1

A Smart Walker to Understand Walking Abilities

Pascal PoupartAssociate Professor

Cheriton School of Computer Science University of Waterloo

Monday, February 15, 2010University of Kentucky

2

Multidisciplinary Team• Computer Science

– Farheen Omar, Mathieu Sinn, Jakub Truskowski, Richard Hu, Allan Caine, Hao Chen, Adam Hartfiel, Adam Fourney, Omar Zia Khan, Jay Black, Richard Mann, Pascal Poupart

• Systems Design Engineering– Samantha Ng, Adel Fakih, John Zelek

• Kinesiology– James Tung, Matt Snyder, Tracy McWhirter, Rhiannon

Rose, Bill McIlroy, Eric Roy,

• Partners– UW Schelgel Research Institute for Aging, Village of

Winston Park, Toronto Rehabilitation Institute, CIHR, NSERC

3

Outline

• Motivation• Smart Walker• User study at the

Village of Winston Park• Behaviour recognition• Limb tracking

4

Disability Statistics• US National Business Service Alliance (July 2006)

5

Disability Statistics• US National Business Service Alliance (July 2006)

Types of disabilities

6

Mobility aids

Encourage walkingIncrease safety

Discourage walking

pp3

Slide 6

pp3 What are the walking aids available to seniors?Pascal Poupart, 11/21/2008

7

Smart Walker

walker devices

+ caregivers

users

Force sensorsLoad sensorsVideo camerasMicrophoneSpeech synthesizerServo-brakesetc.

assist

Smart walker

8

Current Prototype• Designed by James Tung and Bill McIlroy at the

Toronto Rehabilitation Institute

• Monitoring device• Collect information about

– gait (posture, mobility, walking abilities)– walker usage (speed, brakes, load)– environment (obstacles)

• Why?– Detect falls– Monitor users – Ensure safety

9

iWalker (Tung & McIlroy at TRI)

Single-axis load cells (each caster)

Wheel Rotation Counter

(L&R Rear Wheels)

Brake Sensor (x2)

Propulsion forces (x2)

Palm PC with A / D unit

Panoramic hemispherical camera

Battery Unit

Under-Seat Components

2-D accelerometer

Single-axis load cells (each caster)

Wheel Rotation Counter

(L&R Rear Wheels)

Brake Sensor (x2)

Propulsion forces (x2)

Palm PC with A / D unit

Panoramic hemispherical camera

Battery Unit

Under-Seat Components

2-D accelerometervideo camerapointing backward

Video recorder3-D accelerometer

and A / D unitPDA

10

Research questions• How do people use walkers?

– How is balance control affected?– What are the challenges?

• Can we assess walking abilities in a naturalistic environment?– Automated behaviour recognition– Track the 3D pose of lower limbs

• Benefits– Continuous and quantitative monitoring– Improve treatment plans

11

User Study• Village of Winston Park (retirement community)

– 11 walker users (average age: 89) – 10 non-walker users (average age: 83)

• Battery of physical and neuro-psychological tests• Obstacle course

– Straight walking– Turns– Reaching task:

• Press elevator button• Pick up a remote• Open a door

– Go up/down a ramp and curb

12

Prescription of Walkers

• No standardized procedure

• Common clinical measures:– Timed Up and Go (TUG)– Berg Balance Scale (BBS)– Center of pressure variability

13

Timed Up and Go (TUG)• Time to get up, walk 3m, come back and sit

– Normal: < 12 seconds

14

TUG Comparison• Longer time with walker for both groups• Problem: walker appears to decrease

performance

15

Lower Limb Balance Control

16

Upper Limb Balance Control• Center of Pressure (CoP)• Vertical Load (Fz)

17

Center of Pressure Variability

Anterior/PosteriorMedio/Lateral

Hands OFFHands ON

18

Behavioural context and instability triggersInstability triggers:

a: elevator door collisionsb: foot collisions

Behavioural Context:W: straight walkingS: standingR: resting on chair

19

Automated Behaviour Recognition• Segment time by behaviours:

• Manually label behaviours during walking course:– 9 walker users– 8 controls (non-walker users)

1. Not touching walker2. Stop/standing3. Walking forward4. Turning left5. Turning right6. Walking backward

7. Transfer8. Sitting on walker9. Going up ramp10. Going down ramp11. Going up curb12. Going down curb

20

Automated Behaviour Recognition• Problem: infer behaviour from sensor

measurements

• Supervised learning:– Hidden Markov model (HMM)– Conditional random field (CRF)

• Unsupervised learning:– HMM

• Expectation Maximization (EM)• Bayesian Learning

21

Hidden Markov Model (HMM)

• Each sensor:– 50 Hz (50 measurements per second)– 216 values discretized in 20 buckets

• Pr(Oit|Bt) = relative frequency counts

• Pr(Bt+1|Bt) ≈ 0.95 for Bt+1 = Bt

Bt

O1t O2

t O8t…

Bt+1

O1t+1 O2

t+1 O8t+1…

Bt+2

O1t+2 O2

t+2 O8t+2…

22

Conditional Random Field (CRF)

• Features: thresholded average and std– Speed, total load, CoP (m/d, a/p), accel. (x, y, z)– windows of 1, 5, 25 and 50 measurements

• Training: conjugate gradient descent

Bt

F1t F2

t Fnt…

Bt+1

F1t+1 F2

t+1 Fnt+1…

Bt+2

F1t+2 F2

t+2 Fnt+2…

Sensor measurements

23

Behaviour Recognition AccuracyPrediction Accuracy

24

Behaviour Transitions

Precision =# of correct transitions

# of predicted transitionsRecall =

# of correct transitions

# of labelled transitions

TransitionPrecision

TransitionRecall

25

Limb Tracking• Estimate 3D pose• Tapered cylinders

defined by 21 parameters

• HMM: particle filtering– infer pose P from cues Ci

Pt

C1t C2

t Cnt…

Pt+1

C1t+1 C2

t+1 Cnt+1…

Pt+2

C1t+2 C2

t+2 Cnt+2…

26

Appearance Model (Cues)• For each limb segment:

– Texture: histogram of oriented gradients (HOG)– Colour: histogram of colours (RGB)

Ex: histogram of oriented gradients Ex: histogram of colours

27

Uncertainty in Pose Estimate

28

Challenge

Sagittal plane Coronal planeeasy hard

• Motion capture

• Need depth information– Stereo-vision– Time-of-flight camera

29

Conclusion• Smart walker

– Automated behaviour recognition– Limb tracking

• Benefits– Naturalistic gait analysis– Continuous quantitative monitoring

• Next steps– Stereo-vision / time-of-flight camera– Unified model (behaviour recognition and limb tracking)

30

Vision: Robotic Nurse• Navigation assistance

– Obstacle avoidance – Route to some destination

• Personal trainer– Monitor and coach exercises

• Reminder system– Medication, appointments, location

• Alert system– Notify caregiver when in need or fallen

• Conversational companion– Speech recognizer/generator

31

PascalPascal’’s Research Projectss Research Projects• Policy explanation for MDP recommender systems

– Omar Zia Khan (PhD)• Planning as inference for decentralized POMDPs

– Igor Kiselev (PhD)• Bayesian Reinforcement Learning

– Ricardo Salmon (PhD)• Behaviour recognition with a smart walker

– Farheen Omar (PhD), Mathieu Sinn (Postdoc)• Limb tracking with a smart walker

– Richard Hu (MMath)• Automated feature generation in temporal models

– Adam Hartfiel (PhD), Mathieu Sinn (Postdoc)• Voice, Location and Activity Monitoring for Alzheimer

– James Tung (Postdoc)• Unsupervised cluster labelling with HDPs

– Ting Liu (MMath)• Hand-written equation recognition with HDP-HMMs

– Mazen Melibari (MMath)

32

Thank You

Questions?

33

CRF Confusion MatrixNot touching

walkerStanding

/StopWalking Forwards

Turn Left

Turn Right

Walking Backwards

Transfers /Chair

Reaching Tasks

Sitting On Walker

Going Up Ramp

Going Down Ramp

Going Up Curb

Going Down Curb

Predicted Incorrectly

Predicted Correctly

Not touching walker 7916 1265 0 0 0 0 0 0 364 0 0 0 0 1629 7916Standing/Stop 465 44727 1627 914 300 0 0 2483 744 106 17 206 49 6911 44727Walking Forwards 2 2688 60558 2581 2661 0 0 1823 20 578 179 420 78 11030 60558Turn Left 0 1279 4237 10397 523 0 0 238 80 80 19 14 0 6470 10397Turn Right 0 1135 5208 312 7976 0 0 566 14 29 236 65 120 7685 7976Walking Backwards 0 157 230 94 166 0 0 327 0 9 33 29 0 1045 0Transfers/Chair 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0Reaching Tasks 0 5278 1882 911 294 0 0 3876 12 26 8 0 0 8411 3876Sitting On Walker 111 674 67 107 45 0 0 0 70330 0 0 0 0 1004 70330Going Up Ramp 0 154 350 129 8 0 0 52 21 1980 158 137 9 1018 1980Going Down Ramp 0 39 620 0 156 0 0 0 0 0 2517 80 10 905 2517Going Up Curb 0 330 178 20 129 0 0 146 0 85 10 2122 67 965 2122Going Down Curb 0 69 268 0 85 0 0 0 0 0 161 68 821 651 821

Not touching walker

Standing with

Walking Forwards

Turn Left

Turn Right

Walking Backwards

Transfers /Chair

Reaching Tasks

Sitting On Walker

Going Up Ramp

Going Down Ramp

Going Up Curb

Going Down Curb

Predicted Incorrectly

Predicted Correctly

Not touching walker 8514 836 0 0 0 0 0 0 195 0 0 0 0 1031 8514Standing Stop 239 48558 579 431 102 0 0 1352 315 22 17 23 0 3080 48558Walking Forwards 0 1356 67580 959 741 0 0 740 20 16 66 100 10 4008 67580Turn Left 0 710 2191 13531 229 0 0 59 63 52 18 14 0 3336 13531Turn Right 0 702 3342 225 10915 0 0 319 5 15 21 65 52 4746 10915Walking Backwards 0 157 230 94 166 0 0 327 0 9 33 29 0 1045 0Transfers/Chair 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0Reaching Tasks 0 4013 1287 685 185 0 0 6071 12 26 8 0 0 6216 6071Sitting On Walker 71 220 54 107 45 0 0 0 70837 0 0 0 0 497 70837Going Up Ramp 0 87 152 9 8 0 0 52 0 2386 158 137 9 612 2386Going Down Ramp 0 15 216 0 26 0 0 0 0 0 3159 6 0 263 3159Going Up Curb 0 133 37 20 89 0 0 14 0 0 10 2784 0 303 2784Going Down Curb 0 17 168 0 46 0 0 0 0 0 10 19 1212 260 1212

Window with: 0

Window with: 40