Technology & Gamesfor Stroke RehabilitationJames BurkeSchool of Computing &Information EngineeringBurke-J5@email.ulster.ac.uk

University of Ulster, Coleraine

Overview•THE PROBLEM AREA:

▫Stroke is a leading cause of severe physical disability.▫Effective rehabilitation must be early, intensive and

repetitive.▫Traditional tasks often mundane - difficult to maintain

patient motivation.

•THE PROPOSED SOLUTION:▫Games and novel technology may be effective in

optimising engagement.▫We have developed several games upper limb post-

stroke rehabilitation which use novel technology.

Games + Rehabilitation

•MEANINGFUL PLAY▫Effective feedback.▫Conservative response to

failure - reward engagement.

•CHALLENGE▫Challenge should match

player’s abilities.▫Maintains engagement.

Inputs choices

Player GameMeaningful Meaningful playplayFeedback

“Designers of rehabilitation tasks can benefit from examining the formulas that commercial game developers use…” [Rizzo, 2005]

Webcam Games forUpper-Limb Rehabilitation

Webcam Games•Simple 2D games which use a webcam image as player

input.•Games store patient profile.

▫Profile stores reaching ability, play-time, scores, etc.•Played with coloured gloves or any single coloured

object.•No attaching of wires required, untethered movement.•Games can be played standing up or seated.•Potential for home rehabilitation.•No expertise required.

Webcam GamesArchitecture

Player Movement

Wearing Gloves

Image Captured by

USB Webcam

Image Processed –

Pixels identified for each marker

Output to Monitor / Projector

RGB Colour Segmentation• Markers identified through

calibration.• Each frame, each pixel in the camera

image is examined:

rrrrr µ - µ tpt

For each pixel p (pr, pg, pb):

ggggg µ - µ tpt bbbbb µ - µ tpt

Where µ = mean colour vector t = threshold scale value σ = standard deviation

R

G

B

0

µtσ

L

R

Webcam Games• SINGLE ARM EXERCISE:

▫ “Rabbit Chase” Player must catch a rabbit as it peers out of

holes on the screen.▫ “Bubble Trouble”

Player must burst bubbles as they float around the screen.

• TWO ARM EXERCISE:▫ “Arrow Attack”

Player must touch corresponding arrows with both hands simultaneously as they enter boxes.

▫ “Bubble Trouble” (two handed version) Bubbles colour coded and show arrow.

Webcam Games• Initial reach calibration:

▫Player must roll each ball as far as they possibly can.

▫Allows system to determine range of movement and position game elements accordingly.

•Additional tools:▫Camera calibration.▫Profile & data log analyser.

View playing time and movement trends.

Webcam Games DemoVideo: http://www.vimeo.com/4279126

Design Principles•Simple game concepts.•Effective feedback:

▫Sound effects, hit & miss messages, colour indicators, particle effects, timer bar, score graph.

•Clear, simple, friendly user interface.•Manual or automatic difficulty

adjustment.▫Help maintain suitable level of challenge

for level of ability.•No need for computer literacy.

Wii Controller

Wii Remote Features•Face & back buttons (1)• Infra-red camera (2)

▫Uses sensor bar for position•Accelerometer

▫3 degrees of freedom•Vibration feedback•Speaker (3)• LED indicators (4)•Expansion port (5)

▫e.g. Nunchuck, classic controlle

(1)

(5)

(2)

(3)

(4)

Infrared Sensing

• Camera at tip of controller:▫PixArt optical sensor.

• Sensor bar with LED clusters on each side.▫Clusters seen by Wii controller as two bright dots.▫Camera can detect 4 clusters in total.

• Works up to 5 meters.• Allows for depth as well as (x,y) 2D plane:

▫Determined by distance between sensor bar dots.• Sensor bar powered by Wii or USB/battery.

Accelerometer•ADXL330 accelerometer.•Senses linear acceleration

along 3 fixed axis.•Can measure pitch &

roll angles.•Yaw determined

by optical sensor.•Accelerometer can not

determine position in 3D space alone. [Battersby, 2007]

Wii Remote Application•Wii being used with existing games

(e.g. Wii Sports) in clinics and groups.•Optical tracking:

▫Immersive environments. [Murgia, 2008]

▫Virtual cane for the blind. [Evett, 2008]

▫Head tracking. [Chung-Lee, 2007]

•Potential for arm and wrist rehabilitation.▫Accuracy limitations?

•Ongoing studies.

Johnny Chung LeeWii “Head Tracking” Video

Wii Vibraphone Application•Application developed as test

for Wii controller onPC (Windows).

•Wii controller interfaced via Bluetooth.

•Virtual vibraphone instrument allowing for recording and playing tunes.

•Potential for wrist rehabilitation.

Wiibraphone Demo

Conclusions

•Games have potential to engage and motivate during rehabilitation.

•No need for computer or video game experience.•Design games which are friendly for all users.•New innovative controllers at a low price may offer

intuitive control mechanisms suitable for rehabilitation.

• Low-cost off-the-shelf hardware may offer an opportunity for home rehabilitation in addition to traditional therapy.

References• Battersby S. (2008) The Nintendo Wii controller as an adaptive assistive device – a technical

report. http://www.ics.heacademy.ac.uk/events/presentations/670_SJBmiddlesbrough2.doc

• Burke J. et al. (2009) Serious Games for Upper Limb Rehabilitation Following Stroke. IEEE International Conference in Games and Virtual Worlds for Serious Applications (VS Games).

• Chung-Lee J. (2007) Head Tracking for Desktop VR Displays using the WiiRemote. http://www.youtube.com/watch?v=Jd3-eiid-Uw

• Evett L. et al. (2008) Accessible virtual environments for people who are blind – creating an intelligent virtual cane using the Nintendo Wii controller. Proc. 7th ICDVRAT with ArtAbilitation, Maia, Portugal, 2008.

• Murgia A. et al. (2008) Low-cost optical tracking for immersive collaboration in the CAVE using the Wii Remote. Proc. 7th ICDVRAT with ArtAbilitation, Maia, Portugal, 2008.

• Rizzo A., Kim G. J. (2005) A SWOT Analysis of the Field of Virtual Reality Rehabilitation and Therapy. Presence: Teleoperators and Virtual Environments 14: 119-146.