how can we make game audio for the rehabilitation of stroke patients

8/4/2019 How Can We Make Game Audio for the Rehabilitation of Stroke Patients

http://slidepdf.com/reader/full/how-can-we-make-game-audio-for-the-rehabilitation-of-stroke-patients 1/25

How Can We Make Game Audio for

the Rehabilitation of Stroke Patients?

***************************Author*

Eri Shiroyama – Audio Designer

[email protected]

Course*

Utrecht School of the Arts, The

Netherlands

Oude Amersfoortseweg 131,

HilversumGame Design and Development

2009/2010

Client*

FOCAL MEDITECH BV

Team Komodo*

Adriaan de Jongh

Sandra da Cruz Martins

Ronald Houtermans

Tim Remmers

Francis Laclé

Jens van de Water

Supervisor*

Lies van Roessel

Arno Kamphuis



1* Abstract These days the number of stroke patients is increasing in the entire world. Therefore

there is a lot of money invested in research and possible treatments.

With the rise of digital media, companies, and hospitals are looking for new ways of

the rehabilitation. One of these ways is by using game technology as a more fun

way to rehabilitate because they can enjoy the training. This makes the training less

painful and more enjoyable. Since the patients are not familiar with video games,

it’s important the game audio guides them through the training, and makes them

immersion to play game.

In this paper, I will explain how we should make game audio for the rehabilitation

game, how we can help for stroke patient by game audio.

Key Words Game audio, Rehabilitation, Stroke patients, Focal, Arm movement, Immersion,

Relax, Sound, Music, ‘Vogels!’, Instrument, Adobe Audition, Unity,



2* Index 1* Abstract

2* Index

3* Introduction

4* what is a Stroke?

4-1* Physical problems

4-2* Mental problems

5* is game audio important for rehabilitation games?

5-1* Why do especially stroke patients need the audio?

5-2* what kind of sounds is suitable for rehabilitations?

6* Project

6-1* ‘Vogel!’

6-2* what did I do?

6-2-1* Audio trailer

6-2-2* Music for ‘Vogel’

6-2-3* Sound for ‘Vogel’

6-2-4* Managing the audio by Unity

6-3* Audio play test

7* How can we apply game audio for their rehabilitation in the

future?

7-1* Sounds for helping physical rehabilitation

7-2* Music for helping mental rehabilitation

8* Conclusion

9* Reference



3* Introduction We worked for ‘FOCAL MEDITECH BV’ which is a company that develops technical

aids for people with disabilities. These aids assist disabled people in living as

independently as possible and help them in participating in society.

They gave us the assignments of making a serious game for stroke patients by using

their instrument called DYNAMIC ARM SUPPORT TOP/HELP (Presentation from Focal,

Introduction to FOCAL Meditech BV, 2 .2010).

Our purpose was making a game to improve the range of their arm movement for

their rehabilitation. Although posture and gait may improve significantly during their

rehabilitation, recovery of arm - hand function lags behind recovery of other

function. Many patients are left with severe and lasting impairment and disabilities.

That’s whyit will be very helpful for a lot of stroke patients.

(Philips research 2007, Philips Stroke Rehabilitation Exerciser)

The motion of the arm is the one of the most important things in our life. We need it

for eating, lifting things, putting on [taking off] clothes and so on. Also the

rehabilitation game can take away a lot of physical work from therapists, who are

then able to work on multiple patients at the same time.

Our team was made up of 7 students including 2 programmers from Utrecht

University; Adriaan de Jongh as Game Design & Project Coördinator, Sandra da

Cruz Martins as 2D Artist, Tim Remmers as 3D Character Artist, Ronald Houtermans as

3D Environment Artist, Francis Laclé as Programmer, Jens van de Water as Game

Programmer, and Eri Shiroyama as Audio Designer.

I designed all of the sound effects and managed the music with the assistance of a

composer student. For this reason I did research on the stroke patients, the game

audio, the interaction between them, and play testing.

In this paper I will explain what kind of geme audio is suitable and helpful for the

stroke patients on the basis of my research.



4* what is a Stroke? Stroke, Cerebral Vascular Accident

(CVA) is an injury which is damaged by

blocked blood vessel in the brain. It

disturbs cerebral function. If it lasts more

that 24 hours, it leads to death.

Depending on the location which was

affected, the limitation in activity of the

patients might be different.

Strokes are the leading cause of severe

long term disability and high social

costs. In the Netherlands, the amounts

of incidents are estimated at 34,500 per

year and prevalence at 190,000. Also

prevalence will increase with 27% in this

decade.

About 95% of the CVA patients are 45

year and older. And 66% is over 65

years old. In most cases the patients

rehabilitate in either a rehabilitation

center or an elderly home depending

on their age. Most of them need

wheelchairs to move around. Because

this limits them, 70% of them have

problems in Activities of Daily Life (ADL)

and social participation. Of the

consequences of strokes 44% are mild,

21% are moderate, 16% are severe, and

19% are very severe.

(Presentation from Focal, What is a

Stroke, 2.2010)



4-1* Physical problems

Over 80% of all stroke patients suffer from unilateral deficits. (Kamper D 2002,pp702-707)

But there are a lot of differences in the level of impairment.

There is also a huge variety in what stroke patients can feel in their paralyzed limbs.

Their limbs can feel warm or cold, tingling (similar to a ‘sleeping limb’), extra sensitive,

no feeling at all or they can feel a continuous pain.

Mostly because of their age, but also because of their brain damage, the stroke

patients often have trouble dealing with dual tasks ( two separate actions that take

place at the same time). For example walking and swinging your arm at the same

time.

4-2* Mental problems In most cases, not only half of the motor functions of the stroke patients are disabled

but also anything else in their body might be partially disabled like their short

memory, their emotions, their ability to take initiative, their special aptitude, and so

on. If someone would lose half of their vision, for instance, they would only see half

of what we see, which does not simply means that only one eye does not work, but

they literally lose half of their vision.

That’s why they often get depressed and panic easily.

Also, Stroke patients often get cognitive deficits. For example, some patients have

trouble even remembering their training schedule while training.

(Research Document, 2. 2010)



5* is game audio importantfor rehabilitation games? What can mere game audio alone do for the stroke patients?

It’s normal if you think it won’t help anything. But I think game audio can help a lot

for their rehabilitation.

I researched about sound therapy for patients to the Japanese hospital called

Nagao Hospital in Fukuoka/Japan. One doctor and one therapist replied that they

use practically some sound therapy for the patients who are damaged their brain

like a Parkinson as alternative medicine. Though I will introduce what kind of sound

therapy might be possible to apply to game audio in chapter 7, in this chapter I will

explain the importance of audio for game.

5-1* Why do especially stroke patients need the audio? Audio can move the feeling of people like excited, scared, relax, and calm and so

on. So the good game always has the fine sounds and music for making people

being absorbed. But actually sophisticated audio is much more necessary for the

stroke patients for playing the game.

First of the reason, some of them have the ear problem and are much sensitive of

the sounds compare with healthy people.

Second of the reason, in most case they have the problem of their concentration,

so we have to lead them to immersion for playing game by audio. Also

rehabilitation is quite painful and bitter for them. They need to forget the bitterness

as much as possible in the game.

Thirdly, some of them have the eye problem. Therefore we need to make them feel

the circumstances and the atmosphere of game by sound effect and environment

sound.

The last of the reason, according to a research they can understand much easier of

the way of playing game when we explain it by not only letters but also speech.

(Philips research 2007, Philips Stroke Rehabilitation Exerciser)



5-2* what kind of sounds is suitable for rehabilitations?

For the stroke patient, how can we be attentive to when we make the audio?I heard about the experience of the family of a stroke patient when he played the

rehabilitation game. He had a game for the Wii, but he always didn’t want to play it

because of the endless noise of electronic music from the game, and it was hard to

understand the game because his brain got tired easily. Eventually he couldn’t

continue his rehabilitation by using the game.

Therefore when I edit the sounds I always keep the following 5 things in mind

- Not too stimulating,

- Not negative

- Realistic with a touch of fantasy

- Immersion

- Relaxing and calm.

I will explain why and how can I make them in Chapter 6-2-2.



6* Team Komodo Project Our game named ‘Vogels!’ was made for the improvement of the arm movement

of disabled people. The game is a 3D game which is effective because the patients

mind is not focused on the actual training anymore. Instead, it is focused on the

challenge of the fantasy game, as assigned by FOCAL.

We worked at it for 40 hours every week from February to May.

Our team is composed of 7 students. The Game Design & Project Coördinator is

Adriaan de Jongh who led our project, made sure that we proceeded on time, led

each part, contacted all of the other people like a client, helped with other parts,

and programmed the game using Unity. He taught me how to use Unity and how to

advance my work. Our 2D Artist is Sandra da Cruz Martins who created the general

image of the game by designing the 2D graphics and she also designed the

concept-art of characters. Our 3D Character Artist is Tim Remmers who created the

3D characters and the elements of environment, and also created the motion of all

of the elements. Our 3D Environment Artist is Ronald Houtermans who created the

general environment of the game. Our programmers are Francis Laclé and Jens

van de Water who are from Utrecht University.

They made the code for connecting the instrument of arm movement with the

game and programmed the game in Unity.

My role is doing the Audio Design for the game. I designed the character’s voice

sound , the character’s motion sound , the effect sound, and environment sound. I

also took care of the music by asking a composer to make game music that was

suitable for our game and stroke patients. Then I put the audio in the game by using

Unity and adjusted their effects and their volume.

I will explain ‘Vogels’ on the following contexts (7-1*) and my study and the play test

(6-2*, 6-3*).



6-1* ‘Vogels!’

Playing ‘Vogels!’ is very simple. You make the main

character move up, down, right, and left by using the

instrument for arm movement instrument. By doing that,

you help the main character to make contact with the

other birds. When you touch the other bird with your

hand, you get a point. The objective is to get as many

points as possible.

The level of the game can be changed

dynamically according to the level of

impairments of the patients. This is

because depending on the person the

limitation of the moment of the arm is

totally different. Also the game needs

to keep pace with their speed of

recovery.

Therefore in this game you can

calibrate the range of arm movement.

First you need to measure the neutral

position of their arm while keeping in

mind that you should be able to move

right and left after this. Next you set their

limitation of range by moving left, right

and up as much as you can. In this case

it should be much more effective if their

therapist helps them with their aim of

arm range when you calibrate.

Also, in this game the speed of the main

character can be changed. For

instance, the patients are given more

time for moving their arm and flying to

the other birds when you set the speed

to slow.



Also our explanation of the game is written in Dutch, not in English. It’s more

comfortable and useful for the patients because a lot of them are over 65.

‘Vogels!’ has three environments which are Sky, Land, and Water. In each

environment you can see the different characters and different scenery and

different music. By these changes, you can get a new feeling in every world so you

will not (won’t) get bored.

6-2* what did I do? I worked on the audio of ‘Vogels’ in this project. I was an absolute beginner at

game audio when we started the project. So I could learn many things about

audio and game software in the project thanks to the help of a lot of people. I

especially learned a lot from Mr. Richard van Tol who is a game audio designer and

teacher in HKU.

6-2-1* Audio mood board and Audio trailerBefore creating the sounds I designed an audio mood board and an audio trailer in

order to convey the mood of the audio to our client, the composer student, the

other project members and so on. At first I collected a lot of sounds through the

Internet. After that I made the sound fit with the character image table which was

made by our 3d designer. I think it is very important for the game audio that we

always keep contact with our graphic designer; for example we should always ask

about the process of their design. Then you can make a mood board by putting

your image sounds even if you only have a rough sketch from your designer.

(Audio Mood board)



Also I created the audio trailer which was composed with some sounds, some music

and some rough drawings which were made by our 2D designer. I think they workedvery well for designing the music and the sounds in context. It helped me to explain

my game image easily to other project members, so we could synchronize the

difference between the designers and me, which was necessary because they also

had their own idea about the character’s voice. It also made it really easy for me to

communicate my idea about the music to the composer student.

You can see my audio trailer at the following link. http://vimeo.com/10725316

6-2-2* How did I make sound for ‘Vogel!’? I designed the sounds using Adobe Audition, which can record, edit, and creat a lot

of effects.

When I edit the sounds I always keep the following 5 things in mind. (5-2 )

- Not too stimulating,

- Not negative

- Realistic with a touch of fantasy

- Immersion

- Relaxing and calm.

Too many high tones and keys are big problem for the stroke patients because they

might stimulate them too much and surprise them easily. Also I try to erase the

background noises as much as possible. Even if you almost can’t hear these noises,

they might make the patients sensitive, because some stroke patients have ear

problems.

.

The quality of the sounds is also really important for their emersion and the game’s

realism. Therefore we had to be careful of negative change by compressing sound

files. But we didn’t want to end up having big files for only sound because many 3D

graphics and the motion already made it huge.

Keeping these both in consideration, I decided the quality of the sounds should be

standardized to 22050Hz and 16 bit.

I will explain the basics of how I edited the sounds for the patients.



(1), Graphic Equalizer

I used the graphic equalizer for cutting the high frequency and the low frequency.

For example, if you record some bird’s voices, in most case it’s including too

high-pitched voice. We don’t need these sounds for them. That’s way I cut these

high frequencies.

Also according to research, a too low frequency might pose a hazard to health l ike

headaches, even to healthy people. ( Infrasound: http://en.wikipedia.org/wiki/Infrasound )

(2), Delete Noise

(Noise Reduction)



(Spectral Frequency Display)

Noise always bothers us. Almost all sound files contain several place with noise. I

often used the Noise Reduction and Spectral Frequency Display for that. Noise

Reduction can delete the constant noise which expands all over in the sound files.

For the partial noises, you can erase it using Spectral Frequency Display.

(3)Time Stretch and Pitch shift

The real bird’s voices which were recorded were sometimes too stimulating and too

high key and tone. Therefore I changed the pitch and tempo to a more calm voice

by using Stretch.

(4)Echo and Delay



In odder to create feeling of realism and immersion, I gave an echo to the sky birds,

because their twitter should sound faraway sound. Also all of sounds in the cave

need more echo as well because they rebound each other.



6-2-3* Music for ‘Vogel!’ Music is also one of the most important things in our game. I asked the music

composer student that basically we need relaxing music for the patients becausethey need to feel at ease and get comfortable on their rehabilitation without the

beat for of the music over-stimulating them. The beat is normally necessity for

making people excited, but for the patients sometimes it makes too much.

In order to convey our image correctly to the composer I made a music design

document which contains my design image, the format (mp3, wav...), the rate (Hz)

the bit depth (bit) and so on.

This is part of the music design document.

■■■■ Design image for music

Land music (3minutes):

calm, comfortable, slow, peaceful

Sample: land1.mp3, land2.mp3

Water music (3minutes):

relax, clear, refresh

Sample: water1.mp3, water2.mp3

Sky music (3minutes):

freedom, shine, bright

Sample: sky1.mp3, sky2.mp3

■■■■ Technology

Platform - Windows

Sound engine - Unity

Editing tool - Adobe Audition

Format – ○wav △mp3

Rate – 22050Hz

Bit depth - 16bit

Cannels – stereo

But it was still difficult to explain the music image in words. Therefore I include a

music sample for each environment with the music design document.

After that it is also very important we often contacted each other. The composer

student gave me each music demo before he designed it, then we (including our

team members) discussed whether this music was correct or not.



6-2-4* Managing the audio by UnityWe used game Unity to integrate all of the elements for ‘Vogels!’. The following

figure is a picture of the global world of ‘Vogels’. You can see a lot of audio in there.It is essential to put many environment sounds to express a realistic world. Also, lots of

environment sounds will help the patients to feel immersion and fascination.

To put the sounds in Unity, we have to take care of not only their volume but also the

range of the effects (from how far we can hear each sound). For example, the sky

birds make flapping sounds. When they get closer to you, you can hear this flapping

sound gradually. Then it is important to adjust from when till when you can hear it. If

it’s too long, these sounds will become annoying. But if it too short, it lacks realism.

Especially for patients who have weak sight, these sounds let them know that birds

are approaching you.

Also I designed multiple sound effects for the characters, even if they are totally the

same characters and same motions. You can hear the sounds variation when you

approach the birds. In this way multiple sound effects ads realism and avoids

making the patients lose interest.



6-3* Audio play test

we had a play test for audio at 20th

May in an elderly home in Hilversum.

Method8 people played the game while using headphones to listen to the music and

sounds. The first 4 people were shown the game for about 5 minutes because they

weren't allowed to play it. Just watch and listen. The other 4 people were allowed to

play the game while listening to the music.

All of them were given these questions: ’what do you like in the audio? And, what

don’t you like in the audio?’

Result1: Female (58)

What do you like in the audio: The water part was the most clear to her, she liked the waterfalls

What don't you like in the audio: She couldn't recognize the land part

Comments: The world and it's audio gave her a realistic feeling of the world, and it was calming.

2: Female (72)

What do you like in the audio: The music was very clear and suited the environment

What don't you like in the audio: The sound effects were too noisy and chaotic in her opinion

Comments: She couldn't really pay attention to the game, because of the noise of the

environment sounds. She did have some hearing problems.

3: Female (64)

What do you like in the audio: Beautiful music! It suited the environment very well,



What don't you like in the audio: She really liked all the audio in it, even the strange bird sounds.

Comments: She was able to name almost all of the sounds she heard during the playtime.

4: Female (55)

What do you like in the audio: The music and the bird sounds were good together

What don't you like in the audio: She thought some of the birds sounds were a little bit strange

and not very realistic

Comments: The game gave her calming experience.

5: Female (70)

What do you like in the audio: She really liked the animation of the bird together with the music,

the rhythm was good!

What don't you like in the audio: Some of the bird sounds sounded a bit ine panic. That's why she

was avoiding the birds.

Comments: The music, together with the game, really gave her a soothing feeling. She mentioned

that she often listen to this kind of music to relax.

6: Female (67)

What do you like in the audio: She liked the fact that the music/sounds adapted to the

environment, she was able to hear in what part she was flying.

What don't you like in the audio: The sounds of the birds she had to catch sounded like you were

hurting them. She also thought the main character was making that sound instead of the other

birds.

Comments: She thought she was flying in the Grand Canyon. The audio really blended in with the

game.

7: Male (80)

What do you like in the audio: The music, which was like the music on the Discovery Channel

documentaries .

What don't you like in the audio: The flapping sound was a bit annoying, sometimes.

Comments: The environment sound felt very natural and realistic. He could recognize the

landscapes in the music a bit.

8: Female (50)

What do you like in the audio: The realistic feel of the waterfall, she could really feel it coming.

What don't you like in the audio: Although the waterfall felt good, it was a bit too loud in her



opinion. The music is fading away there.

Comments: She was getting very relaxed by the music.

The people who didn't play the game and only watched/listened were able to mention more

environment sounds than the people who were playing the game, they only recognized the

waterfall.

Conclusion and SolutionA lot of people like the music, and some of them said felt relaxed.

But some people think a sky bird sounds panicking and like you were attacking and

wounding the other birds. That‘s why they avoid the birds instead of coming in

contact with them. This was big problem for me. Actually their sounds were

sounding a little bit negative and using too high tones. In order to solve this

problem, first I tried to pull down the pitch, but still the sound was too negative.

Therefore I changed the way in which the pitch was changed.

You can see the shape of the changed the pitch in the figure above. The last part of

the sound file is in a higher pitch than opening parts of it. As a result the tone of

sounds became higher, and sound more positive.



7*

How can we develop game audio for

their rehabilitation in the future? I already explain you that in some hospital and rehabilitation center, they do use

sound therapy in practical for the patients who have the damage in their brain like

a Parkinson and cerebral palsy. They classified the sound therapy as the stimulation

from rhythm and stimulation from music. The rhythm stimulation might be effective

for their physical motion. The music stimulation should affect their mental problem.

Regrettably I couldn’t incorporate the system of these researches to ‘Vogels!’, but

these will be very useful for developing rehabilitation game in future.

I will introduce you what kind of sound could apply to game audio for the

rehabilitation of stroke patients.

7-1* Sounds for helping physical rehabilitationThe surprising thing is that some sounds stimulate and help the physical motion of the

patients on their rehabilitation. It has been proven in a medical dissertation.

(Mutsumi Abiru, 2007, Neurologic Music therapy for Rehabilitation for Stroke patients)

Rhythmic Auditory Stimulation (RES)

In this paper, there are two ways for improving physical motion. One of them is

called Rhythmic auditory stimulation (RAS), which facilitates the rehabilitation of

movements using external and rhythmical stimulation. The sounds are used to

decide the tempo and speed in which is being moved.



RAS was known to be effective against a handicap of the central nervous system.

According to other research, it helps with Parkinson’s disease, stroke, Spinal Cord

Injury and so on.

Patterned sensory enhancement (PSE)

The other way is Patterned sensory enhancement (PSE), which improves motor

dysfunction of stroke patients, using rhythmic, melodic, harmonic and dynamic

elements of music to provide temporal, spatial and kinematic cues.

It affects upward and downward motion, and enables stroke patients to perform a

wide range of motions like for instance a squat. In the case of a squat, if we need

the patients to reach lower positions, we can make the sound go lower and lower

gradually and we can also make it go slower.

I think we can apply these researches to a rehabilitation game quite easily. For

example we can program these sound effects to coincide with the motion of hte

arm.



7-2* Music for helping mental rehabilitation Music has the power to stimulate people easily, like to excite and heal and make

people feel relaxed.I found it interesting that their favorite music works very well for stroke patients not

only for use with physical motion but also especially for the recovery of their

memory.

This was investigated at Helsinki University in Finland. The group of 60 stroke patients

was the target for this research, and all of them had problem with their memory and

trouble with keeping focused.

Everyone was divided between group1 (listening to their favorite music at least 2

hours per day), and group2 (listening to someone reading a book for them), and

group3 (nothing). This figure is the result 3 months of research.

Improvement probability of their memory after 3 months

Group1 60%

Group2 29%

Group3 18%

Also group1 became less depressed than any other group, and the music helped

them to stay forcused.

The treatment is more effective if they include lyrics which they can understand the

meaning and language of.

(http://www.afpbb.com/article/life-culture/health/2353313/2659976)

It should be possible to include their favorite music in game, like for example games

of based on music like Guitar Hero, Dance Dance Revolution and so on.



8* Conclusion

In the introduction I talked about ‘what is a stroke?’, ‘why do we need audio in

games for stroke patients?’, ‘what is ‘Vogels!’?’, ‘how did I make the audio?’ and,

‘how can we develop the game audio of rehabilitation in the future?’

In conclusion, I think game audio is a very effective tool for the rehabilitation game

of the stroke patients, and that’s why we have to be attentive of making good

audio for the stroke patients. This time I took care of the following five things when I

designed the audio; not too stimulating, not negative, realistic with a touch of

fantasy, Immersive, relaxing and calm.

We got a really good response to the game ‘Vogels!’ from the patients.

Also now I know we can help their rehabilitation by implementing good game

audio. Probably we can apply game audio not only to the stroke patients but also

to the patients of other several diseases.

It was the absolute first time for me to make game audio for stroke patients, but I’m

satisfied with our game and the audio. I hope I can continue the research and I will

be able to make the more game for rehabilitation. I appreciate that a lot of people

helped me in making the audio and this paper. Also I’m glad I had a good team

and a nice client.



9* Reference * Paper*Kamper D | Alterations in Reaching after Stroke and Their Relation to Movement

Direction and Impairment Severity. Arch Phys Med Rehabil, o1(83), 2002, pp 702-707

Philips research 2007| Philips Stroke Rehabilitation Exerciser

Mutsumi Abiru, Yutaka Kikuchi, Kouji Tokita, Yoshiko Mihara, Mikio Fujimoto, Ban

Mihara, 2007| Neurologic Music therapy for Rehabilitation for Stroke patients –

Prospects of Music therapy for Gait Disturbance -

Adriaan de Jongh |Komodo Research Document, 2. 2010

*Web site*Infrasound| http://en.wikipedia.org/wiki/Infrasound

脳梗塞患者の回復には好きな音楽が効果的、フィンランドの研究結果|

BB News. 20. Febulary. 2008.

http://www.afpbb.com/article/life-culture/health/2353313/2659976

* Other*Presentation from Focal Meditech BV(2.2010) | Focal gave us the presentation for

introduction.

how can we make game audio for the rehabilitation of stroke patients

Documents