Games and Assistive Technologies for Rehabilitation Mid­project Report Fall Semester 2015

­Full Report­

By: Adam Hicks Bryce Coulson

Christian Rodriguez Conor Hart

Department of Electrical and Computer Engineering Colorado State University Fort Collins, CO, 80523

Project advisor: Sudeep Pasricha

Approved by: Sudeep Pasricha

ABSTRACT Strokes affect nearly 800,000 Americans annually [1], and cerebral palsy (CP) affects 1

in 323 children [2]. Diseases such as these are known to cause upper limb impairments that many people struggle with for years. To help alleviate some of the issues brought upon by these debilitating conditions, physical rehabilitation is needed. However, the rehab process can be quite tedious which can potentially lead to patients not following through with it. This results in a diminished quality of life compared to what they could have. The goal of this project is to help improve the process of rehab and make it a more enjoyable experience for patients. This was an idea that was created four years ago that involved a collaboration of computer and electrical engineering students along with occupational therapy students; the goal was to use their skills to create an interface for these patients to perform their rehab in a new and entertaining environment.

GATOR is a project focused on providing new types of rehabilitation therapy for patients in need of upper limb rehabilitation. GATOR features games and activities that utilize the Leap Motion, Myo Armband, and Kinect v2. The games provide progression incentives as well as an enjoyable experience for the patient. Developed with JavaScript, HTML5, torque script, Construct 2, and Unity 3D, the suite of games utilize both 2D and 3D perspectives of play to give patients the ability to complete their therapeutic exercises in a variety of different ways. The Leap Motion controller is able to track the user's hand movement in a 2D plane while also interpreting various hand gestures. The Myo Armband (Myo for short) is also capable of tracking 2D movement as well as interpreting muscle electrical activity to determine which muscles were used, how frequently they were used, and to what extent the user was exerting themselves. Finally, the Kinect v2 can construct a 3D digital skeletal frame of the user that can be used in any manner of 3D games.

Our initial findings have shown promising results for further development. While we have not had the ability to fully test the Kinect in a 3D space, the 2D games we have created have shown encouraging results. We have tested these games with the Myo, which has worked successfully, and we were able to extract the muscle electrical activity data using a script we wrote. We have also managed to begin incorporating these games into the website and we hope to have them fully implemented very soon. These games should be compatible with the Leap Motion, Myo, and Kinect v2. We have tested these games with all but the Leap, though that will require a plugin. The 3D games are in the building/testing phase and we will soon be able to test the code we have compiled. Unfortunately, due to software constraints, we were not able to test these Kinect v2 games out as we required software that we only have recently acquired. We hope to begin developing fully fledged 3D games within the coming weeks.

Page 1

TABLE OF CONTENTS Title Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 List of Tables and Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 2. State of GATOR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

2.1 History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Initial State. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 2.3 Previous Groups Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3. Summary of Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.1 Research. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 3.2 Leap Motion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 3.3 Myo Armband. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.4 Kinect V2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.5 Games. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.6 Website. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.7 Ethical Concerns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.8 Potential Market, Manufacturability and Marketability. . . . . . . . . . . . . . . . . . . . . . . 15

4. Conclusions and Future Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Appendix A – Abbreviations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Appendix B – Budget. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Appendix C – Project Plan Evolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Appendix D – Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Page 2

LIST OF TABLES AND FIGURES Table 1 ­ Previous Teams Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Table 2 ­ Project Plan Evolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Figure 1 ­ Current Website Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Figure 2 ­ Leap Motion Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Figure 3 ­ 3D Hand Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Figure 4 ­ Myo Armband . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Figure 5 ­ Myo Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Figure 6 ­ Kinect V2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 Figure 7 ­ Kinect Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 Figure 8 ­ Flappy Dolphin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Figure 9 ­ Trespass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Figure 10 ­ Cowboy Runner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

Page 3

1. Introduction Physical therapy is a term that is often used to describe the treatment of a disease or an

injury of the muscles or joints. This process itself, however, is not as straightforward as the definition. Physical therapy can be a grueling process depending on the severity of the patients impairment. These ailments are commonly caused by individuals either experiencing a traumatic event or being innately born with it such as a stroke, cerebral palsy, muscular dystrophy, or other physically debilitating diseases. Whether they are trying to cope or overcome these limitations, patients fight an uphill battle from the start. Physical therapy’s primary goal is to assist these individuals in regaining the mobility and quality of life that they once had.

Most physical therapy programs involve developing short and long term goals for patients that vary depending on the patient’s capabilities. In general, it consists of a series of exercises and routines that are practiced daily so that over time the affected muscle(s) can recover to their normal function. This process can last from weeks, months, or even years depending on the severity of the impairment. For some individuals the practices developed while in therapy are something that they do for their entire lives. Most of these physical therapy routines involve a series of motions and gestures that help maintain and improve the usability of those targeted muscles. The primary goal of GATOR is to aid in this process of therapy by providing a different medium in which patients can complete their therapy.

GATOR is Games and Assistive Technologies for Rehabilitation and by this we simply mean that we develop games that assist patients undergoing physical therapy treatment. These games are designed to help improve the therapy experience for the patients by providing a fun and engaging alternative to conventional techniques. At GATOR’s current state we are mostly focused on the Leap Motion, Myo, and primarily the Kinect.

The Leap has been a staple of GATOR for three years now. It is designed to be cheap and portable while still providing enough functionality to allows users to utilize their hands in a 3D space (Chapter 3.2). This portability allowed for a simplistic design and the development of web­based applications. Since most physical therapy that GATOR focuses on is upper limb rehabilitation, the Leap was a very suitable candidate. Its small size allowed it to be placed close to the user which helped reduced interference from outside factors. One main flaw the Leap has was a lack of precision. This was due to our need to have it placed above the user’s hands in order to make it easier for patients who lack the stamina to keep their arms in the air for extended periods of time. It was because of this lack of precision that caused us to move on to alternative hardware, one such piece being the Myo.

When it came to the games themselves the Myo was not chosen because of its game functionality. In an effort to expand upon the data being given to patients and therapists, we wanted to be able to show patients what they were doing and how well they were doing it. The Myo tracks muscle electrical activity and this is done using eight EMG that it has that can be placed upon either bicep (Chapter 3.3). The data from the Myo could then be extrapolated and displayed in either a table or timeline format. The Myo is, however, capable of playing 2D based games. Due to its ability to sense muscle movement, we were able to enable the Myo to act as the mouse. Because most of the leap games act in this manner, we plan on expanding upon this to the point of allowing us to integrate it with current Leap based games.

At the heart of our current game development lies the Kinect v2. The Kinect v2 is capable of rendering the user in a 3D space which allows us to map in­game actions to specific movements and gestures (Chapter 3.4). It features infrared sensors and a 1080p camera allows

Page 4

for a HD reading of the user as well as encapsulating the surrounding environment in full 3D. It is because of this high end technology that the Kinect v2 is a viable option for GATOR. The ability to map any motions of the user into various game functions opens up a wide array of options in terms of customization and accessibility for the users. This feature can also be applied to 2D games as the Kinect v2 is capable of simple tasks as well such as mapping arm or hand movement to mouse movement. Whether it be 2D or 3D games being developed, GATOR’s goal is still focused on making sure that we create fun and engaging games.

So far much of GATOR’s current games library consist of Leap games. These were developed by previous groups and provide an enjoyable experience for the user. The feedback we received from patients was that these games had a lack of progression and polish. It is because of this feedback that our current games are more in­depth than those previous. Games such as “Flappy Dolphin” and “Trespasser” (Chapter 3.5), while small in scope, play very smoothly and present the player with a sense of accomplishment that we aim to deliver. Both of these games have been created to play with the Leap, Myo, and Kinect as they are all 2D based with mouse control support. Our more recent endeavours have been creating games that are specifically tailored to be played with the Kinect v2. This proved to be challenging at first as the tools we currently used for development were not capable of creating games oriented for the Kinect v2­­that is why we have recently settled on using Unity 3D as our engine of choice for game development. Once the games have been created and are ready to be tested we hope to bring all of them together using the GATOR website.

At the helm of GATOR is the website and infrastructure that is host to the games and patient records. This website was developed using Python and JavaScript and is designed to be able to be compatible with any HTML5 application. We are constantly improving the look and functionality of the website to make it more accessible and compatible with the games we develop. The goal is to have the website support games for the Leap, Myo, and Kinect v2. While currently the website can supply patients and therapists with feedback on the patient's progress with the Leap games, we are looking into integrating the Myo data in a way that is beneficial for both.

Page 5

2. State of GATOR 2.1 History

Project GATOR has been a five year continuation project run by an electrical and computer engineering group supervised by Dr. Sudeep Pasricha and Dr. Matt Malcolm. The goal of the project has been to develop games and adapt them for rehabilitation purposes for occupational therapists to uses as a tool for physical therapy on patients who have suffered traumatic brain injuries. Project GATOR is meant to be a more engaging form of rehabilitation to assist the patients in recovering from their injuries and to also provide information for the occupational therapists helping them.

The project began with rehabilitation in mind, using object tracking via a digital camera to track a patient's movement. A physical object would be viewed by the camera and the motion capture by the camera would then translate into motion captured in the game.[5] This motion capture was used to develop games such as a “Whack­a­Mole” game and a “Pong” clone. These games proved to be successful with patient use; the patients would use their affected limbs to control the game based on supervised proper rehabilitation movements and it was determined that there was a future in this project and that it would continue the following year.

The second iteration of project GATOR was a transition from a digital camera to smartphones and tablets.[6] The decision to switch was made due to the increasing popularity and accessibility of these devices. Unfortunately, the decision was made that the technology wasn’t currently available for the type of game development that was required for project GATOR. It was decided that the next iteration would revert back to a camera based control.

The third iteration of the project was to incorporate two different motion tracking systems into the project and determine which would be a better technology to use in the future. One of the technologies, the Kinect v1 was determined to be too difficult to utilize and the decision was made to use the Leap Motion controller[7], the second of the two technologies. The Leap Motion’s strength is the ability to track the nads of the patients which allowed games to be developed to meet the rehabilitation requirements. The third iteration group developed a number of games working with the occupational therapists and were then able to use those games for rehabilitation on active patients.

The fourth iteration of the project continued with the third iteration to develop a number of games as well as improve some of the existing games. In total they worked on 12 different games including a number of games to be used as rehabilitation tools for the occupational therapists.[8] They also continued the development of the website and data gathering tools from the games for the occupational therapists to use to determine the success of each patient's results. The fourth iteration also began looking at a mind controlled smart home automation, which would split off to form a separate senior design project. Because of the success of the fourth iteration, it was determined that the project would receive another group of students 2.2 Initial State

The initial state of the project was a fully working leap motion based game library with a website that hosts all of the games and acts as a portal for the patients and the occupational therapists. It was clear that the previous group had put in an immense amount of time and energy into their games to try and make the best rehabilitation experience for the patients who

Page 6

had suffered from traumatic brain injury. The table below lists all of the games that the previous team worked on: Table 1 Previous Teams Work [8] Name Movement Type Benefits Descriptions

Alien Invaders Left and Right (dodging hand must keep moving entire time)

Increased range of motion. Improved hand eye coordination. Controlled movement.

Alien Invaders was modeled after space invaders. The gameplay is very similar.

Break Out Left and Right ( ball tracking) Increased range of motion. Critical decision making. Improved hand eye coordination.

Break out is brick breaker style of game. The game features many power ups which forces the user to change their strategy as they play.

Dolphin Run Up and Down Triceps Stimulation. Increased endurance.

Dolphin Run is similar to flappy bird, the user tries to get the dolphin past as many obstacles as possible.

Fruit Viking Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based.

Improved hand eye coordination. Increased range of motion, users have to reach at areas they typically do not access in their field. Vision field improvement

User has to cut as much fruit as possible.

Gestures Pitch and Yaw. Closed Fist. Full palm.

Increased hand strength. Improved endurance. Hand eye coordination

User matches their hand with the oncoming gesture.

Leapfrog Left and Right (controlled movement must move to exact position)

Increased hand eye coordination. Improves Focus and critical decision making.

User controls a frog and must guide it up by hopping to different Lily pads.

Pong Push and Pull. (controlled movement must move to exact position)

Increased hand eye coordination. Increased range of motion. Triceps stimulation.

Classic Pong

Maze Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based.

Improved hand eye coordination. Increased range of motion, users have to reach at areas they typically do not access in their field.

User has to perform the movements that are required to progress through the mazes.

Page 7

Meteors Left and Right. (controlled movements must dodge meteors, no sweeping) Closing fist.

Increased range of motion. Improved hand eye coordination. Controlled movement. Vision field improvement

Meteors has the user controlling a spaceship that‟s primary goal was to dodge the incoming meteors.

Pirates Cove Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based.

Improved hand eye coordination. Increased range of motion, users have to reach at areas they typically do not access in their field. Vision field improvement

User controls a pirate and collects coins that are dispersed in the ocean based on a recurring pattern.

Water Drops Left and Right. Improved hand eye coordination. Increased range of motion, vision field improvement.

Water Drops has the user holding a virtual cup. With the cup the user tries to collect as many water drops as possible.

Whack­a­mole Left and right. Push and pull. Diagonal sweeping. Circular Movements. Pattern based.

Improved hand eye coordination. Increased range of motion, users have to reach at areas they typically do not access in their field. Vision field improvement

WAM has the user controlling a mallet and hitting as many moles as possible.

It was also clear that the previous group had a large prior experience with game development and website development. As a result there was little guide to assist our current team to continue work on the past games or to create new games. In addition, there was almost no guide on how the website and database was structured. 2.3 Previous Groups Recommendations

Though the system that the previous group gave our group was as the described in a working state, they had the vision that the system’s capability and usability could be expanded. Some of the recommendations that the previous group had were: continuing addition of control devices, therapists tools, customizable data analysis, and of course more games.[8] In addition, they recommended to try and automate the system as much as possible to assist the occupational therapists in assigning and monitoring patients with the games.

With regards to the games the previous group stressed the use and implementation of a rewards system as well as an implementation of game progression similar to that in the game “Fruit Viking”. They also stressed the importance of game maintenance, citing that bugs in the system causes the patients to get frustrated and lose interest in the games.

Page 8

3. Summary of Work 3.1 Research

Upon getting accepted into the GATOR project and after meeting with Dr. Sudeep, we were informed that this year's project would take a shift from past years. We as a team would be focusing on incorporating the Kinect v2 as well as continuing development and support of the leap motion games. Upon receiving this news we began researching development for the Kinect v2. Some of the information we found were the minimum system requirements to use a Kinect v2 with a computer which included[9]: 64­bit (x64) processor Physical dual­core 3.1 GHz (2 logical cores per physical) or faster processor USB 3.0 controller dedicate to the Kinect V2 4 GB of RAM Graphics card that supports DirectX 11 Windows 8 or 8.1.

These system requirements didn’t reflect the status of how the Kinect v2 SDK will operate. Our personal computers were simply not powerful enough to support the development of the Kinect v2, and, after discussing the situation with Dr. Pasricha it was decided that we were going to tap into the GATOR funds to purchase an adequate computer.

Unfortunately, Microsoft ended the sales of Kinect v2 for Windows on 2 April 2015 [10]. Though they indicated that support for the sensor would not end[11], as a group we began to look into alternative software that would provide us support for the Kinect v2. After some research and experimentation we settled on Unity 3D Pro being the optimal software for our needs. Unity 3D Pro is fully compatible with the official Kinect v2 SDK which meant that all functions and methods would be inherently supported. This would greatly reduce the complexity to code and speed up the creation process as it will also alleviate some of the learning curve. By using Unity we will be able to create a solid foundation for both our 2D and 3D games that can be built upon for future uses. 3.2 Leap Motion

When we started the project, the leap motion controller was already being utilized. The leap has two cameras and three infrared LEDs. This controller uses these devices in order to track hands with finger detection. This helped with both movement based games in a 3­dimensional space and with games that involved using hand gestures. Most of the games made for this were in 2­dimensional space due to simplicity. The leap motion was built to be used for both overhead and underhand view. Previous groups used it above the hand due to the fact that keeping your hand up the entire time is difficult, especially to the patients. This also made things difficult due to the fact that the leap

Page 9

controller is far better at tracking from below. This controller had some great things about it and some things that made it impractical.

The great thing about the Leap is that it not only tracks movement of the hand but also tracks the fingers. This allows for games developed for the purpose of hand gestures. As you can see to the right, this game gave points for being able to do the hand gesture directed as a certain moment. This is important for people that struggle to even open their hands as it pushes them to make that certain gesture. It also would tell you to rotate your hands to a certain angle. The Leap also had the ability to track your hand in 3­dimensional space for the ability to play a number of different games. This worked really well for the ability to make an multitude of different games; however, there was a setback from using the leap controller.

The problem that came with the leap controller was the fact that it did not track very well from the top. We could not make the patients keep their arms up the entire time during a game, as even for us it was an exhausting process. However, the leap controller is based on finger tracking and a majority of the patients were not able to completely open their hands. This made the tracking of the patient’s hand much less accurate. Therefore, it would lose the hand and the games would then become aggravating. Another problem was that people had to keep their hands on a matte black surface because the glare from the desks would cause even more problems for tracking. Even still, the Leap is a good option for future use in the GATOR project. 3.3 Myo Armband

The myo armband is a new controller that has recently been released that uses various different ways of tracking movement. It has an accelerometer, a gyroscope, and eight different EMG sensors. The accelerometer allows for the motions of the arm to be detected and can sense movement in all three directions. The gyroscope also allows for a better understanding of the movement of the arm. It can detect a tilt in all three directions as well. The EMG sensors make it so that it is possible to understand what muscles are being used. EMG sensors detect

Page 10

electric signal from the muscles and this allows for the sensor to detect different hand gestures. These gestures include but are not limited to: making a fist, opening your hand, turning you hand, flexing you hand right, flexing your hand left, and tapping your thumb and middle finger. These tracking methods are being used for powerpoint presentations, gaming, and a remote controller for motorized devices. We are using this as a way to play games and a way to give us data about our patients.

One of the reasons we want to use this technology is for game play and to make our therapy process more versatile. While we only had the leap motion controller prior to this year, people now have the option to buy different things and use different arm movements. While the leap motion controller only allowed arm movements on a table, we can now use arm movements in any space. Since the Leap only used finger tracking, you could not use multiple arms. The Myo, in comparison, doesn’t register the other arm and this feature is useful because many patients are not able to move their arms by themselves. We have now made it possible for patients to use one arm to aid in the movement of the other if necessary without interfering with the tracking. Therefore, their better arm will help their bad arm become stronger by helping it in its path. This is a type of therapy often used by the patients.

Another reason we want to use this for game play is hand control. As stated before, the Myo has the ability to read hand gestures and with this we can create hand gesture controlled games for therapeutic purposes. This is important because these patients often do not have the ability to do tasks like opening a door or holding a spoon. This type of therapy works on their ability to make hand gestures like this that would apply to their everyday lives. This helps them learn how to use their hands and focus on fine motor skills. Although, this is more advanced for many patients, it is important to have the ability to work on these type of movements.

The final reason why we are working on the Myo is for its ability to collect data coming from the patients’ muscles. Although there is not much research in this area, this opens new opportunities for patients and occupational therapists. We have made it so that we can collect all the EMG data coming from the user’s arm. As you can see on the picture on the left, the Myo armband has eight different segments. Each segment has its own EMG sensor. Therefore, we are able to get readings from each individual sensor. The picture below shows readings graphed of one of us trying this out; the small readings are electrical noise and small movements while the larger readings were large purposeful movements.

Myo Data

Page 11

Each graph is an individual segment. While some of the data is high on one side of the arm, there is almost nothing happening on the other side of the arm. Our hope is that we can send this data in a way that the therapists can understand what is actually happening to the patients and if they are making any progress in the ability to control their arms. Our hope is that this data will truly help those in need by helping the therapists understand what needs to be done based on the information given. 3.4 Kinect V2

With the Leap becoming less viable for what GATOR is striving to achieve, we decided to instead shift our focus to the Kinect v2. The Kinect v2 is, in essence, a specialized webcam. It is capable of producing a 3D model of an individual. This model is in real time and can be used in a variety of ways in terms of gaming. The programmer can dictate what each limb controls in game as well as

have particular gestures pertain to in game actions. This is all done due to the Kinect v2’s 1080p camera working alongside multiple infrared sensors and readers. All of this allows for the Kinect v2 to produce the 3D model while also creating a 3D environment in which the user is in. By creating this environment the Kinect v2 can differentiate between the user and any possible interference. So if there were spectators, poor lighting, or lots of background objects the Kinect v2 will not be interrupted in its current process.

The current games developed for the Kinect v2 are 2D oriented as it was a good starting point. These games were developed using the Construct 2 as it offered an intuitive method of programming that was suitable for the Kinect v2 as well as the Leap and Myo. More recent projects are being done using Unity 3D Pro. Unity allows us to create both 2D and 3D games that are specifically tailored to the Kinect v2. The Kinect v2 can distinguish the user’s fingers, hands, arms, etc all as separate inputs. The Construct was not able to utilize this whereas Unity can. This allows for a level of customization that is not possible using either the Myo or Leap.

Since the Kinect v2 is capable of all these inputs we can program specific user actions or motions into games and change them just as easily. By giving us and patients this level of customization we can create experiences that can be easily adapted depending on the needs of the patient. If a particular action cannot be performed by some patients, we don’t need to create a new game altogether to assist them. Instead, using the Kinect v2’s functionality, we can simply redefine the controls of the game to suit the needs of the patient. Our goal is to be able to have these specific controls and settings save to the patient's account so that there is no need to change it every time they return to play.

Page 12

With all of this hardware being readily available to us it became apparent that we needed games that took full advantage of this. These games needed to be varied and playable on as many devices as possible. With so much new and unfamiliar technology we decided to start out simple and begin with 2D based games. 3.5 Games

Video games allow one’s mind to go into a state where they are almost at a subconscious state of doing things. You are purposely playing the game, but you do not realize that you are moving your body in order to play games. Similar to the Wii Fit making people exercise without knowing it, we are using games for rehabilitation without the patient knowing it. This is the entire purpose of using games: a way of rehabilitation that distracts one’s mind from knowing that they are trying to rehabilitate themselves. Rehabilitation is a difficult and strenuous process beyond comprehension. It is hard to imagine not being able to do simple tasks by yourself let alone trying to strive for improvement without seeing any significant results. GATOR was started in order make this process as relaxed and easy minded as possible. What we must first work on is what kind of games would help.

As we said before, we have a lot of games already developed from previous groups. Our hope is now to add more games that will work more suitably for the Myo and the Kinect v2. Currently we have three games made. The first game is a spin on “Flappy Bird” called “Flipper the Flappy Dolphin”. Another game we have is a maze game called “Trespasser”. The last game is called “Cowboy Runner”.

The first game is called “Flipper the Flappy Dolphin”. This game was created in the program Construct 2 and follows the mouse in order to go through these open spaces. This is different than “Flappy Bird” because flappy bird jumps on a click or tap while this game focuses on up and down movements. As the player moves their arm up and down, the dolphin will follow. This allows for people who struggle with this motion to work specifically on this type for movement. It also works with the Myo, so the patient can also work on using both hands. As

explained above, many patients are not able to use their arm, so they have to use their other arm to direct them on the path. The trouble we had with this game was trying to figure out if it should only follow the mouse at a designated speed or if the dolphin should be at the same height of the mouse always. We decided it would be easier to make it so the dolphin would always be the same height of the mouse. Another thing that this had a problem with was making it so the game did not have collision problems. We could only draw a shape around the dolphin with a few amount of points before the game starts slowing down, but we didn’t want to make the game unfair by it counting it as a hit if you barely miss the pole. This was difficult for a rounded object like the dolphin. However, it works well, and we are proud of the game.

Page 13

Trespass

The pictures above are screenshots of the game “Trespasser”. This game was also made in Construct 2. This a maze game that works specifically on fine motor skills. The player’s purpose is to get the blue square object to the grey circle. If you touch any objects along your path, the blue square will return to the original position. There are stationary objects, moving objects, and even lasers as shown on the right picture. One thing that we warned about was keeping things simple. We have to understand that strokes and things like it also affect the brain and it can be overstimulating to some when too much is going on. The great part about this game is that it has progression. Not only does this game keep your score, but it also has 5 levels, each one being harder than the one previous. Like stated before, one of the goals we had at the beginning of the semester was to work on the progression of our games. A game that has one level and nothing else gets old fast. Even games with the different levels will eventually get boring, but it will definitely lengthen the time of enjoyment of the game. This game is better played on the Kinect v2 because of the fact that you need to make small movements. The Myo device only moves based off of acceleration, so making it move slowly is difficult. This game will hopefully help those that are late in the rehabilitation process working on small movements.

The last game we have is called “Cowboy Runner”. This game was created on the program Unity. The object of the game is to get as far as possible and was based off the game “Maze Runner”. In “Maze Runner” you are supposed to jump over whichever objects come in your way. In this version of the game, we want the jump to be a specific gesture determined by the player. This will allow for the patient to work on any gesture that they need to. This makes this game more versatile than the rest. The game also gets harder as you go along, making the progression of this game very fun. The difficulty

Page 14

we had with this game, and one that is still a problem, is trying to figure out how to make the gesture work as the click. We can have one gesture synced but telling it what gesture to use is far more difficult. 3.6 Website

Website development has been a challenge this semester. There were no guides put into place on how the website is structured, how to make changes, or how to upload new content. As a result we have had to experiment with and analyze the code. It has been a slow process, but we have come to an understanding on how the website is structured. Also, we have begun developing a guide for future groups to use and we have uploaded the games that we have created as a group to the website server. This will then allow users to access the games via the website similarly to how the current games are. As a group we have also begun analyzing a better way to transmit the data gathered from both the Myo and Kinect v2 onto the website to then be published in an easier readable structure such as a graph or chart. We anticipate that this will become a large focus for next semester as we begin to work closer with the occupational therapists.

3.7 Ethical Concerns

Ethics is a large part of any company, especially in engineering, and we want to stick to the engineering state of mind. One of the ethical concerns that we have is about the patients. It is our duty to keep the data that we record of them safe. We have to make sure that none of the information of the trusting patients can be stolen. Also, we have to make sure that any information that we do give is approved, including but not limited to: names, pictures, and personal statistics.

Another one of our main ethical concerns is that we don’t make a profit off of this. We are using pre­written code to manipulate on our code in order to make the process faster and the games better. If we were to sell our games, we would then have to give money to the original code developers, or possibly face lawsuits. Another thing that prevents us from selling our product is the fact that we are using an educational version of Unity. Unity does offer a pro version that gives the license to sell your product, but the educational version does not. Our final reason for not selling the product is that this product is there to help people. We do not want the money of the patients when they already have to pay for so much. 3.8 Potential Market, Manufacturability and Marketability

If this was our own product, we would undoubtedly be able to make a profit. However, this is not what we want to do. That being said, there is potential for this product in the market. While a typical session in OT usually is around $200, Anyone can buy all the equipment they would need from around $1000 including a pretty powerful computer. According to the patient studies from last year, the patients were very happy with the product. It truly did help with their therapy session. With about 800,000 people having a stroke annually[1] and 1.7 million having TBI annually, there are a lot of people out there that need help. Our goal is to market this product in order to make sure people know that this product exists as a potential alternative or addition to their regular therapy sessions. We want the most amount of people to be able to enjoy their therapy sessions.

Page 15

4. Conclusion and Future Work GATOR is designed to be a fun and engaging supplement for patients undergoing

physical therapy. The hardware we have acquired has allowed us to create a set of 2D based games that can be played using either the Leap, Myo, or Kinect v2. By utilizing Construct 2 and Unity 3D Pro we have built some engaging games that we can expand upon further. The website is in the first stages of being modified to support future game development as well as implementation of external hardware and data. While we have accomplished a lot this semester there is still much work to be done.

We are aiming to have some complete Kinect v2 games starting at the beginning of Spring. From there we can begin external testing using real feedback from patients. By using a combination the Myo and Kinect v2 we hope to have a better system in place of extracting data that can be used by both patients and therapist alike. To what extent they use this information will be determined by the therapists themselves and their feedback to us will allow us to help deliver results that they are striving for. While current games are in 2D, we should have some 3D implementations ready sometime in the beginning of next semester. While our experience in 3D development is limited, we have done prior research and are confident in that we can begin development with little hindrance. Once all of this is completed, the final goal would be to have the website be able to support all of this while still having reliable support for legacy code and devices. GATOR is designed to help patients undergoing physical therapy by helping make it an enjoyable experience. By the end of next semester we plan to deliver on this premise.

Page 16

REFERENCES [1] “Stroke Facts,” Centers for Disease Control and Prevention, 2015. [Online]. Available at: http://www.cdc.gov/stroke/facts.htm. [Accessed: Sept­2015]. [2] “Data & Statistics for Cerebral Palsy,” Centers for Disease Control and Prevention, 2015. [Online]. Available at: http://www.cdc.gov/ncbddd/cp/data.html. [Accessed: Nov­2015].

[3]“Kinect for Windows SDK,” Kinect for Windows SDK. [Online]. Available at: https://msdn.microsoft.com/en­us/library/dn799271.aspx. [Accessed: Oct­2015]. [4]P. Bernhardt, “#MyoCraft: Logging IMU and Raw EMG Data,” The Lab, 2015. [Online]. Available at: http://developerblog.myo.com/myocraft­logging­imu­and­raw­emg­data/. [Accessed: Sept­2015]. [5] Poore, Tevfik. Augmented Reality Games for Neurological Rehabilitation. Colorado State University.(2012)[online] http://www.engr.colostate.edu/ece­srdesign/AY11/rehabilitation/summary.html [Accessed: 4 December 2015] [6] Campuzano, Olson, Vlahinos. Augmented Reality Games for Upper­Limb Rehabilitation.Colorado State University. (2013)[online] http://projectsweb.engr.colostate.edu/ece­sr­design/AY12/rehabilitation/summary.html [Accessed: 4 December 2015] [7] Nash, Wallace, Thornton. (2014)Serious Games for Upper Limb Rehabilitation Spring Report.. Colorado State University [online]http://projects­web.engr.colostate.edu/ece­srdesign/AY13/ARGames/Documents/Spring_Final_Report.pdf [Accessed: 4 December 2015] [8]Hesser, LeFevre, Rowack. Games and Assistive Technologies for Rehabilitation. Colorado State University. (2015)[Online] http://projects­web.engr.colostate.edu/ece­sr­design/AY14/GATOR/GATORReport.pdf [Accessed: 4 December 2015] [9] Kinect hardware requirements and sensor setup. Microsoft. (2015) [online] https://dev.windows.com/en­us/kinect/hardware­setup [Accessed: 2 June 2015] [10] Protalinski, Emil. Microsoft stops producing Kinect for Windows v2 sensor, will focus on Kinect for Xbox One and Windows apps. Venturebeat. (2015)[Online]http://venturebeat.com/2015/04/02/microsoft­stops­producing­kinect­for­windows­v2­sensor­will­focus­on­kinect­for­xbox­one­and­windows­apps/ [Accessed: 5 December 2015]

Page 17

[11] Kinect for Windows Frequently asked questions. Microsoft (2015) [online] https://www.microsoft.com/en­us/kinectforwindows/meetkinect/faq.aspx [Accessed: 5 Dececmber 2015] Appendix A ­ Abbreviations GATOR­Games and Assistive Technologies for Rehabilitation PT­Physical Therapy OT­Occupational Therapist EMG­Electromyography V2­Version 2 SDK­Software Development Kit TBI­Traumatic Brain Injuries Appendix B ­ Budget

The cost for this semester has been higher than expected. This is attributed to trying to implement a relatively new technology and as a result the group had to purchase a new computer as well as high game development software. In addition, because Microsoft stopped selling Kinect for Windows v2, we had to purchase a Xbox One Kinect V2 as well as a Kinect for PC Adapter. The table below shows the breakdown of the semester: Table 2 ­ Budget

Starting Budget $3,967

Computer ­$843.95

Myo ­$200.00

Kinect V2 ­$70.00

Kinect PC Adapter ­$130.00

Unity Pro ­$350.00

Remaining Budget $2,373.05

In total, our group spend $1,593.95 this semester. However, because we now have the necessary tools for Kinect development next semester's costs should be highly reduced.

Page 18

Appendix C ­ Project Plan Evolution

10/21 11/5

Page 19

12/5

Page 20

Acknowledgements

As a group we would like to thank Dr. Pasricha for providing this project for our team. We would also like to thank professor Olivera Notaros for her guidance as the the senior design coordinator as well as all the engineers in residence.

Page 21