Can Concussions Be Diagnosed Using the Microsoft Kinect and Machine Learning?By Eric Solender

IntroductionIn today’s world, concussions are being uncovered in all spectrums of life. From car accidents to basketball games, concussions are everywhere. Concussions, while some consider them just a bump on the head, are serious, debilitating brain injuries. They occur when blunt impact hits the head, causing the brain to bounce off of the walls of the skull. Because of the impact the brain has with the skull, bruising is caused, which is what a concussion is – bruising of the brain.

BackgroundThe symptoms of concussions are miserable,including severe headaches, extreme sensitivity to light and sound, dizziness, double vision and many other unpleasant symptoms. They can also affect the chemical makeup of the brain which can cause psychological and cognitive disorders. There is also new research regarding the long term effects of concussions. One of the main findings is that concussions can cause Chronic Traumatic Encephalopathy(CTE), which is a disorder where proteins build up in the brain causing early onset Alzheimer’s, depression, and other mental disorders. Research has also found that when concussions occur too close to one another, severe swelling of the brain can be caused, which in many unfortunate cases, leads to death. A serious problem is that there are not enough reliable test that can be used in the field that can accurately predict whether or not the injured person actually has signs of a concussion, or is even concussed.

Problem/SolutionDue to this major problem, I asked myself the question, can concussions be accurately identified using only a computer and/or hardware? To find out the answer to this question, I created a program that utilizes the Microsoft Kinect, which is a tool that has speech recognition and motion tracking capabilities, to measure: balance, reaction time, memory and coordination on people. The program then records the data from each test, then sends it to a machine learning algorithm to determine whether or not the person does have a concussion.

HypothesisNull: If a concussed victim takes a computer aided concussion test and a non concussed victim takes the same test, the computer won't be able to distinguish the two outcomes to accurately predict whether or not the user has a concussion.

Alternative: If a concussed victim takes a computer aided concussion test and a non concussed victim takes the same test, the computer will be able to distinguish the two outcomes and accurately predict whether or not the user has a concussion.

Method IntroductionTesting for concussions is a complicated process. This is because there are so many different aspects of a person that a concussion could affect. The specific tests in this concussion tests include: pre(right before the test is administered) and post(right after the test is administered) test questionnaires, pre(right before the test is administered) and post(right after the test is administered) memory test, reaction time testing, concentration testing, and coordination testing. The scores for all of these tests are saved as the test progresses until the test is complete. On completion, the scores are then analyzed by a machine learning algorithm to predict whether a person actually has a concussion.

Questionnaire The test starts out by prompting the user for their symptoms. Knowing the user’s symptoms plays a major role in determining whether or not the user actually has a concussion. Generally, if the user rates their symptoms as severe, there is a higher probability that the user has a concussion. Because of this, the questionnaire is in the test. It has the user rate their symptoms on a scale of 0-5, 0 being not experiencing the symptom and 5 being extreme experience of the symptom. The different symptoms that the questionnaire goes through are: Headaches, Feelings of Dizziness, Nausea and/or Vomiting, Noise Sensitivity, Sleep Disturbance, Fatigue, Easily Tiring, Feeling Frustrated or Impatient, Forgetfulness, Poor Memory, Poor Concentration, Taking longer to think, Blurred Vision, Light Sensitivity, Double Vision, and Restlessness. All of these results are saved by the test for later use.

Word MemoryThe next test that the user completes is a word memory test. This test is designed to gauge whether the user’s short term memory has been effected. This is done by providing the user with words and then having them repeat them, in order, back to the test program. The computer then scores the user based on whether they said the correct words, and if they were in the correct order. The scores range from 0-6, 0 being nothing was correct, and 6 being everything was correct.

Coordination: Finger to NoseAfter the word memory test, the next test is the first of the coordination tests. In this test, the user is asked to place either their left or right pointer finger on their nose. This is done because people with concussion symptoms may not be able to perform this task. Therefore, this test is implemented into the overall concussion test as another determining factor as to whether or not the user has a concussion. The first thing this test does is ask the user to stand in a T formation. Once they have held this for 5 seconds, the are given 15 seconds to touch their nose with the indicated finger. This repeats 4 times. The test is then scored on whether they touched their nose, how close they got to their nose, and the route that their hand took to their nose. The scores for this test range from 0-8, 0 being the user was unable to perform the task, and 8 being the user performed the task perfectly.

Coordination: One Foot BalanceWhen a person has a concussion, their balance can become affected. To determine whether the user has this symptom, the program will test the user’s balance. To do this, the user is prompted to stand on one foot for 5 seconds, then repeat this multiple times. The program records the user’s maximum sway to determine how “balanced” the user really is. The user will be asked to do this on both legs a various amount of times until the program is satisfied with the results. The scores will range anywhere from 0 - 8, 0 being unable to perform the task, and 8 being the user perfectly able to perform the task.

Reaction Time WavingAnother symptom of a concussion is an affected reaction time. To test for this, the program will show either red or blue on the screen. If the color is red the user will waive their right hand, if the color is blue, the user will wave their left hand. This will happen a various amount of times until the program can get a definitive average time and average correct hand reaction. Generally the program will cycle through randomly 15 times. The scores for average reaction time, and percent correct are then saved for later use.

Reaction Time MatchingThe next test that user takes is a matching game. The user’s hands are mapped to colored circles on the screen, and the user’s task is to drag the correct color hand into boxes that appear on the screen. The purpose of this test is to measure coordination and reaction time. This adds more data to support the previous reaction time test and aid the algorithm in determining whether the user actually has a concussion.

Concentration: CountdownThe next test is a concentration test. Concentration is tested to determine if a user’s focus has been affected. This is because loss of focus is a major symptom of concussions, as a result of this, the program will determine if a user’s focus has been affected. The way in which the test works is the user will count down from 24-0 for the Kinect. The algorithm will then score this based on 2 conditions: 1)did the use say all of the numbers? 2)Were they said in the correct order. The program will then score this test based on the averages of both accuracy and order.

Concentration: Stroop EffectThe next test is used to determine whether a user can correctly identify different things, and how fast they can be identified. This test has three different stages to it, modeled after the stroop effect. In the first stage the user is prompted with the word representation of various colors in black ink. Their job is to read the words back to the kinect. This is used as a reference for stage two and three to hear the user says the words so that they can be correctly identified. In stage 2, the user is prompted with colored boxes and asked to say the name of the color on the box. In this test, the user’s interpreted color of the presented color is saved to prevent any possibility of errors from partial color blindness. Then in the third portion of the test, the user is given a word representation of a color, in ink that is not necessarily the matching ink. The user is asked to say the color of the word, and not what it actually says. The user’s response for each is stored and referenced with the previous test to determine if they were correct. The test is then scored based on average reaction time, and average correctness. These 2 scores are then saved for later reference.

Post Test MemoryAfter the stroop effect test, the user is then asked to repeat the words from the beginning of the test in the order that they were given. This test is in place to determine if a user can remember something for an extended amount of time, even with distractions going on in the background. The scoring for this is the same as the preliminary word memory test.

Post Test QuestionnaireAfter the word memory test is administered the test then asks the user for their symptoms again. This is to determine if concentration made the user feel worse than they did before the test.

What’s Next?At this point the test is complete for the user, but not for the data. All of the data that the program has been storing up to this point is now transferred to the server for analysis. This is where the program actually determines if the user has a concussion.

Machine LearningThe way in which the server analyzes the data to determine the result is through the use of Machine Learning.

Machine learning, also known as artificial intelligence, is where a computer can learn from examples and make its

own decisions. In this instance, the machine learning is fed results of the test taken, with their medically confirmed

answers to build the training database. The machine learning algorithm then analyzes the training data to

generate its prediction based on a Multiclass Support Vector Machine(MSVM).

An MSVM is a method of supervised learning for classification where there can be multiple results. For the

purposes of the concussion test there are 3 possible results: concussion, no concussion, and intentionally failing.

The program makes its prediction by finding the greatest confidence on the different classes(results).

Machine Learning ContinuedThe equation used in this algorithm essentially finds the deviation between clusters of data, drawing a line between the sets to find the differences. This then allows the program to find the confidence by how close the user’s tested data aligns to the training data. The answer is determined by finding the result with the highest confidence. This result is then sent back to the client and is displayed to the user.

Data CollectionThe data in this project was collected from real concussed and non concussed people. The training set consisted of 150 people. The two accuracy sets consisted of 50 people each. The accuracy of the project was calculated to be 85% accurate!

Stroop Effect Results

Matching Test Results

Accuracy

Discussion/ConclusionIn today’s world, concussions are a serious problem. This is because they go undetected all the time, leading to even worse

conditions such as CTE, Secondary Concussion Syndrome, Depression, etc. Thankfully, this problem is solvable! This is proven

through the use of the Kinect for Windows and Machine learning. Using these tools, I have created a program that can

accurately predict whether or not a user has a concussion!

This is proven through the results of the concussion test. On tested patients: simulated and not simulated. In regards to the

ranges, these are established by the Machine Learning, meaning that they are constantly shifting and evolving based on the data

being used to train the system. Even with these changing ranges, the results are generally the same. For example – concussed

patients rated their symptoms higher than non-concussed patients. This is because concussed patients are actually

experiencing these symptoms, unlike the those who are not concussed. Another area that proves this, are the scores from the

coordination tests. The people that took these tests concussed scored much lower(i.e their times took longer and their scores

were lower) than those without concussions. This pattern can be seen throughout the entire test, the concussed people doing

worse on every test than non-concussed people. This was also proven when the machine learning was tested on sample data,

and created results of the correct output that corresponds to what it was fed.

Discussion/Conclusion ContinuedThe intended use of this research would be for field, home and clinical use. That way anyone can use the test and determine if

they have a concussion. There will also be a system put in place to connect doctor accounts and patient accounts so that

doctors can receive their patients data from wherever their patient took the test. Eventually the test will be portable so that it can

actually be used in the field. The use of machine learning in this research can also be expanded to other areas such as autism

spectrum and epilepsy detection.

Although this project is accurate, there still is some degree of error. One problem is that the speech recognition engine is not

always accurate. It will think you said “tree” instead of “dog” if the word isn’t pronounced clearly enough. Another known error is

that the kinect will misunderstand what it is seeing and display a person that is not in the foreground. The last source of error is

in the machine learning. There will always be some degree of error as the system is collecting more data and learning from it.

Because of this, more testing and data collection is needed to increase the accuracy of the test. As of now, the test must be

administered in a quiet surrounding, but support for loud surroundings is in the works and is being developed.

Discussion/Conclusion ContinuedIn conclusion, the test will continue to be refactored and improved. The data will get more accurate and pieces of the project will grow and become even better than they are. It is the hope that this project will revolutionize concussion detection to show that with technology, anything is possible!

Works CitedCopeland, Marshall. Microsoft Azure: Planning, Deploying, and Managing Your Data Center in the Cloud. Berkeley, CA: Apress, 2015. Print.

Freeman, Adam. Pro ASP.NET MVC 5 Platform. Berkeley, CA: Apress, 2014. Computer software.

Jana, Abhijit. Kinect for Windows SDK Programming Guide. Birmingham: Packt Pub., 2012. Print.

Johnson, Bruce. ASP .NET Web Services. Eden Prairie, MN: Appdev, 2003. Computer software.

Karpov, Valeri, and Diego Netto. Professional AngularJS. N.p.: n.p., n.d. Print.

Kinect. Singapore: Microsoft Game Studios, 2010. Computer software.

“Machine Learning with R - Second Edition.” Machine Learning with R. Paktpublishing, n.d. Web. 22 Jan. 2016.

Noonan, Robert D. HIPPA: Employer’s Guide to Compliance. New Hardford, CT: Executive Greetings, 2002. Print.

“Stack Overflow.” Stack Overflow. N.p., n.d. Web. 22 Jan. 2016.

Standards for Educational and Psychological Testing. Washington, DC: American Psychological Association, 1985. Print.

Webb, Jarrett, and James Ashley. Beginning Kinect Programming with the Microsoft Kinect SDK. New York: Apress, 2012. Print.

R Core Team (2015). R: A language and environment for statistical computing. R Foundation for Statistical Computing,

Vienna, Austria. URL https://www.R-project.org/.

C. R. Souza, “The Accord.NET Framework,” http://accord-framework.net . São Carlos, Brazil. Dec 2014.