Learning using Gesture-based Computing and Brain Computer ...€¦ · • Gesture-based Computing and Brain Computer Interface technologies are: – Affordable: Systems are
21
1 Simulation & Learning using Gesture-based Computing and Brain Computer Interfaces Sridhar Natarajan, Ph.D. Advisor Information Technologies Schawn Thropp Advisor Technologist Alan Hoberney Executive Director, Information Management & Technologies Capability
Simulation & Learning using Gesture-based Computing and
Brain Computer Interfaces Sridhar Natarajan, Ph.D. Advisor Information Technologies
Schawn Thropp Advisor Technologist
Alan Hoberney
Executive Director, Information Management & Technologies Capability
2 2
CTC Overview • 501(c)(3) nonprofit established in 1987
• Staff of 1,400+ professionals
• More than 50 locations
• 900,000 sq. ft., including labs & demonstration space
• Top 100 Government Contractor
• Quality/EH&S Management System comprised of industry-best models: ISO 9001 (Quality) and 14001 (Environmental), AS9100 (Aerospace), and CMMI-SE/SW (Systems/Software Engineering)
• Nationally recognized security capabilities with 300,000+ sq. ft. of Top Secret/Sensitive Compartmented Information Facility Space, JWICS, SIPRNet, and NIPRNet access
3 3
Overview
• Participants in this session will: –Learn about:
CTC employees demonstrate progress on a research and development project using human computer interface technology.
Presenter
Presentation Notes
Point out the technologies in use. I believe there was a video created that demonstrates some of the brain-based and gestured based work we are doing. Need to find it an insert. Check with Sridhar
5 5
Key Message • Gesture-based Computing and Brain Computer Interface
technologies are: – Affordable: Systems are <$1000
– Supported: Open source “ecosystem”
– Mature: Available commercially
– Intuitive: Low cost human capital requirements
• We are seeking opportunities for collaborative research and technology development for applications in Learning & Simulation
To accelerate progress towards the Next Generation Simulation/Learning & Human Performance Systems!
Presenter
Presentation Notes
Market, technology, drivers Themes: doing cool stuff with low cost components. Check with Mark Friedman and Justin D’arcangelo - see what kind of demo can be carried/conducted – check KinectHacks 10x20
Neurofeedback Practice generating the state of mind most beneficial to their sport, art or desired behavior
Adaptive Peak Performance Trainer (APPT)
Real-time analyses of EEG and heart rate to provide automated feedback of trainees’ psychophysiological state in relation to a an expertise profile
Neurostimulation Modulation of the nervous system and electrically activate neurons in the body – including sight and sound
Thought Pattern Recognition
Mental state identification to deliver dynamically adaptive training, gauge operator readiness
Gesture-Based Avatar Training
Train avatar by recording realistic human behavior instead of via code. Intelligent reaction of avatars, avatars that behave in a realistic way through gestures and body language - Kinesics
“Touch Free” Interfaces
Operators in sterile and/or hostile environments who cannot use their hands
10 10
Conclusion
CTC is seeking opportunities for collaborative research and technology development for applications of Gesture-based Computing and Brain Computer Interfaces in Learning & Simulation!
Brain Computer Interface Example • Application: Neurofeedback • Audience: athletes, musicians, dancers, ADD • Result: Practice generating the state of mind most
beneficial to their sport, art or desired behavior • Technology Enabler: presentation in real time of
information about the state of the brain • References: Desney S Tan, Anton Nijholt
Presenter
Presentation Notes
There is evidence that athletes, musicians, dancers and those suffering with behavioural problems like attention deficit disorder amongst others are benefiting from neurofeedback. Neurofeedback is the presentation in real time of information about the state of the brain received via electrodes placed on the scalp. Such systems allow the wearer to practice generating the state of mind most beneficial to their sport, art or desired behaviour. Although there are issues about the reliability, the need for personalisation and time required to be beneficial this technology is becoming more widespread. In this article we look at some of this research and existing systems, their uses, some of the concerns, and the potential this has for education. http://research.microsoft.com/en-us/um/redmond/groups/cue/bci/ http://research.microsoft.com/en-us/um/people/desney/publications.htm
– Performs Real-time analyses of EEG and heart rate to provide automated feedback of trainees’ psychophysiological state (in relation to a psychophysiological profile of expertise.
significantly when novices train with the APPT as compared to controls.
• Technology Enabler: measurements of electroencephalographic (EEG) frequency bands of theta (3-7 Hz) and alpha (8-12 Hz)
• References: Chris Berka, Adrienne Behneman, Natalie Kintz, Robin Johnson and Giby Raphael.
Presenter
Presentation Notes
Interactive Neuro-Educational Technologies (I-NET) are designed to increase the pace and efficiency of skill learning by modeling psychophysiological characteristics of expertise and developing sensor-based feedback to accelerate novice-to-expert transition. A Pre-Shot Peak Performance (PSPP) profile was identified in experts in marksmanship, archery and golf. The PSPP is characterized by an increase in electroencephalographic (EEG) frequency bands of theta (3-7 Hz) and alpha (8-12 Hz), and a local change in heart rate in the seconds leading up to the shot/putt. The PSPP profile of expertise was then incorporated into a novel device called the Adaptive Peak Performance Trainer (APPT). The APPT integrates real-time analysis of EEG and heart rate to provide automated feedback of trainees’ psychophysiological state (in relation to the psychophysiological profile of expertise). Preliminary data indicate that marksmanship learning trajectories improve significantly when novices train with the APPT as compared to controls. http://chrisberka.cgpublisher.com/product/pub.191/prod.70
16 16
Brain Computer Interface Example
• Application: Neurostimulation – Modulation of the nervous system and electrically activate neurons in the
Brain Computer Interface Example • Application: Mental state identification to deliver
dynamically adaptive training, gauge operator readiness • Audience: Drivers, surgeons, first responders, military
personnel • Outcome: Recognizing thought patterns • Technology Enabler: Neuroimaging,
Functional magnetic resonance imaging (FMRI) brain scans, Emotiv Systems headset
• References: Just, Marcel, Mitchell, Tom, Nishimoto, Shinji
Presenter
Presentation Notes
http://en.wikipedia.org/wiki/Thought_identification Operator readiness looking at sleep data and mental state data – driver, surgeons, first responders, doctors, bomb defusion higher order brain patterns during specific situations Image: http://en.wikipedia.org/wiki/File:EPOC_IGN.jpg
18 18
Gesture-based Computing Interface Example
• Application: Avatar training • Audience: Game developers • Outcome: Train avatar by recording realistic human
behavior instead of via code • Technology Enabler: Kinect • References: Demonstration
Presenter
Presentation Notes
Talk to Ron Punako’’s Look at Kinect sites as well.
19 19
Gesture-based Computing Interface Example
• Application: Interrogation or Interviewing Training • Audience: New hires, focus groups, therapy session,
criminal investigations – Observing Kinesics of a person • Outcome: Intelligent reaction of avatars, avatars that
behave in a realistic way through gestures and body language
• Technology Enabler: Microsoft 260 Kinect Gesture Recognition system, Unity, Omek and OpenNI
• References: Demonstration
Presenter
Presentation Notes
Kinesics – the interpretation of body language such as facial expressions and gestures. More formally – non-verbal behavior related to movement, either of any part of the body or the body as a whole (Wikipedia - http://en.wikipedia.org/wiki/Kinesics) Audience: Relevant to anyone who interviews persons for jobs, focus groups, therapy sessions, criminal investigations or who are observing the Kinesics of public figures. Another audience could be consumers. Additionally security cameras could be used to observe the kinesics of individuals in key public places to preempt criminal activity.��Current Outcomes: The outcomes for avatar-based training include the reaction of based avatars to the body language of the observed interview or interrogation facilitator. The avatars disposition towards the facilitator would be modified to improve or deteriorate as a result of the observed behavior of the facilitator. The avatar’s disposition is programmed to be receptive to appropriate sets of gestures at particular points within a virtual interview or interrogation process timeline. In this way the facilitator is trained how to work with people with different dispositions and personality.��The outcomes for therapy could include observation of therapists in their communication with those who have communicative deficits such as autism. This would be a discovery technique to attempt to determine and refine what kinds of kinesics work to best communicate with individuals with these deficits. Advanced/Futuristic Outcomes Personalized, consumer-focused marketing is coming. That is, when you are walking through the mall of the future your presence will be detected through some recognition tech, usually specified as a personally identifiable chip. When the chip is detected you are given a personalized marketing message based upon your demographic and purchasing tendencies. This communication may be presented through super thin and flexible OLED screens embedded in everyday product packaging or on walls in a store. The sound source for the message is focused onto you as a personalized cone based upon your physical coordinates made available through detection of the chip. Imagine now that your current disposition could be read through gesture recognition technology coupled with the information on the personally identifiable chip. Now marketers would have the capability to know who you are, what your purchasing tendencies are and what you want at this very moment through scanning of your kinesics information. Imagine – “Sir, are you feeling tired or down this morning (It knows you are due to your kinesics)? You should refill you Starbucks card and have you favorite cinnamon dolce latte.” - This system would be an absolute gold mine (Marketing holy grail) through corporate sponsorships and through price per/impression. �The outcomes for security industry could come in the form of security cameras that could be used to observe the kinesics of individuals in key public places to preempt criminal activity as a kind of “Minority Report” style of precognition. Gesture Recognition technology already has the capability to simultaneously watch multiple people and interpret their actions. When a “hit” is registered an automated notice could be made available to law enforcement. The advantage of this kind of system is that it is low-cost and ever vigilant supplementing the need for manual observation. A simple example would be a purse snatcher. The act of quickly and violently pulling an object would be interpreted as non-standard crowd activity. A notice could then be generated that a non-standard event had occurred at location x along with a snapshot of the event in question. – We might want to keep this one under our belt as I am not sure if anyone else is doing this. Technology: The Microsoft 360 Kinect Gesture Recognition system is the technology used to observe the facilitators’ Kinesics. The ability to electronically observe gestures provides a natural and realistic interface for the interview or interrogation facilitator to be observed by a computer for interpretation into Kinesic information. We have successfully implemented prototype Kinesics recognition functionality within the Unity environment using both Omek and OpenNI software. The software would need to be trained to detect and interpret the various range of gestures that represent a given communication and would need to be trained to exclude false positives. Cultural considerations must also be taken as Kinesics for the same communication differ across cultures in how they are physically represented.��Resources: A very broad base would be interested such as job interviewers for academia, industry, commerce, psychologists, security industry, PR coaches and criminal investigators.�
20 20
Gesture-based Computing Interface Example
• Application: “Touch Free” Interface • Audience: Operators in sterile or hostile environments who
cannot use their hands • Outcome: Use of more natural movements in training
situations, augmented learning situations • Technology Enabler: Motion detection system (Kinect) • References: Institute of Forensic Medicine – University of
Bern in Switzerland: hands-free way to review radiological images
Summary: Combining Neuro-Technology along with synergistic technologies*, CTC is ushering in a new paradigm of “Human-Machine” interactions within current & future applications to meet and exceed the Healthcare needs of US Veterans. * such as Gesture Recognition, Haptics, Facial Expression Recognition, Voice Recognition and Advanced Data Visualization
VETERANS AFFAIRS (VA) APPLICATIONS OF CTC’s NEURO-BASED HUMAN-MACHINE INTERACTION TECHNOLOGY RESEARCH
Applications: • Prosthetics • Treatment of paralysis • PTSD/TBI monitoring, analysis and treatment • Pain Management • Quality of Life Control (e.g. wheelchair,
automated home) • Re-habilitation: Game-based Learning Relevant VA Initiatives • Veteran Mental Health (IVMH) Initiative • New Health Care Model (NHCM) Initiative • Research & Development (R&D) to Enhance the
Long-term Health and Well-being of Veterans
Details: • Neurotechnology is the application of medical
electronics and engineering to the human nervous system
• Off the Shelf (COTS) technologies that can read and interpret Brain Activity, Brain States and Emotions; and integrate with external software and hardware systems/devices/appliances
• Partnerships with research organizations and technology vendors
• CTC’s Internal Research & Development (IR&D) ongoing efforts: 3-5 Year Outlook for applications in DoD and VA
