ppt - johnny chung lee - human computer interaction research
Post on 18-Oct-2014
622 views
DESCRIPTION
TRANSCRIPT
Using a Low-Cost Electroencephalograph for Task Classification in HCI Research
Johnny C. Lee
Carnegie Mellon University
Desney S. TanMicrosoft Research
UIST 2006, Montreux Switzerland
National Geographic, March 2005 NY Times Magazine, October 16, 2005
Brain-Computer Interfaces (BCI)
A direct technological interface between a brain and computer not requiring any motor output from the user
Example Conferences/Journals with BCI interests:Neural Information Processing Systems (NIPS)IEEE Transactions on Biomedical EngineeringIEEE Transactions on Neural Systems and Rehabilitation Engineering
Why is this relevant to UIST or HCI?
BCI research traditionally focuses on exploratory neuroscience and rehabilitation engineering.
Brain sensing could provide valuable data about:- engagement
- cognitive work load
- surprise
- satisfaction
- frustration
Potentially helpful toContext Sensitive or Evaluation Systems
Values of BCI Values of HCI
VS
To use any means necessary to demonstrate that brain-computer interaction is possible.
To use reasonable means to achieve a practical benefit to many users.
use equipment costing $100K to +$1 million USD
use highly invasive surgical procedures
require hours or days of operant conditioning
remove data from poor performing subjects
It is okay to:
use fairly affordable and accessible equipment
be safe for repeated and extended use
be usable without requiring significant user training
use data from all subjects to evaluate its performance
We’d like to:
Where do we start?
Brain Sensing/Imaging Technologies
MRI
CT
ECoG
SPECT
PET
MEG
fMRI
EROS/fNIR
EEG
Currently Impracticalfor HCI
- Safe, easy, no medical expertise
EEG – Electroencephalograph
the neurophysiological measurement of the electrical activity
of the brain by recording from electrodes placed on the scalp
(skipping the lower level neurophysiology)
- Measures the voltage difference between two locations on the scalp
- Only picks up gross, macroscopic, coordinated, and synchronized firing of neurons near the surface of the brain with perpendicular orientation to the scalp. (thus majority of activity is hidden)
Analogous to holding a thermometer up to the side of a PC case
EEG Devices
Manufacturer: BioSemiChannels: 64-128Cost: ~$30K USD
Manufacturer: EGI SystemsChannels: 128-512Cost: $100K-$250K USD
• Lowest cost FDA approved device• Designed for home and small clinical use.• Only $1500 USD• Specs:
– 2-channels– 8-bit at 4µV resolution– 256 samples/sec
• Has yet to be validated for BCI research work.• If it works, it lowers the entry bar for BCI research.
The Brainmaster
Validating the Device (and ourselves)
1. Validate the device
Can we get useful data from such a low-end device?
2. Validate ourselves
To explore this space, we must be able to collect our own data.
Validating the Device (and ourselves)
Keirn, Z., “A New Mode of Communication Between Man and His Surroundings”, IEEE Transactions on Biomedical Engineering, Vol. 37, No. 12, 1990.
• Data is available for download• Data has not been reproduced in the past 15 years
– Some computational BCI researchers have just used this data.– State of the art does is not a great deal better.
Reproducing the Keirn Data
We adapted procedure from Keirn to better control potential confounds.
3 tasks:• Rest (Baseline): Relaxation and clearing of mind• Math: Mental arithmetic, prompted with “7 times 3 8 5”• Rotation: Mentally rotate an object, prompted with “peacock”
Tasks from the original paper were designed to elicit hemispheric differences.
Experimental Procedure
User is instructed to keep eyes closed, minimize body movement, and not to vocalize part of the tasks.
For each task, a computer driven cue is given: Rest, Math, Rotate
Following Math and Rotate, the experimenter says either the math problem or object
Experimental Procedure
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
session
task (14 seconds)
trial
Block design adapted from Kiern
Experimental Procedure
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
3 sessions per subject
Many short tasks prevent correlation with EEG drift
Experimental Procedure
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Rest Math Rot
Rot Rest Math
Rest Rot Math
Rot Math Rest
Math Rest Rot
Math Rot Rest
Subjects:8 subjects (3 female)29-58 years of ageAll were cognitively and neurologically healthyAll right handed
EEG Setup
International 10-20 EEG electrode placement system
Two channels placed on P3 and P4 with both references tied to Cz.
Electrodes are held in place using conductive paste.
5-10 minute preparation.
Processing the Data
Data Processing
14 secs
Rot
task (14 seconds)
Data Processing
14 secs
Task Cue
Rot
task (14 seconds)
Data Processing
14 secs
Experimenter Prompt
Rot
task (14 seconds)
Data Processing
14 secs
Task Onset
Rot
task (14 seconds)
Data Processing
14 secs
Performing Task
Rot
task (14 seconds)
Data Processing
14 secs
Performing Task
~4 secs
Rot
task (14 seconds)
Data Processing
10 secs
Performing Task
Rot
task (14 seconds)
Removing time for machine learning
Most machine learning algorithms don’t handle time series data very well.
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
10 seconds
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec• Perform signal processing on each of the 9 windows to get our
“time less” feature set
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs
This provides486 windows
per participant
Signal features for each window
Generic signal features such as mean power, peak frequency, peak frequency amplitude, etc.
Features frequently used in EEG signal analysis.
Common EEG Features
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
0 5 10 15 20 25 30 35 40 45 500
1000
2000
3000
4000
5000
Raw EEG
SpectralPower
Gamma (30Hz-50Hz)
Beta High(20Hz-30Hz)
Beta Low(12Hz-20Hz)
Alpha(8Hz-12Hz)
Theta(4Hz-8Hz)
Delta(1Hz-4Hz)
Feature Processing and Selection
The 39 base features from each window are mathematically combined to create 1521 total features.
We used a feature preparation and selection process similar to [Fogarty CHI’05] to reduce the number of features:
23 features for 3-task classification (486 examples)
16.4 features for pair-wise classification (324 examples)
Baseline Results – 3 cognitive tasks
3 taskMath v. Rotate
Rest v. Math
Rest v. Rotate
user 1 67.9% 83.3% 88.0% 85.8%
user 2 70.6% 82.7% 91.4% 84.3%
user 3 77.6% 88.3% 93.8% 86.7%
user 4 63.6% 69.4% 84.9% 86.7%
user 5 66.5% 91.0% 81.2% 80.9%
user 6 59.3% 80.6% 80.2% 68.5%
user 7 71.4% 87.3% 90.4% 86.7%
user 8 69.8% 87.7% 82.4% 83.6%
Average 68.3% 83.8% 86.5% 82.9%
BayesNet classifier
Chance: 33.3% 50% 50% 50%
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
2 secs 86.5%68.3
%
83.8% 82.9%
We can do better…
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
???
0 500 1000 1500 2000 2500 3000-30
-20
-10
0
10
20
30
Throwing time back in…
“Math”
We can average over temporally adjacent windows
to improve classification accuracy
Averaging with Task Transitions
Task transitions result in conflicting data in averaging window.
High density of transitions will result in lower accuracy.
Averaging with Task Transitions
Fewer task transitions will yield better classification accuracy.
Averaging with Task Transitions
No transitions and averaging over all data will be the even better.
Classification Accuracy with Averaging
3 tasks
Math v. Rot
Rest v. Rot
Rest v. Math
100%
95%
90%
85%
80%
75%
70%
65%
60%
55%
50%
Mean Classification Accuracy vs. Averaging Scenarios (Mental Tasks)
Baseline No Averaging
5 windows with transitions
5 windows no transitions
9 windows no transitions
+5.1 to +15.7%for 3-tasks
Error bars represent standard deviation
So, can we really read minds?
3 tasks
Math v. Rot
Rest v. Rot
Rest v. Math
100%
95%
90%
85%
80%
75%
70%
65%
60%
55%
50%
Mean Classification Accuracy vs. Averaging Scenarios (Mental Tasks)
Baseline No Averaging
5 windows with transitions
5 windows no transitions
9 windows no transitions
Error bars represent standard deviation
Possibly not, we might be really detecting subtle motor movements….
Cognitive/Motor “Fabric”
Does this matter to neuroscience? Yes
Does this matter to HCI? Maybe not
Tasks of varying cognitive difficultly are involuntarily coupled with physiological responses, such as minute imperceptible motor activity. [Kramer ’91]
Therefore, it is impossible to completely isolate cognitive activity neurologically intact individuals.
Cognitive/Motor “Fabric”
If motor artifacts are reliably correlated with different types of tasks or engagement, why not use those to help the classifier?
Requiring users to not move is also very impractical.
Non-Cognitive Artifacts detected by EEG:– Blinking– Eye movement– Head movement– Scalpal GSR– Jaw and facial EMG– Gross limb movements– Sensory Response Potentials
Experiment 2 – Game Task
To explore this idea of using non-cognitive artifacts to classify tasks using EEG, we chose a PC-based video game task.
Halo, a PC-based first person shooter game produced by Microsoft Game Studios.
Navigate a 3D environment in an effort to shoot opponents using various weapons.
Relatively high degree of interaction with mouse and keyboard input controls.
Game Tasks
Rest – baseline rest task, relax, fixate eyes on cross hairs on center of screen, do not interact with controls. Game elements do not interact with participant.
Solo – navigate environment, interact with elements in the scene, and collect ammunition. Opponent controlled by expert did not interact with participant.
Play – navigate environment and engage opponent controlled by expert. Expert instructed to play at a level just slightly above skill of participant to optimally challenge them.
Game Experimental Procedure
Setup, design, and procedure was similar to first study.
• Participants had tutorial and practice time with game controls.
• 3 tasks repeated 6 times (counterbalanced)
• Tasks were 24 seconds to allow navigation time.
• Only 2 sessions were run for each participant
• Same 8 participants from first study were run in this study.
• Same data preparation and machine learning procedure.
Results – Game Tasks
3 tasks
Rest v. Play
Rest v. Solo
Solo v. Play
100%
95%
90%
85%
80%
75%
70%
65%
60%
55%
50%
Mean Classification Accuracy vs. Averaging Scenarios (Game Tasks)
Baseline No Averaging
5 windows with transitions
5 windows no transitions
19 windows no transitions
Error bars represent standard deviation
93.1%
Conclusion
This experimental design and data processing procedure can be applied to a much wider range of applications/tasks. Our two experiments were just two examples at different ends of a spectrum.
Compelling results can be achieved with low-cost equipment and without significant medical expertise or training.
Non-cognitive artifacts (inevitable in realistic computing scenarios) can be embraced improve classification power.
To make BCI relevant to HCI, we must challenge traditional assumptions and creatively work with its limitations.
Thanks!
Johnny Chung [email protected]
Desney [email protected]
Thanks to MSR and the VIBEGroup for supporting this work.
Cross-user Classifier
3 Cognitive Tasks
40.00%
45.00%
50.00%
55.00%
60.00%
65.00%
70.00%
75.00%
80.00%
85.00%
90.00%
95.00%
100.00%
Base 5 win 5win no edges Known
3class
MathRotate
RestRotate
RestMath
Cross-user Classifier
3 game tasks
50.00%
55.00%
60.00%
65.00%
70.00%
75.00%
80.00%
85.00%
90.00%
95.00%
100.00%
Base 5 win 5win no edges Known
3class
SoloPlay
RestPlay
RestSolo
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG
SPECT
PET
MEG
fMRI
EROS/fNIR
EEG
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG – highly invasive surgery
SPECT – radiation exposure
PET – radiation exposure
MEG
fMRI
EROS/fNIR
EEG
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG – highly invasive surgery
SPECT – radiation exposure
PET – radiation exposure
MEG – extremely expensive
fMRI – extremely expensive
EROS/fNIR
EEG
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG – highly invasive surgery
SPECT – radiation exposure
PET – radiation exposure
MEG – extremely expensive
fMRI – extremely expensive
EROS/fNIR – currently expensive, still in infancy
EEG – safe, easy, no medical expertise
Other cool things you can do with an EEG device…
Event Related Potentials (ERP)
• Electrical activity related to or in response to the presentation of a stimulus
• Very well studied• Relatively robust• Used daily in clinical settings to check sensory
mechanisms, typically in infants
• Requires averaging over 30-100 windows synchronized with to see response.
ERP - AEP
ERP: Auditory Evoked Potential • Used in clinics/hospitals to check hearing.• Response to clicks in the ear
Bold Lines = no clicksThin Line = with clicksAEP response
AEP response
ERP - VEP
ERP: Visual Evoked Potential • Focusing on a flashing target, the visual cortex will “resonate” at the
stimulus frequency.
Stimulus Frequency
Harmonics
ERP – Auditory and Visual P300
• Well known/studied potential related to “attention” or “surprise”
• Presented with 2 stimuli and instructed to count one of the stimuli
• Positive response will follow the stimulus of interest.
Side note: EEG as ECG
ECG - Electrocardiogram • placing an electrode on the chest provides a clear measure of
cardiac activity.• translation to BPM is a simple autocorrelation
Heart beats
Single Beat period
0.4
µV
un
its
EEG as EMG
EMG - Electromyography • Measures muscular activity
Wrist relaxation (return to straight position)
Tension holding
Wrist inward contraction (toward inner forearm)
Wrist rest state
NOTE: The magnitude of the spikes seem to be proportional to the acceleration involved with the movement.
0.4
µV
un
its
EEG as Blink Detector
• Electrical activity due to muscle movement involved with eye blinks propagate through the head.
• Similarly, eye movements also affect the EEG recording
0 0.5 1 1.5 2 2.5
x 104
-200
-150
-100
-50
0
50
100
150
200
Blinks Blinks
Task Classification Background
• Previous work is split primarily into two camps:
Operant Conditioning Pattern Recognition
Task Classification Background
• Previous work is split primarily into two camps:
Operant Conditioning Pattern Recognition
Human learns how the machine works
Machine learns how the human works
Task Classification Background
• Previous work is split primarily into two camps:
Operant Conditioning Pattern Recognition
Human learns how the machine works
Machine learns how the human works
Relatively newEarly dabbling in the late-80’s
Most work has happened in last 5 years.
Pattern Recognition
Data Collection &
Experimental Design
Signal Processing &
Feature Generation
Machine Learning &
Improving Accuracy
EEG Setup
International 10-20 EEG electrode placement system
Two channels placed on P3 and P4 with both references tied to Cz. Locations selected based on pilot recordings.
Attaching electrodes: Prepare the site with a cleaner, use conductive paste to improve connection and hold electrode in place.
P Paste rinses out with water, non-toxic. 5-10 minute preparation.