an emotion recognition journey
DESCRIPTION
An Emotion Recognition Journey. Lucy Kuncheva. A long time ago in a galaxy far, far away. I’d better move to Cardiff. around the summer of 2008,. we came in contact with the School of Psychology at Bangor University. I’d better move to Brunel. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/1.jpg)
An Emotion Recognition Journey
Lucy Kuncheva
![Page 2: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/2.jpg)
A long time agoin a galaxy far, far
away...
![Page 3: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/3.jpg)
around the summer of 2008,we came in contact with the School of Psychology at Bangor University.
I’d better move to Cardiff...
I’d better move to Brunel...
![Page 4: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/4.jpg)
Areas of activation in the brain in response to emotion stimuli.
Amygdala
![Page 5: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/5.jpg)
Areas of activation in the brain in response to emotion stimuli.
The limbic system
![Page 6: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/6.jpg)
fMRI data were acquired from 16 right-handed healthy US college male students (aged 20–25).
![Page 7: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/7.jpg)
The Dead Salmon Lesson
![Page 8: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/8.jpg)
How far are we from MIND READNING?
this far...
![Page 9: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/9.jpg)
March 2009
EPSRC proposal
New approaches for fMRI data analysis
![Page 10: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/10.jpg)
April 2010
FP7 proposal
Lost an entire month of my life to this....
![Page 11: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/11.jpg)
1. Kuncheva L.I., J. J. Rodriguez, C. O. Plumpton, D. E. J. Linden and S. J. Johnston, Random Subspace Ensembles for fMRI Classification, IEEE Transactions on Medical Imaging, 29 (2), 2010, 531-542.
2. Kuncheva L.I., J. J. Rodriguez, Classifier Ensembles for fMRI Data Analysis: An Experiment, Magnetic Resonance Imaging, 28 (4), 2010, 583-593.
3. Kuncheva L.I. and C. O. Plumpton, Choosing parameters for Random Subspace ensembles for fMRI classification, Proc. Multiple Classifier Systems (MCS'10), Cairo, Egypt, LNCS 5997, 2010, 54-63.
4. Plumpton C. O., L. I. Kuncheva, N. N. Oosterhof and S. J. Johnston, Naive random subspace ensemble with linear classifiers for real-time classification of fMRI data, Pattern Recognition, 45 (6), 2012, 2101-2108.
5. Plumpton C. O., L. I. Kuncheva, D. E. J. Linden and S. J. Johnston, On-line fMRI Data Classification Using Linear and Ensemble Classifiers, Proc. ICPR 2010, Istanbul, Turkey, 2010, 4312-4315.
9
41
70
7
5
![Page 12: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/12.jpg)
Cat Plumpton
PhDreal-time fMRI data analysis
Tom Gardner 10/113rd year projectEnvironments for emotion recognition
Adam Williams 09/103rd year project
Emotion recognition from fMRI data
Jamie Blacker 09/103rd year project
Fractals and fMRI
Joe Freeman 10/113rd year projectfMRI Visualiser
Joey Owen 10/113rd year projectfMRI Voxel selection
Colin Steele 08/093rd year projectfMRI Data analysis
10 20 30 40 50
10
20
30
40
50
10 20 30 40
10
20
30
40
10 20 30 40 50
10
20
30
40
50
50 100 150 200
195
200
205
210
215
![Page 13: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/13.jpg)
2010
End of the fMRI era...
![Page 14: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/14.jpg)
Enter Tom Christy!
Summer 2010
Brain-computer interface through EEG
Game accessories
EEG headsets
Peripheral devices
Accessible
Inexpensive
![Page 15: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/15.jpg)
And just like that...The idea was born...
A game controlled by emotion
Summer 2010
![Page 16: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/16.jpg)
Summer 2010
Sa’ad
Martin
![Page 17: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/17.jpg)
Not as easy as it looked...
Autumn 2010
![Page 18: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/18.jpg)
AFFECTIVE COMPUTING
TAC celebrates its 5th Anniversary
The Galvactivator: A glove that senses and communicates skin conductivity
![Page 19: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/19.jpg)
“Affective Computing is an area of computing that relates to, arises from, or influences emotions.” Rosalind Picard, 1995
HAHV
HALV
POSITIVE
Arousal
LALV
LAHV happy
excited
angry
fearful
content
calm
sad
depressed
PASSIVE
ACTIVE
Valence
NEGATIVE
![Page 20: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/20.jpg)
EXPRESSION OF EMOTION - MODALITIES
facial expression
posture
behaviouralphysiologic
al
peripheral nervous system
central nervous system
EEG
fMRI
Galvanic skin response
blood pressure
skin to
respiration
EMG
speech
gesture
interaction with
the compute
r
pressure on mouse
drag-click speed
eye tracking
fNIRS
pulse rate
pulse variation
dialogue with tutor
![Page 21: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/21.jpg)
facial expression
posture
EEG
fMRI
Galvanic skin response
blood pressure
skin to
respiration
EMGspeech
gesture
pressure on mouse
drag-click-zoom-type
speed
eye tracking
fNIRS
pulse ratepulse variation
dialogue with tutor
![Page 22: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/22.jpg)
![Page 23: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/23.jpg)
Detecting Stress During Real-World Driving Tasks Using Physiological SensorsJennifer A. Healey and Rosalind W. PicardIEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 6, NO. 2, JUNE 2005
The subject wore five physiological sensors:
• an electrocardiogram (EKG) on the chest
• an electromyogram (EMG) on the left shoulder
• a chest cavity expansion respiration sensor (Resp.) around the diaphragm,
• skin conductivity sensor on the left hand
• skin conductivity sensor on the left foot
![Page 24: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/24.jpg)
AMBER Experiment #1
Tom
NeuroSky EEG
Pulse signal
Galvanic skin response
(+) Positive emotionWhale and ocean sounds...
(-) Negative emotion• Michael Jackson• Cheeky girls
![Page 25: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/25.jpg)
3 4 5 6 10 12 15 20
1nn 62.8 64.9 63.4 62.0 61.1 60.9 56.5 59.9
DT 64.2 58.6 67.4 65.9 58.5 62.8 70.0 58.9
RT 60.0 63.0 61.9 62.6 57.1 66.7 66.8 57.1
NB 64.7 63.8 64.5 64.5 65.0 67.8 65.4 61.1
LOG 62.0 60.4 62.6 63.3 59.3 59.2 57.6 57.5
MLP 62.5 59.4 63.3 63.4 63.4 64.2 57.1 58.5SVM-
L 62.1 61.4 63.5 62.4 62.3 59.1 58.7 56.8SVM-
R 50.8 51.2 50.6 50.5 50.2 51.2 51.7 51.3
BAG 65.6 65.6 68.3 67.1 67.4 68.8 66.5 64.4
RAF 64.5 64.7 66.1 65.3 65.9 69.6 67.3 61.6
ADA 63.4 62.2 70.0 67.6 61.1 66.3 73.8 63.3
LB 65.3 62.9 68.8 68.1 62.0 64.0 68.3 60.7
RS 65.0 64.8 66.3 68.2 64.6 67.4 69.0 61.8
ROF 66.9 65.4 66.9 67.2 67.4 69.3 65.5 62.3
Individual classifiers
Ensembles
![Page 26: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/26.jpg)
Serious wired-up man Tom Christy
Very important supervisorLucy Kuncheva
A.M.B.E.R.Advanced Multimodal Biometric Emotion Recognition
NeuroSky
EMOTIV
Nia
My lovely plant
Tom’s R2D2
(show off!)
EDAPulse reader
Cast:
![Page 27: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/27.jpg)
A.M.B.E.R.Advanced Multimodal Biometric Emotion Recognition
Media stars overnight!
![Page 28: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/28.jpg)
HARDWARE
DATA&
CollaboratorsTom Lucy
![Page 29: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/29.jpg)
Tom
HARDWARE201320122011
![Page 30: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/30.jpg)
2013“emotional mouse”
Version 1
EDA sensors(Galvanic skin response)
pulse reader
ReincarnationVersion 2
![Page 31: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/31.jpg)
In the meantime:
Guillaume Thierry
This is what Google returned 2nd on “Guillaume Thierry”!
Me: Would you like to collaborate on emotion recognition from EEG data? Guillaume: Yes, of course, but listen what a fantastic idea occurred to me just now!!!”
Christoph Klein
![Page 32: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/32.jpg)
Stephan Boehm
Let’s give it a go
March 2011
EPSRC proposal
![Page 33: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/33.jpg)
No One is a Prophet in their Own Land
![Page 34: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/34.jpg)
Hello Salzburg!Seminar
May 2012
On-going collaboratio
n with the University of
Salzburg
![Page 35: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/35.jpg)
Hello Juan Rodriguez!
Hello Ramon Mollineda!
![Page 36: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/36.jpg)
happy
elated
excited
alertPOSITIVE AROUSAL
tense
nervous
stressed
upset
UNPLEASANT
sad
depressed
bored
drowsy
NEGATIVE AROUSAL
calm
relaxed
tranquil
contended
PLEASANT
![Page 37: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/37.jpg)
Video: Arch Enemy (My Apocalypse)
Volunteers Participants All
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
6
7
8
9
10
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
6
7
8
9
10
![Page 38: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/38.jpg)
What emotion is being provoked?!?
![Page 39: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/39.jpg)
0 1 2 3 4 5 6 7 8 9 100
1
2
3
4
5
6
7
8
9
10
Valence
Aro
usal
T: Intended emotion
E: EXPERIENCED emotion
P: Reported emotionProvoked?
Acted?
Spontaneous?
Self-reported?
![Page 40: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/40.jpg)
So, WHAT are we RECOGNISING?
1. Emotions are very difficult to define and explicate.2. Experiments for provoking emotion vary considerably, and so do
the results reported in the literature ( from near chance to 95% accuracy).
3. Most emotion measuring modalities are intrusive and annoying.4. Emotions are individual for each person.5. The measured signals are difficult to analyse. There is a
bottleneck of idiosyncratic feature extraction and parameter tuning.
6. There is no unified protocol. 7. Benchmark data collections are not available. 8. There is no consensus about the type of experiment to validate
a hypothesis (provoked, controlled, acted, spontaneous emotion).
![Page 41: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/41.jpg)
Are we doomed?
Maybe not...
We can detect CHANGE in the physiological responses and the EEG, which may be associated with some emotion. If we need to ACT upon detecting an emotion rather than NAMING it, we still may have a chance.
![Page 42: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/42.jpg)
The emotional mouse
Other ingenious input devices
User-friendly EEG headsets
Simple, transparent and generic technologies for feature extraction.
State-of-the-
art data analys
is
LucyTom
Unified protocols, benchmark data
![Page 43: An Emotion Recognition Journey](https://reader036.vdocuments.net/reader036/viewer/2022062722/56813928550346895da0d0a8/html5/thumbnails/43.jpg)
The emotional mouse
Other ingenious input devices
User-friendly EEG headsets
Simple, transparent and generic technologies for feature extraction.
State-of-the-
art data analys
is
LucyTom
Unified protocols, benchmark data
THE END