multicomponent analysis of emotional experience
DESCRIPTION
Multicomponent analysis of emotional experience. M. Mortillaro University of Milan - Bicocca. emotions as multicomponent processes . Reactions to goal-relevant changes in the environment according to different organismic subsystems, that answer functions reflected in five main components - PowerPoint PPT PresentationTRANSCRIPT
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
Multicomponent analysis of emotional experience
M. MortillaroUniversity of Milan - Bicocca
2
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
emotions as multicomponent processes
Reactions to goal-relevant changes in the environment according to different organismic subsystems, that answer functions reflected in five main components
• cognitive appraisal component• subjective feeling component• physiological component• motor expression component• motivational component
(Scherer, 1984, 1987, 2000)
3
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
Most of traditional studies considered only one modality or one component. Authors showed how difficult is the linkage between emotional states and one single modality
In order to overcome these difficulties, emotions should be addressed multimodally, in the sense that signs may appear at the same time in different channels
4
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
• Multimodal• Multicomponent
Ideally research should include all the components at the same time (through physiological measures, voice, gestures, facial expressions, brain activity, self-reports...)
5
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
Difficulties mainly concern how to build an empirical procedure to obtain all these measures in a reliable way (database)
GEMEP
Define one procedure to have multimodal data to be analyzed
6
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
objectives
• Multimodally investigate emotions within a unique research procedure
• Perform cross-component investigation to support the conceptualization of emotion as a whole made by different components
• Suggest multicomponential investigation as an effective way to improve automatic emotion recognition
7
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
• how to obtain multimodal emotional data• which components can be detected• how to synchronize measures• how to integrate them
8
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
how to obtain multimodal dataCombine Velten procedure with standard paradigm
procedure.
Ten narrations (scenarios), each characterized by an univocal emotional episode with a part in first person speech, written in order to describe a situation that can be appraised as joy, anger, etc.
ValidationCultural grounded labels of emotions (script)
9
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
Participants were asked to read aloud trying to identify with the main character (contextualized acting). Velten procedure adding a contextual dimension through narration
Similarity to the work running on in Geneva
10
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
Controlled “in laboratory” situation: to have more reliable values for physiological parameters
Naive participant: non professional emotional expression
11
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
components detectedAlso deals with features to extract
We acquired from the same sequence, simultaneously
• PhysiologicalHeart Rate, HR; Skin conductance, SC; Respiration Rate, RR; Respiration
Amplitude, RA, Finger Blood Amplitude, BA; Electromyography of the extensor muscle of the forearm, EMG); Finger temperature, Temp; PROCOMP (Thought Technology Ltd.)
• ExpressiveFacial expressions FACS and Theme (Noldus)* Vocal acoustic parameters time (total and partial duration, pause, speech and
articulation rate), fundamental frequency (F0) and intensity (mean, sd, range, min and max); CSL (Kay Elemetric Ltd.)
*Facial expressions are not considered within results herein reported
12
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
setting
13
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
how to synchronize those measures
• Texts including a standard utterance. Both vocal, facial and physiological measures considered are extracted during its speech
• timelines can be overlapped• standard utterance lasts between 1 and
1.5 seconds according to the speaker and to the emotion. This short duration allows to consider mean values, but it is also possible to consider signals contour
14
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
NARRATION
Scenario description
First person speech
standard utterance
15
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
how to integrate the information
Measures belonging to both components should be jointly analyzed statistically.
Correlation, patterning, regression
For emotion recognition• Discriminant analysis• Advanced classification algorithms (decision
tree, k-nearest neighbour, bayesian networks) within WEKA environment
16
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
procedure• 34 participants x 10 emotions X 50 measures• introduced in a laboratory setting, briefed about
the sensors and gave consent before being cabled
• baseline for physiological parameters was measured (Berntson, Uchino, Cacioppo, 1994)
• narrations presented in randomized order• read a first time silently figuring out the situations
described, then reading aloud in a natural and spontaneous way trying to identify themselves with the character
17
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
preliminary analysis • Physiological measures: ANOVA statistics showed
significant main effect of emotion in mean values of different measures (SC, Respiration, BA).
• Vocal features: ANOVA statistics showed significant main effect of emotion in every measure considered (except Pause).
• Post hoc analysis showed results mainly consistent with scientific literature, more for the so-called primary emotions included. More problems with physiological measures.
18
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
cross-component correlations We found slight significant cross-component correlations of
measures (acquired in the same time). It suggests that different modalities jointly work to form the emotional experience, showing correspondence in indices variations, but each of them keeping a specific contribution
In particular, concerning correlations among physiological measures and vocal features, articulation rate, variations of F0 and Intensity are clearly reflected in respiration measures. Furthermore, F0 and Intensity correlate with Skin Conductance: these results can be read as reflecting the physiological arousal level
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
19
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
discriminant analysis • Running discriminant analysis for the ten emotions only
on physiological measures it is indicated an overall percentage of 28.4% of correctly reclassified cases
• Including only vocal measures it is obtained an overall percentage of 30.1%
• When all these measures are used at the same time, discriminant analysis outcomes a percentage of an overall correct classification that raises to 38.8% (10% expected by chance). Furthermore, considering 8 out of 10 emotions, the overall correct recognition percentage increases to 47.0%
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
20
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
Advanced classification algorithms are currently being trained on the database
Decision treeBayesian networksK-nearest-neighbour
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
21
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
conclusion
• Preliminary data supported a multicomponent perspective: most of the measures seem clearly influenced by emotional states,using contextualized acting method
• Correlations suggest that considering more components at the same time can provide a clearer definition of emotional experience. Further analyses are needed.
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
22
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
limits • Contextualized acting should be empowered in order to
obtain more wide physiological effects (longer narrations, more detailed character, assessment of transportation tendency of participants)
• Synchronization is still hand-made
• Facial expressions are influenced by reading task
• Wider sample of participants is needed for classification algorithms
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
23
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
Future work
• testing of learning algorithms • integration of facial expressions analysis• contour analysis
24
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
- Multicomponent approach- Questions (procedure, db, features,
synchronization, analysis…)- Attempt
25
Genova, September 2006 – Humaine Summer School – Workshop on synchronization
Thank you