im2 ga meeting, october 18th, 2011 introducing the recola multimodal corpus of remote collaborative...
TRANSCRIPT
IM2 GA meeting, October 18th, 2011
Introducing the RECOLA Multimodal Corpus of Remote Collaborative and Affective Interactions
F. Ringeval, A. Sonderegger, J. Sauer, D. Lalanne
Department of Informatics – Psychology
Université de Fribourg – Universität Freiburg, Switzerland
2nd International Workshop on Emotion Representation, Analysis and Synthesis in Continuous Time and Space, emoSPACE 2013, April 26th,
2013
Corpus Design
• Why creating a new corpus of emotion?– Idea originally comes from the EmotiBoard project (enhancing
emotional awareness for remote collaborative interactions)
– Context of remote collaboration has not been studied so far
– No existing corpus with bothaudio-visual and physiological data, neither with French speakers
• Objective of the corpus– Provide rich and consistently annotated multimodal data of natural
human behaviour in a context of remote dyadic collaboration
Page 2
Incrustation of emotional feedback into audiovisual data of the SEMAINE database; publication submitted to ACII 2013
emoSPACE 2013, Shanghai, China Fabien Ringeval
Corpus Design
• Videoconference situation (2 persons working together)
• 2 x 2 between subjects design
• Independent variables– Emotion feedback (yes/no): study the impact of EmotiBoard
– Emotion manipulation (positive/negative): increase difference in emotional valence between participants of a team
• Participants– 46 students (58.7% female)
– Mean age: 22 years ± 3 (min: 18, max: 32)
– French speakers with different origins: 33 French, 4 Germans, 8 Italians and 1 Portuguese
Page 3 emoSPACE 2013, Shanghai, China Fabien Ringeval
• EmotiBoard: emotional feedback generation– Vertical interactive surface on which multiple users can interact using
different devices
– Java library to transmit and display as client/server wizard-of-oz ratings of user’s emotion (arousal & valence)
Page 4
Corpus Design
EmotiBoard: emotional feedback generation
emoSPACE 2013, Shanghai, China Fabien Ringeval
Corpus Design
• Collaborative task– As simple as possible, while ensuring that people would be both
motivated and sufficiently involved with regard to their emotions
– Winter survival exercise: 15 items have to be ranked according to their significance for survival in a deserted and hostile area (plane crash)
Page 5 emoSPACE 2013, Shanghai, China Fabien Ringeval
• Procedure– 1st self-report: emotion questionnaire (SAM)
– Individual ranking of the items of the survival task; 10 min.
– Display of a film clip for emotion induction; 5 min.
– 2nd self-report: emotion questionnaires (SAM & PANAS)
– Discussion to agree on the final 15 items’ rank; 20 min.
– 3rd self-report: emotion questionnaires (SAM & PANAS), subjective workload, team collaboration and team satisfaction
Corpus Design
Page 6
SAM’s manikins for arousal SAM’s manikins for valence
emoSPACE 2013, Shanghai, China Fabien Ringeval
Corpus Design
• Participant’s location– Separate rooms in semi basement with thick closed curtains and neon
lighting from the ceiling; kept constant all along sessions
Page 7 emoSPACE 2013, Shanghai, China Fabien Ringeval
Multimodal Recordings
• Audio sensor– HQ unidirectional headset + LQ omnidirectional
microphones (built in webcam)
– External sound cards: (1) Phantom alimentation of microphone, (2) Skype videoconference and (3) biosignals synchronisation
– Recording with Audacity software; 44.1kHz, 16bits
Page 8
Audacity audio recording software Lexicon Omega Studio; external sound card
AKG 520L microphone
emoSPACE 2013, Shanghai, China Fabien Ringeval
Multimodal Recordings
• Video sensor– HD 720p webcam; Logitech C270, 1080x720p, 25Hz
– 2 webcams per participant: Skype and video recording
– LQ audio signal captured for post-synchronisation of HQ audio with video data
– Recording with webcam’s software; gain and contrast fixed once and auto-adjustment turned off
Page 9
Logitech webcam’s recording software
Logitech C270 webcam
emoSPACE 2013, Shanghai, China Fabien Ringeval
Multimodal Recordings
• Physiological sensors– ECG: palm of right hand, right and left inner ankles
– EDA: end of the index and middle fingers
– Biopac MP36 unit and Biopac Student Lab software (BSL Pro); 1kHz
– Synchronisation pulses are emitted each second to the external sound card when recording begins (DB9 output → Mono Jack)
Page 10
Back of the BIOPAC MP 36 unit
EDA sensors
BSL Pro recording software; from top to down: EDA, ECG and RR biosignals
emoSPACE 2013, Shanghai, China Fabien Ringeval
Multimodal Recordings
• Data Synchronisation– Video and HQ audio signal: localisation of a sync event in both HQ and
LQ audio signals + inter-correlation maximisation (20ms); precision of 1ms
– Biosignals and HQ audio signal: synchronisation pulses (right channel) make synchronisation trivial; precision of 1ms
Page 11
Left (audio) and right (sync pulses) channels of HQ signalInter-correlation signal between HQ and LQ audio data
emoSPACE 2013, Shanghai, China Fabien Ringeval
Multimodal Recordings
Page 12 emoSPACE 2013, Shanghai, China Fabien Ringeval
Data Annotation
Page 13
• ANNEMO: ANNotating EMOtions– Web-based annotation interface; Google Chrome web-browser
– Emotional behaviours: arousal and valence (continuous time & values)
– Social behaviours: agreement, dominance, engagement, performance and rapport (discrete time & values)
emoSPACE 2013, Shanghai, China Fabien Ringeval
Data Annotation
Page 14
• Annotation Data Collection– 6 French speaking annotators (3M + 3F) annotated all the corpus
– Oral instructions (4 pages document) + practice on 4 sequences
– Automatic check of annotation data by a dedicated algorithm, e.g., blanks, missing sequences, wrong order of annotation, etc.
– Only the first 5 minutes of interaction were annotated
emoSPACE 2013, Shanghai, China Fabien Ringeval
• Post-processing and analysis– Piece-wise cubic interpolation and binning into 40ms frames
– Local normalisations: zero-mean and synchronization
– Good inter-annotator agreement rate for the affective dimensions,and a fairly good one for the social dimensions
Data Annotation
Page 15 emoSPACE 2013, Shanghai, China Fabien Ringeval
Conclusion
• Conclusion: – RECOLA: a new corpus of REmote COLlaborative and Affective
interactions in French
– 3 well synchronized HQ signals: audio, visual and ECG+EDA
– Rich and consistent annotations of socio-affective behaviours; internal (self-reporting) and external (3M+3F)
– From 27 subjects (5.5h of multimodal data) to 34 subjects (7h of audiovisual data) considering positive consent forms
– ANNEMO: a new web-based annotation tool of emotion
ALL WILL BE PUBLICLY MADE AVAILABLE SOON!
Stay informed on: http://diuf.unifr.ch/diva/recola
Page 16 emoSPACE 2013, Shanghai, China Fabien Ringeval