Information Technology – Dialogue SystemsUlm University (Germany)http://www.dialogue-systems.de

Towards investigating effective affective dialogue strategies

Gregor Bertrand, Florian Nothdurft, Steffen Walter, Andreas Scheck, Henrik Kessler and Wolfgang Minker

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 2

Outline

• Introduction

• Role of Emotion in SLDS

• Purpose of our Study

• Design of our Study

• Current Status

• Conclusion / outlook

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 3

Introduction

• Research on Companion Technologies (www.sfb-trr-62.de)

• Speech dialogue one form of UI

• Role of emotion in dialogue:– Agreement / disagreement– Adapt dialogue style– Need of clarification– Misunderstanding

• Additional modality

• Adapt dialogue to user emotion

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 4

Outline

• Introduction

• Role of Emotion in SLDS

• Purpose of our Study

• Design of our Study

• Current Status

• Conclusion / outlook

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 5

Role of Emotion in SLDS

• Spoken Language Dialogue System architecture

• Extend with emotional capabilities

Spoken language dialogue system

Acoustic front-endSpeech recognition

Linguisticanalysis

Dialoguemanagement

Text generationSpeech synthesis

Application

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 6

Role of Emotion in SLDS

• Emotion based on sensor measurements

• React to user emotion in an appropriate way – but how?

Linguisticanalysis

DialogueManagement

Text generation

Acoustic front-endSpeech recognition

Application

Speech synthesis

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 7

Outline

• Introduction

• Role of Emotion in SLDS

• Purpose of our Study

• Design of our Study

• Current Status

• Conclusion / outlook

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 8

Purpose of our study

• Investigation of emotions relevant for speech interaction

• Investigation of biophysiological features connected to emotion– EEG (electro encephalography)– Blood pressure– Heart rate– Skin conductance level– Breathing

• Explorative testing of emotional strategies

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 9

Outline

• Introduction

• Role of Emotion in SLDS

• Purpose of our Study

• Design of our Study

• Current Status

• Conclusion / outlook

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 10

Design of our Study

• User GUI

• Speech Interface provided by Wizard of Oz architecture

Performance feedback

Timer

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 11

Design of our Study

• Wizard GUI

Predefined feedback

Applicationcontrol

Performancerating

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 12

Design of our Study

• Biophysiological data is taken

• 5 Rounds of Memory

• Each Round is supposed to evoke an emotional state (points in PAD-space)

• Classification of the data

• Emotional dialogue strategies- Thankful- Motivating- Empathetic- Neutral

Pleasure

Arousal

Dominance

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 13

Outline

• Introduction

• Role of Emotion in SLDS

• Purpose of our Study

• Design of our Study

• Current Status

• Conclusion / outlook

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 14

Current Status

• 14 Persons (aged between 20 and 50 years)

• SVM - Classifiers for intraindividual emotional features do well

• SVM - Classifiers for interindividual emotional features do not so well

Individuals show different features on biophysiological level

Dialogue strategies need to adapt to individual emotional situation

subject intrai. pos. intrai. neg. interi. pos. interi. neg. 1 99,4 99,5 52,2 0,92 83,3 94,6 0,1 99,63 100 100 98,7 04 65,4 82,9 1 89,85 85,2 97,2 0,9 43,96 93,3 99,3 0 787 52,7 80,6 0 1008 76,4 88,8 83,9 09 92,8 98,6 64,3 0,1

10 80,2 94,4 65,8 39,811 94,1 99 0,1 51,7

sum 83,9 94,1 33,4 45,8

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 15

Outline

• Introduction

• Role of Emotion in SLDS

• Purpose of our Study

• Design of our Study

• Current Status

• Conclusion / Outlook

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 16

Conclusion / Outlook

• WoZ experiment on user emotion

• Speech interface

• Individuality of user emotion

• Requirements for dialogue management

Design of an appropriate dialogue model

Design of an appropriate dialogue management algorithm

Towards investigating effective affective dialogue strategies| LREC 2010 | 2010/05/20Page 17

Thanks for your attention!

…Questions?