Emotions: Emotions: The Art of Visual Communication The Art of Visual Communication for Virtual Agents.for Virtual Agents.

Emmanuel Tanguy

Philip Willis

Joanna J. Bryson

University of Bath

Department of Computer Science

Reseach funded by a studentship from the Department of Computer Science, University of Bath and the ESPRC grant GR/S/79299/01.

19/01/2006 2

Virtual Actors: Text To AnimationVirtual Actors: Text To Animation

In video games, films, educational software,…

Virtual Actor aims to communicate with humans

Democratise the creation of facial animation Text to speech software + facial animation system

Emotionally Expressive Facial Animation System (EE-FAS) producing visual speech from tagged text

19/01/2006 3

Marvin, the depressed robotMarvin, the depressed robot

The Hitchhiker's Guide to the Galaxy:– Marvin: You can blame the Sirius Cybernetics Corporation for making

androids with GPP...– Arthur: Um... what's GPP? – Marvin: Genuine People Personalities. I'm a personality prototype. You

can tell, can't you...?

<comm_func name="personal reaction" max_nb_fd="2" intensity="60">

You can blame Sirius Cybernetics Corporation for making androids <comm_func name="Emphasizer" intensity="30" >

with GPP...

</comm_func>

</comm_func>

Marvin, depressed?

19/01/2006 4

Levels of Control (LoC)Levels of Control (LoC)

What kind of facial representation can be added to the text?

Levels of Control:

1. Static Graphical Representation (LoC1)

2. Graphical Deformation Commands (LoC2)

3. Facial Movements or Facial Signals (LoC3)

4. Facial Meanings (LoC4) To be introduced in texts It is more intuitive than other LoCs To create visual speech

19/01/2006 5

Facial MeaningsFacial Meanings

Two types of facial meaning:– Emotional (ex: Happy, Sad, etc)– Communicative (question mark, emphasise, etc)

A facial meaning has physical implementations, e.g. facial signals

30 facial meanings and 50,000 facial signals (Terzopoulos and Waters, 1990).

One communicative function can have several facial signals. (Bavelas and Chovil, 2000; Pelachaud and Bilvi, 2003)

How to map a small number of facial meanings onto a large number of facial signals?

19/01/2006 6

Emotional ContextEmotional Context

We decided to use emotional contexts

Emotion models are widely used

In emotion models we distinguish two parts: – Emotion elicitation mechanisms and – Emotion representation

The Dynamic Emotion Representation (DER):– Why only the DER?– What for?

19/01/2006 7

Dynamic Emotion RepresentationDynamic Emotion Representation

Foundations of the DER:– Definitions of emotion types– Sloman model of mind processes– Picard’s description of

• emotion impulses

• emotion intensity

19/01/2006 8

Dynamic Emotion RepresentationDynamic Emotion RepresentationConfiguration File

EmotionalImpulse

EmotionIntensity

19/01/2006 9

Emotional ExpressionsEmotional Expressions

BM

NM

GM

19/01/2006 10

Emotional Context for AnimationsEmotional Context for Animations

• 1.30s: <comm_func name="personal reaction" intensity="60" >

• 5.30s: <comm_func name=“emphasizer" intensity="30">

• 10.00s: <comm_func name="personal reaction"

sec_emo_context="happiness" intensity="60" >

• 13.20s: <comm_func name="question marker" intensity="60" >

Context of happiness

Context of sadness

19/01/2006 11

Meanings of Facial ComponentsMeanings of Facial Components

Purely categorical or componential approach? (Smith and Scott (1997)).

An experiment showing videos with individual facial movements:– Eyebrows frown; Lip corners raise; Eyebrows oblique

People’s perception measured on the following dimensions (60 subjects):– Happy; Angry; Sad; – Pleased/Displeased; Energetic/Lethargic; – Friendly/Unfriendly; Sincere/Insincere

19/01/2006 12

Happy

-3

-2

-1

0

1

2

3

0% 40% 80%

Frown

frown+Smile

smile

smile+em

oblique

oblique+smile

Dimension HappyDimension Happy

T-Test: Paired Samples Test (Happy)

-.66667 .89569 .11563 -.89805 -.43529 -5.765 59 .000

-1.40000 .97772 .12622 -1.65257 -1.14743 -11.091 59 .000

-1.33333 1.01958 .13163 -1.59672 -1.06995 -10.130 59 .000

-.73333 1.11791 .14432 -1.02212 -.44455 -5.081 59 .000

1.46667 1.01625 .13120 1.20414 1.72919 11.179 59 .000

1.16667 1.01124 .13055 .90544 1.42790 8.937 59 .000

Neutral - Frown_SmilePair 1

Neutral - SmilePair 2

Frown - SmilePair 3

Frown_Smile - SmilePair 4

Smile - ObliquePair 5

Smile - Oblique_SmilePair 6

Mean Std. DeviationStd. Error

Mean Lower Upper

95% ConfidenceInterval of the

Difference

Paired Differences

t df Sig. (2-tailed)

Lip corners raised involved in happiness expressionSmith and Scott (1997)

19/01/2006 13

Angry

-3

-2

-1

0

1

2

3

0% 40% 80%

Intensity of the expression

Sco

re

Frown

frown+Smile

smile

oblique

oblique+smile

Dimension AngryDimension Angry

T-Test: Paired Samples Test (Angry)

-1.43333 .88999 .11490 -1.66324 -1.20342 -12.475 59 .000

-1.01667 1.22808 .15855 -1.33391 -.69942 -6.412 59 .000

.41667 1.16868 .15088 .11476 .71857 2.762 59 .008

1.56667 .87074 .11241 1.34173 1.79160 13.937 59 .000

1.50000 .91132 .11765 1.26458 1.73542 12.750 59 .000

1.50000 .85371 .11021 1.27946 1.72054 13.610 59 .000

1.15000 1.10200 .14227 .86532 1.43468 8.083 59 .000

1.08333 1.13931 .14708 .78902 1.37765 7.365 59 .000

1.08333 1.07816 .13919 .80482 1.36185 7.783 59 .000

Neutral - FrownPair 1

Neutral - Frown_SmilePair 2

Frown - Frown_SmilePair 3

Frown - SmilePair 4

Frown - ObliquePair 5

Frown - Oblique_SmilePair 6

Frown_Smile - SmilePair 7

Frown_Smile - ObliquePair 8

Frown_Smile -Oblique_Smile

Pair 9

Mean Std. DeviationStd. Error

Mean Lower Upper

95% ConfidenceInterval of the

Difference

Paired Differences

t df Sig. (2-tailed)

Frown involved in anger expressionSmith and Scott (1997)

19/01/2006 14

Sad

-3

-2

-1

0

1

2

3

0% 40% 80%

Intensity of the expression

Sco

re

Frown

frown+Smile

smile

oblique

oblique+smile

Dimension SadDimension Sad

T-Test: Paired Samples Test (Sad)

-1.76667 1.24010 .16010 -2.08702 -1.44631 -11.035 59 .000

-.78333 1.30308 .16823 -1.11996 -.44671 -4.656 59 .000

.98333 1.38383 .17865 .62585 1.34081 5.504 59 .000

-1.95000 1.18501 .15298 -2.25612 -1.64388 -12.746 59 .000

-2.03333 1.11942 .14452 -2.32251 -1.74416 -14.070 59 .000

-2.08333 1.13931 .14708 -2.37765 -1.78902 -14.164 59 .000

Neutral - ObliquePair 1

Neutral - Oblique_SmilePair 2

Oblique - Oblique_SmilePair 3

Frown - ObliquePair 4

Smile - ObliquePair 5

Frown_Smile - ObliquePair 6

Mean Std. DeviationStd. Error

Mean Lower Upper

95% ConfidenceInterval of the

Difference

Paired Differences

t df Sig. (2-tailed)

19/01/2006 15

Dimensions Energy/PleasedDimensions Energy/Pleased

From Russel, J. A. and J. M. Fernandez-Dols (1997). The Psychology of Facial Expression.Cambridge University Press.

Emotional Disque

-2

-1,5

-1

-0,5

0

0,5

1

1,5

2

-2 -1,5 -1 -0,5 0 0,5 1 1,5 2

Unpleaseed/PleasedLet

har

gic

/Ener

get

ic

Frown

Smile

Oblique

neutral

19/01/2006 16

Friendly/Unfriendly

-3

-2

-1

0

1

2

3

0% 40% 80%

Intensity of the expression

Sco

re

Frown

frown+Smile

smile

oblique

oblique+smile

Dimension FriendlyDimension Friendly

T-Test: Paired Samples Test (Friendly/Unfriendly)

-2.55000 1.38301 .17855 -2.90727 -2.19273 -14.282 59 .000

-1.56667 1.29362 .16701 -1.90084 -1.23249 -9.381 59 .000

-1.91667 1.16868 .15088 -2.21857 -1.61476 -12.704 59 .000

-2.20000 1.69546 .21888 -2.63798 -1.76202 -10.051 59 .000

-1.21667 1.82350 .23541 -1.68773 -.74561 -5.168 59 .000

-1.56667 1.92545 .24857 -2.06406 -1.06927 -6.303 59 .000

Frown - SmilePair 1

Frown - ObliquePair 2

Frown - Oblique_SmilePair 3

Frown_Smile - SmilePair 4

Frown_Smile - ObliquePair 5

Frown_Smile -Oblique_Smile

Pair 6

Mean Std. DeviationStd. Error

Mean Lower Upper

95% ConfidenceInterval of the

Difference

Paired Differences

t df Sig. (2-tailed)

19/01/2006 17

Sincere/Insincere

-3

-2

-1

0

1

2

3

0% 40% 80%

Intensity of the expression

Sco

re

Frown

frown+Smile

smile

oblique

oblique+smile

Dimension SincereDimension Sincere

T-Test: Paired Samples Test (Sincere)

-.13333 1.68208 .21716 -.56786 .30119 -.614 59 .542

1.01667 1.92655 .24872 .51899 1.51435 4.088 59 .000

.61667 2.09997 .27111 .07419 1.15915 2.275 59 .027

-.76667 1.56624 .20220 -1.17127 -.36206 -3.792 59 .000

.03333 1.65669 .21388 -.39464 .46130 .156 59 .877

.80000 1.64471 .21233 .37513 1.22487 3.768 59 .000

1.15000 1.72543 .22275 .70427 1.59573 5.163 59 .000

-.40000 2.29369 .29611 -.99252 .19252 -1.351 59 .182

-.58333 2.21162 .28552 -1.15465 -.01201 -2.043 59 .046

Neutral - FrownPair 1

Neutral - Frown_SmilePair 2

Neutral - SmilePair 3

Neutral - ObliquePair 4

Neutral - Oblique_SmilePair 5

Oblique - Oblique_SmilePair 6

Frown - Frown_SmilePair 7

Frown_Smile - SmilePair 8

Smile - Oblique_SmilePair 9

Mean Std. DeviationStd. Error

Mean Lower Upper

95% ConfidenceInterval of the

Difference

Paired Differences

t df Sig. (2-tailed)

19/01/2006 18

EE-FAS ArchitectureEE-FAS Architecture Modular Customisable DER Message passing mechanism Implementation of all LoCs:

– LoC1: 3D Mesh– LoC2: Abstract Muscles– LoC3: Facial Signals– LoC4: Emotions and

Communicative functions

Customisable transformations from one LoC to another

Loose relation between LoCs

19/01/2006 19

Conclusion (1)Conclusion (1)

We developed a Dynamic Emotion Representation (DER) model that enables users– To create their own Emotion Representation based on

different emotion theories.– To represent any interacting variable, such as drives

We integrated an instance of the DER model within the EE-FAS

Technical Report on the DER:“A Dynamic Emotion Representation Model Within a Facial Animation System”, 2005, Emmanuel Tanguy, Joanna Bryson, Philip Willis, University of Bath

19/01/2006 20

Conclusion (2)Conclusion (2)

Developed an Emotionally Expressive Facial Animation System (EE-FAS), which:– Enables non-specialist users to create facial animations– Is customisable through XML dictionaries

EE-FAS extends the number of facial expressions that a such system can produce by displaying:– Emotional expressions– Multiple facial signals for each communicative function– Fake or genuine facial expressions