The Display of Affect through Physical Behavior

CS 543

Overview

Motivation

Psychological View of Nonverbal Behavior

• Encoding Vs. Decoding

• Expression of Emotion through Nonverbal

Behavior

Realizing Emotional Expression

• Animation Techniques

• The Expressive Gaze Model

Conclusion

Motivation

What is interesting about the display of

emotion?

The development of Virtual characters

• Believable

• Capable of open-ended interaction

• Engaging

• For this, a character must be able to signal

its internal state through its external

behavior

“It is the simple idea that "lifelike" doesn't mean "has movement"; "lifelike" means "has a brain." The underlying notion of Pixar and Disney animation is that action is driven by cognitive processes in the character, that there is intelligence and personality and emotion.

Remember that it is crucial that the audience understand the intelligence and personality and emotion. The continual challenge to the animator is in depicting in an unmistakable yet compelling way that the brain is driving the action.”

-Tom Porter

Visual Effects Supervisor

Pixar Studios

Motivation – Continued

Using nonverbal display of emotion to

recognize affective state

• Improve Human-Computer Interaction

• Reliable

• Requires user modeling/theory of mind

Overview

Motivation

Psychological View of Nonverbal Behavior

• Encoding Vs. Decoding

• Expression of Emotion through Nonverbal

Behavior

Realizing Emotional Expression

• Animation Techniques

• The Expressive Gaze Model

Conclusion

Nonverbal Behavior

Encoding

• The display of information through

external behavior

Decoding

• The comprehension of information

through observation of external behavior

Encoding

Types of encoding

• [Ekman & Friesen, 1969] define three

types of coding

• Arbitrary

• No relation between coding and meaning

• Iconic

• Coding is representation of meaning

• Intrinsic

• Coding is meaning

Encoding Research

1. Induce an emotional state into a subject

2. Evaluate the emotional induction.

3. Evaluate the subject‟s behavior within a

specific context

4. Compare the resulting behavior to a

subject that has not gone through the

emotional induction

Encoding – Complications

Encoding may not be deliberate

Encoding may be used to mask information

Men and women encode & decode differently

Different cultures have different coding rules

Encoding may be idiosyncratic

Evaluating Emotion in humans is problematic

• How do we know what lies beneath the

display we‟ve just seen?

Decoding Research

1. Produce stimuli of portrayed nonverbal

behaviors

2. Have subjects rate the emotional content

of the stimuli

Realistic vs. Believable

Decoding research is easier than Encoding research

Generating Believable expressive behavior is easier than generating Realistic expressive behavior

• “Realistic” -> behavior as a human would have performed it

• “Believable” -> behavior understandable by an observer

Overview

Motivation

Psychological View of Nonverbal Behavior

• Encoding Vs. Decoding

• Expression of Emotion through

Nonverbal Behavior

Realizing Emotional Expression

• Animation Techniques

• The Expressive Gaze Model

Conclusion

Nonverbal Communication

Technically, communication without words

• Extremely broad field [Knapp & Hall, 1997]

• Includes:

• Environment

• Appearance

• Dress, grooming

• Height, weight

• Race & gender

• None of these really express emotion

• However, they may affect appraisal ratings

Expression of Emotion

What nonverbal behaviors display emotion?

• Torso Posture

• Head Position

• Facial Expression

• Hand Gestures

• Gaze Behavior

• Prosody

• Movement

Expression of Emotion in Man and Animals [Darwin, 1872].

Posture - Demonstration

Posture - Demonstration

Posture - Demonstration

Torso Posture

Torso Posture is affected less by emphasis

and other speech-related behaviors [Bull

& Connelly, 1985]

Positive and Negative emotions are both

easily distinguished both from static body

postures [Coulson, 2004], [Schouwstra &

Hoogstraten, 1995], and from dynamic

trunk movement [de Meijer, 1989].

Expression of Emotion

What nonverbal behaviors display emotion?

• Torso Posture

• Head Position

• Facial Expression

• Hand Gestures

• Gaze Behavior

• Prosody

• Movement

Head Position

Head Position

Movement of the head is tightly connected

to speech [Hadar et al., 1985],

[McClave, 1999].

Static posture of the head is still useful for

the display of emotions

• Head which is tilted upwards is viewed

as more dominant, and displaying a

more positive than one tilted downwards

[Kappas et al., 1994], [Mignault &

Chaudhuri, 2003].

Expression of Emotion

What nonverbal behaviors display emotion?

• Torso Posture

• Head Position

• Facial Expression

• Hand Gestures

• Gaze Behavior

• Prosody

• Movement

Facial Expression

Facial expression is one of the stronger channels for emotional expression

Paul Ekman

• Performed many of early experiments on facial expression

• Found that facial expression was highly cross cultural

• Developed the Facial Action Coding System (FACS)

Happiness

Anger

Sadness

Fear

Disgust

Facial Expression

Other researchers have shown that

• Children who are Deaf/Blind from birth

share many of the same facial

expressions

• As do very young infants

• As well as monkeys

• [Knapp & Hall, 1997]

Expression of Emotion

What nonverbal behaviors display emotion?

• Torso Posture

• Head Position

• Facial Expression

• Hand Gestures

• Gaze Behavior

• Prosody

• Movement

Hand Gestures

Much of the research on hand gesture focuses

on the association of gestures with the

cognition and communication [McNeill, 1992].

However, there is still evidence that not only the

motion qualities of hand gestures, but the

selection of gestures and hand positions can

be affected by emotional state.

Expression of Emotion

What nonverbal behaviors display emotion?

• Torso Posture

• Head Position

• Facial Expression

• Hand Gestures

• Gaze Behavior

• Prosody

• Movement

Gaze Behavior

What do we mean by gaze behaviors?

• Eye shape and eyebrow position are part of facial expression

• Speed of eye movement is highly stereotypical

• Length of gaze & mutual gaze

• Gaze attraction vs. aversion

• Relation of eye movement to head and body movement

Non-Emotional Gaze

Gaze is closely linked to speech-related

behaviors

• Patterns of gaze for individuals speaking

are often different from patterns of gaze

for listeners [Kleinke, 1986].

Attention and Saliency also control eye

movement [Argyle & Cook, 1973].

Gaze and Dominance

Gaze is very important in signaling

dominance [Exline, 1971]

• However this relationship is not fully

understood

• Seen in the way monkeys perform

dominance displays when looked at by

experimenters

• Highly dominant individuals look less

while listening, and more while speaking

than do individuals of low dominance

[Knapp, 1997].

Gaze and Arousal

Gaze is also closely related to physiological

arousal [Argyle & Cook, 1973]

• Receiving gaze and engaging in mutual

gaze increases arousal

• Gaze is used as a regulation for

physiological arousal

• Increased blinks, and increased pupil

dilation are also strong signals of arousal

Gaze and Preference

Gaze also has a drastic effect on liking

• Individuals who gaze very little at people

they interact with are rated lower on

scales of preference, credibility,

attractiveness, and are less likely to be

hired [Burgoon et al., 1985], [Argyle &

Cook, 1973].

Individuals are also more likely to look at

people that they like [Argyle & Cook,

1973]

Expression of Emotion

What nonverbal behaviors display emotion?

• Torso Posture

• Head Position

• Facial Expression

• Hand Gestures

• Gaze Behavior

• Prosody

• Movement

Prosodic Information

Nonverbal information contained in the voice.

Often examined through a signal processing

paradigm [Bulut, Lee, Narayanan, ‟07].

Expression of Emotion

What nonverbal behaviors display emotion?

• Torso Posture

• Head Position

• Facial Expression

• Hand Gestures

• Gaze Behavior

• Prosody

• Movement

Movement & Locomotion

Emotion can be recognized through how

people walk

But further, emotion can be recognized

through the movement of simple

geometric shapes [Rime et al., 1985]

Movement

Locomotion

Synchrony

The interactions of all of these behaviors is

as important as any single behavior

• If not, emotional signals lack coherency

• The ability of users to recognize emotion

decreases

• Believability and engagement decrease

Conclusion – Nonverbal

While there are many different ways emotion can be signaled through nonverbal behavior

• Often difficult to clearly differentiate between individual emotional categories

• Much nonverbal behavior research done using dimensional models of emotion that are derived from factor analyses of nonverbal behavior results

Overview

Motivation

Psychological View of Nonverbal Behavior

• Encoding Vs. Decoding

• Expression of Emotion through Nonverbal

Behavior

Realizing Emotional Expression

• Animation Techniques

• The Expressive Gaze Model

Conclusion

Virtual Characters

Keyframe Animation

• Classic technique from 2D cell animation

• Skilled animator develops key frames

• Interpolation between them creates movement

• Slow, iterative process

Produces most realistic behavior

Takes very long time to develop

Animations are not reusable

Keyframe Animation

Skilled animators drew

key frames

•Later drew in “in-

betweens”

Developed principles

such as “Squash &

Strech”, “Anticipation”,

and “Follow Through”

Keyframe Animation

Extremely slow process

• Highly iterative

• Highly intuitive

• Very difficult to clearly explain

• Requires great deal of training, and a

certain amount of talent

• Results are excellent

Analyzing 2D Animation

[Lance et al., 2004], and [Chafai et al.,

2006] were attempts to analyze keyframe

animation for information on nonverbal

behavior for gaze and gesture,

respectively

• Moderately successful

• Extremely labor intensive

KeyFrame Animation

Motion Capture

Motion Capture Animation

• Cameras record movement of sensors placed on an

actor

• Software reconstructs movement and

approximation of character‟s anatomy

• Allows for quicker generation of movement

• Requires expensive hardware and analysis tools

• Captured sequences difficult to reuse

• Can be difficult to retarget captured motion to 3D

model being animated

Motion Capture

Procedural Animation

Generation of movement without human oversight

• Virtual Agents

• Video Game Characters

• Can use libraries of keyframe or motion capture animation

• Or movement can be generated through mathematical methods

• Allow for dynamic or interactive environments

• Often lack the quality of previous two methods

Motion Capture Library

Procedural Animation

Overview

Motivation

Psychological View of Nonverbal Behavior

• Encoding Vs. Decoding

• Expression of Emotion through Nonverbal

Behavior

Realizing Emotional Expression

• Animation Techniques

• The Expressive Gaze Model

Conclusion

Gaze Model:

Expressive Gaze Model (EGM)

• Combination of Motion Capture and Procedural animation techniques

• Motion Capture for Head & Torso• [Lance & Marsella, IVA „07]

• Procedural for Eye Movement• [Lance & Marsella, AAMAS „08]

• Produces integrated Eye, Head, and Torso movement for emotionally expressive gaze

The Expressive Gaze Model

-

+

Emotional Movement

Non-Emotional Movement

Procedural Eye Movement

Animated Gaze Shift

+

New Non-Emotional

Movement

Gaze

Warping

Transformation

EGM:

Eye vs. Body Movement

• Use separate methods for modeling the head/torso movement and the eyes.

• Eye movement by itself stereotypical and not expressive

• Eye and head/torso movement function on different time scales with different constraints

• Chosen to model eye and head/torso movement differently, while constraining the relationship between them

GWT:

Neutral Gaze Shift

GWT:

Applied GWT

EGM:

Eye Model Overview

• Procedural model of eye movement

• Basic Eye behaviors

• Integration with head movement

• Based on constraints described in Visual Neuroscience Literature

• Leigh & Zee, 2006; Stark et al., 1980; Zee, 1976; Guitton & Volle, 1987; Gresty, 1974; Uemura et al., 1980

• Relationship between saccades, VOR, and head movement

Eye Model Example

Gaze Model

• Generate gaze to arbitrary location while displaying

desired gaze behaviors

• Provides property of composability of behavior

• Exploited this property to explore relationship

between behavior and emotion

• Using the P-A-D dimensional model of emotion,

found a strong relationship between behaviors and

individual emotional dimensions

• Revealed complexity of multidimensional PAD

space

Conclusion

Many sources of information for the expression of

emotion through human behavior

• Lots of them are qualitative

• Or not experimentally verified

• Or describe more subtle relationships than

direct emotional state -> behavior

• Picture is still very incomplete

Still lots of good information for a starting point!

References

Argyle, M., Cook, M. Gaze and Mutual Gaze. Cambridge University Press. 1976.

Bull, P., Connelly, G. Body Movement and Emphasis in Speech. Journal of Nonverbal Behavior, 9(3). Fall 1985.

Bulut, M., Lee, S., Narayanan, S.: Analysis of emotional speech prosody in terms of part of speech tags. Interspeech, Antwerp, Belgium, August 2007.

Burgoon, J., Manusov, V., Mineo, P., Hale, J. Effects of Gaze on Hiring, Credibility, Attraction and Relational Message Interpretation. Journal of Nonverbal Behavior. 9(3). Fall 1985.

Chafai, N., Pelachaud, C., Pele, D., Breton, G. Gesture Expressivity Modulations in an ECA Application. Proceedings of the Intelligent Virtual Agents IVA 2006 Conference. 2006.

Coulson, M. Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence. Journal of Nonverbal Behavior. 28(2). Summer 2004.

Chi, D., Costa, M., Zhao, L., Badler, N. The EMOTE Model for Effort and Shape. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing, New York, NY. 2000.

Darwin, C., The Expression of Emotions in Man and Animals. 1872.

Ekman P., Friesen, W.V., The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding. Semiotica. 1, 49-98. 1969.

Exline, R. Visual Interaction: The Glances of Power and Preference. In Weitz, S. (ed), Nonverbal Communication: Readings with Commentary. Oxford University Press, 1974.

Hadar, U., Steiner, T. J., Rose, F. C. Head Movement During Listening Turns in Conversation. Journal of Nonverbal Behavior, 9(4). Winter 1985.

Kappas, A., Hess, U., Barr, C., Kleck, R. Angle of Regard: The Effect of Vertical Viewing Angle on the Perception of Facial Expressions. Journal of Nonverbal Behavior. 18(4). Winter 1994.

Kleinke, C. Gaze and Eye Contact: A Research Review. Psychological Bulletin. v. 100, n. 1. 1986.

References (Cont.)

Knapp, M., Hall, J. Nonverbal Communication in Human Interaction. Fourth Edition. Harcourt Brace Publishers. 1997.

Lance, B., Marsella, S., Koizumi, D. Towards Expressive Gaze Manner in Embodied Virtual Agents. AAMAS Workshop on Empathic Agents. 2004.

Lance, B., and S. C. Marsella. 2007. Emotionally Expressive Head and Body Movement During Gaze Shifts. Lecture Notes in Computer Science 4722: 72.

Lance, B. J., and S. C. Marsella. 2008. A Model of Gaze for the Purpose of Emotional Expression in Virtual Embodied Agents. International Foundation for Autonomous Agents and Multiagent Systems Richland, SC , 199-206.

Lasseter, J. Principles of Traditional Animation Applied to 3D Computer Animation. Computer Graphics. v. 21, n 4. July 1987.

Marsella, S., Carnicke, S. M., Gratch, J., Okhmatovskaia, A., Rizzo, A. An Exploration of Delsarte‟s Structural Acting System. Proceedings of the Intelligent Virtual Agents IVA 2006 Conference. 2006.

McClave, E. Linguistic Functions of Head Movements in the Context of Speech. Journal of Pragmatics. v. 32, 2000.

de Meijer, M. The Contribution of General Features of Body Movement to the Attribution of Emotions. Journal of Nonverbal Behavior. 13(4). Winter 1989.

Mignault, A., Chaudhuri, A. The Many Faces of a Neutral Face: Head Tilt and Perception of Dominance and Emotion. Journal of Nonverbal Behavior. 27(2). Summer 2003.

Pollick, F., Paterson, H., Bruderlin, A., Sanford, A. Perceiving Affect from Arm Movement. Cognition. i 82, B51-B61. 2001.

Porter, T. “Creating lifelike characters in Toy Story,” ACM SIGART Bulletin 8 (1997): 10-14.

Rime, B., Boulanger, B., Laubin, P., Richir, M., Stroobantst, K. The Perception of Interpersonal Emotions Originated by Patterns of Movement. Motivation and Emotion. v. 9, n. 3. 1985.

Schouwstra, S., Hoogstraten, J. Head Position and Spinal Position as Determinants of Perceived Emotional State. Perceptual and Motor Skills. v. 81, 1995.

Thomas, F., Johnston, O. The Illusion of Life: Disney Animation. Walt Disney Productions. 1981.

Zhao, L., Badler, N. Acquiring and Validating Motion Qualities from Live Limb Gestures. Graphical Models. v. 67 i. 1. January, 2005.