Affective computing and the role of emotions in computer

science and engineering

Antonio Camurri

Corso di Progettazione e Produzione Multimediale

Goals• Make computers able to - recognize emotions - express emotions - respond intelligently to human emotion - regulate and utilize human emotions

• Enormous potential for – progress in science (understanding of cognitive and emotional

phenomena)– novel computer applications – novel (anthropomorphic) interfaces to existing applications.

• Main “schools”:– (USA) “Affective Computing”, Rosalind W. Picard, 1997, MIT

Press.– (Japan) “KANSEI Information Processing”, Intl. Workshop, 1997.– (EU) Emotion research (Scherer) , Empathy and entrainment

Applications• Applications where computers have a social role (“instructor”,

“tutor”,“helper”, “companion”): to be able to recognize users’ emotions may enhance their functionality.

• Rehabilitation and therapy software: e.g. for autistic children, for monitoring motor and cognitive tasks in patients affected by Parkinson or Alzheimer.

• Affective robots (anthopomorphic or not): emotional behavior to reduce stress, fear, and increase trust in users. For example: support to independent living in elder people (home activities, intelligent prosthesis),

• ECAs (Embodied Conversational Agents): model relationships between emotions and behaviour can help to synthesize emotion-oriented ECAs

• To improve planning and decision systems: emotions influence decision processes: e.g., in game theory: emotion models to boost performance of existing techniques and algorithms.

• Human Computer Interaction (HCI): To measure cognitive and motoric performance in users of computer applications

• To monitor users’ stress level • Social networked media, games, and entertainment• Artistic applications• Edutainment, children educational software and affective robots• Security systems: behavior monitoring of individuals and crowd in public

spaces (e.g. airports)

Emotion theories• Emotions are - cognitive: mental component of emotion is emphasized; appraisal

theories

- physical: bodily component is emphasized; physiological

responses

• Difficulties to define and ambiguous: “emotion” can mean different things, hard to define scientifically:– “I feel happy” (feeling an emotion)– “I want to make understand to others happiness” (cognitive, the

tactor viewpoint in a theatre play)– “I feel engaged with someone” (entrainment, empathy)– “music or artistic emotions” (emotions in audience, between

musicians playing together)

Emotion theories

Computational models and related affective software systems are grounded on results from other disciplines:

• From psychology and cognitive sciences:– discrete emotion theories (basic emotions: e.g. joy, sadness, fear,

anger)– dimensional theories, continuous models (e.g. valence/arousal

spaces)

• From neurosciences:– Novel mind and brain theories: embodiment (enaction, active

perception, mirror neurons, etc.)

• From physics and complex dynamic system theories: – empathy and entrainment: based on theories of synchronisation.

Open problem: how to model an emotional process?

• Psychological models• Physiological models

• Computational models of emotions: – analysis of emotions– synthesis of emotional behavior (in robots, software agents).

• Important for successful approaches: emotions do not refer (only) to states, it is fundamental to take into account the dynamics of an emotional process

The Component-Process Model (from psychology) (1)

• Developed by Klaus Scherer (1984, 2000, 2001; Geneva Emotion Research Group, Dept. Psychology, Univ. Geneve)

• Emotions as constantly changing phenomena integrating more components

• Five subsystems - the cognitive system (appraisal) - the autonomic nervous system (arousal) - the motor system (expression) - the motivational system (action tendencies) - the experiential system (feeling)• Synchronization of all the processes

Component-Process Model (2)

• Emotional processes arise from:

- Appraisal process

- Memory

- Empathy

Component-Process Model (3)

• Corpora for studying and analysing emotions:– How to validate algorithms for emotion analysis and recognition?

• The GEMEP Archive: an annotated collection of short video recordings of emotions, portraied by professional theatre actors, following the Scherer’s emotion model

• (video demo)

Building affective and emotional computing systems

• Affective signals and systems

• Recognizing and expressing affect: from analysis to synthesis

• Validating affective computing systems

• Emotion synthesis

Affective signals and systems

The physiology of emotions

• The limbic system (amygdala) is involved in emotional processes - Through its direct connections with the hypothalamus, it modulates the autonomic

nervous system activity

- It receives information from the thalamus

- Low and high roads to amygdala (LeDoux, 1996)

Sensory cortex

Amygdala

Emotional stimulus Emotional responses

High road

Low road

Sensorythalamus

Properties of affective signals (Picard, 1997)

• Response decay

An emotional response is of relatively short duration, and will fall below a level of perceptibility unless it is re-activated

Properties of affective signals

• Repeated strikes

Rapid repeated activation of an emotion causes its perceived intensity to increase

Properties of affective signals

• Temperament and personality influences

A person’s temperament and personality influence emotion

activation and response

Properties of affective signals

• Non-linearity

The human emotional system is non-linear, but may be approximated as a linear system for a certain range of inputs and outputs

Properties of affective signals

• Time-invariance

The human emotional system can be modeled as independent of time for certain durations

Properties of affective signals

• Activation

Not all inputs can activate an emotion; they have to be of sufficient intensity. This intensity is not a fixed value, but depends on factors such as mood, temperament, cognitive expectation, context

Properties of affective signals

• Saturation

No matter how frequently an emotion is activated, at some point the system will saturate and the response of the person will no longer increase. Similarly, the response cannot be reduced below a “zero” level

A typical representation for emotion signals

• The input of an emotional system is a complex function of cognitive and physical events

• It is often approximated by a simple nonlinear function applied to the inputs to the emotional system:

• X = input stimuli (originating inside and/or outside the person); • Y = the sigmoid function; • S = controls the steepness of the slope (depends on personality); • x0 = shifts the sigmoid left or right; • g = controls the gain applied by the sigmoid; • y0 = shifts the sigmoid up or down

• Different activation and saturation levels

00

1

y

e

gy

s

xx

Properties of affective signals

• Cognitive and physical feedback

Inputs to the system can be initiated by internal cognitive or physical processes.

The physiological expression of an emotion can provide a feedback which acts as another input to the system, generating another emotional response

Properties of affective signals

• Background mood

All inputs contribute to a background mood, whether or not they are below the activation level for emotions. The most recent inputs have the greatest influence on the present mood

Where affective signals come from?• Speech: prosody, syntactic analysis (which

word thesaurus and how concepts are expressed)

• Facial expressions (FACS: Ekman 1977)

• Body movement and gesture (Accelerometers, videocameras, other movement sensors)

• Physiological signals: EMG, EEG, EGC, …

• Social parameters (from analysis of social media)

From inputs to Emotion models

• Affective inputs need to be processed in order to determine higher level parameters, which define the coordinate of emotion spaces.

• Dimensional emotion spaces in the literature:– Russel valence-arousal (2D)– Wundt, Scherer valence-arousal-power (3D)

Valence-Arousal theories

• Use emotional signals to model valence and arousal components

• For both analysis and synthesis purposes

Russel’s valence-arousal model

valence

arousal high activity

low activity

positivenegative

happyfear

angry

sad

neutral

serene

In Russel’s model, Basic emotions are located in a circle centered on the neutral

enthusiasm

Recognizing and expressing affect (from analysis to synthesis)

Multimodal emotion dialogue

Multimodal emotion recognition

• Which cues do reveal emotions, preferences, and attentional states of a person?

• How integrated audio, visual, and possibly physiological inputs can enable more accurate or efficient emotion recognition with respect to using a single modality?

• How to individuate and collect multimodal data responsible for emotional expressions, and how should it be used in an emotional software?

• Are we measuring spontaneous or posed data?

Multimodal emotion recognition• Labelled or continuous emotion? The labeling of emotions

into different states led researchers toward pattern recognition approaches to recognizea nd alcssify emotions, using different modalities as inputs to the emotion recognition models

• Face: optical flow estimation to recognize facial expressions

• Voice: speech rate, pitch average, intensity, prosody cues, etc.

• Physiological patterns: heart rate, breath rate, skin conductance, ...

• Posture and postural attitude• Expressive gesture: fluidity, quantity of motion, repetitivity,

contraction/expansion, symmetry, …

Multimodal approach to emotion recognition

• Affect recognition is more accurate when it combines multiple modalities, information about the user’s context, situation, goal, and preferences

• Probabilistic models:

Collecting multimodal data for emotion recognition

Five factors influence the affective data collection (Picard et

al.)

• Spontaneous vs posed• Lab setting vs real world• Expression vs feeling• Open recording vs hidden recording• Emotion-purpose vs other purpose

From analysis to synthesis

• Manual annotation of affective videos

• Automatic extraction of features in affective videos

Emotion Synthesis

The Ortony Clore Collins (OCC) Cognitive Model

• Aim: cognitive emotions synthesis

• Model of cognitive appraisal for emotions

• Idea: representing emotions not by using sets of basic emotions or dimensioned space, but by grouping emotions according to cognitive eliciting conditions

The Ortony Clore Collins (OCC) Cognitive Model

• Emotions arise from valenced (positive or negative) reactions to situations consisting of events, agents and objects

• Specifications for 22 emotion types

• Disadvantages: low-level details of implementation are not present in the model

(e.g., how emotions interact, mix and change their intensity with time, etc.)

OCC-rule example

D(p, e, t) = desireability– e = event– p = person– t = time

Ig (p, e, t) = global intensity variables, expectedness, proximity, realityPj(p, e, t) = potential for generating a state of joy j Tj = threshold value for joyIj = intensity of joy

IF D(p, e, t) > 0THEN set Pj(p, e, t) =

fj(D(p, e, t), Ig (p, e, t))where fj() is a function specific

to joy j

IF Pj(p, e, t) > Tj(p,t)THEN set Ij(p, e, t) = Pj(p, e, t)

– Tj(p, t)

ELSE set Ij(p, e, t) = 0

Synthesis of Joy = returns positive value if event is expected to have beneficial consequences and viceversa

A computational model for the synthesis of artificial emotions:

an application to interactive multimodal environments (MEs)

(Camurri, Ferrentino, Dapelo; 1997)

Interactive MEs

• We call interactive ME an active space populated by agents observing and interacting with users, allowing them to communicate by means of full-body movement, singing, playing

• Users get feedback from an ME in real time in term of sound, music, visual media and actuators

• ME agents should be capable of changing their character

• Development of a model of artificial emotions to implement the emotional component of such a ME agents architecture

Example of a possible scenario

• Two different agents involved during artistic performances

• An agent capable to extract from humans some gesture and movement features and to control the generation of sound and music

• Another agent based on a robotic platform on wheels that manages the movement of a robot

• Agents communicate to each other

• Social interaction between the robotic agent and the performers

Machine gesture

• ME communicates to humans by means of the movement and gesture of its physical (or robotic) agents

• Robot as multimodal system: integration of movements, audio, music and animated artificial face

• Robot emotional behaviour changes according to the emotional characterization of the stimuli

• Robot emotional behaviour in terms of movement behaviour and sound and music outputs

The model of artificial emotions (1)

• It consists of a two-dimensional space which can be navigated

• A point in such space represents the character of the agent

• The two axes represent the degree of affection of the agent towards itself and towards others, respectively

• We call these two axis “EGO” and “NOS”

• E.g., a point placed in the positive x (EGO) axis represents an agent whose character is in a good disposition towards itself

• The emotion space is partitioned into thirteen regions

The model of artificial emotions (2)

• Each region is labeled by the kind of character the agent simulates

• A point in a region represents the character of the agent and an area around such point represents its mood or disposition

• Each region is defined by a maximum and minimum threshold for the quantities of the EGO and NOS character value

• For kinds of stimuli: positive-EGO, positive-NOS, negative-EGO, negative-NOS

• For each region, the behaviour is defined by the four kinds of stimuli: character moves in response to these four stimuli

• For each region there are four styles of movements to define

Applications: a music example

• Generation of a musical output reflecting the emotional state of an agent-robot navigating on stage

• Aim: obtaining a performance presenting the flow of emotions in the agent by means of the integration of music and movement of the agent-robot