does personality matter?: expressive generation for …...tts text-to-speech synthesis spoken...

NATURAL LANGUAGE AND DIALOGUE SYSTEMS

LAB

UC SANTA CRUZ

Natural Language and Dialogue Systems Lab

Does Personality Matter?:

Expressive Generation for

Dialog Systems IWSDS 2012

Prof. Marilyn Walker

Baskin School of Engineering

University of California, Santa Cruz


LAB

UC SANTA CRUZ


Thank you for inviting me to this lovely

place!


LAB

UC SANTA CRUZ

Expressive Language in Conversation

Expresses Speaker’s Personality & Identity

culture, style, origin, class

Dynamically Adapts to Conversational Partner

Convergent : Matching, e.g. two friends (extraverts)

talking

Divergent: Tailoring, e.g. parent to baby

Controlled by generation parameters

Content: Who is interested in what, who knows what

Linguistic: Lexical and Syntactic Choice

Pragmatic: Personality & Social Relationship

Acoustic: Speaking Rate, Amplitude, Prosody


LAB

UC SANTA CRUZ 4

Personality in Language: People do it

From Mehl et al., 2006. Mairesse etal 2007.

Introvert Extravert

- I don't know man, it is

fine I was just saying I

don't know.

- I was just giving you a

hard time, so.

- I don't know.

- I will go check my e-mail.

- I said I will try to check

my e-mail, ok.

- Oh, this has been

happening to me a lot lately.

Like my phone will ring. It

won't say who it is. It just

says call. And I answer and

nobody will say anything.

So I don't know who it is.

- Okay. I don't really want

any but a little salad.


LAB

UC SANTA CRUZ

Film Characters: Crafted Personalities


LAB

UC SANTA CRUZ

Does personality matter for dialogue

systems?

Yes. Achieving communicative goals in dialogue often relies

on engaging user affect:

persuasion, motivation, increase in self-efficacy beliefs,

learning

People react socially to computational agents, thus social

norms such as liking people like yourself often hold (Nass &

Lee, 2001)

People make attributions beyond social level: task

competence

Personality matching in a robotic exercise coach increased the

time that stroke victims spent on their medically recommended

exercises (Tapus & Mataric 2008)

Tutoring oriented to the student’s ‘face needs’ improved

learning in training and tutoring (Porayska-Pomsta & Mellish

2004; Wang et al., 2004)


LAB

UC SANTA CRUZ

Dialogue Systems Architecture

DM

SLU

TTS

Text-to-

Speech

Synthesis

Spoken Language

Understanding

Dialogue

Management

ASR

SLG Spoken

Language

Generation

Data,

Rules

Words

Meaning

Speech, Nonverbals Speech, Nonverbals

Goal

Words

Speech

Recognition

Personality?


LAB

UC SANTA CRUZ

Outline of my talk

Expressive natural language

generation

PERSONAGE: personality models

for language generation (Mairesse &

Walker, UMUAI 2010; CL 2011, ACL2007; ACL

2008)

Generating nonverbal expression

of personality (Bee, Pollack, Andre’ &

Walker IVA 2010; Neff, Wang, Abbott & Walker

IVA 2010, Neff, Toothman, Grant, Walker IVA

2011)

Generation by learning models of film

characters from corpora (Lin & Walker

AIIDE 2011, Walker, Lin, Wardrip-Fruin, Buell,

Grant ICIDS 2011)


LAB

UC SANTA CRUZ 9

Limitations of previous work

Writing character dialogue is an art: it is not described at a

level that supports computational models

User Experience design is largely based on intuition

Work on narrative (arts and humanities) does not suggest specific

linguistic or behavioral reflexes or parameters

There has been little systematic exploration of personality

or social parameters suggested by psycholinguistic

findings

Unclear which psycholinguistic findings have impact in

particular application domains

Almost no evaluation showing variation system produces

is perceived by the user as the system intended


LAB

UC SANTA CRUZ

Need a general purpose generation

technology that can be easily ported from

one domain/task to another. Expressive

but also for mixed initiative dialogue (any

branching dialogue).

10


LAB

UC SANTA CRUZ

Procedural Language Generation: A Key

Technology

Wide range of generation parameters

Different methods for creating models

that control the parameters

Dynamic Real-Time Adaptation

Trainable: Machine Learning

Techniques

Individual Personalization

11


LAB

UC SANTA CRUZ

Dialogue Systems Architecture

DM

SLU

TTS

Text-to-

Speech

Synthesis

Spoken Language

Understanding

Dialogue

Management

ASR

SLG Spoken

Language

Generation

Data,

Rules

Words

Meaning

Speech, Nonverbals Speech, Nonverbals

Goal

Words

Speech

Recognition

Personality?


LAB

UC SANTA CRUZ 13

Language Generation Module

What to say

Content

Planner

How to Say It

Sentence

Planner

Surface

Realizer

Prosody

Assigner

What is Heard

Speech

Synthesizer


LAB

UC SANTA CRUZ 14

Variation controlled by the Language

Generator

What to say

Content

Planner

How to Say It

Sentence

Planner

Surface

Realizer

Prosody

Assigner

What is Heard

Speech

Synthesizer

Parametrized

Variation

• vary content and form easily depending on any

factor (context, personality, social relationship)


LAB

UC SANTA CRUZ

Expressivity?: Which parameters and

models? Theories and Corpus Studies of Human Dialogue Behavior

Psychology: Big Five Theory of Personality

Sociolinguistics: Politeness Theory

Learn from Film Character Dialogue


LAB

UC SANTA CRUZ

PERSONAGE Generator: BIG FIVE

Theory

Conscientiousness: Dutiful vs. impulsive

Emotional stability: Calm vs. anxious

Openness to experience: Imaginative vs.

conventional

Agreeableness: Kind vs. unfriendly

Extraversion: Sociable, assertive vs. quiet


LAB

UC SANTA CRUZ

Linguistic Reflexes of Personality: 50 yrs of

studies

Extraversion (Furnham, 1990)

Talk more, faster, louder and more repetitively

Fewer pauses and hesitations

Lower type/token ratio

Less formal, more references to context (Heylighen & Dewaele, 2002)

More positive emotion words (Pennebaker & King, 1999)

E.g. happy, pretty, good

Neuroticism (Pennebaker & King, 1999)

1st person singular pronouns

Negative emotion words

Conscientiousness (Pennebaker & King, 1999)

Fewer negations and negative emotion words

Low but significant correlations


LAB

UC SANTA CRUZ

18


LAB

UC SANTA CRUZ

PERSONAGE Architecture: 67 Parameters

Realization

INPUT: Dialog Act,

Content Pool

OUTPUT

UTTERANCE

VERBOSITY

RESTATEMENTS

CONTENT POLARITY

…

SYNTACTIC

COMPLEXITY

SELF-REFERENCE

…

CONTRAST: e.g.

however, but

JUSTIFY: e.g.

so, since

PERIOD

…

EXCLAMATION

HEDGES: e.g. kind of,

rather, basically, you know

FILLED PAUSES: e.g. err…

SWEAR WORDS: e.g. damn

IN GROUP MARKERS: e.g. pal

STUTTERING: e.g. Ri-Ri-River

TAG QUESTIONS

…

FREQUENCY OF USE

WORD LENGTH

VERB STRENGTH

Content

Planner

Pragmatic

Marker

Insertion

Lexical

Choice Aggregation

Syntactic

Template

Selection


LAB

UC SANTA CRUZ 20

Example of Pragmatic Transformation

Negation insertion “X has awful food” “X doesn’t have good food”

Wok Mania class: proper noun

number: sg

have class: verb

awful class:adjective

food class: noun

number: sg

article: none

Obj Subj

ATTR

WordNet

Database Look for antonym

“good”

- Negate verb

- Replace adjective

by antonym Wok Mania class: proper noun

number: sg

have class: verb

negated: true

good class:adjective

food class: noun

number: sg

article: none

Obj Subj

ATTR


LAB

UC SANTA CRUZ 21

Recommendation: A Persuasive Task

Alt Realization Extra

5 Err... it seems to me that Le Marais isn’t as bad as the others. 1.83

4 Right, I mean, Le Marais is the only restaurant that is any good. 2.83

8 Ok, I mean, Le Marais is a quite french, kosher and steak house place, you know and

the atmosphere isn’t nasty, it has nice atmosphere. It has friendly service. It seems to

me that the service is nice. It isn’t as bad as the others, is it?

5.17

9 Well, it seems to me that I am sure you would like Le Marais. It has good food, the food

is sort of rather tasty, the ambience is nice, the atmosphere isn’t sort of nasty, it

features rather friendly servers and its price is around 44 dollars.

5.83

3 I am sure you would like Le Marais, you know. The atmosphere is acceptable, the

servers are nice and it’s a french, kosher and steak house place. Actually, the food is

good, even if its price is 44 dollars.

6.00

10 It seems to me that Le Marais isn’t as bad as the others. It’s a french, kosher and steak

house place. It has friendly servers, you know but it’s somewhat expensive, you know!

6.17

2 Basically, actually, I am sure you would like Le Marais. It features friendly service and

acceptable atmosphere and it’s a french, kosher and steak house place. Even if its

price is 44 dollars, it just has really good food, nice food.

6.17


LAB

UC SANTA CRUZ

Training Models: Human Perceptions. 4

methods

Rule based: take the findings from the psych literature and encode them in a model

Overgenerate and Rank: generate many possibilities, collect human perceptual ratings, learn to select ones that match the human perception you are trying to achieve

Parameter Estimation: Assume parameters are independent and learn to set them as a function of personality desired

Film Corpora: Learn models from film dialogue (Mairesse&Walker07, Mairesse & Walker UMUAI 2010,

Mairesse&Walker08, Lin&Walker11, Walkeretal11,Neffetal10, Neffetal11)


LAB

UC SANTA CRUZ

Corpus Annotation: 3 Human Judges (Ten-Item Personality Inventory, Gosling et al. 03)

Extraversion = 3.5

Neuroticism = 2.0

Agreeableness = 6.5

Conscient. = 4.0

Openness = 1.5


LAB

UC SANTA CRUZ 24

7.0 6.5 6.0 5.5 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1.5 1.0

Extraversion rating

40

30

20

10

0

Utt

era

nc

e

co

un

t

Extravert

Introvert

1st method: Rule-Based Extraversion

Generation Use correlations in literature to set parameters

Significant perceptual differences p < .01

As binary classification, 90% accuracy


LAB

UC SANTA CRUZ

2nd Method: Overgenerate and Rank

Generator

“WokMania is a great

place, because the

food is good, isn’t it?”

Statistical

Regression/Ranking Model Estimates scores from features,

e.g. verbosity, hedges

“Yeah, even if WokMania

is expensive, the food is

nice, I am sure you would

like WokMania!”

“Err… this restaurant’s

not as bad the others.”

…

Input

personality

score

e.g. 2.5 out of 7

Closest estimate, utterance 2:

“Err… this restaurant’s not as bad the others.”

Feature vector 1 Feature vector 3 Feature vector 2 Feature vector n


LAB

UC SANTA CRUZ 26

Statistical Ranking Models of Personality

Data: 160 random utterances rated by 3 judges

Features based on generation decisions

Random utterances less natural

than extreme utterances

Results for predicting extraversion Linear regression, model trees, SVM

Better than inter-rater correlations for random utterances (r = .40)

Correlation Abs. error (1 to 7)

Extraversion .60 .65

Emotional stability .51 .82

Agreeableness .58 .67

Conscientiousness .38 .84

Openness to experience .39 .92


LAB

UC SANTA CRUZ 27

Output

Extraversion

Score


LAB

UC SANTA CRUZ 28

Agreeableness score =

1.9614 +

0.919 * Content Plan Tree:Content Polarity +

0.692 * RST Tree Modification:Polarisation +

0.593 * Hedge Insertion:down_subord_it_seems_to_me_that +

0.432 * Hedge Insertion:in_group_marker=1 +

0.414 * Hedge Insertion:down_err=0 +

0.326 * Tag Question Insertion=0 +

0.314 * Hedge Insertion:down_somewhat=0 +

0.305 * Hedge Insertion:swearing=0 +

…

-0.0878 * Demonstrative Referring Expressions +

-0.0915 * Content Plan Tree:Repetitions Polarity +

-0.1942 * Content Plan Tree:Concessions +

-0.2004 * Positive Content First +

-0.2099 * Aggregation Operation Probabilities:contrast:PERIOD+

-0.3284 * Hedge Insertion:down_subord_it_seems_that +

-0.6367 * Hedge Insertion:competence_mitigation_come_on

Linear regression model for Agreeableness


LAB

UC SANTA CRUZ

Third Method: Parameter Estimation

Models Data: 160 randomly generated utterances + generation

decisions+ ratings

Training Multiple Continuous Parameters Models

Independence assumption between parameters

Best regression models selected through cross-validation

Example: Stuttering


LAB

UC SANTA CRUZ

Utterances express *multiple* personality

traits


LAB

UC SANTA CRUZ

Naïve Subjects Evaluation

Experiment

24 subjects rated 50 utterances

Each utterance hits a combination of

Big Five targets

Correlation between target scores and average ratings

Extraversion = 3.5

Neuroticism = 1.7

Agreeableness = 6.5

Conscientiousn. = 4.0

Openness = 4.5


LAB

UC SANTA CRUZ

So where are we?

A flexible, real-time, generator

Personality parameters

Methods for automatically training

Personalize both content and form

Standard meaning representations: DB Relations,

Content Plan, AI planner


LAB

UC SANTA CRUZ


There are also findings in psychology about

how VOICE and FACE and GESTURE

express personality. Can we use them?


LAB

UC SANTA CRUZ

Psychological Correlates Between Extraversion and

Movement

Introversion Extraversion

Body attitude backward leaning, turning away forward leaning

Gesture amplitude narrow wide, broad

Gesture rate low high

more movements of head, hands

and legs

Gesture speed,

response time

slow fast

Gesture direction more inward, self-contact more outward, table-plane and

horizontal spreading gesture

Gesture connection low smoothness, rhythm

disturbance

smooth, fluent

Body part shoulder erect, limbs spread,

elbows away from body, hands

away from body, legs apart

(References in Neff etal, 2010, 2011)


LAB

UC SANTA CRUZ

Alfred: Facial Dominance Behaviors &

Personality

Extravert, Hi Dominance Introvert, Low Dominance

Bossy or Wimpy: Expressing Social Dominance by Combining Gaze and Linguistic

Behaviors (Bee, Pollack, Andre’ & Walker, Intelligent Virtual Agents 2010)

Hypoth: Persuasiveness could be increased by

expressing dominance with personality (Marwell, 1967,

Mehrabian, 1995)


LAB

UC SANTA CRUZ


Evaluating the Effect of Gesture and

Language on Personality Perception in

Conversational Agents

Intelligent Virtual Agents, 2010

Michael Neff, Yingying Wang(UCD), Rob Abbott, Marilyn

Walker(UCSC)


LAB

UC SANTA CRUZ

Experimental Set-up

Rate: (on seven point scale)

1.Extraverted, enthusiastic: o o o o o o o

2.Reserved, quiet: o o o o o o o

3.Naturalness o o o o o o o


LAB

UC SANTA CRUZ

So where are we now?

Voice realization maintains personality

perceptions

Methodology of mining social psychology

literature for parameters extends to gaze, head

position, body and arm gestures


LAB

UC SANTA CRUZ

Character Creator: Author creativity

Learn models of character voice

(linguistic style) from film

screenplays

Use the learned models to control

the parameters of PERSONAGE

Apply the learned models to

character dialogue in the SpyFeet

story domain

A Different!! Domain

Test human perceptions of the

resulting generated utterances


LAB

UC SANTA CRUZ

Scene from Annie Hall: Lobby of Sports

Club

ALVY: Uh … you-you wanna lift?

ANNIE: Turning and aiming her thumb

over her shoulder

Oh, why-uh … y-y-you gotta car?

ALVY: No, um … I was gonna take a

cab.

ANNIE: Laughing Oh, no, I have a car.

ALVY: You have a car?

Annie smiles, hands folded in front of

her

So … Clears his throat. I don’t

understand why … if you have a car, so

then-then wh-why did you say “Do you

have a car?” … like you wanted a lift?

Annie Hall: Getting a lift


LAB

UC SANTA CRUZ

The Terminator:

getting a lift

Scene from The Terminator: Cigar biker

TERMINATOR: I need your clothes, your boots, and your motorcycle.

CIGAR BIKER: You forgot to say please.

Terminator hurls Cigar, all 230 pounds of him, clear over the bar, through

the serving window into the kitchen, where he lands on the big flat

GRILL. We hear a SOUND like SIZZLING BACON as Cigar screams,

flopping jerking. He rolls off in a smoking heap.


LAB

UC SANTA CRUZ

What can we learn from a corpus?

Reveal Subtext: The way a character says

something is one way to reveal subtext and

character emotion

Short vs. Long turns/sentences => friendliness,

formality

Word choice => level of education,

Disfluencies, Stuttering => anxiety, hesitation

Direct forms vs. indirect forms => extraversion,

aggression

Character Voice: Learning to model specific

characters or sets of characters should produce

individual character voices


LAB

UC SANTA CRUZ

Role playing game

Solve a mystery by

talking to ‘animal

spirits’

Hypotheses: Dynamic

elements will increase

engagement &

immersion

Natural Language and Dialogue Systems http://nlds.soe.ucsc.edu

SPYFEET: Dynamic NPC/Player dialogue


LAB

UC SANTA CRUZ

4. Generate

features

reflecting

linguistic

behaviors

Pulp Fiction

Script

Vincent’s

Dialogue

Jules’

Dialogue

Other’s

Dialogue

Jules’

Dialogue Vincent’s

Dialogue

Jules’

LIWC results Vincent’s

LIWC results

Jules’ Tag

Question

Ratio Vincent’s Tag

Question

Ratio

Jules’

other

features Vincent’s

other

features

Jules’

Overall

Polarity Vincent’s

Overall

Polarity

…

…

1. Collect

movie scripts

from IMSDb

2. Extract

utterances for

each character

3. Select

leading roles

(dialogue > 60

turns)

Method


LAB

UC SANTA CRUZ

Jules’

Learned

Model

Vincent’s

Learned

Model

Generated

features Vincent in

SpyFeet

utterances

PERSONAGE

generator

(ENLG

engine)

Jules’ in

SpyFeet

utterances

Others in

SpyFeet

utterances

…

5. Learn models

of character (z-

scores)

6. Generate new

utterances using learned

models to control

parameters of our dialogue

generator

Story domain:

SpyFeet

utterances

Method (cont)


LAB

UC SANTA CRUZ

Learning Character Models: Z-Scores

Z-scores: individual models trained by normalizing individual

character model against a representative population

Example: normalize Annie (Annie Hall) against all female

characters

z-score >1 or <-1 is more than one standard deviation

away from the average

Indicates parameters that should be high or low

Annie’s

z-score

Annie’s vector Averaged female population

Standard deviation female population


LAB

UC SANTA CRUZ

Example: Model Learned for Annie

PERSONAGE

parameter

Description Sample mapped features

(from character model)

Annie

Verbosity Control # of propositions in the

utterances

Number of sentences per turn,

words per sentence 0.78

Content

polarity

Control polarity of propositions

expressed

Polarity-overall, LIWC-Posemo,

LIWC-Negemo, LIWC-Negate 0.77

Polarization Control expressed polarity as

neutral or extreme

1 if polarity-overall is strong

negative or positive 0.72

Concessions Emphasize one attribute over

another

Category-concession 0.83

Positive

content first

Determine whether positive

propositions – including the

claim – are uttered first

Accept-ratio, Accept-first-

ratio 1.00

… etc.

Map character model to PERSONAGE parameters: weighted average of

features.

Parameters either binary, or scalar range 0…1.


LAB

UC SANTA CRUZ

Annie (Annie Hall)

original dialogue sample

• H’m?

That’s, uh … that’s pretty serious stuff there.

Yeah? Yeah?

M’hm? M’hm.

Yeah.

U-huh. • Hi. Hi, hi.

Well, bye.

Oh, yeah? So do you.

Oh, God, whatta- whatta

dumb thing to say, right? I mean, you say it, “You

play well,” and right

away … I have to say

well. Oh, oh … God,

Annie. Well … oh, well …

la-de-da, la-de-da, la-la

Generated dialogue

(SpyFeet story domain)

• Come on, I don’t

know, do you? People

say Cartmill is strange while I don’t rush to

um.. judgment.

• I don’t know. I think that you brought me

cabbage, so I will tell something to you,

alright?

• Yea, I’m not sure,

would you be? Wolf wears a hard shell but

he is really gentle.

• I see. I am not sure.

Obviously, I respect Wolf. However, he

isn’t my close friend, is

he?

Original and Generated Utterances

Annie’s Learned Z-Score

Model for our ENLG

engine

Verbosity=0.78

Content polarity =0.77

Polarization =0.72

Repetition polarity=0.79

Concessions =0.83

Concessions

Polarity=0.26

Positive content first=1.00

First Person in Claim=0.6

Claim Polarity=0.57

… etc.

Learning

Linguistic

Features

Generation


LAB

UC SANTA CRUZ

Perceptual Experiment Part 3: Can People

Tell? Example: read 3 scenes for Marion, then read 6 sets of

generated utterances to determine similarity of style to

original utterances


LAB

UC SANTA CRUZ

Hypoth 2: Character Models Perceived?

Average similarity scores (1 to 7) between character and character

models.

Perfect Performance: A matrix with highest values along

diagonal

*significant differences between character and character models of

each row

Character Character Models

Alvy Annie Indy Marion Mia Vincent

Alvy 5.2 4.2* 2.1* 2.6* 2.8* 2.3*

Annie 4.2 4.3 2.8* 3.4* 3.9 2.9*

Indy 1.4* 2.2* 4.5 2.8* 3.3* 3.8*

Marion 1.6* 2.8* 3.7 3.1 4.1* 4.2*

Mia 1.7* 2.4* 4.3 3.2 3.6 4.3

Vincent 2.1* 3.2* 4.5 3.5* 3.6* 4.6


LAB

UC SANTA CRUZ

Summary & Future Work

Personality for dialogue agents can help achieve

many conversational goals

Current & Future work:

Methods to make it easy to get content in new domains

into generator

Experiment to test whether helps people authoring

interactive stories

Evaluate use of different personalities for in-vehicle task

dialogue demo

Would like to test in a long-term companion scenario

either phone, car or eldercare robot


LAB

UC SANTA CRUZ

Come Visit !! University of California Santa

Cruz


LAB

UC SANTA CRUZ

Alfred Experiment: Gaze & Personality


LAB

UC SANTA CRUZ

Mary: Gesture & Personality

does personality matter?: expressive generation for …...tts text-to-speech synthesis spoken...

Documents