How Speakers' Eye Movements Reflect

Spoken Language Generation

Zenzi M. GriffinDepartment of Psychology

Intro to Cognitive ScienceUniversity of Texas at Austin

October 14, 2011

TIME

MESSAGE

“blah blah blah…”

?

What the eyes tell us about speaking:

when speakers prepare names for objects

which object they intend to name

possibly when they commit to an order of mention or syntactic structure

Griffin & Bock (2000) Psych Science

Movie Legend

green current fixation

blue earlier fixation

size fixation duration

Speakers gazed at referents in the second before naming them

Griffin & Bock (2000) Psych Science

The girl is kicking the boy

Long gazes followed order of mention

Not left-right or salient-less

salient or agent-patient or big-small or human-

nonhuman

Word selection

Difficulty in selecting a name varies with number & strength of competing candidates

"baby" "TV"or

"television"

Codability

Medium

High

FrequencyLow High

Isolated object naming

Latencies from a large norming study (Griffin & Huitema, 1999)

Huge, additive effects of codability & frequency.

SE

Object categorization task

Is it bigger than a piece of paper?

Codability effect ns Frequency effect ns Interaction ns

SE

YESNO

“The clock and the TV are above the

needle”

Griffin (2001) Cognition

Griffin (2001) Cognition

“The |clock and the TV are above the needle.”

Griffin (2001) Cognition

Gaze duration correlated with name difficulty

Griffin (2001) Cognition

Gaze duration correlated with name difficulty

Griffin (2001) Cognition

Effects of preparing 2nd name only during speech, not before

See also work by Antje Meyer & colleagues on effects of word frequency, phonological priming, image degradation, etc.

Codability: MEDIUM HIGH

Freq:

Gaze

tim

e o

n B

1st noun

Griffin (2001) Cognition

Until ready to describe

The crib & thelimo are above

the needle

time

Prepare then speak

Gaze reflects difficulty when preparing words in advance

Disfluent, Multiple namesFluent, Multiple names

Fluent, One name

Griffin (in prep)

Extemporaneous vs. Prepared speech

Codability: MEDIUM HIGH MEDIUM HIGH

Freq:

Gaze

tim

e o

n B

1st noun

Griffin (in prep)

Not much looking at locations while articulating prepared speech

“The A and the B are above the C”

Gazes Reflect Word Preparation

1. When describing scenes, speakers gaze at objects just before naming them.

2. Time spent gazing at an object reflects how difficult it is to prepare a name.

3. Gazes reflect when words are prepared, not articulated.

Why look?

Gazing at intended object could facilitate name retrieval

leaf

Gazing could prevent interference from other objects

uh leaf

No eye movements

Fixation point

"finger" for "thumb"(half speed then real time)

Timing of gaze: before error = before

correct

Griffin (2004) Psych Science

(real time)

Speakers gazed longer at agents before calling them by inaccurate, similar names

even when only fluent trials considered

Griffin & Oppenheimer (2006) JEPLMC

Timing of gaze offsets was similar

Griffin & Oppenheimer (2006) JEPLMC

(real time)

Speakers even gazed at referent when a better match to name was

visible

Griffin & Oppenheimer (2006) JEPLMC

(half speed then real time)

Speakers also gazed at referent when using a novel word "blick"

Griffin & Oppenheimer (2006) JEPLMC

Name-related eye movements aren't necessary & do not seem to facilitate name preparation

1. Speakers can describe simple scenes without moving their eyes.

2. Gazes before speech errors are similar to those before correct names.

3. Speakers gaze at referents before lying about them.

How early must speakers commit to order-of-mention or syntactic

structure?

A B

C

“The A and the B are above the C”

“The A and the B are above the C”

“The |woman is throwing a dog a bone.”

Referent for

Order of mention/ structure choice

"A woman throwsa bone to a dog."

a dog a bone."

theme-recipient

recipient-theme

1st presentationDirector/

Confederate

Is a man drinking coffee?

Matcher/Participant

No match.

2nd presentation: Biasing Question

Director/Confederate

Matcher/Participant

Is a tall woman throwing a

bone to a dog?

Yes, a tall woman is throwing a bone to

a dog.

Biasing question like having pre-planned order: no uncertainty about order of mention

"Yes, a woman is throwing a bone to a dog"

(real time)

2nd presentation: Neutral Question

Director/Confederate

Matcher/Participant

Is a tall woman swimming?

No, a tall woman is throwing a bone to

a dog.

Emergent argument ordering:

"Yes, a woman is throwing a bone to a dog"

Dependent measure: gaze shifts

"Yes, a woman is throwing a bone to a dog"

(real time, half speed)

More shifts between theme & recipient during articulation of 1st noun phrase

shifts

+ S

E

Match, a woman is throwing a bone to a dog

Griffin, Garton, & Mouzon (in prep)

More shifts/sec during 1st NP

shifts

/sec

+ S

E

Match, a woman isthrowing a bone

to a dog

Griffin, Garton, & Mouzon (in prep)

Conclusion

Speakers may decide order of mention & syntactic structure as needed and still be fluent.

However, additional experiments that manipulate bias in other ways suggest that time of decision varies with other factors.

What the eyes tell us about speaking:

when speakers prepare names for objects

which object they intend to name possibly when they decide order

of mention or syntactic structure

Other current projectsTougher tests of whether eye movements during

production ease cognitive load (in collaboration with Susan Goldin-Meadow & Susan Wagner-Cook)

Do eye movements to empty space during production reflect metaphorical use of space (in collaboration with Daniel

Cassasanto)?

Thanks toKathryn BockDaniel OppenheimerDaniel Spieler

Azucena RangelKristin GartonElisa LawlerEdna ChaviraEllen HamiltonSonia MouzonAmanda SilverioJustin Storbeck

Eye-speech movies, preprints, & posters

available at http://oak.psych.gatech.edu/~zgriffin/