linking vision and language: from infant cognition to eyetracking in the visual world, franklin...

7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world

of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1

Linking vision and language:From infant cognition to eye-tracking in the visual worldFranklin Chang and Andrew Jessop —- University of Liverpool



Vision and Language

How are vision and language linked?·

Meaning mediated account-

2/24



Evidence for a link between vision and language

Caused motion (transitive, 他動詞) -> Spontaneous motion (intransitives, 自動詞)

Children produce FROM passives: I was caught from you before (Clark & Carpenter, 1989)

·

CM I put the doll here -> SP the doll goes here (Bloom, 1993)

Two perspectives on the same visual scene: Child or Doll

-

-

·

Normal BY passives　受け身: I was caught by you

Children never hear caught from in input

Spatial similarity between Agents and Sources

-

-

-

The girl is running from the man (man = SOURCE 起点)

the girl is chased by the man (man = AGENT 動作主)

-

-

3/24


Page 5 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1

Linking Visual-spatial cognition and Language

Standard Approach: Vision and language are separate modules/systems

Alternative Approach: Language is built on pre-existing spatial mechanisms

·

Concepts mediate between vision and language-

·

Use infant/adult visual cognition to motivate a rich set of spatial mechanisms andrepresentations

Show how these can be linked to thematic roles in language

Present a connectionist model that uses these representations to explain eye-tracking inthe visual world

-

-

-

5/24



Thematic roles (主題役割)

Agent 動作主 = doer of action (child in I put the doll here)

Patient 被動者 = affected by action (doll)

Theme 主題 = moved by action (doll in the doll goes here)

How do children figure out if the doll is patient or theme?

Existing theories use abstract features (e.g., volition 意志, sentience 感覚性; Dowty, 1991)

·

·

·

·

Reality: Child moves the doll (child is agent)

Imagination: Doll is moving by itself (doll is theme)

-

-

·

These features are not defined clearly

Vision-based relational features are needed

-

-

6/24



A Vision-based Theory of Thematic Roles

Thematic roles in language make use of mechanisms that developed for object tracking

Vision has evolved powerful mechanisms for tracking objects

·

These two views are consistent with two tigers

Object tracking might show that there is only one tiger

-

-

·

Spatial pointers support object tracking (Pylyshyn & Storm, 1988)- 7/24



Spatial Indexes/Object files (Pylyshyn & Storm, 1988)

Participants see initial scene with identical crosses

Some crosses are highlighted as targets

Crosses move which participants keeps attention in center of screen

Cross is highlighted at test and participant responds whether it is one of the targets

·

·

·

·

8/24



Spatial Pointers and Visual heuristics

Wolf is the agent of chasing (sheep are patients)

Approach visual heuristic

Visual heuristics can be used to identify thematic roles

·

Wolf moves towards sheep (approach)

Wolf and sheep are identical circles (object tracking)

-

-

·

Angle of motion (Gao et al., 2009)

Goal-object motion (Woodward, 1998)

-

-

·

Innate (infants for approach, Luo, 2011)-

9/24



Separation, Distance, and Primacy Heuristic

Separation heuristic

Distance heuristic

Primacy heuristic

·

Reverse of Approach Heuristic-

·

Represents the distance between pairs of objects in the scene

Lew, Bremner, and Lefkovitch (2000) found that 8.5 month old infants encode a hiddentoy’s position between two landmarks after displacement to a new location

-

-

·

Represents the order of motion of objects in scenes

Bullock & Gelman (1979) showed two events where balls went down a ramp, but oneball went down before a jack-in-the-box appeared and another ball went downafterwards. Children labeled the first ball as the cause.

-

-

10/24



A Connectionist Model of Language Acquisition

Simple recurrent network (Elman 1990) ·

11/24



Dual-path model (Chang 2002)

Simple recurrent network learns sequencing constraints

Message: Fast changing weights between roles and concepts

·

·

12/24



Spatial Dual-path model

Roles are replaced with spatial pointers/indexes (e.g., P1)

Visual heuristics are input to SRN and help in identifying thematic roles

·

·

13/24



Explaining Eye-tracking in Production

Griffin & Bock (2000) eye-tracking during sentence production·

Look at subject before they produce head noun-

14/24



Eyetracking during noun production

Visual heuristics activate output pointer P2, which shifts attention to agent before "mouse"produced

·

15/24



Output Message Supports Predictive Eye movements


·

16/24



Output Message Supports Predictive Eye movements


·

17/24



Explaining Eye-tracking in Comprehension

Altmann (2004) Blank Screen Paradigm·

The boy will eat the cake

Screen is blank when eye-tracking is performed

-

-

18/24



Internal object tracking is needed to explain blank screen studies

Input and output messages are set when picture is visible ·

19/24



Input Message Supports Reactive Eye movements

Attention is shifted to corresponding input pointer, even when screen is blank ·

20/24





21/24





22/24



Errors and structural changes

Caused motion (transitive, 他動詞) -> Spontaneous motion (intransitives, 自動詞)

Children produce FROM passives: I was caught from you before (Clark & Carpenter, 1989)

·

CM I put the doll here -> SP the doll goes here (Bloom, 1993)

Changing thematic roles: AGENT=CHILD PATIENT=DOLL -> THEME=DOLL

Spatial similarity in visual heuristics creates these changes in the model

-

-

-

·

Normal BY passives: I was caught by you

Changing AGENT=CHILD -> SOURCE=CHILD

Spatial similarity between agents and sources create these errors in the model

-

-

-

23/24



Spatial Dual-path Model

Object-based pointers and visual heuristics support meaning

Language uses these spatial mechanisms and representations

Powerful generalization abilities have been used to argue for innate language knowledge

·

Infant physical/social cognition (e.g., old-goals have approach information)

Thematic relations (e.g., caused motion = approach + separation)

-

-

·

Eyes move when we understand language (e.g., eye tracking in the visual world)

Spatial biases and errors in development (e.g. FROM passives)

-

-

·

Critics have argued that domain-general statistical learning can explain languageabilities

Language evolved on top of preexisting visual/spatial mechanisms

-

-

24/24

linking vision and language: from infant cognition to eyetracking in the visual world, franklin...

Science