Perspective1

Embodied Coding through Perspective

Brian MacWhinney, CMU

Perspective2

Embodiment

• The problem of the mental code. Something from nothing.

• Mentalese vs. bodyese The perception-motor cycle Seeing things through our actions on them The brain connects to the body Even true on the level of the saccade The mirror system connects the body to others

Perspective3

What does the code do?

• It shapes memory and retrieval.

• It permits resonance.

• It structures mental models Long-term Current conversation

• It gives meaning through the homunculus.

• So, meaning is based in the end on the body.

Perspective4

Bodyese and Perspective

• When we act, we have A stance (balanced, sitting, stretched) A spatial position (in front, behind) A temporal flow (repeat, single, end) Goals and plans (push, open, turn, fall) A social position (giver, brother, trust)

• All of these actions situate us in a perspective

Perspective5

Processing

• Incrementalism In comprehension, we build competing perspectives

and select Picture-sentence matching, sometimes with mental

pictures In production, we enact a walk-through of our past

experiences

• Tip of the iceberg theory Enactive-depictive

• Speech-gesture linkage theory

Perspective

• # switches

• SS: The dog that chased the cat kicked the horse. 0

• OS: The dog chased the cat that kicked the horse. 1-

• OO: The dog chased the cat the horse kicked. 1+

• SO: The dog the cat chased kicked the horse. 2

• SS > OS = OO > SO• The dog the cat the boy liked chased snarled. 4+

• (dog -> cat -> boy -> cat -> dog)

MacWhinney & Pleh 1987

Perspective7

Counting Shifts

• Advantage of first mention - Gernsbacher

• Reflexive shift Mary combed herself.

The desert trains the young to be tough.

• Complex NP shift -- The communist farmers hated died.

• Shift to object, implicit causer Bill chased the cat.

Coming towards him, John saw five bicycles.

Bill criticized Mary, because she/he

Bill apologized to Mary, because she/he

Perspective8

Principles

• Pronouns must be referential

• First mention maintains unless …

• Missing mentions involve costs

• Objects prepare shifts

• Verbs code perspectives

Perspective9

Perspective across Cognition

directexperience

space/timedeixis

socialplans

perspective perspective perspective

language as a functional neural circuit

unified embodied image

perspective

Perspective

How language links to levels

• Words - abandon, beloved

• Linkers - deixis, anaphora

• Phrases - (my (little doll))

• Clauses - grammar

• Discourse - characters on stage

Perspective

Words

• Banana － affordances ， actions

• Navajo table and chair

• Miller and Johnson-Laird’s “table”

• Verbs as links to motor actions -- give, walk, talk, push, run

• Huge schemas -- promise, gravity, a certiorari

Perspective12

Coding and Resonance

• Local organization

• Somatotopic

• Coding through the whole system

• Even crazy resonances facilitate learning

• Is there hard evidence?

Perspective

Clauses: An example

• “The cat licks itself”

Perspective

Depictive Imagery

Perspective

Enactive Imagery

• take stance of cat

• find area to lick

• enact licking

• shift attention to area being licked

• perceive touch through licking

Perspective

enactive

depictive

Two pathways

Perspective

Compatible Images

• He hammered the nail into the wall.

• He hammered the nail into the floor. Nail pointing up. Nail pointing across. Tasks: verify, name

Zwaan, Schwartz, Tversky, Klatzky, Glenberg ….

Perspective

E grasps M grasps E with pliers M grasps

Mirror Neurons - Rizzolatti

Perspective

Monkey grasps in dark

Perspective

Ungrounding

• Identification

• Imagery

• Dreaming

Perspective

Identification

• Imitation and identification require “mapping” of body image

• This requires an accessible body image that can be dynamically remapped

• Once mapped, we can follow another’s motions

• Johnson (1998), Meltzoff and Moore, Csibra et al. Eyes and movement trigger identification Identification is basic, not cue-driven

• Mapping to other Birds “have no hands” (Menn) Child looks for its tail. (Mervis) Child opens mouth for scissors (Fincham)

Perspective

Three spatial perspectives

• Ego-centered -- egocentric Here, go up, go left

• Object-centered -- allocentric On the table, left of the tower Come, go, arrive Ego takes object perspective

• Frame-centered -- geocentric Toward the Bay, north

Perspective

Three temporal perspectives

• Speech time (ego centered) Now, today

• Reference time (object centered) Thursday, graduation, when he arrived …

• Event time against reference Before he arrived, John had lost his wallet After he arrived, John lost his wallet

• We take the perspective of the reference time

Perspective

Aspect

• Aspects are contours of actions or events Terminative: I stop chopping. Inceptive: I began chopping. Iterative: I kept on chopping.

• The whole action is the background and the aspect involves approaching the action with a particular involvement

Perspective

Causal Roles

• Predicates are directly grounded

• Case is routinized perspective

• Grammar arises to encode perspective shifts

• Constructions serve to express perspective modulation

Perspective

Case is routinized perspective

• Arg1 – The initial perspective or starting point• Arg2 – Recipient of shifted perspective• Arg3 – External perspective

• Three roles are enough for an item-based grammar (Construction Grammar, Word Grammar)

Perspective

Item-based Sockets

• Phrasal constructions provide sockets for extension and pushy polysemy (MacWhinney 1989) Give me another sand. A wiser Reagan returned from Rejkjavik.

• Verbs provide sockets for social expectations criticize vs. amazed

Perspective

Constructions

• Passive Adverbialization

• Double Object Attachment

• Relativization Dislocation

• Appositive Clefting

• Fictive Agent Topicalization

• Conflation Possession

• Comparison Ellipsis

• Complementation Coordination ….

Perspective

Ambiguity

• Tim saw the Grand Canyon flying to New York.

• Visiting relatives can be a nuisance.

• The women discussed the dogs on the beach.

• I ordered her pancakes.

• Although John always jogs, a mile seems long.

• The horse raced past the barn fell.

Perspective30

Scope

• Many arrows didn’t hit the target.

• The target wasn’t hit by many arrows.

• Three students read two books.

• Two books were read by three students.

• Every man who owns a Porsche loves it.

Perspective

C-Command

Perspective

C-command as Perspective

• Starting points must be referentially committed (MacWhinney, 1977)

• Pronouns immediately seek their referents McDonald and MacWhinney (1996)

Mary criticized Billi, because hei ...

Maryi amazed Bill, because shei …

• Perspective flows to referents online

Perspective

Centrality forces commitment

• S > DO > IO > Poss• * Hei says Billi came.• * John told himi that Billi was crazy.• ? John said to himi that Billi was crazy.• John told hisi mother that Billi was crazy.

• C-Command blocking depends on how you draw the tree

Perspective34

Backgrounding

• Releasing commitment: After hei came, Johni drank three beers.

Near himi, Johni saw a snake.

• Yes, we can backtrack, but it is marked: *Near Johni, hei saw a snake.

?Near John’si computer desk, hei placed a printer.

?After Johni came, hei drank three beers.

?Hei came and then Johni drank three beers.

Perspective

• Igori hid the book behind himi/himselfi.

• Johni kicked *himi/himselfi.

• Johni ignored the oil on himi/*himselfi.

• Johni heard a story about himi/himselfi.

• Maxi told a story about *himi/himselfi.

Reflexives maintain Perspective

Perspective

Perspective interruption

• *Jessiei stole a photo of heri out of the archives.

• Jessiei stole me a photo of heri out of the archives.

Perspective

Shifts

• Jessie stole a photo of *her/herself out of the archives.

• Jessie stole a silly photo of her/herself out of the archives.

• Anna hid a snapshot of *her/herself under the linoleum.

• Anna hid the snapshot of *her/herself under the linoleum.

• Lucie talked about the operation on *her/herself.

• Lucie talked about the operation on her/herself that Dr. Edward performed.

Perspective

Clitic Assimilation

Why do you want to go? Why do you wanna go? Who(m) do you want to go? * Who(m) do you wanna go? I get ta go. (Privilege) I got ta go. (Privilege) I gotta go. (Obligation)

Perspective39

An example

A cyclone hammered the Bangladesh coast Monday with the force of "hundreds of demons" leveling entire villages of mud and thatch huts, flooding crops, and killing at least six people.

Three men and two children were crushed under collapsed buildings or hit by flying pieces of tin roofs in the southern port of Chittagong. One man died in Teknaf, about 110 miles down the coast, when he was blown off his roof, while trying to secure it.

The storm roared in from the Bay of Bengal with wind gusts of 125 mph, forcing a half-million people to flee their huts and huddle in concrete shelters. Many power and telephone lines were down, so a full account of casualties and damage was not available.

Perspective

Mental Spaces

• A contemporary philosopher, leading a seminar (From Fauconnier and Turner)

I claim that reason is a self-developing capacity. Kant disagrees with me on this point. He says it’s innate, but I answer that that’s begging the question, to which he counters, in Critique of Pure Reason, that only innate ideas have power. But I say to that, what about neuronal group selection? And he gives no answer.

• One space for the teacher, one for Kant, and a comparison space where the debate occurs.

Perspective

Put the bottom block below the apple

Perspective

Linguistic Devices

• Your beloved cat -- Your damned cat.

• John criticized Bill, because he ..

• John apologized to Bill, because he …

• Japanese speaker’s territory

• Evidentials, deontic-evidential

Perspective

Perspectival Domains

• The bicyclist appears to have escaped injury

• Did the bicyclist appear to have escaped injury?

• The reporter said that the bicyclist appeared to have escaped injury.

• The reporter asked if the bicylcist appeared to have escaped injury.

• John said that Marco hates Sarah’s beloved cousin.• The adults in the picture are facing away from us, with the

children hidden behind them. (Cantrall)

Perspective

Logophoricity

• John said, “The bicycle is here.”

• John said that the bicycle is here.

• Unfortunately, John said that the bicycle is behind the damn wall.

• Did John say that the bicycle is evidently behind the damn wall?

Perspective

Frontal support

• Inferior frontal cortex supports code generation through perceptual-motor images for memory lists

• Linkage of frontal areas to temporoparietal areas supports reenactment (simulation)

• Interconnections of frontal motor and attentional areas

• Linkage of prefrontal to subcortical and cingulate gyrus

Perspective46

Evolution

• Apes can shift perspective (Call & Tomasello)

• Gestural communication required further perspective taking

• Creativity revolution: language or symbol?

• Charismatic leaders and priests could use language to motivate the people

Perspective47

Classic Issues

• Formalism vs. Functionalism in Grammar

• Language and Consciousness

• Grounding in Language Development

• Symbol Grounding

• Robotic Grounding

• Language as a Special Gift

Perspective

Assumptions that don’t change

• Basic activation/inhibition framework Power Law Neural Network modelling

• MacWhinney-Bates Competition Model Cue validity, cue strength Lexical grammar Incrementalism

• Barsalou makes mental models safe for AI• Dual code theory still works

Perspective

Conclusions

• Cognition is grounded on perceptual and motoric simulations, triggered by language and gesture

• The forms of language are shaped by the process of perspective-taking

• The brain provides mechanisms to implement simulation and perspective