the neural basis of thought and language
DESCRIPTION
The Neural Basis of Thought and Language. Final Review Session. Administrivia. Final in class next Tuesday, May 9 th Be there on time! Format: closed books, closed notes short answers, no blue books And then you’re done with the course!. Motor Control. Grammar. Metaphor. Bayes Nets. - PowerPoint PPT PresentationTRANSCRIPT
Administrivia
• Final in class next Tuesday, May 9th
• Be there on time!
• Format:
– closed books, closed notes
– short answers, no blue books
• And then you’re done with the course!
The Second Half
Cognition and Language
Computation
Structured Connectionism
Computational Neurobiology
Biology
Midterm Final
abst
ract
ion
Motor Control
Metaphor Grammar
Bailey Model
KARMA ECG
SHRUTI
Bayesian Model of HSP
Bayes Nets
Overview
• Bailey Model
– feature structures
– Bayesian model merging
– recruitment learning
• KARMA
– X-schema, frames
– aspect
– event-structure metaphor
– inference
• Grammar Learning
– parsing
– construction grammar
– learning algorithm
• SHRUTI
• FrameNet
• Bayesian Model of Human Sentence Processing
Full CircleNeural System &
Development
Motor Control & Visual System
Spatial Relation
Psycholinguistics Experiments
Metaphor
Grammar Verbs & Spatial Relation
Embodied Representation
StructuredConnectionism
Probabilistic algorithms
ConvergingConstraints
How can we capture the difference between
“Harry walked into the cafe.”
“Harry is walking into the cafe.”
“Harry walked into the wall.”
Analysis Process
Utterance
Simulation
Belief State
General Knowledg
e
Constructions
SemanticSpecificatio
n
“Harry walked into the café.”
The INTO construction
construction INTO subcase of Spatial-Relation
form selff .orth ← “into”
meaning: Trajector-Landmarkevokes Container as contevokes Source-Path-Goal as spgtrajector ↔ spg.trajectorlandmark ↔ contcont.interior ↔ spg.goalcont.exterior ↔ spg.source
The Spatial-Phrase construction
construction SPATIAL-PHRASE
constructionalconstituents
sr : Spatial-Relationlm : Ref-Expr
formsrf before lmf
meaningsrm.landmark ↔ lmm
The Directed-Motion construction
construction DIRECTED-MOTIONconstructional
constituentsa : Ref-Expm: Motion-Verbp : Spatial-Phrase
form af before mf
mf before pf
meaningevokes Directed-Motion as dmselfm.scene ↔ dm
dm.agent ↔ am
dm.motion ↔ mm
dm.path ↔ pm
schema Directed-Motionroles
agent : Entitymotion : Motionpath : SPG
What exactly is simulation?
• Belief update plus X-schema execution
hungry meeting
cafe
time of day
readystart
ongoingfinish
done
iterate
WALK
at goal
Analysis Process
Utterance
Simulation
Belief State
General Knowledg
e
Constructions
SemanticSpecificatio
n
“Harry is walking to the café.”
“Harry is walking to the café.”
readystart
ongoingfinish
done
iterateabortcancelled
interrupt resume
suspended
WALKwalker=Harry goal=cafe
Analysis Process
Utterance
Simulation
Belief State
General Knowledg
e
Constructions
SemanticSpecificatio
n
“Harry has walked into the wall.”
Perhaps a different sense of INTO?
construction INTO subcase of spatial-prep
form selff .orth ← “into”
meaningevokes Trajector-Landmark as tlevokes Container as contevokes Source-Path-Goal as spgtl.trajector ↔ spg.trajectortl.landmark ↔ contcont.interior ↔ spg.goalcont.exterior ↔ spg.source
construction INTO subcase of spatial-prep
form selff .orth ← “into”
meaningevokes Trajector-Landmark as tlevokes Impact as imevokes Source-Path-Goal as spgtl.trajector ↔ spg.trajectortl.landmark ↔ spg.goalim.obj1 ↔ tl.trajectorim.obj2 ↔ tl.landmark
“Harry has walked into the wall.”
readystart
ongoingfinish
done
iterateabortcancelled
interrupt resume
suspended
WALKwalker=Harry goal=wall
What about…
“Harry walked into trouble”
or for stronger emphasis,
“Harry walked into trouble, eyes wide open.”
Metaphors
• metaphors are mappings from a source domain to a target domain
• metaphor maps specify the correlation between source domain entities / relation and target domain entities / relation
• they also allow inference to transfer from source domain to target domain (possibly, but less frequently, vice versa)
<TARGET> is <SOURCE>
Event Structure Metaphor
• Target Domain: event structure
• Source Domain: physical space• States are Locations
• Changes are Movements
• Causes are Forces
• Causation is Forced Movement
• Actions are Self-propelled Movements
• Purposes are Destinations
• Means are Paths
• Difficulties are Impediments to Motion
• External Events are Large, Moving Objects
• Long-term Purposeful Activities are Journeys
KARMA
• DBN to represent target domain knowledge
• Metaphor maps link target and source domain
• X-schema to represent source domain knowledge
Metaphor Maps
1. map entities and objects between embodied and abstract domains
2. invariantly map the aspect of the embodied domain event onto the target domain
by setting the evidence for the status variable based on controller state (event structure metaphor)
3. project x-schema parameters onto the target domain
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
That’s the Regier model. (first half of semester)
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
VerbLearn
schema elbow jnt posture accel
slide extend palm 6
schema elbow jnt posture accel
slide extend palm 8
schema elbow jnt posture accel
slide 0.9 extend 0.9 palm 0.9 [6]
data #1
data #2
data #3
data #4
schema elbow jnt posture accel
depress 0.9
fixed 0.9 index 0.9 [2]
schema elbow jnt posture accel
slide 0.9 extend 0.9 palm 0.9 [6 - 8]
schema elbow jnt posture
slide 0.9 extend 0.9 palm 0.7
grasp 0.3
schema elbow jnt posture accel
depress fixed index 2
schema elbow jnt posture accel
slide extend grasp 2
Computational Details
• complexity of model + ability to explain data
• maximum a posteriori (MAP) hypothesis
)|( argmax DmPm
rule Bayes'by )()|( argmax mPmDPm
how likely is the data given this model?
penalize complex models – those with too many word senses
wants the best model given data
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
conflation hypothesis(primary metaphors)
How do you learn…
the meanings of spatial relations,
the meanings of verbs,
the metaphors, and
the constructions?
construction learning
Acquisition
Reorganize
Hypothesize
Production
Utterance
(Comm. Intent, Situation)
Generate
Constructions
(Utterance, Situation)
Analysis
Comprehension
Analyze
Partial
Usage-based Language Learning
Main Learning Loop
while <utterance, situation> available and cost > stoppingCriterion
analysis = analyzeAndResolve(utterance, situation, currentGrammar);
newCxns = hypothesize(analysis);
if cost(currentGrammar + newCxns) < cost(currentGrammar)
addNewCxns(newCxns);
if (re-oganize == true) // frequency depends on learning parameter
reorganizeCxns();
Three ways to get new constructions
• Relational mapping
– throw the ball
• Merging
– throw the block
– throwing the ball
• Composing
– throw the ball
– ball off
– you throw the ball offTHROW < BALL < OFF
THROW < OBJECT
THROW < BALL
Minimum Description Length• Choose grammar G to minimize cost(G|D):
– cost(G|D) = α • size(G) + β • complexity(D|G)
– Approximates Bayesian learning; cost(G|D) ≈ posterior probability P(G|D)
• Size of grammar = size(G) ≈ 1/prior P(G) – favor fewer/smaller constructions/roles; isomorphic mappings
• Complexity of data given grammar ≈ 1/likelihood P(D|G)– favor simpler analyses
(fewer, more likely constructions)
– based on derivation length + score of derivation
Connectionist Representation
How can entities and relations be represented at the structured connectionist level?
or
How can we represent Harry walked to the caféin a connectionist model?
SHRUTI
• entity, type, and predicate focal clusters
• An “entity” is a phase in the rhythmic activity.
• Bindings are synchronous firings of role and entity cells
• Rules are interconnection patterns mediated by coincidence detector circuits that allow selective propagation of activity
• An episode of reflexive processing is a transient propagation of rhythmic activity
• asserting that walk(Harry, café)
• Harry fires in phase with agent role
• cafe fires in phase with goal role
+ - ? agt goal
+e +v ?e ?v
+ ?
walk
cafe
Harry
type
entity
predicate
“Harry walked to the café.”
• asserting that walk(Harry, café)
• Harry fires in phase with agent role
• cafe fires in phase with goal role
+ - ? agt goal
+e +v ?e ?v
+ ?
walk
cafe
Harry
type
entity
predicate
“Harry walked to the café.”
Human Sentence Processing
Can we use any of the mechanisms we just discussed
to predict reaction time / behavior
when human subjects read sentences?
Good and Bad News
• Bad news:
– No, not as it is.
– ECG, the analysis process and simulation process are represented at a higher computational level of abstraction than human sentence processing (lacks timing information, requirement on cognitive capacity, etc)
• Good news:
– we can construct bayesian model of human sentence processing behavior borrowing the same insights
Bayesian Model of Sentence Processing
• Do you wait for sentence boundaries to interpret the meaning of a sentence? No!
• As words come in, we construct
– partial meaning representation
– some candidate interpretations if ambiguous
– expectation for the next words
• Model
– Probability of each interpretation given words seen
– Stochastic CFGs, N-Grams, Lexical valence probabilities
SCFG + N-gram
Reduced RelativeMain Verb
S
NP VP
D N VBN
The cop arrested the detective
S
NP VP
NP VP
D N VBD PP
The cop arrested by
Stochastic CFG
SCFG + N-gram
Main VerbReduced Relative
S
NP VP
D N VBN
The cop arrested the detective
S
NP VP
NP VP
D N VBD PP
The cop arrested by
N-Gram
SCFG + N-gram
Main VerbReduced Relative
S
NP VP
D N VBN
The cop arrested the detective
S
NP VP
NP VP
D N VBD PP
The cop arrested by
Different Interpretations
Predicting effects on reading time
• Probability predicts human disambiguation
• Increase in reading time because of...
– Limited Parallelism
• Memory limitations cause correct interpretation to be pruned
• The horse raced past the barn fell
– Attention
• Demotion of interpretation in attentional focus
– Expectation
• Unexpected words