Cognitive Processes during Text Comprehension

Paul van den Broek

University of Minnesota

Outline

1. Components of text (discourse) comprehension1. Product2. processes

2. A framework that captures the complex interactions between on-line processes during reading and the gradual construction of a memory representation of the text

3. Connections to cognitive neuroscience

Cognitive Theory

Computational Model

Neuro-imagingStudies

1A. The product of discourse comprehension: Episodic Memory Representation

A network representation, consisting of elements from text and semantic memory connected by semantic relations (e.g., Kintsch, 1988; Graesser & Clark, 1985; Trabasso & van den Broek, 1985)

During reading, the reader activates information relevant to the current text, establishing semantic relations between parts of the text between text and background knowledge

Referential and Causal relations

• Example

The lady gave the waiter $100He returned to give her the change

• In complete texts, relations are more complex:– May require background knowledge– May extend over long distances– May require coordination of multiple pieces of information

• The language may signal:The moon exerts gravitational pull on the earth, and thus played an important role in the development of life on earth.

1

4

7

2

5

16 15

10

6

3

11

13

9

8

12

18

14

17 19 20

22 21

Memory as a function of number of connections in network

0

10

20

30

40

50

60

1 2 3 4 5+

Number of Connections

Perc

ent

Rec

all

Priming as a function of network distance

0

10

20

30

40

50

60

70

1 2 3 4

Steps

prim

ing

(ms)

priming

Evidence for Network Representation

• The more connections an event has, the better it is remembered

• Reminding someone of one part of the text primes parts that are related more than parts that are not related (e.g., naming times, lexical decisions, speeded recognition)

• Events with many connections are judged more important, more often included in summaries

1b. Cognitive processes during comprehension

• Limited attentional capacity

• As information becomes activated, other information becomes deactivated

• Fluctuating activations of concepts as reader proceeds through text

Fluctuating activations-A view from the sentence

• At each reading cycle, four possible sources of activation:– Current input– Carry over from prior text

– Episodic memory representation of prior text (occasionally physical text)

– Semantic (background) knowledge

• Activation from these sources occurs through distinct processes

Comprehension processes

• Two mechanisms for accessing memory (Hagoort

M):– spread-of-activation (memory-based, associative)

processes– coherence-oriented processes

• Resulting in – Look-backs– Reinstatement from episodic memory – Retrieval from semantic memory– Connecting inferences (content in Working memory)

Testing hypotheses about reading processes

Naming, speeded recognition, lexical decision Speed of response is indication of activation (e.g., O’Brien &

Myers, 1987; McKoon & Ratcliff, 1990; Singer, 1994)

Reading times, Gaze duration/eye fixations, intonation in reading aloud

Example: Lexical Decision

…

…John dropped the banana peel on the floor

Mary fell on her back

…

…

‘slip’

but also ‘eat’, ‘yellow’’, ‘pain’

Memory-based and Coherence-oriented processes

Balancing act:– limited attentional (working memory) capacity– Need for coherence –standards of coherence (Hagoort: C)

• Standards depend on:– Individual differences– Reading goals/instructions– Linguistic input properties

• genre• syntax, vocabulary• connectors• Input order

• Standards of coherence pertain to both– Which types of relations matter (causal, referential, others)– The strength of the required relations

• ‘Good enough’ , shallow vs. deep processing

2. Connecting processes and product –the Landscape Model

• The above mechanisms and processes do not take place in isolation– simultaneously– dynamically (different combinations)– interactively (Memory-based may support/interfere

with coherence-oriented)• To keep track of the various processes: computational

model (cf. Hagoort)

1. A young knight rode through the forest.2. The knight was unfamiliar with the country.3. Suddenly, a dragon appeared.4. The dragon was kidnapping a beautiful princess.5. The knight wanted to free her.6. He wanted to marry her.7. The knight hurried after the dragon.8. They fought for life and death,9. Soon, the night’s armor was completely scorched.10. At last, the knight killed the dragon.11. He freed the princess.12. The princess was very thankful to the knight.13. She married the knight.

Gradual emergence of a memory representation

• At each cycle, activated concepts are added (or strengthened) in memory representation of the text

• Co-activated concepts are connected (or existing connections strengthened) (Hagoort: U): – asymptotic (delta) rule– as function of strength of activations of the concepts

• Computationally: changing connection matrix

• Concepts form interconnected clusters/cohorts– Basis for re-activation in subsequent cycles

Interaction process-product

• Reading is the dynamic interaction between process/activation and gradually emerging mental representation

• Over time:– Fluctuating activations– Different processes (memory-based and coherence-

oriented, with subprocesses)– Changes in episodic memory representation

Cognitive Theory

Computational Model

Neuro-imagingStudies

Connecting to neuroscience

Connections to neuro-imaging studies

• By providing detailed description of component processes discourse models can inform neuro-imaging research (and, at same time, be tested)– Avoid molar questions, such as where does causal coherence

take place, where does referential coherence take place?– Component processes are recruited by both, at different text

points– Basis for experimentation

• Insights about brain function provide constraints for theoretical models

• On-line record of activities can be matched to time course of neural activity

Constraints on theoretical model

• Semantic constraints influence generation of inferences during reading: – the stronger the constraints, the more quickly and

reliably inference is generated• Possible model: particular inference is

generated in particular (set of brain structures, constraints influence efficiency and ease

• But possible hemispheric specialization: generation of associates under high constraints: more LH; low constraints: RH (Beeman, Virtue, Sundermeier, Marsolek, others)

Neural Pathways From the Eye to the Brain

Divided Visual Field Paradigm

Lexical Decision TaskLexical Decision Task

Inference word or Inference word or non-word was non-word was presented quickly to presented quickly to the participant's left the participant's left or right visual fieldor right visual field

Bridging Inference Example (Virtue, van den

Broek, & Linderholm, 2006)

Todd was enjoying a vacation with his friends in the French Riviera. He took off his shirt and shoes, and he went for a walk on the beach. At first, Todd didn't see all of the beautiful shells in the sand as he waded out into the water.

High ConstraintSuddenly he called out to his friends holding his bleeding foot.

Low ConstraintSuddenly he called out to his friends holding a seashell.

Target Word cut

Facilitation for Low & High Constraint Bridging Inferences

-20

0

20

40

60

80

100

120

Ave

rage

Fac

ilit

atio

n (m

s)

Left Hemisphere

High Constraint

Low Constraint

Right Hemisphere

Predictive Inference Example

Tom and Krista were standing together holding hands.Both of them were a little nervous, but mostly excited about today. Tom imagined the future as he looked at Krista.

High ConstraintThey were just pronounced as man and wife.

Low ConstraintThey were just announced as college graduates.

Target Wordkiss

Facilitation for Low & High Constraint Predictive Inferences

0

25

50

75

100

125

Ave

rage

Fac

ilit

atio

n (m

s)

Left Hemisphere

High Constraint

Low Constraint

Right Hemisphere

Conclusions

Inferences under different degrees of constraints involve different processes –not same processes executed with different degrees of efficiency/ease

Neurological processes

• Cognitive processes in computational models can be matched with ongoing neuro-imaging data

Methods • This is the story of a boy named Tuk

who lived in the Artic. He wanted to show that he could be brave by hunting for big animals like his father who was a great hunter. Some people do not like the idea of hunting, but Tuk’s family relies on animals for most of their food and clothing needs…

• Narrative• 118 clauses

• Magnetoencephalography (MEG)– 248 channel axial gradiometer system (Magnes

3600WH, 4d-Neuroimaging, San Diego, CA)– Sampling: 1017.25Hz (0.1 - 400 Hz)– Non-invasive– Real-time recording

• Task– Reading comprehension

Methods

setting events/actions integration (temporal, spatial) (causal, explanations)

Results “Tuk the Hunter”

Conclusion • During reading, multiple processes take place, in dynamic and interatcive ways• Discourse models described processes during reading in considerable detail • Mechanisms

– Spread-of-activation type processes– Coherence-oriented processes

• Fluctuating activations involving– four sources of activation– subprocesses

• Standard of coherence modulate the mix– E.g., Causal, referential

• Gradual emergence of coherent memory representation• We have just scratched the surface of neurological investigations of text

comprehension• Subprocesses (memory-based activation, reinstatement, etc.) rather than global

comprehension processes (causal coherence, spatial layout).