incidental picture exposure affects later reading: evidence from the n400
TRANSCRIPT
Brain & Language 122 (2012) 64–69
Contents lists available at SciVerse ScienceDirect
Brain & Language
journal homepage: www.elsevier .com/locate /b&l
Short Communication
Incidental picture exposure affects later reading: Evidence from the N400
Leonora C. Coppens ⇑, Liselotte Gootjes, Rolf A. ZwaanErasmus University Rotterdam, P.O. Box 1738, 3000 DR Rotterdam, The Netherlands
a r t i c l e i n f o a b s t r a c t
Article history:Accepted 11 April 2012Available online 3 May 2012
Keywords:Embodied cognitionReading comprehensionVisual experienceERPN400
0093-934X/$ - see front matter � 2012 Elsevier Inc. Ahttp://dx.doi.org/10.1016/j.bandl.2012.04.006
⇑ Corresponding author.E-mail addresses: [email protected] (L.C. C
(L. Gootjes), [email protected] (R.A. Zwaan).
Language comprehenders form a mental representation of the implied shape of objects mentioned in thetext. In the present study, the influence of prior visual experience on subsequent reading was assessed. Intwo separate phases, participants saw a picture of an object and read a text about the object, suggestingthe same or a different shape. When the shapes in the two phases mismatched, ERPs during readingshowed a larger N400 amplitude than when the shapes matched, suggesting that a picture presentedincidentally 15 min earlier affected reading. These results further strengthen the case for the interactionof language and visual experience during language comprehension.
� 2012 Elsevier Inc. All rights reserved.
1. Introduction studies have in common is that the linguistic material appears
Words have meanings and senses. For example, the word‘‘bank’’ has at least two distinct meanings. One meaning refers tothe land along the side of a river. The other refers to a financialinstitution. These meanings are unrelated. A word like ‘‘chicken’’has multiple related meanings, or senses, one referring to thefeathered farm animal and the other to a fried piece of meat. Thereare even subtler shades of meaning. Compare In the room was anironing board versus In the closet was an ironing board. In the firstcase, the ironing board is most likely unfolded, whereas it is foldedin the latter case. Thus, the same referent may assume differentshapes depending on the context in which it occurs. Behavioralstudies have demonstrated that language comprehenders are sen-sitive to such subtle shades of meaning. They represent the orien-tation, shape, and motion direction of objects, even though theseare only implied by the text (Hirschfeld & Zwitserlood, 2011; Holt& Beilock, 2006; Stanfield & Zwaan, 2001; Zwaan, Madden, Yaxley,& Aveyard, 2004; Zwaan, Stanfield, & Yaxley, 2002).
In an oft-used paradigm, subjects read or hear sentences and arethen shown a picture. Their task is often to decide if the pictureshows an object in the sentence and in other experiments simplyto name the picture. In the sentences, the shape or orientation ofthe object is implied by the context. Pictures that match the con-text are responded to faster than mismatching pictures (Stanfield& Zwaan, 2001; Zwaan et al., 2002). In similar experiments, theseeffects were shown to be modulated by domain expertise (Holt &Beilock, 2006), reading span (Madden & Zwaan, 2006) and visualhalf-field presentation (Lincoln, Long, & Baynes, 2007). What these
ll rights reserved.
oppens), [email protected]
intermixed with the pictures and that a behavioral response(recognition or naming) to the picture is required. One mighttherefore argue that the results of those earlier studies came aboutbecause subjects were anticipating pictures during reading andtherefore began to process the sentences in a more ‘‘visual’’ man-ner while the experiment progressed. However, a post-hoc analysis(Stanfield & Zwaan, 2001) was not consistent with such a strategicaccount, given that the match effect was not larger in the secondhalf of their experiment than in the first half, which is what thisexplanation would predict. Moreover, Pecher, Van Dantzig, Zwaan,and Zeelenberg (2009) recently found match effects, even thoughthey presented the pictures in a surprise recognition test after allof the sentences had been read. In this study, participants madesensibility judgments about sentences that implied a certain shapeor orientation of an object. Only after all sentences had been pre-sented, matching or mismatching pictures of the objects appearedin a recognition task. Nevertheless, even in this study, a response tothe pictures was required.
A further commonality among earlier studies is that the picturealways followed the sentence (either immediately or after an inter-val). This suggests that the results of linguistic processing influencesubsequent visual processing. In order to firmly demonstrate thatlanguage and visual experience interact, it is also important to showthe reverse: that visual experience influences later language pro-cessing. A recent study demonstrated that prior exposure to picturesaffects eye movements during later reading, even though pictureexposure was incidental to the reading task (Wassenburg & Zwaan,2010). Participants first saw pictures of objects in a particular orien-tation in a word-picture verification task. After a filler task, they readabout the pictured objects in a sentence that constrained the orien-tation. Reading times were longer for phrases that described a pic-ture-mismatching orientation compared to a picture-matching
L.C. Coppens et al. / Brain & Language 122 (2012) 64–69 65
orientation. While informative about online processing, eye track-ing does not provide as much detail on the immediacy of the influ-ence of prior visual experience on language processing as doevent-related potentials (ERPs). The high temporal resolution ofERPs therefore provides a very strong test of the hypothesis that vi-sual experience and language processing closely interact.
In the present experiment, subjects read texts containing wordsthat had previously been associated with pictures of objects in aparticular shape, for example the word ironing board (one wordin Dutch) associated with a picture of a folded or unfolded ironingboard (depending upon the condition). How would incidentalpriming with a picture of a folded out ironing board affect subse-quent processing of the sentence In the closet was an ironing board?Theories that describe language comprehension as partly or en-tirely symbolic, such as grounding by interaction (Mahon & Caram-azza, 2008) or Collins and Loftus’ (1975) spreading-activationtheory of semantic processing, would suggest that seeing a pictureof an object facilitates later reading about the object. In this view,activation of a concept (the ironing board) facilitates reading aboutthe concept, regardless of the shape the item is in. In contrast, the-ories that assume that language comprehension involves a percep-tual simulation of the referential situation (e.g., Barsalou, 1999;Barsalou, 2008; Zwaan, 2004) might predict that priming with apicture would bring a perceptual representation to such a level ofactivation that it is likely to be incorporated in a mental simulationduring language comprehension. Evidently, priming of a contextu-ally appropriate shape would be more useful than priming with acontextually inappropriate shape. Or, to put it differently, primingwith a contextually inappropriate shape should lead to greatersemantic integration problems than priming with a contextuallyappropriate shape. Where can this proposed effect be expected tooccur? Take again the sentence In the closet was an ironing board.When the location is read, the primed concept is not yet known.The conflict between primed and read shape only occurs whenthe reader encounters the noun that refers to the concept, so thedifficulty in semantic integration should be on ironing board.
N400 is the most common ERP measure of ease of semanticintegration (Chwilla, Kolk, & Vissers, 2007; Hagoort, Hald,Bastiaansen, & Petersson, 2004; Van Berkum, Brown, & Hagoort,1999). The N400 is a negative deflection in the signal with a peakat about 400 ms post word onset (Kutas & Hillyard, 1980). Themore difficult it is to integrate a word into the unfolding mentalrepresentation, the larger the amplitude of the N400. Modulationsof the N400 are typically demonstrated by comparing differentwords in the same context and modulating the fit of the word inthe context. In the current study, we predicted N400 modulationsto occur within identical sentences. What makes the conditionsdifferent from one another is the incidental exposure to a picturethat occurs at least 15 min before the reading task. If the shapeof the object in the picture does not match the implied shape inthe text, N400 amplitude should be larger than when the shapesmatch. This would indicate the immediate activation of shape rep-resentation during reading.
Thus, the present research presents an advance on previousstudies on several dimensions. First, it assesses the impact of priorvisual experience on language processing. Second, it providesinformation about the time course of the activation of perceptualinformation during reading.
2. Method
2.1. Participants
Forty-one native Dutch speakers (33 females, mean age21 years, range 18–32 years) without problems with reading or
vision participated for course credit or a reward of €20. Prior tothe experiment, all participants gave written informed consent.
2.2. Materials and design
Sixty critical items were used (examples are included as Supple-mentary materials). Items were objects, animals, or people. Of eachitem, two pictures were selected in which the item was shown in adifferent shape. For example, pictures of the item ironing board de-picted a folded and an unfolded ironing board. Three-sentence pas-sages were written about the critical items. The first two sentencesof these passages provided a context. In the example of the ironingboard, these sentences were: Esther was looking for kitchen steps.She pulled open the hall closet. In the last sentence of the passagethe critical item appeared in a particular location, implying theshape of the item. In case of the ironing board, In the closet wasan ironing board implies the ironing board is folded. The pictureof the folded ironing board, would thus be a match, while the pic-ture of the unfolded ironing board would be a mismatch. Similarmaterials have been shown to elicit mismatch effects in other stud-ies (e.g., Zwaan et al., 2002). Additionally, we selected 70 picturesand 30 passages to be used as fillers. Filler items were intended tomake the experimental manipulation less obvious. The filler pas-sages were similar to the critical passages; after a context descrip-tion of two sentences, a third sentence was presented which endedwith a concrete imageable word.
Two counterbalanced lists were created, each containing 60critical items and 60 filler items. In each list all items appearedonce. Moreover, in each list, half of the pictures of the critical itemsmatched the shape implied in the later presented passage and theother half mismatched the shape. A passage that was paired with amatching picture in List A was paired with a mismatching picturein List B, and vice versa. Filler items were the same in both lists.
2.3. Procedure
The experiment was presented to the participants as threeindependent experiments. The first phase (‘experiment 1’) was aword-picture verification task. All 120 items in one of the counter-balancing lists (60 critical items and 60 filler items) were presentedtwice in random order. Words were presented for 500 ms, followedby a 200 ms mask, a 300 ms picture presentation and again a200 ms mask. Participants indicated as quickly as possible whetherthe word matched the depicted object. A short recognition testfollowed the word-picture verification task, to convince the partic-ipant that the first phase was a complete experiment. Ten newfiller pictures were used for this test in addition to 10 of the fillerpictures that had appeared earlier in the experiment. After this firstphase, participants performed a filler task (the ‘second experi-ment’) for at least 15 min. The filler task consisted of an emotionalword Stroop task, in which emotional and neutral words had to beread and judged. None of the words used in this task occurred inthe present experiment.
In the second phase (the ‘third experiment’), the passages in thelist were presented word by word while EEG signals were being re-corded. Presentation duration of each word was adjusted to wordlength, according to the Variable Serial Visual Presentation (VSVP)procedure described by Otten and Van Berkum (2008). To makesure word length differences could not have an effect on ERPsbecause of different presentation durations, critical words andthe words before and after the critical words were presented fora fixed duration based on the average length of words in thosepositions (400 ms for critical words, 280 ms for adjacent words).To ensure attentive reading, comprehension questions were askeddirectly after 15 of the passages. Participants responded to the yes/no questions by pressing a key on a response box. Note that the
66 L.C. Coppens et al. / Brain & Language 122 (2012) 64–69
sentences presented in the second phase were exactly identical forall participants. Only the pictures in the first phase differed be-tween subjects: either a shape-matching picture or a shape-mis-matching picture of each item was presented.
After the second phase of the experiment, participants wereasked if they had noticed anything special about the experiments,whether they thought any of the three experiments were related toeach other, and what they thought the experiments were about.None of the participants had noticed that the picture-word verifi-cation task and the reading task were related, and only one partic-ipant had noticed that the stimuli partially overlapped. Data fromthis participant did not differ from those of the other participants.
2.4. EEG recording
Participants sat in a comfortable reclining chair in a dimly litroom, separate from the experimenter. They were instructed tominimize movements. EEG was recorded from 64 active Ag/AgClelectrodes (BioSemi, Amsterdam, the Netherlands, ActiveTwoamplifier system) placed in an elastic cap according to the 10/20system. Eye movements and blinks were monitored by four addi-tional electrodes. Recordings were amplified using an ActiveTwoamplifier system and sampled at 256 Hz.
2.5. EEG analysis
EEG data were rereferenced offline to the linked mastoids. Seg-ments of 1100 ms were created, including 100 ms before the targetword onset (i.e., the name of the item). Epochs were filtered with a0.01–40 Hz band filter, and corrected for eye movements using thealgorithm of Gratton, Coles, and Donchin (1983). Epochs contain-ing an EEG signal exceeding 100 lV or a voltage step betweentwo sampling points of more than 50 lV were rejected by auto-matic detection and visual inspection. In each condition the maxi-mum number of data segments per participant was 30. If fewerthan 15 segments remained in any of the conditions after artifactrejection, the data of the participant were excluded from analysis(17 participants). From the remaining 24 participants, 5% of thesegments were rejected because of artifacts. Segments were nor-malized on the basis of the 100 ms pre-stimulus baseline and ERPsfor each participant were calculated by averaging trials for eachelectrode and condition separately. Based on the literature (e.g.,Chwilla et al., 2007), mean amplitude in the 300–500 ms timeinterval was used as a measure of the N400 effect.
2.6. Statistical analyses
First, an overall repeated-measures ANOVA was performed,with condition (match, mismatch) and electrode (Fp1, Fpz, Fp2,AF7, AF3, Afz, AF4, AF8, F7, F5, F3, F1, Fz, F2, F4, F6, F8, FT7, FT8,FC5, FC3, FC1, FCz, FC2, FC4, FC6, C5, C3, C1, Cz, C2, C4, C6, T7,T8, TP7, TP8, CP5, CP3, CP1, CPz, CP2, CP4, CP6, P9, P7, P5, P3, P1,Pz, P2, P4, P6, P8, P10, PO7, PO3, Poz, PO4, PO8, O1, Oz, O2 andIz) as factors. To examine the N400 effect more specifically, we se-lected 30 electrodes for a subsequent planned analysis (F3, F1, Fz,F2, F4, FC3, FC1, FCz, FC2, FC4, C3, C1, Cz, C2 C4, CP3, CP1, CPz, CP2,CP4, P3, P1, Pz, P2, P4, PO7, PO3, POz, PO4 and PO8). Theselocations are in line with the existing literature on the N400, whichsuggests that the N400 response to written words is mostpronounced over centro-parietal sites (Kutas & Federmeier,2011). The scalp distribution of the N400 effect over these 30 elec-trodes was explored using a 6 � 5 � 2 repeated measures ANOVA,with posteriority (frontal, fronto-central, central, centroparietal,parietal, parieto-occipital), laterality (left, left-midline, midline,right-midline, right), and condition (match, mismatch) as factors.The scalp distribution was further examined using five separate 6
(posteriority) � 2 (condition) ANOVAs, one for each level of lateral-ity. An alpha level of .05 was used for all statistical tests reported inthis paper.
3. Results
3.1. Behavioral data
Data from the word-picture verification task were examined toestablish accuracy for each participant and each item. All partici-pants and items had an accuracy score of over .80. To establishwhether both pictures of each object were equally good examplesof the item, we compared reactions to the shape-matching andshape-mismatching pictures. Mean accuracy and reaction times tothe pictures of the 24 participants included in the EEG analysis wereanalyzed separately for both types of pictures. Mean accuracy was.924 (SD = .202) for shape-matching pictures and .928 (SD = .209)for shape-mismatching pictures. Mean response latency for correctresponses was 383 ms (SD = 140) for shape-matching pictures and390 ms (SD = 140) for shape-mismatching pictures. Both paired-samples t-tests yielded no significant results (both ps > .4). To checkwhether these null results arose from a lack of power, the analysiswas repeated using data from all 41 participants. Mean accuracywas .952 (SD = .143) for shape-matching pictures and .957(SD = .143) for shape-mismatching pictures. Mean response latencyfor correct responses was 396 ms (SD = 138) for shape-matchingpictures and 404 ms (SD = 136) for shape-mismatching pictures.Both paired-samples t-tests yielded no significant results (bothps > .2). Subsequent items analyses, both using data from all partic-ipants and using data from the 24 EEG participants also revealed nodifferences between the two types of pictures: separate indepen-dent-samples t-tests on reaction times and accuracy of each itemyielded no significant results (all ps > .05).
3.2. ERP data
In Fig. 1, the grand averaged ERPs are shown. As can be seen,there is a clear difference between match and mismatch wordsat 300–500 ms which is the time interval that is generally usedin literature to represent the N400 (e.g., Chwilla et al., 2007).Fig. 2 shows the scalp distribution for the difference wave in thisinterval and indicates that the N400 difference is most pronouncedover centro-parietal sites, which is typical for N400 responses towritten words (Kutas & Federmeier, 2011). Visual inspection ofthe grand average ERP waveforms also suggests possible differ-ences between 0 and 300 ms after stimulus onset, before theN400 time window. However, an overall 2 � 64 ANOVA on thisearly time window to examine these possible early differencesyielded only a significant main effect of electrode,F(63,1449) = 3.259, p < .001, g2
p = .124, and no main effect of condi-tion or Condition � Electrode interaction (both Fs < 1.1). In addition,at parietal electrodes the difference between match and mismatchseems to sustain beyond 500 ms. An overall 2 � 64 ANOVA on the500–700 ms time window to investigate these possible sustainedeffects yielded only a significant main effect of electrode,F(63,1449) = 6.935, p < .001, g2
p = .232, and no main effect of condi-tion or Condition � Electrode interaction (both Fs < 1.3).
The overall ANOVA examining the N400 effect at all 64 elec-trodes revealed a main effect of electrode, F(63,1449) = 4.084,p < .001, g2
p = .151 and a Condition � Electrode interaction effect,F(63,1449) = 5.432, p < .001, g2
p = .071. The subsequent 6 � 5 � 2ANOVA examining the topography of the N400 effect at 30 elec-trodes revealed a main effect of condition, F(1,23) = 5.499, p =.028, g2
p = .193, a main effect of posteriority, F(5,19) = 4.930, p =.005, g2
p = .565, and an interaction between laterality and condition,
Fig. 1. Grand average event-related potentials at the analyzed electrodes for matching and mismatching pictures.
L.C. Coppens et al. / Brain & Language 122 (2012) 64–69 67
F(4,20) = 4.983, p = .006, g2p = .499, indicating that the N400 match
effect was not evenly distributed in laterality. All other maineffects and interactions were not significant (all Fs < 2.1). The sep-arate ANOVAs examining the outer-left and outer-right electrodesrevealed no effect of condition or interactions with condition.However, the three ANOVAs examining the electrodes around themidline showed only a significant main effect of condition:F(1,23) = 5.811, p = .024, g2
p = .202 for the left-midline electrodes,F(1,23) = 6.437, p = .018, g2
p = .219 for the midline electrodes andF(1,23) = 7.675, p = .011, g2
p = .250 for the right-midline electrodes,indicating that at the midline and next-to-midlines electrodes theamplitude of the N400 was larger in the mismatch condition thanin the match condition.
4. Discussion
Prior exposure to a picture of an object in a specific shape influ-enced the amplitude of the N400 during later reading about thatobject. Words embedded in a text that implied a certain shape
elicited a larger N400 when they had been preceded by a shape-mismatching picture than a shape-matching picture. A largerN400 amplitude indicates more difficulty to integrate the wordwith the sentence (Van Berkum et al., 1999). In the present studyhowever, we found N400 differences between identical wordsand sentences. The only difference between the two conditionswas the presentation of a matching or a mismatching picture15 min beforehand. Therefore, the present results suggest that see-ing a mismatching picture influenced later reading in an ostensiblyunrelated task. Apparently, the shape representation encountered15 min beforehand was active during reading and interfered withforming a coherent representation of the presented sentence. Be-cause the reading task in the current experiment did not requireany visual processing beside reading (there were no pictures tobe rated or studied during reading), it seems that this activationof perceptual representations does not depend on instruction ortask properties.
These results are consistent with the eye tracking study byWassenburg and Zwaan (2010), who found prior exposure to ori-entation matching or mismatching pictures affected later reading
Fig. 2. Scalp distribution for the difference wave in the 300–500 ms interval. Theanalyzed electrode sites are shown in black.
68 L.C. Coppens et al. / Brain & Language 122 (2012) 64–69
times. In that study, the orientation of objects was manipulated (asin Stanfield & Zwaan, 2001), whereas in the present study theshape of objects was manipulated (as in Zwaan et al., 2002). Thepresent results further strengthen the case for the influence of per-ceptual processes on language comprehension, in addition to theinfluence of language on perceptual processes. Moreover, the ERPdata collected in the present study provide insight into the pro-cesses underlying the behavioral outcomes of earlier studies. Theincreased amplitude of the N400 suggests that it was more difficultfor participants to integrate the critical word in the sentence. Thisintegration difficulty could cause longer response latencies andreading times, such as those found by Wassenburg and Zwaan(2010).
The current findings speak to a number of other findings regard-ing the N400 and mental simulation in language comprehension.At first sight, the present results are at odds with findings byHirschfeld, Zwitserlood, and Dobel (2011). These researchers usedmagnetoencephalography (MEG) to examine what stages of visualprocessing are affected by information conveyed through languagein the sentence-picture verification task from Stanfield and Zwaan(2001) and Zwaan et al. (2002). They observed two effects. Theyfound a striking early effect (around 120 ms post picture onset)in the occipital cortex as well as a later effect in the N400 time-window in the left-temporal cortex. The former effect was strongerin the match than in the mismatch condition. In the N400 window,however, the brain response in the left-temporal cortex was insen-sitive to the difference between match and mismatch and the onlydifference was between these two conditions on the one hand anda condition in which a picture unrelated to the sentence wasshown on the other. Our results contrast with this latter finding.We found a clear N400 modulation as a function of match. WhileEEG and MEG are sensitive to different subsets of neuronal activity,a critical difference between the Hirschfeld et al. (2011) findingsand ours is that in their case the N400 response was to the presen-tation of a picture subsequent to the sentence, whereas our N400response occurred during the comprehension of the sentence aftera picture presentation much earlier. Their findings suggest that lin-guistic context has an immediate effect on picture recognition invisual areas of the brain. Hirschfeld and Zwitserlood (2011) explainthis finding by proposing that linguistic context can amplify subse-quent visual processing via reentrant top-down processing fromfrontal cortex to visual processing in the fusiform gyrus. They
showed that the match effect in the picture-verification task(Stanfield & Zwaan, 2001; Zwaan et al., 2002) is due to low-spatialfrequency information, which is processed via the magnocellularpathway. It is this information that is putatively affected by lin-guistic context, leading to faster behavioral responses in the matchthan in the mismatch condition. We surmise that the nature of thisinterpretation would have to be visual or at least in a form that al-lows it to amplify low-frequency visual information. Hirschfeldand Zwitserlood caution that their explanation is silent on whathappens during online sentence comprehension, given that—dueto the nature of the experimental procedure—the sentence is pre-sumably comprehended by the time the picture is shown.
Our use of the visual-memory task (see also Wassenburg &Zwaan, 2010) allowed us to examine the match effect during on-line comprehension. Our findings show that the N400 during lan-guage comprehension is sensitive to a perception-based match.Hirschfeld et al. explain their N400 finding—the lack of a match ef-fect combined with a relatedness effect—by invoking amodal rep-resentations. It would seem that invoking such a level ofrepresentation is unnecessary to explain our finding. Because theverbal stimuli were identical across the match and mismatch con-ditions, prior visual experience must be the source for the differen-tial N400 modulation. This leads to the hypothesis that elements ofthe sensorimotor simulation performed during language compre-hension interact with the visual memory of the object that wastriggered by the target word. In the match condition, this memoryrepresentation was one that could be straightforwardly integratedwith the evolving situation model (Zwaan & Radvansky, 1998). Inthe mismatch condition, however, this memory representationcould not be easily integrated into the situation model. On thisexplanation, there is no need for an amodal level of representation.Our position aligns with that of Kutas and Federmeier (2011), whopropose that the N400 is a temporal interval in which unimodalsensory analysis brings about multimodal associations that are in-formed by and affect long-term memory. Such a multi-modalaccount of long-term memory is consistent with proposals in theembodied-cognition literature (e.g., Barsalou, 1999; Damasio,1989; Glenberg, 1997; Zwaan, 2004).
This interpretation is also consistent with the results of a num-ber of recent studies that examined N400 modulations as a func-tion of context. For example, Knoeferle, Urbach, and Kutas (2011)used a picture-sentence verification task and found that priorexposure to the picture affected ERP responses throughout the sen-tence. Given that the picture, which showed an action involvingtwo individuals (e.g., a gymnast punching vs. applauding a journal-ist), either matched or mismatched the sentence on the verb only,this rules out a simple priming account. Similar to our argumenthere, Knoeferle and colleagues conclude that the effects theyobserve on the N400 responses are visual context effects. The dif-ference between the task they used and our visual-memory taskis that in our case the visual experience was incidental to the read-ing task, whereas it was an explicit and integral part of Knoeferleet al.’s sentence-verification task. Thus, our results suggest thatthese visual context effects are automatic rather than strategic.
The multi-modal integration account further converges withother results, such as those of a story comprehension experiment(Chwilla et al., 2007) in which object affordances were found tomodulate the N400. For example, a sentence stating that a pulloverwas used to paddle produced a larger N400 than a sentence statingthat a frisbee was used for paddling. Given that neither frisbee norpullover are close semantic associates of paddle, this N400 modula-tion has to be due to the affordances of the objects. A pullover isnot sturdy enough to serve as a paddle, whereas a frisbee is. Be-cause the researchers had controlled their sentences and targetwords for semantic associations, the N400 in this study appearsto have been modulated by embodied mental simulations, for this
L.C. Coppens et al. / Brain & Language 122 (2012) 64–69 69
is the only conceivable way that a discrepancy between the condi-tions could come about. Our results paint a similar picture.
Finally, our results converge in a broader sense with two recentstudies (Collins, Pecher, Zeelenberg, & Coulson, 2011; Hald,Marshall, Janssen, & Garnham, 2011) that have used a property-verification task and found that perceptual modalities (specificallyvision) provide a context for the processing of subsequent words,such that words implying the same modality produce a smallerN400 amplitude than words that imply a different modality (e.g.,audition). For example, if the task is to verify that buttermilk isopaque, which activates the visual modality, the N400 amplitudeis smaller when that trial was preceded by a trial in which it hadto be verified that asparagus are green, which also activates the vi-sual modality, than on a trial in which it had to be verified thathigh heels click, which activates the auditory modality.
Thus, there is an emerging body of evidence in support of theidea that language comprehension is a multi-modal integrationprocess and that the N400 indexes this process (Kutas & Federme-ier, 2011). Studies of online comprehension that combine hightemporal with high spatial resolution are needed to further unravelthe inner workings of this process.
Acknowledgments
We would like to thank members of the Language and CognitionGroup for their comments on an earlier draft of this manuscript.
Appendix A. Supplementary material
Supplementary data associated with this article can be found, inthe online version, at http://dx.doi.org/10.1016/j.bandl.2012. 04.006.
References
Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22,577–660. http://dx.doi.org/10.1017/S0140525X99002149.
Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59,617–645. http://dx.doi.org/10.1146/annurev.psych.59.103006.093639.
Chwilla, D. J., Kolk, H. H., & Vissers, C. T. W. M. (2007). Immediate integration ofnovel meanings: N400 support for an embodied view of languagecomprehension. Brain Research, 1183, 109–123. http://dx.doi.org/10.1016/j.brainres.2007.09.014.
Collins, A. M., & Loftus, E. F. (1975). A spreading-activation theory of semanticprocessing. Psychological Review, 82, 407–428. http://dx.doi.org/10.1037/0033-295X.82.6.407.
Collins, J., Pecher, D., Zeelenberg, R., & Coulson, S. (2011). Modality switching in aproperty verification task: An ERP study of what happens when candles flickerafter high heels click. Frontiers in Cognition, 2, 10. http://dx.doi.org/10.3389/fpsyg.2011.00010.
Damasio, A. R. (1989). Time-locked multiregional retroactivation: A systems-levelproposal for the neural substrates of recall and recognition. Cognition, 33,25–62. http://dx.doi.org/10.1016/0010-0277(89)90005-X.
Glenberg, A. M. (1997). What memory is for: Creating meaning in the service ofaction. Behavioral and Brain Sciences, 20, 41–50. http://dx.doi.org/10.1017/S0140525X97470012.
Gratton, G., Coles, M. G., & Donchin, E. (1983). A new method for off-line removal ofocular artifact. Electroencephalography and Clinical Neuropsychology, 55,468–484. http://dx.doi.org/10.1016/0013-4694(83)90135-9.
Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. M. (2004). Integration of wordmeaning and world knowledge in language comprehension. Science, 304,438–441. http://dx.doi.org/10.1126/science.1095455.
Hald, L. A., Marshall, J.-A., Janssen, D. P., & Garnham, A. (2011). Switching modalitiesin a sentence verification task: ERP evidence for embodied language processing.Frontiers in Cognition, 2, 45. http://dx.doi.org/10.3389/fpsyg.2011.00045.
Hirschfeld, G., & Zwitserlood, P. (2011). How vision is shaped by languagecomprehension - top-down feedback based on low-spatial frequencies. BrainResearch, 1377, 78–83. http://dx.doi.org/10.1016/j.brainres.2010.12.063.
Hirschfeld, G., Zwitserlood, P., & Dobel, C. (2011). Effects of languagecomprehension on visual processing: MEG dissociates early perceptual andlate N400 effects. Brain and Language, 116, 91–96. http://dx.doi.org/10.1016/j.bandl.2010.07.002.
Holt, L. E., & Beilock, S. L. (2006). Expertise and its embodiment: Examining theimpact of sensorimotor skill expertise on the representation of action-relatedtext. Psychonomic Bulletin and Review, 13, 694–701. http://dx.doi.org/10.3758/BF03193983.
Knoeferle, P., Urbach, T. P., & Kutas, M. (2011). Comprehending how visual contextinfluences incremental sentence processing: Insights from ERPs and picture-sentence verification. Psychophysiology, 48, 495–506. http://dx.doi.org/10.1111/j.1469-8986.2010.01080.x.
Kutas, M., & Federmeier, K. (2011). Thirty years and counting: finding meaning inthe N400 component of the event-related brain potential (ERP). Annual Reviewof Psychology, 62, 621–647. http://dx.doi.org/10.1146/annurev.psych.093008.131123.
Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentialsreflect semantic incongruity. Science, 207, 203–205. http://dx.doi.org/10.1126/science.7350657.
Lincoln, A. E., Long, D. L., & Baynes, K. (2007). Hemispheric differences in theactivation of perceptual information during sentence comprehension.Neuropsychologia, 45, 397–405. http://dx.doi.org/10.1016/j.neuropsychologia.2006.06.010.
Madden, C. J., & Zwaan, R. A. (2006). Perceptual representation as a mechanism oflexical ambiguity resolution: An investigation of span and processing time.Journal of Experimental Psychology: Learning, Memory, and Cognition, 32,1291–1303. http://dx.doi.org/10.1037/0278-7393.32.6.1291.
Mahon, B. Z., & Caramazza, A. (2008). A critical look at the embodied cognitionhypothesis and a new proposal for grounding conceptual content. Journal ofPhysiology – Paris, 102, 59–70.
Otten, M., & Van Berkum, J. (2008). Discourse-based word anticipation duringlanguage processing: Prediction or priming? Discourse Processes, 45, 464–496.http://dx.doi.org/10.1080/01638530802356463.
Pecher, D., Van Dantzig, S., Zwaan, R. A., & Zeelenberg, R. (2009). Languagecomprehenders retain implied shape and orientation of objects. QuarterlyJournal of Experimental Psychology, 62, 1108–1114. http://dx.doi.org/10.1080/17470210802633255.
Stanfield, R. A., & Zwaan, R. A. (2001). The effect of implied orientation derived fromverbal context on picture recognition. Psychological Science, 12, 153–156. http://dx.doi.org/10.1111/1467-9280.00326.
Van Berkum, J. J. A., Brown, C. M., & Hagoort, P. (1999). Early referential contexteffects in sentence processing: Evidence from event-related brain potentials.Journal of Memory and Language, 41, 147–182. http://dx.doi.org/10.1006/jmla.1999.2641.
Wassenburg, S. I., & Zwaan, R. A. (2010). Readers routinely represent implied objectrotation: The role of visual experience. Quarterly Journal of ExperimentalPsychology, 63, 1665–1670. http://dx.doi.org/10.1080/17470218.2010.502579.
Zwaan, R. A. (2004). The immersed experiencer: Toward an embodied theory oflanguage comprehension. Psychology of Learning and Motivation: Advances inResearch and Theory, 44, 35–62. http://dx.doi.org/10.1016/S0079-7421(03)44002-4.
Zwaan, R. A., Madden, C. J., Yaxley, R. H., & Aveyard, M. E. (2004). Moving words:Dynamic representations in language comprehension. Cognitive Science, 28,611–619. http://dx.doi.org/10.1207/s15516709cog2804_5.
Zwaan, R. A., & Radvansky, G. A. (1998). Situation models in languagecomprehension and memory. Psychological Bulletin, 123, 162–185. http://dx.doi.org/10.1037//0033-2909.123.2.162.
Zwaan, R. A., Stanfield, R. A., & Yaxley, R. H. (2002). Language comprehendersmentally represent the shapes of objects. Psychological Science, 13, 168–171.http://dx.doi.org/10.1111/1467-9280.00430.