56 semantic unification - mpg.pure

Y

56 SemanticUnification

peterhagoort,giosubaggio,androelm.willems

abstract Languageandcommunicationareabouttheexchangeof meaning. A key feature of understanding and producing lan-guageistheconstructionofcomplexmeaningfrommoreelemen-tarysemanticbuildingblocks.Thefunctionalcharacteristicsofthissemantic unification process are revealed by studies using event-related brain potentials. These studies have found that wordmeaning isassembled intocompoundmeaning innotmorethan500ms. World knowledge, information about the speaker, co-occurringvisualinput,anddiscourseallhaveanimmediateimpactonsemanticunificationandtriggerelectrophysiologicalresponsesthat are similar to those triggered by sentence-internal semanticinformation.Neuroimaging studies show thatanetworkofbrainareas, including the left inferior frontal gyrus, the left superior/middletemporalcortex,theleftinferiorparietalcortex,and,toalesser extent, their right-hemispherehomologuesare recruited toperformsemanticunification.

Ultimately, language is the vehicle for the exchange ofmeaningbetweenspeakerandlistener,betweenwriterandreader.Theuniquefeatureofthisvehicleisthatitenablestheassemblyofcomplexexpressionsfromsimplerones.Thecognitive architecture necessary to realize this expressivepower is tripartite in nature, with levels of form (sound,graphemes,manualgestures insign language),syntax,andmeaning as the core components of our language faculty(Jackendoff, 1999, 2002; Levelt, 1999). The principle ofcompositionalityisofteninvokedtocharacterizetheexpres-sivepowerof languageat the level ofmeaning.Themoststrictaccountofcompositionalitystatesthatthemeaningofanexpressionisafunctionofthemeaningsofitspartsandthewaytheyaresyntacticallycombined (Fodor&Lepore,2002;Heim&Kratzer,1998;Partee,1984).Inthisaccount,complexmeaningsareassembledbottom-upfromthemean-ingsofthelexicalbuildingblocksbymeansofthecombina-torialmachineryofsyntax.Thisprocessissometimesreferredtoassimplecomposition(Jackendoff,1997).Thatthisisnotwithout problems can be seen in adjective-noun construc-tions suchas“flat tire,”“flatbeer,”“flatnote,”and soon(Keenan,1979).Inall thesecases, themeaningof“flat” isquitedifferentandstronglycontextdependent.Forthisandotherreasons,simplecompositionseemsnottoholdacross

allpossibleexpressions in the language (foradiscussionofthisandotherissuesrelatedtocompositionality,seeBaggio,vanLambalgen,&Hagoort,inpress).Oneofthechallengesforacognitiveneuroscienceoflanguageistoaccountforthefunctional and neuroanatomical underpinnings of onlinemeaningcomposition.

In linking the requirements of the language systemas instantiated in the finite and real-time machinery ofthe human brain to the broader domain of cognitiveneuroscience, three functional components are consideredto be the core of language processing (Hagoort, 2005).The first is the memory component, which refers to thedifferent types of language information stored in long-term memory (the mental lexicon) and to how this infor-mationisretrieved(lexicalaccess).Theunificationcomponentrefers to the integration of lexically retrieved informationinto a representation of multiword utterances, as well asthe integration of meaning extracted from nonlinguisticmodalities; this component is at the heart of the com-binatorialnatureoflanguage.Finally,thecontrolcomponentrelates language to action, and is invoked, for instance,whenthecorrecttarget languagehastobeselected(inthecase of bilingualism) or for handling turn taking duringconversation.Inprinciple,thisMUC(memory,unification,control) framework applies to both language productionand language comprehension, although details of theirfunctional anatomy within each component will bedifferent. The focus of this chapter is on the unificationcomponent.

Classically, psycholinguistic studies of unification havefocusedonsyntacticanalysis.However,aswesaw,unifica-tionoperationstakeplacenotonlyatthesyntacticprocess-inglevel.Combinatorialityisahallmarkoflanguageacrossrepresentationaldomains (cf. Jackendoff,2002).Thus,alsoatthesemanticandphonologicallevels,lexicalelementsarecombinedandintegratedintolargerstructures(cf.Hagoort,2005). In the remainder of this chapter, we will discusssemanticunification.Semanticunificationreferstotheinte-gration of word meaning into an unfolding representationoftheprecedingcontext.Thisismorethantheconcatena-tion of individual word meanings, as is clear from theadjective-noun examples given earlier. In the interactionwiththeprecedingsentenceordiscoursecontext,theappro-priatemeaningisselectedorconstructed,sothatacoherentinterpretationresults.

hagoort,baggio,andwillems:semanticunification 1

peter hagoort Donders Institute for Brain, Cognition andBehaviour, Rodboud Univerisity; Max Planck Institute forPsycholinguistics,Nijmegen,TheNetherlands.roel m. williams Donders Institute for Brain, Cognition andBehaviour,RodboudUniverisityNijmegen,TheNetherlands.

1

Gazzaniga_56_Ch56.indd 1 2/17/2009 6:58:54 PM

Y

2 language

Hereafterwewillfirstdiscussthefunctionalcharacteristicsof semantic unification as revealed by ERP and MEGstudies.Next,resultsfromfMRIstudieswillbediscussedtoidentifytheneuralnetworksofsemanticunification.Intheremainder we will use the terms unification and integrationinterchangeably.However,inthelastparagraphweproposeto use the terms integration and unification for two differentwaysofcombininginformation.

Functional characteristics of semantic unification

Insightsintothefunctionalcharacteristicsofsemanticunifi-cationhavebeenespeciallygainedthroughaseriesofevent-related potential (ERP) studies. Most studies on semanticunificationexploitthecharacteristicsoftheso-calledN400componentintheERPwaveform.KutasandHillyard(1980)were thefirst toobserve thisnegative-goingpotentialwithanonsetatabout250msandapeakaround400ms(hencetheN400),whoseamplitudewasincreasedwhentheseman-tics of the eliciting word (i.e., socks) mismatched with thesemanticsofthesentencecontext,asin“Hespreadhiswarmbreadwithsocks.”

Since its original discovery in 1980, much has beenlearnedabouttheprocessingnatureoftheN400(forexten-siveoverviews,seeKutas&Federmeier,2000;Kutas,VanPetten,&Kluender,2006;Osterhout,Kim,&Kuperberg,2007).Inparticular,asKutasandHillyard(1984)andmanyothershaveobserved, theN400effectdoesnotdependona semantic violation. For example, subtle differences insemantic expectancy, as between mouth and pocket in thesentencecontext“Jennyputthesweetinhermouth/pocketafter the lesson,” can also modulate the N400 amplitude(Hagoort & Brown, 1994). Specifically, as the degree ofsemanticfitbetweenawordand its context increases, theamplitude of the N400 decreases. This general relationbetweenindividualwordmeaningsandthesemanticsofthecontextisindependentoftypeofcontext.Thatis,itisfoundfor a single-word context (Holcomb,1993), for a sentencecontext(Kutas&Hillyared,1980,1984),andforlargerdis-courses (van Berkum et al., 1999). Because of such subtlemodulations,theN400isgenerallytakentoreflectprocessesinvolved in the integration of the meaning of a word intotheoverallsemanticrepresentationconstructedforthepre-cedinglanguageinput(Brown&Hagoort,1993;Osterhout&Holcomb,1992).However,differentviewsexistastowhatbrings about the N400 integration effect. Federmeier andKutas (1999;Kutas&Federmeier,2000)proposed that inadditiontoitssensitivitytocontext,theN400isalsosensitivetotheeaseofaccessinginformationfromsemanticmemory.Assuch,theN400canbeseentoreflecttheorganizationof(lexical) meaning in semantic memory. According to thisview, the N400 amplitude is modulated by the degree towhich the context contains retrieval cues for accessing or

selecting the stored representation for a particular wordmeaning. Recent evidence in favor of this position wasobtainedinastudybyDeLong,Urbach,andKutas(2005).TheseauthorsfoundanN400effecttoanindefinitearticle(an versus a) that excluded the semantically expected con-tinuation, suchas in“thedaywasbreezy so theboywentout to fly an...,” where kite would be the contextuallyexpectednoun.Thisresultsuggestsacontextualpreactiva-tionof the targetword.However,otherrecentevidence ismore compatible with a unification account. Li, Hagoort,andYang(2008)investigatedtheneurophysiologicalresponseto manipulations of information structure. An importantdistinction at the level of semantic/conceptual structure isthatbetweenconceptualcontentandinformationstructure.Thelatterreferstothedivisionofthecontentofasentenceinto information that is in the foreground or in the back-ground (topic/focus; given/new). In many languages newinformation is accented, whereas old information is deac-cented.Liandcolleagues found that inChinese theN400tonew,accentedinformationwas largerthantheN400tonew,deaccentedinformation,despitethefactthattheaccen-tuationwascontextuallyappropriate,whereas theabsenceofanaccentwasnot.Theauthorsargue that this result isbestexplainedby therecruitmentofadditionalunificationresourcesforinformationthatismarkedasmoresalientbyaccentuation.

OnewaytoreconcilethesedifferentaccountsoftheN400isbyreferencetodifferentrolesfortheleftandrighthemi-spheres (Kutas & Federmeier, 2000; Federmeier, 2007).FedermeierandKutas(1999)didavisual-half-fieldstudyinwhichparticipantsreadsentences suchas“EverymorningJohn makes himself a glass of freshly squeezed juice. Hekeepshisrefrigeratorstockedwith(oranges/apples/carrots).”In this context, “oranges” is the expected continuation,“apples”isaviolationbutwithinthecorrectsemanticcate-gory,and“carrots” isaviolation thatcrosses thecategoryboundary.Theleft-visual-field/right-hemisphere(LVF/RH)results showed a smaller N400 to oranges than to bothwithin-andacross-categoryviolations,butnoN400differ-enceforthetwotypesofviolation.Incontrast,fortheRVF/LHareducedN400wasobtainednotonlyforthepredictedword (“orange”), but also in part for the within-categoryviolation(“apple”)(seefigure56.1).Thislatterresultcanbeexplained as a consequence of a contextual prediction forthe targetconcept.Owing to theorganizationof semanticmemory, thewithin-categorynontarget (“apple”)getsacti-vatedtosomedegreeaswell,resultinginapartiallyreducedN400. Predictive semantic processing might thus be aleft-hemisphere processing mechanism, while the right-hemispherecontributionispresumablystrictlypostlexicalinnature, only contributing to the integration of the wordmeaningfromalexicalitemthatreceivedbottom-upsupportonthebasisofvisualoracousticinput.

3

2


Y


Figure56.1 Participantsreadthesentencesasintheexampleina visual-half-field presentation design. Context words were pre-sentedatcentralfixation,whereassentence-finaltargetwords(e.g.,“oranges”)werepresentedtotheleftorrightoffixation.Asillus-trated,wordspresentedtotheleftvisualfield(LVF)travelinitiallytotherighthemisphere(RH)andviceversa.ERPsareshownherefromarepresentative (rightmedialcentral)siteas indicated.Theresponse to target words presented to the RVF (left hemisphere)(shownon right), yielded the samepatternas thatobservedwithcentral fixation: expected exemplars (solid line) elicited smallerN400s than did violations of either type, but within-categoryviolations (dashed line)alsoelicitedsmallerN400s thanbetween-

category violations (dotted line). This pattern is indicative of a“predictive” strategy, in which semantic information associatedwiththeexpecteditemispreactivatedinthecourseofprocessingthecontextinformation.TheresponsetotargetspresentedtotheLVF/RH (shown on left), however, was qualitatively different:expected exemplars again elicited smallerN400s thanviolations,buttheresponsetothetwotypesofviolationsdidnotdiffer.Thispattern is more consistent with a plausibility-based integrativestrategy.Takentogether,theresultsindicatethatthehemispheresdifferinhowtheyusecontexttoprocesssemanticinformationinonlinelanguageprocessing.(Kutas&Federmeier,2000;reprintedwithpermission.)


Y

4 language

In recent years, the N400 and other language-relevantERPeffectshavebeenexploitedtotestmorespecificideasaboutthefunctionalcharacteristicsofsemanticunification.These include the contribution of world knowledge, theprocessing of silent meaning, the integration of pragmaticinformation, and the syntax-semantics interface. We willdiscussbrieflyeachofthesetheory-drivenissues.

WorldKnowledge AtleastsinceFrege(1892;seeSeuren,1998), theoriesofmeaningmakeadistinctionbetweenthesemanticsofanexpressionanditstruth-valueinrelationtoourmentalrepresentationofthestateofaffairsintheworld(Jackendoff,2002).Forinstance,thesentence“BillClintonisthe43rdpresidentoftheUSA”hasacoherentsemanticinterpretation,butcontainsapropositionthatisfalseinthelight of our knowledge that George W. Bush is the 43rdpresident. The situation is different for the sentence “Thepresidential helicopter is divorced.” Under defaultinterpretation conditions, this sentence has no coherentsemantic interpretation, since the predicate “is divorced”requiresananimateargument.Thedifferencebetweenthesetwo sentences points to the distinction that can be madebetween facts of the world (“world knowledge”) and factsof the words of our language, including their meaning(“linguistic knowledge”). Hagoort, Hald, Bastiaansen, andPetersson (2004)performedacombinedEEG/fMRIstudythatcompared theunificationof linguisticknowledgewiththe unification of world knowledge. While participants’brainactivitywasrecorded,theyreadoneofthreeversionsofasentencesuchas“TheDutchtrainsareyellow/white/sourandverycrowded”(criticalwordsareinitalics).Itisawell-known fact among Dutch people that Dutch trains areyellow, and therefore the first version of this sentence iscorrectlyunderstoodastrue.However,thelinguisticmeaningof the alternative color term white applies equally well totrains as the predicate yellow. It is world knowledge abouttrains in Holland that makes the second version of thissentencefalse.Thisisdifferentforthethirdversion,where(understandardinterpretationconditions)thecoresemanticfeaturesofthepredicatesourdonotfitthesemanticfeaturesofitsargumenttrains.

Figure56.2presentsanoverviewoftheresults.Asexpected,theclassicN400effectwasobtainedforthesemanticviola-tions.Fortheworld-knowledgeviolations,aclearN400effectwasobservedaswell.Crucially, thiseffectwas identical inonsetandpeak latency,andverysimilar inamplitudeandtopographicdistribution to the semanticN400effect.Thisfinding is strong empirical evidence that lexical-semanticknowledgeandgeneralworldknowledgearebothintegratedinthesametimeframeduringsentenceinterpretation.Theresultsofthisworld-knowledgeexperimentprovidefurtherevidenceagainstanaccountofunificationinwhichfirstthemeaning of a sentence is determined, and only then is its

meaningverifiedinrelationtoourknowledgeoftheworld.Semanticinterpretationisnotseparatefromitsintegrationwithnonlinguisticconceptualknowledge.

Further evidence in favor of an enriched compositionaccountcomesfromastudyontheintegrationofinforma-tionaboutthespeaker.Ininterpretingaspeaker’sutterance,wetakenotonlytheprecedingutterancesintoconsideration,but also our knowledge of the speaker. For instance, wemight find it odd for a man, but not for a woman of acertainage,tosay,“IthinkIampregnant.”Atsomepointduring languagecomprehension, the listenercombines theinformationthatisrepresentedinthecontentofasentencewiththeinformationshehasaboutthespeaker.Theques-tionis,Whenexactlydoesthepragmaticinformationaboutthespeakerhaveitsimpactontheunfoldinginterpretationof the utterance? This question was answered in a recentERP studybyVanBerkum,VandenBrink,Tesink,Kos,and Hagoort (2008). Participants listened to sentences,some of which contained a specific word at which themessage content became at odds with inferences aboutthespeaker’ssex,age,andsocialstatus,asinferredfromthespeaker’svoice.

Ifvoice-basedinferencesaboutthespeakerarerecruitedby the same early unification process that combines wordmeanings,thenspeakerinconsistenciesandsemanticanom-alies should elicit the same N400 effect. This was indeedobserved. Reliable effects of speaker inconsistency werealreadyfoundinthe200–300-mslatencyrangeafterwordonset.Thesamelatencyeffectwasobtainedforthestraight-forwardsemanticanomalies.Thesefindingsthereforedem-onstratethatsensemakingdependsonthepragmaticsofthecommunicativesituationrightfromthestart.

As for compositionality, the results of the studies justreviewedmaymean two things,dependingonone’sviewson the lexicon.Onepossibility is that the lexicon includesdeclarativememoryinitsentirety,andthensimplecomposi-tion seems enough to account for the similarity betweentheN400effects.Alternatively,thelexiconincludesinvari-ant (i.e., linguistic) meanings only, and then enrichedcomposition—the thesis that the lexicon is not the onlysourceofsemanticcontent—seemsnecessarytoexplaintheobservedN400effects(Baggioetal.,inpress).

EventKnowledgeandDiscourseModels Unificationoflexical representations ultimately results in a discoursemodel—that is, a representation making what is given asinput true whenever possible (recall the Dutch trainsexamples).Eventsofferavantagepointforinvestigatingtheproperties of discourse models, because natural languageshaveverysophisticateddevicesforcharacterizingtimeandcausation.Oneofthesedevicesisaspect.Thisisthelinguisticmarkingoftheinternalprofileofevents.Ferretti,Kutas,andMcRae (2007) found that readers have least difficulty

4


Y


A

B

Figure 56.2 (A)GrandaverageERPs fora representativeelec-trodesite (Cz)forcorrectcondition(blackline),world-knowledgeviolation(bluedottedline),andsemanticviolation(reddashedline).ERPs are time locked to the presentation of the critical words(underlined).Spline-interpolatedisovoltagemapsdisplaythetopo-graphicdistributionsofthemeandifferencesfrom300to550msbetweensemanticviolationandcontrol (left),andbetweenworld

knowledgeviolationandcontrol(right).(B)Thecommonactivationfor semantic and world-knowledge violations compared to thecorrect condition, based on the results of a minimum-T-fieldconjunctionanalysis.Bothviolationsresultedinasinglecommonactivation (P = 0.043, corrected) in the left inferior frontal gyrus(inorinthevicinityofBrodmann’sarea45).Thecrosshairsindi-catethevoxelofmaximalactivation. 18

integratinglocativenounswhentheaspectofthemainverbisimperfectiveandthedenotedlocationisaprototypicalonegiventheverb’ssemantics.Insentenceswithanimperfective,such as “The diver was snorkeling in the ocean/pond,” alargerN400wasevokedbypond thanbyocean.ThisN400effectwas reduced if theaspectwasperfective,as in“Thediver had snorkeled in the ocean/pond.” Describing aneventasongoingusingtheimperfectiveaspectleadsreaders

toconstructasituationmodelinwhichlocationsandotherdimensions of the action become relevant, while suchdimensionsareignorediftheactionisviewedperfectively.

The imperfective leads also to expectations concerningtheoutcomeoftheeventdescribed.Baggio,vanLambalgen,andHagoort (2008) investigatedwhether, insentences like“Thegirlwaswritingaletterwhenherfriendspilledcoffeeon the tablecloth/paper,” thegoal state (acomplete letter)


Y

6 language

wasrepresentedonlineduringtheunificationprocess.Ifthegoalispredictedtooccurwhenevertheimperfectiveisused,adifferenceshouldbeobservedatthewordpapercomparedtotablecloth.Spillingcoffeeonthepaperimpliesthatthegoalstatewas not attained, and forces the system to revise theearliercommitmenttotheevent’scompletion(Baggio&vanLambalgen,2007).Spillingcoffeeonthetablecloth,however,doesnothavethis implication.Paperdid indeedresult inalargersustainedanteriornegativity(SAN)comparedtotable-cloth,andtheeffectwascorrelatedwiththefrequencywithwhichparticipants concluded that theeventwasnot com-pleted (see figure 56.3). These results again suggest thatsemantic processing is not bound to asserted content, butcan include inferencesanticipating theoutcomeofactionsand events, as well as other inferences invalidating previ-ously drawn conclusions. In this sense, unification can bedescribedasadefeasibleprocess:discoursemodelsbuiltupincrementallyatanyonestagemayhavetoberevisedwhenadditionalinformationbecomesavailable,aswhenthewordpaperisencounteredinthisexample(cf.Carreiras,Garnham,Oakhill,&Cain,1996;Sturt,2007).

Fictional Discourse and Silent Meaning Simplecompositionimpliesthatunificationpreservesthesemanticidentityoftheconstituentexpressions.However,experimentalresearchsuggeststhatdiscoursemayoverrideevensuchcorefeaturesofwordsemanticsasanimacy.NieuwlandandVanBerkum (2006) showed that sentences that make sense ontheirown,like“Thepeanutwassalted,”appearanomalousif theyareembedded inacontext inwhich the inanimatesubject (the peanut) is attributed animate features. In anarrativeinwhichthepeanutdancedandsang,becauseitfellinlovewithanalmondithadmet,thefinalwordin“Thepeanutwas salted”resulted ina largerN400compared to“The peanut was in love” (see figure 56.4). This result istakentoshowthatdiscoursecanoverrideseeminglycontext-invariantsemanticfeaturesofwords.

Another interesting phenomenon is that of silentmeaning—thatis,meaningnotexpressedinthesyntaxandphonologyofanexpression.Anumberoflinguisticdevicesare available to speakers and hearers that allow efficientcommunication of meaning beyond what is explicitlyasserted.Amongthesearecoercingexpressions,functioningasa shorthand for lengthierdefinitedescriptions,as in theclassic examples “The ham sandwich in the corner wantssomemorecoffee,”whereham sandwichinfactreferstothepersonwhoorderedone,and“Platoisonthetopshelfnextto Russell,” where Plato and Russell refer to copies of theworksofthetwophilosophers.Moreextremeformsofcoer-cionarepossible,asin“Fishingtheedgesdry,”wheredryisa condensed expression for thephrase using a dry fly, or inresultative constructions like “Hammering the metal flat,”whereflatdenotesthefinalstateofthemetalafterhammer-

ing. What all these widely used expression types have incommonisasilentsemanticelement,whichhastoberecov-ered (sometimes obligatorily) to make full sense of thesentence.Semanticprocessingmightbe taxedduringsuchrecovery process, and that is indeed what was foundexperimentally. Complement-coercing sentences like “Thejournalistbeganthearticle,”whichpresumablymeansthatshe began writing or typing the article, are more difficultto process than sentences in which the activity is part oftheassertedcontentlike“Thejournalistwrotethearticle.”The processing costs of complement coercion havebeen established using reading times (McElree, Traxler,Pickering, Seely, & Jackendoff, 2001), eye tracking(Traxler, Pickering, & McElree, 2002; Traxler, McElree,Williams, & Pickering, 2005), and MEG (Pylkkänen &McElree, 2007). Pylkkänen and McElree found an MEGresponse that was located in ventromedial prefrontalcortex to coerced sentences, which was different from theM350, the magnetic correlate of the N400. Semanticprocessing beyond the single-word level is therefore notrestricted to processing asserted content as delivered bythe input, but is crucially engaged in recovering silentmeaninginpresuppositions,implicatures,coercions,andsoon.Crucially,recoveredmeaningsaretriggeredbyexpres-sionsthataregivenasinputbutarethemselvesphonologi-cally and syntactically silent, an effect which shows thatsemantics is relatively independent from the two othercomponents of the language system. This “autonomy ofsemantics” isatoddswiththesyntax-semanticshomomor-phismpostulatedby formal semanticists (Montague,1970;Partee, Ter Meulen, & Wall, 1990), as well as with the“interface uniformity” upon which generative grammar isbuilt(Culicover&Jackendoff,2005).

Unification and the Syntax-Semantics Interface Alanguage-relevant ERP effect that has been related tosyntacticprocessingisapositivity,nowadaysreferredtoasP600 or as P600/SPS (Coulson, King, & Kutas, 1998;Hagoort,Brown,&Osterhout,1999;Osterhout,McLaughlin,&Bersick,1997).TheP600isthesyntacticequivalentoftheN400 effect. One of the antecedent conditions of P600effects is aviolationofa syntactic constraint.The relationbetweenN400andtheP600effectsmightprovideinsightsintotheinterplaybetweensemanticandsyntacticunification.Modulations of the P600 have been observed not only tosyntactic violations, syntactic ambiguities, and syntacticcomplexity,butalsotobreakdownsofnormaloperationsatthesyntax-semanticsinterface(forareview,seeKuperberg,2007). For example, Kim and Osterhout (2005) reportedlarger P600s evoked by devouring in “The hearty meal wasdevouring...,”comparedtoeither“Theheartymealwasdevoured...” or “The hungry boys were devouring...”;this despite the fact that the sentence is syntactically well

5


Y


Figure 56.3 (A) Grand-average topographies displaying themean amplitude difference between the ERPs evoked by thesentence-finalverbwhenitterminatedversuswhenitdidnotter-minate theaccomplishments in theprogressive.Circles representelectrodesinasignificant(P<0.05)cluster.(B)Grand-averageERPwaveformsfromarepresentativesite(F3)time-lockedtotheonset(0ms)oftheverbinterminatedversusnonterminatedaccomplish-ments.Negativevaluesareplottedupward. (C )Scatterplotdis-playing the correlation between the amplitude of the sustained

anteriornegativityelicitedbyterminatedaccomplishmentsandthefrequencyofnegativeresponsesinabutton-press,probe-selectiontask(r=−0.415,T(22)=−2.140,P=0.043).Themeandifferenceof negative responses between terminated and nonterminatedaccomplishments isplottedontheabscissa.Themeanamplitudedifferenceatfrontopolarandfrontalelectrodesbetweenterminatedand nonterminated accomplishments in the 500–700-ms intervalfollowing the onset of the sentence-final verb is plotted on theordinate.

A

B C


Y

8 language

formed (see figure 56.5). The semantics of meal and devoursuggestaplausiblethematicroleassignmenttomeal:athemeinsteadofanagentasthesyntaximplies.Inthiscase,semanticplausibility overrides syntactic constraints, and the verbdevouring is presumably perceived as a morphosyntacticviolation indexedbytheP600.Conflictsbetweensyntacticand semantic constraints might result in N400 or P600effectsdependingonwhether,respectively,thesemanticorthesyntacticconstraintsaretheweakest.Incaseswheretheinput isanomalousbecauseofaconflictbetweensemanticandsyntacticcues,themodus operandiofthesystemseemstoobey a “loser takes all” principle. That is, if the semanticcues are stronger than the syntactic cues, the effect willappearatthelevelofsyntacticunification(P600).Kuperberg(2007) argues that there are at least two neural routessubservinglanguagecomprehension:(1)asemantic,memory-basedstreamthatprovideselementarymeaningsaswellasconceptual, categorical, and thematic relations betweenthem; (2) a combinatorial stream that provides analysesbasedonmorphosyntacticconstraintsandthematicrolesasgivenintheinput.TheP600reportedbyKimandOsterhout(2005),forexample,mightbetakentosuggestthatsemanticassociationsbetweenwordsare the strongest constraints—forinstance,becauseinthiscasetheyaretakenintoaccountearlierthanthesyntacticcues.

Conclusion In general, ERP research on semanticprocessing has found that word meaning is very rapidlyassembledintocompoundmeaning.Thisstatementholdsforindividual word meanings in the context of single words,sentences,ordiscourse.Butitalsoholdsformeaningthatisextracted from pictures, co-speech gestures, or stereotypesinferredfromspeakercharacteristics (Willems,Özyürek,&Hagoort,2007,2008;VanBerkumetal.,2008).Theeffectsofsemanticprocessingaremostoftenobservedasmodulationsof theN40amplitude.Thetopographicdistributionof theN400differsslightly fordifferentstimulustypes.It ismoreevenly distributed for auditory than for the visual N400.Picturesandco-speechgestureselicitamore frontalN400than sentences without concomitant nonlinguistic infor-mation.Thisfindingsuggeststhatthesetofneuralgeneratorscontributing to the scalp-recorded N400 is not fullyoverlapping for the different types of meaningful stimuli.ThisresultisconsistentwiththeresultsfromfMRIstudies,showing both overlapping and distinct activations inconnectiontothevarioustypesofmeaningfulinput(seethenext section). Intracranial recordings and MEG studiesindicatethatthescalp-recordedN400iscausedbycoordinatedactivity inanumberofdifferentbrainareas, includingtheanteriorinferotemporalcortex(McCarthy,Nobre,Bentin,&Spencer, 1995), the superior temporal cortex (Dale et al.,

Figure56.4 N400effectstriggeredbyacorrectpredicate(salted)thatis,however,contextuallydisfavoredincomparisontoanincor-rectpredicate(in love).Waveformsarepresentedforrepresentative

electrodesites,time-lockedtotheonsetofthecritical inanimate/animatepredicate in thefifth sentence. (AfterNieuwland&VanBerkum,2006.)

6


Y


2000; Helenius, Salmelin, Service, & Connolly, 1998;Halgren et al., 2002), and the left inferior frontal cortex(Halgrenetal.,1994,2002;Guillem,Rougier,&Claverie,1999).OtherERPeffects(e.g.,anteriornegativities)havealsobeenobservedtoaspectsofpostlexicalsemanticprocessing.How they differ from the N400 effects in their functionalcharacterizationisanissueforfurtherresearch.

The semantic unification network

InrecentyearsaseriesoffMRIstudieswereaimedatiden-tifyingthesemanticunificationnetwork.Thesestudieseithercomparedsentencescontainingsemantic/pragmaticanom-alieswith their correct counterparts (Hagoort et al., 2004;Newman, Pancheva, Ozawa, Neville, & Ullman, 2001;Kuperberg et al., 2000, 2003; Kuperberg, Sitnikova, &Lakshmanan,2008;Nietal.,2000;Baumgaertner,Weiller,&Buchel,2002;Kiehl,Laurens,&Liddle,2002;Friederici,Ruschemeyer, Hahne, & Fiebach, 2003; Ruschemeyer,Zysset,&Friederici,2006)orcomparedsentenceswithandwithout semantic ambiguities (Hoenig & Scheef, 2005;Rodd,Davis,&Johnsrude,2005;Zempleni,Renken,Hoeks,Hoogduin, & Stowe, 2007; Davis et al., 2007). The most

consistentfindingacrossallthesestudiesistheactivationoftheleftinferiorfrontalcortex(LIFC),moreparticularlyBA47 and BA 45. In addition, the left superior and middletemporalcortexisoftenfoundtobeactivated(seefigure56.6foranoverview),aswellasleftinferiorparietalcortex.Forinstance,RoddandcolleagueshadsubjectslistentoEnglishsentencessuchas“Thereweredatesandpearsinthefruitbowl”andcomparedtheBOLDresponseofthesesentencestotheBOLDresponseofsentencessuchas“Therewasbeerand cider on the kitchen shelf.” The crucial differencebetween these sentences is that the former contains twohomophones—“dates” and “pears”—which, when pre-sentedauditorily,havemorethanonemeaning.Thisisnotthecaseforthewordsinthesecondsentence.Thesentenceswith the lexical ambiguities led to increased activations inLIFC and in the left posterior middle/inferior temporalgyrus. In this experiment all materials were well-formedEnglishsentencesinwhichtheambiguityusuallygoesunno-ticed. Nevertheless, the results were very similar to thoseobtained in experiments that used semantic anomalies.Areas involved in semantic unification were found to besensitive to the increase in semantic unification load thatresultedfromtheambiguouswords.

Figure 56.5 At the interface between syntax and semantics.Grand-averageERPsrecordedatthreemidlinesitesandsixmedial-lateral sites. All sentences are syntactically correct. (A) ERPs topassivecontrolverbs(solidline)andthematicviolationverbs(dashedline).(B)ERPstoactivecontrolverbs(solidline)andthematicviola-

tionverbs (dashed line). Inboth cases the inconsistencybetweengrammaticalrolesandthematicrolebiasesresultedinrobustP600effects.Onsetofthecriticalverbsisindicatedbytheverticalbar.Eachhashmarkrepresents100ms.Positivevoltageisplotteddown.(Kim&Osterhout,2005;reprintedwithpermission.)

7


Y

10 language

Inshort,thesemanticunificationnetworkseemstoincludeat least LIFC, left superior/middle temporal cortex, andthe(left)inferiorparietalcortex.Tosomedegree,therighthemispherehomologuesoftheseareasarealsofoundtobeactivated (seefigure56.6). In the following subsectionswewill discuss the possible contributions of these regions tosemanticunification.

The Multimodal Nature of Semantic UnificationAn indication for the respective functional rolesof the leftfrontalandtemporalcorticesinsemanticunificationcomesfrom a few studies investigating semantic unification ofmultimodal information with language. Using fMRI,Willems and colleagues assessed the neural integration ofsemantic information from spoken words and from co-speechgesturesintoaprecedingsentencecontext(Willemset al., 2007).Spoken sentenceswerepresented inwhichacritical word was accompanied by a co-speech gesture.Either the word or the gesture could be semanticallyincongruouswithrespect to theprevioussentencecontext.BothanincongruouswordandanincongruousgestureledtoincreasedactivationinLIFCascomparedtocongruouswordsandgestures (seeWillems et al., 2008, for a similarfindingwithpicturesofobjects).Interestingly,theactivationof the leftposteriorSTSwas increasedbyan incongruousspokenwordbutnotbyanincongruoushandgesture.Thelatterresultedinaspecificincreaseindorsalpremotorcortex

(Willemsetal.,2007).Thisfindingsuggests thatactivationincreasesinleftposteriortemporalcortexaretriggeredmoststrongly by processes involving the retrieval of lexical-semanticinformation.LIFC,however,isakeynodeinthesemanticunificationnetwork,unifyingsemanticinformationfromdifferentmodalities.

Fromthesefindings it seemsthatsemanticunificationisrealized inadynamic interplaybetweenLIFCas amulti-modal unification site on the one hand, and modality-specificareasontheotherhand.

Semantic Unification Beyond the Sentence LevelRecentlyafewstudieshavesetouttoinvestigatetheneuralnetworks involved in semantic processing at the level ofmultisentence utterances, such as short stories. Besidesthenetworkthatisalsoactivatedtosemanticunificationatthesentencelevel,storycomprehensioninvolvesactivationof dorsomedial prefrontal cortex and, presumably, rightinferior frontal cortex. In a recent meta-analysis, Ferstland colleagues report the consistent involvement ofmedialprefrontalcortex, leftSTS/MTG,andLIFCwhenparticipantsprocesscoherenttextascomparedtosentencesthatdonotformacoherentstoryorascomparedtowordlists (Ferstl,Neumann,Bogler,&vonCramon,2008).Inavariant of this line of research, Kuperberg, Lakshmanan,Caplan, and Holcomb (2006) presented participantswith sentence quartets in which the relation of the last

Figure56.6 Overviewoflocalmaximaininferiorfrontalcortexand in temporal cortex in neuroimaging studies employing sen-tenceswithsemanticanomaliesorsemanticambiguities.Thelocalmaxima(inMNIspace)ofeachstudywereoverlaidonarenderingof a brain in MNI space. For local maxima see tables 56.1 and56.2; fora summaryof theresults see table56.3.Renderingwas

madeusingMRIcroN.PleasenotethatthelocalmaximaoftheNiand colleagues (2000) and the Kuperberg and colleagues (2003)studiesaredisplayed,butthatthesearenotbasedoncoordinates,sincenocoordinateswereprovided.Thelocalmaximaaredrawnbyhandbaseduponthefiguresintherespectivepapers.

8

9


Y


Table56.1Involvement of the inferior frontal cortex in fMRI studies of sentence comprehension employing semantic anomalies or semantic ambiguities.

The table shows the studies that were used for the overview in figure 56.6, a brief description of the contrast that was employed ineachofthestudies,thereportedcoordinatesofthelocalmaximaininferiorfrontalcortexinMNIspace,andaverbaldescriptionofthelocationofthelocalmaxima.Whennecessary,TalairachcoordinateswereconvertedtoMNIspaceusingthetransformationsuggestedbyBrett (http://imaging.mrc-cbu.cam.ac.uk/imaging/MniTalairach).Note that in computing themean coordinates thefindings fromKuperbergandcolleagues(2003)andNiandcolleagues(2000)werenottakenintoconsideration,sincenocoordinateswerereportedinthesestudies.

20

Study Comparison Coordinatesxyz(MNI)

Region

Baumgaertneretal.,2002 Sem.incongruent>congruent −5136−6 LeftIFG

Davisetal.,2007 Highambiguity>lowambiguity −402418 LeftIFG

−48634

−401824

463618 RightIFG

Friedericietal.,2003 Sem.incongruent>congruent Noactivation —

Hagoortetal.,2004 Sem.incongruent>congruent∩Worldknowledgeincongruent>congruent

−44308 LeftIFG

Hoenig&Scheef,2005 Sem.incongruent>congruent −5018−14 LeftIFG

−504311

Kiehletal.,2002 Sem.incongruent>congruent −48324 LeftIFG/ant.temporal

3632−16 RightIFG/ant.temporal

Kuperbergetal.,2000 Sem.incongruent>congruent Noactivation —

Pragm.incongruent>congruent Noactivation —

Kuperbergetal.,2003 Pragm.incongruent>congruent (Nocoordinates) LeftIFG

Kuperbergetal.,2008 Pragmaticincongruent>congruent −4325−10 LeftIFG

Sem.incongruent>congruent −49410 LeftIFG

29195 RightIFG

Newmanetal.,2001 Sem.incongruent>congruent −50345 LeftIFG

Nietal.,2000 Sem.incongruencedetection>tonepitchdiscrimination

(Nocoordinates) LeftIFG

RightIFG

Oddballparadigmwithsemanticallyincongruentsentences

(Nocoordinates) LeftIFG

(Nocoordinates) RightIFG

Roddetal.,2005 Highambiguity>lowambiguity −503020 LeftIFG

−561622 LeftIFG

−421432 LeftIFS

36264 RightIFG

503616 RightIFG

Rueschemeyeretal.,2006 Sem.incongruent>synt.incongruent −503015 Leftant.IFG

Willemsetal.,2007 Sem.incongruent>congruent −431127 LeftIFS

Willemsetal.,2008 Sem.incongruent>congruent −451427 LeftIFS

Zemplenietal.,2007 Subordinatemeaning>dominantmeaning −482620 LeftIFG

−521626 LeftIFG3420−10 RightIFG


Y

12 language

Table56.2Involvement of the temporal cortex in fMRI studies of sentence comprehension employing semantic anomalies or semantic ambiguities.

The table shows the studies that were used for the overview in figure 56.6, a brief description of the contrast that was employed ineachofthestudies,thereportedcoordinatesofthelocalmaximaintemporalcortexinMNIspace,andaverbaldescriptionofthelocationof the local maxima. When necessary, Talairach coordinates were converted to MNI space using the transformation suggested byBrett (http://imaging.mrc-cbu.cam.ac.uk/imaging/MniTalairach). Note that in computing the mean coordinates the findings fromKuperbergandcolleagues(2003)andNiandcolleagues(2000)werenottakenintoconsideration,sincenocoordinateswerereportedinthesestudies.Study Comparison Coordinates

xyz(MNI)Region

Baumgaertneretal.,2002 Sem.incongruent>congruent Noactivation —

Davisetal.,2007 Highambiguity>lowambiguity −50−44−12 LeftITG

−54−60−2

Friedericietal.,2003 Sem.incongruent>congruent −60−4220 LeftSTG

63−4020 RightSTG

58−2413 RightSTG

Hagoortetal.,2004 Sem.incongruent>congruent Noactivation —

Hoenig&Scheef,2005 Sem.incongruent>congruent Noactivation —

Kiehletal.,2002 Sem.incongruent>congruent Noactivation —

Kuperbergetal.,2000 Sem.incongruent>congruent 43−11−7 RightMTG

49−174 RightSTG

Pragm.incongruent>congruent −49−319 LeftSTG

Kuperbergetal.,2003 Pragm.incongruent>congruent (Nocoordinates) LeftSTS

Kuperbergetal.,2008 Pragmaticviolations>correctsentences −27−28−19 Leftant.med.temporalcortex

Sem.incongruent>congruent −53−20−1 LeftSTG

58−193 RightSTG

Newmanetal.,2001 Sem.incongruent>congruent 70−36−15 RightMTG

Nietal.,2000 Sem.incongruencedetection>tonepitchdiscrimination

(Nocoordinates) LeftSTG/MTG

(Nocoordinates) RightSTG/MTG

Oddballparadigmwithsemanticallyincongruentsentences

(Nocoordinates) LeftpSTG

Roddetal.,2005 Highambiguity>lowambiguity −52−50−10 LeftpITG

−58−8−6 LeftSTG

Rueschemeyeretal.,2006 Sem.incongruent>synt.incongruent — —

Willemsetal.,2007 Sem.incongruent>congruent −53−522 LeftSTS

Willemsetal.,2008 Sem.incongruent>congruent −53−35−3 LeftSTS

Zemplenietal.,2007 Subordinatemeaning>dominantmeaning −50−48−12 LeftITG/MTG56−34−16 RightITG/MTG


Y


sentence to the previous story context was manipulated.Thelessrelatedsentencesrequiredanextracausalinferenceinorder tomake senseof the story. Itwas found that lessrelated sentences (which evoked more inferencing) led tostronger activations in left and right IFC, left MTG, leftmiddle fontalgyrus,andbilateralmedialprefrontal cortex(Kuperbergetal.;seeHasson,Nusbaum,&Small,2007,fora related result).These andother studies (e.g., StGeorge,Kutas, Martinez, & Sereno, 1999; Xu, Kemeny, Park,Frattali,&Braun,2005;Sieborger,Ferstl,&vonCramon,2007)suggestthatLIFCandleftsuperior/middletemporalcortex are also important for unification of informationbeyondthesentence level. It is interesting tonote that themedial prefrontal cortex, which is found activated fordiscourse but not for sentence-level processing, has beenimplicated in so-called mentalizing tasks, requiring theobserver to take theperspectiveof someoneelse (Buckner,Andrews-Hanna, & Schacter, 2008; Frith & Frith, 2006).According toMasonand Just, thisdomain-general area isrecruitedindiscourseprocessingforthesakeofinterpretingaprotagonist’soragent’sperspective(Mason&Just,2006).Inaddition,right-hemisphereregionsaresometimesbutnotconsistentlyreportedinthecontextofdiscourseprocessing(Maguire, Frith, & Morris, 1999; St George et al.; Ferstletal.;Martin-Loechesetal.,2008;seeFerstletal.;Mason&Just,2006,forextensivereviews).Somestudiesfindthatthetemporalpolesmayberelatedtosuccessfulintegrationduringstorycomprehension(Fletcheretal.,1995;Maguireetal.).The studies that report theseactivationsaremostlydone using PET. It is hard to assess the consistencyof temporal pole activation during story/text comprehen-sion because of the susceptibility to artifacts that theseregionsoftensufferfrominfMRIstudies(butseeXuetal.;Ferstletal.).

Controlled Processing and Selection Accounts forLIFC Although LIFC (including Broca’s area) hastraditionallybeenconstruedasa languagearea, there is awealth of recent neuroimaging data suggesting that itsroleextendsbeyondthelanguagedomain.SeveralauthorshavethereforearguedthatLIFCfunctionisbestcharacter-ized as “controlled retrieval” or “(semantic) selection”(Thompson-Schill, D’Esposito, Aguirre, & Farah, 1997;Wagner, Pare-Blagoev, Clark, & Poldrack, 2001; Badre,Poldrack, Pare-Blagoev, Insler, & Wagner, 2005; Gold,Balota, Kirchoff, & Buckner, 2005; Moss et al., 2005;Thompson-Schill,Bedny,&Goldberg,2005).Forinstance,Thompson-Schill and colleagues showed that LIFC wasmorestronglyactivatedinaverb-generationtaskwhenthenounthatservedasthecueallowedformanydifferentverbresponses,asopposed tonouns thatare reliably related toonlyoneora fewverbs (Thompson-Schilletal.,1997). Inresponse to the noun cue “scissors,” for example, mostparticipants generate the verb “to cut,” whereas the noun“wheel”triggersamorediversesetofresponses.Onthebasisoftheseandotherfindings,itwasarguedthatLIFCguidessemantic selection among competing alternatives, withhigheractivationwhentherearemorecompetitors.

Howdoes theselectionaccountofLIFCfunctionrelateto the unification account? As is discussed in more detailelsewhere, unification often implies selection (Hagoort,2005). For instance, in the study by Rodd and colleaguesdescribedearlier,increasedactivationinLIFCismostlikelyduetoincreasedselectiondemandsinreactiontosentenceswithambiguouswords.Selectionisoften,butnotalways,aprerequisteforunification.Unificationwithorwithoutselec-tionisacorefeatureoflanguageprocessing.Duringnaturallanguage comprehension, information has to be kept inworkingmemoryforacertainperiodoftime,andincoming

Table56.3Summary of the activations in the studies used for the overview in figure 56.6.

Thecoordinatesfromtables56.1and56.2wereused.Table56.3specifiesthemeancoordinatesforleftandrightinferiorfrontalandtemporalcortices,thestandarddeviationinthex,y,andzdirectionsinmilli-meters,themeanEuclidiandistanceofthelocalmaximatothemeancoordinates,thenumberofmaximathatwerereported,andthenumberofstudiesthatreportmaximainthatregion.Notethatthenumberofmaximaishigherthanthenumberofstudies,sinceseveralstudiesreportmorethanonemaximum.NotethatthefindingsfromKuperbergandcolleagues(2003)andNiandcolleagues(2000)werenotusedincomputingthemeancoordinates,sincenocoordinateswerereportedinthesestudies.

10

Mean(xyz)(MNI)

SD(xyz)(mm)

MeanDistancetoMean(mm)

NumberofStudies(outoftotal)

Inferior frontal cortex

Left −472214 4.310.613.9 16.3 14/16

Right 3928 3 7.97.713.6 15.0 6/16

Temporal cortex

Left −51−38−3 8.615.410.9 18.0 10/16 Right 57−26 0 8.810.913.7 17.2 6/16


Y

14 language

informationhastobeintegratedandcombinedwithprevi-ous information. The combinatorial nature of languagenecessitates that a representation be constructed online,withouttheavailabilityofanexistingrepresentationoftheutteranceinlong-termmemory.Inaddition,someinforma-tionsourcesthatareintegratedwithlanguagedonothaveastablerepresentationinlong-termmemorysuchthattheycanbeselected.Forinstance,thereisnostablerepresenta-tionofthemeaningofco-speechgestures,whicharehighlyambiguousoutsideofa languagecontext.Still, inall thesecasesincreasedactivationisobservedinLIFC,suchaswhentheintegrationloadofinformationfromco-speechgesturesis high (Willems et al., 2007). Similarly, it is unlikely thatintegration of information about characteristics of thespeaker as indicated by the acoustics of the voice (e.g.,whetherthespeakerismaleorfemale,childoradult)relieson selection. Nevertheless, increased activation levels areobserved inLIFCwhen integrating speaker characteristicswiththecontentof themessagegetsmoredifficult (Tesinket al., accepted). Therefore, unification is a more generalaccountofLIFCfunction.Itimpliesselection,butitcoversadditionalintegrationprocessesaswell.

IntegrationVersusUnification Wehavesofarusedtheterm “unification” to refer to the assembly of complexmeaning.Althoughtheterm“integration”isoftenusedasasynonymforunification,includingbyourselves,wesuggestthatitisusefultomakeafunctionaldistinctionbetweenthetwo.Semantic integration isat stake ifdifferent sourcesofinformationconvergeonacommonmemoryrepresentation.Anexampleisthesoundandthesightofananimal(e.g.,abarking dog). The sight of a dog, the barking sound, andtheircombinedoccurrencemostlikelyallactivateamemoryrepresentationof“dog”thathasmultimodalcharacteristics.Semantic unification, however, is always a constructiveprocess in which a semantic representation is constructedthat is not already available in memory. This distinctionmakesoppositepredictionsfortheBOLDresponse.Semanticunificationisalwaysharderforsemanticincongruities.TheseshouldresultinastrongerBOLDresponsethansemanticallycongruent items. In contrast, congruent input results inconverging support for a prestored representation, whichmight thus be more strongly activated compared to asituation with incongruent input. Hence, in the case ofintegration the congruent condition will elicit a strongerBOLD response than the incongruent condition. A fewstudies on multimodal integration have indeed reportedactivationincreasestomatchingstimuluscombinations.Forinstance, Van Atteveldt, Formisano, Goebel, and Blomert(2004) observed a higher activation level in left superiortemporal cortex in response to a matching phoneme andletter combination (e.g., letter “p” with phoneme [p]) ascompared to a mismatching combination (e.g., letter “k”

withphoneme[p])(seealsoCalvert,Campbell,&Brammer,2000, for the integration of lip movements and speechsounds).ThesameistrueinthestudybyBeauchamp,Lee,Argall,andMartin (2004),who foundhigheractivation inleftposteriortemporalcortextothematchingcombinationofapictureofanobjectanditssoundversusanincongruentcombination.InarecentpaperHeinandcolleagues(2007)reported an interesting difference between inferior frontalcortex(IFC)andposteriortemporalcortex.TheIFCshoweda stronger response to incongruent familiaranimal soundsand images (e.g., a meowing dog) than to the familiarcombination (a barking dog). This was, however, notobserved inSTGandpSTS.This regionwas found tobemore stronglyactivated tohighly familiar combinationsofobjectsandsoundsascomparedtocombinationsofartificialobjectsandsounds.This result suggestsapossibledivisionof labor between inferior frontal and superior temporalareas,withastrongercontributiontointegrationfortemporalcortexandastrongerrolefortheIFCinunification—thatis, in constructing a common representation that is notalreadyavailableinlong-termmemory.

However, as we have seen, many studies on sentenceprocessing have found increased activation, especially inleft superior/middle temporal cortex when the (semantic)unification load of a word increases given the precedingsentencecontext (e.g.,Bookheimer,2002;Friederici etal.,2003;Kuperbergetal.,2003;Hagoortetal.,2004;Roddetal., 2005; Ruschemeyer, Fiebach, Kempe, & Friederici,2005; Davis et al., 2007; Willems et al., 2007, 2008). WeproposethatthisresultsfromsignalsfromLIFC,indicatingthat in the serviceofunification, lexical-semantic informa-tion needs to be maintained active longer or needs to bereaccessedwhenunification loadincreases (cf.Humphries,Binder,Medler,&Liebenthal,2007). In thisway, it is thedynamic interplaybetweenLIFCand left superior/middletemporal cortex that is necessary for successful semanticunification.

Conclusion

Overandabovetheretrievalofindividualwordmeanings,sentence and discourse processing requires combinatorialoperationsthatresultinacoherentinterpretationofmulti-wordutterances.Theseoperationsdonotadheretoasimpleprinciple of compositionality. World knowledge, informa-tionabout the speaker, co-occurringvisual input, anddis-courseinformationalltriggerelectrophysiologicalresponsessimilar to those triggered by sentence-internal semanticinformation. A network of brain areas, including the leftinferior frontal gyrus, the left superior/middle temporalcortex,theleftinferiorparietalcortex,and,toalesserextent,theirright-hemispherehomologuesarerecruitedtoperformsemantic unification. In line with the MUC framework,

11


Y


semanticunificationoperationsareundertop-downcontrolofleft,andinthecaseofdiscourse,alsorightinferiorfrontalcortex. This contribution modulates activations of lexicalinformation inmemoryas representedby the left superiorand middle temporal cortex, presumably with additionalsupport for unification operations in left inferior parietalareas (e.g., angular gyrus). A more precise account of theindividualcontributionsof thesecorenodes in theunifica-tionnetworkawaitsfurtherresearch.

acknowledgments We thank Jos van Berkum, Karl-MagnusPetersson,andtheNCL-Ph.D.’sfortheircommentsonanearlierversionofthischapter.

REFERENCES

Badre,D.,Poldrack,R.A.,Pare-Blagoev,E.J.,Insler,R.Z.,& Wagner, A. D. (2005). Dissociable controlled retrieval andgeneralized selection mechanisms in ventrolateral prefrontalcortex.Neuron, 47,907–918.

Baggio,G.,&vanLambalgen,M.(2007).Theprocessingconse-quencesoftheimperfectiveparadox.J. Semantics, 24,307–330.

Baggio, G., van Lambalgen, M., & Hagoort, P. (2008).Computingandrecomputingdiscoursemodels:AnERPstudy.J. Mem. Lang., 59,36–53.

Baggio,G.,vanLambalgen,M.,&Hagoort,P. (inpress).Theprocessing consequences of compositionality. In W. Hinzen,E.Machery,&M.Werning(Eds.),The Oxford handbook of compo-sitionality.Oxford,UK:OxfordUniversityPress.

Baumgaertner, C., Weiller, C., & Buchel, C. (2002). Event-related fMRI reveals cortical sites involved in contextual sen-tenceintegration.Neuroimage, 16,736–745.

Beauchamp, M. S., Lee, K. E., Argall, B. D., & Martin,A.(2004).Integrationofauditoryandvisualinformationaboutobjectsinsuperiortemporalsulcus.Neuron, 41,809–823.

Bookheimer, S. (2002). Functional MRI of language: Newapproachestounderstandingthecorticalorganizationofseman-ticprocessing.Ann. Rev. Neurosci., 25,151–188.

Brown, C., & Hagoort, P. (1993). The processing nature oftheN400:Evidencefrommaskedpriming.J. Cogn. Neurosci.,5,34–44.

Buckner,R.L.,Andrews-Hanna,J.R.,&Schacter,D.L.(2008).Thebrain’sdefaultnetwork:Anatomy,function,andrelevancetodisease.Ann. NY Acad. Sci., 1124,1–38.

Calvert, G. A., Campbell, R., & Brammer, M. J. (2000).Evidence from functional magnetic resonance imaging ofcrossmodalbindinginthehumanheteromodalcortex.Curr. Bio.,10,649–657.

Carreiras,M.,Garnham,A.,Oakhill,J.,&Cain,K.(1996).Theuseofstereotypicalgenderinformationinconstructingamentalmodel:Evidence fromEnglishandSpanish.Q. J. Exp. Psychol. [A],49,639–663.

Coulson,S.,King,J.W.,&Kutas,M.(1998).Expecttheunex-pected:Event-relatedbrainresponsetomorphosyntacticviola-tions.Lang. Cogn. Process.,13,21–58.

Culicover, P., & Jackendoff, R. (2005). Simpler syntax. Oxford,UK:OxfordUniversityPress.

Cutler, A., & Clifton, C. E. (1999). Comprehending spokenlanguage: A blueprint of the listener. In C. M. Brown andP. Hagoort (Eds.), The neurocognition of language (pp. 123–166).Oxford,UK:OxfordUniversityPress.

Dale,A.M.,Liu,A.K.,Fischl,B.R.,Buckner,R.L.,Belliveau,J. W., Lewine, J. D., & Halgren, E. (2000). Dynamicstatistical parametric mapping: Combining fMRI and MEGfor high-resolution imaging of cortical activity. Neuron, 26,55–67.

Davis, M. H., Coleman, M. R., Absalom, A. R., Rodd, J. M.,Johnsrude,I.S.,Matta,B.F.,Owen,A.M.,&Menon,D.K.(2007). Dissociating speech perception and comprehension atreduced levels of awareness. Proc. Natl. Acad. Sci. USA, 104,16032–16037.

DeLong,K.A.,Urbach,T.P.,&Kutas,M.(2005).Probabilisticword pre-activation during language comprehension inferredfromelectricalbrainactivity.Nat. Neurosci., 8,1117–1121.

Federmeier, K. D. (2007). Thinking ahead: The role and rootsof prediction in language comprehension. Psychophysiology, 44,491–505.

Federmeier,K.D.,&Kutas,M.(1999).Arosebyanyothername:Long-term memory structure and sentence processing.J. Mem. Lang.,41,469–495.

Ferretti, T. R., Kutas, M., & McRae, K. (2007). Verb aspectandtheactivationofeventknowledge.J. Exp. Psychol. Learn. Mem. Cogn.,33,182–196.

Ferstl, E. C., Neumann, J., Bogler, C., & von Cramon,D.Y.(2008).Theextendedlanguagenetwork:Ameta-analysisof neuroimaging studies on text comprehension. Hum. Brain Mapp.,29,581–593.

Fletcher, P. C., Happe, F., Frith, U., Baker, S. C., Dolan,R.J.,Frackowiak,R.S.,&Frith,C.D.(1995).Othermindsinthebrain:Afunctionalimagingstudyof“theoryofmind”instorycomprehension.Cognition, 57,109–128.

Fodor,J.,&Lepore,E.(2002).The compositionality papers.Oxford,UK:OxfordUniversityPress.

Frege,G.(1892).UberSinnundBedeutung.Zeitschrift für Philoso-phie und philosophische Kritik, 100,25–50.

Friederici, A. D., Ruschemeyer, S. A., Hahne, A., & Fiebach,C.J.(2003).Theroleofleftinferiorfrontalandsuperiortempo-ralcortexinsentencecomprehension:Localizingsyntacticandsemanticprocesses.Cereb. Cortex, 13,170–177.

Frith,C.D.,&Frith,U.(2006).Theneuralbasisofmentalizing.Neuron, 50,531–534.

Gold,B.T.,Balota,D.A.,Kirchhoff,B.A.,&Buckner,R.L.(2005).Commonanddissociableactivationpatternsassociatedwithcontrolledsemanticandphonologicalprocessing:EvidencefromfMRIadaptation.Cereb. Cortex, 15,1438–1450.

Guillem,F.,Rougier,A.,&Claverie,B.(1999).Short-andlong-delay intracranial ERP repetition effects dissociate memorysystemsinthehumanbrain.J. Cogn. Neurosci.,11,437–458.

Hagoort,P.(2005).OnBroca,brain,andbinding:Anewframe-work.Trends Cogn. Sci.,9,416–423.

Hagoort, P., & Brown, C. (1994). Brain responses to lexicalambiguityresolutionandparsing.InC.Clifton,Jr.,L.Frazier,&K.Rayner (Eds.),Perspectives on sentence processing (pp.45–81).Hillsdale,NJ:LawrenceErlbaum.

Hagoort,P.,Brown,C.,&Osterhout,L.(1999).Theneurocog-nitionofsyntacticprocessing.InC.M.BrownandP.Hagoort(Eds.),The neurocognition of language (pp.273–317).Oxford,UK:OxfordUniversityPress.

Hagoort, P., Hald, L., Bastiaansen, M., & Petersson, K. M.(2004). Integration of word meaning and world knowledge inlanguagecomprehension.Science, 304,438–441.

Halgren,E.,Baudena,P.,Heit,G.,Clarke,J.M.,Marinkovic,K., & Chauvel, P. (1994). Spatio-temporal stages in faceand word processing. 2. Depthrecorded potentials in the

12

13


Y

16 language

human frontal and Rolandic cortices. J. Physiol. Paris, 88,51–80.

Halgren, E., Dhond, R. P., Christensen, N., Petten, C. v.,Marinkovic,K.,Lewine,J.D.,&Dale,A.M. (2002).N400-likemagnetoencephalographyresponsesmodulatedbysemanticcontext,wordfrequency,andlexicalclassinsentences.Neuroim-age, 17,1101–1116.

Hasson, U., Nusbaum, H. C., & Small, S. L. (2007). Brain net-workssubservingtheextractionofsentenceinformationanditsencodingtomemory.Cereb. Cortex, 17,2899–2913.

Heim,I.,&Kratzer,A.(1998).Semantics in generative grammar.NewYork:Blackwell.

Hein,G.,Doehrmann,O.,Muller,N.G.,Kaiser, J.,Muckli,L., & Naumer, M. J. (2007). Object familiarity and semanticcongruencymodulateresponses incorticalaudiovisual integra-tionareas.J. Neurosci.,27,7881–7887.

Helenius,P.,Salmelin,R.,Service,E.,&Connolly,J.F.(1998).Distincttimecoursesofwordandcontextcomprehensioninthelefttemporalcortex.Brain, 121,1133–1142.

Hoenig, K., & Scheef, L. (2005). Mediotemporal contributionsto semantic processing: fMRI evidence from ambiguity pro-cessing during semantic context verification. Hippocampus, 15,597–609.

Holcomb,P.(1993).Semanticprimingandstimulusdegradation:Implications for he role of the N400 in language processing.Psychophysiology, 30,47–61.

Humphries, C., Binder, J. R., Medler, D. A., & Liebenthal,E. (2007). Time course of semantic processes during sentencecomprehension:AnfMRIstudy.Neuroimage, 36,924–932.

Jackendoff, R. (1997). The architecture of the language faculty.Cambridge,MA:MITPress.

Jackendoff,R.(1999).Therepresentationalstructuresofthelan-guage faculty and their interactions. In C. M. Brown andP. Hagoort (Eds.), The neurocognition of language (pp. 37–79).Oxford,UK:OxfordUniversityPress.

Jackendoff,R.(2002).Foundations of language: Brain, meaning, grammar, evolution.Oxford,UK:OxfordUniversityPress.

Keenan, E. L. (1979). On surface form and logical form. Stud. Linguist. Sci.,8,163–203.

Kiehl, K. A., Laurens, K. R., & Liddle, P. F. (2002). Readinganomaloussentences:Anevent-relatedfMRIstudyofsemanticprocessing.Neuroimage, 17,842–850.

Kim, A., & Osterhout, L. (2005). The independence of com-binatory semantic processing: Evidence from event-relatedpotentials.J. Mem. Lang.,52,205–225.

Kuperberg,G.R. (2007).Neuralmechanismsof languagecom-prehension:Challengestosyntax.Brain Res., 1146,23–49.

Kuperberg, G. R., Holcomb, P. J., Sitnikova, T., Greve, D.,Dale,A.M.,&Caplan,D. (2003).Distinctpatternsofneuralmodulation during the processing of conceptual and syntacticanomalies.J. Cogn. Neurosci.,15,272–293.

Kuperberg, G. R., Lakshmanan, B. M., Caplan, D. N., &Holcomb, P. J. (2006). Making sense of discourse: An fMRIstudy of causal inferencing across sentences. Neuroimage, 33,343–361.

Kuperberg,G.R.,McGuire,P.K.,Bullmore,E.T.,Brammer,M. J., Rabe-Hesketh, S., Wright, I. C., Lythgoe, D. J.,Williams,S.C.,&David,A.S. (2000).Commonanddistinctneuralsubstratesforpragmatic,semantic,andsyntacticprocess-ing of spoken sentences: an fMRI study. J. Cogn. Neurosci., 12,321–341.

Kuperberg,G.R.,Sitnikova,T.,&Lakshmanan,B.M. (2008)Neuroanatomicaldistinctionswithinthesemanticsystemduring

sentence comprehension: Evidence from functional magneticresonanceimaging.Neuroimage, 40, 367–388.

Kutas,M.,&Federmeier,K.D.(2000).Electrophysiologyrevealssemanticmemoryuseinlanguagecomprehension.Trends Cogn. Sci.,4,463–470.

Kutas,M.,&Hillyard,S.A.(1980).Readingsenselesssentences:Brain potentials reflect semantic anomaly. Science, 207,203–205.

Kutas, M., & Hillyard, S. A. (1984). Brain potentials duringreadingreflectwordexpectancyandsemanticassociation.Nature, 307,161–163.

Kutas,M.,VanPetten,C.,&Kluender,K.R.(2006).Psycho-linguistics electrified. II. 1994–2005. In M. Traxler & M. A.Gernsbacher (Eds.), Handbook of psycholinguistics (2nd ed., pp.659–724).Elsevier.

Levelt,W.J.M.(1999).Producingspokenlanguage:Ablueprintofthespeaker.InC.M.Brown&P.Hagoort(Eds.),The neuro-cognition of language(pp.83–122).Oxford,UK:OxfordUniversityPress.

Li,X.,Hagoort,P.,&Yang,Y. (2008).Eventrelatedpotentialevidence on the influence on the influence of accentuation inspoken discourse comprehension in Chinese. J. Cogn. Neurosci.,20,906–915.

Maguire,E.A.,Frith,C.D.,&Morris,R.G.(1999).Thefunc-tional neuroanatomy of comprehension and memory: Theimportanceofpriorknowledge.Brain, 122,1839–1850.

Mason,R.A.,&Just,M.A.(2006).Neuroimagingcontributionsto the understanding of discourse processes. In M. Traxler &M.A.Gernsbacher(Eds.),Handbook of Psycholinguistics(pp.765–799).Amsterdam:Elsevier.

McCarthy, G., Nobre, A. C., Bentin, S., & Spencer, D. D.(1995).Language-relatedfieldpotentials intheanterior-medialtemporallobe.I.Intracranialdistributionandneuralgenerators.J. Neurosci., 15,1080–1089.

McElree,B.,Traxler,M.J.,Pickering,M.J.,Seely,R.E.,&Jackendoff, R. (2001). Reading time evidence for enrichedcomposition.Cognition, 78,B17–25.

Montague,R.(1970).Universalgrammar.Theoria, 36,373–398.Moss,H.E.,Abdallah,S.,Fletcher,P.,Bright,P.,Pilgrim,L.,

Acres,K.,&Tyler,L.K.(2005).Selectingamongcompetingalternatives: Selection and retrieval in the left inferior frontalgyrus.Cereb. Cortex, 15,1723–1735.

Newman, A. J., Pancheva, R., Ozawa, K., Neville, H. J., &Ullman,M.T.(2001).Anevent-relatedfMRIstudyofsyntacticandsemanticviolations.J. Psycholinguist. Res., 30,339–364.

Ni,W.,Constable,R.T.,Mencl,W.E.,Pugh,K.R.,Fulbright,R. K., Shaywitz, S. E., Shaywitz, B. A., & Gore, J. (2000).An event-related neuroimaging study distinguishing formand content in sentence processing. J. Cogn. Neurosci., 12,120–133.

Nieuwland,M.S.,&VanBerkum,J.J.A.(2006).Whenpeanutsfall in love:N400evidenceforthepowerofdiscourse.J. Cogn. Neurosci.,18,1098–1111.

Osterhout, L., & Holcomb, P. J. (1992). Event-related brainpotentials elicited by syntactic anomaly. J. Mem. Lang., 31,785–806.

Osterhout,L.,Kim,A.,&Kuperberg,G.R.(2007).Theneuro-biologyofsentencecomprehension.InM.Spivey,M.Joanaisse,& K. McRae (Eds.), The Cambridge handbook of psycholinguistics.Cambridge,UK:CambridgeUniversityPress.

Osterhout, L., McLaughlin, J., & Bersick, M. (1997). Event-relatedbrainpotentialsandhumanlanguage.Trends Cogn. Sci.,1, 203–209.

14

16

15


Y


Partee, B. H. (1984). Compositionality. In F. Veltman, &F. Landmand (Eds.), Varieties of formal semantics. Dordrecht:Foris.

Partee, B. H., Ter Meulen, A., & Wall, R. E. (1990).Mathematical methods in linguistics.Kluwer.

Pylkknen, L., & McElree, B. (2007). An MEG study of silentmeaning.J. Cogn. Neurosci.,19,1905–1921.

Rodd,J.M.,Davis,M.H.,&Johnsrude,I.S.(2005).Theneuralmechanismofspeechcomprehension:fMRIstudiesofsemanticambiguity.Cereb. Cortex, 15,1261–1269.

Ruschemeyer, S. A., Fiebach, C. J., Kempe, V., & Friederici,A.D.(2005).Processinglexicalsemanticandsyntacticinforma-tioninfirstandsecondlanguage:fMRIevidencefromGermanandRussian.Hum. Brain Mapp.,25,266–286.

Ruschemeyer,S.A.,Zysset,S.,&Friederici,A.D.(2006).Nativeand non-native reading of sentences: An fMRI experiment.Neuroimage, 31,354–365.

Seuren,P.A.M. (1998).Western linguistics: An historical introduction.Oxford,UK:Blackwell.

Sieborger, F. T., Ferstl, E. C., & von Cramon, D. Y. (2007).Makingsenseofnonsense:AnfMRIstudyoftaskinducedinfer-enceprocessesduringdiscoursecomprehension.Brain Res.,1166,77–91.

StGeorge,M.,Kutas,M.,Martinez,A.,&Sereno,M.I.(1999).Semanticintegrationinreading:Engagementoftherighthemi-sphereduringdiscourseprocessing.Brain, 122,1317–1325.

Sturt, P. (2007). Semantic re-interpretation and garden pathrecovery.Cognition, 105,477–488.

Thompson-Schill,S.L.,Bedny,M.,&Goldberg,R.F. (2005).The frontal lobes and the regulation of mental activity. Curr. Opin. Neurobiol.,15,219–224.

Thompson-Schill, S. L., D’Esposito, M., Aguirre, G. K., &Farah, M. J. (1997). Role of left inferior prefrontal cortex inretrievalofsemanticknowledge:areevaluation.Proc. Natl. Acad. Sci. USA,94,14792–14797.

Traxler, M., McElree, B., Williams, R., & Pickering,M.(2005).Contexteffectsincoercion:Evidencefromeyemove-moents.J. Mem. Lang., 53,1–25.

Traxler,M.,Pickering,M.,&McElree,B.(2002).Coercioninsentence processing: Evidence from eye movements and self-pacedreading.J. Mem. Lang.,47,530–547.

VanAtteveldt,N.M.,Formisano,E.,Blomert,L.,&Goebel,R. (2007).Theeffectof temporalasynchronyon themultisen-sory integration of letters and speech sounds. Cereb. Cortex, 17,962–974.

Van Atteveldt, N., Formisano, E., Goebel, R., & Blomert,L.(2004).Integrationoflettersandspeechsoundsinthehumanbrain.Neuron,43,271–282.

VanBerkum,J.J.A.,VandenBrink,D.,Tesink,C.,Kos,M.,&Hagoort, P. (2008). The neural integration of speaker andmessage.J. Cogn. Neurosci.,20,580–591.

Vigneau, M., Beaucousin, V., Herve, P. Y., Duffau, H.,Crivello,F.,Houde,O.,Mazoyer,B.,&Tzourio-Mazoyer,N.(2006)Meta-analyzinglefthemispherelanguageareas:Pho-nology, semantics, and sentence processing. Neuroimage, 30,1414–1432.

Wagner, A. D., Pare-Blagoev, E. J., Clark, J., & Poldrack,R.A.(2001).Recoveringmeaning:Leftprefrontalcortexguidescontrolledsemanticretrieval.Neuron, 31,329–338.

Willems,R.M.,Özyrek,A.,&Hagoort,P.(2007).Whenlan-guage meets action: The neural integration of gesture andspeech.Cereb. Cortex, 17,2322–2333.

Willems,R.M.,Özyrek,A.,&Hagoort,P.(2008).Seeingandhearing meaning: Event-related potential and functional mag-neticresonanceImagingevidenceofwordversuspictureintegra-tionintoasentencecontext.J. Cogn. Neurosci.,20,1235–1249.

Xu,J.,Kemeny,S.,Park,G.,Frattali,C.,&Braun,A.(2005).Languageincontext:Emergentfeaturesofword,sentence,andnarrativecomprehension.Neuroimage,25,1002–1015.

Zempleni,M.Z.,Renken,R.,Hoeks,J.C.,Hoogduin,J.M.,&Stowe,L.A.(2007).Semanticambiguityprocessinginsentencecontext: Evidence from event-related fMRI. Neuroimage, 34,1270–1279.

17


Y


Y

AUTHOR QUeRy FORMDear Author

During the preparation of your manuscript for publication, the questions listed below have arisen. Please attend to these matters and return this form with your proof.

Many thanks for your assistance.

Query Query RemarksReferences

1. AUTHOR:PlsupdateBaggio…inpressifpossible

2. AUTHOR:Wedon’thavevan(Van?)Berkumetal.’99

3. AUTHOR:“himself”perfigure,OK?


5. AUTHOR:PlsexplainMEG.

6. AUTHOR:“nextsection”OK?Ifnot,whatismeantby“below”?

7. AUTHOR:IsLakshmanan(perp.7-205)correct?OrLakshmann?

8. AUTHOR:OKasedited?

9. AUTHOR:Lakshmann?

10. AUTHOR:Wedon’thaveMartin-Loechesetal.’08

11. AUTHOR:Tesinketal.isn’tintheReferences.

12. AUTHOR:PlsexplainNCL.


14. AUTHOR:IsLakshmananOK?OrLakshmann?

15. AUTHOR:IsLakshmananOK?OrLakshmann?

16. AUTHOR:Kutas,VanP…:plsgiveplaceofpublication

17. AUTHOR:Partee,TerM…:plsgiveplaceofpublication.

18. AUTHOR:Shouldthesourcebegiven?Hagoort…’04?

19. AUTHOR:Shouldthesourcebegivenhere?Baggio,vanLambalgen,andHagoort’08

20. AUTHOR:TablesOKasset?

GAZ_56


56 semantic unification - mpg.pure

Documents