computational models of emotion inference in theory of ... · aa*star artificial intelligence...

Running Head: EMOTION IN THEORY OF MIND

Computational models of emotion inference in Theory of Mind:

A review and roadmap

Desmond C. Onga,b, Jamil Zakic, and Noah D. Goodmanc,d aA*STAR Artificial Intelligence Initiative, Agency for Science, Technology and Research

(A*STAR), Singapore

bInstitute of High Performance Computing, Agency for Science, Technology and Research

(A*STAR), Singapore cDepartment of Psychology, Stanford University

dDepartment of Computer Science, Stanford University

In Press at: Topics in Cognitive Science

Special Issue on: Computational Approaches to Social Cognition

Final version dated: 1 July 2017 (Minor edits to funding acknowledgements)

Address Correspondence to:

Desmond C. Ong A*STAR Artificial Intelligence Initiative Agency for Science, Technology and Research, Singapore 138632 [email protected] Author Contributions: D.C.O., J. Z., and N. D. G. wrote the paper. The authors declare no conflict of interest. Keywords: Emotion; Affective Cognition; Inference; Theory of Mind

EMOTIONINTHEORYOFMIND 2

ABSTRACT

Research on social cognition has fruitfully applied computational modeling approaches to

explain how observers understand and reason about others’ mental states. By contrast, there has

been less work on modeling observers’ understanding of emotional states. We propose an

intuitive theory framework to studying affective cognition---how humans reason about emotions-

--and derive a taxonomy of inferences within affective cognition. Using this taxonomy, we

review formal computational modeling work on such inferences, including: causal reasoning

about how others react to events, reasoning about unseen causes of emotions, reasoning with

multiple cues, as well as reasoning from emotions to other mental states. Additionally, we

provide a roadmap for future research by charting out inferences---such as hypothetical and

counterfactual reasoning about emotions---that are ripe for future computational modeling work.

This framework proposes unifying these various types of reasoning as Bayesian inference within

a common “intuitive Theory of Emotion”. Finally, we end with a discussion of important

theoretical and methodological challenges that lie ahead in modeling affective cognition.


Recentdevelopmentsincomputationalcognitivemodelinghaveallowed

researcherstospecifyandtestprecisehypothesesabouthowpeoplemakeinferences

abouttheirsocialworld.Theseincludeinferencesaboutwhatothersdesireandbelieve

abouttheworld(e.g.,Baker,Jara-Ettinger,Saxe,&Tenenbaum,2017;Goodmanetal,

2006),whatothersmeanwhentheyuselanguagetocommunicate(Goodman&Frank,

2016;Goodman&Stuhlmüller,2013;Frank&Goodman,2012),andwhatfuturedecisions

othersmightmake(Jara-Ettinger,Gweon,Schulz,&Tenenbaum,2016;Jern&Kemp,2015).

Theserecentdevelopmentscomprisethefocusofthecurrentspecialissue.However,these

modelshaveoftenneglectedhowpeoplereasonaboutoneofthemostessentialelements

ofhumanpsychology:emotions.

Emotionsplayacentralroleinoursociallives.Theysignalpeople’simmediate

reactionstoeventsintheworld(e.g.,Ellsworth&Scherer,2003),andcausemany

behaviors---bothintentionalandunintentional(Lerner,Li,Valdesolo,&Kassam,2015;

Loewenstein&Lerner,2003).Giventheimportanceofemotionsinsocialinteractions,itis

hardlysurprisingthatpeoplearenaturallyattunedtoperceivingandunderstanding

emotionsinthosearoundthem(Harris,1989;Zaki&Ochsner,2011).

Inthispaper,weconsiderthepresentandfutureofcomputationalmodelingin

understandinghowpeoplereasonaboutemotionalstates---whatwetermaffective

cognition(Ong,Zaki,&Goodman,2015).Weadoptanintuitivetheoryframeworktothe

studyofaffectivecognition,usingwhichwederiveataxonomyofaffectivecognitive

inferences(inthespiritofAdelson&Bergen,1991,andKemp&Jern,2013,who

respectivelyderivedtaxonomiesofvisualfunctionsandinductiveproblems).This

taxonomyencompassesawiderangeoflayreasoningaboutemotionalstates,suchasusing


observedbehaviororsituationcontexts(orcombinationsthereof)toinferemotionsand

othermentalstates(e.g.,DeMelo,Carnevale,Read,&Gratch,2014;Wu,Baker,Tenenbaum,

&Schulz,2018;Zaki,2013).Finally,weprovidearoadmapforfutureresearch,by

highlightinginferencesthathaveyettobestudiedcomputationally,andbydiscussing

importanttheoreticalandmethodologicalchallengesthatlieahead.

Layingoutanintuitivetheoryofemotions

Peoplehaverichintuitivetheoriesofhowothersaroundthemthinkandbehave,

allowingthemtoinferothers’motivationsandexplainothers’behavior(e.g.,Heider,1958;

Gopnik&Meltzoff,1997;Ross,1977).Anintuitivetheoryofotherpeopleconsistsoffirst,a

structuredontologyofconcepts---forexample,personality,goals,behavior---andsecond,

thecausalrelationshipsrelatingtheseconcepts(Gerstenberg&Tenenbaum,2017).These

intuitivetheoriesallowlaypeopletomakesenseofothersaroundthem,inasimilarfashion

tohowscientifictheoriesallowscientiststoexplainthephysicalworld(Carey,2009;

Wellman&Gelman,1992).

Peoplealsopossessarichintuitivetheoryofemotionsthatcomprisesconceptual

knowledgeaboutdifferentemotionalstates(e.g.,anger,happiness)andhowtheyare

relatedtotheircausesandeffects.People(“observers”)usetheirintuitivetheoryof

emotiontoreasonabouttheemotionalstatesofothers(“agents”)aroundthem,and

therebydecidehowbesttorespondinsocialsituations.Importantly,theseintuitive

theoriescomprisetheobserver’sbeliefsabouthowothers’emotionswork,whichdepend

ontheobserver’spasthistoryandtheirsubjectivebeliefs.Thoughtheobserver’sbeliefs

maynotnecessarilyreflecttherealityofhowemotions“actuallywork”,thesebeliefs


neverthelessformthebasisforhowtheobserverunderstandsandinteractswiththose

aroundthem(e.g.,Gopnik&Meltzoff,1997;Ross,1977).

Anintuitivetheoryofemotioncontainstwoimportanttypesofcausalrelationships.

Thefirstconnectsemotionstotheircauses:what,intheobserver’smind,causesanagent

tofeelanemotion?Thesecondinvolvestheeffectsofemotion:what,intheobserver’s

mind,doesanagent’semotioncausethemtodo?Thesecomponentsandtheirrelationships

arerepresentedinFigure1.

Figure1:Amodelofanintuitivetheoryofemotion,unifyingideasfromdeMeloetal(2014),Ong,Zaki,&Goodman(2015),SaxeandHoulihan(2017),andWuetal(2018).Weusestandardgraphicalmodelnotation:shadedcirclesrepresentobservablevariables,whileunshadedcirclesrepresentlatentvariables.Werendervariablesatthesubsequent“time-step”translucent.Arrowsrepresentadirectedcausalrelationship,andboldedlettersdenotetheabbreviationsusedinequations.Theobserverappliesa“third-personappraisal”processtoreasonabouthow(i)theoutcomeofaneventthatanagentexperiences,and(ii)theagent’smentalstates(beliefsanddesires),togetherresultintheagentexperiencingemotions.Theagent’semotionsinturncausetheagenttodisplayemotionalexpressions,andtakeintentionalactionsthatleadtonewoutcomesandupdatedmentalstates(andnewemotions).

Intuitivecausesofemotions

Peopleintuitivelyexpectthatothers’emotionsariseasareactiontomotivationally

salientevents.Inadditiontotheoutcomeoftheseevents,peoplealsousetheirknowledge

aboutothers’mentalstates—suchasothers’beliefsanddesires—toreasonabouthow

othersfeel.Thisintuitiveunderstandingofhoweventoutcomesandmentalstatesimpact

emotionsemergesearlyinlife(Gross&Ballif,1991;Harris,1989;Lagattuta,Wellman,&


Flavell,1997;Repacholi,Meltzoff,Hennings,&Ruba,2016;Wu,Muentener,&Schulz,

2017):toddlersandinfantsunderstandthatanagent’sdisplayofhappinessreflectsa

satisfieddesire,whileanagent’sdisplayofsadnessreflectsathwartedgoal(Repacholi&

Gopnik,1997;Skerry&Spelke,2014;Wellman,Phillips,&Rodriguez,2000).Bypreschool,

childrenassociateasurprisedemotionalexpressionwithamismatchbetweenrealityand

theagent’spriorbeliefsabouttheworld(Wellman&Banerjee,1991;Wu&Schulz,2017),

andfactoragents’expectationsintoattributionsoftheagent’semotions(Ong,Asaba,&

Gweon,2016).Adultssimilarlydrawrichinferencesfromanagent’semotionalreactionsto

theirlatentbeliefsanddesires(Scherer&Grandjean,2008;Wuetal,2018;VanKleef,De

Dreu,&Manstead,2010).Thus,theintuitivetheoryofemotionrelatesanagent’smental

states(specifically,theirbeliefsabouttheworld,andtheirdesires),andtheoutcomeof

aneventthattheagentexperiences,totheagent’semotions(Fig.1).

Howdopeopleunderstandthewayeventoutcomesandtheagent’smentalstates

giverisetoemotions?Oneideaisthatpeoplemayengageasimilarreasoningprocessto

howtheythemselvesexperienceeventsfirsthand.Accordingtoappraisaltheoriesof

emotion(Arnold,1960;Ellsworth&Scherer,2003;Moors,Ellsworth,Scherer&Frijda,

2013;Ortony,Clore,&Collins,1988;Smith&Lazarus,1993),anagent’semotionsarise

fromanevaluation(“appraisal”)ofoutcomesalongvariousself-relevantdimensions,such

aswhethertheoutcomescontributetoordetractfromtheagent’sgoals(“goal-

conduciveness”).IfSallyjudgesherjobofferasfacilitatinghercareergoals,shewould

likelyfeelhappy.

Manyresearchershaveproposedthatlaypeopleperformasimilarappraisal-like

processtoreasonaboutothers’emotions(DeMeloetal,2014;Ong,Zaki,&Goodman,


2015;Seimer&Reisenzein,2007;Skerry&Saxe,2015;Wondra&Ellsworth,2015;Wuet

al,2018;VanKleefetal,2010)---wetermthisprocess“third-personappraisal”.Thatis,

whenanobserverconsidersanagentexperiencinganoutcome,theobserverfirstreduces

theoutcometoasmallnumberoffeaturesthattheobserverthinksisrelevanttotheagent,

andthenusesthatevaluationtojudgetheagent’semotions.Forexample,observers

intuitivelyevaluate,fromtheagent’sperspective,anoutcome’sgoal-conducivenessand

whethertheoutcomewasexpected.IfBoblearnsthatSallyjustreceivedglowing

performancereviews,andknowsofhergoalsofgettingpromoted,hewouldreasonthat

sheprobablyfeelsapositively-valencedemotion.IfBobadditionallyknowsthatSallydid

notexpecttheperformancereviews,hemightpredictthatshewouldalsofeelsurprised.

Appraisalisespeciallyimportantfromacomputationalstandpoint.Thesituationsthat

agentsexperienceindailylifevarywidelyalonganinnumerablenumberofdimensions,

and(theobserver’stheoryof)anagent’sappraisalreducesthecomplexitiesofeveryday

situationstoasmallsetofrelevant“features”thatareimportantfortheagent’swell-being.

Itisalsoimportanttonotethattheobserver’sappraisalonbehalfoftheagentmaydiffer

fromtheagent’sownappraisalintwokeyways.First,theobserver’ssetofemotionally-

relevantfeatures,encapsulatedwithintheirintuitivetheory,maynotbethesameastheset

offeaturesthattheagentthemselvesconsider,suchaswhenanobserverbringstheir

culturallyheldnormsaboutemotionstoassessanoutcomeexperiencedbyanagentfroma

differentculture.Second,theobserver’sthird-personappraisalsdependontheobserver’s

inferencesoftheagent’spriorbeliefsanddesires,whichmaybedifferentfromtheagent’s

truementalstates.IfBobthoughtthatSallyhadknownaboutherperformancereviews

beforehand,hewouldnothavepredictedhersurprise.


Thesefirstfewpiecesofamodeloftheintuitivetheory(seelefthalfofFig.1),

describetheobserver’srichcausalknowledgeabouthowoutcomesandtheagent’smental

states“cause”emotion.Thisreasoningfromoutcomesandmentalstatestoemotionsrelies

onacontextualized,third-personappraisalprocess,whichwewilldescribeingreater

detaillater.WeturnnexttotherighthalfofFigure1.

Intuitiveeffectsofemotions

Peoplealsointuitivelyreasonaboutwhatbehaviorsemotionscause.Forone,

emotionalagentsproduceawidevarietyofemotionalexpressions.Decadesofscientific

researchhaveshownthatpeoplecanreliablyrecognizeemotionsfromfacialexpressions

(Ekman,Friesen,&Ellsworth,1972;Elfenbein,&Ambady,2002;Russell,Bachorowski,&

Fernández-Dols,2003),bodylanguage(Dael,Mortillaro,&Scherer,2012),aswellas

emotionalvocalizationsandotherparalinguisticchangesinspeech(Banse,&Scherer,

1996;Johnstone&Scherer,2000;Scherer,2003).Alongwithemotionaldisplays,emotions

alsoinfluencetheintentionalactionsthattheagentmaytakenext(Lerneretal,2015;

Loewenstein&Lerner,2003).Fearmayproduceatendencytoflee,angermayprompt

someonetoapproachandconfrontathreat,andhappinessmightengenderprosocial

behavior(e.g.,Frijda,Kuipers,&TerSchure,1989;Isen,1987).Peopleareintuitively

sensitivetohowemotionscaninfluenceothers’actions;forexample,peopleuseapartner’s

emotionaldisplaystoinferhowcooperativetheywouldbe(Krumhuberetal,2007;Levine,

Barasch,Rand,Berman,&Small,2018;VanCleefetal,2010).Intheintuitivetheoryin

Figure1,wehaveplacedemotionalexpressionsandintentionalactionsasdownstream

effectsofemotion,reflectingtheintuitivenotionthatemotions“cause”thesetwotypesof


behavior.Bothtypesofbehaviorarereadilyobservableindailylifeandprovidecrucial

signalsthatallowobserverstoreason“backwards”toinfertheagent’slatentemotion.

ATaxonomyofAffectiveCognitiveInferenceswithintheIntuitiveTheory

InthepreviousSection,welaidoutamodelofanintuitivetheoryofemotion(Figure

1)byconsideringhowemotionsarerelatedtotheirtwoantecedentcausesandtheirtwo

consequenteffects.Usingthisintuitivetheory,wecanderiveataxonomyofinferencesvia

exhaustiveenumeration(seeSupplementaryMaterial).TreatingthemodelinFigure1asa

Bayesiannetwork,wefirstwroteoutallthe47possibleinferencesthathaveemotionas

oneofthevariables---forexample,P(emotion|outcome),inferringemotionsfromoutcomes,

orP(expression|emotion),reasoningaboutexpressionsfromemotions.Next,weremoved

26inferencesthatreducedtosimplerinferencesbasedonconditionalindependence1.

Third,weclassifiedtheremaining21inferencesintosixcategories.Finally,weaddeda

seventhcategorytoaccountforcounterfactualinferences,suchasinferringemotionsifa

givenoutcomehadnotoccurred,P(emotion|notoutcome).ThisissummarizedinTable1.

1Forexample,underthemodel,P(outcome|emotion,expression)reducestoP(outcome|emotion)asoutcomesandexpressionsareindependentgivenemotions(d-separation,e.g.,Koller&Friedman,2009).


Categories Description Inferences ExamplesEmotionRecognition

Inferanagent’semotionsfromemotionalexpressions(facialexpressions,bodylanguage,prosody),orfromtheiractions.

P(e|x)P(e|a)

Givensmile,InferP(happy)Givenscold-others,InferP(anger)

Third-personappraisals

Reason“forward”abouthowaneventwouldcauseanagenttofeel,givenalsotheirmentalstates.

P(e|o)P(e|o,b,d)P(e|b,d)

Givenlose-wallet,InferP(sad)

Inferringcausesofemotions

Reason“backwards”abouttheeventsthatcausedanagent’semotions.

P(o|e)P(o|e,b,d)

Givensad,InferP(lose-wallet)

EmotionalCueIntegration

Givenmultiple,potentiallyconflictingcues(e.g.,multiplebehaviorsand/orcausesofemotion),combinethemandreasonaboutagent’semotions

e.g.,P(e|o,x)P(e|o,a)P(e|a,x)P(e|o,b,d,a,x)

Givensmile+lose-wallet,InferP(happy),P(sad),etc.

ReverseAppraisal

Givenaneventandanagent’semotions,reasonbackwardstomentalstateslikebeliefsanddesires

P(b,d|e,o)P(b,d|e)

Given:receive-gift+surprise+happy,InferGift-Unexpected+Gift-Desired

Predictions(HypotheticalReasoning)

Givenanagent’semotions,predictsubsequentbehavior.Or,givena(hypothetical)situation,predicttheagent’semotions.

P(a|e)P(x|e)

Ifanger,predictP(scold-others)

CounterfactualReasoning(andExplanations)

Givenastateoftheworldandanagent’semotions,reasonaboutemotionsorbehaviorincounterfactualstatesoftheworld.Thisalsoallowsexplanationsofemotionsorbehaviorintermsoftheircauses.

e.g.,P(e|noto)P(a|note)

Given:“lose-walletandsad”,reasonthat:“ifnot-lose-wallet,thennot-sad”or,reasonthat:“sadbecauselose-wallet”

Table1:AtaxonomyofinferenceswithinAffectiveCognition,derivedfromthemodelinFigure1.P(X)denotestheprobabilityofXoccurring.Thelistsofinferencespresentedforeachcategoryareexhaustive(givenourderivation),exceptfortheCueIntegrationandCounterfactualReasoningcategories.


1.EmotionRecognition

Perhapstheeasiestcategoryoflayinferenceisinferringsomeone’semotionalstate

fromtheiremotionalexpressions,oftencalledemotionrecognition(e.g.,Elfenbein&

Ambady,2002).Thedeceptiveeasebywhichlaypeople“read”emotionalexpressionshas

ledmanyresearcherstoassumeperfectemotionrecognitionintheirmodels,without

modelingthisprocessasaninferenceinitsownright(e.g.,deMeloetal,2014;Wuetal,

2018).

Inreality,laypeople’semotionrecognitionisnotperfectlyaccurate,norisit

homogenousacrossallobservers.First,suchinferencesareheavilydependentoncontext:

Bothintensepositiveandnegativeeventsmaysometimesproduceperceptually-similar

facialexpressions,makingitdifficultforlayobserverstoaccuratelyinferemotionssolely

fromexpressions(Aviezer,Trope,&Todorov,2012;Carroll&Russell,1996;Wenzleretal,

2016).Second,peoplefromdifferentculturesdiffersystematicallyinhowtheyperceive

emotionsfromfacialexpressions(Gendron,Roberson,vanderVyver,&Barrett,2014;Jack,

Blais,Scheepers,Schyns,&Caldara,2009;Russell,1994;Yuki,Maddux,&Masuda,2007).

Suchinferencesrelyonlayobservers’intuitivetheories,whichinturndependontheir

culturalbackgroundandpasthistory(Jack,Caldara,&Schyns,2012).Indeed,thesecultural

differenceshavebeenfoundtoimpactevenhowobserversattendtoanduselow-level

facialcueswhenjudgingemotions:forexample,EastAsianparticipantstendtofixate

longerontheeyeregionthanWesternparticipants(Jacketal,2009;Yukietal,2007).

Morerecently,researchershavestartedbuildingformalcomputationalmodelsof

howobserversinferemotionsfromlow-levelfeaturesoffacialexpressions(Delisetal,

2016;Jacketal,2012;Martinez&Du,2012)andbodylanguage(Schindler,VanGool,&de


Gelder,2008).Usingconditionalprobabilitynotation,modelingemotionrecognition

correspondstoestimatingtheprobabilityofalatentemotionegivenanemotional

expressionx,orP(e|x).Thegeneralrecipefollowedbythesestudiesfirstrequiressome

representationoftheemotionalexpression---forexample,apopularrepresentationfor

facialexpressionsistheFacialActionCodingSystem(Ekman&Friesen,1978).Next,one

measureshowhumanobserversclassifystimuliintopre-definedemotioncategories,i.e.,

empiricallymeasuringP(e|x)forsay,thesix“basic”emotioncategories.Finally,onethen

modelstheemotionrecognitionprocessP(e|x)bytrainingclassifiersusingsophisticated

machinelearningtechniques.Suchworkisrelativelynew,andthereisstillmuchmoreto

bedoneinmodelinghumanemotionrecognitionfromexpressionsandalsofromother

behavior:thiswillprovideacrucialfoundationformanyofthemorecomplexinferences

wediscussbelow.

2.Third-personappraisal

Computationalapproachesallowresearcherstoquantitativelymodelhowpeople

reasonaboutanagent’sresponsetoanexperiencedevent.Thisinvolves,asmentioned,a

third-personappraisalprocessthatreducestheoutcomethatanagentexperiencesintoa

smallsetoffeaturesthattheobserverthinksisimportanttotheagent’semotions(e.g.,

Skerry&Saxe,2015).Tostudythisappraisalprocessinacontrolledscenario,wedesigned

agamblingscenariowherewecouldparametricallyvarythepayoffamountsand

probabilities,andwemeasuredhowpeople’sjudgmentsofanagent’semotionsafterthe

gambledependeduponthesegamblefeatures(Ong,Zaki,&Goodman,2015).Wefound

thatparticipants’emotionjudgmentsdependednotonlyontheamountwon,buthow


muchtheagentwoncomparedtotheexpectedvalueofthegamble,oftencalleda

predictionerror(e.g.,Sutton&Barto,1998),suggestingthatparticipantsevaluatedthe

agent’searningsrelativetoareferencepoint,ratherthanjudgingtheirearningsinabsolute

terms.Wealsofoundthatlayparticipantsjudgedagentstoreactmorestronglytonegative

predictionerrorsthantopositivepredictionerrors.Thesethird-personfindingsare

strikinglyconsistentwithinfluentialscientifictheoriesoffirst-personutility,especially

notionsofreference-dependentutilityandlossaversion(Kahneman&Tversky,1979;

Kermer,Driver-Linn,Wilson,&Gilbert,2006;MellersSchwartz,Ho,&Ritov,1997).Finally,

participantsbelievedthatagentswouldfeelworseiftheycamevery“close”towinningthe

nexthigheroutcome,comparedtoifagentswerenotclosetowinningthebetteroutcome

(Ong,Goodman,&Zaki,2015).Thatis,participants’intuitivetheorieswerealsosensitiveto

the“closeness”ofalternativeoutcomes(Gleicheretal,1990;Kahneman&Tversky,1982;

Johnson,1986),whichhaveyettofeatureinfirst-personappraisaltheories.

Thesituationfeatureswestudied---predictionerrorsandthe“closeness”ofthe

outcometoalternativeoutcomes---arewell-definedwithinourexperimentalscenario,but

muchmoreworkisneededtostudythird-personappraisalsinmorecomplex,real-world

situations.Themainchallengesforfutureresearchwouldbedefiningacomprehensive

representationforthemanyappraisaldimensions---usinglistsfromfirst-personappraisal

theory(e.g.,Ellsworth&Scherer,2003;Ortonyetal,1988),orvia“data-driven”approaches

likerepresentationalsimilarityanalyses(Skerry&Saxe,2015)---andmodelinghowpeople

maptheseappraisaldimensionsontoemotions.Therealsoexistsmanytechnical

challengesinautomaticallyextractingtheseappraisaldimensionsfromnaturalisticsources


likevisualscenesorvignettedescriptions,themselvesproblemsattheforefrontofmodern

computerscience.

3.Inferringcausesofemotions

Givenanobservedemotion,peoplecanalsoreason“backwards”aboutpotential

causesofthatemotion.Uponencounteringasadfriend,mostpeoplewouldstart

generatingpotentialcausesofthefriend’ssadness,suchastheirpossiblyfailinganexam.

Intuitively,thesepossibilitiesvaryinhowlikelytheyweretohavehappened:failingan

examismorelikelywhenoneisastudent,butlesslikelyafteronehasgraduated.Theyalso

rangeinhowlikelytheyweretohavecausedtheobservedemotion:ahystericallybawling

friendprobablyexperiencedsomethingworsethanafailedexam.Peopleintuitivelyagree

thatapotentialcausewithahighprobabilityofoccurringandahighprobabilityofcausing

theobservedemotionisamorelikelycandidatethaneitheronewithalowerprobabilityof

occurringoronewithalowercausalstrength.

OnewaytomathematicallydescribethisreasoningisviaBayesianinference.First,

werepresentthethird-personappraisalprocessasP(e|o),acollectionofprobability

distributionsofemotionegivenanoutcomeo2.Then,byBayes’Rule,wecanwritethe

posteriorprobabilityP(o|e),theprobabilityofoutcomeogivenemotione,as:

. (Eqn.1)

Theposteriorprobability(e.g.,P(fail-exam|sad))isproportionaltoboththelikelihoodof

thatoutcomecausingtheemotion(e.g.,howlikelyisitthatfailinganexamwillcauseone’s

2Forthemoment,weleaveouttheagent’sbeliefsanddesiresintheappraisalprocess.Weaddthemlaterinthesectionon“reverseappraisal”.

P (o|e) / P (e|o)P (o)


friendtobesad,P(sad|fail-exam))andthepriorprobabilityofthatoutcomeoccurring(e.g.,

howlikelyisitthattheyhadjustfailedanexam,P(fail-exam)).WetestedwhetherEquation

1providesagoodmodelofparticipants’inferencesoftheposteriorinanexperimentwith

thesamegamblingcontextdescribedintheprevioussection.Participantsobservedthe

emotionsthattheagentostensiblyfeltafterplayingagamble,aswellasthepossible

outcomes,andweretaskedwithinferringhowlikelyitwasthateachoftheoutcomes

occurring.WefoundthatamodelbasedonBayes’rule(Eqn.1)accuratelypredicted

participants’judgmentsoftheposteriorprobabilitiesofunseengambleoutcomes(Ong,

Zaki,&Goodman,2015),lendingsupporttotheclaimthatlaypeople’saffectivecognition

reliesonacoherentuseofarichintuitivetheory.Indeed,recentevidencesuggeststhat

eveninfantsaresensitivetothesecausalprobabilities,activelysearchingforhiddencauses

whenfacedwithanemotionthatisimprobablegiventheobservedcause(Wuetal,2017).

Inferringcausesofemotionsreliesonfirstunderstandingthethird-personappraisal

process,andsothechallengesinherentinmodelingthethird-personappraisalprocessalso

applyhere.Inaddition,Equation1requirestheobservertoconsiderasetofpossible

outcomes.Onechallengeforfutureresearchersistodefinethesetofpossibleoutcomes

thatlaypeopleintuitivelyshortlistandconsider,aswellasthepriorprobabilitiesthat

laypeopleassignthoseoutcomes.Estimatingsuchknowledgeisalsoakeyproblemfaced

whenapplyingBayesianmodelstootherformsofcognition(e.g.,causalinduction;Griffiths,

2017).


4.EmotionalCueIntegration

Peoplehavelittledifficultyreasoningabouttheemotionsthatresultfrom

experiencingsituationoutcomes(P(e|o)),oridentifyingtheemotionsthatproduce

observedfacialexpressions(P(e|x)).Inreallife,however,peopleareoftenpresentedwith

combinationsofmultiplecuestoanagent’semotions,andhavetoreconcilethesemultiple

cues,anissueofemotionalcueintegration(Ong,Zaki,&Goodman,2015;Skerry&Saxe,

2014;Zaki,2013).PerhapsAliceknowsthatafriendhaslosthisjob(anegativeoutcome),

butseesasmileonhisface(apositiveexpression).Howdoesshemakesenseofthis?

Scientistshavelongponderedtherelativeimportanceofsituationcontextsand

emotionalexpressionstoobservers’judgmentsofemotions,especiallyincaseswhere

thesecuesconflict(e.g.,Goodenough&Tinker,1931).Ontheonehand,scholarshave

arguedthatfacialexpressionsevolvedtocommunicate,andhenceare(always)veridical

cuestoanagent’semotion(e.g.,Buck,1994).Thus,sincethejoblessfriendissmiling,they

mustbehappy,becausesmilingevolvedtocommunicatehappiness,andtheywouldnotbe

smilingotherwise.Ontheotherhand,awealthofotherstudiesshowthatlaypeople

sometimesweightthesituationalcontextandotherbehavioralcuesoverfacialexpressions

whenattributingemotions(Aviezeretal.,2008,Caroll&Russell,1996;Kayyal,Widen,&

Russell,2015;seeOng,Zaki,&Goodman,2015,foramoredetaileddiscussion).This

impasseunderscorestheneedforaprecisetheoryofhowdifferentcuesareweightedand

integrated.

Withinourframework,emotionalcueintegrationissimplya“higher-order”

inferencethatreliesonthesimplerinferencesdiscussedearlier.Giventhecausalmodelin

Figure1,wecanwriteoutapreciseandoptimalformulaforhowanobservershould


integratethesedifferentsourcesofinformation:Forexample,theprobabilityofalatent

emotionegivenanobservedoutcomeoandexpressionx,P(e|o,x),isgivenby:

. (Eqn.2)

Notethatthisinferenceintegratesboththeprobabilityoftheemotiongiventheoutcome,

P(e|o),andtheprobabilityoftheemotiongiventhefacialexpression,P(e|x),inferenceswe

discussedearlier.Unlikepriorworkinresolvingcueconflict,thismodeldoesnotassumea

priorithatanyonetypeofcue(e.g.,facialexpressions)ismore“privileged”,butweighsthe

cuesaccordingtotheirreliabilityinpredictingtheagent’slatentemotion.Toverifythis

model,weranaseriesofexperimentswherewepresentedparticipantswithcombinations

ofoutcomes(theoutcomeofagamblethattheagentexperienced)andagents’emotional

expressions(eitherafacialexpressionorverbalutterance).ABayesianmodelbasedon

Equation2accuratelypredictedlayobservers’judgmentsofagents’emotionsgiventhese

multiplecues(Ong,Zaki,&Goodman,2015).Thismodelpredictsthatobserverswillbe

sensitivetothereliabilitiesofcuesincontext,andallowsforobserverstoweighoutcomes,

faces,bodyexpressions,andothercuestodifferingamountsdependingonthecontext.

Emotionalcueintegrationisnotlimitedtosimpletwo-cuecombinations.Underthis

Bayesianframework,cueintegrationisanyhigher-orderinferencethatreliesinparton

single-cueinferenceslikethird-personappraisal(P(e|o,b,d))andemotionrecognition

(P(e|x),P(e|a)):Onceonehasmodelsofthesesimplerinferences,onecanderiveinference

fromanycombinationofcues.

P (e|o, x) / P (e|o)P (e|x)P (e)


5.ReverseAppraisal:Inferringmentalstatesfromemotions

Peoplenotonlydrawinferencesaboutanagent’semotions,butalsouseemotions

toreasonbackwardsaboutanagent’sappraisalsandmentalstates(e.g.,Hareli&

Hess,2010;Scherer&Grandjean,2008).Inparticular,emotionalreactionsprovide

informationastowhetheranagent’sgoalsweremetandwhethertheyexpectedagiven

outcome.Apositive/negativeemotionalexpressionmaysignalthattheoutcomewas

congruent/incongruentwithanagent’sdesires,andasurprisedexpressionmightsignal

thattheoutcomewasunexpectedgiventheirbeliefs.Comparingtothethird-person

appraisalprocessthatmapsoutcomesandmentalstatestoemotions,thiscurrentpieceof

reasoningworksinreverse,takinganoutcomeandobservedexpressiontoinfertheagent’s

mentalstates---whatdeMeloandcolleagues(2014)term“reverseappraisal”.

Extendingtheformalisminprevioussections,givenamodeloftheappraisalprocess

fromtheoutcomeandtheagent’sbeliefsanddesiresP(e|o,b,d),wecanwriteaninference

similartoEquation1fortheagent’sbeliefsanddesires:

.

This,however,requiresaninferenceofthelatentemotion.If,instead,whatweactually

observeisanemotionalexpressionxandtheoutcomeo,andweareonlyinterestedinthe

mentalstates,wecanmarginalizeoutthelatentemotion:

(Eqn.3)

Notethatintheprevioussections(e.g.,Eqns.1-2),theappraisalprocesswasencapsulated

inP(e|o);here,weexplicitlyrepresenttheagent’sbeliefsanddesires,sonowwewritethe

appraisalprocessasP(e|b,d,o).Asdiscussedearlier,thisprocessitselffactorsthrough

P (b, d|e, o) / P (e|b, d, o)P (b, d)

P (b, d|x, o) / P (b, d)X

e

P (x|e)P (e|b, d, o)


quantitiessuchasthepredictionerroroftheoutcome.InEquation3,thegoalofthe

inferenceistheconditionalprobabilityoftheagent’sbeliefsanddesires;theagent’s

emotionsareonlyan“intermediate”quantitythatwecanmarginalize(orsumout).From

righttoleft,theexpressioninEquation3considerstheproductof(i)theprobabilityofan

emotionebeingaresultofabelief,desire,andoutcomeP(e|b,d,o)(i.e.,theappraisal

process)and(ii)theprobabilityoftheemotioneresultingintheobservedexpression

P(x|e).Wethenconsiderallpossibleemotionsebysummingthisproductoverallpossible

emotions(“marginalizing”oute).Finally,thistermismultipliedwiththepriorprobability

ofthatcombinationofbeliefanddesire,P(b,d).

Totestiflaypeople’sinferencescanbemodeledbysuchaBayesianmodel,Wuand

colleagues(2018;Experiment3)constructedscenarioswhereagentsperformedanaction

(puttingawhitepowderintoacolleague’scoffee),witnessedtheresultantoutcome(the

agent’scolleaguelivingordying),anddisplayedanemotionalreaction(e.g.,asurprised

facialexpression).Participantsreadaboutthesesequencesofevents,andprovided

judgmentsoftheagent’sbeliefs(whattheagentthoughtthepowderwas:poisonorsugar)

anddesires(whetherornottheagentwantedtoharmhercolleague).Wuandcolleagues

(2018)presentagraphicalmodelandaninferenceequationsimilartoFigure1and

Equation3.Wenotetwokeydifferences:first,theirposteriorhasanadditionalconditional

dependenceontheagent’sactions(i.e.,P(b,d|x,o,a))---wechosenottoaddactionsinto

Equation3forsimplicity.Second,theyassumedperfectemotionrecognitionandhencedid

notexplicitlymodelthelatentemotionasanintermediatequantitybetweentheoutcome

andtheagent’sexpression.TheirBayesianmodelaccuratelytrackedparticipants’


inferencesoftheagent’sbeliefsanddesires,providingfurtherevidencethatpeoplereason

aboutothers’emotionalandmentalstatesusingacoherentintuitivetheory.

Inaddition,inferringothers’appraisalsisalsoimportantinpredictingtheirfuture

behavior.Forexample,ifsomeonechoosestocooperatewithyouonatask,andsmiles

afterdoingso,youmayinferthattheyhadintendedtocooperate,likedtheoutcome,and

willlikelydosoagain.Bycontrast,iftheyfrownedinsteadofsmiled,yourinferencesand

yoursubsequentbehaviorinresponsewouldbeverydifferent:perhapstheyregretted

cooperating,andmightnotcooperatenexttime(seealsoLevineetal,2018).DeMeloand

colleagues(2014)modeledtheseinferencesofappraisalsfromemotionalexpressionsina

stylizedsocialdilemmagamewheretwoplayerseachchosebetweenamutuallybeneficial

optionandaselfishoptionthatbenefittedoneselfattheexpenseofone’spartner.Some

participantsplayedagainstapartnerwhosmiledaftermutualcooperation,whileother

participantssawtheirpartnerfrownaftermutualcooperation.DeMeloandcolleagues

(2014)proposedandfoundsupportfora“reverseappraisal”modelwherebyparticipants

inferredeitheracooperativeorcompetitiveintentfromtheirpartner’semotional

expressions,andtheseinferencesmediatedparticipants’subsequentbehaviorinfuture

interactions.

Mentalandemotionalstatesareintimatelyrelated,andherewediscussedmodeling

inferencesofmentalstatesfromemotionalinformation.Suchworkiscrucialtointegrate

emotionsintoexistingmodelsofTheoryofMind(e.g.,Baker,Saxe,&Tenenbaum,2009;

Bakeretal,2017).Thereexistmanyopenquestionsthatarisefromsuchintegrationof

emotionsandmentalstates:Forexample,therearemanycomplexinteractionsbetween

desiresandemotions:ontheonehand,peoplehavegoalstoregulateemotions,but


conversely,emotionsthemselvesinfluencehowpeopleprioritizetheirexistinggoals.How

dolaypeoplereasonaboutthese,andhowcanresearchersfitthembothintoacausal

model?Theseareimportantpsychologicalquestionsthatshouldbeaddressedinfuture

modelsofsocialandaffectivecognition.

6.Predictions(HypotheticalReasoning)

Layreasoningaboutothers’emotionsisnotlimitedtothehereandnow;Peoplecan

reasonaboutothers’potentialbehaviorgivenafutureorhypotheticalemotion(P(x|e),

P(a|e)).Reasoninghypotheticallyinvolvesusingone’sintuitivetheorytoreasonabout

novelscenariosthatcouldbeinthepossiblefuture,ormerelyintherealmofimagination.

Planningaromanticmarriageproposalinvolvesincessantsimulationofpossibleworlds

andemotionsinthoseworlds.Indeed,experimentsthatpromptparticipantsforemotion

attributionstoafictionalcharacterinavignetteorcartoonallinvokehypothetical

reasoning,andtheeasewithwhichpeopleengageinhypotheticalreasoningiscrucialto

theirenjoymentoffilmandfiction.

Hypotheticalreasoningcanbemodeledbysimplyallowingtheinferredvariablesto

behypothetical,forexample,estimatingthebehaviorgivenahypotheticalemotionP(a|e).

WithinBayesianmodeling,thiscanbeoperationalizedassampling“posteriorpredictives”

underthemodel(i.e.,conditionalpredictionsofnewdata),andhasbeenappliedtomodel

laypredictionsinotherdomains(Griffiths&Tenenbaum,2006),butnotyetinaffective

cognition.Futureworkshouldfocusonapplyingsuchtechniquestomodellaypeople’s

predictionsof(hypothetical)emotions.Suchworkhasimportantimplicationsfordesigning


computationalagentsthathavetosimulateandforecasttheirhumanuser’semotionsand

behavior.

7.CounterfactualsReasoning(andExplanations)

Thefinalcategoryofinferencesinourtaxonomyiscounterfactualreasoning

aboutemotions.Whilethepreviousfewsectionshavetoucheduponreasoningabout

emotionsgivenastateoftheworld,peoplecanalsoreasonaboutothers’emotionsinstates

oftheworldthataredifferentfromtheexistingreality(Gleicheretal,1990;Johnson,

1986).Indeed,emotionslikeregretarecharacterizedbycounterfactualreasoning(Beck&

Crilly,2009).Counterfactualthinking,likehypotheticalreasoning,reliesonarichcausal

modelthattheobservercanmentallymanipulatetoreasonaboutotherpossibleworlds

(Byrne,2002;Roese,1997).WouldCarolnotbesadifshehadnotlostherwallet(P(sad|

notlose-wallet))?WouldCharliehavebeenlesscurtifhehadnotjustlosthisjob?

Reasoningwithcounterfactualsalsoallowsobserverstoprovideexplanations

(Lagnado,Gerstenberg,&Zultan,2013;Halpern&Pearl,2005)forothers’emotionsand

behavior(Malle,1999,2011).Explanationwithinaffectivecognitioninvolvesassigning

causalityofanemotionorabehaviortotheirpossiblecauses(Böhm&Pfister,2015;Ong,

Zaki,&Goodman,2016).WasTim’sdisappointmentafterhisspeechduetohisstuttering

orthedifficultquestionshereceivedafter(orperhaps,both)?Explanationsareimportant

inchoosinginterventionstoregulatetheagent’semotions.IfonethinksthatTimfeltthat

histalkhadnotgonewellbecauseofthedifficultquestionshereceived,onemighttry

helpingTimreappraisethedifficultquestionsasasignalofengagement,notdisdain,from

theaudience.Formingexplanationsandidentifyingthecausesofbehaviorisalso


importantinassigningresponsibility,asinmoralorlegaljudgments(e.g.,Alicke,Mandel,

Hilton,Gerstenberg,&Lagnado,2015),especiallyinlegalsystemswithlayjuries.

Althoughtherehasnotyetbeenanyworkincomputationallymodeling

counterfactualreasoninginaffectivecognition,theuseofprobabilisticcausalmodelsto

representtheintuitivetheoryofemotionsshouldallowanimmediateapplicationof

existingtechniques(e.g.,Pearl,2001,2009).Similarprobabilisticmodelshaverecently

beenusedinmodelinglaycounterfactualreasoninginotherdomains(Lucas&Kemp,

2015;Oaksford&Chater,2010).Weanticipatethatfutureresearchwillsoonapplyformal

techniquestomodelcounterfactualreasoninginaffectivecognition.

Discussion

Inthispaperwehaveoutlinedaframeworkforunderstandingaffectivecognition

asinferencewithinanintuitivetheoryofemotion.Wederivedataxonomyofinferences

(Table1),andwehavediscussedtheseinferencesbothwithrespecttorecentworkin

computationallymodelingaffectivecognition,aswellasfuturechallengesformodeling

eachoftheseinferences.Importantly,thisunifiedframeworkallowsustodescribeallthe

reasoninginTable1usingthesamegeneralprinciples(i.e.,Bayesianinference)appliedto

different“components”ofthecausalmodelinFigure1.Inotherwords,howpeoplereason

fromoutcomestoemotion,orinferemotionfromexpressions,canallbemodeledwiththe

samedomain-generalinferencemachinery,underacommonBayesian“TheoryofEmotion”

(Ong,Zaki,&Goodman,2015;Saxe&Houlihan,2017).

Throughoutthepaper,wehaveposedaffectivecognitionasa“computationallevel”

problem(Marr,1982),andhavefocusedonreviewingprobabilisticapproaches,which


offeranaturalsolutiontosuchinferentialproblems(e.g.,Griffiths,Kemp,&Tenenbaum,

2008;Oaksford&Chater,2007).TheBayesianapproachesinthestudiesdescribedhere

(deMeloetal,2014;Ong,Zaki,&Goodman,2015;Saxe&Houlihan,2017;Wuetal,2018)

sharemorecommonalitiesthandifferences,andwhatdifferencesexistliemainlyinwhich

variablesthedifferentsetsofauthorschosetoprioritizeorsimplifyout.Indeed,Figure1

resultsfromoureffortstounifythecommonideasfromthesestudies---forinstance,every

studymentionedthecrucialroleofthethird-personappraisalprocessanditscausallinkto

emotion.WethinkthatthiscomputationalBayesianframeworkoffersaprincipled

approachtounifyingtheinferencesdiscussed,butitcanalsobesupplementedby

alternative,non-Bayesianapproaches.Forexample,perhapsthecomplexexpression-to-

emotionmappingmightbebestmodeledwithneural-networkapproaches(e.g.,Schindler

etal,2008).Additionally,processmodelsandneuroscientificapproaches(e.g.,Adolphs,

2002)mayshedmorelightonthe"algorithmic"and"implementation"levelsofanalysis.

Inourview,thebiggestchallengeaheadformodelsofaffectivecognitionisfinding

suitablecomputationalrepresentations,especiallyofemotions:Whatisthespaceof

emotionsintheintuitivetheory?Clearly,binaryormultinomial(“angry”vs.not)labelingis

insufficientlycomprehensive,althoughwidelyusedinmanyclassificationtasks(like

sentimentanalysis).Shouldresearchersthenrepresentemotionsinsomehighdimensional

space,andifso,whatarethosedimensions(Barrett&Russell,1998;Mattek,Wolford,&

Whalen,2017)?Howshouldmixedemotionsberepresented?Definingtherepresentation

spaceisacrucialprerequisiteforprobabilisticmodeling,enablingsamplingfromand

marginalizingoverthespaceofemotions(asinEqn.3).


Butevenavectorinsomehigh-dimensionalspacemaystillbeinadequate.

Intuitively,thethreescenarios(a)“Johnisangry”,(b)“Johnisangryatreceivinganegative

outcome”,and(c)“Johnisangryattheunfairprocessthatledtothenegativeoutcome”,are

allqualitativelydifferentbecauseoftheappraisalthatresultedintheemotions,andthe

differentbehavioralconsequencestheyconnote.Thisinsightdemandsaricher,relational

representationofemotionthatincorporatestarget-andevent-relatedinformation

(minimally,“angry_at_X”,“sad_because_Y”),whichcurrentBayesiangraphicalmodelsdo

notsupport---althoughwethinkthatmodernincarnationslikeprobabilisticprogramming

languagesmayofferasolution(e.g.,Goodman,2013,seealsoGoodman&Frank,2016).In

ouropinion,choosingasuitablerepresentationforemotions(andappraisals)thatcaptures

therichnessoflayaffectivecognitionandthatcanbecomputedoverefficientlymayprove

tobethelargesthurdletoovercome.

Finally,wemuststressthatfutureworkmustprioritizemodelingaffectivecognition

onnaturalisticdata.Thesparsestimuli(e.g.,staticfacialexpressions)andexperimental

scenariosinthestudiesdiscussedaboveareanimportantstartingpoint,buttheypalein

comparisontotherichnessofeverydayemotionalexperiences.Itisimportantforfuture

researchtomodelhowlaypeoplereasonabouttheemotionsofothersinnaturalistic

contexts,suchaswatchingsomeoneinanunscriptedmonologue(e.g.,Zaki,Bolger,&

Ochsner,2008;Devlin,Zaki,Ong,&Gruber,2014,2016)ordialogue(e.g.,Ickes,Stinson,

Bissonnette,&Garcia,1990).Modelingnaturalisticcontextsbringsitsownsetof

challenges---forexample,extractingspontaneousemotionalexpressionsandrepresenting

thecomplexappraisalsofnaturalisticsituations---butthereisalsogreatpayoff.First,they

affordreal-worldtestsofthevalidityofcomputationalmodelsdevelopedon“clean”


laboratorystimuli.Second,theywillinformcomputationaltechnologiesthatcanreason

abouttheirusers’emotionsinahuman-likemanner(e.g.,Picard,1998).

Inconclusion,wehopethattheframeworkandtaxonomyofinferenceswepresent

hereprovidesausefulconceptualizationofworkincomputationalmodelsofaffective

cognition.Wehaveidentifiedmanyimportantchallengesthatlieaheadforthisnascent

fieldofresearch,whichweareconfidentwillbeaddressedbyfutureresearchers.Ata

broaderlevel,ourapproachintegrateswellwithstate-of-the-artcomputationalmodelsin

otherdomainsofsocialcognition(Bakeretal,2017;Goodmanetal,2006;Goodman&

Frank,2016;Jara-Ettingeretal,2016)---thisintegrationwillbecrucialforacumulative

scienceofsocialandaffectivereasoning.


Acknowledgements

WethankPatriciaChenforfeedbackonmultipleversionsofthismanuscript.Thiswork

wassupportedinpartbytheA*STARHuman-CentricArtificialIntelligenceProgramme

(SERCSSFProjectNo.A1718g0048),anA*STARNationalScienceScholarshipanda

StanfordIRiSSComputationalSocialScienceFellowshiptoDCO,NIHGrant

1R01MH112560-01toJZ,andaDARPAPPAMLAgreementnumberFA8750-14-2-0009and

SloanFoundationResearchFellowshiptoNDG.


References

Adelson,E.H.,&Bergen,J.R.(1991).Theplenopticfunctionandtheelementsofearlyvision.InM.LandyandJ.A.Movshon(eds),ComputationalModelsofVisualProcessing(pp.3-20).Cambridge,MA:MITPress.

Adolphs,R.(2002).Recognizingemotionfromfacialexpressions:psychologicalandneurologicalmechanisms.BehavioralandCognitiveNeuroscienceReviews,1(1),21-62.

Alicke,M.D.,Mandel,D.R.,Hilton,D.J.,Gerstenberg,T.,&Lagnado,D.A.(2015).Causalconceptionsinsocialexplanationandmoralevaluation:Ahistoricaltour.PerspectivesonPsychologicalScience,10(6),790-812.

Arnold,M.B.(1960).Emotionandpersonality.NewYork:ColumbiaUniversityPress.Aviezer,H.,Hassin,R.R.,Ryan,J.,Grady,C.,Susskind,J.,Anderson,A.,Moscovitch,M.,&

Bentin,S.(2008).Angry,disgusted,orafraid?studiesonthemalleabilityofemotionperception.PsychologicalScience,19(7),724–732.

Aviezer,H.,Trope,Y.,&Todorov,A.(2012).Bodycues,notfacialexpressions,discriminatebetweenintensepositiveandnegativeemotions.Science,338(6111),1225-1229.

Baker,C.L.,Saxe,R.,&Tenenbaum,J.B.(2009).Actionunderstandingasinverseplanning.Cognition,113(3),329-349.

Baker,C.L.,Jara-Ettinger,J.,Saxe,R.,&Tenenbaum,J.B.(2017).Rationalquantitativeattributionofbeliefs,desiresandperceptsinhumanmentalizing.NatureHumanBehaviour,1,0064.

Banse,R.,&Scherer,K.R.(1996).Acousticprofilesinvocalemotionexpression.JournalofPersonalityandSocialPsychology,70(3),614.

Barrett,L.,&Russell,J.A.(1998).Independenceandbipolarityinthestructureofcurrentaffect.JournalofPersonalityandSocialPsychology,74(4),967.

Beck,S.R.,&Crilly,M.(2009).Isunderstandingregretdependentondevelopmentsincounterfactualthinking?.BritishJournalofDevelopmentalPsychology,27(2),505-510.

Böhm,G.,&Pfister,H.R.(2015).Howpeopleexplaintheirownandothers’behavior:atheoryoflaycausalexplanations.FrontiersinPsychology,6,139.

Buck,R.(1994).Socialandemotionalfunctionsinfacialexpressionandcommunication:Thereadouthypothesis.BiologicalPsychology,38(2),95–115.

Carey,S.(2009).TheOriginofConcepts.OxfordUniversityPress.Carroll,J.M.,&Russell,J.A.(1996).Dofacialexpressionssignalspecificemotions?Judging

emotionfromthefaceincontext.JournalofPersonalityandSocialPsychology,70(2),205.

Dael,N.,Mortillaro,M.,&Scherer,K.R.(2012).Emotionexpressioninbodyactionandposture.Emotion,12(5),1085.

deMelo,C.M.,Carnevale,P.J.,Read,S.J.,&Gratch,J.(2014).Readingpeople’smindsfromemotionexpressionsininterdependentdecisionmaking.JournalofPersonalityandSocialPsychology,106(1),73.

Delis,I.,Chen,C.,Jack,R.E.,Garrod,O.G.,Panzeri,S.,&Schyns,P.G.(2016).Space-by-timemanifoldrepresentationofdynamicfacialexpressionsforemotioncategorization.JournalofVision,16(8),14-14.


Devlin,H.C.,Zaki,J.,Ong,D.C.,&Gruber,J.(2014).Notasgoodasyouthink?Traitpositiveemotionisassociatedwithincreasedself-reportedempathybutdecreasedempathicperformance.PloSone,9(10),e110470.

Devlin,H.C.,Zaki,J.,Ong,D.C.,&Gruber,J.(2016).Trackingtheemotionalhighsbutmissingthelows:Hypomaniariskisassociatedwithpositivelybiasedempathicinference.CognitiveTherapyandResearch,40(1),72-79.

Ekman,P.,Friesen,W.V.,&Ellsworth,P.(1972).Emotioninthehumanface:Guide-linesforresearchandanintegrationoffindings.NewYork:PergamonPress.

Ekman,P.,&Friesen,W.V.(1978).FacialActionCodingSystem.PaloAlto,CA:ConsultingPsychologistsPress.

Elfenbein,H.A.,&Ambady,N.(2002).Ontheuniversalityandculturalspecificityofemotionrecognition:ameta-analysis.PsychologicalBulletin,128(2),203.

Ellsworth,P.C.,&Scherer,K.R.(2003).Appraisalprocessesinemotion.InR.J.Davidson,H.Goldsmith,&K.R.Scherer(Eds.),HandbookofAffectiveSciences(pp.572–595).NewYork,NY:OxfordUniversityPress.

Frank,M.C.,&Goodman,N.D.(2012).Predictingpragmaticreasoninginlanguagegames.Science,336(6084),998-998.

Frijda,N.H.,Kuipers,P.,&TerSchure,E.(1989).Relationsamongemotion,appraisal,andemotionalactionreadiness.JournalofPersonalityandSocialPsychology,57(2),212.

Gendron,M.,Roberson,D.,vanderVyver,J.M.,&Barrett,L.F.(2014).Perceptionsofemotionfromfacialexpressionsarenotculturallyuniversal:evidencefromaremoteculture.Emotion,14(2),251.

Gerstenberg,T.&Tenenbaum,J.B.(2017).Intuitivetheories.InM.Waldmannn(Ed.),OxfordHandbookofCausalReasoning(pp.515-548).OxfordUniversityPress.

Goodenough,F.L.,&Tinker,M.A.(1931).Therelativepotencyoffacialexpressionandverbaldescriptionofstimulusinthejudgmentofemotion.JournalofComparativePsychology,12(4),365.

Goodman,N.D.(2013).Theprinciplesandpracticeofprobabilisticprogramming.ACMSIGPLANNotices,48(1),399-402.

Goodman,N.D.,Baker,C.L.,Bonawitz,E.B.,Mansinghka,V.K.,Gopnik,A.,Wellman,H.,Schulz,L.&Tenenbaum,J.B.(2006).Intuitivetheoriesofmind:Arationalapproachtofalsebelief.InProceedingsoftheTwenty-EighthAnnualConferenceoftheCognitiveScienceSociety(pp.1382-1387).

Goodman,N.D.,&Frank,M.C.(2016).Pragmaticlanguageinterpretationasprobabilisticinference.TrendsinCognitiveSciences,20(11),818-829.

Goodman,N.D.,&Stuhlmüller,A.(2013).Knowledgeandimplicature:Modelinglanguageunderstandingassocialcognition.TopicsinCognitiveScience,5(1),173-184.

Gopnik,A.,&Meltzoff,A.N.(1997).Words,thoughts,andtheories.TheMITPress.Griffiths,T.L.(2017).Formalizingpriorknowledgeincausalinduction.InM.Waldmann,

(Ed),TheOxfordHandbookofCausalReasoning.Oxford:OxfordUniversityPressGriffiths,T.L.,Kemp,C.,&Tenenbaum,J.B.(2008).Bayesianmodelsofcognition.InRon

Sun(ed.),TheCambridgeHandbookofComputationalCognitiveModeling.CambridgeUniversityPress.

Griffiths,T.L.,&Tenenbaum,J.B.(2006).Optimalpredictionsineverydaycognition.PsychologicalScience,17(9),767-773.


Gross,A.L.,&Ballif,B.(1991).Children'sunderstandingofemotionfromfacialexpressionsandsituations:Areview.DevelopmentalReview,11(4),368-398.

Halpern,J.Y.,&Pearl,J.(2005).Causesandexplanations:Astructural-modelapproach.PartII:Explanations.TheBritishJournalforthePhilosophyofScience,56(4),889-911.

Hareli,S.,&Hess,U.(2010).Whatemotionalreactionscantellusaboutthenatureofothers:Anappraisalperspectiveonpersonperception.CognitionandEmotion,24(1),128-140.

Harris,P.L.(1989).Childrenandemotion:Thedevelopmentofpsychologicalunderstanding.BasilBlackwell.

Heider,F.(1958).Thepsychologyofinterpersonalrelations.NewYork:Wiley.Ickes,W.,Stinson,L.,Bissonnette,V.,&Garcia,S.(1990).Naturalisticsocialcognition:

Empathicaccuracyinmixed-sexdyads.JournalofPersonalityandSocialPsychology,59(4),730.

Isen,A.M.(1987).Positiveaffect,cognitiveprocesses,andsocialbehavior.AdvancesinExperimentalSocialPsychology,20,203-253.

Jack,R.E.,Blais,C.,Scheepers,C.,Schyns,P.G.,&Caldara,R.(2009).Culturalconfusionsshowthatfacialexpressionsarenotuniversal.CurrentBiology,19(18),1543-1548.

Jack,R.E.,Caldara,R.,&Schyns,P.G.(2012).Internalrepresentationsrevealculturaldiversityinexpectationsoffacialexpressionsofemotion.JournalofExperimentalPsychology:General,141(1),19.

Jara-Ettinger,J.,Gweon,H.,Schulz,L.E.,&Tenenbaum,J.B.(2016).Thenaïveutilitycalculus:computationalprinciplesunderlyingcommonsensepsychology.TrendsinCognitiveSciences,20(8),589-604.

Jern,A.,&Kemp,C.(2015).Adecisionnetworkaccountofreasoningaboutotherpeople’schoices.Cognition,142,12-38.

Johnstone,T.,&Scherer,K.R.(2000).Vocalcommunicationofemotion.HandbookofEmotions,2,220-235.

Kahneman,D.,&Tversky,A.(1979).Prospecttheory:Ananalysisofdecisionunderrisk.Econometrica,47(2),263–292.

Kahneman,D.,&Tversky,A.(1982).Thesimulationheuristic.InD.Kahneman,P.Slovic,&A.Tversky(Eds.),Judgmentunderuncertainty:Heuristicsandbiases(p.201-208).CambridgeUniversityPress

Kemp,C.,&Jern,A.(2014).Ataxonomyofinductiveproblems.PsychonomicBulletin&Review,21(1),23-46.

Kermer,D.A.,Driver-Linn,E.,Wilson,T.D.,&Gilbert,D.T.(2006).Lossaversionisanaffectiveforecastingerror.PsychologicalScience,17(8),649-653.

Koller,D.,&Friedman,N.(2009).Probabilisticgraphicalmodels:principlesandtechniques.MITpress.

Krumhuber,E.,Manstead,A.S.,Cosker,D.,Marshall,D.,Rosin,P.L.,&Kappas,A.(2007).Facialdynamicsasindicatorsoftrustworthinessandcooperativebehavior.Emotion,7(4),730.

Lagattuta,K.H.,Wellman,H.M.,&Flavell,J.H.(1997).Preschoolers'understandingofthelinkbetweenthinkingandfeeling:Cognitivecuingandemotionalchange.ChildDevelopment,68(6),1081-1104.


Lagnado,D.A.,Gerstenberg,T.,&Zultan,R.I.(2013).Causalresponsibilityandcounterfactuals.CognitiveScience,37(6),1036-1073.

Loewenstein,G.,&Lerner,J.S.(2003).Theroleofaffectindecisionmaking.HandbookofAffectiveScience,619(642),3.

Lerner,J.S.,Li,Y.,Valdesolo,P.,&Kassam,K.S.(2015).Emotionanddecisionmaking.AnnualReviewofPsychology,66,799-823.

Levine,E.E.,Barasch,A.,Rand,D.G.,Berman,J.Z.,&Small,D.A.(2018).SignalingEmotionandReasoninCooperation.JournalofExperimentalPsychology:General,147(5),702.

Lucas,C.G.,&Kemp,C.(2015).Animprovedprobabilisticaccountofcounterfactualreasoning.PsychologicalReview.122(4),700-734.

Malle,B.F.(1999).Howpeopleexplainbehavior:Anewtheoreticalframework.PersonalityandSocialPsychologyReview,3(1),23–48.

Malle,B.F.(2011).Attributiontheories:Howpeoplemakesenseofbehavior.InTheoriesinSocialPsychology(pp.72–95).Wiley-BlackwellMalden,MA.

Marr,D.(1982).Vision.SanFrancisco:W.H.Freeman.Martinez,A.,&Du,S.(2012).Amodeloftheperceptionoffacialexpressionsofemotionby

humans:Researchoverviewandperspectives.JournalofMachineLearningResearch,13,1589-1608.

Mattek,A.M.,Wolford,G.L.,&Whalen,P.J.(2017).Amathematicalmodelcapturesthestructureofsubjectiveaffect.PerspectivesonPsychologicalScience,12(3),508-526.

Mellers,B.A.,Schwartz,A.,Ho,K.,&Ritov,I.(1997).Decisionaffecttheory:Emotionalreactionstotheoutcomesofriskyoptions.PsychologicalScience,8(6),423-429.

Moors,A.,Ellsworth,P.C.,Scherer,K.R.,&Frijda,N.H.(2013).Appraisaltheoriesofemotion:Stateoftheartandfuturedevelopment.EmotionReview,5(2),119-124.

Oaksford,M.,&Chater,N.(2007).Bayesianrationality:Theprobabilisticapproachtohumanreasoning.OxfordUniversityPress,USA.

Oaksford,M.,&Chater,N.(2010).Causationandconditionalsinthecognitivescienceofhumanreasoning.OpenPsychologyJournal,3,105–118.

Ong,D.C.,Asaba,M.,&Gweon,H.(2016).Youngchildrenandadultsintegratepastexpectationsandcurrentoutcomestoreasonaboutothers'emotions.InProceedingsofthe38thAnnualMeetingoftheCognitiveScienceSociety,135-140.

Ong,D.C.,Goodman,N.D.,&Zaki,J.(2015).Near-missesstingevenwhentheyareuncontrollable.InProceedingsofthe37thAnnualMeetingoftheCognitiveScienceSociety,1775-1780.

Ong,D.C.,Zaki,J.,&Goodman,N.D.(2015).Affectivecognition:Exploringlaytheoriesofemotion.Cognition,143,141-162.

Ong,D.C.,Zaki,J.&Goodman,N.D.(2016).Emotionsinlayexplanationsofbehavior.Proceedingsofthe38thAnnualMeetingoftheCognitiveScienceSociety,360-365.

Ortony,A.,Clore,G.L.,&Collins,A.(1988).Thecognitivestructureofemotions.NewYork:CambridgeUniversityPress.

Pearl,J.(2001).Directandindirecteffects.InProceedingsoftheSeventeenthConferenceonUncertaintyinArtificialIntelligence,411-420.

Pearl,J.(2009).Causality:Models,Reasoning,andInference.CambridgeUniversityPress.


Repacholi,B.M.,&Gopnik,A.(1997).Earlyreasoningaboutdesires:evidencefrom14-and18-month-olds.DevelopmentalPsychology,33(1),12.

Repacholi,B.M.,Meltzoff,A.N.,Hennings,T.M.,&Ruba,A.L.(2016).TransferofSocialLearningAcrossContexts:ExploringInfants'AttributionofTrait-LikeEmotionstoAdults.Infancy,21,785--806.doi:10.1111/infa.12136

Ross,L.(1977).Theintuitivepsychologistandhisshortcomings:Distortionsintheattributionprocess.AdvancesinExperimentalSocialPsychology,10,173-220.

Russell,J.A.(1994).Isthereuniversalrecognitionofemotionfromfacialexpression?Areviewofthecross-culturalstudies.PsychologicalBulletin,115(1),102.

Russell,J.A.,Bachorowski,J.A.,&Fernández-Dols,J.M.(2003).Facialandvocalexpressionsofemotion.AnnualReviewofPsychology,54(1),329-349.

Saxe,R.,&Houlihan,S.D.(2017).FormalizingemotionconceptswithinaBayesianmodeloftheoryofmind.CurrentOpinioninPsychology.

Scherer,K.R.(2003).Vocalcommunicationofemotion:Areviewofresearchparadigms.SpeechCommunication,40(1),227-256.

Scherer,K.R.,&Grandjean,D.(2008).Facialexpressionsallowinferenceofbothemotionsandtheircomponents.CognitionandEmotion,22(5),789-801.

Schindler,K.,VanGool,L.,&deGelder,B.(2008).Recognizingemotionsexpressedbybodypose:Abiologicallyinspiredneuralmodel.NeuralNetworks,21(9),1238-1246.

Skerry,A.E.,&Saxe,R.(2014).Acommonneuralcodeforperceivedandinferredemotion.JournalofNeuroscience,34(48),15997-16008.

Skerry,A.E.,&Saxe,R.(2015).Neuralrepresentationsofemotionareorganizedaroundabstracteventfeatures.CurrentBiology,25(15),1945-1954.

Skerry,A.E.,&Spelke,E.S.(2014).Preverbalinfantsidentifyemotionalreactionsthatareincongruentwithgoaloutcomes.Cognition,130(2),204-216.

Smith,C.A.,&Lazarus,R.S.(1993).Appraisalcomponents,corerelationalthemes,andtheemotions.Cognition&Emotion,7(3-4),233-269.

Sutton,R.S.,&Barto,A.G.(1998).Reinforcementlearning:Anintroduction.Cambridge:MITpress.

VanKleef,G.A.,DeDreu,C.K.,&Manstead,A.S.(2010).Aninterpersonalapproachtoemotioninsocialdecisionmaking:Theemotionsassocialinformationmodel.AdvancesinExperimentalSocialPsychology,42,45-96.

Wellman,H.M.,&Banerjee,M.(1991).Mindandemotion:Children'sunderstandingoftheemotionalconsequencesofbeliefsanddesires.BritishJournalofDevelopmentalPsychology,9(2),191-214.

Wellman,H.M.,&Bartsch,K.(1988).Youngchildren’sreasoningaboutbeliefs.Cognition,30(3),239-277.

Wellman,H.M.,&Gelman,S.A.(1992).Cognitivedevelopment:Foundationaltheoriesofcoredomains.AnnualReviewofPsychology,43(1),337-375.

Wellman,H.M.,Phillips,A.T.,&Rodriguez,T.(2000).Youngchildren'sunderstandingofperception,desire,andemotion.Childdevelopment,71(4),895-912.

Wenzler,S.,Levine,S.,vanDick,R.,Oertel-Knöchel,V.,&Aviezer,H.(2016).Beyondpleasureandpain:Facialexpressionambiguityinadultsandchildrenduringintensesituations.Emotion,16(6),807-814.

Wondra,J.D.,&Ellsworth,P.C.(2015).Anappraisaltheoryofempathyandothervicariousemotionalexperiences.PsychologicalReview,122(3),411.


Wu,Y.,Baker,C.L.,Tenenbaum,J.B.,&Schulz,L.(2018).Rationalinferenceofbeliefsanddesiresfromemotionalexpressions.CognitiveScience.42:850-884.doi:10.1111/cogs.12548

Wu,Y.,Muentener,P.,&Schulz,L.(2017).One-tofour-year-oldsconnectdiversepositiveemotionalvocalizationstotheirprobablecauses.ProceedingsoftheNationalAcademyofSciences.doi:10.1073/pnas.1707715114

Wu,Y.,&Schulz,L.(2017).Inferringbeliefsanddesiresfromemotionalreactionstoanticipatedandobservedevents.ChildDevelopment.doi:10.1111/cdev.12759

Yuki,M.,Maddux,W.W.,&Masuda,T.(2007).ArethewindowstothesoulthesameintheEastandWest?CulturaldifferencesinusingtheeyesandmouthascuestorecognizeemotionsinJapanandtheUnitedStates.JournalofExperimentalSocialPsychology,43(2),303-311.

Zaki,J.,Bolger,N.,&Ochsner,K.(2008).Ittakestwo:Theinterpersonalnatureofempathicaccuracy.PsychologicalScience,19(4),399-404.

Zaki,J.,&Ochsner,K.(2011).Reintegratingaccuracyintothestudyofsocialcognition(TargetArticle).PsychologicalInquiry,22(3),159-182.

computational models of emotion inference in theory of ... · aa*star artificial intelligence...

Documents