Evolution of social norms and correlated equilibriaBryce Morskya,1 and Erol Akcaya

aDepartment of Biology, University of Pennsylvania, Philadelphia, PA 19104

Edited by Brian Skyrms, University of California, Irvine, CA, and approved March 20, 2019 (received for review October 7, 2018)

Social norms regulate and coordinate most aspects of humansocial life, yet they emerge and change as a result of individualbehaviors, beliefs, and expectations. A satisfactory account forthe evolutionary dynamics of social norms, therefore, has to linkindividual beliefs and expectations to population-level dynamics,where individual norms change according to their consequencesfor individuals. Here, we present a model of evolutionary dynam-ics of social norms that encompasses this objective and addressesthe emergence of social norms. In this model, a norm is a set ofbehavioral prescriptions and a set of environmental descriptionsthat describe the expected behaviors of those with whom thenorm holder will interact. These prescriptions and descriptions arefunctions of exogenous environmental events. These events haveno intrinsic meaning or effect on the payoffs to individuals, yetbeliefs/superstitions regarding them can effectuate coordination.Although a norm’s prescriptions and descriptions are dependenton one another, we show how they emerge from random accu-mulations of beliefs. We categorize the space of social norms intoseveral natural classes and study the evolutionary competitionbetween these classes of norms. We apply our model to the Gameof Chicken and the Nash Bargaining Game. Furthermore, we showhow the space of norms and evolutionary stability are dependenton the correlation structure of the environment and under whichsuch correlation structures social dilemmas can be ameliorated orexacerbated.

social norms | correlated equilibrium | cooperation | coordination |superstitions

Cooperation and coordination in social settings are funda-mental to the functioning of human and animal communi-

ties. Considering purely self-interested actors, cooperation andcoordination should be rare if there are conflicts of interestsbetween players and if no external incentives (or coercion) areat hand to resolve these conflicts. Yet, we observe many exam-ples of cooperation without external incentives mediated byinformal mechanisms, such as social norms (1–3). It is appar-ent that, while humans and animals do tend to make decisionsto improve their payoffs, individual decision making is perva-sively affected by both prescriptive beliefs on their own behaviorand expectations of others’ behavior (4). These prescriptionsand expectations may or may not be rooted in reality or accu-rately describe others’ behaviors, but regardless, they have a realeffect on behavior (5–8). Importantly, they can induce regular-ities in a population that allow self-interested agents to coop-erate and coordinate with each other efficiently. These beliefscan endow events and cues in the world that are inherentlymeaningless with normative meaning to voluntarily coordinateindividual behavior. We address how social norms that giverise to such beliefs can arise through an evolutionary gametheory model that combines rationality, superstition, and evo-lutionary dynamics to coordinate behaviors on external eventswithout design.

Although social norms coordinate behavior, any individual orthe society at large generally does not explicitly design them;rather, they are a collective phenomenon that emerges from theinteraction of individuals. Thus, game theory is a common toolto model the emergence of social norms (9–11). Many differentgame theoretic conceptions of social norms exist. Some normscan be seen as mechanisms that select from (the typically numer-

ous) Nash equilibria in repeated games (12). Alternatively, somesocial norms consist of conditional behavioral rules (3) that con-vert social dilemmas into coordination games: for example, byexternal enforcement (either through institutions or by others ina group) (13, 14). Our focus here is on social norms that lackexternal enforcement but still are followed, because agents, giventheir beliefs about the world (also induced by social norms), findit in their interest to do so. In other words, we are interestedin both descriptive norms that tell individuals what behaviors toexpect from others and prescriptive norms that tell individualshow to behave and can be predicated on descriptive norms (15).More specifically, we model social norms that act as “choreogra-phers” (16, 17) that induce correlated beliefs in agents, allowingthem to coordinate on a correlated equilibrium (18, 19) ofthe game.

A correlated equilibrium is generated by a correlating device[the choreographer (16, 17)] that suggests pure strategies to theplayers. If the suggestions to each player are correlated in a cer-tain way and the agents know these correlations, it can be optimalfor the players to follow the suggestions of the choreographer.Such behavior is called a correlated equilibrium. The Game ofChicken is an exemplar for the coordinating effects of corre-lated equilibria, as it can feature a scenario in which coordinationwould lead to a socially optimal yet game-theoretically unsta-ble state. Consider two cars approaching an intersection: onenorthbound and the other westbound. The drivers may choose todrive through the intersection or stop. The payoff matrix for thisgame is

Π=

stop go( )6 2 stop7 0 go

, [1]

where πij ∈Π is the payoff to a player playing the row strategyvs. an opponent playing the column strategy. The cooperativestrategy is to stop, which is to “chicken out,” and both players

Significance

We consider social norms as collections of beliefs or super-stitions about events occurring in nature. These events haveno inherent meaning, but individuals can choose to believethat they do. Specifically, individuals can interpret eventsas prescribing behaviors for themselves along with expecta-tions of the behaviors of others. We show how evolutionarycompetition between such beliefs can give rise to both pre-scriptive and descriptive self-enforcing norms that can allowpopulations to coordinate behavior. Our model provides anevolutionary account for how normative meaning emerges inan inherently meaningless world.

Author contributions: B.M. and E.A. designed research, performed research, analyzeddata, and wrote the paper.y

The authors declare no conflict of interest.y

This article is a PNAS Direct Submission.y

Published under the PNAS license.y1 To whom correspondence should be addressed. Email: morsky@sas.upenn.edu.y

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1817095116/-/DCSupplemental.y

Published online April 11, 2019.

8834–8839 | PNAS | April 30, 2019 | vol. 116 | no. 18 www.pnas.org/cgi/doi/10.1073/pnas.1817095116

Dow

nloa

ded

by g

uest

on

July

4, 2

020

Dow

nloa

ded

by g

uest

on

July

4, 2

020

Dow

nloa

ded

by g

uest

on

July

4, 2

020

ECO

NO

MIC

SCIE

NCE

SEV

OLU

TIO

N

stopping is the socially optimal state. The problem is that thisis not a Nash equilibrium of the game. Yet, if somehow aconvention emerges where the westbound car has the right ofway, it can coordinate interactions and avoid collisions. Thisconvention is analogous to the Bourgeois Strategy of evolu-tionary game theory (8, 20), where a norm of respecting cur-rent ownership of a resource may coordinate fighting over it.Another analogous example is when animals use conspicuousnatural landmarks in the environment as focal points for ter-ritory boundaries (21, 22). In all of these cases, an event orcue in the world that has no inherent meaning for the inter-action ends up having normative meaning, because individu-als in the population conventionally condition their behavioron it.

In the classical conception of correlated equilibrium, thereis a third party, the choreographer, that assigns the mean-ings to the cues and decides on the correlations between thesuggestions, which then are assumed to be common knowl-edge between individuals. The city putting a traffic light atour intersection would be an example of this. Here, we areconcerned with the case where there is no central party thatdecides on the normative meaning, and players do not knowcorrelations between cues. Instead, we ask how prescriptiveand descriptive individual beliefs about events in the worldcompete against each other decentrally. In effect, we askwhether natural selection can serve as a “blind choreogra-pher” that shapes individuals’ beliefs about stochastic eventsin the world such that they play correlated equilibria withoutknowledge of the correlation structure of the world or externalenforcement.

Our work belongs to a body of game theory that has stud-ied how correlated equilibria can emerge in a decentralized waythrough evolutionary or learning dynamics (23–32). The firststrand in this body starts with the seminal paper of Selten (23)who proved that evolutionarily stable strategies conditional oncues in the world have to be pure strategies. Later, Cripps (24)and Shmida and Peleg (25) showed that these correspond tocorrelated equilibria of the underlying game (31). Building onthese results, Metzger (32) modeled the evolution of the cuedistribution itself through a centralized mechanism where allplayers have to agree to change the distribution of signals. Asecond strand of literature (26–30) is concerned with learningin repeated games, where the history of the interaction (26–28) or selection of players (29) or an exogenous correlationdevice (30) acts as the choreographer, and it shows that vari-ous learning dynamics can converge to correlated equilibria afterrepeated play.

Our paper is closest in spirit to the first strand. We con-sider nature as providing a collection of correlated events in theworld and ask how behavioral strategies conditional on theseevent spaces might evolve. Our main contribution is that weexplicitly consider prescriptive and descriptive beliefs that indi-viduals might impute from such events. We model the evolutionof such beliefs under the assumption that, given their beliefs,agents make individually rational decisions. This is analogousto the “indirect evolution” approach to the evolution of utilityfunctions (33–36). In this case, we model how evolution drivesindividual behavior indirectly by shaping normative beliefs in apopulation. Our approach is aimed at explaining not only normsas correlated behaviors but also, the belief structures that sup-port them in the absence of central authorities that imposeor justify these beliefs. We show how belief structures aboutevents in the world can emerge and be both individually ratio-nal and evolutionarily stable, although the events are inherentlymeaningless to the interaction. We are particularly interestedin this phenomenon with respect to noncooperative games thatfeature a social dilemma where the socially optimal solutionis evolutionarily unstable. As illustrative examples, we apply

our general framework to the Game of Chicken and the NashBargaining Game.

ModelConsider two players meeting to play a game. Each player pri-vately observes an event, which is irrelevant to the game beingplayed but may be correlated with their opponent’s event andother events in ways unknown to the players. We consider socialnorms that assign to these events a set of labels, prescriptions,and descriptions of opponent behaviors.

Players can assign a private label to these events to differ-entiate between them. The events and labels have no inherentassociation with one another. As a function of the observed label,players’ social norms prescribe a pure strategy recommendation.We assume that players cannot observe the events that theiropponents see or the prescriptions that they may have received.Instead, each players’ norm gives them a (private) conditionalprobability distribution of prescriptions that their opponent maybe receiving given the recommendation that they themselveshave been given. This conditional probability distribution mayor may not correspond to the actual conditional distribution ofprescriptions or behavior of a real opponent. It simply repre-sents a player’s subjective expectation given by its social norm.The players then may obey the norm’s prescription or play adefault strategy from which they will earn a payoff. The pre-scription of the norm is only obeyed if it represents a bestresponse to the description of the opponent behavior given bythe norm.

Fig. 1 illustrates our model graphically in a world with fourevents and two players having differing norms. An event is cho-sen where each player observes a cat. In this case, Player 1observes the black cat, and Player 2 observes the white cat. Player1’s norm prescribes cooperation when it observes black or whitecats and defection for gray and tabby ones. On the descrip-tive side, Player 1’s norm specifies a conditional distribution ofbehaviors of Player 2. Since this distribution is conditional on thelabels, Player 1’s expectation of Player 2’s behavior is the samewhen observing black and white cats. Observing a black or whitecat, Player 1 believes that Player 2 will cooperate with probabil-ity 1/2 and defect otherwise. Notice that a norm’s prescriptionscan be different from a description of those prescriptions. Thisdiscord is illustrated by Player 2’s norm, which could well be

Fig. 1. A pictorial representation of the model. Here, each vertex on thegraph corresponds to the observation of a different colored cat. As an exam-ple, Player 1 and Player 2 meet, and they observe a black (B) cat and a white(W) cat, respectively. Player 1’s norm recommends that they cooperate (rep-resented in white) and given that Player 1 has been told to cooperate, claimsthat Player 2 will cooperate with probability 1/2. Player 2’s norm recommendsthat they defect (represented in black) and claims that Player 1 will alwayscooperate. G, gray; T, tabby.

Morsky and Akcay PNAS | April 30, 2019 | vol. 116 | no. 18 | 8835

Dow

nloa

ded

by g

uest

on

July

4, 2

020

described as impudent; they believe that they should defectregardless of which cat they see and expect their opponent toalways cooperate.

Mathematically, we model the world of events as an undirectedgraph, G =(V ,E), where the edges, {vi , vj}∈E , representinteractions between two players via a game and the verticesprivately observed events for those games. This construction isessentially equivalent to specifying a join probability distribu-tion as in refs. 18, 19, and 23. However, we separate playerinteractions and events so that events may be aggregated underparticular labels, allowing changes in the observed (i.e., label)distribution. Players then play each other a large number of timessuch that they observe all events. When a game is played, eachplayer is equally likely to be privately assigned either vertex of theedge (this model may be extended to incorporate vertex assign-ment dependent on “types,” although we do not consider thiscase here). In the event that the edge is a self-loop, both play-ers are assigned the same vertex (although this is still privatelyobserved). The probability of selecting edge {vi , vj} is aij ∈A,a weighted symmetric adjacency matrix. We limit the numberof ways in which a player can distinguish vertices by defininga set of labels, L, to apply to each vertex. If two vertices havethe same label, then the player cannot differentiate betweenthe two. In matrix notation, we define this as follows. Given aset of labels L, let L be the |L| × |V | label matrix, which par-titions the vertices into |L| labels (i.e., lij =1 if vj has label li ;otherwise, lij =0).

Norms prescribe pure strategy recommendations to labels.We can represent the behaviors that players are prescribed bythe |S | × |L| prescriptive behavior matrix, P, where pij =1 ifthe norm prescribes the player to play si given lj . If |L|= |V |,players can be prescribed a particular strategy for each vertexindividually. In the case where the norm has no prescriptionfor a label, the players play a default strategy, such as a sym-metric (mixed) Nash equilibrium. The expected behavior ofan opponent is determined by the column stochastic |S | × |L|descriptive behavior matrix, D, which is the expected behav-ior of an opponent given a label observed by the focal player.Given these constructions, a norm is defined by the matrixtriplet (L,P,D).

We examine the effects of a variety of competing norms ona given event graph. In playing an opponent with a differingnorm, a player will still follow its own norm and thus, will notnecessarily receive the expected payoff that their norm woulddictate. Norm change can be facilitated by such disparity betweenexpectations generated by descriptive norms and reality (37).However, we assume that players do not reflect on—or at leastdo not do so as to modify—their norms directly but can observethe fitness of competing norms. Thus, norm change is gener-ated by imitation dynamics, whereby players imitate the normsof those with higher fitness (38). The expected payoff of fol-lowing norm n using P playing norm n ′ with P′ is a functionof the prescriptions for each player at each interaction multi-plied by the payoff from that strategy pairing. Mathematically,this is

E[πnn′ ] = tr(ΠP′L′ALTPT ). [2]

The payoff playing the whole population is thus the sum of thisexpected payoff over all norms weighted by their frequency. Notethat this expected payoff does not directly depend on D, thedescriptive behavior matrix, since D only describes the beliefsthat a focal individual has about the behavior of an opponent, notthe actual behaviors. Indirectly, D can affect the payoff, since itcan determine whether a focal individual obeys the prescriptivenorm P or not.

Applying this equation to the example in Fig. 1 with equallyweighted edges, we have

E[πnn′ ] = tr

[6 27 0

][0 01 1

][1 1 0 00 0 1 1

]0 1

60 0

16

0 16

00 1

60 1

6

0 0 16

0

1 00 10 11 0

[0 11 0

]=tr

([43

23

0 0

])=

4

3. [3]

ResultsA Classification of Norms. Our definition of social norms cap-tures the idea that norms imbue an inherently meaningless worldwith prescriptions on self-behavior and expectations about oth-ers’ behaviors. However, we do not assume that the prescriptionsof a social norm are to be blindly obeyed. In particular, weassume that a norm will only be obeyed when—given the expec-tations from the descriptive aspect of the norm—individualscannot improve their payoffs by deviating from the social norm’sprescriptions. In this section, we will first define the rationalitycondition and then, classify several important norms (depictedas a Venn diagram in Fig. 2). In Evolutionary Stability of Norms,we will address evolutionary concerns.

Let (L,P,D) be a social norm. The expected payoff for obeyingthe prescription at label li is (ΠD)∗i · p∗i . It is rational for playersto obey their norm’s prescriptions if each prescription at a label isa best response to an opponent’s expected behavior at that label:that is,

(ΠD)∗i · (p∗i − p∗i)≥ 0 [4]

for every label li and prescription p∗i 6= p∗i . If a norm does notobey [4], then we assume that players who have it will disregardboth the recommended prescriptions and the descriptions to playa default strategy at all events (SI Appendix, section 2 has a dis-cussion of alternative assumptions for this case). We will call suchnorms null norms. If the game features a dominant strategy, weassume that this is the default. However, if the game features asymmetric mixed Nash equilibrium, we assume that that is thedefault. In the case of multiple such strategies, we have a familyof null norms.

Although the descriptive aspect of a norm does not haveto accurately describe opponents’ behavior, there exists a classof norms that does fit the behavior of (possibly hypothetical)opponents. In particular, a norm is empirically validatable ifits prescriptions are rational against the behavior of a ratio-nal norm. This class is of interest, since two norms that are

Fig. 2. A Venn diagram of the set of norms. All norms can be partitionedinto null and rational norms. Empirical norms can be partitioned into con-sistent (which implement a correlated equilibrium when playing itself) andinconsistent norms. Inconsistent norms may implement a correlated equi-librium with complementary inconsistent norms but not with themselves.Best-response norms (BR) are a subset of evolutionary stable norms (ES).

8836 | www.pnas.org/cgi/doi/10.1073/pnas.1817095116 Morsky and Akcay

Dow

nloa

ded

by g

uest

on

July

4, 2

020

ECO

NO

MIC

SCIE

NCE

SEV

OLU

TIO

N

empirically validatable with respect to one another implement acorrelated equilibrium when they play one another (SI Appendix,Lemma S1).

We are also interested in a particular class of norms whereeveryone would obey their norm given that everyone else alsoobeys the same norm (39). The prescriptive part of such normsmust be rational vs. the behavior of opponents that follow theprescriptions of the same norm. In other words, the norm isempirically validated by itself. We call these norms consistentnorms. In contrast, the prescriptive part of an inconsistent normdoes not follow from a description of its own behavior but rather,is rational against a player obeying a different set of prescrip-tions. For example, consider the event pair {black cat, white cat},in which the two cats are completely correlated with one another.Suppose that, when a black cat is observed, a norm prescribesdefection and expects cooperation from the opponent. If thisnorm is consistent, it will prescribe cooperation and expect defec-tion when a white cat is observed. However, an inconsistent normcan hypocritically prescribe its carrier to also defect and expectcooperation in the event that the carrier sees a white cat. Math-ematically, if a norm is empirically validatable for D′=PLALT,then the social norm is consistent. Note that, if a norm is consis-tent, it necessarily implements a correlated equilibrium playingagainst itself. However, if a norm is empirically validatable butis not rational for D′ =PLALT, then the norm is inconsistent.The left-hand norm in Fig. 1 is consistent, and the right-handnorm in Fig. 1 is inconsistent. It is worth a reminder that play-ers do not explicitly know the event graph and therefore, cannotdirectly evaluate whether the norm is consistent or inconsistent.Nonetheless, as we will see in the next section, consistent andinconsistent norms have fundamentally different evolutionaryproperties.

We label a subset of consistent norms where the prescriptionsfor each vertex of an edge are best responses to one anotheras best-response norms. A best-response norm always exists ifthe game admits a single-strategy Nash equilibrium. However,for a two-strategy pure Nash equilibrium, a best-response normexists if and only if the graph is bipartite. Since we may assignto each partition one of the strategies, each vertex along anedge is a best response to each other. It follows that the fit-ness of a best-response norm playing itself is the mean of thepayoffs of the pure Nash equilibria (SI Appendix, Lemmas S2and S3).

Evolutionary Stability of Norms. Similar to the concept of an evo-lutionary stable strategy (40), an evolutionary stable norm isa norm that cannot be invaded by an initially rare norm (SIAppendix, section 1 has the formal definition). We are concernedwith the evolutionary history initiated at a state without pre-scriptions where everyone plays a null norm using a mixed Nashequilibrium. As long as the prescriptions are not obeyed ([4] isnot satisfied), they may accumulate in the population by drift,since they remain playing the mixed Nash strategy. However,on being rational, they are obeyed. If such a norm is a con-sistent norm, it may then invade if its prescriptions are in thehull of the mixed Nash strategy. Any strategy within the sup-port of a mixed Nash playing the mixed Nash strategy receivesthe mixed Nash payoff. Furthermore, as a consistent normimplements a correlated equilibrium, its payoff playing itself isgreater than any alternative norm under the same event label-ing. Since the null norm’s behavior is identical across all events,the labeling is inconsequential and thus, receives a lower fitnessplaying the consistent norm. Therefore, the consistent norm caninvade. More generally, the same arguments applied to a sub-set of events show the emergence of obeyed prescriptions fromnull prescriptions. In other words, the evolutionary dynamicsof norms tend to manufacture beliefs that coordinate inter-actions to correlated equilibria out of inherently meaningless

events (SI Appendix, Lemma S4 has mathematical details of thesearguments).

If the null norm plays a mixed Nash, evolutionarily sta-ble norms must be consistent norms and thereby, implementcorrelated equilibria (SI Appendix, Lemma S6). In this way,evolutionarily stable norms in our model allow the decentral-ized emergence of not only coordinated behaviors but also,coordinated beliefs about the correlation device.

Since consistent norms induce a correlated equilibrium, theycan be evolutionarily stable against norms that share the samerelative labeling of events (i.e., as long as events are labeledsuch that they are differentiated in the same ways). Normswith different labels may invade a consistent norm that isevolutionarily stable against norms with the same labelingunless the resident norm is a best-response norm (SI Appendix,Lemma S7). Unlike consistent norms, inconsistent norms can-not be evolutionary stable, because they can be invaded bya norm that validates their descriptions, which by defini-tion, is different from themselves (SI Appendix, Lemma S5).However, two such inconsistent norms that empirically vali-date one another form a stable polymorphism (SI Appendix,Lemma S8).

We may have many other types of polymorphisms depend-ing on the underlying graph and labeling. Monomorphisms ofconsistent norms are not necessarily stable, since [4] only holdsfor a norm’s labeling. An implication of this is that a con-sistent norm that can invade an inconsistent norm will notnecessarily fix, and a polymorphism may thus result. This isbecause the expected payoff of an inconsistent norm againstthe consistent norm may be higher than the consistent normagainst itself if they have different labelings. However, if theconsistent norm’s labeling is complete (i.e., each vertex has aunique label), then the invading consistent norm will fix (SIAppendix, Lemma S9). Another example of a polymorphism isthat of weakly evolutionarily stable consistent norms with dif-fering prescriptions, which can exist as long as, on average, thepairings of prescriptions are the same or trivially, if the nullnorm and consistent norms’ behaviors are identical (SI Appendix,section 4).

Examples.The Game of Chicken. The existence and evolutionary stabilityof norms for a game are dependent on the underlying eventspace. Consider the bipartite graph G1 in Fig. 3 for the Game ofChicken. There is one behavior that a null norm can play, namelythe mixed Nash equilibrium, where they cooperate at each ver-tex with probability 2/3. The set of consistent norms is composedof five norms represented graphically as n1 . . .n5 in Fig. 3 (note

Fig. 3. Several graphical representations of the prescriptions of consis-tent norms. White corresponds to a prescription to cooperate and blackcorresponds to defect in graph G1. n1, n2, and n3 are evolutionary stablenorms. n1 provides a higher fitness than n2 and n3, which are best-response norms. In contrast, the only evolutionarily stable norms forgraph G2 are best-response norms. Black corresponds to demanding δ

and white corresponds to demanding 1− δ in graph G3 for the NashBargaining Game.

Morsky and Akcay PNAS | April 30, 2019 | vol. 116 | no. 18 | 8837

Dow

nloa

ded

by g

uest

on

July

4, 2

020

that n4 and n5 require an appropriate labeling |L|< |V | to berational). n1, n2, and n3 are evolutionarily stable, while the othertwo norms are not.

Continuing with the example above, we may consider all pre-scriptions as a strategy from which we have a 12-dimensionalreplicator equation, the symmetric Nash equilibria of which arethe fixed points (38). There are 25 fixed points of this model, 3of which are stable (the monomorphic states of the three sta-ble consistent norms). For monomorphic populations, n1 hasa higher fitness than the null norm and the two best-responsenorms, n2 and its inverse. Contrast this result to another bipar-tite graph, G2. The evolutionary stable norms of graph G2 areall best-response norms (the one depicted in Fig. 3 and symme-tries of it), which all have fitness 41/2, lower than the fitness ofn1 of 5.The evolution of fairness in the Nash Bargaining Game. The NashBargaining Game provides us with another interesting example(41). Each player’s strategy is a demand of a portion of a dollar. Ifthe sum of the demands is less than or equal to a dollar, then eachplayer receives their demand. If, however, the sum exceeds a dol-lar, then they both receive nothing. A variety of solutions (corre-sponding to varying definitions of fairness) have been found (41–43). We are interested in how a fair (50/50) division of the dollarcan emerge. It has been shown that a single-population replicatorequation model can converge to such a fair distribution for a sym-metric linear Nash Bargaining Game (44). However, the historyof play matters, and the basin of attraction to a suboptimal poly-morphic stable equilibrium is not negligibly small. A fair divisionof the surplus between tenants and landlords is also stochasti-cally stable if memory is large and noise is small (45). Here,players play the linear symmetric Nash Bargaining Game ateach edge.

Let us first consider fairness across event space (i.e., the vari-ance of the expected payoff) for evolutionarily stable consistentnorms prescribing either δ or 1− δ ∈ [0, 1/2]. If the graph isbipartite, then best-response norms may exist. Therefore, pre-scribing any δ can be evolutionarily stable. Examples includethe ring graph with an even number of vertices and the squaregrid graph. The ring graph with an odd number of vertices (suchas G3 in Fig. 3) and the complete graph are counterexamples.For these graphs, the range of δ for consistent norms is con-strained by a tradeoff between the distributions of each strategyand the demands. For example, for the complete graph, theproportion playing δ is approximately δ/(1− δ) (SI Appendix,section 5, Lemma S10). However, the only distribution that canreach the optimal expected payoff is δ= 1/2, the fair distribution.Since each event is equally correlated with every other event,this result can be interpreted as an expression of the Veil ofIgnorance (46).

To explore the effects of the graph type and number of ver-tices on the evolution of fairness in the Nash Bargaining Game,we used trait substitution simulations on the complete graph, ringgraphs, and small world graphs (47) (details are in SI Appendix,section 5), where the dollar can be divided into either quar-ters or dimes. These simulations permit both monomorphismsof consistent norms and polymorphisms of inconsistent norms.Fig. 4 depicts the results for the complete and degree 2 ringgraph (remaining results are in SI Appendix, section 5). The fairdistribution is frequently reached and is primarily a monomor-phic state of a consistent norm. Polymorphisms of inconsistentnorms can emerge and stabilize, leading to suboptimal mean fit-ness. In general, monomorphic states are more likely to leadto higher fitness than polymorphisms. For example, if the dol-lar can only be split into quarters on the complete graph, wehave 1/3 of the population demanding 1/4 of the dollar, and theremaining demanding 3/4, from which we have a mean fitnessof 1/4. Going from quarters to dimes decreases the frequency

C D

BA

Fig. 4. Mean fitness frequency in the Nash Bargaining Game for the com-plete (A and B) and ring (C and D) graphs and for dollar refinements ofquarters (A and C) and dimes (B and D). Red corresponds to monomorphicstates, and blue corresponds to polymorphic states.

of the fair outcome, as the cost of miscoordination becomessmaller.

DiscussionIn this paper, we show how social norms that coordinate behav-iors can be bootstrapped from random superstitions aboutirrelevant events in the world. A key to our model is thesynthesis of prescriptive and descriptive norms. Inconsisten-cies between prescriptive and descriptive norms have beenexperimentally shown to weaken norm adherence (48, 49). Assuch, our rationality and empirical validation conditions arekey to aligning prescriptive and descriptive norms and induc-ing their obeyance. We show that empirically validatable normsalign not only the behaviors of the players but also, theirbeliefs about others’ behaviors in a correlated equilibrium. Suchsocial norms act as choreographers of the players’ behaviors,potentially ameliorating social dilemmas, a well-studied prod-uct of social norms (50). However, our players are blind tothe true correlations between events in the world, and as such,this coordination is unintentional. Rather than being designed,the choreography of play through consistent belief emergesfrom the accumulation of random superstitions of the environ-ment. In alignment with empirical work (51, 52), our normsstart out at the individual level as subjective norms ascrib-ing meaning to events in the world: whether they be seeinga colored cat or something equally superstitious. If these sub-jective norms are rational to follow given the beliefs that theyinduce and yield a high payoff, they can win in evolutionarycompetition and acquire conventional normative meaning. Inthis way, normative meaning invented at the personal levelprovides the material for conventional meaning to emergethrough the joint action of individual rationality and evolutionarydynamics.

ACKNOWLEDGMENTS. Funding was provided by Defense Advanced Re-search Projects Agency Next Generation Social Science Program GrantD17AC00005 and Army Research Office Grant W911NF-12-R-0012-03.

8838 | www.pnas.org/cgi/doi/10.1073/pnas.1817095116 Morsky and Akcay

Dow

nloa

ded

by g

uest

on

July

4, 2

020

ECO

NO

MIC

SCIE

NCE

SEV

OLU

TIO

N

1. Ostrom E (2000) Collective action and the evolution of social norms. J Econ Perspect14:137–158.

2. Fehr E, Fischbacher U, Gachter S (2002) Strong reciprocity, human cooperation, andthe enforcement of social norms. Hum Nat 13:1–25.

3. Bicchieri C (2006) The Grammar of Society: The Nature and Dynamics of Social Norms(Cambridge Univ Press, Cambridge, UK).

4. Gintis H (2007) A framework for the unification of the behavioral sciences. BehavBrain Sci 30:1–16.

5. Jacobs RC, Campbell DT (1961) The perpetuation of an arbitrary tradition throughseveral generations of a laboratory microculture. J Abnorm Soc Psychol 62:649–658.

6. Cialdini RB, Kallgren CA, Reno RR (1991) A focus theory of normative conduct: Atheoretical refinement and reevaluation of the role of norms in human behavior.Advances in Experimental Social Psychology, ed Zanna MP (Academic, New York), Vol24, pp 201–234.

7. Etzioni A (2000) Social norms: Internalization, persuasion, and history. Law and SocRev 34:157–178.

8. Gintis H (2007) The evolution of private property. J Econ Behav Organ 64:1–16.9. Voss T (2001) Game theoretical perspectives on the emergence of social norms. Social

Norms, eds Hechter M, Opp KD (Russell Sage Foundation, New York), pp 105–136.10. Gintis H, Helbing D (2015) Homo socialis: An analytical core for sociological theory.

Rev Behav Econ 2:1–59.11. Paternotte C, Grose J (2012) Social norms and game theory: Harmony or discord?.

Br J Philos Sci 64:551–587.12. Binmore KG (1998) Game Theory and the Social Contract: Just Playing (MIT Press,

Cambridge, MA), Vol 2.13. Bowles S, Choi J-K, Hopfensitz A (2003) The co-evolution of individual behaviors and

social institutions. J Theor Biol 223:135–147.14. Tavoni A, Schluter M, Levin S (2012) The survival of the conformist: Social pressure

and renewable resource management. J Theor Biol 299:152–161.15. Chung A, Rimal RN (2016) Social norms: A review. Rev Commun Res 4:1–28.16. Gintis H (2010) Social norms as choreography. Polit Philos Econ, 9:251–264.17. Gintis H (2016) Individuality and Entanglement: The Moral and Material Bases of

Social Life (Princeton Univ Press, Princeton).18. Aumann RJ (1974) Subjectivity and correlation in randomized strategies. J Math Econ

1:67–96.19. Aumann RJ (1987) Correlated equilibrium as an expression of bayesian rationality.

Econometrica 55:1–18.20. Smith JM (1982) Evolution and the Theory of Games (Cambridge Univ Press,

Cambridge, UK).21. Eason PK, Cobbs GA, Trinca KG (1999) The use of landmarks to define territorial

boundaries. Anim Behav 58:85–91.22. Mesterton-Gibbons M, Adams ES (2003) Landmarks in territory partitioning: A

strategically stable convention? Am Naturalist 161:685–697.23. Selten R (1980) A note on evolutionarily stable strategies in asymmetric animal

conflicts. J Theory Biol 84:93–101.24. Cripps M (1991) Correlated equilibria and evolutionary stability. J Econ Theory

55:428–434.25. Shmida A, Peleg B (1997) Strict and symmetric correlated equilibria are the distribu-

tions of the ess’s of biological conflicts with asymmetric roles. Understanding Strate-

gic Interaction, eds Albers W, Guth W, Hammerstein P, Moldovanu B, van Damme E(Springer, Berlin), pp 149–170.

26. Foster DP, Vohra RV (1997) Calibrated learning and correlated equilibrium. GamesEcon Behav 21:40–55.

27. Hart S, Mas-Colell A (2000) A simple adaptive procedure leading to correlatedequilibrium. Econometrica 68:1127–1150.

28. Hart S (2005) Adaptive heuristics. Econometrica 73:1401–1430.29. Ianni A (2001) Learning correlated equilibria in population games. Math Soc Sci

42:271–294.30. Arifovic J, Boitnott JF, Duffy J (2019) Learning correlated equilibria: An evolutionary

approach. J Econ Behav Organ 157:171–190.31. Kim C, Wong K-c (2017) Evolutionarily stable correlation. Korean Econ Rev 33:55–94.32. Metzger LP (2018) Evolution and correlated equilibrium. J Evol Econ 28:333–

346.33. Guth W (1995) An evolutionary approach to explaining cooperative behavior by

reciprocal incentives. Int J Game Theor, 24:323–344.34. Dekel E,Ely JC, Ylankaya O (2007) Evolution of preferences. Rev Econ Stud 74:685–

704.35. Akcay E, Van Cleve J, Feldman MW, Roughgarden J (2009) A theory for the evolution

of other-regard integrating proximate and ultimate perspectives. Proc Natl Acad SciUSA 106:19061–19066.

36. Alger I, Weibull JW (2013) Homo moralis—Preference evolution under incompleteinformation and assortative matching. Econometrica 81:2269–2302.

37. Bicchieri C (2016) Norms in the Wild: How to Diagnose, Measure, and Change SocialNorms (Oxford Univ Press, Oxford).

38. Hofbauer J, Karl S (1998) Evolutionary Games and Population Dynamics (CambridgeUniv Press, Cambridge, UK).

39. Bicchieri C (1990) Norms of cooperation. Ethics 100:838–861.40. Smith JM (1974) The theory of games and the evolution of animal conflicts. J Theor

Biol 47:209–221.41. Nash J (1953) Two-person cooperative games. Econometrica 21:128–140.42. Kalai E, Smorodinsky M (1975) Other solutions to nash’s bargaining problem.

Econometrica 43:513–518.43. Harsanyi JC (1977) Rational Behaviour and Bargaining Equilibrium in Games and

Social Situations (Cambridge Univ Press, Cambridge, UK).44. Skyrms B (2000) Stability and explanatory significance of some simple evolutionary

models. Philos Sci 67:94–113.45. Young HP (1993) An evolutionary model of bargaining. J Econ Theor 59:145–168.46. Rawls J (1971) A Theory of Justice (Harvard Univ Press, Cambridge, MA).47. Watts DJ, Strogatz SH (1998) Collective dynamics of ‘small-world’networks. Nature

393:440–442.48. Pillutla MM, Chen X-P (1999) Social norms and cooperation in social dilemmas: The

effects of context and feedback. Organ Behav Hum Decis Process 78:81–103.49. Smith JR, et al. (2012) Congruent or conflicted? The impact of injunctive and

descriptive norms on environmental intentions. J Environ Psychol 32:353–361.50. Van Lange PAM, Joireman J, Parks CD, Van Dijk E (2013) The psychology of social

dilemmas: A review. Organ Behav Hum Decis Process 120:125–141.51. Sherif M (1936) The Psychology of Social Norms (Harper, New York).52. Valsiner J (2001) Comparative Study of Human Cultural Development (Fundacion

Infancia y Aprendizaje, Madrid).

Morsky and Akcay PNAS | April 30, 2019 | vol. 116 | no. 18 | 8839

Dow

nloa

ded

by g

uest

on

July

4, 2

020

Correction

ECONOMIC SCIENCES, EVOLUTIONCorrection for “Evolution of social norms and correlated equi-libria,” by Bryce Morsky and Erol Akçay, which was first pub-lished April 11, 2019; 10.1073/pnas.1817095116 (Proc. Natl.Acad. Sci. U.S.A. 116, 8834–8839).The authors note that Fig. 3 appeared incorrectly due to an

arrangement error during revision. The corrected Fig. 3 and itslegend appear below.

Published under the PNAS license.

First published March 30, 2020.

www.pnas.org/cgi/doi/10.1073/pnas.2004649117

Fig. 3. Several graphical representations of the prescriptions of consistentnorms. White corresponds to a prescription to cooperate and black corre-sponds to defect in graph G1. n1, n2, and n3 are evolutionary stable norms. n1

provides a higher fitness than n2 and n3, which are best-response norms. Incontrast, the only evolutionarily stable norms for graph G2 are best-responsenorms. Black corresponds to demanding δ and white corresponds to de-manding 1 − δ in graph G3 for the Nash Bargaining Game.

www.pnas.org PNAS | April 7, 2020 | vol. 117 | no. 14 | 8213

CORR

ECTION