Worlds with many intelligent agents

An important consideration in AI, as well as games, distributed systems and networking,

economics, sociology, political science, international relations, and other disciplines

Multi-agent Systems, or “Games”Worlds with multiple agents, somtimes called “games”, are VASTLY more complicated than worlds with single agents.

They’re also more interesting and fun!

There are many different aspects to learn about:- Representations (we will consider two, but there are MANY)- Evaluation metrics (I’ll show you a few standard ones, but again there are

MANY)- Inference and planning (we will talk about a few techniques)- Learning (We won’t really cover this)- Communication (We won’t directly cover this)- Computational Complexity (I’ll mention this in passing)- Applications (We’ll look at several examples, but there are TONS)

Example Game (Prisoner’s Dilemma)Student A and Student B have been naughty, cheating on an exam. Their teacher finds some evidence, and sends them to the dean’s office, where the dean grills them individually to find out more.

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Outcome:

Example Game (Prisoner’s Dilemma)Definitions: the set of choices available to each agent is called their action set, denoted Actions(A) or Actions(B).Each square in the grid is called an outcome. Mathematically, let a be an action in Actions(A), and b in Actions(B). O(a, b) is the outcome square. OA(a, b) is the reward for A, and OB(a, b) is the reward for B.This grid is called a normal form representation of the game.

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Outcome:

Quiz: Best outcome?Which outcome is the best, and why?

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Outcome:

Answer: Best outcome?The answer depends. Let’s look at a few definitions of “best”.

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Outcome:

Definition: Dominant StrategyDefinition: A strategy (or pure strategy) for player A (ditto for B) is a selection of one of the actions from Actions(A).

Definition: A’s strategy a ∊ Actions(A) is dominant if∀b ∊ Actions(B), ∀a’≠a ∊ Actions(A) . OA(a, b) > OA(a’, b)

Another way of saying this is: a is a dominant strategy for A if it is better than any other strategy a’ for A, no matter what strategy B chooses.

Quiz: Dominant StrategyDefinition: A’s strategy a ∊ Actions(A) is dominant if∀b ∊ Actions(B), ∀a’≠a ∊ Actions(A) . OA(a, b) > OA(a’, b)Does A have a dominant strategy for the game below? What is it? What about B?

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Outcome:

Answer: Dominant StrategyDefinition: A’s strategy a ∊ Actions(A) is dominant if∀b ∊ Actions(B), ∀a’≠a ∊ Actions(A) . OA(a, b) > OA(a’, b)Does A have a dominant strategy for the game below? What is it? What about B?

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Dominant strategy for A!

Dom

inan

t st

rate

gy fo

r B!

Definition: Pareto OptimalDefinition: An outcome o is called pareto optimal if there is no other outcome o’ such that all players prefer o’ to o.

Notice: pareto optimality can lead to lots of optimal outcomes, not just one.

That’s because o’ is considered better than o only if ALL players prefer o’ to o.

Quiz: Pareto OptimalDefinition: An outcome o is called pareto optimal if there is no other outcome o’ such that all players prefer o’ to o.

Quiz: which outcome(s) is (are) pareto optimal?

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Outcome:

Answer: Pareto OptimalDefinition: An outcome o is called pareto optimal if there is no other outcome o’ such that all players prefer o’ to o.

Quiz: which outcome(s) is (are) pareto optimal?

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Outcome:

Definition: (Nash) EquilibriumDefinition: An outcome O(a, b) is called a Nash Equilibrium (or just equilibrium) if 1. ∀a’≠a ∊ Actions(A) . OA(a, b) >= OA(a’, b) and2. ∀b’≠b ∊ Actions(B) . OB(a, b) >= OB(a, b’)

Another way of saying this is that an outcome is an equilibrium if each player is as happy with this outcome as any other available option, assuming the other player sticks with his equilibrium strategy.

Notice that this means an outcome is stable, in the sense that both players have no incentive to change their strategy.

Quiz: (Nash) EquilibriumDefinition: An outcome O(a, b) is called a Nash Equilibrium if 1. ∀a’≠a ∊ Actions(A) . OA(a, b) >= OA(a’, b) and2. ∀b’≠b ∊ Actions(B) . OB(a, b) >= OB(a, b’)Which outcome(s) is (are) a Nash equilibrium?

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Outcome:

Answer: (Nash) EquilibriumDefinition: An outcome O(a, b) is called a Nash Equilibrium if 1. ∀a’≠a ∊ Actions(A) . OA(a, b) >= OA(a’, b) and2. ∀b’≠b ∊ Actions(B) . OB(a, b) >= OB(a, b’)Which outcome(s) is (are) a Nash equilibrium?

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Outcome:

Answer: (Nash) EquilibriumOne reason that the prisoner’s dilemma is famous:The only outcome that is a Nash equilibrium is also the only outcome that is NOT a pareto optimum.This is not true in all games; it depends on the rewards!For the prisoner’s dilemma, it means that the only stable solution is probably the worst possible solution for the players. (It’s sort of a depressing example.)

Dean now has lots of evidence on both students, both are suspended for the semester.

A: -7, B: -7

Dean has lots of evidence on A, gives B a light punishment for cooperating.

A: -10, B: -1

Dean has lots of evidence on B, gives A a light punishment for cooperating.

A: -1, B: -10

Dean has little evidence on both, gives both some penalty.

A: -3, B: -3

A rats out B A says nothing

B ra

ts

out A

B sa

ys

noth

ing

Quiz: The game of Chicken(a.k.a. “Hawk-Dove”, “snow-drift”)

What are the dominant strategies, pareto optima, and Nash equlibria for the Chicken game?

Neither player wins; a tie.

P1: 0, P2: 0

Player 1 wins.

P1: +1, P2: -1

Player 2 wins.

P1: -1, P2: +1

The players crash; a catastrophe!

P1: -10, P2: -10

Swerve Straight

Swer

veSt

raig

ht

Player 1

Play

er 2

Answer: The game of Chicken(a.k.a. “Hawk-Dove”, “snow-drift”)

What are the dominant strategies, pareto optima, and Nash equlibria for the Chicken game?

Neither player wins; a tie.

P1: 0, P2: 0

Player 1 wins.

P1: +1, P2: -1

Player 2 wins.

P1: -1, P2: +1

The players crash; a catastrophe!

P1: -10, P2: -10

Swerve Straight

Swer

veSt

raig

ht

Player 1

Play

er 2

No dominant strategies!

Pareto Optimum

Nash Equilibrium

Quiz: Coordination GamesWhat are the dominant strategies, pareto optima, and Nash equlibria for each of the games below?

P1: 10, P2: 10 P1: -10, P2: -10

P1: -10, P2: -10 P1: 10, P2: 10

Left Right

Left

Righ

t

10, 5 0, 0

0, 0 5, 10

Party Home

Part

yHo

me

Drive on which side of the road?

“Battle of the Sexes” game

10, 10 0, 0

0, 0 5, 5

Party Home

Part

yHo

me

Pure coordination game

10, 10 7, 0

0, 7 7, 7

Stag HareSt

agHa

re

“Stag Hunt” game

Answer: Coordination GamesWhat are the dominant strategies, pareto optima, and Nash equlibria for each of the games below?

P1: 10, P2: 10 P1: -10, P2: -10

P1: -10, P2: -10 P1: 10, P2: 10

Left Right

Left

Righ

t

10, 5 0, 0

0, 0 5, 10

Party Home

Part

yHo

me

Drive on which side of the road?

“Battle of the Sexes” game

10, 10 0, 0

0, 0 5, 5

Party Home

Part

yHo

me

Pure coordination game

10, 10 7, 0

0, 7 7, 7

Stag HareSt

agHa

re

“Stag Hunt” game

Pareto Optimum

Nash EquilibriumDominant

Strategy

Humans don’t necessarily play Nash equlibria

The “Guess 2/3 of the Average” Game is famous because in experiments with people, they often don’t play Nash equilibrium strategies. (Other games like this are the centipede game and the prisoner’s dilemma.)

Thus Nash equilibria aren’t perfect models of human behavior.

Let’s try it. Here are the rules to “Guess 2/3 of the average”:

1. Everyone guess a number between 0 and 100 (integer or real). Once everyone guesses, we’ll compute the average.

2. The set of people who come closest to 2/3 of the average are the winners (payoff 1), everyone else loses (payoff 0).

Nash equilibrium for “Guess 2/3 of the Average”

If everyone guesses 100, then 2/3 of the average is 66.7, so 66.7 is the biggest possible result, so everyone has an incentive to guess 66.7 or less, so anything more than 66.7 can’t be a Nash equilibrium.

If everyone guesses 66.7, then 2/3 of the average will be around 44.4, so everyone has an incentive to guess lower, so guessing more than 44.4 can’t be a Nash equilibrium.

…

If everyone guesses 0, then we have a Nash equilibrium (they’d all guess right and win, and there’s no incentive for anyone to change their guess assuming everyone else remains at 0).

Constant-Sum Games

0, 0 +1, -1 -1, +1

-1, +1 0, 0 +1, -1

+1, -1 -1, +1 0, 0

Rock Scissors

Player 1

Play

er 2

Paper

Sciss

ors

Pape

rRo

ck

Notice: if you add up the reward for P1 and P2 in each square, you get 0.

The fact that all squares have the same sum makes this a “constant-sum” game.

This game is in fact a “zero-sum” game.

Constant-Sum Games

0, 0 +1, -1 -1, +1

-1, +1 0, 0 +1, -1

+1, -1 -1, +1 0, 0

Rock Scissors

Player 1

Play

er 2

Paper

Sciss

ors

Pape

rRo

ck

Actually, any constant-sum game can be converted to an equivalent zero-sum game by subtracting a constant from every payoff.

They are “equivalent” in that this transformation preserves equilibria, optimal, and dominant strategies.

Constant-Sum Games

0, 0 +1, -1 -1, +1

-1, +1 0, 0 +1, -1

+1, -1 -1, +1 0, 0

Rock Scissors

Player 1

Play

er 2

Paper

Sciss

ors

Pape

rRo

ck

In constant-sum games, any gain for one player is offset by a loss for the other player.

So these are hyper-competitive games.

Quiz: Pure Strategy Nash Equilibria

0, 0 +1, -1 -1, +1

-1, +1 0, 0 +1, -1

+1, -1 -1, +1 0, 0

Rock Scissors

Player 1

Play

er 2

Paper

Sciss

ors

Pape

rRo

ck

Can you find a Nash equilibrium for Rock-Paper-Scissors?

Answer: Pure Strategy Nash Equilibria

0, 0 +1, -1 -1, +1

-1, +1 0, 0 +1, -1

+1, -1 -1, +1 0, 0

Rock Scissors

Player 1

Play

er 2

Paper

Sciss

ors

Pape

rRo

ck

Can you find a Nash equilibrium for Rock-Paper-Scissors?

No “pure” strategy for either player can lead to a Nash equilibrium for this game!

(Our definition of “strategy” is equivalent to “pure strategy”. We’ll talk about unpure strategies next.)

Definition: Mixed StrategyA mixed strategy for player A is a probability distribution over the set of available actions, Actions(A).

For instance, for Rock-Paper-Scissors, the distribution

P(Rock) = 1/3P(Paper) = 1/3P(Scissors) = 1/3

is a mixed strategy.

A pure strategy is also a mixed strategy, but with a probability distribution that places all the probability on one outcome.E.g.: P(Rock) = 0, P(Paper) = 1, P(Scissors) = 0.

Definition: Mixed Strategy Nash Equilibrium

A mixed strategy profile for players A and B is a pair of mixed strategies, distribution PA for player A and distribution PB for player B.

A mixed strategy Nash equilibrium is a mixed strategy profile (PA, PB) such that neither player would gain by choosing a different mixed strategy, assuming the other player’s mixed strategy stays the same.

Quiz: Mixed Strategy Nash Equilibria

0, 0 +1, -1 -1, +1

-1, +1 0, 0 +1, -1

+1, -1 -1, +1 0, 0

Rock Scissors

Player 1

Play

er 2

Paper

Sciss

ors

Pape

rRo

ck

Can you find a mixed-strategy Nash equilibrium for Rock Paper Scissors?

(This can be hard in general, but see if you can guess it for this game.)

Answer: Mixed Strategy Nash Equilibria

0, 0 +1, -1 -1, +1

-1, +1 0, 0 +1, -1

+1, -1 -1, +1 0, 0

Rock Scissors

Player 1

Play

er 2

Paper

Sciss

ors

Pape

rRo

ck

The mixed strategy profile where each player has 1/3 probability for each action is a Nash equilibrium.

(It’s the only Nash equilibrium for this game.)

Nash’s TheoremTheorem (Nash, 1951): Every finite game (finite number of players, finite number of pure strategies) has at least one mixed-strategy Nash equilibrium.

Nash, John (1951) "Non-Cooperative Games" The Annals of Mathematics 54(2):286-295.

(John Nash did not call them “Nash equilibria”, that name came later.)

He shared the 1994 Nobel Memorial Prize in Economic Sciences with game theorists Reinhard Selten and John Harsanyi for his work on Nash equilibria.

He suffered from schizophrenia in the 1950s and 1960s, as depicted in the 1998 film, “A Beautiful Mind”. He nevertheless recovered enough to return to academia and continue his research.

More on Constant-Sum GamesMinimax Theorem (John von Neumann, 1928): For every two-person, zero-sum game with finitely many pure strategies, there exists a mixed strategy for each player and a value V such that:- Given player 2’s strategy, the best possible payoff for player 1 is V- Given player 1’s strategy, the best possible payoff for player 2 is –V.

The existence of strategies part is a special case of Nash’s theorem, and a precursor to it.

This basically says that player 1 can guarantee himself a payoff of at least V, and player 2 can guarantee himself a payoff of at least –V. If both players play optimally, that’s exactly what they will get.

It’s called “minimax” because the players get this value by pursuing a strategy that tries to minimize the maximum payoff of the other player. We’ll come back to this.

Definition: The value V is called the value of the game.Eg: The value of Rock-paper-scissors is 0; the best that P1 can hope to achieve, assuming P2 plays optimally (1/3 probability of each action), is a payoff of 0.