cs344m autonomous multiagent systemstodd/cs344m/slides/week5a-pp4.pdfevolved on huge (at the time)...

75
CS344M Autonomous Multiagent Systems Todd Hester Department or Computer Science The University of Texas at Austin

Upload: others

Post on 17-Jun-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

CS344MAutonomous Multiagent Systems

Todd Hester

Department or Computer ScienceThe University of Texas at Austin

Page 2: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Good Afternoon, Colleagues

Are there any questions?

Todd Hester

Page 3: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Logistics

• Readings

Todd Hester

Page 4: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Logistics

• Readings

– Specify which papers you read!

Todd Hester

Page 5: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Logistics

• Readings

– Specify which papers you read!– 2 case studies and 1 TDP

Todd Hester

Page 6: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Logistics

• Readings

– Specify which papers you read!– 2 case studies and 1 TDP

• How to read a research paper

Todd Hester

Page 7: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Logistics

• Readings

– Specify which papers you read!– 2 case studies and 1 TDP

• How to read a research paper

– Some have too few details...

Todd Hester

Page 8: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Logistics

• Readings

– Specify which papers you read!– 2 case studies and 1 TDP

• How to read a research paper

– Some have too few details...– Others have too many.

Todd Hester

Page 9: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Logistics

• Readings

– Specify which papers you read!– 2 case studies and 1 TDP

• How to read a research paper

– Some have too few details...– Others have too many.

• Next week’s readings posted

Todd Hester

Page 10: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Logistics

• Readings

– Specify which papers you read!– 2 case studies and 1 TDP

• How to read a research paper

– Some have too few details...– Others have too many.

• Next week’s readings posted

• Use the undergrad writing center!

– Friday afternoon workshops (3 p.m.)

Todd Hester

Page 11: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Overview of the Readings• Darwin: genetic programming approach

Todd Hester

Page 12: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Overview of the Readings• Darwin: genetic programming approach

• Stone and McAllester: Architecture for action selection

Todd Hester

Page 13: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Overview of the Readings• Darwin: genetic programming approach

• Stone and McAllester: Architecture for action selection

• Riley et al: Coach competition, extracting models

Todd Hester

Page 14: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Overview of the Readings• Darwin: genetic programming approach

• Stone and McAllester: Architecture for action selection

• Riley et al: Coach competition, extracting models

• Kuhlmann et al: Learning for coaching

Todd Hester

Page 15: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Overview of the Readings• Darwin: genetic programming approach

• Stone and McAllester: Architecture for action selection

• Riley et al: Coach competition, extracting models

• Kuhlmann et al: Learning for coaching

• Withopf and Riedmiller: Reinforcement learning

Todd Hester

Page 16: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Overview of the Readings• Darwin: genetic programming approach

• Stone and McAllester: Architecture for action selection

• Riley et al: Coach competition, extracting models

• Kuhlmann et al: Learning for coaching

• Withopf and Riedmiller: Reinforcement learning

• MacAlpine et al: UT Austin Villa 2011

Todd Hester

Page 17: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Overview of the Readings• Darwin: genetic programming approach

• Stone and McAllester: Architecture for action selection

• Riley et al: Coach competition, extracting models

• Kuhlmann et al: Learning for coaching

• Withopf and Riedmiller: Reinforcement learning

• MacAlpine et al: UT Austin Villa 2011

• Barrett et al: SPL Kicking strategy

Todd Hester

Page 18: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Evolutionary Computation• Motivated by biological evolution: GA, GP

Todd Hester

Page 19: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Evolutionary Computation• Motivated by biological evolution: GA, GP

• Search through a space

Todd Hester

Page 20: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Evolutionary Computation• Motivated by biological evolution: GA, GP

• Search through a space

− Need a representation, fitness function− Probabilistically apply search operators to set of points

in search space

Todd Hester

Page 21: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Evolutionary Computation• Motivated by biological evolution: GA, GP

• Search through a space

− Need a representation, fitness function− Probabilistically apply search operators to set of points

in search space

• Randomized, parallel hill-climbing through space

Todd Hester

Page 22: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Evolutionary Computation• Motivated by biological evolution: GA, GP

• Search through a space

− Need a representation, fitness function− Probabilistically apply search operators to set of points

in search space

• Randomized, parallel hill-climbing through space

• Learning is an optimization problem (fitness)

Todd Hester

Page 23: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Evolutionary Computation• Motivated by biological evolution: GA, GP

• Search through a space

− Need a representation, fitness function− Probabilistically apply search operators to set of points

in search space

• Randomized, parallel hill-climbing through space

• Learning is an optimization problem (fitness)

Some slides from Machine Learning [Mitchell, 1997]

Todd Hester

Page 24: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Darwin United

• More ambitious follow-up to Luke, 97 (made 2nd round)

Todd Hester

Page 25: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Darwin United

• More ambitious follow-up to Luke, 97 (made 2nd round)

• Motivated in part by Peter’s detailed team construction

Todd Hester

Page 26: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Darwin United

• More ambitious follow-up to Luke, 97 (made 2nd round)

• Motivated in part by Peter’s detailed team construction

• Evolves whole teams — lexicographic fitness function

Todd Hester

Page 27: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Darwin United

• More ambitious follow-up to Luke, 97 (made 2nd round)

• Motivated in part by Peter’s detailed team construction

• Evolves whole teams — lexicographic fitness function

• Evolved on huge (at the time) hypercube

Todd Hester

Page 28: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Darwin United

• More ambitious follow-up to Luke, 97 (made 2nd round)

• Motivated in part by Peter’s detailed team construction

• Evolves whole teams — lexicographic fitness function

• Evolved on huge (at the time) hypercube

• Lots of spinning, but figured out dribbling, offsides

Todd Hester

Page 29: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Darwin United

• More ambitious follow-up to Luke, 97 (made 2nd round)

• Motivated in part by Peter’s detailed team construction

• Evolves whole teams — lexicographic fitness function

• Evolved on huge (at the time) hypercube

• Lots of spinning, but figured out dribbling, offsides

• 1-1-1 record. Tied a good team, but didn’t advance

Todd Hester

Page 30: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Darwin United

• More ambitious follow-up to Luke, 97 (made 2nd round)

• Motivated in part by Peter’s detailed team construction

• Evolves whole teams — lexicographic fitness function

• Evolved on huge (at the time) hypercube

• Lots of spinning, but figured out dribbling, offsides

• 1-1-1 record. Tied a good team, but didn’t advance

• Success of the method, but not pursued

Todd Hester

Page 31: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Architecture for Action Selection

• (other slides, video)

Todd Hester

Page 32: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Architecture for Action Selection

• (other slides, video)

• downsides

Todd Hester

Page 33: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Architecture for Action Selection

• (other slides, video)

• downsides

• Keepaway

Todd Hester

Page 34: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coaching• Learn best strategy to play a fixed team

Todd Hester

Page 35: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coaching• Learn best strategy to play a fixed team

• Give high level advice to players at low frequency

Todd Hester

Page 36: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coaching• Learn best strategy to play a fixed team

• Give high level advice to players at low frequency

• Focus on learning formations

Todd Hester

Page 37: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coaching• Learn best strategy to play a fixed team

• Give high level advice to players at low frequency

• Focus on learning formations

• Learn when successful teams passed/kicked

Todd Hester

Page 38: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coaching• Learn best strategy to play a fixed team

• Give high level advice to players at low frequency

• Focus on learning formations

• Learn when successful teams passed/kicked

• Learn when opponent will pass and try to block

Todd Hester

Page 39: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coaching• Learn best strategy to play a fixed team

• Give high level advice to players at low frequency

• Focus on learning formations

• Learn when successful teams passed/kicked

• Learn when opponent will pass and try to block

• What if players switch roles?

Todd Hester

Page 40: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coaching• Learn best strategy to play a fixed team

• Give high level advice to players at low frequency

• Focus on learning formations

• Learn when successful teams passed/kicked

• Learn when opponent will pass and try to block

• What if players switch roles?

• Why just imitate another team?

Todd Hester

Page 41: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coaching• Learn best strategy to play a fixed team

• Give high level advice to players at low frequency

• Focus on learning formations

• Learn when successful teams passed/kicked

• Learn when opponent will pass and try to block

• What if players switch roles?

• Why just imitate another team?

• Other slides

Todd Hester

Page 42: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Reinforcement Learning

• RL Slides

Todd Hester

Page 43: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Reinforcement Learning

• RL Slides

• Extend to grid soccer

Todd Hester

Page 44: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Reinforcement Learning

• RL Slides

• Extend to grid soccer

• Large state space, joint actions

Todd Hester

Page 45: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Reinforcement Learning

• RL Slides

• Extend to grid soccer

• Large state space, joint actions

Todd Hester

Page 46: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

UT Austin Villa 2011

• Other slides

Todd Hester

Page 47: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

UT Austin Villa 2011

• Other slides

• Why not use CMA-ES on role positions as well?

Todd Hester

Page 48: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

UT Austin Villa 2011

• Other slides

• Why not use CMA-ES on role positions as well?

• Changes for 2012?

Todd Hester

Page 49: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Kicking Under Uncertainty• Used by our SPL team

Todd Hester

Page 50: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Kicking Under Uncertainty• Used by our SPL team

• Kick engine to kick at various distances/headings

Todd Hester

Page 51: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Kicking Under Uncertainty• Used by our SPL team

• Kick engine to kick at various distances/headings

• Adjust to seen ball location

Todd Hester

Page 52: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Kicking Under Uncertainty• Used by our SPL team

• Kick engine to kick at various distances/headings

• Adjust to seen ball location

• Select first kick that moves ball up field

Todd Hester

Page 53: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Kicking Under Uncertainty• Used by our SPL team

• Kick engine to kick at various distances/headings

• Adjust to seen ball location

• Select first kick that moves ball up field

• Figure

Todd Hester

Page 54: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Kicking Under Uncertainty• Used by our SPL team

• Kick engine to kick at various distances/headings

• Adjust to seen ball location

• Select first kick that moves ball up field

• Figure

• Emphasis on quickness

Todd Hester

Page 55: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Kicking Under Uncertainty• Used by our SPL team

• Kick engine to kick at various distances/headings

• Adjust to seen ball location

• Select first kick that moves ball up field

• Figure

• Emphasis on quickness

• Now: Better model of opponents -> Know if we have moretime

Todd Hester

Page 56: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Kicking Under Uncertainty• Used by our SPL team

• Kick engine to kick at various distances/headings

• Adjust to seen ball location

• Select first kick that moves ball up field

• Figure

• Emphasis on quickness

• Now: Better model of opponents -> Know if we have moretime

Todd Hester

Page 57: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Learning Commentary

• David Chen and Ray Mooney

Todd Hester

Page 58: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coordination Graphs• n agents, each choose an action Ai

Todd Hester

Page 59: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coordination Graphs• n agents, each choose an action Ai

• A = A1 × . . .×An

Todd Hester

Page 60: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coordination Graphs• n agents, each choose an action Ai

• A = A1 × . . .×An

• Ri(A) 7→ IR

Todd Hester

Page 61: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coordination Graphs• n agents, each choose an action Ai

• A = A1 × . . .×An

• Ri(A) 7→ IR

• Coordination problem: R1 = . . . = Rn = R

Todd Hester

Page 62: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coordination Graphs• n agents, each choose an action Ai

• A = A1 × . . .×An

• Ri(A) 7→ IR

• Coordination problem: R1 = . . . = Rn = R

• Nash equilibrium: no agent could do better given whatothers are doing.

Todd Hester

Page 63: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Coordination Graphs• n agents, each choose an action Ai

• A = A1 × . . .×An

• Ri(A) 7→ IR

• Coordination problem: R1 = . . . = Rn = R

• Nash equilibrium: no agent could do better given whatothers are doing.

• May be more than one (chicken)

Todd Hester

Page 64: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Example from the paper• Understand the rule syntax

Todd Hester

Page 65: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Example from the paper• Understand the rule syntax

• Form the coordination graph

Todd Hester

Page 66: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Example from the paper• Understand the rule syntax

• Form the coordination graph

• First eliminate rules based on context

Todd Hester

Page 67: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Example from the paper• Understand the rule syntax

• Form the coordination graph

• First eliminate rules based on context

• What does it mean for G3 to collect all relevant rules?

Todd Hester

Page 68: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Example from the paper• Understand the rule syntax

• Form the coordination graph

• First eliminate rules based on context

• What does it mean for G3 to collect all relevant rules?

• What does it mean for G3 to maximize over all actions ofa1 and a2?

Todd Hester

Page 69: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Example from the paper• Understand the rule syntax

• Form the coordination graph

• First eliminate rules based on context

• What does it mean for G3 to collect all relevant rules?

• What does it mean for G3 to maximize over all actions ofa1 and a2?

• How are the results propagated back?

Todd Hester

Page 70: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Example from the paper• Understand the rule syntax

• Form the coordination graph

• First eliminate rules based on context

• What does it mean for G3 to collect all relevant rules?

• What does it mean for G3 to maximize over all actions ofa1 and a2?

• How are the results propagated back?

• Let’s try again with G1 eliminated first

Todd Hester

Page 71: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Application to soccer• Make the world discrete by assigning roles, using high-

level predicates

Todd Hester

Page 72: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Application to soccer• Make the world discrete by assigning roles, using high-

level predicates

• Assume global state information

Todd Hester

Page 73: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Application to soccer• Make the world discrete by assigning roles, using high-

level predicates

• Assume global state information

• Finds pass sequences and starts players moving ahead oftime.

Todd Hester

Page 74: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Application to soccer• Make the world discrete by assigning roles, using high-

level predicates

• Assume global state information

• Finds pass sequences and starts players moving ahead oftime.

• Note the results: with and without coordination.

Todd Hester

Page 75: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd

Reactive Deliberation

• A hybrid approach

• Executor: carry out reactive behaviors

• Deliberator: evaluate possible high-level schema withparameters; generate bids

• Deliberator takes time, but something keeps happeningalways.

• In effect: deliberator commits to schema for some time

Todd Hester