cs344m autonomous multiagent systemstodd/cs344m/slides/week5a-pp4.pdfevolved on huge (at the time)...
TRANSCRIPT
![Page 1: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/1.jpg)
CS344MAutonomous Multiagent Systems
Todd Hester
Department or Computer ScienceThe University of Texas at Austin
![Page 2: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/2.jpg)
Good Afternoon, Colleagues
Are there any questions?
Todd Hester
![Page 3: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/3.jpg)
Logistics
• Readings
Todd Hester
![Page 4: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/4.jpg)
Logistics
• Readings
– Specify which papers you read!
Todd Hester
![Page 5: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/5.jpg)
Logistics
• Readings
– Specify which papers you read!– 2 case studies and 1 TDP
Todd Hester
![Page 6: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/6.jpg)
Logistics
• Readings
– Specify which papers you read!– 2 case studies and 1 TDP
• How to read a research paper
Todd Hester
![Page 7: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/7.jpg)
Logistics
• Readings
– Specify which papers you read!– 2 case studies and 1 TDP
• How to read a research paper
– Some have too few details...
Todd Hester
![Page 8: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/8.jpg)
Logistics
• Readings
– Specify which papers you read!– 2 case studies and 1 TDP
• How to read a research paper
– Some have too few details...– Others have too many.
Todd Hester
![Page 9: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/9.jpg)
Logistics
• Readings
– Specify which papers you read!– 2 case studies and 1 TDP
• How to read a research paper
– Some have too few details...– Others have too many.
• Next week’s readings posted
Todd Hester
![Page 10: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/10.jpg)
Logistics
• Readings
– Specify which papers you read!– 2 case studies and 1 TDP
• How to read a research paper
– Some have too few details...– Others have too many.
• Next week’s readings posted
• Use the undergrad writing center!
– Friday afternoon workshops (3 p.m.)
Todd Hester
![Page 11: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/11.jpg)
Overview of the Readings• Darwin: genetic programming approach
Todd Hester
![Page 12: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/12.jpg)
Overview of the Readings• Darwin: genetic programming approach
• Stone and McAllester: Architecture for action selection
Todd Hester
![Page 13: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/13.jpg)
Overview of the Readings• Darwin: genetic programming approach
• Stone and McAllester: Architecture for action selection
• Riley et al: Coach competition, extracting models
Todd Hester
![Page 14: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/14.jpg)
Overview of the Readings• Darwin: genetic programming approach
• Stone and McAllester: Architecture for action selection
• Riley et al: Coach competition, extracting models
• Kuhlmann et al: Learning for coaching
Todd Hester
![Page 15: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/15.jpg)
Overview of the Readings• Darwin: genetic programming approach
• Stone and McAllester: Architecture for action selection
• Riley et al: Coach competition, extracting models
• Kuhlmann et al: Learning for coaching
• Withopf and Riedmiller: Reinforcement learning
Todd Hester
![Page 16: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/16.jpg)
Overview of the Readings• Darwin: genetic programming approach
• Stone and McAllester: Architecture for action selection
• Riley et al: Coach competition, extracting models
• Kuhlmann et al: Learning for coaching
• Withopf and Riedmiller: Reinforcement learning
• MacAlpine et al: UT Austin Villa 2011
Todd Hester
![Page 17: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/17.jpg)
Overview of the Readings• Darwin: genetic programming approach
• Stone and McAllester: Architecture for action selection
• Riley et al: Coach competition, extracting models
• Kuhlmann et al: Learning for coaching
• Withopf and Riedmiller: Reinforcement learning
• MacAlpine et al: UT Austin Villa 2011
• Barrett et al: SPL Kicking strategy
Todd Hester
![Page 18: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/18.jpg)
Evolutionary Computation• Motivated by biological evolution: GA, GP
Todd Hester
![Page 19: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/19.jpg)
Evolutionary Computation• Motivated by biological evolution: GA, GP
• Search through a space
Todd Hester
![Page 20: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/20.jpg)
Evolutionary Computation• Motivated by biological evolution: GA, GP
• Search through a space
− Need a representation, fitness function− Probabilistically apply search operators to set of points
in search space
Todd Hester
![Page 21: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/21.jpg)
Evolutionary Computation• Motivated by biological evolution: GA, GP
• Search through a space
− Need a representation, fitness function− Probabilistically apply search operators to set of points
in search space
• Randomized, parallel hill-climbing through space
Todd Hester
![Page 22: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/22.jpg)
Evolutionary Computation• Motivated by biological evolution: GA, GP
• Search through a space
− Need a representation, fitness function− Probabilistically apply search operators to set of points
in search space
• Randomized, parallel hill-climbing through space
• Learning is an optimization problem (fitness)
Todd Hester
![Page 23: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/23.jpg)
Evolutionary Computation• Motivated by biological evolution: GA, GP
• Search through a space
− Need a representation, fitness function− Probabilistically apply search operators to set of points
in search space
• Randomized, parallel hill-climbing through space
• Learning is an optimization problem (fitness)
Some slides from Machine Learning [Mitchell, 1997]
Todd Hester
![Page 24: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/24.jpg)
Darwin United
• More ambitious follow-up to Luke, 97 (made 2nd round)
Todd Hester
![Page 25: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/25.jpg)
Darwin United
• More ambitious follow-up to Luke, 97 (made 2nd round)
• Motivated in part by Peter’s detailed team construction
Todd Hester
![Page 26: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/26.jpg)
Darwin United
• More ambitious follow-up to Luke, 97 (made 2nd round)
• Motivated in part by Peter’s detailed team construction
• Evolves whole teams — lexicographic fitness function
Todd Hester
![Page 27: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/27.jpg)
Darwin United
• More ambitious follow-up to Luke, 97 (made 2nd round)
• Motivated in part by Peter’s detailed team construction
• Evolves whole teams — lexicographic fitness function
• Evolved on huge (at the time) hypercube
Todd Hester
![Page 28: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/28.jpg)
Darwin United
• More ambitious follow-up to Luke, 97 (made 2nd round)
• Motivated in part by Peter’s detailed team construction
• Evolves whole teams — lexicographic fitness function
• Evolved on huge (at the time) hypercube
• Lots of spinning, but figured out dribbling, offsides
Todd Hester
![Page 29: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/29.jpg)
Darwin United
• More ambitious follow-up to Luke, 97 (made 2nd round)
• Motivated in part by Peter’s detailed team construction
• Evolves whole teams — lexicographic fitness function
• Evolved on huge (at the time) hypercube
• Lots of spinning, but figured out dribbling, offsides
• 1-1-1 record. Tied a good team, but didn’t advance
Todd Hester
![Page 30: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/30.jpg)
Darwin United
• More ambitious follow-up to Luke, 97 (made 2nd round)
• Motivated in part by Peter’s detailed team construction
• Evolves whole teams — lexicographic fitness function
• Evolved on huge (at the time) hypercube
• Lots of spinning, but figured out dribbling, offsides
• 1-1-1 record. Tied a good team, but didn’t advance
• Success of the method, but not pursued
Todd Hester
![Page 31: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/31.jpg)
Architecture for Action Selection
• (other slides, video)
Todd Hester
![Page 32: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/32.jpg)
Architecture for Action Selection
• (other slides, video)
• downsides
Todd Hester
![Page 33: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/33.jpg)
Architecture for Action Selection
• (other slides, video)
• downsides
• Keepaway
Todd Hester
![Page 34: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/34.jpg)
Coaching• Learn best strategy to play a fixed team
Todd Hester
![Page 35: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/35.jpg)
Coaching• Learn best strategy to play a fixed team
• Give high level advice to players at low frequency
Todd Hester
![Page 36: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/36.jpg)
Coaching• Learn best strategy to play a fixed team
• Give high level advice to players at low frequency
• Focus on learning formations
Todd Hester
![Page 37: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/37.jpg)
Coaching• Learn best strategy to play a fixed team
• Give high level advice to players at low frequency
• Focus on learning formations
• Learn when successful teams passed/kicked
Todd Hester
![Page 38: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/38.jpg)
Coaching• Learn best strategy to play a fixed team
• Give high level advice to players at low frequency
• Focus on learning formations
• Learn when successful teams passed/kicked
• Learn when opponent will pass and try to block
Todd Hester
![Page 39: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/39.jpg)
Coaching• Learn best strategy to play a fixed team
• Give high level advice to players at low frequency
• Focus on learning formations
• Learn when successful teams passed/kicked
• Learn when opponent will pass and try to block
• What if players switch roles?
Todd Hester
![Page 40: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/40.jpg)
Coaching• Learn best strategy to play a fixed team
• Give high level advice to players at low frequency
• Focus on learning formations
• Learn when successful teams passed/kicked
• Learn when opponent will pass and try to block
• What if players switch roles?
• Why just imitate another team?
Todd Hester
![Page 41: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/41.jpg)
Coaching• Learn best strategy to play a fixed team
• Give high level advice to players at low frequency
• Focus on learning formations
• Learn when successful teams passed/kicked
• Learn when opponent will pass and try to block
• What if players switch roles?
• Why just imitate another team?
• Other slides
Todd Hester
![Page 42: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/42.jpg)
Reinforcement Learning
• RL Slides
Todd Hester
![Page 43: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/43.jpg)
Reinforcement Learning
• RL Slides
• Extend to grid soccer
Todd Hester
![Page 44: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/44.jpg)
Reinforcement Learning
• RL Slides
• Extend to grid soccer
• Large state space, joint actions
Todd Hester
![Page 45: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/45.jpg)
Reinforcement Learning
• RL Slides
• Extend to grid soccer
• Large state space, joint actions
Todd Hester
![Page 46: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/46.jpg)
UT Austin Villa 2011
• Other slides
Todd Hester
![Page 47: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/47.jpg)
UT Austin Villa 2011
• Other slides
• Why not use CMA-ES on role positions as well?
Todd Hester
![Page 48: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/48.jpg)
UT Austin Villa 2011
• Other slides
• Why not use CMA-ES on role positions as well?
• Changes for 2012?
Todd Hester
![Page 49: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/49.jpg)
Kicking Under Uncertainty• Used by our SPL team
Todd Hester
![Page 50: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/50.jpg)
Kicking Under Uncertainty• Used by our SPL team
• Kick engine to kick at various distances/headings
Todd Hester
![Page 51: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/51.jpg)
Kicking Under Uncertainty• Used by our SPL team
• Kick engine to kick at various distances/headings
• Adjust to seen ball location
Todd Hester
![Page 52: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/52.jpg)
Kicking Under Uncertainty• Used by our SPL team
• Kick engine to kick at various distances/headings
• Adjust to seen ball location
• Select first kick that moves ball up field
Todd Hester
![Page 53: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/53.jpg)
Kicking Under Uncertainty• Used by our SPL team
• Kick engine to kick at various distances/headings
• Adjust to seen ball location
• Select first kick that moves ball up field
• Figure
Todd Hester
![Page 54: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/54.jpg)
Kicking Under Uncertainty• Used by our SPL team
• Kick engine to kick at various distances/headings
• Adjust to seen ball location
• Select first kick that moves ball up field
• Figure
• Emphasis on quickness
Todd Hester
![Page 55: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/55.jpg)
Kicking Under Uncertainty• Used by our SPL team
• Kick engine to kick at various distances/headings
• Adjust to seen ball location
• Select first kick that moves ball up field
• Figure
• Emphasis on quickness
• Now: Better model of opponents -> Know if we have moretime
Todd Hester
![Page 56: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/56.jpg)
Kicking Under Uncertainty• Used by our SPL team
• Kick engine to kick at various distances/headings
• Adjust to seen ball location
• Select first kick that moves ball up field
• Figure
• Emphasis on quickness
• Now: Better model of opponents -> Know if we have moretime
Todd Hester
![Page 57: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/57.jpg)
Learning Commentary
• David Chen and Ray Mooney
Todd Hester
![Page 58: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/58.jpg)
Coordination Graphs• n agents, each choose an action Ai
Todd Hester
![Page 59: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/59.jpg)
Coordination Graphs• n agents, each choose an action Ai
• A = A1 × . . .×An
Todd Hester
![Page 60: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/60.jpg)
Coordination Graphs• n agents, each choose an action Ai
• A = A1 × . . .×An
• Ri(A) 7→ IR
Todd Hester
![Page 61: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/61.jpg)
Coordination Graphs• n agents, each choose an action Ai
• A = A1 × . . .×An
• Ri(A) 7→ IR
• Coordination problem: R1 = . . . = Rn = R
Todd Hester
![Page 62: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/62.jpg)
Coordination Graphs• n agents, each choose an action Ai
• A = A1 × . . .×An
• Ri(A) 7→ IR
• Coordination problem: R1 = . . . = Rn = R
• Nash equilibrium: no agent could do better given whatothers are doing.
Todd Hester
![Page 63: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/63.jpg)
Coordination Graphs• n agents, each choose an action Ai
• A = A1 × . . .×An
• Ri(A) 7→ IR
• Coordination problem: R1 = . . . = Rn = R
• Nash equilibrium: no agent could do better given whatothers are doing.
• May be more than one (chicken)
Todd Hester
![Page 64: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/64.jpg)
Example from the paper• Understand the rule syntax
Todd Hester
![Page 65: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/65.jpg)
Example from the paper• Understand the rule syntax
• Form the coordination graph
Todd Hester
![Page 66: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/66.jpg)
Example from the paper• Understand the rule syntax
• Form the coordination graph
• First eliminate rules based on context
Todd Hester
![Page 67: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/67.jpg)
Example from the paper• Understand the rule syntax
• Form the coordination graph
• First eliminate rules based on context
• What does it mean for G3 to collect all relevant rules?
Todd Hester
![Page 68: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/68.jpg)
Example from the paper• Understand the rule syntax
• Form the coordination graph
• First eliminate rules based on context
• What does it mean for G3 to collect all relevant rules?
• What does it mean for G3 to maximize over all actions ofa1 and a2?
Todd Hester
![Page 69: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/69.jpg)
Example from the paper• Understand the rule syntax
• Form the coordination graph
• First eliminate rules based on context
• What does it mean for G3 to collect all relevant rules?
• What does it mean for G3 to maximize over all actions ofa1 and a2?
• How are the results propagated back?
Todd Hester
![Page 70: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/70.jpg)
Example from the paper• Understand the rule syntax
• Form the coordination graph
• First eliminate rules based on context
• What does it mean for G3 to collect all relevant rules?
• What does it mean for G3 to maximize over all actions ofa1 and a2?
• How are the results propagated back?
• Let’s try again with G1 eliminated first
Todd Hester
![Page 71: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/71.jpg)
Application to soccer• Make the world discrete by assigning roles, using high-
level predicates
Todd Hester
![Page 72: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/72.jpg)
Application to soccer• Make the world discrete by assigning roles, using high-
level predicates
• Assume global state information
Todd Hester
![Page 73: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/73.jpg)
Application to soccer• Make the world discrete by assigning roles, using high-
level predicates
• Assume global state information
• Finds pass sequences and starts players moving ahead oftime.
Todd Hester
![Page 74: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/74.jpg)
Application to soccer• Make the world discrete by assigning roles, using high-
level predicates
• Assume global state information
• Finds pass sequences and starts players moving ahead oftime.
• Note the results: with and without coordination.
Todd Hester
![Page 75: CS344M Autonomous Multiagent Systemstodd/cs344m/slides/week5a-pp4.pdfEvolved on huge (at the time) hypercube Todd Hester Darwin United More ambitious follow-up to Luke, 97 (made 2nd](https://reader033.vdocuments.net/reader033/viewer/2022042318/5f07f24b7e708231d41f8e40/html5/thumbnails/75.jpg)
Reactive Deliberation
• A hybrid approach
• Executor: carry out reactive behaviors
• Deliberator: evaluate possible high-level schema withparameters; generate bids
• Deliberator takes time, but something keeps happeningalways.
• In effect: deliberator commits to schema for some time
Todd Hester