reinforcement learning in real- time strategy games nick imrei supervisors: matthew mitchell &...
Post on 21-Dec-2015
216 views
TRANSCRIPT
Reinforcement Learning in Real-
Time Strategy Games
Nick Imrei
Supervisors: Matthew Mitchell & Martin Dick
Outline
Reasons What this research is about Motivation and Aim
Background RTS games Reinforcement Learning explained Applying RL to RTS
This project Methodology Evaluation Summary
Motivation and Aims
Problem: AI has been a neglected area – game developers
have adopted the “not broken so why fix it” philosophy
Internet Thrashing – my own experience Aim:
Use learning to develop a human-like player Simulate beginner → intermediate level play Use RL and A-life-like techniques
E.g. Black and White, Pengi [Scott]
RTS Games – The Domain
Two or more teams of individuals/cohorts in a war-like situation on a series of battlefields E.g. Command & Conquer, Starcraft, Age of Empires, Red Alert,
Empire Earth Teams can have a variety of:
Weapons Units Resources Buildings
Players required to manage all of the above to achieve the end goal.(Destroy all units, capture flag, etc.)
Challenges offered in RTS games Real time constraints on actions High level strategies combined with low-
level tactics Multiple goals and choices
The Aim and Approach
Create a human-like opponentRealisticDiverse behavior (not boring)This is difficult to do!
Tactics and Strategy Agents will be reactive to environmentLearn rather than code – Reinforcement
learning
The Approach Part 1 – Reinforcement Learning Reward and Penalty
Action Rewards / Penalties Penalize being shot Reward killing a player on the other team
Strategic Rewards / Penalties Securing / occupying a certain area Staying in certain group formations Destroying all enemy units
Aim to receive maximum reward over time Problem: Credit assignment
What rewards should be given to which behaviors?
The Approach Part 2 – Credit Assignment States and actions Decide on a state space and action space Assign values to
States, or States and Actions
Train the agent in this space
Reinforcement Learning example
Reinforcement Learning example
Why use Reinforcement Learning? Well suited to problems where there is a
delayed reward (tactics and strategy) The trained agent moves in (worst case)
linear time (reactive) Problems:
Large state spaces (state aggregation)Long training times (ER and shaping)
The Approach Part 3 – Getting Diversity
Agent
Agent state space
A-life-like behavior using aggregated state spaces
Research Summary:
Investigate this approach using a simple RTS game Issues:
Empirical Research Applying RL in a novel way Not using entire state space Need to investigate
Appropriate reward functions Appropriate state spaces
Problems with Training Will need lots of trials - the propagation problem No. trials can be reduced using Shaping [Mahadevan] and
Experience Replay [Lin] Self play – other possibilities include A* and human opponents
Tesauro, Samuel
Methodology
Hypothesis: “The combination of RL and reduced state spaces in
a rich (RTS) environment will lead to human-like gameplay”
Empirical investigation to test hypothesis Evaluate system behavior
Analyze the observed results Describe interesting phenomenon
Evaluation
Measure the diversity of strategies How big a change (and what type) is required to
change the behaviour – a qualitative analysis of this Success of strategies
I.e. what level of gameplay does it achieve Time to win, points scored, resembles humans
Compare to human strategies “10 requirements of a challenging and realistic
opponent” [Scott]
Summary thus far…
Interested in a human-level game program Want to avoid brittle, predictable programmed
solutions Search program space for most diverse
solutions using RL to direct search Allows specifications of results, without needing to
specify how this can be achieved
Evaluate the results
The Game – Maps and Terrain
2 Armies of equal amount on an n*n map.
Terrain: Grass, Trees, Boundary
Squares and Swamp All units can move on these
squares, however Different terrain types
affect a soldier’s attributes each in a different way
The Game – Soldiers
Soldier Attributes include: Sight Range Weapon Range Fatigue Speed Health Direction Relation Lines
Experiments Part 1: Hand-coded Strategies Create 8 different Hand-coded Strategies
Incl. Horde, Disperse, Central Defense, etc. Test their effectiveness based on:
Time taken to winTime taken to eliminate an enemy once
spottedDamage sustained when victorious
Results of Experiments Part 1
Units deployed closer resulted in quicker games. No strategy was consistently successful against
all others. The 3 most successful were:
Occupy Horde Central Defense
Strategies meant nothing once army sizes were > 150 on a 80*80 map.
Experiments Part 2: Control Architectures Centralized
All units are controlled by one entityOnly do what is commanded (no auto-
behavior)View area = Central controllers ViewscreenGroup FormationUnit SelectionUnit Commanding
Experiments Part 2: Control Architectures Localized
Units are independently maneuvered controlled ala Artificial-life
Viewing space is only what they see individually
Formation ; CohortsUnit Selection & Movement done via an
A-life State Machine
Experiments Part 2: Control Architectures Testing:
Given the best 3 techniques from part1,Program them in a centralized and localized
mannerBase their effectiveness on criteria from part 1Observe the realism of the 6 new hand-coded
strategies
Results of Experiments Part 2
As individual unit sight and weapon range increased, localized performed better.
A-life performed better on rougher terrain, whereas centralized often got stuck.
Centralized formation takes less time, hence it did better in situations where the
ArmySize : MapSize ratio increased.
Results of Experiments Part 2
Realism Evaluation:Localized resembles more a group of soldiersCentralized better resembles human
gameplay. Given its success a local framework is
used as a template for the learning agent
Learning Agents - Architecture
Each agent will work off the same learning table.
Is expected to speed up learning – by learning from everyone’s mistakes rather than just your own
Agents are trained against all opponents from parts 1 and 2
Learning Agents – Representing the world States
Divide sight range up into sectionsEach section can have a combination of an
ally, a health spot, an enemy or none.(On or off a health spot) 288 possible world
states. Actions
Move & Shoot (left, forward, right, back)
Learning Agents – Representing the world Rewards:
Positive: Shooting an enemy Moving to a health spot
Negative: Being shot / killed Being on a health spot when health is full
Reinforcement: Q(s,a) = R(s,a) + γ Σs’P(s’|s,a)Q(s,a,s’)
Results of Experiments Part 3
Learning of behaviors was achieved in only a few simulations
Agents developed the following behaviors: Shoot when seen unless health is low If Health is low, move to health spot Units form a health-spot queue Diversion of a centralized opponents attention
Learning agents were consistently successful against all others bar centralized hording.
Agents told what to do – not how to do it Human testing didn’t prove too successful!
Conclusions
A localized approach was found to be more successful overall than a centralized one.
Given the sans base and resource element of the game, the all out aggressive strategies faired the best
Learning strategies were successful against most programmed ones
Diversion and health spot sharing behaviors were observed
Future Work
Extending the RTS game so it has:Resources and resource gatheringDifferent Unit typesBase building and maintenance
Testing the RL/A-life framework in other game genres including Role Playing Games, Sim games and Sports.
References
Bob Scott. The illusion of intelligence. AI Game Programming Wisdom, pages 16–20, 2002.
Sridhar Mahadevan and Jonathan Connell. Automatic programming of behavior-based robots using reinforcement learning. Artificial Intelligence 55, pages 311–364, 1992.
L Lin. Reinforcement learning for robots using neural networks. PhD thesis, School of Computer Science, Carnegie Mellon University, Pittsburgh USA, 1993.
Mark Bishop Ring. Continual Learning in Reinforcement Environments. MIT Press, 1994.
Stay Tuned!
For more information, seehttp://www.csse.monash.edu.au/~ngi/
Thanks for listening!