![Page 1: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/1.jpg)
ADAPTIVE INTELLIGENT AGENT IN REAL-TIME STRATEGY GAMES
A Hybrid Online Case-Based Planning &Reinforcement Learning Approach
![Page 2: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/2.jpg)
Omar Enayet
Amr Saqr
Ahmed Atta
Abdelrahman Al-Ogail
Dr. Mostafa
Aref
Dr. Ibrahim Fathy
PROJECT MEMBERS
![Page 3: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/3.jpg)
Project Research Area & Domain Motivations. Problem Definition Objectives Related Work Our Methodology.
Offline Stage. Online Stage.
Testing and Results. Conclusion and Future Work. Demo. References.
AGENDA
![Page 4: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/4.jpg)
AI Learning
Make the machine learn.
AI Planning
Plan then re-plan according to new
givens.
Knowledge
SharingLet everyone know instantly what you
knew through experience.
PROJECT RESEARCH AREA
![Page 5: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/5.jpg)
RTS GamesReal-Time Strategy Games.
Severe Time Constraints – Real-Time AI – Many Objects – Imperfect Information – Micro-Actions
PROJECT DOMAIN
![Page 6: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/6.jpg)
RoboticsFor interest for
military which uses battle simulations in training programs.
War Simulation
For the corporation of robots.
Experimental Relevance
They constitute well-defined environments
to conduct experiments.
MOTIVATIONS
![Page 7: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/7.jpg)
Experience Loss Static ScriptsComputer AI relies on
static scripting techniques.
The Absence of sharing experience costs a lot
PROBLEM DEFINITION
![Page 8: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/8.jpg)
Predictability Non-Adaptability
Computer Opponent doesn’t adapt to
changes in human actions.
Computer Opponent actions easily
predicted.
PROBLEM CHALLENGES
![Page 9: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/9.jpg)
Adaptive A.I.Making the Computer
Opponent adapt to changes like human do.
Mobile Experience
Import/Export your experience !
OBJECTIVES
![Page 10: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/10.jpg)
•Eric Kok introduced : Adaptive Reinforcement Learning Agents in RTS Games, which merged BDI Agents technology with Reinforcement Learning, 2009
•Santi Ontanon introduced Darmok 2 which is an Online Case-Based Planning system designed to play Wargus in 2010.
•M.Johansen devised a CBR/RL system for learning micromanagement in real-time strategy games, 2009
RELATED WORK
![Page 11: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/11.jpg)
ARCHITECTURE OVERVIEW
![Page 12: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/12.jpg)
CASE REPRESENTATIONCase
3
0.6Performance
Behavior
BuildBase1TrainForce(TinyLandForce)TrainForce(TinyAirForce)Attack(NearWoodPeon)
Snippet
UnitExist(Peon)
Preconditions
PlayerBuildings > 0
Alive conditions
Rush-AttackStrategy
WinWargusGoal
NumberOfPlayerPeons = 10NumberOfPlayerCanonTower = 3
PlayerHasFortress = 1..
Shallow Features
PathExists = 1DistanceToEmeny = 40
.
.
Deep Features
0.8Prior Confidence
BeginningSituation
Eligibility
![Page 13: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/13.jpg)
PERCEPTION
![Page 14: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/14.jpg)
GAME STATE ANALYZER
![Page 15: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/15.jpg)
Offline Stage:Learning from
human demonstration before shipping
game
![Page 16: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/16.jpg)
CASE ABSTRACTION
![Page 17: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/17.jpg)
CASE ABSTRACTION (CONT’D) Simplify case complexity. Increase the flexibility.
Abstractor
Point(10, 137) Unguarded Region
![Page 18: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/18.jpg)
CASE ACQUISITION
![Page 19: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/19.jpg)
Generates Cases from Human’s game play.
AcquisitionAbstract Trace Casebase
CASE ACQUISITION (CONT’D)
![Page 20: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/20.jpg)
Online Stage:Learning what’s the
best to do while playing
![Page 21: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/21.jpg)
CASE RETRIEVAL
![Page 22: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/22.jpg)
CASE RETRIEVAL – CONT’D
RetrieverCase Base
Goal
Case
![Page 23: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/23.jpg)
CASE ADAPTATION
![Page 24: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/24.jpg)
Adapts Behaviors according to current game state. Removal of unnecessary actions. Adaptation for unsatisfied preconditions.
AdaptationBehavior Adapted Behavior
CASE ADAPTATION(CONT’D)
PlanBuild(Barracks)
PlanBuild(Great-Hall)
Train(Peon)Build(Barracks)
![Page 25: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/25.jpg)
ONLINE PLAN EXPANSION & EXECUTION
![Page 26: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/26.jpg)
ONLINE PLAN EXPANSION & EXECUTION (CONT’D)
![Page 27: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/27.jpg)
ACTION CONTROLLER
![Page 28: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/28.jpg)
CASE CONCRETIZATION
![Page 29: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/29.jpg)
CASE CONCRETIZATION (CONT’D) Adapt the abstract actions to suit current
situation.
Concreter Point(10, 137)Unguarded Region
![Page 30: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/30.jpg)
CASE REVISION
![Page 31: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/31.jpg)
CASE REVISION (CONT’D)
Uses reinforcement learning, TD-learning SARSA( )λ
ReviserCase
Used Case Evaluation
![Page 32: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/32.jpg)
TESTING AND RESULTS
Offline Le
arning
Adap
tation
Retriev
al
Abstr
action
Concre
tizati
on
Build
ing Pl
acemen
t
Respon
ds to
attack
s
Attack
s
Resourc
e Gath
ering
Traini
ng Fo
rces
0102030405060708090
100
Success Percentage
![Page 33: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/33.jpg)
CONCLUSION A Hybrid Architecture of case based reasoning and
reinforcement learning was introduced to play strategy games.
The architecture merged online case based planning with Sarsa(λ) with eligibility traces. The system showed promising simulation of human behavior; however it still needs a lot extra effort and testing to become industrially capable.
Also, the concept of an abstract case base was introduced which opens the door for generic AI engines for games which is never implemented till the date of writing of this document.
![Page 34: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/34.jpg)
Demo!
![Page 35: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/35.jpg)
FUTURE WORK1) Cooperative AI Agents.
2) Opponent Modelling.
3) Strategy visualization tool.
4) Generic situation assessment.
5) Learn weights of Game State through neural network.
6) Online I-Strategizers.
7) Generic Abstraction/ Concretization.
![Page 36: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/36.jpg)
WEB RESOURCES To get introduced for the whole project journey, evolution,
technical summaries, presentations, discussions, meeting minutes and others visit project blog:
http://rtsairesearch.wordpress.com/
For full materials of papers, technical summaries, documentations, articles, external links and running version of WARGUS (our test best) use the repository link:
svn checkout http://rtsairesearch.googlecode.com/svn/trunk/ rtsairesearch-read-only
For downloading the latest source code for the I-Strategizer Project, please use the following:
svn checkout http://istrategizer.googlecode.com/svn/trunk/ istrategizer-read-only
![Page 37: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/37.jpg)
REFERENCES [1] Martin Johansen Gunnerud. A CBR/RL system for learning
micromanagement in real-time strategy games. In Norwegian University of Science and Technology, 2009
[2] Santi Ontañón, Neha Sugandh, Kinshuk Mishra, Ashwin Ram. On-Line Case-Based Planning. In Computational Intelligence, 26(1):84-119, 2010.
[3] Brain Schwab. AI Game Engine Programming. Charles River Media, 2009.
[4] Santi Ontañón and Kane Bonnette and Prafulla Mahindrakar and Marco A. G´omez-Mart´ın and Katie Long and Jainarayan Radhakrishnan and Rushabh Shah and Ashwin Ram. Learning from Human Demonstrations for Real-Time Case-Based Planning. In AAAI 2008
[5] Ralph Bergmann and Wolfgang Wilke, On the role of abstraction in case-based reasoning
![Page 38: Adaptive Intelligent agent in real-time strategy games](https://reader036.vdocuments.net/reader036/viewer/2022062310/568164bb550346895dd6c858/html5/thumbnails/38.jpg)
REFERENCES – CONT. [6] Kinshuk Mishra, Santiago Santi Ontañón, and Ashwin Ram.
Situation Assessment for Plan Retrieval in Real-Time Strategy Games. In 9th European Conference on Case-Based Reasoning (ECCBR 2009), Trier, Germany.
[7] Neha Sugandh and Santiago Santi Ontañón and Ashwin Ram . On-Line Case-Based Plan Adaptation for Real-Time Strategy Games. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008)
[8] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, An Introduction. MIT press, 2005.
[9] Wikipedia, the free encyclopedia. http://www.wikipedia.com
[10] Michael Buro, Call for Research in RTS AI, AAAI 2004