intelligent online case based planning agent model for rts games conference presentation
DESCRIPTION
TRANSCRIPT
Intelligent Online Case Based Planning Agent Model for RTS
Games
Ibrahim Fathy, Mostafa Aref, Omar Enayet, and Abdelrahman Al-Ogail
Faculty of Computer and Information Sciences
Ain-Shams University ; Cairo ; Egypt
Introduction.Problem definition.Objectives.Intelligent OLCBP Agent Model.OLCBP\RL Hybridization .Experiments and Results.Conclusion & Future Work.
Agenda
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Online Case Based Planning is an architecture based on
CBR.
It addresses issues of plan acquisition, on-line plan
execution, interleaved planning and execution and on-line
plan adaptation.
Darmok is a previous system that applies this -without
revision- to RTS Games.
INTRODUCTION: ONLINE CASE-BASED PLANNING
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
A Reinforcement Learning Approach.
Combines temporal difference learning technique “One-
Step SARSA” with eligibility traces to learn state-action
pair values effectively.
It’s an on-policy method.
INTRODUCTION: SARSA(λ) LEARNING
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Considered a challenging domain due to: Severe Time
Constraints – Real-Time AI – Many Objects – Imperfect
Information – Micro-Actions
INTRODUCTION: Real-Time Strategy Games
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Research in learning and planning in real-time
strategy (RTS) games is very interesting.
The research on online case-based planning in RTS
Games does not include the capability of online
learning from experience.
The knowledge certainty remains constant, which
leads to inefficient decisions.
Problem Definition
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Proposing an intelligent agent model based on both
online case-based planning (OLCBP) and
reinforcement learning (RL) techniques.
Increasing the certainty of the case base by learning
from experience.
Increasing both efficiency and effectiveness of the
plan decision making process.
Evaluating the model using empirical simulation on
Wargus. ( A clone of the well-known strategy game
Warcraft 2)
Objectives
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Agent Architecture – Abstract View
Environment ( RTS Game “Wargus”)
Case Base
Offline Phase (Case
Acquisition)
Plan Expansion/Execut
ion Module
Online Case-Based
Learner
Behaviors
Goals Evaluated Case
Retrieved Case
Traces
Cases
Actions
Online Phase
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Agent Architecture – Detailed View
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Case RepresentationCase Goal Strategy SituationState
Shallow Features
Deep Features
BehaviorPre-Conditions
Alive-Conditions
Success-Conditions
Snippet
Learning ParametersCertainty
FactorEligibility
Prior Confidence
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Case Representation - Example
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Case Retrieval Since the evaluation of the case base is changed, the case
retrieval algorithm must change also. The case with the best predicted performance will be
retrieved to be executed.
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
OLCBP/RL Hybridization Algorithm
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
RL Algorithm used: SARSA(λ)
OLCBP/RL Hybridization: The Mapping.
Experiment: Case Study
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Learning Parameter
Value used
Learning Rate (α) 0.1
Discount Rate (γ ) 0.8
Decay Rate (λ ) 0.5
Exploration Rate 0.1
Results of Attack1 & Attack2
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Results of BuildArmy1 & BuildArmy2
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Agent has learnt that building a smaller heavy army in that specific situation (the existence of a towers defense) is more preferable than building a larger light army. Similarly, the agent can evaluate the entire case base and learn the right choices.
Results (Cont’d)
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Online case-based planning was hybridized with reinforcement learning in order to introduce an intelligent agent capable of planning and learning online using temporal difference with eligibility traces: Sarsa (λ) algorithm.
The empirical evaluation has shown that the proposed model –unlike Darmok System - increases the certainty of the case base by learning from experience, and hence the process of decision making for selecting more efficient, effective and successful plans.
The Paper –Conclusion
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Implementing a prototype based on the proposed model.
Developing a strategy/case base visualization tool capable of visualizing agent’s preferred playing strategy according to its learning history. This will help in tracking the learning curve of the agent.
Finally, designing and developing a multi-agent system where agents are able to share their experiences together.
The Paper –Future Work
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10
Thank You !
Questions ?
Intelligent Online Case-Based Planning Agent Model for RTS Games – ISDA’10