Reinforcement Learning approach for Real Time StrategyGames Battle city and S3

Harshit Sethya, Amit Patelb

aCTO of Gymtrekker Fitness Private Limited,Mumbai, India, Email: hsethy1@gmail.com

bAssistant Professor, Department of Computer Science and Engineering, RGUKT IIIT Nuzvid,Krishna-521202 India, Email: amtptl93@gmail.com

In this paper we proposed reinforcement learning algorithms with the generalized reward function. In ourproposed method we use Q-learning and SARSA algorithms with generalised reward function to train the rein-forcement learning agent. We evaluated the performance of our proposed algorithms on two real-time strategygames called BattleCity and S3. There are two main advantages of having such an approach as compared to otherworks in RTS. (1) We can ignore the concept of a simulator which is often game specific and is usually hard codedin any type of RTS games (2) our system can learn from interaction with any opponents and quickly change thestrategy according to the opponents and do not need any human traces as used in previous works.

Keywords : Reinforcement learning, Machine learning, Real time strategy, Artificial intelligence.

1. INTRODUCTION

Existence of a good artificial intelligence(AI)technique in the background of a game is one ofthe major factor for the fun and re-play abilityin commercial computer games. Although AI hasbeen applied successfully in several games suchas chess, backgammon or checkers when it comesto real-time games the pre-defined scripts whichis usually used to simulate the artificial intelli-gence in chess, backgammon etc [11]. does notseem to work. This is because in real-time gamesdecisions has to be made in real-time as well asthe search space is huge and as such they do notcontain any true AI for learning [2]. Traditionalplanning approaches are difficult in case of RTSgames because they have various factors like hugedecision spaces, adversarial domains, partially-observable, non-deterministic and real-time, (realtime means while deciding the best actions, thegame continues running and states change simul-taneously).

1.1. Real Time Strategy GamesToday game developing companies have started

showing more interest in RTS games. Unlike turnbased strategy games, where one has the ability totake ones own time, in real time strategy games,

all movement, construction, combat etc., are alloccurring in real time. In a typical RTS game,the screen contains a map area which consists ofthe game world with buildings, units and terrain.There are usually several players in an RTS game.Other than the players there are various gameentities called participants, units and structures.These are under the control of the players and theplayers need to save their assets and/or destroyassets of the opponent players by making use oftheir control over the entities. We are using 2RTS games (1) BattleCity and (2) S3 game forour evaluation. A snapshot of two RTS gamescalled BattleCity and S3 are given in Figure 1.

1.2. BattleCity GameBattleCity is a multidirectional shooter video

game, which can be played using two basic ac-tions Move and Fire. The player, controlling atank, must destroy enemy tanks or enemy baseand also protect its own base. Player can movetank in four directions (left, right, up and down)and fire bullets in whichever direction the tanklast moved, while bases are static. There arethree types of obstacle. (1) Brick wall tank candestroy it by firing this type wall. (2) Marblewall tank cant destroy it by firing. (3) Water

1

arX

iv:1

602.

0493

6v1

[cs

.AI]

16

Feb

2016

2 Harshit Sethy, Amit Patel

bodies tank can fire through it. Tank cant passthrough any of above obstacle. Only brick wallcan be destroyed by tank so after destroying tankcan pass through it.

(a)

(b)

Figure 1. (a)Snapshot of a BattleCity Game(b)Snapshot of an S3 Game

1.3. S3 GameS3 is a real-time strategy game where each

players goal is to remain alive after destroyingthe rest of the players. Four basic actions inthis game are Harvest : i.e., to gather resources(gold and wood), Build : to build buildings (Bar-rack, Blacksmith, Tower etc) ,Train: to producetroops (archers, footmen, catapults, knights), At-tack : for attacking enemy.

This paper is structured as follows. Apart fromintroduction, there are five more sections.In sec-tion 2 highlights the review of related works. Insection 3 we discuss about reinforcement learningtechniques in real-time-strategy games and out-line the various learning algorithms used in rein-forcement learning. In section 4 we outline im-plementation details related to the proposed re-inforcement learning algorithms with the gener-alized reward function for two real-time-strategygames (1) BattleCity and (2) S3 game. Section5 discusses about the experimental result relatedto our proposed work for BattleCity and S3. Weconclude with section 6.

2. Related Work

One of the major works using Online case-based planning [6] techniques for Real Time Strat-egy Games was published in [9]. On-line case-based planning revises case based planning

for strategic real-time domains involving on-lineplanning.

In [8] a case-based planning system calledDarmok2 is introduced that can play RTSgames. They introduced a set of algorithmsthat can be used to learn plans, represented aspetri-nets, from one or more human demon-strations. Another work by the same authorswhich uses Darmok2 but addresses the issues ofplan acquisition, on-line plan execution, inter-leaved planning and execution and on-line planadaptation is [7].

In [3] the authors summarize their work in ex-ploring the use of the first order inductive learning(FOIL) algorithm for learning rules which can beused to represent opponent strategies.

In [13] the authors improve Darmok2 usinginformation related to sensors of the game. Werefer to that work as PR-Model in this paper.PR-model is capable of learning how to play RTSgames by observing human demonstrations. Us-ing human traces PR-model makes plans to playgames. Prioritize the plan according to the feed-back of the game and feedbacks are decided us-ing some rule which depends on the sensors of thegame.

Reinforcement Learning approach for Real Time Strategy Games like Battle city and S3 3

Drawbacks of all case based learning [4] ap-proaches as mentioned above are (1) It requiresexpert demonstrations for making plans (2) aftertraining is done, no further learning takes place(3) to cover large state spaces it would requirelarge number of rules in the plan base (4) no ex-ploration for optimal solution. Only follows hu-man traces.. Stefan Wender [5] uses Reinforce-ment Learning for City Site Selection in the Turn-Based Strategy Game Civilization IV. Civiliza-tion IV is the strategy game it is a turn-basedgame while Battle City is Real time game.

Stefan Wender [5] uses Reinforcement Learningfor City Site Selection in the Turn-Based Strat-egy Game Civilization IV. Civilization IV is thestrategy game similar to S3 but it is a turn-basedgame while S3 is Real time multi agent game.

In this paper we aim to do away withthe hard coded simulator and propose alearning approach based on Reinforcement

Learning [1](RL) wherein sensor informationfrom the current game-state is used to select thebest action. Reinforcement learning is used be-cause of its advantages over previous strategies.Specifically (1) RL cuts out the need to man-ually specify rules. RL agents learn simply byplaying the game against other human players oreven other RL agents (2) for large state spaces,RL can be combined with a function approxima-tor such as a neural network, to approximate theevaluation function (3) RL agent always exploresfor optimal solution to reach the goal (4) RL hasbeen applied widely to many other fields, suchas robotics, board games ,turn based games andsingle agent games with great results, but hardlyever on RTS multi-agent games.

3. Reinforcement Learning

Reinforcement Learning [1] is the field of Ma-chine Learning which deals with what to do, howto map situations to actions so as to maximizea numerical reward signal.The learner does notknow which actions to take, as in most formsof machine learning, but instead must discoverwhich actions gives the most reward by applyingthem. In the most interesting and challengingcases, actions may affect not only the immediate

reward but also the next situation and, throughthat, all subsequent rewards.

With comparing reinforcement learning [12] toRTS game environment an AI player learns byinteracting with the environment and observingthe feed-backs of these interactions. This is sameas the fundamental way in which humans (andanimals) learn. As a human, we can perform ac-tions and observe the results of these actions onthe environment. The same way RL-agent inter-acts with the environment and observes the re-sult and assign the reward or penalty to state orstate-action pair according to the desirability ofthe resultant state.

(a)

(b)

Figure 2. (a)Reinforcement Learning (b) Archi-tecture for the Reinforcement Learning

3.1. Reinforcement Learning ArchitectureRL Architecture has two main characteristics;

one is learning and the other is playing with

4 Harshit Sethy, Amit Patel

the learnt experiences. Initially RLearner has noKnowledge about the game. So it does randomactions and observe the resultant state using somesensor information of the game and give feedback(in the form of reward which is further used tocalculate the Q-Values for the state-action pairsor Q-Table) of that action to the previous stateaccording to the desirability of the current state.Q-Values of the state-action pairs are known asQ-Table which define a policy. After every actionpolicy updates Q-Values for the state action pairs(Q-Table) this policy is used to predict the bestaction while playing the game. RL agent learnswhile playing so it again gives feedback and thewhole process it going on till the end of the game.

3.2. Basic components of RLReinforcement learning contains five basic com-

ponents which are as listed below.

1. a set of environment states S

2. a set of actions A

3. rules of transitioning between states

4. rules that determine the scalar immediatereward of a transition (Reward Functions)

5. rules that describe what the agent observes(Value Functions)

3.2.1. Reward FunctionThe scalar value which represents the degree to

which a state or action is desirable is known asreward. This scalar reward is assigned to the ac-tion for the particular transition and the resultantstate of the game. If the resultant state is desir-able and safe then positive scalar value as rewardwill be assigned to that action otherwise if state isnot safe or undesirable then some negative scalarvalue as negative reward will be assigned to thataction. We are using 2 types of Reward function(1) Conditional Reward function (2) GeneralisedReward function.

3.2.2. Value FunctionValue Functions are used for mapping from

states or from state-action pairs to real numbers,where the value of a state represents the long-term reward achieved starting from that state (or

state-action), and executing a particular policy.It estimates how good a particular action will bein a given state, or what the return for that ac-tion is expected to be. There are two type ofvalue functions.

1. V π(s) is the value of a state ’s’ under policyπ. The expected return when starting in sand following π thereafter.

2. Qπ(s, a) is the value of taking action ’a’ instate ’s’ under a policy π. The expected re-turn when starting from s taking the actiona and thereafter following policy π.

There are two methods to define these valuefunctions:

1. Monte Carlo [1] Method : In this methodthe agent would need to wait until the finalreward was received before any state-actionpair values can be updated. Once the finalreward is received, the path taken to reachthe final state would need to be traced backand each value updated.

V (st)← V (st) + α[Rt − V (st)] (1)

where st is the state visited at timet, Rt is the reward after time t and α is aconstant parameter.

2. Temporal Difference [1] Method : It is usedto estimate the value functions after eachstep. An estimate of the final reward is cal-culated at each state and the state-actionvalue updated for every step of the way.This reflects a more realistic assignment ofrewards to actions compared to MC, whichupdates all actions at the end directly. TDLearning is nothing but the combinationof dynamic programming with the MonteCarlo method. The formula related to TDlearning is given as

V (st)← V (st)+α[rt+1+γV (st+1)−V (st)](2)

where rt+1 is the observed reward attime t+1.

Reinforcement Learning approach for Real Time Strategy Games like Battle city and S3 5

3.3. Sensor representation for S3 and Bat-tleCity Game

Figure 3. Snapshot of S3, BattleCity Games andthere current 2D maps

We are using two types of sensor informationfor assigning reward in battle city game which areexplained as follows;

1. EnemyInline: If enemy position is directlyin line with player without any block or wallthen sensor is represented by number 2. Ifthere is a wall or block between enemy andplayer then sensor is represented by num-ber 1. If enemy position is not in line withplayer then sensor is 0.

2. EnemyBaseInline: This sensor informationis represented in the same way as above butinstead of taking into consideration position

of enemy, position of enemy-base is takeninto account. If enemy-base position is di-rectly in line with player without any blockor wall then sensor is represented by num-ber 2. If there is a wall or block betweenenemy-base and player then sensor is repre-sented by number 1. If enemy-base positionis not in line with player then sensor is 0.

Sensor information for S3 game

1. Get the current map and store it in a twodimensional array.

2. Gold and Wood sensors are retrieved fromcurrent game-state.

3. Number of peasant and footmen entities forboth enemies and player are retrieved fromentities state.

4. Update two dimensional array with staticentities like goldmine position with ’g’, andbuildings with ’b’.

So far we have outlined our method of obtain-ing sensor information related to two real-timestrategy games, BattleCity and S3.

3.4. Action Selection PoliciesWe have the following action selections policies

which can be used to select desired action accord-ing to the behavior of that particular policy

1. ε−greedy : Most of the time the actionwith the highest estimated reward is cho-sen, called the greediest action. But, with asmall probability ε, an action is selected atrandom to ensure optimal actions are dis-covered.

2. ε−soft : Very similar to ε−greedy. Thebest action is selected with probability 1−εand the rest of the time a random action ischosen uniformly.

3. softmax : One drawback of the abovemethods is that they select random actionswith some probability. So there is a casewhen the worst possible action is selectedas the second best. Softmax remedies this

6 Harshit Sethy, Amit Patel

by assigning a rank or weight to each of theactions, according to their action-value es-timate. So the worst actions are unlikely tobe chosen.

3.5. Steps while learning1. The Rlearner observes an input Game state.

2. The Rlearner then creates a new policybased on the dimensions of the world.

3. Set the parameters (α, γ, ε and number ofepisodes) for the Rlearner and start learn-ing.

4. Start running epochs. You can optionallyrun each epoch individually.

One epoch contains following steps.

1. An action is determined by a decision mak-ing function (e.g. ε−greedy).

2. The action is performed.

3. The Rlearner receives a scalar reward or re-inforcement from the environment accord-ing to reward function.

4. Information about the reward given for thatstate / action pair is recorded.

5. Update the Q-values in Q-table Accordingto Learning Algorithm(e.g. Q-learning orSARSA).

4. Proposed learning algorithm

In this section we outline our proposed learn-ing algorithms which we integrated into the twoRTS games Battlecity and S3. We also providethe implementation details related to selection ofparameters and reward functions.

4.1. ParametersThis section contains the information regarding

the reward algorithms and its parameters whichwe use for the two game BattleCity and S3.

• Learning Rate α : The learning rate0 < α < 1 determines what fraction of theold estimate will be updated with the new

estimate. α = 0 will stop the RL-agent fromlearning anything while α = 1 will com-pletely change the previous values with thenew one.

• Discount Factor γ : The discount factor0 < γ < 1 determines what fraction of theupcoming reward values will be consideredfor evaluation. For γ = 0 all the upcom-ing rewards are ignored. For γ = 1 meansthe RL-Agent will consider the current andupcoming rewards as equal weightage.

• Exploration Rate ε : In action selec-tion policies there is one policy called as εgreedy method which uses the explorationrate 0 < ε < 1 for determining the ratiobetween the exploration and exploitation.We are using ε greedy method for selectingthe best action and to maintain the balancebetween exploration and exploitation.

4.2. Reward function for BattleCityAlgorithm 1: Reward function is for calcu-

lating reward after performing action on currentstate. According to the result of the action re-ward or penalty are assigned. In steps 1 to 9 getthe positions (x-y co-ordinates) of the player, en-emy and enemy base on the map. In steps 10 to16 if game is over and winner is the RL-Agent(player) then add the reward to the total reward(newReward) else deduct penalty from the totalreward. In steps 17 to 18 if enemy is in line withthe RL-Agent deduct penalty from total rewardso it always tries not to be in line with enemy.In steps 19 to 21 if enemy base is in line withthe RL-Agent then calculate the distance betweenthe enemy base and RL-Agent and deduct from2 times of reward and add to total reward. So itpushes the RL-Agent to come closer to the enemybase. Steps 22 to 24 gives the generalized rewardfunction which makes the RL-Agent quickly at-tack the enemy base and prevent attack by theenemy.

%

4.3. Reward function for S3Algorithm 2: In step 1 to 6 get the sensors

related to total gold, total wood and size of troops

Reinforcement Learning approach for Real Time Strategy Games like Battle city and S3 7

Algorithm 1: calcReward for BattleCity

Input: state :- contains positions of entities, reward, penaltysensorsList :- contains sensors of game domain.gameState :- contains state of game is running or notOutput: Reward

1 Playerx = null, Playery = null, Enemyx = null, Enemyy = null ;2 EnemyBasex = null, EnemyBasey = null, winner = null ;3 newReward = 0, distance = 0 ;4 Playerx = getPositionx(state,player) ;5 Playery = getPositiony(state,player) ;6 Enemyx = getPositionx(state,enemy) ;7 Enemyy = getPositiony(state,enemy) ;8 EnemyBasex = getPositionx(state,enemybase) ;9 EnemyBasey = getPositiony(state,enemybase) ;

10 if gameState == "end" then11 winner = getWinner() ;12 if winner == "player" then13 newReward = newReward + reward ;

14 else15 newReward = newReward - penalty ;

16 else17 if sensorList[EnemyInline]==2 then18 newReward = newReward - penalty ;

19 if sensorList[EnemyBaseInline]==2 then

20 distance = 2√

(EnemyBasex − Playerx)2 + (EnemyBasey − Playery)2 ;

21 newReward = newReward + 2 × reward - distance ;

22 newReward = newReward - 4 × distance ;

23 distance = 2√

(Enemyx − Playerx)2 + (Enemyy − Playery)2 ;

24 newReward = newReward + 4 × distance ;

25 return newReward ;

of the player and enemy. In steps 7 to 11 if gameis over and winner is the RL-Agent (player) thenadd the reward to the total reward (newReward)else deduct penalty from the total reward. Insteps 12 to 14 and 17 to 18 if gold and wood forplayer is greater than enemy than add reward tothe total reward otherwise deduct penalty fromtotal reward so it always tries to increase the goldand wood with compare to enemy. In steps 21to 22 if Player troop is bigger than the Enemytroop then add the twice of reward to the totalreward (newReward) else deduct twice of penaltyfrom the total reward. So it pushes the RL-Agentto Attack or build the army to increase the sizeof troop as compared to the enemy. In step 25

Return the total reward.

5. Experimental Results

In the previous section we have discussed howwe successfully applied reinforcement learning intwo real-time strategy games called BattleCityand S3. In this section we outline the experi-mental results related to reinforcement learningin BattleCity and S3.

5.1. BattleCity:We evaluated the performance of RL-Agent

with the help of various maps (e.g. Bridge-26x18,Bridge-metal-26x18, Bridges-34x26 ) as well as

8 Harshit Sethy, Amit Patel

Algorithm 2: calcReward for S3

Input: state :- contains positions of entities, reward, penaltyGlobal access to: sensorsList :- contains sensors of game domaingameState :- contains state of game is running or notOutput: Reward

1 Playerg = 0, Playerw = 0, Enemyg = 0, Enemyw = 0 EnemyTroopLength = 0, PlayerTroopLength = 0,winner = null newReward = 0 Playerg = player.getGold() ;

2 Playerw = player.getWood() ;3 Enemyg = enemy.getGold() ;4 Enemyw = enemy.getWood() ;5 EnemyTroopLength = enemyTroop.size() ;6 PlayerTroopLength = playerTroop.size() ;7 if gameState == "end" then8 winner = getWinner() if winner == "player" then9 newReward = newReward + reward

10 else11 newReward = newReward - penalty

12 else13 if Playerg > Enemyg then14 newReward = newReward + reward

15 else16 newReward = newReward - penalty

17 if Playerw > Enemyw then18 newReward = newReward + reward

19 else20 newReward = newReward - penalty

21 if PlayerTroopLength > EnemyTroopLength then22 newReward = newReward + 2*reward

23 else24 newReward = newReward - 2*penalty

25 return newReward

with two types of opponents called AI-Randomand AI-Follower in each map. We observedthat the Reinforcement Learning Agent won morethan 90% games when played against both op-ponents( AI-Random and AI-Follower) in sim-ple maps and about 80% to 90% when playedagainst AI-Random in complex maps and 60%to 80% when played against AI-Follower in com-plex maps. Statistics about the performance ofthe SARSA[1], Q-Learning[1] and Darmok2 in thevarious maps are represented below in the formof graphs. We observed that performance of RL-Agent under SARSA Learning algorithm is better

than other techniques and also RL-agent trainedby SARSA algorithm takes less time to win thegame.

We performed our evaluation for BattleCitygame against two opponents AI-Random and AI-Follower with three different maps. AI-Randomis the built-in AI which selects random actionalways and AI-Follower is tough to compete be-cause it always follows the opponent and fires atit. It is clear from the experimental results thatreinforcement learning agent with the SARSA [1]algorithm performs better than other techniqueslike Q-Learning [1] and online case based learn-

Reinforcement Learning approach for Real Time Strategy Games like Battle city and S3 9

ing based on Darmok2 [10]. Statistics relatedto performance are given below in the form ofgraphs. Statistics are represented using two typesof graphs. One is time (in milliseconds) takento win the game versus episodes. X-axis repre-sents the number of episode and Y-axis representsthe time in milliseconds. The other is number ofgames won versus episode. Here also X-axis rep-resents the number of episodes and Y-axis rep-resents the total number of games won till thatepisode.

5.1.1. Map: Bridge-26x18This map size is 26x18 (refer Figure 4) so to-

tal state space for this map is total combinationof the x − y co-ordinates of the player and en-emy which is 262x182. This map has a marblewall in between which the tank cannot destroyby firing. So this is an advantage for the tank tohide from opponents and attack when opponentsenters their side.

Figure 4. Map:Bridge-26x18

5.1.2. Map: Bridges-34x24This is the most complex map (refer Figure 9)

among all on which we have performed our eval-uation because of its size and the structure. It isa 34x24 map and it has 342x242 search spaces. Itcontains many brick wall and water bodies. Brickwall can be destroyed by firing. Its size and waterbodies makes it a difficult and complex map.

Figure 5. Map: Bridge-26x18 Against AI-Follower

Figure 6. Map: Bridge-26x18 Against AI-Random

10 Harshit Sethy, Amit Patel

Figure 7. Map: Bridge-metal-26x18 Against AI-Random

Figure 8. Map: Bridge-metal-26x18 Against AI-Follower

Figure 9. Map:Bridge-Metal-34x24

In time versus episodes graph (refer Figure 10and 11) the plot (refer Figure 6 and 5) is show-ing that time to win the game for all strategiesvaries for every episodes. This map has more wa-ter bodies so it is difficult to learn a strategy towin quickly. Against AI-random the performanceof all the strategies are close while in case ofAI-follower SARSA performs well and wins moregame in compared to Q-learning and Darmok2.

Figure 10. Map: Bridges-34x24 Against AI-Random

Reinforcement Learning approach for Real Time Strategy Games like Battle city and S3 11

Figure 11. Map: Bridges-34x24 Against AI-Follower

5.2. S3The maps related to S3 are more complex

than that of Battlecity. We evaluated our ap-proach on various maps against several built-in AI player. In our experiments we built RLagent for S3 game using relative reward func-tion with the Q-learning and SARSA approach asdiscussed earlier. RL-agent learn by playing 10games against built-in-ai called ai-catapult-rushfor the simple map NWTR1 (refer Figure 12) us-ing two approaches Q-Learning and SARSA. Thestate-action pair values (Q-Values) are updatedwhile playing (or Learning as discussed earlierRL-Agent also learns while playing). Using thisupdated Q-Values RL-Agent plays games againstai-catapult-rush as well as another type of built-in-ai called ai-rush.

• ai-catapult-rush is the built-in-ai thatbuilds barracks and lumber-mills at thestarting, this has two peasants for har-vesting gold, and two for harvesting wood.Then it starts building catapults nonstopand also attacks after a while. After some-time it increases the number of peasants to3, and starts building the second barrack.It also looks for goldmines where there gold

is still available. Also, it sends catapults toattack enemies.

• ai-rush is the built-in-ai that builds a bar-rack at the starting. There are two peas-ants at the starting for harvesting gold andwood. After building the barrack ai-rushtrains the footmen. When there are twotrained footmen it starts attacking.

Figure 12. Snapshot of an S3 Game Map:GOW

For our experiment we used three type of maps(refer Figure 3, 1 and 12) according to diffi-culty level (easy-NWTR2, medium-NWTR6 anddifficult-GOW). We performed our experimentswith five games against two built-in-ai whereinthe two approaches are Q-Learning and SARSAfor each map. The comparison statistics are givenin Table 1. We observed that RL-agent withSARSA wins most of the games. Q-Learning andthe previous approach (Darmok2) [10] performsalmost the same but not better than SARSA. ForS3 also SARSA gives the best results.

Table 1 shows the results comparison. By ana-lyzing the results shown in Table 1 we can see thatin most of the maps SARSA has won or drawnthe game. The maps where it has lost we foundthat the built-in-ai was a quick attacker and RL-agent was not able to produce enough number

12 Harshit Sethy, Amit Patel

Table 1Comparison of SARSA and Q-Learning with Darmok2

map Approach Epoch 1 Epoch 2 Epoch 3 Epoch 4 Epoch 5against ai-catapult

NWTR2 SARSA won won won won wonNWTR2 Q-Learning lost won won draw wonNWTR2 Darmok2 won draw won won lostNWTR6 SARSA lost draw won won wonNWTR6 Q-Learning won lost draw lost wonNWTR6 Darmok2 won lost won lost won

GOW SARSA draw lost won draw wonGOW Q-Learning lost lost won lost wonGOW Darmok2 won lost won lost draw

against ai-rush

NWTR2 SARSA won won won won wonNWTR2 Q-Learning won draw won won wonNWTR2 Darmok2 won won won lost wonNWTR6 SARSA won draw won won wonNWTR6 Q-Learning lost lost won won wonNWTR6 Darmok2 won draw won lost won

GOW SARSA draw won won won wonGOW Q-Learning lost won draw won wonGOW Darmok2 won lost won lost won

of troops to defend while the enemy was attack-ing. The RL agent was basically trying to find away to enter through the wall of trees. In somemaps we have shown the results as drawn. Thismeans that resources like wood and gold of bothplayer and enemy got finished and only peasantswere left out at both the sides and they cannotdo anything without the gold and wood.

When compared to previous research onDarmok2 [10], where pre-prepared strategies areused to play the game and plan adaption mod-ule is used to switch strategies in this researchRL-Agent quickly switches the strategies whileplaying, even though we used a simple map fortraining the RL-Agent.

6. Conclusions

In this paper we proposed a reinforcementlearning model for real-time strategy games. Inorder to achieve this end we make use of two re-inforcement learning algorithms SARSA and Q-Learning. The idea is to get the best action usingone of the RL algorithms so as to not make use ofthe traces generated by the players. In previous

works on real-time strategy games using ”on linecase based learning” human traces form an im-portant component in the learning process. In theproposed method we are not making use of anyprevious knowledge like traces and therefore wefollow an unsupervised approach. This researchis with regard to getting the best action usingtwo algorithms (SARSA and Q-Learning) whichcomes under Reinforcement Learning without thetraces generated by the player as proposed in theprevious work ”on line case based learning” us-ing Darmok2. Another major contribution of ourwork is the reward function. Rewards are cal-culated by two types of reward functions calledconditional and generalized reward function. Thesensor information related to game is used for cal-culating the rewards. The reward values are fur-ther used by the two RL algorithms

SARSA and Q-Learning. These algorithmsmake policies according to the reward for thestate-action pair. RL agent choose the action us-ing these policies. We evaluated our approachsuccessfully in two different game domains (Bat-tleCity and S3) and observed that reinforce-ment learning performs better than previous ap-

Reinforcement Learning approach for Real Time Strategy Games like Battle city and S3 13

proaches in terms of learning time and winningratio. In particular SARSA algorithm takes lessertime to learn and start winning very quickly thanQ-Learning and that too for complex maps.

REFERENCES

1. R. S. Sutton and A. G. Barto,ReinforcementLearning: An Introduction. A book publisherMIT Press; 1998.

2. Katie Long Genter. Using first order inductivelearning as an alternative to a simulator in agame arficial intelligence. under-graduate the-sis. In Georgia Institute of Technology, pages1–2, May 2009.

3. Katie Long Genter, Santiago Ontanon, andAshwin Ram. Learning opponent strategiesthrough first order induction. In : FLAIRSConference, pages 1–2, 2011.

4. P. P. Gomez-Martin, D. Llanso, M. A.Gomez-Martin, Santiago Ontanon, and Ash-win Ram. Mmpm: a generic platform forcase-based planning research. In : ICCBR2010 Workshop on Case-Based Reasoning forComputer Games, pages 45-54, July 2010.

5. Stefan Wender and Ian Watson. Using Re-inforcement Learning for City Site Selectionin the Turn-Based Strategy Game Civiliza-tion IV. In : Computational Intelligence andGames (CIG-2008), pages 372-377, 2008.

6. Janet L. Kolodner. An introduction to case-based reasoning. In : Artificial IntelligenceReview, pages 3-34, 1992.

7. Santi Ontanon, Kinshuk Mishra, NehaSugandh, and Ashwin Ram. On-line case-based planning. In : Computational Intelli-gence, pages 84-119, 2010.

8. Santiago Ontanon, K.Bonnette,P.Mahindrakar, M. A. Gomez-Martin,Katie Long Genter, J.Radhakrishnan,R.Shah, and Ashwin Ram. Learning fromhuman demonstrations for real-time case-based planning. In : STRUCK-09 Workshop,colocated with IJCAI, pages 2-3, 2011.

9. Neha Sugandh, Santiago Ontanon, and Ash-win Ram. On-line case-based plan adaptationfor real-time strategy games. In :Associationfor the Advancement of Artificial Intelligence

(AAAI-2008), pages 1-2. AAAI Press, 2008.10. Santiago Ontanon Villar. D2 doc-

umentation. pages 1-6, May 2010,http://heanet.dl.sourceforge.net/project/dar-mok2/D2Documentation.pdf.

11. Marc Ponsen and Pieter Spronck ImprovingAdaptive Game AI with Evolutionary Learn-ing In Computer Games: Artificial Intelli-gence, Design and Education, pages 389-396,2004.

12. Bhaskara Marthi, Stuart Russell, DavidLatham and Carlos Guestrin Concurrent hi-erarchical reinforcement learning Turn-BasedStrategy Game Civilization IV. In Interna-tional Joint Conference on Artificial Intelli-gence, Edinburgh, Scotland, pages 1652-1653,2005.

13. Pranay M. Game AI : Simulator Vs Learnerin Darmok2. In University of Hyderabad asM.Tech. thesis , 2013.

Harshit Sethy is the Co-founder and Chief TechnologyOfficer of Gymtrekker FitnessPrivate Limited, Mumbai, In-dia. He received his Mastersdegree in Artificial Intel-ligence from University ofHyderabad.

Amit Patel is currentlythe Assistant Professor,Rajiv Gandhi University ofKnowledge Technologies, IIITNuzvid, Krishna. He obtainedhis Bachelor of Technologyfrom Uttar Pradesh TechnicalUniversity. He received his

Masters degree in Artificial Intelligence fromUniversity of Hyderabad, Hyderabad.