evolving multimodal networks for multitask games

Evolving Multimodal Networks for Multitask GamesJacob Schrum – schrum2@cs.utexas.eduRisto Miikkulainen – risto@cs.utexas.eduUniversity of Texas at AustinDepartment of Computer Science

Evolution in videogames Automatically learn interesting behavior Complex but controlled environments

Stepping stone to real world Robots Training simulators

Complexity issues Multiple contradictory objectives Multiple challenging tasks

Multitask Games

NPCs perform two or more separate tasks Each task has own performance measures Task linkage

IndependentDependent

Not blended Inherently multiobjective

Test Domains Designed to study multimodal behavior Two tasks in similar environments Different behavior needed to succeed Main challenge: perform well in both

Front Ramming Back Ramming

Front/Back Ramming

Front Ramming Attack w/front ram Avoid counterattacks

Back Ramming Attack w/back ram Avoid counterattacks

Same goal, opposite embodiments

Predator/Prey

Predator Attack prey Prevent escape

Prey Avoid attack Stay alive

Same embodiment, opposite goals

Multiobjective Optimization Game with two objectives:

Damage Dealt Remaining Health

A dominates B iff A is strictly better in one objective and at least as good in others

Population of points not dominated are best: Pareto Front

Weighted-sum provably incapable of capturing non-convex front

Dealt lot of damage,but lost lots of health

Tradeoff between objectives

High health but did not deal much damage

NSGA-II Evolution: natural approach for finding optimal population Non-Dominated Sorting Genetic Algorithm II*

Population P with size N; Evaluate P Use mutation to get P´ size N; Evaluate P´ Calculate non-dominated fronts of {P P´} size 2N New population size N from highest fronts of {P P´}

*K. Deb et al. A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II. Evol. Comp. 2002

Constructive Neuroevolution Genetic Algorithms + Neural Networks Build structure incrementally (complexification) Good at generating control policies Three basic mutations (no crossover used)

Perturb WeightAdd Connection Add Node

Multimodal Networks (1) Multitask Learning*

One mode per task Shared hidden layer Knows current task

Previous work Supervised learning context Multiple tasks learned

quicker than individual Not tried with evolution yet

* R. A. Caruana, "Multitask learning: A knowledge-based source of inductive bias" ICML 1993

Multimodal Networks (2) Mode Mutation

Extra modes evolved Networks choose mode Chosen via preference neurons

MM Previous Links from previous mode Weights = 1.0

MM Random Links from random

sources Random weights Supports mode deletion

Starting network with one mode

MM(R)MM(P)

Experiment Compare 4 conditions:

Control: Unimodal networks Multitask: One mode per task MM(P): Mode Mutation Previous MM(R): Mode Mutation Random + Delete Mutation

500 generations Population size 52 “Player” behavior scripted Network controls homogeneous team of 4

MO Performance Assessment

Reduce Pareto front to single numberHypervolume of

dominated region Pareto compliant

Front A dominates front B implies HV(A) > HV(B)

Standard statistical comparisons of average HV

20 runs

Front/Back Ramming Behaviors

Multitask

MM(R)

Front Ramming Back Ramming

20 runs

Predator/Prey Behaviors

Multitask

MM(R)

Prey Predator

Discussion (1) Front/Back Ramming

Control < MM(P), MM(R) < MultitaskMultiple modes helpExplicit knowledge of task helps

Discussion (2) Predator/Prey

MM(P), Control, Multitask < MM(R)Multiple modes not necessarily helpfulDisparity in relative difficulty of tasks

Multitask ends up wasting effortMode deletion aids search for one good mode

How To Apply Multitask good if:

Task division known, andTasks are comparably difficult

Mode mutation good if:Task division is unknown, or“Obvious” task division is misleading

Future Work Games with more tasks

Does method scale? Control mode bloat

Games with independent tasks Ms. Pac-Man

Collect pills while avoiding ghosts Eat ghosts after eating power pill

Games with blended tasks Unreal Tournament 2004

Fight while avoiding damage Fight or run away? Collect items or seek opponents?

Conclusion Domains with multiple tasks are common

Both in real world and games Multimodal networks improve learning in

multitask games Will allow interesting/complex behavior to

be developed in future

Questions?Jacob Schrum – schrum2@cs.utexas.eduRisto Miikkulainen – risto@cs.utexas.edu

University of Texas at AustinDepartment of Computer Science

Auxiliary Slides