Artificial Life/AgentsCreatures: Artificial Life Autonomous Software Agents for Home EntertainmentStephen Grand, 1997

Learning Human-like Opponent Behaviour for Interactive Computer GamesChristian Bauckhage, Christian Thurau, and Gerhard Sagerer, 2003

Evolving Neural Network Agents in the NERO Video GameKenneth O. Stanley, Bobby D. Bryant, Risto Miikkulainen, 2005

Creatures

• 1997 state of the art Artificial Life game• Creatures with neural network brains and an

evolving genome• Learn by punishment/reward reinforcement• Can learn rudimentary ‘verb-object’ language• Sense of sight, sound, touch• Complex biochemistry (metabolism, immune

system, genetically encoded morphology)

Creatures’ Brains

• Hebbian learning• ~1000 neurons, ~5000 synapses• Organised into ‘lobes’:

Characteristics

• Designed for efficiency (runs on 1997 commodity hardware)

• Limited number of neurons• Brain model is also limited, restricts potential

functions

Learning Human-like Opponent Behaviour

Learning Human-like Opponent Behaviour

• Neural-network control system for a Quake II bot

• Offline, supervised learning• Feed-forward, back-propagation learning,

multilayer perceptron network• One network for moving, one for aiming• Trained to learn one path, then multiple

paths, then moving and aiming

Advantages

• Potentially cheaper and faster than scripting bots

• Generalises to novel situations• More efficient than on-line learning bots• Good introduction to learning agents

Problems

• Paper is horribly structured and hard to read• Assumption: only the agent’s current

state/environmental influences matter!• Experiments didn’t work very well• Bots still static, can’t learn opponent tactics• Maybe difficult to get training data

Evolving Neural Network Agents in NERO

Evolving Neural Network Agents in NERO

• Online, reinforcement learning• Agent fitness increased by learning and

evolution• Player can train teams of bots to compete

against each other in increasingly complex training scenarios

• Won Best Paper Award at the IEEE Symposium on Computer Intelligence and Games

The Network

Learning Method

• rtNeat: basically, a technique for evolving increasingly complex neural networks

• Benefits over traditional RL:– Diversity increased/maintained through speciation– Can keep a memory of past events

• Player provides customised fitness function• NERO removes the worst agents, breeds the

best ones

• Currently NERO is quite simple• Paper presents no quantitative results, but

results seem promising