evolving neural network agents in the nero video
TRANSCRIPT
![Page 1: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/1.jpg)
Evolving Neural Network Agents in
the NERO Video Game
Stelios Petrakis
![Page 2: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/2.jpg)
The Paper
“Evolving Neural Network Agents in the NERO Video Game”
Kenneth O. Stanley / Bobby D. Bryant / Risto Miikkulainen
Winner of ‘Computational Intelligence and Games’ 2005 Best Paper Award
This paper introduces the real-time NeuroEvolution of Augmenting Topologies (rt-NEAT) method for evolving increasingly complex artificialneural networks in real time, as a game is being played.
![Page 3: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/3.jpg)
Problem
In most video games, scripts cannot learn or adapt to control the agents: Opponents will always make the same moves and the game quickly becomes boring.
The rtNEAT method allows agents to change and improve during the game.
In order to demonstrate this concept in the NeuroEvolvingRoboticOperatives (NERO) game, the player trains a team of robots for combat.
![Page 4: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/4.jpg)
Introduction for rtNEAT
rtNEAT is a real-time enhancement of the NeuroEvolution of Augmenting Topologies method (NEAT; Stanley and Miikkulainen2002b, 2004).
NEAT evolves increasingly complex neural networks.
Real-time NEAT (rtNEAT) is able to complexify neural networks as the game is played, making it possible for agents to evolve increasingly sophisticated behaviors in real time.
![Page 5: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/5.jpg)
Properties that games A.I. should have
� Large state / action space
� Diverse behaviors
� Consistent individual behaviors
� Fast Adaptation
� Memory of past states
![Page 6: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/6.jpg)
Learning techniques for NPCs in games
� Supervised techniques
� Backpropagation
� Decision Tree Learning
� Traditional Reinforcement Learning (RL)
� Q-Learning
Not good. Agents need to learn online as game is played
Not suitable due to the constraints explained in
� Sarsa(λ)
� Neuroevolution (NE)
� Artificial Evolution neural networks using a genetic algorithm
constraints explained in previous slide
Suitable! rtNEATtechnique can be used
![Page 7: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/7.jpg)
NEAT
NEAT combines the usual search for the appropriate network weights with complexification of the network structure, allowing the behavior of evolved neural networks to become increasingly sophisticated over generations.
It outperforms other neuroevolution (NE) methods
NEAT was originally designed to run offline and is based on 3 key ideas
![Page 8: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/8.jpg)
NEAT key ideas
• Evolving network structure requires a flexible genetic encoding
� Connection Genes
� In-node
� Out-node
� Connection Weight
� Connection gene expression
� Innovation number
• Individuals compete primarily within their own niches instead of with the population at large
� Protection of Topological innovations (explicit fitness sharing reproduction mechanism)
• NEAT begins with a uniform population of simple networks with no hidden nodes
![Page 9: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/9.jpg)
rtNEAT
In NEAT the entire population is replaced at each generation.In rtNEAT only a single individual is replaced every few game ticks.
One of the worst individuals is removed and replaced with a child of parents chosen from among the best.of parents chosen from among the best.
Challenge : Protection of innovation through speciation and complexification.
Let fi be the fitness of individual i. Fitness sharing adjusts it to fi/|S|, where |S| is the number of individuals in the species. In other words, fitness is reduced proportionally to the size of the species.
![Page 10: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/10.jpg)
rtNEAT Main Loop
� Remove the agent with the worst adjusted fitness from the population assuming one has been alive sufficiently long so that it has been properly evaluated.
� Re-estimate for all speciesF
� Choose a parent species to create the new offspring
� Adjust compatibility threshold Ct dynamically and reassign all agents to species
� Place the new agent in the world
![Page 11: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/11.jpg)
Removing the worst agent
The agent with the worst adjusted fitness should be removed, since adjusted fitness takes into account species size, so that new smaller species are not removed as soon as they appear.removed as soon as they appear.
rtNEAT only removes agents who have played for more than the minimum amount of time m (not young ones).
![Page 12: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/12.jpg)
Re-estimating F
By removing the worst agent from a specie, species size has been decreased, therefore has changed. F
We need to recalculate for the next step; choosing parent species.
F
![Page 13: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/13.jpg)
Choosing the parent species
PF
Fn
tot
kk =
NEAT
Number of offspring The sum of all the average species’
Population size
The average fitnessof species k
Number of offspring assigned to species k
average species’ fitnesses
tot
kk
F
FS =)Pr(
rtNEAT The probability of choosing a given parent species is proportional to its average fitness.
The expected number of offspring for each species is proportional to nk, preserving the speciation dynamics of original NEAT
![Page 14: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/14.jpg)
Dynamic Compatibility Thresholding
Ct determines whether an individual is compatible with a species’ representative.
When there are too many species, Ct is increased to make species more inclusive.
When there are too few, Ct is decreased to be stricter. When there are too few, Ct is decreased to be stricter.
An advantage of this kind of dynamic compatibility thresholding is that it keeps the number of species relatively stable.
After changing Ct in rtNEAT, the entire population must be reassigned to the existing species based on the new Ct.
In original NEAT, if a network does not belong in any species a new species is created with that network as its representative
![Page 15: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/15.jpg)
Replacing the old agent with the new one
� Two options:
� Neural Network can be removed from the body and replaced without any changed in the body of and replaced without any changed in the body of the agent
� Agent dead, so body + neural network are replaced.
![Page 16: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/16.jpg)
![Page 17: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/17.jpg)
Determining Ticks Between Replacements
If agents are replaced too frequently, they do not live long enough to reach the minimum time m to be evaluated.On the other hand, if agents are replaced too infrequently, evolution slows down to a pace that the player no longer enjoys.
A law of eligibility can be formulated that specifies what fraction of the population can be expected to be ineligible once evolution reaches a steady state
nP
mI =
The ticks between replacements
The minimum time alive
The fraction of the population that is too young and therefore cannot be replaced
![Page 18: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/18.jpg)
Let rtNEAT select n
IP
mn
nP
mI =⇒=
It is best to let the user choose I because in general it is most critical to performance; if too much of the population is ineligible at one time, the mating pool is not sufficiently large.
![Page 19: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/19.jpg)
NERO
![Page 20: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/20.jpg)
www.nerogame.org
In NERO, the learning agents are simulated robots, and the goal is to train a team of robots for military combat.
The robots begin the game with no skills and only the ability to learn.
![Page 21: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/21.jpg)
Sensors
Robots have several types of sensors.
![Page 22: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/22.jpg)
DEMO / Trailer
![Page 23: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/23.jpg)
References
� http://www.nerogame.org/
� Evolving Neural Network Agents in the NERO Video � Evolving Neural Network Agents in the NERO Video Game (Stanley, Bryant, Miikkulainen 2005)
![Page 24: Evolving Neural Network Agents In The Nero Video](https://reader033.vdocuments.net/reader033/viewer/2022052823/5551e6f0b4c905f56f8b4b35/html5/thumbnails/24.jpg)
Thanks!