Download - Motivated Reinforcement Learning for Non-Player Characters in Persistent Computer Game Worlds
1Intelligent Database Systems Lab
國立雲林科技大學National Yunlin University of Science and Technology
Motivated Reinforcement Learning for Non-Player Characters in Persistent
Computer Game Worlds
Advisor : Dr. HsuPresenter : Chia-Hao YangAuthor : Kathryn Merrick, Mary Lou Maher
SIGCHI 06
2Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.
Motivation Objective Introduction Method Experiments Discussion Conclusions Habituation SOM Q-learning
Outline
3Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.
Motivation Many NPC possess a fixed set of pre-programmed
behaviors and lack the ability to adapt and evolve in time with their surroundings.
4Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.
Objective To create NPC that can both evolve and adapt with
their environmental.
5Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.Introduction Current technologies for NPCs
─ Reflexive agents Only recognized states will produce a response
State machines & rule-based approaches EX : Baldur Gate & Dungeon Siege
─ Learning agents It can modify their internal structure to respect to some task.
Black and White─ Reinforcement learning agents
The agent records the reward signal. Then chooses an action which attempts to maximize the long-r
un sum of the values of the reward signal. Tao Feng
6Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.Method
Motivated reinforcement learning agents─ It use a motivation function to directs learning.─ Skill development is dependent on the agent’s environment & these
skills are developed progressively over time.
S(t) – S(t-1)
S(t-1) – S(t-2)
Q-learning
7Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.Experiments In order to experiment with MRL agent, we implemented a village s
cenario in Second Life.─ Support character
Trades people Location, object, inventory sensor Move to object, pick up object, use object effector Ex : the pick, when used on the mine, will produce iron which can converted to weapon
s when used near the forge
8Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.Experiments
─ Partner character Vendor character
Location, object sensor Move to object effector Ex : In Ultima Online players can set up vendor characters to sell the goods they have c
rafted.
9Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.Conclusions This paper has presented MRL agents as a means of
creating non-player characters which can both evolve and adapt.
MRL agents explore their environment and learn new behaviors in response to interesting experiences, allowing them to display progressively evolving behavioral patterns.
10Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.Habituation SOM An HSOM consists of a standard Self-Organizing Map with an
additional habituating neuron connected to every clustering neuron of the SOM.
─
─
11Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.Q-Learning
It’s a part of reinforcement learning algorithm which has been widely used for many applications such as robotics, multi agent system, game, and etc.
It allows an agent to learn through training without teacher in unknown environment.─ Modeling the Environment
─ putting similar matrix name Q in the brain of our agent
reference
12Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.Q-Learning ─ algorithm
─ example
reference
……