understanding and improving games through machine learning - natasha latysheva
TRANSCRIPT
Understanding and
improving games through
machine learning
Natasha Latysheva
Data Scientist
Jagex Games Studio
This talk:1. Introduction to Jagex2. Recap of main machine learning
fields + examples3. Fun ML in video games in general4. ML projects at Jagex Data Science
• Identifying player personas• Deep learning player lifecycle
sequences• Automatic bot and abuse
detection• Trading card deck analysis• Quest recommender systems
Jagex Game Studios
• Video game development company based in Cambridge
• Best known for Runescape, the world's largest free-to-play massively multiplayer online role-playing game (MMORPG). First released in 2001!
• Extremely community oriented game
• Old School Runescape was released in 2013
Machine learning recap
Essentially learning patterns from data.
Branches of ML:
• 1. Supervised learning.
• Most familiar names, e.g. linear regression, logistic regression, decision trees, random forests, SVMs, many neural nets.
• Algorithm is presented with input data and explicit labels on that input. Learn signals associating input data with labels.
• Everything from good ole iris classification to detecting diseased patients from complex medical images.
Machine learning recap
2. Unsupervised learning.
• No explicitly labelled data, instead often seeking to find natural sub-groups of similar data points within the data.
• Describing the structure of the dataset.• Objective evaluation of accuracy often
difficult/not possible• Common methods:
• Clustering methods like k-means and hierarchical clustering
• Dimensionality reduction methods like principal components analysis
• Outlier/anomaly detection• Some NNs, e.g. generative adversarial
networks (GANs)
Larsen et al. 2016
Machine learning recap
3. Reinforcement learning.
• Agent must maximise cumulative reward by learning to optimally interact with an environment by choosing ‘good’ actions• E.g. robot must come up with correct limb
positions and joint torque to move efficiently/quickly
• Manufacturing program must maximise total units of product produced each day by controlling/tuning sub-processes
• Video game player agent must process the information from pixels on the screen to operate virtual keyboard + mouse to choose when to turn, apply brakes, gas, etc.
DuskDrive (OpenAI Universe, 2016)
Houthooft et al. 2017
1. DeepMind
• Classic Atari games• Learning from raw pixel feeds to
maximise future cumulative reward (i.e. score), no previous information encoded
• Ability to generalise skill playing across games
• In some cases, outperforming human players
• Deep Q-network approach• Combining deep learning of the
value function Q(s, a) with RL + experience replay
• Deep NN is Q(s, a) approximator, trained with Bellman updates Compulsory DeepMind demo:
https://www.youtube.com/watch?v=V1eYniJ0Rnk
Mnih et al., 2015
• 3. Hearthstone (collectible card game)
Bursztein, 2014
5. Predicting outcome: Domain experts (like casters) generate metrics (per turn + cumulative) that seem important to winning a game:• Mana advantage (delta mana spent in turn)• Board advantage (delta # minions)• Hand size advantage• Board mana advantage (delta total mana spent), etc.• No info on hero class or health
.
Identifying player personas
• Understand different archetypes of players
in Runescape
• Ultimately we would like to have a better
sense of our customer base
• Helps games devs
• Helps us understand the motivations, churn
behaviours, and value of our customers
.
Identifying player personas
• Draw inspiration from Bartel types
• Generated ~280 features to quantify our player behaviour
• Loosely based on Bartle Types (Killers, Achievers, Socializers, Explorers) • Quickly abandoned
• Example behavioural features• Chat Rank Score• Skill Heterogeneity• Forum Interactions • Economic Activity• Social Clustering Coefficients• Abilities Used During Play
.
Identifying player personas
• Data cleaning:
• Normalising by total and that year play time,
logging certain variables
• Dimensionality reduction using principal
components analysis
• Identifying optimal number of clusters using
silhouette plots or tSNE
Then… doing the clustering in PC space!
Life stages
• Differences in playtime both overall and recently are such a strong differentiating factor between
players that you first need to cluster on life cycle stages
Life stages
Name % Median Lifetime Playtime
Median YearlyPlaytime
Little ActivityShort Lifetime Playtime
30.7% 3h 3h
Little ActivityLong Lifetime Playtime
22.1% 590h 4h
ActiveMid Lifetime Playtime
22.2% 75h 35h
Extremely ActiveExtreme Lifetime Playtime
25% 2130h 320h
• The extremely dedicated players make up a huge proportion of total revenue.
Playstyles interacting with lifecycle stages
• Can observe which play styles emerge
or dominate at which lifecycle stages
• Can characterise transitions between
the different personas in consecutive
years
• Importantly: Quantify which types of
players tend to abandon the game
Player lifecycle sequencesMatt Project
• Players can undergo a variety of business relevant lifecycle
transitions.
Player lifecycle sequencesMatt Project
• Number of possible combinations, complexity of transitions,
scale of data, etc. leads to a difficult modelling problem
• Goal: We want to learn more about the player base and
how their lifecycle stages change.
• Two approaches:
• 1. Building transition probability matrices to examine
trends and project player states forward into the future
Deep learning player lifecycle sequences Matt Project
• 2. Building a recurrent neural network (specifically, an LSTM RNN) to learn the types of transitions that
happen and project these forward into the future
• Inherent memory to the system.
• But - difficult and time-consuming to train! GPUs help.
• On the plus side, can generate sequences seeded with multiple states. Also, in theory can learn
quite complex patterns.
• Models operational but need to tune hyperparameters and compare performance to transition
matrices.
[Colah’s blog]
Sentiment, bot and abuse detection workMatt Project
• At the moment we:
• Help community team understand the popularity of
updates using sentiment detection
• Very simple model – Naïve Bayes classifier run over
in-game chat to identify emotional valence
• Detect instances of harassment
• Search through text with preset list of regular
expressions
• Try to learn more complex language models
• Using word2vec to identify words with similar
semantic meanings, identify harassment using
learned vector embeddings
Matt Project
[Saed Sayad]
In progress…
Training deep convolutional neural nets to identify bot-like mouse movements
[Adit Deshpande]
Common CNN architecture
[Also working to automate bot removal in other ways]
.
4. Trading card deck analysis
• Play a sequence of cards in a sequence to
create a quest for your legend
• Many cards have additional effects that buff
your character, provide weapons, alter
creature power, or grief / damage your
opponent
• Games span 5 chapters where each player
places up to 4 cards per chapter
• The gameplay is combo-oriented where card
play sequence is very important
Miro Project
.
Build card vectors with word2vec
• Learn card vectors using Word2Vec algorithm: • Given a card vector, predict the
remaining cards that were played by the player in that chapter.
This worked surprisingly well!
• Cards with similar effects clustered together
• Legend specific cards clustered together• Common cards that combo well with
Legend specific cards are located in a region of space close to the Legend
Miro Project
.
Build deck vectors with word2vec
• Tweak Word2Vec algorithm (more akin to Paragraph2Vec) to learn vector representation for custom decks
• Deck vectors naturally learned archetype clusters without being explicitly taught
• Decks from different legends that play by similar strategies tend to cluster in the same region of space
• Deck vectors can be used down-stream in other ML tasks • Matchmaking• assessing deck quality• recommending cards• predicting churn
Miro Project
• Card and deck vectors userful for game devs for exploring the space of decks real players use• Interactive web application for exploring decks by hovering over
decks, filtering by cards, player rank, and win rate
• Built clustering tool on top of deck vectors to allow devs to quickly identify common cards within a region of ‘deck space’• View summary statistics on the clusters (win rates, play rates,
cluster sizes)
• This is used to identify balance issues in game, identify how players are playing decks, and categorize player play-styles
Miro ProjectChronicle project conclusions
• Ultimate goal: help guide players towards appropriate and interest-relevant content
• Starting point: recommend appropriate quests to players.
• Modelling using 2 approaches:
• 1. Restricted Boltzmann Machine (RBM) trained on past quest data
• 2. Matrix factorisation on past quest data with a REST API built to interface with game
Miro ProjectQuest recommender system
Matt and Miro Project
[deeplearning4j]
Acknowledgements
• Data Science Team!
• Miroslaw Horbal
• Matt Dixon
• Babis Georgiadis
• Katie Scott
• Joel Graham
• Ilian Mitev
• Analytics Team