how rovio teaches angry birds to fly in the cloud using ml

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

How Rovio teaches Angry Birds to fly in the cloud using ML

G A M 3 0 1

David Mason

SVP Technology

Rovio Entertainment Corporation

Asko Relas

Senior Data Engineer

Rovio Entertainment Corporation

Agenda

• Happy Birdday!

• Beacon: Our cloud services

• Our machine learning (ML) story

• Use case: Reinforcement learning (RL) for level difficulty prediction

• Takeaways

Beacon: Our cloud services

Beacon

AWS is our preferred cloud provider

# Player accounts > 1Bn

# Requests per second > 10k

# Analytics events per day > 2Bn

# Amount of data per day > 1TB

# EC2 instances > 1300

Analytics

A/B testing Personalization

Payments Ads + XP

Push

Surveys

Game server infrastructure

LiveOps calendar

Audience builderUA

BuildPrivacy

AI assisted dev

Singular

Player identity

Amazon Elasticsearch

Service (Amazon ES)

Analytics pipeline

Lake of wisdom

Real-time Flink

streams

Amazon EMR

ORC format

with schemas

Amazon S3Secor, Spark, Hive

KPIs, player profiles, ad-hoc queries

Advanced analytics, ML

Spark, R, Python

• Redash• Grafana• Presto

Dashboards, app personalization

Event

gateway Kafka

Mobile client

Services

Amazon

EC2

Services

Amazon

EC2

Services Amazon

EC2

Amazon

EC2

Amazon

EC2

Amazon

EC2

Amazon

EC2

Amazon

Athena

Amazon

Redshift

Amazon

Athena

Amazon

EC2

• Cassandra• Presto• Druid

Amazon

SageMaker

Amazon

EMR

Amazon EMR

Amazon EMR

Game server infrastructure: Reusable templatesPlatform services

Deployment

Network

Management

Amazon

RDS

Amazon

DynamoDB

Amazon

ElastiCache

Amazon Elasticsearch

Service (Amazon ES)

Amazon

CloudWatchAmazon S3

AWS Certificate

Manager (ACM)

AWS

CodeDeployAmazon ECS

Elastic Load

Balancing

Amazon Route

53

AWS Shield

Advanced

Jenkins TerraformAWS Management

Console

Network Load

Balancer

Application Load

Balancer

Service group 1 Service group 2Load test

generators

Logs Monitoring Discovery

AlertsConfig

VPN Gateway

Game server microservices: Chirp

Game serverGame

clients

Consul

Amazon ES

Amazon RDS

Index messages

Store messages

Profanity

check

Community

sift

Discover nodes

Chirp server

Notifications

Firebase cloud

messaging

Push

notifications

Google

Translate

Translate

message

Request

response

Our machine learning (ML) story

ML/AI in gaming

AI creates art,

music, &

narratives

Creative

personalization

Game ideaGame

development Distribution Marketing Live operation Monetization

Market

analysis

Modeling of fun

Bots for QA

& level

balancing

Automatic level

& game object

generation

LTV

modeling

Player

onboarding

Campaign

optimization

(paid/xpromo)

Personalized

events

Moderation

Player in-game

support &

automation

Segmentation

& targeting

Monitoring

& alerting,

anomaly

detection

Pricing

optimization

Ads optimization

Creative

analysis &

optimization

Our goal: Games built for you

MICRO

2011

MACRO

13 15 17 19 21

INDIVIDUAL

Angry Birds Match AI

heuristic bot to play

levels & estimate pass

rate

ML driven market

analysis tooling

Churn prediction used

for cross-promo

LTV prediction in profiles

User-specific app install

propensity prediction for

cross-promotion

Anomaly detection

Fruit Nibblers

using ML for churn

prediction to improve

level funnel

Angry Birds Dream

Blast ML level

balancing

First session spender

prediction

Rovio starts

collecting data

from games

Churn &

conversion

prediction

Project overview

Designing the level funnel of a puzzle game is a balancing act• Too difficult levels give a feeling of pay-to-win and players churn

• Too easy levels make the game boring

Balancing new levels requires repetitive testing from level designers

Being able to predict the difficulty of unreleased levels• Consistent experience for the most advanced players

• Less time spent on manual testing

Measuring difficulty

Overall concept

Level designer

Submits a pack of new levels for evaluation

Bot

Plays the levels and records statistics of them

Prediction

Real players’ pass rates are predicted from the

bot’s statistics

From end users’ point of view

Solution architecture

Amazon API Gateway

Bots API Bots

1. Store levels

2. Run bot

NotificationSubmit levels

HTTP/REST

PredictionsLevels

Regression models

Amazon S3 Amazon RDS

Different bot approaches

Concept Pros Cons

HeuristicScore every available action

based on predefined rules

The simplest to

implement

One policy for the whole

game

Tree searchBuild a game state tree to

find best actions, e.g. MCTS

No understanding of

game mechanics needed

Slow and computationally

heavy

Imitation learning

Collect data on how people

play the game, train a bot to

imitate

Possibly most human-like

Only one policy

More game

instrumentation needed

Reinforcement

learning

The bot learns how to play

the game through

experimentation

Can learn a different

policy for each level

Slower than imitation, but

faster than tree search

Reinforcement learning (RL)

Agent

Environment

ActionReward

Reinforcement learning (RL)

Our adaptation of deep q-learning:

• Feature engineered game state instead of raw graphics

• Feed-forward neural net instead of a deep convolutional net

• Dueling double q-learning with prioritized experience replay

PyTorch as deep learning framework

Data collection

Level 1

Data collection

Level 1

Pass rate: 100%

Goals cleared: 100%

Moves left: 8.9

Extra moves: 0

Data collection

Level 1 Level 2

Pass rate: 100%

Goals cleared: 100%

Moves left: 8.9

Extra moves: 0

Pass rate: 99%

Goals cleared: 99%

Moves left: 7.4

Extra moves: 0

Data collection

Level 1 Level 2 Level 3

Pass rate: 100%

Goals cleared: 100%

Moves left: 8.9

Extra moves: 0

Pass rate: 99%

Goals cleared: 99%

Moves left: 7.4

Extra moves: 0

Pass rate: 97%

Goals cleared: 98%

Moves left: 4.9

Extra moves: 0

Data collection

Level 1 Level 2 Level 3 Level 982

Pass rate: 100%

Goals cleared: 100%

Moves left: 8.9

Extra moves: 0

Pass rate: 99%

Goals cleared: 99%

Moves left: 7.4

Extra moves: 0

Pass rate: 97%

Goals cleared: 98%

Moves left: 4.9

Extra moves: 0

Pass rate: 2%

Goals cleared: 54%

Moves left: 0.02

Extra moves: 7.3

Data collection

Collecting the data set takes time

• For the most accurate statistics, the bot first learns a level, then plays it 1,000 times

• One game round takes 5-10 seconds even with sped-up physics

• Collecting the data set by playing the levels sequentially would take over a year

Solution

• Bundle the game binary and bot Python source code to a Docker container

• Run one container per level using AWS Batch

→ Data collection takes two days for any number of levels

• Learn from multiple parallel instances of the game within one container

→ Data collection takes four hours for any number of levels

Parallelized RL

Agent

Game

Parallelized RL

Agent

Game

Agent

Game

Parallelized RL

Agent

Game

Buffer

Agent

Game

Experience

Parallelized RL

Agent

Game

Learner

Buffer

Agent

Game

ExperienceLearned

NN model

Parallelized RL

Amazon EC2 C5 instance

Agent

Game

Learner

Buffer

Agent

Game

ExperienceLearned

NN model

Parallelized RL

Eight-fold parallelism with a single c5.2xlarge

If we increase the number of agents, the learner becomes a bottleneck

Can scale further through:

● Using a GPU, eg. p3.8xlarge with 32 vCPU

● Scaling out across multiple machines

Amazon EC2 C5 instance

Agent

Game

Learner

Buffer

Agent

Game

ExperienceLearned

NN model

Predicting difficulty

Bot stats

Player pass rates

Learn a regression model from bot stats and player pass rates

Predicting difficulty

New level bot stats

Get prediction from learned regression

model

Player pass rate prediction

Regression modeling

Statistics collected from the bot

• Pass rate

• Moves left

• Level objectives cleared

• Extra moves given

scikit-learn as ML toolkit

Best prediction accuracy with XGBoost

• Tree-based algorithms can’t extrapolate

→ Additional classification between impossible/passable levels is necessary

Prediction accuracy

MAE 0.15 in validation data set

Then what?

Difficulty

Churn Revenue

Then what?

f(x) y(x)

Difficulty

Churn Revenue

Difficulty versus churn

Pass rate

Churn


Level 300

Churn

Pass rate


Level 800

Level 300

Churn

Pass rate


Churn

Pass rate

Difficulty versus revenue

Level 300

Level 1,000

Revenue

Pass rate

Case summary

• AWS Batch is great for parallel workloads, such as having bots play 1,000 levels concurrently

• Parallelization is key to implementing fast-enough RL for business use

• Next step: Procedural content creation?

Building an unfair advantage

• Put the player at the center

• Analytics is foundational

• Have vision, experiment, learn, productize

• ML will change the game

Thank you!


Contact information

David [email protected]

Asko [email protected]

how rovio teaches angry birds to fly in the cloud using ml

Documents