aws webcast - database in the cloud series - scalable games and analytics with aws

AWS Gaming Solutions

Scalable Games and Analytics with AWS

Nate Wiger @nateware Principal Gaming Solutions Architect April 2014

Artist: Abel Oroz Vicente


What's In It For Me?

•  Why AWS for Games? •  Core Game Backend •  Scaling with DynamoDB •  Low-Latency Multiplayer with C3 •  Game Analytics with Redshift and Kinesis


Gratuitous Logo Slide


Handle Viral Spikes


10 Regions

51 CloudFront POPs

Continuous Expansion

Global is Good


Big picture slide


Elastic Beanstalk •  Managed Container •  6 Languages •  Git Push or Zip Upload •  ELB, EC2, DB, DNS •  Web Dashboard •  Same Performance •  Full Access •  So Yeah, Use It


Hill Of Beans

ELB

S3

•  Beanstalk Manages •  ELB •  EC2 •  Auto Scaling •  Monitoring •  RDS

•  Add Other Services •  S3 •  CloudFront •  SNS Mobile Push

EC2

Elastic Beanstalk Container

EC2

CloudFront CDN


Push It

cd MyGame eb init vi app.rb require 'sinatra' get '/hello.json' do {message: "Hello World!"} end git commit -‐m "app updates" -‐a git aws.push

•  Initialize Beanstalk •  Write Code •  Git Commit •  Git Push to Beanstalk •  Coffee / Beer •  Rinse, Repeat


http://bit.ly/ubisoft-aws


Region

Writing is Painful

Availability Zone A

Availability Zone B

S3

EC2

•  Games are Write Heavy •  Constant State Updates •  Key Value Key Value •  Binary Data •  Caching of Limited Use •  Database = Bottleneck

ELB

EC2

CloudFront CDN


DynamoDB

Availability Zone A

Availability Zone B

S3

•  NoSQL datastore •  Fully-managed •  Provisioned throughput •  PUT/GET keys •  Secondary indexes •  Handles any SANE

Gaming use case EC2 EC2

ELB

CloudFront CDN

Elastic Beanstalk Container


DynamoDB: Low Latency at Scale

Docs, SDK's: http://aws.amazon.com/dynamodb/developer-resources/ Auto Scaling: https://github.com/sebdah/dynamic-dynamodb


DynamoDB: Durable at Scale

WRITES Replicated to 2 AZ’s before OK Persisted to disk (SSD)

READS Strong or eventual consistency

No trade-off in latency


Leaderboard in DynamoDB

•  Hash key = Primary key •  Range key = Sub key •  Others attributes are

unstructured •  How to sort based on

High Score per Game? •  Secondary Index on

Game + Score


Python DynamoDB Leaderboard

table = Table('scores', schema=[ HashKey('user'), RangeKey('level') ], throughput={ 'read': 5, 'write': 15 }, global_indexes=[ GlobalAllIndex('highscore', parts=[ HashKey('level'), RangeKey('score', data_type=NUMBER) ], throughput={ 'read': 5, 'write': 15 } ) ])

new_score = Item(table, data={ 'user': user, 'level': level, 'score': score }) new_score.save() topscores = table.query(index='highscore') for ts in topscores: print(ts['user'], ts['score'])


EC2: New C3 Instance

•  High packets per second •  Low latency, no jitter •  Intel Ivy Bridge CPU •  SSD's •  Built for games •  10.5 cents / hour


Enhanced Networking (SR-IOV)

Before: Hypervisor

After: Hardware


Step 1: HVM Machine Image


Step 2: Choose C3 Instance


Step 3: Launch in VPC


Recap: Enabling SR-IOV

①  Use HVM image (Linux or Windows) ②  Select C3 instance ③  Must launch in VPC ④  Verify with CLI

aws ec2 describe-‐instance-‐attribute \ -‐-‐attribute sriovNetSupport -‐-‐instance-‐id i-‐12345678

⑤  Full docs: http://bit.ly/ec2-enhanced-net


EC2: New G2 Instance

•  NVIDIA Kepler GPU •  Game Streaming •  Rendering Assets •  Build Servers •  AppStream! EVE Online:

http://bit.ly/appstream-eve-online


http://bit.ly/red5-aws


http://bit.ly/rovio-aws


Analyze What?

Emotions •  Enjoying game •  Engaged •  Like/dislike new content •  Stuck on a level •  Bored •  Abandonment

Behaviors •  Hours played day/week •  Number of sessions/day •  Level progression •  Friend invites/referrals •  Response to mobile push •  Money spent/week


Analytics Flow

Ingest Store Process Analyze Profit!


Off The Shelf Analytics

•  Easy To Integrate •  Pre-Baked Reports •  Rate Limits •  Retention Windows •  Data Lock-In


•  Write Events File on Device •  Periodically Upload to S3 •  Process into Redshift •  Point GUI Tool to Redshift

Start Simple

2014-‐01-‐24,nateware,e4df,login 2014-‐01-‐24,nateware,e4df,gamestart 2014-‐01-‐24,nateware,e4df,gameend 2014-‐01-‐25,nateware,a88c,login 2014-‐01-‐25,nateware,a88c,friendlist 2014-‐01-‐25,nateware,a88c,gamestart

Profit!


Tableau + Redshift

http://bit.ly/tableau-redshift


•  Also Collect Server Logs •  Periodically Upload to S3 •  Stuff into Redshift •  External Analytics Data Too

More Data Sources

EC2

External Analytics


•  Integrate Game DB •  Load Directly into Redshift •  Redshift does Intelligent Merge •  Tracks DynamoDB Columns

Direct From DynamoDB

EC2


Real-Time Event Stream

2014-‐03-‐17T09:52:08-‐07:00,nateware,e4b5,login 2014-‐03-‐17T09:52:54-‐07:00,nateware,e4b5,gamestart 2014-‐03-‐17T09:53:15-‐07:00,nateware,e4b5,levelup 2014-‐03-‐17T09:54:06-‐07:00,nateware,e4b5,gameend 2014-‐03-‐17T09:54:23-‐07:00,nateware,30a4,gamestart 2014-‐03-‐17T09:55:14-‐07:00,nateware,30a4,gameend 2014-‐03-‐17T09:55:41-‐07:00,nateware,30a4,gamestart 2014-‐03-‐17T09:57:12-‐07:00,nateware,6ebd,levelup 2014-‐03-‐17T09:58:50-‐07:00,nateware,6ebd,levelup 2014-‐03-‐17T09:59:52-‐07:00,nateware,6ebd,gameend


•  Stream Data With Kinesis •  Multiple Writers and Readers •  Still Output to Redshift

Speeding Towards Real-Time

EC2


•  Stream Data With Kinesis •  Multiple Writers and Readers •  Still Output to Redshift •  Stream to Spark on EMR •  Storm via Kinesis Spout •  Custom EC2 Workers

Speeding Towards Real-Time

EC2

EC2


Wrap It Up Already

•  Start Simple With Beanstalk •  Go Directly to DynamoDB, Do Not Pass Go •  Leverage the EC2 C3 Instance •  Analytics 1-2-3: CSV => S3 => Redshift •  Kinesis + Spark/EMR for Real-Time Analytics


http://bit.ly/minecraft-aws

aws webcast - database in the cloud series - scalable games and analytics with aws

Technology

kinesis multiple

availability

low latency

aws

instance

gamestart

score

hashkey