deploying a low-latency multiplayer game globally: loadout
TRANSCRIPT
1
GDC 2015 – Amazon Developer Day
Deploying a Low-Latency Multiplayer Game Globally
Nate WigerPrincipal, Gaming Solutions
@nateware
James GwertzmanCEO, PlayFab
@playfabnetwork
http://playfab.com
AWS Gaming Solutions
[Game] Online Problems Continue; Features
Disabled as [Game Studio] Works on Fixes
“We are working hard to get everyone online
as soon as possible,” [Game Studio] says.
Days later, [Game Studio] leaving
disconnected players stranded
[Game] has a great online mode – if you
can get online, that is
“This is total bull**** I paid for a preorder and
I can’t even connect WTF [Game Studio]??”
AWS Gaming Solutions http://bit.ly/verizon-latencyhttp://bit.ly/superdata-latency
AWS Gaming Solutions
How To Be Awesome
1. Game servers near players
2. Everything else as HTTP APIs
3. Data replication = bad
4. Local caches = good
5. Nobody really “plays everywhere”
AWS Gaming Solutions
VPC Subnet
VPC Subnet
Game API Pods
Availability Zone A Availability Zone B
VPC Subnet
VPC Subnet
Auto Scaling group
WEB
VPC Subnet
WEB
JOBS
Cognito
SNS Mobile
Push
SES
AWS Gaming Solutions
Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
EC2
Game Flow
EC2
EC2
AWS Gaming Solutions
Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over
Game Flow
EC2
EC2
AWS Gaming Solutions
Region
① Login via HTTP API
② Download Game Assets
③ Matchmaking to Game Server
④ Connect to Server
⑤ Hack Apart Your Friends
⑥ Game Over
⑦ Write via HTTP API
Game Flow
EC2
EC2
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Game Server Pods
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
AWS Gaming Solutions
Make. Network. Faster.
• C3/C4 instance family (or R3 for more memory)
• In a VPC
• HVM AMI – not PV
• Enable Enhanced Networking (SR-
IOV)http://bit.ly/ec2-enhanced-net
• Linux: Tweak Networking Stack http://bit.ly/linux-
tuning
AWS Gaming Solutions
VPC Private Subnet VPC Private Subnet
RabbitMQ + Elastic Load Balancing
Availability Zone A Availability Zone B
10.1.0.13 10.2.0.16
TCP 5672
rabbitmq-node1 rabbitmq-node2
TCP 4369
& 25672
AWS Gaming Solutions
RabbitMQ + Elastic Load Balancing
• Clustering Tutorial https://www.rabbitmq.com/clustering.html
• Set Queue HA policy to "all" https://www.rabbitmq.com/ha.html
• Create Internal load balancer
– Listen Port: TCP 5672
• VPC Security Group
– For load balancer: TCP 5672
– Between EC2 nodes: TCP 4369 & 25672
• Set up Client to Heartbeat
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Redis Pub / Sub
Availability Zone A Availability Zone B
VPC Public Subnet
VPC Private Subnet
GAME GAME GAME GAME GAME GAME
Auto Scaling group
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
CloudFormation + Chef
Availability Zone A
GAME GAME GAME
Auto Scaling group
AWS Gaming Solutions
Unreal Engine with CloudFormation
UnrealServer
unreal_linux.template.json
true
my-bucket-name
Unreal-123.zip
sshkeyname
1.2.3.4/24
MyPass123
AWS Gaming Solutions
VPC Subnet
Server Registration
Availability Zone A Availability Zone B
VPC Subnet
Auto Scaling group
WEB WEB
Oregon
Tokyo
VPC SubnetJOBS
AWS Gaming Solutions
Server Registration & Scaling
• HTTPS POST /api/servers/register
• Include an HMAC (RFC 2104)
• Send Server Status– Public IP
– # Players
– Game Modes
• Matchmaking Service– Maintains server list
– Removes servers
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Game Server Logs
Availability Zone A
S3 Bucket
Amazon CloudWatch Logs
GAME GAME GAME
AWS Gaming Solutions
VPC Private Subnet
VPC Public Subnet
Game Server Logs
Availability Zone A
Amazon
Kinesis
S3 Bucket
Amazon Redshift
GAME GAME GAME
AWS Gaming Solutions
What is PlayFab?
PlayFab is a complete live game operations platform.
Focus on the fun – we’ll do the boring bits.
+
Tools for operating &
monetizing live games
Complete set of cloud-based
backend services
Eco-system of pre-integrated
third-party services
.. and
more
+
AWS Gaming Solutions
Some Loadout Stats
• Launched 1/31/14
• Over 1 billion API Calls
• At Peak (Feb 2014):
– 300k DAU
– 120 fixed game servers
– 110 EC2 game servers
AWS Gaming Solutions
PlayFab Core Architecture Tenants
• No scheduled downtime.
• Automatic scaling for load.
• Backend changes should be
invisible to developers.
• Use buffering to smooth over
spikes or failures.
AWS Gaming Solutions
AWS cloud: PlayFab Web Services
US EU
Hardware
Game Servers
Hardware
Game Servers
Game
Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2
(API handling)Matchmaker
Instance Instance
Game Server
Monitor
DynamoDB Amazon RDS Amazon S3
Elastic Load Balancing
Amazon RedshiftReports
service
Instance
LogsA
rch
itec
ture
Overv
iew
Matchmaker
(Secondary)
InstanceUS EU
SA AUS
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers
AWS Gaming Solutions
List of PlayFab Web APIsClient Server
Authentication Title-Wide Data Management Shared Group Data Authentication
AddUsernamePassword GetCatalogItems AddSharedGroupMembers AuthenticateSessionTicket
LoginWithAndroidDeviceID GetStoreItems CreateSharedGroup Account Management
LoginWithFacebook GetTitleData GetSharedGroupData GetUserAccountInfo
LoginWithGameCenter GetTitleNews RemoveSharedGroupMembers SendPushNotification
LoginWithGoogleAccount AddUserVirtualCurrency UpdateSharedGroupData Player Data Management
LoginWithIOSDeviceID Player Item Management Server-Side Game Logic GetLeaderboard
LoginWithPlayFab ConsumeItem GetLogicServerUrl GetLeaderboardAroundUser
LoginWithSteam GetUserInventory GetUserData
RegisterPlayFabUser RedeemCoupon GetUserInternalData
SendAccountRecoveryEmail SubtractUserVirtualCurrency GetUserReadOnlyData
Account Management UnlockContainerItem GetUserStatistics
GetAccountInfo StartPurchase UpdateUserData
GetPlayFabIDsFromFacebookIDs PayForPurchase UpdateUserInternalData
GetUserCombinedInfo ConfirmPurchase UpdateUserReadOnlyData
LinkFacebookAccount PurchaseItem UpdateUserStatistics
LinkGameCenterAccount Friend List Management Title-Wide Data Management
LinkSteamAccount AddFriend GetCatalogItems
UnlinkFacebookAccount GetFriendsList GetTitleData
UnlinkGameCenterAccount RemoveFriend SetTitleData
UnlinkSteamAccount SetFriendTags Player Item Management
UpdateEmailAddress IOS-Specific APIs AddUserVirtualCurrency
UpdatePassword RegisterForIOSPushNotification GetUserInventory
UpdateUserTitleDisplayName ValidateIOSReceipt GrantItemsToUsers
Player Data Management Matchmaking APIs SubtractUserVirtualCurrency
GetFriendLeaderboard GetCurrentGames Matchmaking APIs
GetLeaderboard GetGameServerRegions NotifyMatchmakerPlayerLeft
GetLeaderboardAroundCurrentUser Matchmake RedeemMatchmakerTicket
GetUserData StartGame Steam-Specific APIs
GetUserReadOnlyData Android-Specific APIs AwardSteamAchievement
GetUserStatistics AndroidDevicePushNotificationRegistration
UpdateUserData ValidateGooglePlayPurchase
UpdateUserStatistics Analytics
LogEvent
AWS Gaming Solutions
ELB Logging for Title Tracking
Game ClientAmazon Route 53
(3.playfabapi.com)
API Request:
Elastic Load
Balancing
Logs
Amazon RedshiftAmazon S3
bucket
Reports
service
LogsInstance
Logs include title-specific
endpoint for analytics
AWS wildcard DNS
(*.playfabapi.com)
Log event:
AWS Gaming Solutions
Loadout API Usage Report (Oct 2014)
API NAMETOTAL
CALLS
AVERAGE
BACKEND
TIME (S)
AVERAGE
RESPONSE
TIME (MS)
AVERAGE
REQUEST
TIME (MS)
TOTAL
REQUEST
TIME (M)
AVERAGE
RECIEVED
(BYTES)
AVERAGE
SENT
(BYTES)
TOTAL
ERRORS
adminapi_userinfo 93,081,685 0.03 0.04 0.04 59.83 0 452 1979
gameserver_usercustomdatareadonly 1,582,348 0.06 0.04 0.04 1.06 42 902 290
matchmaker_userinfo 1,138,842 0.20 0.04 0.04 0.81 0 10046 319
matchmaker_playerjoined 1,138,247 0.01 0.04 0.04 0.79 64 2 93
matchmaker_playerleft 1,136,797 0.01 0.04 0.04 0.80 64 2 400
matchmaker_authuser 736,607 0.01 0.03 0.04 0.52 0 48 107
gameserver_usercustomdata 297,735 0.19 0.04 0.04 0.20 0 4080 0
gameserver_usercustomdatainternal 297,469 0.08 0.04 0.04 0.20 0 237 2
AWS Gaming Solutions
Instance
Making a Singleton Highly Available
Health
Check
Auto Scaling group
Auto Scaling
service
Instance
Matchmaker
(Primary)
Instance
Amazon Route 53)
Elastic Load
Balancing
Matchmaker
(Secondary)
1.Auto Scaling is set to keep
two instances up.
2.Elastic Beanstalk Health
Check terminates bad
instance.
3.Amazon Route 53 fails
over all traffic to secondary
server.
4.Auto Scaling detects
failure and starts new
instance.
AWS Gaming Solutions
Monitoring Service Health
Amazon EC2
(API handling)
Elastic Load
Balancing CloudWatch
Health
Check
Email AlertsOn-call Engineer
Mobile Alerts
AWS Gaming Solutions
Handling Spikes Without Losing Data
Amazon EC2 API Server
Server SSD
Amazon Redshift
Local Reports
service Local event
logsInstance
Events:
logins and
purchasesReports
service
Logs
Customer’s game
management dashboard
Analytics
Server
Logs
RabbitMQ
message
queue
LogsInstance
Logstash
service
Logstash
Instance
ElasticSearch
Instance
Instance
AWS Gaming Solutions
Storage in DynamoDB, Amazon RDS, Amazon S3,
Amazon Redshift
DynamoDB Amazon RDS Amazon S3
Player data:
• Accounts
• Profile
• Inventory
• Save data
• Player stats
• Player currency
values
• Auditing
• Log files
• Game replays
Amazon Redshift
• Analytics data
AWS Gaming Solutions
Managing Tables in DynamoDB
Amazon
DynamoDB
• Player data: accounts, profile, inventory, save data
User:
UserID: Hash
TitleID: range key
- DisplayName
- Linked accounts
- …
Custom user data:
UserID: Hash
TitleID: range key
- Key/value 1
- Key/value 2
- …
Inventory:
UserID: Hash
ItemID: range key
- titleID
- Catalog Item
- …
…
AWS Gaming Solutions
Physical Deployment
AZ 1
AZ 2
PlayFab Web
Services
PlayFab Web
Services
N. California
AZ 1
AZ 2
PlayFab Web
Services
PlayFab Web
Services
Oregon
AZ 1
PlayFab Web
Services
Beijing
• Original deployment: us-west-1
• Expanded to us-west-2 in July
• Coming soon: cn-north-1
Amazon EC2
Game Server
Singapore
Game Client
Instance
AWS Gaming Solutions
Managing Cross-Region Data
Player accounts are shared
across all regions:• Cross-regional wrapper for
user.get sets order of
databases and regions to
check
• user.get then checks for
player account data
• High availability, transparent
to customers and players
// Returns the first matching player record, searching DynamoDB and SimpleDB
// in both regions in the specified order.
// Updates the player record in the store and region from which it was loaded.
// Creates a player record in the default store and region.
AWS Gaming Solutions
AWS cloud: PlayFab Web Services
US EU
Hardware
Game Servers
Hardware
Game ServersGame
Client
Amazon Route 53
(3.playfabapi.com)
Amazon EC2
(API handling)Matchmaker
Instance Instance
Game Server
Monitor
DynamoDB Amazon RDS Amazon S3
Elastic Load Balancing
Amazon RedshiftReports
service
Instance
LogsA
rch
itec
ture
Overv
iew
Matchmaker
(Secondary)
InstanceUS EU
SA AUS
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers
Amazon EC2
Game Servers