globalizing player accounts with mysql at riot …...continuent tungsten third-party vendor provides...
TRANSCRIPT
![Page 1: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/1.jpg)
Globalizing Player Accounts with MySQL at Riot Games
Tyler Turk
Riot Games, Inc.
![Page 2: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/2.jpg)
About Me
Senior Infrastructure Engineer
Player Platform at Riot Games
Father of three
Similar talk at re:Invent last year
2
![Page 3: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/3.jpg)
Accounts Team
Responsible for account data
Provides account management
Ensures players can login
Aims to mitigate account compromises
3
![Page 4: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/4.jpg)
Overview
The old and the new
![Page 5: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/5.jpg)
5
League’s growth and shard deployment
Launched in 2009
Experienced rapid growth
Deployed multiple game shards
Each shard used their own MySQL DBs
![Page 6: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/6.jpg)
6
Some context
Hundreds of millions of players worldwide
Localized primary / secondary replication
Data federated with each shard
Account transfers were difficult
![Page 7: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/7.jpg)
7
Why MySQL?
Widely used & adopted at Riot
Used extensively by Tencent
Ensures ACID compliance
![Page 8: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/8.jpg)
8
Catalysts for globalization
General Data Protection Regulation
Decoupling from game platform
Single source of truth for accounts
![Page 9: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/9.jpg)
Globalization of Player Accounts
Migrating from 10 isolated databases to a single globally replicated database
![Page 10: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/10.jpg)
10
Data deployment considerations
Globally replicated, multi-master
Globally replicated, single master
Federated or sharded data
To cache or not to cache
![Page 11: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/11.jpg)
11
Global database expectations
Highly available
Geographically distributed
< 1 sec latency replication
< 20ms read latency
Enables a better player experience
![Page 12: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/12.jpg)
12
Continuent Tungsten
Third-party vendor
Provides cluster orchestration
Manages data replication
MySQL connector proxy
![Page 13: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/13.jpg)
13
Why Continuent Tungsten?
Prior issues with Aurora
RDS was not multi-region
Preferred asynchronous replication
Automated cluster management
![Page 14: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/14.jpg)
14
Explanation & tolerating failure
![Page 15: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/15.jpg)
15
Deployment
Terraform & Ansible (docker initially)
4 AWS regions
r4.8xlarge (10Gbps network)
5TB GP2 EBS for data
15TB for logs / backups
![Page 16: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/16.jpg)
16
Migrating the data
Multi-step migration of data
Consolidated data into 1 DB
Multiple rows for a single account
![Page 17: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/17.jpg)
17
Load testing
![Page 18: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/18.jpg)
18
Chaos testing
![Page 19: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/19.jpg)
19
Monitoring
![Page 20: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/20.jpg)
20
Performing backups
Leverage standalone replicator
Backup with xtrabackup
Compress and upload to S3
Optional delay on replicator
![Page 21: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/21.jpg)
21
Performing maintenance
Cluster policies
Offline and shun nodes
Perform cluster switch
![Page 22: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/22.jpg)
Performing schema changes
Schema MUST be backwards compatible
Order of operations for schema change:
1. Replicas in non-primary region
2. Cluster switch on relay
3. Perform change on former relay
4. Repeat steps 1-3 on all non-primary regions
5. Replicas in primary region
6. Cluster switch on write primary
7. Perform change on former write
22
The Process
• Offline node
• Wait for connections to drain
• Stop replicator
• Perform schema change
• Start replicator
• Wait for replication
• Online node
![Page 23: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/23.jpg)
23
De-dockering
Fully automated the process
One server at a time
Performed live
Near zero downtime
![Page 24: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/24.jpg)
24
Current state
Database deployed on host
No docker for database / sidecars
Accounts are distilled to a single row
Servicing all game shards
![Page 25: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/25.jpg)
Lessons Learned
Avoiding the same mistakes we made
![Page 26: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/26.jpg)
26
Databases in docker
Partially immutable infrastructure
Configuration divergence possible
Upgrades required container restarts
Pain in automating deploys
![Page 27: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/27.jpg)
27
Large data imports
Consider removing indexes
Perform daily delta syncs
Migrate in chunks if possible
![Page 28: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/28.jpg)
28
Think about data needs
Synchronous vs asynchronous
Read heavy vs write heavy
![Page 29: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/29.jpg)
29
Impacts of replication latency
Replication can take >1 second
Impacts strongly consistent expectations
Immediate read-backs can fail
Think about “eventual” consistency
![Page 30: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/30.jpg)
30
WAN replication is fragile
Not completely infallible
Think through your needs
Architect and design accordingly
Even with RiotDirect, it’s not perfect
![Page 31: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/31.jpg)
31
Backup with caution (aka backups v1)
![Page 32: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/32.jpg)
32
Demo Time!
![Page 34: Globalizing Player Accounts with MySQL at Riot …...Continuent Tungsten Third-party vendor Provides cluster orchestration Manages data replication MySQL connector proxy 13 Why Continuent](https://reader033.vdocuments.net/reader033/viewer/2022060503/5f1d20c52dfe2b3b775138f8/html5/thumbnails/34.jpg)
34
Rate My Session