DEPLOYING THE LEAGUE OF LEGENDS DATA PIPELINE WITH CHEF
ENGINEERING MANAGER| NOVEMBER 2013 TROTTER CASHION
2009 LAUNCH
TEAM ORIENTED ONLINE PC
GAME
100+ CHAMPS
LIVE PLAYERS VS. LIVE PLAYERS
Image source: herocomplex.la2mes.com/
WE WORK WITH BIG DATA
SEMI-GLOBAL DEPLOYMENT 0 DOWNTIME
RUNS IN CLOUD (AWS) + DATACENTER
70+ BILLION TESTED @
EVENTS PER DAY
7+ BILLION EVENTS PER DAY 10+ TABLES
100MM TO 1B ROWS/DAY
100+ TABLES
7+ PETABYTE GAME EVENT DATASET
AGENDA CHEF SERVER
CORE CONCEPTS
CHEF SOLO
OVERVIEW
GOLDEN IMAGES
BASED ON REAL WORLD USAGE Big Data Team from April 2012 to June 2013
Also incorporates subsequent work with Player Accounts team
OVERVIEW
OVERVIEW
WHAT DO WE WANT FROM AUTOMATED INFRASTRUCTURE?
OVERVIEW
WHAT DO WE WANT TO BE ABLE TO DO FOR PLAYERS?
TO PLAY LOL
‣ Minimize Unplanned Down2me
‣ Rolling Deploys ‣ Expand Capacity
NEW FEATURES
‣ Low Maintenance ‣ Low Barrier for New Services
WHAT PLAYERS WANT OVERVIEW
OVERVIEW
CORE: ALWAYS UP, EASY TO DEPLOY
AGENDA CHEF SERVER
CORE CONCEPTS
CHEF SOLO
OVERVIEW
GOLDEN IMAGES
UNDERLYING COMPONENTS CORE CONCEPTS
UNDERLYING COMPONENTS CORE CONCEPTS
ELB
EC2
UNDERLYING COMPONENTS CORE CONCEPTS
ELB
EC2
UNDERLYING COMPONENTS CORE CONCEPTS
VPC VPC
ELB
EC2
UNDERLYING COMPONENTS CORE CONCEPTS
VPC VPC
ELB
VGW
EC2
UNDERLYING COMPONENTS CORE CONCEPTS
VPC VPC
ELB
DC
VGW
VPN
S3 DATA STORAGE
EC2
UNDERLYING COMPONENTS CORE CONCEPTS
VPC VPC
ELB
VGW
DATA CENTER VS. CLOUD CORE CONCEPTS
DATA CENTER VS. CLOUD
Min. 2W & 3G
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
Min. 2W & 3G
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W
Min. 2W & 3G
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W
Min. 2W & 3G
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W
G
Min. 2W & 3G
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W
G G
Min. 2W & 3G
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W
G G G
Min. 2W & 3G
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W
G G G
Min. 2W & 3G
85%
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W
G G G
Min. 2W & 3G
85%
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W W
G G G
Min. 2W & 3G
85%
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W W
G G G
MANUAL
manual scaling no Chef required
Min. 2W & 3G
85%
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W W
G G G
MANUAL
manual scaling no Chef required
AUTOSCALING CLOUD
Min. 2W & 3G
85%
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W W
G G G
MANUAL
manual scaling no Chef required
AUTOSCALING CLOUD
Min. 2W & 3G
85%
CORE CONCEPTS
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W W
G G G
MANUAL
manual scaling no Chef required
AUTOSCALING CLOUD
G
Min. 2W & 3G
85%
CORE CONCEPTS
W W W W
G G G G
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W W
G G G
MANUAL
manual scaling no Chef required
AUTOSCALING CLOUD
W W
G G G
W
G G
Min. 2W & 3G
85%
CORE CONCEPTS
W
DATA CENTER VS. CLOUD
DATA CENTER MODEL
W W W
G G G
MANUAL
manual scaling no Chef required
AUTOSCALING CLOUD
W W W
G G G G
AUTO
automatic scaling automated provisioning needed
Min. 2W & 3G
85%
CORE CONCEPTS
W
G
userdata.sh
AMI
START
OPERATING SYSTEM
CORE CONCEPTS
userdata.sh
AMI
START
OPERATING SYSTEM
GOAL
Publish Hello World
• Instantly
• Automatically
CORE CONCEPTS
userdata.sh
AMI
START
OPERATING SYSTEM
GOAL
Publish Hello World
• Instantly
• Automatically
1st BOOT
CORE CONCEPTS
userdata.sh
AMI
START
OPERATING SYSTEM
GOAL
Publish Hello World
• Instantly
• Automatically
Userdata.sh 1st BOOT
CORE CONCEPTS
userdata.sh
GOAL
Publish Hello World
• Instantly
• Automatically
AMI
START
OPERATING SYSTEM
#!/bin/bash
yum install apache
wget http://hw/hello_world.html…
Userdata.sh 1st BOOT
CORE CONCEPTS
AMI
START
userdata.sh
OPERATING SYSTEM
yum install apache
wget s3://hw/hello_world.html… RE
PLAC
ED W
/
GOAL
Publish Hello World
• Instantly
• Automatically
#!/bin/bash
yum install apache
wget s3://hw/hello_world.html…
Userdata.sh 1st BOOT
wget http://hw/first-boot.json
chef-client
CORE CONCEPTS
IMMUTABLE SERVERS LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
CORE CONCEPTS
IMMUTABLE SERVERS LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
CORE CONCEPTS
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVERS
W1 W1 W1
W1 W1
LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
MUTABLE SERVER:
CHANGE EACH BOX TO NEW VERSION
CORE CONCEPTS
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVERS
W2 W1 W1
W1 W1
LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
MUTABLE SERVER:
CHANGE EACH BOX TO NEW VERSION
CORE CONCEPTS
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVERS
W2 W2 W1
W1 W1
LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
MUTABLE SERVER:
CHANGE EACH BOX TO NEW VERSION
CORE CONCEPTS
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVERS
W2 W2 W2
W1 W1
LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
MUTABLE SERVER:
CHANGE EACH BOX TO NEW VERSION
CORE CONCEPTS
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVERS
W2 W2 W2
W2 W1
LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
MUTABLE SERVER:
CHANGE EACH BOX TO NEW VERSION
CORE CONCEPTS
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVERS
W2 W2 W2
W2 W2
LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
MUTABLE SERVER:
CHANGE EACH BOX TO NEW VERSION
CORE CONCEPTS
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVERS LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
W1 W1 W1
W1 W1
IMMUTABLE SERVER:
FLIP ON BOXES WITH NEW VERSION, SHUT OFF EXISTING BOXES
CORE CONCEPTS
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVERS
W2 W2 W2
W2 W2
LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
IMMUTABLE SERVER:
FLIP ON BOXES WITH NEW VERSION, SHUT OFF EXISTING BOXES
W1 W1 W1
W1 W1
CORE CONCEPTS
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVERS
W2 W2 W2
W2 W2
LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU PREVENT UNEXPECTED CHANGES TO SERVERS
IMMUTABLE SERVER:
FLIP ON BOXES WITH NEW VERSION, SHUT OFF EXISTING BOXES
W1 W1 W1
W1 W1
CORE CONCEPTS
IMMUTABLE SERVERS LIMITED TO 2 ACTIONS: 1) Start & provision server 2) Kill it
WITH IMMUTABLE SERVERS, YOU CAN’T SSH INTO BOX & CHANGE THINGS
BENEFITS:
No snowflakes
Easy rollbacks
Cleaner deploys
CORE CONCEPTS
AGENDA CHEF SERVER
CORE CONCEPTS
CHEF SOLO
OVERVIEW
GOLDEN IMAGES
CHEF SOLO WHAT’S REQUIRED
• All cookbooks in one tarball • Somewhere to put the tarball • Tell instances how to provision with the tarball
CHEF SOLO WHAT’S REQUIRED
• All cookbooks in one tarball • Somewhere to put the tarball • Tell instances how to provision with the tarball
PACKAGING
• Use Berkshelf • `berkshelf package COOKBOOK_NAME` • `tar czvf cookbooks-VERSION.tgz ./cookbooks`
CHEF SOLO WHAT’S REQUIRED
• All cookbooks in one tarball • Somewhere to put the tarball • Tell instances how to provision with the tarball
PACKAGING
• Use Berkshelf • `berkshelf package COOKBOOK_NAME` • `tar czvf cookbooks-VERSION.tgz ./cookbooks`
STORAGE OPTIONS
• S3 • `s3cmd` makes it easy to upload to S3
• Internal asset server
CHEF SOLO WHAT’S REQUIRED
• All cookbooks in one tarball • Somewhere to put the tarball • Tell instances how to provision with the tarball
PACKAGING
• Use Berkshelf • `berkshelf package COOKBOOK_NAME` • `tar czvf cookbooks-VERSION.tgz ./cookbooks`
STORAGE OPTIONS
• S3 • `s3cmd` makes it easy to upload to S3
• Internal asset server
PROVISIONING
• userdata.sh • Get cookbooks off s3 and untar to /var/chef/
cookbooks • Get first-boot.json off s3 -> /etc/chef/first-
boot.json • Get solo.rb off s3 -> /etc/chef/solo.rb • Run `chef-solo`
CHEF SOLO
WHERE IT WORKS
• Good when your Chef run is fast • Easy to set up • No single point of failure
CHEF SOLO: PROs VS CONs
WHERE IT FAILS
• When Chef runs occasionally fail • Configuration changes • Service discovery
TO PLAY LOL
ü Minimize Unplanned Down2me
ü Rolling Deploys ü Expand Capacity
NEW FEATURES
q Low Maintenance q Low Barrier for New Services
WHAT PLAYERS WANT CHEF SOLO
RECOMMENDATION:
CHEF SOLO
• Use chef-solo approach to get started
• Stop when you have multiple applications
AGENDA CHEF SERVER
CORE CONCEPTS
CHEF SOLO
OVERVIEW
AMIs
WHAT’S REQUIRED
• Chef Server • Cookbooks uploaded to Chef Server • Tell instances how to provision with Chef Server
CHEF SERVER
WHAT’S REQUIRED
• Chef Server • Cookbooks uploaded to Chef Server • Tell instances how to provision with Chef Server
RUNNING CHEF SERVER
• Run your own • Hosted Chef
CHEF SERVER
WHAT’S REQUIRED
• Chef Server • Cookbooks uploaded to Chef Server • Tell instances how to provision with Chef Server
GETTING COOKBOOKS TO CHEF SERVER
• Use berkshelf • `berks upload COOKBOOK_NAME`
RUNNING CHEF SERVER
• Run your own • Hosted Chef
CHEF SERVER
PROVISIONING
• userdata.sh • Fetch validation.pem • Fetch first-boot.json • Fetch client.rb • Run `chef-client …`
GETTING COOKBOOKS TO CHEF SERVER
• Use berkshelf • `berks upload COOKBOOK_NAME`
WHAT’S REQUIRED
• Chef Server • Cookbooks uploaded to Chef Server • Tell instances how to provision with Chef Server
RUNNING CHEF SERVER
• Run your own • Hosted Chef
CHEF SERVER
WHERE IT WORKS
• Updating feature flag configuration • Warning: Can affect rolling
deploys!
CHEF SERVER: PROs VS CONs
WHERE IT FAILS
• Single point of failure in Chef Server • Long Chef runs cause problems • Can cause problems in large
organizations where multiple teams depend on the same cookbook
CHEF SERVER
TO PLAY LOL
q Minimize Unplanned Down2me
q Rolling Deploys q Expand Capacity
NEW FEATURES
ü Low Maintenance q Low Barrier for New Services
WHAT PLAYERS WANT CHEF SERVER
CHEF SERVER
RECOMMENDATION:
• Use chef-server when managing feature flags
• Not bad when you have multiple applications to deploy
• Pro tips:
• Use a shutdown script to remove instances from the chef server
• Only use chef in daemon mode if you’re certain only feature flags will change
AGENDA CHEF SERVER
CORE CONCEPTS
CHEF SOLO
OVERVIEW
GOLDEN IMAGES
GOLDEN IMAGES
WHAT’S REQUIRED
• Means to create Golden Image • Configuration management
WHAT’S REQUIRED
• Means to create Golden Image • Configuration management
Creating Image
• We use Chef-Solo • Build an RPM • Cookbook installs RPMs
GOLDEN IMAGES
WHAT’S REQUIRED
• Means to create Golden Image • Configuration management
GOLDEN IMAGES
Configuration Management
• Archaius (on disk config) • Chef
Creating Image
• We use Chef-Solo • Build an RPM • Cookbook installs RPMS
WHERE IT WORKS
• Rollbacks are perfect and instances always boot
• Boot times are very fast
CHEF SOLO: PROs VS CONs
WHERE IT FAILS
• Need another method for configuration management
• Need another method for service discovery
• Releasing patches requires entire new build
GOLDEN IMAGES
GOLDEN IMAGES
TO PLAY LOL
ü Minimize Unplanned Down2me
ü Rolling Deploys ü Expand Capacity
NEW FEATURES
ü Low Maintenance q Low Barrier for New Services
WHAT PLAYERS WANT GOLDEN IMAGES
RECOMMENDATION:
• Use when auto scaling is a must
• Requires an up-front effort to get going
• Make sure you can afford it
• Pro tip:
• Use the Netflix stack: Asgard, Aminator, and Archaius
GOLDEN IMAGES
c THANK YOU! QUESTIONS? ENGINEERING MANAGER| NOVEMBER 2013 TROTTER CASHION