(adv402) beating the speed of light with your infrastructure in aws | aws re:invent 2014
DESCRIPTION
With Amazon Web Services it's possible to serve the needs of modern high performance advertising without breaking the bank. This session covers how AdRoll processes more than 60 billion requests per day in less than 100 milliseconds each using Amazon DynamoDB, Auto Scaling, and Elastic Load Balancing. This process generates more than 2 GB of data every single second, which will be processed and turned into useful models over the following hour. We discuss designing systems that can answer billions of latency-sensitive global requests every day and look into some tricks to pare down the costs.TRANSCRIPT
![Page 1: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/1.jpg)
November 14, 2014 | Las Vegas, NV
Valentino Volonghi, CTO, AdRoll
Siva Raghupathy, Principal Solutions Architect, AWS
![Page 2: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/2.jpg)
60 billion requests/day
![Page 3: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/3.jpg)
We
Must
Stay
Up
1% downtime
=
>$1M
![Page 4: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/4.jpg)
No
Infinitely
Deep
Pockets
![Page 5: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/5.jpg)
100ms MAX Latency
![Page 6: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/6.jpg)
Paris-New York: ~6000km
Speed of Light in fiber: 200,000 km/s
RTT latency without hops and copper:
60ms
Paris-New York: ~6000km
Speed of Light in fiber: 200,000 km/s
RTT latency without hops and copper:
60ms6000
km
60 ms
c-RTT
![Page 7: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/7.jpg)
![Page 8: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/8.jpg)
![Page 9: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/9.jpg)
![Page 10: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/10.jpg)
a
a
a
r
• Memcache
aOpen source
aMature
aBlazingly fast
rNo strong guarantees
• Redis
aOpen source
rStorage scale
rNot really distributed
rOperationally intense.
• Hbase (we still use this)
aOpen source
aMaturing quickly
aGreat scale
rReally hard to operate
![Page 11: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/11.jpg)
http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html
Cassandra DynamoDB Delta
10/90 mix, $/month $287,064 $131,040 219%
50/50 mix, $/month $287,064 $280,800 ~0%
10/90, 3-yr reserved $27,075.6
($904k upfront)
$15,736
($504k upfront)
180%
• 10 people Cassandra ops team: $150k/month (fully loaded)
• 0 DynamoDB ops team: $0
![Page 12: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/12.jpg)
![Page 13: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/13.jpg)
![Page 14: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/14.jpg)
![Page 15: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/15.jpg)
![Page 16: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/16.jpg)
![Page 17: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/17.jpg)
Instance
Nginx Erlang Bidder Black Box
Supervisor
ASG
Amazon
Kinesis
Amazon S3
DynamoDB
Elastic Load
Balancing
Instance
Nginx Erlang Bidder Black Box
Supervisor
ASG
Elastic Load
Balancing
![Page 18: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/18.jpg)
Amazon S3
Amazon
Kinesis
DynamoDB
DynamoDB
DynamoDB
![Page 19: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/19.jpg)
![Page 20: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/20.jpg)
HashKey Value
Key {‘segments’: set([‘123’, ‘345’]),
‘timestamp’: ‘1234556’,
‘leads’: ‘to-write-contention’}
HashKey RangeKey Value (just a single
timestamp)
Key Segment 1234554343254
Key Segment1 1231231433235
![Page 21: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/21.jpg)
![Page 22: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/22.jpg)
ArrayList
List WriteRequest ArrayList WriteRequest
Key
AtomicInteger AtomicInteger
Set WriteRequest CollectionsHashSet WriteRequest
String
AmazonDynamoDB
Runnable Runnable run
List WriteRequest
BatchWriteItemResult BatchWriteItemRequest
ImmutableMap List WriteRequest
Map
![Page 23: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/23.jpg)
![Page 24: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/24.jpg)
![Page 25: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/25.jpg)
![Page 26: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/26.jpg)
![Page 27: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/27.jpg)
![Page 28: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/28.jpg)
![Page 29: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/29.jpg)
Not for me!
![Page 30: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/30.jpg)
![Page 31: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/31.jpg)
![Page 32: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/32.jpg)
![Page 33: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/33.jpg)
![Page 34: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/34.jpg)
![Page 35: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/35.jpg)
![Page 36: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/36.jpg)
![Page 37: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/37.jpg)
Data Collection
Data Storage
Global
Distribution
Bid Storage
Bidding
![Page 38: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/38.jpg)
Data Collection
Data Storage
Global
Distribution
Bid Storage
Bidding
![Page 39: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/39.jpg)
Data Collection = Batch Layer Bidding = Speed Layer
Data Collection
Data Storage
Global
Distribution
Bid Storage
Bidding
![Page 40: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/40.jpg)
US East region
Availability Zone Availability Zone
Elastic Load
Balancing
instances instancesAuto Scaling group
Amazon S3Amazon
Kinesis
![Page 41: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/41.jpg)
US East region
Availability Zone Availability Zone
Elastic Load
Balancing
instances instancesAuto Scaling group
Amazon S3Amazon
KinesisApache
StormDynamoDB
US West region
EU West region
DynamoDB
DynamoDB
![Page 42: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/42.jpg)
BiddingData Collection
US East region
Availability Zone Availability Zone
Elastic Load Balancing
instances instances
Auto Scaling group
Amazon
S3
Amazon
KinesisApache
StormDynamoDB
Availability Zone Availability Zone
Auto Scaling group
Elastic Load Balancing
![Page 43: (ADV402) Beating the Speed of Light with Your Infrastructure in AWS | AWS re:Invent 2014](https://reader033.vdocuments.net/reader033/viewer/2022060121/559446b11a28abfc728b4648/html5/thumbnails/43.jpg)
Data Collection
• Amazon EC2, Elastic Load Balancing, Auto Scaling
Store
• Amazon S3 + Amazon Kinesis
Global Distribution
• Apache Storm on Amazon EC2
Bid Store• DynamoDB
Bidding
• Amazon EC2, Elastic Load Balancing, Auto Scaling
Data Collection
Bidding
Ad Network 2Ad Network 1
Auto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling Group Auto Scaling GroupAuto Scaling Group
Auto Scaling GroupAuto Scaling Group Auto Scaling Group
Apache Storm
v2 V3 V3v1 v2 V3 V3v1
V2 V3 V3V1
Auto Scaling Group
V3 V4
Elastic Load Balancing Elastic Load Balancing Elastic Load Balancing Elastic Load Balancing
DynamoDB
Write
Read Read Read ReadRead Read
WriteWrites
WriteWrite
Read
V3`
Elastic Load Balancing
Elastic Load Balancing
Elastic Load Balancing
Elastic Load Balancing
Elastic Load Balancing
Elastic Load Balancing
DynamoDB
Data Collection
Bidding
DynamoDB
Write
Read
Read
Write
Write
WriteAmazon S3
Amazon Kinesis