(pfc308) how dropbox scales massive workloads using amazon sqs | aws re:invent 2014
DESCRIPTION
In this session, learn how Dropbox scales to provide one of the largest cloud storage and file sharing services in the world. Hear how Dropbox leverages Amazon EC2 to run varied workloads including thumbnail generation and document prevent, as well as document indexing to support full-text search. Dropbox presents ''Livefill'' - a generic framework built on top of Amazon SQS. Livefill enables them to trigger customizable data-processing workloads on data stored in Amazon S3 and helps them support more than 200,000 workload requests per second, spread across thousands of machines.TRANSCRIPT
November 12, 2014
Akhil Gupta, Head of Infrastructure, Dropbox
web app
server
Dropbox data center
Databases
S3 Bucket
web app
server
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
web app
server
web app
server
S3 Bucket
Dropbox data center
Databases
web app
server web app
server
S3 Bucket
Dropbox data center
Databases
S3 Bucket
web app
server
Livefill
Server
Livefill
Extractor
Livefill
Extractor
Virtual Cluster #1
Virtual Cluster #3
Virtual Cluster #2
In Queue
Out Queue
Dropbox data center
Databases
S3 Bucket
web app
server
Livefill
Server
Livefill
Extractor
Livefill
Extractor
Virtual Cluster #1
Virtual Cluster #3
Virtual Cluster #2
Process File A
InQueue
Out Queue
S3 Bucket
Livefill
Extractor
Virtual Cluster #1
InQueue
Out Queue
S3 Bucket
Virtual Cluster #1
Read File A
S3 Bucket
Fetch File A
Handler 1Handler 2
Dropbox data center
Databases
S3 Bucket
web app
server
Livefill
Server
Livefill
Extractor
Livefill
Extractor
Virtual Cluster #1
Virtual Cluster #3
Virtual Cluster #2
Store extracted
metadata
InQueue
Out Queue
