engineering v v adam cataldo tuesday, january 24, 2012 quick deploy a distributed systems approach...
Post on 17-Dec-2015
219 Views
Preview:
TRANSCRIPT
EngineeringEngineeringv
Adam Cataldo
Tuesday, January 24, 2012
Quick DeployA distributed systems approach to developer productivity
Engineering 2
Outline
LinkedIn architecture Quick Deploy Cool technical details
Engineering 3
LinkedIn Largest professional networking site
135M+ members 14 languages 200+ countries and
territories Estimated 4B+ people
searches per year
Engineering 4
LinkedIn stack at 20,000 feet
Online system for serving up data to clients– Profile pages, company pages, LinkedIn groups, etc.– Key metric: Latency
Offline system for creating insight from data– People you may know, contact degree of separation,
search indexing, etc.– Key metric: Throughput
Engineering 5
Offline data processingHadoopVoldemort
Map
Map
Map
Map
Reduce
Reduce
Reduce
Engineering 6
Online web applicationClient Presentation Business Logic Data Service
Datastore
Engineering 7
Growing Pains for the Online System
As time has gone by:– We’ve added new
services to support new features
– We’ve added new services to make our site faster and more available
– Our engineering productivity dropped
Engineering 8
Quick Deploy
Developers only locally deploy the services they actually modify
Every service your service depends on is available in some data center
This makes development move much faster!
AngelikaClayton
Engineering 9
Early Integration DatacenterDev Box
QD Proxy
Browser
Local Mid Tier
Shared Frontend
Shared Data Service
DB
Engineering 10
In Detail
Browser ProxyEI
FrontendDev-localMid Tier
EIData Service
QD CookieQD Cookie
QD IC
QD IC
QD IC
QD IC
Engineering 11
Proxy Behavior
Proxy receives request
Has QD cookie?
Route to developer’s local service
Route to service in EI
Read identifier
from cookie
Read identifier from IC
Is service local?
no
yes
yes
no
Engineering 12
QD Cookie and Invocation Context
A developer creates an identifier key, and registers his local services with the proxy
He identifies himself by adding his key to a cookie from his browser
When a front end gets a request with a QD cookie, it puts the key in the invocation context
Services pass the invocation context with every outbound request
Engineering 13
Shared data
OracleMySQL
VoldemortMemcached
Change Alfred’slanguage to
Japaneseきんぎょ
Huh?
Engineering 14
ActiveMQEarly Integration DatacenterDev Box
Producer
Consumer
ConsumerProducesmessage Consumes
message
QueueNever seesmessage
Engineering 15
TimersEarly Integration DatacenterDev Box
Service withTimer
Service
RPC Target
RPC TargetSchedulestime-triggered
RPC call Gets RPC call aftertimer triggers
Never gets RPC call
Engineering 16
Proxy in detail
Admin ConsolePython/Flask Web App
ProxyApache Traffic Server
> 10,000 QPS!
Routing ProfilesApache Zookeeper
ProxyApache Traffic Server
ProxyApache Traffic Server
Engineering 17
Why Apache Zookeeper
Distributed file system for small files Basically, a key-value data store, where the keys
are file names Clients can listen for file change and directory
change events
Great for storing data Also, great publish/subscribe system
Engineering 18
Dynamic DiscoveryEarly Integration DatacenterDev Box
QD ProxyBrowser
Local Mid Tier
Shared Frontend
Shared Data Service DB
Engineering 19
Conclusion
The LinkedIn architecture scales well in production
Quick deploy makes the architecture scale for developers
20% productivity gain across the engineering organization
Engineering 20
Thanks
RobLoh
SweeLim
SteveIhde
JimDumont
top related