bdx 2016- monal daxini @ netflix
TRANSCRIPT
![Page 1: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/1.jpg)
The way to deal with Big data problems
Monal Daxini
March 2016
![Page 2: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/2.jpg)
Monal DaxiniReal Time Data Infrastructure
Senior Software Engineer, Netflixhttps://www.linkedin.com/in/monaldaxini
@monaldax
#Netflix #Keystone
![Page 3: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/3.jpg)
We help Produce,Store,
Process,Move
Events @ scale
![Page 4: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/4.jpg)
Tell me more...
● Big Data Ecosystem @ Netflix● How we built a scalable event pipeline - Keystone - in a year
○ Replaced legacy system without service disruption○ Small team 8 +1
● Netflix Culture○ Relevant tenets tagged on the slides
![Page 5: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/5.jpg)
Global Launch - Jan 6, 2016
![Page 6: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/6.jpg)
Over 75M Members
190 Countries
125M hours/day → 11B hours / quarter
14,269 years / day → 1,255,707 years / quarter
1000+ devices
37% of Internet traffic at peak
![Page 7: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/7.jpg)
Netflix Is a Data Driven Company
Content
Product
Marketing
Finance
Business Development
Talent
Infrastructure
← C
ultu
re o
f Ana
lytics→
![Page 8: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/8.jpg)
Data @ Netflix
Data at Rest (batch)
Data in Motion (streaming)
![Page 9: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/9.jpg)
Big Data Systems - batch
Ingestion / Kafka -> Ursula, Aegisthus
Storage / S3, Teradata, Redshift, Druid
Processing / Pig, Hive, Presto, Spark
Reporting / Microstrategy, Tableau, Sting
Scheduling / UC4
Interface / Big Data Portal, Kragle
Open source &
Community Driven
![Page 10: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/10.jpg)
Big Data Systems - batch
![Page 11: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/11.jpg)
Scale - batch
AWS S3 (instead of HDFS)
40 PB (S3) Compressed
Of which 13 PB events data
![Page 12: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/12.jpg)
Big Data Systems - streaming
Data Pipeline - Keystone
Playback & operational insight - Mantis
Stream Processing* - Spark Streaming
Metrics & monitoring - Atlas
Loosely Coupled
Highly AlignedOpen so
urce &
Community
Driven
![Page 13: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/13.jpg)
What does culture have to do with big data?
![Page 14: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/14.jpg)
Netflix Culture Deck
Netflix CultureFreedom & Responsibility
![Page 15: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/15.jpg)
"It may well be the most important document ever to come out of the Valley." 1
Sheryl SandbergCOO, Facebook
1 Business Insider, 2013
![Page 16: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/16.jpg)
A NETFLIX ORIGINAL SERVICE
How we built an internal facing 1 trillion / day stream processing cloud platform in a year, and how culture played a pivotal role
Freedom & Responsibility
![Page 17: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/17.jpg)
Years ago...
![Page 18: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/18.jpg)
In the Old Days ...
EMR
EventProducers
![Page 19: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/19.jpg)
Chukwa/Suro + Real-Time Branch
![Page 20: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/20.jpg)
About a year ago ...
![Page 21: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/21.jpg)
Chukwa / Suro + Real-Time Branch
EventProducer
Druid
Stream Consumers
EMR
ConsumerKafka
Suro Router
EventProducer
Suro
Kafka
SuroProxy
![Page 22: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/22.jpg)
Support at-least-once processing
Scale, Ease of Operations
Replace dormant open source software - Chukwa
Enable future value adds - Stream Processing As a Service
Seamless transition to the new platform
Context Not Control
![Page 23: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/23.jpg)
Migrate Events to a new Pipeline In flight,while not losing more that 0.1% of them
Context Not ControlHighly A
ligned
Loosel
y Cou
pled
![Page 24: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/24.jpg)
Jan 2016
![Page 25: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/25.jpg)
Keystone
Stream Consumers
SamzaRouter
EMR
FrontingKafka
ConsumerKafka
Control Plane
EventProducer
KS
Prox
y
![Page 26: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/26.jpg)
1 trillion events ingested per day during holiday season
1+ trillion events processed every day
350 billion a year ago 600+ billion events ingested per day
Keystone - Scale - Streaming
![Page 27: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/27.jpg)
11 million events (24 GB per second) peak
Upto 10MB payload / Avg 4K
1.3 PB / day
Keystone - Scale - Streaming
![Page 28: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/28.jpg)
Events & Producers
![Page 29: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/29.jpg)
Keystone
Stream Consumers
SamzaRouter
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control Plane
![Page 30: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/30.jpg)
Event Payload is ImmutableAt-least-once semantics*
* Once the event makes it to Kafka, ther are disaster scenarios where this breaks.
![Page 31: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/31.jpg)
Injected Event Metadata
● GUID
● Timestamp
● Host
● App
![Page 32: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/32.jpg)
Keystone Extensible Wire Protocol
● Backwards and forwards compatibility
● Supports JSON, AVRO on the horizon
● Invisible to source & sinks
● Efficient - 10 bytes overhead per message
○ because message size - hundreds of bytes to 10MB
![Page 33: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/33.jpg)
Netflix Kafka Producer
● Best effort delivery - ack = 1
● Prefer drop event than disrupting producer app
● Resume event production after Kafka cluster restore
● Integration with Netflix Ecosystem
● Configurable topic to Kafka clusters route
![Page 34: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/34.jpg)
Fronting Kafka Clusters
![Page 35: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/35.jpg)
Keystone
Stream Consumers
SamzaRouter
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control Plane
![Page 36: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/36.jpg)
● Pioneer Tax● Started with 0.7● In prod with 0.8.2● Move to 0.9 & VPC in progress
Kafka in the Cloud
![Page 37: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/37.jpg)
Based on topics assigned
● Normal-priority (majority)● High-priority (streaming activities etc.)
Fronting Kafka Topic Classification
![Page 38: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/38.jpg)
● ≅3200 d2.xl brokers for regular, failover, & consumer
● 125 Zookeeper nodes○ Independent zookeeper cluster per Kafka cluster
● 24 island clusters, 8 per region○ 3 ASGs per cluster, 1 ASG per zone○ 24 warm standby 3 node failover clusters
Scale - Kafka (prod)
![Page 39: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/39.jpg)
● No dynamic topic creation● Two copies● Zone aware assignment of Topic partitions and replica
Fronting Kafka Topics
![Page 40: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/40.jpg)
In a distributed system make sure you understand limitations and failures,
even if you don’t know all the features.
- Monal
![Page 41: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/41.jpg)
In addition, we doKafka Kong once a week
![Page 42: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/42.jpg)
Fronting Kafka Failover
Self Service Tool
Blameless Culture
![Page 43: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/43.jpg)
Fronting Kafka Failover
![Page 44: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/44.jpg)
Fronting Kafka Failover
![Page 45: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/45.jpg)
Kafka Management UI (Beta)Open sourcing on the road map
Open source
&
Community
Driven
![Page 46: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/46.jpg)
![Page 47: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/47.jpg)
![Page 48: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/48.jpg)
Kafka AuditorOpen sourcing on the road map
Open source
&
Community
Driven
![Page 49: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/49.jpg)
Kafka Auditor - One pre cluster
● Broker monitoring
● Consumer monitoring
● Heart-beat & Continuous message latency
● On-demand Broker performance testing
● Built as a service deployable on single or multiple instances
![Page 50: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/50.jpg)
Kafka Cluster Size -Tips
● Per Cluster Stay under 10k partitions & 200 brokers
● Leave approx. 40% free disk space on each broker
![Page 51: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/51.jpg)
● Started with AWS zone aware partition assignments
● We have discovered and filed several bugs
○ Details - Upcoming in Netflix Tech blog
Kafka ContributionsOpen source &
Community Driven
![Page 52: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/52.jpg)
Routing Service
![Page 53: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/53.jpg)
Keystone
Stream Consumers
SamzaRouter
EMR
FrontingKafka
EventProducer
ConsumerKafka
Control Plane
![Page 54: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/54.jpg)
Routing Infrastructure
+
CheckpointingCluster
+ 0.9.1Go
C language
![Page 55: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/55.jpg)
Router Job Manager(Control Plane)
EC2 InstancesZookeeper
(Instance Id assignment)
JobJobJob
ksnode
Checkpointing Cluster
ASG
![Page 56: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/56.jpg)
Custom Go Executor
./runJob
Logs
Snapshots
Attach Volumes
./runJob./runJob
Reconcile Loop - 1 minHealth Check
What’s running in ksnode?
Zookeeper(Instance Id assignment)
![Page 57: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/57.jpg)
Logs ZFS Volume Snapshots
Custom Go Executor
./runJo
b.
/runJob
./runJo
b
Go Tools Server��Client ToolsStream Logs
Browse through rotated logs by date
Ksnode Tooling
![Page 58: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/58.jpg)
Yes! You inferred right!
No Mesos & No Yarn
![Page 59: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/59.jpg)
Distributed Systems are HardKeep it Simple
Minimize Moving Parts
![Page 60: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/60.jpg)
● 13,000 docker containers (samza jobs)○ 7,000 - S3 Sink○ 4,500 - Consumer Kafka sink○ 1,500 - Elasticsearch sink
● 1,300 AWS C3-4XL instances
Scale - Routing Service
![Page 61: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/61.jpg)
More Info - Samza Meetup (10/2015)
Samza ver 0.9.1 Contributions
Open source &
Community Driven
![Page 62: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/62.jpg)
Target & Achieved <= 0.1% diff
bw Chukwa & Keystone pipeline,
over 2.6 PB of data / day
Chukwa & Keystone Pipeline Shadowing
![Page 63: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/63.jpg)
Metrics & Monitoring
![Page 64: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/64.jpg)
Keystone
Stream Consumers
SamzaRouter
EMR
FrontingKafka
ConsumerKafka
Control Plane
EventProducer
KS
Prox
y
![Page 65: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/65.jpg)
Customer Facing per topic end-to-end dashboard
![Page 66: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/66.jpg)
Dev facing infrastructure end-to-end dashboard
![Page 67: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/67.jpg)
Scaling Avenues
![Page 68: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/68.jpg)
● Exposed cost attribution per event producers & topic
○ E.g. one producer reduced throughput by 600%
● Automation - frees up additional resources
Scaling Up by Scaling Down
![Page 69: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/69.jpg)
● No dedicated product or project managers
● No separate devops or operational team
● This does not mean we are constantly overworked
○ we make wise and simple choices and
○ lean towards automation & self-healing systems.
We build and run what you saw today!
You build It!
You run it!High Perf
ormance
![Page 70: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/70.jpg)
Not DevOps, but move towards NoOps
You build it! You run it!
![Page 71: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/71.jpg)
● High Performance culture● Communication● No culture of process adherence
○ Creativity & Self Discipline○ Freedom and Responsibility
![Page 72: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/72.jpg)
Looking into the future?
![Page 73: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/73.jpg)
Streaming Processing As a Service
● multi-tenant polyglot support of streaming engines like
Spark Streaming, Mantis, Samza, and may be Flink
Future stepsOpen source &
Community Driven
![Page 74: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/74.jpg)
Messaging As a Service
● Kafka & Others● Spark Streaming, Mantis, Samza, and may be Flink.
Future stepsOpen source &
Community Driven
![Page 75: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/75.jpg)
Data thruway
● Support for schemas - registry, discovery, validation.
Self Service Tooling
Future stepsOpen source &
Community Driven
![Page 76: BDX 2016- Monal daxini @ Netflix](https://reader030.vdocuments.net/reader030/viewer/2022020301/5873bb491a28abbc788b55d3/html5/thumbnails/76.jpg)
More brain food...
Netflix OSS
Samza Meetup Presentation
Netflix Tech Blog
Spark Summit 2015 Talk