net flix kafka seattle meetup
Post on 16-Mar-2018
161 Views
Preview:
TRANSCRIPT
●○○
●○○○
●●●
A NETFLIX ORIGINAL SERVICE
Stream Consumers
Router
EMR
FrontingKafka
EventProducer
ConsumerKafka
Management
HTTPPROXY
Fronting Kafka Clusters Consumer Kafka Clusters
Number of clusters 24 15
Total number of instances 1700+ 1100+
Instance type d2.2xl i2.2xl
Replication factor 2 2
Retention period 8 to 24 hours 2 to 4 hours
●○○○
●○
●
○●
●○
●
●●
○○
●●
●
0.1% 0.5% 1% 5% Percent loss
●●
○○
●
●●
●
●○○
● …
●
○ →
●●
○○○ →
●
●●
○○○
●○○○
●●
●○ →
●○ →
●○ →
●●
○●
○
Rack 0 Rack 1
0
Broker 0 Broker 1 Broker 2 Broker 3
3 0 1 1 2 2 3
N = Partition N for a topic with 2 replicas
0 ← Off line partition
Rack 0 Rack 1
0
Broker 0 Broker 1 Broker 2 Broker 3
3 1 2 0 1 2 3
N = Partition N for a topic with 2 replicas
No offline partition
●
●○
●
●
●○○○
●○○
EventProducer
Kafka
Buffer exhausted and message drop Slow replication
Broker with networking problem
Disk read causes slow responses
X
X
X
28
29
●●
○○○
●○○○
●○○
●●●
●●
●●
RouterFrontingKafka
EventProducer
XConsumer
Kafka
Copy topic metadata
Consumer
●
Fully Automated
top related