kafka, the "dialtone for data": building a self-service, scalable, streaming analytics...

© Copyright 2016 HomeAway, Inc.

Kafka: The “Dial Tone” for Data

HomeAwayThe world leader for vacation

rentals

> 1 million listings(and growing!)

Agenda


• Overview• The Problem• The Experiment• Results: Use Cases• Lessons Learned• Next Steps


Overview

Difference between Dinosaurs and Unicorns


In the old days: “Dial Tone” looked like this


ATDT

Today: Kafka is the modern “Dial Tone” for Data


Producer

Consumer

The Problem


Our original problem/motivation


search head

indexer

indexerapp server forwarder

app server forwarder

1 TB/day ingress and growing!40,000 calls/sec

Also… Historical Analytic Pipeline was slow/expensive


app server

OLTP OLAP

analyticsETL

Fill the Lake! Alternatives

?

Problem: Fill Hadoop!

Problem Data Lake


What we wanted… the Big Idea


If you can log it… … you can analyze it!

How to build self-service?


Hypothesis: Use Kafka!


2 ms medianlatency

http://bit.ly/jay_on_logs the log

2 Million Events / Sec! (3 cheap machines)

http://goo.gl/pv5GoL “Benchmarking Apache

Kafka”

http://bit.ly/jay_on_logs

http://goo.gl/pv5GoL

http://goo.gl/pv5GoL


The Experiment

HACommonsLogging• KafkaAppender

Schema-on-read• KafkaAvroLogger

Schema-on-write

Experiment: Schema-on-Read, Schema-on-Write

Data Lake


SchemaRegistry

Architecture: Kafka + Camus = BigData Ingress


Camus


The Results

Use Cases: ITOA / SLA Reporting


Use Cases: Fraud


Use Cases: Search + ClickStream


User Behavior

Search RequestsA/B Test

Readouts

Proctor

EDAP

Use Cases: Search + ClickStream


Use Cases: Traveler Segmentation


EDAP

Data Mode

l

Lessons Learned


Lesson #1: The Schema [registry] is Everything!

Data Lake


SchemaRegistry

• Decouples producers from consumers

• Enforces backwards compatibility

• Enables self-service / democratization

• SOT for schemas in the pipe

Lesson #2: A Kafka/SR governance module is helpful

Data Lake


• TURN OFF Auto Topic Creation!

• Need a place for developersto request topics• Retention Policy• Expected Load• Compaction• Partition Size / Partition Key• Owner• LTS Date

Lesson #3: Make it easy to do stream processing


SchemaRegistry

• samza-archetype• samza-job-deployer

• Will evaluate k-streams!!!!

http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple

Next Steps


Consistency : 3 types of Data


Event

Document

Transactional

Kafka Producer Spooling


Conclusion


Yesterday


Systems of Record

Today


Systems of Engagement

Tomorrow


Systems of Intelligence

Don’t be a dinosaur…


ATDT

Thank you


End of Presentation


kafka, the "dialtone for data": building a self-service, scalable, streaming analytics...

Engineering