jamie grier - robust stream processing with apache flink

of 24 /24
Robust Stream Processing with Apache Flink Jamie Grier @jamiegrier [email protected]

Upload: flink-forward

Post on 08-Jan-2017

158 views

Category:

Data & Analytics


0 download

TRANSCRIPT

Page 1: Jamie Grier - Robust Stream Processing with Apache Flink

Robust Stream Processing

with Apache Flink

Jamie Grier @jamiegrier [email protected]

Page 2: Jamie Grier - Robust Stream Processing with Apache Flink

Who am I?• Director of Applications Engineering at data

Artisans

• Previously working on streaming computation at Twitter, Gnip and Boulder Imaging

• Involved in various kinds of stream processing for about a decade

• High-speed video, social media streaming, general frameworks for stream processing

Page 3: Jamie Grier - Robust Stream Processing with Apache Flink

Overview• What is Apache Flink?

• What is Stateful Stream Processing?

• Windowed computation over streams

• Robust Time Handling (Event Time vs Processing Time)

• Robust Failure Handling

• Robust Planned Downtime Handling

• Robust Reprocessing

Page 4: Jamie Grier - Robust Stream Processing with Apache Flink

What is Apache Flink?

Apache Flink is an open source platform for distributed stream and batch data processing.

Page 5: Jamie Grier - Robust Stream Processing with Apache Flink

What is Apache Flink?

Page 6: Jamie Grier - Robust Stream Processing with Apache Flink

Stream Processing

Your Code

Data Stream Data Stream

Page 7: Jamie Grier - Robust Stream Processing with Apache Flink

Stateful Stream Processing

Your Code

Data Stream Data Stream

State

Page 8: Jamie Grier - Robust Stream Processing with Apache Flink

More Complex Example

Kafka

Files

RabbitMQ

Filter

Map

Join / Sum

InfluxDB

C*

Page 9: Jamie Grier - Robust Stream Processing with Apache Flink

Distributed and Parallel Deployment

Kafka

Files

Rabbit

MQ

Filter

Parse

Join / Sum

InfluxDB

C*

Page 10: Jamie Grier - Robust Stream Processing with Apache Flink

Robust Stream Processing with Apache Flink

Page 11: Jamie Grier - Robust Stream Processing with Apache Flink

Code Example!

Page 12: Jamie Grier - Robust Stream Processing with Apache Flink

Windowing

Page 13: Jamie Grier - Robust Stream Processing with Apache Flink

Processing Time vs

Event Time

Page 14: Jamie Grier - Robust Stream Processing with Apache Flink

Windowing in Processing Time

0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9

Processing Time

Event Time

Page 15: Jamie Grier - Robust Stream Processing with Apache Flink

Windowing in Event Time

0 1 2 34 56 7 8 9 0 1 2 3 4 5 6 7 8 9

Event Time

Page 16: Jamie Grier - Robust Stream Processing with Apache Flink

Processing Time = Errors!

Page 17: Jamie Grier - Robust Stream Processing with Apache Flink

Event Time = Accuracy

Page 18: Jamie Grier - Robust Stream Processing with Apache Flink

Failure Handling

Page 19: Jamie Grier - Robust Stream Processing with Apache Flink

Downtime Handling

Page 20: Jamie Grier - Robust Stream Processing with Apache Flink

Data Reprocessing

Page 21: Jamie Grier - Robust Stream Processing with Apache Flink

We’re Hiring!http://data-artisans.com/careers

Page 22: Jamie Grier - Robust Stream Processing with Apache Flink

Flink Forward 2016, Berlin Submission deadline: June 30, 2016 Early bird deadline: July 15, 2016 www.flink-forward.org

Page 23: Jamie Grier - Robust Stream Processing with Apache Flink

Questions?

Page 24: Jamie Grier - Robust Stream Processing with Apache Flink

Thanks!