Download - Marton Balassi – Stateful Stream Processing
![Page 2: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/2.jpg)
This talk
Stateful stream processing by example
Open source stream processors
Runtime architecture
Fault tolerance
Stateful processing
Closing
2Flink Forward2015-10-13
![Page 3: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/3.jpg)
Stateful stream processing by example
2015-10-13 Flink Forward 3
![Page 4: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/4.jpg)
Streaming applications
ETL style operations
• Filter incoming data, Log analysis
• No or minimal state
Window aggregations
• Trending tweets, User sessions, Stream joins
• State: Window buffer
2015-10-13 Flink Forward 4
Inpu
tInpu
tInpu
tInput
Process/Enrich
![Page 5: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/5.jpg)
Streaming applications
Machine learning
• Fitting trends to the evolving stream, Stream clustering
• State: Model
Pattern recognition
• Fraud detection, Triggering signals based on activity
• State: Finite state machine
5Flink Forward2015-10-13
![Page 6: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/6.jpg)
State in streaming programs
Almost all non-trivial streaming programs are stateful
Stateful operators (in essence):
𝒇: 𝒊𝒏, 𝒔𝒕𝒂𝒕𝒆 ⟶ 𝒐𝒖𝒕, 𝒔𝒕𝒂𝒕𝒆′
State hangs around and can be read and modified as the stream evolves
Goal: Get as close as possible while maintaining scalability and fault-tolerance
6Flink Forward2015-10-13
![Page 7: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/7.jpg)
Open source stream processors
2015-10-13 Flink Forward 7
![Page 8: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/8.jpg)
Apache Streaming landscape
82015-10-13 Flink Forward
![Page 9: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/9.jpg)
Apache Storm
Started in 2010, development driven by BackType, then Twitter
Pioneer in large scale stream processing
Distributed dataflow abstraction (spouts & bolts)
92015-10-13 Flink Forward
![Page 10: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/10.jpg)
Apache Flink
Started in 2008 as a research project (Stratosphere) at European universities
Unique combination of low latency streaming and high throughput batch analysis
Flexible operator states and windowing
10
Batch data
Kafka, RabbitMQ, ...
HDFS, JDBC, ...
Stream Data
2015-10-13 Flink Forward
![Page 11: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/11.jpg)
Apache Samza
Developed at LinkedIn, open sourced in 2013
Builds heavily on Kafka’s log based philosophy
Pluggable messaging system and execution backend
112015-10-13 Flink Forward
![Page 12: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/12.jpg)
Apache Spark
Started in 2009 at UC Berkley, Apache since 2013
Very strong community, wide adoption
Unified batch and stream processing over a batch runtime
Good integration with batch programs
122015-10-13 Flink Forward
![Page 13: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/13.jpg)
Runtime and programming model
2015-10-13 Flink Forward 13
![Page 14: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/14.jpg)
Native Streaming
2015-10-13 Flink Forward 14
![Page 15: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/15.jpg)
Distributed dataflow runtime
Storm, Samza and Flink
General properties• Long standing operators• Pipelined execution• Usually possible to create
cyclic flows
2015-10-13 Flink Forward 15
Pros
• Full expressivity
• Low-latency execution
• Stateful operators
Cons
• Fault-tolerance is hard
• Throughput may suffer
• Load balancing is an issue
![Page 16: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/16.jpg)
Micro-batching
2015-10-13 Flink Forward 16
![Page 17: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/17.jpg)
Micro-batch runtime
Implemented by Apache Spark
General properties• Computation broken down
to time intervals
• Load aware scheduling
• Easy interaction with batch
2015-10-13 Flink Forward 17
Pros• Easy to reason about• High-throughput• FT comes for “free”• Dynamic load balancing
Cons• Latency depends on
batch size• Limited expressivity• Stateless by nature
![Page 18: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/18.jpg)
Fault tolerance
2015-10-13 Flink Forward 18
![Page 19: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/19.jpg)
Fault tolerance intro
Fault-tolerance in streaming systems is inherently harder than in batch
• Can’t just restart computation• State is a problem• Fast recovery is crucial• Streaming topologies run 24/7 for a long period
Fault-tolerance is a complex issue• No single point of failure is allowed• Guaranteeing input processing• Consistent operator state• Fast recovery • At-least-once vs Exactly-once semantics
2015-10-13 Flink Forward 19
![Page 20: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/20.jpg)
Storm record acknowledgements
Track the lineage of tuples as they are processed (anchors and acks)
Special “acker” bolts track each lineage DAG (efficient xor based algorithm)
Replay the root of failed (or timed out) tuples
2015-10-13 Flink Forward 20
![Page 21: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/21.jpg)
Samza offset tracking
Exploits the properties of a durable, offset based messaging layer
Each task maintains its current offset, which moves forward as it processes elements
The offset is checkpointed and restored on failure (some messages might be repeated)
2015-10-13 Flink Forward 21
![Page 22: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/22.jpg)
Flink checkpointing
Based on consistent global snapshots
Algorithm designed for stateful dataflows(minimal runtime overhead)
Exactly-once semantics
22Flink Forward2015-10-13
![Page 23: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/23.jpg)
Spark RDD recomputation
Immutable data model with repeatable computation
Failed RDDs are recomputed using their lineage
Checkpoint RDDs to reduce lineage length
Parallel recovery of failedRDDs
Exactly-once semantics
2015-10-13 Flink Forward 23
![Page 24: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/24.jpg)
Stateful stream processing
2015-10-13 Flink Forward 24
![Page 25: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/25.jpg)
State in streaming programs
Almost all non-trivial streaming programs are stateful
Stateful operators (in essence):
𝒇: 𝒊𝒏, 𝒔𝒕𝒂𝒕𝒆 ⟶ 𝒐𝒖𝒕, 𝒔𝒕𝒂𝒕𝒆′
State hangs around and can be read and modified as the stream evolves
Goal: Get as close as possible while maintaining scalability and fault-tolerance
25Flink Forward2015-10-13
![Page 26: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/26.jpg)
States available only in Trident API
Dedicated operators for state updates and queries
State access methods• stateQuery(…)
• partitionPersist(…)
• persistentAggregate(…)
It’s very difficult toimplement transactionalstates
Exactly-once guarantee
26Flink Forward2015-10-13
![Page 27: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/27.jpg)
Storm Trident Word Count
27Flink Forward2015-10-13
![Page 28: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/28.jpg)
Stateless runtime by design• No continuous operators
• UDFs are assumed to be stateless
State can be generated as a separate stream of RDDs: updateStateByKey(…)
𝒇: 𝑺𝒆𝒒[𝒊𝒏𝒌], 𝒔𝒕𝒂𝒕𝒆𝒌 ⟶ 𝒔𝒕𝒂𝒕𝒆′𝒌
𝒇 is scoped to a specific key
Exactly-once semantics
28Flink Forward2015-10-13
![Page 29: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/29.jpg)
val stateDstream = wordDstream.updateStateByKey[Int](
newUpdateFunc,
new HashPartitioner(ssc.sparkContext.defaultParallelism),
true,
initialRDD)
val updateFunc = (values: Seq[Int], state: Option[Int]) => {
val currentCount = values.sum
val previousCount = state.getOrElse(0)
Some(currentCount + previousCount)}
Spark Streaming Word Count
29Flink Forward2015-10-13
![Page 30: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/30.jpg)
Stateful dataflow operators(Any task can hold state)
State changes are storedas a log by Kafka
Custom storage engines canbe plugged in to the log
𝒇 is scoped to a specific task
At-least-once processingsemantics
30Flink Forward2015-10-13
![Page 31: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/31.jpg)
Samza Word Count
public class WordCounter implements StreamTask, InitableTask {
//Some omitted details…
private KeyValueStore<String, Integer> store;
public void process(IncomingMessageEnvelope envelope,
MessageCollector collector,
TaskCoordinator coordinator) {
//Get the current count
String word = (String) envelope.getKey();
Integer count = store.get(word);
if (count == null) count = 0;
//Increment, store and send
count += 1;
store.put(word, count);
collector.send(
new OutgoingMessageEnvelope(OUTPUT_STREAM, word ,count));
}}
31Flink Forward2015-10-13
![Page 32: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/32.jpg)
Stateful dataflow operators (conceptually similar to Samza)
Two state access patterns• Local (Task) state
• Partitioned (Key) state
Proper API integration• Java: OperatorState interface
• Scala: mapWithState, flatMapWithState…
Exactly-once semantics by checkpointing
32Flink Forward2015-10-13
0.9.1
![Page 33: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/33.jpg)
Flink Word Count
words.keyBy(x => x).mapWithState {
(word, count: Option[Int]) =>
{
val newCount = count.getOrElse(0) + 1
val output = (word, newCount)
(output, Some(newCount))
}}
33Flink Forward2015-10-13
![Page 34: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/34.jpg)
Local State Example (Java)
public class MySource extends RichParallelSourceFunction {
// Omitted details
private OperatorState<Long> offset;
@Override
public void run(SourceContext ctx) {
Object checkpointLock = ctx.getCheckpointLock();
isRunning = true;
while (isRunning) {
synchronized (checkpointLock) {
offset.update(offset.value() + 1);
//ctx.collect(next);
}
}
}}
34Flink Forward2015-10-13
![Page 35: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/35.jpg)
Internal operators are checkpointed:• Aggregations
• Window operators
• …
KeyValue state• Easing common access patterns
Flexible state backend interface
Removes non-partitioned operator state
35Flink Forward2015-10-13
0.10
![Page 36: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/36.jpg)
Performance (Fault tolerance)
2015-10-13 Flink Forward 36
![Page 37: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/37.jpg)
Performance (Statefullness)
2015-10-13 Flink Forward 37
CheckpointInterval: 5 sec BatchDuration: 5 sec
![Page 38: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/38.jpg)
Closing
2015-10-13 Flink Forward 38
![Page 39: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/39.jpg)
Summary Storm (Trident)
+ Consistent state accessible from outside– Only works well with idempotent states– States are not part of the operators
Spark+ Integrates well with the system guarantees– Limited expressivity– Immutability increases update complexity
Samza+ Efficient log based state updates+ States are well integrated with the operators– Lack of exactly-once semantics– State access is not fully transparent
Flink+ Light-weight, exactly once distributed snaphots+ Flexible checkpointing and state backend interfaces– Has to coordinate with a persistent source– Internal operators not checkpointed in 0.9.1
39Flink Forward2015-10-13
![Page 40: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/40.jpg)
Thank you!
![Page 41: Marton Balassi – Stateful Stream Processing](https://reader033.vdocuments.net/reader033/viewer/2022042907/5871384b1a28abf0568b6319/html5/thumbnails/41.jpg)
List of Figures (in order of usage)
https://upload.wikimedia.org/wikipedia/commons/thumb/2/2a/CPT-FSM-abcd.svg/326px-CPT-FSM-abcd.svg.png
https://storm.apache.org/images/topology.png
https://databricks.com/wp-content/uploads/2015/07/image11-1024x655.png
https://databricks.com/wp-content/uploads/2015/07/image21-1024x734.png
https://people.csail.mit.edu/matei/papers/2012/hotcloud_spark_streaming.pdf, page 2.
http://www.slideshare.net/ptgoetz/storm-hadoop-summit2014, page 69-71.
http://samza.apache.org/img/0.9/learn/documentation/container/checkpointing.svg
https://databricks.com/wp-content/uploads/2015/07/image41-1024x602.png
https://storm.apache.org/documentation/images/spout-vs-state.png
http://samza.apache.org/img/0.9/learn/documentation/container/stateful_job.png
2015-10-13 Flink Forward 41