tiny batches, in the wine: shiny new bits in spark streaming
DESCRIPTION
London Spark Meetup 2014-11-11 @Skimlinks http://www.meetup.com/Spark-London/events/217362972/ To paraphrase the immortal crooner Don Ho: "Tiny Batches, in the wine, make me happy, make me feel fine." http://youtu.be/mlCiDEXuxxA Apache Spark provides support for streaming use cases, such as real-time analytics on log files, by leveraging a model called discretized streams (D-Streams). These "micro batch" computations operated on small time intervals, generally from 500 milliseconds up. One major innovation of Spark Streaming is that it leverages a unified engine. In other words, the same business logic can be used across multiple uses cases: streaming, but also interactive, iterative, machine learning, etc. This talk will compare case studies for production deployments of Spark Streaming, emerging design patterns for integration with popular complementary OSS frameworks, plus some of the more advanced features such as approximation algorithms, and take a look at what's ahead — including the new Python support for Spark Streaming that will be in the upcoming 1.2 release. Also, let's chat a bit about the new Databricks + O'Reilly developer certification for Apache Spark…TRANSCRIPT
![Page 1: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/1.jpg)
Tiny Batches, in the wine: Shiny New Bits in Spark Streaming
London Spark Meetup 2014-11-11 meetup.com/Spark-London/events/217362972/
Paco Nathan @pacoid
1
![Page 2: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/2.jpg)
http://youtu.be/mlCiDEXuxxA
BTW, in case anyone didn’t get the pun…
2
![Page 3: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/3.jpg)
Spark, the elevator pitch
3
![Page 4: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/4.jpg)
Developed in 2009 at UC Berkeley AMPLab, open sourced in 2010, Spark has since become one of the largest OSS communities in big data, with over 200 contributors in 50+ organizations
Spark, the elevator pitch
spark.apache.org
“Organizations that are looking at big data challenges – including collection, ETL, storage, exploration and analytics – should consider Spark for its in-memory performance and the breadth of its model.
It supports advanced analytics solutions on Hadoop clusters, including the iterative model required for machine learning and graph analysis.”
Gartner, Advanced Analytics and Data Science (2014)
4
![Page 5: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/5.jpg)
Spark, the elevator pitch
5
![Page 6: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/6.jpg)
Spark Core is the general execution engine for the Spark platform that other functionality is built atop: !• in-memory computing capabilities deliver speed
• general execution model supports wide variety of use cases
• ease of development – native APIs in Java, Scala, Python (+ SQL, Clojure, R)
Spark, the elevator pitch
6
![Page 7: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/7.jpg)
WordCount in 3 lines of Spark
WordCount in 50+ lines of Java MR
Spark, the elevator pitch
7
![Page 8: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/8.jpg)
Sustained exponential growth, as one of the most active Apache projects ohloh.net/orgs/apache
Spark, the elevator pitch
8
![Page 9: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/9.jpg)
A very brief history
9
![Page 10: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/10.jpg)
Theory, eight decades ago: what can be computed?
Haskell Curry haskell.org
Alonso Churchwikipedia.org
A Brief History: Functional Programming for Big Data
John Backusacm.org
David Turnerwikipedia.org
Praxis, four decades ago: algebra for applicative systems
Pattie MaesMIT Media Lab
Reality, two decades ago: machine data from web apps
10
![Page 11: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/11.jpg)
A Brief History: Functional Programming for Big Data
circa 2002: mitigate risk of large distributed workloads lost due to disk failures on commodity hardware…
Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung research.google.com/archive/gfs.html !MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean, Sanjay Ghemawat research.google.com/archive/mapreduce.html
11
![Page 12: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/12.jpg)
A Brief History: Functional Programming for Big Data
2002
2002MapReduce @ Google
2004MapReduce paper
2006Hadoop @ Yahoo!
2004 2006 2008 2010 2012 2014
2014Apache Spark top-level
2010Spark paper
2008Hadoop Summit
12
![Page 13: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/13.jpg)
A Brief History: Functional Programming for Big Data
MR doesn’t compose well for large applications, and so specialized systems emerged as workarounds
MapReduce
General Batch Processing Specialized Systems: iterative, interactive, streaming, graph, etc.
Pregel Giraph
Dremel Drill
TezImpala
GraphLab
StormS4
F1
MillWheel
13
![Page 14: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/14.jpg)
Spark: Cluster Computing with Working Sets Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scott Shenker, Ion Stoica people.csail.mit.edu/matei/papers/2010/hotcloud_spark.pdf !Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf
circa 2010: a unified engine for enterprise data workflows, based on commodity hardware a decade later…
A Brief History: Functional Programming for Big Data
14
![Page 15: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/15.jpg)
action value
RDDRDDRDD
transformations RDD
// action 1!messages.filter(_.contains("mysql")).count()
// transformed RDDs!val errors = lines.filter(_.startsWith("ERROR"))!val messages = errors.map(_.split("\t")).map(r => r(1))!messages.cache()
TL;DR: Applicative Systems and Functional Programming – RDDs
15
![Page 16: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/16.jpg)
TL;DR: Generational trade-offs for distributing compute tasks
CheapStorage
CheapMemory
CheapNetwork
recompute
replicate
reference
(RDD)
(DFS)
(URI)
16
![Page 17: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/17.jpg)
databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html
TL;DR: Smashing The Previous Petabyte Sort Record
17
![Page 18: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/18.jpg)
Why Streaming?
18
![Page 19: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/19.jpg)
Why Streaming?
Because Machine Data!
I <3 Logs Jay KrepsO’Reilly (2014) shop.oreilly.com/product/0636920034339.do
19
![Page 20: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/20.jpg)
Why Streaming?
Because Google!
MillWheel: Fault-Tolerant Stream Processing at Internet Scale Tyler Akidau, Alex Balikov, Kaya Bekiroglu, Slava Chernyak, Josh Haberman, Reuven Lax, Sam McVeety, Daniel Mills, Paul Nordstrom, Sam Whittle Very Large Data Bases (2013) research.google.com/pubs/pub41378.html
20
![Page 21: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/21.jpg)
Why Streaming?
Because IoT!
kickstarter.com/projects/1614456084/b4rm4n-be-a-cocktail-hero
21
![Page 22: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/22.jpg)
Why Streaming?
Because IoT! (exabytes/day per sensor)
bits.blogs.nytimes.com/2013/06/19/g-e-makes-the-machine-and-then-uses-sensors-to-listen-to-it/
22
![Page 23: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/23.jpg)
Spark Streaming
23
![Page 24: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/24.jpg)
Let’s consider the top-level requirements for a streaming framework:
• clusters scalable to 100’s of nodes
• low-latency, in the range of seconds(meets 90% of use case needs)
• efficient recovery from failures(which is a hard problem in CS)
• integrates with batch: many co’s run the same business logic both online+offline
Spark Streaming: Requirements
24
![Page 25: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/25.jpg)
Therefore, run a streaming computation as: a series of very small, deterministic batch jobs
!• Chop up the live stream into
batches of X seconds
• Spark treats each batch of data as RDDs and processes them using RDD operations
• Finally, the processed results of the RDD operations are returned in batches
Spark Streaming: Requirements
25
![Page 26: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/26.jpg)
Therefore, run a streaming computation as: a series of very small, deterministic batch jobs
!• Batch sizes as low as ½ sec,
latency of about 1 sec
• Potential for combining batch processing and streaming processing in the same system
Spark Streaming: Requirements
26
![Page 27: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/27.jpg)
Data can be ingested from many sources: Kafka, Flume, Twitter, ZeroMQ, TCP sockets, etc.
Results can be pushed out to filesystems, databases, live dashboards, etc.
Spark’s built-in machine learning algorithms and graph processing algorithms can be applied to data streams
Spark Streaming: Integration
27
![Page 28: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/28.jpg)
2012 project started
2013 alpha release (Spark 0.7)
2014 graduated (Spark 0.9)
Spark Streaming: Timeline
Discretized Streams: A Fault-Tolerant Model for Scalable Stream Processing Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, Ion Stoica Berkeley EECS (2012-12-14) www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-259.pdf
project lead: Tathagata Das @tathadas
28
![Page 29: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/29.jpg)
Typical kinds of applications:
• datacenter operations
• web app funnel metrics
• ad optimization
• anti-fraud
• various telematics
and much much more!
Spark Streaming: Requirements
29
![Page 30: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/30.jpg)
Programming Guidespark.apache.org/docs/latest/streaming-programming-guide.html
TD @ Spark Summit 2014 youtu.be/o-NXwFrNAWQ?list=PLTPXxbhUt-YWGNTaDj6HSjnHMxiTD1HCR
“Deep Dive into Spark Streaming”slideshare.net/spark-project/deep-divewithsparkstreaming-tathagatadassparkmeetup20130617
Spark Reference Applicationsdatabricks.gitbooks.io/databricks-spark-reference-applications/
Spark Streaming: Some Excellent Resources
30
![Page 31: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/31.jpg)
import org.apache.spark.streaming._!import org.apache.spark.streaming.StreamingContext._!!// create a StreamingContext with a SparkConf configuration!val ssc = new StreamingContext(sparkConf, Seconds(10))!!// create a DStream that will connect to serverIP:serverPort!val lines = ssc.socketTextStream(serverIP, serverPort)!!// split each line into words!val words = lines.flatMap(_.split(" "))!!// count each word in each batch!val pairs = words.map(word => (word, 1))!val wordCounts = pairs.reduceByKey(_ + _)!!// print a few of the counts to the console!wordCounts.print()!!ssc.start()!ssc.awaitTermination()
Quiz: name the bits and pieces…
31
![Page 32: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/32.jpg)
Because Use Cases
32
![Page 33: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/33.jpg)
Because Use Cases: +40 known production use cases
33
![Page 34: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/34.jpg)
Reasons for adopting/transitioning to Spark Streaming… the unified programming model is particularly relevant for real-time analytics that combine historical data:
• Making data science accessible to non-scientists
• Higher productivity for data workers
• Exactly-once semantics
• No compromises on scalability and throughput
• Ease of operations
Because Use Cases: Analysis
34
![Page 35: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/35.jpg)
Reasons for adopting Spark Streaming:
• Making data science accessible to non-scientists
Spark’s declarative APIs enable users who have domain expertise but lack data science expertise.
In other words, express a business problem and its associated processing algorithm and data pipeline by using simple, high-level operators.
Because Use Cases: Analysis
35
![Page 36: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/36.jpg)
Reasons for adopting Spark Streaming:
• Higher productivity for data workers
Spark’s write-once-run-anywhere approach unifies batch and stream processing.
That ties together the different components ofan analytics pipeline in the same tool – discovery, ETL, data engineering, machine learning model training and execution – across all types of structured and unstructured data.
Because Use Cases: Analysis
36
![Page 37: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/37.jpg)
Reasons for adopting Spark Streaming:
• Exactly-once semantics
Many crucial use cases in business need exactly-once stateful processing
Not at-most-once (which includes zero) or at-least-once (which includes duplicates)
Exactly-once provides users certainty on questions such as the exact number of fraud cases, emergencies, or outages occurring within a time period
Because Use Cases: Analysis
37
![Page 38: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/38.jpg)
Reasons for adopting Spark Streaming:
• No compromises on scalability and throughput
Spark Streaming is designed for hyper-scale environments and combines statefulness and persistence with high throughput.
Because Use Cases: Analysis
38
![Page 39: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/39.jpg)
Reasons for adopting Spark Streaming:
• Ease of operations
Spark provides a unified run time across different processing engines.
One physical cluster and one set of operational processes covers the full spectrum of use cases.
Because Use Cases: Analysis
39
![Page 40: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/40.jpg)
Because Use Cases: Sharethrough
Sharethrough Uses Spark Streaming to Optimize Bidding in Real Time Russell Cardullo, Michael Ruggier 2014-03-25 databricks.com/blog/2014/03/25/sharethrough-and-spark-streaming.html
40
• the profile of a 24 x 7 streaming app is different than an hourly batch job…
• take time to validate output against the input…
• confirm that supporting objects are being serialized…
• the output of your Spark Streaming job is only as reliable as the queue that feeds Spark…
• monoids…
![Page 41: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/41.jpg)
Because Use Cases: Viadeo
Spark Streaming As Near Realtime ETL Djamel Zouaoui 2014-09-18 slideshare.net/DjamelZouaoui/spark-streaming
41
• Spark Streaming is topology-free
• workers and receivers are autonomous and independent
• paired with Kafka, RabbitMQ
• 8 machines / 120 cores
• use case for recommender system
• issues: how to handle lost data, serialization
![Page 42: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/42.jpg)
Because Use Cases: Stratio
Stratio Streaming: a new approach to Spark Streaming David Morales, Oscar Mendez 2014-06-30 spark-summit.org/2014/talk/stratio-streaming-a-new-approach-to-spark-streaming
42
• Stratio Streaming is the union of a real-time messaging bus with a complex event processing engine using Spark Streaming
• allows the creation of streams and queries on the fly
• paired with Siddhi CEP engine and Apache Kafka
• added global features to the engine such as auditing and statistics
![Page 43: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/43.jpg)
Because Use Cases: Ooyala
Productionizing a 24/7 Spark Streaming service on YARN Issac Buenrostro, Arup Malakar 2014-06-30 spark-summit.org/2014/talk/productionizing-a-247-spark-streaming-service-on-yarn
43
• state-of-the-art ingestion pipeline, processing over two billion video events a day
• how do you ensure 24/7 availability and fault tolerance?
• what are the best practices for Spark Streaming and its integration with Kafka and YARN?
• how do you monitor and instrument the various stages of the pipeline?
![Page 44: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/44.jpg)
Because Use Cases: Guavus
Guavus Embeds Apache Spark into its Operational Intelligence Platform Deployed at the World’s Largest Telcos Eric Carr 2014-09-25 databricks.com/blog/2014/09/25/guavus-embeds-apache-spark-into-its-operational-intelligence-platform-deployed-at-the-worlds-largest-telcos.html
44
• 4 of 5 top mobile network operators, 3 of 5 top Internet backbone providers, 80% MSOs in NorAm
• analyzing 50% of US mobile data traffic, +2.5 PB/day
• latency is critical for resolving operational issues before they cascade: 2.5 MM transactions per second
• “analyze first” not “store first ask questions later”
![Page 45: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/45.jpg)
Demos
45
![Page 46: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/46.jpg)
brand new Python support for Streaming in 1.2 github.com/apache/spark/tree/master/examples/src/main/python/streaming
Twitter Streaming Language Classifier databricks.gitbooks.io/databricks-spark-reference-applications/content/twitter_classifier/README.html !!For more Spark learning resources online: databricks.com/spark-training-resources
Demos, as time permits:
46
![Page 47: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/47.jpg)
import sys!from pyspark import SparkContext!from pyspark.streaming import StreamingContext!!sc = SparkContext(appName="PyStreamNWC", master="local[*]")!ssc = StreamingContext(sc, Seconds(5))!!lines = ssc.socketTextStream(sys.argv[1], int(sys.argv[2]))!!counts = lines.flatMap(lambda line: line.split(" ")) \! .map(lambda word: (word, 1)) \! .reduceByKey(lambda a, b: a+b)!!counts.pprint()!!ssc.start()!ssc.awaitTermination()
Demo: PySpark Streaming Network Word Count
47
![Page 48: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/48.jpg)
import sys!from pyspark import SparkContext!from pyspark.streaming import StreamingContext!!def updateFunc (new_values, last_sum):! return sum(new_values) + (last_sum or 0)!!sc = SparkContext(appName="PyStreamNWC", master="local[*]")!ssc = StreamingContext(sc, Seconds(5))!ssc.checkpoint("checkpoint")!!lines = ssc.socketTextStream(sys.argv[1], int(sys.argv[2]))!!counts = lines.flatMap(lambda line: line.split(" ")) \! .map(lambda word: (word, 1)) \! .updateStateByKey(updateFunc) \! .transform(lambda x: x.sortByKey())!!counts.pprint()!!ssc.start()!ssc.awaitTermination()
Demo: PySpark Streaming Network Word Count - Stateful
48
![Page 49: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/49.jpg)
Complementary Frameworks
49
![Page 50: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/50.jpg)
Spark Integrations:
Discover Insights
Clean Up Your Data
RunSophisticated
Analytics
Integrate With Many Other
Systems
Use Lots of Different Data Sources
cloud-based notebooks… ETL… the Hadoop ecosystem… widespread use of PyData… advanced analytics in streaming… rich custom search… web apps for data APIs… low-latency + multi-tenancy…
50
![Page 51: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/51.jpg)
unified compute
Kafka + Spark + Cassandra datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/spark/sparkIntro.html http://helenaedelson.com/?p=991
github.com/datastax/spark-cassandra-connector
github.com/dibbhatt/kafka-spark-consumer
columnar key-valuedata streams
Spark Integrations: Advanced analytics for streaming use cases
51
![Page 52: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/52.jpg)
unified compute
Spark + ElasticSearch databricks.com/blog/2014/06/27/application-spotlight-elasticsearch.html
elasticsearch.org/guide/en/elasticsearch/hadoop/current/spark.html
spark-summit.org/2014/talk/streamlining-search-indexing-using-elastic-search-and-spark
document search
Spark Integrations: Rich search, immediate insights
52
![Page 53: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/53.jpg)
• use Tachyon as a best practice for sharing between two streaming apps
• or write to Cassandra or HBase / then read back
• design patterns for integration: spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations-on-dstreams
Spark Integrations: General Guidelines
53
![Page 54: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/54.jpg)
A Look Ahead…
54
![Page 55: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/55.jpg)
1. Greater Stability and Robustness
• improved high availability via write-ahead logs
• enabled as an optional feature for Spark 1.2
• NB: Spark Standalone can already restart driver
• excellent discussion of fault-tolerance (2012): cs.duke.edu/~kmoses/cps516/dstream.html
• stay tuned: meetup.com/spark-users/events/218108702/
A Look Ahead…
55
![Page 56: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/56.jpg)
2. Support for more environments, i.e., beyond Hadoop
• three use cases currently depend on HDFS
• those are being abstracted out
• could then use Cassandra, etc.
A Look Ahead…
56
![Page 57: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/57.jpg)
3. Improved support for Python
• e.g., Kafka is not exposed through Python yet (next release goal)
A Look Ahead…
57
![Page 58: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/58.jpg)
4. Better flow control
• a somewhat longer-term goal, plus it is a hard problem in general
• poses interesting challenges beyond what other streaming systems have faced
A Look Ahead…
58
![Page 59: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/59.jpg)
A Big Picture
59
![Page 60: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/60.jpg)
A Big Picture…
19-20c. statistics emphasized defensibility in lieu of predictability, based on analytic variance and goodness-of-fit tests !That approach inherently led toward a manner of computational thinking based on batch windows !They missed a subtle point…
60
![Page 61: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/61.jpg)
21c. shift towards modeling based on probabilistic approximations: trade bounded errors for greatly reduced resource costs
highlyscalable.wordpress.com/2012/05/01/probabilistic-structures-web-analytics-data-mining/
A Big Picture… The view in the lens has changed
61
![Page 62: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/62.jpg)
21c. shift towards modeling based on probabilapproximations: trade bounded errors for greatly reduced resource costs
highlyscalable.wordpress.com/2012/05/01/probabilistic-structures-web-analytics-data-mining/
A Big Picture… The view in the lens has changed
Twitter catch-phrase:
“Hash, don’t sample”
62
![Page 63: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/63.jpg)
a fascinating and relatively new area, pioneered by relatively few people – e.g., Philippe Flajolet
provides approximation, with error bounds – in general uses significantly less resources (RAM, CPU, etc.)
many algorithms can be constructed from combinations of read and write monoids
aggregate different ranges by composing hashes, instead of repeating full-queries
Probabilistic Data Structures:
63
![Page 64: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/64.jpg)
Probabilistic Data Structures: Some Examples
algorithm use case example
Count-Min Sketch frequency summaries code
HyperLogLog set cardinality code
Bloom Filter set membership
MinHash set similarity
DSQ streaming quantiles
SkipList ordered sequence search
64
![Page 65: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/65.jpg)
Probabilistic Data Structures: Some Examples
algorithm use case example
Count-Min Sketch frequency summaries code
HyperLogLog set cardinality code
Bloom Filter set membership
MinHash set similarity
DSQ streaming quantiles
SkipList ordered sequence search
65
suggestion: consider these as your most quintessential collections data types at scale
![Page 66: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/66.jpg)
Add ALL the Things: Abstract Algebra Meets Analyticsinfoq.com/presentations/abstract-algebra-analytics Avi Bryant, Strange Loop (2013)
• grouping doesn’t matter (associativity)
• ordering doesn’t matter (commutativity)
• zeros get ignored
In other words, while partitioning data at scale is quite difficult, you can let the math allow your code to be flexible at scale
Avi Bryant@avibryant
Probabilistic Data Structures: Performance Bottlenecks
66
![Page 67: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/67.jpg)
Probabilistic Data Structures: Industry Drivers
• sketch algorithms: trade bounded errors for orders of magnitude less required resources, e.g., fit more complex apps in memory
• multicore + large memory spaces (off heap) are increasing the resources per node in a cluster
• containers allow for finer-grain allocation of cluster resources and multi-tenancy
• monoids, etc.: guarantees of associativity within the code allow for more effective distributed computing, e.g., partial aggregates
• less resources must be spent sorting/windowing data prior to working with a data set
• real-time apps, which don’t have the luxury of anticipating data partitions, can respond quickly
67
![Page 68: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/68.jpg)
Probabilistic Data Structures for Web Analytics and Data MiningIlya Katsov (2012-05-01)
A collection of links for streaming algorithms and data structures Debasish Ghosh
Aggregate Knowledge blog (now Neustar) Timon Karnezos, Matt Curcio, et al.
Probabilistic Data Structures and Breaking Down Big Sequence DataC. Titus Brown, O'Reilly (2010-11-10)
Algebird Avi Bryant, Oscar Boykin, et al. Twitter (2012)
Mining of Massive DatasetsJure Leskovec, Anand Rajaraman, Jeff Ullman, Cambridge (2011)
Probabilistic Data Structures: Recommended Reading
68
![Page 69: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/69.jpg)
Resources
69
![Page 70: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/70.jpg)
databricks.com/blog/2014/07/14/databricks-cloud-making-big-data-easy.html
youtube.com/watch?v=dJQ5lV5Tldw#t=883
70
cloud-based notebooks:
![Page 71: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/71.jpg)
Apache Spark developer certificate program
• http://oreilly.com/go/sparkcert
• defined by Spark experts @Databricks
• assessed by O’Reilly Media
• establishes the bar for Spark expertise
certification:
71
![Page 72: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/72.jpg)
72
• Postcode Anywhere: • Spark, Cassandra, Elasticsearch • And machine learning • Dynamically optimizing website UX
• In house: • Spark, HBase, Elasticsearch • In a Lambda architecture • And data science • Social data mining for ad optimization
• Training: • Introduction to Spark • Regular courses • Next: Dec 5th, Feb 6th • Databricks materials & certified trainers
Big Data Partnership
http://www.bigdatapartnership.com
![Page 73: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/73.jpg)
community:
spark.apache.org/community.html
video+slide archives: spark-summit.org
events worldwide: goo.gl/2YqJZK
resources: databricks.com/spark-training-resources
workshops: databricks.com/spark-training
Intro to Spark
SparkAppDev
SparkDevOps
SparkDataSci
Distributed ML on Spark
Streaming Apps on Spark
Spark + Cassandra
73
![Page 74: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/74.jpg)
books:
Fast Data Processing with Spark Holden Karau Packt (2013) shop.oreilly.com/product/9781782167068.do
Spark in Action Chris FreglyManning (2015*) sparkinaction.com/
Learning Spark Holden Karau, Andy Konwinski, Matei ZahariaO’Reilly (2015*) shop.oreilly.com/product/0636920028512.do
74
![Page 75: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/75.jpg)
events:Big Data Spain Madrid, Nov 17-18 bigdataspain.org
Strata EUBarcelona, Nov 19-21 strataconf.com/strataeu2014 Data Day Texas Austin, Jan 10 datadaytexas.com Strata CA San Jose, Feb 18-20 strataconf.com/strata2015 Spark Summit East NYC, Mar 18-19 spark-summit.org/east
Strata EULondon, May 5-7 strataconf.com/big-data-conference-uk-2015 Spark Summit 2015 SF, Jun 15-17 spark-summit.org
75
![Page 76: Tiny Batches, in the wine: Shiny New Bits in Spark Streaming](https://reader034.vdocuments.net/reader034/viewer/2022042816/559445d11a28ab0e738b45e8/html5/thumbnails/76.jpg)
presenter:
Just Enough Math O’Reilly, 2014
justenoughmath.compreview: youtu.be/TQ58cWgdCpA
monthly newsletter for updates, events, conf summaries, etc.: liber118.com/pxn/
Enterprise Data Workflows with Cascading O’Reilly, 2013
shop.oreilly.com/product/0636920028536.do
76