Download - Who Needs Batch?
![Page 1: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/1.jpg)
SPRINGONE2GX WASHINGTON, DC
Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/
Who Needs Batch? By Michael Minella and Gunnar Hillert
@michaelminella, @ghillert
![Page 2: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/2.jpg)
![Page 3: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/3.jpg)
Michael Minella Twitter: @michaelminella Podcast: http://javaOffHeap.com or @OffHeap Website: http://spring.io
Gunnar Hillert Twitter: @ghillert Website: http://blog.hillert.com Conference: http://devnexus.com
![Page 4: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/4.jpg)
![Page 5: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/5.jpg)
https://github.com/ghillert/s1-2gx2015-who-needs-batch
![Page 6: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/6.jpg)
W E H A V E A N ANNOUNCEMENT
![Page 7: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/7.jpg)
WE HAVE A DISEASE
![Page 8: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/8.jpg)
OBJECTIVIUS SHINIUM SYNDROMUS
![Page 9: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/9.jpg)
Shiny Object Syndrom
SHINY OBJECT SYNDROME
![Page 10: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/10.jpg)
IT MAKES US INNOVATE
![Page 11: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/11.jpg)
ROAD TO RECOVERY
![Page 12: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/12.jpg)
COOL KIDS ARE DOING STREAM PROCESSING
![Page 13: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/13.jpg)
Stream Analytics
Cloud Dataflow
![Page 14: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/14.jpg)
![Page 15: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/15.jpg)
![Page 16: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/16.jpg)
LET’S TAKE A CLOSER LOOK
![Page 17: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/17.jpg)
![Page 18: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/18.jpg)
BATCH AND STREAM DEFINITIONS
![Page 19: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/19.jpg)
LOOK AT COMMON USE CASES
![Page 20: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/20.jpg)
CONCLUSIONS
![Page 21: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/21.jpg)
DEFINITION: STREAM PROCESSING
![Page 22: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/22.jpg)
A system for processing an unbounded set of data in an asynchronous manor.
![Page 23: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/23.jpg)
DEFINITION: BATCH PROCESSING
![Page 24: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/24.jpg)
DEPENDS ON WHO YOU ASK
![Page 25: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/25.jpg)
“Batch … applications run … as special cases of stream processing applications”
- Apache Flink Documentation
![Page 26: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/26.jpg)
Batch processing is the processing of a bounded data set without interruption
or interaction
![Page 27: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/27.jpg)
KEY DIFFERENCES
![Page 28: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/28.jpg)
1 BOUNDED VS UNBOUNDED
![Page 29: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/29.jpg)
2 SYNCHRONOUS VS ASYNCHRONOUS
![Page 30: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/30.jpg)
3 STATEFUL VS STATELESS
![Page 31: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/31.jpg)
STATE OF STREAMING
![Page 32: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/32.jpg)
Stream Analytics
Cloud Dataflow
![Page 33: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/33.jpg)
Stream Analytics
Cloud Dataflow
![Page 34: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/34.jpg)
STATE OF BATCH
![Page 35: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/35.jpg)
JSR-352
![Page 36: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/36.jpg)
WHY THE END OF BATCH?
![Page 37: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/37.jpg)
LATENCY
![Page 38: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/38.jpg)
WHAT DOES THE END OF BATCH MEAN?
![Page 39: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/39.jpg)
L IMIT LATENCY
![Page 40: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/40.jpg)
“ENTERPRISE GRADE”
![Page 41: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/41.jpg)
LOOK AT COMMON USE CASES
![Page 42: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/42.jpg)
E.T.L.
![Page 43: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/43.jpg)
DATABASE TO HDFS
![Page 44: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/44.jpg)
DEMO
![Page 45: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/45.jpg)
APACHE FLINK
Streaming Dataflow Runtime
DataSet API (Batch) DataStream API
Hadoop Table ML Dataflow … Table Dataflow …
Local Remote Yarn Tez Embedded
![Page 46: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/46.jpg)
STREAMING CAN BE USEFUL IN INGESTION USE CASES
![Page 47: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/47.jpg)
TWITTER TO HDFS
![Page 48: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/48.jpg)
DATA SCIENCE
![Page 49: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/49.jpg)
BATCH TRAINING PREDICTIVE MODELS
![Page 50: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/50.jpg)
CONNECTED
CAR
![Page 51: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/51.jpg)
transformer http filter hdfs
type-transformer
python
Gemfire
REST
Hadoop
Gemfire
spark
![Page 52: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/52.jpg)
STREAMING APROXIMIATIONS EXIST ≈
![Page 53: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/53.jpg)
BREADTH AND ACCURACY DON’T MATCH BATCH ≠
![Page 54: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/54.jpg)
WORKFLOW ORCHESTRATION
![Page 55: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/55.jpg)
![Page 56: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/56.jpg)
forEachDir
start
getDirectoryInfo
join
dirSubProcss dirSubProcss …
![Page 57: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/57.jpg)
Condition 1
start
getDirAgeAndNumberOfFiles
Ingest
Condition 2
sendReminderEmail
Archive End
![Page 58: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/58.jpg)
DEMO
![Page 59: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/59.jpg)
HOW IS THIS DONE VIA STREAMING?
![Page 60: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/60.jpg)
NON-INTERACTIVE PROCESSING
![Page 61: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/61.jpg)
SAME AS ETL PROCESSING
![Page 62: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/62.jpg)
PORT SCANNER
![Page 63: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/63.jpg)
DEMO
![Page 64: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/64.jpg)
STREAMING DOESN’T MAKE SENSE
![Page 65: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/65.jpg)
STREAMING DOES MAKE SENSE
![Page 66: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/66.jpg)
WHERE LATENCY IS A PRIORITY
![Page 67: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/67.jpg)
DATA LOSS IS ACCEPTABLE
![Page 68: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/68.jpg)
UNBOUNDED DATASET
![Page 69: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/69.jpg)
BUT DOES NOT IN OTHER USE CASES
![Page 70: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/70.jpg)
HIGHLY COMPLEX
![Page 71: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/71.jpg)
ERROR HANDLING NOT AS ROBUST
![Page 72: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/72.jpg)
DATA GUARANTEES NOT AS ROBUST
![Page 73: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/73.jpg)
WHERE BATCH IS BETTER
![Page 74: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/74.jpg)
F IN ITE DATASET
![Page 75: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/75.jpg)
ROBUST ERROR HANDLING
![Page 76: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/76.jpg)
DATA GUARANTEES BATTLE TESTED
![Page 77: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/77.jpg)
BATCH LOGO SLIDE?
![Page 78: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/78.jpg)
RESOURCES ARE A CONCERN
![Page 79: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/79.jpg)
NOT EITHER OR
![Page 80: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/80.jpg)
COMBINING THE TWO PROVIDES THE MOST POWER
![Page 81: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/81.jpg)
E.T.L.v2
![Page 82: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/82.jpg)
mail http mysql2hdfs
Database to HDFS v2
![Page 83: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/83.jpg)
DEMO
![Page 84: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/84.jpg)
CONSISTENT PROGRAMMING MODEL
![Page 85: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/85.jpg)
SHARE CODE BETWEEN PARADIGMS
![Page 86: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/86.jpg)
![Page 87: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/87.jpg)
BATCH HAS LASTED BECAUSE IT MAKSE SENSE
![Page 88: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/88.jpg)
SPRING XD/DATA FLOW BEST OF BOTH WORLDS
![Page 89: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/89.jpg)
ENTERPRISE GRADE STREAM PROCESSING
![Page 90: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/90.jpg)
BEST BATCH OPTION ON THE JVM
![Page 91: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/91.jpg)
![Page 92: Who Needs Batch?](https://reader031.vdocuments.net/reader031/viewer/2022031518/58ecda771a28ab87618b45e5/html5/thumbnails/92.jpg)
Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 92
Check out Spring Cloud Data Flow on https://spring.io
Apache Spark for Big Data Processing – Salon N-P Up next! High Performance Stream Processing – Salon N-P 8:30AM Tomorrow
Spring Integration Extensions Ecosystem – Here! 12:45 Tomorrow
Learn More. Stay Connected.
@springcentral Spring.io/video