uwe seiler, data architect and trainer at codecentric ag - "hadoop & germany &...
TRANSCRIPT
![Page 1: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/1.jpg)
Hadoop & Germany & 2016
uweseiler
![Page 2: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/2.jpg)
/whoami &
/disclaimer
Hadoop & Germany & 2016
![Page 3: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/3.jpg)
We finally stopped talking infrastructure!
Hadoop & Germany & 2016
![Page 4: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/4.jpg)
We now talk architectures and use cases!
Hadoop & Germany & 2016
![Page 5: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/5.jpg)
#1 The Big Data Lake is an illusion!
Hadoop & Germany & 2016
![Page 6: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/6.jpg)
Da
ta S
ourc
esD
ata
Sys
tem
sA
pp
lica
tion
s
Traditional Sources
RDBMS OLTP OLAP …
Traditional Systems
RDBMS EDW MPP …
Business Intelligence
BusinessApplications
Custom Applications
Operation
Manage &
Monitor
Dev Tools
Build &
Test
New Sources
Logs Mails Sensor …SocialMedia
EnterpriseHadoop Plattform
#1 The Vision of the Big Data Lake
![Page 7: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/7.jpg)
Hadoop is not the one tool to rule them all
#1 Vision & Reality
![Page 8: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/8.jpg)
Embrace heterogeneity! (and learn to deal with the complexity)
#1 After the reality shock…
![Page 9: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/9.jpg)
#1 Real world architecture - Insurance
Da
ta S
ourc
esD
ata
Sys
tem
sA
pp
lica
tion
s
Traditional Sources
RDBMS OLTP OLAP …
Traditional Systems
DWH
BusinessIntelligence
New Sources
Logs Sensor …SocialMedia
Enterprise Hadoop Plattform
SAS LASR Server
Apache Zeppelin
![Page 10: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/10.jpg)
#2 Speed is the new king!
Hadoop & Germany & 2016
![Page 11: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/11.jpg)
#2 The “classic“ Lambda Architecture
![Page 12: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/12.jpg)
Batch Layer
Speed LayerData Ingestion
Data Processing
Data Storage
Data Storage Data Analysis
Visualization
Visualization
…
DataChannels
ms - s
min - h
#2 Lambda in Action - (e)Commerce
![Page 13: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/13.jpg)
SMACK Spark Mesos Akka
Cassandra Kafka
#2 The lust for speed
![Page 14: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/14.jpg)
Data Ingestion
Data Processing
Raw Data
#2 Cassandra & Hadoop - AdServing
Data Processing
User Journey
Aggregated Data
Web Frontend
Aggregated Data< 120 days
Data Science
![Page 15: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/15.jpg)
#3 Data Science to the help!
Hadoop & Germany & 2016
![Page 16: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/16.jpg)
Hadoop is about to become commodity
#3 Let’s face it..
![Page 17: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/17.jpg)
Algorithms will be the new differentiator
#3 We need new challenges…
![Page 18: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/18.jpg)
Batch Layer
Speed LayerData Ingestion
Stream Processing
ms - s
min - h
#3 Fraud detection - Financial services
DataImport
Data Preparation
Model Generation
Model Validation
Feature & Parameter Selection
Manual or automatic Iterations to tune
parameters
Use Model
Refresh Model from latest input data
![Page 19: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/19.jpg)
Every major company is building teams of unicorns
#3 The solution?
![Page 20: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/20.jpg)
#4 Hadoop for good!
Hadoop & Germany & 2016
![Page 21: Uwe Seiler, Data Architect and Trainer at codecentric AG - "Hadoop & Germany & 2016"](https://reader033.vdocuments.net/reader033/viewer/2022042907/58713b011a28abf0568b6cfb/html5/thumbnails/21.jpg)
Hadoop User Group Rhein-Mainhttp://www.meetup.com/de-DE/HUG-Rhein-Main/
Next Meetup: 23.06.2016, Talks welcome