Download - IoT and BigData
![Page 1: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/1.jpg)
![Page 2: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/2.jpg)
WhoAmI
Entrepreneurial geek
Micro-clusters for BigData enthousiasts
BigData Services
Belgian Community
![Page 3: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/3.jpg)
Agenda
Part 1 : BigData
Part 2 : IoT and BigData
Part 3 : Discussions
![Page 4: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/4.jpg)
BigDataPart 1
![Page 5: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/5.jpg)
The Buzz
Started by Google● GFS Paper (2003)● MapReduce Paper (2004)
Storing and processing “Big” Data~ multiple petabytes
![Page 6: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/6.jpg)
The Need
Large amount of data
IOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIIIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIIIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIOIIOIOIOIIIOIOOOOIOIOOIIOIIIIIOIIOIIOIOIOIOIOIOIIOIIOIOIOIIIOIOOOOIOIIOIOIIOIIOIOIOIIIOIOOOOI
![Page 7: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/7.jpg)
The Need
High Speed at which data becomes available IOII OO
IOIIOIII
OIII
IOII OOIOII
OIIIOIII
IOII OOIOII
OIIIOIII
![Page 8: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/8.jpg)
The Need
The Different ways data is structured
IOIIIOII
IOII
IOIIIOIIIOII
IOII
IOII
IOII
IOIIIOII
IOIIIOIIIOIIIOII
IOII
IOIIIOII
IOII
IOII IOII
IOII
IOII
IOIIIOII
IOIIIOIIIOII IOII
![Page 9: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/9.jpg)
Technologies
![Page 10: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/10.jpg)
Technologies
Batch Processing:
Realtime Processing:
STORM
![Page 11: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/11.jpg)
Hadoop
Platform for Batch Processing
Yahoo!Doug CuttingMike Cafarella
Fetching and processing webpages
![Page 12: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/12.jpg)
Storage: HDFS
![Page 13: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/13.jpg)
Processing: YARN
![Page 14: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/14.jpg)
Processing: MapReduce
![Page 15: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/15.jpg)
Apache Storm
Platform for Realtime ProcessingLooks like Complex Event Processing (CEP)
BacktypeNathan MarzAcquired by Twitter
Like “ever-running” MapReduce jobs
![Page 16: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/16.jpg)
Apache Storm
![Page 17: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/17.jpg)
Apache Storm
![Page 18: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/18.jpg)
Apache Storm
![Page 19: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/19.jpg)
Apache Storm
![Page 20: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/20.jpg)
Apache Storm
![Page 21: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/21.jpg)
Myths
“BigData is all about storing data”
“BigData is only for big companies having lots of data”
“There are no BigData use cases in Belgium”
“BigData is only being used for analytics”
![Page 22: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/22.jpg)
Dangers
People are hard to find
Technologies are “different”
Too much technologies
You need hardware
Incorrect information
![Page 23: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/23.jpg)
IoT & BigDataPart 2
![Page 24: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/24.jpg)
Iot as datasource
Use data from “Things”
As raw as possibleNo matter which format
Store everything
![Page 25: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/25.jpg)
Iot as datasource
![Page 26: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/26.jpg)
Iot as datasource
Adapt the processing flow based on the data flowing through it:
- Disable faulty sensors- Take processing shortcuts
![Page 27: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/27.jpg)
Iot as datasource
Control other “Things” based on the data flowing through:
- Rain predicted => disable sprinklers
- Check-out at work, trip home will take 60 min => start heating in 30 min
Would be great to link with a rule-engine (Drools e.g.)
![Page 28: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/28.jpg)
Iot as datasource
Store data for more complex processing:- Pattern or behaviour recognition- Forecasting- Prevention
![Page 29: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/29.jpg)
Iot as datasource
Storage:
Every event is a recordEvery event is immutableEvery event has:
a sourcea timestampa typea value
![Page 30: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/30.jpg)
Iot as datasource
![Page 31: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/31.jpg)
Iot as Platform - Idea
Quantity > Quality
One platform toStoreProcess
Nodes can beAnything with a controllerAnything that can store data
![Page 32: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/32.jpg)
Iot as Platform - Risks
Complex Resource ManagementExploit Capabilities
Complex Data ManagementFragmented Data
Complex Security ModelObfuscate Data
![Page 33: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/33.jpg)
Iot as Platform - Gains
Keep your friends close, but your data closer.
Delegate processing
Use resources in the best way possible
![Page 34: IoT and BigData](https://reader034.vdocuments.net/reader034/viewer/2022051400/54c6ae704a795907748b457e/html5/thumbnails/34.jpg)
What do you think?
Part 3