big data and spark streaming. oil production sensors data monitoring
TRANSCRIPT
Big Data, Spark Streaming, Oil and Gas
Kyiv, Ukraine. 07 June 2016
Oil production sensors data monitoring
Yaroslav Nedashkovsky, System Architect
Shell data project
“Next year, BP will connect 650 wells to the Industrial Internet. If all goes according to plan, the companies will expand the scope to 4,000 BP subsea wells around the world”
Digital Oil Field
How we could handle such huge data flow ?
What kind of streaming technology could we use ?
Stream processing system- Apache Spark Streaming
- Apache Storm- Apache Samza- Azure Stream Analytics- Google Dataflow- Heron
- Apache Flink
What we need from streaming system?- scalable- fault-tolerant- low latency- data distribution- distributed computations- good API- “exactly-once” guarantees or maybe “at most once” or “at least once ” will be enough ?
(near real time, but not real time)
- high-level api (windows, joins, etc.)
- exactly-one semantics (?!), fault tolerant, scalable
- integration with SQL, DataFrames, Mllib, GraphX
How this work ?
Spark 2.0: Structured Streaming
Oil location data flow monitor
“Christmas tree”
IoT (MQTT) + Spark Streaming + Vizualization
Let look at “monitor” implementation and see how it works
