ocean data interoperability platform - big data - streams & workflows
TRANSCRIPT
ODIP II – Data Streams and FlowsSeptember 2015
Adam Leadbetter, Damian Smyth, Rob [email protected]
http://www.confluent.io/blog/apache-kafka-samza-and-the-unix-philosophy-of-distributed-data
20th Century 21st Century
Single machine Distributed processing
One-to-one communication Publish-subscribe pattern
No fault tolerance Replication, auto-recovery
Fixed schema, encoding Schema management, evolvable encoding
20th Century 21st CenturyByte stream Messages
In-memory Durable
Blocking / backpressure Buffering
One-to-one Multi-subscriber
https://github.com/peterataylor/om-json
Further Work
• Look at the architectures for reprocessing historic data
• Incorporate Sensor Web Enablement, OM-JSON
• Deploy on vessels / mobile, remote platforms
• Investigation of Apache NiFi