stream processing with dds and cep
DESCRIPTION
This presentations explains the foundations of Stream Processing and shows how elegant Stream Processing Architectures can be built by using in synergy DDS and CEP.TRANSCRIPT
Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech [email protected]!
海 賊 天 使
Defining "Stream Processing
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Stream Processing ¨ Stream Processing can be seen as an architectural style
for building systems that operate over continuous (theoretically infinite) streams of data
¨ Stream Processing is often reified under one of its many declinations, such as: ¨ Reactive Systems ¨ Signal Processing Systems ¨ Functional Stream Programming ¨ Data Flow Systems
[1/3]
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Stream Processing ¨ Stream Processing Architectures are very natural for
modeling systems needing to react to streams of data produced by the external world, such as the data produced by sensors, a camera or even the data produced by the stock exchange.
¨ Stream Processing Systems usually operate in real-time over streams and generate in turns other streams of data providing information on what is happening or suggesting actions to perform, such as by stock X, raise alarm Y, or detected spatial violation, etc.
[2/3]
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Stream Processing ¨ Stream Processing Systems are
typically modeled as collection of modules communicating via typed data channels
¨ Modules usually play one of the following roles: ¨ Sources: Injecting data into the System
¨ Filters/Actors: Performing some computation over sources
¨ Sinks: Consuming the data produced by the system
[3/3]
Filter
Filter
Filter
source
Sink Channel
Implementing"Stream Processing
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Stream Processing Alternatives ¨ Different possibilities exist to
implement channels as well s filters
¨ DDS is a very good fit for channels due to its strong typing as well as performance and scalability
¨ Filter can be implemented by custom business logic or by leveraging CEP
Filter
Filter
Filter
DDS Writer
DDS Reader DDS Topic
CEP | Custom
Stream Processing "with DDS
Data Distribution in a Nutshell
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Data Distribution in a Nutshell ¨ Data distribution is about
making application defined data available where needed and when needed, while preserving and end-to-end type contract
¨ Data is a first class concept, it can be Created, Read, Updated, and eventually Disposed (CRUD)
¨ The last value (or last N-values) of a Data is always available to applications
Id Temp Scale
101 25 C
202 78 F
303 299 K
DW
DW
DW
DR
DR
DR
struct TempSensor { ! long Id; ! float temp; ! float hum; !} !#pragma keylist TempSensor id!
Fully Distributed Global Data Space DW: DataWriter
DR: DataReader
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
DDS Topics ¨ A Topic defines the subject of
publications and subscriptions
¨ A Topic has associated a user defined type and QoS
¨ The Topic name, type and QoS have a well defined role in matching subscriptions
¨ Topics can be discovered or locally defined
Topic
Name
QoS Type
DURABILITY, DEADLINE, PRIORITY, …
[1/2] “org.opensplice.demo.TTempSensor”
struct TempSensor { ! long Id; ! float temp; ! float hum; !} !#pragma keylist TempSensor id!
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
DDS Topics
¨ DDS Topic types can have associated keys
¨ Each unique key-value identify a Topic Instance – a specific stream of values
[2/2]
Topic
Name
QoS Type
“org.opensplice.demo.TTempSensor”
DURABILITY, DEADLINE, PRIORITY, …
struct TempSensor { ! long Id; ! float temp; ! float hum; !} !#pragma keylist TempSensor id!
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Anatomy of a DDS Application
Domain
Reader/Writers User Defined for Types
Session
val dp = DomainParticipant(0) !
// Create a Publisher / Subscriber!Publisher p = dp.create_publisher(); !Subscriber s = dp.create_subscriber(); !// Create a Topic!Topic<TempSensor> t = ! dp.create_topic<TempSensor>(“com.myco.TSTopic”) !
Domain Participant
Publisher
DataWrter
Topic Subscriber
DataReader // Create a DataWriter/DataWriter!DataWriter<TempSensor> dw = pub.create_datawriter(t); !DataReader<TempSensor> dr = sub.create_datareader(t); !
Gives access to a DDS Domain
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Anatomy of a DDS Application
Domain
Reader/Writers User Defined for Types
Session
val dp = DomainParticipant(0) !
// Create a Publisher / Subscriber !val pub = Publisher(dp) !val sub = Subscriber(dp) !// Create a Topic!val topic = Topic[TempSensor](dp, ! “org.opensplice.demo.TTempSensor”) !
Domain Participant
Publisher
DataWrter
Topic Subscriber
DataReader // Create a DataWriter/DataWriter!DataWriter<TempSensor> dw = pub.create_datawriter(t); !DataReader<TempSensor> dr = sub.create_datareader(t); !
Pub/Sub Abstractions
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Anatomy of a DDS Application
Domain
Reader/Writers for User Defined for Types
Session
Domain Participant
Publisher
DataWrter
Topic Subscriber
DataReader // Create a DataWriter/DataWriter!val writer = DataWriter[TempSensor](pub, topic) !val reader = DataReader[TempSensor](sub, topic) !!// Write data!val t = new TempSensor ts(101, 25, 40) !writer write ts; !
Reader/Writer for application defined Topic Types
// Create a Publisher / Subscriber !val pub = Publisher(dp) !val sub = Subscriber(dp) !// Create a Topic!val topic = Topic[TempSensor](dp, ! “org.opensplice.demo.TTempSensor”) !
val dp = DomainParticipant(0) !
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Domain & Partitions
Topic Instances/Samples
Domain (e.g. Domain 123)
Partition (e.g. Partition “Telemetry”)
Domain Participant
Topic
Publisher
DataWrter
Subscriber
DataReader
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Stream Processing in DDS ¨ DDS naturally supports stream processing through three
main concepts: ¨ Topics
¨ Data Writers
¨ Data Readers
¨ In addition DDS’ dynamic discovery, high availability and high performance make it very easy to architect, deploy and upgrade stream processing systems
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
DDS Topics as Streams ¨ In Stream Processing,
Streams are typed data channel
¨ A DDS Topic provide an elegant way of defining the Streams that make up a Streaming System while capturing their type, as well as their Non-Functional Constraints
Filter
Filter
Filter
DDS Topic
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
DDS Data Writers as Sources
¨ DDS Data Writers make ideal sources for Stream Processing systems due to their: ¨ Dynamic Discovery
¨ Durability
¨ Fault-Tolerance
Filter
Filter
Filter
DDS Writer
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
DDS Data Readers as Sinks
¨ DDS Data Readers make ideal sinks for Stream Processing systems due to their: ¨ Dynamic Discovery
¨ Durability
Filter
Filter
Filter
DDS Reader
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
DDS & Filters ¨ Filters/Actors perform
transformations from one or more input streams to one or more output streams
¨ The combination of DDS ContentFilteredTopics with a powerful language like Scala can make it relatively easy to build custom filters
¨ Let’s explore this path… DDS+Application Logic
Filter
Filter
Filter
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Content Filtered Topics ¨ Content Filtered Topics
provide a way of defining a local projection of the stream of data associated with a given topic
¨ Filters are expressed as the “WHERE” clause of an SQL statement
¨ Filters can operate on any attribute of the type associated with the topic
Example:
// Create a Topic (on default domain)!val topic = Topic[TempSensor](“TTempSensor”) !val ftopic = ! ContentFilteredTopic[TempSensor](“CFTempSensor”, ! topic, ! filter, ! params) !!// - filter is a WHERE-like clause, such as: !// “temp > 20 AND hum > 50” !// “temp > %0” !// “temp > hum” !// “temp BETWEEN (%0 AND %1)!// !// - params is the list of parameters to pass to the !// filter expression – if any!
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Filter Expression Syntax
¨ DDS Filters are condition over a topic type attributes
¨ Temporal properties or causality cannot be captured via DDS filter expression
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
History
¨ DDS provides a way of controlling data windows through the History QoS
The window keeps the last n data samples
Data older than “n samples ago” get’s out the window
past future
now
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
[Putting it All Together] "
TempSensor Moving Average object MovingAverageFilter { ! def main(args: Array[String]) { ! if (args.length < 2) { ! println("USAGE:\n\tMovingAverageFilter <window> <filter-expression>") ! } !! val topic = Topic[TempSensor]("TTempSensor") ! val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1)) !! val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt) ! val reader = DataReader[TempSensor](ftopic, rqos) !! reader.reactions += { ! case e: DataAvailable[TempSensor] => { ! var average: Float = 0 ! val window = e.reader.history! window foreach (average += _.temp) ! average = average / window.length! println("+--------------------------------------------------------") ! println("Moving Average: " + average) ! } ! } ! } !} !
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Key Points So Far
¨ DDS provides some event processing capabilities that facilitate the development of Stream Processing Filters
¨ Higher Level programming languages like Scala can make it very easy to concisely express complex filters
¨ Note: Scala targets the JVM as well as the .Net CLR
Stream Processing "with DDS & CEP
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
CEP In Brief ¨ Complex Event Processing (CEP)
engines provide a declarative way of transforming a set of input streams into one or more output streams
¨ The declarative language provided by CEP is usually based on some extension of SQL to include time, pattern matching and causality
- select * from TempSensor(temp < 30) - select avg(temp) from TempSensor.win:time(1 sec)
CEP
[1/2]
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
CEP In Brief ¨ CEP emerged from Stream
Processing as a way to facilitate the development of “Filters/Actors” and make it “declarative”
¨ In line with the Stream Processing principles, CEP operate on typed data streams
- select * from TempSensor(temp < 30) - select avg(temp) from TempSensor.win:time(1 sec)
CEP
[1/2]
Filter
Filter
Filter
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
DDS + CEP = Stream Processing ¨ DDS provides a powerful Stream
abstraction that is: ¨ Type Safe
¨ High Performance
¨ Highly Available
¨ Decoupled in Time/Space
¨ Dynamically Discoverable
¨ CEP provide a powerful abstraction for processing streams
¨ The combination of DDS + CEP is not only natural but a perfect fit for building high performance, highly available Stream Processing Systems
CEP
CEP
CEP
DDS Writer
DDS Reader DDS Topic
DDS + CEP in Action
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Esper CEP
¨ Esper is an Open Source, pure Java, CEP engine widely used in a wide variety of applications ranging from the Capital Market to Defense and Aerospace ¨ http://www.espertech.com
¨ Esper Rules are expressed EPL (Esper Processing Language) which can be seen as an extension of SQL with support for time, causality and pattern matching
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
OpenSplice + Esper
¨ Esper Provides an EsperIO framework for plugging-in new stream transports
¨ Plugging OpenSplice into Esper is trivial even w/o relying on the EsperIO framework
¨ Let’s have a look…
Demo Application
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
iShapes Application ¨ To explore play with OpenSplice
and Esper, we’ll use the simd-cxx ishapes application
¨ Three Topics ¨ Circle, Square, Triangle
¨ One Type: struct ShapeType { ! string color; ! long x; ! long y; ! long shapesize; !}; !#pragma keylist Shapetype color !
Spotted shapes represent subscriptions
Pierced shapes represent publications
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Esper Setup
Step 1: Register Topic Types
val config = new Configuration !val ddsConf = new ConfigurationEventTypeLegacy!!ddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC) !!config.addEventType("ShapeType", ! classOf[org.opensplice.demo.ShapeType].getName, ! ddsConf) !!val cep: EPServiceProvider = ! EPServiceProviderManager.getDefaultProvider(config) !
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Esper Setup
Step 2: Register a Listener for receiving Esper Events
val listener = new UpdateListener { ! def update(ne: Array[EventBean], oe: Array[EventBean]) { ! ne foreach(e => { !
"// Handle the event ! }) ! } ! } !
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Esper Setup
Step 3: Hook-up DDS to Esper
reader.reactions += { ! case e: DataAvailable[ShapeType] => { ! (e.reader read) foreach(runtime sendEvent _) !! } !} !
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
iShapes FrameRate
¨ Let’s suppose that we wanted to keep under control the iShapes Frame rate for ech given color
¨ In Esper this can be achieved with the following expression:
insert into ShapesxSec !select color, count(*) as cnt !from ShapeType.win:time_batch(1 second) !group by color !
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
iShapes Center of Mass
¨ Suppose that we wanted to compute the center of mass of all the shapes currently displayed over the last second
¨ The Esper expression for this would be:
select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg(y),int), shapesize) as NewShape !from ShapeType.win:time(10 sec) !
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
¥ Fastest growing JVM Language ¥ Open Source ¥ www.scala-lang.org
References
OpenSplice DDS¥ #1 OMG DDS Implementation ¥ Open Source ¥ www.opensplice.org
¥ Scala API for OpenSplice DDS ¥ Open Source ¥ code.google.com/p/escalier
¥ #1 Java-Based CEP Engine ¥ Open Source ¥ www.espertech.com
Summing Up
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
Concluding Remarks
¨ DDS provides a very good foundation to build high performance, highly available and scalable streaming systems
¨ DDS + CEP are the most natural, effective and efficient way of building next generation Stream Processing Systems!
Copyrig
ht 2010, PrismTech – A
ll Rights Reserved.
http://www.opensplice.com/
http://www.opensplice.org/
emailto:[email protected]
http://www.youtube.com/OpenSpliceTube http://opensplice.blogspot.com
http://bit.ly/1Sreg
http://www.slideshare.net/angelo.corsaro
http://twitter.com/acorsaro/
D e l i v e r i n g P e r f o r m a n c e , O p e n n e s s , a n d F r e e d o m
OpenSplice DDS