open splicedds espercep-webinar

Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech [email protected]!

海賊天使

Defining "Stream Processing

Copyrig

ht 2010, PrismTech – A

ll Rights Reserved.

Stream Processing ¨  Stream Processing can be seen as an architectural style

for building systems that operate over continuous (theoretically infinite) streams of data

¨  Stream Processing is often reified under one of its many declinations, such as: ¨  Reactive Systems ¨  Signal Processing Systems ¨  Functional Stream Programming ¨  Data Flow Systems

[1/3]

Copyrig


ll Rights Reserved.

Stream Processing ¨  Stream Processing Architectures are very natural for

modeling systems needing to react to streams of data produced by the external world, such as the data produced by sensors, a camera or even the data produced by the stock exchange.

¨  Stream Processing Systems usually operate in real-time over streams and generate in turns other streams of data providing information on what is happening or suggesting actions to perform, such as by stock X, raise alarm Y, or detected spatial violation, etc.

[2/3]

Copyrig


ll Rights Reserved.

Stream Processing ¨  Stream Processing Systems are

typically modeled as collection of modules communicating via typed data channels

¨  Modules usually play one of the following roles: ¨  Sources: Injecting data into the System

¨  Filters/Actors: Performing some computation over sources

¨  Sinks: Consuming the data produced by the system

[3/3]

Filter

Filter

Filter

source

Sink Channel

Implementing"Stream Processing

Copyrig


ll Rights Reserved.

Stream Processing Alternatives ¨  Different possibilities exist to

implement channels as well s filters

¨  DDS is a very good fit for channels due to its strong typing as well as performance and scalability

¨  Filter can be implemented by custom business logic or by leveraging CEP

Filter

Filter

Filter

DDS Writer

DDS Reader DDS Topic

CEP | Custom

Stream Processing "with DDS

Data Distribution in a Nutshell

Copyrig


ll Rights Reserved.

Data Distribution in a Nutshell ¨  Data distribution is about

making application defined data available where needed and when needed, while preserving and end-to-end type contract

¨  Data is a first class concept, it can be Created, Read, Updated, and eventually Disposed (CRUD)

¨  The last value (or last N-values) of a Data is always available to applications

Id Temp Scale

101 25 C

202 78 F

303 299 K

DW

DW

DW

DR

DR

DR

struct TempSensor { ! long Id; ! float temp; ! float hum; !} !#pragma keylist TempSensor id!

Fully Distributed Global Data Space DW: DataWriter

DR: DataReader

Copyrig


ll Rights Reserved.

DDS Topics ¨  A Topic defines the subject of

publications and subscriptions

¨  A Topic has associated a user defined type and QoS

¨  The Topic name, type and QoS have a well defined role in matching subscriptions

¨  Topics can be discovered or locally defined

Topic

Name

QoS Type

DURABILITY, DEADLINE, PRIORITY, …

[1/2] “org.opensplice.demo.TTempSensor”


Copyrig


ll Rights Reserved.

DDS Topics

¨  DDS Topic types can have associated keys

¨  Each unique key-value identify a Topic Instance – a specific stream of values

[2/2]

Topic

Name

QoS Type

“org.opensplice.demo.TTempSensor”

DURABILITY, DEADLINE, PRIORITY, …


Copyrig


ll Rights Reserved.

Anatomy of a DDS Application

Domain

Reader/Writers User Defined for Types

Session

val dp = DomainParticipant(0) !

// Create a Publisher / Subscriber!Publisher p = dp.create_publisher(); !Subscriber s = dp.create_subscriber(); !// Create a Topic!Topic<TempSensor> t = ! dp.create_topic<TempSensor>(“com.myco.TSTopic”) !

Domain Participant

Publisher

DataWrter

Topic Subscriber

DataReader // Create a DataWriter/DataWriter!DataWriter<TempSensor> dw = pub.create_datawriter(t); !DataReader<TempSensor> dr = sub.create_datareader(t); !

Gives access to a DDS Domain

Copyrig


ll Rights Reserved.


Domain

Reader/Writers User Defined for Types

Session


// Create a Publisher / Subscriber !val pub = Publisher(dp) !val sub = Subscriber(dp) !// Create a Topic!val topic = Topic[TempSensor](dp, ! “org.opensplice.demo.TTempSensor”) !

Domain Participant

Publisher

DataWrter

Topic Subscriber

DataReader // Create a DataWriter/DataWriter!DataWriter<TempSensor> dw = pub.create_datawriter(t); !DataReader<TempSensor> dr = sub.create_datareader(t); !

Pub/Sub Abstractions

Copyrig


ll Rights Reserved.


Domain

Reader/Writers for User Defined for Types

Session

Domain Participant

Publisher

DataWrter

Topic Subscriber

DataReader // Create a DataWriter/DataWriter!val writer = DataWriter[TempSensor](pub, topic) !val reader = DataReader[TempSensor](sub, topic) !!// Write data!val t = new TempSensor ts(101, 25, 40) !writer write ts; !

Reader/Writer for application defined Topic Types

// Create a Publisher / Subscriber !val pub = Publisher(dp) !val sub = Subscriber(dp) !// Create a Topic!val topic = Topic[TempSensor](dp, ! “org.opensplice.demo.TTempSensor”) !


Copyrig


ll Rights Reserved.

Domain & Partitions

Topic Instances/Samples

Domain (e.g. Domain 123)

Partition (e.g. Partition “Telemetry”)

Domain Participant

Topic

Publisher

DataWrter

Subscriber

DataReader

Copyrig


ll Rights Reserved.

Stream Processing in DDS ¨  DDS naturally supports stream processing through three

main concepts: ¨  Topics

¨  Data Writers

¨  Data Readers

¨  In addition DDS’ dynamic discovery, high availability and high performance make it very easy to architect, deploy and upgrade stream processing systems

Copyrig


ll Rights Reserved.

DDS Topics as Streams ¨  In Stream Processing,

Streams are typed data channel

¨  A DDS Topic provide an elegant way of defining the Streams that make up a Streaming System while capturing their type, as well as their Non-Functional Constraints

Filter

Filter

Filter

DDS Topic

Copyrig


ll Rights Reserved.

DDS Data Writers as Sources

¨  DDS Data Writers make ideal sources for Stream Processing systems due to their: ¨  Dynamic Discovery

¨  Durability

¨  Fault-Tolerance

Filter

Filter

Filter

DDS Writer

Copyrig


ll Rights Reserved.

DDS Data Readers as Sinks

¨  DDS Data Readers make ideal sinks for Stream Processing systems due to their: ¨  Dynamic Discovery

¨  Durability

Filter

Filter

Filter

DDS Reader

Copyrig


ll Rights Reserved.

DDS & Filters ¨  Filters/Actors perform

transformations from one or more input streams to one or more output streams

¨  The combination of DDS ContentFilteredTopics with a powerful language like Scala can make it relatively easy to build custom filters

¨  Let’s explore this path… DDS+Application Logic

Filter

Filter

Filter

Copyrig


ll Rights Reserved.

Content Filtered Topics ¨  Content Filtered Topics

provide a way of defining a local projection of the stream of data associated with a given topic

¨  Filters are expressed as the “WHERE” clause of an SQL statement

¨  Filters can operate on any attribute of the type associated with the topic

Example:

// Create a Topic (on default domain)!val topic = Topic[TempSensor](“TTempSensor”) !val ftopic = ! ContentFilteredTopic[TempSensor](“CFTempSensor”, ! topic, ! filter, ! params) !!// - filter is a WHERE-like clause, such as: !// “temp > 20 AND hum > 50” !// “temp > %0” !// “temp > hum” !// “temp BETWEEN (%0 AND %1)!// !// - params is the list of parameters to pass to the !// filter expression – if any!

Copyrig


ll Rights Reserved.

Filter Expression Syntax

¨  DDS Filters are condition over a topic type attributes

¨  Temporal properties or causality cannot be captured via DDS filter expression

Copyrig


ll Rights Reserved.

History

¨  DDS provides a way of controlling data windows through the History QoS

The window keeps the last n data samples

Data older than “n samples ago” get’s out the window

past future

now

Copyrig


ll Rights Reserved.

[Putting it All Together] "

TempSensor Moving Average object MovingAverageFilter { ! def main(args: Array[String]) { ! if (args.length < 2) { ! println("USAGE:\n\tMovingAverageFilter <window> <filter-expression>") ! } !! val topic = Topic[TempSensor]("TTempSensor") ! val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1)) !! val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt) ! val reader = DataReader[TempSensor](ftopic, rqos) !! reader.reactions += { ! case e: DataAvailable[TempSensor] => { ! var average: Float = 0 ! val window = e.reader.history! window foreach (average += _.temp) ! average = average / window.length! println("+--------------------------------------------------------") ! println("Moving Average: " + average) ! } ! } ! } !} !

Copyrig


ll Rights Reserved.

Key Points So Far

¨  DDS provides some event processing capabilities that facilitate the development of Stream Processing Filters

¨  Higher Level programming languages like Scala can make it very easy to concisely express complex filters

¨  Note: Scala targets the JVM as well as the .Net CLR

Stream Processing "with DDS & CEP

Copyrig


ll Rights Reserved.

CEP In Brief ¨  Complex Event Processing (CEP)

engines provide a declarative way of transforming a set of input streams into one or more output streams

¨  The declarative language provided by CEP is usually based on some extension of SQL to include time, pattern matching and causality

-  select * from TempSensor(temp < 30) -  select avg(temp) from TempSensor.win:time(1 sec)

CEP

[1/2]

Copyrig


ll Rights Reserved.

CEP In Brief ¨  CEP emerged from Stream

Processing as a way to facilitate the development of “Filters/Actors” and make it “declarative”

¨  In line with the Stream Processing principles, CEP operate on typed data streams

-  select * from TempSensor(temp < 30) -  select avg(temp) from TempSensor.win:time(1 sec)

CEP

[1/2]

Filter

Filter

Filter

Copyrig


ll Rights Reserved.

DDS + CEP = Stream Processing ¨  DDS provides a powerful Stream

abstraction that is: ¨  Type Safe

¨  High Performance

¨  Highly Available

¨  Decoupled in Time/Space

¨  Dynamically Discoverable

¨  CEP provide a powerful abstraction for processing streams

¨  The combination of DDS + CEP is not only natural but a perfect fit for building high performance, highly available Stream Processing Systems

CEP

CEP

CEP

DDS Writer

DDS Reader DDS Topic

DDS + CEP in Action

Copyrig


ll Rights Reserved.

Esper CEP

¨  Esper is an Open Source, pure Java, CEP engine widely used in a wide variety of applications ranging from the Capital Market to Defense and Aerospace ¨  http://www.espertech.com

¨  Esper Rules are expressed EPL (Esper Processing Language) which can be seen as an extension of SQL with support for time, causality and pattern matching

Copyrig


ll Rights Reserved.

OpenSplice + Esper

¨  Esper Provides an EsperIO framework for plugging-in new stream transports

¨  Plugging OpenSplice into Esper is trivial even w/o relying on the EsperIO framework

¨  Let’s have a look…

Demo Application

Copyrig


ll Rights Reserved.

iShapes Application ¨  To explore play with OpenSplice

and Esper, we’ll use the simd-cxx ishapes application

¨  Three Topics ¨  Circle, Square, Triangle

¨  One Type: struct ShapeType { ! string color; ! long x; ! long y; ! long shapesize; !}; !#pragma keylist Shapetype color !

Spotted shapes represent subscriptions

Pierced shapes represent publications

Copyrig


ll Rights Reserved.

Esper Setup

Step 1: Register Topic Types

val config = new Configuration !val ddsConf = new ConfigurationEventTypeLegacy!!ddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC) !!config.addEventType("ShapeType", ! classOf[org.opensplice.demo.ShapeType].getName, ! ddsConf) !!val cep: EPServiceProvider = ! EPServiceProviderManager.getDefaultProvider(config) !

Copyrig


ll Rights Reserved.

Esper Setup

Step 2: Register a Listener for receiving Esper Events

val listener = new UpdateListener { ! def update(ne: Array[EventBean], oe: Array[EventBean]) { ! ne foreach(e => { !

"// Handle the event ! }) ! } ! } !

Copyrig


ll Rights Reserved.

Esper Setup

Step 3: Hook-up DDS to Esper

reader.reactions += { ! case e: DataAvailable[ShapeType] => { ! (e.reader read) foreach(runtime sendEvent _) !! } !} !

Copyrig


ll Rights Reserved.

iShapes FrameRate

¨  Let’s suppose that we wanted to keep under control the iShapes Frame rate for ech given color

¨  In Esper this can be achieved with the following expression:

insert into ShapesxSec !select color, count(*) as cnt !from ShapeType.win:time_batch(1 second) !group by color !

Copyrig


ll Rights Reserved.

iShapes Center of Mass

¨  Suppose that we wanted to compute the center of mass of all the shapes currently displayed over the last second

¨  The Esper expression for this would be:

select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg(y),int), shapesize) as NewShape !from ShapeType.win:time(10 sec) !

Copyrig


ll Rights Reserved.

¥  Fastest growing JVM Language ¥ Open Source ¥ www.scala-lang.org

References

OpenSplice DDS¥  #1 OMG DDS Implementation ¥ Open Source ¥ www.opensplice.org

¥  Scala API for OpenSplice DDS ¥ Open Source ¥  code.google.com/p/escalier

¥  #1 Java-Based CEP Engine ¥ Open Source ¥ www.espertech.com

Summing Up

Copyrig


ll Rights Reserved.

Concluding Remarks

¨  DDS provides a very good foundation to build high performance, highly available and scalable streaming systems

¨  DDS + CEP are the most natural, effective and efficient way of building next generation Stream Processing Systems!

Copyrig


ll Rights Reserved.

http://www.opensplice.com/

http://www.opensplice.org/

emailto:[email protected]

http://www.youtube.com/OpenSpliceTube http://opensplice.blogspot.com

http://bit.ly/1Sreg

http://www.slideshare.net/angelo.corsaro

http://twitter.com/acorsaro/

D e l i v e r i n g P e r f o r m a n c e , O p e n n e s s , a n d F r e e d o m

OpenSplice DDS

open splicedds espercep-webinar

Technology

val topic

defined topic

dds topics dds topic

dds domain

topic instance

data available

dening stream processing

injecting data