reactive stream processing using dds and rx
Post on 26-Jun-2015
457 Views
Preview:
DESCRIPTION
TRANSCRIPT
Your systems. Working as one.
Reactive Stream Processing
using DDS and Rx
www.rti.com
Sumant Tambe, Ph.D.
Senior Software Research Engineer and Microsoft MVP
Real-Time Innovations, Inc.
@sutambe
Oct. 11, 2014
Outline
• Reactive Systems
• Stream Processing
• Overview of Reactive Extensions
• Overview of DDS
• Streaming Shapes Demo in C#
10/10/2014 2Real-Time Innovations, Inc.
Systems Everyone Wants to Build
• Event-Driven: Modular, pipelined, asynchronous
• Elasic: Scales easily up/down with load and cpu cores.
• Resilient: fault-tolerant
• Responsive: Reacts to events at the speed of environment
10/10/2014 Real-Time Innovations, Inc. 3
Networking Middleware
App
Stream Processing
• Stream Processing is the term used to describe an architectural style that operate on a continuous sequence of data.
10/10/2014 Real-Time Innovations, Inc. 4
Shape of an application
o/p
Where Once CombineLatest Select Scan Merge Raw Data
i/p
*nix command line (pipes and filter)
$ ls -1 | grep “search” | grep “research”
research
$
10/10/2014 Real-Time Innovations, Inc. 6
$ cat - | grep “Real”
R
Real
Real
[Ctrl+C]
$
Reactive Extensions• The Reactive Extensions (Rx) is a
library for composing asynchronous and event-based programs using observable sequences
• Rx = Observables + Composition + Schedulers
• Streams are first-class
• Filter, project, aggregate, compose and perform time-based operations on multiple streams
• Uses Functional Programming Style
• Rx.NET, RxJava, RxJS, RxCpp, RxRuby, RxPyton, and more…
10/10/2014 Real-Time Innovations, Inc. 7
10/10/2014 Real-Time Innovations, Inc. 8
Data Distribution Service (DDS)
A Reactive Middleware
DDS: Standards-based Integration Infrastructure for Critical Applications
© 2009 Real-Time Innovations, Inc.
Streaming
DataSensors Events
Real-Time
Applications
Enterprise
ApplicationsActuators
Systems that interact with the Real World
• Must adapt to changing environment• Cannot stop processing the information• Live within world-imposed timing
Beyond traditional interpretation of real-time
© 2010 Real-Time Innovations, Inc.
Real-Time Data Distribution
10/10/2014 © 2012 RTI • COMPANY 11
RPC over
DDS
2014
DDS
Security
2012
Family of Specifications
© 2009 Real-Time Innovations, Inc. COMPANY 12
DDSImplementation
Network / TCP / UDP / IP
App
DDSImplementation
App
DDSImplementation
DDS Spec
2004
DDS
Interoperablity
2006
UML Profile
for DDS
2008
DDS for
Lw CCM
2009
DDS
X-Types
2010 2010
DDS-STD-C++
DDS-JAVA5
App
Evolution of DataBus
Data-centricity basics
Everyday Example: Schedule Meeting via Emails
Alternative Process #1 (message-centric):
1. Email: “Meeting Monday at 10:00.”
2. Email: “Here’s dial-in info for meeting…”
3. Email: “Meeting moved to Tuesday”
4. You: “Where do I have to be? When?”
5. You: (sifting through email messages…)
14
Everyday Example: Schedule Meeting Using a Calendar
Alternative Process #2:
1. Calendar: (add meeting Monday at 10:00)
2. Calendar: (add dial-in info)
3. Calendar: (move meeting to Tuesday)
4. You: “Where do I have to be? When?”
5. You: (check calendar. Contains consolidated-state)
15
The difference is state!
The infrastructure consolidates changes and maintains it
10/10/2014 © 2012 RTI • COMPANY 16
DDS Communication Model
DDS for Distribution, Rx for Processing
10/10/2014 Real-Time Innovations, Inc. 17
DR
DR
DR
Observable Observer
DW
DW
Processing
Rx4DDS.NET = DDS + Rx
10/10/2014 © 2012 RTI • COMPANY 18
• DDS wrapper for Rx.NET
• In C#
• Anything that produces data is an Observable
– Topics, Discovery, Statuses, statuses, etc.
DDS and Rx: A Great Match
10/10/2014 © 2012 RTI • COMPANY 19
DDS Concept Rx Concept/Type/Operator
Topic of type T An object that implements IObservable<T>, which internally creates a DataReader<T>
Communication status, Discovery event streams
IObservable<SampleLostStatus>
IObservable<SubscriptionBuiltinTopicData>
Topic of type T with key type=Key
IObservable<IGroupedObservable<Key, T>>
Detect a new instance Notify Observers about a new IGroupedObservable<Key, T> with key==instance. Invoke IObserver<IGroupedObservable<Key, T>>.OnNext()
Dispose an instance Notify Observers through IObserver<IGroupedObservable<Key,T>>.OnCompleted()
Take an instance update of type T
Notify Observers about a new value of T using Iobserver<T>.OnNext()
Read with history=N IObservable<T>.Replay(N) (Produces a new IObservable<T>)
DDS and Rx: A Great Match
10/10/2014 © 2012 RTI • COMPANY 20
DDS Concept Rx Concept/Type/Operation
Query Conditions Iobservable<T>.Where(…) OR
Iobservable<T>.GroupBy(…)
SELECT in CFT expression IObservable<T>.Select(...)
FROM in CFT expression DDSObservable.FromTopic(“Topic1”)
DDSObservable.FromKeyedTopic(“Topic2”)
WHERE in CFT expression IObservable<T>.Where(...)
ORDER BY in CFT expression IObservable<T>.OrderBy(...)
MultiTopic (INNER JOIN) IObservable<T>.Join(...)
.Where(...)
.Select(...)
Join between DDS and non-DDS data
Join, CombineLatest, Zip
Living Demo in C#
10/10/2014 Real-Time Innovations, Inc. 21
Background Code
10/12/2014 Real-Time Innovations, Inc. 22
public class ShapeType : ICopyable<ShapeType>{
public string color;public int shapesize;public int x;public int y;
public ShapeType();}
public class ShapeTypeExtended : ShapeType{
public float angle;public ShapeFillKind fillKind;
public ShapeTypeExtended();}
private DDS.TypedDataWriter<ShapeTypeExtended> triangle_writer;
Flower Demo
10/12/2014 Real-Time Innovations, Inc. 23
IDisposable flower(DDS.DomainParticipant participant){
int a = 30, b = 30, c = 10;
returnObservable
.Interval(TimeSpan.FromMilliseconds(1), Scheduler.Immediate)
.Select((long x) =>{
int angle = (int)(x % 360);return new ShapeTypeExtended{
x = (int)(120 + (a + b) * Math.Cos(angle) + b * Math.Cos((a / b - c) * angle)),y = (int)(120 + (a + b) * Math.Sin(angle) + b * Math.Sin((a / b - c) * angle)),color = "GREEN",shapesize = 5
};}).Subscribe(triangle_writer);
}
Simple Square to Triangle Transformation
10/12/2014 Real-Time Innovations, Inc. 24
IDisposable forward_verbose(DDS.DomainParticipant participant){
var rx_reader = DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square");
IDisposable disposable =rx_reader.OnDataAvailable((ShapeTypeExtended shape) =>{
DDS.InstanceHandle_t handle = DDS.InstanceHandle_t.HANDLE_NIL;triangle_writer.write(shape, ref handle);
});
return disposable;}
IDisposable forward_shortest(DDS.DomainParticipant participant){
return DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square").OnDataAvailable(triangle_writer);
}
Swap Square’s x and y and Propagate Stream Dispose Event
10/12/2014 Real-Time Innovations, Inc. 25
IDisposable swap(DDS.DomainParticipant participant){
return DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square").Select(shape => new ShapeTypeExtended{
x = shape.y,y = shape.x,color = shape.color,shapesize = shape.shapesize
}).SubscribeAndDisposeOnCompleted(triangle_writer,
new ShapeTypeExtended { color = "BLUE" });}
Aggregator (Square + Circle = Triangle!)
10/12/2014 Real-Time Innovations, Inc. 26
IDisposable aggregator(DDS.DomainParticipant participant){
var rx_square_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Square");
var rx_circle_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant, "Circle");
return new CompositeDisposable(new IDisposable[] {
rx_square_reader.Subscribe(triangle_writer),rx_circle_reader.Subscribe(triangle_writer)
});
}
Square/Circle Correlator using LINQ
10/12/2014 Real-Time Innovations, Inc. 27
IDisposable selectmany_correlator(DDS.DomainParticipant participant, bool useLinq){
var rx_circle_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant,
"Circle", Scheduler.Default);
var rx_square_reader =DDSObservable.FromTopic<ShapeTypeExtended>(participant,
"Square", Scheduler.Default);
var correlator =from square in rx_square_readerfrom circle in rx_circle_reader.Take(1)where square.color == circle.colorselect new ShapeTypeExtended{
x = square.x,y = square.y,color = square.color,shapesize = circle.x
};
return correlator.Subscribe(triangle_writer);}
Thank You!
• Rx4DDS.NET– https://github.com/rticommunity/rticonnextdds-reactive
• Data Distribution Service– http://portals.omg.org/dds
• Real-Time Innovations, Inc.– www.rti.com
10/12/2014 Real-Time Innovations, Inc. 28
top related