Keith McFarlanekeith dot mcfarlane at gmail dot com

AgendaSensor Fusion ConceptsComplex Event Processing ConceptsEsper CEP EngineEventBotNext Steps

Concepts

What is sensor fusion?“…combining…sensory data or data derived from sensory data

from disparate sources such that the resulting information is in some sense better than would be possible when these sources were used individually.”1

1From Wikipedia article on Sensor Fusion

Sensor readings (bumps, ultrasonic distance measurements, rotations, etc.) give only limited views of the world

When taken together, they can be used to build a model of the world they are sensingAnalogous to perception – the lens through which

the robot views the worldOnce perception is realized, the robot can achieve

situational awarenessFinally, the robot can make predictions based on its

model, act according to its goals, and measure the model’s accuracy

Why give sensor fusion special attention?All robotics systems require some level of sensor

data correlationRequires special-purpose code – even if developed

in a general, parameterized way, it is bound to “bake-in” some assumptions

The application of open source software to the sensor fusion problem advances all at onceNo need to “re-invent the wheel” – address new

problems instead of solved problemsShare fusion advances with a broad community

Benefit from the “wisdom of the crowd”Access to cleaner code over time

Everyone contributes fixes and features, everyone benefits

Principles for Dynamic World Modeling1

1. Express real-world primitives as set of properties

2. Observation and model should be expressed in common coordinate system

3. Observation and model should be expressed in common vocabulary

4. Properties should include an explicit representation of uncertainty

5. Primitives should be accompanied by a confidence factor.

1Crowley et al

The Predict-Match-Update Cycle

Transform MatchObservations Update

World Model

Predict

A model for fusing sensor readings into an actionable view of the world

Sensors gather raw data about the world

The data are put into some form that is compatible with the model

The data are matched against / merged with previous predictions about the world

The model is updated with the new data

Predictions are reset based on updated model

Symbolic vs. Numeric MethodsSymbolic fusion entails the development and

maintenance of a schema-based world modelProperties: attributes of entities in schemaCoordinate system: position information should be

consistent with world modelVocabulary: observations must to be mapped to schema

entities and attributesPrecision: attribute value variance may be provided or

calculated using variance from predictionsConfidence: overall trust in world model may be

calculated and represented as a stateNumerical sensor fusion methods apply for math-

intensive domainsVision-based applications fall into this domain, for

example

Concepts

Enterprise Origins: SOA vs. EDAService-oriented Architecture (SOA) is a style of enterprise

software development and governanceEmphasizes discrete, distributed units of stateless software

logic over monolithic applicationsEncourages information hidingWell supported by open source tools (Apache Tomcat,

Glassfish)Event-driven Architecture (EDA) is a supplemental

approach to enterprise software development and governanceAccounts for the real-time nature of many business

transactionsEncourages discovery of business-relevant situationsNot as well supported by open source (or other) tools, but

support is growingComplex Event Processing (CEP) technologies apply very well

to problems in the EDA domain SOA Tools aren’t always a fit

Complex Event ProcessingThe acquisition, transformation, and correlation of

streams of events for use in real-time or near-real-time applicationsMany common business scenarios can be

interpreted as CEP problems Authentication: if ten password denial events occur

inside of ten seconds, lock the user out Fraud detection: if a credit card is used at locations many

miles apart within a short time period, alert the cardholderMost robotics applications could be described as

CEP applications Robots acquire events via sensors, timers, web services,

mobile phones, etc. Robots transform these into some common internal

representation Robots determine what actions to take based on

interpretations of event content, timing, coincidence, sequence, etc.

EventsFor our purposes, an event is an immutable

historical record of some observed occurrenceHistorical: by definition, events have already

happened Important to track time at the proper resolution

Immutable: no changes allowed Generally, it would not be useful to modify events in real

time Better to generate new events representing a

reinterpretation of event meaningObserved: generated externally or through

correlation Simple events represent some single occurrence

Often useful to consider a variety of sourcesIn enterprise applications, events generally originate

from software services (applications, DB triggers,…)

Event conveyance vs. processingSometimes these concepts become entangled – a

frequent point of confusion in business integration circles

Conveyance: the medium of event movementLocal calls and callbacks – e.g., Java listeners

Typical of the Observer design patternNetwork calls – REST or SOAP Web Services, Java RMI,

CORBAMessage queues – JMS-based, MSMQ, IBM MQ Series

Processing: the application of logic to a set of event streamsEngines: Esper, Coral8, Avaya Event ProcessorGoal: glean information from the chaos of event traffic

Complex EventsComplex events represent the detection of a

related set of other simple or complex eventsOften used to identify detected “situations”

E1 E1 E1 E1 E1

E2 E2 E2 E2

E3

Event Queue 1

Event Queue 2

Event Queue 3

C1

Detect event count

C2

Detect event timing

C3Detect event coincidence

Scenario: Detecting Cheap GasGas Prices

Service

Event Processor

“Cheap Gas Nearby” Alert

New Price

New Price

New Price

New Price

New Price New

Price

New Locatio

n

New Locatio

n

New Locatio

n

New Locatio

n

New Location

Rule: My Location within one mile of gas station with price under $2.75 and gas tank low

Mobile phone GPS

Location Service

New Locatio

n

Sends real-time changes in gas prices

for a set of gas stations

Gas Level

SMS

Event Pattern AbstractionA complex event that represents, denotes, or

summarizes the events of which it is composedSome examples:

A “Completed Loan Origination” event might represent the “New Loan Request,” “Credit Verified,” and “Final Approval” events in sequence

A “Human in Path” event might be generated by a car’s navigation system for coincident infrared, radar and vision events

Event processors generally support the ability to create new event streams based upon a detected set of conditions Streams of events may be cascaded for higher levels of

situational understandingSome event processors support the ability to “drill

down” to the events that triggered the abstract event

Sensor Fusion and CEP Conceptual Relationships

Sensor Fusion CEP

Collects observations Receives Events

Requires normalized observation schema

Requires normalized event data

Dynamic world modeling Event stream maintenance

Matching Event correlation/pattern detection

Situational awareness Event pattern abstraction

Symbolic fusion Event schema

Used to model the world using limited data set and predictive feedback model

Used to detect specific situations

Esper: Open Source CEPEsper is an open source CEP platform with a

rich toolset for processing streaming eventsOpen source version is very capable – all core

functions present Event query language Pattern recognizer Support for JavaBeans, Maps, and XML documents High throughput – up to 500,000 events/sec. for a

two-processor system1

Extensible – not difficult to add new input and output event connectors or event types

Commercial version available that includes HA features

http://docs.codehaus.org/display/ESPER/Esper+performance

Esper Container Architecture

Esper Lightweight CEP Container

Event Stream Connectors

and Adapters

Historical Data Access Layer

Events

Container Core

Event Query and Pattern Language Executive

Event Query and Pattern Language

Esper Engines

StatementsStatementsStatements

StatementsStatements

POJOs Output Adapters

Source: http://www.espertech.com

EPL: Event Processing LanguageUsed to specify the event processor’s behavior

when confronted with events or patterns of eventsSQL-like in appearance – helps SQL programmer’s

see analogous relationships May also confuse the issue for those who have trouble

“letting go”Rather than tables, EPL statements are run

against “streamsSQL idea of a row is replaced by the “event”

conceptIt is possible to include data from JDBC data

sources to augment event processing For example, lookup tables could be used to retrieve a

customer’s current balance based on a customer ID delivered with an event

EPL: Familiar Concepts from SQLClause Description

SELECT Indicates an that events should be selected based on a set of criteria. Can select on insert stream, remove stream, or both.

FROM Defines the event stream(s) or named window(s) from which events should be selected

WHERE Allows filtering of events based on a wide range of criteria

GROUP BY Allows stratification of events by a set of characteristics

HAVING Filters groups specified by the GROUP BY clause

INSERT INTO Creates new events in the given named window, allowing events to cascade through the system

An Example EPL Statementselect * from BatteryChargeEvent(charge<400).win:length(5)

The “select *” statement preamble indicates an event retrieval operation inclusive of all event properties

“from Withdrawal” names the event stream from which to pull events

“amount>=200” filters the events that will be accepted into the stream

“win:length(5)” defines the stream as containing the last five matching events inserted

Event StreamsDefined by EPL statement FROM clause, and

may be one of two types: filter or patternFilter-based event streams apply constraints

to incoming events, limiting stream contentExample:

select * from DistanceEvent(distance>0).win:length(10)

Supports standard comparators (=, !=, >, <, >=, <=) and ranges (between, in, not)

Pattern-based event streams look for the specified pattern of eventsExample:

select * from pattern [every left=LeftBumpEvent or every right=RightBumpEvent].win:time(5 sec)

ViewsUsed to derive or aggregate data from an

event streamMay be “staggered” (chained) within a single

EPL statementExample:

select * from DistanceEvent(distance>0).win:length(10).std:groupby(direction)

Some views delivered with Esper:win:length win:length_batc

hwin:time win:time_batch

win:time_length_batch

win:time_accum win:ext_timed

ext:sort_window

ext:time_order std:unique std:groupby win:firsttime

Hardware El CheapoLaptop

Roo232 USB-to-SerialCableRoomba

Sage

Reversed MacMini-DIN 8 M2M Cable

Roomba ConnectorHome-grown Java library that speaks Roomba SCIMulti-threaded for better processor utilizationCommand pattern applied for command encapsulation

Allow for flexibility in both queuing and macro command building

Commands are queued and processed asynchronouslySensors polled with specified frequency

Current default is 10 polls/secondPolling results in generation of event classes that are

sent to subscribed Java listenersRelevant event parameters bundled in event object

The Command Pattern

EventBot Data Flow vs. Fusion Model

Roomba Connector

Event Transformer

Esper CEP

Event Listeners

Roomba

Transform MatchObservation

sUpdate

World Model

Predict

Events Commands

World Model

Code WalkthroughExample: Governing speed via rotation sensors –

a simple sensor fusion exampleFuses multiple readings into better information

about the environment Uses an average of the last N readings to match speed

and adjust power if neededUses this data to adjust in continuous feedback loopElements:

Connector Coding EPL Statements Listener Coding

Runtime

Next StepsTransducer Markup Language (TML) adapter

Allows wide variety of sensor types to be normalized

More complex fusion exampleMore sensors – simple vision? Ultrasonic?RSS event feedRFIDWS-Eventing adapter

Open source availability of tools

email: keith dot mcfarlane at gmail dot com

ReferencesLinks

Esper home http://esper.codehaus.org

http://www.complexevents.comhttp://www.espertech.comhttp://en.wikipedia.org/wiki/Sensor_fusionhttp://en.wikipedia.org/wiki/Gang_of_Four_(software)

PapersPrinciples and Techniques for Sensor Data Fusion

http://www-prima.imag.fr/jlc/papers/SigProc-Fusion.pdf Crowley et al.

Overview of CEP Concepts http://complexevents.com/wp-content/uploads/2008/07/

overview-of-concepts-of-cep.pdf David Luckham