vldb 2010 event processing tutorial

IBM Haifa Research Lab – Event Processing

© 2010 IBM Corporation

Event processing – past, present, future

VLDB 2010 Tutorial, Singapore, September 15th, 2010

Opher Etzion ([email protected])


© 2010 IBM Corporation2

An example to kick-off this tutorial:

Act: Passenger has been re-routed to another destination – send the luggage

Act: Bag has reached to the wrong aircraft

Notify: Bag has been checked but did not reach the ULD within 20 minutes

Notify: Bag has been checked but did not reach the connecting flight

THE LUGGAGE PERSPECTIVE:

Across the 24 largest airlines more than 5.6 million bags went missing in 2006, this is an average of 15.7 bags per 1,000 travelers. 15% of the bags are never found.

BBC News, April 4, 2007

Event processing can help here..



Outline of this tutorial

What is behind event processing ? How is it related to other computing terms? Where are its roots?

Event processing – architecture, building blocks

The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications

The future: Trends and research challenges



What is behind event processing ? How is it related to other computing terms? Where are its roots?



What is “event processing” anyway?

or

Event processing is a form of computing that performs operations on events



Event processing and Data stream management?

Aliases?

One of them subset of the other?

Totally unrelated concepts?



In computing we processed events since early days

Network and System Management



Emerging technologies in enterprise computing(Gartner Hype Cycle, Summer 2009)



What’s new?The analog: moving from files to DBMS

In recent years – architectures, abstractions, and dedicated commercial products emerge to support functionality that wastraditionally carried out within regular programming. For some applications it is an improvement in TCO; for others is breaking the cost-effectiveness barrier.



What is an event

There are various definitions of events

Event is an occurrence within a particular system ordomain; the word event has double meaning: the real-worldOccurrence as well as its computerized representation



In daily life we often react to events..



Sometime we even react to the occurrence of multiple events

I closedthe deal with the Canadians

We closed two huge deals in a single day,It is a good opportunity to send all the team toSome fun time in Singapore

I closed the deal with the Australians



EventPatterns

Pattern detection is one of the notable functions ofevent processing



What we actually want to react to are – situations

TOLL VILOATOR FRUSTRATED

CUSTOMER

Sometimes the situation is determined by detecting thatsome pattern occurred in theFlowing events.

Toll violation Frustrated customer

Sometimes the events can approximate or indicate withsome certainty that the situation has occurred



Event processing is being used for various reasons


IBM Haifa Research Lab – Event Processing © 2008 IBM Corporation

DetectDecide

Respond

EP Solution Segments – Business Value

Real-Time Operational

Information Dissemination

Observation

Predictive ProcessingActive Diagnostics

Reactions to events are done as part of business transactions – achieving low latency decisions, and quick reaction to threats and opportunities

Getting the

right

information

in the right

granularity

to the right

person at

the right

time

Diagnose

problems

based on

symptoms and

resolve them

Quick observation into exceptional business behavior and notification to the appropriate people.

Mitigate or

eliminate

predicted events



Event Driven Architecture

Event driven architecture: asynchronous, decoupled; each component is autonomic.



Ancestor: active databases

On event

When condition

Do action

With coupling mode

Composite events were inherited to event processing



Ancestor: Data Stream management system

Source: Ankur Jain’s website



Ancestor: Temporal databases

There is a substantial temporal nature to event processing. Recently – also spatial and spatio-temporal functions are being added



Ancestor: Messaging – pub/sub middleware



Ancestor: Discrete event simulation



Ancestor: Network and system management



Event processing – architecture, building blocks

24

Flower StoreVan Driver

Ranking and Reporting

System

Bid Request

Delivery Bid

Assignments,

Bid alerts, Assign Alerts

Control System

GPS Location

Location Service

Location

Driver’s Guild

Ranking and reports

Delivery

confirmation

Pick Up confirmation

Ranked drivers / automatic assignment

Bid System

StorePreferences

Delivery Request

Assignment System

Manual Assignment Assignment

Assignments,

Pick Up Alert

Delivery Alert

Fast Flower Deliveryhttp://www.ep-ts.com/EventProcessingInAction



Event Processing AgentContext

EventChannel

EventConsumer

EventType

EventProducer

GlobalState

The seven

Building blocks



Event processing network

Event Producer 1

Event Consumer 1

Event Consumer 2

Event Producer 2

Event Consumer 3

Agent 2

Channel

Agent 1

State

Agent 3



Example of EPN – part of the FFD example

Automatic assignment

Driver

Bid routing

Store reference

Manual assignment preparationBid Request

channel Bid enrichment

Assignment manager

Alerts channel

Assignment channel

Assignment request channel

Driver status

Driver enrichment



Event type definition

Header Payload Open content

System defined event attributes

Attributes specific to the event type

Additional free format data included in the event instance

Detection time, Occurrence time, source, Certainty…

Stock id, quote, volume… Free

comments…



Producer – State Observer in workflows

State observerPush:

Instrumentation points;

Pull:

Query the state



Producer – Code instrumentation



Producer – syndication



Producers – video streams to events



Producer – sensors



Producer and consumer - Sixth sense



Twitter as a producer and consumer



Consumer - Performance monitoring dashboard



Consumer - Ambient Orb



Event Processing Agent

Filter Transform Detect Pattern

Translate Aggregate Split Compose

Enrich Project

Event Processing Agents



The EPA picture

Output

Not selected

Instanceselection

Context expression

Pattern detect EPA

Relevance filtering

Input terminal filter expression

Relevant event types

DerivationDerivation expression

MatchingPattern signature:

Pattern typePattern parametersRelevant event typesPattern policies

Pattern matching set

Participant events



Filter EPAA filter EPA is an EPA that performs filtering only, and has no matching or derivation steps, so it does not transform the input event.

Filtering

Filter EPA

Filtered Out

Filtered In

Non-Filterable

Principal filter expression



Transform EPA sub types

Translate Compose

Aggregate Enrich

Split

Project



Sample of pattern types

all pattern is satisfied when the relevant event set contains at least one instance of each event type in the participant set

any pattern is satisfied if the relevant event set contains an instance of any of the event types in the participant set

absence pattern is satisfied when there are no relevant events

relative N highest values pattern is satisfied by the events which have the N highest value of a specific attribute over all the relevant events, where N is an argument

value average pattern is satisfied when the value of a specific attribute, averaged over all the relevant events, satisfies the value average threshold assertion.

always pattern is satisfied when all the relevant events satisfy the always pattern assertion

sequence pattern is satisfied when the relevant event set contains at least one event instance for each event type in the participant set, and the order of the event instances is identical to the order of the event types in the participant set.

increasing pattern is satisfied by an attribute A if for all the relevant

events, e1 << e2 e1.A < e2.A

relative max distance pattern is satisfied when the maximal distance between any two relevant events satisfies the max threshold assertion

moving toward pattern is satisfied when for any pair of relevant events

e1, e2 we have e1 << e2 the location of e2 is closer to a certain object then the location of e1.



Pattern detection example

Pattern name: Manual Assignment Preparation

Pattern Type: relative N highest

Context: Bid Interval

Relevant event types: Delivery Bid

Pattern parameter: N = 5; value = Ranking

Cardinality: Single deferred

Find the five highest bids within the bid interval

Taken from the Fast Flower Delivery use case



Why do we need policies - A simple example: heavy trading scenario

Given: A stream of events of a single topic, about the activity in the stock market for a certain stock.

An event is produced every 10 minutes when there is trade in the stock.

Each event consists of: quote (current stock-quote), volume (an accumulated volume of traded events within these 10 minutes).

The pattern based action is: trigger an automatic trade program if the volume exceeds 300,000 3 times within an hour; pass as an argument the last quote and the sum of the 3 volume values.

Event-Id Time-Stamp Quote Volume E1 9:00 33.23 E2 9:10 33.04 320,000 E3 9:20 33.11 280,000 E4 9:30 33.01 400,000 E5 9:40 32.90 315,000 E6 9:50 33.04 320,000 E7 10:00 33.20 303,000 E8 10:10 33.33 219,000 E9 10:20 33.11 301,000 E10 10:40 33.00 210,000 E11 10:50 32.78 400,000 E12 11:00 32.70 176,000

How many times the trade programming is triggered ;Which arguments are used in each triggering?

Why defining patterns is not that easy? Because we need to tune up the semantics



Pattern policies Evaluation policy—This determines when the matching sets are produced

Deferred

Cardinality policy—This determines how many matching sets are produced within a single context partition

Repeated type policy—This determines what happens if the matching step encounters multiple events of the same type

Every - Override – First – Last – with max/min value of…



Pattern policies – cont.

Consumption policy—This specifies what happens to a participant event after it has been included in a matching set

Consume Reuse Bounded reuse

Order policy—This specifies how temporal order is defined

By occurrence time – by detection time – by stream position – by attribute



Our entire culture is context sensitive

In the play “The Tea house of the August Moon” one of the characters says: Pornography question of geography

•This says that in different geographical contexts people view things differently

•Furthermore, the syntax of the language (no verbs) is typical to the way that the people of Okinawa are talking

When hearing concert people are not talking,

eating, and keep their mobile phone on “silent”.



Context has three distinct roles (which may be combined)

Partition the incoming events

The events that relate to each customer are processed separately

Grouping events together

Different processing forDifferent context partitions

Determining the processing

Grouping together events that happened in the same hour at the same location



Context Definition

A context is a named specification of conditions that groups eventinstances so that they can be processed in a related way. It assignseach event instance to one or more context partitions.

A context may have one or more context dimensions.

A context is a named specification of conditions that groups eventinstances so that they can be processed in a related way. It assignseach event instance to one or more context partitions.

A context may have one or more context dimensions.

TemporalTemporal

SpatialSpatial

State OrientedState Oriented

Segmentation OrientedSegmentation Oriented



Context Types Examples

SpatialSpatial


TemporalTemporal

ContextContext

“Every day between 08:00and 10:00 AM”

“A week after borrowing a disk”

“A time window bounded byTradingDayStart andTradingDayEnd events”

“Every day between 08:00and 10:00 AM”

“A week after borrowing a disk”

“A time window bounded byTradingDayStart andTradingDayEnd events”

“3 miles from the trafficaccident location”

“Within an authorized zone ina manufactory”

“3 miles from the trafficaccident location”

“Within an authorized zone ina manufactory”

“All Children 2-5 years old”“All platinum customers”“All Children 2-5 years old”“All platinum customers”

“Airport security level is red”“Weather is stormy”“Airport security level is red”“Weather is stormy”




Context Types

Fixed locationFixed location

Entity distance locationEntity distance location

Event distance locationEvent distance location

SpatialSpatial


Fixed intervalFixed interval

Event intervalEvent interval

Sliding fixed intervalSliding fixed interval

Sliding event intervalSliding event interval

TemporalTemporal


ContextContext



The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications



An ObservationThe Babylon Tower symbolizes the tendencyOf humanity to talk in multiple languages.

The Event Processing area is no different: most languages in the industry really followthe hammer and nails syndrome – and extended existing approaches• imperative script language• SQL extensions• Extension of inference rule language

The epts language analysis workgroup is aimed to understand the various stylesAnd extract common functions that can be used to define what is an event processing language; this tutorial is an interim report

It does not seem that we’ll succeed to settleIn the near future around a single programming style



Existing Styles for EP languages (samples)

InferenceRules

Stateoriented

Agent Oriented

Imperative/Script Based

RuleCore

Oracle

Aleri Streambase

Esper

EventZero

TIBCO

WBE

Apama

Spade

AMiT

Netcool Impact

* - if we add simple and mediated event processing the picture is even more diversified

ECARules

SQL extension

Coral8

Agent LogicStarview

XChangeEQProva



StreamBase Studio



StreamBase Pattern Matching



CCL Studio (Coral8 Sybase)



CCL – Pattern Matching

RFID monitoring application

Checks if a tag has been seen by readers A and B, then C, but not D, within a 10 second window.

Insert into StreamAlerts Select StreamA.id From StreamA a, StreamB b, StreamC c, StreamD d Matching [10 seconds: a && b, c, !d] On a.id = b.id = c.id = d.id

Insert into StreamAlerts Select StreamA.id From StreamA a, StreamB b, StreamC c, StreamD d Matching [10 seconds: a && b, c, !d] On a.id = b.id = c.id = d.id



Microsoft Streaminsights

var topfive = (from window in inputStream.Snapshot() from e in window

orderby e.f ascending, e.i descending select e).Take(5);

var avgCount = from v in inputStream group v by v.i % 4 into eachGroup

from window in eachGroup.Snapshot() select new { avgNumber = window.Avg(e => e.number) };



Esper EPL – FFD Example

/** Not delivered up after 10 mins (600 secs) of the request target delivery time*/insert into AlertW(requestId, message, driver, timestamp)select a.requestId, "not delivered", a.driver, current_timestamp()from pattern[

every a=Assignment (timer:interval(600 + (a.deliveryTime-current_timestamp)/1000) and

not DeliveryConfirmation(requestId = a.requestId) and

not NoOneToReceiveMSG(requestId = a.requestId))];



ruleCore - Reakt Event stream view - a unique context of events

a view contains a window into the inbound stream of events andcontains commonly only semantically related events

Situation - an interesting combination of multiple events as they occur over time

An item with an RFID tag being picked up from the shelf and then moving past the checkout without being paid for

Rule - an active event processing entity reacting to specific combinations of inbound events over time

Action - the last part of a rule's evaluation in response to a detected situation



Amit Terminology

Situation

Conditions

Lifespane5e5 e8e8

Initiator Terminator

Event SelectionEvent Selection ActionsActionsOperationOperation

Keys

e1e1

e3e3

Input events

OperatorJoiningCountingTemporalAbsenceAggregation

Notifications

Messages

Definition updates

User plug-ins

e2e2



Amit - Situation



IBM Websphere Business Events



Apama EPL – FFD Examples



Apama Simulation Studio – cont.



Non functional properties - scalability

scalability in the volume of processed events

Scalability in the quantity of agents

Scalability in the quantity of producers

scalability in the quantity of consumers

scalability in the quantity of context partitions

Scalability in context state size

Scalability in the complexity of computation

scalability in the processor environment

Scalability is the capability of a system to adapt readily to a greater or lesser intensity of use, volume ,or demand while still meeting its business objectives .



Non functional requirements - securityEnsuring only authorized parties are allowed to be event producers

of event consumersEnsuring that incoming events are filtered so that authorized

producers cannot introduce invalid events, or events that they are not entitled to publish

Ensuring that consumers only receive information to which they are entitled. In some cases a consumer might be entitled to see some of the attributes of an event but not others.

Ensuring that unauthorized parties cannot add new EPAs to the system, or make modifications to the EPN itself (in systems where dynamic EPN modification is supported)

Keeping auditable logs of events received and processed, or other activities performed by the system.

Ensuring that all databases and data communications links used by the system are secure.



Performance objectives

Objective

Number

Objective Name Objective metrics

1 MAX input

throughput

Maximize the quantity of input events processed by a

certain system or sub-system within a given time period

2 MAX output

throughput

Maximize the quantity of derived events produced by a

certain system or sub-system within a given time period

3 MIN average

latency

Minimize the average time it takes to process an event

and all its consequences in a certain system or sub-

system

4 Min maximal

latency

Minimize the maximal time it takes to process an event

and all its consequences in a certain system or sub-

system

5 Latency

leveling

Minimize the variance of processing times for a single

event or a collection of events in a certain system or sub-

system

6 Real-time

constraints

Minimize the deviation in latency, from a given value,

for the processing of an event and all its consequences in

a certain system or sub-system.



Optimizations

Optimizations related to EPA assignment: partition, parallelism, Distribution, load balancing.

Optimization related to the execution process: scheduling, routing optimizations and load shedding

Optimizations related to the coding of specific EPAs: code optimization, state management.



Putting derived events in order

Auction Context Interval timeB

id event

Auctionentry

Anomaly 1

Bid

events

Auction

entries

Anomaly 2



Ordering in a distributed environment - possible issues

The occurrence time of an event is accurate, but the event arrives out-of-order and processing that should have included the event might already been executed.

Neither the occurrence time nor detection time can be trusted, so the order of events cannot be accurately determined.



Solutions:

Buffering techniques: Assumptions:

Events are reported by the producers as soon as they occur;

The delay in reporting events to the system is relatively small, and can be bounded by a time-out offset;

Events arriving after this time-out can be ignored.

Principles: Let be the time-out offset,

according to the assumption it is safe to assume that at any time-point t, all events whose occurrence time is earlier than t - have already arrived.

Each event whose occurrence time is To is then kept in the buffer until To+, at which time the buffer can be sorted by occurrence time, and then events can be processed in this sorted order.

Retrospective compensation:

Find out all EPAs that have already sent derived events which would have been affected by the "out-of-order" event if it had arrived at the right time.

Retract all the derived events that should not have been emitted in their current form.

Replay the original events with the late one inserted in its correct place in the sequence so that the correct derived events are generated.



Inexact event processing uncertainty whether an event actually occurred Uncertainty when exactly the event happened inexact content in the event payload inexact matching between derived events and the situations they purport to

describe

Sourcemalfunction

Malicioussource

Projection oftemporal

anomalies

Imprecise source

Sampling orestimate

Propagation of

inexactness

Uncertainevent

Inexact event content



The future: Trends and research challenges



Trend I: Going from narrow to wide

Border security radiation detection

Mobile asset geofence

Logistic and scheduling

Unauthorized use of heavy machinery

Hospital patient and asset tracking

Activity monitoring for taxing and fraud detection

Intelligent CRM in banking

EDA and asynchronous BPM in retail

Situation awareness in energy utilities

Situation awareness in airlines

Reduce cost in injection therapy

Next generation navigation

Real-time management of hazardous materials

Finding anomalies in point of sales in retail stores

Elderly behavior monitoring

Some recently reported applications (EPTS use-cases WG)

Source: ebizQ Event processing market pulse



Trend I: Going from narrow to wide

Taking event processing outside enterprise computing:

Robotics

Bio-Informatics

Socio-technical systems

HomeAutomation



Trend II: Going from monolithic to diversified

Variety of functions

Variety of Quality of Service requirements:

Variety of platforms

“One size fits all” will not work – Instead a collection of building blocksthat can fit together



Trend III: Going from proprietary to standard-based – standard directions

The current situation: Babylon tower: variety of languages, event representation…

Serves as enabler to achieving other trends and general maturity

The shift of vendors from start-up dominant to bigger companies makes the atmosphere more friendly towards standards.

Areas forStandards:ModelingEvent representationInteroperabilityLanguages PIM



Trend IV: Going from programmer centered to semi-technical person centered

Source: ebizQ Event processing market pulse



Trend V: Going from stand-alone toembedded

Packaged applications

Business Activity Monitoring

Sensor Platform

Middleware and platforms



Trend VI: Going from reactive to proactive

TRAFFIC JAM

End of game



Emerging directions: Four directions to observe

Multiple platforms – same look and feel

The engineering of constructing EP applications

Adding intelligence toEvent processing

Tailor-made optimizations



Emerging direction I: multiple platforms – same look and feel

ApplianceAppliance Stream Platform

Stream Platform

Cloud computingPlatform

Cloud computingPlatform

ESB / Messaging Platform

ESB / Messaging Platform

EmbeddedEmbedded

Virtual Event Processing Platform



Emerging direction II: Tailor-made optimizations

Local optimizations: each EPA will be optimized for its own purpose / assumptions / QoS indicators – average/worst case latency, input/output throughput …

Global optimization: scheduling, load balancing, assignment…

EPA EPA

EPA

EPA

Producer

Producer

EPA

Consumer

Consumer

Local optimizations

Global optimizations



Emerging direction III: Event processing software engineering

Best practices

MethodologiesModeling

& meta-modeling

DesignPatterns



Emerging direction IV: Intelligent event processing

Offline and continuous mining of meaningful patterns in event histories

Inexact event processing – handling inexact events and also false positives and false negatives

Causality – a key for proactive, but also vital for provenance



Summary

Event processinghas emerged frommultiple academic disciplines

The state of the Practice is the 1st generation of products – mainlyengineering based

Going to the nextphase – many challenges that require collaborationof research andindustry

Already attracted coverage of analysts and all major software vendors

The Dagstuhl grand challenge

vldb 2010 event processing tutorial

Documents