event processing with the padres publish/subscribe system hans-arno jacobsen bell university...

Event Processing with the PADRES Publish/Subscribe System

Hans-Arno Jacobsen

Bell University Laboratory Chair

Middleware Systems Research Group

University of Toronto

MIDDLEWARE SYSTEMSRESEARCH GROUP

http://www.padres.msrg.utoronto.caThe PADRES Team


Copyright © 2008 - Hans-Arno Jacobsen

Current PADRES Team

Chen Chen Alex Cheung Alton Chiu Amer Farroukh Patrick Lee Guoli Li

Bala Maniymaran Serge Mankovskii, CA Vinod Muthusamy Reza Sherafat Naweed Tajuddin Chunyang Ye Young Yoon

Countless alumni (see our web site.)

PADRES started around 20042CANOE Summer School, Norway, 2009



Try it out: Java Kick Start PADRES!

http://www.padres.msrg.utoronto.ca

3CANOE Summer School, Norway, 2009



Querying the Future




Amazon to Chapters to You ....

Monday, October 10th in Cyberspace

Your book “...”is available

at .... $10 off

Thursday, November 15th,in Toronto



Copyright © 2008 - Hans-Arno JacobsenCopyright © 2008 - Hans-Arno Jacobsen

Border Security Scenario

[class,manifest],[shipID,ACME123],[firm,ACME][class,manifest],[shipID,ACME123],[firm,ACME]

[class,audit], [firm,ACME], [trust, 0][class,audit], [firm,ACME], [trust, 0]

[class, reading],[shipID,ACME123],[level, 4][class, reading],[shipID,ACME123],[level, 4]




elseelse

Business Process Example

Checkscore

Creditcheck

Creditcheck 2

Checkscore 2

Reject

Approve

Send toofficer

Store inDB

…

…

Loan Application Processing

RequestSubmission

RequestSubmission

RequestApprovalRequestApproval

CheckCompletion

CheckCompletion




Large-scale Business ProcessesVendorVendor

SaleSale

ManufactoryManufactory

FinanceFinance

Dispatch B

Out-stock B

Pick-up goods

Packaging

MarketingMarketing

Design

Out-stock B

Target price

Prototype

Out Take

Control

Assign

Confirm

Determinateplan

Check stock

Rawmaterials

Audit

Raw

Determinateplan

Executeplan

Processcontrol

Monitor

Process

Pay

Check

Signature

Print receipt

WarehouseWarehouse

Delivery

FedEx

Pick up

Monitoring

Statistic

Chart

StrategyStrategy DesignDesign MarketingMarketing OrderOrderManufactoryManufactory PaymentPayment

Requirementcollection

Feature selection

Goods selection

Confirmfeatures

Material

Make plan

Feedback

Check order

Fill order

Check dealer Check credit

Approval Approval

Validate

Affirm order

Sale prediction

SignContract

CCCadministrate

Goods delivery

Fill dispatch bill

Fill out-stock bill

Credit card




What is the Common Denominator?

Many applications are driven by asynchronous state transitions. Something happens, … an appropriate reaction is expected

and required.

Asynchronous state transitions represent events. A process is triggered, a request submitted, …

Many applications require event management and processing capabilities to run effectively.




In Terms of the Examples These applications are driven by events

Information matching the query is found and indexed Person walks by a bookstore Loan request is submitted online Radioactivity is detected

Abstractly speaking events are disseminated and filtered against expressions or queries

events queries




What Event Processing Support is Required ? De-coupled & loosely coupled many-to-many interactions

Fine-grained filtering

In-network processing

Composite event detection and event correlation

Event dissemination




Many Applications are Event-based

A

B C

D

E

F

RFID and sensor networksService oriented architectures

Workflows, business processes and job scheduling

Supply chain and logistics

Event-Based

Light

CallbackRazor SKU

Transform

Fault

Temperature

Invoke Loan

Order

Delivered

In flightJob A done

Trigger




What is an event?

Etymology

Latin: eventus;

evenire to happen;

e- + venire to come




Various Interpretations

A postulated outcome, something that follows as a result or consequence, condition, or eventuality; “in the event that I am not there, you may …”

Any of the contests in a program of sports, a social occasion or activity, a world event

The fundamental entity of observed physical reality represented by a point designated by three coordinates of place and one of time in the space-time continuum postulated by the theory of relativity

A subset of the possible outcomes of an experiment (probabilities)

Something that happens, a noteworthy happening, an occurrence




Event – My Working Definition An event is an instantaneous and asynchronous state

transition in the environment of interest to applications, systems, users.

Event versus event occurrence Like type or class versus instance

In my definition events do not extend over time Interval events are defined by starting and ending events

(a.k.a. composite events) My definition does not capture world events

Maybe important for a theory of events, but less so for the effective processing of events

My definition does not capture interval events Events associated with a time span, but those can be

modeled as composite events




Synonymously Used

The terms Event Event occurrence (Event) message Event object

are often used synonymously in practice. Also the pattern for detecting an event is

often simply referred to as event.




Similarly, …

Event source Event producer Event generation Event publisher

… sink … consumer … capture … subscriber




Event Processing

Event Processing is an emerging area and discipline that aims to define and develop Abstractions Architectures Systems Languages Patterns Models Standards

for the processing of events in systems, applications, and by end users.




A Note on CEP

Complex Event Processing (CEP) Event processing must not be complex

Or else, a marketing nightmare, I suspect Meant was probably Complex-event Processing

Or maybe the processing of complex events Not the complex processing of events

Let’s just keep to Event Processing and try to make it as simple as possible,




Events

Many applications are inherently event-based.

An event is an instantaneous and asynchronous state transition in the environment of interest to applications, systems, users.




Agenda

What is the right abstractions for processing events?

Our point of view

PADRES Event & Services Bus (ESB )

Innovative & Advanced Features

Building Applications with PADRES




What Abstractions Do Not Work? Databases

Great for managing historic data But what about future data (e.g., events)

Data streams Great for managing structured streams of tuples But what about un-structured, multi-typed, sporadic, un-

ordered events from many sources Rule-based expert systems

Great for inference and reasoning But what about managing large numbers of fined-grained

filters in distributed environments

Cum gran salis




What Abstractions Enable Event Processing?

The afore-mentioned points can best be addressed by The content-based publish/subscribe model Realized by content-based message routing

Events are conveyed as publications.

Event listening, filtering and correlating is based on content-based subscriptions managed by the pub/sub system.




Publish/Subscribe 101

Not all publish/subscribe is equal

Publish/Subscribe models and evolution Channel-based

OMG CORBA Event Service, … Topic-based

WS Notifications, OMG Data Dissemination Service … Type-based

OMG Data Dissemination Service (partially), … Content-based

The PADRES ESB (see below), … State-based Subject Spaces




Content-based Publish/Subscribe

Publisher Publisher

Subscriber Subscriber

Subscriptions

Publications

NotificationNotification

IBM=84

MSFT=27 INTC=19 JNJ=58ORCL=12

HON=24

AMGN=58

Stock marketsNYSE

NASDAQTSX

Subscriptions:IBM > 85ORCL < 10JNJ > 60

Broker(s)




data tuples subscriptions

query publication

Query and subscription are very similar.Data tuples and publication are very similar.

However, the two problem statements are inverse.

That’s Like Data Base Querying !!

sets of tuples

Abo

ut p

ast

Abo

ut f

utur

e

sets of tuples




The Content-based Model Language and data model

Boolean functions over predicates Predicates are attribute-operator-value triples

[class = reading] Subscriptions are conjunctions of predicates

[class = reading],[level > 3] Publications are sets of attribute-value pairs

[class,reading],[level,3],[id,5637]

P/S

events

notifications




Publish/Subscribe Matching Problem

Given a set of subscriptions, S, and a publication, e, return all s in S matched by e.

e is referred to as event or publication Splitting hairs

The Event is the state transition of interest in the environment

The Publication is the information about the event submitted to the publish/subscribe system

Simple problem statement, widely applicable, and lots of open questions




Scope of the Content-based Model

Matching semantic A subscription matches if all its predicates match Approximate semantic (e.g., close to, cheap, sunny) Semantic and similarity-based matching Probabilistic … Top-k matches

Example Tree-structured data

Graph-structured data

Un-structured data

Regular languages

Relational model

Subscription XPath RDF Query Keywords Regular expressions

SQL

Publication XML RSS feeds Text, documents Sentences over some alphabet

DBs, i.e., tables




Content-based Message Routing

Event-BasedEvent-Based

DecoupledDecoupled

FlexibleFlexible

ResponsiveResponsive

Content RoutingContent Routing

DeclarativeDeclarative

A1

S2

P1 S1

P2S2

[class,=,stock],[symbol,=,YHOO],[price,>,20]

[class,=,stock],[symbol,=,MSFT],[price,>,50]

[class,=,stock],[symbol,=,YHOO]

[class,=,stock],[price,>,40]A2

S1

[class, stock],[symbol, YHOO],[price,25][class, stock],[symbol, YHOO],[price,45]

[class, stock],[symbol, MSFT],[price, 55]




Publication Space

weight

height

Adv: [height > 70],[weight > 25]Sub: [height > 75],[weight > 20]

7075

Sub intersecting Adv

20 25

weight

height

Sub: [height > 75],[weight > 20]

70

90

Pub matching Sub

20 32

Pub: [height , 90],[weight , 32]




Publish/Subscribe in Industry Standards

CORBA Event Service CORBA Notification

Service OMG Data Dissemination

Service Java Messaging Service WS Eventing WS Notification, WS

Topics, WS Brokered Notifications

INFO-D (Grid Forum) AMQP

Emerging technologies RSS aggregators

PubSub.com, FeedTree Real-time data

dissemination TIBCO, RTI Inc.,

Mantara Software Application integration

Softwired Hardware-based brokers

Sarvega (Intel), Solace Systems, DataPower (IBM)




Publish/Subscribe in Academia Research projects

Elvin (Australia) Gryphon (IBM) Hermes (Cambridge) LeSubscribe (INRIA) PADRES (Toronto) REBECA (Darmstadt) SIENA (Boulder) ToPSS (Toronto)

… many more




ToPSS - The Toronto Publish/Subscribe System Family [2000 – present]

Matching algorithms Language expressiveness

vs. efficient matching

Routing protocols Network architectures &

scalability

Higher level abstractions Workflow execution Monitoring

S-ToPSS(semantic)

X-ToPSS(XML matching)

A-ToPSS(approximate)

persistent-ToPSS(subject spaces)

L-ToPSS(location-based)

ToPSS(matching)

M-ToPSS(mobile)

Ad hoc-ToPSS(ad hoc networking)

Federated-ToPSS(federation of ToPSS brokers)

Rb-ToPSS(rule-based)

P2P-ToPSS(peer-to-peer)

LB-ToPSS(load balancing)

FT-ToPSS(fault tolerance)

Historic-ToPSS(historic data)

CS-ToPSS(composite subs)

BPEL-ToPSS(BPEL execution)

JS-ToPSS(job scheduling)




Publish/Subscribe

Decoupling and interaction among many sources & many sinks

Event filtering & matching

Event & data dissemination

Query the future (relaxed later)

Service orchestration and choreography




Agenda

PADRES Event & Services Bus (ESB ) PADRES ESB & Vision PADRES Event Language PADRES Internals


Building Applications with PADRES




PADRES Events & Services Bus

First generation of students, when I looked away Peng Alex David aRno Eli Serge

PADRES is Publish/subscribe Applied to Distributed Resource Scheduling

PAdres is Distributed REsource Scheduling

http://www.padres.msrg.utoronto.ca

Acknowledgements

http://padres.msrg.utoronto.ca



Copyright © 2008 - Hans-Arno Jacobsen 23/4/2138

PADRES Vision

Server Farm

Computers

ComputersDatabase

Laptops

Computers

Workstation

Laptops

DatabaseServer

ServerSwitch

Server

Switch

Computing, Storage, and Networking Resources

Content-based Routing (Publish/Subscribe)

Content-based RouterClients (publisher/subscriber)

PADRES ESB

Business Process Execution

Deploy Control UpdateVisualize

Monitor ...

6

43

7start halt

Business Process Management and Business Activity Monitoring

Redirectresume

addremove

Event Management Framework

Application Events

Business Events

Complex Events

Network and System Events




PADRES Events & Services Bus Consists of pub/sub message brokers

Content-based publish/subscribe interface Content-based message routing Store-and-forward message queuing

Comprised of a federation of brokers deployed as overlay

Offers a slim client library for development

Available under the Eclipse Public License (EPL) – a BSD-like open (source) license modelhttp://padres.msrg.utoronto.ca !39CANOE Summer School, Norway, 2009



PADRES EVENT LANGUAGE




PADRES Event Language Event schema Event instances (a.k.a., publications or events) Subscriptions (a.k.a., atomic subscriptions) Composite subscriptions Historic event queries (a.k.a., historic

subscriptions) Combinations of the former

The above are unified in the PADRES SQL (PSQL), not covered today.




PADRES Event Language

Event schema: Advertisements A1: [class,=,reading], [shipID,=,*], [level,<,10] A2: [class,=,manifest], [shipID,=,*], [firm,=,*], [content,=,*] A3: [class,=,audit], [firm,=,*], [trust,>=,0]

Source: Publications / events P1: [class, reading],[shipID,ACME123],[level, 4] (induced from

A1) P2: [class, manifest],[shipID,ACME123],[firm,ACME] (induced from A2) P3: [class, reading],[shipID,ACME123],[level, 12] (not induced from A1)

Sink: Atomic & (composite) subscriptions S1: [class,=,reading], [level,>,3] S2: [class,=,audit], [firm,=,*],[trust, >=, 3]




Composite Subscription I

AND

OR

S1 S2

OR

S3 S4

AND

S5

CS={ {S1 OR S2} AND {S3 OR S4} AND S5 }

A composite event is the constellation ofevents being detected by the compositesubscription.

Si are atomic subscriptions. I.e., they are satisfied by a single, multi-attribute publication / event.

Composite subscriptions (CS) are used for event correlation, in-network filtering, and thedetection of composite events (complex event).

Use of composite subscriptions: Event correlation Business process management (shown later) Business activity monitoring (shown later)




Composite Subscription II A composite subscription consists of atomic subscriptions

connected by operators AND, OR Variable join

Examples:

[class,=,reading], [level,>,3] OR[class,=,audit], [firm,=,*], [trust, <=, 3]

[class,=,reading], [shipID,=,$X], [level,>,3] AND[class,=,manifest], [shipID,=,$X], [firm,=,$Y], [content,!=,fertilizer]AND[class,=,audit], [firm,=,$Y], [trust,>,7]

AND

AND

S1 S2

S3shipID

firm




Atomic Historic Event Query Future atomic subscription

Historic atomic subscription

Hybrid atomic subscription

now time

# of events per match

Past Future

[class,=,reading], [level,>,3]

[class,=,reading], [level,>,3], [start_time,<,NOW]

[class,=,reading], [level,>,3], [start_time,=,NOW-1h],[end_time,=,NOW+4h]




Composite Historic Event Query Future composite subscription

Historic composite subscription

Hybrid composite subscription

now timePast Future

[class,=,manifest],[firm,=,$X],[start_time,<,NOW]AND[class,=,audit],[firm,=,$X],[trust,>,7],[start_time,<,NOW]

# of events per match

[class,=,reading],[level,>,3],[shipID,=,$X]AND[class,=, manifest], [shipID,=,$X],[content,!=fertilizer]

[class,=,reading],[shipID,=,$X],[level,>,3] AND[class,=,manifest],[shipID,=,$X],[firm,=,$Y], [content,!=,fertilizer]AND[class,=,audit],[firm,=,$Y],[trust,>,7], [start_time,=,now-2 months]




PADRES Event Language

Queries about future events

Queries about past events

Queries that express composite events over past and future




PADRES INTERNALS




B

B

B

S

S

P

B

P

B

input queue

output queue B2

output queue B3

B1

B2

B3

P

S

= publisher

= subscriber

subscription dest

Matching Engine

Routing Table

+

temperature > 37 B2

temperature > 40 B3

Publications

temperature = 38temperature = 42temperature = 36

PADRES Broker




Broker Internals & Routing Tables

the International CANOE Summer School, Norway, 2009

Queue HandlerQueue Handler

Pre

Pro

cess

or

Post

Pro

cess

or

Forw

ard

er

PADRES Broker

Matcher

Input Queue Output Queues

PRT

SRT

Pub/S

ub M

ess

ages

If sub intersects Send to

[class,=,foo],[attr,>,17]

B2

If pub matches Send to

[class,=,foo],[attr,>,20] B3

B1B2

B3B4

Advertisement Subscription Publication

If sub intersects

Send to


B2


B3

sS1

S1


If sub intersects Send to


B2


B1

S1

S2[class,=,foo],[attr,<,4]

[class,foo],[attr,24][class,foo],[attr,11]

sS1S2

S

S1

S2




Algorithm: Subscription Insertion

Root

class,=,reading

AND

class,=,manifest

level,>,3 content,!=,fertilizer

S1 S2

AND

shipID,=,$X

shipID,=,$X

firm,=,$Y

firm,=,$Y

trust,>,7

class,=,audit

CS

S1: [class,=,reading],[level,>,3]S2: [class,=,manifest],[content,!=,fertilizer]

CS: [class,=,reading],[level,>,3],[shipID,=,$X] AND [class,=,manifest],[content,!=,fertilizer] ,[shipID,=,$X],[firm,=,$Y]AND [class,=,audit],[firm,=,$Y],[trust,>,7]

AND

AND

S1 S2

S3shipID

firm




Algorithm: Event Matching

Root

class,=,reading

AND

class,=,manifest

level,>,3 content,!=,fertilizer

S1 S2

AND

shipID,=,$X

shipID,=,$X

firm,=,$Y

firm,=,$Y

trust,>,7

class,=,audit

CS

[class, reading],[level,6],[shipID,ACME123] [class, manifest],[content,stone],[firm,ACME],[shipID,ACME123] [class, audit], [firm,ACME],[trust,10]

AND

AND

S1 S2

S3shipID

firm




Routing & Matching

Content-based matching

Can emulate topic-based model

Built-in composite subscription processing

Initially based on JESS expert system shell

Replaced by our own Rete-like matching due to limited control over JESS engine




Agenda

PADRES Event & Services Bus

Innovative & Advanced Features Composite Event Detection Historic Event Query Robustness and Resilience Cost-based Event Routing & Detection

Building applications with PADRES





A

B C

D

E

F

CompositeEvents

HistoricAccess

Management

Robustness

LoadBalancing

Security




COMPOSITE EVENT DETECTION




Composite Event Detection Simple CS routing

1

43

2P P

P

AND

S1 S2

AND

S3

6

S

p2 p35

cs

cs’

cs

s1 s2p123

s3

p1




Composite Event Detection

p11

43

2P P

P

AND

S1 S2

AND

S3

6

S Topology-based CS routing

p2 p35

cs

cs’ cs

cs’s1s3

s2p12 p123

cs




HISTORIC EVENT QUERY




Historic Event Query

DB Admin

Publisher

Subscriber

P1 P2

P3 P4 P5

Publication space P(A)

1

3

2

4

6

5P

S

• Partition the publication space• Assign partitions

Partition

1

2

3




Assigning A Partition

Advertisement Subscription Publication

DB Admin

Publisher

Subscriber

[class,DB_CONTROL],[command,STORE],[db,B3-DB],[class,DB_CONTROL],[command,STORE],[db,B3-DB],[partition_spec,`[class,=,reading],[level,>,0]`][partition_spec,`[class,=,reading],[level,>,0]`][class,=,reading],[level,>,0][class,=,reading],[level,>,0]

Partition

1

2

3

4

5

1 1

2

• Partitioning

• Partial replication

• Full replication

1

2

3 4

6

5P

S

[class,DB_CONTROL],[command,STORE],[db,B3-[class,DB_CONTROL],[command,STORE],[db,B3-DB],DB],[partition_spec,” [partition_spec,” [class,=,reading],[level,>,0] [class,=,reading],[level,>,0] ”]”]




ROBUSTNESS, RESILIENCE & COST-BASED ROUTING




PPPP

RobustRobust

Self-healingSelf-healing AdaptiveAdaptive

FlexibleFlexible

Congested Link

General Overlay Network

P S

PP63CANOE Summer School, Norway, 2009



Challenges With General Overlays

• Subscriptions are routed in loops

• Brokers receive duplicate subscriptions

• Subscription copies exacerbate the problem

• Same problem for publications

2

3 4

5

1

6

Adv 1

Adv 2

S X

S

S




TID-based Routing Approach Each advertisement

is assigned a unique tree identifier (TID)

Each subscription has a TID predicate with a variable

2

3 4

5

1

6

Adv 1

Adv 2

XS




Subscription Routing

2

3 4

5

1

6

Adv 1

Adv 2

XS: [class=stock][symbol=*] [TID=$Z]

At Broker 1:

Adv1:

[class=stock][symbol=IBM] [TID=Adv1]

Adv2:

[class=stock][symbol=HP] [TID=Adv2]

S matching Adv1:

[class=stock][symbol=*][TID=Adv1]

S matching Adv2: [class=stock][symbol=*] [TID=Adv2]

SA2SA1

S




Publication Routing

Each publication is assigned the TID of its matching advertisement e.g., p [class, stock][symbol,HP][TID,

adv_msg_id] Publications are routed:

Fixed TID routing: a publication is routed to subscribers along its advertisement tree.

Dynamic publication routing: a publication may be routed to subscribers across advertisement trees.




Fixed TID Routing

Property No broker receives

duplicate publication messages

2

3 4

5

1

6

Adv 1

Adv 2

SubX

P

P




Dynamic Publication Routing Publication’s TID is

changeable Routing heuristic

Util = Routput / Rsending

Property Changing a publication’s

TID while in transit will not change the set of notified subscribers.

2

3 4

5

1

6

Adv 1

Adv 2

SubX

P




Side Effect: Faster Matching Subscriptions are augmented with TIDs only

once at the first broker. Other brokers can route the subscription

based on the TID alone. Similar argument applies to publication

routing.




Evaluation

32 overlay brokers, 20 publishers, 30 subscribers

20 machines & PlanetLab Workload

http://research.msrg.utoronto.ca/Padres/DataSets Yahoo!Finance stock quote traces




Dense vs. Sparser Topologies

20%

4%

Note: The benefit is not proportional to the connection degree.




On PlanetLab




High Publication Rate

stabilized




Publication Burst

Burst

1500msg/min




With Broker Failures

1st failure

2nd failure




Advantages of TID-based Routing Retains the publish/subscribe client interface Speeds up subscription and publication

matching Avoids duplicate subscriptions and

publications Routes publications dynamically across

multiple alternative paths Enables routing around failures, congestion

and load imbalances




COST-BASED EVENT ROUTING & DETECTION




Implications for CS Routing

• CS may be routed according to potential publication traffic, bandwidth, latency, etc.

2

1 3

Adv 1

Adv 2

CS={S1 AND S2}

2

1 3

Adv 1

Adv 2

CS={S1 AND S2}

(a) (b)79CANOE Summer School, Norway, 2009



Topology-based CS Routing

2

3 4

5

1

6Adv 1

Adv 2

7

8

9

CS={{S1 AND S2} ANDS3}CS

S2

S3

S1

CS’

Adv 3

Broker 4 and 8 are the join

point brokers




Dynamic CS Routing (DCSR)

2

3 4

5

1

6Adv 1

Adv 2

7

8

9

CS={{S1 AND S2} ANDS3}CS

S2

S3

S1

CS’

Adv 3


point brokers




Dynamic Join Point Movement

2

3 4

5

1

6Adv 1

Adv 2

7

8

9

CS={{S1 AND S2} ANDS3}

CS

S1

CS’

Adv 3


point brokers




Composite Event Detection




BUILDING APPLICATIONS WITH PADRES




Applications Built with PADRES Semantic data fusion in sensor networks

(DEBS’07) Service selection (DEBS’08) Resource discovery (DEBS’09) Job scheduling (Middleware’04) Business process execution Business activity monitoring SLA monitoring & control (CASCON’08 & ’09) …




sub/advsJob D

sub/advsJob C

sub/advsJob B

Business Process Deployment

PADRES Broker Overlay

sub/advsReply

sub/advsReceive

sub/advsAssign

sub/advsInvoke

sub/advsReceive

1

2

3 46

5

BPELReceive

Assign

Flow

Invoke Wait

Reply

Deployer

Activity Agent

Process Deployer




Distributed BPEL Execution

PADRES Broker Overlay1

3 4

5

BPELReceive

Assign

Flow

Invoke Wait

Reply

2

WS Gateway Agent

6

END

WS Client

Web Service

Web ServiceHTTP/SOAPPub/Sub




Evaluation

Setup Cluster with 20 nodes Different approaches

Centralized Clustered Distributed

Workloads BPEL process with 2 external Web services Parallel: BPEL process with 10 parallel branches Sequential: BPEL process with 2 parallel branches




Changing Request Rate




Sequential vs. Parallel

(a) Sequence Process (b) Parallel Process




Publish/Subscribe

Not just message dissemination

Viable solution for building event processing networks and event-based applications

Besides dissemination and decoupling, pub/sub offers efficient Event filtering Event correlation In-network event processing




Summary: Benefits of Content-based Publish/Subscribe Simplifies IT development and maintenance by

decoupling enterprise components Supports sophisticated interactions among

components using expressive subscription languages – going beyond the limits of topics

Allows fine-grained queries and event management

Achieves scalability with in-network filtering and processing




Conclusions

The right abstraction for event processing is content-based publish/subscribe - IMHO

Event processing & publish/subscribe are interesting research areas.

ToPSS and PADRES explore many aspects of these areas.

http://padres.msrg.utoronto.ca




References The PADRES ESB project home

http://padres.msrg.utoronto.ca An eQoSystem for declarative distributed applications with SLAs

http://research.msrg.utoronto.ca/Eqosystem/ The Micro-ToPSS event processing middleware for sensor networks

http://microToPSS.msrg.utoronto.ca/ Mobile-ToPSS – publish/subscribe for mobile and location-based applications

http://research.msrg.utoronto.ca/Mobile/ ToPSS - the Toronto Publish/Subscribe System Family Portal

http://www.ToPSS.biz (coming soon ) Quantifying events in software to increase modularity & customization in C-

based systems and software-based product lines http://www.AspeCtC.net (ACC - the AspeCt-oriented C compiler)

The Middleware Systems Research Group http://www.msrg.utoronto.ca

My web site http://www.eecg.toronto.edu/~jacobsen

@ the University of Toronto

The DEBS Conferencehttp://www.debs.org

July 2009 at Vanderbilt U.

Everything events Portalhttp://www.event-based.org




Questions?

A D R E SP




Cost Model

input queue

output queue B1

output queue B2

subscription dest

Matching Engine

Routing Table

+

temperature > 37 B1

temperature > 40 B2

Routing cost of CS

RC(CS)) =

+

+

Subscription cardinality |P(S)| : The number of matching publications

per unit of time.

|P(S)| =

|P(CS)| = |P(Sl)| + |P(Sr)| if op = OR




Faster Matching with TIDs

Subscriptions are augmented with TIDs only once at the first broker.

Other brokers can route the subscription based on the TID alone.

Similar argument applies to publication routing.

2

3 4

5

1

6

Adv 1

Adv 2

S X

SAdv1

SAdv2




Sparser Topology




Effect of Subscriber Distance

Distance Fixed(ms) Dynamic(ms) Improvement

6 Hops 47.202 47.568 -0.78%

10 Hops 64.477 52.895 17.96%

12 Hops 74.416 60.598 18.57%

Max Diff 57.65% 27.39%




More Publishers




Publication Burst




More Results

Faster matching 1926 publications cause 16997 times of matching

operations in the “Dense Topology” experiment. About 89% of the matching operations can be saved if we

apply the TID-based faster matching.

Overhead of dynamic publication routing Average CPU usage: 6.3% more than the Fixed routing Average memory usage: 8.9% more than the Fixed routing




CS Routing Traffic




Summary

Minimize redundant traffic induced by cycles Improve scalability and robustness of

pub/sub systems by offering routing path alternatives

Enable flexible composite subscription routing

Simplify solutions for failure recovery and load balancing




Cost Model Routing cost of CS

RCB(CS)) = Σi Tin*|P(CSBi)| + Σi Tmatching*|P(CSBi)| + Σi Tout_i*|P(CS)|

Selection factor (SF) SFA(S) = |σS P(A)| / |P(A)| e.g., SFA(a = v) = 1 / (max - min)

Subscription cardinality |P(S)| = Σi ri * SFAi (S)

|P(Sl)| + |P(Sr)| if op = or

|P(CS)| = min (|P(Sl)| , |P(Sr)|) if op = and

Reteinput

outputs




Fast Matching

1926 publications cause 16997 matching operations in the “Dense Topology” experiment.

89% of the matching operations can be saved with TID-based fast matching.




CS Routing Traffic

1. CS Subscriber

3. Join point broker of topology-based

routing

Msgs

Simple routing

Topology-based routing

DCSR

2. Publisher with a higher

publication rate




Dynamic Join Point Movement

Msgs

DCSR (static)

DCSR (changing)

Join Point Movement

Publisher with a higher publication

rate

CS subscriber


event processing with the padres publish/subscribe system hans-arno jacobsen bell university...

Documents

padres teamcanoe summer

hansarno jacobsencopyright

hansarno jacobsenquerying

hansarno jacobsenwhat

hansarno jacobsentry

hansarno jacobsenamazon

event processing

asynchronous state transitions