composite subscriptions in content-based pub/sub systems

University of Toronto

Composite Subscriptions in Content-based Pub/Sub

Systems

Guoli Li

23/4/19Middleware 2005 Grenoble France2

Agenda

Motivation PADRES System Composite Subscriptions Case Study: Workflow Management Evaluations Conclusions and Future Work


Motivation

A large number of atomic events Lack of event correlation Potential Applications:

– Workflow processing and business process execution

– Network management– …


The PADRES System

PADRES: Publish/Subscribe Applied to Distributed REsource Scheduling

Features:– Predicate-based subscription language

(attribute, op, value) pair Extended to supporting composite subscriptions

– Rule-based publication/subscription matching Publications are mapped to facts Subscriptions are mapped to rules

– Advertisement-based routing


Content-based Routing

Advertising

Publisher

Publisher

…PADRES

Distributed Overlay

Broker Network Subscriber

Subscriber

…

Broker Broker

BrokerBroker

Broker

Advertisement



Subscribing

Publisher

Publisher

…PADRES

Distributed Overlay


Subscriber

…

Broker Broker

BrokerBroker

Broker

Subscription



Publishing

Publisher

Publisher

…PADRES

Distributed Overlay


Subscriber

…

Broker Broker

BrokerBroker

Broker

Publication


Several Concepts

Message– Advertisement– Subscription– Publication

Event– Atomic event: publication in pub/sub

literature– Composite event: a pattern of events

Subscriptions– Atomic subscription: atomic event– Composite subscription: composite event


Composite Subscriptions

Composite subscription consists of primitive subscriptions– Provide a higher level view for subscribers– Save network bandwidth

Subscription language features– Operators: AND (&), OR (||)– Variables: ($x)– Advanced event patterns

&

||

S1 S2

||

S3 S4

&

S5

CS={{S1 OR S2} AND{S3 OR S4} AND S5}



Parallelization pattern: s1 & s2

Alternation pattern: s1 || s2

Sequence pattern: (s1;[timespan:ts]s2)[within:wi]

Repetition pattern: Repetition(S, n, attr, v)

e.g. {rule ((job (appl = $X) (name = A) (state = succ)) & (job (appl = $X) (name = B) (state = succ))) => (forward a notification to proper destination)}



Parallelization pattern: s1 & s2

Alternation pattern: s1 || s2

Sequence pattern: (s1;[timespan:ts]s2)[within:wi]

Repetition pattern: Repetition(S, n, attr, v)

e.g. {rule ((job …. (name = A) (time = $Y)) & (job …. (name = B) (time > $Y+ts) (time < $Y+wi))) => (forward a notification to proper destination)}e.g. {rule ((job …. (name = A) (attr = $Z)) & (job …. (name = A) (attr = $Z+v)) & … & (job …. (name = A) (attr = $Z+(n-1)v)) ) => (forward a notification to proper destination)}


Composite Subscription Routing

Distributed Overlay Broker

NetworkB4B3

&

||

S1 S2

S3

S

P2

P3

CS={{S1 AND S2} ANDS3}

B2

B1

B6

B5

P1CS

CS’={S1 AND S2}

P Publishers

S Subscribers

CS’S3S2

S1


Composite Event Detection


NetworkB4B3

AND

AND

S1 S2

S3

S

P2

P3

CS={{S1 AND S2} ANDS3}

B2

B1

B6

B5

P1

CS

CS’={S1 AND S2}

P Publishers

S Subscribers

CS’

S3S2

S1 CS

P1

P2

P12

P3

P123


Advantages

Rule-based approach naturally enables the composite subscription

Event detection results can be shared Network traffic is reduced

– Composite subscription is forwarded into the network as far as possible before it is split

– Composite events are detected close to the data source and not widely disseminated


Case Study: Workflow Management

Decentralized Workflow Management based on Pub/Sub

Functions– Workflow transform– Workflow deployment– Workflow execution– Workflow control and monitoring


Architecture

Server Farm

Computers

ComputersDatabase

Laptops

Computers

Workstation

Laptops

DatabaseServer

ServerSwitch

Server

Switch

Computing, Storage, and Networking Resources


Content-based RouterClients (publisher/subscriber)

Workflow and Business Process Execution

Deploy Control UpdateVisualize

Monitor ...

6

43

7start halt

Workflow Management and Business Activity Monitoring

Redirectresume

addremove

Event ManagementFramework

BusinessEvents

Business ProcessEvents

Business ProcessExecution Events

Network andSystem Events


Workflow Descriptions

Workflows are described using XML-based languages, such as BPEL

Activities defined in a workflow have dependencies among each other

A workflow instance is generated by a trigger

Failures can be handled in the workflow description

A

B C

D

trigger

E

F


Workflow Transform

The first job(s) subscribe to trigger messages e.g. Job A subscribes to trigger publication: {[class,eq,trigger],

[workflow,eq,payroll], [instanceID,=,$x]}

Job dependencies are implied by subscriptions e.g. Job E subscribes to job A failed: {[class,eq,job_info],[workflow,eq,payroll], [nstanceID,=,$x], [job,eq,A],

[status,eq,fail]}

e.g. Job D subscribes to B and C finished, which is a composite subscription

Advertisements are generated from a workflow specification file


sub/advsJob D

sub/advsJob C

sub/advsJob B

Workflow Deployment

A

B C

D

BPEL


NetworkB4

B2

B3

B1

Job C Agent

Job D Agent

Job A Agent

Job B Agent

B6

Deployer

B5

sub/advsJob C

sub/advsJob A

sub/advsJob B

sub/advsJob D

sub/advsJob A


Workflow Execution (Success)

A

B C

D

BPEL


NetworkB4

B2

B3

B1

Job C Agent

Job D Agent

Job A Agent

Job B Agent

B6

Workflow Manager

B5

sub/advsJob C

sub/advsJob A sub/advs

Job B

sub/advsJob D

trigger

A

A

B C

A

BC

END


Workflow Execution (Failure)


NetworkB4

B2

B3

B1 Job D Agent

Job A Agent

Job B Agent

B6

Workflow Manager

B5

sub/advsJob C

sub/advsJob A

sub/advsJob B

sub/advsJob D

trigger

Job E Agent

sub/advsJob E

Job C Agent

Job F Agent

sub/advsJob F

A

B C

D

BPEL

E

F

A

E

END


Workflow Monitoring

Monitor is a Pub/Sub client Monitor/control workflow execution

status– Workflow level– Activity level

Support dynamic workflow modification Visualize topology of broker overlay

network Visualize message routing


Evaluations

Environment– Intel Xeon 3GHz, 1GB RAM– JDK 1.4.2

Workload description– Predicates based on 20 attributes with random

operators– Values are uniformly distributed in the value range– Two workflows with different number of jobs

Metrics– Matching time– Routing delay per composite subscription– Network traffic


Publication Matching Time

0.001

0.01

0.1

1

10

100

1000

0 50,000 100,000 150,000 200,000

Number of Subscriptions

Rou

ting

Tim

e (m

s)(l

og s

cale

)

Naive Matching AlgorithmPredicate Counting

PADRES Broker with Random DataPADRES Broker with Zipf Data


Composite Event Detection

0

50

100

150

200

250

300

350

400

450

500

Number of Composite Subscriptions

Num

ber

of M

atch

ing

Pub

lica

tion

s

0.1

1

10

100

Com

posi

te S

ubsc

ript

ion

Mat

chin

g T

ime(

ms)

Number of Matched Pubs (Uniformly Distr.)

Number of Matched Pubs (Zipf Distr.)

Matching Time (Uniformly Distr.)

Matching Time (Zipf Distr.)


Routing Delay

0

2

4

6

8

10

12

14

1 2 3 4 5 6

Number of Atomic Subscriptions per Composite Subscription

Rou

ting

Del

ay (

ms)

a


Number of Notifications

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

8,000

0 10,000 20,000 30,000 40,000 50,000

Number of Publications

Num

ber

of N

otif

icat

ions

Without Composite Subscription

Composite Subscription


Workflow Traffic

0

500

1000

1500

2000

2500

3000

3500

Network Traffic(KB)

W/t Composite Sub. 1582 3264

Composite Sub. 1084 2026

Workflow A (8 jobs) Workflow B (24 jobs)


Related Work

P.R. Pietzuch, B. Shand, and J. Bacon. Composite event detection as a generic middleware extension. IEEE Network Magazine, Special Issue on Middleware Technologies for Future Communication Networks, Jan 2004

SIENA REBECA


Conclusions

An expressive subscription language A rule-based approach of message

matching and composite event detection Case study: a decentralized workflow

management system based on PADRES Future Works

– Support BPEL4WS– QoS based workflow execution


Questions?

composite subscriptions in content-based pub/sub systems

Documents