Summer School: Events, Publish/Subscribe & Systems

Hans-Arno Jacobsen

On Behalf of the Organizers:Frank Eliassen & Roman Vitenberg

http://canoe.ifi.uio.no/index.php/about-summer-school

VELKOMMEN

For Your Information

• Internet access• Breakfast, lunch, dinner and breaks– 7 AM – 9 AM (Breakfast)– 12:30 (Lunch everyday including Friday)– 7:30 - 9 Monday– 6:30 - 8 Tuesday & Thursday

• Viking Experience: Monday 6 PM at the reception• Wednesday afternoon boat trip (dinner included)– Leave the hotel at 3:45– Leave in front of the city hall in Oslo at 9:15

Around the Sundvolden Hotel

Quick Introductions

• In one minute, please tell us– Who you are– From where you are– Your main research interests– What you expect from the week– Where you heard about the Summer School

(DBWORLD, EPTS, TCCC, etc.)

CANOE

• Canada-Norway Partnership in Higher Education 2008-2011– http://canoe.ifi.uio.no/

• Collaborate in research and teaching on – Distributed systems, networking, and event

processing

• Jointly organize workshops, summer schools, student exchanges, and research visits

Summer School Objectives

• Learn about events, publish/subscribe and systems

• Interact on the above topics in discussions, brainstorming sessions, and poster session

• Network and have fun

Collaborate & Provide Feedback• Collaborate Wiki– http://debs.msrg.utoronto.ca/collaborate– Login via …

• Anonymous feedback form– Feedback URL– Please use as much and as often as you’d like– Many of the questions may only apply towards the

end of the week– Session specific forms available via the Collaborate

Wiki (login required)

THE PROGRAM

Organization & Structure

• Lectures and tutorials– Listen, learn and ask

• Panel, discussion and Q/A session– Ask and participate

• Break-out sessions and social events– Engage, collaborate, and contribute

Events, Publish/Subscribe & Systems

• Events– Are the problem

• Publish/Subscribe, Streams, Rule Engines, Event Processing– Are the abstractions & the paradigms towards a

solution

• Systems– Are the tools to build viable solutions

Applications

A

B C

D

E

F

RFID and sensor networksService oriented architecture

Workflows, business processes and job scheduling

Supply chain and logistics

Event-Based

Light on

Callback received Razor SKU

Transformed

Fault

Temperature high

Loan service invoked

Ordered

Delivered

ShippedJob A done

Triggered

Other Applications• Algorithmic & automated trading• Asset tracking• Audit and compliance checking• Automated security pricing• Business activity monitoring

(BAM)• Business intelligence (BI)• Business process execution• Business activity monitoring• Capital marker surveillance• Content distribution (stock)• Fraud detection and prevention

• Intrusion detection• Information dissemination &

filtering• Location-based services• Luggage handling at airports• Network management• Order management and

advanced pricing• Operational risk assessment• RFID-based systems• Resource management• Sensor networks• System monitoring and

diagnostics

Extracted from a series of use case presentations at EPTS meetings in 2007-present.

(alphabetical order)

Observations

Events in many verticals• Automotive• Banking & Finance• Insurance• Health-care• Transportation• Military• …

Core functions required• Monitor and observe• Detect• Filter• Correlate• Infer and predict• Disseminate

Event Processing Initiatives• ACM Distributed Event-based Systems Conference (DEBS)

– http://www.debs.org/2009

• Event-based Research Portal– http://www.event-based.org

• Complex Event Processing– http://complexevents.com/

• Event Processing Technical Society (EPTS)– http://www.ep-ts.com

OUR PROGRAM

Our Program

Monday• Towards a discipline of “smart” systems by

Mani Chandy, Caltech• Pub/Sub at Google by John Reumann, GoogleTuesday• Discussion & Q/A with John Reumann• Commercial Trends in Event Processing by

Paul Vincent, TIBCO

Our Program

Tuesday (cont.’d)• Poster presentations with all of us• Event Processing with the PADRES Pub/Sub

System by Hans-Arno Jacobsen, University of Toronto

• Break-out sessions with all of us

Our Program

Wednesday• High-performance Event Stream Processing

Infrastructures and Applications by Bugra Gedik, IBM

• Real-time Data Distribution by Angelo Corsaro, PrismTech

• The Next Generation Application Server – How Event-based Processing Yields Scalability by Guy Korland, Gigaspaces

• Field Trip

Our Program

Thursday• Content Routing and Filtering in the Large Scale,

Pascal Felber, University of Neuchatel• Discussions and Reflections with Mani Chandy• Mohammed and the Mountain: Distributing the

Data or Moving the Data by Eric Jul, University of Copenhagen

• Panel: Streams vs. Rules vs. Subscriptions by Bugra Gedik

• Break-out sessions

Our Program

Friday• Scalable Hostting of Web Applications by

Pierre Guillaume, Free University Amsterdam• Break-out Session Summaries & Results with

all of us

POSTER SESSION

Poster Session: Tuesday @ 2-3:30

• Abstracts are posted on our web page• Please set up just after lunch (around 1:45)• We may use walls with hotel provided tape in

the conference room• We start the session with one minute pitches

by authors (alphabetical order)• Please line-up at the front (see web site)• Pitches are performed without slides

BREAK-OUT SESSIONS

Break-outs Organization• Meet in small groups during two slots throughout the

week (evening slots on Tuesday and Thursday)

• Adjourn in plenum on Friday to present discussion and results

• Each group is given some guiding questions, but may define their own

• Groups should be self-organizing, i.e., designate a scribe and a presenter for Friday

http://debs.msrg.utoronto.ca/collaborate

Suggested Discussion Groups

• Simulation & evaluation• Applications• Research agenda & road map• Marketing & advocacy• Practical limitations• Collaborations• Interoperability• Management issues

Others ?

We will discuss the details in the plenum on Tuesdayevening at 8 PM, followed by the group discussions.

We would like tocharter at least

6 groups.

DISCUSSIONS

Discussion Sessions • Tuesday morning with John Reumann about

Events, Pub/Sub and Systems Research Challenges at Google

• Thursday morning with Mani Chandy as Discussion and Reflections about What we Heard and Learned so far

• Thursday afternoon with Bugra Gedic as panel discussion on Streams vs. Rules vs. Subscriptions

Fundamental Questions

• What abstractions support these core functions?• What models, formalisms techniques, algorithms

underlie these abstractions & core functions?• How are these abstractions used? • What abstractions can best be used to build event

processing requiring applications?– Databases? – Messaging middleware? – Transaction processing systems?– ...

• Is there a need for developing an Event Management System?

In analogy to a Database Management System for data.

My Advice

• A Summer School is not a conference

• Please ask questions, any questions

• We want to learn from each other

• Bring yourself in

What is an event?

• Etymology

– Latin: eventus;

• evenire to happen;

• e- + venire to come

Various Interpretations• A postulated outcome, something that follows as a

result or consequence, condition, or eventuality; “in the event that I am not there, you may …”

• Something that happens, a noteworthy happening, an occurrence

• Any of the contests in a program of sports, a social occasion or activity, a world event

• The fundamental entity of observed physical reality represented by a point designated by three coordinates of place and one of time in the space-time continuum postulated by the theory of relativity

• A subset of the possible outcomes of an experiment (probabilities)

Event – My Working Definition• An event is an instantaneous and asynchronous state transition in

the environment of interest to applications, systems, users.• Event versus event occurrence

– Like type or class versus instance• In my definition events do not extend over time

– Interval events are defined by starting and ending events (a.k.a. composite events)

• My definition does not capture world events– Maybe important for a theory of events, but less so for the

effective processing of events• My definition does not capture interval events

– Events associated with a time span, but those can be modeled as composite events

Synonymously Used

The terms• Event• Event occurrence• (Event) message• Event objectare often used synonymously in practice.• Also the pattern for detecting an event is

often simply referred to as event.

Similarly, …

• Event source• Event producer• Event generation• Event publisher

• Sink• Consumer• Capture• Subscriber

Event Processing• Event Processing is an emerging area and discipline that aims

to define and develop– Abstractions– Architectures– Systems – Languages– Patterns– Models– Standards

for the processing of events in systems, applications, and by end users.

A Note on CEP

• Complex Event Processing (CEP)• Event processing must not be complex– Or else, a marketing nightmare, I suspect

• Meant was really Complex-event Processing• Not the complex processing of events

• Let’s just keep to Event Processing and try to make it as simple as possible,

A.k.a.

• Event Processing• Complex Event Processing & CEP• Business Event Processing• Intelligent Event Processing• Event Stream Processing

Marketing Event Processing & CEP are widely used, as opposed to some of the other terms. I am sure there will be more.

Event Processing Models (1970-present)

• Rule-based event processing• Active databases• Publish/Subscribe• Event stream processing (ESP)• Programming language embedded & library

Event Processing Languages

• Rules • ECA rules • Database triggers • Subscriptions• XPath• SQL with stream processing extensions• Graphical notations

Emerging StandardsDissemination & Filtering• OMG DDS• WS Notifications• WS Eventing• INFO-D• AMQP• CORBA Events &

NotificationsAPIs & Formats• CEI & CBE• WSDM & WEF

Languages• StreamSQL• OMG Production Rule

Representation (PRR) and W3C RIF

• OMG Event Metamodel and Profile (EMP)

EP Initiatives• ACM Distributed Event-based Systems Conference (DEBS)

– http://www.debs.org/2009

• Event-based Research Portal– http://www.event-based.org

• Complex Event Processing– http://complexevents.com/

• Event Processing Technical Society (EPTS)– http://www.ep-ts.com

Events in Industry I

• Mostly pre-standardization today• Pre-competitive collaborations• Efforts revolve around activities of EPTS– Gathering of use cases & requirements– Definition of glossary (vocabulary)– Reference architecture– Event processing meta model– Event processing language

• In some related areas standardization started

Events in Industry II

• Predominant trends– Stream-based processing of events through

extensions of SQL• Most dominant database vendors

– Rule-based processing of events through various rule formalisms• Many traditional middleware vendors

– Programming language embedded– Scripted and proprietary

There is no one dominant approach today.

Events in Industry III• Relational algebra, SQL, and SQL extensions– EP uses and builds on stream processing efforts– Industry adoption of SQL is well known– Industry adoption of SQL extensions is not clear at this

state; no de facto standard yet either• Rules– Re-emerging interest in rule-based processing– Intense recent efforts in standardizing

• Flexible rule markup languages• Rule interchange formats

– Industry adoption of standards is not clear at this stage

Predication: We will see many more language dialects until in a few years timea de facto standard emerges. Most likely the approach taken by the surviving ordominant player.

Events in Industry IV

• Efforts underway at (alphabetical)– IBM, Oracle (& BEA), Tibco, – and many more, sorry for omissions

• Start-ups (alphabetical)– Coral8, EventZero, StreamBase– and many more, sorry for omissions

There was/is intense activity – let’s hope recent world events don’t interfere.

Events in Academia I

• Research on events in various sub-areas of computer science and others– Databases

• ECA rules & trigger management • Stream processing• Publish/Subscribe

– Networking• Event correlation and network management

– Systems• Publish/Subscribe• SEDA (staged event-driven architectures)

Events in Academia II

• Distributed systems– Event distribution and

dissemination– Content-based routing– Publish/Subscribe

• Software engineering– Publish/Subscribe

• Embedded systems– Formalization of event-driven

systems– Event-driven programming

• Programming languages– Programming with events– Events and types

• Philosophy– Notion of events

• Events are broadly recognized.

• There is no unifying theory of events.

Events in Academia III

• The DEBS Conference aims to establish a forum that brings together disparate research revolving around events.

• Started in 2002 as workshop until 2006• From 2007 until present an independent

conference.• Obtained ACM sponsorship in 2009.

ACM International Conference on Distributed Event-based Systems.

Data Management versus Event Processing

Data management• Relational algebra & theory• SQL• Started decades ago• 35+ year strong research

community (VLDB, SIGMOD, ICDE et al.)

• Industry community• Several key players and many

players catering to niches• Many textbooks

Event processing• Requires a foundation & theory• … a common language• Recently started• Young and vibrant research

community that’s growing (DEBS)

• EPTS• Interest from key players• Many small emerging players• Essentially none (emerging)

Summary

• Event processing cuts across domains

• Event processing is not new

• New are the concentrated efforts in industry and in academia to develop event processing systems

• Event processing as sub-area of computer science is young and needs time to mature