epts survey results

epts event processing technical society

eptsevent processing technical society

Pedro Bizarro on behalf of theUse Case Working Group

Results of the Survey on Event Processing Use CasesMarch 24, 2011


2

How are event processing technologiesbeing used?

Classify scenarios,help othersselect solutions

Inspire use ofevent processing

Food for thought forresearchers & engineers


3

Earlier versions

• 2007: Kick-off – Problem: world of superficial use cases

• 2008: v1: 6 use cases, 54-questions (@4th epts)

• 2009: v2: 5 use cases 9 lessons learned (@debs2009)

• 2011: v3: 30 use cases plenty of statistics (today!)


4

24 questions~13 minutes

30 use casesEPTS, DEBS

community


Large Variety of Use Cases

5

GridDocument workflow

Patient discharge

Revenues and expensesETL in Telcos

NYC transportation

Content authoring

Gas station networks

Emergency management

Home energy

Testing algorithms…


6

Banking/Financial

Energy/Utilities

Aerospace/Defense

Education

Telecom

Transportation/Logistics

Other

0% 5% 10% 15% 20% 25%25%

18%

11%

7%

7%

7%

25%

Industry background

Computer SoftwareHealthcareManufacturingRetail and distributionetc<4% each


7

Functional area

Operations

Research and Development

Finance

IT Support / Administration

Customer Service and Support

Human Resources

Sales

Marketing

Other

0% 5% 10% 15% 20% 25% 30% 35%

33%

18%

12%

12%

9%

6%

3%

0%

6%


8

Maturity level

A concept

Prototyping - any stage

Pilot

In developement

In production

0% 5% 10% 15% 20% 25% 30% 35%

4%

17%

26%

22%

30%

52%


9

Primary project drivers

Enhance/improve user services

Add new features

Reduce production cost

Faster user service

Reduce deployment cost

Compliance

Other

0% 5% 10% 15% 20% 25% 30% 35% 40%

35%

16%

14%

11%

8%

5%

11%


10

Data sources

Databases

Files

ESB, JMS, other msg protocols

RSS, twitter, stock, news, other feeds

Application or OS events

Video, sound, multimedia streams

Other

0% 5% 10% 15% 20% 25% 30% 35%

31%

18%

18%

10%

5%

0%

18%

Non“streaming”

“streaming”


11

Destinations and actions

Msg people (E-Mail, SMS, MMS)

Trigger other applications

Databases

Dashboards

ESB, JMS, other msg protocols.

RSS, twitter, stock, news, other feeds

Other

0% 5% 10% 15% 20% 25% 30%

24%

24%

20%

17%

7%

4%

4%

someautomation

but peoplestill stronglyin the loop


12

Desired features

Correlations or joins

Summaries or aggregations

Sliding windows

Tumbling windows

Forecasting and predictive functions

Probability or statistical models

Other

0% 5% 10% 15% 20% 25% 30%

24%

24%

17%

16%

9%

9%

2%

simplerbeats

complex


13

Data models and data types

Relational

Column-Oriented

Generic XML

Objects

RDF

XML (industry specific schema)

Extensibility

Other

0% 5% 10% 15% 20% 25% 30% 35%

32%

18%

12%

9%

9%

9%

3%

9%


14

Performance – input events per second

NONE

'1-10

'10-100

'100-1.000

'10.000-100.000

'100.000-1.000.000

1.000.000 or more

No answer

0% 5% 10% 15% 20% 25% 30% 35% 40%

5%

11%

16%

37%

11%

5%

0%

16%

Not that much!


15

Number of data sources

'1-10

'10-100

'100-1.000

'10.000-100.000

'100.000-1.000.000

1.000.000 or more

No answer

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

42%

26%

11%

0%

5%

0%

16%

Or maybe not! Typo?!

Data comes fromfew sources


16

Number of “AI” models

NONE

'1-10

'10-100

'100-1.000

'10.000-100.000

'100.000-1.000.000

1.000.000 or more

No answer

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

47%

16%

11%

0%

0%

0%

0%

26%

3 in 4 don’t answeror don’t use


17

NONE

'1-10

'10-100

'100-1.000

'1.000-10.000

'10.000-more

No answer

0% 5% 10% 15% 20% 25% 30% 35% 40%

5%

11%

16%

11%

5%

16%

37%

Expected data size growth rate/year (in MB)

or not growing much

Hard to forecast

47% increase lessthan 10Gb/year


18

Why did we use log scales?

Because we didn’t knowwhat to expect


19

Enterprise capabilities

High availability

Online system maintenance

Provenance

Fault tolerance

Recovery of any data or meta-data

Security

Encryption

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

39%

24%

24%

21%

21%

12%

0%

Cannot stop!

Not a big concern


20

Implementation constraints

Specific CEP engine

Specific file or database system

Specific operating system

Specific application server

Specific development environment

Specific hardware

Other

0% 5% 10% 15% 20% 25% 30%

27%

19%

15%

12%

12%

8%

8%

surprise


21

The typical use case

• In production to improve banking/utilities user services

• Gets data from databases, files and message queues

• Notifies people and other applications

• Does correlations, joins and aggregations

• Handles less than 1000 events/second (<10 sources)

• Must run a specific CEP engine

• Cannot stop


22

The surprises (personal take)

• Few telcos, transportation and logistics

• Not from IT department

• Not about lower TCO, faster deployments

• “Non-streaming” dominate

• Low throughput, few sources, little growth

• Not about forecasting, predictions

• Very few AI models

• Not about security, encryption


23

Members and acknowledgements

• Pedro Bizarro <[email protected]>

• Christoph Emmersberger <[email protected]>

• Thomas Ertlmaier <[email protected]>

• Matthew Cooper <[email protected]>

• Tina Groves <[email protected]>

• Dieter Gawlick <[email protected]>

• Brian Connell <[email protected]>


eptsevent processing technical society

Q&A?

24

epts survey results

Documents