MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

Big Events

Hans-Arno JacobsenMiddleware Systems Research GroupMSRG.org

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

Big Event Data

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Traditional Big Data Domain vs. Rest of Universe

• There are other emerging domains with needs similar to Big Data– Smart grids– Smart cities …

My first message: There are other relevant Big

Data domains – beware!

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Smart Grids for Taming The Energy Problem

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Relevance of Smart Grids

• Increasing penetration of variable renewable energy sources like wind and solar et al.

• Paradigm shift from demand-following supply to supply-following demand

• Need for new large-scale information system infrastructure to control demand

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Distributed Generation, Flexible Loads and Energy Storage

• Come in big numbers• Show unique behavior (users, weather, equipment, …)• Have to be monitored and controlled

Big event data challenge

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Solar Photovoltaic Power Generation

• High frequency measurements required

• Several metrics of interest, many spatially distributed measurement points

1 127 253 379 505 631 757 883 1009113512611387-200

0

200

400

600

800

1000

1200

direct normal solar irradiance

minutes

watt

s/m

^2

1 121 241 361 481 601 721 841 961 10811201132105

10152025303540

air temperature

minutes

degr

ees c

elsiu

s

1 124 247 370 493 616 739 862 985 1108123113540

50

100

150

200

250

diffuse solar irradiance

minutes

watt

s/m

^2

Source: National Oceanic & Atmospheric Administration (U.S.)

~2.3 TB per year and 1k panels

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Use of PEVs as Grid Resource

-0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 0.6

-0.2

-0.15

-0.1

-0.05

0

0.05

GPS coordinates

Trip 1 Trip 2 Trip 3 Trip 4 Trip 5 Trip 6

delta longitude

dlet

a la

titud

e

• High frequency measurements required

• Important for SG applications: Continuous update of trip destination and energy level at destination

Source: Auto21 Project, University of Winnipeg

~ 0.5 TB per year and 1k vehicles

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Electric Power Consumption

0:00:00 3:39:00 7:18:00 10:57:00 14:36:00 18:15:00 21:54:00012345678

real power

time

kwatt

s

0:00:00 3:30:00 7:00:00 10:30:0014:00:0017:30:0021:00:000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

reactive power

time

kVAR

0:00:00 3:27:00 6:54:00 10:21:0013:48:0017:15:0020:42:00215220225230235240245250255

voltage

Time

Volts

• Very high frequency measurements required (e.g., for inferring device on/off events, grid stability, etc.)

• Several metrics of interest (household electricity meters, single devices, etc.)

Source: UCI Machine Learning Repository

~ 27.5 PB per year and 1k homes

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Traditional Big Data Domain vs. Rest of Universe

My second message: Detecting events in real-

time in the sea of Big Data is just as

important.

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORGTowards Big Events

• Many non-traditional scenarios that require filtering of Big Events at large scales

• … scenarios that require filtering & storage of events at large scales

• Filtering & storage of “event streams”

• Filtering & storage of “event showers”H.-A. Jacobsen

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORGEvent Showers vs. Event Streams

Event Showers• Partially ordered sets of events• No single event schema • Events vary in shape and size

from one to the next• Processing of many event

expressions• Tends to require support for

aggregation• Broader model & paradigm

(dissemination, matching, coordination)

Event Stream Processing• Linearly ordered event

sequences• Schema-based, single schema

per stream• Stream tuples follow schema• More single-expression

processing-based• Aggregation is a key requirement• Focused on processing

queries/expressions over event streams

H.-A. Jacobsen

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Conclusions

• Big Events are Big Data in motion

• Processing Big Data in real-time to detect events of interest is important as well

• There are other emerging application domains; let us watch out for themMy final message: Big

Data Benchmarking efforts should take this

into account.

MIDDLEWARE SYSTEMSRESEARCH GROUP

MSRG.ORG

H.-A. Jacobsen

Acknowledgements

• C. Goebel for help with smart grid slides