how fast data is turned into fast information and timely action - oracle openworld preview amis
DESCRIPTION
Fast data is big data, continuously streaming in, from which information is to be learned in (near) real time. This session demonstrates how Oracle Event Processing is used to analyze live streams of data to find patterns, deviations, and aggregates. The findings are reported in the form of business events that are pushed in live dashboards to Oracle Business Activity Monitoring, which also evaluates business rules on the business events and takes action when required. Examples to be demonstrated in this session include car sensors, website traffic, Twitter feeds, and bank run detection. Oracle SOA Suite 12c, WebSockets, Oracle Application Development Framework (Oracle ADF) active data visualization tools components, and JMS are used to process, forward, and act.TRANSCRIPT
![Page 1: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/1.jpg)
Lucas Jellema
Oracle OpenWorld 2014, San Francisco, CA, USA
How Fast Data Is Turned into Fast
Information and Timely Action
![Page 2: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/2.jpg)
2
![Page 3: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/3.jpg)
Overview
• What is [special about] Fast Data?
– Continuous, Volume|Velocity|Variety, Real Time
• Challenges
– Volatile, non persistent
– Data => Information, Conclusion, Alert, Recommendation, Action
• Strategies
– Smart gathering
– Discard – filter, aggregate, pattern
(and also look for missing events!)
– Promote (process, enrich)
– Visualize
• Technology/Tools
• Demonstration/Cases
![Page 4: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/4.jpg)
4
Fast Data
• Tweet
• Feed
• Beat
• Signal
• Measurement
• Message
• Notification
• Tick
• Pulse
![Page 5: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/5.jpg)
5
New theme (that brings it all together)
![Page 6: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/6.jpg)
6
Some event producing devices
![Page 7: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/7.jpg)
7
![Page 8: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/8.jpg)
Most of these events….
![Page 9: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/9.jpg)
9
![Page 10: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/10.jpg)
10
Fast Data Processing
Fast Data
Smart Processing
• Information
• Conclusion
• Alert
• Recommendation
• Action
![Page 11: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/11.jpg)
11
Fast Data Processing Multi-stage cleansing & aggregation
Fast Data
Smart Processing
• Information
• Conclusion
• Alert
• Recommendation
• Action
![Page 12: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/12.jpg)
12
Typical Flow and Additional Challenge…
Business event
Bu
sin
ess V
alu
e
Data captured
Analysis completed Action taken
Fragmented
event entities
TIME
![Page 13: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/13.jpg)
13
The V-factor
Volume
Velocity
Variety
VALUE
![Page 14: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/14.jpg)
14
Key strategy
• Discard – as early as possible (close to the source)
– Ignore irrelevant events
– Filter out unneeded attributes
– Takes samples instead of entire stream
– Aggregate: merge multiple events into one
![Page 15: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/15.jpg)
15
Fast Data Processing: Oracle Event Processor
Fast Data
Smart Processing
Oracle Event
Processor
RMI
File
REST
HTTP Channel
JMS
Database
Custom (Java)
SOA Suite EDN
Coherence
JMX
QuickFix (financial)
RMI
File
REST
HTTP Channel
JMS
Business Rule
Database
Custom (Java)
SOA Suite EDN
Coherence
JMX
![Page 16: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/16.jpg)
16
Oracle Event Processor
• Light weight, real-time (sub-sub-second), in-memory, continuous query engine
– Available in embedded form – with corresponding licence
• Interacts with many different channels – inbound and outbound
• Has internal caches to enrich events and temporarily retain events
• Uses CQL to:
– Filter, aggregate, enrich and detect patterns (including missing events)
events
Event Processor
![Page 17: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/17.jpg)
Input
Adapter Channel
Input
Adapter Channel
CQL
Processor
CQL
Processor
Channel
CQL
Processor
Channel
Channel
Output
Adapter
Output
Adapter
OSGi Bundle/Spring Application Context
DB
Input adapters connect to data sources Channels help control the flow of data and can be tuned for optimal performance Databases and Coherence caches can be referenced directly in CQL processors CQL processors contain correlation, aggregation and pattern matching business logic Output adapters send data and alerts to downstream systems and business processes
Coherence
Oracle Event Processing Application
![Page 18: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/18.jpg)
18
Fast Data Processing Fusion Middleware Tooling
Fast Data
Smart Processing
Oracle Event
Processor
Coherence
SOA Suite
12c EDN
RMI
File
REST
HTTP Channel
JMS
Database
JMX
Custom (Java)
RMI
File
REST
HTTP Channel
JMS
Business Rule
JMX
Database
Custom (Java)
![Page 19: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/19.jpg)
19
Fast Data Processing Fusion Middleware Tooling
Fast Data Smart Processing
• Information
• Conclusion
• Alert
• Recommendation
• Action OEP
BAM
ADF Coherence SOA
Suite
EDN BPM
Suite
BPEL Task
BI RTD
ODI Golden
Gate
NoSQL
![Page 20: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/20.jpg)
20
Fast Data Example
14,0
16,1
14,1
16,1
16,0
13,1
14,0
16,0
13,1
13,0
14,1
16,0
14,1
13,0
14,1
16,0
13,1
14,0
Smart Processing
Oracle Event
Processor
![Page 21: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/21.jpg)
21
Demonstration: Live Tennis
• Tennis Tournament
• Many matches played in parallel
• The data that is produced:
– At a rate of up to 10 events/minute
Match Id, Player [who scored]
14,0
16,1
14,1
16,1
16,0
13,1
14,0
![Page 22: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/22.jpg)
22
Demonstration: Live Tennis
• The information, conclusions & actions we are looking for:
– Scoreboard per game, set, match
– Match start and completion (action:
inform next players for that court)
– Interrupted match (action: go and check
out the reason for the interruption)
Fast Data
Smart Processing
• Scoreboard
• Match start and
completion
• Interrupted match
![Page 23: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/23.jpg)
23
OEP application to process fast tennis data
• Preparation
– Define event definitions
– Create local, in memory cache with static, enriching data
– Gather (in this case generate) tennis data through adapter
– Create Event Sink to consume all findings and publish to console
TennisMatch
Event
matchId
player
![Page 24: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/24.jpg)
24
Simple Time-slice Aggregation
• Produce aggegrates once every 30 seconds
– Count number of matches going on currently (meaning: in the last 30 seconds)
– Calculate average time per rally (over the last 30 seconds)
– Count total number of points played (over the last 30 seconds)
![Page 25: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/25.jpg)
25
Simple Time-slice Aggregation combined with all-time findings
• Produce aggegrates once every 30 seconds – partially over last 30 seconds and partially over ‘all time’
– Count number of matches going on currently (meaning: in the last 30 seconds)
– Calculate average time per rally and count total number of points played (all time)
– Longest rally played in the tournament
![Page 26: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/26.jpg)
26
Simple Time-slice Aggregation combined with all-time findings
![Page 27: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/27.jpg)
27
Match Level events
![Page 28: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/28.jpg)
28
Rally’s to games
- The first player to have
won more than 4
points
- and have won two or
more points more than
his opponent
TennisMatch
Event
matchId
player
![Page 29: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/29.jpg)
29
Games to Sets
- The first player to have
won more than 5
games
- and have won two or
more games more
than his opponent
![Page 30: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/30.jpg)
30
Detect interrupted matches by ‘finding’ missing events
• When a match is interrupted, obviously no more ‘rally point events’ are produced
• Detecting the absence of these events for a match [that has begun] is equivalent to detecting an interruption of the match
– Unless the match is complete because someone won
![Page 31: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/31.jpg)
31
Detect interrupted matches by ‘finding’ missing events
![Page 32: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/32.jpg)
32
Complete EPN diagram for Tennis Tournament Processor
• A single OEP application that consumes fine grained rally point events and performs three-stage aggregation and enrichment
TennisMatch
Event
matchId
player
New
Match
Match
Finish Interrupted
Match
Set
Won Game
Won
![Page 33: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/33.jpg)
33
Demonstration: Car Parks Management
Fast Data
• Available lots
• Average parking time
• Tow-candidates
(abandoned cars)
Smart
Proces
sing
![Page 34: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/34.jpg)
34
Car Parks Management
Fast Data
• Available lots
• Average parking time
• Tow-candidates
(abandoned cars)
Smart
Proces
sing
Car Entry
- CarParkId
- Licence Plate
Car Exit
- CarParkId
- Licence Plate CarparkCapacity
Alert
- CarParkId
- % full
AbandonedCar
- CarParkId
- Licence Plate
CarStayDone
- CarParkId
- Licence Plate
- Duration
CarParkStatus
- CarParkId
- #cars
- Avg stay
![Page 35: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/35.jpg)
35
Credit Card Theft Detection
• Several situations in the past
– Credit card is stolen in the main terminal building
– Several purchases are made in shops on the way from that area to the main exit • Purchases between $200-$500 dollar
• Purchases made within 5 minutes of each other
• Sometimes the purchases are made in not entirely the direct route to the exit
EXIT
Main
Terminal
![Page 36: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/36.jpg)
36
Credit Card Theft Detection
• Several situations in the past
– Credit card is stolen in the main terminal building
– Several purchases are made in shops on the way from that area to the main exit • Purchases between $200-$500 dollar
• Purchases made within 5 minutes of each other
• Sometimes the purchases are made in not entirely the direct route to the exit
• To catch the perpetrator
– Consume the credit card purchase event stream for airport shops
– Spot situations where three or more purchases of $200-$500 are made within 5
minutes from each other and roughly in the terminal => exit physical order
– Publish an event to alert security staff
• To watch for any further purchases with that credit card
• To inform show staff for that credit card
• To send staff to the exit to try and apprehend the thief
(perhaps based on the shopping bags he is carrying
from the shops he bought stuff at)
![Page 37: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/37.jpg)
37
Catch me if you can
EXIT
Main
Terminal
![Page 38: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/38.jpg)
38
Catch me if you can
EXIT
Main
Terminal
$440
$300
$380
$250
![Page 39: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/39.jpg)
39
Toilet Cleanliness
• Every toilet facility has a customer satisfaction station
– Every 30 seconds, someone can indicate (1-4) their satisfaction
• All entries are collected – signals (toiletId, rating)
• When the rating < 3.3 (average over last 5 signals) – the cleaning staff has to act
• When the rating < 3 and the
• previous signal for that toilet is longer ago than 5 minutes, then also action is required
![Page 40: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/40.jpg)
40
Human consumers
• Slow at data processing
• Not electronically connected
• Visually oriented (1 picture > 1000 words)
• Frequently (though perhaps decreasingly so) the actor or decision maker
• Interact along human communication channels
• Use visualization to present findings, conclusions, recommended actions
– And as a second tier of fast data processing:
Highlight (filter), aggregate, patterns, extrapolate/interpolate, missing elements
• Sometimes take over from humans and just take action
![Page 41: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/41.jpg)
41
Audience Challenge
![Page 42: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/42.jpg)
42
Audience Challenge – 1/2
![Page 43: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/43.jpg)
43
Audience Challenge – 2/2
![Page 44: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/44.jpg)
44
Visualize and Aggregate
![Page 45: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/45.jpg)
45
Trends and Extrapolation
![Page 46: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/46.jpg)
JMS
HTTP
JMX
File
DB
RSS
EDN
…
OEP
SOA Suite
BAM
E
D
N
JMS
@
Alerts
(rules)
Reports
HTTP
Consumers
Web
Application
The OEP to Human interface
![Page 47: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/47.jpg)
47
Real Time – from Event to Task OEP => SOA Suite 12c EDN
Fast Data
Smart Processing
Oracle Event
Processor
SOA Suite 12c
EDN
BPEL Task
BPMN
Medi-
ator
event event event
![Page 48: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/48.jpg)
48
Real Time – from event to UI
Fast Data
Smart Processing
Oracle Event
Processor
WebLogic
JMS
event
msg
WebSocket
Server msg
msg
![Page 49: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/49.jpg)
49
Real Time – from event to UI Business Activity Monitoring
Fast Data
Smart Processing
Oracle Event
Processor
WebLogic
JMS
event
msg BAM msg
![Page 50: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/50.jpg)
50
Summary
• Fast Data (events): Vast, Continuous, Velocity, Variety
– Wanted: Near real time conclusions, recommendations, alerts, actions
• Strategy:
– Discard – as early as possible (Filter, Aggregate)
– Enrich, Pattern Match, Missing Events, Retain, Publish higher level, more coarse
grained business event
– Repeat this cycle multiple times (such as rally point, game, set, match)
• Technology for Fast Data processing: Oracle Event Processor & CQL
– Interacts with JMS, EDN, RMI, HTTP (/REST), JMX, Database, Coherence
• To assist humans in Fast Data and Information Processing: Visualization
– Filter, Aggregate, Enrich, Pattern Match (1 picture > 1000 words)
– Technology: BAM (Dashboard and Rule processing), ADF Data Visualization
– Also: turn findings into actions using Human Task, BPEL and BPM via the SOA Suite
12c Event Delivery Network (EDN)
![Page 51: How fast data is turned into fast information and timely action - Oracle OpenWorld Preview AMIS](https://reader034.vdocuments.net/reader034/viewer/2022051609/5461e9f0af7959477b8b4ca6/html5/thumbnails/51.jpg)