introduction to oracle complex event processing
Post on 26-Mar-2015
150 Views
Preview:
TRANSCRIPT
<Insert Picture Here>
Introduction to Oracle EDA Suite
Kemal Ülkü, kemal.ulku@oracle.comSales Consultancy Team Leader – Oracle Fusion Middleware
Event-Driven Applications
Financial Services
Transportation & Logistics
Public Sector & Military
Manufacturing
Insurance
Telecommunications & Services
Algorithmic trading
Asset management
Distributed order orchestration
‘Negative Working Capital’ inventory management
Reponses to calamities –earthquake, flooding
• Intrusion detection systems• Military asset allocation
Need to support one or more of:
• High volume
• Continuous streaming
• Sub-millisecond latency
• Disparate sources
• Time window processing
• Complex pattern matching
Oracle EDA SuiteComplete Suite for Industrial Event Processing
Routing Transform
Oracle Service BusOracle Service Bus
Mediation
REAL-TIMEVISIBILITY
& PROCESSING
AlertsBusiness Monitoring
BAMBAM
Event FeedsData Streams
Java-based EventServer
CEPCEP
Socket
AdaptersAdapters
DB FileCONNECTIVITY
ROUTING & DATA
SERVICES
Business Rules
J2EE Application Server(Oracle AS, WebLogic, WebSphere, JBoss)
Industry SpecificJMS
CoherenceJRockit VM JRockit RTENTERPRISE PERFORMANCE & SCALABILITY
Apps
AdaptersAdapters B2BB2B
PartnersDB Legacy
CONNECTIVITY
Routing Transform DataQuality
ETL &Replication
Data IntegratorData IntegratorOracle Service BusOracle Service Bus
Mediation
ROUTING & DATA SERVICES
NativeBPEL
Business Rules
Human Workflow
BPEL Process ManagerBPEL Process Manager
ORCHESTRATION
Coherence CacheJ2EE Application Server
(Oracle AS, WebLogic, WebSphere, JBoss)Messaging
Oracle SOA and Event Technologies
GOVERNANCESystem
Monitoring
Enterprise Enterprise ManagerManager
UDDI
WS PoliciesSecurity
Web Services Web Services ManagerManager
RegistryRegistry
GOVERNANCE
SOA lifecyclegovernance
EnterpriseEnterpriseRepositoryRepository
JRockit VM & RT
REAL-TIMEVISIBILITY
& PROCESSING AlertsBusiness Monitoring
BAMBAM
EventsData Streams
CEPCEP
ApplicationDevelopmentFramework
EnterpriseModeling
BPA SuiteBPA Suite
JDeveloperJDeveloper
Business UserModeling
BPM SuiteBPM Suite
EDA Core: A Complex Event ProcessorProcess Static Data, Events, and Streams
� Streams� Continuous input, often in high-volume
� Time ordered
� Does not end
� Impossible to process / analyze in real-
time with traditional relational database
systems
BA BOEING D 77.575 800 20080305 10:03:02:78
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
C CITIGROUP D 34.125 2000 20080305 10:03:03:05
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
C CITIGROUP D 34.125 2000 20080305 10:03:03:05
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
C CITIGROUP D 34.125 2000 20080305 10:03:03:05
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
Event Processing provides a new data management infrastructure to support and analyze Streams in real-time
Event Processing Networks (EPN)
Adapter
Adapter
Processor
Listener- POJO
Event Source
Data Source
Query
Rule
Processor
Query
Query
Rule
Processor
Query
Rule
Processor
Query
Rule
Processor
QueryRule
Processor
Query
Listener- ALSB
EDA is composed of several processing steps intermingled with user logic.
This arrangement or network of event processing components is called an event processing network (EPN).
Event Processing Language (EPL)
Processor
Query
Query
What Is Complex Event Processing?In-Memory, Continuous Queries
� Event Processing Output� Filtering
� New stream filtered for specific criteria, e.g. stock price > $22
� Correlation & Aggregation
� Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour
� Pattern Matching
� Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window
CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46
DO DUPONT D 41.575 3000 20080305 10:03:04:12
AA ALCOA INC D 20.125 1000 20080305 10:03:01:55
AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10
BA BOEING D 77.575 800 20080305 10:03:02:78
EP
Runs In-Memory (not in Database)***Logic is defined through Continuous Queries on the data
Example: Stock Trading “W” Pattern
1 9 12 19
minutes
XY W
Z
Common Concepts
• Extend the relational model to support “continuous” query
• Construct “windows” over event streams• Bounded by time or count
• Partitioned based on values
• Processed incrementally or in batches
• Defines a “working set” of events to then apply queries to
• Use relational operators to characterize the event streams• Filtering, aggregation, and correlation of events
• Add pattern matching• Supports track-and-trace scenarios (e.g. detecting missing events)
Oracle CEP: Sample QueriesIn-Memory, Continuous Queries
� Event Processing Output� Filtering
� New stream filtered for specific criteria, e.g. stock price > $22
� CREATE QUERY f_query AS SELECT * FROM S [Range 10] WHERE S.A > 22;
� Correlation & Aggregation
� CREATE QUERY ca_query AS SELECT AVG(S1.C) FROM
S1 [RANGE 10], S2[RANGE 5] WHERE S1.A = S2.B;
� Pattern Matching – Detect a W Chart Pattern
� CREATE QUERY wp_query AS SELECT MATCH_RECOGNIZE ( MEASURES A.c1 as firstW,
last(Z) as lastZ PATTERN(A W+ X+ Y+ Z+) DEFINE W as W.c2 < prev(W.c2), X as X.c2 > prev(X.c2),
Y as Y.c2 < prev(Y.c2), Z as Z.c2 > prev(Z.c2)) as T;
Runs In-Memory***Logic is defined through Continuous Query Language on the data
Example: Stock Trading “W” Pattern
1 9 12 19
minutes
XY W
Z
Pattern Matching
� Powerful concept that allows identification of complex event patterns
� Defined as regular expressions
PATTERN (X+ Y+)
1 or more X events … … followed by 1 or more Y events
Pattern Matching ExampleStock Trading “W” Pattern
SELECT FIRST(x.time), LAST(z.time)
FROM ticker MATCH_RECOGNIZE (ONE ROW PER MATCH PARTITION BY name
PATTERN (X+ Y+ W+ Z+)
DEFINE X AS (price < PREV(price))
Y AS (price > PREV(price))
W AS (price < PREV(price))
Z AS (price > PREV(price)))
1 9 12 19
days
X
YW
Z
EDA Foundation: A Specialized Server for CEPLow Latency, Extreme High Throughput, and Java-based
Data/Msg.Feeds Listener/SINK:
User Code(Plain Java)
Data Feed Adapters
ProcessEvents(CEP)
Incoming Data Streams1,000,000 Messages/Sec
<1K message size
Aggregate, correlate, filter dataCan Handle 1,000s of Queries
Resulting Data fed to user Java code
<5% data matches filter criteria
Latency< 1 Millisecond Latency (Avg.)10 Millisecond Latency (Max)
EDA Java Application Container
EDA Foundation: A Specialized Server for CEPLow Latency, Extreme High Throughput, and Java-based
Incoming Data Streams1,000,000 Messages/Sec
<1K message size
Aggregate, correlate, filter dataCan Handle 1,000s of Queries
Resulting Data fed to user Java code
<5% data matches filter criteria
Latency< 1 Millisecond Latency (Avg.)10 Millisecond Latency (Max)
Average Latency vs. Injection Rate (10 Connections)
0
10
20
30
40
50
60
70
80
0 200000 400000 600000 800000 1000000 1200000
Injection Rate (events/second)
Average Latency
(microseconds)
Product Architecture
Oracle CEP 10.3
High Throughput
� Hundreds of thousands of events/second
Multiple-choice JVM
� JRockit or WebLogic Real Time
� Unique JVM Tools
Eclipse-based tooling
� Plugins for EDA Application Development
Easy-to-Use Development Environment
� Spring Framework, POJO
Event Processing infrastructure
� CEP Engine
� Event Processing programming constructs and services
� Time-critical streaming
� Well-known event processing language
Lightweight Java Application Server
� Full environment for building and running Java applications
� Services -- Security, Logging, User Mgmt
Advanced Tooling
Late
ncy
Analy
zer
JRockit – Real Time/ Real Time - DGC
Extended Event Infrastructure
Core Event Infrastructure
Oracle Event Stream Processor
Real Time Kernel
Oracle Complex Event Processing
Foundation Services
Config/Admin
Security Logging
Event Infrastructure
Complex Event Processing
Stream Management
Real Time Kernel
App Frameworks
POJO Spring Services
Core Event Infrastructure
Complex Event Processor
Stream Management
Real Time Kernel
ClusterManagement
DataCaching HTTP
Pub/Sub EngineEvent
Repository
Foundation Services
Config/Admin
Coherence Data Cache
Logging & Administration
App Frameworks
Spring ServicesJDBCPOJO
Select New Features Delivered in Oracle CEP 10.3
• Instantaneous In-Memory Data Manipulation
• Programmatic access to local caches
• Pre-built integration with Coherence cache for distributed caching
• Integrate cache with EPN components
• Enterprise Server Topology High Availability
• Managed Oracle CEP Cluster
• Web2.0 Web Client push-based infrastructure
• Event server components can subscribe to and publish to channels, send and receive asynchronous messages over HTTP
• Historical Event Management
• Persist Events, Record/Playback Event activity
• Next Generation Eclipse-based EDA tooling
• Visual EPL and EPN Editing
• Visualizer for Event Data and application Monitoring
Oracle CEP Server Architecture
JVM with DGC
Module Framework
( OSGi)
Service Framework
(Spring DM)
ApplicationsM O D U L E S
Event Processor
Oracle CEP Programming Model
• Event-driven components• Adapter
• Event source– JMS, File, Socket, etc.
• Complex Event Processor• Stream• Continuous Query Language
• Sink• Complex event output– POJO
listener, JMS, etc.
• Event types• Event objects are instances of
event types• Implemented as Beans, Maps,
or ‘Tuples’
Eclipse plug-ins extend WTP, support OSGI development
Oracle CEP Visualizer
Simplified Administration and Deployment Management,
Oracle CEP Visualizer
Advanced EDA Event Processing Network Performance Tuning
Oracle CEP Visualizer
Dynamic Event Processing Language Injection
• Included in Oracle CEP install
• Works with open hardware x86 hardware
• Addresses the key latency issue with Java
• Unique non-intrusive tooling to identify latency bottlenecks
• Extension of the award winning JRockit JVM
0
50000
100000
150000
200000
250000
2P Opteron 2220 4P Opteron 885 2P Intel X5355
SPECjbb2005 bops
Sun JVM
JRockit JVM
CEP Performance Enhanced by JRockitJRockit Real-Time Provides High Throughput, Low Latency JVM
JRockit Real Time Mission Control
Latency Analyzer Tooling
• Visualize thread & transaction execution time lines
• Find sources of latency spikes
• Show Thread Transitions
Oracle CEP Data Caching FrameworkEnhanced by Coherence for Highly Available, Distributed Caching
Data Services Data Services
• Pluggable framework for caching and data grid products• Supports replicated, distributed, multi-tier caching
• Out of the Box integration with Oracle Coherence
• Built-in local cache• Single process caching with paging
Oracle Business Activity Monitoring
Event-Driven Architecture
BAMCEP
BPM Processes
event
JMS
Analy
tic Vie
ws
Excep
tion
Alerts
Integr
ated
Actions
Conte
xt
Enrichm
ent
close loop
Applications
event
Databases
Streams
event
event
Messaging
Event Sources Event Processors
BAM Dashboards
BAM Alerts
BPEL Processes
Response Services
Pattern
Match
ing
across
- time
- chann
els
Correlati
on
windows
• Monitor business processes & services in real-time• Key Performance Indicators (KPIs)
• Service-Level Agreements (SLAs)
• Analyze events as they occur • Correlate events & KPIs
• Identify trends as they emerge
• Alert users to bottlenecks & solutions
• Act on current conditions• Event-driven alerts
• Real-time dashboards
• BPEL processes & web services integration
Oracle Business Activity MonitoringEvent Correlation for Business Users
• Enables business operations workers and managers to • Monitor, analyze and act
• Provides IT with a set of web-based applications to:• Capture real-time data from any database, message queue or application
• Construct data objects for analysis
• Define metrics, dashboards, alerts & automated actions
The Oracle BAM Solution
BAM Architect & Administrator
BAM Active Studio
BAM Active Viewer
• Business User Oriented• Dashboards and alerts without writing a single line of code• Web-based AJAX application, with no downloads or installs
• Real-time Active Data• Applications don’t poll the server for updates• Streams delta changes on continuous hierarchical queries: Active Data• Updates are multiplexed over a single server connection for all open queries• Updates in dimensions create ActiveData in all of the rows of linked fact tables: Active Lookups
• Action Framework• Business users can access web services integration for actions directly from their
dashboards/alerts
• Event Driven Alerts• Immediate evaluation when thresholds are crossed, without running periodic queries.• Alerts on non-events, for example alerts can detect sequences like: event A happened, and then
event B happened, but event C did not happen within 10 minutes.
• Seamless Wireless Recovery• Automatic recovery from losing internet connectivity, without reloading the dashboard and
without losing any messages
• Scalability • Supports thousands of events per second and hundreds of users on a 4x4 box
The Oracle BAM Technical Advantage
Market Validation of Oracle CEP: Awards
The first time a CEP category has been defined and the first time this award has been given in the Financial Services
Industry
Market Validation of Oracle CEP: Customer Testimonials
Tradeweb, a leader in the over-the-counter multi-asset class online marketplace and a pioneer in the development of pre-trade liquidity discovery, electronic trading and trade processing, is using Oracle’s Event-Driven Middleware.
“Oracle CEP 10g R3 is the Cadillac of event-driven solutions in the market,” said Stephani Smith, Ph.D, director and vice president TradewebData Mining Group.
“We recently released SmartIOI, an exciting new product that utilizes technology based upon the industry's first and only low-latency deterministic Java container and runtime for EDA.”
http://finance.yahoo.com/news/Oracle-Unveils-New-prnews-13604303.html
Event-Driven Architecture CustomersSelect Customer List
Other
Finance/ Banking
Public Sector
Telecom High Tech
Travel/ Transport
FXall
Oracle EDA Suite Sample Implementations
• Oi• Largest Brazilian telecommunications company• Monitor payment processing in real-time with Oracle BAM• Large throughput with 30,000 event updates per minute
• Australian Vintage• Second largest vineyard owner and manager in Australia• Monitor order fulfillment process in real-time with Oracle BAM• Oracle BAM was the first component in a SOA implementation
• Tradeweb• Over 2,000 buy-side institutions use for fixed income and derviatives services• Manage Indication of Interest communications with Oracle CEP/ Event Server• Process feeds from 200 per second to over 100,000 per second with latencies of under
10 milliseconds
• FXAll• Leading provider of automated trading and workflow solutions for foreign exchange and
treasury products• Provide real-time VWAP service to customers with Oracle CEP/ Event Server
Real World Oracle CEP DeploymentTradeweb
Tradeweb is a leader in the over-the-counter multi-asset class online marketplace and a pioneer in the development of pre-trade liquidity discovery, electronic trading and trade processing. In 2008, strategic partnership between Thompson and 10 leading global broker dealers
Requirements• When brokers send out IOIs (Indication Of Interest) identify who may
try to take advantage of that information and attempt engaging in arbitrage, to the detriment of the originating broker and restrict accordingly. • Monitor the Thompson orders feed• Identify what parties are interested in an IOI for a particular security• Respond by routing the IOI to only specific parties
• Feed Through-put, Performance• Filter feed at rate over 100,000/Sec to 200/sec for large IOI’s,
latencies < 10ms
Deployment• Event Server: IBM 335s with 2 Xeons at 2.8 GHz and 4 GB RAM,
running 32-bit RedHat ES 4.0
© 2008 Oracle Corporation – Proprietary and Confidential
Smart IOI
The Challenge and Requirements
• High volume of incoming data delivered in bursts
• Low latency, high throughput message processing
• Ability to modify processing parameters on-the-fly
• Quickly distribute processing results to other systems
• Java-based, easy to configure and extend
• Minimal number of vendors to develop solution
• Many different applications and processes consuming the output data
CEP SERVER 0 JRockit Realtime JVM Container
Oracle CEP
Order Message Feed Processing
• Event Processing
• Business Logic
Distributed
Cache
Manager
Message Server
Data Feed Queue 1
Data Feed Queue 2
CEP SERVER 1 JRockit Realtime JVM Container
Oracle CEP
Order Message Feed Processing
• Event Processing
• Business Logic
Distributed
Cache
Manager
Distributed Cache
Application Server
Application Server
Application Server
Application Server…
Production Implementation
• JRockit Realtime JVM
• Oracle CEP (Weblogic Event Server 2.0)
• Standard JMS Implementation
• Third-Party Distributed Cache
• Oracle 11g Database
• Red Hat Enterprise Server 5.0 / 64 bit
• 20GB Heap Size
The Solution
45
r d
46
C ha l le n ge s to me e tc l ie n t n e e d s :- M o re c l i e n t s m e a n s m o re ta ke r s- A s c l i e n t s a re m o v i n g toa l go r i t h m ic t ra d i n g, t h e re a c t i o nt i m e s a re ge t t i n g ve r y s h o r t- C l ie n t s a re se e k i n g m o re l iq u i d i t yo p t i o n s
M a r ke t Ma ke r s a re s t re tc h in g t h e bo un d s :- M o re ba n k s a n d f in a n c ia l in s t. wa n t toma ke ma r ke t s- Ma r ke t - ma ke r so p h i s t i c a t i o n i s d r iv in gq u o te f re q u e n c y h i g h e r- Ma r ke t - ma ke r s a re t i e r in g t h e i r p r i c e s toa c c e s s w i d e r c l i e n t sFXallAccelor(TM)
MarketMaker
MarketMaker
MarketMaker
MarketMaker
MarketMaker
Taker
Taker
Taker
Taker
Taker
Taker
Taker
Taker
Taker
A Taker would like to Hit a pricess soon as it becomes available
FIX Engine
OMS
Credit Engine
Matching Engine
Price Distribution
- En su r e t h e m a r k e tp l a c e i s f a i r a n d c o m p e t i t iv e- H a n d l e t h e g r ow t h, b o t h v e r t i c a l a n d h o r iz on t a l- En su r e l a t en cy w i t h i n t h r e s h o l d- En su r e p a r t i c i p a n t s a r e f o l l ow i n g t h e Ru l e s o f En g a g e m en t
47
48
49
50
51
54
top related