oracle event driven architecture -eda

<Insert Picture Here>

Introduction to Oracle EDA Suite

Kemal Ülkü, [email protected] Consultancy Team Leader – Oracle Fusion Middleware

Event-Driven Applications

Financial Services

Transportation & Logistics

Public Sector & Military

Manufacturing

Insurance

Telecommunications & Services

Algorithmic trading

Asset management

Distributed order orchestration

‘Negative Working Capital’ inventory management

Reponses to calamities –earthquake, flooding

• Intrusion detection systems• Military asset allocation

Need to support one or more of:

• High volume

• Continuous streaming

• Sub-millisecond latency

• Disparate sources

• Time window processing

• Complex pattern matching

Oracle EDA SuiteComplete Suite for Industrial Event Processing

Routing Transform

Oracle Service BusOracle Service Bus

Mediation

REAL-TIMEVISIBILITY

& PROCESSING

AlertsBusiness Monitoring

BAMBAM

Event FeedsData Streams

Java-based EventServer

CEPCEP

Socket

AdaptersAdapters

DB FileCONNECTIVITY

ROUTING & DATA

SERVICES

Business Rules

J2EE Application Server(Oracle AS, WebLogic, WebSphere, JBoss)

Industry SpecificJMS

CoherenceJRockit VM JRockit RTENTERPRISE PERFORMANCE & SCALABILITY

Apps

AdaptersAdapters B2BB2B

PartnersDB Legacy

CONNECTIVITY

Routing Transform DataQuality

ETL &Replication

Data IntegratorData IntegratorOracle Service BusOracle Service Bus

Mediation

ROUTING & DATA SERVICES

NativeBPEL

Business Rules

Human Workflow

BPEL Process ManagerBPEL Process Manager

ORCHESTRATION

Coherence CacheJ2EE Application Server

(Oracle AS, WebLogic, WebSphere, JBoss)Messaging

Oracle SOA and Event Technologies

GOVERNANCESystem

Monitoring

Enterprise Enterprise ManagerManager

UDDI

WS PoliciesSecurity

Web Services Web Services ManagerManager

RegistryRegistry

GOVERNANCE

SOA lifecyclegovernance

EnterpriseEnterpriseRepositoryRepository

JRockit VM & RT

REAL-TIMEVISIBILITY

& PROCESSING AlertsBusiness Monitoring

BAMBAM

EventsData Streams

CEPCEP

ApplicationDevelopmentFramework

EnterpriseModeling

BPA SuiteBPA Suite

JDeveloperJDeveloper

Business UserModeling

BPM SuiteBPM Suite

EDA Core: A Complex Event ProcessorProcess Static Data, Events, and Streams

� Streams� Continuous input, often in high-volume

� Time ordered

� Does not end

� Impossible to process / analyze in real-

time with traditional relational database

systems

BA BOEING D 77.575 800 20080305 10:03:02:78

DO DUPONT D 41.575 3000 20080305 10:03:04:12

AA ALCOA INC D 20.125 1000 20080305 10:03:01:55

AXP AMER EXPRESS CO D 45.875 500 20080305 10:03:02:10

BA BOEING D 77.575 800 20080305 10:03:02:78

C CITIGROUP D 34.125 2000 20080305 10:03:03:05

CAT CATERPILLAR D 22.5 600 20080305 10:03:03:46

DO DUPONT D 41.575 3000 20080305 10:03:04:12

AA ALCOA INC D 20.125 1000 20080305 10:03:01:55


BA BOEING D 77.575 800 20080305 10:03:02:78

C CITIGROUP D 34.125 2000 20080305 10:03:03:05


DO DUPONT D 41.575 3000 20080305 10:03:04:12

AA ALCOA INC D 20.125 1000 20080305 10:03:01:55


BA BOEING D 77.575 800 20080305 10:03:02:78

C CITIGROUP D 34.125 2000 20080305 10:03:03:05


DO DUPONT D 41.575 3000 20080305 10:03:04:12

AA ALCOA INC D 20.125 1000 20080305 10:03:01:55


BA BOEING D 77.575 800 20080305 10:03:02:78

Event Processing provides a new data management infrastructure to support and analyze Streams in real-time

Event Processing Networks (EPN)

Adapter

Adapter

Processor

Listener- POJO

Event Source

Data Source

Query

Rule

Processor

Query

Query

Rule

Processor

Query

Rule

Processor

Query

Rule

Processor

QueryRule

Processor

Query

Listener- ALSB

EDA is composed of several processing steps intermingled with user logic.

This arrangement or network of event processing components is called an event processing network (EPN).

Event Processing Language (EPL)

Processor

Query

Query

What Is Complex Event Processing?In-Memory, Continuous Queries

� Event Processing Output� Filtering

� New stream filtered for specific criteria, e.g. stock price > $22

� Correlation & Aggregation

� Scrolling, time-based window metrics, e.g. average # of stock trades in the last hour

� Pattern Matching

� Notification of detected event patterns, e.g. price changes A, B and C occurred within 15 minute window


DO DUPONT D 41.575 3000 20080305 10:03:04:12

AA ALCOA INC D 20.125 1000 20080305 10:03:01:55


BA BOEING D 77.575 800 20080305 10:03:02:78

EP

Runs In-Memory (not in Database)***Logic is defined through Continuous Queries on the data

Example: Stock Trading “W” Pattern

1 9 12 19

minutes

XY W

Z

Common Concepts

• Extend the relational model to support “continuous” query

• Construct “windows” over event streams• Bounded by time or count

• Partitioned based on values

• Processed incrementally or in batches

• Defines a “working set” of events to then apply queries to

• Use relational operators to characterize the event streams• Filtering, aggregation, and correlation of events

• Add pattern matching• Supports track-and-trace scenarios (e.g. detecting missing events)

Oracle CEP: Sample QueriesIn-Memory, Continuous Queries

� Event Processing Output� Filtering

� New stream filtered for specific criteria, e.g. stock price > $22

� CREATE QUERY f_query AS SELECT * FROM S [Range 10] WHERE S.A > 22;

� Correlation & Aggregation

� CREATE QUERY ca_query AS SELECT AVG(S1.C) FROM

S1 [RANGE 10], S2[RANGE 5] WHERE S1.A = S2.B;

� Pattern Matching – Detect a W Chart Pattern

� CREATE QUERY wp_query AS SELECT MATCH_RECOGNIZE ( MEASURES A.c1 as firstW,

last(Z) as lastZ PATTERN(A W+ X+ Y+ Z+) DEFINE W as W.c2 < prev(W.c2), X as X.c2 > prev(X.c2),

Y as Y.c2 < prev(Y.c2), Z as Z.c2 > prev(Z.c2)) as T;

Runs In-Memory***Logic is defined through Continuous Query Language on the data

Example: Stock Trading “W” Pattern

1 9 12 19

minutes

XY W

Z

Pattern Matching

� Powerful concept that allows identification of complex event patterns

� Defined as regular expressions

PATTERN (X+ Y+)

1 or more X events … … followed by 1 or more Y events

Pattern Matching ExampleStock Trading “W” Pattern

SELECT FIRST(x.time), LAST(z.time)

FROM ticker MATCH_RECOGNIZE (ONE ROW PER MATCH PARTITION BY name

PATTERN (X+ Y+ W+ Z+)

DEFINE X AS (price < PREV(price))

Y AS (price > PREV(price))

W AS (price < PREV(price))

Z AS (price > PREV(price)))

1 9 12 19

days

X

YW

Z

EDA Foundation: A Specialized Server for CEPLow Latency, Extreme High Throughput, and Java-based

Data/Msg.Feeds Listener/SINK:

User Code(Plain Java)

Data Feed Adapters

ProcessEvents(CEP)

Incoming Data Streams1,000,000 Messages/Sec

<1K message size

Aggregate, correlate, filter dataCan Handle 1,000s of Queries

Resulting Data fed to user Java code

<5% data matches filter criteria

Latency< 1 Millisecond Latency (Avg.)10 Millisecond Latency (Max)

EDA Java Application Container

EDA Foundation: A Specialized Server for CEPLow Latency, Extreme High Throughput, and Java-based

Incoming Data Streams1,000,000 Messages/Sec

<1K message size

Aggregate, correlate, filter dataCan Handle 1,000s of Queries

Resulting Data fed to user Java code

<5% data matches filter criteria

Latency< 1 Millisecond Latency (Avg.)10 Millisecond Latency (Max)

Average Latency vs. Injection Rate (10 Connections)

0

10

20

30

40

50

60

70

80

0 200000 400000 600000 800000 1000000 1200000

Injection Rate (events/second)

Average Latency

(microseconds)

Product Architecture

Oracle CEP 10.3

High Throughput

� Hundreds of thousands of events/second

Multiple-choice JVM

� JRockit or WebLogic Real Time

� Unique JVM Tools

Eclipse-based tooling

� Plugins for EDA Application Development

Easy-to-Use Development Environment

� Spring Framework, POJO

Event Processing infrastructure

� CEP Engine

� Event Processing programming constructs and services

� Time-critical streaming

� Well-known event processing language

Lightweight Java Application Server

� Full environment for building and running Java applications

� Services -- Security, Logging, User Mgmt

Advanced Tooling

Late

ncy

Analy

zer

JRockit – Real Time/ Real Time - DGC

Extended Event Infrastructure

Core Event Infrastructure

Oracle Event Stream Processor

Real Time Kernel

Oracle Complex Event Processing

Foundation Services

Config/Admin

Security Logging

Event Infrastructure

Complex Event Processing

Stream Management

Real Time Kernel

App Frameworks

POJO Spring Services

Core Event Infrastructure

Complex Event Processor

Stream Management

Real Time Kernel

ClusterManagement

DataCaching HTTP

Pub/Sub EngineEvent

Repository

Foundation Services

Config/Admin

Coherence Data Cache

Logging & Administration

App Frameworks

Spring ServicesJDBCPOJO

Select New Features Delivered in Oracle CEP 10.3

• Instantaneous In-Memory Data Manipulation

• Programmatic access to local caches

• Pre-built integration with Coherence cache for distributed caching

• Integrate cache with EPN components

• Enterprise Server Topology High Availability

• Managed Oracle CEP Cluster

• Web2.0 Web Client push-based infrastructure

• Event server components can subscribe to and publish to channels, send and receive asynchronous messages over HTTP

• Historical Event Management

• Persist Events, Record/Playback Event activity

• Next Generation Eclipse-based EDA tooling

• Visual EPL and EPN Editing

• Visualizer for Event Data and application Monitoring

Oracle CEP Server Architecture

JVM with DGC

Module Framework

( OSGi)

Service Framework

(Spring DM)

ApplicationsM O D U L E S

Event Processor

Oracle CEP Programming Model

• Event-driven components• Adapter

• Event source– JMS, File, Socket, etc.

• Complex Event Processor• Stream• Continuous Query Language

• Sink• Complex event output– POJO

listener, JMS, etc.

• Event types• Event objects are instances of

event types• Implemented as Beans, Maps,

or ‘Tuples’

Eclipse plug-ins extend WTP, support OSGI development

Oracle CEP Visualizer

Simplified Administration and Deployment Management,


Advanced EDA Event Processing Network Performance Tuning


Dynamic Event Processing Language Injection

• Included in Oracle CEP install

• Works with open hardware x86 hardware

• Addresses the key latency issue with Java

• Unique non-intrusive tooling to identify latency bottlenecks

• Extension of the award winning JRockit JVM

0

50000

100000

150000

200000

250000

2P Opteron 2220 4P Opteron 885 2P Intel X5355

SPECjbb2005 bops

Sun JVM

JRockit JVM

CEP Performance Enhanced by JRockitJRockit Real-Time Provides High Throughput, Low Latency JVM

JRockit Real Time Mission Control

Latency Analyzer Tooling

• Visualize thread & transaction execution time lines

• Find sources of latency spikes

• Show Thread Transitions

Oracle CEP Data Caching FrameworkEnhanced by Coherence for Highly Available, Distributed Caching

Data Services Data Services

• Pluggable framework for caching and data grid products• Supports replicated, distributed, multi-tier caching

• Out of the Box integration with Oracle Coherence

• Built-in local cache• Single process caching with paging

Oracle Business Activity Monitoring

Event-Driven Architecture

BAMCEP

BPM Processes

event

JMS

Analy

tic Vie

ws

Excep

tion

Alerts

Integr

ated

Actions

Conte

xt

Enrichm

ent

close loop

Applications

event

Databases

Streams

event

event

Messaging

Event Sources Event Processors

BAM Dashboards

BAM Alerts

BPEL Processes

Response Services

Pattern

Match

ing

across

- time

- chann

els

Correlati

on

windows

• Monitor business processes & services in real-time• Key Performance Indicators (KPIs)

• Service-Level Agreements (SLAs)

• Analyze events as they occur • Correlate events & KPIs

• Identify trends as they emerge

• Alert users to bottlenecks & solutions

• Act on current conditions• Event-driven alerts

• Real-time dashboards

• BPEL processes & web services integration

Oracle Business Activity MonitoringEvent Correlation for Business Users

• Enables business operations workers and managers to • Monitor, analyze and act

• Provides IT with a set of web-based applications to:• Capture real-time data from any database, message queue or application

• Construct data objects for analysis

• Define metrics, dashboards, alerts & automated actions

The Oracle BAM Solution

BAM Architect & Administrator

BAM Active Studio

BAM Active Viewer

• Business User Oriented• Dashboards and alerts without writing a single line of code• Web-based AJAX application, with no downloads or installs

• Real-time Active Data• Applications don’t poll the server for updates• Streams delta changes on continuous hierarchical queries: Active Data• Updates are multiplexed over a single server connection for all open queries• Updates in dimensions create ActiveData in all of the rows of linked fact tables: Active Lookups

• Action Framework• Business users can access web services integration for actions directly from their

dashboards/alerts

• Event Driven Alerts• Immediate evaluation when thresholds are crossed, without running periodic queries.• Alerts on non-events, for example alerts can detect sequences like: event A happened, and then

event B happened, but event C did not happen within 10 minutes.

• Seamless Wireless Recovery• Automatic recovery from losing internet connectivity, without reloading the dashboard and

without losing any messages

• Scalability • Supports thousands of events per second and hundreds of users on a 4x4 box

The Oracle BAM Technical Advantage

Market Validation of Oracle CEP: Awards

The first time a CEP category has been defined and the first time this award has been given in the Financial Services

Industry

Market Validation of Oracle CEP: Customer Testimonials

Tradeweb, a leader in the over-the-counter multi-asset class online marketplace and a pioneer in the development of pre-trade liquidity discovery, electronic trading and trade processing, is using Oracle’s Event-Driven Middleware.

“Oracle CEP 10g R3 is the Cadillac of event-driven solutions in the market,” said Stephani Smith, Ph.D, director and vice president TradewebData Mining Group.

“We recently released SmartIOI, an exciting new product that utilizes technology based upon the industry's first and only low-latency deterministic Java container and runtime for EDA.”

http://finance.yahoo.com/news/Oracle-Unveils-New-prnews-13604303.html

Event-Driven Architecture CustomersSelect Customer List

Other

Finance/ Banking

Public Sector

Telecom High Tech

Travel/ Transport

FXall

Oracle EDA Suite Sample Implementations

• Oi• Largest Brazilian telecommunications company• Monitor payment processing in real-time with Oracle BAM• Large throughput with 30,000 event updates per minute

• Australian Vintage• Second largest vineyard owner and manager in Australia• Monitor order fulfillment process in real-time with Oracle BAM• Oracle BAM was the first component in a SOA implementation

• Tradeweb• Over 2,000 buy-side institutions use for fixed income and derviatives services• Manage Indication of Interest communications with Oracle CEP/ Event Server• Process feeds from 200 per second to over 100,000 per second with latencies of under

10 milliseconds

• FXAll• Leading provider of automated trading and workflow solutions for foreign exchange and

treasury products• Provide real-time VWAP service to customers with Oracle CEP/ Event Server

Real World Oracle CEP DeploymentTradeweb

Tradeweb is a leader in the over-the-counter multi-asset class online marketplace and a pioneer in the development of pre-trade liquidity discovery, electronic trading and trade processing. In 2008, strategic partnership between Thompson and 10 leading global broker dealers

Requirements• When brokers send out IOIs (Indication Of Interest) identify who may

try to take advantage of that information and attempt engaging in arbitrage, to the detriment of the originating broker and restrict accordingly. • Monitor the Thompson orders feed• Identify what parties are interested in an IOI for a particular security• Respond by routing the IOI to only specific parties

• Feed Through-put, Performance• Filter feed at rate over 100,000/Sec to 200/sec for large IOI’s,

latencies < 10ms

Deployment• Event Server: IBM 335s with 2 Xeons at 2.8 GHz and 4 GB RAM,

running 32-bit RedHat ES 4.0

The Challenge and Requirements

• High volume of incoming data delivered in bursts

• Low latency, high throughput message processing

• Ability to modify processing parameters on-the-fly

• Quickly distribute processing results to other systems

• Java-based, easy to configure and extend

• Minimal number of vendors to develop solution

• Many different applications and processes consuming the output data

CEP SERVER 0 JRockit Realtime JVM Container

Oracle CEP

Order Message Feed Processing

• Event Processing

• Business Logic

Distributed

Cache

Manager

Message Server

Data Feed Queue 1

Data Feed Queue 2

CEP SERVER 1 JRockit Realtime JVM Container

Oracle CEP

Order Message Feed Processing

• Event Processing

• Business Logic

Distributed

Cache

Manager

Distributed Cache

Application Server

Application Server

Application Server

Application Server…

Production Implementation

• JRockit Realtime JVM

• Oracle CEP (Weblogic Event Server 2.0)

• Standard JMS Implementation

• Third-Party Distributed Cache

• Oracle 11g Database

• Red Hat Enterprise Server 5.0 / 64 bit

• 20GB Heap Size

The Solution

45

r d

46

C ha l le n ge s to me e tc l ie n t n e e d s :- M o re c l i e n t s m e a n s m o re ta ke r s- A s c l i e n t s a re m o v i n g toa l go r i t h m ic t ra d i n g, t h e re a c t i o nt i m e s a re ge t t i n g ve r y s h o r t- C l ie n t s a re se e k i n g m o re l iq u i d i t yo p t i o n s

M a r ke t Ma ke r s a re s t re tc h in g t h e bo un d s :- M o re ba n k s a n d f in a n c ia l in s t. wa n t toma ke ma r ke t s- Ma r ke t - ma ke r so p h i s t i c a t i o n i s d r iv in gq u o te f re q u e n c y h i g h e r- Ma r ke t - ma ke r s a re t i e r in g t h e i r p r i c e s toa c c e s s w i d e r c l i e n t sFXallAccelor(TM)

MarketMaker

MarketMaker

MarketMaker

MarketMaker

MarketMaker

Taker

Taker

Taker

Taker

Taker

Taker

Taker

Taker

Taker

A Taker would like to Hit a pricess soon as it becomes available

FIX Engine

OMS

Credit Engine

Matching Engine

Price Distribution

- En su r e t h e m a r k e tp l a c e i s f a i r a n d c o m p e t i t iv e- H a n d l e t h e g r ow t h, b o t h v e r t i c a l a n d h o r iz on t a l- En su r e l a t en cy w i t h i n t h r e s h o l d- En su r e p a r t i c i p a n t s a r e f o l l ow i n g t h e Ru l e s o f En g a g e m en t

oracle event driven architecture -eda

Documents