Download - INNOV-10 Progress® Event Engine™ Technical Overview Prashant Thumma Principal Software Engineer
INNOV-10Progress® Event Engine™Technical Overview
Prashant ThummaPrincipal Software Engineer
2 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
3 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Event Stream Processing (CEP)
ESP is about Operational Business Data– Capture– Query– Analysis
Characteristics– High Volume Streaming Data– Time Critical Analysis– Consistent and Reproducible Results– Highly Available
Simple Structured Data– Data is not Relational or complex Object model
What is ESP?
4 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Event Stream Processing (ESP)
Event Processing Language– Data Driven– Flexible– Complete
Event Persistence Engine– High Performance
Analytical Capabilities– Temporal Data Analysis– Inbound and Outbound Event Processors
High Availability Modeling, Deployment and Development Tools
ESP Platform Requirements
5 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Event Stream Processing (CEP)
Financial Markets– Program Trading– Testing Trading Strategies
RFID– Inventory Logistics– Asset Tracking– Theft Detection
Manufacturing– Capture and analysis of data from PLCs
Telecommunications– Cell Phone Call Tracking– Multimedia Services Billing– Network Management
Scientific– Sensor Data
ESP Applications
6 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
7 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress ESP Products
Event Cache for JMS
Adapter
Event Server
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
XML Configuration FilesJava API
Progress® Event Engine™ (PSEE)
C++ API
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
-<!– simple.xml -->
- <event>
+ <info1>
+ <info2>
</event>
XML Configuration Files
Apama
8 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
9 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
It’s all about Event Data– Collection, storage, management and query
What is Event Data?– Streaming, constantly flowing information
Stock Trades, RFID, IP Packets, Mfg PLC’s– Categorized
Stock tick symbol, RFID tag ID, IP address– Typically Time Ordered
Trade time, RFID scan time
What is it for?
10 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
3. Query
query
client client client
Event Cache
Service
1. CollectEvent Data
Feed store
Convert external events to PSEE events
2. OrganizeStructures data for fast retrieval
sort/mergePersistent Storage
disk disk
disk disk
Data Flow
11 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
High Performance, Scalable– Collection Engine
Store tens of thousands events per second 20 TB+ database size Index data is ~1% of database size
– Query Engine High Performance Pipelined Architecture Extensible Functionality
– Simultaneous Query and Collection Writes do not block reads!
Why PSEE?
12 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
Transparent Data Management– Event Persistence
Requires no coding to persist events
– Transactional Consistent Data Hidden and Managed by PSEE
– Tunable Latency “Real-time” defined by you!
Why PSEE?
13 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
Simple API Supports Multiple Languages
– Java and C++ language support Multiple Platform Support
– Window, Solaris, Linux
– Heterogeneous deployment
Why PSEE?
14 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
Functor: A unit of work in PSEE– 3 Main types (Source, Process, Sink)– Built in library of Functors– Customer coded
Pipeline: A sequence of Functors– Collect, process, transform event streams
Task Manager– Manages pipeline execution
Events: Input and Output of Pipelines
Components
15 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
PSEE Event– XML schema
– Supports All C++ Primitive Types
Event Definition
<EventDefinitions bundleName=“TABundle” …> <Event name=“Trade” type=“TimedCategorizedEvent”> <Member name=“price” type=“Float32”/> <Member name=“volume” type=“UInt32”/> </Event></EventDefinitions>
16 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
class MyCollectorFunctor : public Functor {public:
void initialize(FunctorContext& ctx);void handle(FunctorContext& ctx);void finalize(FunctorContext& ctx);
private:// functor data variables
}
Progress Event Engine
Implements Event Stream Processing Logic Interface For Accessing Events
Functor
17 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event EngineCollector Pipeline
Event-Data File
for Bulk Loading
Event
Stream
StoreRealtime
Functor
Real-Time
Data Feed
Series
(PSEE Database)
MyCollector
Functor
18 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event EngineQuery Pipeline
Series
(PSEE Database)
Retrieve
Functor
Filter
Functor
Sort
Functor
Event
Stream
Event
Stream
Event
Stream
MyXMLwriter
Functor
19 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event Engine
Collector Functors– StoreRealtime
– StoreBulkload
Query Functors– Retrieve
– Filter
– Sort
– Count
– Group Count
Predefined Functors
20 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Progress Event EngineScalable Deployment
OSES
Qu
ery
Ad
apte
r
PS
EE
OSES
Co
llect
ion
Ad
apte
r 2
Co
llect
ion
Ad
apte
r 1 PS
EE
Raw Series
Optimized db’s for query
Series database
OSES
Qu
ery
Ad
apte
r
PS
EE
Qu
ery
Ad
apte
r
OSES
PS
EE
VMMA & CFA Scalability
MessagingScalability
21 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
22 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Performance
Event storage (events/sec)– Real-time collector – pipeline using
StoreRealtime functor– Bulk load collector – pipeline using
StoreBulkload functor Event retrieval (events/sec)
– Threaded execution of query pipeline Random query Query on several (randomly chosen)
categories
Performance Metrics
23 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Performance
Database Characteristics– 160 Million Events
– 20 Byte Event
– 10000 Categories
System Configuration– Sun Fire-V440
– Solaris 8
– 4 CPUs (1GHz)
– 8 GB Memory
Performance Testing Environment
24 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Performance
Real-time collector– 94,062 events / second
Bulk load collector– 191,152 events / second
Query– 213 Queries / second
– 10000 events / query
– 2,134,560 events / sec
Performance Testing Results
25 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Agenda Slide
Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration
26 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
D I S C L A I M E R
Under Development
This talk includes information about potential future products and/or product enhancements.
What I am going to say reflects our current thinking, but the information contained herein is preliminary and subject to change. Any future products we ultimately deliver may be materially different from what is described here.
D I S C L A I M E R
27 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Integration
Provides Event Persistence for Apama Initial Implementation for Customer
– Tick Database for Back-Testing Trading Strategies
– Query Interface Using Apama Event Language
Future– Extend Apama Event Language to support
Persistence– Seamless Configuration via Apama
Management Interfaces
Integration with Apama
28 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Integration
PSEE on the bus using Event Cache Event Cache Features
– Native JMS Message Based Query
– Results are Java Objects
– Batched for Optimal Performance
Integration with Sonic
29 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
PSEE Integration
PSEE based RFID Accelerator provides 4GL Interface
Web Services based Implementation RFID Accelerator as a reference
implementation Further 4GL Integration
Integration with 4GL
30 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
In Summary
Exponential growth of event data requires ESP solutions
PSEE + Apama is an ESP solution
Based on proven technologies Simple! Infrastructure problems
are handled by PSEE It can be integrated into existing
business systems
31 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Questions?
32 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation
Thank you for your time!
33 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation