1

Data and Events Converge: Eventifying Your Data Infrastructure to

Improve Responsiveness

Chris Carlson

Director, Product Evangelist

Complex Event Processing

Informatica

#IW2012

2

Overview

• How responsive enterprises are taking an event-driven

approach to new business requirements.

• How to "eventify" an existing application and data infrastructure

• The components that can assist organizations in leveraging

real-time data

The ability for an organization to respond quickly to changing

business requirements can be facilitated by an event-driven

architecture (EDA). Implementing and growing an event-driven

architecture can be greatly simplified using the Informatica

Platform, from "eventifying" a single business application or

persistence to an entire business process or an entire

organization. This discussion will detail various methods and

examples of benefits from “eventifying” data.

3

Agenda

• What are the business requirements that are

driving Event Driven Architectures?

• What are some examples of event driven

architectures?

• How to “eventify” your applications and systems

4

Business Strategies

Real-Time

Enterprise

Time-Based

Competition

Zero-Latency

Enterprise

Agile/Adaptive

Enterprise

Predictive

Enterprise

Smart Systems

System Requirements

Agility

Speed

Information

Availability

Business Pressures

Increasing

Competition

Globalization

Better

Customer

Service

More

Regulation

Faster Pace of

Business

Market

Consolidation

Architectural Style

Event

Processing

Application Architecture Must Change to Satisfy Escalating Business Requirements

5

“Real-time” Event Processing is Aimed at Operational, not Tactical or Strategic Decisions

Impact on Business/Decision Complexity

Low Medium High

De

cis

ion

Str

uc

ture

Structured

Semi-

Structured

Unstructured

Operational • Loan approvals • Fraud detection • Dynamic pricing • Real-time decisioning

Tactical • Which marketing campaign? • How to treat customer segments? • Which new product features? • Why are sales dropping? • How to handle exceptions? • Agree on a revised forecast? • What should our credit policy be? • How to stop decline in customer satisfaction? • RFP analysis and vendor selection?

Strategic • Acquire a new company? • Change in product line? • Invest in new business? • Enter new territory? • Do we change our brand?

6

Gartner: Use Integrated "Real-Time" Operational Intelligence to Make Business Smarter

Continuous

Embedded

Analytics

"Real-time"

Operational

Decisions

Run-the-Business Operational Systems

Store Data

"at Rest"

Data Mart/ Warehouse

Strategic and

Tactical decisions

Traditional BI

Statistical Analytics

Data Mining

Other Off-Line Analytics

Periodic Reports

Interactive and Collaborative

Analytics

Intelligent

Business

Operations

Cycle (Complex Event

Processing

[CEP] used here)

Decoupled

BI, PM and

Other

Analytics

Capture

Event Data

"in Motion"

7

This is Not About Traditional BI

C-Level

Executives

Middle

Management

Operations

Supervisors

Individual

Contributors

• Complex Event Processing

• Individual process

instances, cases and transactions

• Immediate exceptions, threats and opportunities

• Corporate Performance Management (CPM) • Balanced scorecards • Offline Data Mining • In-memory database

Traditional BI + Big Data Operational Intelligence

8

Where is it Being Used?

• “Operational Technology”: Manufacturing control systems; sensor networks; smart electrical grids, roads, bridges, buildings.

• Location-based Services: Emergency responders; social computing; transportation operations (planes, trains, trucks, ships).

• Customer relationships: Cross-sell/up-sell; loyalty programs; customer service; Web analytics.

• Governance, risk and compliance: Fraud detection; network intrusion; surveillance.

• Trading systems: Financial markets; electricity; online betting; Web auctions.

9

Operational Event Monitoring

(Data Integration and Data Quality)

Adds world class, monitoring

capabilities to protect

Investments in Informatica

PowerCenter and Data Quality

Reduces the time and effort

needed to monitor

environments

Helps companies meet SLA’s

and avoid failed jobs and

downtime

Identifies expense

non-compliance and

abuse violations

• Easily define

expense

compliance rules

• Watches over

17,000 card

administrators

Proactively alerts

when maritime

security is at risk

• Transformed

human intensive

review processes

• Analyzes

geospatial data

• Simplified rule

editing and

creation

Business Event Monitoring

(Payments, Compliance)

Intelligence Monitoring

(End-User, Geospatial)

Monitors payments

for errors, fraud and

abuse

• Enhances the

ability to correct

errors quickly

• Increase loan

volumes

• Increased revenue

• Fewer SLA

violations.

Improved location-

based security and

disaster awareness

• Monitors over 500

diverse sources

• Includes social

media to

classified

intelligence

Informatica CEP – Helping Organizations Become Proactive

10

Design Pattern 1: Front-end Event-processing Subsystem (e.g., Immediate ETL)

Response Event Capture Disseminate and Compute

Raw Events

CEP

Conditioned Events

Filter out irrelevant events and duplicates

Simple calculations

Consolidate to fewer events

Enrich with historical data

Sensor

Business Application or Process

Database or file

Another system

or

or

or

Typically simple computation

Not a complete application

External Source

11

Front-end Event-processing Subsystem Characteristics and Examples

• Variations

• Output to database or event log file

• Output to continuous event stream

• Computation

• Usually or always is “stream” processing not “rules” processing, often emphasizes time windows

• Aggregation (sum, count, average, top-k, etc.)

• No causality tracking or explicit event hierarchy, limited correlation

• Output is cleansed, conditioned and reduced in volume

• Subsequent fraud detection (e.g., PCI) on log files

12

Design Pattern 2: Basic Monitor

Response Event Capture Disseminate and

Compute

Base Events

CEP

Alerts, Notifications

Sensor

Business Application or Process

or

or

One producer

Decision Support

Business Application or Process

or

or

One consumer

Filter events

Calculate

Detect patterns, trends, relationships

among events

Enrich data, put in context

Manual Action

Monitor Facility

Inquiry or Simple

Analytics

13

Examples: Basic Monitors

• Variation 1: Events from physical sensors

• Tsunami warning systems

• Physical intrusion detection systems - accelerometer sensor data, motion detectors or pressure sensors

• Monitoring systems track the location of children at amusement parks using RFID

• Digital Control Systems

• Variation 2: Events from an IT source

• Airport kiosk monitoring system

• Business applications trigger a simple dashboard or e-mail alert

• EPAs listen to Twitter to detect and report on the frequency of use of certain keywords

14

Design Pattern 3: Operations or Process Monitor

Process Monitor

CEP or

BAM

BPMT

Situation Awareness

Fulfillment Shipping Manual

Approval Ordering

Business Process

Cloud

Automated Decisions

Notify person, or trigger external application or

device

Business Application or Process

Actuator or

Machine

Multiple input events from one

process or operation

Feedback to manage the process or operation

15

Examples: Operations or Process Monitor

Purpose: Tracks conditions and operating performance against

targets (thresholds) and historical baselines, problem

determination, sometimes with automated correction.

• Operations monitors (OT) for factory production lines, power

plants, electrical distribution (smart grids), airline baggage

handling and other process plants.

• IT operations monitoring, including network and computer

system monitors (NSM) and “business’ service management

(BSM).

• Application-specific BAM dashboards, fraud.

• Business processing monitors (BPM-style BAM) aligned with a

BPM orchestration engine or workflow engine.

16

Design Pattern 4: Monitoring Heterogeneous Systems in a Virtual Enterprise (e.g., SCM)

Supply Chain Management

CEP or

BAM

Situation Awareness Input events from many

external companies

Internal Systems

Notify person, or trigger external

application

Business Application or Process

Freight Consolidator

Rail Operator

Forwarder

Warehouse

Customer

Air or Express Carrier

Supplier

Agent

17

Examples: Heterogeneous Activity Monitor

Purpose: An operations or process monitor with input from

multiple systems, often multiple companies (B2B).

• Supply Chain Management

• Track and trace (farm to fork)

• Factories with hundreds of disparate devices

• BAM dashboards that monitor heterogeneous applications

• Business processing monitors not aligned with one BPM

orchestration engine or workflow engine

18

Design Pattern 5: Enterprise Nervous System Example: Airlines, Railroads and Retail Banks

Operational Data Store

Enterprise Nervous System

Event

Processor

Operations Control

Seat Reservations Airline Operations

Federal Aviation Administration

Onboard Telemetry

Offline BI

Fueling,

Maintenance

Management

Ramp & Crew Scheduling

Catering, Baggage Handling

Flight Information Displays, Web Access

Flight Status Logs

Response Event Capture Disseminate and Compute

19

Examples: Enterprise Nervous Systems

Nature: Application integration and event management blended. It is an

overlay on other applications, monitoring multiple conditions.

• Airline operations

• Railroad operations

• Shipping operations

• Trucking fleet management

• Retail bank

• Manufacturing Company

• Ultimately all medium and large companies will have EPNs that

function as ENSs

20

“Eventifying” your Applications and Systems

Event Collection Data Replication

Real-time Change Data

Capture from underlying

database persistence layer

Event Transformation

and Routing

Operational Data Integration Hub

Event Distribution Ultra Messaging

21

Data Replication : Business Drivers

Move to Real-Time Data Warehousing for

Operational BI and Analytics • Improve decisions and insight by combining real time and historical data

• Respond quickly to changing business conditions with near real time decisions

Scale Up While Reducing IT Spend • Offload to cost effective systems for analysis and reporting

• Allow more users access without impacting application performance

• Reduce risk and improve compliance with full audit trail of all transaction activity

Continuous Availability for Business Continuity • Eliminate downtime from hardware/software upgrades

• Eliminate downtime from application maintenance

22

Informatica Data Replication

A set of technologies to: Manage the capture, routing and delivery of transactional data in real-time

Optimize delivery to databases, warehouses, and appliances

Automate the cloning of application data – thousands of mixed tables at once

JMS

Sources

Databases

Transaction Logs

Datawarehouses

Appliances

EXTRACT

APPLY

JMS

23

Sources Targets

Replay changes

Merge and apply

Write to Audit tables Databases

Data warehouses

Appliances

Informatica Data Replication Architecture

JMS

EXTRACT APPLY

High Speed Apply

Synchronize sources to targets

Switch to Replication

JMS

Logs

HTTPS

Server Manager Server Manager

Log based Extract

Compression

Encryption

Transaction Integrity

Data Consistency

Recovery and Restart

24

Two Separate Worlds?!

25

Traditional Application Integration Technology Limits

• Corrupted data is propagated in real-time

• Process is valid, business transaction is not

• Cannot process large data sets transformation at

low latency

• Leaving 70% of integration to the application owners

• Lack of visibility of data flow

• Huge security hole and compliance issues

• 60-70% of Integration is Still Batch and File

Oriented

• EAI and ESB fail short

26

What is the Data Integration Hub

• Comprehensive and simplified data integration

solution for moving data across multiple

applications

• Integrates data effectively

• any latency

• any format

• any size

• any protocol

• Supports auditing and quality

Informatica Confidential

27

Data

Persistence

Tracing

Logging

DIH Core Capabilities • Manage RT and Batch

Transformation With Same Design

Paradigm

• Contract Based Data Integration

• What (customer, product, orders,…)

• How (JMS, File, WS, Native, ...)

• When (RT, Scheduled, event-based, …)

Informatica Confidential

ERP Legacy

Application

SaaS Portal

File / Event / 15mn RDBMS / Push / RT

DIH

Universal

MDM EDW

• Governance and End to End

Visibility

• Integrated management and

administration

• Fast Application On-boarding

• Quality-Controlled Data

• Based on Already Well Known

Informatica Technology

• Decouple Applications (support of

pub/sub for batch and real-time)

28

DIH Differentiators vs. Traditional EAI/ESB Offers

• Can manage both bulk and message integration

• Can manipulate the data

• Transform formats

• Transform content

• Can control the quality of the information

• Provide a store and forward engine

• Can deal with native application format without having to

customize application

• Can capture data changes at the source without intrusion

• Provides traceability and audit at the data level for compliance

29

Logging

Data Integration Hub Overview

A1

A2

A3 Canonical

Staging

INBOUND

Data Flow

INBOUND

Data Flow

Data Quality

Control

OUTBOUND

Data Flow

Synchroni-

zation

Control

Providers Consumers Data Integration Hub

30

Ultra Messaging Complete Messaging Solution – Enterprise-Wide

Server / OS:

SUN / Solaris

Server / OS:

HP / Redhat Linux

Server / OS:

Dell / Windows

Server / OS:

IBM / AIX

IBM MQ

NETWORK: Gigabit Ethernet, LAN, WAN, Cisco, Voltaire, 10GiGE, or any other networking environment

Informatica Ultra Messaging®

Streaming Data Delivery

Persistence Durable

Subscriptions

JMS Store & Forward

Queuing Transaction

Management

31

Modern No Limits Design

Informatica Ultra Messaging Built for Modern Times

Reliable & Powerful Reduce Costs

For the next 20 years • Efficient

• Fast

• Unified API

Bet your business • Bullet proof - 100% uptime

• Unmatched throughput

• Flexible

Do more with less • Cut / eliminate hardware

• Consolidate vendors

• Enterprise standard

5X throughput

50% hardware

32

Real-time Aggregated Data Integration PCRE & Ultra Messaging

Server

Server

Server

Server

Server

ServerServer

Server

Server

Server

Server

Server

Server

ServerServer

ServerRAW SYSLOG DATA

BIS Logs

MIGRATION PROCESS Across WAN (Through the Great Divide)

7 Data Centres100's of Servers

Approx 145 TB / Day

Rsyslog Farm50 Servers

EDW

ApplianceSpecificLoading

PowerCenterUM/JMS

LogFile

Drop

B2B DX

ServerPowerCenter8.6.1

UM/JMS

UltraMessaging

UltraMessaging

} UMCache

Informatica Confidential: Internal Use Only

33

Web Sockets Eventing

Desktop

Trading Engine

News Feed

Payment System

Database Storage

Application Logic

Web Service

ERP/CRM System

Trusted Network DMZ Web

Mobile

Browser

Ultra Messaging

Domain

Wide client support – your choice of client technology

Flash/Flex

Mobile

Browser/ Desktop

Cluster

34

Next Steps?

• Are you researching or implementing Event

Driven Architectures?

• Are you receiving business requirements for

data access and processing that is difficult (or

costly) to be met with existing datawarehousing

technologies?

• Would you like to learn more how our customers

are deploying Event Driven Architectures?

35

Marketplace Overview A Trusted, Open Ecosystem

• Virtual Marketplace for Data Integration Apps

• Solutions across all technology areas – DI, DQ, MDM, Cloud, etc.

• Open Ecosystem – Apps from Partners, ISVs, Consultants, and Developers

• Seal of Approval ensures App quality

• More than 600 Apps, over 200 Free!

• 15k visits per month, 2k downloads

http://marketplace.informatica.com

R

M

36

Thank You