Real-Time Data IntegrationBest Practices and Architectures

Presented by Nelson Petracek, Informatica

Real-timeData Quality

Missed Opportunities

Lower Customer

Service Levels

The Need for Timely Information

2

Customers

Indirect Sales Channel

Sales & Marketing

Enterprise

Product Designers

EnterpriseInformation System

Retail Stores

Call Center

Increased Costs

Market Share Erosion

Increased Exposure to

RiskRT Data

Warehouse

Data Synchronization

OperationalData Hub

DataWarehouse

OperationalData Hub

3

Applications created in 2012 using traditional architecture models will be an IT-constraining

legacy by 2016

The leading business applications of 2016 are designed today using Nexus-enabled

application architecture principles

Changing Perspectives on Data

• It is no longer sufficient to view information “after the fact”.

• Business demands information sooner, with more accuracy, in order to meet competitive and regulatory demands.

• Business needs to respond to “threats” and “opportunities sooner.

• Reduce decision latency.

• Proactive Alerts and Notifications

4

What does “Real-Time” mean to you?

5

DaysPerformance Management

Sales by Business Unit

HourInventory Management

Compliance

MinutesOrder ConfirmationOperational Metrics Second

Straight-through ProcessingCall-center Analytics

Point-of-entry Applications

InstantaneousSecurities Exchange

Operational Data Integration

Analytical Data Integration

Application 2

Traditional Integration

6

Application 1

Application Logic

Data

Application Logic

Data

Application 3

Application Logic

Data

Custom Extracts and Interfaces

Query/pull

Interfaces Interfaces

Query/pull

“Siloed” Applications, Embedded Interfaces and Transformation

Interfaces

Traditional Integration: Result…

7

Critical Considerations

• Complexity of source and targets.

• Data formats.

• Level of quality.

• Availability of source/target interfaces.

• Volume, velocity; and variety.

• Delivery requirements.

• Real-time, near real-time, batch/scheduled.

• Loose coupling/reusability.

• Performance/Availability.

8

Real-time Data Integration Patterns

• Transactional Data Processing

• Data Replication

• Data Integration Hub (DIH)

• Event Driven Architecture (EDA)

9

Applications Processes BI Tools Portals Web Services

Batch Trickle Feed Real Time

Applications Databases Messages Flat Files XML UnstructuredData

Mainframe

Data Consumers

Data SourcesOperational Data (Field Devices, Applications, etc.)

LocationContext

(GIS)

BI, DW, Other Targets

Event BasedApplications

Communication “Cloud”

Data Receipt / Transformation & Mapping

Real Time Event Transport / Delivery

Event Processing

Real Time Web Content Delivery

IDRPWC/PX

SourceApplications/Technologies

Interaction management(Publications & subscriptions)

Data Governance

Persistence

Hub Management

ArchivingDelivery & Schedule

CatalogService

Data & Event Monitoring

Validation & Quality Rules

On-boardingMonitoring &

Auditing

Data Transformation & Enrichment

Cloud App (HR)

POS App

CRM

Planning

Master Data

DataWarehouse

Big Data(Analytics)

Finance

Transactional Data Processing

10

1

Operational BI & Real-Time DWProvide freshest information to business w/o impact, off-load reporting, & do high volume updates (ODS/OLTP/DW/Appliance)

2

Operational (Sync)Ensure all users are seeing the same,trusted, up-to-date data that is synchronizedacross the organization.

Transactional/ProductionApplications

Merge/Apply,Reports & Queries

Source System Target Systems

ODS/OLTP/DW/Appliance

TransactionalApplications

Merge/Apply,Reports & Queries

Source System Target Systems

ODS/OLTP/DW

Transactional Data Processing

11

EXTRACT

SERVER

MANAGER

SERVER

MANAGERhttp://

APPLY

Console

Source System Target System

SQL Apply Merge ApplyAudit Apply

Intermediate Files

CommittedCheckpoint

CheckpointHigh SpeedExtraction

High SpeedParallel Apply

JMS*

In-Memory Cache Synchronization

12

Cache Node

Cache Node

Cache Node

Data Sources

Real Time Data

Replication

In Memory Grid

Consuming Applications

Data Integration Hub

13

Data Integration

Hub(Publish/

Subscribe)

Define Topic

Define Publishers

(Connection properties,

Mode, Frequency)

Define Transformation and validation

rules

Define Security &

Access Policies

Define Subscribers (Connection properties,

Mode, Frequency)

Deploy (Instantiate persistence

and data integration

flows)

Monitor, Govern, Alert

& Audit

Cloud App (HR)

POSApp

CRM

Planning

MasterData

DataWarehouse

Big Data(Analytics)

Finance

Data Integration Hub

14

Pu

b.

Ide

nti

fic

ati

on

Da

ta Q

ua

lity

Fir

ew

all

State logging, Error handling

XM

L N

orm

ali

zati

on

Su

b.

Tra

ns

form

ati

on

Sy

nc

-up

Co

ntr

ol

Su

b.

Ide

nti

fic

ati

on

Su

b.

De

liv

ery

Su

b.

Ac

kn

ow

led

ge

me

nt

Command Interface

Monitor and Alert Interface

TransferInterface

Applications Applications

TransferInterface

EnterpriseScheduler

EnterpriseMonitor

File Manager

Connector Connector

Prompt:$ SNMP

Prompt:$

Event Driven Architecture

• An architecture in which the activity is driven by changes in state within an environment.

• Events drive the execution of logic.

15

Ev

en

t S

ou

rce

s (

Pro

du

ce

rs)

De

tec

ted

Situ

atio

ns

(Co

ns

um

ers

)

Inte

gra

tio

n/D

ire

ct In

teg

ratio

n/D

irec

t

Pre-Process

Process

Post-Process

Patterns

Alerts

Event Streams

Actions

Devices

Systems

Applications

People

Sample EDA Reference Architecture

16

Operational Data (Field Devices, Applications, etc.)

LocationContext

(GIS)

BI, DW, Other Targets

Event BasedApplications

Connectivity & Data Replication

Transformation & Mapping

Real Time Delivery

Event Processing

Real Time Web Content Delivery

Event Enablement and Transformation

17

• Event-enable and collect events from underlying systems and applications.• Data Replication

• PowerExchange/PowerCenter

• Transform events to normalized format for downstream processing.• B2B Data Transformation

• Enrich events as needed.

Transformation & Mapping

Connectivity & Data Replication

Event Transport

18

• Deliver transformed events to downstream applications, systems, and event processors.• Ultra Messaging

• Dynamic Routing

• Parallel Persistence

• Guarantees delivery, and decouples event producers from event consumers.

Real Time Delivery

Event Rules and Delivery

19

• Processes events, applies user/business defined rules, and generates responses.

• RulePoint

• Rules may be applied across events and over time/space.

• Alerts/notifications may be sent to BI environments, web applications, etc.

Real Time Web Content Delivery

Event Processing

Architectural Implications

20

Information and Data Architecture

Application Architecture and Development

Infrastructure Architecture

• Shift to modeling events as enterprise assets. Incorporate the processing of immediate, individual events.

• Deliver quality data as it is needed by the business.

• Complement existing SOA strategies with eventdriven applications.

• Move to a single environment that supports different integration delivery models.

• Shift from centralized, database-centric client-server applications to distributed systems.

• Deploy infrastructure that can support data delivery over different latencies.

Summary

• Augment traditional integration solutions with real-time data integration to:

• Reduce decision latency.

• Improve decision making and responsiveness.

• Increase visibility.

• Build upon the principles of SOA and an event-driven architecture (EDA).

• Informatica provides the capabilities for adding real-time data integration to your environment.

21

22

1

2

3

Join our exclusive Potential at Work Communities. Visit the kiosk.

Get a PowerCenter Express demo!Visit the Pavilion kiosk.

Tell us what you think. Click on Evaluations in the IW13 Mobile App.