<Insert Picture Here>

Oracle Data Integrator – Technical Deck

Mark PareSr. Sales ConsultantOracle Higher Education

3

<Insert Picture Here>• 4 Key Differentiators• Architecture• 6 Steps to Production• ODI or ESB?• Popular Usage Scenarios

Agenda

4

Oracle Data Integrator 4 Key Differentiators

5

Data IntegrationData IntegrationData

WarehousingMaster Data Management

Real Time Messaging

FederationMigration

Data in Disparate SourcesData in Disparate Sources

ERP

------------

------

CRM

- - -

Legacy

------------

------ ---------

---

------

Best-of-breed Applications

Information How and Where you Want ItInformation How and Where you Want It

Business IntelligenceCorporate Performance

ManagementBusiness Activity

MonitoringBusiness Process

Management

HAVE…

NEED…

- - -- - - - - -- - -

Data Synchronization

Why Data Integration?

6

Challenges & Emerging SolutionsIn Data Integration

CHALLENGE EMERGING SOLUTION

1. Increasing data volumes; decreasing batch windows

2. Non-integrated integration

3. Complexity, manual effort of conventional ETL design

4. Lack of knowledge capture

Shift from E-T-L to E-LT

Convergence of integration solutions

Shift from custom coding to declarative design

Shift to pattern-driven development

7

Oracle Data Integrator

• Data Movement and Transformation from Multiple Sources to Heterogeneous Targets

1. Performance: Heterogeneous “E-LT”

2. Flexibility: Active Integration Platform

3. Productivity: Declarative Design

4. Hot-Pluggable: Knowledge Modules

BENEFITS KEY DIFFERENTIATED FEATURES

8

Differentiator: E-LT ArchitectureHigh Performance

Conventional ETL Architecture

Extract LoadTransform

Next Generation Architecture

“E-LT”“E-LT”LoadExtract

Transform Transform

Transform in Separate ETL Server• Proprietary Engine• Poor Performance• High Costs• IBM & Informatica’s approach

Transform in Existing RDBMS• Leverage Resources• Efficient• High Performance

BenefitsOptimal Performance & ScalabilityEasier to Manage & Lower Cost

11

9

Enables real-time data warehousing and operational data hubs Services plug into Oracle SOA Suite for comprehensive integration

Oracle Data Integrator

Data-oriented Integration

Event ConductorEvent Conductor

Event-orientedIntegration

Service ConductorService Conductor

Service-orientedIntegration

Declarative Design

Metadata

Data ConductorData Conductor

Differentiator: Active IntegrationBatch, Event-based, and Service-oriented Integration

• Evolve from Batch to Near Real-time Warehousing on Common Platform

• Unify the Silos of Data Integration

• Data Integrity on the Fly• Services Plug into Oracle

SOA Suite

• Benefits

22

10

Differentiator: Declarative DesignDeveloper Productivity

Conventional ETL DesignSpecify ETL Data Flow Graph• Developer must define every step of

Complex ETL Flow Logic• Traditional approach requires specialized

ETL skills• And significant development and

maintenance efforts

Declarative Set-based Design• Simplifies the number of steps• Automatically generates the Data Flow

whatever the sources and target DB• Example: [SALAH]

Benefits Significantly reduce the learning curve Shorter implementation times Streamline access to non-IT pros

ODI Declarative Design

Define How: Built-in Templates

Define What You Want

Automatically GenerateDataflow

11 22

33

11

JournalizeRead from CDC

Source

LoadFrom Sources to

Staging

CheckConstraints before

Load

IntegrateTransform and Move

to Targets

ServiceExpose Data and Transformation

Services

ReverseEngineer Metadata

Tailor to existing best practices Ease administration work Reduce cost of ownership

Reverse

Journalize

Load

Check

IntegrateServices

Pluggable Knowledge Modules Architecture

CDC

Sources

Staging Tables

Error Tables

Target Tables

WS

WS W

S

SAP/R3

Siebel

Log Miner

DB2 Journals

SQL Server Triggers

Oracle DBLink

DB2 Exp/Imp

JMS Queues Check MS Excel

Check Sybase

Oracle SQL*Loader

TPump/ Multiload

Type II SCD

Oracle Merge

Siebel EIM Schema

Oracle Web Services

DB2 Web Services

Sample out-of-the-box Knowledge Modules

Benefits

Differentiator: Knowledge ModulesHot-Pluggable: Modular, Flexible, Extensible

44

12

Oracle Data IntegratorArchitecture

13

• Java design-time environment

• Runs on any platform• Thin client for browsing

Metadata

• Java runtime environment• Runs on any platform• Orchestrates the execution of

data flows

• Metadata repository• Pluggable on many RDBMS• Ready for deployment• Modular and extensible

metadata

Design-Time

Metadata Management

Runtime

Agent

Data Flow Conductor

Service Interfaces and Developer APIs

UserInterfaces

Thin Client

Data Flow Generator

Knowledge Module Interpreter

Knowledge Modules

Master Repository

Work Repositories

Runtime Repositories

Data Flow Generator

Runtime Session

Interpreter

Data Flow

Operator

Designer

Architecture: Conceptual View

14

Architecture: Component View

ODI Design-Time Environment Development Servers and Applications

Design-time Repositories

Code Execution

Execution Log Return Codes

AgentData Flow Conductor

CRM

Legacy

ERP

Data Warehouse

Files / XML

User Interfaces

Administrators

Designers

Topology/Security

Metadata/Rules

Development

ESB

Production Servers and Applications

ODI Runtime Environment

Runtime Repository Return Codes

Code

Execution Log

Execution

Metadata Navigator

Production

CRM

Legacy

ERP

Data Warehouse

Files / XML

ESB

User Interfaces

Administrators

Operators

Thin Client

Data Stewarts

Topology/Security

Execution Log

Metadata Lineage

AgentData Flow Conductor

Scenarios and ProjectsReleases

15

Oracle Data Integrator6 steps to Production

16

ODI Design-Time Environment ODI Runtime Environment

User Interfaces

Overview: 6 steps to Production

1. Retrieve/Enrich metadata

2. Design transformations

3. Orchestrate data flows

4. Generate/Deploy data flows

5. Monitor executions

6. Analyze impact / data lineage

Development

Development Servers and Applications

AgentData Flow Conductor

CRM

Legacy

ERP

Data Warehouse

Files / XML

User Interfaces

Administrators

Designers

ESB

Design-time RepositoriesDesign-time

Repositories

Production

Production Servers and Applications

AgentData Flow Conductor

CRM

Legacy

ERP

Data Warehouse

Files / XML

Operator

Metadata Navigator

ESB

Runtime Repository

17

1. Reverse-engineer Metadata• Automatic• Customizable• 40+ technologies supported

2. Enrich Metadata• Documentation• Declarative rules for Data

Integrity• Cross-technologies references

Design-Time Environment

ODI Designer

Design-time Repositories

Development Servers and Applications

CRM

Legacy

ERP Data Warehouse

Files / XML

ESB

Retrieve/Enrich Metadata11

18

Oracle Data Integrator “Interface”

Declarative Design

11 Define What You Want 33 Automatically GenerateData flows

22 Define How to Do It: Select Template

Bulk Load • Changed Data Capture • Incremental Update • Slowly Changing Dimension

Design Transformations22

19

1. Sequence Transformations

2. Leverage OracleDI Tools• Data Quality Processes• Files/Archives Management• Send/Receive Emails• Web Services Invokation• Event Detection• Create your Own Tools

3. Use Control Structures• Loops• Conditions• Error Handling

33 Orchestrate Data Flows

20

1. Create Scenarios• Compile Data Flows for Run-time

2. Version the Data Flows• Advanced Version Management

3. Deploy to ProductionDesign-time Repositories

Scenarios and ProjectsReleases

Runtime Repository

Generate and Deploy Data Flows44

21

• View sessions running in real-time

• Review generated code• Detailed run-time statistics• Restart failed sessions

55 Monitor Executions

22

Analyze impact / data lineage

• Maintain a large number of data flows in a complex environment

• Web-based end-to-end data lineage1. Understand your data flows2. Follow the path of data3. Drill-down to transformations

?

66

23

Oracle Data IntegratorODI or ESB?

24

What tool is best suited for task X?

Requirement ESB

ODI Recommended

Latency / VolumeSynchronous Integration ESB

Asynchronous Integration with routing and transformation ESB

Asynchronous Integration for Active Data Warehousing (mini-batch) ODI

Batch Integration with High Volume ODI

TransformationsIn-memory XSLT Transformations (XML to XML) ESB

Transformations in App Server ESB

Transformations in Database (E-LT) ODI

Integration TopologyData Warehouse Loading (E-LT) ODI

JMS to JMS ESB

JMS to DB/App with routing and transformation (real-time or synchronous) ESB

JMS to DB/App with bulk transformation (mini-batch) ODI

DB/App to DB/App (batch or mini-batch with CDC) ODI

DB/App to DB/App (synchronous or real-time with CDC Adapters) ESB

…

25

Oracle Data Integrator

ESB and ODI in real-life scenarios

Data Volume Processing

Data Latency

Message by Message

Mini Batches Large Volume(over 1M)

Synchronous(immediate)

Asynchronous

Batch(over 2 hours)

Oracle Enterprise Service Bus

Real-life

Scenarios

26

Oracle Data IntegratorExtended Capabilities

27

Extended Capabilities

• Master Data Management enabled• Common Format Designer• Automated generation of canonical format and transformations• Built-in Data Integrity

• Real-time enabled• Changed Data Capture• Message Oriented Integration (JMS)

• SOA enabled• Generation of Data Services• Generation of Transformation Services

• Extensibility• Knowledge Modules Framework• Scripting Languages• Open Tools

28

Oracle Data IntegratorMaster Data Management Enabled

29

MDM Enabled: Canonical Format Design

• Use in conjunction with packaged MDM solution

• Design and Populate Canonical Format1. Use existing metadata artifacts

to design MDM application (entities, fields, relationships)

2. Generate and maintain Master Data structure

3. Generate and deploy transformations using metadata artifacts

CRM

Enterprise Service Bus

SCM Legacy ERP

Master Data

30

MDM Enabled: Built-in Data Integrity

• Data Integrity Firewall• Auditing, cleansing and

recycling

1. Declare constraints at table level2. Design mappings and check

flow integrity3. Audit, cleanse or recycle

rejected records

Message Id Name City

Duplicated Record 001 John Doe New York

Duplicated Record 022 John Doe Boston

Invalid City Reference 230 Albert Fresh Maris

31

Oracle Data IntegratorReal-time Enabled

32

Real-time enabled: Changed Data Capture

• Publish and Subscribe CDC Framework• Database logs• Triggers• Third-tier solutions

• Ensures “read” transaction integrity across multiple tables

1. Design or generate Mappings

2. Select Journalized Data Only

3. Start Journals

CDC

33

Real-time enabled: Message Oriented Integration

• Connect to Publish and Subscribe JMS Message Providers

• Ensure messages delivery with transaction integrity

• High-volume bulk transformations1. Design complex bulk

transformations mixing Queues, Databases and Applications

2. Use JMS Queues and topics as sources or targets

JMS Provider(MOM, ESB)

CDC

Subscribe

Publish

34

Oracle Data IntegratorSOA Enabled

35

SOA Enabled: Data Access Services

• Generate and share data access services

1. Generate and deploy data services2. Test data services3. Leverage data services in your

SOA infrastructure

SOA Infrastructure

Services

Data Access

Transform

Business

Business Processes

ESB

36

SOA Enabled: Data Flow Services

• Expose transformations as Web Services

1. Orchestrate data flows2. Publish data flows as web services

in your SOA infrastructure

SOA Infrastructure

Services

Data Access

Transform

Business

Business Processes

ESB

Bulk Transf

.

37

Oracle Data IntegratorExtensible Framework

38

JournalizeRead from CDC

Source

LoadFrom Sources to

Staging

CheckConstraints before

Load

IntegrateTransform and Move

to Targets

ServiceExpose Data and Transformation

Services

ReverseEngineer Metadata

Reverse

Journalize

Load

Check

IntegrateServices

Pluggable Knowledge Modules Architecture

CDC

Sources

Staging Tables

Error Tables

Target Tables

WS

WS W

S

Extensibility: Knowledge Modules

• 120+ KMs out-of-the-box Tailor to existing best practices Ease administration work Reduce cost of ownership

• Customizable and extensible

KMInterpreter

KM’s Meta Code

Metadata

Executed Code

39

Extensibility: Scripting Framework

• Extend data flows with scripting procedures

• Leverage all database languages

• SQL, PL/SQL, Transact SQL, etc.

• Use Operating Systems shell scripts

• Win32 DOS, sh, ksh, csh, OS400 commands, JCL, etc.

• Choose from compatible Bean Scripting Framework languages

• Java, JavaScript, Jython (Java Python), Perl, etc.

40

Extensibility: Open Tools

• Extend ODI tools• Add your own tools to the

Design Palette1. Implement OdiOpenToolAbstract

Java Interface2. Register Open Tool in ODI Designer3. Use Open Tool in your design

environment

41

Popular Usage Scenarios

42

E-LT for Data WarehouseCreate Data Warehouse for Business Intelligence Populate Warehouse with High Performance ODI

Heterogeneous sources and targets

Incremental load Slowly changing

dimensions Data integrity and

consistency Changed data capture Data lineage

Data Warehouse

Cube

Cube

Cube

----------------

Op

era

tion

al

An

aly

tics

Metadata

LoadTransformCapture Changes

Incremental UpdateData Integrity

AggregateExport

43

SOA InitiativeEstablish Messaging Architecture for Integration Incorporate Efficient Bulk Data Processing with ODI

Invoke external services for data integration

Deploy data services Deploy transformation

services Integrate data and

transformation services in your SOA infrastructure

Services

Data Access

Transformation

Others

----------------

Op

era

tion

al

Metadata

Generate DataServicesExpose Transformation Services

Deploy and reuse Services

Business Processes

44

Master Data ManagementCreate Single View of the TruthSynchronize Data with ODI

Use in conjunction with packaged MDM solution

Use as infrastructure for designing your own hub

Create declarative data flows Capture changes (CDC) Reconcile and cleanse the

data Publish and share master data Extend metadata definitions

Master Data

----------------

Metadata

Change Data CaptureMaster Data Load

Canonical Format DesignCleansing and Reconciliation

Master Data Publishing

----------------

CDCCDC

CDC

45

MigrationUpgrade Applications or Migrate to New Schema Move Bulk Data Once and Keep in Sync with ODI

Bulk-load historical data to new application

Transform source format to target

Synchronize new and old applications during overlap time

Capture changes in a bi-directional way (CDC)

Old

Ap

plic

atio

ns

Ne

w A

pp

lica

tion

Metadata

Initial bulk loadCDC for synchronization

Transformation to new application format

CDC for loop-back synchronization

CDC

CDC----------------

46

ODI Enhances Oracle BIPopulate Warehouse with High Performance ODI

Oracle Business Intelligence Suite EE:

Simplified Business Model View Advanced Calculation &

Integration Engine Intelligent Request Generation Optimized Data Access

Oracle Data Integrator: Populate Enterprise Data

Warehouse Optimized Performance for Load

and Transform Extensible Pre-packaged E-LT

Content

Siebel CRM

Oracle EBSPeopleSoftSAP/R3

Other Sources

Oracle Data Integrator

E-LT Metadata

E-LT Agent

Oracle BI Enterprise Data

Warehouse

Oracle BI Suite EE

Oracle BI Server

Oracle BI Presentation Server

Answers Interactive Dashboards Publisher Delivers

Bulk E-LT

47

ODI Enhances Oracle SOA SuiteAdd Bulk Data Transformation to BPEL Process

Oracle SOA Suite: BPEL Process Manager for

Business Process Orchestration

Oracle Data Integrator: Efficient Bulk Data Processing

as Part of Business Process Interact via Data Services and

Transformation Services

Oracle SOA Suite

Business Activity Monitoring

Web Services Manager

Business Rules Engine

Enterprise Service Bus

BPEL Process Manager

Bulk Data Processing

Oracle Data Integrator

E-LT Metadata

E-LT Agent

48

ODI Enhances Oracle SOA SuitePopulate BAM Active Data Cache Efficiently

Oracle SOA Suite: Business Activity Monitoring

for Real-time Insight

Oracle Data Integrator: High Performance Loading of

BAM’s Active Data Cache Pre-built and Integrated

Data Warehouse

Oracle SOA Suite

BPEL Process Manager

Web Services Manager

Business Rules Engine

Enterprise Service Bus

Bulk and Real-Time

Data Processing

SAP/R3PeopleSoft

Message Queues

CDC

Business Activity Monitoring

Active Data Cache

Event Engine Report Cache

Event Monitoring Web Applications

Oracle Data Integrator

MetadataAgent

49

Roadmap and Direction

50

Oracle Data Integrator: Roadmap

• Focus Areas for Next Major Release• Deep Integration with Fusion Middleware

• Runtime, Design time, Security, Administration, Events

• Functional Integration with Oracle Warehouse Builder• Runtime Integration, Metadata Sharing, Knowledge Module Sharing

• Deployment of ODI for Embedded Data Integration• OracleBI Enterprise Edition, Data Hubs, Application Migrations

• Enhanced Usability and Debuggability• Wizards, New Views, User-definable Debugging

• Improved Support for Native Oracle Database Features• Oracle OLAP

51

ODI Statement of Direction

• Statement of Direction• http://www.oracle.com/technology/products/odi/statement-of-direction.pdf

• Key Points of Direction• Commitment to heterogeneous systems support

• Including: DB2, Teradata, Netezza, Hyperion, etc.

• Commitment to Fusion design principles• Including: J2EE compliance, container portability

• Commitment to best-of-class E-LT performance• Across platforms, batch & realtime, high complexity