odi overviewlatest
TRANSCRIPT
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 1/51
Oracle Data Integrator
Kiran Kumar R, PMP
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 2/51
INTRODUCTION
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 3/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 4/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 5/51
5
Data Movement and Transformation from MultipleSources to Heterogeneous Targets
BENEFITS KEY DIFFERENTIATED FEATURES
1. Performance Heterogeneous “E-LT” 2. Flexibility Event-Driven Platform
3. Productivity Declarative Design
4. Open 100% Java and SOA Native
5. Hot-Pluggable Knowledge Modules
Customer Value PropositionStrategy: Best-of-Breed Core Architecture & Foundation
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 6/51
Previously Known as “Sunopsis”
• Sunopsis were an ETL tool vendor originally from France,
with offices in the USA and UK• Two major products
– Sunopsis Data Conductor (database integration)
– Sunopsis Active Integration Platform (event and serviced-based
integration)
• Sold on it’s platform independence, Java architecture and EAIcapabilities
• Acquired by Oracle in 2006
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 7/51
ODI Positioning
• Not a replacement for Oracle Warehouse Builder
– OWB has Oracle data warehouse modeling features
– OWB is the obvious choice when target is an Oracle database
• ODI supplements OWB in the following scenarios:
– Target database is non-Oracle (DB2, Teradata, XML, SQL Server etc)
– Sources include Web Services, XML, SOAP etc
– Data integration is complex / real-time
– Multi-platform CDC / SCD / Bulk load and insert / MDM and Data Hubs
– Batch windows require bad source data to be filtered out before loading
• Adds functionality to the Oracle DW ETL stack
– ODI and OWB likely to code-share/merge in future
– In short term, possible replacement for OWB Enterprise ETL Option
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 8/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 9/51
Market Position
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 10/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 11/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 12/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 13/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 14/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 15/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 16/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 17/51
Overview of ODI Studio
17
Navigatorsfor eachfunctionalarea.
ODI operations integrated inIDE: Menu, Gallery, etc.
Intuitive multi-editors supportclose all/save all, docking,etc.
Property inspectors,thumbnails and IDE windowsfully used as needed.
Accordionsfor eachsub-area.
Popup menusand treemarkers.
Interface Editor usingdiagraming.
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 18/51
Interface Editor / Quick Edit
18
Same InterfaceTwo views
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 19/51
Interface Editor
19
Diagram zoomand autolayout.
Datasets management
Lookup wizard
Errors and Autofix.
Lookup
Datasets appear here.
Propertiesinspector forselection.
Sortable andSelectable columns.
DnD from sources tomapping.
Thumbnail: bird’seye view of the
diagram.
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 20/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 21/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 22/51
22
Oracle Data Integrator
SOA Enabled
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 23/51 3
SOA Enabled: Data Access Services
• Generate and share data
access services1. Generate and deploy data services
2. Test data services
3. Leverage data services in your SOA
infrastructure
SOA Infrastructure
Services
Data Access
Transform
Business
BusinessProcesses
ESB
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 24/51 4
SOA Enabled: Data Flow Services
• Expose transformations as
Web Services1. Orchestrate data flows
2. Publish data flows as web services
in your SOA infrastructure
SOA Infrastructure
Services
Data Access
Transform
Business
Business
Processes
ESB
Bulk
Transf.
Extended Capabilities
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 25/51
Extended Capabilities
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 26/51
• Same philosophy as OWB – Use the Database as the ETL engine
– ODI supports heterogeneous databases, not just Oracle
• Built for SOA environments
– Support for Web Services etc.,
• Supports batch, event-based and real-time integration
• Data Integrity Controls create a data “firewall”
– Reduces data prep time by not processing erroneous data• Extensible through “Knowledge Modules”
– Change Data Capture
– Slowly Changing Dimensions
– Bulk load
Data Integrator Overview
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 27/51
Solution Talk – A Case Study
27
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 28/51
Solution Talk – A Case Study
• Industry
• BFSI
• Customer Profile• Our customer is the largest BFSI company in India, providing a wide range of
banking operations and property and casualty insurance products together withaccidental injury insurance and short-term health insurance products
• With 31 sub-companies distributed across the country, this company has beendeveloping a complicated IT infrastructure and an analysis system with a hugevolume of data
• All the database systems are Informix• Business Problem
• KPI reports are significantly delayed due to poor system performance, leading tothe KPI system unreliable for decision-making
• The current system is too overloaded to provide internal users with stableservice based on existing business, not mentioning supporting new business
• Maintenance cost is augmenting due to sub-optimal DW development approach,which has become a manifest issue during the economic downturn
• Product Used
• Oracle Database Enterprise Edition 11.1.0.6 with RAC, Partitioning Options
• Oracle Data Integrator 10.1.3.4
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 29/51
Solution Talk – What Challenges
Were Responded to ??
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 30/51
Customer Challenges -Existing System
Architecture
• Distributed IT infrastructure
• 31 OLTP Informix DBs aredistributed across India, with a
central Informix DB
• Performance-thirsty
transformation
• Data is extracted (from 31 Informixinto ODS) and transformed (finally
into Report Data Layer) in real time
or periodically.
• Non-standardized
implementation• Data extraction and transformation
are implemented via code such as
C,SQL, etc, no ETL tool here.
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 31/51
Solution Talk – What did We
Achieve ??
Copyright © 2008, Oracle and/or its affiliates. All rights reserved. 31
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 32/51
Achievements – ODI Topology Manager
With ODI Topology Manager, wecan easily define all kinds ofdata sources and targettechnologies.
In our case, multiple Informixdatabases can be defined asdata sources thereby UI friendlydata consolidation is achieved.
Here we defineInformix as datasource
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 33/51
Achievements – ODI Designer
Here we designETL processing
ODI Designer is an integratedGUI for users to design ETLprocesses.
In our case, complicated
transformations can be definedin a high efficient manner. Highlyoptimized codes are generatedautomatically by tool.
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 34/51
Achievements – ODI Operator
With ODI Operator, we can
monitor and execute the ETLprocesses conveniently.
In our case, ODI Operator helpsimprove the maintenanceefficiency. For instance, userscan identify the ETL steps witherrors or warnings and pinpointthe problems efficiently.
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 35/51
Key Benefits - Delivered
Key Focus Areas Benefits
High Performance
No analytical report is delayed
The overall performance is dramatically improved and the
hardware resources are utilized with more efficiency
Additional resource is available for processing new business
ETL Efficiency
New business can be developed timely
Distributed data can be easily consolidated
Data processing is easily and flexibly implemented
Transformation code is optimized with high performance
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 36/51
Solution Talk – What did We
Propose ??
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 37/51
Oracle Proposed Solution – System Architecture
• Oracle is introduced as central DB
• As the most powerful DB, Oracle
DW approach helps our customerto solve current performance
problems and maximize system
availability.
• ODI is introduced as ETL tool
• As the most efficient ETL tool,
ODI integrates data from Informix
into Oracle, and subsequent datatransformation,with friendly UI
and efficient code.
• OBIEE would be further introduced
• Other Oracle DB for the
distributed sub-systems and BIEE
for reporting would be further
introduced, concerning that oursolution has successfully solved
current key problems.
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 38/51
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 39/51
Oracle Proposed Solution – How ODI Integrates Data from Informix
• ODI access Informix via JDBC using
Informix Driver. Connections are
managed within Topology, one
component of ODI.
• Complicated transformations are
defined within Designer(the design-time environment of ODI) ,
transformations such as extraction
from Informix(via JDBC), data
aggregation(via SQL).
– Transformations are executed from
Agent(the run-time environment of
ODI), which calls Java,SQL,etc., to
run ETL processes using workflow.
– Transformations are monitored from
Operator(the run-time logger of ODI),which traces each step of the
transformation.
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 40/51
Solution Talk – What Potential This
Solution Has ??
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 41/51
Solution Talk – A Case Study
41
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 42/51
Copyright © 2008, Oracle and/or its affiliates. All rights reserved.4
iTech Solution Profile
Existing Pain
LL RTO’S
SCHEMAS
State
Consolidation
Register
Manual Data
Replication process
STATE LEVEL
RTO’S
SCHEMAS
State Register
Manual Data
Replication process
National Level
RTO Schema
Manual Data Replication
process for Huge volumes
of data
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 43/51
Solution Talk – A Case Study
43
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 44/51
Copyright © 2008, Oracle and/or its affiliates. All rights reserved.4
Existing Data Flow @ HDFC Std Life
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 45/51
Problems with current data flow
• Performance is poor
• Multi step integration – CL + FTP + OWB• More Maintenance effort
• Difficult to incorporate development Best Practices
• No Unified monitoring and scheduling
Current Integration Flow
ODI can make this step as
straight through
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 46/51
Copyright © 2008, Oracle and/or its affiliates. All rights reserved.4
• The proposed data integration solution is based on Oracle DataIntegrator (ODI) product.
• Oracle Data Integrator provided a unique data integration platform, thatwas designed for performance with a high degree of flexibility andmodularity.
Benef i ts
• Performance – Leveraged a high performance E-LT architecture.:
• Productivity- The in-built declarative design was utilised which in-turndrives reusability, accelerates development and maintenance.
• Flexibility – A single platform was presented that natively supportsseveral data integration styles and latencies.
• Modularity – Hot-pluggable knowledge modules were used to bettersupport application and database platform diversity.
Oracle’s Data Integration Solution
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 47/51
47
Advantages of using ODI
• High Performance – using Bulk set operations
• End-to-End integration within ODI
• Easy to maintain because of Repository based architecture and
Impact analysis features• Best Practices can be productized into Knowledge Modules
• End-to-End data lineage features to enable report to source data
lineage
• CDC mechanism for DB2/400, SQL Server and Oracle
Proposed Data Integration Flow
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 48/51
Copyright © 2008, Oracle and/or its affiliates. All rights reserved.4
ODI Screenshots
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 49/51
Proof of concept: Results• Performance Improvement
• NOTE- Execution Times mentioned for OWB does not include the time taken
to extract file from DB2 and FTP the same to Oracle Data Warehouse Server.This currently takes 45 minutes for all mappings put together. ODI accesses
DB2/400 tables directly and does not require an FTP step.
Copyright © 2008, Oracle and/or its affiliates. All rights reserved.4
No. ofRecords
ODI(Secs)
No. ofRecords
OWB(Secs)
%Improvement
ACTIVITY 82911 36 83561 118 ~ 320 %
POLICY_ACCOUNT_DD
533833 718 545028 1693 ~ 230%
S
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 50/51
Summary• Oracle Data Integrator is the new addition to Oracle’s Middleware toolset.
• ODI-EE is the only product that combines the best performance, flexibility
and cost-savings alternatives for data integration business needs. The E-
LT foundation provided by ODI-EE offers Oracle customers a way toleverage existing database
• (Oracle and non-Oracle) investments without sacrificing world-class
functionality that they would expect from a best-of-breed data integration
platform.
• Complements OWB by adding advanced data integration
Functions
– Works in the staging/integration layer
– Pre-built modules for CDC, bulk loading etc
– Supports event-based, service-based and real-time integration
• Adds the ability to target heterogeneous platforms
8/11/2019 ODI OverviewLatest
http://slidepdf.com/reader/full/odi-overviewlatest 51/51
Demonstration