Oracle Data Integrator A Success Story

Jason Jones Key Performance Ideas, Inc.

About Jason Jones

l  Oracle Essbase Certified Developer – experience since 2005

l  Oracle Data Integrator since 2009 l  Extensive experience in developing, tuning and

enhancing Essbase, Hyperion Planning, and ODI

l  Programming expertise: ● Developing software in Java ● Mobile solutions for the iOS platform (Objective-C) ● Relational databases (SQL Server, Oracle) 2

Agenda

l  Company Situation l  What is Oracle Data Integrator? l  Problems & Pain Points l  Solutions & Opportunities l  ODI Functionality & Benefits l  Lessons Learned l  How to Get Started l  Q&A

3

Company Situation

l  Large health services company l  Significant Oracle database implementation l  In process of implementing Essbase l  Transitioning to new database servers l  Data movement processes have grown

organically over the years

What is Oracle Data Integrator?

l  History ● Originally developed by Sunopsis, acquired in 2006 ● Originated Extract Load Transform (ELT) as

alternative to ETL ● Data movement between heterogeneous data

sources l  Future

● Strategic product for data integration ● Significant development resources

ETL vs. E-LT

Source ETL Server Target

Target

ETL Server

Traditional Architecture

E-LT Architecture

Source

Problems & Pain Points

l  SQL procedures being used as de facto ETL solution l  Time-consuming to develop l  Hard to maintain l  Little error visibility l  Difficult to integrate with other data sources l  Have to maintain in multiple environments l  Manual logging facility l  Lacking documentation/comments l  Difficult to troubleshoot

Solutions & Opportunities

l  Easier development and maintenance l  Work with different relational database

technologies l  Load to/from text files, Essbase/Planning l  Email alerts l  Easily switch between physical environments l  Process logging, easily pinpoint errors

Essbase

Typical Data Flow

ODI Server

Data Movement OLTP

(Oracle)

Operational Data Store

(Oracle)

Knowledge Modules

l  RKM: Import existing table/data structure l  LKM: Load data from tables l  IKM: Insert/update data in target table l  JKM: Process new rows of data only l  CKM: Validate data being moved

Models

•  Create models for relational tables, text files, and Essbase dimensions to load data to and from

•  Easily create models automatically based on existing tables with RKM

•  Serve as the source and target of interfaces

Enhanced Functionality

•  Enforce data integrity! •  Exist as logical units irrespective of physical environment

Benefits

•  Define requirements for data entering target tables •  Reuse for multiple interfaces

Health Services Organization Value Achieved

Model Datastore

12

Interfaces

•  Moves data between models •  Source is typically multiple models (tables) •  Destination is always one and only one model (table) •  Declarative!

Enhanced Functionality

•  Easy to create, modify, and maintain •  Work across different technologies •  Document business rules •  Leverage power of existing database server to transform data •  “Auto mapping” speeds development time

Benefits

•  Visual replacement for many SQL procedures •  Quickly define criteria for updating tables •  Choose location of data transformation

Health Services Organization Value Achieved

Interface Screen

14

Interface Flow

•  Validate data being processed •  Automatically recycle data •  Choose loading/integrating strategy

Enhanced Functionality

•  Very easily configure complex actions that would normally be tons of code

•  Select Knowledge Modules (strategy used to load, integrate, validate)

Benefits

•  Turn on data validation, error recycling with mouse click

Health Services Organization Value Achieved

Interface Flow Screen

16

Procedures

•  Useful for performing one-off actions if they can’t be put into interface

•  e.g. call a stored procedure

Enhanced Functionality

•  Perform action that doesn’t fit into an interface

Benefits

•  Reference existing data transformation process without having to rebuild it

Health Services Organization Value Achieved

Interface Overview

18

Procedure Steps

19

Procedure Step Definition

20

Packages

•  Chain together multiple interfaces, procedures, and other steps •  Allow for error control flow

Enhanced Functionality

•  Gracefully handle errors •  Restart process automatically, if desired •  Send an email alert!

Benefits

•  Treat execution of multiple interfaces as one job •  Add email step in case of error (or success)

Health Services Organization Value Achieved

Package Flow

22

Scenarios

•  “Freezes” an interface, procedure, or package in place •  Changes can be made to procedure/interface/package that won’t

affect existing functionality •  Scenarios are unit of work that can be scheduled and called from

command-line

Enhanced Functionality

•  Avoid breaking existing processes when need arises to change/augment functionality in interface

Benefits

•  Call ODI functionality from command-line when needed •  Deploy functionality and not be scared to make changes to it

Health Services Organization Value Achieved

Generated Scenarios

24

Operator

•  Easily view status of all jobs •  Scenarios, packages, interfaces, procedures, load plans •  Replaced need to have manual logging statements

Enhanced Functionality

•  See exactly where and why a process failed

Benefits

•  Insight into currently executing and already executed jobs •  Drill down to exact cause and reason of error

Health Services Organization Value Achieved

Operator Overview

26

Operator Step Detail

27

Operator Step Generated Code

28

Scheduler

•  More robust than Windows Task scheduler •  Easy to set schedule for jobs to run •  Can call jobs from command-line but use scheduler if possible!

Enhanced Functionality

•  Easier to use than setting up Windows Task Scheduler to run a batch file to run a scenario…

Benefits

•  Directly schedule ODI job to run without having to setup batch file •  Run ODI jobs without needed additional deployment step

Health Services Organization Value Achieved

Scheduler

30

Journalized Data

•  Pattern for only consuming updated/inserted rows of data •  Easy to implement •  Single checkbox in interface for using journalized data

Enhanced Functionality

•  Avoid processing all data, block of data by day •  Avoid maintaining timing variables

Benefits

•  Get away from “day of data” processing paradigm •  Move data from one system to another more often

Health Services Organization Value Achieved

Journals

32

Journalized Data in Interface

33

Topologies

•  Logical “Customer” system •  Physical Customer system in Development, QA, Production

Enhanced Functionality

•  Use same logical job for both development and production environments

Benefits

•  Save significant effort not having to copy/deploy code only differing by environment

•  Pick and choose test/production systems

Health Services Organization Value Achieved

Lessons Learned

l  Critical to leverage someone who has done this to lay foundation! Leverage an experienced individual to set critical first steps up correctly

l  Start with simple ETL job and build a roadmap to larger data movement needs

l  First step/phase can be large, subsequent jobs much easier l  Don’t re-implement functionality, build idiomatic ODI jobs l  Always try to use iterative development model l  Reusability, consistency, maintainability l  Implementation duration l  What to look for

How To Get Started

l  ODI is now Oracle’s data movement tool standard l  Serious thought must be made to implement platform -

initial step is typically an architectural discussion l  Utilize experts to help build a roadmap and identify new

idiomatic ODI functionality l  Initial functionality development

● Software install, physical topologies, logical topologies, models reverse engineered

●  Interfaces, procedures, packages, scenarios, solutions

37

WALKTHROUGH OF SQL TO SQL INTERFACE STEPS

Drop Work Table

drop  table  ZODI.C$_0DIM_PETS  

38

Lock Journalized Table

update  ZODI.J$PETS    set  JRN_CONSUMED  =  '1'    where  (1=1)    And  JRN_SUBSCRIBER  =    ‘LAB_RESULTS_TRANSFER_PETS’

39

Create View/Table on Source

create  or  replace  view  ZODI.C$_0DIM_PETS  (  

 C1_PET_ID,  …  )  as  select  *  from  (    select    

 PETS.PET_ID  C1_PET_ID,  …  from  ZODI.JV$PETS      PETS  where  (1=1)    And  JRN_SUBSCRIBER  =  ’LAB_RESULTS_TRANSFER_PETS’    

40

Drop Synonym on Target

drop  synonym  ESSBASE.C$_0DIM_PETS  

41

Create Synonym on Target

create  synonym  ESSBASE.C$_0DIM_PETS  for      ZODI.C$_0DIM_PETS@ZOASIS  

42

Drop Flow Table

drop  table  ESSBASE.I$_DIM_PETS      

43

Create Flow Table

create  table  ESSBASE.I$_DIM_PETS  (  

 PET_ID  NUMBER(38)  NULL,    PET_NAME  VARCHAR2(32)  NULL,  …    JRN_FLAG  VARCHAR2(1)  NULL,    JRN_DATE  DATE  NULL,    DIM_PET_ID  NUMBER  NULL    ,IND_UPDATE    char(1)  

)  NOLOGGING  

44

Insert Flow Into Table

insert  /*+  APPEND  */    into  ESSBASE.I$_DIM_PETS    (    PET_ID,  …………    ,IND_UPDATE    )  

select            C1_PET_ID,  

from  ESSBASE.C$_0DIM_PETS  where  (1=1)  

45

Analyze Integration Table

begin          dbms_stats.gather_table_stats(  

 ownname  =>  'ESSBASE',    tabname  =>  'I$_DIM_PETS',    estimate_percent  =>  

dbms_stats.auto_sample_size          );  end;  

46

Synchronize Deletions from Journal Table

delete  from  ZSTAGING.DIM_PETS  where  exists  (  

 select    'X'    from  ESSBASE.I$_DIM_PETS        I    where  ZSTAGING.DIM_PETS.PET_ID  =  

I.PET_ID    and  IND_UPDATE  =  'D'    )  

47

Create Index on Flow Table

create  index  ESSBASE.I$_DIM_PETS_IDX  on    ESSBASE.I$_DIM_PETS  (PET_ID)  NOLOGGING  

48

Merge Rows

merge  into  ZSTAGING.DIM_PETS  T  using  ESSBASE.I$_DIM_PETS  S  on  (  

   T.PET_ID=S.PET_ID    )  

when  matched  then  update  set  

 T.PET_NAME  =  S.PET_NAME,  …  when  not  matched  then  insert  

 (    T.PET_ID,…  

49

Commit Transaction

/*commit*/  

50

Drop Flow Table

drop  table  ESSBASE.I$_DIM_PETS    

51

Cleanup Journalized Table

delete  from  ZODI.J$PETS  where  JRN_CONSUMED  =  '1'    And  JRN_SUBSCRIBER  =  ‘LAB_RESULTS_TRANSFER_PETS'  /*  AND  JRN_DATE  <  sysdate  */  

52

Drop Synonym on Target

drop  synonym  ESSBASE.C$_0DIM_PETS  

53

Drop View/Table on Source

drop  view  ZODI.C$_0DIM_PETS  

54

Jason Jones Direct: 206.427.1373 Email: jjones@KeyPerformanceIdeas.com

Thank You!