ukoug tech 15 - migration from oracle warehouse builder to oracle data integrator - eurocontrol case...
TRANSCRIPT
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Rittman Mead ConsultingEurocontrol - Migration from OWB to ODI
UKOUG Tech 2015 - BirminghamJérôme Françoisse
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Jérôme Françoisse
•Consultant for Rittman Mead‣Oracle BI/DW Architect/Analyst/Developer
•ODI Trainer•Providing ODI support on OTN Forums•ODI 12c Beta Program Member•Blogger at http://www.rittmanmead.com/blog/•Email : [email protected]•Twitter : @JeromeFr
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Rittman Mead
•Oracle Gold partner with offices in Europe, US (Atlanta)
•UKOUG Partner of the year in 4 categories in 2015
•Provide consulting, training, global managed services for customers around the world
•80+ consultants including 1 Oracle ACE Director and 2 Oracle ACEs
•All expert in Oracle BI, DW, EPM and Analytics tech
•Skills in broad range of supporting Oracle tools: OBIEE, OBIA, ODIEE, Essbase, Oracle OLAP, GoldenGate, Exadata, Endeca
•Blog : http://www.rittmanmead.com/blog/•Twitter : @rittmanmead
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Agenda
•OWB & ODI•Migration Utility•Migration steps•Migration execution•Fixing•Opportunities
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
History of OWB & ODI
•2000 - First release of OWB•2006‣Oracle buys Sunopsis - ODI 10g‣OWB Standard and Enterprise Editions
•2008 - Roadmap to merge the products•2009 - Oracle Data Integrator Enterprise Edition•2010 - ODI 11g•2013‣ODI 12c‣OWB is not shipped with Oracle DB 12c ‣Last OWB release - 11.2.0.4
•2015 - End of standard support for OWB
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Similarities
•Leverage database capabilities as data transformation engine•Objects are stored in the database•Flow-based development•Can use the ELT paradigm
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Flow-based
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ETL vs ELT
•Reduce network transfer•Reduce hardware costs•Set-based operations
•Leverage target platform capabilities•Referential Integrity Check
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ODI Differences
•Heterogeneous (Oracle, IBM DB/2, SQL Server, Flat files, Webservices, Hadoop, Cloud Apps, …)
•No deployment, the code is generated at run-time•Jobs are executed by agents•Repository stored on any RDBMS (not only Oracle)•Separation between Physical and Logical implementation•Customisation and Reusability thanks to Knowledge Modules•Parallelism•Real-time loading
create table<%=odiRef.getTable("L", "INT_NAME", "A")%>(<%=odiRef.getColList("", "[COL_NAME]\t[DEST_WRI_DT] " + odiRef.getInfo("DEST_DDL_NULL"), ",\n\t", "", "INS or UPD")%>)
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ODI Differences
•Heterogeneous (Oracle, IBM DB/2, SQL Server, Flat files, Webservices, Hadoop, ...)
•No deployment, the code is generated at run-time•Jobs are executed by agents•Repository stored on any RDBMS (not only Oracle)•Separation between Physical and Logical implementation•Customisation and Reusability thanks to Knowledge Modules•Parallelism•Real-time loading
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration Utility
•Needs OWB 11.2.0.3+•Patch OWB 17830453or 21977765 (12.2.1)
•ODI 12.1.2 : Patch 17053768•ODI 12.1.3+ : No Patch•Based on ODI SDK•Command Line Interface•3 modes :‣Fast Check‣Dry Run‣Run
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration Utility
•Supported‣Module / Location Model and Topology‣All data objects (except sequences) Datastores‣Mapping (Classic, Code-template and Pluggable) Mapping
•Unsupported‣Process flows‣PL/SQL objects (procs, packages, ...)‣Oracle Discoverer / OBIEE metadata‣Queues, Streams, CDC, User Defined Types‣Dimension and cube, cursors‣Data quality, data profiles, data auditors‣Configuration details (security, OMB scripting, ...)
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration steps• Installing Oracle DB 12c• Installing ODI 12.1.2 and patch 17053768•Plug the homebrew scheduler in ODI•Upgrading a clone OWB repository to 11.2.0.4 and apply patch 17830453•Execute migration utility in Dry Run mode•Fix issues on OWB side• Installing ODI 12.1.3•Build Knowledge Modules for missing loading strategies.•Execute migration utility in Run mode•Clean up topology•Synchronise Test environment and Dev environment•For each mapping : Change knowledge modules, remove unnecessary objects, unit test
•Creation of Load Plans•Testing
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
OWB Architecture
DevelopmentRepository
Deployment
DEV
PROD
DDR_SRC
DDR_STG
DDR
SWH_SRC
SWH_STG
DDR
OWB_REPO
PE21 - Dev
PE5 - Prod
PE7
Sources and Targets
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
ODI Architecture
Repositories ExecutionContext
DEV
PROD
DDR_SRC
DDR_STG
DDR
PE21 - Dev
PE16 - Test
PE9ODI_DEV
Sources and Targets
MASTER
DEV
PE9ODI_TEST
EXEC
PE9ODI_PROD
EXEC
SWH_SRC
SWH_STG
DDR
PE5 - Prod
TEST
HF_TEST
MASTER
MASTER
DDR_SRC
DDR_STG
DDR
DDR_SRC
DDR_STG
DDR
HF_PROD
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration UtilityODI_MASTER_USER=DEV_ODI_REPOODI_MASTER_URL=jdbc:oracle:thin:@porape9.ops.cfmu.eurocontrol.be:1542/odi_devODI_MASTER_DRIVER=oracle.jdbc.OracleDriverODI_USERNAME=SUPERVISORODI_WORK_REPOSITORY_NAME=WORKREP_DEVOWB_WORKSPACE_OWNER=OWB_REPOSOWB_URL=porape9.ops.cfmu.eurocontrol.be:1542:owb2odiOWB_WORKSPACE_NAME=OWB_REPOS
MIGRATION_MODE=DRY_RUN
MIGRATION_OBJECTS=PROJECT.SWH.MODULE.DDR;PROJECT.SWH.MODULE.EUACA.MAPPING.UWB_LDIM_IMATCH_METHOD_010_INS;
MIGRATION_STRATEGY=CREATE
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration UtilityODI_MASTER_USER=DEV_ODI_REPOODI_MASTER_URL=jdbc:oracle:thin:@porape9.ops.cfmu.eurocontrol.be:1542/odi_devODI_MASTER_DRIVER=oracle.jdbc.OracleDriverODI_USERNAME=SUPERVISORODI_WORK_REPOSITORY_NAME=WORKREP_DEVOWB_WORKSPACE_OWNER=OWB_REPOSOWB_URL=porape9.ops.cfmu.eurocontrol.be:1542:owb2odiOWB_WORKSPACE_NAME=OWB_REPOS
MIGRATION_MODE=DRY_RUN
MIGRATION_OBJECTS=PROJECT.SWH.MODULE.DDR;PROJECT.SWH.MODULE.EUACA.MAPPING.UWB_LDIM_IMATCH_METHOD_010_INS;
MIGRATION_STRATEGY=CREATE
oracle@prisme111: cd /opt/oracle/11.2.0.4/owb/bin/unixoracle@prisme111: ./migration.sh ODI_MASTER_REP_PASSWD ODI_SUPERVISOR_PASSWD OWB_WORKSPACE_PASSWD migration.config
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration Utility: Report
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration Utility: Result
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Mappings not migrated
•Errors in input connectors - Fixed with ODI 12.1.3•Procedure inside mappings - Wrapping into functions•No “Delete” loading strategy - Development of new IKM Knowledge Module
•No “Update” only mode with the Merge syntax - Edit of existing IKM
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
PSM - Scheduler
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
PSM - Scheduler
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
PSM - Scheduler
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
PSM - Monitoring
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
PSM - Monitoring
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
PSM - Parameter
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
PSM - Scheduler
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Cleanup Topology
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Migration is a good opportunity for...
•Re-assess the business rules•Define Data Stewardship•Refactoring•Delete unused batch•Complete missing documentation
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Execution OWB jobs from ODI
•Register in Topology•ODI Tool : OdiStartOwbJob•Monitor in Operator
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com
Further reading
•Blog Post by David Allan (@im_dave) and Julien Testut (@JulienTestut) on ODI12c migration : https://blogs.oracle.com/dataintegration/entry/planning_the_journey_from_oracle
•OTN article by Stewart Bryson (@stewartbryson) : Making the Move from Oracle Warehouse Builder to Oracle Data Integrator 12c : http://www.oracle.com/technetwork/articles/datawarehouse/bryson-owb-to-odi-2130001.html
• “OWB – Making the Move to Oracle Data Integrator” by Michael Rainey (@mRainey) at UKOUG 2012:http://www.rittmanmead.com/files/ukoug2012_rainey_odi.pdf
•Blog post series by Peter Scott (@dw_pete) on Rittman Mead blog:http://www.rittmanmead.com/2011/10/oracle-warehouse-builder-and-data-integrator/
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)
E : [email protected] : www.rittmanmead.com