oracle data integration - ad valoremkonferenciak.advalorem.hu/uploads/files/db_24kedd_brooks.pdf ·...

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |

Oracle Data Integration Simplifying Complex IT Platforms with Data Integration

John Brooks Data integration CEE 24 March, 2015

Oracle Confidential


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2


Business Challenges

How do I move to Cloud without impacting my business ?

How do I perform upgrades with minimum downtime?

How can I move my data seamlessly and in real time across heterogeneous system?

How can I run real time reports without impacting production environment?


Oracle Data Integration Solutions can help transition to Cloud

Oracle GoldenGate helps ensure continuous availability of systems during upgrades and migrations

Oracle Data Integration Solutions can move data seamlessly between systems in cloud and on premise in Batch or Real-time

Oracle Goldengate can provides capture technology that is non intrusive and no impact on operational systems

How Oracle Data Integration Solution addresses the Challenge


New DB/HW/OS/APP

Fully Active DB

Reporting Database and/or DR database

Data Warehouse

OGG

OGG

OGG ADG

OGG

Zero Downtime Migrations & Upgrades

Active/Active High Availability

Query Off-Loading and Disaster Recovery

BI&DW Synchronization and Loading

Migrations & Consolidations

ODI

Analytical and Operational Data Integration Solutions

OEDQ

Oracle GoldenGate Real Time Replica Between Any Databases

Oracle Data Integrator ETL Platform

Active Data Guard Disaster Recovery Of Oracle DB EE

Enterprise Data Quality Data Quality Platform

Enterprise Metadata Manager Data Quality Platform

OEMM

ODI OEDQ

OEMM


EOL Oracle Streams customers in ECEMEA! Must upgrade from the product that is going into maintenance mode.

EOL Oracle Warehouse Builder customers in ECEMEA! Must upgrade from the product that is going into maintenance mode. 70% already out of compliance if don’t purchase ODI license!

Oracle GoldenGate Real Time Replica Between Any Databases

Oracle Data Integrator ETL Platform

Active Data Guard Disaster Recovery Of Oracle DB EE

Enterprise Data Quality Data Quality Platform

Analytical and Operational Data Integration Solutions


Oracle (Source) Oracle (Target)

Apply

ASM

Redo/Archive Logs

Capture

LGWR

1

2

6 7

4

5 8

Trail

Capture Delivery

Native TCP/IP 1 2 3

3

ASM

Redo/Archive Logs

LOG

MINER

LGWR

9

10 4

5

Source Server

(CPU %)

Through-

put

(ops/sec)

Streams Latency (secs) GGS Latency

(secs)

~80% 8,000 6 2.7

~45% 5,000 2.7 1.5

~20% 2,500 2 1.3

Features OWB basic ODI-EE

Oracle DB Targets Y Y

Non-Oracle Sources Oracle Gateways Y

Slowly Changing Dimensions (type 1) Y Y

OLAP Targets (Oracle + Essbase) Oracle OLAP only Y

Built-in Scheduler Oracle DBMS only Y

Non-Oracle DB Targets - Y

Changed Data Capture (including OGG integration) - Y

Web Services (API & Src/Trgt) & XML Targets - Y

Data Lineage / Impact Analysis - Y

Declarative Design Environment - Y

Slowly Changing Dimensions (2 & 3) - Y

Load Plans, Parallel Execution and Restartability - Y

JEE Deployment , Clustering and High Availability - Y

Java based SDK for full Programmatic Control - Y

Unified Management and Administration with OEM - Y

Pre-built Integration with OBI-EE, Hyperion, SOA/AIA, etc. - Y

OWB missing these functionalities. ODI has them!

Streams 8000 throughput in 6 seconds. OGG 2.7 seconds!

Streams 9 steps configuration. OGG 5 steps!

Roadmap For Streams & OWB


Data Integrator

New DB/ HW/OS/APP

Fully Active Distributed

DB

Reporting Database

Data Warehouse

ODS

Zero Downtime Upgrade & Migration

Query & Report Offloading

Data Synchronization within the Enterprise

Real-time BI, Operational Reporting, MDM

Event Driven Architecture, SOA

High Availability/ Disaster Recovery

Message Bus

Oracle GoldenGate Low-Impact Heterogeneous Real-Time Data Integration

Global Data

Centers

Low Impact Log Based Capture

JMS


Target DB

OGG

Source DB

First OGG Differentiator Accessing directly transaction

logs

Second OGG Differentiator Moving only committed transactions

Why is GoldenGate Different


Cap

ture

Trai

l

Ro

ute

Del

iver

Pu

mp

#StrataHadoop - Oracle Big Data Architecture

Announcing: Oracle GoldenGate for Big Data

New DB/ HW/OS/APP

Zero Downtime Upgrades & Data Migration

Fully Active Distributed DB

High Availability & Disaster Recovery

Application Offloading

Query & Report Offloading

Big Data, DW & Marts

Real-time BI, Hadoop Data Staging, Data Ingestion

Event Driven Architecture, SOA/JMS, Coherence

Message Bus & Data Grid

Data Synchronization Across the Enterprise

Global Data Centers

Real-time Analytics & Massive Parallelization

Data Streaming

GoldenGate

Real-time Data Delivery


OLTP & ODS Systems Data

Warehouse, Data Mart

Oracle PeopleSoft, Siebel, SAP

Custom Apps

Files Excel XML

Enterprise Performance

Custom Reporting Packaged Applications

Business Intelligence

Analytics

Data Federation

Data Warehousing

Custom

Data Marts Data Access

Data Silos

SQL Java

Batch Scripts

Data Hubs

Data Migration

Data Replication

OLAP

Oracle Data Integrator Replace Manual Coding


OLTP & ODS Systems Data

Warehouse, Data Mart

Oracle PeopleSoft, Siebel, SAP

Custom Apps

Files Excel XML

Enterprise Performance

Custom Reporting Packaged Applications

Business Intelligence

Analytics

OLAP

Oracle Data Integrator

Oracle Data Integrator Replace Manual Coding


Customer ID Customer Name Address 1 Address 2 City State Zip Country Birth Date Gender

AD23298 Mr Peter Mayhew 9407 Main St Fairfax VA 22031-4001 USA 02/23/61 M

VS38611 Dr Ellen Van Der Heijde 144 E Grove St Kingston PA 18704 US 07/12/57

DC18223 Jalila Abdul-Alim (Do Not Call) 4548 Pennsylvania Ave Apt 205 Kansas City MO 64111-3349 USA 02/23/63 F

CO9387A Tayside Computers Inc. 4912 E 41st N Idaho Falls ID 83401 USA 31/03/2007 N/A

TZ35019 Mr Zachary P Jahn 98-1731 Ipuala Loop Aiea Hawaii 96701 1710 United States 06/12/86 Male

CB27843 Mrs Edith Y Baba Junior Baba Real Est. Corp. 209 Stony Point Trl Webster NY USA 11/17/1971 M

OX80306 Andrew & Mary Baxter 14 Oxbridge Way Milfrod NH 03055-4614 US 05/28/67 F

JP70210 Mr RJ & Mrs FB MacDonald 57 Hadleigh Close Westlea Swindon SN5 9BZ MA - USA - Y

RD48107 Mr Andy Baxter 14 Oxbridge Wy Milford NH 3056 USA 01/01/01 M

Inconsistent formats Abbreviations

(often ambiguous)

Attributes non-standard, missing

or invalid

Widespread

duplication

(often hard

to spot)

Compound Names

Embedded Additional Information

Mixed Business & Personal Names

Multiple Names

Mis-Fielded Data

Erroneous Data

International Date Formats

Default or Dummy Data

Enterprise Data Quality Why Use EDQ for CRM


Product data are much more variable and unpredictable than other data types

10hp motor 115V Yoke mount

mtr, ac(115) 10 horsepower 115volts

MOT-10,115V, 48YZ,YOKE

This 10hp yoke mounted motor is rated

for 115V with a 5 year warranty

10 Caballos, Motor, 115 Voltios

TEAO HP = 10.0 1725RPM 115V 48YZ YOKE MTR

Motor, TEAO, 1725 RPM, 48YZ, 15 Voltios,

Montaje de Yugo, hp = 10

Item Motor

Classification 26101600

Power 10 horsepower

Voltage 115

Mounting Yoke

Enterprise Data Quality Why Use EDQ for Logistics


Variation or Error Example Variation or Error Example

Sequence errors • Mark Douglas or Douglas Mark Transcription

mistakes • Hannah, Hamah

Involuntary corrections

• Browne – Brown Missing or extra

tokens • George W Smith, George Smith, Smith

Concatenated names

• Mary Anne, Maryanne Foreign sourced

data • Khader AL Ghamdi, Khadir A. AlGamdey

Nicknames and aliases

• Chris – Christine, Christopher, Tina Unpredictable use of

initials • John Alan Smith, J A Smith

Noise • Full stops, dashes, slashes, titles,

apostrophes Transposed

characters • Johnson, Jhonson

Abbreviations

• Wlm/William, Mfg/Manufacturing Localization

• Stanislav Milosovich – Stan Milo

Truncations • Credit Suisse First Bost Inaccurate dates • 12/10/1915, 21/10/1951, 10121951, 00001951

Prefix/suffix errors • MacDonald/McDonald/Donald Transliteration

differences • Gang, Kang, Kwang

Spelling & typing

errors • P0rter, Beht Phonetic errors • Graeme – Graham

Enterprise Data Quality


Identify and quantify issues

in data

Enterprise Data Quality Profiling


Entry Address : “300 Berry #1210 SF California”

300

Berry St

Unit 1210

San Francisco

CA

94158-1670

PremiseNumber

ThoroughfareName

SubPremise

Locality

AdministrativeArea

PostCode

300

Berry

#1210

SF

California

Parse Validate

Title: Mr First: Robert Last: Fulmar Gender: Male DoB: 12/05/1978 Phone: 555-120-1329 Address: 9405 Main St Fairfax Virginia 22030

First: Bob Last: Fulmar Gender: Male Email: [email protected]

Title: Dr First: R Last: Fulmer DoB: 01/01/1978 Email: [email protected] Address: 9407 Main Street Fairfax VA 22031-4001

Title: Dr First: Robert Last: Fulmar Gender: Male DoB: 12/05/1978 Email: [email protected] Phone: 555-120-1329 Address: 9407 Main St Fairfax VA 22031-4001

Enterprise Data Quality Parsing / Merging


Customer

Place Order

Telephone Network

Bill Customer

Customer Care

Billing

Collect Usage

Self Care CRM Call Center

Customers

ODI

EDQ

Billing R

R

SR ODS

ODI

OGG

OGG OGG OGG R

OGG

EDQ For Customer/Citizen Data


Credit Cards

Loans, Mortgages

Funds, Assets

Business Unit A Marketing

System

Business Unit B Accounting

System

Group Risk Management

Quality Management

CRM

Regulatory reporting

Statutory reporting

Management Information

Systems

Customers

Accounts

ERP

ODI EDQ

Core Banking (Payment

Processing, Accounts)

ODI

Remote

Fraud Detection

OGG OGG OGG OGG

OGG

OGG

OGG

OGG

OGG

EDQ For Customer/Citizen Data


Oracle Data Integration 12c Delivery real-time data integration for Cloud and Big Data

Big Data

Cloud

Apps

Database

• Real-time data replication; optimized for Database 12c and Oracle Exadata

• End-to-end integrated with simplified deployment

• Unified tooling for both structured data sources and Hadoop / NoSQL

• Flexible deployment on-premise or in the Cloud for heterogeneous systems

• Expanded support for 3rd party systems and Oracle Applications in real-time data integration and continuous availability solutions


Oracle GoldenGate

Oracle Enterprise Data

Quality

Oracle Data Services

Integrator

Oracle Metadata

Management


OLTP OLAP Unstructured Legacy

Big Data Connectors

Oracle GoldenGate


Oracle Enterprise Data Quality

Oracle SOA Suite, BPM Suite, Oracle Business Analytics

Metadata Management

Oracle Cloud Application Foundation

Private Cloud

Public Cloud

Zero Downtime Migration Upgrades

Complete Data Services

Real-Time Application Data Synchronization

Across Globe

Oracle Data Integration Next Generation Data Integration for the Enterprise

http://www.peoplesoft.com/corp/en/public_index.jsp


Data Integration Solutions

John Brooks CMC, MBA

Data Integration Manager CEE

Oracle Confidential – Internal/Restricted/Highly Restricted

Enterprise Metadata Management


Oracle Enterprise Metadata Management

Oracle Confidential – Internal/Restricted/Highly Restricted 23

Available H1 FY15

Packaged for two offerings:

Oracle Enterprise Metadata Management (OEMM) Fully featured enterprise edition product

Oracle Metadata Management for Oracle Business Intelligence Applications (OMM) Limited for use with OBIA, no Business Glossary

Key Features: Report to Source Lineage

Impact Analysis

Model Versioning

Annotations and Tagging

Supports Metadata Standards

Business Glossary

3rd Party BI Metadata

3rd Party ETL Metadata

3rd Party DB Metadata

Big Data Ready


Value of Enterprise Metadata Management


ETL

BI Dashboards

App

ETL

ETL

How was sales figure calculated?

What will happen if I change this

table?

What reports use the mainframe

data? Sys Admin

Executive

BI Developer

Where did this data

come from?

Application User

Which reports use this

customer data?

CDC

Hadoop Data Lake

Data Steward

Can I trust the sources of this

customer data?

ETL

Developer

Solves a significant pain point for a wide variety of business consumers and technical staff

I want to design an experiment to measure the

success of a signup page. What data do I have?

Data Scientist

GG


Data Discovery Analytics

Oracle Metadata Management Logical Architecture

Oracle Confidential 27

Actionable Information

Data Lin

eage

Imp

act

An

alys

is

ETL Data Quality

Data Integration

Data Streams Integration Data Enrichment

File Based Cloud

Data Sources

Big Data Relational Applications

Business Glossary

Metadata Repository

Semantic Linking

Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 28

Oracle Metadata Management Physical Architecture

Semantic Mapping

Enterprise Conceptual Model Metadata Repository

Repository Manager

Data Stewardship Governance

Metadata Browser Search

Lineage Impact Analyzer

Any Web Browser

Data Source

ETL

BI - Reporting

Business Glossary


1. Title - the title of the data object

2. Creator - the person or entity responsible for creating the data object

3. Subject - subject terms or keywords that describe the data object

4. Description - a brief description, or abstract, of the data object

5. Publisher - the entity responsible for making the data object available

6. Contributor - a person or entity who contributed to the creation of the data object

7. Date - data of creation, publication, or revision of the data object

8. Type - the type of object. For data this would typically be "dataset"

9. Format - a description of the format or file type(s) of the data object

10. Identifier - a permanent identifier used to locate and identify the data object

11. Language - the language(s) used within the data object (if applicable)

12. Source - a relational element describing the lineage of the data object

13. Relation - a relational element describing the relationship of this data object to other objects, collections, or entities

14. Coverage - describes the spatial and temporal context of the data object

15. Rights - describes any rights, restrictions, or terms of use

Metadata


Metadata Harvesting from all Popular Platforms


Exploring Metadata

Text Search and Metadata Browser,

Model/Diagram Visualizers, Birds-eye View, Metadata Reporting

Data Flow and Lineage Analyzer

Sarbanes-Oxley Act (SOX) Compliance

Versioning and Model Compare

URI Based Call back from BI reports or Business applications

Stewardship Collaboration

Feedback Comments and Review Board

Metadata Tagging with Labels

Multimedia attachments (documents, videos, presentation, code)

URL link references for e-mail, blogging, social networking

Detailed Feature List Lifecycle Change Management

Metadata Harvesting from any DM, DW, ETL/DI, BI Tooling and Big Data/Hadoop

Enterprise Architecture Management (Metadata Configuration Management)

Impact of Change

Data Standardization & Data Governance

Supports Variety of Metadata Standards

Annotations and Tagging

Business Glossary with Semantic Lineage

Oracle Integrated

Oracle Database

Oracle Data Integrator, Oracle GoldenGate

Oracle Business Intelligence Enterprise Edition

Oracle Business Intelligence Applications (ODI and older Informatica based OBIA)


Installing Oracle Enterprise Metadata Manager

Installs as a fully self-contained

Oracle product


OEMM Home with Repository Panel and Bookmarks

Deploys as a web based application

with an attached metadata repository


Graphical Browser for Data Model Diagrams

Birds-eye-view metadata

viewer enables zoom-in / zoom-out

around complex

metadata models


Keyword Based Search on all Metadata

Full index text search

provides simple

navigational access to very

large metadata models


Versioning with Model Comparisons

Easily group and compare data models from similar databases or

between different

versions to determine

crucial differences


Social Collaboration for Stewardship Teams

Complete team-based environment

including support for

reviewing the lifecycle of terms and domains


End-to-End Data Flow Architecture Views

See how systems are connected

while visualizing

annotations and system boundaries


Reporting System Calculations

Drill into formulas and calculations

that are directly

harvested from ETL and BI Reporting engines


Find Data Provenance from the Report Layers

Visualize column and calculation

level dependencies

across systems, databases and

reporting views


Business Glossary is Built In (not a separate add-on)

Going beyond database

dictionaries, link business terminology

across metadata

models and systems


Business Glossary Policies and Reference Data

For code tables and value

ranges, associate glossary

metadata to valid data

Terms, Types and

Relationships


Resources


esource.us.oracle.comTechnology Price List

GoldenGate Pricing FAQ on PM Page

My.oracle.com/site/pd/fmw/products/DI/gg

[email protected] email alias

My.oracle.com/site/pd/fmw/products/DI/odi

[email protected]

Stbeehive.oracle.com/teamcollab/wiki/Oracle+Enterprise+Data+Quality

[email protected]

FY’15 Sales Play Respoistory

Oracle Data Integration OracleDataintegration OracleGoldenGate ORCLGoldenGate blogs.oracle.com/dataintegration

Follow us and connect with our community

Pricing Oracle Data Integrator

Oracle GoldenGate

Oracle Enterprise Data Quality

Sales Play Repository

mailto:[email protected]












http://my.oracle.com/site/pd/fmw/SalesPlays/index.html

http://my.oracle.com/site/pd/fmw/SalesPlays/index.html


Manual Mode

• Little Or No Automation

Siloed, But Automated

• Some Automation

Master Data Management

• More Automation

Integrated Metadata Architecture

• Full Automation

Metadata Maturity


Metadata Requirements Stack

oracle data integration - ad valoremkonferenciak.advalorem.hu/uploads/files/db_24kedd_brooks.pdf ·...

Documents