oracle data integration solutions - doag.org · pdf filethe development, release, ... oracle...
TRANSCRIPT
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Data Integration Solutions
Joachim Jaensch - Principal Sales Consultant
Short Overview
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Agenda
4
1
Oracle OpenWorld 2014
Oracle Data Integration Solutions (DIS)
Oracle Data Integrator (ODI)
Oracle GoldenGate (OGG)
Oracle Enterprise Data Quality (OEDQ)
Oracle Enterprise Metadata Management (OEMM)
Oracle Data Service Integrator (ODSI)
Oracle Active Data Guard (OADG)
2
3
4
5
6
7
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Data Integration Solutions (DIS) Introduction
Oracle Confidential – Internal/Restricted/Highly Restricted 5
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Data Integration Solutions 12c
Deliver real-time data integration for Cloud and Big Data
Big Data
Cloud
Apps
Database
• Real-time data replication; optimized for Database 12c and Oracle Exadata
• End-to-end integrated with simplified deployment
• Unified tooling for both structured data sources and Hadoop / NoSQL
• Flexible deployment on-premise or in the Cloud for heterogeneous systems
• Expanded support for 3rd party systems and Oracle Applications in real-time data integration and continuous availability solutions
Oracle Data Integrator
Oracle GoldenGate
Oracle Enterprise Data
Quality
Oracle Data Services
Integrator
Oracle Metadata
Management
Oracle Active
Data Guard
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Data Integrator
E-LT: Bulk Data Processing and Fast Data Transformation
Big Data
Cloud
Apps
Database
• Certified for leading technologies to deliver fast time to value
• High-performance, low cost of ownership E-LT architecture
• Lightweight deployment
• Flexible, easy to enrich functionality
Oracle Data Integrator
High Performance E-LT
Declarative Design
Data Services
Extensible Knowledge Modules
Structured & Unstructured Data
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 8
Industry Leading Performance
Extremely Fast Execution and Reduced Cost
E-LT provides a flexible architecture for
optimized performance on any platform
Benefits
• Leverages set-based transformations
• Improves performance for loading,
no network hop
• Takes advantage of existing infrastructure:
hardware and software
Conventional ETL Architecture
Extract Load
Transform
Next Generation Architecture
“E-LT”
Load Extract
Transform Transform
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Declarative Design
Improved Developer Productivity For Faster Time to Solution
9
• Easy and universal for designing simple to complex mappings
• Robust and Reusable: Create Mappings once for any physical design
• Quick to define and refactor maps, high degree of automation
• Extensible model for any data integration mechanism
• Benefits
• Significantly reduces the learning curve
• Shortens implementation times
• Simpler maintenance
ODI Declarative Design
Define How: Knowledge Modules
2 1
Automatically
Generate
Dataflow
Define
What You Want
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Reverse Engineer Metadata
Journalize (CDC)
Load from Source to Staging
Check Constraints
Integrate, Transform
Data Service
Knowledge Modules
Simpler Physical Design and Shorter Implementation Time
10
Oracle Sqoop Hive HBase Oracle Merge
SAP ERP
SAP BW Oracle
Datapump Oracle DBLink
JMS External Tables
Teradata
Oracle Spatial
Siebel eBusines
Suite IBM DB2 Netezza SCD
Key Architecture Benefits: • Faster development and simpler
maintenance using templates
• Easy to extend and add new best
practices
• Enforces predictability and reduces cost
of ownership
Pluggable Knowledge Modules Architecture
Sample out-of-the-box Knowledge Modules:
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Big Data: Hive, HBase, Sqoop
Best for Oracle: Merge, Spatial, Multi-Table Insert, Optimizer Hints and more
Real-Time: CDC with GoldenGate
Applications: E-Business Suite, Siebel, PeopleSoft, JD Edwards Enterprise One, JD Edwards World, SAP ERP and SAP BW
Heterogeneous: Optimizations for all major RDBMS: IBM DB2, Microsoft SQL Server, Teradata, Netezza, etc.
Applications
Legacy
Databases
Big Data
Cloud
Extended Connectivity
Leverages Existing IT, Faster Implementation
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
• Key Architecture Benefits: 100% Java, Open APIs, fast E-LT
D A
B
File C
C$_0
C$_1
LKM
LKM
IKM
I$ E$ (Errors)
CKM IKM RKM
JKM
Check-Load Transform Extract-Load
ODI Agent
Application & SOA Business Intelligence & Data Warehouse
ODI Agent may be deployed in any part of the architecture
How ODI Works: E-LT Architecture
12
High Performance, Flexible, Lightweight Architecture
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Integration Between GoldenGate and ODI 12c
• Support real-time and bulk from the same
interface
• Deploy GoldenGate processes via ODI Bulk Load Scenario Real Time Scenario
• Enable GoldenGate journalizing on
interfaces without restructuring.
• Manage GoldenGate Capture/Delivery
instances in ODI Topology
• Set common GoldenGate parameters
graphically
Capture Delivery
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle GoldenGate
Oracle GoldenGate
Log-based Change Data Capture and Delivery
Active-Active Replication
Reliability and Transaction Integrity
Heterogeneous Source and Targets
Big Data/Java/FlatFile Integration
• High-performance, low-
impact real-time data
integration and
replication
• Timely data for
improved business
insight
• Continuous availability
for 24/7 operation Big Data
Cloud
Apps
Database
Heterogeneous Data Replication
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 15
More Information later!
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 16
Oracle Enterprise Data Quality
Profile, Explore, Audit
Parse, Standardize, Cleanse
Governance
Match, Merge, Enrich
Case Management
• Improves data accuracy,
usability and ‘fitness for
purpose
• Unified interface for
ease-of-use, lower TCO
• Depth of capability in
multiple data domains
reduces project risk
Big Data
Cloud
Apps
Database
Oracle Enterprise Data Quality
Data Quality for Customer and Product Data
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted
Data Quality Issues – A simple Customer Table
Name Address City State Zip Phone Email
Bob Williams 36 Jones Avenue Newton MA 02106 617 555 000 [email protected]
Robert Williams 36 Jones Av. MA 02106 617555000
Burkes, Mike and Ilda 38 Jones av. Nweton MA 02106 617-532-9550 [email protected]
Jason Bourne, Bourne & Cie. 76 East 51st Newton MA 617-536-5480 6175541329
… … … … … … …
Mis-fielded data
Matching Records
Typos
Mixed business and contact names
Multiple Names
Non Standard formats
Missing Data
17
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
• Solution
– Profile business data to find the bad data and assess quality over time
– Cleanse, match and merge data before it gets loaded into the Data Mart or Data Warehouse
– Repair data during batch processing flows
• Benefits
– Stop the garbage-in-garbage-out cycle!
– Improve the trust of data within business marts and data warehouses
– Enable IT to deliver value to the lines of business during every data integration flow
– Improve business efficiency with match and de-duplicate redundant records
Profile, Cleanse and Govern Business Data
Oracle Confidential – Internal/Restricted/Highly Restricted 18
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 19
Profiling – Understand the data first
Interactive exploration of data, identifying distribution and outlying values with drill-
downs
Identify and quantify issues in
data
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted
Oracle Enterprise Data Quality – Standardization
• Standardize, Transform and Parse • Split names and name elements • Identify individuals and businesses • Derive additional attributes
Name: Dr Ellen Van Der Heijde
Title: Dr
First: Ellen
Last: Van Der Heijde
Gender: Female
Name: Jalila Abdul-Alim (Do Not Call)
First: Jalila
Last: Abdul-Alim
Gender: Female
Note: Do Not Call
Title: Mr
First: R
Middle: J
Last: MacDonald
Gender: Male
Name: Mr RJ & Mrs FB MacDonald
Title: Mrs
First: F
Middle: B
Last: MacDonald
Gender: Female
20
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Enterprise Data Quality - Matching
•Match & Merge data from disparate sources •Create ‘best’ record based on survivorship rules
Title: Mr First: Robert Last: Fulmar Gender: Male DoB: 12/05/1978 Phone: 555-120-1329 Address: 9405 Main St Fairfax Virginia 22030
First: Bob Last: Fulmar Gender: Male Email: [email protected]
Title: Dr First: R Last: Fulmer DoB: 01/01/1978 Email: [email protected] Address: 9407 Main Street Fairfax VA 22031-4001
Title: Dr First: Robert Last: Fulmar Gender: Male DoB: 12/05/1978 Email: [email protected] Phone: 555-120-1329 Address: 9407 Main St Fairfax VA 22031-4001
Oracle Confidential – Internal/Restricted/Highly Restricted 21
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 22
Oracle Enterprise Metadata Management
Report to Source Lineage
Impact Analysis, Model Versioning
Supports Metadata Standards
Annotations and Tagging
Big Data Enabled Big Data
Cloud
Apps
Database
Oracle Enterprise Metadata Management
Trust your data
• Exploring Metadata
• Sarbanes-Oxley Act (SOX) Compliance
• Stewardship Collaboration
• Lifecycle Change Management
• Data Standardization & Data Governance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Metadata Management
“Metadata Management involves
managing data about other data, whereby this ‘other data’ is generally referred to as content.
- wikipedia
Oracle Confidential – Internal/Restricted/Highly Restricted 23
”
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Value of Enterprise Metadata Management
Oracle Confidential – Internal/Restricted/Highly Restricted 24
ETL
BI Dashboards
App
ETL
ETL
How was sales figure calculated?
What will happen if I change this
table?
What reports use the mainframe
data? Sys Admin
Executive
BI Developer
Where did this data
come from?
Application User
Which reports use this
customer data?
CDC
Hadoop Data Lake
Data Steward
Can I trust the sources of this
customer data?
ETL
Developer
Solves a significant pain point for a wide variety of business consumers and technical staff
I want to design an experiment to measure the
success of a signup page. What data do I have?
Data Scientist
GG
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Data Discovery Analytics
Oracle Metadata Management Logical Architecture
Oracle Confidential
25
Actionable Information
Data Lin
eage
Imp
act
An
alys
is
ETL Data Quality
Data Integration
Data Streams Integration Data Enrichment
File Based Cloud
Data Sources
Big Data Relational Applications
Business
Glossary
Metadata
Repository
Semantic
Linking
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Exploring Metadata
Text Search and Metadata Browser,
Model/Diagram Visualizers, Birds-eye View, Metadata Reporting
Data Flow and Lineage Analyzer
Sarbanes-Oxley Act (SOX) Compliance
Versioning and Model Compare
URI Based Call back from BI reports or Business applications
Stewardship Collaboration
Feedback Comments and Review Board
Metadata Tagging with Labels
Multimedia attachments (documents, videos, presentation, code)
URL link references for e-mail, blogging, social networking
Detailed Feature List
Lifecycle Change Management
Metadata Harvesting from any DM, DW, ETL/DI, BI Tooling and Big Data/Hadoop
Enterprise Architecture Management (Metadata Configuration Management)
Impact of Change
Data Standardization & Data Governance
Supports Variety of Metadata Standards
Annotations and Tagging
Business Glossary with Semantic Lineage
Oracle Integrated
Oracle Database
Oracle Data Integrator, Oracle GoldenGate
Oracle Business Intelligence Enterprise Edition
Oracle Business Intelligence Applications (ODI and older Informatica based OBIA)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 27
Oracle Data Services Integrator
Federated Data Services
Data Virtualization
Bi-directional Reads & Writes
Integrates with Oracle SOA
Integrates with Oracle Business Analytics
• Easily develop data
services and ad-hoc
queries without
infrastructure changes
• Open and standards
based—SOA, Web
Services, Java, Service
Data Object
Big Data
Cloud
Apps
Database
Oracle Data Services Integrator
Data Federation and Virtualization
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 28
Oracle Service Oriented Architecture (SOA) – Product Mapping
Oracle® Reference Architecture, SOA Infrastructure, Release 3.2, 4479-03, February 2012
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 29
• Consolidation: Move data from distributed sources to central hub; consuming applications query hub database
• Data Virtualization / Federation: Query and update distributed sources in-place and provide combined result
• Hybrid Approach: Federation and Consolidation are combined based on best approach for individual use cases
Different Data Access Techniques
Source A
Consolidated Hub
Source B
ODI/OGG
Consumer
Consolidation only
Source A Source B
ODSI
Consumer
Data Virtualization only
Source A
Consolidated Hub
Source B
ODSI
ODI/OGG
Consumer
Source C
Hybrid Consolidation & Data Virtualization
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 30
When to Use Which Integration Technology
Data Virtualization Service Bus / Business Process
• Combine & normalize data access services
(Create, Read, Update, Delete)
• Orchestrate services to business processes
• Service Routing / Mediation
Data Virtualization Consolidation
• Data cannot be replicated (security, ownership)
• Consolidation is not efficient for access pattern
• Real-time access is required and replication is not efficient
• Transactional updates
• Need to change model frequently
• Performance is priority
• Data needs to be offloaded from original source
• Replication is used for redundancy
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 31
Data Virtualization Use Cases
• Data Services Layer – Provide a clean data abstraction for SOA and application developers
– Shield consumers from changes in data topology
• Augmenting Business Intelligence – Extend access to sources outside the data warehouse
• Rapid Prototyping of Data Integration – Quick creation of an interface to integrated data without building a consolidated store
• Cloud Data Integration – Combine onsite and offsite data resources in a unified interface
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 32
• Native SOA Data Integration Native XML type system allows processing and passing of hierarchical & SOA data.
Based on open standards (XML, WS, JDBC/SQL, XQuery)
Tight integration in Oracle SOA: Fast-path with OSB, Governance with OER
• Normalization and Reuse through Data Services Multi-layer approach to normalizing and transforming source data
• Highly Scalable, Industrial-Strength Performance Linear scalability through WLS Clusters
Performance through caching, streaming and query optimization
• Fine-Grained Security and Auditing Row and column-based data security
• Comprehensive Update Capabilities Automated CRUD operations on data services
Distributed XA transactions through WLS Transaction Manager
ODSI Core Strengths
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 33
Oracle Active Data Guard
Snapshot Database
Open for Read-Only Operations
Continuous Recovery
Using for GoldenGate Extract
Automatic Block Repair Big Data
Cloud
Apps
Database
Oracle Active Data Guard
High Availability of the Database
• Main Component of
Maximum Availability
Architecture (MAA)
• Role Transition between
Source and Standby
Database in Case of
Failure
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Note: A single DR copy may be multi-purposed for different combinations of the use cases described
SYNC
or
ASYNC
Data Guard &
Active Data Guard
Use Cases: Data Guard and Active Data Guard Real-Time Data Protection and Availability for Oracle Database
New DB
Version Standby First Patching,
Database Rolling Maintenance
Exact copy
of primary Query & Report Offload
Open Read-Only
Snapshot
Standby
Convert to Test Database (open read-write)
Single Command Refresh
Exact copy
of primary Offload RMAN Backups
Exact copy
of primary Disaster Recovery
Manual or Automatic Failover
Redo Far Sync,
GoldenGate Downstream
Exact copy
of primary Source for thin snaps/clones
Exact copy
of primary Extract offload, source for
GoldenGate ALO mode
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Production
Database
Continuous redo
shipping, validation & apply
Real-time Reporting Fast Incremental
Backups
Physical Standby
Database
Read-write
Workload
Data Guard
Standby Database: Failover Target
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Production
Database
Continuous redo
shipping, validation & apply
Real-time Reporting Fast Incremental
Backups
Active Standby Database
(physical standby open read-only)
Read-write
Workload
Active Data Guard
Standby Database: Offload Production + Failover Target
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Best Data Protection, Auto-Repair, and High ROI
Active Data Guard
Zero data loss at any distance
Primary Data Center DR Data Center
Automatic Block Repair
Enterprise Manager Cloud Control
Offload Fast Incremental
Backups
Offload read-only workload to open standby database
37
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Data Guard and Active Data Guard Feature Sets
Data Guard (included with EE) Data Protection with HA
• Zero or near-zero data loss protection
• Transparent – all datatypes, workloads
• Continuous data validation
• Detect silent corruption
• Automatic database failover
• Dual-purpose DR as test system
• Simple migrations and upgrades
• Oracle Enterprise Manager integration
Active Data Guard (Option) Advanced Protection with High ROI
• Zero data loss at any distance
• Automatic corruption repair
• Auto-replay of inflight transactions
• Offload transport compression*
• Offload read-only workload
• Offload read-mostly workload
• Offload fast incremental backups
• Automation for rolling upgrades
* Also requires Advanced Compression Option
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 39
Active Data Guard: Disaster Recovery and Data Protection
High-performance, simple, drop-in physical replication for HA and DR
Zero data loss, unique data corruption protection, automatic failover, offload read-only workloads
Disaster recovery for all data types, applications and workloads
GoldenGate: Flexible HA, Maintenance and Migrations, Real-Time Data Integration
Logical replication for advanced replication requirements, target open read-write
Enable minimal/zero downtime upgrades and application migration using 2nd database copy
Heterogeneous platforms, bi-directional replication, transformations, subsetting, multiple topologies
Oracle Replication Solutions for Business Continuity
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2014 40
Active Data Guard
Oracle Products for Data Movement
• Optimized Set-based transformation • Highest latency for real-time feeds • Heterogeneous • Supports highly complex transformation
• Not invasive on source • Fastest read and write • Oracle Database only • No transformation; no partial sets • Unidirectional
• Not Invasive on source • Very fast read and write • Heterogeneous or homogeneous • Basic transformation and partial sets • Uni- and Bi-Directional
GoldenGate
Data
Integrator
Comparing Tradeoffs and Benefits Driven by Different Architectures
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Resources
41
Oracle Data Integration Oracle Data Integration OracleGoldenGate ORCL DataIntegration blogs.oracle.com/dataintegration
Oracle Data Integrator
Oracle GoldenGate
Oracle Enterprise Data Quality
Oracle Enterprise Metadata Management
Oracle Data Services Integrator
http://www.oracle.com/us/products/middleware/data-integration/overview/index.html
Data Integration
Oracle OpenWorld 2014
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Questions and Answers
42 Oracle OpenWorld 2014
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 43