oracle data integration management (oemm) overview · pdf file–fix quality at the source...
TRANSCRIPT
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Data Integration Enterprise Metadata Management (OEMM) Overview Dr. Holger Dresing Oracle Data Integration Solutions (DIS) EMEA
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Agenda
4
Datenintegration und Data Governance
OEMM Grundlagen
Schnittstellen/Connectoren
Demo/Powerpoint Screenshots
Q&A
1
2
3
4
5
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Comprehensive Data Integration & Governance Capabilities
5
Real-Time Data Movement – Low impact capture, stage in
Hadoop – Continuous data availability
Data Transformation – Bulk data movement – Pushdown data processing
Data Federation – Virtualized Data Services
Data Quality & Verification – Fix quality at the source – Verify data consistency
Metadata Management – Lineage and Impact Analysis – Business Glossary Semantics
Data Governance Foundation
Oracle Data Integrator (Transformation)
Enterprise Data Quality (Profile, Cleanse, Match and De-duplicate)
Fast Load
Oracle GoldenGate (Movement)
Enterprise Metadata Management & Business Glossary (Business Glossary, Data Lineage, Impact Analysis and Data Provenance)
Data Service Integrator (Federation)
GoldenGate Veridata (Online Data Verification)
ELT Processing on Hadoop or
SQL
Continuous Availability
Big Data Preparation Service
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Data Governance Foundation
Differentiated Technical Approach
Oracle OpenWorld 2014 6
Dynamic Data Movement – Real-time CDC is by default, not ETL – Least invasive on sources – Proven best performance – Integrated Oracle capture/apply
No ETL Engines – Take the processing to the data;
don’t move the data to the process – Leverage your data engines for the
workloads (Hadoop or SQL)
Most Heterogeneous – Leverage open source Hadoop, not
proprietary distributions – Hadoop is the Hub, not ETL tools – Open metadata standards
Oracle Data Integrator * (Transformation)
Enterprise Data Quality (Profile, Cleanse, Match and De-duplicate)
Fast Load
Oracle GoldenGate (Movement)
Enterprise Metadata Management & Business Glossary (Business Glossary, Data Lineage, Impact Analysis and Data Provenance)
Data Service Integrator (Federation)
GoldenGate Veridata (Online Data Verification)
ELT Processing on Hadoop or SQL
Continuous Availability
Big Data Preparation Service
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Enterprise Metadata Management Trust your data – BCBS 239
Executives Dashboards / Reports
!=
!!! ???
•Dashboard does not match report? •Where did this data come from? •Can I trust my other dashboards and reports?
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Enterprise Metadata Management Data Lineage Impact Analysis
DBA’s / ETL / BI Developers Data Mgmnt. / Development Env.
•What impact on does my column change have? •What impact does my ETL mapping have? •I want to create new assets from my metadata!!
?? !! ??
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Value of Enterprise Metadata Management
Oracle Confidential, under Non-Disclosure 9
ETL
BI Dashboards
App
ETL
ETL
How was sales figure calculated?
What will happen if I change this
table?
What reports use the mainframe
data? Sys Admin
Executive
BI Developer
Where did this data
come from?
Application User
Which reports use this
customer data?
CDC
Data Reservoir
Data Steward
Can I trust the sources of this
customer data?
ETL
Developer
Solves a significant pain point for a wide variety of business consumers and technical staff
I want to design an experiment to measure the
success of a signup page. What data do I have?
Data Scientist
GG
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
10
Vertical Lineage Links a business friendly set of terms to the
IT metadata and operational assets Capture Business Glossary, Taxonomy,
Ontology, Conceptual Models
Horizontal Column Level Links the data fields from Business Intelligence
Dashboards or Reports back to the Source Columns Schemas, BI View Layers, ETL Transformations,
Calculations, etc.
Ve
rtic
al L
ine
age
Horizontal Lineage
“NE_SALES”
“SALES”
“NAME” “ACCT_NAME”
“NORTH”
“AGG_TOTAL”
BI Fields to Source Columns
“FNAME|LNAME”
“Customer”
Biz Terms to IT
Two Crucial Styles of Metadata Management
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Open World 2015 11
Introducing: Oracle Enterprise Metadata Management
1. Business Glossary – simple tools to catalog, link and collaborate on business terms
2. Metadata Management – horizontal and semantic data lineage for all big data sources
Business/Metadata Catalog Report to Source Lineage
Impact Analysis
Audit, Versioning & Diff Reports
Social/Collaboration Features
Annotations and Tagging
Automatic Stitching
Comprehensive Harvesting 3rd Party BI Metadata
3rd Party ETL Metadata
3rd Party DB Metadata
3rd Party Modeling Tools
Hadoop & NoSQL Metadata
Metadata Standards
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Oracle Metadata Management Logical Architecture
14
Actionable Information
Data Lin
eage
Imp
act
An
alys
is
ETL Data Quality
Data Integration
Replication Messaging Data Federation
File Based Cloud
Data Sources
Big Data Relational Applications
Metadata Management
Business Glossary
Metadata Repository
Semantic Linking
Data Discovery Analytics
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Data Discovery Analytics
Oracle Metadata Management Logical Architecture
15
Actionable Information
Data Lin
eage
Imp
act
An
alys
is
ETL Data Quality
Data Integration
Replication Messaging Data Federation
File Based Cloud
Data Sources
Big Data Relational Applications
Business Glossary
Metadata Repository
Semantic Linking
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Data Discovery Analytics
Oracle Metadata Management Logical Architecture
16
Actionable Information
Data Lin
eage
Imp
act
An
alys
is
ETL Data Quality
Data Integration
Replication Messaging
Data Federation
File Based Cloud
Data Sources
Big Data Relational Applications
Business Glossary
Metadata Repository
Semantic Linking
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Hadoop HBase Hadoop Hive/Flume HP Enscribe HP NonStop HP Neoview Hypersonic SQL IBM DB2 i Series IBM DB2 UDB IBM DB2 z Series IBM Informix IBM Netezza JMS / MQ Microsoft Access Microsoft SQLServer MySQL Pivotal Greenplum PostgreSQL Salesforce.com SAP BW / BI SAP ERP / ECC SAS SQL/MP SQL/MX Sybase ASE Sybase IQ Teradata
Adaptive Altova Apache Hcatalog Apache Hive/HQL Borland CA ERwin Cloudera Impala COBOL Copybook DataStax Embarcadero EMC ProActivity GentleWare Google BigQuery Grandite Hadapt Hive Hortonworks Hive IBM Cognos IBM DB2 IBM DataStage IBM Discovery IBM Federation Server IBM Lotus Notes IBM Netezza IBM Rational Rose IBM Rational Architect Informatica Metadata Mgr. Informatica PowerCenter
CoSORT ISO SQL Standard (DDL) MapR Hadoop Hive MicroFocus Microsoft Access Microsoft Office Excel Microsoft Visio Microsoft SQL Server Microsoft SSIS Microsoft Visual Studio Microstrategy Magic Draw OMG CWM Standard OMG UML Standard Oracle BI Answers Oracle BI Enterprise Edition Oracle BI Server Oracle DAC Oracle Data Integrator Oracle Data Modeler Oracle Database Oracle Designer Oracle Hyperion Applications Oracle Hyperion Essbase Oracle Warehouse Builder Pivotal Greenplum PostgreSQL
QlikView SAP BO Crystal Reports SAP BO Designer SAP BO Desktop Intelligence SAP BO Repository SAP BO Data Integrator SAP BO Data Steward SAP Master Data Management SAP Sybase PowerDesigner SAP Sybase ASE Database SAS Data Integration Studio SAS BI Server SAS Information Map SAS Metadata Management SAS OLAP Server Select Sparx Architect Syncsort Tableau Talend Teradata Tigris Visible W3C DTD & XSD Schema
Operational Integration (Movement / Transformation) Metadata Harvesting (Glossary, Lineage & Impact Analysis) Oracle Database Oracle Exadata Oracle Big Data Appliance Oracle TimesTen Oracle OLAP Oracle Business Intelligence Oracle BI Applications Oracle E-Business Suite Oracle JD Edwards Enterprise One Oracle JD Edwards World Oracle Fusion Applications Oracle Governance Risk and Compliance Oracle Fusion AIA Oracle Retail Applications Oracle Agile BI / DW Oracle Agile PLM for Process Oracle iFlex FlexCUBE Oracle iFlex Mantas Oracle Hyperion Applications Oracle PeopleSoft Oracle Siebel CRM / OnDemand Oracle Communications Oracle WebLogic Server Oracle Coherence Data Grid Oracle SOA Suite Oracle Enterprise Service Bus
+ open APIs and standards based meta-model
Comprehensive, Open & Heterogeneous Data Integration
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Exploring Metadata
Text Search and Metadata Browser,
Model/Diagram Visualizers, Birds-eye View, Metadata Reporting
Data Flow and Lineage Analyzer
Sarbanes-Oxley Act (SOX) Compliance
Versioning and Model Compare
URI Based Call back from BI reports or Business applications
Stewardship Collaboration
Feedback Comments and Review Board
Metadata Tagging with Labels
Multimedia attachments (documents, videos, presentation, code)
URL link references for e-mail, blogging, social networking
Detailed Feature List Lifecycle Change Management
Metadata Harvesting from any DM, DW, ETL/DI, BI Tooling and Big Data/Hadoop
Enterprise Architecture Management (Metadata Configuration Management)
Impact of Change
Data Standardization & Data Governance
Supports Variety of Metadata Standards
Annotations and Tagging
Business Glossary with Semantic Lineage
Oracle Integrated
Oracle Database
Oracle Data Integrator, Oracle GoldenGate
Oracle Business Intelligence Enterprise Edition
Oracle Business Intelligence Applications (ODI and older Informatica based OBIA)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Metadata Harvesting from all Popular Platforms
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Data Governance Lifecycle Tools
20
Operational Data Flows Business Sources
Quality KPIs Case Management
Governance Cockpit for Data Stewards & Stakeholders
Exception Review
Metadata Management
Business Glossary
Design Time
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 21
Oracle Metadata Management Physical Architecture
Semantic Mapping
Enterprise Conceptual Model Metadata Repository
Repository Manager
Data Stewardship Governance
Metadata Browser Search
Lineage Impact Analyzer
Any Web Browser
Data Source
ETL
BI - Reporting
Business Glossary
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 23
DEMO
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Friendly Views for Business & IT User Collaboration
Oracle Big Data Governance 24
• Havesting Metadaten • Oracle Datenbank • Oracle Data Integrator • Informatica Powercenter • Oracle Business Intelligence • Excel-Datei
• Data Lineage • Impact Analysis • Manual Data Mapping • Business Glossar • Metadata Explorer
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Friendly Views for Business & IT User Collaboration
Oracle Big Data Governance 25
Comprehensive Data Lineage
Business Friendly Navigation
Business & IT Collaboration
Easy to Use, Search Driven
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
End-to-End Data Flow Architecture Views
See how systems are connected
while visualizing
annotations and system boundaries
Oracle Confidential, under Non-Disclosure 26
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Keyword Based Search on all Metadata
Full index text search
provides simple
navigational access to very
large metadata models
Oracle Confidential, under Non-Disclosure 27
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Graphical Browser for Data Model Diagrams
Birds-eye-view metadata
viewer enables zoom-in / zoom-out
around complex
metadata models
Oracle Confidential, under Non-Disclosure 28
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Versioning with Model Comparisons
Easily group and compare data models from similar databases to
determine crucial
differences
Oracle Confidential, under Non-Disclosure 29
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Social Collaboration for Stewardship Teams
Team-based approach is designed in
from the start
Oracle Confidential, under Non-Disclosure 30
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Find Data Provenance from the Report Layers
Visualize column and calculation
level dependencies
across systems, databases and
reporting views
Oracle Confidential, under Non-Disclosure 31
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Business Glossary is Built In (not a separate add-on)
Going beyond database
dictionaries, link business terminology
across metadata
models and systems
Oracle Confidential, under Non-Disclosure 32
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Perform Semantic Analysis
For a Business Term, find all the
physical implementations where the term is
used
Oracle Confidential, under Non-Disclosure 33
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Business Glossary workflow
Workflow for the entire
lifecycle from drafting to the final approval
of business terms
Oracle Confidential, under Non-Disclosure 34
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Connect with Oracle Data Integration
@OracleDI
Blogs.oracle.com/DataIntegration/
Oracle Data Integration
Oracle Data Integration
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Questions & Answers
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 37