v9.1.2 update
DESCRIPTION
v9.1.2 update - Beate porstTRANSCRIPT
© 2013 IBM Corporation
IBM InfoSphere Information Server v9.1.2 releaseupdate and roadmap
Beate Porst ([email protected])InfoSphere Product Management
© 2014 IBM Corporation
IBM Information Server: Simplified Packaging for InformationIntegration and Quality
BusinessInformationExchange
Understanding &Collaboration
• Information blueprints• Relationship discovery
across data sources• IT-to-business mapping
DataQuality
Cleansing &Monitoring
• Analysis & validation• Data cleansing• Data quality rules &
management
DataIntegration
Transformation• Massive scalability• Power for any complexity• Total traceability
Delivery• Data capture at any time• Delivery anywhere• Big data readiness
InfoSphere Information Server EnterpriseEdition:Integrating and transforming data and content to deliveraccurate, consistent, timely and complete information ona single platform unified by a common metadata layer
© 2014 IBM Corporation
Mapping Platform Components to Information ServerPackages
Platform Component \ PackageBusiness
InformationExchange
Data Quality DataIntegration
EnterpriseEdition
Blueprinting and Best Practices (BPD)
Governance Dashboard
Data Discovery
Metadata Management and Lineage (MWB)
Logical and Physical Data Modeling (IDA)
Business Glossary (BG)
Data Cleansing and Enrichment (QS)
Data Quality Validation & Monitoring (IA)
Data Quality Exception Management (DQC)
SOA Deployment (ISD)
Data Specification Mapping (FT)
Extraction, transformation, load (DS)
Self-Service Data Integration (Data Click)
Change Data Delivery (CDD)
© 2014 IBM Corporation
Information ServerWhat’s new in v9.1.2
© 2014 IBM Corporation
TransformationProductivity Connectivity OperationsOverview Performance Admin
2010 2011 2012 2013 2014
Information Server Recent Activity
8.5 FP1
8.7 FP1
9.1
FP2 FP3
FP2
FP1 9.1.2
Data Integration Acceleration- Advanced transformation
features (looping/v.pivot)- zOS File Stage- Integrated Balanced
Optimizer capabilities
Robust Enterprise Support- New Suite Installer- Active/Passive
High Availability support- Source Code Control
Integration
Simple Data Quality- Standardization Quality
Assessment- Match Specification
Report- Match Designer Updates
Stronger Governance- Operations Console- Business Glossary Workflow- Blueprint Task Management- Metadata Asset Manager
Product Integration- Leverage Data Validation
Rules in DataStage Jobs- Advanced Data Replication
integration- Next Generation Netezza
Connectivity & Optimization- HDFS Integration
Advanced Admin & Productivity- Parallel Debugger- New Backup/restore tooling- Maintenance Mode- Stronger Encryption
Agile integration- InfoSphere Data Click- Enhanced Workload Mgmt- ODM Integration- Hadoop Balanced Optimization- HDFS Extensions- InfoSphere Streams Integration
Business Driven Governance- Policy and rules support for
information governance- Web-based blueprints- Integrated metadata mgmt
enhancements
Sustainable Quality- Data Quality Console- Standardization Rules Designer- Data Rules Advancements
- IDA 8.5 support
Anywhere Integration- Big Data Features:
* JSON support*BDFS REST API*JDBC connector
- DB2 on z/OS load optimization- Data Click new data
sources/targets
Sustainable Quality- New QS standardization rulesets
(Thailand , Ireland , update forIndia)
- DQ Exception Mgt for DS/QS- Operational DQ Rules
Business Driven Governance- Bulk metadata import- Governance Dashboard- IDA 8.5 support4
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
5
Information Governance Dashboard
What’s New in Information Server v9.1.2
Usage• Raises data confidence with
visual governance status
Value• Immediate insight into
governance policy status• Interception of issues when they
start, right at the source• Effectively measure results &
compliance of policyenforcement
1000sof data pointsand policiesvisualized
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
6
Support for Information Data Architect 8.5
• Builds on the new metabroker introduced at 9.1 for Information Data Architect which:• introduced better performance at lower resource cost• removed Windows only dependency
• Certification of IDA v 8.5 added• Tolerance for orphaned and invalid objects (ability to ignore those that don’t impact rest of
model)• Improved error/warning logging
What’s New in Information Server v9.1.2
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
7
Metadata Workbench Enhancements
JDBC Connector support• Display details for JDBC Connector stage, including URL Definition, Schema, Table and SQL
statements• Includes inclusion of JDBC (writes/reads) into data lineage flowsXML/JSON Support• Browse, query and detail display for XML/JSON• Displays column level information within asset page• Can be linked via manual binding for lineage
What’s New in Information Server v9.1.2
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
8
Business Glossary Enhancements
Integration with Data Rules• Data rule asset types (including unpublished rules) from InfoSphere Information Analyzer are
now displayed in the Browse All Assets page, can be searched, assigned to terms, governancerules, business labels and data stewards
• Drill down from a GovernanceRule to a Data Rule to theDatabase column to which itsapplied
What’s New in Information Server v9.1.2
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
9
Business Glossary Enhancements
Workflow Roles• Development Log now captures every history event including creation and reviewer comments• Security roles have been changed to provide a higher degree of granularity for existing roles:
Author, Published and Reader• Two new workflow roles:
• Reviewer: can review changes and make comments• Approver: can approve changes to a new or existing term (but no edit abilities themselves)
• Can now add comments at every stage of the workflow process.
Export Development Glossary• Can now export either development or published glossary
What’s New in Information Server v9.1.2
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
10
Data Quality Exception Management for DataStage and QualityStage
What’s New in Information Server v9.1.2
Usage• Collect exception data from any Data
Integration or Data Quality process• Support clerical review• Data Steward Dashboard
Value• Promote consistency in the way data
stewards and business analysts caninvestigate data issues
• Insert good data quality controls andgovernance practices into eachproject
• Support a variety of processingmechanisms at the point of greatestefficiency
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
11
InfoSphere QualityStage Standardization Rules
New Standardization Rules• Country specific rule sets for India, Ireland and Thailand• Provide for data standardization of names (individual and organizational), addresses, phone and
locality (varies per country)• Delivered as archive files in the QSRules folder of the install directory
• Client = ./InformationServer/Clients/Classic/QSRules• Server = ./InformationServer/Server/PXEngine/QSRules
What’s New in Information Server v9.1.2
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
12
JSON Document Support
• Derive metadata format automaticallyfrom sample JSON documents…
• Supports hierarchical formats withsimple fields, objects and arrays
• Schema views• New Parsing and Composing steps for
provide for complex hierarchical data inJSON syntax; with value and structurevalidation options
• Multiple options for reading/writing data:- files directly from disk- as part of a long string- passed in/out as a LOB
What’s New in Information Server v9.1.2
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
13
JDBC Connector
• JDBC Connector provides InformationServer products with access to JDBC datasources
• Supports data read and write operationsand metadata import operations
• Certified in this release with ApacheDerby and IBM Big Insights Big SQLdrivers
• Managed metadata import providedthrough new capabilities in InfoSphereMetadata Asset Manager (IMAM)
• Filtering by asset type and name patterns
What’s New in Information Server v9.1.2
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
14
DB2 Z Bulk Load Optimization
Huge Performance Gains For Load• Moved from a single load stream to parallel streaming via Z pipes.
• Multiple LOAD utilities targeting separate partitions which performsfaster than a single LOAD utility targeting all partitions.
• 9.1.2 is 80 to 160% faster than 9.1.0 (depending on number of partitions.)
• Performance scales almost linearly as you increase the number of partitions, regardless of load method.
• Internal testing loading almost 1TB per hour using 16-way load
Huge Performance Gains For Read• connector determines the number of partitions in the table and dynamically configures the number of
DataStage nodes to match the number of partitions
• Parallel read using the 9.1.2 DB2 connector is 40% faster than the 9.1.2 DB2Z stage, regardless of thenumber of partitions.
Resilience• When Retry on connection failure is set to Yes the connector will try to establish an FTP connection again
when the initial attempt to connect fails.
What’s New in Information Server v9.1.2
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
15 IBM CONFIDENTIAL
What’s New in Information Server v9.1.2
Overview
• Business users need quick and easy access to information to supporttheir analytical projects.
• Organizations need to avoid data sprawl, so governance best practicesmust be ensured
• Originally only supported DB2 or Oracle to Netezza
New in this release
• Universal Connectivity via ODBC to now support DB2, Netezza, Oracle, Teradata, Sybase, SQL Server,Greenplum, and others…. as source or target
• Automatic filtering of columns with data types not supported by the target data store
• Leverages connector framework enhancement for data sampling via “row limits”
http://www.youtube.com/watch?v=hUGGudh2iWI&feature=youtu.be
InfoSphere Data Click - Self Service Data Integration
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
16
Connector Enhancements
Limit number of returned rows• New property to support database sampling(required for feature of Data Click)• Applies to the following Connectors: ODBC, DB2, Netezza, Oracle, Teradata and JDBC
ODBC Connector expanded binary support• The ODBC Connector now supports automatically generated 'CREATE TABLE' statements for
types Binary, VarBinary or LongVarBinary
What’s New in Information Server v9.1.2
Name Label Description Default value
LimitRows Limit number ofreturned rows.
Select Yes to limit the number of rowsthat are returned by the connector.
False (No)
Limit Limit Enter the maximum number of rowsthat will be returned by the connector.
1000
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
17
InfoSphere Metadata Asset Manager - Performance Optimization
What’s New in Information Server v9.1.2
• Performance benefits of BI Simplification in 9.1• 46% reduction in execution time of Express Import
• Performance benefits of physical model import (Erwin)• 44-60% reduction in execution time of Express Import (Erwin)
• IMAM Express Import in 9.1 is 7 - 1200% faster than in 8.7 for the following workloads• IDA:
• Small workload (55K assets): +1200% (Throughput: 450 objects/s)• Large workload (119K – 430K): 9.1 succeeded (Throughput: 245 – 367 objects) whereas 8.7 failed
due to Out of Memory• Erwin (124K assets): +50% (Throughput: 351 objects/s)• BO import (175K assets): +18% (Throughput: 318 objects/s)• DB2 PDR (108K assets): +7% (Throughput: 149 objects/s)• Cognos (141K assets): -20% (Throughput:120 objects/s)
• MITI in 9.1 extracts more metadata (+45% reports &+27x more models, etc) than MITI in 8.7
Note: Performance results may vary in other environments
© 2014 IBM Corporation
Data IntegrationData Governance InfrastructureOverview Data Quality
18
Connectivity Accelerator
What’s New in Information Server v9.1.2
• Growing number of pre-build Connectivity Sample• Cassandra• Hive• Hbase• MongoDB• Avro• Jaql• JMS• WTX• and more…
https://www.ibm.com/developerworks/community/files/app?lang=en#/folder/4645e12a-7bdb-40ed-a103-f1160b707758?sort=collected
© 2014 IBM Corporation
Thank you