data warehousing portfolio update

© 2008 IBM Corporation

®

Data Warehousing Portfolio Update

June 11th 2008Infobahn - Information Management Event 2008Grand Hotel Sofia

Paul Gittins, Data Warehouse Sales, NEIOT: Germany & CEMAAS +44 7802 233328 [email protected]

IBM Software Group | Information Management software

2

Managing Information as a Strategic Asset

Information volumes are growing at the rate of 30% per annum

Managing this growth has its challenges

– Ever-increasing demand to analyse more information

– Ever-increasing demand to manage more types of information

– Growing number of users who need access to information

– Retaining data for compliance requirements

– Total cost of ownership

Key Question: Do you have a strategy for managing this growth ?

– IBM InfoSphere Warehouse is the answer


3

Flexible Architecture for Leveraging Existing Investments

Other Information & Application Sources

Customer & Product Profitability

Workforce Optimization

Dynamic Supply Chain

Multi-ChannelMarketing

Financial Risk Insight

Business Optimization

IBM InfoSphere WarehouseIBM InfoSphere MDM Server

IBM Information Server

IBM Cognos TM1IBM Cognos 8 Planning

IBM Cognos 8 BI

DB2, IMS, InformixIBM Content Manager,

IBM FileNet

IBM Offerings

Industry Models, Blueprints & Frameworks

Information On DemandEnd-to-End Capabilities

Optimization

Automation


4

Topics

Introduction– Example: NYPD (New York Police Department)

Portfolio overview– Data Warehouse Appliance: Infosphere Balanced Warehouse

– Infosphere Data Warehouse

– Information Server

– Cognos

– Data Warehouse Industry Data Models


June 2008 5

Marketplace

Data Warehouse (DW) and Business Intelligence (BI) continues to be a major growth area

– Priority spend for many clients

New clients seek to deliver reporting but facing constraints from their operational systems

– Performance/capacity

– Inflexible operational schema

Focus on improved “time to market”

– Emergence of appliance-based offerings

Desire to simplify existing DW infrastructure

– Data mart proliferation versus Enterprise Data Warehouse (EDW)

– Increasing willingness to review existing DW platform assets

Role of application-specific Data Warehouses

– Example: SAP Business Information Warehouse (BW)


6

Introduction to Business Intelligence and Data Warehousing

Business Users

Data Warehouse

Source Source Source Source Source

Extract, Transform, Load (ETL)

Query Tools

DataMart

DataMart

DataMart

Extract, Transform, Load (ETL)

Applications

Federation

Data Mining / Statistics

Dat

a -

Info

rmat

ion

Data Model

operational

informational

History

Detail

OLAPStar Schema

Normalised

FlexibleAnalysis

Optimised

Business Peformance Management


7

Example: NY Police (NYPD) Crime Information Warehouse (CIW)

Challenge • Lots of data providing historical access only• Difficult to access, disparate systems: Oracle & SQL Server• Absence of integrated view• Use of legacy tools

Solution• Integrated Data Warehouse• Improved delivery of data using

• Dashboard• Drill-down• Geo-spatial

Benefits• Move use of data from “reactive” to “pro-active”• Wider use of data • Improved performance of NYPD organisation

Technology: IBM DB2; Cognos; Crime Data Model; Information Server; IBM GBS Consultancy and Server

DB2 Warehouse

Linux Balance WarehouseInformation

Server

Mining and Advanced Analytics


8

CIW User Interface

http://ciwdemo.dfw.ibm.com/cognos8/


9

NYPD Crime Information Warehouse (CIW) in action

A source for this video : http://www.youtube.com/watch?v=qy7MCu-FhKk


10

Leveraging Information to Create Business ValueInsightful, relevant information: when and where it is needed

OLAP & Data Mining− Merchandising, Inventory,

Operations

Help Solve Crimes by Delivering Suspect List to Detectives Arriving

at the Crime Scene

Optimizing Police Force Deployments

Information On Demand− Optimize Each Transaction− Call Centers, Field Ops

Query & Reporting− Financials, Sales

Crime Rate Reports


11

The IBM Balanced Warehouse is the complete data warehousing solution comprised of pre-tested, scalable and fully-integrated system components of InfoSphere Warehouse, Server and Storage.

Balanced Warehouse

IBM Balanced Warehouses include everything required to serve as a foundation for your business intelligence solutions. Some of the included highlights are:

• Robust InfoSphere Warehouse software• The latest in IBM Server technology• High performance IBM storage• Comprehensive total solution support

Real time, ready to go data warehousing solutionsThe InfoSphere Balanced Warehouse


12

InfoSphere Balanced WarehouseA fast track to warehousing

IBM modules

Preconfigured, pretested allocation of software, storage and hardware to support a specified combination of function and scale

Simple

Optimized

Flexible

BusinessRequirements

WarehouseFunction

WarehouseOperations

BusinessFutures

WarehousePlatform


13

InfoSphere Balanced Warehouse Better than an appliance

Flexibility for growth Add modules to address increasing demands

Multiple on-ramps for different needs

Reliable, nonproprietary hardware for reusability

Optimized performance Preconfigured and certified for guaranteed performance

Based on best practices for reduced risk

Simplicity Predefined configurations for reduced complexity

One number to contact for complete solution support

Balanced Warehouse


14

IBM Balanced Warehouse – Solution Classes

1 3 10 30 100

C-ClassBusiness Partner / IBM

D-Class

Modular for flexible Scale out

E-Class

Modular for flexible Scale out

Terabytes of User data

Workload complexity

D5100 for high availabilityD5000 for price-performance

E7100 for large scale and complex workloads

C4000 – IBM/BP - up to 4TB C3000 – BP - up to 1.5TBC1000 - BP - up to 350GB

C1000 C3000 C4000

The IBM Balanced Warehouse provides 3 different solution classes that each target and serve a distinct warehousing market segment


15

Balanced Warehouse Offerings Note: Infosphere Data Warehouse software also available separately

C-Class D-Class E-Class

Enterprise SolutionsGrowth SolutionsApplication Solutions

Class Name

Highly Scalable Enterprise Data Warehouses

Advanced Departmental Data Marts and Growing Data Warehouses

Easy to Deploy Warehousing Applications and Reporting Tools

with a Fixed Raw Data Limit

~$15 to $250K USD $890K to over $1.8M USD (min config)

Scalability High – BCU and Module Very High – BCU and ModuleFixed Raw Data Limit

Size 1TB and up 1-2TB and up50GB to ~5TB

Primarily Partners (except C4000) IBM Direct Sales and partners

Users No Hard LimitsUp to 250 Users

$220K to $280K USD (min config)

Delivery


16

What is the Value of an IBM Balanced Warehouse ?A ‘Ready-to-Go’ Warehouse that is Simple, Flexible and Optimized

1 C-Class offerings do not scale through the addition of BCUs due to their specialized form factor, targeting smaller installations. 2 C-Class offerings are supported through associated reseller/distributor service agreements

Simple:

Reduced Complexity: Ships configured and ‘Ready to go’

Pre-tested: Validated by IBM

Full integration: High Performance DB2 Warehouse delivered ‘load ready’

Predictable, modular growth through the usage of BCU1

One number: Single point of contact for all support issues2

Flexible

Modular scalability: Grows with your demands, not your vendor’s

Open and Reusable: Ensures that you are not locked in to proprietary limitations

Spectrum of offerings: Multiple on-ramps to warehousing

Ability to plan for and price the growth of your company’s warehousing needs

Optimized:

IBM Balanced Warehouse solutions are thoroughly tested and tuned for performance

Solutions developed through IBM Best Practices of successful client implementations

Low Risk: Guaranteed performance


17

“No Copy Analytics” Bridging the Operational Insight Divide

The No Copy approach to delivering business analytics has many benefits

•Enabling real-time Analytics•Increasing Flexibility•Reducing the Development and Management Effort•Flexibility to add and change Analytical Applications cost effectively

This Approach has had Outstanding Business Results

• Improved Customer Service• Fraud Detection• Customer Churn Prediction• Event Based Decision Support

Overall Greater Return On Investmentand Lower Cost of Ownership !


June 2008 18

Infosphere Data Warehouse A complete, integrated platform

AIXHP/UX Linux/x86SolarisWindows/x86

MPP Data Server

Analytical AccelerationNo CopyAnalytics

Advanced Design & Management

Extreme Performance

C-Class D-Class E-Class Linux/Windows

AIX/SUN/HPPlatform

Universal Access

Advanced Capability

Portals & Web Apps Reporting Solutions MS Office / Share-point

On-line Analytical Processing (OLAP)

Data Mining

Design Studio

Data Compression

Workload Management

Data RetentionRemote Data Access

Web Services SQL/MDX MDX

Text Analytics

Embedded Data

Movement


June 2008 19

Components

Standard Eclipse-based GUI


June 2008 20

SQL Warehousing (SQW)

Design Project

DB source/target design (RDA)

SQL Operator

PropertyProduction server view

Connect to DataStage

Server

SQL Operator palette

Integrated data modeling and SQL tooling in DWE Design Studio complements ETL tool for complete warehouse building– Integrated RDA-based physical data modeling– SQL Warehousing Tool with Eclipse-based data flow and control flow design, web-based administration

• Deep DB2 exploitation (e.g. support MERGE syntax)• Websphere App Server-controlled runtime

– Special integration with IIS when deployed together

Manipulation within the RDBMS– Especially relevant for maintaining derived structures & data marts

Design Studio

Run Time

Control Flow Editor

Data Flow Editor

Extract SQL JoinSQL

Lookup

DS

subflow

FTP

SQL DF

DS Job DS Job

Email Verify

MetaData (EMF) CodeGen/Optimizer

DWE Admin Console

Websphere

Application Server

DataStage

Server

DB2

Control Flow Editor

Data Flow Editor

Extract SQL JoinSQL

Lookup

DS

subflowExtract SQL Join

SQL

Lookup

DS

subflow

FTP

SQL DF

DS Job DS Job

Email Verify

FTP

SQL DF

DS Job DS Job

Email Verify

MetaData (EMF) CodeGen/Optimizer

DWE Admin Console

Websphere

Application Server

DataStage

Server

DB2

DataStage

Server

DB2

Import DataStageJob as visual

Subflow

Export SQL to DataStage as CMD Operator

Import DataStageJob as opaque Runtime object

Infosphere Warehouse Integration with IBM Information Server


June 2008 21

OLAP

New OLAP acceleration

Multiple approaches

– MQTs & Query re-write

– Cube Views

Cubing Services

– Delivers mainstream MDX Function / Robust Data Cubes and support for MDX client tools.

– Integrates Alphablox cubing technology, DB2 Cube Views and DB2 optimization technology

– Provides native MDX interface

– Addresses 75% of OLAP applications • Exception is specialised forecasting and writeback

– 2Q08 support for Excel

– Benefits• Improve Time-to-Value, Ease of Use / Deployment, lower TCO

New


June 2008 22

Inline Analytics

Toolbox of pre-built components for analytic functionality (blox)

Scaleable, J2EE-based uses Websphere, Weblogic or Tomcat web-app servers

Rapid application development platform

Enables creation of customised analytical components, embedded into existing business processes and web applications

DWE now ships connectors for non-DB2 RDBMS sources

Complementary to Cognos capabilities

DataLayoutBlox

PageBlox

ToolbarBlox

ToolbarBlox

GridBlox ChartBlox

DataLayoutBlox

PageBlox

ToolbarBlox

ToolbarBlox

DataLayoutBloxDataLayoutBlox

PageBloxPageBlox

ToolbarBloxToolbarBlox

ToolbarBloxToolbarBlox

GridBlox ChartBloxGridBlox ChartBlox

Access

Analyze

Present

Access

Analyze

Present

Web Servers on Windowsor UNIX


RDBMS

MDB

CustomAnalytic

Applications

PresentationApplication ServicesData Sources

Multiple Data Sources:

RDBMS, OLAP

Internet Explorer,Netscape


Application Server

DB2 Alphablox



RDBMS

MDBMDB

CustomAnalytic

Applications

PresentationApplication ServicesData Sources

Multiple Data Sources:

RDBMS, OLAP



Application Server

DB2 AlphabloxDB2 Alphablox


June 2008 23

Data Mining & Visualisation

Supports Traditional and Embedded Mining– Scoring, Modelling and Visualisation components using DWE

workbench– Algorithms include Segmentation; Classification Tree,

Regression and Association

ExtractedData

Data Mining Workbench

Select Transform Mine Assimilate

Transformed Data

Extracted

Information

Assimilated

Information

Selected Data

Data Warehouse

Traditional Data MiningTraditional Data Mining Embedded Data MiningEmbedded Data Mining

Easy MiningIM Modeling IM Scoring

IM Visualizations

DB2

QueryOLAP

Simple Mining

Reporting Tool

Traditional

– Predictive, Statistical Base, API interface

– PMML standard interface available

– Partners

Quadstone

Embedded

– Discovery: patterns

– Easier

– SQL based calls

– Exploiters

Newer


June 2008 24

Unstructured Analytics

Rich analysis interface for combining structured and unstructured data

Combines search, text analytics and data visualization

Min

ing

engin

eM

inin

g

engin

e

Category ItemCategory Item

[Call Taker] James [Date] 2002/08/30[CustomerID] ADC00123

[type] Claim[Involved Parties]John Jones Jack Smith [Vehicle type] Truck[Incident Type] Parking

Extractedmetadata

[Call Taker] James [Date] 2002/08/30[CustomerID] ADC00123

[type] Claim[Involved Parties]John Jones Jack Smith [Vehicle type] Truck[Incident Type] Parking

Extractedmetadata

Search, visualization and interactive miningSearch, visualization and interactive mining

Call Taker: James Date: Aug. 30, 2002Duration: 10 min.CustomerID: ADC00123

D: Details of incident in motor vehicle claim including involved parties

Unstructured data

Structured Data

Original Data

Unstructured analytics framework Analysis tools

Linguisticanalysis

Jack Smith suspected Involvement in

Insurance = Investigate Claim

Warehouseapplication

Design Time Runtime Analysis / Reporting

DataMining

ReportingOLAP

OLAP

Intelligent Miner

EDW

Configureannotators(controlledVocabulary,regularexpressionstaxonomies)

DefineDataflowCreatemetadata

Deployment preparation

UIMA annotator,dictionaries

PackageAnalysis eng.

GenerateWarehouseapplication

UIMA Tools Design studio

Transformation engine

Warehouseapplication

UIMA

Configuredannotators

EDWSource DB(with text fields)


June 2008 25

Transparent access to non-DB2 dataInformation Integrator (II) Connectors component of IIS included in Infosphere Warehouse (Enterprise Edition)

Potential Uses

– Ad-hoc access

– Propagation

– Prototyping

– Productivity

Not a virtual data warehouse

Superior to brute-strength tool-based approach

Examples

– Taikang Life (China Insurer)

– UK Bank

Advanced SQLRecursive SQL

User Defined Func.Common Table Exp.

Advanced SQLRecursive SQL

User Defined Func.Common Table Exp. Read Write

Informix IDS

Cost based optimizationRetrieve data statisticsRewrite/ Optimize Query« Push-Down » processingLock ManagementBuffer ManagementRead/Write accessAST Support

Cost based optimizationRetrieve data statisticsRewrite/ Optimize Query

Push-Down » processingLock ManagementBuffer ManagementRead/Write accessMQT Support

Cost based optimizationRetrieve data statisticsRewrite/ Optimize Query« Push-Down » processingLock ManagementBuffer ManagementRead/Write accessAST Support

Cost based optimizationRetrieve data statisticsRewrite/ Optimize Query

Push-Down » processingLock ManagementBuffer ManagementRead/Write accessMQT Support

OracleInformixSybaseMicrosoftDB2 S/390ODBC

OracleInformixSybaseMicrosoftDB2 S/390ODBC


26

IBM Information Server Delivering information you can trust

Industry leading data-integration platform

Scaleable

Platform/vendor-neutral

– Synergy with Infosphere Warehouse

Can exploit new blade technologies


27

All capabilities against any combination of data sources (OLAP or relational); removes barriers to usage Hides complexity from users and ensures consistency, integrity and accuracy of information

Business Performance Management (BPM) Industry Solutions

Cognos 8 Starter Edition ships with

Infosphere Warehouse

Cognos 8 Business Intelligence Suite

USER

Purpose-Built WebServices Architecture

SERVICES

Zero Footprint Task-Based Interfaces

Open Data AccessDATA

Warehouses Warehouses (Relational & OLAP)(Relational & OLAP)

Transaction Transaction SystemsSystems

Flat, Legacy Flat, Legacy or Modernor Modern

…SecuritySecurity PresentationPresentation QueryQuery ScheduleSchedule& Burst& Burst

MetadataMetadata

ReportingReporting

AnalysisAnalysisScorecardingScorecarding

DashboardsDashboardsSearchSearch

Event Event ManagementManagement

Office Office ConnectionConnection

QueryQuery

…

REPORTING & DASHBOARDING ANALYSIS EVENT MANAGEMENTEVENT MANAGEMENT SCORECARDING

New Pre-Integrated Software Offerings

Data Warehousing


28

IBM Industry Data Models

Industry-proven models, including KPIs and compliance metrics

– Trusted, single analytical view of the business

Proven data model methodology

– Accelerated, business-centric development

Models automatically populate and generate metadata in IBM Information Server

– Reduces project complexity and risk

Models: Data Warehouse, Process

Industries

– Banking

– Financial Markets

– Insurance

– Telecommunications

– Retail

– Healthcare

– Crime

– Airlines

500+ clients

•Customer Interaction Analysis•Customer Investment Profile•Individual Customer Profile•Wallet Share Analysis

Relationship Relationship MarketingMarketing

•Outstandings Analysis •Portfolio Credit Exposure•Security Analysis•Liquidity Risk•Collections Analysis•Insurance Risk Profile

RiskRisk

•Transaction Analysis•Activity Based Costing Analysis•Insurance Product Analysis•Investment Arrangement Analysis

ProfitabilityProfitability

•Capital Allocation Analysis•Capital Procurement•Credit Loss Provision•Funds Maturity Analysis•Income Analysis

Asset & Liability Asset & Liability Management Management

•Interest Rate Risk Analysis•Credit Risk Profile•Credit Risk Assessment•Credit Risk Mitigation Assessment•Asset Securitization Analysis •Operational Risk Assessment

•European Central Bank Reporting•Financial Capital Adequacy Analysis•Structure Of Regulatory Capital •Foreign Financial Account Analysis•Suspicious Activity Analysis•Transaction Activity Analysis•SOA Balance Sheet Analysis•SOA Cash Flow Analysis•SOA Statement Of Change In Shareholders' Equity Analysis•SOA Statement Of Income Analysis

ComplianceCompliance

Balance Sheet Portfolio Basis Approach AnalysisBalance Sheet Classified Approach AnalysisBalance Sheet Order Of Liquidity Approach AnalysisBalance Sheet Net Assets Approach AnalysisCash Flow Direct AnalysisCash Flow Indirect Analysis

•Customer Complaints •Delinquency Analysis•Customer Loyalty•Market Analysis

•Campaign Analysis•Cross Sell Analysis•Customer Attrition •Customer Behavior•Lead Analysis

•Product Profitability•Product Analysis•Organization Unit Profitability•Performance Measurement•Business Procedure Performance

•Profitability Analysis•Channel Profitability•Customer Lifetime Value •Customer Profitability•Location Profitability

•Authority Profiling•Credit Risk Analysis•Debt Restructuring •Involved Party Exposure•Location Exposure•Non Performing Loan•Operational Risk Loss Analysis

•Interest Rate Sensitivity •Liquidity Analysis•Short Term Funding Management•Financial Management Accounting

•Net Interest Margin Variance•Structured Finance Analysis•Equity Position Exposure•Position Valuation Analysis

Cash Flow Direct Financial Institution AnalysisCash Flow Indirect Financial Institution AnalysisIncome Statement By Function AnalysisIncome Statement By Nature AnalysisIncome Statement Financial Institution Approach AnalysisStatement Of Changes In Equity Analysis

Example: Banking DW Model, Business Solution Templates (BSTs)


29

Summary

IBM offers a best-of-breed portfolio for you to deliver a cost-effective reporting platform– We have all the components necessary– Valid even for clients who have a heritage with other RDBMS eg. Oracle– Strong delivery capabilities (not covered in this presentation)

Consider the benefits of using an appliance-based solution– Focus on solution delivery not individual components– Minimal additional effort to manage an IBM data warehouse

Consultancy effort typically the major element in data warehouse delivery– Assess suitability of existing data schema and relevance of off-the-shelf data model– Understand business reporting requirements

Questions .. ?

data warehousing portfolio update

Documents