data virtualization and etl

1

Data Virtualization and ETL

2© 2014 Rocket Software, Inc. All Rights Reserved.

Data


Real-time


Cost Efficient


Business Insight

6

What is Wrong with Status Quo?

“There is not enough time in

the day to move all the data.”

“My mobile users expect to see

current data, not yesterday’s

data.”

7

Reporting

Ad-hoc

OLAP

Data WarehouseStaging

Server

Staging

Server

Staging

Server

Data Movement Using ETL Tools

Represents ETL

System ofRecord

S

Q

L

OLTP

Files

Files

Files

Current Data Integration Limitations

ETL Server

Complex, high mainframe costsData inconsistency – High latency

8

ETL Drives Up Mainframe Costs

ETL costs found in three areas

Additional hardware, storage and

networking costs

Labor involved in managing file

transfers

Wasted systems cycles (MIPs)

Clabby Analytics “The ETL Problem, October 2013

IBM study found that to move one terabyte of data,

with three derivative copies each day, amortized over

a four year period added up to $8,269,335.

9

Eliminate Data Movement

Data virtualization enables

data structures that were

designed independently to be

leveraged together, from a

single source, in real time,

and without data movement

Mainframe

Web/ Mobile

RDBMS

Cloud Data

Big Data

Unstructured

Logical Data Source

© 2015 Rocket Software, Inc. All Rights Reserved.

10

What is Rocket Data Virtualization?

The industry’s only mainframe-resident data virtualization

solution for real-time, universal access to data, regardless of

location or format.

Makes data as simple to access as an Excel

spreadsheet, with reduced complexity and costs

Eliminates data movement and

dramatically reduces mainframe TCO

11

Rocket Data Virtualization Architecture

MQ

TSO/SPUFI

IMS

DB2

CICS

Rocket

DV Client

SMF Tape

Sys Logs

12© 2014 Rocket Software, Inc. All Rights Reserved - Company Confidential

How We Lower Mainframe TCO

GPP zIIP

Eligible Workloads Can Run

Outside of GPP within zIIP Mainframes have multiple processors

• General purpose processor

all processing counts against capacity

• Specialty Engines

Eligible workloads don’t count against capacity

Rocket DV has patented technology that allows it to run 99% of its own processing in the zIIP engine

• Enables mainframe data to be integrated in-place without processing “penalty


ETL and Data

Virtualization Scenarios


Typical ETL Process

Adabas

VSAMSequential

DB2 for z/OSCICS

IMS

Natural

IDMS

Data Warehouse

Data extracts from mainframe

and non-mainframe data sources

Transformation of data into

compatible formats

Analytics, Search

Staging

Warehouse

Oracle Informix

DerbyDB2 LUW

SQL Server

IBM Federation Server

Load into Data Warehouse

Complex process

Prone to errors

Costly - high MIPS

usage

Issues with data

inconsistency

Data not timely


SQLJDBC/ODBC/DRDA

NoSQLMongoDB API

Services

SOAP/REST/HTMLWebHTTP

EventsCDC/Data Streams

Rocket

Data Virtualization

ServerMap/Reduce Query

OptimizationParallel/IO

Data Mapping Caching

Design

Security

Metadata

Monitoring

Augmenting ETL with Data Virtualization

Adabas

VSAMSequential

DB2 for z/OSCICS

IMS

Natural

IDMS

Analytics, Search

Oracle Informix

DerbyDB2 LUW

SQL Server


IBM zIIP Specialty Engine

Join mainframe and non-

mainframe data

Combined data delivered to

analytics

All data

transformations run

on zIIP specialty

engine for

significantly

reduced MIPS

capacity usage

Information

delivered in right

format, in real-time


Augmenting Data Warehouse via DVS

SQLJDBC/ODBC/DRDA

NoSQLMongoDB API

Services



Rocket

Data Virtualization




Design

Security

Metadata

Monitoring

Adabas

VSAMSequential

DB2 for z/OSCICS

IMS

Natural

IDMS

Data Warehouse

Join VSAM with DW data

Combined data delivered to

analytics


Analytics, Search


Complex ETL Script

Vendor Systems

Source SystemPre-Landing

ETL

(Flow 1)

Landing

ETL

(Flow2)

Staging

ETL

(Flow 3)

Vendor

Extract

ETL

(Flow 4)

Vendor

Landing

ETL

(Flow 5)

Vendor

Updates

(Flow 6)

Pre-Landing Landing Staging

Enterprise Data Exchange Interface

Landing Cross-Ref

Cross-Ref

Extract

Program

Source

System

Staging

Hub Key

Generation

Services

Hub Key

Generation

Services

Services

Environment

Database

Environment

ETL

Environment


SQLJDBC/ODBC/DRDA

NoSQLMongoDB API

Services



Rocket

Data Virtualization




Design

Security

Metadata

Monitoring

SQL Insert Into Select Statement

Adabas

VSAMSequential

DB2 for z/OSCICS

IMS

Natural

IDMS

Analytics, Search

Oracle Informix

DerbyDB2 LUW

SQL Server



Replaces complex

and hard to manage

ETL scripts with

ELT (Extract Load

Transform)SQL Insert Into Select

Statement

Big Insights

Big SQL

MongoDB

Cloud

Data

IBM

Pure Data

Web/Mobile Analytics, Search ESB, ETL Transactional DataTransactional DataTransactional Data


A Unique ETL Solution

Providing a unique data virtualization solution that

keeps data on the mainframe

• save processing time, lower IT costs, and minimize

risk by eliminating replication and migration

Further reducing mainframe TCO

• ability to off load up to 99% of data integration

processing to System z specialty engine

Reduces complexity, simplifies management

• provides the data mapping tooling needed to access all mainframe

data sources with SQL or JSON


Where to Find Rocket Data Virtualization

RocketOn

Click rotating banner on

the RocketOn homepage

Click the Data Virtualization

tile under Rocket Solutions

http://www.rocketsoftware.com/solutions/data-virtualization

http://www.rocketsoftware.com/solutions/data-virtualization

data virtualization and etl

Software

yesterdays data

transfer of data

volume of data

data structures

terabyte of data

etling data

operational data

realtime data access