introduction to heterogeneous data replication spring common 1999 richard sinn ibm santa teresa lab

Introduction to Introduction to Heterogeneous Heterogeneous Data ReplicationData Replication

Spring COMMON 1999Spring COMMON 1999

Richard SinnRichard Sinn

IBM Santa Teresa LabIBM Santa Teresa Lab

AgendaAgenda

What is and Why Replication ?What is and Why Replication ? Introduction to IBM Data Propagator Introduction to IBM Data Propagator

Relational (DPropR, DPR)Relational (DPropR, DPR)– DB2 to DB2 family including DB2/400DB2 to DB2 family including DB2/400

Introduction to DataJoinerIntroduction to DataJoiner– DB2 to non-DB2DB2 to non-DB2– non-DB2 to DB2non-DB2 to DB2– non-DB2 to non-DB2non-DB2 to non-DB2

Putting Them TogetherPutting Them Together

What is Replication ?What is Replication ?

Moving Data from Moving Data from one place to one place to anotheranother

Replication vs Replication vs CopyingCopying

From Source to From Source to TargetTarget

Maintain “changes” Maintain “changes” automaticallyautomatically

Why Replication ?Why Replication ?

Data is everywhereData is everywhere Use Information Use Information

Intelligently and Intelligently and improve decision improve decision makingmaking

Stay competitive with Stay competitive with both Data access and both Data access and Data movementData movement

Speedup application Speedup application deploymentdeployment

Reengineer business Reengineer business processesprocesses

Increase online Increase online throughputthroughput

Improve system Improve system availabilityavailability

Support audit Support audit requirementsrequirements

Support Data Support Data WarehousingWarehousing

IBM Data Propagator IBM Data Propagator SolutionSolution

Multiple platforms Multiple platforms (AS/400, NT, AIX, etc)(AS/400, NT, AIX, etc)

Supported by IBMSupported by IBM Simple AdministrationSimple Administration Proven TechnologyProven Technology Speed Implementation Speed Implementation

with IBM services with IBM services organizationorganization

Built on standard SQLBuilt on standard SQL

DPropR ComponentsDPropR Components

Capture

Apply

Admin ToolsSource Server

Source ServerTarget ServerControl Server(Flexible location)

Windows NT, Windows 95,OS/2 PlatformWeb Browser

Replication Admin ToolsReplication Admin Tools

Define replication sourcesDefine replication sources– Also called “Registration”Also called “Registration”– This is what Capture works with logicallyThis is what Capture works with logically

Defines subscriptionsDefines subscriptions– This is what Apply works with logicallyThis is what Apply works with logically

Create replication control tablesCreate replication control tables– AS/400 DPR install program will create all AS/400 DPR install program will create all

the necessary replication control tablesthe necessary replication control tables

Admin Tool: DJRAAdmin Tool: DJRA

Define RegistrationDefine Registration

““Register” a source table as a Register” a source table as a replication sourcereplication source

Process data using vertical Process data using vertical fragmentationfragmentation– Replicate only selected columnsReplicate only selected columns

Select before and after image of a Select before and after image of a columncolumn

RegistrationRegistration

Capture

Registered Source Tables

At the Source Server

Registration Using DRJARegistration Using DRJA

Define SubscriptionDefine Subscription

““Subscribe” to registered tableSubscribe” to registered table Defines source and target Defines source and target

relationshiprelationship– For example, many target tables can For example, many target tables can

subscribe to one source tablesubscribe to one source table Horizontal fragmentationHorizontal fragmentation

– Columns can be further sub-selectedColumns can be further sub-selected

SubscriptionSubscription

Apply

Target Tables

Subscription Definitions


Subscription Using DJRASubscription Using DJRA

Replication FlowReplication Flow Source data changes are Source data changes are

“captured” by “captured” by CaptureCapture component according to component according to “registration”“registration”

““Change Data” are then put Change Data” are then put into Staging Tablesinto Staging Tables

According to “subscription”, According to “subscription”, Apply Apply component “apply” component “apply” changes to target(s) using changes to target(s) using the data in Staging Tablesthe data in Staging Tables

The Capture ComponentThe Capture Component

Journals,Logs Capture


Staging Tables

Unit of WorkTable

At the Source Server

RegisterTable

Journal/Log based CaptureJournal/Log based Capture

Capture reads the DB2/400 JournalCapture reads the DB2/400 Journal– Non-AS/400 DB2 will be reading LogNon-AS/400 DB2 will be reading Log

Inserts rows into the change data tableInserts rows into the change data table– One of the staging tablesOne of the staging tables

Time stamps or relative byte address is Time stamps or relative byte address is used as a log sequence numberused as a log sequence number

Prunes the change data tablePrunes the change data table Communicates with Apply via Communicates with Apply via

ASN/IBMSNAP_PRUNCNTL tableASN/IBMSNAP_PRUNCNTL table

The Apply ComponentThe Apply Component

Apply

RegisterTable

Staging Tables

Unit of WorkTable

Target Tables

Subscription Definitions

Capture and ApplyCapture and Apply

Unit of Work Table

Log, Journal Capture

Registered Source Table

Staging Table

Apply

Target Table

ASN.IBMSNAP_REGISTER

ASN.IBMSNAP_SUBS_SET

ASN.IBMSNAP_PRUNCNTL

source server

control server

target server

DPropR FunctionsDPropR Functions

Subscription SetSubscription Set Time based Time based

ReplicationReplication Event based Event based

ReplicationReplication Multiple Target Multiple Target

TypesTypes Blocking FactorBlocking Factor

Update AnywhereUpdate Anywhere Programming Programming

CapabilityCapability– SQL Statement / SQL Statement /

Call ProcedureCall Procedure– ASNDONEASNDONE– ASNLOADASNLOAD

Support Support Transaction Transaction ConsistencyConsistency

Different Target TypesDifferent Target Types User copy tablesUser copy tables

– Represent source data at a Represent source data at a particular point in time particular point in time

Point-in-time tablesPoint-in-time tables– Represent source data at a Represent source data at a

particular point in time particular point in time plus some overhead plus some overhead columnscolumns

Replica tablesReplica tables– Updates to a Replica table Updates to a Replica table

are automatically applied are automatically applied to the original source table to the original source table of the Replica, Auto of the Replica, Auto conflicts detectionconflicts detection

Consistent Change DataConsistent Change Data Consistent Change Consistent Change

Data (CCD)Data (CCD) Maintain complete Maintain complete

histories of data histories of data changes (Audit)changes (Audit)

Condense “hot spot” Condense “hot spot” updates before updates before transmitting datatransmitting data

““Loading” dockLoading” dock Used by NotesPump Used by NotesPump

or other applicationsor other applications

AgendaAgenda

What is and Why Replication ?What is and Why Replication ? Introduction to IBM Data Introduction to IBM Data

Propagator Relational (DPropR, Propagator Relational (DPropR, DPR)DPR)– DB2 to DB2 family including DB2/400DB2 to DB2 family including DB2/400

What if I have non-DB2 What if I have non-DB2 data?data?

What Happens to my:What Happens to my: Oracle DatabaseOracle Database Microsoft SQL Microsoft SQL

ServerServer Sybase SQL ServerSybase SQL Server Informix DatabaseInformix Database And all the And all the

applications that applications that built with them ?built with them ?

Introduction to DataJoinerIntroduction to DataJoiner

Transparent Access to Heterogeneous Transparent Access to Heterogeneous Data SourcesData Sources

Transparent DDL SupportTransparent DDL Support Heterogeneous Database JoinsHeterogeneous Database Joins Global OptimizationGlobal Optimization Two-Phase Commit SupportTwo-Phase Commit Support Integrated Data ReplicationIntegrated Data Replication Spatial Data Management/AccessSpatial Data Management/Access

DataJoiner OverviewDataJoiner Overview

SetupSetup NT Server (on IPCS)NT Server (on IPCS) Install DJInstall DJ Install other DB’s Install other DB’s

protocolsprotocols– SQL*Net for OracleSQL*Net for Oracle– Informix-net for Informix-net for

InformixInformix– ODBC drivers or Sybase ODBC drivers or Sybase

open client for MS SQL open client for MS SQL Server or Sybase SQL Server or Sybase SQL ServerServer

Basic Data AccessBasic Data Access On NT with DJ:On NT with DJ:

– Create databaseCreate database– Connect to DBConnect to DB– Create Server Create Server

Mapping (Create Mapping (Create “Nickname”)“Nickname”)

Setup client to go to Setup client to go to NT with DJNT with DJ

All access will “map” All access will “map” to “real” target DBto “real” target DB

DataJoiner Overview DataJoiner Overview

NT Server

IPCS

DataJoinerDB

Protocol

Client System

Non-DB2(e.g. Oracle)

Non-DB2(e.g. Oracle)

Total Access with Total Access with DataJoinerDataJoiner

DataJoinerAIX or NT

OS/2

WindowsDOS

HP-UX

MacintoshSolaris

AIXWeb

Browser

DB2 for MVSOS/390 DB2 for

VM/VSE

DB2 forOS/400

DB2 for NT

DB2 for HP-UXDB2 for Solaris

VSAMIMSOracle

Oracle Rdb

SybaseSQL ServerSybase SQL Anywhere

MicrosoftSQL Server

Informix

DB2/Non-DB2 ReplicationDB2/Non-DB2 Replication

AS/400 with DPR/400(Capture/400 Running;

Could be any DB2 family)

AS/400 with DPR/400(Capture/400 Running;

Could be any DB2 family)

Source Server

DataJoiner on NT IPCSApply Running


Sybase, Informix, OracleMicrosoft SQLServer,

DB2 Family


DB2 Family

Target ServerAdmin (DJRA)

Non-DB2/Non-DB2 Non-DB2/Non-DB2 ReplicationReplication




DB2 Family


DB2 Family

Target Server

Capture Triggers•Using CCD•Sybase, Microsoft•Oracle, Informix•SQL Anywhere

Source Server

Admin (DJRA)

Trigger Based Capture ITrigger Based Capture I

Trigger: Wake up and do actions based Trigger: Wake up and do actions based on event (such as insert, delete, on event (such as insert, delete, update)update)

Insert, Update, and Delete triggers Insert, Update, and Delete triggers created on the source tablecreated on the source table

Committed changes inserted into the Committed changes inserted into the CCD tableCCD table

Capture both before and after image of Capture both before and after image of the columnthe column

Trigger Based Capture IITrigger Based Capture II

Prunes CCD tables via trigger on Prunes CCD tables via trigger on IBMSNAP_PRUNCNTL tableIBMSNAP_PRUNCNTL table– Clean up CCDClean up CCD

Update Synchpoint via Trigger on Update Synchpoint via Trigger on IBMSNAP_REG_SYNCH TableIBMSNAP_REG_SYNCH Table– Communication with CaptureCommunication with Capture

Trigger Based Capture IIITrigger Based Capture III

Log sequence numbers generated Log sequence numbers generated dependent on OEM’sdependent on OEM’s

Pre-existing TriggersPre-existing Triggers– Run-time errorRun-time error– Check by DJRA and inform usersCheck by DJRA and inform users– Depends on OEM DBDepends on OEM DB

Heterogeneous ReplicationHeterogeneous Replication

Up

dat

e Tr

igg

er

Up

dat

e Tr

igg

er

Source table

InsertUpdateDelete

TRIGGERS

ibmsnap_pruncntl

ASN.IBMSNAP_REGISTER

ASN.IBMSNAP_PRUNCNTL

ibmsnap_reg_synch

CCD table

ASN.IBMSNAP_REG_SYNCH

Data source

DataJoiner Apply

Nickname for

ccd table

Nickname for

source table

Heterogeneous Replication Heterogeneous Replication DJRA tolerates the DJRA tolerates the

pre-existing target pre-existing target tabletable

DataJoiner does the DataJoiner does the data type mappingdata type mapping

Read only copies Read only copies (one direction with (one direction with User Copy, PIT)User Copy, PIT)

No Support for No Support for Replica yetReplica yet

Target table created Target table created in data sourcein data source

Nickname for target Nickname for target table created in DJ table created in DJ DatabaseDatabase


ReplicationReplication DPR on AS/400 and DPR on AS/400 and

other platformsother platforms Integrated support with Integrated support with

DPNR, Data Refresher, DPNR, Data Refresher, NotesPumpNotesPump

Expert Info and Support Expert Info and Support from IBMfrom IBM

Proven TechnologyProven Technology

Heterogeneous Heterogeneous

Data AccessData Access DataJoiner on DataJoiner on

IPCSIPCS Server Server

consolidation consolidation advantages with advantages with IPCSIPCS


Data from Any to Data from Any to AnyAny

Replicate “Changes”Replicate “Changes” Time & Event based Time & Event based

replicationreplication Update AnywhereUpdate Anywhere Data Transformation Data Transformation

for Datawarehousefor Datawarehouse Build Powerful Build Powerful

ApplicationsApplications

Reference InformationReference Information

http://www.software.ibm.com/data/http://www.software.ibm.com/data/dproprdpropr

http://www.software.ibm.com/data/http://www.software.ibm.com/data/datajoinerdatajoiner

http://www.as400.ibm.comhttp://www.as400.ibm.com http://as400bks.rochester.ibm.com/http://as400bks.rochester.ibm.com/

Trademark InformationTrademark Information AIX, AS/400, DataHub, DataJoiner, DataPropagator, DataRefresher, AIX, AS/400, DataHub, DataJoiner, DataPropagator, DataRefresher,

DB2, DProp, DRDA, IBM, IMS, MVS, OS/2, OS/400, RS/6000, VM/ESA DB2, DProp, DRDA, IBM, IMS, MVS, OS/2, OS/400, RS/6000, VM/ESA and VSE/ESA are trademarks or registered trademarks of the IBM and VSE/ESA are trademarks or registered trademarks of the IBM Corpopration in the United States or other countries or both.Corpopration in the United States or other countries or both.

Microsoft, Windows, and the Windows 95 and the Windows NT logos Microsoft, Windows, and the Windows 95 and the Windows NT logos are trademarks or registered trademarks of Microsoft Corporation.are trademarks or registered trademarks of Microsoft Corporation.

UNIX is a registered trademark in the United States and other UNIX is a registered trademark in the United States and other countries licensed exclusively through X/Open Company Limited. countries licensed exclusively through X/Open Company Limited.

Lotus Notes is a trademark of the Lotus Development Corporation.Lotus Notes is a trademark of the Lotus Development Corporation.

Other company, product, and service names may be trademarks or Other company, product, and service names may be trademarks or service marks of others. service marks of others.

introduction to heterogeneous data replication spring common 1999 richard sinn ibm santa teresa lab

Documents

n data

target n

subscription n

ipcs n

agenda n

data n loading dock

capture n capture

n replication vs