gummadi-47-shadowbase-technical-overview.final

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

HP47: HP Shadowbase Technical Overview

Ajaya GummadiWW NonStop Data & Cloud Product Manager

Paul J. HolensteinEVP, Shadowbase Products Group, Gravic Inc.

November 18, 2014


Introducing HP Shadowbase• Sophisticated replication capabilities

• Rich data and application integration options

• Available for NonStop and other servers (Unix, Linux, Windows)

• Numerous databases supported: NonStop SQL/MX, SQL/MP, Enscribe, IBM DB2, Oracle Enterprise, Oracle MySQL, SAP Sybase, Microsoft SQL Server

• Zero Downtime Migrations to migrate to a new platform or release

• Post replication database compare• Management utilities• Attractive Cost of Ownership (COO)• Backed by premium HP Support and Consulting Services

… and we’re just getting started


HP Shadowbase Portfolio

Business Continuity

• HP Shadowbase Basic Data Replication

• HP Shadowbase Advanced Data Replication

Data & ApplicationIntegration

• HP Shadowbase Basic Data and App Integration

• HP Shadowbase Advanced Data and App Integration

Data Utilities

• HP Shadowbase Data Management Utilities

• HP Shadowbase Compare

Proven in the Market

• Over 55 customers worldwide

• Around 300+ licenses• Shadowbase has

been in the market for over 25+ years

Best in Class products for a nonstop world

© Copyright 2014 Gravic, Inc. The information contained herein is subject to change without notice.4

HP Shadowbase Supported Platforms & Databases

Any ODBC Target Platform/Database (e.g., Teradata)

HP NonStop

SQL/MPSQL/MX

Enscribe

SAPSybase

HP NonStop

SQL/MPSQL/MX

Enscribe

Microsoft Windows

MicrosoftSQL

Server

SAPSybase

Unix/Linux

OracleEnterprise

OracleMySQL

IBMDB2

HP NonStop

SQL/MPSQL/MX

Enscribe

HP NonStop

SQL/MPSQL/MX

Enscribe

Uni-directional Replication and Data Integration

Bi-directional Replication and Data Integration

Sour

ceD

atab

ase Target D

atabase

Sour

ce/T

arge

t Dat

abas

e Source/Target Database

Linux, Unix

OracleEnterprise

Linux, Unix

OracleEnterprise

Microsoft Windows

MicrosoftSQL

Server

Linux, Unix

OracleEnterprise

Microsoft Windows

MicrosoftSQL

Server

Microsoft Windows

MicrosoftSQL

Server

HP Shadowbase

HP Shadowbase


Business Continuity Journey: End to end coverage

Business Case Due Diligence Delivery & Support

• Pre-Sales Discovery Service

• Gather customer requirements

• Determine RTO and RPO for each mission critical business service

• Gap analysis

• Business case creation

• Offered by Pre-Sales, typically at no charge

• Current environment analysis

• Determine the Business Continuity architecture

• Risk analysis

• Proposal development

• NonStop ATC support for POCs, Pilots, Migration

• Implementation Services (Installation, Configuration, User Exit Development, Performance Tuning)

• Support Customer testing

• Production cut-over support

• Training and Education

• Support services provided by HP Global NonStop Support Center ( Level 0 and 1 Support on NonStop and Level 0 for other servers)

Best in class services for a nonstop world


An Architectural Overview of Shadowbase

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7

Reading Change Data from the Source EnvironmentShadowbase Replication Technology

Shadowbase NonStop Audited Source Replication

• Shadowbase uses HP TMF ARLIB audit reading routines to capture/replicate DML/DDL changes

Shadowbase NonStop Non-Audited Source Replication

These OPTIONS allow Shadowbase to capture/replicate DML/DDL changes for non-audited files/tables:

• Option 1: Use HP AutoTMF to convert non-audited I/O into audited I/O

− Then use standard Shadowbase for replication

• Option 2: Use Shadowbase SOLV “Snap-Shot” File/Table Loading/Refresh

• Option 3: Use Shadowbase “File Chaser” utility (SOLV Manager)

Shadowbase Other Server Source Replication

• Oracle*, SQL Server Source: Trigger-based I/O capture

• Sybase Source: Log-based capture using Sybase Replication Server feed

− (*Note: Oracle Log-based source capture being released now)

Shadowbase Custom API Source Replication

• Programmatic method to inject I/O events into Shadowbase replication stream for non-supported sources

− For example, inject received middleware events into Shadowbase replication

− Typically implemented as a Professional Service…


Option 2 TCP/IP

AUDMON

Shadowbase for NonStop Server to NonStop Server

Shadowbase Replication Technology

System \A

CONSUMER

TARGETDATABASE

System \B

SOURCEDATABASE

AUDCOM

APPLICATION

TMFAUDITTRAIL

CONSUMER

AUDMON

COLLECTOR

Option 1 EXPAND

Standard HP Audit Read Routines

(ARLIBn)

Either approach can preserve source TX boundary and only replay committed TX data.


NonStop Server to NonStop Server - QMGR on Receiving Side


System \A

TARGETDATABASE

System \B

SOURCEDATABASE

APPLICATION

TMFAUDITTRAIL

COLLECTOR

EXPAND

CONSUMER

QMGRFILES

QMGR

QMGR Features:• Optional, on receiving side• Decouples delivery from data replay• Queues data to disk in Q Files (parallel writes)• Won’t decrease overall replay thruput but does add

some system overhead

QMGR Located on Target System


(ARLIBn)


NonStop Server to NonStop/SBOS Platforms – QMGR on Sending Side


System \A


(ARLIBn)

SOURCEDATABASE

APPLICATION

TMFAUDITTRAIL

COLLECTOR

CONSUMERQMGRFILES

QMGR

QMGR Features:• Optional, on sending side• Decouples delivery from data replay• Queues data to disk in Q Files (parallel writes)• Won’t decrease overall replay thruput but does add

some system overhead

QMGR Located on Source System

Option 2 TCP/IP

Option 1 EXPAND

Option 2Option 1 & 2Option 1


Bi-Directional NonStop Replication Architecture


System \A System \B

APPLICATION B

TMF AUDITA

APPLICATION A

TMF AUDITB

CONSUMER B

CONSUMER A

TRANSLOG(Cut-off)

A

TRANSLOG(Cut-off)

B

COLLECTOR A

DATABASEA

DATABASEB

COLLECTOR B


Option 3

Option 2bOther

TargetEnvironment

NonStop Server to Other Platforms

System \SOURCE Windows, UNIX, Linux, OS400, Open VMS, etc.

APPLICATION

COLLECTOR

Option 1

SOURCEDATABASE

TMFAUDITTRAIL

TCP/IP

CONSUMER

DOC

TARGETDATABASE

Other Server orNonStop Target

DOC WRITER

DIRECT WRITER

READER

TRS

Option 2a

Option 2


OPTION 1 does not preserve source TX boundary, and replays both committed as well as aborted data.

OPTION 2 & 3 can preserve source TX boundary and only replay committed TX data.


Option 3bOther

TargetEnvironment

Windows, UNIX, Linux Windows, UNIX, Linux, OS400, Open VMS, etc.

APPLICATION

COLLECTOR

Option 2

SOURCEDATABASE

TCP/IPTRANSACTIONFORWARD SERVER

DOC

TARGETDATABASE


DOC WRITER

DIRECT WRITER

READER

TRS

Option 3a

Option 3

EXTERNAL ROUTINES

Option 4

LOCAL orREMOTE DB

TRANSACTIONREPLAY SERVERDOC

Shadowbase for Oracle, SQL Server, and Sybase

Option 1



Shadowbase Scaling for Load (NonStop to NonStop)

Server \SOURCE

APPLICATION

SOURCEDATABASE

TMFAUDITTRAIL

CONSUMERCONSUMERCONSUMER

CONSUMERCONSUMERCOLLECTOR

1

1

Server \TARGET


CONSUMERCONSUMERCONSUMER

TARGETDATABASE

1N

1

1

Option 2TCP/IP

Option 1 EXPAND

N1


DOCDOC

Option 2b

Shadowbase Scaling for Load (NonStop to Other Server)

Server \SOURCE

APPLICATION

Option 1

SOURCEDATABASE

TMFAUDITTRAIL

TCP/IP

CONSUMER

DOC

TARGETDATABASE


DOC WRITER

DIRECT WRITER

TRS

Option 2a

Option 2

OtherTarget

EnvironmentCONSUMERCONSUMER

CONSUMERCONSUMERCOLLECTOR

DOC WRITER

DIRECT WRITER

DOC WRITER

DIRECT WRITER

TRSTRS

1N

11

N to 1or

1 to 1

1 N

N

N

1

1

Windows, UNIX, Linux, etc.


N


Database Event Capture and Delivery (NonStop Source, Replay from Guardian Space)

Shadowbase for Application Integration (1)

SelectedDB Activity

ApplicationIPC Feed

MiddlewareFeed

AND/OR AND/OR

Business Logic

CONSUMERCOLLECTOR

CustomerApplication

CustomerDatabase

Audit Trail(Changes)

TCP/IPFeed

AND/OR

Function:Shadowbase “sees” all changes to the customer’s database…and can act on them in real-time.

Uses:Shadowbase acts as a capture process for change events from the database/audit trail and notifies or delivers them to downstream files, applications, or middleware.

This architecture replays both committed as well as aborted TX data.




Database Event Capture and Delivery (SBOS Source)


APPLICATION

COLLECTORSOURCEDB

TRANSACTIONFORWARD SERVER

EXTERNAL ROUTINES

TRANSACTIONREPLAY SERVER

SelectedDB Activity

ApplicationIPC Feed

MiddlewareFeed

TCP/IPFeed

DOC

OPTION 2b

OPTION 1b

OPTION 1a

OPTION 2a

Both options only replay committed TX data.




DOC

Database Event Capture and Delivery (NonStop Source to SBOS Target)


TCP/IP

DOC WRITER

DIRECT WRITER

Option 1

Option 2a

Other Server or NonStop OSS Target

SelectedDB Activity

ApplicationIPC Feed

MiddlewareFeed

TCP/IPFeed

Option 2b

Option 2c

OPTION 1 replays both committed as well as aborted TX data.

OPTION 2 only replays committed TX data.

Business Logic

CONSUMER

APPLICATION AUDITTRAIL

COLLECTORSOURCEDB

NonStop Source

TRS


Feeding Data into Active/MQ and JMS

Shadowbase and JMS Queues (1)

For replication from the NonStop to an Active/MQ or JMS target environment, Shadowbase uses our standard architecture for interfacing via messages. In this architecture, a replication thread is configured from the NonStop source system to a DOC Writer on the Other Server target.

Only committed transactions will be read from the DOC by a ‘consumptive’ Transaction Replay Server (TRS) and converted into messages using a Shadowbase provided User Exit function. In this example, these messages will be sent to the Apache Active/MQ service directly, using Apache’s C++ API for Active/MQ. This architecture is shown in the following drawing.

Alternatively, if you need vender independence from the JMS provider, Gravic can also provide a Java bridge into JMS. In this architecture, the messages will be sent via TCP/IP from the consumptive TRS’s user exit to a Java program, which will then use JMS to send them to the remote system as shown in a subsequent diagram.


Feeding NonStop Data into Active/MQ via SBOS


NonStop Server (Production or Backup) Shadowbase Other Server Platform

APPLICATION

SOURCEDATABASE

DATABASE

TCP/IP

DOC

AUDITTRAIL

CONSUMER

TMF

DOC WRITEROTHER JMS

SYSTEMS

USER EXIT

CONSUMPTIVETRS

ACTIVE MQ

MQ Client ConnectionCOLLECTOR

This architecture only replays committed TX data.


Feeding NonStop Data into JMS via SBOS


NonStop Server (Production or Backup) Shadowbase Other Server Platform

APPLICATION

SOURCEDATABASE

DATABASE

TCP/IP

DOC

AUDITTRAIL

TMF

DOC WRITER

USER EXIT

CONSUMPTIVETRS

ACTIVE MQ orOther Provider

USER EXIT

JMS GATEWAY

TCP/IP

CONSUMER

COLLECTOR

OTHER JMS

SYSTEMS

This architecture only replays committed TX data.


Shadowbase Loading Technologies

Shadowbase Loading Technologies

• Offline Loading (Bulk Loading)

• Online Loading (SOLV Loading)


Bulk Loading via Utilities (Target Offline During Load)

Shadowbase Initial/Refresh Data Loading (1)

Sequence & Comments:• Configure Shadowbase, note TIMESTAMP of the Start of the Load…• Start & Run Bulk Extract/Bulk Loads Using External Utility (They Typically PURGEDATA on Targets) …TRANSFORMS needed?• During Bulk Load, Target Unavailable and Changes Queue in Audit Trails…• After Bulk Load, Replay Queue of Change Data into Target…TRANSFORMS needed?• Target Then Synchronized and Available for Work

TARGETDATABASE

1-Configure SB(Note Timestamp)

2b-Bulk Load

3a-Change Data

Change Data SHADOWBASE REPLICATION

ENGINESOURCE

DATABASE

APPLICATION

2a-Bulk Extract• FUP or SQL DUP or Copy/Load• PAK/UNPAK• Backup/Restore• ETL, etc.

3b-Replay Changes into Target

AUDIT TRAILS


Bulk Loading via Utilities (Target Online During Load)

Shadowbase Initial/Refresh Data Loading (2)

Sequence & Comments:• Configure & Start Shadowbase Change Data Capture (CDC) Replication…TRANSFORMS needed? CDC & Load use Same Set…• Start & Run Shadowbase Loads Using SOLV…Simultaneous with CDC/Replay…• During Load, Target Fully Available and CDC Consumed• After Load, Target Immediately “Caught Up” to Source’s Changes and NO Backlog CDC QUEUE Needs to be Drained

TARGETDATABASE

AUDIT TRAILS

SHADOWBASE REPLICATION

ENGINE

APPLICATION

SOLV

Change Data Change Data

Merged Data(Patented)

SOURCEDATABASE


Shadowbase Compare Technologies

Shadowbase Compare Technologies

• NonStop to NonStop

– Enscribe to Enscribe

– SQL/MP to SQL/MP

– SQL/MX to SQL/MX


NonStop Server to NonStop ServerShadowbase Compare Technologies

• Enscribe to Enscribe, SQL/MP to SQL/MP, or SQL/MX to SQL/MX Like-to-Like Only at this time− Full File/Table or Key-Range Comparisons Supported

• Non-Audited and Audited Enscribe Files and SQL Tables Supported• Same Node and Cross Node Comparisons, Using Expand (Partial TCP/IP Support at this time)• Source vs Target Configuration Determines Comparison ‘Polarity’: Both Source and Target can be OPEN…

− Additional, Missing, and Mismatched Records/Rows Identified• Application(s) and Any Data Replication Product Can Be Active, However False Positives May Result During Replication Latency

READ ONLY READ ONLY

APPLICATION

SOURCEDATABASE

SHADOWBASECOMPARE

TARGETDATABASE

SHADOWBASECOMPARE

NonStop Server NonStop Server

REPORT


Shadowbase Audit Compliance (1)

Shadowbase Audit Reader (SAR)

• NonStop TMF Audit Reading Tool

• Mine TMF Audit Trail Data Using SQL-like Syntax

Shadowbase Audit Log (SAL)

• Create an “Archive-Log” Database of Source Database Activity

• NonStop Source (Enscribe, SQL/MP, and SQL/MX)

• SBOS Target – Available Now (Oracle, SQL Server)

• NonStop Target – Future


ARLIB

Shadowbase Audit Reader (SAR)


APPLICATION

• SAR (TP-SPY) Reads Local/Active + Dumped TMF Audit Trail Files (Will request Restore if Needed)• MAT and AUX Trails Supported• Selective Event Display (All Audit Trail Events Supported, Including DDL and DML)• All File/Table Types Supported (Enscribe, SQL/MP, SQL/MX, Unstructured)• SAR Can Also Read “Foreign” Audit Trail Files Generated on Another SYSTEM (must be local to SAR though)• Uses an SQL-like Syntax for File/Table Selection, and Supports WHERE Clauses for Filtering• Determine When Your Data Was Changed, How it Was Changed, and Analyze TX Details (Long Running TX’s, Aborted TX’s,

Misbehaving Application Activity, etc.)

AUDITTRAIL

SAR READERSAR

CMD/DISPLAYREPORTS

DISPLAY

SOURCEDATABASE

TMF


PRODTARGET

Shadowbase Audit Log (SAL) – NonStop to SBOS Target


SAL Target Database• Typically One SAL Table Per Target Table (Uses Similar

Schema with Header Columns Added)• Enscribe, SQL/MP, SQL/MX Source Files/Tables

Supported• Captures Committed TX Data (Only)• Prepends SOURCE TX ID, TS, ADT LOC, etc.• Capture All or Selected Source File/Table Activity• One SAL Target INSERT per IUD• I/U = After Image, D = Before Image• Can use same or new DOC Writer/TRS Replication

Threads

System \A

SBOS Target

SOURCEDB

TM/MPAUDITTRAIL

COLL

APP DOC WRITER

DOCSALTRS

SALTARGET

PRODTRS

CONS


Shadowbase Data Recovery Software (1)

Shadowbase REDO Technology

• Rolling a database forward from a recovery point to re-apply selected changes

• Useful when schema/structure has changed or massive data changes have occurred

• Offline database recovery technique (application is down)

Shadowbase UNDO Technology

• Rolling selected database changes backward after corruption is detected

• Useful when schema hasn’t changed and not “too much” data has to be rolled back

• Online database recovery technique (application remains active)



Offline Database Recovery Technique

• Application does not remain active during the recovery

• Use ZDM if able, REDO is an (inferior) alternate approach

Basically, a Technique/Procedure to Recover a Damaged Database and Roll it Forward, Selectively Preserving Desired Updates (Skipping Those You Do Not Want):

• Preserve a Recovery Point Copy of your database (BACKUP, ONLINE DUMP, SPLIT MIRROR, etc.)

• Do the upgrade (usually disruptive, e.g., schema changes)

• Start the application, run for a while…

• Determine to “Fallback”

• Recover/Restore DB from Recovery Point Copy

• Roll-forward through changes to keep selected changes (either from audit or from REDO queue)

• User Exits needed??? Likely…especially if schema changed

Shadowbase REDO



Online Database Recovery Technique

• Application & database can remain online and active during the recovery

Basically, a Technique/Procedure to Recover a Damaged Database by Selectively Rolling Parts of it Backward:

• At any point when you notice database damage…usually after an application upgrade or a buggy program corrupts your data…

• Configure and run Shadowbase UNDO (e.g., setting time-based criteria)

• Review the reports/statistics

• Determine if correct, if not adjust and re-run Shadowbase UNDO

• When satisfied, tell Shadowbase to apply the UNDO queue changes

• Can fix-up source and/or target database (if replicating source to target, fixing source fixes target)

• User Exits needed??? Perhaps, but most likely not…

Shadowbase UNDO


Changes

Shadowbase UNDO

REPORTSReverse Corrupting Changes

UNDOQUEUE

DB

APPLICATION

COLLECTOR QMGR

USEREXITS

CONSUMERAUDITTRAIL

Shadowbase Data Recovery Software (4)Shadowbase UNDO

a) Decide to UNDO after upgrade operation (AUDIT saves database changes)

b) Determine to selectively roll back certain changes from AUDIT by capturing changes into UNDO queue; check report for accuracy of UNDO events…

c) Selectively roll-back changes while application stays active

d) Database selectively recovered from UNDO queue (complete)

e) If also replicating to a target environment, the changes made at source will propagate to target as part of normal replication operation…


Shadowbase UNDO


Original Change(Original Order)

UNDO Change(Reverse Operation)

Effect

Begin transaction (1) Commit transaction (5) Commit an UNDO transaction

Insert Row A (2) Delete Row A (4) An erroneously inserted row is deleted

Update Row B from x to y (3) Update Row B from y to x (3) An erroneous update is reversed

Delete Row C (4) Insert Row C (2) An erroneously deleted row is reinserted

Commit transaction (5) Begin transaction (1) Begin an undo transaction

Replay OrderOriginal Order


Worldwide Contacts and Resources

• Ajaya Gummadi, NonStop Data Product Manager ([email protected])

• Paul Holenstein, EVP, Shadowbase, Gravic Inc. ([email protected])

• www.hp.com/go/NonStopContinuity

(Customer accessible web portal)

• www.hp.com/go/NonStop-Docs

(All Shadowbase publications are customer accessible)

mailto:[email protected]

mailto:[email protected]

http://www.hp.com/go/NonStopContinuity

http://www.hp.com/go/NonStop-Docs


Thank you

gummadi-47-shadowbase-technical-overview.final

Documents

premium hp support

licenses shadowbase

shadowbase products

support training

nonstop world copyright

education support services

nonstop sqlmx

started copyright