hp dba v.6.2 technical slides

HP Database Archiving - technical details

These are technical slides to understand ‘how things work’ under the covers. This is

not a complete presentation by itself. Incorporate into your customer facing slide

decks & demos.

Confidential & Proprietary2

Technical Topics

• Deployment Scenarios• Data Selection & Reporting• Data Movement• Archive Access• Note: the above topics assume a working

knowledge of Designer to model tables and rules.

© 2008 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice

Deployment Scenarios


Maintain Online/Offline Storage

Typical Deployment Scenarios

DB DB

DB XML

DB DBDB XML

DB DBXML DBLoad

DB DB

XML

Performance Optimization, Subsetting

Compliance, Application Retirement, Long-term storage

Multi-tiered Archiving (full lifecycle)

Compliance Archiving with event-based load (to a temporary DB)

Copy

Maintain copy for long-term compliance(delete from archive DB as required)

Reload by Archive Run

or by Transaction (DB to DB only)


Data Selection


Data Selection• Standardized across all data movement methods

• Creates driving rows for next step in the process (data movement)

• Can be run standalone if desired (without automatically going to the data movement step)

• Eligibility Analytics™ reports provided for the business to investigate and clean-up transactions that are ‘stuck’ (i.e ineligible)

• Allows reporting on eligible and ineligible records

• Supports complex chained relationships between transactions with pre-built algorithms


Standard Selection• Selection determines archive candidates for the data

movement step and Eligibility Analytics™ (optional)• The Selection step is run for each cartridge and always

runs two processes – Selection and Eligibility

• Tables created at deployment time in the ‘interface’ schema

• One table for each transaction table• Selection rows populated during

selection & eligibility steps• Exclusion table populated during

eligibility step (optional)

Interface Schema

SelectionTables

ExclusionTables

Flags-Eligibility Analytics one

Keep Results

<source tablename>_SEL

<PK1 Column><PK2 Column>

OBT_ROW_SEQOBT_ROW_EXCLUDEDOBT_WF_RUN_IDOBT_GROUP_IDOBT_SAVED_ROWID (*)

<…>

<source tablename>_EXC

ELIGIBILITY _ RULEOBT _ROW _SEQOBT _WF _RUN _IDTABLE _CATALOGTABLE _OWNERTABLE _NAME

Identification by ROWID (Oracle only)

REPOSITORY


Standard Selection• Selection Process populates the selection tables by choosing

the rows that match the given policy and ‘unrecorded’ rules (as designated in Designer)

• Eligibility process uses ‘recorded’ eligibility rules to determine exclusions and records reasons (rows that cannot be archived)

SelectionProcess

Transaction TablesPolicy

Unrecorded Rules

EligibilityProcess

Transaction Tables

Recorded Rules

* Eligibility Analytics™ must be turned on in the Console to populate exclusion tables

Disqualified by Rule*

REPOSITORY

SelectionTables

ExclusionTables

Determines ‘scope’ of rows to act upon (e.g older than Jan-01-2005 and ‘closed’)

Run Eligibility

Analytics™ reports and/or

continue to data

movement step


Data Movement


Transactional Data Movement• No special coding required (standard platform

feature)• Ensures complete transactions (all related rows

across Designer model) are committed together• Maintains consistency of the production database

and archive at all times (even in the event of a failure)

• Provides high degree of scalability using job engine and database parallelism

• Supports ‘online’ archiving (end-users can remain on the system)

• Allows reload of individual transactions or entire archive runs

Confidential & Proprietary11 Confidential & Proprietary11

Data Movement Options

• Database to Database−Transactional (standard option)

−Advanced Options for Oracle (with additional licensing)BulkPartition-based

• Database to XML−Transactional only

• Platform provides both DB to DB and DB to XML capabilities (single platform license)

• Selection process is shared across all data movement techniques


Transactional Data Movement

SourceSchema

InterfaceSchema

SelectionTables

ExclusionTables

SELECTION VIEW

HistorySchema

Eligible Rows

COPY1

Commit size set in Console (# of driving table rows)

3

Archive target-In source or archive database

Next n rows

DELETE 2

1-Step Data Movement-Archive data inserted directly into archive tables.

First n rows


Data Movement Parallelism

SourceSchema

InterfaceSchema

SelectionTables

ExclusionTables

SELECTION VIEW

HistorySchemaCOPY

DELETE

Data movement throughput can be simply adjusted by the number of threads, batch size and/or database parameters based on the available # of CPUs.

Database Parallelism Method

Java Threads

Start Row-Sequence

End Row-Sequence

Thread#1 1 MAX

Database Threads Spawned and Managed by Database

Job Parallelism Method

Java Threads

Start Row-Sequence

End Row-Sequence

Thread#1 1 100

Thread#2 101 200

Thread#3 201 300

… … …

HP Job Engine uses workers to spawn threads & optimize throughput


Chaining ConceptChaining- Additional business relationships exists in addition to table relationships that inter-relate transactions. Any single transaction in a chain that is ineligible makes the entire chain ineligible.


Oracle EBS/AR Chaining Example

HP Database Archiving provides built-in algorithms to analyze chains based on the Designer model (no coding required)


Advanced Data Movement Options

• Bulk

• Partition-based (PDM)

• These options are available for Oracle database implementations only

Note: if Bulk Data Movement is used one-time as part of the services engagement then the customer does not have to license this option


Bulk data movement

Active Rows

Archive Eligible Rows

Active Data



Active Data


Source Table

Table to archive has

high percentage of eligible

rows (~50% or higher)

Goals: 1. Archive data quickly by eliminating deletes 2. Automatically re-organize source tables 3. Automate the process

Transportable Tablespace

Relocate to archive database

Drop source & rename or truncate & copy data back to original source

SelectionTables

ExclusionTables

StandardSelection

RetainTable

Copy Exclude

dRows

ArchiveTable

Copy EligibleRows

ActiveData


Adding value to Oracle Partitioning• Analyzes partitions for eligibility (partial or complete)• Identifies complete sets of data across partitioned and

non-partitioned tables (based on Designer model)• Allows full lifecycle management not just storage tiering

(until destruction)

ARCHIVEDATASTORE

Related non-partitioned data

Partitioned table

libretti

corrected caps


Partitioned data movement

Goals: 1. Leverage existing partitioning scheme (key range) 2. Consistent enforcement archiving policies & rules 3. Automated process

Copy to archive

database (using Data

Pump)

Swap Partitions

SelectionTables

ExclusionTables

PDMSelection

Sales Order Table

Copy Exclude

dRows

Copy EligibleRows

Archive Table

Retain Table

P0

…

P10

P11

P12

P13

P14P0

P0

…

… P10

P10

P11

P11

Partitioning key = Create Date

Current Year

CY -1

CY -2

CY - 3

Next Year

Status

Ship DatePolicy:Ship Date < today – 2 years

Today

P0

…

P10

P11

P12

P13

P14P0 … P10 P11

Truncate Tables


Archive Access


• XML Search/Retrieve−Long-term preservation of data−Search and view archive records (ASCII text view)−Load to database for more intense research/inquiries

• BI/Reporting tools access−Run custom queries against XML or Database

Archive−‘History Only’ access to data

• Transparency−Native application access & security (in addition to

BI/Reporting tools access)−Combined Reporting™ capabilities (current +

archive data in a single report)

Archive Access Options


XML Search/Retrieve

Search & Retrieve

.XSD

.XML.XSD

.XML

.XSD

.XML

Database to File

.XSD

.XML

.XSD

.XML.XSD

.XML

.XSD

.XML

Database to File

.XSD

.XML

.XSD

.XML.XSD

.XML

.XSD

.XML

Database to File

.XSD

.XML

Archive Data StoreArchive Data Store

REPOSITORY

INDEXES

Reload

SOURCE

TEMP DB


Reporting/BI Access

ArchiveDatabase

COLLECTIONS

POsPOs

ARCHIVE QUERY SERVER

SalesOrders

SalesOrders

InvoicesInvoices

Online/Offline StorageOnline/Offline Storage

.XSD

.XML.XSD

.XML.XSD

.XML

Database to File

.XSD

.XML

.XSD

.XML.XSD

.XML.XSD

.XML

Database to File

.XSD

.XML

.XSD

.XML.XSD

.XML.XSD

.XML

Database to File

.XSD

.XML

ReloadDatabase

Archive Data StoreArchive Data Store

DynamicIndexes

Archive AccessArchive Access

‘History Only Queries’SQL SQL

XSDTablesViews

Users


Transparency Defined

• Application level access− Leverages application security model− Provides users with same navigation as production system− Allows access to standard screens and reports in read-only mode− Preserves use of existing queries/reports (including third-party

reporting tools)

• Combined Reporting™− Allows current + archive data to be queried in a single

screen/report− Highly optimized across distributed databases− ‘History only’ option available (archive only transactions)

• Automated generation & synchronization− Archive schema creation− Creates all named objects− Re-runable processes to maintain synchronization with patches

etc.

More than union views!


Archive Access Schema

Transparency Architecture

Transparency

Generator

Application Metadata

Application Security

Transaction Data

Master Data

Reference Data

Packages Views Synonyms

ProductionSchema

Relocation tablespaceArchive tables

SCHEMA1

IndexesNon-managedtables

Managed tables

SCHEMA1

Indexes

Non-managedtables

Managed tables

SCHEMA2

Indexes

Non-managedtables

Managed tables

SCHEMA…

Indexes

Archive tables

SCHEMA2

Indexes

Archive tables

SCHEMA…

Indexes

Data Movement

Archive Access schemas are created using the Transparency Generator and are distinct from the archive data. Each production object is cloned and generated.Key Features-•Drop and generate AA schemas at any time•Two modes – Full or Net Change (just new objects)•Archive schema can be created in SOURCE or HISTORY database (configurable)



Transparency

Generator

REPOSITORY



Transaction Data

Master Data

Reference Data


ProductionSchema

Relocation tablespace


Models/table list

Managed tables

SCHEMA1


Managed tables

SCHEMA1

Indexes

Non-managedtables

Managed tables

SCHEMA2

Indexes

Non-managedtables

Managed tables

SCHEMA…

Indexes

Managed tables

SCHEMA2

Indexes

Managed tables

SCHEMA3

Indexes

Data Movement

BASE VIEWS

SubQuery

SubQuery

All managed tables are automatically generated as views – joined views if Combined Reporting™ is selected.



Transparency

Generator

REPOSITORY



Transaction Data

Master Data

Reference Data


ProductionSchema



Models/table list

Managed tables

SCHEMA1


Managed tables

SCHEMA1

Indexes

Non-managedtables

Managed tables

SCHEMA2

Indexes

Non-managedtables

Managed tables

SCHEMA…

Indexes

Managed tables

SCHEMA1

Indexes

Managed tables

SCHEMA1

Indexes

Data Movement

SubQuery

SubQuery

Dependent objects are automatically generated from the database catalogs.

BASE VIEWS

VIEWS PACKAGESSYNONYMS



Transparency

Generator

REPOSITORY



Transaction Data

Master Data

Reference Data


ProductionSchema



Models/table list

Managed tables

SCHEMA1


Managed tables

SCHEMA1

Indexes

Non-managedtables

Managed tables

SCHEMA2

Indexes

Non-managedtables

Managed tables

SCHEMA…

Indexes

Managed tables

SCHEMA2

Indexes

Managed tables

SCHEMA3

Indexes

Data Movement

SubQuery

SubQuery

Remaining objects are generated as synonyms to production objects (tables/views/synonyms etc.)

SYNONYM

S

Synonyms

BASE VIEWS


Non-managed objects



Transparency

Generator



Transaction Data

Master Data

Reference Data


ProductionSchema



Read/write access Read-only access

Managed tables

SCHEMA1


Managed tables

SCHEMA1

Indexes

Non-managedtables

Managed tables

SCHEMA2

Indexes

Non-managedtables

Managed tables

SCHEMA…

Indexes

Managed tables

SCHEMA2

Indexes

Managed tables

SCHEMA3

Indexes

Data Movement

SubQuery

SubQuery

SYNONYM

S

Synonyms

BASE VIEWS


10,000 objects = 10,000 objects

APPLICATION

Non-managed objects

Production Connection(apps server/db server)

Archive Connection(apps server/db server)


Key Transparency Features

• Full support for native application access and security for archive inquiries/reports

• Current + archive data in a single report/inquiry using Combined Reporting™ capabilities

• Session and query optimization

• Read-only access (including prevention of updates to production)

• Support for schema evolution (due to patching etc.)

• Run Transparency Generator in Full or Net Change mode

• Multi-schema support

hp dba v.6.2 technical slides

Technology

ig ib il

ig ib

io nr

im iz

end row

tms orb

xml database

tms