www.lamkodb.com jos van lammeren [email protected]

1

Oracle GoldenGate

www.LamkoDB.com www.MonitorMyDatabase.com

Jos van [email protected]

2

Background GoldenGate• Overview• Setup• Operations• Troubleshooting• Monitoring• Challenges

Q&A


Agenda

3

Using Oracle GoldenGate since 2010 At clients• Oracle 9i, 10g near-zero downtime migrations to 11g RAC• Oracle 11g RAC setup for DR and EDW load• Upgrade OGG 10.4 to 11.2 for Oracle 11g clusters (no app

downtime)• Oracle 9i, 10g, 11g EDW load• SQL Server 2008 to Oracle 11g EDW load

At LamkoDB• Oracle 10g, 11g, 12c, SQL Server 2008, 2012, MySQL tests with

OGG 11g and 12c


Background

4

What GoldenGate does• Continuous data replication from one database to another• Source and target databases can be of different

vendors/releases• Non-intrusive, low-impact, sub-second latency • Maintains transactional integrity - Resilient against interruptions

and failures • Data transformation on the fly• Few or many tables in a schema• DDL replication if desired• Used for EDW feeds, DR, DB version upgrades, DB migrations

What GoldenGate does not• Is not a replacement for Data Guard


Overview

5www.LamkoDB.com www.MonitorMyDatabase.com

Overview

6

Use of replication


Overview

7

Source Databases/Platforms supported


Overview

8

Target Databases/Platforms supported


Overview

9

Processes/Files involved

Classic Capture or Integrated Capture mode Nonintegrated Replicat or Integrated Replicat


Overview

10

Observations• Not all data types are supported• Need data type translation in heterogeneous environments• Encryption of data in files and across network• Optional compression of data copied by pump process• Cryptic names for processes: 8 characters• Cryptic names of trail files e1xxxxxx (2 characters followed by

incrementing number)• Administrator must like command line tool (similar to sqlplus)


Overview

11

Installation GG software (source & target server)• Download software from Oracle site. Software to use is Oracle

DBMS release specific.• On Linux/Unix create account that owns the software & add to

dba group• Create profile to set $ORACLE_HOME, $PATH, etc. for the

account• Unzip & untar software or use Oracle Universal Installer• Configure GLOBALS and mgr.prm files

Installation in DB• DB in archive log mode (at least source instance)• Turn off recyclebin (in source instance for DDL replication)• Put DB in supplemental logging mode (source instance)• Run OGG supplied installation scripts (source & target)


Setup

12

How many schemas/tables to replicate & how much data will be replicated?

Have tables the same structure in source and target database or do target tables have additional columns?

Do tables have PKs? Are there referential constraints to deal with if not all

tables are being replicated? Delete Cascade constraints When bi-directional replication• Sequences• Triggers firing on either database


Setup items to keep in mind

13

Setup replication flow & initial load of data• Create/start Extract on source database• Create/start Pump process on source database• Do initial data load from source to target database• Create/start Replicat on target database


Replication Setup Example

14

Extract• Extract process E1ORA1• E1ORA1 extracts data from MMDBORA1 (Oracle) to local trail

files in directory/gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1


Setup

15

EXTRACT

E01ORA1.obey

ADD EXTRACT E01ORA1, TRANLOG, BEGIN NOW, params /gg/product/11.2.1_11g/ggs/dirprm/MMDBORA1/E01ORA1.prm

ADD EXTTRAIL /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1, EXTRACT E01ORA1

E01ORA1.prm

EXTRACT E01ORA1

. . .

USERID ggate, PASSWORD abc

EXTTRAIL /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1 megabytes 100

. . .

TABLE CHINOOK.*;

start E01ORA1


Setup

16

Pump• Pump process P1ORA1 copies local trail file contents to remote

trail files on another server in directory C:\gg\dirdat\MMDBORA1\p1


Setup

17

PUMP

P01ORA1.obey

ADD EXTRACT P01ORA1, EXTTRAILSOURCE /gg/product/11.2.1_11g/ggs/dirdat/MMDBORA1/e1, params /gg/product/11.2.1_11g/ggs/dirprm/MMDBORA1/P01ORA1.prm

ADD RMTTRAIL C:\gg\dirdat\MMDBORA1\p1, EXTRACT P01ORA1, MEGABYTES 100

P01ORA1.prm

EXTRACT P01ORA1

RMTHOST 192.168.21.94, MGRPORT 7809

RMTTRAIL C:\gg\dirdat\MMDBORA1\p1, megabytes 100

PASSTHRU

TABLE CHINOOK.*;

start P01ORA1


Setup

18

Initial load of data• Export/Import• Data pump• RMAN• Load data over a DB link• bcp• OGG utilities• many more …


Setup

19

Replicat• Replicat process R01ORA1 (SQL Server 2012) reads the remote

trail file contents and applies rows to the database


Setup

20

REPLICAT

R01ORA1.obey

DBLOGIN SOURCEDB GGATE, USERID ggate, PASSWORD abcadd replicat R01ORA1, exttrail C:\gg\dirdat\MMDBORA1\p1 params C:\gg\dirprm\CHINOOK\R01ORA1.prm

R01ORA1.prm

REPLICAT R01ORA1

SOURCEDB GGATE, USERID ggate, PASSWORD abc

. . .

MAP CHINOOK.ALBUM, TARGET dbo.ALBUM;

MAP CHINOOK.ARTIST, TARGET dbo.ARTIST;

MAP CHINOOK.CUSTOMER, TARGET dbo.CUSTOMER;

MAP CHINOOK.TRACK, TARGET dbo.TRACK;

start R01ORA1


Setup

21

Replication flow complete


Setup

22

ggsci command line interface• info all• info * or info E*, etc• info E01ORA1• info E01ORA1 detail• start E01ORA1• stop E01ORA1• stats E01ORA1• send E01ORA1 status• many more …


Operations

23

ggsci commands

GGSCI (MMDBLNXDB01) 1> info all

Program Status Group Lag at Chkpt Time Since Chkpt

MANAGER RUNNING

EXTRACT RUNNING E01ORA1 00:00:00 00:00:04

EXTRACT RUNNING E02ORA1 00:00:00 00:00:04

EXTRACT RUNNING P01ORA1 00:00:00 00:00:09

EXTRACT RUNNING P02ORA1 00:00:00 00:00:00

REPLICAT RUNNING R01SQL1 00:00:00 00:00:04

GGSCI (MMDBLNXDB01) 2> info E01ORA1

EXTRACT E01ORA1 Last Started 2014-02-20 15:49 Status RUNNING

Checkpoint Lag 00:00:00 (updated 00:00:02 ago)

Log Read Checkpoint Oracle Redo Logs

2014-03-13 21:52:56 Seqno 3726, RBA 25596928

SCN 0.127734271 (127734271)


Operations

24

Network issues causing pump to crash• Usually restart pump• Use AUTORESTART in mgr.prm


Operational Issues

25


Server reboot and processes not starting• Use AUTOSTART in mgr.prm


Operational Issues

26



Extract performance bad• Use dedicated volumes where redo/archived logs are stored• Have separate volumes for local trail files• Use multiple extracts


Operational Issues

27



Extract performance bad• Use dedicated volumes where redo/archived logs are stored• Have separate volumes for local trail files• Use multiple extracts

Pump performance bad• Compress data when sending data across• Have separate volumes for local/remote trail files• Use multiple extracts/pumps


Operational Issues

28

Replicat performance bad • Check for tables with missing PKs• Use batchsql in prm file of replicat• Have separate volumes for remote trail files• Use multiple replicats


Operational Issues

29

Replicat performance bad • Check for tables with missing PKs• Use batchsql in prm file of replicat• Have separate volumes for remote trail files• Use multiple replicats

Disk full for local and/or remote trail files• Occasionally need to setup replication again, with or without

initial load• Monitor disk space usage by storage group or sysadmins• Have separate volumes for local and remote trail files• Use PURGEOLDEXTRACTS in mgr.prm


Operational Issues

30

Crashing replicats when tables added/changed• Add/modify tables on target database, possibly update data

definition file, and restart replicat• Make extract process stop when it notices DDL


Operational Issues

31



Extract process does not stop• Commit open transactions in DB• Kill session having open transaction• Force stop of Extract process if not many archived logs to re-

read at next Extract startup


Operational Issues

32



Extract process does not stop• Commit open transactions in DB• Kill session having open transaction• Force stop of Extract process if not many archived logs to re-

read at next Extract startup Crashing replicats because of manual data changes in

target database• Do more manual changes to fix the data• Possibly reload affected tables• Educate users


Operational Issues

33

Where to look for information• Report file of process (view report <process>)• Discard file• OGG Error log• Oracle alert log / SQL Server ERRORLOG

Compare source and target data after fixing issue• Veridata• select data from source table MINUS select data from target

table• Oracle DBMS_COMPARISON• Other tools (TOAD, …)


Troubleshooting

34

Setup Heartbeat in replication flow• Document ID: 1299679.1 OGG_Best_Practice_-_heartbeat_table_using_DBMS_SCHEDULER-_V11_0_ID1299679.1.pdf

• Update heartbeat record in source database (every minute)• Extract picks up record in source database, adds information

and write to local trail file• Pump picks up record, adds information and write to remote

trail file• Replicat picks up record, adds information and writes to target

database (as an insert)


Monitoring

35

Monitor using Oracle tool (Director)• Need to install Java agent in every GG instance & maintain

bunch of servers to keep tool running (Weblogic, …) (Remote) Monitor using own developed tools• Monitor status of processes• Monitor number of rows replicated• Report on Heartbeat data• Queries against heartbeat table in target database• Graphs generated based on data in heartbeat table (lag

times)• ggsci-like GG Dashboard for all GG instances monitored


Monitoring

36

OGG Management Pack


Monitoring

37

LamkoDB Monitoring


Monitoring

38

Historical stats for Extract process


Monitoring

39

Lag times for Extract/Pump/Replicat


Monitoring

40

GG Dashboard


Monitoring

41

GG process detail information


Monitoring

42

GG process relationships


Monitoring

43

Who supports GoldenGate?• DBA• Middleware group


Challenges

44


What to do about table structure changes• software releases need to be evaluated/tested/coordinated


Challenges

45



What do developers, support, clients know about OGG?• Educate personnel


Challenges

46




Monitor what is running and what is broken• Have proper monitoring in place with alerting


Challenges

47




Monitor what is running and what is broken• Have proper monitoring in place with alerting

Keeping track of new, existing and changing replication flows. Number of processes quickly becomes a spaghetti bowl.• Have automated tool to show relationships


Challenges

48

Questions?


Q&A

Jos van Lammeren

[email protected]

www.lamkodb.com jos van lammeren [email protected]

Documents

g racoracle

g clusters

oracle site

source databasesplatforms

data types

g edw loadsql server

source instanceturn

g rac setup