aw-ogg-201504

Copyright 2015, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracles products remains at the sole discretion of Oracle.

2


Oracle Advisor Webcast Oracle GoldenGate Troubleshooting Kee Gan Senior Principal Technical Support Engineer April 21, 2015

Copyright 2015, Oracle and/or its affiliates. All rights reserved. | 4

Agenda

1

2

3

4

OGG basics

Common OGG issues

Researching issues

Diagnostic settings

First Steps in troubleshooting an OGG issue

4

5


OGG Basics


dml TXN 2

dml

dml

dml dml

dml dml

dml TXN 3 dml

dml dml dml

dml dml dml

dml dml dml

commit dml dml

dml dml

commit

dml dml

dml

dml commit

EXTRACT TXN 1

TXN 1

TXN 2

TXN 3

TXN 3 dml, dml, dml,..



Transaction Log (Redo)

OGG trail

Capturing Transactions and Writing Trails dml

dml BEGIN


TXN 3

dml, dml, dml,..

TXN 2

dml, dml, dml,..


OGG trail Replicat

TXN 3

dml, dml, dml,..

TXN 2

dml, dml, dml,..

TXN 1

dml, dml, dml,..

BEGIN TXN

COMMIT TXN

Delivering Transactions to database

TARGET DATABASE

dml, dml, dml,..


Extract Pump connection with Remote Server

MANAGER

7809 EXTRACT PUMP

SERVER

OGG TRAILS

2015-04-02 11:59:36 INFO OGG-00963 Oracle GoldenGate Manager for Oracle, MGR.prm: Command received from EXTRACT on host [10.187.14.86]:63229 (START SERVER CPU -1 PRI -1 TIMEOUT 300 PARAMS ).

OGG-01677 Oracle GoldenGate Collector for Oracle: Waiting for connection (started dynamically). OGG-00963 Oracle GoldenGate Manager for Oracle, MGR.prm: Command received from SERVER on host [127.0.0.1]:63230 (REPORT 8240 7819). OGG-00974 Oracle GoldenGate Manager for Oracle, MGR.prm: Manager started collector process (Port 7819).

OGG TRAILS


Common OGG Issues

Database errors and database issues

Non-database errors

Hanging OGG groups

Performance issues

OGG group abending abruptly

Communication errors

Hardware failures (disk, server) and corruption

9


Database errors and database issues

Possible Causes: o Out of Sync with Source

o Foreign constraints

o Unique Indexes

o DDL change

o Target database updates by applications

o Resource unavailable, archived logs not present, privilege issues

o .many more


Database errors and database issues (contd) Examples:

2015-01-02 03:07:49 WARNING OGG-01004 Aborted grouped transaction on 'QDENA_ODS.QTPQITA', Database error 1403 (OCI Error ORA-01403: no data found, SQL ). 2015-01-30 16:47:42 ERROR OGG-00519 Fatal error executing DDL replication: error [Error code [942], ORA-00942: table or view does not exist SQL DROP TABLE SCANDS1 /* GOLDENGATE_DDL_REPLICATION */], no error handler present. 2014-12-05 11:37:50 ERROR OGG-00446 Oracle GoldenGate Capture for Oracle, eext01.prm: No valid log files for current redo sequence 18082, thread 1, error retrieving redo file name for sequence 18082, archived = 0, use_alternate = 0 Not able to establish initial position for begin time 2014-12-05 11:03:55.


Non-database errors Too many to pigeon hole! Examples: 2014-12-23 10:10:19 ERROR OGG-01028 Incompatible record (104) in /u01/app/oracle/goldengate/dirdat/tr008496, rba 98847946 (getting header). OGG-01006: Maximum records exceeded in discard file ({0,number,0})

Diagnostics: Check the report file* for errors. Fix the external cause and restart. Might need recovery of trails.

*Disk full/permissions/etc report file cannot be created or completed.


Hanging OGG groups

GGSCI > SEND MYEXT, STATUS Sending STATUS request to EXTRACT MYEXT ... ERROR: sending message to EXTRACT MYEXT (Timeout waiting for message). GGSCI > INFO , showch Note: Recovery Checkpoint for each thread as Re-starting can take a long time if recovery archived log sequence is far behind .etc Recovery Checkpoint (position of oldest unprocessed transaction in the data source): Thread #: 1 Sequence #: 643 RBA: 27038224 etc


Hanging OGG groups (contd)

GGSCI > SEND MYREP, STATUS Sending STATUS request to REPLICAT MYREP... ERROR: sending message to REPLICAT MYREP(Timeout waiting for message). Most common reason is locks on tables or rows by other apps running

Kill and start group to recover.


Performance issues

Common Causes:

Long transaction

Doing lots of fetch( FETCH parameter),

Full Table scans

Slow Database


Performance issues (contd)

Diagnostics:

10046 trace

AWR report

Health checks

ggsci > send status/showtrans > lag

Parameters REPORTCOUNT , REPORT


OGG group abending abruptly

Restart group If it ends abruptly very quickly, run from Unix shell > ./extract paramfile dirprm/myext.prm OR >./replicat paramfile dirprm/myrep.prm Check last lines for why OS terminate this process


OGG group abending abruptly (contd)

Example: /home/ogg > replicat paramfile dirprm/myrep.prm *********************************************************************** Oracle GoldenGate Delivery for Sybase Version 11.1.1.1.2 OGGCORE_11.1.1.1.2_PLATFORMS_111004.2100 etc Wildcard MAP resolved (entry dbo.*): Memory Fault


Communication errors

Examples:

2015-04-02 10:41:42 ERROR OGG-01224 TCP/IP error 10060 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.), endpoint: 12.1.2.3:7809; retries exceeded. 2015-04-02 11:59:45 ERROR OGG-01233 Send TCP params error: TCP/IP error 10054 (An existing connection was forcibly closed by the remote host.), endpoint: KKGAN-AU:7819.


Hardware failures (disk, server) and corruption

Most likely to get corruption on trails Check what is committed on target Rebuilt trail Adjust Replicat checkpoints

Resync / Recover Data ( Tables , Trails ) Trails Are Missing, Damaged, Corrupt or Suspect for Extract or Replicat (Doc ID 1614665.1)


Researching Issues Report files, useful for 99% of all OGG issues for errors and warnings.

OGG error log, ggserr.log. - timeline

Examine trails using logdump.

Discard files, text display of trail records that has issues.

Database alert logs. eg ORA-600

System alert logs, useful for performance issues, resource issues, pstack

Database performance utilities, health checks, AWR for performance issues, logminer

21


Diagnostic settings

Set reporting parameters and options

DDL REPORT

REPORT

REPORTCOUNT

SHOWSYNTAX

Avoid REPORTROLLOVER

22


First Steps in troubleshooting an OGG issue What was the error? Which group it is? Where can I find more detail of this error? the report file? What type of error is this? Is this error documented in the Error Messages Manual or a Knowledge Note?


Submitting an effective SR A Good description, version, platform, group, error, timeline. Report file


Summary What we covered today

Where to look for diagnostics info

Interpreting messages

Where to look for solutions

25


Take Away

Report file

Documentation

Knowledge Note

26


Top Articles and Community Links

27

http://docs.oracle.com/goldengate/1212/gg-winux/docs.htm (search Oracle GoldenGate Documentation)

https://community.oracle.com/community/business_intelligence/system_management_and_integration/goldengate (search Oracle GoldenGate User forum

www.oracle.com/goldengate


Learn More Available References and Resources to Get Proactive

My Oracle Support Essentials Webcast Series https://support. oracle.com | Doc ID: 553747.1

My Oracle Support Blog https://blogs.oracle.com/supportportal/

Get Proactive in My Oracle Support https://support. oracle.com | Doc ID: 432.1

Ask the Get Proactive Team [email protected]

28


Accessing My Oracle Support Community 1. Via My Oracle Support -> Community Tab

2. Directly : https://community.oracle.com/community/support

29


Session Related Community Links

The following thread will have a copy of the presentation and can be used for additional questions or discussions on this topic.

https://community.oracle.com/thread/3702944

30


Oracle Advisor Webcast Program Locating Current Schedule & Archived Recordings

My Oracle Support https://support.oracle.com

Document ID: 740966.1

Select your Product Area

31


Q & A

To ask a question on the phone line, select *1 on your phone.

To ask a question online, use the Q&A area at the top.

Your question will be read aloud in the order received.

Question can also be asked on the My Oracle Support Communities

32


THANK YOU

33

aw-ogg-201504

Documents

writing trails dml dml

database target database

oracle andor

txn commit txn

extract txn

oracle goldengate manager

ogg trails copyright

ogg basics copyright