h6540 oracle backup recovery perform avamar rman wp

Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN

Best Practices Planning

Abstract

This white paper provides an in-depth review of the capabilities of the EMC Avamar Oracle plug-in. The primary focus of this paper is two-fold:

1. Provide the reader with a complete understanding of Avamar Oracle plug-in capabilities including Oracle backup and recovery features and advanced concepts such as performance tuning

2. Describe how to protect large Oracle databases with the Avamar Oracle plug-in via multiple streams.

September 2009

Copyright 2009 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com

All other trademarks used herein are the property of their respective owners.

Part Number h6540

Oracle Backup, Recovery, and Performance Tuning using EMC Avamar and Oracle RMAN Best Practices Planning 2

Table of Contents Executive summary ............................................................................................5 Introduction.........................................................................................................5

Audience ...................................................................................................................................... 5 Benefits of using EMC Avamar..........................................................................5 Avamar and large databases .............................................................................6 Key features of using RMAN with the Avamar Oracle plug-in ........................7 Oracle database objects explained ...................................................................7

System Global Area (SGA) .......................................................................................................... 7 Oracle control files ....................................................................................................................... 7 Oracle datafile.............................................................................................................................. 7 Oracle tablespace ........................................................................................................................ 8 Oracle online redo logs ................................................................................................................ 8 Oracle archived redo logs ............................................................................................................ 8 Oracle parameter file (init(SID).ora or pfile)................................................................................. 8 Other Oracle objects .................................................................................................................... 8

Recovery catalog ................................................................................................9 Oracle hot or cold backups................................................................................9

Online (hot) backups.................................................................................................................... 9 Offline (cold) backups .................................................................................................................. 9

Target database preparation............................................................................10 Install the Avamar operating system file system agent. ............................................................ 10 Install the SBT library on the target database............................................................................ 10 Create users on the Avamar Grid, target database, and recovery catalog ............................... 11

Create a user called backup in the Avamar GUI ................................................................. 11 Create a user in the target DB................................................................................................ 11 Create users in the recovery catalog ..................................................................................... 12

Archive Log Mode.............................................................................................13 Oracle password file explained.................................................................................................. 14 Create your Oracle password file .............................................................................................. 14

RMAN and Oracle backups ..............................................................................15 RMAN backup script broken down ............................................................................................ 15

Avamar-specific parts in the RMAN script.............................................................................. 16 Universal parts of the RMAN script ........................................................................................ 16 RMAN backup script execution and sample output ............................................................... 16

Avamar and RMAN backup example......................................................................................... 17 My-Avtar-Flags.txt explained ...........................................................................18

Avamar and RMAN recovery scenarios for datafile recovery.................................................... 18 Recovery types in Oracle ....................................................................................................... 19 Avamar and RMAN Oracle disaster recovery ........................................................................ 19 Oracle Open Reset Logs command....................................................................................... 20

Performance tuning Oracle backups ..............................................................20 Identifying the client-side physical system bottleneck ............................................................... 20


Physical bottleneck................................................................................................................. 20 Avamar cache sizing for databases ................................................................22

Rules for sizing hash caches properly ....................................................................................... 22 Examples of properly sized hash cache files ......................................................................... 23

The case for multiple RMAN channels............................................................23 Example scripts and multiple Avamar cache files...................................................................... 24

Multiple channel RMAN Avamar backup script ...................................................................... 24 Multiple channel RMAN restore script.................................................................................... 25 Multiple channel Avamar My-Avtar-Flags file......................................................................... 25

Sample outputs .................................................................................................26 Conclusion ........................................................................................................28 Appendix A: Avamar Oracle plug-in recovery catalog creation ...................29

Create the catalog database instance in Oracle 9i .................................................................... 29 Create a Oracle database in Oracle 9i ...................................................................................... 29 Create catalog users.................................................................................................................. 30

Create the catalog schema .................................................................................................... 30 Register the target database.................................................................................................. 30

Appendix B: Avamar Oracle plug-in backup and recovery scripts ..............31 Appendix C: Avamar Administrator Oracle GUI backups .............................35

Snapup options .......................................................................................................................... 35 Restore options.......................................................................................................................... 36 Troubleshooting GUI operations................................................................................................ 37

Appendix D: Avamar and Oracle theory of operations..................................37 Avoracle theory of operations .................................................................................................... 37 Avoracle operations ................................................................................................................... 38

Browse.................................................................................................................................... 38 Backup.................................................................................................................................... 38 Restores ................................................................................................................................. 39

Appendix E: Troubleshooting, hints, and tips................................................40 Diagnostic commands: A sample session ................................................................................. 40 Different Var directories depending on operating system.......................................................... 41


Executive summary This white paper details procedures to enhance the performance and scalability of the EMC Avamar Oracle plug-in for larger Oracle databases. In most cases the Oracle database size was greater than 1 terabyte. The main goal of the testing was to demonstrate that the Avamar Oracle plug-in can address the data protection needs of these larger databases. The test sample database was approximately 5 terabytes in size. Multiple streams were used to achieve the recovery time objectives that were required.

Introduction This white paper describes backup and recovery options for Oracle via Avamar and Oracle Recovery Manager (RMAN) using the SBT_TAPE interface supplied in Avamar.

The primary focus of this paper is two-fold:

1. Provide the reader with a complete understanding of the Avamar Oracle plug-in capabilities including Oracle backup and recovery concepts and advanced concepts such as performance tuning

2. Describe how to protect large Oracle databases with the Avamar Oracle plug-in via multiple streams.

It should be noted that Avamar complies with the standards that are set forth by Oracle. A shared library is installed into the Oracle environment. This library allows Oracle RMAN to invoke the SBT interface. In addition, there are several field-tested samples of scripts and configurations.

Audience This white paper is intended for the technicians and database administrators (DBAs) who want to understand how to utilize Avamar for their Oracle data protection needs. This group includes customers, backup administrators, DBAs, technical consultants, delivery engineers, technical support engineers, and EMC internal professionals.

Benefits of using EMC Avamar The Avamar Oracle plug-in utilizes Avamars powerful source, global data deduplication technology. Unlike traditional backup solutions, EMC Avamar identifies redundant data segments at the source (client) before they are transferred across the network. By moving only new, unique subfile data segments, Avamar delivers fast daily full backups while reducing the required daily network bandwidth by up to 500X. This capability allows companies to utilize existing network bandwidth for backup and disaster recovery of remote offices and data centers, despite slow or congested networks and infrastructure. Data can be encrypted both in flight and at rest for added security, and centralized management makes protecting hundreds of remote offices easy and efficient.

By storing just a single instance of each subfile data segment globally, Avamar also reduces total back-end storage by up to 50X, enabling cost-effective disk-based recovery over extended periods of time. Although Avamar backs up data to disk, it can also work with existing tape and traditional backup software such as EMC NetWorker. In addition, EMC Avamars grid architecture provides online scalability, and patented redundant array of independent nodes (RAIN) technology delivers high availability.


Avamar and large databases Any data protection discussion should focus on the customers business requirements such as desired backup and recovery time objectives. Environmental factors should also be considered, especially for databases larger than 500 GB. Key factors and questions to consider are the following:

Database size and version? Available CPU and memory? What type of disk subsystem is available? What type of Avamar environment is in place and how large is it? Was the Oracle database installed on the storage system using Oracles best practices? By allocating multiple channels, and also having the proper environmental factors in place, Avamar is a perfect fit for larger Oracle environments.

Several rounds of testing have been done on 4 TB and 5 TB Oracle databases. Avamar was used to back up the database within a 10-12 hour nightly backup window.

Following are three real-world customer examples.

Example No. 1 Customer environment: 5 TB database hosted on a Sun Solaris machine with 128 GB of RAM (64 GB of free RAM) and 12 core processors. The back-end disk is EMC Symmetrix DMX. Customer has a recovery time objective of 24 hours and the backup window is 12 hours. Can the Avamar Oracle plug-in be leveraged to protect this configuration? Answer: Avamar would be a good fit here. You can allocate multiple RMAN channels to get the backups and restores done within their windows.

Example No. 2 Customer environment: Sun Solaris machine with 128 GB of RAM running 25 instances of Oracle. The system has 12 KB of RAM free. Can the Avamar Oracle plug-in be leveraged to protect this configuration? Answer: In this case, the machine is already memory constrained with its current load. You can try Avamar for backup and recovery, but the RAM bottleneck will still be there.

Example No. 3 Customer environment: 30 TB data warehouse hosted on a large AIX server with plenty of RAM and processors, and a very fast disk subsystem. Can the Avamar Oracle plug-in be leveraged to protect this configuration? Answer: It would be very difficult to complete the backup within the standard 10-12 hour backup window. Avamar is likely not the best fit.


Key features of using RMAN with the Avamar Oracle plug-in The Avamar Oracle plug-in is a fully integrated third-party module that is designed to leverage Oracles RMAN. By using RMAN, all of the key features of Oracles backup utility are available to Avamar. RMAN is Oracles preferred method to back up and restore an Oracle database. You essentially run RMAN and call for a SBT device type. This means RMAN data is sent to a third-party backup provider like Avamar.

Following is a short list of key RMAN features.

Online backups Full and Incremental Offline backups Full and Incremental Archived redo log backups Dedicated Oracle database to track backups recovery catalog Oracle block-level corruption detection Restore capabilities Full system restore Datafile restore Tablespace level restore Point-in-time restore Archived redo log restores In recent versions of Oracle, RMAN provides some advanced functionality. Please refer to Oracle documentation for a complete listing.

Oracle database objects explained The following section provides an explanation of all physical objects in the database that need to be protected.

System Global Area (SGA) When you mount or open an Oracle database, it invokes some background process, then reads what is called a control file and loads the SGA. In simple terms the SGA is the area where Oracle does its work. The more work you can do in RAM the faster the work will get done. The SGA is where your queries are processed, it is a temporary area for your data blocks, and it is also where a good deal of the RMAN work is processed.

Oracle control files This file is a small binary file. It keeps track of the state of the entire database environment, and the system change numbers (SCNs) of all objects. The database is useless without the control file. There is typically a group of at least three of these. Oracle recommends them to be on separate spindles. They are the registry of the Oracle system. An example is /u01/oracle10g/ctrl1.ctl.

Oracle datafile This is a physical file that typically has tables of data inside it. It is where the data is essentially stored. This object can be put online or offline. A single datafile can be backed up or restored.

An example is F:\oradata\payroll.dbf.


Oracle tablespace A tablespace is a logical collection of datafiles. It is a logical object, not a physical object. It can be confused with an Oracle data table. It is useful to recover an entire tablespace depending on where the database corruption occurred and how far it spread. The entire tablespace can be taken offline and online. Examples are /u01/oradata/payroll.dbf, /u01/oradata/currency.dbf, and /u01/oradata/taxes.dbf.

Oracle online redo logs These objects are where data is stored temporarily before it is committed to a datafile. It allows you to roll back the database to a point in time in case of a corrupted block or if a mistake is made.

Online redo logs are typically in groups; however, there is a limited amount of space in the online redo logs. Lets say we have three online redo logs. You fill the first, then the second, then the third. You would then start back and overwrite the first log. This is fine for a smaller database with a small amount of change. You typically do not back up online redo logs unless you shut the database down. This is because they are constantly changing

To keep a history of changes in the database, activate Archive Log Mode. Archived Log Mode allows you to make an archive copy of the log before it is overwritten. Examples of online redo logs are E:\oradata\logs\rd_1.log, E:\oradata\logs\rd_2.log, and E:\oradata\logs\rd_3.log.

Oracle archived redo logs Archived redo logs (also referred to as archive logs) keep a history of the database in case you want to roll back to a point in time. These archive logs are key objects to protect in the backup and recovery design. RMAN does a great job of protecting these and also utilizing them during recovery efforts. Examples are E:\oradata\arch\log_1.arc, E:\oradata\arch\log_2.arc, and E:\oradata\arch\log_3.arc.

Oracle parameter file (init(SID).ora or pfile) The parameter file tells the Oracle database about the environment when it loads the SGA, and reads the control files. It sets important parameters such as Archive Log Mode, how much RAM to use for SGA, and many others. It is similar to an .ini file in Windows.

This used to be a flat file but has changed to be more like a control file in later versions of Oracle. This file is protected in Oracle 9i and later by using the control file Autobackup function.

Other Oracle objects Oracle objects (control files, datafiles, and archived redo logs) make up the physical objects that are protected with Avamar and Oracle RMAN. The following objects are objects inside the database. They help keep the system running efficiently. Tablespace: Mentioned previously, it is a logical grouping of datafiles. Index: Allows you to be more efficient with queries. Table: Where you store data. It is comprised of rows and columns. Various datatypes. Constraints: A DBA term for rules with data. Views: Logical views of specific data. For example, V$Database allows you to see statistics on the

database. Sometimes referred to the system schema. Instance: This is the actual target database you are working with. There may be a production instance,

a test instance, and also a development instance. This is typically referenced by a System Identifier (SID) name. A good example of this may be Payroll_Prod.


Recovery catalog The recovery catalog database is an Oracle database that is created and used to keep track of the RMAN objects and allows you to recover them if you lose your target database control file. The target database is the database that you are backing up. This catalog also allows you to perform advanced reporting on backups and lists the target database schema.

If you choose not to use the recovery catalog database, your RMAN information is then stored in the target databases control file.

Later versions of Oracle 9i have a featured called control file Autobackup, which makes a snapshot copy of the control file and parameter file, and stores them in a location that you can take with the normal Avamar file system backup. You can then recover the control file directly from the Autobackup location. This is a configurable location.

It is important to understand that Oracle recommends that all production RMAN environments use a recovery catalog. However Test and Dev environments running Oracle 9i and later can use the option. Be careful on what the DBAs perception is. Most will push to use the recovery catalog either way.

The recovery catalog database should be running on a separate machine from the target database, which will allows you to get your environment up and running quicker. Imagine the difficulty in having to recover the catalog and then the target database.

Oracle hot or cold backups

Online (hot) backups A backup of your environment while it is running is referred to as a hot backup or online backup. The database must be running in Archive Log Mode to run a hot backup. This means that you are making copies of online redo logs and archiving them before you overwrite them. It is a setting within the database.

RMAN is not needed to perform this hot backup. Simply put your Oracle tablespace in hot backup mode and then perform a backup of the Oracle objects using the regular file system backup technology.

An example is as follows: Alter Table space Payroll begin backup within the Oracle system.

To use RMAN to perform hot backups, the database must be running in Archive Log Mode.

Offline (cold) backups A backup of your environment while it is shut down is an offline backup or cold backup. You do not need to be running in Archive Log Mode to do this. Simply issue the proper shutdown command within Oracle and then back up physical objects using the Avamar File System plug-in. While the database is shutting down, all of the SGA and buffers are flushed to their logs and datafiles.

RMAN can be used to perform a cold backup. This is a little tricky because you then embed a SQL statement into the RMAN script. Shut down the database normally, then start up the Oracle system with the Mount Option. This is like a single user mode in UNIX. Then use RMAN to back up the objects.


Target database preparation The following are the steps involved in getting the target database ready for a backup with Avamar and RMAN.

1. Ensure that the Avamar File System client is installed and the client is registered to the Avamar Utility Node. Please refer to the EMC Avamar Oracle Client Installation Guide and Reference Manual.

2. Install the Avamar SBT_TAPE library into your target database. This is outlined well in the EMC Avamar Oracle Client Installation Guide and Reference Manual.

3. Create a user within the target database and grant that user privileges. Also create the same user in Avamar. If you are using a recovery catalog database, create that user in the recovery catalog as well.

4. Put your target database into Archive Log Mode. This is required for hot backups.

5. Create an Oracle password file on the target database.

Install the Avamar operating system file system agent. This is a key step to make sure the client and server are communicating. The EMC Avamar Oracle Client Installation Guide and Reference Manual has more details. You should be able to test a small backup of the file system before you begin to test the Oracle backups. It is recommended that you back up and restore the operating system data (file or file system) before you attempt to back up and restore the Oracle data.

Install the SBT library on the target database This is somewhat similar and well documented for each operating system. The installer installs the Avamar Library into a directory like /usr/local/avamar/lib. Reference the EMC Avamar Oracle Client Installation Guide and Reference Manual for specific installation instructions.

This depends on the OS. You may need to shut down your target database and bring it back up to bind the SBT_TAPE library with the Oracle kernel.

An example of this file in UNIX is called libobk_avamar.so. Once this library is in place you will be able to reference it in the Avamar RMAN script.

The following samples are performed on an Oracle 10g system with Avamar version 4.x.

Example of the installation on Red Hat. rpm -hi AVAMARRMAN.rpm

You can then log in to the database with SQL plus.

oradba@oratest1# sqlplus /nolog

connect / as sysdba shutdown immediate startup exit

This step may be necessary to bind the SBT library with the Oracle kernel.


Create users on the Avamar Grid, target database, and recovery catalog You will need to create users within Avamar, the target database, and the recovery catalog. You will also need to grant the proper privileges to these users. Follow these examples.

Create a user called backup in the Avamar GUI The following are steps to create a user by utilizing the Avamar Administrator.

1. Start Avamar Administrator. Choose Navigation > Administration or click the Administration launcher button. The Administration window appears.

2. Click the Account Management tab. In the tree, select the client you want to add the new user to.

3. Choose Actions > Account Management > New User(s) or click the toolbar icon shown immediately to the left. The New User dialog box appears. Use the Axion Authentication System. It is the default. 4. Create a user backup with a password backup. Type the password backup to confirm. If you need help with these steps please consult the EMC Avamar System Administration Manual. There are some graphical examples in this guidel

Create a user in the target DB Create a user backup in the target database and grant this user privileges. The samples below show proper syntax. This procedure may vary a little depending on which version of Oracle you are running. However, these examples should get you fairly close. Please check with your Oracle DBA if you find issues. This is an example from UNIX creation. These commands are run from the target system.


SQL*Plus: Release 9.2.0.1.0 - Production on Wed Apr 23 21:33:25 2003 Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

SQL> connect / as sysdba

Connected to: Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.1.0 Production

SQL> alter user backup identified by backup; Oracle 9 User altered.

SQL> alter user backup account unlock; Oracle 9 User altered

You may need to identify the users default tablespace. Here is an example.

SQL> alter user backup identified by backup DEFAULT TABLESPACE users;

User altered. Here is a more elaborate example of the preceding tablespace declaration.


SQL> create user username identified by password default tablespace user_tablespace temporary tablespace temp_tablespace;

Remember to unlock the account!

SQL> alter user backup account unlock; Oracle 9i and above. User altered.

SQL> grant sysdba to backup; Universal Grant succeeded.

SQL> exit Disconnected from Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.1.0 - Production

Create users in the recovery catalog You can create a user/password called backup in the recovery catalog and grant this user the proper privileges. It is essentially the same procedure used to create the user backup with a password backup in the target database. You will need to log in to the machine that is running the recovery catalog database. This procedure may vary a little depending on which version of Oracle you are running. However, these examples should get you fairly close. Please check with your Oracle DBA if you find issues. This is an example from UNIX creation. These commands are run from the target system.

oradba@rcvcat# sqlplus /nolog


SQL>connect / as sysdba

Connected to: Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.1.0 Production

SQL> alter user backup identified by backup; Oracle 9 User altered.

SQL> alter user backup account unlock; Oracle 9 User altered.

You may have to identify the users default tablespace. Here is an example.

SQL> alter user backup identified by backup DEFAULT TABLESPACE users;

User altered.

Here is a more elaborate example of the preceding tablespace declaration.

SQL> create user username identified by password default tablespace user_tablespace temporary tablespace temp_tablespace;

Remember to unlock the account!


SQL> alter user backup account unlock; Oracle 9i and above. User altered.

SQL> grant sysdba to backup; Universal Grant succeeded.

SQL> grant connect,resource,recovery_catalog_owner to backup; Grant succeeded.

SQL> exit Disconnected from Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.1.0 Production

Archive Log Mode If you will be doing hot (online) backups, which is likely, you will need to put your target database in Archive Log Mode. Here are some examples of doing so. This procedure may also vary with different versions of Oracle.

Archive logging is enabled by updating the log startup parameter.

If you are running Oracle 9i with a PFILE, add the following entry to the init.ora file for the database to be backed up. Example: /space/home/oracle/admin/eval/pfile/init(sid).ora

log_archive_start = true If you are running Oracle 9i with an SPFILE do the following. You will not need to do this with Oracle 10g and later.


SQL*Plus: Release 9.2.0.1.0 - Production on Thu Apr 24 17:38:56 2003 Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected to: Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the Partitioning, OLAP and Oracle Data Mining options SQL> connect / as sysdba SQL> alter system set log_archive_start=true scope=spfile; System altered. SQL> exit

Disconnected from Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release 9.2.0.1.0 Production.

Now we can actually put the database into Archive Log Mode. oracle@oratest1# setenv ORACLE_SID targetdb oradba@oratest1# sqlplus /nolog SQL*Plus: Release 9.2.0.1.0 - Production on Thu Apr 24 17:38:56 2003 Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected to: Oracle9i Enterprise Edition Release 9.2.0.1.0 - Production With the Partitioning, OLAP and Oracle Data Mining options


SQL> shutdown immediate Database closed.


Database dismounted. ORACLE instance shut down

SQL> startup mount ORACLE instance started. (This is like single user mode in UNIX.)

SQL> alter database archivelog; Database altered.

SQL> alter database open; Database altered.

SQL> archive log list Database log mode Archive Mode Automatic archival Enabled Archive destination /space/home/oracle/product/9.2.0/dbs/arch Oldest online log sequence 1 Next log sequence to archive 3

SQL> exit

Oracle password file explained You can create a password file on the target database system, as well as the recovery catalog system. This will help avoid authentication problems when trying to make connections to get backups and recoveries started.

The following is a short explanation of the password file and what it is used for.

If the DBA wants to start up an Oracle instance there must be a way for Oracle to authenticate this user. That is, if they are allowed to do so.

Obviously, the users password cannot be stored in the database, because Oracle cannot access the database before the instance is started up. Therefore, the authentication of the user must happen outside of the database.

There are two distinct mechanisms to authenticate the user: using the password file or through the operating system. The init parameter remote_login_passwordfile specifies if a password file is used to authenticate the DBA or not. If set either to shared or exclusive, then a password file will be used.

The following is an appropriate example of the creation of this password file. You will use the orapwd utility.

Create your Oracle password file These are the commands required to create the password file on the Oracle database system.

oracle@oratest1# setenv ORACLE_SID targetdb (or catalog db ) oracle@oratest1# cd $ORACLE_HOME/dbs /oracle/product/10.1.0/Db_1/dbs

oracle@oratest1# orapwd file=/oracle/product/10.1.0/Db_ 1/dbs/orapw password=PASSWORD

Where PASSWORD is the sys account password on the database that is pointed to by ORACLE_SID. Verify that the Oracle password file was created by entering:


ls -la orapw

The following information appears in your command shell:

-rwSr----- 1 oracle oracle 1536 Oct 1 17:02 orapw

Now we can have Oracle read the passwd file when Oracle starts up. This next step is required for the password file to be read during database initilization. oracle@oratest1# vi /oracle/product/10.1.0/Db_1/dbs/initdbname.ora (or the equivalent initSID.ora file for your database) un-comment the following entryin the init.ora file

REMOTE_LOGIN_PASSWORDFILE=EXCLUSIVE

Restart the database in normal mode. oradba@rcvcat# sqlplus /nolog



SQL> shutdown immediate;

SQL> startup;

SQL> Exit

RMAN and Oracle backups There are a few different ways to invoke RMAN backups with Avamar.

Invoke RMAN from the Avamar Administrator. Please review Appendix C: Avamar Administrator Oracle GUI backups on page 35.

Invoke RMAN manually from the command interpreter (good for testing) Invoke RMAN via the OS scheduler (cron) using RMAN cmdfiles. The Oracle 9i syntax is:

o oradba@targ1# rman cmdfile=fullbackup.rman o oradba@targ1# rman @`fullbackup.rman` (the most popular option)

These commands can also be invoked by the OS scheduler (Cron). Invoke RMAN via a stored script inside the recovery catalog.

o oradba@targ1#rman target backup/backup@payroll catalog backup/backup@rcvcat o RMAN>run rmanscriptname

RMAN backup script broken down Here is an example of a basic RMAN backup script. The Avamar-specific lines are in blue.

run


{ allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so, ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';

backup database; release channel T1;

}

Avamar-specific parts in the RMAN script This line helps RMAN find the libobk sbt library.

PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so, ENV=(PATH=/bin:/usr/bin)";

This should be on a single line. The "ENV=(PATH=/bin:/usr/bin)" sets the environment variable so that the Avamar script can find "avtar" and "uname".

This line helps with the proper Avamar Avtar flags.

send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';

Universal parts of the RMAN script run

{ allocate channel T1 type 'SBT_TAPE' backup database; release channel T1; }

Plenty of other backup and recovery options can be added to this script and most DBAs will have their own scripts already written. The only action needed is to copy the PARMS= line and the send line to their script. The my-avtar-flags.txt file will also need to be created.

RMAN backup script execution and sample output (These samples are performed on an Oracle 9i system with EMC NetWorker for sample purposes.)

oracle@legato1% rman cmdfile backup_database.rman

Recovery Manager: Release 9.2.0.1.0 - Production Copyright (c) 1995, 2002, Oracle Corporation. All rights reserved. RMAN> connect target rman/rman@eval 2> connect catalog rman/rman@rman 3> run { 4> allocate channel T1 type 'SBT_TAPE'; 5> backup database plus archivelog delete input; 6> } 7> connected to target database: EVAL (DBID=1928152189)


connected to recovery catalog database allocated channel: T1 channel T1: sid=12 devtype=SBT_TAPE channel T1: NMO v4.0.0.0 Starting backup at 23-APR-03 current log archived channel T1: starting archive log backupset channel T1: specifying archive log(s) in backup set input archive log thread=1 sequence=6 recid=1 stamp=492128357 channel T1: starting piece 1 at 23-APR-03 channel T1: finished piece 1 at 23-APR-03 piece handle=04elai38_1_1 comment=API Version 2.0,MMS Version 4.0.0.0 channel T1: backup set complete, elapsed time: 00:00:25 channel T1: deleting archive log(s) archive log filename=/space/home/oracle/product/9.2.0/dbs/arch1_6.dbf recid=1 stamp=492128357 Finished backup at 23-APR-03 Starting backup at 23-APR-03 channel T1: starting full datafile backupset channel T1: specifying datafile(s) in backupset including current SPFILE in backupset including current controlfile in backupset input datafile fno=00001 name=/space/home/oracle/oradata/eval/system01.dbf input datafile fno=00002 name=/space/home/oracle/oradata/eval/undotbs01.dbf input datafile fno=00005 name=/space/home/oracle/oradata/eval/example01.dbf input datafile fno=00011 name=/space/home/oracle/oradata/eval/test1a.dbf input datafile fno=00012 name=/space/home/oracle/oradata/eval/test1b.dbf channel T1: starting piece 1 at 23-APR-03 channel T1: finished piece 1 at 23-APR-03 piece handle=05elai44_1_1 comment=API Version 2.0,MMS Version 4.0.0.0 channel T1: backup set complete, elapsed time: 00:01:26 Finished backup at 23-APR-03 Document Number: 1-1033-01 NetWorker Module for Oracle 8i and 9i on Solaris Evaluation Guide 15 channel T1: backup set complete, elapsed time: 00:00:07 channel T1: deleting archive log(s) archive log filename=/space/home/oracle/product/9.2.0/dbs/arch1_7.dbf recid=2 stamp=492128476 Finished backup at 23-APR-03 released channel: T1 Recovery Manager complete.

Avamar and RMAN backup example The following is a sample RMAN script that will back up the entire database to Avamar, including the archived redo logs. Backup database and archived logs This could be saved to a file called fullbkavmar.rman.

connect target backup/backup@evaldb


connect catalog backup/backup@rcvcat

run { allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; backup database plus archivelog delete input; release channel T1; }

Then invoke the script like this. oracle@test1# rman cmdfile fullbkavamar.rman

My-Avtar-Flags.txt explained The following is a sample copy of the My-Avtar-Flags file with flag descriptions and values.

--debug --pidname=Oracle --pidnum=1002 --logfile=/usr/local/avamar/var/avtar.log --vardir=/usr/local/avamar/var --id=backup --ap=backup --path=/clients/vaerp-db4.va.company.org

Where --id is the username within the Avamar utility node. If you specify "--id=backup"you are implicitly specifying "--id=backup@avamar/". The "avamar" denotes that the "backup" user is in the default Avamar authentication system, rather than, say, in an external authentication system. The "/" signifies that the "backup" user is a root-level user within the Avamar application. While it is acceptable to perform the initial tests with a root level user (in this case, "backup"), for security reasons, it is best to define an account user that is specifically authorized to perform only backups for that particular client account. In this case, that is the following: "--id=backup@avamar/clients/clientname".

--ap is the password for the user. --pidnum is a number for the OS of the target database.

Linux 1002 Solaris 2002 Windows 3002 HP-UX 4002 AIX 5002

--path is the Avamar domain and client of the target database. /clients/clientname

Avamar and RMAN recovery scenarios for datafile recovery Recovery scenarios are more detailed in Oracle due to the rules to recover. A couple of terms are important to point out here.


RMAN Restore Command This command initiates the actual physical rebuild of the Oracle object

that you need. It is rebuilding the data from the Avamar Storage Node. This is the physical restore. RMAN Recovery Command This command allows you to bring the database to a consistent point

in time. Depending on the options, you may need to apply the archived redo logs. This is the logical recovery.

Recovery types in Oracle It is important to discuss the different restore/recovery scenarios in Oracle. This may help you understand the rules to recovery when approached by each scenario. You will want the DBA to help you understand what type of recovery scenario is required for the Oracle system.

Using existing control and parameter files If you have control files and Pfiles in place you are in good shape for most recovery scenarios. You will be able to bring your database to a point in time or recover a datafile or tablespace. Following is an example of how this would look in an Oracle RMAN script. (Most recoveries are done in this fashion.)

connect target backup/backup@targetdb connect catalog backup/backup@rcvcat

run { allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; restore datafile '/d1/dev3data/usr01.dbf' ; recover datafile '/d1/dev3data/usr01.dbf'; release channel T1; }

NOTE: You might need to change the "--id" and "--ap" parameters in the "my-avtar-flags.txt" file to specify a user that is authorized to perform restore operations.

Avamar and RMAN Oracle disaster recovery With a complete system outage the control file must be restored/recovered before anything else can be done. Start up the database in nomount mode then recover the control file. Then mount the database and then restore/recover the database. The following is an example of how this would look in a RMAN script. The DBA should be nearby for this test.

Note the resetlogs option here.

connect target backup/backup@targetdb connect catalog backup/backup@rcvcat

set DBID = 92755261

startup force nomount; run { allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; restore controlfile; alter database mount; recover database; alter database open resetlogs; release channel T1; }


You will find various permutations of this exercise depending on what needs to be recovered and rolled back. You can set the SCN to find a point in time to recover the database from.

It is also important to note that Oracle 9i and later have the auto control file backup option. If this is being used the syntax to restore the control file is slightly different.

Oracle Open Reset Logs command The previous example showed an example of the alter database open resetlogs. The example above reflects an incomplete recovery scenario. This is basically a recovery of an Oracle object that may be out of sync with the rest of the database. This could be due to the fact that you have lost an archive log or a user caused database corruption.

If we take this down another level, the system change numbers (SCNs) of various objects are out of sync with the rest of the database, and the resetlogs command should be used to synchronize the database.

The resetlogs command starts a new database logical life or a new set of SCNs. It records the beginning of a new DB life and the old SCNs are maintained. The counter is not reset back to zero.

The old SCNs are left behind, which are sometimes referred to as incarnations of the Oracle system. Once the database is opened with the resetlogs command, the entire database should be backed up immediately. It is a difficult task to restore from a previous incarnation of the database.

Performance tuning Oracle backups

Identifying the client-side physical system bottleneck It is apparent that when architecting the proper Oracle backup and recovery solution, there will be a some bottlenecks within the Oracle system. This lends to a discussion around what exactly are the environmental limitations that you will have to work with. There are several variables that are set inside the database itself, including the System Global Area size. This area is where the main RMAN I/O originates. This tuning is typically the job of the DBA, and is not in the scope of this paper. A great reference book that covers this tuning in detail is Oracle Database 10G RMAN Backup & Recovery by Matthew Hart and Robert Freeman.

Physical bottleneck The system in question here would be the Avamar backup client system that hosts the Oracle database. It could be a physical or virtual machine with various hardware components. Every system built will have a slowest component that translates into a bottleneck. The goal is to identify the bottleneck to improve the efficiency of the backup and recovery jobs. Often, once you have addressed one bottleneck, another bottleneck will quickly surface. The process is outlined in the following sections. Table 1 shows some examples of physical system bottlenecks. Your backup and recoveries will go only as fast as the system bottleneck. Every system will have a physical bottleneck.


Table 1. Potential physical system bottlenecks

Operation type Physical component Backup Memory Backup CPU Backup I/O Subsystem Backup Network (First Backup) Restores Network Restores I/O Subsystem Avamar will process Oracle backups as fast as the memory and CPU will allow. Be aware that Avamars initial first backup will appear to utilize more network and server resources as the effect of the deduplication will not be fully realized. Subsequent backups will process faster and utilize less resources as the local cache files and data store are populated. During backups, the I/O subsystem read performance may be the bottleneck and will need to provide exceptional read and seek performance. EMCs experience at customer sites is that database backup performance ranges from approximately 60 to 100 GB/hour per backup stream. Performance may even be in excess of 200 GB/hour per stream. EMC has, in numerous cases, observed a linear increase in backup performance as multiple parallel backup streams are launched to back up the data. Allocating multiple RMAN channels to boost the overall backup performance is definitely worth exploring under the following conditions: If the I/O subsystem is well architected and can support multiple concurrent backup streams that are

each capable of streaming backup data in excess of 100 GB per hour.

NOTE: "Well architected" in this case means that backing up the data through multiple streams should not create contention in the I/O subsystem and therefore, slow down, rather than improve, the overall backup performance.

If the host has multiple processors available during the backup window. By default, the Avamar client

will distribute the SHA-1 hashing and compression out to all the processors on the client. However, the main avtar thread is mapped to a single processor, and on some clients with slower individual processors, this could be the bottleneck. In this case, allocating multiple RMAN channels creates multiple avtar processes, each of which will have a main thread that is mapped to a different processor, thus boosting overall performance.

As described in the next section "Avamar cache sizing for databases, the maximum amount of

memory that will be consumed by the Avamar process to optimally perform the Avamar backups of the database data will be approximately 0.001 (1/1,000) of the total size of the database. For example, to back up a 5 TB Oracle database, the maximum amount of memory consumed will be approximately 5 GB total for all concurrent streams.

Although the examples later are based on using four RMAN channels to back up the Oracle database data, the optimum number of required RMAN channels needs to be determined based on the following factors: The optimum number of separate backup streams that can be supported by the I/O subsystem before

excessive contention slows down performance. The CPU utilization will tend to continue to improve as more channels are added, until the number of

channels equals the number of processors.


As seen in the earlier formula, the amount of memory that needs to be allocated for the backup process

is independent of the number of channels that are used to perform the backup. For reasons that will become apparent in the next section, enough RMAN channels will need to be

allocated so that the average amount of data per channel does not exceed 1.5 TB. Based on current lab testing, the Avamar/Oracle recovery should go as fast as the network will allow. This pertains to the client-side network interface, which in the testing to date has been a single GigE interface. A second client-side network interface should help reduce this bottleneck.

Avamar cache sizing for databases For more background information about the client cache files, please refer to the EMC Avamar Operational Best Practices manual.

The hash cache file will allow you to tune your backups to run faster. They do not have any effect on restores (p_cache.dat).

For database backups, having properly sized hash cache files (p_cache.dat files) is extremely important. The hash cache stores the hashes of the chunks and composites that have been sent to the Avamar server. The hash cache is used to quickly identify which chunks or composites have previously been backed up to the Avamar server.

It is very important that the hash caches be sized appropriately. If the hash cache is not large enough, the overall performance of the backups will slow down. This is because if the avtar process finds that a hash of a chunk that is not contained in the hash cache, it queries the Avamar server for the presence of the hash. These "lookups" of the hashes on the Avamar Grid require random seeks, which detract from the performance of the Avamar Grid.

Conversely, if the hash caches are oversized, then the hash caches, which are loaded into the client's memory at the beginning of the backup, will consume too much memory. This, in turn, could cause swapping on the client, which slows down the backup performance, or could even cause the backups to fail because of memory allocation failures.

NOTE: Properly sizing the hash caches is particularly important when backing up Oracle databases using multiple RMAN channels. If the hash caches are undersized, then the combined effect of multiple avtar processes querying the Avamar Grid for the presence of hashes of chunks that were previously backed up could cause the Avamar Grid to become the bottleneck. This slows down overall performance. On the other hand, if we over allocate memory for the hash caches, then the client could run out of memory resources to support all concurrent avtar streams.

Rules for sizing hash caches properly There are a few rules that you need to know to size the hash caches appropriately:

1) By default, the hash cache could consume up to 1/16th of the physical RAM on the Avamar client.

2) If the hash cache files already exist and are larger than the required size, the cache files (p_cache.dat) will need to be deleted and re-created to be resized to a smaller size.

3) The hash cache doubles in size each time it needs to grow. The current hash cache sizes are 24 MB, 48 MB, 96 MB, 192 MB, 384 MB, 768 MB, and 1,536 MB. At this time, the maximum hash cache size that is supported is 1,536 MB.

4) The hash cache includes only one 20-byte SHA-1 hash per chunk, which is the hash of the contents of the chunk, plus another 4 bytes of overhead per chunk. Therefore, each entry in the hash cache consumes 24 bytes per entry.

5) During Oracle backups, the average chunk size (before compression) is 24 KB per chunk.

6) To control the size of the hash cache used for a dataset, you can specify the "--hashcachemax=" option. The value to which you set this attribute is the "not to exceed" size of the hash cache file in


MBs. Therefore, for example, setting "--hashcachemax=500" will allow the hash cache file to grow to 384 MB (because 384 MB is the largest hash cache file less than 500 MB).

From rules (3), (4), (5), and (6), we can derive the following rule for setting the "hashcachemax" option:

--hashcachemax = 2 x MB So if the database is 700 GB the formula should look like this.

Hash Cache Max should be less than 2 x 700 MB (Converted GB to MB) = 1400 MB

Examples of properly sized hash cache files Here are some examples of how to properly specify the "hashcachemax" option and the resulting size of the hash cache: Table 2. Sample hash cache sizes based on stream size and hashcachemax

Stream size Hashcachemax Hash cache size

10 GB DB Stream --hashcachemax=20 24 MB 100 GB DB Stream --hashcachemax= 200 192 MB 500 GB DB Stream --hashcachemax= 1000 768 MB 700 GB DB Stream --hashcachemax=1400 768 MB 1000 GB DB Stream --hashcachemax=2000 1536 MB 1536 GB DB Stream --hashcachemax=3072 1536 MB So you can see as we get to the larger dataset size the maximum hash cache will be likely 1,536, which is the largest it can possibly get today. A dataset over 2 TB that uses a single hash cache will likely slow down because it does not have a large enough hash cache. This is where the multiple stream discussion becomes relevant.

You can specify the "--hashcachemax=" parameter in the my-avtar-flags file.

You will have to rename the original p_cache.dat file and your next backup will build a new one.

When you allocate multiple RMAN channels to back up the Oracle database, each channel will process roughly the same amount of data. This means that you can specify the same "--hashcachemax=" in the my-avtar-flags.txt file.

When backing up large databases (greater than 2 TB databases), it might be important to minimize the amount of memory consumed on the host. In this case, the amount of data backed up per RMAN channel should be ~750 GB or ~1.5 TB. This ensures that we can effectively utilize a 768 MB or 1,536 MB hash cache file with very little wasted space in the hash cache file.

It is also important to understand that if your cache files are close to these values and not exactly 100 percent the same, you should be safe. You should be aware of major deviations from the rules.

The case for multiple RMAN channels When the system bottleneck has been identified, you can test a single channel RMAN script for the backup and recovery of the database. It appears that a single RMAN stream will run around 60-100 GB per hour. This may be adequate to satisfy the backup and recovery time objectives for your database.

The issue here is that the larger databases do require the backup and restores to run at faster speeds to satisfy the backup and recovery time objectives.

For example a 2 TB database will require at least 21 hours to restore if using a single stream. In the same context a 4 TB database would require at least 42 hours.


This leads to the case for allocating multiple channels, which is equivalent to generating multiple streams or multiple Avtar processes. This is definitely needed when it comes to a restore of a larger database.

Example scripts and multiple Avamar cache files When architecting an Oracle backup and recovery that will use multiple RMAN channels in Avamar, use multiple sets of Avamar cache files. This is similar to the way multiple cache files are invoked for multiple Avtar processes for very dense file system backups. As a matter of fact, we will allocate multiple Avtar processes in the end. This is simplified by using the Avamar "- - cacheprefix" flag. This is an actual Avtar flag. Read more about the "--cacheprefix" flag in EMC Avamar Operational Best Practices.

The good news here is by using this option there is not a lot of heavy lifting or changes that need to be made to allow for multiple cache files.

The actual cache files sit in the /usr/local/avamar/var directory. Of course, the cache files are f_cache and p_cache. To be more specific the cache files actually sit in the /var/avamar directory. They are referenced by a symbolic link in the /usr/local/avamar directory.

Multiple channel RMAN Avamar backup script run {

configure controlfile autobackup on; set controlfile autobackup format for device type sbt to "CONTROLFILE.%F"; allocate channel c1 type sbt PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so ENV=(PATH=/bin:/usr/local/avamar/bin)"; allocate channel c2 type sbt PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so ENV=(PATH=/bin:/usr/local/avamar/bin)"; allocate channel c3 type sbt PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so ENV=(PATH=/bin:/usr/local/avamar/bin)"; allocate channel c4 type sbt PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so ENV=(PATH=/bin:/usr/local/avamar/bin)"; set controlfile autobackup format for device type sbt to "CONTROLFILE.%F"; send channel='c1' '"--flagfile=/scripts/my-avtar-flags.txt" "--cacheprefix=c1" "--logfile=/usr/local/avamar/var/c1_avoracle.log"'; send channel='c2' '"--flagfile=/scripts/my-avtar-flags.txt" "--cacheprefix=c2" "--logfile=/usr/local/avamar/var/c2_avoracle.log"'; send channel='c3' '"--flagfile=/scripts/my-avtar-flags.txt" "--cacheprefix=c3" "--logfile=/usr/local/avamar/var/c3_avoracle.log"'; send channel='c4' '"--flagfile=/scripts/my-avtar-flags.txt" "--cacheprefix=c4" "--logfile=/usr/local/avamar/var/c4_avoracle.log"'; backup database plus archivelog; release channel c1; release channel c2; release channel c3; release channel c4;

The blue text outlines the unique parts of this script. This is essentially how to address the multiple channels and map them to the multiple cache files. The cacheprefix=c(n) creates a unique set of cache files for that channel. There are four in this example.

The logfile= creates a unique set of log files for each channel.

All of these files are stored in the standard /usr/local/avamar/var directory.


Multiple channel RMAN restore script

run {

configure controlfile autobackup on; set controlfile autobackup format for device type sbt to "CONTROLFILE.%F"; allocate channel c1 type sbt PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so ENV=(PATH=/bin:/usr/local/avamar/bin)"; allocate channel c2 type sbt PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so ENV=(PATH=/bin:/usr/local/avamar/bin)"; allocate channel c3 type sbt PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so ENV=(PATH=/bin:/usr/local/avamar/bin)"; allocate channel c4 type sbt PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar64.so ENV=(PATH=/bin:/usr/local/avamar/bin)"; set controlfile autobackup format for device type sbt to "CONTROLFILE.%F"; send channel='c1' '"--flagfile=/scripts/my-avtar-flags.txt" "--logfile=/usr/local/avamar/var/c1_avoracle.log"'; send channel='c2' '"--flagfile=/scripts/my-avtar-flags.txt" "--logfile=/usr/local/avamar/var/c2_avoracle.log"'; send channel='c3' '"--flagfile=/scripts/my-avtar-flags.txt" "--logfile=/usr/local/avamar/var/c3_avoracle.log"'; send channel='c4' '"--flagfile=/scripts/my-avtar-flags.txt" "--logfile=/usr/local/avamar/var/c4_avoracle.log"'; restore database; recover database; release channel c1; release channel c2; release channel c3; release channel c4; }

This script is used for the restore and looks very much like the backup script. Always make sure the DBA is involved with the restore and recovery of an Oracle database. There are many restore scenarios and the DBA will need to be aware of the state of the database before you begin.

The main concept to gather from this example is that the recovery will come back in multiple streams. This is just how the backup was performed.

Multiple channel Avamar My-Avtar-Flags file --hashcachemax= --expires=30 --server=avamar220.lss.emc.com --pidname=Oracle --pidnum=1002 --logfile=/usr/local/avamar/var/avtar.log --vardir=/usr/local/avamar/var --id=backup --ap=backup --path=/Oracle/avamar161.lss.emc.com This looks similar to earlier examples. The hashcachemax flag was added to accommodate the large database files. The rest is standard.

By using the cache prefix option, only one my-avtar-flags file is needed. Each channel reads this file.


Sample outputs

Figure 1. Prebackup

Figure 2. Start backup

Figure 3. Backup output multiple channels


Figure 4. Check for multiple avtars The first channel completes fast because it handles the archived redo logs and the control file backups. Figure 5 shows how to see the progress in the Avamar GUI.

Figure 5. Avamar GUI


Conclusion As can be seen by examples in this white paper, Avamar can be used as a viable solution for larger database environments. By using multiple RMAN channels you can easily achieve respectable backup and recovery time objectives.


Appendix A: Avamar Oracle plug-in recovery catalog creation

Create the catalog database instance in Oracle 9i RMAN tracks backed up initialization files, control files, datafiles, and logs using the RMAN repository. The RMAN repository facilitates automated identification of files required for a given backup or restore operation.

The RMAN repository can be written to the target database control file by running RMAN in no catalog mode or to a separate database instance known as the catalog database. A catalog database is recommended for the RMAN repository if you are backing up multiple database instances or if a single database instance has more than 20 datafiles.

If you choose to run RMAN in no catalog mode, it is critical that the control file is backed up frequently and regularly since the backup repository is in the control file. If the control file is lost or damaged, it must be recovered from a non-RMAN backup before a recovery of the database is possible. The use of RMAN in no catalog mode is not shown in the examples. This, of course, changes in Oracle 9i with the control file Autobackup option.

If you choose to run RMAN with a backup repository in a separate database instance, it is recommended that the catalog database be on a separate server from any target database that is registered with that catalog database. The examples provided use a target database called eval and a catalog database called rman.

Create a Oracle database in Oracle 9i 1. Log in to the Oracle server as oracle.

SunOS 5.8 login: oracle Password: Last login: Wed Jan 21 08:06:31 from legato1 Sun Microsystems Inc. SunOS 5.8 Generic February 2000 Test1% 2. Start the Oracle Database Configuration Assistant.

oracle@test1% setenv ORACLE_SID rman oracle@test1% /space/home/oracle/product/9.2.0/bin/dbca & 3. Select Next on the Welcome screen.

4. Select Create a database and click Next.

5. Select General Purpose on the Database Templates screen and click Next.

6. Enter the Database Identification information and click Finish. Example:

Global Database Name: rman.legato.com SID: rman 7. Confirm the database creation by clicking OK on the Summary screen.


8. Enter password information on the Oracle Database Configuration Assistant screen and click Exit.

Example: SYS password: backup Confirm SYS password: backup SYSTEM password: backup Confirm SYSTEM password: backup

Create catalog users

Create the catalog schema Create a catalog database user A catalog database user for use by the RMAN facility must be created in the catalog database instance. The user in this example is called backup. This user will require connect, resource, and recovery_catalog_owner privileges. It is also recommended to grant this user sysdba privileges. See the Create users in the recovery catalog section for examples. Create the recovery catalog Connect as the new user to the recovery catalog database and populate the recovery catalog tablespace.

Oracle 9.x oracle@test1% rman target backup/backup@eval catalog backup/backup@rman

Recovery Manager: Release 9.2.0.1.0 - Production Copyright (c) 1995, 2002, Oracle Corporation. All rights reserved. connected to target database: EVAL (DBID=1928152189) connected to recovery catalog database RMAN> create catalog recovery catalog created RMAN> exit Recovery Manager complete.

Register the target database A target database must be registered with the catalog database before RMAN can perform any backup operations for the target. Execute the following commands to register the target database: oracle@legato1% rman target backup/backup@eval catalog backup/backup@rman Recovery Manager: Release 9.2.0.1.0 - Production Copyright (c) 1995, 2002, Oracle Corporation. All rights reserved. connected to target database: EVAL (DBID=1928152189) connected to recovery catalog database RMAN> register database; database registered in recovery catalog starting full resync of recovery catalog full resync complete RMAN> exit Recovery Manager complete. oracle@test1%

RMAN> resync catalog;

If you have not executed the register database command prior to executing the backup, restore, or resync catalog commands you will get the following error:

RMAN-20001: target database not found in recovery catalog


Appendix B: Avamar Oracle plug-in backup and recovery scripts This section shows examples of scripts used for various backup and recovery scenarios.

Back up all and delete the logs connect target backup/backup@evaldb connect catalog backup/backup@rcvcat

run { allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; backup database plus archivelog delete input; release channel T1; } Crosscheck archive logs connect target backup/backup@evaldb connect catalog backup/backup@rcvcat

run { allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; crosscheck archivelog all ; release channel T1; } Back up the datafile connect target backup/backup@evaldb connect catalog backup/backup@rcvcat

{ allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; backup datafile '/d1/dev3data/usr01.dbf' ; release channel T1; } Recover the datafile connect target backup/backup@evaldb connect catalog backup/backup@rcvcat run { allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; restore datafile '/d1/dev3data/usr01.dbf' ;


recover datafile '/d1/dev3data/usr01.dbf'; release channel T1; } Back up the tablespace connect target backup/backup@evaldb connect catalog backup/backup@rcvcat

run {

allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';

backup tablespace DEMO; release channel T1;

} Recover the tablespace connect target backup/backup@evaldb connect catalog backup/backup@rcvcat

run {

allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)";

send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; restore tablespace DEMO; recover tablespace DEMO; sql "alter tablespace DEMO online";

release channel T1; } Restore archive log all run { allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; restore archivelog all ; release channel T1; } Disaster recovery using Autobacked up control files connect target backup/backup@evaldb connect catalog backup/backup@rcvcat

set DBID = 92755261 startup force nomount;


run {


#restore spfile; restore spfile from autobackup; release channel T1;

} shutdown immediate; startup nomount;

Disaster recovery restoring a control file

run {


#restore controlfile; restore controlfile from autobackup; alter database mount; restore database; recover database; release channel T1; } Simpler DR example of incomplete recovery connect target backup/backup@evaldb connect catalog backup/backup@rcvcat

connect target /

set DBID = 92755261 startup force nomount;

run { allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"';

#restore controlfile; restore controlfile from autobackup; alter database mount; recover database; alter database open resetlogs; } Perform cold backup using RMAN connect target backup/backup@evaldb connect catalog backup/backup@rcvcat

run { shutdown immediate; startup mount;


allocate channel T1 type 'SBT_TAPE' PARMS="SBT_LIBRARY=/usr/local/avamar/lib/libobk_avamar.so,ENV=(PATH=/bin:/usr/bin)"; send '"--flagfile=/usr/local/avamar/lib/my-avtar-flags.txt"'; backup (DATABASE format 'df_t%t_s%s_s%p'); release channel t1; sql Alter Database Open;

} Oracle on Windows example connect target backup/backup@evaldb connect catalog backup/backup@rcvcat

run { configure controlfile autobackup on; allocate channel c1 type sbt PARMS="SBT_LIBRARY=C:\PROGRA~1\avs\bin\orasbt.dll,ENV=(PATH=C:\PROGRA~1\avs\bin)" format '%d_%U'; set controlfile autobackup format for device type sbt to "CONTROLFILE.DUBLIN.%F"; send '"--flagfile=C:\TEMP\FlagFile.txt" --debug'; backup database plus archivelog delete input; } Notice the short pathname for SBT_LIBRARY=C:\PROGRA~1\avs\bin\orasbt.dl

Flagfile located at C:\TEMP\FlagFile.txt: --debug --pidname=Oracle --pidnum=3002 --logfile=C:\TEMP\avtar.log --vardir="C:\Program Files\avs\var" --id=root --ap=8RttoTriz --path=/clients/vm-qasys-02


Appendix C: Avamar Administrator Oracle GUI backups

Snapup options The options in defined Figure 6 can be specified during on-demand Oracle database backup operations, or within the datasets. They are explained below.

Figure 6. Options for on-demand Oracle database backup

Oracle Instance Name specifies the Oracle server name. This field can normally be left blank because the Oracle instance name is automatically stored when the target client is browsed during on-demand snapup or when a dataset is applied to a member of a group during a scheduled snapup.

User name specifies the username used to authenticate with the Oracle database. If this field is left blank, RMAN tries to log in with the same username and password that the Avamar client agent is running under, and attempts to assume SYSDBA privileges. Typically, this field should contain the special account name (backupuser) manually created as part of database preparation. Username and Password comprise a connection string to Oracle. The connection string must specify a user who has backup privileges for the database.

Password specifies the password for the username account. The Delete Archive Logs after backup checkbox, if selected, automatically deletes Oracle archive

logs immediately following a successful database backup. Enhanced Commonality Factoring enables or disables enhanced data deduplication. During snapups,

enhanced data deduplication typically reduces the amount of client data that must be sent to the server but requires additional client CPU resources. Choices are: Disabled Default (use the global enhanced data deduplication setting already set on the server) Enabled

The Enable debugging messages checkbox, if selected, adds all available debugging output to both avtar and avoracle. This is useful when submitting a problem to EMC Technical Support so that an issue can be more easily diagnosed.

The Use Recovery Catalog checkbox is followed by three recovery catalog fields. The Recovery Catalog Server Name, Recovery Catalog Username and Recovery Catalog Password settings are used to specify a Recovery Catalog Server connection string for RMAN. By using a Recovery Catalog Server, many specialized features of RMAN may be used. A thorough discussion of these features is beyond the scope of this paper.


Recovery Catalog Server Name specifies the recovery catalog server name. Recovery Catalog Username the specifies the recovery catalog username. Recovery Catalog password specifies the recovery catalog password.

Restore options The options in Figure 7 can be specified during the restore operations. They are explained below.

Figure 7. Options for restore operations

Oracle Instance Name specifies the Oracle server name. This field can normally be left blank because the Oracle instance name is automatically stored when the target client is browsed during restore operations

User name specifies the username used to authenticate with the Oracle database. If the field is left blank, RMAN tries to log in with the same username and password the Avamar client agent is running under, and attempts to assume SYSDBA privileges. Typically, this field should contain the special account name (backupuser) manually created as part of database preparation. Username and Password comprise a connection string to Oracle. The connection string must specify a user that has backup privileges for the database.

Password specifies the password for the username account. Recovery Until SCN is for the recovery portion, so you can play back archive log files until the

transaction with this SCN is reached. If this field is left blank, avoracle will scan through the known backups in the backup catalog (either in the system control file or the recovery catalog) and find the highest SCN associated with a backup file.

The Enable debugging messages checkbox, if selected, adds all available debugging output to both avtar and avoracle. This is useful when submitting a problem to EMC Technical Support so that an issue can be more easily diagnosed.

The Only restore control file checkbox, if selected, restores only the control file, and known backups in the backup catalog (either in the system control file or the recovery catalog) are listed. This is useful when the user wants to find an SCN for a database or archive log without actually performing the restore.


Restore Command Options is for additional command line options that will be passed to the RMAN

restore command. Recover Command Options is for additional command line options that will be passed to the RMAN

recover command. For descriptions of Use Recovery Catalog, Recovery Catalog Server Name, Recovery Catalog

Username, and Recovery Catalog Password, see the Snapup options section.

Troubleshooting GUI operations If you are encountering issues with browsing your database(s) within the GUI, this will help you identify what the issue is. Avoracle looks at the oratab file for database names and then cross-references those names to the tnsnames file. Local databases that resolve this way are then presented in the browser window for selection for backups. Avoracle defaults to the $ORACLE_HOME/network/admin directory. So, make sure the tnsnames file is in the $ORACLE_HOME/network/admin directory. If these instructions do not resolve the browse issue, please do the following:

1. Create the file /opt/AVMRclnt/var/avoracle.cmd

2. Add the following two lines to the file: --debug --logfile=/opt/AVMRclnt/var/avoracle_browse.log

3. Try to browse again. The browse attempt will be logged in avoracle_browse.log.

4. Look in the avoracle_browse.log file and you will be able to identify the error you have and take the necessary steps to correct it.

Appendix D: Avamar and Oracle theory of operations

Avoracle theory of operations Avoracle is normally invoked whenever any operations on the Oracle database are carried out through the Avamar Administrator. The three operations supported by Avoracle are: Browse Backups Restores While the functionality of Backup and Restore should be fairly obvious, Browse is an intermediate and not so visible operation. When an Avamar Oracle operation is selected in Avamar Administrator, it sends a message to the concerned client node. This message is called a workorder, and it contains all the information a client needs to perform an operation. Communication between the Avamar Administrator and the client node is always initiated through Avagent. Avagent runs as a daemon (service) and listens for messages from the Administrator.


Avagent inspects the workorder received, and uses the information to decide which of the Avamar clients needs to be invoked. If the operation is meant for the Oracle plug-in, Avagent starts Avoracle and passes it the workorder.

Avoracle operations

Browse The first step in starting a backup of an Oracle database is the Browse operation. When the Oracle plug-in is selected in the Backup panel of Avamar Administrator, it sends a Browse request to Avoracle (through Avagent). This triggers a database discovery process in Avoracle. Avoracle reads the /etc/oratab to get a list of databases on the system and their home directories. Using the Oracle home directories it then looks for the tnsnames.ora file. The special fields of interest in tnsnames.ora are HOST and SERVICE. Using this, Avoracle generates a list of databases that are local to the system. This list is then returned to Avamar Administrator in a message. Avamar Administrator then uses this list to populate the Backup panel.

Backup When one or more databases are selected for backup, a new message is sent to Avoracle. This message contains the list of databases to be backed up, together with any options that the user specifies. The message also contains credentials that Avoracle can use to communicate with the Avamar Server. In the current implementation Avoracle can only do backups and restores sequentially. So, even if multiple databases have been specified, they are done one by one. The following description is for one database. For multiple databases the sequence is just repeated. Avoracle uses RMAN and Oracles SBT feature for taking backups. For this Avoracle specifies a loadable library that can communicate with the Avamar Server. This library implements certain entry points that have been written to Oracles specification. The library path and some other parameters are communicated to RMAN through a script that Avoracle creates. In Oracle 10g and later, Oracle has a restriction that prohibits RMAN from running as root. Avoracle scripts are run as the default user oracle unless an alternate user id has been specified. The RMAN script, credentials, and logfiles are created and stored in the Avamar var directory. Then avoracle creates another process and starts RMAN, telling it what script to use. Avoracle then goes into a loop waiting for a connection from avtar (more on this later) or for RMAN to complete its operations and exit. Once RMAN starts, it communicates with Oracle and tells it which library needs to be loaded. Oracle loads this library (which is actually an Avoracle component) and uses it for backup. After this point the backup process is controlled by RMAN. RMAN, by default, backs up archive logs, the database file, and the control file. Based on certain factors RMAN splits up the entire backup into one of more backup pieces. For each backup piece (which is actually opaque to Avoracle), RMAN begins by communicating the piece name to libobk. It then performs the write using multiple calls to libobk, ending the operation with a close command. When libobk receives the backup command it creates an avtar process. The name of the backup piece is communicated to avtar together with some other information.


Avtar is also told how to connect with the already running Avoracle. This last step is required so that avtar can get the wor

h6540 oracle backup recovery perform avamar rman wp

Documents

emc avamar oracle plug

oracle datafile

oracle tablespace

large oracle databases

oracle parameter file

oracle database objects

oracle control files

emc software