emc avamar integration with the ibm lotus domino server · pdf fileavamar® backup and...

$: EMC Avamar Integration with the IBM Lotus Domino Server · PDF fileAvamar® backup and recovery software leverages its IBM Lotus Domino server ... \Program Files\IBM ... namely whether$
White Paper

Abstract

This white paper provides a technical overview of how EMC® Avamar® backup and recovery software leverages its IBM Lotus Domino server and Notes client. This includes an in-depth look at how restores differ from point-in-time recoveries and other robust backup functionality provided to Notes administrators and users. November 2010

EMC AVAMAR INTEGRATION WITH THE IBM LOTUS DOMINO SERVER

2 EMC Avamar Integration with the IBM Lotus Domino Server

Copyright © 2010 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate of its publication date. The information is subject to change without notice. The information in this publication is provided “as is”. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. Part Number h8108


Table of Contents

Executive summary.................................................................................................. 4

Introduction ............................................................................................................ 4

Audience ............................................................................................................................ 4

Terminology ....................................................................................................................... 4

EMC Avamar Notes agent fundamentals ................................................................... 6

Restore vs. recovery............................................................................................................ 6

Speeds and feeds ................................................................................................... 6

Backup speeds ................................................................................................................... 7

Restore speeds ................................................................................................................... 7

Point-in-time recovery speeds ............................................................................................ 7

Deduplication rates ............................................................................................................ 7

Scheduling .............................................................................................................. 7

Daily operations ...................................................................................................... 8

Recoveries .......................................................................................................................... 8

Restores ............................................................................................................................. 8

Restores from an Avamar /REPLICATE domain ................................................................ 9

Critical information ................................................................................................ 10

File handling and locations............................................................................................... 10

Notes file filtering ......................................................................................................... 10

Non-default notes.ini file location ................................................................................ 11

Point-in-time recoveries .................................................................................................... 11

Notes.ini file location is not needed for point-in-time recoveries .................................. 11

Point-in-time recoveries to other Domino servers is not supported ............................... 11

Troubleshooting .................................................................................................... 11

Finding the notes.ini file ................................................................................................... 12

Determining the Domino version number and DAOS status .............................................. 12

Determining the transaction log type and its location ....................................................... 13

Best practices for Lotus Domino ............................................................................. 14

Cold backups and other alternatives ...................................................................... 15

Conclusion ............................................................................................................ 17


Executive summary EMC® Avamar® released its Lotus Domino server and Notes 8.5 client in May 2010.

This white paper provides guidance to the questions that may arise during implementation, giving EMC’s Backup and Recovery Systems (BRS) technical consultants (TC) and prospective Domino administrators the ability to ensure the Domino server environment is configured correctly to leverage new Avamar features and creating common ground to discuss the environment. This paper begins with checking prerequisites that determine which backup and recovery options are possible. These prerequisites include how to check for the notes.ini path, whether the correct type of Notes transaction logging is enabled, and if Direct Attached Object Storage (DAOS) has been implemented. The balance of the paper covers the features and benefits between restores and recoveries, particularly point-in-time recoveries (PITR) leveraging transaction log replay.

With these prerequisites and backup/recovery options understood, the rest of the paper explains how the Avamar Notes client achieves typical day-over-day database commonality around 98 percent, leveraging a set of robust application-specific choices for restores, recoveries, or both, depending on how the Domino environment is initially set up and the customer’s objectives and future plans.

Introduction This white paper provides a technical overview of the EMC Avamar Lotus Notes Backup Agent, including how to use it for both restores and conducting point-in-time recoveries from both a primary Avamar Grid and a remote Avamar target. This white paper is not intended to replace any of the Avamar Lotus Domino client documentation (such as the User Guide), but it is a summation of findings for usability, tech tips, and troubleshooting experience gained since the last publication of this documentation. This white paper does not focus on the Avamar AIX 5.3 support for RS/6000 clients.

Audience

This white paper is intended for backup administrators or technical staff seeking a more in-depth review of how EMC Avamar's deduplication technology is applied to Lotus Notes and Lotus Domino data.

Terminology

The following terms are used throughout this document:

Archival logging — The most common form of logging for larger enterprises, archival logging ensures that point-in-time recoveries can be achieved by replay of logs forward to a user-specified date/time. Archival logging is essential for point-in-time recovery since all other forms of Notes logging (circular and linear) result


in truncated logs and thus leave no logs for Avamar to replay forward to a user-defined point in time.

.BOX file — A file extension denoting an outgoing mailbox file. Most Notes backups use the .NSF database file extension, but .BOX and .DIR file extensions are the other two main data or metadata file types.

Compaction — A Domino server utility to save space on database files. More importantly for backup and recovery, there are compaction operations that can preserve or change the DBIID (see next definition) of the original database file signature. Use of these by the Notes administrator can impact point-in-time recovery capabilities since a DBIID change without its transaction log companion files could negatively impact a recovery using transaction logs (a roll forward of the logs beyond the date/time of the original backup). While this actual scenario is beyond the scope of this paper, understand that forced DBIID changes are typically made either via a compaction routine (or the administrator simply adding new messages to a user’s database) to meet the objective of generating incremental data for Avamar to deduplicate.

Database Instance Identification (DBIID) – A change in this ID signals, via the Notes Backup API, to back up this database. Unlike Exchange where all users reside in a monolithic .edb file and accompany logs, each Notes client database resides on a Domino server where each user or resource is assigned its own database name, that is, jsmith.nsf, busytime.nsf, and so on. This database modularity provides distinct benefits for more surgical recoveries either in-place or out-of-place.

Default data directory – For Windows, the default data directory is x:\Program Files\IBM\Lotus\Domino\data where is x is the drive letter the administrator elected to install the data directory to. This directory, or a non-default directory, typically contains the .NSF database files, that is, jsmith.nsf, jjones.nsf, and so on.

.DIR – A database file extension denoting a Lotus Notes directory link. .DIR files contain the full path to a directory located outside of the data directory.

Domino – The server component that provides Notes desktop clients with database files (.NSF, .BOX, and others).

Fixup – A Domino server routine to automatically or manually fix inconsistent databases, but for Avamar purposes, another routine that induces a DBIID change to legitimately trigger an incremental backup and start filling transaction logs with Notes data.

notes.ini file – In its default condition, this text file is located one directory above the default data directory (x:\Program Files\IBM\Lotus\Domino\data) so it is typically found at x:\Program Files\IBM\Lotus. The notes.ini file controls most of the vital functions of the Domino server. This file has some essential information for the field TC, namely whether the Notes administrator set this Domino server up for transaction logging and what directory the transaction log (.TXN) files are


located in. Any procedures resulting in a notes.ini change typically require a reboot/restart of the supported Windows server.

.NSF file - Typical file extension for a Lotus Notes file/database (NotesSystemFile). All users and resources in Notes have distinct filenames, that is, jsmith.nsf, log.nsf, names.nsf, busytime.nsf (including mail.box for outgoing STMP mail), and so on, ensuring greater precision when restoring and recovering database files.

Point-in-time recovery (PITR) – This feature is provided as a result of an Avamar backup using the archival logging transaction type. Using this logging type enables administrators to conduct a point-in-time recovery where log replay occurs from the date/time of the last backup to the specified date/time required for the recovery.

Root-to-root replication (R2R) – This mode of Avamar replication ensures that a copy of the primary Avamar Grid’s root-level domains are replicated to its target grid.

.TXN – This file extension is for a Notes transaction log, uncommitted data contained in a pre-allocated 64 MB file.

EMC Avamar Notes agent fundamentals This section covers the basic concepts and fundamental principles of Avamar’s Lotus Domino agent.

Restore vs. recovery

Restore is retrieving the backed -p dataset from the Avamar server to the original, or an alternate directory, on the same Domino server.

Recovery is the application of transaction logs to a restored database file to bring the selected Notes database back to a user-defined point in time. Most POCs and deployments use the flexibility of PITR for recovering database files.

In fact, most enterprises prefer to conduct a PITR to an alternate directory and then selectively choose the missing item(s) followed by copying them into the live user’s .NSF database (while the file remains open by that user).

Speeds and feeds The Domino Backup API provided by IBM Lotus to certified backup vendors is a single stream network-based API call. This backup API was introduced in 1999. The pledge for a follow-on snaphot-based backup API, such as Exchange has with Volume Shadow Copy (VSS), has not been fulfilled to date. Various back-end architectures such as DB2/WebSphere have been proposed to provide snapshot services, but there is not a single mechanism today providing both snapshot for speed and database consistency.


The Domino Backup API is not dependent on Microsoft services, including VSS. In fact, this API is used across the board on different platforms and operating systems ranging from Windows to AIX and other supported combinations.

Backup speeds

Typical backup speeds observed vary, but average between 50-70 GB/hour. Any variation here is mostly related to the customer’s fragmentation or network throughput to and from the Avamar Grid.

Restore speeds

Restore speeds are typically in the 50 GB/ hour range, with variables such as network throughput and the number of active nodes in the Avamar Grid playing major roles.

Point-in-time recovery speeds

Point-in-time recoveries are about the the same speed as restores, depending on the number of log files that require rolling forward. The roll forward is achieved as the .TXN logs are replayed toward the end of the recovery, against the database file selected from the backup.

In Figure 1, the 64 MB logs in this directory are replayed in ascending order (S0000000.TXN, S0000001.TXN, S0000002.TXN) until the date-time marker is reached.

The recovery order for a point-in-time recovery restores the .NSF database files first followed by a varying number of transaction logs replayed forward as the very last step before the database is recovered to the user-defined point in time.

Deduplication rates

Domino server data adheres to typical day-over-day deduplication or commonality percentages, namely in the 97 percent to 98 percent range. Initial commonality tends to be higher than other databases like Exchange due to the distributed name and resource .NSF file architecture.

Scheduling Typically customers will schedule a full backup starting on Friday with subsequent daily backups for the remainder of the week.

This scheduling provides the dual benefits of speedy incremental backups for six to seven days a week plus ensuring that any planned or inadvertant DBIID changes are captured in full backups. Although not a requirement of the Avamar Domino agent, a full backup has the advantage of reconciling transaction log headers to their parent database files. Thus, a full database backup at least once a week avoids a worst-case scenario of undetected DBIID changes that may have occurred with a compaction or fixup.


Some customers further mitigate against this risk with a more aggressive full backup schedule regimen by conducting at least two full backups per week.

Daily operations

Recoveries

Recovering data to a point in time uses archival transaction logging to replay the .NSF database to a user-defined date/time. If Domino is set up for circular logging, this type of logging does not support point-in-time recoveries.

Figure 1. Domino transaction logs. This path is found in the notes.ini file shown in Figure 6.

See Figure 6 on page 14 for notes.ini’s actual syntax pointing to this directory.

Restores

Restores are a fairly simple operation whether they are in-place, or to an alternate directory. Archival logging is not required to support restores since restores have no point-in-time capability.

While one option is to select Restore everything to its original location, the typical procedure is to Restore everything to a different location where one or more user/resource database files are selected and restored to the user-defined alternate directory.

The Domino administrator then manually adds the objects (emails, calendar entries) back into a “live” copy of the user/resource .NSF database file, thus having zero impact in terms of downtime or any action required by the Notes client or its user during restores. The ease of use in this restore model is that the Notes user is simply able to find, view, and read the restored objects.


Figure 2. Domino backup administrators typically select “Restore everything to a different location” and then manually add the objects into a “live” copy of the user’s .NSF database file. This means that there is zero impact to the user during restores.

Restores from an Avamar /REPLICATE domain

Restores from a /REPLICATE domain on an Avamar target grid configured for root-to-replicate are effective to an alternate directory on the original Domino server. Like the recoveries described previously, the Create New DBIID during recovery checkbox must be selected. However, there is no Enable Transaction Logging checkbox here since transaction log functionality does not apply here.


Figure 3. Restoring from an Avamar /REPLICATE domain requires a new DBIID since the databases are restoring to the same Domino server where they were backed up from.

Critical information This section covers how to look up information critical for backups and recoveries, plus some not so intuitive, but otherwise "working as designed" behaviors of the Avamar Domino agent.

File handling and locations

Notes file filtering

All files contained in the Domino data directory get backed up, not just the .NSF and other database files. If temporary files such as .sh or .tmp files are getting caught in the backup and impacting restores, these can be excluded in the dataset.

One can use the .dir link file type to point to other databases outside of this data directory.


Non-default notes.ini file location

The notes.ini file must be explicitly included in the Domino dataset whether it is installed to the default location or to a non-default path. This default location (shown in Figure 4) is x:/Lotus/Domino where x is the drive letter selected for the notes.ini file.

Non-default notes.ini locations are supported as well, but need to be called out in the dataset as shown in the figure.

Figure 4. The notes.ini file must be included in the dataset. The Domino data directory is also necessary.

Point-in-time recoveries

Notes.ini file location is not needed for point-in-time recoveries

The notes.ini path is not required for either in-place or alternate directory (out-of-place) point-in-time recoveries. As long as the recovery is taking place on the same Domino client, specifying this path is not necessary.

Point-in-time recoveries to other Domino servers is not supported

For point-in-time recoveries to succeed, they must be performed on the same Domino server. This restriction is due to the fact that the DBIID is keyed to that particular Domino server. Export of the DBIID to a foreign host is a disaster recovery operation and is out of scope for this document.

Troubleshooting This section covers various troubleshooting tasks and their procedures. The main focus here is to offer top troubleshooting tips. It is not intended to be a comprehensive summary.


Finding the notes.ini file

The default location for the Domino notes.ini file is:

x:\Lotus\IBM\Domino

This path can vary by version, platform, and administrator's preference.

Determining the Domino version number and DAOS status

Presently, Avamar supports Domino versions 6.5 through 8.5 on Windows 2003 and 2008. In 2009, IBM Lotus released version 8.5, which has a DAOS capability. Although DAOS begin with Domino 8.05, it is fully realized in version 8.5. Enabling DAOS allocates a dedicated .NSF database file for single instance storage since many Notes users share the same Office documents, presentations, and spreadsheets. This DAOS storage path is located in the notes.ini file, which is why this file needs to be included in the Avamar dataset. Avamar parses this .ini file to see if DAOS is enabled and then parses for the path location of this file, typically ending with the file extension .NLO.

The Avamar Domino agent fully leverages the Lotus APIs that support DAOS/.NLO attachment extensions for Domino 8.5 and later. In order to provide DAOS functionality, the Domino administrator must enable archival transaction logging.

To determine the Domino version number, reference the pre-site qualifier or ask the administrator. If you have access to the Domino server console, a “show server” command line (see Figure 5) will show the present version as well.


Figure 5. Determining Domino's version number and whether DAOS is enabled

Determining the transaction log type and its location

To determine whether transaction logging is enabled and where it is located, review the notes.ini file found using the Finding the notes.ini file section for the two main settings.

Under the transaction_logstyle line in Figure 6, if the number is 1, then transaction logs are set to Archival mode. The default is 0, meaning that it is configured for circular logging – the default must be changed to gain PITR functionality.

Once archival transaction logging is enabled, the second requirement is for the Domino/Notes administrator to define where the logs are located. This path is recorded in the notes.ini file as well. The Avamar Notes dataset needs to align with condition of the transaction logging type. With the TRANSLOG_Style variable set to 1 as shown in Figure 6 below (first arrow), Domino point-in-time recoveries are now possible using Avamar. The value 1 set for this variable means that the Domino server is set for archival transaction logging.

The abridged notes.ini below highlights both of these settings, Notice that Domino-specific settings exist, such as whether the transaction logs can use all of the disk space and schedule, console, catalog settings, and so on. These settings are not vital to backup and recoveries and are thus beyond the scope of this paper.


During troubleshooting sessions involving Avamar Support, they may request the prior state of both the log path and transaction logging type as shown in Previous_TRANSLOG_Path and Previous_TRANSLOG_Style just below the second arrow in Figure 6. Best practice is to simply send the entire notes.ini file unless the business unit’s security practices prevent doing so.

Figure 6. notes.ini file showing that transaction logging is enabled and the location of the log directory

Best practices for Lotus Domino If long backups are planned or occurring, consider increasing the timeout setting

from 15 (minutes) to 60.

a. If a database backup (username.nsf, resourcename.nsf, and so on) is going beyond 15 minutes, then it will fail with a backup timeout error message since 15 minutes is the default per database.

b. To address this contingency, modify notes.ini by including a "BACKUP_NO_TIMEOUT=1" line. You can also specify BACKUP_NO_TIMEOUT=60 or other intervals, but a 1 here disables the timeout completely.


c. An authoritative source for enabling this setting is located at:

http://www-01.ibm.com/support/docview.wss?uid=swg21229170

Avoid Domino backups during compaction schedules. Veteran Notes administrators will understand this; Domino administrators standing in for proof-of-concept may not understand the impact of conducting backups during compaction windows.

a. Compaction is an intensive process reclaiming white space within each .NSF database file. As mentioned earlier, some forms of compaction (such as compaction –B) can alter the DBIID of the database. When this occurs, it is the Notes administrator’s responsibility to conduct a full database backup to reconcile database and transaction headers.

b. Knowledgeable Domino backup administrators typically use a non-DBIID-changing compaction method (compaction –c, for example) that does not alter the DBIID.

Since a DBIID-altering compaction method can impact successful recoveries, the BRS field TC can inquire as to what type of compactions occurred since the last full backup. In the end, though, it is the customer’s responsibility to keep the Domino backup out of any compaction or fixup window that could alter DBIIDs.

c. One other process similar to compaction is called fixup. Certain fixup process can alter DBIIDs as well, so for backup and recovery purposes, inquire about both of these processes if you’re in troubleshooting mode.

Cold backups and other alternatives With Avamar’s Domino support spanning version 6.5 through 8.5 running on Windows 2003 and 2008, a number of platforms and operating system with agent support remain. This relatively narrow band of support has led a number of field personnel astray, so please check the version using the methods described above.

With this abbreviated support list, this leads to many asking for workarounds to use Avamar to deduplicate the inherently repetitive data behavior that characterizes Domino databases. IBM introduced DAOS to catch some of this deduplication benefit and make it native to the application. Many Domino administrators knew that version 8.5 would be introducing this support, so there was a burst of migrations from Domino 6.x (end-of-lifed for some time now) and 7.x, bypassing 8.x and 8.1 in favor of going straight to version 8.5, which has a full implementation of DAOS.

The workarounds mentioned here respect the following caveats:

Domino has no official snapshot support today.

Customers must realize that they will not get point-in-time recoveries without a certified platform/operating system combination as found on the IBM Lotus Domino Support Matrix.

http://www-01.ibm.com/support/docview.wss?uid=swg21229170


Second, the backup vendor must support the correct combinations, namely calling and writing to the Lotus C API.

Presently, Avamar support for this API is for Windows 2003 and 2008 up to the latest Domino 8.5 and excluding Domino 6.x and earlier versions.

Many customers implement transaction logging to get a performance boost (~15 percent to 20 percent on average) that logging provides and not just the PITR and DR benefits.

Using a sequestered Notes cluster node, and one dedicated explicitly to the backup process (set such that no users sign on), is one of the best work-arounds for conducting a cold backup and then presenting it to Avamar for backup.

This cold backup would be where all user and resource database files are shut down and thus consistent.

Cold backups are operating system-specific in their implementation, but fairly simple to execute. Typically a cron script executes at a specified recurring cycle to shut down the Domino services for different operating systems. Two sample shutdown commands follow:

“net stop <Lotus Domino server service>” commands for non-supported Windows operating systems

“server stop <Lotus Domino daemon>” for Domino on supported LINUX and UNIX operating systems

With this orderly shutdown all database buffers are flushed and I/O is quiesced. This means a set of consistent database file(s) is presented to Avamar file system backup. With this cold/consistent backup condition, no Domino agent-based backup is needed.

Granularity of restores is the same as that of backups. At the operating system, the granularity of the backup is an .NSF database file, so restore granularity is the same – the entire database file(s).

Without Domino snapshot support, hot backups on EMC Symmetrix® business continuance volumes (BCVs) during a low I/O watermark have reportedly been successful, but such practices are the customer’s responsibility.

Although this type of backup is not officially supported by IBM Lotus, BCVs presented to mount hosts are then chkdsk’ed or fsck'ed for disk integrity before the actual backup takes place.

Hot snapshot backups from BCVs (or other array mirrors) have worked well for sites where they chose, due to risk mitigation, to not implement transaction logging.

Customers conducting hot backups regard archival transaction logging as too risky to provide a consistent database.


While hot snapshot backups gain the speed of a snapshot, they do so at the cost of a point-in-time recovery. As noted earlier, this means that the granularity of the backup dictates the recovery granularity in this scenario. Although an .NSF database backup means an .NSF database restore, the benefit of an administrator opening this file database from an alternate directory to restore the missing objects (email, calendar entries, documents) is that the process works flawlessly.

Conclusion For anyone familiar with Exchange, SQL Server, Oracle or SharePoint, one can see that Lotus Notes has a somewhat simpler architecture that provides backup, restore, and recovery equivalents. Avamar's Domino client supporting version 8.5 provides the ability to tap the deduplication potential inherent in a Domino server’s databases, whether the customer is using single instance DAOS/NLO attachment database files or not. With either method, Avamar will yield the same commonality, but DAOS does reduce the amount of deduplicated data that must go over IP to the Avamar Grid.

emc avamar integration with the ibm lotus domino server · pdf fileavamar® backup and...

Documents