domino database maintenance: best practices › web › cms.nsf › web › openmic_dbmainten... ·...

Domino Database Maintenance: Best Practices

Amy KnoxAdvisory Software Engineer, IBM Notes/Domino L2 Support

2 | © 2015 IBM Corporation

Domino Database Maintenance Best Practices - Agenda

■ General Maintenance Tools and Best Practices■ DBMT:

● DBMT – Summary and Review● DBMT – Deeper Dive● DBMT – Tips and Best Practices

■ Maintaining Databases● All Databases● Mail Databases● System Databases● Custom Databases

■ Additional Resources■ Q&A

*Note: Tips and Best Practices noted in italicized green text throughout the presentation


General Maintenance Tools and Best Practices


Domino General Maintenance Tools and Best Practices

“Traditional” Maintenance Tasks:■ Fixup:

● Many improvements to data validation over the past few major releases● Run it manually when:

● Messages output to the console, indicating that some other process cannot complete as expected due to perceived corruption

● Database symptoms point to corruption that server consistency check didn't resolve● Run fixup on system dbs with server down (only as needed)● TIP: There is no longer a need to schedule regular fixup in 9.x!

■ Compact: ● Ideal to compact most* dbs about once per month (*depending on need/usage)● TIP: Still need to run compact on system dbs with server down● TIP: Regularly Compact dbs that see high volume of document updates

■ Updall: ● Can still be run nightly on all dbs (including system dbs)● TIP: Remove the server ini param (i.e. ServerTasksAtTwo=updall) if using dbmt



9.x Domino Maintenance Features:■ DBMT

● Preferred method for managing Domino db maintenance in 9.x● Easy way to schedule regular compact and updall threads for processing dbs● Has built-in mechanisms that ensure proper mail delivery and failover

■ Compact Replication:● Helps prevent fragmented id table - Makes a db replica, reorders db id table and replaces db

TIP: Run compact -REPLICA on system dbs, with Domino up and running● Why this is a good idea: Minimal downtime -

● All work done elsewhere in memory● Automated restart of server for NSF rename/replace to complete



Other Items that Support Good Database Health:

■ Transaction logging: The current recommendation is to enable/run it:● Helps reduce need for fixup● Saves processing time/resources by deferring db updates to non-peak times● Can simplify/maximize the backup processes (incremental backups vs full)

■ Upgrading: Important to upgrade to latest MRs, fixpacks, and IFs (901FP4/ODS52):● DBMT has some recent fixes; Frequent security-related updates in Notes/Domino/JVM

■ DDM/Event Monitoring: Create probes or event monitors to alert you when certain error messages are detected

■ Domino DB Size: Honor the DB Size hard limit (64 gig)● Enable DAOS on dbs with shared medium/large sized attachments (>=1MB)

■ Defrag: Physical drives approximately once per year (*depends on the system/OS)

● TIP: Defrag before running DBMT the first time■ Backups, Backups, Backups: Not all issues can be prevented

● Backup server AND local databases and important data

http://www-01.ibm.com/support/docview.wss?uid=swg27009309


DBMT


DBMT - Summary and Review

DBMT Fun Facts:■ DBMT was designed with mail files in mind■ DBMT uses copy style compact (compact -c)

● Can affect system performance (affects disk i/o subsystem), but effective on large data sets

■ DBMT is failover ready and has a compact 'anti-kill' switch, with notes.ini parameters■ DBMT is cluster aware

● DBMT reviews CLDBDIR.NSF to ensure a replica is available at all times:● DBMT will never compact the same replica across clusters at the same time

● Exception: Clustermate server is down, DBMT running compact on the only available replica = db possibly unavailable to users at that time

■ Starting with 901FP2, UPDALL executes first when Compact and Updall occur in the same command line

■ Can run multiple instances simultaneously, as long as db list is unique AND machines' hardware can handle the load

■ If using DBMT to compact dbs, no need to schedule any other compacts



■ DBMT Maintenance options include:● Copy-style Compacting● Incremental View Indexing● Full Text Indexing● Folder Reorganizing● Deletion Stub Purging● Expiring of Soft Deleted Entries● Updating of Unread Lists● Fixup of Corruption

■ How to run it:

● Command line (server down)● Program document● Domino console command line (server up and running)


DBMT – Deeper Dive

More on -UpdallThreads and View Indexing via DBMT:■ By default, specific View Indices in mail databases are created, brought up-to-

date and marked “non-discardable” through DBMT● Dramatically improves the end user's fail-over experience

● Larger dbs can have larger views that take longer to build● TIP: Run -updallThreads on clustermate or DR servers regularly, so that in a

failover situation, replicas' view indices are already built and ready to go. ● Otherwise: users could experience long delays opening dbs/views after failover, and

Domino server could see heavy loads, trying to rebuild views. ● Non-discardable views are:

● ($Inbox)● ($Drafts)● ($Sent)● ($All)● ($RepeatLookup)● ($ToDo)● ($Calendar)

● (Haiku_TOC)● ($Alarms)● (iNotes)● ($Users)● (iNotes_Contacts)● ($ThreadsEmbed)


DBMT - Deeper Dive

DBMT, Updall, and Unread Marks: Additional information in TN# 1663952■ By default, DBMT processes the db Unread Table

● Not typically an issue with mail files (1-2 users per db)

● Potentially long updall times, for multi-user, large business applications (Ex: During an upgrade)

● Can be an issue for failover, if a user was on vacation or out of office for an extended duration● Performance effect can be similar to view rebuilds

■ Solution: Upgrade to 901FP3 and run DBMT with the new -nounread option ● Example (command line): >ndbmt indfile4.ind -compactthreads 0 -updallthreads 8 -nounread

● 901FP3 Also includes OPTIONAL new notes.ini parameter (for debugging DBMT/unread marks)

DEBUG_UPDATE_UNREAD_TABLE=1

■ Workarounds:

a.) (Pre 901FP3) Run updall with the “-nounread” flag, outside of DBMT

Example: >nupdall.exe ind1.ind -nounread -nodbmt

*NOTE: When using the -nounread here you must use the -dbmt switch also!

b.) When running dbmt, use updallThreads 0

c.) Request hotfix for SPR# GFUR9N6LD4 for your version of 9.x, in order to use the -nounread flag with DBMT


http://www-10.lotus.com/ldd/fixlist.nsf/SPRNum/E834F75711C0250785257DCC0074AC1F?OpenDocument&ca=drs-fo



DBMT Command line Options:

*Not the same as RAM or “number of processors” of the VM or machine; relates to Redundant Array of Independent (Inexpensive) Disks (RAID). A certain number of disk arms is needed for optimum performance on each processor level.

● Guidance: WINDOWS , AIX , Domino on iSeries (page 26), SAN, Linux

-compactThreads <n> Use <n> threads for compact, where n is between 0 & 100 inclusive● Default is 1 thread. If 0 is specified, no compact operations performed.TIP: Base the number relative to the # of disks arms* backing the data directory, and monitor the system for optimal performance

-updallThreads <m> Use <m> threads for updall, where m is between 0 & 100 inclusive *Default is 1. Base it on the number of disks backing the data directory.

-ftiThreads <g> Use <g> threads for the rebuild of the full text indices, where g is between 1 & 100 inclusive. *Configures number of threads for rebuild of full text indices if corrupt or -ftiNDays specified. Default is 1; 0 not allowed.

BEST PRACTICE: Use default for -ftiThreads, unless specifying -ftiNDays

https://msdn.microsoft.com/en-us/library/dd163558.aspx

http://www.ibm.com/developerworks/aix/tutorials/au-aix-raid/

ftp://public.dhe.ibm.com/systems/i/perfmgmt/V5R2FiSArmct.pdf

http://www-01.ibm.com/support/docview.wss?uid=swg27038065&aid=1

http://www-01.ibm.com/support/knowledgecenter/ssw_i5_54/rzalm/rzalmdynamicallyaddingdisks.htm



DBMT Schedule Options:-force <n> Take the DBs offline for fixup & compact if flagged for force. Where n is the day

of week to do the force (1 == Sunday, 2 == Monday, ...), 0 = “any” day● After 5 failures/10 days, triggers the fixup task when compact does not complete

-compactNdays <p> Compact DBs that have not been compacted in last <p> days

-ftiNdays <f> Rebuild the full text indices every <f> days from creation *Default is to rebuild full text indexes only when found corrupt.

-timeLimit <q> Only run compact for <q> minutes● DBMT process will exit once -timeLimit expires and all active DBMT processing

completes

-range <s> <e> Run DBMT tasks between times <s> and <e> daily. ● Minimum of 10 minutes between start time <s> and end time <e> ● Times must be specified in 12hour format; AM/PM required (i.e.11:50PM).● DBMT process remain loaded but dormant until -range time occurs the next day ● *Compact honors the end time; however, updall and fti operations will complete, as

they run faster and probably not many corrupt FTIs to process

-stoptime <e> Only run compact between now and <e> once● Time must be specified in 12hour format with a required AM/PM (e.g. 6:00AM)● DBMT process will exit once -stoptime is reached and all active DBMT processing

completes

-noCompactLimit Allows compact to run past the end time, but no new compacts will be started.



Recommended Notes.ini Parameters:

*Best Practice: In a clustered environment, use MailFileDisableCompactAbort=1 and MailFileEnableDeliveryFailover=1 together

MailFileDisableCompactAbort=1 New mail arriving in mail.box will wait in current server mail.box, undelivered, while db is being compacted. Router will retry delivery periodically, will not succeed until current compact completes, then mail is finally delivered at next router retry.

● By default (or =0), when new mail is delivered by the mail router to a mail file that is being compacted, the current compact operation will be aborted.

● Note: For large mail files, this may be an unacceptable extended period of time to go without receiving new email

MailFileEnableDeliveryFailover=1 Router feature that allows mail delivery to failover to an available clustermate replica, when the current replica is unavailable.● OOO still works● Mail rules still work with 1 caveat:

● Each Cluster Mate needs all other Cluster Mates specified in “Allowed to use monitors” configuration in it's Server Record → Security Tab.

● $MailClusterFailover item appended to the doc; ● value = Domino Server name where doc delivered



Other Notes.ini Parameters:

*Note: notes.ini setting(s) are restricted to 128 characters

DBMT_MAILTemplate= <customTemplateName>

By default, DBMT looks for dbs built from these templates [StdR7Mail, StdR8Mail, StdR85Mail, StdR9Mail] to ensure that certain critical views are built (i.e. $inbox, $sent, $all, etc.). To ensure that dbs built from custom mail templates receive the same attention, add this parameter and set the value to the template name, as it appears on the Design tab of Application properties. DBMT will recognize it as a mail file and process the same standard views for -updallthreads.● Separate entries with a comma (for multiple custom dbs)Example: DBMT_MAILTemplate=CustomMail90,CustomMail85

DBMT_TemplateName=ViewNameOrAlias1;ViewNameOrAlias2...

Specify views in system dbs that updall must build for that db. ● Once built, these are then added to the NonDiscardableView' list● Separate entries with “;” OK for view names to have spaces.Examples: DBMT_StdNotesLog=MiscellaneousEvents;SecurityEvents; ReplicationEvents;MailRoutingEventsDBMT_StdR9Mail=By Category;CurrentClientDetails

DBMT_COMPACT_FILTER=<list of dbs to exclude from compact>

● Db names are relative to data directory● Separate db names with a comma● TIP: Include Secondary address books in this list● Used mostly for BLACKLIST situations● Same effect as the dbmt_compact_filter.ind file



dbmt_compact_filter.ind file = list of dbs that DBMT compact threads will skip

● Similar effect as the DBMT_COMPACT_FILTER= notes.ini parameter● The Domino server code automatically creates and maintains this file● These db names get added to this list:

● Dbs whose Compact exceeds the specified -timelimit ● Dbs whose Compact take longer than the specified -range and compact

terminated because end of time range was reached● Database(s) < ODS41

● Best Practice: Don't manually edit this file! But you could periodically delete it, if you feel there are dbs in this list that should not be.

■ Idea: If you wish to compact some dbs from the dbmt_compact_filter.ind file:● build a new .ind file● add those dbs to it● remove them from the dbmt_compact_filter.ind file● run dbmt -compactThreads against this new .ind file


DBMT - Deeper Dive

What happens if the DBMT task is interrupted?■ If DBMT compact is interrupted or time limit reached, DBMT is paused. The

next time it runs on these same dbs, it will pick up from where it stopped.● Uses the 'last compact time' to determine which dbs get compacted next

● 'Last compact time' column added to 9.x Domino Admin client● No need for further administrative action● MailFileDisableCompactAbort=1 will prevent this behavior

● If you use this parameter, also use MailFileEnableDeliveryFailover=1● DBs will only be compacted every other day, due to the built-in cluster awareness

● i.e: if compact is interrupted on Tuesday, next compact attempt is Thursday● *NOTE: If compact time limits are consistently insufficient for completion, it is

possible that not all dbs can be compacted as expected! Monitor and adjust as needed

■


DBMT – Tips and Best Practices

■ TIP: When possible, upgrade to 901FP4:● In 8.5x (fixed 9.0): If DAOS-enabled dbs had missing nlos, compact operations would fail. In 9.x, all

compact operations (includes dbmt-induced) process/compact dbs in spite of missing nlos.

● FP2: Updall now runs before compact, when both occur in same program/command line

● FP4: Fix for (low-risk) DBMT crash when processing large dbmt_compact_filter.ind file (SPR# JPAI9VYQ8T)

● FP4: Prevent DBMT from running against daoscat.nsf (SPR#JPAI9WMMG5)

■ TIP: Upgrade databases to ODS 52 as well● Set it using Create_R9_Databases=1 in server notes.ini + a compact -c

● To prevent Rare note corruption issue updating a note in Medium or Strong encrypted databases

■ TIP: Never run 2 instances of DBMT on the same db at the same time● It is acceptable to run multiple DBMT programs simultaneously, as long as the db lists are different!

● Only run one DBMT program on all dbs at any given time

● Can run as many DBMT threads as # of disk arms backing the data directory

■ TIP: Use .ind files for organization of file lists● Easy to manage, easy to edit

http://www-10.lotus.com/ldd/fixlist.nsf/Public/F9C5303D5AFC0B1B85257CFC006E05AE?OpenDocument

http://www-10.lotus.com/ldd/fixlist.nsf/8ed1b46cfdba8957852570c90054623b/ebce9e7615d5f02f85257e5b00005e91?OpenDocument

http://www-10.lotus.com/ldd/fixlist.nsf/8ed1b46cfdba8957852570c90054623b/f8a722002b31d30685257e65006e23de?OpenDocument

http://www-10.lotus.com/ldd/dominowiki.nsf/dx/CREATE_R9_DATABASES


Maintaining Databases


Maintaining Databases - Any Database

Best Practices – Maintaining ANY database: ■ Not a general need to compact a db more than once per week

● TIP: Schedule 20% of dbs/data each weekday night; thus each db processed weekly

● TIP: Run DBMT with “-compactNdays 7” option on Sundays -- catches any dbs not compacted in the last 7 days

● TIP: On server startup: run dbmt -updallthreads N (with -compactThreads 0) on mail files (or any dbs) to ensure that critical views are up to date.

■ Note times of backups and all other server tasks and processes● Schedule dbmt, compact replication, OR compact at times that do NOT

overlap with those other tasks and processes■ If using DBMT, do not also run nightly in-place compacts■ For ease of use for maintenance tasks: Use folder names OR ind files


Maintaining Databases - Mail Databases

■ Recommendation: ● 8.5x: compact weekly/monthly, updall nightly via notes.ini, only run fixup only as needed● 9.x: DBMT regularly on all mail files (compact and updall), only run fixup as needed

■ Sample command lines in DBMT program documents:

*Example 1: Program name = dbmt; Run on day: Saturday.

Start time = 1 AM: Mail1 -compactThreads 4 -updallthreads 0 -compact Ndays 7 -timelimit 360

Start time = 7:15 AM: Mail1 -compactThreads 0 -updallthreads 4

OPTION: Run compact and updall threads in separate instances at separate times, to prevent errors such as “Database is already in use by you or another user”

*Example 2: Run these simultaneously:

Mail1 -compactThreads 2 -updallthreads 0 -range 12:01AM 11:59AM -force 1



12 hours later:

Mail1 -compactThreads 0 -updallthreads 2




Maintaining Databases - System Databases

2 Options in 9.x: Use either Compact Replication OR DBMT to compact system dbs:

● Compact replication runs with server up and running = minimal down time● Both are effective – use whichever is more comfortable and convenient for you

DBMT and System Databases – Tips:■ TIP: For each system database, run a separate thread of dbmt compact; kick off

multiple simultaneous threads, comparable to available resources.■ TIP: Run dbmt compact against system dbs with the server down. Run compact

replication against select system databases with server up and running, and use the -restart operation to complete the rename process.



■ DBMT by default, does not compact System Databases with Domino up and running, but will perform -updall and -fti operations on these:● Reason: DBMT Compact makes the dbs unavailable – these need to be available at all times

admin4.nsfbusytime.nsfcatalog.nsfcldbdir.nsfclubusy.nsfdaoscat.nsfddm.nsfdbdirman.nsfdircat.nsfdomlog.nsfevents4.nsf log.nsflndfr.nsfmtdata\mtsore.nsfnames.nsfstatrep.nsf

*Implications:● If you schedule dbmt for compact and/or

updall for all dbs, the dbmt compact task will NOT process these dbs.

● This also applies to dbmt -compactthreads if server is down.

● Thus, if you wish to compact system dbs, they must be handled separately and intentionally by the Domino admin.

*TIP: Create an .ind file with this list of dbs. During next server down time, run dbmt against this .ind file. **NOTE: Not recommended to compact daoscat.nsf●



NAMES.NSF:■ 8.5x (Tips for all System dbs):

● Run all maintenance with server down:● Run maintenance at least once per month

● Compact -c● Updall -R● Fixup -f (no trans logging) fixup -j (with trans logging) As needed

■ 9.x, Using DBMT:

● If running nightly or weekly updall via dbmt for all dbs, names.nsf IS processed at that time

● Best Practice: Run dbmt processes for names.nsf by itself ● Example: (win OS) Server down:

Ndbmt.exe names.nsf -compactthreads 1 -updallthreads 0 ...

Ndbmt.exe names.nsf -compactthreads 0 -updallthreads 1 ...



LOG.NSF Options:■ Can use either dbmt or compact replication in 9.x■ Can also remove/recreate log.nsf at each server restart

● Using DBMT: ● Run dbmt on server log.nsf by itself, server down, without other system dbs

■ Using Compact Replication (My preferred):● Requires less down time● Experiences 'daily' deletions (based on log= notes.ini parameter), thus at greater risk of

fragmented ID Table



■ LOG.NSF - Sample Output:

After -Restart option:

■

■

■


Maintaining Databases - Custom Databases

■ As with any other db, these should also be regularly compacted■ Determine if the document count and activity level of each custom database

is considered “high”:● Run compact replication against the most frequently used databases with highest doc

counts● Example:


Maintaining Databases - Summary

Summary of 8.5x Database Maintenance:1)Compact should be run against all dbs, except daoscat.nsf, at least once per month.

Daily compacts have been shown to trigger corruption and cause DAOS catalog synch issues.

2)Take the Domino server down to run maintenance on system databases, including updall -R.

3)Fixup task is intended as a solution/repair path, not a preventative measure.

4)Nightly Updall executed via the server notes.ini parameter “ServerTasksAtN=” (Default is 'ServerTasksAt2=').

5) If you receive the error “Entry not found in index” against the ID Vault database (i.e. change user status from inactive/active OR doc deletion), run updall -R (Or ctrl+shift+F9) against the vault db.


Maintaining Databases - Summary

Summary of 9.x Database Maintenance:1)Compact (DBMT preferred) should be run against all dbs at least once per month.

2)Run dbmt on names.nsf and log.nsf independently of any other compact or dbmt operations; run -updallthreads in a separate instance from -compactthreads.

3)Run compact replication (or dbmt) on log.nsf, coordinated with a server restart.

4)Run DBMT against an ind file that includes all other system dbs, EXCEPT daoscat.nsf.

5)Use DBMT to regularly compact mail databases.

6)Use compact replication to maintain dbs that experience “frequent” updates (i.e creation of new docs, edits to existing docs, deletions) and have high doc counts.

7)Use the -force option of DBMT to manage fixup needs. Then, only run fixup if an error is returned OR if all other tasks failed to resolve a database/document corruption issue.

8)Monitor the first few executions of DBMT, and adjust # of -compactthreads as needed, based on the available system resources (disk arms).

9) If using DBMT for updall, remove it from the 'ServerTasksAt2=' notes.ini parameter.

10)Upgrade to 901FP4 and ODS 52 as soon as it is convenient.


Additional Resources


Domino Database Maintenance: Best Practices – Additional Resources

Using DBMT and Compact Replication - Open Mic 2013

Q&A From the Open Mic

Domino Administrator Help – 9.x

John Paganetti's Presentation on Upgrading to Domino 9.x – Best Practices

Administrator Guide for Domino Server maintenance

Wiki article: Best practices for database maintenance

Database Corruption Troubleshooting Guide

How to resolve corruption on a DAOS enabled database

8.x Maintenance tasks

Domino Compact Options

http://www-01.ibm.com/support/docview.wss?uid=swg27039379&aid=1


http://www-12.lotus.com/ldd/doc/domino_notes/9.0/help9_admin.nsf/b3266a3c17f9bb7085256b870069c0a9/5e0ecdb4cff85e9f85257b19005b5854?OpenDocument

http://www-12.lotus.com/ldd/doc/domino_notes/9.0/help9_admin.nsf/b3266a3c17f9bb7085256b870069c0a9/5e0ecdb4cff85e9f85257b19005b5854?OpenDocument

https://maironmendes.files.wordpress.com/2014/02/id104.pdf


http://www-10.lotus.com/ldd/dominowiki.nsf/dx/best-practices-for-database-maintenance

http://www-10.lotus.com/ldd/dominowiki.nsf/dx/Database_Corruption_Troubleshooting_Guide

https://www-304.ibm.com/support/docview.wss?uid=swg21420023

http://www-10.lotus.com/ldd/dominowiki.nsf/dx/maintenance-tasks

https://www-01.ibm.com/support/knowledgecenter/SSKTMJ_9.0.1/admin/tune_compactoptions_r.dita


Q & A


Press *1 on your telephone to ask a question.

Visit our Support Technical Exchange page or our Facebook page for details on future events.

To help shape the future of IBM software, take this quality survey and share your opinion of IBM software used within your organization: https://ibm.biz/BdxqB2

IBM Collaboration Solutions Support page http://www.facebook.com/IBMLotusSupport

IBM Collaboration Solutions Supporthttp://twitter.com/IBM_ICSSupport

http://www.ibm.com/support/entry/portal/scheduled_tech_exchanges/software/lotus/lotus_brand_support_(general)

http://www.facebook.com/IBMLotusSupport

https://ibm.biz/BdxqB2

http://www.facebook.com/IBMLotusSupport

http://twitter.com/IBM_ICSSupport

domino database maintenance: best practices › web › cms.nsf › web › openmic_dbmainten... ·...

Documents