open source data backup, or: how to sleep better at night (oscon 2005)

Open Source Data Backup, or:How To Sleep Better At Night

Fran FabrizioSenior Systems Administrator

Dept. of Computer and Information SciencesUniversity of Alabama at Birmingham

O’Reilly Open Source Convention, August 1-5, 2005

Open Source Data Backup (OSCON 2005), Slide 2

Talk Overview• Introduction to Amanda

– What it is, how it thinks

• Amanda In Action– Real world examples

• Configuring Amanda– Quick look at the config files

• Introduction to Bacula– What it is, how it thinks

• Bacula vs. Amanda– Major differences

• For More Info


Introduction To Amanda

What is Amanda?How does it work?

What is its philosophy?


What is Amanda?

• Cross-Platform• Scalable• Automated• Flexible• Robust

Amanda is the Advanced Maryland Automatic Network Disk Archiver. It has the followingfeatures:


What is Amanda? (Cont)

• A set of CLI utilities written in C• Its own protocols on top of TCP and UDP• Client and server components that partner to

stream and store data for backup and recovery


HoldingDisk

TapeDrive

OS X Client

Solaris Client

Linux Client WindowsClient

Amanda

Amanda Serveramandad

Samba

amandad

amandad

Sample Topology


Typical Sequence of Events

Amanda Server Amanda Client

amdump

planner

dumper(s)

driver

taper

amandad

amandad

launches

passes schedule

spawns

writesto disk

writes totaper

flush to tape

requests estimate

returns estimate

requests backup

returns image

(many clients serviced in parallel)


How Does It Work?

• gtar and dump• Uses standard backup levels• Manages tapes• Balances resources• Supports compression and encryption• Degrades gracefully


Some Amanda Terminology

• Dump cycle– How often do you want a full backup?

• Disklist Entry / DLE / Target– Something (partition, filesystem) you want to backup


Amanda's Philosophy

• “You tell me how often you want a full backup, and I'll worry about everything else”

• At least one full backup of each DLE per cycle• You don't get to say when full backups happen• Sounds scary, but usually sufficient


Real World Examples – CIS @ UAB

• Before....

(And this isn't even the whole dirty truth)



• So we bought...

Great! Onlyone problem...



• This hardware is very expensive• I work for a public university...• ....in Alabama ;-)• We had no more money• Amanda saved the day• Usage at CIS

– 43 filesystems/partitions on 14 Linux, Solaris and Windows clients

– ~ 60GB of data per night– Fully automated, requiring about 2 hours of attention so

far this year


Real World Examples - Others

• Other Examples– Much Larger

• One user wrote about a 700GB nightly dump

– Much Smaller• one client systems

– Vtape setups• virtual tapes on disk• periodically burn to DVD


Typical Amanda Daily Operation

• At 2:30pm, am check utility runs via cron job and informs me via email if there are any problems

• At 2:00am, am dum p utility runs to kick off the Amanda backup process

• Sometime in the morning, amdump sends an email summarizing last night's activity.


Sample amdump Email OutputThese dumps were to tape CIS-004.The next tape Amanda expects to use is: CIS-005.

STATISTICS: Total Full Daily -------- -------- --------Estimate Time (hrs:min) 1:00Run Time (hrs:min) 2:57Dump Time (hrs:min) 3:24 2:02 1:22Output Size (meg) 20957.1 17678.4 3278.8Original Size (meg) 38473.8 31408.7 7065.1Avg Compressed Size (%) 54.5 56.3 46.4 (level:#disks ...)Filesystems Dumped 41 11 30 (1:28 3:1 4:1)Avg Dump Rate (k/s) 1753.8 2474.3 682.5

Tape Time (hrs:min) 0:21 0:15 0:05Tape Size (meg) 20957.2 17678.4 3278.8Tape Used (%) 4.3 3.6 0.7 (level:#disks ...)Filesystems Taped 41 11 30 (1:28 3:1 4:1)Avg Tp Write Rate (k/s) 17361.9 19931.6 10242.2

USAGE BY TAPE: Label Time Size % Nb CIS-004 0:21 20957.2 4.3 41


Sample amdump Email Output (Continued)

NOTES: planner: Full dump of virginia:/hc promoted from 4 days ahead. planner: Full dump of florida:/var promoted from 4 days ahead. planner: Full dump of alabama:/etc promoted from 4 days ahead. planner: Full dump of florida:/etc promoted from 1 day ahead. planner: Full dump of alabama:/home promoted from 1 day ahead. planner: Full dump of georgia:/home promoted from 4 days ahead. planner: Full dump of illinois:/home promoted from 4 days ahead. planner: Full dump of newyork:/home promoted from 4 days ahead. planner: Full dump of newjersey:/ promoted from 4 days ahead. taper: tape CIS-004 kb 21462176 fm 41 [OK]

DUMP SUMMARY: DUMPER STATS TAPER STATS HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s-------------------------- --------------------------------- ------------missouri.hpcl.c -xport/home 1 24671 13205 53.5 3:42 59.6 0:018848.1missouri.hpcl.c /export/opt 1 1055 49 4.6 0:44 1.1 0:01 42.8missouri.hpcl.c /var/mail 1 1538303 828724 53.9 13:171040.4 1:299345.3nevada.cis.ua / 4 2271760 881675 38.8 5:092850.4 1:0813033.1ohio.cis.u /home 1 1540 247 16.0 0:01 401.8 0:04 69.7maine.cis. /home 1 82640 59078 71.5 0:144300.2 0:087671.2florida.cis.u /etc 0 5200 1613 31.0 0:03 625.5 0:011186.9florida.cis.u /he 1 8780 614 7.0 0:26 23.4 0:01 512.8florida.cis.u /hf 1 16440 2066 12.6 2:02 16.9 0:021249.4etc....

(brought to you by Amanda version 2.4.4p2)


Amanda's planner In Action

planner clientA

“Please estimate level 0, level 1 and level 2 backups for /home.”Let's assume that last night Amanda performed a level 1 backup of clientA's /home DLE...

clientA returns the info

“Will promoting this to a level 0 (full) dump lead tomore balance over the dump cycle?”

“Will we save a significant amount of tape space bygoing to a level 2 incremental instead of level 1?”

SchedulesLevel 0 dump

SchedulesLevel 2 dump

Schedules level 1 dump

Yes

Yes

No

No


More about planner

• Guarantees one full dump per cycle• If the tape is too small for the run's data, it will

delay some of the dumps in the least disruptive way

• Looks at past dumps to determine optimal balancing behavior

• Tries to stay as close to level 0 as possible to reduce need to use multiple tapes for restore


Restoring One Or A Few Files

[root@alabam a /tm p]# amrecover -C CIS -s amanda.cis.uab.edu -t amanda.cis.uab.edu -d /dev/nst0AMRECOVER Version 2.4.4p1. Contacting server on am anda.cis.uab.edu ...220 keep AMANDA index server (2.4.4p2) ready.200 Access OKSetting restore date to today (2004-06-28)200 Working date set to 2004-06-28.Scanning /dum ps/am anda... 20040622: found Am anda directory.200 Config set to CIS.200 Dum p host set to alabam a.cis.uab.edu.am recover> setdisk /var/spool/mail200 Disk set to /var/spool/m ail.am recover> setdate 2004-06-25200 Working date set to 2004-06-25.

Problem: User 'bryant' requests that you restore his INBOX to its state on 2004-06-25.Solution: Use the amrecover utility


Restoring One Or A Few Files (Con't)

am recover> ls[...]2004-06-25 brockhw2004-06-25 brownta2004-06-25 bryant2004-06-25 byrdv[...]am recover> add bryantAdded /bryantam recover> extract Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu.The following tapes are needed: CIS-024 Restoring files into directory /tm pContinue [?/Y/n]? Y Extracting files using tape drive /dev/nst0 on host am anda.cis.uab.edu.Load tape CIS-024 nowContinue [?/Y/n/s/t]? Y./bryantam recover> quit200 Good bye.[root@alabam a /tm p]# ls -l bryant-rw------- 1 bryant disk 14533946 Jun 24 19:10 bryant


Restoring An Entire Backup TargetProblem: The disk holding /etc on a critical server has failed Solution: Use the amrestore utility.

[root@am anda testrestore]# amadmin CIS info alabama.cis.uab.edu '/etc$' Current info for alabam a.cis.uab.edu /etc: Stats: dum p rates (kps), Full: 481.0, 320.0, 350.0 Increm ental: 19.0, 13.0, 11.0 com pressed size, Full: 20.6%, 20.6%, 20.6% Increm ental: 5.4%, 5.4%, 5.4% Dum ps: lev datestm p tape file origK com pK secs 0 20040623 CIS-022 23 18690 3849 8 1 20040628 CIS-027 10 700 38 0[root@am anda testrestore]# su - amanda-bash-2.05b$ am tape CIS label CIS-022am tape: scanning for tape with label CIS-022am tape: slot 26: date 20040623 label CIS-022 (exact label m atch)am tape: label CIS-022 is now loaded.-bash-2.05b$ exit


Restoring An Entire Backup Target (Con't)

[root@am anda testrestore]# amrestore -p /dev/nst0 alabama.cis.uab.edu /etc > etc.0.tar

am restore: 0: skipping start of tape: date 20040623 label CIS-022am restore: 1: skipping am anda.cis.uab.edu.__texas_dfs_hom e_undergrad.private.20040623.1am restore: 2: skipping florida.cis.uab.edu._etc.20040623.1am restore: 3: skipping alabam a.cis.uab.edu._root.20040623.1[...]amrestore: 21: skipping missouri.cis.uab.edu._.20040623.0amrestore: 22: skipping oregon.cis.uab.edu._home.20040623.1amrestore: 23: restoring alabama.cis.uab.edu._etc.20040623.0amrestore: 24: reached end of information

[root@am anda testrestore]# ls -ltotal 18716-rw-r--r-- 1 root root 19138560 Jun 28 14:40 etc.0.tar


Other Ways To Restore Files

• Amanda not available? root@am anda testrestore]# mt -f /dev/nst0 rewind[root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1AMANDA: TAPESTART DATE 20040623 TAPE CIS-022

1+0 records in1+0 records out[root@am anda testrestore]# mt -f /dev/nst0 fsf 01[root@am anda testrestore]# dd if=/dev/nst0 bs=32k count=1AMANDA: FILE 20040623 am anda.cis.uab.edu //texas/dfs/hom e/undergrad.public lev 1 com p .gz program /usr/bin/sm bclientTo restore, position tape at start of file and run: dd if=<tape> bs=32k skip=1 | /bin/gzip -dc | usr/bin/sm bclient -f... -

1+0 records in1+0 records out


When Things Go Wrong

• Data backup is a complex interaction between a lot of players, and things -will- go wrong...– Hosts will be down, or away (laptops)– Tapes will go bad– You'll change a password and then forget to tell Amanda

you changed it– Your holding disk might be too small

• Amanda tries to be proactive by running am check and giving you time to fix problems


Sample amcheck Problem Report

• Host downAmanda Tape Server Host Check-----------------------------Holding disk /dumps/amanda: 60967688 KB disk space available, that's plentyamcheck-server: slot 5: date 20040520 label CIS-018 (exact label match)NOTE: skipping tape-writable testTape CIS-018 label okServer check took 175.627 seconds

Amanda Backup Client Hosts Check--------------------------------WARNING: vermont.cis.uab.edu: selfcheck request timed out. Host down?Client check: 13 hosts checked in 30.210 seconds, 1 problem found(brought to you by Amanda 2.4.4p2)


More amcheck Sample MessagesNOTE: index dir /usr/local/etc/amanda/CIS/index/amanda.cis.uab.edu/__texas_dfs_home_undergrad.private: does not exist

ERROR: georgia.cis.uab.edu: [access as amanda not allowed from amanda@amanda] amandahostsauth failed

ERROR: virginia.cis.uab.edu: [dir /etc needs 64KB, only has 5KB

WARNING: holding disk /dumps/amanda: only 50254708 KB free (52428800 KB requested)

amcheck-server: slot 19: rewinding tape: No medium foundamcheck-server: slot 19: date 20040330 label CIS-027 (active tape)amcheck-server: fatal slot 20: slot 20 move failedERROR: label CIS-016 or new tape not found in rack (expecting tape CIS-016 or a new tape)

amcheck-server: could not get changer info: could not read result from "/usr/local/libexec/chg-scsi"

WARNING: skipping tape test because amdump or amflush seem to be runningWARNING: if they are not, you must run amcleanup


amcheck Is The Best Thing Since Sliced Bread

• Cron this script to email you every day before you leave the office

• Run this script manually any time you touch a config file

• Run this script manually any time you add or change a client

• It will save you hours of troubleshooting• One of the best features of Amanda


amcheck Can't Do Everything

• am check cannot catch problems that arise after it runs

• am check doesn't check for everything• Amanda's other utilities are really good at telling

you why they could not do their job


Revisiting The amdump Email Report

• Very rarely is the email report as uneventful as was presented earlier. It typically will include something like this:

These dumps were to tape CIS-004.The next tape Amanda expects to use is: CIS-005.

FAILURE AND STRANGE DUMP SUMMARY: alabama.cis.u /usr lev 1 STRANGE

STATISTICS: Total Full Daily -------- -------- --------Estimate Time (hrs:min) 1:00Run Time (hrs:min) 2:57

etc....


Revisiting The amdump Email Report (Con't)

• And then later on, it will explain itself....

FAILED AND STRANGE DUMP DETAILS:

/-- alabama.cis.u /usr lev 1 STRANGEsendbackup: start [alabama.cis.uab.edu:/usr level 1]sendbackup: info BACKUP=/bin/gtarsendbackup: info RECOVER_CMD=/bin/gzip -dc |/bin/gtar -f... -sendbackup: info COMPRESS_SUFFIX=.gzsendbackup: info end? gtar: ./local/majordomo-1.94.5/log/mdlog: file changed as we read it| Total bytes written: 207923200 (198MB, 3.1MB/s)sendbackup: size 203050sendbackup: end\--------

Amanda is telling us a file was in use as it tried to grab it. This is usually harmless and constitutes 99% of STRANGE results.


Revisiting The amdump Email Report (Con't)• Sometimes, there are more serious failures

FAILURE AND STRANGE DUMP SUMMARY: amanda.cis.u //texas/dfs/simnetxpcd lev 0 FAILED [disk //texas/dfs/simnetxpcd, all estimate failed] amanda.cis.u //texas/dfs/classfiles lev 0 FAILED [disk //texas/dfs/classfiles, all estimate failed] amanda.cis.u //texas/dfs/officefiles lev 0 FAILED [disk //texas/dfs/officefiles, all estimate failed] amanda.cis.u //texas/dfs/scripts lev 0 FAILED [disk //texas/dfs/scripts, all estimate failed] amanda.cis.u //texas/dfs/home/graduate.private lev 0 FAILED [disk //texas/dfs/home/graduate.private, all estimate failed] amanda.cis.u //texas/dfs/home/graduate.public lev 0 FAILED [disk //texas/dfs/home/graduate.public, all estimate failed] amanda.cis.u //texas/dfs/home/undergrad.private lev 0 FAILED [disk //texas/dfs/home/undergrad.private, all estimate failed]

This was because I changed the password to the SAMBA share that Amanda was using to back up a Windows server. The email clued me in and the problem was resolved quickly.


Handling Tape Failures

• Amanda keeps going and stores as much as possible in the holding disk

• You can then use am flush to flush data to tape

-bash-2.05b$ amflush CIS Scanning /dum ps/am anda... 20040516: found Am anda directory. 20040517: found Am anda directory.

Multiple Am anda directories, please pick one by letter: A. 20040516 B. 20040517 Select directories to flush [A..B]: [ALL]

Today is: 20040517 Flushing dum ps in 20040516, 20040517 using tape changer "chg-scsi". Expecting tape CIS-015 or a new tape. (The last dum ps were to tape CIS-014) Are you sure you want to do this [yN]? y Running in background, you can log off now. You'll get m ail when am flush is finished.


Amanda Prerequisites

• A server that is mostly idle during the times that you want to do your backups

• Enough disk space for a suitable holding disk• GNU tar• Samba (for Windows clients)• A large capacity tape drive (typically)• GNUplot


Configuration Roadmap

• Set up server– Create amanda user and assign to group with permission

to use the tape and changer devices (e.g. 'disk' on Linux)– Gather info on tape and changer devices - mt and mtx

are handy scripts to have here– Open ports (10080,10082,10083) and set up services

(amanda, amandaidx, amidxtape)– Configure amanda.conf & changer (e.g. chg-scsi.conf)

• Set up clients– Create amanda user– Config ports, services, files, directories– Allow access (typically .amandahosts)


Things To Think About Before Proceeding

• What should my cycle be?– Once per night? Every three nights?

• If your cycle seems convoluted, try using two:– One for daily backups, once per night, one week cycle– One for archives, always full dumps, run manually when

you need it

• Find the right balance for the cycle length– Short cycles eat up lots of resources doing full dumps– Long cycles can be a pain to restore from, you might

need 4 tapes for example

• How many tapes to use?– How far back into the past do you want to go?


amanda.conf

• Amanda's main configuration file• Many options• Well documented• Cannot begin to cover everything here, definitely

read the documentation first!• This is where you define your cycle's parameters

(length, number of runs, number of tapes, etc...)


amanda.conf - dumptypes

• Different rules for each DLE

define dum ptype root-tar { global program "GNUTAR" com m ent "root partitions dum ped with tar" com press none index yes exclude list "/usr/local/lib/am anda/exclude.gtar" priority low}

define dum ptype com p-high { global com m ent "very im portant partitions on fast m achines" com press client best priority high}


disklist

• Defining your DLEs / targets

# File form at is:## hostnam e diskdev dum ptype [spindle [interface]] # the tape server itselfam anda.cis.uab.edu / com p-root-tar-exclude-holdingdisk # the directory servernewjersey.cis.uab.edu / com p-root-tar # the file servervirginia.cis.uab.edu /m z com p-user-tarvirginia.cis.uab.edu /root com p-root-tarvirginia.cis.uab.edu /usr com p-user-tarvirginia.cis.uab.edu /hc com p-root-tar


Configuring Tape Changers

• Amanda has a generic interface to tape changers• Tape changer configuration is stored in chg-

scsi.conf• chg-scsi.conf is one of many changer scripts that

come with Amanda• This is where you tell Amanda how many drives

you have, which tapes go to which drives, barcode support, etc...


Tape Drive Configuration Heads-Up

• There are many choices of changer configuration scripts (chg-scsi, chg-multi, chg-mtx, chg-manual, chg-disk, etc...)

• You may have to use one even if you don't have an actual changer (chg-manual, chg-disk)

• Many options, many chances for confusion• Please read docs/TAPE.CHANGERS and other

sources of information (listed at end of talk)


Configuring Your Tape Collection

• Use amlabel to label new tapes– am label CIS CIS-000 slot 0

• Then use amtape to build the tapelist– am tape CIS updateam tape: scanning all 30 slots in tape-changer rack:

slot 26: date 20040623 label CIS-022

etc...


Special Case Configurations

• Windows Clients– Use Samba to backup Windows clients– Configure the Amanda server or other Unix client with

Samba shares– Then in the disklist point to that server and share

•am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar


Special Case Configurations

• Windows Clients– Use Samba to backup Windows clients– Configure the Amanda server or other Unix client with

Samba shares– Then in the disklist point to that server and share

•am anda.cis.uab.edu //texas/dfs/officefiles com p-user-tar

• Firewalls– --with-tcpportrange=40000,40030 ( something > 1024)– --with-udpportrange=920,940 (something < 1024)– or, iptables has amanda support you can enable


Introduction To Bacula

What is Bacula?How is it different?


What is Bacula?

• “The Network Backup Tool for Linux, Unix, Mac and Windows.”

• Another open-source project that aims to provide a robust network-based, multiplatform backup solution

• Newer than Amanda (started in 2000)

“It comes in the night and sucks the essence from your computers.” - Kern Sibbald


Bacula Features

• Modular, scalable components• Its own protocols on top of TCP and UDP• Client and server components that partner to

stream and store data for backup and recovery• Clean component separation - all communication

between them goes over the network• Threaded rather than multiple processes• Excellent documentation


How is Bacula Different than Amanda?

• It can support multiple volumes• You may find it easier to setup• There are both command line and GUI

configuration tools available• Scheduler gives you more control over what jobs

run at which times• It will reuse a tape on multiple nights until it is

full• Support for automated restores from bare metal• Native support for Windows (no Samba/NFS)• SQL database support


Sample Bacula Topology

DirectorDaemon

DatabaseServer

File DaemonWindows

File DaemonLinux

File DaemonUnix

File DaemonOS X

StorageDaemon

TapeDevice

StorageDaemon

TapeChanger

StorageDaemon

DiskDevice

AdminConsole


Component Roles

• The Director manages all scheduling and job creation. It is via an administration interface talking to the Director that the backup administrator controls the backup process.

• The Storage Daemon is responsible for writing data out to disk/tape/changer

• The Database keeps the catalog of what has been backed up and where

• The File Daemon streams the data to be backed up from the client to the Storage Daemon


Prerequisites

• Bacula currently works with SQLite, MySQL and PostgreSQL

• GNU C++ 2.95 or higher to compile• Other software may be necessary depending on

configuration. If burning DVDs, you need the dvd+rw-tools. If using the GUI console, you need recent GNOME and GTK+ libs.


Configuration Overview

• Bacula is configured via a series of config files– bacula-dir.conf, bacula-fd.conf, bacula-sd.conf,

console.conf

• Examples coming here


Storage Organization

• Bacula organizes tapes (volumes) into pools. It will use one volume up until it is full, and then move on to the next. You can give more control if you want to use a new tape each night

• Similar to Amanda, each Bacula volume get a unique label and added to the pool. However, unlike Amanda, there is no set rotation. Bacula will use one until full, and then go look for another, etc...

• Can use multiple pools to ensure that a new tape is used each day. e.q. Setup a Monday pool, a Tuesday pool, etc...


Pre and Post Job Scripts

• Bacula's File Daemon has the ability to run a script before and after a job

• This can be used to shutdown a database in order to take a safe backup of it, for example

• Use the bacula-fd.conf directives “Run Before Job” and “Run After Job”


Take Home Points...

• Amanda is very robust, highly scalable, almost infinitely configurable, and can very likely handle your data backup situation

• If you find Amanda limiting or do not agree with the scheduling philosophy, Bacula may be for you. Momentum -may- be headed to Bacula.

• Once you get either of these configured, you can trust it, and move on with your life. They work, are battle-tested and just as reliable, if not more so, than the expensive commercial products.

• Test your backup and recovery system and strategy early and often!!


For More Information

• http://www.amanda.org/• Documentation

– README, INSTALL, docs/* (esp. FAQ and TAPE.CHANGERS), example/*

– The AMANDA section in O'Reilly's “Unix Backup and Recovery”

– man pages

• http://www.bacula.org/– Documentation - Tutorial, Quick Start, User Guide

• The user communities are wonderful– Sign up for [email protected] at

http://www.amanda.org/support/mailinglists.php– Sign up for [email protected] at

http://lists.sourceforge.net/lists/listinfo/bacula-users

http://www.amanda.org/

http://www.bacula.org/

mailto:[email protected]

http://www.amanda.org/support/mailinglists.php



My Information

• My email is [email protected]• This presentation is available at

http://www.cis.uab.edu/fran/• More Amanda and Bacula information can be

found at the above URL, including:– A document detailing every step of my Amanda

configuration, along with complete sample amanda.conf and chg-scsi.conf

– Notes on configuring Samba to backup Windows shares– Sample Bacula configuration files for each component– Notes on strategies for setting up volumes and pools for

Bacula


http://www.cis.uab.edu/fran/


My Information

• My email is [email protected]• This presentation is available at

http://www.cis.uab.edu/fran/• Much more Amanda information can be found at

the above URL, including:– A document detailing every step of my configuration– A complete sample amanda.conf and chg-scsi.conf– Notes on configuring Samba to backup Windows shares

The End - Thank You!


http://www.cis.uab.edu/fran/

open source data backup, or: how to sleep better at night (oscon 2005)

Technology

gb of data

store data

standard backup levels

money amanda

real world examples

amanda quick look

amanda major differences

action real world examples