BackupPC

LINSAM-Infosessie

19/04/2007

Agenda

• Backup@mech

• BackupPC- Basics- User Interface- Future

• Vragen???

Backup@mech

• Hardware- Rack-model van TTEC (5.700,- euro incl. BTW)- 15 SATA2 slots

• 2x 80GB in mirror voor systeemsoftware• 8x 500GB in RAID5 + 1x 500GB hotspare• nog 4 slots vrij• Max. bruto opslagcapaciteit 5,5 TB

- Uitbreidbaar met extra disk-kabinet- OS: RedHat Fedora 6

Backup@mech

• Backup Linux-servers (RHEL4)- rdiff-backup+ssh

• system configs, databases, websites, ftp- en smb-shares• volume: ~590GB

- rsync+ssh• maildir:

– rotatie met hardlink copy: 6 daily, 3 weekly, 1 monthly

– volume: ~134GB (200 gebruikers)

• subversion repositories:– volume: ~6,4GB

Backup@mech

• Backup van eindgebruiker PC’s- BackupPC (versie 3.0.0):

• backup-to-disc systeem voor WinXX, Linux en Mac clients

• CGI-webinterface voor gebruikers en admins• Volume: ~750GB

Backup@mech

• TSM- Incrementele backup van dagelijkse rdiff-backup en

rsync snapshots naar TSM• ~740GB, > 2.000.000 bestanden• Duur incrementele backup: ~1u20

- TSM-backup van BackupPC data-pool problematisch omwille van intensief hardlink-gebruik

• Rsync van BackupPC data-pool naar remote server?

BackupPC

• BackupPC is a high-performance, enterprise-grade system for backing up Linux and WinXX PCs and laptops to a server's disk. BackupPC is highly configurable and easy to install and maintain.

• BackupPC is written in Perl and extracts backup data via SMB using Samba, tar over ssh/rsh/nfs, or rsync. It is robust, reliable, well documented and freely available as Open Source on SourceForge.

BackupPC Basics

• Full Backup- configurable interval (eg. monthly)

- configurable number of full backups to keep

- Exponential expiry supported• eg., a settable number of most recent montly fulls, plus a

settable number of older fulls that are 2, 4, 8, or 16 weeks apart

• Our config: 1 recent monthly full, and 2 older fulls that are 2 and 4 months apart

BackupPC Basics

• Incremental Backup- backup of files that have changed since the last successful full or

incremental backup.

- BackupPC 3.0 supports multi-level incrementals.

- full backup has level 0. new incremental of level N will backup all files that have changed since the most recent backup of a lower level.

- interval and number of incrementals to keep is configurable

BackupPC Basics

• Incremental Backups- For SMB and tar, BackupPC uses the modification time (mtime)

to determine which files have changed since the last lower-level backup.

- SMB and tar incrementals are not able to detect deleted files, renamed files or new files whose modification time is prior to the last lower-level backup.

- Rsync is more clever: any files whose attributes have changed (ie: uid, gid, mtime, modes, size) since last full are backed up. Deleted, new and renamed files are detected.

BackupPC Basics• Partial Backup

- When full backup fails or is canceled, and some files have already been backed up, BackupPC keeps a partial backup containing just the files that were backed up successfully.

- Partial backup is removed when the next successful backup completes, or if another full backup fails resulting in a newer partial backup.

- Failed incremental backups are not saved.

- Partial backups may be browsed or used to restore files just like a successful full or incremental backup.

- With rsync transfer method the partial backup is used to resume the next full backup, avoiding the need to retransfer the file data already in the partial backup.

BackupPC Basics• Pooling common files

- BackupPC pools identical files using hardlinks.• “identical files'' means files with identical contents.• file meta-data (permissions, ownership, and modification time) stored

separately from file contents

- Hashing function = MD5 digest of file length and contents• Number of bytes used for MD5 digest is configurable and determines the

uniqueness of the hash• complete file comparison is always done to verify if two matching files are

really the same

• Compression- Compression provides additional reductions in storage (around 40%)

BackupPC Basics

• Some figures (19/04/2007)

- 79 PC’s and laptops (WinXP and Linux),- 202 full backups of total size 2313.48GB (prior to

pooling and compression),- 459 incr backups of total size 145.66GB (prior to

pooling and compression),- Pool is 501.16GB comprising 887041 files and 4369

directories,- Pool hashing gives 738 repeated files with longest

chain 46,

BackupPC Basics

• Transfer protocols- WinXX clients:

• smb• rsyncd (cygwin)

– rsyncd is the better method:» more efficient transport and more intellegent algorithm for

incremental backup– standalone rsyncd service to be installed on client

» pre-packaged zip file on http://backuppc.sourceforge.net.

- linux, unix or MacOSX clients:• rsync over ssh/rsh/nfs• tar over ssh/rsh/nfs

BackupPC Basics

• Operation- wakes up periodically to queue backups on all the

PCs (usually every hour)- maintains queues of pending backup requests, user

backup requests and administrative commands- expired backups are removed in background.- Once each night, a process is run to complete some

additional administrative tasks, such as cleaning the pool

BackupPC Basics• Periodical email reminders

- send to users when their PC has not recently been backed up.

- Email message, timing and policies are configurable.

• Powerfull HTTP/CGI interface for administrators and users- allows administrators to view the current status, edit configuration,

add/delete hosts, view log files, initiate and cancel backups and browse and restore files from backups

- allows users to view the current status, edit configuration, initiate and cancel backups and browse and restore files for their own PC(s).

BackupPC Server Status

BackupPC Host Summary

Main Config Editor

Add/Delete Hosts

Host Backup Summary

Host Config

LOG File

BackupPC User Interface

• Restore- Single files can be downloaded from any backup

directly from the CGI interface.

- Zip or Tar archives for selected files or directories from any backup can also be downloaded.

- Direct restore to the client machine for selected files or directories is also supported.

Browse Backups

Restore

Future of BackupPC• Features are steadily being added to BackupPC.• Currently, big development activity is around BackupPCd

- BackupPCd is a client for the BackupPC server that will handle all the issues involved in dealing with the client.

- Own transport protocol with the server.• Protocol is being based on the rsync protocol, ensuring a reliable

and efficient transport.- Provides a uniform backup protocol across different client

operating systems, and provides better performance.- BackupPCd will allow ACLs and other file metadata to be

backed up, avoiding the need to install cygwin on Windows machines for rsync

- Will enable backup of open files

Vragen???