BPR476 Windows Sharepoint Services And Sharepoint Portal Server: Data Protection, Disaster Recovery, Site ArchivalAnd Beyond

Joel S. OlesonJoel S. OlesonTechnologistTechnologistMicrosoft CorporationMicrosoft Corporation

OverviewMicrosoft IT Case StudyMicrosoft IT Case Study

MS SharePoint EnvironmentMS SharePoint EnvironmentIT Pain PointsIT Pain PointsData Protection / BackupsData Protection / BackupsDisaster RecoveryDisaster RecoveryData Recovery / RestoreData Recovery / Restore

Demo TextCopyDemo TextCopyIT ImprovementsIT Improvements

Chofu

Dublin

Singapore

SharePoint Deployment

Redmond

Stats:Stats:10 WSS Web Farms, 6 SPS Web Farms10 WSS Web Farms, 6 SPS Web Farms7 SQL Clusters; 4 Search clusters, 8 Index boxes7 SQL Clusters; 4 Search clusters, 8 Index boxes2.5 TB of database content2.5 TB of database content~320 SPS 2003 Portals~320 SPS 2003 Portals50,000+ Site Collections50,000+ Site Collections80,000+ User Profiles80,000+ User Profiles

Americas; Europe, Middle East, Africa (EMEA); Americas; Europe, Middle East, Africa (EMEA); South Pacific; Far EastSouth Pacific; Far East

4 Regional Intranet Deployments4 Regional Intranet Deployments3 Regional Extranet Deployments3 Regional Extranet Deployments

Americas Intranet Hardware

MSWeb Shared Services

2 WFE

2 Search

2 Index

2 WFE

Portal 100

2 WFE

Portal 200

2 WFE

Upgraded WSS Farm

SQL (A/P)

Team SQL Cluster

SharePoint Portal Server 2003 SharePoint Services

2 WFE

Team Farm

2 WFE

Host Header

SQL (A/P)

Portal SQL Cluster

Regional SharePoint

SQL (A/P)

Consolidated SQL Cluster

Intranet Deployment

SQL 2000

Extranet Deployment

Shared Services and Portal Farm

2 WFE

2 Search

2 Index

2 WFE

Corp Hosting

2 WFE

Host Header

2 WFE

Extranet WSS

IT SharePoint Storage4 TB SAN (DB)4 TB SAN (DB)2 TB per SQL Cluster 2 TB per SQL Cluster

(DB) (DB)

Plan to Reclaim 50%Plan to Reclaim 50%

Site Backups Site Backups

  Data (29%)Data (29%)

  

  DB Dump (29%) DB Dump (29%)

  

  Site Dump (29%)Site Dump (29%)

  

Transaction Logs &Transaction Logs &System databases (13%)System databases (13%)

Backup Infrastructure

WSS Backup / Restore Legend

1.0 DB Dump to SAN (attached to SQL Backend)2.0 – 2.2 Site Collection Backup to SAN3.0 – 3.1 Site Collection / DB Backup to Tape4.0 Restore of Site Collection to any Share4.1 Restore of DB to Disk (a. for Disaster & Recovery b. Restore files from DB)

Dump

specific site

SQL Backend

Dump Dump Dump

Tape BackupSite Collection WinDisk

2.0 2.1 2.2

DB DB DB

SAN

Site Collection

Database WinDisk DB Dump WinDisk

WSS Frontend

1.02.0

Backup

3.03.14.1

4.0

Tape Content

Service Growth # of MySites

05,000

10,00015,00020,00025,00030,000

Jul Aug Sep Oct Nov Dec Jan Feb Mar

# of WSS Sites

010,00020,00030,00040,00050,00060,000

Oct Nov Dec Jan Feb Mar

# of WSS Extranet Sites

0

500

1000

1500

2000

2500

Jul Aug Sep Oct Nov Dec Jan Feb Mar

# of Portals

050

100150200250300350

Jul Aug Sep Oct Nov Dec Jan Feb Mar

# of TB in Sharepoint

0

0.5

1

1.5

2

2.5

Jul Aug Sep Oct Nov Dec Jan Feb Mar

# of TB in fileshare

0.0

2.0

4.0

6.0

8.0

10.0

Jul Aug Sep Oct Nov Dec Jan Feb Mar

IT Pain PointsTapeTape

Tape Backups long runningTape Backups long running300-500MB/min300-500MB/minHigh NIC utilization by Tape backupHigh NIC utilization by Tape backup

DatabaseDatabaseDatabase backups long running Database backups long running

1TB 10hr (100 GB/hr)1TB 10hr (100 GB/hr)SPSBackup backup duplication of databasesSPSBackup backup duplication of databases

Site CollectionSite CollectionSTSADM doesn’t scale well over the network STSADM doesn’t scale well over the network

~300GB in 24/hr (12MB/min)~300GB in 24/hr (12MB/min)STSADM full 2-3 days (1TB)STSADM full 2-3 days (1TB)

Restore management painsRestore management pains

Data ProtectionWhat to backupWhat to backupDatabase BackupsDatabase BackupsToolsTools

SPS ToolsSPS ToolsWSS ToolsWSS ToolsSTSADM demoSTSADM demoThird Party Backup VendorsThird Party Backup VendorsScripts/Dev IdeasScripts/Dev Ideas

Disk to TapeDisk to TapeHot Swappable FarmHot Swappable Farm

SharePoint Storage ObjectsDatabase

Site Collections

Document Libraries and Lists

Files/Documents and List Items

What to BackupDatabasesDatabasesWSS/SPSWSS/SPS

CONFIG DBCONFIG DB

STS_W3SVC#_#STS_W3SVC#_#SPSSPS

PROF DBPROF DB

SERVDBSERVDB

SITEDBSITEDB

*External to Db*External to DbSystem StateSystem State

SPS: Indexes & SPS: Indexes & WebStore Doc LibWebStore Doc Lib

IIS Meta baseIIS Meta base

Web Part PackagesWeb Part Packages

Home DirectoriesHome Directories

Database BackupsDatabase to dump on diskDatabase to dump on disk

DatabaseBackup Optimization

Database sizesDatabase sizes25-50GB Database Storage (>500 25-50GB Database Storage (>500 sites)sites)Number of databases (>300)Number of databases (>300)

Cost SavingsCost SavingsDirect to Media?Direct to Media?Backup compression/SpeedBackup compression/Speed

SQL LiteSpeed – 25-75% compression SQL LiteSpeed – 25-75% compression expectedexpected

Snapshots or ClonesSnapshots or Clones

SharePoint Portal ToolsSPSBackup.exeFull Farm BackupFull Farm BackupSpsbackup.exeSpsbackup.exe /all /file <Backup path and file>/all /file <Backup path and file>Central Portal Index Backup:Central Portal Index Backup:

Spsbackup.exeSpsbackup.exe /portal http://portal /service index /file \\svr\shr\backup /overwrite /portal http://portal /service index /file \\svr\shr\backup /overwrite

DatabasesDatabasesBackward-compatible Doc LibBackward-compatible Doc LibSPS IndexesSPS IndexesXML manifestXML manifest

schema versionschema versionportal GUID portal GUID Path to DB backup filePath to DB backup fileResultResult

Windows SharePoint Services ToolsSTSAdm.exeSTSAdm.exeExample: stsadm.exe –o backup –url Example: stsadm.exe –o backup –url

http://team/sites/sitename -f sitename.bakhttp://team/sites/sitename -f sitename.bakHelp: stsadm.exe –help backupHelp: stsadm.exe –help backup

WSS sites – full fidelity w/ securityWSS sites – full fidelity w/ securityMy sites - content backed upMy sites - content backed up

Personalization, web part customizations, and profile Personalization, web part customizations, and profile changes are not preserved in migrationschanges are not preserved in migrations

Sites under portals – full fidelity w/ security Sites under portals – full fidelity w/ security Smigrate.exe Smigrate.exe

Files & Lists copied to disk with manifest Files & Lists copied to disk with manifest Note: Security on sites not preserved from WSS to WSSNote: Security on sites not preserved from WSS to WSSFrontPage2003 Backup/RestoreFrontPage2003 Backup/Restore

SharePoint BackupThird Party VendorsAvePoint *AvePoint *

DocAve – SPS 2003 – Item level backup/restore/archive with DocAve – SPS 2003 – Item level backup/restore/archive with multiple migration options (supports SQL & Webstore)multiple migration options (supports SQL & Webstore)Downloadable Demo AvailableDownloadable Demo Available

Computer Associates (CA) *Computer Associates (CA) *BrightStor ARCserve Backup BrightStor ARCserve Backup

Backup/Restore SharePoint databases hosted on the various SQL Backup/Restore SharePoint databases hosted on the various SQL servers distributed across the enterprise.servers distributed across the enterprise.Advanced document level backup and restore supportAdvanced document level backup and restore support

CommVault *CommVault *GA, Galaxy- SPS 2003 database level backup and recovery GA, Galaxy- SPS 2003 database level backup and recovery productproduct

Enterprise wide backup and recovery of SPS 2003 data (SQL and Enterprise wide backup and recovery of SPS 2003 data (SQL and indices). Site and sub-site backup and recovery; Webstore backup indices). Site and sub-site backup and recovery; Webstore backup supportsupport

Product in Beta for “Doc Level” backup/restoreProduct in Beta for “Doc Level” backup/restoreVeritas *Veritas *

Backup Exec 10.0 (Late Q3 04) - SPS Server Level backupBackup Exec 10.0 (Late Q3 04) - SPS Server Level backup

* Here at Tech ED 2004* Here at Tech ED 2004

Hot Swappable SQLVia SQL Log Shipping

Mirror ServersMirror ServersSet DB Full or Bulk Logged recovery modeSet DB Full or Bulk Logged recovery modeMirror DataMirror Data

Full Backup of FarmFull Backup of FarmFull Restore of FarmFull Restore of Farm

Enable SQL Log shippingEnable SQL Log shippingSITE, SERV, PROF & Content dbs SITE, SERV, PROF & Content dbs (NOT CONFIG)(NOT CONFIG)

Set DBs on Mirror to Standby or No RecoverySet DBs on Mirror to Standby or No RecoveryConfigure FrequencyConfigure FrequencyCreate Farm, Customization/Portal & Virtual Server Create Farm, Customization/Portal & Virtual Server configconfig

Recommend: Periodic execution of “delete portals and Recommend: Periodic execution of “delete portals and detach databases; reconnect them to mirror farm detach databases; reconnect them to mirror farm config DB” config DB”

Incremental Site BackupIncremental Site BackupBackup/Restore SiteBackup/Restore Site

Site ArchivalCleanup Storage

Analyze Site UsageAnalyze Site UsageLast ModifiedLast ModifiedIIS usageIIS usage

Identify sites to “cleanup”Identify sites to “cleanup”Backup targeted sitesBackup targeted sitesDelete unused SitesDelete unused Sites

Monitoring BackupsSharePoint Backup Relevant MOM Events

WSS MOM PackWSS MOM PackDatabase Capacity ReachedDatabase Capacity ReachedDatabase Warning ReachedDatabase Warning Reached

SPS MOM PackSPS MOM PackSPSBackup.exeSPSBackup.exe backup success/fail backup success/fail

Indexes or databaseIndexes or databaseSQL Server 2000 MOM PackSQL Server 2000 MOM Pack

SQL Server database and transaction log SQL Server database and transaction log space warnings and errors.space warnings and errors.SQL Server failed backupsSQL Server failed backupsSQL Server failed agent jobsSQL Server failed agent jobs

Backup MediaDisk to TapeDisk to Tape

TapeBackup Optimization

Network Upgrade - Over NetworkNetwork Upgrade - Over NetworkPrivate NetworkPrivate NetworkSAN TapeSAN Tape

1GB/Min (db dump to tape)1GB/Min (db dump to tape)Max drive or share sizeMax drive or share size

100-300 GB for quicker restart100-300 GB for quicker restart

Script/Dev ideasIncremental Site Backup – Incremental Site Backup – backup changed backup changed sitessitesDocument backup – Document backup – Web folder backup of Web folder backup of documents or object model iterating through documents or object model iterating through documentsdocumentsIncremental Document backupIncremental Document backup – Backup – Backup changed documents using Object Modelchanged documents using Object ModelArchive – Archive – Use visits w/ modify date to Use visits w/ modify date to determine what sites should be archived determine what sites should be archived (backed up to disk) then deleted(backed up to disk) then deleted

Data RecoveryDisaster RecoveryDisaster RecoveryRecovery Scenarios & RequestsRecovery Scenarios & RequestsRecovery FlowRecovery Flow

DatabaseDatabaseGranularGranular

Demo: Textcopy Demo: Textcopy Doc restore from Content DB Doc restore from Content DB

Portal RestorePortal RestoreService LevelsService Levels

Disaster Recovery: ServerWeb Front End – Web Front End –

A. Restore failed drives and system state (metabase) A. Restore failed drives and system state (metabase) B. Reinstall, remove/re-add topology reconfigure, re-add web parts, B. Reinstall, remove/re-add topology reconfigure, re-add web parts, recreate virtual servers, extend and map to existing virtual serversrecreate virtual servers, extend and map to existing virtual servers

Search – Search – A. Restore failed drives and system state (metabase) propegateA. Restore failed drives and system state (metabase) propegateB. reinstall, remove/add back to topology and propagateB. reinstall, remove/add back to topology and propagate

Index – Index – A. Restore failed drives and system state (metabase) restore Index A. Restore failed drives and system state (metabase) restore Index using using spsbackup.exespsbackup.exereinstall, remove/add back to topology, restore indexes using reinstall, remove/add back to topology, restore indexes using spsbackup.exespsbackup.exe, and propagate, and propagate

SQLSQL – Restore failed databases (except Config db) – Restore failed databases (except Config db)Disconnect reconnect content databasesDisconnect reconnect content databases

Backward Compatible Doc LibBackward Compatible Doc Lib – –Disconnect Doc libs, remove from topology & farmDisconnect Doc libs, remove from topology & farmRe-add new server to farm & topologyRe-add new server to farm & topologyRestore using Restore using SPSBackup.exeSPSBackup.exe restore restore

Recovery of Hot Swap SQL Via SQL Log ShippingSQLSQL

Close Open ConnectionsClose Open ConnectionsChange role of Secondary SQL to Primary Change role of Secondary SQL to Primary (or keep dbs read only until recovery of (or keep dbs read only until recovery of primary)primary)

SharePointSharePointAs needed Create Portal from DBs As needed Create Portal from DBs (connect)(connect)Disconnect/Reconnect Content DbsDisconnect/Reconnect Content Dbs

WINS/DNS to point to new Farm VIPWINS/DNS to point to new Farm VIP-Start a Full Crawl-Start a Full Crawl

Recovery ScenariosDatabase -Database - most reliable disaster recovery most reliable disaster recovery scenarioscenario

Site Collections – Site Collections – STSAdm.exe STSAdm.exe restorerestoreDatabase restoreDatabase restore

Document Library, List, Doc recovery Document Library, List, Doc recovery – – Database restoreDatabase restoreSTSAdm.exeSTSAdm.exe restore restoreDocumentDocument- - Textcopy or other API can make retrieval Textcopy or other API can make retrieval from DB possiblefrom DB possible

Restore Requests

Default 15000 Sites MaxPer Content DB

Avg 20GB

>1% of Restore Requests

Includes Sub sites Workspaces

Avg 30-65 MB

41% Restores68/167 site68/167 site

Doc Libraries, list items, lists, events,Surveys, forms, etc..

Avg 1.5 MB

49% Restores82/167 doc82/167 doc

Database RestoreTape to Disk to Online dbTape to Disk to Online db

Database RecoverySITE / STS_W3SVC#_#SITE / STS_W3SVC#_# – restore database, – restore database, remove from config db using admin UI, re-add remove from config db using admin UI, re-add using admin UIusing admin UI

PROF DB – PROF DB – restore profile database, start full restore profile database, start full import, remove objects missing from import import, remove objects missing from import using UIusing UI

SERV DB – SERV DB – restore services database, bring restore services database, bring online (keep in sync w/ SITE DB)online (keep in sync w/ SITE DB)

Granular RecoveryPortal and below

Restore relevant databases (SITE, SERV, PROF, Restore relevant databases (SITE, SERV, PROF, STS_W3SVC#_#)STS_W3SVC#_#)

Restore to standby environment using Restore to standby environment using SPSBackup.exeSPSBackup.exe /Restore tab GUI /Restore tab GUI

STSAdmSTSAdm for site collection migration for site collection migration

SmigrateSmigrate for area or sub site migration (reapply for area or sub site migration (reapply security)security)

TextCopyTextCopy for doc restore for doc restore

Portal Restore From DBDBs Only (no index)

Central AdminCentral AdminClick - Create a portal siteClick - Create a portal siteChoose - Restore a portal siteChoose - Restore a portal siteSelect Servers and databasesSelect Servers and databasesChoose Virtual Server and specify URLChoose Virtual Server and specify URLExtend and MapExtend and Map

Granular RecoverySite/Sub siteSite CollectionSite Collection

Backed up Backed up Stsadm.exe –o restore –url Stsadm.exe –o restore –url http://http://sitenamesitename –f –f filename –overwritefilename –overwriteRequires 2Requires 2ndnd content db content db

No Site BackupNo Site BackupRestore content databaseRestore content databaseAttach database to Attach database to standbystandby virtual server/Lab virtual server/Lab environmentenvironmentBackup site using Backup site using stsadm.exestsadm.exeRestore using Restore using stsadm.exestsadm.exe with overwrite option with overwrite option

Sub site - use Sub site - use smigratesmigrate to move data to to move data to production with overwriteproduction with overwrite

TextCopyTextCopyRestoring document from DBRestoring document from DB

Demo – TextCopyIdentify Database NameIdentify Database NameUSE @WSS_configUSE @WSS_configSELECT nameSELECT nameFROM sites s, databases dFROM sites s, databases dWHERE s.databaseid = d.databaseidWHERE s.databaseid = d.databaseidAND urlpath = @urlpathAND urlpath = @urlpath

Identify Document/FileIdentify Document/FileUSE [X1_SITE]USE [X1_SITE]SELECT content SELECT content FROM docsFROM docsWHERE dirname = @dirnameWHERE dirname = @dirnameAND leafname = @leafnameAND leafname = @leafname

> textcopy /s server /u @user /P @password /d “database" /t docs /c content > textcopy /s server /u @user /P @password /d “database" /t docs /c content /W "select * from docs where leafname= ‘@leafname’ and dirname= /W "select * from docs where leafname= ‘@leafname’ and dirname= ‘@dirname’ /F c:\temp\leafname /O /Z‘@dirname’ /F c:\temp\leafname /O /Z

Service LevelsCharge back – Charge back – charge the user cost charge the user cost center for restorecenter for restoreSponsor – Sponsor – ask for corporate sponsor on ask for corporate sponsor on restore requestrestore requestQuantify cost of restoreQuantify cost of restoreFull Portal or Virtual ServerFull Portal or Virtual ServerCap off Max Site Collection sizeCap off Max Site Collection size

IT ImprovementsTapeTape

SAN Tape/Fiber Attached BackupSAN Tape/Fiber Attached BackupThroughput 1.2 GB/Min, 2-3xThroughput 1.2 GB/Min, 2-3x

Split drive into ~300GB shares for Tape BackupSplit drive into ~300GB shares for Tape BackupGB Network upgrade on SQL boxesGB Network upgrade on SQL boxes

DatabaseDatabaseReduce database sizes Max 25-50GBReduce database sizes Max 25-50GBReduce DB to >300DB per SQL instanceReduce DB to >300DB per SQL instanceSPSBackup.exeSPSBackup.exe to backup indexes only to backup indexes only

Site CollectionSite CollectionIncremental BackupsIncremental BackupsCleanup/AutoDelete – ArchiveCleanup/AutoDelete – Archive

Restore process flow better understoodRestore process flow better understood

© 2004 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.

Q1:Q1: Overall satisfaction with the sessionOverall satisfaction with the session

Q2:Q2: Usefulness of the informationUsefulness of the information

Q3:Q3: Presenter’s knowledge of the subjectPresenter’s knowledge of the subject

Q4:Q4: Presenter’s presentation skillsPresenter’s presentation skills

Q5:Q5: Effectiveness of the presentationEffectiveness of the presentation

Please fill out a session evaluation on CommNetPlease fill out a session evaluation on CommNet