Post Katrina Disaster Post Katrina Disaster Recovery PlansRecovery PlansEducause Western Regional Educause Western Regional

Conference Conference April 26, 2006April 26, 2006

San Francisco, CaliforniaSan Francisco, California

Copyright StatementCopyright Statement

• Copyright Ann Dobson 2006. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Panel TopicPanel Topic

• Panel discussion on how Universities have revised Disaster Recovery and Business Continuity plans in light of Hurricane Katrina and the lessons learned by Tulane and other Universities

Panel TopicPanel Topic

• Most Universities have Disaster Recovery and Business Continuity plans for IT.  Lessons learned from Hurricane Katrina exposed weaknesses in those plans.  For example, most plans don’t include closure for an entire semester.  This discussion will center around how Universities are applying the lessons learned from Katrina.  We will share ideas, help others get started, answer questions, and brainstorm solutions.

Panel PresentersPanel Presenters

• Kevin Barney, UC San Francisco

• Jacqueline Craig, UC Office of the Pres.

• Ann Dobson, UC Berkeley

• Nina Hundley, UC Berkeley

• Randy Jones, UC San Francisco Med Ctr

• Paul Weiss, UC Office of the Pres.

Information Resources and Communications University of California, Office of the President

Jacqueline CraigDirector of Policy, IR&COffice of the PresidentUniversity of California

Emergency Planning and Disaster Recovery Policy

University of California

Information Resources and Communications University of California, Office of the President

Current IT policy framework• Business and Finance Bulletin, IS-3,

Electronic Information Security– Risk Assessment and criticality

classification• Essential• Required• Deferrable

– Essential services must be included in Disaster Recovery Plans

Emergency Planning and Disaster Recovery Policy

University of California

Information Resources and Communications University of California, Office of the President

Current IT policy framework

Campus planning must include:– provisions for running essential

applications– emergency response procedures– backup of data and software– vendor requirements

Emergency Planning and Disaster Recovery Policy

University of California

Information Resources and Communications University of California, Office of the President

New Policy Framework

Presidential Policy on Safeguards, Security, and Emergency Management– each campus must implement

comprehensive and effective program– conduct risk assessment, risk mitigation,

emergency preparedness and response, and business recovery

Emergency Planning and Disaster Recovery Policy

University of California

Information Resources and Communications University of California, Office of the President

New UC IT Policy• Inventory and classify critical assets

• Continuity Planning at UC includes:– mitigation– preparedness– response– recovery

Emergency Planning and Disaster Recovery Policy

University of California

Information Resources and Communications University of California, Office of the President

Mitigation

• deployment of protective measures appropriate to each classification level

• infrastructure– communications resources– data centers

Emergency Planning and Disaster Recovery Policy

University of California

Information Resources and Communications University of California, Office of the President

Preparedness

• GOAL - reduce risk and minimize disruption of all campus programs

• risk assessment/business impact analysis

• components of continuity planning

Emergency Planning and Disaster Recovery Policy

University of California

Information Resources and Communications University of California, Office of the President

Response

• implement plans, operating procedures, or protocols– priorities for short and long term impacts

• establish processes• build in flexibility – cannot foresee all

potential circumstances

Emergency Planning and Disaster Recovery Policy

University of California

Information Resources and Communications University of California, Office of the President

Recovery• GOAL – re-establish operational

capability identified in planning priorities– people– facilities/property– financial and information technology

systems

Emergency Planning and Disaster Recovery Policy

University of California

UC Wide Disaster Recovery UC Wide Disaster Recovery Before KatrinaBefore Katrina

• Minimal sharing between UC entities

• Each campus & medical center had own plans and approach

• Office of the President had offsite vendor contract for critical systems and, cost if ever used DR site for real, is extremely expensive.

UC Wide Disaster Recovery After UC Wide Disaster Recovery After KatrinaKatrina

• The primary reactions - Increased urgency & enhanced spirit of SHARING across UC system!

• Joint Data Center Managers Group (comprised of all UC entities) begins work to consider sharing opportunities within UC. Two goals:– Enhance IT DR preparedness by eliminating

duplicate efforts– Consider redirecting spend from outside

vendors to within UC system

UC Wide Disaster Recovery After UC Wide Disaster Recovery After KatrinaKatrina

• IT Leadership Council (UC wide CIO working group) sponsors all day meeting on business continuity planning

• Guest speaker John Lawson (CIO at Tulane) speaks at meeting and shares two key lessons from his experience:– Communication is key– Pay your personnel

UC Wide Disaster Recovery After UC Wide Disaster Recovery After KatrinaKatrina

• JDCMG takes Tulane lessons and begins work on two efforts:– Determine if can back up all mainframes w/in

the UC system– Set up shared web site, registration site, and

email emergency site that any entity w/in UC can utilize if their local capabilities are disabled.

UC Berkeley Disaster Recovery UC Berkeley Disaster Recovery Before KatrinaBefore Katrina

• Emergency Web Site

• Emergency Purchasing Procedures

• On-site Backups

• Off-site Backups

• Off-site Vital Records

UC Berkeley Disaster Recovery UC Berkeley Disaster Recovery Before KatrinaBefore Katrina

• Business Impact Analysis

• IBM Hot Site Contract

• Annual Hot Site Tests

• New Applications Added Each Year

UC Berkeley Disaster Recovery UC Berkeley Disaster Recovery After KatrinaAfter Katrina

• Servers for redundancy at sister campus– CalNet authentication and directory services– Web communications tools (home page)– People locator– Distance learning (Sakai, Webcast)– Other applications, e.g. student registration

UC Berkeley Disaster Recovery UC Berkeley Disaster Recovery After KatrinaAfter Katrina

• Additional satellite phones

• Laptops, printers for response teams

• Data backup pilot for researchers

• Portable generators

• Planning for Wide-spread Regional Impact• Multi-pronged approach to Business Continuity

Risk Assessment - Analysis Prevention and Mitigation – New Data Center Emergency Preparedness – Emergency Operations Plan Emergency Communications – Immediate Information Business Resumption – Resume Business Functions Disaster Recovery – Recover Technical Environment

Administrative Computing

Planning for Business Continuity: “Lessons Learned from Katrina”

Identify critical business functions Determine priorities Establish recovery point objective (RPO) Establish recovery time objective (RTO) Determine disaster recovery strategy Develop disaster recovery procedures Contract recovery services

Business Continuity Governance

Align IT and business activities around business resumption requirements

Ensure responsibility and accountability are accepted by key stakeholders

Joint development of technical and business recovery plans

Integrate planning and testing Validate capabilities and compliance without

crippling productivity

Specific Challenges

Establish priorities Validate plan Justify funding and staffing Coordinate multiple plans

Information Technology Services• Associate Vice Chancellor, ITS• Director of Administrative

Computing• Administrative Computing Chief

Technology Officer• Administrative Computing

Operations Manager Controller’s Office

• Associate Vice Chancellor & Controller

• Asst. Controller, Fin. & Admin. Systems

• Asst. Controller, Disbursements• Asst. Controller, Acctng. / Reptng.

Business Continuity Committee

Develop plans Test & maintain procedures Conduct the recovery

Information Technology Services

• ITS Application Manager

• ITS Application Programmer/Analyst

• Open System Administrator

• MVS System Programmer Controller’s Office

• Manager Functional Unit

• Analyst Functional Unit

Business Continuity Team

Payroll / Personnel• Resident on the UCOP system • UCOP hot site contract covers 5 campuses on the base payroll

system + UCSF payroll• Last recovery test performed Fall 2005• RPO = 24 to 48 hours before disaster• RTO = 3 days after disaster

• Other Mainframe Applications• Need to acquire UCSF hot site contract• Develop recovery procedures for system and remaining

applications• Test and maintain procedures

Mainframe Recovery

Establish Hot Site Contact

• Recovery of critical business functions within five days

Emergency Communications – University-wide Solution

• Immediate availability

• Emergency communications website

• Emergency email

• Employee Registration

Establish Cold Site recovery at other UC campus

• Longer term recovery (Hitching-post site)

Second Site at UCSF

• Renovate former Data Center as a recovery site for limited localized outages

Backup Printing

• Backup printers at UCSF second site

• Backup printing at other UC campus

Open Systems Recovery Proposal

UCSF Medical Center Disaster Recovery Plan before Katrina

Overview of UCSF Medical CenterOverview of UCSF Medical Center Install generator.Install generator. Hire DRP Coordinator in 2002.Hire DRP Coordinator in 2002. Analysis of all application backups. Analysis of all application backups.

Make changes.Make changes. Internal Business Impact Analysis.Internal Business Impact Analysis. Recovery Plans for Specific ApplicationsRecovery Plans for Specific Applications Recover applications locally, Document Recover applications locally, Document

recoveries.recoveries. IBM BCRS Contract for 5 critical systems.IBM BCRS Contract for 5 critical systems.

UCSF Medical Center Disaster Recovery Plan before Katrina

Completed 5 Recovery Tests since Completed 5 Recovery Tests since December 2002.December 2002.

Recovery Plans for Specific Applications.Recovery Plans for Specific Applications. Cross train staff.Cross train staff. Iron Mountain 7 days a week.Iron Mountain 7 days a week. During Analysis Phase of an Application, During Analysis Phase of an Application,

complete high level assessment of DR complete high level assessment of DR implications, including Senior implications, including Senior Management Ranking.Management Ranking.

Examine hot site options.Examine hot site options.

UCSF Medical Center Disaster Recovery Plan after Katrina

Increase of Senior Management Increase of Senior Management awareness and concern.awareness and concern.

IBM Consultancy for facilities IBM Consultancy for facilities analysis, business impact analysis, business impact analysis, and recommendations analysis, and recommendations and plan going forward.and plan going forward.

Begin re-examine hot site Begin re-examine hot site options.options.