principles of incident response and disaster recovery chapter 8 disaster recovery: operation and...

Principles of Incident Response and

Disaster Recovery

Chapter 8Disaster Recovery: Operation and

Maintenance

Principles of Incident Response and Disaster Recovery 2

Objectives

• Understand the key challenges an organization faces when engaged in disaster recovery operations

• Know what actions organizations take to prepare for the activation of the DR plan

• Recognize what critical elements compose the response phase of the DR plan

• Know what occurs in the recovery phase of the DR plan


Objectives (continued)

• Understand how an organization uses the resumption phase of the DR plan

• Know how an organization resumes normal operations using the restoration phase of the DR plan


Introduction

• An organization should operate on the premise that it is only a matter of time until a disaster strikes

• Proper response to a disaster requires meticulous preparation and ongoing diligence

• In the event of a total loss, an organization must be prepared to promptly reestablish operations at a new permanent location


Facing Key Challenges

• Disasters are not confined to the IT department or limited to the assets of the organization

• Disasters may also affect the community and employees personally, as well as vendors and suppliers

• In a major or widespread disaster, there may be challenges associated with local emergency services, service providers, and other-non business issues


Facing Key Challenges (continued)

• Areas possibly affected in a major disaster:– Basic emergency and transportation services– Food and survival supplies– Water supplies and sanitation– Electrical power– Products and services delivered by vendors and

suppliers– Telecommunications services (land and cellular)– Transportation services (freeways, highways, and

local streets)


Facing Key Challenges (continued)

• Major disaster can result in:– Declaration of state of emergency– Imposition of martial law– Restrictions on movement or quarantines

• DR plan typically involves 5 phases:– Preparation– Response– Recovery– Resumption– Restoration


Preparation: Training the DR Team and the Users

• In DR planning, there is no prevention phase• Take steps during preparation to minimize losses• Preparation: making an organization ready for

possible contingencies that escalate to disaster• Preparation phase is continuous, but other phases

are activated by triggers such as:– Management notification– Employee notification– Emergency management notification– Local emergency services– Media outlets


Disaster Recovery Planning as Preparation

• 3 primary objectives of the DR plan:– Eliminate or reduce potential for injuries or loss of

life, damage to facilities, and loss of assets and records to minimize disruption and financial loss and reduce or limit liability exposure

– Stabilize the effects of the disaster to allow recovery efforts to begin

– Implement DR procedures


Disaster Recovery Planning as Preparation (continued)

• Recovery efforts must be prioritized as follows:– Employees– Customers– Facilities– Assets– Records

• CP team creates scenario development and impact analysis, and categorizes the level of threat for each potential disaster



• Key features of the DR plan:– Clear delegation of roles and responsibilities– Execution of the alert roster and notification of key

personnel– Use of employee check-in systems– Clear establishment and communication of business

resumption priorities– Complete and timely documentation of the disaster– Preparations for alternative implementations



• All employees should have 2 types of emergency information in possession at all times:– Personal emergency information (who to notify)– Instructions on what to do in the event of an emergency

(snapshot of the DR plan)• Emergency info should include contact number or

hotline for the organization, emergency services numbers, evacuation and assembly locations, disaster recovery coordinator, etc.

• Crisis management: focused steps that deal with safety of people who are involved in the disaster


DR Training and Awareness

• DR training focuses on the roles each individual is expected to execute during an actual disaster

• For most employees, training is limited to awareness• General job function training is key to being prepared

for disaster recovery actions• Cross-training should also be considered, both

vertically and horizontally, to deal with personnel shortages

• Training should include operating in degraded mode


DR Training and Awareness (continued)

• Disaster management team (command and control group) training is primarily about communication

• Communications team training involves preparing information notices, news releases, and internal memorandums and directives

• Hardware recovery team training may include training to rebuild damaged systems by scavenging from other damaged systems



• Systems recovery team training is mostly the same as their normal operations training

• Network recovery team training may include wireless network installation as a quick recovery mechanism, walkie-talkie deployment, and other connectivity mechanisms

• Storage recovery team training may include rebuilding damaged storage systems and recovering data from offsite



• Applications recovery team training primarily consists of skills used in normal operations

• Data management team training focuses on rapid data restoration and recovery from backup

• Vendor contact team training focuses on methods of obtaining resources as quickly as possible

• Damage assessment and salvage team training primarily consists of hardware repair skills that enable team members to determine if items are repairable or not



• Business interface team training includes communication skills and mechanisms for assisting with routine needs

• Logistics team training includes training in purchasing and procurement and providing rest and comfort for other workers


DR Plan Testing and Rehearsal

• Testing of the plan and the training and rehearsal of the plan can overlap

• Testing can involve several levels of assessment:– Employee self-assessments– Peer evaluations– Formally appointed internal assessors– External certification or accreditation groups

• Classroom training should come first before actual rehearsals


DR Plan Testing and Rehearsal (continued)

• Testing strategies include:– DR plan desk check: individual review of plan– DR plan structured walk-through: group exercise– DR plan simulation: each individual works

independently– DR plan parallel testing: act as if the disaster had

occurred but do not interfere with normal operations– DR plan full interruption: act as if disaster had

occurred, and perform all steps including data recovery

– DR plan war gaming: few tools available for this in the private sector


Rehearsal and Testing of the Alert Roster

• Alert roster must be tested more often than other plan components due to employee turnover

• Quarterly testing is recommended• Alert message contains just enough information to

allow employees to determine which part of the DR plan to implement

• Auxiliary phone alert and reporting system: automated system for activating the alert roster

• You are never completely ready for a disaster• Key skills to retain from rehearsals are flexibility,

decisive decision making, and professionalism


Disaster Response Phase

• Response phase: the phase associated with implementing the reaction to a disaster

• Response phase focuses on controlling or stabilizing the situation for the purposes of:– Protecting human life and well-being– Limiting or containing damage to facilities and

equipment– Managing communications with employees and other

stakeholders


Recovery Phase• Recovery phase:

– Initiates the recovery of the most time-critical business functions

– Focuses on getting up and running as quickly as possible, even in degraded mode; less critical operations must wait for the resumption phase

• Primary goals of the recovery phase:– Recover critical business functions– Coordinate recovery efforts– Acquire resources to replace damaged or destroyed

equipment or materials– Evaluate whether to implement the business

continuity plan


Resumption Phase

• Resumption phase: focuses on non-critical functions

• BIA should guide in the prioritization of critical and secondary functions

• Goals of the resumption phase:– Initiate implementation of secondary functions– Finalize implementation of primary functions– Identify additional needed resources– Continue planning for restoration


Restoration Phase• Restoration phase: the final phase of disaster

recovery• Primary goals of restoration phase:

– Repair all damage to primary site or select or build a replacement facility

– Replace damaged or destroyed contents of primary site including supplies, equipment, and material

– Coordinate relocation from temporary offices to primary site or suitable new replacement facility

– Restore normal operations at primary site, beginning with critical functions, then secondary operations

– Stand down the DR team and conduct the after-action review


Repair or Replacement

• Two possibilities in restoration phase:– Reestablish operations at primary site– Establish operations at a new permanent site

• Reestablish operations at primary site:– Must be able to rebuild damaged facilities– May need to relocate administrative functions to

provide space to the operational functions while rebuilding is underway

• New permanent site options:– New location– Complete rebuild on site of destroyed facilities


Restoration of the Primary Site

• After physical facilities are rebuilt, the contents must be replaced, including:– Office furniture, PCs, photocopies, filing systems,

office supplies, etc.• Must assess what will be covered by insurance and

service contracts


Relocation from Temporary Offices

• Transition back to the primary site must be carefully coordinated to minimize additional disruptions to business functions

• If data management functions must move, may want to use a movement coordinator to plan the relocation of personnel, equipment, materials, and data back to the primary site


Resumption at the Primary Site

• Must reestablish all normal operations, including tertiary operations that may have been suspended due to relocation, such as:– Managing employee benefit packages– Employee training and awareness programs– Organizational planning retreats and meetings– Routine progress meetings and reports– Long-term planning activities– Research and development activities


Standing Down and the After-Action Review

• Standing down: the deactivation of the disaster recovery team, releasing individuals back to normal duties

• After-action review provides a method for management to obtain input and feedback from each group represented in the team

• AAR log serves as legal and planning record and tool for future training

• Official report should include AAR and reports from individual teams


Summary

• An organization should operate on the premise that it is only a matter of time until a disaster strikes

• 5 phases in the DR plan: preparation, response, recovery, resumption, restoration

• Goals of DR and business resumption planning: eliminate or reduce potential for injuries or loss of life, stabilize the effects of the disaster, implement the DR plan based on type and impact of disaster

• Recovery phase attempts to recover the most critical business functions immediately


Summary (continued)

• Resumption focuses on the remaining unrestored functions

• Restoration seeks to:– Repair all damage to primary site or arrange a

replacement facility– Replace all damaged or destroyed contents– Coordinate relocation from temporary back to primary

site– Restore normal operations at primary site– Stand down the DR teams and conduct the AAR

principles of incident response and disaster recovery chapter 8 disaster recovery: operation and...

Documents