it contingency implementation guide 1 running head: it contingency implementation...

IT Contingency Implementation Guide 1

Running head: IT Contingency Implementation Guide

IT Contingency Planning & Disaster Recovery Implementation Guide

David Goldsmith

University of Advancing Technology


Abstract

One major issue facing companies that utilize technology is the fact that a disaster holds the

possibility of wiping out a majority of the companies electronic assets (data, hardware, etc). As a

result of this it is important for a company to have a well documented contingency plan of what

to do in the event of a disaster or other disruption. This document is a basic guide to

implementing the recommendations found in the Special Publications 800-34 document by the

National Institute of Standards and Technology entitled: Contingency Planning Guide for

Information Technology Systems.


Overall Summary

All companies are open to the possibility of a natural or man made disaster. In the event

that one does occur, companies that have a contingency plan (commonly known as a disaster

recovery plan) are more likely to survive the disaster intact and return quickly to business.

Companies that do not have a plan in place are more likely to suffer severe losses, and even to go

out of business.

One entity that is well aware of the possible problems that could be caused by a disaster and of

the importance of having a plan ready for use is the National Institute for Standards and

Technology (N.I.S.T.). Aware of the special issues faced by companies who use a great deal of

technology they developed a special publication entitled: SP 800-34 Contingency Planning

Guide for Information Technology Systems. The intent of the document is to assist IT personnel

in preparing for disaster. This intent of this document is to a) provide general information that

can be used to build an IT Contingency Plan, and b) to provide simple instructions that will

enable a company to build a basic plan; mapping out both what form of alternate sites to use and

actually writing the plan. As well as simple instructions and recommendations that can be used to

ensure proper functioning of IT related computers.


Plan Types

While the term “disaster recovery plan” is often used as the only term for a guide dealing

with how to recover from a disaster, there are actually a number of different documents that go

into disaster recovery; each one with a very specific scope. These include Business Continuity

Plans, Crisis Communications Plans, and the ever mentioned Disaster Recovery Plan.

Different Plan Types

• Business Continuity Plan – The overall purpose of this plan is to provide details and

procedures for keeping the business operating while it recovers from a disaster or other

business disruption. When it comes to details on IT related issues they are focused only

on supporting business processes.

• Business Recovery Plan – Similar to the Business Continuity Plan this plan gives

procedures to follow for recovering the operations of a business following disaster. It also

does not have much IT related support.

• Continuity of Operations Plan – This plan provides the necessary procedures to allow an

organization to remain running for at least thirty days at an alternate site. As it is written

on a company-wide level it rarely is IT focused.

• IT Contingency Plan – This provides procedures for recovering applications and general

support systems by IT staff. It is designed to assist IT personnel and as such is much

more focused on IT than other documents.


• Crisis Communications Plan – This document provides procedures and methods for

keeping personnel and the public apprised to the status of the company during and

following a disaster. It is not focused on IT related issues.

• Cyber Incident Response Plan – This plan, which typically focuses on the security issues

inherent to IT, provides methods to detect respond to and limit issues caused by cyber-

attacks.

• Disaster Recovery Plan – This plan provides procedures and techniques to recovery the

capabilities of a company at an alternate site following a disaster. While it usually has a

great deal of information regarding IT related issues it is typically only for long-term

disruptions.

Recommended Documents for IT to Focus On

While all of the documents listed above are important documents and will aid the company in

recovering from a disaster, the IT Contingency Plan is the most important to returning IT

systems to full functionality following a disaster, as a result it is recommended that the IT

staff focus on creating a full and detailed IT Contingency Plan in a timely manner before

disaster occurs.


Alternate Sites

Occasionally a disaster will force the business to relocate to an alternate site for a period

of time. Having an understanding of the various types of alternate sites, as well as a plan for how

to move into one is important for a company in dealing with disasters. This includes the IT staff

of the company because often times computer systems and data will be needed to keep the

company functioning at the alternate sites. Below is a brief definition of each site, followed by a

table demonstrating the abilities and costs of each alternate site as they relate to IT functions.

Site Descriptions

• Cold Site – This typically consists of a facility with adequate space and infrastructure to

support the IT System. It does not contain IT equipment or office automation equipment.

• Warm Site - Similar to a cold site, this site contains some of the hardware and software

and other equipment needed to maintain IT functionality. It is usually prepared in

advance and as a result can be activated faster than a cold site.

• Hot Site – This is a fully functional office space which is often staffed 24 hours a day. It

contains all necessary equipment and software and can be activated as soon as a

contingency plan is started.

• Mobile Site – This is self-contained and transportable and contains the equipment needed

to meet necessary system requirements. It can be moved if needed to an alternate

location.

• Mirrored Site – This is the most expensive form of alternate site, but also allow for the

highest degree of data and hardware availability. All information passing into the primary

location (which may be unavailable during a disaster) is also passed into the mirror site.


Alternate Site Criteria Selection

Site Cost Hardware / Equipment Telecommunications Setup Time Location

Cold Site Low None None Long Fixed

Warm Site Medium Partial Partial/Full Medium Fixed

Hot Site Medium/High Full Full Short Fixed

Mobile Site High Dependent Dependent Dependent Not Fixed

Mirrored Site High Full Full None Fixed

(National Institute of Standards and Technology, 2002)

Recommended Form of Alternate Site

While all the alternate sites shown above are designed with IT in mind, I would

personally recommend that a mid to large sized company with moderate technology needs

prepare for disasters by planning for a hot site. While the costs are higher than some of the other

sites, the alternate site still allows for some flexibility in preparing an alternate site. It still,

however, provides a reasonably quick activation time to allow the company to be functional

again in a small amount of time. If a company does a great deal of IT related work, such as a

hosting company; then a mirrored site would be the best type of alternate site to use because in

the event of an emergency the mirrored site allows the company to continue hosting data from

the alternate site while simultaneously recovering the primary site. Additionally, the mirrored

site can be used in the ordinary course of business.


Roles and Responsibility Reference

In addition to the actual hardware and software used in IT functions, one of the most

important assets of any company is the employees who work for it. In the event of an emergency

it will be up to the employees to ensure that any alternate and backup equipment functions

properly. To ease the burden on employees it is recommended that they be split into groups, each

of which will support a specific feature of the IT systems. A full listing of groups is available in

the original document, but to reiterate from my point of view, the most important groups and

roles (i.e. the ones that all companies should have regardless of size) are:

• Senior Management Official – This is the individual who ensures that all of the other

contingency groups perform there tasks on time, and in the proper manner. The overall

success or failure of the plan rests at least partially on their shoulders.

• Damage Assessment Team – The importance of this group cannot be stressed strongly

enough as it is this team which examines the primary site following a disaster and

determines whether business functions are still available or if the IT staff should begin to

transfer operations to an alternate location.

• Server Recovery Team – This team is tasked with recovering hardware and data (if

necessary) from the servers following a disaster.

• Database Recovery Team – This team is tasked with recovering any needed information

from company databases following an emergency.

• LAN/WAN Recovery Team - This team ensures that local and wide area networks are

functional following a disaster, whether it is at an alternate site or the primary site.


• Administrative Support Team – This team provides necessary support for administrative

duties and overall operations.

• Network Operations Recovery Team – This team ensures that the company network is

functioning properly following a disaster. If it is not, then they are tasked with bringing it

back on line as quickly as possible.

• Media Relations Team – This team ensures that the media is kept up to date as to the

status of the company following an emergency; this prevents rumors from starting and

allows the public and other company personnel to feel more at ease with the situation.

• Legal Affairs Team – This team

Note: While not all personnel provide direct IT support, they are still essential figures.

.


Contingency Planning Guide

All information given up until this point has been summary and reference information.

Now comes the important part of this document, which is a step by step guide as to how to build

an IT Contingency Plan. This plan is designed to address IT related issues, and as such is

somewhat narrow in its scope; this guide should NOT be the contingency plan created by a

company; and likewise while this guide attempts to touch on all important topics there may be

procedures that are not mentioned here which should still be documented in a contingency plan.

All plans should be submitted to IT administrators to ensure that all necessary procedures are

outlined.

This plan consists of two major parts. Part one is discussion of how to create a basic

contingency plan for a company (focusing on IT). Part two is how to create an IT Contingency

Plan, and focuses on dealing with technology related issues.

Writing the Contingency Plan

Develop Policy Statement

The policy statement is the department level document which describes the overall intent and

purpose of the disaster recovery plan. I recommend that before it is accepted as full policy

that it be reviewed by all departmental leaders to ensure that it is supported by them. In

following the original document, I recommend that the following questions be answered

about the company that is building the plan.


1A. Policy Statement – Determining Information

a. Determine Roles and Responsibilities – Straight forward, this means that a clear idea

should be made of who is responsible for what in the creation and maintenance of the

contingency guide.

b. Determine the scope of the document – This ensures that the policies created in

making the contingency guide are appropriate.

c. Determine the necessary resources required by the guide – if there are any specific

pieces of software, hardware they should be mentioned in this document; as well as

listing the manpower required to make the document.

d. Determine training requirements – A clear indication should be made as to how

employees will be trained in the procedures within the contingency guide as well as

how often they will need to repeat the training (if necessary). I would recommend that

all employees be tested at least twice a year to ensure that they are up to date on the

procedures.

e. Determine the testing schedule for the document – Since a plan is not very useful if

nobody knows how to implement it the policy statement should address how often the

document will be tested to ensure that it still fits the needs of the company. I would

recommend that the plan be tested at least once a year, if not more.

f. Determine how the plan will be maintained – Technologies change, as do employees,

and keeping current lists of both is important to the plan so it is important to decide

how often the plan will be revised and changed, as well as who will be responsible for

making changes.


g. Determine how often data will be backed up, and how backups will be stored.

1B. Policy Statement – Write the Statement

Once the data above has been chosen write the statement; it should typically be around

one to two paragraphs and address all of the information that has been gathered.

Identify Critical IT Resources

1. Create a list of all IT related assets, including hardware, software and personnel. Also list

the functions of each item (whether it be a mechanical function or one that is performed

manually by a staff member) – This is important for several reasons. One reason being

its use in this contingency plan; another useful function of the list is that it can be used in

other emergency plans created by the company.

2. Rank the listed assets (keeping in mind their functions) according to importance; keeping

in mind what would be most needed to keep the company running in a disaster situation.

– In a disaster this information could prove very valuable because it will let the company

know what systems need to be brought back online quickly; a printer might not be

incredibly important but a router just might be.

Identity Disruptions and Allowable Outages

IT systems tend to be prone to disruption and outages, even in a perfectly designed

system. Determining possible chokepoints in the network and associated hardware and

software that could cause the system to fail will allow a company to plan for that possibility

and develop failsafe methods.

1. Using the IT Resource list created in the last step, determine the estimated amount of

time that could be lost if critical equipment were to fail. Also determine other


possible problems that could result from such a failure – This information is valuable

because as stated above it gives the company a chance to prepare for problems. If for

instance the company finds a damaged router could cripple the network they may opt

to include an additional router to prevent the problem from occurring.

2. Determine the amount of time that a system could be offline while causing minimal

damage to the company. Responding quickly to problems can be expensive;

determining how long a particular piece of equipment could be offline without

problems can allow for other more important pieces of equipment to be checked.

3. Document your decisions

Develop Recovery Priorities

1. Again, using the information gathered in the previous step (outage times and

allowable outages) determine which systems should be focused on in a disaster,

and in what order they should be checked. - Setting priorities for how these

systems should be checked can help the disaster recovery team in recovering from

a disaster quickly and with less overall cost.

2. Document your decisions

Identify Preventative Controls

Preventative controls are techniques, strategies, and hardware / software that can help

to lessen or eliminate problems caused by disasters. These can include backup power

supplies offsite data storage and emergency generators.

1. Determine which resources are likely to require backup supplies; this can include

needs for emergency power, storage or any other functions.


2. Determine the proper method in which the supplies can be delivered (i.e. emergency

power might be provided by an emergency generator).

3. Purchase the necessary supplies

4. Place the supplies in the proper location so that they can be reached quickly in an

emergency. Proper location will vary depending on the equipment. – It is important to

make sure that the supplies are placed in a location which will be accessible in an

emergency (it is not possible to ensure that they are, however) so that the supplies

can actually be used. If for instance a generator is in a blocked off area it is

essentially useless in an emergency.

5. In the contingency plan document the equipment, what it is used for, and where it is

located

Develop Recovery Strategies

These provide means to restore IT to full operations as quickly and effectively as

possible following a disruption due to any number of reasons (whether they be natural

disasters or man made)

1. Determine order of recovery – This is simply choosing which systems will be worked

on first, to ensure that the overall system functions properly. These decisions were

already documented in the section entitled “Develop Recovery Priorities” above.

2. If specific vendors are being used for services you should document their contact

information. I recommend that when documenting vendor information at the very

least the company name, company representative, phone number, address and

service contract number be documented.


Choose a Backup Method

In an IT environment data is essential. Losing data due to corruption of data, or

damaged disks can be devastating to a company which relies on the information. Because of

this it is very important to backup data according to a specified plan, and to ensure that the

data that is backed up is stored properly.

1. Examine how much data the company stores. Then use that information to decide on

the proper method of backing up data. Tape backups are a very common method of

backing up data, and provide a reasonably high amount of storage for a modest price.

2. Determine how often the data will be backed up. This is very important, if it is not

backed up enough than important data could be lost. I recommend that data be backed

up at least three times per business week. This would mean that backups would be

created on Monday, Wednesday and Friday.

3. Determine if backups will be stored on or off site. On site storage means that the data

is available quickly in the result of corrupted or lost data, it also means, however, that

in the event of a large scale disaster (such as a flood or fire) the data could be lost. My

personal recommendation would be that companies should store there data offsite.

While the cost might be higher, the long-term benefits seem to outweigh the costs.

4. Document how often data will be backed up, to what media it will be backed up to,

and how the media will be stored to ensure that the backed up data is not corrupted or

destroyed itself.


Choose an Alternate Site

While not a requirement, it is a good idea for a company to have an alternate site, and

a plan for activating it. This site will be used in the event that the primary location becomes

unavailable during a major disruption or disaster

1. Determine if an alternate site is needed – There is no simple criteria for determining

if an alternate site is needed, but things that should be taken into consideration

include the cost of keeping and equipping an alternate site, the losses that might be

incurred during a disaster if the company does not have an alternate site, and the

possibility of data being lost in an emergency if an alternate site is not used.

2. Assuming that an alternate site is being used, choose the type of alternate site (see the

section regarding alternate sites at the top of this document. As I stated there I

recommend that for IT purposes a hot site be used as it allows for quick resumption of

business as it relates to IT).

3. Equip the Alternate Site, or Plan to supply it – This is simple enough, based on the

needs of the IT staff at the company purchase extra servers, software, and other items

that an alternate site uses.

4. Document the site setup – So that the site can be activated as quickly as possible it is

important to document what is located at the alternate site as well as important phone

numbers.

• Create Staff Lists & Contact Plans – I recommend that these plans include the employees

name, office location, telephone number, work hours, job function, and training. It should

also include the duties that the employee has in the event of the contingency plan being

activated.


Testing & Servicing the Plan

Exercises and Recommendations

Once the plan has been created it is important to test it and ensure that the plan is kept

up to date with staff and technology changes. Making sure that this is done can help to

prevent problems in the future when attempts are made to use the plan and it refers to

employees who no longer work at the company or technology that is no longer used.

1. Determine elements to be tested – This will vary from company to company but

may include such things as ensuring that a backup is performed or testing how

network traffic could be switched to another location (alternate site). I would

recommend that simple IT tasks be tested on a regular basis to ensure that

employees know the procedures as well as to ensure that the equipment is

functioning properly.

2. Determine Test Frequency – Tests should be performed regularly. I would

recommend that depending on the IT function tests should be performed at least

twice per year to ensure that the plan works with the current technology.

3. Test the Plan – Actually test the plan according to the set schedule, employees

should then be allowed to make recommendations and suggestions as to how the

plan might be improved. I would recommend that employees have the option to

make these suggestions in a confidential manner as to avoid feeling intimidation

from managers. They should be allowed to comment on any aspect of IT.


How to Update the Plan

As time passes the technology used at the company will change, as will the

employees working at the company. It is important to ensure that these changes are

documented so that the plan may be used.

1. Determine the frequency of the updates – This is simple enough, and ensures that a

time is set when the plan will be checked and updated. For a large scale IT company I

would recommend that the plan be checked at least every six months if not more,

because the chances of new employees and new systems seems quite high.

2. Assign employees to perform updates – Once an update is needed employees should

be given the task of checking to ensure that the data is correct. If the data needs to be

updated, it should be.

3. Update the plan – As the employees assigned to monitor the plan for changes find

technologies, employees, and other things that have changed in the plan they should

update the plan as needed

4. Document changes – Just as important as updating the plan is documenting what

changes have been made to it, so that people at the manager level can immediately

see what has changed.

Activating and Using the Plan

The activation and use of this plan is somewhat beyond the scope of the document, but as

follows are some basic recommendations and procedures for how the plan may be activated.

How to Activate & Use the Plan

1. If there is risk to human life immediately evacuate the area


2. Once the threat to individual life is negated the plan should be activated by a

manager. – They can do this by documenting the activation of the plan and telling

the necessary employees (listed in the plan) to begin following contingency plan

procedures.

3. Necessary employees should begin to contact other needed employees as well as

begin the process (if needed) of setting up alternate sites. Employees should also

begin to assess possible damage and begin restoring data from backups if needed.

Sample Recovery Procedure

Samples of procedures that could be used to coordinate recovery from a disaster might include

Server Team

• Ensure that the network is plugged in by checking behind the computer

and ensuring the cable is fully plugged into the Ethernet port

• Open Server Operations Software and ensure that all necessary services

are running

1. Check that DNS is running

2. Check that Port 80 is open (or whatever other ports are needed)

3. Check that Active Directory is functional and allowing users to log

into and out of the network

4. Etc.


Technical Contingency Planning

Below is what I would consider to be the minimum amount of documentation that should

be prepared for a medium sized business with moderate IT use. Depending on the company their

may be fewer or additional procedures to be performed. As a side note it should be mentioned

that the topics listed below are all based on the NIST Special Publication. Some of the individual

steps are based on experience and other sources.

Servers

Document System and App Configurations

• Whenever a piece of software is installed the following should be documented so that

if the system needs to be rebuilt it can be with a minimum of hassle and guess work.

Application Name and Function – This seems like a simple thing to

document and it is, but doing so will allow you to quickly determine

what software each server is running.

Installation Location – If a patch is needed it will help to have the

location that applications were installed to so that it can be done so

quickly. I can say from experience that this saves a lot of time in a

large situation.

Any Configuration Changes Made to the base install. – If any changes

were made they should be documented so that if the employee who

installed the software leaves, or the software has to be worked on by

another employee the configuration is known.

Record the Vendor information. - This will often consist of examining

the materials that come with the server. The serial number of the


system (location varies by vendor) should be documented because

when calling in for technical support it is often required. Additionally,

the contact number of the vendor should be recorded. At the very least

the following should be documented

Server Configuration Number – This is if the computer is purchased

from a company like Dell or Gateway, you should record the model

name and configuration number so that if additional servers are

purchased you can order identical systems.

Purchase Numbers – This will allow you to document what has been

ordered and when, so that it can be accounted for when needed.

Vendor Contact Information – In case of a problem I think that you

should have the contact information readily available so that the

problem can be resolved quickly.

Computer Serial Number – When calling for technical support it is

important to have the serial number available in case it is asked for by

the company you purchased the system from

Standardize Hardware

• All servers should utilize similar hardware – This is to ensure that there is no

compatibility problems between the various servers run by the company. It also

makes it easier to replace broken components by making it easy to know exactly what

brand and model each component is.

Implementation Notes: Many companies (even small businesses) have servers, and it

seems that ensuring that they are running properly is something that is commonly done.


A problem that I have seen which I have tried to give steps to avoid (in my step by step

instructions above) is that equipment will be purchased at times without regard to other

equipment, I think that this is a huge mistake that should be avoided as there is always

the possibility that there will be incompatibilities between the server hardware.

Backup Data

If the media to be backed up to is a DVD follow the following procedure

o Insert DVD into the DVD burner.

o Open Backup Software (or DVD Burning software)

o Select data to be backed up

o Burn data to disc

o Remove disc and label it

o Store disc in proper location until it is needed. (preferably offsite)

o Log the backup as completed, while also documenting what files were backed

up.

If the media being backed up to is an online server follow the following procedure

o Log into online server using appropriate credentials

o Select data that is to be backed up to the server

o Copy data to the server using the tools provided by the hosting company (or if

using simple FTP by dragging and dropping the files)

o Document which files were backed up to the server

Implementation Notes: I have found that while data is often considered very valuable it

also tends to be treated very poorly. In too many situations data is either not backed up,

or if it is backed up it is stored in the same location as the computer containing the data.


For the purpose of dealing with a disaster I think that this is rather stupid because if a

disaster takes out the system containing the data it will likely also take out the backed up

data; therefore I strongly recommend that any backups be placed offsite.

Web Sites

Document the Site

• Hardware & Software: The hardware and software used to create the website should

be documented to allow for future editing if needed.

• Document Web Server Settings – The physical location as well as the IP address of

the server hosting the website should be documented in the contingency plan. This is

important in a disaster because knowing which servers in which locations are hosting

which websites will better allow the sites to be kept online or brought back online as

quickly as possible

1. Check the software version by loading the editor software that is used for web

editing. The software version can usually be found by clicking the “about”

button under the help menu in the application toolbar.

2. Fill in the appropriate form to document the server information with the make

and model of the system hosting the website. Also write down and important

information such as usernames and passwords.

3. Document any changes to the web host that are not standard configurations.

This could include custom scripts that are installed, and additional scripting

languages that might be available on the server.


LAN

Choose a Topology & Network Architecture

1. First determine the number of computers that will be on the network

2. Next determine what security is needed on the network. This could include various

forms of logging, as well as preventing files from being accessed or changed.

3. Using the information above decide whether the network should be peer to peer or

server based. I would recommend that server based networks should be used as it will

often provide faster service because one computer can be designed to serve

information to all of the other computers on the network.

• Document and Map the Network

1. Write down the computer name and location of all computers on the

network. This could take a while if it is a large network.

2. Write down the location and name of all routers, servers, and other

network based hardware other than computers

3. Separate the compiled data into functional groups, such as ordering

by building, or by department.

4. Draw out the various connections within the network, for instance all

of the computers in one office might be connected to a single server,

which is connected through a firewall, to the internet. Doing this will

enable you to quickly pinpoint possible problems, such as security

risks and chokepoints.


5. Log and document all IP addresses, subnets, and other network

related configurations in the contingency plan so that the information

will be immediately available during a crisis.

General Note

This contingency plan guide is written for the average business in mind, including small

to medium sized businesses. For this reason mainframes are not written about because they seem

somewhat out of the scope that this paper is designed for. The Special Publication does contain

contingency planning information for mainframe systems and can be consulted if the business

implementing the plan needs to.


Final Notes

This implementation plan has been designed to provide basic information that any

company creating a contingency plan should be able to use. It has included step by step

instructions for documenting and implementing the plan, as well as basic information regarding

the various types of plans that can be created as well as the types of alternate sites that could be

used. Any plan created by a company will likely contain some or all of the documentation items

that have been discussed. It is important to remember, however, that there may be additional

pieces of documentation that are required in the plan.

Regardless of its importance, a contingency and disaster recovery plan for a company is

only a small piece of the overall issue. Employees must be made aware of the importance of

contingency planning and be well trained in how to respond to events. If employees don’t know

how to use it, or if the plan is not made available to them, then even the best plan will be doomed

to failure.

To briefly summarize an important point that I have tried to make, I consider proper

documentation to be essential to a company surviving a disaster. As a result I have tried to

continuously stress this throughout the document, which has been a steady mix of both the

recommendations made in the special publication and my opinions on the matters involved,

where possible I have tried to distinguish between my the official recommendations and my

personal opinions by putting my opinions in italic font.


References

1. National Institute of Standards and Technology. (2002). Contingency Planning Guide for

Information Technology Systems, (SP 800-34). Washington DC: U.S. Government

Printing Office.

2. Ali Pabrai, U. O. Contingency Planning and Security. from

http://www.ihatoday.org/services/business/contplanpaper.pdf