it continuity of operations

62
IT C ti it f IT Continuityof Operations Operations Mi i i th G Bt Y Minimize the Gaps Between Your Recovery Capabilities and Functional Requirements of the Enterprise Requirements of the Enterprise Presented By Jeff Roseman Director, IT Infrastructure Patterson Medical March 10, 2009

Upload: jeff-roseman

Post on 29-Nov-2014

1.007 views

Category:

Technology


0 download

DESCRIPTION

 

TRANSCRIPT

Page 1: IT Continuity of Operations

IT C ti it fIT Continuity of OperationsOperations

Mi i i th G B t YMinimize the Gaps Between Your Recovery Capabilities and Functional 

Requirements of the EnterpriseRequirements of the Enterprise

Presented ByyJeff Roseman

Director, IT Infrastructure ‐ Patterson Medical

March 10, 2009

Page 2: IT Continuity of Operations

Semper Paratus: Are You Ready?Semper Paratus: Are You Ready?

• Annual Budget for Disaster Recovery• Annual Budget for Disaster Recovery & Business Continuity?

• Experienced a Disaster?• Experienced a Disaster?• Declared a Disaster in Last Year?• Disaster Recovery Plan?• Updated DR Plan in Last Year? Let Me 

S Y• Tested DR Plan in Last Year?• Business Continuity Plan?

See Your Hands

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 2

y

Semper Paratus (Latin: Always Prepared; U.S. Coast Guard motto)

Page 3: IT Continuity of Operations

Disaster Recovery vs.Business Continuity

• Disaster Recovery (DR)• Disaster Recovery (DR)– Evolved from Data Center operations– Strictly a “technical” solutionStrictly a  technical  solution – Over time, it was realized that recovery of the platforms did not mean recovery of the business

• Business Continuity (BC)– Addresses those “non‐technical” functions                       th t i d t t b ithat are required to restore business

– Not just actions taken during a disaster– An enterprise‐wide project not just IT

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 3

– An enterprise‐wide project, not just IT

Page 4: IT Continuity of Operations

2008 AT&T BusinessC i i S dContinuity Study

• One in five businesses does not have a businessOne in five businesses does not have a business continuity plan in place

• For the third year in a row, the survey finds that nearly 30 % of U S businesses don't considernearly 30 % of U.S. businesses don t consider business continuity planning a priority

• Six out of 10 companies have made some type of p ypbusiness change in the past year, but only 28 % updated their plans

• One‐fourth (28 %) have insufficientOne fourth (28 %) have insufficient                           storage space

• The vast majority (79 %) have special             t f i ti ith

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 4

arrangements for communicating with                         key executives during a natural disaster

Page 5: IT Continuity of Operations

Every Business Needs a PlanEvery Business Needs a Plan• A generic DR plan is better than nothing, but it may stress elements thatA generic DR plan is better than nothing, but it may stress elements that 

are less important to your business, or worse, leave out critical aspects• Every organization, regardless of size or industry should have a Business 

Continuity Plan (BCP).o u y a ( )• Needs vary from business to business and a good availability plan should 

be designed for the individual business's needs– Service Delivery / Call Center / eCommerceService Delivery / Call Center / eCommerce– Manufacturing / Distribution– Multi‐Site & International Operations

• A Business Continuity Plan is the least expensive insurance• A Business Continuity Plan is the least expensive insurance                       any company can have (especially for small companies, as                            it costs next to nothing to produce)– Treat it as an investment not an expense

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 5

Treat it as an investment not an expense– Many businesses NEVER recover from a serious incident

Page 6: IT Continuity of Operations

Taking Your Business Continuityto the Next Level

• It is a huge mistake to 

EnterpriseAvailability

gdevelop a business continuity plan and not make it integral with your daily

BusinessContinuity

with your daily business operations

• Availability planning is an investment in the                 

DisasterRecovery

Days                          Hours              Minutes

continuing operations of the business

• Transform your Business Continuity y

Data‐Centric     Business Function‐Centric

Functionality/Cost/Recovery Time Objective — RTOBusiness Continuity Plan  into an Enterprise Availability Plan

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 6

Page 7: IT Continuity of Operations

Enterprise Availability PlansEnterprise Availability Plans

• The ProcessThe Process– Understand Your Enterprise Requirements – Prioritize and Map Enterprise Requirements – Minimize the Gaps between Requirements and Capabilities

– Test and Modify the Plan to Prevent Future GapsTest and Modify the Plan to Prevent Future Gaps • The Results

– Incident Management Plan – Focused on Crisis Management

– Business Availability Plan – Focused on Work Area Recovery

– Technology Availability Plan – Focused on Technology Recovery

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 7

Page 8: IT Continuity of Operations

U d t d YUnderstand Your EnterpriseEnterprise 

Requirements

Page 9: IT Continuity of Operations

Document Past Downtime EventsDocument Past Downtime Events• A list of known downtime events  Common Downtime Eventsand their associated costs will 

help you identify common problems and develop solutions that will improve availability 24/7

Common Downtime Events(My Personal Stream of Misfortune)

that will improve availability 24/7– Power Loss– Communication Outage– Hardware FailuresHardware Failures– Scheduled Maintenance

• Your physical location can have a lot to do with it

Hardware FailurePower Outage

– Multi‐tenant Spaces– Construction– Weather Patterns

gWeather / FloodMalicious ActsFire / BuildingSoftware Failure

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 9

Software FailureOther

Page 10: IT Continuity of Operations

Identify Systems And Recovery Procedures

• Disaster Recovery Plan (You already have one right?)• Disaster Recovery Plan (You already have one, right?)• How‐To Guides & Instructions• Technology Profile• Technology Profile

– Team Members & Skill Sets– Systems Diagram

See Appendix for Technology

Profiles Examplesy g– Hardware Inventory– DataComm Inventory

Profiles Examples

– Critical Applications– PBX Configurations– Vendors/Partners

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 10

Vendors/Partners– Vital Records

Page 11: IT Continuity of Operations

Define Business FunctionsDefine Business Functions

• Scope of Business Operations• Scope of Business Operations– Locations (Single, Multi‐Site, International operations)– Departments / Teams (How is the company organized?)Departments / Teams (How is the company organized?) – Processes / Tasks (What does the department do all day?)– Schedules (Period Close, Peak Seasons, etc.)– Dependencies (Order processing affected by credit dept.)

• Organization charts and process flow diagrams      can really help IT understand the business.

• Are there manual work arounds?

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 11

Page 12: IT Continuity of Operations

Identify Critical Business RequirementsIdentify Critical Business Requirements

• Document internal key personnel and• Document internal key personnel and functions (who is their backup?) Id if h l• Identify who can telecommute

• Document external contacts• Document critical equipment• Identify critical documentsIdentify critical documents• Identify contingency equipment optionsId tif ti l ti

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 12

• Identify your contingency location

Page 13: IT Continuity of Operations

Document Key Internal Personnel and Functions 

• Consider which job functions are critically necessary,Consider which job functions are critically necessary, every day, not just in an emergency 

• Think about who fills those positions when the primary job holder is on vacationprimary job‐holder is on vacation

• These are people who fill positions without which your business absolutely cannot function – make the y ylist as large as necessary, but as small as possible

• Decide what non‐critical employees                             should do in the event of a disastershould do in the event of a disaster.                              If there is no place for them to work,                                will they be in the way of more critical                   business functions?

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 13

business functions?

Page 14: IT Continuity of Operations

Identify Who Can TelecommuteIdentify Who Can Telecommute• Some people in your company might be perfectlySome people in your company might be perfectly capable of conducting business from a home office

• Find out who can and who cannot work remotelyh d h l ff• You might consider assuring that your critical staff 

(identified in Step 1) can all telecommute if necessaryy

• This is an easy piece that you can build                          into your daily operations

• Key personnel who cannot telecommute• Key personnel who cannot telecommute                    will likely need a workstation at your                   contingency site

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 14

Page 15: IT Continuity of Operations

Document Critical External ContactsDocument Critical External Contacts 

• Your business partners and vendors can really make or break p yyour recovery– Build a contact list that includes a contact information and a 

description services they providel d i li l lik h i– Include in your list people like the insurance company, attorneys, 

bankers, IT consultants, electricians...anyone that you might need to call to assist with various operational issues

– Don’t forget utility companies, municipal and community offices, the g y p , p y ,post office and FedEx/UPS. 

• Keep a list of key customers who you                                              will want to notify in an emergency

• Create a “Yellow Pages” of external                                     contacts by function and a “White Pages”                                  by name

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 15

Page 16: IT Continuity of Operations

Document Critical EquipmentDocument Critical Equipment • Personal computers often contain critical information (You doPersonal computers often contain critical information (You do 

have off‐site backups, don’t you?)• Some businesses cannot function even for a few hours 

without a Fax machine (i.e. 25% of orders come by fax)• Do you have special printers you absolutely must have? • What about security and encryption keys?• What about security and encryption keys?• Do you have hardware license dongles?• Don’t forget software – that would oftenDon t forget software  that would often                                      

be considered critical equipment especially                                 if it is specialized software or if it cannot                                     b l d

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 16

be replaced.

Page 17: IT Continuity of Operations

Identify Critical DocumentsIdentify Critical Documents • You need to have everything available that would beYou need to have everything available that would be 

necessary to start your business over again – Articles of incorporation and other legal papers– Insurance policies banking information building lease papersInsurance policies, banking information, building lease papers– HR documents, government mandated records, tax returns– Software Licenses, technical documents and source code

• Remember you might be dealing with a total facility loss• Remember, you might be dealing with a total facility loss• You keep copies of your DR Plan off‐site, why wouldn’t you do 

the same for your critical business documents?• Store PDFs of critical documents on a                                 

secure, off‐site server that you can access                                     via the Internet in an emergency

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 17

g y

Page 18: IT Continuity of Operations

Identify Contingency Equipment Options

• IT Equipmentq p– Where would you rent computers? – Who can provide equipment such as servers on very short notice?  

(i.e. CDW has same day service in Chicago)– Are there components with a particularly long lead time? What are 

the alternatives?• Telecom

Does your call center require special equipment?– Does your call center require special equipment?– Can your telecom partner provide you with a loaner?– What is the turn‐around time to set‐up a new phone                               

system?y• Other Equipment

– Can you use a business service outlet like Kinko’s for copies, fax, printing, and other critical functions? 

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 18

– Where would you rent trucks, air conditioners, generators, etc.? 

Page 19: IT Continuity of Operations

Identify your Contingency LocationIdentify your Contingency Location

• This is the place you will conduct business while your primary p y y p yoffices are unavailable– It could be a hotel, an adjacent vacant space,  or even someone’s 

home for a small businessld b h ffi l i– It could be another company office location

– Or a 3rd party site or mobile service like IBM or SunGard– Perhaps telecommuting for everyone is a viable option. D idi WHERE t d d th d f th b i• Deciding WHERE to go depends on the needs of the business– How much space do you need?– What facilities and services do you need?

Will the facility be available to you in a regional disaster?– Will the facility be available to you in a regional disaster?– What solution will get you back up and running fastest?

• Wherever it is, make sure you have all the                 appropriate contact information and include

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 19

appropriate contact information and include                              a map in your BCP

Page 20: IT Continuity of Operations

P i i i Y E iPrioritize Your Enterprise Requirements and MapRequirements and Map 

into Your Plan 

Page 21: IT Continuity of Operations

Conduct a Business Impact AnalysisConduct a Business Impact Analysis

• Reveal vulnerabilities and potential risks of• Reveal vulnerabilities and potential risks of worst case scenariosM i f fi• Measure impact on safety, finances, marketing, legal compliance, and quality assurance   

• Identify the organization’s business               unit processes and the estimated                 recovery time frame for each business unit

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 21

Page 22: IT Continuity of Operations

Identify Risks And ExposuresIdentify Risks And Exposures• We confuse the concept of risk—the probability of success or p p y

failure—with the concept of exposure— what is at stake• From a business continuity standpoint, your risks are what is 

likely to fail:y– Hardware failure (minimized with redundant hardware such as dual 

power supplies, RAID arrays, clustered servers)– Power failure (UPS and/or backup generator)

C iti l d t t t d i fi f f– Critical documents not stored in fire‐proof safe• Your exposure is what is at stake:

– Lost data and informationL f b i l d– Loss of business, sales and revenue

– Government penalties (IRS, SOX, HIPPA)• Understanding the risks and exposures of the business are 

fundamental in setting priorities

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 22

fundamental in setting priorities

Page 23: IT Continuity of Operations

Priority MetricsPriority Metrics

• Recovery Point Objective – RPO (data)Recovery Point Objective  RPO (data)– The acceptable level of data loss exposure following an unplanned 

eventThis is the point in time (prior to the disaster) to hich lost data can– This is the point in time (prior to the disaster) to which lost data can be restored, typically the last backup taken offsite

• Recovery Time Objective – RTO (business process)– The maximum acceptable length of time that can                            

elapse before the lack of critical business functions                          severely impacts the viability of the business

– This is the total time required to recover critical services  – Measured form the time of disaster to resumption of                           

critical operations (a.k.a – Maximum Allowable Downtime)

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 23

Page 24: IT Continuity of Operations

Prioritize Your RequirementsPrioritize Your Requirements• Are there existing Service Level Agreements (SLAs) in place?• Each business unit should rank their business functions based on most 

critical to the organization– Financial Impact

O ti l I t– Operational Impact– Reputation Impact– Regulatory Impact

• What are interdependencies between business units?What are interdependencies between business units?• Set Recovery Time Objectives (RTOs) for business functions and the 

applications they depend on– < 4 Hrs– < 24 Hrs– < 72 Hrs– < 7 days

7 14 days

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 24

– 7‐14 days– > 15 days

Page 25: IT Continuity of Operations

Prioritization ProcessPrioritization Process

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 25

Page 26: IT Continuity of Operations

Associate Business Functions With Applications & Data Sets

• Let the business set the recovery requirements, not the y q ,technical capabilities of the organization– RTO for business function drives RTO for systems– These gaps are natural– Gaps will force the technology to improve to meet                                  

the business needs• Mapping is complicated process

Wh b i i d d i ?– What are business process interdependencies? – What are hardware/software dependencies? – Organize applications in tiers based on business priority

(10 departments X 10 task) X 5 applications X 10 locations = a very– (10 departments X 10 task) X 5 applications X 10 locations = a very complex relationship

• You will quickly learn to– Isolate what are the key resources to recover and in what order

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 26

Isolate what are the key resources to recover and in what order– Build recovery strategies around those priorities

Page 27: IT Continuity of Operations

Build a Relationship DiagramBuild a Relationship Diagram

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 27

Page 28: IT Continuity of Operations

Evaluate ScenariosEvaluate Scenarios• What are the most likely scenarios you will face?

– Power Loss– Telecom Interruption– Hardware Failure

Severe Weather– Severe Weather• What are the most catastrophic scenarios?

– Regional Incidents– Short Term Loss of Facility Availability– Complete Facility Destruction– Global Pandemic

• As your business changes, so will the                                                       scenarios you facescenarios you face

• It’s hard to prepare for every conceivable disaster, so start of with the most common outages and work your way up

• These scenarios will be key in putting together your Enterprise Availability 

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 28

Plan

Page 29: IT Continuity of Operations

Develop Strategies for Minimizing Risk

• Do nothing; assume the risk• Do nothing; assume the risk• Revert to manual processing• Be self recoverable via multiple sites• Contract with a hot‐site/cold‐site vendor

• Contract a mobile recovery facilityContract a mobile recovery facility• Establish a quick‐ship agreement

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 29

Page 30: IT Continuity of Operations

There Is No "One Size Fits All" Solution

• It is normal for there to be a mixture of• It is normal for there to be a mixture of solution types within an organizationB ild l i d l ifi h• Build a solution and plan specific to each business function

• Assume business and technology requirements will evolve over time

• Think scalability• Think flexibility

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 30

Think flexibility

Page 31: IT Continuity of Operations

Building Your Enterprise Availability Plan

• So you have you a concept of what you will do in a disasterSo you have you a concept of what you will do in a disaster and tons of supporting documentation, NOW WHAT?

• It’s not enough to just throw it all in a big binder and say DONE!DONE!

• Start off with the Incident Management Plan– In a crisis, it the first step to recovery– Most “good practice” standards specify Incident Response       

planning now (Sarbanes‐Oxley, ISO, IEEE, ITIL, Payment                     Card Industry, etc.)Developing your response as an incident is occurring– Developing  your response as an incident is occurring                         probably will create more stress, cost more, take                                    more time and not be as well executed

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 31

Page 32: IT Continuity of Operations

Incident Management PlanIncident Management Plan• The Incident Management Plan (IMP) is your Management Playbook

– An enterprise‐wide action plan to help your senior management effectively and efficiently respond to an incident. 

– Your plan includes checklists of required activities, an explanation of roles and a definition of your resourcesy

• Incident Management Coordinator is the Quarterback– Management Action Team – Damage Assessment Team

R T– Recovery Team• Incidents usually require a time‐sensitive response –

if staff don’t know what to do, critical information and                               options may be lost

• Under stress it is good to know who is capable and                                   permitted to decide time‐critical issues

• Have an Incident Operations Hub (the “War Room”)                                   with specific outgoing channels and messages

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 32

with specific outgoing channels and messages

Page 33: IT Continuity of Operations

Incident Management OverviewIncident Management Overview

ControlCommand Communicate

Select Recovery

Plan

ImpactAssess-

ment

PlanExecution

Escalate to Mgmt

Incident Occurs

Mobilize Response

Pending Crisis

Post IncidentAnalysis

RecoveryMgmt.

• Establish command and responsibility for managing the incident then mobilize the response

• Determine the scope of the issue, set priorities, appropriateDetermine the scope of the issue, set priorities, appropriate response, and take control the overall recovery process

• Coordinate internal and external communication

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 33

Page 34: IT Continuity of Operations

Business Availability PlanBusiness Availability Plan• Business Availability Plan (BAP) is an action plan                                           

focused on maintaining the availability of critical                                              business processes when situations—ranging from                                             minor outages to major disasters—threaten to                                                disrupt them. 

• A detailed series of responses, checklists and                                           action steps to deal with situations that might                                           otherwise affect routine work activities

• Each business unit or department should have• Each business unit or department should have                                                  their own plan that meets their particular needs                                         and rolls up into the Enterprise Plan

• Individual plan also allow you to spread the work around and make it relevant to the business process ownersrelevant to the business process owners

• Added Bonus : You’re better prepared to meet regulatory, legal and internal audit compliance requirements, with thorough documentation 

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 34

Page 35: IT Continuity of Operations

Getting Your Business RestartedGetting Your Business Restarted• Business Function Priorities

– What are the functions most critical to the operation (Consistent with your technology recovery priories?)

– What processes can be done manually?• Facilities

– Where can the employees work?– How do they get there?

• Workstations– What office equipment do we need? – What supplies to we need to function?

• Vital Records– What documents do we need to function?– How do we write and deposit checks?

h i i li ?

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 35

– Where is our insurance policy?

Page 36: IT Continuity of Operations

Technology Availability PlanTechnology Availability Plan• Your existing disaster recovery plan is a good starting point for building a 

Technology Availability Plan (TAP)• It’s a defensive measure that prepares your IT management and team 

members to respond to—even help prevent—interruptions• All‐inclusive it covers your entire infrastructure as well as• All‐inclusive, it covers your entire infrastructure as well as 

telecommunications, systems, applications and data within the data center.

• A detailed series of action steps, activity checklists, personnel role d fi iti id tifi tidefinitions, resource identification

• Technology recovery priorities• Benefits of a comprehensive TAP

– Better preparedness for IT disruptionsBetter preparedness for IT disruptions – More agile, more effective response – Reduced severity and duration of incidents – Greater ability to mitigate risk—and the                                                             

associated increased confidence

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 36

associated increased confidence 

Page 37: IT Continuity of Operations

L th GLessen the Gaps Between CapabilitiesBetween Capabilities and Requirements 

Page 38: IT Continuity of Operations

For Most Businesses, 100% Availability Is a Myth

• In a perfect world, you would have 100% availability, but whoIn a perfect world, you would have 100% availability, but who can afford complete redundancy?– Smaller business have tighter budgets, but tend to be less complicated– Large corporations have higher requirements and budgetsLarge corporations have higher requirements and budgets– The mid‐market tends to be in the most challenging position

• The most we can hope for is to lessen the gaps between the needs and capabilities of the businessneeds and capabilities of the business

• How do you make it a reality?– Management Buy‐in and Support– Allocation of Resources– Build Availability into Systems– Hard Work and Persistence

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 38

Page 39: IT Continuity of Operations

Selling Availability In Your Organization

• Management EducationManagement Education– Downtime impact on the business– Informed managers make better decisions

• Risks and Exposures• Risks and Exposures• Goal:  RTO/RPO acceptance

– What managements needs to approve– Communicate in business terms ($$$)

• Cost of Ownership– Initial costs– Ongoing costs

• Return on investmentR bilit & M U ti

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 39

– Recoverability  & More Uptime– Customer Service / Satisfaction

Page 40: IT Continuity of Operations

Cost Of Downtime AnalysisCost Of Downtime Analysis

• The more complexThe more complex your environment the more resource i i dintensive and expensive it is to keep availableavailable

• High availability is not cheap, but that is nothing compared to a business interruption

Cost of Prevention

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 40

interruption

Page 41: IT Continuity of Operations

Example: Downtime Cost to a $$500M Organization

Cost of Outage = $250K/HrLength of Outage w/o Preparation (5 days) = $10MLength of Outage w/ Preparation (1 day) = $2M

SAVINGS = $8M

Cost of Preparation = $75K/year

Odds of Outage 1 in 25 = 4.0%4.0% x $8M = $320K

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 41

Prevention is actually quite cost effective!

Page 42: IT Continuity of Operations

Determining ROI Of AvailabilityDetermining ROI Of Availability

• 'Disaster Driven' ROI Solutions• Disaster‐Driven  ROI Solutions… – If Your Business Continuity Solution only Addresses UNPLANNED UNPREDICTABLEAddresses UNPLANNED, UNPREDICTABLE DOWNTIME (Less than 5% of Downtime) it will take a disaster to find ROI

• ROI from Everyday SolutionsIf Your Business Continuity solution also– If Your Business Continuity solution also  addresses PLANNED, PREDICTABLE DOWNTIME. (95+% of All Downtime), you'll find everyday ROI 

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 42

( ), y y ywithout the disaster!

Page 43: IT Continuity of Operations

Achieving Management Buy‐InAchieving Management Buy In• Management support of availability solutions requires g pp y q

understanding the business requirements– What are the drivers of the business? Speak the language of business 

not just ITh i h f d i ?– What is the cost of downtime?

– What are the other non‐technical effects of business interruption?• Availability is an investment, not an expense

B ild b i i i il bili l i– Build a business case to invest in availability solutions– What is the ROI from implementing availability                                     

solutions?• Strike when the iron is hot there is no• Strike when the iron is hot, there is no                                            

better time to pitch availability than                                        after an outage (even a small one)

• Build consensus form the bottom up and the top down

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 43

Build consensus form the bottom up and the top down

Page 44: IT Continuity of Operations

K C t U d tKeep Current: Update Your Plan to PreventYour Plan to Prevent Gaps from Developing 

Page 45: IT Continuity of Operations

Putting it All TogetherPutting it All Together

• Your AEP is useless if all the information is scatteredYour AEP is useless if all the information is scattered about in different places

• Make it easy to update• Make plenty of copies and give one to each of your key personnel– Make hard‐copy emergency “grab binders”– Keep copies on USB flash drives

• Keep several extra copies off‐site• Keep several extra copies off‐site– Keep copies at home, in your car, and/or in a safety‐deposit box. 

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 45

– Upload a copy to a web‐accessible server hosted off‐site

Page 46: IT Continuity of Operations

Communicate, Communicate, Communicate

• Share your plan don’t just lock it in a• Share your plan, don t just lock it in a desk drawer!M k i• Make sure everyone in your company is familiar with the Availability Plan

• Hold mandatory training classes for every employee whether they are on the critical  list or not

• Keep availability on everyone's radar

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 46

p y y

Page 47: IT Continuity of Operations

Test Your PlanTest Your Plan• You’ve put really good ideas down, accumulated all yourYou ve put really good ideas down, accumulated all your 

information, identified contingency locations, put your contact lists in place, but can you pull it off? 

• One thing you will definitely learn in the test is that youOne thing you will definitely learn in the test is that you haven’t gotten it all just exactly right

• Don’t wait until disaster strikes to figure out what you should do differently next timeshould do differently next time

• If you make any major changes, run it again• Even after you have a solid plan, you should                  

test it annually• Run desktop simulations: call your team into a 

conference room and run through a mock disaster

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 47

g

Page 48: IT Continuity of Operations

Plan to Change the PlanPlan to Change the Plan

• “No battle plan survives contact with theNo battle plan survives contact with the enemy.” ‐‐Helmuth von Moltke the Elder

h d l i d• No matter how good your plan is, and no matter how smoothly your test runs, it is likely there will be events outside your planthere will be events outside your plan– The hotel that was to be your DR site is booked up– A key member of the recovery team is on vacationy y– Your backup tape was defective– The one weekend, you leave your laptop               at the office the building burns down

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 48

at the office, the building burns down

Page 49: IT Continuity of Operations

Review Revise and RedistributeReview, Revise and Redistribute

• Every time something changes update all copies of your EAPEvery time something changes, update all copies of your EAP– New hardware / new software– More importantly…new business processes

• Constant updating can be time consuming, consider using a software tool to manage and update your plans

• Schedule regular reviews of your plan and stick to theSchedule regular reviews of your plan and stick to the schedule

• Never let it get out of date…It is a living document

• An out‐of‐date plan can be worse than useless: it makes you feel safe when you are anything but!

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 49

feel safe when you are anything but!

Page 50: IT Continuity of Operations

IT Continuity of Operations: Lessons Learned

• Get out of your comfort zone and focus on the business, not y ,just technology

• Embrace availability as a discipline or methodologygy

• Build higher availability into every project• Business needs will change over time• Think flexibility scalability• Think flexibility, scalability• Strive for continuous improvement• Test frequently

Y d ’ l d illi d ll l i b d• You don’t always need a million dollar solution, but you need an annual budget

• No matter how prepared you think you are, the unexpected ill l h M h O ti i t!

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 50

will always happen…Murphy was an Optimist!

Page 51: IT Continuity of Operations

Questions & AnswersQuestions & Answers“I always tried to turn every disaster into anI always tried to turn every disaster into an opportunity.” ‐‐ John D. Rockefeller

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 51

Page 52: IT Continuity of Operations

AppendixAppendix

Page 53: IT Continuity of Operations

Where Else Can I Get Information?Where Else Can I Get Information?

• Web Sites• Web Sites– www.drj.com

ti l i– www.contingencyplanning.com– www.globalcontinuity.com– www.recovery.sungard.com– www.disaster‐resource.com– www.businesscontinuitytoday.com

• Professional Organizations 

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 53

• Consultants

Page 54: IT Continuity of Operations

More of My Favorite DR PagesMore of My Favorite DR Pages

• Downtime Calculator• Downtime Calculator– www.visionsolutions.com/Solutions/Disaster‐Recovery‐toolkit‐downtime‐calc.aspxy p

• Glossary of Terms– www.continuitycentral.com/DRGlossaryofTerms.pwww.continuitycentral.com/DRGlossaryofTerms.pdf

• Business Continuity and Resiliency Self‐y yAssessment Tool– www.ibm.com/services/us/bcrs/self‐assessment

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 54

Page 55: IT Continuity of Operations

Document Collection WorksheetsDocument Collection Worksheets

• Applications• Computer 

• Vital Records• Employee Contact p

EquipmentOffi E i t

p yInfoE l C ll T• Office Equipment

• Telecom/Voice• Employee Call Trees• 3rd Party Info

• Office Supplies • Alternate Site Space

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 55

Page 56: IT Continuity of Operations

ApplicationsApplications– Business FunctionBusiness Function– Recovery Priority– Application RTO– Manual Procedures in Place– Inter‐dependant Applications– VendorVendor– Version– # Licenses– Install Key– Serial NumberMedia Off Site

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 56

– Media Off Site

Page 57: IT Continuity of Operations

Computer EquipmentComputer Equipment• FunctionFunction• IP Address• Description• Service Tag / Code• Warranty expires• OS / Service PackOS / Service Pack• Memory• Hard Drive ‐ number & capacity• Specialty cards• Applications supported• Business function

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 57

• Business function

Page 58: IT Continuity of Operations

Telecom/VoiceTelecom/Voice• Site Name• Site Name• Circuit Size• Equipment• Circuit ID• Vendor• Contact Number• Contact Number

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 58

Page 59: IT Continuity of Operations

Vital RecordsVital Records• Description• Description• Location• Required By• Responsible Partyp y

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 59

Page 60: IT Continuity of Operations

Employee Contact & Call TreesEmployee Contact & Call Trees• NameName• Role / Title• Address• Phone

– Office Phone– Cell Phone– Alternate Phone

• E‐mail– Office E‐mailOffice E mail– Personal E‐mail– Alternate E‐mail

• Expertise / Notes

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 60

• Expertise / Notes

Page 61: IT Continuity of Operations

3rd Party Info3rd Party Info• Name• Name• Customer #• Telephone• Contact• Comments• Service / Product Provided• Service / Product Provided• Used in this Recovery Activity

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 61

Page 62: IT Continuity of Operations

Alternate Site Space

• Workstation TypeWorkstation Type– Hardware/Software– Phone 

• Shared Resources– Phone SystemP i– Printers

– Faxes / Copiers• Seats required by departmentSeats required by department

– Match to RTOs (24 hrs, 72 Hrs, etc)– Not everyone needs to be there Day 1

March 10, 2009 CAMP IT Conferences ‐ IT Continuity of Operations 62