exercising - moving from good to great

MidAmerica Contingency Planning Forum - Not to be used or reproduced without written authorization.

1

Exercising -Moving From Good to Great

A member discussion facilitated by the MCPF BoardMarch 17, 2011


2

Overview

The purpose of this 90 minute facilitated discussion is to look at why and how we conduct business continuity and emergency management exercises and to share “best practices” that can in turn be used to improve our own exercises.

Active Participation = Greater Success


3

Agenda

• Review September 2010 meeting• Provide updates• Facilitated Group Activity• Wrap Up• Close


4

Review - Types

Exercise Format

• Talk-Through / Table Top• Simulation / Connectivity• Integrated • Live

Testing

• Building / Data Center Infrastructure

• Business Continuity • Crisis Management • Emergency Responder • Executive • Human Capital • Supply Chain • Technology• Third Party Vendor


5

Review Types - Updated

Exercise Format• Talk-Through / Table Top• Simulation / Connectivity• Integrated • Live

• Orientation• Discussion based• Functional• Drill• Full-scale

Of Testing

• Building / Data Center Infrastructure

• Business Continuity • Crisis Management • Emergency Responder • Executive • Human Capital • Supply Chain • Technology• Third Party Vendor


6

Review Exercise Phases• Planning

• Preparing

• Executing / Conducting

• Follow-up / Resolution

• Closure / Next Exercise Date


7

Review Exercise Phases – Updated

• Planning

• Preparing

• Executing / Conducting

• Follow-up / Resolution

• Closure / Next Exercise Date


8

Review - Testing Partners

• Business Leaders

• Customers

• Internal Auditors

• Third Party Vendors


9

Review - Testing PartnersUpdated

• Business Leaders

• Customers

• Internal Auditors

• Third Party Vendors

Don’t forget about:• Local, state, federal

government• Regulators• First responders• Utility providers• Suppliers• Media (cautiously)


10

Review - Discussion Questionsfor the Types of Testing

• Assumptions

• Challenges

• Actions/Tasks

• Budget

• Resource Commitments

• Requirements – Goal

What else?


11

Types of Testing

Building/Data Center Infrastructure: Assumptions:• Do you have a back-up building?• Valid security policies and monitoring of data center entry• When does timing of RTO start – point of disaster or disaster

declaration?Challenges:• Measure against RTO• Tenant of a building - POP of telecommunication• Identifying test timeActions/Tasks:• Ensure diesel fuel supply in case of emergency• Map on infrastructure from generator – ask data center facilities

team to ensure this exists (what is failing over and what is not).• Communication, communication, communication…Budget: • Part of IT budgetResource Commitments:• Timing – weekend or off hours, peak versus non-peak season• Customer communicationRequirements – Goals

Considerations:• Telecommunications• Underload• Third-party vendors• Fire control/suppression systems• Generator – failover to UPS or generator• Ensure emergency responders understand location of data

center, critical infrastructure – NO WATER in data center!• Announced versus un-announced for building• Reasonable test if thinking of unannounced – ramifications• Tabletop Exercise – un-announced – after fire drill, then

notify team going to conduct a tabletop exercise – do not allow them access to work area to get plan.

Question & Answers:• Does generator need to cycle gas? Can use stabilizer, run

once a month or week?Lessons Learned:• Need to determine method to get folks back to work after an

evacuation like a bomb scare.• Ensure the right folks can get access to facility after an

incident.

Disaster Recovery Power Down


12

Types of Testing

Types of Test:• Talk-Through / Tabletop• Simulation / Connectivity• Integrated • LiveOptions:• Alternate Work Area – Internal, 3rd party• Work from Home – ensure capability is set up prior to event, test

capacity of VPN, training employees on process, security guidelinesAssumptionsChallenges:• Maintaining a living document• Business to own their plan• Actions/TasksBudgetResource CommitmentsRequirements – Goals:• Meet RTO• Ensure business understands RTOLessons Learned:• Transportation to alternate siteReference:• Red Cross Ready Rating Program

Assumptions:Define crisis for your organization ChallengesActions/Tasks• Training employees on how to

react to an emergency• Quick reference cards/wallet cards• Communication - practiceBudget Resource CommitmentsRequirements – Goal

Business Continuity Crisis Management


13

Types of Testing

Tabletop exercise• Scenario, walk-through guides and wallet cardsEngage ExecutivesAssumptions:• If plan is not in place, executives will take charge, take

over.Action Items:• Ensure executives know and understand their roles -

training• Executives can talk publicly in front of a camera• Separate command center for executive/management

team and technical teams• Train assistants that manage logistics for the executivesRequirements/Goals:• Test support personnel• Engage them in the process• Test executive responseLessons Learned:• Do not have executives in command center

Assumptions:• If vendor does not have plan, they don’t test.Actions:• Send questionnaires/risk assessment to top #

vendors• Test with a couple of vendors• Ensure contractual agreements with vendors

include disaster recovery testing• Ability to audit vendorChallenges:• Coordinating test dates, time, resources with

vendor• Include vendor in your own disaster planning and

testing• Identify secondary and tertiary individuals to fill key

roles• “Rehearsals” – have to rehearse enough that

employees can seamlessly move into crisis response

Executive Third-Party Vendor


14

Things That Contributeto a Good Exercise

• An understanding by all that there are no wrong answers or actions. The purpose of the exercise is to identify what works and doesn't work.

• Realistic exercise scenario with no "cutesy gotchas" or trick injects.• It should be Realistic, Relevant and Revealing• No worst-case scenarios (because I already know how to pray, and that is the only solution

to the worst case scenario). • An invested planning group• Have a good planning committee that is committed to the overall outcome of the exercise• As a planner be flexible and look for workable solutions to benefit all• Challenging for the participants/responders• SMART Objectives - additionally, make sure that your objectives identify the Audience, the

Behavior that you what the audience to perform, under what Condition it (the behavior) is to be performed under, and to what Degree of accuracy it (the behavior) is to be performed.

• Make it fun• Putting into practice learned lessons.


15

SMART Objectives

• Specific – Is the wording precise and unambiguous?• Measurable – How will achievements be measured?• Action Oriented – Is an action verb used to describe

expected accomplishments?• Realistic – Is the outcome achievable with given

available resources?• Time Sensitive – What is the timeframe (if

applicable)?


16

Things That Can Hurt an Exercise• Trying to cram too much into the exercise or making it too complex - "KISS“• Time frame too short for adequate planning will kill an exercise...adequate

time for planning is a must• Inadequate preparation by all participants prior to the exercise• Less than 100% by all participants in terms of physical and mental

involvement.• Lack of Commitment, Communication, and Coordination• Unrealistic scenario for the players or jurisdiction• Too many people on the planning committee• Trying to make the exercise "'everything to everybody," and too much

artificiality.• Any exercise not tied into a cycle of training and exercises is a waste of time.


17

Which of the following are you most worried about being not up-to-speed in your organization should an incident occur?

Security Executive Council Poll 03/15/3011

www.securityexecutivecouncil.com


18

Group ActivityCongratulations – you are a member of the MCPF Exercise Team!

1. Break up into five groups – Planning, Preparing (Design & Development), Executing & Conducting, Follow-up & Resolution (Evaluation), Closure & Next Exercise (Improvement Planning).

2. Select a Spokesperson and a Documenter for your group3. Your goal is to brainstorm (WHAT?) related to your team

assignment4. Use the easel paper and markers to capture your ideas5. You will have 30 minutes to complete this task and then

participate in a group debriefing


19

Today’s Scenario• Earlier this morning at 9:00 AM, a 7.7 magnitude earthquake occurred along the New Madrid

Seismic Zone (NMSZ). Fifteen county regions along the Mississippi River, with a population of approximately 1.9 million people are seriously affected. The counties include Dunklin, Pemiscot, New Madrid, Stoddard, Butler, Mississippi, Scott, Bollinger, Cape Girardeau, Perry, Ste. Genevieve, Jefferson, St. Louis, St. Louis City, and Wayne.

• Our region is considered to be in the affected area.• Police and MODOT authorities are asking motorists to stay in place and not drive unless it’s

an emergency, as inspectors are assessing damage to roadways and bridges.• Utility outages are widespread throughout the metro St. Louis area.• The area within 50-75 miles of the epicenter was subjected to shaking on the Modified

Mercalli scale at an intensity of VII or greater, strong enough to destroy well-built structures, damage dams and reservoirs, cause landslides, and severely damage or destroy transportation structures such as roads, highways, bridges, and railroad tracks.

• Soil-liquefaction occurred in some areas, thereby increasing the level of destruction as quicksand-like conditions contributed to the destabilization and collapse of numerous buildings, transportation, and utility structures.

• Assume current day, time, and weather conditions.


20


21


22


23


24


25


26


27


28

Debriefing

• What did we set out to do?• What actually happened?• Why did it happen?• What are we going to do differently next time?• Are there lessons learned that should be

shared? • What follow-up is needed?


29

Additional Resources• Disaster Recovery Journal: www.drj.com• Disaster Resource Guide: www.disaster-resource.com• Continuity insights: www.continuityinsights.com/• FEMA HSEEP: https://hseep.dhs.gov/pages/1001_HSEEP7.aspx• MCPF Group in Linked In: www.linkedin.com• CUSEC: www.cusec.org• USGS: www.usgs.gov• Missouri SEMA: www.sema.dps.mo.gov• Natural Hazards Observer: http://www.colorado.edu/hazards

• Each other!

http://www.drj.com/

http://www.disaster-resource.com/

http://www.continuityinsights.com/

http://www.continuityinsights.com/

https://hseep.dhs.gov/pages/1001_HSEEP7.aspx

http://www.linkedin.com/

http://www.cusec.org/

http://www.usgs.gov/

http://www.sema.dps.mo.gov/

http://www.colorado.edu/hazards

http://www.colorado.edu/hazards


30

NLE 2011• In May 2011, the Federal Emergency Management Agency (FEMA) will

conduct the National Level Exercise 2011 (NLE 2011; www.ready.gov/nle2011/index.html ). The purpose of the exercise is to prepare and coordinate a multiple-jurisdictional integrated response to a national catastrophic event.

• NLE 2011 is a White House directed Congressionally-mandated exercise that will focus on regional catastrophic response and recovery activities between federal, regional, state, tribal, local and private sector participants.

• The focus of the exercise will simulate the catastrophic nature of a major earthquake in the central United States region of the New Madrid Seismic Zone (NMSZ). The year 2011 is the bicentennial anniversary of the 1811 New Madrid earthquake, for which the NMSZ is named. NLE 2011 will be the first NLE to simulate a natural hazard.

http://www.ready.gov/nle2011/index.html

exercising - moving from good to great

Documents

written authorization

review exercise phasesplanning

exercise date

emergency management

party vendors dont

followup resolution

diesel fuel supply

greata member discussion