mock disaster exercises – how to test true capability€¦ · • police and fire are on the...

© 2018 MHA CONSULTING. ALL RIGHTS RESERVED.

Mock Disaster Exercises –How to Test True Capability

Richard Long, Senior Advisory Consultant, MHA Consulting

July 11, 2018

© 2018 MHA CONSULTING. ALL RIGHTS RESERVED. 2

19-year proven track record of applying

industry standards and best practices

across a diverse pedigree of clients.

A simple mission: Ensure the continuous

operations of our clients’ critical

processes.

60% of revenue comes from Business

Resiliency, 30% from IT Disaster

Recovery, and 10% from SaaS tools.

SaaS Tools: BIA On-Demand, Compliance

Confidence, Residual Risk.

SAASCompliance

and risk tools.

CAPABLEComprehensive suite of services.

20Average years

industry experience.

Years inoperation.

GLOBALDiverse, global

client base.

19

Richard Long, Practice Leader & Senior Advisory ConsultantPhoenix, Arizona www.mha-it.com

KEY FACTS

SENIOR LEADERSHIP

MHA Consulting’s senior team has an average of over 20 years of industry relevant experience in the areas of Business Continuity, Disaster Recovery, and Project Management.

COMPANY BACKGROUND

http://www.mha-it.com/


HEALTHCARE EDUCATION FINANCIAL INSTITUTIONS

CONSUMER PRODUCTS INSURANCE TRAVEL & ENTERTAINMENT GOVERNMENT/UTILITY

SERVICES

DIVERSE, GLOBAL CLIENT BASE


Business Recovery Strategies & Solutions

Data Center Recovery Strategies

Current State Assessment

Policy & Standards

Business Impact Analysis

Threat & Risk Assessment

BCMMETRICSTM

BIA On-Demand (BIAOD)

BCMMETRICSTM

Compliance Confidence (C2)

BCMMETRICSTM

Residual Risk (R2)

Training & Awareness

Mock Disaster Exercises

Plan Functional Walkthroughs

Alternate Worksite Exercises

Crisis Management

Business Recovery

IT Disaster Recovery

Update Recovery Plans

Update Current State Assessment

Update Business Impact Analysis & Threat Assessment

Third Party Assessments

EXERCISES MAINTAIN & IMPROVEASSESS THE CURRENT

ENVIRONMENTRECOVERY STRATEGIES &

SOLUTIONSRESPONSE & RECOVERY

PLANS

ROBUST SUITE OF SERVICES


… a simulation of an unplanned disruption that

requires participants to identify the actions and

steps they would take to successfully respond,

assess impacts, activate resources, and recover in

a timely manner.

WHAT IS A MOCK DISASTER EXERCISE?

MDE


A P P R O A C H & M E T H O D O L O G Y

E X E R C I S E T Y P E S

• The RIGHT people.

• The RIGHT planning.

• Increasingly complex exercises to heighten skills.

• Properly training participants to maximize learnings.

• Conduct maturity-appropriate exercises.

• Assess and document the results.

• Resolve the findings and update the process.

• Tabletop Exercise/Structured Walkthrough Test

• Walkthrough Drill/Simulation Test

• Functional Drill/Parallel Test

• Full Interruption/Full Scale Test

THE SESSION


T R E N D S I M P A C T I N G P L A N N I N G

N E X T E V O L U T I O N O F P L A N N I N G

• Few organizations have best of class capabilities.

• Management places no emphasis on

training/exercises.

• Believe events will never happen to them.

• Attention spans, availability, accountability,

participation.

• Refuse to exercise at the highest levels.

• Don’t incorporate external agencies and partners.

• Lack of willingness to use a standard IM process.

• Leaders can’t lead the team.

• Scenario agnostic – it’s about the impacts.

• Test real-time capabilities in unplanned

exercises.

• Fully integrate external agencies and partners.

• Certify IM team members on a regular basis.

• Conduct virtual reality exercises.

• Mini and full exercises.

• Use “Checklist Manifesto” approach.

TRENDS & THE FUTURE


THE BIG PICTURE

Risk

Resilience


• It’s simple, you must have the right people in the right seats.

• Don’t pick people simply based on title; must consider fit and capability.

• Remember the team takes on the personality of its leaders.

• Train your Incident Management leaders on how to run a team.

• Be prepared for political, hierarchy, generational, and cultural issues.

• Keep structure simple and easy to understand.

• Teams who work together more often are proven to be more successful.

• Remember attention spans are short; train appropriately.

DEALING WITH THE HUMAN FACTOR

T H E H U M A N F A C T O R


E X E R C I S E M U S T N O T B E T A K E N L I G H T L Y

D O C U M E N T A L L E X E R C I S E P L A N S

CONSIDERATIONS

• Scenario complexity increases planning time

• Exercise planning team experts

• Maturity of the team being exercised

• Time available for the exercise

• Areas of weakness to address from past exercises

• Resources required (people, process, technology)

• Exercise type

• Budget

PLANNING DOCUMENT

• Purpose

• Scope

• Objectives

• Assumptions

• Risks

• Resources (People, Places, Things)

• Action Item Lists

RESOURCES

• DHS Exercise Planning Guidelines

PLANNING THE EXERCISE


WHAT TYPE OF EXERCISE?

Consider the maturity of the team; are you and your team members beginners, intermediates or advanced?01

What key objectives are you trying to achieve with the exercise? 02

Build on increasingly complex exercises to achieve success over time (Tabletop, Walkthrough, Functional, etc.).

03CONSIDER

It’s okay to use the same exercise type more than once to build confidence and success, but move on when you have met your objectives.

04

Sometimes, you have to go back to a less complexexercise to reinforce the key objectives you are tryingto achieve before you can move forward again.

05

Remember the mission is to build the capabilityof your team - not to make them fail.

06


E X E R C I S E T Y P E

• Tabletop Exercise/Structured Walkthrough Test

• Novice/Days/Risk Low

• Walkthrough Drill/Simulation Test

• Novice to Moderate/Days to Weeks/Risk Low

• Functional Drill/Parallel Test

• Moderate to Advanced/Months/Risk Moderate to

High

• Full Interruption/Full Scale Test

• Advanced/Months/Risk High

APPROPRIATE EXERCISE TYPE FOR TEAM



EXERCISE THE TEAM AND PLAN

• Standalone – Test the ability to recover the system/application on its own.

• Integrated – Test the ability to recover the system/application with one or more of its critical upstream/downstream dependencies.

• Business Process – Test the ability to recover the system/application as part of a complex business process with many systems/applications involved from end to end.

• Complexity – Increase complexity over time to heighten ability to validate your recovery plan and strategy for critical systems/applications.

U S E I N C R E A S I N G L Y C O M P L E X D R E X E R C I S E S


• Perform a single large integrations-based exercise annually.

• Perform limited application/department exercises monthly/quarterly.

• Tabletop on new app or changed technologies.

• Emergency/Crisis Exercises:• Corporate

• IT

• Coordinated

PERFORM MULTIPLE EXERCISES



T H E S C E N A R I O S H O U L D T E L L A S T O R Y

CONSIDERATIONS

• It’s about getting them to reach new capabilities.

• Ensure the scenario elicits the actions you want to happen.

• Use a relevant scenario that makes sense for your company.

• Do your research on the scenario.

• Doesn’t have to be complex to be successful.

• Use pauses and breaks.

OUTPUT

• Best possible exercise to heighten team capabilities by providing them with

points of reference should the real event occur.

THE SCENARIO


B U I L D I N G T H E S C E N A R I O

SCENARIOS

• Determine if natural, man-made or technological event.

• Build summary of the simulated event.

• Align objectives to the scenario.

• Build injects and timeline that tell the story.

• Use quality over quantity of injects.

• Identify actions to be taken based on injects.

• Validate scenario with your planning team.

• Consider doing it (or a portion of it) in real time.

RESOURCES


THE SCENARIO


A G O O D F A C I L I T A T O R

CONSIDERATIONS

• Choose a primary facilitator.

• He or she must be good at leading the exercise, but not

directing.

• Complex exercises may require multiple facilitators.

• Must keep it on time and on point.

• Be able to make decisions to adjust the exercise as needed.

• Ensure its success.

REMEMBER

• If you are NOT a good facilitator, find someone who is to lead

your exercises. Even a good exercise will fail with a bad

facilitator.

FACILITATING THE EXERCISE


C O N S I D E R A T I O N S

CONDUCT THE EXERCISE

• Have fun!

• Start on time and end on time.

• Keep it on track based on your timeline.

• Adjust based on progress made or lack of it.

• Be prepared to add or delete injects on the fly.

• Don’t be afraid to stop and do a RESET.

• Breaks should be short.

• Be prepared for real life emergencies.

• Let participants take longer if they are being

successful or stop them if they need to move on.

• Focus on identifying issues, not binary results

(success or failure).

• Debrief at the end.


POST-EXERCISE DOCUMENT

• Purpose

• Scope

• Objectives

• Assumptions

• Successes and Opportunities

• Participating Resources (People, Places, Things)

• Action Item Lists with Responsible Parties/Due Dates

RESOURCES


DOCUMENT THE EXERCISE

D O C U M E N T T H E R E S U L T SO F T H E E X E R C I S E


• Adjust the Strategy - May require modifications to the recovery strategy based on results and needs.

• Adjust Recovery Procedures and Documentation -A major benefit is updating and ensuring functional procedures.

• Review and update coordination between plans -Review and validate the integration and validity of BC plans while recovery occurs.

• Budget - Honest results help in determining both BC and DR budgets and approval from management.

USE THE RESULTS

H O W T O U S E T H E R E S U L T S O F T H E E X E R C I S E


Life Safety

Incident Stabilization

Property Preservation

Restoration of the Business

INCIDENT PRIORITIES & OBJECTIVES


(A)ssess

• What is the current situation?

(P)lan

• What steps are needed to address the current situation?

(I)mplement

• What resources do we need to assign to execute steps to address the current situation?

(E)valuate

• How well did we execute the plan and what needs to be addressed?

A.P.I .E.

U s e t h e A s s e s s , P l a n , I m p l e m e n t , a n d E v a l u a t e a p p r o a c h t o s i z e u p t h e i n c i d e n t a n d d e f i n e a p l a n o f a c t i o n . A . P . I . E . c o n s i s t s o f :


EXAMPLE SCENARIOS


Y O U A R E A S C H O O L D I S T R I C TI N C I D E N T M A N A G E M E N T T E A M

SCENARIO #1

• 5 elementary schools

• 2 junior high schools

• 1 high school

• Each school has its own Emergency Response Team

• Incident Command System is new to the district

• Command Center is at the district office

• School wide emergency notification system

• School resource officers available at each school


S C E N A R I O A C T I O N S

• A commercial tanker truck carrying ammonia hydroxide was diverted off the freeway to local roads due to an accident.

• The tanker truck was making a left turn when a car cut in front of it and crashed on its side; passenger car is under the tanker truck.

• Police and fire are on the scene. A hazmat team from a nearby city is en route; ETA is 30 minutes.

• Fire Incident Commander advises Dispatch there are 3 schools downwind within 2 miles of the scene (elementary, junior high, and a private K-8 school) with a total of around 500 students.

• Fire Incident Commander has not detected leakage from the tanker at this time.

• ESTABLISH command.

• ASSESS the strategic risks to the school district.

• PLAN your approach to address the strategic

risks.

• IMPLEMENT your approach.

• EVALUATE your performance.

SCENARIO #1


Y O U A R E A M O B I L E P A Y M E N T S C O M P A N Y I N C I D E N T M A N A G E M E N T T E A M

SCENARIO #2

• Headquartered in downtown San Francisco

• 1,000 employees

• Multi-tenant facility

• BCM program is new to the organization

• Critical operations have been identified

• Can work from home but no dedicated backup sites

• Data centers are in colocation sites

• Primary call center is at headquarters – no backup



• Protestors against the North Dakota pipeline have taken

control of your building overnight.

• They have come to protest against the Bureau of Land

Management, headquartered in your building.

• They got past security and up to the BLM office using stolen

access cards.

• The building is closed to all tenants until this is resolved;

could be days.

• Police have been contacted.

• Protestors are inside and outside the building.


• ASSESS the strategic risks to the organization.


risks.



SCENARIO #2


Y O U A R E A G L O B A L R E A L E S T A T E C O M P A N Y I N C I D E N T M A N A G E M E N T T E A M

SCENARIO #3

• Headquartered in downtown Austin, Texas

• 3,000 employees

• Global real estate offices


• Crisis Team has been implemented but not tested

• Data center is in headquarters




• The company is holding its annual global corporate get

together for its employees and real estate agents.

• There are over 7,500 people in attendance at a downtown

conference center 10 miles from headquarters.

• CNN is reporting that a gunman entered the conference

center and began shooting during a keynote speech.

• Police and fire are on-scene attempting to address the

situation.

• On-scene employees are reporting the CEO has been shot

along with at least 15 to 20 others.

• Your team has been assembled to address the situation.




risks.



SCENARIO #3


Y O U A R E A M O B I L E P A Y M E N T S C O M P A N Y I N C I D E N T M A N A G E M E N T T E A M

SCENARIO #4

• Headquartered in downtown San Francisco

• 1,000 employees

• Multi-tenant facility

• BCM program is new to the organization


• Can work from home but no dedicated backup sites

• Data centers are in colocation sites




• At approximately 6:00 AM today, the IT Help Desk began

handling several calls related to Alert Errors being received

as employees were attempting to log in to the network

and various Point of Sale (POS) systems.

• Although trouble-shooting efforts began immediately

upon notification of these Alerts Errors, the cause is

unknown and has been escalated to the ISERT.

• Based on the Severity Level, the ISERT has escalated to

the Oversight Committee.




risks.



SCENARIO #4



• The ISERT is reporting that a disgruntled (IT) employee who

thought he might be terminated (and eventually was)

developed a program that would change passwords on

critical SQL databases on a specific date and time. It appears

the script may not be on the backups so it will be safest to

restore at the Alternate site with the 7/10 backup.

• Any data that was entered into the POS Systems on 7/11 is no

longer available. Although attempts have been made to

contact the (terminated) employee he has directed all

inquiries to his attorney. At this point, this issue has been

referred to Legal for further review and action.

• At a minimum, the ISERT is reporting that the POS Systems

will remain off-line until noon on Thursday 7/12.




risks.



SCENARIO #4 - INJECT


Y O U A R E A S C H O O L D I S T R I C TI N C I D E N T M A N A G E M E N T T E A M

SCENARIO #5

• 5 elementary schools

• 2 junior high schools

• 1 high school

• Each school has its own Emergency Response Team

• Incident Command System is new to the district

• Command Center is at the district office

• School wide emergency notification system

• School resource officers available at each school



• A man has been shot at a local convenience store and the

shooter has evaded capture.

• Local elementary school is placed on lockdown.

• A mother of 3 is shot dead at her home; her 3 children attend

elementary school, junior high, and high school.

• News outlets are reporting that a close friend of the dead

mother stated that her ex-husband had threated to come

back to kill her and her boyfriend, and to get his 3 kids back.

• Two school principals report a black Impala slowly driving by

the schools and asking children if they know Billy, Cindy, or

Joanne.


• ASSESS the strategic risks to the school district.


risks.



SCENARIO #5


T H I N G S T O T H I N K A B O U T

• Get the RIGHT humans in the right seats.

• Focus on making incremental gains over time.

• Build on increasingly complex exercises that heighten

performance.

• Create an easy to use Incident Management process – think

Checklist Manifesto.

• It’s not about the number of injects or length of the exercise

but the quality.

• Perfect is the enemy of the good.

• Focus on the outcome, become target focused, and

concentrate on the ability to successfully recover.

FINAL THOUGHTS


• Human Factor.

• Plan Your Exercise.

• Select the Right Exercise Type.

• Build an Appropriate Scenario.

• Find the Right Facilitators.

• Conduct the Exercise.

• Document the Exercise and Update Your Process.

SUMMARY

TEST TRUE CAPABILITY

mock disaster exercises – how to test true capability€¦ · • police and fire are on the...

Documents