velocity eu 2012 escalating scenarios: outage handling pitfalls

Escalating Scenarios

A Deep Dive Into Outage Pitfalls

John AllspawVelocity

London 2012

Wednesday, October 3, 12

TROUBLESHOOTING

This is NOT about troubleshooting

Or, not just about troubleshooting

LAYOUT

• Criteria• Situational Awareness• HROs• Decision Making• Communication• Team Coordination• A little bit of psychology

How important is this?

Oct 2011

Sept 2012

Where to learn from?

Kegworth 1989

Dr. Richard Cook, Velocity US 2012http://www.youtube.com/watch?v=R_PDc0HFdP0

“The Self-Designing High-Reliability Organization: Aircraft Carrier Flight Operations at Sea”Rochlin, La Porte, and Roberts. Naval War College Review 1987

http://govleaders.org/reliability.htm

What Goes On In Our Heads?

Jens Rasmussen, 1983Senior Member, IEEE

“Skills, Rules, and Knowledge; Signals, Signs, and Symbols, and Other Distinctions in Human Performance Models”IEEE Transactions On Systems, Man, and Cybernetics, May 1983

SKILL - BASED

Simple, routineRULE - BASED

Knowable, but unfamiliarKNOWLEDGE - BASED

WTF IS GOING ON?(Reason, 1990)

Situational Awareness"the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future,” - (Endsley, 1995)

"keeping track of what is going on around you in a complex, dynamic environment" (Moray, 2005, p. 4)

"knowing what is going on so you can figure out what to do" (Adam, 1993)

OODA Loop

Observe Orient Decide Act

MetricsMonitoringAlertingAlarming

AnalysisVisualizationCorrelation

PlanningResourcing

Execution

credit: http://blog.b3k.us/ooda.htmlWednesday, October 3, 12

Canonical Work

“Towards a Theory of Situational Awareness”Mica Endsley, Human Factors (1995)

http://www.satechnologies.com/Papers/pdf/Toward%20a%20Theory%20of%20SA.pdf

Situational AwarenessLevel I

Perception

Level IIComprehension

Level IIIProjection

Situational Awareness

Projectionof future status

LEVEL III

Comprehension of current situation

LEVEL II

Perception of elements in current situation

LEVEL I

DecisionPerformance

of actionsState of the environment

System capabilityInterface designStress and workloadComplexityAutomation

Goals andobjectives

Preconceptions (expectations)

Information processing mechanisms

Long term memory states Automaticity

Feedback

- Abilities- Experience- Training

Task/System Factors

Individual Factors

(Endsley)Wednesday, October 3, 12

Level One: Perception

Context

Can you spot the anomaly?Wednesday, October 3, 12

24 hours

Context

7 daysWednesday, October 3, 12

Context

NormalBut

NoisyWednesday, October 3, 12

Level Two

ComprehensionWednesday, October 3, 12

Level Two

Mental Models

• Categorization & Comprehension

• Mental “map” or “schema”

• Informed by experience, stored in memory

Mental Models

Level Three

Mental Models

Level Three

• Ambiguity • Fixation • Confusion • Lack of Information• Failure to maintain • Failure to meet expected checkpoint or target• Failure to resolve discrepancies • A bad gut feeling that things are not quite right

Common Clues you’re losing SA at this level

Characteristics of response to escalating scenarios

...tend to neglect how processes develop within time (awareness of rates) versus assessing how things are in the moment

“On the Difficulties People Have in Dealing With Complexity” Dietrich Doerner, 1980

...have di!culty in dealing with exponential developments (hard to imagine how fast something can change, or accelerate)

...inclined to think in causal SERIES, instead of causal NETS.

A therefore B,

instead of

A, therefore B and C (therefore D and E), etc.

Requisite Memory Trap

SA Pitfalls

Workload, anxiety, fatigue, other stressors

SA Pitfalls

Data Overload

SA Pitfalls

Misplace Salience

SA Pitfalls

http://www.perceptualedge.com/articles/Whitepapers/Dashboard_Design.pdf

Complexity Creep“Everything should be as simple as it can be, but not simpler.”- paraphrased, Einstein

SA Pitfalls

Poor Mental Models

SA Pitfalls

Out-Of-The-Loop Syndrome

SA Pitfalls

Refusal to make decisions

SA Pitfalls

Non-communicating lone wolf-isms

Heroism

Irrelevant noise in comm channels

Distraction

• Divide and conquer applied to problem space, division of labor

• Incident resolution vs. Problem resolution

• Reproducibility

• Fault Tolerance E!ects

Shotgun debugging

• Interpredictability

• Common Ground

• Directability

JOINTACTIVITY

http://csel.eng.ohio-state.edu/woods/distributed/CG%20final.pdf

Interpredictability

Common GroundWednesday, October 3, 12

Directability

Improvisation

IMPROVISATION

Improvisation

“...you can’t improvise on nothing; you got to improvise on something.”

Charles Mingus

Diagnose the problem

Represent the problem

Detect the Problem/Opportunity

Generate acourse of

action

ApplyLeveragePoints

Evaluate

CommunicationRecommendations

•Explicitness•Assertiveness•Timing

Assertiveness

• Passive

• Assertive

• Aggressive

Exercise

Communication

• IRC?

• Face-To-Face?

• Conference Call?

• Morse Code?

Kegworth 1989

MeaningEncode

Sender ReceiverDecode

Meaning

Transmission

MeaningEncode

Meaning

Transmission

MeaningEncode

Meaning

Transmission

Meaning

DecodeReceiver Sender

Encode

Meaning

Transmission

MeaningEncode

Meaning

Transmission

Meaning

DecodeReceiver Sender

Encode

Meaning

Transmission

FeedbackInformational

FeedbackCorrective

FeedbackReinforcing

Decision Making Naturalistic Decision Making (NDM)Gary Klein

Decision Making

Step One: What is the problem?

Decision Making

Step Two: What shall I do?

Recognition-Primed Decisions

Decision Making

Rule-Based Decisions

Decision Making

Choice decisions

Decision Making

Creative decisionsDecision Making

Decision Making

Creative Choice Rule-Based RPD

Decreasing cognitive effortDecreasing effects of stress

Increasing cognitive effortIncreasing effects of stress

PRE-Mortem

Decision Making

Tooling

MetricTimePeriod

Controls

ALERTS• Meant to boost SA

• Alarm overload

• High false alarm rates

• Routinely disable alerts

Alert ReliabilityWednesday, October 3, 12

ALERT DESIGN

• Signal:Noise can be di"cult

• Easy to err on more false alarms

• Decay in trust

• Origins: Undetectable conditions

ALERT DESIGN

Confirmation

ALERT DESIGN

Expectancy

ALERT DESIGN

• Don’t make people singularly reliant on alarms

• Support alarm confirmation activities

• Make alarms unambiguous

• Reduce, reduce, reduce false alerts

• Set missed/false alert trade-o!s appropriately

ALERT DESIGN

• Use multiple modalities

• Minimize alarm disruptions to ongoing activities

• Support the assessment/diagnosis of multiple alerts

• Support global SA of systems in an alarm state

Mature Role of Automation

http://www.bainbrdg.demon.co.uk/Papers/Ironies.html

“Ironies of Automation” - Lisanne Bainbridge

Mature Role of Automation

• Moves humans from manual operator to supervisor

• Extends and augments human abilities, doesn’t replace it

• Doesn’t remove “human error”

• Are brittle

• Recognize that there is always discretionary space for humans

• Recognizes the Law of Stretched Systems

SUMMARY

So what can we do?

“In preparing for battle, I have always found that plans are useless but planning is indispensable.”

- Eisenhower

So what can we do?We develop our Non-Technical Skills

• Situational Awareness

• Communication

• Decision Making

• Improvisation

• Crew Resource Management (CRM)

So what can we do?

We tailor our environment to adapt

• Tooling to support SA

• Learning from outages (PostMortem)

• Anticipating problems (PreMortem)

• Gather Meta-Metrics

THE ENDWednesday, October 3, 12

Credits

• http://www.flickr.com/photos/28650594@N03/2718027136/

• http://www.flickr.com/photos/telstar/2816887784/

• http://www.flickr.com/photos/soldiersmediacenter/3855375117/

• https://www.flickr.com/photos/spencediddy/3197199659/sizes/l/in/faves-allspaw/

• http://www.flickr.com/photos/splorp/64027565/sizes/l/in/photostream/

velocity eu 2012 escalating scenarios: outage handling pitfalls

human factors

dynamic environment

ieee skills

knowledge signals

near future

theory of sa

velocity us

deep dive

Business

ge- outage schedule

outage management system

the escalating importance ofthe escalating importance of...

post outage feedback

always report your power outage power outage 715-372-4047

understanding and managing escalating...

module 3 | part 6 managing conflict: escalating and de...

outage investigation larkin substation outage on april 21

study guide on escalating inequality

disneyjunior - sorry - outage

escalating cancer drug prices

outage coordination

reducing outage durations

raising and escalating concerns - rcn

economic outage evaluation woody rickerson ercot outage...

outage schedule timeline

escalating the war

material costs escalating - owens corning

de-escalating volatile situations

muscle logic escalating density training