velocity eu 2012 escalating scenarios: outage handling pitfalls

Escalating Scenarios

A Deep Dive Into Outage Pitfalls

John AllspawVelocity

London 2012

Wednesday, October 3, 12

TROUBLESHOOTING

This is NOT about troubleshooting

Or, not just about troubleshooting


LAYOUT

• Criteria• Situational Awareness• HROs• Decision Making• Communication• Team Coordination• A little bit of psychology


How important is this?


Oct 2011

Sept 2012


Where to learn from?


TMI


Kegworth 1989


Dr. Richard Cook, Velocity US 2012http://www.youtube.com/watch?v=R_PDc0HFdP0


http://www.youtube.com/watch?v=R_PDc0HFdP0

http://www.youtube.com/watch?v=R_PDc0HFdP0

“The Self-Designing High-Reliability Organization: Aircraft Carrier Flight Operations at Sea”Rochlin, La Porte, and Roberts. Naval War College Review 1987

http://govleaders.org/reliability.htm




What Goes On In Our Heads?


Jens Rasmussen, 1983Senior Member, IEEE

“Skills, Rules, and Knowledge; Signals, Signs, and Symbols, and Other Distinctions in Human Performance Models”IEEE Transactions On Systems, Man, and Cybernetics, May 1983


SKILL - BASED

Simple, routineRULE - BASED

Knowable, but unfamiliarKNOWLEDGE - BASED

WTF IS GOING ON?(Reason, 1990)


Situational Awareness"the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future,” - (Endsley, 1995)

"keeping track of what is going on around you in a complex, dynamic environment" (Moray, 2005, p. 4)

"knowing what is going on so you can figure out what to do" (Adam, 1993)


OODA Loop

Observe Orient Decide Act

MetricsMonitoringAlertingAlarming

AnalysisVisualizationCorrelation

PlanningResourcing

Execution

credit: http://blog.b3k.us/ooda.htmlWednesday, October 3, 12

http://blog.b3k.us/ooda.html

http://blog.b3k.us/ooda.html

Canonical Work

“Towards a Theory of Situational Awareness”Mica Endsley, Human Factors (1995)

http://www.satechnologies.com/Papers/pdf/Toward%20a%20Theory%20of%20SA.pdf






Situational AwarenessLevel I

Perception

Level IIComprehension

Level IIIProjection


Situational Awareness

Projectionof future status

LEVEL III

Comprehension of current situation

LEVEL II

Perception of elements in current situation

LEVEL I

DecisionPerformance

of actionsState of the environment

System capabilityInterface designStress and workloadComplexityAutomation

Goals andobjectives

Preconceptions (expectations)

Information processing mechanisms

Long term memory states Automaticity

Feedback

- Abilities- Experience- Training

Task/System Factors

Individual Factors

(Endsley)Wednesday, October 3, 12

Level One: Perception


Context

Can you spot the anomaly?Wednesday, October 3, 12

24 hours

Context


Context

7 daysWednesday, October 3, 12

Context

NormalBut

NoisyWednesday, October 3, 12

Level Two

ComprehensionWednesday, October 3, 12

Level Two


Mental Models

• Categorization & Comprehension

• Mental “map” or “schema”

• Informed by experience, stored in memory


Mental Models


Level Three


Mental Models


Level Three

• Ambiguity • Fixation • Confusion • Lack of Information• Failure to maintain • Failure to meet expected checkpoint or target• Failure to resolve discrepancies • A bad gut feeling that things are not quite right

Common Clues you’re losing SA at this level


Characteristics of response to escalating scenarios


...tend to neglect how processes develop within time (awareness of rates) versus assessing how things are in the moment


“On the Difficulties People Have in Dealing With Complexity” Dietrich Doerner, 1980


...have di!culty in dealing with exponential developments (hard to imagine how fast something can change, or accelerate)




...inclined to think in causal SERIES, instead of causal NETS.

A therefore B,

instead of

A, therefore B and C (therefore D and E), etc.




Requisite Memory Trap

SA Pitfalls


Workload, anxiety, fatigue, other stressors

SA Pitfalls


Data Overload

SA Pitfalls


Misplace Salience

SA Pitfalls


http://www.perceptualedge.com/articles/Whitepapers/Dashboard_Design.pdf




Complexity Creep“Everything should be as simple as it can be, but not simpler.”- paraphrased, Einstein

SA Pitfalls


Poor Mental Models

SA Pitfalls


Out-Of-The-Loop Syndrome

SA Pitfalls


Refusal to make decisions

SA Pitfalls


Non-communicating lone wolf-isms

Heroism


Irrelevant noise in comm channels

Distraction


• Divide and conquer applied to problem space, division of labor

• Incident resolution vs. Problem resolution

• Reproducibility

• Fault Tolerance E!ects

TEAMS


Shotgun debugging

TEAMS


• Interpredictability

• Common Ground

• Directability

JOINTACTIVITY

http://csel.eng.ohio-state.edu/woods/distributed/CG%20final.pdf




Interpredictability


Common GroundWednesday, October 3, 12

Directability


Improvisation


IMPROVISATION


Improvisation

“...you can’t improvise on nothing; you got to improvise on something.”

Charles Mingus


Diagnose the problem

Represent the problem

Detect the Problem/Opportunity

Generate acourse of

action

ApplyLeveragePoints

Evaluate


CommunicationRecommendations

•Explicitness•Assertiveness•Timing


Assertiveness

• Passive

• Assertive

• Aggressive


Exercise


Communication

• IRC?

• Face-To-Face?

• Conference Call?

• Morse Code?


Kegworth 1989


MeaningEncode

Sender ReceiverDecode

Meaning

Transmission


MeaningEncode

Sender ReceiverDecode

Meaning

Transmission

Meaning

DecodeReceiver Sender

Encode

Meaning

Transmission


FeedbackInformational


FeedbackCorrective


FeedbackReinforcing


Decision Making Naturalistic Decision Making (NDM)Gary Klein


Decision Making

Step One: What is the problem?


Decision Making

Step Two: What shall I do?


Recognition-Primed Decisions

Decision Making


Rule-Based Decisions

Decision Making


Choice decisions

Decision Making


Creative decisionsDecision Making


Decision Making

Creative Choice Rule-Based RPD

Decreasing cognitive effortDecreasing effects of stress

Increasing cognitive effortIncreasing effects of stress


PRE-Mortem

Decision Making


Tooling


??


MetricTimePeriod


Controls


ALERTS• Meant to boost SA

• Alarm overload

• High false alarm rates

• Routinely disable alerts


Alert ReliabilityWednesday, October 3, 12

ALERT DESIGN

• Signal:Noise can be di"cult

• Easy to err on more false alarms

• Decay in trust

• Origins: Undetectable conditions


ALERT DESIGN

Confirmation


ALERT DESIGN

Expectancy


ALERT DESIGN


ALERT DESIGN

• Don’t make people singularly reliant on alarms

• Support alarm confirmation activities

• Make alarms unambiguous

• Reduce, reduce, reduce false alerts

• Set missed/false alert trade-o!s appropriately


ALERT DESIGN

• Use multiple modalities

• Minimize alarm disruptions to ongoing activities

• Support the assessment/diagnosis of multiple alerts

• Support global SA of systems in an alarm state


Mature Role of Automation

http://www.bainbrdg.demon.co.uk/Papers/Ironies.html

“Ironies of Automation” - Lisanne Bainbridge




Mature Role of Automation

• Moves humans from manual operator to supervisor

• Extends and augments human abilities, doesn’t replace it

• Doesn’t remove “human error”

• Are brittle

• Recognize that there is always discretionary space for humans

• Recognizes the Law of Stretched Systems


SUMMARY


So what can we do?

“In preparing for battle, I have always found that plans are useless but planning is indispensable.”

- Eisenhower


So what can we do?We develop our Non-Technical Skills

• Situational Awareness

• Communication

• Decision Making

• Improvisation

• Crew Resource Management (CRM)


So what can we do?

We tailor our environment to adapt

• Tooling to support SA

• Learning from outages (PostMortem)

• Anticipating problems (PreMortem)

• Gather Meta-Metrics


THE ENDWednesday, October 3, 12

Credits

• http://www.flickr.com/photos/28650594@N03/2718027136/

• http://www.flickr.com/photos/telstar/2816887784/

• http://www.flickr.com/photos/soldiersmediacenter/3855375117/



• https://www.flickr.com/photos/spencediddy/3197199659/sizes/l/in/faves-allspaw/

• http://www.flickr.com/photos/splorp/64027565/sizes/l/in/photostream/


http://www.flickr.com/photos/28650594@N03/2718027136/


http://www.flickr.com/photos/telstar/2816887784/










https://www.flickr.com/photos/spencediddy/3197199659/sizes/l/in/faves-allspaw/

https://www.flickr.com/photos/spencediddy/3197199659/sizes/l/in/faves-allspaw/

http://www.flickr.com/photos/splorp/64027565/sizes/l/in/photostream/

http://www.flickr.com/photos/splorp/64027565/sizes/l/in/photostream/

velocity eu 2012 escalating scenarios: outage handling pitfalls

Business

human factors

dynamic environment

ieee skills

knowledge signals

near future

theory of sa

velocity us

deep dive