![Page 1: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/1.jpg)
Escalating Scenarios
A Deep Dive Into Outage Pitfalls
John AllspawVelocity
London 2012
Wednesday, October 3, 12
![Page 2: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/2.jpg)
TROUBLESHOOTING
This is NOT about troubleshooting
Or, not just about troubleshooting
Wednesday, October 3, 12
![Page 3: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/3.jpg)
LAYOUT
• Criteria• Situational Awareness• HROs• Decision Making• Communication• Team Coordination• A little bit of psychology
Wednesday, October 3, 12
![Page 4: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/4.jpg)
Wednesday, October 3, 12
![Page 5: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/5.jpg)
Wednesday, October 3, 12
![Page 6: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/6.jpg)
How important is this?
Wednesday, October 3, 12
![Page 7: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/7.jpg)
Wednesday, October 3, 12
![Page 8: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/8.jpg)
Wednesday, October 3, 12
![Page 9: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/9.jpg)
Wednesday, October 3, 12
![Page 10: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/10.jpg)
Wednesday, October 3, 12
![Page 11: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/11.jpg)
Wednesday, October 3, 12
![Page 12: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/12.jpg)
Oct 2011
Sept 2012
Wednesday, October 3, 12
![Page 13: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/13.jpg)
Wednesday, October 3, 12
![Page 14: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/14.jpg)
Wednesday, October 3, 12
![Page 15: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/15.jpg)
Wednesday, October 3, 12
![Page 16: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/16.jpg)
Wednesday, October 3, 12
![Page 17: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/17.jpg)
Where to learn from?
Wednesday, October 3, 12
![Page 18: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/18.jpg)
TMI
Wednesday, October 3, 12
![Page 19: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/19.jpg)
Wednesday, October 3, 12
![Page 20: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/20.jpg)
Wednesday, October 3, 12
![Page 21: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/21.jpg)
Kegworth 1989
Wednesday, October 3, 12
![Page 22: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/22.jpg)
Dr. Richard Cook, Velocity US 2012http://www.youtube.com/watch?v=R_PDc0HFdP0
Wednesday, October 3, 12
![Page 23: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/23.jpg)
“The Self-Designing High-Reliability Organization: Aircraft Carrier Flight Operations at Sea”Rochlin, La Porte, and Roberts. Naval War College Review 1987
http://govleaders.org/reliability.htm
Wednesday, October 3, 12
![Page 24: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/24.jpg)
Wednesday, October 3, 12
![Page 25: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/25.jpg)
Wednesday, October 3, 12
![Page 26: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/26.jpg)
What Goes On In Our Heads?
Wednesday, October 3, 12
![Page 27: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/27.jpg)
Jens Rasmussen, 1983Senior Member, IEEE
“Skills, Rules, and Knowledge; Signals, Signs, and Symbols, and Other Distinctions in Human Performance Models”IEEE Transactions On Systems, Man, and Cybernetics, May 1983
Wednesday, October 3, 12
![Page 28: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/28.jpg)
SKILL - BASED
Simple, routineRULE - BASED
Knowable, but unfamiliarKNOWLEDGE - BASED
WTF IS GOING ON?(Reason, 1990)
Wednesday, October 3, 12
![Page 29: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/29.jpg)
Situational Awareness"the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future,” - (Endsley, 1995)
"keeping track of what is going on around you in a complex, dynamic environment" (Moray, 2005, p. 4)
"knowing what is going on so you can figure out what to do" (Adam, 1993)
Wednesday, October 3, 12
![Page 30: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/30.jpg)
OODA Loop
Observe Orient Decide Act
MetricsMonitoringAlertingAlarming
AnalysisVisualizationCorrelation
PlanningResourcing
Execution
credit: http://blog.b3k.us/ooda.htmlWednesday, October 3, 12
![Page 31: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/31.jpg)
Canonical Work
“Towards a Theory of Situational Awareness”Mica Endsley, Human Factors (1995)
http://www.satechnologies.com/Papers/pdf/Toward%20a%20Theory%20of%20SA.pdf
Wednesday, October 3, 12
![Page 32: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/32.jpg)
Situational AwarenessLevel I
Perception
Level IIComprehension
Level IIIProjection
Wednesday, October 3, 12
![Page 33: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/33.jpg)
Situational Awareness
Projectionof future status
LEVEL III
Comprehension of current situation
LEVEL II
Perception of elements in current situation
LEVEL I
DecisionPerformance
of actionsState of the environment
System capabilityInterface designStress and workloadComplexityAutomation
Goals andobjectives
Preconceptions (expectations)
Information processing mechanisms
Long term memory states Automaticity
Feedback
- Abilities- Experience- Training
Task/System Factors
Individual Factors
(Endsley)Wednesday, October 3, 12
![Page 34: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/34.jpg)
Level One: Perception
Wednesday, October 3, 12
![Page 35: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/35.jpg)
Wednesday, October 3, 12
![Page 36: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/36.jpg)
Context
Can you spot the anomaly?Wednesday, October 3, 12
![Page 37: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/37.jpg)
24 hours
Context
Wednesday, October 3, 12
![Page 38: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/38.jpg)
Context
7 daysWednesday, October 3, 12
![Page 39: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/39.jpg)
Context
NormalBut
NoisyWednesday, October 3, 12
![Page 40: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/40.jpg)
Level Two
ComprehensionWednesday, October 3, 12
![Page 41: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/41.jpg)
Level Two
Wednesday, October 3, 12
![Page 42: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/42.jpg)
Mental Models
• Categorization & Comprehension
• Mental “map” or “schema”
• Informed by experience, stored in memory
Wednesday, October 3, 12
![Page 43: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/43.jpg)
Mental Models
Wednesday, October 3, 12
![Page 44: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/44.jpg)
Level Three
Wednesday, October 3, 12
![Page 45: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/45.jpg)
Mental Models
Wednesday, October 3, 12
![Page 46: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/46.jpg)
Level Three
• Ambiguity • Fixation • Confusion • Lack of Information• Failure to maintain • Failure to meet expected checkpoint or target• Failure to resolve discrepancies • A bad gut feeling that things are not quite right
Common Clues you’re losing SA at this level
Wednesday, October 3, 12
![Page 47: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/47.jpg)
Characteristics of response to escalating scenarios
Wednesday, October 3, 12
![Page 48: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/48.jpg)
...tend to neglect how processes develop within time (awareness of rates) versus assessing how things are in the moment
Characteristics of response to escalating scenarios
“On the Difficulties People Have in Dealing With Complexity” Dietrich Doerner, 1980
Wednesday, October 3, 12
![Page 49: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/49.jpg)
...have di!culty in dealing with exponential developments (hard to imagine how fast something can change, or accelerate)
Characteristics of response to escalating scenarios
“On the Difficulties People Have in Dealing With Complexity” Dietrich Doerner, 1980
Wednesday, October 3, 12
![Page 50: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/50.jpg)
...inclined to think in causal SERIES, instead of causal NETS.
A therefore B,
instead of
A, therefore B and C (therefore D and E), etc.
Characteristics of response to escalating scenarios
“On the Difficulties People Have in Dealing With Complexity” Dietrich Doerner, 1980
Wednesday, October 3, 12
![Page 51: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/51.jpg)
Requisite Memory Trap
SA Pitfalls
Wednesday, October 3, 12
![Page 52: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/52.jpg)
Workload, anxiety, fatigue, other stressors
SA Pitfalls
Wednesday, October 3, 12
![Page 53: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/53.jpg)
Data Overload
SA Pitfalls
Wednesday, October 3, 12
![Page 54: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/54.jpg)
Misplace Salience
SA Pitfalls
Wednesday, October 3, 12
![Page 55: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/55.jpg)
http://www.perceptualedge.com/articles/Whitepapers/Dashboard_Design.pdf
Wednesday, October 3, 12
![Page 56: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/56.jpg)
http://www.perceptualedge.com/articles/Whitepapers/Dashboard_Design.pdf
Wednesday, October 3, 12
![Page 57: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/57.jpg)
Wednesday, October 3, 12
![Page 58: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/58.jpg)
Complexity Creep“Everything should be as simple as it can be, but not simpler.”- paraphrased, Einstein
SA Pitfalls
Wednesday, October 3, 12
![Page 59: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/59.jpg)
Poor Mental Models
SA Pitfalls
Wednesday, October 3, 12
![Page 60: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/60.jpg)
Out-Of-The-Loop Syndrome
SA Pitfalls
Wednesday, October 3, 12
![Page 61: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/61.jpg)
Refusal to make decisions
SA Pitfalls
Wednesday, October 3, 12
![Page 62: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/62.jpg)
Non-communicating lone wolf-isms
Heroism
Wednesday, October 3, 12
![Page 63: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/63.jpg)
Irrelevant noise in comm channels
Distraction
Wednesday, October 3, 12
![Page 64: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/64.jpg)
Wednesday, October 3, 12
![Page 65: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/65.jpg)
• Divide and conquer applied to problem space, division of labor
• Incident resolution vs. Problem resolution
• Reproducibility
• Fault Tolerance E!ects
TEAMS
Wednesday, October 3, 12
![Page 66: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/66.jpg)
Shotgun debugging
TEAMS
Wednesday, October 3, 12
![Page 67: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/67.jpg)
• Interpredictability
• Common Ground
• Directability
JOINTACTIVITY
http://csel.eng.ohio-state.edu/woods/distributed/CG%20final.pdf
Wednesday, October 3, 12
![Page 68: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/68.jpg)
Interpredictability
Wednesday, October 3, 12
![Page 69: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/69.jpg)
Common GroundWednesday, October 3, 12
![Page 70: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/70.jpg)
Directability
Wednesday, October 3, 12
![Page 71: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/71.jpg)
Improvisation
Wednesday, October 3, 12
![Page 72: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/72.jpg)
IMPROVISATION
Wednesday, October 3, 12
![Page 73: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/73.jpg)
IMPROVISATION
Wednesday, October 3, 12
![Page 74: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/74.jpg)
Improvisation
“...you can’t improvise on nothing; you got to improvise on something.”
Charles Mingus
Wednesday, October 3, 12
![Page 75: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/75.jpg)
Diagnose the problem
Represent the problem
Detect the Problem/Opportunity
Generate acourse of
action
ApplyLeveragePoints
Evaluate
Wednesday, October 3, 12
![Page 76: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/76.jpg)
CommunicationRecommendations
•Explicitness•Assertiveness•Timing
Wednesday, October 3, 12
![Page 77: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/77.jpg)
Assertiveness
• Passive
• Assertive
• Aggressive
Wednesday, October 3, 12
![Page 78: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/78.jpg)
Wednesday, October 3, 12
![Page 79: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/79.jpg)
Exercise
Wednesday, October 3, 12
![Page 80: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/80.jpg)
Communication
• IRC?
• Face-To-Face?
• Conference Call?
• Morse Code?
Wednesday, October 3, 12
![Page 81: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/81.jpg)
Kegworth 1989
Wednesday, October 3, 12
![Page 82: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/82.jpg)
MeaningEncode
Sender ReceiverDecode
Meaning
Transmission
Wednesday, October 3, 12
![Page 83: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/83.jpg)
MeaningEncode
Sender ReceiverDecode
Meaning
Transmission
Wednesday, October 3, 12
![Page 84: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/84.jpg)
MeaningEncode
Sender ReceiverDecode
Meaning
Transmission
Meaning
DecodeReceiver Sender
Encode
Meaning
Transmission
Wednesday, October 3, 12
![Page 85: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/85.jpg)
MeaningEncode
Sender ReceiverDecode
Meaning
Transmission
Meaning
DecodeReceiver Sender
Encode
Meaning
Transmission
Wednesday, October 3, 12
![Page 86: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/86.jpg)
FeedbackInformational
Wednesday, October 3, 12
![Page 87: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/87.jpg)
FeedbackCorrective
Wednesday, October 3, 12
![Page 88: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/88.jpg)
FeedbackReinforcing
Wednesday, October 3, 12
![Page 89: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/89.jpg)
Decision Making Naturalistic Decision Making (NDM)Gary Klein
Wednesday, October 3, 12
![Page 90: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/90.jpg)
Decision Making
Step One: What is the problem?
Wednesday, October 3, 12
![Page 91: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/91.jpg)
Decision Making
Step Two: What shall I do?
Wednesday, October 3, 12
![Page 92: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/92.jpg)
Recognition-Primed Decisions
Decision Making
Wednesday, October 3, 12
![Page 93: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/93.jpg)
Rule-Based Decisions
Decision Making
Wednesday, October 3, 12
![Page 94: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/94.jpg)
Choice decisions
Decision Making
Wednesday, October 3, 12
![Page 95: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/95.jpg)
Creative decisionsDecision Making
Wednesday, October 3, 12
![Page 96: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/96.jpg)
Decision Making
Creative Choice Rule-Based RPD
Decreasing cognitive effortDecreasing effects of stress
Increasing cognitive effortIncreasing effects of stress
Wednesday, October 3, 12
![Page 97: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/97.jpg)
PRE-Mortem
Decision Making
Wednesday, October 3, 12
![Page 98: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/98.jpg)
Tooling
Wednesday, October 3, 12
![Page 99: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/99.jpg)
??
Wednesday, October 3, 12
![Page 100: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/100.jpg)
MetricTimePeriod
Wednesday, October 3, 12
![Page 101: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/101.jpg)
Controls
Wednesday, October 3, 12
![Page 102: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/102.jpg)
ALERTS• Meant to boost SA
• Alarm overload
• High false alarm rates
• Routinely disable alerts
Wednesday, October 3, 12
![Page 103: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/103.jpg)
Alert ReliabilityWednesday, October 3, 12
![Page 104: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/104.jpg)
ALERT DESIGN
• Signal:Noise can be di"cult
• Easy to err on more false alarms
• Decay in trust
• Origins: Undetectable conditions
Wednesday, October 3, 12
![Page 105: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/105.jpg)
ALERT DESIGN
Confirmation
Wednesday, October 3, 12
![Page 106: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/106.jpg)
ALERT DESIGN
Expectancy
Wednesday, October 3, 12
![Page 107: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/107.jpg)
ALERT DESIGN
Wednesday, October 3, 12
![Page 108: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/108.jpg)
ALERT DESIGN
• Don’t make people singularly reliant on alarms
• Support alarm confirmation activities
• Make alarms unambiguous
• Reduce, reduce, reduce false alerts
• Set missed/false alert trade-o!s appropriately
Wednesday, October 3, 12
![Page 109: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/109.jpg)
ALERT DESIGN
• Use multiple modalities
• Minimize alarm disruptions to ongoing activities
• Support the assessment/diagnosis of multiple alerts
• Support global SA of systems in an alarm state
Wednesday, October 3, 12
![Page 110: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/110.jpg)
Mature Role of Automation
http://www.bainbrdg.demon.co.uk/Papers/Ironies.html
“Ironies of Automation” - Lisanne Bainbridge
Wednesday, October 3, 12
![Page 111: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/111.jpg)
Mature Role of Automation
• Moves humans from manual operator to supervisor
• Extends and augments human abilities, doesn’t replace it
• Doesn’t remove “human error”
• Are brittle
• Recognize that there is always discretionary space for humans
• Recognizes the Law of Stretched Systems
Wednesday, October 3, 12
![Page 112: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/112.jpg)
SUMMARY
Wednesday, October 3, 12
![Page 113: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/113.jpg)
So what can we do?
“In preparing for battle, I have always found that plans are useless but planning is indispensable.”
- Eisenhower
Wednesday, October 3, 12
![Page 114: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/114.jpg)
So what can we do?We develop our Non-Technical Skills
• Situational Awareness
• Communication
• Decision Making
• Improvisation
• Crew Resource Management (CRM)
Wednesday, October 3, 12
![Page 115: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/115.jpg)
So what can we do?
We tailor our environment to adapt
• Tooling to support SA
• Learning from outages (PostMortem)
• Anticipating problems (PreMortem)
• Gather Meta-Metrics
Wednesday, October 3, 12
![Page 116: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/116.jpg)
Wednesday, October 3, 12
![Page 117: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/117.jpg)
THE ENDWednesday, October 3, 12
![Page 118: Velocity EU 2012 Escalating Scenarios: Outage Handling Pitfalls](https://reader031.vdocuments.net/reader031/viewer/2022013003/54c7fa7a4a79594f2e8b4611/html5/thumbnails/118.jpg)
Credits
• http://www.flickr.com/photos/28650594@N03/2718027136/
• http://www.flickr.com/photos/telstar/2816887784/
• http://www.flickr.com/photos/soldiersmediacenter/3855375117/
• http://www.flickr.com/photos/19743256@N00/2640217771/
• http://www.flickr.com/photos/29456680@N06/6148754691/
• https://www.flickr.com/photos/spencediddy/3197199659/sizes/l/in/faves-allspaw/
• http://www.flickr.com/photos/splorp/64027565/sizes/l/in/photostream/
Wednesday, October 3, 12