failure as success devops roadtrip seattle 2016

‘Failure’ as ‘Success’: The Mindset, The Methods, and The Land Mines

J. Paul Reed / Release Engineering Approaches

VictorOps DevOps Roadtrip 2016, Seattle

Get a Buddy!

#DevOpsRoadTrip @jpaulreed

Get Some Paper!


Thought Experiment: An Incident in the JFK “Datacenter”

Dev OpsFrom: Is Your Team Instrument Rated?

vimeo.com/69081899#DevOpsRoadTrip @jpaulreed

https://vimeo.com/69081899

A Question

Was this event a Success

or a Failure?


J. Paul Reed

✤ @jpaulreed on

✤ Alum of The Ship Show

✤ 15 Years in Build/Release Engineering

✤ Now, a DevOps Consultant™

✤ A Master of Science candidate in Human Factors & Systems Safety


http://twitter.com/jpaulreed

http://theshipshow.com/

“Tell Me About Your Feelings About Failure…”


‘Failure’ as ‘Success’: The

Mindset#DevOpsRoadTrip @jpaulreed

Success

Failure

Accident

Human Error

Incident

BlameRetrospective


Success

Failure

Accident

Safety

Human Error

Incident

BlameRetrospective


Safety as “Energy & Barriers”

✤ “Failure” as a release of energy

✤ “Design” safety in

✤ “Defense in depth”


Safety Impossible Due to “Normal Accidents”✤ Degree of linearity vs.

complexity

✤ Degree of system “coupling”

✤ Certain systems—complex, tightly coupled ones—will have accidents

✤ This is “normal.”(Also? Unavoidable.)


Complexity + Tight Coupling =‘Normal’ Disasters


Safety in “High Reliability Orgs”

✤ Constant, active learning

✤ Decentralized, active review

✤ Rank deemphasized (oddly?)

✤ Crew rotation

✤ “Success” may be failure


Energy & Barriers

“Normal” Accidents

High Reliability Orgs


Energy & Barriers



Static Processes & (Repeated) Defenses

Active Defenses & Processes


Energy & Barriers



Technical / Engineering Solutions

Solutions Inherent to People, the Org, the

Way of Working


Energy & Barriers



Backups, Redundancy,

DR sites

Drilled Incident Response,

Retrospectives, Red-teaming, Value Streams

“Simplify,” Decompose,

Microservices



Methods#DevOpsRoadTrip @jpaulreed

Successful Failure: The Methods

✤ Inject Failure



✤ Inject Failure

✤ Incident Command & Crews



✤ Inject Failure

✤ Incident Command& Crews

✤ Postmortems!



✤ Inject Failure


✤ (Blameless?)Postmortems!


Blameless?

http://jpaulreed.com/blame-aware-postmortems#DevOpsRoadTrip @jpaulreed

http://jpaulreed.com/blame-aware-postmortems

Debrief the Actors


Gather the Data


Create a Timeline

–The Field Guide to Understanding Human Error#DevOpsRoadTrip @jpaulreed

Let’s Create a Timeline!

Write

It On

A Sticky



✤ Inject Failure


✤ (Blameless?) Postmortems!

✤ “Retrospective- ready” Infrastructure


Your “Black Boxes”

✤ Chat Ops

✤ TMate

✤ Incident response / management tools


http://tmate.io/


Land Mines#DevOpsRoadTrip @jpaulreed

Successful Failure: The Land Mines

✤ OrganizationalIncompatibility


The best single question for testing an organization’s character is:

What happens when people make mistakes?

— Robert I. Sutton



✤ Organizational Incompatibility

✤ Only Certain Groups“Get” to Fail


An Antipattern: Stopping the Line is a “Privilege”



✤ Organizational Incompatibility

✤ Only Certain Groups“Get” to Fail

✤ Forgetting to Dampen Failure (Where Possible)



✤ Only Reviewing Failure




✤ Forgetting About Bias


A Bit About Biases

Hindsight Bias#DevOpsRoadTrip @jpaulreed

A Bit About Biases

Correspondence Bias#DevOpsRoadTrip @jpaulreed

A Sea Of Biases

Attention Bias

Backfire Effect

Belief Bias

Confirmation Bias

Congruence Bias Decoy Effect

Contrast Effect

Expectation Bias

Hindsight Effect

IKEA Effect

Impact Bias

Information Bias

Sunk Cost EffectNegativity Effect

Irrational Escalation Effect

Pro-innovation Bias

Optimism Bias

Omission Bias

Rhyme-as-reason Effect

Social Comparison Bias

Social Desirability Bias

Time-Saving Bias

Survivorship Bias

Zero-Risk Bias

Illusion of Truth Effect

Stereotype Bias


A Simple, Surprising Example

Bias is built in to the way our brains function.




✤ Forgetting About Bias

✤ De-prioritizing Retrospectives / Learning Processes


So What Now?


Takeaway # 1

Stop thinking about incidents as “events that went wrong.”

Start thinking about incidents in terms of your team’s response.

The key to reframing “failure”:


Takeaway # 1

The key to reframing “failure”:Develop Your

“Incident Immune System” (It Gets Better

the More You Use It!)#DevOpsRoadTrip @jpaulreed

Takeaway # 2

To make this practical, your operations and infrastructure

need to be “retrospective-ready.”


J. Paul Reed www.jpaulreed.com

@jpaulreed

www.release-approaches.comSimply Ship. Every Time.

http://www.jpaulreed.com

http://www.release-approaches.com

Adoption of Agile & DevOps methodologies is a journey, and every person and organization’s journey is going to be different.

-J. Paul Reed

DevOps in Practice

J. Paul Reed

http://jpaulreed.com/devops-in-practice

http://jpaulreed.com/devops-in-practice

failure as success devops roadtrip seattle 2016

Technology