failure as success devops roadtrip seattle 2016

‘Failure’ as ‘Success’: The Mindset, The Methods, and The Land Mines

J. Paul Reed / Release Engineering Approaches

VictorOps DevOps Roadtrip 2016, Seattle

Get a Buddy!

#DevOpsRoadTrip @jpaulreed

Get Some Paper!

#DevOpsRoadTrip @jpaulreed

Thought Experiment: An Incident in the JFK “Datacenter”

Dev OpsFrom: Is Your Team Instrument Rated?

vimeo.com/69081899#DevOpsRoadTrip @jpaulreed

#DevOpsRoadTrip @jpaulreed

A Question

Was this event a Success

or a Failure?

#DevOpsRoadTrip @jpaulreed

J. Paul Reed

✤ @jpaulreed on

✤ Alum of The Ship Show

✤ 15 Years in Build/Release Engineering

✤ Now, a DevOps Consultant™

✤ A Master of Science candidate in Human Factors & Systems Safety

#DevOpsRoadTrip @jpaulreed

“Tell Me About Your Feelings About Failure…”

#DevOpsRoadTrip @jpaulreed

#DevOpsRoadTrip @jpaulreed

#DevOpsRoadTrip @jpaulreed

#DevOpsRoadTrip @jpaulreed

‘Failure’ as ‘Success’: The

Mindset#DevOpsRoadTrip @jpaulreed

Success

Failure

Accident

Human Error

Incident

BlameRetrospective

#DevOpsRoadTrip @jpaulreed

Success

Failure

Accident

Safety

Human Error

Incident

BlameRetrospective

#DevOpsRoadTrip @jpaulreed

Safety as “Energy & Barriers”

✤ “Failure” as a release of energy

✤ “Design” safety in

✤ “Defense in depth”

#DevOpsRoadTrip @jpaulreed

#DevOpsRoadTrip @jpaulreed

Safety Impossible Due to “Normal Accidents”✤ Degree of linearity vs.

complexity

✤ Degree of system “coupling”

✤ Certain systems—complex, tightly coupled ones—will have accidents

✤ This is “normal.”(Also? Unavoidable.)

#DevOpsRoadTrip @jpaulreed

Complexity + Tight Coupling =‘Normal’ Disasters

#DevOpsRoadTrip @jpaulreed

Safety in “High Reliability Orgs”

✤ Constant, active learning

✤ Decentralized, active review

✤ Rank deemphasized (oddly?)

✤ Crew rotation

✤ “Success” may be failure

#DevOpsRoadTrip @jpaulreed

Energy & Barriers

“Normal” Accidents

High Reliability Orgs

#DevOpsRoadTrip @jpaulreed

Energy & Barriers

“Normal” Accidents

High Reliability Orgs

Static Processes & (Repeated) Defenses

Active Defenses & Processes

#DevOpsRoadTrip @jpaulreed

Energy & Barriers

“Normal” Accidents

High Reliability Orgs

Technical / Engineering Solutions

Solutions Inherent to People, the Org, the

Way of Working

#DevOpsRoadTrip @jpaulreed

Energy & Barriers

“Normal” Accidents

High Reliability Orgs

Backups, Redundancy,

DR sites

Drilled Incident Response,

Retrospectives, Red-teaming, Value Streams

“Simplify,” Decompose,

Microservices

#DevOpsRoadTrip @jpaulreed

‘Failure’ as ‘Success’: The

Methods#DevOpsRoadTrip @jpaulreed

Successful Failure: The Methods

✤ Inject Failure

#DevOpsRoadTrip @jpaulreed

#DevOpsRoadTrip @jpaulreed

Successful Failure: The Methods

✤ Inject Failure

✤ Incident Command & Crews

#DevOpsRoadTrip @jpaulreed

Successful Failure: The Methods

✤ Inject Failure

✤ Incident Command& Crews

✤ Postmortems!

#DevOpsRoadTrip @jpaulreed

Successful Failure: The Methods

✤ Inject Failure

✤ Incident Command& Crews

✤ (Blameless?)Postmortems!

#DevOpsRoadTrip @jpaulreed

Blameless?

http://jpaulreed.com/blame-aware-postmortems#DevOpsRoadTrip @jpaulreed

Debrief the Actors

#DevOpsRoadTrip @jpaulreed

Gather the Data

#DevOpsRoadTrip @jpaulreed

Create a Timeline

–The Field Guide to Understanding Human Error#DevOpsRoadTrip @jpaulreed

Let’s Create a Timeline!

Write

It On

A Sticky

#DevOpsRoadTrip @jpaulreed

Successful Failure: The Methods

✤ Inject Failure

✤ Incident Command& Crews

✤ (Blameless?) Postmortems!

✤ “Retrospective- ready” Infrastructure

#DevOpsRoadTrip @jpaulreed

Your “Black Boxes”

✤ Chat Ops

✤ TMate

✤ Incident response / management tools

#DevOpsRoadTrip @jpaulreed

‘Failure’ as ‘Success’: The

Land Mines#DevOpsRoadTrip @jpaulreed

Successful Failure: The Land Mines

✤ OrganizationalIncompatibility

#DevOpsRoadTrip @jpaulreed

The best single question for testing an organization’s character is:

What happens when people make mistakes?

— Robert I. Sutton

#DevOpsRoadTrip @jpaulreed

Successful Failure: The Land Mines

✤ Organizational Incompatibility

✤ Only Certain Groups“Get” to Fail

#DevOpsRoadTrip @jpaulreed

An Antipattern: Stopping the Line is a “Privilege”

#DevOpsRoadTrip @jpaulreed

Successful Failure: The Land Mines

✤ Organizational Incompatibility

✤ Only Certain Groups“Get” to Fail

✤ Forgetting to Dampen Failure (Where Possible)

#DevOpsRoadTrip @jpaulreed

Successful Failure: The Land Mines

✤ Only Reviewing Failure

#DevOpsRoadTrip @jpaulreed

#DevOpsRoadTrip @jpaulreed

Successful Failure: The Land Mines

✤ Only Reviewing Failure

✤ Forgetting About Bias

#DevOpsRoadTrip @jpaulreed

A Bit About Biases

Hindsight Bias#DevOpsRoadTrip @jpaulreed

A Bit About Biases

Correspondence Bias#DevOpsRoadTrip @jpaulreed

A Bit About Biases

Correspondence Bias#DevOpsRoadTrip @jpaulreed

A Sea Of Biases

Attention Bias

Backfire Effect

Belief Bias

Confirmation Bias

Congruence Bias Decoy Effect

Contrast Effect

Expectation Bias

Hindsight Effect

IKEA Effect

Impact Bias

Information Bias

Sunk Cost EffectNegativity Effect

Irrational Escalation Effect

Pro-innovation Bias

Optimism Bias

Omission Bias

Rhyme-as-reason Effect

Social Comparison Bias

Social Desirability Bias

Time-Saving Bias

Survivorship Bias

Zero-Risk Bias

Illusion of Truth Effect

Stereotype Bias

#DevOpsRoadTrip @jpaulreed

A Simple, Surprising Example

Bias is built in to the way our brains function.

#DevOpsRoadTrip @jpaulreed

Successful Failure: The Land Mines

✤ Only Reviewing Failure

✤ Forgetting About Bias

✤ De-prioritizing Retrospectives / Learning Processes

#DevOpsRoadTrip @jpaulreed

#DevOpsRoadTrip @jpaulreed

So What Now?

#DevOpsRoadTrip @jpaulreed

Takeaway # 1

Stop thinking about incidents as “events that went wrong.”

Start thinking about incidents in terms of your team’s response.

The key to reframing “failure”:

#DevOpsRoadTrip @jpaulreed

Takeaway # 1

The key to reframing “failure”:Develop Your

“Incident Immune System” (It Gets Better

the More You Use It!)#DevOpsRoadTrip @jpaulreed

Takeaway # 2

To make this practical, your operations and infrastructure

need to be “retrospective-ready.”

#DevOpsRoadTrip @jpaulreed

J. Paul Reed www.jpaulreed.com

@jpaulreed

www.release-approaches.comSimply Ship. Every Time.

Adoption of Agile & DevOps methodologies is a journey, and every person and organization’s journey is going to be different.

-J. Paul Reed

DevOps in Practice

J. Paul Reed

http://jpaulreed.com/devops-in-practice