devops days tel aviv 2013: what exactly is anti-fragile in devops? - asher sterkin

Cisco Confidential © 2013 Cisco and/or its affiliates. All rights reserved. 1

What is Exactly Anti-Fragile in DevOps? Asher Sterkin

Distinguished Engineer, SPVSS, Cisco Video Systems, Israel

September 30, 2013


Antifragile

Some things benefit from shocks…

volatility, randomness, disorder,

and stressors and love adventure,

risk, and uncertainty… there is no

word for the exact opposite of

fragile. Let’s call it antifragile. Nassim N. Taleb, “Antifragile. Things that gain from

disorder”


The Book and Reactions

I think this concept is incredibly

powerful when applied to systems

and organizational architecture.

Jez Humble, “On Antifragility in Systems and

Organizational Structure”

http://continuousdelivery.com/2013/01/on-antifragility-in-systems-and-organizational-architecture/







The Netflix cloud architecture is

anti-fragile… The Netflix culture is

anti-fragile… Getting stronger

through failure is the basis of anti-

fragility. Avoiding failure at all costs

… makes you brittle and

vulnerable...

Adrian Cockcroft, “Looking back at 2012 with

pointers to 2013”

http://perfcap.blogspot.co.il/2013/01/looking-back-at-2012-with-pointers-to.html

http://perfcap.blogspot.co.il/2013/01/looking-back-at-2012-with-pointers-to.html



If the idea is nice and neat,

however, the book that houses it

is just the opposite. It is a big,

baggy, sprawling mess.

David Runciman, review of the book in Guardian

November 21, 2012

http://www.theguardian.com/books/2012/nov/21/antifragile-how-to-live-nassim-nicholas-taleb-review


Larger Body of Knowledge

• Complex Adaptive Systems

• Highly-Optimized Tolerance

• Technology Development Cycle

• Disruptive Innovations

• Product Development Flow

• Lean Start Up


For Today • De-fragilization

• Skin in the Game

• Barbell

• Asymmetric Pay-off


DevOps Areas

OPS DEV

Area 1: extend delivery to

production

Area 2: extend operations

feedback to project

Area 3: embed project knowledge into operations

Area 4: embed operations knowledge into project

Patrick Debois: “Codifying devops practices”

http://www.jedi.be/blog/2012/05/12/codifying-devops-area-practices/





De-Fragilization


“Beauty plus pity-that is the closest we can get to a definition of art. Where there is beauty there is pity for the simple reason that beauty must die: beauty always dies, the manner dies with the matter, the world dies with the individual.”

V. Nabokov, “Lecture on Metamorphosis”

http://www.vahidnab.com/kafka.htm


DevOps Areas

OPS DEV


production


feedback to project




Continuous Delivery

... to exert a constant stress on

your delivery and deployment

process to reduce its fragility so

that releasing becomes a

boring, low-risk activity.

Jez Humble, “On Antifragility in Systems and

Organizational Structure”








Large batches increase cycle time

Large batches increase variability in flow

John Allspaw: “Ops Meta-Metrics”,

slides 103-109

http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change




Reducing batch size accelerates feedback

Reducing batch size reduces overhead

Reducing batch size reduces risk


Batch size and Bottlenecks Reduce batch

size before you

attack

bottlenecks


DevOps Areas

OPS DEV


production


feedback to project




“Resilience through Failure”

http://www.slideshare.net/atseitlin/resiliency-through-failure-qconny-2013

http://www.slideshare.net/atseitlin/resiliency-through-failure-qconny-2013


Resilience to failure is a

lofty goal. It enables a

system to survive and

withstand failure. There's an

even higher peak to strive

for, however: making the

system stronger and better

with each failure.

A. Tseitlin, “The Antifragile Organization”

Culture of Continuous Learning

http://queue.acm.org/detail.cfm?id=2499552





E. Deming, “Out of Crisis”

“Confusing common causes with special causes will only make things worse


Resilience, Reliability, Robustness

Normal Operation

MTBF

Failure!

MTTD MTTR

Max Downtime

Normal Operation

MTBF

Max Data

Lost


It’s me

“Obsessive protection of the system against extremely rare events makes it more fragile. Resilience comes before the last percentiles of reliability.


DevOps Areas

OPS DEV


production


feedback to project




Give ‘em pagers!


“DevOps: architects, developers, QA/QC, system engineers, and IT cooperate to maximize the company value

It’s me again


Barbell


DevOps Areas

OPS DEV


production


feedback to project



© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32 John Alsspaw, “Dev and Ops Collaboration”

http://www.slideshare.net/jallspaw/dev-and-ops-collaboration-and-awareness-at-etsy-and-flickr





Developers

• Responding to outages, takes on-call

• Alerting systems thresholding, design

• Architecture design and review

• Building metrics collection

• Application configuration

• Shipping public-facing code

• Responding to outages, takes on-call

• Alerting systems thresholding, design

• Architecture design and review

• Building metrics collection

• Application configuration

• Infrastructure buildout/management

Operations

John Allspaw, “Reply to NoOps @ Netflix”

http://perfcap.blogspot.co.il/2012/03/ops-devops-and-noops-at-netflix.html





DevOps Areas

OPS DEV


production


feedback to project


Area 4: embed operations knowledge into project No, not really


Every analogy has its limit.

One just needs to learn where

to stop.


N. Taleb, “Antifragile”

“The first step to anti-fragility consists in decreasing downside…This brings us to the solution in the form of barbell … Away from Golden Middle.


Away from golden Middle

Risk

Aversion

Risk

Loving


Niek Bartholomeus,

“DevOps For Dinosaurs”

https://speakerdeck.com/niekbartho/devops-for-dinosaurs




Cisco Confidential © 2013 Cisco and/or its affiliates. All rights reserved. 40 Jeremy Edberg, “DevOps at Netflix”

Risk Aversion Risk Loving

http://www.slideshare.net/jedberg/devops-at-netflix-reinvent






N. Taleb, “Antifragile”

“The downside/lost should be known and protected, not probability


It’s me

“ You never know where from your next outage or cyber attack will come from and when. The maximal downtime and maximal data lost should be known and guaranteed regardless of probabilities.


Asymmetric Pay-Offs

x

Gain

/Loss f

(x)

Pain

Gain


DevOps Areas

OPS DEV


production


feedback to project




N. Taleb

“Never be sucker. Period!

Thanks