winnipeg isaca security is dead, rugged devops

109
Session ID: Gene Kim Winnipeg ISACA April 26, 2012 Infosec In The New World Order: Rugged DevOps and More…

Upload: gene-kim

Post on 08-May-2015

1.044 views

Category:

Business


0 download

TRANSCRIPT

Page 1: Winnipeg ISACA Security is Dead, Rugged DevOps

Session ID:

Gene Kim

Winnipeg ISACA

April 26, 2012

Infosec In The New World Order: Rugged DevOps and More…

Page 2: Winnipeg ISACA Security is Dead, Rugged DevOps

Where Did The High Performers Come From?

Page 3: Winnipeg ISACA Security is Dead, Rugged DevOps

Agenda

Background of research The big unsolved problem What is Rugged? What is DevOps? How do you do Rugged DevOps? Things you can do right away

3

Page 4: Winnipeg ISACA Security is Dead, Rugged DevOps

High Performing IT Organizations High performers maintain a posture of compliance

Fewest number of repeat audit findings One-third amount of audit preparation effort

High performers find and fix security breaches faster 5 times more likely to detect breaches by automated control 5 times less likely to have breaches result in a loss event

When high performers implement changes… 14 times more changes One-half the change failure rate One-quarter the first fix failure rate 10x faster MTTR for Sev 1 outages

When high performers manage IT resources… One-third the amount of unplanned work 8 times more projects and IT services 6 times more applications

Source: IT Process Institute, 2008

Page 5: Winnipeg ISACA Security is Dead, Rugged DevOps

Visible Ops: Playbook of High Performers

The IT Process Institute has been studying high-performing organizations since 1999 What is common to all the high

performers? What is different between them

and average and low performers?

How did they become great? Answers have been codified in

the Visible Ops Methodology

www.ITPI.org

Page 6: Winnipeg ISACA Security is Dead, Rugged DevOps

2007: Three Controls Predict 60% Of Performance

To what extent does an organization define, monitor and enforce the following? Standardized configuration strategy Process discipline Controlled access to production systems

Source: IT Process Institute, 2008

Page 7: Winnipeg ISACA Security is Dead, Rugged DevOps

The Downward SpiralOperations Sees… Fragile applications are prone to failure Long time required to figure out “which

bit got flipped” Detective control is a salesperson Too much time required to restore

service Too much firefighting and unplanned

work Urgent security rework and

remediation Planned project work cannot complete Frustrated customers leave Market share goes down Business misses Wall Street

commitments Business makes even larger promises

to Wall Street

Dev Sees… More urgent, date-driven projects

put into the queue Even more fragile code (less

secure) put into production More releases have increasingly

“turbulent installs” Release cycles lengthen to

amortize “cost of deployments” Failing bigger deployments more

difficult to diagnose Most senior and constrained IT ops

resources have less time to fix underlying process problems

Ever increasing backlog of work that cold help the business win

Ever increasing amount of tension between IT Ops, Development, Design…

These aren’t IT or Infosec problems…These are business problems!

Page 8: Winnipeg ISACA Security is Dead, Rugged DevOps

10

My Mission: Figure Out How Break The IT Core Chronic Conflict

Every IT organization is pressured to simultaneously: Respond more quickly to urgent business needs Provide stable, secure and predictable IT service

Source: The authors acknowledge Dr. Eliyahu Goldratt, creator of the Theory of Constraints and author of The Goal, has written extensively on the theory and practice of identifying and resolving core, chronic conflicts.

Words often used to describe process improvement:“hysterical, irrelevant, bureaucratic, bottleneck, difficult to understand, not

aligned with the business, immature, shrill, perpetually focused on irrelevant technical minutiae…”

Page 9: Winnipeg ISACA Security is Dead, Rugged DevOps

Good News: It Can Be Done

Bad News: You Can’t Do It Alone

Page 10: Winnipeg ISACA Security is Dead, Rugged DevOps

Ops

Page 11: Winnipeg ISACA Security is Dead, Rugged DevOps

QA And Test

Source: Flickr: vandyll

Page 12: Winnipeg ISACA Security is Dead, Rugged DevOps

Development

Page 13: Winnipeg ISACA Security is Dead, Rugged DevOps

Infosec

Page 14: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: Flickr: birdsandanchors

Product Management And Design

Page 15: Winnipeg ISACA Security is Dead, Rugged DevOps

But…

18

Page 16: Winnipeg ISACA Security is Dead, Rugged DevOps

Ludicrous Speed?

19

Page 17: Winnipeg ISACA Security is Dead, Rugged DevOps

Ludicrous Speed

20

Page 18: Winnipeg ISACA Security is Dead, Rugged DevOps

Ludicrous Speed!

21

Page 19: Winnipeg ISACA Security is Dead, Rugged DevOps

Ludicrous Fail?!

22

Page 20: Winnipeg ISACA Security is Dead, Rugged DevOps

DevOps:The Shining Beacon Of Hope

Page 21: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: John Allspaw

Page 22: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: John Allspaw

Page 23: Winnipeg ISACA Security is Dead, Rugged DevOps
Page 24: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: John Allspaw

Page 25: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: John Allspaw

Page 26: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: Theo Schlossnagle

Page 27: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: Theo Schlossnagle

Page 28: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: Theo Schlossnagle

Page 29: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: John Jenkins, Amazon.com

Page 30: Winnipeg ISACA Security is Dead, Rugged DevOps

What Is Rugged?

33

Page 31: Winnipeg ISACA Security is Dead, Rugged DevOps

Rugged Software Development

Joshua Corman, David Rice, Jeff Williams2010

Page 33: Winnipeg ISACA Security is Dead, Rugged DevOps
Page 34: Winnipeg ISACA Security is Dead, Rugged DevOps

RUGGED SOFTWARE

Page 35: Winnipeg ISACA Security is Dead, Rugged DevOps

…so software not only needs to be…

Page 36: Winnipeg ISACA Security is Dead, Rugged DevOps

FAST

Page 37: Winnipeg ISACA Security is Dead, Rugged DevOps

AGILE

Page 38: Winnipeg ISACA Security is Dead, Rugged DevOps

Are You Rugged?

Page 39: Winnipeg ISACA Security is Dead, Rugged DevOps

HARSH

Page 40: Winnipeg ISACA Security is Dead, Rugged DevOps

UNFRIENDLY

Page 41: Winnipeg ISACA Security is Dead, Rugged DevOps

THE MANIFESTO

Page 42: Winnipeg ISACA Security is Dead, Rugged DevOps
Page 43: Winnipeg ISACA Security is Dead, Rugged DevOps

I recognize that my code will be used in ways I cannot anticipate, in ways it was not designed,

and for longer than it was ever intended.

Page 44: Winnipeg ISACA Security is Dead, Rugged DevOps
Page 45: Winnipeg ISACA Security is Dead, Rugged DevOps

www.ruggedsoftware.orgCrossTalk

http://www.crosstalkonline.org/issues/marchapril-2011.html

Page 46: Winnipeg ISACA Security is Dead, Rugged DevOps

What Is Rugged DevOps?

49

Page 47: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: James Wickett

Page 48: Winnipeg ISACA Security is Dead, Rugged DevOps

Source: James Wickett

Page 49: Winnipeg ISACA Security is Dead, Rugged DevOps
Page 50: Winnipeg ISACA Security is Dead, Rugged DevOps

DevOps: It’s A Real Movement

I would never do another startup that didn’t employ DevOps like principles

It’s not just startups – it’s happening in the enterprise and in public sector, too

I believe working in DevOps environments will be a necessary skillset 5 years from now

Just as Agile helped Dev regain trust with the business, DevOps will help all of IT

Page 51: Winnipeg ISACA Security is Dead, Rugged DevOps

How Do You Do DevOps?

67

Page 52: Winnipeg ISACA Security is Dead, Rugged DevOps

The Prescriptive DevOps Cookbook

“DevOps Cookbook” Authors

Patrick DeBois, Mike Orzen, John Willis

Goals

Codify how to start and finish DevOps transformations

How does Development, IT Operations and Infosec become dependable partners

Describe in detail how to replicate the transformations describe in “When IT Fails: The Novel”

Page 53: Winnipeg ISACA Security is Dead, Rugged DevOps

“The Goal” by Dr. Eliyahu Goldratt

Page 54: Winnipeg ISACA Security is Dead, Rugged DevOps

71

Page 55: Winnipeg ISACA Security is Dead, Rugged DevOps

72

Page 56: Winnipeg ISACA Security is Dead, Rugged DevOps

The First Way:Systems Thinking

Page 57: Winnipeg ISACA Security is Dead, Rugged DevOps

The First Way:Systems Thinking

(Business) (Customer)

Page 58: Winnipeg ISACA Security is Dead, Rugged DevOps

The First Way:Systems Thinking (Left To Right)

Never pass defects to downstream work centers Never allow local optimization to create global

degradation Increase flow: elevate bottlenecks, reduce WIP,

throttle release of work, reduce batch sizes Understanding where reliance is placed

Page 59: Winnipeg ISACA Security is Dead, Rugged DevOps

Phase 1: Extend the Agile CI/CR Processes

Assign Ops person into Dev team Create one-step Dev, Test and Production

environment creation procedure in Sprint 0 Create the one-step automated code

deployment procedure Define roles of Dev, QA, Prod Mgmt and Infosec

Page 60: Winnipeg ISACA Security is Dead, Rugged DevOps

The First Way:Systems Thinking: Infosec Insurgency

Have infosec attend the daily Agile standups Gain awareness of what the team is working on

Find the automated infrastructure project team (e.g., puppet, chef) Provide hardening guidance Integrate and extend their production configuration

monitoring Find where code packaging is performed

Integrate security testing pre- and post-deployment Integrate into continuous integration and release

process Add security test scripts to automated test library

Page 61: Winnipeg ISACA Security is Dead, Rugged DevOps

The First Way:Outcomes

Determinism in the release process

Continuation of the Agile and CI/CR processes

Creating single repository for code and environments

Packaging responsibility moves to development

Consistent Dev, QA, Int, and Staging environments, all properly built before deployment begins

Decrease cycle time

Reduce deployment times from 6 hours to 45 minutes Refactor deployment process that had 1300+ steps

spanning 4 weeks Faster release cadence

Page 62: Winnipeg ISACA Security is Dead, Rugged DevOps

The Second Way:Amplify Feedback Loops

Page 63: Winnipeg ISACA Security is Dead, Rugged DevOps

The Second Way:Amplify Feedback Loops (Right to Left)

Protect the integrity of the entire system of work, versus completion of tasks

Expose visual data so everyone can see how their decisions affect the entire system

Page 64: Winnipeg ISACA Security is Dead, Rugged DevOps

Phase 2: Extend Release Process And Create Right -> Left Feedback Loops

Embed Dev into Ops escalation process Invite Dev to post-mortems/root cause analysis

meeting Create necessary rollback procedures (instead

of fixing forward) Create application monitoring/metrics to aid in

Ops work (e.g., incident/problem management) Actively manage flow of work across org

boundaries

Page 65: Winnipeg ISACA Security is Dead, Rugged DevOps

The Second Way:Amplify Feedback Loops: Infosec Insurgency

Extend criteria of what changes/deploys cannot be made without triggering full retest

Create reusable Infosec use and abuse stories that can be added to every project “Handle peak traffic of 4MM users and constant 4-6

Gb/sec Anonymous DDoS attacks” Integrate Infosec and IR into the Ops/Dev escalation

processes (e.g., RACI) Pre-enable, shield streamline successful audits

Document separation of duty and compensating controls Don’t let them disrupt the work

Page 66: Winnipeg ISACA Security is Dead, Rugged DevOps

The Second Way:Outcomes

Andon cords that stop the production line Kanban to control work Project freeze to reduce work in process Eradicating “quick fixes” that circumvent the process Ops user stories are part of the Agile planning

process Better build and deployment systems More stable environment Happier and more productive staff

Page 67: Winnipeg ISACA Security is Dead, Rugged DevOps

Definition: Kanban Board

Signaling tool to reduce WIP and increase flow

84

Page 68: Winnipeg ISACA Security is Dead, Rugged DevOps

The Third Way:Culture Of Continual Experimentation And Learning

Page 69: Winnipeg ISACA Security is Dead, Rugged DevOps

The Third Way:Culture Of Continual Experimentation And Learning

Foster a culture that rewards: Experimentation (taking risks) and learning from

failure Repetition is the prerequisite to mastery

Why? You need a culture that keeps pushing into the danger

zone And have the habits that enable you to survive in the

danger zone

Page 70: Winnipeg ISACA Security is Dead, Rugged DevOps

You Don’t Choose Chaos Monkey…Chaos Monkey Chooses You

Page 71: Winnipeg ISACA Security is Dead, Rugged DevOps

Phase 3: Organize Dev and Ops To Achieve Organizational Goals

Allocate 20% of Dev cycles to non-functional requirements

Build Ops user stories and environments in Dev that can be reused across all projects (e.g., deployment, capacity, security)

Integrate fault injection and resilience into design, development and production (e.g., Chaos Monkey)

Prioritize backlog to manage technical debt

Page 72: Winnipeg ISACA Security is Dead, Rugged DevOps
Page 73: Winnipeg ISACA Security is Dead, Rugged DevOps

The Third Way:Culture Of Continual Experimentation And Learning: Infosec

Add Infosec fixes to the Agile backlog Make technical debt visible Help prioritize work against features and other non-functional requirements

Weaponize the Security Monkey Evil/Fuzzy/Chaotic Monkey Eridicate SQLi and XSS defects in our lifetime

Let loose the Security Monkies and the Simian Army Eliminate needless complexity Become the standard bearer: 20% of Dev cycles spent on non-

functional requirements Take work out of the system Keep decreasing cycle time: it increases work that the system

can achieve

Page 74: Winnipeg ISACA Security is Dead, Rugged DevOps

The Third Way:Outcomes

Dedicated time spent on improving daily work (best practice: 20% of Dev dedicated to non-functional requirements)

Continual reduction of unplanned work

More cycles for planned work

Projects completed to pay down technical debt and increase flow

Elimination of needless complexity

More resilient code and environments

Balancing nimbleness and practiced repetition

Enabling wider range of risk/reward balance

Page 75: Winnipeg ISACA Security is Dead, Rugged DevOps

What Does Transformation Feel Like?

92

Page 76: Winnipeg ISACA Security is Dead, Rugged DevOps

Find What’s Most Important First

Page 77: Winnipeg ISACA Security is Dead, Rugged DevOps

Quickly Find What Is Different…

Page 78: Winnipeg ISACA Security is Dead, Rugged DevOps

Before Something Bad Happens…

Page 79: Winnipeg ISACA Security is Dead, Rugged DevOps

Find Risk Early…

Page 80: Winnipeg ISACA Security is Dead, Rugged DevOps

Communicate It Effectively To Peers…

Page 81: Winnipeg ISACA Security is Dead, Rugged DevOps

Hold People Accountable…

Page 82: Winnipeg ISACA Security is Dead, Rugged DevOps

Based On Objective Evidence…

Page 83: Winnipeg ISACA Security is Dead, Rugged DevOps

Answer Important Questions…

Page 84: Winnipeg ISACA Security is Dead, Rugged DevOps

Recognize Compounding Technical Debt…

Page 85: Winnipeg ISACA Security is Dead, Rugged DevOps

That Gets Worse…

Page 86: Winnipeg ISACA Security is Dead, Rugged DevOps

And Fixing It…

Source: Pingdom

Page 87: Winnipeg ISACA Security is Dead, Rugged DevOps

Have What We Need, When When We Need It…

Page 88: Winnipeg ISACA Security is Dead, Rugged DevOps

Big Things Get Done Quickly…

Page 89: Winnipeg ISACA Security is Dead, Rugged DevOps

Ever Increasing Situational Mastery…

Page 90: Winnipeg ISACA Security is Dead, Rugged DevOps

Help The Business Win…

Page 91: Winnipeg ISACA Security is Dead, Rugged DevOps

With Support From Your Peers…

Page 92: Winnipeg ISACA Security is Dead, Rugged DevOps

And Do More With Less Effort…

Page 93: Winnipeg ISACA Security is Dead, Rugged DevOps

This Is An Important ProblemOperations Sees… Fragile applications are prone to failure Long time required to figure out “which

bit got flipped” Detective control is a salesperson Too much time required to restore

service Too much firefighting and unplanned

work Urgent security rework and

remediation Planned project work cannot complete Frustrated customers leave Market share goes down Business misses Wall Street

commitments Business makes even larger promises

to Wall Street

Dev Sees… More urgent, date-driven projects

put into the queue Even more fragile code (less

secure) put into production More releases have increasingly

“turbulent installs” Release cycles lengthen to

amortize “cost of deployments” Failing bigger deployments more

difficult to diagnose Most senior and constrained IT ops

resources have less time to fix underlying process problems

Ever increasing backlog of work that cold help the business win

Ever increasing amount of tension between IT Ops, Development, Design…

Page 94: Winnipeg ISACA Security is Dead, Rugged DevOps

When IT Fails: The Novel and The DevOps Cookbook

Coming in July 2012

“In the tradition of the best MBA case studies, this book should be mandatory reading for business and IT graduates alike.”Paul Muller, VP Software Marketing, Hewlett-Packard

“The greatest IT management book of our generation.”Branden Williams, CTO Marketing, RSA

Gene Kim, Tripwire founder, Visible Ops co-author

Page 95: Winnipeg ISACA Security is Dead, Rugged DevOps

When IT Fails: The Novel and The DevOps Cookbook

Our mission is to positively affect the lives of 1 million IT workers by 2017

If you would like the “Top 10 Things You Need To Know About DevOps,” sample chapters and updates on the book:

Sign up at http://itrevolution.com Email [email protected] Hand me a business card

Gene Kim, Tripwire founder, Visible Ops co-author

Page 96: Winnipeg ISACA Security is Dead, Rugged DevOps

Thank You

113

Page 97: Winnipeg ISACA Security is Dead, Rugged DevOps

Appendix

114

Page 98: Winnipeg ISACA Security is Dead, Rugged DevOps

Resources From the IT Process Institute

www.itpi.org Both Visible Ops Handbooks ITPI IT Controls Performance Study

Rugged Software by Corman, et al: http://ruggedsoftware.org

“Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation” by Humble, Farley

Follow us… @JoshCorman, @RealGeneKim mailto:[email protected] http://realgenekim.me/blog

Page 99: Winnipeg ISACA Security is Dead, Rugged DevOps

Common Traits of High Performers

Source: IT Process Institute

Change management

Causality

Compliance and continual reduction of operational variance

Culture of…

Integration of IT operations/security via problem/change management Processes that serve both organizational needs and business objectives Highest rate of effective change

Highest service levels (MTTR, MTBF) Highest first fix rate (unneeded rework)

Production configurations Highest level of pre-production staffing Effective pre-production controls Effective pairing of preventive and detective controls

Page 100: Winnipeg ISACA Security is Dead, Rugged DevOps

Visible Ops: Playbook of High Performers

The IT Process Institute has been studying high-performing organizations since 1999 What is common to all the high

performers? What is different between them and

average and low performers? How did they become great?

Answers have been codified in the Visible Ops Methodology

The “Visible Ops Handbook” is available from the ITPI

www.ITPI.org

Page 101: Winnipeg ISACA Security is Dead, Rugged DevOps

IT Operations Increases Process Rigor

Standardize deployment Standardize unplanned work: make it repeatable Modify first response: ensure constrained

resources have all data at hand to diagnose Elevate preventive activities to reduce incidents

Page 102: Winnipeg ISACA Security is Dead, Rugged DevOps

Help Development…

Help them see downstream effects Unplanned work comes at the expense of planned

work Technical debt retards feature throughput Environment matters as much as the code

Allocate time for fault modeling, asking “what could go wrong?” and implementing countermeasures

Page 103: Winnipeg ISACA Security is Dead, Rugged DevOps

Help QA…

Ensure test plans cover not only code functionality, but also: Suitability of the environment the code runs in The end-to-end deployment process

Help find variance… Functionality, performance, configuration Duration, wait time and handoff errors, rework, …

Page 104: Winnipeg ISACA Security is Dead, Rugged DevOps

John Pesche, CISO

CISO for 12 years 39 years old Aggressive career

climber Ex-Big Four auditor

Page 105: Winnipeg ISACA Security is Dead, Rugged DevOps

John Pesche, CISO

Page 106: Winnipeg ISACA Security is Dead, Rugged DevOps
Page 107: Winnipeg ISACA Security is Dead, Rugged DevOps
Page 108: Winnipeg ISACA Security is Dead, Rugged DevOps

John Pesche, CISO

Page 109: Winnipeg ISACA Security is Dead, Rugged DevOps

John Pesche, CISO