7 lessons learned when implementing devops. toyota . production. system. ... qa inlcuded down -...

7 Lessons Learned when Implementing DevOps

BRIAN ROCHE@BRIANROCHEBOS

EMILY KAISER© Copyright 2017 Dell Inc.

@DELLEMCDOJO

BRIAN ROCHEDELL EMC DOJO

[email protected]

TWITTER@BRIANROCHEBOS

YOUR PRESENTERS7 LESSONS LEARNEDWHEN IMPLEMENTINGDEVOPS + BONUS

EMILY KAISERDELL EMC DOJO

[email protected]

TWITTER@DELLEMCDOJO



@DELLEMCDOJO

THE TOPICS. 01. WHY ARE WE HERE?

02. WHAT IS DEVOPS?

03. WHY DEVOPS WORKS?

04. USER OBSESSION

05. THE GLUE

06. WE MESS UP A LOT

07. SCALING DEVOPS

08. TRANSFORMATION

09. CLOSING

7 LESSONS LEARNEDWHEN IMPLEMENTINGDEVOPS +BONUS



@DELLEMCDOJO

01. WHY ARE WE HERE?BUSINESS DRIVERS FOR DEVOPSHOW TO GET STARTEDREVENUE IS SCORECARD



@DELLEMCDOJO

AMOON MISSION

SEEMS TOOEASY.

I N T H E ‘ 6 0 ST E C H N O L O G Y

P U T A M A N O N T H E M O O N .

W E H A V E C O N T I N U E DT O A I M F O R T H E S T A R S .

W I T H A D V A N C E S I N T E C H N O L O G Y T O D A Y ,



@DELLEMCDOJO

THEFUTURE

LOOKSBRIGHT.

M O O N M I S S I O NS P A C E T R A V E L

C A R S T H A T F L YN E U R O H A C K I N G

R O B O T SS M A R T C L O T H I N G

A R T I F I C I A L I N T E L L I G E N C E



@DELLEMCDOJO

ATTHE CURRENT

RATE OF CHURN,75% OF THE S&P

WILL BE REPLACEDBY 2027

I N 1 9 6 0 , T H E A V E R A G E L I F E S P A N W A S6 0 Y E A R S .

T O D A Y ,I T I S 1 2 Y E A R S .



@DELLEMCDOJO

OUR WORLD HAS CHANGED.MANY PEOPLE

HAVE NOT.THEIR CAREERS

MAY FACE EXTINCTION.

THERE IS A BETTER WAY…

LEARNBY

DOINGT R A N S F O R M Y O U R T E A M

T R A N S F O R M Y O U R C O M P A N YT R A N S F O R M Y O U R S E L F



@DELLEMCDOJO

DELL EMC PIVOTALC L O U D F O U N D RY D O J O

Contribute to Cloud Foundry Foundation sanctioned OS projects.Train new Cloud Foundry contributors to commit to OS Cloud Foundry.Innovate around the platform/infrastructure.

CLOUD FOUNDRY

OUR MISSION

A Japanese term which literally means "place of the way”.A room in which martial arts are practiced

DOJO - noun \ˈdō-(ˌ)jō\

In our world, the dojo is the place where we practice modern software development

Practice a modern software development methodology known as ‘the way’ (XP, Lean Startup). Evangelize new work patterns ‘the way’ to internal EMC product teams.

R&D MODERNIZATION



@DELLEMCDOJO

TAKE THE

FIRSTSTEP

HOW DO I GET STARTED? DIVE RIGHT INACT YOUR WAY TO NEW THINKING, LEARN BY ‘DOING’

LEARN BY

DOING

KEEP LEARN-

ING



@DELLEMCDOJO

STEPS TO TRANSFORMATION

BALANCED TEAMS

BUILD, MEASURE, LEARN

BMLTEST DRIVEN

DEVELOPMENT

TDD

MINIMUM VIABLE PRODUCT

MVPTIME, INVENTORY

CONTINUOUS DELIVERY

CD

FEEDBACK

PAIRING

ELIMINATE WASTE

EXTREME PROGRAMMING

CONTINUOUS IMPROVEMENT

SUSTAINABLE PACE

The Importance of EmpathyConscious and deliberate empathy is at the heart of everything we do. When wehave empathy for each other, then we can have empathy for the customer andtheir needs. We favor people and teams who want to work together toward a

shared goal. We dislike selfishness and value teamishness. If we placecustomer, team and self in this order we usually achieve great results and have

fun.

EMPATHY



@DELLEMCDOJO

THE LEAN STARTUPOUR GUIDING

PRINCIPLES

RUNNING LEANTHE PRACTITIONERS

GUIDE

TOYOTA PRODUCTION

SYSTEMREDUCE WASTE

LET’S BREAK DOWN WHAT WE’RE ABOUT



@DELLEMCDOJO

02. WHAT IS DEVOPS?WHAT IT LOOKS LIKEWHAT IT IS NOTEFFECTS OF DEVOPS



@DELLEMCDOJO

DEV

FEATURE IMPLEMENTED

NIGHTLYBUILD

PROVISIONSENVIRONMENT

OPSREL.ENG

BUILDDEPLOYED

QA

FEATURE NOT TO SPEC. BUG CREATED.

WEEK01.

DEV

BUGFIXED

NIGHTLYBUILD

REL.ENG

TESTBUG

QA

NOT FIXEDSENT BACK.

WEEK02. DEV

NEEDMORE INFO.

INFOPROVIDED

QA

FEAT URECOMPLETE.

WEEK03. DEV

BUGFIXED

TESTBUG

QA

TYPICALLIFECYCLEOF ANEW FEATURE

03WEEKAVERAGE

01

WITH NO TDD OR AUTOMATION UP-FRONT,

DEVELOPERS DELIVER FEATURES AND HOPE

FOR THE BEST.

02

WITHOUT THE ABILITY TO SELF PROVISION

ENVIRONMENTS, TESTING IS DELAYED.

03

QA INLCUDED DOWN-STREAM & LATE IN

THE DEV CYCLE. THEY DON’T HAVE FULL

CONTEXT, LEADING TO CONFUSION.

04

DEV AND QA EVENTUALLY ALIGN. THE FEATURE IS TESTED BUT NOT RELEASED MISSING CUSTOMER FEEDBACK.



@DELLEMCDOJO

DEVOPS CULTURE,

PRACTICES, TOOLS.

RAPID DELIVERY

OFAPPS

& SERVICES AT

HIGH VELOCITY

D E L I V E R A N D I M P R O V E P R O D U C T SA T A F A S T E R P A C E T H A N Y O U R C O M P E T I T O R S .

S P E E D E N A B L E S Y O U T O B E T T E R S E R V E C U S T O M E R SA N D C O M P E T E M O R E E F F E C T I V E L Y .

ADOPT

MOVEMENTA DEVOPS



@DELLEMCDOJO

RELENG

DEV

TDD, PAIR PROGRAMMING

OPS

INFRASTRUCTURE AS CODE. GRE

QA

QA EMBEDDED AS PART OF DEVOPS

TEAM

DEVELOPERS WRITE

DEPLOYMENT AUTOMATION

SHARED VISIONEMPATHY AND TRUST PLAYS AN IMPORTANT ROLE. LEADERSHIP RECOGNIZES THE IMPORTANCE OF THIS AND IS 100% BOUGHTIN. WITH DEVOPS UNIFIED IT IS THE HERO AND NO ONE IS SINGULARLY TO BLAME FOR PROBLEMS.



@DELLEMCDOJO

VISIONALIGNMENT

AUTOMATION



@DELLEMCDOJO

THE PERFECT MARRIAGE

BUSINESSVALUE

CULTURETOOLS

COLLABORATION

$

$

QUALITYVELOCITY

PREDICTABILITY

EFFICIENCYEFFECTIVENESS

DEVOPSORGANIZATIONS CHOOSE DEVOPS

FOR A PURPOSE AND WITH A PURPOSE.THAT PURPOSE IS TO DELIVER

USER AND BUSINESS VALUE.



@DELLEMCDOJO

INNOVATIONACCOUNTING



@DELLEMCDOJOBRIAN ROCHE

@BRIANROCHEBOSEMILY KAISER

© Copyright 2017 Dell Inc.@DELLEMCDOJO

SUCCESSMEASURING

LINES OF CODE# OF COMMITS

MILESTONESPROGRESS WITH TECHNICAL DEBT

VANITY METRICS

GROUPS OF ENGINEERS THAT

ARE ‘MINE’PRODUCTS MIRROR

ORG STRUCTURE.

CONWAY’S LAW

TECHNICAL DEBTFAILURE TO AUTOMATE

‘EVERYTHING’NOT FIXING

PROBLEMS NOW!

WASTE

BUILDING PRODUCTS

WITHOUT USER DATA IS POINTLESS

NO DATA

EXTENSIVE UP-FRONT PLANNINGPREDICTING TIME,

FEATURE & RELEASE DATE

WATERFALL

don’t do these things



@DELLEMCDOJO

SUCCESSMEASURING



@DELLEMCDOJO

LOOK FOR COMMUNICATIONMEASURE TEAMS

ON HOW THEY WORK TOGETHER

EMPATHY

LOOK FOR EVIDENCE OF RETROS AND CONTINUOUS

IMPROVEMENT

IMPROVEMENT

STOP AND FIX PIPELINE ISSUES

DISALLOW TECHNICAL DEBT TO

OCCUR

WASTE

THE USER IS OMNIPRESENT

WE BUILD, LEARN AND PIVOT OR

PERSEVERE

USER

BUILD, MEASURE, LEARN IN SMALL

CHUNKS. CODE IS A LIABILITY –

IMPLEMENT ONLY WHAT YOU NEED

ACCOUNTING

GOdo these things

always



@DELLEMCDOJO

03. WHY DEVOPS WORKS?DEVOPS ISN’T EASYBUY-IN AT ALL LEVELSPAIR PROGRAMMING



@DELLEMCDOJO

CHANGE IS HARD(IMPOSSIBLE FOR SOME).

CULTURE SHIFT (PEOPLE, PROCESS, TECH.)

POLYLINGUALPACE IS EXHAUSTING

CONFLICT IS A POSITIVE

LIVEINUNCERTAINTY

UNCOMFORTABLE

DEVOPSIS



@DELLEMCDOJO

WHAT OUR FRIENDS THINK WE DO

WHAT OUR COMPANY THINKS WE DO

WHAT EXTERNAL LEADERSHIP THINKS WE DO

WHAT OUR FAMILIESTHINK WE DO

WHAT WE ACTUALLY DO

WHAT WE THINK WE DO

DELL EMC PIVOTALC L O U D F O U N D RY D O J O






ALLFOR

ONEONEFORALL

S P M & T P M

D E V E L O P E R S

U X / D E S I G N

A N C H O R

D E V O P S T E A MS H A R E D C O N T E X T

C O L L E C T I V E LY P O LY L I N G U A LR A P I D U P - L E V E L I N G






buy-in at all levels

dev team

middle management

executives

operational support

data metric support

fiscal support






PAIR PROGRAMMINGT W O D E V E L O P E R S W R I T E C O D E A T O N E M A C H I N E

O N E D E S I G N A T E D D R I V E R I M P L E M E N T S T H E C O D EO N E O B S E R V E R N A V I G A T E S A N D R E V I E W S T H E C O D E

B O T H A R E A C T I N G A N D R E A C T I N G S T R AT E G I C A L LY



@DELLEMCDOJO

04. USER OBSESSIONscopingthe user is omnipresenttdd is non-negotiable



@DELLEMCDOJO

“THE LONGER IT TAKES FOR A

BUG TO SURFACE, THE

HARDER IT IS TO FIND.”

- ROEDY GREEN



@DELLEMCDOJO

POSSIBILITIES ARE LIMITLESSBUT TIME, MONEY AND HUMAN

CAPITAL AREN’T SO PRIORITIES MUST

BE SET

scope | (v) to assess or investigate

business goalsproduct goals

architectural goalsanti-goals

tweet length description

current solutionsprograms

risksusersretro



@DELLEMCDOJO

Write automated test code

Test DrivenDevelopment

1

2

3

4

Run the test = Red (Failed)

Write some code = Green (Pass)

Clean up the code = Refactor (Pass)

Benefits Quality | Fast Feedback Lean Code | Safety Net =



@DELLEMCDOJO

USER

Build

Measure

Learn

Ris

k =

Un-

valid

ated

Effo

rt

Build

Learn Learn Learn

Measure

Learn

Build

Measure

Learn

Build

Measure

Build

Measure

Build

MeasureBuild

Measure

Learn

USER

USER

USER

IS OMNIPRESENT



@DELLEMCDOJO

05. THE GLUEFEEDBACKEMPATHYBALANCED TEAMSTEAMISHNESS



@DELLEMCDOJO

FORMULA 1

THE FORMULA TEAMS TELL THEIR OWN TRANSFORMATION STYORY. PITSTOPS IN THE 60S OFTEN TOOK LONGER THAN

BLINK DURING A FORMULA 1 PIT-STOP AND YOU’LL PROBABLY MISS IT. BUT THIS WASN’T ALWAYS THE CASE. FIFTY YEARS AGO, A PIT-CREW WOULD TAKE OVER A MINUTE TO CHANGE THE WHEELS AND REFUEL. TODAY, ANYTHING MORE THAN THREE SECONDS IS CONSIDERED A FAIL.

UNSCHEDULEDSTOPS

PIT CREWSUNCOORDINATED

TOO MUCH FUELADDS WEIGHT

LACK OF PRACTICELEAD TO UNPREDICTABLE

RESULTS

TRANSFORMATIONSTORY

FORMULA 1THE



@DELLEMCDOJO

180 5 50% 0Teams were examined Research lasted 5 year Time spent collaborating

Is up to 50% or more. ¾ of our day is spent collaborating

Patterns werefound

Google Research

Zero patterns were found...

Was it all a waste of time?

What did we learn?

What makes a great team?



@DELLEMCDOJO

EMPATHYTHE BEST OUTCOMES

ARE THE CONFLUENCE OF EVERYONE’S OPINION

WE

ARE

GREATBECAUSE

FEEDBACKWE SHARE OUR FEELINGS

ALL THE TIME. WE HELP EACH OTHER SUCCEED.

FEELINGSPSYCHOLTOGICAL SAFETY

HEARD. TRUST. DEPENDABILITYLIKE SUPERHEROES … TOGETHER



@DELLEMCDOJO

COLLABORATIONTEAMS BREAK DOWN PROBLEMS AND MAKE IMPROVEMENTS.

RETRO (RETROSPECTIVES)

ACTION

FEEDBACKTEAM ISSUES ARE AS CHALLENGING AS TECHNICAL ISSUES – IF NOT MORE SO.

EMPATHYWHEN WE HAVE EMPATHY FOR EACH OTHER, WE HAVE EMPATHY FOR USERS.

UP-LEVEL TEAMTEAMS INVEST IN EACH OTHER; WE FOCUS ON WHOLE-TEAM LEARNING.

WHATCAN WELEARN



@DELLEMCDOJO



@DELLEMCDOJO

BALANCED TEAMS“ D E V O P S T E A M S A C H I E V E P R E D I C T A B L E R E S U L T S B E C A U S E

T H E Y M O V E A T A S U S T A I N A B L E P A C E ”

D E V E L O P E R S U X / D E S I G NS P M & T P M A N C H O R

DEVOPS TEAM MAKEUP

At the Dell EMC Dojo we promote the notion of a balanced team. A balanced team is an autonomous group of people with a variety of skills and perspectives that support each other towards a common shared goal. It has all the resources and authority it needs to complete projects on its own. It values continuous delivery, building in small batch sizes and collaboration. We believe the best customer outcomes happen when we structure our work around Balanced Teams, employing Agile XP and Lean practices.

What is a Balanced Team?

“Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.” — the Agile Manifesto

Team members take joint responsibility for the work, sharing in both successes and failures. We have empathy for each other and want to see each other succeed. We trust that each team member will take ownership of her domain (e.g. — the product manager owns the backlog, the developers own the codebase), but will also reach across disciplines to help wherever needed. Each team member keeps the success of the product and team in mind, not the success of the individual.

It’s also vitally important that stakeholders trust the team. A tenet of the Balanced Team is that those closest to the product and its intended users are autonomous and empowered to make all product, design, and technical decisions. While outside or higher-level stakeholders can (and should) have input, they need to trust that the product team has the best information to make the best decisions.

Why are we so darn happy all the time?



@DELLEMCDOJO

06. WE MESS UP A LOTQUICK DECISIONSGOOD ENOUGH IS OKFAST IS ONLY PART OF ITNO INTERRUPTIONS



@DELLEMCDOJO

“Only those who dare to fail greatly can ever achieve greatly.” - Robert F. Kennedy



@DELLEMCDOJO

CUSTOMERFEEDBACK

CONTINUOUS DELIVERY

CONTINUOUS

INTEGRATION

Run Test Automation

Write the Code

Continuous Integration

Deploy



@DELLEMCDOJO

+ + =FIGURE OUT THE

PROBLEM THAT NEEDS TO BE SOLVED AND

DEVELOP A (MVP) TO BEGIN THE PROCESS OF

LEARNING AS QUICKLY AS POSSIBLE.

01DID OUR SOLUTION

SOLVE THE CUSTOMER’S NEEDS?

IS THE SOLUTION VALUABLE?

02DID WE VALIDATE OUR HYPOTHESIS ABOUT

THE PRODUCT STRATEGY AND

ENGINE OF GROWTH?

03OUR ULTIMATE GOAL

IS TO BUILD A SUSTAINABLE

BUSINESS WITH MAXIMUM

ACCELERATION

04THE PROBLEM THE VALUE PIVOT OR PERSEVERE SUSTAINABLE BUSINESS

MINIMUM VIABLE PRODUCT (MVP)



@DELLEMCDOJO

8a.m.-5p.m.

Work Day

12:30p.m.-

1:30p.m.Lunch

Stand Up, IPM,Retro

Meetings

9Hours

x 5Days

1Hours

x 5Days

x15 min 5 Days

30 min 1 Day

1 Hour 1 Day

= 45Hours of Work per Week

= 5Lunch Hours per Week

= 2.75Meeting Hours per Week

DEVOPS FOCUSED WORK WEEK 37.25Hours of

PURE BUSINESS VALUE



@DELLEMCDOJO

SHARED SPACE

PM, UX, Dev all sit together, everyone

around us is working on the same goal. The

space is really important

ONE GOAL

Small teams work on a single shared problem.

AVOID WASTE

Meetings are waste, keep them to a

minimum. IPM, Retro, Standup

ELIMINATE IDLE TIME

Long running tasks >20 minutes = pickup

another story from the backlog

No Interruptions



@DELLEMCDOJO

07. SCALING DEVOPSLEARNING CYCLESFAIL ALL THE TIMENO WATER FALLINGBUILD IN SMALL CYCLES



@DELLEMCDOJO

“IT’S ALL TALK UNTIL

THE CODERUNS.”- WARD

CUNNINGHAM



@DELLEMCDOJO

APIsARE THEBUILDING BLOCKSWHEN SCALINGDEVOPS

PRODUCTION

APIAPP#1

APP#2

APP#3

API

FEATURETEAM 1

FEATURETEAM 2

FEATURETEAM 3

FEATURETEAM 4

API API API

RELEASE ENG.PACKAGING

APIs

APIs GOVERN INTERACTION

BETWEEN COMPONENTS

TDD

TDD ENSURES QUALITYAT THE FEATURE

LEVEL

INTEGRATION

SUBSET OF (TDD) TESTS BECOME PART

OF INTEGRATION SUITE

RELEASE

RELEASEENGINEERING TEAMINTEGRATES FINAL

RELEASE

DEVELOPERS AREN’T HARDWIRED TO THINK LIKE

QA. QA DOES NOT BECOME SUBSUMED BY

DEV.

MINDSET



@DELLEMCDOJO

DEV+OPSWANT

THESAME

THING …APPS

RUNNING INPRODUCTION

OPERATORSADOPT ADEVMENTALITY

FULLY AUTOMATED

HEALTHMONITORING

VERSIONED INFRA.CHANGES

INFRASTRUCTURE-AS-CODE

GOOGLE SITE RELIABILITY

ENGINEERING

EMPATHYRETROS WITH

BEER



@DELLEMCDOJO

DEVOPS

CONTAINERS

MICROSERVICES

CONTINUOUSDELIVERY

CLOUD NATIVE APPLICATIONS

PURPOSE-BUILT FOR THE CLOUD.Exploiting the advantages of a cloud computing model.

CLOUD NATIVEPLATFORM CONCEPTS

CLOUD NATIVE PLATFORMS HAVE THE POTENTIAL TO MAKE THE SOFTWARE BUILD, TEST, DEPLOY, AND SCALE CYCLE SIGNIFICANTLY FASTER. THEY REMOVE MANY HURDLESIN DEPLOYING SOFTWARE, ENABLING YOU TO RELEASE SOFTWARE AT WILL.

OSS IAAS ABSTRACTION

CONTAINERS

HEALTHMONITORING

DEPLOYMENTSUPPORT

APP LIFECYCLEMANAGEMENT

NETWORKING SELF HEALING SCALING

CLOUD-NATIVE APPS USE AN ELASTIC INFRASTRUCTURE

CLOUD-NATIVE APPS NEED TO PROVISION INSTANCES OF THEMSELVES THROUGH AN API

CLOUD-NATIVE APPS NEED TO SCALE UP AND DOWN AT A RAPID RATE

AUTOMATIC DETECTION AND WORKAROUND OF FAILURES



@DELLEMCDOJO



@DELLEMCDOJO

08. TRANSFORMATIONLEARN BY DOINGSCALING DEVOPS IN THE ENTERPRISECONTINUOUS DELIVERYNO TRAINING PROGRAMS



@DELLEMCDOJO

LEARNBY

DOING

T R A N S F O R M Y O U R C O M P A N YT R A N S F O R M Y O U R T E A M

T R A N S F O R M Y O U R S E L F

B o o s t s P r o b l e m - S o l v i n g S k i l l s a n d M o t i v a t i o n

D e e p e n s C o n c e p t u a l U n d e r s t a n d i n g

I m p r o v e s K n o w l e d g e T r a n s f e r t o n e w s i t u a t i o n s

a n d p e o p l e

S p a r k s h i g h e r - o r d e r t h i n k i n g a n d d e e p e r i n q u i r y



@DELLEMCDOJO

BASICS

XP, LEAN STARTUP, RUNNING LEAN, LEAN MANUFACTURING

THE

STARTED

DOJO ENGAGEMENTIMMERSION (6 WEEKS)INTENSE PRACTICE

GETTING

PROFICIENT

DAILY PRACTICECONTINUOUS IMPROVEMENTUNDERSTAND NUANCES

PROMOTER

PRACTICE LEADDEEP UNDERSTANDINGEVANGELIST

DOJOPATH TO

ENLIGHTENMENT

THE



@DELLEMCDOJO

Basics Engaged Proficient Promoter

Scale Potential

DevOpsScaling

TheArt ofAb

ility

to S

cale



@DELLEMCDOJO

DO

traditional world devops world

SAY



@DELLEMCDOJO

09. CLOSINGbringingit alltogether



@DELLEMCDOJO

THE

EVOLUTION



@DELLEMCDOJO

DON’T SWIM HALF WAY AND TURN BACK

TRANSFORM YOURSELF



@DELLEMCDOJO

OUR WORLD HAS CHANGED.MANY PEOPLE

HAVE NOT.THEIR CAREERS

MAY FACE EXTINCTION.

THERE IS A BETTER WAY…

LEARNBY

DOINGT R A N S F O R M Y O U R T E A M

T R A N S F O R M Y O U R C O M P A N YT R A N S F O R M Y O U R S E L F



@DELLEMCDOJO

BRIAN ROCHETWITTER@BRIANROCHEBOS

EMILY KAISERTWITTER@DELLEMCDOJO

THANK YOU7 LESSONS LEARNEDWHEN IMPLEMENTINGDEVOPS + BONUS

7 lessons learned when implementing devops. toyota . production. system. ... qa inlcuded down -...

Documents