a case study in metrics-driven devops

DT2Session6/9/1611:30AM

ACaseStudyinMetrics-DrivenDevOps

Presentedby:

BenVining

ElectronicArts

Broughttoyouby:

350CorporateWay,Suite400,OrangePark,FL32073888---268---8770··[email protected]://www.techwell.com/

BenViningElectronicArtsBenViningisadevelopmentmanagerofaDevOpsteamatVancouver-basedElectronicArts.BenhasworkedongamesforEAfranchisessuchasFIFA,NHL,UFC,andPlantsvs.Zombies.Inapreviouslife,hewasabusinesssystemsanalystwithJELD-WENWindows&DoorsinAustralia,helpingtodevelop,implement,andmaintaintheirin-houseERPsystem.Benbelievesinworkingsmart,takingtimetocelebratewins,andlearningfrommistakes.ThisisBen'sfirsttimepresentingataconference,sopleasegoeasyonhim.YoucanfindhimonLinkedIn.

5/30/16

1

A Case Study in Metrics-Driven DevOps BEN VINING | ELECTRONIC ARTS DEVOPS WEST | JUNE 2016

Overview WHYYOU’REHERE You’reinterestedinusingmetricstoimproveyourteam’s:◦ Performance◦ ReputaDon◦ Happiness

WHATTOEXPECT IntroducDon TheCaseStudy ExampleObjecDves BigBenefits&KeyTakeaways QuesDonTime(attheend)

5/30/16

2

IntroducKon BEN VINING | ELECTRONIC ARTS

IntroducKon – Ben Vining

3years B.Sciencein

InformaDonSystemsUniversityofNewSouthWalesSydney,AU

8years Business

SystemsAnalystJELD-WENWindows&DoorsGoldCoast,AU

3years Development

ManagerElectronicArtsVancouver,CA

5/30/16

3

IntroducKon – Electronic Arts (EA)  ElectronicArtsInc.isaleadingglobalinteracDveentertainmentsoewarecompany.EAdeliversgames,contentandonlineservicesforInternet-connectedconsoles,personalcomputers,mobilephonesandtablets.

The Case Study THE TEAM | WHERE WE WERE | BECOMING METRICS-DRIVEN |

WHERE WE ARE NOW

5/30/16

4

Development & Release Engineering (DRE)  Weprovidebuild&testautomaDonservicestoourcustomers. Ourcustomersaregame,content,andengineteamsatEA. 70staffin6locaDonsaroundtheglobe. Ourvision:EAdeveloperscanworktotheirfullestpotenDal,becauseoursystemstakecareofeverythingelse. Ourmission:makelifesimplerfordevelopers.

Overall Team Structure

Sub-TeamLeads

Leadership

SeniorLeadership GroupDD&TD

DDs&TDs

Canada USA Europe Engine Infra.

5/30/16

5

Sub-Team Structure

Projects

Sub-TeamLeads Canada

1xLarge 4xMedium

4xMedium 6xSmall

Project Structure  Eachcustomeristreatedasauniqueproject

 Manyteam-membersareassignedtoeachproject

 Eachteam-memberisassignedtomanyprojects

 FrequencyofinteracDons(meeDngs,reports,communicaDons,etc.)isdependantonsizeandcultureofthecustomerteam

PointofContact

Leads

AddiDonalSEs

5/30/16

6

Where We Where  WearenowinthefourthyearofourapproachtometricscollecDon&reporDng.

 Beforethat:◦ OurautomaDonsystemsweren’treliableenough.Oneinfivejobswouldfailtoproducemeaningfulresults

◦ Wedidn’tknowwhereourDmewent.Wehadnovisibilityonwhichprojectsdemandedthemosteffortorhowlongwespentonsupport

◦  Itwasnotuncommonforengineerstoleavetoworkonagameteam◦ Wewereasmallerteamworkingacrossthreestudiosandaselectrangeofprojects

Becoming Metrics-Driven

EvaluaDon&EvoluDonLeadership

ImplementaDon&OperaDonSub-TeamLeads

DesignSeniorLeadership

5/30/16

7

Design  CorePillars ObjecDvesDefiniDons ClassificaDon DataPoints Storage SummaryReport

Design – Core Pillars

Healthy Reliable Efficient Responsible

Predictable Responsive InnovaDve Respected

5/30/16

8

Design – ObjecKve DefiniKons

DescripDon Source

Intent Target

Name

ObjecKve – AutomaKon Reliability

Descrip0on:PercentageofautomaDonrunsthataresuccessful

Source:StatsarepulledfromtheautomaDonsystemweekly

Intent:OurautomaDonsystemiscloselymonitoredandmaintained

Target:97%Note:failuresduetothegamecode

countasasuccesstous

AutomaDonReliability

5/30/16

9

ObjecKve – Time Logged

Descrip0on:PercentageofavailableDmeloggedintheDckeDngsystem

Source:StatsarepulledfromtheDckeDngsystemweekly

Intent:WeknowwherewespendourDme

Target:100%(ofsixhoursaday).Note:noneedtolog:meagainstmee:ngsandotherinterrup:ons

TimeLogged

ObjecKve – Team Health Descrip0on:AverageresultsperteammemberfromtheTeam

HealthSurvey

Source:Surveyiscompletedtwiceperyear

Intent:Ourteamishealthy&happy Target:4/5

TeamHealth

5/30/16

10

ObjecKve – External RecogniKon Descrip0on:NumberofDmesour

teamispraisedbycustomers(attheseniorleadershiplevel)

Source:Casesaremanuallyrecordedasrequired

Intent:Ourbrandiswell-respectedaroundthecompany Target:4peryear

ExternalRecogniDon

Design – ObjecKve ClassificaKon

CustomerFacing

InternalOperaDons Strategic

Sub-Team

5/30/16

11

Design – Data Points  Baseline Target Current Previous Currentvs.Target Currentvs.Previous

Design – Storage  TherawdataiscomingfrommulDplesources(automaDonsystem,DckeDngsystem,surveys,etc.)

 WeuseExcelasthecentralstoragelocaDonforourmetrics

 Eachmetrichasatablethatstorestherawdata

 Separateentryforeachsub-team,eachperiod

 Wehaveconsideredusingothertools,butExceldoesthejobandweareallfamiliarwithusingit

5/30/16

12

Design – Summary Report  DisplaystheaggregatedresultsforeachobjecDvewiththedatapointsasdiscussedearlier

Baseline Target Current Previous Currentvs.Target

Currentvs.Previous

Customer

Reliability 95.0% 97.0% 98.1% 95.3% 1.1% 2.8%

TicketAck. 8.0 4.0 4.5 4.8 0.5 -0.3

InternalOps

TeamHealth 3.8 4.0 4.2 3.8 0.2 0.4

ImplementaKon & OperaKon  Training GatherBaselines SprintProcess DataCollecDon FocusedReporDng

5/30/16

13

ImplementaKon – Training  Geteveryoneontheteamon-board◦ Whywearedoingthis?◦ ExpectaDons◦ Workflowchanges◦ Systemchanges

ImplementaKon – Gather Baselines  Beforewestartedregularlyrecordingthemetrics,wefirstgatheredthebaselinestounderstandwherewearenow

 Determiningthebaselinewastrickyforcertainmetrics:◦  Inaccuratetracking◦ Notrackingatall◦ Nofocusonit◦ Dataisthere,butcollecDngitisinefficient

 SDll,havingthesebaselinesmadeiteasytoshowimprovementsearlyintheprocess

5/30/16

14

ImplementaKon – Sprint Process  Sub-teamrunsaweeklysprint

 FibonacciSequence(1,2,3,5)foresDmaDngcomplexityoftasks

 Tasksinthesprintcovertherangeofprojectsthatsub-teamisresponsiblefor

 Onlycommittasksthatwepromisetofinish

 We’reaDevOpsteam–berealisDcinhowmuchworkwecommitto

 Combinedsprintisinternal,i.e.notsharedwithcustomers

ImplementaKon – Sprint Admin (Time) Name Days Hours/Day LoggedHoursAngus 4.5 6 29Julia 4 6 22

5/30/16

15

ImplementaKon – Sprint Admin (Tasks) Name Task Points StatusAngus TASK-1 3 CompleteAngus TASK-2 3 UnderesDmatedJulia TASK-3 5 De-prioriDzedJulia TASK-4 2 Complete

OperaKon – Data CollecKon  Warning:metricsdatacollecDonintroducesoverhead

 Weautomatewhatwecan,butthedatacomesfrommulDplesourcesandismanuallyentered

 It’simportantforeachobjecDvetohaveaclearowner

 ObjecDveowners:justgetinanddoit.Onceyouletitsliponeweek,it’seasytoletitslipagain.Andagain…

5/30/16

16

OperaKon – Focused ReporKng  TheresultsofeachautomaDonrunarestoredinacentraldatabase.

 ThisallowsustocloselymonitortheresultswithfocusedreporDngvia:◦  InteracDveReports◦ TVDashboard◦ DailyAutomatedEmails

Reliability ReporKng – InteracKve Reports AutomaDonReliability

Success GameFail InfraFail

FailureCount

FailureDescrip0on

70 GAME–compileerror20 INFRA–nodiscspace10 INFRA–networkerror

5/30/16

17

Reliability ReporKng – TV Dashboards

Perceived100%

Tests93.1%

Builds98.1%

Reliability ReporKng – Automated Emails

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ProjectA ProjectB ProjectC ProjectD

TestReliability-Yesterday

Success Failure

5/30/16

18

EvaluaKon & EvoluKon  Sub-TeamReviews

 QuarterlyUpdates AnnualRefresh

EvaluaKon – Sub-Team Reviews  Leadershipmeetsregularly,rotaDngamongstthesub-teams

 Comparecurrentresultsvs.previous&target

 WhatacDonshavebeentakensincethepreviousreview?

 WhatacDonswillbetakeninthefuture?

 Focusisnotnecessarilyonthenumbers,butmoreontheacDons

5/30/16

19

EvaluaKon – Quarterly Updates  Anupdateonthetrendofkeymetricsispresentedintheteam’squarterlytownhallmeeDng

 Followed-upwithemail

91%

92%

93%

94%

95%

96%

97%

98%

99%

100%

Q1 Q2 Q3 Q4

AutomaDonReliability

CAN USA EUR Target

EvoluKon – Annual Refresh  Towardstheendoftheyear,theteamhastheopportunitytoupdatetheobjecDves:◦ AddnewobjecDves◦ RemovestaleobjecDves◦ MoveobjecDvesbetweencore&sub-team◦ Modifytargets◦ Re-baseline

5/30/16

20

Where We Are Now  WearenowinthefourthyearofourapproachtometricscollecDon&reporDng.

 Improvementswe’veseen:◦ OurautomaDonsystemsarereliable.Weregularlyhitourtargetsand97%ofrunsproducemeaningfulresultstothecustomer

◦ WeknowwhereourDmegoes.WecantrackDmespentperproject,type(supportvs.value-add),orarea(builds,test,infrastructure,reporDng,etc.)

◦ Theteamishappyandconsistentlyscoresover4/5intheteamhealthsurvey◦ TheteamhasexpandedtoworkonaddiDonalprojectsinotherstudios

Example ObjecKves OVERVIEW | DETAIL

5/30/16

21

Overview of Example ObjecKves  AutomaDonReliability TimeLogged TeamHealth ExternalRecogniDon CustomerSaDsfacDon BudgetVariance

 TicketAcknowledgement SupportTicketClosure TimeinWork Overhang PlannedWork SupportWork

ObjecKve – Customer SaKsfacKon Descrip0on:AverageresultsperrespondentfromtheCustomer

SaDsfacDonSurvey

Source:Surveyiscompletedtwiceperyear

Intent:Ourcustomersarehappy Target:4/5

CustomerSaDsfacDon

5/30/16

22

ObjecKve – Budget Variance

Descrip0on:Percentagethatwearewithinourbudget

Source:Statsarepulledfromthefinancesystemquarterly

Intent:WesDcktoourbudget Target:0%

BudgetVariance

ObjecKve – Ticket Acknowledgement Descrip0on:AverageDmecustomercreatedDcketsspendinthe‘new’

state

Source:StatsarepulledfromtheTickeDngsystemweekly

Intent:Ourcustomersarepromptlyrespondedto Target:4hours

TicketAcknowledgement

5/30/16

23

ObjecKve – Support Ticket Closure

Descrip0on:AverageDmeittakestoclosesupportDckets


Intent:Brokenthingsarefixedpromptly Target:4days

SupportTicketClosure

ObjecKve – Time in Work

Descrip0on:AverageDmeDcketsspendinthe‘inprogress’state


Intent:TasksaresizedintoDcketsthatarecompletedinasprint Target:4days

TimeinWork

5/30/16

24

ObjecKve – Overhang Descrip0on:Percentageof

commitedtasksthatareincompleteattheendofasprint


Intent:Wecompleteourplannedworkeachweek Target:10%

Overhang

ObjecKve – Planned Work Descrip0on:PercentageofDmespentonthecommitedtasksina

sprint


Intent:WemaximizetheDmespentoncommitedtasks Target:40%

PlannedWork

5/30/16

25

ObjecKve – Support Work

Descrip0on:PercentageofDmespentonfixingbrokenthings


Intent:WeminimizetheDmespentfixingthings Target:10%

SupportWork

Wrap-Up BIG BENEFITS | KEY TAKEAWAYS | QUESTION TIME

5/30/16

26

Big Benefits

Data-DrivenDecisionMaking

• Wehaveevolvedtomakedecisionsbasedonourdata,ratherthanouremoDons

Confidence

• Wecanvalidatethatwe’reasgoodaswethinkweare

ReputaDon

• Weareseenasateamthathastheirstufftogether

Key Takeaways  BeforeyoudefineindividualobjecDves,considerthecorepillarsofyourteam

 IfindividualsontheteamcannotinfluenceanobjecDve,itshouldn’tbethere

 Don’thavecompeDngobjecDves

 EducatetheexisDngteam&onboardnewstaff

 ThebehavioursthattheobjecDvesdrivearemoreimportantthanthenumbers

5/30/16

27

QuesKon Time PRESENTATIONTITLE ACaseStudyinMetrics-DrivenDevOps

CONTACTINFO BenVining ElectronicArts [email protected]

a case study in metrics-driven devops

Software