devops pipelines and metrics driven feedback loops

57
Metrics-Driven Pipelines or Testing & Measures in DevOps Andreas Grabner: @grabnerandi, [email protected] Slides: http://www.slideshare.net/grabnerandi Podcast: https://www.spreaker.com/show/pureperformance

Upload: andreas-grabner

Post on 15-Apr-2017

1.427 views

Category:

Software


3 download

TRANSCRIPT

Page 1: DevOps Pipelines and Metrics Driven Feedback Loops

Metrics-Driven Pipelinesor

Testing & Measures in DevOpsAndreas Grabner: @grabnerandi, [email protected] Slides: http://www.slideshare.net/grabnerandiPodcast: https://www.spreaker.com/show/pureperformance

Page 2: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

The Story started in 2009

Page 3: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Page 4: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

“The stuff we did when we were a Start Up

and we All wereDevs, Testers and Ops”

Quote from Andreas Grabner back in 2013 @ DevOps Boston

Page 5: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Page 6: DevOps Pipelines and Metrics Driven Feedback Loops

Utmost goal: minimize cycle time (= Lead Time)

timefeature cycle time

minimize Users

This is where youcreate value!

Page 7: DevOps Pipelines and Metrics Driven Feedback Loops

From the DevOps Webinar with Gene & Mark

Mark TomlinsonPerformance Sherpa

@mark_on_task

Andi GrabnerPerformance Advocate

@grabnerandi

Gene Kim, CTOResearcher and Author

@RealGeneKim

Webinar Recording: https://info.dynatrace.com/apm_wc_gene_kim_na_registration.html

Page 8: DevOps Pipelines and Metrics Driven Feedback Loops

High Performers Are …

200x 2,555xmore frequent deployments faster lead times than their peers

Source: Puppet Labs 2015 State Of DevOps Report: https://puppet.com/resources/white-paper/2016-state-of-devops-report

More Agile

3x 24xlower change failure rate faster Mean Time to Recover

More Reliable

Page 9: DevOps Pipelines and Metrics Driven Feedback Loops

24 “Features in a Box” Ship the whole box!

Very late feedback

F r u s t r a t i o n !

Page 10: DevOps Pipelines and Metrics Driven Feedback Loops

„1 Feature at a Time“

„Optimize before Deploy“„Immediate Customer Feedback“

Continuous Innovation and Optimization

Page 11: DevOps Pipelines and Metrics Driven Feedback Loops

DevOps Adoption

Page 12: DevOps Pipelines and Metrics Driven Feedback Loops

700 deployments / YEAR

10 + deployments / DAY

50 – 60 deployments / DAY

Every 11.6 SECONDS

Innovators (aka Unicorns): Deliver value at the speed of business

Page 13: DevOps Pipelines and Metrics Driven Feedback Loops

Early

Adopt

ers

Page 14: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

DevOps @ Targetpresented at Velocity, DOES and more …

http://apmblog.dynatrace.com/2016/07/07/measure-frequent-successful-software-releases/

“We increased from monthly to 80 deployments per week

… only 10 incidents per month …

… over 96% successful! ….”

Page 15: DevOps Pipelines and Metrics Driven Feedback Loops

“We Deliver High Quality Software,Faster and Automated using New Stack“

„Shift-Left Performance to Reduce Lead Time“

Adam Auerbach, Sr. Dir DevOps

https://github.com/capitalone/Hygieia & https://www.spreaker.com/user/pureperformance

“… deploy some of our most critical production workloads on the AWS platform …”, Rob Alexander, CIO

Page 16: DevOps Pipelines and Metrics Driven Feedback Loops

From 0 to DevOps in 80 daysLessons learnt from shifting an on-prem to a cloud cultureBernd Greifeneder, CTO

http://dynatrace.com/trial

Webinar: http://ow.ly/cEYo305kFEyPodcast: http://bit.ly/pureperf

Page 17: DevOps Pipelines and Metrics Driven Feedback Loops

2 major releases/yearcustomers deploy & operate on-prem

26 major releases/year170 prod deployments/dayself-service online sales SaaS & Managed

2011 2016

Page 18: DevOps Pipelines and Metrics Driven Feedback Loops

18 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

believe in the mission impossible

6 monthsmajor/minor release+ intermediate fix-packs+ weeks to months rollout delay

sprint releases (continuous-delivery)1h : code to production

Page 19: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandihttps://dynatrace.github.io/ufo/

“In Your Face” Data!

Page 20: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Time of D

eployment

Availability dropped to 0%

#1: Availability -> Brand Impact

Page 21: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

New Deployment + Mkt Push

Increase # of unhappy users!

Decline in Conversion Rate

Overall increase of Users!

#2: User Experience -> Conversion

Spikes in FRUSTRATED Users!

Page 22: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

#3: Resource Cons -> Cost per Feature

4x $$$ to IaaS

Page 23: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

#4: Performance -> Behavior

Page 24: DevOps Pipelines and Metrics Driven Feedback Loops

Dynatrace Transformation by the numbers

23x

170

more releases

Deployments / Day

31000 60hUnit+Int Tests / hour UI Tests per Build

More Quality

~200 340code commits / day Stories per sprint

More Agile

93%Production bugs found by Dev

@grabnerandi

More Stability 450 99.998%Global EC2 Instances Global Availability

Page 25: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Not every Sprint ends without bruises!

Page 26: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Richard DominguezDeveloper in OperationsPrep Sportswear

„In 2013 business demanded to go from monthly to daily deployments“

„80% failed!“

Page 27: DevOps Pipelines and Metrics Driven Feedback Loops

Understanding Code Complexity• 4 Millions Lines of Monolith Code• Partially coded and commented in

Russian

From Monolith to Microservice• Initial devs no longer with company• What to extract withouth breaking it?

Shift Left Quality & Performance• No automated testing in the pipeline• Bad builds just made it into production

Cross Application Impacts• Shared Infrastructure between Apps• No consolidated monitoring strategy

Page 28: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Scaling an Online Sports Club Search Service

2015201420xx

Response Time

2016+

1) 2-Man Project 2) Limited Success

3) Start Expansion

4) Performance Slows Growth Users

5) Potential Decline?

Page 29: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Early 2015: Monolith Under Pressure

Can‘t scale vertically endlessly!

May: 2.68s 94.09% CPU Bound

April: 0.52s

Page 30: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

From Monolith to Services in a Hybrid-Cloud

Front End inGeo-Distributed

Cloud

Scale Backendin ContainersOn Premise

Page 31: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Go live – 7:00 a.m.

Page 32: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Go live – 12:00 p.m.

Page 33: DevOps Pipelines and Metrics Driven Feedback Loops

What Went Wrong?

Page 34: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

26.7s Load Time5kB Payload

33! Service Calls99kB - 3kB for each call!

171! Total SQL Count

Architecture ViolationDirect access to DB from frontend service

Single search query end-to-end

Page 35: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

The fixed end-to-end use case“Re-architect” vs. “Migrate” to Service-Orientation

2.5s (vs 26.7) 5kB Payload

1! (vs 33!) Service Call5kB (vs 99) Payload!

3! (vs 177) Total SQL Count

Page 36: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Page 37: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

You measure it! from Dev (to) Ops

Page 38: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Build 17 testNewsAlert OK

testSearch OK

Build # Use Case Stat # APICalls # SQL Payload CPU

1 5 2kb 70ms

1 35 5kb 120ms

Use Case Tests and Monitors Service & App Metrics

Build 26 testNewsAlert OK

testSearch OK

Build 25 testNewsAlert OK

testSearch OK

1 4 1kb 60ms

34 171 104kb 550ms

Ops#ServInst Usage RT

1 0.5% 7.2s

1 63% 5.2s

1 4 1kb 60ms

2 3 10kb 150ms

1 0.6% 3.2s

6 75% 2.5s

Build 35 testNewsAlert -

testSearch OK

- - - -

2 3 7kb 100ms

- - -

4 80% 2.0s

Continuous Innovation and Optimization

Re-architecture into „Services“ + Performance Fixes

Scenario: Monolithic App with 2 Key Features

Page 39: DevOps Pipelines and Metrics Driven Feedback Loops

Where to Start?Where to Go?

Page 40: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Page 41: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

„Always seek to Increase Flow“

„Understand and Respond to Outcome“

„Culture on Continual Experimentation“

Page 42: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

„Always seek to Increase Flow“

Testing: Ensure Success in The First Way

Removing Bottlenecks

Eliminating Technical Debt

Enable Successful Cloud & Miroservices Migration

Shift-Left Quality

Reduce Code Complexity

Page 43: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandiAND MANY MORE

Page 44: DevOps Pipelines and Metrics Driven Feedback Loops

Manual Code/Architectural Bottleneck Detection• Blog & YouTube Tutorial:

• http://apmblog.dynatrace.com/2016/06/23/automatic-problem-detection-with-dynatrace/• http://bit.ly/dttutorials

• Metrics• # SQL, # of Same SQLs, # Threads, # Web Service/API Calls # Exceptions, # of Logs• # Bytes Transferred, Total Page Load, # of JavaScript/CSS/Images ...

Page 45: DevOps Pipelines and Metrics Driven Feedback Loops

Automated Code/Archiecture Bottleneck Detection

Locate Architectural Hotspots

Automated Filtering by Database Patterns

Problem Details

Page 46: DevOps Pipelines and Metrics Driven Feedback Loops

Remove Database Bottlenecks

cite the database as the most common challenge or issue with application performance

88%

Page 47: DevOps Pipelines and Metrics Driven Feedback Loops

Manual Database Bottleneck Detection• Blog & YouTube Tutorial:

• http://apmblog.dynatrace.com/2016/02/18/diagnosing-java-hotspots/• http://bit.ly/dttutorials -> Database Diagnostics

• Patterns• N+1 Query, Unprepared SQL, Slow SQL, Database Cache, Indices, Loading Too Much

Data ...

Page 48: DevOps Pipelines and Metrics Driven Feedback Loops

Automated Filtering by Pattern

Automated Database Bottleneck Detection

SQL+Bind + Execution Plans

Tuning Insights for

MSSQL, Oracle, MySQL ...

Page 49: DevOps Pipelines and Metrics Driven Feedback Loops

“To Deliver High Quality Working Software Faster“

„We have to Shift-Left Performance to Optimize Pipelines“http://apmblog.dynatrace.com/2016/10/04/scaling-continuous-delivery-shift-left-performance-to-improve-lead-time-pipeline-flow/

Page 50: DevOps Pipelines and Metrics Driven Feedback Loops

= Functional Result (passed/failed)+ Web Performance Metrics (# of Images, # of JavaScript, Page Load Time, ...)+ App Performance Metrics (# of SQL, # of Logs, # of API Calls, # of Exceptions ...)

Auto Baseline

Auto Alert

Fail the build early!

Page 51: DevOps Pipelines and Metrics Driven Feedback Loops

Reduce Lead Time: Stop 80% of Performance Issues in your Integration Phase

CI/CD: Test Automation (Selenium, Appium, Cucumber, Silk, ...) to detect functional and

architectural (performance, scalabilty) regressions

Perf: Performance Test (JMeter, LoadRunner, Neotys, Silk, ...) to detect

tough performance issues

Page 52: DevOps Pipelines and Metrics Driven Feedback Loops

Shift-Left Performance results in Reduced Lead Timepowered by Dynatrace Test Automation

http://apmblog.dynatrace.com/2016/10/04/scaling-continuous-delivery-shift-left-performance-to-improve-lead-time-pipeline-flow/

Page 53: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Fast Response to Outcome: Address Deployment Impact

User Experience, Conversion Rate

Costs and EfficiencyAvailability

Page 54: DevOps Pipelines and Metrics Driven Feedback Loops

@grabnerandi

Real User Feedback: Building the RIGHT thing RIGHT!

Experiment & innovate on new ideas

Optimizing what is not perfect

Removing what nobody needs

Page 55: DevOps Pipelines and Metrics Driven Feedback Loops

Faster Lead Times to User Value!Results in Business Success!

Page 56: DevOps Pipelines and Metrics Driven Feedback Loops

QuestionsSlides: slideshare.net/grabnerandiGet Tools: bit.ly/dtpersonalWatch: bit.ly/dttutorialsFollow Me: @grabnerandiRead More: blog.dynatrace.comListen: http://bit.ly/pureperfMail: [email protected]

Page 57: DevOps Pipelines and Metrics Driven Feedback Loops

Andreas GrabnerDynatrace Developer Advocate@grabnerandihttp://blog.dynatrace.com