metrics-driven devops: delivering high quality software faster!

57
Metrics Driven DevOps Delivering High Quality Software Faster! Andreas Grabner (@grabnerandi) – [email protected] Your Dynatrace Personal License: http://bit.ly/dtpersonal

Upload: dynatrace

Post on 15-Apr-2017

509 views

Category:

Software


0 download

TRANSCRIPT

Page 1: Metrics-Driven Devops: Delivering High Quality Software Faster!

Metrics Driven DevOpsDelivering High Quality Software Faster!Andreas Grabner (@grabnerandi) – [email protected] Dynatrace Personal License: http://bit.ly/dtpersonal

Page 2: Metrics-Driven Devops: Delivering High Quality Software Faster!
Page 3: Metrics-Driven Devops: Delivering High Quality Software Faster!

700 deployments / year

10 + deployments / day

50 – 60 deployments / day

Every 11.6 seconds

Page 4: Metrics-Driven Devops: Delivering High Quality Software Faster!

Not only fast delivered but also delivering fast!

Page 5: Metrics-Driven Devops: Delivering High Quality Software Faster!

60% Rate performance/response time as the #1 mobile app

expectation- ahead of features and functionality -

Source: Forrester Research 2014

Page 6: Metrics-Driven Devops: Delivering High Quality Software Faster!
Page 7: Metrics-Driven Devops: Delivering High Quality Software Faster!

It‘s not about blind automation of pushing more bad code on new stacks through a pipeline

Page 8: Metrics-Driven Devops: Delivering High Quality Software Faster!

It‘s not about blindly giving everyone Ops powerto deploy changes only tested locally

Page 9: Metrics-Driven Devops: Delivering High Quality Software Faster!

Technical Debt

Business Debt

Organizational Rust

Page 10: Metrics-Driven Devops: Delivering High Quality Software Faster!

Status Quo

NEXT EXIT

Page 11: Metrics-Driven Devops: Delivering High Quality Software Faster!

„All-In Agile: Across the Pipeline“„We don‘t log bugs! We fix them!“

„All Manual Testers Automate!“

Adam Auerbach@bugman31

Page 12: Metrics-Driven Devops: Delivering High Quality Software Faster!

All-In Agile: Use application metrics as Feedback Loop & Quality Gates

Dev&Test: Personal License to Stop Bad Code

when it gets created!Tip: Dont leave your IDE!

Continuous Integration: Auto-Stop Bad Builds based on AppMetrics from Unit-, Integration, - Perf Tests

Tip: integrate with Jenkins, Bamboo, ...

Prod: Monitor Usage and Runtime Behavior per Service, User Action,

Feature ...Tip: Stream to ELK, Splunk and Co ...

Automated Tests: Identify Non-Functional Problems by looking at App Metrics

Tip: Feed data back into your test tool!

Page 13: Metrics-Driven Devops: Delivering High Quality Software Faster!

14

What you currently measure

What you should measure

Quality Metrics in your pipeline # Test Failures

Execution Time per test# calls to API# executed SQL statements# Web Service Calls# JMS Messages# Objects Allocated# Exceptions# Log Messages# HTTP 4xx/5xxRequest/Response SizePage Load/Rendering Time…

WHO is using it?Did we build the RIGHT thing?

Page 14: Metrics-Driven Devops: Delivering High Quality Software Faster!

Online Sportsclub Search Service

2015201420xx

Response Time

2016+

1) Started as a small project

2) Slowly growing user base

3) Expanding to new markets –

1st performance degradation!

4) Adding more markets – performance becomes

a business impact Users

4) Potentially start loosing users

Page 15: Metrics-Driven Devops: Delivering High Quality Software Faster!

Early 2015: Monolithic App

Can‘t scale vertically endlessly!

2.68s Load Time

94.09% CPU Bound

Page 16: Metrics-Driven Devops: Delivering High Quality Software Faster!

Proposal: Service approach!

Front Endto Cloud

Scale Backendin Containers!

Page 17: Metrics-Driven Devops: Delivering High Quality Software Faster!

7:00 a.m.Low Load and Service runningon minimum redundancy

12:00 p.m.Scaled up service during peak loadwith failover of problematic node

7:00 p.m.Scaled down again to lower loadand move to different geo location

Testing the Backend Service alone scales well …

Page 18: Metrics-Driven Devops: Delivering High Quality Software Faster!

Go live – 7:00 a.m.

Page 19: Metrics-Driven Devops: Delivering High Quality Software Faster!

Go live – 12:00 p.m.

Page 20: Metrics-Driven Devops: Delivering High Quality Software Faster!

What Went Wrong?

Page 21: Metrics-Driven Devops: Delivering High Quality Software Faster!

26.7s Load Time5kB Payload

33! Service Calls

99kB - 3kB for each call!

171! Total SQL Count

Architecture ViolationDirect access to DB from frontend service

Key Metrics: Top Search Query end-to-end

Page 22: Metrics-Driven Devops: Delivering High Quality Software Faster!

The fixed end-to-end use case“Re-architect” vs. “Migrate” to Service-Orientation

2.5s (vs 26.7) 5kB Payload

1! (vs 33!) Service Call

5kB (vs 99) Payload!

3! (vs 177) Total SQL Count

Page 23: Metrics-Driven Devops: Delivering High Quality Software Faster!

Build # Use Case Stat # API Calls # SQL Payload CPU

Use Case Tests and Monitors Service & App Metrics Ops#ServInst Usage RT

With App Metrics from&for Dev(to)OpsScenario: Monolithic App with 2 Key Features

Page 24: Metrics-Driven Devops: Delivering High Quality Software Faster!

Build 17 testNewsAlert OKtestSearch OK

Build # Use Case Stat # API Calls # SQL Payload CPU

1 5 2kb 70ms1 3 5kb 120ms

Use Case Tests and Monitors Service & App Metrics Ops#ServInst Usage RT

1 0.5% 7.2s

1 63% 5.2s

With App Metrics from&for Dev(to)OpsScenario: Monolithic App with 2 Key Features

Page 25: Metrics-Driven Devops: Delivering High Quality Software Faster!

Build 17 testNewsAlert OKtestSearch OK

Build # Use Case Stat # API Calls # SQL Payload CPU

1 5 2kb 70ms1 3 5kb 120ms

Use Case Tests and Monitors Service & App Metrics Ops#ServInst Usage RT

1 0.5% 7.2s

1 63% 5.2s

With App Metrics from&for Dev(to)Ops

Re-architecture -> Performance Fixes

Scenario: Monolithic App with 2 Key Features

Page 26: Metrics-Driven Devops: Delivering High Quality Software Faster!

Build 17 testNewsAlert OKtestSearch OK

Build # Use Case Stat # API Calls # SQL Payload CPU

1 5 2kb 70ms1 3 5kb 120ms

Use Case Tests and Monitors Service & App Metrics

Build 25 testNewsAlert OKtestSearch OK

1 4 1kb 60ms34 171 104kb 550ms

Ops#ServInst Usage RT

1 0.5% 7.2s

1 63% 5.2s

With App Metrics from&for Dev(to)Ops

Re-architecture -> Performance Fixes

Scenario: Monolithic App with 2 Key Features

Page 27: Metrics-Driven Devops: Delivering High Quality Software Faster!

Build 17 testNewsAlert OKtestSearch OK

Build # Use Case Stat # API Calls # SQL Payload CPU

1 5 2kb 70ms1 3 5kb 120ms

Use Case Tests and Monitors Service & App Metrics

Build 26 testNewsAlert OKtestSearch OK

Build 25 testNewsAlert OKtestSearch OK

1 4 1kb 60ms34 171 104kb 550ms

Ops#ServInst Usage RT

1 0.5% 7.2s

1 63% 5.2s

1 4 1kb 60ms2 3 10kb 150ms

1 0.6% 4.2s

5 75% 2.5s

With App Metrics from&for Dev(to)Ops

Re-architecture -> Performance Fixes

Scenario: Monolithic App with 2 Key Features

Page 28: Metrics-Driven Devops: Delivering High Quality Software Faster!

Build 17 testNewsAlert OKtestSearch OK

Build # Use Case Stat # API Calls # SQL Payload CPU

1 5 2kb 70ms1 3 5kb 120ms

Use Case Tests and Monitors Service & App Metrics

Build 26 testNewsAlert OKtestSearch OK

Build 25 testNewsAlert OKtestSearch OK

1 4 1kb 60ms34 171 104kb 550ms

Ops#ServInst Usage RT

1 0.5% 7.2s

1 63% 5.2s

1 4 1kb 60ms2 3 10kb 150ms

1 0.6% 4.2s

5 75% 2.5s

Build 35 testNewsAlert -testSearch OK

- - - -2 3 10kb 150ms

- - -

8 80% 2.0s

With App Metrics from&for Dev(to)Ops

Re-architecture -> Performance Fixes

Scenario: Monolithic App with 2 Key Features

Page 29: Metrics-Driven Devops: Delivering High Quality Software Faster!

Build 17 testNewsAlert OKtestSearch OK

Build # Use Case Stat # API Calls # SQL Payload CPU

1 5 2kb 70ms1 3 5kb 120ms

Use Case Tests and Monitors Service & App Metrics

Build 26 testNewsAlert OKtestSearch OK

Build 25 testNewsAlert OKtestSearch OK

1 4 1kb 60ms34 171 104kb 550ms

Ops#ServInst Usage RT

1 0.5% 7.2s

1 63% 5.2s

1 4 1kb 60ms2 3 10kb 150ms

1 0.6% 4.2s

5 75% 2.5s

Build 35 testNewsAlert -testSearch OK

- - - -2 3 10kb 150ms

- - -

8 80% 2.0s

With App Metrics from&for Dev(to)Ops

Re-architecture -> Performance Fixes

Scenario: Monolithic App with 2 Key Features

Page 30: Metrics-Driven Devops: Delivering High Quality Software Faster!

How

gets you there!

Page 31: Metrics-Driven Devops: Delivering High Quality Software Faster!

I am a Developer ...

... and want to only check-in good code!

Page 32: Metrics-Driven Devops: Delivering High Quality Software Faster!

#1: Don’t Check In Bad CodeStep #1: Execute your Tests just as you always do ...

Step #2: ... but DO IT WITH

Dynatrace!!

Page 33: Metrics-Driven Devops: Delivering High Quality Software Faster!

#1: Don’t Check In Bad CodeStep #1: Execute your Tests just as you always do ...

Step #2: ... but DO IT WITH

Dynatrace!!

Step #3: Verify Code works as intended?

Page 34: Metrics-Driven Devops: Delivering High Quality Software Faster!

I am a Functional Tester ...

... and want to find non-functional problems

Page 35: Metrics-Driven Devops: Delivering High Quality Software Faster!

#2: Identify Non-Functional Problems1: Loading of Homepage

2: Click on Search

WPO: Analyze W3C Timings for

any obvious mistakes, e.g: too many images, ...

Architecture: Identify common problem patterns,

e.g: N+1 Query Problem

Page 36: Metrics-Driven Devops: Delivering High Quality Software Faster!

I am a Test Automation Engineer ...

... and want to integrate App Metrics into my

tests

Page 37: Metrics-Driven Devops: Delivering High Quality Software Faster!

#3: Integrate with your Testing Tools

HTTP-based Testing: Leverage the X-Dynatrace HTTP

Header Integration

Page 38: Metrics-Driven Devops: Delivering High Quality Software Faster!

#3: Integrate with your Testing Tools

HTTP-based Testing: Leverage the X-Dynatrace HTTP

Header Integration

HTTP-based Testing: Makes analyzing these requests much easier

in Dynatrace

Page 39: Metrics-Driven Devops: Delivering High Quality Software Faster!

#3: Integrate with your Testing Tools Pull Your Test Tool Results

into DynatraceCompare with

Dynatrace Data

App Metrics DB, Exceptions, CPU, ...

for each Test Step

Page 40: Metrics-Driven Devops: Delivering High Quality Software Faster!

I am a Build Engineer ...

... and want to integrate App

Metrics into my Pipeline to stop

bad builds!

Page 41: Metrics-Driven Devops: Delivering High Quality Software Faster!

#1: Analyzing every Unit, Integration & REST API test

#2: Key Architectural Metrics for each test

#3: Detecting regression based on measure per Checkin

#4: Automatically Stop Bad Builds in CI

Page 42: Metrics-Driven Devops: Delivering High Quality Software Faster!

#4: Integration Powered by REST API

Page 43: Metrics-Driven Devops: Delivering High Quality Software Faster!

I am a (Dev/App)Ops ...

... and want to identify good/bad

features & deployments

based on Usage and App Metrics!

Page 44: Metrics-Driven Devops: Delivering High Quality Software Faster!

#5: Monitor your Services/Users in Prod #1: Usage

Tip: UEM Conversion!#2: Load vs Response

Tip: See unusual spikes

#3: Architectural Metrics DB, Exceptions, Web

Service Calls

Page 45: Metrics-Driven Devops: Delivering High Quality Software Faster!

I am a (Biz)Ops ...

... and want to improve user experience &

behavior

Page 46: Metrics-Driven Devops: Delivering High Quality Software Faster!

#6: UX Analysis based on UEM Data

Page 47: Metrics-Driven Devops: Delivering High Quality Software Faster!

#6: UX Analysis based on UEM Data

Page 48: Metrics-Driven Devops: Delivering High Quality Software Faster!

We are DevOps

... and only deploy software that approved by Dynatrace!

Page 49: Metrics-Driven Devops: Delivering High Quality Software Faster!

Confidential, Dynatrace, LLC

„Level-Up my friends! Work on your own bright

future! Dont‘ fight the change!

#devops“Andreas Grabner (@grabnerandi)

Page 50: Metrics-Driven Devops: Delivering High Quality Software Faster!
Page 51: Metrics-Driven Devops: Delivering High Quality Software Faster!

Use application metrics as additional Quality Gates

Page 52: Metrics-Driven Devops: Delivering High Quality Software Faster!

Use application metrics as additional Quality GatesDev&Test: Personal

License to Stop Bad Code when it gets created!

Tip: Dont leave your IDE!

Page 53: Metrics-Driven Devops: Delivering High Quality Software Faster!

Use application metrics as additional Quality GatesDev&Test: Personal

License to Stop Bad Code when it gets created!

Tip: Dont leave your IDE!

Automated Tests: Identify Non-Functional Problems by looking at App Metrics

Tip: Feed data back into your test tool!

Page 54: Metrics-Driven Devops: Delivering High Quality Software Faster!

Use application metrics as additional Quality GatesDev&Test: Personal

License to Stop Bad Code when it gets created!

Tip: Dont leave your IDE!

Continuous Integration: Auto-Stop Bad Builds based on AppMetrics from Unit-, Integration, - Perf Tests

Tip: integrate with Jenkins, Bamboo, ...

Automated Tests: Identify Non-Functional Problems by looking at App Metrics

Tip: Feed data back into your test tool!

Page 55: Metrics-Driven Devops: Delivering High Quality Software Faster!

Use application metrics as additional Quality GatesDev&Test: Personal

License to Stop Bad Code when it gets created!

Tip: Dont leave your IDE!

Continuous Integration: Auto-Stop Bad Builds based on AppMetrics from Unit-, Integration, - Perf Tests

Tip: integrate with Jenkins, Bamboo, ...

Prod: Monitor Usage and Runtime Behavior per Service, User Action,

Feature ...Tip: Stream to ELK, Splunk and Co ...

Automated Tests: Identify Non-Functional Problems by looking at App Metrics

Tip: Feed data back into your test tool!

Page 56: Metrics-Driven Devops: Delivering High Quality Software Faster!

QuestionsSlides: slideshare.net/grabnerandiGet Tools: bit.ly/dtpersonalYouTube Tutorials: bit.ly/dttutorialsContact Me: [email protected] Me: @grabnerandiRead More: blog.dynatrace.com

Page 57: Metrics-Driven Devops: Delivering High Quality Software Faster!

Andreas GrabnerDynatrace Developer Advocate@grabnerandihttp://blog.dynatrace.com