Performance Quality Metrics along CD

Pipeline@grabnerandi

http://apmblog.compuware.com

And just very recently @ Euro Song Contest

What you should check out …

Tonight we are here for …

Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers

Automated Acceptance

Testing

Automated Capacity Testing

Manual testing• Key showcases• Exploratory testing Release

Cont. Delivery / Deploy

Or – The Evolution of Software Engineering

Who is doing it? How many successful deployments can they do?

300 Deployments / Year

50-60 Deployments / Day

10+ Deployments / Day

Every 11.6 seconds

More on Amazons Story

75% fewer outages since 2006

90% fewer outage minutes

~0.001% of deployments cause a problem

Instantaneous automatic rollback

Deploying every 11.6s

Testing is Important – and gives Confidence

But are we ready for “The Real” world?

Measure Performance during the game

Ball Possession: 40 : 60Fouls: 0 : 0Score: 0 : 0

Minute 1 - 5

Measure Performance during the game

Minute 6 - 35

Ball Possession: 80 : 20Fouls: 2 : 12Score: 0 : 0

Deep Dive Analysis

Options “To Fix” the situation

Not always a happy ending

Minute 90

Ball Possession: 80 : 20Fouls: 4 : 25Score: 3 : 0

FRUSTRATED FANS!!

23

How does that relate to

Software?

How does that relate to

Software?

From Deploy to …

Deploy Promotion/Event Problems Ops Playbook War Room

Timeline

The “War Room” – back then

'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970

The “War Room” – back then

'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970

The “War Room” – NOW

Facebook – December 2012

The “War Room” – NOW

Facebook – December 2012

4 Situations on

WHY this happens,

HOW to avoid it

4 Situations on

WHY this happens,

HOW to avoid it

“Blindly” (Re)use Existing

Components

Requirement: We need a report

Using Hibernate results in 4k+ SQL Statements to display 3 items!

Hibernate Executes 4k+ Statements

Individual Execution VERY

FAST

But Total SUM takes 6s

Requirement: We need a fancy UI

Using Telerik Controls Results in 9s for Data-Binding of UI Controls

#1: Slow Stored ProcedureDepending on Request

execution time of this SP varies between 1 and 7.5s

#2: 240! Similar SQL StatementsMost of these 240! Statements are

not prepared and just differ in things like Column Names

Metrics: # Total SQLs# SQLs / Web Request# Same SQLs / Request

Transferred Rows

Test: With realistic Data

Dev: “Learn” Frameworks

Implementation Flaws

Business Impact requires Action!

Solution: Cache to the RESCUE!!

Implementation and Rollout

Implemented InMemory Cache

Worked well in Load Testing

Result: Out of Memory Crashes!!

Still crashes

Problem fixed!Fixed Version Deployed

Metrics: Heap Size, # Objects Allocated,# Objects in Cache

Cache Hit Ratio

Test: With realistic Data

12 000 000 $

#No “Agile” Deployment

Ad on air

Availability dropped to 0%

Load Spike resulted in Unavailability

Alternative: “GoDaddy goes DevOps”

Response time improved 4x

1h before SuperBowl KickOff

1h after Game ended

Behind the Scenes

Metrics: AvailabilityPage Size, # Objects

# Hosts, # Connections

DevOps: “Feature” Switches

#Push without a Plan

Mobile Landing Page of Super Bowl Ad

434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …

Total size of ~ 20MB

m.store.com redirects to www.store.com

ALL CSS and JS files are redirected to the www domain

This is a lot of time “wasted” especially on high latency mobile

connections

Metrics: Load Time, # Resources (Images, …),

# HTTP 3xx, 4xx, 5xx

Dev: Build for Mobile

Test: Test on Mobile

Ops: Monitor Mobile

#1: Define Measures# of Requests / User

# of Log Messages

# of Exceptions

# Objects Allocated

# Objects In Cache

Cache Hit Ratio

# of Images

# of SQLs

# SQLs per RequestAvailability

# HTTP 3xx, 4xx

Page Size

#2: Automate

Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers

Automated Acceptance

Testing

Automated Capacity Testing

Manual testing• Key showcases• Exploratory testing Release

Unit & Integration Tests

Functional Tests

Performance TestsProductionMonitoring

Functional Tests

57

#3: Share Results

#4: Integrate

If we do all that

CANCELLED

Which gives you more time for the real important things in life …

Want MORE of these and more details?

http://apmblog.compuware.com

FREE Products & More Info

• dynaTrace Enterprise– Full End-to-End Visibility in your Java, .NET, PHP Apps

– Sign up for a 15 Days Free Trial on http://compuwareapm.com

• dynaTrace AJAX Edition– Browser Diagnostics for IE + FF

– Download @ http://ajax.dynatrace.com

• Our Blog: http://apmblog.compuware.com