DevOps and PerformanceWhy, How & Best Practices
@grabnerandihttp://apmblog.compuware.com
What you may have heard about Austrians
And just very recently @ Euro Song Contest
How we would like the world to see us
What we are also proud of
What you should check out …
Tonight we are here for …
Others Call ItThe stuff we did
when we were a Start Up and we All were
Devs, Testers and Ops
Or – The Evolution of Software Engineering
YOU ARE NOT ALONE: Popularity on Google
Who is doing it? How many successful deployments can they do?
300 Deployments / Year
50-60 Deployments / Day
10+ Deployments / Day
Every 11.6 seconds
More on Amazons Story
75% fewer outages since 2006
90% fewer outage minutes
~0.001% of deployments cause a problem
Instantaneous automatic rollback
Deploying every 11.6s
Testing is Important – and gives Confidence
But are we ready for “The Real” world?
Measure Performance during the game
Ball Possession: 40 : 60Fouls: 0 : 0Score: 0 : 0
Minute 1 - 5
Measure Performance during the game
Minute 6 - 35
Ball Possession: 80 : 20Fouls: 2 : 12Score: 0 : 0
Deep Dive Analysis
Options “To Fix” the situation
Not always a happy ending
Minute 90
Ball Possession: 80 : 20Fouls: 4 : 25Score: 3 : 0
FRUSTRATED FANS!!
25
How does that relate to
Software?
From Deploy to …
Deploy Promotion/Event Problems Ops Playbook War Room
Timeline
The “War Room” – back then
'Houston, we have a problem‘NASA Mission Control Center, Apollo 13, 1970
The “War Room” – NOW
Facebook – December 2012
3 Situations on
WHY this happens,
HOW to avoid it
Image taken from https://www.scriptrock.com/blog/devops-whats-hype-about/
#Disconnected Teams
“Teamwork” between Dev and Ops
SEV1 Problem in Production
Need access to log files
Where are they? Can’t get them
Need to increase log level
Can’t do! Can’t change config files in prod!
“Solution”: Implement a Custom “On Demand” Remote Logger
Implementation and Rollout
Implemented Custom Logger
Worked well in Load Testing
What happened?
~ 1Mio Lock Exceptions in 30 mins
Root Cause: A special WebSphere Setting!
Log Service provides a synchronized log file across ALL JVMs
Log Service provides a synchronized log file across
ALL JVMs
Metrics: # Log Messages, # Exceptions
Share: Same Server Settings
DevOps: Agree on Data for Troubleshooting
12 000 000 $
#No “Agile” Deployment
Ad on air
Availability dropped to 0%
Load Spike resulted in Unavailability
Alternative: “GoDaddy goes DevOps”
Response time improved 4x
1h before SuperBowl KickOff
1h after Game ended
Behind the Scenes
Metrics: AvailabilityPage Size, # Objects
# Hosts, # Connections
DevOps: “Feature” Switches
#Push without a Plan
Mobile Landing Page of Super Bowl Ad
434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~ 20MB
m.store.com redirects to www.store.com
ALL CSS and JS files are redirected to the www domain
This is a lot of time “wasted” especially on high latency mobile
connections
Critical Pages not Optimized!
Browse, Search and Product Info
performs well
… because they don’t follow best practices: 87 Requests, 28
Redirects, …
Critical Pages such as Shopping Cart are very
slow …
Metrics: Load Time, # Resources (Images, …),
# HTTP 3xx, 4xx, 5xx
Dev: Build for MobileTest: Test on MobileOps: Monitor Mobile
#2: Define Measures# of Requests / User
# of Log Messages
# of Exceptions
# Objects Allocated
# Objects In Cache
Cache Hit Ratio
# of Images
# of SQLs
# SQLs per RequestAvailability
# HTTP 3xx, 4xx
Page Size
54
#3: Share Results
#4: Automate
Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers
Automated Acceptance
Testing
Automated Capacity Testing
Manual testing• Key showcases• Exploratory testing Release
Unit & Integration Tests
Functional Tests
Performance Tests ProductionMonitoring
Functional Tests
#5: Integrate
If we do all that
CANCELLED
Which gives you more time for the real important things in life …
Recommended Book
https://itrevolution.wufoo.com/forms/phoenix-project-ebook-offer/
FREE Products & More Info
• dynaTrace Enterprise– Full End-to-End Visibility in your Java, .NET, PHP Apps– Sign up for a 15 Days Free Trial on http://compuwareapm.com
• dynaTrace AJAX Edition– Browser Diagnostics for IE + FF– Download @ http://ajax.dynatrace.com
• Our Blog: http://apmblog.compuware.com