the secret art of agile performance testing
TRANSCRIPT
The Secret Art of Agile Performance TestingAlois Reitbauer, dynaTrace Software
The Classical Approach
Waterfalls are pretty
But might get scary
Dream
Comfort Zone
Reality
Iteration
Iteration
Iteration
Test
Test
Dream
Comfort Zone
Reality
Iteration
Iteration
Iteration
Test
Test
Iteration
Iteration
Iteration
NFRs and Maslow
The Focus Problem
Methodology
The Agile Approach
Defining our Strategy
Start Early
Test Continuously
Break in Pieces
Creating FocusRi
sk
Impact
AutomateRegular TestingSkip
Frequency vs. Granularity
Frequency
Gra
nu
lari
ty
JUnit-basedTests (2x day)
Total SystemTests
Long-running Stabiltiy Tests(2 w duration)
Granularity
Comparability
Complexity
Quality
Define KPIs
Throughput
Response Time
Memory Consumption
Other KPIs …
MonitorGain Visibility
ResolveReduce Resolution
Time
PreventProactively Avoid
Problems
Follow a Strategy
Avoid Re-Runs
• What could happen?• Which information do you
want?• What describes your system?• What is different from the last
run?
Don´t Forget the Real World
Lessons learned
Profiling is not enough
Good for finding problems
Result comparison hard
Only valid until next check-in
Too much work
Measurements
Stability of Tests
Use Dedicated Hardware
Comparability
Stability
Efficiency
Trends in Unstable Tests
Create Instability
.. adding some volatility increases the likelyness to discover problems …“
Testing scalability
Small Dump Operations Big Dump Operations
Understand your measurements
Response Time only Response Time and GC
Motivate your team
How to make developers write tests
#1 Heroism#2 Boomerang#3 The other guy#4 Bug me not#5 Feedback#6 Code vs. Wine#7 Newb vs. Noob
Test Case Complexity
FirstStart dynaTrace infrastructureWhen readyStart n WebSphere instances on servers … When readyStart Loadtest against WebSphere serversAfter loadtest startExecute test case
Making complex things easy
@DtdRemoteSud( host = "lab2", name = "WAS7.0", startupPriority = 1, postStartClosure = WaitForWebSphereSudIsUp.class ) private SudInterface webSphereSud;
Finding the responsible code
Version Control History Lookup
E-Mail Notification
… better humans
Wrap Up1. Say „Yes“2. Focus3. Define KPIs4. Redefine Done5. Remove Barriers6. Automate7. Monitor8. Resolve9. Prevent
