how to get automatic analysis for load test results
TRANSCRIPT
![Page 1: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/1.jpg)
Automatic Analysis for Load Test Results
by Andrey PokhilkoChief Scientist, BlazeMeter Inc.
February 3rd, 2016Mountain View, CA
![Page 2: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/2.jpg)
About Me
My key interest: tools that help people enjoy their job
Current: Chief Scientist for BlazeMeter
Past: Performance Engineering in Yandex
JMeter Plugins project founder http://jmeter-plugins.org/
Loadosophia author http://loadosophia.org/
Taurus Tool lead dev http://gettaurus.org/
![Page 3: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/3.jpg)
Look Out of the Window
1. It’s the era of total speed-up
2. Which is solved by total automation
3. Which leads to CI/CD as standard practices
You can’t have a human in the loop anymore
![Page 4: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/4.jpg)
Understand how humans do it, then automate the process
![Page 5: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/5.jpg)
Analysis
Raw DataInformation DecisionsFacts
True Analysis is Here
![Page 6: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/6.jpg)
What is our “Load Test Results Analysis” ?
1. Decide if results are reliable
2. Break complex test timeline into components - surprisingly important
3. Decide if it had hit bottleneck, had memory leak etc
4. Understand APM correlations
5. Decide if something has changed since the last test / over recent N tests
![Page 7: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/7.jpg)
When do we do it?
1. Online while test is still running
2. Post-test
3. Comparing tests to a specific baseline
4. Observing KPI history from a series of tests
Unconsciously most of the time
Speed Constraint
mainchallenge
ResourcesUsed
![Page 8: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/8.jpg)
Challenge Accepted
Isn’t it trivial?
Wasn’t it done before?
Why can’t we reuse what APM vendors do these days?
![Page 9: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/9.jpg)
How can we do it?
![Page 10: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/10.jpg)
“Automated” vs “Automatic”
Disambiguation:
![Page 11: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/11.jpg)
Apply the right test signal
First and foremost:
![Page 12: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/12.jpg)
Right test signal
Load Profile
Real test intention
MeasuredResponseSystem
![Page 13: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/13.jpg)
Common Test Signals aka Load Profiles
Stress, Max Throughput => Capacity, “how much”
Constant, Soak, Sustain => Quality, “how good”
Spike, recovery
Combinations => reality requires to adapt
Stress
Constant
Mixed
Recovery
![Page 14: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/14.jpg)
Each testing process stage has own techniques
![Page 15: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/15.jpg)
In-Test Automation
Instant threshold
Windowed threshold => average / instant
Steadiness threshold => stop constant test earlier
Decisions to be made: stop test, mark it failed, mark it passed
![Page 16: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/16.jpg)
In-Test Automation: Implementations
JMeter-Plugins Autostop - show it
Taurus Pass/Fail criteria - show it
BlazeMeter Thresholds
![Page 17: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/17.jpg)
Post-Test Analysis
Stage breakdown
1. Ramp-Up Stage
2. Constant Stage
3. Leftovers
POI detection
APM correlation- “measure everything” costs you in the analysis stage
![Page 18: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/18.jpg)
Post-Test Analysis: Implementations
BlazeMeter App => Thresholds
Loadosophia.org:
stages breakdown
test classification
refined KPIs - it’s all about speed
monitoring correlation
85% to 95% correct
![Page 19: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/19.jpg)
Test Comparison & Series Trend
Have reference test (aka “baseline”)
Raise flag on too much difference
Notify on trend outlier
Notify on trend reaching limit
It’s all easy to automate with APIs + simple scripts
![Page 20: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/20.jpg)
We’ve Learned Today
1. Why to automate results analysis
2. What our “result analysis” is
3. Prerequisites to make it automatable
4. Particular techniques & implementations
![Page 21: How to Get Automatic Analysis for Load Test Results](https://reader033.vdocuments.net/reader033/viewer/2022051706/58f012371a28aba50b8b4671/html5/thumbnails/21.jpg)
Now Q&A (my favorite!)