![Page 1: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/1.jpg)
01 February 2017 www.retit.de 1
Continuous Performance Evaluationusing Open Source Tools
Markus Dlugi, RETIT GmbH
![Page 2: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/2.jpg)
01 February 2017 www.retit.de 2
• Continuous Integration (CI) has reached widespread adoption
• Functional requirements are tested with automated tests
• Non-functional requirements such as performance are often neglected
How can we track performance if the application changes every day?
Every hour?
Every minute?
Motivation
![Page 3: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/3.jpg)
01 February 2017 www.retit.de 3
• Modern APM tools often have features to monitor the performance over
multiple application versions
• However, they generally come with a high price tag
Maybe we can also do it with Open Source Software?
Motivation
![Page 4: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/4.jpg)
01 February 2017 www.retit.de 4
General workflow
VCS CI system
Load testing tool
Application
builds &
deploys
runs
tests
Performance
Plugin
analyzes
runs
Performance analysis tool
APM tool
sends datanotifies
![Page 5: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/5.jpg)
01 February 2017 www.retit.de 5
General workflow
VCS CI system
Load testing tool
Application
builds &
deploys
runs
tests
Performance
Plugin
analyzes
runs
Performance analysis tool
APM tool
sends datanotifies
Performance Analyst
Developer /
Architect /
Tester
![Page 6: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/6.jpg)
01 February 2017 www.retit.de 6
Performance Analyst
Developer /
Architect /
Tester
builds &
deploys
Alternatives: Version Control System
VCS CI system
Load testing tool
Application
runs
tests
Performance
Plugin
analyzes
runs
Performance analysis tool
APM tool
sends datanotifies
SubversionGit CVS
CVS
Mercurial
and
many
more
![Page 7: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/7.jpg)
01 February 2017 www.retit.de 7
Performance Analyst
Developer /
Architect /
Tester
Alternatives: Continuous Integration Systems
VCS CI system
Load testing tool
Application
builds &
deploys
runs
tests
Performance
Plugin
analyzes
runs
Performance analysis tool
APM tool
sends datanotifies
Jenkins
Bamboo
Closed Source alternatives:
TeamCity
CruiseControl
Open Source alternatives:
![Page 8: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/8.jpg)
01 February 2017 www.retit.de 8
Performance Analyst
Developer /
Architect /
Tester
Alternatives: Load Testing Tools
VCS CI system
Load testing tool
Application
builds &
deploys
runs
tests
Performance
Plugin
analyzes
runs
Performance analysis tool
APM tool
sends datanotifies
JMeter GatlingLoadRunner
Closed Source alternatives:
Silk Performer
Open Source alternatives:
The Grinder
![Page 9: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/9.jpg)
01 February 2017 www.retit.de 9
Performance Analyst
Developer /
Architect /
Tester
Alternatives: APM Tools
VCS CI system
Load testing tool
Application
builds &
deploys
runs
tests
Performance
Plugin
analyzes
runs
Performance analysis tool
APM tool
sends datanotifies
inspectITKiekerZipkinICINGAPinPointZMON
ZMON
Closed Source alternatives:
AppDynamics Dynatrace New RelicInstana
and
many
more
Open Source alternatives:
![Page 10: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/10.jpg)
Demo
https://github.com/RETIT/continuous-performance-evaluation
![Page 11: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/11.jpg)
01 February 2017 www.retit.de 11
• Pipelines in Jenkins allow to specify all steps using Groovy scripts
Avoid manual configuration
Treat pipeline as code
Tip #1: Use Pipelines
![Page 12: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/12.jpg)
01 February 2017 www.retit.de 12
• Load tests only yield meaningful results if used
correctly
• When designing your tests, make sure that:
• Your workload resembles how users would
access the application
• You achieve steady state performance (test
runs long enough?)
• Your test environment is comparable to your
production environment
• Your test environment has no external
influences (e.g., from the load test
environment)
Tip #2: Use Load Tests correctly
![Page 13: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/13.jpg)
01 February 2017 www.retit.de 13
• Thresholds are a good way of notifying
developers about performance problems
• However, good thresholds are hard to find
• If the threshold is too low, developers will
start ignoring the warnings
• If the threshold is too high, potential
problems might go unnoticed
• Each application and load test is different
• Finding good thresholds takes time
• Consider using absolute thresholds
• You might also need to remove ramp-up
using custom scripts
Tip #3: Find good Thresholds
![Page 14: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/14.jpg)
01 February 2017 www.retit.de 14
• Ideally, you would run your performance
tests after every commit
• If you can’t do that, minimum is once a
day
• Try to minimize the pipeline run time so
you can run it more often
Tip #4: Run the Pipeline often
![Page 15: Continuous Performance Evaluation using Open Source · PDF file• Functional requirements are tested with automated tests ... Load testing tool Application builds & deploys runs](https://reader031.vdocuments.net/reader031/viewer/2022022419/5a78df817f8b9a70238db597/html5/thumbnails/15.jpg)
01 February 2017 www.retit.de 15
• Minimize data which needs to be processed, sent
or stored
• Use JMeter’s CSV output instead of XML
• Don’t store every Jenkins build, introduce log
rotation
• Use partial database resets (e.g., only delete
data generated during the last test execution)
instead of full restores
• Use caching effectively
• Maven’s local repository
• Docker’s image cache
Tip #5: Optimize your Pipeline