load testing using continuous integration tools
TRANSCRIPT
![Page 1: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/1.jpg)
LOAD TESTINGusing continuous integration
toolsRichard Pitts
Sr. Load Test Engineer | CommerceHub
![Page 2: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/2.jpg)
Agenda
Our experience adding NeoLoad to our Continuous Integration process.
1. Benefits of using CI to run NeoLoad2. Why we run NeoLoad using CI3. How NeoLoad + CI has helped us4. Common pitfalls to avoid5. Getting started + best practices
![Page 3: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/3.jpg)
About CommerceHub
Hosted services for e-commerce & retail. [B2B] Connection between 200+ retailers and 8,900 brands/distributors. Translate & route drop shipment orders amongst business partners. Goal: Facilitate e-commerce sales growth for our customers Processed $10bil GrossMerchValue for our customers in 2014 [33.7mil
orders]
Why performance matters to us: “retail busy season”: Black Friday Christmas 38k+ users 8.25 mil logins/yr 2 mil hits per month on most visited web page
![Page 4: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/4.jpg)
Benefits of using CI to run NeoLoad
Same benefit as any CI Job Easy to execute Easy to schedule Easy to monitor Build history Results in Workspace
![Page 5: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/5.jpg)
Benefits of using NeoLoad Plugin Retrieve
Neoload results back into to Jenkins
Display Trends: Avg Response Time & Error Rate
Maintain build history & preserve Neoload reports for each build.
![Page 6: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/6.jpg)
Benefits of using NeoLoad Plugin
Collaborate.Easy to share.Link to the build.
![Page 7: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/7.jpg)
Why you should run NeoLoad using CI If load testing occurs less frequently than desired New development methodologies more frequent releases Scheduled automation avoids waiting on humans, keeps a
faster pace.
Our Story… before NeoLoad with NeoLoad with NeoLoad plus CI
![Page 8: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/8.jpg)
Load Testing before NeoLoad
• Close to the release date• Late testing Late Discoveries Delayed Releases.
feature complete build
Monthly releases
Performance test@ end of dev cycle
![Page 9: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/9.jpg)
Load Testing before NeoLoad An automated tool could help us load test. Hopefully one that would inspire us to load test earlier + more
often.
Started to Trial + Assess several load testing tools
![Page 10: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/10.jpg)
NeoLoad pilot
Began recording &designing our Neoload scripts
![Page 11: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/11.jpg)
A better Load Test environment
Properly designed for load testing @ high volume [150+ users]
Used CI to automate deployment & configuration of this env
On-premise Load Generators Web cluster behind Load Balancer
![Page 12: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/12.jpg)
NeoLoad pilot: real progress
Started generating performance test results
![Page 13: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/13.jpg)
Success implement an automated load testing tool library of scripts now easier to include load testing for releases load testing more things, more often stockpile load test results for future benchmark comparisons
![Page 14: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/14.jpg)
NeoLoad + Waterfall: How to?
So, how did we regularly include NeoLoad tests as part of our waterfall releases?
![Page 15: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/15.jpg)
NeoLoad + Waterfall: Option 1Three options to include Neoload testing for a release:
1) author new NeoLoad scripts designed to target specific concerns. Record/Playback Configure for multi-user concurrency & unique data Run scripts Share results
![Page 16: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/16.jpg)
NeoLoad + Waterfall: Option 2Three options to include Neoload testing for a release:
2) run existing NeoLoad scripts + compare w/ previous release results
Try/validate the scripts Update/fix outdated scripts Run scripts Share results
![Page 17: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/17.jpg)
NeoLoad + Waterfall: Option 3Three options to include Neoload testing for a release:
3) run NeoLoad scripts on project vs mainline branches, comparing results
Run scripts on mainline branch to obtain baseline results Run scripts on project branch Compare project results to baseline results Is performance of the project branch better/no worse/worse than
mainline
![Page 18: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/18.jpg)
Results: Still a Waterfall Performance testing still occurred @ end of dev cycles Limited time to re-work performance issues before release dates. Often, the tests were short samples due to time constraints. The most recent set of results for comparison was often many weeks
old. Centralized around performance engineer (bottleneck risk) Sometimes it was “testing for the sake of testing”
![Page 19: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/19.jpg)
Meanwhile: Along comes Agile Continuous Integration of code Automated CI builds Continuous Delivery to test environments Automated regressions using CI builds Cross functional team members “Everyone is responsible for quality” approach
Result: Sped up development + shortened our dev lifecycle!
Release more frequently
![Page 20: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/20.jpg)
Load Testing + Agile Teams Team mindset: “Performance is everybody's job” But, most team members had little load testing experience.
Result: Still had all of the same problems getting Load Tests done.
Completed “in the margins” by perf engineer + agile team member. Relied upon Ops and/or performance engineer (bottleneck risk)
Load testing wasn’t keeping pace with our Agile teams. Needed a way to bake Load Testing into our Agile process.
![Page 21: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/21.jpg)
Load Testing + Agile Teams How could we include LT more often, under Agile process?
Manual performance testing? [+leverage firebug, Yslow, Fiddler, etc.]
Collect performance metrics while we run our automated regression?
WebPagetest.org? Schedule NeoLoad to run using Cron jobs? Schedule NeoLoad to run using NeoLoad internal scheduler? Schedule Neoload to run using CI? Hmmmm… we do have
Jenkins.
![Page 22: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/22.jpg)
Jenkins NeoLoad plugin v4.2 [Nov 2013]
Retrieve Neoload results back into to Jenkins
Display Trends: Avg Response Time & Error Rate
Maintain build history & preserve Neoload reports for each build.
![Page 23: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/23.jpg)
Gains from NeoLoad + CI
Daily performance “pulse”
![Page 24: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/24.jpg)
Gains from NeoLoad + CIJUnit test results based on SLA profiles
![Page 25: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/25.jpg)
Gains from NeoLoad + CI
Collaborate.Easy to share.Link to the build.
Increased Collaboration.
![Page 26: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/26.jpg)
NeoLoad Tests: Controlled by CIScheduled nightly.
![Page 27: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/27.jpg)
Lessons learnedPitfalls to Avoid:
Using only “happy path” scripts Comparing results over long gaps time risky Comparing multiple short samples inaccurate Swap sharing the Load Testing environment bottlenecks Requiring involvement from Ops or perf. engineer bottlenecks
![Page 28: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/28.jpg)
Your turn! Run NeoLoad using Continuous Integration
![Page 29: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/29.jpg)
Step1: get a CI Server, setup for NeoLoad
We used an existing remote Jenkins server.
Use your existing remote CI serveror install Jenkins along side NeoLoad.
![Page 30: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/30.jpg)
Step2: add CI step to run NeoLoadCmd
Neotys’ Jenkins Integration Guide explains how to configure –options
![Page 31: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/31.jpg)
Step3: Install NeoLoad plugin for Jenkins
![Page 32: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/32.jpg)
Step4: retrieve Neoload results
Jenkins Integration Guide explains how to incorporate NeoLoad results
![Page 33: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/33.jpg)
Where to go from here?Best practices & Pro Tips:
Incorporate performance testing into every DevOps team Intelligent testing: only run load scripts that test the code that was
changed Reduce the amount of time & resources needed for load testing
For “hardening sprints”- setup a Stress test scenario. Run on-demand using CI
Make use of email alerts and chat notifications in CI
![Page 34: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/34.jpg)
Best Practices: Setup SLA profiles
Setup SLA profiles so pass/fail criteria are bubbled up to CI results
![Page 35: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/35.jpg)
Best Practices: Increase collaboration
![Page 36: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/36.jpg)
Best Practices: Smoke test Smoke test first (low volume), then run full
Load Test
![Page 37: Load Testing using Continuous Integration tools](https://reader035.vdocuments.net/reader035/viewer/2022062821/588886e61a28ab3e658b4fc9/html5/thumbnails/37.jpg)
Richard Pitts We are hiring:[email protected] http://www.commercehub.com/about-us/careers/
Share with me:How are you using CI to drive your NeoLoad process? Let me know.
Learn more:http://www.neotys.com/introduction/neoload-continuous-integration.htmlhttp://
www.neotys.com/webcast/Continuous-Performance-Testing-with-NeoLoad.htmlhttps://wiki.jenkins-ci.org/display/JENKINS/NeoLoad+Pluginhttps://github.com/jenkinsci/neoload-plugin