distributed testing environment
DESCRIPTION
Distributed Testing Environment with Jenkins and virtual machines. Presented on 3-rd PTaQ meetup.TRANSCRIPT
![Page 1: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/1.jpg)
Distributed Testing EnvironmentŁukasz Morawski
![Page 2: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/2.jpg)
The Stories
• John The Tester is a manual tester,
• He is ordered to do an automation,
• So he does,
• He develop some tests, he runs iton his box
![Page 3: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/3.jpg)
The Stories
• The more tests he develops, the more time tests lasts,• At first, it takes couple minutes,• Later, it takes dozen,• It is a lot of time!
He cannot afford to stare at running tests
![Page 4: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/4.jpg)
...and the crucial is...
![Page 5: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/5.jpg)
...and the crucial is
![Page 6: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/6.jpg)
The problem
• QA team needs to provide test feedback as soon as it’s possible,
• Automated tests need to be “Fast and Furious”.
![Page 7: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/7.jpg)
![Page 8: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/8.jpg)
Distributed testing environment on virtual machines.
![Page 9: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/9.jpg)
...but at the beginning there was Eternity...
• server hosting Hudson,• test execution testbed for Selenium IDE tests,
• but one executor means one test at a time.
![Page 10: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/10.jpg)
VM and Hyper-V• QA team uses Hyper-V virtualization software,
• It let us to run ~14 VM,• CPU is not the problem,• Memory is the problem,• 80% free memory rule – swapping leads to serious
performance lost,• We have GEB and BES servers available for us:• BES – manual tests environments for testers and developers
(different OS, different browsers, mobiles),
![Page 11: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/11.jpg)
VM and Hyper-V• GEB – automated tests environments.• Slaves for us and our clients,• Jenkins,• Applications – QA Lab uses this machine to provide tools and
services for our test framework. So far we have W3C Markup Validation service – our Validation Tool uses it.
![Page 12: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/12.jpg)
VM and Hyper-V
All machines are under Nagios supervision, It let us to control if the environment is up and ready to go, Notifies by e-mail if something is wrong.
![Page 13: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/13.jpg)
![Page 14: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/14.jpg)
VM Future
• More tools available for our testing framework (W3C tools),• Different browsers/browsers version,• Maybe different virtualization system – bare metal hyper-
visors seems to look promising,• Cloud – as a main resource, as a backup resource.
![Page 15: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/15.jpg)
Centralized Automated test driving with Jenkins CI
• QA Team use Jenkins CI tool to drive tests,• We have Jenkins deployed in a Tomcat container,• We use SLAVE AGENTS as a communication channel,• It’s not greatest solution because CI tools are rather build
tools not testing tools.
![Page 16: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/16.jpg)
How we use Jenkins...
• We use Views to manage projects
![Page 17: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/17.jpg)
How we use Jenkins...
• We use “distributed builds” feature to manage parallel testing (speed up) with several slaves,
![Page 18: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/18.jpg)
How we use Jenkins...
• We use labels to nickname a slaves (ff36, ff4, winxp).
• We use “Restrict where this project can be run” to configure a test job run.
![Page 19: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/19.jpg)
How we use Jenkins...
• For performance testing we use JMeter and JMeter plugin for Jenkins.
![Page 20: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/20.jpg)
How we use Jenkins...• We use “Configuration Matrix” for some jobs.
![Page 21: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/21.jpg)
How we use Jenkins...• We use “Discard old builds” to save disk space.
![Page 22: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/22.jpg)
How we use Jenkins...
• We use “Trigger build remotely” to run build from outside:
• ...and tools like curl/wget to trigger it:
curl --user user:password JENKINS/view/project/job/build?token=token
• We use Jenkins Remote Access API,
![Page 23: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/23.jpg)
How we use Jenkins...
00:00 to 08:00 - Automated/scheduled tests, nightly test,08:00 to 21:00 - work day, manual/scheduled test,21:00 to 00:00 - daily/weekly maintenance time,
• We try follow some time line guidelines:
![Page 24: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/24.jpg)
Maintenance problem
• We suffer with maintenance problem while managing so many VM – we use STAF for performing actions on all machines.
![Page 25: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/25.jpg)
Maintenance problem
• These machines lives and to prevent deterioration we enforce some maintenance tasks.
![Page 26: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/26.jpg)
Maintenance problem
• Daily tasks (21:00):• Restart,• Set screen resolution,• Resources.
Weekly tasks (21:00 Sunday): Cleaner, Defragmentation, Anti-Virus Scan, Windows updates.
![Page 27: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/27.jpg)
![Page 28: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/28.jpg)
Jenkins future
• Browsers auto updating,• Dynamic slave management,• Custom framework – leave Jenkins bottlenecks behind and
build your own,
![Page 29: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/29.jpg)
The answer
• Distributed testing environment on virtual machines,• Centralized automated test driving with Jenkins CI.
![Page 30: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/30.jpg)
Benefits of presented solution
• Easy way to extend test environment (just clone VM and voila),
• ...which let us to speed up tests by splitting and paralleling,• Easy way to restore corrupted system (revert snapshot, copy
VM disk file),
• HQ for managing tests with different tools (Selenium, Webdriver, Jmeter, Wapiti, Test Complete, AutoIt...),
• Built-in features and plugins (e.g.: SVN client),
![Page 31: Distributed Testing Environment](https://reader034.vdocuments.net/reader034/viewer/2022042714/556613c5d8b42a7d608b4824/html5/thumbnails/31.jpg)