salesforce selenium-saucelabs-webinar-april-2014
DESCRIPTION
Engineers David Louvton and Sagar Wanaselja will show you Salesforce's best practices and how they automate their Selenium tests at scale to ensure success. Topics covered will also include how to architect your test harness for success, Selenium design patterns and how to overcome test failures.TRANSCRIPT
![Page 1: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/1.jpg)
Selenium atSalesforce Scale
David LouvtonLead Engineer
Sagar WanaseljaLead Engineer
Corporate Pres – PowerPoint Template – 4x3 FY14.pptx
![Page 2: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/2.jpg)
![Page 3: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/3.jpg)
• About Salesforce engineering• Operating a selenium farm• Selenium design patterns• Overcoming test failures• Q&A
Agenda
![Page 4: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/4.jpg)
Salesforce Scale
• 3:2 ratio for developers : quality engineers• everyone is a software engineer; role varies• selenium committers on staff (Jim Evans, Luke Inman)
• Data centers with thousands of VMs for automation• multimillion dollar investment• using vmware (VShere), OpenStack and EC2
• 500 commits per day• ~120 software engineering teams
![Page 5: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/5.jpg)
Why Salesforce loves WebDriver
WebDriver is the only test framework in our current arsenal that truly mimics an end-user’s experience• We don’t mock anything• We test against a live app• We suffer the pain of latency• We test end to end on almost every action
WebDriver tests can catch things other levels of tests can’t• Multi-platform consistency• DOM consistency• Rendering issues• Catching inconsistency between
app layerssource: watirmelon.com © Alister Scott
![Page 6: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/6.jpg)
UI Testing Portfolio
• 50,000 unique Selenium tests• 7,500 unique WebDriver tests
• full “browser compatibility”: IE6-IE11, Firefox, Chrome, Safari• some can run in headless browser like Phantom JS
• 2,000 javascript unit tests• no DOM required• 10ms average execution time
source: watirmelon.com © Alister Scott
![Page 7: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/7.jpg)
1,000,000 browser tests per day
• Click here to add bullets
![Page 8: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/8.jpg)
The Selenium Infrastructure● Automatically creating, assigning, closing and reopening
bug reports for test failures and flapping tests .
● Test jobs must meet failure thresholds (99%+). Code lines are locked for teams under thresholds
● Support for all combinations of OS & Browser using portable VM Templates, creating deterministic and identical test environments
![Page 9: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/9.jpg)
![Page 10: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/10.jpg)
Issues of scale
• Assigning test failures to the responsible engineer• Non deterministic tests (flappers) and re-runs• Selenium jar upgrades are challenging• Browser upgrades are challenging
• IT and automation ops provide different versions• Large suites: adjust run frequency, deprecate old tests
![Page 11: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/11.jpg)
Salesforce UI Testing Pattern
• Click here to add bullets
![Page 12: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/12.jpg)
Proper Page Objects Encapsulate Selenium
• Make an API of your page for your test• No imports for “openqa” or “selenium.org” in your test• All references to driver & selenium in the page object• Fail fast - call the page object factory in test setup• No assertions in page objects (or any test utility)• Share page objects across engineering teams
• developer complete UI should have a page object• if you can’t make one it’s not a testable page• dependency challenges at scale
![Page 13: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/13.jpg)
Challenges for Mobile Platforms
• Adapting page objects to platform specific UI elements• Screen sizes and adaptive layouts
• “stage left” - switching into contained apps• lightweight page objects = widgets
• Avoid browser or platform specific code in actual tests• establish context elsewhere in utils or page objects
• Automated tests in emulators (99%+)• manual tests on devices before release• exception: testing video
![Page 15: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/15.jpg)
Problematic WebDriver Test
![Page 16: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/16.jpg)
Better WebDriver Test
![Page 17: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/17.jpg)
PageObject WebDriver Test
![Page 18: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/18.jpg)
Diagnosing Test Failures
• Screenshot and HTML capture at failure point• AssertionError or RuntimeException
• Differences between localhost and automation VM• Optimistic timing assumptions
• javascript, rendering, server response, etc.• Defensive “negative” testing• Getting to 100% tests passing
![Page 19: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/19.jpg)
Pro Tips
• Version tests and app code (and configuration) together• Use the Page Object model• Use the API to set up the test, not the UI• Anticipate Selenium and Browser releases• Allocate capacity to browsers most used by customers
• log the user agent on every page view
![Page 20: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/20.jpg)
More Pro Tips
• Avoid using xpath selectors• Complex DOM makes xpath evaluation much slower• Some xpaths are not properly evaluated in WebDriver
• When working with popups make sure you return focus to the main window
• For static/simple HTML pages consider the HtmlUnitDriver
![Page 21: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/21.jpg)
![Page 22: Salesforce selenium-saucelabs-webinar-april-2014](https://reader034.vdocuments.net/reader034/viewer/2022052504/554a12d5b4c9058c5d8b4c20/html5/thumbnails/22.jpg)
Mature Quality Strategies
• Selenium for dynamic security vulnerability analysis• Pixel perfect image diffs• Accessibility testing (tab completion)• Testing javascript with unit tests
• Selenium is for testing browser compatibility and DOM