seconf2015: advanced automated visual testing with selenium
TRANSCRIPT
![Page 1: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/1.jpg)
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
ADVANCED AUTOMATED VISUAL TESTING WITH
SELENIUM
![Page 2: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/2.jpg)
YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!
![Page 3: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/3.jpg)
![Page 4: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/4.jpg)
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
Q & A
![Page 5: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/5.jpg)
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
![Page 6: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/6.jpg)
A VISUAL BUG
![Page 7: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/7.jpg)
AND ANOTHER…
![Page 8: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/8.jpg)
AMAZON PRIME DAY – JULY 15th, 2015
![Page 9: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/9.jpg)
WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO COVER MANUALLY Web browsers Devices Operating systems Screen resolutions Responsive design L10n 3rd Party upgrades
![Page 10: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/10.jpg)
WHY SHOULD IT BE AUTOMATED?
![Page 11: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/11.jpg)
WHY SHOULD IT BE AUTOMATED?
NATIVE / HYBRID MOBILE APPS Harder to roll back changes
Can’t push daily
Updates take battery and data
Higher quality bar
![Page 12: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/12.jpg)
WHY SHOULD IT BE AUTOMATED?
Many are already doing it…
PhantomCSSFighting Layout Bugs
CSS Critc Wraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
EyesHuxley
FBSnapshotTestCase
GeminiSelenium Visual Diff
VisualCeption
Specter
Snap And Compare
kobold
![Page 13: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/13.jpg)
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
Q & A
![Page 14: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/14.jpg)
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 16: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/16.jpg)
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 17: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/17.jpg)
SCREENSHOTS
Full page? Regions? Frames? Device pixel ratio? Rotation? Viewport size? Page preparation? Page stabilization?
![Page 18: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/18.jpg)
QUICK FEEDBACK TOOLS
MOTIVATION
Get fast feedback on code
changes
Run tests locally in the
background
SETUP
Render screenshots with a
headless browser
PhatomJS, SlimerJS
Configuration file driven tests
CONS
Partial coverage
Chrome, IE?
Limited navigation
Tests stale browser versions
![Page 19: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/19.jpg)
VISUAL COVERAGE TOOLS
MOTIVATION
Verify that the app / site looks right
in all real execution environments
Test as many UI states as possible
Execute many tests in parallel
SETUP
Render screenshots using real
browsers on a variety of
operating systems and devices
in parallel
WebDriver, Grid
Code driven tests
WebDriver, DSLCONS
Requires test infrastructure
Implement and maintain test
code
![Page 20: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/20.jpg)
SELENIUM BASED TOOLS
CODE / SCRIPT
Needle (Python WD)
WebDriverCss (JS WebDriverIO)
Gemini (JS DSL)
Selenium Visual Diff (Java WD)
VisualCeption (PHPCodeCeption)
Pix-Diff (JS Protractor)
Shoov (JS WebDriverIO)
Vizregress (.NET WD)
Rspec Page Regression(Capibara)
Applitools Eyes (All WD + Appium)
Fighting Layout Bugs (JavaWebDriver)
Huxley (Record Playback)
CONFIGURATION
Viff
CSS Visual Test
GreenOnion
Wraith-Selenium
![Page 21: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/21.jpg)
NON-SELENIUM BASED TOOLS
CODE / SCRIPT
PhantomCSS (JS CasperJS)
Specter (JS DSL)
FBSnapshotTestCase (XCTest)
CONFIGURATION
Wraith
Dpdxt
Grunt PhotoBox
Grunt-Vigo
Snap And Compare
BackstopJS
CSSCritic
Kobold
DiffCop
SUCCSS
MANUAL
Eyes Express (Chrome)
![Page 22: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/22.jpg)
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 23: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/23.jpg)
![Page 24: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/24.jpg)
WHY NOT?
FALSE POSITIVES
![Page 25: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/25.jpg)
ANTI-ALIASING?
![Page 26: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/26.jpg)
ANTI ALIASING 1/2
![Page 27: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/27.jpg)
ANTI ALIASING 2/2
![Page 28: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/28.jpg)
BRIGHTNESS 1/2
![Page 29: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/29.jpg)
BRIGHTNESS 2/2
![Page 30: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/30.jpg)
TEXT POSIONING 1/2
![Page 31: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/31.jpg)
TEXT POSIONING 2/2
![Page 32: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/32.jpg)
IMAGE SCALING 1/2
![Page 33: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/33.jpg)
IMAGE SCALING 2/2
![Page 34: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/34.jpg)
ANDMORE…
1 pixel offsets in element positioning
Dynamic content
Moving elements
Images of different size
Performance
![Page 35: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/35.jpg)
![Page 36: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/36.jpg)
Image Comparison APIs
![Page 37: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/37.jpg)
ImageMagick A powerful command line tool for image processing.
APIs are available for most programming languages.
Fuzzing is used to eliminate slight color differences
An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
![Page 38: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/38.jpg)
The Javascript Engines Resemble.js
Pixel by pixel + error ratio + anti-aliasing
http://huddle.github.io/Resemble.js
Blink-DiffPixel by pixel + error ratio + anti-aliasing + perceptual color distance computation
https://github.com/yahoo/blink-diff
Looks-SamePixel by pixel + perceptual color distance computation + ignore Caret
https://github.com/gemini-testing/looks-same
![Page 39: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/39.jpg)
Applitools Eyes A specialized image processing stack designed to compare computer
generated UI images
Handles anti-aliasing, pixel offsets, color similarity, and image scaling
Dynamic and moving content
Compare images of different sizes
No error ratio configuration required
Validates full UI pages
Fast!
Supports layout matching
![Page 40: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/40.jpg)
DEMO
![Page 41: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/41.jpg)
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 42: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/42.jpg)
REPORT DIFFERENCESAs files on the file system (combined with source control)
![Page 43: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/43.jpg)
REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)
![Page 44: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/44.jpg)
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 45: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/45.jpg)
UPDATE THE BASELINE
Rename or commit individual image files
GUI (Gemini GUI)
![Page 46: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/46.jpg)
UPDATE THE BASELINE
Overwrite mode
Automatic maintenance (Applitools Eyes)
![Page 47: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/47.jpg)
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
Q & A
![Page 48: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/48.jpg)
WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA
Visual testing of frontend components by
frontend developers
![Page 49: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/49.jpg)
WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others
Full or partial validation of application
screens
A collaboration tool
![Page 50: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/50.jpg)
WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA
Validate your staging deployment using your
production deployment as a baseline
![Page 51: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/51.jpg)
WHERE DOES IT FIT?
• Monitoring
• Ops
• QA
No missing resources in production
No corruption due to 3rd party data
No corruption due to browser / OS upgrades
![Page 52: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/52.jpg)
QUESTIONS?
![Page 53: SeConf2015: Advanced Automated Visual Testing With Selenium](https://reader033.vdocuments.net/reader033/viewer/2022050613/589af0ac1a28ab3e288b4bd9/html5/thumbnails/53.jpg)
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
THANK YOU