advanced visual test automation with selenium
TRANSCRIPT
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
ADVANCED VISUAL TEST AUTOMATION WITH
SELENIUM
YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
A VISUAL BUG
AND ANOTHER…
WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY Web browsers Devices Operating systems Screen resolutions Responsive designs L10n
WHY SHOULD IT BE AUTOMATED?
WHY SHOULD IT BE AUTOMATED?
NATIVE / HYBRID MOBILE APPS Harder to roll back changes
Can’t push daily
Updates take battery and data
Higher quality bar
WHY SHOULD IT BE AUTOMATED?
Many are already doing it (and sharing)…
PhantomCSSFighting Layout Bugs
CSS Critc Wraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
EyesHuxley
FBSnapshotTestCase
GeminiSelenium Visual Diff
VisualCeption
Specter
Snap And Compare
kobold
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
DRIVING AND CAPTURING
OTHER
• Wraith (URLs)
• PhantomCSS (CJS)
• dpdxt (URLs)
• Grunt PhotoBox (URLs)
• Snap And Compare
(URLs)
• Specter (JS)
• Gemini (JS)
• Grunt-Vigo (URLs)
• WebDriverCSS (JS)
• Fighting Layout Bugs
(Java)
• Wraith-Selenium (Ruby)
• Selenium Visual Diff
(Java)
• Needle (Python)
• VisualCeption (PHP)
• Huxley (RP)
• Gemini (JS)
• Vizregress (.NET)
• Visualci (Java)
• Viff (JS)
• Pix-Diff (JS)
• Applitools Eyes (All)
• CSS Critic (URLs)
• FBSnapshotTestCa
se (XCTest)
• Kobold (folders)
• Applitools Eyes
(All)
SCREENSHOTS
Real browsers? Full page? Frames? Regions? Viewport size? Page stabilization? Page preparation?
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
WHY NOT?
FALSE POSITIVES
ANTI ALIASING 1/2
ANTI ALIASING 2/2
BRIGHTNESS 1/2
BRIGHTNESS 2/2
SUB PIXEL SHIFTS 1/2
SUB PIXEL SHIFTS 2/2
IMAGE SCALING 1/2
IMAGE SCALING 2/2
ANDMORE…
1 pixel offsets in element positioning
Dynamic content
Moving elements
Images of different size
Performance
Image Comparison APIs
ImageMagick A powerful command line tool for image processing. APIs are available for most programming languages. Fuzzing is used to eliminate slight color differences An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
Resemble.js An image comparison library implemented in Javascript
Used by PhantomCSS and other tools.
Good antialiasing support
An error ratio is usually used to determine a match
http://huddle.github.io/Resemble.js/
Blink-Diff An image comparison library implemented in Javascript
Can be operated from the command line using node
Perceptual color distance computation
Anti-aliasing support
An error ratio is used to determine a match
https://github.com/yahoo/blink-diff/
Applitools Eyes A specialized image processing stack designed to compare
computer generated UI images
Anti-aliasing, pixel offsets, and image scaling with low false negative rate
Dynamic and moving content
Compare images of different sizes
No error ratio configuration required
Fast!
Layout matching
Available as a cloud service
DEMO
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
REPORT DIFFERENCESAs files on the file system (combined with source control)
REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
UPDATE THE BASELINE
Rename or commit individual image files
Overwrite mode
UPDATE THE BASELINE
GUI (Gemini GUI)
Automatic maintenance (Applitools Eyes)
AGENDA
Why automated visual testing?
Tools & Technology
Where does it fit?
WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA
WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others
WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA
WHERE DOES IT FIT?
• Monitoring
• Ops
• QA
QUESTIONS?
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
THANK YOU