automated visual testing that does not suck
TRANSCRIPT
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
AUTOMATED VISUAL TESTING THAT DOES
NOT SUCK
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY Web browsers
Operating systems
Screen resolutions
Responsive designs
L10n
WHY SHOULD IT BE AUTOMATED?
Many are already doing it (and sharing)…
PhantomCSS Fighting Layout Bugs
CSSCritique
Wraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
EyesHuxley
FBSnapshotTestCase
GeminiSelenium Visual Diff
VisualCeption
Specter
Snap And Compare
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
DRIVING AND CAPTURING
OTHER
PhantomCSS (CasperJS)
Fighting Layout Bugs (Java)CSSCritique (URLs)Wraith (URLs)
Needle (Python)Grunt PhotoBox (URLs)
WebdriverCSS (JS)
Eyes (All)
Huxley (RP)
FBSnapshotTestCaseSelenium Visual Diff (Java)
VisualCeption (PHP)
Specter (JS)
Wraith-Selenium (Ruby)
Snap And Compare (URLs)
Eyes (All)
Gemini (JS)
dpdxt (URLs)
Gemini (JS)
SCREENSHOTS
Real browsers? Full page? Frames? Regions? Viewport size? Page stabilization? Page preparation?
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
ANDMORE…
1 pixel offsets in element positioning
Dynamic content
Moving elements
Images of different size
ImageMagick A powerful command line tool for image processing. APIs are available for most programming languages. Fuzzing is used to eliminate slight color differences An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
Resemble.js An image comparison tool implemented in Javascript
Used by PhantomCSS and other tools.
Good de-antialiasing
An error ratio is usually used to determine a match
http://huddle.github.io/Resemble.js/
Applitools Eyes A specialized image processing stack designed to compare
computer generated UI images
Anti-aliasing
Partial and full pixel offsets
Images of different size
Dynamic content
Moving elements
Layout matching
Available as a cloud service
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
THANK YOU