automated tests - symbio · driverbindings for node.js. •with all the goodies that selion has,...
TRANSCRIPT
AUTOMATEDTESTS
How We Learned To Love Test Automation for Localization
2
In general, localization testing is manual – as text needs to be read, layout needs to be checked, and localization needs to be verified – all of which require a person to manually perform. But, automation is possible in localization testing too, enabling the localization testers to be more efficient.
We wrote this white paper to help you understand the importance of an integrated test automation framework, and to elaborate on the idea how we learned to love test automation, and how you can learn to love it too.
This white paper will guide you through the steps we took with our clientsto build UI Automation, to make localization testing more effective and efficient.
This transition – from manual to automation testing – reduced our client’s test cases by 60%, increased test coverage by 23%, andsaved time by 97.5%.
Executive Summary
• There was lot of technical debt in the code
• There was a need for lot of manual efforts to run automation
• Silk lacked a way to run tests in parallel
• Silk lacked data driven capabilities
• Silk lacked multi-browser support and backward compatibility with IE
• Developers couldn’t reproduce bugs found in Silk, since Silk environment was complex
• Developers did not want to use point & click tools, they wanted to code
• There were 80,000 NRT test cases and a 20-day NRT cycle.
• Each of our teams took 14-20 days to run NRT, with localization tests taking the longest time
We Also Wanted to Go Agile
• GUI tests are brittle, or was there a way to make it less brittle through design?
• The goal was to have manual testing team focus on new features using exploratory testing, and automate as much of the tests as possible
• Focus on Functional, GUI, and Content tests.
• Business facing + critique product
quadrant must be manual,
or are there ways to make
manual linguistic testing
more effective?3
What We Hated About Test Automation
Testing Pyramid Testing Matrix
Solid
Brittle Tests
Automated Unit Tests
Automated API Tests
AutomatedGUI Tests
Automated Integration Tests
Automated Component Tests
FitNess REST API
Cucumber
Selenium SeLion
xUnit TestNG
Manual Session
4
Based TestingTesting Tools
Before Agile transformation, most of thetesting logic was written in silk and allbusiness logic, integration logic, and GUI logicwere tested in silk at the GUI layer. Bydefault, GUI tests are brittle, since any change
in the UI will break the tests. SeLionenforces the best practice of separating theUI elements and the UI logic, which willincrease the robustness of the GUI tests.
Agile Testing Pyramid
Symbio Automation Framework
TEP(Test Execution
Platform)
Dashboard
Framework
Jenkins
o Webo Mobile
Test ScriptsRepository
Product
JIRA
Build Deploy
Metrics &Reports Screenshots
LQA Review
Defect reporting
5
Symbio Automation Dashboard
Web
Mobile
Desktop
6
Start -> Search -> Shop -> Sort -> Select -> Buy -> Configure -> Cart
All test flow pages are captured as screenshots
Screenshot library
7
Review and Bug reporting
• Bug review window gives clear view of details with Markup highlight feature
• Bugs can be reported from Dashboard to JIRA, filling most of the data automatically
8
Selenuim is a great open source solution, however:
• All tests were flow-based vs page-based
• Selenium only supports web based testing
• Selenium didn’t support multi devices
• Selenium didn’t support multi lingual
This took us part of the way there. Luckily, Selenium gave us more options:
• Much easier, cheaper to add new core functionality
• The large Selenium community provided us with drivers to drive native apps on iOS and Android
• We could easily add whatever we were missing, then roll it out to the community for their use, in turn allowing them to add on, and improve it
Browser
Phone Tablet TV
Desktop
Multi-device
Web
Mobile
Desktop
Multi-platform
zh j a ko th h i
ar ms p t ru v i
t r p l cs f i f a
sv da n l i d ro
Multi-lingual
9
UI Test Automation with SeLion
Solution: Extend Selenium Into “SeLion”
We added what was missing, added more features, and released it into the open source community. This gave us:
• A separation of concerns:
• YAML to handle page objects + multilingual support
• Test cases focused on test logic
• YAML provided the ability to fix page changes in one place instead of having to fix in every script
• Very readable test scripts
• Testing grid, Jenkins + Selenium grid to run tests
• Testing environment component to provide data driven tests
• Added Android + iOS drivers that plugged into Selenium grid, providing the multiplatform support
10
baseClass: “com.paypal.selion.testcomponents.BasicPageImpl” pageTitle:US: “Official Apple Store” defaultLocale: “US” elements:
officialAppleStoreAppleStoreLink: locators:US: “//div[@id=’page’]/div[2]/div/div[1]/h1/a/span”
ZH: “//div[@id=’page’]/div[2]/div/div[1]/h1/a/span” searchSubmitButton:locators:
US: “//button[@class=’gh-search-submit gh-search-magnify’]”ZH: “//button[@class=’gh-search-submit gh-search-magnify’]” siteSearchQueryTextField:
locators:US: “//input[@id=’site-search-query’]” ZH: “//input[@id=’site-search-query’]”
pageValidators: [ officialAppleStoreAppleStoreLink]
How Did We Do It?
Example Test Case Script
YAML: Page Objects + Locale
Test Code
Test NG
SuiteTests
Page Object
Repository
Preprocessor
Yaml Yaml Yaml
UI in multiple
languages
- Web pages- Mobile screens- Desktop screens
Template- Page Objects- Multi-lingual support
Tests- Page flows- End-2-End test
11
SeLion Design
• Nemo combines mocha.js, the confit configuration engine, plugin architecture, and flexible parallel options to the selenium-web driver bindings for node.js.
• With all the goodies that SeLion has, Nemo is smaller and more Node.js friendly
• Allows automation testing to be part of your Node.js app’s development life cycle.
• For Node.js application
12
UI Test Automation with Nemo.JS
Why Nemo?
Nemo.JS is a open source
NPM package.
We will create functions that
fit test projects, and released
it as Nemo plugins. We will be able
to:
• Aseparation of concerns:
• Data Provider to handle
test case level and locale
default data to feed the
test.
• Test suites focused ontest
logic
• Page Object to handle
page locaters and page
level common steps.
• Less maintenance effort.
• Testing on cloud
• Test run in parallel in data
driven.
13
How Did We Do It?
Example Test Case Script
JSON: page locators, localized data
Validation Layer(Screenshots + Check Points)
Reporting Layer
14
Nemo Work Flow
JENKINSTest Suites
N
e
m
o
A
P
P
L
I
C
A
T
I
O
N
Data Provider
Case Data
Locale Data
Page Object
Test Reports
NPM
Screenshots Stack TracesEmail/Slack Notifications
•Cucumber uses Gherkin to define test cases, which is a language designed to be human readable, and easily accessible to non-technical people
•Dev and QA can help to write automation scripts easily
•Multi-languages, Multi-platforms, Multi-devices support as well as SeLion platform
UI Automation with Cucumber/Appium
Write test scripts in human language
Browser
Phone Tablet TV
Desktop
Web
Mobile
Desktop
Native/Hybrid/Webview Apps supported
15
Write Test Scripts in Human Language
We created functions and test steps that fitted test projects, and implemented to framework.
• Supported data driven tests
• Test steps were defined and centralized
• Human readable scripts
• Easy onboarding
• Supported Native/Hybrid/Webview Apps, and
• Supported multiple OS/system on devices farm
16
How Did We Do It?
Example Test Case Script
Steps pre-defined
Test cases
Testcases
Testcases
Tests
Appium/UIAutomator2/Frank
Multiple
languages cases
- Native case- WebView test - Hybrid
Run tests on multiple devices/OS
17
Cucumber/Appium Framework Design
Device Farm/Cloud
APP/Browser
JENKINS
Support Libraries
Predefined Steps
• Running tests in parallel greatly decreased total run time
• Full automated NRT cycle down to 2 days, shortening the cycle
• For localization tests, 20 days went down to 4 hours
• 60% reduction of test cases, one team went from 3000 to 800 cases
• Product and testing teams were able to develop test cases
• Every release, 90% of the tests were run using automation
• New tests per release focused on new features, and were SBTM based
• Automation cost was lower than manual testing cost, yet the coverage was much higher. The test cases were cumulative over time
$
$
$
$
$
$
$20 days 4 hours
Time Saving by 97.5%
3000 800
Test Cases Reduction by60% Cost Coverage
80%
98%
18
How We Learned To Love Test Automation?
Results
• Symbio UI Automation solution emerged out of the need for a better test automation framework
• Symbio UI Automation solution can help you to drastically improve your processes
• Symbio UI Automation solution will scale your testing and QA, and you will be able to test more and test better in a shorter period of time
• Symbio UI Automation solution will help your testing teams to be more efficient
• Symbio UI Automation solution will increase the test coverage, and will minimize test cases
• SeLion will also drastically lower the cost of testing
Multi-device
en f rdei tes f i
Multi-lingual
Manual Testing
Automated Testing
Multi-platform
Symbio Can Help You to Implement UI Automation
How can we help you?
SYMBIOAutomation
19
Symbio is the global leader in innovation, co-creation and seamless augmentation. Weprovide insight and strategy, design and research, productization, softwaredevelopment, quality assurance, globalization, R&D operations, systems managementand support operations for over 400 customers worldwide.
Visit us online at www.symbio.com or contact us directly at [email protected]
Contact Us