end-to-end test automation for both horizontal and vertical scale
TRANSCRIPT
![Page 1: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/1.jpg)
End to End Test Automation for Both Horizontal and Vertical Scale
Erdem YILDIRIM
![Page 2: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/2.jpg)
2000 – Software Developer
2006 – Discovered Test Automation &
Decided to be Test Engineer
2010 – Test Lead at Defence Technologies and Engineering (STM)
2013 – Test Lead at İnnova(Telecommunication)
About Me
Linkedin: https://tr.linkedin.com/in/erdem-yildirim-0b5ab466
![Page 3: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/3.jpg)
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales)
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
Agenda
![Page 4: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/4.jpg)
Agenda
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales)
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
![Page 5: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/5.jpg)
Machines are great for runningrepetetive / straightforward scenariosthat don’t require any intelligence orexperience.
Test Automation
![Page 6: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/6.jpg)
End to end testing is verifying that all units of anapplication and all subsystems of a system interact asexpected with each other that the system as a wholeworks as intended.
E2E Test Automation
![Page 7: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/7.jpg)
Ideal Software Test Automation Pyramid
UI
Tests
Integration
Tests
Unit Tests
Manual
(Scripted)
Test
![Page 8: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/8.jpg)
It is OK that automating whole Unit, Inregration, UI levels is great.
Can we improve it?
![Page 9: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/9.jpg)
More Ideal One
CI –
Co
ntin
uo
us In
teg
ratio
n
Exploratory
Test
![Page 10: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/10.jpg)
An Extra Level: Test Data Preparation
How to prepare test dataset and bringdatabase to known state for test cases before
every test run?
Is Teardown Method OK?
![Page 11: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/11.jpg)
Test Data Preparation Automation
Teardown Method with Automation
Setup
Exercise
Verify
Teardown
SUT
Install
Create
Return
ValuesIndirect
Inputs
Record
Mock
Object
![Page 12: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/12.jpg)
Advantages of AutomatingTest Data Preparation Phase
Faster coding,
Faster test executions,
Clean test environment,
Testing with real and more complex test data,
Loose coupling between data and code layers
Facilitates testing for multiple customers (each with a DB)
![Page 13: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/13.jpg)
So, we were able to automate theentire vertical E2E testing:
Unit, Inregration, UI +Test Data Preparation levels
Can we improve it further?
![Page 14: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/14.jpg)
E2E Test AutomationHorizontal & Vertical Scale
UI
Tests
Integration
Tests
Unit Tests
Test Data Preparation
CI –
Co
ntin
uo
us In
teg
ratio
nHorizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay
Vertical E2E
Implementing
tests on all level
of the pyramid
![Page 15: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/15.jpg)
E2E Test AutomationHorizontal & Vertical Scale
UI
Tests
Integration
Tests
Unit Tests
Test Data Preparation
CI –
Co
ntin
uo
us In
teg
ratio
nHorizontal E2E (UI & Integration Layer) Login / Add to Chart / Checkout / Pay
Vertical E2E
Implementing
tests on all level
of the pyramid
![Page 16: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/16.jpg)
Prerequisites of Horizontal E2E Tests
All system and subsystems’ test environments shouldbe ready.
If they are not ready, mocking work should be performed.
![Page 17: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/17.jpg)
Prerequisites of Vertical E2E Tests
Having a supporting test/development strategy likeTDD, BDD, Continuous Testing
It should be driven by all stakeholders (dev, tester, pm, tl) in Project.
![Page 18: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/18.jpg)
Advantages of Vertical andHorizontal E2E Tests
Advantages of Horizontal E2E Tests:
High business logic coverage,
Testing from the user perspective,
Prevents production incidents,
Gives the confidence that everything is OK
Advantages of Vertical E2E Tests:
High code coverage,
Fast execution,
More determined and focussed tests,
Useful for especially safety critical softwares
![Page 19: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/19.jpg)
UI
Tests
Integration
Tests
Unit Tests
Test
Type
Low
Medium
High
Business Logic
Coverage
Code
Coverage
Low
Medium
High
Execution
Time/Costs
Short/
Low
Medium
Long / High
![Page 20: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/20.jpg)
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
Agenda
![Page 21: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/21.jpg)
E2E Test Strategy
Which projects’ tests strategy should be designed as E2E?
Can all tests be written as E2E ?
![Page 22: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/22.jpg)
Test Strategy
I Put That Sh t on Everthing*
![Page 23: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/23.jpg)
Alternate Test Automation Approach
Continuous Testing
To develop software within shorter delivery cycles; Agile, DevOpsand Continuous Delivery approaches are on the forefront.
Continuous Delivery supports Continuous Testing which is a testing strategy that consists of a large number of automated unit and acceptance tests but a small number of automated end-to-
end tests.
Continuous Testing claims that E2E tests are inefficient and theyshould be limited.
It is a means of using Vertical E2E Test Automation effectively but avoiding Horizontal E2E Test Automation
![Page 24: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/24.jpg)
Which Test Automation Strategyis the Best?
There isn’t one absolute strategy that is valid forevery SUT
![Page 25: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/25.jpg)
Criteria / PrinciplesTo Form Test Strategy
Test Types / Methodologies that your staff is strong,
Coverage Strategy: Business Logic Coverage or CodeCoverage
Execution Time Priority
Cost
These Criteria should be considered with the dynamics and needsof Project, Staff, Company, Customer.
![Page 26: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/26.jpg)
It is not an obligation to select & implement a strategystrictly.
You can mix the approaches and form your ownstrategy based on your needs.
It’s about finding the right balance.
Criteria / PrinciplesTo Form Test Strategy
![Page 27: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/27.jpg)
Key Factors To Form & Implement a Successful Test Automation Strategy
Manage test releases effectively
Establish test measurement strategy & metrics
Establish test management strategy
Choose your test delivery model
Determine & implement test tools strategy
Assess your test & process maturity
![Page 28: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/28.jpg)
Feasibility Analysis
Evaluation of Tools / Methodologies
Forming Automation Strategy (Plan, Design)
Implementation
Test & Maintain
Key Factors To Form & Implement a Successful Test Automation Strategy
![Page 29: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/29.jpg)
Based on Delivery Strategy
Automated test executions should be planned andclear CI roadmap should be outlined based on
needs.
![Page 30: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/30.jpg)
An ExampleAutomated Test Execution/CI Strategy
Based on Below Criteria
Delivery Velocity; should be fast so
smoke Unit test first, smoke UI test second,
full test sets last-after deploy in order.
Test Risk Level: Medium; smoke tests for all test types runsbefore deployment. But it is risky to run full sets after
deployment, since bug could stay undetected.
![Page 31: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/31.jpg)
Dev TeamVersionControl
Build & UnitTests
UI Test Automation
(Smoke)Deploy (Prod)
UI Test Automation
(Smoke)
UI Test Automation
(Full Set)Monitoring
Dev CI Pre-Production Production
Check inTrigger
Trigger
Approval
Approval
Approval
Approval
Feedback
Feedback
Feedback
Feedback
FeedbackAutomated Test RunsAfter Nightly Build [1]• Unit Test
Automation• Web Service Test
Automation• UI Test Automation
(Full Set)• Cross Browser &
Platform Tests
Automated Test RunsAfter Weekly Build [2]• Automated Non
Functional Tests(Performance & Load Tests)
• Static Code QualityAnalysis
Feedback
Feedback
Feedback
![Page 32: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/32.jpg)
Communication
Collaboration
Automation
Living Documentation
Drives whole project team
Healthy test basis / requirements
Replaces scripted test with automation scripts
To Strengthen the Automation Process:BDD – Behaviour Driven Development
![Page 33: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/33.jpg)
Benefitsof E2E Test Automation
Velocity
Stability
Efficiency
Money
![Page 34: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/34.jpg)
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
Agenda
![Page 35: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/35.jpg)
E2E Test AutomationExperiences
7 active projects’ tests that are being automated
as E2E
![Page 36: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/36.jpg)
ROI
Project Name # of E2E Test
Scenarios
Manual Test
Execution Time
Automation
Execution Time
Ratio of Saving
Legalite 235 6 Days (48 hrs) 1,5 Hour 32 times
HOPE 85 2 Days (16 hrs) 1 Hour 16 times
PROTTON 1200 5 Days (40 hrs) 2 Hours 20 times
TTSis 2000 8 Days (64 hrs) 3 Hours 21 times
Planor 110 3 Days (24 hrs) 2 Hours 12 times
TTS 120 2 Days (16 hrs) 1 Hour 16 times
ATS 30 1 Day (8 hrs) 1 Hour 8 times
![Page 37: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/37.jpg)
Challenges / Problems FacedIn the Process
TechnicalStaff Training and MotivationChoice of Tool and Methodology
PlanningDoing Feasibility Analysis Unproperly.Wrongly designed test automation strategy.
ManagerialDoing incrementally and showing the work & results.
![Page 38: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/38.jpg)
Test Team (Agile) Organizational Structure
PM
PM
PM
PM
Team
Team
Team
Team
T L
Test Engineers TL: Test Lead
PM: Project Manager
Me!
![Page 39: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/39.jpg)
E2E Test Automation
(Definition, Improvements, Vertical & Horizontal Scale, Prerequisites, Advantages of Each Test Levels & Both Scales
Test Automation Strategy
(Approaches, How to Form, Criteria/Principles, Key Factors For a Successful Strategy, Prerequisites, Gains)
E2E Test Automation Experiences
(ROI, The Problems Faced In the Process)
Need for Both Manual and Automated Testing
(Is Manual Tests Still Needed? Cross Tests)
Agenda
![Page 40: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/40.jpg)
Testing cost: testing accounts for roughly 26% of IT budgets
*According to a 2014 industrial survey of 1543 executives from 25
countries
Lack of testing is even more costly; the global cost of locating
and removing bugs from software has risen to $312 billion
annually and it makes up half of the development time of the
average project.
* A 2013 study by the Cambridge University
Testing Cost vs Lack of Testing Cost
![Page 41: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/41.jpg)
Trying to find defects on System UnderTest using human intelligence andexperience.
Manual testing
![Page 42: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/42.jpg)
Is Manual Test Still Needed?
Mostly: Yes!
Avoid Scripted Testing;
Automate it instead
Concantrate on Exploratory Testing;
Machines are great for running straigtforward
tests
Humans are great finding new defects
![Page 43: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/43.jpg)
Support
your team
by moving testers
across teams
and a little
bit of fun
![Page 44: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/44.jpg)
At the End of Bug Fest
Team Facility;Playing Tabu, Surprise Gifts,
Coffee, Free Time
![Page 46: End-to-End Test Automation for Both Horizontal and Vertical Scale](https://reader035.vdocuments.net/reader035/viewer/2022062306/58ce5b241a28ab2f268b49f1/html5/thumbnails/46.jpg)
• V. Garousi and M. V. Mäntylä, "When and what to automate in software testing? A multi-vocal literature review," Information and Software Technology, vol. 76, pp. 92-117, 2016.
• Testistanbul 2016 - Keynote: "Why Automated Verification Matters" by Kristian Karl
• V. Garousi and D. Pfahl, "When to Automate Software Testing? A Decision-Support Approach Based on Process Simulation," Wiley Journal of Software: Evolution and Process, vol. 28, pp. 272-285, 2016.
Effective Software Testing: 50 Ways to Improve Your Software Testing, Addison-Wesley Longman Publishing Co., Inc. Boston, MA, USA ©2002
End-to-End Integration Testing Design, Computer Software and Applications Conference, 2001. COMPSAC 2001. 25th Annual International
Distributed End-to-End Testing Management, Enterprise Distributed Object Computing Conference, 2001
HP Capgemini. Sogetti, World quality report 2014-2015. www.sogeti.com/solutions/testing/wqr/, Last accessed: Sept. 2015.[2] T. Britton, L. Jeng, G. Carver, P. Cheak, and T. Katzenellenbogen, “Reversibledebugging software,” University of Cambridge, Judge Business School, TehnicalReport, 2013
How Google Tests Software, James A. Whittaker
Behavior Driven Development Webapps mit Groovy Spock und Geb
http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/
http://techarcis.com/services/end-to-end-test-strategy/
References