fergal hynes softtest conference
TRANSCRIPT
Cloud Based Test Automation
Fergal Hyneswww.ExpertSoftwareTest.ie
Goals & Context
Frameworks
Cloud Intro
Case Studies
Cloud Based Auto Test Execution
Presentation Structure
Our Goals Today
• Understand (some of !) what's possible re. testing on the cloud
• Presenter to avoid product placement and remain agnostic w.r.t. open source \ commercial
• Allow you to decide if cloud based test automation can help you
Context
Planning and Control
Analysis and Design
Test Implementatio
n and Execution
Evaluating exit criteria and Reporting
Test Closure activities
Frameworks
Framework 1(a): Test Cases \ Test Suites
• Focus on the E2E Processes
• Analyse and prioritise the E2E Business Processes
• Assign each test to either Smoke, Core or Full Test Suite
• Separate the E2E Processes out by functional area
Auto the ST first!
• Write the tests to test the E2E Business Processes
Framework 1(b): Test Suites \ Test Cases
Resultant Product Risk Profile:
Framework 2: Test Automation Project \ Scripts
Test Cases
Business Process Layer
Action Layer
Abstracted Data
Framework 3(a): Test Environment \ SW & ..HW
VCS
Version Control Solution Build Management Solution
Execution Management Solution
Reporting Solution
Execution Machines
Software Software
Software Software
Hardware
So…..
Get your 3 Frameworks Right First :
(a)Test Suite Framework(b)Test Auto Framework(c)Test Environment Framework
Therefore at this point you will have:VCS
OFFICE
CLOUD
The Reality…
Cloud
The Top 3 Advantages of Cloud Based Test Execution
• Saves Test Analyst Time
• Inherently Supports Scalability
• Is Cheap (no capex and low running cost if managed properly)
Cloud in 30 secs…
• Virtual machines in a specific remote location & managed via your web based account
Simple!
• Connect to machines via (for example) RDP for windows and putty \ SSH for linux
• File transfer by (for e.g.) FTP
Clarification
The Test Machines
The Product
Machines
Test Automation on the Cloud 1
The Test Machines
• Security Group• Each orange box ~ 1 or >1 Machines• VM management (e.g. create \ destroy)
Test Automation on the Cloud 2
The Test Machines
The Product
Machines
• Product can be cloud or not cloud!• Trigger on event\schedule\on-demand
Case Study 1 – API & Mobile Apps
• Microsoft house• Locally Hosted API (i.e. Non Cloud based )• Event Driven and On-Demand Functional and Non-Functional Test Exe
Case Study 2 – API & Web
• Open source house• 100% Cloud based product under test• Functional and Non-Functional Test Exe On Demand
Special Note: Non-Functional Testing(Perf, Load, Stress)
• Regional load generation capability supported on cloud• SAAS
The Top 3 Pros \ Cons of Cloud Based Test Execution
Priority Pros Cons
1 Time. Backup, Reuse, Version Control via VM management Console
Your company and it’s security policies need to be flexible. Over to you…
2 100% Scalable As secure as you are - Choose your security level carefully – Broadcasting is rife
3 Cost – No capex and environment prototyping is essentially free. Billing reports are good.
Beware of Mobile Device Farms – Reliability and Cost(Can be discussed later if you want )
More Pros \ Cons of Cloud Based Test ExecutionPros Cons
Personal computers, internal and public internet networks can be abstracted from the test execution environment avoiding latency and local\network failures and false negatives
Domain knowledge inertia
Accessible anywhere - Walk away, interrogate results\progress from home
Cost can spiral if not controlled and monitored
Machine resources are configurable (almost) on the fly
Images (backups) become unsupported over time
Regionally distributed worldwide iOS not well supported
No physical machine maintenance overhead
Ability to share machines between cloud accounts
Provides many useful facilities, e.g. machine resource monitoringPlatinum services available for enterprise applications
Drives Continuous Integration best practices (local dev and remote execution)Not much you can't do...
Main Takeaways
3 Key Takeaways
Get your 3 Frameworks Right First :
(a) Test Suite (b) Test Auto(c) Test Env
Prototype your Test Auto
Solution on Free Machines on
your Cloud Account
If you do nothing else…..get the following right:
Idealfor
Perf, Load and Stress Testing
!
CONTACT US !
Expert Software TestDalys Lane, Main Street, Killorglin, Kerry, Ireland.V93 RT2P
(066)[email protected]
Q & A!