framework for automation testing practice sharing
DESCRIPTION
This is the presentation that Mr. Duong Phan and Mr. Vu Duong shared at KMS's public seminar on Saturday, June 15, 2013.TRANSCRIPT
FRAMEWORK FOR AUTOMATION TESTING – PRACTICE SHARING
June 2013KMS Technology: http://
kms-technology.com
AGENDA
Duong Phan – QA Architect Introduction Challenges and Solutions KMS Test Automation Framework Advanced Features
Vu Duong – Engineering Manager ROI and Metrics
2
KMS TECHNOLOGY OVERVIEW
3
US Company/Passionate Global Workforce• 400 Resources & Growing in Vietnam and the US
• 160 Testers ~ 50% Workforce• Proven Leadership TeamWorld Class Infrastructure• Built for ISO 27001, Planned Certification in 2013Best-in-Class SDLC Practices• CMMI and Agile focus• QASymphony - Commercial Agile Testing Solutions
recognized by Forrester with over 4,500 users
Best Clients – Raving Fans• 100% Referenceable and Ecstatic• 100% in long-term dedicated team
A CASE STUDY
4
About the Clientis a leader in eCommerce software and solutions, Provide multifaceted eCommerce platform, reliable 24/7/365 service, scalable IT infrastructure, integration with third party, and custom-built features, Serve over 160 retail sites
Technology: Platform: Windows XP, Windows 7,
Linux, OSX, iOS and Android. Technology: Java, .NET/C#, Oracle Testing Tools: JIRA, Testlink, KMS
Automation Framework
Achievements KMS is responsible for automation of
client Core Platform Product and also for distribution platforms
Reusable Automation Framework can stand independent with KMS run-time plug-in (support web testing) although can adapt other tools (Selenium)
Framework support multiple OS: Windows, Linux and OSX, iOS and Android
Execute test daily, Automation script is able to find a lot of defects
Confidential
Automation Goals Reduce cost of development and quality
assurance Continuous platform upgrade to provide
new features and patch releases Testing is performed on core platform
and more than 200 of distributions Testing on many browsers (IE, Firefox,
Chrome, Safari)
CHALLENGES AND OUTCOMES
Project Specification Multiple Instance Application Types
Flexible Components
Regular Sanity Test
Automation must find bugs Customer Satisfaction
Cost Saving
Product Quality Improvement
Bug Escape Reduction
5
KEY FEATURES…
Framework using Excel as Scripting tool• Easy to use and learn• Easy to apply Data Driven and Keyword Driven
ideasPrototype plug-in• Able to develop wrapper plug-in with several
tools• One script can run under any plug-ins
Log System• General report and log types• History report
6
KMS TEST AUTOMATION FRAMEWORK
7
EXECUTION MODEL AND PLUG-INS
8
SCRIPTING
9
© 2013 KMS Technology
ADVANCED FEATURES
SMART OBJECT RECOGNITION
Multiple GUI instances 1 GUI for many kinds of instance objects, apply for
multi sites product Example: 1 gui object txt_Email may be detect for 4 sites
(gmail, yahoo mail, facebook, Linkedin)
11
KEYWORD DRIVEN - FLEXIBLE REUSABILITY
12
A library can use other libraries Test case can use any library Test case can call other test cases Data-driven on test case calls test cases
DATA DRIVEN - FLEXIBLE CONFIGURATION
13
Multiple Datasets
DATA DRIVEN - FLEXIBLE CONFIGURATION
14
Able to switch runtime engines Sample case
• Configure steps in IE• Execute some steps in Firefox• Verify result steps on Chrome
LOG AND REPORT
15
LOG AND REPORT
History report (excel, html and email) Helpful to follow bug fixing HTML format can be opened anywhere Email (HTML format) is portable
Various type of reports/logs Statistic of result Detail report/log HTML / Excel format report
Reminders Send email notification Define groups to send notification
16
LOG AND REPORT
Follow Drill-down ideaHistory Report Step report Detail report
17
© 2013 KMS Technology
ROI & METRICS
VU DUONG
ROI – WHAT?
19
ROI – WHAT?
20
ROI – WHY?
21
ROI – WHY?
22
ROI – HOW?
23
ROI ANALYSIS OVERVIEW
24
To calculate ROI, the benefit (return) of an investment is divided by the cost of the investment; the result is expressed as a Percentage or a Ratio.
ROI – BEST PRACTICE
25
ROI (Effort) ROI (Cost)
ROI - EFFORT
26
ESV (Effort saved from using automation)ROI (effort) = ------- = (-------------------------------------------------)
ESP (Effort involved in automation)
ESV = Effort saved as a result of test execution using automation (in hours)
ESP = Effort involved in automation like: Framework Development, Automation Scripts Development, Automated Script Execution, Automation Scripts Maintenance, Automation Training …
ROI ANALYSIS – EFFORT
27
Parameters:
ROI ANALYSIS – EFFORT
28
Pay-Back Period (Effort)
ROI ANALYSIS – COST
29
Parameters:
ROI ANALYSIS – COST
30
Pay-Back Period (Cost)
METRICS
31
METRICS – EFFORT/COST
32
Metrics for Effort/Cost Running Time Per Month Critical/High Defect Per Component Critical/High Defect Per Month
METRICS – QUALITY
33
Metrics for Quality Defect Escape % Automation Coverage Per Sites/Features Defect Found by Automation Defect Rate (Compare with Manual) …….
© 2013 KMS Technology
THANK YOU
34