test automation - business v2

8/6/2019 Test Automation - Business v2

http://slidepdf.com/reader/full/test-automation-business-v2 1/36

Test AutomationTest Automation

Let¶s talk businessLet¶s talk business

Igor GershovichIgor GershovichConnected Testing, Inc.Connected Testing, Inc.www.connectedtesting.comwww.connectedtesting.comigershovich@[email protected]



Over view

Test Automation Myths

Return on Inv

estment (RO

I) Frameworks



Test Automation Myths



Myth #1 - Test Automation is

simple, that every tester can do it This myth is promoted by the tool sales people.

They are trying to promote the following testautomation process: Record the script Enhance the script by adding functions and data

driving Run the scripts Report results

Under the influence of this myth the QAmanager can proudly report: All our testers aredeveloping test automation.



Reality - Test automation is a

software development task Automation should be designed, developed and

tested

You need to have some kind of aprogramming background to implement testautomation. Test Automation is not as complexas C++/C#/Java development.

Test automation standards should be developed

Automated test components are assets thatshould be treated like application source code



Myth #2 ± Commercial test tools

are expensive Under the influence of this myth some

companies, especially the small ones:

Try to develop their own test automation tools

Use scripting languages like Perl and Ruby

Use shareware test tools

Do not consider test automation at all



Reality ± Commercial tools are

cheap Per seat license for the most expensive automation tool

is $8K

This tool will be used for 5 years.

Maintenance/Support fees are 20% of tool cost or $1,800 per year

The cost of this tool is $8K/5+$1,800 = $3,100 per year

The automation developer cost with overhead is $100K

per year The cost of this tool is just 3% of the person who uses it,

but productivity gain can be very significant



Commercial Tool Benefits: Customer support. Many of the open source tools come and go with

little to no support Most commercial tools are constantly being updated as technologies

change Most commercial tools usually have more functionality (QTP can test

various GUI applications: Web, .Net, Java, VB, C/C++,PowerBuilder, etc. vs. WATIR ± Web only)

Commercial tools usually have a large community of users, whichtranslates into better availability of qualified resources

Commercial tools require less advanced programming More test automation frameworks are available for commercial tools Commercial tools are integrated with Test Management tools which

makes reporting and execution much simpler.



Over 300 Test Tools are available

(http://www.softwareqatest.com) Load/Performance tools ± 54

Web Functional/Regression ± 60

Java Test tools - 48

Other Web tools ± 76

Which tool is right for you?



Absolutely Unscientific research:

Employer demand by tool expertise(May, 2008)



Dice search results across US (30 days)

Tool Search string Matches

QTP Qtp OR quicktest OR "quick test³ 613

Functional

Tester

"rational robot" OR "functional

tester´

118

SilkTest silktest OR "silk test´ 86

TestComplete testcomplete OR "test complete´ 26

TestPartner testpartner OR "test partner³ 19

VSTE VSTE AND test 45

WATIR watir 26

Selenium Selenium 108



De a e l

HP P

60

Open ou ce

I

2c osoft

4

uto ated

est o p ete

2

o pu a eestPar tner

2

orland

lk est

8

Iunct onal

ester

12

Open ource

elen u

10P

Funct onal ester

Silk est

est o plete

estPar tner

VSTE

WATI

Seleniu



Excerpt from IDC Report:

³Worldwide AutomatedSoftware Quality 2007-2011Forecast and 2006 Vendor

Shares«´



Think of yourself like a business Facts:

Average IT job tenure is about 2 years Competitive intensity is rising

What you should do: Invest your time/money in learning mainstream

automation Create a niche for yourself - add Configuration

Management, Performance Testing or ProjectManagement to your skillset Market yourself



Return On Investment (ROI)



Classic ROI Calculation

ROI = BENEFIT/COST

Automation Cost = Price Of HW + Price of SW + Development Cost +Maintenance Cost + Execution Cost

Manual Testing Cost = Development Cost + Maintenance Cost +Execution Cost

ROI = (Manual Testing Cost - Automation Cost)/Automation Cost

Looks right, Doesn¶t it?



Problems with Classic ROI Calculation

You can¶t compare Automated Testingand Manual Testing. They are not the

same and they provide differentinformation about the AUT.

You can¶t compare cost of multipleexecution of automated tests vs. manual

tests. You would never dream of executing that many test cases manually.



Automation Real RO

I ROI value IS NOT the value of Automationvs. Cost of executing these tests manually

Automation ROI value IS the benefit of this type of testing, and it can be:

Reducing Time to Market

Increased Test Efficiency (Productivity)

Increased Test Effectiveness



Reduced Time to Market Can get a greater market share

Makes people available to work onother projects

Higher margins, if no competitive

products are currently available



Productiv

ity and Effectiv

eness More testing gets done faster, increasing the odds of

finding defects

Defects found early have better chances of being fixed Manual Testers can concentrate on clever ways to

finding defects, instead of typing test inputs and verifyoutput.

About 7% of bug fixes create new bugs, sometimes inalready tested parts of the system. With automation youcan rerun tests for those modules. This almost never happens when testing is done manually.



RO

I summary Each project requires different types of

automation - there is often no easy

formula available to calculate ROI

Performing ROI calculation can help todetermine upfront what type of

automation, what level of skills, what toolswill be required.



Automation Frameworks



Test Automation Framework A Test Automation Framework is a set

of assumptions, concepts and tools that

provide support for Automated SoftwareTesting.

A correctly implemented Test AutomationFramework can further improve ROI byreducing the development andmaintenance costs.



Types of Test Automation

Frameworks Modular

Data-Driven

Keyword±Driven

Model-Based



Modular framework The Modular framework is the natural

progression derived from Record-and-Playback

The modular framework seeks to minimizerepetition of code by grouping similar actionsinto ³modules´ (e.g.: login)

Test Data is in a script or in an internal table

This is what tool sales people are promoting.



Data-Driven/Keyword Driven

These frameworks are similar in that thedata is separated from the test script

The script is just a "driver" or deli

verymechanism for the data.

The difference:

In keyword-driven testing, the navigation dataand test data are contained in the data source

In data-driven testing, only test data iscontained in the data source.



Model-Based testing

Model-based testing is software testingin which test cases are derived in whole or

in part from a model that describes some(usually functional) aspects of the system.

Model-based testing for complex software

systems is still an evolving field.



We¶ll be concentrating onKeyword-Driven testing since it

is the most beneficial frameworkfor large-scale test automation.



Advantages of Keyword-Driven

Test automation This Framework addresses the most common problem

with test automation: Automation Engineers do not have domain knowledge

and the End Users (Subject Matter Experts/TestEngineers) usually do not have automation expertise. When properly implemented and maintained, it presents

a superior ROI because each business event isdesigned, automated and maintained as a discreteentity.

Keywords can then be used to design test cases, but thedesign and automation overhead for the keyword hasalready been paid.



Advantages of Keyword-Driven

Test automation Reduced the cost and time spent maintaining

and updating tests The modular structure of keyword-driven testing

means that new tests can easily be created frompre-existing modules

The test team is capable of entirely automatingtests, even without programming knowledge

Can be easily modified to use with different testtool Reusability across different projects



Classic Keyword-Driven

ExampleObject Action Data

Textfield (username) Enter Text <username>



Recent Success with Keyword-Driven

approach Estimate using Modular Framework: 1 year

Estimate with Keyword-Driven Framework: 6 months

Project was completed in 4 months by 1 person, 2 monthahead of schedule.

This was the first project for the automation developer utilizing the Keyword-Driven Framework.

170 components were developed (1 component per Webpage).

109 Regression Test cases were automated.



Example of Test Data for Keyword-Driven testautomation developed by Connected Testing Inc.

for TestComplete and QTP



Resources

³Everything you want to know about Test Automation«´ by Brian LeSuer

³Manager¶s Guide to GUI Test Automation´ by Yury Makedonov ³The ROI of Test Automation´ by Michael Kelly ³What is my ROI?´ by mVerify Corporation SQA Forums ³An Over view of Test Automation Frameworks´ by Nathaniel

Ritmeyer ³The benefits of keyword-based software test automation´ by David

W. Johnson Wikipedia



Igor Gershovich

Company: Connected Testing, Inc.

E-Mail: [email protected]

Phone: 720-933-9395

Website: www.connectedtesting.com

test automation - business v2

Documents