©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

Selenium vs. Quick Test Professional: the hidden costs of open-source test automation

Brad DavidowitzManager, QM Services, VIP

Joseph FordPrincipal, QM Services, VIP

VIP at a Glance

• IT Solutions Provider

• Founded 1996

• Over 900 Staff

• 11 Offices– Sacramento Metro (2 offices)– Washington DC Metro (2 offices)– New York City– Dallas Metro– Houston Metro– Los Angeles Metro– Minneapolis Metro– Denver Metro– Bangalore, India

• 82% of Revenue From Repeat Business

• Successfully Delivered Solutions to 900+ Clients in Commercial & Public Sectors

• Solution Offerings by Practice– Management Consulting– Technology Integration/App Dev– Business Intelligence and DW– SAP Consulting Services– Quality Management

3

• Strategic Planning

• Testing

• Test Automation

• Velocity Performance Testing Program

• Production Monitoring (BAC)

• HP Tool Sales and Implementation

VIP Quality Management Solutions

4

Why the Presentation?

• Functional Test Automation is a Large Part of Our Business– Hundreds of client engagements

– Trusted advisor

• The Big Question:

5

“Can’t I just use

a free tool?”

Why the Presentation? (Cont.)

• Lack of Good Comparison Studies

• Evidence Can Be Found to Support any Theory

Image Source: Total Cost of Ownership:

http://www.netc.org/openoptions/pros_cons/tco.html

Needed a better answer!

6

7

Comparison Process

• Typical Proof of Concept Process

– Define steps in PoC test

– Define comparison criteria

– Select tools for comparison

– Execute tests independently

– Compare results

– Draw conclusions

8

Proof of Concept Test Steps

• Launch Browser

• Validate Logo

• Login and Input Data

• Navigate Within Site

• Validate Text

• Playback Test as Recorded

• Parameterize and Data Drive

• Cause and Detect Failure

• View and Interpret Test Results

• Edit/Enhance Script and Re-run

http://newtours.demoaut.com

9

Comparison Criteria

• Ability to Execute Proof of Concept Steps

• Ease of Test Automation

– Installation and configuration

– Record/playback and advanced scripting

– Results

• Versatility

– Support for emerging technologies

– Integration with other tools

• Resources

– Availability of experienced professionals

– Help, documentation, published materials

– Technical support, tool maintenance, training

• Hard/Soft Costs

10

Tool Selection: Why Selenium?

• Free & Open Source & Free of Charge

• Functional Test Automation Tool

– Supports web testing

– Supports record/playback and adv. scripting

• Recent Releases Available for Download

• Active Community of Users

• Peer Recommendations

• Popular Topic

11

Tool Selection: Why QTP?

• Market Leader in Test Automation

• Supports Largest Array of Technologies

• Graphical, Keyword-Driven Test Development

• Personal Experiences With it

• Peer Recommendations

• Part of HP’s ‘Functional Test’ Suite

12

Selenium Components

• Basic Recording– Firefox

– Selenium IDE

• Custom Coding– Firebug

– Java

– Eclipse IDE

– JUnit libraries

– Selenium libraries

• Playback– Selenium Remote Control

13

Installing and Configuring Selenium

• Download and Install– Firefox

– Java Development Kit

– Eclipse IDE for Java Developers

• Download and Unzip– Selenium RC

• Install Firefox Plug-ins– Selenium IDE

– Firebug (optional)

• Configure– Eclipse IDE

• Start Selenium RC

14

Installing and Configuring Selenium

Installing and Configuring QTP

16

Comparison: Install and Configure

Selenium

• Firefox

• Selenium IDE

• Selenium RC

• Firebug

• Java

• Eclipse

• Configure Eclipse

QTP

• QTP

• Configure QTP– None required

Automating with Selenium vs. QTP

17

Video: Automating With Selenium

Video: Automating With QTP

25

Comparison: Test Automation Process

Selenium• Launch Firefox

• Invoke Selenium

• Record Script

• Convert to Java

• Launch Eclipse

• Copy/Paste Script

• Fix Errors

• Custom Code to Parameterize

• Custom Code to Data-Drive

• Run Test

• View Results

QTP

• Launch QTP

• Record Script

• Parameterize

• Data-Drive

• Run Test

• View Results

26

Comparison: PoC Results

Selenium PoC Test Step QTP

Green Launch Browser Green

Red Validate Image Green

Green Input Data Green

Yellow Navigate Green

Yellow Playback Green

Yellow Parameterize Green

Red Data Drive Green

Green Detect Failure Green

Yellow Results Green

Red Edit Script Green

Key

Green completed with ease

Yellow required extra effort

Red result not desirable

28

Comparison: Versatility

Selenium• Web Testing Only

• Must Record in Firefox,

Playback in Many OS, Browser

• No Built-In Object or Data

Management Concepts

• Some Emerging Technologies

Supported by Community

• Can Be Integrated With Many

Development Tools

QTP• Automates Many Technologies

• Must Record in IE*, Playback

in Many Windows Browsers

• Facilitates Best Practices via

Object Repository, Data Table,

Parameterization

• Native Support for Emerging

Technologies + Extensibility

• Tight Integration With Other

HP Tools

29

Comparison: Resources

Selenium• Resources Must Have

Significant Development Skills

• Unofficial Training Only

• Community Driven or Pay for

Commercial Support for

Multiple Tools

• Documentation for Complete

Selenium Solution Does Not

Exist

QTP• Development Skills Not

Required

• Official Training Available

• Professional Support Included

• Documentation is Mature and

Complete

30

The Total Cost of Ownership

• Hard Costs– Licensing

– Support and maintenance

– Staffing

• Hidden Costs– Staffing

– Time

– Complexity

– Internal desktop policies

– Legal policies

– Lost opportunity

31

• Licensing− Selenium: $0

− QTP: ~ $8000* (one-time fee, varies by scope of use and region purchased)

• Support and Maintenance− Selenium: $0

− QTP: ~ $1600/year*

• Staffing− Selenium: dev or sr. qa resource

− QTP: qa resource

− Varies by region

TCO: Hard Costs

*Not an official quote.

Represents example list

price as of April 2010 for one

“HP Functional Test” site

license. Contact HP or HP

Partner for latest pricing as

discounts may be available.

32

Comparison: Hard Costs (Cont.)Selenium QTP

• Regional Example− Selenium $90,000

− QTP $70,000

Image Source: http://www.salary.com/

33

TCO: Hard Costs (Cont.)

34

TCO: Hard Costs (Cont.)

35

TCO: Hidden Costs of Selenium

• Staffing− Selenium career path vs. QTP career path

− Finding Selenium resources is more difficult

− Selenium detour creates higher turnover

• Time– Longer learning curve

– Time spent getting Selenium to work = less time testing

• Complexity– Solutions take longer to build, maintain

– Requires integration of many components

– New challenges will require more components and more custom development

36

TCO: Hidden Costs of Selenium

• Internal Desktop and Legal Policies– Many components for IT to support

– Open-source may not be permitted

– Firefox may not be permitted

• Lost Opportunity– Only supports web, does not solve other automation

challenges

– Organizational challenges around reuse, standardization, centralization, visibility will be harder to accomplish

37

Conclusions

• Free Does Not Mean Free– Free doesn’t even mean cheaper

– Cost of the tool is just one small factor

• Understand Your Expectations– From the software producer, from the tool itself

• Examine How the Software Will Be Used– Who will use, support and maintain it

– What will it be used for in the near and long term

• Our Conclusions:– Selenium may be right for some specific situations

– QTP can be the right choice for a much larger array of situations

38

Questions?

Solution ShowcaseBrad DavidowitzJoe Ford

39 ©2010 Hewlett-Packard Development Company, L.P.

To learn more on this topic, and to connect with your peers after

the conference, visit the HP Software Solutions Community:

www.hp.com/go/swcommunity

41

References

• The True Cost of Software:

http://www.computerworld.com/s/article/81590/The_True_Costs_of_Software

• Total Cost of Ownership:

http://www.netc.org/openoptions/pros_cons/tco.html

• The Free Software Definition:

http://www.gnu.org/philosophy/free-sw.html

• Why Open Source Software:

http://www.dwheeler.com/oss_fs_why.html

• Repository of Open Source Tools Including Some Reviews:

http://opensourcetesting.org/

• Eclipse: http://www.eclipse.org

• Selenium HQ: http://seleniumhq.org/

• Salary Ranges: http://salary.com

42

Appendix A: QTP and Web 2.0

• Adobe® Flex/Flash

• Microsoft® Silverlight™

• AJAX in Various Forms

– Dojo Toolkit

– Yahoo! User Interface (YUI)

– ASP.NET AJAX

– Google® Web Toolkit (GWT)

• Web 2.0 Extensibility

– Build your own custom add-in

43

Appendix B: QTP Add-Ins

• QTP Add-Ins Included With Purchase– ActiveX

– VisualBasic

– Web

– Delphi

– Java

– .NET

– Oracle

– PeopleSoft

– PowerBuilder

– SAP

– Siebel

– Stingray

– Terminal Emulator

– Extensibility

• QTP Tool Integrations– HP Quality Center

– HP LoadRunner

– HP Business Process Testing (BPT)

– Many BPT Accelerators