169655_performance testing tool evaluation

8/3/2019 169655_Performance Testing Tool Evaluation

http://slidepdf.com/reader/full/169655performance-testing-tool-evaluation 1/12

Performance Testing Tool Evaluation for Web Performance Suite 3.599

White Paper of 12

Performance Testing Tool Evaluation

for Web Performance Suite 3.599

By

Arkadyuti Roy (169655)

Perform an ce Tes t ing CoE

PDF created with pdfFactory trial version www.pdffactory.com

http://www.pdffactory.com/






White Paper of 12

Abstract

This paper discusses the details of the evaluation of Web Performance Suite 3.599, a

performance testing tool targeted to test web sites that use standard technologies such as web

forms, J2EE, ASP, .NET/C#, PHP, Ruby, ColdFusion, Java, etc. A comparative study of the tool

has been done with respect to other well-known performance testing tools like LoadRunner 7.8,

Silk Performer 6.0 and IBM Rational Robot 7 satisfying criteria used by Cognizant for

evaluation of performance testing tools.








White Paper of 12

Table of Contents

Abstract ................................................................................................................2 1.0 Introduction ................................................................................................4 2.0 Cognizant Evaluation: Criteria used by Cognizant for Evaluation .........4

2.1 Key Factors considered ........................................................................................ 4 2.2 Other factors (to be considered) .......................... ........................... ...................... 5

3.0 Load and Stress Test Capabilities ............................................................5 4.0 Tool Reporting Capability ..........................................................................7 5.0

Tools Ranking .............................................................................................7

6.0 Tools / Supported Protocols .....................................................................8 7.0 Performance Monitor Test Features .........................................................8 8.0 Consulting Requirements ..........................................................................9 9.0 Vendor Qualifications ..............................................................................10 10.0 Vendor Support ........................................................................................10 11.0 Conclusion ................................................................................................11

11.1 Potential Benefits ................................................................................................11 11.2 Potential Limitations ........................... ........................... ........................... ...........11 11.3 Constraints on Current Evaluation ................................................. ....................12

12.0 References ................................................................................................12








White Paper of 12

1.0 Introduction

The Web Performance Suite™ is web testing software targeted at web sites thatuse standard technologies such as web forms, J2EE, ASP, .NET/C#, PHP, Ruby,ColdFusion, Java, etc. Its growing list of modules currently can tackle everythingfrom baseline performance analysis to a load test with tens of thousands ofsimulated users.

The evaluation of web performer began with the need to evaluate the tool as aload generator tool in place of the Web Load tool that is currently being used forthe Web Re-launch project. To evaluate the tool a Cognizant benchmark criteriais followed and certain functionalities of the tool (Web Performance Suite 3.5xxDemo Version) have been evaluated and compared with other standard tools,like LoadRunner 7.8 (the most popular Performance testing tool in the market),

Silk Performer 6.0 ( the most powerful performance testing tool in the market)and IBM Rational Robot 7. The LoadRunner, Silk Performer and Rational Robotare already evaluated tools.

The problem with the most commonly used Load Testing Tools (likeLoadRunner, Silk Performer or Rational) is that they are costly. GenerallyLicenses are given on basis of the number of VUsers to be used for the tests, orin a timeline basis. That’s why the search for a cheaper tool like web Performer isencouraged. The license of Web Performance Suite is cheaper and it’s for Lifetime. Scripting is simpler in case of web performer in comparison to the othertools. But the tool has got some limitations while modifying the test cases as

there is no customized scripting as we get in case of LR, Silk or Rational.

2.0 Cognizant Evaluation: Criteria used byCognizant for Evaluation

While evaluating any load testing tool, following evaluation criteria are used byCognizant. The team has reviewed the product based on these key factors:

2.1 Key Factors consideredo Load and Stress Test Capabilitieso Performance Monitor Test Featureso Tool Reporting Capability

In addition certain additional factors will also need to be considered to ensure acomprehensive review of the product and the vendor (for due diligence):








White Paper of 12

2.2 Other factors (to be considered)

o Consulting Requirementso Maturity of vendoro Market share of vendoro Vendor Qualifications (e.g. Financial stability)o Vendor support

3.0 Load and Stress Test Capabilities

FEATURE

WEBPERFORM-

ER 3.5LOAD-

RUNNERRATIONAL

7SILK

PERFOR-MER

Scripting and Debugging Capabilities

Allows for reusing scripts fromfunctional test suite Yes Yes Yes Yes

Tool Customization No Yes Yes Yes

Test Language Features (FullDebugging Capabilities) No Yes Yes Yes

Ability to have data-drivenscripts that can use a storedpool of data Yes Yes Yes Yes

Actions in a script can be

iterated any specified numberof times without programmingor rerecording of the script Yes Yes Yes Yes

Cookies and sessions IDsautomatically correlatedduring recording and playbackfor dynamically changing Webenvironments Yes Yes Yes Yes

Different modem connectionspeeds and browser typescan be applied to a script

without any rerecording Yes Yes Yes YesAutomatic correlation orhandling dynamic data No Yes Yes Yes

Load Test Scenario Creation Capabilities

All users can be queued toexecute a specified action atthe same time. No Yes Yes Yes








White Paper of 12

Automatic load scenariogeneration based on loadtesting goals: hits/second,number of concurrent users

before specified performancedegradation, and so on Yes Yes Yes Yes

Load runs and groups ofusers within load runs can bescheduled to execute atdifferent times Yes Yes Yes Yes

Scales to 500–1,000 virtualusers Yes Yes Yes Yes

Thread-based virtual usersimulation Yes Yes Yes Yes

Process-based virtual usersimulation No Yes Yes Yes

Centralized load test controller Yes Yes Yes Yes

Allows for variable accessmethods and ability to mixaccess methods in a singlescenario: modem simulationor various line speedsimulation Yes Yes Yes Yes

Allows for throttle control fordynamic load generation No Yes No Yes

Allows for automatic service-level violation (boundaryvalue) checks No Yes Yes Yes

Allows for variable recordinglevels (network, Web, API,and so on) Yes Yes Yes Yes

Load Test Analysis Capabilities

Allows for transactionbreakdown/drilldowncapabilities for integrityverification at the per client,

per session, and per instancelevel for virtual users Yes Yes Yes Yes

Load Test Execution Capabilities

Adding or removing usersduring test run

No Yes No Yes

Snapshot on error Yes Yes Yes Yes

Distributing load on across Yes yes yes yes








White Paper of 12

agents

Load Test Feedback

Allows for Web applicationserver integration:ColdFusion, NetDynamics,Dynamo, and so on Yes Yes Yes Yes

Can run tests on varioushardware and softwareconfigurations Yes Yes Yes Yes

Compatible with SSLrecording Yes Yes Yes YesCompatible with one or moreof the relevant technologies:Linux, UNIX, NT, XWindows,

Windows CE, Win3.1, Win95,Win98, in2000, WinME Yes Yes Yes Yes

CITRIX / Microsoft TerminalServices Compatibility Yes Yes Yes Yes

4.0 Tool Reporting Capability

FEATURE WEBPERFORME

R 3.5

LoadRunner

7.8

Rational7

SilkPerform

er6.0

Summary Level Reporting Yes Yes Yes Yes

Test Report Presentation ,Customization Of Reports

Yes Yes Yes Yes

Automatic HTML ReportGeneration Settings

Yes Yes Yes Yes

Web page/componentbreakdown with component

size Yes Yes Yes YesMerging graphs No Yes No YesZoom in/out in the graphs,along with granularitydefinition No Yes Yes YesFiltering graphs in differenttimestamps. No Yes No Yes








White Paper of 12

5.0 Tools / Supported Protocols

Tools/

SupportedProtocol

WEB

PERFORMER3.5

Load

Runner7.8

Rational

7

Silk

Performer6.0

HTTP √ √ √ √

HTTPS √ √ √ √

COM/DCOM X √ √ √

RMI/IIOP X √ √ √

Corba/IIOP X √ X √

Winsock X √ √ √

ODBC X √ √ √

People soft X √ √ √

Siebel Web X √ X √

The “ √” symbol means that the testing tool supports the corresponding protocol.

6.0 Performance Monitor Test Features

FEATURE WEBPERFORME

R 3.5

LOAD-RUNNE

R

RATIONAL

SILKPERFOR

-MER

Monitors various tiers: Web server,database server, and app server

separately

No Yes Yes Yes

Supports monitoring for one or moreof ColdFusion, Broadvision, BEAWebLogic, Silverstream, ATG

Dynamo, Apache, IBM Websphere,Oracle RDBMS, MS SQL Server,Real Media Server, IIS, Netscape

Web Server

No Yes Yes No








White Paper of 12

Supports monitoring for one or moreof the relevant technologies: Linux,

NT, UNIX, XWindows, Windows CE,Win3.1, Win95/98, Win2000

Yes Yes Yes No

Monitors network segments Yes Yes Yes No

Supports resource monitoring Yes Yes Yes Yes

Synchronization ability in order todetermine locking, deadlock

conditions, and concurrency controlproblems

Yes Yes Yes Yes

Ability to correlate any metrics fromall monitors to identify performance

bottlenecks

No Yes Yes Yes

Ability to detect when events havecompleted in a reliable fashion

Yes Yes Yes Yes

Ability to provide client-to-serverresponse times

No Yes Yes Yes

Ability to provide graphical resultsand export them to common formats

Yes Yes Yes Yes

7.0 Consulting Requirements

FEATURE WEBPERFORMER

3.5

LoadRunner

7.8

Rational7

SilkPerformer

6.0

Maturity ofvendor

Tool Hasbeen

supportedsince 2004

Tools havebeen

supportedsince early

1990s.

Varioustesting

products havebeen on themarket sinceearly 1980s

Silk Test/QAPartner

products onthe marketsince early

1990s.

Market shareof vendor

Sustaining Considerable Sustaining Sustaining








White Paper of 12

8.0 Vendor Qualifications


3.5

LoadRunner

7.8

Rational7

SilkPerformer

6.0Financialstability of

vendor

WebPerformanceIncorporated

Continued/consistentgrowth pattern overlast several years

Stable / ProfitableCompany

Stable / ProfitableCompany

Timeexisted

1999 1989 1980 1990

9.0 Vendor Support


3.5

LoadRunner

7.8

Rational7

SilkPerformer

6.0Support Yes Yes Yes Yes

Onsite and PublicTraining around the

worldNo Yes Yes Yes

Awards Received SSPA STARAward in the

last 2 years forexcellence in

Support

Servicehas

receivedmany

awards

GoodTech

Support

10.0 Tools Ranking

Criteria WEBPERFORME

R 3.5

LoadRunner

7.8

Rational7

SilkPerfor

mer 6.0

Scripting 2 4 4 4

Load Test Scenario Creation 3 4 4 4

Load Test Analysis 3 4 3 4

Load Test Execution 2 4 3 4








White Paper of 12

Ø There is not customized scripting. On the contrary, it uses a set of screens

/ dialogue boxes to construct the scripts.

Ø Test cases cannot be modified to take randomized query parameter

values. It uses a table of pre-defined parameter sets to vary the requestquery string.

Ø Does not allow adding or removing a virtual user during execution.

Ø Does not support protocols like COM/DCOM, RMI/IIOP, Corba/IIOP,

Winsock, ODBC, People soft, Siebel Web.

Ø Does not support monitoring for one or more of ColdFusion, Broadvision,

BEA WebLogic, Silverstream, ATG Dynamo, Apache, IBM Websphere,

Oracle RDBMS, MS SQL Server, Real Media Server, IIS, Netscape Web

Server.

Ø Does not allow all users to be queued to do a specific action at a specific

time (inability to produce Rendezvous’ point).

11.3 Constraints on Current Evaluation

Ø The evaluation copy only allows a limited number of users. This is not

suitable to scale up the number of users to test under high load.

Ø Several features as documented in the website are unavailable on this

evaluation version.

12.0 References

Ø http://www.webperformanceinc.com/

http://www.webperformanceinc.com/

http://www.webperformanceinc.com/

169655_performance testing tool evaluation

Documents