why & how-performance test

8/10/2019 Why & How-Performance Test

1/25

1

Why & How : Performance Test

For: QAAC

By: Deepak


2/25

2

Performance Test References

Investigate Performance Early; Validate Performance Last- by Scott Barber is the Chief Technologist, CEO, and President of PerfTestPlus,

Inc. Reference article: Performance Investigatorin Better Software Magazine(March 06 issue).

To derive a good testing strategy out of scalability andperformance testing, there needs to be testable and

measurable requirements - by Srinivasan Desikan is the director ofquality engineering at Siebel Systems, Inc. Reference article: Scalability andPerformance Testing of Server Software (Website http://www.stickyminds.com

published July 05).
http://www.stickyminds.com/BetterSoftware/magazine.asp?fn=cifea&id=74http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/BetterSoftware/magazine.asp?fn=cifea&id=74


3/25

3

Table of Contents

Performance Test description and its need

Performance Test Objectives

Performance Testing Engagements

PVS Test Terminology

PVS Test Requirements

PVS Test Process

PVS Tool-set

Transaction Response Time

Response Times Vs Service-level agreements (SLAs)

Monitor Application and Database Server UsagePerformance Testing Best Practices

Benefits of Performance Testing

Summary and Conclusion


4/254

What is Performance Testing?

The process of exercising an application by

emulating actual users. This can be used tomonitor how the database, application code andhardware will act under load.

Performance testing verifies that the applicationmeets specified requirements under normalconditions.


5/25

5

The Need For Performance Testing

No matter how rich application is functionally, if

it fails to meet the performance expectations(SLA) of customer, the application will bebranded a failure.


6/25

6

Performance Test Objectives

Determine business transaction response time

Identify system bottlenecks

- Server resource usage e.g. CPU, Memory and Disk Space- Network Bandwidth(throughput) and Latency(delay)

e.g. Limited network throughput speeds serve to introduce alatency(delay) when transmitting larger amounts of data at specific

location

Determine system (hardware/software) optimalconfiguration

Verify current system capacity and scalability for futuregrowth

- Determine how many users the system can support

- Determine if the application will meet its SLA


7/25

7

Performance Test Engagements

Performance Risk Assessment

-Determine if application needs to be PVS tested bycompleting performance risk assessment matrix

Performance Volume Stress(PVS) Testing

- Test Strategy with Action plan

-Script development

- Test execution- Resource monitoring

-Analysis & Recommendations

Load-balancing and Fail-over Test

Network Analysis


8/25

8

PVS Test TerminologyTypes of Tests

Load Test: To determine the impact on an applications servicelevel for a specific number of users and user types.

Volume Test : Inject heavy workload (e.g. future volumes) toshow if the system can handle the peak volume of data. Testsserver response time under heavy load and varying user levels,measures round-trip response time and determines maximumnumber of users.

Stress Test: Test the application or server to determine a loadpoint at which the application suffers severe degradation ormalfunction.

Network Application Analysis Test: To capture the network

characteristics of an application and to proactively seekinefficiencies to forecast potential performance and network risk.

Latency: The amount of time it takes a packet to travel fromsource to destination. Together, latency and bandwidth define the

speed and capacity of a network.


9/25

9

Other Terminology

Workload: The total user load exercised in a productionenvironment.

Workload Model: A subset of actual workload used forperformance testing (80/20 rule: 80% of the workload is

generated by 20% of the transactions mix).Test Cycle: A group of testing scenarios designed to accomplish aspecific threshold or condition (production level conditions, futureproduction conditions, end-state conditions).

Scenario: A logical unit of work or a transaction sequence toexercise a type of performance test that achieves a performancetest objective.

Script: Automated set of instructions to drive the performance test

during execution.Virtual User: A software process that simulates real userinteractions with the application.

Response Time: The time it takes between initial request and

complete download of response (rendering of entire web page).


10/25

10

PVS Test Requirements/Prerequisites

Technical design and architecture, Non-functional businessrequirements and Data flow diagrams.

Hardware/software configuration and access to monitorresource usage.

Establish and document SLAs (e.g. scope and objectives,

response time, availability, business scenarios, acceptancecriteria and project timelines etc).

Stable test environment, test data, user IDs or any otherunique data required for the Performance test types.

Use-cases: Test scripts reflecting how the end-users are

using the system.


11/25

11

Perfo

rman

ceTestingPha

ses

Performance Testing Processes

Templates/Documents Needed

Performance Testing Questionnaire

Test Cases


Performance Test Strategy

Effort Estimation Guideline


Performance Test Report

RequirementGathering

Effort Estimation And

Planning

Scripting And Execution

Result Analysis , Reporting And DefectTracking

Templates/Documents Needed Performance Testing Best Practices

Scripting Standards and Naming

Conventions Guidelines

Workload calculation


12/25

12

PVS Test Process

Gather and ReviewChecklist Requirements

Test Strategy/ApproachDocument-Scope and Objectives

-Identify needed test

resources-Milestones

Script Development

-Design test-Develop workload model-Build test bed

-Create script test data

Test Execution-Performance/Load test-Review initial output

-Volume and Endurance test-Stress test

Result Analysis and FinalReport-Executive summary

-Recommendations toimprove application

performance-Results and Analysis


13/25

13

Performance Engineering Methodology

Performance Engineering

Understand System (Sys.

Architecture, Workload,

Performance Goals)

Develop Test Assets

(Test Plan, Test Scripts,

Monitoring Strategy etc)

Execute Tests Collect monitoring metrics

Analyze Results andmonitoring metrics

Performance criteria are met

TUNE

Criteria not met

Re-test after tuning

Complete Testing


14/25

14

PVS Tool-set

Mercury-HPs LoadRunner, Borland-Segues SilkPerformer,Compuwares QA Load, IBMs RPT, and Empirixs eLoad

-Load injection and scenario design mirrors production

-Round trip transaction times-Monitor application and architectural components

Perfmon/Sitescope

-Performance logs and alerts supports detailed monitoring of theutilization of operating system resources.Web Analytics/Trends/Topaz/Gomez

-Interactive web analytics reporting(monitor production usagepattern)Vantage/Sniffer/Netsense

-Network Management - Analysis for bandwidth usage andlatency issues


15/25

15

Multi-Tier Architecture

For instance, its veryimportant to performancetest a complex architecture

which support highlyscalable and robust load-balancing mechanisms at alltiers of the architecture using

a combination of hardwareand software load-balancing.

All tiers can be scaled

individually by buildingclusters of servers for load-balancing and highredundancy.


16/25

16

End-to-End Interface Architecture Diagram

PWMCICS

[Mainframe]

JSP MQ

MQ

SeeBeyondAdmin ServerHTTP call

[XML]

MQ Wrapper program

Request[CobolCopybook]

Response[CobolCopybook]

AgentInsured

Home office employee

Policy Request


17/25

17

Monitor Server Usage

Timings-Response time-Execution time-Wait and connection times

Resource Consumption

-CPU, Disk Space and Memory usage-Number of connections, active threads, timeouts and Servlets

requested, loaded and active-Web page counts, size and timingsDatabase Performance

-Physical reads & writes

-Access paths and connections-Locking performance- I/O, buffer and wait counts and timing

Additional application specific metrics

f


18/25

18

Measurement of Application Business TransactionResponse Time


19/25

19

Response Times Vs Service-level agreements

(SLAs)Average Transact ion Response Time

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

ResponseTime

(Seconds)

Cycle 1-1300 users 0.632 0.587 0.07 0.066 0.59 0.658 0.594 0.622 0.595 0.594

Cycle 2-1500 users 0.609 0.892 0.068 0.063 0.533 0.536 0.581 0.611 0.586 0.586

Claims

Login

Corres.

Login

Claims

Enter ID

Corres.

Enter ID

Claims

View

Image

Corres.

View

Image

Claims

Item

Close

Corres.

Item

Close

Claims

Logout

Corres.

Logout

Login Acceptance

Criteria 10 seconds

View Image

Acceptance

Criteria 30

seconds

Enter ID

Acceptance

Criteria 5

seconds

Logout

Acceptance

Criteria 5

seconds

Item Close

Acceptance Cri teria

10 to 60 second s


20/25

20

Monitor Database Server


21/25

21

Monitor Database Server on Mainframe

Database Server Timings Application AAA

Batch Job / MetricCycle: 1B

3.5mil records09/15 09/22

Cycle: 25.4mil records09/22 90/30

Cycle: 311.6mil records

10/01 10/10

GXX0000

SQL Total Elapsed Time 01:44:03.25 00:55:29.78 06:10:12.24

SQL CPU Time 00:04:27.76 00:01:53.88 00:31:02.91

SQL I/O Elapsed Time 01:15:38.35 00:38:32.57 03:38:11.97

SQL Wait for Read 00:00:13.71 00:00:07.06 00:00:08.20

SQL Lock/Latch Time 00:00:00.00 00:00:00.09 00:00:00.57

Total SQL Elapsed Time 01:57:23.10 01:12:36.65 06:26:08.99

DML Comparisons Application AAA

Batch Job / MetricCycle: 1B

3.5mil records09/15 09/22

Cycle: 25.4mil records09/22 90/30

Cycle: 311.6mil records

10/01 10/10

GXX0000V

Total Cursor Open / Closes 3,235 1,336 8,462Total Fetch 7,035 3,323 25,443Total Select 441,554 170,307 2,284,024Total Insert, Prepare, Delete 0 0 0Locks Suspended / Timeout 0 0 0Max Page Locks Held 1 1 1Total Waits for Lock/Latch 132 620 752Locks Requested 112,028 5,594 2,149,614


22/25

22

Performance Testing Best Practices

Performance testing requires a different mind set and skill-set tothat of functional testing.

-Best started early in the development life cycle wheneverpossible.

Understanding business requirements and expectations.

-User activities and behaviors is key to designing suitable tests.

-Ensure tests represent realistic usage of the application.Automated test tools coupled with fast backup and restoremechanisms are essential due to the need to repeat tests multiple

times.System bottlenecks can rapidly become very technical in natureand consume considerable resources and efforts to diagnose.

-Resolution may require considerable re-work and even redesign.


23/25

23

Benefits of Performance Testing

The ultimate goal of all performance testing is to optimize performancewhile minimizing costs.

Does the application perform as required?. Will it continue to perform asrequired?.

Will it meet current and future performance requirements for the leastcost?.

Improve new or upgraded applications and third-party services quality,workflow and reporting.

Reduce system downtime by isolating and resolving infrastructurebottlenecks, if any.

Improve quality from a users prospective.

Early identification of major application defects and architectural issuesReduce cost of change, system costs and increase profits.

Cost per Defect

Design Test Production$1x $10x $100x


24/25

24

Summary and Conclusion

In this presentation we have covered

Performance Testing and its need to start early in the

development process.

Performance Testing Objectives and Engagements.

PVS Testing Terminology, Requirements, TestProcess, Tool-set, Transaction Response Time,Service-level agreements (SLAs), Application andDatabase Server Usage.

Performance Testing Best Practices.

Benefits of Performance Testing.


25/25

25

Questions ?For More Information contact:Deepak Sharma

Email: [email protected]: (860)4032195(O), (919)9397091(C)
mailto:[email protected]:[email protected]