concepts on software performance testing

testrun.co.nz@testrunnzConcepts on

Software Performance

Testing

Chris SAUNDERS 2014

testrun.co.nz@testrunnz

testrun.co.nz@testrunnz

Presentation's Purpose

testrun.co.nz@testrunnz

Presentation's Purpose

• To talk about software performance testing

• To explore concepts, ideas and things to consider

• Not talking about tools

testrun.co.nz@testrunnz

What is Software Performance?

testrun.co.nz@testrunnz

What is Software Performance?Software performance is the manner in which

software functions, operates, or behaves; based on a given set of factors.

Software performance testing is to determine how a system performs in terms of responsiveness and stability with a defined set of factors.

What factors?

Each person takes two sticky notes and writes their answer down, and places it on the wall

Note this question may be asked again later

testrun.co.nz@testrunnz

General Measures

testrun.co.nz@testrunnz

General Measures

Measures Description

1. Known measures Measure where the system meets a given predefined performance criteria

For example: The CPU & Memory resources will not exceed 80% for more than 5 second duration.

testrun.co.nz@testrunnz

General Measures

Measures Description

1. Known measures Measure where the system meets a given predefined performance criteria

2. Comparable measures Measure between the performance of two systems to determine which has better performance

For example: The new calculation engine will have a 20% increase in processing time and data volume

throughput.

testrun.co.nz@testrunnz

General Measures

Measures Description

1. Known measures Measure where the system meets a given predefined performance criteria

2. Comparable measures Measure between the performance of two systems to determine which has better performance

3. Unknown measures Measure where the system has a possible performance issue in a given area

(One does not exactly no where or how it will fail)

For example: Determine the system performance on the system due to teamed network card failure.

testrun.co.nz@testrunnz

The Business Case

testrun.co.nz@testrunnz

The Business Case

“What is the business case for performance testing?”

“How does it add business value?”

Need to define “realistic performance related goals”

testrun.co.nz@testrunnz

Performance Goals5 Examples Types

testrun.co.nz@testrunnz

Performance Goals:Water pipe analogy

Consider software performance testing like Christchurch’s water pipe system

testrun.co.nz@testrunnz

Performance Goals:Response Time

Measuring the duration of time between two points

For example: The duration for a user waiting on a file to upload

A BDuration

0:00:00.29 or 290ms

testrun.co.nz@testrunnz

Performance Goals:Concurrency

Measuring the effect on the system with concurrent processes

For example: The effect of the different number of users trying to upload files at almost the same time

testrun.co.nz@testrunnz

Performance Goals:Throughput

Measuring the effect on the system with throughput of information (volume)

For example: The effect of different sized files being uploaded

20 l/s98 l/s

testrun.co.nz@testrunnz

Performance Goals:Interdependence

Measuring how the system responds to different factors

For example: Consider the relationship between the following1. Response time 2. Concurrency 3. Throughput

A BDuration

testrun.co.nz@testrunnz

Performance Goals:Interdependence

Compare these two scenarios:1. 100 users are each uploading 20MB files, start

uploading files within a 5 minutes duration.2. 10 users are each uploading 200MB files, start

uploading files within a 10 seconds duration.

Which scenario has better performance?

testrun.co.nz@testrunnz

Performance Goals:MTBF & MTTR

Mean Time Between Failures & Mean Time To Repair

Determine the performance of when the system fails and how long does it take to repair?

ConsIder the relationship of the system to other systems or internal components

testrun.co.nz@testrunnz

Performance Goals:MTBF & MTTR

Source : http://www.greenm3.com/gdcblog/2012/7/31/netflix-sets-chaos-monkey-free-for-all-to-use-next-comes-mor.html

The Chaos Monkey is a software tool that was

developed by Netflix engineers to test

the resiliency and recoverability of their Amazon Web Services

Open Source https://github.com/Netflix/SimianArmy

Netflix engineers develop automatic recovery mechanisms

testrun.co.nz@testrunnz

Performance Goals:Summary

1. Response Time2. Concurrency3. Throughput4. Interdependence5. MTBF & MTTR6. …

testrun.co.nz@testrunnz

Some Performance Testing Approaches

testrun.co.nz@testrunnz

Performance Testing Approach:Load Testing

Measure how the system behaves under a specific expected load.

Load characteristics: Define number of concurrent users, with a determined throughput for a given duration

Could be looking for bottlenecks

testrun.co.nz@testrunnz

Performance Testing Approach:Stress Testing

Measure how the system behaves with loads beyond normal operational capacity.

Measure modes of failure.

Determine the robustness of the system: How, when & where is it going to fail

testrun.co.nz@testrunnz

Performance Testing Approach:Spike Testing

Measure how the system behaves with the sudden increase in load

Determine system response; performance profile or system failure

testrun.co.nz@testrunnz

Performance Testing Approach:Endurance (Soak) Testing

Measure how the system operates under a continuous expected load

Determine performance degradation

testrun.co.nz@testrunnz

Environmental changes to determine system performance

System controlled failure

System configuration changes

testrun.co.nz@testrunnz

Performance Testing Approach:Summary

1. Load Testing2. Stress Testing3. Spike Testing4. Endurance Testing5. Environmental changes to

determine system performance6. …

testrun.co.nz@testrunnz

What’s Software Performance?

testrun.co.nz@testrunnz

What is Software Performance Testing?

Software performance testing is to determine how a system performs in terms of responsiveness and stability with a defined set of factors.

What factors?(Each person can change their two sticky notes on the wall)

Discuss software performance testing…

testrun.co.nz@testrunnzConcepts on

Software Performance

Testing

Chris SAUNDERS 2014