1132 performance testing ebooklet albert witt final

7/29/2019 1132 Performance Testing Ebooklet Albert Witt Final

1/15

EuroSTARSoftware TestingC o m m u n i t y

Performance Testing,

a Practical Guide and

Approach

Albert Witteveen

Owner/Founder Pluton IT


2/15

Performance testing, a practical guide and approach

2

PA

GE

Abstract

This eBook contains a number o chapters rom

Albert Witteveens book entitled Perormancetesting, a practical guide and approach.

This book was written to give people a start in

perormance testing. It should help people to

understand the basics o how to do perormance

testing and hopeully how to do a good job.

Another target audience is anyone dealing with

perormance demands or problems. How can

you tell i the testers are going to deliver a resultthat gives tangible insight in the perormance?

The book aims to give you some handle with

which you can judge the approach and see i

you get your moneys worth.

2. An Overview o

the Basic Activities

2.1 Introduction

What should you learn rst? To navigate or to

sail? I you learned how to navigate perectly

and then discover that you dont last on

a sailing boat or longer than 10 minutes,those navigation skills are a bit useless. Thats

why in this book we rst get into the actual

perormance testing itsel beore discussing

planning, modeling and so-orth. The basics

will teach you how to simulate many users with

tools and what you need to know beore you

can judge i you created a good test script.

2.2 A ew defnitions in perormancetesting

First it is time to set some denitions. Especially

since some words we use oten are dened in

many other ways. Which denition is right is not

the point, but is important that we dene here

what denition we use.

2.2.1 Perormance testing

A denition o perormance testing is that it

is about testing i a system accomplishes its

designated unctions within given constraints

regarding processing time and throughput

rate.

Perormance testing is a superset containing

other tests such as load testing, stress testing

endurance testing etc.

2.2.2 Load testing

When we are using the term Load we are talking

about workload put on a system. Typically this

is the load provided by the multiple users thatwe expect when a system is in production.

When you a have a web application o which

you expect that under normal circumstances it

is used by 200 simultaneous users, those users

provide the load.

What we test or is the perormance o sotware

in terms o response times, throughput etc. at

the given load.

So when we are discussing load testing,

we discuss testing on the perormance o

the application in terms o response times,

throughput etc. when we apply a load that is the

same as what we expect during production.

2.2.3 Stress testing

Stress testing has many similarities with load

testing. It is about testing with high loads. The

big dierence is that with stress testing we go


3/15


3

PA

GE

way beyond expected load and apply load until

the system cant handle this anymore. This is

done or a ew reasons: we want to know when

a system breaks. But most o all it is to determine

what happens at that moment. Is there a ullsystem outage? Do we lose transactions? Are

there data synchronization issues? Additionally

we can monitor to see which part o the system

and resources is the bottleneck. That will give a

clear indication on what to improve or expand

i we have to.

Stress testing is thereore about nding out

when it breaks and what happens when it

breaks.

2.2.4 Endurance testing

A little less known, but certainly related and

important is endurance testing. This is about

testing with a given load, usually not all that

high, but signicant nonetheless, over a

prolonged period o time.

The reason or perorming these tests is to nd

out the problems and deects that occur over

time. A well known example o such a problem

is the memory leak. Due to deects, over time,

with the load not increasing, the memory

consumption does increase. That is a clear

indication that there is a memory leak. I this

happens in production, the systems need to berestarted to remedy this.

2.3 Load generation

What all tests share in common is that the aim

is to provide load similar to what we expect in

production. It is not really easible to do this by

hand. You could do that in theory, but it would

require many people to do the work. The

common approach is to generate the load using

a load and stress test tool.

2.3.1 Simple example o a webapplication

To explain what a load generation tool does

we will use the example o testing a webapplication. For this example we will use a airly

simple setup or a web application. There are two

servers, a web server and a database server. On

the web server the sotware contains and runs

the code that provides the web application. The

data is stored in the database server and the

web application queries this database when it

requires data.

When this application is in production, we

expect that roughly 100 users will be using the

application at the same time. That means that

by 100 users their web browsers will request

pages rom the web server. In production, that

would mean 100 computers and 100 people in

many dierent places.

2.3.2 Simulating the users

What we do to simulate this is that we use a

tool to simulate the HTTP trac between the

browsers and the web server. The web server

cannot tell the dierence between the requests

by real users and by that o our tool. The requests

are multiplied by the number o users we aim

to simulate. Usually this is reerred to as virtual

users.

2.3.3 Record and playback

Creating this trac by hand is a daunting task

that would not only require a lot o knowledge

o the HTTP protocol, but also it would be

nearly impossible to imitate exactly what the

browsers and users would do. For that the loadgenerating tools provide record and playback

unctionality.

When the tool is in recording mode, you have


4/15


4

PA

GE

to use your browser and perorm the actions as

the real user would do. All the trac between

the browser and the server is recorded. Then

the tool will turn the recorded trac into a test

script that it can rerun to playback the exactsame request as were perormed during the

recording.

2.3.4 Parametrization

Now what would happen i you recorded

the creation o a new user account? During

recording you would or instance create the

user: testuser1. When you rerun the exact

same request, you would try to create the

user testuser1 one again. This time however

the server would reply something like user

already exists. All the coming requests would

subsequently receive error messages, eectively

making sure that the load on the server is not

what you wanted to generate. And this is or the

simple test o making a new user. For changing

data, or most tests, data needs to be used thatis available and does not confict with certain

business rules.

There is more. Much o the requests that

a browser makes are based on returning

inormation the browser received rom that

server such as cookies and session-ids. Many

applications even have their own inormation

going back and orth the keep track onrequests.

For this a script nearly always needs

parametrization beore it can be rerun,

especially beore it can be rerun under load.

Parametrization is one o the most dicult and

time consuming part o load and stress testing.

2.3.5 Content checks in the script

So when weve done all that, the script runs

perectly. The Load test tool when playback

tells you that the script is passed. We however

still dont know i it really passes. We know that

we send the same requests, but we dont i the

answers that we get are the correct ones.

Take the ollowing example o a web application.

You have created a test script that tests the login

procedure o the application. The script sends

the requests. But when the login is requested

the server, due to a parametrization omission

doesnt answer with a page that youre logged

in, but it answers instead: login denied. On the

server side this means it probably does a lot less

than when the login is successul, giving you

dierent results than what you should get.

Thereore in any test script you need to build

content checks. For instance i the screen o a

successul login shows the text: welcome user

Username1, build in a check that tests that

the response o the server contains the words

welcome user.

Also it can be necessary to check in systemsthat should be updated by your test that the

updates are done.

2.3.6 Test the water

When this is all ready and the script works or

one virtual user, it makes sense to try it with

just a ew virtual users, or instance threeusers. Then run the test with the three virtual

users, and validate in the system under test

that everything was updated as expected. This

is done to nd errors in your scripts such as

orgetting to ensure that the script is set to use

unique values rom the test data. All too oten

testers parameterized everything correctly,

lled all the test data les, to later discover that

the script used the rst entry in the test data le

or all virtual users. Oten this requires setting

the properties o the parameters correctly. The

reason to not jump in right away with a ull

blown test is that detecting and troubleshooting


5/15


5

PA

GE

issues that indeed exist is much easier with just

a ew virtual users. It also prevents you rom

burning your test data.

2.3.7 Summarizing the steps ocreating a test script

So these are the steps in creating a valid test

script in load generating tool or testing load o

multiple users:

Record the activities by setting the tool on

record and perorm the actions within the

system under test.

Parametrize the script so that when we run the

load test every virtual user has dierent data,

that the script captures server responses to be

resent.

Collect test data or the parametrized items that

need to be lled with data.

Build in checks to see i the server gave the

correct responds.

Veriy the parametrization by running a test

with a ew virtual users to detect parameter

property errors.

There is much more to tell about creating test

scripts. But these are the basic steps.

2.4 Setting up monitoring

Recording beginning and end times are a basic

unction o the load test tools on the market.

But when the time comes to analyze the results

we will want more data. For that we need to

utilize monitoring sotware.

2.4.1 Monitoring tools

Monitoring tools will continuously monitor the

systems and record the resource usage that

they were made or. Common things to monitoror are:

CPU usage

Memory usage

Swap space used

IO usage

Database activities

Network activity

Web server connections

Some load test tools have monitoring build-

in. Other tools dont provide this and then you

have to rely on monitoring outside o the test

tool. Even or the tools that have monitoring

build-in it oten makes sense to also set up some

monitoring outside o the test tool. For instance

i you want monitoring on the database, most

database products eature their own monitoring

tools that provide more in depth inormation

and most o all in a ormat that is well known

to the database experts that you will rely on to

help you to nd and solve issues.

2.4.2 Selection o monitoring items

S Some tools make it really easy to monitor

each and everything. It then becomes verytempting to turn them all on. Especially i there

analyzing tools available allowing you to select

and deselect every monitoring item during the

analyzing phase. Nonetheless too much oten

is too much. It is not so much that there is not

enough space or all the data or that it becomes

unusable. Rather it is about that with so much

inormation, people stop to look careully and

see patterns. I you have to deselect all the lessrelevant stu or every test it doesnt take long

beore you stop looking at all and just go to the

summary page. In the rare cases that a test is

very dicult to rerun you may want to enable a


6/15


6

PA

GE

bit too much just to be certain. In all other cases,

keep it limited to what you would look at beore

detecting an issue in the rst place. When you do

detect an issue that needs urther investigation,

rerun the test with added monitors.

2.5 Reporting

When everything worked out and we nished

a test, it needs to be reported. As always with

reporting, what you report depends greatly

on who you report to. You may have to report

to an accepting party, sometimes a sotware

development project needs a report to show

it has met its targets, oten you are involved

in testing to reproduce perormance issues

on production and veriy that a solution will

provide improvements. All o these dierent

reasons or testing require a dierent ocus in

your report. Obviously there are also overlaps

and similarities, but reporting is important and

is equally important to report the right thing.

Also reporting issues needs to be done right,

although in many ways similar to reporting

regular issues or ndings, perormance testing

puts some strong demands on what you include

in the issue report. We will explain much more

in the chapter on reporting.

2.6 Conclusion

The basics were explained here to provide

an overview o the activities. Perormance

testing can mean load testing, stress testing

or endurance testing. To perorm a test, a

tester needs to perorm quite a ew steps.

Recording is just one step, the scripts needs to

be customized to handle dierent data, server

responses and check or real results. To provide

insightul reports we need proper monitoring.

This needs to be set up. When a test is done, the

results need to be reported and reporting on

perormance depends on the purpose o the

test and to whom to report. All these items will

now be explained in more detail.


7/15


7

PA

GE

5. Get in Line

5.1 Introduction

Computers behave in a very simple way. They

stand still or they run at ull speed. Yet we oten

consider the systems not to be running at ull

capacity. The systems are quite oten waiting

or something. To understand perormance

and to describe the perormance aecting

items queuing theory is a good approach. At

minimum a perormance tester and or engineer

should understand queuing in our systems.

Even better is i the tester applies queuing

models and monitors all components. That way

the behavior gets documented much better and

allows us to optimize perormance. This chapter

explains the basics, but we encourage you to

learn more about queuing theory. The book

calledAnalyzing Computer Systems Performance

with Perl::PDQ by Neil J. Gunther does this verywell.

5.2 Queuing theory or computersystems

In our day to day lies much o our time is spent

on waiting. Waiting or the elevator, or the

checkout line and so on. The queuing conceptis also very applicable or computer systems.

When we request a page, queuing happens

throughout the systems. It can happen at the

nano second level at CPUs, but it can also be

seen in as high a level as components as a web

server, a database server etc. The use o the

queuing theory started as early as 1917 or

understanding the availability o telephone

networks. Later it started getting applied or

computer systems.

5.3 A simple queue at thesupermarket

We all are amiliar with queues in supermarkets.

In a very simple model there is one cashier anda waiting line. You have customers arriving and

customers leaving.

The customer that is being served is spending

time on unloading the cart, paying the cashier

and packing up, the service time. The other

customers in the line are waiting. The total time

it takes you rom arrival to departure is called

residence time.

Algorithm 5.1 Residence time = total waiting

time + service time.

5.3.1 Small shop with one checkoutlane

I we draw the queuing model or a small shop

with just one checkout lane we get:

5.3.2 Multiple lanes

Any larger sized supermarket there will be more

checkout lines. When you arrive you pick your

lane. You do this based on the number o other

customers in the lines and how ull their carts

are. You cannot infuence your service time [B]

[B] to keep it simple we will ignore that some

cashiers are very notably slower or aster thanthe others, but you can infuence your waiting

time. And this depends on number o customers

ahead o you and their service time.

Figure 5.1 simple que


8/15


8

PA

GE

This model turns out like this:

5.3.3 One lane, multiple cashiers

And in the larger super markets the cheese

sections oten use a system where you get a

number and wait until your number is up. The

good thing is that you do not need to asses or

yoursel which line will be the astest or you.

Rather than having three queues there is one

queue that is handled by multiple people.

The model would look like this:

5.4 Queues in a server environment

The queues in the supermarket can be

compared to queues in our computer systems.

Lets compare a ew common examples with

the models rom the supermarket.

5.4.1 One CPU system

On a computer system with one (single core) CPU

at the CPU level the queuing model resembles

the model o 5.1 As soon as a processor gets a

task it will do that task, nish, and take on the

next task.

Other processes will have to wait. Anyone

that encoded a video le on older PCs will

understand the eect. As soon as the encoding

process starts, anything else will hardly get

through. Although operating systems do havesome methods to allow or processes to get

some CPU time. Now i the service time o the

processes is small you may hardly notice. The

wait time gets to be so small that the residence

time ends up being small as well.

5.4.2 Web server with multipleprocesses

I we look at a higher level at a system with

multiple web servers serving pages to clients,

the model will resemble the supermarket with

multiple checkout lanes. I there are multiple

web servers, some load balancing service is in

place that will receive the request and give it

to one o the servers. Like with the customer

deciding on the number o preceding customers

and the content o their shopping cart, the loadbalancing service will also make a decision to

which server to hand it and basically that means,

in what queue it ends up.

Interesting to see above is that in this very

simple model you can already see that there are

multiple locations where queues can occur that

cause wait time.

5.4.3 One queue, multiple

processes

And the last model can be seen as well. I there

is one service that can handle multiple requests

and make sure that such requests are dealt with

Figure 5.2 Multiple checkout lanes

Figure 5.3 Drawing numbers

Figure 5.4 Three webservers


9/15


9

PA

GE

by multiple worker processes, such as most

web serving sotware (such as Apache and

Microsots IIS), you will something similar. I the

sotware has many worker processes available

the queue may be small, but there is a centralqueue where the requests arrive. It should be

noted though that i a single server is setup to

have many worker processes, on the system

level there will be queuing at or instance the

CPUs or or memory.

5.5 Understanding queuing at theCPU level

I once had a colleague report that a certain

process was taking almost 100% CPU time. The

administrator dryly responded: good then were

using the CPU eciently. He was right, yet or

good reasons i a server reports 100% CPU time,

we oten view this as a problem.

Like we said beore, a computer component

basically runs at ull speed or does nothing. Iyoure CPU monitoring tool tells you the CPU

is running at 25% it does not mean that it runs

25% o its possible speed. It means that or 25%

o the measured time it was running at 100%

and or 75% it was doing nothing, or as this is

usually called: idle.

I we draw a queuing center or a CPU it looks

like this:

For most processes this waiting line hardly has

any impact. The service time or most requests is

so small that the waiting line is hardly noticeable.Even CPU intensive processes do not send one

large request, but many small ones. As a result,

even i the CPU is busy, processes can get their

requests in.

However as the waiting line is larger, the

residence time or a request becomes large.

Even though the processor did, when it got

around to it, calculates just as ast when it was

less busy. The service time stays the same.

As a result, when the CPU is starting to get more

requests, the chance that requests o other

dierent processes are there increases resulting

in that processes will nd queues at the CPU.

There will however still be moments that the

CPU is idle until the load is so high that it is

continuously occupied.

At that time the waiting time is usually large and

the overall residence time at the CPU is large aswell reducing response times.

Overall it eels as i the CPU is doing its

calculations slower, whilst really we just have to

wait longer.

5.5.1 Multicore or multiprocessors

Server systems usually have more than one CPU.And modern CPUs oten have multiple cores.

Multiple cores eectively work as multiple

CPUs.

At the CPU level, or at the CPU core level, the

queuing center does not change. We just have

multiple centers. Like in the supermarket where

more open checkout lanes do not make the

service time itsel shorter. But the wait time isshorter. In a system where we add equally ast

CPUs, the residence time at the CPU will decrease

i there are enough processes running.

There is a catch though. Not every process

will divide its workload over the extra CPUs.

Suppose you have a ully loaded shopping

cart in the supermarket and there are six open

checkout lanes, your residence time would not

be shorter i you were the only one in the shop.

In that case six open lanes will be just as ast as

one lane. Unless you nd some way to divide

your cart in six smaller carts and

Figure 5.5 Queueing center for a single core CPU


10/15


10

PA

GE

checkout in six lanes. In computer systems this

is the same. A process will be handled by one

processor or core unless the application was

build in such a way that the workload is divided

over multiple processes or threads. For mostprocesses you will not end up in the situation

that one CPU is very busy whereas the others

are idle. But on the process level the queuing

center or single threaded applications look like

this:

I a process ends up in a lane it will be in

that queue and handled ully by that core.

Multithreaded programs get this center:

The process will end up using both CPUs. For

situations where a server has many smaller

processes, there is not much dierence. There

is however a big dierence i we have large CPU

intensive processes. Multithreaded processes

will use as much o all o the CPUs as they can.

You can compare this with unloading your cartdivided over all the checkout lanes. I you are

the only on the shop that is the most eective

and overall residence time will be the smallest.

However i you are in a supermarket other

customers will not like this. And sometimes in

computer systems we dont like this either. When

we or instance have a maintenance batch job

running on the server that is CPU intensive this

will lead to a degradation o the other services.

While making all CPUs chip in will shortenthe time o the batch job, the degradation o

perormance may be a problem. I you make sure

the batch job is not multithreaded or divided

over multiple processes, you will make sure it

will use only one core o a CPU. It will then run

much slower i you have many cores, but you

will have little impact on the other processes.

5.6 Queuing on higher levels

The principals described here work at higher

level in the same way. You can look at a system

on inrastructure level, describing the servers,

network and the same principal applies. I

you or instance have a couple o web servers,

database servers and a load balancer, i you

apply load you should be able to detect queuing

somewhere in the system. Even though possibly

none o the systems are stressed in any way, i

you analyze the processes as they go through

the entire system it must be possible to nd

out the residence times and service times. In

act there is queuing occurring at least in one

location even i it is hard to detect. I there was

no queuing anywhere, each process would

have nished in the blink o an eye.

I you nd a queuing center at a higher level, inmany cases it makes sense to go down a level at

the queue to nd out more. In the example we

used we could or instance see queuing in the

web servers. That would mean we would have

to look into the monitoring o the web servers.

There we would have to nd queuing in or

instance the CPU or memory. This would tell us

much about where the most time is used and

possibly where it can be improved.

5.7 Is there always a queue?

We once tested a batch process that was taking

a long time to complete. We started to run this

process in parallel on one server and we noticed

that i we ran three processes instead o one it

did indeed nish nearly three times as ast. This

is not weird on a multi core CPU or indeed a

multi CPU system, but we were not seeing high

CPU utilization. Now i there had been another

queuing center than the CPU, we suspected not

to see almost the same increase in perormance

Figure 5.6 Single thread or process

Figure 5.7 Multithreaded


11/15


11

PA

GE

as the amount o processes we ran.

So we did a deeper monitoring o the CPU. What

we then noticed is that the process had many

locks. The program had internally a loop with asleep in it. As this program was called through

the batch process in sequence, although it

required a small service time, the sleep time

meant that it waited until completing. As a

result, we got a throttled process which took a

long time to complete but was not continuously

stressing the CPU. The solution to increasing

the perormance was laying in getting the sleep

time out o the program.

What is important to remember is that you

should always nd a queuing center. I there

doesnt seem to be one, continue investigating

until you do. Once ound, you may nd an option

to optimize the sotware. At that moment it is

running at ull speed. So i you can, especially

reporting to technical teams, report where the

queuing center is or where you ound out that

the sotware is not perorming at its maximumperormance. You may enable the team to

greatly improve the perormance.

O course in the case o the batch process,

the throttling may be the desired behavior.

Considering that it was doing its job without

putting much load on the CPU, it also didnt

hinder other unctionality. I you do use

maximum perormance, the batch processwould end earlier but would have led to other

users getting a much slower system.

5.8 Conclusion

Similar to queues in day to day lie, computer

systems behave in a way that mimics these

queues. Computer components dont perorm

at a certain percentage o their capacity. They

are perorming at a certain percentage o time.

The total time it takes a process to complete is

called residence time, but the service time may

be much smaller i it has to wait to be serviced.

Waiting is done in queues. When load and stress

testing using at the very least the concept o

queuing helps us nd where the most time is

spent. O reporting to technical team, it makessense to make a model and show where the

queuing takes place. I you cant nd any

component that has its resources ully utilized,

you should investigate urther until the cause

is ound, especially i you do see decreasing

perormance with added load.


12/15


12

PA

GE

10. For Managers

10.1 Introduction

I you are a typical (test) manager you probably

skipped the rest o the book and went straight

or this chapter. That is OK; this chapter is

here to outline what you should look or in a

perormance team. We want to give you some

handles on how to see i you are getting good

work rom your team without getting in their

way.

10.2 Intake

I you are involved in the selection o the team,

the vision and technical experience o the team

are very important. There are some questions

you can ask to help you evaluate this even i you

are not an expert yoursel:

1. Ask or their vision. A team or tester that will

repeat they had the expensive training with

the expensive tool and years o experience

o using it shows no vision. They only show

that they can show up or class and work.

2. I they show: an approach, mention how

they usually derive test cases, their

approach on monitoring, report,

requirements rom other teams and

expertise desired, etc. they show better

understanding o perormance testing and

not just their tool o choice.

3. Another interesting question is what load

testing tool they preer. There is not best

tool or every situation. It does make sense

that they have a preerence or a tool with

which they have a lot o experience, but orsome application specialized tools may

exist. Sometimes a competitor supports the

protocol you need much better than the

one you are amiliar with. Hire a

perormance test team, not a tool operating

team.

4. Experienced perormance testers have

lessons learned or interesting anecdotes.Eager and enthusiastic testers will oten talk

about them even when not being asked.

10.3 The Plan

Check or the ollowing important items:

1. Planning: the planning should take into

account time to prepare, time to learn how

to record your application, time or analysis,

time or extra test based on the results

(exploratory testing) and time or reporting.

2. The test environment: expect the test team

to want test environment that is as

production like as possible. I they do not,

check i this is explained properly why they

can do without.

3. Cooperation with other teams: i the

perormance team locks itsel up in a room

and has little interaction with other teams

there is a good chance they will not get a

good grip on the business unctionality.

Check i they address that the team will

require time and input rom others in the

organization, particular the other testteams.

4. Monitoring: Check or a strategy that clearly

explains how and what they wish to

monitor.

5. Support: Unless the development team

perorms the test, the perormance test

team should perorm the test, not x issues.

The team and all their tools and hardware

are usually expensive. They should be

able to rely on support o the systems are

not working which prevents them rom


13/15


13

PA

GE

testing. Also remember that specialist may

be called in to help such as DBAs. Check i

they have taken this into account arranged

or this support.

10.4 The progress

So the plan is right, the team seems to know

what they are doing. During this phase it is

important to monitor progress.

1. Deliverables to the team: The biggest

problems or team are usually deliverables

o other teams to the perormance test

team. Test environments are not available,

releases dont work, systems appear to

rely on other systems that everyone orgot

to mention etc. I you want to get your

moneys worth, help the team get what

they need.

2. Finish the ull rst test as soon as possible:

When starting the recording and creatingthe test scripts they will encounter many

hurdles, this is because every application

has its own peculiarities. But as soon as the

rst script is ully working and the rst load

or stress test was done, things get easier.

See i you can get to that stage as quickly as

possible. Especially i not everything has

been delivered. Getting to the rst nished

test will deal with hardest parts or the teamand make sure that when the real deliveries

are there, they can move at better speed.

Focus on a ull test. Not just a working

script, but a script that ran with multiple

virtual users.

3. Check i they analyze on the go: A good test

team will analyze the results as they go.

Even i a test seems to perorm well on

items such as response times, analyzing

may trigger good testers to investigate

something important they hadnt thought

o. And you also dont want to nd out in

the end that perormance is not good

enough when even the rst test already

showed issues.

4. Checks: The results can be very deceiving;the tools may report excellent behavior,

only to nd out later that nothing really

happened. See i they checked that the

tests really perormed the ull business

process they set out to.

Do not accept the line: just wait until the

report.

10.5 The test report

A perormance test should produce a test

report. Here are some tips to assess the report.

1. Short and understandable: Perormance

testers can very easily obuscate sub-par

work by a thick report lled with techno-

babble, graphs and huge tables. Dontaccept that. Tell them you expect a

legible and short report. Graphs are great

or illustrating a point, not or hiding it.

Appendices exist to add literal weight to

a report (and provide inormation or

anyone that wants to draw their own

conclusions based on the data).

2. Resource utilization: It is important torealize that not every good perormance

tester uses the terminology and approach

o this book. So they may not know or use

the term queuing center. But you may

ask them to report on what exactly in an

application was the bottleneck that limited

the perormance. This can be as simple as

a report that says the application was CPU

bound or Memory bound. I they have no

idea, than their monitoring was not good

enough or a technical report. Lack o these

items is oten also a good indication that

their conclusions on response times in


14/15


14

PA

GE

relation to the actual production situation

are not that solid.

3. Modesty: Perormance testing cannot

give exact results or how the applicationwill behave in production. The report

should make the limitations clear. A good

indicator o a serious team is that there is

modesty in conclusions and statements.

Biography

Albert Witteveen has been

working both as an operations

manager and a proessional

tester or nearly two decades

now. The combination o setting

up complex server environments

and proessional testing almost automaticallyled to a specialisation in load and stress testing.

He wrote a practical guide to load and stress

testing which is available at Amazon. Lately his

ocus has been on perormance o applications

hosted with Cloud technology which led to

insight on the changing needs in perormance

testing or Cloud based applications.


15/15

w w w . e u r o s t a r c o n f e r e n c e s . c o m

Join the EuroSTAR CommunityAccess the latest testing news! Our Community strives to provide test

professionals with resources that prove benecial to their day-to-day roles.These resources include catalogues of free on-demand webinars, ebooks,

videos, presentations from past conferences and much more...

Follow us on Twitter @esconfsRemember to use our hash tag #esconfs when tweeting

about EuroSTAR 2013!

Become a fan of EuroSTAR onFacebook

Join ourLinkedIn Group

Add us to your circles

Contribute to the Blog

Check out our free Webinar Archive

Download our latest eBook
https://twitter.com/esconfshttps://twitter.com/esconfshttps://twitter.com/esconfshttps://www.facebook.com/EuroSTARSoftwareTestingConferencehttps://www.facebook.com/EuroSTARSoftwareTestingConferencehttp://www.linkedin.com/groups/EuroSTAR-Software-Testing-Community-1798888http://www.linkedin.com/groups/EuroSTAR-Software-Testing-Community-1798888https://plus.google.com/113080904826107701288/postshttp://www.eurostarconferences.com/bloghttp://www.eurostarconferences.com/community/member/webinar-archivehttp://www.eurostarconferences.com/community/member/ebook-libraryhttp://www.eurostarconferences.com/community/member/ebook-libraryhttp://www.eurostarconferences.com/community/member/webinar-archivehttp://www.eurostarconferences.com/bloghttps://plus.google.com/113080904826107701288/postshttp://www.linkedin.com/groups/EuroSTAR-Software-Testing-Community-1798888https://www.facebook.com/EuroSTARSoftwareTestingConferencehttps://twitter.com/esconfs

1132 performance testing ebooklet albert witt final

Documents