TestingMichael Heron

Introduction• Getting a program running is only the first step.• Getting it working is the next… and this is often more costly.• Having a structured approach to systems testing is one of the

key requirements for building robust and correct software.

Testing• Testing helps to ensure that a program is correct.

• But what do we mean by correct?

• There are a range of criteria that we can apply to any given program to judge its correctness.

Testing

• Correctness:• It must handle all the requested functionality for all

valid data.• It must not generate unexpected behaviour.• It must respond appropriately to all anticipated data.• It must have sufficient error checking to ensure the

program will not produce unexpected results in the case of being given unexpected input.

Testing

• This is a very tall order for most software projects.• Sometimes it’s even a tall order for very simple

projects.

• It is impossible to exhaustively test a program of any significant complexity.• There are just far too many variables.• Not all of the variables are found in the program itself.

Testing

• The goal of testing is to reveal inadequacies in a piece of software.• This is an important point.

• Programmers are often not the best people to write the test cases for their own programs.

• Many large companies have dedicated testing departments to ensure a minimalisation of bias.

Testing• As a developer, it is tempting to write biased test cases that

code is very likely to pass.• A good test however is one that reveals a problem.

• A structured approach to testing involves creating a number of strong test cases for a program and comparing the actual results against the expected results.

Test Sets

• For each test case in a test set, documentation should include:• The exact input data and the conditions under which

the data were inputted.• The purpose of the test.• The exact output expected from the program with the

given test data.

• Once the test is completed, the actual output is compared to the expected output.

Test Strategies• The kind of test cases used in a given test set depends on the

kind of testing strategy adopted.• There are two main families of testing:

• Black box testing• White box testing

• We shall look at these in turn.

Black Box Testing• Black box (or functional) testing is centred around creating a

set of test cases based on sample data that is representative of all data.

• Black box testing requires no knowledge of the program architecture… the program is a ‘black box’ that we cannot see into.• Think of Searle’s Chinese room.

Black Box Testing• In Black Box testing, all we consider are the inputs and the

outputs.• Because it is impossible to exhaustively test a program, we

must choose data that is representative of other data.• We must also consider special cases where errors are

particularly likely.

Black Box Testing• Consider a program designed to multiply two numbers

together.• If it works for 1 * 1, and 2 * 2, you may think it is safe to think

it will work for any positive integer.• But this is not always true.

• Consider the following table

Black Box Testing

Number 1 Number 2 Result

1 1 1

2 2 4

3 3 27

Black Box Testing• It is important that you choose a large enough sample from

any representative set.• It is also important to check for specific kinds of input:

• Boundaries• Maximum and minimums• Invalid input

Black Box Testing• Boundaries are those areas where errors are very likely to

creep in because of the transition between sets.• Between positive and negative• Between valid and invalid

• These areas are often fruitful tests because programmers often make errors in boundary checking:

Examples

• If (a = a) {do_something();

}

vs

if (a == a) {do_something();

}

Examples• If (a > b) {

do_something();}

vs

if (a >= b) {do_something();

}

Examples

• For (int I = 0; I < 100; i++) {do_something_with (i);

}

vs

for (int I = 0; I <= 100; i++) {do_something_with(i);

}

Black Box Testing• It is very easy for these kind of errors to slip into a program.• The ramifications can often be very subtle.• A good test case will take numerous samples from a

representative set and also check at the boundaries of each set.

• Remember – good test cases are those that reveal errors.

White Box Testing

• A counterpoint to black box testing is White Box (or structural) testing.

• In White Box testing we do not care about the matching of expected output to actual output.

• We do care that every statement in a program gets executed at some point.

• White Box testing requires significant knowledge of a program’s architecture.

White Box Testing• In White Box testing, test cases are developed to ensure every

path of execution through a program is touched.• Where there are loops and conditionals, test cases should be

devised to ensure that these are executed at some point with valid test data.

White Box Testing• Consider the following piece of code:

public static void main (String args[]) {int a, b;

a = Integer.parseInt (args[0]);b = Integer.parseInt (args[1]);

if (a < b); {System.out.println (“A is less than B”);

}if (a > b) {

System.out.println (“A is greater than B”);}else {

System.out.println (“A is equal to B”);}

}

White Box Testing

Value of A

Value of B

Expected Actual

1 2 A is less than B

A is less than B

A is equal to B

2 1 A is greater than B

A is less than B

A is greater than B

White Box Testing• White box testing allows us to test every conditional to ensure

that it gets called with the right sets of data.• White box testing is not a replacement for black box testing.• The two approaches work best as complements to each other.

Complementary Approaches• Why use White Box Testing in addition to Black Box Testing?

• Logic errors are usually made when coding for ‘special cases’. The correct execution of these logic paths should be tested.

• Assumptions about execution paths can be incorrect, and white box testing shows this.

• Errors are random, and just as likely to be on an obscure path as a mainstream path.

Complementary Approaches• Why use Black Box Testing in addition to White Box Testing?

• We can check to see how tightly a method conforms to the software specification

• We can ensure reliability of user input.• We cannot assume that just because an execution path is

followed that the logic will be correct. Black Box Testing highlights this.

Unit Testing• Most programs are more complex than the ones shown in this

lecture.• How is it possible to test these using proper test cases?• It is not possible to test most complex programs as a whole.

• It’s just too complicated.

Unit Testing• Instead, we use a technique called unit testing.• We take each of the methods or classes in turn and test them

with all valid sets of data in a harness.• A harness is just a small program that calls the method with

various appropriate sets of data and outputs the returned values.

Unit Testing• This takes advantage of the modularity of modern software

development.• Once a unit has been tested with white and black box testing

and has deemed to be ‘correct’, then it is usually safe to assume the unit will work correctly when called by the program itself.

• However, this is not always true!

Integration Testing• However, we can apply a more structured approach to this

also with Integration Testing.• We start off with a unit that has no dependencies on other

units and test that to ensure that it works.• Then we test any units that make use of that unit.

Integration Testing• Once we have tested these, then we use Integration Testing to

test them together to ensure that the result is correct.• This is a time consuming, but very valuable process.• In this way, a whole program can be broken up into chunks

and then reintegrated with a good degree of confidence.

Summary• Testing is vital in ensuring correctness in programming.• It is however very time consuming and often frustrating.• It is possible to develop a structured approach to testing that

makes debugging an easier process.