Software Testing

Testing

▪ Testing = Executing software with test input and checking whether it

does what we want

▪ Example:

• We have the program “division.exe”

• What we want: the program should print the quotient of two numbers

▪ Let’s test it:

> division.exe 6 3

2

> division.exe 12 3

4

> division.exe 4 0

Exception in line 5: Division by zero

good✓

good✓

bad

Testing (2)

▪ Obviously, we can only write and test a program if we have a complete

idea of what it should do

▪ Different possibilities to specify what a program should do

1. Formal specification:

𝑑𝑖𝑣𝑖𝑠𝑖𝑜𝑛 𝑎, 𝑏 = ቐ

𝑎

𝑏, 𝑖𝑓 𝑏 ≠ 0

𝑒𝑟𝑟𝑜𝑟, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒2. Specification document:

“The program should print the quotient of …”

3. User requirement:

“The user wants a calculator”

Functional vs non-functional tests

▪ Tests can be done to check whether a program satisfies functional

requirements or non-functional requirements

▪ Examples for functional requirements:

• "The program should calculate 𝑎/𝑏"

• "The program should sort a list"

• "The program should print all prime numbers"

• …

▪ Examples for non-functional requirements:

• "The program should have complexity 𝑂(𝑛)“

• “The program should be written in Java”

• "The program should be easy to use"

• "The program should not contain a virus"

• …

Finding input values for tests

▪ To test whether a program fulfills the requirements, we have to test it

with input values from its input domain

• In our “division.exe” example, the input domain is ℤ × ℤ

▪ Do we have to test all possible input values?

• Hopefully not! We expect that if division.exe works for 𝑎 = 5, b = 7it will also work for 𝑎 = 12, 𝑏 = 25

▪ So, our approach to find useful input values for our tests is:

1. Look at the input domain of the programℤ × ℤ

2. Split the input domain into interesting sub-domains

Two sub-domains: 𝑎 ∈ ℤ, 𝑏 ∈ ℤ ∖ {0} and 𝑎 ∈ ℤ, 𝑏 = 0

3. Choose test input values from each sub-domain:

𝑎 ∈ ℤ, 𝑏 ∈ ℤ ∖ {0} → 𝑎 = 5, 𝑏 = 7𝑎 ∈ ℤ, 𝑏 = 0 → 𝑎 = 3, 𝑏 = 0

Quiz

▪ Let‘s say you want to test the following method:

int[] sortArray(int[] array)

a) What is the input domain?

ℤ𝑛 𝑤ℎ𝑒𝑟𝑒 𝑛 ∈ ℕ 𝑖𝑠 𝑡ℎ𝑒 𝑙𝑒𝑛𝑔𝑡ℎ 𝑜𝑓 𝑡ℎ𝑒 𝑎𝑟𝑟𝑎𝑦

b) What are possible sub-domains of the input domain?

1. Empty array (𝑛 = 0)

2. Array with one element (𝑛 = 1)

3. Unsorted array with 𝑛 > 1

4. Array already sorted in ascending order with 𝑛 > 1

5. Array already sorted in descending order with 𝑛 > 1

It’s always good to have disjoint sub-domains that cover the entire input

domain!

We can test a program at different levels

▪ Unit testing = testing a single method

“Does the method give the correct result?”

▪ Module testing = testing a module (in Java: module ≈ class)

“Does the class work correctly? Does it have the required methods?”

▪ Integration testing = testing several modules together

“Do the modules work together correctly? Do all modules have theright methods? Do the modules use the methods of the other modulescorrectly?”

▪ System testing = testing the entire system or program

“Does the system follow the specification?”

▪ Acceptance testing = testing at the customer

“Does software do what the user wants?”

Test levels

▪ Unit testing

▪ Module testing

▪ Integration testing

▪ System testing

▪ Acceptance testing

Tests can be done very

early (as soon as you

have written a method)

and frequently

Tests are difficult: The

software has to be

installed, users have

to “play“ with it,…

Test difficulty If you find a

bug…

Very easy to fix

Very expensive to fix

Who does the tests?

▪ Unit and module tests

The author of the unit or module

▪ Integration tests

• Done by the developer team

▪ System test

• Done by the test team

▪ Acceptance test

• Done by the customer or by people who know what the customer needs (“domain knowledge”)

How to do automated unit testing

Testing is repetitive

▪ Testing is a repetitive task

• You have to test new software that you have written

• You also have to test the software everytime after you have made a change or you have fixed a bug (this is called “Regression testing”)

▪ Conclusion: Try to automate the testing as much as possible, so you can

redo the tests everytime you change the program

• How? Write a program to test your program!

Automated unit testing

▪ Example: We want to test this method in class Main:static int min(int a, int b)

▪ Our test program:// test case 1

int r1=Main.min(3,5);

if(r1!=3) {

System.out.println(“Test 1 failed”);

}

// test case 2

int r2=Main.min(5,3):

if(r2!=3) {

System.out.println(“Test 2 failed”);

}

▪ We can run this test program everytime we have made a change in the method min

▪ This is a lot of code for just three simple tests...• Let’s make life easier: Let’s use the JUnit unit testing tool

Write unit tests with JUnit4

▪ The static assertEquals method does the same as our code on the

previous slide.

▪ There are also methods to test inequality etc. Check the documentation

on https://junit.org/junit4/javadoc/4.12/org/junit/Assert.html

import static org.junit.Assert.*;

public class MainTest {

@org.junit.Testpublic void testFirstNumberLessThanSecondNumber() {

assertEquals("Minimum of 3 and 5 should be 3", 3, Main.min(3,5));}

@org.junit.Testpublic void testFirstNumberGreaterThanSecondNumber() {

assertEquals("Minimum of 5 and 3 should be 3", 3, Main.min(5,3));}

}

How to know whether we have tested our

program sufficiently

Whitebox vs blackbox tests

▪ Our examples with “division.exe”, “sortArray”, “min” are blackbox tests

• Blackbox = We don’t look at the source code of the program to design tests

• Very hard to tell whether we have tested the program sufficiently: How do we know that “division.exe” does not have a fault for 𝑎 =1353885, 𝑏 = −45242 ?

▪ In this course, we will also do whitebox tests

• Whitebox = We can see the source code of the program.

• Seeing the source code can help us to choose good test values!

Testing everything?

▪ Our faulty code:

int min(int a, int b) {

int m;

if(a<b)

m=a;

else

m=a;

return m;

}

▪ Input domain: ℤ × ℤ

▪ Let’s choose 𝑎 = 3, 𝑏 = 5. Result: 4. The program works!?

▪ How do we make sure that we have not forgotten a test case during our

tests?

▪ Obvious truth of testing 1: We can only find a bug in a program if the

program reaches the faulty location with our test input values.

Control Flow Graph and Node Coverage

▪ The control flow graph (CFG) tells us if our tests are complete:

• Test case 1: a=3, b=5

The program will go through nodes 0,1,3 of the CFG

⇒ We have not tested what happens in node 2!

• Test case 2: a=5, a=3

The program will go through nodes 0,2,3 of the CFG

→ We have achieved 100% node coverage

int min(int a, int b) {

int m;

if(a<b)

m=a;

else

m=a;

return m;

}

0

1 2

3return m

m=am=a

a<b a≥b

Control Flow Graph of a for-loop

▪ Remarks:

• Nodes can have more than one statement in one node. In theControl Flow Graph, we are only interested in changes of the controlflow (“branching”).Group of statements without branching = “Basic block”

• Nodes can have no statements. This node only makes a decision.

int sum(int[] x) {

int sum=0;

for(int i=1;i<x.length;i++){

sum+=x[i];

}

return sum;

}

1

2

3return sum

sum+=x[i]

i++

i≥x.lengthi<x.length

0sum=0

i=1

Edge Coverage

▪ If we have 100% node coverage, can we be sure that our program is

correct?

▪ We can cover all statements with the test case a=3, b=5

→ Program goes through nodes 0,1,2 ⇒ 100% node coverage

But: We have not tested the direct path 0→2 !

▪ Instead of 100% node coverage, our goal should be 100% edge coverage

⇒ We have to choose test cases that cover all edges in the control flow

graph.

int min(int a, int b) {

int m=0;

if(a<b) {

m=a;

}

return m;

}

0

1

2 return m

m=a

a<b

m=0

a≥b

Is edge coverage enough?

▪ If we have 100% edge coverage, can we be sure that our program is

correct?

▪ Test case 𝑛 = 0: Covered edges: 0 → 1, 1 → 3. Result is correct: sum=0

▪ Test case 𝑛 = 1: Covered edges: 0 → 1, 1 → 2, 1 → 3. Result is correct:

sum=1

▪ We have covered all edges with our two tests, but the program is wrong

for 𝑛 > 1!

// sum of 1..n

int sum(int n) {

int sum=0;

for(int i=1;i<=n;i++){

sum=i;

}

return sum;

}

1

23return sum

sum=i

i++

i>ni≤n

0sum=0

i=1

Path Coverage

▪ To be sure that our program is correct, we would have to test all

possible paths through the code:

n=0: 0 → 1 → 3, n=1: 0 → 1 → 2 → 1 → 3, n=2: 0 → 1 → 2 → 1 →2 → 1 → 3, ...

▪ In practice, 100% path coverage is often not feasible if your program has

loops. Too many possible paths! In that case, people are just happy with

node coverage or edge coverage.

// sum of 1..n

int sum(int n) {

int sum=0;

for(int i=1;i<=n;i++){

sum=i;

}

return sum;

}

1

23return sum

sum=i

i++

i>ni≤n

0sum=0

i=1

▪ If your goal is to design tests with 100% path coverage, you should be

aware that not all paths are possible in some programs:

▪ In this program, the path 0 → 2 → 4 is not possible

▪ Therefore, when we say “𝑛% path coverage“, we usually mean “𝑛% of

the reachable paths“

Unreachable paths

if(x<0) {

y=1;

}

z=5;

if(x>0) {

z++;

}

0

1

2 z=5

y=1

x>0

x≥0

3z++ x≤0

4

x<0

Quiz

Yes or No:

a) If we have 100% edge coverage, we also have 100% node coverage.

Yes: 100% edge coverage ⇒ 100% node coverage

b) In a program without loops, all existing paths in the CFG can be

checked.

No. The program can have unreachable paths.

Coverage test on Inginious

▪ Inginious uses the tool JaCoCo for coverage tests

▪ JaCoCo calculates two metrics:

• JVM bytecode instruction coverage: this is node coverage.

Note: JVM bytecode instruction ≠ Java statementFor example, the Java statement

a=b+2;

is compiled to four bytecode instructions!

• Branch coverage: this is edge coverage for if and switch statements

Software testing = doing coverage tests ?

▪ The goal of testing is not to reach 100% coverage!

• Coverage is not a goal. Coverage is a tool that helps you to definetest cases

▪ What to do in practice:

1. Identify the input domain

2. Divide the input domain into blocks

3. Create test cases with test values that come from the blocks

4. Run tests

5. Check the coverage. Not happy? Go back to step 1 or 2.

Practical issues

Write code that is easy to unit test

▪ We should write code such that we can easily provide test values

▪ We should write code such that we can easily check the result

class BadClass {

int method(int v1) {

something using v1;

int v2=...

something using v2;

return result;

}

}

class GoodClass {

int method1(int v1) {

something using v1;

int v2=...

return v2;

}

int method2(int v2) {

something using v2;

return result;

}

}

Difficult to test whether v2 is correct.

Add a print(v2) statement?

You can easily check the

results for v2

Difficult to test this part for different v2 values

You can easily provide your

own test values

Testing methods that need an object

▪ If we test non-static methods, we have to work with objects

▪ We want to test the add method for

• Test case 1: empty lists

• Test case 2: non-empty lists

class LinkedList {

Element head=null;

void add(Element e) {

if(head!=null) {

e.next=head;

}

head=e;

}

}

class Element {

Element next;

int value;

Element(int value) {

this.value=value;

}

}

Test case 1: Add element to empty list

class LinkedList {

Element head=null;

void add(Element e) {

if(head!=null) {

e.next=head;

}

head=e;

}

}

class Element {

Element next;

int value;

Element(int value) {

this.value=value;

}

}

@Testpublic void addToEmptyList() {

LinkedList list=new LinkedList();Element e=new Element(2);list.add(e);assertSame("Empty list must contain element after add",list.head, e);

}

assertSame(...) checks

whether two references

point to the same object

Test case 2: Add element to non-empty list

class LinkedList {

Element head=null;

void add(Element e) {

if(head!=null) {

e.next=head;

}

head=e;

}

}

class Element {

Element next;

int value;

Element(int value) {

this.value=value;

}

}

@Testpublic void addToNonEmptyList() {

LinkedList list=new LinkedList();Element e1=new Element(2);list.add(e1);Element e2=new Element(3);list.add(e2);

assertSame(“Must contain new element as head after add", list.head, e2);assertSame(“Old element must be second element after add",list.head.next,e1);

}

More JUnit features

▪ Sometimes, you want to test whether a method throws an exception:

▪ In JUnit:

▪ You can also give a time limit :

void myMethod(int i) {

if(i<0)

throw new NumberFormatException();

}

@Test(expected = NumberFormatException.class)

public void myTest() {

MyClass.myMethod(-1);

}

@Test(timeout = 1000) // <- 1000 milliseconds

public void myTest() {

...

}

More JUnit features (2)

▪ Sometimes there are things that you want to do before or after each

test

▪ Example:

• You have written a program that writes a .png file

• You want that the png file is deleted after a test

• In JUnit:

@After

public void after() {

// delete the file here

...

}

▪ See

https://www.tutorialspoint.com/junit/junit_execution_

procedure.htm

Complex methods are difficult to test

▪ In this course, we focus on simple unit testing

▪ In reality, testing a method can be very complex:

int myMethod1(int a) {

int b=myMethod2(a);

return myMethod3(b);

}

▪ What is the CFG of this code?

• CFG of myMethod1 = CFG of myMethod2 + CFG of myMethod3 !

▪ Simple methods are easier to test

Be careful with ”Hidden” if

▪ Be careful with statements that look like basic statements...

▪ Example: r = a/b;

If b can be zero, this is what is actually happening in the program:

if(b==0)

throw new DivisionByZeroException()

else

r=a/b

▪ Example: name = animal.getName();

If animal can be null, this is what is actually happening in the program:if(animal==null)

throw new NullPointerException()

else

name = animal.getName()