How much testing is enough?

• Suppose that we have a function

float divide (float x, float y)

• This function performs 64-bit IEEE 754

compliant floating point division

• Testing it requires 264 test cases (clearly not

feasible)

• From a practical and economic perspective, we

know that exhaustive testing is not normally

possible, so we ask:

– Which pieces of software should we test?

– Which test cases should we pick?

Coverage

• How completely a set of test cases (a test suite)

exercises the capabilities of a piece of software

is called its coverage.

– Each line of code should be executed at least once • We saw this with "white box" testing

– One test case should be constructed from each

specified requirement

• We need to use testing techniques that reduce

the number of test cases while at the same time

providing the broadest testing coverage with the

least effort.

Black Box Testing

• Focuses on the functional requirements of the software

• Complement to white box testing

• Goal is to find – Incorrect or missing functions

– Interface errors

– Errors in data structures or external database access

– Behavioral or performance errors

– Initialization and termination errors

• Want each test to – Reduce the number of additional tests needed for

reasonable testing

– Tell us about presence or absence of a class of errors

Black Box Testing - Techniques

• Functional Analysis – Analyze the expected behavior of the system according to its

functional specification

– Generate a test procedure for each of the possible usage scenarios

• Equivalence Partitioning – Identifies ranges of input and initial conditions that are expected

to produce the same result

– It is adequate to use only a single representative of the equivalence class

• Boundary Value Analysis – Based on experience / heuristics

• Testing boundary conditions of equivalence classes is more effective

– Choose input boundary values as equivalence classes representatives

– Choose inputs that invoke output boundary values

Functional System Testing

• Testing of an application to determine that it

provides all of the behaviors specified in the

system requirements

• Along the way --

– Testing of completed increments that provide some

degree of end-user functionality

– Test that end user functionality that is present

• Search for defects

– defects are variances between the actual operation of

the system and the requirements for the system

• System is treated as a black box

Technique: Functional Analysis

• Analyze the expected behavior of the system

according to its functional specification

• Generate a test procedure for each of the

possible usage scenarios

– Corresponds to use case scenarios

– Analyze how a change in one part of the system

affects other parts

– “Grand tour” test cases: the result of one test case

produces the data that is the input to the next test

case

Example 1: Functional Analysis (i)

Use Case: Dismiss Parking Ticket 1. User calls for deletion of the chosen Parking Ticket

2. System prompts the User for confirmation

3. User confirms

4. System requests the TicketDB to delete the Parking

Ticket from the Offendor's record

5. OffendorsDB confirms that the Parking Ticket has been

deleted

6. System allows the User to look up a new Violation as

described in the "Lookup Violation" Use Case

Data For Testing

Name: test.db

Description: Violation records design for validating “Violation

Lookup”

Violation ID Offendor’s

First Name

Offendor’s

Last Name

Issuing

Person ID

243567 Elizabeth Jones 8700342

237812 David Smith 6386541

264683 David Long 1346329

255245 Lee Jeans 8245731

000345 Lazarus Long 8700342

… … … …

Example 1: Functional Analysis (ii)

Test Case ID: 00105

Preconditions: 1: System initialized with test.db

2: Violation 264683 displayed in the “Lookup Violation”

dialog

Related Use Cases: “Lookup Violation”, “Remove Ticket”

Action Expected Result

Press “Delete” button Confirmation dialog displayed

Press the “Yes” button “Lookup Violation” dialog is displayed

Enter “264683” in the

“Violation ID” text field and

press the “Search” button

A message dialog stating that violation

“264683” is not stored in the system

Test Results

Passed

Failed

Actual results:

Defect Diagnosis

Example 1: Functional Analysis (iii)

Test Case ID: 00105

Preconditions: 1: System initialized with test.db

2: Violation 264683 displayed in the “Lookup

Violation” dialog

Related Use Cases: “Lookup Violation”, “Remove Ticket”

Action Expected Result

Press “Delete” button Confirmation dialog displayed

Press the “Yes” button “Lookup Violation” dialog is

displayed

Enter “264683” in the

“Violation ID” text field and

press the “Search” button

A message dialog stating that

violation “264683” is not stored in

the system

Test Results

Passed

Failed

Actual results:

Defect Diagnosis

How do we know that violation

264683 is in the system?

Verify effects

of change

Fill in when

test case is

executed

Run a query on

OffendorsDB too?

Can a tester diagnose

the cause of a defect?

Technique: Equivalence Partitioning

• Identifies ranges of input and initial conditions

that are expected to produce the same result

• A group of test cases form an equivalence

class if:

– They test the same feature/scenario

– If one test reveals a fault, the other ones (probably)

will too

– If a test does not reveal a fault, the other ones

(probably) will not either

• It is adequate to use only a single representative

of the equivalence class

Sample Equivalence Classes

• Valid data – User supplied commands

– Responses to system prompts

– File names

– Computational data • Physical parameters

• Bounding values

• Initial values

– Output data formatting

– Responses to error messages

– Graphical data (e.g. mouse picks)

• Invalid data – Data outside program bounds

– Physically impossible data

– Proper data supplied in the wrong place

Example 2: Equivalence Partitioning (i)

• Input value specification for "Lookup Violation" form:

Field Name Valid Values

Violation ID [0-9] {0, 9}

Offendor's First Name [a-zA-Z] {0, 10}

Offendor's Last Name [a-zA-Z] {0, 10}

Example 2: Equivalence Partitioning (ii)

Field

Valid

Equivalent

Classes

Valid

Representative

Values

Invalid

Equivalent

Classes

Invalid

Representative

Values

Violation ID

Known

Violation

00243567 ID <0 or ID>

999999999

-12

1234567890

Unknown

Violation

9912367 Non-numeric

ID

23@41.C

Empty “”

Offendor’s

first name

Unknown

violation

Donald Character >

10

Veryverylongna

me

Single known

violation

Elizabeth Invalid

character

adm@moore

Many known

violations

David

Empty “”

… … … … …

Example 2: Equivalence Partitioning (iii)

• Considering this analysis, the number of test cases to choose from can be reduced to: – (3+2) (for the violation ID) times

– (4+2) (for the offendor's first name) times

– (4+2) for the offendor's last name)

– This makes a total of 180

• This is a lot, but it can be further reduced – Consider only one invalid field per test case

– Prioritize by importance of use case

– Base on most frequent input

– Eliminate infeasible test cases

– etc.

Guidelines:

Generating Tests using Equivalence Partitioning:

• If an input condition specifies a range, one valid and two invalid equivalence classes are defined

• If an input condition requires a specific value, one valid and two invalid conditions are defined

• If an input condition specifies a member of a set, one valid and one invalid equivalence class are defined

• If an input condition is Boolean, one valid and one invalid class are defined

Pick test cases so that the largest number of attributes of an equivalence class are exercised at once

Technique: Boundary Value Analysis

• More errors occur at the boundaries of the input domain than in the middle

• This technique leads to choice of test cases at bounding values

• This technique can (in some sense) be regarded as an extension of equivalence class partitioning – Choose one (or more) arbitrary values(s) in each

equivalence class

– Choose valid values exactly on lower and upper boundaries of equivalence classes

– Choose invalid values immediately above and below each boundary (if this is possible)

Guidelines: Boundary Value Analysis

1. If an input condition specifies a range bounded by

values a and b, test cases should be designed with a

and b and values just above and just below a and b

2. If an input condition specifies a number of values, test

cases should be developed that exercise the min and

max numbers as well as values just above and below

the max and min

3. Apply (1) and (2) to output conditions as well – design

test cases that will produce the largest and smallest

possible output values, for example

4. If internal program data structures have prescribed

boundaries, be sure to design test cases to exercise

the data structure at its boundaries.

Example: Gym or Fitness Club Fees

Day Sunday - Thursday Friday - Saturday

Guest

status

Visitor Member Student Visitor Member Student

Age

(years)

Admission fee

[0, 16) 25 10 20 35 10 30

[16, 60) 50 25 45 70 25 65

[60, 120] 35 15 30 50 15 45

Specification

Example: Gym or Fitness Club

Equivalence Classes

& Boundaries

Field Valid

equivalent

classes

Valid

representative

values

Invalid

equivalent

classes

Invalid

representative

values

Day Sun - Thu Mon, Sun, Thu

Fri - Sat Fri, Sat

Guest

status

Visitor Visitor

Member Member

Student Student

Age [0, 16) 2, 0, 15 Non-numeric

value

14@a

[16, 60) 34, 16, 59 Age < 0 or

Age > 120

-1, 121

[60, 120] 100, 60, 120

A combo box is used

for choosing the day

and guest status

Test case ID Day Guest status Age Result

1 Mon Visitor 2 25

2 Fri Member 34 25

3 Mon Student 100 30

4 Sun Visitor 0 25

5 Sat Member 16 10

6 Thu Student 60 30

7 Sun Member 15 10

8 Sat Student 120 45

9 Thu Visitor 59 50

10 Mon Member 14@a Invalid age

11 Fri Student -1 Invalid age

12 Fri Visitor 121 Invalid age

valid

valid

(boundary)

invalid

Example: Gym or Fitness Club

Test Suite