© andrew irelandsoftware design f28sd2 software design (f28sd2): life-cycle perspective - part 2...

© Andrew IrelandSoftware Design F28SD2

Software Design (F28SD2):Life-Cycle Perspective - Part 2

Andrew IrelandSchool of Mathematical & Computer Sciences

Heriot-Watt UniversityEdinburgh


Outline

• A strategy for dynamic analysis - testing

• Designing test cases – specification and construction of test case data


Testing (Dynamic Analysis)

• Testing involves system operation, i.e. code execution.

• Testing that does not exploit the internal structure of the code is known as specification-based, functional, black-box, behavioural, requirement-based, …

• Testing that does exploit the internal structure of the code is known as structure-based, structural, white-box, glass-box, coverage based, …

• Testing will typically involve the construction of additional code to facilitate the testing process.


A Classification of Test Case Design Techniques

The aim of design techniques is to provide a systematic basis for developing effective test cases, i.e. test cases that will provide a high yield in terms of defect detection.

Specification-based Structure-based Experience-based

Equivalence analysis Statement coverage Error guessing

Boundary value analysis

Decision coverage

… Condition/Decision coverage

… …


A Strategy for Dynamic Analysis

1. Use specification and experience-based techniques to design initial test cases.

2. Use coverage metrics to determine the effectiveness of the initial test cases.

3. Use structure-based techniques to increase coverage if judged necessary.

Note: Both positive (desirable behaviours) and negative (undesirable behaviours) test cases should be used.


Error Guessing• Error guessing is driven by experience and will

typically cover both functional and structural perspectives.

• To illustrate, given a software component to test that exploits dynamic storage allocation, then an experienced tester will typically focus upon the correct use of resource allocation and deallocation, e.g. seek out memory leakage defects.

• By its very nature, error guessing is hard to teach. However, check lists designed around language features and related defects can play a useful role in the transfer of experience.


Structural/Functional Balance

• While internal knowledge can greatly simplify the task of dynamic testing - over reliance on the actual code should be avoided, i.e. risk of designing test cases that demonstrate that the code does what the code does!

• The primary source of test cases should be the functional specification. The process of designing tests based upon functional specification will typically expose many defects before coding gets underway.


Equivalence Partitioning

• A technique that partitions the space of possible program inputs (and potentially outputs) into a finite set of equivalence classes – a set of data values for which the program will perform the same function

• For example, consider a program that accepts numbers, and doubles the odd numbers – this suggests partitioning the inputs into two equivalences classes, i.e. odd and even


Equivalence Classes (EC)

Equivalence partitioning can significantly prune the space of possible test cases - assuming the equivalence partitioning is performed correctly!

0

8

…

4 6

2

1

9

…

5 7

3

Odd EC Even EC

14


Valid and Invalid ECs• Parts of the input space which a program should

accept give rise to Valid Equivalence Classes• Parts of the input space which a program should

accept give rise to Invalid Equivalence Classes

0

-4

…

-2 -3

-1

1

5

…

3 4

2

Positive Not positive


Some Guidance for Partitioning

• Ranges: if a requirement specifies a range then three ECs are required, one valid and two invalid. Note that there are always implementation dependent boundaries, e.g. 16-bit integer gives rise to a partition bounded by 32767 and -32768.

• Size: if a requirement specifies a size then three ECs are required, one valid and two invalid, e.g. a four character password gives rise to a valid EC which contains all four character alpha numeric sequences, and two invalid ECs containing less-than and greater-than four character alpha numeric sequences

• Sets: if a requirement specifies a set then two equivalence classes are required, one valid (member) and one invalid (non-member).


Boundary Value Analysis


Test Case Effectiveness

• When have we tested enough?• Coverage metrics provide one of the most widely used

approaches to judging the effectiveness of testing.• A coverage metric (CM) represents the ratio of metric items

executed at least once (EM) to the total number of metric items (TM): CM = EM/TM

• For example, if the code under test contains 1000 statements (TM), are your test cases execute 900 of the statements (EM), then you have achieved 90% statement coverage, i.e. CM = 0.9

• Coverage metrics are provide a criteria for specifying and constructing structural test cases ...


Statement CoverageDefinition: Every statement is executed at least once.Advantage: understandable & relatively simple to achieve 100% coverage.Disadvantage: Even with 100%, statement coverage is weak, e.g.

1. int* ptr = NULL; 2. if (flag) ptr = &result;

3. *ptr = 666;

100% statement coverage only requires that the execution path including flag set to true is achieved.

Note that if flag is set to false, then the code crashes! That is, statement coverage misses the bug!

statements flag

lines 1, 2, 3 true


Decision Coverage• Definition: Every statement and every decision (if-

statement, while-statement, etc) is executed at least once.• Advantage: easy to understand and relatively simple to

achieve 100% coverage and overcomes the deficiency of statement coverage.

• Disadvantage: 100% coverage does not mean that the deeper logical structure of the decision point will be adequately explored, e.g.

if (condition1 || condition2) statement1; else statement2;

case (condition1 || condition2) branch

1 true true

2 false false

Note: no guarantee that the decision point will be tested when condition2 is true …


Condition/Decision Coverage• Definition: Every statement and every branch is

executed at least once, with each condition within each branch taking on all possible values at least once.

• Advantage: Addresses the deficiency of decision coverage to some extent, i.e. explores the deeper structure of the decision point to some extent.

• Disadvantage: Can still leave untested combinations of conditions, e.g.

if (condition1 && (condition2 || condition3)) ...

case condition1 condition2 condition3 branch

1 true true true true

2 false false false false


Multiple Condition Coverage


Multiple Condition Coverage

if (condition1 && (condition1 || condition2)) …


1 true true true true

2 true true false true

3 true false true true

4 true false false false

5 false true true false

6 false true false false

7 false false true false

8 false false false false


Modified Condition/Decision Coverage (MC/DC)

• Definition: Every condition within a decision is executed to shown that it can independently effect the outcome of the decision.

• Advantage: A compromise requiring fewer cases than multiple condition coverage, i.e. minimum of N+1 and a maximum of 2N cases where N denotes the number of Boolean operands involved.

• Disadvantage: Time consuming to calculate. No real gains when N is small.


MC/DC Coverage







Cases 2 and 6 show independence of condition1•Value of condition1 changed•Values of condition2 and condition3 fixed•Both true and false branches executed


MC/DC Coverage









MC/DC Coverage







• Cases 2 and 6 show independence of condition1 • Cases 2 and 4 show independence of condition2 • Cases 3 and 4 show independence of condition3


Test Case Specifications to Test Cases• Construct test cases for the following if-statement

using Condition/Decision coverage:

if !(closed) && ((n < max) || flag) …• Where max denotes the constant 100, closed, flag are

Boolean variables and n is an integer variable.• Test case specification:

• Test cases, e.g.• TC1: closed == false, n == 99, flag true• TC2: closed == true, n == 100, flag == false

case closed (n < max) flag branch

TC1 false true true true

TC2 true false false false


Summary • Learning outcomes:

– A strategy for testing– An understanding of the complementary nature of specification-based

and structure-based testing methods– Equivalence partitioning & boundary value analysis– Coverage metrics for structure-based testing

• Recommended reading:– “Software Testing: An ISTQB-ISEB Foundation Guide” (2nd

Edition). BCS, 2010.– “Software Testing in the Real World”, E. Kit, Addison-Wesley,

1995.– “Black Box Testing”, B. Beizer. Wiley & Sons, 1995.– “Applicability of modified condition/decision coverage to

software testing'', J.J. Chilensky and S.P. Miller. Software Engineering J. 1994.


Summary • Recommended reading:

– “Software Testing: An ISTQB-ISEB Foundation Guide” (2nd Edition). BCS, 2010.

– “Software Testing in the Real World”, E. Kit, Addison-Wesley, 1995.

– “Black Box Testing”, B. Beizer. Wiley & Sons, 1995.– “Applicability of modified condition/decision coverage to

software testing'', J.J. Chilensky and S.P. Miller. Software Engineering J. 1994.

© andrew irelandsoftware design f28sd2 software design (f28sd2): life-cycle perspective - part 2...

Documents