defect testing testing programs to establish the presence of system

©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 1

Defect testing

l Testing programs to establishthe presence of system defects


Objectives

l To understand testing techniques that are gearedto discover program faults

l To introduce guidelines for interface testing

l To understand specific approaches to object-oriented testing

l To understand the principles of CASE toolsupport for testing


Topics covered

l Defect testing

l Integration testing

l Object-oriented testing

l Testing workbenches


The testing process

l Component testing• Testing of individual program components

• Usually the responsibility of the component developer (exceptsometimes for critical systems)

• Tests are derived from the developer’s experience

l Integration testing• Testing of groups of components integrated to create a system or

sub-system

• The responsibility of an independent testing team

• Tests are based on a system specification


Testing phases

Componenttesting

Integrationtesting

Software developer Independent testing team


Defect testing

l The goal of defect testing is to discover defects inprograms

l A successful defect test is a test which causes aprogram to behave in an anomalous way

l Tests show the presence not the absence ofdefects


l Only exhaustive testing can show a program isfree from defects. However, exhaustive testingis impossible

l Tests should exercise a system's capabilitiesrather than its components

l Testing old capabilities is more important thantesting new capabilities

l Testing typical situations is more important thanboundary value cases

Testing priorities


l Test data Inputs which have been devised totest the system

l Test cases Inputs to test the system and thepredicted outputs from these inputs if thesystem operates according to its specification

Test data and test cases


The defect testing process

Design testcases

Prepare testdata

Run programwith test data

Compare resultsto test cases

Testcases

Testdata

Testresults

Testreports


Black-box testing

l An approach to testing where the program isconsidered as a ‘black-box’

l The program test cases are based on the systemspecification

l Test planning can begin early in the softwareprocess


Black-box testing

Ie

Input test data

OeOutput test results

System

Inputs causinganomalousbehaviour

Outputs which revealthe presence ofdefects


Equivalence partitioning

l Input data and output results often fall intodifferent classes where all members of a class arerelated

l Each of these classes is an equivalence partitionwhere the program behaves in an equivalent wayfor each class member

l Test cases should be chosen from each partition



System

Outputs

Invalid inputs Valid inputs


l Partition system inputs and outputs into‘equivalence sets’• If input is a 5-digit integer between 10,000 and 99,999,

equivalence partitions are <10,000, 10,000-99, 999 and >10, 000

l Choose test cases at the boundary of thesesets• 00000, 09999, 10000, 99999, 10001



Equivalence partitions

Between 10000 and 99999Less than 10000 More than 99999

999910000 50000

10000099999

Input values

Between 4 and 10Less than 4 More than 10

34 7

1110

Number of input values


Search routine specification

procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;

Pre-condition-- the array has at least one elementT’FIRST <= T’LAST

Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)

or -- the element is not in the array( not Found and

not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))


l Inputs which conform to the pre-conditions

l Inputs where a pre-condition does not hold

l Inputs where the key element is a member ofthe array

l Inputs where the key element is not a memberof the array

Search routine - input partitions


Testing guidelines (sequences)

l Test software with sequences which have only asingle value

l Use sequences of different sizes in different tests

l Derive tests so that the first, middle and lastelements of the sequence are accessed

l Test with sequences of zero length


Search routine - input partitionsArray ElementSingle value In sequenceSingle value Not in sequenceMore than 1 value First element in sequenceMore than 1 value Last element in sequenceMore than 1 value Middle element in sequenceMore than 1 value Not in sequence

Input sequence (T) Key (Key) Output (Found, L)17 17 true, 117 0 false, ??17, 29, 21, 23 17 true, 141, 18, 9, 31, 30, 16, 45 45 true, 717, 18, 21, 23, 29, 41, 38 23 true, 421, 23, 29, 33, 38 25 false, ??


l Sometime called white-box testing

l Derivation of test cases according to programstructure. Knowledge of the program is used toidentify additional test cases

l Objective is to exercise all program statements(not all path combinations)

Structural testing


White-box testing

Componentcode

Testoutputs

Test data

DerivesTests

Binary search (Java)

class BinSearch {

// This is an encapsulation of a binary search function that takes an array of// ordered objects and a key and returns an object with 2 attributes namely// index - the value of the array index// found - a boolean indicating whether or not the key is in the array// An object is returned because it is not possible in Java to pass basic types by// reference to a function and so return two values// the key is -1 if the element is not found

public static void search ( int key, int [] elemArray, Result r ){

int bottom = 0 ;int top = elemArray.length - 1 ;int mid ;r.found = false ; r.index = -1 ;while ( bottom <= top ){

mid = (top + bottom) / 2 ;if (elemArray [mid] == key){

r.index = mid ;r.found = true ;return ;

} // if partelse{

if (elemArray [mid] < key)bottom = mid + 1 ;

elsetop = mid - 1 ;

}} //while loop

} // search} //BinSearch


l Pre-conditions satisfied, key element in array

l Pre-conditions satisfied, key element not inarray

l Pre-conditions unsatisfied, key element in array

l Pre-conditions unsatisfied, key element not inarray

l Input array has a single value

l Input array has an even number of values

l Input array has an odd number of values

Binary search - equiv. partitions


Binary search equiv. partitions

Mid-point

Elements < Mid Elements > Mid

Equivalence class boundaries


Binary search - test cases

Input array (T) Key (Key) Output (Found, L)17 17 true, 117 0 false, ??17, 21, 23, 29 17 true, 19, 16, 18, 30, 31, 41, 45 45 true, 717, 18, 21, 23, 29, 38, 41 23 true, 417, 18, 21, 23, 29, 33, 38 21 true, 312, 18, 21, 23, 32 23 true, 421, 23, 29, 33, 38 25 false, ??


Path testing

l The objective of path testing is to ensure that theset of test cases is such that each path through theprogram is executed at least once

l The starting point for path testing is a programflow graph that shows nodes representingprogram decisions and arcs representing the flowof control

l Statements with conditions are therefore nodes inthe flow graph


l Describes the program control flow. Each branchis shown as a separate path and loops are shownby arrows looping back to the loop conditionnode

l Used as a basis for computing the cyclomaticcomplexity

l Cyclomatic complexity = Number of edges -Number of nodes +2

Program flow graphs


l The number of tests to test all controlstatements equals the cyclomatic complexity

l Cyclomatic complexity equals number ofconditions in a program

l Useful if used with care. Does not implyadequacy of testing.

l Although all paths are executed, all combinationsof paths are not executed

Cyclomatic complexity

Binary search flow graph

1

2

3

4

65

7

while bottom <= top

if (elemArray [mid] == key

(if (elemArray [mid]< key8

9

bottom > top


l 1, 2, 3, 8, 9

l 1, 2, 3, 4, 6, 7, 2

l 1, 2, 3, 4, 5, 7, 2

l 1, 2, 3, 4, 6, 7, 2, 8, 9

l Test cases should be derived so that all of thesepaths are executed

l A dynamic program analyser may be used tocheck that paths have been executed

Independent paths


Integration testing

l Tests complete systems or subsystems composedof integrated components

l Integration testing should be black-box testingwith tests derived from the specification

l Main difficulty is localising errors

l Incremental integration testing reduces thisproblem


Incremental integration testing

T3

T2

T1

T4

T5

A

B

C

D

T2

T1

T3

T4

A

B

C

T1

T2

T3

A

B

Test sequence1

Test sequence2

Test sequence3


Approaches to integration testing

l Top-down testing• Start with high-level system and integrate from the top-down

replacing individual components by stubs where appropriate

l Bottom-up testing• Integrate individual components in levels until the complete

system is created

l In practice, most integration involves acombination of these strategies


Top-down testing

Level 2Level 2Level 2Level 2

Level 1 Level 1Testingsequence

Level 2stubs

Level 3stubs

. . .


Bottom-up testing

Level NLevel NLevel NLevel NLevel N

Level N–1 Level N–1Level N–1

Testingsequence

Testdrivers

Testdrivers


Tetsing approaches

l Architectural validation• Top-down integration testing is better at discovering errors in

the system architecture

l System demonstration• Top-down integration testing allows a limited demonstration at

an early stage in the development

l Test implementation• Often easier with bottom-up integration testing

l Test observation• Problems with both approaches. Extra code may be required to

observe tests


l Takes place when modules or sub-systems areintegrated to create larger systems

l Objectives are to detect faults due to interfaceerrors or invalid assumptions about interfaces

l Particularly important for object-orienteddevelopment as objects are defined by theirinterfaces

Interface testing


Interface testingTestcases

BA

C


Interfaces types

l Parameter interfaces• Data passed from one procedure to another

l Shared memory interfaces• Block of memory is shared between procedures

l Procedural interfaces• Sub-system encapsulates a set of procedures to be called by

other sub-systems

l Message passing interfaces• Sub-systems request services from other sub-systems


Interface errors

l Interface misuse• A calling component calls another component and makes an

error in its use of its interface e.g. parameters in the wrong order

l Interface misunderstanding• A calling component embeds assumptions about the behaviour

of the called component which are incorrect

l Timing errors• The called and the calling component operate at different speeds

and out-of-date information is accessed


Interface testing guidelines

l Design tests so that parameters to a calledprocedure are at the extreme ends of their ranges

l Always test pointer parameters with null pointers

l Design tests which cause the component to fail

l Use stress testing in message passing systems

l In shared memory systems, vary the order inwhich components are activated


Stress testing

l Exercises the system beyond its maximum designload. Stressing the system often causes defects tocome to light

l Stressing the system test failure behaviour..Systems should not fail catastrophically. Stresstesting checks for unacceptable loss of service ordata

l Particularly relevant to distributed systemswhich can exhibit severe degradation as anetwork becomes overloaded


l The components to be tested are object classesthat are instantiated as objects

l Larger grain than individual functions soapproaches to white-box testing have to beextended

l No obvious ‘top’ to the system for top-downintegration and testing

Object-oriented testing


Testing levels

l Testing operations associated with objects

l Testing object classes

l Testing clusters of cooperating objects

l Testing the complete OO system


Object class testing

l Complete test coverage of a class involves• Testing all operations associated with an object

• Setting and interrogating all object attributes

• Exercising the object in all possible states

l Inheritance makes it more difficult to designobject class tests as the information to be tested isnot localised


Weather station object interface

l Test cases are needed for alloperations

l Use a state model to identifystate transitions for testing

l Examples of testingsequences

• Shutdown ♦ Waiting ♦ Shutdown

• Waiting ♦ Calibrating ♦ Testing ♦Transmitting ♦ Waiting

• Waiting ♦ Collecting ♦ Waiting ♦Summarising ♦ Transmitting ♦ Waiting

identifier

reportWeather ()calibrate (instruments)test ()startup (instruments)shutdown (instruments)

WeatherStation


Object integration

l Levels of integration are less distinct in object-oriented systems

l Cluster testing is concerned with integrating andtesting clusters of cooperating objects

l Identify clusters using knowledge of the operationof objects and the system features that areimplemented by these clusters


Approaches to cluster testing

l Use-case or scenario testing• Testing is based on a user interactions with the system

• Has the advantage that it tests system features as experienced byusers

l Thread testing• Tests the systems response to events as processing threads

through the system

l Object interaction testing• Tests sequences of object interactions that stop when an object

operation does not call on services from another object


Scenario-based testing

l Identify scenarios from use-cases and supplementthese with interaction diagrams that show theobjects involved in the scenario

l Consider the scenario in the weather stationsystem where a report is generated


Collect weather data:CommsController

request (report)

acknowledge ()report ()

summarise ()

reply (report)

acknowledge ()

send (report)

:WeatherStation :WeatherData


Weather station testing

l Thread of methods executed• CommsController:request ♦ WeatherStation:report ♦

WeatherData:summarise

l Inputs and outputs• Input of report request with associated acknowledge and a final

output of a report

• Can be tested by creating raw data and ensuring that it issummarised properly

• Use the same raw data to test the WeatherData object


Testing workbenches

l Testing is an expensive process phase. Testingworkbenches provide a range of tools to reducethe time required and total testing costs

l Most testing workbenches are open systemsbecause testing needs are organisation-specific

l Difficult to integrate with closed design andanalysis workbenches


A testing workbench

Dynamicanalyser

Programbeing tested

Testresults

Testpredictions

Filecomparator

Executionreport

Simulator

Sourcecode

Testmanager Test data Oracle

Test datagenerator Specification

Reportgenerator

Test resultsreport


Tetsing workbench adaptation

l Scripts may be developed for user interfacesimulators and patterns for test data generators

l Test outputs may have to be prepared manuallyfor comparison

l Special-purpose file comparators may bedeveloped


Key points

l Test parts of a system which are commonly usedrather than those which are rarely executed

l Equivalence partitions are sets of test cases wherethe program should behave in an equivalent way

l Black-box testing is based on the systemspecification

l Structural testing identifies test cases which causeall paths through the program to be executed


Key points

l Test coverage measures ensure that all statementshave been executed at least once.

l Interface defects arise because of specificationmisreading, misunderstanding, errors or invalidtiming assumptions

l To test object classes, test all operations,attributes and states

l Integrate object-oriented systems around clustersof objects

defect testing testing programs to establish the presence of system

Documents