09 testing
DESCRIPTION
software engineeringTRANSCRIPT
COMP 211COMP 211 TESTINGTESTING1
TESTINGTESTINGTESTINGTESTING
COMP 211COMP 211INTRODUCTION TOINTRODUCTION TO
SOFTWARE ENGINEERINGSOFTWARE ENGINEERING
COMP 211COMP 211INTRODUCTION TOINTRODUCTION TO
SOFTWARE ENGINEERINGSOFTWARE ENGINEERING
COMP 211COMP 211 TESTINGTESTING3
TESTING OVERVIEWTESTING OVERVIEW [9.1]
testing is the process of finding differences between the specified (expected) and the observed (existing) system behavior
usually done by developers that were not involved in system implementation
to test a system effectively, must have a detailed understanding of the whole system
not a job for novices
it is impossible to completely test a nontrivial system
systems often deployed without being completely tested
Goal:Goal: design tests that will systematically find defects aim is to break the system (make it fail)
Goal:Goal: design tests that will systematically find defects aim is to break the system (make it fail)
COMP 211COMP 211 TESTINGTESTING4
the time uncertainty in testing is the debug part!
TESTING OVERVIEWTESTING OVERVIEW [9.2]
Softwareconfiguration
Testconfiguration
Testresults
Expectedresults
Errors
Errorratedata
Corrections
Testing Evaluation
Reliability& quality
model
Debug
Done
No errors
COMP 211COMP 211 TESTINGTESTING5
TESTING — VERIFICATION & VALIDATIONTESTING — VERIFICATION & VALIDATION
verification is the process of making sure that we have built the product right (i.e., it meets its stated requirements) most of the testing workflow is targeted at doing verification
validation is the process of making sure that we have built the right product (i.e., it is fit for its purpose) acceptance tests deal mainly with validation
testing verifies the results of implementation by testing each build as well as final versions of the system by– planning the tests required for each iteration
– designing and implementing the tests by creating test cases that specify what to test test procedures that specify how to test test components to automate the testing, if possible
COMP 211COMP 211 TESTINGTESTING6
Inception Elaboration Construction Transition
TESTING LIFE CYCLE ROLETESTING LIFE CYCLE ROLE
PhasesCore Workflows
Requirements
Analysis
Design
Implementation
Testing
iter.#1
iter.#2
— — — — —iter.#n-1
iter.#n
Increments
Iter
atio
n
COMP 211COMP 211 TESTINGTESTING8
TESTING ARTIFACTS & WORKERSTESTING ARTIFACTS & WORKERS
responsible for
TestEngineer
ComponentEngineer
responsible for
TestCase
XTest
ModelTest
EvaluationTest
Procedure
XTestPlan
XDefect
TestComponent
IntegrationTester
SystemTester
responsible for
COMP 211COMP 211 TESTINGTESTING11
UP — TESTING PROCESSUP — TESTING PROCESS
IntegrationTester
TestEngineer
SystemTester
PlanTests
PerformIntegration
Tests
ImplementTests
PerformSystemTests
EvaluateTests
ComponentEngineer
DesignTests
PerformUnit Test
COMP 211COMP 211 TESTINGTESTING12
TESTING TRUTHSTESTING TRUTHS
testing cannot show the absence of software errors;it can only show that software errors are present!
unlike software construction,testing is a destructive process–> try to make the software fail
a test is successful only if it uncovers a defect (bug)
hence, good testing has the opposite natureof good software engineering
(i.e., for good software engineering, success –> no defect foundfor good testing, success –> defect found)
COMP 211COMP 211 TESTINGTESTING13
It is impossible to completely test the entire system!
PLAN TESTSPLAN TESTS [9.5.1]
Testing strategy: specifies the criteria and goals of testing– what kinds of tests to perform and how to perform them
– the required level of test and code coverage
– the percentage of tests that should execute with a specific result
Estimate of resources required: human/system
Schedule for the testing: when to run which tests
Overall goalOverall goal: design a set of tests that has the highest likelihood of uncovering defects with the minimum amount of time and effort since resources are usually limited (up to 40% of project effort often devoted to testing)
Overall goalOverall goal: design a set of tests that has the highest likelihood of uncovering defects with the minimum amount of time and effort since resources are usually limited (up to 40% of project effort often devoted to testing)
COMP 211COMP 211 TESTINGTESTING14
DESIGN TESTSDESIGN TESTS [9.3.2]
test case: one way of testing the system (what, conditions, how)test case: one way of testing the system (what, conditions, how)
knowing the internal workings of a component, design test cases to ensure the component performs according to specification
White Box Testing: “testing-in-the-small”Derive test cases to verify component logic based on data/control structures. –> Availability of source code is required.
knowing the specified functionality of a component, design test cases to demonstrate that the functionality is fully operational
Black Box Testing: “testing-in-the-large”Derive test cases to verify component functionality based on the inputs and outputs. –> Availability of source code is not required!
Regression Testing: selective White Box and Black Box re-testing to ensure that no new defects are introduced after a change
COMP 211COMP 211 TESTINGTESTING15
WHITE BOX TESTINGWHITE BOX TESTING
Goal:Goal: ensure that we have executed/exercised:– all independent paths in the code at least once– all logical decisions on their true and false sides– all loops at their boundaries and within their bounds– internal data structures to ensure their validity
Goal:Goal: ensure that we have executed/exercised:– all independent paths in the code at least once– all logical decisions on their true and false sides– all loops at their boundaries and within their bounds– internal data structures to ensure their validity
Why is this important?
logic errors and incorrect assumptions are inversely proportional to a path’s execution probability
errors more likely in less used parts of the code
we often believe that a path is not likely to be executed reality is often counter intuitive
typos are random its likely that untested paths will contain some
COMP 211COMP 211 TESTINGTESTING16
There are 1014 possible paths!It takes 3,170 years to exhaustively test the program if each test requires 1 ms!!
WHY NOT EXHAUSTIVE CODE TESTING?WHY NOT EXHAUSTIVE CODE TESTING?
loop < 20 timesHow long would it take to exhaustively test this program?
COMP 211COMP 211 TESTINGTESTING17
Even though the False branch executes no additional statements, it needs to be tested!
WHY NOT JUST STATEMENT TESTING?WHY NOT JUST STATEMENT TESTING?
i = j + 1if (A && B {
x = y + z;}n = p + 1;
i = j + 1
A && B
x = y + z
n = p + 1
TrueFalse
suppose we modify the program to:
i = 0if (A && B {
i = 1;}x = y / i;
COMP 211COMP 211 TESTINGTESTING20
BASIS PATH TESTINGBASIS PATH TESTING [9.4.2]
GoalGoal: exercise each independent path in the code at least onceGoalGoal: exercise each independent path in the code at least once
1. From the code, draw a corresponding flow graph.
Sequence
If-then-else
...
Case
Do-untilDo-while
COMP 211COMP 211 TESTINGTESTING21
BASIS PATH TESTING EXAMPLEBASIS PATH TESTING EXAMPLE
procedure: process records1. do while records remain2. read record;3. if record field 1 = 04. then store in buffer;5. increment counter;6. elseif record field 2 = 07. then reset counter;8. else store in file;9. endif10. endif11. enddo
end
COMP 211COMP 211 TESTINGTESTING24
BASIS PATH TESTINGBASIS PATH TESTING
2. Determine the cyclomatic complexity of the flow graph.
cyclomatic complexity: a quantitative measure of thelogical complexity of code
Ways to compute cyclomatic complexity V(G):
V(G) = number of regions (areas bounded by nodes and edges—area outside graph is also a region)
V(G) = number of edges - number of nodes + 2
V(G) = number of (simple) predicate nodes + 1
provides an upper bound on the number of pathsthat need to be tested in the code
COMP 211COMP 211 TESTINGTESTING26
BASIS PATH TESTINGBASIS PATH TESTING
3. Determine a basis set of linearly independent paths.
independent path any path that introduces at least one new set of processing statements or a new condition
basis set set of independent paths through the code
test cases derived from a basis set are guaranteed to execute every statement at least one time during testing
basis set is not unique
COMP 211COMP 211 TESTINGTESTING27
1-2-3-6-7-9-10-1-11
BASIS PATH TESTING — EXAMPLEBASIS PATH TESTING — EXAMPLE
1-2-3-6-8-9-10-1-11 1-2-3-4-5-10-1-11 1-11
1
91011
2
4
587
3
6
COMP 211COMP 211 TESTINGTESTING28
BASIS PATH TESTINGBASIS PATH TESTING
4. Prepare test cases that force the execution of each path in the basis set.
Notes:
you do not need an activity diagram, but the picture will help when you trace component paths
count each logical test—compound tests count as the number of Boolean operators + 1 (i.e., count each simple predicate)
basis path testing should be applied to all components, if possible, and to critical components always
path testing does not test all possible combinations of all paths through the code; it just tests every path at least once
COMP 211COMP 211 TESTINGTESTING29
ASU — REGISTER FOR A COURSE PROCEDUREASU — REGISTER FOR A COURSE PROCEDURE
A student can register for a course if he has taken the prerequisites. However, he may not register for a course if he has already taken a course that is an exclusion. A student who does not have the prerequisites can register for a course if he is currently registered in the prerequisite course and has the permission of the instructor. Notwithstanding the preceding, the instructor may waive the prerequisite for a student.
Register for course
If (enrolment is open) AND (student does not have exclusions) AND ((student has prerequisites) OR (student has permission))Registerreturn(success)
Elsereturn(failure)
Endif
COMP 211COMP 211 TESTINGTESTING30
IS BASIS PATH TESTING ENOUGH?IS BASIS PATH TESTING ENOUGH?
int Q;. . .i = j + 1;if ( A() && B() ) {
x = y + z;}n = p + 1;. . .
Boolean A() { Boolean B() {if (Flag1) { if (Flag2) {
Q = 0; return TRUE;return TRUE; } else {
} else { int x = 10/Q;return FALSE; return FALSE;
} }} }
i = j + 1
A()&&B()
x = y + z
n = p + 1
TrueFalse
Flag1=FALSE Flag1=TRUEFlag 2=TRUE
What happens when Flag1=TRUE and Flag2=FALSE?
This isThis issufficientsufficientfor basisfor basispath testingpath testing
COMP 211COMP 211 TESTINGTESTING31
CONDITION TESTINGCONDITION TESTING
Goal:Goal: further exercises the true and false value of each simple logical condition in the component
Goal:Goal: further exercises the true and false value of each simple logical condition in the component
condition testing
– simple condition: (a rel-op b) where rel-op={<, ≤, =, ≠, ≥, >} (may be negated with NOT), e.g., a≤b; NOT(a≤b)
– compound condition: two or more simple conditions connected with AND, OR, e.g., (a>b) AND (c<d)
– relational expression: (E1 rel-op E2) where E1 and E2 are arithmetic expressions, e.g., ((a*b+c)>(a+b+c))
errors to test for include (incorrect/missing/extra):– Boolean operator – relational operator– Boolean variable – arithmetic expression– Boolean parenthesis
COMP 211COMP 211 TESTINGTESTING32
branch testing
for a compound condition C, test true and false branches of C and every simple condition of C
e.g., for C = (a>b) AND (c<d) we test for: C TRUE and FALSEa>b TRUE and FALSEc<d TRUE and FALSE
CONDITION TESTING (cont’d)CONDITION TESTING (cont’d)
domain testing
for an expression E1 rel-op E2, test for E1 greater than, equal to, or less than E2
– guarantees detection of rel-op error if E1 and E2 are correct
– to detect errors in E1 and E2, the difference between E1 and E2 for the tests E1 greater or less than E2 should be as small as possible
– for an expression with n variables, 2n tests are required
COMP 211COMP 211 TESTINGTESTING33
Goal:Goal: execute loops at their boundaries and within their boundsGoal:Goal: execute loops at their boundaries and within their bounds
Nested Loops1 conduct simple loop tests for the innermost loop
while holding the outer loops at their minimumiteration
2 work outward, conducting tests for the nextinnermost loop
3 continue until all the loops have been tested
tests grow geometrically as the level of nesting increases
LOOP TESTINGLOOP TESTING
Simple Loops (n iterations)1 skip the loop entirely2 only one pass through the loop3 two passes through the loop4 m passes through the loop where m < n5 n-1, n, n+1 passes through the loop
COMP 211COMP 211 TESTINGTESTING34
LOOP TESTING (cont’d)LOOP TESTING (cont’d)
Concatenated Loops independent loops simple loop testing dependent loops nested loop testing
Unstructured Loops redesign!!!
COMP 211COMP 211 TESTINGTESTING35
DATA FLOW TESTINGDATA FLOW TESTING
GoalGoal: ensure that the value of a variable is correct at certain points of execution in the code
GoalGoal: ensure that the value of a variable is correct at certain points of execution in the code
select test paths according to the locations ofdefinitions and uses of a variable
DEF(S) = {X | S contains a definition of X} locations of definition of X
USE(S´) = {X | S´ contains a use of X} locations of use of X
Data Use (DU) Chain (X) is the set of [X, S, S´] where X is “live”(i.e., not redefined between S and S´)
one testing strategy: every data use chain must be covered once for every variable, do a test along the path from where the
variable is defined to the statement(s) where the variable is used (can be combined with basis path testing)
COMP 211COMP 211 TESTINGTESTING36
BLACK BOX TESTINGBLACK BOX TESTING
black box techniques attempt to find:– incorrect or missing functions– interface errors– data structure or external database access errors– performance errors– initialization and termination errors
Codeinput
actual ? predictedoutput = output
Test cases should:– reduce by more than 1 the number of additional test cases required (i.e.,
a test case should cover a range of input or output values)– tell us something about presence or absence of a class of errors (e.g., all
character data is correctly/incorrectly processed)
COMP 211COMP 211 TESTINGTESTING37
all possible input/output values
EQUIVALENCE PARTITIONINGEQUIVALENCE PARTITIONING [9.4.2]
design test cases which are based on valid and invalid inputs/outputs
userqueries
numericaldata output
formatrequests
commandkey input
mousepicks onmenu
responsesto prompts
partition inputs and outputs to create thorough test coverage of a class of errors
partition inputs and outputs to create thorough test coverage of a class of errors
COMP 211COMP 211 TESTINGTESTING38
not member of set
SELECTING EQUIVALENCE CLASSESSELECTING EQUIVALENCE CLASSES
1. If input is a range –> one valid and two invalid equivalence classes:
2. If input is a specific value –> one valid and two invalid equivalence classes:
3. If input is a set of related values –> one valid and one invalid class:
4. If input is Boolean –> one valid and one invalid class:
in range greater than rangeless than range
value greater than valueless than value
member of set
BooleannonBoolean
COMP 211COMP 211 TESTINGTESTING39
userqueries
SELECTING TEST VALUES — BOUNDARY VALUESSELECTING TEST VALUES — BOUNDARY VALUES[9.4.2]
as well as selecting test data “inside” an equivalence class, we also select data at the “edges” of the equivalence class
numericaldata output
formatrequests
responsesto prompts
commandkey input
mousepicks onmenu
more errors occur at the boundaries of an input/output domain than in the “center”
more errors occur at the boundaries of an input/output domain than in the “center”
COMP 211COMP 211 TESTINGTESTING40
not member of set
BOUNDARY VALUE SELECTION — GUIDELINESBOUNDARY VALUE SELECTION — GUIDELINES
1. If the input is a range and is bounded by a and b, then use a, b, and values just above and just below a and b, respectively.
2. If the input is a number of discrete values, use the minimum and the maximum of the values and values just above and just below them, respectively. (Can also be applied to a single specific input value.)
3. For inputs that are a set of values, test all values in the set (if possible) and one value outside the set.
a b
min max
member of set
COMP 211COMP 211 TESTINGTESTING41
BOUNDARY VALUE SELECTION — GUIDELINES BOUNDARY VALUE SELECTION — GUIDELINES (cont’d)(cont’d)
5. Apply guidelines 1 and 2 to create output values at the minimum and maximum expected values.
e.g., if the output is a table, create a minimum size table (1 row) and a maximum size table
6. If data structures have boundaries, test these boundary values and values just above and just below them, respectively.
e.g., for an array of bounds 1 to 10 — test array index equal to 0, 1,2, 9, 10, 11
1 100 2 9 11
4. For inputs that are Boolean, test for both Boolean values (T, F) and for a non-Boolean value.
nonBoolean Boolean
COMP 211COMP 211 TESTINGTESTING42
Comparison Testing– when reliability is absolutely critical, write several versions of the
software — perhaps to be used as backups– run same test data on all versions; cross-check results for
consistency Caution: if the error is from a bad specification and all
versions are built from the same specification, then the same bug will be present in all copies!
Cause-Effect Graphing1. causes (input conditions) and effects (actions) are identified for a
subsystem2. a cause-effect graph is developed3. the graph is converted to a decision table4. decision table rules are converted into test cases
OTHER BLACK BOX TECHNIQUESOTHER BLACK BOX TECHNIQUES [9.4.2]
Error Guessing– use application domain experience or knowledge to select test values
COMP 211COMP 211 TESTINGTESTING43
THREAD TESTINGTHREAD TESTING
event-based approach where tests are based on events which trigger system actions particularly appropriate for object oriented systems
used after classes have been unit tested and integrated into subsystems
need to identify and execute each possible processing thread from use cases
may not be possible to do complete thread testing too many input/output combinations
focus on most commonly exercised threads basic paths through flow of events
COMP 211COMP 211 TESTINGTESTING44
THREAD TESTINGTHREAD TESTING
single thread: obj3obj2obj4
obj1
obj3
obj2
obj4
obj5
I1(obj1)
I2(obj1)
I3(obj1)
I1(obj3)
I1(obj2)
O1(obj4)
O2(obj4)
O1(obj5)
multiple thread: obj1obj2{obj5, obj4} multi-input thread: into obj1
COMP 211COMP 211 TESTINGTESTING45
STATE-BASED TESTINGSTATE-BASED TESTING [9.4.2]
focuses on comparing the resulting state of a class with the expected state
derive test cases using statechart diagram for a class
derive a representative set of “stimuli” (events) for each transition
check the attributes of the class after applying a “stimuli” to determine of the specified state has been reached
first need to put the class in the desired state before applying the “stimuli”
COMP 211COMP 211 TESTINGTESTING46
TEST CASE DOCUMENTATIONTEST CASE DOCUMENTATION
Test case name name of the test case
Description of the test what the test is designed to do
Target class/component/subsystem name name of the thing to be tested
Target class/component/subsystem operation name of the operation to be tested
Test type black box/white box; valid/invalid input
Test value(s) inputs to be used for the test
Verification expected result
COMP 211COMP 211 TESTINGTESTING47
IMPLEMENT TESTSIMPLEMENT TESTS
GoalGoal: automate test procedures as much as possibleGoalGoal: automate test procedures as much as possible
running test cases can be very tedious and time consuming– many possible input values and system states to test
test component: a program that automates one or several test procedures or parts of them
there are tools available to help write test components:– perform actions for a test case and the tool “records” the actions
– parameterize the recorded script to accept a variety of input values
spreadsheets and/or database applications can be used to store the required input data and the results of each test
COMP 211COMP 211 TESTINGTESTING48
PERFORM TESTS — TESTING STRATEGIESPERFORM TESTS — TESTING STRATEGIES
we need to integrate test cases into a well-planned series of steps that are taken to test a component/subsystem –> test procedure– what are the steps that need to be conducted to test a component?– when are the steps planned and undertaken?– how much effort, time and resources will be required to do the steps?
the testing steps start at the component/subsystem level and work outward toward the integration of the entire system
a testing strategy indicates which testing techniques (white box, black box, thread) are appropriate at which point in time
need to balance:flexibility & creativity with planning & management
testing often occurs when deadline pressures are most severe(progress must be measurable and problems identified as early as possible)
COMP 211COMP 211 TESTINGTESTING49
A TESTING STRATEGYA TESTING STRATEGY
Requirements capture R
U Unit testing
AAnalysis
We developthe systemfrom theoutside in
We testthe systemfrom theinside outDDesign
CImplementation(coding)
I Integration testing
SdSystem testing(developer)
ScSystem testing(customer)
COMP 211COMP 211 TESTINGTESTING51
Test cases
UNIT TESTINGUNIT TESTING [9.4.2]
emphasis is on White Box techniques
interfacelocal data structuresboundary conditionsindependent pathserror-handling paths
Component
COMP 211COMP 211 TESTINGTESTING53
Test cases
UNIT TESTING — PROCEDURESUNIT TESTING — PROCEDURES
driver and/or stubs must be developed for each unit test
interfacelocal data structuresboundary conditionsindependent pathserror-handling paths
Driver
Stub StubResults
driver: component that calls component to be testedstub: component called by component to be tested
Componentto be tested
COMP 211COMP 211 TESTINGTESTING54
INTEGRATION TESTINGINTEGRATION TESTING [9.4.3]
If components all work individually, why more testing?
Big Bang!! Incremental“builds”
Incrementalconstruction
strategy
Regressiontesting
Interaction errors cannot be uncovered by unit testing(e.g., interface misuse, interface misunderstanding, timing errors)
integration approaches:
versus
COMP 211COMP 211 TESTINGTESTING55
S2S2
S5S4
S3
S6 S7S6 S7
TOP DOWN INTEGRATION TESTINGTOP DOWN INTEGRATION TESTING
pro –> early testing and error detection of user interface components;can demonstrate a complete function of the system early
con –> cannot do significant low-level processing until late in the testing;need to write and test stubs
usually not appropriate for OO development(except to test operations of classes)
top subsystem is tested with stubs
stubs are replaced one at a time“depth-first” or “breadth-first”
as new subsystems are integrated, some subset of previous tests is re-run (regression testing)
S1
S3
S4 S5
COMP 211COMP 211 TESTINGTESTING56
D2
BOTTOM UP INTEGRATION TESTINGBOTTOM UP INTEGRATION TESTING
pro –> interaction faults are more easily found;easier test case design and no need for stubs
con –> user interface components are tested last
appropriate for OO development;test classes using their own test driver
drivers are replaced one ata time “depth-first”
subsystems are groupedinto builds and integrated
S2 S6 S7
S1
S3
S4 S5
COMP 211COMP 211 TESTINGTESTING57
pro –> parallel testing activities
con –> need many drivers and stubs
can significantly shortentotal testing time
D2
SANDWICH INTEGRATION TESTINGSANDWICH INTEGRATION TESTING
lower-level subsystems are groupedinto builds and tested with drivers
top-level subsystemsare tested with stubs
S2 S6 S7
S1
S3
S4 S5
S3
COMP 211COMP 211 TESTINGTESTING58
INTEGRATION TESTING — CRITICAL SUBSYSTEMSINTEGRATION TESTING — CRITICAL SUBSYSTEMS
Critical subsystems should be tested as early as possible! Critical subsystems should be tested as early as possible!
Critical subsystems are those that:
1. have high risk
2. address several software requirements (i.e., several use cases)
3. have a high level of control
4. are complex or error prone high cyclomatic complexity
5. have specific performance requirements
Regression testing is a must for these subsystems!
COMP 211COMP 211 TESTINGTESTING59
SYSTEM TESTINGSYSTEM TESTING [9.4.4]
We need to test the entire system to be sure thesystem functions properly when integrated
We need to test the entire system to be sure thesystem functions properly when integrated
some specific types of system tests:
Functional –> developers verify that all user functions work as specified in the system requirements specification
Performance –> developers verify that the nonfunctional requirements are met
Pilot –> a selected group of end users verifies common functionality in the target environment
Acceptance –> customer verifies usability, functional and nonfunctional requirements against system requirements specification
Installation –> customer verifies usability, functional and nonfunctional requirements in real use
COMP 211COMP 211 TESTINGTESTING60
PERFORMANCE TESTINGPERFORMANCE TESTING
stress tests –> verify that the system can continue functioning when confronted with many simultaneous requests How high can we go? Do we fail-soft or collapse?
volume tests –> verify that the system can handle large amounts of data, high complexity algorithms, or high disk fragmentation
security tests –> verify that access protection mechanisms work make penetration cost more than value of entry
timing tests –> verify that the system meets timing constraints usually for real-time and embedded systems
recovery tests –> verify that the system can recover when forced to fail in various ways database recovery is particularly important
COMP 211COMP 211 TESTINGTESTING61
PILOT TESTINGPILOT TESTING
Alpha test: a test in a controlled environment so thatdevelopers can observe users
software
Developer site Customer site
customer tests
Beta test: a test in a real environment so that bugs areuncovered from regular usage patterns
Developer site
developerreviews software
Customer site
customertests
COMP 211COMP 211 TESTINGTESTING62
ACCEPTANCE TESTINGACCEPTANCE TESTING
the process of ensuring that the software meets the user’s reasonable expectations
Can be very subjective good requirements specification from the start is important!
The method of defect resolution should be negotiated before the transition phase!
acceptance tests (black box) cover classes of tests such as:– functionality – documentation– performance – reliability, etc.
configuration review (audit)– ensures that all elements of the software delivered to the customer
(the software configuration) have been properly developed and documented
COMP 211COMP 211 TESTINGTESTING63
EVALUATE TESTSEVALUATE TESTS
We need to– evaluate the results of our testing
– compare the results with the goals outlined in the test plan
– prepare metrics that let the test engineers determine the current quality of the software
How do we know when to stop testing?
We can consider:
– testing completeness: % of test cases run and % of code tested
– reliability: based on testing error rate of previous projects
COMP 211COMP 211 TESTINGTESTING64
TESTING ERROR RATETESTING ERROR RATE
Past history can be used to plot expected failure rate.We compare this with actual failure rate for this project.
Testing time
Fai
lure
s pe
r te
st h
our
This project
Expected failure rate
Possible outcomes:• perform additional tests to find more defects• relax the test criteria, if set too high• deliver acceptable parts of the system;
continue testing unacceptable parts
Possible outcomes:• perform additional tests to find more defects• relax the test criteria, if set too high• deliver acceptable parts of the system;
continue testing unacceptable parts
COMP 211COMP 211 TESTINGTESTING65
SPECIAL CONSIDERATIONS FOR OO TESTINGSPECIAL CONSIDERATIONS FOR OO TESTING
What to unit test?– unit test has to be at least a class, but object state makes testing
difficult a class must be tested in every state it can ever enter
(i.e., state-based testing)
Encapsulation– encapsulation hides what is inside an object hard to know its state
need to provide a method, for testing only, that reports all of an object’s state
Inheritance and polymorphism– If a subclass overrides methods of an already tested superclass,
what needs to be tested — only the overridden methods? No! All of a subclass’s methods need to be tested again
due to dynamic binding and substitutability
COMP 211COMP 211 TESTINGTESTING66
TESTING — FINAL THOUGHTSTESTING — FINAL THOUGHTS
All test are importantAll test are important, but if you need to restrict testing … , but if you need to restrict testing … All test are importantAll test are important, but if you need to restrict testing … , but if you need to restrict testing …
testing a system’s capabilities (functionality) is more important than testing its components– Identify things that will stop users doing their jobs.
testing old capabilities is more important than testing new capabilities– Users expect existing features to keep working!
testing typical situations is more important than testing boundary cases– Normal usage patterns are more important than atypical ones.
COMP 211COMP 211 TESTINGTESTING67
DEBUGGINGDEBUGGING [9.3.4]
a consequence of testing, but a separate activity not testing
can be one of the most exasperating parts of development easy to lose focus or follow the wrong path!
time required tocorrect the error
time required todetermine the natureand location of theerror
Debugging islike trying tocure a sickperson.
We see thesymptomsand try tofind thecause.
COMP 211COMP 211 TESTINGTESTING68
BE SYSTEMATIC!
DEBUGGING — SYMPTOMS AND CAUSESDEBUGGING — SYMPTOMS AND CAUSES
symptom and cause may begeographically separated
symptom may disappear whenanother problem is fixed
cause may be due to humanerror that is hard to trace
cause may be due to assumptionsthat everyone believes
symptom may be intermittent
cause may be due to a systemor compiler error
symptom
cause
So, what can we do?
pressure to fix bugs often causes new bugs to be introduced
COMP 211COMP 211 TESTINGTESTING69
DEBUGGING TECHNIQUESDEBUGGING TECHNIQUES
1. brute force: try to let the computer find the error– run-time traces everywhere
The hope is that somewhere in all this informationwe’ll find the bug!
2. backtracking: start where the problem occurs and read the code backward until the problem is found
OK for small programs.
3. cause elimination: try to create input test data sets that would prove or disprove a particular theory
Difficult, but sometimes your best bet!
COMP 211COMP 211 TESTINGTESTING70
DEBUGGING — FINAL THOUGHTSDEBUGGING — FINAL THOUGHTS
debugging tools can help gain more insight into the problem
talk to your colleagues — They can see things you’ve grown too tired to see!
isolate and focus –> Think through the problem—some of the best debugging happens away from the computer!
Be absolutely sure to conduct regression tests when you “fix” the bug!
COMP 211COMP 211 TESTINGTESTING71
TESTING SUMMARYTESTING SUMMARY
Plan your tests carefully know what you are trying to test for
Use the right test in the right situation white box, black box, regression
Decide when enough testing has been done have clear criterion for when to stop testing
Be systematic in debugging