software testing. 2 cmsc 345, version 4/12 topics the testing process unit testing integration and...

Software Testing

2CMSC 345, Version 4/12

Topics The testing process

unit testing integration and system testing acceptance testing

Test case planning and tracking Testing types

black box vs. white box testing coverage testing regression testing stress testing

Test case prioritization


Testing

Goal is to discover defects The only validation technique for non-

functional requirements Successful test: causes a program to

behave in an anomalous way Testing shows the presence, not the

absence, of defects.


Unit Testing

Integration and System Testing

Acceptance Testing

Testing Process Overview

unit1

unit2

unit3

unitn

unitunit

unit

Subsystem1

unitunit

Subsystem2

unit

unitunit

unit

Subsystemm

Final System Final System


unitunit

Subsystem

The Testing Process Elaborated Unit (component) testing

Testing of individual program components (e.g., functions, methods, or classes)

Usually the responsibility of the component’s developer Must be structured White box testing

Integration and system testing Testing of groups of components integrated to create a subsystem or

the entire system Sometimes the responsibility of an independent testing team Tests are based on system specification (black box testing) Is iterative (keeps building) May include alpha and beta testing

Acceptance testing Run in the presence of customer or by customer Used to validate all system requirements

unit

Final System$$$$!


Unit (Component) Testing

You may do this on your own, but always … have a plan keep records automate!

JUnit (for Java) or other toolseven #ifdef - #endif pairs (C/C++) main( ) method (Java)


Integration Testing Approaches – Procedural Programming

Top-down testing Start with high-level components and integrate from top down Uses stubs

Bottom-up testing Start with low-level components and integrate from bottom up Uses drivers

Sandwich testing Combination of top-down and bottom-up

Thread testing Tests one piece of functionality at a time


n

Level

0

1

Top-down Integration & Testing

Being Tested

Stub StubStub

Stub: 1) says “I’ve been called!”, 2) confirms receipt of inputs, if any, and 3) returns hard-coded outputs, if necessary


n

Level

0

1


Tested

Being Tested

Being Tested

Being Tested

Stub Stub StubStub



n

Level

0

1


Tested

Tested TestedTested

Tested TestedTestedTested

Being Tested

Being Tested

Being Tested

Being Tested



n

Level

0

1

Bottom-up Integration & Testing

Being Tested

Being Tested

Being Tested

Being Tested

Driver Driver

Driver: 1) Calls functions, 2) passes in inputs, if any, and 3) confirms receipt of outputs, if any


n

Level

0

1


Being Tested

Being Tested

Being Tested

Being Tested




n

Level

0

1




Driver DriverDriver



n

Level

0

1

Sandwich Integration & TestingUses both stubs and drivers


n

Level

0

1

Thread Integration & TestingMay be done top-down, bottom-up, or sandwich


Exercise (on your own!)What are some of the benefits associated with

each of the following integration and testing methods?

Top-down

Bottom-up

Sandwich

Thread


Integration Testing Approaches – Object-oriented Programming

Use based Start with classes that have no dependencies Move “out” to next “level” of classes, and so on Similar to bottom-up for procedural programming

Cluster Test clusters of associated classes Integrate the clusters May be combined with use based testing (above)

Scenario Based on functional specifications (possibly expressed as use

cases) Similar to thread testing for procedural programming


System Testing Alpha

Internal to development organization, but not the developers themselves

BetaExternal to development organization

AcceptancePerformed in the presence of or possibly by

the customerDetermines final “acceptance” by the

customer


Test #

Description

Input Values

Expected Output

Actual Output

P/F Criteria

Comments

Test Case Planning and Tracking

Natural language description

IDSpecific values (if possible)

Specific values (if possible)

Specific values (if possible)

Pass, Fail, other notes

How to determine Pass or Fail


Test #

Description

Input Values

Expected Output

Actual Output

P/F Criteria

Comments

1 Tests that a string will be displayed centered and in 12 point

font = 12 alignment = “C” string = “CMSC 345”

“CMSC 345” is displayed and centered in 12 point

Actual = Expected

A Sample Testing Form Before Testing


Test #

Description

Input Values

Expected Output

Actual Output

P/F Criteria

Comments




“CMSC 345 is displayed and centered in 12 point

Actual = Expected

Pass

A Sample Testing Form After Testing


Test #

Description

Input Values

Expected Output

Actual Output

P/F Criteria

Comments





Actual = Expected

Pass

2 Tests that the largest integer in the list is located

intList = 5, -2,100,16, -51,12 listSize = 6

100 Actual = Expected

A Sample Testing Form Before Testing


Test #

Description

Input Values

Expected Output

Actual Output

P/F Criteria

Comments




“CMSC 345 is displayed and centered in 12 point

Actual = Expected

Pass

2 Tests that the largest integer in the list is located

intList = 5, -2,100,16, -51,12 listSize = 6

100 -51 Actual = Expected

Fail

A Sample Testing Form After Testing


White Box Testing

Sometimes called structural testing Can “see inside” the code unit Objectives:

Exercise unique algorithmsExercise all program statements at least once

Typically tests small program units such as functions or class methods

code code code code code code code code code codecode code code code codecode code code code code

input output


Coverage Testing A technique for white box testing Ensures that each statement is executed at least

once Use a program flow graph to derive paths Find minimum paths needed to “cover” all

statements (there are tools to do this) Drive the code through all of these paths All branches are executed, but not all

combinations of branches. Some paths may be impossible to test.

#include <iostream>using namespace std;void triangle (int, int, int); // parameters are sides of the triangle

int main(){ int s1, s2, s3; char repeat;

cout << "This program will accept the sides of a triangle\n" << " and tell you the type of triangle.\n" << endl; do {

cout << "Enter the 3 sides: " ;cin >> s1 >> s2 >> s3;triangle(s1, s2, s3);cout << "Want to do another? (Y or N) ";cin >> repeat;

cout << endl << endl; } while (repeat =='Y' || repeat == 'y'); cout <<"Program terminated" << endl << endl; cout << "Enter any character to continue" << endl; cin >> repeat; return 0;}

void triangle (int side1, int side2, int side3){ // Error checking/***********************************/ // Check sides for validity if (side1 <= 0 || side2 <= 0) // 1 cout << "Illegal value for side" << endl << endl; // 2 // Check Triangle inequality if (side1+side2 <= side3 || side2+side3 <= side1) // 3 cout << "Triangle inequality violation" << endl << endl; // 4/***********************************/ // Pick triangle type if (side1 == side2 && side2 == side3) // 5 cout << "Triangle is equilateral" << endl; // 6 else if (side1 == side2 || side2 == side3 || side1 == side3) // 7 cout << "Triangle is isosceles" << endl; // 8 else if (side1*side1 + side2*side2 == side3*side3 || // 9 side1*side1 + side3*side3 == side2*side2) cout << "Triangle is right" << endl; //10 else cout <<"Triangle is scalene" << endl; //11return; //12} D. Cheslock, Northrup Grumman, 2009

triangle.cpp

CMSC 345, Version 4/12 26

triangle.exe Control Graph

Vertices= 18Edges = 28Regions = 12

V(g) = 28 - 18 + 2 (e-n+2) = 11 + 1 (p+1) = 12

1

7c

7b

5b

1b

3b

1a

2

3a

4

5a

67a

8

9a

10

1112

9b

2

3

4

5

6

7 8

9

11

10

12

Baseline path

D. Cheslock, Northrup Grumman, 2009CMSC 345, Version 4/12 27

A Set Of Basis Paths For triangle.cpp 1) 1a - 1b - 3a - 3b - 5a - 7a - 7b -7c -9a - 9b - 11 - 12

sample data: 3, 5, 7 (baseline path)

2) 1a - 2 - 3a - 3b - 5a - 7a - 7b -7c -9a - 9b - 11 - 12

sample data: -3, 7, 3

3) 1a - 1b - 2 - 3a - 3b - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12

sample data: can’t perform this test

4) 1a - 1b - 3a - 4 - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12

sample data: 2, 3, 7

5) 1a - 1b - 3a - 3b - 4 - 5a - 7a - 7b -7c - 9a - 9b - 11 - 12


6) 1a - 1b - 3a - 3b - 5a - 5b - 6 - 12


7) 1a - 1b - 3a - 3b - 5a - 5b - 7a - 7b -7c - 9a - 9b - 11 - 12

sample data: can’t perform this test

8) 1a - 1b - 3a - 3b - 5a - 5b - 7a - 8 - 12


9) 1a - 1b - 3a - 3b - 5a - 7a - 7b - 8 - 12


10) 1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 8 - 12


11) 1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 9a - 10 - 12


12) 1a - 1b - 3a - 3b - 5a - 7a - 7b -7c - 9a - 9b - 10 - 12

sample data: 3, 5, 4 D. Cheslock, Northrup Grumman, 2009CMSC 345, Version 4/12 28


Metrics – Cyclomatic Complexity

The minimum number of tests needed to execute all statements

CC = number_edges – number_nodes + 2

OR

CC = number_decisions + 1

OR

the number of regions


Black Box Testing

Cannot “see inside” code unit Test cases are:

based on the system specification and/or interfaces (preconditions, postconditions, invariants, and parameters)

Use equivalence partitions when conducting black box testing

input output


Equivalence Partitions and Boundary Testing

Equivalence Partition A group of input values that will produce

equivalent behavior Example

Square root function the set of all numbers >= zero the set of all negative numbers

Test cases should be chosen from each partition, especially at the boundaries.


Equivalence Partitions and Boundary Testing

Exercise A system accepts 4 to 10 inputs that are 5-

digit integers >= 10,000 Partition the system inputs into groups

(partitions) that should cause equivalent behavior. Include both valid and invalid inputs.

Ian Somerville, Software Engineering, 6th edition


Between 10000 and 99999Less than 10000 More than 99999

999910000 50000

10000099999

Input values

Between 4 and 10Less than 4 More than 10

34 7

1110

Number of input values

Ian Somerville, Software Engineering, 6th edition

The six partitions with corresponding boundary values


Regression Testing

Code A

Test 1

Test 2

Code A

Code B

Code C

Test 1

Test 2

Test 3

Test 4

Test 5

Code A

Code B

Code C

Code D

Test 2

Test 3

Test 4

Test 5

Test 1

Test 6

Test 7

Test 8

… and so on


Stress Testing Example: A child’s car seat Exercises the system beyond its maximum

design load. Examples: Exceed string lengths Store/manipulate more data than in specification Load system with more users than in specification

Often causes defects to come to light Systems should not fail catastrophically.

Stress testing checks for unacceptable loss of service or data.


Test Case PrioritizationWhy? To locate defects quickly to:

begin debugging ASAP debug and deliver in increments ASAP

To locate the majority of errors quickly Other

How? Can prioritize according to:

module or functional dependencies module size module complexity module suspected to be the most error-prone most “important” module or functionality other


References

Somerville, I., Software Engineering, 6th ed SourceForge.net, JUnit,

http://junit.sourceforge.net, accessed 11/6/07

JUnit.org, http://www.junit.org, accessed 11/6/07

software testing. 2 cmsc 345, version 4/12 topics the testing process unit testing integration and...

Documents

sandwich integration

receipt of outputs

receipt of inputs

software testing

hardcoded outputs

independent testing

calls functions

components developer