1841f06detprob3

MM Stroustrup Ch26

Comments?

Agree or disagree with his testing approach?

2841f06detprob3

MM Stroustrup ch26

Additional topics?

3841f06detprob3

L2asg C++

Good job – very interesting programs Common mistakes

– No failure set or not well labelled– Easy fault = large failure set

4841f06detprob3

Failure Set

-15

-10

-5

0

5

10

15

-15 -10 -5 0 5 10 15

Series1

5841f06detprob3

Failure Sets

What was the shape of your failure sets?

How could test case selection criterion improve the probability of finding these faults?

6841f06detprob3

From last year’s 1-minute paper

How do we measure test adequacy? Or what is test adequacy?

7841f06detprob3

Test terms

Fault

Failure

Error

8841f06detprob3

Test terms

Test selection criterion – a methodology for selecting test cases

Test adequacy criterion – a methodology for deciding when to stop testing

9841f06detprob3

Testing Basics

Detection probability

10841f06detprob3

Detection Probability

The probability of detecting a specific fault using a specified testing strategy

OrThe probability that a randomly-generated test set

that satisfies a specific criterion will detect a specified fault

11841f06detprob3

Operational Profile

The set of inputs from which operational data is chosen is called the operational profile– It usually includes a description of the

probability of which points will be used Is 7.99 more likely to be encountered at a

WalMart POS terminal than at an exclusive store that only sells fur coats?

12841f06detprob3

Methodology to evaluate testing criterion

Pick sample program, sample faults, sample operational profile, two criteria.

For each criterion, randomly generate test suites that satisfy the criterion.

Calculate the percentage of the suites that detect each of the faults.

The criterion that has the higher percentage is better.

13841f06detprob3

Triangle Example

cin >> a >> b >> c ;type = “scalene”;if (a == b || a == c || b == c) type= “isosceles”;if (a == b && a == c) type = “equilateral”;if (a >= b+c || b >= a+c || c >= a+b) type=“not a triangle”;if (a <= 0 || b <= 0 || c <= 0) type=“bad input”;cout<< type;

14841f06detprob3

Control Flow Graph

I: cout

B: type = isoscles

D: type = equilateral

F: type = not a triangle

H: type = bad input

A: type = scalene

C: if

E: if

G: if

Operational profile

3,3,3 abcdegi equi3,3,4 abcegi isos3,3,5 abcegi isos3,3,6 abcefgi not3,4,3 abcegi isos3,4,4 abcegi isos3,4,5 acegi scal3,4,6 acegi scal

All inputs are equally likely

15841f06detprob3

What are the failure probability for each color (separately)?

cin >> a >> b >> c ;type = “scalene”;if (a == b || a == c && b == c) type= “isosceles”;if (a == b || a == c) type = “equilateral”;if (a >= b+c || b >= a+c || c > a+b) type=“not a triangle”;if (a <= 0 || b <= 0 || c <= 0) type=“bad input”;cout<< type;

Blue Green Red

16841f06detprob3

TTYP – probability of detection

What is the probability of detection with one randomly chosen test case?

What is the probability of detection with two randomly chosen test cases?

17841f06detprob3

TTYP – per path

What is the probability of detection with one randomly chosen test case per path?

What is the probability of detection with an equal number of randomly chosen test cases?

18841f06detprob3

TTYP – smaller subdomains

What might be better smaller subdomains?

19841f06detprob3

TTYP - subdomains

Are paths the best subdomains? Would a functional decomposition be

better? Should we re-define the term subdomain?

20841f06detprob3

fault 1 fault 2 fault 3 fault 4

criterion 1 0.08 0.03 0.04 0.85

criterion 2 0.07 0.1 0.05 0.91

criterion 3 0.06 0.12 0.08 0.88

criterion 4 0.05 0.4 0.06 0.89

Comparing Criteria

21841f06detprob3

Solving this choice

What are the assumptions?

Analogy with dice?

22841f06detprob3

TTYP – smaller subdomains

What might be better smaller subdomains? Would MCC (multiple condition coverage)

be better subdomains

23841f06detprob3

TTYP2 – C0 and C1 coverage

How do we deal with C0 and C1 coverage since they are not subdomain testing methodologies?

24841f06detprob3

TTYP3

How could you estimate the det prob of C0 or C1 testing?

25841f06detprob3

Marble Problem

Assume that there is a bag of marbles from which marbles are drawn with replacement.

What is the maximum likelihood estimate of p (the probability of drawing a purple) marble if you draw exactly n purple marbles in a row?

? P such that (p)n(1-p) is max or ? P such that (p)n = .5

26841f06detprob3

Evaluating Testing Methods by Delivered Reliability

Frankl, Hamlet, Littlewood, Strigini

IEEE TOSE Aug98

For Thursday, Aug 30 – study through section 2.3