w04-dataflowcoverageagurfink/ece653/assets/pdf/w04-dataflowcoverage.pdftitle: w04-dataflowcoverage...

Testing: Dataflow Coverage

Testing, Quality Assurance, and MaintenanceWinter 2018

Arie Gurfinkel

based on slides by Thibaud Lutellier, Marsha Chechik and Prof. Lin Tan

2 2

Non-looping Path Selection Problem

x : =1

2 3

4

5 6

7

:= x

All branches 1, 2, 4, 5, 71, 3, 4, 6, 7

does not exercise the relationship between thedefinition of X in statement 2 and the reference to X in

statement 6.

3 3

Def, Use

• def: a location where a value for a variable is stored in memory.• use: a location where a variable's value is accessed.• def(n): The set of variable defined by node n.• use(n): the set of variable used by node n

Goal: Try to ensure that values are computed and used correctly.

4 4

Def-Use CFG

def(q0) = {z} use(q0) = {x, y}

def(q1) = {}use(q1) = {z, w}

def(q2) = {}use(q2) = {z}

def(q3) = {}, use(q3) = {w}

def(q4) = {}, use(q4) = {}

5 5

DU-path, Def-clear and Reach

l A definition-clear path (def-clear) p with respect to x is a sub-path where x is not defined at any of the nodes in p

l A du-path is a simple path where the initial node of the path is the only defining node of x in the path.

l Reach: if there is a def-clear path from the nodes m to p with respect to x, then the def of x at m reaches the use at p.

Example for z.:- du-path: q0,q1,q2- def-clear: q1,q2,q4- q0 reaches q2.

6 6

Does the def at n2 reach the use at n3?

7 7

no edges of the form (n,nstart) or (nfinish ,n)

no edges of the form (n,n)

there is at most one edge (m,n) for all m,n

every control graph is well-formed• Connected• Single start and single final node

every loop has a single entry and a single exit

Assumptions (1/2)

8 8

Assumptions (2/2)

at least one variable is associated with a node representing a predicate

no variable definitions are associated with a node representing a predicate • no predicates of the form: (x++ > 3)

every definition of a variable reaches at least one use of that variable

every use is reached by at least one definition

every control graph contains at least one variable definition

9 9

All-Defs-Coverage (ADC)l All-Defs-Coverage (ADC):

l Some def-clear sub-path from each definition to some use reached by that definition

x:= := x. . .def-clear

10 10

All-Defs Coverage: Example

Requires:d1(x) to a use

Satisfactory Path:[1, 2, 4, 6]

1

2 3

4

5

6

d1(x)

u3(x)

u5(x)

u2(x)

11 11

All-Uses Coverage (AUC)

All-Uses Coverage (AUC)Some definition-clear subpath from each definition to each use reached by that definition and each successor node of the use

:= x. . .

:= x

x:= . . .. . .

:= xdef-clear

def-clear

def-clear

12 12

All-Uses Coverage: Example

Requires:• d1(x) to u2(x)• d1(x) to u3(x)• d1(x) to u5(x)

Satisfactory Paths:• [1, 2, 4, 5, 6]• [1, 3, 4, 6]

1

2 3

4

5

6

d1(x)

u3(x)

u5(x)

u2(x)

13 13

All-Du-Paths Coverage (ADUPC)

• All-Du-Paths Coverage (ADUPC):– All def-clear sub-paths that are cycle-free or simple-cycles from each definition to

each use reached by that definition and each successor node of the use

. . .x:= . . .

. . .

def-clear

def-clear

def-clear

cycle-free or simple-cycles

cycle-free or simple-cycles

cycle-free or simple-cycles

:= x

14 14

All-Du-Paths Coverage: Example

Requires:• All d1(x) to u2(x): [1,2]• All d1(x) to u3(x): [1,3]• All d1(x) to u5(x): [1,2,4,5], [1,3,4,5]

Satisfactory Paths:• [1, 2, 4, 5, 6]• [1, 3, 4 ,5, 6]

1

2 3

4

5

6

d1(x)

u3(x)

u5(x)

u2(x)

15 15

Introduction to Software Testing (Ch 2)

Data Flow Test Criteria

• Then we make sure that every def reaches all possible uses:• All-Uses Coverage (AUC) : For each set of du-paths to uses S =

du (ni, nj, v), TR contains at least one path d in S.

• Finally, we cover all the du-paths between defs and uses:• All-du-Paths Coverage (ADUPC) : For each set S = du (ni, nj, v),

TR contains every path d in S.

• First, we make sure every def reaches a use:• All-Defs Coverage (ADC) : For each set of du-paths S = du (n,

v), TR contains at least one path d in S.

!X

Assume definitions occur on nodes only, for the following definitions.

16 16

Problems with data flow coverage criteria

Infeasible paths•Don’t usually get 100% coverage

Need to understand fault detection ability

Artificially combines control with data flow

17 17

Graph Coverage Criteria Subsumption

18 18

AUC & EC

Under what additional assumptions, AUC subsumes EC?• For every node with multiple outgoing edges, at least one

variable is used on each out edge, (and the same variables are used on each out edge).

Reasoning (key points, not a full proof):• Assume test set T satisfies AUC. • Each edge e1 must be either a branch edge or not.• If e1 is a branch edge, then it has a use, thus it must be covered

by T.• If e1 is not a branch edge, the closest branch edge before e1,

denoted as e2, should be covered T, and if e2 is covered by a test, e1 must also be. If the closed branch edge doesn’t exit, then the program has no branch edges; therefore e1 must be covered by T.

19 19

Compiler tidbit

In a compiler, we use intermediate representations to simplify expressions, including definitions and uses.

For instance, we would simplify:• x = foo(y + 1, z * 2)

To:• a = y+1• b = z*2• x = foo(a,b)

20 20

Exercise (1/3)

Answer questions (a)-(f) for the graph defined by the following sets:• N ={0, 1, 2, 3, 4, 5, 6, 7}• N0 = {0}• Nf = {7}• E = {(0,1), (1, 2), (1, 7), (2, 3), (2, 4), (3, 2), (4, 5), (4, 6), (5, 6), (6, 1)}• def(0)=def(3)=use(5)=use(7) = {X}

Also consider the following test paths:• t1 = [0,1,7]• t2 = [0,1,2,4,6,1,7]• t3 = [0,1,2,4,5,6,1,7] • t4 = [0,1,2,3,2,4,6,1,7]• t5 = [0,1,2,3,2,3,2,4,5,6,1,7]• t6 = [0,1,2,3,2,4,6,1,2,4,5,6,1,7]

21 21

Exercise (2/3)

(a) Draw the graph.

(b) List all of the du-paths with respect to x. (Note: Include all du-paths, even those that are subpaths of some other du-paths).

(c) For each test path, determine which du-paths that test path du-tours. Consider direct touring only. Hint: A table is a convenient format for describing the relationship.

22 22

Exercise (3/3)

(d) List a minimal test set that satisfies all-defs coverage with respect to x (Direct tours only). Use the given test paths.

(e) List a minimal test set that satisfies all-uses coverage with respect to x. (Direct tours only). Use the given test paths.

(f) List a minimal test set that satisfies all-du-paths coverage with respect to x. (Direct tours only). Use the given test paths.

23 23

Exercise - cont.(b) du (0, 5, x): i: [0,1,2,4,5] du (0, 7, x): ii: [0,1,7]du (3, 5, x): iii: [3,2,4,5]du (3, 7, x): iv: [3,2,4,6,1,7]

v: [3,2,4,5,6,1,7](c)

Test Path Du-tour directly

t1 = [0,1,7] iit2 = [0,1,2,4,6,1,7] /t3 = [0,1,2,4,5,6,1,7] it4 = [0,1,2,3,2,4,6,1,7] ivt5 = [0,1,2,3,2,3,2,4,5,6,1,7] iii, vt6 = [0,1,2,3,2,4,6,1,2,4,5,6,1,7] /

24 24

Exercise - Partial Solutions.

(d) List a minimal test set that satisfies all-defs coverage with respect to x (Direct tours only). Use the given test paths.• {t1, t4} or {t1, t5} or {t3, t4} or {t3, t5}• Why isn’t {t1, t6} a correct answer?– t6 does not du-tour any path in du(3, x).

(e) List a minimal test set that satisfies all-uses coverage with respect to x. (Direct tours only). Use the given test paths.• {t1, t3, t5}

(f) List a minimal test set that satisfies all-du-paths coverage with respect to x. (Direct tours only). Use the given test paths.• {t1, t3, t4, t5}

36 36

Strengths and Weaknesses of Graph Coverage:

Must create graphNode coverage is usually easy, but cycles make it hard to get good coverage in general.Incomplete node or edge coverage point to deficiencies in a test set.

37 37

Summary

Summarizing Structural Coverage:• Generic; hence broadly applicable• Uses no domain knowledge

Summarizing Dataflow Coverage:• Definitions and uses are useful but hard to reason.

Miscellaneous other notes:• Control-flow graphs are manageable for single methods, but not generally more than

that.• Use call graphs to represent multiple methods, hiding details of each method.• When we want to test du-paths across multiple elements, use first-use/last-def

heuristic.