computer science 2212a/b - uwo1 structural testing motivation the gcov tool an example using gcov...
TRANSCRIPT
Computer Science 2212a/b - UWO 1
Structural Testing
Motivation
The gcov Tool
• An example using gcov
• How does gcov do it
• gcov subtleties
Further structural testing measures
• Flowcharts
• Structural testing and flowcharts
• Minimal test suites
• Path coverage
• Testing loops
Time to cover: 25-35 minutes
Computer Science 2212a/b - UWO 2
Structural Testing: Motivation
Consider a typical programming project:
• We have designed code based on requirements
• We have also built in some additional features
• Additional features interact with required features in various ways
• We need to test code of additional features
• We don’t remember what all of them are
• Because new features are not mentioned in requirements, testing based on requirements will not help us
Consider a function to sort an array of integers
• Requirements state only that resulting array must preserve original elements and be sorted
• Many different algorithms exist for doing this
• A test suite which tests one algorithm thoroughly may not test another
Computer Science 2212a/b - UWO 3
Structural Testing: Motivation continued
In both these cases, the essential problem is: How do we know we have tested all of our code?
• For all we know, some code may never have been executed by any test that we have run
• User may do something that executes this untested code • It will therefore be run for the first time by the user • Code run for the first time often has bugs!
Have we tested all of our code? • Tools exist to help us answer this question
These tools generally called “code coverage tools” • Help us to see what percentage of code has been “covered” by some
test case • These tools usually targeted to individual programming languages
Good free code coverage tools exist for C • Seems only commercial tools exist for Java • We will look at one for C
Computer Science 2212a/b - UWO 4
gcov: A Structural Testing Tool
gcov: GNU Project coverage utility • Based on earlier utility tcov • GNU Project:
—Supplies free software —Created gcc compiler for C —Most Linux utility programs come from GNU Project
To use gcov (overview): • Compile your C program using gcc with special switches • Run your program normally • While running, your program will write information to special “log
files”• After running test cases, run gcov • gcov will generate a coverage report listing which lines have been
executed We will look at each of these steps in more detail
Computer Science 2212a/b - UWO 5
Normal Compilation of a Program
In the above diagram, • Solid lines = inputs • Dashed lines = outputs
Normally, we just: • Compile a program from source code • Execute it on some selected test cases • Look at the output for correctness
Computer Science 2212a/b - UWO 6
Compilation Using gcov
• Compilation / execution with gcov is basically the same • However, extra switches to gcc cause extra things to happen • Switches: fprofile arcs ftest coverage
Computer Science 2212a/b - UWO 7
Compilation Using gcov continued
When we compile with the extra switches, gcc does the following
extra things:
• Generates “map file” of the blocks of code in our source files
• Generates object code which counts the number of times each block of code has been executed
“Block of code”:
• Any sequence of statements such that executing one of them guarantees we must execute the next one
Example (program wordcount.c):
• Instead of just “gcc wordcount.c”, we say gcc fprofile arcs ftest coverage wordcount.c
• gcc generates map file wordcount.gcno
• Exact format of the “map file” is not important to us
Computer Science 2212a/b - UWO 8
Running a gcov-Compiled Program
Run program as normal on test cases
First time program is run, a coverage data file appears (with extension .gcda)
• Coverage data file contains information about how many times each line of code has been executed
Example:
• After running wordcount, file wordcount.gcda is created
Coverage data is updated every subsequent time program is run
Coverage data is cumulative; e.g.
• Line 42 executed 5 times on first run;
• Line 42 executed 10 times on second run;
• Therefore, coverage data stored in .gcda file shows 15 executions of line 42
Computer Science 2212a/b - UWO 9
Getting Information from gcov
Run gcov with source file name as argument (e.g. gcov wordcount.c)
gcov writes some statistics to screen, creates a file with extension “.gcov” (e.g. wordcount.c.gcov)
.gcov file contains:
• On right: source code
• On left: number of times each line has been executed
• If lines have never been executed, .gcov file shows ##### to highlight it
Computer Science 2212a/b - UWO 10
Example wordcountProgram count the number of lines, words, chars in input file
Original Source file:
#include <stdio.h> #define TRUE 1 #define FALSE 0 main() { int nl=0, nw=0, nc=0; int inword = FALSE; char c; c = getchar(); while (c != EOF) { ++nc; if (c == ‘\n’) {++nl;} if (c==‘ ‘ || c==‘\n’ || c ==‘\t’) {inword = FALSE;} else if (!inword) {inword = TRUE; ++nw;} c = getchar(); } printf(“%d lines, %d words, %d chars\n”, nl, nw, nc);}
Computer Science 2212a/b - UWO 11
wordcount Example: Compiling and Running
We compile with:
gcc fprofile arcs ftest coverage o wordcount wordcount.c
We get:
• Executable in wordcount
• System files wordcount.gcno
We run wordcount • We give it as input two empty lines (two carriage returns)
File wordcount.da is created
We now run gcov wordcount.c
gcov tells us:
• 86.67% of 15 source lines executed in file wordcount.c Creating wordcount.c.gcov.
Computer Science 2212a/b - UWO 12
-: 0:Source:wordcount.c -: 0:Graph:wordcount.gcno -: 0:Data:wordcount.gcda -: 0:Runs:1 -: 0:Programs:1 -: 1:#include <stdio.h> -: 2:#define TRUE 1 -: 3:#define FALSE 0 1: 4:main() { 1: 5: int n1=0, nw=0, nc=0; 1: 6: int inword=FALSE; -: 7: char c; -: 8: 1: 9: c=getchar(); 4: 10: while (c != EOF) { 2: 11: ++nc; 2: 12: if (c=='\n') 2: 13: {++n1;} 4: 14: if (c==' ' || c=='\n' || c=='\t') 2: 15: {inword=FALSE; } #####: 16: else if (!inword) #####: 17: {inword=TRUE; ++nw; } 2: 18: c=getchar(); -: 19: } 1: 20: printf("%d lines, %d words, %d chars\n", -: 21: n1, nw, nc); -: 22:}
Computer Science 2212a/b - UWO 13
wordcount Example: Explanation of the Output
The two lines marked with # have never been executed
The other lines have been executed
For instance:
• The initial c = getchar() has been executed once (1 in left margin)
• The ++nc has been executed twice (2 in left margin)
—This is because each carriage return counts as 1 character
The declaration lines:
int nl=0, nw=0, nc=0;
int inword = FALSE;
are marked as executed because they contain initialization code
The declaration line
char c;
is not marked at all because it is just a declaration
Computer Science 2212a/b - UWO 14
wordcount Example: Carrying On
We now run wordcount again, this time giving just the line Zippy as input
We now run gcov wordcount.c gcov tells us: 100.00% of 15 source lines executed in file wordcount.c Creating wordcount.c.gcov.
Computer Science 2212a/b - UWO 15
Wordcount Example: New .gcov File -: 0:Source:wordcount.c -: 0:Graph:wordcount.gcno -: 0:Data:wordcount.gcda -: 0:Runs:2 -: 0:Programs:1 -: 1:#include <stdio.h> -: 2:#define TRUE 1 -: 3:#define FALSE 0 2: 4:main() { 2: 5: int n1=0, nw=0, nc=0; 2: 6: int inword=FALSE; -: 7: char c; -: 8: 2: 9: c=getchar(); 12: 10: while (c != EOF) { 8: 11: ++nc; 8: 12: if (c=='\n') 3: 13: {++n1;} 11: 14: if (c==' ' || c=='\n' || c=='\t') 3: 15: {inword=FALSE; } 5: 16: else if (!inword) 1: 17: {inword=TRUE; ++nw; } 8: 18: c=getchar(); -: 19: } 2: 20: printf("%d lines, %d words, %d chars\n", -: 21: n1, nw, nc); -: 22:}
Computer Science 2212a/b - UWO 16
wordcount Example: New Output
We have given the system an input file of 6 characters on this run (5 letters + carriage return) Along with the 2 characters we gave on the previous run, the runs of wordcount have read 8 characters Hence, ++nc line has been executed 8 times in total
• Counts on other lines have similarly been updated We have now executed the case where we have a non whitespace character
• Therefore, the lines else if (!inword) { inword = TRUE; ++nw; }
have been executed • Therefore, 100% of the lines of the program have now been
executed
Computer Science 2212a/b - UWO 17
Measuring the Usefulness of a Test Suite A good structural test suite will execute most lines of code
• It may not be possible to execute all lines • Executing a large percentage may be acceptable • Hence, measure of code covered (% of lines executed) is a good
measure of the usefulness of such a test suite However, with gcov:
• As we keep executing test cases, the .gcda file keeps getting updated Therefore, to measure goodness of a test suite, what we would like to do is:
• Reset all counts in .gcda file to 0 • Run test suite • Run gcov and look at % of lines covered
To reset all counts in the .gcda file: • Just remove the file!
Your compiled code will: • Assume this is the first time it has been run again • Create the .gcda file anew
Computer Science 2212a/b - UWO 18
How Does gcov Do It?
Each block in code numbered gcc inserts some code into your program at beginning of main, beginning of each block, end of main
• At beginning of main: —Program zeros out a big array
• During execution: —Every time block n executed, entry n of array incremented
• At end of main: —If .gcda file does not exist, program creates it —Otherwise, program reads .gcda file, increments counts in
array, writes out .gcda file again Does similar processing for calls to exit() as at end of main Map files map source line number to block
• gcov program matches block execution counts in .da file with source lines
Computer Science 2212a/b - UWO 19
gcov SubtletiesIf we compile and link many source files with the “gcov options”:
• Each source file gets a .gcno file (the map file) • When we run the program, each source file gets a .gcda file (the coverage data
file) Thus, we can/must get a separate report from gcov on each separate source file
• Useful if we are interested in coverage of each module of our program Example:
• We have a main program main.c and two modules, StackImplementation.c and IO.c
• On compilation, we get main.gcno, StackImplementation.gcno and IO.gcno • Every time we run the program, files main.gcda, StackImplementation.gcda,
and IO.gcda are updated • We can say gcov StackImplementation.c
—Get a report on StackImplementation.c separately —Useful if that is the module we are interested in
Computer Science 2212a/b - UWO 20
#include <stdio.h>
int main()
{
int x;
int y;
int z=0;
printf("\nEnter x:");
scanf ("%d", &x);
printf("\nEnter y:");
scanf("%d", &y);
z = x+y;
if (z>x)
printf("\nHi");
else
printf("\nBye");
if (z>y) printf ("\nYes");
else
printf("\nNo");
printf("\nAll Done");
return 0;
}
Let’s try it out on the gaul network:
lsgcc -fprofile-arcs -ftest-coverage junk2.c
Computer Science 2212a/b - UWO 21
gcov, Lines and Statements
Consider the following code: scanf(“%d”, &x); /* Read x from terminal */ if (x >= 0) {
pos = 1; }printf(“%d %d \n”, x, pos);
• We can execute 100% of lines in this code only if we input a positive number for x sometime
Now consider the following code: scanf(“%d”, &x); /* Read x from terminal */ if (x >= 0) {pos = 1;} printf(“%d %d\n”, x, pos);
• We can execute 100% of lines in this code even if we input only negative numbers for x
Computer Science 2212a/b - UWO 22
gcov, Lines and Statements continued
Reason:
• In second code fragment, the sub statement pos = 1 of the if is on the same line as the if
• We have executed the if in the sense that we have evaluated its condition (x>=0)
• Line is considered executed even when only some of the code on it is executed
Moral: to ensure all statements executed:
• Must put each statement on a separate line
Computer Science 2212a/b - UWO 23
Is gcov Enough?gcov is OK for a simple analysis of the thoroughness of a test suite
• It tells us what percentage of the lines of code we have executed However, a test suite executing 100% of lines may still not catch some problems Example: consider again the code
scanf(“%d”, &x); /* Read x from terminal */ if (x>=0) { pos = 1; }printf(“%d %d\n”, x, pos);
• If all tests in our test suite set x to a positive number, we can achieve 100% line coverage
• However, we have still not tested the case in which x is negative or zero • That case might be handled incorrectly
—For instance, maybe pos is supposed to be 1 if x is positive, and 0 otherwise
To talk about stronger measures of thoroughness, it is useful to have a graph of the program control flow
Computer Science 2212a/b - UWO 24
int main() {
int n1, n2, n3, ans; printf("\n\nEnter a value for n1:"); scanf("%d",&n1); printf("\n\nEnter a value for n2:"); scanf("%d",&n2); printf("\n\nEnter a value for n3:"); scanf("%d",&n3); ans = getnum(n1, n2, n3); printf("\nThe number was: %d\n”,ans); return 0;}
int getnum(int num1,int num2,int num3){ if (num1>num2) return num1; else if (num2>num3) return num2; else { if (num1==num2) return num3; else return num1+num2; } }
Consider the code to the right and answer the following questions:
• gcov would count 16 lines, which lines do you think get counted?
•What % coverage would the following test case give:n15, n24, n33
•Can we get 100% coverage? If so, list each of the test cases you would use to get that coverage, give the minimal number of test cases in your answer needed to get 100% coverage.
•Junit info
Review Questions