11

Chapter 8: Testing and Debugging Chapter 8: Testing and Debugging

Chapter GoalsChapter Goals• To learn how to design test harnesses To learn how to design test harnesses

for testing components of your for testing components of your programs in isolation programs in isolation

• To understand the principles of test case To understand the principles of test case selection and evaluation selection and evaluation

• To be able to use assertions to To be able to use assertions to document program assumptions document program assumptions

• To become familiar with the debugger To become familiar with the debugger • To learn strategies for effective To learn strategies for effective

debugging debugging

22

Unit TestsUnit Tests In a unit test, a function or procedure is In a unit test, a function or procedure is

compiled outside the program in which it compiled outside the program in which it will be used, together with a will be used, together with a test harnesstest harness that feeds arguments to it. that feeds arguments to it.

Once you have confidence that the Once you have confidence that the function is working correctly, you can plug function is working correctly, you can plug it into your program. it into your program.

Test data can be entered manually (see Test data can be entered manually (see sqrtest1.cpp), in a loop (see sqrtest2.cpp), sqrtest1.cpp), in a loop (see sqrtest2.cpp), or using random number generation (see or using random number generation (see sqrtest3.cpp). sqrtest3.cpp).

33

Unit Tests (sqrtest1.cpp)Unit Tests (sqrtest1.cpp)/* Function to be tested *//* Function to be tested */

   /**/**     Computes the square root using Heron's formula   Computes the square root using Heron's formula     @param a an integer >= 0   @param a an integer >= 0     @return the square root of a   @return the square root of a*/*/double double squarerootsquareroot(double a) {  (double a) {          ifif (a == 0) (a == 0) returnreturn 0; 0;      double xnew = a;    double xnew = a;    double xold;    double xold;          do do {  {         xold = xnew;       xold = xnew;       xnew = (xold + a / xold) / 2;       xnew = (xold + a / xold) / 2;    }    }        whilewhile (! (!approx_equalapprox_equal(xnew, xold));(xnew, xold));          returnreturn xnew; xnew;}}   

44

Unit Tests (sqrtest1.cpp – cont.)Unit Tests (sqrtest1.cpp – cont.)

/* Test harness *//* Test harness */ int int mainmain() { () {

double x; double x; whilewhile (cin >> x) { (cin >> x) {

double y = double y = squarerootsquareroot(x); (x); cout << "squareroot of " cout << "squareroot of " << x << " = " << y << "\n"; << x << " = " << y << "\n";

} } returnreturn 0; 0;

} }

55

Unit Tests (sqrtest2.cpp – cont.)Unit Tests (sqrtest2.cpp – cont.)/* Test harness *//* Test harness */

int int mainmain() { () {

double x; double x;

forfor (x = 0; x <= 10; x = x + 0.5) { (x = 0; x <= 10; x = x + 0.5) {

double y = double y = squarerootsquareroot(x); (x);

cout << "squareroot of " << x << " = " cout << "squareroot of " << x << " = "

<< y << "\n"; << y << "\n";

} }

returnreturn 0; 0;

} }

66

Unit Test - Another AlternativeUnit Test - Another Alternative Example generating random inputs Example generating random inputs

Let’s first review how to generate random numbersLet’s first review how to generate random numbers

/**/**     Sets the seed of the random number generator.   Sets the seed of the random number generator.*/*/void void rand_seedrand_seed(){  (){     int seed =    int seed = static_caststatic_cast<int>(<int>(timetime(0));(0));      srandsrand(seed);(seed);}}  /** /**    Compute a random floating point number in a range   Compute a random floating point number in a range   @param a the bottom of the range   @param a the bottom of the range   @param b the top of the range   @param b the top of the range   @return a random floating point number x,    @return a random floating point number x,    a <= x and x <= b   a <= x and x <= b*/*/double double rand_doublerand_double(double a, double b) {  (double a, double b) {        returnreturn a + (b - a) * a + (b - a) * randrand() * (1.0 / RAND_MAX);() * (1.0 / RAND_MAX);}}  

77

Unit Tests (sqrtest3.cpp) Unit Tests (sqrtest3.cpp)

/* Test harness *//* Test harness */

int int mainmain() {  () {  rand_seedrand_seed();();       int i;int i;       forfor (i = 1; i <= 100; i++) {  (i = 1; i <= 100; i++) {         double x =        double x = rand_doublerand_double(0, 1E6);(0, 1E6);       double y =        double y = squarerootsquareroot(x);(x);       cout << "squareroot of " << x        cout << "squareroot of " << x

<< " = " << y << "\n";<< " = " << y << "\n";    }    }          returnreturn 0; 0;}}   

88

Selecting Test CasesSelecting Test Cases Positive testsPositive tests consist of legitimate inputs that you consist of legitimate inputs that you

expect the program handles correctly. expect the program handles correctly. Boundary casesBoundary cases are also legitimate inputs that you are also legitimate inputs that you

expect the program to handle in a trivial way. expect the program to handle in a trivial way. Negative casesNegative cases are inputs that you expect the are inputs that you expect the

program to fail on. program to fail on. Keeping test data in a file is smart because you can Keeping test data in a file is smart because you can

use it to review every version of the program. use it to review every version of the program. When you find a bug, you should record the data that When you find a bug, you should record the data that

caused the bug, and retest your program with that caused the bug, and retest your program with that data after you have tried to fix the bug. data after you have tried to fix the bug.

The phenomenon of fixing a bug, only to have it The phenomenon of fixing a bug, only to have it reappear after further program modifications is called reappear after further program modifications is called cyclingcycling. .

The process of testing against a set of past failures is The process of testing against a set of past failures is called called regression testingregression testing. .

99

Test Case EvaluationsTest Case Evaluations Once you have determined what Once you have determined what inputsinputs

are needed to test the program, you need are needed to test the program, you need to decide if the to decide if the outputsoutputs are correct. are correct.

Sometimes you can verify output by Sometimes you can verify output by calculating the correct values by hand. calculating the correct values by hand.

Sometimes a computation does a lot of Sometimes a computation does a lot of work, and it is not practical to do the work, and it is not practical to do the computation manually. computation manually.

We could write a test program that verifies We could write a test program that verifies that the output values fulfill certain that the output values fulfill certain properties. properties. • We next present some alternatives for this We next present some alternatives for this

particular case to test the square root function. particular case to test the square root function.

1010

Testing Result Values (example 1)Testing Result Values (example 1) Here we test the squareroot() function by comparing the square of Here we test the squareroot() function by comparing the square of

the returned number with the original input. (See sqrtest4.cppthe returned number with the original input. (See sqrtest4.cpp).).

/* Test harness *//* Test harness */  int int mainmain() {  () {     int i;   int i;      forfor (i = 1; i <= 100; i++) {  (i = 1; i <= 100; i++) {         double x =        double x = rand_doublerand_double(0, 1E6);(0, 1E6);       double y =        double y = squarerootsquareroot(x);(x);              ifif (! (!aprox_equalaprox_equal(y * y, x)) (y * y, x))           cout << "Test failed. ";          cout << "Test failed. ";              elseelse           cout << "Test passed. ";          cout << "Test passed. ";       cout << "squareroot of " << x << " = "        cout << "squareroot of " << x << " = "

<< y << "\n";<< y << "\n";   }   }        returnreturn 0; 0;}}   

1111

Testing Result Values (example 2)Testing Result Values (example 2) Another method is to use a slower but reliable Another method is to use a slower but reliable

procedure (called an procedure (called an oracleoracle) to verify the results. (In ) to verify the results. (In this case, we will use pow() as an oracle). (See this case, we will use pow() as an oracle). (See sqrtest5.cpp) sqrtest5.cpp)

/* Test harness *//* Test harness */ int int mainmain() {  () {  rand_seedrand_seed();(); int i; int i; forfor (i = 1; i <= 100; i++)  {  (i = 1; i <= 100; i++)  {       double x =      double x = rand_doublerand_double(0, 1E6);(0, 1E6);      double y =       double y = squarerootsquareroot(x);(x);            ifif (! (!approx_equalapprox_equal(y, (y, powpow(x, 0.5))) (x, 0.5)))          cout << "Test failed. ";         cout << "Test failed. ";            elseelse          cout << "Test passed. ";         cout << "Test passed. ";      cout << "squareroot of " << x << " = "       cout << "squareroot of " << x << " = "

<< y << "\n";<< y << "\n"; } } returnreturn 0; 0;} }

1212

Assertions Assertions Functions often contain implicit assumptions (denominators Functions often contain implicit assumptions (denominators

should be nonzero, salaries should not be negative). should be nonzero, salaries should not be negative). Such illegal values can creep into a program due to an input Such illegal values can creep into a program due to an input

or processing error - with distressing regularity! or processing error - with distressing regularity! Assertions provide a valuable sanity check. Assertions provide a valuable sanity check.

void raise_salary(Employee& e, double by){void raise_salary(Employee& e, double by){    assert(e.get_salary() >= 0 );    assert(e.get_salary() >= 0 );    assert(by >= -100);    assert(by >= -100);    double new_salary =     double new_salary =

e.get_salary() * (1 + by / 100);e.get_salary() * (1 + by / 100);    e.set_salary(new_salary);    e.set_salary(new_salary);} }

If an assertion is not satisfied, the program terminates with If an assertion is not satisfied, the program terminates with a useful error message showing the line number and the a useful error message showing the line number and the code of the failed assertion: code of the failed assertion:

assertion failed in file finclac.cpp line 61: by >= -100assertion failed in file finclac.cpp line 61: by >= -100

1313

Program TracesProgram Traces To get a printout of the program flow, you can insert trace To get a printout of the program flow, you can insert trace

messages into the beginning and end of every procedure. messages into the beginning and end of every procedure.

It is also useful to print the input parameters when a It is also useful to print the input parameters when a procedure is entered and to print return values when a procedure is entered and to print return values when a function is exited. function is exited.

ExampleExample: : string int_name(int n) { string int_name(int n) { cout << "Entering digit_name. n = " cout << "Entering digit_name. n = " << n << "\n"; << n << "\n";

……… … Other statements in functionOther statements in function … …

…… cout << "Exiting digit name. Return value = “cout << "Exiting digit name. Return value = “

<< s << "\n"; << s << "\n"; return s; return s; } }

1414

Program Traces (cont.)Program Traces (cont.) To get a proper trace, you must locate To get a proper trace, you must locate eacheach function exit function exit

point. point.

Sample output for n = 12305.Sample output for n = 12305.

Inside int_name. Thousands.Inside int_name. Thousands.Entering int_name. n = 12Entering int_name. n = 12Inside int_name. Teens.Inside int_name. Teens.Entering teen_name. n = 12Entering teen_name. n = 12Exiting teen_name. Return value = twelveExiting teen_name. Return value = twelveExiting digit_name. Return value = twelveExiting digit_name. Return value = twelveInside int_name. Hundreds.Inside int_name. Hundreds.Entering digit_name. n = 3Entering digit_name. n = 3Exiting digit_name. Return value = threeExiting digit_name. Return value = threeInside int_name. Ones.Inside int_name. Ones.Entering digit_name. n = 5Entering digit_name. n = 5Exiting digit_name. Return value = fiveExiting digit_name. Return value = fiveExiting int_name. Return value = twelve thousand Exiting int_name. Return value = twelve thousand three hundred five three hundred five

1515

Program Traces (Problems with Trace Program Traces (Problems with Trace Messages)Messages)

Program traces can be time-consuming to find out which Program traces can be time-consuming to find out which trace message to insert. trace message to insert.

If you insert too many messages, you produce a flurry of If you insert too many messages, you produce a flurry of output that is hard to analyze. output that is hard to analyze.

If you insert to few, you many not have enough information If you insert to few, you many not have enough information to spot the cause of the error. to spot the cause of the error.

When you are done with the program, you need to remove When you are done with the program, you need to remove all the trace messages. all the trace messages.

When you find a new error, you need to stick the print When you find a new error, you need to stick the print statement back in. statement back in.

Many profession programmers use a Many profession programmers use a debuggerdebugger, not trace , not trace messages, to locate errors in their code. messages, to locate errors in their code.

1616

The DebuggerThe Debugger Modern development environments contain Modern development environments contain

special programsspecial programs, called , called debuggersdebuggers, that help , that help you locate bugs by letting you follow the you locate bugs by letting you follow the execution of a program. execution of a program.

You can stop and restart your program and see You can stop and restart your program and see the contents of variables whenever your program the contents of variables whenever your program is temporarily stopped. is temporarily stopped.

Using a debugger is not cost free - it takes time to Using a debugger is not cost free - it takes time to set up and carry out an effective debugging set up and carry out an effective debugging session. session.

Large programs are harder to debug Large programs are harder to debug without using a debugger!!without using a debugger!!• learn how to use one, is time well spent!! learn how to use one, is time well spent!!

1717

The DebuggerThe Debugger Debuggers vary wildly from one system to another. Debuggers vary wildly from one system to another.

If you use an integrated development environment, which If you use an integrated development environment, which contains an editor, compiler, and a debugger, finding and contains an editor, compiler, and a debugger, finding and starting the debugger is usually very easy. starting the debugger is usually very easy.

On a UNIX system, you must manually build a debug On a UNIX system, you must manually build a debug version of your program and invoke the debugger. version of your program and invoke the debugger.

You can go a long way with just three commands: You can go a long way with just three commands: • Run until this line. Run until this line. • Step to next line. Step to next line. • Inspect Variable. Inspect Variable.

The exact names for these commands will vary from The exact names for these commands will vary from debugger to debugger, but all debuggers support them. debugger to debugger, but all debuggers support them.

1818

The DebuggerThe Debugger The "run until this line" command is the most important. The "run until this line" command is the most important.

Select a line with the mouse or cursor, then hit a key or Select a line with the mouse or cursor, then hit a key or select a menu command to run the program to the select select a menu command to run the program to the select line. line.

It's possible that the program terminates normally without It's possible that the program terminates normally without reaching that line, but that could be informative. reaching that line, but that could be informative.

1919

The DebuggerThe Debugger The "step to next line" command executes the current line The "step to next line" command executes the current line

and stops at the next program line. and stops at the next program line.

Once the program has stopped, you can look at the current Once the program has stopped, you can look at the current values of variables. values of variables.

Some debuggers require you select the variable the choose Some debuggers require you select the variable the choose a command such as "inspect variable." a command such as "inspect variable."

Some debuggers require you to type the name of the Some debuggers require you to type the name of the variable into a dialogue box. variable into a dialogue box.

Some variables automatically show the values of all Some variables automatically show the values of all variables. variables.

2020

The DebuggerThe Debugger Many debuggers also have "step over" and Many debuggers also have "step over" and

"step into" commands. "step into" commands. The "step over" command advances to the The "step over" command advances to the

next program line: next program line: • ExampleExample: If the current line is : If the current line is

r = future_value(balance, p, n);r = future_value(balance, p, n); cout << setw(10) << r; cout << setw(10) << r;

"step over" advances the program to"step over" advances the program to

r = future_value(balance, p, n); r = future_value(balance, p, n); cout << setw(10) << r;cout << setw(10) << r;

2121

The Debugger (cont.)The Debugger (cont.) The "step into" command advances into The "step into" command advances into

function calls. function calls. • ExampleExample: if the current line is : if the current line is

r = future_value(balance, p, n);r = future_value(balance, p, n); cout << setw(10) << r;cout << setw(10) << r;

"step into" advances the program to "step into" advances the program to

double future_value(double initial_balance, double future_value(double initial_balance, double p, int n) {double p, int n) {

double b = initial_balance*pow((1+p/100),n);double b = initial_balance*pow((1+p/100),n); return b; return b;

} }

2222

The Debugger (cont.)The Debugger (cont.) You should not "step into" system functions like setw. You should not "step into" system functions like setw.

Some debuggers have a "run to end of function" or "step Some debuggers have a "run to end of function" or "step out" command to get out of a function. out" command to get out of a function.

Most debuggers can show you a call stack: a listing of all Most debuggers can show you a call stack: a listing of all currently pending function calls. currently pending function calls.

My selecting the function in the middle of the call stack, you My selecting the function in the middle of the call stack, you can jump to the code line containing that function call. can jump to the code line containing that function call.

2323

The DebuggerThe Debugger All debuggers support a navigational All debuggers support a navigational

approach by inserting approach by inserting breakpointsbreakpoints in the in the code. code.

When the program reaches any When the program reaches any breakpoint, execution stops. breakpoint, execution stops.

Breakpoints are particularly useful when Breakpoints are particularly useful when you know at which point your program you know at which point your program starts doing the wrong thing. starts doing the wrong thing.

Some debuggers let you set Some debuggers let you set conditional conditional breakpointsbreakpoints • the program stops only when a certain the program stops only when a certain

condition, such as n == 0, is met. condition, such as n == 0, is met.

2424

The Debugger (cont.)The Debugger (cont.) When inspecting an object variable, all the fields are When inspecting an object variable, all the fields are

displayed.displayed.

With some variables you must "open up" the object, usually With some variables you must "open up" the object, usually by clicking on a tree node. Here, we have opened up the by clicking on a tree node. Here, we have opened up the name field of the harry object.. name field of the harry object..

2525

StrategiesStrategies

Reproduce the error. Reproduce the error. • What numbers did you enter? What numbers did you enter? • Where did you click the mouse? Where did you click the mouse? • Run the same program again; type in Run the same program again; type in

exactly the same input. exactly the same input. Divide and Conquer Divide and Conquer

• Step over features using debugger, but Step over features using debugger, but don't step into them. don't step into them.

• Locate the last procedure called before Locate the last procedure called before failure. failure.

• Step into that procedure, and repeat. Step into that procedure, and repeat.

2626

StrategiesStrategies Know what your program should do. Know what your program should do.

• Does your program do what's expected? Does your program do what's expected? • Before you inspect a variable, decide what Before you inspect a variable, decide what

value is should have. value is should have. If value is correct, look elsewhere for the bug. If value is correct, look elsewhere for the bug. If value is incorrect, double check your computation, If value is incorrect, double check your computation,

then try to find out why your program comes up with then try to find out why your program comes up with a different value. a different value.

• Look for off by one errors. Look for off by one errors. • Look for computation errors (is the formula Look for computation errors (is the formula

typed in correctly). typed in correctly). • Make "sign calculations" when the numbers get Make "sign calculations" when the numbers get

nasty. ("Should this number be positive or nasty. ("Should this number be positive or negative?") negative?")

2727

Strategies (cont.)Strategies (cont.)

Avoid quick fixes - they tend to Avoid quick fixes - they tend to create problems elsewhere. create problems elsewhere.

Develop a thorough understanding of Develop a thorough understanding of the problem and the program before the problem and the program before you try to fix it. you try to fix it.

Sometimes fix and fix only causes Sometimes fix and fix only causes the problem to move around. the problem to move around. Possibly the program logic is Possibly the program logic is incorrect (does the program need incorrect (does the program need redesign?). redesign?).

2828

Debugger LimitationsDebugger Limitations Break points in recursive procedures interrupt the program Break points in recursive procedures interrupt the program

each time through the procedure.each time through the procedure. When using the debugger on a recursive procedure, watch When using the debugger on a recursive procedure, watch

the call stack carefully. the call stack carefully. Sometimes the compiler generates faster code by keeping a Sometimes the compiler generates faster code by keeping a

variable in a processor register rather than reserving a variable in a processor register rather than reserving a memory location for it. memory location for it. • The debugger cannot find that variable or displays a wrong The debugger cannot find that variable or displays a wrong

value for it. value for it. • You can try turning off all compiler optimizations and recompile. You can try turning off all compiler optimizations and recompile. • You can open a special register window that shows all processor You can open a special register window that shows all processor

registers (advanced). registers (advanced).

Some errors show up when you run the program normally, Some errors show up when you run the program normally, but go away under the debugger. but go away under the debugger. • Usually an uninitialized variable is the cause. Usually an uninitialized variable is the cause. • Inspect all variables manually and check that they are Inspect all variables manually and check that they are

initialized. initialized. • Insert print statements if you are desperate. Insert print statements if you are desperate.