Chapter 10 – Testing Chapter 10 – Testing and Debuggingand Debugging

Chapter GoalsChapter Goals►Learn techniques to test your codeLearn techniques to test your code

►Learn to carry out unit testsLearn to carry out unit tests

►Understand principles of test case Understand principles of test case selection and evaluationselection and evaluation

►Become familiar with a debuggerBecome familiar with a debugger

10.1 Unit Tests10.1 Unit Tests

►The single most important testing tool The single most important testing tool

►Checks a single method or a set of Checks a single method or a set of cooperating methods cooperating methods

►You don't test the complete program You don't test the complete program that you are developing; you test the that you are developing; you test the classes in isolation classes in isolation

Unit testingUnit testing

►Avoids confusion of interactionsAvoids confusion of interactions

►More startup cost, but less work in the More startup cost, but less work in the endend

►Analogy: Testing carAnalogy: Testing car

Our ExampleOur Example

►To compute the square root of To compute the square root of aa use a use a common algorithm: common algorithm: Guess a value Guess a value xx that might be somewhat that might be somewhat

close to the desired square root (close to the desired square root (xx = = aa is ok) is ok) Actual square root lies between Actual square root lies between xx and and a/xa/x Take midpoint (Take midpoint (x x + + aa//xx) / 2 as a better ) / 2 as a better

guess guess Repeat the procedure. Stop when two Repeat the procedure. Stop when two

successive approximations are very close to successive approximations are very close to each other each other

►Method converges rapidly. Square root Method converges rapidly. Square root of 100: Guess #1: 50.5of 100: Guess #1: 50.5Guess #2: 26.24009900990099Guess #2: 26.24009900990099Guess #3: 15.025530119986813Guess #3: 15.025530119986813Guess #4: 10.840434673026925Guess #4: 10.840434673026925Guess #5: 10.032578510960604Guess #5: 10.032578510960604Guess #6: 10.000052895642693Guess #6: 10.000052895642693Guess #7: 10.000000000139897Guess #7: 10.000000000139897Guess #8: 10.0 Guess #8: 10.0

Test HarnessTest Harness

►For each test, you provide a simple For each test, you provide a simple class called a class called a test harnesstest harness

►Test harness feeds parameters to the Test harness feeds parameters to the methods being tested methods being tested

HarnessHarness

►What is needed to make the testing What is needed to make the testing more robust?more robust? More testsMore tests

►Limitation: Hard to replicate tests when Limitation: Hard to replicate tests when bugs are fixedbugs are fixed

►Solution: Use test harness to repeat Solution: Use test harness to repeat valuesvalues

10.2 Providing Test Input10.2 Providing Test Input

►Solution #1: Hardwire series of testsSolution #1: Hardwire series of tests

►No need to memorize series of tests, No need to memorize series of tests, already taken care of. Just run tester already taken care of. Just run tester class.class.

public class RootApproximatorHarness1public class RootApproximatorHarness1{{public static void main(String[] args)public static void main(String[] args){{

double[] testInputs = double[] testInputs = { 100, 4, 2, 1, 0.25, 0.01 };{ 100, 4, 2, 1, 0.25, 0.01 };

for (int x = 0; x < testInputs.length; x++)for (int x = 0; x < testInputs.length; x++){{

RootApproximator r = new RootApproximator r = new RootApproximator(RootApproximator(testInputs[x]testInputs[x]););

double y = r.getRoot();double y = r.getRoot();System.out.println("square root of " + System.out.println("square root of " +

testInputs[x] + " = " + y);testInputs[x] + " = " + y);}}

}}}}

Generate Test Cases Generate Test Cases AutomaticallyAutomatically

► Instead of hard-coding array of valuesInstead of hard-coding array of values1.1. Loop through a sample range of valuesLoop through a sample range of values

for (for (int x = MIN; x <= MAX; x += INCREMENTint x = MIN; x <= MAX; x += INCREMENT){){

RootApproximator r = new RootApproximator(x);RootApproximator r = new RootApproximator(x);

double y = r.getRoot();double y = r.getRoot();

System.out.println("square root of " + x + System.out.println("square root of " + x +

" = " + y);" = " + y);

}}

Generate Test Cases Generate Test Cases AutomaticallyAutomatically

► Instead of hard-coding array of valuesInstead of hard-coding array of values2.2. Use random number generatorUse random number generator

final int SAMPLES = 100;final int SAMPLES = 100;

Random generator = new Random();Random generator = new Random();

for (for (int i = 0; i < SAMPLES; i++int i = 0; i < SAMPLES; i++){){

double x = 1000 * double x = 1000 * generator.nextDouble();generator.nextDouble();

RootApproximator r = new RootApproximator(x);RootApproximator r = new RootApproximator(x);

double y = r.getRoot();double y = r.getRoot();

System.out.println("square root of " + x + System.out.println("square root of " + x + " = " + y); " = " + y);

}}

What to Test for?What to Test for?

►Good testing requires testing good casesGood testing requires testing good cases Reduces debugging time and ensures better Reduces debugging time and ensures better

productproduct

►Test all of the features of the methodTest all of the features of the method Positive tests – normal behavior of methodPositive tests – normal behavior of method Boundary test cases (e.g. x = 0), make sure Boundary test cases (e.g. x = 0), make sure

end conditions are correctend conditions are correct Negative cases – program should rejectNegative cases – program should reject

10.3 Test Case Evaluation10.3 Test Case Evaluation►How do you know whether the output How do you know whether the output

is correct? is correct? Calculate correct values by hand Calculate correct values by hand

E.g., for a payroll program, compute taxes E.g., for a payroll program, compute taxes manually manually

Supply test inputs for which you know the Supply test inputs for which you know the answer answer E.g., square root of 4 is 2 and square root of E.g., square root of 4 is 2 and square root of 100 is 10 100 is 10

Verify that the output values fulfill certain Verify that the output values fulfill certain properties properties E.g., square root squared = original valueE.g., square root squared = original value

for (int i = 1; i <= SAMPLES; i++) {for (int i = 1; i <= SAMPLES; i++) {double x = 1000 * generator.nextDouble(); double x = 1000 * generator.nextDouble(); RootApproximator r = new RootApproximator(x);RootApproximator r = new RootApproximator(x);double y = r.getRoot(); double y = r.getRoot(); if (Numeric.approxEqual(y * y, x)){if (Numeric.approxEqual(y * y, x)){

System.out.print("Test passed: "); System.out.print("Test passed: "); passcount++;passcount++;} else {} else {

System.out.print("Test failed: "); System.out.print("Test failed: "); failcount++;failcount++;

}}System.out.println("x = " + x + ", root System.out.println("x = " + x + ", root squared = " + y * y);squared = " + y * y);

} }

►Use an Oracle: a slow but reliable Use an Oracle: a slow but reliable method to compute a result for testing method to compute a result for testing purposes purposes E.g., use Math.pow to slower calculate E.g., use Math.pow to slower calculate xx1/2 1/2 (equivalent to the square root of (equivalent to the square root of xx) )

for (int i = 1; i <= SAMPLES; i++) {for (int i = 1; i <= SAMPLES; i++) {

double x = 1000 * generator.nextDouble(); double x = 1000 * generator.nextDouble(); RootApproximator r = new RootApproximator(x);RootApproximator r = new RootApproximator(x);double y = r.getRoot(); double y = r.getRoot(); double oracleValue = Math.pow(x, 0.5);double oracleValue = Math.pow(x, 0.5);if (Numeric.approxEqual(y,oracleValue)){if (Numeric.approxEqual(y,oracleValue)){

System.out.print("Test passed: "); System.out.print("Test passed: "); passcount++;passcount++;} else {} else {

System.out.print("Test failed: "); System.out.print("Test failed: "); failcount++;failcount++;

}}} }

10.4 Regression Testing and 10.4 Regression Testing and Test CoverageTest Coverage

►Save test cases Save test cases Particularly tests that reveal bugsParticularly tests that reveal bugs

►Use saved test cases in subsequent Use saved test cases in subsequent versions versions

►A A test suitetest suite is a set of tests for is a set of tests for repeated testing repeated testing

Why keep a test case?Why keep a test case?►Very common for bugs to show up laterVery common for bugs to show up later

We often think we fixed a bug, but just We often think we fixed a bug, but just covered it upcovered it up

►Cycling = bug that is fixed but reappears Cycling = bug that is fixed but reappears in later versions in later versions

►Regression testingRegression testing: repeating previous : repeating previous tests to ensure that known failures of tests to ensure that known failures of prior versions do not appear in new prior versions do not appear in new versions versions

Test CoverageTest Coverage

►Test coverageTest coverage: measure of how : measure of how many parts of a program have been many parts of a program have been tested tested Good testing tests all parts of the Good testing tests all parts of the

programprogram E.g. every line of code is executed in at E.g. every line of code is executed in at

least one testleast one test

►Example: Test all possible branches Example: Test all possible branches inside of an if statementinside of an if statement

Testing LevelsTesting Levels

►The part of the program that The part of the program that is being tested.is being tested.

UnitUnit SystemSystem

Unit Level TestingUnit Level Testing

►Test individual units (classes) in Test individual units (classes) in isolation.isolation.

►Test features of the unit.Test features of the unit.

►Multiple test cases for each feature.Multiple test cases for each feature.

System Level TestingSystem Level Testing

►Test interactions of the various parts Test interactions of the various parts of the application.of the application.

►Completed after the unit tests pass.Completed after the unit tests pass.

Test CoverageTest Coverage

►Black-box testingBlack-box testing: test functionality : test functionality without consideration of internal structure without consideration of internal structure of implementation of implementation Useful since this is what the user interacts withUseful since this is what the user interacts with

►White-box testingWhite-box testing: take internal : take internal structure into account when designing structure into account when designing tests tests Unit testingUnit testing Why? Can not prove absence of bugs, only Why? Can not prove absence of bugs, only

presencepresence

Black Box TestingBlack Box Testing

►a.k.a. behavioral, functional, closeda.k.a. behavioral, functional, closed

►Test against the specification Test against the specification

►does not need the program sourcedoes not need the program source

► internal implementation is unknowninternal implementation is unknown

Black Box TestingBlack Box Testing

►Advantages: Advantages: user's point of view user's point of view less biased less biased can discover if part of the specification can discover if part of the specification

has not been fulfilled. has not been fulfilled. ►Disadvantages: Disadvantages:

may be redundant may be redundant can be difficult to design. can be difficult to design. testing every possible input stream is testing every possible input stream is

unrealisticunrealistic

White Box TestingWhite Box Testing

►a.k.a. structural, clear box, opena.k.a. structural, clear box, open

►tests against the implementationtests against the implementation

►every line of source code is executed every line of source code is executed at least once.at least once.

►tests exception handling code.tests exception handling code.

White Box TestingWhite Box Testing

►AdvantagesAdvantages can discover if any part is faultycan discover if any part is faulty test coverage (test all paths)test coverage (test all paths)

►DisadvantagesDisadvantages will not discover if part is missingwill not discover if part is missing only if the programmer knows what the only if the programmer knows what the

program is supposed to doprogram is supposed to do code must also be visible to the tester. code must also be visible to the tester.

"Fully" Tested"Fully" Tested

►To fully test a software product,To fully test a software product,the following are required:the following are required: unit testing and system level testingunit testing and system level testing black and white box testingblack and white box testing

►Does this ensure there are no bugs?Does this ensure there are no bugs?

10.6 Debugger10.6 Debugger

►To avoid/find problems, most To avoid/find problems, most professionals use a debuggerprofessionals use a debugger

►Programs rarely (never) run perfectly Programs rarely (never) run perfectly the first timethe first time Think of perfect first draft to paperThink of perfect first draft to paper

►The larger your programs, the harder The larger your programs, the harder it is to debug them simply by printing it is to debug them simply by printing out values or looking at the codeout values or looking at the code

►DebuggerDebugger - program to run your program - program to run your program and analyze its run-time behaviorand analyze its run-time behavior

►Debuggers can be part of your IDE (Eclipse, Debuggers can be part of your IDE (Eclipse, BlueJ) or separate programs (JSwat) BlueJ) or separate programs (JSwat)

► Three key concepts: Three key concepts: Breakpoints Breakpoints Stepping Stepping Inspecting variables Inspecting variables

BreakpointBreakpoint

►Breakpoint – Breakpoint – the debugger stops the debugger stops running the program when it hits a running the program when it hits a Setup by programmerSetup by programmer

►Once stopped, you can see the state of Once stopped, you can see the state of all the variablesall the variables

SteppingStepping

►Once a breakpoint is hit, two optionsOnce a breakpoint is hit, two options Step through the next few statements Step through the next few statements

carefully inspectingcarefully inspecting►Slow, but useful in heavy calculationsSlow, but useful in heavy calculations►Single-step – Single-step – line by line executionline by line execution►Step intoStep into – doesn’t just skip line to line, – doesn’t just skip line to line,

but goes into method calls within each linebut goes into method calls within each line►Step over – Step over – Execute the method and stopExecute the method and stop

Run at full speed until the next breakpointRun at full speed until the next breakpoint

ExampleExample

►Current line: Current line: String input = in.next();String input = in.next();Word w = new Word(input);Word w = new Word(input);int syllables = w.countSyllables();int syllables = w.countSyllables(); System.out.println("Syllables in " + input System.out.println("Syllables in " + input

+ ": " + syllables); + ": " + syllables);

Step OverStep Over

►Next step: Next step: String input = in.next();String input = in.next();Word w = new Word(input);Word w = new Word(input);int syllables = w.countSyllables(); int syllables = w.countSyllables(); System.out.println("Syllables in " + input System.out.println("Syllables in " + input

+ ": " + syllables);+ ": " + syllables);

Step intoStep into

public int countSyllables()public int countSyllables()

{{int count = 0;int count = 0;

int end = text.length() - 1; int end = text.length() - 1;

. . .. . .

} }

Which to chooseWhich to choose

► If the method is suspect (may be If the method is suspect (may be cause of problem) – step intocause of problem) – step into

► If you are sure the method works If you are sure the method works correctly – step overcorrectly – step over

What if a test fails?What if a test fails?

► It's time to debug!It's time to debug!

►We will want (need) a strategy.We will want (need) a strategy.

►Debugging StrategyDebugging Strategy A systematic way to find and fix bugs.A systematic way to find and fix bugs.

Debug Strategy 1: Trial & Debug Strategy 1: Trial & ErrorError

1.1. Write codeWrite code2.2. Run programRun program3.3. Fix bugsFix bugs

► AdvantagesAdvantages no planning requiredno planning required

► DisadvantagesDisadvantages Spend more time trying to find and fix bugs Spend more time trying to find and fix bugs

than you did writing the program.than you did writing the program.

Debug Strategy 2: Debug Strategy 2: Trial & Error with a DebuggerTrial & Error with a Debugger

1.1. Write codeWrite code2.2. Run programRun program3.3. Use Debugger program to fix bugsUse Debugger program to fix bugs

► AdvantagesAdvantages no planning requiredno planning required

► DisadvantagesDisadvantages Spend lots of time in the debugger program Spend lots of time in the debugger program

trying to find and fix the bugs (but maybe less trying to find and fix the bugs (but maybe less than the first strategy)than the first strategy)

Debug Strategy 3:Debug Strategy 3:Incremental Development with TestingIncremental Development with Testing

1.1. Write small amount of code with testing Write small amount of code with testing and debugging in mind.and debugging in mind.

2.2. Document and write tests for each new Document and write tests for each new piece of code.piece of code.

3.3. Run tests on new codeRun tests on new code

4.4. Fix any bugs before adding new code.Fix any bugs before adding new code.

Debug Strategy 3:Debug Strategy 3:Incremental Development with TestingIncremental Development with Testing

► DisadvantagesDisadvantages Takes more time (harder) to design the solutionTakes more time (harder) to design the solution Must write more code (tests)Must write more code (tests) Only possible if you're the one who wrote the Only possible if you're the one who wrote the

programprogram

► AdvantagesAdvantages A small amount of code to review and fix.A small amount of code to review and fix. Bug is fixed before it get hidden by some other bug.Bug is fixed before it get hidden by some other bug. If small parts work, the bigger parts will usually work If small parts work, the bigger parts will usually work

too. If not, it's only the public interfaces that have too. If not, it's only the public interfaces that have to be debugged.to be debugged.

General Debugging StrategyGeneral Debugging Strategy

1.1. Identify the bug. Must be repeatable and Identify the bug. Must be repeatable and testable.testable.

2.2. Write test(s) that fails for the bug.Write test(s) that fails for the bug.

3.3. Locate the source of the bug. HOW?Locate the source of the bug. HOW?

4.4. Understand the problem before you fix it.Understand the problem before you fix it.

5.5. Try your fix and run all tests.Try your fix and run all tests.

6.6. Repeat until all tests pass.Repeat until all tests pass.

How do we find the sourceHow do we find the sourceof the bug?of the bug?

► Pick a point to start lookingPick a point to start looking Beginning (main method)Beginning (main method) Divide and conquerDivide and conquer

► Trace the code. Trace the code. ((Look for variables with unexpected values)Look for variables with unexpected values) Manual traceManual trace Trace messagesTrace messages

►System.out.println(…)System.out.println(…)►Use Use java.util.logging.Loggerjava.util.logging.Logger class class

Debugger programDebugger program