hypothesize - free university of bozen-bolzano › ~russo › ttst ›...
TRANSCRIPT
![Page 1: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/1.jpg)
Tools and Techniques for Software Testing - Barbara Russo SwSE - Software and Systems Engineering group
�31
Hypothesize
![Page 2: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/2.jpg)
�32
• The point where the program actually failed, by throwing an exception or failing, isn’t necessarily where the bug is
• The buggy code may have propagated some bad values through good parts of the program before it eventually failed
The rationalebuglocalization
bugpropagation
![Page 3: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/3.jpg)
�33
• So the hypotheses should be about where the bug actually is (or is not), and what might have caused it
The rationale
![Page 4: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/4.jpg)
�34
• First, think about a program as modules, or steps in an algorithm, and try to rule out whole sections of the program at once
• The goal is to localize the bug!
Hypothesize
![Page 5: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/5.jpg)
�35
• Look at the flow of data in mostCommonWord() • If the symptom of the bug is an exception in countOccurrences(), then you can rule out everything downstream: • for example findMostFrequent()
Example
public static String mostCommonWord(String text) { List<String> words = splitIntoWords(text); Map<String,Integer> frequencies = countOccurrences(words); String winner = findMostFrequent(frequencies); return winner; }
![Page 6: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/6.jpg)
�36
• Second, choose a hypothesis that tries to localize the bug even further
• For instance, hypothesize that the bug is in splitIntoWords(), corrupting its results, which then cause the exception in countOccurrences()
Example
public static String mostCommonWord(String text) { List<String> words = splitIntoWords(text); Map<String,Integer> frequencies = countOccurrences(words); String winner = findMostFrequent(frequencies); return winner; }
![Page 7: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/7.jpg)
�37
• Third, run test cases: • If the hypothesis is true, then countOccurrences()is not the source of the problem
• If it is false, then splitIntoWords()is not the source of the problem
Example
![Page 8: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/8.jpg)
�38
• Bug localization by binary search. For example, if in mostCommonWords(), the data flows through three helper methods, do a binary search by • dividing the workflow in half, guessing that the bug is
found somewhere between the first helper method call and the second, and insert probes (like breakpoints, print statements, or assertions) there to check the results
• From the answer to that experiment, further divide in half
Other strategies
![Page 9: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/9.jpg)
�39
• Prioritize the hypotheses. When making the hypothesis, keep in mind that different parts of the system have different likelihoods of failure. For example, • Old, well-tested code is probably more
trustworthy than recently-added code • Java library code is probably trustworthy, because
they are more tried and tested
Other strategies
![Page 10: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/10.jpg)
Tools and Techniques for Software Testing - Barbara Russo SwSE - Software and Systems Engineering group
�40
Hypothesize: Delta debugging
![Page 11: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/11.jpg)
�41
• The process of isolating a small test case may also give the data that helps form a hypothesis, for example
• It may uncover two closely-related test cases for which one succeeds and one fails
• For example, maybe mostCommonWords("cc,b") fails, but mostCommonWords("ccb") passes
Rationale
![Page 12: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/12.jpg)
�42
• Examine the difference between the execution of these two test cases to help form a hypothesis • Which code is executed for the passing test
case, but skipped for the failing test case, and vice versa?
Rationale
![Page 13: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/13.jpg)
�43
• One hypothesis is that the bug lies in those lines of code, the delta, between the passing run and the failing run
Delta debugging
Andreas Zeller, Why Programs Fail
![Page 14: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/14.jpg)
�44
• Delta debugging defines the cause of a bug as a difference between successful execution and failing execution, and then searches for a minimal cause in order to find the problem
• Delta debugging tools can automate this process, but they are not widely used yet. Only one old implementation in Python (2005). Does it work on new applications? No plug-in for Eclipse!
Delta debugging
Perfect thesis work!!
![Page 15: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/15.jpg)
�45
• Delta Debugging automates the scientific method of debugging: • Set a hypothesis on why something does not
work • Test this hypothesis • Refine or reject it depending on the test outcome
Scientific method of debugging
![Page 16: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/16.jpg)
�46
• Consider a program and a number of changes to the program code. • After applying the changes, the program no longer works
• Isolation of failure-inducing executed statements - that is, the events during execution which were critical for producing the failure
• Identification of failure-inducing schedules (e.g. race conditions due to nondeterministic behavior) or the isolation of failure-inducing control statements (i.e. which branches taken were relevant and which not).
Applications
![Page 17: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/17.jpg)
Tools and Techniques for Software Testing - Barbara Russo SwSE - Software and Systems Engineering group
�47
Hypothesize: Slicing
![Page 18: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/18.jpg)
�48
• Finding the parts of a program that contributed to computing a particular value
• When a failure occurs, the slice for that value consists of the lines of the program that helped compute that bad value
• The bug that caused the bad value lies somewhere in that slice, so that’s your search space
Slicing
![Page 19: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/19.jpg)
�49
int x = 0; // must be >= 0 ... System.out.println("x=" + x); // prints a negative number
Exercise
![Page 20: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/20.jpg)
�50
int x = 0; // must be >= 0 Lines that directly change the value of x… x += bonus; ... System.out.println("x=" + x); // prints a negative number
Example
![Page 21: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/21.jpg)
�51
lines that helped compute bonus at that point
int x = 0; // must be >= 0 final int bonus = getBonus(); ... x += bonus; ... System.out.println("x=" + x); // prints a negative number
![Page 22: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/22.jpg)
�52
int x = 0; final int bonus = getBonus(); ... if (isWorthABonus(s)) { x += bonus; } ... System.out.println("x=" + x); // prints a negative number
![Page 23: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/23.jpg)
�53
control statements that affected the execution of statements already in the slice
int x = 0; final int bonus = getBonus(); ... for (final Sale s : salesList) { ... if (isWorthABonus(s)) { x += bonus; } ... } ... System.out.println("x=" + x); // prints a negative number
![Page 24: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/24.jpg)
�54
int calculateTotalBonus(final List<Sale> salesList) { ... int x = 0; final int bonus = getBonus(); ... for (final Sale s : salesList) { ... if (isWorthABonus(s)) { x += bonus; } ... } ... System.out.println("x=" + x); // prints a negative number ... }
![Page 25: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/25.jpg)
Tools and Techniques for Software Testing - Barbara Russo SwSE - Software and Systems Engineering group
�55
Experiment
![Page 26: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/26.jpg)
�56
• A good experiment is an observation of the system without disturbing it much. For example: • Run a different test case with reduced input • Insert a print statement or assertion in the
running program, to check its internal state • Set a breakpoint using a debugger, then
inspect the code with single-step and look at variables and object values
Experiment
![Page 27: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/27.jpg)
�57
• Do not work around your code before understanding your code : • Risk to mask the real bug • Increase the technical debt
• For example, when getting an ArrayOutOfBoundsException, don’t just add code that avoids or catches the exception, without fixing the real problem!
Watch out!
![Page 28: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/28.jpg)
�58
• Swap components. Swapping an alternative component implementing the interface. • If you suspect java.util.ArrayList, you could
swap in java.util.LinkedList instead. • If you suspect the Java runtime, run with a
different version of Java.
• Watch out!: have good reason to suspect a component as it takes time!
Other common experiments
![Page 29: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/29.jpg)
�59
• Make sure source code and object code are up to date: • Not running the right code: pull the latest
version from the repository, and delete all your binary files and recompile everything (in Eclipse, this is done by Project → Clean)
Other common experiments
![Page 30: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/30.jpg)
�60
The quadraticRoots function produces wrong answers sometimes /** * Solves quadratic equation ax^2 + bx + c = 0. * * @param a quadratic coefficient, requires a != 0 * @param b linear coefficient * @param c constant term * @return a list of the real roots of the equation */ public static List<Double> quadraticRoots(int a, int b, int c) { ... }
Put the following items in the order that you should try them: 1, 2, 3 • Change the code from using ArrayList to using LinkedList • Insert println() statements throughout the method to display the
intermediate values of the calculation • Write a test case that causes the bug to happen
Exercise mentimeter.com 800660
![Page 31: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/31.jpg)
�61
• 3 Change the code from using ArrayList to using LinkedList
• 2 Put println() statements throughout the method to display the intermediate values of the calculation
• 1 Write a test case that causes the bug to happen
Explain
![Page 32: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/32.jpg)
�X
Change your code from using ArrayList to using LinkedList.4Put println() statements throughout your method to display the intermediate values of the calculation.3Write a test case that causes the bug to happen.1Run a code coverage tool to see if there are lines that your test suite wasn’t reaching.2
Explain
![Page 33: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/33.jpg)
�62
• Having a failing test case makes it much easier to debug, and is always the best thing to start with
• Inserting print statements to collect information takes more effort but the effort pays off
• Changing from ArrayList to LinkedList is a good example of the “swap components” technique, but these are trusted components that are supposed to behave alike, so they would be the last things to try.
Explanation
![Page 34: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/34.jpg)
�63
• Classify the bug as a coding error, like a misspelled variable or interchanged method parameters, or a design error, like an underspecified or insufficient interface
• Verify whether the bug has any relatives. If I just found a divide-by-zero error here, did I do that anywhere else in the code? Also consider what effects your fix will have. Will it break any other code?
Fix the Bug and bug propagation
![Page 35: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/35.jpg)
�64
• After you have applied your fix, add the bug’s test case to your regression test suite, and run all the tests to assure yourself that
• The bug is fixed, and • No new bugs have been introduced
Run the test suite
![Page 36: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/36.jpg)
Tools and Techniques for Software Testing - Barbara Russo SwSE - Software and Systems Engineering group
�65
Coverage Testing
![Page 37: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/37.jpg)
�66
• One way to judge a test suite is to ask how thoroughly it exercises the program
• This is called coverage
Coverage
![Page 38: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/38.jpg)
Barbara Russo�67
1intfoo(inta,intb,intc,intd,floate){2if(a==0){
3return0;
4}5intx=0;
6if((a==b)||((c==d)&&bug(a))){
7x=1;8}
9e=1/x;
10returne;11}
Example text
bug(): if a=1 it returns true
and false if a!=1.
![Page 39: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/39.jpg)
BarbaraRusso �68
• Coverage is a measure of the completeness of the set of test cases • Method coverage • Statement coverage • Branch coverage • Condition coverage
Coverage
![Page 40: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/40.jpg)
BarbaraRusso �69
• Measure: percentage of methods that have been executed at least once by test cases
• Tests should call 100% of the methods • It seems irresponsible to deliver methods in the
product when testing never used these methods • you need to ensure you have 100% method
coverage
Method coverage
![Page 41: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/41.jpg)
BarbaraRusso �70
• There is only one method • int foo (int a, int b, int c, int d, float e) • for a=0 foo returns 0 no matter the values of the
other parameters • calling foo with input (0,0,0,0,0) we attain
100% method coverage in our example
Test Case 1
![Page 42: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/42.jpg)
BarbaraRusso �71
• Measure: percentage of statements that have been executed by test cases • Achieve 100% statement coverage. Count the
number of statements and cover all of them with a test
Statement coverage
![Page 43: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/43.jpg)
BarbaraRusso �72
• With Test Case 1, we executed the program statements on lines 1-4 out of 11 lines of code
• As a result, we had 42% (5/12) statement coverage from Test Case 1
Example
![Page 44: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/44.jpg)
BarbaraRusso �73
• We can attain 100% statement coverage by one additional test case,
• Test Case 2: foo(1, 1, 1, 1,1), expected return value of 1.
• we have now executed the program statements on lines 5-11
Example
![Page 45: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/45.jpg)
BarbaraRusso �74
• Measure: percentage of the decision points have been evaluated as both true and false in test cases.
• Two decision points – one on line 2 and the other on line 6
• 2 if (a == 0) {} • 6 if ((a==b) OR ((c == d) AND bug(a) )) {}
Branch Coverage
![Page 46: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/46.jpg)
BarbaraRusso �75
• For decision/branch coverage, we evaluate an entire Boolean expression as one true-or-false predicate
• We need to ensure that each of these predicates (compound or single) is tested as both true and false
![Page 47: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/47.jpg)
BarbaraRusso �76
Line# Predicate True False
3 (a==0) TestCase1foo(0,0,0,0,0)return0
TestCase2foo(1,1,1,1,1)return1
7 ((a==b)OR((c==d)ANDbug(a)))
TestCase2foo(1,1,1,1,1)return1
![Page 48: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/48.jpg)
BarbaraRusso �77
• With TewstCase1 and TestCase2 we have executed three of the four necessary conditions • we have achieved 75% branch coverage so far
• TestCase3 foo(1, 2, 1, 2, 1) return ?? • in calculating the output, we discover a division
by 0 that can cause future failures! • That was due to a local variable that we could
not control before!
TestCase3
![Page 49: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/49.jpg)
BarbaraRusso �78
Line# Predicate True False
3 (a==0) TestCase1foo(0,0,0,0,0)return0
TestCase2foo(1,1,1,1,1)return1
7 ((a==b)OR((c==d)ANDbug(a)))
TestCase2foo(1,1,1,1,1)return1
TestCase3 foo(1, 2, 1, 2, 1) division by zero!
![Page 50: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/50.jpg)
BarbaraRusso �79
• Measure: percentage of Boolean sub-expressions of the program that have been evaluated as both true or false outcome in test cases • applies to compound predicates
• Condition coverage measures the outcome of each of these sub-expressions independently of each other
Condition Coverage
![Page 51: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/51.jpg)
BarbaraRusso �80
Predicate True False
(a==b) TestCase2foo(1,1,1,1,1)return1
TestCase3foo(1,2,1,2,1)divisionbyzero!
(c==d) TestCase3foo(1,2,1,2,1)divisionbyzero!
bug(a)
Onlythe50%coverage!
![Page 52: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/52.jpg)
BarbaraRusso �81
• We examine our available information on the bug method and determine that is should return true when a=1
• foo(1, 2, 1, 1, 1), expected return value 1
TestCase4
![Page 53: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/53.jpg)
BarbaraRusso �82
Predicate True False
(a==b) TestCase2foo(1,1,1,1,1)return1
TestCase3foo(1,2,1,2,1)divisionbyzero!
(c==d) TestCase4foo(1,2,1,1,1)return1
TestCase3foo(1,2,1,2,1)divisionbyzero!
bug(a) TestCase4foo(1,2,1,1,1)return1
![Page 54: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/54.jpg)
BarbaraRusso �83
• To finalize our condition coverage, we must force bug(a) to be false
• We again examine our bug() method, which informs us that it should return a false value if fed any integer a different from 1
• So we create Test Case 5, foo(3, 2, 1, 1, 1), expected return value “division by zero”.
TestCase5
![Page 55: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/55.jpg)
�84
• We could have (2,2,1,1,1). The input would have been fine but we would never reach the AND condition. Thus, we must make the (a==b) false to be sure to test the AND condition for FALSE.
• The same applies for (c==d): we need to have it TRUE to be sure that FALSE is due to the bug(a) condition.
Note
![Page 56: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/56.jpg)
Barbara Russo�85
Predicate True False
(a==b) TestCase2foo(1,1,1,1,1)return1
TestCase3foo(1,2,1,2,1)divisionbyzero!
(c==d) TestCase4foo(1,2,1,1,1)return1
TestCase3foo(1,2,1,2,1)divisionbyzero!
bug(a) TestCase4foo(1,2,1,1,1)return1
TestCase5foo(3,2,1,1,1)divisionbyzero!
01/04/14
Traceability matrix
![Page 57: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/57.jpg)
�86
• Path coverage is every possible combination of branches — every path through the program — taken by some test case
• McCabe complexity is used to determine how many complete execution paths (i.e. test built on them) a tester need to consider
• As with code coverage this is a measure that approximates exhaustiveness
Path coverage
![Page 58: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/58.jpg)
�87
• It is an Eclipse plug-in • In the node build and sub-node plugins of the
POM file include <plugin>
<groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.8.2</version> <executions>
<execution> <goals>
<goal>prepare-agent</goal> </goals>
</execution> <!-- attached to Maven test phase --> <execution> <id>report</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution>
</executions> </plugin>
JaCoCo
![Page 59: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/59.jpg)
�X
Branch coverage is stronger (requires more tests to achieve) than statement coverage, and path coverage is stronger than branch coverage. In industry, 100% statement coverage is a common goal, but even that is rarely achieved due to unreachable defensive code (like “should never get here” assertions). 100% branch coverage is highly desirable, and safety critical industry code has even more arduous criteria (e.g., MC/DC, modified condition/decision coverage). Unfortunately 100% path coverage is infeasible, requiring exponential-size test suites to achieve.A standard approach to testing is to add tests until the test suite achieves adequate statement coverage: i.e., so that every reachable statement in the program is executed by at least one test case. In practice, statement coverage is usually measured by a code coverage tool, which counts the number of times each statement is run by your test suite. With such a tool, white box testing is easy; you just measure the coverage of your black box tests, and add more test cases until all important statements are logged as executed.
![Page 60: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/60.jpg)
�88
public class Hailstone { public static void main(String[] args) { int n = 3; while (n != 1) { if (n % 2 == 0) { n = n / 2; } else { n = 3 * n + 1; } } } }
Exercise
![Page 61: Hypothesize - Free University of Bozen-Bolzano › ~russo › TTST › SystematicDebuggingCode... · 2019-12-09 · • If you suspect java.util.ArrayList, you could swap in java.util.LinkedList](https://reader033.vdocuments.net/reader033/viewer/2022053008/5f0bdda57e708231d432991e/html5/thumbnails/61.jpg)
�89
• Run this class with JaCoCo code coverage highlighting turned on, by choosing Run → Coverage As → Java Application.
• By changing the initial value of n, you can observe how JaCoCo highlights different lines of code differently.
When n=3 initially, what color is the line n = n/2 after execution?
When n=16 initially, what color is the line n = 3 * n + 1 after execution?
What initial value of n would make the line while (n != 1) yellow after execution?