coverage tools
DESCRIPTION
Coverage tools. Program is typically compiled with special options, to add extra source or object code. Additional data structures, such as a flow graph, may also be created. Program is run, possibly via test cases During execution, information is accumulated and written to an output file. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/1.jpg)
Coverage tools
• Program is typically compiled with special options, to add extra source or object code.
– Additional data structures, such as a flow graph, may also be created.
• Program is run, possibly via test cases
– During execution, information is accumulated and written to an output file.
• Post-processing phase:
– User report is generated from output file.
![Page 2: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/2.jpg)
How to measure coverage?
• Possibilities:
– Instrument the source code before compilation
– Instrument the virtual machine or object code.
– The Apache Byte Code Engineering Library (BCEL) is used by several Java coverage tools
– http://jakarta.apache.org/bcel
– Use a customized virtual machine.
![Page 3: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/3.jpg)
Coverage filters
• Normally, a filtering function is provided to determine for which classes coverage is to be measured:
– Include or exclude specified packages
– Filter out calls to provided class libraries or framework.
– Exclude certain source code statements– Example: When a Java class is not
intended to have instances created, the default constructor is often set to be private. This would show up as uncovered code.
![Page 4: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/4.jpg)
Source code instrumentation
• Create a copy of the source code, instrument the copy, and compile the instrumented copy to .class files.
– This is the approach used by Clover.
• Typical method:
– Set up an array of counters.
– Insert source code at points of interest that increment specific counters.
– Coverage tool calls main() method
– After main() method returns, dump values of counters.
![Page 5: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/5.jpg)
Source code before instrumentation
public final void add( final MatrixRow<T> newValues ){ if ( getNumColumns( ) == 0 ) { contents = newValues.copy( ).getContents( ); } else { contents.addAll( newValues.copy( ).getContents( ) ); }}
• A counter is inserted for each method, statement, and branch.
• Statement counters are before each statement, in case the statement throws an exception.
• Branch counters need to identify true and false branches taken.
![Page 6: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/6.jpg)
Same code after instrumentation
public final void add( final MatrixRow<T> newValues ){ CLOVER.M[348]++; CLOVER.S[1814]++; if ( ( getNumColumns( )==0 && ++CLOVER.CT[338]!=0 | true )
|| ( ++CLOVER.CF[338] == 0 & false ) ) { CLOVER.S[1815]++; contents = newValues.copy( ).getContents( ); } else { CLOVER.S[1816]++; contents.addAll( newValues.copy( ).getContents( ) ); }}
methodcounter
statementcounters
branchcounters
![Page 7: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/7.jpg)
Counting branches
• Count number of times that a branch condition evaluates to true and false.
if ( ( aBoolean && ++CT[338]!=0 | true )
|| ( ++CF[338] == 0 & false ) )
• If aBoolean is true, then the && is also evaluated, which will increment the CT counter. We want the expression’s overall evaluation to be true.
• If aBoolean is false, then execution switches to the right side of the || clause. Increment the CF counter, but then ensure that the entire expression still evaluates to false to take the else branch.
• Short-circuit evaluation means that as soon as the expression result can be deterimined, evaluation stops. This avoids:– Incrementing the true counter if aBoolean is false.– Incrementing the false counter if aBoolean is true.
![Page 8: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/8.jpg)
Analysis
• Advantages:– Results are directly mapped to source code.– If a statement throws an exception, it will be counted
as having been executed.
• Disadvantages:– Requires keeping a second version of source code,
and ensuring that original source is not over-written.– Instrumented code must not affect original code
execution.– Management of .class files from compiling alternate
source code. – Memory usage: for statement coverage alone, every
statement requires one integer as a counter.
![Page 9: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/9.jpg)
Byte code instrumentation
• Can be done in two ways:
– Instrument object files in advance to produce a alternate version.
– Example: create instrumented .class files for Java.
– Instrument object files as they are used.
– Example: create customized .class file loader that inserts the instrumentation at run time.
![Page 10: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/10.jpg)
Analysis
• Pre-instrumentation:
– Advantages:
– source code is not modified.
– can be done as a batch process before execution
– Disadvantages:
– management of two .class file versions.
– mapping of source code statements to/from object code
– counting source code statements can be difficult
– object code may need extra information such as special compiler options
![Page 11: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/11.jpg)
Clover
• http://www.atlassian.com/software/clover
– Commercial tool: requires license
• Supports method, statement, and branch coverage
• Reports its findings in multiple formats
– From project level down to individual lines of source code
• Shows list of test cases that hit each line of course code
• Historical charting of code coverage and other metrics
• Plug-in versions available for Eclipse, NetBeans
![Page 12: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/12.jpg)
Clover plugin for Eclipse
• Image source: atlassian.com
![Page 13: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/13.jpg)
Clover coverage filters
• One can choose not to instrument certain types of blocks in the code (e.g. assertions and exception catching), in order to focus on the coverage of interest.
![Page 14: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/14.jpg)
Emma
• Open-source tool
– Supports class, method, “basic block”, and line coverage.
– “Fractional” line coverage supported, but not branch coverage.
– Standalone version works with Ant builds
– http://emma.sourceforge.net
– Eclipse plugin EclEmma also available
– http://www.eclemma.org
![Page 15: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/15.jpg)
Emma
• Uses byte code instrumentation.
– Classes can be instrumented in advance, or during class loading.
• Tool keeps a metadata file to associate byte code with source code
– A regular compile will allow for only limited metadata.
– A compile with debug option will provide more information to support branch coverage.
![Page 16: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/16.jpg)
Block Coverage
• Block coverage assumes that if a block of statements without branches is entered and exited, all statements in the block were executed.
– That is, the counter is at the end of the block, instead of before the source code statement.
• Result: If an exception occurs in the block, the entire block is not recorded as having executed.
– This may be fine for application source code, but it does not work well with JUnit test source code or in code for which exceptions are commonplace.
– JUnit throws exceptions internally when tests fail, so the test may not have appeared to be executed.
![Page 17: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/17.jpg)
Emma coverage report
![Page 18: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/18.jpg)
Emma source code annotations
![Page 19: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/19.jpg)
Fractional line coverage
Only partof conditionalexecuted
Loop incrementnot executed
![Page 20: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/20.jpg)
Anomalies with byte code coverage
• There are cases where the compiler may emit multiple copies of object code for a single instance of source code.
• In the above example, there is an instance variable initialized for each of two constructors. However, only one constructor is called. The result is the yellow annotation at initialization statement 17.
![Page 21: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/21.jpg)
Coverlipse
• Open source Eclipse plug-in
– http://coverlipse.sourceforge.net
• Provides block coverage, all-uses data flow coverage
– No branch coverage
– Uses Apache BCEL to instrument bytecode
• Appears to be dormant, and not compatible with Java 6 or Eclipse 3.3.
![Page 22: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/22.jpg)
Line coverage report
![Page 23: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/23.jpg)
All-uses coverage report
![Page 24: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/24.jpg)
All-uses coverage report
![Page 25: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/25.jpg)
CodeCover
• Open source Eclipse plug in
– Web site: http://codecover.org
– Version 1.0.0.0 available Nov. 2007
• Performs source-code instrumentation to obtain:
– Statement coverage
– Branch coverage
– Loop coverage: loop executed zero/once/many times.
– Multiple-condition coverage (called “strict condition” coverage by CodeCover)
![Page 26: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/26.jpg)
CodeCover summary report
![Page 27: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/27.jpg)
CodeCover detailed report
![Page 28: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/28.jpg)
CodeCover features
• Boolean value analyzer: shows how many Boolean combinations in conditions have been covered.
• Code “hot spots”: highlighting of code that is executed more frequently than most.
• Test correlation matrix: for each pair of test cases, the overlap in coverage for the two test cases is shown.
• Source instrumentation is done following a statement.– The following statements do not appear as coverable
items:– throw statements– return statements
– Methods that throw exceptions are marked as not covered.
![Page 29: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/29.jpg)
CodeCover Boolean analyzer
• For the compound condition shown, combinations of atomic conditions that have occurred are shown.
– The x shows a short-circuit evaluation.
![Page 30: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/30.jpg)
CodeCover Correlation view
• The colours give an indication of the overlap for pairs of test cases.
– The selected square shows that for these two test cases, they have 24 and 25 coverable items, and that 19 are shared, for an overlap of 88.1%
![Page 31: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/31.jpg)
Cobertura
• Open source tool for Java code coverage
– http://cobertura.sourceforge.net
• Measures line coverage, branch coverage, and McCabe cyclomatic complexity metric for methods.
– Branch coverage requires compilation with debug option.
• Must be run as an Ant task, with four stages
– Compile source code
– Instrument classes
– Run code
– Generate report
![Page 32: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/32.jpg)
Cobertura summary report
![Page 33: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/33.jpg)
Cobertura detail report
![Page 34: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/34.jpg)
Code Coverage: Benefits
• If you haven’t exercised part of the code, it could contain a bug.
• Can be used to locate “dead” or unexecutable code.
– WARNING: Be very careful when removing code, if the purpose is to test for some exceptional condition that is difficult to trigger.
![Page 35: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/35.jpg)
Code Coverage: Caveats
• Coverage tests that the code has been exercised, and not that you built what the customer wanted.
• If the logical structure of the code contains errors such as a missed case in a switch statement, the absence of code will not be detected.
• 100% coverage is often unrealistic, especially for fine-grained coverage measures
– Effort to achieve high coverage may be unjustified in terms of:
– time spent– producing unreliable test cases
– 80% coverage is a “rule of thumb” often used in industry
![Page 36: Coverage tools](https://reader030.vdocuments.net/reader030/viewer/2022032708/56812ee4550346895d94814a/html5/thumbnails/36.jpg)
References
• Clover: http://www.atlassian.com/software/clover
• Emma:– http://emma.sourceforge.net– http://www.eclemma.org
• Coverlipse: http://coverlipse.sourceforge.net
• CodeCover: http://codecover.org
• Cobertura: http://cobertura.sourceforge.net
• A. Glover, “Don’t be fooled by the Coverage Report”, IBM developer works articlehttp://www-128.ibm.com/developerworks/java/library/j-cq01316
• S. Gornett, “Code Coverage Analysis”– http://www.bullseye.com/coverage.html