1. Copyright 2014, Oracle and/or its affiliates. All rights reserved. |

2. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Java Code Coveragewith JCovImplementation Details and Use CasesAlexandre (Shura) IlineJDK Test ArchitectJDK SQE teamOct 1, 2014 3. Dmitry FazunenkoCopyright 2014, Oracle and/or its affiliates. All rights reserved. 3JCov developmentAlexey Fedorchenko 4. Safe Harbor StatementThe following is intended to outline our general product direction. It is intended forinformation purposes only, and may not be incorporated into any contract. It is not acommitment to deliver any material, code, or functionality, and should not be reliedupon in making purchasing decisions. The development, release, and timing of anyfeatures or functionality described for Oracles products remains at the sole discretion ofOracle.Copyright 2014, Oracle and/or its affiliates. All rights reserved.4 5. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Program AgendaHistory, factsGetting the data with JCovApplied to OpenJDKUsing the dataLinks, more information123455 6. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 6Facts JCov is a Java Code Coverage tool. The code coverage tool for JCK The code coverage tool for Oracle JDK. Other products:JavaFXSceneBuilder... 7. History 1996: First prototype 1997: Integration with JDK 1.1 1998: Used in production JDK 1.2 JDK 7 2014: Open-source as part of OpenJDK codetools project. 2014: JCov 3.0 a public release supporting JDK 8 JDK 9 in progressCopyright 2014, Oracle and/or its affiliates. All rights reserved. 7 8. Copyright 2014, Oracle and/or its affiliates. All rights reserved.More on JCov dev1.5 engineer at a time, on average :)Leonid Arbouzov, Alexander Petrov, Vladimir Generalov, Serguei Chukhontsev, OlegUliankin, Gregory Steuck, Pavel Ozhdikhin, Konstantin Bobrovsky, Robert Field,Alexander Kuzmin, Leonid Mesnik, Sergey Borodin, Andrey Titov, Dmitry Fazunenko,Alexey Fedorchenko 9. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Code coverage is An information on what source code is exercised in execution9 10. most often ...Copyright 2014, Oracle and/or its affiliates. All rights reserved.Code coverage is An information on what source code is exercised in testing10Testing is Activities to prove that the code behaves as expectedwhere ... 11. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Other possible usages. Usage monitoring Save data in a test environment Check coverage from generated logic Fuzzing Load testing Regression test generation 12. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Program AgendaHistory, factsGetting the data with JCovApplied to OpenJDKUsing the dataLinks, more information2121345 13. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 13Getting the data A simplest use case A pick or two under the hood More possibilities Dynamic vs. static instrumentationGrabber Individual test coverage Abstract coverage Drop points Direct coverage Running big test suites 14. Simplest use case Compile java files as usual Instrument the byte codejava -jar jcov.jar Instr Run the codejava -classpath ...:jcov_file_saver.jar ... Create a reportjava -jar jcov.jar RepGen demoCopyright 2014, Oracle and/or its affiliates. All rights reserved. 15. Copyright 2014, Oracle and/or its affiliates. All rights reserved.A pick under the hood15 16. Copyright 2014, Oracle and/or its affiliates. All rights reserved.JCov bytecode insertionA pick under the hoodpublic int getX();Code:0: ldc #31 // int 22: invokestatic #29 // Method com/sun/tdk/jcov/runtime/Collect.hit:(I)V5: aload_06: getfield #2 // Field x:I9: ireturn16demoWarning: the code is a subject to change!!! 17. Copyright 2014, Oracle and/or its affiliates. All rights reserved.JCov bytecode insertionAs if it was in Javapublic int getX() {com.sun.tdk.jcov.runtime.Collect.hit(2);return x;}17Warning: the code is a subject to change!!! 18. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Increasing the countOne step deeperpublic class Collect {...private static long counts[];...public static void hit(int slot) {counts[slot]++;}...}18demo 19. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Saving the dataFinally ...Runtime.getRuntime().addShutdownHook(new Thread() {public void run() {...Collect.disable();Collect.saveResults();Collect.enable();...}});19 20. Performance impact Instrumentation phase:proportional to the application code size. Execution phase:Counters increments - very low. Depends on the size of blocks.Saving data from significant to blocking.Copyright 2014, Oracle and/or its affiliates. All rights reserved. 20 There is a cure! :) More on next slides. Report generation phase. 21. Copyright 2014, Oracle and/or its affiliates. All rights reserved.XML coverage data...21demoWarning: the format is a subject to change!!! 22. Back to the simplest use case Compile java files as usual Instrument the byte codejava -jar jcov.jar Instr Run the codejava -classpath ...:jcov_file_saver.jar ... Create a reportjava -jar jcov.jar RepGen demoCopyright 2014, Oracle and/or its affiliates. All rights reserved.Incomplete data!!! 23. Code coverage template Describes all code there is to cover Created during instrumentation:java -jar jcov.jar Instr -t template.xml Created explicitly:java -jar jcov.jar TmplGen -t template.xml A regular JCov XML file.Copyright 2014, Oracle and/or its affiliates. All rights reserved. Merge Filter etc. etc. 24. Simplest use case correcteddemo Compile java files as usual Instrument the byte codejava -jar jcov.jar Instr -t template.xml Run the codejava -classpath ...:jcov_file_saver.jar ... Merge with the templatejava -jar jcov.jar Merger -o merge.xml template.xml result.xml Create a reportjava -jar jcov.jar RepGen result.xmlCopyright 2014, Oracle and/or its affiliates. All rights reserved. 25. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Simplest use case with filtering Compile java code Instrument the byte codejava -jar jcov.jar Instr -t template.xml -i com.company.product Run the code, merge with the template Create a reportjava -jar jcov.jar RepGen -e com.company.product.test result.xmldemo 26. Copyright 2014, Oracle and/or its affiliates. All rights reserved.More on getting the data26 27. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Dynamic instrumentation Compile java files as usual Run with an extra VM optionjava -classpath ... -javaagent:jcov.jar ... result.xml generated. Merge and filter by the templatejava -jar jcov.jar Merger -o merge.xml -t template.xml result.xmldemodemo 28. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Dynamic vs. static instrumentationDynamic StaticModification of the testedapplicationNot needed NeededCoverage collected for All code(*) Instrumented code onlyTest execution Modified to pass more optionsModified to add jcov code tothe classpath (**)Performance Slows the classloading Faster28* There are options to limit the instrumented code** Or inject the classes into the application itself 29. Network grabberdemo Start the grabber on the networkjava -jar jcov.jar Grabber -hostname host123 -port 3333 Copyright 2014, Oracle and/or its affiliates. All rights reserved.-t template.xml Run the testsjava -classpath ... -javaagent:jcov.jar=grabber,host=host123,port=3333 ... Stop the grabber a data file is generatedjava -jar jcov.jar GrabberManager -kill Merge, generate report 30. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Individual code coverage Track code coverage on a per-test level Usefulness of the testsTest 2CodeCodeCodeTest 1 31. Individual test coverage with JCov Data is encoded into the XML file:...Copyright 2014, Oracle and/or its affiliates. All rights reserved.... 32. Test scales with files Run tests separately, save result.xml's$ java -classpath ... -javaagent:jcov.jar ... my.tests.TestN$ cp result.xml result_TestN.xml Merge data files$ java -jar jcov.jar Merger -o merge.xml Copyright 2014, Oracle and/or its affiliates. All rights reserved.-outTestList testlist.txt -t template.xml result_*.xmldemo 33. Test scales with grabber Start the grabber$ java -jar jcov.jar -scale -outTestList testlist.txt-t template.xml Run tests separately, let the grabber know test names$ java -classpath ... -javaagent:jcov.jar:grabber Copyright 2014, Oracle and/or its affiliates. All rights reserved.-Djcov.testname=TestN ... my.tests.TestN Stop the grabber, generate report.demo 34. Field coverage Generate templatejava -jar jcov.jar TmplGen -field on -t template.xml Instrumentationldc // int 1951invokestatic demo// Method com/sun/tdk/jcov/runtime/CollectDetect.invokeHit:(I)getfield // Field x:ICopyright 2014, Oracle and/or its affiliates. All rights reserved. 35. Abstract API coveragedemo Generate templatejava -jar jcov.jar TmplGen -abstract on -t template.xml Run with abstractjava -classpath ... -javaagent:jcov.jar=abstract=on ... Instrumentationldc // int 3012invokestatic // Method com/sun/tdk/jcov/runtime/CollectDetect.invokeHit:(I)invokeinterface // InterfaceMethod ...Copyright 2014, Oracle and/or its affiliates. All rights reserved. 36. Save points Instrumentjava -jar jcov.jar Instr -savebegin demoCopyright 2014, Oracle and/or its affiliates. All rights reserved.-t template.xml Instrumentation for the method:0: ldc // int 142: invokestatic // Method com/sun/tdk/jcov/runtime/Collect.hit:(I)V5: invokestatic // Method com/sun/tdk/jcov/runtime/Collect.saveResults:()V 37. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Direct coverage Only report code called directly from tests Fair coverage Controlled environment Meaningful parametersTestCodeCodeCode 38. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Direct coverage with no inner invocation Instrument code only (not tests)java -jar jcov.jar Instr -type method -innerinvocation off -t template.xml Instrumentationldc // int -1invokestatic // Method com/sun/tdk/jcov/runtime/CollectDetect.setExpected:(I)V... // Some other codeldc // int 0invokestatic // Method com/sun/tdk/jcov/runtime/CollectDetect.setExpected:(I)Vreturndemo 39. demoCopyright 2014, Oracle and/or its affiliates. All rights reserved.Direct coverage with caller include Run tests, instrumenting code and the testsjava -classpath ...-javaagent:jcov.jar=ci=,type=method ... Instrumentation of test codeinvokestatic // Method com/sun/tdk/jcov/runtime/CollectDetect.setExpected:(I)Vinvokevirtual // some product code call Instrumention of product codeinvokestatic // Method com/sun/tdk/jcov/runtime/CollectDetect.hit:(III)V 40. Collecting data for big suites Decide on dynamic vs static Save data periodicallyEven in case of same vmOtherwise risking loosing the data Using fileOne file for all tests getting bigger in time huge performance impact, ifmultiple VMsSeparate files for tests takes a lot of time to merge Use the grabber!Run test consequently for individual test coverage.Copyright 2014, Oracle and/or its affiliates. All rights reserved. 41. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 41Getting the data Instrument dynamically or statically Save the data onto a file or grabber Individual test coverage Abstract coverage Fields coverage Data save points Direct coverage 42. Copyright 2014, Oracle and/or its affiliates. All rights reserved.There is more.Exec Executes a command collecting coverage data in a grabberAgent print help on usage jcov in dynamic modeInstr instruments classfiles and creates template.xmlJREInstr instrumenter designed for instumenting rt.jarProductInstrInstr2 instrumenter designed for abstract, native methods and TmplGen generates the jcov template.xmlGrabber gathers information from JCov runtime via socketsGrabberManager control commands to the Grabber serverMerger merges several jcov data filesRepMerge merges jcov data files at method level not caring of blocks Filter filters out result dataDiffCoverage check whether changed lines were coveredRepGen generates text or HTML (or custom) reportsJCov gets product coverage with one command 43. There is more. TmplGenVerbosity: -verboseTemplate specification: -template(t) 'string value'Type of template: -type [all|branch|block|method]Filtering conditions:-include(i) 'string value', -exclude(e) 'string value',-include_list 'string value', -exclude_list 'string value'Specify which items should be additionally included in template:-abstract [on|off], -native [on|off], -field [on|off]-synthetic [on|off], -anonym [on|off]Flush instrumented classes: -flush 'string value'Basic options: -help(h, ?), -help-verbose(hv)-print-env(env), -propfile 'string value'-plugindir 'string value', -log.file 'string value'-log.level(log) 'string value'Copyright 2014, Oracle and/or its affiliates. All rights reserved. 44. There is more. InstrOutput: -instr.output(output, o) 'string value'Verbose mode: -verboseType of template: -type [all|branch|block|method]Filtering conditions: -include(i) 'string value, -include_list 'stringvalue, -exclude(e) 'string value, -caller_include(ci) 'string value,-caller_exclude(ce) 'string value, -exclude_list 'string value'Save points: -savebegin 'string value, -saveatend 'string value'Template specification: -template(t) 'string value, -subsequentItems to be included: -abstract [on|off], -native [on|off], -field [on|off],-synthetic [on|off], -anonym [on|off], -innerinvocation [on|off]Flush instrumented classes: -flush 'string value'Runtime management: -implantrt(rt) 'string value, -recursiveBasic options: -help(h, ?), -help-verbose(hv), -print-env(env), -propfile'string value, -plugindir 'string value, -log.file 'string value,-log.level(log) 'string value'Copyright 2014, Oracle and/or its affiliates. All rights reserved. 45. There is more. Merger.Output file: -merger.output(output, o) 'string value'File to read jcov input files from: -filelist 'string value'Filtering conditions: -include(i) 'string value'-exclude(e) 'string value', -fm 'string value'-include_list 'string value', -exclude_list 'string value'-fm_list 'string value'Process/generate test scales: -scale, -outTestList 'string value'Verbose mode: -verbose(v)Looseness level: -loose [0|1|2|3|blocks]Compress test scales: -compressBreak on error: -breakonerror [file|error|test|skip|none], -critwarnTemplate path: -template(tmpl, t) 'string value'Skipped files: -outSkipped 'string value'Basic options: -help(h, ?) -help-verbose(hv) -print-env(env)-propfile 'string value' -plugindir 'string value-log.file 'string value' -log.level(log) 'string valueCopyright 2014, Oracle and/or its affiliates. All rights reserved. 46. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Program AgendaHistory, factsGetting the dataApplied to OpenJDKUsing the dataLinks, more information2461345 47. OpenJDK coverage. Very simpledemo Templatejava -jar jcov.jar TmplGen -t template.xml rt.jar Start grabberjava -jar jcov.jar Grabber -v -t template.xml start Execute testsjtreg ... -javaoptions:"-javaagent:$JCOV_JAR=grabber" Stop grabber, merge, generate reportjava -jar jcov.jar GrabberManager -killjava -jar jcov.jar Merger -o merged.xml -t template.xml Copyright 2014, Oracle and/or its affiliates. All rights reserved.result.xmljava -jar jcov.jar RepGen -src jdk/src/share/classes merged.xml 48. OpenJDK coverage. Static. Instrumentjava -jar jcov.jar JREInstr ... -implant jcov.jar Copyright 2014, Oracle and/or its affiliates. All rights reserved.-t template.xml .../jre Run tests, etc 49. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Program AgendaHistory, factsGetting the dataApplied to OpenJDKUsing the dataLinks, more information2491345 50. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 50Using the data 100% coverage. Needed? Possible? Enough? Prioritizing Public API Monitoring coverage Speed up test execution More on test development prioritization Comparing suites, runs 51. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 51Block coverage target valueCost of testing-Defects found * COD=Return 52. 100% block coverage 1false 1 test 100% pass rate 100% coverage. The bug is not discoveredCopyright 2014, Oracle and/or its affiliates. All rights reserved. 52 53. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 53100% block and branch coverage 1true-1false 2 tests 100% pass rate 100% block coverage. 100% branch coverage. The bug is not discovered 54. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 54Prioritizing test development 100% coverage is not the goal Too much code to choose from Filter the coverage data to leave only code to coverPublic APIUIController code (as in MVC) Prioritize codeBy ageBy complexityBy bug density 55. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 55Public API 56. Public API Methods which supposed to be called directly from user code Every method in public API needs to be called at least once (*) 100% public API does not prove anything Necessary. Not sufficient. Rather blunt solution:java -jar jcov.jar RepGen -publicapi ...In JDK8: public and protected methods of public and protected class inCopyright 2014, Oracle and/or its affiliates. All rights reserved. 56java.* and javax.* (**)In JDK9: something different* More later** Not completely accurate 57. demoJCov API com.sun.tdk.jcov.instrument.DataRootLoad, save com.oracle.java.sqe.inspection.JCovFinder,JCovInspector, JCovVisitorwalk over coverage data com.sun.tdk.jcov.instrument.DataBlock, DataBranch,DataClass, DataMethod, DataPackageInvestigate, edit com.sun.tdk.jcov.filter.MemberFilter, FilterSPI,Filter JCov vommandfilterCopyright 2014, Oracle and/or its affiliates. All rights reserved. 57 58. Abstract public API Code coverage Good API is abstract An abstract public API is covered when at least one of its implementationsis covered. Done with JCov abstract coverage and filteringCopyright 2014, Oracle and/or its affiliates. All rights reserved. 58 59. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 59Public API implementation 60. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 60Public API implementation coverage Create template with abstract API Filter, only leavingImplementations of public APIExtensions of public API Filter coverage data by the template 61. Public API implementation closure It may be required to extend the original set//original public APIpublic interface Action { public abstract void perform();}//some implementationpublic class MySomethingAction implements Action{Copyright 2014, Oracle and/or its affiliates. All rights reserved. 61@Overridepublic void perform() {//prepare somethingdoPerform();}protected abstract void doPerform();}add tofilter 62. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 62Public API closure A set of methods implementing library interfaceNon-abstract public APIImplementations of abstract public APIOverrides of non-abstract public APIDelegation API closure of the delegation APIdemo 63. demoSane public APIEvery method in public API needs to be called at least once - Is this so? Some code is trivialOne line gettersOverloads Different code requires different techniques.java.lang.Object.hashCode() Some code has low impactException constructors MoreCopyright 2014, Oracle and/or its affiliates. All rights reserved. 63 64. UI code coverage Similar to public API:If there is a form in the product UI, it needs to be covered at least once Identify UI codeconstructing UI objectsdisplaying UI objectsactions with UICopyright 2014, Oracle and/or its affiliates. All rights reserved. 64 javax.sing.Action.actionPerformed 65. Controller code coverage Per MVCController accepts inputs and and converts to commands to view orCopyright 2014, Oracle and/or its affiliates. All rights reserved. 65model Very little boilerplate code In some cases identifiable by classes and packages Could be marked explicitly. @Important 66. Linking CC to other characteristics of the source codeCopyright 2014, Oracle and/or its affiliates. All rights reserved. 66Characteristic ReasoningAgeNew code is better to be tested before getting to customer.Old code is either already tested or not needed. :)Number of changesMore times the code was changed,more atomic improvements were implementedBug densityMany bugs already found means there are morehidden by existing bugs and more could be introduced with fixesComplexityAssuming similar engineering talent and the same technology more bugs are likely to exist in complex code 67. Linking CC to other characteristics of the source codeCopyright 2014, Oracle and/or its affiliates. All rights reserved. 67 A formula (1 cc) * (a1*x1 + a*x2 + a2*x33+ ...)cc code coverage (0, 1)x- characteristiciai importance coefficient Coefficients are important 68. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 68Code coverage monitoring Build by build Platform to platform JCov comparison report Homegrown database solution (*) Requires an apparatus to compare the coverage data* More at the end of the presentation 69. Compare two run on the same build Comparison reportjava -jar jcov.jar RepGen file1.xml file2m.xmldemoCopyright 2014, Oracle and/or its affiliates. All rights reserved. 69 70. demoCopyright 2014, Oracle and/or its affiliates. All rights reserved. 70Compare two runs of different builds Bytecode is not comparable. Loose the detailsjava -jar jcov.jar Merger -loose blocks file1.xml file2m.xml Only methods leftComparison reportjava -jar jcov.jar RepGen file1.xml file2m.xml 71. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 71Subtraction Compare two suites A and B In (cca ccb) only that code which is covered in ccaand not in ccb Using JCov APILoad cc, ccabWalk over ccaSearch in ccbIf found null the coverage Review (cca cc) and (ccbb cc)ademo 72. Test base reduction Collect coverage for tests individually and together. Compare coverage from every test to the combined coverage.if a test does not add coverage, execute it less Conscious choice.Remember that code coverage proves nothing.Only use for acceptance test cycles.Do not throw tests away run every test although less often.Analyze tests with give no additional coverage.Copyright 2014, Oracle and/or its affiliates. All rights reserved. 72 73. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 73Using the data 100% coverage. Needed? Possible? Enough? Getting coverage subsets to cover 100% Public API UI Controller Monitoring coverage Test base reduction Prioritizing test dev by ranking Comparing suites, runs 74. More info Wiki: https://wiki.openjdk.java.net/display/CodeTools/jcov Source: http://hg.openjdk.java.net/code-tools/jcov Tutorial: http://hg.openjdk.java.net/code-tools/jcov/raw-file/tip/examples/tutorial/ How to build JCov: https://wiki.openjdk.java.net/display/CodeTools/How+To+Build+ Copyright 2014, Oracle and/or its affiliates. All rights reserved. 74Pragmatic code coverage on slideshare.net 75. Copyright 2014, Oracle and/or its affiliates. All rights reserved. 75More to come PluginsIDE: Netbeans, Eclipse, Intellij Ideamaven BackendCoverage storage databaseTrends, statisticsRanking Test base reductionHey, it's open-source! Contribute. 76. Copyright 2014, Oracle and/or its affiliates. All rights reserved.Java Code Coveragewith JCovImplementation Details and Use CasesAlexandre (Shura) IlineJDK Test ArchitectJDK SQE teamSept 1, 2014