Develop Reference

AGP 7.2.x jacoco coverage data is unexpectedly low

Starting with AGP 7.2.0-alpha01 and onwards (including 7.2.0, 7.3.x, and 7.4.x preview builds), jacoco coverage is not working correctly despite all config/information seemingly being setup/working as expected.
On AGP 7.1.3, coverage shows up to the expected %s (very high for my app I am using - above 90% lets say) - things are healthy. When I switch to any newer AGP version, coverage drops to nearly something like 10% or less. Some coverage data does show up, but most of the classes in the app show as uncovered. I don't see a pattern for why some data shows up but some does not. Classes tested only via instrumentation tests and other classes tested only via regular unit tests both have data missing - and the opposite exists too (meaning the data shows up for a small number of classes in these cases), so it's very confusing...
I setup my jacoco task like so:
task developDebugCodeCoverageReport(type: JacocoReport, dependsOn: ['testDevelopDebugUnitTest', 'createDevelopDebugCoverageReport']) {
group = 'Code Coverage'
description = 'Generate Jacoco coverage reports for the developDebug build'
reports {
html.required.set(true)
xml.required.set(true)
}
def excludes = [
// list of files I want to exclude
]
def javaClasses = fileTree(dir: "${buildDir}/intermediates/javac/developDebug/classes", excludes: excludes)
def kotlinClasses = fileTree(dir: "${buildDir}/tmp/kotlin-classes/developDebug", excludes: excludes)
classDirectories.setFrom(files([
javaClasses,
kotlinClasses
]))
sourceDirectories.setFrom("${project.projectDir}/src/main/java")
def junitTestsData = "${buildDir}/outputs/unit_test_code_coverage/developDebugUnitTest/testDevelopDebugUnitTest.exec"
def androidTestsData = fileTree(dir: "${buildDir}/outputs/code_coverage/developDebugAndroidTest/connected/", includes: ["**/*.ec"])
executionData.setFrom(files([
junitTestsData,
androidTestsData
]))
}
I assume AGP 7.2.x made some sort of change that broke how jacoco is able to read the coverage data. The exec and ec files that are generated are different sizes after updating the AGP version... but I don't know enough about jacoco/AGP configs to continue on my own.
I'm trying to setup a small sample app that demonstrates the issue, I fear it's a related complexity with one of the dependencies I am using, but we'll see... in the meantime - anyone have any thoughts?

Run all subprojects tests using Gradle

I have a multi-module android project with the structure:
:a
:b
:c
:d
:e
I'm trying to run a jacoco report on module :b so that it runs on :b, :c, :d, and :e without running :a. I want all of the xml reports to be in a common folder with names of their project.xml (e.g. b.xml, c.xml, etc.) I have a pretty standard jacoco setup
task jacocoTestReport(type: JacocoReport, dependsOn: ['testDebugUnitTest', 'createDebugCoverageReport']) {
reports {
xml.enabled = true
xml.destination = file(allTestCoverageDir + project.name + ".xml")
html.enabled = true
}
def fileFilter = [
'**/R.class',
'**/R$*.class',
'**/BuildConfig.*',
'**/Manifest*.*',
'**/*Test*.*',
'android/**/*.*',
//Dagger 2
'**/*Dagger*Component*.*',
'**/*Module.*',
'**/*Module$*.*',
'**/*MembersInjector*.*',
'**/*_Factory*.*',
'**/*Provide*Factory*.*',
]
def kotlinDebug = [fileTree(dir: "${project.buildDir}/tmp/kotlin-classes/debug", excludes: fileFilter)]
def mainSrc = files([
"$project.projectDir/src/main/java",
"$project.projectDir/src/main/kotlin"
])
sourceDirectories = files([mainSrc])
classDirectories = files(kotlinDebug)
executionData = fileTree(dir: project.buildDir, includes: [
'jacoco/testDebugUnitTest.exec', 'outputs/code-coverage/connected/*coverage.ec'
])
}
But when I try to loop through subprojects in a doLast block, the doLast block never runs and trying to access subprojects before that also shows that :a has no subprojects.
Edit I am able to run these for each sub project with ./gradlew b:jacocoTestReport or ./gradlew c:jacocoTestReport and all the reports and in a folder with the correct names. But as my project grows I don't want to have to run dozens of commands (one for each module) I want a single command ./gradlew b:jacocoTestReport (or something similar) which runs for b and it's subtree

As far as I understand at the moment you're using allprojects {} to configure all of the sub projects. Although this has been the canonical to configure a group of projects in the past, it is now discouraged. Furthermore projects should use publications to interface with each other instead of copying files across project boundaries. So you need to do two things:
Instead of configuring the sub projects from the root root project, you should create a plugin to configure jacoco and create configuration that will hold the reports.
To do this, create a pre-compiled script plugin in your project. The idea is to have a kotlin build script in the buildSrc project and create a Gradle plugin from that file on the fly. So you should move the logic that configures jacoco to the file buildSrc/src/main/kotlin/jacoco-conventions.gradle.kts:
plugins {
jacoco
}
val jacocoTestReport by tasks.getting(JacocoReport::class) {
// jacoco configuration
}
configurations.create("jacocoReports") {
isCanBeResolved = false
isCanBeConsumed = true
attributes {
attribute(Usage.USAGE_ATTRIBUTE, project.objects.named(Usage::class, "jacocoReports"))
}
outgoing.artifact(jacocoTestReport.reports.xml.destination) {
builtBy(jacocoTestReport)
}
}
The last part creates a new configuration in the project the pre-compiled script plugin is applied to. This configuration uses the xml destination file which is builtBy the jacoco report task as an outgoing artifact. The important part here is the USAGE_ATTRIBUTE because we will need this later on to consume the files.
The precompiled script plugin can now be applied in the projects where you want to gather jacoco metrics by:
// for example in c/build.gradle.kts
plugins {
`jacoco-conventions`
}
Now you have configured the sub projects to put the Jacoco xml reports into configurations with usage attribute jacocoReports.
In the root project create a task that copies the report from the configuration.
To do this we need to setup a configuration that consumes the jacocoReports variants and then depend on the variants of the sub projects:
// main build file
val jacocoReports by configurations.creating {
isCanBeResolved = true
isCanBeConsumed = false
attributes {
attribute(Usage.USAGE_ATTRIBUTE, project.objects.named(Usage::class, "jacocoReports"))
}
}
dependencies {
jacocoReports(project(":b:c"))
jacocoReports(project(":b:d:e"))
// other jacocoReports you want to consume
}
tasks.register<Copy>("aggregateJacocoReports") {
from(jacocoReports)
into(file("$buildDir/jacoco"))
}
As you can see, the jacocoReports configuration has the same usage attribute, so it can be used to resolve files that are in configurations with the same attribute. Then we need to define which project reports we want to consume. This is cone by defining project dependencies using the jacocoReports configuration. The last step is a simple copy task that copies the files into the build directory of the root project. So now when you call ./gradlew aggregateJacocoReports this task resolves all the files from the jacocoReports configuration, which in turn will create the jacoco report for all projects that are the root project has a dependency on.
Why is this better than cross configuration? If projects are not entangled by cross configuration and by tasks that copy stuff between projects, gradle can more efficiently schedule and parallelize the work that has to be done.
I have a created a minimal example that should help you to setup your project this way: https://github.com/britter/gradle-jacoco-aggregate. I have removed the android specific configuration to keep it simple, but I'm sure you will figure it out.

Powermock Jacoco Gradle 0% Coverage For Android Project

We have an Android project, and we're using Powermock for some of our test cases and Jacoco for coverage report. We noticed that some our classes are returning as 0% coverage although they are indeed covered. We also observed the message below for affected classes.
"Classes ... do no match with execution data."
A few searches online show that Powermock and Jacoco don't play well and that Offline Instrumentation is a possible workaround.
Has anyone used gradle Offline Instrumentation script for android projects before?

In hindsight, I guess this can be solved with enough android experience and online perusing. However, I was (and still am) relatively new to Android, gradle, and groovy when this fell on my lap, so I'm writing this for the next me :-D
WHAT IS HAPPENING IN A NUTSHELL (excerpt from a jacoco forum)
Source file is compiled into non-instrumented class file
Non-instrumented class file is instrumented (either pre-instrumented offline, or automatically at runtime by Java agent)
Execution of instrumented classes collected into exec file
report decorates source files with information obtained from analysis of exec file and original non-instrumented class files
Message "Classes ... do no match with execution data." during generation of report means that class files used for generation of report are not the same as classes prior to instrumentation.
SOLUTION
The Jacoco Offline Instrumentation page provides the main steps that should occur for offline instrumentation in this excerpt:
For such scenarios class files can be pre-instrumented with JaCoCo,
for example with the instrument Ant task. At runtime the
pre-instrumented classes needs be on the classpath instead of the
original classes. In addition jacocoagent.jar must be put on the
classpath.
The script below does exactly that:
apply plugin: 'jacoco'
configurations {
jacocoAnt
jacocoRuntime
}
jacoco {
toolVersion = "0.8.1"
}
def offline_instrumented_outputDir = "$buildDir.path/intermediates/classes-instrumented/debug"
tasks.withType(Test) {
jacoco.includeNoLocationClasses = true
}
def coverageSourceDirs = [
'src/main/java'
]
task jacocoTestReport(type: JacocoReport, dependsOn: "test") {
group = "Reporting"
description = "Generate Jacoco coverage reports"
classDirectories = fileTree(
dir: 'build/intermediates/classes/debug',
excludes: ['**/R.class',
'**/R$*.class',
'**/BuildConfig.*',
'**/MainActivity.*']
)
sourceDirectories = files(coverageSourceDirs)
executionData = files('build/jacoco/testDebugUnitTest.exec')
}
jacocoTestReport {
reports {
xml.enabled true
html.enabled true
html.destination file("build/test-results/jacocoHtml")
}
}
/* This task is used to create offline instrumentation of classes for on-the-fly instrumentation coverage tool like Jacoco. See jacoco classId
* and Offline Instrumentation from the jacoco site for more info.
*
* In this case, some classes mocked using PowerMock were reported as 0% coverage on jacoco & Sonarqube. The issue between PowerMock and jacoco
* is well documented, and a possible solution is offline Instrumentation (not so well documented for gradle).
*
* In a nutshell, this task:
* - Pre-instruments the original *.class files
* - Puts the instrumented classes path at the beginning of the task's classpath (for report purposes)
* - Runs test & generates a new exec file based on the pre-instrumented classes -- as opposed to on-the-fly instrumented class files generated by jacoco.
*
* It is currently not implemented to run prior to any other existing tasks (like test, jacocoTestReport, etc...), therefore, it should be called
* explicitly if Offline Instrumentation report is needed.
*
* Usage: gradle clean & gradle createOfflineInstrTestCoverageReport & gradle jacocoTestReport
* - gradle clean //To prevent influence from any previous task execution
* - gradle createOfflineInstrTestCoverageReport //To generate *.exec file from offline instrumented class
* - gradle jacocoTestReport //To generate html report from newly created *.exec task
*/
task createOfflineTestCoverageReport(dependsOn: ['instrument', 'testDebugUnitTest']) {
doLast {
ant.taskdef(name: 'report',
classname: 'org.jacoco.ant.ReportTask',
classpath: configurations.jacocoAnt.asPath)
ant.report() {
executiondata {
ant.file(file: "$buildDir.path/jacoco/testDebugUnitTest.exec")
}
structure(name: 'Example') {
classfiles {
fileset(dir: "$project.buildDir/intermediates/classes/debug")
}
sourcefiles {
fileset(dir: 'src/main/java')
}
}
//Uncomment if we want the task to generate jacoco html reports. However, the current script does not exclude files.
//An alternative is to used jacocoTestReport after this task finishes
//html(destdir: "$buildDir.path/reports/jacocoHtml")
}
}
}
/*
* Part of the Offline Instrumentation process is to add the jacoco runtime to the class path along with the path of the instrumented files.
*/
gradle.taskGraph.whenReady { graph ->
if (graph.hasTask(instrument)) {
tasks.withType(Test) {
doFirst {
systemProperty 'jacoco-agent.destfile', buildDir.path + '/jacoco/testDebugUnitTest.exec'
classpath = files(offline_instrumented_outputDir) + classpath + configurations.jacocoRuntime
}
}
}
}
/*
* Instruments the classes per se
*/
task instrument(dependsOn:'compileDebugUnitTestSources') {
doLast {
println 'Instrumenting classes'
ant.taskdef(name: 'instrument',
classname: 'org.jacoco.ant.InstrumentTask',
classpath: configurations.jacocoAnt.asPath)
ant.instrument(destdir: offline_instrumented_outputDir) {
fileset(dir: "$buildDir.path/intermediates/classes/debug")
}
}
}
Usage
The script can be copied into a separate file. For instance: jacoco.gradle
Reference the jacoco file in your build.gradle. For instance: apply from: jacoco.gradle
Ensure proper dependencies: jacocoAnt 'org.jacoco:org.jacoco.ant:0.8.1:nodeps'
In command line run: gradle clean & gradle createOfflineTestCoverageReport & gradle jacocoTestReport
gradle clean will wipe out any previous gradle execution artifacts
gradle createOfflineTestCoverageReport will create offline instrumentation, change order of classpath, generate .exec file
gradle jacocoTestReport will run test and generate jacoco report based on previously generated .exec file
Feeling Lost?
I've put together a github Jacoco Powermock Android project with sample scripts to reproduce and fix the issue. It also contains more information about the solution.
REFERENCE
https://github.com/powermock/powermock/wiki/Code-coverage-with-JaCoCo
https://www.jacoco.org/jacoco/trunk/doc/classids.html
https://www.jacoco.org/jacoco/trunk/doc/offline.html
https://github.com/powermock/powermock-examples-maven/tree/master/jacoco-offline
https://automated-testing.info/t/jacoco-offline-instrumentations-for-android-gradle/20121
https://stackoverflow.com/questions/41370815/jacoco-offline-instrumentation-gradle-script/42238982#42238982
https://groups.google.com/forum/#!msg/jacoco/5IqM4AibmT8/-x5w4kU9BAAJ

Jacoco code covergage file empty leading to false coverage reports

A Github repo with the code used for this question can be found here:
https://github.com/thenewmr/UnitTestCoverageExample
We've been having serious issues trying to get a code coverage report via Jacoco generated correctly.
We've followed various guides on the internet including this one from Patrick McLaren that he linked to in his answer to this question.
We've also looked at various questions on stack overflow but no joy so far.
Here's what we have so far (in bullet point form so as not to make this question too long to read):
Adding testCoverageEnabled = true to the debug closure created a task called "createDebugCoverageReport"
Running this task:
Produces a report for our Android tests at: app/build/outputs/reports/androidTests/connected/index.html with an accurate report of test passes and failures etc
But an inaccurate coverage report (0% coverage) at: app/build/outputs/reports/coverage/debug/index.html
It also produces what appears to be empty coverage data at: app/build/outputs/code-coverage/connected/coverage.ec
Now, if we add the following:
apply plugin: 'jacoco'
//specify which directories should be examined by jacoco
def coverageSourceDirs = [
'src/main/java'
]
task jacocoTestReport(type:JacocoReport, dependsOn: "testDebug") {
group = "Reporting"
description = "Generate Jacoco coverage reports"
classDirectories = fileTree(
dir: 'build/intermediates/classes/debug',
excludes: ['**/R.class',
'**/R$*.class',
'**/*$ViewInjector*.*',
'**/BuildConfig.*',
'**/Manifest*.*']
)
additionalSourceDirs = files(coverageSourceDirs)
sourceDirectories = files(coverageSourceDirs)
executionData = files('build/jacoco/testDebug.exec')
reports {
xml.enabled = true
html.enabled = true
}
}
to the build.gradle file for the app (with or without the bit above) and run jacocoTestReport, we get:
A test report for the vanilla Unit tests at: app/build/reports/tests/debug/index.html
An accurate code coverage report at app/build/reports/jacoco/jacocoTestReport/html/index.html
So, we get the correct stuff for the vanilla unit tests but not for the Android unit tests.
The blog post mentioned above talks about how to combine the two reports. But it seems it would be pointless if we can’t get the Android test coverage report produced in the first place.
The problem appears to be due to the empty coverage.ec file as mentioned earlier.
This answer says this used to be a bug:
https://stackoverflow.com/a/28080713/487812
and this bug report says this problem was fixed:
https://code.google.com/p/android/issues/detail?id=78556
But perhaps it's been reintroduced since as a regression? Are we missing something obvious?

Though nius' answer is correct, the info below was the solution to our particular problem.
Turns out that, for some bizarre reason, running the tests on Samsung devices yields empty coverage files. Running the same tests on an emulator or non-Samsung phone produced the desired results.
Including this info here so that people are aware of this.

Now (Oct 2015) you can use it as android team has fixed the bug.
android {
...
buildTypes {
debug {
testCoverageEnabled true
}
}
...
dependencies{
androidTestCompile 'com.android.support.test:runner:0.4.1'
// Set this dependency to use JUnit 4 rules
androidTestCompile 'com.android.support.test:rules:0.4.1'
// Set this dependency to build and run Espresso tests
androidTestCompile 'com.android.support.test.espresso:espresso-core:2.2.1'
// Espresso-contrib for DatePicker, RecyclerView, Drawer actions, Accessibility checks, CountingIdlingResource
androidTestCompile 'com.android.support.test.espresso:espresso-contrib:2.2.1'
}
}
Then just use ./gradlew createDebugCoverageReport. Find the reports in app/build/reports/coverage/debug/index.html.

Combine jacoco coverage from androidTest and test

Since the release of 'com.android.tools.build:gradle:1.1.0' I'm moving most of my java test code from androidTest to the test folder because the JVM tests are a lot faster. But I cannot move all tests. I really need the device tests because of some ContentProvider stuff.
I've had 100% code coverage before I started migrating. When I'm currently running the jacoco code coverage I get 40% for the androidTest folder and 71% for the test folder. My code is 100% tested but I have no report proofing this.
Is there a way to combine both reports? I found JacocoMerge but couldn't get it to work.
Here is the output of the androidTest folder: build/outputs/reports/coverage/debug/index.html
And here the output of the test folder
build/reports/jacoco/generateJacocoTestReports/html/index.html
generated with this gradle task:
def coverageSourceDirs = [
'../library/src/main/java'
]
task generateJacocoTestReports(type: JacocoReport, dependsOn: "test") {
group = "Reporting"
description = 'Generate Jacoco Robolectric unit test coverage reports'
classDirectories = fileTree(
dir: '../library/build/intermediates/classes/debug',
excludes: ['**//*R.class',
'**//*R$*.class',
'***/*//*$ViewInjector*.*',
'**//*BuildConfig.*',
'**//*Manifest*.*']
)
sourceDirectories = files(coverageSourceDirs)
additionalSourceDirs = files(coverageSourceDirs)
executionData = files('../library/build/jacoco/testDebug.exec')
}

Not sure if you still need this, but I recently published Gradle plugin which might help you with that: https://github.com/paveldudka/JacocoEverywhere

There is also the gradle plugin https://github.com/palantir/gradle-jacoco-coverage that acording to docs can do the job, too.
I haven-t tried it for one submodul with two different test-parts but it works well for merging the testparts of to two submoduls.
See Gradle jacoco coverage report with more than one submodule(s)? for details

JacocoMerge task can be used to merge 2 or more jacoco execution data.
Below task can be added to the root gradle file and on successful execution of this task, merged execution data can be found under root build directory. (build/jacoco/mergeJacocoReport.exec)
evaluationDependsOnChildren()
//Missing this might be a problem in fetching JacocoReport tasks from sub-modules.
task mergeJacocoReport(type: org.gradle.testing.jacoco.tasks.JacocoMerge) {
group "Jacoco Report"
description "Merge Jacoco Code Coverage Report"
def executionFiles = fileTree("$rootProject.rootDir", {
includes = ['**/*.exec']
})
setExecutionData(executionFiles)
}
subprojects.each { $project ->
def tasks = $project.tasks.withType(JacocoReport)
if (tasks != null) {
mergeJacocoReport.dependsOn << tasks
}
}

In case you use Jenkins with the JaCoCo plugin you can just configure all jacoco.exec and emma.ec files in "Path to exec files" to have a combined coverage reported.
connectedAndroidTest will result in emma.ec files somewhere in "outputs" by default.