I am trying to define the location, where jacoco will create the coverage file for instrumentation tests running on real devices.
From the --debug run of the gradle task I see this log:
[DEBUG] [org.gradle.api.Task] DeviceConnector 'Nexus 5X - 6.0.1': installing /home/martin/workspace/lib/my-lib/build/outputs/apk/my-lib-debug-androidTest-unaligned.apk
[INFO] [org.gradle.api.Task] Starting 1 tests on Nexus 5X - 6.0.1
[INFO] [org.gradle.api.Task] de.my.lib.utils.UtilsTest testMyTest[Nexus 5X - 6.0.1] [32mSUCCESS [0m
[DEBUG] [org.gradle.api.Task] DeviceConnector 'Nexus 5X - 6.0.1': fetching coverage data from /data/data/de.my.lib.test/coverage.ec
[DEBUG] [org.gradle.api.Task] DeviceConnector 'Nexus 5X - 6.0.1': uninstalling de.my.lib.test 13:46:14.538
[DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter] Finished executing task ':my-lib:connectedDebugAndroidTest'
I tried 3 ways to define the location:
Using the <instrumentation> tag in the manifest file didn't change anything.
<?xml version="1.0" encoding="utf-8"?>
<manifest
package="de.my.lib.test"
xmlns:android="http://schemas.android.com/apk/res/android">
<instrumentation
android:name="android.support.test.runner.AndroidJUnitRunner"
xmlns:tools="http://schemas.android.com/tools"
android:targetPackage="de.my.lib.test"
tools:replace="android:targetPackage">
<meta-data
android:name="coverage"
android:value="true" />
<meta-data
android:name="coverageFile"
android:value="/sdcard/coverage.ec" />
</instrumentation>
</manifest>
I tried it with gradle but the output was the same:
defaultConfig {
// unimportant stuff
testApplicationId "de.my.lib.test"
testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
testInstrumentationRunnerArgument('coverageFile', '/sdcard/coverage.ec')
}
And finally I tried it with adb command:
adb shell am instrument -w -e coverage true -e coverageFile /sdcard/coverage.ec de.my.lib.test/android.support.test.runner.AndroidJUnitRunner
But there I get 2 errors:
de.my.lib.utils.UtilsTest:.
Could not find class: org.jacoco.agent.rt.internal_773e439.CoverageTransformer
.
Time: 0,072
OK (1 test)
Error: Failed to generate emma coverage.
I am completely lost here. Any ideas?
Background Why I need it to have it stored in another place: There is a bug with adb shell run-as command on some devices and Android version so I have devices in my test device farm which return 0% coverage because the file can't be pulled. So I need the file to be stored in a publicly available location.
However, the coverage report is not generated yet. To enable this option, we need to add a property to our debug build variant. Using the Android plugin DSL, we can enable the coverage through the testCoverageEnabled property:
android {
...
buildTypes {
debug {
testCoverageEnabled true
}
...
}
}
To enable the coverage report for local tests when using version 2.2.+ of Android Gradle plugin, you need to enable it in your app’s build.gradle:
android {
//...
testOptions {
unitTests.all {
jacoco {
includeNoLocationClasses = true
}
}
}
}
The instrumentation performed by Jacoco produces execution files that contain the necessary data to create the report (HTML, XML, etc.). The problem here, is that Espresso generates .ec file, while the unit tests execution generates .exec file… we have different formats!
As we are not able to configure the coverage task of Espresso, we must ensure that it is executed first. Next, we need to run unit tests, and then create the coverage data with both files (ec and exec).
To enable this, we need to edit our task once more and add the coverage.ec file as a parameter in executionData property:
apply plugin: 'jacoco'
task jacocoTestReport(type: JacocoReport, dependsOn: ['testDebugUnitTest', 'createDebugCoverageReport']) {
reports {
xml.enabled = true
html.enabled = true
}
def fileFilter = ['**/R.class', '**/R$*.class', '**/BuildConfig.*', '**/Manifest*.*', '**/*Test*.*', 'android/**/*.*']
def debugTree = fileTree(dir: "${buildDir}/intermediates/classes/debug", excludes: fileFilter)
def mainSrc = "${project.projectDir}/src/main/java"
sourceDirectories = files([mainSrc])
classDirectories = files([debugTree])
executionData = fileTree(dir: "$buildDir", includes: [
"jacoco/testDebugUnitTest.exec",
"outputs/code-coverage/connected/*coverage.ec"
])
}
As Android Gradle plugin 2.2.+ now generates a coverage file for each execution, using the device / emulator in the file name, now we need to pass every file to execution data, as the file name is now dynamic. In addition, I added the createDebugCoverageReport task as a dependency of our custom task, so we don’t need to run it manually :)
Finally, when we execute both tasks, running the Espresso tests first, we will get the unified coverage report!
gradle clean jacocoTestReport
For detailed explanation see here.
Edit:
So, the command adb shell am instrument does not generates the report, for this you have to handle it using gradle solution. If you really want to test it your self, then you've to use a custom runner.
References:
1) How to Generate Android Testing Report in HTML Automatically
2) How to retrieve test results when using "adb shell am instrument"
3) https://github.com/jsankey/android-junit-report
Test from the command line
Updated to Command line method follow these steps.
Running the instrumented application
Once we have the EmmaInstrumentation class (is in ref link) in place we need a few more adjustments to be able to get the coverage report of the running application.
Firstly, we need to add the new Activity to the manifest. Secondly, we should allow our application to write to the sdcard if this is where we decided to generate the coverage report. To do it you should grant the android.permission.WRITE_EXTERNAL_STORAGE permission.
Then, it's time to build and install the instrumented apk:
$ ant clean
$ ant instrument
$ ant installi
Everything is ready to start the instrumented application
$ adb shell am instrument -e coverage true \
-w com.example.i2at.tc/\
com.example.instrumentation.EmmaInstrumentation
Here is Source code.
the missing CoverageTransformer class hints for a missing dependency to the JaCoCo agent runtime. one simple solution approach might be, to provide it with what it demands (to be available on device):
testImplementation "org.jacoco:org.jacoco.agent:0.8.3"
in case you get a FileNotFoundException, when the ClassNotFoundException was fixed, make sure to have the permission to write to SD card, which appears to be another possible pitfall:
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
as I've just seen, there is also an org.jacoco.report package, which could be optionally added:
testImplementation "org.jacoco:org.jacoco.report:0.8.3"
package org.jacoco.agent is in every case the one which contains class CoverageTransformer.
For us it was an AOSP app. We were generating the test and module apks with disabling Jack compiler which was causing this further issue. Because for us Jack compiler was needed to generate coverage.em. But the error was not quite relevant. So had no clue to go any further.
So the fix was to run the jack compiler server and get the apks generated using it.
Related
I'm trying to migrate our app to using Espresso for UI testing, but I'm unable to make Gradle find my tests with a device running Android 4.4 (API 19, our minimum deployment target). Tests on Android 6.0 (API 23) run fine. I added the JUnit runner and the dependencies to app/build.gradle according to this and this (I excluded annotations because of version conflicts between the modules):
android {
...
defaultConfig {
...
testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
}
}
dependencies {
...
androidTestCompile("com.android.support.test.espresso:espresso-core:2.2.2") {
exclude module: 'support-annotations'
}
androidTestCompile("com.android.support.test:runner:0.5") {
exclude module: 'support-annotations'
}
androidTestCompile("com.android.support.test:rules:0.5") {
exclude module: 'support-annotations'
}
}
Then I created the required directory structure app/src/androidTest/java/ and a package com.companyname.appname and a java class EspressoTest.java with some mock test code:
#RunWith(AndroidJUnit4.class)
public class EspressoTest {
#Rule
public ActivityTestRule<TermsOfUse> termsOfUseActivityTestRule = new ActivityTestRule<>(TermsOfUse.class);
#Test
public void iAmAtTouView() {
onView(withId(R.id.terms_of_use_content)).check(matches(isDisplayed()));
}
}
If I right-click on the test class EspressoTest and choose "Run 'EspressoTest'" I get an error:
$ adb shell am instrument -w -r -e package com.companyname.appname -e debug false com.companyname.appname.qa.test/android.support.test.runner.AndroidJUnitRunner
Client not ready yet..
Started running tests
Test running failed: Instrumentation run failed due to 'Process crashed.'
Empty test suite.
Also if I enter ./gradlew connectedAndroidTest in the command line I get:
Starting 0 tests on GT-I9305 - 4.4.4
Tests on GT-I9305 - 4.4.4 failed: Instrumentation run failed due to 'Process crashed.'
com.android.builder.testing.ConnectedDevice > No tests found.[GT-I9305 - 4.4.4] FAILED
No tests found. This usually means that your test classes are not in the form that your test runner expects (e.g. don't inherit from TestCase or lack #Test annotations).
:app:connectedAuditDebugAndroidTest FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:connectedAuditDebugAndroidTest'.
> There were failing tests. See the report at:
...
So to me it seems like Gradle doesn't recognize my test class when trying to execute test tasks with an Android device. How can I fix this?
Any chance you had to enable multidex, because your *-androidTest.apk contained more than 64k methods?
If so, you need to make sure that the first dex file (classes.dex) contains all the *Test classes that you want to run. You can configure this like this:
android {
defaultConfig {
multiDexEnabled true
multiDexKeepProguard file('multidex-config.pro')
}
}
and in your multidex-config.pro you add regular -keep rules for your test classes, e.g.
-keep class path.to.my.package.**.*Test { *; }
If your test classes have dependencies outside of your own package path, be sure to include those as well with separate -keep statements. Look out for java.lang.ClassNotFoundException in adb logcat to identify missing classes.
First check what tasks are avaliable for gradle to run by:
gradle <app-module>:tasks
or
gradle --gui
See if the task you are running is in the list of tasks. Also which machine are you trying to run: MacOS or Windows? Always run gradle with 'sudo' if not at least sync your project from android studio always a good idea before running the tests
Added additional info for Thomas Keller's answer:
-keep path.to.my.package.**.*Test { *; }
has syntax error, it should be:
-keep class path.to.my.package.**.*Test { *; }
According to Android guide:
https://developer.android.com/studio/build/multidex?#multidexkeepproguard-property
It worked for me. I run instrument test on Android 4.4.
When I found the key message in logcat that the test runner only search classpath in the main DEX path, even though the secondary dex is already processed by androidx.multidex.
D/dalvikvm( 7758): DexOpt: --- END 'com.byted.bytexx.test-1.apk.classes2.zip' (success) ---
D/dalvikvm( 7758): DEX prep '/data/data/com.byted.bytexx.test/code_cache/secondary-dexes/com.byted.bytexx.test-1.apk.classes2.zip': unzip in 109ms, rewrite 1745ms
I/MultiDex( 7758): install done
I/MonitoringInstr( 7758): Instrumentation started!
I/MultiDex( 7758): Installing instrumentation
I/MultiDex( 7758): Installation done
...
I/TestRequestBuilder( 7758): Scanning classpath to find tests in paths [/data/app/com.byted.bytexx.test-1.apk]
...
D/TestExecutor( 7758): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
I/TestRunner( 7758): run started: 1 tests
I/TestRunner( 7758): run finished: 0 tests, 0 failed, 0 ignored
I/MonitoringInstr( 7758): waitForActivitiesToComplete() took: 0ms
D/AndroidRuntime( 7741): Shutting down VM
And I found my test code is not in the main DEX.
I'm building an Android application using Jenkins pipeline and Gradle.
I'd like to run Lint tests on the code and so the command which is used for that matter is:
./gradlew lintStagingDebug
For some reason this Gradle task is not creating the report.
I know that if I'd run lint like so:
lint <project_dir> --xml xml_dest_path
It would create the report.
But since through out the whole pipeline I'm using gradlew to run the relevant tasks (clean, lint, compile, unittest, assemble) I'd like to use gradlew for this task as well but I'm not sure how to tell Gradle to export the lint report.
Another thing I've tried is to edit the project/build.gradle file and change this section:
lintOptions {
abortOnError false
}
to (according to Lint official documentation)
lintOptions {
abortOnError false
xmlOutput projectDir/lint-results.xml
}
But then I get an error:
A problem occurred evaluating project ':App01'.
> No signature of method: java.io.File.div() is applicable for argument types: (com.android.build.gradle.tasks.Lint_Decorated) values: [task ':App01:lint']
Anyone knows how it can be done?
You can provide the XML report output file in Gradle in the following way:
lintOptions {
xmlOutput file("lint-results.xml")
}
The xmlOutput method requires a file type argument. I have used the relative path in the argument. You can also use it with projectDir.
lintOptions {
xmlOutput file("$projectDir/lint-results.xml")
}
Here's how I solved it.
stage('Lint run') {
sh """
export HOME=$GRADLE_USER_HOME
export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"
OUTPUTFILE="lint-results.txt"
touch \$OUTPUTFILE
./gradlew lint${BUILDFLAV}${BUILDTYPE} -x lint 2>&1 | tee \$OUTPUTFILE
"""
})
Explanation:
In order to keep using gradlew to run lint test on the code, I've defined an output file and added at the end of the gradlew command redirection of stdOut and stdErr to this output file using the Linux command "tee".
How do I find the code coverage for instrumentation test case, the run tests with code coverage button is disabled.
Please help.
Before that you need to enable Coverage in gradle file at app level .
buildTypes {
debug
{
testCoverageEnabled = true
}
}
Once done you can run below command in Terminal
gradlew createDebugCoverageReport
You can check it at below path
YoutProjectName\app\build\reports\coverage\debug
and check index.html file in your browser.
I try to run an Instrumentation Test for a certain flavor of my android app and always get this:
Test running failed: Unable to find instrumentation info for: ComponentInfo{<packackename>/android.support.test.runner.AndroidJUnitRunner}
Empty test suite.
This happens only in a certain productFlavor where I had to change the packagename in the build.gradle "manually" with:
applicationVariants.all { variant ->
def flavorName = variant.getVariantData().getVariantConfiguration().getFlavorName()
def mergedFlavour = variant.getVariantData().getVariantConfiguration().getMergedFlavor();
if (flavorName.toLowerCase().contains("foobar")) {
mergedFlavour.setApplicationId(mergedFlavour.getApplicationId() + ".foobar")
}
}
I already tried to do the same thing in the android-testing.gradle but actually all packagenames are looking fine:
Installing de.test.foobar.debug
DEVICE SHELL COMMAND: pm install -r "/data/local/tmp/de.test.foobar.debug"
pkg: /data/local/tmp/de.test.foobar.debug
Success
Installing APK:
Uploading file to: /data/local/tmp/de.test.foobar.debug.test
Installing de.test.foobar.debug.test
DEVICE SHELL COMMAND: pm install -r "/data/local/tmp/de.test.foobar.debug.test"
pkg: /data/local/tmp/de.test.foobar.debug.test
Success
If I remove the "manual" change of the packagename of this specific productflavor, all tests are executed fine.
I already tried to change the instrumentationRunner and made sure that it is the same in the configuration of the test execution - but unfortunately without luck...
Is there maybe a possibility to override the changed applicationId if the test is executed?
Thanks for every input!
if you recently upgraded to Android Studio 2.x and have enabled MultiDex (instant run need it at some point), you need to add the following in your gradle.build
defaultConfig {
...
multiDexEnabled true // <- if you have this you need the following
testInstrumentationRunner "com.android.test.runner.MultiDexTestRunner"
...
}
The testApplicationId needs to match the applicationId for each flavor you modify so the Test Runner can find the tests. You can do this by using testVariants along with applicationVariants.
Something like this in your case:
testVariants.all { variant ->
def flavorName = variant.getVariantData().getVariantConfiguration().getFlavorName()
def mergedFlavour = variant.getVariantData().getVariantConfiguration().getMergedFlavor();
if (flavorName.toLowerCase().contains("foobar")) {
mergedFlavour.setApplicationId(mergedFlavour.getApplicationId() + ".foobar")
}
}
Can I have gradle tests report for android tests? I want to have the report directly in the terminal. Every time a test is passed, failed or skipped I want to have the report printed in the terminal with the name of the test and the status.
For Unit tests I can add this to the gradle file to have the reports:
tasks.withType(Test) {
testLogging {
exceptionFormat "full"
events "passed", "skipped", "failed"
}
}
I didn't find any solution to the android tests.
You can try this (add to app/library build.gradle)
tasks.withType(com.android.build.gradle.internal.tasks.AndroidTestTask) { task ->
task.doFirst {
logging.level = LogLevel.INFO
}
task.doLast {
logging.level = LogLevel.LIFECYCLE
}
}
There's a bug filed to the Platform tools team that is still not resolved:
https://code.google.com/p/android/issues/detail?id=182307
You can vote it to help. The workaround didn't work for me, but if you use Gradle Plugin >= 1.5.0 you can run the command with info logs enabled like this:
./gradlew cC --info
and then you'll see the tests being printed with no extra configuration.
Hope it helps!
Have you tried looking into app/build/reports/tests/debug/index.html or app/build/test-results/?
By using the new test support library you should find your reports there, with no extra-configuration needed.
I've made some investigation, and universal variant to show unitTests and AndroidTests in the terminal is put '--debug' option on the gradle terminal command.
Example:
./gradlew cleanTestDebugUnitTest testDebugUnitTest cleanConnectedDebugAndroidTest connectedDebugAndroidTest --debug