Develop Reference

Gradle ConnectedAndroidTest - Attach Test Listener

I'm working on an SDK that uploads real-time Android test results to a backend. I'd like to be able to attach a test listener to the instrumented tests that run when the connectedAndroidTest gradle command is invoked.
I can annotate the tests with #RunWith and have them use a custom runner with its own listener, but this solution is not ideal, as users implementing this SDK will have to annotate each and every test. We're looking for something that works at a global level.
For unit tests, we can hook into the gradle test API and listen for test results. But, instrumented tests use different gradle APIs, provided by the Android Gradle Plugin (this is where the connectedAndroidTest task comes from). There doesn't seem to be an equivalent gradle-based test listener.
I can set a JUnit test listener via the testInstrumentationRunner in build.gradle - but this listener runs on the Android device under test - I'm looking for something that can listen to test results on the local machine, rather than the Android device.
Android Studio seems to have this capability - when running Instrumented Tests via a run configuration, the IDE can report on instrumented test results in real-time. I've had a dig around in the source code, but it's not clear to me how this works.
Any help would be greatly appreciated!

Android: configuring CICD pipeline to support/ run Integration or UI Tests on Bitrise

I am working on an Android project. I am trying to build a CICD pipeline for my application on the Bitrise. I am following an example on the Bitrise official page, https://devcenter.bitrise.io/testing/device-testing-for-android/. So I added Android Build and [Beta] Virtual Device Testing steps into the workflow in my Bitrise Dashboard.
But when I build the app, it is only running the tests within the app/src/test folder. I have also written UI Tests using the Expresso framework in the app/src/androidTest folder. But the ones in the app/src/androidTest are not run. What changes do I have to make in order to run those tests?

There are three types of Android tests that can be run with the Virtual Device Testing step:
Robo, the testing monkey that can be configured for end-to-end testing.
Game Loop, which I suppose you would use if your app was a game.
Instrumentation, this will run tests written in androidTest/
In order to run instrumentation tests you will need to build both an app.apk and appTest.apk, conveniently Bitrise has a step called Android Build for UI Testing
Make sure to change the Test type input variable.

Porting JUnit tests to Android Testing Framework

I'm planning to port my JUnit tests to Android Testing framework.
Some of the tests only involve the JVM but not the Android system, is it still necessary to port them? Android is using Dalvik and will replace it with ART(Android Runtime) in Lollipop, both of which are different from the JVM (Please correct if I'm wrong.). In this sense, it seems necessary to port all the JUnit tests to the Android Testing framework.
However, some tutorial argues that
"If possible, you should prefer to run your unit tests directly on the
JVM as the test execution is much faster compared to the time required
to deploy and run the test on an Android device."
http://www.vogella.com/tutorials/AndroidTesting/article.html#androidtesting
I'm not an expert on JVM, Dalvik nor ART. Can someone clarify this issue? Is it necessary to port the tests that only involve JVM to Android Testing Framework?
Thanks!

This depends on what you would like to test and the environment in which you run your tests in.
In theory, pure non-Android-specific Java code (aka. POJOs) you write will work on whatever virtual machine you run it on--JVM, Dalvik, or ART.
If you agree with that theory, then if you are already running your tests on a JVM, there is no need to run it on Dalvik or ART, especially if you consider the speed at which JVM will execute the tests. (Running it on Dalvik involves building the apk, and installing it, and running an adb shell command which can take some time...)
Of course, if you feel that the behavior of your Java code may change depending on the runtime, then you should probably port those tests over to your Android tests.

Recently, I checked out Android Testing Support provided in Android SDK.
As far as I know, you can write Unit Tests, which can be executed directly on JVM in your computer or CI Server (e.g. Jenkins) without Android device or emulator. The only requirement is the fact that these tests and code, which is tested have to be written in pure Java without any dependencies to Android SDK and Android Context. These tests have to be located at src/test/java directory. You can switch test artifact in Build Variants in Android Studio to "Unit Tests" and run your tests with IDE. you can also run them via Gradle wrapper with the command: ./gradlew test
If your tests or code, which is going to be tested depend on Android SDK and Android Context, you should use Android Instrumentation tests, which have to be located in src/androidTest/java directory. These tests have to be executed on Android device or emulator. You can switch test artifact in Android Studio to "Android Instrumentation Tests" and run your tests with IDE or use Gradle wrapper command ./gradlew connectedCheck
You can check official article about Unit Testing at: http://tools.android.com/tech-docs/unit-testing-support . It's worth reading. It's a bit outdated, because now you can use build tools: 1.1.0 instead of 1.1.0-rc1, but rest of the information is up to date. After reading this article, you should understand this topic better.

Run robolectric and android unit test from Android studio

I currently have various AndroidTestCases which use Mockito and the module dexmaker-mockito. I am now trying to add Robolectric test cases using the robolectric plugin.
I have everything running fine except that when I try to mock methods with no arguements I get an IllegalArgumentException from Dexmaker. Removing the dexmaker-mockito dependency and using mockito version 1.9.0 instead of 1.9.5 causes the error to go away but causes all of my AndroidTestCases to fail. Is there anyway in gradle to have dexmaker-mockito exclude from my robolectric test cases? Or does anyone know of any other way to run both android unit test and robolectric unit test from android studio?
Error:
java.lang.IllegalArgumentException at com.google.dexmaker.mockito.InvocationHandlerAdapter.invoke
Update
I tried gradle-android-test-plugin and robolectric-gradle-plugin but was unable to to get rid of the IllegalArguementException without changing the version of mockitio to 1.9.0 and removing dexmaker, dexmaker-mockito from my dependencies.
I tried the guide but was unable to get Android Studio to recognize my source directories without adding them to main or test source sets which break my build because robolectric is not compiled for those gradle tasks and I do not want it to be.
I ended up following the guide here this time the error disappeared when I ran the tests from command line but persisted when running from android studio which I found odd. Again changing the version of mockitio to 1.9.0 and removing dexmaker, dexmaker-mockito from my dependencies in my Android App removed the error. For now I will settle for not being able to run the unit tests from android studio until I find a better answer.

If I understand correctly what dexmaker-mockito is supposed to do, it's meant to be used with Android Unit Tests that run on device, so it's good for your Android Test Cases.
Robolectric tests run on the JVM and should be run as JUnit test, not Android tests. They should not be using dexmaker-mockito as they never get deployed to device, they can just use the normal Mockito libraries, with scope set to test.
I think you may not be able to mix the two, you may have two keep your robo tests separate from your Android unit tests.

Is there a way to only run a specific set of instrumentation tests in an Android Gradle project?

I have an Android/Gradle project. Whenever I want to run tests, I run:
./gradlew connectedInstrumentTest
which runs all my tests under the test folder of my project.
My test folder has several automation tests as well as non-automation tests. I'm mostly interested in running the fast non-automation tests without the slow automation tests.
Is there a way to run just a specific set of tests, such as from one specific class or anything similar? I'm basically asking about any kind of separation so that I can choose to run just a few tests when I want to.
Created a sample project here.
Edit local.properties to point at your Android SDK.
Next, start up an emulator or connect a phone to your computer. Then you can run tests using ./gradlew connectedInstrumentTest --info. This runs all tests.
What I am unable to figure out is how to only run tests in, say, one class and not all tests.

Since Android Gradle Plugin 1.3.0
Starting from version 1.3.0 you can (finally!) specify the arguments the Android Gradle Plugin have to pass to the InstrumentationTestRunner.
For example, if you want to run only the tests annotated with #SmallTest and ignore the others:
android {
//....
defaultConfig {
//....
testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
testInstrumentationRunnerArgument "size", "small"
}
}
Old workaround
Prior to plugin 1.3.0 is not possible to do that but I've found a little workaound. Basically I've annotated with the #SmallTest annotation the fast tests and using a custom subclass of the InstrumentationTestRunner I'm able to run just them and not the whole suite.
You can found the example code in this gist.

Yes, please have a look here. It should also work with an android project. Unfortunately as far as I know there's no way to rung single method - You can limit to whole suit only.

Also you can install test suit on the device and run it manually through adb. See android docs