I'm working on an SDK that uploads real-time Android test results to a backend. I'd like to be able to attach a test listener to the instrumented tests that run when the connectedAndroidTest gradle command is invoked.
I can annotate the tests with #RunWith and have them use a custom runner with its own listener, but this solution is not ideal, as users implementing this SDK will have to annotate each and every test. We're looking for something that works at a global level.
For unit tests, we can hook into the gradle test API and listen for test results. But, instrumented tests use different gradle APIs, provided by the Android Gradle Plugin (this is where the connectedAndroidTest task comes from). There doesn't seem to be an equivalent gradle-based test listener.
I can set a JUnit test listener via the testInstrumentationRunner in build.gradle - but this listener runs on the Android device under test - I'm looking for something that can listen to test results on the local machine, rather than the Android device.
Android Studio seems to have this capability - when running Instrumented Tests via a run configuration, the IDE can report on instrumented test results in real-time. I've had a dig around in the source code, but it's not clear to me how this works.
Any help would be greatly appreciated!
Related
There are some tests in my android project and configured CI
Before deploying tests are launched using gradle wrapper(4.10.3)
95% of tests are robolectric tests
And there are cases when these tests are not passing because of different reasons, but the problem is that when i run them from android studio GUI tests are working fine(passing).
For example i use mockito for mocking and mock-maker-inline, i have this MockMaker in core module that is used in all other modules, if i launch tests from android studio - they are passing, if i launch using gradle wrapper - they are all failing. It can be solved by adding MockMaker in module, where i have tests, but still why it is working from android studio, but not working from console?
My test running options:
Case when i remove mock maker from locations module(dependent on core module with enabled mock-maker)
All tests are filing with this error
But when i run with android studio tests are passing and everything is fine.
What is the difference between these two launches?(default launch type from android studio and gradlew testDebug).
And is there a way to connect my custom gradle test task with android studio tests gui?
The biggest difference between the GUI and running from terminal is that the Gradle wrapper will execute the tests of different modules concurrently if you have it enabled in gradle.properties:
org.gradle.parallel=true
If you have tests in different modules acting upon the same singleton, or the same mocks, then you can run into any number of conflicts.
I have created a cordova plugin which has an android implementation. This Java code has calls into an SDK that interfaces with specific hardware that the mobile device is connected to. I would like to write unit tests for my Java code and mock all the calls into the SDK so that I can run the tests during CI.
When looking into this, I found the cordova-plugin-test-framework, but from what I can tell, this is for tests that you would write against the javascript code, not the actual platform implementations. I'm sure I could use it and write some tests, but it would require the mobile device to be connected to the hardware, and I don't want to have actual calls into the SDK. In other words, I don't have a way of mocking the SDK calls using this.
I thought maybe I could just run JUnit from the command line with my Java code, but then I was getting errors because it couldn't find org.apache.cordova.CallbackContext. I tried faking my own object, but as I continued, I found more dependencies that my code in isolation just doesn't know about.
Next I figured it would be best to test it within the android project under /platforms in my ionic mobile app, since all the dependencies would be visible at that point. I'm able to run ./gradlew test (https://developer.android.com/studio/test/command-line.html) from here, but I can't figure out how to implement the test cases so that this call will pick them up. Is there some kind of config I can add so it knows where to look for tests? I'd be fine if there was just a certain place I had to put them.
I'm open to any options with this. I just want to be able to run JUnit test as part of our CI process.
I currently have various AndroidTestCases which use Mockito and the module dexmaker-mockito. I am now trying to add Robolectric test cases using the robolectric plugin.
I have everything running fine except that when I try to mock methods with no arguements I get an IllegalArgumentException from Dexmaker. Removing the dexmaker-mockito dependency and using mockito version 1.9.0 instead of 1.9.5 causes the error to go away but causes all of my AndroidTestCases to fail. Is there anyway in gradle to have dexmaker-mockito exclude from my robolectric test cases? Or does anyone know of any other way to run both android unit test and robolectric unit test from android studio?
Error:
java.lang.IllegalArgumentException at com.google.dexmaker.mockito.InvocationHandlerAdapter.invoke
Update
I tried gradle-android-test-plugin and robolectric-gradle-plugin but was unable to to get rid of the IllegalArguementException without changing the version of mockitio to 1.9.0 and removing dexmaker, dexmaker-mockito from my dependencies.
I tried the guide but was unable to get Android Studio to recognize my source directories without adding them to main or test source sets which break my build because robolectric is not compiled for those gradle tasks and I do not want it to be.
I ended up following the guide here this time the error disappeared when I ran the tests from command line but persisted when running from android studio which I found odd. Again changing the version of mockitio to 1.9.0 and removing dexmaker, dexmaker-mockito from my dependencies in my Android App removed the error. For now I will settle for not being able to run the unit tests from android studio until I find a better answer.
If I understand correctly what dexmaker-mockito is supposed to do, it's meant to be used with Android Unit Tests that run on device, so it's good for your Android Test Cases.
Robolectric tests run on the JVM and should be run as JUnit test, not Android tests. They should not be using dexmaker-mockito as they never get deployed to device, they can just use the normal Mockito libraries, with scope set to test.
I think you may not be able to mix the two, you may have two keep your robo tests separate from your Android unit tests.
I have an Android/Gradle project. Whenever I want to run tests, I run:
./gradlew connectedInstrumentTest
which runs all my tests under the test folder of my project.
My test folder has several automation tests as well as non-automation tests. I'm mostly interested in running the fast non-automation tests without the slow automation tests.
Is there a way to run just a specific set of tests, such as from one specific class or anything similar? I'm basically asking about any kind of separation so that I can choose to run just a few tests when I want to.
Created a sample project here.
Edit local.properties to point at your Android SDK.
Next, start up an emulator or connect a phone to your computer. Then you can run tests using ./gradlew connectedInstrumentTest --info. This runs all tests.
What I am unable to figure out is how to only run tests in, say, one class and not all tests.
Since Android Gradle Plugin 1.3.0
Starting from version 1.3.0 you can (finally!) specify the arguments the Android Gradle Plugin have to pass to the InstrumentationTestRunner.
For example, if you want to run only the tests annotated with #SmallTest and ignore the others:
android {
//....
defaultConfig {
//....
testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
testInstrumentationRunnerArgument "size", "small"
}
}
Old workaround
Prior to plugin 1.3.0 is not possible to do that but I've found a little workaound. Basically I've annotated with the #SmallTest annotation the fast tests and using a custom subclass of the InstrumentationTestRunner I'm able to run just them and not the whole suite.
You can found the example code in this gist.
Yes, please have a look here. It should also work with an android project. Unfortunately as far as I know there's no way to rung single method - You can limit to whole suit only.
Also you can install test suit on the device and run it manually through adb. See android docs
I have a Robolectric test project setup, but I'd like to also run these tests on my device to check that I don't get bit by JVM vs Dalvik implementation differences.
Unlike robolectric tests, I won't run these tests frequently. My concern is that there's little effort to maintain the test suite and that they verify actual device functionality.
What's the best way to do that?
What I've currently got:
My robolectric test project as a test case TestPackage. I created an Android Test project with a test case TestRoboOnAndroid. It creates a TestPackage and has a test for each test in TestPackage.
Right now, every time I add a test to my robolectric suite, I need to manually add it to my device suite. Is there some way to do that automatically with reflection?
Also, Robolectric uses JUnit 4 (by default) and Android uses JUnit 3. So I have to write all of my Robolectric tests using JUnit 3 style (importing from junit.framework instead of org.junit).
The whole point of Robolectric is NOT to run it on the device and the design is based on that. If you want to run something on the device look at the default instrumentation tests from the SDK or Robotium.
It is totally feasible to run your Robolectric tests on the JVM and in addition create Robotium tests that rely on device specific interaction (e.g. creating screenshots, hardware differences...) and run on devices and emulators all combined into one build.
The easiest way to do that is to use the Android Maven Plugin.
I use a tiered system, where I prefer earlier tiers where possible:
Pure unit tests. I try to make as much code as possible fully independent of Android APIs, and then use "pure" unit tests which can run on any JVM. These tests are the fastest, and it helps keep code that has no need to be Android-specific portable.
Robolectric-supported unit tests. Where my code has only small dependencies on Android APIs, that can be satisfied by Robolectric shadows, I test it with Robolectric. There is a little more setup time for Robolectric compared to pure tests, but it's still faster than starting/running on an emulator.
Android framework tests. Where Robolectric doesn't cut it - either because the shadows don't exist, or because I'm heavily using Android APIs (and therefore want to test against the Real Thing) - I write test that run on the emulator/device with the default framework.
The point of the tiers is to keep things as simple as possible, which keeps the full suite faster and helps promote cleaner code.
We use Robolectric for unit testing against the JVM and Calabash-Android for system testing against Dalvik. Both can be integrated into our Jenkins CI build and between the two tools I feel that we cover all the bases.