I've not found any post on Stack Overflow that helped, so I'm posting my own question.
I have a custom testInstrumentationRunner that when set doesn't run any tests.
The logs says:
App restart successful without requiring a re-install.
Running tests
adb shell am instrument -w -m --no-window-animation -e debug false -e class 'com.my.app.MyFragmentTest' com.my.app.debug.test/com.my.app.CustomTestRunner
Connected to process on device
When running the normal androidx.test.runner.AndroidJUnitRunner the tests run, but fail, because I need the custom runner.
The runner is set up like the androidx one, in:
defaultConfig {
// testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner
testInstrumentationRunner "com.my.app.CustomTestRunner"
}
It's also located in the com.my.app package in androidTest-folder
The code for the runner is:
package com.my.app
import ...
class CustomTestRunner: AndroidJUnitRunner() {
override fun newApplication(cl: ClassLoader?, className: String?, context: Context?): Application {
DexOpener.install(this)
Timber.i("This should print hopefully")
return super.newApplication(cl, ApplicationTest::class.java.canonicalName, context)
}
}
Not sure if it matters, but here's the code for the application class, which extends the application class used for the app itself:
package com.my.app
class ApplicationTest: MyApplication() {
override fun onCreate() {
super.onCreate()
appContext = applicationContext
initializeDependencyInjection()
}
override fun initializeDependencyInjection() {}
}
I have a test-case I'm trying to run, which contains 7 tests. The androidx runner fails on 7/7 tests. But when I run with the custom runner, I get duration 0ms and 0/0 tests.
So it seems like Android isn't detecting this runner. I also get the same results using a random name in the testInstrumentationRunner, like testInstrumentationRunner "not.a.TestRunner"
I'm sure you've already solved the issue or switched jobs to growing potatoes or something more productive, but I just spent my whole morning debugging this same exact scenario and managed to find a solution:
The custom instrumentation runner you have is not ignored, but there is something broken in your custom Test Application initialization. Probably in your replacement dependency injections, like it was in my case. When you run the tests using the custom runner, the custom Test Application silently crashes immediately, but this is not indicated in the test results or in any other easily visible place. The crash stack trace is, however, accessible in logcat. So solve that first, and then try using your custom test runner again.
Do not mix between Instrumentation Tests and Unit Tests:
Unit Tests
Unit tests runs in local JVM and minimizes the execution time.
Unit Tests cannot test UI of the app without mocking activity objects.
Unit test are used for white box testing for testing code.
Most of the time Unit Tests are written by Developers.
You don’t need a device connected for execution of Unit Tests.
Instrumentation Tests
Instrumentation tests are used for black box testing.
It is used to test GUI of the application along with its functionality in real environment.
For execution of Instrumentation tests you need a device /emulator on which the application is first installed and then tests are executed.
Automation Testers are involved in writing instrumentation framework.
When testing a library module, you can link the test application in src/androidTest/AndroidManifest.xml:
<manifest
xmlns:android="http://schemas.android.com/apk/res/android">
<application
android:name="com.my.app.ApplicationTest"/>
</manifest>
Related
I have written multiple test methods in a single android instrumented test class, when I am trying to run a single test method it will run all methods exists in that class.
I want to run only one.
Earlier I was able to run all, but somehow configuration settings have been changed
class HistoryTest{
#Test
fun openHistoryTest{
}
#Test
fun closeHistoryTest{
}
#Test
fun editHistoryTest{
}
}
I want to run a specific single test method say openHistoryTest.
Currently getting an error - the command line is too long shorten the command line for test "testname"
I updated Android Studio to canary and can run the whole class or a single method as instrumental test. Currently using AS 3.6 Canary 12.
Still doesn't work on Android Studio 3.5. I can't run each method as an
instrumental test, only the whole class.
I gave a try to Android Test Orchestrator and it doesn't see any tests if Application class were changed. Pretty easy to reproduce.
In Android Studio 3 preview Beta 6 create simple project with wizard
Create custom runner like:
class CustomTestRunner : AndroidJUnitRunner() {
override fun newApplication(cl: ClassLoader?, className: String?, context: Context?): Application {
return super.newApplication(cl, TestApplicationClass::class.simpleName, context)
}
}
Replace instrumentation runner with new one
No test found after running instrumented tests
Any ideas? Looks like the Orchestrator depends on application class name from manifest.
I use this configuration to use special Dagger dependencies for tests.
I had the similar issue, with a custom test runner. Make sure that your TestApplicationClass does not crash at runtime. If the custom runner crashes then orchestrator will not be able to get information about the tests and will return the message:
No tests found. This usually means that your test classes are not in the form that your test runner expects (e.g. don't inherit from TestCase or lack #Test annotations).
That is what happened to me in my custom runner.
Good luck!
I am interested in how to run Espresso tests from command line (gradle task) individually (run group/suite tests then close app and then run another group/suite of tests).
Found that it is feasible to implement JUnit Test Suites but do not really understand how does it looks like under the hood in a context of instrumentation tests. Does it starts separate processes per Test suite? There is sample application on Github but how to execute it from terminal?
Another interesting discovery is Sharding tests. However, it one sentence documentation.
May be somebody can share with any experience of running Espresso tests individually.
Most of this is documented as part of AndroidJUnitRunner: https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner.html
The key piece that is missing is how to pass those parameters via Gradle. You can do that by specifying the options at the commandline as such:
./gradlew connectedAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=*The full name of your test suite goes here*
I would recommend using the Spoon library to run your tests individually or in parallel on multiple devices. You can either use the jar file or use the Spoon gradle plugin mentioned on the same page. Both of them have adequate documentation to help you set it up.
You can also use Spoon to run an individual test and the command would look something like this:
./gradlew yourSpoonTaskName -PspoonClassName=com.yourPackageName.blah.ClassName
-PspoonMethodName=methodName
In order to know what yourSpoonTaskName is run ./gradlew tasks.
Also, in your build.gradle file add the following spoon configuration:
spoon {
// for debug output
debug = true
// To grant permissions to Android M >= devices
grantAllPermissions = true
// for sharding
/*
this will execute tests in parallel on multiple devices.
*/
shard = true
// Add this to run a specific test class & method
if (project.hasProperty('spoonClassName')) {
className = project.spoonClassName
}
if (project.hasProperty('spoonMethodName')) {
methodName = project.spoonMethodName
}
}
If you are not interested in Spoon and just want a simple solution, then use the following command to run an individual test:
am instrument -w -r -e class com.packageName.blah.TestName#methodName com.packageName.blah.YourIntrumentationRunnerName
You can easily determine these values if you right click the test name in AndroidStudio and run it. In the console, you will see the entire command being printed when the test is bring run.
I've got a couple of AndroidTestCase subclasses in separate packages of my project:
However, whenever I run Android Tests configuration from Android Studio, I see that my regular app is starting as well. I see that the onCreate method is fired up inside my Application class (which is really bad since I am loading some additional resources there).
Why is Android Studio/gradle running my app as well? Can I programatically detect if I am inside test or regular configuration? Can I stop my regular app from being booted before running tests?
In addition, when I am running tests in debug mode it doesn't stop on breakpoints placed inside the application's onCreate method. Why is this happening?
Edit:
Body of test class doesn't really matter, it can be something like:
public class SimpleTest extends AndroidTestCase {
public void testSample()
{
assertEquals(true, false);
}
}
Executing only this simple test fires up onCreate method inside application class.
Gradle console prints out:
Executing tasks: [:app:assembleDebug, :app:assembleDebugTest]
I guess that first task creates instance of my Application class - is it expected behavior?
Why is Android Studio/gradle running my app as well?
Can I stop my regular app from being booted before running tests?
AndroidTestCase is extension of JUnit TestCase which is aware of your android application. In case you don't need to test your android application and want to test plain java only you should use JUnit framework. Create regular JUnit tests, do not use android classes there and run JUnit test configuration like this:
You should treat AndroidTestCase as instrumentation tests which will build android app and run that tests on it. This is usefull with combination of Espresso and Robotium. Both are working on top of base android test classes and both will build and run your application before testing it. Real device or emulator is needed.
Use plain JUnit tests or Robolectric to test java on your desktop JVM.
Can I programatically detect if I am inside test or regular configuration?
You can use power of gradle to provide such info with autogenerated BuildConfig file.
At your build.gradle
android {
defaultConfig {
testPackageName "com.foo.test"
}
}
At your code:
BuildConfig.PACKAGE_NAME.equals("com.foo.test")
The AndroidTestCase is an unit test that unfortunately runs on the device (either virtual or real).
I think what you want to have is a UnitTestFramework that runs in the JVM (local on your machine). The TestFramework Robolectric can do this.
I have started a gitHub project to show how to setup the gradle test file and the project structure if you want to have UnitTests and InstrumentationTests side by side. If you want to look its AndroidGradleTests
When I create parameterized test cases in JUnit 3.x, I usually create a TestSuite with something like
public static Test suite() {
TestSuite s = new TestSuite();
for (int i = MIN; i < MAX; ++i) {
s.addTest(new MyTest(i));
}
}
This suite() method is called correctly when running JUnit from a desktop command-line. When I tried this with my Android test project, the tests don't run. How do I get my tests to run on the emulator? Or is there a different way to create parameterized tests for Android?
More thoughts:
Typically I run my tests with the command line:
adb shell am instrument -w [-e class <fully qualified test class name>[#<test method name>()]] <Android package name>/android.test.InstrumentationTestRunner
This allows me to select which tests to run from my test suite. Ideally, I want to run the the parameterized tests in this way as well. The link in the comment from #Appu describes building a separate app that runs JUnit tests. As part of that, this app has a custom TestRunner. I can very likely borrow these ideas to create a TestRunner which I can use in place of android.test.InstrumentationTestRunner. This seems like a lot of work for a not uncommon task. I prefer not to reinvent the wheel if there is already a similar solution in the Android API. Does anyone know of such a thing? Also, other alternative solutions will be helpful.
Nevermind, it looks like #dtmilano already posted this as an answer...
You can implement a test runner to be able to pass parameters to Android tests.
See the example at how to pass an argument to a android junit test (Parameterized tests).
Or is there a different way to create parameterized tests for Android?
We (Square) wrote a library called Burst for this purpose. If you add enum parameters in your test constructor, Burst's test runner will generate a test for each combination of enum values. For example:
public class ParameterizedTest extends TestCase {
enum Drink { COKE, PEPSI, RC_COLA }
private final Drink drink;
// Nullary constructor required by Android test framework
public ConstructorTest() {
this(null);
}
public ConstructorTest(Drink drink) {
this.drink = drink;
}
public void testSomething() {
assertNotNull(drink);
}
}
Quite a while after originally writing this question, I discovered that I can directly run a test class which contains a static suite() method:
adb shell am instrument -w -e class <fully qualified test class name> <Android package name>/android.test.InstrumentationTestRunner
However, the test suite doesn't run when I try to run all the tests in a given package.
Of course, this has been a while. Now I am using Android Studio instead of the command-line. I can still run the test class individually, but it still doesn't run when I select a package or try to run all of my tests.
A potential alternative is to write a master test class with a suite() method which adds all the tests to the returned TestCase. Unfortunately, this requires some manually editing every time I add a new test class to my suite.