For Android Espresso Testing, i'm using the following commands to run our tests:
./gradlew pixel4api30DebugAndroidTest
-Pandroid.testInstrumentationRunnerArguments.class=com.aaa.bbb.ccc.Suites.TestSuite
-Pandroid.experimental.androidTest.numManagedDeviceShards=2
This splits all the tests in the TestSuites across 2 pixel 4 emulators.
If I have the following in the TestSuite:
TestClass1
Test1
Test2
Test3
TestClass2
Test1
Test2
Test3
It may run TestClass1.Test1 on 1 emulator and TestClass1.Test2 on emulator 2. (it could pick any emulator to run any of the tests on).
Is there a way to force it to split the shards by TestClasses? So it will run TestClass1 and all tests on one emulator, TestClass2 and all tests on an emulator.
I believe if you have used Spoon in the past, you may have been able to accomplish this by using singleInstrumentationCall property as noted here: https://github.com/jaredsburrows/gradle-spoon-plugin
Related
Background
I want to run my Android Instrumented tests on Jenkins on different emulators. Say I have 100 tests and 4 emulators, I want to run 25 tests on each.
I perform ./gradlew connectedDebugAndroidTest in Jenkins Pipeline's parallel for 4 emulators
stage('Instrumented Tests') {
parallel(
emu1: {
runInstrumentedTestOnEmu(...)
},
emu2: {
runInstrumentedTestOnEmu(...)
}
...
)
}
connectedDebugAndroidTest will spawn other commands in order to setup the environment for running instrumented tests.
...
:app:transformNativeLibsWithMergeJniLibsForDebugAndroidTest
:app:processDebugAndroidTestJavaRes NO-SOURCE
:app:transformResourcesWithMergeJavaResForDebugAndroidTest
:app:validateSigningDebugAndroidTest
:app:packageDebugAndroidTest
:app:assembleDebugAndroidTest
:app:connectedDebugAndroidTest
And when environment is ready then it performes :app:connectedDebugAndroidTest which will start running tests on emulator.
I do not want to run these procedure for all my parallel calls (in this case it would be 4 of them), because obviously I'm doing the exact same job multiple times. Theoretically, the best option would be to perform setup before parallel and when everything is ready for running tests, then go into parallel step and start tests on each emulator.
Question
Is it possible to perform all the pre-setup steps of connectedDebugAndroidTest without performing itself?
Additionally, if I run connectedDebugAndroidTest parallel on 4 emulators the build crashes, because gradle tries to read a file from intermediate directory, when other parallel build has already removed that file, which results in crash.
You can view this test project in github with setup mentioned above.
Is it possible to perform all the pre-setup steps of connectedDebugAndroidTest without performing itself?
Yes, you can run assembleDebugAndroidTest, which as your build log shows, is the last prerequisite to running the device tests. Running that will build both the app and test APKs.
Though AFAIK, there isn't a way of sharding your tests across multiple emulators when using Gradle — you would have to install both of the APKs onto each emulator and use adb shell am instrument with the numShards and shardIndex options.
Not a unit test, but an instrumentation test.
My app has multiple flavors, so I run:
./gradlew connectedClientDebugAndroidTest to run my instrumentation tests (flavor name is client).
But I want to run one particular instrumentation test case class called MyActivityTestCase.java. Is this possible? If it is what is the command to run this
With Gradle, you can run a single test by using the test.single system property. You set it from the command-line with the -D option. For example
$ gradle -Dtest.single=MyActivityTestCase connectedClientDebugAndroidTest
For more details, see http://mrhaki.blogspot.com/2013/05/gradle-goodness-running-single-test.html.
The general syntax using adb shell is :
adb shell am instrument -w <test_package_name>/<runner_class>
where is the Android package name of your test application, and is the name of the Android test runner class you are using.
More details at: http://developer.android.com/tools/testing/testing_otheride.html
So the way I fixed this was by putting the single TestCase that I want into a specific folder by flavor.
So in this example, my flavor is named client, so I put the single test that I want to run for this flavor into app/src/androidTestClient/MyActivityTestCase.java
Then run ./gradlew connectedClientDebugAndroidTest
Not a solution if you have multiple test cases for a specific build flavor, but for me that wasn't the case so it works.
All my other tests are in my flavor called other, in the folder:
app/src/androidTestOther/. So to run the other instrumentation tests I just run the command ./gradlew connectedOtherDebugAndroidTest
I have created Android unit tests project. They need ADB running for the tests to run successfully.
I have also used the command line tool of Android Debug Bridge for unit testing the Android Unit tests projects.
In above both the cases ADB needs to be running.
Is it possible to run the tests without running ADB. This will be useful for build systems.
If you want to get the errors then you have to use ADB to get the errors.
As this was not answered (Maybe did I not find it) previously, I investigated on the following question :
How to perform automated functional tests on Android devices with robotium, and report them to continuous integration server like TeamCity?
As I did not find any answer on that specific question, I investigated. Here is the output of my investigation and a quick How-To in order to help people perform automated functional tests on Android applications using robotium, and then report the results to a continuous integration server like TeamCity. Please note that this may not be the best solution, but I felt that people might be in the same situation as me. So here it is !
The following libraries have been used :
Robotium (https://code.google.com/p/robotium/) : This is an Android test automation framework. It helps you to perform automated tests like click on buttons, fill text automatically, and a lot of other things.
Android Junit Report
(http://zutubi.com/source/projects/android-junit-report/) : This library is very useful to publish test result to an exploitable xml format. If you want to run your tests through Eclipse, you will see the results of your tests on the go, but in order to export them, this library is very useful
Assuming that you have an Android project to test, create an Android Test Project (Eclipse does have a nice workflow to create it for you) and set it up to work with Robotium. A pretty clear detailed instruction on how to do it can be found here : https://code.google.com/p/robotium/wiki/Getting_Started
Then, you need to add Android Junit Report to your project in order to be able to fetch the results of your tests. In order to do so, add the Android Junit Report *.jar library in your lib folder and add it to your build path (in Eclipse : Project -> Properties -> Java Build Path -> Libraries -> Add External Jar).
You also have to change the test runner of your project. In the AndroidManifest.xml of your test project add the following :
<instrumentation
android:name="com.zutubi.android.junitreport.JUnitReportTestRunner"
android:targetPackage="<insert your package ex:com.alth.myproject" />
Once this is done, you should be able to run your tests properly. The results of the tests should be available in your device (in the following folder /data/data//files/junit-report.xml)
The next step is to configure your TeamCity build steps to perform all the different needed action to run your tests. Please note that my solution might not be the optimal one !
Build step 1 : Clean - Command Line runner - This build step may be optional depending of how you decide to create your build.xml files and such build decisions.
rm -rf <report folder>
rm -rf <Project build.xml>
rm -rf <Test project build.xml>
android update project -p <Path to your project>
android update test-projecct -m <Path to your project, relative to the test project> -p <Path to your test project>
Build step 2 : Launch AVD - Command Line runner - This build step launches the android virtual device. This step may be optional if you decide to run the tests on an actual device.
emulator -avd <nameOfYourAvd> -no-boot-anim &
sleep 45
The & avoids build to be interrupted by the virtual device launch (It is basic shell command). The sleep command is used to try to let the AVD be ready for the next build step
Build step 3 : Test app release - Ant runner : Build the test project, install it on the virtual device
Path to build xml file : <Path to your test project>/build.xml
Additional Ant command line parameters : -f <Path to your test project>/build.xml clean debug install -Dsdk.dir=<Path to your android sdk>
Build step 4 : AVD Unlock - Command line runner : Unlock the AVD screen for testing purpose
bash avdUnlock.sh
Body of avdUnlock.sh here : (http://pastie.org/7919761). This script is sending informations on regular AVD port in order to unlock the screen. This may be improved by sending the command only to a specific port and changing build step 2 to add a specific port to the emulator launch. This is however not really part of this how-to
Build step 5 : Launch tests - Command line runner : Launch the tests
adb shell pm list instrumentation
adb shell am instrument -w <insert your test package ex:com.alth.myproject.test>/com.zutubi.android.junitreport.JUnitReportTestRunner
The first adb command could be removed. This is only for debug purpose in order to see which instrumentation has been installed on the device.
Build step 6 : Fetch tests - Command line runner : Retrieve tests report from the device
adb pull /data/data/<insert your project package ex:com.alth.myproject>/files/junit-report.xml <report folder>/junit-report.xml
Build step 7 : Final emulator kill - Command line runner : Kill the running android virtual device
adb emu kill
Additional Build Features : XML report processing - Report type : Ant JUnit
Monitoring rules : <report folder>/*.xml
This How-to is clearly not optimal but answer the original question. Doing so, it is possible to fetch the android functional tests report and feed it to teamcity in order to monitore test results.
I hope this will help someone, and I would try to answer to your questions if you have some.
Al_th
I developed for my application a small suite of Android tests written in Scala that uses the Robotium library. The suite is for all intents and purposes a standard Android JUnit test project and runs successfully if launched from Eclipse.
I've already successfully built and run my main Android application with sbt android-plugin. The main application is located in [ProjectDir]/src/main. I was also able to successfully build my Android test application that is located in the [ProjectDir]/tests/src/main directory. I checked the emulator, and the test application appears to have been correctly installed with android-plugin's tests/android:install-emulator command. However, when I try to run the test project via sbt tests/android:test-emulator, I get:
...
Test results for InstrumentationTestRunner=
Time: 0.001
OK (0 tests)
How I can get sbt android-plugin to recognize that the project contains JUnit tests and run them?
The naming convention used here is the same as the normal JUnit and as such you need to name the tests xxxTest.class. They also need to extend TestCase (AndroidTestCase, InstrumentationTestCase etc...).
To reiterate, eclipse will run a command which will look like:
adb shell am instrument -w -e class com.android.foo.FooTest,com.android.foo.TooTest com.android.foo/android.test.InstrumentationTestRunner
It will append the classes name to the command so naming convention might not apply.
If you run from sbt, it will run
adb shell am instrument -w com.android.foo/android.test.InstrumentationTestRunner
which will find all the classes under the package name of the application com.android.foo which finishes with someClassNameTest.