Background
I want to run my Android Instrumented tests on Jenkins on different emulators. Say I have 100 tests and 4 emulators, I want to run 25 tests on each.
I perform ./gradlew connectedDebugAndroidTest in Jenkins Pipeline's parallel for 4 emulators
stage('Instrumented Tests') {
parallel(
emu1: {
runInstrumentedTestOnEmu(...)
},
emu2: {
runInstrumentedTestOnEmu(...)
}
...
)
}
connectedDebugAndroidTest will spawn other commands in order to setup the environment for running instrumented tests.
...
:app:transformNativeLibsWithMergeJniLibsForDebugAndroidTest
:app:processDebugAndroidTestJavaRes NO-SOURCE
:app:transformResourcesWithMergeJavaResForDebugAndroidTest
:app:validateSigningDebugAndroidTest
:app:packageDebugAndroidTest
:app:assembleDebugAndroidTest
:app:connectedDebugAndroidTest
And when environment is ready then it performes :app:connectedDebugAndroidTest which will start running tests on emulator.
I do not want to run these procedure for all my parallel calls (in this case it would be 4 of them), because obviously I'm doing the exact same job multiple times. Theoretically, the best option would be to perform setup before parallel and when everything is ready for running tests, then go into parallel step and start tests on each emulator.
Question
Is it possible to perform all the pre-setup steps of connectedDebugAndroidTest without performing itself?
Additionally, if I run connectedDebugAndroidTest parallel on 4 emulators the build crashes, because gradle tries to read a file from intermediate directory, when other parallel build has already removed that file, which results in crash.
You can view this test project in github with setup mentioned above.
Is it possible to perform all the pre-setup steps of connectedDebugAndroidTest without performing itself?
Yes, you can run assembleDebugAndroidTest, which as your build log shows, is the last prerequisite to running the device tests. Running that will build both the app and test APKs.
Though AFAIK, there isn't a way of sharding your tests across multiple emulators when using Gradle — you would have to install both of the APKs onto each emulator and use adb shell am instrument with the numShards and shardIndex options.
Related
I am performing instrumented test execution using gradlew connectedAndroidTest on physical devices. But while doing this, I observe that irrespective of number of devices connected to my machine, at max 3-4 devices only have the tests running.
For Example - If I have 10 android devices connected to my machine, and I start execution of instrumented tests using gradlew connectedAndroidTest, it first starts the instrumented test on 3-4 devices. Then once any of the device is finished execution, it randomly picks up another device and starts execution.
Thus would like to understand that is there a way to control that on how many devices the instrumented test would be running?
Is there a way to call the task connectedAndroidTest and skip the uninstall task at the end of the process ?
At the end of the test execution, the app is uninstalled from the device, but I would like to keep the app on the device.
from http://tools.android.com/tech-docs/new-build-system/user-guide#TOC-Running-tests :
As mentioned previously, checks requiring a connected device are launched with the anchor task called connectedCheck. This depends on the task connectedDebugAndroidTest and therefore will run it. This task does the following:
Ensure the app and the test app are built (depending on assembleDebug and assembleDebugAndroidTest).
Install both apps.
Run the tests.
Uninstall both apps.
Looking at the sorce of gradle plugin there is no way to prevent uninstalling app at the end of test task. You can check that in SimpleTestCallable class of android gradle plugin.
From what i see there are two options to acchive what you want.
First one is to reinstall app after your connected check is done. Command to do that would look something like this. ./gradlew connectedCheck installDebug installDebugAndroidTest This will execute test on device and delete apps from it. But after that it will reinstall app and test app. So app will still be removed and then installed which means a bit of owerhead but at least apps will not be recompiled twice since you are executing in same gradle execution.
Second option is to not use gradle for executing tests but use adb instead.
To do this you first need to install app and test app through gradle.
./gradlew installDebug installDebugAndroidTest
After that you can execute tests through adb. by caling adb shell am instrument -w com.example.test/android.support.test.runner.AndroidJUnitRunner.
When this is done you can run your cli tests since both app and test app are still installed.
With second approach you would lose all the benefits of executing test wit gradle. Such as code coverage and executing in multiple proceses, etc.
I have a strange situation with Jenkins when trying to run the tests.
The configuration of the job clearly specifies to run the tests on this specific avd: "hudson_en-US_160_HVGA_android-18_armeabi-v7a", but for some odd reasons, during the build process, 2 devices are found. One of the devices is the previous one, but the other is called "unknown-sdk".
The tests are successfully executed on the specific avd, and fail on "unknown-sdk":
The relevant portion of the log:
[SR.runTests] Executing instrumentation suite on 2
device(s).
[SR.runTests] [emulator-8215] Starting
execution.
[SR.runTests] [localhost:8216] Starting
execution.
Running am instrument -w -r -e class com.smoke.tests.LoginTest
com.muume.dev.test/android.support.test.runner.AndroidJUnitRunner on
unknown-sdk-localhost:8216
2015-06-17 11:54:05 [SDR.printStream] [localhost:8216] STDOUT 11:54:05
I/InstrumentationResultParser: test run failed: 'Unable to find
instrumentation info for:
ComponentInfo{com.muume.dev.test/android.support.test.runner.AndroidJUnitRunner}
As you can see, it detected 2 devices: emulator-8215 which is the known avd, and localhost-8216 which is the unknown and causes the problems.
My question is how to restrict the tests to run only one single emulator, and from where did the "unknown-sdk-localhost" comes from.
"unknown-sdk" is what it tells adb while the emulator is still busy loading itself.
Does anyone have any good resources for setting up Bamboo to do CI with Android projects? I have mine setup to pull source and compile it with ant. But I would love to know how to setup JUnit tests, where the tests are in a separate project.
Thanks
I have figured out how to do it using Bamboo CI and new Android Studio projects with gradle. Bamboo does not have nice drop in tasks yet but you can leverage the script runner to do it. We setup our basic build tasks as follows:
Source Code Checkout.
Script task:
Script Location: Inline
Script Body: gradlew.bat assembleDebug test (our Bamboo server is Windows so we use the bat file, linux use the ./gradlew assembleDebug test command)
Then we add a final task of JUnit parser, and we use the result directory line of: **/test-results/debug/*.xml
As for testing we use Robolectric Gradle tests, which generate JUnit test results.
I hope this helps anyone else who is looking into how to setup Bamboo with Android, hopefully they will add support one day like they do for .NET where its just a single task that builds and tests. The script command feels kind of a hack.
If someone is looking for Ant style tests, I can share that too but hopefully by now everyone has moved to Android Studio from eclipse. I will say the steps required for Ant and Instrumentation take a lot more time to setup and I had to use an emulator running on the server to do the tests.
In addition to using Bamboo to build the APK for my Android project, I also wanted to use Bamboo to run the JUnit based tests against an Android emulator. After quite a bit of "trial and error" primarily around finding a reliable way to start and stop the Android emulator, here is what I came up with for my Bamboo build plan. See Bamboo waits for script task to terminate, although it is run in the background for additional background information regarding why I take the approach described below.
My Bamboo plan has one stage with two jobs. The jobs run using two agents that execute on the same system. Both jobs start and run in parallel. One job starts the Android emulator using the Android SDK emulator command. The other job waits for the emulator to start, builds the mobile app, runs the tests against the emulator and then stops the running emulator using a final task that is always executed even if a previous task in the build job fails.
The emulator job does get "stuck" after starting the emulator because it is waiting for the emulator process to finish. When the build job runs, the final task in the build job stops the emulator which causes the emulator job to finish because the emulator process is no longer running.
Here are the key task details for the build job:
First task is a script task that waits for the emulator to start. The adb -s command below will fail causing this task to fail if the emulator failed to start.
echo "Waiting 60 seconds for the Android emulator to start"
sleep 60
echo "See if Emulator is up and running"
${bamboo.ANDROID_HOME}/platform-tools/adb -s emulator-5554 shell getprop dev.bootcomplete
The second and third tasks check out the source and build the app using Gradle. The build runs the JUnit tests against the running emulator.
The fourth task which is configured as a final task is a script task that stops the emulator.
echo "Stopping the Android emulator"
${bamboo.ANDROID_HOME}/platform-tools/adb -s emulator-5554 emu kill
I have created Android unit tests project. They need ADB running for the tests to run successfully.
I have also used the command line tool of Android Debug Bridge for unit testing the Android Unit tests projects.
In above both the cases ADB needs to be running.
Is it possible to run the tests without running ADB. This will be useful for build systems.
If you want to get the errors then you have to use ADB to get the errors.