On http://tools.android.com/tech-docs/new-build-system/user-guide, the following tasks are defined:
assemble The task to assemble the output(s) of the project
check The
task to run all the checks.
connectedCheck Runs checks that requires a
connected device or emulator. they will run on all connected devices
in parallel.
deviceCheck Runs checks using APIs to connect to remote
devices. This is used on CI servers.
build This task does both
assemble and check
clean This task cleans the output of the project
I am now setting up a Jenkins CI to run my (Espresso) tests and the deviceCheck task description seems related to that. But I couldn't find any further documentation or examples on how to use this task on CI server to execute the tests on a remote device. Does anyone know how to use it?
I would recommend running the tests with: Spoon. It's easy to setup, has lots of nice out of the box features. e.g. good test reports, screenshots ++.
I am using it on my Jenkins CI, and it works really well! I have connected multiple devices to the CI-server, and the tests runs on all of them.
Related
When running android espresso tests via gradle managed devices, i.e. by running:
./gradlew pixel4api30DebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.aaa.bbb.ccc.Suites.TestSuite -Pandroid.experimental.androidTest.numManagedDeviceShards=3
Is it possible to have it display the emulators while running the tests so that you are able to see what is happening?
You can use the --enable-display switch:
./gradlew pixel4api30DebugAndroidTest --enable-display
But keep in mind that this will not work with ATD images.
Since some time, when I run an instrumentation test on a device (emulator) and try to start another test on a second device in parallel, I get the following exception:
FAILURE: Build failed with an exception.
What went wrong: Execution failed for task ':app:connectedDebugAndroidTest'.
java.nio.file.FileSystemException: D:\Android\MyApp\app\build\outputs\androidTest-results\connected\name_of_the_first_device\logcat-MyApp.TestClassName-TestName.txt:
The process cannot access the file because it is being used by another process
The problem seems to be, that the build process tries to access a file from the first test device, which is already blocked. But I'm using a different device and try to run a different test.
The reason why I do that is because some of the tests are very large and therefore I try to spread them to several devices, so I get my results faster. The machine I run the tests on, has a hell of performance, so it can't be a problem of the resources ;-).
When I did the described actions some months ago, it worked properly, so I guess it's due to some updates (probably in Gradle or whatever). But I installed several updates since then, so it can't just be a problem of the current version.
Update - Two more inputs:
I recently switcht to using hilt and therefore created my own TestRunner, because an Application annotated with #HiltApplication cannot be used for tests, so I had to create a custom one for testing which needs to be run in a custom TestRunner as described here.
When I run the same tests on multiple devices using "Select Multiple Devices" dropdown in Android Studio it works properly.
The error message indicates a conflict over a build-and-run output file on your development computer, not on the Android test device. That is, it's trying to do another build-and-run in the same build output directory that the running test is using for purposes like the test's logcat output.
My first thought is to try duplicating androidTest to another build target. Same contents, different target name, different output directory. That workaround seems likely to get past this limitation (but with new limitations) and to help debug the problem. But the name_of_the_first_device subdirectory suggests that it can keep a logcat output file per test device, so the real question is how to get the test runner to use a separate subdirectory per test device.
Second thought is to point you to the the docs, Run Tests with Multiple Devices and Shard Tests.
With a custom TestRunner, see Write a Sharded IRemoteTest Test Runner (and other pages in the "Tradefed" section) on making it a "shardable" test runner which enables the test infrastructure to distribute the full test execution over several devices.
I have an unused mac mini, so I installed Jenkins on it, and I'm trying to configure it so that it is always running monkey tests on our latest Android build. I can't find a way to do that though. I've found the Android Emulator Plugin which allows for the booting of an emulator and then running a Monkey Test on the build, but it only seems to be configurable for a certain number of events, not time based.
The ideal Jenkins configuration would poll a git server. When it sees a new commit on a specific branch, it would build the APK, run the emulator, and then start the monkey test. I'm sure crashes would happen, and every time a crash happens the test should restart. It should continue running like this indefinitely. Then when a new commit is detected, it would shut down the emulator, and start the build all over again.
Are there any creative ways that a Jenkins build could be configured to continually run monkey tests?
When I run the following command I get a description of connectedCheck but I'm still not sure what it's used for. Could anyone give me a real world example?
./gradlew tasks prints
...
Verification tasks
------------------
check - Runs all checks.
connectedCheck - Runs all device checks on currently connected devices.
connectedInstrumentTest - Installs and runs the tests for Build 'Debug' on connected devices.
deviceCheck - Runs all device checks using Device Providers and Test Servers.
...
Command ./gradlew connectedCheck executes instrumentation tests located in src/androidTests/ directory on connected Android device or emulator. Such tests can have dependencies to Android API. These tests can be simple assertions or UI tests with Espresso framework or something similar. Yesterday I wrote post about Android automated tests including more detailed description of them. You can check it out here.
I am using Espresso. I have multiple tests written, all in different classes but the same project. When I run I remove all but one class and try running, it runs fine. However, when I have them all in my project and try running the second one always fails.
All of my test are pretty basic right now. They just have setUp() and one testMethod().
Can someone tell me what I am doing wrong?
NOTE: I run via terminal with command gradle appName:cC
I found my answer. I need to add the Spoon Gradle Plugin
That's one option, but you don't have to, you can still use gradle command connectedCheck
Find out how to run the tests (including running tests for a specific module and on multiple devices) here at this Google example:
https://github.com/googlesamples/android-testing-templates/tree/master/AndroidTestingBlueprint