When running android espresso tests using gradle managed devices, how can you clear the test results that are cached if you want to force the test(s) to be re-ran?
Per https://developer.android.com/studio/test/gradle-managed-devices:
Caches test results and reruns only tests that are likely to provide different results
Related
Since some time, when I run an instrumentation test on a device (emulator) and try to start another test on a second device in parallel, I get the following exception:
FAILURE: Build failed with an exception.
What went wrong: Execution failed for task ':app:connectedDebugAndroidTest'.
java.nio.file.FileSystemException: D:\Android\MyApp\app\build\outputs\androidTest-results\connected\name_of_the_first_device\logcat-MyApp.TestClassName-TestName.txt:
The process cannot access the file because it is being used by another process
The problem seems to be, that the build process tries to access a file from the first test device, which is already blocked. But I'm using a different device and try to run a different test.
The reason why I do that is because some of the tests are very large and therefore I try to spread them to several devices, so I get my results faster. The machine I run the tests on, has a hell of performance, so it can't be a problem of the resources ;-).
When I did the described actions some months ago, it worked properly, so I guess it's due to some updates (probably in Gradle or whatever). But I installed several updates since then, so it can't just be a problem of the current version.
Update - Two more inputs:
I recently switcht to using hilt and therefore created my own TestRunner, because an Application annotated with #HiltApplication cannot be used for tests, so I had to create a custom one for testing which needs to be run in a custom TestRunner as described here.
When I run the same tests on multiple devices using "Select Multiple Devices" dropdown in Android Studio it works properly.
The error message indicates a conflict over a build-and-run output file on your development computer, not on the Android test device. That is, it's trying to do another build-and-run in the same build output directory that the running test is using for purposes like the test's logcat output.
My first thought is to try duplicating androidTest to another build target. Same contents, different target name, different output directory. That workaround seems likely to get past this limitation (but with new limitations) and to help debug the problem. But the name_of_the_first_device subdirectory suggests that it can keep a logcat output file per test device, so the real question is how to get the test runner to use a separate subdirectory per test device.
Second thought is to point you to the the docs, Run Tests with Multiple Devices and Shard Tests.
With a custom TestRunner, see Write a Sharded IRemoteTest Test Runner (and other pages in the "Tradefed" section) on making it a "shardable" test runner which enables the test infrastructure to distribute the full test execution over several devices.
I have functional autotests for some SDK (as aar-libs) and TeamCity CI server with several Android devices and emulators connected on.
If I use gradle to run my tests (./gradlew connectedDebugAndroidTest), all tests will run on each device, but each next results will overwrite previous ones and TeamCity will show only the last results.
I have to distinct these results and should know which results from what device/emulator.
Any ideas?
Decided to create additional build configuration on Team City, which will parse all connected devices from adb devices and trigger all other configurations passing device name as parameter
I am doing a simple series of tests with Robotium Recorder on Android Studio. I have created my test cases and had a few questions:
One of my tests is to click a URL link and to make sure that the default application dialogue/internet browser is displayed. I don't see an option to do this from the recorder (I wouldn't expect it). I have been looking through the documentation for the solo object and do not see anything that would help me out. There has to be a way to check this.
Once a test is ran from Android Studio, where would the file containing the test results reside?
According to second question: I'm not sure that Robotium already creates any tests reports. You may use for it additional addons like Spoon or Emma.
Read: Creating Test Reports for Android with Spoon and Emma
I know also that Robotium can do a screenshots during the run of tests, but already I don't use this framework so please find on your own how to do it and where screenshots are placec if you feel a bit interesting.
Already, for creating instrumentation tests report I use Gradle. It generates quite good HTML reports. There are at least two ways to make it
in Android Studio, on the left panel, choose Gradle then app, verification, finally connectedAndroidTest
if you're familiar with Unix console go to your project location and just run the command:
./gradlew connectedAndroidTest
NOTE: On the first time it might be needed to run before chmod u+x gradlew, if console would say that you don't have enough permissions to do it.
SUGGESTION: if you make one of your test failed, in generated by Gradle test report you would find a location of HTML generated tests report.
Gradle test report would look like this if all would run successful or this, if at least one would fail.
On http://tools.android.com/tech-docs/new-build-system/user-guide, the following tasks are defined:
assemble The task to assemble the output(s) of the project
check The
task to run all the checks.
connectedCheck Runs checks that requires a
connected device or emulator. they will run on all connected devices
in parallel.
deviceCheck Runs checks using APIs to connect to remote
devices. This is used on CI servers.
build This task does both
assemble and check
clean This task cleans the output of the project
I am now setting up a Jenkins CI to run my (Espresso) tests and the deviceCheck task description seems related to that. But I couldn't find any further documentation or examples on how to use this task on CI server to execute the tests on a remote device. Does anyone know how to use it?
I would recommend running the tests with: Spoon. It's easy to setup, has lots of nice out of the box features. e.g. good test reports, screenshots ++.
I am using it on my Jenkins CI, and it works really well! I have connected multiple devices to the CI-server, and the tests runs on all of them.
Is it possible to resume on failure, while running gradle connectedCheck?
I want to use it in hudson, but it stopped execution after the first failure. How can I configure it to do the next test on failure? This way, I see all failing tests at the end and not only the first one.
This may not answer the question in the way you wanted but here goes anyway. What I did to solve this problem was to use the Spoon testing framework as a wrapper around the Android test framework. It gives you lots of advantages:
You can use the regular tests you've already created
Spoon produces permanent HTML output that you can save
You can add screenshots to your test output
The test output contains the log messages for failures
And, most relevant to your question, it continues on even when a test fails.
You can find spoon at: http://square.github.io/spoon/