Currently I run Espresso tests on Firebase / Google Test Lab, but it takes time to run them all, as they run in sequentially.
Is there a way to run them in parallel?
You would need to shard your tests so that you can kick off multiple batches of test that can run in parallel on different devices. Firebase doesn't provide a way to do this directly, but there is an open source project called Flank that can help.
Related
A basic Flutter app failed to launch on Firebase Test Lab for Android. I can see in the video that the app never came up so it never even tried running my tests, it just timed out. In the logs, I do not see any errors from my application; there are errors and warnings but I can't understand any of them nor do I know if they're ok to ignore. I used Firebase Test Lab on other projects and I'm accustomed to seeing lots of errors which I assume come from Firebase Test Lab's infrastructure. I typically look through the errors to try to find something that might pertain to my app. In this case I don't see anything that seems to be related to my app.
The app is from the code lab https://codelabs.developers.google.com/codelabs/flutter-codelab-first#0 .
The instructions for Firebase setup are at https://github.com/flutter/flutter/tree/main/packages/integration_test#firebase-test-lab .
The test passes when run locally by running the following command on my Mac:
./gradlew app:connectedAndroidTest -Ptarget=`pwd`/../integration_test/app_test.dart
The command I run from my CI (gitlab) is as follows:
gcloud firebase test android run --type instrumentation --app build/app/outputs/apk/debug/app-debug.apk --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk
I got the same timeout by initiating the test through the Firebase Test Lab UI.
I also posted this question at https://firebase-community.slack.com/archives/C1MTSQ5QT/p1674438388007639 .
Any ideas for what I could investigate?
I was able to get Firebase Test Lab to test a Flutter App by using this Espresso "bridge" https://pub.dev/packages/espresso. So instead of writing Flutter tests with the integrationDriver from package:integration_test/integration_test_driver.dart as described in https://github.com/flutter/flutter/tree/main/packages/integration_test#driver-entrypoint, I can use the enableFlutterDriverExtension() from package:flutter_driver/driver_extension.dart .
Not ideal.
I need to run integration tests for an Android library that leverages native
code via JNI. This has a few implications:
This is a library, and as such has no GUI whatsoever. This means that test
frameworks like Espresso, Appium and the like are of no use, since there's no
GUI to test against.
This gives me some headaches with popular cloud services like App Center and
Firebase Test Lab, as they require both an UI instrumented test apk, written
with one of the supported UI testing frameworks, and an app apk to run these
tests against.
As said, the library contains C code loaded via JNI. Unfortunately, this
excludes the possibility of using Robolectric, since it doesn't support JNI.
Now there are a few questions:
Are there any Android simulation testing frameworks that support JNI?
Is there a cloud service out there which allows me to simply run a pre-built
Android instrumented unit tests apk, without any UI instrumented test
framework and without an actual app to be tested?
Have I missed a much simpler solution to my problem?
The problem with cloud services is that currently I only have an instrumented
unit test apk, which as such has no actual GUI testing. Furthermore, that's the
sole apk I have, since there's no actual Android app to be tested.
I am not aware of any Android mocking test frameworks, like Robolectric, that
support JNI, which is why I am looking into instrumented tests, on real or
emulated Android devices, in the first place.
Google released Android Testing Support Library 1.0 recently. After reading the overview, I'm a little confused with Android Test Orchestrator.
It said
Typically, AndroidJUnitRunner runs all tests in the same instrumentation process, which can cause a number of problems.
Can you explain which kinds of problems will be caused by using the same instrumentation process?
if one test crashes, it prevents the remainder of the test suite from running
Through my experience, one test crash won't prevent from other test cases from running. Please point out what I misunderstood here?
And from Android Testing Orchestrator developer guide,
For completeness, Android Test Orchestrator runs pm clear after each test.
So Android Test Orchestrator will run pm clear [test_package_name] after each test, right?
Through my test, pm clear [app_package_name] won't be executed after each test. That means the data of application under test will not be cleared. So test cases might still have dependencies on each other. For example:
Test case A stores a SharedPreference key-value
Test case B which runs after test case A can read out the value stored by test case A
Overall, after some trial, I did not find any advantage of Android Test Orchestrator. Can somebody help to address my confusion? Thanks.
After researching the issue a bit I can provide the following answers:
Typically, AndroidJUnitRunner runs all tests in the same instrumentation process, which can cause a number of problems.
As mentioned, AndroidJUnitRunner runs on the same instrumentation process so basically your tests are running state-full, this might wreak havoc if your tests have some sort of dependency on the process state. In Android test orchestrator each test is run in its own process so dependencies aren't an issue.
if one test crashes, it prevents the remainder of the test suite from running
The crash in question here is a crash of the process, not the activity/application. You can test this by inserting in one your tests System.exit(0); Typically, this will stop the whole test run while in Android test orchestrator the tests continue as expected.
For completeness, Android Test Orchestrator runs pm clear after each test.
This is an oversight of google and has been retracted from the official documentation as can be observed here.
Basically, The advantages of using Android test orchestrator is all about the separate process for each test which improves stability and ensures full execution of tests.
Android Test Orchestrator is a tool which allows you to run each of your app’s tests within its own invocation of Instrumentation.
This means that each test (method annotated with #Test ) will be run on a separate instance of AndroidJUnitRunner.
What issues does it solve?
When dealing with UI tests we identified 2 major problems which occur from time to time when run on CI or locally:
Occasional crashes which stop the entire test suite.
Test overlap.
Orchestrator can prevent crash to interrupt the whole test, eg. native crash.
But it may slow the tests
I have a web app that is using Firebase FCM to communicate with my android device.
On receiving a certain data message through Firebase, I need android to run Instrumentation tests (such tests are run in a separate test application or "test APK" which is developed in the androidTest folder).
How can I make an Intent that will launch the test APK?
You can't run instrumentation tests in response to an external stimulus like this. Instrumentation tests are always driven through a JUnit test harness, which requires yet some other controlling process to kick them off using the am instrument command line program on the device. This is typically Android Studio, which will install both your app APK and your test APK, then kick off the tests. Firebase Test Lab will also take both APKs kick off your tests on physical and virtual devices.
I have multiple tests running as a part of Android Jenkins build including Unit test and Functional tests. I am able to publish the test results on Jenkins successfully but I want to see seperate result graph for Unit tests and Functional Test. Jenkins JUnit publisher only shows one graph for multiple XML files. Any help?