I have a web app that is using Firebase FCM to communicate with my android device.
On receiving a certain data message through Firebase, I need android to run Instrumentation tests (such tests are run in a separate test application or "test APK" which is developed in the androidTest folder).
How can I make an Intent that will launch the test APK?
You can't run instrumentation tests in response to an external stimulus like this. Instrumentation tests are always driven through a JUnit test harness, which requires yet some other controlling process to kick them off using the am instrument command line program on the device. This is typically Android Studio, which will install both your app APK and your test APK, then kick off the tests. Firebase Test Lab will also take both APKs kick off your tests on physical and virtual devices.
Related
A basic Flutter app failed to launch on Firebase Test Lab for Android. I can see in the video that the app never came up so it never even tried running my tests, it just timed out. In the logs, I do not see any errors from my application; there are errors and warnings but I can't understand any of them nor do I know if they're ok to ignore. I used Firebase Test Lab on other projects and I'm accustomed to seeing lots of errors which I assume come from Firebase Test Lab's infrastructure. I typically look through the errors to try to find something that might pertain to my app. In this case I don't see anything that seems to be related to my app.
The app is from the code lab https://codelabs.developers.google.com/codelabs/flutter-codelab-first#0 .
The instructions for Firebase setup are at https://github.com/flutter/flutter/tree/main/packages/integration_test#firebase-test-lab .
The test passes when run locally by running the following command on my Mac:
./gradlew app:connectedAndroidTest -Ptarget=`pwd`/../integration_test/app_test.dart
The command I run from my CI (gitlab) is as follows:
gcloud firebase test android run --type instrumentation --app build/app/outputs/apk/debug/app-debug.apk --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk
I got the same timeout by initiating the test through the Firebase Test Lab UI.
I also posted this question at https://firebase-community.slack.com/archives/C1MTSQ5QT/p1674438388007639 .
Any ideas for what I could investigate?
I was able to get Firebase Test Lab to test a Flutter App by using this Espresso "bridge" https://pub.dev/packages/espresso. So instead of writing Flutter tests with the integrationDriver from package:integration_test/integration_test_driver.dart as described in https://github.com/flutter/flutter/tree/main/packages/integration_test#driver-entrypoint, I can use the enableFlutterDriverExtension() from package:flutter_driver/driver_extension.dart .
Not ideal.
I working on developing Android Instrumentation Tests written in Espresso. As part of the effort, I am uploading and running in Firebase Test Lab using the Firebase Console. It would save me a lot of time if I didn't have to upload the App APK every time I want to re-run a test.
Does anybody know if Firebase Test Lab stores uploaded App and Test APKs so that a test can be re-run without requiring APKs to be uploaded again? Or an option in the Firebase Test Lab console to access an App APK that was previously uploaded?
I tried to use Firebase Storage but I can't access the files from Test Lab.
Maybe using Cloud SDK command line interface it best option since the test run will be automatically initiated once the APKs load?
Files (and APKs) that you pass to gcloud firebase test android run can be both from the local file system, as well as files stored in Google Cloud Storage. E.g. see the official documentation for the --app argument.
If you don't want to pass these files for every test invocation, upload them first to GCS with gsutil:
gsutil cp myapp.apk gs://my-bucket/myapp.apk
Then, use them for every test run:
gcloud firebase test android run --app gs://my-bucket/myapp.apk
So I want to build my tests and app, and then run the instrumentation tests whenever I want, even when I do not have access to the code (and hence, cannot use cAT task)
I have tried using assembleAndroidTest to generate test apk. But when I install this apk (as well as the app apk) and run am instrumentation with correct arguments (-e emma true, and with correct runner class verified using pm list instrumentation), I get no .ec file under /data/app/<package-name>. There is no error however.
I am using JaCoCo and when I run the tests using cAT, the coverage report is generated so I guess my JaCoCo configs are Ok.
I have made a library project consisting of an Android module and no application. This module relies on Android-specific functionality.
How do I run these tests with Firebase?
It seems like Firebase requires an app apk and a test apk, while I only have a test apk (*-androidTest.apk). I normally run tests locally without issues using ./gradlew connectedCheck.
Google released Android Testing Support Library 1.0 recently. After reading the overview, I'm a little confused with Android Test Orchestrator.
It said
Typically, AndroidJUnitRunner runs all tests in the same instrumentation process, which can cause a number of problems.
Can you explain which kinds of problems will be caused by using the same instrumentation process?
if one test crashes, it prevents the remainder of the test suite from running
Through my experience, one test crash won't prevent from other test cases from running. Please point out what I misunderstood here?
And from Android Testing Orchestrator developer guide,
For completeness, Android Test Orchestrator runs pm clear after each test.
So Android Test Orchestrator will run pm clear [test_package_name] after each test, right?
Through my test, pm clear [app_package_name] won't be executed after each test. That means the data of application under test will not be cleared. So test cases might still have dependencies on each other. For example:
Test case A stores a SharedPreference key-value
Test case B which runs after test case A can read out the value stored by test case A
Overall, after some trial, I did not find any advantage of Android Test Orchestrator. Can somebody help to address my confusion? Thanks.
After researching the issue a bit I can provide the following answers:
Typically, AndroidJUnitRunner runs all tests in the same instrumentation process, which can cause a number of problems.
As mentioned, AndroidJUnitRunner runs on the same instrumentation process so basically your tests are running state-full, this might wreak havoc if your tests have some sort of dependency on the process state. In Android test orchestrator each test is run in its own process so dependencies aren't an issue.
if one test crashes, it prevents the remainder of the test suite from running
The crash in question here is a crash of the process, not the activity/application. You can test this by inserting in one your tests System.exit(0); Typically, this will stop the whole test run while in Android test orchestrator the tests continue as expected.
For completeness, Android Test Orchestrator runs pm clear after each test.
This is an oversight of google and has been retracted from the official documentation as can be observed here.
Basically, The advantages of using Android test orchestrator is all about the separate process for each test which improves stability and ensures full execution of tests.
Android Test Orchestrator is a tool which allows you to run each of your app’s tests within its own invocation of Instrumentation.
This means that each test (method annotated with #Test ) will be run on a separate instance of AndroidJUnitRunner.
What issues does it solve?
When dealing with UI tests we identified 2 major problems which occur from time to time when run on CI or locally:
Occasional crashes which stop the entire test suite.
Test overlap.
Orchestrator can prevent crash to interrupt the whole test, eg. native crash.
But it may slow the tests