I am running my Android Instrumentation tests with AWS Devicefarm but I want to know if there is a way to provide the ScheduleRun with a callback for status of the tests
Related
A basic Flutter app failed to launch on Firebase Test Lab for Android. I can see in the video that the app never came up so it never even tried running my tests, it just timed out. In the logs, I do not see any errors from my application; there are errors and warnings but I can't understand any of them nor do I know if they're ok to ignore. I used Firebase Test Lab on other projects and I'm accustomed to seeing lots of errors which I assume come from Firebase Test Lab's infrastructure. I typically look through the errors to try to find something that might pertain to my app. In this case I don't see anything that seems to be related to my app.
The app is from the code lab https://codelabs.developers.google.com/codelabs/flutter-codelab-first#0 .
The instructions for Firebase setup are at https://github.com/flutter/flutter/tree/main/packages/integration_test#firebase-test-lab .
The test passes when run locally by running the following command on my Mac:
./gradlew app:connectedAndroidTest -Ptarget=`pwd`/../integration_test/app_test.dart
The command I run from my CI (gitlab) is as follows:
gcloud firebase test android run --type instrumentation --app build/app/outputs/apk/debug/app-debug.apk --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk
I got the same timeout by initiating the test through the Firebase Test Lab UI.
I also posted this question at https://firebase-community.slack.com/archives/C1MTSQ5QT/p1674438388007639 .
Any ideas for what I could investigate?
I was able to get Firebase Test Lab to test a Flutter App by using this Espresso "bridge" https://pub.dev/packages/espresso. So instead of writing Flutter tests with the integrationDriver from package:integration_test/integration_test_driver.dart as described in https://github.com/flutter/flutter/tree/main/packages/integration_test#driver-entrypoint, I can use the enableFlutterDriverExtension() from package:flutter_driver/driver_extension.dart .
Not ideal.
Currently I run Espresso tests on Firebase / Google Test Lab, but it takes time to run them all, as they run in sequentially.
Is there a way to run them in parallel?
You would need to shard your tests so that you can kick off multiple batches of test that can run in parallel on different devices. Firebase doesn't provide a way to do this directly, but there is an open source project called Flank that can help.
I have a web app that is using Firebase FCM to communicate with my android device.
On receiving a certain data message through Firebase, I need android to run Instrumentation tests (such tests are run in a separate test application or "test APK" which is developed in the androidTest folder).
How can I make an Intent that will launch the test APK?
You can't run instrumentation tests in response to an external stimulus like this. Instrumentation tests are always driven through a JUnit test harness, which requires yet some other controlling process to kick them off using the am instrument command line program on the device. This is typically Android Studio, which will install both your app APK and your test APK, then kick off the tests. Firebase Test Lab will also take both APKs kick off your tests on physical and virtual devices.
I am running instrumentation tests on AWS Device Farm and can download the results of the tests but do not see any *.exec or *.ec files in the results. This is what I typically see when running tests locally with an attached device/emulator using Gradle.
Does AWS Device Farm support code coverage?
I work for the AWS Device Farm team.
Unfortunately, that artifact isn't exposed yet on device farm.
I have made a note of it in our product backlog.
I am curious if you can obtain the same thing if you were not running from Android studio at all.
I want to integrate my Jenkins build with AWS device farm. So, while creating a project on AWS device farm, it asks to upload the apk to test. I want that the apk should be picked from Jenkins server location and I don't have to manually upload apk to AWS device farm.
Is this possible?
Yes, you can configure Jenkins to upload your compiled application to AWS Device Farm. Install the AWS Device Farm Jenkins plugin and set Application to **/app-debug-unaligned.apk.
You can also watch our Jenkins integration webinar for more information.
To create a Project in AWS device farm is mandatory to upload an APK to proceed with the project creation.
The same will happen with the device pool, you can create a device pool only after creating a Run.
After you create that all this information will be available in Jenkins Server (of course after Authentication and plug in installed)
Another was is using AWS CLI but I can't give you more information as I haven't used yet