I am using Firebase test labs along with Jenkins to run a Robo test after each build of my Android app. Currently I use a Yaml file to setup all the configurations I need. I supply a username and password using the Robo Directives. However, I now want to supply a Robo Script that controls the sign in process a little bit more than just supplying the login details. I have made the script in Android studio, but I don't see anywhere in the documentation that allows me to supply the location of the Robo Script from the Yaml file or command line, does anyone know if this is possible?
I have tried the documentation and various links from here - https://firebase.google.com/docs/test-lab/command-line
This feature is available today in beta, but not as a full release. You can use the --robo-script flag to specify a script to upload when using
gcloud beta firebase test android run
It is not available in the release version:
gcloud firebase test android run
After some time in beta, new features will be promoted to full release. See the CLI docs for the beta version here.
I know this is an old question but I was looking for the same thing and I find it :
https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--robo-script
Using like this
gcloud firebase test android run --app app-release.apk --robo-script script.json
Related
A basic Flutter app failed to launch on Firebase Test Lab for Android. I can see in the video that the app never came up so it never even tried running my tests, it just timed out. In the logs, I do not see any errors from my application; there are errors and warnings but I can't understand any of them nor do I know if they're ok to ignore. I used Firebase Test Lab on other projects and I'm accustomed to seeing lots of errors which I assume come from Firebase Test Lab's infrastructure. I typically look through the errors to try to find something that might pertain to my app. In this case I don't see anything that seems to be related to my app.
The app is from the code lab https://codelabs.developers.google.com/codelabs/flutter-codelab-first#0 .
The instructions for Firebase setup are at https://github.com/flutter/flutter/tree/main/packages/integration_test#firebase-test-lab .
The test passes when run locally by running the following command on my Mac:
./gradlew app:connectedAndroidTest -Ptarget=`pwd`/../integration_test/app_test.dart
The command I run from my CI (gitlab) is as follows:
gcloud firebase test android run --type instrumentation --app build/app/outputs/apk/debug/app-debug.apk --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk
I got the same timeout by initiating the test through the Firebase Test Lab UI.
I also posted this question at https://firebase-community.slack.com/archives/C1MTSQ5QT/p1674438388007639 .
Any ideas for what I could investigate?
I was able to get Firebase Test Lab to test a Flutter App by using this Espresso "bridge" https://pub.dev/packages/espresso. So instead of writing Flutter tests with the integrationDriver from package:integration_test/integration_test_driver.dart as described in https://github.com/flutter/flutter/tree/main/packages/integration_test#driver-entrypoint, I can use the enableFlutterDriverExtension() from package:flutter_driver/driver_extension.dart .
Not ideal.
How can we have another apps installed in devices that is used in testing process( robotests , ...) , before starting testing our app?
I mean my app needs some other apps to be installed in the device to work completely and correctly .
Is it possible?
i'm using firebase test lab
You cannot upload additional APKs via the Firebase web console, but you can interact with Test Lab by using the gcloud command-line app too.
The "beta" version of the gcloud command-line has an option to add additional APKs. If you run
gcloud beta firebase test android run --help
You will find the following option that allows you to install more apks:
--additional-apks=APK,[APK,...]
A list of up to 100 additional APKs to install, in addition to those
being directly tested. The path may be in the local filesystem or in
Google Cloud Storage using gs:// notation.
Here's an example on how you would use this to start a test:
gcloud beta firebase test android run \
--app=build/outputs/apk/debug/app-debug.apk \
--test=build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=walleye,version=28 \
--additional-apks=path/to/another.apk
My company is trying out Flutter to create mobile applications and I'm currently in the process of trying to distribute both iOS and Android beta builds through Fabric.
The flutter plugin repo doesn't have any Fabric plugins and I can't seem to find any guides at the flutter home page nor elsewhere online on how to achieve this.
I had planned to focus first on iOS since I'm an iOS developer and figure android out later and use fastlane-match to manage my codesigning on iOS. What I've come up with so far is that I'm gonna try to create post-build-scripts discussed in this fabric guide and thus upload my builds manually after flutter builds and creates an IPA file. I'd say this is not optimal since it differs from the rest of our build setup, as in the fabric kit is not directly embedded in to our iOS project.
So I'm wondering if anybody has any experience in distributing flutter apps with Fabric and if you had any tips on how to achieve this, preferably with Fabric integrated into my iOS and android project respectively.
Edit:
For anybody interested in the solution, I ended up adding the Fabric and Crashlytics cocoapod into the generated cocoapod file by flutter and installing them via pod install. I'm not sure if that cocoapod file will get overwritten in the future as my experience with Flutter is quite limited.
After that I was able to import both Fabric and Crashlytics into my AppDelegate and Initializing the Fabric kit from there just like in a normal iOS project, as well as adding the runscript phase and plist entries as recommended by the fabric iOS install guide.
I also set the project up to use manual code signing and managed the codesigning with fastlane match.
After that I created a fastlane script that uploads to crashyltics. The build script I ended up with is:
flutter build ios --release --no-codesign
cd ios
fastlane ios beta
where ios references to my platform and beta is the name of the lane that exectues the fabric upload. The beta lane first builds with gym (to sign the IPA file) and then that signed IPA file is uploaded to crashlytics.
Fabric is now integrated to Firebase. It was deprecated after March 31, 2020. With this, you can now refer to Firebase App Distribution as an official documentation.
Flutter also has a bunch of documentation regarding distribution of Flutter applications to beta testers.
Continuous delivery with Flutter
Follow continuous delivery best practices with Flutter to make sure
your application is delivered to your beta testers and validated on a
frequent basis without resorting to manual workflows.
You can also check this blog for some useful tips and more information on Deploying Flutter apps to Firebase App Distribution using Fastlane:
Deploying Flutter app to Firebase App Distribution using Fastlane
In this article, I will be showing how to set up and deploy your
Flutter app to Firebase App Distribution with the help of
fastlane.
I tried to test an app with Firebase Test lab but get an error saying "testOnly APKs are not allowed."
C:\temp>gcloud firebase test android run --type instrumentation --app myapp.apk --test myapp-androidTest.apk --device model=Nexus10,version=22,locale=en,orientation=landscape --timeout 300s
Have questions, feedback, or issues? Get support by visiting:
https://firebase.google.com/support/
Uploading [myapp.apk] to Firebase Test Lab...
Uploading [myapp-androidTest.apk] to Firebase Test Lab...
Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-j9zwyqscmy0rw-k53tazzivjxvu/2017-10-16_11:56:43.691000_AcnJ/]
Test [matrix-1vxb29yr0b2z7] has been created in the Google Cloud.
Firebase Test Lab will execute your instrumentation test on 1 device(s).
Creating individual test executions...failed.
ERROR: (gcloud.firebase.test.android.run)
Matrix [matrix-1vxb29yr0b2z7] failed during validation: "testOnly" found in the Manifest. testOnly APKs are not allowed.
C:\temp>
But the firebase test works when I run it in Android Studio 3.0 RC1 by setting the deployment target options to "Firebase Test Lab Device Matrix".
I searched "testOnly" in my project but not found it. It looks like it was gradle injected the attribute into AndroidManifest.xml.
Does anyone know how to resolve it?
I have the same probllem. I finally resolved by generating the apk with 'Build > Build APK(s)'
The APk is generated at : app\build\outputs\apk\debug
The apk is generated witouth the 'testOnly' label and it runs on firebase.
I found this info on https://developer.android.com/studio/run/index.html
It was Android Studio added the field. Refer to the document developer.android.com/guide/topics/manifest/… "Android Studio automatically adds this attribute when you click Run". Not using Android Studio, I built an APK from the command line then I can upload it to Firebase Test Lab for testing.
When you use the instant run option and the APK generated in this fashion cannot be used for any kind of firebase test.
Instead select the build option above and then select the build APK option and the the APK generated in this fashion can be used for the firebase test.
I am looking at ways to automate Android build deployment for Beta/QA Testers via Jenkins. Are there any solutions out there that will allow me to deploy my android apk via jenkins to a test harness of some sort, from where my QA and beta testers could download and install the application ?
Everything almost work out of the box for Android on Jenkins now. You can :
use the Android Emulator Plugin and perform your tests using ant debug test
get emma coverage and reporting (you must export the emma reports as artefacts) using ant emma installd test
you can even get your unit test reports inside jenkins (export the reports as artefacts as well) using a special test runner : either the missing Android Test Runner or Zutubi test runner.
you can get lint analysis and results thanks to the Lint Plugin for Jenkins as well.
Everything that can be done with ant will also work with maven for more industrial configurations.
We are actually working on a better integration of other testing technologies and a better intregration with Sonar, this topic should get some progress during 2013.
For distributing your APK to beta testers, you can try services like HockeyApp or Google Play alpha/beta distribution, both of which have Jenkins plugins to upload your builds.
By using those services and their SDKs in your app, you automatically get usage statistics and crash reports from your testers.