I'm investigating a cloud-based solution for our UIAutomator 2.0 testing needs, and I'm having quite a few issues with uploading APKs for our test project.
We have a stand-alone test project that exists separately from the main Android application project. The package for the project is in the format com.company.project and the package for the test project is com.company.project.test.
1) If I attempt to upload a test APK that has all the tests in androidTest (and the gradle uses androidTestCompile for dependencies), Firebase console complains that:
"Unable to find instrumentation package for com.company.project.test"
This is likely because our test project is standalone, and the tests actually live under com.company.project.test.test.
2) On the other hand, if I change the location of tests to java main folder in the project and change androidTestCompile to compile for the gradle dependencies, when I attempt to upload the test APK Firebase console complains again:
"We experienced an error while validating your APK. Please verify the APK is correct and upload again."
I can't get a single test running since I'm blocked both ways.
Anyone else attempt to use Firebase Test Lab with a similar project hierarchy? Help!
Finally figured it out. Don't even bother with the web UI, use the gcloud terminal to run your tests instead. It's WAY more customizable.
https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run
You can specify the --test-package to whatever your test package is, whether it's a standalone project or not.
Related
I have setup a project in Firebase to run Instrumented Test for my app. I created the project from Android Studio, then I went to the Firebase Console and moved the project to Blaze Plan. I logged in and setup the project in Android Studio, so far so good, the thing is that when I try to run the test in Firebase using the Sample Spark Configuration but Android Studio never finishes to upload the APK, I don't have any error neither, what I get in Android Studio RUN console are the following lines:
Testing started at 11:29 ...
Using Cloud Storage Bucket location test-bucket-id
Uploading app APK ...
So, it seems there are some issue witht the bucket or with the project, but I'm not able to read any error anywhere. So, where can I access any log to know what is going on? Does a firebase project expose logs though its web UI or does Android Studio allows me to get some kind of logs besides what I get on the build window?
I had the same issue. Turned out there wasn't anything wrong with the project or the bucket. All I had to do was specify the package of my instrumented tests instead of using the "all in module" option in the run configuration.
I tried to test an app with Firebase Test lab but get an error saying "testOnly APKs are not allowed."
C:\temp>gcloud firebase test android run --type instrumentation --app myapp.apk --test myapp-androidTest.apk --device model=Nexus10,version=22,locale=en,orientation=landscape --timeout 300s
Have questions, feedback, or issues? Get support by visiting:
https://firebase.google.com/support/
Uploading [myapp.apk] to Firebase Test Lab...
Uploading [myapp-androidTest.apk] to Firebase Test Lab...
Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-j9zwyqscmy0rw-k53tazzivjxvu/2017-10-16_11:56:43.691000_AcnJ/]
Test [matrix-1vxb29yr0b2z7] has been created in the Google Cloud.
Firebase Test Lab will execute your instrumentation test on 1 device(s).
Creating individual test executions...failed.
ERROR: (gcloud.firebase.test.android.run)
Matrix [matrix-1vxb29yr0b2z7] failed during validation: "testOnly" found in the Manifest. testOnly APKs are not allowed.
C:\temp>
But the firebase test works when I run it in Android Studio 3.0 RC1 by setting the deployment target options to "Firebase Test Lab Device Matrix".
I searched "testOnly" in my project but not found it. It looks like it was gradle injected the attribute into AndroidManifest.xml.
Does anyone know how to resolve it?
I have the same probllem. I finally resolved by generating the apk with 'Build > Build APK(s)'
The APk is generated at : app\build\outputs\apk\debug
The apk is generated witouth the 'testOnly' label and it runs on firebase.
I found this info on https://developer.android.com/studio/run/index.html
It was Android Studio added the field. Refer to the document developer.android.com/guide/topics/manifest/… "Android Studio automatically adds this attribute when you click Run". Not using Android Studio, I built an APK from the command line then I can upload it to Firebase Test Lab for testing.
When you use the instant run option and the APK generated in this fashion cannot be used for any kind of firebase test.
Instead select the build option above and then select the build APK option and the the APK generated in this fashion can be used for the firebase test.
I have made a library project consisting of an Android module and no application. This module relies on Android-specific functionality.
How do I run these tests with Firebase?
It seems like Firebase requires an app apk and a test apk, while I only have a test apk (*-androidTest.apk). I normally run tests locally without issues using ./gradlew connectedCheck.
I would like to run Android UIAutomator Tests with AWS Device Farm. The Tests needs to be uploaded as a separate JAR to AWS. In Android Studio (1.4) UIAutomator Tests are part of the app project itself, so no dedicated JAR is generated.
How can I generate a JAR, that just contains the UIAutomator Tests and meets the requirements of AWS Device Farm?
You'll still be able to execute these tests using AWS Device Farm without converting them to a JAR.
On March 12, 2015, Google announced uiautomator 2.0. Without going into too much detail, the significant change made in this new version is that these tests are based on Android instrumentation (generated as APK files) instead of the previously used system of uiautomator 1.0 (generated as JAR files).
I would have to examine the gradle/build configuration further, but my guess is that you're using the new Android testing libraries and thus, using uiautomator 2.0. When you build a project such as this, it will generate two APK files, one for your application and another for your instrumentation tests.
When using AWS Device Farm, you'll want to take these two APK files and upload them using the INSTRUMENTATION test type. This test type works for all Instrumentation based frameworks/tools such as Espresso, uiautomator 2.0, and Robotium. The UIAUTOMATOR test type is specifically for older uiautomator 1.0 projects that still build and use JAR files for their test packages.
I am looking at ways to automate Android build deployment for Beta/QA Testers via Jenkins. Are there any solutions out there that will allow me to deploy my android apk via jenkins to a test harness of some sort, from where my QA and beta testers could download and install the application ?
Everything almost work out of the box for Android on Jenkins now. You can :
use the Android Emulator Plugin and perform your tests using ant debug test
get emma coverage and reporting (you must export the emma reports as artefacts) using ant emma installd test
you can even get your unit test reports inside jenkins (export the reports as artefacts as well) using a special test runner : either the missing Android Test Runner or Zutubi test runner.
you can get lint analysis and results thanks to the Lint Plugin for Jenkins as well.
Everything that can be done with ant will also work with maven for more industrial configurations.
We are actually working on a better integration of other testing technologies and a better intregration with Sonar, this topic should get some progress during 2013.
For distributing your APK to beta testers, you can try services like HockeyApp or Google Play alpha/beta distribution, both of which have Jenkins plugins to upload your builds.
By using those services and their SDKs in your app, you automatically get usage statistics and crash reports from your testers.