I need to run integration tests for an Android library that leverages native
code via JNI. This has a few implications:
This is a library, and as such has no GUI whatsoever. This means that test
frameworks like Espresso, Appium and the like are of no use, since there's no
GUI to test against.
This gives me some headaches with popular cloud services like App Center and
Firebase Test Lab, as they require both an UI instrumented test apk, written
with one of the supported UI testing frameworks, and an app apk to run these
tests against.
As said, the library contains C code loaded via JNI. Unfortunately, this
excludes the possibility of using Robolectric, since it doesn't support JNI.
Now there are a few questions:
Are there any Android simulation testing frameworks that support JNI?
Is there a cloud service out there which allows me to simply run a pre-built
Android instrumented unit tests apk, without any UI instrumented test
framework and without an actual app to be tested?
Have I missed a much simpler solution to my problem?
The problem with cloud services is that currently I only have an instrumented
unit test apk, which as such has no actual GUI testing. Furthermore, that's the
sole apk I have, since there's no actual Android app to be tested.
I am not aware of any Android mocking test frameworks, like Robolectric, that
support JNI, which is why I am looking into instrumented tests, on real or
emulated Android devices, in the first place.
Related
I am a tester, but putting my hands on in Android app development and have developed a EdTech app "Class eLearn Mobile app"
As of now, i am manually testing the application and pushing the updates to production.
But now i want to write automated cases in the same development project and implement CI/CD or DevOps. How can we do that? I know in Maven projects, we can have a separate test folder, where we can write functional tests. How can i do the same with Android apps?
Your inputs are highly appreciated.
I have made a library project consisting of an Android module and no application. This module relies on Android-specific functionality.
How do I run these tests with Firebase?
It seems like Firebase requires an app apk and a test apk, while I only have a test apk (*-androidTest.apk). I normally run tests locally without issues using ./gradlew connectedCheck.
Google released Android Testing Support Library 1.0 recently. After reading the overview, I'm a little confused with Android Test Orchestrator.
It said
Typically, AndroidJUnitRunner runs all tests in the same instrumentation process, which can cause a number of problems.
Can you explain which kinds of problems will be caused by using the same instrumentation process?
if one test crashes, it prevents the remainder of the test suite from running
Through my experience, one test crash won't prevent from other test cases from running. Please point out what I misunderstood here?
And from Android Testing Orchestrator developer guide,
For completeness, Android Test Orchestrator runs pm clear after each test.
So Android Test Orchestrator will run pm clear [test_package_name] after each test, right?
Through my test, pm clear [app_package_name] won't be executed after each test. That means the data of application under test will not be cleared. So test cases might still have dependencies on each other. For example:
Test case A stores a SharedPreference key-value
Test case B which runs after test case A can read out the value stored by test case A
Overall, after some trial, I did not find any advantage of Android Test Orchestrator. Can somebody help to address my confusion? Thanks.
After researching the issue a bit I can provide the following answers:
Typically, AndroidJUnitRunner runs all tests in the same instrumentation process, which can cause a number of problems.
As mentioned, AndroidJUnitRunner runs on the same instrumentation process so basically your tests are running state-full, this might wreak havoc if your tests have some sort of dependency on the process state. In Android test orchestrator each test is run in its own process so dependencies aren't an issue.
if one test crashes, it prevents the remainder of the test suite from running
The crash in question here is a crash of the process, not the activity/application. You can test this by inserting in one your tests System.exit(0); Typically, this will stop the whole test run while in Android test orchestrator the tests continue as expected.
For completeness, Android Test Orchestrator runs pm clear after each test.
This is an oversight of google and has been retracted from the official documentation as can be observed here.
Basically, The advantages of using Android test orchestrator is all about the separate process for each test which improves stability and ensures full execution of tests.
Android Test Orchestrator is a tool which allows you to run each of your app’s tests within its own invocation of Instrumentation.
This means that each test (method annotated with #Test ) will be run on a separate instance of AndroidJUnitRunner.
What issues does it solve?
When dealing with UI tests we identified 2 major problems which occur from time to time when run on CI or locally:
Occasional crashes which stop the entire test suite.
Test overlap.
Orchestrator can prevent crash to interrupt the whole test, eg. native crash.
But it may slow the tests
I have a web app that is using Firebase FCM to communicate with my android device.
On receiving a certain data message through Firebase, I need android to run Instrumentation tests (such tests are run in a separate test application or "test APK" which is developed in the androidTest folder).
How can I make an Intent that will launch the test APK?
You can't run instrumentation tests in response to an external stimulus like this. Instrumentation tests are always driven through a JUnit test harness, which requires yet some other controlling process to kick them off using the am instrument command line program on the device. This is typically Android Studio, which will install both your app APK and your test APK, then kick off the tests. Firebase Test Lab will also take both APKs kick off your tests on physical and virtual devices.
I would like to run Android UIAutomator Tests with AWS Device Farm. The Tests needs to be uploaded as a separate JAR to AWS. In Android Studio (1.4) UIAutomator Tests are part of the app project itself, so no dedicated JAR is generated.
How can I generate a JAR, that just contains the UIAutomator Tests and meets the requirements of AWS Device Farm?
You'll still be able to execute these tests using AWS Device Farm without converting them to a JAR.
On March 12, 2015, Google announced uiautomator 2.0. Without going into too much detail, the significant change made in this new version is that these tests are based on Android instrumentation (generated as APK files) instead of the previously used system of uiautomator 1.0 (generated as JAR files).
I would have to examine the gradle/build configuration further, but my guess is that you're using the new Android testing libraries and thus, using uiautomator 2.0. When you build a project such as this, it will generate two APK files, one for your application and another for your instrumentation tests.
When using AWS Device Farm, you'll want to take these two APK files and upload them using the INSTRUMENTATION test type. This test type works for all Instrumentation based frameworks/tools such as Espresso, uiautomator 2.0, and Robotium. The UIAUTOMATOR test type is specifically for older uiautomator 1.0 projects that still build and use JAR files for their test packages.