I have a strange situation with Jenkins when trying to run the tests.
The configuration of the job clearly specifies to run the tests on this specific avd: "hudson_en-US_160_HVGA_android-18_armeabi-v7a", but for some odd reasons, during the build process, 2 devices are found. One of the devices is the previous one, but the other is called "unknown-sdk".
The tests are successfully executed on the specific avd, and fail on "unknown-sdk":
The relevant portion of the log:
[SR.runTests] Executing instrumentation suite on 2
device(s).
[SR.runTests] [emulator-8215] Starting
execution.
[SR.runTests] [localhost:8216] Starting
execution.
Running am instrument -w -r -e class com.smoke.tests.LoginTest
com.muume.dev.test/android.support.test.runner.AndroidJUnitRunner on
unknown-sdk-localhost:8216
2015-06-17 11:54:05 [SDR.printStream] [localhost:8216] STDOUT 11:54:05
I/InstrumentationResultParser: test run failed: 'Unable to find
instrumentation info for:
ComponentInfo{com.muume.dev.test/android.support.test.runner.AndroidJUnitRunner}
As you can see, it detected 2 devices: emulator-8215 which is the known avd, and localhost-8216 which is the unknown and causes the problems.
My question is how to restrict the tests to run only one single emulator, and from where did the "unknown-sdk-localhost" comes from.
"unknown-sdk" is what it tells adb while the emulator is still busy loading itself.
Related
I saw some links and posts about running AndroidTest in Docker. Like:
https://dzone.com/articles/running-android-tests-in-docker
https://github.com/ksoichiro/android-tests/tree/master/docker-emulator
And some more. However, they all seem inappropriate for AndroidTest in CI, since they require an actual VM, or simply they are too old.
I tried the following lines Docker Image (Partial):
RUN /opt/adk/tools/bin/sdkmanager "emulator" "build-tools;${BUILD_TOOLS}" "platforms;${ANDROID_PLATFORM}" "system-images;${ANDROID_PLATFORM};google_apis;armeabi-v7a" \
&& echo no | /opt/adk/tools/bin/avdmanager create avd -n "Android" -k "system-images;${ANDROID_PLATFORM};google_apis;armeabi-v7a" \
And then I run the emulator using:
/opt/adk/emulator/emulator #Android -writable-system -nocache -no-snapstorage &
However, when trying to run connectedAndroidTest there were some weird exceptions, like:
Caused by: com.android.ddmlib.InstallException: Unknown failure: Error: Could not access the Package Manager. Is the system running?
Or, if I try to manually install the test APK, and run am instrument:
android.util.AndroidException: Can't connect to activity manager; is the system running?
So my question: Is anyone is running AndroidTest on emulator as part of the CI on docker? Means, an image that you just need to mount your project and run gradle connectedAndroidTest, and everything works? Is there a working example for that?
I run AndroidTest on emulator as part of the CI on docker using this image: https://hub.docker.com/r/chrisss404/android-emulator
The problem you might encounter is that hardware acceleration is not available on your host because of missing nested KVM (just a guess, but that was the problem I was facing). In this case you have to use software rendering, which can take a significant amount of time.
With software rendering, the startup takes about 40 minutes until the emulator is usable (on my host). However, then you can run the instrumentation tests as usual in an acceptable timeframe.
I am unable to run instrumentation tests in my Android project in two cases:
When running individual tests (i.e. a single #Test)
When running all tests in a class
I am able to run instrumentation tests by right clicking on the folder the test is in and selecting the Run 'Tests in PACKAGE_NAME' option
Here's an example of running an individual test:
Testing started at 6:35 PM ...
12/30 18:35:05: Launching getId()
$ adb push /Users/zach/Developer/Code/testapp/app/build/outputs/apk/develop/debug/app-develop-debug.apk /data/local/tmp/com.zao.testapp.develop
$ adb shell pm install -t -r "/data/local/tmp/com.zao.testapp.develop"
Success
APK installed in 4 s 42 ms
No apk changes detected since last installation, skipping installation of /Users/zach/Developer/Code/testapp/app/build/outputs/apk/androidTest/develop/debug/app-develop-debug-androidTest.apk
Running tests
$ adb shell am instrument -w -r -e debug false -e class 'com.zao.testapp.models.entities.impl.EntityParseTest#getId' com.zao.testapp.develop.test/com.zao.testapp.TestRunner
Client not ready yet..
Started running tests
Test running failed: Fatal exception when running tests
java.lang.IllegalArgumentException: Ambiguous arguments: cannot provide both test package and test class(es) to run
at android.support.test.internal.runner.TestRequestBuilder.validate(TestRequestBuilder.java:773)
at android.support.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:742)
at android.support.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:354)
at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:260)
at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2075)
Empty test suite.
And when I run via the right click option to run all tests in the package, here is what I see:
Testing started at 6:40 PM ...
12/30 18:40:38: Launching Tests in 'com.zao.testapp.models.entities.impl'
$ adb push /Users/zach/Developer/Code/testapp/app/build/outputs/apk/develop/debug/app-develop-debug.apk /data/local/tmp/com.zao.testapp.develop
$ adb shell pm install -t -r "/data/local/tmp/com.zao.testapp.develop"
Success
APK installed in 4 s 471 ms
No apk changes detected since last installation, skipping installation of /Users/zach/Developer/Code/testapp/app/build/outputs/apk/androidTest/develop/debug/app-develop-debug-androidTest.apk
Running tests
$ adb shell am instrument -w -r -e package com.zao.testapp.models.entities.impl -e debug false com.zao.testapp.develop.test/com.zao.testapp.TestRunner
Client not ready yet..
Started running tests
Tests ran to completion.
The only "gotcha" here might be that I am using a custom test runner (TestRunner.java that is referenced above). I'm not sure that matters too much though. I'll try removing this (some of the tests need the custom runner, but I can disable those for now...)
Any thoughts? Thanks!
I debugged this issue and figured out the root cause. I was using a custom runner that was ignoring a package from being tested. I implemented a workaround described in https://github.com/mockito/mockito/issues/922 to get my mockito stuff working with the custom runner, and this resulted in the error.
Putting arguments.putString("notPackage", "net.bytebuddy"); in the runner's onCreate resulted in TestRequestBuilder.java throwing an exception in the validate method, which does this:
if ((!mIncludedPackages.isEmpty() || !mExcludedPackages.isEmpty()) && !classNames.isEmpty()) {
throw new IllegalArgumentException(AMBIGUOUS_ARGUMENTS_MSG);
}
Where mExcludedPackages has a size of 1 (net.bytebuddy), while classNames is also not empty and contains the class I am running tests in.
Therefore to fix this you can either do one of two things:
Remove the arguments.putString("notPackage", "net.bytebuddy") workaround (which makes mockito not work in some cases for my tests).
Refactor your tests to not need the custom runner, thereby doing step 1 as well.
Not be able to run tests for specific tests/files.
I ended up doing step 2, FWIW.
Background
I want to run my Android Instrumented tests on Jenkins on different emulators. Say I have 100 tests and 4 emulators, I want to run 25 tests on each.
I perform ./gradlew connectedDebugAndroidTest in Jenkins Pipeline's parallel for 4 emulators
stage('Instrumented Tests') {
parallel(
emu1: {
runInstrumentedTestOnEmu(...)
},
emu2: {
runInstrumentedTestOnEmu(...)
}
...
)
}
connectedDebugAndroidTest will spawn other commands in order to setup the environment for running instrumented tests.
...
:app:transformNativeLibsWithMergeJniLibsForDebugAndroidTest
:app:processDebugAndroidTestJavaRes NO-SOURCE
:app:transformResourcesWithMergeJavaResForDebugAndroidTest
:app:validateSigningDebugAndroidTest
:app:packageDebugAndroidTest
:app:assembleDebugAndroidTest
:app:connectedDebugAndroidTest
And when environment is ready then it performes :app:connectedDebugAndroidTest which will start running tests on emulator.
I do not want to run these procedure for all my parallel calls (in this case it would be 4 of them), because obviously I'm doing the exact same job multiple times. Theoretically, the best option would be to perform setup before parallel and when everything is ready for running tests, then go into parallel step and start tests on each emulator.
Question
Is it possible to perform all the pre-setup steps of connectedDebugAndroidTest without performing itself?
Additionally, if I run connectedDebugAndroidTest parallel on 4 emulators the build crashes, because gradle tries to read a file from intermediate directory, when other parallel build has already removed that file, which results in crash.
You can view this test project in github with setup mentioned above.
Is it possible to perform all the pre-setup steps of connectedDebugAndroidTest without performing itself?
Yes, you can run assembleDebugAndroidTest, which as your build log shows, is the last prerequisite to running the device tests. Running that will build both the app and test APKs.
Though AFAIK, there isn't a way of sharding your tests across multiple emulators when using Gradle — you would have to install both of the APKs onto each emulator and use adb shell am instrument with the numShards and shardIndex options.
We have a library project and multiple applications depends on it. And the unit tests are in the library project.
We're able to run the tests from dependent projects in Android Studio, but
./gradlew :[DependentProject]:connectedAndroidTest
always returns "No test found, nothing to do”.
Through observation, I found in Android Studio, seems that it only executes gradle tasks:
:[DependentProject]:assembleDebug, :[DependentProject]assembleDebugTest
then uses adb to install the target and test apk, and adb shell am instruments to run the tests.
Since connectedAndroidTest depends on these two tasks, I install the target and test apks it produced, and manually invoked instrument command, tests got started.
adb shell am instrument -w com.package.test/android.test.InstrumentationTestRunner
Then the question comes, where does connectedAndroidTest look for tests, and why it cannot find the tests while adb instrument can? How to resolve this issue?
I have the same issue and I solve it by add a method starting with "test"
#Test
public void testWTF() throws Exception {
assertTrue(true);
}
And all other method with #Test annotation work too !
Amazing no ? I found the answer here : No tests found with test runner 'JUnit 4'
I first ran into this problem when trying to run Robotium UI tests on a MacOS Mini with no monitor. I added Robotium code to test UI clicks on screen, since we want to add some UI automation to our regression testing. The error I get on the Jenkins server is when solo.clickInList(X) is run:
junit.framework.AssertionFailedError: AbsListView is not found!
[exec] at com.jayway.android.robotium.solo.Waiter.waitForAndGetView(Waiter.java:492)
[exec] at com.jayway.android.robotium.solo.Clicker.clickInList(Clicker.java:406)
[exec] at com.jayway.android.robotium.solo.Clicker.clickInList(Clicker.java:388)
[exec] at com.jayway.android.robotium.solo.Solo.clickInList(Solo.java:1081)
[exec] at com.cars.android.robotium.TestApplication.testSearch(TestApplication.java:29)
I found out that Jenkins needs Xvnc plugin to fake a UI so that the emulator can use it. Make sure to check "Show emulator window" in the specific job. Now doing that I currently get this error.
/Applications/RealVNC/VNC\ Server.app :$DISPLAY_NUMBER -localhost -geometry 1280x1024 -depth 24 -SecurityTypes None
I now get this error in Jenkins
Starting xvnc
[workspace] $ "/Applications/RealVNC/VNC Viewer.app" :77 -localhost -geometry 1280x1024 -depth 24 -SecurityTypes None
FATAL: Cannot run program "/Applications/RealVNC/VNC Viewer.app" (in directory "/Users/Shared/Jenkins/Home/jobs/Android/workspace"): error=13, Permission denied
java.io.IOException: Cannot run program "/Applications/RealVNC/VNC Viewer.app" (in directory "/Users/Shared/Jenkins/Home/jobs/Android/workspace"): error=13, Permission denied
You don't say whether the Robotium tests work on your local machine, so I don't know whether the Robotium code itself is correct.
But regarding the Xvnc setup, I see a couple of probable issues:
"VNC Viewer.app" sounds like a VNC client, rather than an X11 server.
As far as I know, you can't just run "/Applications/Foo.app" on the command line. You usually need to run the actual binary; something like "/Applications/Foo.app/Contents/MacOS/foo".