I have seen several post regarding making a build in Jenkins fail if the unit test execution fail (e.g. this one). It turns out that by default Jenkins reports builds with failing tests as unstable and some people do not like that. This, however, will be perfectly fine for me. I just want to be able to easily differentiate builds with passing tests from such with failing tests.
And here is the catch: I am developing for Android so my build is configured following this page. Basically the tests are run with the following command:
ant all clean emma debug install test
As result coverage report is generated and published in Jenkins.
All posts I have read about configuring the Jenkins result according to tests results
were dealing with ant task manipulation. However, if we look at the android build.xml the Android tests are run with adb command: adb shell am instrument .... I don't know how to configure this command to print the tests results. It can be configured to print the coverage report. I have already done that, but was never able to make the build fail according to the coverage report.
I hope somebody else also faced the same problem and managed to solve it. Any guidance will be most appreciated.
Consider using 'the-missing-android-xml-junit-test-runner' - the idea being to produce jUnit reports that Jenkins understands. Disclaimer: I do have no personal experience with adb. However, I do know from experience that Jenkins deals with properly generated jUnit reports very gracefully.
This is possible, configure the server using the following documentation:
https://wiki.jenkins-ci.org/display/JENKINS/Android+Emulator+Plugin
See "Parsing monkey output" in the link.
Addition, Jenkins can test instrumentation, below have a link of a good place to start in the learning of instrumentation:
http://developer.android.com/resources/tutorials/testing/helloandroid_test.html
I hope it is useful.
Related
I've got my travis CI set up with espresso instrumental testing for my android library but some test fail on Travis CI while on my local machine (emulator and physical device) everything is okay and all test are passed..
I don't understand why on the server side someone fail, while on the local machine in my ide, with the same script command everything is okay.
Can someone help me out?
Thanks
Instrumented UI tests using Espresso Testing Framework depend on the emulator that you created.
We normally use Android Studio locally, and a different one in CI build servers.
The chosen server is not the reason of the failure, probably the screen size of the emulator.
I'm planning to port my JUnit tests to Android Testing framework.
Some of the tests only involve the JVM but not the Android system, is it still necessary to port them? Android is using Dalvik and will replace it with ART(Android Runtime) in Lollipop, both of which are different from the JVM (Please correct if I'm wrong.). In this sense, it seems necessary to port all the JUnit tests to the Android Testing framework.
However, some tutorial argues that
"If possible, you should prefer to run your unit tests directly on the
JVM as the test execution is much faster compared to the time required
to deploy and run the test on an Android device."
http://www.vogella.com/tutorials/AndroidTesting/article.html#androidtesting
I'm not an expert on JVM, Dalvik nor ART. Can someone clarify this issue? Is it necessary to port the tests that only involve JVM to Android Testing Framework?
Thanks!
This depends on what you would like to test and the environment in which you run your tests in.
In theory, pure non-Android-specific Java code (aka. POJOs) you write will work on whatever virtual machine you run it on--JVM, Dalvik, or ART.
If you agree with that theory, then if you are already running your tests on a JVM, there is no need to run it on Dalvik or ART, especially if you consider the speed at which JVM will execute the tests. (Running it on Dalvik involves building the apk, and installing it, and running an adb shell command which can take some time...)
Of course, if you feel that the behavior of your Java code may change depending on the runtime, then you should probably port those tests over to your Android tests.
Recently, I checked out Android Testing Support provided in Android SDK.
As far as I know, you can write Unit Tests, which can be executed directly on JVM in your computer or CI Server (e.g. Jenkins) without Android device or emulator. The only requirement is the fact that these tests and code, which is tested have to be written in pure Java without any dependencies to Android SDK and Android Context. These tests have to be located at src/test/java directory. You can switch test artifact in Build Variants in Android Studio to "Unit Tests" and run your tests with IDE. you can also run them via Gradle wrapper with the command: ./gradlew test
If your tests or code, which is going to be tested depend on Android SDK and Android Context, you should use Android Instrumentation tests, which have to be located in src/androidTest/java directory. These tests have to be executed on Android device or emulator. You can switch test artifact in Android Studio to "Android Instrumentation Tests" and run your tests with IDE or use Gradle wrapper command ./gradlew connectedCheck
You can check official article about Unit Testing at: http://tools.android.com/tech-docs/unit-testing-support . It's worth reading. It's a bit outdated, because now you can use build tools: 1.1.0 instead of 1.1.0-rc1, but rest of the information is up to date. After reading this article, you should understand this topic better.
We're trying to get code coverage analysis of a set of manual tests run on an Android application running on a device. We build our application using maven, so the cool emma and ant solution isn't viable.
Does anybody have explicit experience getting emma or cobertura (or any other open source coverage tool) working for manual testing of Android applications? We can't afford to purchase a solution (so no Clover, or Java Test Coverage Tool, etc.).
I am doing a little research on how to setup Continuos delivery pipeline for Android apps. I am aware of existing approaches of using Jenkins and Ant scripts for making a build. What I'd really like to do is when I do a git push on say a development branch, I'd like Jenkins to do a build on bunch of emulators (With different resolutions) run my Robotium scripts and email me results.
I am looking for end-to-end setup preferably the server would run on Ubuntu.
Siddharth,
You may want to consider CloudMunch for this need. All you need to do is launch the emulators as part your pipeline to test various stacks and finally get an update over mail. Robotium scripts can be called over shell and reports will come to you as mail at the end of the test.
After much trial and error I've got emma (code coverage tool) with ant for Android dev working purely on command line. Using Eclipse to try and run the test suite I get:
Conversion to Dalvik format failed with error 1
Which could possibly be part of the problem.
Any I've ran all the commands I've found such as
ant emma debug clean
ant emma debug
I use to run the test suite:
ant emma debug install test
Problem
However the report states that code is still untested even though I've written many new tests. IMO, I believe a new debug version isn't being complied, and I'm wondering if anyone has had similar problems. This could be due the the error I linked above, so currently I'm trying various remedies to resolve this first.
Many thanks
EDIT: could this be due to the generated build.xml files?
In my case, the problem was the Robotium tests whilst working perfectly fine, were was it not to compile. To be honest I'm not entirely sure why but removing the tests using Robotium library worked.