I have a Robolectric test project setup, but I'd like to also run these tests on my device to check that I don't get bit by JVM vs Dalvik implementation differences.
Unlike robolectric tests, I won't run these tests frequently. My concern is that there's little effort to maintain the test suite and that they verify actual device functionality.
What's the best way to do that?
What I've currently got:
My robolectric test project as a test case TestPackage. I created an Android Test project with a test case TestRoboOnAndroid. It creates a TestPackage and has a test for each test in TestPackage.
Right now, every time I add a test to my robolectric suite, I need to manually add it to my device suite. Is there some way to do that automatically with reflection?
Also, Robolectric uses JUnit 4 (by default) and Android uses JUnit 3. So I have to write all of my Robolectric tests using JUnit 3 style (importing from junit.framework instead of org.junit).
The whole point of Robolectric is NOT to run it on the device and the design is based on that. If you want to run something on the device look at the default instrumentation tests from the SDK or Robotium.
It is totally feasible to run your Robolectric tests on the JVM and in addition create Robotium tests that rely on device specific interaction (e.g. creating screenshots, hardware differences...) and run on devices and emulators all combined into one build.
The easiest way to do that is to use the Android Maven Plugin.
I use a tiered system, where I prefer earlier tiers where possible:
Pure unit tests. I try to make as much code as possible fully independent of Android APIs, and then use "pure" unit tests which can run on any JVM. These tests are the fastest, and it helps keep code that has no need to be Android-specific portable.
Robolectric-supported unit tests. Where my code has only small dependencies on Android APIs, that can be satisfied by Robolectric shadows, I test it with Robolectric. There is a little more setup time for Robolectric compared to pure tests, but it's still faster than starting/running on an emulator.
Android framework tests. Where Robolectric doesn't cut it - either because the shadows don't exist, or because I'm heavily using Android APIs (and therefore want to test against the Real Thing) - I write test that run on the emulator/device with the default framework.
The point of the tiers is to keep things as simple as possible, which keeps the full suite faster and helps promote cleaner code.
We use Robolectric for unit testing against the JVM and Calabash-Android for system testing against Dalvik. Both can be integrated into our Jenkins CI build and between the two tools I feel that we cover all the bases.
Related
I'm a fanboy of Test-pyramid application in software development and want the same to be applied in Mobile software development as well.
For this, I'm desiring to leverage both Appium (for functional tests) and Espresso (for Integration Tests), and have the following questions:
The structure of my tests in Android Studio with Gradle has only 2 folders: app/src/test and app/src/androidTest.
The former test is meant for writing Unit tests with JUnit. The later for functional tests; and I'm using espresso tests for integration-testing making use of the latter directory.
I know I can create a separate project for Appium and write functional tests. But prefer to have those as part of the project in separate test source directory, so that it can be run at different intervals in my CI pipeline. How do I go about doing this? Is it possible to create another source test directory like app/src/functionalTest? Pointers to sample projects in github or resources is well appreciated.
I'm planning to port my JUnit tests to Android Testing framework.
Some of the tests only involve the JVM but not the Android system, is it still necessary to port them? Android is using Dalvik and will replace it with ART(Android Runtime) in Lollipop, both of which are different from the JVM (Please correct if I'm wrong.). In this sense, it seems necessary to port all the JUnit tests to the Android Testing framework.
However, some tutorial argues that
"If possible, you should prefer to run your unit tests directly on the
JVM as the test execution is much faster compared to the time required
to deploy and run the test on an Android device."
http://www.vogella.com/tutorials/AndroidTesting/article.html#androidtesting
I'm not an expert on JVM, Dalvik nor ART. Can someone clarify this issue? Is it necessary to port the tests that only involve JVM to Android Testing Framework?
Thanks!
This depends on what you would like to test and the environment in which you run your tests in.
In theory, pure non-Android-specific Java code (aka. POJOs) you write will work on whatever virtual machine you run it on--JVM, Dalvik, or ART.
If you agree with that theory, then if you are already running your tests on a JVM, there is no need to run it on Dalvik or ART, especially if you consider the speed at which JVM will execute the tests. (Running it on Dalvik involves building the apk, and installing it, and running an adb shell command which can take some time...)
Of course, if you feel that the behavior of your Java code may change depending on the runtime, then you should probably port those tests over to your Android tests.
Recently, I checked out Android Testing Support provided in Android SDK.
As far as I know, you can write Unit Tests, which can be executed directly on JVM in your computer or CI Server (e.g. Jenkins) without Android device or emulator. The only requirement is the fact that these tests and code, which is tested have to be written in pure Java without any dependencies to Android SDK and Android Context. These tests have to be located at src/test/java directory. You can switch test artifact in Build Variants in Android Studio to "Unit Tests" and run your tests with IDE. you can also run them via Gradle wrapper with the command: ./gradlew test
If your tests or code, which is going to be tested depend on Android SDK and Android Context, you should use Android Instrumentation tests, which have to be located in src/androidTest/java directory. These tests have to be executed on Android device or emulator. You can switch test artifact in Android Studio to "Android Instrumentation Tests" and run your tests with IDE or use Gradle wrapper command ./gradlew connectedCheck
You can check official article about Unit Testing at: http://tools.android.com/tech-docs/unit-testing-support . It's worth reading. It's a bit outdated, because now you can use build tools: 1.1.0 instead of 1.1.0-rc1, but rest of the information is up to date. After reading this article, you should understand this topic better.
I have some plain old Java classes and simple JUnit tests in my Android Studio project. They currently live inside my Android Module. If I run them as Android tests they pass fine, but they're really slow as it requires launching or connecting to the emulator.
If I try to run one as a JUnit test, I hit the !!! JUnit version 3.8 or later expected JUnit version problem. This answer explains how to workaround that issue. Unfortunately, I then hit the java.lang.NoClassDefFoundError: junit/textui/ResultPrinter, which I can't figure out how to fix.
I then tried to move my JUnit tests into a separate Java module that depends on the Android Module. Everything compiles fine. I added a new configuration to launch the JUnit tests (:MyJUnitTests:test), but the tests fail with package com.myproject.util does not exist. It looks like maybe the classpath isn't including the classes from the dependent Android Module.
Does anybody know how to fix this classpath issue? I've looked at lots of related answers and none of them seem to work for me. Or is it a just a bad idea to try and keep my JUnit tests in their own Module?
I can get the JUnit tests to run fast and pass if I move my plain Java classes and their JUnit tests into a completely separate Module (e.g. here), and reverse the dependency so the Android Module depends on the Java Module. Is that the preferred solution?
The basic problem is that Android Framework classes don't work well outside the context of Android OS, so any code that has dependencies on Framework classes doesn't run in a regular JUnit environment, as you've found.
Your solution of trying to move the JUnit tests into a separate module won't work because you can't have a plain Java module depend on an Android module. Android Gradle modules don't act like Java modules, because Android builds are much more complex, and because the end result of an Android module build will be an APK or an AAR, which other module types won't understand.
If you can move your plain Java classes and unit tests into a plain Java module and have the Android modules depend on that, it will be the best approach that will get the most support from official features in the IDE. It will also have an architectural benefit in that it will keep those classes free from Android abstractions, making it more portable, and enforcing a separation of business logic from UI or storage or other things more device-specific.
If that's difficult for you to do, then a lot of developers in your shoes go with Robolectric, which is a test harness allowing code depending on many parts of the Android Framework to run in a JUnit environment without an emulator or device. It's not officially supported by Google, but Google is aware that it's widely used and tries hard to not wantonly break it.
Why do you need to have an emulator or device plugged in for the testing to take place? Why can't I just generate coverage for my tests without the use of these?
It all depends on your Unit-Test.
If you're testing classes that dont use at all the Android Framework, and only the standard Java classes, then you could run the Unit test and get a code coverage (althouhg I don't have a step by step procedure to give you).
But if you're using any part of the Android Framework (that is, any class in an android.* or com.android.* package), then you'll need to run in a DalvikVM, meaning an Android emulator.
Is anyone using Junit and Android? Or is that just a worthy hope? Is there a tutorial anywhere?
Android has great support for JUnit 3
From Testing Fundamentals on Android Developers:
The Android testing framework, an integral part of the development environment, provides an architecture and powerful tools that help you test every aspect of your application at every level from unit to framework.
The testing framework has these key features:
Android test suites are based on JUnit. You can use plain JUnit to test a class that doesn't call the Android API, or Android's JUnit extensions to test Android components. If you're new to Android testing, you can start with general-purpose test case classes such as AndroidTestCase and then go on to use more sophisticated classes.
The Android JUnit extensions provide component-specific test case classes. These classes provide helper methods for creating mock objects and methods that help you control the lifecycle of a component.
Test suites are contained in test packages that are similar to main application packages, so you don't need to learn a new set of tools or techniques for designing and building tests.
The SDK tools for building and tests are available in Eclipse with ADT, and also in command-line form for use with other IDES. These tools get information from the project of the application under test and use this information to automatically create the build files, manifest file, and directory structure for the test package.
The SDK also provides monkeyrunner, an API testing devices with Python programs, and UI/Application Exerciser Monkey, a command-line tool for stress-testing UIs by sending pseudo-random events to a device.
This document describes the fundamentals of the Android testing framework, including the structure of tests, the APIs that you use to develop tests, and the tools that you use to run tests and view results. The document assumes you have a basic knowledge of Android application programming and JUnit testing methodology.
I have been using Roboelectric which is awesome because it does not launch the simulator making the run time for the tests very very quick. A sample project is provided as an example and can be found here at github
You can also use Robotium to drive the UI from within JUnit for more functional style testing.
"Note that the Android testing API supports JUnit 3 code style, but not JUnit 4." (Source)
If you want to use JUnit4 or have existing JUnit4 tests you can use JUnit4Android.