Wondering which one is the better choice to write unit test cases for android apps and libraries: Using Robolectric library or sticking with Android Testing framework. I want to run test suite at commandline and want it be independent of need of configuring emulator or letting a device attached with build machine. Does anyone of you run a comparative analysis on both of these or something better? Your experiences will be great help me to decide on the better solution.
I use a tiered system, where I prefer earlier tiers where possible:
Pure unit tests. I try to make as much code as possible fully independent of Android APIs, and then use "pure" unit tests which can run on any JVM. These tests are the fastest, and it helps keep code that has no need to be Android-specific portable.
Robolectric-supported unit tests. Where my code has only small dependencies on Android APIs, that can be satisfied by Robolectric shadows, I test it with Robolectric. There is a little more setup time for Robolectric compared to pure tests, but it's still faster than starting/running on an emulator.
Android framework tests. Where Robolectric doesn't cut it - either because the shadows don't exist, or because I'm heavily using Android APIs (and therefore want to test against the Real Thing) - I write test that run on the emulator/device with the default framework.
The point of the tiers is to keep things as simple as possible, which keeps the full suite faster and helps promote cleaner code.
I worked on both, what i found is :-
1) Robolectric do not support API 19, it's mention in its document -
http://robolectric.org/eclipse-quick-start/. It's a great
disadvantage of it.
2) Robolectric run on JVM not on DVM. So we can not
detect that on that particular time GPS is enable in device or not
etc. We can only pass our pre-decided value for it.
3) Code writing in Robolectric is complex than junit specially for
fragment there are lot of complexity and issues.
4) Robolectric needs external jar and configuration and for junit test
we do not need any external library.
5) Robolectric is faster because it runs on JVM but this have
disadvantage too, we can not see UI on our device, what screen code
is executing.
For Android, i like jUnit test.
Related
I have go through Test Android Apps in Android document. Google introduced AndroidX testing, but I am still very of confused of its implications.
I am using MVVM Android architecture components with ViewModel, LiveData.
Which parts of the Android X Framework do I need to use?
View(Activity.Fragments) --> Espresso?
ViewModel(with LiveData) --> RoboElectric?
Model --> JUnit or RoboElectric?
Why is Google still confusing us with lot of libraries, why can't they move all Local test, Instrumentation test, and everything under the same Framework(Espresso)?
The Google Testing team has made many improvements to their testing framework AndroidX.
Why Google still confusing with lot of libraries, why cant they move all Local test, Instrumentation test everything under same Framework(Espresso)?
With AndroidX Test, they are actually attempting to solve this!
First off, regular "Junit" unit Tests are unchanged. You should test your java code on the JVM the same way you were doing.
But now, with AndroidX, they have made the Robolectric 4.0 API the same as Espresso's so it is now possible to run your Espresso Tests as Robolectric tests. This means when developing locally, you can iterate more quickly because the Instrumentation Tests can run so much faster instead of having to run on emulators. It also makes it a lot easier to learn Robolectric, because you are able to use the same API as espresso.
When you get to the PR or CI phase, you can then submit those same tests that ran on Robolectric JVM to run on your CI on real devices using Espresso.
I suggested checking out Testing Rebooted (with AndroidX Test) (Android Dev Summit '18) for more context.
In this picture, the failing UI and Unit tests would represent the same test but represented as Espresso/Robolectric Instrumentation Test depending on what you want to run and when. For clarity, you would want to run it as Robolectric when you are doing local development especially with TDD. When ready to submit a PR, you can then run the same test on CI with emulator or real device.
So, while AndroidX.test can be confusing, I suggest watching this video to get a bit more of a grasp on Google's intent with the framework.
Also, I agree they can be providing more examples and guidance on using the new Androidx.Test!
Does Robolectric provide any clear benefits compared to Android Test Framework ? I've read the docs regarding both the frameworks but as far as i can see the only clear cut benefit regarding Robolectric is that it runs on JVM rather than the DalvikVM, making it faster than Android framework.
Are there any other major benefits that stand out ?
Update Apr-2015: Gradle build tools and Android Studio now officially support unit testing and prevent android.jar from throwing stub (no real implementation) error. So, yes its possible to run tests on Java VM, when stubs are appropriately mocked. Its a start but is still not comparable with Robolectric's power. There is also a third alternative, Scroll to bottom of this answer.
Now, about Robolectric :
Pros: here are a few points about how it has proved to be useful in unit testing:
You need not run an emulator, So you can test some of the project's non UI parts without requiring an emulator or device. This also applies to running tests on continuous integration/ build servers, no emulator instances need be launched.
With Android Studio, you can quickly run one particular test class, when you are working on the implementation to satisfy the test cases. You can debug as you write code. This is a massive productivity gain.
Can fake almost every android related thing as shadow objects, even SQLite. Plus each shadow object exposes many useful functions which their normal android counterparts don't offer. With shadow counterparts of android Object, you can do internal checking or call special methods.
Really shines when testing multi threaded code like AsyncTasks, Loopers and Handlers etc. You can pause and fast-forward thread Loopers, even main thread. Excellent for Handler based callback testing.
JUnit 4 format supported. Android is still holding onto JUnit 3 last time I checked.
Can be combined with other test tools like Mockito, Espresso etc etc.
Supports mock Activity instance creationRobolectric.buildActivity() and its control via ActivityController. Fragment/View manipulation also works on such mock activity instances.
There are now provided add-on modules that cover multi-dex, v4-support, play services, maps and http client as well. So, its now easy to test code using these library functions as well.
Cons: Where I found it not so good:
Robolectric excels at aiding Unit testing, but does not cover all the functionality a real device or emulator can offer. For example sensors, gps, open-gl etc etc.
You'll need an emulator or real device when doing integration or UI testing, so that Activities and services can interact with full android environment (other apps, like using camera app to get a picture for your app), not a limited one. Here you'll need to use the default test framework as it has functions to test UI as well.
JNI loading seems not to be supported. So code with native dependency can't be tested.
As of now, Robolectric has a hard wired dependency on google maps jar to work. And will download another android.jar from maven. So, project setup may require a bit of a tinkering. Update: as of v3 it seems to pull all dependencies via Gradle without much fuss.
Newer Android tools support coverage and reports generation etc, but only when test are run on a device. So with Robolectric you'll have to create extra Gradle tasks (run Jacoco) to do it for you. Update: Gradle 2.9 + ships with jacoco plugin.
As both gradle and android build tools are shipping out newer build versions at a fast rate, stable Robolectric versions will sometimes start having problems with the changed build tooling. Most typical problems include: sdk version incompatible, manifest not found, build output paths mismatch, resources not loading, build config issues etc. Some issues are also related to bugs in android tools. At times you may even have to write your own custom test runner or apply workarounds till next version fixes those issues. Check out open issues and configure tests accordingly.
Another alternative is simply mock stuff on your own, no frameworks involved. Its the "hard way" but the most customizable way. Its plain JUnit with JMockit:
#RunWith(JMockit.class)
public class OtherTest {
public void testHandlerCallback(#Mocked final FragmentTransaction transaction,
#Mocked final FragmentManager manager,
#Mocked final Activity activity,
#Mocked final LayoutInflater inflater,
#Mocked final ViewGroup parent) {
final List<Fragment> fragments = new ArrayList<>();
new Expectations() {{
activity.getFragmentManager(); result = manager;
manager.beginTransaction(); result = transaction;
transaction.add(withCapture(fragments), anyString);
transaction.commit(); result = new Delegate<Void>() {
public int commit() {
View v = fragments.get(0).onCreateView(inflater,parent,null);
Deencapsulation.invoke(v,"onMeasure",0,0);
return 0;
}
};
}};
}
}
Above is a crude and inline example. You can actually create proper re-usable classes (say FragmentTestHarness) that will take a component (say a Fragment) under test and wrap it in completely isolated environment, preparing it for tests.
To share how i do...
Robolectric
For SQL, activities flow, for those objects that needed context.
JUnit4
for api's java module to make sure data are return correctly.
Espresso
For checking ui display correctly.
When I modified api...I only run jUnit4.
When I modified the data binding between api and UI or Sqlite, then I will only run Robolectric.
When I modified UI I only run Espresso.
Sometimes I will run Robolectric and espresso together, but very rare.
But I will run all before publish to play store.
Because I think there is not real benefit for now. But see how u use it to speed up your product quality and development speed.
Correct me if I am wrong.
You would use Robolectric only in cases you need to mock or fake the Android framework, e.g. if you need a context. When using the Android Test Framework you would have to run Instrumented tests for that which is ultra slow.
If you write your tests for letting them run frequently, e.g. because you follow a tdd approach this is not an option. So in this case Robolectric comes in handy.
So the main benefit about Robolectric is that it is a lot faster than Espresso or Instrumented tests in general.
The downside is that it fakes an Android environment which you should be aware of. To validate real world problems, better use a classic Android Framework approach.
The best is still to write your code in a way that you can unit test it and that you don't need a context or any other Android framework dependencies.
Robolectric is getting integrated into the Android Testing Framework since I/O 2018 - check out more at the official Robolectric page and watch this video from I/O 2018
The major benefit of Robolectric is speed. Unit tests with Robolectric do not require a running emulator or device to run the tests and are thus much much faster.
You might still want to have a suite of integration tests that run against a real device, just a much smaller suite.
I'm taking over an android project and I wish to introduce unit tests to the project, to help avoid possible regressions.
For normal java projects, I have two source folders: src and test. The src source folder contains all of my source files and my test source folder contains all of my unit tests, which I believe is pretty standard for keeping test separate from the source, so you don't have to ship with them.
I've been doing some reading online and the approach with android apps looks to be a little bit different. Several examples talk about setting up a 2nd project for an android test project and then referencing the android project.
I wish to confirm a few things:
Is having a 2nd project for the testing the appropriate thing to do when it comes to testing android projects or am I just finding bad examples?
Should all unit tests be android unit tests? E.g. Yes they should all be, or no I should mix between android unit tests and junit because junits have less overhead.
What additional benefits do android unit tests give over junit tests? E.g. Handles to the emulator, etc.
Is having a 2nd project for the testing the appropriate thing to do
when it comes to testing android projects or am I just finding bad
examples?
Yes, it's typical to have a separate "test project" for testing Android specific code. http://developer.android.com/tools/testing/testing_android.html
Should all unit tests be android unit tests? E.g. Yes they should all
be, or no I should mix between android unit tests and junit because
junits have less overhead.
You'll usually have a mix because you can't test Android specific code on a standard JVM with regular ole' JUnit (not without some helper libraries, more on that in a moment).
In practice I've found that it makes sense to divide your application into plain JVM components and Android portions. For instance, if you need to communicate with a REST API you can have a separate component that does only this and which is plain Java. These types of components can easily be tested with standard JUnit. This type of architecture also leads to a clearer separation of responsibility and often and easier to understand and maintain design as well. (Such components can be included in your Android app as regular JARs.)
What additional benefits do android unit tests give over junit tests?
E.g. Handles to the emulator, etc.
Android testing can be slow and painful because a full Android test runs the Android stack in an emulator (or on a device). Android tests are however necessary for testing the parts of your application that are Android specific, such as Context/Activity/Service and so on.
Because of the cumbersome and slow nature of native Android tests several frameworks have been created that mock or stub parts of the SDK and take different approaches to help. You may want to look into Robolectric and Robotium, for instance. (These each have their own pros and cons.)
Can we use JUnit for test driven development in Android ? If not, is there an alternative similar to JUnit ?
I searched a bit on google and also read a SO post Android Test Driven Development
It looks like Android was never made with TDD in mind. I wanted to be sure before I begin learning TDD and doing Android development at the same time.
I think you can completely rely on Robolectric for running your tests in JVM. You get to use JUnit4 for testing your POJOs and Robolectric provides you the support for testing the Android components.
I am also a beginner in TDD for Android Development. I find Robolectric really useful for test driving my code.
This video will tell you almost everything it provides you for unit testing the Android code.
UPDATE:
With the Android studio support and the new Android ecosystem, now unit testing can be done as a first class practice. Refer http://developer.android.com/training/testing/unit-testing/local-unit-tests.html for more details.
There are couple of good approaches to test drive the android code. The most effective ones I have found so far is using MVVM(model-view-viewmodel) or MVP(model-view-presenter) approach, where the business logic as well as presentation logic is decoupled from the view and can be easily be unit tested.
Yeah we can use JUnit for test driven development. To initiate with you can refer to following
link : http://developer.android.com/tools/testing/testing_android.html#JUnit
Following the documentation we can use the junit.framework to have unit testing done.
Here is a bit of explanation of the problem space: http://www.techwell.com/2013/04/applying-test-driven-development-android-development
The conclusion is you should use Robolectric. Unfortunately Robolectric is slow, and if you follow TDD even on a pragmatic level, you will end up with a couple of hundreds of test, that will run for couple of 10 seconds. And that is not you want to do with TDD. TDD test package should run at most in seconds.
My advise is:
Create wrapper classes around Android classes that simply call the Android class.
Write your app logic in pure Java.
Use Junit (or TestNG or whatever pleases you) to test you model/business logic
Use occasionally Robolectric for the wrapper classes (probably you won't have to use)
(You can write integration tests that use multiple classes, Robolectric, etc. but only run it on a separate Continous Integration server eg. hourly)
With this approach your logic will be more portable as well.
I'm new to testing.I've developed a application,now i need to test.I googled about testing for some time,learnt ,what different types of testing are there in general.I wrote few test cases.
Three things,i would like to know,
Is there any different types of testing for android,if yes,can you give me some links which could help me to refer.
How do generally a android user test his apps,Will he uses test frame works or generally write test cases and testing that on real phone to see how they are performing.
Is there any sample test cases written document which will give me some basic idea.
For integration testing I use Robotium. It is a nice convenient layer on top of the build in instrumentation testing. These Tests need to be running in an emulator or on a real device. It is recommended in both cases to have an extra test project (producing an additional APK) that depends on the project under test.
Personally I like to partition my app so I have one or more libraries that do not depend on Android specific classes and can therefore be tested in a regular JVM using JUnit.
There is a third way to test and that is by mocking the android classes and have the tests run in a JVM. I have not yet used it but I hear Roboelectric is a framework that allows for this kind of testing.
Android Monkey tool can be a handy little tool. I find it handy the pseudo random fashion is handy for generating unusual use cases.
http://developer.android.com/guide/developing/tools/monkey.html