Does Robolectric provide any clear benefits compared to Android Test Framework ? I've read the docs regarding both the frameworks but as far as i can see the only clear cut benefit regarding Robolectric is that it runs on JVM rather than the DalvikVM, making it faster than Android framework.
Are there any other major benefits that stand out ?
Update Apr-2015: Gradle build tools and Android Studio now officially support unit testing and prevent android.jar from throwing stub (no real implementation) error. So, yes its possible to run tests on Java VM, when stubs are appropriately mocked. Its a start but is still not comparable with Robolectric's power. There is also a third alternative, Scroll to bottom of this answer.
Now, about Robolectric :
Pros: here are a few points about how it has proved to be useful in unit testing:
You need not run an emulator, So you can test some of the project's non UI parts without requiring an emulator or device. This also applies to running tests on continuous integration/ build servers, no emulator instances need be launched.
With Android Studio, you can quickly run one particular test class, when you are working on the implementation to satisfy the test cases. You can debug as you write code. This is a massive productivity gain.
Can fake almost every android related thing as shadow objects, even SQLite. Plus each shadow object exposes many useful functions which their normal android counterparts don't offer. With shadow counterparts of android Object, you can do internal checking or call special methods.
Really shines when testing multi threaded code like AsyncTasks, Loopers and Handlers etc. You can pause and fast-forward thread Loopers, even main thread. Excellent for Handler based callback testing.
JUnit 4 format supported. Android is still holding onto JUnit 3 last time I checked.
Can be combined with other test tools like Mockito, Espresso etc etc.
Supports mock Activity instance creationRobolectric.buildActivity() and its control via ActivityController. Fragment/View manipulation also works on such mock activity instances.
There are now provided add-on modules that cover multi-dex, v4-support, play services, maps and http client as well. So, its now easy to test code using these library functions as well.
Cons: Where I found it not so good:
Robolectric excels at aiding Unit testing, but does not cover all the functionality a real device or emulator can offer. For example sensors, gps, open-gl etc etc.
You'll need an emulator or real device when doing integration or UI testing, so that Activities and services can interact with full android environment (other apps, like using camera app to get a picture for your app), not a limited one. Here you'll need to use the default test framework as it has functions to test UI as well.
JNI loading seems not to be supported. So code with native dependency can't be tested.
As of now, Robolectric has a hard wired dependency on google maps jar to work. And will download another android.jar from maven. So, project setup may require a bit of a tinkering. Update: as of v3 it seems to pull all dependencies via Gradle without much fuss.
Newer Android tools support coverage and reports generation etc, but only when test are run on a device. So with Robolectric you'll have to create extra Gradle tasks (run Jacoco) to do it for you. Update: Gradle 2.9 + ships with jacoco plugin.
As both gradle and android build tools are shipping out newer build versions at a fast rate, stable Robolectric versions will sometimes start having problems with the changed build tooling. Most typical problems include: sdk version incompatible, manifest not found, build output paths mismatch, resources not loading, build config issues etc. Some issues are also related to bugs in android tools. At times you may even have to write your own custom test runner or apply workarounds till next version fixes those issues. Check out open issues and configure tests accordingly.
Another alternative is simply mock stuff on your own, no frameworks involved. Its the "hard way" but the most customizable way. Its plain JUnit with JMockit:
#RunWith(JMockit.class)
public class OtherTest {
public void testHandlerCallback(#Mocked final FragmentTransaction transaction,
#Mocked final FragmentManager manager,
#Mocked final Activity activity,
#Mocked final LayoutInflater inflater,
#Mocked final ViewGroup parent) {
final List<Fragment> fragments = new ArrayList<>();
new Expectations() {{
activity.getFragmentManager(); result = manager;
manager.beginTransaction(); result = transaction;
transaction.add(withCapture(fragments), anyString);
transaction.commit(); result = new Delegate<Void>() {
public int commit() {
View v = fragments.get(0).onCreateView(inflater,parent,null);
Deencapsulation.invoke(v,"onMeasure",0,0);
return 0;
}
};
}};
}
}
Above is a crude and inline example. You can actually create proper re-usable classes (say FragmentTestHarness) that will take a component (say a Fragment) under test and wrap it in completely isolated environment, preparing it for tests.
To share how i do...
Robolectric
For SQL, activities flow, for those objects that needed context.
JUnit4
for api's java module to make sure data are return correctly.
Espresso
For checking ui display correctly.
When I modified api...I only run jUnit4.
When I modified the data binding between api and UI or Sqlite, then I will only run Robolectric.
When I modified UI I only run Espresso.
Sometimes I will run Robolectric and espresso together, but very rare.
But I will run all before publish to play store.
Because I think there is not real benefit for now. But see how u use it to speed up your product quality and development speed.
Correct me if I am wrong.
You would use Robolectric only in cases you need to mock or fake the Android framework, e.g. if you need a context. When using the Android Test Framework you would have to run Instrumented tests for that which is ultra slow.
If you write your tests for letting them run frequently, e.g. because you follow a tdd approach this is not an option. So in this case Robolectric comes in handy.
So the main benefit about Robolectric is that it is a lot faster than Espresso or Instrumented tests in general.
The downside is that it fakes an Android environment which you should be aware of. To validate real world problems, better use a classic Android Framework approach.
The best is still to write your code in a way that you can unit test it and that you don't need a context or any other Android framework dependencies.
Robolectric is getting integrated into the Android Testing Framework since I/O 2018 - check out more at the official Robolectric page and watch this video from I/O 2018
The major benefit of Robolectric is speed. Unit tests with Robolectric do not require a running emulator or device to run the tests and are thus much much faster.
You might still want to have a suite of integration tests that run against a real device, just a much smaller suite.
Related
I have go through Test Android Apps in Android document. Google introduced AndroidX testing, but I am still very of confused of its implications.
I am using MVVM Android architecture components with ViewModel, LiveData.
Which parts of the Android X Framework do I need to use?
View(Activity.Fragments) --> Espresso?
ViewModel(with LiveData) --> RoboElectric?
Model --> JUnit or RoboElectric?
Why is Google still confusing us with lot of libraries, why can't they move all Local test, Instrumentation test, and everything under the same Framework(Espresso)?
The Google Testing team has made many improvements to their testing framework AndroidX.
Why Google still confusing with lot of libraries, why cant they move all Local test, Instrumentation test everything under same Framework(Espresso)?
With AndroidX Test, they are actually attempting to solve this!
First off, regular "Junit" unit Tests are unchanged. You should test your java code on the JVM the same way you were doing.
But now, with AndroidX, they have made the Robolectric 4.0 API the same as Espresso's so it is now possible to run your Espresso Tests as Robolectric tests. This means when developing locally, you can iterate more quickly because the Instrumentation Tests can run so much faster instead of having to run on emulators. It also makes it a lot easier to learn Robolectric, because you are able to use the same API as espresso.
When you get to the PR or CI phase, you can then submit those same tests that ran on Robolectric JVM to run on your CI on real devices using Espresso.
I suggested checking out Testing Rebooted (with AndroidX Test) (Android Dev Summit '18) for more context.
In this picture, the failing UI and Unit tests would represent the same test but represented as Espresso/Robolectric Instrumentation Test depending on what you want to run and when. For clarity, you would want to run it as Robolectric when you are doing local development especially with TDD. When ready to submit a PR, you can then run the same test on CI with emulator or real device.
So, while AndroidX.test can be confusing, I suggest watching this video to get a bit more of a grasp on Google's intent with the framework.
Also, I agree they can be providing more examples and guidance on using the new Androidx.Test!
Wondering which one is the better choice to write unit test cases for android apps and libraries: Using Robolectric library or sticking with Android Testing framework. I want to run test suite at commandline and want it be independent of need of configuring emulator or letting a device attached with build machine. Does anyone of you run a comparative analysis on both of these or something better? Your experiences will be great help me to decide on the better solution.
I use a tiered system, where I prefer earlier tiers where possible:
Pure unit tests. I try to make as much code as possible fully independent of Android APIs, and then use "pure" unit tests which can run on any JVM. These tests are the fastest, and it helps keep code that has no need to be Android-specific portable.
Robolectric-supported unit tests. Where my code has only small dependencies on Android APIs, that can be satisfied by Robolectric shadows, I test it with Robolectric. There is a little more setup time for Robolectric compared to pure tests, but it's still faster than starting/running on an emulator.
Android framework tests. Where Robolectric doesn't cut it - either because the shadows don't exist, or because I'm heavily using Android APIs (and therefore want to test against the Real Thing) - I write test that run on the emulator/device with the default framework.
The point of the tiers is to keep things as simple as possible, which keeps the full suite faster and helps promote cleaner code.
I worked on both, what i found is :-
1) Robolectric do not support API 19, it's mention in its document -
http://robolectric.org/eclipse-quick-start/. It's a great
disadvantage of it.
2) Robolectric run on JVM not on DVM. So we can not
detect that on that particular time GPS is enable in device or not
etc. We can only pass our pre-decided value for it.
3) Code writing in Robolectric is complex than junit specially for
fragment there are lot of complexity and issues.
4) Robolectric needs external jar and configuration and for junit test
we do not need any external library.
5) Robolectric is faster because it runs on JVM but this have
disadvantage too, we can not see UI on our device, what screen code
is executing.
For Android, i like jUnit test.
Can we use JUnit for test driven development in Android ? If not, is there an alternative similar to JUnit ?
I searched a bit on google and also read a SO post Android Test Driven Development
It looks like Android was never made with TDD in mind. I wanted to be sure before I begin learning TDD and doing Android development at the same time.
I think you can completely rely on Robolectric for running your tests in JVM. You get to use JUnit4 for testing your POJOs and Robolectric provides you the support for testing the Android components.
I am also a beginner in TDD for Android Development. I find Robolectric really useful for test driving my code.
This video will tell you almost everything it provides you for unit testing the Android code.
UPDATE:
With the Android studio support and the new Android ecosystem, now unit testing can be done as a first class practice. Refer http://developer.android.com/training/testing/unit-testing/local-unit-tests.html for more details.
There are couple of good approaches to test drive the android code. The most effective ones I have found so far is using MVVM(model-view-viewmodel) or MVP(model-view-presenter) approach, where the business logic as well as presentation logic is decoupled from the view and can be easily be unit tested.
Yeah we can use JUnit for test driven development. To initiate with you can refer to following
link : http://developer.android.com/tools/testing/testing_android.html#JUnit
Following the documentation we can use the junit.framework to have unit testing done.
Here is a bit of explanation of the problem space: http://www.techwell.com/2013/04/applying-test-driven-development-android-development
The conclusion is you should use Robolectric. Unfortunately Robolectric is slow, and if you follow TDD even on a pragmatic level, you will end up with a couple of hundreds of test, that will run for couple of 10 seconds. And that is not you want to do with TDD. TDD test package should run at most in seconds.
My advise is:
Create wrapper classes around Android classes that simply call the Android class.
Write your app logic in pure Java.
Use Junit (or TestNG or whatever pleases you) to test you model/business logic
Use occasionally Robolectric for the wrapper classes (probably you won't have to use)
(You can write integration tests that use multiple classes, Robolectric, etc. but only run it on a separate Continous Integration server eg. hourly)
With this approach your logic will be more portable as well.
I have considerable experience in making Android applications. For my new project, we have decided to do Test Driven Development (TDD). I have been getting my hands wet on Robotium for User Scenario Testing, and it works fine and looks easy too.
For unit testing, I tried to mock Context using (MockContext Android Class) but I am unable to do so. I went through this blog http://sites.google.com/site/androiddevtesting/ and through this http://sdudzin.blogspot.com/2011/01/easy-unit-testing-for-android.html , which suggests that mocking in Android apps is still very limited and hard, and have suggested to use PowerMock, jMockit, JeasyTest, or Roboelectric (in combination with Mockito and Maven) and even RoboGuice.
I would like to get any suggestions from you guys on which unit testing framework in your opinion is the best for testing Android applications. (particularly testing Android classes, possibly giving mock Contexts and other mocking features so that I can make my test cases as independent as possible). Any suggestions or pointers would be helpful . Thanks
For off-device testing, look at Robolectric
For on-device testing, look at Borachio
Bottom line: it's still very, very difficult to do well. Things are improving (the situation is dramatically better today than it was 6 months ago) but Android is comfortably the most test-hostile environment I've ever written programs for.
To do TDD in Android, I personally use all of the following:
assertj-android: assertions for Android
Mockito: Mocking Framework
Robolectric: Unit testing framework that runs without the need of Android emulator
Robotium: UI tests (Needs emulator or device to run)
Also: Using dependency injection libraries such as Dagger or Roboguice will greatly simplify your unit/integration tests. To run tests on multiple devices, consider using Spoon.
I use ActivityInstrumentationTestCase2 in the case of Activities for TDD (or BDD rather), and write normal unit tests for all logic. This also helps me separate logic from Activities.
Mobile applications by nature are UI centric. Therefore it does not
make sense to mock out the UI, even if it makes the Unit test look
like a Functional Test.
For adding Extras to intents, you can set a custom intent for the test, or do it for all tests by overriding setup.
Mocks give a lot of issues on Android, so I use stubs.
An example is given below. The Activity shows Hello World when you click a button -
public class HelloWorldActivityTest extends
ActivityInstrumentationTestCase2<HelloWorldActivity> {
private HelloWorld activity;
public HelloWorldActivityTest() {
super(HelloWorldActivityTest.class);
}
public void testShouldRenderGreetingOnButtonClick() {
activity = this.getActivity();
Button button = (Button) activity.findViewById(R.id.btn_greet);
TouchUtils.clickView(this, button);
assertEquals("Hello World!",
((TextView) activity.findViewById(android.R.id.greeting_text))
.getText());
}
}
EDIT: Things have changed since I posted the answer. Mockito now has reasonably good support for Android. And for tests, we've moved from ActivityInstrumentationTestCase2 to Robolectric, just in order to tap the sheer speed of JVM in the development phase. The Android Testing Framework is great for Integration and Functional testing, just not for Unit Tests.
To apply TDD for android, Android Testing Codelab will be very helpful to you. Code lab shows a use of the testing tool and how you can apply TDD for android.I tried it and it was very helpful to me.
Bonus: Check Clean Architecture
Android Testing Support Library provides an extensive framework for testing Android apps. This library provides a set of APIs that allow you to quickly build and run test code for your apps, including JUnit 4 and functional user interface (UI) tests. You can run tests created using these APIs from the Android Studio IDE or from the command line.
Read more about:
Espresso
AndroidJUnitRunner
JUnit4 Rules
UI Automator
Thank you :)
we have
https://developer.android.com/training/testing/start/index.html
and can able to test local(Runs on JVM) and instrumental test (Runs on Device or Emulator )
For this we need to add
Android Testing Support Library
The Android SDK includes two tools for functional-level app testing
Monkey and monkeyrunner
I am new to Android test frameworks ,Would like to know the differences between existing test frameworks : Monkey , CTS ,Instrumentation Framework & Robotium ?
Instrumentation is a category of testing, opposite to Unit-testing.
The framework provides hooks for instrumentation testing, but you are going to need an additional third-party framework to really get going.
Robotium is such a framework. It allows you to write "scripts" that run through the user interface, saying "click this", "type that", etc. Well-written it can take you through your usecases and thus provide a good feeling that your app isn't broken. It also allows you to test multiple activities and activities interacting.
Unit-testing in my experience is very hard for Android, especially for the "regular" code dealing with UI, databases, activity state, etc., unless you write your code for testability.
The Android Monkey also uses instrumentation to run through your user interface but it does not follow a script. It does this randomly, with the idea that whatever it does it should not crash your app. By generating 100000's of events it tries to get coverage as high as possible, based on statistics. Other than Robotium, the monkey never leaves your app (that would be dangerous). It's a perfect complement though and it comes nearly for free (the setup is really cheap and there is no maintenance).
CTS is only relevant to the operating system and framework itself.
You'll probably also want to know about mocks?
Observe the testing Pyramid below:
Manual testing - self explanatory
Functional testing - testing a feature
Integration testing - checking the units play nicely
Unit tests - make sure an individual unit works as expected (See SRP)
It suggests how many tests you should have of each level. Below the pyramid are the Android frameworks that you can use at each level.
In Android, the following frameworks are commonly used for each section:
Functional:
Monkey runner "kind of" falls under this section, it basically just bashes around the app to see if any combination of interactions crashes it
Integration:
Instrumentation falls under this category.
Espresso (Made by Google, recommended, uses Hamcrest matchers)
Robotium
Unit:
JUnit4
Mockito, Powermock, other mocking libraries
Matching frameworks like Hamcrest, Fest, AssertJ
Robolectric (provides Android specific methods)