I'll be developing an Android app in Kotlin and I'm trying to figure out how to initialize emulated dependencies. For example, the app will make API calls to a server, obtain the user's location from a location provider, pull down images from a content management system, store data locally in a database and in Android's Shared Preferences, and do math based on the current date/time. Thus there are a lot of external dependencies I want to emulate, including the current date/time so I can verify age calculation, etc.
My goal for testing is just to validate my app's screens using an Android instrumented test. I don't want any dependency on real external systems because testing those systems is the responsibility of the developers of those systems.
While reading Android'd documentation Consider whether to use test doubles, I noticed it offered a good tip: "Tip: Check with the library authors to see if they provide any officially-supported testing infrastructures, such as fakes, that you can reliably depend on." However, the documentation doesn't really explain how to initialize a 3rd party test infrastructure.
Below are what I understand so far about my options, but they all come back to a fundamental thing I don't understand: how does an Android app under test know it should operate in a test mode vs. a production mode?
Mocking such as Mockito or MockK: Mocking seems like a special case of Dependency Injection tailored for testing. The examples I've seen involve testing a class or a method, not a full scale system. The examples typically show how you mock a class and pass it to the class/method under test. But with a full scale system, the test code operates on widgets referenced via Espresso. There is no access to the classes where the logic is. My impression is mocking is for unit testing, not UI testing. But maybe someone can explain how to use mocking for UI testing:
a) Suppose an external dependency is initialized deep in the call stack. If I define a mock in my test code's setup function (e.g. a method annotated with #Before), how do I pass that down to the place in the code that depends on it?
b) I keep reading that mocks don't work in Kotlin because Kotlin defines all classes as final. There seem to be a few work arounds. But does does Google/Android officially recommend one of these (I haven't read it in their documentation).
Dependency Injection such as Dagger 2: If mocking isn't viable for UI testing, then should I use Dependency Injection? From what I understand, it seems Dagger 2 deals with issue 1.a above by defining a top-level Component and a tree of Modules which can provide dependencies at any layer of the stack. For testing, it seems like I would just provide a different Component that emulates the real dependencies.
a) In an Android instrumented test, where do instantiate a Dagger 2 Component designed for testing. How do I make sure that Component is used rather than the Component intended for production?
Prepare before launching test: I can see how I could customize build.gradle to prepare my test environment before my application is launched. For example, I could pass a flag to my app so that when the Application's onCreate() gets called, I can configure my system to prepare emulated dependencies via Dependency Injection, Mocking, or even just a custom implementation. For example, some external dependencies have a test mode where I would need to pass a flag to them so they work in test mode. I'm not clear how that sort of thing reconciles with Dependency Injection or Mock but I guess I could see how I could use those mechanisms as a wrapper to pass the test flag or not. In the following post someone wanted to mock a location provider and to do that they modified their build.gradle file to set things up before the Android test infrastructure started.
How to set Allow Mock Location on Android Device before executing AndroidTest with uiautomator and espresso?
In conclusion, I want to test a Kotlin Android app's UI using Android instrumented test with Espresso but I don't know how to setup the test so that external dependencies use emulation code rather than production code. Should I use mocking, Dependency Injection, or customize via build.gradle? Can someone help me get my thinking on track?
After much searching, I've discovered that the Android ActivityTestRule allows you to defer launching the Activity. This gives the test code time to initialize emulated dependencies as demonstrated in Fast Mocked UI Tests on Android Kotlin.
Related
Is there a way to simulate or mock FingerprintManager.authenticate()? I want to write instrumented tests for my fingerprint authenticator.
I'm fine if there's a solution with a restriction that the tests can be run either on an emulator or a device. I use JUnit 4.12.
I might suggest that you don't mock it, in the spirit of not mocking things you don't own. What I might suggest instead (from the above link):
The prescription implied by "don't mock what you don't own" is to introduce your own shim/wrapper/adapter around it. This effectively cordons off the dependency to a single place in your codebase and contextualizes it in the consistent and easy-to-use style you're trying to promote within your codebase. If there's anything awkward about how one needs to invoke the dependency (maybe a chaining API, multi-step invocation, repetitive default configuration, etc.), it can be swept under the rug into that common adapter.
This feels like an instance where the humble object pattern might be appropriate.
I am wondering if there is an option to test client library which uses Android (mock) and calls backend running on some endpoint?
The use case could be:
Have a library called 'A' that requires android's context (extracts data from it) and Google Services (could be mock). Then call A.sendData(context) that collects data such as advertising id, phone info, ...
So the test lifecycle could be:
A.sendData(context) → waits for server response → assert result.
I've read everywhere that Robolectric can't be used as an integration framework. I've created an Instrumented test with some UI which did the trick BUT running it takes ages and doesn't work properly on remote CI (timeouts, sometimes it passes sometimes it doesn't).
I need to mock 'just' part of Android device (no need for UI) and test if backend returns valid data so probably no need for emulator. OR if the emulator is needed I would like to create a test that uses emulator but there is no need to write UI test.
Is out there anything like this?
It seems to me all you're trying to do is validate a network component sends the appropriate data to a server and handles the appropriate response. For this kind of test, I think Robolectric would suit the needs just fine. It should be able to fill the Context of your app which can then be injected in to your component. All while being run through the JUnit framework which is significantly faster than instrumentation tests.
Though, if all you're trying to do is make sure your network component works. Then you may not actually need the Android framework. You could instead rewrite your class using pure Java. Just strip out the items you need from Context, and inject those in to your classes. Then you can use the JUnit testing framework which runs on the computer instead of the device. It's much faster and doesn't require a phone. This works as long as they're not Android specific components such as Drawable. This will also not work if you require Google Services.
A very tedious method is to use Mockito and PowerMock to mock the methods of your Context and Play Services. You just mock the methods that your Context class is going to use. This can get really complicated though. You can use this alongside Robolectric so you only have to mock what you need to mock.
As far as integrations tests are concerned, there's absolutely no need to write a UI test when writing integration tests. Basically what the integration tests do is create a test Application that runs on the phone. If you don't provide an UI, then you won't use an UI. It'll just show a blank screen while the test runs. You can use InstrumentationRegistery#getContext() to retrieve the context that's used in your application. Then inject the context in to your components that need it.
Though, this can sometimes even be a pain and instrumentation tests take longer even without any UI. But, if you want to make sure it works with an actual Android Context with actual Google Services, this is the best way.
So the basic strategy is to write a whole bunch of small, fast unit tests using Robolectric on JUnit framework which can be run a bunch of times. Then write your comprehensive validation tests using instrumentation framework to make sure everything works together nicely.
Does Robolectric provide any clear benefits compared to Android Test Framework ? I've read the docs regarding both the frameworks but as far as i can see the only clear cut benefit regarding Robolectric is that it runs on JVM rather than the DalvikVM, making it faster than Android framework.
Are there any other major benefits that stand out ?
Update Apr-2015: Gradle build tools and Android Studio now officially support unit testing and prevent android.jar from throwing stub (no real implementation) error. So, yes its possible to run tests on Java VM, when stubs are appropriately mocked. Its a start but is still not comparable with Robolectric's power. There is also a third alternative, Scroll to bottom of this answer.
Now, about Robolectric :
Pros: here are a few points about how it has proved to be useful in unit testing:
You need not run an emulator, So you can test some of the project's non UI parts without requiring an emulator or device. This also applies to running tests on continuous integration/ build servers, no emulator instances need be launched.
With Android Studio, you can quickly run one particular test class, when you are working on the implementation to satisfy the test cases. You can debug as you write code. This is a massive productivity gain.
Can fake almost every android related thing as shadow objects, even SQLite. Plus each shadow object exposes many useful functions which their normal android counterparts don't offer. With shadow counterparts of android Object, you can do internal checking or call special methods.
Really shines when testing multi threaded code like AsyncTasks, Loopers and Handlers etc. You can pause and fast-forward thread Loopers, even main thread. Excellent for Handler based callback testing.
JUnit 4 format supported. Android is still holding onto JUnit 3 last time I checked.
Can be combined with other test tools like Mockito, Espresso etc etc.
Supports mock Activity instance creationRobolectric.buildActivity() and its control via ActivityController. Fragment/View manipulation also works on such mock activity instances.
There are now provided add-on modules that cover multi-dex, v4-support, play services, maps and http client as well. So, its now easy to test code using these library functions as well.
Cons: Where I found it not so good:
Robolectric excels at aiding Unit testing, but does not cover all the functionality a real device or emulator can offer. For example sensors, gps, open-gl etc etc.
You'll need an emulator or real device when doing integration or UI testing, so that Activities and services can interact with full android environment (other apps, like using camera app to get a picture for your app), not a limited one. Here you'll need to use the default test framework as it has functions to test UI as well.
JNI loading seems not to be supported. So code with native dependency can't be tested.
As of now, Robolectric has a hard wired dependency on google maps jar to work. And will download another android.jar from maven. So, project setup may require a bit of a tinkering. Update: as of v3 it seems to pull all dependencies via Gradle without much fuss.
Newer Android tools support coverage and reports generation etc, but only when test are run on a device. So with Robolectric you'll have to create extra Gradle tasks (run Jacoco) to do it for you. Update: Gradle 2.9 + ships with jacoco plugin.
As both gradle and android build tools are shipping out newer build versions at a fast rate, stable Robolectric versions will sometimes start having problems with the changed build tooling. Most typical problems include: sdk version incompatible, manifest not found, build output paths mismatch, resources not loading, build config issues etc. Some issues are also related to bugs in android tools. At times you may even have to write your own custom test runner or apply workarounds till next version fixes those issues. Check out open issues and configure tests accordingly.
Another alternative is simply mock stuff on your own, no frameworks involved. Its the "hard way" but the most customizable way. Its plain JUnit with JMockit:
#RunWith(JMockit.class)
public class OtherTest {
public void testHandlerCallback(#Mocked final FragmentTransaction transaction,
#Mocked final FragmentManager manager,
#Mocked final Activity activity,
#Mocked final LayoutInflater inflater,
#Mocked final ViewGroup parent) {
final List<Fragment> fragments = new ArrayList<>();
new Expectations() {{
activity.getFragmentManager(); result = manager;
manager.beginTransaction(); result = transaction;
transaction.add(withCapture(fragments), anyString);
transaction.commit(); result = new Delegate<Void>() {
public int commit() {
View v = fragments.get(0).onCreateView(inflater,parent,null);
Deencapsulation.invoke(v,"onMeasure",0,0);
return 0;
}
};
}};
}
}
Above is a crude and inline example. You can actually create proper re-usable classes (say FragmentTestHarness) that will take a component (say a Fragment) under test and wrap it in completely isolated environment, preparing it for tests.
To share how i do...
Robolectric
For SQL, activities flow, for those objects that needed context.
JUnit4
for api's java module to make sure data are return correctly.
Espresso
For checking ui display correctly.
When I modified api...I only run jUnit4.
When I modified the data binding between api and UI or Sqlite, then I will only run Robolectric.
When I modified UI I only run Espresso.
Sometimes I will run Robolectric and espresso together, but very rare.
But I will run all before publish to play store.
Because I think there is not real benefit for now. But see how u use it to speed up your product quality and development speed.
Correct me if I am wrong.
You would use Robolectric only in cases you need to mock or fake the Android framework, e.g. if you need a context. When using the Android Test Framework you would have to run Instrumented tests for that which is ultra slow.
If you write your tests for letting them run frequently, e.g. because you follow a tdd approach this is not an option. So in this case Robolectric comes in handy.
So the main benefit about Robolectric is that it is a lot faster than Espresso or Instrumented tests in general.
The downside is that it fakes an Android environment which you should be aware of. To validate real world problems, better use a classic Android Framework approach.
The best is still to write your code in a way that you can unit test it and that you don't need a context or any other Android framework dependencies.
Robolectric is getting integrated into the Android Testing Framework since I/O 2018 - check out more at the official Robolectric page and watch this video from I/O 2018
The major benefit of Robolectric is speed. Unit tests with Robolectric do not require a running emulator or device to run the tests and are thus much much faster.
You might still want to have a suite of integration tests that run against a real device, just a much smaller suite.
Unit testing android application is often more difficult than expected because there are classes that do not have a public constructor (like NotificationManager) and classes that have methods that is not possible to override like Context.getText.
For NotificationManager I used a delegator. I would do the same for the context but that means that in all the classes that use the context (many) I need to use my own context that can't even be derived from Context. Then whenever I need to pass the context to an Android API I need to get the real context from within my wrapper.
Is this correct? Is there another way to do it? What is the rationale of having those methods declared as final?
Is there someone that has really written unit tests for big applications for Android?
EDIT
I found the reason why the getText method is defined as final: it's a template method. So it is enough to override the called non final methods
I also found this that works but it is a bit weird
Question is a little bit vague so I am going to write a long answer.
there are classes that do not have a public constructor (like NotificationManager)
Because there are some system-wide resources/components (for instance NotificationManager) which Android doesn't like application to construct/instantiate on demand, instead, these resources are centralized and managed by system and application should always use system provided API to acquire them.
and classes that have methods that is not possible to override like Context.getText
Generally in Java, method marked as final means they are protected by API and API developer doesn't want the default behaviour to be overridden by application developer. In particular for this method Context.getText(), it actually use Template method pattern, check out herschel's comment below and the source code for more details.
Then the question is how do we properly test our code written by using these android context based resources/components?
The answer given by Android SDK is Test Project (via InstrumentationTestRunner). Generally, when using InstrumentationTestRunner, behind the sense, system will install both test.apk (build from Test Project) and app.apk (build from Application Project), and use test.apk manipulate app.apk for testing. This is happened Regardless of whether you use AndroidTestCase API (referred as JUnit in diagram below) or InstrumentationTestCase API (referred as Instrumentation in diagram below) The corresponding testable android context based resoures/components (i.e. Activity, NotificationManager or TextView) are all acquired from the real running app.apk instance. In case you create/use your own MockContext in Test Project, you can see from the diagram that mock object is injected into the running application by InstrumentationTestRunner.
The drawback of using instrumentation test is about efficiency, it goes into the complete running life cycle (start emulator -> install and start both test.apk and app.apk and fire InstrumentationTestRunner) even though you just need test a single method from a single class (as you mentioned in comment). I agree with Ollie, it is a hostile design.
This is where Robolectric come in and play, quouting from their website: Robolectric is a unit test framework that de-fangs the Android SDK jar so you can test-drive the development of your Android app. Tests run inside the JVM on your workstation in seconds.
From my own experience, I have an application consists of around 10 activities and several Service with some other custom views/widgets doing mainly http communication with remote server, and use both instrumentation test (via Test Project) and Robolectic testing Android Context based code. I can pretty much test almost every single class/public method of my application as long as time is allowed.
Android is pretty hostile to testing in many respects.
You might want to take a look at Roboelectric
http://pivotal.github.com/robolectric/
I assume you've read these:
http://developer.android.com/guide/topics/testing/index.html
http://developer.android.com/resources/tutorials/testing/helloandroid_test.html
The *TestCase classes that Android provides for you should be able to solve the testing issues you are encountering. In most of these instances it will spawn a context for you that you can override if you wish. I would suggest starting there, or asking more specific questions about how you would test one thing or another.
As the title says, I want to know, if possible, how can I create a stub from Bundle to run at JVM (so not Dalvik). Since its part of android.os package, may be that is related to that impossibility.
My point is due to the fact that I want to use a mocking framework, which run at JVM, to test the Activity.onCreate() method.
I've already found this link but it seems they use a stripped custom android.jar since its rather experimental and unproven (yet I could not reproduce even with their jar, maybe I need some sleep).
If there is another alternative to mock testing this, any insights will as always be appreciated.
Thanks.
I've been successfully writing unit tests on the java jvm for classes that interact with android.jar classes. I use Powermock to mock out all android classes (like Bundle).
This link talks about using a version of the android jar that does not throw the "stub" exception on the java jvm. I have not needed to use a different version of the android jar. Using Powermock I've been able to successfully mock out anything I need to. For me this unit testing technique is not experimental.
Having said that - the unit tests for classes that extend activities, fragments, etc. become very much mock heavy. I feel it's good to move as much logic as you can into pojo classes, keeping your android extended classes as thin as possible. You can decide then if you feel it's worth isolation junit testing the android extended classes. It can certainly be done if you wish. You can also consider an integration testing framework like the one provided by Android or Robotium to test your android extended classes and their lifecycles.