I am new to Android test frameworks ,Would like to know the differences between existing test frameworks : Monkey , CTS ,Instrumentation Framework & Robotium ?
Instrumentation is a category of testing, opposite to Unit-testing.
The framework provides hooks for instrumentation testing, but you are going to need an additional third-party framework to really get going.
Robotium is such a framework. It allows you to write "scripts" that run through the user interface, saying "click this", "type that", etc. Well-written it can take you through your usecases and thus provide a good feeling that your app isn't broken. It also allows you to test multiple activities and activities interacting.
Unit-testing in my experience is very hard for Android, especially for the "regular" code dealing with UI, databases, activity state, etc., unless you write your code for testability.
The Android Monkey also uses instrumentation to run through your user interface but it does not follow a script. It does this randomly, with the idea that whatever it does it should not crash your app. By generating 100000's of events it tries to get coverage as high as possible, based on statistics. Other than Robotium, the monkey never leaves your app (that would be dangerous). It's a perfect complement though and it comes nearly for free (the setup is really cheap and there is no maintenance).
CTS is only relevant to the operating system and framework itself.
You'll probably also want to know about mocks?
Observe the testing Pyramid below:
Manual testing - self explanatory
Functional testing - testing a feature
Integration testing - checking the units play nicely
Unit tests - make sure an individual unit works as expected (See SRP)
It suggests how many tests you should have of each level. Below the pyramid are the Android frameworks that you can use at each level.
In Android, the following frameworks are commonly used for each section:
Functional:
Monkey runner "kind of" falls under this section, it basically just bashes around the app to see if any combination of interactions crashes it
Integration:
Instrumentation falls under this category.
Espresso (Made by Google, recommended, uses Hamcrest matchers)
Robotium
Unit:
JUnit4
Mockito, Powermock, other mocking libraries
Matching frameworks like Hamcrest, Fest, AssertJ
Robolectric (provides Android specific methods)
Related
Can Android acceptance tests be written using Robolectric? It seems to be classed only as a unit-testing framework.
Why can it not be classed as acceptance or "end-to-end" testing framework? (Can it be adapted for that purpose?)
I think this may be a bit subjective, but I feel it's answerable, so here's my take on it.
Is it an e2e testing framework?
It cannot be "end-to-end" because the application doesn't actually run on a device, the dependencies may be mocked out, network is usually mocked out and the tests don't simulate exactly how a user would work with the app. The lifecycle is simulated and controlled by you in the tests. It doesn't even run on the dalvik-VM and there are difference between dalvik and the JVM (here's a simple but relevant example: British Summer Time - BST not recognised by SimpleDateFormat timezone)
For me, an end-to-end test can only be achieved with a deployed application, using the network, connecting to the server it needs to, getting the information, showing it correctly on screen and the test verifying what can be seen on screen. Again, this may be subjective, but that's what e2e means to me.
Can it be used to write acceptance tests?
That depends on what you want from your acceptance tests, as I've seen people use this term differently, but I would say the answer is a definite yes. We use robolectric to test most of our business logic as we have found UI tests to not be reliable.
You can instantiate your UI elements with the right data, interact with UI elements, check what they show on screen. I would say this covers a lot of ground.
That said, sometimes these tests were very difficult to write, and we got a lot out of architecting our code differently (Model-View-Presenter is most commonly suggested for that)
The key for us was to have pretty "dumb" views and to let the presenter determine what will be shown on screen. Then you test that presenter heavily with robolectric tests and you can definitely write tests of the type "given this series of inputs, I expect to see these outputs on screen". Testing the view further adds little value because it's just a shell for what your presenter serves it.
I think describing how to apply MVP to your project is outside the scope of a SO answer, but I hope this helps.
I am using AndroidAnnotations(2.5) in a sample project I am currently working on.
Since you can annotate your classes with for example #EActivity,#ViewById,#Click which will all lead to generated compile-time code, I was wondering how one would go about creating unit tests / functional tests for any Android Annotations powered application.
I would love to hear some opinions on this matter.
Cheers,
I responded to a similar post here.
There are a couple of options available to you. You can, of course, test your code pre-generation in, what I claim, is a more unit testing style. This should test the Java code in isolation, preferably without generated code involved.
You can also test the code post-generation. The MyActivity_ classes generated by AA can be instantiated directly after compile time and test them accordingly. I claim this is edging towards an integration testing style.
While, I think it is always better to test than not to test, I think for integration tests, you should test on the hardware in a situation similar to production. This gives you a total picture of how your application will behave in a real world situation. Thus, for integration tests I prefer high level "is everything working well together" tests.
Robolectric and Robotium can help greatly in these two efforts. Robolectric allows you to instantiate you Activities in a unit test, while Robotium allows you to test selenium style directly on a device.
To recap, I prefer to heavily unit test my code without generation then do some light integration testing to make sure everything is working well together.
I'm new to testing.I've developed a application,now i need to test.I googled about testing for some time,learnt ,what different types of testing are there in general.I wrote few test cases.
Three things,i would like to know,
Is there any different types of testing for android,if yes,can you give me some links which could help me to refer.
How do generally a android user test his apps,Will he uses test frame works or generally write test cases and testing that on real phone to see how they are performing.
Is there any sample test cases written document which will give me some basic idea.
For integration testing I use Robotium. It is a nice convenient layer on top of the build in instrumentation testing. These Tests need to be running in an emulator or on a real device. It is recommended in both cases to have an extra test project (producing an additional APK) that depends on the project under test.
Personally I like to partition my app so I have one or more libraries that do not depend on Android specific classes and can therefore be tested in a regular JVM using JUnit.
There is a third way to test and that is by mocking the android classes and have the tests run in a JVM. I have not yet used it but I hear Roboelectric is a framework that allows for this kind of testing.
Android Monkey tool can be a handy little tool. I find it handy the pseudo random fashion is handy for generating unusual use cases.
http://developer.android.com/guide/developing/tools/monkey.html
I have been developing Android application for a small company and during the development process we need to do repetitive testing of some modules, So i searched tools for doing automation testing (unit testing) of the app. Android has a unit test tool however to write those test cases will itself take more time then to actually test it by hand.
I found some apps which do some great stuff and provide good charts for example Robolectric, robotium, fonemonkey4android, but am confused to what to be used, any one with any experience with the same can help.
I checked for previous questions on the similar terms like below
https://stackoverflow.com/questions/522312/best-practices-for-unit-testing-android-apps
But all the threads are very old and not so informative to decide on which to choose..
I think first you need figure out which part of your code you want to test.
For codes which doesn't related to user interface, you can test them with Robolectric. With Robolectric, the unit test code is the same to those written for java application. But it's not suitable for test ui components.
If you want to test ui, then you can choose robotium. But i always doubt whether it's worth writing tests for ui, they change too often..
What are different points to note down while testing android applications? What are different techniques or test cases used for android?
Good question. Here's some lists of things to consider, with links to tools that can help.
As for implementating tests, you can read some of the tutorials listed and use frameworks like Robotium to simplify the writing of tests.
Test types
Here is a non-exhaustive list of the types of tests that should be relevant for testing an Android application.
Unit tests
Non-Android specific, i.e. business logic
Android unit tests, e.g. testing UI components
Functional tests
Android instrumentation tests
Integration tests
Testing the interaction of app components
System tests
Black-box testing that tests the whole app and its dependencies
Accesibility tests
Do UI components have the correct labels, descriptions and hint texts included?
Are there any potential problems that could affect people, e.g. colour blindness?
Security and reliability tests
Are inputs validated before use, e.g. in a local database, or before sending to a server?
Does the UI reliably handle all events, e.g. config changes, hardware events?
It depends on what exactly your application does, but it should be possible to test much of this automatically using some of the tools listed below.
Variables
Software and hardware features differ between the various Android devices.
You should test taking into account these, in conjunction with the types of devices your target market will be using.
OS version
Screen density
Screen resolution (small, normal phones, large [tablets])
Locale
Important is to make sure you support multiple screen types, particularly making sure you provide the right resources to support devices with different screen densities and physical screen sizes.
In general, you want to include as few graphics as possible, but make use of the various Android Drawable types, which often let you define the graphics you need via XML. Also make good use of layouts and images that automatically scale themselves no matter what device they're being used on.
Doing so will make testing across different devices simpler.
Tools
Writing tests
Integrated JUnit support for unit testing
TestCase classes for testing Android components
Robotium — a library that makes it very simple to write black-box functional tests that can also cross multiple activities
Using mocks
Although Android provides a few mock objects that can be used to fake components for test purposes, many more would be useful.
For this reason (and in general), designing your application with testability in mind is a good idea. For example, don't directly access ConnectivityManager, but instead create an interface that defines the method calls you need. Then write two implementations of that interface: one that wraps the Android ConnectivityManager and another, mock version. Choose the implementation you want to use at runtime, depending on whether you're running unit tests or not.
Running tests
InstrumentationTestRunner — the default utility for running automated Android tests
Android Emulator Plugin for Jenkins — automates the creation and execution of Android emulators with various configurations, so you can test one APK across multiple device types very easily
monkey — sends random commands to your app; a form of fuzzing
Other
Integrated EMMA support for measuring code coverage
Reference
Beyond all the links above, here are a few specific articles available:
Activity Testing Tutorial — Android's quick overview of setting up a test project, writing and running tests
Testing and Instrumentation — Android's overview of the test APIs and troubleshooting tips
Activity Testing — Further information from Android on writing and running Activity tests