What are different points to note down while testing android applications? What are different techniques or test cases used for android?
Good question. Here's some lists of things to consider, with links to tools that can help.
As for implementating tests, you can read some of the tutorials listed and use frameworks like Robotium to simplify the writing of tests.
Test types
Here is a non-exhaustive list of the types of tests that should be relevant for testing an Android application.
Unit tests
Non-Android specific, i.e. business logic
Android unit tests, e.g. testing UI components
Functional tests
Android instrumentation tests
Integration tests
Testing the interaction of app components
System tests
Black-box testing that tests the whole app and its dependencies
Accesibility tests
Do UI components have the correct labels, descriptions and hint texts included?
Are there any potential problems that could affect people, e.g. colour blindness?
Security and reliability tests
Are inputs validated before use, e.g. in a local database, or before sending to a server?
Does the UI reliably handle all events, e.g. config changes, hardware events?
It depends on what exactly your application does, but it should be possible to test much of this automatically using some of the tools listed below.
Variables
Software and hardware features differ between the various Android devices.
You should test taking into account these, in conjunction with the types of devices your target market will be using.
OS version
Screen density
Screen resolution (small, normal phones, large [tablets])
Locale
Important is to make sure you support multiple screen types, particularly making sure you provide the right resources to support devices with different screen densities and physical screen sizes.
In general, you want to include as few graphics as possible, but make use of the various Android Drawable types, which often let you define the graphics you need via XML. Also make good use of layouts and images that automatically scale themselves no matter what device they're being used on.
Doing so will make testing across different devices simpler.
Tools
Writing tests
Integrated JUnit support for unit testing
TestCase classes for testing Android components
Robotium — a library that makes it very simple to write black-box functional tests that can also cross multiple activities
Using mocks
Although Android provides a few mock objects that can be used to fake components for test purposes, many more would be useful.
For this reason (and in general), designing your application with testability in mind is a good idea. For example, don't directly access ConnectivityManager, but instead create an interface that defines the method calls you need. Then write two implementations of that interface: one that wraps the Android ConnectivityManager and another, mock version. Choose the implementation you want to use at runtime, depending on whether you're running unit tests or not.
Running tests
InstrumentationTestRunner — the default utility for running automated Android tests
Android Emulator Plugin for Jenkins — automates the creation and execution of Android emulators with various configurations, so you can test one APK across multiple device types very easily
monkey — sends random commands to your app; a form of fuzzing
Other
Integrated EMMA support for measuring code coverage
Reference
Beyond all the links above, here are a few specific articles available:
Activity Testing Tutorial — Android's quick overview of setting up a test project, writing and running tests
Testing and Instrumentation — Android's overview of the test APIs and troubleshooting tips
Activity Testing — Further information from Android on writing and running Activity tests
Related
I'm taking over an android project and I wish to introduce unit tests to the project, to help avoid possible regressions.
For normal java projects, I have two source folders: src and test. The src source folder contains all of my source files and my test source folder contains all of my unit tests, which I believe is pretty standard for keeping test separate from the source, so you don't have to ship with them.
I've been doing some reading online and the approach with android apps looks to be a little bit different. Several examples talk about setting up a 2nd project for an android test project and then referencing the android project.
I wish to confirm a few things:
Is having a 2nd project for the testing the appropriate thing to do when it comes to testing android projects or am I just finding bad examples?
Should all unit tests be android unit tests? E.g. Yes they should all be, or no I should mix between android unit tests and junit because junits have less overhead.
What additional benefits do android unit tests give over junit tests? E.g. Handles to the emulator, etc.
Is having a 2nd project for the testing the appropriate thing to do
when it comes to testing android projects or am I just finding bad
examples?
Yes, it's typical to have a separate "test project" for testing Android specific code. http://developer.android.com/tools/testing/testing_android.html
Should all unit tests be android unit tests? E.g. Yes they should all
be, or no I should mix between android unit tests and junit because
junits have less overhead.
You'll usually have a mix because you can't test Android specific code on a standard JVM with regular ole' JUnit (not without some helper libraries, more on that in a moment).
In practice I've found that it makes sense to divide your application into plain JVM components and Android portions. For instance, if you need to communicate with a REST API you can have a separate component that does only this and which is plain Java. These types of components can easily be tested with standard JUnit. This type of architecture also leads to a clearer separation of responsibility and often and easier to understand and maintain design as well. (Such components can be included in your Android app as regular JARs.)
What additional benefits do android unit tests give over junit tests?
E.g. Handles to the emulator, etc.
Android testing can be slow and painful because a full Android test runs the Android stack in an emulator (or on a device). Android tests are however necessary for testing the parts of your application that are Android specific, such as Context/Activity/Service and so on.
Because of the cumbersome and slow nature of native Android tests several frameworks have been created that mock or stub parts of the SDK and take different approaches to help. You may want to look into Robolectric and Robotium, for instance. (These each have their own pros and cons.)
Wondering which one is the better choice to write unit test cases for android apps and libraries: Using Robolectric library or sticking with Android Testing framework. I want to run test suite at commandline and want it be independent of need of configuring emulator or letting a device attached with build machine. Does anyone of you run a comparative analysis on both of these or something better? Your experiences will be great help me to decide on the better solution.
I use a tiered system, where I prefer earlier tiers where possible:
Pure unit tests. I try to make as much code as possible fully independent of Android APIs, and then use "pure" unit tests which can run on any JVM. These tests are the fastest, and it helps keep code that has no need to be Android-specific portable.
Robolectric-supported unit tests. Where my code has only small dependencies on Android APIs, that can be satisfied by Robolectric shadows, I test it with Robolectric. There is a little more setup time for Robolectric compared to pure tests, but it's still faster than starting/running on an emulator.
Android framework tests. Where Robolectric doesn't cut it - either because the shadows don't exist, or because I'm heavily using Android APIs (and therefore want to test against the Real Thing) - I write test that run on the emulator/device with the default framework.
The point of the tiers is to keep things as simple as possible, which keeps the full suite faster and helps promote cleaner code.
I worked on both, what i found is :-
1) Robolectric do not support API 19, it's mention in its document -
http://robolectric.org/eclipse-quick-start/. It's a great
disadvantage of it.
2) Robolectric run on JVM not on DVM. So we can not
detect that on that particular time GPS is enable in device or not
etc. We can only pass our pre-decided value for it.
3) Code writing in Robolectric is complex than junit specially for
fragment there are lot of complexity and issues.
4) Robolectric needs external jar and configuration and for junit test
we do not need any external library.
5) Robolectric is faster because it runs on JVM but this have
disadvantage too, we can not see UI on our device, what screen code
is executing.
For Android, i like jUnit test.
I'm new to testing.I've developed a application,now i need to test.I googled about testing for some time,learnt ,what different types of testing are there in general.I wrote few test cases.
Three things,i would like to know,
Is there any different types of testing for android,if yes,can you give me some links which could help me to refer.
How do generally a android user test his apps,Will he uses test frame works or generally write test cases and testing that on real phone to see how they are performing.
Is there any sample test cases written document which will give me some basic idea.
For integration testing I use Robotium. It is a nice convenient layer on top of the build in instrumentation testing. These Tests need to be running in an emulator or on a real device. It is recommended in both cases to have an extra test project (producing an additional APK) that depends on the project under test.
Personally I like to partition my app so I have one or more libraries that do not depend on Android specific classes and can therefore be tested in a regular JVM using JUnit.
There is a third way to test and that is by mocking the android classes and have the tests run in a JVM. I have not yet used it but I hear Roboelectric is a framework that allows for this kind of testing.
Android Monkey tool can be a handy little tool. I find it handy the pseudo random fashion is handy for generating unusual use cases.
http://developer.android.com/guide/developing/tools/monkey.html
For my CI infrastrucure I need to automatically deploy and run complex tests between two (or more) android phones from a controler machine (linux).
So far android monkeyrunner in combination with android unit tests is not a satisfactory solution:
monkeyrunner does not call single methods from a remote app and can't address UI components by their id
writing java tests cases with ActivityInstrumentationTestCase2 is too expensive (loC) and does not provide enough flexibility for interacting between two devices
robotium.org is a step in the right direction but is in java (not dynamic) and not multidevice yet
all other solutions I found does not allow interactive testing thx to dynamic scripting
Any suggestion?
There is a tool called AndroidViewClient which extends the usability of monkeyrunner. You can address UI components by ID with it, check their properties and so on.
Here's the link:
https://github.com/dtmilano/AndroidViewClient
Have you looked at MonkeyTalk?
Here:
http://www.gorillalogic.com/testing-tools/monkeytalk
I am new to Android test frameworks ,Would like to know the differences between existing test frameworks : Monkey , CTS ,Instrumentation Framework & Robotium ?
Instrumentation is a category of testing, opposite to Unit-testing.
The framework provides hooks for instrumentation testing, but you are going to need an additional third-party framework to really get going.
Robotium is such a framework. It allows you to write "scripts" that run through the user interface, saying "click this", "type that", etc. Well-written it can take you through your usecases and thus provide a good feeling that your app isn't broken. It also allows you to test multiple activities and activities interacting.
Unit-testing in my experience is very hard for Android, especially for the "regular" code dealing with UI, databases, activity state, etc., unless you write your code for testability.
The Android Monkey also uses instrumentation to run through your user interface but it does not follow a script. It does this randomly, with the idea that whatever it does it should not crash your app. By generating 100000's of events it tries to get coverage as high as possible, based on statistics. Other than Robotium, the monkey never leaves your app (that would be dangerous). It's a perfect complement though and it comes nearly for free (the setup is really cheap and there is no maintenance).
CTS is only relevant to the operating system and framework itself.
You'll probably also want to know about mocks?
Observe the testing Pyramid below:
Manual testing - self explanatory
Functional testing - testing a feature
Integration testing - checking the units play nicely
Unit tests - make sure an individual unit works as expected (See SRP)
It suggests how many tests you should have of each level. Below the pyramid are the Android frameworks that you can use at each level.
In Android, the following frameworks are commonly used for each section:
Functional:
Monkey runner "kind of" falls under this section, it basically just bashes around the app to see if any combination of interactions crashes it
Integration:
Instrumentation falls under this category.
Espresso (Made by Google, recommended, uses Hamcrest matchers)
Robotium
Unit:
JUnit4
Mockito, Powermock, other mocking libraries
Matching frameworks like Hamcrest, Fest, AssertJ
Robolectric (provides Android specific methods)