Can Android acceptance tests be written using Robolectric? It seems to be classed only as a unit-testing framework.
Why can it not be classed as acceptance or "end-to-end" testing framework? (Can it be adapted for that purpose?)
I think this may be a bit subjective, but I feel it's answerable, so here's my take on it.
Is it an e2e testing framework?
It cannot be "end-to-end" because the application doesn't actually run on a device, the dependencies may be mocked out, network is usually mocked out and the tests don't simulate exactly how a user would work with the app. The lifecycle is simulated and controlled by you in the tests. It doesn't even run on the dalvik-VM and there are difference between dalvik and the JVM (here's a simple but relevant example: British Summer Time - BST not recognised by SimpleDateFormat timezone)
For me, an end-to-end test can only be achieved with a deployed application, using the network, connecting to the server it needs to, getting the information, showing it correctly on screen and the test verifying what can be seen on screen. Again, this may be subjective, but that's what e2e means to me.
Can it be used to write acceptance tests?
That depends on what you want from your acceptance tests, as I've seen people use this term differently, but I would say the answer is a definite yes. We use robolectric to test most of our business logic as we have found UI tests to not be reliable.
You can instantiate your UI elements with the right data, interact with UI elements, check what they show on screen. I would say this covers a lot of ground.
That said, sometimes these tests were very difficult to write, and we got a lot out of architecting our code differently (Model-View-Presenter is most commonly suggested for that)
The key for us was to have pretty "dumb" views and to let the presenter determine what will be shown on screen. Then you test that presenter heavily with robolectric tests and you can definitely write tests of the type "given this series of inputs, I expect to see these outputs on screen". Testing the view further adds little value because it's just a shell for what your presenter serves it.
I think describing how to apply MVP to your project is outside the scope of a SO answer, but I hope this helps.
Related
Our team is considering starting testing based on user scenarios. So, we are picking a E2E framework.
Searching for UI test lead to the following:
So, I found this this
It says
UI testing: user interface testing. In other words, you have to make sure that all buttons, fields, labels and other elements on the screen work as assumed in a specification.
GUI testing: graphical user interface. You have to make sure that all elements on the screen work as mentioned in a specification and also color, font, element size and other similar stuff match design.
Functional testing: the process of quality assurance of a product that assumes the testing of the functions/functionalities of component or system in general, according to specification requirements.
E2E testing: it needs for identifying system dependencies and ensuring that the right information is passed through multiple components and systems.
I don't get the difference between UI Testing & E2E Testing.
I wrote UI Test Code in Android Studio. And I need to write a code for each and every click and view, etc. I feel why do we need this? I'd rather test with my finger directly and dynamically.
Let's start by looking at what the difference is between E2E (end to end) testing and UI testing:
End to end testing is checking the whole system / product behaves
correctly when used in the way it will be deployed.
UI testing ensuring that the UI works correctly.
Essentially UI testing is focusing on the UI component of the product. It would be entirely appropriate to do this testing with a mocked backend to avoid needing to run the entire system to check the UI.
There is an amount of overlap between UI testing and E2E testing. I.E. if you test a form in UI testing I would also expect that form to be tested in an E2E scenario. The main difference would be the coverage, the E2E test would try to cover the scenario which may be one use of the form. Where as, the UI testing would cover all the things that the user can do with the form, including entering bad data.
One of the problems with E2E testing is often when a test fails you need to spend some time working out which component has caused the failure. By having tests for each component (including UI testing) there should be a corresponding failure in the tests for one of the components.
Imagine that your E2E test has failed because the login button has disappeared from the UI. Your E2E test will say user cannot log in. Is that because of the UI, the API, the credential store (e.g. database), connection to external service when using SSO (e.g. LDAP)? When looking at the UI tests it will also say user cannot log in, but now you know it's a UI problem.
I wrote UI Test Code in Android Studio. And I need to write a code for each and every click and view, etc. I feel why do we need this? I'd rather test with my finger directly and dynamically.
As with all testing the depth you go into with each type of testing is a decision based heavily on your needs. There are many cases where the manual testing you describe is entirely adequate, typically because the system is not safety critical and the cost to fix issues is low. However, if the cost of fixing issues is high, or the system is safety critical you might wish to invest in writing tests for every click or view could be useful.
If there are any points you wish me to expand on please leave comment.
I've in the past year written an Android app without doing any automated testing at all. I'm now learning about testing in Android but I still have some questions I can't seem to find answers to online.
I have unit-tested my presenters and since they don't use any Android-classes it has been pretty straight forward to write these tests.
I have also written some UI tests for my view (Fragment) using Espresso, which I also feel like I have a good grip on.
What I don't really understand is if I should write unit tests for all the methods in my Fragments as well? The UI tests check all the possible user interactions that can be done. But my Fragments still have some basic methods that are called from the presenter after it has fetched some data from model. These methods in my Fragment are really simple, they take data from presenter and use it to fill a view or toggle visibility or something similar. Should I write unit tests for these types of methods? They seem really simple but I want to make sure I understand Android testing completely.
Thanks!
UI tests are always very brittle creatures. I once was called by my boss to check why the ui tests were failing. In the end it was because the testdevice was not connected by usb to the testserver anymore.
Also ui tests take a lot of time to run. So you have to think about what you want to achieve? When do you run your tests? Where do you want to run the tests
Where? Either use some kind of device lab, or if money is tight run tests on headless emulators that you create every time
What? Think about the big picture. Features You have already covered the logic of your presenters in your unit tests, now try to cover the flows in the view
When? If you have a big test suite, it may take hours to run. In some companies even days. Schedule the whole test suite as a nightly build. But make sure to add a reduced set of tests, the absolute core of your product to run after every commit
Unit tests are generally pointless on trivial methods. If you have a method that just unconditionally sets a visible flag on an item, what's there to really test?
Unit testing your models and presenters should cover most potential bugs. I wouldn't worry much about your views, especially since you're already doing UI tests. You might want to write tests to cover bugs that come up more than once, but apart from that your time is better spent elsewhere.
Your question is about test coverage. If you want to cover more of your codes, you should write tests for every method even fragment methods using mocking tools.
Suppose you are tasked with adding a testing framework to an existing code base that has very very little unit testing coverage. The code base isn't insanely large yet, however, it does have areas where it's not super clean, or not very OOP or testable.
I've read a couple of good answers:
Adding unit tests to legacy code
How to approach unit testing in a large project
Best Option for Retrospective application of TDD into C# codebase
But the project I'm working on is an Android app, so it's slightly different (given lots more UI components).
I have some questions that all relate to the same issue:
What's the best way to do go back and put a bunch of tests in place?
How do I prioritize which parts to test first?
do I start with areas where it's getting called a lot (and is there a tool for code analysis like this)?
or do I go back and look at classes which in the past have had the most number of bugs?
Should I write integration tests first (given this is an Android app and also integration tests may prevent broken unit tests/refactorings) and then work in the unit tests?
Sorry for all the questions, just really looking for a good approach, more specifically geared towards retrospectively testing an Android app.
P.S. I'm curious if people know of good tools they can use for code analysis that can help bring to attention the areas in a code base in which unit testing would be most helpful.
Usually when starting with unit testing on an existing application you would want to do it iteratively - you probably do not have the time or the man power for a big upfront investment and so you want to add unit tests as part of the required work:
Write unt tests When adding a new feature - or better yet use TDD
When fixing a bug - write unit test(s) that fails de to the bug before fixing it
When refactoring old code wriute unit tests to make sure no regression bus were introduced
Id the whole team follow the three steps above in a matter of weeks you should have good coverage for the code you've been changing - depending on the size of the project.
I don't have an Android-specific answer. I hope someone comes along and adds a good one. Meanwhile: write acceptance (integration) tests for your most important features (from your product owner's point of view, not from some tool's). You'll get the most coverage for your effort, and be the most likely to prevent bugs that customers care about. Don't worry so much about unit tests; those are for details.
[That's the answer given in the third post you cite, but we're still waiting for that Android-savvy answer so let's not call this a duplicate yet.]
Do write acceptance tests and, as necessary, unit tests for new features, and write unit tests when fixing bugs.
Regarding how to find areas of the code that need tests, the first thing to do is measure code coverage. (You can do that in the Android SDK with ant emma debug install test.) Obviously code with no coverage needs tests.
My apps are mostly GUIs that communicate to a server for most of their information. If anything goes wrong it will usually be in the network call or making a wrong assumption about a JSON object.
Unit Tests are not good for these network-related and i/o related tasks, otherwise, they won't be called unit tests.
SO I am trying to gather the point of Unit Tests in my case. Why would I test if an Android button can click or an EditText can see what I type? I just don't understand the utility of implementing these tedious tests
private void initElements(){
placeButton = (Button) findViewById(R.id.currplace);
placeButton.setText(MainActivity.this.getString(R.string.findingLocation));
placeButton.setEnabled(false);
selectplaceLayout = (LinearLayout)findViewById(R.id.selectplaceLayout);
selectplaceLayout.setVisibility(View.GONE);
splash = (RelativeLayout)findViewById(R.id.splashbg);
infoLayout = (LinearLayout)findViewById(R.id.infoLayout);
}
if this above method passed, which all my activities run in onCreate, then I know the app works. A unit test of this would be a redundant time-consuming thing to create. Time-consuming because I am not familiar with all the methods in the jUnit and Android testing framework.
So, long story short, what's the point? Is there a particular way I should be thinking about these tests? All examples and tutorials I've seen so far only talk about the simplest examples, for the sake of brevity, but I cannot think of any practical uses for unit tests in a predominately client-server app.
What am I expected to discover by accessing the android views that I already know I declared and initialized? I must be thinking about this in a too limited way
so, insight appreciated
There are lots of facets in your question, but to my opinion - you probably don't need unit-tests in your project.
Unit tests really shine when you need lots of business logic to your project. In this case you probably want to divide your application into multiple layers (say, 3-tier architecture) to, among other, add some natural isolation for business-logic layer and cover it with safety net of unit tests.
This safety net covers your ass during refactor of the business layer, and that's one of the main things you what from unit tests (TDD can offer some nice extra side effects though).
However, it's not all unicorns and rainbows and unit-test may cost, and sometimes they cost a lot. Good unit tests are isolated (i.e. deal with small chunks of code). This means that you have to add layers of abstraction in order to put your classes under the test.
This may have positive affect on your system or negative one. Layering makes your system more flexible with cost of increased complexity.
Having that said - the value of the unit-tests is proportional to the amount of abstract business-logic you are going to introduce in your project. You may think of it also this way - if it is overkill to add abstract layers to your architecture - don't add unit-tests - they will only make things more complicated (architecture and build wise).
Based on your description - your typical app tend to be pretty much presentation layer for some external server-side. It does not that much except presenting information on android handset and transforms user actions to commands to server-side where main business logic gets done (controlling).
With this approach most of the code you probably write is related to "how to display this and that" or "how to signal server in this and that case". This sort of code is obviously heavily depend on platform and this mean that if you do want to put it under test you'll have to mock lots and lots of Android specific code\behavior.
Now, Android is somewhat specific platform. It was designed to be both performance optimized and allow developers to start and produce apps quickly. Often this means some amount of "swiss-knife" classes you have use\extend and generally this speeds up writing code, but mocking those classes can become a real hell. Not to mention that you have to have understanding of how platform works under the hood to make those mock useful. In other words overhead from making those test is going to be high.
Another thing that is wrong with testing presentation layers is that they tend to change much more dynamically than business layers. And, of course this mean that you'll have to refactor thee tests which adds even more overhead.
I have to say one thing about various utility/helper classes though. Even if these classes are belong to presentation layer and do depend Android code, but do some rather non-trivial logic and it is easy to mock and write unit tests for them, it might actually be a good idea to do this. However, if you do have lots of such code - this is might be a signal that you haven't designed your architecture/layering well, and need to rethink what you are doing.
In the end to answer your question you have to answer these questions first:
Will it be overdesign to add abstract layer that is separated from the platform to your application (seems like in your case it will)? If yes - do not use unit-tests - they will only slow you down. If no - do use them.
Are you going to refactor a lot? If that's large project with lots of code and thus maintenance - you probably will, so invest in layering and unit-tests (but, at a glance, it does not seem that this is your case). If that is not your case - do not bother with unit-tests and go fast.
Do you need to mock platform a lot to write your unit tests? If yes (seems to be your case) - don't write unit tests - they do not worth the effort.
Hope this will help.
UTesting in Android generally takes relevance when you are using some layering with a good architecture. Nowadays the most popular is Clean Arch, which is all about maintainability and testability. Every part of the architecture has exactly one purpose. We just need to specify it and check that it actually does its job every time.
When testing use cases we should test that use case calls correct methods in repositories or executes other use cases. We should also test that use case returns proper callback.
When testing repositories, you should arrange DAOs – make them return or receive some dummy data, and check that repository is handling the data in a proper way.
When testing mappers (converters), specify input to the mapper, and exact output you expect from the mapper, then assert they are equal. Do the same for services, parsers etc.
Leaving the architecture besides any good modular and decoupled design will contain business logic in functions and classes, and if those follow single responsibility principle (they should), then likely should be tested.
Think about testing before and during coding. That way you can write testable and decoupled code. Use your tests as class specification, and if possible write them before the code.
Examples and expanded info: https://five.agency/android-architecture-part-5-test-clean-architecture/
from Android.Docs
For testing Android apps, you typically create these types of automated unit tests:
Local tests: Unit tests that run on your local machine only. These tests are compiled to run locally on the Java Virtual Machine (JVM) to minimize execution time. Use this approach to run unit tests that have no dependencies on the Android framework or have dependencies that can be filled by using mock objects.
Instrumented tests: Unit tests that run on an Android device or emulator. These tests have access to instrumentation information, such as the Context for the app under test. Use this approach to run unit tests that have Android dependencies which cannot be easily filled by using mock objects.
Note: Unit tests are not suitable for testing complex UI interaction events. Instead, you should use the UI testing frameworks, as described in Automating UI Tests.
I am new to Android test frameworks ,Would like to know the differences between existing test frameworks : Monkey , CTS ,Instrumentation Framework & Robotium ?
Instrumentation is a category of testing, opposite to Unit-testing.
The framework provides hooks for instrumentation testing, but you are going to need an additional third-party framework to really get going.
Robotium is such a framework. It allows you to write "scripts" that run through the user interface, saying "click this", "type that", etc. Well-written it can take you through your usecases and thus provide a good feeling that your app isn't broken. It also allows you to test multiple activities and activities interacting.
Unit-testing in my experience is very hard for Android, especially for the "regular" code dealing with UI, databases, activity state, etc., unless you write your code for testability.
The Android Monkey also uses instrumentation to run through your user interface but it does not follow a script. It does this randomly, with the idea that whatever it does it should not crash your app. By generating 100000's of events it tries to get coverage as high as possible, based on statistics. Other than Robotium, the monkey never leaves your app (that would be dangerous). It's a perfect complement though and it comes nearly for free (the setup is really cheap and there is no maintenance).
CTS is only relevant to the operating system and framework itself.
You'll probably also want to know about mocks?
Observe the testing Pyramid below:
Manual testing - self explanatory
Functional testing - testing a feature
Integration testing - checking the units play nicely
Unit tests - make sure an individual unit works as expected (See SRP)
It suggests how many tests you should have of each level. Below the pyramid are the Android frameworks that you can use at each level.
In Android, the following frameworks are commonly used for each section:
Functional:
Monkey runner "kind of" falls under this section, it basically just bashes around the app to see if any combination of interactions crashes it
Integration:
Instrumentation falls under this category.
Espresso (Made by Google, recommended, uses Hamcrest matchers)
Robotium
Unit:
JUnit4
Mockito, Powermock, other mocking libraries
Matching frameworks like Hamcrest, Fest, AssertJ
Robolectric (provides Android specific methods)