Develop Reference

Gradle task to only execute uiTests

I've got this Android, with the default Gradle-tasks. In the Android-project, there is a androidTest package, containing integrationTests and uiTests. I have also two Kotlin classes containing a suite of test classes to be called.
However, ./gradlew connectedAndroidTest runs both integrationTests and uiTests, I want to separate this. I came up with multiple solutions:
Android Studio's run configurations. However, this isn't checked into VCS, therefore we have no access on our Jenkins build server.
Add Gradle tasks in the Groovy language, however, it's not encouraged to call another gradle task in a new task.
So I'm looking for a way to only test either integrationTests or uiTests. How can I do this?

I'm going to give you a cheap and cheerful answer now. Maybe someone else will be able to provide a fuller one.
Since all the tests appear to be part of the same source set, you need to distinguish between them some other way. The most appropriate solution is whatever mechanism your testing library has for grouping, which you can then utilise from Gradle.
Alternatively, use something like a naming convention to distinguish between UI tests and integration tests.
What you do then depends on how you want the build to deal with these different categories. The main options include:
Using test filtering from the command line — via the --tests option — to run just integration or UI tests. Note that filtering only works via class name, so you'd have to go with the naming convention approach.
Configure the appropriate Test task — is that connectedAndroidTest? — so that if you set a particular project property it will run either the integration tests or the UI tests based on the value of that project property. This involves using an if condition in the configuration. This approach works with both filtering and grouping.
Add two extra Test tasks, one that executes the integration tests and one that executes the UI tests. You would leave connectedAndroidTest untouched. This is my preferred approach, but requires a bit more code than the others.
This answer is missing a lot of detail on how to implement those solutions, but I'm afraid filling that detail out is too time-consuming for me right now. As I said, hopefully someone will come along with a fuller answer.

TeamCity trigger for Github comment on pull request

I have my CI on TeamCity. It currently triggers builds on different events (Pull Request creations, merges into Develop branch, etc.). However, I was wondering if it is possible to trigger a specific build as a result of writing a specific comment/labelling a Pull Request.
The goal is to run a set of Automated UI tests when a Pull Request has been approved (from a coding correctness point of view) and that branch is ready to be merged into Develop. I don't want to run that set of Automated UI tests for every commit on that branch as it takes around 1h to run them and I don't want to run it only after merging that PR so we avoid merging anything that breaks the UI tests into Develop.
The desired flow would be to write a special comment on that PR such as run_UI_test or labelling the PR with a custom label so the tests are executed on CI and the feedback is displayed on the PR on Github.
Thank you very much in advance.

I don't believe TeamCity has any awareness of the comments on Github since the comments themselves aren't stored directly in the branch(es). My assumption is that you have a VCS Root similar to this:
+:refs/heads/master
+:refs/heads/develop
+:...
+:refs/pull/*/head
It is possible to access the comments of a pull request through the GitHub "Issues API", however, I think this would add unnecessary complexity to your build process and it would obscure how the builds are getting triggered.
My suggestion would be to follow a more tradition CI process and create another layer of "integration" through a new branch. So basically your flow would look like this:
master (merge pull requests for release)
integration (Run UI automation test)
develop (Run basic unit tests and other stuff)
So basically all of your development happens on develop or some other "feature" branch. When all basic tests are passing and you are ready to "promote" you could merge into the integration branch which will then trigger your UI tests. I also want to point out that this "integration" branch could in fact just be "pull requests" and no static branch is actually required depending on how you have your development flow setup.
it's much easier to setup trigger rules and branch filters in TC then it would be to write some custom REST API script to work with the GitHub Issues API.

You can achieve similar functionality by setting up a separate build configuration which utilizes a specific VCS Trigger to run the UI tests. This build configuration would also have different build steps which would include the commands to execute your tests.
For example: In Triggers, you can add a new VCS Trigger with +:comment=presubmit:** This would look for any commit message that contains "presubmit" and would trigger your UI test suite to run.
I have seen a few repositories use tools such as https://danger.systems to write custom rules which can look for the text in Github comments and trigger interactions based on the comment.

How to retroactively add tests to a code base?

Suppose you are tasked with adding a testing framework to an existing code base that has very very little unit testing coverage. The code base isn't insanely large yet, however, it does have areas where it's not super clean, or not very OOP or testable.
I've read a couple of good answers:
Adding unit tests to legacy code
How to approach unit testing in a large project
Best Option for Retrospective application of TDD into C# codebase
But the project I'm working on is an Android app, so it's slightly different (given lots more UI components).
I have some questions that all relate to the same issue:
What's the best way to do go back and put a bunch of tests in place?
How do I prioritize which parts to test first?
do I start with areas where it's getting called a lot (and is there a tool for code analysis like this)?
or do I go back and look at classes which in the past have had the most number of bugs?
Should I write integration tests first (given this is an Android app and also integration tests may prevent broken unit tests/refactorings) and then work in the unit tests?
Sorry for all the questions, just really looking for a good approach, more specifically geared towards retrospectively testing an Android app.
P.S. I'm curious if people know of good tools they can use for code analysis that can help bring to attention the areas in a code base in which unit testing would be most helpful.

Usually when starting with unit testing on an existing application you would want to do it iteratively - you probably do not have the time or the man power for a big upfront investment and so you want to add unit tests as part of the required work:
Write unt tests When adding a new feature - or better yet use TDD
When fixing a bug - write unit test(s) that fails de to the bug before fixing it
When refactoring old code wriute unit tests to make sure no regression bus were introduced
Id the whole team follow the three steps above in a matter of weeks you should have good coverage for the code you've been changing - depending on the size of the project.

I don't have an Android-specific answer. I hope someone comes along and adds a good one. Meanwhile: write acceptance (integration) tests for your most important features (from your product owner's point of view, not from some tool's). You'll get the most coverage for your effort, and be the most likely to prevent bugs that customers care about. Don't worry so much about unit tests; those are for details.
[That's the answer given in the third post you cite, but we're still waiting for that Android-savvy answer so let's not call this a duplicate yet.]
Do write acceptance tests and, as necessary, unit tests for new features, and write unit tests when fixing bugs.
Regarding how to find areas of the code that need tests, the first thing to do is measure code coverage. (You can do that in the Android SDK with ant emma debug install test.) Obviously code with no coverage needs tests.

Robolectric vs Android Test Framework

Does Robolectric provide any clear benefits compared to Android Test Framework ? I've read the docs regarding both the frameworks but as far as i can see the only clear cut benefit regarding Robolectric is that it runs on JVM rather than the DalvikVM, making it faster than Android framework.
Are there any other major benefits that stand out ?

Update Apr-2015: Gradle build tools and Android Studio now officially support unit testing and prevent android.jar from throwing stub (no real implementation) error. So, yes its possible to run tests on Java VM, when stubs are appropriately mocked. Its a start but is still not comparable with Robolectric's power. There is also a third alternative, Scroll to bottom of this answer.
Now, about Robolectric :
Pros: here are a few points about how it has proved to be useful in unit testing:
You need not run an emulator, So you can test some of the project's non UI parts without requiring an emulator or device. This also applies to running tests on continuous integration/ build servers, no emulator instances need be launched.
With Android Studio, you can quickly run one particular test class, when you are working on the implementation to satisfy the test cases. You can debug as you write code. This is a massive productivity gain.
Can fake almost every android related thing as shadow objects, even SQLite. Plus each shadow object exposes many useful functions which their normal android counterparts don't offer. With shadow counterparts of android Object, you can do internal checking or call special methods.
Really shines when testing multi threaded code like AsyncTasks, Loopers and Handlers etc. You can pause and fast-forward thread Loopers, even main thread. Excellent for Handler based callback testing.
JUnit 4 format supported. Android is still holding onto JUnit 3 last time I checked.
Can be combined with other test tools like Mockito, Espresso etc etc.
Supports mock Activity instance creationRobolectric.buildActivity() and its control via ActivityController. Fragment/View manipulation also works on such mock activity instances.
There are now provided add-on modules that cover multi-dex, v4-support, play services, maps and http client as well. So, its now easy to test code using these library functions as well.
Cons: Where I found it not so good:
Robolectric excels at aiding Unit testing, but does not cover all the functionality a real device or emulator can offer. For example sensors, gps, open-gl etc etc.
You'll need an emulator or real device when doing integration or UI testing, so that Activities and services can interact with full android environment (other apps, like using camera app to get a picture for your app), not a limited one. Here you'll need to use the default test framework as it has functions to test UI as well.
JNI loading seems not to be supported. So code with native dependency can't be tested.
As of now, Robolectric has a hard wired dependency on google maps jar to work. And will download another android.jar from maven. So, project setup may require a bit of a tinkering. Update: as of v3 it seems to pull all dependencies via Gradle without much fuss.
Newer Android tools support coverage and reports generation etc, but only when test are run on a device. So with Robolectric you'll have to create extra Gradle tasks (run Jacoco) to do it for you. Update: Gradle 2.9 + ships with jacoco plugin.
As both gradle and android build tools are shipping out newer build versions at a fast rate, stable Robolectric versions will sometimes start having problems with the changed build tooling. Most typical problems include: sdk version incompatible, manifest not found, build output paths mismatch, resources not loading, build config issues etc. Some issues are also related to bugs in android tools. At times you may even have to write your own custom test runner or apply workarounds till next version fixes those issues. Check out open issues and configure tests accordingly.
Another alternative is simply mock stuff on your own, no frameworks involved. Its the "hard way" but the most customizable way. Its plain JUnit with JMockit:
#RunWith(JMockit.class)
public class OtherTest {
public void testHandlerCallback(#Mocked final FragmentTransaction transaction,
#Mocked final FragmentManager manager,
#Mocked final Activity activity,
#Mocked final LayoutInflater inflater,
#Mocked final ViewGroup parent) {
final List<Fragment> fragments = new ArrayList<>();
new Expectations() {{
activity.getFragmentManager(); result = manager;
manager.beginTransaction(); result = transaction;
transaction.add(withCapture(fragments), anyString);
transaction.commit(); result = new Delegate<Void>() {
public int commit() {
View v = fragments.get(0).onCreateView(inflater,parent,null);
Deencapsulation.invoke(v,"onMeasure",0,0);
return 0;
}
};
}};
}
}
Above is a crude and inline example. You can actually create proper re-usable classes (say FragmentTestHarness) that will take a component (say a Fragment) under test and wrap it in completely isolated environment, preparing it for tests.

To share how i do...
Robolectric
For SQL, activities flow, for those objects that needed context.
JUnit4
for api's java module to make sure data are return correctly.
Espresso
For checking ui display correctly.
When I modified api...I only run jUnit4.
When I modified the data binding between api and UI or Sqlite, then I will only run Robolectric.
When I modified UI I only run Espresso.
Sometimes I will run Robolectric and espresso together, but very rare.
But I will run all before publish to play store.
Because I think there is not real benefit for now. But see how u use it to speed up your product quality and development speed.
Correct me if I am wrong.

You would use Robolectric only in cases you need to mock or fake the Android framework, e.g. if you need a context. When using the Android Test Framework you would have to run Instrumented tests for that which is ultra slow.
If you write your tests for letting them run frequently, e.g. because you follow a tdd approach this is not an option. So in this case Robolectric comes in handy.
So the main benefit about Robolectric is that it is a lot faster than Espresso or Instrumented tests in general.
The downside is that it fakes an Android environment which you should be aware of. To validate real world problems, better use a classic Android Framework approach.
The best is still to write your code in a way that you can unit test it and that you don't need a context or any other Android framework dependencies.
Robolectric is getting integrated into the Android Testing Framework since I/O 2018 - check out more at the official Robolectric page and watch this video from I/O 2018

The major benefit of Robolectric is speed. Unit tests with Robolectric do not require a running emulator or device to run the tests and are thus much much faster.
You might still want to have a suite of integration tests that run against a real device, just a much smaller suite.

How to do Cucumber Android integration testing?

I'm trying to set up continuous integration with an Android project and Cucumber.
The idea is to write tests in Cucumber and run the tests on my Android build via Cuke4Duke and NativeDriver for Android.
When I have this running, I plan to use Maven and a Jenkins server to automate the testing, so it is run every time i commit to the Subversion repo.
Has this been done before? Is there a good guide somewhere? Or is it a bad idea to do it this way?

We're doing exactly what you're planning to do with Maven, Jenkins, and Git. The missing ingredient is the android/cucumber integration from lesspainful.com.

I don't think that what you've planned is a bad idea. But I don't know of anyone that's doing Android CI with that particular setup.
You may also want to take a look at Robotium, it's like Selenium for Android and offers a very rich DSL that will help out with your cuke4duke step implementations.

In my company we use a little different setup (but probably you will have to solve similar challenges): Jenkins + Jenkins Android Plugin + Robotium + Ant. We find out that ant is hard to maintain when you try to use it to something more complicated then simple build and we are rewriting our scripts to gradle.
It works quite well, however you should be aware of two potential problems:
1. emulator is slow (even on fast server) - you can consider attaching physical device to your server.
2. you probably have to setup lock (or use only one executor) for emulator since using multiple emulator instance is hard/tricky.

What we have done is write an test instrumentation engine above Robotium. This engine is mainly a state machine reading keywords from a text file and converting them into Robotium API calls. We did initially notice that inputs and outputs were the same: user taps on the screen, a new screen is displayed or new text is displayed.
That allows us to implement keyword test driven but it runs on the device so not remotely.
It is 20% of effort to get 80% of the benefit: easy to write/add new tests that are readable by anybody. Of course there are limitations but our goal was achieved.
Cheers
Ch