Develop Reference

Firebase test lab showing unexpected test failures

I've started using Firebase Test Lab recently with espresso. All the test cases pass on my local device but when it comes to testing devices present on google cloud, some test cases fail with exceptions like "NoMatchingViewException". I've tried increasing the wait time before the test runs and the fragment loads, but that too does not seem to help. Please help if you know how to resolve this issue. Thank you!

I had the same problem when I tried my app. The problem is that I used UDPSocket in my application.. kinda weird but there was this exception thrown : NoMatchingViewException.
So, I Mocked the UDP call. So at least I was able to do the UI automated test.

How to test MVP presenter method which has database operation

I have implement unit test with MVP Architecture and every thing went well
with Junit4 and Mokito
But my problem my that when i need to test methods which has database operation
i'm facing two issue
1-getting null pointer Exception when using #RunWith(MockitoJUnitRunner.class)
and mocking the Presenter and View
2-using InstrumentationTestCase and MockContext i'm able to open the data base with no error but when i call verify on database related method the test success but in the coverage i see that this method in the presenter is not covered
Thanks

Without seeing your code I can't really provide a detailed answer, but in general, you could use Robolectric (which uses a real SQLite db) in order to unit-test classes which depend on SQLite database.
Keep in mind, though, that these tests might be slow. In my practice, a unit test that uses SQLite and run by Robolectric test runner can take up to 10 seconds and more.

ContentProvider starting service before Junit tests in test enviornment

I have a whole bunch of existing tests for an aidl service. The test class extends ServiceTestCase. They were all working fine until I recently added a new ContentProvider in the same package. Now when I run the tests one or two of the existing tests fail randomly.
The reason for this fail is that I am binding to the service from onCreate() of the ContentProvider. When the test framework restarts the process the ContentProvider is getting created automatically and it starts the aidl service BEFORE the test has executed setup. This results in the service being running even when I have not started it from the particular test. This causes timing issues which cause the random test to fail.
I tried out following things :
If I comment out the bind call from the onCreate of the new ContentProvider then all tests still pass successfully.
(Note : None of the existing tests needs this provider - I still need to add the tests for this provider)
I tried setting some boolean from tests and checking that in onCreate() of the provider - but that does not work since provider is initialized before the setUp() of the junit test
If I add startservice() in tests that did not call it previously then the service actually gets started 2 times and I can see 2 instances running during the unit test

AWS Device farm seems to be ignoring TestNG annotations

I have successfully uploaded and run my tests on AWS device farm. Locally, I'm using fun things like #Test(enabled = false, dependsOnGroups = "Login") to mark which tests to run at the time, and what order they should execute. Locally, this all works fine and dandy as expected. The problem happens after I upload the zip of the maven build to device farm and perform a test run.
Looking at the logs from device farm, it doesn't care if "enabled" is set to true or false, it'l run things regardless. It also ignores the "group=" and "dependsOnGroups" annotations. This is super important, since all other tests will fail if I'm not logged in first. Worse, the subsequent failing tests will not be skipped, so AWS is happily charging me more money off of this.
I tried using #Test(priority=blah), but it's ignoring that too. The only thing it seems to be respecting are things like #BeforeSuite and #AfterSuite.
Anyone run into this or have any ideas why this is happening?

I am an engineer working on AWS Device Farm.
1) "enabled" annotation flag
I just verified that you are correct about our TestNG parser ignoring the "enabled" flag on annotations and always including the test, even if it is disabled. At first glance, this appears to be a simple issue with a straight forward fix. Assuming best case scenario, we will try and have it fixed and live in production as soon as we can.
2) "dependsOnGroups" annotation field
The answer to this one is a bit more complex. As of today, AWS Device Farm does not support the dependsOnGroups or dependsOnMethods annotation fields.
There are a few reasons for this, with the main reason being that AWS Device Farm executes each #Test method individually, each one using a fresh instance of the Appium server. There are pros/cons to this approach that I won't delve into here, but I will say that it does have benefits on both a technical and feature level. When executing individual methods with the TestNG runner, it will only load the context of an individual method, and not all tests/suites/groups of the specified dependencies found within the .jar file. Additionally, since each one of these test methods is executed in a different Java process each time, the TestNG runner does not maintain any "state" in memory. This means that it does not know that it has already executed a test previously (in a different process) so it will attempt to run it again.
3) Executing "groups" of tests
We currently do not expose groups/excludegroups to the user to allow them to pick specific collections of tests to execute or skip. However, this should be possible by configuring your <groups> entries in a testng.xml file placed in the root of your *-tests.jar file that is uploaded in your test package archive. In this case, our parser should only "discover" tests defined within those groups and not all #Test annotated methods. I have not tried this specific case however, so your YMMV.
Hope that helps! If you have any additional questions or have a specific run/test package you'd like us to look at, feel free to reach out or paste a URL to a previously executed run.

I added testng.xml in the root of *-tests.jar and checked.
But device farm is not running the tests listed in testng.xml. It's still running all the classes with #Test annotation

Android JUnit testing not firing test methods

I don't know if I'm really rusty with JUnit or their is a concept with Android Testing in particular I'm not familiar with but:
I'm finding it very difficult to understand how my tests get run.
I've created a Test Project based on my main project, and created a class which extends ActivityInstrumentationTestCase2<SinglePaneActivity> and in this Test Case I've implemented setUp(), testPort() and tearDown() methods.
When I run the project as a Android JUnit test it all tests correctly.
However, adding another class extending ServiceTestCase<NativeService> with the same setUp(), testStart() and tearDown() methods implemented, the test isn't performed.
Looking through the documentation I can't find anything which states how the tests are run, I'm assuming since their is no specific setup it done via reflection.
Given that as the case however, I don't understand the documentation on TestSuites or why my Service test case isn't running.
Am I the only one that's finding the usually very well written Android Documentation lacking when it comes to testing?

As #Blackbelt says, the was a warning in the Log's indicating that the Test wasn't running.
My problem was that I had used the constructor auto generated for me by eclipse
public NativeServiceTestCase(Class<NativeService> activityClass) {
And the error was output as a warning in LogCat explaining you need to have an empty argument constructor (But that error wasn't repeated anywhere else with any visibility).