I'm starting using Device Farm. My problem is that it is ignoring the order that I used to write tests. In local all works fine because Espresso executes tests in a certain order. To declare that order I used alphabetical order. All my classes starts with a letter (E.g. "A_EspressoTest") so I can choose which class has to be ran first.
Into my classes I use
#FixMethodOrder(MethodSorters.NAME_ASCENDING)
to declare in which order my tests have to be ran.
It seems also like Device Farm ignores all my annotations (E.g. "#Test" ) because it is running also methods that doesn't have that annotation.
Lorenzo,
As of today, there is no way to specify the order of test execution with Espresso. Additionally, your observation about the #Test annotation is correct, we don't currently use that when discovering what test classes/test methods are selected.
AWS Device Farm currently discovers tests based on JUnit 3 style naming conventions (classes starting/ending with the word Test and methods within those classes starting with the word test.
For example:
// This class would be automatically discovered (by name).
public class LoginTests extends InstrumentationTestCase {
// This method would be automatically discovered (by name).
public void testLoginFormUsingInvalidCredentials() {
// ...
}
// This method would **not** be automatically discovered.
#Test
public void loginWithValidCredentials() {
// ...
}
}
// This class would **not** be automatically discovered.
public class Login extends InstrumentationTestCase {
// This method would **not** be automatically discovered since the class was not discovered.
public void testLoginFormWithValidCredentials() {
// ...
}
}
With all that said, we've heard plenty of feedback and requests for supporting test discovery using all JUnit 4 annotations and it's an area of improvement we're definitely taking a look at.
Hope that helps!
Best,
Andrew # AWS Device Farm
Related
I have a few "connected" tests that are only relevant to run on a specific device model or on a specific brand and should be skipped on other brands/models.
I may be missing something, but this kind of filtering seems not possible out-of-the-box with AndroidJUnitRunner (by using annotation and/or passing appropriate arguments to it).
So, I was thinking to extend the AndroidX test framework to support this kind of filtering. In the end, I would like to be able to filter test with something like this
#TargetDeviceFilter(brand="SAMSUNG",model="XCover3")
#Test
public void myTestToRunOnSamsungXCover3DeviceOnly(){
...
}
My question: is there any way to accomplish this kind of filtering without extending AndroidX test framework? And if writing my own AndroidJUnitRunner and/or my own annotations is required, how should I start ?
I found a few interesting base classes that I may need to extend like :
androidx.test.internal.runner.TestRequestBuilder
androidx.test.internal.runner.TestRequestBuilder.DeviceBuild
but as those classes are in a "internal" package: attempting to extend them is probably not a good idea?
Any advice on how to deal with that problem is welcome.
I think, you may use org.junit.Assume.
Create a helper class DeviceHelper to detect mobile device informations for convenience.
Your test logic will be executed only if the assumption is correct.
#Test
public void myTestToRunOnSamsungXCover3DeviceOnly() {
// adapt this part to your business need
org.junit.Assume.assumeTrue(
DeviceHelper.isBrand("SAMSUNG") &&
DeviceHelper.isModel("XCover3")
);
// i.e. you can filter whatever you want test's according to device sdk_int
assumeTrue(SomeHelper.getDeviceSdk() >= 21);
// your test code
}
Jake Wharton delivered a fascinating talk where he proposes some smart ways to improve our UI tests by abstracting the detail of how we perform UI out of the tests: https://news.realm.io/news/kau-jake-wharton-testing-robots/
An example he gives is a test that looks as follows, where the PaymentRobot object contains the detail of how the payment amount & recipient are entered into the UI. Putting that one place makes a lot of sense so when the UI inevitably changes (e.g. renaming a field ID, or switching from a TextEdit to a TextInputLayout), it only needs updating in one place not a whole series of tests. It also makes the tests much more terse and readable. He proposes using Kotlin to make them even terser. I do not use Kotlin but still want to benefit from this approach.
#Test public void singleFundingSourceSuccess {
PaymentRobot payment = new PaymentRobot();
ResultRobot result = payment
.amount(42_00)
.recipient("foo#bar.com")
.send();
result.isSuccess();
}
He provides an outline of how the Robot class may be structured, with an explicit isSuccess() response, returning another Robot which is either the next screen, or the state of the current one:
class PaymentRobot {
PaymentRobot amount(long amount) { ... }
PaymentRobot recipient(String recipient) { .. }
ResultRobot send() { ... }
}
class ResultRobot {
ResultRobot isSuccess() { ... }
}
My questions are:
How does the Robot interface with the Activity/Fragment, and specifically where is it instantiated? I would expect that happens in the test by the runner, but his examples seem to suggest otherwise. The approach looks like it could be very useful, but I do not see how to implement it in practice, either for a single Activity/Fragment, or for a sequence of them.
How can this approach be extended so that the isSuccess() method can handle various scenarios. e.g. if we're testing a Login screen, how can isSuccess() handle various expected results like: authentication success, API network failure, and auth failed (e.g. 403 server response)? Ideally the API would be mocked behind Retrofit, and each result tested with an end to end UI test.
I've not been able to find any examples of implementation beyond Jake's overview talk.
I had totally misunderstood how Espresso works, which led to even more confusion in my mind about how to apply it to the page object pattern. I now see that Espresso does not require any kind of reference to the Activity under test, and just operates within the context of the runner rule. For anyone else struggling, here is a fleshed-out example of applying the robot/page object pattern to a test of the validation on a login screen with a username and password field where we are testing that an error message is shown when either field is empty:
LoginRobot.java (to abstract automation of the login activity)
public class LoginRobot {
public LoginRobot() {
onView(withId(R.id.username)).check(matches(isDisplayed()));
}
public void enterUsername(String username) {
onView(withId(R.id.username)).perform(replaceText(username));
}
public void enterPassword(String password) {
onView(withId(R.id.password)).perform(replaceText(password));
}
public void clickLogin() {
onView(withId(R.id.login_button)).perform(click());
}
}
(Note that the constructor is testing to make sure the current screen is the screen we expect)
LoginValidationTests.java:
#LargeTest
#RunWith(AndroidJUnit4.class)
public class LoginValidationTests {
#Rule
public ActivityTestRule<LoginActivity> mActivityTestRule = new ActivityTestRule<>(LoginActivity.class);
#Test
public void loginPasswordValidationTest() {
LoginRobot loginPage = new LoginRobot();
loginPage.enterPassword("");
loginPage.enterUsername("123");
loginPage.clickLogin();
onView(withText(R.string.login_bad_password))
.check(matches(withEffectiveVisibility(ViewMatchers.Visibility.VISIBLE)));
}
#Test
public void loginUsernameValidationTest() {
LoginRobot loginPage = new LoginRobot();
loginPage.enterUsername("");
loginPage.enterPassword("123");
loginPage.clickLogin();
onView(withText(R.string.login_bad_username)).check(matches(withEffectiveVisibility(ViewMatchers.Visibility.VISIBLE)));
}
}
Abstracting the mechanisms to automate the UI like this avoids a mass of duplication across tests, and also means changes are less likely to need to be reflected across many tests. e.g. if a layout ID changes, it only needs updating in the robot class, not every test that refers to that field. The tests are also significantly shorter and more readable.
The robot methods, e.g. the login button method, can return the next robot in the chain (ie that operating on the activity after the login screen). e.g. LoginRobot.clickLogin() returns a HomeRobot (for the app's main home screen).
I've put the assertions in the tests, but if assertions are reused in many tests it might make sense to abstract some into the robot.
In some cases it might make sense to use a view model object to hold a set of fake data that is reused across tests. e.g. if testing a registration screen with many fields that is operated on by many tests it might make sense to build a factory to create a RegistrationViewModel that contains the first name, last name, email address etc, and refer to that in tests rather than duplicating that code.
How does the Robot interface with the Activity/Fragment, and
specifically where is it instantiated? I would expect that happens in
the test by the runner, but his examples seem to suggest otherwise.
The approach looks like it could be very useful, but I do not see how
to implement it in practice, either for a single Activity/Fragment, or
for a sequence of them.
The Robot is supposed to be a utility class used for the purpose of the test. It's not supposed to be a production code included as a part of your Fragment/Activity or whatever you wanna use. Jake's analogy is pretty much perfect. The Robot acts like a person interacting with the screen of the app. So the exposed api of a Robot should be screen specific, regardless of what your implementation is underneath. It can be across multiple activities, fragments, dialogs, etc. or it can reflect an interaction with just a single component. It really depends on your application and test cases you have.
How can this approach be extended so that the isSuccess() method can
handle various scenarios. e.g. if we're testing a Login screen, how
can isSuccess() handle various expected results like: authentication
success, API network failure, and auth failed (e.g. 403 server
response)? Ideally the API would be mocked behind Retrofit, and each
result tested with an end to end UI test.
The API of your Robot really should specify the what in your tests. Not the how. So from the perspective of using a Robot it wouldn't care if you got the API network failure or the auth failure. This is how. "How did you end up with a failure". A human QA tester (=== Robot) wouldn't look at http stream to notice the difference between api failure or http timeout. He/she would only see that your screen said Failure. The Robot would only care if that was a Success or Failure.
The other thing you might want to test here is whether your app showed a message notifying user of a connection error (regardless of the exact cause).
class ResultRobot {
ResultRobot isSuccess() { ... }
ResultRobot isFailure() { ... }
ResultRobot signalsConnectionError() { ... }
}
result.isFailure().signalsConnectionError();
I was wondering if it was good practice to subclass the test cases on Android. I mean, I need to test a lot of Parcelable objects and I could create a class like GenerericParcelableAndroidTestCase to test all these objects.
I also have a problem implementing it, I have something like this:
public class GenericParcelableTest extends AndroidTestCase {
private Parcelable p = null;
GenericParcelableTest(Parcelable p) {
this.p = p;
}
public void testDescribeContents() throws Exception {
assertEquals(0, p.describeContents());
}
}
And that:
public class AttachmentTest extends GenericParcelableTest {
public AttachmentTest() {
super(new Attachment());
}
}
Attachment implements Parcelable of course.
It returns me this error:
junit.framework.AssertionFailedError: Class GenericParcelableTest has no public constructor TestCase(String name) or TestCase()
I mean, I know that I created no empty constructor but why would I need one?
And generally, is there some known issues with this approach? If not why is there very few article on this topic on the internet (and actually some say even that it's not a good idea).
I have this conversation quite often when introducing new team members to unit testing. The way I explain it is by stating that your tests are first class citizens of your code base (no pun intended), they are susceptible to the same technical debt as any other part of your code base and have equivalent (maybe more?!) importance as that of the runtime code.
With this mindset, the questions begins to answer itself; if it makes sense from an OO perspective to use inheritance (i.e. your subclass is a insert name of test superclass) then subclass away. However, like any abuse of inheritance ever, be careful...the minute you add a test case that doesn't rely upon that superclass behaviour you may have a code smell.
In this scenario, it's likely (perhaps 90% of the time?) it is a separation of concern issue within the code being placed under test, i.e. the "unit" under test isn't actually (one) unit but has combinatorial behaviour. Refactoring that code to do one thing would be a good way of allowing your super-class test case to live on. However, watch this super class test case like a hawk...the minute you see booleans being added to signatures to "allow that similar but not the same" test case to run under your once unpolluted super class then you have a problem, a tech debt problem that is no different to your runtime code.
At last check AndroidTestCase depends on an Activity context so it's likely best described as an integration test which tend to regularly have boilerplate super-class test behaviour. In this case, try to narrow the focus of your superclass to the use case under test...i.e. extends LoginUseCase or extends LoginScenario to better "bucket" those subclasses in the first instance. This will help guide would be extenders as to whether they should be using it for their non-login scenario. Hopefully, conversation will ensue and tech debt accumulation be avoided!
Regarding your error, in JUnit3 do what #Allen recommends, if moving to JUnit4 with something like Robolectric then explore using Rules as well as #BeforeClass.
Personal note
I have only felt the need to write test super classes for pseudo-unit tests that mock an API end point (akin to MockWebServer if you are familiar with that product) and DAO integration tests whereby an in-memory db is started and torn down over the lifecycle of each test (warning - slow (but useful) tests!)
junit.framework.AssertionFailedError: Class GenericParcelableTest has no public constructor TestCase(String name) or TestCase()
You get this error because JUnit needs to be able to construct an instance of your test class. It only knows how to do this using no-arg, or single string constructors.
Instead of performing initialization in your constructor, you should put it in the setUp() method. This will let you use the default constructor while still initializing the object before the test method is called.
I have subclass of Application called myApplication, and i have a method in there to setup crashlytics. But in my unit tests everytime i run a test its restarting a new Application so then it tries to setup crashlytics again each time and then eventually i just get Out of memory exception. How can i force a method to only be executed once since its in Application which is the foundation of the application.
my subclass if necessary , looks like this:
public class MyApplication extends Application {
#Override
public void onCreate() {
super.onCreate();
setupCrashlytics();
}
/**
* sets up Crashlytics.
*/
private void setupCrashlytics() {
Crashlytics.Builder crashlytics= new Crashlytics.Builder();
Fabric.Builder fabric= new Fabric.Builder(this);
fabricBuilder.kits(crashlytics.build());
Fabric.with(fabric.build());
}
}
I assume you're running instrumentation tests, which are dependent of the configuration you apply in your Application subclass.
Although the tests should be run in an environment as close as possible to the production, you can differentiate your test environment by exposing a MyApplication.setIsTestEnvironment() method and using the flag to enable/disable third party dependencies when you're testing.
Alternatively, some testing libraries (i.e. Mockito) allow you to fully or partially mock an object. You can do that to prevent your real Application class from initializing unwanted dependencies when testing.
I am slowly becoming obsessed with unit testing. I am trying to develop as much software as I can using test-driven development. I am using JUnit to unit test my android applications.
I have been working on an app that uses bluetooth and am having a hard time unit testing it. I have an Activity that uses BluetoothAdapter to obtain a list of paired and discovered devices. Although it works, I would like to know how to unit test it.
To get the list of paired devices, I call getBondedDevices() on the instance of BluetoothAdapter. The problem is I don't know how to stub or mock this method (or any other bluetoothAdapter method that my Activity calls) so I can't test my Activity against different lists of paired devices.
I thought about using Mockito or trying to subclass BluetoothAdapter to somehow stub out the methods I'm interested in, but it's a final class so I can't do either.
Any ideas on how I could test programs that use BluetoothAdapter or other resources that are (as far as I know) difficult or impossible to stub or mock? As another example, how would you test a program that uses sockets?
thanks in advance for any help
aleph_null
To test your activity, you may refactor your code. Introduce a BluetoothDeviceProvider with a default implementation.
public interface BluetoothDeviceProvider {
Set<BluetoothDevice> getBluetoothDevices();
}
public class DefaultBluetoothDeviceProvider implements BluetoothDeviceProvider {
public Set<BluetoothDevice> getBluetoothDevices() {
return new BluetoothAdapter.getDefaultAdapter().getBondedDevices();
}
}
then inject this new interface, in the activity :
public class MyActivity extends Activity {
private BluetoothDeviceProvider bluetoothDeviceProvider;
public MyActivity(BluetoothDeviceProvider bluetoothDeviceProvider) {
this.bluetoothDeviceProvider = bluetoothDeviceProvider;
}
protected void onStart() {
Set<BluetoothDevice> devices = bluetoothDeviceProvider.getBluetoothDevices();
...
}
...
}
Now the activity seems unit-testable. But the BluetoothDevice is still final and you can't inject a mock in your activity. So you have to refactor this new code and introduce a new interface that wraps the BluetoothDevice... -> an abstraction layer upon the core android classes.
At the end the activity behavior can be checked via various unit tests... So the implementations of the newly introduced interfaces remains to test. To do this, you can :
keep them not (unit) tested, not a big problem for me since they just do delegation
look at PowerMock.
Also check this wiki page about mocking final classes using PowerMock.