I know that, in test automation, we have to avoid sequential test-cases.So, the order of running the test-cases are not important.
I believe in some cases the sequential test-cases is unavoidable:
1.Consider a scenario which a user needs to take some previous steps in order to complete a final goal. For example, a user needs to be logged in, so that he can purchase.
Given User is logged in
When User adds an item into its basket
And User Complete his purchase
Then He receives an Email
So, in above scenario, each part (Given, When, And, or Then) are seperate testcases. But still the order of testcase is crucial.
2.Also, Junit team provides a method called #FixMethodOrder(MethodSorters.NAME_ASCENDING) for such usage, but i am not sure when we are allowed to use this?
So, how do you write independent test-cases in end-2-end testing?
Tests are meant to be atomic, in the sense that they should NOT rely on a status that was generated by a previous test. That's why sequential test-cases should be avoided. In case any part of the sequence fails, then the rest of the tests fail, so it really doesn't make much sense to have them sepparate (they could be just one test).
If you need preconditions on your tests you can include them in: 1) before all tests, 2) at start of the test suite, 3) before your specific test, 4) inside your specific test. No matter where/when this is done, the important thing is that you assume that all that's needed to generate that status works as expected, so you just focus on what you are testing.
Now, if you want to test a complete flow of a system, I would recommend you to put that inside one unique test. In case there are variations, then you would have multiple tests.
Let's say for example that you want to test D, but D needs C, and then C needs B, and finally B needs A; one strategy would be to generate a test in which you would do A->B->C->D. Another strategy is to generate multiple tests in which you test the individual steps: test 1 does A; test 2 does A->B (but doesn't care if A is ok); test 3 does A->B->C (but doesn't care if A and B are ok); and so on. Which strategy to use would depend on your goals and the size of the scenario. I personally prefer the second option for big things, and the first one for simple/short ones.
The espresso UI test framework allows to implement such core-user-journey scenarii.
If you want unit tests, then you should set the expected preconditions at the beginning of each.
The saying of "avoid sequential test cases" only apply to unit tests. And that is where Junit and mockito shine. In these cases, we are only worried about a unit correctness. We don't care about other units fail or pass or whatever, we will mock other unit behaviors, so that we can focus on testing exact behaviors of current test.
The case you described don't fall into this category. In your case, you need an integration test suite since you need to test end to end flow. You can use the following tools for integration test.
Cucumber
Soap UI Test
Serenity
etc
Related
I'm writing data driven testing of my android app, and have begun writing a CSV to espresso testing framework which will allow me to quickly design and alter tests. Espresso can verify basic things about the UI such as successful clicks and text entry, but can it handle more advanced logic?
For example, I'm attempting to create a receipt based on some clicks that are made during the test. This will have been costed before runtime and entered into the CSV for automatic comparison of actual vs expected outcomes. In order to do this, I will need to access the receipt object to check its values? Or at least the textbox which displays it (easy, but not really what I'm after).
So, is there any way to get objects stored in memory from the main application into the espresso testing framework?
Found the answer to my question.
In order to get to any objects/variables, you need to get to a class in which they exist. This is achieved in espresso with the #Rule tag:
#Rule
public ActivityTestRule<MainActivity> mainActivityTestRule = new ActivityTestRule<MainActivity>(MainActivity.class);
From here you can do:
mainActivityTestRule.getActivity();
and from there all objects can be accessed
EDIT:
Primitives seem to come through fine, but I'm still trying to confirm if objects are actually passed through. It seems that the objects that are in my Espresso class are default, as though the constructor has just been run. Will update with more info.
Edit 2:
This does NOT work for objects. Putting a break-point in the espresso class and one in the main application and comparing them shows different object IDs. Getting the object from the main application doesn't return the object in the same state.
I am heavily testing my application with unit tests and Espresso tests. My next step is to make sure my application hits all required apis. For that reason I am looking for a way to write a test, which will verify the api calls.
I would really appreciate any suggestions.
What you are describing is called a "unit test". Unit tests are meant to test as many lines of code as possible regardless of UI.
Espresso tests are "instrumentation tests" (or "UI tests") intended to check if the app is responding to UI events correctly. They're not meant to verify the correctness of code, but the correctness of the functionality of the app as used by the user.
You can read about both at the official documentation. You'll find that unit tests are very different than instrumentation tests, and often harder to write because they require more engineering of your code to do correctly. You will likely have to "mock" the various parts of your application to make sure their APIs were called exactly as you expected.
There are 2 main goal when I was writing api tests:
First is component based. The goal was to make sure each class / component makes an api call when certain criteria is met (for example calling an api A when onDestroy() is called)
Second, is to make sure the Apis are called is certain order for the analytics purposes.
The first step I achieved by using unit tests with injected mock objects via Mockito and PowerMockito. PowerMockito was used primarily to mock static methods and to make sure the methods were called at least n times.
For the second step, UI test could be used, since it runs the real application. I have implemented the helper class, which was recording the instances when api requests were made. The script in Espresso was validating the order of api calls by referring the helper class.
I am using Robotium Solo to test the app
since I'm fairly new to app test, and Robotium
I have 3 methods in my test case - however, I want to run those tests, under certain conditions
otherwise they fail
I can do that if i write the entire test in one method, but i don't want to run it in one method, I
want to separate it into 3 methods
how do I make sure that the tests run only if I called the test methods, and not one after the other
In Robotium Test Application, every public method starting with keyword 'test' will be considered as one test case.
So, you can try using 2-3 methods with names starting by keyword test.( e.g. testA(), testB() etc.)
So, even if testA() fails, automation testing will continue to run for testB().
And results will be shown to the JUnit viwe.
Note:- the test cases are executed in the alphabetical order as per their names.
Hope this helps , ask if you are stuck somewhere.
I have troubles with my Robolectric unit test.
I can assert without problem that a click started a new activity when the listener uses the method startActivity( Intent )
But it seems Robolectric has trouble when a new activity is started with the method startActivityForResult(Intent, int) : putting some breaks in the code made me figure out that the activity wasn't started ( and just changing for method startActivity( Intent ) made the assert pass).
Is that normal ? It's a pity since the first activity of my app uses startActivityForResult(Intent, int).
Did someone manage to make tests with this way of launching activities ?
Thanks for your help ..
The short Answer to your question is that, due to the way Robolectric converts Android classes to code that executes in the JVM, a lot of their functionality doesn't behave as you'd expect. Many system callbacks won't execute, and you'll have to rely on what Robolectric provides in their Shadow implementation of classes. (See the link provided by #Steven_BDawg).
The long answer: It may be possible to implement this whole flow in one big test, but it's not what the framework is designed for.
Robolectric and Unit Testing in general aren't meant to be used in the way that you describe. The Unit Testing page on wikipedia states that one can view a unit as the smallest testable part of an application. A unit testing suite should contain many lightweight tests, where each test isolates a bit of functionality in your app and ensure it's working properly.
Consider a basic Application that contains two Activities, A and B. Activity A displays some information about a topic, and Activity B allows the user to select which topic to show in A. When the user moves from Activity A to Activity B, B gets called with startActivityForResult() and should return to A with the selected topic.
Now say we want to Unit Test this flow of A getting the result from B and displaying the data. We can break this up into two tests:
Activity Under Test - Activity A. In our test, we'll create a new instance of Activity A. In the Robolectric Test, we create the Intent that we expect B to return to A, and call the shadow method receiveResult() for A, filling out the arguments with a result code of OK and this Intent. After receiveResult(), run your assertions. You now know that Activity A handles the result properly!
Activity Under Test - Activity B. In our test, we'll create a new instance of Activity B, setting it up as if it were started for result from Activity A. In the Robolectric Test, we'll perform all actions needed to select the data, create the intent we'll send back, then run assertions on the intent to ensure it was created correctly.
This is a very simple example. These two steps could probably be broken out into many more tests, as, again, each unit test should only be testing the smallest unit of functionality that your app can be broken into. The example is mainly to help you start thinking in a unit testing kind of way. I've found that as my understand of unit testing deepens, the way I write code has changed. I try to avoid writing methods and classes in such a way that they do too much work and cannot be properly unit tested. As a rule of thumb, code that's easy to unit test performs very specific operations which are readily apparent when reading the code for the first time.
Finally, if you want to take this a step further, mocking frameworks can greatly aid your ability to Unit Test. Mockito is a mocking framework I've had success with in the past. The purpose of a mocking framework is to create stub Objects whose behavior you tightly control. Mockito (or any other Mocking Framework), will allow you to define an object that extends from any type you need and only implement the methods you need. You'll be able to directly control the response to any of these method invocations. This aids Unit Testing because the only real object that you'll need is the Object Under Test; by Mocking all other objects, you'll have a better sense of whether or not the Object Under Test is behaving properly because all other behavior is explicitly defined by you, the tester. (And yes, this does lead to lots of extra code, but such is the life of a good unit tester. However, as previously stated, as you get more comfortable with unit testing, you may find yourself writing methods that require less mocking and are more conducive to writing tests. Some coders will even write their unit tests BEFORE they code, in order to keep their code tight and focused on a single purpose)
Hope this helps!
I did some Google'ing for you. I don't know if this will definitely help you or not, but I think it is a good start!
Roboelectric: Testing startActivityForResult() and onActivityResult()
I am new for Junit testing framework. Is it possible to test a flow of activities with assertions to check if each is launched using only exisiting classes provided in android API.
It seems we can only test at activity level. But can we automate a flow : like spawning front and back between activities ?
Yes, through the use of Android instrumentation, which provides you with methods that put you into the drivers seat when interacting with activities as the system would through it's instrumentation framework.
Basically, your able to trigger and manage every aspect of an activities life cycle, including introspecting results from activities calling each other. So if you've worked with the ActivityTestCase that's suitable when testing a single activity, you can have a look at the InstrumentationTestCase that provides you with a low-level approach to work with multiple activities at once.