Develop Reference

Android Espresso FlakyTest annotation - how to filter tests on execution?

From what i read from these docs we can annotate any flakytests and then there should be a way to filter them out and run them alone.I was thinking firebase robo tests will know about this and re -tests the flaky ones only, but the following statements have confused me :
Can then be used to filter tests on execution using -e annotation or -e notAnnotation as desired.
What is this switch -e ? How can i filter tests ? the comment leads me to confusion on how to fitler tests on execution. is it done on the gradle command line ? Can i get an example ?

enter code hereI finally found out how to do this. When you put annotations on test methods you can run a group of methods that have the same annotation. Reading the docs we find out how to do this. so if i were to mark many tests as #FlakyTest then i can run all of the FlakyTest with adb like this:
adb shell am instrument -w -e annotation android.support.test.filters.FlakyTest
Here is an the part of the AndroidJUnitRunner docs explaining this:
Filter test run to tests with given annotation: adb shell am instrument -w -e annotation com.android.foo.MyAnnotation com.android.foo/android.support.test.runner.AndroidJUnitRunner

Android Manifest Instrumentation

I have a android test project setup to test my android project. In the android test project's manifest, I have an entry for instrumentation. It looks like:
<instrumentation
android:name="android.test.InstrumentationTestRunner"
android:targetPackage="com.company.android" />
I'm curious what the point of this entry is, and particuarly what the purpose of android:targetPackage="com.company.android" is for. I ask, because I refactored the old project and put classes into different packages, so I'm curious on what I need to update this value to... is it suppose to point the the package where the class that extends android.app.Application is at?

It tells the build system where to access the actual project you are going to test.
This is necessary because you need access to all your Activities and classes without having extra copies around.
Info about it is scattered around in: http://developer.android.com/tools/testing/testing_android.html

InstrumentationTestRunner is something you use to write Android unit tests.
From the documentation:
Typical Usage
Write TestCases that perform unit, functional, or performance tests against the classes in your package.
Typically these are subclassed from:
ActivityInstrumentationTestCase2
ActivityUnitTestCase
AndroidTestCase
ApplicationTestCase
InstrumentationTestCase
ProviderTestCase
ServiceTestCase
SingleLaunchActivityTestCase
In an appropriate AndroidManifest.xml, define the this instrumentation with the appropriate android:targetPackage set.
Run the instrumentation using "adb shell am instrument -w", with no optional arguments, to run all tests (except performance tests).
Run the instrumentation using "adb shell am instrument -w", with the argument '-e func true' to run all functional tests. These are tests that derive from InstrumentationTestCase.
Run the instrumentation using "adb shell am instrument -w", with the argument '-e unit true' to run all unit tests. These are tests that do notderive from InstrumentationTestCase (and are not performance tests).
Run the instrumentation using "adb shell am instrument -w", with the argument '-e class' set to run an individual TestCase.

Test suite issue in robotium

I have one test case file with around 20 methods (test cases) which extends ActivityInstrumentationTestCase2. I need to write a suite which will call only selected test case methods, I know in junit there is one method which accepts the methods to be executed
suite.addTest( new AllTestCases("testcase1"));
Is there a similar way to do stuff in android robotium? If yes, please help me out with a way to fix this. Thanks.

You can't make a call like new AllTestCases("testcase1"); because all Android related test classes inherit from either AndroidTestCase or InstrumentationTestCase and neither of these classes expose a constructor that takes a string as an argument.
You could take a look at android.test.suitebuilder.TestSuiteBuilder but even this class does not allow for the running of individual test methods, it accepts tests at the package level.
You might have some luck achieving your goal by using the Android test annotations such as #SmallTest, #MediumTest, #LargeTest etc. These will allow you to target only the specified annotated methods using the follwing command:
adb shell am instrument -w -e size <small|medium|large> com.youproject.test/android.test.InstrumentationTestRunner
Finally, its possible to target individual tests methods or classes directly from within eclipse.

To run an individual test case directly from command line:
adb shell am instrument -w -e class <Test-Class-With-Package-Name>#<Test-Method-Name> <Package-Name-Of-Test-App>/<Instrumentation-Name-Defined-In-Manifest>
Example:
adb shell am instrument -w -e class com.myapp.test.ActivityFragmentTest#testLogin com.myapp.test/android.test.InstrumentationTestRunner

You can run individual test cases programmatically with "-e" arguments to the "adb shell am instrument" command. For example, for a method 'testFoo()' in 'com.foo.bar.FooTest' you could run:
adb shell am instrument -w \
-e "class com.foo.bar.FooTest#testFoo" \
com.foo.bar.test/android.test.InstrumentationTestRunner
http://developer.android.com/guide/developing/testing/testing_otheride.html

How we can skip any Test Cases during CTS Run?

How we can do the following task:
During the running time we can skip the any package or case...?

You can not skip the particular test cases directly in CTS. For that you have to execute the Test Cases manually which you want to execute. Since there are thousands of test cases so there is a short way to execute test cases, use the short package name which is common.
eg. you can use $ start --plan CTS -p android.app
So this will executes all the test cases which starts with name android.app, like
android.app.cts.ActivityGroupTest
android.app.cts.AlarmManagerTest
android.app.cts.AlertDialogTest
android.app.cts.InstrumentationTest
and so on...

while running CTS locally we can actually write a .xml file (say foo.xml) which cab be kept under android-cts/repository/plans directory. Test cases under <Entry exclude="class#method;class#method name="package"/> will not be executed for the package.
And then we can run like below example
cts run -s device_ip:port --plan foo
This is helpful while debugging CTS issues

We can skip the particular test case by editing the xml file in the Plans folder .
For eg in the folder
android-cts/repository/plans/CTS.xml
This contains list of all the packages to be executed.Simply delete the package which you want to exclude and save it with other name like
CTS_1.xml and run.
run cts --plan CTS1

getAllTests and getTestSuite in android.test.InstrumentationTestRunner

I've created a test runner extending android.test.InstrumentationTestRunner. I'm looking for a way to define the set of tests to get executed based on a set of configurations.
I thought I may be able to override the below methods to return my custom test suite, however, these are not getting called! Just wondering whats the use of these:
public TestSuite getAllTests ()
public TestSuite getTestSuite ()
Any clues? Any other alternatives I can use to define a custom test suite at runtime?
Thanx

I also dont know... Another solution is to define your own annotation and annotate the test you want to run. Then you can run only the tests with that annotation using:
Filter test run to tests with given annotation: adb shell am instrument -w -e annotation com.android.foo.MyAnnotation com.android.foo/android.test.InstrumentationTestRunner
If used with other options, the resulting test run will contain the union of the two options. e.g. "-e size large -e annotation com.android.foo.MyAnnotation" will run only tests with both the LargeTest and "com.android.foo.MyAnnotation" annotations.
Filter test run to tests without given annotation: adb shell am instrument -w -e notAnnotation com.android.foo.MyAnnotation com.android.foo/android.test.InstrumentationTestRunner