I recently started looking into automating accessibility testing on Android. There isn't much information out there on the web.
Has anyone explored this or are currently doing this? If so, can you share your ideas/approach?
Seems like Android's uiautomator relies on Accessibility features working but it doesn't support testing Accessibility. If it relies on Accessibility features, does that mean that basic validation like accessible label exists, etc. can be done by just executing UI tests using uiautomator?
This is a new area for me so any information could be helpful.
Here's a great introduction to Accessibility testing in Android. It basically boils down to:
Manually test your app for visual issues with Accessibility Scanner
Turn on TalkBack and manually test your app to find hearing impaired issues
To find font scaling and layout issues, use Large Text
Definitely lint check, but make sure that 'Image without contentDescription' is set to Severity = Error
Any/all accessibility issues you find or that are reoccurring, write an Espresso test to fail when that accessibility issue is violated in the future
For automation, you'll also need to consider how to perform visual validation of certain screen artifacts and audio analysis if hearing impaired functionality is a requirement.
Also, I recommend watching this presentation from GTAC 2015 on accessibility testing for some great context on the topic.
For automated tests that check for accessibility, I'd very much recommend starting with issues that can be identified in elements that are shared across screens (menus, layouts, themes, custom controls). While they won't catch the one-off errors that will occasionally pop up, they'll address issues that happen everywhere in your app, a 'prioritize by volume' approach if you will.
Also, if your team uses Android Studio then you definitely want to push for the ability to write Espresso tests that reside with the code. QA are part of the development process, period. Getting access to a subfolder where your tests reside shouldn't be a problem unless there some legal bologna to deal with. For instance, split out the 'androidTest' folder as a submodule where you have pull/push rights as a tester, but only read rights to the rest of the app so you can compile and run yourself. If you're writing Appium tests, it may be harder to ask your dev team to run them as part of their own BVT/smoke test process during builds, but not unheard of.
As for visual analysis and audio injection/confirmation, these are advanced capabilities that you'll probably need to use some service or commercial tool for.
Best of luck!
I agree with Paul's answer in its entirety and he links to some extremely helpful resources (so please have a look at them!), but if all you're looking for is basic accessibility test coverage as you suggest (e.g. checking for accessible labels on all your components), your use case might be a good one for something like Continuum for Mobile, specifically the Android variant. You can then do more manual passes once you've found the more basic violations that can be detected using automated tools; as of right now, manual testing is always necessary for total compliance with accessibility standards, but something like this will get you closer.
Related
Is there any chance to take a snapshot from Android native camera from your own app, and avoid INJECT_EVENTS Security Exception.
Advise something, please.
P.S I use Robotium and failed to do that
Robotium is based on instrumentation which was designed to test just one activity, if I'm not wrong. Robotium is just a wrapper for instrumentation itself and provides a lot of useful methods. The issue is that instrumentation is bind strictly to target application and you cannot invoke anything for another context. Basically writing, if you leave your target application - press back, open another application like camera you are out of the target context and you cannot do anything. Sending any mouse events ends up with security exceptions etc.
There are other frameworks designed for cross application testing.
First of all there is monkey runner, which uses jython (everything the worst from python and java put together). The issue is that MR allows you to just click XY and it doesn't know about screen size, so those cannot be scalable. Another issue is that clicks and drags are almost everything what MR offers. More you can find here
Second one is UIAutomator. It's quite powerful and well designed, however it supports api level 16 and above. More details here.
There is also appium, which I heard allows to cross apps testing, however I don't have enough knowledge about it, so help yourself and go to appium.io
Anyway I suggest to read a bit more about android testing. Maybe this discussion is going to be helpful as well.
What I mean is automating test writing itself. Tester can perform some actions, they get recorded, and then can be used as a base for writing tests. I wasn't able to find any existing solutions, is it that hard to implement? Or did I miss something?
I mean it might be as simple as logging all touch/key events, and write tests using for example https://github.com/square/javawriter
UPD: I should've mentioned that I know of standard testing approaches:
UiAutomator
Roboelectric
Espresso (android-test-kit)
But that wasn't what I meant. Writing tests by hand is a pain, and I wander if this process can be simplified.
UPD1: If anyone stumbles apon this question, this is what I meant. It's non-free, so I'd gladly check out free/opensource analog.
You can try to use Appium (iOS/Android) which is free and has a feature that lets you record actions, and run them through. However this only works for UI functionality and any standard Unit tests will still need to be developed the standard way (manually). As a note, Appium is new (~1 year old) so there may still be bugs (such as not running in iOS devices with Xcode 5.0.1 +)
appium.io
I feel it's worthwhile to mention that appium doesn't require any modifications/additions to your android/iOS projects like a few of the other solutions I have found.
Please Calash https://github.com/calabash which is open and compatible with both iOS and Android.
I have googled about testing uni and systematic testing. But I am confused what exactly is systematic testing? I’m developing a mobile app and like to know how I can approach this technique testing.
Any ideas? Thanks in advance.
Systematic testing is a much more exhaustive means of debugging software. The goal is to test the software against a variety inputs in order to find as many defects/errors as possible. The difficult part about systematic testing is usually coming up with the inputs/situations to test against (especially in the case of Android development, as a number of things can go wrong). Does your app work correctly after configuration changes? Do your EditText boxes specify the correct inputType? Systematic testing provides a means of uncover such errors, and often results in a more secure/robost application.
The Android team provides several tools that can automate this kind of testing (although obviously there is no guarantee that it will cover ALL of the cases). One such tool is monkeyrunner. Check out the Android developer's guide on testing for more information.
A systematic test is a test, in which
the setup is defined,
the inputs are chosen systematically,
the results are documented and evaluated by
criteria being defined prior to the test
I found couple of difference frameworks which can test Android apps, but all of them are limited on testing of just one app (because mainly they use Instrumentation)
My app contains a service which could be called by other apps and I want to automate testing of this too. So, I would like to be able to write some tests which automate UI in other apps.
Have you seen anything, except MonkeyRunner? I looked at it, but the API is quite poor.
Take a look at Sikuli IDE it's easy enough to use and is based on Python.
You can bascially run integration tests using it, (kind of like Selenium for desktops).
There is also Selenium Android Driver if you want to run automaton tests from the Android WebView!
You are right, a bare monkeyrunner is perhaps not enough, but if you combine it with other tools perhaps you can find your way. For example, AndroidViewClient is a library that adds the ability of:
finding Views by ID, very much like you normally do in your Android Activity (using ViewClient.findViewById())
obtaining the value for almost any of the Views properties (using for example View.isChecked() or View.mText())
sending touch events to the Views by simply invoking View.touch()
More details an a complete example can be found at http://dtmilano.blogspot.com/2012/02/monkeyrunner-interacting-with-views.html
I have a fairly basic application. I'm loading it up on 2.1+ on emulators and 2.3.3 on an n1 and basically just making sure everything looks right and does what it's supposed to.
I was reading about activity testing on the dev site and it talks about creating test applications and such. I don't get the point of testing whether a textview = a resource string that you set it to or not.
How deep do you really need to test fairly basic apps before letting other people use them. Is there really anything that I won't catch using emulators and an n1 that test applications and such will?
The general rule of thumb is test anything that is complex enough to go wrong. If you don't have alternate resources, it is generally too simple to break.
Creating programs to test other programs? The only purpose I would see to this would be testing security of your app.
I've never had to create anything like this, and I publish many types of apps on the marketplace.
In my opinion, it is always best to do testing on an actual phone. Emulators arnt fully functional and possibly slower than the phone itself.