Is there any chance to take a snapshot from Android native camera from your own app, and avoid INJECT_EVENTS Security Exception.
Advise something, please.
P.S I use Robotium and failed to do that
Robotium is based on instrumentation which was designed to test just one activity, if I'm not wrong. Robotium is just a wrapper for instrumentation itself and provides a lot of useful methods. The issue is that instrumentation is bind strictly to target application and you cannot invoke anything for another context. Basically writing, if you leave your target application - press back, open another application like camera you are out of the target context and you cannot do anything. Sending any mouse events ends up with security exceptions etc.
There are other frameworks designed for cross application testing.
First of all there is monkey runner, which uses jython (everything the worst from python and java put together). The issue is that MR allows you to just click XY and it doesn't know about screen size, so those cannot be scalable. Another issue is that clicks and drags are almost everything what MR offers. More you can find here
Second one is UIAutomator. It's quite powerful and well designed, however it supports api level 16 and above. More details here.
There is also appium, which I heard allows to cross apps testing, however I don't have enough knowledge about it, so help yourself and go to appium.io
Anyway I suggest to read a bit more about android testing. Maybe this discussion is going to be helpful as well.
Related
I'm attempting to add some test automation to my XF app, but I'm having some difficulty as I've never used Appium before so I'm hoping that someone with experience may be able to help with my issues.
When I'm setting up my tests - if I set the "automationName" to be "UiAutomator2", then I can launch my app without problems and interact with the UI fine.
I need to do some tests with specific methods inside my app, but Appium only allows interaction with the UI. Doing some searching, I found that Appium contains an Espresso driver which is supposed to allow interaction with specific methods (this sounds exactly what I'm after).
The suggestion is to change the "automationName" from "UiAutomator2" to "Espresso" - the example I was going through in the Appium documentation was a simple test that just launched the app. It suggested that just by changing this setting, that the same test would work regardless.
I have created a simple test that just launches my app - this works fine with "UiAutomator2" but as soon as I change to "Espresso", my app doesn't launch.
The Appium server mentions not being able to find the signed .apk.
According to the documentation, this should all be working fine but as it isn't I'm guessing that there's something else that I need to do so I'm hoping that someone with experience with this will be able to help me get the tests to work.
I don't know whether I need to add a reference to Espresso somewhere in my XF app or if I need to build the .apk in a special way that will make it work.
Any advice or help on this matter would be grateful.
IMHO there are better options to achieve what you want. I'd always test the methods that are not platform specific separately, e.g. with with NUnit. You can run Unit- and Integration-Tests created with NUnit with the NUnit GUI or ReSharper if you've got a license. Unless you are using very exotic functions (or relying on timings, etc.) this should work well across all platforms.
If you are not sure that your methods will behave the same across all platforms (better to be sure), I'd create a separate App that runs tests on those methods on an actual device. There is a runner for NUnit-Tests for Xamarin.Forms, you could use for this purpose. The last release is from 2017, but I'd give it a shot. This way you can make sure that your methods work as intended, but avoid having to access them via a framework that is aimed at UI-tests.
I recently started looking into automating accessibility testing on Android. There isn't much information out there on the web.
Has anyone explored this or are currently doing this? If so, can you share your ideas/approach?
Seems like Android's uiautomator relies on Accessibility features working but it doesn't support testing Accessibility. If it relies on Accessibility features, does that mean that basic validation like accessible label exists, etc. can be done by just executing UI tests using uiautomator?
This is a new area for me so any information could be helpful.
Here's a great introduction to Accessibility testing in Android. It basically boils down to:
Manually test your app for visual issues with Accessibility Scanner
Turn on TalkBack and manually test your app to find hearing impaired issues
To find font scaling and layout issues, use Large Text
Definitely lint check, but make sure that 'Image without contentDescription' is set to Severity = Error
Any/all accessibility issues you find or that are reoccurring, write an Espresso test to fail when that accessibility issue is violated in the future
For automation, you'll also need to consider how to perform visual validation of certain screen artifacts and audio analysis if hearing impaired functionality is a requirement.
Also, I recommend watching this presentation from GTAC 2015 on accessibility testing for some great context on the topic.
For automated tests that check for accessibility, I'd very much recommend starting with issues that can be identified in elements that are shared across screens (menus, layouts, themes, custom controls). While they won't catch the one-off errors that will occasionally pop up, they'll address issues that happen everywhere in your app, a 'prioritize by volume' approach if you will.
Also, if your team uses Android Studio then you definitely want to push for the ability to write Espresso tests that reside with the code. QA are part of the development process, period. Getting access to a subfolder where your tests reside shouldn't be a problem unless there some legal bologna to deal with. For instance, split out the 'androidTest' folder as a submodule where you have pull/push rights as a tester, but only read rights to the rest of the app so you can compile and run yourself. If you're writing Appium tests, it may be harder to ask your dev team to run them as part of their own BVT/smoke test process during builds, but not unheard of.
As for visual analysis and audio injection/confirmation, these are advanced capabilities that you'll probably need to use some service or commercial tool for.
Best of luck!
I agree with Paul's answer in its entirety and he links to some extremely helpful resources (so please have a look at them!), but if all you're looking for is basic accessibility test coverage as you suggest (e.g. checking for accessible labels on all your components), your use case might be a good one for something like Continuum for Mobile, specifically the Android variant. You can then do more manual passes once you've found the more basic violations that can be detected using automated tools; as of right now, manual testing is always necessary for total compliance with accessibility standards, but something like this will get you closer.
What I mean is automating test writing itself. Tester can perform some actions, they get recorded, and then can be used as a base for writing tests. I wasn't able to find any existing solutions, is it that hard to implement? Or did I miss something?
I mean it might be as simple as logging all touch/key events, and write tests using for example https://github.com/square/javawriter
UPD: I should've mentioned that I know of standard testing approaches:
UiAutomator
Roboelectric
Espresso (android-test-kit)
But that wasn't what I meant. Writing tests by hand is a pain, and I wander if this process can be simplified.
UPD1: If anyone stumbles apon this question, this is what I meant. It's non-free, so I'd gladly check out free/opensource analog.
You can try to use Appium (iOS/Android) which is free and has a feature that lets you record actions, and run them through. However this only works for UI functionality and any standard Unit tests will still need to be developed the standard way (manually). As a note, Appium is new (~1 year old) so there may still be bugs (such as not running in iOS devices with Xcode 5.0.1 +)
appium.io
I feel it's worthwhile to mention that appium doesn't require any modifications/additions to your android/iOS projects like a few of the other solutions I have found.
Please Calash https://github.com/calabash which is open and compatible with both iOS and Android.
I found couple of difference frameworks which can test Android apps, but all of them are limited on testing of just one app (because mainly they use Instrumentation)
My app contains a service which could be called by other apps and I want to automate testing of this too. So, I would like to be able to write some tests which automate UI in other apps.
Have you seen anything, except MonkeyRunner? I looked at it, but the API is quite poor.
Take a look at Sikuli IDE it's easy enough to use and is based on Python.
You can bascially run integration tests using it, (kind of like Selenium for desktops).
There is also Selenium Android Driver if you want to run automaton tests from the Android WebView!
You are right, a bare monkeyrunner is perhaps not enough, but if you combine it with other tools perhaps you can find your way. For example, AndroidViewClient is a library that adds the ability of:
finding Views by ID, very much like you normally do in your Android Activity (using ViewClient.findViewById())
obtaining the value for almost any of the Views properties (using for example View.isChecked() or View.mText())
sending touch events to the Views by simply invoking View.touch()
More details an a complete example can be found at http://dtmilano.blogspot.com/2012/02/monkeyrunner-interacting-with-views.html
I am new to Android test frameworks ,Would like to know the differences between existing test frameworks : Monkey , CTS ,Instrumentation Framework & Robotium ?
Instrumentation is a category of testing, opposite to Unit-testing.
The framework provides hooks for instrumentation testing, but you are going to need an additional third-party framework to really get going.
Robotium is such a framework. It allows you to write "scripts" that run through the user interface, saying "click this", "type that", etc. Well-written it can take you through your usecases and thus provide a good feeling that your app isn't broken. It also allows you to test multiple activities and activities interacting.
Unit-testing in my experience is very hard for Android, especially for the "regular" code dealing with UI, databases, activity state, etc., unless you write your code for testability.
The Android Monkey also uses instrumentation to run through your user interface but it does not follow a script. It does this randomly, with the idea that whatever it does it should not crash your app. By generating 100000's of events it tries to get coverage as high as possible, based on statistics. Other than Robotium, the monkey never leaves your app (that would be dangerous). It's a perfect complement though and it comes nearly for free (the setup is really cheap and there is no maintenance).
CTS is only relevant to the operating system and framework itself.
You'll probably also want to know about mocks?
Observe the testing Pyramid below:
Manual testing - self explanatory
Functional testing - testing a feature
Integration testing - checking the units play nicely
Unit tests - make sure an individual unit works as expected (See SRP)
It suggests how many tests you should have of each level. Below the pyramid are the Android frameworks that you can use at each level.
In Android, the following frameworks are commonly used for each section:
Functional:
Monkey runner "kind of" falls under this section, it basically just bashes around the app to see if any combination of interactions crashes it
Integration:
Instrumentation falls under this category.
Espresso (Made by Google, recommended, uses Hamcrest matchers)
Robotium
Unit:
JUnit4
Mockito, Powermock, other mocking libraries
Matching frameworks like Hamcrest, Fest, AssertJ
Robolectric (provides Android specific methods)