I have googled about testing uni and systematic testing. But I am confused what exactly is systematic testing? I’m developing a mobile app and like to know how I can approach this technique testing.
Any ideas? Thanks in advance.
Systematic testing is a much more exhaustive means of debugging software. The goal is to test the software against a variety inputs in order to find as many defects/errors as possible. The difficult part about systematic testing is usually coming up with the inputs/situations to test against (especially in the case of Android development, as a number of things can go wrong). Does your app work correctly after configuration changes? Do your EditText boxes specify the correct inputType? Systematic testing provides a means of uncover such errors, and often results in a more secure/robost application.
The Android team provides several tools that can automate this kind of testing (although obviously there is no guarantee that it will cover ALL of the cases). One such tool is monkeyrunner. Check out the Android developer's guide on testing for more information.
A systematic test is a test, in which
the setup is defined,
the inputs are chosen systematically,
the results are documented and evaluated by
criteria being defined prior to the test
Related
Suppose you are tasked with adding a testing framework to an existing code base that has very very little unit testing coverage. The code base isn't insanely large yet, however, it does have areas where it's not super clean, or not very OOP or testable.
I've read a couple of good answers:
Adding unit tests to legacy code
How to approach unit testing in a large project
Best Option for Retrospective application of TDD into C# codebase
But the project I'm working on is an Android app, so it's slightly different (given lots more UI components).
I have some questions that all relate to the same issue:
What's the best way to do go back and put a bunch of tests in place?
How do I prioritize which parts to test first?
do I start with areas where it's getting called a lot (and is there a tool for code analysis like this)?
or do I go back and look at classes which in the past have had the most number of bugs?
Should I write integration tests first (given this is an Android app and also integration tests may prevent broken unit tests/refactorings) and then work in the unit tests?
Sorry for all the questions, just really looking for a good approach, more specifically geared towards retrospectively testing an Android app.
P.S. I'm curious if people know of good tools they can use for code analysis that can help bring to attention the areas in a code base in which unit testing would be most helpful.
Usually when starting with unit testing on an existing application you would want to do it iteratively - you probably do not have the time or the man power for a big upfront investment and so you want to add unit tests as part of the required work:
Write unt tests When adding a new feature - or better yet use TDD
When fixing a bug - write unit test(s) that fails de to the bug before fixing it
When refactoring old code wriute unit tests to make sure no regression bus were introduced
Id the whole team follow the three steps above in a matter of weeks you should have good coverage for the code you've been changing - depending on the size of the project.
I don't have an Android-specific answer. I hope someone comes along and adds a good one. Meanwhile: write acceptance (integration) tests for your most important features (from your product owner's point of view, not from some tool's). You'll get the most coverage for your effort, and be the most likely to prevent bugs that customers care about. Don't worry so much about unit tests; those are for details.
[That's the answer given in the third post you cite, but we're still waiting for that Android-savvy answer so let's not call this a duplicate yet.]
Do write acceptance tests and, as necessary, unit tests for new features, and write unit tests when fixing bugs.
Regarding how to find areas of the code that need tests, the first thing to do is measure code coverage. (You can do that in the Android SDK with ant emma debug install test.) Obviously code with no coverage needs tests.
I recently started looking into automating accessibility testing on Android. There isn't much information out there on the web.
Has anyone explored this or are currently doing this? If so, can you share your ideas/approach?
Seems like Android's uiautomator relies on Accessibility features working but it doesn't support testing Accessibility. If it relies on Accessibility features, does that mean that basic validation like accessible label exists, etc. can be done by just executing UI tests using uiautomator?
This is a new area for me so any information could be helpful.
Here's a great introduction to Accessibility testing in Android. It basically boils down to:
Manually test your app for visual issues with Accessibility Scanner
Turn on TalkBack and manually test your app to find hearing impaired issues
To find font scaling and layout issues, use Large Text
Definitely lint check, but make sure that 'Image without contentDescription' is set to Severity = Error
Any/all accessibility issues you find or that are reoccurring, write an Espresso test to fail when that accessibility issue is violated in the future
For automation, you'll also need to consider how to perform visual validation of certain screen artifacts and audio analysis if hearing impaired functionality is a requirement.
Also, I recommend watching this presentation from GTAC 2015 on accessibility testing for some great context on the topic.
For automated tests that check for accessibility, I'd very much recommend starting with issues that can be identified in elements that are shared across screens (menus, layouts, themes, custom controls). While they won't catch the one-off errors that will occasionally pop up, they'll address issues that happen everywhere in your app, a 'prioritize by volume' approach if you will.
Also, if your team uses Android Studio then you definitely want to push for the ability to write Espresso tests that reside with the code. QA are part of the development process, period. Getting access to a subfolder where your tests reside shouldn't be a problem unless there some legal bologna to deal with. For instance, split out the 'androidTest' folder as a submodule where you have pull/push rights as a tester, but only read rights to the rest of the app so you can compile and run yourself. If you're writing Appium tests, it may be harder to ask your dev team to run them as part of their own BVT/smoke test process during builds, but not unheard of.
As for visual analysis and audio injection/confirmation, these are advanced capabilities that you'll probably need to use some service or commercial tool for.
Best of luck!
I agree with Paul's answer in its entirety and he links to some extremely helpful resources (so please have a look at them!), but if all you're looking for is basic accessibility test coverage as you suggest (e.g. checking for accessible labels on all your components), your use case might be a good one for something like Continuum for Mobile, specifically the Android variant. You can then do more manual passes once you've found the more basic violations that can be detected using automated tools; as of right now, manual testing is always necessary for total compliance with accessibility standards, but something like this will get you closer.
What I mean is automating test writing itself. Tester can perform some actions, they get recorded, and then can be used as a base for writing tests. I wasn't able to find any existing solutions, is it that hard to implement? Or did I miss something?
I mean it might be as simple as logging all touch/key events, and write tests using for example https://github.com/square/javawriter
UPD: I should've mentioned that I know of standard testing approaches:
UiAutomator
Roboelectric
Espresso (android-test-kit)
But that wasn't what I meant. Writing tests by hand is a pain, and I wander if this process can be simplified.
UPD1: If anyone stumbles apon this question, this is what I meant. It's non-free, so I'd gladly check out free/opensource analog.
You can try to use Appium (iOS/Android) which is free and has a feature that lets you record actions, and run them through. However this only works for UI functionality and any standard Unit tests will still need to be developed the standard way (manually). As a note, Appium is new (~1 year old) so there may still be bugs (such as not running in iOS devices with Xcode 5.0.1 +)
appium.io
I feel it's worthwhile to mention that appium doesn't require any modifications/additions to your android/iOS projects like a few of the other solutions I have found.
Please Calash https://github.com/calabash which is open and compatible with both iOS and Android.
I'm working with Android in Netbeans and trying to decide which testing framework to implement for my application (I've never done TDD in Android before). I've been using this thread to look into different resources. I'm mainly familiar with whitebox, so I feel that I would be more comfortable with the built-in Instrumentation Framework. However, this is the first time I've heard of blackbox testing (Robotium), but it looks like it would be really useful as well. Is it common practice to implement both whitebox and blackbox tests? Or is only one really necessary? If both, what things are best to be whitebox tested and which are best to be blackbox tested? Or is this a totally useless question as it's completely application dependent and I should just pick one and start messing with it?
[EDIT]: I also want to add that I don't have any experience with JUnit since a lot of explanations seem to assume a basic understanding of it.
The two options you suggest (Instrumentation and Robotium) are actually pretty much the same thing, the closes to whitebox (or true tdd/unit tests would probably be roboelectric).
Is it common to do both? Yes i think that is usually a good approach normally you want to test things as low down as possible and then have fewer of the big blackbox tests, android doesn't make this the easiest if you ask me though so you may have more luck with the instrumentation/robotium tests cases as these are fairly easy to understand and they match up with what you can see on the screen fairly easily.
With regards to junit, android is bundled with junit3 which has much less features/complexity as jUnit4, and for the most part the only thing you will need to know is to name your testmethods with the word "test" in front of it e.g. public void testXXX().
I have a fairly basic application. I'm loading it up on 2.1+ on emulators and 2.3.3 on an n1 and basically just making sure everything looks right and does what it's supposed to.
I was reading about activity testing on the dev site and it talks about creating test applications and such. I don't get the point of testing whether a textview = a resource string that you set it to or not.
How deep do you really need to test fairly basic apps before letting other people use them. Is there really anything that I won't catch using emulators and an n1 that test applications and such will?
The general rule of thumb is test anything that is complex enough to go wrong. If you don't have alternate resources, it is generally too simple to break.
Creating programs to test other programs? The only purpose I would see to this would be testing security of your app.
I've never had to create anything like this, and I publish many types of apps on the marketplace.
In my opinion, it is always best to do testing on an actual phone. Emulators arnt fully functional and possibly slower than the phone itself.