I have a fairly basic application. I'm loading it up on 2.1+ on emulators and 2.3.3 on an n1 and basically just making sure everything looks right and does what it's supposed to.
I was reading about activity testing on the dev site and it talks about creating test applications and such. I don't get the point of testing whether a textview = a resource string that you set it to or not.
How deep do you really need to test fairly basic apps before letting other people use them. Is there really anything that I won't catch using emulators and an n1 that test applications and such will?
The general rule of thumb is test anything that is complex enough to go wrong. If you don't have alternate resources, it is generally too simple to break.
Creating programs to test other programs? The only purpose I would see to this would be testing security of your app.
I've never had to create anything like this, and I publish many types of apps on the marketplace.
In my opinion, it is always best to do testing on an actual phone. Emulators arnt fully functional and possibly slower than the phone itself.
Related
I recently started looking into automating accessibility testing on Android. There isn't much information out there on the web.
Has anyone explored this or are currently doing this? If so, can you share your ideas/approach?
Seems like Android's uiautomator relies on Accessibility features working but it doesn't support testing Accessibility. If it relies on Accessibility features, does that mean that basic validation like accessible label exists, etc. can be done by just executing UI tests using uiautomator?
This is a new area for me so any information could be helpful.
Here's a great introduction to Accessibility testing in Android. It basically boils down to:
Manually test your app for visual issues with Accessibility Scanner
Turn on TalkBack and manually test your app to find hearing impaired issues
To find font scaling and layout issues, use Large Text
Definitely lint check, but make sure that 'Image without contentDescription' is set to Severity = Error
Any/all accessibility issues you find or that are reoccurring, write an Espresso test to fail when that accessibility issue is violated in the future
For automation, you'll also need to consider how to perform visual validation of certain screen artifacts and audio analysis if hearing impaired functionality is a requirement.
Also, I recommend watching this presentation from GTAC 2015 on accessibility testing for some great context on the topic.
For automated tests that check for accessibility, I'd very much recommend starting with issues that can be identified in elements that are shared across screens (menus, layouts, themes, custom controls). While they won't catch the one-off errors that will occasionally pop up, they'll address issues that happen everywhere in your app, a 'prioritize by volume' approach if you will.
Also, if your team uses Android Studio then you definitely want to push for the ability to write Espresso tests that reside with the code. QA are part of the development process, period. Getting access to a subfolder where your tests reside shouldn't be a problem unless there some legal bologna to deal with. For instance, split out the 'androidTest' folder as a submodule where you have pull/push rights as a tester, but only read rights to the rest of the app so you can compile and run yourself. If you're writing Appium tests, it may be harder to ask your dev team to run them as part of their own BVT/smoke test process during builds, but not unheard of.
As for visual analysis and audio injection/confirmation, these are advanced capabilities that you'll probably need to use some service or commercial tool for.
Best of luck!
I agree with Paul's answer in its entirety and he links to some extremely helpful resources (so please have a look at them!), but if all you're looking for is basic accessibility test coverage as you suggest (e.g. checking for accessible labels on all your components), your use case might be a good one for something like Continuum for Mobile, specifically the Android variant. You can then do more manual passes once you've found the more basic violations that can be detected using automated tools; as of right now, manual testing is always necessary for total compliance with accessibility standards, but something like this will get you closer.
I'm working with Android in Netbeans and trying to decide which testing framework to implement for my application (I've never done TDD in Android before). I've been using this thread to look into different resources. I'm mainly familiar with whitebox, so I feel that I would be more comfortable with the built-in Instrumentation Framework. However, this is the first time I've heard of blackbox testing (Robotium), but it looks like it would be really useful as well. Is it common practice to implement both whitebox and blackbox tests? Or is only one really necessary? If both, what things are best to be whitebox tested and which are best to be blackbox tested? Or is this a totally useless question as it's completely application dependent and I should just pick one and start messing with it?
[EDIT]: I also want to add that I don't have any experience with JUnit since a lot of explanations seem to assume a basic understanding of it.
The two options you suggest (Instrumentation and Robotium) are actually pretty much the same thing, the closes to whitebox (or true tdd/unit tests would probably be roboelectric).
Is it common to do both? Yes i think that is usually a good approach normally you want to test things as low down as possible and then have fewer of the big blackbox tests, android doesn't make this the easiest if you ask me though so you may have more luck with the instrumentation/robotium tests cases as these are fairly easy to understand and they match up with what you can see on the screen fairly easily.
With regards to junit, android is bundled with junit3 which has much less features/complexity as jUnit4, and for the most part the only thing you will need to know is to name your testmethods with the word "test" in front of it e.g. public void testXXX().
I am thinking of developing a program within ASP.NET. But that I could port easily to Apple, Android and Windows phones and tablets. I also want the program to be able to run on PCs (MAC´s?).
This seems to be fine. However, the normal GUI of Windows does not look at all like the GUI from e.g. iOS or Android. And this would be a big show stopper. It should be at least optimized for touch mode. So all on screen widgets should be a bit bigger.
Are there libraries, providing controls, which looks like normal ones on the operating system? It would be best if this works kind of automatically. So the program should find out where it runs and load the standard GUI best fitting to the device, where it is being started.
I could of course develop own controls, but this may be a bit too much of effort.
It would also be very nice if the library would handle different screens or modes like portrait and landscape. So that it may switch depending on the orientation of the tablet, or phone for instance.
I am searching for something like that, which provides a good result, without the programmer needing too much to do by himself.
I also like if one can recommend a library he is using already by a bigger program. From my own experience, you get to know the strengh and weakness of such libraries with the time passing by.
Best Regards.
I already discovered some tools like PhoneGap, Sencha, Mono. I do not know, if these are well suited for me.
I like to create an app most likely for all plattforms at the same time. Is this the case with e.g. PhoneGap?
I am not familar with JavaScript, or CSS or HTML development.
I am an experienced C# programmer and I am quite familar with Java. So VisualStudio is my daily working environment, and I am using Eclipse also a lot.
Mono:
I only read about it and found out this aspects:
cost 999$ to use it in Visual Studio. Hope that this price includes development for all plattforms and not just one. Did not yet check this.
The coding looks very good.
There are APIs for Android, iOS, and Windows Phone to setup controls. So these parts needs to be rewritten for any supported application. One may create a core library, which can be used for all components. This aspect is not so nice.
However, I would prefer a solution, where the GUI is even described on another level. And then it should be brought to the current plattform automatically, if not specific libraries are used. So HTML5 may be more flexible.
Best Regards,
Patrick
I have googled about testing uni and systematic testing. But I am confused what exactly is systematic testing? I’m developing a mobile app and like to know how I can approach this technique testing.
Any ideas? Thanks in advance.
Systematic testing is a much more exhaustive means of debugging software. The goal is to test the software against a variety inputs in order to find as many defects/errors as possible. The difficult part about systematic testing is usually coming up with the inputs/situations to test against (especially in the case of Android development, as a number of things can go wrong). Does your app work correctly after configuration changes? Do your EditText boxes specify the correct inputType? Systematic testing provides a means of uncover such errors, and often results in a more secure/robost application.
The Android team provides several tools that can automate this kind of testing (although obviously there is no guarantee that it will cover ALL of the cases). One such tool is monkeyrunner. Check out the Android developer's guide on testing for more information.
A systematic test is a test, in which
the setup is defined,
the inputs are chosen systematically,
the results are documented and evaluated by
criteria being defined prior to the test
A coworker and I were talking (after a fashion) about an article I read (HTC permission security risk). Basically, the argument came down to whether or not it was possible to log every action that an application was doing. Then someone (an abstract theroetical person) would go through and see if the app was doing what it was supposed to do and not trying to be all malicious like.
I have been programming in Android for a year now, and as far as I know if -- if -- that was possible, you would have to hack Dalvik and output what each process was doing. Even if you were to do that, I think it would be completely indecipherable because of the sheer amount of stuff each process was doing.
Can I get some input one way or the other? Is it completely impractical to even attempt to log what a foriegn application is doing?
I have been programming in Android for a year now, and as far as I know if -- if -- that was possible, you would have to hack Dalvik and output what each process was doing.
Not so much "hack Dalvik" but "hack the android.* class library, and perhaps a few other things (e.g., java.net).
Even if you were to do that, I think it would be completely indecipherable because of the sheer amount of stuff each process was doing.
You might be able to do some fancy pattern matching or something on the output -- given that you have determined patterns of inappropriate actions. Of course, there is also the small matter of having to manually test the app (to generate the output).
Is it completely impractical to even attempt to log what a foriegn application is doing?
From an SDK app? I damn well hope so.
From a device running a modded firmware with the aforementioned changes? I'd say it is impractical unless you have a fairly decent-sized development team, at which point it is merely expensive.
This is both possible and practical if you are compiling your own ROM. Android is based on Linux and I know several projects like this for Linux, like Linux Trace Toolkit. I also know of research into visualizing the results and detecting malicious apps from the results as well.
Another thing functionality like this is often used for is performance and reliability monitoring. You can read about the DTRACE functionality in Solaris to learn more about how this sort of stuff is used in business rather than academia.