I am working on automating monkey testing for Android apps.
My question is, if I obtain the seed when monkey causes a crash, what are the requirements to use the seed to reproduce the crash again across devices?
Like Android os version, exact emulator. Is it even possible?
I couldn't find the exact seed requirements for using another device.
Is it even possible?
That depends entirely on the nature of the crash and the steps the monkey took to trigger it on the original device.
For example, if launching Screen 3 crashes immediately, so long as the monkey does something that triggers your app to navigate to Screen 3, it will reproduce the crash. However, whether the original seed will lead to that outcome is impossible to know in the abstract. All you can do is try it and see what happens.
what are the requirements to use the seed to reproduce the crash again across devices?
If it is not absolutely identical, running absolutely identical code, the monkey actions based on that seed might not reproduce the crash. Then again, it might. We have no way to know.
In general, since the monkey is doing UI input, you would want the devices to be as close as possible from a UI standpoint, such as screen size and density. As a counter-example, if the devices have differing amounts of RAM or flash storage space, it is possible that those differences will not impact the monkey results.
Related
Does android somehow keep track of which apps are/were running and if so when it was in foreground or background, etc?
I already had a look at adb logcat, which gives a rather overwhelming amount of data and I'm not quite sure if I actually will be able to deduce what was started when, but that might be possible.
Seeing how that log disappears after a while and does not survive reboots, I was curious if I'm maybe there might other places where android keeps information about running apps. For instance a colleague mentioned that iOS running apps can usually be deduced from the automatic screenshots it takes for app switching.
I've developed an application which seems to work on most tablets/phones I've tested it on (S2/S3/S4/Xoom/some emulator configurations etc)
However, I've noticed a few complaints around a "Pantech Burst" - I can't seem to find any of these phones to pick one up (possibly it's specific to the US) and thought perhaps I could simulate one.
I know its 480 x 800 pixels, and has 1GB of memory
http://www.gsmarena.com/pantech_burst-4429.php
Is it possibly to simulate this kind of phone?
Or are some phones inherently different based on hardward that you could never simulate?
(I have a gut feeling it might be related to mp3's and Soundpools, but I'd rather prove it)
Short answer: no. In my experience if you have device-specific problems really the best way to debug them is to get your hands on the specific device.
Failing that, I can recommend integrating some kind of crash-reporting framework into your app, if you haven't already. These really help in capturing, tracking, and sending errors (with stacktraces) to you and have helped me fix problems on devices I can't get my hands on.
One I use is bugsense, there is also ACRA and others.
http://www.bugsense.com/
https://github.com/ACRA/acra
If you're having a problem on one particular device, then it is likely a hardware + software bug, and simply simulating the hardware configuration will not solve your problem.
That said, you can always duplicate the hardware by setting the RAM, screen size, storage etc. to its specifications. You probably won't get the same processing speed due to the fact that you're on an emulator.
If getting device is not really an option for you, you might want to consider using the Apkudo service, assuming they have the device your app is having trouble with.
You submit your app, and they run it on their set of devices using Monkey, returning to you a logcat and a stack trace when the application crashes on a particular device.
I have written an android app that generates a strange exception on the LG-Phone I have just started testing on. In DDMS the tag is "ISP_LOG_MW_DEBUG" and the text is "AMI_IsWindowSearch()." It is generated about 10 times per second while the app is running. It doesn't seem to interfere with the program itself, but I would rather not have this error. but I can not find any information on this on the web.
If a click the home button (the app continues running in the background) and start the app again (new UI, with the old services) the amount of these messages increases each time. So it is reasonable to assume that they are generated by something in the UI.
Has anybody experienced this error or has any idea how to avoid it?
Real devices can sometimes make a lot of noise. Sometimes the vendors just leave a lot of debugging turned on. If your app is not crashing, or showing performance issues, I would say it's safe to ignore.
Also, try with another device, or try with the emulator. If you don't see those errors with another device or the emulator, I think it's safe to say the vendor developers of the the LG device are just lazy and didn't turn off all their debugging like a good developer should. :D
db
I want to create a couple of functional tests for an Android application to run them on a continuous integration server. As far as I understand, there are two main approaches: monkeyrunner and test cases via instrumentation.
At the moment, I can't see any advantages of monkeyrunner, but I might be missing something. What is it good for?
I like to use MonkeyRunner because its really portable (Linux, Mac and Windows), easy to setup and can work easily across many different devices and emulators. Also, sometimes with instrumentation you get crashes that are unrelated to the app, but are rather because of the instrumentation implementation. With MonkeyRunner you will know what caused the crash.
From my experience, monkey testing is really good for detecting flaws of the application in terms of:
Memory leaks: sometimes it is impossible to track scenarios generating excessive memory usage (say basic fast rotation, subsequent button clicks etc.).
Monkey also helps identifying test cases; unintended, strange uses of the applications that lead eventually to crashes.
Using monkey tests you can also somehow mesure performance of the application, when used by "heavy" users.
I would say, that monkey testing does not stand in opposition to unit/instrumenation testing, but it is yet another way to test, that your application is working as intended.
Of course it also depends on the software is about to be tested, but in my opinion it is not always that easy to determine what happens if your button is clicked, then 9px above the button is touched and eventually a phone activity is run. :) That what monkey tests are for...
I started developing android applications. And am testing with the android emulator. Do I really need android phone before releasing it for public usage?
Short answer No. You can test and build a android application package with the SDK and an emulator. But I would say there are usually many things which it would be wise to test on a device.
Personally I have noticed that the emulator does not give a good indication of response times for UI controls. It is usually necessary to move functionality which has long processing times into background threads to maintain user interactivity without the 'force close' pop-up. Testing the effectiveness of your UI responsiveness must be done on a phone to be meaningful.
Network connectivity is another aspect which can be vastly different on a phone, 3G or wifi.
Device sizes and Android platform versions can be tested effectively on the emulator.
Some phone allow hot-swapping of the SD card (replacing the SDcard without turning off the phone). I am not sure how to replicate this on the emulator.
There may be many more things which may only become apparent when using your application on a real device. I would strongly suggest to always test under real conditions when feasible for any commercial project.
From a technical perspective there's no reason why you can't develop purely on the emulator. You're not going to be able to test on every available device, so there's always going to be possibility of device specific bugs that you've missed.
However, I'd strongly recommend getting an actual phone to test your application on.
For me the biggest difference between an actual device and the emulator is the difference between using the interface with your fingers and using a mouse. Interactions which make sense in the emulator sometimes don't work as well when you start using touch on the screen. So if you develop purely on an emulator you'll won't lots of little improvements to your UI that would obvious when you used your app on a phone.
You can't feel a real app in your hands until you have a real phone. (I'm telling you as an Android developer)
So, developing w/o real phone is possible, but real phone gives you a lot more experience, fun & usefulness.
It depends on what type of application you're developing, for serious ones you need at least one device to test it on. For complex applications you would need a range of devices, for example with or without hardware keyboard, different navigational button etc. For basic, simple applications you'll probably do fine with just the emulator.
I would imagine with games you would definitely need to test on real devices.
Thanks to you all. I am going to get HTC Legend and test it, so that I can hope that my apps can be used by others :)
You guys suggest me HTC Desire or HTC Legend?