Is it possible to automatically test Android apps on different vendor Android versions?
I'd like to set up unit tests in Jenkins which run on different vendor ROMs, above all on Samsung. Is there a way to download those versions?
I found some sites to add to user defined sites, e.g:
http://developer.lgmobile.com/sdk/android/repository.xml
http://developer.samsung.com/sdk-manager/repository/Samsung-SDK.xml
http://developer.sonymobile.com/edk/android/repository.xml
http://dl.htcdev.com/sdk/addon.xml
I also could download stuff from there:
But how do I use those things for testing?
I am not a fan of long answers, but I failed keeping it short.
TL:DR:
Prefer functional testing over unit testing when testing compatibility with devices.
Testing on emulators does not provide you the same level of confidence than testing on real devices.
Probably the best option would be testing on a small set of virtual devices and then running the same tests on the real reference devices.
Long answer:
First of all, are you already using a solution to automate functional testing? Robotium is a great option for that and there are plenty of tutorials online. I cannot see the point on unit testing (instead of functional testing) on different devices, except for testing compatibility with different android api versions.
Secondly, testing on emulators does not provide you the same level of confidence than testing on real devices. Some things are not the same, for example there is no wifi manager on simulators. Given that, even if you create a virtual device with a custom vendor ROM for each device you are supporting, it does not substitute testing on the real devices.
Finally, in my opinion considering my lack of knowledge in your specific project, probably the best option would be testing on a small set of virtual devices that simulates the higher, medium and lower devices you are supporting as a first pass, and then running the same tests on real reference devices connected to your testing server as a definitive test.
And yes, I know not everyone are willing to buy a bunch of devices for testing purpose, but this is a trade of as you can only guarantee it really works as expected in a specific device when running on that device.
Hope it helps you. :)
Try with Xamarin TestCloud testing using real devices with 788 android devices and 21 OS versions at this moment.
So, you can use too Genymotion, a faster Android emulator, with a lot of devices, very useful Java API. The following articles explains what you need:
Use Genymotion with Jenkins for Android testing
Calabash, Genymotion and Jenkins
Check testdroid cloud.
From wikipedia:
Testdroid Cloud contains real Android and iOS powered devices, some of
which are available for users.Testdroid Cloud lets users run tests
simultaneously on cloud-based service
Related
I was asked to test some Android apps, but I do not have any andoid device currently. There are a loads of different Android emulators and/or dev frameworks, so I was wondering if tests performed on such virtual system is reliable in terms of later work on actual devices?
Can I use these emulators for test purposes without testing on any actual device? What are strategies for such tests?
Yes, you can use emulators.
Of course, Eclipse and Android Studio has there own, but I would recommend you Genymotion as faster one.
I want to get into developing apps for Android because of its large userbase. However, at the moment I do not have a smartphone. I'm also not too keen on getting an Android phone in the future.
What problems could I encounter when developing apps for Android solely using an emulator? I could use my parents' Android phones for the occasional testing, but will an emulator prove sufficient otherwise? Furthermore, is it possible to set up an Android Virtual Device with low-end hardware on my computer to test out apps with low-end phones?
note0: I do not plan on developing apps which require use of the phone's sensors (such as gyros, GPS, etc.) I only plan on developing apps which require internet access for API calls and perhaps microphone usage (I can't give away too many details right now ;) )
note1: I know this question has been asked before on stackoverflow (albeit 4 years ago). However, I feel it is important to revisit the question because emulators, hardware, and Android itself have changed enormously in the last few years.
edit: My main issues are problems like the app looking weird on the phone despite it looking fine on the emulator or working slowly on the phone but working fine on the emulator. Will every app which works fine on the emulator work fine on the phone as well with a similar hardware configuration as the emulator? I'm in particular targetting devices with Android 4.x/5.x.
Without a doubt, the biggest problem is that many functions like push
notifications, maps, geo-location, in-app vending etc. cannot be
tested with an emulator. The reason is that all these functions are
present in the Google Play Services library which is not present on
emulators.
Another important reason for testing on hardware devices is that the
OS implementation is changed by manufacturers like Samsung, HTC and
others, and therefore, certain problems & exceptions are not observed
till a real device is used.
But the most important reason has to be that running apps on an
actual phone is a hell of a lot more fun that it is with the painfully
slow and retarded emulators! One of the most important things about
Android is that anybody with an Android phone can easily explore
computer science concepts by running simple programs on their phone.
It is incredibly easy, even for a novice, to just fire up their
Android device and start exploring the world of programming! Till just a few years back, it was incredibly difficult to debug an app on a Symbian or Palm or WinCE device. iOS is still as retarded when it comes to running the simplest apps, you have to create a provisioning profile to run a Hello World app on an iPhone!
EDIT:
If your processor supports Intel virtualization, then the emulator can be made to run as fast as a real device. You need to check that out. If your PC is low-end in the sense that it does not support such virtualization, then running and testing Android apps will be a painful, tedious experience that will suck all the fun out of the development process.
Emulators do have one significant advantage over real devices - an emulator is able to access web services on a localhost. If you are still in development phase on your local server and have not yet shifted to a publicly hosted environment, the emulator can easily access web services on a local server. All you have to do is put your computer on the same network as the local server and you're good to go. A real device cannot access a localhost; its impossible, as far as I know.
The Android emulator is a true emulator - it emulates sensors as well, so in fact you can test sensor programming (with some limitations) on an emulator. It cannot do things like GPS, maps etc. because those features have been migrated to the Google Play Services library, and they require a valid Google account to be used, which is why that library is not present on an emulator. If you only plan to work with web services, APIs', GUIs' then I suppose an emulator is more than sufficient. Unlike the iOS simulator, an Android emulator generally exhibits the WYSIWYG behavior, so your GUI will not differ on actual devices, although you do need to test your GUI on multiple AVDs' with different screen form factors.
We currently have a lot of cucumber tests that we use to test our mobile web-app on desktop browsers. However, i recently heard a rumor that it is possible to use some sort of remote web-driver to run our tests on actual devices instead of just on the desktop browser. However when I look up the android or iPhone web-drivers, they seem to be deprecated and recommend me using selendroid or appium, which neither seem to support browser-testing.
Today, we are only able to use our cucumber-tests for acceptance testing and as a first line of defense when doing regression tests. We still have to spend countless hours doing manual testing on around 10-20 different devices with different OS versions and browsers. Being able to run our cucumber-tests on the actual devices themselves could save us tons of time, as we'd be able to run these tests in parallel with the manual tests, and only focus our manual tests on detecting visual issues instead of also having to test the functionality of each page in our web-app.
Is it possible to run our cucumber tests on phones and tablets, and in that case how?
I realize this is an old post, but in case this helps folks, here's some basic information:
Take a look at Appium, which provides automation (with cucumber) for Android and iOS applications or mobile-web and for real devices or emulation/simulation.
There's a decent setup tutorial here, for a web-app. Setup for a mobile-website is similar but not as easy to find. Take a look here if you want Appium on SauceLabs.
Most sites you visit will say "it's easy!". I found that not to be the case; you'll see many requests for help and the specific devices and versions that you're working with all make a big difference. This is especially true if you are working with older devices, in which case you'll need to look at Selendroid for Android.
Assuming you're working with more than just a small group, you'll need to become familiar with configuring Cucumber to work with different devices and configurations: profiles may become your new best friend. There's also some profile help in this post.
I am trying to test my app and I have lots of problems with Android emulators. Finally, I got one emulator running using Android 2.2 . So I used this one and my app works fine.
I also tested my app using real device, Nexus 7 , Android OS 4.2. App works fine in both versions.
It is safe to consider my app will run fine for OS versions within 4.2 - and 2.2 band? The reason I am asking is because using emulator makes my computer very slow and it takes forever.
In the best case: yes. In worst case: no. It completely depends on the parts of the Android API you're using in your app. For example, private APIs might very well break or change. You should be fine with public APIs, but there's always the chance of a bug in one of the OS versions...
If you want to test your app on various devices and Android versions, you should give apkudo.com and testdroid.com a try.
You can also try https://www.elusivestars.com it allows you to select devices based on specs like OS version, brand... and offering 5 free reviews
I have heard good things about testflightapp, they used to be iOS only and now doing Android.
Something like http://www.perfectomobile.com/ might also be a good way to go.
Here is a list of tools, including test tools
http://www.apptamin.com/blog/app-developer-tools/
Cloud based real device testing is typically either devices you can access and manipulate/automate or users with devices. Personally I think starting with automated testing then finishing off with real user (usability testing) may give you good coverage.
I am the Founder of elusivestars
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Android Emulator vs Real Device
Is it better to test your app on an emulator or on an actual Android device? As in to test your app while you are still developing it.
In addition, does testing on the emulator gives you more computing power than actual devices?
It is always better to test on a real device, as the emulator lacks certain inputs like sensors etc. (especially the older version of it). You will also have problem testing services like In app billing and LVM as the emulator doesn't come with Google Play, and the accounts that can be configured on it are limited.
The best solution is to have a few devices to test on, and use the emulator for testing different screen sizes and how they react to your layouts. There was a Google I/O talk this year that gave you a breakdown of what devices you needed to test on to make sure you have maximum compatibility. I don't remember which one though, sorry.
Concerning the development period,you should test the application on emulator first so that you can test it on multiple targets having different versions eg:2.2, 2.3, 4.0 to find out the loopholes and the resolution issues.
After completion of development you can test it on your actual device for hands on experience.
I would say it certainly depends on your needs. Since you are developing an application that will be consumed not by yourself actually, it will be consumed by your targeted audience and what will they have? They will definitely have real devices to use your application.
There are some features that are not available or really hard to configure on the emulator, like for example you are developing an app that uses a bluetooth feature.
The emulator is certainly designed for developers to test their apps in a virtual environment with limited functionality.
It really depends on the kind of app you are creating. If you are creating a game that requires the accelerometer then a real device would be beneficial. Yet if you are creating an app with a simple layout and you want to test for different screen dimensions and so on, the emulator is pretty good with this.
Yet you should always have a real device, since the emulator is very limiting.
Android simulator provides a platform very similar to the actual phone, so if your application is running in the simulator, it goes to 94% given the same result in the actual phone, except that sometimes there are modules that must be compulsorily test on a phone like the GMap, Push, screen size..