We currently have a lot of cucumber tests that we use to test our mobile web-app on desktop browsers. However, i recently heard a rumor that it is possible to use some sort of remote web-driver to run our tests on actual devices instead of just on the desktop browser. However when I look up the android or iPhone web-drivers, they seem to be deprecated and recommend me using selendroid or appium, which neither seem to support browser-testing.
Today, we are only able to use our cucumber-tests for acceptance testing and as a first line of defense when doing regression tests. We still have to spend countless hours doing manual testing on around 10-20 different devices with different OS versions and browsers. Being able to run our cucumber-tests on the actual devices themselves could save us tons of time, as we'd be able to run these tests in parallel with the manual tests, and only focus our manual tests on detecting visual issues instead of also having to test the functionality of each page in our web-app.
Is it possible to run our cucumber tests on phones and tablets, and in that case how?
I realize this is an old post, but in case this helps folks, here's some basic information:
Take a look at Appium, which provides automation (with cucumber) for Android and iOS applications or mobile-web and for real devices or emulation/simulation.
There's a decent setup tutorial here, for a web-app. Setup for a mobile-website is similar but not as easy to find. Take a look here if you want Appium on SauceLabs.
Most sites you visit will say "it's easy!". I found that not to be the case; you'll see many requests for help and the specific devices and versions that you're working with all make a big difference. This is especially true if you are working with older devices, in which case you'll need to look at Selendroid for Android.
Assuming you're working with more than just a small group, you'll need to become familiar with configuring Cucumber to work with different devices and configurations: profiles may become your new best friend. There's also some profile help in this post.
Related
Hi need list of trusted website/app/platform where i can test in a reliable way((should be confident after using the platform)) on ios and android devices using selenium ,dont want to do test on actual physical devices need simulator for the following reasons firstly with the cost involved and secondly dont want to maintain the OS versions.
I came across this resource but have no idea about any of the listed ones there.
https://www.guru99.com/top-10-cross-browser-testing-tools.html
Is it possible to automatically test Android apps on different vendor Android versions?
I'd like to set up unit tests in Jenkins which run on different vendor ROMs, above all on Samsung. Is there a way to download those versions?
I found some sites to add to user defined sites, e.g:
http://developer.lgmobile.com/sdk/android/repository.xml
http://developer.samsung.com/sdk-manager/repository/Samsung-SDK.xml
http://developer.sonymobile.com/edk/android/repository.xml
http://dl.htcdev.com/sdk/addon.xml
I also could download stuff from there:
But how do I use those things for testing?
I am not a fan of long answers, but I failed keeping it short.
TL:DR:
Prefer functional testing over unit testing when testing compatibility with devices.
Testing on emulators does not provide you the same level of confidence than testing on real devices.
Probably the best option would be testing on a small set of virtual devices and then running the same tests on the real reference devices.
Long answer:
First of all, are you already using a solution to automate functional testing? Robotium is a great option for that and there are plenty of tutorials online. I cannot see the point on unit testing (instead of functional testing) on different devices, except for testing compatibility with different android api versions.
Secondly, testing on emulators does not provide you the same level of confidence than testing on real devices. Some things are not the same, for example there is no wifi manager on simulators. Given that, even if you create a virtual device with a custom vendor ROM for each device you are supporting, it does not substitute testing on the real devices.
Finally, in my opinion considering my lack of knowledge in your specific project, probably the best option would be testing on a small set of virtual devices that simulates the higher, medium and lower devices you are supporting as a first pass, and then running the same tests on real reference devices connected to your testing server as a definitive test.
And yes, I know not everyone are willing to buy a bunch of devices for testing purpose, but this is a trade of as you can only guarantee it really works as expected in a specific device when running on that device.
Hope it helps you. :)
Try with Xamarin TestCloud testing using real devices with 788 android devices and 21 OS versions at this moment.
So, you can use too Genymotion, a faster Android emulator, with a lot of devices, very useful Java API. The following articles explains what you need:
Use Genymotion with Jenkins for Android testing
Calabash, Genymotion and Jenkins
Check testdroid cloud.
From wikipedia:
Testdroid Cloud contains real Android and iOS powered devices, some of
which are available for users.Testdroid Cloud lets users run tests
simultaneously on cloud-based service
Is there any benchmark tests results comparing this two emulators ?
I know that Genymotion is seems to be more faster when loading and rebooting the emulator,
but is there any tests show result like, Camera, GPS, networking and etc. ?
Have a look here for some informal benchmark comparisons. However, take these with a grain of salt because benchmark scores will vary greatly depending on the host system.
As for the items you want benchmarks for, not sure that camera/gps can even be benchmarked (what would you test), and networking is perfect since it uses VirtualBox, which has always had great virtual network support.
As a graphics/game developer, I find Genymotion unparalleled for running OpenGL code - on Genymotion it runs much smoother framerate wise, than using an x86+HAXM+HW GPU android emulator. Also, the integration of Google Apps/Services in Genymotion is fantastic.
Compatibility is the only downside, since Genymotion only provides v4.1+ images there is no way to test compatibility on older android versions - but for these tests I just use the standard android emulator (which is fine since it is not as frequent as normal testing).
Update: As of January 2014, Genymotion supports Android v2.3.7 as well as v4.3 (with v4.4 in preview). This means it now covers all relevant android versions except Froyo (v2.2), which at this time accounts for only 1-2% of all devices :)
I've been investigating whether to use Genymotion, with a particular focus on running automated Robotium tests.
The Genymotion startup time is around 5 seconds (compared to at least 30 seconds for the Android emulator). Genymotion also runs Robotium tests visibly faster, and with a smoother UI.
However, be warned there are some noticable differences between Genymotion and the default Android emulators:
You cannot use the special 10.0.2.2 IP address with Genymotion to access localhost on the machine you are running the emulator from. This means for local tests you may need to explicitly configure the IP address of the machine the emulator should connect to.
The difference in emulator speeds will expose timing issues in your tests. The most common one is list views needing to load their data.
I think that some Robotium Solo utilities might behave slightly differently between Genymotion and the default Android emulators. On Genymotion, I vaguely recall some problems with searchText(String) scrolling my screen to the bottom, even when the text I am searching for is already in view. However I suspect this was just another symptom of timing issues than anything being fundamentally different though (in that particular case you could always call searchText(String, true) to prevent the scrolling).
Use Android Emulator
I've personally tried both and android emulator is way better.
You just have to configure the emulator right.
My use case is mostly to manual test things here and there and run automation tests. I've benchmarked them both, android emulator is faster when running tests.
Almost all the sensors are present in android emulator and you also get an awesome command line tool with it.
For more detail, check out my blog benchmarking this.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Android Emulator vs Real Device
Is it better to test your app on an emulator or on an actual Android device? As in to test your app while you are still developing it.
In addition, does testing on the emulator gives you more computing power than actual devices?
It is always better to test on a real device, as the emulator lacks certain inputs like sensors etc. (especially the older version of it). You will also have problem testing services like In app billing and LVM as the emulator doesn't come with Google Play, and the accounts that can be configured on it are limited.
The best solution is to have a few devices to test on, and use the emulator for testing different screen sizes and how they react to your layouts. There was a Google I/O talk this year that gave you a breakdown of what devices you needed to test on to make sure you have maximum compatibility. I don't remember which one though, sorry.
Concerning the development period,you should test the application on emulator first so that you can test it on multiple targets having different versions eg:2.2, 2.3, 4.0 to find out the loopholes and the resolution issues.
After completion of development you can test it on your actual device for hands on experience.
I would say it certainly depends on your needs. Since you are developing an application that will be consumed not by yourself actually, it will be consumed by your targeted audience and what will they have? They will definitely have real devices to use your application.
There are some features that are not available or really hard to configure on the emulator, like for example you are developing an app that uses a bluetooth feature.
The emulator is certainly designed for developers to test their apps in a virtual environment with limited functionality.
It really depends on the kind of app you are creating. If you are creating a game that requires the accelerometer then a real device would be beneficial. Yet if you are creating an app with a simple layout and you want to test for different screen dimensions and so on, the emulator is pretty good with this.
Yet you should always have a real device, since the emulator is very limiting.
Android simulator provides a platform very similar to the actual phone, so if your application is running in the simulator, it goes to 94% given the same result in the actual phone, except that sometimes there are modules that must be compulsorily test on a phone like the GMap, Push, screen size..
I am looking for a way (maybe impossible) to test my PhoneGap Android application on different phones / hardware. As I am able to obviously test it only on my phone, but some people are reporting that my app doesn't work / crashes / explodes on their phone. Is there any way how can I test my app on different phones without buying actual device ? Maybe virtually like running Android in an emulator or somehow ?
It is kind of frustrating knowing that the app works perfectly, but some people are not able to even start it...
EDIT: I am certainly testing the app in Android emulator, but that only helps to test on different Android versions. But I am looking for a way how to test in on different phones without actually buying it - if it is possible ?
There is a company called DeviceAnywhere: http://www.deviceanywhere.com/ that seems to be doing that.
But it may cost you. If you don't want to buy all those devices, somebody else will have to. Plus they have to build the infrastructure to support remote access. Does not sound like something you would get for free.
Like DeviceAnywhere, there is another company that will let you test your application on multiple hardware devices. It https://appthwack.com/ . They do have 100 mins of free plan to test out, so go ahead and try this out.
You can run it through the emulator that is provided with the Android development kit through Eclipse. But this will only help test the code on different versions of Android not necessarily on the different hardware so if it is really a hardware problem this will not help but if it is related to the different versions of android this should help you find the problem.
Here is information on the emulator and how to use it: http://developer.android.com/guide/developing/building/building-eclipse.html#RunningOnEmulatorEclipse