My team is responsible for maintaining a C++ library built, amongst other targets, for Android using NDK. We have unit tests that are based on googletest/gtest and are built as simple executables. Our automated build system currently runs these tests on a physical device by running adb commands.
It has been suggested that we investigate using Android Device Farm, at least as an option - we occasionally get trouble with specific versions of Android, and it would allow us to test more widely. However the examples I am seeing reflect running proper Android apps via a generated apk - they would need an App entry point.
I am thinking this would need a dummy app to call the unit test code via an JNI entry point and then exit, or is that over complicating something that is better supported and I've not noticed.
Has anybody tried doing this?
Related
While developing an app I more and more stumble upon the fact that I need to run interaction tests where two emulators are running the app to ensure the entire integration is working by testing the interaction between those (and not mock one of the devices)
During research on the topic (which is quite hard to search for because I do not want to run the same tests on multiple devices) I found a couple of solutions, however none of them seem to integrate very well into an Android Studio workflow. Some tools I've found:
UiAutomator
UI/Application Exerciser Monkey
Appmium
Some blend of adb, ruby, python etc.
Some projects on github
My expectation may be off but is there no framework or tool with a similar level of integration like running an instrumented tests that allows me to connect to several devices and perform interactions on them together?
Some of those approaches above require out-of-the-tool tinkering with adb or require a lot of environment to be set up.
Good answer to this question here:
https://stackoverflow.com/a/49647422/14338677
In short:
1.Create communication proxy that will be between sender and receiver
2.Add build type autoresponder, and communication proxy code get empty if (!BuildConfig.AUTORESPONDER)
3.Install autoresponder app to emulator or device when you will be testing
4.Run another test app with Espresso that can communicate with autoresponder app
5.Use Thread.sleep(...) to wait response of autoresponder app
I am currently testing an app for Android phones using the Espresso library with local devices as well as the Google Firebase Test Lab. For a good portion of these tests to be functional the phone running the test needs to connect to another phone or at least receive feedback from another device. I have been trying to do some research but really can't find anything on what I'm looking to do, seeing that it isn't a common practice.
Is there away to run Espresso or any small script directly on an Android device? I just need a script to sit idle or looping until it receives the information from the tested device, accepts it so the test can continue. I am not trying to alter any direct code in my app to auto accept things, I want to keep the app itself as untouched as possible. And I could manually do it on a small scale, but with Firebase I would like to test lots of devices at once or will run tests daily, sometimes when I'm not there for feedback. I want to automate this process as much as possible, so anything that could do that would help.
I've automated this process completely. The best thing is that you don't have to keep busy of your Android Studio and resources. Everything is done remotely.
You can use this plugin to setup Firebase test lab:
https://github.com/piotrmadry/firebase-test-lab-gradle-plugin
Hopefully you have CI (Continuous Integration) for your project.
So after your setup for test lab (you can define as many devices as you want)
You can create 2 parallel jobs on you CI:
First - it would build your project in normal way
Second - would run android task created by plugin ./gradlew uploadTestLab. After this job you will be able to look at results in Firebase Console.
I have recently been looking in to Espresso, which looks for me very promising.
I am recently working in an Android test project, and we are recently using Espresso as we see some benefits with that. Unfortunately the setup in the company I work for is such that I do not have access to the entire code for the Application we are testing.
So how I can create a test using android studio?
You can use UIAutomator to get the ids of various views.
See here
With view ids you can perform assertions/click and other actions on them
An example
onView(withId(R.id.play_search_container)).perform(click());
Hence you can go through the whole UI flow by clicking on various elements.
if you want to write a test outside of application's code, I mean: pure black-boxing, Espresso would be a pretty hard to implement as it needs a bit of application architecture (the same thing is with Robotium or `UiAutomator): names of Activities, views texts, contentDescriptions or idies.
You can use instead of Espresso:
UI/Application Exerciser Monkey
The Monkey is a program that runs on your emulator or device and
generates pseudo-random streams of user events such as clicks,
touches, or gestures, as well as a number of system-level events. You
can use the Monkey to stress-test applications that you are
developing, in a random yet repeatable manner.
monkeyrunner(requires Python language programming basics)
The monkeyrunner tool provides an API for writing programs that
control an Android device or emulator from outside of Android code.
With monkeyrunner, you can write a Python program that installs an
Android application or test package, runs it, sends keystrokes to it,
takes screenshots of its user interface, and stores screenshots on the
workstation.
The monkeyrunner tool is primarily designed to test
applications and devices at the functional/framework level and for
running unit test suites, but you are free to use it for other
purposes.
Calabash
Calabash is a framework that enables Automated UI Acceptance Tests
written in Cucumber to be run on iOS and Android applications.
Calabash works by enabling automatic UI interactions within an
application such as pressing buttons, entering text, validating
responses, etc.
From: Introduction to
Calabash
Hope it will help
I'm final testing a project I've been working on. So far, I've tested the app by installing on various virtual & real devices, and user testing all possible outcomes of the functionality.
Going through the release preparation doc, android test project is the method used in the guide.
I've started to write out some test cases but I can't see the benefit, or additional options from using a test project, that I have not already covered in device/user testing.
Am i wrong to just stick with the device/user tests, or is the test project method just a coded version of the tests I've already completed?
The big advantage of unit-tests is that you can run them always after you did some changes in your code.
This way, it is a lot easier to find a bug which breaks already working functionality.
I'm currently looking for a solution to test Android Apps automatically, native Android apps as well as hybrid Cordova/Phonegap apps.
The UI Automator from Google looks pretty fine, but can not deal with WebViews, which makes it unusable for me.
Then I found the promising automation framework Appium based on Selenium, which should also be able to deal with hybrid apps, but I couldn't get it to work with WebViews, it seems to not be implemented fully yet, as also several posts prove.
After trying Appium I found a hint for Selendroid, that also should work for hybrid apps, but it seems pretty much buggy and keeps crashing my whole computer all the time.
So, does anyone here know a working solution for testing Android Apps with WebViews? I'm working on Windows 7. Thanks a lot in advance for any help.
In first case we need to know what kind of tests you like to drive. I think you like to run functional tests on the UI? Thats what i can think about after reading your question.
So lets split this by the given environment:
native iOS
unit tests -> do it in xCode. http://www.apple.com
functionality tests -> http://appium.io/
native Android
unit tests -> JUnit, http://developer.android.com/tools/testing/testing_android.html
functionality tests -> http://selendroid.io/
native Web like phonegap/cordova Apps
unit tests -> jsUnit, http://jsunit.berlios.de/
QA Testing and Checkstyle -> jsHint/jsLint, http://www.jshint.com/ & http://www.jslint.com/
functionality tests -> http://docs.seleniumhq.org/ & https://npmjs.org/package/ripple-
functionality tests (E2E) -> https://angular.github.io/protractor/#/ or https://github.com/jasmine/jasmine
Here is a article about JavaScript QA / QS
native X - The out of the Box solution for all Platforms
Here we are. There is none atm. But .. footage is that one here:
Gappium based on appium uses json for writing your tests. As i get the thread right, it was testet with Web/iOS for this moment. Android/Win seems not be integrated for this moment:
https://github.com/appium/io.appium.gappium.sampleapp
Note for phonegap & apache cordova functional tests:
Its possible to use ripple and selenium for testing hybrid apps. Keep in mind that some native function/plugins (hehe native functions on hybrid apps ... =) cant be tested or need to be mocked on a unknown way. Thats the footage for this moment.
Maybe usefull for you Continous Integration Server:
https://github.com/linslin/PHPUnit-selenium-HTMLGallery
Generally, testing hybrid apps is quite a challenge with API-based approaches. It can be done, but typically you need to integrate the UI technologies (e.g. WebView, Native, OpenGL, ...) with several APIs or some kind of wrapper (like Selendroid).
Here at TestObject (I'm one of the co-founders) we've created an Image-based UI testing approach specifically for cases like hybrid apps, interaction between apps, and so forth. This black-box approach is solely based on capturing the screen - we don't care what happens underneath.
Feel free to give it a try (http://testobject.com/) with your hybrid Cordova/Phonegap app and let me know how it worked out in comparison to UI Automator, Selendroid, Appium. Let us know if you stumble over any issues.
I faced this challenge with our own Cordova app ( sourceforge.net/p/ustadmobil/codehtml5cordova/ci/master/tree/ ) ; and I needed a way to integrate this with Jenkins CI. We have Cordova specific code (e.g. filesystem access) so just using PhantomJS and Grunt is not an option.
Essentially what I did was to launch the app with the test page as index.html in the emulator, run an http server, and then have the test page pass the results back using AJAX.
Script launches an emulator with the package
$ /opt/adt/sdk/tools/emulator-x86 -avd $AVDNAME -qemu -m 2047 &
$ EMULATEPID=$!
$ adb wait-for-device #wait for device to start
$ adb shell input keyevent 82 #unlock screen
Make a test page using qunit, which when finished will communicate the test results with an AJAX call.
Make a basic nodejs server script that will listen for that ajax call and save the results to a file. A bash script can then read that file and determine if the tests passed or failed : see http://sourceforge.net/p/ustadmobil/codehtml5cordova/ci/master/tree/setup/node-qunit-server/node-qunit-server.js
In the build script wait for the result file to come, then kill the emulator and the NodeJS server - see http://sourceforge.net/p/ustadmobil/codehtml5cordova/ci/master/tree/setup/android/unit-test-setup-android.sh
Also worth noting: this approach works for the cross platform nature of Cordova: it should be possible to take the same approach running iOS, Windows Phone emulators etc (I also use this test method against our NodeWebKit desktop version which shares most of the code).
Hope this helps,
-Mike