How do we generate reports in calabash android ? I looked in to "The cucumber recipe" book on how to generate report. But that didn't help me completely.
I want to generate report on why login failed :
For eg :
1) User enter valid credential and tries to Login
2) But, Login fails due to server_error/user_not_found or some exceptions . I get error xml/ statusCode from server when login fails.
How to generate report using that xml/status code in calabash ?
Please help !!
Use this command to execute your test:
calabash-android run <apkfile>.apk --format html --out reports.html
<%
date = Time.now.strftime('%m_%d-%H:%M:%S')
default_report = "./reports/output_#{date}.html"
default_rerun = "./rerun.txt"
%>
common: NO_STOP='1'
rerun_out: -f rerun -o <%= default_rerun %>
html_report: -f html -o <%= default_report %>
run: -p common -p rerun_out -p html_report
Here is my calabash-android profile config, you can put it in ./.config/cucumber.yml file under your calabash android project.
using the following command will export the test result into a rerun.txt file and a ./reports/output_#{date}.html file at the same time:
calabash-android run *.apk -p run
For HTML report
> calabash-android run <apkfile>.apk --format HTML --out
For JSON report
> calabash-android run <apkfile>.apk --format JSON --out
These are the otehr formats in cucumber,
Pretty
Progress
Usage
JUnit
Rerun
Read more from here: https://cucumber.io/docs/reference#reports
For HTML report use the following:
calabash-android run ${APKFile} -f html -o test-result.html
Related
I'm trying to generate code coverage report for my native components with AOSP source code using soong build system.
I have extended aosp vhal but unit test cases are same as in below link.
http://androidxref.com/8.1.0_r33/xref/hardware/interfaces/automotive/vehicle/2.0/default/tests/
Tried adding below to cc_test, cc_binary in Android.bp
native_coverage : true,
cflags: [
"-g",
"-O0",
"-fprofile-arcs",
"-ftest-coverage",
],
ldflags : [
"-fprofile-arcs",
"-ftest-coverage",
],
Native binary unit-tests-coverage is generated in out/target/product but I can't find gcno intermediates for this.
Running below command gives me *.gcda files for each test files.
adb shell \
GCOV_PREFIX=/data/local/tmp \
GCOV_PREFIX_STRIP=`echo $ANDROID_BUILD_TOP | grep -o / | wc -l` \
/data/local/tmp/unit-tests-coverage
I have tried below links but not sure how to proceed :(
http://logan.tw/posts/2015/04/28/check-code-coverage-with-clang-and-lcov/
https://android.googlesource.com/platform/bionic.git/+/master-soong
https://android.googlesource.com/platform/build/soong/+/581341d%5E%21/
https://android.googlesource.com/platform/external/e2fsprogs/+/fedfb27%5E%21/
https://android.googlesource.com/platform/development/+/master/scripts/acov#23
http://androidxref.com/9.0.0_r3/xref/bionic/README.md#293
I'm not sure if google's vts framework can be used here to generate native code coverage.
https://codelabs.developers.google.com/codelabs/android-vts-8/#6
"gcnodir" is generated but not sure how to make use of it.
/coverage/data/nativetest64/vehicle-unit-tests-coverage/unit-tests-coverage.gcnodir
Posting answer to my question for other users on SO.
Install coverage tool :
sudo apt-get install lcov (This should install lcov-1.12)
sudo apt-get install gcc-4.6 (Clang generates .gcno approximately equal to gcc 4.2 that aren't compatible
with gcov-4.8. Installing gcc-4.6 to get gcov-4.6 and invoking lcov with '--gcov-tool /usr/bin/gcov-4.6')
Download LLVM 3.8 for llvm-cov to work : http://releases.llvm.org/download.html
All native unit test cases i.e instrumented binary needs to be executed on target. To build and emit clang's instrumentation based profiling. Example: http://androidxref.com/9.0.0_r3/xref/hardware/interfaces/automotive/vehicle/2.0/default/Android.bp#82 (Renamed to vehicle-manager-unit-test for shorter name)
export NATIVE_COVERAGE=true
Add native_coverage: true to test module in Android.bp
Go to: module-name/test
Use mm or make command to build native binary
Ex: For hardware/interfaces/automotive/vehicle/2.0/default/tests/ :
mma or make vehicle-manager-unit-test -j32
Copy coverage enabled instrumented binary to target
adb push out/target/product/product_name/data/nativetest64/vendor/vehicle-manager-unit-test /data/nativetest64/vehicle-manager-unit-test
adb shell chmod +x /data/nativetest64/vehicle-manager-unit-test
Run test cases and generate .gcda files
adb shell \
GCOV_PREFIX=/data/local/tmp \
GCOV_PREFIX_STRIP=echo $ANDROID_BUILD_TOP | grep -o / | wc -l \
/data/nativetest64/vehicle-manager-unit-test
adb shell find -iname *.gcda
adb pull /data/local/tmp/proc/self/cwd/out/soong/.intermediates/hardware/interfaces/automotive/vehicle/2.0/default/vehicle-manager-unit-test/android_x86_64_silvermont_vendor_cov/obj/hardware/interfaces/automotive/vehicle/2.0/default/tests/ .(Destination folder)
Extract GCNO files from GCNODIR (archive file generated at
out/overage/data/nativetest64/vendor/vehicle-manager-unit-test ) to
same folder with GCDA files
llvm-cov gcov -f -b *.gcda (https://llvm.org/docs/CommandGuide/llvm-cov.html )
lcov --directory . --base-directory . --gcov-tool /usr/bin/gcov-4.6 --capture -o cov.info (http://ltp.sourceforge.net/coverage/lcov.php)
genhtml cov.info -o output
Here's the script which wraps all these commands:
https://gist.github.com/pankajgangwar/f070b8b54e83543f8e3638dcd2cae1b8
here it is explaned how to generate coverage reports, which do require GTest:
these flags enable the generation of test coverage: -fprofile-arcs -ftest-coverage
then one has to use gcov: gcov main_test.cpp
which's output then can be passed on to lcov (for reference):
$ lcov --coverage --directory . --output-file main_coverage.info
from which one can generate an lcov coverage report in HTML format:
$ genhtml main_coverage.info --output-directory out
these .gcda files in .gcnodir are gcov data files. gcov also has an output option --json-format, which might come handy when wanting to consume the coverage data with a web-service.
one of the examples from the links you've provided can be used to generate it for a whole project:
Collect the code coverage results:
$ lcov --directory . \
--base-directory . \
--gcov-tool gcov.sh \
--capture -o cov.info
Generate HTML files:
$ genhtml cov.info -o output
where the only difference is, that the wrapper script would need to be adjusted to call gcov. probably one could even omit the wrapper passed with option --gcov-tool, since it should be directly called.
since one can only prepare the coverage report by adding the compiler flags, the gcov and lcov commands should be setup as post-build script, so that they would automatically generate the report.
I'm trying to set up a basic BrowserStack sample App with Jenkins a Jenkinsfile for the Pipeline.
Reading the documentation of the BrowserStack plugin for Jenkins I came up with the following step:
// ...
stages {
stage('assemble') {
// Here the App gets assembled
}
stage('upload-to-browserstack'){
steps{
browserstack(credentialsId: '<credentials>'){
sh 'test -e app/build/outputs/apk/flavorProduction/debug/browserstack-sample-debug.apk && echo exists || echo does not exist'
browserstackAppUploader('app/build/outputs/apk/flavorProduction/debug/browserstack-sample-debug.apk'){
}
}
}
}
}
// ...
This results in the following error message:
[BrowserStack] Starting upload process.
[BrowserStack] Uploading app app/build/outputs/apk/flavorProduction/debug/browserstack-sample-debug.apk to Browserstack.
[BrowserStack] [ERROR] File not found at app/build/outputs/apk/flavorProduction/debug/browserstack-sample-debug.apk
[BrowserStack] ERROR : App Id empty. ABORTING!!!
I can see, that the file exists on my Jenkins instance. How can I upload the APK to the BrowserStack server or am I using a wrong syntax for this?
You can use any of the two scripts in your Jenkins build step in order to upload an app and use the hash ID from the response.
result="$(curl -u ":" -X POST "https://api-cloud.browserstack.com/app-automate/upload" -F "file=#/Users/Downloads/BStackSampleApp_1.ipa" | jq -r '.app_url')"
result="$(curl -u ":" -X POST "https://api-cloud.browserstack.com/app-automate/upload" -F "file=#/Users/Downloads/BStackSampleApp_1.ipa" | jq '.app_url' | tr -d \")"
You can then export the value of the result as the environment variable for the Jenkins configuration.
I try to install Fastlane in a pre build step on Buddybuild.
My pre build script looks like this:
#!/bin/bash
if ! which fastlane >/dev/null; then
echo "Installing fastlane this may need sudo"
sudo gem install fastlane
else
echo "Updating fastlane this may need sudo"
sudo gem update fastlane
fi
I get the following error:
ERROR: Error installing fastlane:
ERROR: Failed to build gem native extension.
/usr/bin/ruby2.1 mkrf_conf.rb .
Building native extensions. This could take a while...
/usr/lib/ruby/2.1.0/rubygems/ext/builder.rb:89:in `run': ERROR: Failed to build gem native extension. (Gem::Ext::BuildError) .
/usr/bin/ruby2.1 extconf.rb .
mkmf.rb can't find header files for ruby at
/usr/lib/ruby/include/ruby.h
extconf failed, exit code 1
How can I solve this?
Rest of the Log
Gem files will remain installed in /var/lib/gems/2.1.0/gems/unf-0.2.0.beta2/ext/gems/gems/unf_ext-0.0.7.4 for inspection.
Results logged to /var/lib/gems/2.1.0/gems/unf-0.2.0.beta2/ext/gems/extensions/x86_64-linux/2.1.0/unf_ext-0.0.7.4/gem_make.out
from /usr/lib/ruby/2.1.0/rubygems/ext/ext_conf_builder.rb:38:in block in build
from /usr/lib/ruby/2.1.0/tempfile.rb:324:in 'open'
from /usr/lib/ruby/2.1.0/rubygems/ext/ext_conf_builder.rb:17:in 'build'
from /usr/lib/ruby/2.1.0/rubygems/ext/builder.rb:161:in 'block (2 levels) in build_extension'
from /usr/lib/ruby/2.1.0/rubygems/ext/builder.rb:160:in 'chdir'
from /usr/lib/ruby/2.1.0/rubygems/ext/builder.rb:160:in 'block in build_extension'
from /usr/lib/ruby/2.1.0/rubygems/ext/builder.rb:159:in 'synchronize'
from /usr/lib/ruby/2.1.0/rubygems/ext/builder.rb:159:in `build_extension'
from /usr/lib/ruby/2.1.0/rubygems/ext/builder.rb:198:in 'block in build_extensions'
from /usr/lib/ruby/2.1.0/rubygems/ext/builder.rb:195:in 'each'
from /usr/lib/ruby/2.1.0/rubygems/ext/builder.rb:195:in 'build_extensions'
from /usr/lib/ruby/2.1.0/rubygems/installer.rb:677:in 'build_extensions'
from /usr/lib/ruby/2.1.0/rubygems/installer.rb:232:in 'install'
from /usr/lib/ruby/2.1.0/rubygems/resolver/specification.rb:78:in 'install'
from /usr/lib/ruby/2.1.0/rubygems/request_set.rb:206:in 'block in install_into'
from /usr/lib/ruby/2.1.0/rubygems/request_set.rb:198:in 'each'
from /usr/lib/ruby/2.1.0/rubygems/request_set.rb:198:in 'install_into'
from /usr/lib/ruby/2.1.0/rubygems/request_set.rb:119:in 'install'
from /usr/lib/ruby/2.1.0/rubygems/dependency_installer.rb:389:in 'install'
from mkrf_conf.rb:15:in ''
rake failed, exit code 1
Gem files will remain installed in /var/lib/gems/2.1.0/gems/unf-0.2.0.beta2 for inspection.
Results logged to /var/lib/gems/2.1.0/extensions/x86_64-linux/2.1.0/unf-0.2.0.beta2/gem_make.out
Change your script to:
if ! which fastlane >/dev/null; then
echo "Installing fastlane..."
echo password | sudo -S gem install fastlane
else
echo "Updating fastlane..."
echo password | sudo -S gem update fastlane
fi
Our recommendation (and, as it turns out, fastlane's) is to use bundler and a Gemfile to achieve that. It lets you use a specific version of fastlane and, as a bonus, don't require sudo.
Let me know if this helps!
Command is gradlew assembleQW >a.txt.
When success, the log is all in a.txt.
But when failure, the reason for the failure is printed in the console.
Using > only redirects STDOUT from the console to the file. If you want to also redirect STDERR you will need to use 2>.
See windows or unix
eg:
gradlew assemble 1> a.txt 2>&1
I was trying to run a a sample python program using monkey runner but unfortunately throwing an error of this type :
Can't open specified script file
Usage: monkeyrunner [options] SCRIPT_FILE
-s MonkeyServer IP Address.
-p MonkeyServer TCP Port.
-v MonkeyServer Logging level (ALL, FINEST, FINER, FINE, CONFIG, INFO, WARNING, SEVERE, OFF)
Exception in thread "main" java.lang.NullPointerException
so any one can guide me how to resolve this and how to use monkey runner to execute this type of things
I've found that making the path to the script absolute helped monkeyrunner.
I am invoking the runner from a Python script where a helper class has a startMonkey method:
def startMonkey(self, monkeyScript):
command = [ self.env + '\\tools\\monkeyrunner.bat' ]
command.append( os.path.abspath( monkeyScript ) )
return startProcess( command )
This seems to be working for me.
Script file should be a full path file name try below monkeyrunner c:\test_script\first.py
So go to the folder \ sdk \ tools
and press shift and right click to open command prompt and type monkeyrunner c:\test_script\python_file_name.py