Develop Reference

Jenkins pipeline job executing commands in tmp folder

MacOS Monterey version 12.4
I'm trying to run a simple pipeline script on my jenkins job
pipeline {
agent any
stages {
stage('Build') {
steps {
sh('emulator -list-avds')
}
}
}
But it throws an error:
/Users/<my_username>/.jenkins/workspace/<my_job_name>#tmp/durable-22217e91/script.sh: line 1: emulator: command not found
My question is: why is it executing commands in the tmp folder? Anything "emulator" related does work when I run commands via terminal.
Following this answer, I've confirmed I'm in the correct dir
Why Jenkins mounts a temporary volume in addition to the workspace?

You are getting this error because the emulator executable is not set in the PATH. Try something like the below.
Try setting your PATH variable to add emulator executable.
environment {
PATH = "/PATH_EMULATOR/bin:${env.PATH}"
}
or something like the below.
withEnv(["PATH+EMULATOR=/PATH_EMULATOR/bin"]) {
sh('emulator -list-avds')
}
or you can also use the full qualified path to the executable
sh('/PATH_TO_EMULATOR/bin/emulator -list-avds')

Shell script in Gradle build: java.io.IOException

I have the following lines in my build.gradle file of my Android project:
"./prebuild.sh".execute()
But during the build I get this error:
java.io.IOException: Cannot run program "./prebuild.sh": error=2, No such file or directory
The prebuild.sh script is in the root directory of the app and executable.
What's weird is that this exact build works for everyone on the team, just not on my machine (M1). I also remember that this used to work months ago.
This happens on a fresh clone of the repository and a fresh install of Android Studio.
I think I've narrowed it down to a problem with the working directory. If I try to print it like this:
println new File(".").absolutePath
I get the following:
/Users/ale/.gradle/daemon/6.5/.
Which is obviously not my project directory.
Any hints on what I could do to fix it?

Assuming a functional shell prompt; pass the absolute path instead of . current working directory:
if(rootProject.file('prebuild.sh').exists()) {
commandLine 'sh', rootProject.file('prebuild.sh').absolutePath
} else {
println "missing: prebuild.sh"
}
Or how you start it as process, one can also pass the current working directory as second argument:
def proc = "./prebuild.sh".execute([], rootProject.absolutePath)
proc.waitForProcessOutput(System.out, System.err)
I'd run cd first, then pwd should return the expected value:
def proc = "cd ${rootProject.absolutePath} && pwd".execute()
...

Check if the file has DOS line endings (\r\n). This can lead to a confusing "no such file or directory", because it searches for a file called /bin/sh\r (ending with an actual carriage return), which does not exist.

Install APK using root, handling new limitations of "/data/local/tmp/" folder

Background
So far, I was able to install APK files using root (within the app), via this code:
pm install -t -f fullPathToApkFile
and if I want to (try to) install to sd-card :
pm install -t -s fullPathToApkFile
The problem
Recently, not sure from which Android version (issue exists on Android P beta, at least), the above method fails, showing me this message:
avc: denied { read } for scontext=u:r:system_server:s0 tcontext=u:object_r:sdcardfs:s0 tclass=file permissive=0
System server has no access to read file context u:object_r:sdcardfs:s0 (from path /storage/emulated/0/Download/FDroid.apk, context u:r:system_server:s0)
Error: Unable to open file: /storage/emulated/0/Download/FDroid.apk
Consider using a file under /data/local/tmp/
Error: Can't open file: /storage/emulated/0/Download/FDroid.apk
Exception occurred while executing:
java.lang.IllegalArgumentException: Error: Can't open file: /storage/emulated/0/Download/FDroid.apk
at com.android.server.pm.PackageManagerShellCommand.setParamsSize(PackageManagerShellCommand.java:306)
at com.android.server.pm.PackageManagerShellCommand.runInstall(PackageManagerShellCommand.java:884)
at com.android.server.pm.PackageManagerShellCommand.onCommand(PackageManagerShellCommand.java:138)
at android.os.ShellCommand.exec(ShellCommand.java:103)
at com.android.server.pm.PackageManagerService.onShellCommand(PackageManagerService.java:21125)
at android.os.Binder.shellCommand(Binder.java:634)
at android.os.Binder.onTransact(Binder.java:532)
at android.content.pm.IPackageManager$Stub.onTransact(IPackageManager.java:2806)
at com.android.server.pm.PackageManagerService.onTransact(PackageManagerService.java:3841)
at android.os.Binder.execTransact(Binder.java:731)
This seems to also affect popular apps such as "Titanium backup (pro)", which fails to restore apps.
What I've tried
Looking at what's written, it appears it lacks permission to install APK files that are not in /data/local/tmp/.
So I tried the next things, to see if I can overcome it:
set the access to the file (chmod 777) - didn't help.
grant permissions to my app, of both storage and REQUEST_INSTALL_PACKAGES (using ACTION_MANAGE_UNKNOWN_APP_SOURCES Intent) - didn't help.
create a symlink to the file, so that it will be inside the /data/local/tmp/, using official API:
Os.symlink(fullPathToApkFile, symLinkFilePath)
This didn't do anything.
create a symlink using this :
ln -sf $fullPathToApkFile $symLinkFilePath
This partially worked. The file is there, as I can see it in Total Commander app, but when I try to check if it exists there, and when I try to install the APK from there, it fails.
Copy/move (using cp or mv) the file to the /data/local/tmp/ path, and then install from there. This worked, but it has disadvantages: moving is risky because it temporarily hides the original file, and it changes the timestamp of the original file. Copying is bad because of using extra space just for installing (even temporarily) and because it wastes time in doing so.
Copy the APK file, telling it to avoid actual copy (meaning hard link), using this command (taken from here) :
cp -p -r -l $fullPathToApkFile $tempFileParentPath"
This didn't work. It got me this error:
cp: /data/local/tmp/test.apk: Cross-device link
Checking what happens in other cases of installing apps. When you install via via the IDE, it actually does create the APK file in this special path, but if you install via the Play Store, simple APK install (via Intent) or adb (via PC), it doesn't.
Wrote about this here too: https://issuetracker.google.com/issues/80270303
The questions
Is there any way to overcome the disadvantages of installing the APK using root on this special path? Maybe even avoid handling this path at all?
Why does the OS suddenly require to use this path? Why not use the original path instead, just like in the other methods of installing apps? What do the other methods of installing apps do, that somehow avoids using the spacial path?

One solution, in case you don't mind the moving procedure, is to also save&restore the timestamp of the original file, as such:
val tempFileParentPath = "/data/local/tmp/"
val tempFilePath = tempFileParentPath + File(fullPathToApkFile).name
val apkTimestampTempFile = File(context.cacheDir, "apkTimestamp")
apkTimestampTempFile.delete()
apkTimestampTempFile.mkdirs()
apkTimestampTempFile.createNewFile()
root.runCommands("touch -r $fullPathToApkFile ${apkTimestampTempFile.absolutePath}")
root.runCommands("mv $fullPathToApkFile $tempFileParentPath")
root.runCommands("pm install -t -f $tempFilePath")
root.runCommands("mv $tempFilePath $fullPathToApkFile")
root.runCommands("touch -r ${apkTimestampTempFile.absolutePath} $fullPathToApkFile")
apkTimestampTempFile.delete()
It's still a bit dangerous, but better than copying files...
EDIT: Google has shown me a nice workaround for this (here) :
We don't support installation of APKs from random directories on the device. They either need to be installed directly from the host using 'adb install' or you have to stream the contents to install --
$ cat foo.apk | pm install -S APK_SIZE
While I think this is incorrect that they don't support installing of APK files from random paths (always worked before), the workaround does seem to work. All I needed to change in the code of installing an APK file is as such:
val length = File(fullPathToApkFile ).length()
commands.add("cat $fullPathToApkFile | pm install -S $length")
Thing is, now I have some other questions about it :
Does this workaround avoid the moving/copying of the APK into storage, and without affecting the original file ? - seems it does
Will this support any APK file, even large ones? - seems it succeeds in doing it for an APK that takes 433MB, so I think it's safe to use for all sizes.
This is needed only from Android P, right? - so far seems so.
Why does it need the file size as a parameter ? - No idea, but if I remove it, it won't work

Thanks for the answers! I looked everywhere else as well to get a whole setup for OTA to work for Android 10 and so on. It 100% works on Samsung Galaxy Tab 10.1 running Android 10.
Here is a medium article with the code:
https://medium.com/#jnishu1996/over-the-air-ota-updates-for-android-apps-download-apk-silent-apk-installation-auto-launch-8ee6f342197c
The magic is running this command with root access:
process = Runtime.getRuntime().exec("su");
out = process.getOutputStream();
DataOutputStream dataOutputStream = new DataOutputStream(out);
// Get all file permissions
dataOutputStream.writeBytes("chmod 777 " + file.getPath() + "\n");
// Perform silent installation command, all flags are necessary for some reason, only this works reliably post Android 10
String installCommand = "cat " + file.getAbsolutePath() + "| pm install -d -t -S " + file.length();
// Data to send to the LaunchActivity to the app knows it got updated and performs necessary functions to notify backend
// es stands for extraString
// In LaunchActivity onCreate(), you can get this data by running -> if (getIntent().getStringExtra("OTA").equals("true"))
String launchCommandIntentArguments = "--es OTA true --es messageId " + MyApplication.mLastSQSMessage.receiptHandle();
// Start a background thread to wait for 8 seconds before reopening the app's LaunchActivity, and pass necessary arguments
String launchCommand = "(sleep 8; am start -n co.getpresso.Presso/.activities.LaunchActivity " + launchCommandIntentArguments + ")&";
// The entire command is deployed with a ";" in the middle to launchCommand run after installCommand
String installAndLaunchCommand = installCommand + "; " + launchCommand;
// begins the installation
dataOutputStream.writeBytes(installAndLaunchCommand);
dataOutputStream.flush();
// Close the stream operation
dataOutputStream.close();
out.close();
int value = process.waitFor();

Gradle task using adb to install apk not being executed

I am writing a gradle task to install an apk unto emulators before espresso tests are run.
This is the task I have so far.
task installButlerApk {
doLast {
println "Verifying test-butler installation in emulators"
final adb = "$android.sdkDirectory.absolutePath/platform-tools/adb"
final String[] split = ["$adb", "devices", "-l"].execute().text.split("\\r?\\n")
split.each {
if (it.isEmpty())
return;
println "Emulator: $it"
final emu = it.split("\\s")[0]
checks whether the APK is already installed
if (["$adb", "-s", "$emu", "shell", "pm", "list", "packages"].execute().text.contains(butlerPackage))
return;
final installResult = ["$adb", "-s", "$emu", "install", "$butlerApkPath"].execute().text
if (!installResult.contains("Success"))
println "Could not install APK. Install output:\n$installResult"
else
println "Installed $butlerApkPath in $emu successfully"
}
}
}
However when I run it via the terminal the task ends up freezing. I am not sure why. I did some research about it and at one point I thought the command that was being passed to ProcessGroovyMethods' execute was failing because it was being passed as a string (execute(String self)) so I then used the array representation of execute (execute(String[] commandArray)) to see if that would work but I am still ending up with the same result so I am just asking for someone who has experience writing these tasks to give me some assistance. So far, I am printing the result of the command and it hasn't shown any errors. It's just stuck at the building process for hours.
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.
C:\Users\Joel\Documents\Projects\Forms>gradlew installButlerApk
Picked up _JAVA_OPTIONS: -XX:ParallelGCThreads=2
To honour the JVM settings for this build a new JVM will be forked. Please consider using the daemon:
https://docs.gradle.org/2.14.1/userguide/gradle_daemon.html.
Incremental java compilation is an incubating feature.
:app:installButlerApk
Verifying test-butler installation in emulators
Emulator: List of devices attached
> Building 0% > :app:installButlerApk

Well, this is expected behavior.
If you look at your output closely, you see
Emulator: List of devices attached
So following your code:
println "Emulator: $it"
outputs that line I quoted
final emu = it.split("\\s")[0]
takes the first space separated token which is List
checks whether the APK is already installed
this will not even compile, but I guess you just forgot the comment characters that you added in the question as explanation
if (["$adb", "-s", "$emu", "shell", "pm", "list", "packages"].execute().text.contains(butlerPackage))
return;
Now you execute adb -s List shell pm list
Executed manually this two times prints error: device not found for me and then exits, so your contains condition is false and the return is not done.
final installResult = ["$adb", "-s", "$emu", "install", "$butlerApkPath"].execute().text
Now you execute adb -s List install butler.apk
Executed manually this three times prints out error: device not found, then one time - waiting for device - and then sits there waiting until you cancel it, or a device with serial number List becomes available which of course will never happen and thus your task hangs until you kill it.
You have to skip the header line when you work through the list of devices, as this is of course not a device.
Besides this, you can of course use the Groovy standard ways to execute external commands. Yet while in Gradle, I'd rather use the Gradle variants. If you only want to execute one thing it would be a task of type Exec, if you want to execute multiple things like in your case, it is the Project.exec() or Script.exec() methods, so you would do something like
def output
new ByteArrayOutputStream().withStream { baos ->
exec {
executable adb
args "-s", emu, "shell", "pm", "list", "packages"
standardOutput os
}.assertNormalExitValue()
output = baos.toString()
}
if (output.contains(butlerPackage)) {
return
}
exec {
executable adb
args "-s", emu, "install", butlerApkPath
}.assertNormalExitValue()

How to order test cases for Spoon Automated Testing in Android?

I have added a suite() method to order my tests the way I want them and thus when I run it through Android JUnit they are executed accordingly. But then I noticed that when I use the Spoon execution, the one using cmd, my test cases are executed alphabetically which is the default order.
Why does this happen and how would you counter it without renaming my test cases?

I have the same issue as you; I require a specific order that my test need to be ran in. The app I am testing is too complicated to run in an unpredictable order. My solution was this:
Add this to your build.gradle:
spoon {
if (project.hasProperty('spoonClassName')){
className = project.spoonClassName
}
}
Now, you can execute a specific class with a command like so:
gradle spoon -PspoonClassName=< com.your.pakage.ClassName>
Next, create a file at the root of your Android project: runAllTests.sh
Edit your .sh to look like this:
#!/bin/sh
date +%b-%dT%H.%M > timestamp.out
sites="$HOME"/path/to/project/root
timestamp="$(cat "$sites"/timestamp.out)"
result_folder="$sites"/results
destdir="$result_folder/Results-$timestamp"
mkdir -p "$destdir"
echo "Directory created: ${destdir##*/}"
<---------- Here you start running the test --------------->
echo "Starting Master Setup"
gradle spoon -PspoonClassName=com.espresso.test.MasterSetup
cp -r "$sites"/app/build/spoon "$destdir"/MasterSetup
echo "Results saved to MasterSetup"
echo "Starting WorkoutSchedule"
gradle spoon -PspoonClassName=com.espresso.test.WorkoutSchedule
cp -f "$sites"/app/build/spoon "$destdir"/WorkoutSchedule
echo "Results saved to WorkoutSchedule"
echo "Starting Setting.test"
gradle spoon -PspoonClassName=com.espresso.test.Settings
cp -r "$sites"/app/build/spoon "$destdir"/Settings
echo "Results saved to Settings"
Then, give the script permissions
cd to the script
type chmod u+x runAllTest.sh
You're set. Now just cd to your root, then to execute your test, type . runAllTest.sh.
So, what this does:
First, it creates a timestamp.out. I use this so I can save my results to a file over and over without previous results being overwritten. You do not need this part.
Next, it creates a result folder in the root of your project if it is not already there.
Then, it will make a folder inside the results folder named Results-SOME-DATE.
Lastly, each test will run, saving the results to the normal spot on your project. (Inside build/spoon) Once test are complete it will copy the results to the results folder, and name each test result appropriately so it is easy to see all your tests ran.
NOTE: This script was wrote for MAC. If your on windows or anything else, this script may need modifications.
Additionally: You will find it is inconvenient to open in to each folder to get the index.html opened. So I wrote this script to add to your bash_profile:
function open-results () {
# the browser to open up `index.html' in.
browser='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
# let the user know what directory we're looking in
printf "looking in %s" "$(pwd)"
echo ...
for paths in $(find ./ -name 'debug' -type d); do
for files in $(find "$paths" -name 'index.html'); do
open -a "$browser" "$files"
done
done
echo done
}
Now, cd in terminal to the Results-SOME-DATE, and type open-results. Again, this was written for terminal. You may need to modify depending on your OS. But the structure should be the same
I hope this helps.

The jUnit testing philosophy is that test cases should not depend on each other so order shouldn't be important. That's why you're finding it hard to do. You might want to consider using the "setUp" method to create initial conditions for your test cases rather than having them build on each other.