Develop Reference

Execute multiple adb shell command in bash script [duplicate]

I'm writing a simple Bash script that simply the call of HadnBrakeCli for render videos.
I also implemented a simple queue option: the queue file just store the line-command it has to call to start a render.
So I wrote a while-loop to read one line at time, eval $line and repeat untill file ends.
if [[ ${QUEUE_MODE} = 'RUN' ]]; then
QUEUE_LEN=`cat ${CONFIG_DIR}/queue | wc -l`
QUEUE_POS='1'
printf "Queue lenght:\t ${QUEUE_LEN}\n"
while IFS= read line; do
echo "--Running render ${QUEUE_POS} on ${QUEUE_LEN}..."
echo "++" && echo "$line" && echo "++"
eval "${line}"
tail -n +2 "${CONFIG_DIR}/queue" > "${CONFIG_DIR}/queue.tmp" && mv "${CONFIG_DIR}/queue.tmp" "${CONFIG_DIR}/queue"
echo "--Render ended"
QUEUE_POS=`expr $QUEUE_POS + 1`
done < "${CONFIG_DIR}/queue"
exit 0
The problem is that any command makes the loop to work fine (empty line, echo "test"...), but as soon a proper render is loaded, it is launched and finished correctly, but also the loops exists.
I am a newbie so I tried some minor changes to see what effect I got, but nothing change the result.
I commented the command tail -n +2 "${CONFIG_DIR}/queue" > "${CONFIG_DIR}/queue.tmp" && mv "${CONFIG_DIR}/queue.tmp" "${CONFIG_DIR}/queue" or I added/removed IFS= in the while-loop or removed the -r in read command.
Sorry if the question is trivial, but I'm really missing some major part in how it works, so I have no idea even how to search for the solution.
I'll put a sample of a general render in the queue file.
HandBrakeCLI -i "/home/andrea/Videos/done/Rap dottor male e mini me.mp4" -o "/hdd/Render/Output/Rap dottor male e mini me.mkv" -e x265 -q 23 --encoder-preset faster --all-audio -E av_aac -6 dpl2 --all-subtitles -x pmode:pools='16' --verbose=0 2>/dev/null

HandBrakeCLI reads from standard input, which steals the rest of the queue file before read line can see it. My favorite solution to this is to pass the file over something other than standard input, like file descriptor #3:
...
while IFS= read line <&3; do # The <&3 makes it read from FD #3
...
done 3< "${CONFIG_DIR}/queue" # The 3< redirects the file into FD #3
Another way to avoid the problem is to redirect input to the HandBrakeCLI command:
...
eval "${line}" </dev/null
...
There's some more info about this in BashFAQ #89: I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!
Also, I'm not sure I trust the way you're using tail to remove lines from the queue file as they're executed. I'm not sure it's really wrong, it just looks fragile to me. Also, I'd recommend using lower- or mixed-case variable names, since there are a bunch of all-caps names with special meanings, and re-using one of them by mistake can have weird consequences. Finally, I'd recommend running your script through shellcheck.net, as it'll make some other good recommendations.
[BTW, this question is a duplicate of "Bash script do loop exiting early", but that doesn't have any upvoted or accepted answers.]

Extra ":" at the end of output from sudo su -c ls, only when globbing is used

Using adb shell to run commands on an android device, I get different results when running ls with or without a wildcard ( globbing, i.e * ).
When running ls without a wildcard, the last path is displayed properly. When running ls with a wildcard, the path is displayed with an : in the end of it for some reason. The actual file does not have a : in its path.
My issue is specifically with the last file: /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-com.com.kauf.wrapmyFaceFunphotoeditor-2020-05-17-17-44-30-DEBUG.txt:
it has an : in the end which isn't supposed to be there
Why does using a wildcard in ls add characters to the result path?
Edit, environment details: Windows 10 / Android 7, the code is running on sh. I've ran adb shell to get to this command prompt, and doing it in one line (i.e adb shell su -c ls ...) returns similar results, same for adb shell command ...; also clarified the question.

As described in Why you shouldn't parse the output of ls, ls's behavior is not always well-defined. It's generally safer to use NULs (if you don't have any control or knowledge of filenames) or newlines (if you have reason to be certain that filenames can't contain them) to directly delimit a list of values emitted by the shell. Consider, then:
# output is separated by NULs, which cannot possibly exist in filenames
printf '%s\0' /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
...or...
# output is separated by newlines; beware of a file named DV-evil<newline>something-else
printf '%s\n' /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
Note that if you're passing this through extra unescaping layers, it may be necessary to double up your backslashes -- if you see literal 0s or ns separating filenames in your output, that's evidence of same.
Note also that if no matching files exist, a glob will expand to itself, so you can get an output that contains only the literal string /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*; in bash this can be suppressed with shopt -s nullglob, but with /bin/sh (particularly the minimal busybox versions more likely to be available on Android) this may not be available. One way to work around this is with code similar to the following:
# set list of files into $1, $2, etc
set -- /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
# exit immediately if $1 does not exist
if [ "$#" -le 1 ] && [ ! -e "$1" ]; then
exit
fi
# otherwise, print the list in our desired format
printf '%s\0' "$#"

Time cannot run brunch?

I'm having an issue with the time command.
I use /usr/bin/time so i can use the -f and -o modifiers.
My script is:
make clean
repo sync -j5
. ./platform_manifest/manual_add_kernel_manifest.sh \
&& . build/envsetup.sh \
&& /usr/bin/time -o log$day$month.log -f "Buildtime: %E" brunch aokp_mako-eng
The script is part of an automated build system for an android ROM.
When it gets to the time part, i get:
/usr/bin/time: cannot run brunch: No such file or directory
Command exited with non-zero status 127
Buildtime: 0:00.00
Brunch works fine with the regular time command, but I cannot route its output to a file, which is why I am using /usr/bin/time in the first place. If there is a way to do this, that is fine for me as I can trim off the real: header with | awk '{ print $2 }'.
Any help is appreciated!
Ubuntu 12.04, all updates installed

I don't know what brunch is, but the most likely reason that a non-builtin time program can't find it, even when the built-in time can, is that it's a shell function or a Bash alias.
Whether or not that's the case . . .
[…] I cannot route [the regular time command's] output to a file […]
You can write:
{ time brunch aokp_mako-eng ; } 2>"log$day$month.log"
to wrap time (and everything else) into a command-list whose STDERR has already been redirected to the file.

Android ant and bash scripting

I've had 0 exposure to BASH scripting and this is something I would love to learn. I can't figure out how to run a conditional statement based on the output of ant debug on an Android build.
I would like to essentially say something like
if(`ant debug` == SUCCESS) {
// EXECUTE THESE COMMANDS
} else {
// EXECUTE THESE COMMANDS
}
How can I determine if the ant debug has passed or failed in shell script?
SOLUTION
Okay here is what I have:
ant clean
if ant debug; then
echo "success"
else
echo "failure"
fi

I'll give a quick summary for you.
In Bash, conditionals are based around the exit codes of programs. An exit code of 0 is accepted as true, while everything else is accepted as false.
For example, the true program always exits with an exit code of 0, which means that something like this is possible:
if true;
echo "It is true"
fi
Most commands honor this system, but not every program does. The first thing to check is what exit code ant returns on success and failure. You can check the exit code of the previous command with $?.
Here is an example:
$ true
$ echo $?
0
$ false
$ echo $?
1
If ant does honor the exit code system properly, then something like the following should be possible:
if ant debug; then
echo success
else
echo failure
fi

I know nothing about Ant debugging, but there are two approaches to doing what you want to do in Bash. The first is to test output like you've shown:
if test $(ant debug) == 'SUCCESS'; then
# do stuff
else
# do other stuff
fi
You can make your shell script portable to other variants on the Bourne shell by using backticks instead of $(.....) like you wrote in your question, but that starts to become a hassle if your commands later involve nested quotes.
The second way, which is a little more robust, is to test the exit value of the commands instead of their output. This depends on Ant (or whatever) having exit codes that are documented and stable, but it means that if the output of the commands changes, your scripts will continue to work. For example, the POSIX standard says that if a programs succeeds in doing whatever it's supposed to do, it should exit() with a value of zero:
ant debug > /dev/null
ant_exit_code=$?
# other commands can safely go here now that $? is captured
if test $ant_exit_code -eq 0; then
# do stuff
else
# do other stuff
fi
And yes, Bourne shell really does end an if block with "fi". :-)

A quick man ant shows that ant debug invokes Ant with the debug task. Ant's tasks are kinda of user-defined XML scripts, and by default Ant searches a build.xml file in the current directory. You can generate the file with the android tools, however a template is kept in android-sdk/tools/ant and you can view it online (line 1005 defines the debug target).
So ant debug isn't really a command, and should not be put in a script toolchain. However, if you find your output to be stable and feel brave, you can always compare string. This is the definitve guide.
if [ `ant debug` = $SOMETHING ]; then
echo Success
else
echo Failure
fi

Trouble calling bash from within python

I am trying to use python to help do some automation around an incremental build function in the Android build system. Generally, from a given directory, I would execute the following command to build whatever is in that directory and subdirectories:
mm -j8
This is analogous to a "make" command, only it is incremental build and is defined as a function in a bash file called envsetup.sh. What is does it not important, just know that it's a function defined in a bash script somewhere in the file system. To execute this, I can also do:
bash -c ". /path/to/envsetup.sh; mm -j8"
This method of calling it will be important in calling the function from python. I have followed the solution here which shows how to call a function within a bash script from python. I have used this method in a simple script that, in theory, should just spit out the STDOUT and STDERR from executing the command:
import subprocess
command = ['bash', '-c', '. /path/to/envsetup.sh; mm -j8']
(stdout, stderr) = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
print 'stdout: ' + stdout
print 'stderr: ' + stderr
The call to Popen, however, never returns. What am I doing wrong that would allow bash to execute the command properly, but Python hangs when executing the command?

tl; dr:
Your issue is the use of shell=True. Set it to shell=False and it'll work.
With this option set, python will just run the first element of the command array, i.e. bash as a shell script. So currently, python is launching a shell of its own, in order to run your command (bash). It'll run bash with no arguments, and bash will then wait for input, blocking your python script.
The shell=True setting is for use cases where you are passing a shell script in as a single string. When you're explicitly specifying a shell and its parameters as the process to invoke, as you are doing above, you should set shell=False.
>>> import subprocess
>>> subprocess.Popen(['bash', 'whatever'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
Here's what the proces tree looks like when I run the above:
\_ python
\_ /bin/sh -c bash whatever
\_ bash
The whatever is actually passed in, but it's a parameter to the sh, not a parameter to the inner bash, so the command being run is effectively ['/bin/sh', '-c', 'bash', 'whatever'], which is quite different from ['/bin/sh', '-c', 'bash whatever']