Sorry if the question is easy. I am kind of a beginner in shell scripting, and I need to write a script which will work on a text file on a machine which mksh installed, but no working version of sed or most gnu utilities or compatible.
There is also no version of dos2unix installed.
The script receives a file which is dos formatted, but is quite simple in the kind of characters it contains (only letters and numbers, lenght of each line below 20 characters, few than 1000 lines), and it reads the file character by character adding the character to a "line" variable. When it reaches a carriage return it prints the line. Usage will be sh script.sh file.txt > newfile.txt.
The script does not work as intended and I am not really sure why:
#!/bin/sh
riga="";
nomefile="$1";
while IFS='' read -r -n1 carattere;
do
if [[ $carattere != *$'\r'* ]]; then
riga="${riga}carattere";
elif [[ $carattere == *$'\r'* ]]; then
print "${riga%$}";
riga="";
fi
done < "$nomefile"
This is the output of the script as originally written:
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
I read the excellent man pages of ksh at https://www.freebsd.org/cgi/man.cgi?query=ksh and modified the script as follows:
#!/bin/sh
#!i=0
#!
riga="";
nomefile="$1";
while IFS='' read -r -n1 carattere;
do
if [[ $carattere != *$'\r'* ]]; then
riga="${riga}$carattere";
elif [[ $carattere == *$'\r'* ]]; then
print "${riga%$}";
riga="";
fi
done < "$nomefile"
The output is similar to what I intend, but blank lines are inserted between printed lines:
ID
1
2
3
4
5
6
7
8
Ok, after more reading of the ksh man pages and some tests, I found out what wasnt working. The first version of my script did not correctly append the characters I read to the riga variable, because I did not use a substitution ($) to append the carattere variable.
The second version worked as intended, but did not take into consideration the fact that dos files terminate lines with CRLF: I was thus checking for the \r character, and this meant the \n character was added to my riga variable. I modified the first if condition to check if the character is different from \n too.
Then an additional problem in my script was that, in the structure of files I want to convert, there are n lines and the last line is not terminated. This means my script would not write the characters of the last line in the riga variable, but it would not print the last line. I solved this adding a print instruction after the while cycle, using the -n parameter to avoid printing a newline after the string.
The final version of the script is:
#!/bin/sh
#!i=0
#!^M
riga="";
nomefile="$1";
while IFS='' read -r -n1 carattere;
do
if [[ $carattere != *$'\r'* ]] && [[ $carattere != *$'\n'* ]]; then
riga="${riga}$carattere";
elif [[ $carattere == *$'\r'* ]]; then
print "${riga%$}";
riga="";
fi
done < "$nomefile"
print -n "${riga%$}";
Thanks to Benjamin whose comments put me in the right direction to solve my problem.
Related
I'm writing a simple Bash script that simply the call of HadnBrakeCli for render videos.
I also implemented a simple queue option: the queue file just store the line-command it has to call to start a render.
So I wrote a while-loop to read one line at time, eval $line and repeat untill file ends.
if [[ ${QUEUE_MODE} = 'RUN' ]]; then
QUEUE_LEN=`cat ${CONFIG_DIR}/queue | wc -l`
QUEUE_POS='1'
printf "Queue lenght:\t ${QUEUE_LEN}\n"
while IFS= read line; do
echo "--Running render ${QUEUE_POS} on ${QUEUE_LEN}..."
echo "++" && echo "$line" && echo "++"
eval "${line}"
tail -n +2 "${CONFIG_DIR}/queue" > "${CONFIG_DIR}/queue.tmp" && mv "${CONFIG_DIR}/queue.tmp" "${CONFIG_DIR}/queue"
echo "--Render ended"
QUEUE_POS=`expr $QUEUE_POS + 1`
done < "${CONFIG_DIR}/queue"
exit 0
The problem is that any command makes the loop to work fine (empty line, echo "test"...), but as soon a proper render is loaded, it is launched and finished correctly, but also the loops exists.
I am a newbie so I tried some minor changes to see what effect I got, but nothing change the result.
I commented the command tail -n +2 "${CONFIG_DIR}/queue" > "${CONFIG_DIR}/queue.tmp" && mv "${CONFIG_DIR}/queue.tmp" "${CONFIG_DIR}/queue" or I added/removed IFS= in the while-loop or removed the -r in read command.
Sorry if the question is trivial, but I'm really missing some major part in how it works, so I have no idea even how to search for the solution.
I'll put a sample of a general render in the queue file.
HandBrakeCLI -i "/home/andrea/Videos/done/Rap dottor male e mini me.mp4" -o "/hdd/Render/Output/Rap dottor male e mini me.mkv" -e x265 -q 23 --encoder-preset faster --all-audio -E av_aac -6 dpl2 --all-subtitles -x pmode:pools='16' --verbose=0 2>/dev/null
HandBrakeCLI reads from standard input, which steals the rest of the queue file before read line can see it. My favorite solution to this is to pass the file over something other than standard input, like file descriptor #3:
...
while IFS= read line <&3; do # The <&3 makes it read from FD #3
...
done 3< "${CONFIG_DIR}/queue" # The 3< redirects the file into FD #3
Another way to avoid the problem is to redirect input to the HandBrakeCLI command:
...
eval "${line}" </dev/null
...
There's some more info about this in BashFAQ #89: I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!
Also, I'm not sure I trust the way you're using tail to remove lines from the queue file as they're executed. I'm not sure it's really wrong, it just looks fragile to me. Also, I'd recommend using lower- or mixed-case variable names, since there are a bunch of all-caps names with special meanings, and re-using one of them by mistake can have weird consequences. Finally, I'd recommend running your script through shellcheck.net, as it'll make some other good recommendations.
[BTW, this question is a duplicate of "Bash script do loop exiting early", but that doesn't have any upvoted or accepted answers.]
Using adb shell to run commands on an android device, I get different results when running ls with or without a wildcard ( globbing, i.e * ).
When running ls without a wildcard, the last path is displayed properly. When running ls with a wildcard, the path is displayed with an : in the end of it for some reason. The actual file does not have a : in its path.
My issue is specifically with the last file: /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-com.com.kauf.wrapmyFaceFunphotoeditor-2020-05-17-17-44-30-DEBUG.txt:
it has an : in the end which isn't supposed to be there
Why does using a wildcard in ls add characters to the result path?
Edit, environment details: Windows 10 / Android 7, the code is running on sh. I've ran adb shell to get to this command prompt, and doing it in one line (i.e adb shell su -c ls ...) returns similar results, same for adb shell command ...; also clarified the question.
As described in Why you shouldn't parse the output of ls, ls's behavior is not always well-defined. It's generally safer to use NULs (if you don't have any control or knowledge of filenames) or newlines (if you have reason to be certain that filenames can't contain them) to directly delimit a list of values emitted by the shell. Consider, then:
# output is separated by NULs, which cannot possibly exist in filenames
printf '%s\0' /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
...or...
# output is separated by newlines; beware of a file named DV-evil<newline>something-else
printf '%s\n' /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
Note that if you're passing this through extra unescaping layers, it may be necessary to double up your backslashes -- if you see literal 0s or ns separating filenames in your output, that's evidence of same.
Note also that if no matching files exist, a glob will expand to itself, so you can get an output that contains only the literal string /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*; in bash this can be suppressed with shopt -s nullglob, but with /bin/sh (particularly the minimal busybox versions more likely to be available on Android) this may not be available. One way to work around this is with code similar to the following:
# set list of files into $1, $2, etc
set -- /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
# exit immediately if $1 does not exist
if [ "$#" -le 1 ] && [ ! -e "$1" ]; then
exit
fi
# otherwise, print the list in our desired format
printf '%s\0' "$#"
This is really odd...
I can't get this test to result in true in my linux shell and I can't figure out why.
#!/bin/bash
a=$(adb shell getprop ro.product.brand)
adb shell getprop ro.product.brand
if [ "$a" == "Huawei" ]; then
echo "Success"
else
echo "Failed"
fi
The script just outputs:
Huawei
Failed
Whereas this script:
b=$(whoami)
whoami
if [ "$b" == "amo" ]; then
echo "Success"
else
echo "Failed"
fi
...outputs:
amo
Success
Can anyone help me understand that?
I already tried cutting away spaces or line breaks in $a by piping to cut or sed but I get the same result...
I suggest this as a way to remove leading/trailing whitespace :
# Trims $1
# If $2 supplied, assigns result to variable named $2
# If $2 not present, echoes the value to stdout
trim()
{
if
[[ $1 =~ ^[[:blank:]]*(.*[^[:blank:]])[[:blank:]]*$ ]]
then
local result="${BASH_REMATCH[1]}"
else
local result="$1"
fi
if
(( $# > 1 ))
then
printf -v "$2" %s "$result"
else
printf %s "$result"
fi
}
This function uses no external program, so has low overhead.
Maybe a quick explanation of the regular expression...
^[[:blank:]]*(.*[^[:blank:]])[[:blank:]]*$
It matches all leading and trailing whitespace (no surprise there)
In the middle, it matches any string of characters that ends with a non-blank and saves that as a sub-expression for access with BASH_REMATCH
If there were no "non-blank" character specified to end the middle portion, the greedy .* would eat everything up until the end of the string, including trailing blanks.
The .* is, on the other hand, certain to begin with a non-blank, because the greedy initial [[:blank:]]* will only stop when encountering a non-blank.
Depending on your need, you may also use [[:space:]] instead of [[:blank:]] (difference explained here : https://en.wikipedia.org/wiki/Regular_expression#Character_classes). Basically, [[:blank:]] matches tabs and spaces, and [[:space:]] also matches newlines, carriage returns, and a few more.
I'm running stock android (5.1.1) on my nexus 5 and trying to use a script to kill a process (kill_process.sh).
while true; do
PROCESS=$(pgrep -l whatsapp)
if [[ $PROCESS == *"whatsapp"* ]]; then
kill $(pgrep -f whatsapp)
fi
done
I'm able to have scripts running at boot using the app "Universal Init.d", but this particular script does not seem to be working at boot. It does work as long as I ssh into my phone and run it manually. E.g., as soon as I launch whatsapp it's being killed right away.
Does anybody have any ideas what I could do to make it work at boot?
Also, a while loop is probably not the most efficient way to do this...so any ideas are welcome.
Edit1:
changed the if statement to
case $PROCESS in *"whatsapp"*)
kill $(pgrep -f whatsapp)
esac
Edit2: is there a way to daemonize a script?
Edit3: I set up another script containing only one line start-stop-daemon --exec /path_to_main_script/kill_process.sh -S. I only put the new script in the /system/etc/init.d directory and the kill_process.sh script into a different subdirectory in /system/etc/. This way the new script is being run at boot which then calls the kill_process.sh script. But it's not running stable, which is caused by the "Universal Init.d" app: when the app crashes the script stops running. Any ideas?
Disclaimer: I never used the app Universal Init.d nor do I have a lot of Android experience. But I did some tests on my desktop:
me:~$ bash -x /tmp/x
+ a=abcd
+ [[ abcd == *bc* ]]
+ echo 0
0 # correct
me:~$ dash -x /tmp/x
+ a=abcd
+ [[ abcd == *bc* ]]
/tmp/x: 3: /tmp/x: [[: not found
+ echo 127
127 # [[ not supported
me:~$ busybox sh -x /tmp/x
+ a=abcd
+ [[ abcd == *bc* ]]
+ echo 1
1 # wrong!
The script:
a=abcd
[[ $a == *bc* ]]
echo $?
Conclusion: You should check the shell which the app uses. It's likely that the [[ syntax in combination with * fails. I think it's busybox or dash there...
For saving you battery, I recommend at least replacing while true by
while sleep 1; do
But also this is just a hack. You should look for an event-based solution (cannot help you there)
This question already has answers here:
How to compare strings in Bash
(12 answers)
Closed 8 years ago.
I want to simply test if an android device is rooted from a computer (for a root/backup/flash script).
So i have to test some locations for the su binary. Here is my code :
#!/bin/sh
#####################
## ROOT CHECK
SU_LOCATIONS="/system/bin/su /system/xbin/su"
check() {
echo "Checking for SU binary…"
for file in $SU_LOCATIONS
do
suFileResult=`$ADB shell "ls $file"`
echo "suFileResult = $suFileResult"
echo "file = $file"
if [ "$suFileResult" == "$file" ];
then echo "Su trouvé à $suFileResult"
fi
done
echo "$suFileResult" > tmpbak/suFil
}
The problem is that even if file == suFileResult, "if" return false. If i remove the spaces around ==, "if" will always return TRUE…
What am i doing wrong ? If you give an other way to test the file (and where), it would be perfect.
Thanks for your answers !
PS : the string could contain spaces here, such as :
/system/bin/su: No such file or directory
EDIT After answer :
By the way, i figured how to remove this annoying \r character : add
| tr -d '\r'
will remove this character in ALL the string (not only on the end). So in my case :
suFileResult=$(adb shell "[[ -e $file ]]; echo \$?;" | tr -d '\r')
By some weird reason adb returns strings with DOS line-ending (\r\n).
$ suFileResult="$(adb shell "ls $file")"
$ set | grep suFileResult
suFileResult=$'/system/bin/su\r'
So the proper condidtion statement may look like:
[[ $suFileResult == ${file}$'\r' ]]
Also, I should note that using ls for checking file existence is a quite odd approach, even though adb does not return an error code, you could print it to STDOUT:
suFileResult=$(adb shell "[[ -e $file ]]; echo \$?")
if [[ $suFileResult == $'0\r' ]]; then
echo "Su trouvé à $file"
fi