Develop Reference

Reading character by character with android shell (mksh)

Sorry if the question is easy. I am kind of a beginner in shell scripting, and I need to write a script which will work on a text file on a machine which mksh installed, but no working version of sed or most gnu utilities or compatible.
There is also no version of dos2unix installed.
The script receives a file which is dos formatted, but is quite simple in the kind of characters it contains (only letters and numbers, lenght of each line below 20 characters, few than 1000 lines), and it reads the file character by character adding the character to a "line" variable. When it reaches a carriage return it prints the line. Usage will be sh script.sh file.txt > newfile.txt.
The script does not work as intended and I am not really sure why:
#!/bin/sh
riga="";
nomefile="$1";
while IFS='' read -r -n1 carattere;
do
if [[ $carattere != *$'\r'* ]]; then
riga="${riga}carattere";
elif [[ $carattere == *$'\r'* ]]; then
print "${riga%$}";
riga="";
fi
done < "$nomefile"
This is the output of the script as originally written:
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
caratterecaratterecaratterecaratterecaratterecaratterecaratterecaratterecarattere
I read the excellent man pages of ksh at https://www.freebsd.org/cgi/man.cgi?query=ksh and modified the script as follows:
#!/bin/sh
#!i=0
#!
riga="";
nomefile="$1";
while IFS='' read -r -n1 carattere;
do
if [[ $carattere != *$'\r'* ]]; then
riga="${riga}$carattere";
elif [[ $carattere == *$'\r'* ]]; then
print "${riga%$}";
riga="";
fi
done < "$nomefile"
The output is similar to what I intend, but blank lines are inserted between printed lines:
ID
1
2
3
4
5
6
7
8

Ok, after more reading of the ksh man pages and some tests, I found out what wasnt working. The first version of my script did not correctly append the characters I read to the riga variable, because I did not use a substitution ($) to append the carattere variable.
The second version worked as intended, but did not take into consideration the fact that dos files terminate lines with CRLF: I was thus checking for the \r character, and this meant the \n character was added to my riga variable. I modified the first if condition to check if the character is different from \n too.
Then an additional problem in my script was that, in the structure of files I want to convert, there are n lines and the last line is not terminated. This means my script would not write the characters of the last line in the riga variable, but it would not print the last line. I solved this adding a print instruction after the while cycle, using the -n parameter to avoid printing a newline after the string.
The final version of the script is:
#!/bin/sh
#!i=0
#!^M
riga="";
nomefile="$1";
while IFS='' read -r -n1 carattere;
do
if [[ $carattere != *$'\r'* ]] && [[ $carattere != *$'\n'* ]]; then
riga="${riga}$carattere";
elif [[ $carattere == *$'\r'* ]]; then
print "${riga%$}";
riga="";
fi
done < "$nomefile"
print -n "${riga%$}";
Thanks to Benjamin whose comments put me in the right direction to solve my problem.

How to handle .tar.md5 files

I was wondering about how to create / extract / verify .tar.md5 files. These files are used when flashing images to android devices, see here for example.
As far as I can tell the checksum is appended to the file like this:
cp file.tar file.tar.md5
md5sum file.tar >> file.tar.md5
Firstly I would like to know how to extract the file. Can I simply use tar -xf on the file.tar.md5?
How can I verify the integrity of the file? I would like to remove the last bytes (containing the checksum) from the file to obtain the original file back. I guess you would have to use a regexp to match the checksum file.tar? Is something like this implemented somewhere already?

First of all tar -xf should work since tar continues while it matches its' packing algorithm. If the file stops matching so would tar.
Also most archive managers such as 7-zip or winrar will open it if you remove the ".md5".
They might print error regarding mismatch with the end of the file, ignore it.
As for verifying the file:
print out the stored md5sum: tail -z -n 1 [File name here].tar.md5
calculate the md5sum of the tar part of the file: head -z -n -1 [File name here].tar.md5 | md5sum

What works for me with Ubuntu 19.10 is:
download single-file 4 GiB zip from sammobile com
unzip to several *.tar.md5
run the below command-line
.
for F in *.tar.md5; do echo -n "$F " &&
EXP=($(tail --lines=1 "$F")) &&
ACT=($(head --lines=-1 "$F" | md5sum)) &&
if [ ${EXP[0]} = ${ACT[0]} ]; then echo -n "md5ok " &&
tar --extract --file "$F" && echo "done"
else echo "FAIL"; fi; done &&
unlz4 --multiple --verbose *.lz4
AP_G965U1UEU3ARL1_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT_meta.tar.md5 md5ok done
BL_G965U1UEU3ARL1_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT.tar.md5 md5ok done
CP_G965U1UEU3ARL1_CP11407818_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT.tar.md5 md5ok done
CSC_OMC_OYM_G965U1OYM3ARL1_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT.tar.md5 md5ok done
HOME_CSC_OMC_OYM_G965U1OYM3ARL1_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT.tar.md5 md5ok done
…
But we should all try to get away from bash

GCC thinks I'm running x86

I am trying to compile Cyanogenmod on Linux Mint 15. And receive the following error.
host StaticLib: libmincrypt (/home/benji/Source/out/host/linux-x86/obj/STATIC_LIBRARIES/libmincrypt_intermediates/libmincrypt.a)
ERROR: prebuilts/tools/gcc-sdk/../../gcc/linux-x86/host/x86_64-linux-glibc2.7-4.6/bin/x86_64-linux-ar only run on 64-bit linux
make: *** [/home/benji/Source/out/host/linux-x86/obj/STATIC_LIBRARIES/libmincrypt_intermediates/libmincrypt.a] Error 1
make: *** Waiting for unfinished jobs....
# In case value of PACKAGES is empty.
-
benji#ultranoid ~/Source/prebuilts/tools/gcc-sdk $ ./gcc
ERROR: ./../../gcc/linux-x86/host/x86_64-linux-glibc2.7-4.6/bin/x86_64-linux-gcc only run on 64-bit linux
I can't figure out what is causing this. I am on a 64 bit install. Please advise.
benji#ultranoid ~/Source $ uname -p
x86_64

Workaround
Comment out lines 23-38 of prebuilts/tools/gcc-sdk/gcc
Example: http://pastebin.com/qH0BYcSF

As pointed out here, within Android build system the test for x32 vs x64 host is based on the output of file -L "$SHELL" | grep -q "x86[_-]64". In other words it tests whether or not current shell is a 64 bit binary.
So a possible answer is : check out if value of environmental variable $SHELL is a valid path to a shell executable.
You may try to change shell with chsh or
create a link to your favorite shell so that $SHELL is satisfied. The latter did the trick for me - I've simply linked /bin/bash to where $SHELL pointed.

use uname -m to check system.
#file -L "$SHELL" | grep -q "x86[_-]64"
#if [ $? != 0 ]; then
# $SHELL is not a 64-bit executable, so assume our userland is too.
# echo "ERROR: $MY_TOOL only run on 64-bit linux"
# exit 1
#fi
changed to:
ARCH_OS=uname -m | tr '[:upper:]' '[:lower:]'
if [ "$ARCH_OS" != "x86_64" ] ; then
echo "ERROR: $MY_TOOL only run on 64-bit linux from uname -m"
exit 1
fi

shell script in android gives [: not found

I have this script which works on my linux machine
#!/bin/sh
c=1
if [ $c == 1 ]
then
echo c is 1
else
echo c is 0
fi
But when I use this in android as follows:
#!/system/bin/sh
c=1
if [ $c == 1 ]
then
echo c is 1
else
echo c is 0
fi
It gives an error like:
[: not found
EDIT
Is there any other logic to check the value of $c, whether it is 1 or 0 ?
Android shell have problem with [] in if so is there any other way to check the value of c ?

andriod shell sh is actually a link to busybox, and it is invoked as
busybox sh
you need setup [ applets manually
busybox ln -s /your_original_sh_path/busybox [
if you don't know where busybox is put, try list the /system/bin/sh which you give
ls /system/bin/sh
busybox which busybox

generally [ is an alias for test,
in Linux machine test is at
/usr/bin/test
and
if [ $c == 1 ]
is evaluated as
if test "$c" = 1
BUT here in android there is no test
so if with [] will not work in any case...
i will cross compile test for android and check it....!!!

Android does not provide a full UNIX environment, it is not a UNIX operating system. It has some similarities, much like how Windows also has some similarities to UNIX. Some Android devices and ROMs try to provide more of a UNIX-like environment that others, but you cannot rely on most of the standard shell scripting tools being installed if you are thinking about cross-device compatibility.
So for example, if you look at your GNU/Linux system, you can see that test and [ are actually programs. Try this: ls -l /usr/bin/[. Most Android installs do not include test or [. That means that if you want to try to do actual programming with Android's minimal shell environment, you have to use lots of odd tricks. You can install busybox to get a full UNIX shell environment, or you can even build busybox into your app. I do that when I need to include shell scripts in an app (for example, Lil' Debi and Commotion MeshTether).
Here's an example of writing a killall in Android's /system/bin/sh environment: http://en.androidwiki.com/wiki/Android_Shell_tips_and_tricks You can also use the various parameter expansions to create some logic, you can see an example of that in the Barnacle Wifi Tether scripts.

Use bash:
#!/system/bin/bash
or
#!/system/xbin/bash
You can check where your sh binary is pointing to on your Linux machine:
ls -l /bin/sh
Edit
BTW, use:
c=1
if [ $c -eq 1 ]
then
echo c is 1
else
echo c is 0
fi

Think you using the wrong arithmetic operator and there is a syntax error of a missing ";": try
[ $c -eq 1 ];
Also your location for Bash (sh) might be wrong at the top of your file:
#!/system/bin/sh

How about checking that the .sh file doesn't contain a carriage return before line feed.
Windows \r\n -> CR LF
Unix \n -> LF

use /system/bin/cmp for equality test.
if you need numerically test, substitute $(($c == 1)) with $c
#!/system/bin/sh
echo $c >/tmp/a
echo 1 >/tmp/b
if cmp /tmp/a /tmp/b
echo c is 1
else
echo c is 0
fi

I run into this issue also and found a solution (on another site)
if [[ $b -gt 0]]
then
echo 'Hooray it works'
else
echo 'still works'
fi

adb push/pull with progress bar

It is really annoying if you adb push/pull large files to the device that there's no way to now how far along it is. Is it possible to run adb push or adb pull and get a progress bar using the 'bar' utility?
The main issue here is I think that adb expects two file names, if the input file could be replaced by stdin you could pipe through the 'bar' utility and get a progress bar. So far I haven't succeeded in doing so, but I'm not really a shell guru which is why I'm asking here :)
Note that I'm on Linux using bash.

It looks like the latest adb has progress support.
Android Debug Bridge version 1.0.32
device commands:
adb push [-p] <local> <remote>
- copy file/dir to device
('-p' to display the transfer progress)
However, the answers above also work for 'adb install' which do not have a progress option. I modified the first answer's script to work this way:
Create "adb-install.sh" somewhere in your PATH and run "adb-install.sh " instead of "adb install -f "
#!/bin/bash
# adb install with progressbar displayed
# usage: <adb-install.sh> <file.apk>
# original code from: http://stackoverflow.com/questions/6595374/adb-push-pull-with-progress-bar
function usage()
{
echo "$0 <apk to install>"
exit 1
}
function progressbar()
{
bar="================================================================================"
barlength=${#bar}
n=$(($1*barlength/100))
printf "\r[%-${barlength}s] %d%%" "${bar:0:n}" "$1"
# echo -ne "\b$1"
}
export -f progressbar
[[ $# < 1 ]] && usage
SRC=$1
[ ! -f $SRC ] && { \
echo "source file not found"; \
exit 2; \
}
which adb >/dev/null 2>&1 || { \
echo "adb doesn't exist in your path"; \
exit 3; \
}
SIZE=$(ls -l $SRC | awk '{print $5}')
export ADB_TRACE=all
adb install -r $SRC 2>&1 \
| sed -n '/DATA/p' \
| awk -v T=$SIZE 'BEGIN{FS="[=:]"}{t+=$7;system("progressbar " sprintf("%d\n", t/T*100))}'
export ADB_TRACE=
echo
echo 'press any key'
read n

Currently I have this little piece of bash:
function adb_push {
# NOTE: 65544 is the max size adb seems to transfer in one go
TOTALSIZE=$(ls -Rl "$1" | awk '{ sum += sprintf("%.0f\n", ($5 / 65544)+0.5) } END { print sum }')
exp=$(($TOTALSIZE * 7)) # 7 bytes for every line we print - not really accurate if there's a lot of small files :(
# start bar in the background
ADB_TRACE=adb adb push "$1" "$2" 2>&1 | unbuffer -p awk '/DATA/ { split($3,a,"="); print a[2] }' | unbuffer -p cut -d":" -s -f1 | unbuffer -p bar -of /dev/null -s $exp
echo # Add a newline after the progressbar.
}
It works somewhat, it shows a progress bar going from 0 to 100 which is nice. However, it won't be correct if you do a lot of small files, and worse, the bytes/s and total bytes shown by 'bar' aren't correct.
I challenge you to improve on my script; it shouldn't be hard! ;)

Here is my solution, it will show a simple progressbar and current numeric progress
[==================================================] 100%
Usage
./progress_adb.sh source destination
progress_adb.sh
#!/bin/bash
function usage()
{
echo "$0 source destination"
exit 1
}
function progressbar()
{
bar="=================================================="
barlength=${#bar}
n=$(($1*barlength/100))
printf "\r[%-${barlength}s] %d%%" "${bar:0:n}" "$1"
# echo -ne "\b$1"
}
export -f progressbar
[[ $# < 2 ]] && usage
SRC=$1
DST=$2
[ ! -f $SRC ] && { \
echo "source file not found"; \
exit 2; \
}
which adb >/dev/null 2>&1 || { \
echo "adb doesn't exist in your path"; \
exit 3; \
}
SIZE=$(ls -l $SRC | awk '{print $5}')
ADB_TRACE=adb adb push $SRC $DST 2>&1 \
| sed -n '/DATA/p' \
| awk -v T=$SIZE 'BEGIN{FS="[=:]"}{t+=$7;system("progressbar " sprintf("%d\n", t/T*100))}'
echo
Testing on Ubuntu 14.04
$ bash --version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
TODO
directory support
progressbar size change when screen size change

Well I can give you an Idea:
ADB_TRACE=adb adb push <source> <destination>
returns logs for any command, so for example the copy command, which looks like:
writex: fd=3 len=65544: 4441544100000100000000021efd DATA....#....b..
here you can get the total bytes length before, with ls -a, then parse the output of adb with grep or awk, increment an interneral counter and send the current progress to the bar utility.
When you succeeded, please post the script here.