I have a weird issue where I am reading a binary file with cat. Piping the output is adding extra characters, specifically 0D0D. So I can read a file in a binary editor and it looks like this
...82 FF B3 C9 0A 97....
However, when I cat the output to a text file it mysteriously adds 0D0D like this
...82 FF B3 C9 0D 0D 0A 97...
I am reading a btsnoop_hci.log file generated by an android phone. These are the actual characters in question. I know for a fact that this happens in the middle of a string of characters that make up a Bluetooth link key, so those extra characters should DEFINITELY not be getting added. Does anyone know what is going on, and why cat is adding extra characters??
You are getting two carriage return bytes inserted before a line feed byte. These characters are significant, in different combinations, as line terminators for various operating systems. (Windows uses the two-byte terminator 0D 0A, Unix and OS X use just 0A, and the original MacOS used just 0D).
It looks, therefore, as if you are getting some kind of text mode processing of your binary data, though why you get that specific processing I couldn't say. I'm somewhat surprised by your claim that cat is performing such munging, but if it's really cat and not something else in the pipeline then you probably need to use a different tool.
cat is not adding any characters. Try the following:
$ adb shell
android$ md5 yourfile
android$ cat yourfile > newfile
android$ md5 newfile
Here's an example from my device:
shell#hammerhead:/mnt/sdcard $ md5 /system/bin/sh
cf3301e2cf56e5edc88ee20e3899a31e /system/bin/sh
shell#hammerhead:/mnt/sdcard $ cat /system/bin/sh > newfile
shell#hammerhead:/mnt/sdcard $ md5 newfile
cf3301e2cf56e5edc88ee20e3899a31e newfile
shell#hammerhead:/mnt/sdcard $
You'll notice that the checksums are identical. cat did not modify the file in any way.
Now let's try adb:
$ adb shell cat /system/bin/sh | md5sum
23c023cc5eaca953d4b1733afcdd1097 -
The checksum is different.
The problem is not with cat. Delete this question and ask a followup about adb.
Related
I'm writing a simple Bash script that simply the call of HadnBrakeCli for render videos.
I also implemented a simple queue option: the queue file just store the line-command it has to call to start a render.
So I wrote a while-loop to read one line at time, eval $line and repeat untill file ends.
if [[ ${QUEUE_MODE} = 'RUN' ]]; then
QUEUE_LEN=`cat ${CONFIG_DIR}/queue | wc -l`
QUEUE_POS='1'
printf "Queue lenght:\t ${QUEUE_LEN}\n"
while IFS= read line; do
echo "--Running render ${QUEUE_POS} on ${QUEUE_LEN}..."
echo "++" && echo "$line" && echo "++"
eval "${line}"
tail -n +2 "${CONFIG_DIR}/queue" > "${CONFIG_DIR}/queue.tmp" && mv "${CONFIG_DIR}/queue.tmp" "${CONFIG_DIR}/queue"
echo "--Render ended"
QUEUE_POS=`expr $QUEUE_POS + 1`
done < "${CONFIG_DIR}/queue"
exit 0
The problem is that any command makes the loop to work fine (empty line, echo "test"...), but as soon a proper render is loaded, it is launched and finished correctly, but also the loops exists.
I am a newbie so I tried some minor changes to see what effect I got, but nothing change the result.
I commented the command tail -n +2 "${CONFIG_DIR}/queue" > "${CONFIG_DIR}/queue.tmp" && mv "${CONFIG_DIR}/queue.tmp" "${CONFIG_DIR}/queue" or I added/removed IFS= in the while-loop or removed the -r in read command.
Sorry if the question is trivial, but I'm really missing some major part in how it works, so I have no idea even how to search for the solution.
I'll put a sample of a general render in the queue file.
HandBrakeCLI -i "/home/andrea/Videos/done/Rap dottor male e mini me.mp4" -o "/hdd/Render/Output/Rap dottor male e mini me.mkv" -e x265 -q 23 --encoder-preset faster --all-audio -E av_aac -6 dpl2 --all-subtitles -x pmode:pools='16' --verbose=0 2>/dev/null
HandBrakeCLI reads from standard input, which steals the rest of the queue file before read line can see it. My favorite solution to this is to pass the file over something other than standard input, like file descriptor #3:
...
while IFS= read line <&3; do # The <&3 makes it read from FD #3
...
done 3< "${CONFIG_DIR}/queue" # The 3< redirects the file into FD #3
Another way to avoid the problem is to redirect input to the HandBrakeCLI command:
...
eval "${line}" </dev/null
...
There's some more info about this in BashFAQ #89: I'm reading a file line by line and running ssh or ffmpeg, only the first line gets processed!
Also, I'm not sure I trust the way you're using tail to remove lines from the queue file as they're executed. I'm not sure it's really wrong, it just looks fragile to me. Also, I'd recommend using lower- or mixed-case variable names, since there are a bunch of all-caps names with special meanings, and re-using one of them by mistake can have weird consequences. Finally, I'd recommend running your script through shellcheck.net, as it'll make some other good recommendations.
[BTW, this question is a duplicate of "Bash script do loop exiting early", but that doesn't have any upvoted or accepted answers.]
Using adb shell to run commands on an android device, I get different results when running ls with or without a wildcard ( globbing, i.e * ).
When running ls without a wildcard, the last path is displayed properly. When running ls with a wildcard, the path is displayed with an : in the end of it for some reason. The actual file does not have a : in its path.
My issue is specifically with the last file: /data/data/com.kauf.wrapmyFaceFunphotoeditor/files/DV-com.com.kauf.wrapmyFaceFunphotoeditor-2020-05-17-17-44-30-DEBUG.txt:
it has an : in the end which isn't supposed to be there
Why does using a wildcard in ls add characters to the result path?
Edit, environment details: Windows 10 / Android 7, the code is running on sh. I've ran adb shell to get to this command prompt, and doing it in one line (i.e adb shell su -c ls ...) returns similar results, same for adb shell command ...; also clarified the question.
As described in Why you shouldn't parse the output of ls, ls's behavior is not always well-defined. It's generally safer to use NULs (if you don't have any control or knowledge of filenames) or newlines (if you have reason to be certain that filenames can't contain them) to directly delimit a list of values emitted by the shell. Consider, then:
# output is separated by NULs, which cannot possibly exist in filenames
printf '%s\0' /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
...or...
# output is separated by newlines; beware of a file named DV-evil<newline>something-else
printf '%s\n' /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
Note that if you're passing this through extra unescaping layers, it may be necessary to double up your backslashes -- if you see literal 0s or ns separating filenames in your output, that's evidence of same.
Note also that if no matching files exist, a glob will expand to itself, so you can get an output that contains only the literal string /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*; in bash this can be suppressed with shopt -s nullglob, but with /bin/sh (particularly the minimal busybox versions more likely to be available on Android) this may not be available. One way to work around this is with code similar to the following:
# set list of files into $1, $2, etc
set -- /data/data/com.kauf.wrapmyfacefunphotoeditor/files/DV-*
# exit immediately if $1 does not exist
if [ "$#" -le 1 ] && [ ! -e "$1" ]; then
exit
fi
# otherwise, print the list in our desired format
printf '%s\0' "$#"
i'm writing a shell script for android and i need to get the last created directory, i would usually use
ls -t | head -1 but ls -t gives me the error "ls: Unknown option '-t'"
is there another shell command that can order the files by time-stamp or another way to do that in android? the busy box is more limited
It looks like stat is available in BusyBox, which means that you could do something like this:
stat -c '%Y %n' */ | sort -nr | cut -d' ' -f2-
This passes the names of all directories (paths ending in a slash) to stat, which prints the last modification time (seconds since the UNIX epoch) and the filename. These are sorted in reverse numerical order and then the time field is stripped from each line.
This assumes that your directory names don't contain newlines, otherwise the sorting would be messed up.
I am seeking a way to measure launch times of general apps in Android, and happen to know that
adb logcat -b events | grep am_activity_launch_time
might be one answer.
But I failed to find out what duration the command above shows (i.e., from what event to what event). Any help?
You can use this command
cmd0=add logcat -c
to clear the logs and then execute the below command
cmd1 = "grep am_activity_launch_time Example.txt | grep com.example.example| cut -d ',' -f4-4 > Example.csv"
The values are then saved in a csv file.
Then you can the python commands
os.system(cmd0);
os.system(cmd2);
Then use matplotlib to plot the graph from the generated csv files.
axes=plt.gca()
axes.set_xlim([1,20])
axes.set_ylim([0,4000])
plt.title("Performance Analysis ")
plt.ylabel('Time in milliseconds')
plt.xlabel('Number of itirations (DEVICE:Samsung Galaxy Note 4)')
data1=np.genfromtxt("Example.csv)"
plt.plot(data1,label="Example",linewidth=2.0,color='m')
plt.legend(bbox_to_anchor=(1,1),loc=1,borderaxespad=0.)
plt.show()
Once this is done execute this file in the python cmd.
adb logcat -b all | egrep -n --color 'Displayed *'
12461:11-26 22:43:05.316 1110 1167 I ActivityTaskManager: Displayed mobi.goldendict.android.free/mobi.goldendict.android.GoldenDict: +1s494ms
You could measure app launches time in android by this.
I tried create an OBB file and then, dump this file to see all works fine.
I have a folder called "HOLA" in C:\Users\abreu20011\Desktop\HOLA with the next distribution:
HOLA
\_>hola.txt (i wrote "hello world" inside)
\_>emulator-arm.exe
\_>emulator-mips.exe
\_>emulator-x86.exe
I put emulators file, because when I trying to compress with the text file only, I received the next error when I write the sentence:
jobb -pn es.test.test -pv 1 -d C:\Users\abreu20011\Desktop\HOLA -o hola.obb
Slop:0 Directory Overhead: 0
Slop:289 Directory Overhead: 224
java.io.IOException: no data clusters
at de.waldheinz.fs.fat.Fat.<init>(Fat.java:129)
at de.waldheinz.fs.fat.Fat.create(Fat.java:96)
at de.waldheinz.fs.fat.Fat.SuperFloppyFormatter.format(SuperFloppyFormatter.java:236)
at com.android.jobb.Main.main(Main.java:414)
So, I fattened the directory with this emulators, and the jobb tool worked fine:
Slop:0 Directory Overhead: 0
Slop:2453 Directory Overhead: 608
But the real problems arise when I try to dump the obb file. Since the jobb tool creates the obb file in the same directory of the tool, I wrote the dump sentence:
jobb -dump hola.obb -d C:\Users\abreu20011\Desktop
The program extracts a partial file of emulator-arm.exe and wrote the next error:
Package Name: es.test.test
Package Version: 1
LFN = emulator-arm.exe / SFN = ShortName [/☺[←►▬←☺ifl -- 2f 1 5b 1b 10 16 1b 1 6 9 66 6c ]
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2335
at de.waldheinz.fs.fat.ClusterChain.readData(ClusterChain.java:225)
at de.waldheinz.fs.fat.FatFile.read(FatFile.java:126)
at com.android.jobb.Main.dumpDirectory(Main.java:137)
at com.android.jobb.Main.main(Main.java:315)
Why I can't create an obb file of his size is smaller?
and why I can't dump the original file?
Thanks for your help and for your time! :)