I am trying to run a bash script which will run on the current directory, find files and do some operation on them and when a directory is found do the same for it and over and over again.
The thing that got me confused was my operation, it is supposed to convert each graphic file, resize it, and put it in a parallel directory structure, doing that for four parallel directories, meaning it will mimic the original directory structure creating folders as it goes.
The problem is it doesn't work and also it keeps recursing into the newly created directories..
Can you help make it right?
recurse() {
for i in "$1"/*;do
if [ -d "$i" ];then
echo "dir: $i"
mkdir "res-ldpi/$i"
mkdir "res-hdpi/$i"
mkdir "res-mdpi/$i"
mkdir "res-xhdpi/$i"
recurse "$i";
elif [ -f "$i" ]; then
convert ./"$i" -resize 38% -unsharp 0x1 res-ldpi/"$i"
fi
done
}
recurse .
Perhaps something along these lines (not tested...):
while read fn
do
dn=$(dirname "${fn}")
bn=$(basename "${fn}")
[[ -d "res-ldpi/${dn}" ]] || mkdir -p "res-ldpi/${dn}"
convert "${fn}" -resize 38% -unsharp 0x1 "res-ldpi/${dn}/${bn}"
done < <(find "${1}"/. -type f -print)
The idea is to 1) use find to just give you the name of every file in the directory and its subdirectories, 2) split that into a directory and file name, 3) check that the destination directory exists and create it if necessary, then 4) run convert as desired.
That didn't bother with the res-hdpi, res-mdpi and res-xhdpi directories, since your sample code didn't use them anyway - it shouldn't be too difficult to figure out how to add those in...
Related
Story
I take photos and record videos with my phone camera and keep all of them on my internal storage/sdcard. I periodically back them up on my PC, so I keep these camera photos on PC storage in sync with phone storage.
For years, I've been backing up my phone camera photos to my PC in the following way:
Plug in phone into PC and allow access to phone data
Browse phone storage → DCIM → Camera
Wait several minutes for the system to load a list of ALL photos
Copy only several latest photos which haven't been backed up yet
I figured that waiting several minutes for all photos to load is an unnecessary drag so I downloaded adb platform tools. I've added the folder bin to my Path environment variable (i.e. %USERPROFILE%\Tools\adb-platform-tools_r28.0.3) so that I can seamlessly use adb and not write its full path each time.
The script
I wrote the following script for Git Bash for Windows. It is also compatible with Unix if you change the $userprofile variable. Essentially, the script pulls camera photos between two dates from phone storage to PC.
# Attach device and start deamon process
adb devices
# Initialize needed variables
userprofile=$(echo "$USERPROFILE" | tr "\\" "/") # Windows adjustments
srcFolder="//storage/06CB-C9CE/DCIM/Camera" # Remote folder
dstFolder="$userprofile/Desktop/CameraPhotos" # Local folder
lsFile="$dstFolder/camera-ls.txt"
filenameRegex="2019061[5-9]_.*" # Date from 20190615 to 20190619
# Create dst folder if it doesn't exist
mkdir -p "$dstFolder"
# 1. List contents from src folder
# 2. Filter out file names matching regex
# 3. Write these file names line by line into a ls file
adb shell ls "$srcFolder" | grep -E "$filenameRegex" > "$lsFile"
# Pull files listed in ls file from src to dst folder
while read filename; do
if [ -z "$filename" ]; then continue; fi
adb pull "$srcFolder/$filename" "$dstFolder" # adb: error: ...
done < "$lsFile"
# Clean up
rm "$lsFile"
# Inform the user
echo "Done pulling files to $dstFolder"
The problem
When I run the script (bash adb-pull-camera-photos.sh), everything runs smoothly except for the adb pull command in the while-loop. It gives the following error:
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190618_204522.jpg
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190619_225739.jpg
I am not sure why the output is broken. Sometimes when I resize the Git Bash window some of the text goes haywire. This is the actual error text:
adb: error: failed to stat remote object '//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg': No such file or directory
adb: error: failed to stat remote object '//storage/06CB-C9CE/DCIM/Camera/20190618_204522.jpg': No such file or directory
adb: error: failed to stat remote object '//storage/06CB-C9CE/DCIM/Camera/20190619_225739.jpg': No such file or directory
I am sure that these files exist in the specified directory on the phone. When I manually execute the failing command in bash, it succeeds with the following output:
$ adb pull "//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg" "C:/Users/User/Desktop/CameraPhotos/"
//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg: 1 file pulled. 15.4 MB/s (1854453 bytes in 0.115s)
The question
I can't figure out what's wrong with the script. I thought the Windows system might be causing a commotion, because I don't see the reason why the same code works when entered manually, but doesn't work when run in a script. How do I fix this error?
Additional info
Note that I had to use // in the beginning of an absolute path on Windows because Git Bash would interpret / as its own root directory (C:\Program Files\Git).
I've echoed all variables inside the script and got all the correct paths that otherwise work via manual method.
camera-ls.txt file contents
20190618_124656.jpg
20190618_204522.jpg
20190619_225739.jpg
Additional questions
Is it possible to navigate to external sdcard without using its name? I had to use /storage/06CB-C9CE/ because /sdcard/ navigates to internal storage.
Why does tr "\\" "/" give me this error: tr: warning: an unescaped backslash at end of string is not portable?
Windows batch script
Here's a .bat script that can be run by Windows Command Prompt or Windows PowerShell. No Git Bash required.
:: Start deamon of the device attached
adb devices
:: Pull camera files starting from date
set srcFolder=/storage/06CB-C9CE/DCIM/Camera
set dstFolder=%USERPROFILE%\Desktop\CameraPhotos
set lsFile=%USERPROFILE%\Desktop\CameraPhotos\camera-ls.txt
set dateRegex=2019061[5-9]_.*
mkdir %dstFolder%
adb shell ls %srcFolder% | adb shell grep %dateRegex% > %lsFile%
for /F "tokens=*" %%A in (%lsFile%) do adb pull %srcFolder%/%%A %dstFolder%
del %lsFile%
echo Done pulling files to %dstFolder%
Just edit the srcFolder to point to your phone camera folder,
plug a pattern into the dateRegex for matching the date interval and
save it as a file with .bat extension, i.e: adb-pull-camera-photos.bat.
Double-click the file and it will pull filtered photos into CameraPhotos folder on Desktop.
Keep in mind that you still need have adb for Windows on your PC.
The problem was with Windows line delimiters.
Easy fix
Just add the IFS=$'\r\n' above the loop so that the read command knows the actual line delimiter.
IFS=$'\r\n'
while read filename; do
if [ -z "$filename" ]; then continue; fi
adb pull "$srcFolder/$filename" "$dstFolder"
done < "$lsFile"
Explanation
I tried plugging the whole while-loop into the console and it failed with the same error:
$ bash adb-pull-camera-photos.sh
List of devices attached
9889db343047534336 device
tr: warning: an unescaped backslash at end of string is not portable
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190618_124656.jpg
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190618_204522.jpg
': No such file or directoryemote object '//storage/06CB-C9CE/DCIM/Camera/20190619_225739.jpg
Done pulling files to C:/Users/User/Desktop/CameraPhotos
This time I started investigating why the output was broken. I remembered that windows uses \r\n as newline, which means Carriage Return + Line Feed, (CR+LF), so some text must have been overwritten.
It was because of broken values stored inside the $filename variable.
This is the loop from the script:
while read filename; do
if [ -z "$filename" ]; then continue; fi
adb pull "$srcFolder/$filename" "$dstFolder"
done < "$lsFile"
Since each iteration of the while-loop reads a line from $lsFile in the following form:
exampleFilename.jpg\r\n
It misinterprets the newline symbols as part of the file name, so adb pull tries to read files with these whitespaces in their names, but fails and it additionally writes a broken output.
Adb Photo Sync
This might not be the answer but might be useful for others looking for android photo/files backup solution.
I use this script on my Windows with git bash. This can be easily used for Linux. A common issue with a long backup process is that it might get interrupted and you might have to restart the entire copy process from start.
This script saves you from this trouble. You can restart the script or interrupt in between but it will resume copy operation from the point it left.
Just change the rfolder => android folder, lfolder => local folder
#!/bin/sh
rfolder=sdcard/DCIM/Camera
lfolder=/f/mylocal/s8-backup/Camera
adb shell ls "$rfolder" > android.files
ls -1 "$lfolder" > local.files
rm -f update.files
touch update.files
while IFS= read -r q; do
# Remove non-printable characters (are not visible on console)
l=$(echo ${q} | sed 's/[^[:print:]]//')
# Populate files to update
if ! grep -q "$l" local.files; then
echo "$l" >> update.files
fi
done < android.files
script_dir=$(pwd)
cd $lfolder
while IFS= read -r q; do
# Remove non-printable characters (are not visible on console)
l=$(echo ${q} | sed 's/[^[:print:]]//')
echo "Get file: $l"
adb pull "$rfolder/$l"
done < "${script_dir}"/update.files
I was wondering about how to create / extract / verify .tar.md5 files. These files are used when flashing images to android devices, see here for example.
As far as I can tell the checksum is appended to the file like this:
cp file.tar file.tar.md5
md5sum file.tar >> file.tar.md5
Firstly I would like to know how to extract the file. Can I simply use tar -xf on the file.tar.md5?
How can I verify the integrity of the file? I would like to remove the last bytes (containing the checksum) from the file to obtain the original file back. I guess you would have to use a regexp to match the checksum file.tar? Is something like this implemented somewhere already?
First of all tar -xf should work since tar continues while it matches its' packing algorithm. If the file stops matching so would tar.
Also most archive managers such as 7-zip or winrar will open it if you remove the ".md5".
They might print error regarding mismatch with the end of the file, ignore it.
As for verifying the file:
print out the stored md5sum: tail -z -n 1 [File name here].tar.md5
calculate the md5sum of the tar part of the file: head -z -n -1 [File name here].tar.md5 | md5sum
What works for me with Ubuntu 19.10 is:
download single-file 4 GiB zip from sammobile com
unzip to several *.tar.md5
run the below command-line
.
for F in *.tar.md5; do echo -n "$F " &&
EXP=($(tail --lines=1 "$F")) &&
ACT=($(head --lines=-1 "$F" | md5sum)) &&
if [ ${EXP[0]} = ${ACT[0]} ]; then echo -n "md5ok " &&
tar --extract --file "$F" && echo "done"
else echo "FAIL"; fi; done &&
unlz4 --multiple --verbose *.lz4
AP_G965U1UEU3ARL1_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT_meta.tar.md5 md5ok done
BL_G965U1UEU3ARL1_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT.tar.md5 md5ok done
CP_G965U1UEU3ARL1_CP11407818_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT.tar.md5 md5ok done
CSC_OMC_OYM_G965U1OYM3ARL1_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT.tar.md5 md5ok done
HOME_CSC_OMC_OYM_G965U1OYM3ARL1_CL14745140_QB21029084_REV01_user_low_ship_MULTI_CERT.tar.md5 md5ok done
…
But we should all try to get away from bash
I want to iterate through all files in a specified directory including hidden files in a script
i tried something like this:
for f in /home/pedja/test/*; do
Which shows only visible files (files that don't start with .)
If i do this:
for f in /home/pedja/test/.*; do
It shows only hidden files (files that starts with .)
How can i list all files in a single for loop
IFS=$'\n'; for f in $(ls -a);do echo "$f"; done
Try this:
for f in `find /home/pedja/test -maxdepth 1`; do
use brace expansion:
for f in /home/pedja/test/{,.}*; do echo "$f"; done
This form runs it on a subshell but that's the safest you can have when your shell does not support process substitution.
find /home/pedja/test/ -mindepth 1 -maxdepth 1 -name '.*' | while read -r file; do
...
done
How can I remove all dirs that start with a prefix, So I have something like this
xyzdirblah
xyzdir2
xyzdir3
xyzdir4
..more
want to do via adb shell , android
Here is the correct command:
rm -r xyzdir*
Jonathan's answer won't work, because adb does not recognize the -f switch.
EDIT: I believe it is not really the point of the question, but if you want to spare files called xyzdir* then use Jonathan's command without -f:
for i in xyzdir*; do [ -d $i ] && rm -r $i; done
rm -rf xyzdir*
rm is the command to delete things.
-r means recursively delete directories.
-f means don't ask questions, and don't complain.
-rf means both of those things together.
xyzdir* expands to the names of files that start with "xyzdir" (including directory names, since in Unix-like systems -- including Android -- directories are really just a special kind of file.)
If there are nondirectory files with the same prefix, like "xyzdir.txt", which you don't want to remove, you have to be more careful:
for i in xyzdir*; do [ -d $i ] && rm -rf $i; done
This loops over every file-or-directory with a name starting with "xyzdir" (the for i in part), checks whether it is a directory (the [ -d $i ] part), and if so (the && part), removes it recursively (the rm -rf $i part).
Right now I am using rm -r /blaa/* to delete all folders and files in the blaa directory. What I am looking for is a way to remove all folders and files in the blaa directory except if the folder is named abc.
Any ideas?
In Linux:
There are many ways to this; however I believe the best way to do this is to simply use the "find" tool.
find ! -iname "abc" -exec rm -rf {} \;
We can easily find and delete the every file and folder that is not named "abc".
find - to find files
! -iname - to filter files/folders, the "!" means not
-exec - to execute a command on every file
rm -rf - remove/delete files -r for folders as well and -f for force
"{} \;" - allows the commands to be used on every file
In Android:
Since you can't use "rm -rf" and when you use "rm -r" it will delete the folder "." which ends up deleting everything.
I am guessing you have "root" on your phone because you can use the "find" tool.
find ! -iname "abc" | sed 1d | xargs rm -r
find - to find files
! -iname - to filter files/folders, the "!" means not
| - pipe sends data to next command
sed - replace text/output
"1d" - removes first line when you do "find ! -iname" by itself
xargs - runs commands after pipe
rm -r - remove/delete files, "-r" for recursive for folders
Edit: Fixed and tested in Android
You can easily change this to suite your needs, please let me know if this helps!
The Adopted Solution
...and the final hoorah... This is what worked for the use case (helps to sum up the comments below as well):
find ! -iname "abc" -maxdepth 1 -depth -print0 | sed '$d' | xargs -0 rm -r;
Notes:
-depth — reverses the output (so you dont delete sub-dirs first
-maxdepth 1 — kind of voids the use of -depth, but hey... this says only output contents of the current directory and not sub-dirs (which are removed by the -r option anyway)
-print0 and -0 — splits on line feeds instead of white space (for dirs with space in the name)
sed "$d" — says to remove the last line (cause its now reversed). The last line is just a period which including would make the call delete everything in the directory (and subs!)
I am sure someone can tighten this up, but it works and was a great learning op!
Thanks again to Jared Burrows(and the Unix community in general — go team!) — MindWire