Develop Reference

Problem with special filenames when trying to pull android files into windows pc recursively using adb

I want to download all files from /data/data/foo folder to my pc. I am using adb to connect with my phone.
cd /data/data/foo
ls -R
.:
cache
databases
files
lib
shared_prefs
./cache:
./databases:
errors.db
errors.db-journal
logs.db
logs.db-journal
./files:
cache_state.info
./shared_prefs:
LANG_CODE.xml
PPP.xml
CADD
.xml
To download files I am using:
adb pull "/data/data/foo" %USERPROFILE%/foo
And here's response:
pull: building file list...
skipping special file 'lib'
pull: /data/data/foo/files/cache_state.info -> C:\Users\root/foo/files/cache_state.info
pull: /data/data/foo/shared_prefs/CADD
.xml -> C:\Users\root/foo/shared_prefs/CADD
.xml
cannot create 'C:\Users\root\foo\shared_prefs\CADD
.xml': No such file or directory
And in fact, only this is copied:
foo/files/cache_state.info
foo/shared_prefs
What I am doing wrong here?

The problem:
What you're doing is right (the command is correct), but the problem comes from the nature of the task you're trying to achieve.
You're trying to copy files from a unix device (an ext filesystem) into your windows pc (with NTFS filesystem). This usually works fine but there are cases when it doesn't: different filesystems have different rules as to which filenames are valid, and if a filename is valid in the source but not in the destination, there's no way to copy it preserving it's name.
From the error I understand you have a file named CADD
.xml (with line break included in the name) that is valid in your android device but will give an error when trying to copy it into your windows filesystem (see Which file systems support newlines in file names?).
Possible solutions:
Copy the files one by one and set a different destination name for the file causing trouble (CADD\n.xml -> CADD.xml)
Get some ext filesystem (either create it in your disk or mount from an external disk) and copy the files there
Copy the file into the ext filesystem used by your in-windows bash shell (see How to Access Your Ubuntu Bash Files in Windows )

How to read a file extension from a variable in Bash?

I am working on a major update for my Android utility that you flash in recovery and it uses Bash Scripting... Problem is, I kinda know nothing about bash!
Right now, I have two files... A common file that contains the variables and a script file that actually holds the file lists.
Here is the problem part of the common script:
if [ ${android_version} == 5.* ]; then
rm -rf /system/$file_name;
else
if [ -e ${file_name}.* ]; then
rm -rf /system/$file_name;
else
rm -f /system/$file_name.apk;
fi;
fi;
And here's an example of where the $file_name's come from:
# AOSP
aosp_remove_list="BootAnimation
Browser
...";
BootAnimation_list="media/bootanimation.zip";
Browser_list="app/Browser
app/ChromeBookmarksSyncAdapter";
So in the common file, it looks to see if it is Lollipop. If not, (this is the issue) it is supposed to look at the file name from the individual app lists and see if it contains a file extension. If it doesn't then it will remove the $file_name + .apk.
So in the above example, Browser's files get .apk added and removed from the system but the bootanimation .zip doesn't get removed.

You should maybe use suffix truncation to perform this.
With ${your_var_name%.zip} you'll get ride of everything after the .zip including the zip, if present of course.
You can try this in your terminal:
a="toto.zip"
echo ${a%.zip} #= > toto
You'll get more details about the possibilities here: http://www.tldp.org/LDP/abs/html/parameter-substitution.html

Gsutil downloading Android review stats csv files as binary

I'm trying to download the review data for my Android app.
But the files gsutil downloads are not text files!
FWIW, I'm running Ubuntu 14.04 with gsutil 4.7.
Below is an example
stats$ gsutil cat gs://pubsite_prod_rev_xxxxxxxxx/stats/ratings/ratings_com.yyyyyyyyyy_201501_overview.csv
��
�#��uO�X���]]GD/ �FP_G�%�fם���9;:��
�� �5�����sv���g�T��S;�����v�jZ}�={H0B�y�cD~M�O���wC���2F����ZI��9�.p�wM�e��p�3���Rj#^�,�²Ԍ��bY�V`�jy���^�X-�p�2��b�&jg8�+�uV�|�~�N߰����xY:}��_��t�F[�,�������F���xY:}�aY�e���|U:

I just ran gsutil for the first time and saw the same thing - something that looks like binary rubbish. Except...
$ gsutil cp gs://pubsite_prod_rev_xxxx/reviews/reviews_com.yyyy.csv .
$ file reviews_com.yyyy.csv
$ reviews_com.yyyy.csv: Little-endian UTF-16 Unicode text, with very long lines
So it is text, you just need to use the right tool to view it. I managed to load it into gedit:
$ gedit reviews_com.yyyy.csv
Then save as something else. I am sure there is a better way.

It seems that the data you have in your bucket, is encrypted or some other format. I tried to download the files in same environment and it works well for me. So i would suggest check your file contents, by just downloading it manually from the bucket.

I just downloaded three .csv review files for our app and noticed the same. I'm running Mac OS X (yosemite/10.10.1) and for me the file is identified as gzipped file:
$ file Dec2014.csv
$ Dec2014.csv: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT)
I managed to turn this into readable file by adding .zip to the end of filename and unzipping it. Then it is a perfectly ok text file for, for example, TextEdit or Numbers.

It's likely utf-16 encoded. You can run this to convert on the Mac:
iconv -f UTF-16 <file>.csv
Google unfortunately makes it much more complicated to get reviews than Apple does with their RSS feeds…

Where to I place the OBB file to test Android Expansion Pack Files (OBB) on my Nexus 4?

I'm trying to test the Expansion Pack Files (OBB) In Android following the guide here:
http://developer.android.com/google/play/expansion-files.html
I'm in the step where I need to test my app on my Nexus 4.
I generated my .obb file with jobb and adb-pushed it in the device in this location:
/mnt/shell/emulated/0/Android/obb/my.package/main.1.my.package.obb
When the app run it doesn't find the file.
Method:
Helpers.doesFileExist(context, fileName, xf.mFileSize, false)
return false for my file.
I debugged and found out it is looking for the file in:
/storage/emulated/0/Android/obb/my.package/main.1.my.package.obb
Specifically this is the path returned by:
Helpers.generateSaveFileName(c, fileName)
The /storage/emulated/0, returned by Environment.getExternalStorageDirectory() doesn't exist browsing the device with adb shell.
But it DOES at runtime, inside the app, I also checked what it contains: it contains almost the same things I found in /mnt/shell/emulated/0, it contains the Android/obb dir, which is empty.
How I found out the path /mnt/shell/emulated/0/Android/obb/my.package/main.1.my.package.obb where I placed my obb file:
$ adb shell
$ ls -ld sdcard
lrwxrwxrwx root root 2013-10-16 17:34 sdcard -> /storage/emulated/legacy
$ ls -ld /storage/emulated/legacy
lrwxrwxrwx root root 2013-10-16 17:34 legacy -> /mnt/shell/emulated/0
And inside that I already found the Android/obb directory, empty.
So the question is: where should I put my obb file for it to be in the right position at runtime?
I did everything said there:
created a draft application in the Market to get the public key
generated a random array of 20 byte (salt)
integrated play_licensing/library and play_apk_expansion/download_library
wrote my Service / Receiver
did the check using the Helpers etc.. exactly like the documentation say.
I suppose everything works but I can't just yet release on Play Store! I need to test locally and I'll have the need to change my obb file pretty often in this initial phase of development.
I can't test on the Emulator because I use 3D and device camera.

Since Android 4.2 multi users support have been added.
To support that Android mount an emulated disk for each users acting as a sandbox layer around the actual filesystem: this let Android handle gracefully either sharing of files between users either personal files.
Long story short:
/storage/emulated
is the emulated filesystem.
if you enter that directory from adb shell you may see a
/storage/emulated/obb
directory. Sometimes it doesn't show up, for some reason (see below for what to do if this happen)
It's not in /Android/obb but that's the right directory where to place your app package / obb file!
If you don't see that directory try looking in:
/mnt/shell/emulated/obb
You should be able to put your file there.
It will be correctly picked up at runtime ending at the
/storage/emulated/0/Android/obb/my.package/main.1.my.package.obb
path.
I think the Android documentation should explain this.
(I answer my own question because I found out how to solve it while writing it.)

For me the correct location is : mnt/sdcard/Android/obb/nameofyourpackage/
NOT "/mnt/shell"

How can I copy the contentsof data/ to sdcard/ without using adb?

Hi I need to copy/move the contents of data/tombstones to sdcard/tombstones
I'm using the command below:
mv data/tombstones /sdcard/tombstones
"failed on 'tombstones' - Cross-device link"
but I'm getting above error.

You have a SANE VERSION of the mv command
paraphrasing a few bits from lbcoder from xda and darkxuser from androidforums
"failed on 'tombstones' - Cross-device link"
It means that you can't create a hard link on one device (filesystem) that refers to a file on a different filesystem.
This is an age-old unix thing. You can NOT move a file across a filesystem using most implementations of mv. mv is not made to copy data from device to device, it simply changes a file's location within a partition. Since /data and /sdcard are different partitions, it's failing.
Consider yourself fortunate that you have a SANE VERSION of the mv command that doesn't try anyway -- some old versions will actually TRY to do this, which will result in a hard link that points to NOTHING, and the original data being INACCESSIBLE.
The mv command does NOT MOVE THE DATA!!! It moves the HARDLINK TO
THE DATA.
If you want to move the file to a different filesystem, you need to use the "cp" command. Copy the file to create a SECOND COPY of it on a different filesystem, and then DELETE the OLD one with the "rm" command.
A simple move command:
#!/bin/bash
dd if="$1" of="$2"
rm -f "$1"
You will note that the "cp" command returns true or false depending on the successful completion of the copy, therefore the original will only be removed IF the file copied successfully.
OR
#!/bin/bash
cat data/tombstones > sdcard/tombstones
rm data/tombstones
These script can be copied into some place referenced by the PATH variable and set executable.
Different Interface
If you need a different interface from adb you may move files using the FileExplorer in DDMS View.
Side note:
You can move a file into a folder when:
You're root;
It is your app directory;
You've used chmod from adb or elsewhere to change permissions

Basically you don't have permission to access /data/tombstones in a production version .
It seems we have to 'root' the device first.
But I failed to root my Samsung S4 which is using Android 4.3