Develop Reference

Crawling Android File System gets stuck in possible SymLink loop

I'm trying to crawl the entire file system of an android device, both directories and files, without the benefit of NIO, to build a tree of it. If I had NIO then I could use WalkTree or similar, but I don't.
The problem I am having (on the Nexus 5 API 23 x86 emulator) is in /sys/bus/pci/devices and possibly other directories (eg /proc/self) - it doesn't complete before the app times out/quits/crashes (unknown which), possibly getting into some kind of loop or something (the path may change in a repetitive fashion but the canonical path varies little or not at all) .
However if I rule out Symbolic links then that problem goes away but I get what is only some of the files on the device rather than all - for example lacking files under /data (or /data/media/0) and those files not showing up elsewhere - not to mention it looks completely different from the file system that most file managers show. The former is strange as I'd understood Symbolic Links pointed to files and folders that were still present in the file system, but just made them look as if they were elsewhere.
What's the solution? Do I have to code exceptions or special handling for /sys/bus/pci/devices, /proc/self and others? I'd prefer to keep Symbolic Links being followed if I can, and I'd prefer to crawl as many files and folders as I can (so starting in a sub-folder is not preferred).
And a few related questions that might affect the approach I eventually take - if I DO keep SymLinks then does that mean that some things will be crawled twice or more? Is there a way to avoid that? Is there a way to detect when something is the TARGET of a SymLink, other than following the SymLink and checking the CanonicalPath?
Here's my code:
I get the root (I understand that in Android, the first and likely only root is the valid one):
File[] roots = File.listRoots();
String rootPath = "";
try {
rootPath = roots[0].getCanonicalPath();
} catch (IOException e) {
// do something
}
Then I start the crawl (note the boolean to choose whether to ignore simlinks or not):
try {
// check if the rootPath is null or empty, and then...
File rootFile = new File(rootPath);
rootNode = new FileFolderNode(rootFile, null, true, false); // last param may be true to ignore sim links
//FileFolderNode(String filePath, FileFolderNode parent, boolean addChildren, boolean ignoreSimLinks)
} catch (Exception e) {
// do something
}
That uses the FileFolderNode, which has constructor:
public FileFolderNode(File file, FileFolderNode parent, boolean addChildren, boolean ignoreSimLinks) throws IOException {
if (file == null)
throw new IOException("File is null in new FileFolderNode");
if (!file.exists())
throw new IOException("File '" + file.getName() + "' does not exist in new FileFolderNode");
// for now this uses isSymLink() from https://svn.apache.org/repos/asf/commons/_moved_to_git/io/trunk/src/main/java/org/apache/commons/io/FileUtils.java adjusted a bit to remove Java 7 and Windows mentions
if (!ignoreSimLinks)
if (FileUtils.isSymlink(file))
return;
this.name = file.getName();
if (this.name.equals("") && ! file.getCanonicalPath().equals("/"))
throw new IOException("Name is empty in new FileFolderNode");
this.isDirectory = file.isDirectory();
if (this.isDirectory) {
this.children = new ArrayList<FileFolderNode>();
if (addChildren) {
File[] files = file.listFiles();
if (files == null) {
// do something
} else {
// add in children
for (File f : files) {
FileFolderNode child = null;
try {
child = new FileFolderNode(f, this, addChildren, ignoreSimLinks);
} catch (Exception e) {
child = null;
}
if (child != null)
children.add(child);
}
}
}
}
}
Given the lack of answers here, I've broken this question down into areas needing clarification, and am trying to get answers to those - please do see if you can help with those:
Get Android Filing System root
Android SymLinks to hidden or separate locations or partitions
Avoiding Android Symbolic Link loop

Unable to make new folder in Apps cache folder

I have written a caching system. It saves data to the cache folder of the App using platform / Android methods to find the folder. I have an issue that I cannot create a new folder in the Apps cache folder.
This error is reporting in Crashlytics (via non-fatal error reporting) for some devices and not for the majority.
The method that contains the issue is my getCacheFolder() method.
An example folder path that does not work on these devices: /storage/emulated/0/Android/data/app.package.name/cache/stream-cache.
Reported versions affected: 4.4.2; 5.0.1; 6.0, 6.0.1.
Reported devices affected: LENOVO YOGA Tablet 2-1050L, MOTOROLA MotoE2(4G-LTE), Samsung SM-N910F (accounts for 98% of faults reported).
My suspicions are that there might be an issue relating to removable sdcards.
#NonNull
public File getCacheFolder() {
// If this is the first run of this library function, we should get our cache folder
if (mCacheFolder == null || !mCacheFolder.exists()) {
File dir = null;
// Prioritise the external sdcard to store cached files
dir = mContext.get().getExternalCacheDir();
if(dir == null) {
// There was no external sdcard - instead use internal storage
dir = mContext.get().getCacheDir();
}
// Still couldn't get a cache location, go to fallback plan and throw exception
if(dir == null) {
throw new IllegalStateException("Could not create a location to cache. " +
"This leaves the caching library in a bad state.");
}
// Point to our specific caching folder for this system
mCacheFolder = new File(dir, "stream-cache");
}
// If it doesn't exist, we will need to make it
if (!mCacheFolder.exists()) {
// If we cannot make the directory, go to fallback plan and throw an exception
if (!mCacheFolder.mkdirs()) {
// HERE IS WHERE THE FAULT IS REPORTED
throw new IllegalStateException("Could not create a location to cache. " +
"This leaves the caching library in a bad state.: "+mCacheFolder);
}
}
return mCacheFolder;
}
Update 1
I updated the code to write out various information about the state:
#NonNull
public File getCacheFolder() {
if (mCacheFolder == null || !mCacheFolder.exists()) {
File dir = mContext.get().getExternalCacheDir();
if(dir == null || !Utils.isExternalStorageWritable()) {
dir = mContext.get().getCacheDir();
}
mCacheFolder = new File(dir, "stream-cache");
}
if (!mCacheFolder.exists()) {
if (!mCacheFolder.mkdirs()) {
// Problems writing to cache - test writing to file path.
File filePath = mContext.get().getExternalFilesDir(null);
File cachePath = new File(filePath, "stream-cache");
throw new IllegalStateException("Could not create a location to cache. " +
"This leaves the caching library in a bad state.: "+mCacheFolder+ ". " +
"filePath="+filePath+". " +
"cachePath="+cachePath+". " +
"cachePath-exists="+cachePath.exists()+". " +
"cachePath-mkdirs="+cachePath.mkdirs());
}
}
return mCacheFolder;
}
The important results from this:
filePath=null - Context.getExternalFilesDir(null); returns a null. Breaking the rest of the tests.

Instead of using
// Point to our specific caching folder for this system
mCacheFolder = new File(dir, "stream-cache");
try using this:
// Point to our specific caching folder for this system
mCacheFolder = new File(dir + "/stream-cache");
in your first code snippets.

Android service file observer strange behavior

I need to implement a service in android that must be able to monitor a folder to detect a certain file and read what it contains. I'm having a strange behavior with my code and I can't find the reason. This is my relevant code.
public void onCreate(){
lectorFichCSV = new LectorFichCSV(); //object to read CSV files
ftpFileObserver = new FileObserver(filePath.getAbsolutePath()){
public void onEvent(int event, String file) {
if((FileObserver.CREATE & event) != 0){
Log.i("INFO: ", filePath.getAbsolutePath() + "/" + file + " is created");
if(file.substring(0,3).equals("RVE")){ //If file is created and the one I expect
try{
Log.i("INFO: ", "We have a RVE answer");
is = new FileInputStream(filePath + "/" + file);
lineaVent = lectorFichCSV.parseCSVFileAsList(is); //Get information in a list
//Get dao from ORMLite
dao = getHelper().getLineaVentDao();
Iterator<String[]> iterator = lineaVent.iterator();
if(iterator.hasNext()){
String[] aux = iterator.next();
Log.i("INFO:", "CodLineaVent "+aux[0]);
if(aux[2].equals("S")){
//Update DB information accordin to my file
UpdateBuilder<LineaVent, Integer> updateBuilder = dao.updateBuilder();
updateBuilder.where().eq("_id", aux[0]);
updateBuilder.updateColumnValue("valido", true);
updateBuilder.updateColumnValue("saldo", true);
updateBuilder.update();
lineaVent.clear();
}else if(aux[2].equals("N")){
UpdateBuilder<LineaVent, Integer> updateBuilder = dao.updateBuilder();
updateBuilder.where().eq("_id", aux[0]);
updateBuilder.updateColumnValue("saldo", false);
updateBuilder.update();
lineaVent.clear();
}
File fileToDel = new File(filePath + "/" + file);
fileToDel.delete();
}
}catch(FileNotFoundException e){
e.printStackTrace();
}catch(SQLException e){
e.printStackTrace();
}
}
I debugged the code and sometimes is working and sometimes I get lineaVent.size() == 0. I'm going crazy with this, I'm thinking, is it possible that events occurs faster than the creation of my file? that would be the reason when I tried to parse my CSV file into my List object is size = 0? In that case I'm not getting any FileNotFoundException.
Any help will be appreciate. Thank you.

I am not an expert with the inotify POSIX API that, IIRC, underlies FileObserver. However, given that there are separate events for CREATE, MODIFY, and CLOSE_WRITE, it stands to reason that the CREATE event is solely for file creation -- in other words, allocating a new entry in the filesystem for the file. That would either create an empty file, or perhaps a file with some initial load of bytes, but where other MODIFY calls might be needed to write out the full contents. CLOSE_WRITE would then be called to indicate that whoever was writing to the file has now closed their file handle.
Hence, if you are watching for some file to be created, to read it in, watch for CREATE, then watch for CLOSE_WRITE on that same file, and then try to read it, and see if that works better.

Copy the shared preferences XML file from /data on Samsung device failed

There's an exporting feature in my application. It's just a copy operation since all my settings are store in shared preference.
I just copy the xml file from /data/data/package.name/shared_prefs/settings.xml to SD card. It works fine on my HTC desire. However, it might not work on Samsung devices, and i got the following error while I try to copy the file.
I/System.out( 3166): /data/data/package.name/shared_prefs/settings.xml (No such file or directory)
in the directory.
Anyone know how to fix it, or is there another simple way to store the shared preference ?
Thanks.

Never never never never never never never never never hardwire paths.
Unfortunately, there's no getSharedPreferenceDir() anywhere that I can think of. The best solution I can think of will be:
new File(getFilesDir(), "../shared_prefs")
This way if a device manufacturer elects to change partition names, you are covered.
Try this and see if it helps.

CommonsWare's suggestion would a be clever hack, but unfortunately it won't work.
Samsung does not always put the shared_prefs directory in the same parent directory as the getFilesDir().
I'd recommend testing for the existence of (hardcode it, except for package name):
/dbdata/databases/<package_name>/shared_prefs/package.name_preferences.xml and if it exists use it, otherwise fall back to either CommonsWare's suggestion of new File(getFilesDir(), "../shared_prefs") or just /data/data/<package_name>/shared_prefs/package.name_preferences.xml.
A warning though that this method could potentially have problems if a user switched from a Samsung rom to a custom rom without wiping, as the /dbdata/databases file might be unused but still exist.
More details
On some Samsung devices, such as the Galaxy S series running froyo, the setup is this:
/data/data/<package_name>/(lib|files|databases)
Sometimes there's a shared_prefs there too, but it's just Samsung's attempt to confuse you! Don't trust it! (I think it can happen as a left over from a 2.1 upgrade to 2.2, but it might be a left over from users switching roms. I don't really know, I just have both included in my app's bug report interface and sometimes see both files).
And:
/dbdata/databases/<package_name>/shared_prefs
That's the real shared_prefs directory.
However on the Galaxy Tab on Froyo, it's weird. Generally you have: /data/data/<package_name>/(lib|shared_prefs|files|databases)
With no /dbdata/databases/<package_name> directory, but it seems the system apps do have:
/dbdata/databases/<package_name>/yourdatabase.db
And added bonus is that /dbdata/databases/<package_name> is not removed when your app is uninstalled. Good luck using SharedPreferences if the user ever reinstalls your app!

Try using
context.getFilesDir().getParentFile().getAbsolutePath()

Best way to get valid path on all devices - run method Context.getSharedPrefsFile defined as:
/**
* {#hide}
* Return the full path to the shared prefs file for the given prefs group name.
*
* <p>Note: this is not generally useful for applications, since they should
* not be directly accessing the file system.
*/
public abstract File getSharedPrefsFile(String name);
Because of it hidden need use reflection and use fallback on fail:
private File getSharedPrefsFile(String name) {
Context context = ...;
File file = null;
try {
if (Build.VERSION.SDK_INT >= 24) {
try {
Method m = context.getClass().getMethod("getSharedPreferencesPath", new Class[] {String.class});
file = (File)m.invoke(context, new Object[]{name});
} catch (Throwable e) {
Log.w("App TAG", "Failed call getSharedPreferencesPath", e);
}
}
if (file == null) {
Method m = context.getClass().getMethod("getSharedPrefsFile", new Class[] {String.class});
file = (File)m.invoke(context, new Object[]{name});
}
} catch (Throwable e) {
Log.w("App TAG", "Failed call getSharedPrefsFile", e);
file = new File(context.getFilesDir(), "../shared_prefs/" + name + ".xml");
}
return file;
}
On some Samsungs implements like this:
public File getSharedPrefsFile(String paramString) {
return makeFilename(getPreferencesDir(), paramString + ".xml");
}
private File getPreferencesDir() {
synchronized (this.mSync) {
if (this.mPreferencesDir == null) {
this.mPreferencesDir = new File("/dbdata/databases/" + getPackageName() + "/", "shared_prefs");
}
File localFile = this.mPreferencesDir;
return localFile;
}
}
On other Android like this:
public File getSharedPrefsFile(String name) {
return makeFilename(getPreferencesDir(), name + ".xml");
}
private File getPreferencesDir() {
synchronized (mSync) {
if (mPreferencesDir == null) {
mPreferencesDir = new File(getDataDirFile(), "shared_prefs");
}
return mPreferencesDir;
}
}
private File getDataDirFile() {
if (mPackageInfo != null) {
return mPackageInfo.getDataDirFile();
}
throw new RuntimeException("Not supported in system context");
}
After while Google change API for level 24 and later:
https://android.googlesource.com/platform/frameworks/base/+/6a6cdafaec56fcd793214678c7fcc52f0b860cfc%5E%21/core/java/android/app/ContextImpl.java

I've tested in Samsung P1010 with:
//I'm in a IntentService class
File file = this.getDir("shared_prefs", MODE_PRIVATE);
I got:
"/data/data/package.name/app_shared_prefs"
It works fine to me. I can run ffmpeg in this folder.
Look:
Context.getDir

You have to create the shared_prefs directory:
try{
String dir="/data/data/package.name/shared_prefs";
// Create one directory
boolean success = (new File(dir)).mkdirs();
if (success) {
// now copy the file
}
}catch (Exception e){//Catch exception if any
System.err.println("Error: " + e.getMessage());
}
Also... the package of your app is package.name? Make sure you are referring to the right package.

Creating and Storing Log File on device in Android

I am planning to automate the testing of an application by creating a log to store some results of execution of the app and latter on parse it using a piece of python code and plot a graph.
The application is a WiFi fingerprinter i.e it collects info such as mac id, rss(recieved signal strength and rank(normalized rss) about the wifi devices in the surrounding environment. So to test this application I would have to take it to the location and record the results(as of now manually). So logcat wouldn't serve the purpose.
Automation requires
1. Storing the log in the device
2. Access to the log file in the system through usb
Format of the Log file:
Snapshot: 1
Fingerprint: 1, Rank: 0.23424, Boolean: true
Fingerprint: 2, Rank: 0.42344, Boolean: false
Fingerprint: 3, Rank: 0.23425, Boolean: true
Snapshot: 2
Fingerprint: 1, Rank: 0.75654, Boolean: false
Fingerprint: 2, Rank: 0.23456, Boolean: true
Fingerprint: 3, Rank: 0.89423, Boolean: true
................
Now I know there are basically 3 approaches for persistent storage(SharedPrefs wouldn't suit this scenario anyway). I tried Internal Storage, but even after setting the mode of the file as MODE_WORLD_READABLE it was impossible to read the file using Device File Explorer in Eclipse.
I am still wary of using external storage for storing the log. Any tutorial on how to write to a file in usb of the device will definitely help.
I thought of structuring the data to be stored so as to use SQLite for storage. But this establishing many unnecessary relations(foreign and domestic) between data and make it complex. If there is no way around, then here be dragons.
Basically I want to write to a file(easier I suppose) in the device and latter on read it in my system by connecting to it via usb. Any help on how to do it would be much appreciated.

Wary or not, External Storage still may be the only way to go. Without root access on the device, you can't really get at anything "Internal" unless you're going to be okay with reading within an application on the device. The docs provide pretty solid guidelines for where to create external files, and if you are using API Level 8 or higher, there are a couple of extra functions that can be used. I'm sure you know this page, but here it is anyway: http://developer.android.com/guide/topics/data/data-storage.html#filesExternal
If you're in need of any file io example code... I think I could dig some up...
EDIT - I would start by following the guidelines in the above docs to first confirm the state of the storage. I unfortunately don't have any experience with appending a file in Java, so someone else would definitely be more qualified to answer. This doesn't cover appending, but I have a backup routine in one of my personal apps that looks something like this.
File backupPath = Environment.getExternalStorageDirectory();
backupPath = new File(backupPath.getPath() + "/Android/data/com.maximusdev.bankrecord/files");
if(!backupPath.exists()){
backupPath.mkdirs();
}
FileOutputStream fos;
try {
fos = new FileOutputStream(backupPath.getPath() + "/recordsbackup.txt");
if(okaytowrite){
for(int i = 0; i < count; ++i){
entry = adapter.getItem(i);
fos.write(entry.toString().getBytes());
fos.write("\n".getBytes());
fos.write(String.valueOf(entry.dateTime).getBytes());
fos.write("\n".getBytes());
fos.write(String.valueOf(entry.sign).getBytes());
fos.write("\n".getBytes());
fos.write(String.valueOf(entry.cleared).getBytes());
fos.write("\n".getBytes());
fos.write(String.valueOf(entry.transDate).getBytes());
fos.write("\n".getBytes());
fos.write(entry.category.getBytes());
fos.write("\n".getBytes());
}
}
fos.close();
Toast.makeText(this, "Backup Complete", Toast.LENGTH_SHORT).show();
} catch (FileNotFoundException e) {
e.printStackTrace();
AlertDialog.Builder delmessagebuilder = new AlertDialog.Builder(this);
delmessagebuilder.setCancelable(false);
delmessagebuilder.setMessage("File Access Error");
delmessagebuilder.setNeutralButton("Okay", new DialogInterface.OnClickListener() {
public void onClick(DialogInterface dialog, int id) {
dialog.dismiss();
}
});
delmessagebuilder.create().show();
} catch (IOException e) {
e.printStackTrace();
AlertDialog.Builder delmessagebuilder = new AlertDialog.Builder(this);
delmessagebuilder.setCancelable(false);
delmessagebuilder.setMessage("File Access Error");
delmessagebuilder.setNeutralButton("Okay", new DialogInterface.OnClickListener() {
public void onClick(DialogInterface dialog, int id) {
dialog.dismiss();
}
});
delmessagebuilder.create().show();
}
Once I'm ready to write, I'm pulling a custom object (entry) out of an ArrayAdapter (adapter) and converting field valuse to strings and using getBytes() to pass to the FileOutputStream write function. I've done some research and there are quite a few other options for file writing in Java/Android... the FileWriter Class for instance, so it bears further research.

I used a very simple approach to write String messages to the log file by creating a FileWriter object.
public static BufferedWriter out;
private void createFileOnDevice(Boolean append) throws IOException {
/*
* Function to initially create the log file and it also writes the time of creation to file.
*/
File Root = Environment.getExternalStorageDirectory();
if(Root.canWrite()){
File LogFile = new File(Root, "Log.txt");
FileWriter LogWriter = new FileWriter(LogFile, append);
out = new BufferedWriter(LogWriter);
Date date = new Date();
out.write("Logged at" + String.valueOf(date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds() + "\n"));
out.close();
}
}
Now the function to write a new message to the log file.
public void writeToFile(String message){
try {
out.write(message+"\n");
out.close();
} catch (IOException e) {
e.printStackTrace();
}