Develop Reference

Crawling Android File System gets stuck in possible SymLink loop

I'm trying to crawl the entire file system of an android device, both directories and files, without the benefit of NIO, to build a tree of it. If I had NIO then I could use WalkTree or similar, but I don't.
The problem I am having (on the Nexus 5 API 23 x86 emulator) is in /sys/bus/pci/devices and possibly other directories (eg /proc/self) - it doesn't complete before the app times out/quits/crashes (unknown which), possibly getting into some kind of loop or something (the path may change in a repetitive fashion but the canonical path varies little or not at all) .
However if I rule out Symbolic links then that problem goes away but I get what is only some of the files on the device rather than all - for example lacking files under /data (or /data/media/0) and those files not showing up elsewhere - not to mention it looks completely different from the file system that most file managers show. The former is strange as I'd understood Symbolic Links pointed to files and folders that were still present in the file system, but just made them look as if they were elsewhere.
What's the solution? Do I have to code exceptions or special handling for /sys/bus/pci/devices, /proc/self and others? I'd prefer to keep Symbolic Links being followed if I can, and I'd prefer to crawl as many files and folders as I can (so starting in a sub-folder is not preferred).
And a few related questions that might affect the approach I eventually take - if I DO keep SymLinks then does that mean that some things will be crawled twice or more? Is there a way to avoid that? Is there a way to detect when something is the TARGET of a SymLink, other than following the SymLink and checking the CanonicalPath?
Here's my code:
I get the root (I understand that in Android, the first and likely only root is the valid one):
File[] roots = File.listRoots();
String rootPath = "";
try {
rootPath = roots[0].getCanonicalPath();
} catch (IOException e) {
// do something
}
Then I start the crawl (note the boolean to choose whether to ignore simlinks or not):
try {
// check if the rootPath is null or empty, and then...
File rootFile = new File(rootPath);
rootNode = new FileFolderNode(rootFile, null, true, false); // last param may be true to ignore sim links
//FileFolderNode(String filePath, FileFolderNode parent, boolean addChildren, boolean ignoreSimLinks)
} catch (Exception e) {
// do something
}
That uses the FileFolderNode, which has constructor:
public FileFolderNode(File file, FileFolderNode parent, boolean addChildren, boolean ignoreSimLinks) throws IOException {
if (file == null)
throw new IOException("File is null in new FileFolderNode");
if (!file.exists())
throw new IOException("File '" + file.getName() + "' does not exist in new FileFolderNode");
// for now this uses isSymLink() from https://svn.apache.org/repos/asf/commons/_moved_to_git/io/trunk/src/main/java/org/apache/commons/io/FileUtils.java adjusted a bit to remove Java 7 and Windows mentions
if (!ignoreSimLinks)
if (FileUtils.isSymlink(file))
return;
this.name = file.getName();
if (this.name.equals("") && ! file.getCanonicalPath().equals("/"))
throw new IOException("Name is empty in new FileFolderNode");
this.isDirectory = file.isDirectory();
if (this.isDirectory) {
this.children = new ArrayList<FileFolderNode>();
if (addChildren) {
File[] files = file.listFiles();
if (files == null) {
// do something
} else {
// add in children
for (File f : files) {
FileFolderNode child = null;
try {
child = new FileFolderNode(f, this, addChildren, ignoreSimLinks);
} catch (Exception e) {
child = null;
}
if (child != null)
children.add(child);
}
}
}
}
}
Given the lack of answers here, I've broken this question down into areas needing clarification, and am trying to get answers to those - please do see if you can help with those:
Get Android Filing System root
Android SymLinks to hidden or separate locations or partitions
Avoiding Android Symbolic Link loop

Google Drive API - get list of files including folders

I need to list all images in a users drive cloud.
I use https://www.googleapis.com/drive/v2/files with the correct filter to query all images. I need to group the result in my app in folders. I know a file can have multiple parents, that's fine.
I would like to avoid making and calls (for every single file a single call) to get a files folder via https://developers.google.com/drive/v2/reference/files/get using the parent id from the first call.
Is there a network friendly way to get all files inclusive there folders?
EDIT
A simple solution would be to get all folders with ids in one query and lookup folder names in this result. Maybe that's somehow possible?

As you answered yourself in the comments above (but you can't match names, you have to match IDs; names aren't unique).
Step 1: get all your folders in one shot (paging results, filtering unneeded fields, skipping the trashed ones):
private static Drive mGOOSvc;
....
static ArrayList<ContentValues> getFolders() {
ArrayList<ContentValues> cvs = new ArrayList<>();
if (mGOOSvc != null) try {
Drive.Files.List qry = mGOOSvc.files().list()
.setQ("mimeType = 'application/vnd.google-apps.folder'")
.setFields("items(id,labels/trashed,parents/id,title),nextPageToken");
String npTok = null;
if (qry != null) do {
FileList gLst = qry.execute();
if (gLst != null) {
for (File gFl : gLst.getItems()) {
if (gFl.getLabels().getTrashed()) continue;
for (ParentReference parent : gFl.getParents())
cvs.add(newContentValues(gFl.getTitle(), gFl.getId(), parent.getId()));
}
npTok = gLst.getNextPageToken();
qry.setPageToken(npTok);
}
} while (npTok != null && npTok.length() > 0);
} catch (Exception e) { /* handle Exceptions */ }
return cvs;
}
Step 2: Parse the resulting ArrayList to build the tree structure (match ParentIds, handle multiple parents)
Step 3: Do the same for files with mime type ""image/jpeg", "image/png", ... "whatever img mimetype" (just modify the code above to get files) and parse again.
Of course the 'execute()' method will produce exceptions that should be handled as pointed out here.
... and you can take the 'not so network friendly' approach of iterating down the folder tree as seen in the 'testTree()' method here. Recursion is necessary if you have no knowledge how deep your tree structure is.
Good Luck

SharedPreferences in Android not persisted to disk when key contains newline

In Android, I'd like to write SharedPreferences key-value pairs where the keys are Base64 strings.
// get a SharedPreferences instance
SharedPreferences prefs = getSharedPreferences("some-name", Context.MODE_PRIVATE);
// generate the base64 key
String someKey = new String(Base64.encode("some-key".getBytes("UTF-8"), Base64.URL_SAFE), "UTF-8");
// write the value for the generated key
prefs.edit().putBoolean(someKey, true).commit();
In the last line, the call to commit returns true. So this key-value pair should have been saved successfully.
When I close and destroy the Activity where this piece of code was used and then re-create the Activity (running this code again), the specified value is returned for the key that we used.
But it turns out that, when I destroy the whole application/process (e.g. using "Force stop" in app settings), the value for our key is lost on the next launch of the Activity.
When I don't use Base64.URL_SAFE but Base64.URL_SAFE | Base64.NO_WRAP as the flags for the Base64 encoding, it works fine.
So this problem has been caused by the newlines at the end of the Base64 keys. Keys like abc can be written without any problems. But when the key is abc\n, it fails.
The problem is that it appears to work without problems first, returning true on commit() and returning the correct preference value on subsequent calls. But when the whole application is destroyed and re-started, the value has not been persisted.
Is this intended behaviour? A bug? Does the documentation say anything about valid key names?

I took a look at GrepCode and see that the operations will be the following (I do not mention useless ones) :
android.app.SharedPreferencesImpl.commit()
android.app.SharedPreferencesImpl.commitToMemory()
android.app.SharedPreferencesImpl.queueDiskWrite(MemoryCommitResult,Runnable)
3.1. XmlUtils.writeMapXml(Map, OutputStream)
3.2. XmlUtils.writeMapXml(Map, String, XmlSerializer)
3.3. XmlUtils.writeValueXml(Object v, String name, XmlSerializer ser)
First : how your data are converted ?
The method XmlUtils.writeValueXml writes the Object value in a XML tag with the attribute name set to the String value. This String value contains exactly the value you specified at the SharedPreference's name.
(And I confirmed this by doing a step-by-step debug with your piece of code).
The XML will be with an unescaped line break character. Actually, the XmlSerializer instance is a FastXmlSerializer instance and it does not escape the \n character (see the link for this class at the end if you want to read the source code)
Interesting piece of code :
writeValueXml(Object v, String name, XmlSerializer out) {
// -- "useless" code skipped
out.startTag(null, typeStr);
if (name != null) {
out.attribute(null, "name", name);
}
out.attribute(null, "value", v.toString());
out.endTag(null, typeStr);
// -- "useless" code skipped
}
Second : why the result is true ?
The commit method has the following code :
public boolean commit() {
MemoryCommitResult mcr = commitToMemory();
SharedPreferencesImpl.this.enqueueDiskWrite(
mcr, null /* sync write on this thread okay */);
try {
mcr.writtenToDiskLatch.await();
} catch (InterruptedException e) {
return false;
}
notifyListeners(mcr);
return mcr.writeToDiskResult;
}
So it returns the mcr.writeToDiskResult which is set in the SharedPreferencesImpl.writeToFile(MemoryCommitResult) method. Interesting piece of code :
writeToFile(MemoryCommitResult mcr) {
// -- "useless" code skipped
try {
FileOutputStream str = createFileOutputStream(mFile);
if (str == null) {
mcr.setDiskWriteResult(false);
return;
}
XmlUtils.writeMapXml(mcr.mapToWriteToDisk, str);
FileUtils.sync(str);
str.close();
ContextImpl.setFilePermissionsFromMode(mFile.getPath(), mMode, 0);
try {
final StructStat stat = Libcore.os.stat(mFile.getPath());
synchronized (this) {
mStatTimestamp = stat.st_mtime;
mStatSize = stat.st_size;
}
} catch (ErrnoException e) {
// Do nothing
}
// Writing was successful, delete the backup file if there is one.
mBackupFile.delete();
mcr.setDiskWriteResult(true);
return;
} catch (XmlPullParserException e) {
Log.w(TAG, "writeToFile: Got exception:", e);
} catch (IOException e) {
Log.w(TAG, "writeToFile: Got exception:", e);
}
// -- "useless" code skipped
}
As we see at the previous point : the XML writing is "ok" (do not throw anything, do not fails), so the sync in the file will be too (just a copy of a Stream in another one, nothing checks the XML content here !).
Currently : your key was converted to (badly formatted) XML and correctly wrote in a File. The result of the whole operation is true as everything went OK. Your changes are comitted to the disk and in the memory.
Third and last : why do I get back the correct value the first time and a bad one the second time
Take a quick look to what happen when we commit the changes to memory in SharedPreferences.Editor.commitToMemory(...) method (interesting part only... :)):
for (Map.Entry<String, Object> e : mModified.entrySet()) {
String k = e.getKey();
Object v = e.getValue();
if (v == this) { // magic value for a removal mutation
if (!mMap.containsKey(k)) {
continue;
}
mMap.remove(k);
} else {
boolean isSame = false;
if (mMap.containsKey(k)) {
Object existingValue = mMap.get(k);
if (existingValue != null && existingValue.equals(v)) {
continue;
}
}
mMap.put(k, v);
}
mcr.changesMade = true;
if (hasListeners) {
mcr.keysModified.add(k);
}
}
Important point : the changes are commited to the mMap attribute.
Then, take a quick look to how we get back a value :
public boolean getBoolean(String key, boolean defValue) {
synchronized (this) {
awaitLoadedLocked();
Boolean v = (Boolean)mMap.get(key);
return v != null ? v : defValue;
}
}
We are taking back the key from mMap (no reading of the value in the file for now). So we have the correct value for this time :)
When you reload your application, you will load the data back from the disk, and so the SharedPreferencesImpl constructor will be called, and it will call the SharedPreferencesImpl.loadFromDiskLocked() method. This method will read the file content and load it in the mMap attribute (I let you see the code by yourself, link provided at the end).
A step-by-step debug shown me that the abc\n was written as abc (with a whitespace character). So, when you will try to get it back, you will never succeed.
To finish, thank you to #CommonsWare to give me a hint about the file content in the comment :)
Links
XmlUtils
FastXmlSerializer
SharedPreferencesImpl
SharedPreferencesImpl.EditorImpl.commit()
SharedPreferencesImpl.EditorImpl.commitToMemory()
SharedPreferencesImpl.enqueueDiskWrite(MemoryCommitResult, Runnable)
SharedPreferencesImpl.writeToFile(MemoryCommitResult)
SharedPreferencesImpl.loadFromDiskLocked()

Parsing local gpx file in Android

I followed this example to parse a local GPX file in Android:
http://android-coding.blogspot.pt/2013/01/get-latitude-and-longitude-from-gpx-file.html
All works fine to access "lat" and "long" but I need also to get the "ele" value but all my tentatives were unsuccessful.
Anyone can give me some hits to do that?
Thanks in advance!
Best regards,
NR.

I will add my library for GPX parsing to these answers: https://github.com/ticofab/android-gpx-parser. It provides two ways to parse you GPX file: once you obtain / create a GPXParser object (mParser in the examples below), you can then either parse directly your GPX file
Gpx parsedGpx = null;
try {
InputStream in = getAssets().open("test.gpx");
parsedGpx = mParser.parse(in);
} catch (IOException | XmlPullParserException e) {
e.printStackTrace();
}
if (parsedGpx == null) {
// error parsing track
} else {
// do something with the parsed track
}
or you can parse a remote file:
mParser.parse("http://myserver.com/track.gpx", new GpxFetchedAndParsed() {
#Override
public void onGpxFetchedAndParsed(Gpx gpx) {
if (gpx == null) {
// error parsing track
} else {
// do something with the parsed track
}
}
});
Contributions are welcome.

you have the "Node node = nodelist_trkpt.item(i);" in your first loop.
Get the child elements from this node an run through these child elements.
e.g.:
NodeList nList = node.getChildNodes();
for(int j=0; j<nList.getLength(); j++) {
Node el = nList.item(j);
if(el.getNodeName().equals("ele")) {
System.out.println(el.getTextContent());
}
}

Update: I've added parsing "ele" element as well, so this code could match your requirements.
I will propose different approach: https://gist.github.com/kamituel/6465125.
In my approach I don't create an ArrayList of all track points (this is done in the example you posted). Such a list can consume quite a lot of memory, which can be an issue on Android.
I've even given up on using regex parsing to avoid allocating too many objects (which causes garbage collector to run).
As a result, running Java with 16Mb heap size, parsing GPX file with over 600 points, garbage collector will be run only 12 times. I'm sure one could go lower, but I didn't optimize it heavily yet.
Usage:
GpxParser parser = new GpxParser(new FileInputStream(file));
TrkPt point = null;
while ((point = parser.nextTrkPt()) != null) {
// point.getLat()
// point.getLon()
}
I've successfully used this code to parse around 100 Mb of GPX files on Android. Sorry it's not in the regular repo, I didn't plan to share it just yet.

I've ported the library GPXParser by ghitabot to Android.
https://github.com/urizev/j4gpx

Not finding local data files saved in my application

The process seemed quite simplistic at first, but there must be something that I am missing going forward with this task. There was a settings file that I wanted to create local to my application for storing a whole bunch of data (not preference worthy). I ended up saving the file with the following code snippet.
protected File createSettingsFileLocation(String fileNameF)
{
File directoryFile = context_.getDir("settings", Context.MODE_PRIVATE);
File settingsFile;
settingsFile = new File(directoryFile, fileNameF);
if (!settingsFile.exists())
{
try
{
settingsFile.createNewFile();
} catch(IOException e)
{
Log.e(MyConstants.LOG_TAG, "Could not create the file as intended within internal storage.");
return null;
}
}
return settingsFile;
}
and then proceeded to retrieve the file later by looking for it locally with the following code snippets.
public String getCurrentFileContainingSettings()
{
List<String >settingFilesInFolder = getLocalStorageFileNames();
if (settingFilesInFolder == null || settingFilesInFolder.isEmpty())
{
return null;
}
String pathToCurrentSettingsFile = settingFilesInFolder.get(0);
return pathToCurrentSettingsFile;
}
protected List<String> getLocalStorageFileNames()
{
return Arrays.asList(context_.fileList());
}
However, the settingFilesInFolder always returns no entries, so I get null back from the getCurrentFileContainingSettings(). As what I could see from the documentation it seems as thought I was doing it right. But, I must be missing something, so I was hoping that someone could point something out to me. I could potentially hard-code the file name once it has been created within the system in a preference file for access later the first time that the settings are created, but I shouldn't have to do something like that I would think.

fileList() only looks in getFilesDir(), not in its subdirectories, such as the one you created via getDir(). Use standard Java file I/O (e.g., list()) instead.