Develop Reference

Volley - download directly to file (no in memory byte array)

I'm using Volley as my network stack in a project I'm working on in Android. Part of my requirements is to download potentially very large files and save them on the file system.
Ive been looking at the implementation of volley, and it seems that the only way volley works is it downloads an entire file into a potentially massive byte array and then defers handling of this byte array to some callback handler.
Since these files can be very large, I'm worried about an out of memory error during the download process.
Is there a way to tell volley to process all bytes from an http input stream directly into a file output stream? Or would this require me to implement my own network object?
I couldn't find any material about this online, so any suggestions would be appreciated.

Okay, so I've come up with a solution which involves editing Volley itself. Here's a walk through:
Network response can't hold a byte array anymore. It needs to hold an input stream. Doing this immediately breaks all request implementations, since they rely on NetworkResponse holding a public byte array member. The least invasive way I found to deal with this is to add a "toByteArray" method inside NetworkResponse, and then do a little refactoring, making any reference to a byte array use this method, rather than the removed byte array member. This means that the transition of the input stream to a byte array happens during the response parsing. I'm not entirely sure what the long term effects of this are, and so some unit testing / community input would be a huge help here. Here's the code:
public class NetworkResponse {
/**
* Creates a new network response.
* #param statusCode the HTTP status code
* #param data Response body
* #param headers Headers returned with this response, or null for none
* #param notModified True if the server returned a 304 and the data was already in cache
*/
public NetworkResponse(int statusCode, inputStream data, Map<String, String> headers,
boolean notModified, ByteArrayPool byteArrayPool, int contentLength) {
this.statusCode = statusCode;
this.data = data;
this.headers = headers;
this.notModified = notModified;
this.byteArrayPool = byteArrayPool;
this.contentLength = contentLength;
}
public NetworkResponse(byte[] data) {
this(HttpStatus.SC_OK, data, Collections.<String, String>emptyMap(), false);
}
public NetworkResponse(byte[] data, Map<String, String> headers) {
this(HttpStatus.SC_OK, data, headers, false);
}
/** The HTTP status code. */
public final int statusCode;
/** Raw data from this response. */
public final InputStream inputStream;
/** Response headers. */
public final Map<String, String> headers;
/** True if the server returned a 304 (Not Modified). */
public final boolean notModified;
public final ByteArrayPool byteArrayPool;
public final int contentLength;
// method taken from BasicNetwork with a few small alterations.
public byte[] toByteArray() throws IOException, ServerError {
PoolingByteArrayOutputStream bytes =
new PoolingByteArrayOutputStream(byteArrayPool, contentLength);
byte[] buffer = null;
try {
if (inputStream == null) {
throw new ServerError();
}
buffer = byteArrayPool.getBuf(1024);
int count;
while ((count = inputStream.read(buffer)) != -1) {
bytes.write(buffer, 0, count);
}
return bytes.toByteArray();
} finally {
try {
// Close the InputStream and release the resources by "consuming the content".
// Not sure what to do about the entity "consumeContent()"... ideas?
inputStream.close();
} catch (IOException e) {
// This can happen if there was an exception above that left the entity in
// an invalid state.
VolleyLog.v("Error occured when calling consumingContent");
}
byteArrayPool.returnBuf(buffer);
bytes.close();
}
}
}
Then to prepare the NetworkResponse, we need to edit the BasicNetwork to create the NetworkResponse correctly (inside BasicNetwork.performRequest):
int contentLength = 0;
if (httpResponse.getEntity() != null)
{
responseContents = httpResponse.getEntity().getContent(); // responseContents is now an InputStream
contentLength = httpResponse.getEntity().getContentLength();
}
...
return new NetworkResponse(statusCode, responseContents, responseHeaders, false, mPool, contentLength);
That's it. Once the data inside network response is an input stream, I can build my own requests which can parse it directly into a file output stream which only hold a small in-memory buffer.
From a few initial tests, this seems to be working alright without harming other components, however a change like this probably requires some more intensive testing & peer reviewing, so I'm going to leave this answer not marked as correct until more people weigh in, or I see it's robust enough to rely on.
Please feel free to comment on this answer and/or post answers yourselves. This feels like a serious flaw in Volley's design, and if you see flaws with this design, or can think of better designs yourselves, I think it would benefit everyone.

Caching ListFragment data

My application generates ListFragments based on user type (determined by the DeviceID) and then fetch images from a web server. These images will then be displayed as a list.
My question is how can I cache those images as well as fragments (generated dynamically) to be displayed in offline mode. For instance when user opens the application without having an active internet connection, it should display the images within the fragments generated dynamically last time.
At the moment my app just download the images from a web service each time.
code for generating fragments dynamically in the MainActivity each time when the application loads.
//generating the views based on JSON data
try {
JSONObject resultObject = new JSONObject(result.toString());
boolean success = resultObject.getBoolean("success");
JSONArray jArray = resultObject.getJSONArray("data");
if (success == true) {
//save menu
for (int i = 0; i < jArray.length(); i++) {
postObject = jArray.getJSONObject(i);
if (postObject.has("ev_count")) {
categoriesSet.put("Events", "Events");
mTabsAdapter.addTab(bar.newTab().setText("Events"), EventsFragment.class, null);
}
if (postObject.has("pl_count")) {
categoriesSet.put("Places", "Places");
mTabsAdapter.addTab(bar.newTab().setText("Places"), PlacesFragment.class, null);
}
if (postObject.has("gn_count")) {
categoriesSet.put("General", "General");
mTabsAdapter.addTab(bar.newTab().setText("General"), GeneralFragment.class, null);
}
}
}
//saving values to the shared preferences (hashmap as a string)
categoriesPreferences = PreferenceManager.getDefaultSharedPreferences(this);
SharedPreferences.Editor editor = categoriesPreferences.edit();
editor.putString("categories", categoriesSet.toString());
editor.commit();
}catch (JSONException e) {
Log.e("ALLOCATE_DAT_ERROR", e.toString());
}

I would just advice to use some image downloading library which also often handles caching for you..
Here is a short list of some of them:
Volley - directly from google, its more like a whole networking stack, but it allows you to easily download Images (see NetworkImageView) and also to cache them (you need to provide a cache implemtantion - google will help you)
Picasso - nice library from square with very straightforward API - I would advice you to go with it, it might be the easiest way to go
Universal Image Loader - another option, it has really a lot of options and is also easy to use
Here is a simple sample code which should advice you how to use volley to load and cache images.
NetworkImageView mage = (NetworkImageView) view.findViewById(...);
image.setImageUrl("http://someurl.com/image.png",mImageLoader);
You need an ImageLoader instance then..
public ImageLoader getImageLoader() {
if (mImageLoader == null) {
mImageLoader = new ImageLoader(getImageRequestQueue(), new DiskBitmapCache(getCacheDir(), 50 * 1024 * 1024));
}
return mImageLoader;
}
imageRequestQueue is standard queue you should have already initialised somewhere in your app if already using volley for networking stuff
As a DiskCache you can use this

How to correctly use com.octo.android.robospice to cache bitmaps

I'm using the example found here:
https://github.com/octo-online/robospice/wiki/Starter-Guide
and have attempted to extend it to support bitmap requests with caching. With LogCat set to verbose, I see log messages indicating that the bitmaps are being put into cache successfully, but attempts to read the bitmaps back from cache on subsequent calls are failing with the following log message:
11-15 09:14:02.694: D//RequestRunner.java:102(5462): com.octo.android.robospice.persistence.exception.CacheLoadingException: org.codehaus.jackson.map.JsonMappingException: No suitable constructor found for type [simple type, class android.graphics.Bitmap]: can not instantiate from JSON object (need to add/enable type information?)
Here's the relevant code from the JsonSpiceFactory:
#Override
public CacheManager createCacheManager(Application application) {
CacheManager cacheManager = new CacheManager();
try {
final int maxMemory = (int) (Runtime.getRuntime().maxMemory() / 1024);
final int cacheSize = maxMemory / 8;
JacksonObjectPersisterFactory jacksonObjectPersisterFactory = new JacksonObjectPersisterFactory(application);
cacheManager.addPersister( jacksonObjectPersisterFactory );
// Try to add support for bitmap caching here
InFileBitmapObjectPersister filePersister = new InFileBitmapObjectPersister(application);
LruCacheBitmapObjectPersister memoryPersister = new LruCacheBitmapObjectPersister(filePersister, cacheSize);
cacheManager.addPersister(memoryPersister);
} catch (CacheCreationException e) {
e.printStackTrace();
}
return cacheManager;
}
And here's the code that invokes the spiceManager call to load the image from cache or network:
public void getNetworkBitmap(String url, String cacheKey, GetBitmapListener listener) {
GetMerchantLogoRequest request = new GetMerchantLogoRequest(url);
spiceManager.execute(request, cacheKey, DurationInMillis.ONE_HOUR, new NetworkBitmapRequestListener(listener));
}
private class NetworkBitmapRequestListener implements RequestListener<Bitmap> {
private GetBitmapListener listener;
public NetworkBitmapRequestListener(GetBitmapListener listener) {
this.listener = listener;
}
#Override
public void onRequestFailure(SpiceException spiceException) {
listener.onImageRequestFailure(spiceException);
}
#Override
public void onRequestSuccess(Bitmap logo) {
listener.onImageRequestSuccess(logo);
}
}
Do I need a factory to reconstitute the bitmap from the cache data? Since this isn't a Json object, what do I need to do to correct the attempt to use Jackson Json mapping? Will Jackson do the right thing if I add/enable type information? How do I do that?
Thanks for reading this far!
-Eric

I found the answer reading through the CacheManager (link below) and stumbled over the line "All elements [persisters] ... are questioned in order" so I re-ordered the entries in the CreateCacheManager method and now it's working.
http://grepcode.com/file/repo1.maven.org/maven2/com.octo.android.robospice/robospice-cache/1.3.1/com/octo/android/robospice/persistence/CacheManager.java

you can use BitmapRequest, here is an example:
https://github.com/octo-online/robospice/blob/release/robospice-core-parent/robospice-core-test/src/main/java/com/octo/android/robospice/request/BitmapRequestTest.java?source=cc

Volley: How to set up a Cache.Entry

I am using Google's Volley Library as my design for getting network data;
I have set up a RequestQueue
requestQueue = new RequestQueue(new DiskBasedCache(new File(context.getCacheDir(),
DEFAULT_CACHE_DIR)), new BasicNetwork(new
HttpClientStack(AndroidHttpClient.newInstance(userAgent))));
I have also subclassed Request, and have data coming back from the network just fine. My issue is with caching: in parseNetworkResponse() which is overridden in my subclass of Request, when I call
return Response.success(list, HttpHeaderParser.parseCacheHeaders(response));
HttpHeaderParser.parseCacheHeaders(response) returns null since the server is set up for "no caching" in its response header... Regardless I still would like to cache this data for a variable set number of hours (24 hours probably), How can I do this by creating a volley Cache.Entry... It is my understanding that the URL is used as the cache key value (and I would like it to be the URL).
To sum up, since HttpHeaderParser.parseCacheHeaders(response) returns null, I would like to create a new Cache.Entry that is set up for expiring after 24 hours, and the cache key being the URL of the request.
Any thoughts?
Thanks!

I've had the same issue and ended up with this solution:
#Override
protected Response<String> parseNetworkResponse(NetworkResponse response) {
// Create a FakeCache that invalidate the data after 24 hour
Cache.Entry mFakeCache = HttpHeaderParser.parseCacheHeaders(response);
mFakeCache.etag = null;
mFakeCache.softTtl = System.currentTimeMillis() + 86400 * 1000;
mFakeCache.ttl = mFakeCache.softTtl;
return Response.success(response.data, mFakeCache);
}

Create a fast file search function in android

I'm new to android and I'm trying to develop file explorer which includes search function. I'm using a recursive search function that works fine in folders with a few subfolders and files, but for some reason it's EXTREMELY SLOW and could "Force Close" in folders with lots of subfolders and files, because there's not enough memory. I do the search by creating ArrayList where the results will be placed, and then calling the recursive function that will fill the list. The "path" argument is the file where the search will start from, and "query" is the search query.
ArrayList<File> result = new ArrayList<File>();
fileSearch(path, query, result);
this is what the recursive function looks like:
private void fileSearch(File dir, String query, ArrayList<File> res) {
if (dir.getName().toLowerCase().contains(query.toLowerCase()))
res.add(dir);
if (dir.isDirectory() && !dir.isHidden()) {
if (dir.list() != null) {
for (File item : dir.listFiles()) {
fileSearch(item, query, res);
}
}
}
}
If someone could point me to a way of performing a faster and/or more efficient file search, I would really appreciate that.
EDIT:
This is how I tried to do the job with AsyncTask:
private class Search extends AsyncTask<File, Integer, Void> {
String query;
ArrayList<File> result = new ArrayList<File>();
public Search(String query){
this.query = query;
setTitle("Searching");
}
#Override
protected Void doInBackground(File... item) {
int count = item.length;
for (int i = 0; i < count; i++) {
fileSearch(item[i], query, result);
publishProgress((int) ((i / (float) count) * 100));
// Escape early if cancel() is called
if (isCancelled()) break;
}
return null;
}
protected void onProgressUpdate(Integer... progress) {
setProgress(progress[0]);
}
protected void onPostExecute() {
searchResults = new ListItemDetails[result.size()];
for (int i = 0; i < result.size(); i++) {
File temp = result.get(i);
if (temp.isDirectory())
searchResults[i] = new ListItemDetails(temp.getAbsolutePath(),
R.drawable.folder, temp.lastModified(), temp.length());
else {
String ext;
if (temp.getName().lastIndexOf('.') == -1)
ext = "";
else
ext = temp.getName().substring(
temp.getName().lastIndexOf('.'));
searchResults[i] = new ListItemDetails(temp.getAbsolutePath(),
getIcon(ext), temp.lastModified(), temp.length());
}
}
finishSearch();
}
}
public void finishSearch() {
Intent intent = new Intent(this, SearchResults.class);
startActivity(intent);
}
The call to finishSearch() is just so I can create the Intent to show the results in other Activity. Any ideas, suggestions, tips? Thanks in advance

It is possible that you are hitting symbolic links and going into an infinitive loop with your search function and depleting available memory to your application.
I would suggest you to keep a separate list containing canonical paths (File.getCanonicalPath()) of directories you've visited and avoid visiting them over and over again.

Why don't you use Apache Commons IO? It has some functions to deal with searching.
I also suggest using the method FileUtils.listFiles, which takes a folder, a search query and a directory filter as parameters.
The following example returns you a list of all file's paths that matched according to a regex. Try adding it in doInBackground of your AsyncTask:
Collection files = FileUtils.listFiles(new File(yourRootPath),
new RegexFileFilter(searchQuery),
DirectoryFileFilter.DIRECTORY);

Have you looked into Lucene?
It is especially designed to index and query large numbers of free-text documents, so many of the I/O streaming and indexing tasks have already been solved for you. If you remove the recursion and do the document indexing using a Lucene index in a purely iterative fashion, memory issues may be mitigated.
Look into this thread:
Lucene in Android

Do it in the background, and starting from Android O (API 26) , you can use Files.find API. Exmaple:
Files.find(
Paths.get(startPath), Integer.MAX_VALUE,
{ path, _ -> path.fileName.toString() == file.name }
).forEach { foundPath ->
Log.d("AppLog", "found file on:${foundPath.toFile().absolutePath}")
}