I'm using Volley as my network stack in a project I'm working on in Android. Part of my requirements is to download potentially very large files and save them on the file system.
Ive been looking at the implementation of volley, and it seems that the only way volley works is it downloads an entire file into a potentially massive byte array and then defers handling of this byte array to some callback handler.
Since these files can be very large, I'm worried about an out of memory error during the download process.
Is there a way to tell volley to process all bytes from an http input stream directly into a file output stream? Or would this require me to implement my own network object?
I couldn't find any material about this online, so any suggestions would be appreciated.
Okay, so I've come up with a solution which involves editing Volley itself. Here's a walk through:
Network response can't hold a byte array anymore. It needs to hold an input stream. Doing this immediately breaks all request implementations, since they rely on NetworkResponse holding a public byte array member. The least invasive way I found to deal with this is to add a "toByteArray" method inside NetworkResponse, and then do a little refactoring, making any reference to a byte array use this method, rather than the removed byte array member. This means that the transition of the input stream to a byte array happens during the response parsing. I'm not entirely sure what the long term effects of this are, and so some unit testing / community input would be a huge help here. Here's the code:
public class NetworkResponse {
/**
* Creates a new network response.
* #param statusCode the HTTP status code
* #param data Response body
* #param headers Headers returned with this response, or null for none
* #param notModified True if the server returned a 304 and the data was already in cache
*/
public NetworkResponse(int statusCode, inputStream data, Map<String, String> headers,
boolean notModified, ByteArrayPool byteArrayPool, int contentLength) {
this.statusCode = statusCode;
this.data = data;
this.headers = headers;
this.notModified = notModified;
this.byteArrayPool = byteArrayPool;
this.contentLength = contentLength;
}
public NetworkResponse(byte[] data) {
this(HttpStatus.SC_OK, data, Collections.<String, String>emptyMap(), false);
}
public NetworkResponse(byte[] data, Map<String, String> headers) {
this(HttpStatus.SC_OK, data, headers, false);
}
/** The HTTP status code. */
public final int statusCode;
/** Raw data from this response. */
public final InputStream inputStream;
/** Response headers. */
public final Map<String, String> headers;
/** True if the server returned a 304 (Not Modified). */
public final boolean notModified;
public final ByteArrayPool byteArrayPool;
public final int contentLength;
// method taken from BasicNetwork with a few small alterations.
public byte[] toByteArray() throws IOException, ServerError {
PoolingByteArrayOutputStream bytes =
new PoolingByteArrayOutputStream(byteArrayPool, contentLength);
byte[] buffer = null;
try {
if (inputStream == null) {
throw new ServerError();
}
buffer = byteArrayPool.getBuf(1024);
int count;
while ((count = inputStream.read(buffer)) != -1) {
bytes.write(buffer, 0, count);
}
return bytes.toByteArray();
} finally {
try {
// Close the InputStream and release the resources by "consuming the content".
// Not sure what to do about the entity "consumeContent()"... ideas?
inputStream.close();
} catch (IOException e) {
// This can happen if there was an exception above that left the entity in
// an invalid state.
VolleyLog.v("Error occured when calling consumingContent");
}
byteArrayPool.returnBuf(buffer);
bytes.close();
}
}
}
Then to prepare the NetworkResponse, we need to edit the BasicNetwork to create the NetworkResponse correctly (inside BasicNetwork.performRequest):
int contentLength = 0;
if (httpResponse.getEntity() != null)
{
responseContents = httpResponse.getEntity().getContent(); // responseContents is now an InputStream
contentLength = httpResponse.getEntity().getContentLength();
}
...
return new NetworkResponse(statusCode, responseContents, responseHeaders, false, mPool, contentLength);
That's it. Once the data inside network response is an input stream, I can build my own requests which can parse it directly into a file output stream which only hold a small in-memory buffer.
From a few initial tests, this seems to be working alright without harming other components, however a change like this probably requires some more intensive testing & peer reviewing, so I'm going to leave this answer not marked as correct until more people weigh in, or I see it's robust enough to rely on.
Please feel free to comment on this answer and/or post answers yourselves. This feels like a serious flaw in Volley's design, and if you see flaws with this design, or can think of better designs yourselves, I think it would benefit everyone.
Related
EDIT: This is a bug in Android version <4.3 Kitkat. It relates to the libjpeg library in Android, which can't handle JPEGs with missing EOF/EOI bits, or apparently with metadata/EXIF data that it doesn't like.
https://code.google.com/p/android/issues/detail?id=9064
ORIGINAL QUESTION:
I have an issue when loading an image in my app.
My endpoint sends JSON which contains a BASE64 encoded image. Depending on the REST call, these images can be PNG or JPG. Some of the JPG files suffer from an issue where they are missing an EOF bit at the end. The PNG files work, and some JPG files work, but unfortunately a lot of these JPG files with the issue are present in the Oracle DB (stored as BLOB). I don't have control of the DB.
I have been looking through Google bugs here:
https://code.google.com/p/android/issues/detail?id=9064
and here:
https://code.google.com/p/android/issues/detail?id=57502
The issue is also seen where the encoding is CYMK using a custom ICC profile.
Decoding the image the standard way returns false:
byte[] imageAsBytes = Base64.decode(base64ImageString, Base64.DEFAULT);
return BitmapFactory.decodeByteArray(imageAsBytes, 0, imageAsBytes.length);
According to the bug reports above, the built in JPG parser in Android is to blame.
I'm trying to figure out a workaround for my device, which is stuck on 4.2.2. I have no other option on this OS version.
I thought it might be a good idea to try and use an image loader library like Universal Image Loader, but it requires I either have the image stored locally, or stored on a URL. As I get the data in BASE64 from the REST server, I can't use this. An option is to support decodeByteArray in a custom class that extends BaseImageDecoder, as stated by the dev at the bottom here: https://github.com/nostra13/Android-Universal-Image-Loader/issues/209
Here's where I get stuck. I already have a custom image decoder to try handle the issue of the missing EOF marker in the JPG file, but I don't know how to edit it to add support for decodeByteArray.
Here is my CustomImageDecoder:
public class CustomImageDecoder extends BaseImageDecoder {
public CustomImageDecoder(boolean loggingEnabled) {
super(loggingEnabled);
}
#Override
protected InputStream getImageStream(ImageDecodingInfo decodingInfo) throws IOException {
InputStream stream = decodingInfo.getDownloader()
.getStream(decodingInfo.getImageUri(), decodingInfo.getExtraForDownloader());
return stream == null ? null : new JpegClosedInputStream(stream);
}
private class JpegClosedInputStream extends InputStream {
private static final int JPEG_EOI_1 = 0xFF;
private static final int JPEG_EOI_2 = 0xD9;
private final InputStream inputStream;
private int bytesPastEnd;
private JpegClosedInputStream(final InputStream iInputStream) {
inputStream = iInputStream;
bytesPastEnd = 0;
}
#Override
public int read() throws IOException {
int buffer = inputStream.read();
if (buffer == -1) {
if (bytesPastEnd > 0) {
buffer = JPEG_EOI_2;
} else {
++bytesPastEnd;
buffer = JPEG_EOI_1;
}
}
return buffer;
}
}
}
By the way, using the above custom class, I am trying to load my byte array like this:
byte[] bytes = Base64.decode(formattedB64String, Base64.NO_WRAP);
ByteArrayInputStream is = new ByteArrayInputStream(bytes);
String imageId = "stream://" + is.hashCode();
...
ImageLoader imageLoader = ImageLoader.getInstance();
imageLoader.displayImage(imageId, userImage, options);
and I get this error:
ImageLoader: Image can't be decoded [stream://1097215584_656x383]
Universal Image loader does not allow the stream:// schema, so I created a custom BaseImageDownloader class that allows it:
public class StreamImageDownloader extends BaseImageDownloader {
private static final String SCHEME_STREAM = "stream";
private static final String STREAM_URI_PREFIX = SCHEME_STREAM + "://";
public StreamImageDownloader(Context context) {
super(context);
}
#Override
protected InputStream getStreamFromOtherSource(String imageUri, Object extra) throws IOException {
if (imageUri.startsWith(STREAM_URI_PREFIX)) {
return (InputStream) extra;
} else {
return super.getStreamFromOtherSource(imageUri, extra);
}
}
}
So if anyone can help me create a better CustomImageDecoder that handles a BASE64 encoded string, or a byte[] containing an image so I can use decodeByteArray, I would be grateful!
Thank you.
UnversalImageLoader uses the following schemes to decode the files
"h t t p ://site.com/image.png" // from Web
"file:///mnt/sdcard/image.png" // from SD card
"file:///mnt/sdcard/video.mp4" // from SD card (video thumbnail)
"content://media/external/images/media/13" // from content provider
"content://media/external/video/media/13" // from content provider (video thumbnail)
"assets://image.png" // from assets
"drawable://" + R.drawable.img // from drawables (non-9patch images)
your scheme is stream://
Hope that helps.
Just to close this off:
The issue here is actually a bug in Android <4.3 where Android can't display images that either aren't closed properly (missing end bytes) or contain certain metadata that, for some reason, it doesn't like. I'm not sure what metadata this is, however. My issue was with JPEGs not being terminated properly.
The bug is fixed in Android 4.3 anyway.
I have the following requirements for image download:
ignoring SSL errors (yes I am aware of the risks)
using a session cookie
I tried to adapt Picasso 2.4.0 to do that, below is my approach:
public static Picasso getPicasso(Context context) {
/* an OkHttpClient that ignores SSL errors */
final OkHttpClient client = getUnsafeOkHttpClient();
return new Picasso.Builder(context)
.downloader(new OkHttpDownloader(client) {
#Override
public Response load(Uri uri, boolean localCacheOnly) throws IOException {
final String RESPONSE_SOURCE_ANDROID = "X-Android-Response-Source";
final String RESPONSE_SOURCE_OKHTTP = "OkHttp-Response-Source";
HttpURLConnection connection = openConnection(uri);
connection.setRequestProperty("Cookie", getCookieHandler().
getCookieStore().getCookies().get(0).toString());
connection.setUseCaches(true);
if (localCacheOnly)
connection.setRequestProperty("Cache-Control", "only-if-cached,max-age=" + Integer.MAX_VALUE);
int responseCode = connection.getResponseCode();
if (responseCode == 401)
relogin();
else if (responseCode >= 300) {
connection.disconnect();
throw new ResponseException(responseCode + " " + connection.getResponseMessage());
}
String responseSource = connection.getHeaderField(RESPONSE_SOURCE_OKHTTP);
if (responseSource == null)
responseSource = connection.getHeaderField(RESPONSE_SOURCE_ANDROID);
long contentLength = connection.getHeaderFieldInt("Content-Length", -1);
boolean fromCache = parseResponseSourceHeader(responseSource);
return new Response(connection.getInputStream(), fromCache, contentLength);
}
}).build();
}
The only thing that I changed from the original source is adding a Cookie for the HttpURLConnection. I also copied (unchanged) the parseResponseSourceHeader() method since it has private access.
Note that the approach given here does NOT work (response code 401).
The image loading basically works, but there are major issues:
caching doesn't work (fromCache is always false and Picasso always reloads an image which has already been downloaded)
there's no "Content-Length" header, so contentLength is always -1
though the cache doesn't work, the RAM usage increases when loading next image (into exactly the same or any other ImageView), it seems the Bitmap object stays somewhere in the memory
when used inside the BaseAdapter of a GridView, it seems that Picasso tries to load all (or at least as many as the number of times getView() was called) images at the same time. Those images appear, then the app freezes and closes with the following (OOM?) log:
A/Looper﹕ Could not create wake pipe. errno=24
or
A/Looper﹕ Could not create epoll instance. errno=24
The described issues occur no matter if I use a custom Target of just an ImageView.
It seems I have broken some of Picasso mechanisms by overriding the load() method of the OkHttpDownloader, but I'm not getting what's wrong since I did minimal changes. Any suggestions are appreciated.
In case someone has a similar problem: it was a really lame mistake of mine. I was creating multiple Picasso instances which is complete nonsense. After ensuring the singleton pattern with a helper class that returns a single Picasso instance everything works as intended.
Sometimes randomly Volley crashes my app upon startup, it crashes in the application class and a user would not be able to open the app again until they go into settings and clear app data
java.lang.OutOfMemoryError
at com.android.volley.toolbox.DiskBasedCache.streamToBytes(DiskBasedCache.java:316)
at com.android.volley.toolbox.DiskBasedCache.readString(DiskBasedCache.java:526)
at com.android.volley.toolbox.DiskBasedCache.readStringStringMap(DiskBasedCache.java:549)
at com.android.volley.toolbox.DiskBasedCache$CacheHeader.readHeader(DiskBasedCache.java:392)
at com.android.volley.toolbox.DiskBasedCache.initialize(DiskBasedCache.java:155)
at com.android.volley.CacheDispatcher.run(CacheDispatcher.java:84)
The "diskbasedbache" tries to allocate over 1 gigabyte of memory, for no obvious reason
how would I make this not happen? It seems to be an issue with Volley, or maybe an issue with a custom disk based cache but I don't immediately see (from the stack trace) how to 'clear' this cache or do a conditional check or handle this exception
Insight appreciated
In the streamToBytes(), first it will new bytes by the cache file length, does your cache file was too large than application maximum heap size ?
private static byte[] streamToBytes(InputStream in, int length) throws IOException {
byte[] bytes = new byte[length];
...
}
public synchronized Entry get(String key) {
CacheHeader entry = mEntries.get(key);
File file = getFileForKey(key);
byte[] data = streamToBytes(..., file.length());
}
If you want to clear the cache, you could keep the DiskBasedCache reference, after clear time's came, use ClearCacheRequest and pass that cache instance in :
File cacheDir = new File(context.getCacheDir(), DEFAULT_CACHE_DIR);
DiskBasedCache cache = new DiskBasedCache(cacheDir);
RequestQueue queue = new RequestQueue(cache, network);
queue.start();
// clear all volley caches.
queue.add(new ClearCacheRequest(cache, null));
this way will clear all caches, so I suggest you use it carefully. of course, you can doing conditional check, just iterating the cacheDir files, estimate which was too large then remove it.
for (File cacheFile : cacheDir.listFiles()) {
if (cacheFile.isFile() && cacheFile.length() > 10000000) cacheFile.delete();
}
Volley wasn't design as a big data cache solution, it's common request cache, don't storing large data anytime.
------------- Update at 2014-07-17 -------------
In fact, clear all caches is final way, also isn't wise way, we should suppressing these large request use cache when we sure it would be, and if not sure? we still can determine the response data size whether large or not, then call setShouldCache(false) to disable it.
public class TheRequest extends Request {
#Override
protected Response<String> parseNetworkResponse(NetworkResponse response) {
// if response data was too large, disable caching is still time.
if (response.data.length > 10000) setShouldCache(false);
...
}
}
I experienced the same issue.
We knew we didn't have files that were GBs in size on initialization of the cache. It also occurred when reading header strings, which should never be GBs in length.
So it looked like the length was being read incorrectly by readLong.
We had two apps with roughly identical setups, except that one app had two independent processes created on start up. The main application process and a 'SyncAdapter' process following the sync adapter pattern. Only the app with two processes would crash.
These two processes would independently initialize the cache.
However, the DiskBasedCache uses the same physical location for both processes. We eventually concluded that concurrent initializations were resulting in concurrent reads and writes of the same files, leading to bad reads of the size parameter.
I don't have a full proof that this is the issue, but I'm planning to work on a test app to verify.
In the short term, we've just caught the overly large byte allocation in streamToBytes, and throw an IOException so that Volley catches the exception and just deletes the file.
However, it would probably be better to use a separate disk cache for each process.
private static byte[] streamToBytes(InputStream in, int length) throws IOException {
byte[] bytes;
// this try-catch is a change added by us to handle a possible multi-process issue when reading cache files
try {
bytes = new byte[length];
} catch (OutOfMemoryError e) {
throw new IOException("Couldn't allocate " + length + " bytes to stream. May have parsed the stream length incorrectly");
}
int count;
int pos = 0;
while (pos < length && ((count = in.read(bytes, pos, length - pos)) != -1)) {
pos += count;
}
if (pos != length) {
throw new IOException("Expected " + length + " bytes, read " + pos + " bytes");
}
return bytes;
}
Once the problem occurs, it seems to recur on every subsequent initialization, pointing to an invalid cached header.
Fortunately, this issue has been fixed in the official Volley repo:
https://github.com/google/volley/issues/12
See related issues in the android-volley mirror:
https://github.com/mcxiaoke/android-volley/issues/141
https://github.com/mcxiaoke/android-volley/issues/61
https://github.com/mcxiaoke/android-volley/issues/37
I´ve been reading the questions that are in StackOverFlow about Volley and I´m still unsure if the approach that I have in mind of how to use it, is correct or not.
My usecase is the following: I first download an index (text file) that contains the set of URLs from the files that I have to download, then I read it and put its content in a list (in the following code; "urls"). Lastly, my idea is to do something like this:
RequestQueue queue = Volley.newRequestQueue(this);
for (int i = 0; i < urls.length; i++) {
String url = urls[i];
final int p = i;
if (urls[i].contains("json")) {
JsonObjectRequest jsObjRequest = new JsonObjectRequest(
Request.Method.GET, url, null,
new Response.Listener<JSONObject>() {
#Override
public void onResponse(JSONObject response) {
// Save the data or whatever
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
// Handle the error
}
}
);
queue.add(jsObjRequest);
} else if (urls[i].contains("png") || urls[i].contains("jpeg")) {
int memClass = ((ActivityManager) this.getSystemService(Context.ACTIVITY_SERVICE))
.getMemoryClass();
// Use 1/8th of the available memory for this memory cache.
int cacheSize = 1024 * 1024 * memClass / 8;
ImageLoader imageLoader = new ImageLoader(queue, new BitmapLruCache(cacheSize));
imageLoader.get(urls[p],
ImageLoader.getImageListener(imageDisplay,
R.drawable.ic_launcher,
R.drawable.ic_launcher)
);
} else {
// The file is not a json or an image
// What should I do?
}
}
So basically my questions are:
1.- Is it a good idea to have this kind of separation depending on the kind of request that you are going to make? Is there a better/cleaner way? In the examples that I´ve reading, they always cover just one scenario (either json, string or image download) and I´m unsure how to mix them.
2.- For the audio and video files that I would need to download from my index, considering that Volley is not meant for large downloads, should I just use an AsyncTask and OKHttp and execute it within that last "else"?
Any input about Volley will be appreciated.
We usually get data from server response in android development.
/*
* get server response inputStream
*/
InputStream responseInputStream;
Solution1: get response string by multiple read.
/*
* get server response string
*/
StringBuffer responseString = new StringBuffer();
responseInputStream = new InputStreamReader(conn.getInputStream(),"UTF-8");
char[] charBuffer = new char[bufferSize];
int _postion = 0;
while ((_postion=responseInputStream.read(charBuffer)) > -1) {
responseString.append(charBuffer,0,_postion);
}
responseInputStream.close();
Solution2: get response only one read.
String responseString = null;
int content_length=1024;
// we can get content length from response header, here assign 1024 for simple.
responseInputStream = new InputStreamReader(conn.getInputStream(),"UTF-8");
char[] charBuffer = new char[content_length];
int _postion = 0;
int position = responseInputStream.read(charBuffer)
if(position>-1){
responseString = new String(charBuffer,0,position );
}
responseInputStream.close();
Which solution has better performance? why?
Notes: server response json format data that less than 1M bytes.
Why you're reinventing a wheel? ;)
If you're using HttpClient then just use EntityUtils.toString(...).
I guess you're using HttpURLConnection. Then look at EntityUtils.toString(...) from Apache HttpClient - source code. Your first approach is similar to it.
BTW, the second code is worse because:
new String(charBuffer,0,position ) runs garbage collector
In both and even in EntityUtils:
int content_length = 1024; in most cases 8192 is default for socket buffer, so your code might run while loop 8 times more often than it could.
I would recommend the second method IF you do not want to display the amount of data downloaded/transferred . As the object is read as a whole and since the size of your JSON string is comparable to 1M, it will take some time to download. At that time you can, atmost, put up a text for the user saying downloading... You cannot notify the user the amount downloaded.
But if you want to display the amount of data downloaded, use the first method that you gave. Where the you read the data from the server in parts. You can update the UI, with the amount downloaded. For eg 25 % downloaded...
char[] charBuffer = new char[bufferSize];
int _postion = 0;
int i=0;
while ((_postion=responseInputStream.read(charBuffer)) > -1) {
//((i*buffer_size)/content_length) * 100 % completed..
i++;
}
So, I would say the seconds method is better.
BTW Did you consider this?
ObjectInputStream in = new InputStreamReader(conn.getInputStream(),"UTF-8");
if(resposeCode==200)
{
String from_server=(String) in.readObject();
}
Reading the input String as an object. Any object whoe class implements serializable can be passed using ObjectOutputStream and received using ObjectInputStream()
I Think First one is good
Because, in First that will reading your response in char to char method .
Where , Second that will try to read whole response object or as key Filed of Object.
So ,As i think and as per my knowledge First is Better to camper with second.If anyone want to edit then it will truly appreciated.