I have the following requirements for image download:
ignoring SSL errors (yes I am aware of the risks)
using a session cookie
I tried to adapt Picasso 2.4.0 to do that, below is my approach:
public static Picasso getPicasso(Context context) {
/* an OkHttpClient that ignores SSL errors */
final OkHttpClient client = getUnsafeOkHttpClient();
return new Picasso.Builder(context)
.downloader(new OkHttpDownloader(client) {
#Override
public Response load(Uri uri, boolean localCacheOnly) throws IOException {
final String RESPONSE_SOURCE_ANDROID = "X-Android-Response-Source";
final String RESPONSE_SOURCE_OKHTTP = "OkHttp-Response-Source";
HttpURLConnection connection = openConnection(uri);
connection.setRequestProperty("Cookie", getCookieHandler().
getCookieStore().getCookies().get(0).toString());
connection.setUseCaches(true);
if (localCacheOnly)
connection.setRequestProperty("Cache-Control", "only-if-cached,max-age=" + Integer.MAX_VALUE);
int responseCode = connection.getResponseCode();
if (responseCode == 401)
relogin();
else if (responseCode >= 300) {
connection.disconnect();
throw new ResponseException(responseCode + " " + connection.getResponseMessage());
}
String responseSource = connection.getHeaderField(RESPONSE_SOURCE_OKHTTP);
if (responseSource == null)
responseSource = connection.getHeaderField(RESPONSE_SOURCE_ANDROID);
long contentLength = connection.getHeaderFieldInt("Content-Length", -1);
boolean fromCache = parseResponseSourceHeader(responseSource);
return new Response(connection.getInputStream(), fromCache, contentLength);
}
}).build();
}
The only thing that I changed from the original source is adding a Cookie for the HttpURLConnection. I also copied (unchanged) the parseResponseSourceHeader() method since it has private access.
Note that the approach given here does NOT work (response code 401).
The image loading basically works, but there are major issues:
caching doesn't work (fromCache is always false and Picasso always reloads an image which has already been downloaded)
there's no "Content-Length" header, so contentLength is always -1
though the cache doesn't work, the RAM usage increases when loading next image (into exactly the same or any other ImageView), it seems the Bitmap object stays somewhere in the memory
when used inside the BaseAdapter of a GridView, it seems that Picasso tries to load all (or at least as many as the number of times getView() was called) images at the same time. Those images appear, then the app freezes and closes with the following (OOM?) log:
A/Looper﹕ Could not create wake pipe. errno=24
or
A/Looper﹕ Could not create epoll instance. errno=24
The described issues occur no matter if I use a custom Target of just an ImageView.
It seems I have broken some of Picasso mechanisms by overriding the load() method of the OkHttpDownloader, but I'm not getting what's wrong since I did minimal changes. Any suggestions are appreciated.
In case someone has a similar problem: it was a really lame mistake of mine. I was creating multiple Picasso instances which is complete nonsense. After ensuring the singleton pattern with a helper class that returns a single Picasso instance everything works as intended.
Related
I am working on an app that uses a Recyclerview to display mp3 files, providing its cover art image along with other info. It works but is slow once it starts dealing with a dozen or more cover arts to retrieve, as I am currently doing this from the id3 on the main thread, which I know is not a good idea.
Ideally, I would work with placeholders so that the images can be added as they become available. I've been looking into moving the retrieval to a background thread and have looked at different options: AsyncTask, Service, WorkManager. AsyncTask seems not to be the way to go as I face memory leaks (I need context to retrieve the cover art through MetadataRetriever). So I am leaning away from that. Yet I am struggling to figure out which approach is best in my case.
From what I understand I need to find an approach that allows multithreading and also a means to cancel the retrieval in case the user has already moved on (scrolling or navigating away). I am already using Glide, which I understand should help with the caching.
I know I could rework the whole approach and provide the cover art as images separately, but that seems a last resort to me, as I would rather not weigh down the app with even more data.
The current version of the app is here (please note it will not run as I cannot openly divulge certain aspects). I am retrieving the cover art as follows (on the main thread):
static public Bitmap getCoverArt(Uri medUri, Context ctxt) {
MediaMetadataRetriever mmr = new MediaMetadataRetriever();
mmr.setDataSource(ctxt, medUri);
byte[] data = mmr.getEmbeddedPicture();
if (data != null) {
return BitmapFactory.decodeByteArray(data, 0, data.length);
} else {
return null;
}
}
I've found many examples with AsyncTask or just keeping the MetaDataRetriever on the main thread, but have yet to find an example that enables a dozen or more cover arts to be retrieved without slowing down the main thread. I would appreciate any help and pointers.
It turns out it does work with AsyncTask, as long as it is not a class onto itself but setup and called from a class with context. Here is a whittled down version of my approach (I am calling this from within my Adapter.):
//set up titles and placeholder image so we needn't wait on the image to load
titleTv.setText(selectedMed.getTitle());
subtitleTv.setText(selectedMed.getSubtitle());
imageIv.setImageResource(R.drawable.ic_launcher_foreground);
imageIv.setAlpha((float) 0.2);
final long[] duration = new long[1];
//a Caching system that helps reduce the amount of loading needed. See: https://github.com/cbonan/BitmapFun?files=1
if (lruCacheManager.getBitmapFromMemCache(selectedMed.getId() + position) != null) {
//is there an earlier cached image to reuse? imageIv.setImageBitmap(lruCacheManager.getBitmapFromMemCache(selectedMed.getId() + position));
imageIv.setAlpha((float) 1.0);
titleTv.setVisibility(View.GONE);
subtitleTv.setVisibility(View.GONE);
} else {
//time to load and show the image. For good measure, the duration is also queried, as this also needs the setDataSource which causes slow down
new AsyncTask<Uri, Void, Bitmap>() {
#Override
protected Bitmap doInBackground(Uri... uris) {
MediaMetadataRetriever mmr = new MediaMetadataRetriever();
mmr.setDataSource(ctxt, medUri);
byte[] data = mmr.getEmbeddedPicture();
Log.v(TAG, "async data: " + Arrays.toString(data));
String durationStr = mmr.extractMetadata(MediaMetadataRetriever.METADATA_KEY_DURATION);
duration[0] = Long.parseLong(durationStr);
if (data != null) {
InputStream is = new ByteArrayInputStream(mmr.getEmbeddedPicture());
return BitmapFactory.decodeStream(is);
} else {
return null;
}
}
#Override
protected void onPostExecute(Bitmap bitmap) {
super.onPostExecute(bitmap);
durationTv.setVisibility(View.VISIBLE);
durationTv.setText(getDisplayTime(duration[0], false));
if (bitmap != null) {
imageIv.setImageBitmap(bitmap);
imageIv.setAlpha((float) 1.0);
titleTv.setVisibility(View.GONE);
subtitleTv.setVisibility(View.GONE);
} else {
titleTv.setVisibility(View.VISIBLE);
subtitleTv.setVisibility(View.VISIBLE);
}
lruCacheManager.addBitmapToMemCache(bitmap, selectedMed.getId() + position);
}
}.execute(medUri);
}
I have tried working with Glide for the caching, but I haven't been able to link the showing/hiding of the TextViews to whether there is a bitmap. In a way though, this is sleeker as I don't need to load the bulk of the Glide-library. So I am happy with this for now.
I have an Android app which is performing HTTP get requests in the background. I'll give some details of how this is performed shortly. I'll focus on describing the issue first.
I've observed in the debugger that OkHttp ConnectionPools are stacking up in the Threads window in Android Studio. This is concerning me as it seems to indicate a resource leak and over time these will increase indefinitely (it seems).
My main problem is that I have no idea why this would be happening. I'm almost certain that it's my fault, but I don't have a lead on what to look for. So my general question is:
"What might cause a new pool to be created each time a HTTP request is made?"
Firstly, here's the code I use to do the get request. Note that it's simplified, but I've run with this exact code and see the problem.
private TestResult test() {
final long startTime = SystemClock.elapsedRealtime();
final String server = "connectivitycheck.gstatic.com";
URL url;
try {
url = new URL("http", SOME_PATH);
} catch (MalformedURLException e) {
// We shouldn't get here since the url is well known
return new TestResult(
"failed to generate url " + e,
SystemClock.elapsedRealtime() - startTime
);
}
HttpURLConnection urlConnection = null;
int httpResponseCode;
try {
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setInstanceFollowRedirects(false);
urlConnection.setConnectTimeout(SOCKET_TIMEOUT_MS);
urlConnection.setReadTimeout(SOCKET_TIMEOUT_MS);
urlConnection.setUseCaches(false);
InputStream is = urlConnection.getInputStream();
httpResponseCode = urlConnection.getResponseCode();
is.close();
} catch (IOException e) {
return new TestResult(
e,
SystemClock.elapsedRealtime() - startTime
);
} finally {
if (urlConnection != null)
urlConnection.disconnect();
}
if (httpResponseCode != HttpURLConnection.HTTP_NO_CONTENT) {
return new TestResult(
"invalid error code, expected 204 but got " + httpResponseCode,
SystemClock.elapsedRealtime() - startTime
);
}
return new TestResult(
SystemClock.elapsedRealtime() - startTime
);
}
So, when is this code run...?
Firstly let me say that I've tried running this code from my home activity on my app (in a separate thread - as HTTP isn't allowed on the main android thread). When I do that, everything is fine. I can make multiple requests and never see more than one pool.
The situation in which I run the code and see the issues is as follows:
I have a background Service
In that service I start a new Thread
In that thread I create an ExecutorService:
ExecutorService executorService = Executors.newCachedThreadPool();
CompletionService<FutureResult> completionService = new ExecutorCompletionService<>(executorService);
Using that ExecutorService I submit a callable task which runs my test method (above).
Note that this may all sound convoluted but there's purpose to all of this. For one, the ExecutorService will be performing many other tasks other than just running the test method.
I've had a good look for hanging references that somehow might be keeping the underlying HTTP resources alive but I can't see any.
Does anyone with good knowledge of OkHttp know why this might happen? I'm hoping for some insight to help me uncover the problem.
The easiest way to find out is to set a breakpoint in the ConnectionPool constructor and run your app. That'll show who is creating it. My guess is it's an unrelated class like a crash reporter or analytics library.
I am using a binding adapter to load images in a recycler view. Images appear fine. While fast scrolling I noticed sometimes I was getting a 'connection leaked' message from Picasso.
The problem comes from dead image links, hardcoding all of my image urls to point nowhere produces the error for every image after scrolling the first couple off the screen.
W/OkHttpClient: A connection to https://s3-eu-west-1.amazonaws.com/ was leaked. Did you forget to close a response body?
The code is basically identical to this sample.
BindingUtils.kt
object BindingUtils {
#BindingAdapter("imageUrl")
#JvmStatic
fun setImageUrl(imageView: ImageView, url: String) {
Picasso.with(imageView.context).load(url).into(imageView)
}
xml
<ImageView
android:id="#+id/imageview_merchant_background"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:background="#color/primary"
android:scaleType="centerCrop"
app:imageUrl="#{viewModel.background}"/>
gradle
implementation "com.squareup.retrofit2:retrofit:$rootProject.retrofitVersion"
implementation "com.squareup.retrofit2:adapter-rxjava2:$rootProject.retrofitVersion"
implementation "com.squareup.retrofit2:converter-gson:$rootProject.retrofitVersion"
implementation "com.squareup.okhttp3:logging-interceptor:$rootProject.okhttpLoggingVersion"
implementation "com.squareup.picasso:picasso:$rootProject.picassoVersion"
retrofitVersion = '2.3.0'
okhttpLoggingVersion = '3.6.0'
picassoVersion = '2.5.2'
I can see several references to people needing to closing connections for standard Okhttp requests but seeing as that Picasso load call is a one-liner how can this be leaking?
Under the hood Picasso is using okhttp3 for handling its network requests. See here the code for Picasso's NetworkRequestHandler class: https://github.com/square/picasso/blob/0728bb1c619746001c60296d975fbc6bd92a05d2/picasso/src/main/java/com/squareup/picasso/NetworkRequestHandler.java
There is a load function that handles an okhttp Request:
#Override public Result load(Request request, int networkPolicy) throws IOException {
okhttp3.Request downloaderRequest = createRequest(request, networkPolicy);
Response response = downloader.load(downloaderRequest);
ResponseBody body = response.body();
if (!response.isSuccessful()) {
body.close();
throw new ResponseException(response.code(), request.networkPolicy);
}
// Cache response is only null when the response comes fully from the network. Both completely
// cached and conditionally cached responses will have a non-null cache response.
Picasso.LoadedFrom loadedFrom = response.cacheResponse() == null ? NETWORK : DISK;
// Sometimes response content length is zero when requests are being replayed. Haven't found
// root cause to this but retrying the request seems safe to do so.
if (loadedFrom == DISK && body.contentLength() == 0) {
body.close();
throw new ContentLengthException("Received response with 0 content-length header.");
}
if (loadedFrom == NETWORK && body.contentLength() > 0) {
stats.dispatchDownloadFinished(body.contentLength());
}
InputStream is = body.byteStream();
return new Result(is, loadedFrom);
}
I am not too familiar with the Picasso project, but it seems like the response body object is not closed in all cases. You may have spotted a bug in Picasso and may want to file an issue at picasso's github
Wild guess, if it has to do anything with the leaking of the context of your activity. Try with applicationContext
Picasso.with(imageView.context.applicationContext).load(url).into(imageView)
I'm using Volley as my network stack in a project I'm working on in Android. Part of my requirements is to download potentially very large files and save them on the file system.
Ive been looking at the implementation of volley, and it seems that the only way volley works is it downloads an entire file into a potentially massive byte array and then defers handling of this byte array to some callback handler.
Since these files can be very large, I'm worried about an out of memory error during the download process.
Is there a way to tell volley to process all bytes from an http input stream directly into a file output stream? Or would this require me to implement my own network object?
I couldn't find any material about this online, so any suggestions would be appreciated.
Okay, so I've come up with a solution which involves editing Volley itself. Here's a walk through:
Network response can't hold a byte array anymore. It needs to hold an input stream. Doing this immediately breaks all request implementations, since they rely on NetworkResponse holding a public byte array member. The least invasive way I found to deal with this is to add a "toByteArray" method inside NetworkResponse, and then do a little refactoring, making any reference to a byte array use this method, rather than the removed byte array member. This means that the transition of the input stream to a byte array happens during the response parsing. I'm not entirely sure what the long term effects of this are, and so some unit testing / community input would be a huge help here. Here's the code:
public class NetworkResponse {
/**
* Creates a new network response.
* #param statusCode the HTTP status code
* #param data Response body
* #param headers Headers returned with this response, or null for none
* #param notModified True if the server returned a 304 and the data was already in cache
*/
public NetworkResponse(int statusCode, inputStream data, Map<String, String> headers,
boolean notModified, ByteArrayPool byteArrayPool, int contentLength) {
this.statusCode = statusCode;
this.data = data;
this.headers = headers;
this.notModified = notModified;
this.byteArrayPool = byteArrayPool;
this.contentLength = contentLength;
}
public NetworkResponse(byte[] data) {
this(HttpStatus.SC_OK, data, Collections.<String, String>emptyMap(), false);
}
public NetworkResponse(byte[] data, Map<String, String> headers) {
this(HttpStatus.SC_OK, data, headers, false);
}
/** The HTTP status code. */
public final int statusCode;
/** Raw data from this response. */
public final InputStream inputStream;
/** Response headers. */
public final Map<String, String> headers;
/** True if the server returned a 304 (Not Modified). */
public final boolean notModified;
public final ByteArrayPool byteArrayPool;
public final int contentLength;
// method taken from BasicNetwork with a few small alterations.
public byte[] toByteArray() throws IOException, ServerError {
PoolingByteArrayOutputStream bytes =
new PoolingByteArrayOutputStream(byteArrayPool, contentLength);
byte[] buffer = null;
try {
if (inputStream == null) {
throw new ServerError();
}
buffer = byteArrayPool.getBuf(1024);
int count;
while ((count = inputStream.read(buffer)) != -1) {
bytes.write(buffer, 0, count);
}
return bytes.toByteArray();
} finally {
try {
// Close the InputStream and release the resources by "consuming the content".
// Not sure what to do about the entity "consumeContent()"... ideas?
inputStream.close();
} catch (IOException e) {
// This can happen if there was an exception above that left the entity in
// an invalid state.
VolleyLog.v("Error occured when calling consumingContent");
}
byteArrayPool.returnBuf(buffer);
bytes.close();
}
}
}
Then to prepare the NetworkResponse, we need to edit the BasicNetwork to create the NetworkResponse correctly (inside BasicNetwork.performRequest):
int contentLength = 0;
if (httpResponse.getEntity() != null)
{
responseContents = httpResponse.getEntity().getContent(); // responseContents is now an InputStream
contentLength = httpResponse.getEntity().getContentLength();
}
...
return new NetworkResponse(statusCode, responseContents, responseHeaders, false, mPool, contentLength);
That's it. Once the data inside network response is an input stream, I can build my own requests which can parse it directly into a file output stream which only hold a small in-memory buffer.
From a few initial tests, this seems to be working alright without harming other components, however a change like this probably requires some more intensive testing & peer reviewing, so I'm going to leave this answer not marked as correct until more people weigh in, or I see it's robust enough to rely on.
Please feel free to comment on this answer and/or post answers yourselves. This feels like a serious flaw in Volley's design, and if you see flaws with this design, or can think of better designs yourselves, I think it would benefit everyone.
Sometimes randomly Volley crashes my app upon startup, it crashes in the application class and a user would not be able to open the app again until they go into settings and clear app data
java.lang.OutOfMemoryError
at com.android.volley.toolbox.DiskBasedCache.streamToBytes(DiskBasedCache.java:316)
at com.android.volley.toolbox.DiskBasedCache.readString(DiskBasedCache.java:526)
at com.android.volley.toolbox.DiskBasedCache.readStringStringMap(DiskBasedCache.java:549)
at com.android.volley.toolbox.DiskBasedCache$CacheHeader.readHeader(DiskBasedCache.java:392)
at com.android.volley.toolbox.DiskBasedCache.initialize(DiskBasedCache.java:155)
at com.android.volley.CacheDispatcher.run(CacheDispatcher.java:84)
The "diskbasedbache" tries to allocate over 1 gigabyte of memory, for no obvious reason
how would I make this not happen? It seems to be an issue with Volley, or maybe an issue with a custom disk based cache but I don't immediately see (from the stack trace) how to 'clear' this cache or do a conditional check or handle this exception
Insight appreciated
In the streamToBytes(), first it will new bytes by the cache file length, does your cache file was too large than application maximum heap size ?
private static byte[] streamToBytes(InputStream in, int length) throws IOException {
byte[] bytes = new byte[length];
...
}
public synchronized Entry get(String key) {
CacheHeader entry = mEntries.get(key);
File file = getFileForKey(key);
byte[] data = streamToBytes(..., file.length());
}
If you want to clear the cache, you could keep the DiskBasedCache reference, after clear time's came, use ClearCacheRequest and pass that cache instance in :
File cacheDir = new File(context.getCacheDir(), DEFAULT_CACHE_DIR);
DiskBasedCache cache = new DiskBasedCache(cacheDir);
RequestQueue queue = new RequestQueue(cache, network);
queue.start();
// clear all volley caches.
queue.add(new ClearCacheRequest(cache, null));
this way will clear all caches, so I suggest you use it carefully. of course, you can doing conditional check, just iterating the cacheDir files, estimate which was too large then remove it.
for (File cacheFile : cacheDir.listFiles()) {
if (cacheFile.isFile() && cacheFile.length() > 10000000) cacheFile.delete();
}
Volley wasn't design as a big data cache solution, it's common request cache, don't storing large data anytime.
------------- Update at 2014-07-17 -------------
In fact, clear all caches is final way, also isn't wise way, we should suppressing these large request use cache when we sure it would be, and if not sure? we still can determine the response data size whether large or not, then call setShouldCache(false) to disable it.
public class TheRequest extends Request {
#Override
protected Response<String> parseNetworkResponse(NetworkResponse response) {
// if response data was too large, disable caching is still time.
if (response.data.length > 10000) setShouldCache(false);
...
}
}
I experienced the same issue.
We knew we didn't have files that were GBs in size on initialization of the cache. It also occurred when reading header strings, which should never be GBs in length.
So it looked like the length was being read incorrectly by readLong.
We had two apps with roughly identical setups, except that one app had two independent processes created on start up. The main application process and a 'SyncAdapter' process following the sync adapter pattern. Only the app with two processes would crash.
These two processes would independently initialize the cache.
However, the DiskBasedCache uses the same physical location for both processes. We eventually concluded that concurrent initializations were resulting in concurrent reads and writes of the same files, leading to bad reads of the size parameter.
I don't have a full proof that this is the issue, but I'm planning to work on a test app to verify.
In the short term, we've just caught the overly large byte allocation in streamToBytes, and throw an IOException so that Volley catches the exception and just deletes the file.
However, it would probably be better to use a separate disk cache for each process.
private static byte[] streamToBytes(InputStream in, int length) throws IOException {
byte[] bytes;
// this try-catch is a change added by us to handle a possible multi-process issue when reading cache files
try {
bytes = new byte[length];
} catch (OutOfMemoryError e) {
throw new IOException("Couldn't allocate " + length + " bytes to stream. May have parsed the stream length incorrectly");
}
int count;
int pos = 0;
while (pos < length && ((count = in.read(bytes, pos, length - pos)) != -1)) {
pos += count;
}
if (pos != length) {
throw new IOException("Expected " + length + " bytes, read " + pos + " bytes");
}
return bytes;
}
Once the problem occurs, it seems to recur on every subsequent initialization, pointing to an invalid cached header.
Fortunately, this issue has been fixed in the official Volley repo:
https://github.com/google/volley/issues/12
See related issues in the android-volley mirror:
https://github.com/mcxiaoke/android-volley/issues/141
https://github.com/mcxiaoke/android-volley/issues/61
https://github.com/mcxiaoke/android-volley/issues/37