I will soon work on a project, which uses a lot of HTTPRequests for mainly JSONs and Images, so I thought it is a good idea to think about caching. Basically I'm looking for a solution for
Start a HTTPRequest with a given lifetime (f.e. 3,6,12 hours)
Check, if that Request is available in the Cache and still valid (lifetime)
If Request is still valid, take it from Cache, otherwise make the Request and save its Response
I found HttpResponseCache class in Android. It is working, however it is not working like I'm expecting.
My test case is an AsyncTask to cache several Images. Code looks like the following:
URL url = new URL(link);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
Bitmap myBitmap;
try {
connection.addRequestProperty("Cache-Control","only-if-cached");
//check if Request is in cache
InputStream cached = connection.getInputStream();
//set image if in cache
myBitmap = BitmapFactory.decodeStream(cached);
} catch (FileNotFoundException e) {
HttpURLConnection connection2 = (HttpURLConnection) url.openConnection();
connection2.setDoInput(true);
connection2.addRequestProperty("Cache-Control", "max-stale=" + 60);
connection2.connect();
InputStream input = connection2.getInputStream();
myBitmap = BitmapFactory.decodeStream(input);
}
return myBitmap;
} catch (IOException e) {
e.printStackTrace();
}
Two questions:
I set max-stale=60seconds for testing purposes. However, if I call the same URL 5minutes later, it tells me, it is loading the image from cache. I would assume, that the image is reloaded because the HTTPRequest in the cache is out of date? Or do I have to clean the cache myself?
In my catch block, I have to create a second HttpURLConnection, because I cannot add properties, after I opened an URLConnection (this happens in the connection.getInputStream()?!). Is this bad programming?
After all, I find that HttpResponseCache poorly documented. I came across Volley: Fast Networking, but this seems even less documented, even if it is offering exactly the things I need (Request queuing and prioritization...). What do you use for caching? Any links to libraries, tutorials, are welcome.
UPDATE
I'm not targeting Android versions lower than 4.0 (still maybe intresting for other users?)
Both HttpResponseCache and volley are poorly documented. However, I have found that you
can very easily extend and tweak volley. If you explore source code of volley, especially of: CacheEntry, CacheDispatcher and HttpHeaderParser, you can see how it is implemented.
A CacheEntry holds serverDate, etag, ttl and sofTtl which can represent cache state pretty well, also it has isExpired() and refreshNeeded() methods as convenience.
CacheDispatcher is implemented accurately as well:
// Attempt to retrieve this item from cache.
Cache.Entry entry = mCache.get(request.getCacheKey());
if (entry == null) {
request.addMarker("cache-miss");
// Cache miss; send off to the network dispatcher.
mNetworkQueue.put(request);
continue;
}
// If it is completely expired, just send it to the network.
if (entry.isExpired()) {
request.addMarker("cache-hit-expired");
request.setCacheEntry(entry);
mNetworkQueue.put(request);
continue;
}
// We have a cache hit; parse its data for delivery back to the request.
request.addMarker("cache-hit");
Response<?> response = request.parseNetworkResponse(
new NetworkResponse(entry.data, entry.responseHeaders));
request.addMarker("cache-hit-parsed");
if (!entry.refreshNeeded()) {
// Completely unexpired cache hit. Just deliver the response.
mDelivery.postResponse(request, response);
} else {
// Soft-expired cache hit. We can deliver the cached response,
// but we need to also send the request to the network for
// refreshing.
request.addMarker("cache-hit-refresh-needed");
request.setCacheEntry(entry);
// Mark the response as intermediate.
response.intermediate = true;
// Post the intermediate response back to the user and have
// the delivery then forward the request along to the network.
mDelivery.postResponse(request, response, new Runnable() {
#Override
public void run() {
try {
mNetworkQueue.put(request);
} catch (InterruptedException e) {
// Not much we can do about this.
}
}
});
}
One interesting tidbit: If cache is "soft expired", volley will deliver data from local cache immediately, and re-deliver it from server again after some time, for single request.
Finally, HttpHeaderParser does its best to cope to server headers:
headerValue = headers.get("Date");
if (headerValue != null) {
serverDate = parseDateAsEpoch(headerValue);
}
headerValue = headers.get("Cache-Control");
if (headerValue != null) {
hasCacheControl = true;
String[] tokens = headerValue.split(",");
for (int i = 0; i < tokens.length; i++) {
String token = tokens[i].trim();
if (token.equals("no-cache") || token.equals("no-store")) {
return null;
} else if (token.startsWith("max-age=")) {
try {
maxAge = Long.parseLong(token.substring(8));
} catch (Exception e) {
}
} else if (token.equals("must-revalidate") || token.equals("proxy-revalidate")) {
maxAge = 0;
}
}
}
headerValue = headers.get("Expires");
if (headerValue != null) {
serverExpires = parseDateAsEpoch(headerValue);
}
serverEtag = headers.get("ETag");
// Cache-Control takes precedence over an Expires header, even if both exist and Expires
// is more restrictive.
if (hasCacheControl) {
softExpire = now + maxAge * 1000;
} else if (serverDate > 0 && serverExpires >= serverDate) {
// Default semantic for Expire header in HTTP specification is softExpire.
softExpire = now + (serverExpires - serverDate);
}
Cache.Entry entry = new Cache.Entry();
entry.data = response.data;
entry.etag = serverEtag;
entry.softTtl = softExpire;
entry.ttl = entry.softTtl;
entry.serverDate = serverDate;
entry.responseHeaders = headers;
So, ensure the server sends proper headers as well as honors etag,time-stamp and cache control headers.
Finally, you can override getCacheEntry() of Request class to return custom CacheEntry make cache behave exactly according to your needs.
Soryy. But why don't you use third-party libs for this? Try to use Volley lib for this. It maintains a cache out of the box and it is async out of the box. It works really good. Tutorials for Volley: one (with caching demonstration), two.
And there is another good one async, caching lib for android with good documentation - Retrofit. Here is Retrofit Caching Example.
And here is their comparison.
To enable caching, all you need to do is just install HTTP response cache at application startup by using below code:
File httpCacheDir = new File(context.getCacheDir(), "http");
long httpCacheSize = 10 * 1024 * 1024; // 10 MiB
HttpResponseCache.install(httpCacheDir, httpCacheSize);
Whether the resource need to be fetched from network or cache, is taken care by HttpResponseCache. The age of cache is specified in the response headers of the resource request. For example this image, specifies cache age of 43200 seconds.
You can verify whether the resource is fetched from cache or network, by using following apis:
getHitCount : Number of Http requests which were served by cache.
getNetworkCount : Number of Http requests which were served by network.
Regarding max-stale, you have misunderstood it's purpose. It is used to permit stale cache responses. Here is it's definition from the rfc documentation :
Indicates that the client is willing to accept a response that has
exceeded its expiration time. If max-stale is assigned a value, then
the client is willing to accept a response that has exceeded its
expiration time by no more than the specified number of seconds. If no
value is assigned to max-stale, then the client is willing to accept a
stale response of any age.
Regarding cache control directive only-if-cached, only use it when you need to show something while your application is downloading latest content. So the question of handling new HttpUrlConnection in exception handler does not arise. From the docs:
Sometimes you'll want to show resources if they are available
immediately, but not otherwise. This can be used so your application
can show something while waiting for the latest data to be downloaded.
To restrict a request to locally-cached resources, add the
only-if-cached directive
One suggestion, add finally block, where you release the connection by calling disconnect.
Related
The code on the CacheDispatcher is as below. And focus on that release previous request object to avoid leaking request object when mQueue is drained.
I do not know why. Can anyone tell me the reason?
Thanks in advance.
java
Request<?> request;
while (true) {
// release previous request object to avoid leaking request object when mQueue is drained.
request = null;
try {
// Take a request from the queue.
request = mCacheQueue.take();
} catch (InterruptedException e) {
// We may have been interrupted because it was time to quit.
if (mQuit) {
return;
}
continue;
}
}
request = mCacheQueue.take(); is a blocking call so when the queue is empty(drained) we will still keep reference of the old request while waiting if we didn't do request = null;
this however is not the best way to achieve this behavior and it was changed in the new Volley code to :
while (true) {
try {
// Get a request from the cache triage queue, blocking until
// at least one is available.
final Request<?> request = mCacheQueue.take();
I have a recyclerview in wich each row contains an url, and the urls can expire in any moment.
I would like the user to not be forced to click on the url to check if it's not valid but I'd rather change the row's color in that case to notify the user.
This is the method I use to check if the url is still valid:
private boolean isUrlStillValid(String url) {
try {
HttpURLConnection.setFollowRedirects(false);
HttpURLConnection con = (HttpURLConnection) new URL(url).openConnection();
con.setRequestProperty("Accept-Encoding", "");
int responseCode = con.getResponseCode();
con.disconnect();
return responseCode == HttpURLConnection.HTTP_OK;
}catch(Exception e) {
return false;
}
}
I initially thought of launching an AsyncTask in the onBindViewHolder method of the recycler view where I check the url validity, but this will open a lot of connections simultaneously every time a row is shown and will cause very bad performance and memory issues I think.
Do you have any tip/suggestions on how to achieve this?
Do you have any tip/suggestions on how to achieve this?
for each url create a runnable and run all of them on fix thread pool with ExecutorService.
for example you can have 5 threads at a time with below code:
ExecutorService executor = Executors.newFixedThreadPool(5);
and also I recommend to save the result somewhere so when the user plays with scrollbar it dose not check the url again and again and again ...
by running netstat on the server:
UNTIL A FEW DAYS AGO: I could see the connection being ESTABLISHED only for about a second, and then it would disappear from the list
NOW: it stays as ESTABLISHED for about 10 seconds, then it goes into FIN_WAIT1 and FIN_WAIT2
the Android code is the same, the server is still the same
is it possible that some kind of Android update might have changed things?
I can't really explain it.
I report the code below. The urlConnection.disconnect() gets executed, but the connection remains established on the server.
HttpURLConnection urlConnection = null;
System.setProperty("http.keepAlive", "false");
try {
URL url = new URL(stringUrl);
urlConnection = (HttpURLConnection) url.openConnection();
InputStream instream = new BufferedInputStream(urlConnection.getInputStream());
...
instream.close();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (urlConnection!=null) {
urlConnection.disconnect();
}
}
The moment all data on input stream is consumed, the connection is released automatically and added to the connection pool. The underlying socket connection is not released, assuming the connection will be reused in near future. It is a good practice to call disconnect in finally block, as it takes care of connection release in case of exceptions.
Here is the implementation of read method of FixedLengthInputStream:
#Override public int read(byte[] buffer, int offset, int count) throws IOException {
Arrays.checkOffsetAndCount(buffer.length, offset, count);
checkNotClosed();
if (bytesRemaining == 0) {
return -1;
}
int read = in.read(buffer, offset, Math.min(count, bytesRemaining));
if (read == -1) {
unexpectedEndOfInput(); // the server didn't supply the promised content length
throw new IOException("unexpected end of stream");
}
bytesRemaining -= read;
cacheWrite(buffer, offset, read);
if (bytesRemaining == 0) {
endOfInput(true);
}
return read;
}
When bytesRemaining variable becomes 0, endOfInput is called which will futher call release method with true parameter, which will ensures the connection is pooled.
protected final void endOfInput(boolean reuseSocket) throws IOException {
if (cacheRequest != null) {
cacheBody.close();
}
httpEngine.release(reuseSocket);
}
Here is the release method implementation. The last if check ensures whether connection need to be closed down or added to the connection pool for reuse.
public final void release(boolean reusable) {
// If the response body comes from the cache, close it.
if (responseBodyIn == cachedResponseBody) {
IoUtils.closeQuietly(responseBodyIn);
}
if (!connectionReleased && connection != null) {
connectionReleased = true;
// We cannot reuse sockets that have incomplete output.
if (requestBodyOut != null && !requestBodyOut.closed) {
reusable = false;
}
// If the headers specify that the connection shouldn't be reused, don't reuse it.
if (hasConnectionCloseHeader()) {
reusable = false;
}
if (responseBodyIn instanceof UnknownLengthHttpInputStream) {
reusable = false;
}
if (reusable && responseBodyIn != null) {
// We must discard the response body before the connection can be reused.
try {
Streams.skipAll(responseBodyIn);
} catch (IOException e) {
reusable = false;
}
}
if (!reusable) {
connection.closeSocketAndStreams();
connection = null;
} else if (automaticallyReleaseConnectionToPool) {
HttpConnectionPool.INSTANCE.recycle(connection);
connection = null;
}
}
}
Note: I had previously answered couple of SO questions related to HttpURLConnection, which can help you in understanding the underlying implementation. Here are the links : Link1 and Link2.
As per how the TCP protocol works, when you close a connection, it doesn't automatically disappear from within your socket list.
When you send the termination signal to the other part, there starts a protocol (a procedure, morelike), where the first step is precisely your intention of closing the connection. You send a signal to the other node and that would involve the FIN_WAIT1 status.
When the user has received that signal, the next step is to acknowledge it from the remote side. This means that the opposite server sends you another signal symbolizing that the node is ready to close the connection too. That would be the FIN_WAIT2 status.
Between these two steps, it might happen that the remote node hasn't responded yet (so you're not been acknowledged that you want to close the connection). In that time, you would be in an intermediate state called CLOSE_WAIT (resuming: once you've sent the FIN signal to the remote server and they haven't responded yet).
The TIME_WAIT state would mean that you are giving some graceful time to the server before definitely closing it to receive some packets. You do this because connection anomalies might happen, and the remote server could have not received the 'disconnection' message and send you some packet. So when that happens, instead of creating a new socket between both nodes, you associate it to the one you have in the TIME_WAIT state and simply discard that packet because probably the sequence number will not be ordered.
There are some other states you might see, but according to the way you describe it, it seems pretty normal to me, unless when you call the .disconnect() method, the ESTABLISHED status would last. In that case something's not working as expected (it might be related to some kind of overloading or non-optimized code which might make your execution very slow).
You have to disconnect your open UrlConnection
HttpURLConnection urlConnection = null;
System.setProperty("http.keepAlive", "false");
try {
URL url = new URL(stringUrl);
urlConnection = (HttpURLConnection) url.openConnection();
InputStream instream = new BufferedInputStream(urlConnection.getInputStream());
...
instream.close();
urlConnection.disconnect(); //HERE
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (urlConnection!=null) {
urlConnection.disconnect();
}
}
I am trying to perform a simple get request using Apache HTTPClient however it seems as if all the code after the HTTPResponse response = client.execute(get); is being skipped. I am not able to access the contents of the response object,they are all null. However when I use debug mode and I explore the object I see all the data. This function is wrapped in an async task so I am wondering the task itself is not waiting on it to be executed or something I am not sure.
Something similar happened here:
Android code after httpclient.execute(httpget) doesn't get run in try (using AsyncTask)
Here is the code.
#Override
public T execute()
{
utils = new GeneralUtils();
if(getURL() == null)
{
throw new NullPointerException("No path specified");
}
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet(getURL());
Log.e(TAG,"client created");
if(getHeaders()!=null)
{
Log.e(TAG,"client created");
for(Map.Entry<String,String> header:getHeaders().entrySet())
{
get.addHeader(header.getKey(),header.getValue());
}
}
try
{
HttpResponse response = client.execute(get);
Log.e(TAG,"executed");
if(response==null)
Log.v(TAG,"its null as heell");
Log.v(TAG,response.getStatusLine().getReasonPhrase());
Log.v(TAG,String.valueOf(response.getStatusLine().getStatusCode()));
Log.e(TAG,getURL());
Log.v(TAG,"everything else is dead");
for(Header header:response.getAllHeaders())
{
Log.v(TAG,header.getName()+" "+header.getValue());
}
if(response.getStatusLine().getStatusCode() == 200)
{
if(response.getEntity().getContent()!=null)
{
try
{
if(utils.isExternalStorageWritable())
{
String path = getContext().getFilesDir().getAbsolutePath()+"/"+getFileCategory()+"/" +getAlarmId()+getFileExtension();
media = new File(path);
/**
* if the directory has not being created this function does the creation.
*/
media.mkdirs();
FileOutputStream fileOutputStream = new FileOutputStream(media);
IOUtils.copy(response.getEntity().getContent(),fileOutputStream);
fileOutputStream.close();
Log.e(TAG,media.getAbsolutePath());
return (T)media;
}
return null;
}
catch (ClientProtocolException e)
{
Log.v(TAG,e.getMessage());
}
catch (IOException e)
{
Log.v(TAG,e.getMessage());
}
}
}
}
catch (IOException e)
{
Log.e(TAG, e.getCause().getMessage());
e.printStackTrace();
}
return null;
}
The code is not throwing any exceptions so I am not sure about what's happening.
All the code after the response object does not work. It just returns null, As in as soon as I try to obtain a value from response like so response.getStatusCode(), it seems as if the code goes dead and just returns null.
Why don't you use a library that will handle all these restful connections?
I would recommend a couple:
https://github.com/darko1002001/android-rest-client (this is mine i have to mention it first :). I have built this library for the projects i build. For your case you would supply a parser which will give you an InputStream which you will just save as a file (as you do it now with IO utils). It handles the Asynchronous part of the whole thing and generally gives you a nice way to organize code.
http://square.github.io/retrofit/ - is another one that i have been playing around with. i think it is pretty well made and should be able to do whatever you want with it.
http://java.dzone.com/articles/android-%E2%80%93-volley-library - Volley is a project that came out straight from Google and it was demoed at the last Google IO conference. It handles all the async operations for you as well and enables you to do all these things. One thing that i am not really sure about is whether or not it will enable you to parse the responses in the background thread.
I would strongly suggest for you to use one of these as they might save you a lot of time.
If you do want to continue with your code then i would suggest to first investigate if some of the "if" blocks you have are skipped, use the debugger or add log messages to see if it enters the blocks. Go step by step and see what goes wrong.
I am doing something similar in my project, check out this file:
https://github.com/darko1002001/android-rest-client/blob/master/android-rest-lib/src/main/java/com/dg/libs/rest/client/BaseRestClient.java
I'm developing an app which needs to get music file by streaming for playing live.
In the request song api I can specify the bandwith (eg: 50kbps, 100kbps, 300, 600 or 1 Mbps).
The more the bandwith is big, the more the file will get time to be fetched. As I don't want the users to be restricted about that I have multiple choices to deal with it:
Detect wether the phone is using 3g, wifi or Edge and specify an bandwith for each connection speed.
Let the user decide the quality of the song he will get: like youtube (but the users won't be people that know much about computing: the more easy is the software, the more it will fit)
Having a way to evaluate properly the connection speed: Like fetching a file, measure the time that it took and set the bandwith.
I know that connection speed could vary a lot if user loose the wifi, or is using 3g moving in the street. And the thing is that I can't change the bandwidth when the song will be playing.
Maybe you have experience about that you would like to share?
Thank you!
Facebook released a library for this:
https://github.com/facebook/network-connection-class
this wasn't existing in 2011..
why not try to change your view of things.
Try to flow with your users. say your user wishes to download 128 kbit quality song. you start the download , WHILE downloading you make an average download time, take a few seconds for this average to stabilize, and if it's below certain value make a pop up to tell the user that his connection is too slow for the current bandwidth and ask him if to lessen the quality or to keep downloading slowly.
This will:
let the users the option to always assume they can get the best quality media.
let u do your check in runtime and change the quality accordingly while downloading without the need to pre check.
keeps your app simple to users.
I know i'm not answering your specific requirement, i'm just offering a different view.
protected String doInBackground(String... urls) {
String response = "";
startTime = System.currentTimeMillis();
for (String url : urls) {
DefaultHttpClient client = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
try {
HttpResponse execute = client.execute(httpGet);
InputStream content = execute.getEntity().getContent();
BufferedReader buffer = new BufferedReader(
new InputStreamReader(content));
String s = "";
while ((s = buffer.readLine()) != null) {
response += s;
}
endTime = System.currentTimeMillis();
} catch (Exception e) {
e.printStackTrace();
}
}
return response;
}
#Override
protected void onPostExecute(String result) {
// TODO Auto-generated method stub
long dataSize = result.length() / 1024;
takenTime = endTime - startTime;
long s = takenTime / 1000;
double speed = dataSize / s;
Toast.makeText(context, "" + s + "kbps", Toast.LENGTH_SHORT).show();
}
Detect network connection type on Android
You can check all available options here: http://developer.android.com/reference/android/telephony/TelephonyManager.html
This can fix the mobile network type but can't help you with the Wifi speed, you should code it by downloading something from a server you know and calculate the time.
I hope it helps.