Develop Reference

Android HttpDefaultClient times out only after another device makes a similar but not identical request

Some quick background. We have multiple devices running a scanner app which checks against a database to see whether an id has been scanned in or not. I can scan in with Device A as many times as I like without issue. I then pick up Device B and scan in, also as many or few times as I like. If I pick Device A back up and scan, the HttpClient will hang for approximately 60 seconds refusing to send any further requests. The code below has commented the point of failure.
// Asynchronous get request
private class aGETRequest extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
String response = "";
for (String url : urls) {
DefaultHttpClient client = new DefaultHttpClient();
client.setKeepAliveStrategy(null);
HttpGet httpGet = new HttpGet(url);
try {
HttpResponse httpResponse = client.execute(httpGet); //Hangs Here
HttpEntity httpEntity = httpResponse.getEntity();
InputStreamReader isr = new InputStreamReader(httpEntity.getContent());
BufferedReader buffer = new BufferedReader(isr);
String s = "";
while ((s = buffer.readLine()) != null) {
response += s;
}
httpEntity.consumeContent();
isr.close();
} catch (Exception e) {
e.printStackTrace();
}
httpGet.abort();
client.getConnectionManager().shutdown();
}
return response;
}
#Override
protected void onPostExecute(String result) {
results(result);
}
}
The client hangs and even snooping traffic shows no requests sent at all from Device A after the failure. You ready for the best part? If the devices are going through a Proxy server, it works. W.T.F?

Android is java 6 compat . right.
BufferedReader on java 7 makes me nervous and the while read loop appears to be whats hanging.....
I would try a different different read loop class thats solid on java 6 or i would find someone else's pattern for httpclient that's solid.
My wild guess is that your code is never getting out of the following...
while ((s = buffer.readLine()) != null)
Maybe the server is returing chunked encoding or something like that with a diff protocol ( pattern of length=0 followed by \r\n or something.

How to check for unrestricted Internet access? (captive portal detection)

I need to reliably detect if a device has full internet access, i.e. that the user is not confined to a captive portal (also called walled garden), i.e. a limited subnet which forces users to submit their credentials on a form in order to get full access.
My app is automating the authentication process, and therefore it is important to know that full internet access is not available before starting the logon activity.
The question is not about how to check that the network interface is up and in a connected state. It is about making sure the device has unrestricted internet access as opposed to a sandboxed intranet segment.
All the approaches I have tried so far are failing, because connecting to any well-known host would not throw an exception but return a valid HTTP 200 response code because all requests are routed to the login page.
Here are all the approaches I tried but they all return true instead of false for the reasons explained above:
1:
InetAddress.getByName(host).isReachable(TIMEOUT_IN_MILLISECONDS);
isConnected = true; <exception not thrown>
2:
Socket socket = new Socket();
SocketAddress sockaddr = new InetSocketAddress(InetAddress.getByName(host), 80);
socket.connect(sockaddr, pingTimeout);
isConnected = socket.isConnected();
3:
URL url = new URL(hostUrl));
URLConnection urlConn = url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) urlConn;
httpConn.setAllowUserInteraction(false);
httpConn.setRequestMethod("GET");
httpConn.connect();
responseCode = httpConn.getResponseCode();
isConnected = responseCode == HttpURLConnection.HTTP_OK;
So, how do I make sure I connected to an actual host instead of the login redirection page? Obviously, I could check the actual response body from the 'ping' host I use but it does not look like a proper solution.

For reference, here is the 'official' method from the Android 4.0.1 AOSP code base:
WifiWatchdogStateMachine.isWalledGardenConnection(). I am including the code below just in case the link breaks in the future.
private static final String mWalledGardenUrl = "http://clients3.google.com/generate_204";
private static final int WALLED_GARDEN_SOCKET_TIMEOUT_MS = 10000;
private boolean isWalledGardenConnection() {
HttpURLConnection urlConnection = null;
try {
URL url = new URL(mWalledGardenUrl); // "http://clients3.google.com/generate_204"
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setInstanceFollowRedirects(false);
urlConnection.setConnectTimeout(WALLED_GARDEN_SOCKET_TIMEOUT_MS);
urlConnection.setReadTimeout(WALLED_GARDEN_SOCKET_TIMEOUT_MS);
urlConnection.setUseCaches(false);
urlConnection.getInputStream();
// We got a valid response, but not from the real google
return urlConnection.getResponseCode() != 204;
} catch (IOException e) {
if (DBG) {
log("Walled garden check - probably not a portal: exception "
+ e);
}
return false;
} finally {
if (urlConnection != null) {
urlConnection.disconnect();
}
}
}
This approach relies on a specific URL, mWalledGardenUrl = "http://clients3.google.com/generate_204" always returning a 204 response code. This will work even if DNS has been interfered with since in that case a 200 code will be returned instead of the expected 204. I have seen some captive portals spoofing requests to this specific URL in order to prevent the Internet not accessible message on Android devices.
Google has a variation of this theme: fetching http://www.google.com/blank.html will return a 200 code with a zero-length response body. So if you get a non-empty body this would be another way to figure out that you are behind a walled garden.
Apple has its own URLs for detecting captive portals: when network is up IOS and MacOS devices would connect to an URL like http://www.apple.com/library/test/success.html, http://attwifi.apple.com/library/test/success.html, or http://captive.apple.com/hotspot-detect.html which must return an HTTP status code of 200 and a body containing Success.
NOTE:
This approach will not work in areas with regionally restricted Internet access such as China where the whole country is a walled garden, and where some Google/Apple services might be blocked. Some of these might not be blocked: http://www.google.cn/generate_204, http://g.cn/generate_204, http://gstatic.com/generate_204 or http://connectivitycheck.gstatic.com/generate_204 — yet these all belong to google so not guaranteed to work.

Another possible solution might be to connect via HTTPS and inspect the target certificate. Not sure if walled gardens actually serve the login page via HTTPS or just drop the connections. In either case, you should be able to see that your destination is not the one you expected.
Of course, you also have the overhead of TLS and certificate checks. Such is the price of authenticated connections, unfortunately.

I believe preventing redirection for your connection will work.
URL url = new URL(hostUrl));
HttpURLConnection httpConn = (HttpURLConnection)url.openConnection();
/* This line prevents redirects */
httpConn.setInstanceFollowRedirects( false );
httpConn.setAllowUserInteraction( false );
httpConn.setRequestMethod( "GET" );
httpConn.connect();
responseCode = httpConn.getResponseCode();
isConnected = responseCode == HttpURLConnection.HTTP_OK;
If that doesn't work, then I think the only way to do it is to check the body of the response.

This has been implemented on Android 4.2.2+ version - I find their approach fast and interesting :
CaptivePortalTracker.java detects walled garden as follows
- Try to connect to www.google.com/generate_204
- Check that the HTTP response is 204
If the check fails, we are in a walled garden.
private boolean isCaptivePortal(InetAddress server) {
HttpURLConnection urlConnection = null;
if (!mIsCaptivePortalCheckEnabled) return false;
mUrl = "http://" + server.getHostAddress() + "/generate_204";
if (DBG) log("Checking " + mUrl);
try {
URL url = new URL(mUrl);
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setInstanceFollowRedirects(false);
urlConnection.setConnectTimeout(SOCKET_TIMEOUT_MS);
urlConnection.setReadTimeout(SOCKET_TIMEOUT_MS);
urlConnection.setUseCaches(false);
urlConnection.getInputStream();
// we got a valid response, but not from the real google
return urlConnection.getResponseCode() != 204;
} catch (IOException e) {
if (DBG) log("Probably not a portal: exception " + e);
return false;
} finally {
if (urlConnection != null) {
urlConnection.disconnect();
}
}
}

if you are already using retrofit you can do it by retrofit. just make a ping.html page and send an head request to it using retrofit and make sure your http client is configured like below: (followRedirects(false) part is the most important part)
private OkHttpClient getCheckInternetOkHttpClient() {
return new OkHttpClient.Builder()
.readTimeout(2L, TimeUnit.SECONDS)
.connectTimeout(2L, TimeUnit.SECONDS)
.followRedirects(false)
.build();
}
then build your retrofit like below:
private InternetCheckApi getCheckInternetRetrofitApi() {
return (new Retrofit.Builder())
.baseUrl("[base url of your ping.html page]")
.addConverterFactory(GsonConverterFactory.create(new Gson()))
.client(getCheckInternetOkHttpClient())
.build().create(InternetCheckApi.class);
}
your InternetCheckApi.class would be like:
public interface InternetCheckApi {
#Headers({"Content-Typel: application/json"})
#HEAD("ping.html")
Call<Void> checkInternetConnectivity();
}
then you can use it like below:
getCheckInternetOkHttpClient().checkInternetConnectivity().enqueue(new Callback<Void>() {
public void onResponse(Call<Void> call, Response<Void> response) {
if(response.code() == 200) {
//internet is available
} else {
//internet is not available
}
}
public void onFailure(Call<Void> call, Throwable t) {
//internet is not available
}
}
);
note that your internet check http client must be separate from your main http client.

This is best done here as in AOSP :
https://github.com/aosp-mirror/platform_frameworks_base/blob/6bebb8418ceecf44d2af40033870f3aabacfe36e/core/java/android/net/captiveportal/CaptivePortalProbeResult.java#L61
https://github.com/aosp-mirror/platform_frameworks_base/blob/e3a0f42e8e8678f6d90ddf104d485858fbb2e35b/services/core/java/com/android/server/connectivity/NetworkMonitor.java
private static final String GOOGLE_PING_URL = "http://google.com/generate_204";
private static final int SOCKET_TIMEOUT_MS = 10000;
public boolean isCaptivePortal () {
try {
URL url = new URL(GOOGLE_PING_URL);
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setConnectTimeout(SOCKET_TIMEOUT_MS);
urlConnection.setReadTimeout(SOCKET_TIMEOUT_MS);
urlConnection.setUseCaches(false);
urlConnection.getInputStream();
return (urlConnection.getResponseCode() != 204)
&& (urlConnection.getResponseCode() >= 200)
&& (urlConnection.getResponseCode() <= 399);
} catch (Exception e) {
// for any exception throw an exception saying check was unsuccesful
} finally {
if (urlConnection != null) {
urlConnection.disconnect();
}
}
}
Please note this will probably not work on a proxy network and something more advanced as in the AOSP url needs to be done

Android HttpURLConnection throwing EOFException

Here's the question in simplest way.
I create a HTTPS connection to my server through proxy using HttpUrlConnection Object.
My proxy closes the connection but my code still tries to reuse the same connection. And so I get EOFException.
How do I handle such cases?

I'd recommend disabling the http.keepalive system property. The performance section of the documentation indicates that socket connections will be reused when possible. It sounds like it is trying to reuse the connection, but the proxy has already closed it. On this post, Darrell also indicates that changing the system property solved his problem.
System.setProperty("http.keepAlive", "false");

Turns out they've fixed this issue in Android on Jan 8th [1]. The fix basically marks the old connection as recycled and internally retries the request.
To fix this for now, I would suggest retrying requests if an EOFException is encountered with a retry limit to prevent stackoverlows.
https://android.googlesource.com/platform/libcore/+/19aa40c81c48ff98ccc7272f2a3c41479b806376

I had this problem with normal HTTP connections as well. The first request went OK, the second failed with an EOFException.
Eventuelly I fixed it by removing...
urlConnection.setChunkedStreamingMode(0);
...from the HttpUrlConnection.
I could be that the webserver I'm calling can't handle chuncked data well. Don't know.

If you don't want to reuse the connection then release it.
finally {
urlConnection.disconnect();
}

You can use this method to pick data from server then you convert the inputs trim to string then you can parse for further use.`
public static InputStream getUrlData(final String url)
throws URISyntaxException, ClientProtocolException, IOException {
final DefaultHttpClient client = new DefaultHttpClient();
final HttpGet method = new HttpGet(new URI(url));
final HttpResponse res = client.execute(method);
return res.getEntity().getContent();
}

Maybe httpClient "has more bugs" and is deprecated, but this problem with JellyBean is a showstopper. I am using Ksoap2 so I tried all the suggested answers that I could.
System.setProperty("http.keepAlive", "false");
httpTransportSE.getServiceConnection().setRequestProperty("Connection", "close");
httpTransportSE.getServiceConnection().disconnect();
Nothing worked - my solution was to rollback the version of Ksoap2 I'm using from 3.1.1 to 2.6.5. Using 2.6.5 the problem is substantially reduced. Still testing but maybe even solved.

I found that retrying the connection fixes the issue as seen here: https://stackoverflow.com/a/20302767/2520390
Make sure you close off the connection before your recursive call.
Also, I added the following to the connection to close the connection, though I'm not sure if it helps:
if (retries > 0) {
connection.setRequestProperty("Connection", "close");
}

You shouldn't be attempting to reuse the same HttpURLConnection instance. The docs in the very bottom line of the "Class Overview" say
Each instance of HttpURLConnection may be used for one
request/response pair.
Keep-Alive connections work at a different level, see the disconnect docs:
http://developer.android.com/reference/java/net/HttpURLConnection.html#disconnect()
Unlike other Java implementations, this will not necessarily close
socket connections that can be reused. You can disable all connection
reuse by setting the http.keepAlive system property to false before
issuing any HTTP requests.
So you should always use a fresh HttpURLConnection and let the socket pool handle re-use.
There were apparently bugs with keep-alive connections pre-Froyo (2.2) so it is recommended to disable keep-alive on those old devices.
In my case the EOFException was caused by my server not sending a full response, see the details here: https://stackoverflow.com/a/27845172/2335025

You shouldn't be attempting to reuse the same HttpURLConnection instance. The docs in the very bottom line of the "Class Overview" say
Each instance of HttpURLConnection may be used for one
request/response pair.
Keep-Alive connections work at a different level, see the disconnect docs:
http://developer.android.com/reference/java/net/HttpURLConnection.html#disconnect()
Unlike other Java implementations, this will not necessarily close
socket connections that can be reused. You can disable all connection
reuse by setting the http.keepAlive system property to false before
issuing any HTTP requests.
So you should always use a fresh HttpURLConnection and let the socket pool handle re-use. There are perhaps issues if it tries to reuse a socket that has been closed by the server, which the answers to this question deal with: Android HttpUrlConnection EOFException
There were apparently bugs with keep-alive connections pre-Froyo (2.2) so it is recommended to disable keep-alive on those old devices.
In my case the EOFException was caused by my server not sending a full response, see the details here: https://stackoverflow.com/a/27845939/2335025

if (Build.VERSION.SDK != null
&& Build.VERSION.SDK_INT > 13) {
con.setRequestProperty("Connection", "close");
}

Try this code:`
Httppost method:
HttpParams httpParams = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(httpParams, TIMEOUT_MILLISEC);
HttpConnectionParams.setSoTimeout(httpParams, TIMEOUT_MILLISEC);
HttpClient client = new DefaultHttpClient(httpParams);
HttpPost request = new HttpPost("put_url");
request.setHeader("Content-Type", "application/xml");
String file = resourceXml();
StringEntity se = null;
try {
se = new StringEntity(file);
} catch (UnsupportedEncodingException e1) {
e1.printStackTrace();
}
se.setContentEncoding("UTF-8");
se.setContentType("application/xml");
request.setEntity(se);
HttpResponse response = null;
try {
response = client.execute(request);
} catch (ClientProtocolException e1) {
e1.printStackTrace();
} catch (IOException e1) {
e1.printStackTrace();
}
HttpEntity entity = response.getEntity();
InputStream is = null;
try {
is = entity.getContent();
} catch (IllegalStateException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
String _response = convertStreamToString(is);
Log.i(TAG, "Response:" + _response);
// Check if server response is valid code
int res_code = response.getStatusLine().getStatusCode();
Log.i(TAG, "status_code" + res_code);
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
`

to convert stream to string:`
private static String convertStreamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is),
8192);
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append((line + "\n"));
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}`

Android HTTP Client freezes

I'm experiencing some odd behavior in my HTTP requests. I have some users that are saying that this call isn't ever coming back (the spinner marking it's asynchronous call never goes away). I have seen this happen before, but I attributed it to the emulator going through Charles Proxy. I haven't yet seen it on actual phone until now.
I'm not sure what would cause this to happen, which is why I'm posting it here. Here's the call, using Jackson to deserialize the result into a Value Object. The two spots I saw the emulator freeze are httpclient.execute(httpGet); and getObjectMapper().readValue(jp, SyncVO.class);.
While debugging, stepping over the offending statement caused the debugger to never gain control back of stepping. Meanwhile, I see the request go out AND come back from the server through Charles. It's just that the app doesn't seem to get the response and just sits there.
So, here's the code. Thanks for any help!
public SyncVO sync(String userId, long lastUpdate, boolean includeFetch) throws IOException {
SyncVO result = null;
String url = BASE_URL + "users/" + userId + "/sync" + "?" + "fetch=" + includeFetch;
if (lastUpdate > 0) {
url += "&updatedSince=" + lastUpdate;
}
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
httpGet.setHeader("Accept", "application/json");
httpGet.setHeader("Accept-Encoding", "gzip");
httpGet.setHeader(AUTHORIZATION, BEARER + " " + mOAuthToken);
httpclient.getParams().setParameter(CoreProtocolPNames.USER_AGENT, USER_AGENT_STRING);
httpclient.getParams().setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, false);
HttpResponse response = httpclient.execute(httpGet);
if (isUnauthorized(response)) {
APPLICATION.needReauthentication();
return null;
}
if (response != null) {
InputStream stream = response.getEntity().getContent();
Header contentEncoding = response.getFirstHeader("Content-Encoding");
if (contentEncoding != null && contentEncoding.getValue().equalsIgnoreCase("gzip")) {
stream = new GZIPInputStream(stream);
}
InputStreamReader inReader = new InputStreamReader(stream, "UTF-8");
JsonParser jp = mJsonFactory.createJsonParser(inReader);
result = getObjectMapper().readValue(jp, SyncVO.class);
}
return result;
}
private ObjectMapper getObjectMapper() {
return (new ObjectMapper()
.configure(Feature.AUTO_DETECT_FIELDS, true)
.configure(Feature.FAIL_ON_UNKNOWN_PROPERTIES, false)
.configure(JsonParser.Feature.ALLOW_UNQUOTED_CONTROL_CHARS, true));
}

don't forget to consume entities content after each request.
HttpEntity entity = response.getEntity();
try {
if (entity != null)
entity.consumeContent();
} catch (IOException e) {
e.printStackTrace();
}

You should definitely use connection timeout and socket read and be prepared for the worst from the server. Network operations will never be 100% predictable and there is not much your client can do then so make sure you code optimally.
httpParameters = httpclient.getParams();
HttpConnectionParams.setConnectionTimeout(httpParameters, 5000);
HttpConnectionParams.setSoTimeout(httpParameters, 10000);
You can also cancel a task with asyncTask.cancel(true);

The reason is because you have left stream open. As such, the response is left in limbo. This means your global variable httpClient is also left in limbo, and unable to get a new entity when it re-uses the client.
You should call close() after finishing with the stream.
stream.close();

Network calls take a while and will block the UI thread. Same with your jackson deserialization code. This stuff needs to be put on a separate thread. See AsyncTask for an easy way to do it.

HttpURLConnection with POST "times out" on ICS, works in HC

My Android tablet application does not work with ICS due to a Login problem. When I looked at my code and ran it under debug mode on an ICS tablet, I see the problem but I don't understand it. The code functions correctly on all Honeycomb models that i have tested and in fact I have two tablets hooked up to my computer (one Samsung Galaxy Tab running 3.2, and a Motorola Xoom wifi running 4.0.3) and the code fails on ICS and works on HC.
The failure is a Socket Timeout exception. The timeout was 2000ms, but I upped it to 100000ms to test and it had no impact.
Using the browser on the ICS tablet, I can go to the URL and it responds, so it doesn't appear to be network related.
I am running on a background thread using AsyncTask.
Slurp just takes all of the input from the InputStream and using StringBuilder creates a string representation. Its not actually useful in this request but I added it to see what the server was replying with.
I am POSTing to the page the same way a user authenticates using the form, which is why I am using x-www-form-urlencoded.
Again, this code functions perfectly on Honeycomb but fails on ICS.
The code makes a connection but fails when it asks for a response from the server, almost like the server is still waiting for something... anyway, here is the code:
static public String authenticate(String service_url, String username, String password) throws IOException {
if (username == null || password == null)
throw new IOException();
String charset = "UTF-8";
String query = String.format("Email=%s&Password=%s",URLEncoder.encode(username, charset),URLEncoder.encode(password, charset));
byte [] data = query.getBytes(charset);
URL url = new URL(service_url);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-Length", Integer.toString(data.length));
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.setReadTimeout(5000); // 2 second timeout.
try {
connection.connect();
DataOutputStream pw = new DataOutputStream (connection.getOutputStream());
pw.writeBytes(query);
pw.flush();
pw.close();
int code = connection.getResponseCode(); //SOCKET TIMEOUT HERE
if (code == 200 || code == 302)
{
InputStream is = connection.getInputStream();
String value = slurp(is);
List<String> cookies = connection.getHeaderFields().get("Set-Cookie");
if (cookies == null)
throw new IOException();
for (String cookie : cookies) {
if (cookie.startsWith("cpms")) {
cookieTime = new DateTime(); //crazy but the expires time in the cookie is not actually accurate.
return cookie; // this is the only correct path out.
}
}
}
else
Logger.e(StaticUtils.class, "Invalid response code while logging in: " + code);
}
catch (IOException ioe)
{
Logger.e(StaticUtils.class, ioe);
throw ioe; // log it and then throw it back.
} finally {
connection.disconnect();
}
return null;
}