Develop Reference

HTTP CONNECT method with Android

I want to extend an existing Android app which sets up a http connection to a remote device which it sends commands to and receives values from.
The feature consists of a tunneled connection via a custom proxy server that has been set up. I have the http header format given which should make the proxy server create and provide a tunnel for my app.
CONNECT <some id>.<target ip>:80 HTTP/1.1
Host: <proxy ip>:443
Authorization: basic <base64 encoded auth string>
# Here beginns the payload for the target device. It could be whatever.
GET / HTTP/1.1
Host: 127.0.0.1:80
The app uses the Apache HttpClient library to handle it's connections, and I would like to integrate with that. This is not mandatory, however.
The authorization is standard conform basic auth.
I have trouble implementing this because it is not clear to me how the HttpClient is intended to be used for such behaviour.
There is no CONNECT method in the library, only GET, POST and so on. I figured this would then be managed by the proxy settings of the HttpClient instance.
The problem here is that the request line is not standard, since the CONNECT line contains an id which the custom proxy then would parse and interpret.
I now would like to know if there is any intended method to implement this using the Apache HttpClient and what it would look like with this sample data given, or if I have to implement my own method for this. And if so, which interface (there are a few that would sound reasonable to inherit from) it should implement.
Any explanation, snippet or pointer would be appreciated.
UPDATE:
I now have a small snippet set up, without Android. Just plain Java and Apache HttpClient. I still think the Host mismatch in the request is a problem, since I can't manage to establish a connection.
final HttpClient httpClient = new DefaultHttpClient();
// Set proxy
HttpHost proxy = new HttpHost (deviceId + "." + "proxy ip", 443, "https");
httpClient.getParams().setParameter (ConnRoutePNames.DEFAULT_PROXY, proxy);
final HttpGet httpGet = new HttpGet("http://" + "target device ip");
httpGet.addHeader ("Authorization", "Basic" +
Base64.encodeBase64String((username + ":" + password).getBytes()));
// Trying to overvrite the host in the header containing the device Id
httpGet.setHeader("Host", "proxy ip");
System.out.println("Sending request..");
try {
final HttpResponse httpResponse = httpClient.execute (httpGet);
final InputStream inputStream = httpResponse.getEntity ().getContent ();
final InputStreamReader inputStreamReader =
new InputStreamReader(inputStream, "ISO-8859-1");
final BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
final StringBuilder stringBuilder = new StringBuilder ();
String bufferedStrChunk = null;
while ((bufferedStrChunk = bufferedReader.readLine ()) != null) {
stringBuilder.append (bufferedStrChunk);
}
System.out.println("Received String: " + stringBuilder.toString());
}
catch (final ClientProtocolException exception) {
System.out.println("ClientProtocolException");
exception.printStackTrace();
}
catch (final IOException exception) {
System.out.println("IOException");
exception.
}
This looks fairly good to me in the way of "it could actually work".
Anyways, I receive the following log and trace:
Sending request..
2015/03/03 13:16:16:199 CET [DEBUG] ClientParamsStack - 'http.route.default-proxy': https://"device id"."proxy ip":443
2015/03/03 13:16:16:207 CET [DEBUG] SingleClientConnManager - Get connection for route HttpRoute[{}->https://"device id"."proxy ip":443->http://"target device ip"]
2015/03/03 13:16:16:549 CET [DEBUG] ClientParamsStack - 'http.tcp.nodelay': true
2015/03/03 13:16:16:549 CET [DEBUG] ClientParamsStack - 'http.socket.buffer-size': 8192
2015/03/03 13:16:16:563 CET [DEBUG] DefaultClientConnection - Connection shut down
2015/03/03 13:16:16:563 CET [DEBUG] SingleClientConnManager - Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter#bc6a08
Exception in thread "main" java.lang.IllegalStateException: Unable to establish route.
planned = HttpRoute[{}->https://"device id"."proxy ip":443->http://"target device ip"]
current = HttpRoute[{s}->https://"device id"."proxy ip":443->http://"target device ip"]
at org.apache.http.impl.client.DefaultRequestDirector.establishRoute(DefaultRequestDirector.java:672)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:385)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
at run.main(run.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Any ideas what goes wrong?
UPDATE: As stated here, this might be due to a bug when redirecting. The fact that the target does redirect tells me, that I do not have reached the correct target, implying that the Host parameter may have not been overwritten.

In fact, this can't be done with the HttpClient at all, I tried at the wrong level. It works when done with a TcpSocket (or a SSLSocket). The custom CONNECT header can simply be assembled and sent like that:
final Socket tcpSocket = SSLSocketFactory.getDefault().createSocket("host ip", 443);
String connect = "custom CONNECT header";
tcpSocket.getOutputStream().write((connect).getBytes());
tcpSocket.getOutputStream().flush();
The response from the server can then be read with a BufferedReader or whatever.

Do Android apps force the use of SSL 3?

In light of the recent POODLE vulnerability, the servers that I talk to in my Android app are disabling SSL 3.
The code where I make my HTTP requests looks something like this
AndroidHttpClient c = AndroidHttpClient.newInstance("My Client/1.0");
try {
SyncBasicHttpContext httpcontext = getHttpContext();
HttpResponse jsonResponse = c.execute(request, httpcontext);
Status status = jsonResponse.getStatus();
} catch (Exception e) {
// ...etc
}
Is there anything I will need to do to my code to continue to be able to talk to the severs?

Android HttpUrlConnection: Post Multipart

I'm trying to post a test File to a spring rest servlet deployed on tomcat using Android. I'm developing on Android 4.1.2, but I have verified same problem on 4.0.3.
The problem is that the file upload requires a very long time (about 70 seconds for a 4MB file), also in local network. The time is equiparable using a 3g connection. I've excluded that it could be a server problem: executing the same call with curl it takes 1 / 2 seconds, and using apache as backend results are the same.
Using HttpClient works fine.
I'm using Spring Android RestClient 1.0.1.RELEASE and, given Android version and the fact that I'm not overriding default behaviour, it uses HttpUrlConnection instead of HttpClient to make http requests.
I have also implemented my custom ClientHttpRequestFactory in order to manipulate some details of SSL connection and I have defined my own implementation of ClientHttpRequestInterceptor in order to modify authentication header.
I have also set setBufferRequestBody(false) in order to avoid OutOfMemoryException on big files. But this property have no effects on time required.
MyClientHttpRequestFactory:
public class MyClientHttpRequestFactory extends SimpleClientHttpRequestFactory{
#Override
protected void prepareConnection(HttpURLConnection connection, String httpMethod) throws IOException {
super.prepareConnection(connection, httpMethod);
connection.setConnectTimeout(240 * 1000);
connection.setReadTimeout(240 * 1000);
if ("post".equals(httpMethod.toLowerCase())) {
setBufferRequestBody(false);
}else {
setBufferRequestBody(true);
}
}
#Override
protected HttpURLConnection openConnection(URL url, Proxy proxy) throws IOException {
final HttpURLConnection httpUrlConnection = super.openConnection(url, proxy);
if (url.getProtocol().toLowerCase().equals("https")
&&
settings.selfSignedCert().get())
{
try {
((HttpsURLConnection)httpUrlConnection).setSSLSocketFactory(getSSLSocketFactory());
((HttpsURLConnection)httpUrlConnection).setHostnameVerifier(new NullHostnameVerifier());
} catch (Exception e) {
MyLog.e(LOG_TAG, "OpenConnection", e);
}
}
return httpUrlConnection;
}
MyClientHttpRequestInterceptor:
public class MyClientHttpRequestInterceptor implements ClientHttpRequestInterceptor {
#Override
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution) throws IOException {
final HttpHeaders headers = request.getHeaders();
headers.setAuthorization(new HttpBasicAuthentication( settings.username().get(), settings.password().get()));
if (settings.enable_gzip().get()) {
headers.setAcceptEncoding(ContentCodingType.GZIP);
}
return execution.execute(request, body);
}
}
And here my Rest call:
List<ClientHttpRequestInterceptor> interceptors = Arrays.asList((ClientHttpRequestInterceptor)myClientHttpRequestInterceptor);
RestTemplate restTemplate = new RestTemplate();
restTemplate.getMessageConverters().add(new FormHttpMessageConverter());
restTemplate.getMessageConverters().add(new StringHttpMessageConverter());
restTemplate.getMessageConverters().add(new MappingJackson2HttpMessageConverter());
restTemplate.setInterceptors(interceptors);
MultiValueMap<String, Object> parts = new LinkedMultiValueMap<String, Object>();
parts.add("file", new FileSystemResource("/sdcard/test/4MB_file"));
HttpEntity<MultiValueMap> requestEntity = new HttpEntity<MultiValueMap>(parts);
restTemplate.exchange(myUrl, HttpMethod.POST, requestEntity, Integer.class).getBody();
}
Looking at Spring Android source code, the next lines of code my request is passing through are:
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
HttpURLConnection connection = openConnection(uri.toURL(), this.proxy);
prepareConnection(connection, httpMethod.name());
if (this.bufferRequestBody) {
return new SimpleBufferingClientHttpRequest(connection);
} else {
return new SimpleStreamingClientHttpRequest(connection, this.chunkSize);
}
}
Because of this.bufferRequestBody is false, return new SimpleStreamingClientHttpRequest(connection, this.chunkSize); is executed (with chunkSize = 0)
SimpleStreamingClientHttpRequest(HttpURLConnection connection, int chunkSize) {
this.connection = connection;
this.chunkSize = chunkSize;
// Bugs with reusing connections in Android versions older than Froyo (2.2)
if (olderThanFroyo) {
System.setProperty("http.keepAlive", "false");
}
}
and then:
ClientHttpRequest delegate = requestFactory.createRequest(request.getURI(), request.getMethod());
delegate.getHeaders().putAll(request.getHeaders());
if (body.length > 0) {
FileCopyUtils.copy(body, delegate.getBody());
}
return delegate.execute();
From here is all android subsystem I think..
I have dumped tcp traffic and analyzed it:
POST /urlWherePost HTTP/1.1
Content-Type: multipart/form-data;boundary=nKwsP85ZyyzSDuAqozCTuZOSxwF1jLAtd0FECUPF
Authorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxxx=
Accept-Encoding: gzip
User-Agent: Dalvik/1.6.0 (Linux; U; Android 4.1.2; sdk Build/MASTER)
Host: 192.168.168.225:8080
Connection: Keep-Alive
Content-Length: 4096225
--nKwsP85ZyyzSDuAqozCTuZOSxwF1jLAtd0FECUPF
Content-Disposition: form-data; name="file"; filename="4MB_file"
Content-Type: application/octet-stream
Content-Length: 4096000
I've tryed to re-create similar request with curl:
curl --verbose
-H "Connection: Keep-Alive"
-H "Content-Type: multipart/form-data"
-H "Accept-Encoding: gzip"
-H "Content-Disposition: form-data; name=\"file\"; filename=\"4MB_file\""
-H "Content-Type: application/octet-stream"
--user xxx:xxx
-X POST
--form file=#4MB_file
http://192.168.168.225:8080/urlWherePost
but with curl the post is ok.
Posting json data is not a problem (maybe small body size). But when I try to send "big" files the time increase.
Looking in DDMS shell, on Network Statistics I've also found that the network throughput is never over 250kb in TX. There seems to be a bootleneck, but how to investigate it? Where I can look, which parameter can I change?
Thank you for any suggestion!

Have you tried using the MultipartEntity method? I had the same problem when downloading big amounts of JSON data from the server, but I switched to this method and caught all the data that the server provided me.
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost("http://myurl.com");
try {
MultipartEntity entity = new MultipartEntity();
entity.addPart("type", new StringBody("json"));
entity.addPart("data", new JSONObject(data));
httppost.setEntity(entity);
HttpResponse response = httpclient.execute(httppost);
} catch (ClientProtocolException e) {
} catch (IOException e) {
} catch (JSONException e){
}

To upload big files, You can use this library android-async-http

For simple and easy to use, I recommend this lib https://github.com/koush/ion.
I use it on my project and it works perfectly.

How to use Socket.IO with server demanding authentication

This is where I got the socketIO files from.
https://github.com/Gottox/socket.io-java-client/tree/master/src/io/socket
I am on the client side.
I know connecting works when the server does not need authentication.
But when it needs authentication (Username and password), I get a handshaking error message.
How do I get passed authentication?? Could it be a server side error? Would the server side of things change if authentication was added?
This is the function that throws an error...I did not write it.
This line is the one causing problems: InputStream stream = connection.getInputStream();
It says it is caused by: java.io.FileNotFoundException: url:80/socket.io/1/
private void handshake() {
URL url;
String response;
URLConnection connection;
try {
setState(STATE_HANDSHAKE);
url = new URL(IOConnection.this.url.toString() + SOCKET_IO_1);
connection = url.openConnection();
if (connection instanceof HttpsURLConnection) {
((HttpsURLConnection) connection)
.setSSLSocketFactory(sslContext.getSocketFactory());
}
connection.setConnectTimeout(connectTimeout);
connection.setReadTimeout(connectTimeout);
/* Setting the request headers */
for (Entry<Object, Object> entry : headers.entrySet()) {
connection.setRequestProperty((String) entry.getKey(),
(String) entry.getValue());
}
InputStream stream = connection.getInputStream();
Scanner in = new Scanner(stream);
response = in.nextLine();
String[] data = response.split(":");
sessionId = data[0];
heartbeatTimeout = Long.parseLong(data[1]) * 1000;
closingTimeout = Long.parseLong(data[2]) * 1000;
protocols = Arrays.asList(data[3].split(","));
} catch (Exception e) {
error(new SocketIOException("Error while handshaking", e));
}
}

Problem solved (sort of), here: Android developpement, Gottox socket.io-java-client: file not fount Exception /socket.io/1/
(try using an earlier version of socket.io - by first deleting socket.io folder from node_modules and then install an older version, e.g., 0.9.16, using this command: npm install socket.io#0.9.16)

Android HttpURLConnection throwing EOFException

Here's the question in simplest way.
I create a HTTPS connection to my server through proxy using HttpUrlConnection Object.
My proxy closes the connection but my code still tries to reuse the same connection. And so I get EOFException.
How do I handle such cases?

I'd recommend disabling the http.keepalive system property. The performance section of the documentation indicates that socket connections will be reused when possible. It sounds like it is trying to reuse the connection, but the proxy has already closed it. On this post, Darrell also indicates that changing the system property solved his problem.
System.setProperty("http.keepAlive", "false");

Turns out they've fixed this issue in Android on Jan 8th [1]. The fix basically marks the old connection as recycled and internally retries the request.
To fix this for now, I would suggest retrying requests if an EOFException is encountered with a retry limit to prevent stackoverlows.
https://android.googlesource.com/platform/libcore/+/19aa40c81c48ff98ccc7272f2a3c41479b806376

I had this problem with normal HTTP connections as well. The first request went OK, the second failed with an EOFException.
Eventuelly I fixed it by removing...
urlConnection.setChunkedStreamingMode(0);
...from the HttpUrlConnection.
I could be that the webserver I'm calling can't handle chuncked data well. Don't know.

If you don't want to reuse the connection then release it.
finally {
urlConnection.disconnect();
}

You can use this method to pick data from server then you convert the inputs trim to string then you can parse for further use.`
public static InputStream getUrlData(final String url)
throws URISyntaxException, ClientProtocolException, IOException {
final DefaultHttpClient client = new DefaultHttpClient();
final HttpGet method = new HttpGet(new URI(url));
final HttpResponse res = client.execute(method);
return res.getEntity().getContent();
}

Maybe httpClient "has more bugs" and is deprecated, but this problem with JellyBean is a showstopper. I am using Ksoap2 so I tried all the suggested answers that I could.
System.setProperty("http.keepAlive", "false");
httpTransportSE.getServiceConnection().setRequestProperty("Connection", "close");
httpTransportSE.getServiceConnection().disconnect();
Nothing worked - my solution was to rollback the version of Ksoap2 I'm using from 3.1.1 to 2.6.5. Using 2.6.5 the problem is substantially reduced. Still testing but maybe even solved.

I found that retrying the connection fixes the issue as seen here: https://stackoverflow.com/a/20302767/2520390
Make sure you close off the connection before your recursive call.
Also, I added the following to the connection to close the connection, though I'm not sure if it helps:
if (retries > 0) {
connection.setRequestProperty("Connection", "close");
}

You shouldn't be attempting to reuse the same HttpURLConnection instance. The docs in the very bottom line of the "Class Overview" say
Each instance of HttpURLConnection may be used for one
request/response pair.
Keep-Alive connections work at a different level, see the disconnect docs:
http://developer.android.com/reference/java/net/HttpURLConnection.html#disconnect()
Unlike other Java implementations, this will not necessarily close
socket connections that can be reused. You can disable all connection
reuse by setting the http.keepAlive system property to false before
issuing any HTTP requests.
So you should always use a fresh HttpURLConnection and let the socket pool handle re-use.
There were apparently bugs with keep-alive connections pre-Froyo (2.2) so it is recommended to disable keep-alive on those old devices.
In my case the EOFException was caused by my server not sending a full response, see the details here: https://stackoverflow.com/a/27845172/2335025

You shouldn't be attempting to reuse the same HttpURLConnection instance. The docs in the very bottom line of the "Class Overview" say
Each instance of HttpURLConnection may be used for one
request/response pair.
Keep-Alive connections work at a different level, see the disconnect docs:
http://developer.android.com/reference/java/net/HttpURLConnection.html#disconnect()
Unlike other Java implementations, this will not necessarily close
socket connections that can be reused. You can disable all connection
reuse by setting the http.keepAlive system property to false before
issuing any HTTP requests.
So you should always use a fresh HttpURLConnection and let the socket pool handle re-use. There are perhaps issues if it tries to reuse a socket that has been closed by the server, which the answers to this question deal with: Android HttpUrlConnection EOFException
There were apparently bugs with keep-alive connections pre-Froyo (2.2) so it is recommended to disable keep-alive on those old devices.
In my case the EOFException was caused by my server not sending a full response, see the details here: https://stackoverflow.com/a/27845939/2335025

if (Build.VERSION.SDK != null
&& Build.VERSION.SDK_INT > 13) {
con.setRequestProperty("Connection", "close");
}

Try this code:`
Httppost method:
HttpParams httpParams = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(httpParams, TIMEOUT_MILLISEC);
HttpConnectionParams.setSoTimeout(httpParams, TIMEOUT_MILLISEC);
HttpClient client = new DefaultHttpClient(httpParams);
HttpPost request = new HttpPost("put_url");
request.setHeader("Content-Type", "application/xml");
String file = resourceXml();
StringEntity se = null;
try {
se = new StringEntity(file);
} catch (UnsupportedEncodingException e1) {
e1.printStackTrace();
}
se.setContentEncoding("UTF-8");
se.setContentType("application/xml");
request.setEntity(se);
HttpResponse response = null;
try {
response = client.execute(request);
} catch (ClientProtocolException e1) {
e1.printStackTrace();
} catch (IOException e1) {
e1.printStackTrace();
}
HttpEntity entity = response.getEntity();
InputStream is = null;
try {
is = entity.getContent();
} catch (IllegalStateException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
String _response = convertStreamToString(is);
Log.i(TAG, "Response:" + _response);
// Check if server response is valid code
int res_code = response.getStatusLine().getStatusCode();
Log.i(TAG, "status_code" + res_code);
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
`

to convert stream to string:`
private static String convertStreamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is),
8192);
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append((line + "\n"));
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}`