Develop Reference

Implementing SSL pinning using Trustkit with Google Cloud Endpoints

I am using Google Cloud Endpoints to interact with the app engine backend for my Android app. I want to implement public key/SSL pinning. It's easy to do this for Android N and above, but i want to implement pinning for earlier versions of Android. It seems like a common way to do this is using Trustkit.
The Getting Started notes on the Trustkit link, describe how to set it up by setting the SSLSocketFactory, for example
// HttpsUrlConnection
HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
connection.setSSLSocketFactory(TrustKit.getInstance().getSSLSocketFactory(serverHostname));
// OkHttp 2.x
OkHttpClient client =
new OkHttpClient()
.setSSLSocketFactory(TrustKit.getInstance().getSSLSocketFactory(serverHostname));
But i'm not sure how to apply this to Google Cloud Endpoints because of the way the connection is set up, for example here's how its set up for Google Cloud Endpoints in my code
import com.xxxx.backend.myApi.MyApi;
//I believe the MyApi class is generated automatically as part of the Google Cloud Endpoints integration in Android Studio.
class EndpointsAsyncTask extends AsyncTask<HashMap, Void, String> {
#Override
protected String doInBackground(HashMap... params) {
HashMap<String, String> dict = params[0];
String data = dict.get("dataString");
MyApi.Builder builder = new MyApi.Builder(AndroidHttp.newCompatibleTransport(), new AndroidJsonFactory(), null)
.setRootUrl("https://xxxxx").setApplicationName("xxxx");
myApiService = builder.build();
//This calls the API method
return myApiService.customApiMethodName(data).execute().getDict().toString();
}
}
What I want to find out is how i can I implement SSL pinning in early versions of Android e.g. API16-API23 while still connecting to Google Cloud Endpoints in the same way my App does as shown above?
If possible i'd like to use Trustkit, maybe there is a different way i can set up how to connect with my Api methods so i can use it? If this is not possible is there an alternative to Trustkit i can use? or just a completely different way to implement SSL pinning while still using Google Cloud Endpoints in this way?

You would have to stop using the AndroidHttp helper and write your own, based off of it.
Instead of just calling new NetHttpTransport() or new ApacheHttpTransport(), you must use their builders instead, for example:
public static HttpTransport newCompatibleTransport(String hostname) {
SSLSocketFactory factory = TrustKit.getInstance().getSSLSocketFactory(serverHostname);
if (AndroidUtils.isMinimumSdkLevel(9)) {
return new NetHttpTransport.Builder()
.setSslSocketFactory(factory)
.build();
}
return new ApacheHttpTransport.Builder()
.setSslSocketFactory(factory)
.build();
}

HttpClient is removed from Android 6.0, how do i use it with RestTemplate?

I have the need to use the HttpClient to set certain parameters while using Spring's RestTemplate.
I currently do this via:
HttpClient httpClient = new HttpClient();
httpClient.getParams().setSoTimeout(prefs.getServerTimeout());
httpClient.getParams().setConnectionManagerTimeout(3000);
httpClient.getParams().setContentCharset("UTF-8");
httpClient.getParams().setCredentialCharset("ISO-8859-1", )
...
CommonsClientHttpRequestFactory requestFactory = new CommonsClientHttpRequestFactory(httpClient);
requestFactory.setReadTimeout(prefs.getServerTimeout());
RestTemplate restTemplate = new RestTemplate(requestFactory);
The HttpClient currently used everywhere, and for example in the
HttpComponentsClientHttpRequestFactory.getHttpClient()
Is pointing to the deprecated one shipped with Android.
Since it's deprecated, and removed from Android in 6.0, how do i go about continuing to use a HttpClient object with RestTemplate?
Since they share the same package (org.apache.http.client), i'm not sure how to make this work in pre/post 6.0.
(I tried using httpclient-android and HttpComponentsClientHttpRequestFactory without setting HttpClient, and it then seems to be using CloseableHttpClient. But the method signature is the deprecated HttpClient as mentioned.)
Pointers would be much appreciated.

You can add it's dependency explicitly in gradle or add jar but try considering modern options like volley 'com.android.volley:volley:1.0.0'
or okHttp 'com.squareup.okhttp3:okhttp:3.4.1' or you can use Rest-Template for android 'org.springframework.android:spring-android-rest-template:2.0.0.M3'.

I think you can use SimpleClientHttpRequestFactory from springframework (org.springframework.http.client.SimpleClientHttpRequestFactory) .
override the prepareConnection method to set timeout and other parameters.
public class MySimpleClientHttpRequestFactory extends SimpleClientHttpRequestFactory {
#Override
protected void prepareConnection(HttpURLConnection connection, String httpMethod) throws IOException {
super.prepareConnection(connection, httpMethod);
connection.setConnectTimeout(5000);
connection.setReadTimeout(10000);
}
}
Then set it to RestTemplate.
SimpleClientHttpRequestFactory factory = new MySimpleClientHttpRequestFactory();
RestTemplate template = new RestTemplate(factory)

Mock HttpResponse with Robolectric

Using Robolectric 2.3-SNAPSHOT, I want to test an object that'll execute a request in the background. In order to isolate it, I'm trying to mock the HttpResponse returned, without success after some hours invested.
I've created a project that anyone can clone. Simly run ./gradlew check https://github.com/Maragues/RobolectricDummyProject (git clone https://github.com/Maragues/RobolectricDummyProject.git)
I've tried
Robolectric.setDefaultHttpResponse(200, "my_mocked_word");
MockWebServer (https://code.google.com/p/mockwebserver/)
But the tests fail because they query the real URL
private static final String MOCKED_WORD = "MOCKED";
#Test
public void mockedRequestUsingMockServer() throws Exception {
mMockWebServer.enqueue(new MockResponse().setBody(MOCKED_WORD));
mMockWebServer.play();
Robolectric.getFakeHttpLayer().interceptHttpRequests(false);
Robolectric.getFakeHttpLayer().interceptResponseContent(false);
String result = request.loadDataFromNetwork();
assertEquals(MOCKED_WORD, result);
}
#Test
public void mockedRequestUsingRobolectric() throws Exception {
Robolectric.setDefaultHttpResponse(200, MOCKED_WORD);
String result = request.loadDataFromNetwork();
assertEquals(MOCKED_WORD, result);
}
The code executing the request
public String loadDataFromNetwork() throws Exception {
// With Uri.Builder class we can build our url is a safe manner
Uri.Builder uriBuilder = Uri.parse("http://robospice-sample.appspot.com/reverse").buildUpon();
uriBuilder.appendQueryParameter("word", word);
String url = uriBuilder.build().toString();
HttpURLConnection urlConnection = (HttpURLConnection) new URL(url)
.openConnection();
String result = IOUtils.toString(urlConnection.getInputStream());
urlConnection.disconnect();
return result;
}
Possibly related questions
Can't capture HTTP request with robolectric (I've tried that without success. Perhaps I'm missing something)
Anyone had success mocking HttpRequests with Robolectric? (I'm not using eclipse)

You're disabling Roboelectric's HTTP layer, so you're using the real HTTP layer. This means that there's no clever magic happening under the hood of your test: when you send an HTTP request, it's really going out onto the internet (as you are seeing).
MockWebServer doesn't stop this. It just sets up a web server locally, that your test can connect to.
So to resolve this problem, you need to stop attempting to connect to a real server, and instead, connect to the mock server. To do this, yo need to inject/set the URL in the request.
#Test
public void mockedRequestUsingMockServer() throws Exception {
mMockWebServer = new MockWebServer();
mMockWebServer.play();
mMockWebServer.enqueue(new MockResponse().setResponseCode(200).setBody(MOCKED_WORD));
request.myUrl = mMockWebServer.getUrl("/");
String result = request.loadDataFromNetwork();
assertEquals(MOCKED_WORD, result);
mMockWebServer.shutdown();
}

It turns out that Robolectric's FakeHttpLayer only works with Apache's HttpClient, which is highly discouraged on versions greater than Froyo. Extracted from Robolectric's Google Group
That being said, the usage of HttpUrlConnection will cause you trouble. I'd try to use Android's implementation of HttpClient where possible, since Robolectric intercepts all calls to that library and lets you set up canned responses to your HTTP calls. We're looking at doing the same for HttpUrlConnection, though it's not clear when that'll happen.
Apart from that, a unit test should not need to mock the HTTP layer. My approach was wrong from the beginning.

You can try this(ref:https://github.com/square/okhttp/tree/master/mockwebserver).
// Create a MockWebServer. These are lean enough that you can create a new
// instance for every unit test.
MockWebServer server = new MockWebServer();
// Schedule some responses.
server.enqueue(new MockResponse().setBody("it's all cool"));
// Start the server.
server.play();
// Ask the server for its URL. You'll need this to make HTTP requests.
//Http is my own http executor.
Http.Response response = http.get(server.getUrl("/"));
then, you can compare the response to server.enqueue(new MockResponse().setBody("it's all cool"));
MockWebServer is a part of okhttp https://github.com/square/okhttp/tree/master/mockwebserver. the URLConnectionImpl in android 4.4 have been changed from defaultHttpClient to Okhttp.

Android HttpUrlConnection EOFException

I would like to know if there are known issues on Android with HttpUrlConnection and POST requests. We are experiencing intermittent EOFExceptions when making POST requests from an Android client. Retrying the same request will eventually work. Here is a sample stack trace:
java.io.EOFException
at libcore.io.Streams.readAsciiLine(Streams.java:203)
at libcore.net.http.HttpEngine.readResponseHeaders(HttpEngine.java:579)
at libcore.net.http.HttpEngine.readResponse(HttpEngine.java:827)
at libcore.net.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:283)
at libcore.net.http.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:497)
at libcore.net.http.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:134)
There are many similar bug reports and posts to stack overflow but I cannot understand if there really is an issue and if so, what versions of Android are affected and what the proposed fix/work around is.
Here are some of the similar reports I am referring to:
Android HttpsUrlConnection eofexception
Android HttpURLConnection throwing EOFException
EOFException and FileNotFoundException in HttpURLConnection getInputStream()
https://code.google.com/p/google-http-java-client/issues/detail?id=213
https://code.google.com/p/android/issues/detail?id=29509
https://code.google.com/p/google-http-java-client/issues/detail?id=230
https://code.google.com/p/android/issues/detail?id=41576
Here is a potential Android framework fix
https://android.googlesource.com/platform/libcore/+/19aa40c81c48ff98ccc7272f2a3c41479b806376
I do know there was an issue with poisoned connections in the connection pool in pre-Froyo but these issues are occurring on new ICS+ devices exclusively. If there were a problem on later devices I would expect some kind of official Android documentation of the issue.

Our conclusion is that there is an issue in the Android platform. Our workaround was to catch the EOFException and retry the request N number of times. Below is the pseudo code:
private static final int MAX_RETRIES = 3;
private ResponseType fetchResult(RequestType request) {
return fetchResult(request, 0);
}
private ResponseType fetchResult(RequestType request, int reentryCount) {
try {
// attempt to execute request
} catch (EOFException e) {
if (reentryCount < MAX_RETRIES) {
fetchResult(request, reentryCount + 1);
}
}
// continue processing response
}

HttpURLConnection library internally maintains a pool of Connections. So, whenever a request is send, it first checks if there is an existing connection already present in the pool, based on which it decides to create a new one.
These connections are nothing but sockets, and this library by default does not closes these sockets. It may sometimes happen that a connection (socket) which is not currently being used and is present in the pool is no longer usable as the Server may choose to terminate the connection after some time. Now, since the connection even though is closed by the server, the library does not knows about it and assumes the connection/socket to be still connected. Thus it sends the new request using this stale connection and hence we get EOFException.
The best way to handle this is to check the Response Headers after each request you send. The server always sends a "Connection: Close" before terminating a connection (HTTP 1.1). So, you can use getHeaderField() and check for "Connection" field. Another thing to note is that server ONLY sends this connection field when it is about to terminate the connection. So, you need to code around this with the possibility of getting a "null" in the normal case (when server is not closing the connection)

This workaround tends to be reliable and performant:
static final int MAX_CONNECTIONS = 5;
T send(..., int failures) throws IOException {
HttpURLConnection connection = null;
try {
// initialize connection...
if (failures > 0 && failures <= MAX_CONNECTIONS) {
connection.setRequestProperty("Connection", "close");
}
// return response (T) from connection...
} catch (EOFException e) {
if (failures <= MAX_CONNECTIONS) {
disconnect(connection);
connection = null;
return send(..., failures + 1);
}
throw e;
} finally {
disconnect(connection);
}
}
void disconnect(HttpURLConnection connection) {
if (connection != null) {
connection.disconnect();
}
}
This implementation relies on the fact that the default number of connections that can be opened with a server is 5 (Froyo - KitKat). This means that up to 5 stale connections may exist, each of which will have to be closed.
After each failed attempt, the Connection:close request property will cause the underlying HTTP engine to close the socket when connection.disconnect() is called. By retrying up to 6 times (max connections + 1), we ensure that the last attempt will always be given a new socket.
The request may experience additional latency if no connections are alive, but that is certainly better than an EOFException. In that case, the final send attempt won't immediately close the freshly opened connection. That's the only practical optimization that can be made.
Instead of relying on the magic default value of 5, you may be able to configure the system property yourself. Keep in mind that this property is accessed by a static initializer block in KitKat's ConnectionPool.java, and it works like this in older Android versions too. As a result, the property may be used before you have a chance to set it.
static final int MAX_CONNECTIONS = 5;
static {
System.setProperty("http.maxConnections", String.valueOf(MAX_CONNECTIONS));
}

Yes. There is a problem in the Android platform, specifically, in Android libcore with version 4.1-4.3.
The problem is introduced in this commit: https://android.googlesource.com/platform/libcore/+/b2b02ac6cd42a69463fd172531aa1f9b9bb887a8
Android 4.4 switched http lib to "okhttp" which doesn't have this problem.
Problem explained as follow:
On android 4.1-4.3, when you are using URLConnection/HttpURLConnection to POST with "ChunkedStreamingMode" or "FixedLengthStreamingMode" set, URLConnection/HttpURLConnection will not do silent retry if the reused connection is stale. You should retry POST at most "http.maxConnections+1" times in your code, just as previous answers suggest.

I suspect it might be the server that is at fault here, and the HttpURLConnection is not as forgiving as other implementations. That was the cause of my EOFException. I suspect in my case this would not be intermittent (fixed it before testing the N retry workaround), so the answers above relate to other issues and be a correct solution in those cases.
My server was using python SimpleHTTPServer and I was wrongly assuming all I needed to do to indicate success was the following:
self.send_response(200)
That sends the initial response header line, a server and a date header, but leaves the stream in the state where you are able to send additional headers too. HTTP requires an additional new line after headers to indicate they are finished. It appears if this new line isn't present when you attempt to get the result body InputStream or response code etc with HttpURLConnection then it throws the EOFException (which is actually reasonable, thinking about it). Some HTTP clients did accept the short response and reported the success result code which lead to me perhaps unfairly pointing the finger at HttpURLConnection.
I changed my server to do this instead:
self.send_response(200)
self.send_header("Content-Length", "0")
self.end_headers()
No more EOFException with that code.

This worked for me.
public ResponseObject sendPOST(String urlPrefix, JSONObject payload) throws JSONException {
String line;
StringBuffer jsonString = new StringBuffer();
ResponseObject response = new ResponseObject();
try {
URL url = new URL(POST_URL);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setReadTimeout(10000);
connection.setConnectTimeout(15000);
connection.setRequestMethod("POST");
connection.setRequestProperty("Accept", "application/json");
connection.setRequestProperty("Content-Type", "application/json; charset=UTF-8");
OutputStream os = connection.getOutputStream();
os.write(payload.toString().getBytes("UTF-8"));
os.close();
BufferedReader br = new BufferedReader(new InputStreamReader(connection.getInputStream()));
while ((line = br.readLine()) != null) {
jsonString.append(line);
}
response.setResponseMessage(connection.getResponseMessage());
response.setResponseReturnCode(connection.getResponseCode());
br.close();
connection.disconnect();
} catch (Exception e) {
Log.w("Exception ",e);
return response;
}
String json = jsonString.toString();
response.setResponseJsonString(json);
return response;
}

connection.addRequestProperty("Accept-Encoding", "gzip");
is the answer

HttpUrlConnection.openConnection fails second time

I know this issue should be fixed with System.setProperty("http.keepAlive", "false"); before openConnection, but that didn't work to me. First try on this code works, second one fails. Even if i try this request after less than 5 seconds, it also works. If i wait more than that, it fails again
This is my code:
System.setProperty("http.keepAlive", "false");
HttpURLConnection conn = (HttpURLConnection) mURL.openConnection();
conn.setUseCaches(false);
conn.setRequestProperty("Connection","Keep-Alive");
conn.setRequestProperty("User-Agent", useragent);
conn.setConnectTimeout (30000) ;
conn.setDoOutput(true);
conn.setDoInput(true);
consumer.sign(conn);
InputSource is = new InputSource(conn.getInputStream());
I get the exception on last line:
java.io.IOException: Write error: I/O error during system call, Broken pipe
W/System.err( 2164): at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl.nativewrite(Native Method)
W/System.err( 2164): at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl.access$600(OpenSSLSocketImpl.java:55)
W/System.err( 2164): at org.apache.harmony.xnet.provider.jsse.OpenSSLSocketImpl$SSLOutputStream.write(OpenSSLSocketImpl.java:583)
W/System.err( 2164): at java.io.OutputStream.write(OutputStream.java:82)
W/System.err( 2164): at org.apache.harmony.luni.internal.net.www.protocol.http.HttpURLConnectionImpl.sendRequest(HttpURLConnectionImpl.java:1332)
W/System.err( 2164): at org.apache.harmony.luni.internal.net.www.protocol.http.HttpURLConnectionImpl.doRequestInternal(HttpURLConnectionImpl.java:1656)
W/System.err( 2164): at org.apache.harmony.luni.internal.net.www.protocol.http.HttpURLConnectionImpl.doRequest(HttpURLConnectionImpl.java:1649)
W/System.err( 2164): at org.apache.harmony.luni.internal.net.www.protocol.http.HttpURLConnectionImpl.getInputStream(HttpURLConnectionImpl.java:1153)
W/System.err( 2164): at org.apache.harmony.luni.internal.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:253)
Does someone have an idea about what's wrong here?. Thanks!

The connection pool used by HttpURLConnection when it is keeping connections alive is broken such that it tries to use connections that have been closed by the server. By default Android sets KeepAlive on all connections.
System.setProperty("http.keepAlive", "false"); is a workaround that disables KeepAlive for all connections so then you avoid the bug in the connection pool.
conn.setRequestProperty("Connection","Keep-Alive"); turns KeepAlive on for this particular connection, essentially reversing what System.setProperty("http.keepAlive", "false"); does.
Also I always explicitly call connect() as it makes it clear where you are ending your connection setup. I'm not sure if calling this method is optional or not.
System.setProperty("http.keepAlive", "false");
HttpURLConnection conn = (HttpURLConnection) mURL.openConnection();
conn.setUseCaches(false);
conn.setRequestProperty("User-Agent", useragent);
conn.setConnectTimeout(30000);
conn.setDoOutput(true);
conn.setDoInput(true);
consumer.sign(conn);
conn.connect();
InputSource is = new InputSource(conn.getInputStream());

You dont need the System.setProperty("http.keepAlive", "false");
All you need is conn.setRequestProperty("connection", "close");
this fixes the issue but effectively kills keep alives and therefore potentially makes multiple connections slower (which is a shame). I was looking through the harmony bug tracker but could not really find anything.
#fonetik, do you know whether this is already raised with harmony? I mean not that it helps much since another http related luni defect is still unassigned after more than a month.

I solved the problem. Here I leave you the code, in case it might be helpful for someone. Basically I see a trend on Google for using HttpClient/HttpGet instead of HttpUrlConnection. So I tried with those classes, and everything worked:
final HttpClient client = new DefaultHttpClient();
final HttpGet conn = new HttpGet(mURL.toString());
OAuthConsumer consumer = mOAuthManager.getPostConsumer();
consumer.sign(conn);
HttpResponse response = client.execute(conn);
InputSource is = new InputSource(response.getEntity().getContent());

this bug had beed fixed in Android2.3 version,as we know System.setProperty("http.keepAlive", "false"); is not a very good solution ,because on mobile device ,create each connection is every time is high cost.

I believe your problem lies in the order of your code. Check those methods in the URLConnection JavaDocs - setRequestProperty should not be called after the connection is made on mUrl.openConnection(). It may be working the first time because the connection is made, then you are changing settings that are not affecting anything until the next time you try to make a connection. Try using the HttpURLConnection constructor instead so you can call connect() after you have set the properties.

When i am trying to open https connection it is working fine but second time it fails because i have set the system property value instead of HttpsURLConnection connection. I have got the java.io.IOException: Write error: I/O issue while opening the https connection second time. I have used following code in my applicaiton.
System.setProperty("http.proxyHost", proxy);
System.setProperty("http.proxyPort", port);
But when i changed the same to below it works fine.
javax.net.ssl.HttpsURLConnection ucon = (javax.net.ssl.HttpsURLConnection) urlWPF.openConnection(proxyserver);
ucon.setRequestProperty("http.proxyHost", proxy);
ucon.setRequestProperty("http.proxyPort", port);
If you set the system property, it will applicable for entire application. If you want reset the same, you can follow two ways. One is you have to take server refresh and second one is you have to change the HttpsURLConnection.setRequestProperty which is mentioned above where required.