Develop Reference

Posting data to web service from Android doesn't get sent

I am working on a project which was working fine before Christmas but suddenly doesn't without any changes being made.
The project involves a C++ which listens on a particular port and listens to post requests in a REST API to process the data and store in a database.
It consists of an Android library which gathers information and then sends this as an HTTP POST to the rest API on the C++ app.
The C++ app prints out the HTTP response that was received straight from receiving it on the socket before any processing done. First the android app has to send an initalisation request to the C++ app, the C++ prints the request and shows post data was sent, and successfully initalises and sends a response back to android including a session cookie. I then re-use the HttpClient within Android to post the next request which contains a fair amount of data but this request doesn't work.
When stepping through the android library I can see the post values have been successfully set and are being used to perform the HTTP request, however, the C++ app only receives the HTTP headers, not any post data.
If in the second request, I replace the post data with only a couple of post fields, the C++ then sees the post data, so it looks like the DefaultHTTPClient in Android isn't sending the post data if the post data is quite large.
Below is how I am posting the data in Android
if (httpClient == null)
{
//AndroidHttpClient client = new AndroidHttpClient();
httpClient = new DefaultHttpClient();
}
HttpParams httpParams = httpClient.getParams();
HttpConnectionParams.setConnectionTimeout(httpParams, 3000);
HttpConnectionParams.setSoTimeout(httpParams, 3000);
HttpPost httpPost = new HttpPost(serverURL);
//httpPost.setHeader("Content-Type", "application/x-www-form-urlencoded");
httpPost.setHeader("Authorisation-Token", authorisationToken);
httpPost.setHeader("Connection", "close");
//httpPost.setHeader("Cookie", "SESSIONID=zd8d5n3kucysl4idug1911m7ye");
httpPost.setEntity(new UrlEncodedFormEntity(postData, "UTF-8"));
String headers = "";
responseHandler = new ResponseHandler<String>()
{
#Override
public String handleResponse(HttpResponse httpResponse) throws ClientProtocolException, IOException
{
return EntityUtils.toString(httpResponse.getEntity());
}
};
responseBody = httpClient.execute(httpPost, responseHandler);
Log.d("Response", responseBody);
//httpClient = null;
if (responseBody != null && !responseBody.isEmpty())
{
httpClient.getConnectionManager().closeIdleConnections(0, TimeUnit.NANOSECONDS);
JSONObject jsonObject = new JSONObject(responseBody);
return jsonObject;
}
else
{
httpClient.getConnectionManager().closeIdleConnections(0, TimeUnit.NANOSECONDS);
return null;
}
As an example the post data is being created as follows:
postData = new ArrayList<>();
postData.add(new BasicNameValuePair("Test", "Item 1"));
postData.add(new BasicNameValuePair("Test 2", "Item 2"));
and the above is passed in to the execute command of the ASyncTask.
Below is how I am receiving the data on the socket from C++
string LinuxSocket::receiveDataOnSocket(int *socket)
{
string receiveData = "";
char * temp = NULL;
int bytesReceived = 0;
do
{
bytesReceived = recv(*socket, this->buffer, this->bufferLength, 0);
if (bytesReceived < 0)
{
stringstream logstream;
logstream << "Failed to read data on socket. Error: " << strerror(bytesReceived);
this->bitsLibrary->writeToLog(logstream.str(), "LinuxSocket", "receiveDataOnSocket");
this->closeSocket(socket);
throw SocketException(strerror(bytesReceived));
}
//If we got here then we should be able to get some data
temp = new char[bytesReceived + 1];
strncpy(temp, this->buffer, bytesReceived);
temp[bytesReceived] = '\0';
receiveData.append(temp);
delete[] temp;
temp = NULL;
memset(this->buffer, 0, this->bufferLength);
} while (bytesReceived == this->bufferLength);
return receiveData;
}
The post data that I am sending is a follows:
POST /crash HTTP/1.1
User-Agent: My Android User Agent
Authorisation-Token: DlzSIkx4ro*OatHCV6epfWY0F
Connection: close
Content-Length: 1231
Content-Type: application/x-www-form-urlencoded
Host: 192.168.1.123:500
Cookie: SESSIONID=6rc5q1db0z8lupe5uij5ten3mw
Cookie2: $Version=1
Severity=Critical&DeviceID=36e85611db16c7fa&VersionName=1.5&DeviceType=Android&ROMBuild=sdk_gphone_x86-userdebug+8.0.0+OSR1.170901.056+4497355+dev-keys&KernelVersion=3.18.81%2B&DeviceBrand=google&DeviceModel=Android+SDK+built+for+x86&APILevel=26&ScreenResolution=1080+x+1776&Locale=English&MobileNetwork=Android&CrashType=Handled&ExceptionType=java.lang.Exception&Stacktrace=java.lang.Exception%3A+Standard+Exception+been+thrown%0A%09at+com.MyCompany.MyApp.MainActivity%242.onClick%28MainActivity.java%3A67%29%0A%09at+android.view.View.performClick%28View.java%3A6256%29%0A%09at+android.view.View%24PerformClick.run%28View.java%3A24701%29%0A%09at+android.os.Handler.handleCallback%28Handler.java%3A789%29%0A%09at+android.os.Handler.dispatchMessage%28Handler.java%3A98%29%0A%09at+android.os.Looper.loop%28Looper.java%3A164%29%0A%09at+android.app.ActivityThread.main%28ActivityThread.java%3A6541%29%0A%09at+java.lang.reflect.Method.invoke%28Native+Method%29%0A%09at+com.android.internal.os.Zygote%24MethodAndArgsCaller.run%28Zygote.java%3A240%29%0A%09at+com.android.internal.os.ZygoteInit.main%28ZygoteInit.java%3A767%29%0A&CustomProperty=%7B%22Test+Non+Json+Property%22%3A%22Here+is+my+value%22%7D&AppID=15869700
As an example the post data that successfully works is a follows:
POST /initialise HTTP/1.1
Authorisation-Token: DlzSIkx4ro*OatHCV6epfWY0F
Connection: close
Content-Length: 48
Content-Type: application/x-www-form-urlencoded
Host: 192.168.1.123:500
User-Agent: My Android User Agent
ApplicationID=15869700&DeviceID=36e85611db16c7fa
I've also captured the request from Android and sent it from a Rest API Test Client Insomnia.Rest and sent this to the Rest API and the C++ successfully sees all of the post data, so it looks like the problem is the Android library won't send post data if its of a certain size.
Is this the case, and how can I get round this?

I found the issue with this, there were two separate things that I've done to resolve this.
The issue with no post data at all being sent, I changed from the org.apache.DefaultHTTPClient to the OkHTTPClient. From looking at Google it looks like the apache version has been deprecated and the OkHTTPClient is the preferred HTTP client anyway. This then sent some post data but not everything. This brings me to the second fix (I don't think the issue was actually with the apache client, I think it was my C++ receiving data on the socket).
The second fix I did was to change my receiveDataOnSocket function in the C++ app.
When I've worked on socket to socket communication then normally if the buffer is full, then usually I can expect to receive more data, if the buffer is only partly full, then all data is sent and I return the received data.
It looks like HTTP doesn't necessarily send as much as the buffer on receiving socket can handle, I've therefore changed my receive data on socket to process the data as its being received to look for the Content-Length header, and the length of the body that I may have, then on each subsequent recv call, I increment the current body count by the receivedBytes count and if the body length is equal to the content length I stop receiving data on the socket and return the HTTP request.

Django: resetting password without a CSRF token

I have a Django website that manages Users. Using the built-in functionality, users can request a password reset from the website and that works great. I have implemented it according to this tutorial so I am using the built-in password reset functionality.
I have an Android app from which users should also be able to request a password reset. The problem is that I do not have a CSRF token in the application, and the the built-in password_reset method has the #csrf_protect decorator. This means that I cannot access it without a CSRF token and I also can't modify it with the #csrf_exempt decorator.
So the next idea is to create a function, which generates a CSRF token, stores it in the request and redirects to the correct URL which sends the reset email. The problem is that according to this, django does not allow to pass POST parameters further in a redirect.
Therefore my question is how can I request a password reset in Django without a CSRF token? Alternatively, what is the correct way to request this from an application?

I found a solution myself. Please feel free to post any alternative solutions. One that doesn't require two separate requests would be particularly great.
If you look at the password_reset method, you can see that it only tries to process the request as a reset request if the request method is POST. Otherwise it just returns a TemplateResponse containing a form. This also contains the CSRF token as a cookie.
So first, I send a GET request to http://myaddress.com/user/password/reset/ and extract the CSRF cookie from the response. Then I send a POST request containing the cookie, the email address and 2 headers (see below).
This is the code I've implemented to achieve this from Android (trimmed):
String url = "http://myaddress.com/user/password/reset/";
GET Request:
HttpClient httpClient = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
CookieStore cookieStore = new BasicCookieStore();
HttpContext localContext = new BasicHttpContext();
localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
HttpResponse httpResponse = httpClient.execute(httpGet, localContext);
Cookie csrfCookie = null;
for (Cookie cookie : cookieStore.getCookies()) {
if (cookie.getName() == "csrftoken") {
csrfCookie = cookie;
break;
}
}
if (csrfCookie == null) {
throw new NullPointerException("CSRF cookie not found!");
}
return csrfCookie;
Note that you want the CookieStore from org.apache.http.client.
POST Request:
HttpClient httpClient = new DefaultHttpClient();
HttpContext localContext = new BasicHttpContext();
HttpPost httpPost = new HttpPost(url);
// Prepare the cookie store to receive cookies.
CookieStore cookieStore = new BasicCookieStore();
cookieStore.addCookie(csrfCookie);
httpPost.setHeader("Referer", url);
httpPost.setHeader("X-CSRFToken", csrfCookie.getValue());
localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE);
builder.addTextBody("email", emailAddressToReset);
httpPost.setEntity(builder.build());
HttpResponse httpResponse = httpClient.execute(httpPost, localContext);
if (httpResponse.getStatusLine().getStatusCode() != 200) {
throw new Exception("Could not reset password!");
}
Toast.makeText(context, "Password reset requested! Please check your email inbox!", Toast.LENGTH_LONG).show();

How to unit test a class that uses HttpClient in Android using the built-in framework?

I've got a class:
public class WebReader implements IWebReader {
HttpClient client;
public WebReader() {
client = new DefaultHttpClient();
}
public WebReader(HttpClient httpClient) {
client = httpClient;
}
/**
* Reads the web resource at the specified path with the params given.
* #param path Path of the resource to be read.
* #param params Parameters needed to be transferred to the server using POST method.
* #param compression If it's needed to use compression. Default is <b>true</b>.
* #return <p>Returns the string got from the server. If there was an error downloading file,
* an empty string is returned, the information about the error is written to the log file.</p>
*/
public String readWebResource(String path, ArrayList<BasicNameValuePair> params, Boolean compression) {
HttpPost httpPost = new HttpPost(path);
String result = "";
if (compression)
httpPost.addHeader("Accept-Encoding", "gzip");
if (params.size() > 0){
try {
httpPost.setEntity(new UrlEncodedFormEntity(params, "UTF-8"));
} catch (UnsupportedEncodingException e1) {
e1.printStackTrace();
}
}
try {
HttpResponse response = client.execute(httpPost);
StatusLine statusLine = response.getStatusLine();
int statusCode = statusLine.getStatusCode();
if (statusCode == 200) {
HttpEntity entity = response.getEntity();
InputStream content = entity.getContent();
if (entity.getContentEncoding() != null
&& "gzip".equalsIgnoreCase(entity.getContentEncoding()
.getValue()))
result = uncompressInputStream(content);
else
result = convertStreamToString(content);
} else {
Log.e(MyApp.class.toString(), "Failed to download file");
}
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return result;
}
private String uncompressInputStream(InputStream inputStream)
throws IOException {...}
private String convertStreamToString(InputStream is) {...}
}
I cannot find a way to test it using a standard framework. Especially, I need to simulate total internet lost from inside the test.
There are suggestions to manually turn the Internet in the emulator off while performing the test. But it seems to me as not quite a good solution, because the automatic tests should be... automatic.
I added a "client" field to the class trying to mock it from inside the test class. But implementation of the HttpClient interface seems quite complex.
The Robolectric framework allows the developers to test Http connection as far as I know. But I guess there is some way to write such a test without using so big additional framework.
So are there any short and straightforward ways of unit testing classes that use HttpClient? How did you solve this in your projects?

I added a "client" field to the class trying to mock it from inside the test class. But implementation of the HttpClient interface seems quite complex.
I am a little bit confuse about this statement. From the question title, you are asking about unit-testing httpClint, by mocking a FakeHttpClient may help you unit-testing other part of app except httpClient, but doesn't help anything for unit-testing httpClient. What you need is a FakeHttpLayer for unit-testing httpClient (no remote server, network requires, hence unit-testing).
HttpClient Dummy Test:
If you only need examine app behavior in the situation that internet is lost, then a classic Android Instrument Test is sufficient, you can programmatically turn the Internet in the emulator off while performing the test:
public void testWhenInternetOK() {
... ...
webReader.readWebResource();
// expect HTTP 200 response.
... ...
}
public void testWhenInternetLost() {
... ...
wifiManager = (WifiManager) this.getSystemService(Context.WIFI_SERVICE);
wifiManager.setWifiEnabled(false);
webReader.readWebResource();
// expect no HTTP response.
... ...
}
This requires the remote http server is completely setup and in a working state, and whenever you run your test class, a real http communication is made over network and hit on http server.
HttpClient Advanced Test:
If you want to test app behavior more precisely, for instance, you want to test a http call in you app to see if it is handle different http response properly. the Robolectric is the best choice. You can use FakeHttpLayer and mock the http request and response to whatever you like.
public void setup() {
String url = "http://...";
// First http request fired in test, mock a HTTP 200 response (ContentType: application/json)
HttpResponse response1 = new DefaultHttpResponseFactory().newHttpResponse(HttpVersion.HTTP_1_1, 200, null);
BasicHttpEntity entity1 = new BasicHttpEntity();
entity1.setContentType("application/json");
response1.setEntity(entity1);
// Second http request fired in test, mock a HTTP 404 response (ContentType: text/html)
HttpResponse response2 = new DefaultHttpResponseFactory().newHttpResponse(HttpVersion.HTTP_1_1, 404, null);
BasicHttpEntity entity2 = new BasicHttpEntity();
entity2.setContentType("text/html");
response2.setEntity(entity2);
List<HttpResponse> responses = new ArrayList<HttpResponse>();
responses.add(response1);
responses.add(response2);
Robolectric.addHttpResponseRule(new FakeHttpLayer.UriRequestMatcher("POST", url), responses);
}
public void testFoo() {
... ...
webReader.readWebResource(); // <- a call that perform a http post request to url.
// expect HTTP 200 response.
... ...
}
public void testBar() {
... ...
webReader.readWebResource(); // <- a call that perform a http post request to url.
// expect HTTP 404 response.
... ...
}
Some pros of using Robolectric are:
Purely JUnit test, no instrument test so don't need start emulator (or real device) to run the test, increase development speed.
Latest Robolectric support single line of code to enable/disable FakeHttpLayer, where you can set http request to be interpreted by FakeHttpLayer (no real http call over network), or set the http request bypass the FakeHttpLayer(perform real http call over network). Check out this SO question for more details.
If you check out the source of Robolectric, you can see it is quite complex to implement a FakeHtppLayer properly by yourself. I would recommend to use the existing test framework instead of implementing your own API.
Hope this helps.

Using HttpGet returning complete HTML code

I am trying to invoke a private web-service in which there's one link I've to access using GET method. While using the direct URL on browser (needs to login first), I get the data in JSON format. The URL I am invoking is like this
http://www.example.com/trip/details/860720?format=json
The url is working fine, but when I invoke this using HttpGet, I am getting the HTML coding of the webpage, instead of the JSON String. The code I am using is as follows:
private String runURL(String src,int id) { //src="http://www.example.com/trip/details/"
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(src);
String responseBody="";
BasicHttpParams params=new BasicHttpParams();
params.setParameter("domain", token); //The access token I am getting after the Login
params.setParameter("format", "json");
params.setParameter("id", id);
try {
httpget.setParams(params);
HttpResponse response = httpclient.execute(httpget);
responseBody = EntityUtils.toString(response.getEntity());
Log.d("runURL", "response " + responseBody); //prints the complete HTML code of the web-page
} catch (Exception e) {
e.printStackTrace();
}
return responseBody;
}
Can you tell me what am I doing wrong here??

Try specify Accept & Content-Type in you http header:
httpget.setHeader("Accept", "application/json"); // or application/jsonrequest
httpget.setHeader("Content-Type", "application/json");
Note that you can use tools like wireshark capture and analyse the income and outcome http package, and figure out the exact style of the http header that returns your json response from a standard browser.
Update:
You mentioned need login first when using browser, the html content returned is probably the login page (if use basic authentication type, it returns a short html response with status code 401, so a modern browser knows how to handle, more specifically, pop up login prompt to user), so the first try would be checking the status code of your http response:
int responseStatusCode = response.getStatusLine().getStatusCode();
Depend on what kind of authentication type you use, you probably need specify login credentials in your http request as well, something like this (if it is a basic authentication):
httpClient.getCredentialsProvider().setCredentials(
new AuthScope("http://www.example.com/trip/details/860720?format=json", 80),
new UsernamePasswordCredentials("username", "password");

How to prevent Android from returning a cached response to my HTTP Request?

I'm writing a client that is making repeated http requests for xml data that is changing over time. It looks like the Android stack is caching my page requests and returning the same page repeatedly. How do I make sure it gets a fresh page each time?
-- code ---
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(url);
HttpResponse response;
response = client.execute(request);
InputStream in;
in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
Thanks,
Gerry

Append an unused parameter on the end of the URL:
HttpGet request = new HttpGet(url + "?unused=" + someRandomString());
where someRandomString() probably involves the current time.
It's crude, but it's pretty much guaranteed to work regardless of all the outside factors that can make a "proper" solution fail, like misconfigured or buggy proxies.

add a HTTP header:
Cache-Control: no-cache
and see if that works.

Hint: to get the random string
HttpGet request = new HttpGet(url + "?unused=" + UUID.randomUUID().toString());