Develop Reference

Android could not receive form submit response

I have a python webserver, which is listening for connections, and responds to them based on the request. The portion of code on python server of interest is POST request (http://192.168.0.1/conf_wifi_app.html). It takes 4 arguments
username (admin)
password (admin)
essid (Home wifi network SSID)
passphrase (Home wifi network password)
On python server, after the post body parameters are validated, a response is to be sent like so (notice I've put logs for debugging):
json_response = '{"error":false,' + '"code":"' + str(activation_code) + '","mac":"' + str(macaddress) + '","message":"Device configured successfully"}'
bytes_sent = client_s.send(json_response)
client_s.close()
print("Bytes sent " + str(bytes_sent))
print("json_response : " + json_response)
where client_s is the client socket. "send" function on socket should send the response (json_response), and then we close the socket. Logs print the number of bytes which is actually sent.
The client responds perfectly well when POST request is done from the web browser or from postman plugin. Just for some reference, I've put the raw request when invoked from postman plugin on chrome browser:
POST /conf_wifi_app.html HTTP/1.1
Host: 192.168.0.1
Connection: keep-alive
Content-Length: 67
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/68.0.3440.75 Chrome/68.0.3440.75 Safari/537.36
Cache-Control: no-cache
Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop
Postman-Token: 4f4a14a7-857d-666f-a2db-279731c83b4a
Content-Type: application/x-www-form-urlencoded
Accept: /
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
essid=NETGEAR-XXXX&passphrase=XXXXXXXX&username=admin&password=admin&submit=submit
The response is received when the POST request is made from web browser (or postman). Now I was trying to create an android app which does the same POST request as follows:
HttpURLConnection urlConnection = null;
try {
Map<String,Object> params = new LinkedHashMap<>();
params.put("user", Constants.DEVICE_DEFAULT_USER);
params.put("username", Constants.DEVICE_DEFAULT_USER);
params.put("password", Constants.DEVICE_DEFAULT_PASSWORD);
params.put("essid", homeWifiSSID.replaceAll("^\"|\"$", ""));
params.put("passphrase", homeWifiPassword);
StringBuilder urlParameters = new StringBuilder();
for (Map.Entry<String,Object> param : params.entrySet()) {
if (urlParameters.length() != 0) urlParameters.append('&');
urlParameters.append(URLEncoder.encode(param.getKey(), "UTF-8"));
urlParameters.append('=');
urlParameters.append(URLEncoder.encode(String.valueOf(param.getValue()), "UTF-8"));
}
byte[] postData = urlParameters.toString().getBytes("UTF-8");
int postDataLength = postData.length;
URL url = new URL(Constants.DEVICE_CONFIG_URL);
urlConnection = (HttpURLConnection) network.openConnection(url);
urlConnection.setDoInput( true );
urlConnection.setDoOutput( true );
urlConnection.setInstanceFollowRedirects( false );
urlConnection.setRequestMethod( "POST" );
urlConnection.setRequestProperty( "Content-Type", "application/x-www-form-urlencoded");
urlConnection.setRequestProperty( "charset", "utf-8");
urlConnection.setRequestProperty( "Content-Length", Integer.toString( postDataLength ));
urlConnection.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8");
urlConnection.setRequestProperty("Accept-Encoding", "gzip, deflate");
urlConnection.setUseCaches( false );
urlConnection.setConnectTimeout(OuroborosAPI.CONNECTION_TIMEOUT);
urlConnection.setReadTimeout(OuroborosAPI.CONNECTION_TIMEOUT);
try( DataOutputStream wr = new DataOutputStream( urlConnection.getOutputStream())) {
wr.write( postData );
wr.flush();
wr.close();
}
Reader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream(), "UTF-8"));
for (int c; (c = in.read()) >= 0;) {
System.out.print((char) c);
}
}
catch (Exception e) {
}
From android app, the post raw data received is as follows:
POST /conf_wifi.html HTTP/1.1
Content-Type: application/x-www-form-urlencoded
charset: utf-8
Content-Length: 85
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8
User-Agent: Dalvik/2.1.0 (Linux; U; Android 7.1.1; Moto G (5S) Plus Build/NPSS26.116-61-11)
Host: 192.168.0.1
Connection: Keep-Alive
Accept-Encoding: gzip
essid=NETGEAR-XXXX&passphrase=XXXXXXXX&username=admin&password=admin&submit=submit
The python socket in this case does send data (as confirmed from the logs), but the android errors out saying unexpected end of string.
I've literally tried every thing (like adding extra headers, etc) but to no avail. Please help or suggest where I may be going wrong.

The problem was not that I was not sending \n ended lines, or not using readline()
I was not sending the HTML raw headers like so
HTTP/1.1 200 OK
Content-Length: 9328
Content-Type: text/html
... Actual content/payload....
Once I sent the headers also, the code worked without any problems.

Pulling in JSON data gives response code of 200, but content length of -1 which breaks everything

I am building an android app that pulls in and displays data from a JSON api.
Specifically, the JSON api that I am pulling from is here
URL blogFeedUrl = new URL("http://www.johncorser.com/?json=1&count=15/");
HttpURLConnection connection = (HttpURLConnection)blogFeedUrl.openConnection();
connection.connect();
responseCode = connection.getResponseCode();
Log.i(TAG, "Code: " + responseCode);
Log.i(TAG, "Length " + connection.getContentLength());
if (responseCode == HttpURLConnection.HTTP_OK){
InputStream inputStream = connection.getInputStream();
Reader reader = new InputStreamReader(inputStream);
int contentLength = connection.getContentLength();
char[] charArray = new char[contentLength]; //Line 97, where the error is thrown because contentLength == -1
reader.read(charArray);
String responseData = new String(charArray);
jsonResponse = new JSONObject(responseData);
}
else {
Log.i(TAG, "Unsuccessful HTTP response code: " + responseCode);
}
When I run this, the logcat gives me this stacktrace:
10-15 11:36:00.512 8704-8718/com.johncorser.johncorser I/MainListActivity﹕ Code: 200
10-15 11:36:00.512 8704-8718/com.johncorser.johncorser I/MainListActivity﹕ Length -1
10-15 11:36:00.512 8704-8718/com.johncorser.johncorser E/MainListActivity﹕ Exception caught:
java.lang.NegativeArraySizeException: -1
at com.johncorser.johncorser.MainListActivity$GetBlogPostTask.doInBackground(MainListActivity.java:97)
at com.johncorser.johncorser.MainListActivity$GetBlogPostTask.doInBackground(MainListActivity.java:80)
at android.os.AsyncTask$2.call(AsyncTask.java:288)
at java.util.concurrent.FutureTask.run(FutureTask.java:237)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:231)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
at java.lang.Thread.run(Thread.java:841)
The strangest part is, if I swap the api with one structured in a nearly identical way, like the one here, the code executes without errors.
Any idea why I can't load this specific JSON data?

The issue is that service returns data in chunks (Transfer-Encoding:chunked in response headers), so it does not know content-length at the time it sends headers.
To handle such situation you can just wrap your InputStreamReader in BufferedReader and read it line by line:
BufferedReader br = new BufferedReader(new InputStreamReader(inputStream));
StringBuilder responseBuilder = new StringBuilder();
String chunk;
while ((chunk = br.readLine()) != null) {
responseBuilder.append(chunk);
}
String responseData = responseBuilder.toString();

See the api description: http://developer.android.com/reference/java/net/URLConnection.html#getContentLength()
"Returns the content length in bytes specified by the response header field content-length or -1 if this field is not set."
The http response is not sending content length header back

If the content length isn't known when the server sends the headers, then it doesn't tell the client the length; that means that the client doesn't know the length, and the call returns -1. Details are in the Javadoc for URLConnection.
What this means in practical terms is that you can't do it with a fixed length char array. You have to read the response line by line, into a StringBuilder, until you've exhausted the input. Then you can put the result into your JSONObject constructor.
Remember that some things on the web stream live content, so it's impossible for them to know the total length at the point when they start transmitting. That means you can't ever rely on being told the length; and because of that, some things are lazy and don't bother telling you the length even when they do know it.
Here are the headers your johncorser.com web site returns for that page:
HTTP/1.1 200 OK
Date: Wed, 15 Oct 2014 15:40:15 GMT
Server: Apache
Set-Cookie: PHPSESSID=82ivean17opsfkqjf9n4itdog2; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
X-Pingback: http://www.johncorser.com/xmlrpc.php
Content-Type: application/json; charset=UTF-8
Note there's nothing about length there. But the teamtreehouse.com site you also linked to, with a similar structure, returns this:
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 15 Oct 2014 15:46:59 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 4422
Connection: keep-alive
Keep-Alive: timeout=20
X-Pingback: http://blog.teamtreehouse.com/xmlrpc.php
X-Cacheable: SHORT
Vary: Accept-Encoding,Cookie
Cache-Control: max-age=600, must-revalidate
Accept-Ranges: bytes
X-Cache: HIT: 2
X-Cache-Group: normal
X-Type: default
Note that this one includes a Content-Length: 4422.

Correct way of setting up cache for OkHttp in Android

I'm trying to set up the cache for OkHttp, so it only requests to the server the first time I try to retrieve a response from the server until the expires header date, or cache-control header that comes from the server invalidates the response from the cache.
Currently, its caching the response, but not using it when requesting the resource again. May be this is not the way it was supposed to be used.
I'm setting up OkHttpClient with cache like this:
public static Cache createHttpClientCache(Context context) {
try {
File cacheDir = context.getDir("service_api_cache", Context.MODE_PRIVATE);
return new Cache(cacheDir, HTTP_CACHE_SIZE);
} catch (IOException e) {
Log.e(TAG, "Couldn't create http cache because of IO problem.", e);
return null;
}
}
This is used like this:
if(cache == null) {
cache = createHttpClientCache(context);
}
sClient.setCache(cache);
This is how I make one of the requests to the server with OkHttp that's actually failing to use the cache:
public static JSONObject getApi(Context context)
throws IOException, JSONException, InvalidCookie {
HttpCookie sessionCookie = getServerSession(context);
if(sessionCookie == null){
throw new InvalidCookie();
}
String cookieStr = sessionCookie.getName()+"="+sessionCookie.getValue();
Request request = new Request.Builder()
.url(sServiceRootUrl + "/api/"+API_VERSION)
.header("Accept", "application/json")
.header("Cookie", cookieStr)
.build();
Response response = sClient.newCall(request).execute();
if(response.code() == 200){
String charset = getResponseCharset(response);
if(charset == null){
charset = "utf-8";
}
String responseStr = new String(response.body().bytes(), charset);
response.body().close();
return new JSONObject(responseStr);
} else if(response.code() == 401){
throw new InvalidCookie();
} else {
return null;
}
}
If I get to the directory I specified to be the cache for OkHttp I can see the journal file and 4 other files that contains the responses for some of the requests. This request (the /api one I've just pasted the code) is stored on the cache directory so it was really cached, but the filename has a .tmp at the end, like if it wasn't properly saved to the final file, like other request I made.
This is how it looks like the headers of the server response for the request:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Expires: Sat, 09 Aug 2014 19:36:08 GMT
Cache-Control: max-age=86400, must-revalidate
Last-Modified: Sun, 04 Aug 2013 15:56:04 GMT
Content-Length: 281
Date: Fri, 08 Aug 2014 19:36:08 GMT
And this is how OkHttp stores it on the cache:
{HOST}/api/0.3
GET
0
HTTP/1.1 200 OK
9
Server: Apache-Coyote/1.1
Expires: Sat, 09 Aug 2014 19:36:08 GMT
Cache-Control: max-age=86400, must-revalidate
Last-Modified: Sun, 04 Aug 2013 15:56:04 GMT
Content-Length: 281
Date: Fri, 08 Aug 2014 19:36:08 GMT
OkHttp-Selected-Protocol: http/1.1
OkHttp-Sent-Millis: 1407526495630
OkHttp-Received-Millis: 1407526495721
After OkHttp creates this file, it keeps requesting to the server the same resource. I can see those messages in Wireshark.
What am I doing wrong?
UPDATE:
This is now the server response after Jesse suggestion:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Expires: Thu, 14 Aug 2014 18:06:05 GMT
Last-Modified: Sun, 10 Aug 2014 12:37:06 GMT
Content-Length: 281
Date: Wed, 13 Aug 2014 18:06:05 GMT
UPDATE 2:
Tried the code version and found out that it is quite probable there is a bug somewhere regarding the cache. This is what I've got from the Maven output:
Results :
Failed tests:
CacheTest.conditionalHitUpdatesCache:1653 expected:<[A]> but was:<[B]>
Tests in error:
CallTest.tearDown:86 » IO failed to delete file: C:\Users\Adrian\AppData\Local...
Tests run: 825, Failures: 1, Errors: 1, Skipped: 17
A more complete log can be seen here: https://gist.github.com/16BITBoy/344ea4c22b543f397f53

I just solved the problem. It was somewhat misleading that the cache tests where failing when I tried to use OkHttp from source.
The problem was quite easy and it was that other of the request methods was getting a body on the response, and it wasn't closed at the end. That explains why I saw the ".tmp" file in the cache, but still a confusing and misleading because of the fact that this request method was consuming and closing the body from the response. Its like the lock or monitor for the cache editor is global for all requests, instead of being by request. I though it wasn't when I read the code, when it used a hash for the request as a key.
Anyway, that was it :D
From now on, I'll try to stick to a pattern like this...
String respBody = null;
if(response.body() != null) {
respBody = response.body().string();
response.body().close();
}
...before handling each case for response code. That way I won't be missing a close call to the response body.

Your server is forcing cache validation with this response header:
Cache-Control: max-age=86400, must-revalidate
Remove that and you should be good to go.

I have the same problem, then I debug the okhttp source code, you can look at the CacheStrategy.cacheResponseAge(), okhttp will use the nowMillis and servedDate, the servedDate is get from you server http header "Date", the nowMillis is get from your android device. So when the server time is later than device time, okhttp will not get from cache if max-age is small.
forgive my poor english ^_^

Android DefaultHTTPConnection with Keep-Alive does not reuse TCP connection

I am trying to reuse HTTP connections with Keep Alive. The server supports Keep Alive, I send the Connection: Keep-Alive HTTP headers and still the connection is FIN (initiated by the Android client) right after the response is received. Some help would be appreciated.
Here some code:
DefaultHttpClient client = new DefaultHttpClient();
client.setKeepAliveStrategy(new DefaultConnectionKeepAliveStrategy());
client.setReuseStrategy(new DefaultConnectionReuseStrategy());
HttpParams params = client.getParams();
ClientConnectionManager mgr = client.getConnectionManager();
client = new DefaultHttpClient(new ThreadSafeClientConnManager(params, mgr.getSchemeRegistry()), params);
HttpPost post = new HttpPost(URL);
post.setHeader("Connection","Keep-Alive");
ByteArrayEntity entity = new ByteArrayEntity(requestBundle.postString().getBytes());
entity.setContentType("application/xml");
post.setEntity(entity);
HttpResponse response = client.execute(post);
When I check the HTTP request headers I see these:
Connection: Keep-Alive\r\n
Content-Length: 459\r\n
Content-Type: application/xml\r\n
Host: xxx.yyy.com\r\n
User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4)\r\n
In the HTTP response headers I see these:
HTTP/1.1 200 OK\r\n
Content-Type: application/xml\r\n
Content-Length: 8049\r\n
Date: Tue, 22 Jan 2013 03:14:40 GMT\r\n
Server: lighttpd/1.4.31\r\n
Looking at the responses and other TCP packets going over the line, it is clear that the Android client initiates a FIN based on some HTTP client settings that I am not aware of, which are not caused by the Keep-Alive settings on the server (the HTTP response does not contain header "Connection: close").
Please enlighten me!
Cheers,
Andrej

Android HttpPost failes - server receives null query string parameter

I have a simple WCF web service on my machine which I have developed to serve Android and IOS devices.
The service has a single method as following :
[OperationContract]
[WebInvoke(RequestFormat = WebMessageFormat.Json, ResponseFormat = WebMessageFormat.Json, BodyStyle = WebMessageBodyStyle.Wrapped, UriTemplate = "/?message={message}")]
string ServiceMessage(string message);
I have 3 clients , one .NET test client using HttpWebRequest which works fine , one IOS client which works fine and one Android client which I have developed with the HttpPost and HttpClient classes that fails.
Using Fiddler reverse proxy I have debugged the output of the .net client :
POST http://127.0.0.1:8888/Service1.svc/?message=_|JSON MESSAGE BODY|_HTTP/1.1
Content-Type: application/json; charset=utf-8
Host: 127.0.0.1:8888
Content-Length: 338
Expect: 100-continue
Connection: Keep-Alive
_|JSON MESSAGE BODY|_
On the other hand , this is the output of the Android HTTP Post :
POST http://10.0.2.2:8888/Service1.svc/ HTTP/1.1
Content-Type: application/json; charset=utf-8
Content-Length: 139
Host: 10.0.2.2:8888
Connection: Keep-Alive
User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4)
Expect: 100-Continue
message=_|JSON MESSAGE BODY|_
As you can see , .Net puts the message parameter on the Post at the top and does not
put the message variable name at the bottom while Android does not put the message body at the Post at the top and does put the message variable name at the bottom.
This is my Android post code ,
HttpPost httpPost = new HttpPost(url);
String messageBody = "message=" + jsonMessageParameter;
StringEntity entity = new StringEntity(messageBody);
httpPost.setEntity(entity);
httpPost.setHeader("Content-Type", "application/json; charset=utf-8");
HttpResponse response = hc.execute(httpPost);
InputStream content = response.getEntity().getContent();
String json = ConvertStreamToString(content);
When calling with this code , the server method is called but the message method parameter is null.
I tried playing with the Uri.Builder class to also make the android post put the message at the header , but doesnt quite work.
If can someone help me out here , I am stuck on this for hours over hours.
Thank you in advance ,
James
EDIT :
I changed the Android code to :
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(1);
nameValuePairs.add(new BasicNameValuePair("message", jsonMessageParameter));
HttpPost httpPost = new HttpPost(url);
UrlEncodedFormEntity urlEncodedFromEntity = new UrlEncodedFormEntity(
nameValuePairs);
urlEncodedFromEntity.setContentType(new BasicHeader("Content-Type",
"application/json; charset=utf-8"));
httpPost.setEntity(urlEncodedFromEntity);
InputStream postStream = httpPost.getEntity().getContent();
String postOutput = ConvertStreamToString(postStream);
HttpResponse response = hc.execute(httpPost);
InputStream content = response.getEntity().getContent();
String json = ConvertStreamToString(content);
But still the Fiddler monitoring is as following :
POST http://10.0.2.2:8888/Service1.svc/ HTTP/1.1
Content-Length: 189
Content-Type: application/json; charset=utf-8
Host: 10.0.2.2:8888
Connection: Keep-Alive
User-Agent: Apache-HttpClient/UNAVAILABLE (java 1.4)
Expect: 100-Continue
message=_|MESSAGE_JSON|_

There a number of points here.
First, you specifically added message= to your Android POST body:
String messageBody = "message=" + jsonMessageParameter;
This can be also problematic since you specified Content-Type: application/json but by adding message= you are not providing a valid JSON object.
Second, why does the .Net implementation replicate the JSON object both as parameter in the URL and in the body? This looks strange and very uncommon for a POST request, and can cause problem if your JSON object makes the URL exceed the maximum URL length.
So I would try removing message= in the body and removing the JSON object as a URL parameter, since servers processing POSTs should read the body and not the URL.

HttpPost is for posting, afaik.
String url;
input = url + URLEncoder.encode(messageBody, "utf-8");
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(input);
InputStream stream = httpclient.execute(httpget).getEntity().getContent();
String s = streamToString(stream);
JSONObject jObject = new JSONObject(s);
This very exact code (plus some try/catch) works for me for querying GoogleBooks, probably you can adapt it to your service with little effort?
Best regards.