Develop Reference

Android Http Response is incomplete. Returns an unterminated json object

I am using HttpClient 4.3.6 to perform http GET and POST requests. Right now I am using multipartentity to send a few string parameters and an image in the form of a file. I am able to successfully post the data but my problem comes in when I get the HTTP response. The response contains json data.
What happens is the HTTP response is incomplete and when i try to create a json object with the data i get jsonexception error saying:
Unterminated object at character 407.
I noticed that the response does not contain closed braces. Is this a problem on android or should I check the server? Because I am able to see the data properly on postman and on ios. I have never faced this issue before and don't know how to solve this.
This is my code to post and get the response:
#Override
protected String doInBackground(String... params) {
try {
String url = params[0];
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(url);
MultipartEntity entity = new MultipartEntity();
ByteArrayOutputStream baos = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.JPEG, 100, baos);
byte[] imageBytes = baos.toByteArray();
ByteArrayBody bab = new ByteArrayBody(imageBytes, "image.jpg");
entity.addPart("image_data", bab);
entity.addPart("action", new StringBody("1", "text/plain", Charset.forName("UTF-8")));
entity.addPart("name", new StringBody("asdfg", "text/plain", Charset.forName("UTF-8")));
entity.addPart("user_id", new StringBody("157", "text/plain", Charset.forName("UTF-8")));
entity.addPart("birthday", new StringBody("18-04-1995", "text/plain", Charset.forName("UTF-8")));
entity.addPart("gender", new StringBody("male", "text/plain", Charset.forName("UTF-8")));
entity.addPart("is_jlpt_student", new StringBody(String.valueOf(0), "text/plain", Charset.forName("UTF-8")));
entity.addPart("relationship", new StringBody("Father", "text/plain", Charset.forName("UTF-8")));
entity.addPart("relationship_id", new StringBody(String.valueOf(10002), "text/plain", Charset.forName("UTF-8")));
entity.addPart("is_creator", new StringBody(String.valueOf(1), "text/plain", Charset.forName("UTF-8")));
entity.addPart("email", new StringBody(email, "text/plain", Charset.forName("UTF-8")));
httppost.setEntity(entity);
HttpResponse resp = httpclient.execute(httppost);
String response = EntityUtils.toString(resp.getEntity());
Log.i("HttpResponse", response);
return response;
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
#Override
protected void onPostExecute (String result) {
super.onPostExecute(result);
JSONObject jsonObject = null;
try {
jsonObject = new JSONObject(result);
JSONObject json_data = jsonObject.getJSONObject("data");
String json_userid = json_data.getString("user_id");
String json_username = json_data.getString("name");
String json_email = json_data.getString("email");
String json_country = json_data.getString("country_code");
String json_imagefilename = json_data.getString("image_filename");
String json_imgurl = json_data.getString("image_url");
Toast.makeText(ParentGuardianProfile.this, "ImageFile " + json_imagefilename, Toast.LENGTH_SHORT).show();
User new_user = userdao.createUser(json_userid, json_username, json_email,json_imagefilename,json_country,selectedImageUri.toString(), 1);
Log.i("SQLITE", "added user : " + new_user.getmUserName() + new_user.getmId());
} catch (JSONException e) {
e.printStackTrace();
}
}
And my json response is :
{"status":1,"message":"success","data":{"child_id":"381","name":"asdfg","image_filename":"C201603021734476.jpg","image_url":"https:\/\/innokid.blob.core.windows.net\/media\/child\/381.jpg","birthday":"18-04-1995","gender":"male","is_jltp_student":"0","relationship":"Father","relationship_id":"10002","is_creator":1,"rank":1,"qrcode_url":"http:\/\/innokid.azurewebsites.net\/uploads\/qrcode\/child_381.png"
I tried using String buffer as suggested in this post String is being truncated when its too long . But i still get the same result.

Code looks ok at first glance.
How do you got know that the json data is cut? Logcat can truncate text. Debugger should be more reliable in this case.
Try to generate this same request with some tools like curl / SoapUI and validate JSON you got with some formatter / validator (you'll easily find a few of such tools).
It's beyond the range of question, but using raw Android built-in communication libraries seems to be a little bit masochistic. Have you ever consider to use Retrofit?

I think this code is problematic String response = EntityUtils.toString(resp.getEntity());
may be you should use some other function to convert response toString...
Apparently the json is missing two curly brackets '}}' at the end, which can happen due to some bug in the toString code.

I pulled up an old project that was using the org.apache.http stuff and below is how I was parsing the response. As you can see it is rather cumbersome. There are many tested and maintained libraries out there that are better suited to this kind of heavy-lifting.
// Get hold of the response entity (-> the data):
HttpEntity entity = response.getEntity();
if (entity != null) {
// Read the content stream
InputStream instream = entity.getContent();
Header contentEncoding = response.getFirstHeader("Content-Encoding");
if (contentEncoding != null && contentEncoding.getValue().equalsIgnoreCase("gzip")) {
instream = new GZIPInputStream(instream);
}
// Convert content stream to a String
resultString = convertStreamToString(instream);
instream.close();
// Do stuff with resultString here
// Consume Content
entity.consumeContent();
}
And the convertStreamToString() method:
private static String convertStreamToString(InputStream is) {
/*
* To convert the InputStream to String we use the
* BufferedReader.readLine() method. We iterate until the BufferedReader
* return null which means there's no more data to read. Each line will
* appended to a StringBuilder and returned as String.
*
* (c) public domain:
* http://senior.ceng.metu.edu.tr/2009/praeda/2009/01/
* 11/a-simple-restful-client-at-android/
*/
BufferedReader reader = new BufferedReader(new InputStreamReader(is), 8192);
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}

I finally solved this issue by replacing httpclient library with Android Asynchronous Http Client. Now it works fine. Thanks a lot for your help!
However, I still dont understand why the response was truncated when i used httpclient.

How to fix "400 bad request" with uploading file on android with java HttpClient?

I need to upload a file to server. If i use the "curl -i -F filedata=#"PATH TO FILE" http://█.199.166.14/audiostream " it return a 200 OK code (Or may be this command incorrect) .
But when I use java function
public String send()
{
try {
url = "http://█.199.166.14/audiostream";
File file = new File(Environment.getExternalStorageDirectory().getAbsolutePath(), "test.pcm");
try {
Log.d("transmission", "started");
HttpClient httpclient = new DefaultHttpClient();
HttpPost httppost = new HttpPost(url);
ResponseHandler Rh = new BasicResponseHandler();
InputStreamEntity reqEntity = new InputStreamEntity(new FileInputStream(file), -1);
reqEntity.setContentType("binary/octet-stream");
reqEntity.setChunked(true); // Send in multiple parts if needed
httppost.setEntity(reqEntity);
HttpResponse response = httpclient.execute(httppost);
response.getEntity().getContentLength();
StringBuilder sb = new StringBuilder();
try {
BufferedReader reader =
new BufferedReader(new InputStreamReader(response.getEntity().getContent()), 65728);
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line);
}
}
catch (IOException e) { e.printStackTrace(); }
catch (Exception e) { e.printStackTrace(); }
Log.d("Response", sb.toString());
Log.d("Response", "StatusLine : " + response.getStatusLine() + " Entity: " + response.getEntity()+ " Locate: " + response.getLocale() + " " + Rh);
return sb.toString();
} catch (Exception e) {
// show error
Log.d ("Error", e.toString());
return e.toString();
}
}
catch (Exception e)
{
Log.d ("Error", e.toString());
return e.toString();
}
}
It's return 400 Bad request.
I'm also not sure that server proceed correctly my attempts to upload this file, but I can't check it.

From the error received its likely a bad formatted HTTP query. If audiostream is a php, write the full link.
Also it seems that there might be a wrong/bad encoded char at "http://█.199.166.14/audiostream, the link should be http://(IP or DNS)/(rest of URL)(the URI)
You should erase the link, then manually writte it again.
If those didnt fix the issue, its also possible that the Server (or its path equipment) might be blocking you. Check from the Access Log and the security rules of its accesses, that you are not blocked (some routers may block users from performing repeated querys as a sort of anti "Denial of Service" measure)

Getting a JSON data and reading as a string is slow (compared to my browser)

I have read many questions on SO about like this, this, this and this.
My problem is the following: I have json data that reside on a server. Tha data are cached and are requested as gzip, so that the whole size is about 110kb. If you try to retrieve this data with firefox (and count the time with firebug) it takes about 3 seconds to download and display.
The very same data take about 10 - 15 seconds (or even more sometimes) to get downloaded and converted to string on my android app. (Note that I am using a Nexus 10 tablet which is one of the fastest in temrs of processing of its kind).
See my code below:
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(2);
// ... nameValuePairs omitted
HttpParams params = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(params, 3500);
HttpConnectionParams.setSoTimeout(params, 14000);
HttpConnectionParams.setTcpNoDelay(params, false);
params.setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
params.setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE, true);
HttpClient httpclient = new DefaultHttpClient(params);
HttpPost httppost = new HttpPost(mContext.getResources().getString(R.string.newsUrl));
httppost.addHeader("Accept-Encoding", "gzip");
try {
// Add your data
httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs, "UTF-8"));
Log.i(TAG, "NewsFetcher, Request started at: " + (int)(System.currentTimeMillis() / 1000));
// Execute HTTP Post Request
HttpResponse response = httpclient.execute(httppost);
Log.i(TAG, "NewsFetcher, Request ended at: " + (int)(System.currentTimeMillis() / 1000));
if (response.getStatusLine().getStatusCode() == 200 || response.getStatusLine().getStatusCode() == 204 || response.getStatusLine().getStatusCode() == 304 ) {
Header contentEncoding = response.getFirstHeader("Content-Encoding");
String webAppResponse;
if (contentEncoding != null && contentEncoding.getValue().equalsIgnoreCase("gzip")) {
Log.i(TAG, "NewsFetcher, InputStream(gzip) received at : " + (int)(System.currentTimeMillis() / 1000));
webAppResponse = IOUtils.toString(new GZIPInputStream(response.getEntity().getContent()), "UTF-8");
} else {
Log.i(TAG, "NewsFetcher, InputStream(plain) received at : " + (int)(System.currentTimeMillis() / 1000));
webAppResponse = IOUtils.toString(response.getEntity().getContent(), "UTF-8");
}
Log.i(TAG, "NewsFetcher, InputStream ended at : " + (int)(System.currentTimeMillis() / 1000));
// try to detect any error
try {
JSONArray webAppResultJSON = new JSONArray(webAppResponse);
ArrayList<NewsItem> parsedItems = parseJSON(webAppResultJSON);
return new FetchResult(UpdateStatus.FETCH_OK, parsedItems);
// ... exceptions omitted
} catch (Exception e) {
Log.e(TAG, "General Exception 1");
e.printStackTrace();
return new FetchResult(UpdateStatus.FETCH_GENERAL_ERROR, null);
}
} else {
return new FetchResult(UpdateStatus.FETCH_HTTP_ERROR, null);
}
// ... exceptions omitted
} catch (Exception e) {
Log.e(TAG, "General Exception 2");
e.printStackTrace();
return new FetchResult(UpdateStatus.FETCH_GENERAL_ERROR, null);
}
And here are the results from logcat
06-28 10:43:11.486: I/com.package.my.logic.NewsFetcher(3102): NewsFetcher, Request started at: 1372405391
06-28 10:43:17.986: I/com.package.my.logic.NewsFetcher(3102): NewsFetcher, Request ended at: 1372405397
06-28 10:43:17.986: I/com.package.my.logic.NewsFetcher(3102): NewsFetcher, InputStream(gzip) received at : 1372405397
06-28 10:43:30.266: I/com.package.my.logic.NewsFetcher(3102): NewsFetcher, InputStream ended at : 1372405410
If you subtract 1372405391 from 1372405410 it is 19 seconds! Can you suggest any optimization on the code above ?
I have also tried to use HttpURLConnection instead of HttpPost but no luck...

It seems that
webAppResponse = IOUtils.toString(new GZIPInputStream(response.getEntity().getContent()), "UTF-8");
takes most of the time (13 seconds).
I once wrote this method which works pretty well. maybe it will work better than IOUtils:
public String inputStreamToString (InputStream inputStream) throws IOException {
StringBuilder builder = new StringBuilder();
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null) {
builder.append(line);
}
return builder.toString();
}
Just pass your GZIPInputStream into it.
Hope it will help

HttpUrlConnection.getInputStream returns empty stream in Android

I make a GET request to a server using HttpUrlConnection.
After connecting:
I get response code: 200
I get response message: OK
I get input stream, no exception thrown but:
in a standalone program I get the body of the response, as expected:
{"name":"my name","birthday":"01/01/1970","id":"100002215110084"}
in a android activity, the stream is empty (available() == 0), and thus I can't get
any text out.
Any hint or trail to follow? Thanks.
EDIT: here it is the code
Please note: I use import java.net.HttpURLConnection; This is the standard
http Java library. I don't want to use any other external library. In fact
I did have problems in android using the library httpclient from apache (some of their anonymous .class can't be used by the apk compiler).
Well, the code:
URLConnection theConnection;
theConnection = new URL("www.example.com?query=value").openConnection();
theConnection.setRequestProperty("Accept-Charset", "UTF-8");
HttpURLConnection httpConn = (HttpURLConnection) theConnection;
int responseCode = httpConn.getResponseCode();
String responseMessage = httpConn.getResponseMessage();
InputStream is = null;
if (responseCode >= 400) {
is = httpConn.getErrorStream();
} else {
is = httpConn.getInputStream();
}
String resp = responseCode + "\n" + responseMessage + "\n>" + Util.streamToString(is) + "<\n";
return resp;
I see:
200
OK
the body of the response
but only
200
OK
in android

Trying the code of Tomislav I've got the answer.
My function streamToString() used .available() to sense if there is any data received,
and it returns 0 in Android. Surely, I called it too soon.
If I rather use readLine():
class Util {
public static String streamToString(InputStream is) throws IOException {
StringBuilder sb = new StringBuilder();
BufferedReader rd = new BufferedReader(new InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
sb.append(line);
}
return sb.toString();
}
}
then, it waits for the data to arrive.
Thanks.

You can try with this code that will return response in String:
public String ReadHttpResponse(String url){
StringBuilder sb= new StringBuilder();
HttpClient client= new DefaultHttpClient();
HttpGet httpget = new HttpGet(url);
try {
HttpResponse response = client.execute(httpget);
StatusLine sl = response.getStatusLine();
int sc = sl.getStatusCode();
if (sc==200)
{
HttpEntity ent = response.getEntity();
InputStream inpst = ent.getContent();
BufferedReader rd= new BufferedReader(new InputStreamReader(inpst));
String line;
while ((line=rd.readLine())!=null)
{
sb.append(line);
}
}
else
{
Log.e("log_tag","I didn't get the response!");
}
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return sb.toString();
}

The Stream data may not be ready, so you should check in a loop that the data in the stream is available before attempting to access it.
Once the data is ready, you should read it and store in another place like a byte array; a binary stream object is a nice choice to read data as a byte array. The reason that a byte array is a better choice is because the data may be binary data like an image file, etc.
InputStream is = httpConnection.getInputStream();
byte[] bytes = null;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] temp = new byte[is.available()];
while (is.read(temp, 0, temp.length) != -1) {
baos.write(temp);
temp = new byte[is.available()];
}
bytes = baos.toByteArray();
In the above code, bytes is the response as byte array. You can convert it to string if it is text data, for example data as utf-8 encoded text:
String text = new String(bytes, Charset.forName("utf-8"));

HttpGet - weird encoding/characters

I have a following class
public class MyHttpClient {
private static HttpClient httpClient = null;
public static HttpClient getHttpClient() {
if (httpClient == null)
httpClient = new DefaultHttpClient();
return httpClient;
}
public static String HttpGetRequest(String url) throws IOException {
HttpGet request = new HttpGet(url);
HttpResponse response = null;
InputStream stream = null;
String result = "";
try {
response = getHttpClient().execute(request);
if (response.getStatusLine().getStatusCode() != 200)
response = null;
else
stream = response.getEntity().getContent();
String line = "";
StringBuilder total = new StringBuilder();
BufferedReader rd = new BufferedReader(new InputStreamReader(stream));
while ((line = rd.readLine()) != null) {
total.append(line);
}
// Return full string
result = total.toString();
} catch (ClientProtocolException e) {
response = null;
stream = null;
result = null;
} catch (IllegalStateException e) {
response = null;
stream = null;
result = null;
}
return result;
}
}
and a web-service which response header is (I can't provide direct link because of privacy)
Status: HTTP/1.1 200
OK Cache-Control: private
Content-Type: application/json;
charset=utf-8
Content-Encoding: gzip
Server: Microsoft-IIS/7.5
X-AspNetMvc-Version: 3.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sun, 03
Jul 2011 11:00:43 GMT
Connection: close
Content-Length: 8134
In the end I get as a result series of weird, unreadable characters (I should get regular JSON like I get in regular desktop browser).
Where is the problem? (The same code for ex. google.com works perfectly and I get nice result)
EDIT: Solution (see below for description)
Replace
HttpGet request = new HttpGet(url);
with
HttpUriRequest request = new HttpGet(url);
request.addHeader("Accept-Encoding", "gzip");
and replace
stream = response.getEntity().getContent();
with
stream = response.getEntity().getContent();
Header contentEncoding = response.getFirstHeader("Content-Encoding");
if (contentEncoding != null && contentEncoding.getValue().equalsIgnoreCase("gzip")) {
stream = new GZIPInputStream(stream);
}

The problem is here:
Content-Encoding: gzip
This means the weird characters you are getting are the gzip compressed version of the expected JSON. Your browser does the decmpression so there you see the decoded result. You should have a look at your request header and at the configuration of your server.

Well, gzip encoding is generally good practice - for JSON data (especially big one) it can actually get between 10x and 20x decrease in the amount of data to transfer (which is a GOOD THING).. So better is to let HttpClient to handle GZIP compression nicely. For example here:
http://forrst.com/posts/Enabling_GZip_compression_with_HttpClient-u0X
BTW. It seems wrong however on the server side to provide GZIP compressed data when the client does not say "Accept-Encoding: gzip", which seems to be the case... So some things have to be corrected on the server as well. The example above adds Accept-Encoding header for you.