I'm using this code to parse a JSON array I'm getting from my server.
try {
URL u = new URL("http://54.68.139.250/get_user_likes");
HttpURLConnection conn = (HttpURLConnection) u.openConnection();
conn.setRequestMethod("GET");
conn.connect();
InputStream is = conn.getInputStream();
byte[] b = new byte[1024];
ByteArrayOutputStream baos = new ByteArrayOutputStream();
while ( is.read(b) != -1)
baos.write(b);
String JSONResp = new String(baos.toByteArray());
JSONArray arr = new JSONArray(JSONResp);
for (int i=0; i < arr.length(); i++) {
result.add(convertArticle(arr.getJSONObject(i)));
}
return result;
}
catch(Throwable t) {
t.printStackTrace();
}
return null;
This code works great on my phone. Unfortunately, when I'm using a Genymotion emulator with the virtual device of Google Nexus 7, the JSON array is slightly altered. 95% of the JSON array is fine, but it is truncated near the very end and is randomly missing about 4 characters of the json array at character 1253 so I'm getting:
org.json.JSONException: Expected ':' after top_id at character 1253 of [{"top_id":6,"top_url":
I'm thinking this is some memory problem with the emulator. Its base memory is 1024. Increasing that amount though doesn't change anything.
Any tips as to the reason behind the problem would be greatly appreciated. Also, feel free to comment on my code if you see room for improvement. :)
That's weird.
I can think of two things to try:
Check the encoding of the server and the encoding of the String constructor.
It's possible that the server is decoding with, say, Latin-1 (ISO-8859-1) and the String is encoding with UTF-8. But JSON is supposed to be sent in Unicode and the default charset for Android is UTF-8. Check the HTTP Content-type response header; it should say: application/json; charset=utf-8. If the charset part isn't there, do some investigation and find out what character set your server is using for decoding to HTTP stream. And check that the default charset on the phone and the emulator is the same; it should be UTF-8.
Try calling flush() on the ByteArrayOutputStream after you read the data and before you construct the String.
It may be that there is a slight difference between the phone OS and the emulator OS on how stream data is transferred/buffered/flushed.
If flush() doesn't work, try rewriting the code without using ByteArrayOutputStream. You could, for example, wrap the input stream with an InputStreamReader, which reads characters, not bytes, then append the characters using a StringBuilder or StringBuffer.
One way you could make the code better is to use JSONReader instead of JSONArray or JSONObject. The JSONReader would wrap an InputStreamReader which in turn wraps the HTTP input stream. It can be faster and more memory efficient since you don't have to read the entire input stream before starting to parse the data. When the server is sending a LOT of JSON data, that can make a big difference.
You should check the return value of is.read(). Change
while ( is.read(b) != -1)
baos.write(b);
to
int nread;
while ( (nread=is.read(b)) != -1)
baos.write(b, 0, nread);
Related
I'm struggling with an issue while sending Json data to the server. I guess there is some issue with the bad characters which are not expected at start of UTF-8 format.
I used CharDecoder to replace all the malformed utf-8 characters and here is the code.
// Construct the Decoder
CharsetDecoder utf8Decoder = Charset.forName("UTF-8").newDecoder();
utf8Decoder.onMalformedInput(CodingErrorAction.REPLACE);
utf8Decoder.onUnmappableCharacter(CodingErrorAction.REPLACE);
// Configure to replace Malformed input with space
utf8Decoder.replaceWith(" ");
// Construct ByteBuffer
ByteBuffer byteBuff = ByteBuffer.wrap(text.getBytes());
try {
// Process the text.
CharBuffer parsed = utf8Decoder.decode(byteBuff);
return new String(parsed.array());
} catch (CharacterCodingException e) {
e.printStackTrace();
}
This is not helping me. When I look at the column line of Json post data where parser is complaining, it is a space character.
Json to post is
{"body":{"messageSegments":[{"type":"Text","text":"This is a link "},{"type":"Mention","id":"005GGGGGG02g6MMIAZ"},{"type":"Text","text":" ish"}]},"capabilities":{"questionAndAnswers":{"questionTitle":"https:\/\/www.google.co.nz\/webhp?sourceid=chrome-instant&amp;amp;ion=1&amp;amp;espv=2&amp;amp;ie=UTF-16"}}}
Error is
[{"errorCode":"JSON_PARSER_ERROR","message":"Invalid UTF-8 start byte 0xa0 at [line:1, column:139]"}]
Any leads please.
Thanks,
Sree
0xa0 is a Unicode Character 'NO-BREAK SPACE' (U+00A0)
not the usual space 0x20
visually hard to notice the difference, but some frameworks do not like it.
Convert characters - the other characters and test it.
Helpful link http://www.ietf.org/rfc/rfc4627.txt
My android application uses a web service.The web service returns response in json format (which is UTF8 encoded). Here I am using the same for decoding the json data. still some special symbols(eg degree celcius symbol) are displays a question mark
InputStream is = con.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(is,"UTF-8"));
JSON:
{
"option1":"109.5?",
"option2":"109?",
"option3":"120?",
"option4":"180?",
"ans_option":"",
"qd_id":76,
"questions":"In alkanes the bond angle is"
}
You have to use "UTF-8" Mark for this issue:
http://developer.android.com/reference/java/nio/charset/Charset.html
You have to encode for your expected character like this way :
URLEncoder.encode("Your Special Character", "UTF8");
Check this question as well:
Android: Parsing special characters (ä,ö,ü) in JSON
Sorry for asking this question twice times, but I did research 2 hours...
I have problem with Base64 decode String in my Android.
For example:
encoded string: VgFzJ1+TrFa7WsXS5w==
The results in javascript and PHP: Vs'_¬V»ZÅÒç
The result in my Android: Vs'_��V�Z���
I found a solution from this page Android PHP Base64 decode with different results
They say
By this You convert the input String content to ISO-8859-1 encoded byte stream that will be decoded from base64.
This is my function decoder based on what they say:
byte[] b = Base64.decode(data.getBytes("ISO-8859-1"), Base64.DEFAULT);
return new String(b, "UTF-8");// event if remove UTF8
But still got the same error
Sorry, I have resolved by myself
It's very simple to get string from ISO-8859-1
return new String(b, "ISO-8859-1")
I am getting an OutOfMemoryException when processing a very long JSON response
returned by the code below:
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpGETRequest = new HttpGet(url);
HttpResponse response = httpclient.execute(httpGETRequest);
return EntityUtils.toString(response.getEntity());
Even though I use System.gc() before I call the http request, the exception still occurs.
I tried to look for the cause of the problem by saving the response on a notepad and check its size, just to find out that it is a 5mb file.
What are the things that should be done in order to avoid the exception?
Thanks for your help in advance.
In this cause, we should avoid large data responses from server. I requested a pagination functionality from the server and problem solved.
I don't recommend that you use System.gc() on Android. What you would need to do is use streaming and avoid loading the whole content in memory if it can be avoided.
https://google-gson.googlecode.com/svn/trunk/gson/docs/javadocs/com/google/gson/JsonStreamParser.html
If you need to access parts of the JSON data more readily rather than doing it one chunk at at time, you can transform the data as it is being read and store it in a SQL database that you can query later.
http://developer.android.com/training/basics/data-storage/databases.html
However, if you have embedded binary data such as an image as part of the JSON object you can't do anything about it anymore because it will read the embedded binary data into memory as one chunk. There's no easy way around this aside from writing your own JSON streamer.
Generally
This general technique can be applied not just to Android, but even application servers dealing with large input data such as XML as well.
Calling System.GC() is not guaranteed to run GC. In fact its not a good practice to programmatically invoke GC. For parsing large json data, use streaming APIs in whichever library you use to encode/decode json. One such popular library is Jackson processor.
5mb is not that big data to cause out of memory, maybe dump the memory of the app and analyze for memory leaks. You could sump using adb dumpsys
Tool. Find more about how to use that in What's the Android ADB shell "dumpsys" tool and what are its benefits?
try {
HttpResponse response = httpclient.execute(httppost);
HttpEntity entity = response.getEntity();
inputStream = entity.getContent();
// json is UTF-8 by default
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"), 8);
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
result = sb.toString();
inputStream.close();
} catch (Exception e) {
Log.d(TAG, "String was to large" + e.toString());
}
I think if you have problem to load 5Mb string into memory (totally reasonable size), streaming JSON parser will probably not help, because you need to have the result of parsing in memory, which will not necessarily be much smaller. I'd suggest you to do the memory profiling using Android Studio built-in memory monitor, to check how much free memory you have and how you're using it in general.
I sometimes get an OutOfMemoryError when posting a large file in Android. This is the code I'm using. Am I doing something wrong?
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setDoInput(true);
con.setDoOutput(true);
ostream = new DataOutputStream(con.getOutputStream());
byte[] buffer = new byte[1536];
int count;
while ((count = fileInput.read(buffer)) != -1) {
ostream.write(buffer, 0, count); // This sometimes causes OutOfMemoryError
}
ostream.flush();
Would calling ostream.flush() inside the while loop do any good?
If you do it like that, the whole POST must be buffered in memory. This is because it needs to send the Content-Length header first.
Instead, I think you want to use the Apache HTTP libraries, including FileEntity. That will let it figure out the length before reading the file. You can use this answer as a starting point. But the second parameter to the FileEntity constructor should be a mime type (like image/png, text/html, etc.).
HTTP connection is fine, but HTTPS will trigger Out of Memory error because there is a bug in HttpsURLConnectionImpl.java in Android 2.3.4 (verified on my tablet), and it's fixed in Android 4.1 (I have checked the source code).
By the way, he should avoid buffering the whole POST data in RAM by adding "con.setChunkedStreamingMode(0);" immediately after "con.setDoOutput(true);" statement.