I'm creating an android application which should parse a Json from a file or url to a jsonarray and jsonobjects.
The problem is that my json is 3.3 mb and when i use a simple code, something like this: (can't acces my real code now because im at work, copied some code from tutorial; so there might be some errors in it)
(assuming i already have my inputstream content)
InputStream content = entity.getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(content));
String line;
while ((line = reader.readLine()) != null) {
builder.append(line);
String twitterfeed = builder.toString();
}
JSONArray jsonArray = new JSONArray(twittefeed);
Log.i(ParseJSON.class.getName(),
"Number of entries " + jsonArray.length());
for (int i = 0; i < jsonArray.length(); i++) {
JSONObject jsonObject = jsonArray.getJSONObject(i);
Log.i(ParseJSON.class.getName(), jsonObject.getString("text"));
When i run this code on my android device, i get an OutOfMemory error when parsing the string to the jsonArray.
I logged some things i found that my total string is 17 mb (of a 3.3 mb json file?!) When i use a small json file, like a twitterfeed or so, the code works fine.
When i got this 17 mb string in my memory i can't parse the json, because then i run out of memory.
After a lot of research i found that jackson might be my solution, because i understood that it is possible to parse an inputstream. This should help, because than i don't need the 17 mb string in my memory; and this is not the most efficient way i gues.... But i can't get it clear of this really will work, and didn't get it running myself.
Does anyone know of this is really will work, and where i can find a tutorial?
I found the "createJsonParser -- public JsonParser createJsonParser(InputStream in)" and think this is my way to go... but i don't know how to implement this in my code, and can't find an example. Does anyone know how this works?
You should use json streaming either with gson or jackson. With Jackson you can use a hybrid approach as well. This would reduce your memory consumption significantly as only the portion of json being parsed is loaded into memory.
https://sites.google.com/site/gson/gson-user-guide
http://jackson.codehaus.org/
A jackson example http://www.mkyong.com/java/jackson-streaming-api-to-read-and-write-json/
The hybrid streaming approach with Jackson is a good option. That way, you can advance a pointer through your raw JSON input (validating it as you go) and, when you detect an input chunk that needs to be processed, you can read it into a tree-hierarchy so that you can pull out any data you want from it.
Related
I have tried several of the examples listed and none of them seem to work for some reason so I am posting a new question.
I have a JSON string coming to me however it is all objects (it is not formatted in an array). I need to parse out the Team names from this. I know I need to iterate over them as they are the keys and then store the objects in a new array.
Here is the JSON (a sample of it)
{"query":{"printrequests":[{"label":"","typeid":"_wpg","mode":2,"format":false}],"results":{"Team:Kubbchucks":{"printouts":[],"fulltext":"Team:Kubbchucks","fullurl":"http://wiki.planetkubb.com/wiki/Team:Kubbchucks","namespace":822,"exists":true},"Team:Kubbchucks
IDP":{"printouts":[],"fulltext":"Team:Kubbchucks
IDP","fullurl":"http://wiki.planetkubb.com/wiki/Team:Kubbchucks_IDP","namespace":822,"exists":true}},"serializer":"SMW\Serializers\QueryResultSerializer","version":0.5,"meta":{"hash":"8407a177d701d746edc3066a012c17d2","count":2,"offset":0}}}
What I have so far to dig down to that level of the JSON (data is the string above)
JSONObject parsing = new JSONObject(data);
JSONObject query = parsing.getJSONObject("query");
JSONObject results = query.getJSONObject("results");
This allows me to manually check for a team using :
JSONObject teamone = results.getJSONObject("Team:Kubbchucks");
This is of course not ideal. The full JSON file has almost 3000 entries and I do not know what those entries all are. I need to iterate over the results object to get the keys that start with "Team:"
I've tried a few different iterator samples but none seem to be working for me and it is starting to really frustrate me. Believe me, I've tried many things before asking the question. Any help is appreciated.
I had a similar problem some time ago.
You can do like this:
JSONObject results = query.getJSONObject("results");
Iterator<String> iterator = results.keys();
while (iterator.hasNext()) {
JSONObject team = results.getJSONObject(iterator.next());
// do stuff
}
Hi I am parsing data from JSON. While parsing data I am getting OUT OF MEMORY error and dalvik-heap. How to over come from this problem. and I search in net i didn't found any solution please help me.
Thanks in advance
03-06 08:19:00.618: E/dalvikvm-heap(1415): Out of memory on a 2506782-byte allocation.
JSONObject jsonObj1 = new JSONObject(jsonStr1);
ArrayList<String> matter1= new ArrayList<String>();
String stat = jsonObj1.getString(TAG_stat);
String suc=jsonObj1.getString(TAG_success);
// Getting JSON Array node
matter = jsonObj1.getJSONArray(TAG_mdata);
// looping through All Contacts
dbdata =new ArrayList<String>();
for (int i = 0; i < matter.length(); i++) {
JSONObject j = matter.getJSONObject(i);
String ddesc= j.getString(TAG_ddesc);
String spd= j.getString(TAG_sped);
String gzid=j.getString(TAG_geozid);
String dev = j.getString(TAG_devid);
matter1.add(stat);
matter1.add(suc);
Log.v("contact", ""+matter1);
dbdata.add(ddesc);
dbdata.add(spd);
dbdata.add(gzid);
dbdata.add(dev);
dbdata1.add(dbdata);
One strange thing about this is that the crash only occurs every 2nd or 3rd time the app is run, leaving me to believe that the memory consumed by the app is not being garbage collected each time the app closes.
That is certainly possible, and if it is the case then it probably due to a memory leak that can be traced back to something that your application is doing. I think you should focus your initial efforts into investigating this aspect ... rather than loading the file in chunks. (I am not familiar with the Android tool-chain, but I am sure it includes memory usage profilers or memory dump analysers.)
EDIT
In response to your followup comment, the fact that it works 2 times in 3 suggests that your app ought to work roughly as-is. Admittedly, you don't have much leeway if the input file gets bigger.
A couple of ideas though:
Rather than reading the file into a String and running the JSON parser on the String, use a parser that can read directly from a stream. Your current solution needs space for two complete copies of the data in memory while you are doing the parsing.
If the file gets much bigger, you may need to think of a design that doesn't create a complete in-memory representation of the data.
I'm not sure that it is a good idea to read a JSON file in "chunks". This could present problems for parsing the JSON ... depending on exactly what you mean by reading in chunks.
Use Gson library. Will optimize all your process.
I know how to get jsonarray from jsonobject. I am doing like below code to get jsonarray.
JSONObject recvJson = new JSONObject(holder.toString());
String numberByte= (String) recvJson.get("data");
String ts = (String) recvJson.get("time");
JSONObject temp2 = new JSONObject("{ \"data\" : " + numberByte+ "}");
JSONArray recvJarray = temp2.getJSONArray("data");
for (int i = 0; i < recvJarray.length(); i++)
{
byteArray[i] = (byte) recvJarray.getInt(i);
}
But don't want to use for-loop, without using for-loop or any other loop want to retrieve jsonarray data values.
How should I do ?? I have done google & saw many forums but dint succeed to retrieve data without using for-loop.
for example : I ll be getting 80 - 100 packets of 1024bytes per second from server, I want to retrieve this data & store it into bytearray. By usingf for-loop its taking around 300ms- 400ms and I am loosing many packets between that. So I want to use different approach. If any Idea or solution for cracking this.
Help will be appreciated !!
You can use GSON to parse the json objects. It is much faster and easier to decode json data.
Check these links.
Gson1, Gson2
try usig gson
it uses java reflection to convert objects to json and json to object with simple methods
toJson(),fromJson() (and it works for object arrays and lists too)
but you have to write the proper classes for the jsons (with all fields)
read the documentation.
I have a question that I am a little bit confused about. I am quite new to JSON and getting JSON values in the android API. I am trying to access an array within the response I get. the JSON code I am getting is something like this:
Response:
{
"event": {
"participants": []
},
"status": "success"
}
How would I access the participants array and store their values. This is what I am trying at the moment... but I dont appear to be getting what I want.
try{
//get the JSON values from the URL.
JSONObject json = jParser.getJSONFromUrl("http://somesite.com/api/find?"+"somevar="+someJavaStringVar);
json_event = json.getJSONObject("event");
JSONArray json_array_participants = json_event.getJSONArray("participants");
} catch(JSONException e) {
}
The thing I am mostly confused about is... what is the arrays type equivalent to. Any advice or reasoning as to the correct way to get ahold of that variables value would be great... thanks guys.. :).
Think JSON is really just a key-value pairing. The JSONArray type is just an array full of objects (like Object[]) - it has no idea what the objects it contains are or what they're to be used for. Its up to you to assign meaning to the JSON stream based on what you know of the source. From what I see of your code, most of it looks fine, though I don't know what your jParser.getJSONFromURL() is doing. Typically, you would build the JSON from the response string like so:
String jsonString = getJSONFromUrl("http://somesite.com/api/find?"+"somevar="+someJavaStringVar);
JSONObject json = new JSONObject(jsonString)
JSONObject json_event = json.getJSONObject("event");
JSONArray json_array_participants = json_event.getJSONArray("participants");
You can iterate through the array like any other array to get subobjects or whatever:
for(int i=0; i < json_array_participants.getLength(); i++) {
JSONObject participant = json_array_participants.getJSONObject(i);
// Do stuff
}
As a side note - I WOULDN'T use GSON until you understand the underlying protocol, at least a little - because you never know when you might want to parse your JSON from a different language for some reason.
I would strongly recommend to use gson instead as your preferred parser since it will do all the job of serializing and deserializing for you except creating the domain objects.
This tutorial should get you going:
http://www.javacodegeeks.com/2011/01/android-json-parsing-gson-tutorial.html
This will depend on what the server is supposed to return. It could be an array of anything and if this is a public service, there should be a specification to go off of.
If you are in charge of the server portion as well, and you have a backing object, Google's GSON library is extremely easy to use. It will also keep type information straight.
I'm using JSON Framework to update the data of client side from server. I have 30 Pojo Classes and i received the Http Response from server. I create the Object of Reader by using the method
InputStream instream = entity.getContent();
reader = new InputStreamReader(instream, "UNICODE");
And i pass it in the Json Object like this
synchronizationResponse = gson.fromJson(reader, SynchronizationResponse.class);
But this line is giving outOfMemory Exception.
I write the response in a file and found the size of file is around 12MB.
So is their any way to split the response in multiple response.So that i can read and write simultaneously to avoid OOM exception.
Looking for Help
Difficult to say, as long as we do not know, what your SynchronisationResponse Class looks like.
Whatever your content, if you can split it into sub-sections, do so and serialize those.
You can use a JsonReader to parse a large stream of JSON without running out of memory. The details are in the documentation, but basically you create a JsonReader:
JsonReader reader = new JsonReader(new InputStreamReader(instream, "UNICODE"));
and write a recursive descent parser using reader.nextName() to get keys, reader.nextString(), reader.nextInt(), etc. to get values, and reader.beginObject(), reader.endObject(), reader.beginArray(), reader.endArray() to mark object boundaries.
This allows you to read and process one piece at a time, at any granularity, without ever holding the entire structure in memory. If the data is long-lived, you can store it in a database or a file.