In my application I am downloading very large, complex, JSON files (over 100MB) from the server. The structure of these files can differ and I don't always know the key names. Because of this I cannot create a custom object to hold the data. The one thing that I do always know is that the file contains an array of objects. What I need to do is convert each object into a JsonObject and add it to a Kotlin List to be used in a RecyclerView and other places throughout the app.
What I do currently is download the JSON as a Reader object using OKHttp like this:
val jsonStream = response.body!!.charStream()
From there I use Gson's JsonReader to iterate through the file and create my JSON objects.
val array = mutableListOf<JsonObject>()
JsonReader(jsonStream).use {reader ->
reader.beginArray()
while (reader.hasNext()) {
val json = JsonParser.parseReader(reader).asJsonObject
array.add(json)
}
}
Here is an example of what an object looks like:
{
"employee" : [
{
"person_id" : 1441815,
"id" : 1441815,
"first" : "Steve",
"last" : "Eastin",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 1470429,
"id" : 1470429,
"first" : "Kerry",
"last" : "Remsen",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 1471551,
"id" : 1471551,
"first" : "Clu",
"last" : "Gulager",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 1604199,
"id" : 1604199,
"first" : "Brian",
"last" : "Wimmer",
"movie_custom_id" : 3916884,
"middle" : "",
"job" : "actor"
},
{
"person_id" : 1632559,
"id" : 1632559,
"first" : "Lyman",
"last" : "Ward",
"movie_custom_id" : 3916884,
"middle" : "",
"job" : "actor"
},
{
"person_id" : 1788526,
"id" : 1788526,
"first" : "Christie",
"last" : "Clark",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 1869213,
"id" : 1869213,
"first" : "Sydney",
"last" : "Walsh",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 1892343,
"id" : 1892343,
"first" : "Robert",
"last" : "Rusler",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 1961713,
"id" : 1961713,
"first" : "Jack",
"last" : "Sholder",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 2476997,
"id" : 2476997,
"first" : "Tom",
"last" : "McFadden",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 3401109,
"id" : 3401109,
"first" : "Allison",
"last" : "Barron",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 8201549,
"id" : 8201549,
"first" : "JoAnn",
"last" : "Willette",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 27936448,
"id" : 27936448,
"first" : "Melinda",
"last" : "Fee",
"custom_id" : 3916884,
"middle" : "O."
},
{
"person_id" : 40371176,
"id" : 40371176,
"first" : "Steven",
"last" : "Smith",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 45323542,
"id" : 45323542,
"first" : "Kimberly",
"last" : "Lynn",
"custom_id" : 3916884,
"middle" : ""
},
{
"person_id" : 45323546,
"id" : 45323546,
"first" : "Jonathan",
"last" : "Hart",
"custom_id" : 3916884,
"middle" : ""
}
],
"id" : 3916884,
"array1" : [
"3",
"4",
"5"
],
"date_added" : "2020-10-10 15:26:09",
"number1" : 1985,
"verified" : 0,
"number2" : 14446757,
"string1" : "test string 1",
"null1" : null,
"string2" : "test string 2",
"null2" : null,
"number3" : 0,
"array2" : [
"1",
"2",
"3"
],
"string3" : "test string 3",
"string4" : "test string 4",
"null3" : null,
"null4" : null,
"number4" : 1
}
My JSON files can contain 10,000+ objects. The issue I am having is that I'm running out of memory. Through a lot of testing I've determined that it is because of the nested array of employee objects. Is there a way to parse this file more efficiently, to prevent running out of memory, or am I going to have to come up with a different solution to handle this amount of data?
Related
I have very big problem. I have this structure in my Firebase:
I would like to save this structure in JSON file/String no matter. I would get such a result:
{
"-Ka8_NYNxaFCcZQ3eaTp" : {
"Persons" : {
"-KaHUTUan-l06OVjsUlu" : {
"birthDay" : "13.01.2017",
"deathDay" : "06.01.2017",
"lastName" : "LastName1",
"latitude" : "54.2039383",
"longitude" : "16.1930868",
"name" : "name1"
},
"-KaHU_nPOE2NlqTbhLxT" : {
"birthDay" : "11.01.2017",
"deathDay" : "03.01.2017",
"lastName" : "LastName2",
"latitude" : "54.2039383",
"longitude" : "16.1930868",
"name" : "name2"
}
},
"city" : "Wroclaw",
"county" : "county1",
"nameCemetery" : "nameCemetery1",
"province" : "province1",
"street" : "street1"
}
I tried through DataSnaphot but the result was not in the format JSON, e.g:
{-Ka8_NYNxaFCcZQ3eaTp={nameCemetery=nameCemetery1, street=street1, county=county1, province=province1, Persons={-KaHU_nPOE2NlqTbhLxT={name=Damian, latitude=54.2039383, lastName=Szeeczyk, birthDay=11.01.2017, longitude=16.1930868, deathDay=03.01.2017}, -KaHUTUan-l06OVjsUlu={name=name1, latitude=54.2039383, lastName=LastName1, birthDay=13.01.2017, longitude=16.1930868, deathDay=06.01.2017}}, city=Wroclaw}}
The most important for me is not to create JSONArray Persons, because then my application will run incorrectly.
In short, I would like to make a copy node (in my example key -"Ka8_NYNxaFCcZQ3eaTp"
Having the following data:
"animals" : {
"a1" : {
"name" : "Dog"
},
"a2" : {
"name" : "Cat"
},
"a3" : {
"name" : "Cow"
}
},
"users" : {
"u1" : {
"favAnimals" : {
"a1" : true,
"a2" : true
},
"name" : "Danilo"
}
}
How can I display for the user, using the FirebaseListAdapter, the complete list of animals, highlighing user's favorite animals?
Is it possible to query and join them, to get a Database Reference that would watch for changes both in "/animals" and "/users/$uid/favAnimals" and return something like:
"animals" : {
"a1" : {
"name" : "Dog",
"fav" : true
},
"a2" : {
"name" : "Cat"
"fav" : true
},
"a3" : {
"name" : "Cow"
}
}
I am using the original retrofit to call the Instagram api. The call returns this reponse
{ "data" : [ { "attribution" : null,
"caption" : { "created_time" : "1460953831",
"from" : { "full_name" : "Edward ♕",
"id" : "282525924",
"profile_picture" : "https://scontent.cdninstagram.com/t51.2885-19/s150x150/12519400_165282517199271_1363863979_a.jpg",
"username" : "manlike_eddy"
},
"id" : "17847512245075242",
"text" : "Oh girl"
},
"comments" : { "count" : 1,
"data" : [ { "created_time" : "1460953885",
"from" : { "full_name" : "Edward ♕",
"id" : "282525924",
"profile_picture" : "https://scontent.cdninstagram.com/t51.2885-19/s150x150/12519400_165282517199271_1363863979_a.jpg",
"username" : "manlike_eddy"
},
"id" : "17847512257075242",
"text" : "#love #TagsForLikes #TagsForLikesApp #instagood #me #smile #follow #cute #photooftheday #tbt #followme #girl #beautiful #happy #picoftheday #instadaily #food #swag #amazing #TFLers #fashion #igers #fun #summer #instalike #bestoftheday #smile #like4like #friends #instamood"
} ]
},
"created_time" : "1460953831",
"filter" : "Normal",
"id" : "1230892414266144399_282525924",
"images" : { "low_resolution" : { "height" : 320,
"url" : "https://scontent.cdninstagram.com/t51.2885-15/s320x320/e35/12965764_932183320233898_1928407701_n.jpg?ig_cache_key=MTIzMDg5MjQxNDI2NjE0NDM5OQ%3D%3D.2.l",
"width" : 320
},
"standard_resolution" : { "height" : 640,
"url" : "https://scontent.cdninstagram.com/t51.2885-15/s640x640/sh0.08/e35/12965764_932183320233898_1928407701_n.jpg?ig_cache_key=MTIzMDg5MjQxNDI2NjE0NDM5OQ%3D%3D.2.l",
"width" : 640
},
"thumbnail" : { "height" : 150,
"url" : "https://scontent.cdninstagram.com/t51.2885-15/s150x150/e35/c257.0.565.565/12950340_1566398773652655_1170974002_n.jpg?ig_cache_key=MTIzMDg5MjQxNDI2NjE0NDM5OQ%3D%3D.2.c",
"width" : 150
}
},
"likes" : { "count" : 0,
"data" : [ ]
},
"link" : "https://www.instagram.com/p/BEVAiEVDnaP/",
"location" : null,
"tags" : [ "beautiful",
"cute",
"fashion",
"love",
"summer",
"food",
"instalike",
"tbt",
"tagsforlikesapp",
"igers",
"follow",
"instadaily",
"instamood",
"friends",
"girl",
"me",
"swag",
"like4like",
"tflers",
"followme",
"instagood",
"tagsforlikes",
"amazing",
"bestoftheday",
"fun",
"smile",
"photooftheday",
"picoftheday",
"happy"
],
"type" : "image",
"user" : { "full_name" : "Edward ♕",
"id" : "282525924",
"profile_picture" : "https://scontent.cdninstagram.com/t51.2885-19/s150x150/12519400_165282517199271_1363863979_a.jpg",
"username" : "manlike_eddy"
},
"user_has_liked" : false,
"users_in_photo" : [ ]
} ],
}
}
As you can see some of the labels are using snakecase, such as standard_resolution. When I plug this data into Json Pogo it creates two classes for standard_resolution. One named StandardResolution and one StandardResolution. When I call .getStandardResolution() on the respective object null is returned, even though it should have valid data.
This question already has answers here:
Sending and Parsing JSON Objects in Android [closed]
(11 answers)
Closed 7 years ago.
I have a json+collection string as following:
//code formated
{ "collection" :
{
"version" : "1.0",
"href" : "http://example.org/friends/",
"links" : [
{"rel" : "feed", "href" : "http://example.org/friends/rss"}
],
"items" : [
{
"href" : "http://example.org/friends/jdoe",
"data" : [
{"name" : "full-name", "value" : "J. Doe", "prompt" : "Full Name"},
{"name" : "email", "value" : "jdoe#example.org", "prompt" : "Email"}
],
"links" : [
{"rel" : "blog", "href" : "http://examples.org/blogs/jdoe", "prompt" : "Blog"},
{"rel" : "avatar", "href" : "http://examples.org/images/jdoe", "prompt" : "Avatar", "render" : "image"}
]
},
{
"href" : "http://example.org/friends/msmith",
"data" : [
{"name" : "full-name", "value" : "M. Smith", "prompt" : "Full Name"},
{"name" : "email", "value" : "msmith#example.org", "prompt" : "Email"}
],
"links" : [
{"rel" : "blog", "href" : "http://examples.org/blogs/msmith", "prompt" : "Blog"},
{"rel" : "avatar", "href" : "http://examples.org/images/msmith", "prompt" : "Avatar", "render" : "image"}
]
},
{
"href" : "http://example.org/friends/rwilliams",
"data" : [
{"name" : "full-name", "value" : "R. Williams", "prompt" : "Full Name"},
{"name" : "email", "value" : "rwilliams#example.org", "prompt" : "Email"}
],
"links" : [
{"rel" : "blog", "href" : "http://examples.org/blogs/rwilliams", "prompt" : "Blog"},
{"rel" : "avatar", "href" : "http://examples.org/images/rwilliams", "prompt" : "Avatar", "render" : "image"}
]
}
],
"queries" : [
{"rel" : "search", "href" : "http://example.org/friends/search", "prompt" : "Search",
"data" : [
{"name" : "search", "value" : ""}
]
}
],
"template" : {
"data" : [
{"name" : "full-name", "value" : "", "prompt" : "Full Name"},
{"name" : "email", "value" : "", "prompt" : "Email"},
{"name" : "blog", "value" : "", "prompt" : "Blog"},
{"name" : "avatar", "value" : "", "prompt" : "Avatar"}
]
}
}
}
sample linK: http://amundsen.com/media-types/collection/examples/
Can you please help me on how to decode this message. I have to read items , queries into my code and also version etc. I have to do this on android. I am not able to get how to use JSONObject to fetch the items, queries into my code.
For the array objects use JSONArray. Method arr.length() returns elements count, arr.get<type>(position) or just arr.get(position) for getting element.
you may use:
import org.json.JSONException;
import org.json.JSONObject;
in the following way:
try {
JSONObject myObj = new JSONObject("your json string");
} catch (JSONException e) {
e.printStackTrace();
}
I'm a beginner in android development ... i want to know how to create a json structure, with using arraylists and hashmaps ... you can find the structure as follows ...
NOTE: I don't want to use JSONArray or JSONObject
[
[
{
"blog_owner" : "user",
"icalIssueEventID" : "",
"updated_date" : "2014-02-17 07:05:53 +0000",
"icalEventID" : "",
"blog_stage" : "Cancelled",
"blog_ID" : "GMpk20140206160708281",
"created_date" : "2014-02-06 10:37:38 +0000",
"emailFlag" : false
}
],
[
{
"blog_iNotes_ID" : "",
"blog_name" : "",
"blog_status" : "",
"blog_duration" : ""
},
{
"pid" : "0",
"isOpen" : true,
"id" : "R1",
"level" : 0,
"value" : [
{
"level" : 0,
"id" : "MA20140206181038936",
"type" : "meetingAgenda",
"isOpen" : true,
"value" : [
{
"pid" : "MA20140206181038936",
"level" : 1,
"id" : "CN20140207211127374",
"isOpen" : false,
"value" : [
],
"line_number" : 13,
"keyItem" : "Fgh",
"isCategorySelected" : false
},
{
"line_number" : 14,
"level" : 1,
"id" : "CN20140217123343769",
"isOpen" : true,
"value" : [
],
"keyItem" : "Fghh",
"isCategorySelected" : false,
"pid" : "MA20140206181038936"
},
{
"pid" : "MA20140206181038936",
"level" : 1,
"id" : "ISS20140217123542229",
"isOpen" : true,
"type" : "Issues",
"value" : [
],
"type_KeyItem" : {
"tfImpact" : "",
"tvIssueDescription" : "",
"tfActionRequired" : "Ffg",
"tfResponsible" : "",
"targetDate" : "Month DD YYYY"
},
"keyItem" : "Issues",
"isCategorySelected" : true,
"line_number" : 15
}
],
"type_KeyItem" : {
"tfAgendaTitle" : "New agenda for magic minutes",
"lblDuration" : "00:00",
"tfAgendaName" : ""
},
"line_number" : 11,
"isCategorySelected" : false,
"pid" : "R1"
}
],
"isCategorySelected" : false,
"keyItem" : "root",
"line_number" : 10
}
],
[
]
]
You can use google-gson library for converting JSON to Java objects and vice-versa.
Some other helpful links are :
1) http://www.javacreed.com/simple-gson-example/
2) http://www.mkyong.com/java/how-do-convert-java-object-to-from-json-format-gson-api/
Hope this will help you.