I am trying to parse the very large JSON file (for me around 100MB, ~4 million lines) that you receive when downloading your personal location history from Google Takeout and put in a local database in an Android application.
The structure of the JSON file looks like this, and I put a very small sample on Pastebin to show the structure.
{
"locations" : [ {
"timestampMs" : "1518722738911",
"latitudeE7" : 123456789,
"longitudeE7" : 123456789,
"accuracy" : 21
}, {
"timestampMs" : "1518722617339",
"latitudeE7" : 123456789,
"longitudeE7" : 123456789,
"accuracy" : 21,
"activity" : [ {
"timestampMs" : "1518722603634",
"activity" : [ {
"type" : "TILTING",
"confidence" : 100
} ]
}, {
"timestampMs" : "1518722660455",
"activity" : [ {
"type" : "STILL",
"confidence" : 55
} ]
} ]
} ]
}
When parsing this in Java with GSON, you will get a LinkedTreeMap. I know that GSON is not the fastest parser out there, but that is another issue.
The purpose of storing all the JSON data into a SQLite database is to have it available much faster and analyze it, so I would like to have the data as practical as possible.
I tried creating two separate tables, one for simple timestamps and their longitude and langitude, and one for the activities associated with timestamps themselves.
However, I can't quite wrap my head around of how to iterate through this TreeMap correctly, converting those entries into simple rows, since it's quite a mess with many subarrays.
Is SQLite even the right datbase, or should I use something else for this kind of data? How would you parse such a file?
Any tip or suggestion is very much appreciated, thanks.
You could use http://www.jsonschema2pojo.org/ which converts Json to Pojo automatically.
Related
I am stuck in one case of firebase operation in android.
My Requirement :
I have table Named : "Values"
This value table contains following type of data.
{
"menu": [
{
"category": "cocktails",
"product": [
{
"id": "123",
"name": "test1"
},
{
"id": "456",
"name": "test2"
}
]
},
{
"category": "vodka",
"product": [
{
"id": "789",
"name": "test3"
},
{
"id": "901",
"name": "test4"
}
]
}
]
}
Now, I want to update id : 123 with name = "test5" , How to update this nested array object
in firebase database in android?
I tried following but it update/add Entire menu array not perticular product array object.
I just want to update in product array object.
Following are the code that i tried, but it update all menu array.
val listPorduct = ArrayList<HashMap<String, Any>>()
for ((i, product) in productList.withIndex()) {
var productMap = HashMap<String, Any>()
productMap = hashMapOf(
"id" to "123",
"name" to "test5")
listPorduct.add(productMap)
}
val map = hashMapOf<String, Any>()
map.put("category", list[position].category)
map.put("product", listPorduct)
repository.getTabs().update("menu", FieldValue.arrayUnion(map))
if i am try with
repository.getTabs().update("menu/product", FieldValue.arrayUnion(map))
or
repository.getTabs().update("menu.product", FieldValue.arrayUnion(map))
then getting issue / or dot not allowed. i have latest firebase gradle file.
How update particular position of product object?
Please anyone can help me to solve out this?
Image of firebase database.
The solution to this problem can be found in my answer from the following post:
firestore -- Add element in a field in hashmap
There is a slightly smaller difference. However, for your use-case, please note that the document that you gave us as an example contains a property named menu which is of type array. In this array, you have added some objects that contain two properties, a String property named category and an array named product. This array contains in term two other properties, one is the id and the other one is the name, both being of type String.
None of those arrays can be updated using FieldValue.arrayUnion(map). The only option that you have is to get the menu array as a list of HashMap<String, Any>. Once you have this map, you can simply iterate through it and get the desired data.
So to update the value of a property inside a Map, you should first get the map, do the changes and put it back. In the end, after you've done all the necessary changes, write the document back to Firestore.
Edit:
According to your comment:
In my case I have more than 5000 records in a list with a nested array.
It's hard to believe that you can nest 5000 elements into a single document because it might not fit since the maximum size for a document is 1 MiB (1,048,576 bytes). Please see usage and limits. So nesting 5000 objects into an array isn't a solution at all. In this case, the single solution that you have is to use a sub-collection and add each item from the array as a document. In this case, there are no limitations.
As a conclusion, the only highly scalable way to store an unknown large list of items is using documents in a collection. Array type fields do not scale for lists the are constantly growing because the items will eventually exceed that (1 MiB), which will obviously cause problems in the future.
I am stuck in one case of firebase operation in android.
My Requirement :
I have table Named : "Values"
This value table contains following type of data.
{
"menu": [
{
"category": "cocktails",
"product": [
{
"id": "123",
"name": "test1"
},
{
"id": "456",
"name": "test2"
}
]
},
{
"category": "vodka",
"product": [
{
"id": "789",
"name": "test3"
},
{
"id": "901",
"name": "test4"
}
]
}
]
}
Now, I want to update id : 123 with name = "test5" , How to update this nested array object
in firebase database in android?
I tried following but it update/add Entire menu array not perticular product array object.
I just want to update in product array object.
Following are the code that i tried, but it update all menu array.
val listPorduct = ArrayList<HashMap<String, Any>>()
for ((i, product) in productList.withIndex()) {
var productMap = HashMap<String, Any>()
productMap = hashMapOf(
"id" to "123",
"name" to "test5")
listPorduct.add(productMap)
}
val map = hashMapOf<String, Any>()
map.put("category", list[position].category)
map.put("product", listPorduct)
repository.getTabs().update("menu", FieldValue.arrayUnion(map))
if i am try with
repository.getTabs().update("menu/product", FieldValue.arrayUnion(map))
or
repository.getTabs().update("menu.product", FieldValue.arrayUnion(map))
then getting issue / or dot not allowed. i have latest firebase gradle file.
How update particular position of product object?
Please anyone can help me to solve out this?
Image of firebase database.
The solution to this problem can be found in my answer from the following post:
firestore -- Add element in a field in hashmap
There is a slightly smaller difference. However, for your use-case, please note that the document that you gave us as an example contains a property named menu which is of type array. In this array, you have added some objects that contain two properties, a String property named category and an array named product. This array contains in term two other properties, one is the id and the other one is the name, both being of type String.
None of those arrays can be updated using FieldValue.arrayUnion(map). The only option that you have is to get the menu array as a list of HashMap<String, Any>. Once you have this map, you can simply iterate through it and get the desired data.
So to update the value of a property inside a Map, you should first get the map, do the changes and put it back. In the end, after you've done all the necessary changes, write the document back to Firestore.
Edit:
According to your comment:
In my case I have more than 5000 records in a list with a nested array.
It's hard to believe that you can nest 5000 elements into a single document because it might not fit since the maximum size for a document is 1 MiB (1,048,576 bytes). Please see usage and limits. So nesting 5000 objects into an array isn't a solution at all. In this case, the single solution that you have is to use a sub-collection and add each item from the array as a document. In this case, there are no limitations.
As a conclusion, the only highly scalable way to store an unknown large list of items is using documents in a collection. Array type fields do not scale for lists the are constantly growing because the items will eventually exceed that (1 MiB), which will obviously cause problems in the future.
I'm no back-end developer. So perspective is always appreciated.
I have written a script which requests from an API and creates this huge JSON file I want to save in firebase, how can I accomplish this? And would it be possible to filter this json with python for example; when I add region=eu in the url this returns the objects which have Europe as region or do I absolutely need to request the entire json file and parse in my code (java android) ?
Since there are a few parts to your question:
You can save JSON to Firebase and the data will be mapped to child locations:
Using PUT, we can write a string, number, boolean, array or any JSON object to our Firebase database...When a JSON object is saved to the database, the object properties are automatically mapped to child locations in a nested fashion.
https://firebase.google.com/docs/database/rest/save-data
And for your next question:
And would it be possible to filter this json with python for example; when I add region=eu in the url this returns the objects which have Europe as region
Looks like you should be able to jimmy something together with Firebase's filters, startAt and endAt:
We can combine startAt and endAt to limit both ends of our query.
https://firebase.google.com/docs/database/rest/retrieve-data#section-rest-filtering
For your example you might do something like this:
curl 'https://yourfirebase.firebaseio.com/yourendpoint.json?orderBy="$REGION_NAME"&startAt="EU"&endAt="EU"&print=pretty'
...or do I absolutely need to request the entire json file and parse in my code (java android) ?
The facts that JSON objects are stored hierarchically in Firebase and that you can filter based on those object values makes me think you do not, in fact, have to request the entire JSON file. However, I don't have personal experience with this particular aspect of Firebase, so give it a shot!
As #ackushiw mentions in the comments, you can also use the equalTo query (https://firebase.google.com/docs/reference/js/firebase.database.Query#equalTo):
curl 'https://yourfirebase.firebaseio.com/yourendpoint.json?orderBy="$REGION_NAME"&equalTo="EU"&print=pretty'
It really depends on how you are structuring your JSON. It's generally recommended to make your JSON tree as shallow as possible since all children are loaded when you have a matching query.
FIREBASE DATA:
{
"-id1": {
"region": "eu" // bear in mind queries are case sensitive
"title": "Foo"
"nested": {
"city": "berlin"
}
},
"-id2": {
"region": "other"
"title": "Bar"
"nested": {
"city": "berlin"
}
},
"-id3": {
"region": "eu"
"title": "Baz"
"nested": {
"city": "paris"
}
}
}
Querying with (using the Android API)
.orderByChild("region").equalTo("eu")
would return "-id1" and "-id3"
with
.orderByChild("nested/city").equalTo("berlin")
would return "-id1" and "-id2"
The REST API Returns Unsorted Results: JSON interpreters do not enforce any ordering on the result set. While orderBy can be used in combination with startAt, endAt, limitToFirst, or limitToLast to return a subset of the data, the returned results will not be sorted. Therefore, it may be necessary to manually sort the results if ordering is important.
If you're using a more complex structure I recommend watching this video
https://www.youtube.com/watch?v=vKqXSZLLnHA
I'd also recommend using the firebase library for Android
https://firebase.google.com/docs/android/setup
And Firebase-UI, It does a lot for you.
https://firebaseopensource.com/projects/firebase/firebaseui-android/
I have a json file on server:
{"images":[
{"url":"...", "likes":"123"},
{"url":"...", "likes":"234"},
{"url":"...", "likes":"345"}
]}
I get the json file on android read it, but if someone likes a picture i want to change the value of the first picture from 123 to 124, is this possible and how can i do this?
The whole point is to change a json value on server,from client side.
Also if this isn't possible how can i make this happen?
Also if i want to get the Top50 rated pictures,how can i sort them and get only the 50 picture,without getting all pictures and sorting them on android ?
Which one is better,initializing the Top50 images,when the user starts the app,or when the user click on the button Top50.I assume that if its when he click the button,there might be some performance issues server side?
My other idea is to have a function server side,which every 10 min,executes automatically and gets the Top50 rated and makes something like json file.So it all happens server side automatically.
To make this happen, client should expose some interface, i.e. function that will allow to modify file on server side. The interface and implementation of this function greatly depends on server itself, i.e. which protocols it handles, what built-in or external modules it supports, which languages are supported, etc... For example, the classic scenario is using apache as HTTP server, CGI enabled, and write CGI function in perl. So, in this case interface would look like http://server.name/like.cgi?image=image123.
How to modify the values on the server ?
For this every like of a photo should be a post request of this sort.
{
"data": [
{
"image_id": 3133456,
"likes": 343
},
{
"image_id": 3133456,
"likes": 343
}
]
}
On parsing this request server updates the corresponding image's like on the server.
How to get the top 50 rated/liked images from the server ?
Again you send a get request to such a url
http://server.getsomething.com/getTop50Images
On server side
On receiving such a request you make a query on the table in your database something like this
select image_id , image_url, likes from image_table limit 50 ORDER BY likes ASC
Now getting those query results and returning them as a json would not be a performance hit until you have huge bulk of data. like some million rows may be in your database.
Response can be something like this
{
"result": [
{
"image_id": 3133456,
"likes": 34400,
"url": "http://flickr.com/someimage"
},
{
"image_id": 3133456,
"likes": 34380,
"url": "http://flickr.com/someimage"
}
]
}
You still avoid using a database yourself but can lease it from clouds services like parse.
However if you won't be using those services then you can take a look at ftp packages for js. Like the neo JavaScript library by apache.
But still a good choice will be to go with the database approach (is quiet simpler).
I am working with an api where I am retrieving JSON data. I've come across a data type that I am unfamiliar with and I would like some insight into what it means.
An example is shown below.
{
"data":{
"id":"92",
"name":"harry",
"friends":"a:2:{i:0;s:1:\"1\";i:1;s:2:\"15\";}"
"enemies":"a:0:{}"
},
"error":false
}
I am unsure how to interpret the "friends" and "enemies" fields. I am aware that arrays are represented by [ ] and objects by { }.
Could anyone explain what they mean?
Thank you.
The fields contain arrays that have been serialized by PHP.
For example
<?php
$friends = "a:2:{i:0;s:1:\"1\";i:1;s:2:\"15\";}";
$arr = unserialize($friends);
var_dump($arr);
?>
gives
array(2) {
[0]=> string(1) "1"
[1]=> string(2) "15"
}
What these values represent will be application-specific.
The value of friends is a string with the value: a:2:{i:0;s:1:"1";i:1;s:2:"15";}
What this string represents depends on what the application does with it