I will develop an android application with a lot of data (json files with some rows and CSV for graphics data with a lot of rows) , this data change every 5 minutes and replaces all the previous data (or mostly).
What are the best approaches to design this ? I have 2 options:
Save all the data in a sqlite db, and sync this by a IntentService.
save the data in json and csv files and replace this every 5 minutes.
Which approach will the best performance?
This considering the time to parse the files, sorting data, the download time and the consistency of data.
any other ideas?
PD:I need a cache system too, in case if i don't have internet and I need the previous stored data
Advantages of SQLite:
Changes are ACID
You can make complex requests faster (e.g. "give me only fields A,B from items with (C/D)>E")
I would bet more compact for big data (integers are stored as integers instead of a string of individual digit)
Only one file
You can merge old data with new data easily
You can update current data, even while using it
Concurrency can be handled
Advantages for JSON/CSV:
Easier to debug (plain text)
Faster to make a whole update (copy the new file + delete the old one)
For the original question the whole delete/replace of the data makes JSON/CSV the winner.
However, if the application was to retrieve partial data every 10s and merge/update it with the previous one, SQLite would be a better option.
Sqlite is mostly used when you want data to be saved and used in future. In your case data is changing every 5 minutes so its better to have JSON because every time to make the Database connection store and retrieve after 5 minutes will take some time.
UPDATE:
I also had the same Application in which the data was changing every time. In that case I used Map<K, V> and ArrayList to maintain the values, because as the data is changing everytime I think its not feasible to store the data in Sqlite everytime. It needs much time to perform DB Connection store, retrieve, update the data in Sqlite.
I recommend using JSON or some type of object serialisation unless:
You need ACID compliance for write operations
You need to report against the data which may involve copying the data to an external RDBMS
or
You wish to join those complicit in the overuse / abuse of databases, as commonly seen nowadays
Ideally this should depend on whether you need the previous data, for maybe comparing it with current data and so on. As a thumb rule, I use SQLite when you need the data to be stored and retrieved at a later stage. In case the data is only for display, I would rather keep it in program memory. Mind you this does not involve file operation.
Purpose of JSON and SQLite is completely different from each other
JSON = is used send and receive data between server and client.
SQLite= is used to store data.
Related
I'm working on an Android app for homework management. I'm a senior in college, so my experience on larger projects is very limited, but I'd like to design all parts of this app well instead of just throwing something together. This includes the way data is stored.
We have two main objects/entities: Task and Subject. Even if someone uses the app for the whole time they're in college and never deletes anything, I'm guessing there would be a maximum of a few thousand tasks and a couple hundred subjects (not all subjects would be shown at once). The initial version of the app won't sync data with a server, but this is a definite possibility in the future, so I'd like to design with that in mind. We might also have an option for users to send tasks to each other.
Here are my questions:
Would a SQLite database be the best option for storing the amount of data we're likely to have, or would something like serializing it to XML or JSON then loading it into memory when the app starts work?
I'm used to thinking in terms of objects. This means that if I use a database and it has a Task table and a Subject table, my first instinct is to convert each database table row into a corresponding object for viewing/editing. (The objects' setters would contain validation logic.) Is this a good/helpful/necessary way to think? If not, what is the alternative?
Thanks for your help!
This question is broad so may comments below may not be 100% correct as I don't have all the information about your system.
SQLite is better suited for storing thousands of records than files (be it JSON or XML). This is especially true if your data is not static, i.e. will be changed during the usage of your app (which is the case for you, I believe). You can take advantage of existing functionality for records inserts, updates, deletions, using indexes, etc.
Yes, you generally create objects similar to your database. But you don't usually need to convert each and every record from the database into your objects. You usually query the database for a limited number of objects, depending on what you want to show in the UI. Of course, if you need to show all, let's say, tasks, you need to get them all.
1. Would a SQLite database be the best option for storing the amount of data we're likely to have, or would something like serializing it to XML or JSON then loading it into memory when the app starts work?
Yes SQlite will be the option for you.It will give you a structured format and in future if you want to access data from remote end the same structure of tables can be used without much change in the code.
2. I'm used to thinking in terms of objects. This means that if I use a database and it has a Task table and a Subject table, my first instinct is to convert each database table row into a corresponding object for viewing/editing. (The objects' setters would contain validation logic.) Is this a good/helpful/necessary way to think? If not, what is the alternative?
you can simply execute queries to manipulate data.
But dont forget to encryt your database if you storing it in mobile itself.
In my android app I load some data (JSON) from wed server. And I need to store that data.
All data, which I want to save - it is fixed number (for ex. 5) of JSONObjects with 30-50 fields.
Is this good idea to save each JSONObject in Preference? Or will be better to use SQLite?
note: why I ask?
save in Preference - implementation take 10 minutes.
use SQLite - implementation take one or two days, and required to
edit database, when I will add new fields.
I'd prefer database, and if you need 2 whole days for a reasonably simple database setup, you should perhaps take any chance to practice with it.
However, if you're sure you're never going to have any advantage of (complexer) queries, then saving it into a database might not give you any real advantages.
I would prefer sharedpreference here , because size of data is not too big. If size of data was too big and had many fields then database would be first choice. You can save the whole json String into preference and use it after parsing at run time.
My implementation choice would be a database table that stores the object as something you can then transform into a native intermediate object. By keeping the local data stored as JSON you're potentially locking yourself into keeping a JSON parser within the app for the foreseeable future even if your web service changes its format.
For example, if your web service changes to XML you're still going to have to keep the JSON parser in the app to handle those users who upgrade the app and have to do a data migration.
You could save JSON directly in SQLite database. If you parse data and then save in database then it would take some time for you to make everything work. But think about saving JSON directly in database. It will require two fields only in database link and json response.
Advantages:
- Big time saving
- You could save the same solution in your future apps for saving JSON in database.
Disadvantages
- You will not be able to query data
- You will have to parse every-time
In terms of storing data in Android, would it be more efficient to use a large ArrayList or setup an SQLite database? I have ~9000 bus stops and stop names (both in String) that I need to store and I was wondering which method would be the quickest or most efficient.
An ArrayList is probably a very bad idea. I assume you want the data to be persistent, so if your user kills your app your data will be lost if you use an ArrayList. A database is persistent(unless the user clears the cache of the app). So I would highly recommend using a database over an ArrayList.
If your data does not change then you could probably have it read on startup and kept in memory while the App runs. For example, having a .json file with all the stops, read it on startup, put the data in a HashMap or something that is easy to use since you will probably lookup stuff. It would probably work fine with ~9000 entries.
If you are going to add or update information about the stops during usage then the SQLite is the better choice though.
1.) Store and retrieve your data from a SQLite DB since you want persistance. And since you say you have 9k+ rows a simple select will give you everything at once and you can easily filter the data as well if you need to
2.) At startup, put all your data into efficient memory structures like HashMaps(or in your case arraylists) and reference them throughout the app. This way you'll only do one time access.
3.) When in doubt build a DB. No harm, more efficient and easier to handle than files
I have an ArrayList of Location objects. The ArrayList typically has
from 1,000 to 10,000 Location objects.
I want to be able to persist the ArrayList and then read it back in
later.
I am assuming that it would be much faster to write out the entire
ArrayList of Location objects as a single file rather than storing
each Location object in SQLiteDB.
Can someone show me some example code to read/write an ArrayList of
Location objects, or suggest a better solution?
Are you really going to have to store all of the 10k locations, delete them all and store a new set of 10k locations every time? Or just make some updates occasionally? (such as delete or add several rows). If the latter, than you want to be using SQLite as it will be monumentally easier to do those actions with the Db.
Also note that SQLite is also just a stored file on the filesystem, there isn't a DB server in the way there is with MySQL, for example.
That aside, this has been discussed before on SO, so you may want to look at these:
Android data storage - File vs SQLite
Sqlite vs File based data storing?
We've got an android app and an iPhone app (same functionality) that use sqlite for local data storage. The apps initially come with no data, then on the first run they receive data from a remote server and store it in a sqlite database. The sqlite database is created by the server and the apps download it as one file, which is then used buy the apps. The database file is not very large by today's standards, but not a tiny one either - about 5-6 MB.
Now, once in a while, the apps need to refresh the data from the server. There a few approaches I can think of:
Download a new full database from the server and replace the existing one. This one sounds like the simplest way to deal with the problem were it not for a repeated 5-6 MB downloads. The apps do prompt the user whether they want to download the updates, so this may not be too much of a problem.
Download a delta database from the server, containing only the new/modified records and in some form information about what records to delete. This would lead to a much smaller download size, but the work on the client side is more complicated. I would need to read one database and, based on what is read, update another one. To the best of my knowledge, there's not way with sqlite to do something like insert into db1.table1 (select * from db2.table1) where db1 and db2 are two sqlite databases containing table1 of the same structure. (The full sqlite database contains about 10 tables with the largest one probably containing about 500 records or so.)
Download delta of the data in some other format (json, xml, etc.) and use this info to update the database in the app. Same as before: not to much problem on the server side, smaller download size than the full database, but quite a painful process to do the updates.
Which of the three approaches you recommend? Or maybe there's yet another way that I missed?
Many thanks in advance.
After much considerations and tries-and-errors, I went for a combination of options (2) and (3).
If no data is present at all, then the app downloads a full database file from the server.
If data is present and an update is required, the app downloads some database from the server. And checks the content of a particular value in a particular table. That value will state whether the new database is to replace the original or whether it contains deletions/updates/inserts
This turns out to be the fastest way (performance-wise) and leaves all the heavy lifting (determining whether to put everything into one database or just an update) to the server. Further, with this approach, if I need to modify the algorithm to, say, always download the full database, it would only be a change on the server without the need to re-compile and re-distribute the app.
Is there a way you can have a JSON field for each of the tables? For instance, if you got a table named users, have a column named "json" that stores the JSON for each of the users. In essence, it would contain the information the rest of the fields have.
So when you download the delta in JSON, all you got to do is insert the JSON's into the tables.
Of course with this method, you will need to do additional work in parsing the JSON and creating the model/object from it, but it's just an extra 3-4 small steps.
I will recommend approach 3, because app will download the json file more fast and local db will be updated more easily avoid overhead of more internet usages.
Just create a empty db initially according to server db and then regularly updated the same by fetching json