I have an external database on Azure that holds large amounts of information. My app needs to be able to synchronize a selection of this data and store it locally on the app's database. I currently use web services to make calls to the database. When the app starts for the first time, it will have a large amount of data that would need to be downloaded. What's the most efficient way of downloading a large number of rows from the external database? I was thinking an XML file might be the best way, but I'm not sure. There could be thousands of rows that need downloading so I'm not sure which method would be the most appropriate.
There isn't 300 ways to download data, so you'll basically have to call your API and get the data. For obvious performance reasons, I would avoid XML and prefere JSON instead. So once you've got your JSON file, you parse it and put it in the locale database. I also would suggest you to use service as it won't be interupt, and don't forget to warn the user that you gonna download massive data ;)
If you wan't to minimize the amount of data downloaded, you can store a JSON file for example in the assets that will contains all the things "static" that won't change online.
I would offload the work to some sort of service. Services have the benefit of running in the background without interacting with the application. You would still need to create another thread to do the work. When the download is finished, it will simply destroy itself.
Data should be downloaded anyway whatever method you use, maybe you may prepare the initial data as XML file, download it the first time and store the data in the database showing a progress bar to the user
Related
I am creating a data repository layer in my application which in turn serves data from two sources, either network API calls or getting data from local storage. When there is a request to the data repository layer, I first need to check if the data is present in the local storage or not. If not, I then make a call to my API to get the data. To check if the data is in local storage or not, I was thinking of two ways to implement this:
On app startup, I load all the data in the database into the memory (eg. lists, maps etc.) and then whenever I have to check of existence of data in the local storage, I check from the memory. This has a possible problem that I have faced earlier as well. When the app is in background, Android system, might clear up this allocated memory to serve memory to other apps. This makes things complicated for me.
Whenever I want to check the existence of data in the local storage, I directly make queries to SQL tables and decide based upon that. This is much more leaner and cleaner solution than the above mentioned case. The only worry for me here is the performance hit. I wanted to know if the SQLite database runs after being loaded into the memory or not? If yes, then the memory layer of data that I had created above is useless. If not, is it safe to keep on querying the SQLite database for each and every small data request?
SQLite caches some data, and the Android OS will keep the database file in its file cache, as long as there is enough memory.
So when using SQLite, your app's performance is automatically optimized, depending on available memory.
I have an issue of saving app data internally on device, Well this go like below:
App is service based. We send a request a for particular home project and it will respond with an image of the project and a list of operations. And they want every link to be clickable and on click of that show more links with some text and images. Means lot of data to be handle.
The data which store internally should also get updates. So what could be the better option of saving it. Download files, download more files when update is required or go with SQLite database, which will update the existing db only.
Till now I am finding to create sqlite db as a better option, but still I want to know which could be the better option. I know this would create a memory issue, but this is what client want.
In terms of storing data in Android, would it be more efficient to use a large ArrayList or setup an SQLite database? I have ~9000 bus stops and stop names (both in String) that I need to store and I was wondering which method would be the quickest or most efficient.
An ArrayList is probably a very bad idea. I assume you want the data to be persistent, so if your user kills your app your data will be lost if you use an ArrayList. A database is persistent(unless the user clears the cache of the app). So I would highly recommend using a database over an ArrayList.
If your data does not change then you could probably have it read on startup and kept in memory while the App runs. For example, having a .json file with all the stops, read it on startup, put the data in a HashMap or something that is easy to use since you will probably lookup stuff. It would probably work fine with ~9000 entries.
If you are going to add or update information about the stops during usage then the SQLite is the better choice though.
1.) Store and retrieve your data from a SQLite DB since you want persistance. And since you say you have 9k+ rows a simple select will give you everything at once and you can easily filter the data as well if you need to
2.) At startup, put all your data into efficient memory structures like HashMaps(or in your case arraylists) and reference them throughout the app. This way you'll only do one time access.
3.) When in doubt build a DB. No harm, more efficient and easier to handle than files
We've got an android app and an iPhone app (same functionality) that use sqlite for local data storage. The apps initially come with no data, then on the first run they receive data from a remote server and store it in a sqlite database. The sqlite database is created by the server and the apps download it as one file, which is then used buy the apps. The database file is not very large by today's standards, but not a tiny one either - about 5-6 MB.
Now, once in a while, the apps need to refresh the data from the server. There a few approaches I can think of:
Download a new full database from the server and replace the existing one. This one sounds like the simplest way to deal with the problem were it not for a repeated 5-6 MB downloads. The apps do prompt the user whether they want to download the updates, so this may not be too much of a problem.
Download a delta database from the server, containing only the new/modified records and in some form information about what records to delete. This would lead to a much smaller download size, but the work on the client side is more complicated. I would need to read one database and, based on what is read, update another one. To the best of my knowledge, there's not way with sqlite to do something like insert into db1.table1 (select * from db2.table1) where db1 and db2 are two sqlite databases containing table1 of the same structure. (The full sqlite database contains about 10 tables with the largest one probably containing about 500 records or so.)
Download delta of the data in some other format (json, xml, etc.) and use this info to update the database in the app. Same as before: not to much problem on the server side, smaller download size than the full database, but quite a painful process to do the updates.
Which of the three approaches you recommend? Or maybe there's yet another way that I missed?
Many thanks in advance.
After much considerations and tries-and-errors, I went for a combination of options (2) and (3).
If no data is present at all, then the app downloads a full database file from the server.
If data is present and an update is required, the app downloads some database from the server. And checks the content of a particular value in a particular table. That value will state whether the new database is to replace the original or whether it contains deletions/updates/inserts
This turns out to be the fastest way (performance-wise) and leaves all the heavy lifting (determining whether to put everything into one database or just an update) to the server. Further, with this approach, if I need to modify the algorithm to, say, always download the full database, it would only be a change on the server without the need to re-compile and re-distribute the app.
Is there a way you can have a JSON field for each of the tables? For instance, if you got a table named users, have a column named "json" that stores the JSON for each of the users. In essence, it would contain the information the rest of the fields have.
So when you download the delta in JSON, all you got to do is insert the JSON's into the tables.
Of course with this method, you will need to do additional work in parsing the JSON and creating the model/object from it, but it's just an extra 3-4 small steps.
I will recommend approach 3, because app will download the json file more fast and local db will be updated more easily avoid overhead of more internet usages.
Just create a empty db initially according to server db and then regularly updated the same by fetching json
I will develop an android application with a lot of data (json files with some rows and CSV for graphics data with a lot of rows) , this data change every 5 minutes and replaces all the previous data (or mostly).
What are the best approaches to design this ? I have 2 options:
Save all the data in a sqlite db, and sync this by a IntentService.
save the data in json and csv files and replace this every 5 minutes.
Which approach will the best performance?
This considering the time to parse the files, sorting data, the download time and the consistency of data.
any other ideas?
PD:I need a cache system too, in case if i don't have internet and I need the previous stored data
Advantages of SQLite:
Changes are ACID
You can make complex requests faster (e.g. "give me only fields A,B from items with (C/D)>E")
I would bet more compact for big data (integers are stored as integers instead of a string of individual digit)
Only one file
You can merge old data with new data easily
You can update current data, even while using it
Concurrency can be handled
Advantages for JSON/CSV:
Easier to debug (plain text)
Faster to make a whole update (copy the new file + delete the old one)
For the original question the whole delete/replace of the data makes JSON/CSV the winner.
However, if the application was to retrieve partial data every 10s and merge/update it with the previous one, SQLite would be a better option.
Sqlite is mostly used when you want data to be saved and used in future. In your case data is changing every 5 minutes so its better to have JSON because every time to make the Database connection store and retrieve after 5 minutes will take some time.
UPDATE:
I also had the same Application in which the data was changing every time. In that case I used Map<K, V> and ArrayList to maintain the values, because as the data is changing everytime I think its not feasible to store the data in Sqlite everytime. It needs much time to perform DB Connection store, retrieve, update the data in Sqlite.
I recommend using JSON or some type of object serialisation unless:
You need ACID compliance for write operations
You need to report against the data which may involve copying the data to an external RDBMS
or
You wish to join those complicit in the overuse / abuse of databases, as commonly seen nowadays
Ideally this should depend on whether you need the previous data, for maybe comparing it with current data and so on. As a thumb rule, I use SQLite when you need the data to be stored and retrieved at a later stage. In case the data is only for display, I would rather keep it in program memory. Mind you this does not involve file operation.
Purpose of JSON and SQLite is completely different from each other
JSON = is used send and receive data between server and client.
SQLite= is used to store data.