I have a remote database that can change at any time. I need to find a way to keep my sqlite database update with all the changes in the better and optimised way possible.
I thought having a single timestamp per table and send only the updated table to the client would be one solution, or maybe having one timestamp per row and sending only the updated row to the client..
But
- how can i manage deleted items, for example?
- how can i manage the technical update on Android?
Basically, in a few words, at some point in the app, i need to download the changes with an API and update the local db.
Does anyone have some ideas?
Thanks
Since it seems this is a one-way only sync, I would:
Make each table on the server DB have a UUID column and a last
updated date/time column.
Create a REST call that will request updated data. Ideally, it would
take the time the client was updated. This date should come from the
server during the update call.
The server would scan each table searching for rows that have a date > than the date passed. Serialize the data and return it as JSON along with the server date/time for the next request.
Now tracking deletes is a bit more work. You can either:
Never delete data and only mark the rows as deleted, this is the easiest, but depending on your data may end up with lots of rows in the DB.
Delete the data and then track the deleted rows in another table. These can be deleted after some time and if/when all clients have been updated.
Have a second update call that will push down all the UUID values and time stamps for each row. The client could then figure out which rows need to be deleted.
Though it's the most complex, I'd probably opt for option 3, as I don't like old data hanging around. The direction I would go would also depend on how many clients will be syncing with the DB.
I've decided to follow another approach:
I've created an api that parse the Mysql db into an SQLITE db on the SERVER side. There is a "last_update_timestap" that will be updated every time that something will be changed.
Another api call, gives in the header an extra field containing that timestamp.
On the client side there is a Sync process that do the follow when needed:
Api call retrieving just the HEADER, getting the last update timestamp and checking if an update is needed;
If is needed download the database from the other api
Write the database into a temporary file and copy that temporary file at the position and with the same name of the real database
Use the database
Thanks everyone for the help but this turned out to be the suitable approach for my project structure.
Related
I currently have a local SQLite database that will update periodically from a server.
Initially, the local database will be empty. The app will fetch entire server database as json, and insert all the new rows locally.
After some time the app will fetch the entire server database as json again, and insert any new rows, or update any existing rows that have changed.
Now, the issue is if the server's database has removed an item (row). The app will never be aware of these removed items, and it will exist forever locally.
What's the best way to handle this?
A few ideas:
Add a "is deleted" flag on each item. This would allow the app to remove any items, but this seems like it would be harder to maintain, and a lot of changes for each table. Although it would allow you to undelete items as well.
Same idea as the flag, just modify as existing field and return it. For example, return a negative id ( id = 150 -> id = -150 ). This wouldn't require any database changes, but is confusing to anyone who doesn't know what's happening.
Modify the result to return a list of "deleted" items. This wouldn't be too much work, but doesn't seem like a good solution either.
Is there any standard pattern to follow for this?
Thanks.
You can use Realm instead of SQLite for better experience, And When the app is fetching the data from server clear all data in the realm and insert new records, It will not save the removed item.
Thank you.
I have a SQLite database on Android, which I'm designing to be available offline, and is getting data through a REST API on the server side. But is there any way I can only get the data that has changed since last upload? For example, if there is 5 customers registered in the MySQL database, and one of them change the address, then I only want to update the one with new address, and not the other 4.
Using timestamp field will be a solution for downloading incremental changes from the server to the client. For uploading data from the client to the server - some "Dirty" flag can be used to designate new records for upload.
Have a look at this question: Client-server synchronization over REST, which addresses the similar problems.
There is no build in API.
However you can create a timestamp column for every table which gets set on every update and insert.
When syncing, you can ask the server to give you only the rows with a timestamp greater than the last sync.
You might get in trouble when doing updates in both directions.
With your help, I have successfully created a web page which has its own MySQL DB and then uses a JSON web service to pass the values my Android application.
The next stage is to copy them into a local SQLite DB so the phone does not require an internet connection to view the list.
I have successfully implemented code that creates a table, and inserts values into this table.
The problem is, I don't want it to necessarily re-write the whole DB at a time, which is the current idea to keep it current.
I might have things in the DB deleted or more things added, and want the local DB to reflect this.
What steps can I take to delete things that are no longer present in the MySQL DB in the SQLite one, and add new things?
Cheers
The best way to do this should be to have a date time column in your database and then just fetch the posts that are newer then the last time you fetched from the database, in your android app you could save an date time in your preferences
I'm sorry I missed that you wanted to delete items as well, than this probably isn't the best approach. Are you using the mySQL only for the API? Then you could probably mark posts as deleted inserted odd feeling them.
You can think this approach, parse your JSON response into objects of your data model and then create the content values holding that data model to work with the database in your device.
Once you have the content values, use a content resolver to work with your database and try an update of each data parsed from the JSON response (each content value), update method of the content resolver will return an integer value that tell us the number of rows updated in the database, if such number is 0 means that no data were in the database, so make an insertion, simply.
In that way you first search if the current data is present in the database, if it is, update it, if not, insert it.
We've got an android app and an iPhone app (same functionality) that use sqlite for local data storage. The apps initially come with no data, then on the first run they receive data from a remote server and store it in a sqlite database. The sqlite database is created by the server and the apps download it as one file, which is then used buy the apps. The database file is not very large by today's standards, but not a tiny one either - about 5-6 MB.
Now, once in a while, the apps need to refresh the data from the server. There a few approaches I can think of:
Download a new full database from the server and replace the existing one. This one sounds like the simplest way to deal with the problem were it not for a repeated 5-6 MB downloads. The apps do prompt the user whether they want to download the updates, so this may not be too much of a problem.
Download a delta database from the server, containing only the new/modified records and in some form information about what records to delete. This would lead to a much smaller download size, but the work on the client side is more complicated. I would need to read one database and, based on what is read, update another one. To the best of my knowledge, there's not way with sqlite to do something like insert into db1.table1 (select * from db2.table1) where db1 and db2 are two sqlite databases containing table1 of the same structure. (The full sqlite database contains about 10 tables with the largest one probably containing about 500 records or so.)
Download delta of the data in some other format (json, xml, etc.) and use this info to update the database in the app. Same as before: not to much problem on the server side, smaller download size than the full database, but quite a painful process to do the updates.
Which of the three approaches you recommend? Or maybe there's yet another way that I missed?
Many thanks in advance.
After much considerations and tries-and-errors, I went for a combination of options (2) and (3).
If no data is present at all, then the app downloads a full database file from the server.
If data is present and an update is required, the app downloads some database from the server. And checks the content of a particular value in a particular table. That value will state whether the new database is to replace the original or whether it contains deletions/updates/inserts
This turns out to be the fastest way (performance-wise) and leaves all the heavy lifting (determining whether to put everything into one database or just an update) to the server. Further, with this approach, if I need to modify the algorithm to, say, always download the full database, it would only be a change on the server without the need to re-compile and re-distribute the app.
Is there a way you can have a JSON field for each of the tables? For instance, if you got a table named users, have a column named "json" that stores the JSON for each of the users. In essence, it would contain the information the rest of the fields have.
So when you download the delta in JSON, all you got to do is insert the JSON's into the tables.
Of course with this method, you will need to do additional work in parsing the JSON and creating the model/object from it, but it's just an extra 3-4 small steps.
I will recommend approach 3, because app will download the json file more fast and local db will be updated more easily avoid overhead of more internet usages.
Just create a empty db initially according to server db and then regularly updated the same by fetching json
HI All,
I am developing an application which would need to regularlay update the database from the server. I am planning to get the database from server in json format.
But I want to update the data for those records which have been updated on the server. I do not want to download the complete database but only the differential database so that data transfer is minimum each time.
What is the best way to achieve this? Any sample code available?
Rgds,
Sapan
Simply put a *modified_date* column in the corresponding tables in the database and ensure it is updated by server(or some other mechanism such as using database provided facilities) whenever a row is updated/added. At any time you can request the records which have *modified_date* greater than the time when last query was executed. On your application you have to maintain the time when the data was last fetched.