I have an in-house app that reads data from a WCF server. It keeps a local copy of important items, but once it uploads successfully, it deletes the local copy. This works great as long as there is cell coverage. I have figured out that I need to keep a local copy of all recently accessed data so that the tablet isn't rendered useless if it loses cell coverage. I was wondering if anyone had already written or thought through a system that would manage local data efficiently. There are several important aspects I would like to see:
Whenever a record is read from the server, a local copy is created on "disk"
Also, when data is read from the server, it checks to see if the local copy has been successfully updated to the server before it overwrites the local copy. If the local copy hasn't been updated on the server, it needs to use the local copy.
If it tried to update to the server and fails, then there needs to be a background process that tries to send it later when a cell signal becomes available.
It needs to be able to handle different record types and different key types for looking up the records.
It needs to be able to purge local copies of the records if they have not been accessed for a certain period of time.
Number 4 is my big sticking point. Is there a good way to keep a collection of different types of records and different key types and numbers of keys in order to access them?
I wound up using Akavache github.com/akavache/Akavache to implement my scenario. It is a really cool set of libraries. You can load it using NuGet. Here is a link to a blog that has a basic demo of how to use it:
https://codemilltech.com/akavache-is-aka-awesome/.
Related
In my Android App I save data I receive directly into a DB (room) and use it as a single source of truth for the UI etc. This make the architecture quite nice because the data is available offline by default and I can listen in a reactive fashion on the DB.
Now some of the data I receive is paginated, of course this is where the trouble begins with my architecture. Querying data before the first entry I have locally and after the last entry I have locally is not a challenge because I could simple send their IDs or timestamps, the real problem arises when there is a possibility that new entries can appear between the items I already have saved locally.
My question now is if there is a known technique or algorithm or even library on how I could synchronise the data I have locally and the data I have on the server? Of course there is always the option of sending all IDs that I have locally or a similar solution, but these don't really scale well if you have thousands IDs locally and then have to find the diff between server and client.
One direction I was thinking about was somehow using the updated-at timestamp which is available in the dataset, I didn't come up with something concrete yet tho.
The Setup
I have native iOS and Android apps which sync data to and from my webserver. A requirement of the apps is that they work offline so data is stored on the apps in sqlite databases.
The apps communicate with the server with a series of REST calls which send JSON from the server for the apps to store in their databases.
My Problem
The scale of this data is very large, some tables can have a million records, and the final size of the phone databases can approach 100mb.
The REST endpoints must limit their data and have to be called many times with different offsets for a whole sync to be achieved.
So I'm looking for ways to improve the efficiency of this process.
My Idea
An idea I had was to create a script which would run on the server which would create an sqlite file from the servers database, compress it and put it somewhere for the apps to download. Effectively creating a snapshot of the server's current data.
The apps would download this snapshot but still have to call their REST methods in case something had changed since the snapshot happened.
The Question
This would add another level of complexity to my webapp and I'm wondering if this is the right approach. Are there other techniques that people use when syncing large amounts of data?
This is a complex question, as the answer should depend on your constraints:
How often will data change? If it is too often, then the snapshot will get out of date really fast, thus apps will be effectively updating data a lot. Also, with the big volume of data, an application will waste CPU time on synchronization (even if user is not actively using all of that data!), or may become quickly out of sync with the server - this is especially true for iOS where Applications have very limited background capabilities (only small window, which is throttled) compared to Android apps.
Is that DB read-only? Are you sending updates to the server? If so, then you need to prepare conflict resolution techniques and cover cases, in which data is modified, but not immediately posted to the server.
You need to support cases when DB scheme changes. Effectively in your approach, you need to have multiple (initial) databases ready for different versions of your application.
Your idea is good in case there are not too many updates done to the database and regular means of download are not efficient (which is what you generally described: sending millions of records through multiple REST calls is quite a pain).
But, beware of hitting a wall: in case data changes a lot, and you are forced to update tens/hundreds of thousands of records every day, on every device, then you probably need to consider a completely different approach: one that may require your application to support only partial offline mode (for most recent/important items) or hybrid approach to data model (so live requests performed for most recent data in case user wants to edit something).
100mb is not so big. My apps have been synching many GBs at this point. If your data can be statically generated and upated , then one thing you can do is write everything to the server, (json, images, etc...) and then sync all on your local filesystem. In my case I use S3. At a select time or when the user wants to, they sync and it only pulls/updates what's changed. AWS actually has an API call called sync on a local/remote folder or bucket. A single call. I do mine custom, but essentially it's the same, check the last update date and file size locally and if it's different, you add that to the download queue.
I am developing an Android app that has a list, I would like this list to be synced between multiple users - can it be done with out server side?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database.
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: http://code.google.com/p/google-gson/, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data..
I am working on an application that will, basically, allow people to create, join and manage groups of other people. The people within the groups can also message each other.
I have been wondering which path would be better:
Keep a remote database with all the information, including messages sent to and from users. And have the app query the server every time it needs information. Even information it has seen before.
Keep a remote database with all the information, including messages sent to and from users. Also keep a local copy of the remote database and just keep it synced with the remote database. Whenever the app needs to query for information, it does a query to see if the local table is up to date. If it is not up to date, it updates the table and runs the query on the local table. This way it will keep a local copy and the app will have fast queries when there is not an update to the remote table.
What is generally done with mobile applications and remote databases?
Would it be "bad practice" if i just did number 1?
From my point of view, in most cases, the database in the mobile is just a cache of the real database, the one in the server. So, my suggestion will be to keep locally all data that you need syncing with the server. This allows you to show information even when no connection and show something to the user while the info is updated.
Also, this approach makes the local data volatile without risk, as it's stored in the server. So:
All info is in the server
With a background process (service, thread, intentservice, whatever best suits you) you sync this information with the local database
UI is always showing info from local database
Of course, this is a very general approach, and needs to be examined for each case as different situations may need different approaches.
My base response is that I would keep the data in one place and access it remotely unless there is a major reason to keep it locally. There would have to be extenuating circumstances to mandate that I keep a copy of the data locally. Just make sure your queries are accurate and concise. Don't pull over more data than you need to.However, you can have a subset of data kept locally. Items that are specific to the user (like messages), but keeping data that is not relevant just adds overhead and bloat.
I have made an application which is about pets, at first I had kept the database local to the phone, however as new features arrive I want to make it a network based application with remote database. However I have around 500 downloads on the play store and I dont want my previous users to lose data. I came up with an idea of rolling out an update in which I copy all the databases to the SD card and then mail them back to me and update them in the remote database. I wonder if there is a better way to go around this. Help will be highly appreciated.
A better way would be to write a webservice to which you can send the database row by row. The webservice will then update your Server database(s).
This not only prevents duplication of your database between internal and external memory, but it also allows automation and more flexibility in the update process. You can also pause the transfer, and pick up from whichever row was last sent easily.