I have mobile app. Something like to do list or calendar. Teoretically user can have a few devices with that application on a defferent platforms and so on. I would like to create a automatic synchronization between them through a own server. What is the best practice: update all the information or only the changes? On the one hand usually there is no a lot of data when it's about a to do list but who knows?
The correct approach is not date/time as others suggest, as time can go out of sync. The right algorithm is to keep the checksum of the data entries during last synchronization. On next synchronization you compare current checksums with stored ones, then you know whether the entry has been changed on the server, on the client or both.
Our open-source Rethync SDK lets you implement the above approach quite easily and is available for Android (not for iOS at the moment).
I am doing something similar in my application. I have a last modified date field with each entity that I need to sync. Then periodically, I post this data to the server (actual data + date and time). Now the server can do one of two things. It will check the corresponding data on server side and compare the last modified date. If what the server is latest, it will return the latest data in response. If not, it will update its data and send a response indicating what client has is latest.
Of course you can do several optimization. That is, mark the data as "dirty" so you know whether to even send your data to server. If the phone does not have modified data, your sync is basically getting the latest data from server.
Basically server does the heavy lifting and does all the logic necessary to maintain the latest data on its end and send responses to client appropriately.
Good Luck
Best approach is use a time stamp to handle this.
Initial request to server with time stamp value 0.
Server will give the all the data first time with Time-stamp.
Store the Time stamp to sharedpreferences.
In All next request pass the time stamp back to the server
Server will send only those data which are add/update/ after that
given time stamp
That is it.
There is a new alternative to the syncing problem. It's called EnduroSync from Orando Labs. You can sync object data stores between devices on Android and iOS now, with others coming soon.
Full Disclosure: I work for Orando Labs.
The EnduroSync clients allow you to create object data stores on the local devices. The clients are fairly sophisticated - data is modeled as native objects for each client we support (iOS and Android now, more coming). The clients work offline and online. The data is saved to an sqlite database locally.
As you change objects in your model, the deltas are recorded on the device. At some point, you can 'sync' the object data store. Syncing uses a commit/push/pull process (like git), but this is invisible to you. The sync brings your local copy up to date with whatever is on the server, and sends up any changes you have made. Conflicts are resolved using a timestamp based merge, so newer data is not overwritten by older data.
EnduroSync is an online service, so there is no server setup on your end.
There is also a flexible permission system which lets you share the object data stores in a variety of ways. For instance, most applications will have one or more object data stores for each user, for preferences, notes, tags, etc. You can also share object data stores per app, per user type, and with wild cards, many other ways.
So basically you use our client SDK's to model your data on the device. Modeling is with simple objects in the native programming language of the device. If you sign up for the syncing service, you get the syncing also.
Here is another approach.
Issue :I need to have the appointments of doctors syned to client (mobile device) from the server. Now the appointments can drop off or the data could possibly change on the server. Having the client to know what change and sending a request back to server could be an expensive service.
Possible approach : Have the server do the heavy lifting. Keep a table which stores values of time stamp and if a change happened with regard to an appointment - cancellation / reschedule etc. The client would then look at this table to see if anything changed. In reality we don't need to sync anything but only the delta which server can provide to the client based on what it has and what is at Client. There is one aspect which needs to be taken care of is updation of info from client to server and traditional conflict management can be done where client can update the server when a data connectivity between client and server exists.
Essentially the approach is to have only the deltas synced by maintaining a checksum or data change log to PUSH changes to the client.
Related
I have been trying to implement SyncAdapter to sync data between client(android device) and server using sqlite.The same data will be shared between different devices.
On Update:
As soon as a particular record is updated on one client device it will be pushed to the server. At that moment the updated record should also be synced from the server to other connected devices.
Now, when we send sync request from other connected devices, what is the best way to know which record get updated on the server?
Here is what I would do to achieve this:
First, consider using timestamps. Here is how that could work;
In your database table (remotely), have a column updated_at which as soon as you sync your fresh data up, you should set a new timestamp for it.
In your android code, you can simply store lastSyncTime in your preferences then compare with what you get from your server column.
You can easily pull the value from an endpoint /last_sync/ to see if the sync_time has changed.
By comparing the two timestamps, you can then decide as to whether you want to sync down new data or keep the old until the new ones are synced.
I hope this helps. Good luck!
I am working on a Point of Sale application that needs to be very good syncing mechanism. We have Magento Database.The android device have SQLite local Db. Now we need to sync in the following way:
Local ------Sync To---------------> Server (Up Sync)
Server------Sync To---------------> Locals (Down Sync)
There are 2 things:
1) write-to (How to take care??)
For every change that i do on local ,it will directly sync my local to server
2) write-back (How to take care???)
Whenever there is a change in server, we need to sync all our locals with server.
So, the task is: to identify a server update
And sync our locals.
Like there are 4 devices are running in a store and we have added one new customer through one device. Now i want that the three other devices local db also updated with the information about that customer and server also updated.
I heard about the background threads and run threads after a time interval. But what is the best way to do that which don't affect the application. Also the all Big Retail stores uses the syncing process. what they used for that ?
Any help is appreciated.
It fully depends on you database structure...
you have DATABASE in LOCAL (device) and on SERVER
NOW
You need to have TIMESTAMP fieLd added to the TABLES which actually you want to keep in SYNC.
When ever you will make any changes on server the TIMESTAMP will be updated there and same will be the case for the local database also what you have to do is now.
Run a service in the background which will keep on comparing the TIMESTAMPS of LOCAL with that of SERVER.
Now you have to put condition that if TIMESTAMP of SERVER is newer to that of LOCAL then bring changes from SERVER to LOCAL,
and vice versa will be the condition to take changes from LOCAL to SERVER.
Further you have to decide how frequently you want to run this SERVICE.
ALTERNATIVELY:
You can make the table there on SERVER which will store LAST_SYNCHED date for particular device
Whenever you will login in you device (or any other particular event on which you want it to perform this) the server will check-
when this device was LAST_SYNCHED
then it will compare it to TODAYS DATE
and will check what upadets actualy happened between these dates and
will send the changes to the LOCAL (device)
and vice versa for LOCAL (device) to SERVER
you have to play with TIMESTAMPS rest you can have your own logic how to structure the database.
I told you what I have have observed, when I have been a part of such similar project
EDIT
The above Process defines how to sync the devices with server I mean the strategy..
If you want your devices to get notified from server when to sync instead of hitting the WEB-SERVICE recurrently ..
You can make use of PUSH NOtification, GCM is one of them that send push notification to devices, you can integrate it to your project
For syncing you need to handle following two situations.
How and when to receive server updates
How to identify local non-synced data
How and when to receive server updates:
For receiving updates, we can use GCM (Google Cloud Messaging). If any updates made in server, server sends a push message to all devices. Devices will receive that push and based on the message, devices will download the data from server. (I think this is better approach than continuous hitting service for some particular intervals like polling)
For receiving only updated data from server, server maintains modified_timestamp column for all tables. First time devices will send empty timestamp, so that server sends all data to the device with server timestamp. Device receives the new data and updates local db and saves the latest server timestamp. For next time to get server updates, device will send stored server timestamp then server will send only modified data after that timestamp only. For each response server sends server timestamp, devices needs to store that timestamp and needs to use while calling service.
How to identify local non-synced data:
For sending local updates, local db needs to maintain one 'isSynced' column in tables. If any row modified in local isSynced will be false, after successful syncing local data to server isSynced will be true. so that we can handle local data up to date with server.
Updated:
You can find more information on this developer link
Have you considered using commercial solution?
http://www.mobeelizer.com/ seems like what you want to achieve. There are probably many other.
Note: no affiliation with that company.
I would say that the problem statement is incomplete. In above described setup what is missing is what actually you are going to synchronise.
Usual case in POS is that there exist few indices (id,value,...) tables that shall be distributed from central server to the client devices. In most cases it is price list, stock list, etc. Those tables should rarely be modified on client devices (actually could but then has to be redistributed from central server and acknowledged by client devices).
The other direction tend to be also pretty straightforward on client device you generate bills or invoices. These are again local stuff that shall be propagated towards server. Thus you actually store them locally and at sync point dispatch them to the server. Later on you might receive your own items back from server as an acknowledge.
EDIT: to detect changes, on-write timestamps as mentioned above is a must.
So far above described is the data flow.
Next you have to move into solution domain and implement these rules. There is couple of sync approaches (i.e.SyncML). On the other hand keeping it simple rulez. Thus the main concern should be some kind of locking and queueing that makes the thing robust.
It could also use the agent based client, in such case each device has it own agent (could be replica of last known state of the device db) but I would consider this as an advanced feature that might come in future release:-)
I am also working on the sales app in which i have to my local goals to server and server goals to my local goals
My proceder is that when ever my app is started i get the latest data from my server of my all my member and update my local data base with this data and when ever i change data in my local data base also update on sever side
also i used a sync button which will fetch latest data from the server if my team member changes its goal or profile
IF you want updated data on all the devices, why don't you use remote database only, why are you introducing local database for this.
For your case i will suggest you to work with only remote database directly so the things can be done real time.
I implemented an Android application that requires a set of data, taken by a SQL Server database. The application obtains the data calling a WS. After a first call to WS when the application start the first time, I need to maintain the data updated, according to the modify that may happens server-side (SQL server database).
For obtaining this result I perform, with a with a predefined frequency, a WS call, for knowing if data on database are changed. If new data are available, other web service is called for obtaining them.
This solution works fine for my ( I don't require real-time update). But, I think that this solution is too expensive in term of energy consumption, cpu consumption and network traffic.
Since, I immagine this is a very common problem I would know if exists a generic way to deal with it.
I suggest you to use extra fields. Add four colums to your local tables in Android :
TRANSACTING_FLAG : Set it to true when you are posting or updating this resource on the server
REQUEST_STATE : Set this flag to POSTING / UPDATING / DELETING etc.
RESULT_CODE : Set this field to the last result code received by the server for this particular resource.
TIMESTAMP : Period after wich data has to be updated
Workflow is simple :
When you retrieve data for your server just check if the last updated timestamp of your resource is superior to the cache timestamp you have defined before. If the timestamp is superior perform a request to update data. The transacting boolean let you know that a particular resource is actually synchronizing with the server. The result code lets you know if the request has failed or not and enventually retry it when the network is available. Doing this way you will maintain the persitence between your local and remote database because at any moment you can check the "synchronized state" of any local resource thanks to extra fields seen before.
I made a library to handle this. Take a look to RESTDroid. Even if the cache functionnality is not handles, you will be able to simply add it.
What you do is ok for most cases. You can take advantage of Google Cloud Messaging, but it needs time and effort to get implemented. I would stay with your solution.
You could look into Query Notifications, using something like SqlDependency - http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldependency.aspx. You can use this to track whether the results of a query change and to inform an application when this happens.
There are restrictions on the query you can use, and cost on the server is similar to an indexed view. You need .NET for this, by the way. If implemented in your Web Service, you would have to implement some kind of subscribe feature for your android, so that notifications could be pushed to it.
Another option to reduce the cost of checking for changes could be SQL Server Change Tracking - http://msdn.microsoft.com/en-us/library/bb933875.aspx
The android app stores the data in SQLite database during the offline mode. When online I want the app to sync (in both direction) with the datastore(database) in cloud server (App Engine). How do I implement this functionality, so that I can show the data captured on phone on a web application. Also please suggest any simple alternative way if any..
I just write out my data as a String (using a format I can reconstruct my data with), pass that to AE, parse it and store/display it.
You could use json too.
or try http://mylifewithandroid.blogspot.jp/2010/10/client-server-communication-with-json.html
for the sync part I use a timestamp. If the timestamped result isn't recorded on the server, I record it. I send back the recorded stamps to the client and delete them from the store. Of course the server can also send back new results if a user's records were updated from a different client.
In general, you should implement some complex algorithm, that will be doing synchronization depending on your needs, and then make it in code on both side (server and client). This not quite a simple task, in general. Useful keywords for googling: SOAP, REST, JSON, ...
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I want to sync data (such as db record, media) between an Android App and a Server. If you've seen Evernote or similar Applications, you certainly understand what I mean.
I have some question (imagine we want to sync DB records):
Every user has a part of server space for himself (such as Evernote or Dropbox). Maybe the user creates new records by cellphone and creates new records in server. How can I match these records together? If there are records with same ID What algorithms do you suggest me?
Except JSON, Are there any way for send data between cellphone device and server?
If SyncAdapter and ContentProvider can solve my problems, please explain exactly for me. (If you could offer some samples or tutorials to me OR Any advice or keywords to help broaden/guide my search would be appreciated as well).
I'll try to answer all your questions by addressing the larger question: How can I sync data between a webserver and an android app?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database. A decent tutorial for a content provider can be found here: http://thinkandroid.wordpress.com/2010/01/13/writing-your-own-contentprovider/
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: https://github.com/google/gson, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data.
If we think about today, accepted answer is too old. As we know that we have many new libraries which can help you to make this types of application.
You should learn following topics that will helps you surely:
SyncAdapter: The sync adapter component in your app encapsulates the code for the tasks that transfer data between the device and a server. Based on the scheduling and triggers you provide in your app, the sync adapter framework runs the code in the sync adapter component.
Realm: Realm is a mobile database: a replacement for SQLite & Core Data.
Retrofit Type-safe HTTP client for Android and Java by Square, Inc. Must Learn a-smart-way-to-use-retrofit
And your sync logic for database like: How to sync SQLite database on Android phone with MySQL database on server?
Best Luck to all new learner. :)
If you write this yourself these are some of the points to keep in mind
Proper authentication between the device and the Sync Server
A sync protocol between the device and the server. It will usually go in 3 phases, authentication, data exchange, status exchange (which operations worked and which failed)
Pick your payload format. I suggest SyncML based XML mixed with JSON based format to represent the actual data. So SyncML for the protocol, and JSON for the actual data being exchanged. Using JSON Array while manipulating the data is always preferred as it is easy to access data using JSON Array.
Keeping track of data changes on both client and server. You can maintain a changelog of ids that change and pick them up during a sync session. Also, clear the changelog as the objects are successfully synchronized. You can also use a boolean variable to confirm the synchronization status, i.e. last time of sync. It will be helpful for end users to identify the time when last sync is done.
Need to have a way to communicate from the server to the device to start a sync session as data changes on the server. You can use C2DM or write your own persistent tcp based communication. The tcp approach is a lot seamless
A way to replicate data changes across multiple devices
And last but not the least, a way to detect and handle conflicts
Hope this helps as a good starting point.
#Grantismo provides a great explanation on the overall. If you wish to know who people are actually doing this things i suggest you to take a look at how google did for the Google IO App of 2014 (it's always worth taking a deep look at the source code of these apps that they release. There's a lot to learn from there).
Here's a blog post about it: http://android-developers.blogspot.com.br/2014/09/conference-data-sync-gcm-google-io.html
Essentially, on the application side: GCM for signalling, Sync Adapter for data fetching and talking properly with Content Provider that will make things persistent (yeah, it isolates the DB from direct access from other parts of the app).
Also, if you wish to take a look at the 2015's code: https://github.com/google/iosched
For example, you want to sync table todoTable from MySql to Sqlite
First, create one column name version (type INT) in todoTable for both Sqlite and MySql
Second, create a table name database_version with one column name currentVersion(INT)
In MySql, when you add a new item to todoTable or update item, you must upgrade the version of this item by +1 and also upgrade the currentVersion
In Android, when you want to sync (by manual press sync button or a service run with period time):
You will send the request with the Sqlite currentVersion (currently it is 1) to server.
Then in server, you find what item in MySql have version value greater than Sqlite currentVersion(1) then response to Android (in this example the item 3 with version 2 will response to Android)
In SQLite, you will add or update new item to todoTable and upgrade the currentVersion
Look at parseplatform.org.
it's opensource project.
(As well as you can go for commercial package available at back4app.com.)
It is a very straight forward and user friendly server side database service that gives a great android client side API
one way to accomplish this to have a server side application that waits for the data. The data can be sent using HttpRequest objects in Java or you can write your own TCP/IP data transfer utility. Data can be sent using JSON format or any other format that you think is suitable. Also data can be encrypted before sending to server if it contains sensitive information. All Server application have to do is just wait for HttpRequests to come in and parse the data and store it anywhere you want.
I would suggest using a binary webservice protocol similar to Hessian. It works very well and they do have a android implementation. It might be a little heavy but depends on the application you are building. Hope this helps.
#Grantismo gives a great overview of Android sync components.
SyncManagerAndroid library provides a simple 2-way sync implementation to plug into the Android Sync framework (AbstractThreadedSyncAdapter.OnPerformSync).
https://github.com/sschendel/SyncManagerAndroid