I have an application in hand where we need to use a tab for data entry. The tab loads initial data from the remote server. Subsequently remote server needs to be updated, inserted (for new data) as the user inserts/updates data on the tab. Out database server is SQL Server 2008.
As suggested by the many experts at stackoverflow we are going to use Webservices at the server to facilitate data interchange. However I am still not sure of the following points -
1. whats the best mechanism for authentication in such case.
2. should i take a chunk of data from sqlite table at android, convert to JSON and pass it on to the Webservices for insert/update operation or take single row and update. Though I think sending single row would not be efficient.
3.How I manage failure to upgrade remote server. This is easier in case I use single row. My plan is to set status flag for sqlite records to 1 (default is 0) for records being updated/inserted to remote server. If update/insert fails I change the status flag back to 0 so that i can use them again next time. In case of success change the flag to 2.
thanks in advance
UPDATE
Doen some study and tried to use SampleSyncAdapter. Still some confusion about the whole operation. My Sqlite database is created by a program and content provider class exists in that application. Package name for the application is com.xylo.pds. I am trying to write a sync application which attempts to sync the data used in the first application. If I follow the SampleSyncAdapter sample - I need to develop server side application for authentication and then uploading android data to the server(in my case one way is sufficient). I can do that with the help of server side code given with the sample.
So I just copied codes of the sample code for my Authentication and Sync. My authenticator.xml has the existing entries-
android:contentAuthority="com.android.contacts"
android:accountType="com.example.android.samplesync
So now my application can add account and sync the contact. And no wonder it works with dummy server id given with the sample.
Now I need to put my own code in the application so that I can load my local database to the server. In order to that I need to add codes at onPerformSync of SyncAdapter. In order to use existing ContentProvider I have the following entries in the manifest file
<uses-permission android:name="com.xylo.pds.RCDataProvider" />. The application which defines the ContentProvider has the following entries -
<provider android:name=".RCDataProvider"
android:authorities="com.xylo.pds.provider"
android:exported="true"
android:readPermission="android.permission.permRead" />
Now if I have added a call to the contentresolver inside SyncAdapter keeping every thing else same just to check things are ok. So that, it is ok, I can change onPerformSync to add codes for uploading data. However now the application stops sysnc the contacts. What I am missing
Please enlighten me. Thanks
1) whats the best mechanism for authentication in such case.
You could/should use OAuth2. either implement your own token on web service website or use common OAuth2 web services in conjunction with the Android Account Manager.
The reason for suggesting this approach is really down to the suggested/recommended way of handling user authentication as per the Google docs.
See "Remembering your user" here http://developer.android.com/training/id-auth/identify.html
Which leads nicely on to your next questions
2) should i take a chunk of data from sqlite table at android, convert to JSON and pass it on to the Webservices for insert/update operation
or take single row and update. Though I think sending single row would
not be efficient.
You should use the android sync adapter which will make use of the account manager functionality described in the link I gave you in answer to question 1
You can code your android service in whatever way you wish but you should be using JSON rather than XML in both directions.
The really neat thing about using the account manager with a sync adapter is that your SQLite content provider methods can use the notifyChange method to tell the sync adapter to update the web service.
You can tell the sync adapter to get the latest data from your web service at the same time or you can schedule syncs.
3) How I manage failure to upgrade remote server. This is easier in case I use single row. My plan is to set status flag for sqlite
records to 1 (default is 0) for records being updated/inserted to
remote server. If update/insert fails I change the status flag back to
0 so that i can use them again next time. In case of success change
the flag to 2.
This is explained by Virgil in the Google I/O video embeded into into this sync adapter tutorial https://sites.google.com/site/andsamples/concept-of-syncadapter-androidcontentabstractthreadedsyncadapter
An alternative solution to using a sync adapter to get the data FROM your web service would be to use GCM (Google Cloud Messaging service. http://developer.android.com/google/gcm/gs.html
Basically Account Manager/sync adapter/gcm is the functionality that Android recommend you use and these are the services that Google uses itself for things like GMail and give your users the ability to keep their data intact even after clearing their data or uninstalling the app and re-installing the app and it also allows for a user to be able to install the app on a new phone and keeping their data.
Hope that helps
UPDATE in response to comments
You should always try hard to minimise traffic and size of data being sent in either direction. I would send the whole lot in one single JSON request gzipped.
Your web server should be able to automatically handle gzipped requests and if gzip is not installed on your server it's simple enough to add
A reference to creating a gzipped output stream from your app can be found here
http://developer.android.com/reference/java/util/zip/GZIPOutputStream.html
For the web server the solution you need will depend on the server you use and you should check with your host about gzip but here are a couple of links for the most popular web servers
Apache - http://howtounix.info/howto/Apache-gzip-compression-with-mod_deflate
NGinx Tutorial - http://www.howtoforge.com/how-to-save-traffic-with-nginxs-httpgzipmodule-debian-squeeze
GZip is the most popular solution for web servers and is very simple to implement.
I don't really have enough info to provide much more advice other than to say that I normally use Ruby on Rails for stuff like this and gzip is handled very simply with the ative support gzip library using something similar to this contacts = ActiveSupport::JSON.decode(gzipped_contacts.gsub("+", ""))
UPDATE 2 notifyChange()
Just to pick up on your point about notifyChange not being appropriate in the content provider due to there being no internet access.
It is fine to use notifyChange() in your content provider as it will tell the sync adapter to update as soon as it is appropriate to do so which means when the device is not too busy and as soon as an internet connection becomes available. That's the whole point of the sync adapter. Should you need to make use of notifyChange in your content provider for other services such as array adapters but you do not want the sync adapter to be told to update then there is a little documented boolean parameter that you can add to the end of the notifyChange params list. set it to false and the content provider will ignore the notifyChange
e.g. instead of the usual getContext().getContentResolver().notifyChange(uri, null);You can use getContext().getContentResolver().notifyChange(uri, null, false);
a) You could ask the users to sign in / sign up from the mobile apps, exactly as you would do it for a web site.
b) Take a look at this http://android-developers.blogspot.ro/2013/01/verifying-back-end-calls-from-android.html
Send more records in the same request, the idea is to make as few requests as possible.
I don't think you should keep the flag for error the same as the default value, it should be a different value so that you will be able to handle it more easily.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I want to sync data (such as db record, media) between an Android App and a Server. If you've seen Evernote or similar Applications, you certainly understand what I mean.
I have some question (imagine we want to sync DB records):
Every user has a part of server space for himself (such as Evernote or Dropbox). Maybe the user creates new records by cellphone and creates new records in server. How can I match these records together? If there are records with same ID What algorithms do you suggest me?
Except JSON, Are there any way for send data between cellphone device and server?
If SyncAdapter and ContentProvider can solve my problems, please explain exactly for me. (If you could offer some samples or tutorials to me OR Any advice or keywords to help broaden/guide my search would be appreciated as well).
I'll try to answer all your questions by addressing the larger question: How can I sync data between a webserver and an android app?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database. A decent tutorial for a content provider can be found here: http://thinkandroid.wordpress.com/2010/01/13/writing-your-own-contentprovider/
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: https://github.com/google/gson, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data.
If we think about today, accepted answer is too old. As we know that we have many new libraries which can help you to make this types of application.
You should learn following topics that will helps you surely:
SyncAdapter: The sync adapter component in your app encapsulates the code for the tasks that transfer data between the device and a server. Based on the scheduling and triggers you provide in your app, the sync adapter framework runs the code in the sync adapter component.
Realm: Realm is a mobile database: a replacement for SQLite & Core Data.
Retrofit Type-safe HTTP client for Android and Java by Square, Inc. Must Learn a-smart-way-to-use-retrofit
And your sync logic for database like: How to sync SQLite database on Android phone with MySQL database on server?
Best Luck to all new learner. :)
If you write this yourself these are some of the points to keep in mind
Proper authentication between the device and the Sync Server
A sync protocol between the device and the server. It will usually go in 3 phases, authentication, data exchange, status exchange (which operations worked and which failed)
Pick your payload format. I suggest SyncML based XML mixed with JSON based format to represent the actual data. So SyncML for the protocol, and JSON for the actual data being exchanged. Using JSON Array while manipulating the data is always preferred as it is easy to access data using JSON Array.
Keeping track of data changes on both client and server. You can maintain a changelog of ids that change and pick them up during a sync session. Also, clear the changelog as the objects are successfully synchronized. You can also use a boolean variable to confirm the synchronization status, i.e. last time of sync. It will be helpful for end users to identify the time when last sync is done.
Need to have a way to communicate from the server to the device to start a sync session as data changes on the server. You can use C2DM or write your own persistent tcp based communication. The tcp approach is a lot seamless
A way to replicate data changes across multiple devices
And last but not the least, a way to detect and handle conflicts
Hope this helps as a good starting point.
#Grantismo provides a great explanation on the overall. If you wish to know who people are actually doing this things i suggest you to take a look at how google did for the Google IO App of 2014 (it's always worth taking a deep look at the source code of these apps that they release. There's a lot to learn from there).
Here's a blog post about it: http://android-developers.blogspot.com.br/2014/09/conference-data-sync-gcm-google-io.html
Essentially, on the application side: GCM for signalling, Sync Adapter for data fetching and talking properly with Content Provider that will make things persistent (yeah, it isolates the DB from direct access from other parts of the app).
Also, if you wish to take a look at the 2015's code: https://github.com/google/iosched
For example, you want to sync table todoTable from MySql to Sqlite
First, create one column name version (type INT) in todoTable for both Sqlite and MySql
Second, create a table name database_version with one column name currentVersion(INT)
In MySql, when you add a new item to todoTable or update item, you must upgrade the version of this item by +1 and also upgrade the currentVersion
In Android, when you want to sync (by manual press sync button or a service run with period time):
You will send the request with the Sqlite currentVersion (currently it is 1) to server.
Then in server, you find what item in MySql have version value greater than Sqlite currentVersion(1) then response to Android (in this example the item 3 with version 2 will response to Android)
In SQLite, you will add or update new item to todoTable and upgrade the currentVersion
Look at parseplatform.org.
it's opensource project.
(As well as you can go for commercial package available at back4app.com.)
It is a very straight forward and user friendly server side database service that gives a great android client side API
one way to accomplish this to have a server side application that waits for the data. The data can be sent using HttpRequest objects in Java or you can write your own TCP/IP data transfer utility. Data can be sent using JSON format or any other format that you think is suitable. Also data can be encrypted before sending to server if it contains sensitive information. All Server application have to do is just wait for HttpRequests to come in and parse the data and store it anywhere you want.
I would suggest using a binary webservice protocol similar to Hessian. It works very well and they do have a android implementation. It might be a little heavy but depends on the application you are building. Hope this helps.
#Grantismo gives a great overview of Android sync components.
SyncManagerAndroid library provides a simple 2-way sync implementation to plug into the Android Sync framework (AbstractThreadedSyncAdapter.OnPerformSync).
https://github.com/sschendel/SyncManagerAndroid
I am working on an android fitness app and I want to get the hang of using ContentProviders. I was thinking about using the myfitnesspal api to get a list of exercises, but the api is private and my request has not been addressed yet. Then I considered scraping exercises from a website--but I am a little concerned about the reliability of this approach (if the site goes down, app won't keep working).
What is the best way to go about this? Is it "safe" to get information from a website (rather than an api) with a ContentProvider?
Correct me If I am wrong. You want user to acces your data even if there is not web API active. So my thought for it would be
Download all the data from web API and store it in database
create an node to check if data has been updated in the API if updated download new data in background and update the database else show same data from database
Benefits
User don't need to download data all the time they use app and their volume of internet would be saved
User don't need to see blank page if there's not any data
If im wrong, i would really like to know.. but the answer is simple: a ContentProvider allows only the connection to a database in your cellphone. You can't use a ContentProvider to get data from the internet. What you're trying to do is achievable with a WebService, which is an application running on a determined domain on the Internet and allows you to call some pre-defined methods linked to URLs of this same domain (but i imagine you know about that, right?).
I implemented an Android application that requires a set of data, taken by a SQL Server database. The application obtains the data calling a WS. After a first call to WS when the application start the first time, I need to maintain the data updated, according to the modify that may happens server-side (SQL server database).
For obtaining this result I perform, with a with a predefined frequency, a WS call, for knowing if data on database are changed. If new data are available, other web service is called for obtaining them.
This solution works fine for my ( I don't require real-time update). But, I think that this solution is too expensive in term of energy consumption, cpu consumption and network traffic.
Since, I immagine this is a very common problem I would know if exists a generic way to deal with it.
I suggest you to use extra fields. Add four colums to your local tables in Android :
TRANSACTING_FLAG : Set it to true when you are posting or updating this resource on the server
REQUEST_STATE : Set this flag to POSTING / UPDATING / DELETING etc.
RESULT_CODE : Set this field to the last result code received by the server for this particular resource.
TIMESTAMP : Period after wich data has to be updated
Workflow is simple :
When you retrieve data for your server just check if the last updated timestamp of your resource is superior to the cache timestamp you have defined before. If the timestamp is superior perform a request to update data. The transacting boolean let you know that a particular resource is actually synchronizing with the server. The result code lets you know if the request has failed or not and enventually retry it when the network is available. Doing this way you will maintain the persitence between your local and remote database because at any moment you can check the "synchronized state" of any local resource thanks to extra fields seen before.
I made a library to handle this. Take a look to RESTDroid. Even if the cache functionnality is not handles, you will be able to simply add it.
What you do is ok for most cases. You can take advantage of Google Cloud Messaging, but it needs time and effort to get implemented. I would stay with your solution.
You could look into Query Notifications, using something like SqlDependency - http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldependency.aspx. You can use this to track whether the results of a query change and to inform an application when this happens.
There are restrictions on the query you can use, and cost on the server is similar to an indexed view. You need .NET for this, by the way. If implemented in your Web Service, you would have to implement some kind of subscribe feature for your android, so that notifications could be pushed to it.
Another option to reduce the cost of checking for changes could be SQL Server Change Tracking - http://msdn.microsoft.com/en-us/library/bb933875.aspx
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I want to sync data (such as db record, media) between an Android App and a Server. If you've seen Evernote or similar Applications, you certainly understand what I mean.
I have some question (imagine we want to sync DB records):
Every user has a part of server space for himself (such as Evernote or Dropbox). Maybe the user creates new records by cellphone and creates new records in server. How can I match these records together? If there are records with same ID What algorithms do you suggest me?
Except JSON, Are there any way for send data between cellphone device and server?
If SyncAdapter and ContentProvider can solve my problems, please explain exactly for me. (If you could offer some samples or tutorials to me OR Any advice or keywords to help broaden/guide my search would be appreciated as well).
I'll try to answer all your questions by addressing the larger question: How can I sync data between a webserver and an android app?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database. A decent tutorial for a content provider can be found here: http://thinkandroid.wordpress.com/2010/01/13/writing-your-own-contentprovider/
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: https://github.com/google/gson, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data.
If we think about today, accepted answer is too old. As we know that we have many new libraries which can help you to make this types of application.
You should learn following topics that will helps you surely:
SyncAdapter: The sync adapter component in your app encapsulates the code for the tasks that transfer data between the device and a server. Based on the scheduling and triggers you provide in your app, the sync adapter framework runs the code in the sync adapter component.
Realm: Realm is a mobile database: a replacement for SQLite & Core Data.
Retrofit Type-safe HTTP client for Android and Java by Square, Inc. Must Learn a-smart-way-to-use-retrofit
And your sync logic for database like: How to sync SQLite database on Android phone with MySQL database on server?
Best Luck to all new learner. :)
If you write this yourself these are some of the points to keep in mind
Proper authentication between the device and the Sync Server
A sync protocol between the device and the server. It will usually go in 3 phases, authentication, data exchange, status exchange (which operations worked and which failed)
Pick your payload format. I suggest SyncML based XML mixed with JSON based format to represent the actual data. So SyncML for the protocol, and JSON for the actual data being exchanged. Using JSON Array while manipulating the data is always preferred as it is easy to access data using JSON Array.
Keeping track of data changes on both client and server. You can maintain a changelog of ids that change and pick them up during a sync session. Also, clear the changelog as the objects are successfully synchronized. You can also use a boolean variable to confirm the synchronization status, i.e. last time of sync. It will be helpful for end users to identify the time when last sync is done.
Need to have a way to communicate from the server to the device to start a sync session as data changes on the server. You can use C2DM or write your own persistent tcp based communication. The tcp approach is a lot seamless
A way to replicate data changes across multiple devices
And last but not the least, a way to detect and handle conflicts
Hope this helps as a good starting point.
#Grantismo provides a great explanation on the overall. If you wish to know who people are actually doing this things i suggest you to take a look at how google did for the Google IO App of 2014 (it's always worth taking a deep look at the source code of these apps that they release. There's a lot to learn from there).
Here's a blog post about it: http://android-developers.blogspot.com.br/2014/09/conference-data-sync-gcm-google-io.html
Essentially, on the application side: GCM for signalling, Sync Adapter for data fetching and talking properly with Content Provider that will make things persistent (yeah, it isolates the DB from direct access from other parts of the app).
Also, if you wish to take a look at the 2015's code: https://github.com/google/iosched
For example, you want to sync table todoTable from MySql to Sqlite
First, create one column name version (type INT) in todoTable for both Sqlite and MySql
Second, create a table name database_version with one column name currentVersion(INT)
In MySql, when you add a new item to todoTable or update item, you must upgrade the version of this item by +1 and also upgrade the currentVersion
In Android, when you want to sync (by manual press sync button or a service run with period time):
You will send the request with the Sqlite currentVersion (currently it is 1) to server.
Then in server, you find what item in MySql have version value greater than Sqlite currentVersion(1) then response to Android (in this example the item 3 with version 2 will response to Android)
In SQLite, you will add or update new item to todoTable and upgrade the currentVersion
Look at parseplatform.org.
it's opensource project.
(As well as you can go for commercial package available at back4app.com.)
It is a very straight forward and user friendly server side database service that gives a great android client side API
one way to accomplish this to have a server side application that waits for the data. The data can be sent using HttpRequest objects in Java or you can write your own TCP/IP data transfer utility. Data can be sent using JSON format or any other format that you think is suitable. Also data can be encrypted before sending to server if it contains sensitive information. All Server application have to do is just wait for HttpRequests to come in and parse the data and store it anywhere you want.
I would suggest using a binary webservice protocol similar to Hessian. It works very well and they do have a android implementation. It might be a little heavy but depends on the application you are building. Hope this helps.
#Grantismo gives a great overview of Android sync components.
SyncManagerAndroid library provides a simple 2-way sync implementation to plug into the Android Sync framework (AbstractThreadedSyncAdapter.OnPerformSync).
https://github.com/sschendel/SyncManagerAndroid
I'm trying to add a synchronization function to my app which will work like this :
I want it to be login free, so no new account will be needed.
The data will be assigned to the user's google account on which he's logged in.
On server side, there will be a mySQL database wich will hold the user's data.
I didn't find a useful tutorial. http://code.google.com/p/openmobster/wiki/AndroidSyncApp This one uses some weird servlet or whatever, and the official google tut is very briefly explained.
Could anyone help me ?
I could imagine synchronization through xml - like implementing a function that would create an xml file from my database, upload it to the server and parse it/put it my online database.
This would be useful, if I only wanted to be able to modify data on one end - the phone - but then it wouldn't be called syncing, rather backup.
Or maybe I could request a similar xml file from the server. I think it should be possible to send some kind of query from the device to the server which would call a function that creates an xml file from database entries and then download the created file, parse it, compare with the device's database and update the database if some file was updated (it's 'last edited' time was changed).
So on each sync cycle I would first get the server-side xml file, update the local DB, then create the xml file from local DB, send it to the server and update the online DB. Only files with newer "last edited" time would be updated.
What do you think ?
Just for the communication between App and Server, you could use XML or JSON. But if you also want to write to the Database and/or transfer files to/from the Server, you're going to need something with Sockets.
I have the same issue.
In my case I have two separate parts to the app, one is a genreal information feed, the other is secured user data.
I need them both to sync, but the general feed needs to sync even if the user has not logged in.
There are two things I'm going to try:
1) Add the sync but don't ask the AccountManager for the token. This might skip the whole login prompt.
2) USe a separate service that can wake up and sync the general data. (I suspect this is the correct way to go).