Develop Reference

asynchronous programming between android and mysql [duplicate]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I want to sync data (such as db record, media) between an Android App and a Server. If you've seen Evernote or similar Applications, you certainly understand what I mean.
I have some question (imagine we want to sync DB records):
Every user has a part of server space for himself (such as Evernote or Dropbox). Maybe the user creates new records by cellphone and creates new records in server. How can I match these records together? If there are records with same ID What algorithms do you suggest me?
Except JSON, Are there any way for send data between cellphone device and server?
If SyncAdapter and ContentProvider can solve my problems, please explain exactly for me. (If you could offer some samples or tutorials to me OR Any advice or keywords to help broaden/guide my search would be appreciated as well).

I'll try to answer all your questions by addressing the larger question: How can I sync data between a webserver and an android app?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database. A decent tutorial for a content provider can be found here: http://thinkandroid.wordpress.com/2010/01/13/writing-your-own-contentprovider/
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: https://github.com/google/gson, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data.

If we think about today, accepted answer is too old. As we know that we have many new libraries which can help you to make this types of application.
You should learn following topics that will helps you surely:
SyncAdapter: The sync adapter component in your app encapsulates the code for the tasks that transfer data between the device and a server. Based on the scheduling and triggers you provide in your app, the sync adapter framework runs the code in the sync adapter component.
Realm: Realm is a mobile database: a replacement for SQLite & Core Data.
Retrofit Type-safe HTTP client for Android and Java by Square, Inc. Must Learn a-smart-way-to-use-retrofit
And your sync logic for database like: How to sync SQLite database on Android phone with MySQL database on server?
Best Luck to all new learner. :)

If you write this yourself these are some of the points to keep in mind
Proper authentication between the device and the Sync Server
A sync protocol between the device and the server. It will usually go in 3 phases, authentication, data exchange, status exchange (which operations worked and which failed)
Pick your payload format. I suggest SyncML based XML mixed with JSON based format to represent the actual data. So SyncML for the protocol, and JSON for the actual data being exchanged. Using JSON Array while manipulating the data is always preferred as it is easy to access data using JSON Array.
Keeping track of data changes on both client and server. You can maintain a changelog of ids that change and pick them up during a sync session. Also, clear the changelog as the objects are successfully synchronized. You can also use a boolean variable to confirm the synchronization status, i.e. last time of sync. It will be helpful for end users to identify the time when last sync is done.
Need to have a way to communicate from the server to the device to start a sync session as data changes on the server. You can use C2DM or write your own persistent tcp based communication. The tcp approach is a lot seamless
A way to replicate data changes across multiple devices
And last but not the least, a way to detect and handle conflicts
Hope this helps as a good starting point.

#Grantismo provides a great explanation on the overall. If you wish to know who people are actually doing this things i suggest you to take a look at how google did for the Google IO App of 2014 (it's always worth taking a deep look at the source code of these apps that they release. There's a lot to learn from there).
Here's a blog post about it: http://android-developers.blogspot.com.br/2014/09/conference-data-sync-gcm-google-io.html
Essentially, on the application side: GCM for signalling, Sync Adapter for data fetching and talking properly with Content Provider that will make things persistent (yeah, it isolates the DB from direct access from other parts of the app).
Also, if you wish to take a look at the 2015's code: https://github.com/google/iosched

For example, you want to sync table todoTable from MySql to Sqlite
First, create one column name version (type INT) in todoTable for both Sqlite and MySql
Second, create a table name database_version with one column name currentVersion(INT)
In MySql, when you add a new item to todoTable or update item, you must upgrade the version of this item by +1 and also upgrade the currentVersion
In Android, when you want to sync (by manual press sync button or a service run with period time):
You will send the request with the Sqlite currentVersion (currently it is 1) to server.
Then in server, you find what item in MySql have version value greater than Sqlite currentVersion(1) then response to Android (in this example the item 3 with version 2 will response to Android)
In SQLite, you will add or update new item to todoTable and upgrade the currentVersion

Look at parseplatform.org.
it's opensource project.
(As well as you can go for commercial package available at back4app.com.)
It is a very straight forward and user friendly server side database service that gives a great android client side API

one way to accomplish this to have a server side application that waits for the data. The data can be sent using HttpRequest objects in Java or you can write your own TCP/IP data transfer utility. Data can be sent using JSON format or any other format that you think is suitable. Also data can be encrypted before sending to server if it contains sensitive information. All Server application have to do is just wait for HttpRequests to come in and parse the data and store it anywhere you want.

I would suggest using a binary webservice protocol similar to Hessian. It works very well and they do have a android implementation. It might be a little heavy but depends on the application you are building. Hope this helps.

#Grantismo gives a great overview of Android sync components.
SyncManagerAndroid library provides a simple 2-way sync implementation to plug into the Android Sync framework (AbstractThreadedSyncAdapter.OnPerformSync).
https://github.com/sschendel/SyncManagerAndroid

How to synchronize SQLite and MySQL databases using push notifications?

I have an SQLite database on Android and a MySQL database on a server. I want to synchronize these databases when a user edits data on their phone or edits data on a website.
I know how to update the MySQL database on the server when a user makes changes on their phone but I don't know how to update the Android database when a user makes changes on the website.
I have read into push notification and believe this to be a good path to follow but I have a few questions about it:
When a user updates data through a website it will send a push notification to that user's phone saying changes have been made. Can this push notification trigger to update the Android's database with the new changes made on the Server database?
What if a user turns off push notifications? Will I still be able to trigger for their Android database to be updated?
I have also read up on SQLite and MySQL database synchronization and found this post SQLite and MySQL sync but did not find the post helpful for my situation.
Are push notifications a good way to go or should I be using a different approach?
In a nutshell - I want a way for the Android device to detect changes on the MySQL database and update its SQLite database without the user initiating the synchronization.

I'm afraid I've not used push notifications. But a solution might be: You could create an early method call to an Asynchronous polling event from the launcher onCreate() that looks up the server to see if any changes have been registered (though an API of some sort) in the MySQL, and then update the SQLite that way? Since it's the first thing that happens on launch, technically the user isn't initiating it. Granted this won't update during use of the app, unless you repeat poll at regular intervals?

Token based pagination approach.
Assumptions: or calls you need to take
One of the databases will the source of truth, in case of differences in the two, which data is true, and which will be overwritten? - assuming remote database is source of truth
What's the frequency of data changes? - assuming its not realtime critical
How much stale data are we OK with dealing on the app. - assuming we're OK with a few minutes of difference
How to ensure data is consistent
Find a method to associate a token, which can be used to identify till which record is the data in sync. This is important no matter how assured you are of web requests, they will fail. So the best method is to send the last token that have stored, and the web endpoint will return data from that token to the latest value.
If the volume of data is too much here, sending chunks > sending all of it. Chunks work the same way, based on tokens.
These tokens can be simple PK auto increment column as well.
How to deal with time difference, stale data
If your application demands some data to be near realtime, better to categorize data based on a fiew screens, and whenever the user comes to the said screen, send a request in background to fetch related data columns only. Will ensure the data stays in sync w.r.t to the important columns. This is a classic push/pull approach. Can also be done on the splash screen itself.
as a rule of thumb, if you need something urgent, pull.
if it can wait, wait for a push.
Push notifications are OK as far as:
they should be silent.
there'a a limit on the number of push notifications that you can send
have costs associated
what's the fail - check mechanism? What if the requests fail?

How do you handle stale cache records in mobile app

I am in the process of creating an android app (my first) that consumes a REST API.
I use a background job to fetch content and I plan to use a GET request with a from_id parameter in order to get more content. Of course anything fetched from the API gets stored in the SQLite db (I am using greendao) and the app only uses data that is already present there, in order to be snappy.
So, the question is: What happens if a given record is updated on the server? If records once read are cached, how come the app will notice that there are changes to sync? Which strategies are feasible solutions?
Thanks.
EDIT:
As Satish P points out in his answer, the client-server communication is handled with ETag (and I must add the possibility of using If-Modified-Since).
But my main concern, is how to mix this with the app UI. Given this example:
A list of elements, which have been retrieved from the REST service but client-side are read from the local database to make the app more responsive.
User clicks in one of those elements and a detailed view is show. Again, the data is loaded from the local database. I guess that at this point a GET request for the specific record is requested, either with ETag or If-Modified-Since headers.
It happens that the server returns a modified record, thus the local data is modified, so now it's time to update whatever the user is seeing.
Problem: If the detailed view is already populated because the local database read was already done when the remote request returns, how can I update the view? I don't think that just replacing current data with the fresher one is acceptable, the user would see a change out of the blue.

Satish's answer is absolutely right in terms of what you need your server to do. The gist is that it needs to support ETags and 304 response codes in case the content hasn't changed since the last time you got it from the server. On the client side now, there are essentially three strategies you can follow (each with it's own pros and cons):
Only use the cache if the content hasn't changed. That means you will always do a request and will display a progress bar to the user. If the server returns 304, then your content hasn't changed, and the request will be pretty fast (the moment you see that, you display the cached content). If the server actually returns new content, you continue showing the progress bar, and when the content is loaded you display the new content. The good thing about this is that the user will only ever see valid content, therefore avoiding a lot of headaches on your part. The bad thing is that the app does not appear that fast (especially if the content has changed and you are in a very slow connection).
Use only the cache for a predefined period and then fallback to first case. There are a couple of cache header to define that period ('max-age' and 'Expires'). Before that period you always use the cache (without doing a request), and after that you do a request and see if the content has changed. The good thing about this method is that for during the period mentioned above, the app is really fast. The bad thing is that there is a possibility that the user is looking at incorrect content.
Use both the cache and the network for a predefined period, and then fallback to the first case. You can use the cache headers mentioned earlier in a different way. Instead of only showing the cached content, you can actually display the cached content AND do a request in the background. If that request comes back with a 304, fine, else you will have to update you UI with the new data (expect two responses, one with the cached data and one with the newly retrieved data). The positive with this is that you get both a fast experience and valid data (most of the time). The negative is that you add a lot of complexity to your app (what happens if the user interacts with the stale data, and then a second response comes in etc).
All in all, every strategy is valid depending on the use case. For example, if the user can't interact with the screen that displays the data (like a tv program), the third option is pretty good. If it is crucial that the user sees correct data (a financial app let's say), then the first option is best. If speed is more important than having the latest data (a game or something) then the second option is your best choice.

How efficient the client can do caching is solely dependent on how much support you get from the REST API your client is accessing.
Using ETag is the industry standard to make caching on the client side more efficient and also server to serve the request faster. In short ETag is LIKE an MD5 hash of the content returned. More about ETag here: http://en.wikipedia.org/wiki/HTTP_ETag
If it is a popular API like Google, Facebook etc they inherently support ETags.
Please look at links below:
ETag usage best explained here: https://developers.facebook.com/docs/reference/ads-api/etags-reference
When the client does a GET on the a particular resource, the server when responding back with the content should include an ETag.
The client should store the ETag for that resource against the data cached.
Whenever the client is using the cache information, it should verify the cache using the ETag. Can work in multiple ways depending on the service implementation again
Make a usual GET on the resource and include the ETag as part of the request. If the content did not change the service will ideally no return any data but will give a specific code like (304 - Not Modified). Client knows that the cache is still valid and continues to use it
Make a HEAD call on the resource and the ETag is returned. It is part of the standard HTTP Specification. http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html (Refer 9.4). In this case Client will verify the ETag and decide whether or not to make the GET call.
Sample of a resource in the above explaination is like below
GET http://serverapi.com/employees/2312312312

The screen update could be handled fairly gracefully by Javascript, rewriting specific elements in the DOM -- optionally applying CSS formatting to call attention to the change in the UI -- if each datum in the UI has a unique ID/container ID or can otherwise be targeted by JS.
A simple/istic way to defeat some caching is to append a query string to the resource you want. For example, a file named testfile.csv can be accessed as easily at testfile.csv?12345 -- and the next time you want to bypass the cache, just update the query string, e.g., testfile.csv?23456. If updating a query string manually is arduous in your context, get a bit more clever at the cost of a modest hit in performance via PHP: call on the resource as testfile.csv to cause the query string to auto-update on every query after the resource is modified; the updated version gets served instead of the cached one.

Lately what I've been doing is using GraphQL with Apollo. It handles all this stuff automatically which is plain awesome.

Sync data between client and server

I have mobile app. Something like to do list or calendar. Teoretically user can have a few devices with that application on a defferent platforms and so on. I would like to create a automatic synchronization between them through a own server. What is the best practice: update all the information or only the changes? On the one hand usually there is no a lot of data when it's about a to do list but who knows?

The correct approach is not date/time as others suggest, as time can go out of sync. The right algorithm is to keep the checksum of the data entries during last synchronization. On next synchronization you compare current checksums with stored ones, then you know whether the entry has been changed on the server, on the client or both.
Our open-source Rethync SDK lets you implement the above approach quite easily and is available for Android (not for iOS at the moment).

I am doing something similar in my application. I have a last modified date field with each entity that I need to sync. Then periodically, I post this data to the server (actual data + date and time). Now the server can do one of two things. It will check the corresponding data on server side and compare the last modified date. If what the server is latest, it will return the latest data in response. If not, it will update its data and send a response indicating what client has is latest.
Of course you can do several optimization. That is, mark the data as "dirty" so you know whether to even send your data to server. If the phone does not have modified data, your sync is basically getting the latest data from server.
Basically server does the heavy lifting and does all the logic necessary to maintain the latest data on its end and send responses to client appropriately.
Good Luck

Best approach is use a time stamp to handle this.
Initial request to server with time stamp value 0.
Server will give the all the data first time with Time-stamp.
Store the Time stamp to sharedpreferences.
In All next request pass the time stamp back to the server
Server will send only those data which are add/update/ after that
given time stamp
That is it.

There is a new alternative to the syncing problem. It's called EnduroSync from Orando Labs. You can sync object data stores between devices on Android and iOS now, with others coming soon.
Full Disclosure: I work for Orando Labs.
The EnduroSync clients allow you to create object data stores on the local devices. The clients are fairly sophisticated - data is modeled as native objects for each client we support (iOS and Android now, more coming). The clients work offline and online. The data is saved to an sqlite database locally.
As you change objects in your model, the deltas are recorded on the device. At some point, you can 'sync' the object data store. Syncing uses a commit/push/pull process (like git), but this is invisible to you. The sync brings your local copy up to date with whatever is on the server, and sends up any changes you have made. Conflicts are resolved using a timestamp based merge, so newer data is not overwritten by older data.
EnduroSync is an online service, so there is no server setup on your end.
There is also a flexible permission system which lets you share the object data stores in a variety of ways. For instance, most applications will have one or more object data stores for each user, for preferences, notes, tags, etc. You can also share object data stores per app, per user type, and with wild cards, many other ways.
So basically you use our client SDK's to model your data on the device. Modeling is with simple objects in the native programming language of the device. If you sign up for the syncing service, you get the syncing also.

Here is another approach.
Issue :I need to have the appointments of doctors syned to client (mobile device) from the server. Now the appointments can drop off or the data could possibly change on the server. Having the client to know what change and sending a request back to server could be an expensive service.
Possible approach : Have the server do the heavy lifting. Keep a table which stores values of time stamp and if a change happened with regard to an appointment - cancellation / reschedule etc. The client would then look at this table to see if anything changed. In reality we don't need to sync anything but only the delta which server can provide to the client based on what it has and what is at Client. There is one aspect which needs to be taken care of is updation of info from client to server and traditional conflict management can be done where client can update the server when a data connectivity between client and server exists.
Essentially the approach is to have only the deltas synced by maintaining a checksum or data change log to PUSH changes to the client.

Few issues regarding accessing remote SQL Server from Android

I have an application in hand where we need to use a tab for data entry. The tab loads initial data from the remote server. Subsequently remote server needs to be updated, inserted (for new data) as the user inserts/updates data on the tab. Out database server is SQL Server 2008.
As suggested by the many experts at stackoverflow we are going to use Webservices at the server to facilitate data interchange. However I am still not sure of the following points -
1. whats the best mechanism for authentication in such case.
2. should i take a chunk of data from sqlite table at android, convert to JSON and pass it on to the Webservices for insert/update operation or take single row and update. Though I think sending single row would not be efficient.
3.How I manage failure to upgrade remote server. This is easier in case I use single row. My plan is to set status flag for sqlite records to 1 (default is 0) for records being updated/inserted to remote server. If update/insert fails I change the status flag back to 0 so that i can use them again next time. In case of success change the flag to 2.
thanks in advance
UPDATE
Doen some study and tried to use SampleSyncAdapter. Still some confusion about the whole operation. My Sqlite database is created by a program and content provider class exists in that application. Package name for the application is com.xylo.pds. I am trying to write a sync application which attempts to sync the data used in the first application. If I follow the SampleSyncAdapter sample - I need to develop server side application for authentication and then uploading android data to the server(in my case one way is sufficient). I can do that with the help of server side code given with the sample.
So I just copied codes of the sample code for my Authentication and Sync. My authenticator.xml has the existing entries-
android:contentAuthority="com.android.contacts"
android:accountType="com.example.android.samplesync
So now my application can add account and sync the contact. And no wonder it works with dummy server id given with the sample.
Now I need to put my own code in the application so that I can load my local database to the server. In order to that I need to add codes at onPerformSync of SyncAdapter. In order to use existing ContentProvider I have the following entries in the manifest file
<uses-permission android:name="com.xylo.pds.RCDataProvider" />. The application which defines the ContentProvider has the following entries -
<provider android:name=".RCDataProvider"
android:authorities="com.xylo.pds.provider"
android:exported="true"
android:readPermission="android.permission.permRead" />
Now if I have added a call to the contentresolver inside SyncAdapter keeping every thing else same just to check things are ok. So that, it is ok, I can change onPerformSync to add codes for uploading data. However now the application stops sysnc the contacts. What I am missing
Please enlighten me. Thanks

1) whats the best mechanism for authentication in such case.
You could/should use OAuth2. either implement your own token on web service website or use common OAuth2 web services in conjunction with the Android Account Manager.
The reason for suggesting this approach is really down to the suggested/recommended way of handling user authentication as per the Google docs.
See "Remembering your user" here http://developer.android.com/training/id-auth/identify.html
Which leads nicely on to your next questions
2) should i take a chunk of data from sqlite table at android, convert to JSON and pass it on to the Webservices for insert/update operation
or take single row and update. Though I think sending single row would
not be efficient.
You should use the android sync adapter which will make use of the account manager functionality described in the link I gave you in answer to question 1
You can code your android service in whatever way you wish but you should be using JSON rather than XML in both directions.
The really neat thing about using the account manager with a sync adapter is that your SQLite content provider methods can use the notifyChange method to tell the sync adapter to update the web service.
You can tell the sync adapter to get the latest data from your web service at the same time or you can schedule syncs.
3) How I manage failure to upgrade remote server. This is easier in case I use single row. My plan is to set status flag for sqlite
records to 1 (default is 0) for records being updated/inserted to
remote server. If update/insert fails I change the status flag back to
0 so that i can use them again next time. In case of success change
the flag to 2.
This is explained by Virgil in the Google I/O video embeded into into this sync adapter tutorial https://sites.google.com/site/andsamples/concept-of-syncadapter-androidcontentabstractthreadedsyncadapter
An alternative solution to using a sync adapter to get the data FROM your web service would be to use GCM (Google Cloud Messaging service. http://developer.android.com/google/gcm/gs.html
Basically Account Manager/sync adapter/gcm is the functionality that Android recommend you use and these are the services that Google uses itself for things like GMail and give your users the ability to keep their data intact even after clearing their data or uninstalling the app and re-installing the app and it also allows for a user to be able to install the app on a new phone and keeping their data.
Hope that helps
UPDATE in response to comments
You should always try hard to minimise traffic and size of data being sent in either direction. I would send the whole lot in one single JSON request gzipped.
Your web server should be able to automatically handle gzipped requests and if gzip is not installed on your server it's simple enough to add
A reference to creating a gzipped output stream from your app can be found here
http://developer.android.com/reference/java/util/zip/GZIPOutputStream.html
For the web server the solution you need will depend on the server you use and you should check with your host about gzip but here are a couple of links for the most popular web servers
Apache - http://howtounix.info/howto/Apache-gzip-compression-with-mod_deflate
NGinx Tutorial - http://www.howtoforge.com/how-to-save-traffic-with-nginxs-httpgzipmodule-debian-squeeze
GZip is the most popular solution for web servers and is very simple to implement.
I don't really have enough info to provide much more advice other than to say that I normally use Ruby on Rails for stuff like this and gzip is handled very simply with the ative support gzip library using something similar to this contacts = ActiveSupport::JSON.decode(gzipped_contacts.gsub("+", ""))
UPDATE 2 notifyChange()
Just to pick up on your point about notifyChange not being appropriate in the content provider due to there being no internet access.
It is fine to use notifyChange() in your content provider as it will tell the sync adapter to update as soon as it is appropriate to do so which means when the device is not too busy and as soon as an internet connection becomes available. That's the whole point of the sync adapter. Should you need to make use of notifyChange in your content provider for other services such as array adapters but you do not want the sync adapter to be told to update then there is a little documented boolean parameter that you can add to the end of the notifyChange params list. set it to false and the content provider will ignore the notifyChange
e.g. instead of the usual getContext().getContentResolver().notifyChange(uri, null);You can use getContext().getContentResolver().notifyChange(uri, null, false);

a) You could ask the users to sign in / sign up from the mobile apps, exactly as you would do it for a web site.
b) Take a look at this http://android-developers.blogspot.ro/2013/01/verifying-back-end-calls-from-android.html
Send more records in the same request, the idea is to make as few requests as possible.
I don't think you should keep the flag for error the same as the default value, it should be a different value so that you will be able to handle it more easily.