I know that this question has been asked too many times, but I think the issues I'm trying to target are a little bit different, maybe more complicated.
I am going to develop an application that uses a RESTful Web Service and needs to have the following requirements:
the app should show some books, their authors and their editors in lists and in detail
the app should also allow searching for a book
books, authors and editors are fetched from a RESTful web service
every entity has to be cached so that when I open an Activity I see the old data first (if any), while the new one updates from the network.
every time an entity is updating, the interested parties should be notified (ContentObserver? A regular Listener implementation?)
if a call is already executing (say to api/books/1337 or to api/editors) the caller should be notified that it is loading data and should be given the old one (if it exists), as if it was the original caller.
some data (only books and authors) should be updated every N minutes (decided by the user) and the observers should be notified (SyncAdapter?)
Questions:
After watching and studying all of the components proposed by Virgil Dobjanschi at Google I/O 2010 here are my doubts:
How can I transparently handle the "entity-is-updating" concept for any caller? Should I use ContentObserver on a ContentProvider I will have to implement?
If I use a ContentObserver I can easily set a status-flag for the single entity (as suggested by Dobjanschi), for example UPDATING, INSERTING, and so on. But how should I handle list? Say I want a list of books, where should I put the status flag? Should I put it in a status table for lists only? If so, I should observe two Cursors, one for the status and one for the actual list (i.e., the table/Content URI). And what if the entity I'm asking for does not exists (yet) or the REST call returns a 404? How do I handle the callback?
If I put all of my REST methods in a **SyncAdapter**, can I "force" the SyncAdapter to update an entity/entity list from the network (and therefore put it into the proper table)? This way, the status flag would be useful.
Can the SyncAdapter work on multiple entities (actually, entity lists, as I want to update books and editors every now and then), since it only has a performSync method?
If my SyncAdapter implementation has been disabled by the user in the device settings it won't update anything (and that's fine). But if the user clicks on an "update books" button in an Activity, can I still call the performSync method, or will it be disabled as well?
SyncAdapter is a design pattern involving five components:
An app. This uses a set of Activity along with Cursor and ContentObserver and maybe CursorAdapter and some to provide a UI to the locally stored data from a ContentProvider.
ContentProvider The local device's data-store. Handles CRUD calls, handles notifying SyncAdapter of the need to push an update to the server.
Account The user identity on the remote server.
SyncAdapter A background process which runs, and keeps the local datastore in sync with the server.
The server itself.
So. To the questions:
"Is-updating" means, "has local changes which have not yet been pushed to the server. It's a flag you set on a row in your database. It's set in ContentProvider when you Create/Update/Delete a row. When SyncAdapter runs, it sees the flag, pushes the update to the server, clears the flag. The flag itself does two things:
a. Tells the user the app is busy saving the change, and when that's done.
b. Marks the row as changed so SyncAdapter knows to push it to the server.
Read here for more details.
If you're not syncing the entire catalog, then your client will directly query the server and cache the results by putting them into the ContentProvider. There is no status flag there, since they're coming from the server and therefore match the server state. Write your SyncAdapter to ignore them, or perhaps discard them after they've been cached a few days.
a. To ensure your local updates get sent to the server, you write your ContentProvider to notify the SyncAdapter during the ContentProvider's Create/Update/Delete calls. (Read here...)
b. To ensure you get updates from the server periodically, you configure the account for automatic sync. (Read Here...)
Yes. performSync is just a function call. Write it to do what you want. Have it fetch table 1 from the server and put it into one table in your ContentProvider. Then have it fetch table 2, and put it into a different table. Etc.
a. You can force a sync by calling ContentResolver.RequestSync() with ContentResolver.SYNC_EXTRAS_MANUAL in the extras bundle.
b. You can manually fetch something with client code and directly push it into the ContentProvider.
If you take a look at Running Sync Adapter section in Google Developers Training, you will find a whole detailed description of how to:
Run the Sync Adapter When Server Data Changes
Run the Sync Adapter When Content Provider Data Changes
Run the Sync Adapter After a Network Message
Run the Sync Adapter Periodically
Run the Sync Adapter On Demand
Related
I have a question about real time notification in MySQL.
I want that when there is a change in a specific table of the MySQL db, the db notify my android app that do something.
Is it possibile to do? I have listen that trigger can help, but i didn't found anything about that in web, especially for android.
I'm creating a real-time chat app for an android project at the university. It is required to use a servlet in the project, so my servlet makes requests to the database and sends the data to the app. To do this the app makes a GET request to the servlet every second to get the new messages. I would like to avoid making these requests and updating the message list only when there is a change in the database. I know it can be done with firebase for example, but having to use the servlet I would like to know if there is a way to notify the app when this change occurs.
The messages are insert in the db with a POST request to the servlet that update db; information about messages are taken from an edit text in the app and with a send button i make the request, so the thing that i have to do is to push the android app when new messages are detected in the db.
Presumably you have some code component sending commands to the database. If yes, then that is where you could add code to post notifications to your app via websockets or something similar. Triggers in your database come at a cost and should be avoided if there is going to be a lot of them required.
As already mentioned, the app constantly polling, looking for a change is also expensive and probably best avoided.
'Is it possible?' questions are always difficult to answer. The answer is almost always 'yes', but 'is it advisable', 'is it a good idea', 'is it recommended', that is where it gets more complicated. I don't know of any way that a MySQL database change could notify individual Android app installations after a change. However, a running instance of an Android app could poll the MySQL database and look for the structure of the table, and if the structure of that table differed from the last time it polled, it could do something.
This would have a large overhead, in having the app continuously polling the MySQL database to check the structure of the table. That would add a lot of demands on your MySQL installation, eat up a lot of the app user's bandwidth and data plan.
So, 'is it possible', yes. 'is it recommended', no.
One question would be, what is the scenario where you think the table structure might change, and how often might that occur? What many people do, is check to see a database's table structures only once, when the app starts up a new session. If a change is detected on start-up, it could make the necessary local changes to its logic, then, assume the structure will remain constant through the rest of the session. This would eliminate the constant polling and other negative behaviors I mentioned before.
I have an SQLite database on Android and a MySQL database on a server. I want to synchronize these databases when a user edits data on their phone or edits data on a website.
I know how to update the MySQL database on the server when a user makes changes on their phone but I don't know how to update the Android database when a user makes changes on the website.
I have read into push notification and believe this to be a good path to follow but I have a few questions about it:
When a user updates data through a website it will send a push notification to that user's phone saying changes have been made. Can this push notification trigger to update the Android's database with the new changes made on the Server database?
What if a user turns off push notifications? Will I still be able to trigger for their Android database to be updated?
I have also read up on SQLite and MySQL database synchronization and found this post SQLite and MySQL sync but did not find the post helpful for my situation.
Are push notifications a good way to go or should I be using a different approach?
In a nutshell - I want a way for the Android device to detect changes on the MySQL database and update its SQLite database without the user initiating the synchronization.
I'm afraid I've not used push notifications. But a solution might be: You could create an early method call to an Asynchronous polling event from the launcher onCreate() that looks up the server to see if any changes have been registered (though an API of some sort) in the MySQL, and then update the SQLite that way? Since it's the first thing that happens on launch, technically the user isn't initiating it. Granted this won't update during use of the app, unless you repeat poll at regular intervals?
Token based pagination approach.
Assumptions: or calls you need to take
One of the databases will the source of truth, in case of differences in the two, which data is true, and which will be overwritten? - assuming remote database is source of truth
What's the frequency of data changes? - assuming its not realtime critical
How much stale data are we OK with dealing on the app. - assuming we're OK with a few minutes of difference
How to ensure data is consistent
Find a method to associate a token, which can be used to identify till which record is the data in sync. This is important no matter how assured you are of web requests, they will fail. So the best method is to send the last token that have stored, and the web endpoint will return data from that token to the latest value.
If the volume of data is too much here, sending chunks > sending all of it. Chunks work the same way, based on tokens.
These tokens can be simple PK auto increment column as well.
How to deal with time difference, stale data
If your application demands some data to be near realtime, better to categorize data based on a fiew screens, and whenever the user comes to the said screen, send a request in background to fetch related data columns only. Will ensure the data stays in sync w.r.t to the important columns. This is a classic push/pull approach. Can also be done on the splash screen itself.
as a rule of thumb, if you need something urgent, pull.
if it can wait, wait for a push.
Push notifications are OK as far as:
they should be silent.
there'a a limit on the number of push notifications that you can send
have costs associated
what's the fail - check mechanism? What if the requests fail?
I am working on an application that will, basically, allow people to create, join and manage groups of other people. The people within the groups can also message each other.
I have been wondering which path would be better:
Keep a remote database with all the information, including messages sent to and from users. And have the app query the server every time it needs information. Even information it has seen before.
Keep a remote database with all the information, including messages sent to and from users. Also keep a local copy of the remote database and just keep it synced with the remote database. Whenever the app needs to query for information, it does a query to see if the local table is up to date. If it is not up to date, it updates the table and runs the query on the local table. This way it will keep a local copy and the app will have fast queries when there is not an update to the remote table.
What is generally done with mobile applications and remote databases?
Would it be "bad practice" if i just did number 1?
From my point of view, in most cases, the database in the mobile is just a cache of the real database, the one in the server. So, my suggestion will be to keep locally all data that you need syncing with the server. This allows you to show information even when no connection and show something to the user while the info is updated.
Also, this approach makes the local data volatile without risk, as it's stored in the server. So:
All info is in the server
With a background process (service, thread, intentservice, whatever best suits you) you sync this information with the local database
UI is always showing info from local database
Of course, this is a very general approach, and needs to be examined for each case as different situations may need different approaches.
My base response is that I would keep the data in one place and access it remotely unless there is a major reason to keep it locally. There would have to be extenuating circumstances to mandate that I keep a copy of the data locally. Just make sure your queries are accurate and concise. Don't pull over more data than you need to.However, you can have a subset of data kept locally. Items that are specific to the user (like messages), but keeping data that is not relevant just adds overhead and bloat.
I have an application in hand where we need to use a tab for data entry. The tab loads initial data from the remote server. Subsequently remote server needs to be updated, inserted (for new data) as the user inserts/updates data on the tab. Out database server is SQL Server 2008.
As suggested by the many experts at stackoverflow we are going to use Webservices at the server to facilitate data interchange. However I am still not sure of the following points -
1. whats the best mechanism for authentication in such case.
2. should i take a chunk of data from sqlite table at android, convert to JSON and pass it on to the Webservices for insert/update operation or take single row and update. Though I think sending single row would not be efficient.
3.How I manage failure to upgrade remote server. This is easier in case I use single row. My plan is to set status flag for sqlite records to 1 (default is 0) for records being updated/inserted to remote server. If update/insert fails I change the status flag back to 0 so that i can use them again next time. In case of success change the flag to 2.
thanks in advance
UPDATE
Doen some study and tried to use SampleSyncAdapter. Still some confusion about the whole operation. My Sqlite database is created by a program and content provider class exists in that application. Package name for the application is com.xylo.pds. I am trying to write a sync application which attempts to sync the data used in the first application. If I follow the SampleSyncAdapter sample - I need to develop server side application for authentication and then uploading android data to the server(in my case one way is sufficient). I can do that with the help of server side code given with the sample.
So I just copied codes of the sample code for my Authentication and Sync. My authenticator.xml has the existing entries-
android:contentAuthority="com.android.contacts"
android:accountType="com.example.android.samplesync
So now my application can add account and sync the contact. And no wonder it works with dummy server id given with the sample.
Now I need to put my own code in the application so that I can load my local database to the server. In order to that I need to add codes at onPerformSync of SyncAdapter. In order to use existing ContentProvider I have the following entries in the manifest file
<uses-permission android:name="com.xylo.pds.RCDataProvider" />. The application which defines the ContentProvider has the following entries -
<provider android:name=".RCDataProvider"
android:authorities="com.xylo.pds.provider"
android:exported="true"
android:readPermission="android.permission.permRead" />
Now if I have added a call to the contentresolver inside SyncAdapter keeping every thing else same just to check things are ok. So that, it is ok, I can change onPerformSync to add codes for uploading data. However now the application stops sysnc the contacts. What I am missing
Please enlighten me. Thanks
1) whats the best mechanism for authentication in such case.
You could/should use OAuth2. either implement your own token on web service website or use common OAuth2 web services in conjunction with the Android Account Manager.
The reason for suggesting this approach is really down to the suggested/recommended way of handling user authentication as per the Google docs.
See "Remembering your user" here http://developer.android.com/training/id-auth/identify.html
Which leads nicely on to your next questions
2) should i take a chunk of data from sqlite table at android, convert to JSON and pass it on to the Webservices for insert/update operation
or take single row and update. Though I think sending single row would
not be efficient.
You should use the android sync adapter which will make use of the account manager functionality described in the link I gave you in answer to question 1
You can code your android service in whatever way you wish but you should be using JSON rather than XML in both directions.
The really neat thing about using the account manager with a sync adapter is that your SQLite content provider methods can use the notifyChange method to tell the sync adapter to update the web service.
You can tell the sync adapter to get the latest data from your web service at the same time or you can schedule syncs.
3) How I manage failure to upgrade remote server. This is easier in case I use single row. My plan is to set status flag for sqlite
records to 1 (default is 0) for records being updated/inserted to
remote server. If update/insert fails I change the status flag back to
0 so that i can use them again next time. In case of success change
the flag to 2.
This is explained by Virgil in the Google I/O video embeded into into this sync adapter tutorial https://sites.google.com/site/andsamples/concept-of-syncadapter-androidcontentabstractthreadedsyncadapter
An alternative solution to using a sync adapter to get the data FROM your web service would be to use GCM (Google Cloud Messaging service. http://developer.android.com/google/gcm/gs.html
Basically Account Manager/sync adapter/gcm is the functionality that Android recommend you use and these are the services that Google uses itself for things like GMail and give your users the ability to keep their data intact even after clearing their data or uninstalling the app and re-installing the app and it also allows for a user to be able to install the app on a new phone and keeping their data.
Hope that helps
UPDATE in response to comments
You should always try hard to minimise traffic and size of data being sent in either direction. I would send the whole lot in one single JSON request gzipped.
Your web server should be able to automatically handle gzipped requests and if gzip is not installed on your server it's simple enough to add
A reference to creating a gzipped output stream from your app can be found here
http://developer.android.com/reference/java/util/zip/GZIPOutputStream.html
For the web server the solution you need will depend on the server you use and you should check with your host about gzip but here are a couple of links for the most popular web servers
Apache - http://howtounix.info/howto/Apache-gzip-compression-with-mod_deflate
NGinx Tutorial - http://www.howtoforge.com/how-to-save-traffic-with-nginxs-httpgzipmodule-debian-squeeze
GZip is the most popular solution for web servers and is very simple to implement.
I don't really have enough info to provide much more advice other than to say that I normally use Ruby on Rails for stuff like this and gzip is handled very simply with the ative support gzip library using something similar to this contacts = ActiveSupport::JSON.decode(gzipped_contacts.gsub("+", ""))
UPDATE 2 notifyChange()
Just to pick up on your point about notifyChange not being appropriate in the content provider due to there being no internet access.
It is fine to use notifyChange() in your content provider as it will tell the sync adapter to update as soon as it is appropriate to do so which means when the device is not too busy and as soon as an internet connection becomes available. That's the whole point of the sync adapter. Should you need to make use of notifyChange in your content provider for other services such as array adapters but you do not want the sync adapter to be told to update then there is a little documented boolean parameter that you can add to the end of the notifyChange params list. set it to false and the content provider will ignore the notifyChange
e.g. instead of the usual getContext().getContentResolver().notifyChange(uri, null);You can use getContext().getContentResolver().notifyChange(uri, null, false);
a) You could ask the users to sign in / sign up from the mobile apps, exactly as you would do it for a web site.
b) Take a look at this http://android-developers.blogspot.ro/2013/01/verifying-back-end-calls-from-android.html
Send more records in the same request, the idea is to make as few requests as possible.
I don't think you should keep the flag for error the same as the default value, it should be a different value so that you will be able to handle it more easily.
I implemented an Android application that requires a set of data, taken by a SQL Server database. The application obtains the data calling a WS. After a first call to WS when the application start the first time, I need to maintain the data updated, according to the modify that may happens server-side (SQL server database).
For obtaining this result I perform, with a with a predefined frequency, a WS call, for knowing if data on database are changed. If new data are available, other web service is called for obtaining them.
This solution works fine for my ( I don't require real-time update). But, I think that this solution is too expensive in term of energy consumption, cpu consumption and network traffic.
Since, I immagine this is a very common problem I would know if exists a generic way to deal with it.
I suggest you to use extra fields. Add four colums to your local tables in Android :
TRANSACTING_FLAG : Set it to true when you are posting or updating this resource on the server
REQUEST_STATE : Set this flag to POSTING / UPDATING / DELETING etc.
RESULT_CODE : Set this field to the last result code received by the server for this particular resource.
TIMESTAMP : Period after wich data has to be updated
Workflow is simple :
When you retrieve data for your server just check if the last updated timestamp of your resource is superior to the cache timestamp you have defined before. If the timestamp is superior perform a request to update data. The transacting boolean let you know that a particular resource is actually synchronizing with the server. The result code lets you know if the request has failed or not and enventually retry it when the network is available. Doing this way you will maintain the persitence between your local and remote database because at any moment you can check the "synchronized state" of any local resource thanks to extra fields seen before.
I made a library to handle this. Take a look to RESTDroid. Even if the cache functionnality is not handles, you will be able to simply add it.
What you do is ok for most cases. You can take advantage of Google Cloud Messaging, but it needs time and effort to get implemented. I would stay with your solution.
You could look into Query Notifications, using something like SqlDependency - http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldependency.aspx. You can use this to track whether the results of a query change and to inform an application when this happens.
There are restrictions on the query you can use, and cost on the server is similar to an indexed view. You need .NET for this, by the way. If implemented in your Web Service, you would have to implement some kind of subscribe feature for your android, so that notifications could be pushed to it.
Another option to reduce the cost of checking for changes could be SQL Server Change Tracking - http://msdn.microsoft.com/en-us/library/bb933875.aspx