I'm writing an app that will periodically send information out to a remote server, and then get relevant information about other users from that server back to the local database. What's the best way to handle sending out this info (i.e.: XML or binary) and writing it to the remote server.
Also, how can I assure that, when 500+ users' data get's to the server or FTP (or better alternative?) at once, the appropriate fields gets overwritten or added, without skipping any or overwriting the entire thing? Thanks for the help.
i was looking for that answer too and i found some nice tutorials. Unfortenly they are not in english, but you can understand a princip.
This (http://vimeo.com/29332913) is the forth video of tutorials. Just click on authors name and start 3 tutorials before.
Hope it will help you XD
Janez
Have you considered JSON? FTP is generally a solution when it comes to binary data (large files, etc.), however I assume you want to exchange data that contains information in a textual format, so for a website HTTP will be a good way (the best way depends entirely on your server setup, your data format and model).
I have to add, that YAML is also a very nice data format tool if you are using Ruby frameworks, which I prefer over JSON (personal experience).
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I want to sync data (such as db record, media) between an Android App and a Server. If you've seen Evernote or similar Applications, you certainly understand what I mean.
I have some question (imagine we want to sync DB records):
Every user has a part of server space for himself (such as Evernote or Dropbox). Maybe the user creates new records by cellphone and creates new records in server. How can I match these records together? If there are records with same ID What algorithms do you suggest me?
Except JSON, Are there any way for send data between cellphone device and server?
If SyncAdapter and ContentProvider can solve my problems, please explain exactly for me. (If you could offer some samples or tutorials to me OR Any advice or keywords to help broaden/guide my search would be appreciated as well).
I'll try to answer all your questions by addressing the larger question: How can I sync data between a webserver and an android app?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database. A decent tutorial for a content provider can be found here: http://thinkandroid.wordpress.com/2010/01/13/writing-your-own-contentprovider/
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: https://github.com/google/gson, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data.
If we think about today, accepted answer is too old. As we know that we have many new libraries which can help you to make this types of application.
You should learn following topics that will helps you surely:
SyncAdapter: The sync adapter component in your app encapsulates the code for the tasks that transfer data between the device and a server. Based on the scheduling and triggers you provide in your app, the sync adapter framework runs the code in the sync adapter component.
Realm: Realm is a mobile database: a replacement for SQLite & Core Data.
Retrofit Type-safe HTTP client for Android and Java by Square, Inc. Must Learn a-smart-way-to-use-retrofit
And your sync logic for database like: How to sync SQLite database on Android phone with MySQL database on server?
Best Luck to all new learner. :)
If you write this yourself these are some of the points to keep in mind
Proper authentication between the device and the Sync Server
A sync protocol between the device and the server. It will usually go in 3 phases, authentication, data exchange, status exchange (which operations worked and which failed)
Pick your payload format. I suggest SyncML based XML mixed with JSON based format to represent the actual data. So SyncML for the protocol, and JSON for the actual data being exchanged. Using JSON Array while manipulating the data is always preferred as it is easy to access data using JSON Array.
Keeping track of data changes on both client and server. You can maintain a changelog of ids that change and pick them up during a sync session. Also, clear the changelog as the objects are successfully synchronized. You can also use a boolean variable to confirm the synchronization status, i.e. last time of sync. It will be helpful for end users to identify the time when last sync is done.
Need to have a way to communicate from the server to the device to start a sync session as data changes on the server. You can use C2DM or write your own persistent tcp based communication. The tcp approach is a lot seamless
A way to replicate data changes across multiple devices
And last but not the least, a way to detect and handle conflicts
Hope this helps as a good starting point.
#Grantismo provides a great explanation on the overall. If you wish to know who people are actually doing this things i suggest you to take a look at how google did for the Google IO App of 2014 (it's always worth taking a deep look at the source code of these apps that they release. There's a lot to learn from there).
Here's a blog post about it: http://android-developers.blogspot.com.br/2014/09/conference-data-sync-gcm-google-io.html
Essentially, on the application side: GCM for signalling, Sync Adapter for data fetching and talking properly with Content Provider that will make things persistent (yeah, it isolates the DB from direct access from other parts of the app).
Also, if you wish to take a look at the 2015's code: https://github.com/google/iosched
For example, you want to sync table todoTable from MySql to Sqlite
First, create one column name version (type INT) in todoTable for both Sqlite and MySql
Second, create a table name database_version with one column name currentVersion(INT)
In MySql, when you add a new item to todoTable or update item, you must upgrade the version of this item by +1 and also upgrade the currentVersion
In Android, when you want to sync (by manual press sync button or a service run with period time):
You will send the request with the Sqlite currentVersion (currently it is 1) to server.
Then in server, you find what item in MySql have version value greater than Sqlite currentVersion(1) then response to Android (in this example the item 3 with version 2 will response to Android)
In SQLite, you will add or update new item to todoTable and upgrade the currentVersion
Look at parseplatform.org.
it's opensource project.
(As well as you can go for commercial package available at back4app.com.)
It is a very straight forward and user friendly server side database service that gives a great android client side API
one way to accomplish this to have a server side application that waits for the data. The data can be sent using HttpRequest objects in Java or you can write your own TCP/IP data transfer utility. Data can be sent using JSON format or any other format that you think is suitable. Also data can be encrypted before sending to server if it contains sensitive information. All Server application have to do is just wait for HttpRequests to come in and parse the data and store it anywhere you want.
I would suggest using a binary webservice protocol similar to Hessian. It works very well and they do have a android implementation. It might be a little heavy but depends on the application you are building. Hope this helps.
#Grantismo gives a great overview of Android sync components.
SyncManagerAndroid library provides a simple 2-way sync implementation to plug into the Android Sync framework (AbstractThreadedSyncAdapter.OnPerformSync).
https://github.com/sschendel/SyncManagerAndroid
I am interested in making android app like crickbuzz that will fetch live content from server and also there is option of like and comment for users for feedback. So, can anyone tell that how to implement this i.e (how to fetch data from server) how that data will fit in app(confused about this how data get fit in instagram app means profile, home, search tab)
thanks
I suggest you check out this links
Android, PHP & MySQL
Fetch Data From Server
Ok, let's break it down:
The data is obtained from the server using http requests. It is usually received in the form of JSON String, basically a ordered, human-readable data format which is easy to read and parse.
On the app side, you need to implement the layout, which is done in xml and Java. You also need some logic which will obtain the data with the aforementioned requests, parse it, turn into Java Objects and put everything into the layout.
The process is not really complex, could be completed by a single person, but you will need experience, time and good architectural decisions.
If this is your first time with Android app programming, do some tutorials (especially ones that cover networking), learn some libraries (Volley, Retrofit, org.json, RXJava, Picasso will be really useful for what you ask about) and code several small apps from start to end. After that you may be ready to tackle the app you described :-)
In Formhub, in the form management page, there was a button on the right to connect to bamboo.io . Doing that gave me a handy URL having all the collected data in a JSON, through which I could dynamically load latest collected survey data in my application. See screenshot.
As an example, all entries submitted on this form (and its ODK Collect counterpart), were collecting at this URL in JSON format which I could then import into my javascript and do mapping, data visualization, etc.
I'm using formhub's successor, ona.io now because formhub.org isn't loading half the time, and I read in some article that they don't intend to keep it up and one should go to ona.io now to do the same thing. It's the same thing (at much better speed, I can see that formhub.org was having server problems), But it's missing the easy bamboo.io integration. There is a "REST Services" tab mentioned below in the form's dashboard..
..that I can't make sense out of. What am I supposed to enter as "service URL" ?
I tried looking up ona.io's documentation but it's all for API and involved CURL commands etc. I'm checking out bamboo.io and even there I'm not getting any help. Everywhere they're going into advanced usage of custom servers, API's CURL GET etc. But I just want the simple live JSON data through a simple link that I was getting in formhub.
I'm open to any other methods if available.. the end goal is to have a URL that carries all the accumulated data submitted by the form in JSON or CSV format. I want to import JSON (or CSV) from this set URL in my javascript on a web page hosted on my own server.
Note that it's not possible for me to self-host an ODK Aggregate instance. Ona.io is doing that for me.
In the traditional way of exporting data at ona.io (and formhub before it), the system creates a new file each time you export, with a timestamped file name. It doesn't auto-generate any single URL having latest data.. one has to go there and generate manually. That won't do... need a fixed URL which will have all latest data JSON or CSV.
Found a way. ona.io provides the same thing natively; I don't need bamboo.io (besides, even that is being retired same as formhub!)
All you need to do to get started, is to go here:
https://ona.io/api/v1/data?owner=[your usename]
From there you'll find URL's to the data collected by your forms, and you'll get the direct URLs for getting the data in JSON / CSV / many other formats.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I want to sync data (such as db record, media) between an Android App and a Server. If you've seen Evernote or similar Applications, you certainly understand what I mean.
I have some question (imagine we want to sync DB records):
Every user has a part of server space for himself (such as Evernote or Dropbox). Maybe the user creates new records by cellphone and creates new records in server. How can I match these records together? If there are records with same ID What algorithms do you suggest me?
Except JSON, Are there any way for send data between cellphone device and server?
If SyncAdapter and ContentProvider can solve my problems, please explain exactly for me. (If you could offer some samples or tutorials to me OR Any advice or keywords to help broaden/guide my search would be appreciated as well).
I'll try to answer all your questions by addressing the larger question: How can I sync data between a webserver and an android app?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database. A decent tutorial for a content provider can be found here: http://thinkandroid.wordpress.com/2010/01/13/writing-your-own-contentprovider/
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: https://github.com/google/gson, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data.
If we think about today, accepted answer is too old. As we know that we have many new libraries which can help you to make this types of application.
You should learn following topics that will helps you surely:
SyncAdapter: The sync adapter component in your app encapsulates the code for the tasks that transfer data between the device and a server. Based on the scheduling and triggers you provide in your app, the sync adapter framework runs the code in the sync adapter component.
Realm: Realm is a mobile database: a replacement for SQLite & Core Data.
Retrofit Type-safe HTTP client for Android and Java by Square, Inc. Must Learn a-smart-way-to-use-retrofit
And your sync logic for database like: How to sync SQLite database on Android phone with MySQL database on server?
Best Luck to all new learner. :)
If you write this yourself these are some of the points to keep in mind
Proper authentication between the device and the Sync Server
A sync protocol between the device and the server. It will usually go in 3 phases, authentication, data exchange, status exchange (which operations worked and which failed)
Pick your payload format. I suggest SyncML based XML mixed with JSON based format to represent the actual data. So SyncML for the protocol, and JSON for the actual data being exchanged. Using JSON Array while manipulating the data is always preferred as it is easy to access data using JSON Array.
Keeping track of data changes on both client and server. You can maintain a changelog of ids that change and pick them up during a sync session. Also, clear the changelog as the objects are successfully synchronized. You can also use a boolean variable to confirm the synchronization status, i.e. last time of sync. It will be helpful for end users to identify the time when last sync is done.
Need to have a way to communicate from the server to the device to start a sync session as data changes on the server. You can use C2DM or write your own persistent tcp based communication. The tcp approach is a lot seamless
A way to replicate data changes across multiple devices
And last but not the least, a way to detect and handle conflicts
Hope this helps as a good starting point.
#Grantismo provides a great explanation on the overall. If you wish to know who people are actually doing this things i suggest you to take a look at how google did for the Google IO App of 2014 (it's always worth taking a deep look at the source code of these apps that they release. There's a lot to learn from there).
Here's a blog post about it: http://android-developers.blogspot.com.br/2014/09/conference-data-sync-gcm-google-io.html
Essentially, on the application side: GCM for signalling, Sync Adapter for data fetching and talking properly with Content Provider that will make things persistent (yeah, it isolates the DB from direct access from other parts of the app).
Also, if you wish to take a look at the 2015's code: https://github.com/google/iosched
For example, you want to sync table todoTable from MySql to Sqlite
First, create one column name version (type INT) in todoTable for both Sqlite and MySql
Second, create a table name database_version with one column name currentVersion(INT)
In MySql, when you add a new item to todoTable or update item, you must upgrade the version of this item by +1 and also upgrade the currentVersion
In Android, when you want to sync (by manual press sync button or a service run with period time):
You will send the request with the Sqlite currentVersion (currently it is 1) to server.
Then in server, you find what item in MySql have version value greater than Sqlite currentVersion(1) then response to Android (in this example the item 3 with version 2 will response to Android)
In SQLite, you will add or update new item to todoTable and upgrade the currentVersion
Look at parseplatform.org.
it's opensource project.
(As well as you can go for commercial package available at back4app.com.)
It is a very straight forward and user friendly server side database service that gives a great android client side API
one way to accomplish this to have a server side application that waits for the data. The data can be sent using HttpRequest objects in Java or you can write your own TCP/IP data transfer utility. Data can be sent using JSON format or any other format that you think is suitable. Also data can be encrypted before sending to server if it contains sensitive information. All Server application have to do is just wait for HttpRequests to come in and parse the data and store it anywhere you want.
I would suggest using a binary webservice protocol similar to Hessian. It works very well and they do have a android implementation. It might be a little heavy but depends on the application you are building. Hope this helps.
#Grantismo gives a great overview of Android sync components.
SyncManagerAndroid library provides a simple 2-way sync implementation to plug into the Android Sync framework (AbstractThreadedSyncAdapter.OnPerformSync).
https://github.com/sschendel/SyncManagerAndroid
Below I explained what I need and in the answer I would like to get information what technology, what kind of protocols, services etc should I use.
Also I know that there is a massive amount of information on the internet, but because there are so many choices I'm unable to make a decision. Here's what I want to have:
Android application which will sent and receive information from the internet/server
Of course server, which will do some rudimentary computation with the obtained information, then store it and at the end send it back to the client application
I have to admit that the subjects of networking, socketing, protocols, ciphering etc are the ones I've always run away from. Therefore advices such as what domains, databases etc utilize are highly welcome
Update: After a bit of a research I've implemented mechanism based on information from this site [1]. I already have a working MySql database with several tables inside. Also I've added php scripts to my Apache webspace and implemented all that's necessary within my Android app.
Now, as I understand communication between php's scripts and MySql database is safe (I've hardcoded the usr/pass within scripts). Therefore the only thing that has left to be done to secure the connection between my Android application and php scripts (I want to prevent the situation when everybody has an access to my php scripts). So my question should be pretty straightforward now, how can it be achieved?
[1] http://blog.sptechnolab.com/2011/02/10/android/android-connecting-to-mysql-using-php/
I think a simple TCP/IP connection via Sockets will do just fine for your purposes.
The lesson (with more information and some examples) from the Java-Docs can be found here.
The basic workflow is as follows:
Your Android-App opens a Socket-connection to your server on a
given port where a server-application listens.
The Android-App sends it's data (whatever that might be) to the
Server.
The Server reads the send data,
processes it,
stores it...
...then it sends back some response (maybe the computed values) to your
Android-App.
Your app can then figure out if everything went okay and use the
given data.
Answering my second question, all the php scripts has been put into directory with the password on my Apache server. To be able to trigger them I need to give this password which is hardcoded in my application. All the connection is done through HTTPS. Now everything works as planned!