I am dealing with huge data (downloading from webserver to client/phone). Currently I am trying to parse and load complete data into sqlite database when the application launches.
This is taking a lot of time. I want to load the data based on the screen navigation.
How do I achieve that?
You'll want to pass information between your server and the device as XML.
This way the device can request URLs for specific information such as example.com/news/latest (return headlines of the latest 20 news articles), example.com/news/americas (the latest 20 article headlines from the Americas) or example.com/news/article/177309/ (one full article).
The server can then query its database for the needed information based on the query and output it for the client as XML. The client can parse the XML and add the data to it's local SQLiteDatabase.
Is this not restricting the dataset returned from the source first, then downloading it onto the client device?
If you are looking for performance boost up , i prefer running an async task in which the data gets uploaded or downloaded to the server. Its similar to threads but much better.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I want to sync data (such as db record, media) between an Android App and a Server. If you've seen Evernote or similar Applications, you certainly understand what I mean.
I have some question (imagine we want to sync DB records):
Every user has a part of server space for himself (such as Evernote or Dropbox). Maybe the user creates new records by cellphone and creates new records in server. How can I match these records together? If there are records with same ID What algorithms do you suggest me?
Except JSON, Are there any way for send data between cellphone device and server?
If SyncAdapter and ContentProvider can solve my problems, please explain exactly for me. (If you could offer some samples or tutorials to me OR Any advice or keywords to help broaden/guide my search would be appreciated as well).
I'll try to answer all your questions by addressing the larger question: How can I sync data between a webserver and an android app?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database. A decent tutorial for a content provider can be found here: http://thinkandroid.wordpress.com/2010/01/13/writing-your-own-contentprovider/
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: https://github.com/google/gson, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data.
If we think about today, accepted answer is too old. As we know that we have many new libraries which can help you to make this types of application.
You should learn following topics that will helps you surely:
SyncAdapter: The sync adapter component in your app encapsulates the code for the tasks that transfer data between the device and a server. Based on the scheduling and triggers you provide in your app, the sync adapter framework runs the code in the sync adapter component.
Realm: Realm is a mobile database: a replacement for SQLite & Core Data.
Retrofit Type-safe HTTP client for Android and Java by Square, Inc. Must Learn a-smart-way-to-use-retrofit
And your sync logic for database like: How to sync SQLite database on Android phone with MySQL database on server?
Best Luck to all new learner. :)
If you write this yourself these are some of the points to keep in mind
Proper authentication between the device and the Sync Server
A sync protocol between the device and the server. It will usually go in 3 phases, authentication, data exchange, status exchange (which operations worked and which failed)
Pick your payload format. I suggest SyncML based XML mixed with JSON based format to represent the actual data. So SyncML for the protocol, and JSON for the actual data being exchanged. Using JSON Array while manipulating the data is always preferred as it is easy to access data using JSON Array.
Keeping track of data changes on both client and server. You can maintain a changelog of ids that change and pick them up during a sync session. Also, clear the changelog as the objects are successfully synchronized. You can also use a boolean variable to confirm the synchronization status, i.e. last time of sync. It will be helpful for end users to identify the time when last sync is done.
Need to have a way to communicate from the server to the device to start a sync session as data changes on the server. You can use C2DM or write your own persistent tcp based communication. The tcp approach is a lot seamless
A way to replicate data changes across multiple devices
And last but not the least, a way to detect and handle conflicts
Hope this helps as a good starting point.
#Grantismo provides a great explanation on the overall. If you wish to know who people are actually doing this things i suggest you to take a look at how google did for the Google IO App of 2014 (it's always worth taking a deep look at the source code of these apps that they release. There's a lot to learn from there).
Here's a blog post about it: http://android-developers.blogspot.com.br/2014/09/conference-data-sync-gcm-google-io.html
Essentially, on the application side: GCM for signalling, Sync Adapter for data fetching and talking properly with Content Provider that will make things persistent (yeah, it isolates the DB from direct access from other parts of the app).
Also, if you wish to take a look at the 2015's code: https://github.com/google/iosched
For example, you want to sync table todoTable from MySql to Sqlite
First, create one column name version (type INT) in todoTable for both Sqlite and MySql
Second, create a table name database_version with one column name currentVersion(INT)
In MySql, when you add a new item to todoTable or update item, you must upgrade the version of this item by +1 and also upgrade the currentVersion
In Android, when you want to sync (by manual press sync button or a service run with period time):
You will send the request with the Sqlite currentVersion (currently it is 1) to server.
Then in server, you find what item in MySql have version value greater than Sqlite currentVersion(1) then response to Android (in this example the item 3 with version 2 will response to Android)
In SQLite, you will add or update new item to todoTable and upgrade the currentVersion
Look at parseplatform.org.
it's opensource project.
(As well as you can go for commercial package available at back4app.com.)
It is a very straight forward and user friendly server side database service that gives a great android client side API
one way to accomplish this to have a server side application that waits for the data. The data can be sent using HttpRequest objects in Java or you can write your own TCP/IP data transfer utility. Data can be sent using JSON format or any other format that you think is suitable. Also data can be encrypted before sending to server if it contains sensitive information. All Server application have to do is just wait for HttpRequests to come in and parse the data and store it anywhere you want.
I would suggest using a binary webservice protocol similar to Hessian. It works very well and they do have a android implementation. It might be a little heavy but depends on the application you are building. Hope this helps.
#Grantismo gives a great overview of Android sync components.
SyncManagerAndroid library provides a simple 2-way sync implementation to plug into the Android Sync framework (AbstractThreadedSyncAdapter.OnPerformSync).
https://github.com/sschendel/SyncManagerAndroid
So the app is for a Parrot Rescue. It will contain profiles for each of the birds that we currently have for adoption. So a picture of the bird, and then basic info about the bird. It needs to be regularly updated as the birds are adopted out and new ones come in. What route should I go? My main concern is that I want it to be relatively easy to update so that some of the less technically inclined could use it, as I won't always be available to help them.
Instead of loading a new version of the application every time, you should implement a database on a parrot information server. Then the application (the client) will ask the server to receive parrot information.
Just implement a mySQL database and a PHP page that returns the information in JSON format. Then through the application you will make a request to the server using the volley library or okHTTP and enter the information into a list.
As there are also pictures, use the glide library to download the images.
Also remember to upload small images to the server.
This way the app is always up to date
The easiest way is to update the app on the play store with the newest db, and let auto-update update the app with the new db.
Next easiest is to have a nightly (or whatever frequency) service download a new db and replace the old one from your servers
If you want it to be as easy as possible to update for everyone I would recommend a little app that has two main activities. One for displaying the entries of the database (and maybe also giving the ability to delete the entry) and another with a template to fill with all information and the image.
Then like Raffaele D'Arco explained use a little server. You could also delay the updates to be in a 5 minute routine or the like so that you put as little stress as possible on the server.
There are multiple good ways to update the interface without updating the app constantly.
Use a database which the app will download after a fixed amount of time which will be located on the server.
Secondly, if you have a website you can convert the web pages to XML or JSON or something like that and can update the app UI as per the data downloaded from the website.
I would recommend the second option as it is quite easy than the first one...
In my android application im using web-service to get information about food the problem that if i write appl instead of apple,row,skin it dosent fill automatically and an error message will be shown,, what i did i stored 2000 name from web service in sqlite and search through them is it wrong to do this ? and how can i search with misspelling in web-service without the error message ?
The api url http://api.nal.usda.gov/ndb/search/?format=json&q=apple&sort=n&max=25&offset=0&api_key=DEMO_KEY
Salam Enas,
If the data do change frequently (like a movies list) then it's not ok to store it in the sqlite and it's better to be stored in the server database.
If the data does not change frequently, then it's ok to store them on the sqlite database although the app size will increase by 2 Mb which is a drawback.
You may also want to consider other APIs if you are not committed to use this one.
When storing data on the server, HTTP requests burden can be alleviated by caching data on the mobile.
good luck :)
I have developed an app which make connection to MySQL database and display some information from there. Is there a way to test performance of app like simulation of different amount of users request.
500 HTTP requests
1000, 1500 request.
Yes it is possible ... Performance Testing a Mobile Application is different from server or other applications.
Simulating the load can be done by pushing dummy records to SQLite Table and doing your testing. However, Since you mentioned that
app which make connection to MySQL database and display some information from there
I would like to suggest you to look for the following.
Analyze the data parsing logic and how the data is rendered on device.
Data Retrieval Thread should be separate from Main Thread.
Data Parsing should occur in parts. User should not wait for all 500 records to be parsed before rendering on screen. To do this, in the code, based on device screen size data retrieval count should be managed.
What are the views used to render the data on screen ? If list View, see how the data is getting recycled.
In SQLite, if they are storing images, see if BLOB datatype is used.
Check the frequency of database interaction. This would impact the application performance highly.
Hope this helps.
Try with JMeter. It's free tool. I've been using it to simulate http request to pages. Hope its help.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I want to sync data (such as db record, media) between an Android App and a Server. If you've seen Evernote or similar Applications, you certainly understand what I mean.
I have some question (imagine we want to sync DB records):
Every user has a part of server space for himself (such as Evernote or Dropbox). Maybe the user creates new records by cellphone and creates new records in server. How can I match these records together? If there are records with same ID What algorithms do you suggest me?
Except JSON, Are there any way for send data between cellphone device and server?
If SyncAdapter and ContentProvider can solve my problems, please explain exactly for me. (If you could offer some samples or tutorials to me OR Any advice or keywords to help broaden/guide my search would be appreciated as well).
I'll try to answer all your questions by addressing the larger question: How can I sync data between a webserver and an android app?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database. A decent tutorial for a content provider can be found here: http://thinkandroid.wordpress.com/2010/01/13/writing-your-own-contentprovider/
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: https://github.com/google/gson, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data.
If we think about today, accepted answer is too old. As we know that we have many new libraries which can help you to make this types of application.
You should learn following topics that will helps you surely:
SyncAdapter: The sync adapter component in your app encapsulates the code for the tasks that transfer data between the device and a server. Based on the scheduling and triggers you provide in your app, the sync adapter framework runs the code in the sync adapter component.
Realm: Realm is a mobile database: a replacement for SQLite & Core Data.
Retrofit Type-safe HTTP client for Android and Java by Square, Inc. Must Learn a-smart-way-to-use-retrofit
And your sync logic for database like: How to sync SQLite database on Android phone with MySQL database on server?
Best Luck to all new learner. :)
If you write this yourself these are some of the points to keep in mind
Proper authentication between the device and the Sync Server
A sync protocol between the device and the server. It will usually go in 3 phases, authentication, data exchange, status exchange (which operations worked and which failed)
Pick your payload format. I suggest SyncML based XML mixed with JSON based format to represent the actual data. So SyncML for the protocol, and JSON for the actual data being exchanged. Using JSON Array while manipulating the data is always preferred as it is easy to access data using JSON Array.
Keeping track of data changes on both client and server. You can maintain a changelog of ids that change and pick them up during a sync session. Also, clear the changelog as the objects are successfully synchronized. You can also use a boolean variable to confirm the synchronization status, i.e. last time of sync. It will be helpful for end users to identify the time when last sync is done.
Need to have a way to communicate from the server to the device to start a sync session as data changes on the server. You can use C2DM or write your own persistent tcp based communication. The tcp approach is a lot seamless
A way to replicate data changes across multiple devices
And last but not the least, a way to detect and handle conflicts
Hope this helps as a good starting point.
#Grantismo provides a great explanation on the overall. If you wish to know who people are actually doing this things i suggest you to take a look at how google did for the Google IO App of 2014 (it's always worth taking a deep look at the source code of these apps that they release. There's a lot to learn from there).
Here's a blog post about it: http://android-developers.blogspot.com.br/2014/09/conference-data-sync-gcm-google-io.html
Essentially, on the application side: GCM for signalling, Sync Adapter for data fetching and talking properly with Content Provider that will make things persistent (yeah, it isolates the DB from direct access from other parts of the app).
Also, if you wish to take a look at the 2015's code: https://github.com/google/iosched
For example, you want to sync table todoTable from MySql to Sqlite
First, create one column name version (type INT) in todoTable for both Sqlite and MySql
Second, create a table name database_version with one column name currentVersion(INT)
In MySql, when you add a new item to todoTable or update item, you must upgrade the version of this item by +1 and also upgrade the currentVersion
In Android, when you want to sync (by manual press sync button or a service run with period time):
You will send the request with the Sqlite currentVersion (currently it is 1) to server.
Then in server, you find what item in MySql have version value greater than Sqlite currentVersion(1) then response to Android (in this example the item 3 with version 2 will response to Android)
In SQLite, you will add or update new item to todoTable and upgrade the currentVersion
Look at parseplatform.org.
it's opensource project.
(As well as you can go for commercial package available at back4app.com.)
It is a very straight forward and user friendly server side database service that gives a great android client side API
one way to accomplish this to have a server side application that waits for the data. The data can be sent using HttpRequest objects in Java or you can write your own TCP/IP data transfer utility. Data can be sent using JSON format or any other format that you think is suitable. Also data can be encrypted before sending to server if it contains sensitive information. All Server application have to do is just wait for HttpRequests to come in and parse the data and store it anywhere you want.
I would suggest using a binary webservice protocol similar to Hessian. It works very well and they do have a android implementation. It might be a little heavy but depends on the application you are building. Hope this helps.
#Grantismo gives a great overview of Android sync components.
SyncManagerAndroid library provides a simple 2-way sync implementation to plug into the Android Sync framework (AbstractThreadedSyncAdapter.OnPerformSync).
https://github.com/sschendel/SyncManagerAndroid