I'm trying to figure out the best strategy for data caching in Android, particularly in view of using the data binding library.
The Android documentation really seems to put a lot of emphasis on using the built-in SQLite database and to access this with the help of content providers. However, assuming that my app retrieves data in JSON format from a REST API, using this strategy would entail the following:
Make HTTP request --> Parse JSON --> Insert results into DB
--> Call content provider --> Build model from cursor --> Bind to view
Not only does this seem like a very roundabout way of doing something relatively simple, but assuming all this happens when the user first opens the app the result would be lots and lots of waiting before anything useful appears on the screen.
To speed things up I may then decide to create my model earlier and let the caching take place in a separate thread, like this:
Make HTTP request --> Build model from JSON --> Bind to view
-->(NEW THREAD) --> Insert results into DB
Once the data is cached, the next time the user opens the app the following would take place:
Call content provider --> Build model from cursor --> Bind to view
But of course this would add even more complexity, for example forcing me to maintain code to build the model from two sources: JSON and the cursors returned by the content provider.
Given the above I am even more tempted to do away with the SQLite/ContentProvider model and instead do the following:
Make HTTP request --> Build Model and Store JSON to file --> Bind to view
But while this would greatly reduce boilerplate, parsing (plenty of libraries available for JSON parsing) and overall complexity it would also mean I cannot take advantage of content providers nor of SQLite's functionality.
So the question is, which model should I follow? Are there situations in which one is better than the other? Or are there better ways of handling this process that I am not aware of?
A possible solution to your problem could be the one used in the Google IO app. They parse JSON available on a remote server and display that content directly. In order to avoid empty screens they use a bootstrap json file containing an image of the data available on the server. All this info is available here. More concretely:
Bootstrap data
When the user runs the app for the first time, they expect to see data. However, if we relied only on the sync mechanism to bring data into the app, a first-time user would stare at a blank screen while waiting for a sync to happen, and this would be a bad user experience.
This is why IOSched ships with preloaded "bootstrap data", which is essentially a preloaded offline snapshot of the JSON data. This data is parsed by the app and saved to the database on first execution.
You can find this file in res/raw/bootstrap.json. It is simply a text file with a combined snapshot of the JSON files on the server.
Related
About Single Sources of Truth Google document said:
Using this model, the database serves as the single source of truth, and other parts of the app access it using our UserRepository. Regardless of whether you use a disk cache, we recommend that your repository designate a data source as the single source of truth for the rest of your app
https://developer.android.com/jetpack/guide?gclid=CjwKCAjwo4mIBhBsEiwAKgzXOH1Pq--Ws1PLzUiSP4RmDE6ByKfEi6mdXu5g86btqveIdJvvrgYuxBoCz8wQAvD_BwE&gclsrc=aw.ds#connect-viewmodel-repository
According to the document I save all data when I fected data from remote server and I only get data from room When I need to use in acitivty(In fact I collect flow which is defined in viewmodel).
It seems so good! It avoids the different data sources mix up together! But actually I found some strange question gradually:
In my App, I have a list that the server may change it(Because we have data manager website that admin can update or delete data). So in order to get the newest list data from server, I must clear all data stored in room and fect data again from remote server. This operation seems redundant: "why could I get data directly from remote server", I mean, I only get data from remote source is also a single sources truth. And also it cause a promble: my app will flash a moment because clear data make list empty and fect data from server make list full!
The most important thing is that it seems like the local data is not necessary because I must stay the newest list from remote server.
Some people may say that save data into room can make us app available offlice. I agree that, But in this place, my item of list is represent a image url, and after click the item, the app will jump to a new activity and display a ImageView base on the url we get from the list. If app offlice, the ImageView couldn't load the url also.
I am so confused I couldn't load all image url(use base64-url to avoid load invalid) in a moment also, because the data is so much. And if I say I need a search function in this list and I need load so much unbelievable data into my room, It seems so unreal and event fantasy!
In brief：
Room is a nessary? Couldnt just fect data from remote?
If room is nessary, how to solve problem I met, do my incorrect useage cause the problem?
Hi #psycongroo as I Understood your problem, and I want to share my experience:
You can handle any error with loading URL with placeholder I mean if you got an error with no Internet connection user will see placeholder, but in general libs like Picasso or Glide can cache images when it`s load one time, so the user will see the Image.
The question about why we need to use room instead of fetch data from remote directly. So from your question I don`t understand why you need to drop your local changes even they are completely new, user can have a low internet connection so he will see an empty list instead of previous data with for example progress indicator. And also if the user doesn't have the internet at all you can show some dialog to explain what the problem but old data is still present. If you are using, for example, RecyclerView you can update data with Paging 3 from google, and they update the only necessary items from your list.
P.S. let me know if that help, or you have another question.
Is it possible to download a form, a form has input fields where a user capture data and submits to dB etc, is it possible to download those fields with its id’s into android from web server without creating inputs for offline data capture then upload the captured data when internet is available. I have seen one working from dhis2 event capture where it downloads a data set for offline capture - https://github.com/dhis2/dhis2-android-eventcapture?files=1
Another example is how kobo collect works - https://www.kobotoolbox.org
I don’t know if the same can work but with php. Anyone with an idea how it’s done.
if I understand your question, you want to know if you can perform the following actions using DHIS2:
Download the definition of a form (For example a data set)
Use the definition to render the form on a client (For example Android)
Perform data entry offline using this form
Upload data from the offline data entry when the client reconnects to the internet
If these are the things you are asking about, the answer is yes, it is possible. This is a very common use-case for DHIS2 clients, since they are often using in areas with poor or no internet connection.
In DHIS2 there will be 3 main types of form definitions you may work with:
1. Metadata (Any metadata, used to define the other forms)
2. Aggregate data (Data entry using data elements and data sets)
3. Tracker data (Data entry using the tracker models: Tracked entities, enrolments and events)
Depending on which types of forms you want to work with, their definition is found in different places.
Information about metadata can be found here:
../api/schemas
Information about data sets, data elements, etc. (Check docs for other relevant endpoints):
../api/dataSets
../api/dataSets/<uid>
../api/dataElements/<uid>
Using these endpoints should be sufficient to gather all definitions you need to recreate any data entry form for a given data set.
When you have collected data offline and want to end it back to DHIS2, you can use the following endpoint to POST your data:
../api/dataValueSets
This should give you the main steps to solve your problem, but for more in-depth descriptions of the api or data models, you should have a look at the docs:
Developer documentation
I am developing an Android app that has a list, I would like this list to be synced between multiple users - can it be done with out server side?
Syncing data between your webserver and an android app requires a couple of different components on your android device.
Persistent Storage:
This is how your phone actually stores the data it receives from the webserver. One possible method for accomplishing this is writing your own custom ContentProvider backed by a Sqlite database.
A ContentProvider defines a consistent interface to interact with your stored data. It could also allow other applications to interact with your data if you wanted. Behind your ContentProvider could be a Sqlite database, a Cache, or any arbitrary storage mechanism.
While I would certainly recommend using a ContentProvider with a Sqlite database you could use any java based storage mechanism you wanted.
Data Interchange Format:
This is the format you use to send the data between your webserver and your android app. The two most popular formats these days are XML and JSON. When choosing your format, you should think about what sort of serialization libraries are available. I know off-hand that there's a fantastic library for json serialization called gson: http://code.google.com/p/google-gson/, although I'm sure similar libraries exist for XML.
Synchronization Service
You'll want some sort of asynchronous task which can get new data from your server and refresh the mobile content to reflect the content of the server. You'll also want to notify the server whenever you make local changes to content and want to reflect those changes. Android provides the SyncAdapter pattern as a way to easily solve this pattern. You'll need to register user accounts, and then Android will perform lots of magic for you, and allow you to automatically sync. Here's a good tutorial: http://www.c99.org/2010/01/23/writing-an-android-sync-provider-part-1/
As for how you identify if the records are the same, typically you'll create items with a unique id which you store both on the android device and the server. You can use that to make sure you're referring to the same reference. Furthermore, you can store column attributes like "updated_at" to make sure that you're always getting the freshest data, or you don't accidentally write over newly written data..
I am in the process of creating an android app (my first) that consumes a REST API.
I use a background job to fetch content and I plan to use a GET request with a from_id parameter in order to get more content. Of course anything fetched from the API gets stored in the SQLite db (I am using greendao) and the app only uses data that is already present there, in order to be snappy.
So, the question is: What happens if a given record is updated on the server? If records once read are cached, how come the app will notice that there are changes to sync? Which strategies are feasible solutions?
Thanks.
EDIT:
As Satish P points out in his answer, the client-server communication is handled with ETag (and I must add the possibility of using If-Modified-Since).
But my main concern, is how to mix this with the app UI. Given this example:
A list of elements, which have been retrieved from the REST service but client-side are read from the local database to make the app more responsive.
User clicks in one of those elements and a detailed view is show. Again, the data is loaded from the local database. I guess that at this point a GET request for the specific record is requested, either with ETag or If-Modified-Since headers.
It happens that the server returns a modified record, thus the local data is modified, so now it's time to update whatever the user is seeing.
Problem: If the detailed view is already populated because the local database read was already done when the remote request returns, how can I update the view? I don't think that just replacing current data with the fresher one is acceptable, the user would see a change out of the blue.
Satish's answer is absolutely right in terms of what you need your server to do. The gist is that it needs to support ETags and 304 response codes in case the content hasn't changed since the last time you got it from the server. On the client side now, there are essentially three strategies you can follow (each with it's own pros and cons):
Only use the cache if the content hasn't changed. That means you will always do a request and will display a progress bar to the user. If the server returns 304, then your content hasn't changed, and the request will be pretty fast (the moment you see that, you display the cached content). If the server actually returns new content, you continue showing the progress bar, and when the content is loaded you display the new content. The good thing about this is that the user will only ever see valid content, therefore avoiding a lot of headaches on your part. The bad thing is that the app does not appear that fast (especially if the content has changed and you are in a very slow connection).
Use only the cache for a predefined period and then fallback to first case. There are a couple of cache header to define that period ('max-age' and 'Expires'). Before that period you always use the cache (without doing a request), and after that you do a request and see if the content has changed. The good thing about this method is that for during the period mentioned above, the app is really fast. The bad thing is that there is a possibility that the user is looking at incorrect content.
Use both the cache and the network for a predefined period, and then fallback to the first case. You can use the cache headers mentioned earlier in a different way. Instead of only showing the cached content, you can actually display the cached content AND do a request in the background. If that request comes back with a 304, fine, else you will have to update you UI with the new data (expect two responses, one with the cached data and one with the newly retrieved data). The positive with this is that you get both a fast experience and valid data (most of the time). The negative is that you add a lot of complexity to your app (what happens if the user interacts with the stale data, and then a second response comes in etc).
All in all, every strategy is valid depending on the use case. For example, if the user can't interact with the screen that displays the data (like a tv program), the third option is pretty good. If it is crucial that the user sees correct data (a financial app let's say), then the first option is best. If speed is more important than having the latest data (a game or something) then the second option is your best choice.
How efficient the client can do caching is solely dependent on how much support you get from the REST API your client is accessing.
Using ETag is the industry standard to make caching on the client side more efficient and also server to serve the request faster. In short ETag is LIKE an MD5 hash of the content returned. More about ETag here: http://en.wikipedia.org/wiki/HTTP_ETag
If it is a popular API like Google, Facebook etc they inherently support ETags.
Please look at links below:
ETag usage best explained here: https://developers.facebook.com/docs/reference/ads-api/etags-reference
When the client does a GET on the a particular resource, the server when responding back with the content should include an ETag.
The client should store the ETag for that resource against the data cached.
Whenever the client is using the cache information, it should verify the cache using the ETag. Can work in multiple ways depending on the service implementation again
Make a usual GET on the resource and include the ETag as part of the request. If the content did not change the service will ideally no return any data but will give a specific code like (304 - Not Modified). Client knows that the cache is still valid and continues to use it
Make a HEAD call on the resource and the ETag is returned. It is part of the standard HTTP Specification. http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html (Refer 9.4). In this case Client will verify the ETag and decide whether or not to make the GET call.
Sample of a resource in the above explaination is like below
GET http://serverapi.com/employees/2312312312
The screen update could be handled fairly gracefully by Javascript, rewriting specific elements in the DOM -- optionally applying CSS formatting to call attention to the change in the UI -- if each datum in the UI has a unique ID/container ID or can otherwise be targeted by JS.
A simple/istic way to defeat some caching is to append a query string to the resource you want. For example, a file named testfile.csv can be accessed as easily at testfile.csv?12345 -- and the next time you want to bypass the cache, just update the query string, e.g., testfile.csv?23456. If updating a query string manually is arduous in your context, get a bit more clever at the cost of a modest hit in performance via PHP: call on the resource as testfile.csv to cause the query string to auto-update on every query after the resource is modified; the updated version gets served instead of the cached one.
Lately what I've been doing is using GraphQL with Apollo. It handles all this stuff automatically which is plain awesome.
I'm in the process of implementing my first ContentProvider. Since I'm quite inexperienced I wanted to check if my general design is sound.
So these are my requirements:
The web-service providing the original data might update the data it holds once per day. I can check for such a version change.
From the app's point of view, the data is read-only (think of weather or stocks data).
I already have a class that can take queries from the application domain, transform them into matching HTTP requests and asynchronously return domain objects. As you might guess, this is what the app currently uses to obtain data directly from an activity.
Now I want to do better and employ a ContentProvider to persist data locally.
Once per day the app would start a Service, that checks the data version on the server and when needed, update the data contained in the ContentProvider.
The following is the design that I've come up with:
Arrows indicate data-flow, not code-dependency.
I've noticed, I forgot to add a layer on top of the ContentProvider to convert Cursors to domain objects, but ignoring these details, do you think, this is a sound way of adding a ContentProvider to an app's architecture?
Any advice is greatly appreciated!