I am fetching a content of an Internet feed and storing it into a local database (caching). the feed does not have a tag which will tell me when a specific entry in it was created (no date tag). All it has is a title, content, url and image.
Now, I want to make this: if there is no new content on the feed, the app will load from the cache during the specific session. But if I see that there is a new content on the feed, then the app will load from the feed and all later loads (during one session) will go from the cache.
How can I detect in this specific case if there is a new content on the feed or not? Shall I do string comparison of the first X lines of both cache and remote feed? Or there is a better and proper way to do it?
Using your approach you will have to download the entries and compare them to the data stored in your cache. This pretty much renders the cache useless, as one of the things you want to prevent by using a caching mechanism is unnecessary downloads.
Are you sure that there are no header fields that you can use to check for changes? Good candidates are Last-Modified, If-Modified-Since and If-Not-Modified. See Wikipedia's list of header fields for more detailed information on these fields.
Related
About Single Sources of Truth Google document said:
Using this model, the database serves as the single source of truth, and other parts of the app access it using our UserRepository. Regardless of whether you use a disk cache, we recommend that your repository designate a data source as the single source of truth for the rest of your app
https://developer.android.com/jetpack/guide?gclid=CjwKCAjwo4mIBhBsEiwAKgzXOH1Pq--Ws1PLzUiSP4RmDE6ByKfEi6mdXu5g86btqveIdJvvrgYuxBoCz8wQAvD_BwE&gclsrc=aw.ds#connect-viewmodel-repository
According to the document I save all data when I fected data from remote server and I only get data from room When I need to use in acitivty(In fact I collect flow which is defined in viewmodel).
It seems so good! It avoids the different data sources mix up together! But actually I found some strange question gradually:
In my App, I have a list that the server may change it(Because we have data manager website that admin can update or delete data). So in order to get the newest list data from server, I must clear all data stored in room and fect data again from remote server. This operation seems redundant: "why could I get data directly from remote server", I mean, I only get data from remote source is also a single sources truth. And also it cause a promble: my app will flash a moment because clear data make list empty and fect data from server make list full!
The most important thing is that it seems like the local data is not necessary because I must stay the newest list from remote server.
Some people may say that save data into room can make us app available offlice. I agree that, But in this place, my item of list is represent a image url, and after click the item, the app will jump to a new activity and display a ImageView base on the url we get from the list. If app offlice, the ImageView couldn't load the url also.
I am so confused I couldn't load all image url(use base64-url to avoid load invalid) in a moment also, because the data is so much. And if I say I need a search function in this list and I need load so much unbelievable data into my room, It seems so unreal and event fantasy!
In brief：
Room is a nessary? Couldnt just fect data from remote?
If room is nessary, how to solve problem I met, do my incorrect useage cause the problem?
Hi #psycongroo as I Understood your problem, and I want to share my experience:
You can handle any error with loading URL with placeholder I mean if you got an error with no Internet connection user will see placeholder, but in general libs like Picasso or Glide can cache images when it`s load one time, so the user will see the Image.
The question about why we need to use room instead of fetch data from remote directly. So from your question I don`t understand why you need to drop your local changes even they are completely new, user can have a low internet connection so he will see an empty list instead of previous data with for example progress indicator. And also if the user doesn't have the internet at all you can show some dialog to explain what the problem but old data is still present. If you are using, for example, RecyclerView you can update data with Paging 3 from google, and they update the only necessary items from your list.
P.S. let me know if that help, or you have another question.
Context
I have a news activity which contains a ListView . Each item in the list contains some text and one image. Now every time the user opens the activity, the application makes a request to the server for getting data. Unfortunately if the user doesn't have an active internet connection, I simply put a Toast saying "Please check your internet connection"
Requirements
I would like to show news article from the last session if the user doesn't have an active internet connection. I define session as the time when user launches the activity and exits it by pressing back button.
Libraries used
I am using Retrofit Library for making Http Requests and Picasso for loading images.
What I have tried so far
While exploring Retrofit, I found out that retrofit by default caches the response for some time. So without any internet connection also I am able to get the data. In Picasso also, I found that we can save the image to any location on the device.
How to go for its Implementation ?
Since it will be a big change in my application, I wanted to know what is the best way to enable offline reading using the above two libraries? How do other applications manage to do so? Also if I can get some references or some blogs regarding this implementation, then it would be great.
Use the internal storage of the application.
Parse the image to a byteArray and the text to a single String so you can easily convert this also to a byteArray.
Name the files so you can easily retrieve them and link them back together.
You can storage the last updated data on local, with SharedPreferences or json file.
Instead of show toast, you can load the last updated data from local.
Picasso is a good choice for images, it saves images locally and reuse it automatically later.
For your items, I suggest to use a small local DataBase : in Android, we use SQLiteDataBase. here is a small tutorial : http://www.androidhive.info/2011/11/android-sqlite-database-tutorial/
So, I suggest you this pattern
user opens activity
system retrieves data from database
in parallel, system starts to download the new items.
When new items have been downloaded, you should notice user like 9gag or Facebook apps do.
I am in the process of creating an android app (my first) that consumes a REST API.
I use a background job to fetch content and I plan to use a GET request with a from_id parameter in order to get more content. Of course anything fetched from the API gets stored in the SQLite db (I am using greendao) and the app only uses data that is already present there, in order to be snappy.
So, the question is: What happens if a given record is updated on the server? If records once read are cached, how come the app will notice that there are changes to sync? Which strategies are feasible solutions?
Thanks.
EDIT:
As Satish P points out in his answer, the client-server communication is handled with ETag (and I must add the possibility of using If-Modified-Since).
But my main concern, is how to mix this with the app UI. Given this example:
A list of elements, which have been retrieved from the REST service but client-side are read from the local database to make the app more responsive.
User clicks in one of those elements and a detailed view is show. Again, the data is loaded from the local database. I guess that at this point a GET request for the specific record is requested, either with ETag or If-Modified-Since headers.
It happens that the server returns a modified record, thus the local data is modified, so now it's time to update whatever the user is seeing.
Problem: If the detailed view is already populated because the local database read was already done when the remote request returns, how can I update the view? I don't think that just replacing current data with the fresher one is acceptable, the user would see a change out of the blue.
Satish's answer is absolutely right in terms of what you need your server to do. The gist is that it needs to support ETags and 304 response codes in case the content hasn't changed since the last time you got it from the server. On the client side now, there are essentially three strategies you can follow (each with it's own pros and cons):
Only use the cache if the content hasn't changed. That means you will always do a request and will display a progress bar to the user. If the server returns 304, then your content hasn't changed, and the request will be pretty fast (the moment you see that, you display the cached content). If the server actually returns new content, you continue showing the progress bar, and when the content is loaded you display the new content. The good thing about this is that the user will only ever see valid content, therefore avoiding a lot of headaches on your part. The bad thing is that the app does not appear that fast (especially if the content has changed and you are in a very slow connection).
Use only the cache for a predefined period and then fallback to first case. There are a couple of cache header to define that period ('max-age' and 'Expires'). Before that period you always use the cache (without doing a request), and after that you do a request and see if the content has changed. The good thing about this method is that for during the period mentioned above, the app is really fast. The bad thing is that there is a possibility that the user is looking at incorrect content.
Use both the cache and the network for a predefined period, and then fallback to the first case. You can use the cache headers mentioned earlier in a different way. Instead of only showing the cached content, you can actually display the cached content AND do a request in the background. If that request comes back with a 304, fine, else you will have to update you UI with the new data (expect two responses, one with the cached data and one with the newly retrieved data). The positive with this is that you get both a fast experience and valid data (most of the time). The negative is that you add a lot of complexity to your app (what happens if the user interacts with the stale data, and then a second response comes in etc).
All in all, every strategy is valid depending on the use case. For example, if the user can't interact with the screen that displays the data (like a tv program), the third option is pretty good. If it is crucial that the user sees correct data (a financial app let's say), then the first option is best. If speed is more important than having the latest data (a game or something) then the second option is your best choice.
How efficient the client can do caching is solely dependent on how much support you get from the REST API your client is accessing.
Using ETag is the industry standard to make caching on the client side more efficient and also server to serve the request faster. In short ETag is LIKE an MD5 hash of the content returned. More about ETag here: http://en.wikipedia.org/wiki/HTTP_ETag
If it is a popular API like Google, Facebook etc they inherently support ETags.
Please look at links below:
ETag usage best explained here: https://developers.facebook.com/docs/reference/ads-api/etags-reference
When the client does a GET on the a particular resource, the server when responding back with the content should include an ETag.
The client should store the ETag for that resource against the data cached.
Whenever the client is using the cache information, it should verify the cache using the ETag. Can work in multiple ways depending on the service implementation again
Make a usual GET on the resource and include the ETag as part of the request. If the content did not change the service will ideally no return any data but will give a specific code like (304 - Not Modified). Client knows that the cache is still valid and continues to use it
Make a HEAD call on the resource and the ETag is returned. It is part of the standard HTTP Specification. http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html (Refer 9.4). In this case Client will verify the ETag and decide whether or not to make the GET call.
Sample of a resource in the above explaination is like below
GET http://serverapi.com/employees/2312312312
The screen update could be handled fairly gracefully by Javascript, rewriting specific elements in the DOM -- optionally applying CSS formatting to call attention to the change in the UI -- if each datum in the UI has a unique ID/container ID or can otherwise be targeted by JS.
A simple/istic way to defeat some caching is to append a query string to the resource you want. For example, a file named testfile.csv can be accessed as easily at testfile.csv?12345 -- and the next time you want to bypass the cache, just update the query string, e.g., testfile.csv?23456. If updating a query string manually is arduous in your context, get a bit more clever at the cost of a modest hit in performance via PHP: call on the resource as testfile.csv to cause the query string to auto-update on every query after the resource is modified; the updated version gets served instead of the cached one.
Lately what I've been doing is using GraphQL with Apollo. It handles all this stuff automatically which is plain awesome.
I have an application that reads information of songs and singers from Mysql DB and display it in listView ... my question(s)
is it a good idea to cache the data that the user already retrieved so he does not need to retrieve it every time?
how to do that chasing if it is a good idea(give me hints only)?
how to match between what is already cached and what is need to be retrieved from Mysql
It's always a good idea to cache values coming from external sources, as it reduces the time the application takes to display them.
This can be implemented in multiple ways. Generally said:
Binary data (e.g. Images) should be cached on internal/external storage. See "Caching Bitmaps"
Textual data (e.g. Articles) should be cached in memory (for example in a Map)
That said, the important and tricky part of a caching system is to determine, when a cached values is no longer "new" enough. While choosing the specific criteria depends on your application and personal taste, here are some general hints:
The age of the data (you can give server-side hints with the HTTP Response-Header Expires)
The priority of the data. The more important the data, the more up-to-date it should be.
The likelihood of any changes at all. The comments on a article are more likely to change frequently than the article itself.
A nice implementation would be to just delegate all network requests to a caching-aware method/class (implemented from the above hints), which would then decide if the request needed to be done in the first place and return either fresh or cached data.
I have an Android application which talks to a public Data-API by calling URLs. The API returns XML which describes search results or detailed Data of a particular dataset.
In my Application i retrieve this data and parse it to Java Objects to display them in a ListView for example.
edit: Just to make it clearer: In my Application you can search for music Artists and get their whole discographic information. I Retrieve the list of Releases and display them in a ListView. (No problem right here, because the results of the same search request can change any minute i have to retrieve this data everytime a search request is issued).
Now i have a list with all the LPs the Beatles produced (for example). I can then click one particular LP and view the Details such as the Release Year and the Tracklist.
What i want to cache is the Details data and i'm currently thinking of which is the best way to do this. I thought of:
Retrieving the XML data once and store the XML Data in the SQLite Database (that would imply, that i have to parse the data everytime i want to access it again).
Retrieving the XML data once, parsing it once and somehow store the serialized JavaObject into the SQLite Database as ByteStream. (so all the time consuming work would be done just once).
What do you think is the best version or is there maybe another better way to achieve caching the results?
serializing an object would be quick solution but that could not be effective solution. Every a time you need to load entire object, while in this case if you are storing your data set into database then, using cursor/queries data handling will be smoother.
CursorAdapter will allow you to plug database cursor directly to list in GUI. I would suggest you to use database approach.