I'm using the RemoteMediator in an app to load page keyed data. Everything works fine, except when after process death, the data is refreshed.
My current implementation is :
val results = savedStateHandle.get<String>("query").flatMapLatest { query ->
repository.getPager(
query = query,
)
}.cachedIn(viewModelScope)
I do know about the initialize() function of RemoteMediator, but how do I tie it in with process death?
As you found out, .cachedIn just operates in memory, so it won't survive process death. You cannot rely on Paging's internal cache of items in memory for this, you need to cache the loaded items on disk.
I would recommend using something like Room or some dedicated persistence layer that is actually built to handle large lists of arbitrary data classes.
I would not recommend to try to serialize and stash the entire list of data into SavedState as this could become prohibitively expensive quite quickly.
For your other point on RemoteMediator - it is just a "dumb" callback which has no influence on what Paging actually loads or displays. It's simply a way for your to write custom logic which is triggered during edge-case conditions in Paging. You probably only want this if you are already using a layered approach and trying to skip remote REFRESH. If that is your case, the RemoteMediator.intiailize function is guaranteed to complete before Paging starts loading, which means you can check whether you are coming from SavedState and there is already cached data, and if so, you can skip remote REFRESH by returning InitializeAction.SKIP_INITIAL_REFRESH.
Related
i have a data class
data class Holder(uri: String, title: String, desc: String, source: String, color: String?) that i have from screen a, that i want to pass along to screen b. i have the code set up as follows:
#Composable
fun A(navigateToScreenB: (holder: Holder) -> Unit) {...}
#Composable
fun B(holder: Holder) {...}
in my nav file:
composable(
route = Screen.b.route
) {
B(how to get `Holder` to pass here)
}
composable(
route = Screen.A.route
) {
A(
navigateToScreenB = { it ->
// `it` is my data class `Holder`, but how to pass it to screen b?
navController.navigate(Screen.B.route)
}
)
}
any insight on this?
tl/dr: you shouldn't be passing the object at all, the same way you don't see a website that is example.com/data/{the whole object in the URL}, you'd see example.com/data/{a unique ID used to retrieve the object}
As per this thread, if the object exists only at the UI layer (i.e., it isn't coming from a remote database or any other source of truth), you can just pass the object directly by following the documentation and making your class Parcelable and serializing it to include it as part of your Screen.b.route just like any other argument.
However, you've said that your Holder object actually comes from a remote database. That same thread goes on to discuss exactly this case:
A better way to start the conversation is figuring out how StoreList is going to recreate its list after process death and recreation. Your objects have been wiped from memory entirely, which means that ViewModel needs to be talking to the actual source of truth (i.e., your repository layer - note that doesn't mean "save to disk" or "database", it just means the layer responsible for getting your data)
Once you have a repository layer anyways (since a ViewModel shouldn't be doing network calls or disk access directly), that's also the layer that can do in memory caching
Keep in mind that every list already has a way to uniquely define each item by the index in the list
So at that point you already have everything you need - a repository both destinations can talk to that is the single source of truth for Store objects and an index into the list that you can pass to the details screen to define you element
And if you start persisting those objects into a database, the rest of the layers don't have to care
(of course, you probably will add a unique ID to each element if you do add it to a database at some point, which then makes the index based approach unnecessary)
So how you should actually be doing this is:
Create a data layer that is responsible for loading your data. No other layer should know that your data is actually from a remote database. Okhttp caching is a simple way to add caching to prevent redownloading data in a way that is transparent to the other layers.
Both Screen A and Screen B would talk to the same data layer, generally, as per the UI Layer documentation, would be by using a state holder such as a ViewModel that is responsible for loading data from the data layer.
2a) Screen A would request the entire list of Holder objects from the data layer, which would load the entire list from the remote database.
2b) Screen B would request just a single Holder object, either by using the index in the list or a unique key if one is available (it isn't clear from your Holder class if a unique key exists)
Instead of passing the entire Holder as part of Screen B's route argument, you'd pass only that unique key, which could simply be the index in the list or something more complicated if your remote database has such a unique key (note that a Uri would need to be encoded via Uri.encode if included in a route)
This approach follows the approach of a single source of truth. If you later change your data layer to store data locally, none of the rest of your layers need to change. If Screen B gains the ability to edit your Holder objects, then Screen A (assuming your data layer uses a Flow or similar observable structure) would automatically update, without needing to pass data back or anything complicated like that.
It also means that cases which are generally hard to handle, such as configuration changes (i.e., rotating your device) or process death and recreation (which can happen at any screen, meaning you can't rely on Screen A to have loaded your data into memory) are also handled without any additional work.
I am looking for a way to update specific items in my PagingDataAdapter from the Paging 3 library. The recommended way at the moment seems to be to invalidate the PagingSource but this causes the adapter to fetch the whole data set again, which is not efficient and also shows my loading spinner.
However, I noticed that I can access and modify items in the adapter using the peek() method and it seems to work quite well. Am I missing anything here? Will this fall apart in certain scenarios? I know that it's good practice to keep data classes immutable but this approach makes my life a lot easier.
Here is an example of my usage and it seems to work quite well:
viewModel.chatMessageUpdateEvents.collect { messageEvent ->
when (messageEvent) {
is FirestoreChatMessageListener.ChatMessageUpdateEvent.MessageUpdate -> {
val update = messageEvent.chatMessage
val historyAdapterItems = chatMessagesHistoryAdapter.snapshot().items
val updatedMessage =
historyAdapterItems.find { chatMessage ->
chatMessage.documentId == messageEvent.chatMessage.documentId
}
if (updatedMessage != null) {
val messagePosition = historyAdapterItems.indexOf(updatedMessage)
chatMessagesHistoryAdapter.peek(messagePosition)?.unsent = update.unsent
chatMessagesHistoryAdapter.peek(messagePosition)?.imageUrl = update.imageUrl
chatMessagesHistoryAdapter.notifyItemChanged(messagePosition)
}
}
}
}
I replied in a separate comment but wanted to post here for visibility.
This is really not recommended and a completely unsupported usage of paging.
One of the primary ways of restoring state if Pager().flow() hasn't been cleaned up (say if ViewModel hasn't been cleared yet) is via the .cachedIn(scope) method, which will cache out-of-date data in your case. This is also the only way to multicast (make the loaded data in PagingData re-usable) for usage in Flow operations like .combine() that allow you to mix transformations with external signals.
You'll also need to handle races between in-flight loads, what happens if you get a messageEvent the same time an append finishes? Who wins in this case and is it possible between taking the .snapshot() a new page is inserted so your notify position is no longer correct?
In general it's much simpler to have a single source of truth and this is the recommended path, so the advice has always been to invalidate on every backing dataset update.
There is an open FR in Paging's issue tracker to support Flow<Item> or Flow<Page> style data to allow granular updates, but it's certainly a farther future thing: https://issuetracker.google.com/160232968
I use Android Architecture Components to build my app. There is Paging Library to load items with Room generated DataSource. Also there is BoundaryCallback to get new data from server and store it in the database. It works fine, all is reactive, changes in the database come into PagedList.
But now I need to these items get some additional data, some calculations before they come into PagesList and RecyclerView. These calculations is not so fast to executing them on main thread in RecyclerView ViewHolder (actually I need to get additional data from the database or even from the server). So I supposed that I need to write my custom DataSource and make calculations there and then pass these processed items to PagedList.
I created my ItemKeyedDataSource (I'm not sure this is correct, because I load data from database, but this data source type is designed for network, but I don't think this is critical), and make queries in Dao that return List of items. After I got a "page", I make calculations to items and then pass it to callback. It works, PagedList gets processed items.
But unfortunately there is no reactivity with this approach. No changes in database come to my PagedList. I tried to return LiveData<List> from Dao and add observeForever() listener in DataSource but it fails since you can't run it on background thread.
I watched Room generated DataSource.Factory and LimitOffsetDataSource but it doesn't look good to me since you need to pass table names to observe changes and other unclear things.
I suppose that I need to use invalidate(), but I don't because I have no idea where it should be.
There is 3 main questions:
Is it right to process items in DataSource before they come to RecyclerView or there is a better place?
Should I use PositionalDataSource instead of ItemKeyedDataSource?
How can I add Room reactivity to custom DataSource?
It seems that I've found a mistake in my DataSource.Factory. Instead of creating DataSource object in create() method I just returned object which was passed to that factory (I saw it in one popular article on Medium). And because of that I couldn't invalidate my DataSource. But now I create DataSource in that method and invalidation works.
The only problem is to understand where and when to invalidate. For now I've found some workaround: make a query in Dao that returns LiveData of last item, and then observe it in my Activity to understand that data was modified and call invalidate(). But I'm not sure this is a good solution. Maybe you know a better one.
You may add invalidationTracker in your DataSource:
dbRoom.getInvalidationTracker().addObserver(
object : InvalidationTracker.Observer("your_table") {
override fun onInvalidated(#NonNull tables: Set<String>) {
invalidate()
}
})
For the first time I want to retrieve data from server cache it and next times show data on UI from local storage and request from server and update local storage and UI as
I have tried
(getCachedData()).concatWith(getRemoteData())
getCachedData returns Single
return apiSeResource.getData()
.doAfterSuccess { response ->
saveData(response.body())
}
}
.onErrorReturn {
return#onErrorReturn emptyList()
}
}```
The problem with `concat` is that the subsequent observable doesn't even start until the first Observable completes. That can be a problem. We want all observables to start simultaneously but produce the results in a way we expect.
I can use `concatEager` : It starts both observables but buffers the result from the latter one until the former Observable finishes.
Sometimes though, I just want to start showing the results immediately.
I don't necessarily want to "wait" on any Observable. In these situations, we could use the `merge` operator.
However the problem with merge is: if for some strange reason an item is emitted by the cache or slower observable after the newer/fresher observable, it will overwrite the newer content.
So none of mentioned above solution is not proper ,what is your solution?
Create 2 data sources one local data source and one remote and use the flatMap for running the Obervables. You can publish the data from the cache and when u get data from remote save data to cache and publish.
Or you can also try Observable.merge(dataRequestOne, dataRequestTwo) . run both the Observables on different threads
I'm still fairly new to RxJava and I'm using it in an Android application. I've read a metric ton on the subject but still feel like I'm missing something.
I have the following scenario:
I have data stored in the system which is accessed via various service connections (AIDL) and I need to retrieve data from this system (1-n number of async calls can happen). Rx has helped me a ton in simplifying this code. However, this entire process tends to take a few seconds (upwards of 5 seconds+) therefore I need to cache this data to speed up the native app.
The requirements at this point are:
Initial subscription, the cache will be empty, therefore we have to wait the required time to load. No big deal. After that the data should be cached.
Subsequent loads should pull the data from cache, but then the data should be reloaded and the disk cache should be behind the scenes.
The Problem: I have two Observables - A and B. A contains the nested Observables that pull data from the local services (tons going on here). B is much simpler. B simply contains the code to pull the data from disk cache.
Need to solve:
a) Return a cached item (if cached) and continue to re-load the disk cache.
b) Cache is empty, load the data from system, cache it and return it. Subsequent calls go back to "a".
I've had a few folks recommend a few operations such as flatmap, merge and even subjects but for some reason I'm having trouble connecting the dots.
How can I do this?
Here are a couple options on how to do this. I'll try to explain them as best I can as I go along. This is napkin-code, and I'm using Java8-style lambda syntax because I'm lazy and it's prettier. :)
A subject, like AsyncSubject, would be perfect if you could keep these as instance states in memory, although it sounds like you need to store these to disk. However, I think this approach is worth mentioning just in case you are able to. Also, it's just a nifty technique to know.
AsyncSubject is an Observable that only emits the LAST value published to it (A Subject is both an Observer and an Observable), and will only start emitting after onCompleted has been called. Thus, anything that subscribes after that complete will receive the next value.
In this case, you could have (in an application class or other singleton instance at the app level):
public class MyApplication extends Application {
private final AsyncSubject<Foo> foo = AsyncSubject.create();
/** Asynchronously gets foo and stores it in the subject. */
public void fetchFooAsync() {
// Gets the observable that does all the heavy lifting.
// It should emit one item and then complete.
FooHelper.getTheFooObservable().subscribe(foo);
}
/** Provides the foo for any consumers who need a foo. */
public Observable<Foo> getFoo() {
return foo;
}
}
Deferring the Observable. Observable.defer lets you wait to create an Observable until it is subscribed to. You can use this to allow the disk cache fetch to run in the background, and then return the cached version or, if not in cache, make the real deal.
This version assumes that your getter code, both cache fetch and non- catch creation, are blocking calls, not observables, and the defer does work in the background. For example:
public Observable<Foo> getFoo() {
Observable.defer(() -> {
if (FooHelper.isFooCached()) {
return Observable.just(FooHelper.getFooFromCacheBlocking());
}
return Observable.just(FooHelper.createNewFooBlocking());
}).subscribeOn(Schedulers.io());
}
Use concatWith and take. Here we assume our method to get the Foo from the disk cache either emits a single item and completes or else just completes without emitting, if empty.
public Observable<Foo> getFoo() {
return FooHelper.getCachedFooObservable()
.concatWith(FooHelper.getRealFooObservable())
.take(1);
}
That method should only attempt to fetch the real deal if the cached observable finished empty.
Use amb or ambWith. This is probably one the craziest solutions, but fun to point out. amb basically takes a couple (or more with the overloads) observables and waits until one of them emits an item, then it completely discards the other observable and just takes the one that won the race. The only way this would be useful is if it's possible for the computation step of creating a new Foo to be faster than fetching it from disk. In that case, you could do something like this:
public Observable<Foo> getFoo() {
return Observable.amb(
FooHelper.getCachedFooObservable(),
FooHelper.getRealFooObservable());
}
I kinda prefer Option 3. As far as actually caching it, you could have something like this at one of the entry points (preferably before we're gonna need the Foo, since as you said this is a long-running operation) Later consumers should get the cached version as long as it has finished writing. Using an AsyncSubject here may help as well, to make sure we don't trigger the work multiple times while waiting for it to be written. The consumers would only get the completed result, but again, that only works if it can be reasonably kept around in memory.
if (!FooHelper.isFooCached()) {
getFoo()
.subscribeOn(Schedulers.io())
.subscribe((foo) -> FooHelper.cacheTheFoo(foo));
}
Note that, you should either keep around a single thread scheduler meant for disk writing (and reading) and use .observeOn(foo) after .subscribeOn(...), or otherwise synchronize access to the disk cache to prevent concurrency issues.
I’ve recently published a library on Github for Android and Java, called RxCache, which meets your needs about caching data using observables.
RxCache implements two caching layers -memory and disk, and it counts with several annotations in order to configure the behaviour of every provider.
It is highly recommended to use with Retrofit for data retrieved from http calls. Using lambda expression, you can formulate expression as follows:
rxCache.getUser(retrofit.getUser(id), () -> true).flatmap(user -> user);
I hope you will find it interesting :)
Take a look at the project below. This is my personal take on things and I have used this pattern in a number of apps.
https://github.com/zsiegel/rxandroid-architecture-sample
Take a look at the PersistenceService. Rather than hitting the database (or MockService in the example project) you could simply have a local list of users that are updated with the save() method and just return that in the get().
Let me know if you have any questions.