Develop Reference

Android Paging 3 remoteMediator infinitely load(APPEND) data with Compose

I'm trying to implement a list with Paging 3 library using RemoteMediator.
Initial loading is OK. Mediator loaded only 3 pages, which is set in 'PageConfig'. In my case, there are 10 data per page.
But when scroll the list to trigger extra loading for more data, then Mediator starts loading data infinitely until it returns MediatorResult.Success(endOfPaginationReached = true) (Which means all data in remote was loaded). Even though scrolling is stopped.
I have no idea what makes Mediator keep loading.
I want Mediator to load data only needed along scrolling.
Here is my code:
#OptIn(ExperimentalPagingApi::class)
class PostRemoteMediator(
private val postApi: ApiInterface,
private val database: PostDatabase
) : RemoteMediator<Int, Post>() {
override suspend fun load(loadType: LoadType, state: PagingState<Int, Post>): MediatorResult {
return try {
val userId = when (loadType) {
LoadType.REFRESH -> {
logd(">> loadType.REFRESH")
STARTING_USER_ID
}
LoadType.PREPEND -> {
return MediatorResult.Success(endOfPaginationReached = true)
}
LoadType.APPEND -> {
logd(">> loadType.APPEND")
val lastItem = state.lastItemOrNull()
?: return MediatorResult.Success(endOfPaginationReached = true)
lastItem.userId + 1
}
}
logd(">> load data with userId = $userId")
val response = postApi.getUserPosts(userId)
database.withTransaction {
if (loadType == LoadType.REFRESH) {
database.postsDao().clearAll()
}
database.postsDao().insertAll(response?.body() ?: emptyList())
}
MediatorResult.Success(
endOfPaginationReached = response.body().isNullOrEmpty()
)
} catch (e: IOException) {
MediatorResult.Error(e)
} catch (e: HttpException) {
MediatorResult.Error(e)
}
}
}
#OptIn(ExperimentalPagingApi::class)
class PostRepositoryImpl #Inject constructor(
private val remoteApi: ApiInterface,
private val database: PostDatabase
) : PostRepository {
override fun getUserPosts(): Flow<PagingData<Post>> {
return Pager(
config = PagingConfig(
pageSize = 1
),
remoteMediator = PostRemoteMediator(
remoteApi,
database
)
) {
// returns all data in table as PagingSource<Int, Post>
database.postsDao().getPosts()
}.flow
}
}
#HiltViewModel
class PostViewModel #Inject constructor(
private val postRepository: PostRepository
) : ViewModel() {
private val TAG = PostViewModel::class.simpleName
val postFlow: Flow<PagingData<Post>> = postRepository.getUserPosts().cachedIn(viewModelScope)
}
This is UI code:
#Composable
fun PostList(postsFlow: Flow<PagingData<Post>>) {
val posts = postsFlow.collectAsLazyPagingItems()
LazyColumn(contentPadding = PaddingValues(horizontal = 8.dp)) {
items(posts, key = { it.id }) { post ->
post?.also { PostItem(userId = it.userId, id = it.id, content = it.body) }
}
posts.apply {
when {
loadState.mediator?.refresh is LoadState.Loading -> {
item { LoadingView(modifier = Modifier.fillParentMaxSize()) }
}
loadState.mediator?.append is LoadState.Loading -> {
item { LoadingView(modifier = Modifier.wrapContentHeight()) }
}
loadState.mediator?.refresh is LoadState.Error -> {
val error = loadState.refresh as LoadState.Error
item { ErrorView(error.error.localizedMessage!!, modifier = Modifier.fillParentMaxSize()) { retry() } }
}
loadState.mediator?.append is LoadState.Error -> {
val error = loadState.append as LoadState.Error
item { ErrorView(error.error.localizedMessage!!, modifier = Modifier.wrapContentHeight()) { retry() } }
}
}
}
}
}
Thanks for any answer

Increase the pageSize when creating the PageConfig.
According to the documentation:
Should be several times the number of visible items onscreen.
Configuring your page size depends on how your data is being loaded and used. Smaller page sizes improve memory usage, latency, and avoid GC churn. Larger pages generally improve loading throughput, to a point (avoid loading more than 2MB from SQLite at once, since it incurs extra cost).
If you're loading data for very large, social-media style cards that take up most of a screen, and your database isn't a bottleneck, 10-20 may make sense. If you're displaying dozens of items in a tiled grid, which can present items during a scroll much more quickly, consider closer to 100.

Data not updating with Paging 3 and Firestore

So i have a RecyclerView and i fetch all the data, then if a user clicks on some buttons it triggers the same function but with a different filter.
The issue i'm having is it won't clear the old data when the new query returns empty.
So if it load all the data and they i click on a filter that has no data, it won't clear the data that is the screen.
I tested it and when i select a filter that has data it will clear the old data and show the new one.
I tried this with room and it work as inteded out of the box but i don't know how to make this with Firestore.
Any ideas on how can i fix this?
This is my repo, here it checks if it has to fetch the data from Firestore or Room.
override suspend fun getEventPagingDataFlow(filter: String): Flow<PagingData<Event>> = withContext(Dispatchers.IO){
Pager(
PagingConfig(
pageSize = 30,
enablePlaceholders = true,
maxSize = 200)
) {
if(currentBusinessType() == "ONLINE"){
val query = firestoreQueries.getEventsCollectionQuery()
if (filter == "ALL") FirestoreEventsPagingSource(query)
else FirestoreEventsPagingSource(query.whereEqualTo("status", filter))
}
else{
if (filter == "ALL") eventsDao.getAllPaged()
else eventsDao.getPagedEventsWithFilter(filter)
}
}.flow
}
This is the Firestore paging source
class FirestoreEventsPagingSource(
private val query: Query
) : PagingSource<QuerySnapshot, Event>() {
override suspend fun load(params: LoadParams<QuerySnapshot>): LoadResult<QuerySnapshot, Event> {
return try {
val currentPage = params.key ?: query
.get()
.await()
val lastDocumentSnapshot = currentPage.documents[currentPage.size() - 1]
val nextPage = query.startAfter(lastDocumentSnapshot)
.get()
.await()
LoadResult.Page(
data = currentPage.toObjects(Event::class.java),
prevKey = null,
nextKey = nextPage
)
} catch (e: Exception) {
LoadResult.Error(e)
}
}
override fun getRefreshKey(state: PagingState<QuerySnapshot, Event>): QuerySnapshot? {
return null
}
}
Here is where i update the adapter:
lifecycleScope.launchWhenStarted {
viewModel.events.collectLatest {
calendarEventsAdapter.submitData(it)
}
}

How to go to a position with view pager 2 that uses paging 3 to load data?

I am using ViewPager2 to display data that I fetch from a server and save to a Room database using the Paging 3 library. My question is, how do I navigate to a specific view pager item through code? If I use viewPager.setCurrentItem(position, false) then this does not work. For example if there are 3000 items total which I dynamically load while swiping left/right, how do I set the position to 1000 and then navigate/load data in both directions from there? I cannot get this to work.
P.S: In the DogPagingMediator class below, I have also tried to set some starting number in the refresh block instead of the latest(highest) number, but when loading the app the view pager will only start at this position if higher numbered items don't exist locally in the database, otherwise it will always start at the highest numbered item regardless of the page returned in refresh(I assume since dogDao.getDogs() fetches all items in the database in descending order).
P.P.S: The reason why I am using live data and not flow is because flow for some reason causes NullPointerException when I swipe.
Code from onCreateView within the fragment containing the view pager:
lifecycleScope.launch {
// Fetch the latest dog item from the network (data is sorted by descending)
if (!browseDogsViewModel.latestDogIsFetched()) {
browseDogsViewModel.setLatestDogNumber()
}
browseDogsViewModel.pagingDataStream.observe(viewLifecycleOwner) {
adapter.submitData(viewLifecycleOwner.lifecycle, it)
}
}
From the view model:
val pagingDataStream = repository.getAllDogsPagingData()
suspend fun setLatestDogNumber() {
latestDogNumber = repository.getLatestDogNumber()
}
From the repository:
fun getAllDogsPagingData() = Pager(
config = PagingConfig(pageSize = PAGE_SIZE),
remoteMediator = dogPagingMediator,
pagingSourceFactory = { dogDao.getDogs() }
).liveData
The Mediator (similar to googles paging3 codelab example except it sorts by descending): https://codelabs.developers.google.com/codelabs/android-paging/#0):
#OptIn(ExperimentalPagingApi::class)
class DogPagingMediator #Inject constructor(
private val dogDatabase: DogDatabase,
private val dogDao: DogDao,
private val remoteKeysDao: RemoteKeysDao,
private val service: DogService,
) : RemoteMediator<Int, Dog>() {
override suspend fun load(loadType: LoadType, state: PagingState<Int, Dog>): MediatorResult {
try {
val page = when (loadType) {
LoadType.REFRESH -> {
val remoteKeys = getRemoteKeyClosestToCurrentPosition(state)
remoteKeys?.nextKey?.plus(PAGE_SIZE) ?: BrowseDogsViewModel.latestDogNumber
}
LoadType.PREPEND -> {
val remoteKeys = getRemoteKeyForFirstItem(state)
if (remoteKeys == null) {
// The LoadType is PREPEND so some data was loaded before,
// so we should have been able to get remote keys
// If the remoteKeys are null, then we're an invalid state and we have a bug
throw InvalidObjectException("Remote key and the prevKey should not be null")
}
// If the previous key is null, then we can't request more data
remoteKeys.prevKey
?: return MediatorResult.Success(endOfPaginationReached = true)
remoteKeys.prevKey
}
LoadType.APPEND -> {
val remoteKeys = getRemoteKeyForLastItem(state)
if (remoteKeys?.nextKey == null) {
throw InvalidObjectException("Remote key should not be null for $loadType")
}
remoteKeys.nextKey
}
}
val dogs: MutableList<Dog> = mutableListOf()
for (i in page downTo page - PAGE_SIZE) {
try {
val response = service.geDogWithNumber(i)
dogs.add(convertFromDto(response))
} catch (ex: HttpException) {
// Will be 404 when requesting a dog out of range
if (ex.code() != 404) {
throw ex
}
}
}
val endOfPaginationReached = dogs.isEmpty()
dogDatabase.withTransaction {
val prevKey =
if (page == BrowseDogsViewModel.latestDogNumber) null else page + PAGE_SIZE
val nextKey = if (endOfPaginationReached) null else page - PAGE_SIZE
val keys = dogs.map {
RemoteKeys(dogNum = it.number, prevKey = prevKey, nextKey = nextKey)
}
remoteKeysDao.insertAll(keys)
dogDao.insertAll(dogs)
}
return MediatorResult.Success(
endOfPaginationReached = endOfPaginationReached
)
} catch (exception: IOException) {
return MediatorResult.Error(exception)
} catch (exception: HttpException) {
return MediatorResult.Error(exception)
}
}
private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, Dog>): RemoteKeys? {
// Get the last page that was retrieved, that contained items.
// From that last page, get the last item
return state.pages.lastOrNull { it.data.isNotEmpty() }?.data?.lastOrNull()
?.let { dog->
// Get the remote keys of the last item retrieved
remoteKeysDao.remoteKeysDogNum(dog.number)
}
}
private suspend fun getRemoteKeyForFirstItem(state: PagingState<Int, Dog>): RemoteKeys? {
// Get the first page that was retrieved, that contained items.
// From that first page, get the first item
return state.pages.firstOrNull { it.data.isNotEmpty() }?.data?.firstOrNull()
?.let { dog->
// Get the remote keys of the first items retrieved
remoteKeysDao.remoteKeysDogNum(dog.number)
}
}
private suspend fun getRemoteKeyClosestToCurrentPosition(
state: PagingState<Int, Dog>
): RemoteKeys? {
// The paging library is trying to load data after the anchor position
// Get the item closest to the anchor position
return state.anchorPosition?.let { position ->
state.closestItemToPosition(position)?.number?.let { num ->
remoteKeysDao.remoteKeysDogNum(num)
}
}
}
private fun convertFromDto(dogDto: DogDto): Dog {
return Dog(...)
}
}
adapter:
class DogPagingAdapter() :
PagingDataAdapter<Dog, DogPagingAdapter.ViewPagerViewHolder>(DogDiffUtilCallback) {
override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ViewPagerViewHolder {
return ViewPagerViewHolder(
ItemDogViewPagerBinding.inflate(
LayoutInflater.from(parent.context),
parent,
false
)
)
}
override fun onBindViewHolder(holder: ViewPagerViewHolder, position: Int) {
holder.bind(getItem(position))
}
inner class ViewPagerViewHolder(private val binding: ItemDogViewPagerBinding) :
RecyclerView.ViewHolder(binding.root) {
fun bind(dog: Dog?) {
binding.dog = dog
binding.executePendingBindings()
}
}
}

The official documentation says:
Set the currently selected page. If the ViewPager has already been
through its first layout with its current adapter there will be a
smooth animated transition between the current item and the specified
item. Silently ignored if the adapter is not set or empty. Clamps
item to the bounds of the adapter.
Before you call this method you must ensure that Nth item exists in your adapter.

I managed to achieve this by creating my own paging source. As you already mentioned, the default paging source of a dao will start loading at the beginning of the table. Therefore you have to create a paging source which handles loading data from the database from a specific position.
To do that have to implement a paging source that queries the database with an offset and limit:
PagingSource
class DogsPagingSource(
private val database: DogDatabase,
private val dogDao: DogDao,
private var startPos: Int
) : PagingSource<Int, Dog>() {
// to show new loaded data, you have to invalidate the paging source after data in db changed
init {
val tableObserver = object : InvalidationTracker.Observer("dogs") {
override fun onInvalidated(tables: MutableSet<String>) {
invalidate()
}
}
database.invalidationTracker.addObserver(tableObserver)
}
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Dog> {
return try {
// params.key is null when paging source gets loaded the first time
// for this case use your start position = number of rows that should be skipped in your table (offset)
val position =
if (params.key == null) startPos
else maxOf(0, params.key ?: 0)
// load one item from your db by using a limit-offset-query
// limit is your page size which has to be 1 to get this working
val dogs = dogDao.getDogsPage(params.loadSize, position)
// to load further or previous data just in-/decrease position by 1
// if you are at the start/end of the table set prev-/nextKey to null to notify RemoteMediator to load new data
// nextKey = null will call APPEND, prevKey = null will call PREPEND in your RemoteMediator
Page(
data = dogs,
prevKey = if (position == 0) null else position.minus(1),
nextKey = if (position == dogDao.count()) null
else position.plus(1),
)
} catch (e: IOException) {
LoadResult.Error(e)
} catch (e: HttpException) {
LoadResult.Error(e)
}
}
override fun getRefreshKey(state: PagingState<Int, Item>): Int? =
state.anchorPosition?.let {
state.closestPageToPosition(it)?.prevKey?.minus(1)?.minValue(0)
}
override val jumpingSupported: Boolean
get() = true
}
For this you also will need a new query for your dogs table:
DAO
#Dao
interface DogDao {
...
// just define the order which the data should be arranged by. limit and offset will do the rest
#Query("SELECT * FROM `dogs` ORDER BY id ASC LIMIT :limit OFFSET :offset")
suspend fun getDogsPage(limit: Int, offset: Int): List<Dog>
#Query("SELECT COUNT(*) FROM `dogs`")
suspend fun count(): Int
}
This only works if your page size is 1, but this should be the case if you use it for a viewpager. If not, you have to modify the paging source a little bit.

Paging 3 - Why does my retry footer not call my PagingSource's load method?

I've implemented Paging 3 into my app following a codelab and added a footer with a retry button via withLoadStateHeaderAndFooter:
recycler_view_results.adapter = adapter.withLoadStateHeaderAndFooter(
header = UnsplashLoadStateAdapter { adapter.retry() },
footer = UnsplashLoadStateAdapter { adapter.retry() }
)
When I click the retry button in my footer's ViewHolder, adapter.retry() is indeed called, so the setup there is correct. However, this method never ends up calling my PagingSource's load method as it normally should.
My PagingSource (I checked that the LoadResult.Error is returned correctly in an error case):
class UnsplashPagingSource(
private val unsplashApi: UnsplashApi,
private val query: String
) : PagingSource<Int, UnsplashPhoto>() {
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, UnsplashPhoto> {
val position = params.key ?: UNSPLASH_STARTING_PAGE_INDEX
return try {
val response = unsplashApi.searchPhotos(query, position, params.loadSize)
val photos = response.results
LoadResult.Page(
data = photos,
prevKey = if (position == UNSPLASH_STARTING_PAGE_INDEX) null else position - 1,
nextKey = if (photos.isEmpty()) null else position + 1
)
} catch (exception: IOException) {
return LoadResult.Error(exception)
} catch (exception: HttpException) {
return LoadResult.Error(exception)
}
}
}
My repository:
class UnsplashRepository #Inject constructor(private val unsplashApi: UnsplashApi) {
fun getSearchResultStream(query: String): Flow<PagingData<UnsplashPhoto>> {
return Pager(
config = PagingConfig(
pageSize = NETWORK_PAGE_SIZE,
enablePlaceholders = false
),
pagingSourceFactory = { UnsplashPagingSource(unsplashApi, query) }
).flow
}
companion object {
private const val NETWORK_PAGE_SIZE = 20
}
}
And in my fragment I do this:
private fun searchPhotos(query: String) {
searchJob?.cancel()
searchJob = lifecycleScope.launch {
viewModel.searchPhotos(query).collectLatest {
adapter.submitData(it)
}
}
}
Interestingly the retry button for an empty list works:
retry_button.setOnClickListener {
adapter.retry()
// this works
}

It works now after I updated the paging dependency from '3.0.0-alpha02' to '3.0.0-alpha03'. Looks like this was a bug in the library.
Afterward I also found the corresponding bug report: https://issuetracker.google.com/issues/160194384

InvalidObjectException("Remote key and the prevKey should not be null") while creating RemoteMediator android

I am trying to cache list from api response and sync the data with the server, I followed up with Codelab to create the single source of truth as android documentation shows up using paging 3, I followed the steps and i was surprised with the results, but when i tried to cache the paging data i'v got this thrown error while just the first ever running and continue work when the app crashes or while the first running without internet connection InvalidObjectException("Remote key and the prevKey should not be null") and it seams came from :
private suspend fun getRemoteKeyForFirstItem(state: PagingState<Int, MohItem>): RemoteKeys? {
return state.pages.firstOrNull { it.data.isNotEmpty() }?.data?.firstOrNull()
?.let { mohItem ->
database.remoteKeysDao().remoteKeysId(mohItem.id)
}
}
It's convenience for me that it's since it's trying to insert keys which isn't received from empty or null list as the snipped code shows up :
#ExperimentalPagingApi
class MohRemoteMediator(
private val context: Context,
private val database: IbnsinaDatabase,
private val apiService: ApiService,
private val query: MohSearchQueryRequest
) : RemoteMediator<Int, MohItem>() {
override suspend fun load(
loadType: LoadType,
state: PagingState<Int, MohItem>
): MediatorResult {
val page = when (loadType) {
LoadType.REFRESH -> {
val remoteKeys = getRemoteKeyClosestToCurrentPosition(state)
remoteKeys?.nextKey?.minus(1) ?: Constants.PAGING_STARTING_PAGE_INDEX
}
LoadType.PREPEND -> {
val remoteKeys = getRemoteKeyForFirstItem(state)
?: throw InvalidObjectException("Remote key and the prevKey should not be null")
remoteKeys.prevKey ?: return MediatorResult.Success(endOfPaginationReached = true)
remoteKeys.prevKey
}
LoadType.APPEND -> {
val remoteKeys = getRemoteKeyForLastItem(state)
if (remoteKeys?.nextKey == null) {
throw InvalidObjectException("Remote key should not be null for $loadType")
}
remoteKeys.nextKey
}
}
try {
val apiResponse = apiService.getMohList(
pageIndex = page,
title = query.title,
number = query.number,
month = query.month,
year = query.year
)
val mohs = apiResponse.data ?: emptyList()
val endOfPaginationReached = mohs.isEmpty()
database.withTransaction {
if (loadType == LoadType.REFRESH) {
database.remoteKeysDao().clearRemoteKeys()
database.mohDao().clearMohs()
}
val prevKey = if (page == Constants.PAGING_STARTING_PAGE_INDEX) null else page - 1
val nextKey = if (endOfPaginationReached) null else page + 1
val keys = mohs.map {
RemoteKeys(mohId = it.id, prevKey = prevKey, nextKey = nextKey)
}
database.remoteKeysDao().insertAll(keys)
database.mohDao().insertAllMohs(mohs)
}
return MediatorResult.Success(endOfPaginationReached = endOfPaginationReached)
} catch (exception: IOException) {
return MediatorResult.Error(Throwable(context.getString(R.string.no_internet_connection)))
} catch (exception: HttpException) {
return MediatorResult.Error(exception)
}
}
private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, MohItem>): RemoteKeys? {
return state.pages.firstOrNull { it.data.isNotEmpty() }?.data?.firstOrNull()
?.let { repo ->
// Get the remote keys of the first items retrieved
database.remoteKeysDao().remoteKeysId(repo.id)
}
}
private suspend fun getRemoteKeyForFirstItem(state: PagingState<Int, MohItem>): RemoteKeys? {
return state.pages.firstOrNull { it.data.isNotEmpty() }?.data?.firstOrNull()
?.let { mohItem ->
database.remoteKeysDao().remoteKeysId(mohItem.id)
}
}
private suspend fun getRemoteKeyClosestToCurrentPosition(
state: PagingState<Int, MohItem>
): RemoteKeys? {
return state.anchorPosition?.let { position ->
state.closestItemToPosition(position)?.id?.let { repoId ->
database.remoteKeysDao().remoteKeysId(repoId)
}
}
}
}

Do you actually want to support PREPEND here or is the first page you load always going to be the first page (no prepend from network)?
This code in your RemoteMediator:
val prevKey = if (page == Constants.PAGING_STARTING_PAGE_INDEX) null else page - 1
seems to imply that your prevKey will always be null after initial REFRESH so that in your load method,
LoadType.PREPEND -> {
val remoteKeys = getRemoteKeyForFirstItem(state)
?: throw InvalidObjectException("Remote key and the prevKey should not be null")
remoteKeys.prevKey ?: return MediatorResult.Success(endOfPaginationReached = true)
remoteKeys.prevKey
}
will always throw after initial REFRESH?
If you don't support PREPEND from network (you can still set maxSize and drop, then reload from PagingSource), then the simplest fix would be to return MediatorResult.Success(true) on PREPEND in RemoteMediator.load() immediately, instead of trying to fetch a key that should never exist.

This is the issue in all the version of Paging above 3.0.0-alpha02.
So you can change your paging library version to 3.0.0-alpha02. I have also created an issue on google issue tracker, you can track the progress of the issue.
https://issuetracker.google.com/issues/170025945