Develop Reference

LoadAppend In Paging library 3 is never called

I want to store in room database by triggering the online source , I have set up a remoteMediator to grab the data and save it into database , upon doing that it only triggers Refresh for first time and it only save the first page items but upon scrolling down , LoadAppen is never triggered not any data is saved , i need some help , Thank you
Mediator Class
#ExperimentalPagingApi
class PopularMediator(
private var movieDatabase: MovieDatabase,
private var moviesDao: MoviesDao,
private var itemsKeyDao: ItemsKeyDao,
private var authService: AuthService
) : RemoteMediator<Int, Result>() {
override suspend fun load(loadType: LoadType, state: PagingState<Int, Result>): MediatorResult {
return try {
val currentPage = when(loadType){
LoadType.REFRESH -> 1
LoadType.PREPEND -> {
return MediatorResult.Success(endOfPaginationReached = true)
}
LoadType.APPEND -> {
val key = movieDatabase.withTransaction {
itemsKeyDao.getAllKeys("Popular").lastOrNull()
}
if(key!!.nextKey == null){
return MediatorResult.Success(endOfPaginationReached = true)
}
key.nextKey
}
}
// GET DATA AND PUSH INTO DATABASE
Log.d("TAG","Current Paging Value $currentPage")
val response = authService.getPopular(Utils.MOVIE_API_KEY,"en-US", page = currentPage!!)
movieDatabase.withTransaction {
if(loadType == LoadType.REFRESH){
itemsKeyDao.deleteAllKeys()
}
response.results.forEach {
it.movieCategory = "Popular"
}
moviesDao.insertMovies(response.results)
response.results.forEach {
itemsKeyDao.insertItems(MovieItemKey(
category = "Popular",
nextKey = it.id,
previousKey = null
))
}
}
MediatorResult.Success(endOfPaginationReached = true)
}catch (ex : Exception){
MediatorResult.Error(ex)
}
}

Android Paging3 with RemoteMediator: items appear in the middle of the list or above visible ones, looks messy

I have a list of items from API. I'm using Paging3 with RemoteMedatior to maintain network with local caching in database using Room. My issue is when data is loading recyclerView looks messy and glitchy, items appear in the middle of the list and above visible ones, and recyclerView doesn't scroll to the top of the list. Looks like this.
I've tried with various values for pageSize, maxSize, and prefetchDistance for pageConfig but it doesn't help.
setHasFixedSize on recylerview also doesn't work.
Flow is collected with collectLatest in the fragment. I've tried also normal collect and changing flow into liveData. Same result. Any idea how to correct this?
load function in RemoteMediator
#ExperimentalPagingApi
class RecipesMediator(
private val recipesApi: RecipesApi,
private val appDatabase: RecipesDatabase,
private val queries: HashMap<String, String>,
) : RemoteMediator<Int, Recipe>() {
override suspend fun load(loadType: LoadType, state: PagingState<Int, Recipe>): MediatorResult {
val queryId = queries[QUERY_DIET] + queries[QUERY_MEAL]
val page = when (loadType) {
LoadType.REFRESH -> 0
LoadType.PREPEND -> return MediatorResult.Success(endOfPaginationReached = true)
LoadType.APPEND -> appDatabase.remoteKeysDao().remoteKeysRecipeId(queryId).nextKey
}
queries[QUERY_OFFSET] = page.toString()
queries[QUERY_NUMBER] = "20"
try {
val response = recipesApi.getRecipes(queries)
appDatabase.withTransaction {
if (loadType == LoadType.REFRESH) {
appDatabase.remoteKeysDao().clearRemoteKeys()
appDatabase.recipesDao().clearRecipes()
}
val nextKey = page + 20
val keys = response.recipes.map {
RemoteKeys(query = queryId, nextKey = nextKey)
}
appDatabase.remoteKeysDao().insertAll(keys)
appDatabase.recipesDao().insertAllRecipes(response.recipes)
}
return MediatorResult.Success(endOfPaginationReached = response.recipes.isEmpty())
} catch (exception: IOException) {
return MediatorResult.Error(exception)
} catch (exception: HttpException) {
return MediatorResult.Error(exception)
}
}
I use string for remote key ids, because when i used item id from API = LoadType.Append was called many times and only first page was loaded. Its explained in this issue
nextPage is "page + 20" because this API (Spoonacular) doesn't provide pagination, although provides parameters offset (number of results to skip) and number(number of expected results), so i used them for my paging.
UI states in fragment
recipesAdapter.addLoadStateListener { loadState ->
binding.apply {
recipesProgressBar.isVisible = loadState.mediator?.refresh is LoadState.Loading
recipesProgressBar.isVisible = loadState.source.refresh is LoadState.Loading
recipesRecyclerView.isVisible =loadState.mediator?.refresh is LoadState.NotLoading
recipesRecyclerView.isVisible =loadState.source.refresh is LoadState.NotLoading
buttonRecipesRetry.isVisible =loadState.mediator?.refresh is LoadState.Error
buttonRecipesRetry.isVisible =loadState.source.refresh is LoadState.Error
textViewRecipesError.isVisible =loadState.mediator?.refresh is LoadState.Error
textViewRecipesError.isVisible =loadState.source.refresh is LoadState.Error
if (loadState.source.refresh is LoadState.NotLoading &&
loadState.mediator?.refresh is LoadState.NotLoading &&
loadState.append.endOfPaginationReached &&
recipesAdapter.itemCount < 1
) {
recipesRecyclerView.isVisible = false
textViewEmpty.isVisible = true
} else {
textViewEmpty.isVisible = false
}
}
}
Remote Keys
#Entity(tableName = "remoteKeys")
data class RemoteKeys
(#PrimaryKey val query: String, val nextKey: Int)
Remote Keys Dao
#Dao
interface RemoteKeysDao {
#Insert(onConflict = OnConflictStrategy.REPLACE)
suspend fun insertAll(remoteKey: List<RemoteKeys>)
#Query("SELECT * FROM remoteKeys WHERE `query` = :query")
suspend fun remoteKeysId(query: String): RemoteKeys
#Query("DELETE FROM remoteKeys")
suspend fun clearRemoteKeys()
}

Paging 3 - How to handle old APPEND RemoteMediator requests that come trough since beta01?

Since version beta01 of Paging 3, when refreshing the PagingData from a RemoteMediator, it sometimes happens that the old APPEND request from the previous generation is still executed after the refresh has completed. This seems to be expected behavior reading from this commit.
When this happens, the old APPEND request calls the RemoteMediator's load method but with an outdated PagingState. This outdated PagingState can cause bugs and crashes if we are using the information in it in the load function (for example, the code snippet below uses lastItemOrNull to find the RemoteKeys for an item in the database). This breaking change (which also breaks the corresponding codelab) is not mentioned in the release notes at all. How are we supposed to handle this?
Here is an example of a RemoteMediator that breaks with beta01. The getRemoteKeyForLastItem method can return null (because the old PagingState is looking for a database entry that was deleted in the previous REFRESH) causing the InvalidObjectException to be thrown.
private const val GITHUB_STARTING_PAGE_INDEX = 1
#OptIn(ExperimentalPagingApi::class)
class GithubRemoteMediator(
private val query: String,
private val service: GithubService,
private val repoDatabase: RepoDatabase
) : RemoteMediator<Int, Repo>() {
override suspend fun load(loadType: LoadType, state: PagingState<Int, Repo>): MediatorResult {
val page = when (loadType) {
LoadType.REFRESH -> GITHUB_STARTING_PAGE_INDEX
LoadType.PREPEND -> return MediatorResult.Success(true)
LoadType.APPEND -> {
// this can run with an outdated PagingState from the previous RemoteMediator instance, causing the Exception to be thrown
val remoteKeys = getRemoteKeyForLastItem(state)
if (remoteKeys == null || remoteKeys.nextKey == null) {
throw InvalidObjectException("Remote key should not be null for $loadType")
}
remoteKeys.nextKey
}
}
val apiQuery = query + IN_QUALIFIER
try {
val apiResponse = service.searchRepos(apiQuery, page, state.config.pageSize)
val repos = apiResponse.items
val endOfPaginationReached = repos.isEmpty()
repoDatabase.withTransaction {
if (loadType == LoadType.REFRESH) {
repoDatabase.remoteKeysDao().clearRemoteKeys()
repoDatabase.reposDao().clearRepos()
}
val prevKey = if (page == GITHUB_STARTING_PAGE_INDEX) null else page - 1
val nextKey = if (endOfPaginationReached) null else page + 1
val keys = repos.map {
RemoteKeys(repoId = it.id, prevKey = prevKey, nextKey = nextKey)
}
repoDatabase.remoteKeysDao().insertAll(keys)
repoDatabase.reposDao().insertAll(repos)
}
return MediatorResult.Success(endOfPaginationReached = endOfPaginationReached)
} catch (exception: IOException) {
return MediatorResult.Error(exception)
} catch (exception: HttpException) {
return MediatorResult.Error(exception)
}
}
private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, Repo>): RemoteKeys? {
return state.pages.lastOrNull() { it.data.isNotEmpty() }?.data?.lastOrNull()
?.let { repo ->
repoDatabase.remoteKeysDao().remoteKeysRepoId(repo.id)
}
}
}

I talked to Dustin Lam and Yigit Boyar and apparently, the best way to handle this is to make prepend an append not depend on the PagingState. This means we should store our remote keys in a table related to a query rather than on the item level.
Exmaple:
#Entity(tableName = "search_query_remote_keys")
data class SearchQueryRemoteKey(
#PrimaryKey val searchQuery: String,
val nextPageKey: Int
)

How to go to a position with view pager 2 that uses paging 3 to load data?

I am using ViewPager2 to display data that I fetch from a server and save to a Room database using the Paging 3 library. My question is, how do I navigate to a specific view pager item through code? If I use viewPager.setCurrentItem(position, false) then this does not work. For example if there are 3000 items total which I dynamically load while swiping left/right, how do I set the position to 1000 and then navigate/load data in both directions from there? I cannot get this to work.
P.S: In the DogPagingMediator class below, I have also tried to set some starting number in the refresh block instead of the latest(highest) number, but when loading the app the view pager will only start at this position if higher numbered items don't exist locally in the database, otherwise it will always start at the highest numbered item regardless of the page returned in refresh(I assume since dogDao.getDogs() fetches all items in the database in descending order).
P.P.S: The reason why I am using live data and not flow is because flow for some reason causes NullPointerException when I swipe.
Code from onCreateView within the fragment containing the view pager:
lifecycleScope.launch {
// Fetch the latest dog item from the network (data is sorted by descending)
if (!browseDogsViewModel.latestDogIsFetched()) {
browseDogsViewModel.setLatestDogNumber()
}
browseDogsViewModel.pagingDataStream.observe(viewLifecycleOwner) {
adapter.submitData(viewLifecycleOwner.lifecycle, it)
}
}
From the view model:
val pagingDataStream = repository.getAllDogsPagingData()
suspend fun setLatestDogNumber() {
latestDogNumber = repository.getLatestDogNumber()
}
From the repository:
fun getAllDogsPagingData() = Pager(
config = PagingConfig(pageSize = PAGE_SIZE),
remoteMediator = dogPagingMediator,
pagingSourceFactory = { dogDao.getDogs() }
).liveData
The Mediator (similar to googles paging3 codelab example except it sorts by descending): https://codelabs.developers.google.com/codelabs/android-paging/#0):
#OptIn(ExperimentalPagingApi::class)
class DogPagingMediator #Inject constructor(
private val dogDatabase: DogDatabase,
private val dogDao: DogDao,
private val remoteKeysDao: RemoteKeysDao,
private val service: DogService,
) : RemoteMediator<Int, Dog>() {
override suspend fun load(loadType: LoadType, state: PagingState<Int, Dog>): MediatorResult {
try {
val page = when (loadType) {
LoadType.REFRESH -> {
val remoteKeys = getRemoteKeyClosestToCurrentPosition(state)
remoteKeys?.nextKey?.plus(PAGE_SIZE) ?: BrowseDogsViewModel.latestDogNumber
}
LoadType.PREPEND -> {
val remoteKeys = getRemoteKeyForFirstItem(state)
if (remoteKeys == null) {
// The LoadType is PREPEND so some data was loaded before,
// so we should have been able to get remote keys
// If the remoteKeys are null, then we're an invalid state and we have a bug
throw InvalidObjectException("Remote key and the prevKey should not be null")
}
// If the previous key is null, then we can't request more data
remoteKeys.prevKey
?: return MediatorResult.Success(endOfPaginationReached = true)
remoteKeys.prevKey
}
LoadType.APPEND -> {
val remoteKeys = getRemoteKeyForLastItem(state)
if (remoteKeys?.nextKey == null) {
throw InvalidObjectException("Remote key should not be null for $loadType")
}
remoteKeys.nextKey
}
}
val dogs: MutableList<Dog> = mutableListOf()
for (i in page downTo page - PAGE_SIZE) {
try {
val response = service.geDogWithNumber(i)
dogs.add(convertFromDto(response))
} catch (ex: HttpException) {
// Will be 404 when requesting a dog out of range
if (ex.code() != 404) {
throw ex
}
}
}
val endOfPaginationReached = dogs.isEmpty()
dogDatabase.withTransaction {
val prevKey =
if (page == BrowseDogsViewModel.latestDogNumber) null else page + PAGE_SIZE
val nextKey = if (endOfPaginationReached) null else page - PAGE_SIZE
val keys = dogs.map {
RemoteKeys(dogNum = it.number, prevKey = prevKey, nextKey = nextKey)
}
remoteKeysDao.insertAll(keys)
dogDao.insertAll(dogs)
}
return MediatorResult.Success(
endOfPaginationReached = endOfPaginationReached
)
} catch (exception: IOException) {
return MediatorResult.Error(exception)
} catch (exception: HttpException) {
return MediatorResult.Error(exception)
}
}
private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, Dog>): RemoteKeys? {
// Get the last page that was retrieved, that contained items.
// From that last page, get the last item
return state.pages.lastOrNull { it.data.isNotEmpty() }?.data?.lastOrNull()
?.let { dog->
// Get the remote keys of the last item retrieved
remoteKeysDao.remoteKeysDogNum(dog.number)
}
}
private suspend fun getRemoteKeyForFirstItem(state: PagingState<Int, Dog>): RemoteKeys? {
// Get the first page that was retrieved, that contained items.
// From that first page, get the first item
return state.pages.firstOrNull { it.data.isNotEmpty() }?.data?.firstOrNull()
?.let { dog->
// Get the remote keys of the first items retrieved
remoteKeysDao.remoteKeysDogNum(dog.number)
}
}
private suspend fun getRemoteKeyClosestToCurrentPosition(
state: PagingState<Int, Dog>
): RemoteKeys? {
// The paging library is trying to load data after the anchor position
// Get the item closest to the anchor position
return state.anchorPosition?.let { position ->
state.closestItemToPosition(position)?.number?.let { num ->
remoteKeysDao.remoteKeysDogNum(num)
}
}
}
private fun convertFromDto(dogDto: DogDto): Dog {
return Dog(...)
}
}
adapter:
class DogPagingAdapter() :
PagingDataAdapter<Dog, DogPagingAdapter.ViewPagerViewHolder>(DogDiffUtilCallback) {
override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ViewPagerViewHolder {
return ViewPagerViewHolder(
ItemDogViewPagerBinding.inflate(
LayoutInflater.from(parent.context),
parent,
false
)
)
}
override fun onBindViewHolder(holder: ViewPagerViewHolder, position: Int) {
holder.bind(getItem(position))
}
inner class ViewPagerViewHolder(private val binding: ItemDogViewPagerBinding) :
RecyclerView.ViewHolder(binding.root) {
fun bind(dog: Dog?) {
binding.dog = dog
binding.executePendingBindings()
}
}
}

The official documentation says:
Set the currently selected page. If the ViewPager has already been
through its first layout with its current adapter there will be a
smooth animated transition between the current item and the specified
item. Silently ignored if the adapter is not set or empty. Clamps
item to the bounds of the adapter.
Before you call this method you must ensure that Nth item exists in your adapter.

I managed to achieve this by creating my own paging source. As you already mentioned, the default paging source of a dao will start loading at the beginning of the table. Therefore you have to create a paging source which handles loading data from the database from a specific position.
To do that have to implement a paging source that queries the database with an offset and limit:
PagingSource
class DogsPagingSource(
private val database: DogDatabase,
private val dogDao: DogDao,
private var startPos: Int
) : PagingSource<Int, Dog>() {
// to show new loaded data, you have to invalidate the paging source after data in db changed
init {
val tableObserver = object : InvalidationTracker.Observer("dogs") {
override fun onInvalidated(tables: MutableSet<String>) {
invalidate()
}
}
database.invalidationTracker.addObserver(tableObserver)
}
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Dog> {
return try {
// params.key is null when paging source gets loaded the first time
// for this case use your start position = number of rows that should be skipped in your table (offset)
val position =
if (params.key == null) startPos
else maxOf(0, params.key ?: 0)
// load one item from your db by using a limit-offset-query
// limit is your page size which has to be 1 to get this working
val dogs = dogDao.getDogsPage(params.loadSize, position)
// to load further or previous data just in-/decrease position by 1
// if you are at the start/end of the table set prev-/nextKey to null to notify RemoteMediator to load new data
// nextKey = null will call APPEND, prevKey = null will call PREPEND in your RemoteMediator
Page(
data = dogs,
prevKey = if (position == 0) null else position.minus(1),
nextKey = if (position == dogDao.count()) null
else position.plus(1),
)
} catch (e: IOException) {
LoadResult.Error(e)
} catch (e: HttpException) {
LoadResult.Error(e)
}
}
override fun getRefreshKey(state: PagingState<Int, Item>): Int? =
state.anchorPosition?.let {
state.closestPageToPosition(it)?.prevKey?.minus(1)?.minValue(0)
}
override val jumpingSupported: Boolean
get() = true
}
For this you also will need a new query for your dogs table:
DAO
#Dao
interface DogDao {
...
// just define the order which the data should be arranged by. limit and offset will do the rest
#Query("SELECT * FROM `dogs` ORDER BY id ASC LIMIT :limit OFFSET :offset")
suspend fun getDogsPage(limit: Int, offset: Int): List<Dog>
#Query("SELECT COUNT(*) FROM `dogs`")
suspend fun count(): Int
}
This only works if your page size is 1, but this should be the case if you use it for a viewpager. If not, you have to modify the paging source a little bit.

Paging 3 - Why does my retry footer not call my PagingSource's load method?

I've implemented Paging 3 into my app following a codelab and added a footer with a retry button via withLoadStateHeaderAndFooter:
recycler_view_results.adapter = adapter.withLoadStateHeaderAndFooter(
header = UnsplashLoadStateAdapter { adapter.retry() },
footer = UnsplashLoadStateAdapter { adapter.retry() }
)
When I click the retry button in my footer's ViewHolder, adapter.retry() is indeed called, so the setup there is correct. However, this method never ends up calling my PagingSource's load method as it normally should.
My PagingSource (I checked that the LoadResult.Error is returned correctly in an error case):
class UnsplashPagingSource(
private val unsplashApi: UnsplashApi,
private val query: String
) : PagingSource<Int, UnsplashPhoto>() {
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, UnsplashPhoto> {
val position = params.key ?: UNSPLASH_STARTING_PAGE_INDEX
return try {
val response = unsplashApi.searchPhotos(query, position, params.loadSize)
val photos = response.results
LoadResult.Page(
data = photos,
prevKey = if (position == UNSPLASH_STARTING_PAGE_INDEX) null else position - 1,
nextKey = if (photos.isEmpty()) null else position + 1
)
} catch (exception: IOException) {
return LoadResult.Error(exception)
} catch (exception: HttpException) {
return LoadResult.Error(exception)
}
}
}
My repository:
class UnsplashRepository #Inject constructor(private val unsplashApi: UnsplashApi) {
fun getSearchResultStream(query: String): Flow<PagingData<UnsplashPhoto>> {
return Pager(
config = PagingConfig(
pageSize = NETWORK_PAGE_SIZE,
enablePlaceholders = false
),
pagingSourceFactory = { UnsplashPagingSource(unsplashApi, query) }
).flow
}
companion object {
private const val NETWORK_PAGE_SIZE = 20
}
}
And in my fragment I do this:
private fun searchPhotos(query: String) {
searchJob?.cancel()
searchJob = lifecycleScope.launch {
viewModel.searchPhotos(query).collectLatest {
adapter.submitData(it)
}
}
}
Interestingly the retry button for an empty list works:
retry_button.setOnClickListener {
adapter.retry()
// this works
}

It works now after I updated the paging dependency from '3.0.0-alpha02' to '3.0.0-alpha03'. Looks like this was a bug in the library.
Afterward I also found the corresponding bug report: https://issuetracker.google.com/issues/160194384