Develop Reference

Paging 3 refresh in the middle

I'm having a problem with refreshing the paged data and I'm not sure how I need to set the refresh key so it works correctly. The docs aren't clear at all.
I have this base class for offset paging, so it goes 0-40, 40-60, 60-80, and so on. And that works, but when I'm in the middle of the collection and I want to refresh the data, either with invalidate() or adapter.refresh(), it crashes with following message:
java.lang.IllegalStateException: The same value, 40, was passed as the prevKey in two sequential Pages loaded from a PagingSource. Re-using load keys in PagingSource is often an error, and must be explicitly enabled by overriding PagingSource.keyReuseSupported.
When it is enabled, it doesn't crash, but its behavior is weird because it starts paging from the middle and I can't go back to the beginning of the collection, as it is constantly paging the same items.
Example:
prevKey null, nextKey: 40 -> prevKey 40, nextKey: 60 -> prevKey 60,
nextKey: 80
Invalidate()
prevKey 40, nextKey: 80 -> prevKey 40, nextKey: 60 // This is the case
when it crashes without the keyReuseSupported flag.
And as I want to go back to the beginning it is stuck on prevKey 40, nextKey: 60
abstract class OffsetPagingSource<Value : Any>(
private val coroutineScope: CoroutineScope
) : PagingSource<Int, Value>() {
abstract suspend fun queryFactory(
size: Int,
after: Int,
itemCount: Boolean
): PagingResult<Value>?
abstract suspend fun subscriptionFactory(): Flow<Any>?
init {
initSubscription()
}
private fun initSubscription() {
coroutineScope.launch {
try {
subscriptionFactory()?.collect {
invalidate()
}
} catch (e: Throwable) {}
}
}
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Value> {
val size = params.loadSize
val after = params.key ?: 0
return try {
val result = queryFactory(size, after, true) ?: return LoadResult.Page(emptyList(), null, null)
val totalCount = result.pageInfo.itemCount
val nextCount = after + size
val prevKey = if (after == 0) null else after
val nextKey = if (nextCount < totalCount) nextCount else null
LoadResult.Page(result.edges.mapNotNull { it.node }, prevKey = prevKey, nextKey)
} catch (e: Exception) {
LoadResult.Error(e)
}
}
override fun getRefreshKey(state: PagingState<Int, Value>): Int? {
return state.anchorPosition?.let { anchorPosition ->
val anchorPage = state.closestPageToPosition(anchorPosition)
anchorPage?.prevKey
}
}
}
Does anyone know how to fix this? I'm not sure if this is a bug or not. Paging 3 kinds of forces a way the paging works and not enabling a different approach.

The exception being thrown is warning you that you are using the same key to load different data / pages. In your example after invalidate:
{ prevKey 40, nextKey: 80}, { prevKey 40, nextKey: 60 }
Implies that you would want to use params.key = 40 for the page before the first page and also to reload the first page. I.e., you have:
page0: not loaded yet
page1: { prevKey 40, nextKey: 80 }
page2: { prevKey 40, nextKey: 60 }
Implies you want to use params.key to load both page0 and page1. This might be what you intend, but is generally a mistake which is what that exception is warning you about.
The reason you are probably reloading the same page over and over again, is that you interpret prepending and appending with the same logic that loads from key...key+pageSize and always passing prevKey = key. You probably need to either check for LoadParams is LoadParams.Prepend and offset the key in your load logic, or offset the key before you pass it back to Paging as prevKey.

Android Paging 3 library PagingSource invalidation, causes the list to jump due to wrong refresh key (not using room)

Since I'm currently working on a Project with custom database (not Room), I'm testing whether we could use the Paging 3 library in the Project.
However, I run into the issue, that if you make changes to the data and therefore invalidate the paging source, the recreation of the list is buggy and jumps to a different location. This is happening because the Refresh Key calculation seems to be wrong, which is most likely caused by the fact that the initial load loads three pages worth of data, but puts it into one page.
The default Paging Source looks like this:
override fun getRefreshKey(state: PagingState<Int, CustomData>): Int? {
// Try to find the page key of the closest page to anchorPosition, from
// either the prevKey or the nextKey, but you need to handle nullability
// here:
// * prevKey == null -> anchorPage is the first page.
// * nextKey == null -> anchorPage is the last page.
// * both prevKey and nextKey null -> anchorPage is the initial page, so
// just return null.
return state.anchorPosition?.let { anchorPosition ->
val anchorPage = state.closestPageToPosition(anchorPosition)
anchorPage?.prevKey?.plus(1) ?: anchorPage?.nextKey?.minus(1)
}
}
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, CustomData> {
var pagePosition = params.key ?: STARTING_PAGE_INDEX
var loadSize = params.loadSize
return try {
val dataResult = dataRepository.getPagedData(
pagePosition = pagePosition,
loadSize = loadSize,
pageSize = pageSize
)
val nextKey = if (dataResult.isEmpty() || dataResult.size < pageSize) {
null
} else {
pagePosition + (loadSize / pageSize)
}
Log.i(
"RoomFreePagingSource",
"page $pagePosition with size $loadSize publish ${dataResult.size} routes"
)
return LoadResult.Page(
data = dataResult,
prevKey = if (pagePosition == STARTING_PAGE_INDEX) null else pagePosition - 1,
nextKey = nextKey
)
} catch (exception: Exception) {
LoadResult.Error(exception)
}
}
The dataRepository.getPagedData() function simply accesses a in memory list and returns a subsection of the list, simulating the paged data.
For completeness here is the implementation of this function:
fun getPagedData(pageSize:Int, loadSize:Int, pagePosition: Int): List<CustomData>{
val startIndex = pagePosition * pageSize
val endIndexExl =startIndex + loadSize
return data.safeSubList(startIndex,endIndexExl).map { it.copy() }
}
private fun <T> List<T>.safeSubList(fromIndex: Int, toIndex: Int) : List<T>{
// only returns list with valid range, to not throw exception
if(fromIndex>= this.size)
return emptyList()
val endIndex = if(toIndex> this.size) this.size else toIndex
return subList(fromIndex,endIndex)
}
The main Problem I currently face is, that the getRefreshKey function doesn't return a correct refresh page key, which results in the wrong page being refreshed and the list jumping to the loaded page.
For example if you have 10 items per page.
The first page 0 contains 30 items
The next page would be 3 and contains 10 items.
If you don't scroll (only see the first 7 items) and invalidate then the anchorPosition will be 7 and the refresh key will be 2.
(anchorPage.prevKey = null => anchorPage.nextKey = 3 => 3-1 is 2)
However, at this point we would like to load page 0 not page 2.
Knowing the cause of the issue I tried to adapt the default implementation to resolve it and came up with a lot of different versions. The one below currently works best, but still causes the jumping from time to time. And also sometimes a part of the list flickers, which is probably caused when not enough items are fetched to fill the view port.
override fun getRefreshKey(state: PagingState<Int, CustomData>): Int? {
return state.anchorPosition?.let { anchorPosition ->
val closestPage = state.closestPageToPosition(anchorPosition)?.prevKey
?: STARTING_PAGE_INDEX
val refKey = if(anchorPosition>(closestPage)*pageSize + pageSize)
closestPage+1
else
closestPage
Log.i(
"RoomFreePagingSource",
"getRefreshKey $refKey from anchorPosition $anchorPosition closestPage $closestPage"
)
refKey
}
}
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, CustomData> {
var pagePosition = params.key ?: STARTING_PAGE_INDEX
var loadSize = pageSize
return try {
when (params) {
is LoadParams.Refresh -> {
if (pagePosition > STARTING_PAGE_INDEX) {
loadSize *= 3
} else if (pagePosition == STARTING_PAGE_INDEX) {
loadSize *= 2
}
}
else -> {}
}
val dataResult = dataRepository.getPagedData(
pagePosition = pagePosition,
loadSize = loadSize,
pageSize = pageSize
)
val nextKey = if (dataResult.isEmpty() || dataResult.size < pageSize) {
null
} else {
pagePosition + (loadSize / pageSize)
}
Log.i(
"RouteRepository",
"page $pagePosition with size $loadSize publish ${dataResult.size} routes"
)
return LoadResult.Page(
data = dataResult,
prevKey = if (pagePosition == STARTING_PAGE_INDEX) null else pagePosition - 1,
nextKey = nextKey
)
} catch (exception: Exception) {
LoadResult.Error(exception)
}
}
In theory I found that the best solution would be to just calculate the refresh PageKey like this anchorPosition/pageSize then load this page, the one before and after it. Hower, calculating the refresh key like this does not work since the anchorPosition isn't the actual position of the item in the list. After some invalidations, the anchor could be 5 even if you are currently looking at item 140 in the list.
So to sum it up:
How can I calculate the correct refresh page after invalidation when I don't use Room, but another data source, like in this sample case a in memory list which I access through getPagedData?

Found the solution my self in the meantime, but forgot to add the answer here.
The anchorPosition in getRefreshKey() was wrong, because the the itemsBefore argument was not set for each Page returned by the paging source. This behaviour does not seem to be documented.
LoadResult.Page(
itemsBefore = pagePosition * pageSize,
data = dataResult,
prevKey = if (pagePosition == STARTING_PAGE_INDEX) null else pagePosition - 1,
nextKey = nextKey
)
Now one can actually determine the refresh key based on the anchorPosition and the pageSize(defined as PagingSource constructor parameter in this example, but can also be a constant etc.)
override fun getRefreshKey(state: PagingState<Int, CustomData>): Int? {
return state.anchorPosition?.let { anchorPosition ->
anchorPosition / pageSize
}
}
No the list does not jump any longer when refreshing/ invalidating it. However sometimes the list might flickr or a whole part of the list won't be visible when invalidating. This is due to the fact, that the refresh page is too small and does not cover the whole view port. Hence one can see the paging in action for the items outside of the refresh page. This behaviour can be fixed by making the refresh page (loadSize) 3 times larger and reducing the pagePosition by one. 3 times is necessary, as the page before and after the actual refresh page could be visible at in the view port.
if (params is LoadParams.Refresh) {
loadSize *= 3
pagePosition = max(0, pagePosition - 1)
}
This solution works just fine now, however, it feels like this can't/ shouldn't be the official solution to this problem. As all these adaptations are not necessary when using paging and room. So, I'm open to other/ better solutions.
Here the full code of working PagingSource with all mentioned changes:
override fun getRefreshKey(state: PagingState<Int, CustomData>): Int? {
return state.anchorPosition?.let { anchorPosition ->
anchorPosition / pageSize
}
}
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, CustomData> {
var pagePosition = params.key ?: STARTING_PAGE_INDEX
var loadSize = pageSize
return try {
if (params is LoadParams.Refresh) {
// make sure everything visible in the view port is updated / loaded
loadSize *= 3
pagePosition = max(0, pagePosition - 1)
}
val dataResult = dataRepository.getPagedData(
pagePosition = pagePosition,
loadSize = loadSize,
pageSize = pageSize
)
val nextKey = if (dataResult.isEmpty() || dataResult.size < pageSize) {
null
} else {
pagePosition + (loadSize / pageSize)
}
LoadResult.Page(
itemsBefore = pagePosition * pageSize,
data = dataResult,
prevKey = if (pagePosition == STARTING_PAGE_INDEX) null else pagePosition - 1,
nextKey = nextKey
)
} catch (exception: Exception) {
LoadResult.Error(exception)
}
}

You can define initial load size in PagingConfig object when creating Pager. Like
Pager(
config = PagingConfig(pageSize = PAGE_SIZE, initialLoadSize = PAGE_SIZE),
initialKey = INITIAL_PAGE,
remoteMediator = mediator,
pagingSourceFactory = {
...
})

Android Paging 3 library loading infinitely without scroll with Jetpack Compose

I'm attempting to make a paged list of books using Jetpack Compose and Android's Paging 3 library. I am able to make the paged list and get the data fine, but the load() function of my paging data source is being called infinitely, without me scrolling the screen.
My paging data source looks like this:
class GoogleBooksBookSource #Inject constructor(
private val googleBooksRepository: GoogleBooksRepository,
private val query: String
): PagingSource<Int, Book>() {
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Book> {
val position = params.key ?: 0
return try {
val response = googleBooksRepository.searchForBookStatic(query, position)
if (response is Result.Success) {
LoadResult.Page(
data = response.data.items,
prevKey = if (position == 0) null else position - 1,
nextKey = if (response.data.totalItems == 0) null else position + 1
)
} else {
LoadResult.Error(Exception("Error loading paged data"))
}
} catch (e: Exception) {
Log.e("PagingError", e.message.toString())
return LoadResult.Error(e)
}
}
override fun getRefreshKey(state: PagingState<Int, Book>): Int? {
return state.anchorPosition?.let { anchorPosition ->
val anchorPage = state.closestPageToPosition(anchorPosition)
anchorPage?.prevKey?.plus(1) ?: anchorPage?.nextKey?.minus(1)
}
}
}
and this is the UI:
Column() {
// other stuff
LazyColumn(
modifier = Modifier.padding(horizontal = 24.dp),
content = {
for (i in 0 until searchResults.itemCount) {
searchResults[i]?.let { book ->
item {
BookCard(
book = book,
navigateToBookDetail = { navigateToBookDetail(book.id) }
)
}
}
}
}
)
}
As far as I can tell, the data loads correctly and in the correct order, but when I log the API request URLs, it's making infinite calls with an increasing startIndex each time. That would be fine if I was scrolling, since Google Books searches often return thousands of results, but it does this even if I don't scroll the screen.

The issue here was the way I was creating elements in the LazyColumn - it natively supports LazyPagingItem but I wasn't using that. Here is the working version:
LazyColumn(
modifier = Modifier.padding(horizontal = 24.dp),
state = listState,
content = {
items(pagedSearchResults) { book ->
book?.let {
BookCard(
book = book,
navigateToBookDetail = { navigateToBookDetail(book.id) }
)
}
}
}
)

In your original example, you have to use peek to check for non-null and access the list as you do only inside item block, which is lazy. Otherwise the paging capabilities will be lost and it will load the entire dataset in one go.

In Paging3, after deleting the item, how to return the recyclerview to its original position?

I imitated the codelab and implemented the getRefreshKey() method, but since the params.loadSize is 3*PAGE_SIZE after refresh(that is, to delete an item or edit an item), most of the probability of my recyclerview will not return to the original position. What should I do?
This is my pagingSource:
class PasswordPagingSource2(
val type: String,
val service: PasswordService,
val PAGE_SIZE:Int) :
PagingSource<Int, Any>() {
private val PAGE_INDEX = 0
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Any> {
val page = params.key ?: PAGE_INDEX
return try {
val list =
service.getPasswords(
"{\"type\":\"$type\"}",
params.loadSize, page * PAGE_SIZE).items
val preKey = if (page > 0) page - 1 else null
val nextKey = if (list.isNotEmpty()) page + (params.loadSize/PAGE_SIZE)else null
return LoadResult.Page(data =list, prevKey = preKey, nextKey = nextKey)
} catch (exception: IOException) {
return LoadResult.Error(Throwable("IO Error"))
} catch (exception: HttpException) {
return LoadResult.Error(Throwable("Network Error"))
}
}
override fun getRefreshKey(state: PagingState<Int, Any>): Int? {
val page = state.anchorPosition?.let { anchorPosition->
state.closestPageToPosition(anchorPosition)?.prevKey?.plus(1)
?:state.closestPageToPosition(anchorPosition)?.nextKey?.minus(1)
}
return page
}
}

It is your responsibility to ensure that initialLoadSize will load enough items to cover the viewport so that on invalidation, you will load enough items such that the previous list, and the initial refresh after invalidation overlap so that DiffUtil can help you resume scrolling position.
If you want to keep your pageSize the same, you can modify initialLoadSize in PagingConfig instead. As a rule of thumb, you want it to load at least 2x the number of visible items at once, both before and after the last PagingState.anchorPosition.

Android - How to load several lists from API?

I have an API (https://www.thecocktaildb.com/api.php) I want to load all lists one by one. There's a request where I can find all categories and the only difference beetween lists in the filter in URL.request function What should i use to achieve such functionality? The sketch of how it should be. Maybe Paging library can be useful? Please help!

I believe that you can use Paging 3 for that.
I'll assume that you will start this by having all the categories in a list.
class DrinkSource(
private val categories: List<String>
) : PagingSource<String, Drink>() {
override suspend fun load(
params: LoadParams<String>
): LoadResult<String, Drink> {
val result = requestFromAPI(params.key ?: categories[0])
val index = categories.indexOf(params.key)
val previous = if (index == 0) null else categories[index - 1]
val next = if (index == categories.size - 1) null else categories[index + 1]
return LoadResult.Page(result, prevKey = previous, nextKey = next)
}
private suspend fun requestFromAPI(category: String): List<Drink> {
// replace this with an API call
return listOf(Drink(1, ""))
}
}
From what i saw of your API there were no pagination in the category query, so this solution would work.