Develop Reference

The same value, 16, was passed as the prevKey in two sequential Pages loaded from a PagingSource

I have this pager with Paging 3 that keeps either repeating pages (if I allow keyReuseSupported) or returning the IllegalStateException because of "The same value, 16, was passed as the prevKey in two sequential Pages loaded from a PagingSource." I am not sure why this is happening, even if I do understand the theory.
This is the paging data source:
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, CameraManagerEvent> {
val key = params.key ?: parseDateFrom(date)
return runCatching {
service.getEvents(
dateJump = this.date,
one = true
)
}
}.fold(
onFailure = { LoadResult.Error(it) },
onSuccess = {
LoadResult.Page(
data = it.events,
prevKey = if(date>18) null else parseDate(date) + 1,
nextKey = if(it.events.isEmpty()) null else parseDate(date) - 1
)
}
)
}
When I send for example date 15, the prevKey is 16, the nextKey is 17, but it seems to return 16 twice... why is the prevKey and the nextKey not working properly? I need to be able to return the next and previous date with my paging but if I do this:
prevKey = if(date>18) null else key + 1,
nextKey = if(it.events.isEmpty()) null else key - 1
It just keeps returning the same two pages (16 and 17 for example). What's going on?

Can't understand code of getRefreshKey in Android paging3 codelab

// GitHub page API is 1 based: https://developer.github.com/v3/#pagination
private const val GITHUB_STARTING_PAGE_INDEX = 1
class GithubPagingSource(
private val service: GithubService,
private val query: String
) : PagingSource<Int, Repo>() {
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Repo> {
val position = params.key ?: GITHUB_STARTING_PAGE_INDEX
val apiQuery = query + IN_QUALIFIER
return try {
val response = service.searchRepos(apiQuery, position, params.loadSize)
val repos = response.items
val nextKey = if (repos.isEmpty()) {
null
} else {
// initial load size = 3 * NETWORK_PAGE_SIZE
// ensure we're not requesting duplicating items, at the 2nd request
position + (params.loadSize / NETWORK_PAGE_SIZE)
}
LoadResult.Page(
data = repos,
prevKey = if (position == GITHUB_STARTING_PAGE_INDEX) null else position - 1,
nextKey = nextKey
)
} catch (exception: IOException) {
return LoadResult.Error(exception)
} catch (exception: HttpException) {
return LoadResult.Error(exception)
}
}
// The refresh key is used for subsequent refresh calls to PagingSource.load after the initial load
override fun getRefreshKey(state: PagingState<Int, Repo>): Int? {
// We need to get the previous key (or next key if previous is null) of the page
// that was closest to the most recently accessed index.
// Anchor position is the most recently accessed index
return state.anchorPosition?.let { anchorPosition ->
state.closestPageToPosition(anchorPosition)?.prevKey?.plus(1)
?: state.closestPageToPosition(anchorPosition)?.nextKey?.minus(1)
}
}
}
This is the code of getRefreshKey function of the paging3 codelab.
I thought that it's just okay to return state.anchorPosition. But why is this returning closestPagetToPosition's previous key plus 1??
This is the link of the paging 3 code lab.

Well, state.anchorPosition is index of the first item you see in this list, getRefreshKey should return the key to load on refresh.
In this example, key is index of page and each page typically contains many items, lets say 20. Now loading two pages would mean using keys 1 and 2 and having 40 items loaded.
In this situation, when you are looking at item on index 30, state.anchorPosition is 30 as well, but you definitely don't want your refresh key to be 30, you need it to be 2 because item on index 30 comes from page 2. This is what closestPageToPosition does.

Android Paging 3 library PagingSource invalidation, causes the list to jump due to wrong refresh key (not using room)

Since I'm currently working on a Project with custom database (not Room), I'm testing whether we could use the Paging 3 library in the Project.
However, I run into the issue, that if you make changes to the data and therefore invalidate the paging source, the recreation of the list is buggy and jumps to a different location. This is happening because the Refresh Key calculation seems to be wrong, which is most likely caused by the fact that the initial load loads three pages worth of data, but puts it into one page.
The default Paging Source looks like this:
override fun getRefreshKey(state: PagingState<Int, CustomData>): Int? {
// Try to find the page key of the closest page to anchorPosition, from
// either the prevKey or the nextKey, but you need to handle nullability
// here:
// * prevKey == null -> anchorPage is the first page.
// * nextKey == null -> anchorPage is the last page.
// * both prevKey and nextKey null -> anchorPage is the initial page, so
// just return null.
return state.anchorPosition?.let { anchorPosition ->
val anchorPage = state.closestPageToPosition(anchorPosition)
anchorPage?.prevKey?.plus(1) ?: anchorPage?.nextKey?.minus(1)
}
}
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, CustomData> {
var pagePosition = params.key ?: STARTING_PAGE_INDEX
var loadSize = params.loadSize
return try {
val dataResult = dataRepository.getPagedData(
pagePosition = pagePosition,
loadSize = loadSize,
pageSize = pageSize
)
val nextKey = if (dataResult.isEmpty() || dataResult.size < pageSize) {
null
} else {
pagePosition + (loadSize / pageSize)
}
Log.i(
"RoomFreePagingSource",
"page $pagePosition with size $loadSize publish ${dataResult.size} routes"
)
return LoadResult.Page(
data = dataResult,
prevKey = if (pagePosition == STARTING_PAGE_INDEX) null else pagePosition - 1,
nextKey = nextKey
)
} catch (exception: Exception) {
LoadResult.Error(exception)
}
}
The dataRepository.getPagedData() function simply accesses a in memory list and returns a subsection of the list, simulating the paged data.
For completeness here is the implementation of this function:
fun getPagedData(pageSize:Int, loadSize:Int, pagePosition: Int): List<CustomData>{
val startIndex = pagePosition * pageSize
val endIndexExl =startIndex + loadSize
return data.safeSubList(startIndex,endIndexExl).map { it.copy() }
}
private fun <T> List<T>.safeSubList(fromIndex: Int, toIndex: Int) : List<T>{
// only returns list with valid range, to not throw exception
if(fromIndex>= this.size)
return emptyList()
val endIndex = if(toIndex> this.size) this.size else toIndex
return subList(fromIndex,endIndex)
}
The main Problem I currently face is, that the getRefreshKey function doesn't return a correct refresh page key, which results in the wrong page being refreshed and the list jumping to the loaded page.
For example if you have 10 items per page.
The first page 0 contains 30 items
The next page would be 3 and contains 10 items.
If you don't scroll (only see the first 7 items) and invalidate then the anchorPosition will be 7 and the refresh key will be 2.
(anchorPage.prevKey = null => anchorPage.nextKey = 3 => 3-1 is 2)
However, at this point we would like to load page 0 not page 2.
Knowing the cause of the issue I tried to adapt the default implementation to resolve it and came up with a lot of different versions. The one below currently works best, but still causes the jumping from time to time. And also sometimes a part of the list flickers, which is probably caused when not enough items are fetched to fill the view port.
override fun getRefreshKey(state: PagingState<Int, CustomData>): Int? {
return state.anchorPosition?.let { anchorPosition ->
val closestPage = state.closestPageToPosition(anchorPosition)?.prevKey
?: STARTING_PAGE_INDEX
val refKey = if(anchorPosition>(closestPage)*pageSize + pageSize)
closestPage+1
else
closestPage
Log.i(
"RoomFreePagingSource",
"getRefreshKey $refKey from anchorPosition $anchorPosition closestPage $closestPage"
)
refKey
}
}
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, CustomData> {
var pagePosition = params.key ?: STARTING_PAGE_INDEX
var loadSize = pageSize
return try {
when (params) {
is LoadParams.Refresh -> {
if (pagePosition > STARTING_PAGE_INDEX) {
loadSize *= 3
} else if (pagePosition == STARTING_PAGE_INDEX) {
loadSize *= 2
}
}
else -> {}
}
val dataResult = dataRepository.getPagedData(
pagePosition = pagePosition,
loadSize = loadSize,
pageSize = pageSize
)
val nextKey = if (dataResult.isEmpty() || dataResult.size < pageSize) {
null
} else {
pagePosition + (loadSize / pageSize)
}
Log.i(
"RouteRepository",
"page $pagePosition with size $loadSize publish ${dataResult.size} routes"
)
return LoadResult.Page(
data = dataResult,
prevKey = if (pagePosition == STARTING_PAGE_INDEX) null else pagePosition - 1,
nextKey = nextKey
)
} catch (exception: Exception) {
LoadResult.Error(exception)
}
}
In theory I found that the best solution would be to just calculate the refresh PageKey like this anchorPosition/pageSize then load this page, the one before and after it. Hower, calculating the refresh key like this does not work since the anchorPosition isn't the actual position of the item in the list. After some invalidations, the anchor could be 5 even if you are currently looking at item 140 in the list.
So to sum it up:
How can I calculate the correct refresh page after invalidation when I don't use Room, but another data source, like in this sample case a in memory list which I access through getPagedData?

Found the solution my self in the meantime, but forgot to add the answer here.
The anchorPosition in getRefreshKey() was wrong, because the the itemsBefore argument was not set for each Page returned by the paging source. This behaviour does not seem to be documented.
LoadResult.Page(
itemsBefore = pagePosition * pageSize,
data = dataResult,
prevKey = if (pagePosition == STARTING_PAGE_INDEX) null else pagePosition - 1,
nextKey = nextKey
)
Now one can actually determine the refresh key based on the anchorPosition and the pageSize(defined as PagingSource constructor parameter in this example, but can also be a constant etc.)
override fun getRefreshKey(state: PagingState<Int, CustomData>): Int? {
return state.anchorPosition?.let { anchorPosition ->
anchorPosition / pageSize
}
}
No the list does not jump any longer when refreshing/ invalidating it. However sometimes the list might flickr or a whole part of the list won't be visible when invalidating. This is due to the fact, that the refresh page is too small and does not cover the whole view port. Hence one can see the paging in action for the items outside of the refresh page. This behaviour can be fixed by making the refresh page (loadSize) 3 times larger and reducing the pagePosition by one. 3 times is necessary, as the page before and after the actual refresh page could be visible at in the view port.
if (params is LoadParams.Refresh) {
loadSize *= 3
pagePosition = max(0, pagePosition - 1)
}
This solution works just fine now, however, it feels like this can't/ shouldn't be the official solution to this problem. As all these adaptations are not necessary when using paging and room. So, I'm open to other/ better solutions.
Here the full code of working PagingSource with all mentioned changes:
override fun getRefreshKey(state: PagingState<Int, CustomData>): Int? {
return state.anchorPosition?.let { anchorPosition ->
anchorPosition / pageSize
}
}
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, CustomData> {
var pagePosition = params.key ?: STARTING_PAGE_INDEX
var loadSize = pageSize
return try {
if (params is LoadParams.Refresh) {
// make sure everything visible in the view port is updated / loaded
loadSize *= 3
pagePosition = max(0, pagePosition - 1)
}
val dataResult = dataRepository.getPagedData(
pagePosition = pagePosition,
loadSize = loadSize,
pageSize = pageSize
)
val nextKey = if (dataResult.isEmpty() || dataResult.size < pageSize) {
null
} else {
pagePosition + (loadSize / pageSize)
}
LoadResult.Page(
itemsBefore = pagePosition * pageSize,
data = dataResult,
prevKey = if (pagePosition == STARTING_PAGE_INDEX) null else pagePosition - 1,
nextKey = nextKey
)
} catch (exception: Exception) {
LoadResult.Error(exception)
}
}

You can define initial load size in PagingConfig object when creating Pager. Like
Pager(
config = PagingConfig(pageSize = PAGE_SIZE, initialLoadSize = PAGE_SIZE),
initialKey = INITIAL_PAGE,
remoteMediator = mediator,
pagingSourceFactory = {
...
})

In Paging3, after deleting the item, how to return the recyclerview to its original position?

I imitated the codelab and implemented the getRefreshKey() method, but since the params.loadSize is 3*PAGE_SIZE after refresh(that is, to delete an item or edit an item), most of the probability of my recyclerview will not return to the original position. What should I do?
This is my pagingSource:
class PasswordPagingSource2(
val type: String,
val service: PasswordService,
val PAGE_SIZE:Int) :
PagingSource<Int, Any>() {
private val PAGE_INDEX = 0
override suspend fun load(params: LoadParams<Int>): LoadResult<Int, Any> {
val page = params.key ?: PAGE_INDEX
return try {
val list =
service.getPasswords(
"{\"type\":\"$type\"}",
params.loadSize, page * PAGE_SIZE).items
val preKey = if (page > 0) page - 1 else null
val nextKey = if (list.isNotEmpty()) page + (params.loadSize/PAGE_SIZE)else null
return LoadResult.Page(data =list, prevKey = preKey, nextKey = nextKey)
} catch (exception: IOException) {
return LoadResult.Error(Throwable("IO Error"))
} catch (exception: HttpException) {
return LoadResult.Error(Throwable("Network Error"))
}
}
override fun getRefreshKey(state: PagingState<Int, Any>): Int? {
val page = state.anchorPosition?.let { anchorPosition->
state.closestPageToPosition(anchorPosition)?.prevKey?.plus(1)
?:state.closestPageToPosition(anchorPosition)?.nextKey?.minus(1)
}
return page
}
}

It is your responsibility to ensure that initialLoadSize will load enough items to cover the viewport so that on invalidation, you will load enough items such that the previous list, and the initial refresh after invalidation overlap so that DiffUtil can help you resume scrolling position.
If you want to keep your pageSize the same, you can modify initialLoadSize in PagingConfig instead. As a rule of thumb, you want it to load at least 2x the number of visible items at once, both before and after the last PagingState.anchorPosition.

Android: Paging3: Duplicates items

Problem:
I get 40 items at the beginning of the list, then it starts to count from 11, and after this, everything is good. So, 1...40,11,12,13,...,300.
And when I scroll a lot to the bottom and then scroll up to see first items, the items have been changed to 1,2,...,10,1,2,...,10,1,2,...,10,11,12,...,300.
But, when I pass false to enablePlaceholders in the PagingConfig, when I scroll to the bottom, I see the issue as I said above(1,2,..,40,11,...,300) and suddenly the 40 items vanish and I only see 1,2,...,10 + 11,12,...,300(the correct way); And it doesn't change or get worse again.
ProductsPagingSource:
#Singleton
class ProductsPagingSource #Inject constructor(
private val productsApi: ProductsApi
//private val query: String
) : RxPagingSource<Int, RecyclerItem>() {
override fun loadSingle(params: LoadParams<Int>): Single<LoadResult<Int, RecyclerItem>> {
val position = params.key ?: STARTING_PAGE_INDEX
//val apiQuery = query
return productsApi.getBeersList(position, params.loadSize)
.subscribeOn(Schedulers.io())
.map { listBeerResponse ->
listBeerResponse.map { beerResponse ->
beerResponse.toDomain()
}
}
.map { toLoadResult(it, position) }
.onErrorReturn { LoadResult.Error(it) }
}
private fun toLoadResult(
#NonNull response: List<RecyclerItem>,
position: Int
): LoadResult<Int, RecyclerItem> {
return LoadResult.Page(
data = response,
prevKey = if (position == STARTING_PAGE_INDEX) null else position - 1,
nextKey = if (response.isEmpty()) null else position + 1,
itemsBefore = LoadResult.Page.COUNT_UNDEFINED,
itemsAfter = LoadResult.Page.COUNT_UNDEFINED
)
}
}
ProductsListRepositoryImpl:
#Singleton
class ProductsListRepositoryImpl #Inject constructor(
private val pagingSource: ProductsPagingSource
) : ProductsListRepository {
override fun getBeers(ids: String): Flowable<PagingData<RecyclerItem>> = Pager(
config = PagingConfig(
pageSize = 10,
enablePlaceholders = true,
maxSize = 30,
prefetchDistance = 5,
initialLoadSize = 40
),
pagingSourceFactory = { pagingSource }
).flowable
}
ProductsListViewModel:
class ProductsListViewModel #ViewModelInject constructor(
private val getBeersUseCase: GetBeersUseCase
) : BaseViewModel() {
private val _ldProductsList: MutableLiveData<PagingData<RecyclerItem>> = MutableLiveData()
val ldProductsList: LiveData<PagingData<RecyclerItem>> = _ldProductsList
init {
loading(true)
getProducts("")
}
private fun getProducts(ids: String) {
loading(false)
getBeersUseCase(GetBeersParams(ids = ids))
.cachedIn(viewModelScope)
.observeOn(AndroidSchedulers.mainThread())
.subscribe {
_ldProductsList.value = it
}.addTo(compositeDisposable)
}
}
ProductsListFragment:
#AndroidEntryPoint
class ProductsListFragment : Fragment(R.layout.fragment_product_list) {
private val productsListViewModel: ProductsListViewModel by viewModels()
private val productsListAdapter: ProductsListAdapter by lazy {
ProductsListAdapter(::navigateToProductDetail)
}
override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
super.onViewCreated(view, savedInstanceState)
setupRecycler()
setupViewModel()
}
private fun setupRecycler() {
itemErrorContainer.gone()
productListRecyclerView.adapter = productsListAdapter
}
private fun setupViewModel() {
productsListViewModel.run {
observe(ldProductsList, ::addProductsList)
observe(ldLoading, ::loadingUI)
observe(ldFailure, ::handleFailure)
}
}
private fun addProductsList(productsList: PagingData<RecyclerItem>) {
loadingUI(false)
productListRecyclerView.visible()
productsListAdapter.submitData(lifecycle, productsList)
}
...
BASE_DIFF_CALLBACK:
val BASE_DIFF_CALLBACK = object : DiffUtil.ItemCallback<RecyclerItem>() {
override fun areItemsTheSame(oldItem: RecyclerItem, newItem: RecyclerItem): Boolean {
return oldItem.id == newItem.id
}
override fun areContentsTheSame(oldItem: RecyclerItem, newItem: RecyclerItem): Boolean {
return oldItem == newItem
}
}
BasePagedListAdapter:
abstract class BasePagedListAdapter(
vararg types: Cell<RecyclerItem>,
private val onItemClick: (RecyclerItem, ImageView) -> Unit
) : PagingDataAdapter<RecyclerItem, RecyclerView.ViewHolder>(BASE_DIFF_CALLBACK) {
private val cellTypes: CellTypes<RecyclerItem> = CellTypes(*types)
override fun getItemViewType(position: Int): Int {
getItem(position).let {
return cellTypes.of(it).type()
}
}
override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): RecyclerView.ViewHolder {
return cellTypes.of(viewType).holder(parent)
}
override fun onBindViewHolder(holder: RecyclerView.ViewHolder, position: Int) {
getItem(position).let {
cellTypes.of(it).bind(holder, it, onItemClick)
}
}
}

I had to set the same number for pageSize and initialLoadSize. Also, I had to set the enablePlaceholders to false.
config = PagingConfig(
pageSize = 10,
enablePlaceholders = false,
maxSize = 30,
prefetchDistance = 5,
initialLoadSize = 10
),
But still, I want to know if it's the normal way? If yes, I couldn't find anywhere point to this! If not, why's that? Why initialLoadSize can not have value more than the pageSize?!
As we can see, the default value for the initialLoadSize is:
internal const val DEFAULT_INITIAL_PAGE_MULTIPLIER = 3
val initialLoadSize: Int = pageSize * DEFAULT_INITIAL_PAGE_MULTIPLIER,

Your issue is most likely because at the repository layer your results page numbers are indexed with numbers derived from the total divided by your page size.
This page number scheme assumes that you will page through the items with pages of the same size. However, Paging wants to get an initial page that's three times bigger by default, and then each page should be the page size.
So, the indexes might be like 0 through totalElements/PageSize, with the number for page that includes a given position equaling itemsIndex/PageSize.
This part is especially relevant:
return LoadResult.Page(
data = response,
prevKey = if (position == STARTING_PAGE_INDEX) null else position - 1,
nextKey = if (response.isEmpty()) null else position + 1,
itemsBefore = LoadResult.Page.COUNT_UNDEFINED,
itemsAfter = LoadResult.Page.COUNT_UNDEFINED
)
Let's imagine a pagesize of 10, an initial load of 30, and 100 elements total. We start with page 0, and we request 30 items. Position is 0, and params.loadSize is 30.
productsApi.getBeersList(0, 30)
Then we return a page object with those 30 elements and a nextPage key of 1.
If we imagine all our objects as list, the first page would be the asterisks in this span: [***-------]
Let's get the next page:
productsApi.getBeersList(1, 10)
That returns elements that are this from our span: [-*--------]
And then you get: [--*-------]
So the 0th page is fine, but the 1th and 2nd page overlap with it. Then, pages 3 and onward contain new elements.
Because you're getting the keys for the next and previous pages by adding or subtracting to get the adjacent keys to the current page of the current length, the index can't scale with the page size. This isn't always easy without knowing what is inside the PagingConfig.
However you can make it work with dynamic page sizes if you can guarantee your initial load size is your regular page size times some integer and that you'll always get the requested number of items except for the last page, you could store an offset as your next/previous page keys, and turn that offset into the next page at load like:
/* Value for api pageNo */
val pageNo = (params.key/params.loadSize + STARTING_PAGE_INDEX) ?: STARTING_PAGE_INDEX
productsApi.getBeersList(pageNo, params.loadSize) // and so on
/* for offset keys in PageData */
nextKey = (pageNo + 1) * loadSize
prevKey = (pageNo - 1) * loadSize // Integer division rounds down so larger windows start where they should