I'm trying to paginate my REST API with the paging 3 library, specifically with the RemoteMediator class.
What I want to achieve is that my local database is being used for the initial presentation of my UI, which gets updated reactively with a rest call if there are any changes in the shown data.
The strange behaviour that I have right now is, that the pagination only works like expected for the first time: Pages are loaded incrementally (meaning a REST call is fired) when the user scrolls the RecyclerView.
After having data in my local database, there is only a single rest call in the beginning (with the initial load size) and at the end where like 4 requests are fired successively. So it does not request my API incrementally like the first time, it just fires all requests at the end.
Is this behaviour normal ? Or is this a bug ?
My RemoteMediator code:
#ExperimentalPagingApi
class PageableRemoteMediator<T : Any>(
private val pageKeyId: String,
private val apiRequestFun: suspend (PageKey?, Int) -> ApiResult<Page<T>>,
private val entityPersisterFun: suspend (Page<T>) -> Unit,
application: Application
) : RemoteMediator<Int, T>() {
private val db = DatabaseProvider.getDatabase(application)
private val pageKeyDao = db.pageKeyDao()
override suspend fun initialize(): InitializeAction {
return InitializeAction.LAUNCH_INITIAL_REFRESH
}
override suspend fun load(loadType: LoadType, state: PagingState<Int, T>): MediatorResult {
val pageKey = getPageKeyOrNull(loadType)
/**
* If the next pageKey is null and loadType is APPEND, then the pagination end is reached.
* Also a prepend is seen as the end, since the RecyclerView always start from the beginning after a refresh.
*/
if ((loadType == LoadType.APPEND && pageKey?.next == null) || loadType == LoadType.PREPEND) {
return MediatorResult.Success(endOfPaginationReached = true)
}
val limit = if (loadType == LoadType.REFRESH) {
state.config.initialLoadSize
} else {
state.config.pageSize
}
return when (val result = apiRequestFun(pageKey, limit)) {
is ApiResult.Success -> {
db.withTransaction {
val page = result.value
removeAllPageKeysIfRefreshRequested(loadType)
// This function just does an #Insert query for the specific type in a Room DAO class.
// I'm not calling something like a BaseDAO function, since sometimes I need to use insert queries with multiple models in the args.
// E.g. #Insert insertUniversityWithStudents(university : University, students : List<Student>) in UniversityDAO
entityPersisterFun(page)
saveNewPageKey(page)
val endOfPaginationReached = page.cursor.next == null
MediatorResult.Success(endOfPaginationReached)
}
}
is ApiResult.Failed -> {
MediatorResult.Error(result.getException())
}
}
}
private suspend fun getPageKeyOrNull(loadType: LoadType): PageKey? {
return if (loadType != LoadType.REFRESH) {
pageKeyDao.findByKey(pageKeyId)
} else {
null
}
}
private suspend fun removeAllPageKeysIfRefreshRequested(loadType: LoadType) {
if (loadType == LoadType.REFRESH) {
pageKeyDao.removeAllByKey(pageKeyId)
}
}
private suspend fun saveNewPageKey(entityPage: Page<*>) {
val key = PageKey(pageKeyId, entityPage.cursor.next, entityPage.cursor.prev)
pageKeyDao.asyncInsert(key)
}
}
EDIT:
I found this code snippet in the Android codelab
// clear all tables in the database
if (loadType == LoadType.REFRESH) {
repoDatabase.remoteKeysDao().clearRemoteKeys()
repoDatabase.reposDao().clearRepos()
}
Wouldn't this delete all my local data after an app restart ? Im not getting why I should use the RemoteMediator at all, if it just removes all entities anyway and wait for the response of my api-request ?
Check the #4 (Paging Library Components) in the Android Codelab you shared.
The remote mediator's behavior depends on the configuration of your implementation of the PagingSource and your PagingConfig.
The PagingSource manages the local data access, e.g. Sqlite database, while the RemoteMediator is for requesting data from the network when the particular conditions are satisfied within the PagingSource.
Related
I was trying to implement an approach to fetch products from two Data sources (Room & FirebaseFirestore) using Flows.
It was working fine until I noticed that the debugger was returning to the same break point infinitely. When the execution of "ViewmMdel.insertProducts(products)" ends, the debugger returns to Repository.getProducts(//) & repeats.
I changed the approach using only suspending functions & coroutines & works fine but I am curious about how I must to use Flows to implement this approach.
Maybe is only that flatMapMerge is in preview version.
Thanks in advance :D
This one is the implementation:
ViewModel:
fun getProductNames(companyName: String) {
viewModelScope.launch {
repository.getProducts(companyName).catch {
_event.value = AddSaleEvents.ShouldShowLoading(false)
_event.value = AddSaleEvents.ProductsFailureResponse(it.message.toString())
}.collect { products ->
productsList = products
if (products != emptyList<Product>()) {
_event.value = AddSaleEvents.ShouldShowLoading(false)
_event.value = AddSaleEvents.ProductsSuccessfulResponse(products)
insertProducts(products)
} else {
_event.value = AddSaleEvents.ShouldShowLoading(false)
_event.value = AddSaleEvents.ProductsSuccessfulResponse(products)
}
}
}
}
Repository:
#OptIn(FlowPreview::class)
override suspend fun getProducts(compnayName: String): Flow<List<Product>> {
return localDataSource.getProducts().flatMapMerge { list -> // LINE RUNNING INFINITELY
getProductsFromFirebase(list, compnayName)
}.flowOn(Dispatchers.IO).catch {
Log.d("Error", it.message.toString())
}
}
private fun getProductsFromFirebase(products: List<Product>, compnayName: String) = flow {
if (products.isEmpty()) {
remoteDataSource.getProducts(compnayName).collect {
emit(it)
}
} else {
emit(products)
}
}
LocalDataSource with Room:
override suspend fun getProducts(): Flow<List<Product>> = saleDao.getProducts()
Firebase Data Source:
override suspend fun getProducts(company: String): Flow<List<Product>> = flow {
val response = fireStore.collection("products").whereEqualTo("company", company).get()
response.await()
if (response.isSuccessful && !response.result.isEmpty) {
emit(response.result.toObjects(FirebaseProduct::class.java).toEntity())
}
}.catch {
Log.d("Error", it.message.toString())
}
How can I chain the response of a flow to trigger another one inside the MVVM Architecture + Clean Architecture?
6 if it is possible, I want to understand the reason the code is repeating infinitely.
Looks like insertProducts(products) triggers room's DAO.
So localDataSource.getProducts() is a observable read query
Observable queries are read operations that emit new values whenever there are changes to any of the tables that are referenced by the query.
Try to change LocalDataSource
interface SaleDao {
// fun getProducts(): Flow<List<Product>>
suspend fun getProducts(): List<Product>
}
I have a common situation of getting data. I use the Kotlin Coroutines.
1 variant:
class SomeViewModel(
private val gettingData: GetDataUseCase
) : ViewModel() {
lateinit var data: List<String>
init {
viewModelScope.launch {
data = gettingData.get()
}
}
}
2 variant:
class SomeViewModel(
private val gettingData: GetDataUseCase
) : ViewModel() {
val data = MutableStateFlow<List<String>?>(null)
init {
viewModelScope.launch {
data.emit(gettingData.get())
}
}
}
How can I initialize a data field not delayed, but immediately, with the viewModelScope but without a lateinit or nullble field? And without LiveData, my progect uses Coroutine Flow
I can't return a result of viewModelScope job in .run{} or by lazy {}.
I cant return a result drom fun:
val data: List<String> = getData()
fun getData(): List<String> {
viewModelScope.launch {
data = gettingData.get()
}
return ???
}
Also I can't make suspend fun getData() because I can't create coroutineScope in initialisation'
You're describing an impossibility. Presumably, gettingData.get() is defined as a suspend function, meaning the result literally cannot be retrieved immediately. Since it takes a while to retrieve, you cannot have an immediate value.
This is why apps and websites have loading indicators in their UI.
If you're using Flows, you can use a Flow with a nullable type (like in your option 2 above), and in your Activity/Fragment, in the collector, you show either a loading indicator or your data depending on whether it is null.
Your code 2 can be simplified using the flow builder and stateIn with a null default value:
class SomeViewModel(
private val gettingData: GetDataUseCase
) : ViewModel() {
val data = flow<List<String>?> { emit(gettingData.get()) }
.stateIn(viewModelScope, SharingStarted.Eagerly, null)
}
In your Activity or Fragment:
viewLifecycleOwner.lifecycleScope.launch {
viewModel.data
.flowWithLifecycle(viewLifecycleOwner.lifecycle, Lifecycle.State.STARTED)
.collect { list ->
if(list == null) {
// Show loading indicator in UI
} else {
// Show the data
}
}
}
If your data loads pretty quickly, instead of making the type nullable, you can just make the default value emptyList(). Then your collector can just not do anything when the list is empty. This works if the data loads quickly enough that the user isn't going to wonder if something is wrong because the screen is blank for so long.
You have to use SharedFlow with replay 1 (to store last value and replay it for a new subscriber) to implement it.
My sample:
interface DataSource {
suspend fun getData(): Int
}
class DataViewModel(dataSource: DataSource): ViewModel() {
val dataField =
flow<Int> {
emit(dataSource.getData())
}.shareIn(viewModelScope, SharingStarted.WhileSubscribed(1000), 1)
}
I'm investigating the use of Kotlin Flow within my current Android application
My application retrieves its data from a remote server via Retrofit API calls.
Some of these API's return 50,000 data items in 500 item pages.
Each API response contains an HTTP Link header containing the Next pages complete URL.
These calls can take up to 2 seconds to complete.
In an attempt to reduce the elapsed time I have employed a Kotlin Flow to concurrently process each page
of data while also making the next page API call.
My flow is defined as follows:
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val internalWorkWorkState = MutableStateFlow<Response<List<MyPage>>?>(null)
private val workWorkState = internalWorkWorkState.asStateFlow()
private val myJob: Job
init {
myJob = GlobalScope.launch(persistenceThreadPool) {
workWorkState.collect { page ->
if (page == null) {
} else managePage(page!!)
}
}
}
My Recursive function is defined as follows that fetches all pages:-
private suspend fun managePages(accessToken: String, response: Response<List<MyPage>>) {
when {
result != null -> return
response.isSuccessful -> internalWorkWorkState.emit(response)
else -> {
manageError(response.errorBody())
result = Result.failure()
return
}
}
response.headers().filter { it.first == HTTP_HEADER_LINK && it.second.contains(REL_NEXT) }.forEach {
val parts = it.second.split(OPEN_ANGLE, CLOSE_ANGLE)
if (parts.size >= 2) {
managePages(accessToken, service.myApiCall(accessToken, parts[1]))
}
}
}
private suspend fun managePage(response: Response<List<MyPage>>) {
val pages = response.body()
pages?.let {
persistResponse(it)
}
}
private suspend fun persistResponse(myPage: List<MyPage>) {
val myPageDOs = ArrayList<MyPageDO>()
myPage.forEach { page ->
myPageDOs.add(page.mapDO())
}
database.myPageDAO().insertAsync(myPageDOs)
}
My numerous issues are
This code does not insert all data items that I retrieve
How do complete the flow when all data items have been retrieved
How do I complete the GlobalScope job once all the data items have been retrieved and persisted
UPDATE
By making the following changes I have managed to insert all the data
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val completed = CompletableDeferred<Int>()
private val channel = Channel<Response<List<MyPage>>?>(UNLIMITED)
private val channelFlow = channel.consumeAsFlow().flowOn(persistenceThreadPool)
private val frank: Job
init {
frank = GlobalScope.launch(persistenceThreadPool) {
channelFlow.collect { page ->
if (page == null) {
completed.complete(totalItems)
} else managePage(page!!)
}
}
}
...
...
...
channel.send(null)
completed.await()
return result ?: Result.success(outputData)
I do not like having to rely on a CompletableDeferred, is there a better approach than this to know when the Flow has completed everything?
You are looking for the flow builder and Flow.buffer():
suspend fun getData(): Flow<Data> = flow {
var pageData: List<Data>
var pageUrl: String? = "bla"
while (pageUrl != null) {
TODO("fetch pageData from pageUrl and change pageUrl to the next page")
emitAll(pageData)
}
}
.flowOn(Dispatchers.IO /* no need for a thread pool executor, IO does it automatically */)
.buffer(3)
You can use it just like a normal Flow, iterate, etc. If you want to know the total length of the output, you should calculate it on the consumer with a mutable closure variable. Note you shouldn't need to use GlobalScope anywhere (ideally ever).
There are a few ways to achieve the desired behaviour. I would suggest to use coroutineScope which is designed specifically for parallel decomposition. It also provides good cancellation and error handling behaviour out of the box. In conjunction with Channel.close behaviour it makes the implementation pretty simple. Conceptually the implementation may look like this:
suspend fun fetchAllPages() {
coroutineScope {
val channel = Channel<MyPage>(Channel.UNLIMITED)
launch(Dispatchers.IO){ loadData(channel) }
launch(Dispatchers.IO){ processData(channel) }
}
}
suspend fun loadData(sendChannel: SendChannel<MyPage>){
while(hasMoreData()){
sendChannel.send(loadPage())
}
sendChannel.close()
}
suspend fun processData(channel: ReceiveChannel<MyPage>){
for(page in channel){
// process page
}
}
It works in the following way:
coroutineScope suspends until all children are finished. So you don't need CompletableDeferred anymore.
loadData() loads pages in cycle and posts them into the channel. It closes the channel as soon as all pages have been loaded.
processData fetches items from the channel one by one and process them. The cycle will finish as soon as all the items have been processed (and the channel has been closed).
In this implementation the producer coroutine works independently, with no back-pressure, so it can take a lot of memory if the processing is slow. Limit the buffer capacity to have the producer coroutine suspend when the buffer is full.
It might be also a good idea to use channels fan-out behaviour to launch multiple processors to speed up the computation.
I was working with Android Paging 3 as you can see in https://developer.android.com/topic/libraries/architecture/paging/v3-network-db.
When I run the app RemoteMediator requests server to get List until there is nothing on the server.
The point is here that we want to get the list when user reached the bottom of the list, but RemoteMediator is requesting until get all of list on the server!
RemoteMediator.kt
#OptIn(ExperimentalPagingApi::class)
class ExampleRemoteMediator(
private val query: String,
private val database: RoomDb,
private val networkService: ExampleBackendService
) : RemoteMediator<Int, User>() {
val userDao = database.userDao()
val remoteKeyDao = database.remoteKeyDao()
override suspend fun load(
loadType: LoadType,
state: PagingState<Int, User>
): MediatorResult {
return try {
// The network load method takes an optional String
// parameter. For every page after the first, pass the String
// token returned from the previous page to let it continue
// from where it left off. For REFRESH, pass null to load the
// first page.
val loadKey = when (loadType) {
LoadType.REFRESH -> null
// In this example, you never need to prepend, since REFRESH
// will always load the first page in the list. Immediately
// return, reporting end of pagination.
LoadType.PREPEND -> return MediatorResult.Success(
endOfPaginationReached = true
)
// Query remoteKeyDao for the next RemoteKey.
LoadType.APPEND -> {
val remoteKey = database.withTransaction {
remoteKeyDao.remoteKeyByQuery(query)
}
// You must explicitly check if the page key is null when
// appending, since null is only valid for initial load.
// If you receive null for APPEND, that means you have
// reached the end of pagination and there are no more
// items to load.
if (remoteKey.nextKey == null) {
return MediatorResult.Success(
endOfPaginationReached = true
)
}
remoteKey.nextKey
}
}
// Suspending network load via Retrofit. This doesn't need to
// be wrapped in a withContext(Dispatcher.IO) { ... } block
// since Retrofit's Coroutine CallAdapter dispatches on a
// worker thread.
val response = networkService.searchUsers(query, loadKey)
// Store loaded data, and next key in transaction, so that
// they're always consistent.
database.withTransaction {
if (loadType == LoadType.REFRESH) {
remoteKeyDao.deleteByQuery(query)
userDao.deleteByQuery(query)
}
// Update RemoteKey for this query.
remoteKeyDao.insertOrReplace(
RemoteKey(query, response.nextKey)
)
// Insert new users into database, which invalidates the
// current PagingData, allowing Paging to present the updates
// in the DB.
userDao.insertAll(response.users)
}
MediatorResult.Success(
endOfPaginationReached = response.nextKey == null
)
} catch (e: IOException) {
MediatorResult.Error(e)
} catch (e: HttpException) {
MediatorResult.Error(e)
}
}
}
LoadType.PREPEND and LoadType.APPEND are constantly calling!
After testing the whole code finally I founded why this is happening.
The problem was in "layout.xml"! You should not put RecyclerView in NestedScrollView!!
But why are the RecyclerView and NestedScrollView bugs not completely resolved?!
I can't figure out how to do a "simple" operation with Room and MVVM pattern.
I’m fetching some data with Retrofit. A “proper” response triggers an observer in the activity and a small part of the response itself is inserted in the database using Room library, wiping all previous values stored and inserting the fresh ones. Otherwise old values are retained on DB.
Just after that, I would like to check for a field in the database, but I’m not able to force this operation to wait until the previous one is completed.
Models
#Entity(tableName = "licence")
data class Licence(
#PrimaryKey(autoGenerate = true)
#ColumnInfo(name = "licence_id")
var licenceId: Int = 0,
#Ignore
var config: List<LicenceConfig>? = null,
.......
//all the others attributes )
#Entity(foreignKeys = [
ForeignKey(
entity = Licence::class,
parentColumns = ["licence_id"],
childColumns = ["licence_reference"],
onDelete = ForeignKey.CASCADE
)],tableName = "licence_configurations")
data class LicenceConfig(
#PrimaryKey(autoGenerate = true)
#ColumnInfo(name = "licence_config_id")
var licenceConfigId: Int,
#ColumnInfo(name="licence_reference")
var licenceReference: Int,
Observer in the activity
loginViewModel.apiResponse.observe(this, Observer { response ->
response?.let {
loginViewModel.insertLicences(response.licence)
}
//here I need to wait for the insertion to end
loginViewModel.methodToCheckForTheFieldOnDatabase()
})
ViewModel
fun insertLicences(licences: List<Licence>) = viewModelScope.launch {
roomRepository.deleteAllLicences()
licences.forEach { licence ->
roomRepository.insertLicence(licence).also { insertedLicenceId ->
licence.config?.forEach { licenceConfiguration ->
roomRepository.insertLicenceConfiguration(
licenceConfiguration.apply { licenceReference = insertedLicenceId.toInt() }
)
}
}
}
}
Room Repository
class RoomRepository(private val roomDao: RoomDao) {
val allLicences: LiveData<List<Licence>> = roomDao.getAllLicences()
suspend fun insertLicence(licence: Licence): Long {
return roomDao.insertLicence(licence)
}
suspend fun insertLicenceConfiguration(licenceConfiguration: LicenceConfig){
return roomDao.insertLicenceConfiguration(LicenceConfig)
}
}
RoomDao
#Dao
interface RoomDao {
#Query("select * from licence")
fun getAllLicences(): LiveData<List<Licence>>
#Insert(onConflict = OnConflictStrategy.REPLACE)
suspend fun insertLicence(licence: Licence): Long
#Insert
suspend fun insertLicenceConfiguration(licence: LicenceConfig)
#Query("DELETE FROM licence")
suspend fun deleteAllLicences()
}
Set an observer to the "allLicences" LiveData or directly on that field on DB is not an option because the operations will be performed just after the activity creation and I have to wait until the API response to perform them.
In another project, without Room, I have used async{} and .await() to perform sequential operations while working with coroutines but I can't really make it works here. When I pause the debugger just after the insertion method the value of "allLicences" it's always null but after resuming and exporting the DB the data are properly inserted. I also tried adding .invokeOnCompletion{} after the ViewModel method but with the same result.
Basically I would like to wait for this method to end to do another operation.
Any suggestions?
EDIT
I totally forgot to report the models! Each licence have a list of configurations. When I perform a licence insert I take the autogenerated id, I apply it to the licenceConfig and then I perform the insert for each licenceConfig object (the code in the nested forEach loop of the ViewModel method). The problem seems to be that performing this nested loop breaks the "synchronicity" of the operation
To wait until insertion is completed, you need to move the coroutine creation from insertLicences() to your observer and also make the insertLicences() a suspend function.
loginViewModel.apiResponse.observe(this, Observer { response ->
lifecycleScope.launch {
response?.let {
loginViewModel.insertLicences(response.licence)
}
//here I need to wait for the insertion to end
loginViewModel.methodToCheckForTheFieldOnDatabase()
}
})
and
suspend fun insertLicences(licences: List<Licence>) {
roomRepository.deleteAllLicences()
licences.forEach { licence ->
roomRepository.insertLicence(licence).also { insertedLicenceId ->
licence.config?.forEach { licenceConfiguration ->
roomRepository.insertLicenceConfiguration(
licenceConfiguration.apply { licenceReference = insertedLicenceId.toInt() }
)
}
}
}
}
Alternative Solution
You can shift all of the code present in the observer into ViewModel.
loginViewModel.apiResponse.observe(this, Observer { response ->
loginViewModel.refreshLicenses(response)
})
and in ViewModel
fun refreshLicenses(response:Response?){
viewModelScope.launch{
response?.let {
insertLicences(response.licence)
}
methodToCheckForTheFieldOnDatabase()
}
}
and also make insertLicences as suspend function
suspend fun insertLicences(licences: List<Licence>) {
roomRepository.deleteAllLicences()
licences.forEach { licence ->
roomRepository.insertLicence(licence).also { insertedLicenceId ->
licence.config?.forEach { licenceConfiguration ->
roomRepository.insertLicenceConfiguration(
licenceConfiguration.apply { licenceReference = insertedLicenceId.toInt() }
)
}
}
}
}
Edit: Didn't read your conclusion before I reply but, I still think that your answer lies in coroutines
Using callbacks or promises, won't your function be executed when the insert query is finished?
Callbacks
With callbacks, the idea is to pass one function as a parameter to
another function, and have this one invoked once the process has
completed.
fun postItem(item: Item) {
preparePostAsync { token ->
submitPostAsync(token, item) { post ->
processPost(post)
}
}
}
fun preparePostAsync(callback: (Token) -> Unit) {
// make request and return immediately
// arrange callback to be invoked later
}
I would prefer promises to be honest
Promises
The idea behind futures or promises (there are also other terms these
can be referred to depending on language/platform), is that when we
make a call, we're promised that at some point it will return with an
object called a Promise, which can then be operated on.
fun postItem(item: Item) {
preparePostAsync()
.thenCompose { token ->
submitPostAsync(token, item)
}
.thenAccept { post ->
processPost(post)
}
}
fun preparePostAsync(): Promise<Token> {
// makes request an returns a promise that is completed later
return promise
}
Do your work and when the promise is fullfilled, proceed to data validation.
You can read more about coroutines here