Develop Reference

Combine data from Kotlin Flow/LiveData

I've got a flow from my repository that looks something like this:
val userListFlow: Flow<List<User>> = channelFlow<List<User>> {
source.setOnUserUpdatedListener { userList ->
trySend(userList)
}
awaitClose {
logger.info("waitClose")
source.setOnUserUpdatedListener(null)
}
}.stateIn(
scope = externalScope,
started = SharingStarted.WhileSubscribed(5000),
initialValue = emptyList()
)
suspend fun getUserThumbnail(user: User): File {
return getUserThumbFromCache(user) ?: run {
fetchUserThumbnailRemote()
}
}
private suspend fetchUserThumbnailRemote(user: User): Bitmap {
thumbnailService.getUserThumbnailBitmap(user.id)
}
fun getUserThumbFromCache(user: User) {
val thumbFile = getThumbFile(user)
return if (thumbFile.exists() && thumbFile.size() > 0) {
thumbFile
} else null
}
private fun getThumbFile(user: User): File {
return File(cacheDir, "${user.id}.jpg")
}
}
For each of these users I can call the suspend function to get a thumbnail for the user.
I don't want to wait for the thumbnail before showing the list of users though, I'd rather it show the users and then when the thumbnail is fetched, update the list.
However I'd like the list to be updated when a thumbnail is fetched..
From my ViewModel I have something like
data class UserWithThumb(user: User, thumb: File?)
val userLiveData = repo.userListFlow.map {
UserWithThumb(it, repo.getUserThumbFromCache(it))
}.asLiveData()
So then from my Fragment I do
viewModel.userLiveData.observe(viewLifecycleOwner) {
userListAdapter.submitList(it)
}
My thumbnails are all null though as I need to fetch them from remote. However if I call that function then that will delay my list from getting to the UI until the thumbnail is fetched. How can I get the thumbnail to the UI in a clean way? I realize that I need to have my livedata or flow update itself once the thumbnail is fetched but I have no idea how to hook that into my code. Any ideas would be appreciated.
I suppose one way to think about this is I'd like my upstream (repository) flow to contain the list of users but then I'd like to update the list given to the view not just when the upstream (repo) flow gets new data but when new thumbnails are downloaded as well..

What I understood from the question is, you have a list of UserWithThumb that is created once you set Users list and you want to show it to the UI immediately. In the background you want to fetch User thumbnails and once you receive them, you want to update the list again.
One way to achieve what you want is:
val userLiveData = flow {
repo.userListFlow.collect { users ->
val initialList = users.map { UserWithThumb(it, repo. getUserThumbFromCache(it)) }
emit(initialList)
coroutineScope {
val finalList = users.map {
async(Dispatchers.IO) { // fetch all thumbnails in parallel
UserWithThumb(it, repo. getUserThumbnail(it))
}
}.awaitAll() // wait until all thumbnails have been fetched
emit(finalList)
}
}
}.asLiveData()

How to continue paging after swipe to refresh error?

I am implementing swipe to refresh with the PagingLibrary 3.0.
My repository returns a flow of PagingData<Item> which is then exposed as LiveData by a viewmodel.
Repository
override fun getItems(): Flow<PagingData<Item>> {
val pagingConfig = PagingConfig(pageSize = 20, enablePlaceholders = false)
val pager = Pager(pagingConfig) {
IndexedPagingSource(remoteDataSource)
}
return pager.flow
}
ViewModel
val itemsStream: LiveData<PagingData<Item>> = repository.getItems()
.asLiveData()
.cachedIn(viewModelScope)
Fragment
private fun FragmentItemListBinding.bindView() {
list.adapter = adapter
list.layoutManager = LinearLayoutManager(context)
swipeRefresh.setOnRefreshListener { onRefresh() }
adapter.loadStateFlow
.onEach { resolveLoadState(it) }
.launchIn(viewLifecycleOwner.lifecycleScope)
viewModel.itemsStream.observeWithViewLifecycleOwner {
adapter.submitData(viewLifecycleOwner.lifecycle, it)
}
}
private fun onRefresh() {
adapter.refresh()
}
private fun FragmentItemListBinding.resolveLoadState(loadState: CombinedLoadStates) {
val adapterEmpty = adapter.itemCount < 1
swipeRefresh.isRefreshing = loadState.refresh is LoadState.Loading && !adapterEmpty
// resolve all other states here...
}
The problem is that while the refresh is in progress, the paging stops working (and any ongoing requests are cancelled while the UI stays the same - talking about you LoadStateFooter). And it stops until the refresh succeeds, which includes failure.
If I don't have any data, I simply display an error screen. But in this case I want to see the previous items and continue paging even after error.
Is there a way to continue paging in the case of a refresh error?
The official architecture components sample for paging has the same behavior.

Currently not supported. The generation simply stops working when refresh fails.
https://issuetracker.google.com/u/3/issues/189967519

How to complete a Kotlin Flow in Android Worker

I'm investigating the use of Kotlin Flow within my current Android application
My application retrieves its data from a remote server via Retrofit API calls.
Some of these API's return 50,000 data items in 500 item pages.
Each API response contains an HTTP Link header containing the Next pages complete URL.
These calls can take up to 2 seconds to complete.
In an attempt to reduce the elapsed time I have employed a Kotlin Flow to concurrently process each page
of data while also making the next page API call.
My flow is defined as follows:
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val internalWorkWorkState = MutableStateFlow<Response<List<MyPage>>?>(null)
private val workWorkState = internalWorkWorkState.asStateFlow()
private val myJob: Job
init {
myJob = GlobalScope.launch(persistenceThreadPool) {
workWorkState.collect { page ->
if (page == null) {
} else managePage(page!!)
}
}
}
My Recursive function is defined as follows that fetches all pages:-
private suspend fun managePages(accessToken: String, response: Response<List<MyPage>>) {
when {
result != null -> return
response.isSuccessful -> internalWorkWorkState.emit(response)
else -> {
manageError(response.errorBody())
result = Result.failure()
return
}
}
response.headers().filter { it.first == HTTP_HEADER_LINK && it.second.contains(REL_NEXT) }.forEach {
val parts = it.second.split(OPEN_ANGLE, CLOSE_ANGLE)
if (parts.size >= 2) {
managePages(accessToken, service.myApiCall(accessToken, parts[1]))
}
}
}
private suspend fun managePage(response: Response<List<MyPage>>) {
val pages = response.body()
pages?.let {
persistResponse(it)
}
}
private suspend fun persistResponse(myPage: List<MyPage>) {
val myPageDOs = ArrayList<MyPageDO>()
myPage.forEach { page ->
myPageDOs.add(page.mapDO())
}
database.myPageDAO().insertAsync(myPageDOs)
}
My numerous issues are
This code does not insert all data items that I retrieve
How do complete the flow when all data items have been retrieved
How do I complete the GlobalScope job once all the data items have been retrieved and persisted
UPDATE
By making the following changes I have managed to insert all the data
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val completed = CompletableDeferred<Int>()
private val channel = Channel<Response<List<MyPage>>?>(UNLIMITED)
private val channelFlow = channel.consumeAsFlow().flowOn(persistenceThreadPool)
private val frank: Job
init {
frank = GlobalScope.launch(persistenceThreadPool) {
channelFlow.collect { page ->
if (page == null) {
completed.complete(totalItems)
} else managePage(page!!)
}
}
}
...
...
...
channel.send(null)
completed.await()
return result ?: Result.success(outputData)
I do not like having to rely on a CompletableDeferred, is there a better approach than this to know when the Flow has completed everything?

You are looking for the flow builder and Flow.buffer():
suspend fun getData(): Flow<Data> = flow {
var pageData: List<Data>
var pageUrl: String? = "bla"
while (pageUrl != null) {
TODO("fetch pageData from pageUrl and change pageUrl to the next page")
emitAll(pageData)
}
}
.flowOn(Dispatchers.IO /* no need for a thread pool executor, IO does it automatically */)
.buffer(3)
You can use it just like a normal Flow, iterate, etc. If you want to know the total length of the output, you should calculate it on the consumer with a mutable closure variable. Note you shouldn't need to use GlobalScope anywhere (ideally ever).

There are a few ways to achieve the desired behaviour. I would suggest to use coroutineScope which is designed specifically for parallel decomposition. It also provides good cancellation and error handling behaviour out of the box. In conjunction with Channel.close behaviour it makes the implementation pretty simple. Conceptually the implementation may look like this:
suspend fun fetchAllPages() {
coroutineScope {
val channel = Channel<MyPage>(Channel.UNLIMITED)
launch(Dispatchers.IO){ loadData(channel) }
launch(Dispatchers.IO){ processData(channel) }
}
}
suspend fun loadData(sendChannel: SendChannel<MyPage>){
while(hasMoreData()){
sendChannel.send(loadPage())
}
sendChannel.close()
}
suspend fun processData(channel: ReceiveChannel<MyPage>){
for(page in channel){
// process page
}
}
It works in the following way:
coroutineScope suspends until all children are finished. So you don't need CompletableDeferred anymore.
loadData() loads pages in cycle and posts them into the channel. It closes the channel as soon as all pages have been loaded.
processData fetches items from the channel one by one and process them. The cycle will finish as soon as all the items have been processed (and the channel has been closed).
In this implementation the producer coroutine works independently, with no back-pressure, so it can take a lot of memory if the processing is slow. Limit the buffer capacity to have the producer coroutine suspend when the buffer is full.
It might be also a good idea to use channels fan-out behaviour to launch multiple processors to speed up the computation.

android -MutableLiveData doesn't observe on new data

I'm using mvvm and android architecture component , i'm new in this architecture .
in my application , I get some data from web service and show them in recycleView , it works fine .
then I've a button for adding new data , when the user input the data , it goes into web service , then I have to get the data and update my adapter again.
this is my code in activity:
private fun getUserCats() {
vm.getCats().observe(this, Observer {
if(it!=null) {
rc_cats.visibility= View.VISIBLE
pb.visibility=View.GONE
catAdapter.reloadData(it)
}
})
}
this is view model :
class CategoryViewModel(private val model:CategoryModel): ViewModel() {
private lateinit var catsLiveData:MutableLiveData<MutableList<Cat>>
fun getCats():MutableLiveData<MutableList<Cat>>{
if(!::catsLiveData.isInitialized){
catsLiveData=model.getCats()
}
return catsLiveData;
}
fun addCat(catName:String){
model.addCat(catName)
}
}
and this is my model class:
class CategoryModel(
private val netManager: NetManager,
private val sharedPrefManager: SharedPrefManager) {
private lateinit var categoryDao: CategoryDao
private lateinit var dbConnection: DbConnection
private lateinit var lastUpdate: LastUpdate
fun getCats(): MutableLiveData<MutableList<Cat>> {
dbConnection = DbConnection.getInstance(MyApp.INSTANCE)!!
categoryDao = dbConnection.CategoryDao()
lastUpdate = LastUpdate(MyApp.INSTANCE)
if (netManager.isConnected!!) {
return getCatsOnline();
} else {
return getCatsOffline();
}
}
fun addCat(catName: String) {
val Category = ApiConnection.client.create(Category::class.java)
Category.newCategory(catName, sharedPrefManager.getUid())
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ success ->
getCatsOnline()
}, { error ->
Log.v("this", "ErrorNewCat " + error.localizedMessage)
}
)
}
private fun getCatsOnline(): MutableLiveData<MutableList<Cat>> {
Log.v("this", "online ");
var list: MutableLiveData<MutableList<Cat>> = MutableLiveData()
list = getCatsOffline()
val getCats = ApiConnection.client.create(Category::class.java)
getCats.getCats(sharedPrefManager.getUid(), lastUpdate.getLastCatDate())
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ success ->
list += success.cats
lastUpdate.setLastCatDate()
Observable.just(DbConnection)
.subscribeOn(Schedulers.io())
.subscribe({ db ->
categoryDao.insert(success.cats)
})
}, { error ->
Log.v("this", "ErrorGetCats " + error.localizedMessage);
}
)
return list;
}
I call getCat from activity and it goes into model and send it to my web service , after it was successful I call getCatsOnline method to get the data again from webservice .
as I debugged , it gets the data but it doesn't notify my activity , I mean the observer is not triggered in my activity .
how can I fix this ? what is wrong with my code?

You have made several different mistakes of varying importance in LiveData and RxJava usage, as well as MVVM design itself.
LiveData and RxJava
Note that LiveData and RxJava are streams. They are not one time use, so you need to observe the same LiveData object, and more importantly that same LiveData object needs to get updated.
If you look at getCatsOnline() method, every time the method gets called it's creating a whole new LiveData instance. That instance is different from the previous LiveData object, so whatever that is listening to the previous LiveData object won't get notified to the new change.
And few additional tips:
In getCatsOnline() you are subscribing to an Observable inside of another subscriber. That is common mistake from beginners who treat RxJava as a call back. It is not a call back, and you need to chain these calls.
Do not subscribe in Model layer, because it breaks the stream and you cannot tell when to unsubscribe.
It does not make sense to ever use AndroidSchedulers.mainThread(). There is no need to switch to main thread in Model layer especially since LiveData observers only run on main thread.
Do not expose MutableLiveData to other layer. Just return as LiveData.
One last thing I want to point out is that you are using RxJava and LiveData together. Since you are new to both, I recommend you to stick with just one of them. If you must need to use both, use LiveDataReactiveStreams to bridge these two correctly.
Design
How to fix all this? I am guessing that what you are trying to do is to:
(1) view needs category -> (2) get categories from the server -> (3) create/update an observable list object with the new cats, and independently keep the result in DB -> (4) list instance should notify activity automatically.
It is difficult to pull this off correctly because you have this list instance that you have to manually create and update. You also need to worry about where and how long to keep this list instance.
A better design would be:
(1) view needs category -> (2) get a LiveData from DB and observe -> (3) get new categories from the server and update DB with the server response -> (4) view is notified automatically because it's been observing DB!
This is much easier to implement because it has this one way dependency: View -> DB -> Server
Example CategoryModel:
class CategoryModel(
private val netManager: NetManager,
private val sharedPrefManager: SharedPrefManager) {
private val categoryDao: CategoryDao
private val dbConnection: DbConnection
private var lastUpdate: LastUpdate // Maybe store this value in more persistent place..
fun getInstance(netManager: NetManager, sharedPrefManager: SharedPrefManager) {
// ... singleton
}
fun getCats(): Observable<List<Cat>> {
return getCatsOffline();
}
// Notice this method returns just Completable. Any new data should be observed through `getCats()` method.
fun refreshCats(): Completable {
val getCats = ApiConnection.client.create(Category::class.java)
// getCats method may return a Single
return getCats.getCats(sharedPrefManager.getUid(), lastUpdate.getLastCatDate())
.flatMap { success -> categoryDao.insert(success.cats) } // insert to db
.doOnSuccess { lastUpdate.setLastCatDate() }
.ignoreElement()
.subscribeOn(Schedulers.io())
}
fun addCat(catName: String): Completable {
val Category = ApiConnection.client.create(Category::class.java)
// newCategory may return a Single
return Category.newCategory(catName, sharedPrefManager.getUid())
.ignoreElement()
.andThen(refreshCats())
.subscribeOn(Schedulers.io())
)
}
}
I recommend you to read through Guide to App Architecture and one of these livedata-mvvm example app from Google.

Implementing search that pushes results to list as soon as they become available using rxJava

I need to implement a search on a large data set that can take some time to complete on mobile devices. So I want to display each matching result as soon as it becomes available.
I need to fetch all available data from a data store that decides whether to get them from network or from the device. This call is an Observable. As soon as the data from that Observable becomes available I want to loop over it, apply a search predicate and notify any Observers for any match found.
So far my idea was to use a PublishSubject to subscribe to and call its onNext function every time the search finds a new match. However I can't seem to get the desired behavior to work.
I'm using MVVM + Android Databinding and want to display every matched entry in a RecyclerView so for every onNext event that is received by the observing viewModel I have to call notifyItemRangeInserted on the RecyclerView's adapter.
class MySearch(val dataStore: MyDataStore) {
private val searchSubject = PublishSubject.create<List<MyDto>>()
fun findEntries(query: String): Observable<List<MyDto>> {
return searchSubject.doOnSubscribe {
// dataStore.fetchAll returns an Observable<List<MyDto>>
dataStore.fetchAll.doOnNext {
myDtos -> if (query.isNotBlank()) {
search(query, myDtos)
} else {
searchSubject.onNext(myDtos)
}
}.subscribe(searchSubject)
}
}
private fun(query: String, data: List<MyDto>) {
data.forEach {
if (it.matches(query)) {
// in real life I cache a few results and don't send each single item
searchSubject.onNext(listOf(it))
}
}
}
fun MyDto.matches(query: String): Boolean // stub
}
-
class MyViewModel(val mySearch: MySearch, val viewNotifications: Observer<Pair<Int, Int>>): BaseObservable() {
var displayItems: List<MyItemViewModel> = listOf()
fun loadData(query: String): Subscription {
return mySearch.findEntries(query)
.observeOn(AndroidSchedulers.mainThread())
.doOnNext(this::onSearchResult)
.doOnCompleted(viewNotifications::onCompleted)
.doOnError(viewNotifications::onError)
.subscribe()
}
private fun onSearchResult(List<MyDto> data) {
val lastIndex = displayItems.lastIndex
displayItems = data.map { createItem(it) }
notifyChange()
viewNotifications.onNext(Pair(lastIndex, data.count()))
}
private fun createItem(dto: MyDto): MyItemViewModel // stub
}
The problem I have with the above code is that with an empty query MyViewModel::onSearchResult is called 3 times in a row and when the query is not empty MyViewModel::onSearchResult isn't called at all.
I suspect the problem lies somewhere in the way I have nested the Observables in findEntries or that I'm subscribing wrong / getting data from a wrong thread.
Does anyone have an idea about this?