Develop Reference

How to complete a Kotlin Flow in Android Worker

I'm investigating the use of Kotlin Flow within my current Android application
My application retrieves its data from a remote server via Retrofit API calls.
Some of these API's return 50,000 data items in 500 item pages.
Each API response contains an HTTP Link header containing the Next pages complete URL.
These calls can take up to 2 seconds to complete.
In an attempt to reduce the elapsed time I have employed a Kotlin Flow to concurrently process each page
of data while also making the next page API call.
My flow is defined as follows:
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val internalWorkWorkState = MutableStateFlow<Response<List<MyPage>>?>(null)
private val workWorkState = internalWorkWorkState.asStateFlow()
private val myJob: Job
init {
myJob = GlobalScope.launch(persistenceThreadPool) {
workWorkState.collect { page ->
if (page == null) {
} else managePage(page!!)
}
}
}
My Recursive function is defined as follows that fetches all pages:-
private suspend fun managePages(accessToken: String, response: Response<List<MyPage>>) {
when {
result != null -> return
response.isSuccessful -> internalWorkWorkState.emit(response)
else -> {
manageError(response.errorBody())
result = Result.failure()
return
}
}
response.headers().filter { it.first == HTTP_HEADER_LINK && it.second.contains(REL_NEXT) }.forEach {
val parts = it.second.split(OPEN_ANGLE, CLOSE_ANGLE)
if (parts.size >= 2) {
managePages(accessToken, service.myApiCall(accessToken, parts[1]))
}
}
}
private suspend fun managePage(response: Response<List<MyPage>>) {
val pages = response.body()
pages?.let {
persistResponse(it)
}
}
private suspend fun persistResponse(myPage: List<MyPage>) {
val myPageDOs = ArrayList<MyPageDO>()
myPage.forEach { page ->
myPageDOs.add(page.mapDO())
}
database.myPageDAO().insertAsync(myPageDOs)
}
My numerous issues are
This code does not insert all data items that I retrieve
How do complete the flow when all data items have been retrieved
How do I complete the GlobalScope job once all the data items have been retrieved and persisted
UPDATE
By making the following changes I have managed to insert all the data
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val completed = CompletableDeferred<Int>()
private val channel = Channel<Response<List<MyPage>>?>(UNLIMITED)
private val channelFlow = channel.consumeAsFlow().flowOn(persistenceThreadPool)
private val frank: Job
init {
frank = GlobalScope.launch(persistenceThreadPool) {
channelFlow.collect { page ->
if (page == null) {
completed.complete(totalItems)
} else managePage(page!!)
}
}
}
...
...
...
channel.send(null)
completed.await()
return result ?: Result.success(outputData)
I do not like having to rely on a CompletableDeferred, is there a better approach than this to know when the Flow has completed everything?

You are looking for the flow builder and Flow.buffer():
suspend fun getData(): Flow<Data> = flow {
var pageData: List<Data>
var pageUrl: String? = "bla"
while (pageUrl != null) {
TODO("fetch pageData from pageUrl and change pageUrl to the next page")
emitAll(pageData)
}
}
.flowOn(Dispatchers.IO /* no need for a thread pool executor, IO does it automatically */)
.buffer(3)
You can use it just like a normal Flow, iterate, etc. If you want to know the total length of the output, you should calculate it on the consumer with a mutable closure variable. Note you shouldn't need to use GlobalScope anywhere (ideally ever).

There are a few ways to achieve the desired behaviour. I would suggest to use coroutineScope which is designed specifically for parallel decomposition. It also provides good cancellation and error handling behaviour out of the box. In conjunction with Channel.close behaviour it makes the implementation pretty simple. Conceptually the implementation may look like this:
suspend fun fetchAllPages() {
coroutineScope {
val channel = Channel<MyPage>(Channel.UNLIMITED)
launch(Dispatchers.IO){ loadData(channel) }
launch(Dispatchers.IO){ processData(channel) }
}
}
suspend fun loadData(sendChannel: SendChannel<MyPage>){
while(hasMoreData()){
sendChannel.send(loadPage())
}
sendChannel.close()
}
suspend fun processData(channel: ReceiveChannel<MyPage>){
for(page in channel){
// process page
}
}
It works in the following way:
coroutineScope suspends until all children are finished. So you don't need CompletableDeferred anymore.
loadData() loads pages in cycle and posts them into the channel. It closes the channel as soon as all pages have been loaded.
processData fetches items from the channel one by one and process them. The cycle will finish as soon as all the items have been processed (and the channel has been closed).
In this implementation the producer coroutine works independently, with no back-pressure, so it can take a lot of memory if the processing is slow. Limit the buffer capacity to have the producer coroutine suspend when the buffer is full.
It might be also a good idea to use channels fan-out behaviour to launch multiple processors to speed up the computation.

android -MutableLiveData doesn't observe on new data

I'm using mvvm and android architecture component , i'm new in this architecture .
in my application , I get some data from web service and show them in recycleView , it works fine .
then I've a button for adding new data , when the user input the data , it goes into web service , then I have to get the data and update my adapter again.
this is my code in activity:
private fun getUserCats() {
vm.getCats().observe(this, Observer {
if(it!=null) {
rc_cats.visibility= View.VISIBLE
pb.visibility=View.GONE
catAdapter.reloadData(it)
}
})
}
this is view model :
class CategoryViewModel(private val model:CategoryModel): ViewModel() {
private lateinit var catsLiveData:MutableLiveData<MutableList<Cat>>
fun getCats():MutableLiveData<MutableList<Cat>>{
if(!::catsLiveData.isInitialized){
catsLiveData=model.getCats()
}
return catsLiveData;
}
fun addCat(catName:String){
model.addCat(catName)
}
}
and this is my model class:
class CategoryModel(
private val netManager: NetManager,
private val sharedPrefManager: SharedPrefManager) {
private lateinit var categoryDao: CategoryDao
private lateinit var dbConnection: DbConnection
private lateinit var lastUpdate: LastUpdate
fun getCats(): MutableLiveData<MutableList<Cat>> {
dbConnection = DbConnection.getInstance(MyApp.INSTANCE)!!
categoryDao = dbConnection.CategoryDao()
lastUpdate = LastUpdate(MyApp.INSTANCE)
if (netManager.isConnected!!) {
return getCatsOnline();
} else {
return getCatsOffline();
}
}
fun addCat(catName: String) {
val Category = ApiConnection.client.create(Category::class.java)
Category.newCategory(catName, sharedPrefManager.getUid())
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ success ->
getCatsOnline()
}, { error ->
Log.v("this", "ErrorNewCat " + error.localizedMessage)
}
)
}
private fun getCatsOnline(): MutableLiveData<MutableList<Cat>> {
Log.v("this", "online ");
var list: MutableLiveData<MutableList<Cat>> = MutableLiveData()
list = getCatsOffline()
val getCats = ApiConnection.client.create(Category::class.java)
getCats.getCats(sharedPrefManager.getUid(), lastUpdate.getLastCatDate())
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
{ success ->
list += success.cats
lastUpdate.setLastCatDate()
Observable.just(DbConnection)
.subscribeOn(Schedulers.io())
.subscribe({ db ->
categoryDao.insert(success.cats)
})
}, { error ->
Log.v("this", "ErrorGetCats " + error.localizedMessage);
}
)
return list;
}
I call getCat from activity and it goes into model and send it to my web service , after it was successful I call getCatsOnline method to get the data again from webservice .
as I debugged , it gets the data but it doesn't notify my activity , I mean the observer is not triggered in my activity .
how can I fix this ? what is wrong with my code?

You have made several different mistakes of varying importance in LiveData and RxJava usage, as well as MVVM design itself.
LiveData and RxJava
Note that LiveData and RxJava are streams. They are not one time use, so you need to observe the same LiveData object, and more importantly that same LiveData object needs to get updated.
If you look at getCatsOnline() method, every time the method gets called it's creating a whole new LiveData instance. That instance is different from the previous LiveData object, so whatever that is listening to the previous LiveData object won't get notified to the new change.
And few additional tips:
In getCatsOnline() you are subscribing to an Observable inside of another subscriber. That is common mistake from beginners who treat RxJava as a call back. It is not a call back, and you need to chain these calls.
Do not subscribe in Model layer, because it breaks the stream and you cannot tell when to unsubscribe.
It does not make sense to ever use AndroidSchedulers.mainThread(). There is no need to switch to main thread in Model layer especially since LiveData observers only run on main thread.
Do not expose MutableLiveData to other layer. Just return as LiveData.
One last thing I want to point out is that you are using RxJava and LiveData together. Since you are new to both, I recommend you to stick with just one of them. If you must need to use both, use LiveDataReactiveStreams to bridge these two correctly.
Design
How to fix all this? I am guessing that what you are trying to do is to:
(1) view needs category -> (2) get categories from the server -> (3) create/update an observable list object with the new cats, and independently keep the result in DB -> (4) list instance should notify activity automatically.
It is difficult to pull this off correctly because you have this list instance that you have to manually create and update. You also need to worry about where and how long to keep this list instance.
A better design would be:
(1) view needs category -> (2) get a LiveData from DB and observe -> (3) get new categories from the server and update DB with the server response -> (4) view is notified automatically because it's been observing DB!
This is much easier to implement because it has this one way dependency: View -> DB -> Server
Example CategoryModel:
class CategoryModel(
private val netManager: NetManager,
private val sharedPrefManager: SharedPrefManager) {
private val categoryDao: CategoryDao
private val dbConnection: DbConnection
private var lastUpdate: LastUpdate // Maybe store this value in more persistent place..
fun getInstance(netManager: NetManager, sharedPrefManager: SharedPrefManager) {
// ... singleton
}
fun getCats(): Observable<List<Cat>> {
return getCatsOffline();
}
// Notice this method returns just Completable. Any new data should be observed through `getCats()` method.
fun refreshCats(): Completable {
val getCats = ApiConnection.client.create(Category::class.java)
// getCats method may return a Single
return getCats.getCats(sharedPrefManager.getUid(), lastUpdate.getLastCatDate())
.flatMap { success -> categoryDao.insert(success.cats) } // insert to db
.doOnSuccess { lastUpdate.setLastCatDate() }
.ignoreElement()
.subscribeOn(Schedulers.io())
}
fun addCat(catName: String): Completable {
val Category = ApiConnection.client.create(Category::class.java)
// newCategory may return a Single
return Category.newCategory(catName, sharedPrefManager.getUid())
.ignoreElement()
.andThen(refreshCats())
.subscribeOn(Schedulers.io())
)
}
}
I recommend you to read through Guide to App Architecture and one of these livedata-mvvm example app from Google.

RxJava2 doOnNext returns later than intended

So I'm pretty sure that I'm kind of at a loss here.
The expected behavior is:
Get data from API -> save it in the local DB -> load data from the
local DB and display it
First of all in my Fragment I call a function that does this:
mLiveData = viewModel.fetchAllCategories(getString(R.string.lang_code))
mCategoryLiveData!!.observe(this, Observer<Array<Category>> { it ->
if (it != null) {
this#CategoryListFragment.allCategories = SparseArray(it.size)
for (category in it) {
allCategories[category.id] = category
}
displayedCategory = allCategories[1]
this#CategoryListFragment.mLoadingCircle.visibility = View.GONE
this#CategoryListFragment.displayCategoryChildren()
}
})
fetchAllCategories calls a function in the ViewModel which calls this function:
fun getAllCategoriesFromAPI(language: String): Flowable<Array<Category>> {
return service.getAllCategories(language)
.doOnNext {
Log.e("Repository", "Fetched ${it.size} Categories from the API ")
storeCategoriesInDb(it)
}
}
However the function displayCategoryChildren() fires before the onNext finishes which results in an error since the data the app is supposed to get from the db is not saved there yet.
If it is in any way relevant I can also post the fuction in the ViewModel

Loading data from Database + Network (Room + Retrofit + RxJava2)

I have a sample API request which returns a list of user's watchlist. I want to achieve the following flow when the user loads the watchlist screen:
Load the data from DB cache immediately.(cacheWatchList)
Initiate the RetroFit network call in the background.
i. onSuccess return apiWatchList
ii. onError return cacheWatchList
Diff cacheWatchList vs apiWatchList
i. Same -> all is well since data is already displayed to the user do nothing.
ii. Differs -> Save apiWatchList to a local store and send the apiWatchList to the downstream.
What I have done so far?
Watchlist.kt
data class Watchlist(
val items: List<Repository> = emptyList()
)
LocalStore.kt (Android room)
fun saveUserWatchlist(repositories: List<Repository>): Completable {
return Completable.fromCallable {
watchlistDao.saveAllUserWatchlist(*repositories.toTypedArray())
}
}
RemoteStore.kt (Retrofit api call)
fun getWatchlist(userId: UUID): Single<Watchlist?> {
return api.getWatchlist(userId)
}
DataManager.kt
fun getWatchlist(userId: UUID): Flowable<List<Repository>?> {
val localSource: Single<List<Repository>?> =
localStore.getUserWatchlist()
.subscribeOn(scheduler.computation)
val remoteSource: Single<List<Repository>> = remoteStore.getWatchlist(userId)
.map(Watchlist::items)
.doOnSuccess { items: List<Repository> ->
localStore.saveUserWatchlist(items)
.subscribeOn(scheduler.io)
.subscribe()
}
.onErrorResumeNext { throwable ->
if (throwable is IOException) {
return#onErrorResumeNext localStore.getUserWatchlist()
}
return#onErrorResumeNext Single.error(throwable)
}
.subscribeOn(scheduler.io)
return Single.concat(localSource, remoteSource)
}
The problem with the above flow is, it calls onNext twice for each stream source to the downstream(presenter) even though both the data are same.
I can do the data diff logic in the presenter and update accordingly but I want the DataManager class to handle the logic for me(CleanArchitecture, SOC).
My Questions?
What's the best possible way to implement the above logic?
Am I leaking the inner subscriptions in DataManager (see: doOnSuccess code) ?. I'm disposing of the outer subscription when the presenter is destroyed.

fun getWatchlist(userId: UUID): Observable<List<Repository>?> {
val remoteSource: Single<List<Repository>> =
remoteStore.getWatchlist(userId)
.map(Watchlist::items)
.subscribeOn(scheduler.io)
return localStore.getUserWatchlist()
.flatMapObservable { listFromLocal: List<Repository> ->
remoteSource
.observeOn(scheduler.computation)
.toObservable()
.filter { apiWatchList: List<Repository> ->
apiWatchList != listFromLocal
}
.flatMapSingle { apiWatchList ->
localSource.saveUserWatchlist(apiWatchList)
.andThen(Single.just(apiWatchList))
}
.startWith(listFromLocal)
}
}
Explanation step by step:
Load data from localStore
Use flatMapObservable to subscribe to remoteSource each time the localStore emits data.
As there are more than one emission from inner observable(initial data from local and new data in case of updated data from the remoteSource) transform Single to Observable.
Compare data from remoteSource with data from the localStore and proceed data only in case if newData != localData.
For each emission after the filter initiate the localSource to save data and on a completion of this operation proceed saved data as Single.
As requested, at the beginning of remote request data from localStore should be proceeded and it is simply done be adding startWith at the end of the operators chain.

Implementing search that pushes results to list as soon as they become available using rxJava

I need to implement a search on a large data set that can take some time to complete on mobile devices. So I want to display each matching result as soon as it becomes available.
I need to fetch all available data from a data store that decides whether to get them from network or from the device. This call is an Observable. As soon as the data from that Observable becomes available I want to loop over it, apply a search predicate and notify any Observers for any match found.
So far my idea was to use a PublishSubject to subscribe to and call its onNext function every time the search finds a new match. However I can't seem to get the desired behavior to work.
I'm using MVVM + Android Databinding and want to display every matched entry in a RecyclerView so for every onNext event that is received by the observing viewModel I have to call notifyItemRangeInserted on the RecyclerView's adapter.
class MySearch(val dataStore: MyDataStore) {
private val searchSubject = PublishSubject.create<List<MyDto>>()
fun findEntries(query: String): Observable<List<MyDto>> {
return searchSubject.doOnSubscribe {
// dataStore.fetchAll returns an Observable<List<MyDto>>
dataStore.fetchAll.doOnNext {
myDtos -> if (query.isNotBlank()) {
search(query, myDtos)
} else {
searchSubject.onNext(myDtos)
}
}.subscribe(searchSubject)
}
}
private fun(query: String, data: List<MyDto>) {
data.forEach {
if (it.matches(query)) {
// in real life I cache a few results and don't send each single item
searchSubject.onNext(listOf(it))
}
}
}
fun MyDto.matches(query: String): Boolean // stub
}
-
class MyViewModel(val mySearch: MySearch, val viewNotifications: Observer<Pair<Int, Int>>): BaseObservable() {
var displayItems: List<MyItemViewModel> = listOf()
fun loadData(query: String): Subscription {
return mySearch.findEntries(query)
.observeOn(AndroidSchedulers.mainThread())
.doOnNext(this::onSearchResult)
.doOnCompleted(viewNotifications::onCompleted)
.doOnError(viewNotifications::onError)
.subscribe()
}
private fun onSearchResult(List<MyDto> data) {
val lastIndex = displayItems.lastIndex
displayItems = data.map { createItem(it) }
notifyChange()
viewNotifications.onNext(Pair(lastIndex, data.count()))
}
private fun createItem(dto: MyDto): MyItemViewModel // stub
}
The problem I have with the above code is that with an empty query MyViewModel::onSearchResult is called 3 times in a row and when the query is not empty MyViewModel::onSearchResult isn't called at all.
I suspect the problem lies somewhere in the way I have nested the Observables in findEntries or that I'm subscribing wrong / getting data from a wrong thread.
Does anyone have an idea about this?