Develop Reference

How to interpect when a coroutine in an observable has finished?

I have an observable in my foreground service which fetch data from a paging API and save it to the database, the foreground service shows a notification with a progress bar with number of saved items vs the total amount.
Observable which fetch all the data looks like this:
private fun getAllProducts(): Observable<Response<List<ProdottoBarcode>>> {
val lastId = intArrayOf(0)
return Observable.range(1, Integer.MAX_VALUE - 1)
.concatMap { currentPage -> getProducts(currentPage, lastId[0]) }
.takeUntil { response -> response.body()?.isEmpty() == true }
.doOnNext { response ->
lastId[0] = response.headers().get("lastId")?.toInt()!!
}
}
Then the subscription is done in onCreate() like this:
override fun onCreate() {
super.onCreate()
...
getAllProducts().subscribeWith(object: DisposableObserver<Response<List<ProdottoBarcode>>>() {
override fun onNext(response: Response<List<ProdottoBarcode>>) {
if (response.isSuccessful) {
val products = response.body()
val totalItems = response.headers().get("items")?.toInt()
insertProducts(totalItems, products)
}
}
override fun onError(e: Throwable) {
stopService()
}
override fun onComplete() {
}
})
}
And the method which saves all the data to the database looks like this:
private fun insertProducts(totalItems: Int?, products: List<ProdottoBarcode>?) {
if (products != null) {
CoroutineScope(Dispatchers.IO).launch {
for (product in products) {
repository.insert(product)
savedItems += 1
val notification =
totalItems?.let { items ->
NotificationCompat.Builder(baseContext, "progress_channel")
.setSmallIcon(R.drawable.ic_box)
.setContentTitle("Sincronizzati: $savedItems prodotti su $totalItems")
.setProgress(items, savedItems, false)
.setOngoing(true)
.build()
}
notificationManager.notify(notificationId, notification)
}
// TODO: stop the service and dismiss the notification when all items has been saved
if (savedItems == totalItems) {
stopService()
}
}
}
}
The stopService() in insertProducts not always works, while if I try to put stopService in onComplete() it will be executed once all subscriptions are done and NOT when all the items has been saved.
So my question is:
How can I stop my services by using the Coroutine inside the Observable? I need to know when all items from all observables are insert in database and only then to dismiss the service.

Side note: you don't need to do Int wrapping like this in Kotlin like you would in Java. Kotlin has implicit variable wrapping, so you can simply use a var local variable and it will be captured by whatever function you use it in.
val lastId = intArrayOf(0) // can just be var lastId = 0
Starting with getProducts() for fetching a page. I think the code you linked is OK provided your Retrofit service's getProducts function is marked suspend, so it's not blocking. No changes here.
private suspend fun getProducts(
page: Int,
lastId: Int,
itemsPerPage: Int = 50
): Response<List<ProdottoBarcode>> {
val prefs = PreferenceManager.getDefaultSharedPreferences(this)
val urlServer = prefs.getString("server", "http://127.0.0.1/")!!
return RetrofitClient.getInstance().getService()
.getProducts(urlServer, "A", page, lastId, itemsPerPage)
}
Your getAllProducts in your linked code doesn't need backing StateFlows that are never collected--you're using them simply as mutable Int wrappers, which are unnecessary in Kotlin as mentioned way above. I'm not exactly sure how you're consuming these pages, since I'm not very familiar with Rx, but I take the use of concatMap to mean that the Observable is queuing up pages as fast as it can into a buffer, and you are reading out these pages to some local property that the UI uses. I think a buffer should be added so we can be inserting in the database in parallel with fetching the next page. Default buffer arguments are probably appropriate.
private val allProducts: Flow<Response<List<ProdottoBarcode>>> = flow {
var lastId = 0
for (currentPage in 1 until Int.MAX_VALUE) {
emit(getProducts(currentPage, lastId))
lastId = response.headers().get("lastId")!!.toInt()
}
}
.takeWhile { response -> !response.body().isNullOrEmpty() }
.buffer()
Usually, when you collect your flow, you should use an appropriate coroutine scope provided by the Android framework, so it will automatically cancel collection once it goes out of scope. If you inherit your service from LifecycleService, you can use the existing lifecycleScope. This is maybe not so critical in a service in this case, since I think you are only calling stopService() when your flow is complete, but it would make it a little more robust against potential mistakes, I think.
.launchIn is a shortcut that is like wrapping everything above it in launch and calling collect() on it. I prefer the syntax because it has less nesting of code.
override fun onCreate() {
super.onCreate()
// ...
allProducts.onEach { response ->
if (response.isSuccessful) {
val products = response.body()
val totalItems = response.headers().get("items")?.toInt()
insertProducts(totalItems, products)
}
}
.catch { Log.e(TAG, "Failed collecting page.", it) }
.onCompletion { stopService() }
.flowOn(Dispatchers.Default) // don't use main thread since this is a service
.launchIn(lifecycleScope)
}
Since we're using buffer() in the fetching flow, we don't need to launch other coroutines when inserting in the database to achieve parallelism. We can simplify this into a suspend function. We are handling stopping the service in the flow collector, so we don't need to do that here either. I'm assuming repository.insert is a suspend function, not blocking.
private suspend fun insertProducts(totalItems: Int?, products: List<ProdottoBarcode>?) {
if (totalItems == null) {
Log.e(TAG, "Tried to insert items without any item count. Skipping.")
return
}
if (products == null) {
Log.e(TAG, "Tried to insert null products list. Skipping.")
return
}
for (product in products) {
repository.insert(product)
savedItems += 1
val notification = NotificationCompat.Builder(baseContext, "progress_channel")
.setSmallIcon(R.drawable.ic_box)
.setContentTitle("Sincronizzati: $savedItems prodotti su $totalItems")
.setProgress(items, savedItems, false)
.setOngoing(true)
.build()
}
notificationManager.notify(notificationId, notification)
}
}

When will awaitWorkAvailable resume from runRecomposeAndApplyChanges in Recomposer?

I saw that changes were tracked and requested invalidation through Recompoer#runRecomposeAndApplyChanges. This function tracks changes and requests invalidation in the while statement and remains suspended until new changes are made through waitWorkAvailable() in the while statement.
awaitWorkAvailable() works with suspendCancellableCoroutine and assigns Continuation to workContinuation if there is no scheduled task. The workContinuation is resumed in the invokeOnCompletion of a Job called effectJob. Therefore, in order for workContinuation to resume, effectJob must be completed. Recomposer#close, which completes it, is used only in the withRunningRecomposer function, but withRunningRecomposer is not used.
So where does workContinuation resume? I had the same concern a month ago, but I still haven’t solved it.
(Job.cancel is also possible to call invokeOnComplete, but cancel is used only for lifecycle handling through Owner)
full-code for
runRecomposeAndApplyChanges: cs.android.com
awaitWorkAvailable: cs.android.com
effectJob: cs.android.com
close: cs.android.com
withRunningRecomposer: cs.android.com

I finally solved it.
A workContinuation instance was being returned from deriveStateLocked(), and workContinuation is resumed whenever invalidation is required thereafter.
private fun deriveStateLocked(): CancellableContinuation<Unit>? {
// ...
return if (newState == State.PendingWork) {
workContinuation.also {
workContinuation = null
}
} else null
}
#OptIn(ExperimentalComposeApi::class)
private suspend fun recompositionRunner(
block: suspend CoroutineScope.(parentFrameClock: MonotonicFrameClock) -> Unit
) {
val parentFrameClock = coroutineContext.monotonicFrameClock
withContext(broadcastFrameClock) {
val callingJob = coroutineContext.job
registerRunnerJob(callingJob)
val unregisterApplyObserver = Snapshot.registerApplyObserver { changed, _ ->
synchronized(stateLock) {
if (_state.value >= State.Idle) {
snapshotInvalidations += changed
deriveStateLocked()
} else null
}?.resume(Unit)
}
// ...
}
internal override fun invalidate(composition: ControlledComposition) {
synchronized(stateLock) {
if (composition !in compositionInvalidations) {
compositionInvalidations += composition
deriveStateLocked()
} else null
}?.resume(Unit)
}
internal override fun invalidateScope(scope: RecomposeScopeImpl) {
synchronized(stateLock) {
snapshotInvalidations += setOf(scope)
deriveStateLocked()
}?.resume(Unit)
}
// ... more invalidation methods are using deriveStateLocked().

How to complete a Kotlin Flow in Android Worker

I'm investigating the use of Kotlin Flow within my current Android application
My application retrieves its data from a remote server via Retrofit API calls.
Some of these API's return 50,000 data items in 500 item pages.
Each API response contains an HTTP Link header containing the Next pages complete URL.
These calls can take up to 2 seconds to complete.
In an attempt to reduce the elapsed time I have employed a Kotlin Flow to concurrently process each page
of data while also making the next page API call.
My flow is defined as follows:
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val internalWorkWorkState = MutableStateFlow<Response<List<MyPage>>?>(null)
private val workWorkState = internalWorkWorkState.asStateFlow()
private val myJob: Job
init {
myJob = GlobalScope.launch(persistenceThreadPool) {
workWorkState.collect { page ->
if (page == null) {
} else managePage(page!!)
}
}
}
My Recursive function is defined as follows that fetches all pages:-
private suspend fun managePages(accessToken: String, response: Response<List<MyPage>>) {
when {
result != null -> return
response.isSuccessful -> internalWorkWorkState.emit(response)
else -> {
manageError(response.errorBody())
result = Result.failure()
return
}
}
response.headers().filter { it.first == HTTP_HEADER_LINK && it.second.contains(REL_NEXT) }.forEach {
val parts = it.second.split(OPEN_ANGLE, CLOSE_ANGLE)
if (parts.size >= 2) {
managePages(accessToken, service.myApiCall(accessToken, parts[1]))
}
}
}
private suspend fun managePage(response: Response<List<MyPage>>) {
val pages = response.body()
pages?.let {
persistResponse(it)
}
}
private suspend fun persistResponse(myPage: List<MyPage>) {
val myPageDOs = ArrayList<MyPageDO>()
myPage.forEach { page ->
myPageDOs.add(page.mapDO())
}
database.myPageDAO().insertAsync(myPageDOs)
}
My numerous issues are
This code does not insert all data items that I retrieve
How do complete the flow when all data items have been retrieved
How do I complete the GlobalScope job once all the data items have been retrieved and persisted
UPDATE
By making the following changes I have managed to insert all the data
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val completed = CompletableDeferred<Int>()
private val channel = Channel<Response<List<MyPage>>?>(UNLIMITED)
private val channelFlow = channel.consumeAsFlow().flowOn(persistenceThreadPool)
private val frank: Job
init {
frank = GlobalScope.launch(persistenceThreadPool) {
channelFlow.collect { page ->
if (page == null) {
completed.complete(totalItems)
} else managePage(page!!)
}
}
}
...
...
...
channel.send(null)
completed.await()
return result ?: Result.success(outputData)
I do not like having to rely on a CompletableDeferred, is there a better approach than this to know when the Flow has completed everything?

You are looking for the flow builder and Flow.buffer():
suspend fun getData(): Flow<Data> = flow {
var pageData: List<Data>
var pageUrl: String? = "bla"
while (pageUrl != null) {
TODO("fetch pageData from pageUrl and change pageUrl to the next page")
emitAll(pageData)
}
}
.flowOn(Dispatchers.IO /* no need for a thread pool executor, IO does it automatically */)
.buffer(3)
You can use it just like a normal Flow, iterate, etc. If you want to know the total length of the output, you should calculate it on the consumer with a mutable closure variable. Note you shouldn't need to use GlobalScope anywhere (ideally ever).

There are a few ways to achieve the desired behaviour. I would suggest to use coroutineScope which is designed specifically for parallel decomposition. It also provides good cancellation and error handling behaviour out of the box. In conjunction with Channel.close behaviour it makes the implementation pretty simple. Conceptually the implementation may look like this:
suspend fun fetchAllPages() {
coroutineScope {
val channel = Channel<MyPage>(Channel.UNLIMITED)
launch(Dispatchers.IO){ loadData(channel) }
launch(Dispatchers.IO){ processData(channel) }
}
}
suspend fun loadData(sendChannel: SendChannel<MyPage>){
while(hasMoreData()){
sendChannel.send(loadPage())
}
sendChannel.close()
}
suspend fun processData(channel: ReceiveChannel<MyPage>){
for(page in channel){
// process page
}
}
It works in the following way:
coroutineScope suspends until all children are finished. So you don't need CompletableDeferred anymore.
loadData() loads pages in cycle and posts them into the channel. It closes the channel as soon as all pages have been loaded.
processData fetches items from the channel one by one and process them. The cycle will finish as soon as all the items have been processed (and the channel has been closed).
In this implementation the producer coroutine works independently, with no back-pressure, so it can take a lot of memory if the processing is slow. Limit the buffer capacity to have the producer coroutine suspend when the buffer is full.
It might be also a good idea to use channels fan-out behaviour to launch multiple processors to speed up the computation.

How to use Kotlin Coroutine.LiveData + Retrofit for API call while showing a progress bar?

I am trying to do an API call using coroutine and retrofit with the MVVM architecture. I would like to show a progress bar while waiting for the API response to be ready (with a timeout of 3 seconds).
In the View Model I am using Coroutine.LiveData
class BootstrapViewModel: ViewModel() {
private val repository : ConfigRepository =
ConfigRepository()
val configurations = liveData(Dispatchers.IO) {
val retrievedConfigs = repository.getConfigurations(4)
emit(retrievedConfigs)
}
}
What I have so far in the activity is just a simulation of API call to update progress bar:
launch {
// simulate API call
val configFetch = async(Dispatchers.IO) {
while (progressState.value != 100) {
progressState.postValue(progressState.value?.plus(1))
delay(50)
}
}
// suspend until fetch is finished or return null in 3 sec
val result = withTimeoutOrNull(3000) { configFetch.await() }
if (result != null) {
// todo: process config... next steps
} else {
// cancel configFetch
configFetch.cancel()
// show error
}
}
I can also observe the livedata as below and works fine:
bootstrapViewModel.configurations.observe(this, Observer {
//response is ready
})
Everything works fine separated. However, when I try to use the livedata inside coroutine scope things get messy. Is there anyway to await() for a coroutine livedata (like how I did for configFetch)?

You can do it like this:
val _progressBarVisibility = MutableLiveData<Int>() // Use this with postValue
val progressBarVisibility: LiveData<Int> = _progressBarVisibility
val configurations = liveData(Dispatchers.IO) {
_progressBarVisibility.postValue(View.VISIBLE)
// you can just stimulate API call with a delay() method
delay(3000) //3 seconds
val retrievedConfigs = repository.getConfigurations(4)
_progressBarVisibility.postValue(View.GONE)
emit(retrievedConfigs)
}
After that in your Activity:
viewModel.progressBarVisibility.observe(this, Observer{
pbVisibilityView.visibity = it
}
If you are asking about retrofit in particular, this is how you can do it.
In your DataApi interface, you just mark the method as suspended:
interface DataApi{
#GET("endpointHere")
suspend fun getData() : Result<Data>
}
The rest is just as I described above. Just replace getConfigurations(4) with getData()

Kotlin corountines : Note: end time exceeds epoch:

Combine runBlocking and withContext seems to dispatch the message
Note: end time exceeds epoch:
class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
runBlocking {
withContext(DefaultDispatcher) {
null
}
}
}
}
I use many coroutines like this and the logcat is spammed, any idea to avoid this ? Another way to do this, for example :
var projects: List<ProjectEntity>? = runBlocking {
withContext(DefaultDispatcher) {
//Get the ProjectEntity list
}
}
projects?.let {
onResult(projects)
}
EDIT
I try something based on your comments (thank you), but I can't get a similar result as my example above :
Log.d("Coroutines", "getMostRecent start")
var localeProject: ProjectEntity? = null
launch {
withContext(CommonPool) {
Log.d("Coroutines", "getRecentLocaleProject")
localeProject = getRecentLocaleProject()
}
}
Log.d("Coroutines", "check localeProject")
if (localeProject != null) {
//Show UI
}
In Logcat :
D/Coroutines: getMostRecent start
D/Coroutines: check localeProject
D/Coroutines: getRecentLocaleProject
I want to separate async and sync stuff, there is no way like this ? I really want to avoid all the callbacks things in my repositories when possible.

Markos comment is right, you should not block the UI thread.
You should use launch or async and use withContext to switch back to the UI thread.
You find some examples here: https://github.com/Kotlin/kotlinx.coroutines/blob/master/ui/coroutines-guide-ui.md#structured-concurrency-lifecycle-and-coroutine-parent-child-hierarchy
class MainActivity : ScopedAppActivity() {
fun asyncShowData() = launch { // Is invoked in UI context with Activity's job as a parent
// actual implementation
}
suspend fun showIOData() {
val deferred = async(Dispatchers.IO) {
// impl
}
withContext(Dispatchers.Main) {
val data = deferred.await()
// Show data in UI
}
}
}
Be aware, that the example uses the new coroutine API (>0.26.0), that renamed the Dispatchers. So Dispatchers.Main corresponds to UI in older versions.

var localeProject: ProjectEntity? = null
launch {
withContext(CommonPool) {
localeProject = getRecentLocaleProject()
}
}
if (localeProject != null) {
//Show UI
}
I want to separate async and sync stuff, there is no way like this ?
When you launch a coroutine, semantically it's like you started a thread. Intuition tells you that you can't expect localeProject != null just after you've started the thread that sets it, and this is true for the coroutine as well. It's even stronger: you are guaranteed not to ever see localeProject != null because launch only adds a new event to the event loop. Until your current method completes, that event won't be handled.
So you can forget about top-level vals initialized from async code. Not even lateinit vars can work because you have no guarantee you'll see it already initialized. You must work with the loosest kind: nullable vars.