Develop Reference

How to interpect when a coroutine in an observable has finished?

I have an observable in my foreground service which fetch data from a paging API and save it to the database, the foreground service shows a notification with a progress bar with number of saved items vs the total amount.
Observable which fetch all the data looks like this:
private fun getAllProducts(): Observable<Response<List<ProdottoBarcode>>> {
val lastId = intArrayOf(0)
return Observable.range(1, Integer.MAX_VALUE - 1)
.concatMap { currentPage -> getProducts(currentPage, lastId[0]) }
.takeUntil { response -> response.body()?.isEmpty() == true }
.doOnNext { response ->
lastId[0] = response.headers().get("lastId")?.toInt()!!
}
}
Then the subscription is done in onCreate() like this:
override fun onCreate() {
super.onCreate()
...
getAllProducts().subscribeWith(object: DisposableObserver<Response<List<ProdottoBarcode>>>() {
override fun onNext(response: Response<List<ProdottoBarcode>>) {
if (response.isSuccessful) {
val products = response.body()
val totalItems = response.headers().get("items")?.toInt()
insertProducts(totalItems, products)
}
}
override fun onError(e: Throwable) {
stopService()
}
override fun onComplete() {
}
})
}
And the method which saves all the data to the database looks like this:
private fun insertProducts(totalItems: Int?, products: List<ProdottoBarcode>?) {
if (products != null) {
CoroutineScope(Dispatchers.IO).launch {
for (product in products) {
repository.insert(product)
savedItems += 1
val notification =
totalItems?.let { items ->
NotificationCompat.Builder(baseContext, "progress_channel")
.setSmallIcon(R.drawable.ic_box)
.setContentTitle("Sincronizzati: $savedItems prodotti su $totalItems")
.setProgress(items, savedItems, false)
.setOngoing(true)
.build()
}
notificationManager.notify(notificationId, notification)
}
// TODO: stop the service and dismiss the notification when all items has been saved
if (savedItems == totalItems) {
stopService()
}
}
}
}
The stopService() in insertProducts not always works, while if I try to put stopService in onComplete() it will be executed once all subscriptions are done and NOT when all the items has been saved.
So my question is:
How can I stop my services by using the Coroutine inside the Observable? I need to know when all items from all observables are insert in database and only then to dismiss the service.

Side note: you don't need to do Int wrapping like this in Kotlin like you would in Java. Kotlin has implicit variable wrapping, so you can simply use a var local variable and it will be captured by whatever function you use it in.
val lastId = intArrayOf(0) // can just be var lastId = 0
Starting with getProducts() for fetching a page. I think the code you linked is OK provided your Retrofit service's getProducts function is marked suspend, so it's not blocking. No changes here.
private suspend fun getProducts(
page: Int,
lastId: Int,
itemsPerPage: Int = 50
): Response<List<ProdottoBarcode>> {
val prefs = PreferenceManager.getDefaultSharedPreferences(this)
val urlServer = prefs.getString("server", "http://127.0.0.1/")!!
return RetrofitClient.getInstance().getService()
.getProducts(urlServer, "A", page, lastId, itemsPerPage)
}
Your getAllProducts in your linked code doesn't need backing StateFlows that are never collected--you're using them simply as mutable Int wrappers, which are unnecessary in Kotlin as mentioned way above. I'm not exactly sure how you're consuming these pages, since I'm not very familiar with Rx, but I take the use of concatMap to mean that the Observable is queuing up pages as fast as it can into a buffer, and you are reading out these pages to some local property that the UI uses. I think a buffer should be added so we can be inserting in the database in parallel with fetching the next page. Default buffer arguments are probably appropriate.
private val allProducts: Flow<Response<List<ProdottoBarcode>>> = flow {
var lastId = 0
for (currentPage in 1 until Int.MAX_VALUE) {
emit(getProducts(currentPage, lastId))
lastId = response.headers().get("lastId")!!.toInt()
}
}
.takeWhile { response -> !response.body().isNullOrEmpty() }
.buffer()
Usually, when you collect your flow, you should use an appropriate coroutine scope provided by the Android framework, so it will automatically cancel collection once it goes out of scope. If you inherit your service from LifecycleService, you can use the existing lifecycleScope. This is maybe not so critical in a service in this case, since I think you are only calling stopService() when your flow is complete, but it would make it a little more robust against potential mistakes, I think.
.launchIn is a shortcut that is like wrapping everything above it in launch and calling collect() on it. I prefer the syntax because it has less nesting of code.
override fun onCreate() {
super.onCreate()
// ...
allProducts.onEach { response ->
if (response.isSuccessful) {
val products = response.body()
val totalItems = response.headers().get("items")?.toInt()
insertProducts(totalItems, products)
}
}
.catch { Log.e(TAG, "Failed collecting page.", it) }
.onCompletion { stopService() }
.flowOn(Dispatchers.Default) // don't use main thread since this is a service
.launchIn(lifecycleScope)
}
Since we're using buffer() in the fetching flow, we don't need to launch other coroutines when inserting in the database to achieve parallelism. We can simplify this into a suspend function. We are handling stopping the service in the flow collector, so we don't need to do that here either. I'm assuming repository.insert is a suspend function, not blocking.
private suspend fun insertProducts(totalItems: Int?, products: List<ProdottoBarcode>?) {
if (totalItems == null) {
Log.e(TAG, "Tried to insert items without any item count. Skipping.")
return
}
if (products == null) {
Log.e(TAG, "Tried to insert null products list. Skipping.")
return
}
for (product in products) {
repository.insert(product)
savedItems += 1
val notification = NotificationCompat.Builder(baseContext, "progress_channel")
.setSmallIcon(R.drawable.ic_box)
.setContentTitle("Sincronizzati: $savedItems prodotti su $totalItems")
.setProgress(items, savedItems, false)
.setOngoing(true)
.build()
}
notificationManager.notify(notificationId, notification)
}
}

Is there any way to return value and end a function in callback?

I am struggling with Firestore.
suspend fun writeClientMemo(userId: String, memo: Memo): MemoWriteResponse {
val response = MemoWriteResponse()
val docRef = Firebase.firestore
.collection(COLLECTION_USER)
.document(userId)
val result = Tasks.await(docRef.get())
if (result.exists() && result.data != null) {
Log.d("memo", "data exists ")
val res = docRef.update(FIELD_MEMO_LIST, FieldValue.arrayUnion(memo))
res.addOnSuccessListener {
Log.d("memo", "success")
response.isSuccessful = true
// What I want to do is something like these:
// 1. return#addOnSuccessListener response
// 2. return response
}.addOnFailureListener { e ->
Log.d("memo", "failed")
response.isSuccessful = false
response.errorMessage = "${e.toString}"
}
}
return response // At the moment this always get false(default is false)
}
I tried to make this synchronous but didn't work.
when {
res.isSuccessful -> {
response.isSuccessful = true
Log.d("memo", "true: ${response.toString()}")
}
res.isCanceled -> {
response.run {
isSuccessful = false
errorMessage = "ADD: Error writing document}"
}
Log.d("memo", "false: ${response.toString()}")
}
res.isComplete -> {
Log.d("memo", "false2: ${response.toString()}")
}
else->{
Log.d("memo", "false3: ${response.toString()}")
}
}
I always get false3 here.

Is there any way to return value and end a function in a callback?
No, you cannot simply return a value of an asynchronous operation as a result of a method. Firebase API, as most modern cloud/ APIs are asynchronous. This means that it takes some time until the operation completes. Since you are using the Kotlin programming language, I recommend you consider using Kotlin Coroutines and call .await() like this:
docRef.get().await();
This means that you suspend the operation until the data is available. This means that this value can be returned as a result of a method.
For more information you can check my answers from the following post:
How to return a list from Firestore database as a result of a function in Kotlin?
Or read the following article:
How to read data from Cloud Firestore using get()?
Where I have explained four ways in which you can deal with the Firestore using get().
P.S.
I tried to make this synchronous but didn't work
Never try to do that. It's best to get used to dealing with Firestore in such a way. Hopefully, my explanation from the above answer and article can help you a little bit.

You can achieve this using suspendCoroutine() utility function. It is a common practice to use it to convert asynchronous callback-based API to synchronous suspend API. You use it like this:
suspend fun loadSomeData(): String {
return suspendCoroutine { cont ->
loadSomeDataAsync(
onSuccess = {
cont.resume("ok!")
},
onFailure = {
cont.resume("failed!")
// OR:
cont.resume(null)
// OR:
cont.resumeWithException(Exception("failed"))
}
)
}
}
suspendCoroutine() suspends execution in outer scope until one of resume*() function is called. Then it resumes with the provided result.

FireStore not return all results

I want to get data from firestore, pretend that I have 10 documents in a collection. after that I must get data inside every document, so I save data in ArrayList. But FireBase never return all documents in a collection. Sometimes it returns only 5 ,6 docs in collection that has 10 docs.
my fireBaseUtil :
fun getDocumentByQueryAList( idQuery: List<String>, callBack: (ArrayList<DocumentSnapshot>) -> Unit) {
val listDocumentSnapshot = ArrayList<DocumentSnapshot>()
collRef = fireStore.collection("myCollection")
val size = idQuery.size
for (i in 0 until size) {
val query = collRef.whereEqualTo("fieldQuery", idQuery[i])
query.get().addOnSuccessListener { documents ->
for (document in documents) {
listDocumentSnapshot.add(document)
if (i == size - 1) {
callBack.invoke(listDocumentSnapshot)
}
}
}
}
}
I log out when size = 10 , but i = 8 it called invoke....
UserRepository:
FireBaseUtil.getDocumentByQueryAList{
// myList.addAll(listGettedByCallback)
}
->> when I want to have data in my list I call FireBaseUtil.getDocumentByQueryAList. I know firebase return value async but I dont know how to get all my doc then receiver.invoke("callbackValue").
Please tell me is there any solution. Thank in advance.

The problem you are experiencing is that you are expecting the queries to be run in order like so:
get idQuery[0], then add to list, then
get idQuery[1], then add to list, then
get idQuery[2], then add to list, then
...
get idQuery[8], then add to list, then
get idQuery[9], then add to list, then
invoke callback
But in reality, all of the following things happen in parallel.
get idQuery[0] (add to list when finished)
get idQuery[1] (add to list when finished)
get idQuery[2] (add to list when finished)
...
get idQuery[8] (add to list when finished)
get idQuery[9] (add to list and invoke callback when finished)
If the get idQuery[9] finishes before some of the others, you will be invoking your callback before the list is completely filled.
A primitive way to fix this would be to count the number of finished get queries, and when all of them finish, then invoke the callback.
fun getDocumentByQueryAList( idQuery: List<String>, callBack: (ArrayList<DocumentSnapshot>) -> Unit) {
val listDocumentSnapshot = ArrayList<DocumentSnapshot>()
collRef = fireStore.collection("myCollection")
val size = idQuery.size
val finishedCount = 0
for (i in 0 until size) {
val query = collRef.whereEqualTo("fieldQuery", idQuery[i])
query.get().addOnSuccessListener { documents ->
for (document in documents) {
listDocumentSnapshot.add(document)
}
if (++finishedCount == size) { // ++finishedCount will add 1 to finishedCount, and return the new value
// all tasks done
callBack.invoke(listDocumentSnapshot)
}
}
}
}
However, this will run into issues where the callback is never invoked if any of the queries fail. You could use a addOnFailureListener or addOnCompleteListener to handle these failed tasks.
The more correct and proper way to do what you are expecting is to make use of Tasks.whenAll, which is used in a similar fashion to how you see JavaScript answers using Promise.all. I'm still new to Kotlin syntax myself, so expect the following block to potentially throw errors.
fun getDocumentByQueryAList( idQueryList: List<String>, callBack: (ArrayList<DocumentSnapshot>) -> Unit) {
val listDocumentSnapshot = ArrayList<DocumentSnapshot>()
collRef = fireStore.collection("myCollection")
val getTasks = new ArrayList<Task<Void>>();
for (idQuery in idQueryList) {
val query = collRef.whereEqualTo("fieldQuery", idQuery)
getTasks.add(
query.get()
.onSuccessTask { documents ->
// if query succeeds, add matching documents to list
for (document in documents) {
listDocumentSnapshot.add(document)
}
}
)
}
Tasks.whenAll(getTasks)
.addOnSuccessListener { results ->
callback.invoke(listDocumentSnapshot)
}
.addOnFailureListener { errors ->
// one or more get queries failed
// do something
}
}
Instead of using the callback, you could return a Task instead, where the last bit would be:
return Tasks.whenAll(getTasks)
.onSuccessTask { results ->
return listDocumentSnapshot
}
This would allow you to use the following along with other Task and Tasks methods.
getDocumentByQueryAList(idQueryList)
.addOnSuccessListener { /* ... */ }
.addOnFailureListener { /* ... */ }

Use smth like this using RxJava:
override fun getAllDocs(): Single<List<MyClass>> {
return Single.create { emitter ->
db.collection("myCollection").get()
.addOnSuccessListener { documents ->
val list = mutableListOf<MyClass>()
documents.forEach { document ->
list.add(mapDocumentToMyClass(document))}
emitter.onSuccess(list)
}
.addOnFailureListener { exception ->
emitter.onError(exception)
}
}
}
private fun mapDocumentToMyClass(documentSnapshot: QueryDocumentSnapshot) =
MyClass(
documentSnapshot.get(ID).toString(),
documentSnapshot.get(SMTH).toString(),
// some extra fields
null
)
Or smth like this using coroutine:
override suspend fun getAllDocs(): List<MyClass> {
return try {
val snapshot =
db.collection("myCollection")
.get()
.await()
snapshot.map {
mapDocumentToMyClass(it)
}
} catch (e: Exception) {
throw e
}
}
This functions helps you to get all data from one doc

Kotlin Coroutines - Suspend function returning a Flow runs forever

I am making a network repository that supports multiple data retrieval configs, therefore I want to separate those configs' logic into functions.
However, I have a config that fetches the data continuously at specified intervals. Everything is fine when I emit those values to the original Flow. But when I take the logic into another function and return another Flow through it, it stops caring about its coroutine scope. Even after the scope's cancelation, it keeps on fetching the data.
TLDR: Suspend function returning a flow runs forever when currentCoroutineContext is used to control its loop's termination.
What am I doing wrong here?
Here's the simplified version of my code:
Fragment calling the viewmodels function that basically calls the getData()
lifecycleScope.launch {
viewModel.getLatestDataList()
}
Repository
suspend fun getData(config: MyConfig): Flow<List<Data>>
{
return flow {
when (config)
{
CONTINUOUS ->
{
//It worked fine when fetchContinuously was ingrained to here and emitted directly to the current flow
//And now it keeps on running eternally
fetchContinuously().collect { updatedList ->
emit(updatedList)
}
}
}
}
}
//Note logic of this function is greatly reduced to keep the focus on the problem
private suspend fun fetchContinuously(): Flow<List<Data>>
{
return flow {
while (currentCoroutineContext().isActive)
{
val updatedList = fetchDataListOverNetwork().await()
if (updatedList != null)
{
emit(updatedList)
}
delay(refreshIntervalInMs)
}
Timber.i("Context is no longer active - terminating the continuous-fetch coroutine")
}
}
private suspend fun fetchDataListOverNetwork(): Deferred<List<Data>?> =
withContext(Dispatchers.IO) {
return#withContext async {
var list: List<Data>? = null
try
{
val response = apiService.getDataList().execute()
if (response.isSuccessful && response.body() != null)
{
list = response.body()!!.list
}
else
{
Timber.w("Failed to fetch data from the network database. Error body: ${response.errorBody()}, Response body: ${response.body()}")
}
}
catch (e: Exception)
{
Timber.w("Exception while trying to fetch data from the network database. Stacktrace: ${e.printStackTrace()}")
}
finally
{
return#async list
}
list //IDE is not smart enough to realize we are already returning no matter what inside of the finally block; therefore, this needs to stay here
}
}

I am not sure whether this is a solution to your problem, but you do not need to have a suspending function that returns a Flow. The lambda you are passing is a suspending function itself:
fun <T> flow(block: suspend FlowCollector<T>.() -> Unit): Flow<T> (source)
Here is an example of a flow that repeats a (GraphQl) query (simplified - without type parameters) I am using:
override fun query(query: Query,
updateIntervalMillis: Long): Flow<Result<T>> {
return flow {
// this ensures at least one query
val result: Result<T> = execute(query)
emit(result)
while (coroutineContext[Job]?.isActive == true && updateIntervalMillis > 0) {
delay(updateIntervalMillis)
val otherResult: Result<T> = execute(query)
emit(otherResult)
}
}
}

I'm not that good at Flow but I think the problem is that you are delaying only the getData() flow instead of delaying both of them.
Try adding this:
suspend fun getData(config: MyConfig): Flow<List<Data>>
{
return flow {
when (config)
{
CONTINUOUS ->
{
fetchContinuously().collect { updatedList ->
emit(updatedList)
delay(refreshIntervalInMs)
}
}
}
}
}
Take note of the delay(refreshIntervalInMs).

Android Paging library does not work with Asynchronous requests

The Android Paging Library does not work when making asynchronous network calls using Retrofit. I am using the Google's sample code for Architecture Components on Github, and modified it for my needs.
I had faced the same issue previously but got around it by making synchronous call since the use-case allowed it. But in the current scenario, there are multiple network calls required and the data repository returns the combined result. I am using RxJava for this purpose.
Initially it seemed like a multi-threading issue, but this answer suggests otherwise. Observing the RxJava call on the Main Thread also does not work.
I have added the relevant code below. I stepped into the callback.onResult while debugging and everything works as expected. But ultimately it does not notify the Recycler View Adapter.
View Model snippet:
open fun search(query : String, init : Boolean = false) : Boolean {
return if(query == searchQuery.value && !init) {
false
} else {
searchQuery.value = query
true
}
}
fun refresh() {
listing.value?.refresh?.invoke()
}
var listing : LiveData<ListingState<T>> = Transformations.map(searchQuery) {
getList() // Returns the Listing State from the Repo snippet added below.
}
Repository snippet:
val dataSourceFactory = EvaluationCandidateDataSourceFactory(queryParams,
Executors.newFixedThreadPool(5) )
val pagelistConfig = PagedList.Config.Builder()
.setEnablePlaceholders(true)
.setInitialLoadSizeHint(5)
.setPageSize(25)
.setPrefetchDistance(25).build()
val pagedList = LivePagedListBuilder<Int, PC>(
dataSourceFactory, pagelistConfig)
.setFetchExecutor(Executors.newFixedThreadPool(5)).build()
val refreshState = Transformations.switchMap(dataSourceFactory.dataSource) {
it.initialState
}
return ListingState(
pagedList = pagedList,
pagingState = Transformations.switchMap(dataSourceFactory.dataSource) {
it.pagingState
},
refreshState = refreshState,
refresh = {
dataSourceFactory.dataSource.value?.invalidate()
},
retry = {
dataSourceFactory.dataSource.value?.retryAllFailed()
}
)
Data Source snippet :
override fun loadInitial(params: LoadInitialParams<Int>, callback: LoadInitialCallback<Int, PC>) {
try {
queryMap = if (queryMap == null) {
hashMapOf("page" to FIRST_PAGE)
} else {
queryMap.apply { this!!["page"] = FIRST_PAGE }
}
initialState.postValue(DataSourceState.LOADING)
pagingState.postValue(DataSourceState.LOADING)
val disposable : Disposable = aCRepositoryI.getAssignedAC(queryMap)
.subscribeOn(AndroidSchedulers.mainThread())
.observeOn(AndroidSchedulers.mainThread())
.subscribe({
if(it.success) {
// remove possible retries on success
retry = null
val nextPage = it.responseHeader?.let { getNextPage(it, FIRST_PAGE) } ?: run { null }
val previousPage = getPreviousPage(FIRST_PAGE)
callback.onResult(it.response.pcList, previousPage, nextPage)
initialState.postValue(DataSourceState.SUCCESS)
pagingState.postValue(DataSourceState.SUCCESS)
} else {
// let the subscriber decide whether to retry or not
retry = {
loadInitial(params, callback)
}
initialState.postValue(DataSourceState.failure(it.networkError.message))
pagingState.postValue(DataSourceState.failure(it.networkError.message))
Timber.e(it.networkError.message)
}
}, {
retry = {
loadInitial(params, callback)
}
initialState.postValue(DataSourceState.failure(it.localizedMessage))
pagingState.postValue(DataSourceState.failure(it.localizedMessage))
})
} catch (ex : Exception) {
retry = {
loadInitial(params, callback)
}
initialState.postValue(DataSourceState.failure(ex.localizedMessage))
pagingState.postValue(DataSourceState.failure(ex.localizedMessage))
Timber.e(ex)
}
}
Can someone please tell what is the issue here. There is a similar issue I mentioned above, but it recommends using synchronous calls. How can we do it using asynchronous calls or with RxJava.

I don't understand why you want to go to the main thread. The loading methods in the DataSource are running in a background thread. This means you can do synchronous work on this thread without blocking the main thread, which means you could just think of a solution without RxJava. Something like:
override fun loadInitial(params: LoadInitialParams<Int>, callback: LoadInitialCallback<Int, PC>) {
try {
val result = repository.fetchData(..)
// post result values and call the callback
catch (e: Exception) {
// post error values and log and save the retry
}
}
Then in your repository you can do this because we are not on the main thread.
fun fetchData(...) {
val response = myRetrofitService.someBackendCall(..).execute()
response.result?.let {
return mapResponse(it)
} ?: throw HttpException(response.error)
}
I might have messed up the syntax but I hope you get the point. No callbacks, no subscribing/observing but simple straightforward code.
Also if you start threading inside the loadInitial(..) method your initial list will be empty, so doing things synchronously also avoids seeing empty lists.