I'm investigating the use of Kotlin Flow within my current Android application
My application retrieves its data from a remote server via Retrofit API calls.
Some of these API's return 50,000 data items in 500 item pages.
Each API response contains an HTTP Link header containing the Next pages complete URL.
These calls can take up to 2 seconds to complete.
In an attempt to reduce the elapsed time I have employed a Kotlin Flow to concurrently process each page
of data while also making the next page API call.
My flow is defined as follows:
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val internalWorkWorkState = MutableStateFlow<Response<List<MyPage>>?>(null)
private val workWorkState = internalWorkWorkState.asStateFlow()
private val myJob: Job
init {
myJob = GlobalScope.launch(persistenceThreadPool) {
workWorkState.collect { page ->
if (page == null) {
} else managePage(page!!)
}
}
}
My Recursive function is defined as follows that fetches all pages:-
private suspend fun managePages(accessToken: String, response: Response<List<MyPage>>) {
when {
result != null -> return
response.isSuccessful -> internalWorkWorkState.emit(response)
else -> {
manageError(response.errorBody())
result = Result.failure()
return
}
}
response.headers().filter { it.first == HTTP_HEADER_LINK && it.second.contains(REL_NEXT) }.forEach {
val parts = it.second.split(OPEN_ANGLE, CLOSE_ANGLE)
if (parts.size >= 2) {
managePages(accessToken, service.myApiCall(accessToken, parts[1]))
}
}
}
private suspend fun managePage(response: Response<List<MyPage>>) {
val pages = response.body()
pages?.let {
persistResponse(it)
}
}
private suspend fun persistResponse(myPage: List<MyPage>) {
val myPageDOs = ArrayList<MyPageDO>()
myPage.forEach { page ->
myPageDOs.add(page.mapDO())
}
database.myPageDAO().insertAsync(myPageDOs)
}
My numerous issues are
This code does not insert all data items that I retrieve
How do complete the flow when all data items have been retrieved
How do I complete the GlobalScope job once all the data items have been retrieved and persisted
UPDATE
By making the following changes I have managed to insert all the data
private val persistenceThreadPool = Executors.newFixedThreadPool(3).asCoroutineDispatcher()
private val completed = CompletableDeferred<Int>()
private val channel = Channel<Response<List<MyPage>>?>(UNLIMITED)
private val channelFlow = channel.consumeAsFlow().flowOn(persistenceThreadPool)
private val frank: Job
init {
frank = GlobalScope.launch(persistenceThreadPool) {
channelFlow.collect { page ->
if (page == null) {
completed.complete(totalItems)
} else managePage(page!!)
}
}
}
...
...
...
channel.send(null)
completed.await()
return result ?: Result.success(outputData)
I do not like having to rely on a CompletableDeferred, is there a better approach than this to know when the Flow has completed everything?
You are looking for the flow builder and Flow.buffer():
suspend fun getData(): Flow<Data> = flow {
var pageData: List<Data>
var pageUrl: String? = "bla"
while (pageUrl != null) {
TODO("fetch pageData from pageUrl and change pageUrl to the next page")
emitAll(pageData)
}
}
.flowOn(Dispatchers.IO /* no need for a thread pool executor, IO does it automatically */)
.buffer(3)
You can use it just like a normal Flow, iterate, etc. If you want to know the total length of the output, you should calculate it on the consumer with a mutable closure variable. Note you shouldn't need to use GlobalScope anywhere (ideally ever).
There are a few ways to achieve the desired behaviour. I would suggest to use coroutineScope which is designed specifically for parallel decomposition. It also provides good cancellation and error handling behaviour out of the box. In conjunction with Channel.close behaviour it makes the implementation pretty simple. Conceptually the implementation may look like this:
suspend fun fetchAllPages() {
coroutineScope {
val channel = Channel<MyPage>(Channel.UNLIMITED)
launch(Dispatchers.IO){ loadData(channel) }
launch(Dispatchers.IO){ processData(channel) }
}
}
suspend fun loadData(sendChannel: SendChannel<MyPage>){
while(hasMoreData()){
sendChannel.send(loadPage())
}
sendChannel.close()
}
suspend fun processData(channel: ReceiveChannel<MyPage>){
for(page in channel){
// process page
}
}
It works in the following way:
coroutineScope suspends until all children are finished. So you don't need CompletableDeferred anymore.
loadData() loads pages in cycle and posts them into the channel. It closes the channel as soon as all pages have been loaded.
processData fetches items from the channel one by one and process them. The cycle will finish as soon as all the items have been processed (and the channel has been closed).
In this implementation the producer coroutine works independently, with no back-pressure, so it can take a lot of memory if the processing is slow. Limit the buffer capacity to have the producer coroutine suspend when the buffer is full.
It might be also a good idea to use channels fan-out behaviour to launch multiple processors to speed up the computation.
Related
I have an observable in my foreground service which fetch data from a paging API and save it to the database, the foreground service shows a notification with a progress bar with number of saved items vs the total amount.
Observable which fetch all the data looks like this:
private fun getAllProducts(): Observable<Response<List<ProdottoBarcode>>> {
val lastId = intArrayOf(0)
return Observable.range(1, Integer.MAX_VALUE - 1)
.concatMap { currentPage -> getProducts(currentPage, lastId[0]) }
.takeUntil { response -> response.body()?.isEmpty() == true }
.doOnNext { response ->
lastId[0] = response.headers().get("lastId")?.toInt()!!
}
}
Then the subscription is done in onCreate() like this:
override fun onCreate() {
super.onCreate()
...
getAllProducts().subscribeWith(object: DisposableObserver<Response<List<ProdottoBarcode>>>() {
override fun onNext(response: Response<List<ProdottoBarcode>>) {
if (response.isSuccessful) {
val products = response.body()
val totalItems = response.headers().get("items")?.toInt()
insertProducts(totalItems, products)
}
}
override fun onError(e: Throwable) {
stopService()
}
override fun onComplete() {
}
})
}
And the method which saves all the data to the database looks like this:
private fun insertProducts(totalItems: Int?, products: List<ProdottoBarcode>?) {
if (products != null) {
CoroutineScope(Dispatchers.IO).launch {
for (product in products) {
repository.insert(product)
savedItems += 1
val notification =
totalItems?.let { items ->
NotificationCompat.Builder(baseContext, "progress_channel")
.setSmallIcon(R.drawable.ic_box)
.setContentTitle("Sincronizzati: $savedItems prodotti su $totalItems")
.setProgress(items, savedItems, false)
.setOngoing(true)
.build()
}
notificationManager.notify(notificationId, notification)
}
// TODO: stop the service and dismiss the notification when all items has been saved
if (savedItems == totalItems) {
stopService()
}
}
}
}
The stopService() in insertProducts not always works, while if I try to put stopService in onComplete() it will be executed once all subscriptions are done and NOT when all the items has been saved.
So my question is:
How can I stop my services by using the Coroutine inside the Observable? I need to know when all items from all observables are insert in database and only then to dismiss the service.
Side note: you don't need to do Int wrapping like this in Kotlin like you would in Java. Kotlin has implicit variable wrapping, so you can simply use a var local variable and it will be captured by whatever function you use it in.
val lastId = intArrayOf(0) // can just be var lastId = 0
Starting with getProducts() for fetching a page. I think the code you linked is OK provided your Retrofit service's getProducts function is marked suspend, so it's not blocking. No changes here.
private suspend fun getProducts(
page: Int,
lastId: Int,
itemsPerPage: Int = 50
): Response<List<ProdottoBarcode>> {
val prefs = PreferenceManager.getDefaultSharedPreferences(this)
val urlServer = prefs.getString("server", "http://127.0.0.1/")!!
return RetrofitClient.getInstance().getService()
.getProducts(urlServer, "A", page, lastId, itemsPerPage)
}
Your getAllProducts in your linked code doesn't need backing StateFlows that are never collected--you're using them simply as mutable Int wrappers, which are unnecessary in Kotlin as mentioned way above. I'm not exactly sure how you're consuming these pages, since I'm not very familiar with Rx, but I take the use of concatMap to mean that the Observable is queuing up pages as fast as it can into a buffer, and you are reading out these pages to some local property that the UI uses. I think a buffer should be added so we can be inserting in the database in parallel with fetching the next page. Default buffer arguments are probably appropriate.
private val allProducts: Flow<Response<List<ProdottoBarcode>>> = flow {
var lastId = 0
for (currentPage in 1 until Int.MAX_VALUE) {
emit(getProducts(currentPage, lastId))
lastId = response.headers().get("lastId")!!.toInt()
}
}
.takeWhile { response -> !response.body().isNullOrEmpty() }
.buffer()
Usually, when you collect your flow, you should use an appropriate coroutine scope provided by the Android framework, so it will automatically cancel collection once it goes out of scope. If you inherit your service from LifecycleService, you can use the existing lifecycleScope. This is maybe not so critical in a service in this case, since I think you are only calling stopService() when your flow is complete, but it would make it a little more robust against potential mistakes, I think.
.launchIn is a shortcut that is like wrapping everything above it in launch and calling collect() on it. I prefer the syntax because it has less nesting of code.
override fun onCreate() {
super.onCreate()
// ...
allProducts.onEach { response ->
if (response.isSuccessful) {
val products = response.body()
val totalItems = response.headers().get("items")?.toInt()
insertProducts(totalItems, products)
}
}
.catch { Log.e(TAG, "Failed collecting page.", it) }
.onCompletion { stopService() }
.flowOn(Dispatchers.Default) // don't use main thread since this is a service
.launchIn(lifecycleScope)
}
Since we're using buffer() in the fetching flow, we don't need to launch other coroutines when inserting in the database to achieve parallelism. We can simplify this into a suspend function. We are handling stopping the service in the flow collector, so we don't need to do that here either. I'm assuming repository.insert is a suspend function, not blocking.
private suspend fun insertProducts(totalItems: Int?, products: List<ProdottoBarcode>?) {
if (totalItems == null) {
Log.e(TAG, "Tried to insert items without any item count. Skipping.")
return
}
if (products == null) {
Log.e(TAG, "Tried to insert null products list. Skipping.")
return
}
for (product in products) {
repository.insert(product)
savedItems += 1
val notification = NotificationCompat.Builder(baseContext, "progress_channel")
.setSmallIcon(R.drawable.ic_box)
.setContentTitle("Sincronizzati: $savedItems prodotti su $totalItems")
.setProgress(items, savedItems, false)
.setOngoing(true)
.build()
}
notificationManager.notify(notificationId, notification)
}
}
I'm trying to show a user information in DetailActivity. So, I request a data and get a data for the user from server. but in this case, the return type is Flow<User>. Let me show you the following code.
ServiceApi.kt
#GET("endpoint")
suspend fun getUser(#Query("id") id: Int): Response<User>
Repository.kt
fun getUser(id: Int): Flow<User> = flow<User> {
val userResponse = api.getUser(id = id)
if (userResponse.isSuccessful) {
val user = userResponse.body()
emit(user)
}
}
.flowOn(Dispatchers.IO)
.catch { // send error }
DetailViewModel.kt
class DetailViewModel(
private val repository : Repository
) {
val uiState: StateFlow<User> = repository.getUser(id = 369).stateIn(
scope = viewModelScope,
started = SharingStarted.WhileSubscribed(5000),
initialValue = User() // empty user
)
}
DetailActivity.kt
class DetailActivity: AppCompatActivity() {
....
initObersevers() {
lifecycleScope.launch {
// i used the `flowWithLifecycle` because the data is just a single object.
viewModel.uiState.flowWithLifecycle(lifecycle, Lifecycle.State.STARTED).collect { state ->
// show data
}
}
}
...
}
But, all of sudden, I just realized that this process is just an one-shot operation and thought i can use suspend function and return User in Repository.kt.
So, i changed the Repository.kt.
Repository.kt(changed)
suspend fun getUser(id: Int): User {
val userResponse = api.getUser(id = id)
return if(userResponse.isSuccessful) {
response.body()
} else {
User() // empty user
}
}
And in DetailViewModel, i want to convert the User into StateFlow<User> because of observing from DetailActivity and I'm going to use it the same way as before by using flowWithLifecycle.
the concept is... i thought it's just one single data and i dind't need to use Flow in Repository. because it's not several items like List.
is this way correct or not??
Yeap, this one-time flow doesn't make any sense - it emits only once and that's it.
You've got two different ways. First - is to create a state flow in your repo and emit there any values each time you're doing your GET request. This flow will be exposed to the use case and VM levels. I would say that it leads to more difficult error handling (I'm not fond of this way, but these things are always arguable, haha), but it also has some pros like caching, you can always show/get the previous results.
Second way is to leave your request as a simple suspend function which sends a request, returns the result of it back to your VM (skipping error handling here to be simple):
val userFlow: Flow<User>
get() = _userFlow
private val _userFlow = MutableStateFlow(User())
fun getUser() = launch(viewModelScope) {
_userFlow.value = repository.getUser()
}
This kind of implementation doesn't provide any cache out of scope of this VM's lifecycle, but it's easy to test and use.
So it's not like there is only one "the-coolest-way-to-do-it", it's rather a question what suits you more for your needs.
I am reading this article to fully understand the dos and donts of using Flow while comparing it to my implementation, but I can't grasp clearly how to tell if you are wasting resource when using Flow or flow builder. When is the time a flow is being release/freed in memory and when is the time that you are wasting resource like accidentally creating multiple instances of flow and not releasing them?
I have a UseCase class that invokes a repository function that returns Flow. In my ViewModel this is how it looks like.
class AssetViewModel constructor(private val getAssetsUseCase: GetAssetsUseCase) : BaseViewModel() {
private var job: Job? = null
private val _assetState = defaultMutableSharedFlow<AssetState>()
fun getAssetState() = _assetState.asSharedFlow()
init {
job = viewModelScope.launch {
while(true) {
if (lifecycleState == LifeCycleState.ON_START || lifecycleState == LifeCycleState.ON_RESUME)
fetchAssets()
delay(10_000)
}
}
}
fun fetchAssets() {
viewModelScope.launch {
withContext(Dispatchers.IO) {
getAssetsUseCase(
AppConfigs.ASSET_BASE_URL,
AppConfigs.ASSET_PARAMS,
AppConfigs.ASSET_SIZES[AppConfigs.ASSET_LIMIT_INDEX]
).onEach {
when(it){
is RequestStatus.Loading -> {
_assetState.tryEmit(AssetState.FetchLoading)
}
is RequestStatus.Success -> {
_assetState.tryEmit(AssetState.FetchSuccess(it.data.assetDataDomain))
}
is RequestStatus.Failed -> {
_assetState.tryEmit(AssetState.FetchFailed(it.message))
}
}
}.collect()
}
}
}
override fun onCleared() {
job?.cancel()
super.onCleared()
}
}
The idea here is we are fetching data from remote every 10 seconds while also allowing on demand fetch of data via UI.
Just a typical useless UseCase class
class GetAssetsUseCase #Inject constructor(
private val repository: AssetsRepository // Passing interface not implementation for fake test
) {
operator fun invoke(baseUrl: String, query: String, limit: String): Flow<RequestStatus<AssetDomain>> {
return repository.fetchAssets(baseUrl, query, limit)
}
}
The concrete implementation of repository
class AssetsRepositoryImpl constructor(
private val service: CryptoService,
private val mapper: AssetDtoMapper
) : AssetsRepository {
override fun fetchAssets(
baseUrl: String,
query: String,
limit: String
) = flow {
try {
emit(RequestStatus.Loading())
val domainModel = mapper.mapToDomainModel(
service.getAssetItems(
baseUrl,
query,
limit
)
)
emit(RequestStatus.Success(domainModel))
} catch (e: HttpException) {
emit(RequestStatus.Failed(e))
} catch (e: IOException) {
emit(RequestStatus.Failed(e))
}
}
}
After reading this article which says that using stateIn or sharedIn will improve the performance when using a flow, it seems that I am creating new instances of the same flow on-demand. But there is a limitation as the stated approach only works for variable and not function that returns Flow.
stateIn and shareIn can save resources if there are multiple observers, by avoiding redundant fetching. And in your case, you could set it up to automatically pause the automatic re-fetching when there are no observers. If, on the UI side you use repeatOnLifecycle, then it will automatically drop your observers when the view is off screen and then you will avoid wasted fetches the user will never see.
I think it’s not often described this way, but often the multiple observers are just observers coming from the same Activity or Fragment class after screen rotations or rapidly switching between fragments. If you use WhileSubscribed with a timeout to account for this, you can avoid having to restart your flow if it’s needed again quickly.
Currently you emit to from an external coroutine instead of using shareIn, so there’s no opportunity to pause execution.
I haven't tried to create something that supports both automatic and manual refetching. Here's a possible strategy, but I haven't tested it.
private val refreshRequest = Channel<Unit>(Channel.CONFLATED)
fun fetchAssets() {
refreshRequest.trySend(Unit)
}
val assetState = flow {
while(true) {
getAssetsUseCase(
AppConfigs.ASSET_BASE_URL,
AppConfigs.ASSET_PARAMS,
AppConfigs.ASSET_SIZES[AppConfigs.ASSET_LIMIT_INDEX]
).map {
when(it){
is RequestStatus.Loading -> AssetState.FetchLoading
is RequestStatus.Success -> AssetState.FetchSuccess(it.data.assetDataDomain)
is RequestStatus.Failed -> AssetState.FetchFailed(it.message)
}
}.emitAll()
withTimeoutOrNull(100L) {
// drop any immediate or pending manual request
refreshRequest.receive()
}
// Wait until a fetch is manually requested or ten seconds pass:
withTimeoutOrNull(10000L - 100L) {
refreshRequest.receive()
}
}
}.shareIn(viewModelScope, SharingStarted.WhileSubscribed(4000L), replay = 1)
To this I would recommend not using flow as the return type of the usecase function and the api call must not be wrapped inside a flow builder.
Why:
The api call actually is happening once and then again after an interval it is triggered by the view model itself, returning flow from the api caller function will be a bad usage of powerful tool that is actually meant to be called once and then it must be self-reliant, it should emit or pump in the data till the moment it has a subscriber/collector.
One usecase you can consider when using flow as return type from the room db query call, it is called only once and then the room emits data into it till the time it has subscriber.
.....
fun fetchAssets() {
viewModelScope.launch {
// loading true
val result=getusecase(.....)
when(result){..process result and emit on state..}
// loading false
}
}
.....
suspend operator fun invoke(....):RequestStatus<AssetDomain>{
repository.fetchAssets(baseUrl, query, limit)
}
.....
override fun fetchAssets(
baseUrl: String,
query: String,
limit: String
):RequestStatus {
try {
//RequestStatus.Loading()//this can be managed in viewmodel itself
val domainModel = mapper.mapToDomainModel(
service.getAssetItems(
baseUrl,
query,
limit
)
)
RequestStatus.Success(domainModel)
} catch (e: HttpException) {
RequestStatus.Failed(e)
} catch (e: IOException) {
RequestStatus.Failed(e)
}
}
I have still a little bit of trouble putting all information together about the thread-safety of using coroutines to launch network requests.
Let's say we have following use-case, there is a list of users we get and for each of those users, I will do some specific check which has to run over a network request to the API, giving me some information back about this user.
The userCheck happens inside a library, which doesn't expose suspend functions but rather still uses a callback.
Inside of this library, I have seen code like this to launch each of the network requests:
internal suspend fun <T> doNetworkRequest(request: suspend () -> Response<T>): NetworkResult<T> {
return withContext(Dispatchers.IO) {
try {
val response = request.invoke()
...
According to the documentation, Dispatchers.IO can use multiple threads for the execution of the code, also the request function is simply a function from a Retrofit API.
So what I did is to launch the request for each user, and use a single resultHandler object, which will add the results to a list and check if the length of the result list equals the length of the user list, if so, then all userChecks are done and I know that I can do something with the results, which need to be returned all together.
val userList: List<String>? = getUsers()
val userCheckResultList = mutableListOf<UserCheckResult>()
val handler = object : UserCheckResultHandler {
override fun onResult(
userCheckResult: UserCheckResult?
) {
userCheckResult?.let {
userCheckResultList.add(
it
)
}
if (userCheckResultList.size == userList?.size) {
doSomethingWithResultList()
print("SUCCESS")
}
}
}
userList?.forEach {
checkUser(it, handler)
}
My question is: Is this implementation thread-safe? As far as I know, Kotlin objects should be thread safe, but I have gotten feedback that this is possibly not the best implementation :D
But in theory, even if the requests get launched asynchronous and multiple at the same time, only one at a time can access the lock of the thread the result handler is running on and there will be no race condition or problems with adding items to the list and comparing the sizes.
Am I wrong about this?
Is there any way to handle this scenario in a better way?
If you are executing multiple request in parallel - it's not. List is not thread safe. But it's simple fix for that. Create a Mutex object and then just wrap your operation on list in lock, like that:
val lock = Mutex()
val userList: List<String>? = getUsers()
val userCheckResultList = mutableListOf<UserCheckResult>()
val handler = object : UserCheckResultHandler {
override fun onResult(
userCheckResult: UserCheckResult?
) {
lock.withLock {
userCheckResult?.let {
userCheckResultList.add(
it
)
}
if (userCheckResultList.size == userList?.size) {
doSomethingWithResultList()
print("SUCCESS")
}
}
}
}
userList?.forEach {
checkUser(it, handler)
}
I have to add that this whole solution seems very hacky. I would go completely other route. Run all of your requests wrapping those in async { // network request } which will return Deferred object. Add this object to some list. After that wait for all of those deferred objects using awaitAll(). Like that:
val jobs = mutableListOf<Job>()
userList?.forEach {
// i assume checkUser is suspendable here
jobs += async { checkUser(it, handler) }
}
// wait for all requests
jobs.awaitAll()
// After that you can access all results like this:
val resultOfJob0 = jobs[0].getCompleted()
What I'm trying to do
I have an app that's using Room with Coroutines to save search queries in the database. It's also possible to add search suggestions and later on I retrieve this data to show them on a list. I've also made it possible to "pin" some of those suggestions.
My data structure is something like this:
#Entity(
tableName = "SEARCH_HISTORY",
indices = [Index(value = ["text"], unique = true)]
)
data class Suggestion(
#PrimaryKey(autoGenerate = true)
#ColumnInfo(name = "suggestion_id")
val suggestionId: Long = 0L,
val text: String,
val type: SuggestionType,
#ColumnInfo(name = "insert_date")
val insertDate: Calendar
)
enum class SuggestionType(val value: Int) {
PINNED(0), HISTORY(1), SUGGESTION(2)
}
I have made the "text" field unique to avoid repeated suggestions with different states/types. E.g.: A suggestion that's a pinned item and a previously queried text.
My Coroutine setup looks like this:
private val parentJob: Job = Job()
private val IO: CoroutineContext
get() = parentJob + Dispatchers.IO
private val MAIN: CoroutineContext
get() = parentJob + Dispatchers.Main
private val COMPUTATION: CoroutineContext
get() = parentJob + Dispatchers.Default
And my DAOs are basically like this:
#Insert(onConflict = OnConflictStrategy.REPLACE)
suspend fun insert(obj: Suggestion): Long
#Insert(onConflict = OnConflictStrategy.REPLACE)
suspend fun insert(objList: List<Suggestion>): List<Long>
I also have the following public functions to insert the data into the database:
fun saveQueryToDb(query: String, insertDate: Calendar) {
if (query.isBlank()) {
return
}
val suggestion = Suggestion(
text = query,
insertDate = insertDate,
type = SuggestionType.HISTORY
)
CoroutineScope(IO).launch {
suggestionDAO.insert(suggestion)
}
}
fun addPin(pin: String) {
if (pin.isBlank()) {
return
}
val suggestion = Suggestion(
text = pin,
insertDate = Calendar.getInstance(),
type = SuggestionType.PINNED
)
CoroutineScope(IO).launch {
suggestionDAO.insert(suggestion)
}
}
fun addSuggestions(suggestions: List<String>) {
addItems(suggestions, SuggestionType.SUGGESTION)
}
private fun addItems(items: List<String>, suggestionType: SuggestionType) {
if (items.isEmpty()) {
return
}
CoroutineScope(COMPUTATION).launch {
val insertDate = Calendar.getInstance()
val filteredList = items.filterNot { it.isBlank() }
val suggestionList = filteredList.map { History(text = it, insertDate = insertDate, suggestionType = suggestionType) }
withContext(IO) {
suggestionDAO.insert(suggestionList)
}
}
}
There are also some other methods, but let's focus on the ones above.
EDIT: All of the methods above are part of a lib that I made, they're are not made suspend because I don't want to force a particular type of programming to the user, like forcing to use Rx or Coroutines when using the lib.
The problem
Let's say I try to add a list of suggestions using the addSuggestions() method stated above, and that I also try to add a pinned suggestion using the addPin() method. The pinned text is also present in the suggestion list.
val list = getSuggestions() // Getting a list somewhere
addSuggestions(list)
addPin(list.first())
When I try to do this, sometimes the pin is added first and then it's overwritten by the suggestion present in the list, which makes me think I might've been dealing with some sort of race condition. Since the addSuggestions() method has more data to handle, and both methods will run in parallel, I believe the addPin() method is completing first.
Now, my Coroutines knowledge is pretty limited and I'd like to know if there's a way to enqueue those method calls and make sure they'll execute in the exact same order I invoked them, that must be strongly guaranteed to avoid overriding data and getting funky results later on. How can I achieve such behavior?
I'd follow the Go language slogan "Don't communicate by sharing memory; share memory by communicating", that means instead of maintaining atomic variables or jobs and trying to synchronize between them, model your operations as messages and use Coroutines actors to handle them.
sealed class Message {
data AddSuggestions(val suggestions: List<String>) : Message()
data AddPin(val pin: String) : Message()
}
And in your class
private val parentScope = CoroutineScope(Job())
private val actor = parentScope.actor<Message>(Dispatchers.IO) {
for (msg in channel) {
when (msg) {
is Message.AddSuggestions -> TODO("Map to the Suggestion and do suggestionDAO.insert(suggestions)")
is Message.AddPin -> TODO("Map to the Pin and do suggestionDAO.insert(pin)")
}
}
}
fun addSuggestions(suggestions: List<String>) {
actor.offer(Message.AddSuggestions(suggestions))
}
fun addPin(pin: String) {
actor.offer(Message.AddPin(pin))
}
By using actors you'll be able to queue messages and they will be processed in FIFO order.
By default when you call .launch{}, it launches a new coroutine without blocking the current thread and returns a reference to the coroutine as a Job. The coroutine is canceled when the resulting job is canceled.
It doesn't care or wait for other parts of your code it just runs.
But you can pass a parameter to basically tell it to run immediately or wait for other Coroutine to finish(LAZY).
For Example:
val work_1 = CoroutineScope(IO).launch( start = CoroutineStart.LAZY ){
//do dome work
}
val work_2 = CoroutineScope(IO).launch( start = CoroutineStart.LAZY ){
//do dome work
work_1.join()
}
val work_3 = CoroutineScope(IO).launch( ) {
//do dome work
work_2.join()
}
When you execute the above code first work_3 will finish and invoke work_2 when inturn invoke Work_1 and so on,
The summary of coroutine start options is:
DEFAULT -- immediately schedules coroutine for execution according to its context
LAZY -- starts coroutine lazily, only when it is needed
ATOMIC -- atomically (in a non-cancellable way) schedules coroutine for execution according to its context
UNDISPATCHED -- immediately executes coroutine until its first suspension point in the current thread.
So by default when you call .launch{} start = CoroutineStart.DEFAULT is passed because it is default parameter.
Don't launch coroutines from your database or repository. Use suspending functions and then switch dispatchers like:
suspend fun addPin(pin: String) {
...
withContext(Dispatchers.IO) {
suggestionDAO.insert(suggestion)
}
}
Then from your ViewModel (or Activity/Fragment) make the call:
fun addSuggestionsAndPinFirst(suggestions: List<Suggestion>) {
myCoroutineScope.launch {
repository.addSuggestions(suggestions)
repository.addPin(suggestions.first())
}
}
Why do you have a separate addPin() function anyways? You can just modify a suggestion and then store it:
fun pinAndStoreSuggestion(suggestion: Suggestion) {
myCoroutineScope.launch {
repository.storeSuggestion(suggestion.copy(type = SuggestionType.PINNED)
}
}
Also be careful using a Job like that. If any coroutine fails all your coroutines will be cancelled. Use a SupervisorJob instead. Read more on that here.
Disclaimer: I do not approve of the solution below. I'd rather use an old-fashioned ExecutorService and submit() my Runnable's
So if you really want to synchronize your coroutines in a way that the first function called is also the first one to write to your database. (I'm not sure it is guaranteed since your DAO functions are also suspending and Room uses it's own threads too). Try something like the following unit test:
class TestCoroutineSynchronization {
private val jobId = AtomicInteger(0)
private val jobToRun = AtomicInteger(0)
private val jobMap = mutableMapOf<Int, () -> Unit>()
#Test
fun testCoroutines() = runBlocking {
first()
second()
delay(2000) // delay so our coroutines finish
}
private fun first() {
val jobId = jobId.getAndIncrement()
CoroutineScope(SupervisorJob() + Dispatchers.Default).launch {
delay(1000) // intentionally delay your first coroutine
withContext(Dispatchers.IO) {
submitAndTryRunNextJob(jobId) { println(1) }
}
}
}
private fun second() {
val jobId = jobId.getAndIncrement()
CoroutineScope(SupervisorJob()).launch(Dispatchers.IO) {
submitAndTryRunNextJob(jobId) { println(2) }
}
}
private fun submitAndTryRunNextJob(jobId: Int, action: () -> Unit) {
synchronized(jobMap) {
jobMap[jobId] = action
tryRunNextJob()
}
}
private fun tryRunNextJob() {
var action = jobMap.remove(jobToRun.get())
while (action != null) {
action()
action = jobMap.remove(jobToRun.incrementAndGet())
}
}
}
So what I do on each call is increment a value (jobId) that is later used to prioritize what action to run first. Since you are using suspending function you probably need to add that modifier to the action submitted too (e.g. suspend () -> Unit).