Can't get previous emitted values from Flow.
class TestActivity: ComponentActivity() {
...
private val flowA = MutableStateFlow(0)
private val flowB = MutableStateFlow("")
init {
flowB.onEach { Log.d("flowtest", "test - $it") }
.launchIn(lifecycleScope)
}
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
flowB.value = "1"
flowB.value = "2"
flowB.value = "3"
flowA.map { "ignore" }
.onEach {
flowB.value = "4"
flowB.value = "5"
flowB.value = "6"
}
.launchIn(lifecycleScope)
flowA.value = 0
...
}
expect
test - 1
test - 2
test - 3
test - 4
test - 5
test - 6
result
test - 1
test - 2
test - 3
test - 6
What is missing point that concept of Flow?
How can I get previous emitted values?
This is a bit of a guess since I'm not bothering to put together a project and test it.
First, there are two things to remember about StateFlow.
It is limited to a history of 1. It can only replay a single value to new subscribers.
It is conflated. This means that if a collector is slower than the emissions coming in, the collector will miss some of those emitted values and only get the latest one.
So, looking at your code, at first glance I would expect you to see only:
test - 3
test - 6
This is because you're emitting on the main thread and collecting on the main thread, so whenever you have multiple StateFlow value changes in a row, I would expect only the last one called in a method to "stick" since the collector is having to wait its turn for the main thread to be relinquished before it can collect its next value.
So why do 1 and 2 appear?
Well, actually, lifecycleScope doesn't use Dispatchers.Main. It uses Dispatchers.Main.immediate, which behaves a little differently. The immediate version runs suspending code immediately in place if you are already on the Main thread, instead of yielding to other coroutines first.
So, I'm theorizing that when you change the value on the main thread, but you are collecting flowB's onEach on Dispatchers.Main.immediate, so it gets a chance to immediately run its onEach right in place each time you emit a value to flowB.
But, a reminder, I haven't actually tested flows with immediate to test this hypothesis. It's just the only reason I can think of that would explain the behavior. To test this theory yourself, you can change to using launchIn(lifecycleScope + Dispatchers.Main) in both places and see if 1 and 2 disappear.
In fact, there is something wrong with this code because the value property is a stateFlow property, not a sharedFlow property. In addition, you can't add value to statedFlow after collecting data if the collector and suscriber in the same coroutine. So the right code will be:
val flowA = MutableSharedFlow<Int>()
val flowB = MutableStateFlow("")
flowA.map { "test $it" }
.onEach { flowB.value = it }
flowB.value = "1"
flowB.value = "2"
flowB.value = "3"
flowB.collect { println(it) }
and the result will be: 3
because the stateFlow just keeps the last emitted one
if you want to get all values you can use SharedFlow like that:
suspend fun main():Unit = coroutineScope {
val flowA = MutableSharedFlow<Int>()
launch {
flowA.collect {
println(it)
}
}
launch {
flowA.emit(1)
flowA.emit(2)
flowA.emit(3)
}
And you can check the documentation to get more Info SharedFlow StateFlow
Related
I was testing some Kotlin Flows according to what the document mentioned in this part.
I created a FakeRepository and a FakeViewModel like this:
class FakeRepository {
private val flow = MutableSharedFlow<Int>()
suspend fun emit(value: Int) = flow.emit(value)
val scores: Flow<Int> = flow
}
class FakeViewModel(private val repository: FakeRepository) : ViewModel() {
val score: StateFlow<Int> = repository.scores
.stateIn(viewModelScope, SharingStarted.WhileSubscribed(), 0)
fun initialize() {
viewModelScope.launch {
//delay(100)
repository.emit(1)
//delay(100)
repository.emit(2)
//delay(100)
repository.emit(3)
//delay(100)
}
}
}
I also created a test class to test this score flow like this:
#ExperimentalCoroutinesApi
class FakeViewModelTest {
/*
Replace real dispatchers with instances of TestDispatchers to ensure that all
code runs on the single test thread.
*/
#get:Rule
val mainDispatcherRule = MainDispatcherRule()
#Test
fun testFakeRepository() = runTest {
val fakeRepository = FakeRepository()
val viewModel = FakeViewModel(fakeRepository)
val items = mutableListOf<Int>()
val job = launch(StandardTestDispatcher()) {
viewModel.score.toList(items)
}
viewModel.initialize()
advanceUntilIdle()
MatcherAssert.assertThat(items.size, Matchers.equalTo(4))
job.cancel()
}
}
When I run the test, surprisingly, it fails and says that the list size is 1.
But after uncommenting the delays, the test will be passed.
Why the test behave like this? As the document mentioned here, TestDispatcher will skip delays.
Did I misunderstand something? How can I fix it?
As the StateFlow documentation says:
Updates to the value are always conflated. So a slow collector skips fast updates, but always collects the most recently emitted value.
A StateFlow has no memory whatsoever of previous values. In your current code, the collector and the initialize function's coroutine are racing each other. There's no guarantee that the emitter will wait for an item to be collected before emitting the next.
The possible reason the delay calls make it work is that even with the TestDispatcher, it might be suspending (yielding the thread), which would give the other coroutine a chance to collect each item. Not sure, because I don't know the details of what these TestDispatchers do under the hood.
If you want to guarantee values aren't dropped, you need to use a SharedFlow with BufferOverflow.SUSPEND. Or you could give it a replay of Int.MAX_VALUE, but this could be unsustainable depending on your use case.
I have an instance of CoroutineScope and log() function which look like the following:
private val scope = CoroutineScope(Dispatchers.IO)
fun log(message: String) = scope.launch { // launching a coroutine
println("$message")
TimeUnit.MILLISECONDS.sleep(100) // some blocking operation
}
And I use this test code to launch coroutines:
repeat(5) { item ->
log("Log $item")
}
The log() function can be called from any place, in any Thread, but not from a coroutine.
After a couple of tests I can see not sequential result like the following:
Log 0
Log 2
Log 4
Log 1
Log 3
There can be different order of printed logs. If I understand correctly the execution of coroutines doesn't guarantee to be sequential. What it means is that a coroutine for item 2 can be launched before the coroutine for item 0.
I want that coroutines were launched sequentially for each item and "some blocking operation" would execute sequentially, to always achieve next logs:
Log 0
Log 1
Log 2
Log 3
Log 4
Is there a way to make launching coroutines sequential? Or maybe there are other ways to achieve what I want?
Thanks in advance for any help!
One possible strategy is to use a Channel to join the launched jobs in order. You need to launch the jobs lazily so they don't start until join is called on them. trySend always succeeds when the Channel has unlimited capacity. You need to use trySend so it can be called from outside a coroutine.
private val lazyJobChannel = Channel<Job>(capacity = Channel.UNLIMITED).apply {
scope.launch {
consumeEach { it.join() }
}
}
fun log(message: String) {
lazyJobChannel.trySend(
scope.launch(start = CoroutineStart.LAZY) {
println("$message")
TimeUnit.MILLISECONDS.sleep(100) // some blocking operation
}
)
}
Since Flows are sequential we can use MutableSharedFlow to collect and handle data sequentially:
class Info {
// make sure replay(in case some jobs were emitted before sharedFlow is being collected and could be lost)
// and extraBufferCapacity are large enough to handle all the jobs.
// In case some jobs are lost try to increase either of the values.
private val sharedFlow = MutableSharedFlow<String>(replay = 10, extraBufferCapacity = 10)
private val scope = CoroutineScope(Dispatchers.IO)
init {
sharedFlow.onEach { message ->
println("$message")
TimeUnit.MILLISECONDS.sleep(100) // some blocking or suspend operation
}.launchIn(scope)
}
fun log(message: String) {
sharedFlow.tryEmit(message)
}
}
fun test() {
val info = Info()
repeat(10) { item ->
info.log("Log $item")
}
}
It always prints the logs in the correct order:
Log 0
Log 1
Log 2
...
Log 9
It works for all cases, but need to be sure there are enough elements set to replay and extraBufferCapacity parameters of MutableSharedFlow to handle all items.
Another approach is
Using Dispatchers.IO.limitedParallelism(1) as a context for the CoroutineScope. It makes coroutines run sequentially if they don't contain calls to suspend functions and launched from the same Thread, e.g. Main Thread. So this solution works only with blocking (not suspend) operation inside launch coroutine builder:
private val scope = CoroutineScope(Dispatchers.IO.limitedParallelism(1))
fun log(message: String) = scope.launch { // launching a coroutine from the same Thread, e.g. Main Thread
println("$message")
TimeUnit.MILLISECONDS.sleep(100) // only blocking operation, not `suspend` operation
}
It turns out that the single thread dispatcher is a FIFO executor. So limiting the CoroutineScope execution to one thread solves the problem.
I have an instance of CoroutineScope and log() function which look like the following:
private val scope = CoroutineScope(Dispatchers.IO)
fun log(message: String) = scope.launch { // launching a coroutine
println("$message")
TimeUnit.MILLISECONDS.sleep(100) // some blocking operation
}
And I use this test code to launch coroutines:
repeat(5) { item ->
log("Log $item")
}
The log() function can be called from any place, in any Thread, but not from a coroutine.
After a couple of tests I can see not sequential result like the following:
Log 0
Log 2
Log 4
Log 1
Log 3
There can be different order of printed logs. If I understand correctly the execution of coroutines doesn't guarantee to be sequential. What it means is that a coroutine for item 2 can be launched before the coroutine for item 0.
I want that coroutines were launched sequentially for each item and "some blocking operation" would execute sequentially, to always achieve next logs:
Log 0
Log 1
Log 2
Log 3
Log 4
Is there a way to make launching coroutines sequential? Or maybe there are other ways to achieve what I want?
Thanks in advance for any help!
One possible strategy is to use a Channel to join the launched jobs in order. You need to launch the jobs lazily so they don't start until join is called on them. trySend always succeeds when the Channel has unlimited capacity. You need to use trySend so it can be called from outside a coroutine.
private val lazyJobChannel = Channel<Job>(capacity = Channel.UNLIMITED).apply {
scope.launch {
consumeEach { it.join() }
}
}
fun log(message: String) {
lazyJobChannel.trySend(
scope.launch(start = CoroutineStart.LAZY) {
println("$message")
TimeUnit.MILLISECONDS.sleep(100) // some blocking operation
}
)
}
Since Flows are sequential we can use MutableSharedFlow to collect and handle data sequentially:
class Info {
// make sure replay(in case some jobs were emitted before sharedFlow is being collected and could be lost)
// and extraBufferCapacity are large enough to handle all the jobs.
// In case some jobs are lost try to increase either of the values.
private val sharedFlow = MutableSharedFlow<String>(replay = 10, extraBufferCapacity = 10)
private val scope = CoroutineScope(Dispatchers.IO)
init {
sharedFlow.onEach { message ->
println("$message")
TimeUnit.MILLISECONDS.sleep(100) // some blocking or suspend operation
}.launchIn(scope)
}
fun log(message: String) {
sharedFlow.tryEmit(message)
}
}
fun test() {
val info = Info()
repeat(10) { item ->
info.log("Log $item")
}
}
It always prints the logs in the correct order:
Log 0
Log 1
Log 2
...
Log 9
It works for all cases, but need to be sure there are enough elements set to replay and extraBufferCapacity parameters of MutableSharedFlow to handle all items.
Another approach is
Using Dispatchers.IO.limitedParallelism(1) as a context for the CoroutineScope. It makes coroutines run sequentially if they don't contain calls to suspend functions and launched from the same Thread, e.g. Main Thread. So this solution works only with blocking (not suspend) operation inside launch coroutine builder:
private val scope = CoroutineScope(Dispatchers.IO.limitedParallelism(1))
fun log(message: String) = scope.launch { // launching a coroutine from the same Thread, e.g. Main Thread
println("$message")
TimeUnit.MILLISECONDS.sleep(100) // only blocking operation, not `suspend` operation
}
It turns out that the single thread dispatcher is a FIFO executor. So limiting the CoroutineScope execution to one thread solves the problem.
Is following code safe and why?
val flow: Flow<String> = ...
val allStrings = mutableListOf<String>()
var sum = 0
flow.transform {
allStrings += it
emit(it.toInt())
}.collect {
sum += it
}
Following test demonstrates collect {} body being called from different threads:
val ctx = newFixedThreadPoolContext(32, "my-context")
runBlocking(ctx) {
val f = flow<Int> {
(1 .. 1000).forEach {
emit(it)
}
}
var t: Thread? = null
f.collect {
delay(1)
// this requirement will fail
if (t == null) t = Thread.currentThread() else require(t == Thread.currentThread())
}
}
And another one that tests publication:
fun main(args: Array<String>) {
val ctx = newFixedThreadPoolContext(32, "my-context")
runBlocking(ctx) {
val f = flow<Int> {
(1 .. 1000000).forEach {
emit(it)
}
}
var c = 0
f.transform {
c += 1
boo()
c += 1
emit(it)
c += 1
}.collect {
c += 1
boo()
c += 1
}
println(c) // prints 5_000_000
}
}
suspend fun boo() {
withContext(Dispatchers.IO) {
}
}
Therefore it seems kotlin flow ensures publication between coroutine invocations but is this intentional (or even documented) or implementation side effect?
Yes, the code is safe. Flows are sequential by default.
Each individual collection of a flow is performed sequentially unless
special operators that operate on multiple flows are used. The
collection works directly in the coroutine that calls a terminal
operator (collect). No new coroutines are launched by default. Each
emitted value is processed by all the intermediate operators from
upstream to downstream and is then delivered to the terminal operator
after.
Kotlin Flows are based on suspending functions and they are completely sequential, but not single-threaded.
Therefore it seems kotlin flow ensures publication between coroutine invocations but is this intentional (or even documented) or implementation side effect?
Here it says:
a flow is a type that can emit multiple values sequentially
According to this I understand that no new value will be collected until previous value is processed, no matter how much Threads are involved.
As Roman mention in his comment, here is a good article about sequential execution. A great quote from there:
Even though a coroutine in Kotlin can execute on multiple threads it is just like a thread from a standpoint of mutable state. No two actions in the same coroutine can be concurrent. And just like with threads you should avoid sharing your mutable state between coroutines or you’ll have to worry about synchronization yourself.
Avoid sharing mutable state. Confine each mutable object to either a single thread or to a single coroutine and sleep well.
And this is applicable to Flows, because collection of Flow works directly in the coroutine that calls a terminal operator. No new coroutines are launched by default.
I have a connection to a Bluetooth device that emits data every 250ms
In my viewmodel I wish to subscribe to said data , run some suspending code (which takes approximatelly 1000ms to run) and then present the result.
the following is a simple example of what I'm trying to do
Repository:
class Repo() : CoroutineScope {
private val supervisor = SupervisorJob()
override val coroutineContext: CoroutineContext = supervisor + Dispatchers.Default
private val _dataFlow = MutableSharedFlow<Int>()
private var dataJob: Job? = null
val dataFlow: Flow<Int> = _dataFlow
init {
launch {
var counter = 0
while (true) {
counter++
Log.d("Repo", "emmitting $counter")
_dataFlow.emit(counter)
delay(250)
}
}
}
}
the viewmodel
class VM(app:Application):AndroidViewModel(app) {
private val _reading = MutableLiveData<String>()
val latestReading :LiveData<String>() = _reading
init {
viewModelScope.launch(Dispatchers.Main) {
repo.dataFlow
.map {
validateData() //this is where some validation happens it is very fast
}
.flowOn(Dispatchers.Default)
.forEach {
delay(1000) //this is to simulate the work that is done,
}
.flowOn(Dispatchers.IO)
.map {
transformData() //this will transform the data to be human readable
}
.flowOn(Dispatchers.Default)
.collect {
_reading.postValue(it)
}
}
}
}
as you can see, when data comes, first I validate it to make sure it is not corrupt (on Default dispatcher) then I perform some operation on it (saving and running a long algorithm that takes time on the IO dispatcher) then I change it so the application user can understand it (switching back to Default dispatcher) then I post it to mutable live data so if there is a subscriber from the ui layer they can see the current data (on the Main dispatcher)
I have two questions
a) If validateData fails how can I cancel the current emission and move on to the next one?
b) Is there a way for the dataFlow subscriber working on the viewModel to generate new threads so the delay parts can run in parallel?
the timeline right now looks like the first part, but I want it to run like the second one
Is there a way to do this?
I've tried using buffer() which as the documentation states "Buffers flow emissions via channel of a specified capacity and runs collector in a separate coroutine." but when I set it to BufferOverflow.SUSPEND I get the behaviour of the first part, and when I set it to BufferOverflow.DROP_OLDEST or BufferOverflow.DORP_LATEST I loose emissions
I have also tried using .conflate() like so:
repo.dataFlow
.conflate()
.map { ....
and even though the emissions start one after the other, the part with the delay still waits for the previous one to finish before starting the next one
when I use .flowOn(Dispatchers.Default) for that part , I loose emissions, and when I use .flowOn(Dispatchers.IO) or something like Executors.newFixedThreadPool(4).asCoroutineDispatcher() they always wait for the previous one to finish before starting a new one
Edit 2:
After about 3 hours of experiments this seems to work
viewModelScope.launch(Dispatchers.Default) {
repo.dataFlow
.map {
validateData(it)
}
.flowOn(Dispatchers.Default)
.map {
async {
delay(1000)
it
}
}
.flowOn(Dispatchers.IO) // NOTE (A)
.map {
val result = it.await()
transformData(result)
}
.flowOn(Dispatchers.Default)
.collect {
_readings.postValue(it)
}
}
however I still haven't figured out how to cancel the emission if validatedata fails
and for some reason it only works if I use Dispatchers.IO , Executors.newFixedThreadPool(20).asCoroutineDispatcher() and Dispatchers.Unconfined where I put note (A), Dispatchers.Main does not seem to work (which I expected) but Dispatchers.Default also does not seem to work and I don't know why
First question: Well you cannot recover from an exception in a sense of continuing
the collection of the flow, as per docs "Flow collection can complete with an exception when an emitter or code inside the operators throw an exception." therefore once an exception has been thrown the collection is completed (exceptionally) you can however handle the exception by either wrapping your collection inside try/catch block or using the catch() operator.
Second question: You cannot, while the producer (emitting side) can be made concurrent
by using the buffer() operator, collection is always sequential.
As per your diagram, you need fan out (one producer, many consumers), you cannot
achieve that with flows. Flows are cold, each time you collect from them, they start
emitting from the beginning.
Fan out can be achieved using channels, where you can have one coroutine producing
values and many coroutines that consume those values.
Edit: Oh you meant the validation failed not the function itself, in that case you can use the filter() operator.
The BroadcastChannel and ConflatedBroadcastChannel are getting deprecated. SharedFlow cannot help you in your use case, as they emit values in a broadcast fashion, meaning producer waits until all consumers consume each value before producing the next one. That is still sequential, you need parallelism. You can achieve it using the produce() channel builder.
A simple example:
val scope = CoroutineScope(Job() + Dispatchers.IO)
val producer: ReceiveChannel<Int> = scope.produce {
var counter = 0
val startTime = System.currentTimeMillis()
while (isActive) {
counter++
send(counter)
println("producer produced $counter at ${System.currentTimeMillis() - startTime} ms from the beginning")
delay(250)
}
}
val consumerOne = scope.launch {
val startTime = System.currentTimeMillis()
for (x in producer) {
println("consumerOne consumd $x at ${System.currentTimeMillis() - startTime}ms from the beginning.")
delay(1000)
}
}
val consumerTwo = scope.launch {
val startTime = System.currentTimeMillis()
for (x in producer) {
println("consumerTwo consumd $x at ${System.currentTimeMillis() - startTime}ms from the beginning.")
delay(1000)
}
}
val consumerThree = scope.launch {
val startTime = System.currentTimeMillis()
for (x in producer) {
println("consumerThree consumd $x at ${System.currentTimeMillis() - startTime}ms from the beginning.")
delay(1000)
}
}
Observe production and consumption times.