Confusion on RxJava onBackPressureBuffer and onBackPressureDrop - android

I want my code work like this graph, but it does not work ...
My Code:
private fun <T> register(cls: Class<T>): Flowable<Pair<T, Long>> {
return FlowableFromObservable(mRelay).onBackpressureBuffer(4).filter(
/* filter target event. */
EventPredictable(cls)
).cast(
/* cast to target event */
cls
).onBackpressureDrop {
Log.i(TAG, "drop event: $it")
}.concatMap { data ->
/* start interval task blocking */
val period = 1L
val unit = TimeUnit.SECONDS
MLog.d(TAG, "startInterval: data = $data")
Flowable.interval(0, period, unit).take(DURATION.toLong()).takeUntil(
getStopFlowable()
).map {
Pair(data, it)
}
}
}
private fun getStopFlowable(): Flowable<StopIntervalEvent> {
return RxBus.getDefault().register(StopIntervalEvent::class.java)
.toFlowable(BackpressureStrategy.LATEST)
}
when I send 140 event in 10 ms, my code drop 12 event, not dropping 140 - 4 = 136 event that I expect. Why my code don't work like the graph above? Thank for your watching and answers!

onBackpressurDrop is always ready to receive items thus onBackpressureBuffer has no practical effect in your setup. onBackpressurBuffer(int) would fail on overflow so you'd never se the expected behavior with it. In addition, concatMap fetches 2 items upfront by default so it will get source items 1 and 2.
Instead, try using the overload with the backpressure strategy configurable:
mRelay
.toFlowable(BackpressureStaregy.MISSING)
.onBackpressureBuffer(4, null, BackpressureOverflowStrategy.DROP_LATEST)
.flatMap(data ->
Flowable.intervalRange(0, DURATION.toLong(), 0, period, unit)
.takeUntil(getStopFlowable())
.map(it -> new Pair(data, it))
, 1 // <--------------------------------------------------- max concurrency
);

Related

How to define a dependency between two coroutines?

Consider the following code:
init {
// coroutine 1
// this is and needs to be in a separate coroutine as the collection runs indefinite
viewModelScope.launch {
myService.someSharedFlow.collect {
// handle values
}
}
// coroutine 2
viewModelScope.launch {
// this shall not be executed before the subscription to the SharedFlow in coroutine 1 is set up
// to make sure I don't miss any emitted values
withContext(Dispatchers.IO) {
myService.initialize() // will send a value through the flow after initialization
}
}
}
How can I let coroutine 2 wait until the subscription to the SharedFlow in coroutine 1 is set up?
If you want to wait for a collector to subscribe on your flow before pushing values in it, I see two solutions:
If you have a MutableSharedFlow, you can use subscriptionCount property to know if your flow has been subscribed to.
As #broot said, there's also the possibility to use onSubscription on a SharedFlow. This does not require to expose mutable API. However, if there can be multiple collectors, and you want to ensure initialization is triggered only once, you will have to add some manual checks over it.
Here is a sample program that show how to use both solutions:
import kotlinx.coroutines.*
import kotlinx.coroutines.flow.*
import kotlin.time.Duration.Companion.seconds
fun main() : Unit = runBlocking {
val flow = MutableSharedFlow<Int>()
println(
"""
Solution 1: subscriptionCount
-----------------------------
""".trimIndent())
waitForSubsciptionCount(flow)
println(
"""
Solution 2: onSubscription
-----------------------------
""".trimIndent())
reactToSubscription(flow)
}
suspend fun CoroutineScope.waitForSubsciptionCount(flow: MutableSharedFlow<Int>) {
val collectJob = collectAfterDelay(flow)
println("Waiting for subscription")
flow.subscriptionCount.filter { it > 0 }.first()
println("Subscription detected")
flow.initialize()
// Specific to this test program, to avoid it to hang indefinitely
collectJob.cancel()
}
suspend fun CoroutineScope.reactToSubscription(flow: SharedFlow<Int>) {
println("Waiting for subscription")
// This is optional. Helps to detect "end" of the flow.
val initialized = MutableStateFlow(false)
val initializableFlow = flow.onSubscription {
println("Subscription detected")
initialize()
initialized.emit(true)
}
val collectJob = collectAfterDelay(initializableFlow)
// Specific to this test program, to avoid it to indefinitely
initialized.filter { it == true }.first()
collectJob.cancel()
}
private fun CoroutineScope.collectAfterDelay(flow: Flow<Int>) = launch {
delay(1.seconds)
print("Start collecting")
flow.collect { println("collected: $it") }
}
suspend fun FlowCollector<Int>.initialize() {
for (i in 1..3) {
emit(i)
println("Emitted $i")
}
}
The output is:
Solution 1: subscriptionCount
-----------------------------
Waiting for subscription
Start collectingSubscription detected
collected: 1
Emitted 1
collected: 2
Emitted 2
collected: 3
Emitted 3
Solution 2: onSubscription
-----------------------------
Waiting for subscription
Start collectingSubscription detected
collected: 1
Emitted 1
collected: 2
Emitted 2
collected: 3
Emitted 3
NOTE: Both solutions above should also allow you to move initialization trigger inside your service, and hide it completely from consumers. This way, code calling your service would not require to mangle with your service initialization.
EDIT: Here is an example of a service that triggers flow emission after a subscriber start collecting:
class SubscriptionCountService {
private val _flow = MutableSharedFlow<Int>()
public val flow = _flow.asSharedFlow()
init {
_flow.subscriptionCount.filter { it > 0 }
.take(1)
.onCompletion {
println("First subscription detected: initialize")
_flow.initialize()
}
.launchIn(CoroutineScope(Dispatchers.IO))
}
}
fun main() : Unit = runBlocking {
val flow = SubscriptionCountService().flow
launch {
delay(1.seconds)
println("Start collecting")
// Collect a limited number of entries to terminate main program easily
flow.take(3)
.collect { println("collected: $it") }
}
}
The output is:
Start collecting
collected: 1
Emitted 1
collected: 2
Emitted 2
collected: 3
Emitted 3

How to throttle Rx stream conditionally based on model field?

I had two classes Error and Alert, and a bunch of Rx streams of Error type.
For simplification let's agreed upon two streams.
private val exampleErrorStream1 = PublishSubject.create<Error>()
private val exampleErrorStream2 = PublishSubject.create<Error>()
The whole point is to map error streams to alert streams with according names:
private val exampleAlertStream1 = PublishSubject.create<Alert>()
private val exampleAlertStream2 = PublishSubject.create<Alert>()
Also, I declared a map where:
key is pair of those streams
value is mapper function for each stream transformation
private val errorToAlerts = mutableMapOf<Pair<Subject<Error>, Subject<Alert>>, (Error) -> Alert>(
Pair(exampleErrorStream1, exampleAlertStream1) to { Alert(it.message, 1)},
Pair(exampleErrorStream2, exampleAlertStream2) to { Alert(it.message, 2)}
)
Finally, I run the method once on the app start for mapping those streams:
fun mapErrorsToAlerts() {
errorToAlerts.forEach { (streams, toAlert) ->
val (errorStream, alertStream) = streams
errorStream.map(toAlert).throttleByPriority()
.doOnNext {
Log.d("Alert","New alert: $it")
}
.subscribe(alertStream)
}
}
The only problem I had is throttling alerts depending on one of the Alert fields.
The throttleByPriority() is an extension function:
private fun Observable<Alert>.throttleByPriority(): Observable<Alert> {
return this.flatMap {
val time = if (it.priority == 1) 5L else 10L
throttleFirst(time, TimeUnit.SECONDS)
}
}
The throttling is not working how I imagined it.
I have 10 seconds window but the same events are emitted one by one despite that.
I assumed that the problem is with flatMap where all previous Observables are kept alive, but I'm not sure of that and I don't know how to achieve this differently.

How do you make make a subscriber to a kotlin sharedflow run operations in parallel?

I have a connection to a Bluetooth device that emits data every 250ms
In my viewmodel I wish to subscribe to said data , run some suspending code (which takes approximatelly 1000ms to run) and then present the result.
the following is a simple example of what I'm trying to do
Repository:
class Repo() : CoroutineScope {
private val supervisor = SupervisorJob()
override val coroutineContext: CoroutineContext = supervisor + Dispatchers.Default
private val _dataFlow = MutableSharedFlow<Int>()
private var dataJob: Job? = null
val dataFlow: Flow<Int> = _dataFlow
init {
launch {
var counter = 0
while (true) {
counter++
Log.d("Repo", "emmitting $counter")
_dataFlow.emit(counter)
delay(250)
}
}
}
}
the viewmodel
class VM(app:Application):AndroidViewModel(app) {
private val _reading = MutableLiveData<String>()
val latestReading :LiveData<String>() = _reading
init {
viewModelScope.launch(Dispatchers.Main) {
repo.dataFlow
.map {
validateData() //this is where some validation happens it is very fast
}
.flowOn(Dispatchers.Default)
.forEach {
delay(1000) //this is to simulate the work that is done,
}
.flowOn(Dispatchers.IO)
.map {
transformData() //this will transform the data to be human readable
}
.flowOn(Dispatchers.Default)
.collect {
_reading.postValue(it)
}
}
}
}
as you can see, when data comes, first I validate it to make sure it is not corrupt (on Default dispatcher) then I perform some operation on it (saving and running a long algorithm that takes time on the IO dispatcher) then I change it so the application user can understand it (switching back to Default dispatcher) then I post it to mutable live data so if there is a subscriber from the ui layer they can see the current data (on the Main dispatcher)
I have two questions
a) If validateData fails how can I cancel the current emission and move on to the next one?
b) Is there a way for the dataFlow subscriber working on the viewModel to generate new threads so the delay parts can run in parallel?
the timeline right now looks like the first part, but I want it to run like the second one
Is there a way to do this?
I've tried using buffer() which as the documentation states "Buffers flow emissions via channel of a specified capacity and runs collector in a separate coroutine." but when I set it to BufferOverflow.SUSPEND I get the behaviour of the first part, and when I set it to BufferOverflow.DROP_OLDEST or BufferOverflow.DORP_LATEST I loose emissions
I have also tried using .conflate() like so:
repo.dataFlow
.conflate()
.map { ....
and even though the emissions start one after the other, the part with the delay still waits for the previous one to finish before starting the next one
when I use .flowOn(Dispatchers.Default) for that part , I loose emissions, and when I use .flowOn(Dispatchers.IO) or something like Executors.newFixedThreadPool(4).asCoroutineDispatcher() they always wait for the previous one to finish before starting a new one
Edit 2:
After about 3 hours of experiments this seems to work
viewModelScope.launch(Dispatchers.Default) {
repo.dataFlow
.map {
validateData(it)
}
.flowOn(Dispatchers.Default)
.map {
async {
delay(1000)
it
}
}
.flowOn(Dispatchers.IO) // NOTE (A)
.map {
val result = it.await()
transformData(result)
}
.flowOn(Dispatchers.Default)
.collect {
_readings.postValue(it)
}
}
however I still haven't figured out how to cancel the emission if validatedata fails
and for some reason it only works if I use Dispatchers.IO , Executors.newFixedThreadPool(20).asCoroutineDispatcher() and Dispatchers.Unconfined where I put note (A), Dispatchers.Main does not seem to work (which I expected) but Dispatchers.Default also does not seem to work and I don't know why
First question: Well you cannot recover from an exception in a sense of continuing
the collection of the flow, as per docs "Flow collection can complete with an exception when an emitter or code inside the operators throw an exception." therefore once an exception has been thrown the collection is completed (exceptionally) you can however handle the exception by either wrapping your collection inside try/catch block or using the catch() operator.
Second question: You cannot, while the producer (emitting side) can be made concurrent
by using the buffer() operator, collection is always sequential.
As per your diagram, you need fan out (one producer, many consumers), you cannot
achieve that with flows. Flows are cold, each time you collect from them, they start
emitting from the beginning.
Fan out can be achieved using channels, where you can have one coroutine producing
values and many coroutines that consume those values.
Edit: Oh you meant the validation failed not the function itself, in that case you can use the filter() operator.
The BroadcastChannel and ConflatedBroadcastChannel are getting deprecated. SharedFlow cannot help you in your use case, as they emit values in a broadcast fashion, meaning producer waits until all consumers consume each value before producing the next one. That is still sequential, you need parallelism. You can achieve it using the produce() channel builder.
A simple example:
val scope = CoroutineScope(Job() + Dispatchers.IO)
val producer: ReceiveChannel<Int> = scope.produce {
var counter = 0
val startTime = System.currentTimeMillis()
while (isActive) {
counter++
send(counter)
println("producer produced $counter at ${System.currentTimeMillis() - startTime} ms from the beginning")
delay(250)
}
}
val consumerOne = scope.launch {
val startTime = System.currentTimeMillis()
for (x in producer) {
println("consumerOne consumd $x at ${System.currentTimeMillis() - startTime}ms from the beginning.")
delay(1000)
}
}
val consumerTwo = scope.launch {
val startTime = System.currentTimeMillis()
for (x in producer) {
println("consumerTwo consumd $x at ${System.currentTimeMillis() - startTime}ms from the beginning.")
delay(1000)
}
}
val consumerThree = scope.launch {
val startTime = System.currentTimeMillis()
for (x in producer) {
println("consumerThree consumd $x at ${System.currentTimeMillis() - startTime}ms from the beginning.")
delay(1000)
}
}
Observe production and consumption times.

Testing RxJava repeatWhen with Mockk returnsMany

I'm trying to test multiple server responses with Mockk library. Something like I found in this answer for Mockito.
There is my sample UseCase code, which every few seconds repeats call to load the system from a remote server and when the remote system contains more users than local it stops running (onComplete is executed).
override fun execute(localSystem: System, delay: Long): Completable {
return cloudRepository.getSystem(localSystem.id)
.repeatWhen { repeatHandler -> // Repeat every [delay] seconds
repeatHandler.delay(params.delay, TimeUnit.SECONDS)
}
.takeUntil { // Repeat until remote count of users is greater than local count
return#takeUntil it.users.count() > localSystem.users.count()
}
.ignoreElements() // Ignore onNext() calls and wait for onComplete()/onError() call
}
To test this behavior I'm mocking the cloudRepository.getSystem() method with the Mockk library:
#Test
fun testListeningEnds() {
every { getSystem(TEST_SYSTEM_ID) } returnsMany listOf(
Single.just(testSystemGetResponse), // return the same amount of users as local system has
Single.just(testSystemGetResponse), // return the same amount of users as local system has
Single.just( // return the greater amount of users as local system has
testSystemGetResponse.copy(
owners = listOf(
TEST_USER,
TEST_USER.copy(id = UUID.randomUUID().toString())
)
)
)
)
useCase.execute(
localSystem = TEST_SYSTEM,
delay = 3L
)
.test()
.await()
.assertComplete()
}
As you can see I'm using the returnsMany Answer which should return a different value on every call.
The main problem is that returnsMany returns the same first value every time and .takeUntil {} never succeeds what means that onComplete() is never called for this Completable. How to make returnsMany return a different value on each call?
You probably don't understand how exactly .repeatWhen() works. You expect cloudRepository.getSystem(id) being called every time repetition is requested. That is not correct. Repeated subscription is done all the time on the same instance of mocked Single – first Single.just(testSystemGetResponse) in your case.
How to make sure, getSystem() is called every time? Wrap your Single into Single.defer(). It's similar to Single.fromCallable() but there is a difference between the return type of passed lambda. Lambda passed to the .defer() operator must return Rx type (Single in our case).
Final implementation (I have made a few changes to make it compile successfully):
data class User(val id: String)
data class System(val users: List<User>, val id: Long)
class CloudRepository {
fun getSystem(id: Long) = Single.just(System(mutableListOf(), id))
}
class SO63506574(
private val cloudRepository: CloudRepository
) {
fun execute(localSystem: System, delay: Long): Completable {
return Single.defer { cloudRepository.getSystem(localSystem.id) } // <-- defer
.repeatWhen { repeatHandler ->
repeatHandler.delay(delay, TimeUnit.SECONDS)
}
.takeUntil {
return#takeUntil it.users.count() > localSystem.users.count()
}
.ignoreElements()
}
}
And test (succeeds after ~8s):
class SO63506574Test {
#Test
fun testListeningEnds() {
val TEST_USER = User("UUID")
val TEST_SYSTEM = System(mutableListOf(), 10)
val repository = mockk<CloudRepository>()
val useCase = SO63506574(repository)
val testSystemGetResponse = System(mutableListOf(), 10)
every { repository.getSystem(10) } returnsMany listOf(
Single.just(testSystemGetResponse), // return the same amount of users as local system has
Single.just(testSystemGetResponse), // return the same amount of users as local system has
Single.just( // return the greater amount of users as local system has
testSystemGetResponse.copy(
users = listOf(
TEST_USER,
TEST_USER.copy(id = UUID.randomUUID().toString())
)
)
)
)
useCase.execute(
localSystem = TEST_SYSTEM,
delay = 3L
)
.test()
.await()
.assertComplete()
}
}

Caching items from observable until first subscription to deal with race condition

I am using Reactive Extensions (RxJava 2) to perform an RPC call to a Bluetooth device, resulting in an incoming data stream, which I subsequently parse, also uxing Rx. The resulting API is a simple Flowable<DownloadedRecord>. For this, I am building on top of the Rx API of the Sweetblue library for Android.
My problem is there is a race condition between 'requesting' the device to start streaming, and subscribing to the stream in time to make sure no packets are missed.
I use a Completable to first perform an RPC call to request data streaming to commence, andThen( readRecords ). A race condition seems to occur where some packets are emitted by Sweetblue, before readRecords had time to subscribe to this stream, thereby 'breaking' readRecords.
To abstract away from this concrete scenario, take the following stand alone code:
val numbers = PublishSubject.create<Int>()
var currentTotal = 0
val sumToTen = numbers
.doOnNext { currentTotal += it }
.doOnNext { println( "Produced $it" ) }
.takeUntil { currentTotal >= 10 }
.doOnComplete { println( "Produced a total of $currentTotal." ) }
Completable.fromAction { numbers.onNext( 9 ) } ) // Mimic race condition.
.andThen( sumToTen )
.subscribe { println( "Observed: $it, Current total: $currentTotal" ) }
numbers.onNext( 1 )
The numbers.onNext( 9 ) call mimics the race condition. This number is never observed by sumToTen, since sumToTen is only subscribed to on the next line. Thus, the stream never completes.
After some investigating, I understand I can 'solve' this problem by using replay and connect.
val numbers = PublishSubject.create<Int>()
var currentTotal = 0
val sumToTen = numbers
.doOnNext { currentTotal += it }
.doOnNext { println( "Produced $it" ) }
.takeUntil { currentTotal >= 10 }
.doOnComplete { println( "Produced a total of $currentTotal." ) }
.replay( 1 ) // Always replay last item upon subscription.
Completable.fromAction { sumToTen.connect() }
.andThen( Completable.fromAction { numbers.onNext( 9 ) } )
.andThen( sumToTen )
.subscribe { println( "Observed: $it, Current total: $currentTotal" ) }
numbers.onNext( 1 )
Now the sumToTen stream completes, since, by first connecting to sumToThen prior to 'starting to stream data' (onNext( 9 )), this stream subscribes to numbers, thus the intended side effects occur (currentTotal). But, '9' is only observed when the replay buffer is big enough (in this case it is). For example, replacing replay( 1 ) with publish will make the stream complete ("Produced a total of 10"), but will not observe '9'.
I am not fully satisfied with this solution for two reasons:
This simply minimizes the chance of the race condition occurring. How large to set the replay buffer is arbitrary.
This will always keep the specified number of elements in replay in memory, even though the intent is only to do so until subscribed to.
Practically speaking neither of these are a real problem, but this is an eye soar from a maintainability perspective: the code does not clearly communicate the intent.
Is there a better way to deal with this scenario? E.g.:
A replay operator which only replays for one subscriber (thus drops the cache once emitted for the first time).
An entirely different approach than what I explored here with publish/connect?

Categories

Resources