I've witnessed a weird behavior with onBackpressureBuffer, I'm not sure if it is a valid behavior or a bug.
I'm having a tcp call that is emitting items in a certain rate (using streaming and inputStream but that just for some info)
On top of it I've created an observable using create that will emit an item each time it is ready.
Let's call it messages().
Then I'm doing this:
messages()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe({//do some work});
I've noticed using analytics tools that MissingBackPressureException is thrown rarely, so I've added onBackpressureBuffer to the call.
If I'm adding it after observeOn:
messages()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.onBackpressureBuffer()
.subscribe({//do some work})
everyting works fine, but it means it will buffer only after it get's to the UI Main thread, so I prefered it to be like this:
messages()
.onBackpressureBuffer()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe({//do some work});
And that where things start to get weird.
I've noticed the while messages() keeps emitting item, at some point they will stop being delivered to the subscriber.
More precisely after exactly 16 items, what is apparently happing is that the buffer will start holding items without passing them forward.
Once I cancel the messages() with some sort of timeout mechanism, it will cause messages() to emit onError() and the buffer will emit immediately all the items it has kept (they will be handled).
I've checked to see if it is the subscriber fault of doing too much work but it not, he is finished and still he doesn't get the items...
I've also tried using the request(n) method in the subscriber asking for one item after onNext() is finished but the buffer doesn't behave.
I suspect that the messaging system of the Main Android UI Thread with the backpressure causing this, but I can't explain why.
can someone explain why this is happening? is this a bug or a valid behaviour?
Tnx!
Not knowing how messages(), based on the behavior described, this is a similar same-pool deadlock as with this question
The workaround, what you didn't try, is to put the .onBackpressureBuffer between the subscribeOn and observeOn.
messages()
.subscribeOn(Schedulers.io())
.onBackpressureBuffer() // <---------------------
.observeOn(AndroidSchedulers.mainThread())
.subscribe({//do some work});
Question is different but answer comes down to the same thing.
The implementation of observeOn constructor: OperatorObserveOn(Scheduler scheduler, boolean delayError, int bufferSize):
public OperatorObserveOn(Scheduler scheduler, boolean delayError, int bufferSize) {
this.scheduler = scheduler;
this.delayError = delayError;
this.bufferSize = (bufferSize > 0) ? bufferSize : RxRingBuffer.SIZE;
}
The last line points to the buffer size.
The buffer size on Android is 16.
The solution is simply passing a bigger buffer size to observeOn(Scheduler scheduler, int bufferSize) operator:
messages()
.observeOn(AndroidSchedulers.mainThread(), {buffer_size})
Be careful not to put too high value, as Android has limited memory.
Related
I'm trying for some time now to implement an extension function (just becuse it's easier to me) that is capable of delaying both normal item emissions and errors. The existing delay operators only delays normal item emissions, errors are delivered ASAP.
For context, I'm trying to immitate an Android LiveData's behavior (kinda). LiveDatas are a observable pattern implementation that is lifecycle aware. Their observers are only notified if they are in a state where they can process that emission. If they are not ready, the emission is cached in the livedata and delivered as soon as they become ready.
I created a BehaviourSubject that emits the state of my Activities and Fragments when it changes. With that I created a delay operator like this:
fun <T> Flowable<T>.delayUntilActive(): Flowable<T> = delay { lifecycleSubject.toFlowable(BackpressureStrategy.LATEST).filter { it.isActive } }
and then use it like this
myUseCase.getFlowable(Unit)
.map { it.map { it.toDisplayModel() } }
.delayUntilActive()
.subscribe({
view.displaySomethings(
}, { }).addTo(disposables)
So even if myUseCase emits when the view is not ready to display somethings, the emission won't reach onNext() until the view does become ready. The problem is that I also want the view to displayError() when onError is triggered, but that too is lifecycle sensitive. If the view isn't ready, the app will crash.
So I'm looking for a way to delay both emissions and errors (onComplete would be good too). Is this possible?
I tried some things with zip, onErrorReturn, delay inside delay, but nothing seemed right. I'd be equally unimpressed if this had a really easy solution I'm overlooking, or is impossible. Any ideas are welcome.
Bonus: any better way to do that for Single and Completable too? currently I'm just converting them to flowable.
Thanks in advance!
You can handle the error via onErrorResumeNext, then taking the same error and delaying it via delaySubscription until your desired signal to emit said error happens:
source
.onErrorResumeNext({ error ->
Observable.error(error)
.delaySubscription(lifecycleSubject.filter { it.Active } )
})
I am writing a sample app, that processes the bitmap. The process can be controlled by a slider, so when the slider position is changed, I generate another bitmap.
When the user drags the slider, it emits some 10-20 events per second. Processing the bitmap takes about 1 second, so the processing queue becomes quickly stuck with requests.
It seems like a good backpressure example to me, but I couldn't figure out how to use stuff like Flowable and BackpressureStrategy to handle it properly. Moreover, I couldn't make this small sample work:
val pubsub = PublishSubject.create<Int>()
pubsub
.toFlowable(BackpressureStrategy.LATEST)
.observeOn(computation())
.subscribe {
Timber.d("consume %d - %s", it, Thread.currentThread().name)
Thread.sleep(3000)
}
for (i in 0 .. 1000) {
Timber.d("emit %d - %s", i, Thread.currentThread().name)
pubsub.onNext(i)
}
Well, I expect this code to emit 1000 integers through PublishSubject, but as long as processing each takes 3 seconds, 999 of integers should be dropped, only "0" and "1000" should be processed...
But in the logs I see, that all my integers are slowly processed, one by one, and the backpressure strategy is ignored. Actually, toFlowable(...) expression seems to do nothing. With or without backpressure, I see 1000 emissions followed by the several minutes of consumption.
What am I missing here? How can I drop the intermediate elements and consume only the latest available?
solved:
observeOn(computation()) is actually observeOn(computation(), delayErrors = false, bufferSize = 128). To see real backpressure, decrease the bufferSize, when you call observeOn(...)
This might be related to observeOn(computation()). Depending on the backing thread, this might be throttled automatically. The emission of the items is queued. Therefore there's no backpressure on the Flowable.
Try putting these thread changes before toFlowable(LATEST) or use a different Scheduler which is not as forgiving or put even more items to pubsub.
Also you could use observeOn(Scheduler scheduler, boolean, int) to enforce a bufferSize.
While subscribing to a Reactive Extensions Flowable stream, I noticed the stream halts/hangs (no more future items are emitted, and no error is returned) after 128 items have been returned.
val download: Flowable<DownloadedRecord> = sensor.downloadRecords()
download
.doOnComplete { Log.i( "TEST", "Finished!" ) }
.subscribe(
{ record ->
Log.i( "TEST", "Got record: ${record.record.id}; left: ${record.recordsLeft}" )
},
{ error ->
Log.i( "TEST", "Error while downloading records: $error" )
} )
Most likely, this is related to Reactive Extensions. I discovered the default buffer size of Flowable is set to 128; unlikely to be a coincidence.
While trying to understand what is happening, I ran into the following documentation on Flowable.subscribeOn.
If there is a create(FlowableOnSubscribe, BackpressureStrategy) type source up in the chain, it is recommended to have requestOn false to avoid same-pool deadlock because requests may pile up behind an eager/blocking emitter.
Although I do not quite understand what a same-pool deadlock is in this situation, it looks like something similar is happening to my stream.
1. What is a same-pool deadlock in Reactive Extensions? What would be a minimal code sample to recreate it (on Android)?
Currently at a loss, I tried applying .subscribeOn( Schedulers.io(), false ) before .subscribe, without really understanding what this does, but my stream still locks up after 128 items have been emitted.
2. How could I go about debugging this issue, and how/where can it be resolved?
What is a same-pool deadlock in Reactive Extensions?
RxJava uses single threaded executors in the standard schedulers. When a blocking or eager source is emitting items, it occupies this single thread and even though the downstream requests more, subscribeOn will schedule those requests behind the currently running/blocking code that then never gets notified about the new opportunities.
What would be a minimal code sample to recreate it (on Android)?
Why would you want code that deadlocks?
I tried applying .subscribeOn( Schedulers.io(), false )
What is your actual flow? You likely applied subscribeOn too far from the source and thus it has no effect. The most reliable is to put it right next to create.
How could I go about debugging this issue, and how/where can it be resolved?
Putting doOnNext and doOnRequest at various places and see where signals disappear.
I tried to make a shorter code of what I have , but it seems not enough to make it readable so I ended up with the following:
Flowable
.defer (return new outer observable upon subscribing)
.retryWhen ( ->
Flowable.flatMap throwable when recieve a 400
valve.onNext(false)
-> Flowable.defer ( return new Network_A_Observable)
.retryWhen ( ->
Flowable.flatmap throwable when receive 500)
valve.onNext(true)
)
.compose(flowable valve (i intentionally put a false here))
.subscribe(new subscriber)
This is the very short version of my long-non-lambda code that performs a series of network calls and retrying appropriately on certain conditions.
I have no problems with retrying and emitting a new outer observable for each retry(as this was solved on my other post, but not sure if it has something to do with my current issue), now I noticed that when I perform, say, 2 asynchrounous network calls, it returns two different values, although those two values are valid, and ofcourse the stream of observable works as expected and that was what I intended to do(retry when some error happens), now I realize that I should "PAUSE" the next stream of calls, searching like "how to pause an observable, until I found FlowableTransformer.valve(), which is available on RxJava2Extensions, I ran a code snippet from a blog where it pauses/continues a stream, but when I tried it on the above code, event if the valve is set to false, it keeps on finishing the whole stream of flowable.
Am i missing something?
Any help would be greatly appreciated.
I'm feeling a bit curious about how .subscribeOn() actually works on RxJava.
I have this piece of code that works as intended:
return endpoints.getRecentConversations(page)
.map().flatMap().doOnNext() //etc etc...
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
I was reading this article, trying to understand what the difference between subscribeOn and observeOn when this line caught my eye:
Position does not matter
subscribeOn can be put in any place in the stream because it affects
only the time of subscription
Which sounds perfectly fine. But I was feeling a bit skeptical I decided to test it. So I changed the code above (switched lines 2 and 3):
return endpoints.getRecentConversations(page)
.subscribeOn(Schedulers.io())
.map().flatMap().doOnNext() //etc etc...
.observeOn(AndroidSchedulers.mainThread())
As a result, I get an premature onComplete() on my subscriber. onNext() is never called and no errors are present in my logcat.
I can leave things the way they were, but I'd like to know why this is happening. Is it true that position doesn't matter? Is it something wrong with my code? Here's how my code looks
Yes, the position does matter a lot in RxJava, It's called upstream & downstream.
It's because you are subscribing to a hot observable
Watch this video to understand it better: Common RxJava Mistakes
https://www.youtube.com/watch?v=QdmkXL7XikQ&t=768s
There are two types of observables: Hot & cold.
A “hot” Observable may begin emitting items as soon as it is created,
and so any observer who later subscribes to that Observable may start
observing the sequence somewhere in the middle. A “cold” Observable,
on the other hand waits until an observer subscribes to it before it
begins to emit items, and so such an observer is guaranteed to see the
whole sequence from the beginning.