Hi I'm using RxJava for my disk storage get and set operations. Basically I have a method like this:
public Observable<String> getStorageItem(String id, String type) {
return Observable.defer(new Func0<Observable<String>>() {
// Run db operations to get storage item.
}
}
The problem is that it's possible this method getStorageItem(...) gets subscribed to multiple times in a row. And the DB operations within the observable cannot run concurrently. What's my best option here? Should I manually create some sort've queue? Or does RxJava have some kind of tool that allows me to block the operation until a previous one is complete?
You can use a subscribeOn with a single-threaded scheduler created from an ExecutorService to make sure there's only one DB operation in progress:
ExecutorService exec = Schedulers.newSingleThreadExecutor();
Scheduler s = Schedulers.from(exec);
public Observable<String> getStorageItem(String id, String type) {
return Observable.fromCallable(() -> {
// Do DB operations
});
}
getStorageItem("1", "2").subscribeOn(s).subscribe(...);
getStorageItem("2", "4").subscribeOn(s).subscribe(...);
getStorageItem("3", "6").subscribeOn(s).subscribe(...);
But note that by moving the computation off the caller's thread, it may execute any time. If you need to wait for it individually (because the getStorageItem is already called on some thread), you can apply toBlocking() after subscribeOn.
Related
I have some expensive operations that only need to be performed once (e.g. load/ download large files, load large ML models, or calculate optimized data structure based on some other data). I want to use this for every value the Observable/ Flowable generates:
The following code works, but it runs heavyProcessing() and heavyProcessing2() on the caller's thread. In my case, I can't choose what my callers thread (its the main thread because I am using WorkManager's RxWorker, which calls createWork from main). Therefore, start blocks the main thread. How do I get heavyProcessing to be performed in the background with RxJava and also available to the subsequent RxJava chain?
fun start(): Observable<Unit> {
val heavy = heavyProcessing() // the heavy value i want to use everywhere!
val anotherHeavyObject = heavyProcessing2()
val items = Observable.fromIterable(listOfHundredsOfItems)
.map { doSomeWork(it, heavy) }
.map { doSomeWork(it, anotherHeavyObject) }
}
My attempts has so far not worked:
Create a wrapper around the existing function: The issue with this code is the Observable returned by start() does not get observed, so the doSomeWork doesn't actually get done. I only know this because I put breakpoints in at doSomeWork, and it never gets called.
fun startInBackground(): Single<Unit> {
return Single.fromCallable {
start()
}
}
I've been trying to find ways of 'unnesting' the inner Observable (inside the Single), as that's probably the issue here. The inner Observable is not being observed.
This RxJava stuff is very unintuitive even after reading the guide
Yes, it was related to Deferred-dependent. The example in the docs state:
Sometimes, there is an implicit data dependency between the previous sequence and the new sequence that, for some reason, was not flowing through the "regular channels". One would be inclined to write such continuations as follows:
AtomicInteger count = new AtomicInteger();
Observable.range(1, 10)
.doOnNext(ignored -> count.incrementAndGet())
.ignoreElements()
.andThen(Single.defer(() -> Single.just(count.get())))
.subscribe(System.out::println);
Actually, all I needed the caller to do is:
Single.defer { start() }.map { doMoreWork() }
instead of
start().map { doMoreWork() }
Let me simplify my example. Imagine I have two buttons.
btn_A.setOnClickListener(v -> {
// update ROOM DB value to A
});
btn_B.setOnClickListener(v -> {
// update ROOM DB value to B
});
Button A will set DB value to 'A'.
Button B will set DB value to 'B'.
Since I cannot access DB from the Main/UI thread, I execute it from the new thread:
btn_A.setOnClickListener(v -> {
new Thread(() -> {
// update ROOM DB value to A
}).start();
});
btn_B.setOnClickListener(v -> {
new Thread(() -> {
// update ROOM DB value to B
}).start();
});
If I press Button A, B, A, B..., then I am worried that new threads will not execute consecutively(in sequence).
I thought about using ExecutorService with Executors.newFixedThreadPool(1) but I am not sure if this will run the worker thread in sequence. Also the DB modification commends are called from different Activities/Service so I should make the ExecutorService static. I am worried if I make the ExecutorService static and don't call 'shutdown()', then it may result in memory leak.
I want to avoid implementing my own producer/consumer design pattern with the queue logic.
It feels like there is a simple way to do it but I can't figure it out. :(
You need to run all updates on a single thread. One way is to use Looper and Handler in order to send messages to the database update thread and it executes those messages in the order received. At a higher level, you can use an event bus library like Otto to help facilitate this inter-thread communication.
Brief introduction for those not familiar with Android and/or Firebase development:
In Android development, you should always manipulate your application's views from the main thread (also called UI thread), but if your application needs to make some heavy processing, it should use a background thread, otherwise the app would seem unresponsive.
Firebase is a service that offers a way to store and sync data with a NoSQL database in the cloud. It also offers an Android SDK to manage this database. Every time this SDK is used to make an operation, like a query, Firebase avoids those threading pitfalls by making all of its heavy processing on its own internal background thread and by always calling its callbacks on the main thread.
Example:
Query postsQuery = FirebaseDatabase.getInstance().getReference("posts");
ValueEventListener postListener = new ValueEventListener() {
#Override
public void onDataChange(DataSnapshot dataSnapshot) {
// This is always called on the main thread
// Get Post object and use the values to update the UI
Post post = dataSnapshot.getValue(Post.class);
// ...
}
#Override
public void onCancelled(DatabaseError databaseError) {
// Getting Post failed, log a message
printError(databaseError.toException());
// ...
}
};
postsQuery.addValueEventListener(postListener);
The actual issue I'm facing:
I'm trying to wrap Firebase's query listeners with RxJava using a method like this:
private static Observable<DataSnapshot> queryObservable(final Query query) {
return Observable.fromEmitter(emitter -> {
// This is called on the Scheduler's thread defined with .subscribeOn()
printThread("emitter");
final ValueEventListener listener = new ValueEventListener() {
#Override public void onDataChange(final DataSnapshot dataSnapshot) {
// This is always called on the main thread
printThread("onDataChange");
emitter.onNext(dataSnapshot);
}
#Override public void onCancelled(final DatabaseError databaseError) {
// This is called on the main thread too
emitter.onError(databaseError.toException());
}
};
query.addValueEventListener(listener);
emitter.setCancellation(() -> query.removeEventListener(listener));
}, Emitter.BackpressureMode.BUFFER);
}
But because the Observable is emitting items from inside the Firebase's callback (called on the main thread) any further .subscribeOn() operators are going to be ignored.
For example, calling the above method like this:
Query postsQuery = FirebaseDatabase.getInstance().getReference("posts");
queryObservable(postsQuery).doOnSubscribe(() -> printThread("onSubscribe"))
.subscribeOn(Schedulers.io())
.subscribe(dataSnapshot -> printThread("onNext"));
Would print the following:
onSubscribe Thread: RxIoScheduler-2
emitter Thread: RxIoScheduler-2
onDataChange Thread: main
onNext Thread: main
From what I understand, when Firebase's SDK calls the onDataChange() callback and switches from its own internal background thread to the main thread, it also makes the Observable emit new items on the main thread, rendering useless any .subscribeOn() operator down the stream.
The actual question:
What can I do to not only correctly wrap listeners like this into an Observable but also make them conform to the Scheduler defined by .subscribeOn()?
Thank you!
Update:
I know .observeOn() gives me the ability to process the data returned by Firebase on another thread. That's what I'm doing already, but it just isn't the point of this question. The point is: when I pass a Scheduler through .subscribeOn() I expect the upstream to conform to that Scheduler's thread but that doesn't happen when the Observable has an internal listener that is being triggered from a callback on a different thread. When that happens, I lose the .subscribeOn() guarantee.
The severity of this issue may not seem obvious at first, but what if that Observable was part of a library? What's the best practice there? Should the library enforce its clients to always call an .observeOn() after any call to that method? Should the library call an .observeOn() itself and call it a "default Scheduler"? In any of these cases the .subscribeOn() is just useless, and that doesn't seem right to me.
Just use observeOn in IO and subscribeOn in Main Thread, in that way you can manage your recieved that in MainThread and move the firebase work to a different Thread.
Remember to import rxAndroid to your gradle(Rxjava or RxJava 2):
compile 'io.reactivex.rxjava2:rxandroid:2.0.1'
Also suggest you to check as reference(or just use it) one of the next libraries:
RxJava : https://github.com/nmoskalenko/RxFirebase
RxJava 2.0: https://github.com/FrangSierra/Rx2Firebase
One of them works with RxJava and the other one with the new RC of RxJava 2.0. If you are interested of it, you can see the differences between both here.
I Had the same problem, finally I combined with Coroutines to run the listener on background.
To do that, simple add coroutine background work in your onDataChange
Kind regards
I pretty much understand the concept of subscribe (any code below subscribeOn will be performed in that particular thread) and observe (same with subscribeOn) in rxandroid/rxjava.
What I want to happen is to perform long io operation in background thread then notify the main thread if the operations is finished. To do that, I'm thinking of having a flatmap which is subscribed in Schedulers.io() then observe a subscribe in AndroidSchedulers.mainThread(), something like this:
Observable.just(1)
.subscribeOn(Schedulers.io())
.flatMap(o -> {
longIO();
return null;})
.observeOn(AndroidSchedulers.mainThread())
.subscribe(//i want to notify user here);
This is actually performing the longIO() in a different thread, thus not blocking the main thread, my problem is, this doesn't notify the main thread that longIO() is finished, note that android doesn't allow notifying user by creating Toast or AlertDialog if not in main thread. The code doesn't seem to pass through subscribe
Note: I used just(1) even though I don't use the integer 1 because I want the method inside flatMap to be performed. If I used empty it won't go through flatMap
The return type of flatMap is Observable. If the flatMap returns a null Observable, the subscriber won't get notified. Change the return statement to return Observable.just(null);
But, it's preferred to use Observable.fromCallable() to wrap your longIO() method, so just(1) would be obsolete and code looks cleaner. Note: the return type offromCallable() isn't Observable, so the subscriber would get notified even null is returned. It would look like:
Observable.fromCallable(() -> {
longIO;
return null;
})
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(
I think that you are wrong in few things. IMO everything ABOVE subscribeOn() will be done in specific thread from thread pool. And of course everything BELOW observeOn should be pass into UI Thread.
Second thing - You cannot perform that your flatMap operator is returning null. You need to return Observable. If you don't need to pass data you can use : Observable.just(null) or Observable.never().
I think that better solution would be:
Observable.create(new Observable.OnSubscribe<Object>() {
#Override
public void call(final Subscriber<? super Object> subscriber) {
longIO();
}
})
.startWith(new Object()) //if you want to run it once
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe();
I am working on an Android application that uses greenDAO as a data persistence layer. The application downloads data from various different sources across multiple threads (determined by a thread pool), each piece of data is inserted into the database in a transaction using insertOrReplaceInTx. This is working fine.
My question is whether it is technically possible, using greenDAO, to encapsulate these different transactions (which occur on different threads) into an overall transaction, using nested transactions. I know in theory it is possible to do this if all the transactions were taking place on a single thread, however I am unsure if this possible with the insertOrReplaceInTx calls occurring on different threads.
The reason I wish to encapsulate these into a single overall transaction is because they represent a synchronisation process within an app. In the event of any single part of the import failing, I wish to abort and rollback all of the modifications within the overall transaction.
If I begin a transaction with db.beginTransaction on the main thread where I initiate the import process, this creates a deadlock when another thread tries to insertOrReplaceInTxt.
Is the correct way to counter this to ensure that all greenDAO transactions are taking place on the same thread?
Afaik, you cannot because each thread manages its own connection.
If you have such dependency between these operations, you probably want to sync them anyways.
e.g. what if Job A finishes way before Job B and Job B's db connection fails. Your data will go out of sync again. You still need some logic for the other job.
Also, writers are mutually exclusive.
I would suggest creating a utility class that can run a list of runnables in a transaction. Each job, when finished, enqueues a Runnable to this utility. These runnables include the actual database commands.
When the last one arrives (this depends on your dependency logic), the utility will run all runnables in a transaction.
A sample implementation may look like this: (I used a simple counter but you may need a more complex logic)
class DbBundle {
AtomicInteger mLatch;
List<Runnable> mRunnables = new ArrayList();
DbBundle(int numberOfTx) {
mLatch = new AtomicInteger(numberOfTx);
}
void cancel() {
mLatch.set(-1); // so decrement can never reach 0 in submit
}
boolean isCanceled() {
mLatch.count() < 0;
}
void submit(Runnable runnable) {
mRunnables.add(runnable);
if (mLatch.decrementAndGet() == 0) {
db.beginTransaction();
try {
for (Runnable r : mRunnables) r.run();
db.setTransactionSuccessful()
} finally {
db.endTransaction();
}
}
}
}
When you create each job, you pass this shared DbBundle and the last one will execute them all.
So a job would look like:
....
try {
if (!dbBundle.isCanceled()) { // avoid extra request if it is already canceled
final List<User> users = webservice.getUsers();
dbBundle.submit(new Runnable() {
void onRun() {
saveUsers(users);//which calls db. no transaction code.
});
});
} catch(Throwable t) {
dbBundle.cancel();
}