I am trying to implement an Android app which needs to obtain a big amount of data from a backend service and save it to a db to later work on it.
The below code describes the process:
itemsService
.getAllItemIds() //This returns Single<List<Int>> from backend
.subscribeOn(Schedulers.io())
.subscribe({
Observable.fromIterable(it)
.map({
itemsService
.getItemById(it) //This gets one item details from backend
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe({
//Add item details to db
}, {
//Some error
})
})
}, {
//Some error
})
I obtain a list of ids and then map each of these ids to a network call to obtain the full object.
This works for a test set of, say, 10 items, but the production set contains over 50 000 ids. It works initially, saving the items, but around 5-10% it grinds to a halt and the app dies.
I assume the reason here would be that Rx keeps the reference between the source and the mapped value.
My question is: is there a way to "pool" the source emissions to, let's say, 10 at a time? Or maybe there is some other mechanism I am not aware of?
You didn't mention what exactly "grinds to a halt" means, but it makes sense that you will get out of memory in real case of 50,000 items, cause you will basically try to create 50,000 threads at once to fetch each items details.
moreover, instead of chaining Observables using operators, you're creating nested chains at subscribe/map, you can read here why you shouldn't.
regarding limiting the work to 10 at a time, there is an flatMap overload for that, at the end it might look something like this:
itemsService
.getAllItemIds() //This returns List<Int> from backend
.flatMapIterable { t -> t }
.flatMap({
itemsService
.getItemById(it) //This gets one item details from backend
.subscribeOn(Schedulers.io())
}, 10) //limit flat map parallelism by desired value
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe({
//Add item details to db
}, {
//Some error
})
Related
So, I have an API call which returns a list of Dog breeds,
and another call that takes the dogBreedIds and fetches dog names for each of these breed IDs.
In onNext, I add these dog names into a list,
and in the doOnComplete, I display these in a recycler view.
Both getDogBreeds and getDogNames return an Observable<List> is an example of rx chain i accomplish this with:
petsRepository.getDogBreeds()
.map { breeds ->
breeds.items.map {
it.id
}
}
.flatMapIterable { listOfIds -> listOfIds }
.flatMap { dogId -> getDogNames(dogId) }
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.doOnComplete { displayDogNames() }
.subscribe(this::onDogNamesLoaded, this::onError)
What I'm trying to do is, to re-trigger this whole process once in every 60 seconds, and then, compare the response list with the first response I have and display it to the user if the response list has differences (this part is perhaps irrelevant)
I have tried to use Observable.interval(), however I could not succeed in doing this. I have checked plenty of examples with no success.
Please let me know if I should be providing more information on this.
Thanks!
You could go for interval like this:
Observable.interval(0, 60, TimeUnit.SECONDS)
.flatMap {
petsRepository.getDogBreeds()
}.map {...}
The rest should remain pretty much the same.
As for the displaying part - I'm going to assume you're using RecyclerView(correct me if not). If so, then use ListAdapter, DiffUtils will handle displaying differences in lists.
Hope this helps!
I am investigating the use of RxJava in my latest Android application.
I have a two lists of related updated Database model objects
ListDB1 and ListDB2
the logic I am attempting to implement is as follows
1). For each item in ListDB1
1.1). Transform it to a Network model object
1.2). Execute an Update RESTful API
2). Once all network updates have completed successfully
2.1). Persist ListDB1 to my local database.
2.2). Persist ListDB2 to my local database.
So far I have this code which should call my network API's
Observable.just(getDatabaseList())
.subscribeOn(Schedulers.io())
.flatMapIterable(x -> x)
.flatMap(database -> transformDatabase(database, DB_1_MAPPER))
.doOnNext(NetworkController::updateRemote)
.observeOn(AndroidSchedulers.mainThread())
.doOnComplete(getOnComplete())
.doOnError(getOnError())
.subscribe();
No API calls are executed though
I would rather use Single that Observable as my API calls respond with Single<Response>, however I couldnt see how to achieve Observable.just(<>) with a Single.
Also I cannot see how to commence my Rx process by processing each List item separately for the Network calls, then performing Database calls with a complete list as I am using Realm as my local database which can accept lists of database objects in a single update.
In pseudo code my process resembles:
for each database list item
convert to network model item
call remote update API
When all all network calls are successful
update database with ListDB1
update database with ListDB2
end
Is this possible in one Rx process "stream"?
1.
I would rather use Single that Observable as my API calls respond with Single, however I couldnt see how to achieve Observable.just(<>) with a Single.
You can do this:
Single.just(getDatabaseList()) // Single<>
However, Single is not suitable in this case because you are not working with a single item, and what you need instead is to iterate through multiple items and work on the items one by one.
2.
processing each List item separately for the Network calls, then performing Database calls with a complete list
You can use toList() operator which emits entire list of items when the observable completes.
3.
The purpose of do operators such as doOnNext, doOnComplete, and doOnError is to create side effect that does not affect the stream. An example of this kind of operations is logging. You should not do any meaningful operation that affect the stream in such operators.
Instead you should be using operators such as map, flatMap, etc.
4.
Putting everything together:
Observable.fromIterable(getDatabaseList())
.subscribeOn(Schedulers.io())
.flatMap(database -> transformDatabase(database, DB_1_MAPPER))
.flatMap(NetworkController::updateRemote)
.toList() // This has type of Single<List<>>
.flatMap(list -> {
// Update db1 and db2 with the result of server update.
return Single.zip(updateDb1(list), updateDb2(list), (a, b) -> {
// Combine result of db updates
});
})
.observeOn(AndroidSchedulers.mainThread())
.subscribe(result -> {
// Handle success case
}, err -> {
// Handle error case
});
My observable looks like following:
obs
.doOnNext { logger("Items changed (${it.size})") }
.distinctUntilChanged()
.doOnNext { logger("Items changed (${it.size})- EMIITTED") }
Log looks like following:
Items changed (7)
Items changed (7)- EMIITTED
Items changed (8)
// => missing EMIITTED message although it.size has changed => WHY?
Using the default comparator with a list of comparable items seems to fail here. Why? If the observables emitted list item size changed, the data is different, so distinctUntilChanged should not filter out the new list. But it seems like this happens here. Why?
Do I really need to provide my own comparator for distinctUntilChanged if I emit a list of items that compares the list size and the items one by one?
Edit
My obs basically looks like following:
obs = Observable.combineLatest(
RxDBDataManager.appsManager.observeList(),
RxDBDataManager.widgetsManager.observeList(),
RxDBDataManager.shortcutsManager.observeList(),
RxDBDataManager.customItemsManager.observeList(),
RxDBDataManager.foldersManager.observeList(),
Function5<List<IDBApp>, List<IDBWidget>, List<IDBShortcut>, List<IDBCustomItem>, List<IDBFolder>, List<IFolderOrSidebarItem>> { t1, t2, t3, t4, t5 ->
val list = ArrayList<IFolderOrSidebarItem>()
list.addAll(t1)
list.addAll(t2)
list.addAll(t3)
list.addAll(t4)
list.addAll(t5)
list
}
.flatMapSingle {
Observable.fromIterable(it)
.filter { it.parentType == parentType && it.parentId == parentId }
.cast(T::class.java)
.toList()
}
.flatMapSingle {
Observable.fromIterable(it)
.sorted(comparator)
.toList()
}
Additionally I apply some sorting and filtering on this data with
Based on the exchange in the comments:
RxJava users are encouraged to use immutable data types in its flows which prevents concurrency issues such as modifying the same object at different stages from different threads resulting in broken operator behavior and seemingly impossible business logic failures.
In this case, distinctUntilChanged didn't work as expected because mutable items were changed in a way that two subsequent onNext signals basically had the same content and the operator filtered them out as being non-distinct.
A way to detect if the items involved are in fact the same unintentionally is to use the bi-predicate version of the operator and then placing a breakpoint in the custom lambda. This lets one inspect the previous and current values and see if they are truly equal even if they shouldn't be:
source.distinctUntilChanged((prev, curr) -> {
// breakpoint the next line
return prev.equals(curr);
});
As in this case, broken behavior was due to a mutable item changed somewhere and thus evaluating as the same as the current/previous. With Lists, it is often not practical to breakpoint all mutation methods (such as add, addAll, set, remove etc.) but one can turn a mutable list into an immutable one and send it along the sequence. The built-in way is to convert it via the Collections::unmodifiableList:
source
.toList()
.map(Collections::unmodifiableList)
;
This will crash whenever a mutation is attempted on the now unmodifiable list instance, pointing to the logic that should be investigated further.
Here is my use case:
I am developing an app that communicates with a server via a REST API and stores the received data in a SQLite database (it's using it as a cache of some sorts).
When the user opens a screen, the following has to occur:
The data is loaded from the DB, if available.
The app call the API to refresh the data.
The result of the API call is persisted to the DB.
The data is reloaded from the DB when the data change notification is intercepted.
This is very similar to the case presented here, but there is a slight difference.
Since I am using SQLBrite, the DB observables don't terminate (because there is a ContentObserver registered there, that pushes new data down the stream), so methods like concat, merge, etc. won't work.
Currently, I have resolved this using the following approach:
Observable.create(subscriber -> {
dbObservable.subscribe(subscriber);
apiObservable
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.subscribe(
(data) -> {
try {
persistData(data);
} catch (Throwable t) {
Exceptions.throwOrReport(t, subscriber);
}
},
(throwable) -> {
Exceptions.throwOrReport(throwable, subscriber);
})
})
It seems like it's working OK, but it just doesn't seem elegant and "correct".
Can you suggest or point me to a resource that explains what's the best way to handle this situation?
The solution to your problem is actually super easy and clean if you change the way of thinking a bit. I am using the exact same data interaction (Retrofit + Sqlbrite) and this solution works perfectly.
What you have to do is to use two separate observable subscriptions, that take care of completely different processes.
Database -> View: This one is used to attach your View (Activity, Fragment or whatever displays your data) to the persisted data in db. You subscribe to it ONCE for created View.
dbObservable
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(data -> {
displayData(data);
}, throwable -> {
handleError(throwable);
});
API -> Database: The other one to fetch the data from api and persist it in the db. You subscribe to it every time you want to refresh your data in the database.
apiObservable
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.subscribe(data -> {
storeDataInDatabase(data);
}, throwable -> {
handleError(throwable);
});
EDIT:
You don't want to "transform" both observables into one, purely for the reason you've included in your question. Both observables act completely differently.
The observable from Retrofit acts like a Single. It does what it needs to do, and finishes (with onCompleted).
The observable from Sqlbrite is a typical Observable, it will emit something every time a specific table changes. Theoretically it should finish in the future.
Ofc you can work around that difference, but it would lead you far, far away from having a clean and easily readable code.
If you really, really need to expose a single observable, you can just hide the fact that you're actually subscribing to the observable from retrofit when subscribing to your database.
Wrap the Api subscription in a method:
public void fetchRemoteData() {
apiObservable
.subscribeOn(Schedulers.io())
.observeOn(Schedulers.io())
.subscribe(data -> {
persistData(data);
}, throwable -> {
handleError(throwable);
});
}
fetchRemoteData on subscription
dbObservable
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.doOnSubscribe(() -> fetchRemoteData())
.subscribe(data -> {
displayData(data);
}, throwable -> {
handleError(throwable);
});
I suggest you really think about all that. Because the fact that you're forcing yourself into the position where you need a single observable, might be restricting you quite badly. I believe that this will be the exact thing that will force you to change your concept in the future, instead of protecting you from the change itself.
I have the following code to fetch a list of items from the internet.
Observable<RealmList<Artist>> popArtists = restInterface.getArtists();
compositeSubscription.add(popArtists.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread()).subscribe(artistsObserver));
The trouble is the list has over 80 items and i only want to get the first 5 items. What is the best way to achieve this?
takeis the operator you looking for. (see documentation here : http://reactivex.io/documentation/operators/take.html )
The flatMapIterable transform your RealmList (which implement Iterable, that's why flatMapIterable can be used) to an Observable which emiting all items of your list
Subscription subscription = restInterface.getArtists()
.flatMapIterable(l -> l)
.take(5)
.subscribeOn(Schedulers.io())
.observeOn(androidSchedulers.mainThread())
.subscribe(artistsObserver);
compositeSubscription.add(subscription);
I guess you have no control over server side, so solution is to take first 5 items from received result:
Observable<RealmList<Artist>> popArtists = restInterface.getArtists();
compositeSubscription.add(
popArtists.flatMap(list-> Observable.from(list).limit(5)).subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(artistsObserver));