I am trying to figure out what is the best way to open/close Realm instances. We are using Realm in 3 places - App, Activities and Worker thread. However, it might be pretty cool to access data from Realm from any place. Thus, we wrapped all our Realm queries/writes in a Data Layer. Is it ok (from performance point of view) to do the following for every query (and similar on each write)?
try (Realm realm = getRealm()) {
return realm.copyFromRealm(realm.where(Task.class).isNull("name")
.findAllSorted("createdAt", Sort.DESCENDING));
}
How do you do something similar in async transaction? Are there better approaches? (We are using RxJava).
Thank you!
EDIT: After quick response from #christian-melchior we will use it like this:
try (Realm realm = getRealm()) {
return realm.copyFromRealm(realm.where(Task.class).isNull("name")
.findAllSortedAsync("createdAt", Sort.DESCENDING));
}
EDIT2: We actually settled on this:
Realm realm = getRealm();
return realm.where(Task.class).isNull("name")
.findAllSortedAsync("createdAt", Sort.DESCENDING)
.asObservable()
.filter(RealmResults::isLoaded)
.first()
.map(realmObjects -> {
if (realmObjects.isEmpty()) {
realm.close();
return null;
}
List<Task> res = realm.copyFromRealm(realmObjects);
realm.close();
return res;
});
Since we want our onComplete to be called after each query.
Using copyFromRealm will give you an in-memory copy of all the data, so as long as you are not loading huge lists, it should work fine. You will use more memory though, and there is a slight overhead in opening and closing Realms all the time.
One idea for controlling the Realm lifecycle can be seen here: https://realm.io/docs/java/latest/#controlling-the-lifecycle-of-realm-instances
Also note you in many cases can use our async API if you are concerned about loading data on the UI thread: https://realm.io/docs/java/latest/#asynchronous-queries
Related
I'm trying to implement a simple chat application on web sockets in Clean Architecture. I had to choose a db for caching all information, so I decided to use Realm, because I heard it was pretty good database for any kind of mobile applications. But when I actually faced the Realm, it turned out to be really painful experience for me to implement caching logic with it.
All problems come from applying transaction to database which then must be synced on all threads working with Realm. There seems to some kind of synchronization problem with my code. For example, I want to save my object to Realm and then query it out of.
Here I have two simple functions to save and to get chat:
fun getBackgroundLooper(): Looper {
val handlerThread = HandlerThread("backgroundThread")
if (!handlerThread.isAlive)
handlerThread.start()
return handlerThread.looper
}
fun saveChat(chat: Chat): Completable {
val realmChat = ChatMapper.domainToCache(chat)
return Completable.create { e ->
val realm = Realm.getDefaultInstance()
realm.executeTransactionAsync({
it.insertOrUpdate(realmChat)
}, {
realm.close()
e.onComplete()
}, {
realm.close()
e.onError(it)
})
// Subscribe on background looper thread
// to be able to execute async transaction
}.subscribeOn(AndroidSchedulers.from(getBackgroundLooper()))
}
fun getSingleChat(chatId: String): Single<Chat> {
return Single.defer {
val realm = Realm.getDefaultInstance()
realm.isAutoRefresh = true
val realmChat = realm.where(RealmChat::class.java)
.equalTo("id", chatId).findFirstAsync()
if (realmChat.isValid) {
realmChat.load()
val chat = ChatMapper.cacheToDomain(realmChat)
realm.close()
Single.just(chat)
}
realm.close()
Single.error<Chat>(ChatNotExistException())
// Subscribe on background looper thread
// to be able to execute auto refreshing
}.subscribeOn(AndroidSchedulers.from(getBackgroundLooper()))
}
So, when I try to run simple code like this
remote.getChat().flatMap {
cache.saveChat(it) //save chat to realm
.andThen(cache.getSingleChat(it.id)) //then query it by id
}
I always get no matter of what ChatNotExistException, but if I try to run query again in another attempt or after restarting the application, then the chat object gets found
I also tried many different approaches to execute this code:
I tried to use realm.refresh() in getSingleChat or not use it at all.
I tried to query chat synchronously with findFirst() and findAll() instead of findFirstAsync().
I tried querying chat on current thread without .subscribeOn().
I tried to use realm.executeTransaction() instead of async transactions.
I tried to add thread sleep between saving and querying, so that transaction may take some time to get applied and I need to wait before attempting to query the chat
I'm begging anybody to explain me what am I doing wrong and how to make this code working. I can't change the architecture of my application and use Realm objects as my view models, I need to find solution in these conditions.
But when I actually faced the Realm, it turned out to be really painful experience for me to implement caching logic with it.
Reading the docs regarding best practices help. For example, the default idea is that you define a RealmResults using an async query on the UI thread, add a change listener to it, and observe the latest emission of the database.
There is no "caching" involved in that beyond saving to the database and observing the database. Any additional complexity is added by you and is completely optional.
All problems come from applying transaction to database which then must be synced on all threads working with Realm.
All looper threads automatically make the Realm auto-refresh, therefore if addChangeListener is used as intended in the docs, then there is no need for trickery, Realm will manage the synchronization between threads.
I want to save my object to Realm and then query it out of.
realm.executeTransactionAsync({
No reason to use executeTransactionAsync when you are already on a background thread.
try(Realm realm = Realm.getDefaultInstance()) {
realm.executeTransaction((r) -> {
// do write here
});
}
realm.where(RealmChat::class.java)
If you do import io.realm.kotlin.where, then you can do realm.where<RealmChat>().
.findFirstAsync()
No reason to use findFirstAsync() instead of findFirst() when you are already on a background thread. Also no reason to use load() when you're on a background thread, because you should be using findFirst() in the first place anyway.
You are also most likely missing a return#defer Single.just(chat) to actually return the chat if it's found. That is most likely what your original problem is.
With the handler thread things you're doing though, you might want to consider taking a look at this project called "Monarchy", as it intends to set up the ability to run queries on a background looper thread while still observing the results. It is labelled stagnant but the ideas are sound.
I am using RxJava in my Android project and I'm happy about it. I'm currently using it to make all my DAO methods asynchronous and make UI listens on them.
But I have a big problem, that is, when I retrieve some data from database using Observable<List<User>> getLists(), I need to use List<User> in my ViewModels, but I cannot extract it from the observable.
I would like to know what is the common approach to solve this kind of problem ? I searched on Internet and people said it's not recommended to extract the objects, but in this case how can I use the data from database and at the same time still enable the observers listening ?
Should I create another method using AsyncTask ??
Thanks.
In my UserRepo.java
public Observable<List<User>> getUsers() {
return colisDao.getUsers();
}
In HomeScreenViewModel.java:
public List<User> getUsers() {
return userRepo.getUsers(); // do not work because I need a List<User>
}
In HomeActivity.java:
UserListAdapter userListAdapter = new UserListAdapter(this,
vm.getUsers());
Central idea of reactive extensions is to make use of events' streams observation and timely processing.
So actually, if you need to retrieve data in a straightforward way, I'd say you don't need RxJava2 at all. Still, if you want to stick to the reactive approach, the data stream should be listened to instead.
All RxJava2 types provide a subscribe method that "notifies" the source of data that's lazy by nature that here is an observer that wants to receive the data, so all the data processing flow described by use of RxJava2 operators will become alive.
The most painless approach is to change HomeActivity's code to this:
vm.getUsers()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(userListAdapter::populateWithNewDataSet);
, assuming that adapter will have the mentioned method that will update the UI data set using something like notifyDataSetChanged() (or DiffUtil, for instance) internally.
By doing that the data source is now observed and every time the update is emitted the UI will be repopulated with the most recent data.
P.S.: I've just demonstrated the simplest way to do the thing, but it is up to the developer where to place RxJava-related code: be it ViewModel, Activity, or even some other component. RxJava is a convenient tool to use and it can make complicated asynchronous flow simple, but the problem with RxJava arises when all the code base is dependent on it. The code base can then quickly become unmanageable, fragile and rigid if the tool was used in an improper place.
Adding on #AndreyIlyunin very good answer, You could also use MutableLivedata in your Viewmodel to save the List in the viewmodel as Livedata and observe changes to it in your Activity. This is suggested by Google as a way to maintain MVVM architecture. Something like:
In HomeScreenViewModel.java:
private final MutableLivedata<List<User>> users = new MutableLivedata<>();
public void getUsers() {
return userRepo.getUsers()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(this::onUsers)
}
private void onUsers(List<> list){
users.setValue(list);
}
public MutableLivedata<List<User>> getUserList(){
return users;
}
In HomeActivity.java, in onCreate() add:
vm.getUserList().observe(this,this::onUserList);
and add following methods to activity:
private void onUserList(List<> list){
userListAdapter = new UserListAdapter(this,list);
}
and then from your activity call:
vm.getUsers();
The getUsers() call is made asynchronously in the background, and you get the userList reactivly.
A bit of context, I’ve tried to apply some clean-architecture to one of my projects and I’m having trouble with the (Realm) disk implementation of my repository. I have a Repository which pulls some data from different DataStores depending on some conditions (cache). This is the theory, the problem comes when mixing all of this with UseCases and RxJava2.
First I get the list of objects from Realm and then I manually create an Observable of it. But the subscribe (as expected) is executed on a different thread so realm ends up crashing… (second block of code)
This is the code I use to create the Observables (from an abstract class DiskStoreBase):
Observable<List<T>> createListFrom(final List<T> list) {
return Observable.create(new ObservableOnSubscribe<List<T>>() {
#Override
public void subscribe(ObservableEmitter<List<T>> emitter) throws Exception {
if (list != null) {
emitter.onNext(list);
emitter.onComplete();
} else {
emitter.onError(new ExceptionCacheNotFound());
}
}
});
}
How can I deal with this scenario?
More code of DiskStoreForZone:
#Override
public Observable<List<ResponseZone>> entityList() {
Realm realm = Realm.getDefaultInstance();
List<ResponseZone> result = realm.where(ResponseZone.class).findAll();
return createListFrom(result);
}
The exact crash:
E/REALM_JNI: jni: ThrowingException 8, Realm accessed from incorrect thread.
E/REALM_JNI: Exception has been thrown: Realm accessed from incorrect thread.
It doesn't work because despite using Rx, your data layer is not reactive.
Realm by its nature is a reactive datasource, and its managed objects by nature are also mutable (updated in place by Realm), and thread-confined (can only be accessed on the same thread where the Realm was opened).
For your code to work, you'd need to copy out the data from the Realm.
#Override
public Single<List<ResponseZone>> entityList() {
return Single.fromCallable(() -> {
try(Realm realm = Realm.getDefaultInstance()) {
return realm.copyFromRealm(realm.where(ResponseZone.class).findAll());
}
});
}
I took the liberty and represented your Single as a Single, considering it's not an Observable, it does not listen for changes, there is only 1 event and that is the list itself. So sending it through an ObservableEmitter doesn't really make sense as it does not emit events.
Therefore, this is why I said: your data layer is not reactive. You are not listening for changes. You are just obtaining data directly, and you are never notified of any change; despite using Rx.
I drew some pictures in paint to illustrate my point. (blue means side-effects)
in your case, you call a one-off operation to retrieve the data from multiple data-sources (cache, local, remote). Once you obtain it, you don't listen for changes; technically if you edit the data in one place and another place, the only way to update is by "forcing the cache to retrieve the new data manually"; for which you must know that you modified the data somewhere else. For which you need a way to either directly call a callback, or send a message/event - a notification for change.
So in a way, you must create a cache invalidation notification event. And if you listen to that, the solution could be reactive again. Except you're doing this manually.
----------------------------------------------------------------------
Considering Realm is already a reactive data source (similarly to SQLBrite for SQLite), it is able to provide change notifications by which you can "invalidate your cache".
In fact, if your local data source is the only source of data, and any write from network is a change that you listen to, then your "cache" can be written down as replay(1).publish().refCount() (replay latest data for new subscribers, replace data with new if new data is evaluated) which is RxReplayingShare.
Using a Scheduler created from the looper of a handler thread, you can listen to changes in the Realm on a background thread, creating a reactive data source that returns up-to-date unmanaged copies that you can pass between threads (although mapping directly to immutable domain models is preferred to copyFromRealm() if you choose this route - the route being clean architecture).
return io.reactivex.Observable.create(new ObservableOnSubscribe<List<ResponseZone>>() {
#Override
public void subscribe(ObservableEmitter<List<ResponseZone>> emitter)
throws Exception {
final Realm observableRealm = Realm.getDefaultInstance();
final RealmResults<ResponseZone> results = observableRealm.where(ResponseZone.class).findAllAsync();
final RealmChangeListener<RealmResults<ResponseZone>> listener = results -> {
if(!emitter.isDisposed()) {
if(results.isValid() && results.isLoaded()) {
emitter.onNext(observableRealm.copyFromRealm(results));
}
}
};
emitter.setDisposable(Disposables.fromRunnable(() -> {
if(results.isValid()) {
results.removeChangeListener(listener);
}
observableRealm.close();
}));
results.addChangeListener(listener);
// initial value will be handled by async query
}
}).subscribeOn(looperScheduler).unsubscribeOn(looperScheduler);
Where looper scheduler is obtained as
handlerThread = new HandlerThread("LOOPER_SCHEDULER");
handlerThread.start();
synchronized(handlerThread) {
looperScheduler = AndroidSchedulers.from(handlerThread.getLooper());
}
And that is how you create reactive clean architecture using Realm.
ADDED:
The LooperScheduler is only needed if you intend to actually enforce Clean Architecture on Realm. This is because Realm by default encourages you to use your data objects as domain models and as a benefit provides lazy-loaded thread-local views that mutate in place when updated; but Clean Architecture says you should use immutable domain models instead (independent from your data layer). So if you want to create reactive clean architecture where you copy from Realm on a background thread any time when Realm changes, then you'll need a looper scheduler (or observe on a background thread, but do the copying from a refreshed Realm on Schedulers.io()).
With Realm, generally you'd want to use RealmObjects as your domain models, and rely on lazy-evaluation. In that case, you do not use copyFromRealm() and you don't map the RealmResults to something else; but you can expose it as a Flowable or a LiveData.
You can read related stuff about this here.
I am using Realm 3.0.0 as the DB of my Android app. It's like a questionnaire application, in which the user navigates inside the app a lot. When I use the app (go back and forth) continuously, I get the following error:
Fatal Exception: io.realm.exceptions.RealmError: Unrecoverable error. mmap() failed: Out of memory size: 1073741824 offset: 0 in /Users/cm/Realm/realm-java/realm/realm-library/src/main/cpp/io_realm_internal_SharedRealm.cpp line 109
at io.realm.internal.SharedRealm.nativeGetSharedRealm(SharedRealm.java)
at io.realm.internal.SharedRealm.(SharedRealm.java:187)
at io.realm.internal.SharedRealm.getInstance(SharedRealm.java:229)
at io.realm.internal.SharedRealm.getInstance(SharedRealm.java:204)
at io.realm.RealmCache.createRealmOrGetFromCache(RealmCache.java:124)
at io.realm.Realm.getDefaultInstance(Realm.java:210)
Now I know the main cause of this is not closing Realm instances. But I've already checked for that multiple times. And I am positive that I close every instance I open.
The app has many activities and fragments that all get a Realm instance on their onCreate and close it on their onDestroy. There are also other background network jobs that run to upload data that get Realm instances. These jobs close their Realm instances when they've finished running or when they cancel.
All of the above get their Realm instance thru injection via Dagger 2:
#Provides
#Nullable
static Realm realm(#Nullable RealmConfiguration configuration) {
if (configuration != null) {
Realm.setDefaultConfiguration(configuration);
return Realm.getDefaultInstance();
}
return null;
}
Configuration is also provided in the same Dagger Module.
To be more specific, a Questionnaire consists of many Question Fragments displayed in a ViewPager. Each Fragment gets injected with a realm. Many interactions in a given Question Fragment write data to the DB (some async, some blocking). These Fragments also query the database on onResume to get their updated Data. Some of this data is also copied out of Realm via realm.copyFromRealm(). Now at any given time of these happening, an upload job is most likely running and reading data from the DB and uploading it to a server. When an upload job finishes, it then writes to the DB.
I think I can have up to 7-12 fragment/activities holding a realm reference on the UI thread at a given moment. And 0-6 other references on 0-3 other threads (Background Jobs).
Moreover, I compact my realm DB via Realm.compactRealm(realmConfiguration) on every app launch (perhaps as a separate problem, this doesn't seem to do it's job consistently).
Above I've tried to describe my Realm usage descriptively without going into details. Now my problem is, when a user excessively uses the app (going back and forth between activities/fragments (realm injection + DB read query), uploading data (realm injection + DB read&write query)), I get the above posted Out of Memory Error.
I am also using Leak Canary, and it hasn't detected any leaks. (Not sure if it can anyway)
Am I using Realm in a way it's not supposed to be used? Should I close Realm instances onPause instead of onDestroy? Should I have only one realm instance in an activity and have all it's fragmetns (up to 5 in my case) use this instance? What kind of changes can I make in my app, and perhaps my app architecture to solve this problem?
I appreciate any help in trying to solve this problem.
EDIT: I'm sharing the realm open-close logic in my background threads.
All my jobs share the same realm usage, which is the following:
Realm is injected lazily via:
#Inject protected transient Lazy<Realm> lazyRealm;
The realm object reference is held at the private transient Realm realm; field. I am using Android Priority Job Queue. When the job is added:
#Override
public void onAdded() {
realm = lazyRealm.get();
realm.executeTransaction(realm1 -> {
//write some stuff into realm
});
realm.close();
}
And when the job is run realm is retreived once, and every possible ending of this method has a call to realm.close()
#Override public void onRun() throws Throwable {
synchronized (syncAdapterLock) {
realm = lazyRealm.get();
Answer answer = realm.where(Answer.class).equalTo(AnswerQuery.ID, answerId).findFirst();
if (answer == null) {
realm.close();
throw new RealmException("File not found");
}
final File photoFile = new File(answer.getFilePath());
final Response response = answerService.uploadPhotoAnswer(answerId, RequestBody.create(MediaType.parse("multipart/form-data"), photoFile)).execute();
if (!response.isSuccessful()) {
realm.close();
throw new HttpError(statusCode);
}
realm.executeTransaction(realm1 -> {
answer.setSyncStatus(SyncStatus.SYNCED.getCode());
});
}
realm.close();
}
}
As you can see, these background threads do close their realm instances properly as far as I'm concerned.
While it was true that all my background tasks did call realm.close(), one of them called it too late in it's lifecycle. That was my GPSService, which is a background service. The problem was that GPS service is initialized at the launch of the App as an Android Service, which is rarely destroyed. I was injecting a realm instance onCreate and closing it onDestroy. After the comments of #EpicPandaForce and reading his articles about using realm properly. I realized that this was the cause of the leak. A non-looper thread was keeping an open realm reference for an extremely long time, thus, the mmap was bloating every time a write transaction occures. Now that I moved the realm get/close to happen every time the service runs, my problem is fixed.
My take away is that one needs to treat background thread realm access very delicately. Thank you both for your quick responses and help!
In my Android projects, I use realm as my data storage engine. I love it!
I also use RxJava because it makes "threading" so much easier, and I really like the whole "reactive mindset". I love it!
I use an MVP pattern + some "Clean architecture" ideas to build my apps.
My Interactors are the only ones who know about Realm. I expose data with the help of Observable, like this:
#Override
public Observable<City> getHomeTown() {
final Realm realm = Realm.getDefaultInstance();
return realm.where(City.class).equalTo("name", "Cluj-Napoca").findAllAsync().asObservable()
.doOnUnsubscribe(new Action0() {
#Override
public void call() {
realm.close();
}
})
.compose(new NullIfNoRealmObject<City>());
}
The problem is my doOnUnsubscribe side-effect gets called before Realm can do its thing, handling the exposed observable:
Caused by: java.lang.IllegalStateException: This Realm instance has already been closed, making it unusable.
at io.realm.BaseRealm.checkIfValid(BaseRealm.java:344)
at io.realm.RealmResults.removeChangeListener(RealmResults.java:818)
at io.realm.rx.RealmObservableFactory$3$2.call(RealmObservableFactory.java:137)
at rx.subscriptions.BooleanSubscription.unsubscribe(BooleanSubscription.java:71)
at rx.internal.util.SubscriptionList.unsubscribeFromAll(SubscriptionList.java:124)
at rx.internal.util.SubscriptionList.unsubscribe(SubscriptionList.java:113)
at rx.Subscriber.unsubscribe(Subscriber.java:98)
at rx.internal.util.SubscriptionList.unsubscribeFromAll(SubscriptionList.java:124)
at rx.internal.util.SubscriptionList.unsubscribe(SubscriptionList.java:113)
at rx.Subscriber.unsubscribe(Subscriber.java:98)
at rx.subscriptions.CompositeSubscription.unsubscribeFromAll(CompositeSubscription.java:150)
at rx.subscriptions.CompositeSubscription.unsubscribe(CompositeSubscription.java:139)
at ro.tudorluca.realm.sandbox.city.CityPresenter.onDestroy(CityPresenter.java:62)
at ro.tudorluca.realm.sandbox.city.CityActivity.onDestroy(CityActivity.java:35)
I created a sandbox project for this use case.
I really like using Realm+RxJava, but I can't seem to find a clean solution to close the Realm instance when I unsubscribe (I usually unsubscribe when the activity gets destroyed). Any ideas?
Edit 1: https://github.com/realm/realm-java/issues/2357
Edit 2: thanks to the very active realm team, there is already a pull request to fix this issue.
21 hours later and this is what I came up with:
#Override
public Observable<City> getHomeTown() {
return getManagedRealm()
.concatMap(new Func1<Realm, Observable<City>>() {
#Override
public Observable<City> call(Realm realm) {
return realm.where(City.class).equalTo("name", "Cluj-Napoca").findAllAsync().asObservable()
.compose(new NullIfNoRealmObject<City>());
}
});
}
private static Observable<Realm> getManagedRealm() {
return Observable.create(new Observable.OnSubscribe<Realm>() {
#Override
public void call(final Subscriber<? super Realm> subscriber) {
final Realm realm = Realm.getDefaultInstance();
subscriber.add(Subscriptions.create(new Action0() {
#Override
public void call() {
realm.close();
}
}));
subscriber.onNext(realm);
}
});
}
I tried something like this before posting the question on stackoverflow, but my mistake was using flatMap(), instead of concatMap().
Unlike flatMap(),concatMap() will keep the order of the emissions which, in my case, means that my Action0 -> realm.close() will be the last action being called after unsubscribing from the stream, after Realm's Action0 -> results.removeChangeListener(listener) which was causing the problem.
A full example can be found on github.
Edit: thanks to the very active realm team, there is already a pull request to fix this issue.
Since you said that only the Interactor "knows" about the Realm framework I would say not to even return a managed Realm object, instead return an unmanaged copy of the results using copyFromRealm. That way you don't have to care about the Realm instance being open or closed in the Presenter.
At the same time I would let the Presenter chose wether the call should be done asynchronous or not since RxJava does that pretty cool and easy and you won't have issues calling the Interactor load method within another thread(which can be avoided using Loopers but why overcomplicate the situation if you can make it simpler :P ).
So I would go for:
Override
public Observable<City> getHomeTown() {
final Realm realm = Realm.getDefaultInstance();
City city = realm.where(City.class).equalTo("name", "Cluj-Napoca").findFirst();
// make sure we don't send back Realm stuff, this is a deep copy that will copy all referenced objects (as the method doc says)
City cityUnmanaged = realm.copyFromRealm(city);
// safe to close the realm instance now
realm.close();
return Observable.just(cityUnmanaged);
}
I am curious to see more options :).
As per me, one of the major thing to look after in a good architecture is modularity. All the major modules(or libraries) should be isolated from rest of the code. Since Realm, RealmObject or RealmResult can not be passed across threads, it is even more important to make Realm & Realm related operations isolated from rest of the code.
Keeping this philosophy in mind, I came up with the following approach.
For every jsonModel class, we create a realmModel class and a DAO (Data Access Object) class. Idea here is that other than DAO class none of the class must know or access realmModel or Realm entities. DAO class takes jsonModel, converts it to realmModel, performs read/write/edit/remove operations & for read operations DAO converts resulted realmModel to jsonModel and returns them.
This way it is easy to maintain Realm, avoid all thread related issues, easy to test and debug.
Here is an article about Realm best practices with a good architechture https://medium.com/#Viraj.Tank/realm-integration-in-android-best-practices-449919d25f2f
Also a sample project demonstrating Integration of Realm on Android with MVP(Model View Presenter), RxJava, Retrofit, Dagger, Annotations & Testing. https://github.com/viraj49/Realm_android-injection-rx-test