Develop Reference

What's the proper way to add/update data in Firestore when offline?

When my app is offline and I am adding or updating a document the memory increases. Also, showing lists of documents take longer to load. If I run the same code when the device is online the memory stays consistent as well as the speed of the activities that have lists of documents.
I'm currently doing saves like the following:
collectionRef.document(id).set(obj, SetOptions.merge());
Or for batching a couple of records:
batch.set(docRef1, obj1);
batch.set(docRef2, obj2);
batch.commit();
I had listeners for onComplete but in the accepted answer for this question, it seems to indicate that listeners are unnecessary in most situations and that you can't wait for it to complete anyway when you're offline.
In another question they indicate in code that a "Snapshot" is required to properly do online and offline saving: Offline issue with Firestore vs Firebase. But I can't find anywhere else indicating if that will make a difference. I think of Snapshots as being something you attach to a document or query when you want to be notified of changes to it and attaching a listener like that will result in a memory leak if it isn't removed.
Another part of all of this is how this slowness might affect data integrity. When I watch in the profiler in Android Studio I see that the FirestoreWorker can get to a point where it is constantly working even if I don't do anything in the app. I'm not just talking a few seconds, more like a minute. There isn't any ordering guarantee of writes when it is offline that I can find. Trying to stop and restart the app doesn't seem to have any effect on the slowness (although it will reset the memory).
So all of this leads to the question: what is the proper way to add/update data in Firestore when offline so that the app's memory doesn't grow unbounded and slow down?

Cloud Firestore uses SQLite for its persistence mechanism. So for intermittent periods of offline activity, you shouldn't have problems with performance or durability.
However, if you intend to use a Firestore database for very long periods of time, there are some things you should be aware of. Cloud Firestore was not built as an offline database, is an online database that continues to work when you're offline for short or longer periods of time. When offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations.
So I suggest use this database for its online capabilities. As one of the Firebase engineers said and I quote, "It is impossible to build a slow query in Firestore". So, the performance comes from the new indexing capabilities on the backend and these optimizations don't exist when you're offline.
One more thing, if you have many offline clients who are trying to write to the same document, only the last one will be actually be written to servers when the state is changed.
So, to answer your question, there is no proper way to add/update data in Firestore when offline, to have a less memory usage. Just go online and that's it!

How to use Firebase RealtimeDatabase like a common Database?

The Firebase RealtimeDatabase works with Listeners, when a event happen so the function is triggered, but I wants a option to update the data when a Button is clicked, for example. I learned about this and found the method DatabaseReference.addListenerForSingleValueEvent, but keeping be a listener. So, exists a way to solve my problem? I'm working with Android.

when a Button is clicked you can use method DatabaseReference.addValueEventListener() to get data from firebase.
And whenever you need cancel this listener: call DatabaseReference.removeEventListener(valueEvenListener);
Sorry my bad English :( :( :(

The main goal of Firebase databases is to work as Real-time databases and not to work as offline databases. Firebase applications work even if your app temporarily loses its network connection. Cached data is available while offline and Firebase resends any writes when network connectivity is restored.
When you enable disk persistence, your app writes the data locally to the device so your app can maintain state while offline. You can enable disk persistence like this:
FirebaseDatabase.getInstance().setPersistenceEnabled(true);
If you want to be compltely offilne you can use goOffline() method
This method shuts down our connection to the Firebase Database backend until goOnline() is called.
However, i don't recomand you using Firebase as an offline-only database. It is really designed as an online database that came work for short to intermediate periods of being disconnected. While offline it will keep queue of write operations. As this queue grows, local operations and app startup will slow down. Nothing major, but over time these may add up.
Hope it helps.

Maintaining connection to Firebase realtime database after app is closed if there is data that is still queued to be stored?

I'm using Firebase's realtime database on Android and the way I understand how it works is that even if the app disconnects from the network, Firebase will simply queue the transactions that the user has initiated and then perform then when connectivity is resumed. This works really well but if the app is closed then this queue seems to be discarded.
The Firebase docs on handling offline capabilities states the following:
Transactions are not persisted across app restarts
Even with persistence enabled, transactions are not persisted across
app restarts. So you cannot rely on transactions done offline being
committed to your Firebase Realtime Database. To provide the best user
experience, your app should show that a transaction has not been saved
into your Firebase Realtime Database yet, or make sure your app
remembers them manually and executes them again after an app restart.
But as far as I know, there is no way of knowing whether or not data has finished being written to the database.
How exactly would you go about making the app manually remembering what still needs to be written to the database? Is there some way of accessing the queue of transactions that is yet to be carried out? Or is there some way of keeping the app running in the background after being closed that could just sync the data when connectivity resumes?
Thanks in advance.

But as far as I know, there is no way of knowing whether or not data has finished being written to the database.
There actually is. The Transaction.Handler interface has a [onComplete method](https://firebase.google.com/docs/reference/android/com/google/firebase/database/Transaction.Handler.html#onComplete(com.google.firebase.database.DatabaseError, boolean, com.google.firebase.database.DataSnapshot)). The boolean that is passed to that argument is a flag to indicate if the transaction was committed:
committed
True if the transaction successfully completed, false if it was aborted or an error occurred
For more information, see the Firebase documentation on transactions.

I think I had the problem you are facing, in my case was a simple confusion. That Firebase warning is not about "transactions" in general, is about the "transaction" method provided by them.
In Android this is reference().runTransaction().
The "transaction" method is used to validate data first, by example, if more than one user can subscribe to an event simultaneously, you can make sure that the last vacant was available.
Since the "transaction" method query the database gives you the data, and the upload data, if there is no network connectivity there is no way to make sure that will work on app restart because there was never a first query to see the data you have to validate.
This seems logical to me, a "transaction" method will create a sort of bridge between the client and the database, this is not random, but because is part of the business logic, then you should warn the user visually that their changes might not be saved since it is offline, or even if it is sensitive not allow the user to do it.
In other cases, the data is indeed stored locally and then uploaded when the app is restarted. So if you do something like
reference.child(key).setValue(myObject);
Thant change will be local until the next time user has an internet connection.
You have to make sure to add the keepSynced to the references you actually need. Setting the syncing to the root, won't solve the problem as a waterfall, make sure to be specific with nodes you need to keep synced, this way the user will see the changed reflected visually in the app.
//Won't work
DatabaseReference root = FirebaseDatabase.getInstance().getReference();
root.keepSynced(true);
//This will work
root.child("event_list").keepSynced(true);
root.child("user_events").child(uid)keepSynced(true);

Firebase : What is the difference between setPersistenceEnabled and keepSynced?

I thought the whole time when I used the following all data for chat conversation will be available offline at any time. Which somehow isn't and all nodes are loaded from the server.
FirebaseDatabase.getInstance().setPersistenceEnabled(true);
Then, according to DOCS :
Persistence Behavior : By enabling persistence, any data that we sync while online will be persisted to disk and available offline, even when we restart the app. This means our app will work as it would online using the local data stored in the cache.
If data will already be available offline when setPersistenceEnabled(true); , why do I need keepSynced(true) ?
DatabaseReference r = FirebaseDatabase.getInstance().getReference("chat/roomid");
r.keepSynced(true);

The Firebase Database client in your app can keep the data from the database in two places: in memory and/or on disk.
When you attach a listener, it synchronizes data from the database to an in memory representation in your app.
If you've enable persistence, the data is automatically also persisted on disk.
When you detach the last listener from a location, the data for that location is flushed from memory. But it is not deleted from disk.
When you keep a location synchronized, the client essentially attaches an empty listener to that location. So the data in the app will always be up to date with what's in the database on the server (as long as there is a network connection). If you haven't enabled persistence, the data will just be kept up to date in memory. If you've enabled persistence, it will also be kept up to date on disk.
While it's most common to use keepSynced with persistence, there are also use-cases without persistence.
For example, if you have a master-detail app, where you frequently bounce from a list of item names to the details of each item. In that case keeping the list of item names synchronized will save you from having to reload that data when the user comes back from the detail screen.
You could of course also simply keep a listener on the data, which is essentially what keepSynced does behind the scenes.

According to Firebase documentation
By default the Firebase client will keep data in memory while your application is running, but not when it is restarted. By setting this value to true, the data will be persisted to on-device (disk) storage and will thus be available again when the app is restarted (even when there is no network connectivity at that time). Note that this method must be called before creating your first Firebase reference and only needs to be called once per application. If your app uses Firebase Authentication, the client will automatically persist the user's authentication token across restarts, even without persistence enabled. But if the auth token expired while offline and you've enabled persistence, the client will pause write operations until you successfully re-authenticate (or explicitly call unauth) to prevent your writes from being sent unauthenticated and failing due to security rules.
Note that it says the data will persist on disk and be available when apps restart. If you look at the life cycle of an Activity, you'll see that an activity stop when you change to other activity. So, as long as your app is still open and user only navigated to others activities, data will remain persisted.
But don't say anything about killing your app and persist data. That's why you need keepSynced():
By calling keepSynced(true) on a location, the data for that location will automatically be downloaded and kept in sync, even when no listeners are attached for that location. Additionally, while a location is kept synced, it will not be evicted from the persistent disk cache.
Pay attencion when it says "while location is kept synced, it will not be evicted from the persistent disk cache", this means that if you don't use keepSynced(true) your data could be flushed way when the app is killed/closed.
So, to persist and continue with data after your app is killed, you need to use both FirebaseDatabase.getInstance().setPersistenceEnabled(true); and keepSynced(true).

setPersistanceEnabled(true) It will store the data for offline use.
keepsynced(true) by default, Firebase keeps 10mb data in cache,if it grows further it will replace by new data.To avoid the same and keep the whole data keepsynced(true) will help You.

Is SQLite appropriate for off-line storage before replication to a server?

I am planning on writing an application that saves a fair amount of data. Historically, I have simply written data directly to a server, and only used some simple key/value storage with shared preferences for local storage.
I am considering this time, instead, using SQLite to save the information at first, and sync the data to the server in the background later. This will benefit the user in a few ways: 1) can use the app offline 2) don't have to worry about data being saved right away, it happens when ever it can 3) more reliability.
My approach will be to get/set data from SQLite during UI usage, and use a background process to find new rows and put them on the server, flagging them as synced when it happens.
Does this sound reasonable?

You can use SQLIte for your scenario. But, while implementing, you can follow any one of this approach.
Approach #1: Use an Abstract Factory to Instantiate the SQLiteOpenHelper.
Approach #2: Wrap the SQLiteDatabase in a ContentProvider
Refer to this link for how to implement these 2 approaches. http://www.androiddesignpatterns.com/2012/05/correctly-managing-your-sqlite-database.html
Key points to be noted while using SQLite
Sqlite takes care of the file level locking.
Many threads can read,one can write. The locks prevent more than one
writing.
Android implements some java locking in SQLiteDatabase to help keep
things straight.
If we handle the database incorrectly from many threads and mess up the code, your
database will not be corrupted. Only few updates will be lost.
How "Multiple Threads - DB access" can be used for your scenario
The SqliteOpenHelper object holds on to one database connection.
If you try to write to the database from actual distinct connections (multiple threads) at the same time, one will fail. It will not wait till the first is done and then write. It will simply not write your change. Worse, if you don’t call the right version of insert/update on the SQLiteDatabase, you won’t get an exception. You’ll just get a message in your LogCat, and that will be it.
So recommended to write using single thread and read from multiple threads if necessary for faster access.

Does this sound reasonable?
Yes. Note that the synchronization process can get tricky (e.g., what happens if the server hiccups halfway through?), but that has mostly to do with synchronization and little to do with SQLite.

We implemented a solution that used a SQLite db on the device to sync data via a web service to the master database. We did this for a couple reasons: offline, poor connection, manual sync.
For our solution we had a flag on the table that determined if the data was pushed to the web service. Our web service also provided data back to our application to let us know if the data was received and processed correctly. This allowed us to clean up the data on the device, send notifications if there were failures, and resubmit the data if there were previous failures.
You can use push notifications as well if you have fixed the issues on the backend and have the device resend the data to the web service. This worked really well for us.