I recently learnt room database and made a note making application. The app follows MVVM design pattern and the user can insert/delete/update notes.
The problem is that for every insert/update/delete , the app's size is increasing.
I checked the phone settings-> Applications-> SimpleNoteApp -> Storage and these are my observations.
The "App size" is fixed. - 15.16 mb
The "user data" keeps on increasing for every database operation.
"cache" doesn't increase no matter how many times I perform db operations
The total app's size is a sum of 1),2), 3)
So this means if I keep using the app for quite a while and even if I delete all the notes in the app, the "User data" will be a cumulative sum of every db operation performed. Correct me if I'm wrong.
There is a similar question on Stack OverFlow here, but it doesn't solve the issue.
As for the coding part, there isn't much going around as these are basic examples of room database, I use " #Update #Insert #Delete " commands.
So this means if I keep using the app for quite a while and even if I delete all the notes in the app, the "User data" will be a cumulative sum of every db operation performed. Correct me if I'm wrong.
This is incorrect. Deleting the rows should remove the data from storage...more or less. There is a write-ahead logging/rollback journal that may mean it persists temporarily, so if it's important for security, you'll have to be more careful. This may also result in temporary per-database-operation increases in the short-term. However, in the long term, if your app's storage size is continuing to grow despite deleting all content from the database, something else is wrong.
Since the developer is not responsible for collecting the garbage, this is actually normal as suggested by #MaximFirsoff in the comments
I ran a test which did 100_000 inserts, and deletions and observed the changes to the storage.
Observation
The garbage is collected after the "UserData" reaches a threshold limit and the data is cleared.
Related
As far as I understand, a SQLite Cursor is a snapshot of queried data.
So, as the data is a snapshot, is it correct to assume that deletions/updates/iterations made by a third-party to the database, will not affect the cursor snapshot when iterating?
Since a SQLite cursor doesn't hold all the data into memory, as it loads the records when necessary while iterating, is the queried snapshot data completely isolated from the database? or will it be some illegal state situation if a third-party performs some update/delete on the records which aren't yet into memory as iteration didn't reach them yet?
SQLite has two ways of handling concurrent access :
A database lock : This is the simpler and older mechanism. Basically the lock is shared (between readers) or exclusive (held by a single writer), which means that the data can not be modified until the cursor is closed.
Write-Ahead Logging. This newer mechanism (activated by default since Android 9) allows concurrent readers and (one) writer while maintaining consistency : The cursor will retrieve the data without the changes made by later commits. From 2.2 Concurency :
When a read operation begins on a WAL-mode database, it first
remembers the location of the last valid commit record in the WAL.
Call this point the "end mark". Because the WAL can be growing and
adding new commit records while various readers connect to the
database, each reader can potentially have its own end mark. But for
any particular reader, the end mark is unchanged for the duration of
the transaction, thus ensuring that a single read transaction only
sees the database content as it existed at a single point in time.
The whole page is worth reading if you're interested in how SQLite works
When my app is offline and I am adding or updating a document the memory increases. Also, showing lists of documents take longer to load. If I run the same code when the device is online the memory stays consistent as well as the speed of the activities that have lists of documents.
I'm currently doing saves like the following:
collectionRef.document(id).set(obj, SetOptions.merge());
Or for batching a couple of records:
batch.set(docRef1, obj1);
batch.set(docRef2, obj2);
batch.commit();
I had listeners for onComplete but in the accepted answer for this question, it seems to indicate that listeners are unnecessary in most situations and that you can't wait for it to complete anyway when you're offline.
In another question they indicate in code that a "Snapshot" is required to properly do online and offline saving: Offline issue with Firestore vs Firebase. But I can't find anywhere else indicating if that will make a difference. I think of Snapshots as being something you attach to a document or query when you want to be notified of changes to it and attaching a listener like that will result in a memory leak if it isn't removed.
Another part of all of this is how this slowness might affect data integrity. When I watch in the profiler in Android Studio I see that the FirestoreWorker can get to a point where it is constantly working even if I don't do anything in the app. I'm not just talking a few seconds, more like a minute. There isn't any ordering guarantee of writes when it is offline that I can find. Trying to stop and restart the app doesn't seem to have any effect on the slowness (although it will reset the memory).
So all of this leads to the question: what is the proper way to add/update data in Firestore when offline so that the app's memory doesn't grow unbounded and slow down?
Cloud Firestore uses SQLite for its persistence mechanism. So for intermittent periods of offline activity, you shouldn't have problems with performance or durability.
However, if you intend to use a Firestore database for very long periods of time, there are some things you should be aware of. Cloud Firestore was not built as an offline database, is an online database that continues to work when you're offline for short or longer periods of time. When offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations.
So I suggest use this database for its online capabilities. As one of the Firebase engineers said and I quote, "It is impossible to build a slow query in Firestore". So, the performance comes from the new indexing capabilities on the backend and these optimizations don't exist when you're offline.
One more thing, if you have many offline clients who are trying to write to the same document, only the last one will be actually be written to servers when the state is changed.
So, to answer your question, there is no proper way to add/update data in Firestore when offline, to have a less memory usage. Just go online and that's it!
When my app is offline and I am adding or updating a document the memory increases. Also, showing lists of documents take longer to load. If I run the same code when the device is online the memory stays consistent as well as the speed of the activities that have lists of documents.
I'm currently doing saves like the following:
collectionRef.document(id).set(obj, SetOptions.merge());
Or for batching a couple of records:
batch.set(docRef1, obj1);
batch.set(docRef2, obj2);
batch.commit();
I had listeners for onComplete but in the accepted answer for this question, it seems to indicate that listeners are unnecessary in most situations and that you can't wait for it to complete anyway when you're offline.
In another question they indicate in code that a "Snapshot" is required to properly do online and offline saving: Offline issue with Firestore vs Firebase. But I can't find anywhere else indicating if that will make a difference. I think of Snapshots as being something you attach to a document or query when you want to be notified of changes to it and attaching a listener like that will result in a memory leak if it isn't removed.
Another part of all of this is how this slowness might affect data integrity. When I watch in the profiler in Android Studio I see that the FirestoreWorker can get to a point where it is constantly working even if I don't do anything in the app. I'm not just talking a few seconds, more like a minute. There isn't any ordering guarantee of writes when it is offline that I can find. Trying to stop and restart the app doesn't seem to have any effect on the slowness (although it will reset the memory).
So all of this leads to the question: what is the proper way to add/update data in Firestore when offline so that the app's memory doesn't grow unbounded and slow down?
Cloud Firestore uses SQLite for its persistence mechanism. So for intermittent periods of offline activity, you shouldn't have problems with performance or durability.
However, if you intend to use a Firestore database for very long periods of time, there are some things you should be aware of. Cloud Firestore was not built as an offline database, is an online database that continues to work when you're offline for short or longer periods of time. When offline, pending writes that have not yet been synced to the server are held in a queue. If you do too many write operations without going online to sync them, that queue will grow fast and it will not slow down only the write operations it will also slow down your read operations.
So I suggest use this database for its online capabilities. As one of the Firebase engineers said and I quote, "It is impossible to build a slow query in Firestore". So, the performance comes from the new indexing capabilities on the backend and these optimizations don't exist when you're offline.
One more thing, if you have many offline clients who are trying to write to the same document, only the last one will be actually be written to servers when the state is changed.
So, to answer your question, there is no proper way to add/update data in Firestore when offline, to have a less memory usage. Just go online and that's it!
I've got a table with about 7 million rows in it. I'm inserting on average about one row every second into the database. When I do this, I am noticing that it is taking an incredibly long time (as much as 15 seconds) to run a simple SELECT against the database, e.g. something like:
SELECT * FROM table WHERE rowid > 7100000
This select often returns no rows of data as sometimes no data has been inserted in this particular table. It is often happening even when the table I'm writing to isn't even actually inserting rows into the table I am reading.
The idea is that there are two separate processes, one is adding data, the other is trying to get all new data that has not yet been read. But the read side is connected to a UI and any noticable lag is intolerable, much less 15 seconds. This is being run under Android and the the UI thread doesn't like being blocked for that long either and it is wreaking havoc.
My initial thought was maybe the insert is requiring an update to the indicies as originally I had the index on a different field (a time field). This seems at least partially confirmed because if I use a database with only a few rows each select completes in a few milliseconds. But when I re-created the table to only have the rowid as primary key it actually got slower. I would expect inserting a new row at the end would always result in very fast reads when just comparing on the rowid as primary key.
I have tried enabling write ahead logging, but it appears that SQLCipher doesn't support this, at least not directly, as it doesn't adhere to the lastest API for android.database.sqlite.SQLiteDatabase. Even using "PRAGMA journal_mode = WAL" in the postKey hook hasn't made any difference.
What's going on here? How can I speed up my selects?
Update: I tried getting rid of sqlcipher and just using plain sqlite to see if that was a factor. I used sqlcipher_export to export to a plaintext database, and then used the default android.database.sqlite.SQLCipher. The delay time dropped from 10-20s to 1.8-2.8s. I then removed write-ahead and it dropped further to 1.3-2.7s. So the issue is still noticably there, although it did get a lot better.
SQLite is ultimately file-based, and there is no portable mechanism to communicate to another process which part of a file has changed. So when one process has written something, all other processes must drop their caches when they access the database file the next time.
If possible, modify your architecture that both parts of the code are in the same process and share the same database connection. (With multiple threads, this requires locking, but SQLite has not much concurrency anyway.)
Alternatively, write the new data into a separate database, and let the UI app move it to its own database.
I don't know why SQLCipher is so much slower (it's unlikely to be the CPU overhead of the decryption).
I have a SQLite database with with just over 6,000 rows of addresses in a table. This is a read-only database - no updates or changes after the app is built and deployed. I have an index on the state field. My app uses a simple select statement to get all rows that match the given state. I have used the explain and explain query plan statements to see that my query is using the index.
Most of the time the query comes back in under a second - not great, but good enough for my application.
Every so often the query takes longer - even up to 14 seconds, often 3-4 seconds. Exact same query on the exact same read-only database (and table) on the same phone, invoked by the exact same binary.
I can see that no garbage collection is occurring, and no exceptions are being generated from monitoring logcat
There is just a variation that sometimes occurs. A variation that creates an inconsistent user experience.
It appears that the SQLite database system is being shared by other apps - such as the email client. Could it be that my query is being queued behind another app's queries and thus the variation is due to when the shared SQLite database system actually gets to run my query? If this is the case, is it possible to "create my own SQLite instance" so that I can get consistent performance?
If it is not a shared SQLite database system (and thus I do have my own instance) what else could be causing such a large variation in query performance given that everything else is equal?
Note that I can't easily bring the data into memory to run the query there as the rows are pretty long (have more information than just the address) and I have a number of other parts of my code that make use of more complex select queries. I've narrowed the performance variation down to just the simplest "select where state = " query for this question (plea for help).
It appears that the SQLite database system is being shared by other apps - such as the email client.
Not exactly. Storage is shared by other apps. And on Android 1.x and most 2.x devices, internal storage is formatted YAFFS2, which only allows one process to access the storage at a time. This should be less of a problem on Android 3.0+ devices (and some 2.3 devices) that are running ext4 instead of YAFFS2.
Could it be that my query is being queued behind another app's queries and thus the variation is due to when the shared SQLite database system actually gets to run my query?
Not exactly. Your disk I/O could be queued behind another app's disk I/O, though.