I have a SQLite database with with just over 6,000 rows of addresses in a table. This is a read-only database - no updates or changes after the app is built and deployed. I have an index on the state field. My app uses a simple select statement to get all rows that match the given state. I have used the explain and explain query plan statements to see that my query is using the index.
Most of the time the query comes back in under a second - not great, but good enough for my application.
Every so often the query takes longer - even up to 14 seconds, often 3-4 seconds. Exact same query on the exact same read-only database (and table) on the same phone, invoked by the exact same binary.
I can see that no garbage collection is occurring, and no exceptions are being generated from monitoring logcat
There is just a variation that sometimes occurs. A variation that creates an inconsistent user experience.
It appears that the SQLite database system is being shared by other apps - such as the email client. Could it be that my query is being queued behind another app's queries and thus the variation is due to when the shared SQLite database system actually gets to run my query? If this is the case, is it possible to "create my own SQLite instance" so that I can get consistent performance?
If it is not a shared SQLite database system (and thus I do have my own instance) what else could be causing such a large variation in query performance given that everything else is equal?
Note that I can't easily bring the data into memory to run the query there as the rows are pretty long (have more information than just the address) and I have a number of other parts of my code that make use of more complex select queries. I've narrowed the performance variation down to just the simplest "select where state = " query for this question (plea for help).
It appears that the SQLite database system is being shared by other apps - such as the email client.
Not exactly. Storage is shared by other apps. And on Android 1.x and most 2.x devices, internal storage is formatted YAFFS2, which only allows one process to access the storage at a time. This should be less of a problem on Android 3.0+ devices (and some 2.3 devices) that are running ext4 instead of YAFFS2.
Could it be that my query is being queued behind another app's queries and thus the variation is due to when the shared SQLite database system actually gets to run my query?
Not exactly. Your disk I/O could be queued behind another app's disk I/O, though.
Related
As far as I understand, a SQLite Cursor is a snapshot of queried data.
So, as the data is a snapshot, is it correct to assume that deletions/updates/iterations made by a third-party to the database, will not affect the cursor snapshot when iterating?
Since a SQLite cursor doesn't hold all the data into memory, as it loads the records when necessary while iterating, is the queried snapshot data completely isolated from the database? or will it be some illegal state situation if a third-party performs some update/delete on the records which aren't yet into memory as iteration didn't reach them yet?
SQLite has two ways of handling concurrent access :
A database lock : This is the simpler and older mechanism. Basically the lock is shared (between readers) or exclusive (held by a single writer), which means that the data can not be modified until the cursor is closed.
Write-Ahead Logging. This newer mechanism (activated by default since Android 9) allows concurrent readers and (one) writer while maintaining consistency : The cursor will retrieve the data without the changes made by later commits. From 2.2 Concurency :
When a read operation begins on a WAL-mode database, it first
remembers the location of the last valid commit record in the WAL.
Call this point the "end mark". Because the WAL can be growing and
adding new commit records while various readers connect to the
database, each reader can potentially have its own end mark. But for
any particular reader, the end mark is unchanged for the duration of
the transaction, thus ensuring that a single read transaction only
sees the database content as it existed at a single point in time.
The whole page is worth reading if you're interested in how SQLite works
I've got a table with about 7 million rows in it. I'm inserting on average about one row every second into the database. When I do this, I am noticing that it is taking an incredibly long time (as much as 15 seconds) to run a simple SELECT against the database, e.g. something like:
SELECT * FROM table WHERE rowid > 7100000
This select often returns no rows of data as sometimes no data has been inserted in this particular table. It is often happening even when the table I'm writing to isn't even actually inserting rows into the table I am reading.
The idea is that there are two separate processes, one is adding data, the other is trying to get all new data that has not yet been read. But the read side is connected to a UI and any noticable lag is intolerable, much less 15 seconds. This is being run under Android and the the UI thread doesn't like being blocked for that long either and it is wreaking havoc.
My initial thought was maybe the insert is requiring an update to the indicies as originally I had the index on a different field (a time field). This seems at least partially confirmed because if I use a database with only a few rows each select completes in a few milliseconds. But when I re-created the table to only have the rowid as primary key it actually got slower. I would expect inserting a new row at the end would always result in very fast reads when just comparing on the rowid as primary key.
I have tried enabling write ahead logging, but it appears that SQLCipher doesn't support this, at least not directly, as it doesn't adhere to the lastest API for android.database.sqlite.SQLiteDatabase. Even using "PRAGMA journal_mode = WAL" in the postKey hook hasn't made any difference.
What's going on here? How can I speed up my selects?
Update: I tried getting rid of sqlcipher and just using plain sqlite to see if that was a factor. I used sqlcipher_export to export to a plaintext database, and then used the default android.database.sqlite.SQLCipher. The delay time dropped from 10-20s to 1.8-2.8s. I then removed write-ahead and it dropped further to 1.3-2.7s. So the issue is still noticably there, although it did get a lot better.
SQLite is ultimately file-based, and there is no portable mechanism to communicate to another process which part of a file has changed. So when one process has written something, all other processes must drop their caches when they access the database file the next time.
If possible, modify your architecture that both parts of the code are in the same process and share the same database connection. (With multiple threads, this requires locking, but SQLite has not much concurrency anyway.)
Alternatively, write the new data into a separate database, and let the UI app move it to its own database.
I don't know why SQLCipher is so much slower (it's unlikely to be the CPU overhead of the decryption).
I have several lists that I want to store in my sqlite db in a multithreading fashion using activeAndroid. I run a thread for each list to persist.
The body of each thread look like this.
ActiveAndroid.beginTransaction();
try {
for (MyObjToPersist e : myListOfObjToPersist){
e.save();
}
ActiveAndroid.setTransactionSuccessful();
}
finally {
ActiveAndroid.endTransaction();
}
The transaction seems to add a lock to the db since each thread run one after the other.
Without using transaction things work as expected but the tasks are pretty slow (a hundred time slowest).
This subject are discuss here.
Anybody know how to avoid this behaviour?
Thanks
TL;DR: Only one thread can make modification, but all can read the data.
Can multiple applications or multiple instances of the same application access a single database file at the same time?
Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.
SQLite uses reader/writer locks to control access to the database. (Under Win95/98/ME which lacks support for reader/writer locks, a probabilistic simulation is used instead.) But use caution: this locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. You should avoid putting SQLite database files on NFS if multiple processes might try to access the file at the same time. On Windows, Microsoft's documentation says that locking may not work under FAT filesystems if you are not running the Share.exe daemon. People who have a lot of experience with Windows tell me that file locking of network files is very buggy and is not dependable. If what they say is true, sharing an SQLite database between two or more Windows machines might cause unexpected problems.
We are aware of no other embedded SQL database engine that supports as much concurrency as SQLite. SQLite allows multiple processes to have the database file open at once, and for multiple processes to read the database at once. When any process wants to write, it must lock the entire database file for the duration of its update. But that normally only takes a few milliseconds. Other processes just wait on the writer to finish then continue about their business. Other embedded SQL database engines typically only allow a single process to connect to the database at once.
However, client/server database engines (such as PostgreSQL, MySQL, or Oracle) usually support a higher level of concurrency and allow multiple processes to be writing to the same database at the same time. This is possible in a client/server database because there is always a single well-controlled server process available to coordinate access. If your application has a need for a lot of concurrency, then you should consider using a client/server database.
But experience suggests that most applications need much less
concurrency than their designers imagine.
When SQLite tries to access a file that is locked by another process, the default behavior is to return SQLITE_BUSY. You can adjust this behavior from C code using the sqlite3_busy_handler() or sqlite3_busy_timeout() API functions.
https://www.sqlite.org/faq.html#q5
I'm curious if anyone has done any performance testing on querying a ContentProvider via ContentResolver vs querying a SQLiteDatabase object in the same process. I'm guessing that a ContentResolver query passes back a Cursor that communicates with the database through a Binder (Android IPC). This means that if I read the contents of 100 records through the Cursor that would result 100 Binder method calls. Are my guesses correct and if so would that be significantly slower than accessing the database in the same process?
I have not done exactly that meassure. What I did was to meassure the performance of multiple inserts via a ContentProvider or directly via a SQLite database.
I inserted around 1000 items (one by one). It was much slower to insert via a ContentProvider. In my test almost 10% slower.
There's no definitive answer and results depends on what and how you do it.
For example, I want to share preferences between apps so ContentProvider seems the perfect answer. Yes if I don't mind a longer delay on first read as just connecting to a ContentProvider takes 120ms on a S10+ these days!
So if you have a UI depends on those settings, you'd better copy the preference file between apps (using a ContentProvider) and then read from the file directly otherwise the UI will be delayed before showing with appropriate theme. Fact is the onStart() will have already been called for the newly created activity.
On the opposite side doing DB operations (if done right) will not change much of the results, unless you need to re-connect frequently as it will add a non neglectable overhead.
I'm developing an Android application. It has multiple threads reading from and writing to the Android SQLite database. I am receiving the following error:
SQLiteException: error code 5: database is locked
I understand the SQLite locks the entire db on inserting/updating, but these errors only seem to happen when inserting/updating while I'm running a select query. The select query returns a cursor which is being left open quite a wile (a few seconds some times) while I iterate over it. If the select query is not running, I never get the locks. I'm surprised that the select could lock the db. Is this possible, or is something else going on?
What's the best way to avoid such locks?
You are probably opening and closing multiple database connections in your various threads. This is a bad idea. Just open a single database connection, and reuse it everywhere; SQLite will then ensure that concurrent accesses are serialized correctly.
As with jcwenger's answer, using a ContentProvider is another way of achieving this, but will require much more intrusive changes to your code.
By avoiding leaving cursors open for "quite a while". If you can afford to have all your data in memory all at once, then do so.
If you can't, then try increasing the busy timeout.
Migrate to a ContentProvider rather than directly accessing the DB. ContentResolver marshals away all the threading issues for you and also allows for other useful features like sharing data between apps or syncing with a server.
The api overhead of ContentResolver is minimal. You just need to define an AUTHORITY string (A unique string identifying the "kind" of your data -- use a "com.example.myapp.contacts" type of string) and use ContentResolver.bla rather than db.bla.
Its caused by beginTransaction() function.Look at your code, the problem is solved for my app to making a comment line this function(beginTransaction) line