I am very much confused regarding access to SQLiteDatabase. Either it should be one connection or multiple connections to have access from multiple threads. I have read many articles including following two.
https://stackoverflow.com/a/3689883/3027124
http://touchlabblog.tumblr.com/post/24474398246/android-sqlite-locking
Both of these recommends to have single connection. Even my own answer to same question was accepted by OP. I used the Singleton approach to access SQLiteopenHelper class.
https://stackoverflow.com/a/35358702/3027124
But I am still confused after reading documentation of enableWriteAheadLogging which states
This method enables parallel execution of queries from multiple
threads on the same database. It does this by opening multiple
connections to the database and using a different database connection
for each query. The database journal mode is also changed to enable
writes to proceed concurrently with reads.
Now this is the confusing part. If I want to access database from simultaneous multiple threads, I have Singleton access to SQLiteOpenHelper which in my understanding means the serial execution of insertions while Simultaneous reads can be done with no-errors. But above documentation says, in order to have Simultaneous access, enableWriteAheadLogging should be called which in returns create multiple connections. What is this going on?? What does it mean if I do insertions by calling getWritableDatabase() using Singleton SQLiteOpenHelper from multiple threads? Will the calls be serial? Should enableWriteAheadLogging be called?
Please clarify.
I would use the singleton instance whether I'm using enableWriteAheadLogging or not when dealing with threading which is the case in most apps unless it's a very trivial app like a sample.
Using a singleton instance is ensuring Thread Safety: The singleton instance ensures that synchronization works across that instance, that means when you have a read and write methods calling the database at the same time from different threads, one of them should wait for the other as the database gets locked while writing to it.
It's really clear that this is the case as written in the documentation and quoted below
it is not
possible for reads and writes to occur on the database at the same
time. Before modifying the database, the writer implicitly acquires an
exclusive lock on the database which prevents readers from accessing
the database until the write is completed.
enableWriteAheadLogging is actually altering the above behavior as the above statement is only true When write-ahead logging is not enabled (the default).
So what happens when you enable write-ahead logging through enableWriteAheadLogging?
It's actually changing the default behavior enabling actual parralism as it changes the underlying database journal file to enable doing a write and a read at the same time, but to do that it needs more memory than usual. Read the documentation quote below to know more!
In contrast, when write-ahead logging is enabled (by calling this
method), write operations occur in a separate log file which allows
reads to proceed concurrently. While a write is in progress, readers
on other threads will perceive the state of the database as it was
before the write began. When the write completes, readers on other
threads will then perceive the new state of the database.
It is a good idea to enable write-ahead logging whenever a database
will be concurrently accessed and modified by multiple threads at the
same time. However, write-ahead logging uses significantly more memory
than ordinary journaling because there are multiple connections to the
same database. So if a database will only be used by a single thread,
or if optimizing concurrency is not very important, then write-ahead
logging should be disabled.
Related
I know transactions are supposed to enforce ACID properties but w.r.t transaction in SQLite, a guy here warns about the transaction done on same connection to be visible to others:
By default — changes that are being done in a transaction on a single
SQLite database connection can be visible to other transactions on
that connection immediately — even before calling
SQLiteDatabase.endTransaction()
which is on Medium and since no one has pointed out, seems to be authentic advice.
Now, I was reading about enableWriteAheadLogging() as a solution for concurrent Db access from official docs and found this:
This method enables parallel execution of queries from multiple
threads on the same database. It does this by opening multiple
connections to the database and using a different database connection
for each query. The database journal mode is also changed to enable
writes to proceed concurrently with reads.
When write-ahead logging is not enabled (the default), it is not possible for reads and writes to occur on the database at the same
time. Before modifying the database, the writer implicitly acquires an
exclusive lock on the database which prevents readers from accessing
the database until the write is completed.
In contrast, when
write-ahead logging is enabled (by calling this method), write
operations occur in a separate log file which allows reads to proceed
concurrently. While a write is in progress, readers on other threads
will perceive the state of the database as it was before the write
began. When the write completes, readers on other threads will then
perceive the new state of the database.
Now, if you read the highlighted part above, you see that by default, without enabling write ahead logging, the default behavior prevents concurrent read/write access and blocks until current operation has completed. If you contrast this with what the gut above said, it seems that with transaction, there is no such blocking.
How can this be possible that non-transactional behavior prevents you from reading or writing to the Db but the transactional one does?
There is no "non-transactional behaviour":
No changes can be made to the database except within a transaction. Any command that changes the database (basically, any SQL command other than SELECT) will automatically start a transaction if one is not already in effect. Automatically started transactions are committed when the last query finishes.
What that guy says is technically wrong; there are no "other transactions on that connection".
A connection can have only one active transaction at the same time.
When multiple threads share the same connection, they share the transaction.
And because they are in the same transaction, they are not isolated from each other. Any thread that executes a BEGIN/COMMIT/ROLLBACK, or any other SQL statement, affects all other threads on the same connection.
By default — changes that are being done in a transaction on a single
SQLite database connection can be visible to other transactions on
that connection immediately — even before calling
SQLiteDatabase.endTransaction()
You may be READING (pun intentional) more into what has been said. i.e. The changes applied will be available i.e. they can be SEEN/READ within the connection BUT they have not been written/committed.
It's only when/if
a) the outer AND ALL inner/nested transactions have been marked as clean/OK by a setTransactionSucccessful AND
b) the endTransaction is invoked
that the changes are committed (written to disk). i.e. it is only at/during the END that data is written to disk.
I have a ContentProvider access in a AsyncTask which generally works fine while looking up some data stored in sqlite on the phone.
However I also have some other background services that do network checks and download and update some data.
It is observed that this network related background activity is affecting my ContentProvider. The ContentProvider which entirely works on offline data either delays the response or just never returns for a long time.
Are then any locking/queuing mechanisms on the ContentProvider or the sqlite database? The writes do not happen to the same tables as the reads.
Are then any locking/queuing mechanisms on the ContentProvider or the sqlite database? The writes do not happen to the same tables as the reads.
ContentProvider implementations require additional synchronization to be thread-safe, i.e the developer is in charge of it
SQLite is synchronized, i.e. writes exclude reads
In your case you might want to enable write-ahead-logging (WAL), api 11+. This enables writes to happen in a separate journal in parallel: holding the lock and excluding parallel reads is not necessary anymore.
See for instance:
SQLite, Content Providers, & Thread Safety
Official docs for SQLite locking
Good explanation of WAL
My code:
SQLiteDatabase db = ...;
db.beginTransaction();
try{
db.update(...);
db.setTransactionSuccessful();
}finally{
db.endTransaction();
}
Now the problem is that endTransaction occasionally throws SQLiteDatabaseLockedException, and I don't know reason, or how to repeat same exception.
From SQLiteDatabaseLockedException I read:
Thrown if the database engine was unable to acquire the database locks
it needs to do its job.
And from beginTransaction I read:
Begins a transaction in EXCLUSIVE mode.
From SQLite manual I read:
An EXCLUSIVE lock is needed in order to write to the database file.
Only one EXCLUSIVE lock is allowed on the file and no other locks of
any kind are allowed to coexist with an EXCLUSIVE lock. In order to
maximize concurrency, SQLite works to minimize the amount of time that
EXCLUSIVE locks are held.
So how can DB lock not be acquired in endTransaction when I hold exclusive lock from beginTransaction?
Android version where this happens is 4.0.4 (I have crash report, but not able to repeat this).
Need to say that I enabled SQLiteDatabase.enableWriteAheadLogging on the DB, maybe it matters? My app accessess DB in multiple threads.
Anyway, I'd like to get clear explanation, and make simple example that can repeat conditions repeating the problem, so that I can make real fix.
Thanks.
IMO your code is correct in a singular thread app, so it must be an enableWriteAheadLogging related issue. Maybe this can help:
SOURCE
... The maximum number of connections used to execute queries in
parallel is dependent upon the device memory and possibly other
properties.
...
Writers should use beginTransactionNonExclusive() or
beginTransactionWithListenerNonExclusive(SQLiteTransactionListener) to
start a transaction. Non-exclusive mode allows database file to be in
readable by other threads executing queries.
...
As I understand in serializable (default) mode of sqlite the locks refer not to different threads but to connections (this mode even knows nothing about threads). So if you are using the same connection (and all SqliteDatabase objects, produced by one sqliteOpenHelper instance share the same connection) across multiple threads you are totally unprotected.
Use either connection per thread or non-database synchronization locks, if you insist on multithreaded use of Sqlite. I would prefer a wrapper singleton with a single thread lock protecting its every transaction-like method. But this depends on your app specifics.
Please read this extensive answer and links in it for details and best practices for Sqlite multithreading.
I've read a lot of topics, but can't figure out answer for question: is it possible to read and write simultaneous?
I have background thread that updates some data and UI needs small piece of data stored in DB. So in UI thread SELECT operation is performed. But it blocks when update is in progress. As result, UI freezes for several seconds.
Does anyone has success in reading from DB when writing?
Its possible to read and write to DB on iPhone. Does the reason of such difference is in synchronious implementation of wrapper on native sqlite functions?
On Android 3.0 and higher SQLiteDatabases support WAL mode (write-ahead logging):
When write-ahead logging is not enabled (the default), it is not
possible for reads and writes to occur on the database at the same
time. Before modifying the database, the writer implicitly acquires an
exclusive lock on the database which prevents readers from accessing
the database until the write is completed.
In contrast, when write-ahead logging is enabled, write operations
occur in a separate log file which allows reads to proceed
concurrently. While a write is in progress, readers on other threads
will perceive the state of the database as it was before the write
began. When the write completes, readers on other threads will then
perceive the new state of the database.
http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html#enableWriteAheadLogging()
To start a transaction in WAL mode use beginTransactionNonExclusive() instead of beginTransaction().
While beginTransaction() starts a transaction in EXCLUSIVE mode, beginTransactionNonExclusive() starts one in IMMEDIATE mode
EXCLUSIVE mode uses exclusive locks (http://www.sqlite.org/lockingv3.html#excl_lock) meaning no other database connection except for read_uncommitted connections will be able to read the database and no other connection without exception will be able to write the database until the transaction is complete
IMMEDIATE mode uses reserved locks (http://www.sqlite.org/lockingv3.html#reserved_lock) meaning no other database connection will be able to write to the database or do a BEGIN IMMEDIATE or BEGIN EXCLUSIVE, other processes can continue to read from the database, however.
In simpler words: call beginTransactionNonExclusive() for IMMEDIATE mode and we can read while another thread is writing (the state before the write transaction started because we won't use read_uncommitted connections -> http://en.wikipedia.org/wiki/Isolation_%28database_systems%29#Dirty_reads).
Starting with API 11 Android has support for WAL mode. It keeps original data untouched during transaction, so other threads can read when transaction is running. You can check my article for more details about WAL mode:
http://www.skoumal.net/en/parallel-read-and-write-in-sqlite/
You should also avoid running database queries in UI thread. It could always become sluggish and block your UI.
You cannot read and write at the same time. SQLite is a serverless, file-based database.
From the SQLite FAQ:
"When any process wants to write, it must lock the entire database file for the duration of its update. But that normally only takes a few milliseconds. Other processes just wait on the writer to finish then continue about their business. Other embedded SQL database engines typically only allow a single process to connect to the database at once."
It is not possible to read & write simultaneously. However, if you have your SQLite database classes set up correctly (single instance of your DB & helper classes), different threads should be able to grab the DB connection synchronously so that there is not any noticeable lag.
It also sounds like you're trying to do backend work (write to the db) with your UI thread. You should not be doing this. Create an AsyncTask to handle this instead of having your UI thread handle it.
Refer to the SQLiteOpenHelper documentation. Here's a previous post that talks about this as well: What are the best practices for SQLite on Android?
I have an app (Android 2.2 Google API Level 8) that has multiple activities pulling data from a content provider (SELECT only database access).
It also has a service with a central blocking task queue accepting any database write tasks; activities can fire a service request (As intent) which places a task on a blocking queue for sequential retrieval by a single thread and execution. Database is around 4mb.
There is a single database helper which the service uses to call methods to interact with the database including writing to it; all SQL writes are carried out within the database helper.
All database writes are surrounded by a transaction.
All database reads have the cursor closed at the end of the method.
None of the Activities has a handle to the database object, they can only communicate via the content provider or the service.
Any AlarmManager fired tasks - like Activities - only use the service to pop an appropriate task onto the queue.
The service is the only class that has a handle to the database helper.
All database writes are only carried out via a task placed on a queue; I have exhaustedly checked that task execution is sequential being well aware of it being essential to avoid concurrent writes to an SQLite database.
During a run of task executions I consistently get one or two "database is locked" errors on attempting to write to the database triggered by a tasks execution of 'begin transaction'.
In attempting to track down the source of the lock I found that using dbhelper.inTransaction(), dbhelper.isLockedByThisThread(), dbhelper.isLockedByOtherThread() didn't help as they wouldn't indicate an unexpected database lock.
What I did find that worked in detecting a lock early was to create a method with beginTransaction() and setTransactionSuccessful without any actual SQL write code, within a try catch block that would log the issue - always triggered by beginTransaction().
I placed this database lock trap either side of each of the blocking queue task methods in the expectation/hope that I would find a singular culprit that was leaving the database in a locked state after finishing.
I could not find a consistent culprit. After drilling down through from the start of the task call through to the database write I found that a database lock could occur seemingly out of the blue without having been locked by the previously run task (All these tasks run in sequence under the same singular thread).
After looking at a number of other peoples experiences with database locking issues I've tried closing the database connection directly after the transaction has completed on all tasks but this didn't help, if anything seemed to get more database locking occurrences. Tried added a sleep between each task execution; not exhaustively tested but generally found that a delay of 3 seconds or above seemed to stop the database locks appearing. Tried disabling alarm manager fired tasks - didn't make any difference.
Impression I have is that some form of maintenance task external to my application is dropping in and locking the database periodically - perhaps delayed writing of logs. Obviously I'm less than keen on setting a task processing delay so I'm considering having a database lock retry task queue to reattempt database writing as necessary; much prefer to resolve but am running out of ideas.
Can anyone think of some principle or gotcha I've missed?
Is it in reality normal within Android and larger SQLite databases that you'll get occasional database locks?
Thanks
SQLite guarantees sequential access from multiple threads as long as you use a single database connection. How and where are you opening and closing the database connection?
I generally recommend opening the database once on startup, and never closing it. There's no benefit to closing, since the transactional nature of SQLite means that writes are flushed to persistent storage as soon as possible anyway.
I would check if some activity that calls the DB or calls other activity that calls the DB, has only one instance. Otherward it can lock itself, in some sense.
With regards to
Is it in reality normal within Android and larger SQLite databases that you'll get occasional database locks?
No, it is definitely not normal to get occasional database locks. From reading your story you say that you have both a service and a content provider pulling from the database, so it is possible that you are locking the database between the two accesses.
What I generally do is ensure that I handle all of my database access through the content provider. By having a single point of entry to the database you can ensure that every software component is using the same logic to access the DB. Would it be possible to have your service access the DB through the content provider?
It's also good to remember that by placing your DB behind a content provider, it can still be accessed by multiple threads at once. To ensure that you are accessing the DB only one thread at a time you could place synchronized constructs on the DB inside of your content provider. Obviously if you are doing lots of long writes/reads to the DB, locking in this fashion will absolutely destroy your app. Putting all of your DB code inside of the content provider will also give you a single point of debugging which would help you figure out if multiple threads are accessing the DB.