I have a really long running transaction in background thread which...
iterates through around batches of 400 records
checks if it exists
update if it exists, create new otherwise
iterate through all of them again, to do some operation
and number of batches may vary.
The problem is: as long as the transaction is running, I cannot access database. I'm using a single instance of SQLiteOpenHelper to access database throughout application.
I've tried yieldIfContendedSafely() method, but it doesn't seem to work (or am I using it incorrectly?)
If I try to access database when the transaction is running, it does not throw any exception. It just sits there, like in a queue, and runs immediately when transaction ends.
PS: It is safe to access the database in this scenario, since transaction is running on one table, and I'm trying to read another table.
Related
I know transactions are supposed to enforce ACID properties but w.r.t transaction in SQLite, a guy here warns about the transaction done on same connection to be visible to others:
By default — changes that are being done in a transaction on a single
SQLite database connection can be visible to other transactions on
that connection immediately — even before calling
SQLiteDatabase.endTransaction()
which is on Medium and since no one has pointed out, seems to be authentic advice.
Now, I was reading about enableWriteAheadLogging() as a solution for concurrent Db access from official docs and found this:
This method enables parallel execution of queries from multiple
threads on the same database. It does this by opening multiple
connections to the database and using a different database connection
for each query. The database journal mode is also changed to enable
writes to proceed concurrently with reads.
When write-ahead logging is not enabled (the default), it is not possible for reads and writes to occur on the database at the same
time. Before modifying the database, the writer implicitly acquires an
exclusive lock on the database which prevents readers from accessing
the database until the write is completed.
In contrast, when
write-ahead logging is enabled (by calling this method), write
operations occur in a separate log file which allows reads to proceed
concurrently. While a write is in progress, readers on other threads
will perceive the state of the database as it was before the write
began. When the write completes, readers on other threads will then
perceive the new state of the database.
Now, if you read the highlighted part above, you see that by default, without enabling write ahead logging, the default behavior prevents concurrent read/write access and blocks until current operation has completed. If you contrast this with what the gut above said, it seems that with transaction, there is no such blocking.
How can this be possible that non-transactional behavior prevents you from reading or writing to the Db but the transactional one does?
There is no "non-transactional behaviour":
No changes can be made to the database except within a transaction. Any command that changes the database (basically, any SQL command other than SELECT) will automatically start a transaction if one is not already in effect. Automatically started transactions are committed when the last query finishes.
What that guy says is technically wrong; there are no "other transactions on that connection".
A connection can have only one active transaction at the same time.
When multiple threads share the same connection, they share the transaction.
And because they are in the same transaction, they are not isolated from each other. Any thread that executes a BEGIN/COMMIT/ROLLBACK, or any other SQL statement, affects all other threads on the same connection.
By default — changes that are being done in a transaction on a single
SQLite database connection can be visible to other transactions on
that connection immediately — even before calling
SQLiteDatabase.endTransaction()
You may be READING (pun intentional) more into what has been said. i.e. The changes applied will be available i.e. they can be SEEN/READ within the connection BUT they have not been written/committed.
It's only when/if
a) the outer AND ALL inner/nested transactions have been marked as clean/OK by a setTransactionSucccessful AND
b) the endTransaction is invoked
that the changes are committed (written to disk). i.e. it is only at/during the END that data is written to disk.
I have an android application in which I am using Content Provider on top of the database. However, I see that database gets created only when I insert the first record in the database through the Insert of Content Provider. In this scenario, if there's an error in database creation that would remain pending till the insertion of a record.
I would like to know if there's a way I can create the database when the app is accessed for the first time, so, any possible errors in the database creation appear at the earlier stage.
Within our app we show a splash screen specifically for the purpose of creating the db and initialising some application singletons. I'd recommend something similar.
You can start a db (on)Create or db (on)Upgrade by calling SQLiteDatabase#getWritableDatabase or equivalent. Remember to perform this on a background thread and use standard eventing / callbacks to understand when this (synchronous) method has completed.
If something goes wrong then you can catch that exception during start-up although you can't really recover by the sounds of it so you'll be best off not catching any exceptions and making sure your critical init code is bullet proof.
I am creating an Android app for which I need to create a SQLite DB and pre-populate it with some values.
The Android documentation says this about what to do in "onCreate" of the SQLiteOpenHelper:
Called when the database is created for the first time. This is where the creation of tables and the initial population of the tables should happen.
Reference - http://developer.android.com/reference/android/database/sqlite/SQLiteOpenHelper.html#onCreate(android.database.sqlite.SQLiteDatabase)
I am doubtful about the following 2 things -
What is meant by "when database is created for the first time"? Is this done on the first launch of the app or only when the first DB request (read/write etc) is done.
If it is the latter, I fear that it may take quite some time to create DB, pre-populate it with values (I have about 60 rows to be inserted into 1 table) and then read the DB to show it. Is this the best practice?
I have been doing all my DB operations in AsyncTasks. But I am doing the table creations in onCreate using "db.execSQL" statements. Is this fine (in terms of convention/ performance) or should I go for an AsyncTask here as well?
Any help is appreciated.
1) The later. It is done on the first read or write to the DB.
Your fear might be correct, this is why you can ship your app with a database that's already populated. Or you can launch an AsyncTask with a simple SELECT 1 FROM anytable query. More about shipping with DB here. (60 rows is nothing to fear about tho, and you can safely just keep using AsyncTasks).
2) Yes it is fine. The onCreate logic will run when you first read/write the DB, so it if you always use AsyncTasks onCreate will run in an AsyncTask also.
What is meant by "when database is created for the first time"? Is this done on the first launch of the app or only when the first DB request (read/write etc) is done.
It happens when you first query from database in general term. After that only Upgrade method is called that too when you change the db version.
If it is the latter, I fear that it may take quite some time to create DB, pre-populate it with values (I have about 60 rows to be inserted into 1 table) and then read the DB to show it. Is this the best practice?
60 rows insertion is not a big task. More you can read about beginTransaction(),commitTransaction and endTransaction for insertion. It will make your insertion task lighting fast.
I have been doing all my DB operations in AsyncTasks. But I am doing the table creations in onCreate using "db.execSQL" statements. Is this fine (in terms of convention/ performance) or should I go for an AsyncTask here as well?
It good you are doing you Db operation in AsyncTask and its completely fine.
Speaking of DB operations:
Performing DB operations in AsyncTask is not a good approach, generally. As you might encounter a problem called "memory leak", and it might come as a silent assassin in the night.
There's lot written on this issue. Just google "asynctask leak context" and here you go.
So how to perform DB operations?
Using Loader API in conjunction with ContentProvider is considered good approach for querying database. Loader asynchronously queries your database and delivers the result to specified subscribers. Configuration changes or other sudden stuff does not bother it.
And it is really convenient to query your data using loader API once you know how to do it.
Single inserts/updates/deletes might be done directly from the main thread via ContentResolver. These calls will be blocking (synchronous), but I bet you user would never notice anything while the amount of data is not large.
If you're operating on a large dataset, and you fear you'll be significantly blocking UI thread, I'd suggest using IntentService or any custom Service capable of doing operations in background (note that by default Service operates on main UI thread and you have to specify background operation yourself or use IntentService)
Speaking of DB initialisation:
You might create a one-time IntentService, if you're initialising a large set of data. It will handle your request asynchronously and, for example, perform a broadcast that the application is set up and ready, so you might stop a "wait a sec, performing app initialisation" screen and show user your data.
There's also nothing wrong with shipping your database along with application, though it appears to be a bit hackish solution.
Either way, you choose what is more suitable for you.
I have a long-running operation which I perform in a background thread. As it is important for the operation to either complete successfully or not at all, I am wrapping the entire operation in a transaction.
Aspects of the UI need read-only access to the database during this time. To avoid blocking the UI, I am experimenting with inserting calls to db.yieldIfContendedSafely() in the main loop of the background operation.
This does what I want in that the UI is no longer blocked, but it's not completely clear to me if this is risking a loss of data integrity.
The javadoc for yieldIfContendedSafely() says:
Temporarily end the transaction to let other threads run. The
transaction is assumed to be successful so far. Do not call
setTransactionSuccessful before calling this. When this returns a new
transaction will have been created but not marked as successful. This
assumes that there are no nested transactions (beginTransaction has
only been called once) and will throw an exception if that is not the
case.
Does this mean that my long-running operation is actually being committed to the database in separate chunks, or is the overall transaction maintaining enough state to commit the whole lot in one go at the end, thus preserving data-integrity?
Does this mean that my long-running operation is actually being committed to the database in separate chunks
Yes. Within yieldIfContendedSafely(), Android calls setTransactionSuccessful(), endTransaction(), and begins a new transaction -- committing your statements in the process. There is no mechanism to rollback the "real" transaction after it ends.
This behavior only occurs if there is another thread waiting on the database, otherwise yieldIfContendedSafely() does nothing.
I checked this with the following scenario. I started two threads: one inserted data into a table using a transaction, another read data out of the same table. The transaction didn't call setTransactionSuccessful() so normally everything is rolled back at the end, leaving the table empty. I added a call to yieldIfContendedSafely(), and afterwards the table was not empty and had data from the transaction.
I have an app (Android 2.2 Google API Level 8) that has multiple activities pulling data from a content provider (SELECT only database access).
It also has a service with a central blocking task queue accepting any database write tasks; activities can fire a service request (As intent) which places a task on a blocking queue for sequential retrieval by a single thread and execution. Database is around 4mb.
There is a single database helper which the service uses to call methods to interact with the database including writing to it; all SQL writes are carried out within the database helper.
All database writes are surrounded by a transaction.
All database reads have the cursor closed at the end of the method.
None of the Activities has a handle to the database object, they can only communicate via the content provider or the service.
Any AlarmManager fired tasks - like Activities - only use the service to pop an appropriate task onto the queue.
The service is the only class that has a handle to the database helper.
All database writes are only carried out via a task placed on a queue; I have exhaustedly checked that task execution is sequential being well aware of it being essential to avoid concurrent writes to an SQLite database.
During a run of task executions I consistently get one or two "database is locked" errors on attempting to write to the database triggered by a tasks execution of 'begin transaction'.
In attempting to track down the source of the lock I found that using dbhelper.inTransaction(), dbhelper.isLockedByThisThread(), dbhelper.isLockedByOtherThread() didn't help as they wouldn't indicate an unexpected database lock.
What I did find that worked in detecting a lock early was to create a method with beginTransaction() and setTransactionSuccessful without any actual SQL write code, within a try catch block that would log the issue - always triggered by beginTransaction().
I placed this database lock trap either side of each of the blocking queue task methods in the expectation/hope that I would find a singular culprit that was leaving the database in a locked state after finishing.
I could not find a consistent culprit. After drilling down through from the start of the task call through to the database write I found that a database lock could occur seemingly out of the blue without having been locked by the previously run task (All these tasks run in sequence under the same singular thread).
After looking at a number of other peoples experiences with database locking issues I've tried closing the database connection directly after the transaction has completed on all tasks but this didn't help, if anything seemed to get more database locking occurrences. Tried added a sleep between each task execution; not exhaustively tested but generally found that a delay of 3 seconds or above seemed to stop the database locks appearing. Tried disabling alarm manager fired tasks - didn't make any difference.
Impression I have is that some form of maintenance task external to my application is dropping in and locking the database periodically - perhaps delayed writing of logs. Obviously I'm less than keen on setting a task processing delay so I'm considering having a database lock retry task queue to reattempt database writing as necessary; much prefer to resolve but am running out of ideas.
Can anyone think of some principle or gotcha I've missed?
Is it in reality normal within Android and larger SQLite databases that you'll get occasional database locks?
Thanks
SQLite guarantees sequential access from multiple threads as long as you use a single database connection. How and where are you opening and closing the database connection?
I generally recommend opening the database once on startup, and never closing it. There's no benefit to closing, since the transactional nature of SQLite means that writes are flushed to persistent storage as soon as possible anyway.
I would check if some activity that calls the DB or calls other activity that calls the DB, has only one instance. Otherward it can lock itself, in some sense.
With regards to
Is it in reality normal within Android and larger SQLite databases that you'll get occasional database locks?
No, it is definitely not normal to get occasional database locks. From reading your story you say that you have both a service and a content provider pulling from the database, so it is possible that you are locking the database between the two accesses.
What I generally do is ensure that I handle all of my database access through the content provider. By having a single point of entry to the database you can ensure that every software component is using the same logic to access the DB. Would it be possible to have your service access the DB through the content provider?
It's also good to remember that by placing your DB behind a content provider, it can still be accessed by multiple threads at once. To ensure that you are accessing the DB only one thread at a time you could place synchronized constructs on the DB inside of your content provider. Obviously if you are doing lots of long writes/reads to the DB, locking in this fashion will absolutely destroy your app. Putting all of your DB code inside of the content provider will also give you a single point of debugging which would help you figure out if multiple threads are accessing the DB.