I have a long-running operation which I perform in a background thread. As it is important for the operation to either complete successfully or not at all, I am wrapping the entire operation in a transaction.
Aspects of the UI need read-only access to the database during this time. To avoid blocking the UI, I am experimenting with inserting calls to db.yieldIfContendedSafely() in the main loop of the background operation.
This does what I want in that the UI is no longer blocked, but it's not completely clear to me if this is risking a loss of data integrity.
The javadoc for yieldIfContendedSafely() says:
Temporarily end the transaction to let other threads run. The
transaction is assumed to be successful so far. Do not call
setTransactionSuccessful before calling this. When this returns a new
transaction will have been created but not marked as successful. This
assumes that there are no nested transactions (beginTransaction has
only been called once) and will throw an exception if that is not the
case.
Does this mean that my long-running operation is actually being committed to the database in separate chunks, or is the overall transaction maintaining enough state to commit the whole lot in one go at the end, thus preserving data-integrity?
Does this mean that my long-running operation is actually being committed to the database in separate chunks
Yes. Within yieldIfContendedSafely(), Android calls setTransactionSuccessful(), endTransaction(), and begins a new transaction -- committing your statements in the process. There is no mechanism to rollback the "real" transaction after it ends.
This behavior only occurs if there is another thread waiting on the database, otherwise yieldIfContendedSafely() does nothing.
I checked this with the following scenario. I started two threads: one inserted data into a table using a transaction, another read data out of the same table. The transaction didn't call setTransactionSuccessful() so normally everything is rolled back at the end, leaving the table empty. I added a call to yieldIfContendedSafely(), and afterwards the table was not empty and had data from the transaction.
Related
I know transactions are supposed to enforce ACID properties but w.r.t transaction in SQLite, a guy here warns about the transaction done on same connection to be visible to others:
By default — changes that are being done in a transaction on a single
SQLite database connection can be visible to other transactions on
that connection immediately — even before calling
SQLiteDatabase.endTransaction()
which is on Medium and since no one has pointed out, seems to be authentic advice.
Now, I was reading about enableWriteAheadLogging() as a solution for concurrent Db access from official docs and found this:
This method enables parallel execution of queries from multiple
threads on the same database. It does this by opening multiple
connections to the database and using a different database connection
for each query. The database journal mode is also changed to enable
writes to proceed concurrently with reads.
When write-ahead logging is not enabled (the default), it is not possible for reads and writes to occur on the database at the same
time. Before modifying the database, the writer implicitly acquires an
exclusive lock on the database which prevents readers from accessing
the database until the write is completed.
In contrast, when
write-ahead logging is enabled (by calling this method), write
operations occur in a separate log file which allows reads to proceed
concurrently. While a write is in progress, readers on other threads
will perceive the state of the database as it was before the write
began. When the write completes, readers on other threads will then
perceive the new state of the database.
Now, if you read the highlighted part above, you see that by default, without enabling write ahead logging, the default behavior prevents concurrent read/write access and blocks until current operation has completed. If you contrast this with what the gut above said, it seems that with transaction, there is no such blocking.
How can this be possible that non-transactional behavior prevents you from reading or writing to the Db but the transactional one does?
There is no "non-transactional behaviour":
No changes can be made to the database except within a transaction. Any command that changes the database (basically, any SQL command other than SELECT) will automatically start a transaction if one is not already in effect. Automatically started transactions are committed when the last query finishes.
What that guy says is technically wrong; there are no "other transactions on that connection".
A connection can have only one active transaction at the same time.
When multiple threads share the same connection, they share the transaction.
And because they are in the same transaction, they are not isolated from each other. Any thread that executes a BEGIN/COMMIT/ROLLBACK, or any other SQL statement, affects all other threads on the same connection.
By default — changes that are being done in a transaction on a single
SQLite database connection can be visible to other transactions on
that connection immediately — even before calling
SQLiteDatabase.endTransaction()
You may be READING (pun intentional) more into what has been said. i.e. The changes applied will be available i.e. they can be SEEN/READ within the connection BUT they have not been written/committed.
It's only when/if
a) the outer AND ALL inner/nested transactions have been marked as clean/OK by a setTransactionSucccessful AND
b) the endTransaction is invoked
that the changes are committed (written to disk). i.e. it is only at/during the END that data is written to disk.
I've read the warnings about keeping long-lived Realm instances on background non-Looper threads. I've also seen the suggestion that opening and closing Realm instances quickly is not the best idea. Given those constraints (if either is invalid, let me know), I'm trying to identify the best way to use Realm with websockets, where websocket events contain information that requires access to the Realm.
Particularly, what is the right way to go between these options (or something else entirely?):
Open and close the Realm on every event
Open the Realm at the start of the thread, periodically (every 30s or so) begin and commit transaction on the thread to bring the Realm up to date.
Allocate a Looper for the thread that handles websocket messages. Create the Realm instance once on the thread, and leave it open for the thread's lifetime, using the Looper to keep it up to date.
Other things worth noting:
The client will already have a Realm open on the UI thread. So, as far as I understand, this means the background threads at least do not need to pay the price of schema validation.
There's no way to predict the frequency of websocket events. In the typical case, there may be no more than one or two events per second. However, in the case that a large operation happens on the server which changes a bunch of objects, the client may receive hundreds or thousands of websocket events fairly rapidly.
Open and close the Realm on every event
Personally, I find that to be reliable according to the people who say "their Realm is out of date", so this only makes sense if you actually call refresh() after Realm.getDefaultInstance().
Open the Realm at the start of the thread, periodically (every 30s or so) begin and commit transaction on the thread to bring the Realm up to date.
Periodic updates would cause either needless transactions, or be "unaware of change" every now and then.
Also, you actually don't need to commit the transaction to force an update, because beginTransaction() already brings the Realm to the new version.
realm.beginTransaction();
realm.cancelTransaction();
This would be sufficient on its own for that - but technically, this is a workaround to replace using refresh() (not part of public API), with also blocking the thread if a transaction is open on another thread.
Allocate a Looper for the thread that handles websocket messages. Create the Realm instance once on the thread, and leave it open for the thread's lifetime, using the Looper to keep it up to date.
If you want to ensure at all times that the current thread is up to date reliably and also without blocking other threads in the process, then this is the way to go.
So the two ways are either:
1.) keep a newSingleThreadedExecutor() that does queries to the Realm only inside a transaction
2.) keep a HandlerThread alive for as long as needed, and execute the web socket + query stuff there
(Personally, I've only ever used HandlerThread with RxJava + AndroidSchedulers.from(handlerThread.getLooper())).
I have a really long running transaction in background thread which...
iterates through around batches of 400 records
checks if it exists
update if it exists, create new otherwise
iterate through all of them again, to do some operation
and number of batches may vary.
The problem is: as long as the transaction is running, I cannot access database. I'm using a single instance of SQLiteOpenHelper to access database throughout application.
I've tried yieldIfContendedSafely() method, but it doesn't seem to work (or am I using it incorrectly?)
If I try to access database when the transaction is running, it does not throw any exception. It just sits there, like in a queue, and runs immediately when transaction ends.
PS: It is safe to access the database in this scenario, since transaction is running on one table, and I'm trying to read another table.
I am reading SQLite in Android, I see that there is transaction methods to do CRUD operation on SQLite in Android, same thing can be done without transactions methods.
Transaction methods : beginTransaction, setTransactionSuccessfull, endTransaction.
In what cases should we use the transactions method over simple approach.
All sqlite writes are in transactions. If you don't start one explicitly yourself, one will be started and committed for you implicitly. This implicit transaction finishes automatically when the statement finishes. Further reading.
Generally, you want explicit transactions when you want to group more than one database operation as one to maintain ACID properties of your data model. As a consequence, you also gain performance benefit since you don't need to wait for I/O after each statement, only at commit.
It's also worth noting that sqlite does not support nested transactions but the Android sqlite API emulates them using nest level counting. As a consequence, any nested Android sqlite transaction really commits or rollbacks when the outermost transactions is committed or rolled back.
I'm processing a lot of XML data that validates the local data storage within an AsyncTask object. First I tried to use transactions for these operations but while a transaction is in progress any other actions by the user will make the app freeze and wait for the transaction to finish, and sometimes even make the app stop responding.
The transactions are divided into several steps sometimes counting to a couple of hundreds per iteration. Because of the problems I went from using transactions to real-time queries which is very slow but solves the freezing - resulting in a very battery consuming application.
My question is; Is there a way to stop the transactions from locking the database? Or is my problem a result of poor preparation before the transactions?
Transactions are tend to lock your table(s) while doing their business, so there's no way that you can play transactions and non-transactional queries on a same instance at same time.
However, what you need to do is to process your data (xml) first (which might be time consuming) and later kick-in transaction once you've got the data ready.
P.S. I would personally suggest you to make use of transactions (in an efficient way) when it comes to insert multiple records because it creates a single Journal-file to handle all the insertion and speeds up SQLite operations a lot.
make your transactions smaller and don't forget (like I did) to still end transaction if exception thrown - use try, catch, finally...