Develop Reference

SQlite transactions: change visibility

I know transactions are supposed to enforce ACID properties but w.r.t transaction in SQLite, a guy here warns about the transaction done on same connection to be visible to others:
By default — changes that are being done in a transaction on a single
SQLite database connection can be visible to other transactions on
that connection immediately — even before calling
SQLiteDatabase.endTransaction()
which is on Medium and since no one has pointed out, seems to be authentic advice.
Now, I was reading about enableWriteAheadLogging() as a solution for concurrent Db access from official docs and found this:
This method enables parallel execution of queries from multiple
threads on the same database. It does this by opening multiple
connections to the database and using a different database connection
for each query. The database journal mode is also changed to enable
writes to proceed concurrently with reads.
When write-ahead logging is not enabled (the default), it is not possible for reads and writes to occur on the database at the same
time. Before modifying the database, the writer implicitly acquires an
exclusive lock on the database which prevents readers from accessing
the database until the write is completed.
In contrast, when
write-ahead logging is enabled (by calling this method), write
operations occur in a separate log file which allows reads to proceed
concurrently. While a write is in progress, readers on other threads
will perceive the state of the database as it was before the write
began. When the write completes, readers on other threads will then
perceive the new state of the database.
Now, if you read the highlighted part above, you see that by default, without enabling write ahead logging, the default behavior prevents concurrent read/write access and blocks until current operation has completed. If you contrast this with what the gut above said, it seems that with transaction, there is no such blocking.
How can this be possible that non-transactional behavior prevents you from reading or writing to the Db but the transactional one does?

There is no "non-transactional behaviour":
No changes can be made to the database except within a transaction. Any command that changes the database (basically, any SQL command other than SELECT) will automatically start a transaction if one is not already in effect. Automatically started transactions are committed when the last query finishes.
What that guy says is technically wrong; there are no "other transactions on that connection".
A connection can have only one active transaction at the same time.
When multiple threads share the same connection, they share the transaction.
And because they are in the same transaction, they are not isolated from each other. Any thread that executes a BEGIN/COMMIT/ROLLBACK, or any other SQL statement, affects all other threads on the same connection.

By default — changes that are being done in a transaction on a single
SQLite database connection can be visible to other transactions on
that connection immediately — even before calling
SQLiteDatabase.endTransaction()
You may be READING (pun intentional) more into what has been said. i.e. The changes applied will be available i.e. they can be SEEN/READ within the connection BUT they have not been written/committed.
It's only when/if
a) the outer AND ALL inner/nested transactions have been marked as clean/OK by a setTransactionSucccessful AND
b) the endTransaction is invoked
that the changes are committed (written to disk). i.e. it is only at/during the END that data is written to disk.

SQLite selecting very slow when recent inserts have been made

I've got a table with about 7 million rows in it. I'm inserting on average about one row every second into the database. When I do this, I am noticing that it is taking an incredibly long time (as much as 15 seconds) to run a simple SELECT against the database, e.g. something like:
SELECT * FROM table WHERE rowid > 7100000
This select often returns no rows of data as sometimes no data has been inserted in this particular table. It is often happening even when the table I'm writing to isn't even actually inserting rows into the table I am reading.
The idea is that there are two separate processes, one is adding data, the other is trying to get all new data that has not yet been read. But the read side is connected to a UI and any noticable lag is intolerable, much less 15 seconds. This is being run under Android and the the UI thread doesn't like being blocked for that long either and it is wreaking havoc.
My initial thought was maybe the insert is requiring an update to the indicies as originally I had the index on a different field (a time field). This seems at least partially confirmed because if I use a database with only a few rows each select completes in a few milliseconds. But when I re-created the table to only have the rowid as primary key it actually got slower. I would expect inserting a new row at the end would always result in very fast reads when just comparing on the rowid as primary key.
I have tried enabling write ahead logging, but it appears that SQLCipher doesn't support this, at least not directly, as it doesn't adhere to the lastest API for android.database.sqlite.SQLiteDatabase. Even using "PRAGMA journal_mode = WAL" in the postKey hook hasn't made any difference.
What's going on here? How can I speed up my selects?
Update: I tried getting rid of sqlcipher and just using plain sqlite to see if that was a factor. I used sqlcipher_export to export to a plaintext database, and then used the default android.database.sqlite.SQLCipher. The delay time dropped from 10-20s to 1.8-2.8s. I then removed write-ahead and it dropped further to 1.3-2.7s. So the issue is still noticably there, although it did get a lot better.

SQLite is ultimately file-based, and there is no portable mechanism to communicate to another process which part of a file has changed. So when one process has written something, all other processes must drop their caches when they access the database file the next time.
If possible, modify your architecture that both parts of the code are in the same process and share the same database connection. (With multiple threads, this requires locking, but SQLite has not much concurrency anyway.)
Alternatively, write the new data into a separate database, and let the UI app move it to its own database.
I don't know why SQLCipher is so much slower (it's unlikely to be the CPU overhead of the decryption).

Android: how can I lock SQLite database accessible with ContentProvider (or other way of executing atomic conditional operations)

I got two tables in my SQLite DB: entities and user_actions. Their approximate schemes:
The flow of the program is something like this (all DB accesses handled by ContentProvider):
The user performs some action which modifies one of the entities
The corresponding entity is updated in entities immediately. locally_modified value of this entity is set to 1
The information about user's action is stored in user_actions
At some point in future a sync session with the server is being initiated (I use SyncAdapter framework)
User's actions from user_actions are uploaded to the server one by one and removed from the DB in a background thread
When the uploading completed, I need to clear locally_modified flags in entities
At this point I encounter my atomicity issue: the synchronization with the server happens in a background thread, therefore the user can use the app and perform additional actions. As a consequence, right before I clear locally_modified flag for an entity, I must check that there are no records in user_actions corresponding to this entity. These three steps must be executed atomically for each entity having locally_modified set to 1:
Query user_actions for entries corresponding to entity's _id
Test whether the query from #1 returned an empty set
Clear locally_modified of that entity to 0
Given the above scenario, I have three questions:
Q1: Is there a way to lock SQLite DB accessed over ContentProvider in Android such that it can be accessed only by the locking thread?
Q2: If the answer to Q1 is positive, what happens if some other thread tries to access a locked DB? What precautions should I take to ensure reliable operation?
Q3: It is possible to execute atomic transactions with conditional logic using ContentProviderOperation? You can use "back-references" as described in this answer and this blog post to reference the result of a previous operations, but is there a way to use that result in some kind of if-else statement?
Thanks

Is there a way to lock SQLite DB in Android such that it can be accessed only by the locking thread?
Yes, have a look at SQLiteDatabase.beginTransaction() (source). I believe you need SQLite's exclusive transactions, but you need to study that a bit more for your exact usage.
If the answer to Q1 is positive, what happens if some other thread tries to access a locked DB? What precautions should I take to ensure reliable operation?
There's an SQLite.amIInTransaction() method that you could check, or just catch an SQLiteDatabaseLockedException (more SQLite exceptions that you should look up)
It is possible to execute atomic transactions with conditional logic using ContentProviderOperation? You can use "back-references" as described in this answer and this blog post to reference the result of a previous operations, but is there a way to use that result in some kind of if-else statement?
Never done that, but it seems that overriding ContentProvider's applyBatch and wrapping it in a transaction should work:
Android: SQLite transactions when using ContentResolver

The answer turned out to be pretty simple, but it is kind of a "hack" - just add additional Uri to ContentProvider.
For example: initially my ContentProvider supported the following URIs:
Uri.withAppendedPath(MyContract.CONTENT_URI, "entities")
Uri.withAppendedPath(MyContract.CONTENT_URI, "user_actions")
In order to support the atomic operation described in the question I added an additional Uri:
Uri.withAppendedPath(MyContract.CONTENT_URI, "clear_modified_flag")
When this Uri is updated through:
getContentResolver().update(
MyContract.ClearModifiedFlag.CONTENT_URI,
new ContentValues(),
null,
null);
my ContentProvider executes an SQLite transaction that locks the database for the duration of the operation and rolls it back in case of any errors (as described in this answer).
That's it.
P.S. my ContentProvider is not exported (i.e. other apps can't access and use it), therefore it is safe to add this new Uri to it. But keep in mind that if you do export your ContentProvider, then exposing functionality like this one could be problematic.

Why to use sqlite Database in Android?

Why do we use the sqlite data base in android.I am developing an android application where the data is to be fetched from the server and do some data calculation and show it on the UI.
Is it good for me to fetch the data into the sqlite DB and update the UI on regular interval from the sqlite in evry 20 minutes or will it be good to sent the Http get request to the server and update the data from teh response on the UI.
I wanted to know which one will be better and why?Why to involve sqlite DB?
The data corresponds to some 40X40 table data on which some heavy mathematical processing is to be done and then displayed on the UI(similar to Stock market application) and data needs to be cleared after every 12 hours.
plz advice
Rgds,
Raul

It is good to use database in your case.
Pros:
If your application gets closed the in memory data will be lost, but after that you will be able to restore the state from the database if you have one
Especially for the case of complex calculations it is good to store the result once in the database and not recalculate it multiple times on demand
The database will untie your UI from the internet connection and thus you will be able to display results even if there is not internet connection
Using database you will be able to fetch the updated data from a background service, without impacting your UI
Organizing your data in database usually makes it a lot easier to manage all the application data.
Cons:
Adding database will require a bit of additional effort on your side
As you can see my list proves you SHOULD use database in your case. Maybe I am biased, but at least I provide you with things to consider.

It's really a design decision, SQLite offers a very robust way to organize and persist your data, you're only other options are to write to a file, or to save in SharedPrefs, both methods become a lot harder to manage once the size of your data begins to grow, as you must manually keep a list of objects and manage their names etc etc. 40 x 40 table data is large enough to justify using SQLite, even if you are dropping and recreating the table every 12 hours.
You might want to consider using an ORM library to make fetching and saving data from the DB simpler, ORMLite is good and compatible with Android
http://ormlite.com/

If your application relies heavily on an internet connection you don't need to buffer information in the database. However if you want to use the app where you have bad or no signal you might want to used cached values from the sqlite database.
With slow internet connection your application may be unresponsive so caching may be a good idea - but it doesn't necessarily be in the sqlite database. You should use the sqlite database for data that is required by the device frequently and that is irrelevant to your server component.
If the data is updated frequently but only while the application runs you might want to cache in the devices memory. I assume your app is not running all the time within the 12 hours but is called regularly instead to check something.

12hrs is a long time, so rather than leaving your data wander in RAM, i would suggest you to use database. Because you never know when you may need to read it again.
Otherwise, if your purpose is only to downloaded data, process it and display in activity, then you dont need to involve database because if your app is closed (due to user or low memory), in anyway your app will be downloading fresh data from server... am i right?

> update the UI on regular interval from the sqlite in evry 20 minutes
Dont expect your app to be open for such a long duration.
To precisely suggest to your case
Avoid DB
Fetch Data at app start or at appropriate time when app is opened
and save it in plain java objects.
Define Methods within it that perform operation in it.
Make map or list to save those POJO
Define Seprate Controller Classes within your project to update map of pojo at any
specific change to make fresh data available to UI.

Android variable SQLite select query performance - any explanations?

I have a SQLite database with with just over 6,000 rows of addresses in a table. This is a read-only database - no updates or changes after the app is built and deployed. I have an index on the state field. My app uses a simple select statement to get all rows that match the given state. I have used the explain and explain query plan statements to see that my query is using the index.
Most of the time the query comes back in under a second - not great, but good enough for my application.
Every so often the query takes longer - even up to 14 seconds, often 3-4 seconds. Exact same query on the exact same read-only database (and table) on the same phone, invoked by the exact same binary.
I can see that no garbage collection is occurring, and no exceptions are being generated from monitoring logcat
There is just a variation that sometimes occurs. A variation that creates an inconsistent user experience.
It appears that the SQLite database system is being shared by other apps - such as the email client. Could it be that my query is being queued behind another app's queries and thus the variation is due to when the shared SQLite database system actually gets to run my query? If this is the case, is it possible to "create my own SQLite instance" so that I can get consistent performance?
If it is not a shared SQLite database system (and thus I do have my own instance) what else could be causing such a large variation in query performance given that everything else is equal?
Note that I can't easily bring the data into memory to run the query there as the rows are pretty long (have more information than just the address) and I have a number of other parts of my code that make use of more complex select queries. I've narrowed the performance variation down to just the simplest "select where state = " query for this question (plea for help).

It appears that the SQLite database system is being shared by other apps - such as the email client.
Not exactly. Storage is shared by other apps. And on Android 1.x and most 2.x devices, internal storage is formatted YAFFS2, which only allows one process to access the storage at a time. This should be less of a problem on Android 3.0+ devices (and some 2.3 devices) that are running ext4 instead of YAFFS2.
Could it be that my query is being queued behind another app's queries and thus the variation is due to when the shared SQLite database system actually gets to run my query?
Not exactly. Your disk I/O could be queued behind another app's disk I/O, though.