I have an SQLite database, which receives some heavy use in my app. It is accessed from:
Various activities, to load data into the UI via ASyncTasks (select statements only)
Several background processes (only one of which can run at a time), which do many inserts/updates
I call getWritableDatabase() once and use this everywhere. Each of the background processes uses an IMMEDIATE (i.e. non-locking) transaction, which speeds some of them up from minutes to seconds.
This all worked perfectly pre-Jellybean; the background processes had the speed benefit of the transaction, but this did not block Activites from loading data via ASyncTasks.
Post-Jellybean, each ASyncTask loading data for the GUI is blocked while a transaction is in progress, so often does not return for several seconds. This makes navigation slow and frustrating. I've isolated the blocking to the rawQuery database call.
I can't find any documentation which specifies exactly what changed in Jellybean, other than several methods being deprecated - http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html#isDbLockedByOtherThreads() - with comments like "There is no longer the concept of a database lock, so this method always returns false."
Is there a simple way to mimic the pre-Jellybean behaviour?
It seems that pre-Jelly Bean async tasks were running in parallel all of the time. Now you've got an Executor under the covers running async tasks serially. Good news is, you can specify how your tasks run by branching based on OS version.
Read this article
I've found the solution to my own question. I'm still not sure exactly what Google have changed in Jellybean to stop what worked before working now - but the solution is to use enableWriteAheadLogging().
I had tried this before, to no avail - I was still getting blocked reads while a transaction was in progress. The important bit for me was (in the enableWriteAheadLogging() documentation)
If the database has any attached databases, then execution of queries in parallel is NOT possible.
Unfortunately, my application has two separate databases which are attached together to allow joined queries - enableWriteAheadLogging() returns false. I have prototyped removing the attachment, and this resolves the blocking issue; my select queries all run without delay while another thread runs an IMMEDATE transaction.
Sadly, I'll have to redesign my database schemas to implement a full fix.
Related
I'm wondering what the possible drawbacks are to running my queries on the main thread. At the moment I am loading data from a database using Room and am using this data to generate graphs for the user.
I have already tried putting my queries into separate threads using a runnable, but I am having issues with the application attempting to use data which has not yet been loaded due to the separate thread not finishing it's operation in time. I understand that I could implement some sort of listener or notification, but even then, the user will have to wait an undetermined amount of time before they may view the content because the data for the graphs that they are wanting to view never loads before the button that loads them is clicked.
I have done quite a bit of research and found that it is unadvisable to put the queries on the main thread because it could possibly hang up the thread for a long time and give a poor user experience. So, assuming that I will never exceed 1400 data members, should I worry about this? Are there any caveats that I have not been informed of?
Edit: I am testing the device on a Samsung SM-J106B which is running Android 6.0.1 and I have not noticed any impact on performance even when loading the max of 1400 data members.
DB queries could take long enough that they would cause the UI to skip frames causing a bad visual experience or worse trigger an app not responding exception.
My app has a pretty big database, especially one of the tables - it has like 2 million entities. In the next app release we are doing a huge update - dropping one of the columns and replacing it with another one. For the biggest table this takes ~ 13 sec.
The problem is even though i have put this update in a background thread(still it is in the onUpgrade method of the SQLLiteDatabase), it still blocks the UI thread and causes an android os popup to appear - prompting to either kill the app or wait.
Is the onUpgrade method actually blocking the UI thread, or am I doing something wrong?
Use StrictMode API to detect the exact cause of the UI thread blocking.
I would recommend you to a service and initiate this db upgrade if it's going to be huge one, also provide a loading screen if you don't want see the UI immediately.
Hope this helps.
There was actually another database process that was blocking the UI thread. I managed to identify the problem using the StrictMode API. Thanks #albeee
I have a lot of data that is stored in a CSV file (about 20,100 rows), which I need to insert into a sqlite database.
This insert is taking very long to complete. What is the fastest way to insert this data?
As you have suggested, number of rows are huge I will recommend not to use AsyncTask, as its not tied to your Activity lifecycle i.e if you activity which started it dies, it doesnt mean AsyncTask dies as well, so if you try initiate a AsyncTask and somehow if your activity dies e.g screen rotation or back key pressed, upon restarting another AsyncTask will get spawned rather then it getting linked to already executing AsyncTask. hence duplicating same operations.
So, all in all I would recommend following approach
(A)
Create a IntentService, it's handleIntent() api already executes in a worker thread so you don't have to worry about any thing, and once all messaged in its queue are finished it automatically dies, so no worry at all about leaking any resources.
write your logic for inserting rows in bulk, use content resolver bulkInsert() api for same. I will recommend inserting in 100 roes per batch, you can implement rollback and error checks to make sure insert goes normally.
Once all insert is finish, you can post back to your UI using Handler and Messengers.
with all this you will achieve two major challenge
Not to hang up your UI, escaping any possible ANR
Even if back key is pressed, ensured that db operation goes on smoothly as it was taken up in background task.
Using AsyncTask<>, insert 20,100 rows inserts in database. Using this asynctask whole work run in background. For more information follow this link
The best solution would be using services and executor because as OP described, process can take a lot time. Thanks that You will be able to close app or move it to background with no worried Your long process is destroyed.
Using AsyncTask is not a good idea because it was designed for short operations as it is described on http://developer.android.com/reference/android/os/AsyncTask.html You must also be careful with using it. Changing orientation screen cause recreating view and also task of asynctask.
AsyncTasks should ideally be used for short operations (a few seconds
at the most.) If you need to keep threads running for long periods of
time, it is highly recommended you use the various APIs provided by
the java.util.concurrent package such as Executor, ThreadPoolExecutor
and FutureTask.
I have an application that is doing a LOT of sqllite transactions, I currently have a bit of a hang because I am doing the sqllite actions on the UI thread... yes bad...
so I made each item have a thread and execute on it assuming sqllite api was smart enough to FIFO them.. nope ... now I get database is locked exceptions
this says it should work
without completely rewriting my code, and having a list of transactions queue up and execute them all on the same thread (many different classes, would be kind of a pain)
is there a way for me to check, and not execute a thread unless there isnt a lock? a lock check per se, or something similar that would get this to work, is efficient and isn't a huge rewrite?
Thanks
My answer that you quoted seems to be confusing. You don't have to do anything special when you are accessing the same Android database using the same database object with multiple threads. Under the covers, Sqlite has it's own locking to guarantee that the database will not be corrupted. To quote my answer;
Sqlite under Android is single threaded. Even if multiple threads were using the same database connection, my understanding is that they would be blocked from running concurrently. There is no way to get around this limitation
It has it's own locking which serializes the requests. This means that adding multiple threads will not increase the performance of the database unfortunately.
As my other answer mentions, you cannot use multiple database objects to the same database from multiple threads since there is no locking and you will corrupt your database.
I have a certain update method in my Android App, which generates quite an amount of data - up to hundreds of database entries.
I also have a background service, aside from the UI thread. Both threads have to execute the update method, sometimes even at the same time - basically, this is about generating and caching data to display. Both the UI and background service need this data.
Currently, I have wrapped the method's execution in an ORMLite transaction, which maps to an ordinary SQLite transaction. However, I am afraid that this will bite me in the butt one day, when some race condition screws up the data cache.
The question: Do SQLite transactions protect me from concurrent execution, or should I rather implement some kind of worker thread which is spawned when the generator-method shall start, or blocking if the generator-method is already running?
UPDATE:
I have decided to not rely on SQLite logic for the protection of my high-level Java method. The solution was for me as follows:
Wrap the generating part of the method with synchronized
Introduce a variable which tracks the last time of executing the method (set at the end of the method, so it is the marker of execution END)
First thing in the synchronized section, check if the last execution is in a specific threshold (e.g. <= 100ms in the past)
If yes, skip generation
If no, perform generation
In this way, duplicate generation should not take place, since when the method is accessed from two threads at the same time, the first will generate, but the second will not. The most important part for me here is that it is still blocking, since both threads rely on the generation having taken place after they have called the method.
EDIT:
It seems I'm wrong in my below statement: The SQLite implementation is, according to many, thread safe. I have, however, bitterly experienced threading issues, especially when testing database access, but that must have been caused by other factors in my code then, I assume.
Sorry for the misleading answer.
ORIGIN:
Good question!
You should be very careful here because the standard Android database access objects (such as SQLiteDatabase, Cursor etc) are not thread-safe by default. Not even ContentProvider's seem to give you a complete protection unless you explicitly write them with multithreading in mind.
According to Android documentation on ContentProvider's and threading (almost at the end of the page):
"Because these methods [update() is one of the functions] might be called from any number of threads at the same time, they too must be implemented to be thread-safe."
I don't know if there is any explicit locking mechanism to SQLiteDatabases (as in locking the actual database file). I would assume that a transaction itself would lock, at least the very handle you access your database with. I don't know what is true for the case where you have multiple handles to your database.
Maybe you could try to implement some singleton object (A ContentProvider maybe?) to access your database with, but even then you'd have to manage some sort of "request queue" I suppose.
You should also consider not to make any calls to the file system (the database is on the file system) from the UI-thread, what-so-ever. There is no guarantee that the database will answer in time and you're likely to end up with an ANR (especially as you write "...which generates quite an amount of data").