I have a certain update method in my Android App, which generates quite an amount of data - up to hundreds of database entries.
I also have a background service, aside from the UI thread. Both threads have to execute the update method, sometimes even at the same time - basically, this is about generating and caching data to display. Both the UI and background service need this data.
Currently, I have wrapped the method's execution in an ORMLite transaction, which maps to an ordinary SQLite transaction. However, I am afraid that this will bite me in the butt one day, when some race condition screws up the data cache.
The question: Do SQLite transactions protect me from concurrent execution, or should I rather implement some kind of worker thread which is spawned when the generator-method shall start, or blocking if the generator-method is already running?
UPDATE:
I have decided to not rely on SQLite logic for the protection of my high-level Java method. The solution was for me as follows:
Wrap the generating part of the method with synchronized
Introduce a variable which tracks the last time of executing the method (set at the end of the method, so it is the marker of execution END)
First thing in the synchronized section, check if the last execution is in a specific threshold (e.g. <= 100ms in the past)
If yes, skip generation
If no, perform generation
In this way, duplicate generation should not take place, since when the method is accessed from two threads at the same time, the first will generate, but the second will not. The most important part for me here is that it is still blocking, since both threads rely on the generation having taken place after they have called the method.
EDIT:
It seems I'm wrong in my below statement: The SQLite implementation is, according to many, thread safe. I have, however, bitterly experienced threading issues, especially when testing database access, but that must have been caused by other factors in my code then, I assume.
Sorry for the misleading answer.
ORIGIN:
Good question!
You should be very careful here because the standard Android database access objects (such as SQLiteDatabase, Cursor etc) are not thread-safe by default. Not even ContentProvider's seem to give you a complete protection unless you explicitly write them with multithreading in mind.
According to Android documentation on ContentProvider's and threading (almost at the end of the page):
"Because these methods [update() is one of the functions] might be called from any number of threads at the same time, they too must be implemented to be thread-safe."
I don't know if there is any explicit locking mechanism to SQLiteDatabases (as in locking the actual database file). I would assume that a transaction itself would lock, at least the very handle you access your database with. I don't know what is true for the case where you have multiple handles to your database.
Maybe you could try to implement some singleton object (A ContentProvider maybe?) to access your database with, but even then you'd have to manage some sort of "request queue" I suppose.
You should also consider not to make any calls to the file system (the database is on the file system) from the UI-thread, what-so-ever. There is no guarantee that the database will answer in time and you're likely to end up with an ANR (especially as you write "...which generates quite an amount of data").
Related
I would like to add a SyncAdapter to my app to update server-side data with minimal battery-use. For this, I need to have my application's main activity write data to some shared storage location that the SyncAdapter can then read in the onPerformSync(...) method.
I'm trying to figure out where best to store this data and have the following questions:
Could the SyncAdapter ever be called in a background thread while my main activity is executing in the foreground thread? I.e. do I need to worry about thread-safety of my storage access between the SyncAdapter and my main activity?
Are there ever any situations in which two instances of my main activity could be in the "Created" state? I.e. do I need to worry about thread-safety between two instances of my main activity?
Assuming that I do have to worry about any sort of thread safety, I have the following questions about the different data storage options:
Files in internal storage:
Are there any atomic operations I can perform on files in internal storage?
How about file locks?
SharedPreferences:
If two editors in different threads simultaneously modify different!!! keys in the Shared-Preferences, could two simultaneous commits lead to the loss of one of the changes?
SQL database:
Is the Android SQL Lite framework thread-safe if I simultaneously open the same SQL Lite database file from different threads?
Are there other ways to share data among SyncAdapters and (multiple instances of (if that is even possible)) the main activity?
Aside: To maximize compatibility of my app, I would like to not use any APIs greater than level 5.
I think, it should be possible to fix this issue by using a ContentProvider.
ContentProviders don't inherently solve the multithreading issue, as stated in the documentation:
Data access methods (such as insert(Uri, ContentValues) and update(Uri, ContentValues, String, String[])) may be called from many threads at once, and must be thread-safe. Other methods (such as onCreate()) are only called from the application main thread, and must avoid performing lengthy operations. See the method descriptions for their expected thread behavior.
But, unless the ContentProvider is declared with android:multiprocess=true, there should only ever be a single instance of the ContentProvider (if I understand this correctly), or at least all instances will live in the same process, i.e. they should have access the same static fields in the class definition.
With this, it should be possible to use the standard Java synchronization features to manage access to the storage-backed resources.
Please comment if I'm missing something...
Aside: Why all the "should's"?
Because, unfortunately, with Android's documentation, I'm never 100% sure that it's reliable or complete. Example here: The quote above states that "[o]ther methods (such as onCreate()) are only called from the application main thread"... What happens when I set android:multiprocess=true? The docs there state that "if this flag is set to 'true', the system can create an instance in every process where there's a client that wants to interact with it". Would this lead to a call to onCreate() from a thread other than this application's main thread? Probably...
I have a lot of data that is stored in a CSV file (about 20,100 rows), which I need to insert into a sqlite database.
This insert is taking very long to complete. What is the fastest way to insert this data?
As you have suggested, number of rows are huge I will recommend not to use AsyncTask, as its not tied to your Activity lifecycle i.e if you activity which started it dies, it doesnt mean AsyncTask dies as well, so if you try initiate a AsyncTask and somehow if your activity dies e.g screen rotation or back key pressed, upon restarting another AsyncTask will get spawned rather then it getting linked to already executing AsyncTask. hence duplicating same operations.
So, all in all I would recommend following approach
(A)
Create a IntentService, it's handleIntent() api already executes in a worker thread so you don't have to worry about any thing, and once all messaged in its queue are finished it automatically dies, so no worry at all about leaking any resources.
write your logic for inserting rows in bulk, use content resolver bulkInsert() api for same. I will recommend inserting in 100 roes per batch, you can implement rollback and error checks to make sure insert goes normally.
Once all insert is finish, you can post back to your UI using Handler and Messengers.
with all this you will achieve two major challenge
Not to hang up your UI, escaping any possible ANR
Even if back key is pressed, ensured that db operation goes on smoothly as it was taken up in background task.
Using AsyncTask<>, insert 20,100 rows inserts in database. Using this asynctask whole work run in background. For more information follow this link
The best solution would be using services and executor because as OP described, process can take a lot time. Thanks that You will be able to close app or move it to background with no worried Your long process is destroyed.
Using AsyncTask is not a good idea because it was designed for short operations as it is described on http://developer.android.com/reference/android/os/AsyncTask.html You must also be careful with using it. Changing orientation screen cause recreating view and also task of asynctask.
AsyncTasks should ideally be used for short operations (a few seconds
at the most.) If you need to keep threads running for long periods of
time, it is highly recommended you use the various APIs provided by
the java.util.concurrent package such as Executor, ThreadPoolExecutor
and FutureTask.
In my application, there exist more than one process, and in each process, I need access the same SQLite database (of course, it means more than 2 theads), so I worried about not only the thread-safety about the SQLite, but also the process-safety.
One solution for this case is using content-provider. But from android sdk, it warns that its methods may be called from multiple threads and therefore must be thread-safe. If content provider itself not necessarily means thread-safe, how can I assume it is process-safe?
The article also clarifies that SQLiteDatabase itself is synchronized by default, thus guaranteeing that no two threads will ever touch it at the same time. What if in the multi-process case? Can two processes modify the same table at the same time? Will it crash?
Multiple processes behave just like multiple threads, i.e., their transactions are safe from being interfered with by each other.
You can find answer here: https://www.sqlite.org/faq.html (point 5). Briefly:
Multiple processes can have the same database open at the same time. Multiple
processes can be doing a SELECT at the same time. But only one process can be
making changes to the database at any moment in time, however.
If you use Android Room, see https://issuetracker.google.com/issues/62334005
I have an SQLite database, which receives some heavy use in my app. It is accessed from:
Various activities, to load data into the UI via ASyncTasks (select statements only)
Several background processes (only one of which can run at a time), which do many inserts/updates
I call getWritableDatabase() once and use this everywhere. Each of the background processes uses an IMMEDIATE (i.e. non-locking) transaction, which speeds some of them up from minutes to seconds.
This all worked perfectly pre-Jellybean; the background processes had the speed benefit of the transaction, but this did not block Activites from loading data via ASyncTasks.
Post-Jellybean, each ASyncTask loading data for the GUI is blocked while a transaction is in progress, so often does not return for several seconds. This makes navigation slow and frustrating. I've isolated the blocking to the rawQuery database call.
I can't find any documentation which specifies exactly what changed in Jellybean, other than several methods being deprecated - http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html#isDbLockedByOtherThreads() - with comments like "There is no longer the concept of a database lock, so this method always returns false."
Is there a simple way to mimic the pre-Jellybean behaviour?
It seems that pre-Jelly Bean async tasks were running in parallel all of the time. Now you've got an Executor under the covers running async tasks serially. Good news is, you can specify how your tasks run by branching based on OS version.
Read this article
I've found the solution to my own question. I'm still not sure exactly what Google have changed in Jellybean to stop what worked before working now - but the solution is to use enableWriteAheadLogging().
I had tried this before, to no avail - I was still getting blocked reads while a transaction was in progress. The important bit for me was (in the enableWriteAheadLogging() documentation)
If the database has any attached databases, then execution of queries in parallel is NOT possible.
Unfortunately, my application has two separate databases which are attached together to allow joined queries - enableWriteAheadLogging() returns false. I have prototyped removing the attachment, and this resolves the blocking issue; my select queries all run without delay while another thread runs an IMMEDATE transaction.
Sadly, I'll have to redesign my database schemas to implement a full fix.
I have an application that is doing a LOT of sqllite transactions, I currently have a bit of a hang because I am doing the sqllite actions on the UI thread... yes bad...
so I made each item have a thread and execute on it assuming sqllite api was smart enough to FIFO them.. nope ... now I get database is locked exceptions
this says it should work
without completely rewriting my code, and having a list of transactions queue up and execute them all on the same thread (many different classes, would be kind of a pain)
is there a way for me to check, and not execute a thread unless there isnt a lock? a lock check per se, or something similar that would get this to work, is efficient and isn't a huge rewrite?
Thanks
My answer that you quoted seems to be confusing. You don't have to do anything special when you are accessing the same Android database using the same database object with multiple threads. Under the covers, Sqlite has it's own locking to guarantee that the database will not be corrupted. To quote my answer;
Sqlite under Android is single threaded. Even if multiple threads were using the same database connection, my understanding is that they would be blocked from running concurrently. There is no way to get around this limitation
It has it's own locking which serializes the requests. This means that adding multiple threads will not increase the performance of the database unfortunately.
As my other answer mentions, you cannot use multiple database objects to the same database from multiple threads since there is no locking and you will corrupt your database.