I have several lists that I want to store in my sqlite db in a multithreading fashion using activeAndroid. I run a thread for each list to persist.
The body of each thread look like this.
ActiveAndroid.beginTransaction();
try {
for (MyObjToPersist e : myListOfObjToPersist){
e.save();
}
ActiveAndroid.setTransactionSuccessful();
}
finally {
ActiveAndroid.endTransaction();
}
The transaction seems to add a lock to the db since each thread run one after the other.
Without using transaction things work as expected but the tasks are pretty slow (a hundred time slowest).
This subject are discuss here.
Anybody know how to avoid this behaviour?
Thanks
TL;DR: Only one thread can make modification, but all can read the data.
Can multiple applications or multiple instances of the same application access a single database file at the same time?
Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.
SQLite uses reader/writer locks to control access to the database. (Under Win95/98/ME which lacks support for reader/writer locks, a probabilistic simulation is used instead.) But use caution: this locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. You should avoid putting SQLite database files on NFS if multiple processes might try to access the file at the same time. On Windows, Microsoft's documentation says that locking may not work under FAT filesystems if you are not running the Share.exe daemon. People who have a lot of experience with Windows tell me that file locking of network files is very buggy and is not dependable. If what they say is true, sharing an SQLite database between two or more Windows machines might cause unexpected problems.
We are aware of no other embedded SQL database engine that supports as much concurrency as SQLite. SQLite allows multiple processes to have the database file open at once, and for multiple processes to read the database at once. When any process wants to write, it must lock the entire database file for the duration of its update. But that normally only takes a few milliseconds. Other processes just wait on the writer to finish then continue about their business. Other embedded SQL database engines typically only allow a single process to connect to the database at once.
However, client/server database engines (such as PostgreSQL, MySQL, or Oracle) usually support a higher level of concurrency and allow multiple processes to be writing to the same database at the same time. This is possible in a client/server database because there is always a single well-controlled server process available to coordinate access. If your application has a need for a lot of concurrency, then you should consider using a client/server database.
But experience suggests that most applications need much less
concurrency than their designers imagine.
When SQLite tries to access a file that is locked by another process, the default behavior is to return SQLITE_BUSY. You can adjust this behavior from C code using the sqlite3_busy_handler() or sqlite3_busy_timeout() API functions.
https://www.sqlite.org/faq.html#q5
Related
I am planning on writing an application that saves a fair amount of data. Historically, I have simply written data directly to a server, and only used some simple key/value storage with shared preferences for local storage.
I am considering this time, instead, using SQLite to save the information at first, and sync the data to the server in the background later. This will benefit the user in a few ways: 1) can use the app offline 2) don't have to worry about data being saved right away, it happens when ever it can 3) more reliability.
My approach will be to get/set data from SQLite during UI usage, and use a background process to find new rows and put them on the server, flagging them as synced when it happens.
Does this sound reasonable?
You can use SQLIte for your scenario. But, while implementing, you can follow any one of this approach.
Approach #1: Use an Abstract Factory to Instantiate the SQLiteOpenHelper.
Approach #2: Wrap the SQLiteDatabase in a ContentProvider
Refer to this link for how to implement these 2 approaches. http://www.androiddesignpatterns.com/2012/05/correctly-managing-your-sqlite-database.html
Key points to be noted while using SQLite
Sqlite takes care of the file level locking.
Many threads can read,one can write. The locks prevent more than one
writing.
Android implements some java locking in SQLiteDatabase to help keep
things straight.
If we handle the database incorrectly from many threads and mess up the code, your
database will not be corrupted. Only few updates will be lost.
How "Multiple Threads - DB access" can be used for your scenario
The SqliteOpenHelper object holds on to one database connection.
If you try to write to the database from actual distinct connections (multiple threads) at the same time, one will fail. It will not wait till the first is done and then write. It will simply not write your change. Worse, if you don’t call the right version of insert/update on the SQLiteDatabase, you won’t get an exception. You’ll just get a message in your LogCat, and that will be it.
So recommended to write using single thread and read from multiple threads if necessary for faster access.
Does this sound reasonable?
Yes. Note that the synchronization process can get tricky (e.g., what happens if the server hiccups halfway through?), but that has mostly to do with synchronization and little to do with SQLite.
We implemented a solution that used a SQLite db on the device to sync data via a web service to the master database. We did this for a couple reasons: offline, poor connection, manual sync.
For our solution we had a flag on the table that determined if the data was pushed to the web service. Our web service also provided data back to our application to let us know if the data was received and processed correctly. This allowed us to clean up the data on the device, send notifications if there were failures, and resubmit the data if there were previous failures.
You can use push notifications as well if you have fixed the issues on the backend and have the device resend the data to the web service. This worked really well for us.
My code:
SQLiteDatabase db = ...;
db.beginTransaction();
try{
db.update(...);
db.setTransactionSuccessful();
}finally{
db.endTransaction();
}
Now the problem is that endTransaction occasionally throws SQLiteDatabaseLockedException, and I don't know reason, or how to repeat same exception.
From SQLiteDatabaseLockedException I read:
Thrown if the database engine was unable to acquire the database locks
it needs to do its job.
And from beginTransaction I read:
Begins a transaction in EXCLUSIVE mode.
From SQLite manual I read:
An EXCLUSIVE lock is needed in order to write to the database file.
Only one EXCLUSIVE lock is allowed on the file and no other locks of
any kind are allowed to coexist with an EXCLUSIVE lock. In order to
maximize concurrency, SQLite works to minimize the amount of time that
EXCLUSIVE locks are held.
So how can DB lock not be acquired in endTransaction when I hold exclusive lock from beginTransaction?
Android version where this happens is 4.0.4 (I have crash report, but not able to repeat this).
Need to say that I enabled SQLiteDatabase.enableWriteAheadLogging on the DB, maybe it matters? My app accessess DB in multiple threads.
Anyway, I'd like to get clear explanation, and make simple example that can repeat conditions repeating the problem, so that I can make real fix.
Thanks.
IMO your code is correct in a singular thread app, so it must be an enableWriteAheadLogging related issue. Maybe this can help:
SOURCE
... The maximum number of connections used to execute queries in
parallel is dependent upon the device memory and possibly other
properties.
...
Writers should use beginTransactionNonExclusive() or
beginTransactionWithListenerNonExclusive(SQLiteTransactionListener) to
start a transaction. Non-exclusive mode allows database file to be in
readable by other threads executing queries.
...
As I understand in serializable (default) mode of sqlite the locks refer not to different threads but to connections (this mode even knows nothing about threads). So if you are using the same connection (and all SqliteDatabase objects, produced by one sqliteOpenHelper instance share the same connection) across multiple threads you are totally unprotected.
Use either connection per thread or non-database synchronization locks, if you insist on multithreaded use of Sqlite. I would prefer a wrapper singleton with a single thread lock protecting its every transaction-like method. But this depends on your app specifics.
Please read this extensive answer and links in it for details and best practices for Sqlite multithreading.
I have an application that is doing a LOT of sqllite transactions, I currently have a bit of a hang because I am doing the sqllite actions on the UI thread... yes bad...
so I made each item have a thread and execute on it assuming sqllite api was smart enough to FIFO them.. nope ... now I get database is locked exceptions
this says it should work
without completely rewriting my code, and having a list of transactions queue up and execute them all on the same thread (many different classes, would be kind of a pain)
is there a way for me to check, and not execute a thread unless there isnt a lock? a lock check per se, or something similar that would get this to work, is efficient and isn't a huge rewrite?
Thanks
My answer that you quoted seems to be confusing. You don't have to do anything special when you are accessing the same Android database using the same database object with multiple threads. Under the covers, Sqlite has it's own locking to guarantee that the database will not be corrupted. To quote my answer;
Sqlite under Android is single threaded. Even if multiple threads were using the same database connection, my understanding is that they would be blocked from running concurrently. There is no way to get around this limitation
It has it's own locking which serializes the requests. This means that adding multiple threads will not increase the performance of the database unfortunately.
As my other answer mentions, you cannot use multiple database objects to the same database from multiple threads since there is no locking and you will corrupt your database.
My app requires high data throughput. It receives an incoming data stream over bluetooth and has to parse it, scale it, display it, and store the data.
After reading up on different storage methods, I've decided to try to use Sqlite for data storage. I've read up and it seems to get the best write performance, I should use transactions.
But before I even go there, I'm wondering if the DB handler should just be in it's own thread. The BT data processing is already in a separate thread and works well. I see a lot of discussions about accessing sqlite from multiple threads, but I'm thinking I want one thread handling the DB connection and just use intents to get/store data..mostly just to disconnect the display from the data storage.
Will this increase my performance, or is it not worth it?
I'm guessing I'll be writing 20-50 rows a second with up to 19 fields.
Remember that SQLite is totally memory-resident. There is no disk latency time to slow up processing. As such, I would first try to keep the architecture as simple as possible to avoid creating any unnecessary overhead.
I have a an application that has 2 parts.
A service which creates content.
An application that uses the content
Each of these run as different processes. The problem is that both of them share a database. And I frequently get database locked error, both when the service tries to write something and the UI is reading data. Also vice versa.
How do go about this?
The class used to access DB is a singleton class. But since both UI & the service are 2 different processes, there are 2 singletons I presume. So that doesn't help.
Even synchronise won't help I suppose, since again because of 2 different processes.
Content Providers maybe an option, but since I use complex queries to dig info, it would be really hard to use that too.
How do I get the two processes share the database.
Any cues would be greatly appreciated.
Using a content provider is one option. Another is to take a look at Berkeley DB. The BDB SQL API is SQLite compatible and the BDB lock manager allows multiple threads and/or processes to read/write to the database concurrently.
close the connection after each operation
catch the database locked error and try to reconnect after 50ms
or let the service handle the database and the activity ask the service for data
may be there is isDatabaseInUseMethod ?
You should use a content provider to funnel your database queries through one source. Inside of the content provider you can use any locking mechanisms you would like to ensure you're not having concurrent access. You may also think about using content observers to coordinate service actions with changes to the database.
The following is a great article on how locking works with SQLite on Android and what things to be aware of: http://kagii.squarespace.com/journal/2010/9/10/android-sqlite-locking.html
I would think you'll find some answers there :)