I am developing a multithreaded server that stores and reads information from a db. The db is implemented with RocksDB.
The problem i am having is that when i access the db from more than one thread at a time, i get that error.
Usually, it ment that the db wasn't deleted after usign it, but now it doesn't make sense, of course it isn't deleted, it's being used by another thread.
I know that the db itself is accessed in secuential order, so tecnically you will never get two threads reading it at the same time, but at least one of them should wait until the lock is released and access it then
anyone has any idea what might happen?
The problem was that i had 2 rocksdb::DB* objects in different threads with the same db name and was opening the db from both threads at the same time
Related
I have an Android Application that holds a fairly large Database which is accessed from multiple threads. I am starting to get Database locked Exceptions. My question, is there a way of creating a database Queue (similar to FMDataBase.Queue in iOS)?
I have done some research and I don't want to create a helper class as a lot of my database queries and inserts are different so creating a method for each query is not feasible.
I could put
if(!db.isLocked)
{
//exec(BLAH)
}
else
{
//try{thread.sleep(1000);}... then some retry code
}
on every database function but there must be a better way than this.
moreover if the thread is locked after is has slept for 1000ms, it will still crash and if the db is locked for 10ms I will be waiting for 990ms for the query to run, which is not great for user experience.
Is is possible to create a Queue so that any commands sent to the db will be executed once the the db becomes unlocked?
Any advice would be greatly appreciated
My question, is there a way of creating a database Queue
Move all your database writes to an IntentService, a regular service hosting a LinkedBlockingQueue, a LinkedBlockingQueue held by your singleton SQLiteOpenHelper, etc.
a lot of my database queries and inserts are different so creating a method for each query is not feasible.
The overhead to wrap a database I/O in a method not especially much, less than what you were proposing doing in your question.
if the db is locked for 10ms I will be waiting for 990ms for the query to run, which is not great for user experience
Tune your database access (e.g., EXPLAIN keyword, then set up appropriate indices). Use Traceview to determine exactly where your problems lie. If your problems are in a database transaction (e.g., you open your own transaction and do a ton of work inside of it), consider using yieldIfContendedSafely().
I need to call multiple api's each executing in separate thread and insert the respective data from response to sqlite database without causing locks. Can anyone help me in this regard with a working example that I can refer to.
I need to call multiple api's each executing in separate thread and insert the respective data from response to sqlite database without causing locks.
You don't need to do anything special. Multiple threads can make use of the same SQLiteDatabase object without you doing locking at the application level. Sqlite does it's own locking under the covers. You should never get deadlocks but one thread will have to wait for the other thread to finish making its insert.
See these questions/answers:
Can two AsyncTasks share the same SQLiteDatabase object?
What are the best practices for SQLite on Android?
Sqlite under Android is single threaded. Even if multiple threads were using the same database connection, my understanding is that they would be blocked from running concurrently. There is no way to get around this limitation. If you open two connections to the same database, this would corrupt the database because database updates would not be coordinated.
I am trying to understand the possible ways to work with SQLite when there can be multiple threads work on DB.
Based on various responses in stackoverflow and other sites, it appears that there will be locking issue when same sqlitehelper instance is used from multiple threads. In a typical java application, I would expect instance to mean single object of type sqlite helper to be used by different threads of application.In such cases, the locks ,I guess, are a matter of correctly using the synchronized blocks. [Correct me here as I am not comfortable with this way of looking at sqliethelper instance here]
My concern is with sharing same data base : when one instantiate sqlite helper in different threads [ie each thread has its own object instance] but working on same Database [this I guess is more inline with having same db instance].
In such cases I'm getting frequent database lock errors. This occurs even when the threads are working on different tables of database.
In my application database can be updated by user interaction through application or by getting data through server [periodic synchronization]. And some time when synchronization process and user activity overlaps, I get the lock issues. As this pattern of data processing seems to be common in application synchronizing with server, would like to know how do lock issue due the concurrency is to be handled.
I would like to understand this since if this is bound to happen always then probably need to make only one handler over database and implement queue over that to avoid lock. But that will mean the complete application needs to be aware that the database may not get updated immediately and they need to implement listener to know when the data is actually updated in database.
thanks
pradeep
As far as I know sqlite is intended for single process usage. No matter what you will always need to access the database from one thread at a time. You can do selects from multiple clients but can only write from one at a time. And other readers and writers will ahve to lock in the mean time.
As a side note - database access can hardly ever be considered instantaneous.
I have an app (Android 2.2 Google API Level 8) that has multiple activities pulling data from a content provider (SELECT only database access).
It also has a service with a central blocking task queue accepting any database write tasks; activities can fire a service request (As intent) which places a task on a blocking queue for sequential retrieval by a single thread and execution. Database is around 4mb.
There is a single database helper which the service uses to call methods to interact with the database including writing to it; all SQL writes are carried out within the database helper.
All database writes are surrounded by a transaction.
All database reads have the cursor closed at the end of the method.
None of the Activities has a handle to the database object, they can only communicate via the content provider or the service.
Any AlarmManager fired tasks - like Activities - only use the service to pop an appropriate task onto the queue.
The service is the only class that has a handle to the database helper.
All database writes are only carried out via a task placed on a queue; I have exhaustedly checked that task execution is sequential being well aware of it being essential to avoid concurrent writes to an SQLite database.
During a run of task executions I consistently get one or two "database is locked" errors on attempting to write to the database triggered by a tasks execution of 'begin transaction'.
In attempting to track down the source of the lock I found that using dbhelper.inTransaction(), dbhelper.isLockedByThisThread(), dbhelper.isLockedByOtherThread() didn't help as they wouldn't indicate an unexpected database lock.
What I did find that worked in detecting a lock early was to create a method with beginTransaction() and setTransactionSuccessful without any actual SQL write code, within a try catch block that would log the issue - always triggered by beginTransaction().
I placed this database lock trap either side of each of the blocking queue task methods in the expectation/hope that I would find a singular culprit that was leaving the database in a locked state after finishing.
I could not find a consistent culprit. After drilling down through from the start of the task call through to the database write I found that a database lock could occur seemingly out of the blue without having been locked by the previously run task (All these tasks run in sequence under the same singular thread).
After looking at a number of other peoples experiences with database locking issues I've tried closing the database connection directly after the transaction has completed on all tasks but this didn't help, if anything seemed to get more database locking occurrences. Tried added a sleep between each task execution; not exhaustively tested but generally found that a delay of 3 seconds or above seemed to stop the database locks appearing. Tried disabling alarm manager fired tasks - didn't make any difference.
Impression I have is that some form of maintenance task external to my application is dropping in and locking the database periodically - perhaps delayed writing of logs. Obviously I'm less than keen on setting a task processing delay so I'm considering having a database lock retry task queue to reattempt database writing as necessary; much prefer to resolve but am running out of ideas.
Can anyone think of some principle or gotcha I've missed?
Is it in reality normal within Android and larger SQLite databases that you'll get occasional database locks?
Thanks
SQLite guarantees sequential access from multiple threads as long as you use a single database connection. How and where are you opening and closing the database connection?
I generally recommend opening the database once on startup, and never closing it. There's no benefit to closing, since the transactional nature of SQLite means that writes are flushed to persistent storage as soon as possible anyway.
I would check if some activity that calls the DB or calls other activity that calls the DB, has only one instance. Otherward it can lock itself, in some sense.
With regards to
Is it in reality normal within Android and larger SQLite databases that you'll get occasional database locks?
No, it is definitely not normal to get occasional database locks. From reading your story you say that you have both a service and a content provider pulling from the database, so it is possible that you are locking the database between the two accesses.
What I generally do is ensure that I handle all of my database access through the content provider. By having a single point of entry to the database you can ensure that every software component is using the same logic to access the DB. Would it be possible to have your service access the DB through the content provider?
It's also good to remember that by placing your DB behind a content provider, it can still be accessed by multiple threads at once. To ensure that you are accessing the DB only one thread at a time you could place synchronized constructs on the DB inside of your content provider. Obviously if you are doing lots of long writes/reads to the DB, locking in this fashion will absolutely destroy your app. Putting all of your DB code inside of the content provider will also give you a single point of debugging which would help you figure out if multiple threads are accessing the DB.
We all learn that resources, such as database connections, should be acquired late and released early.
Yet applying this principle to SQLite database connections on Android have caused me some headache.
I have an app that download updates from a backend server in a service working in the background, writing updates to the database at a regular basis. The problem I experience arise when the following sequence occurs:
Service opens a writable database connection
Some activity opens a readable database connection
Service closes its database connection concurrently with the activity reading data
Activity fails due to its database connection was closed
Both the service and the activity uses the same SQLiteOpenHelper class, though different instances, to open their connections. My initial assumption was that this should work just fine, but somehow it seems that the underlying connection is shared between the two database instances.
To work around the problem I ended up not closing the database objects, only closing any opened cursors. This seems to work, though I'm not sure that I'm not leaking memory here.
Is there something obvious I am missing here?
Is there something obvious I am missing here?
I'd say no. Looking at the source code to SQLiteOpenHelper, I can't see how two instances could be sharing a SQLiteDatabase object.
Some diagnostic suggestions:
Dump the toString() value of each SQLiteDatabase, which should give you a Java instance ID. If they are the same, that is where your problem lies, and you will need to work your way upstream to figure out how the heck this is happening (e.g., you really are using the same instance of the SQLiteOpenHelper).
With your database in a stable state (i.e., no need to create or upgrade), flip one of your two database spots to use SQLiteDatabase directly, rather than via SQLiteOpenHelper, and see if that changes matters.