I'm processing a lot of XML data that validates the local data storage within an AsyncTask object. First I tried to use transactions for these operations but while a transaction is in progress any other actions by the user will make the app freeze and wait for the transaction to finish, and sometimes even make the app stop responding.
The transactions are divided into several steps sometimes counting to a couple of hundreds per iteration. Because of the problems I went from using transactions to real-time queries which is very slow but solves the freezing - resulting in a very battery consuming application.
My question is; Is there a way to stop the transactions from locking the database? Or is my problem a result of poor preparation before the transactions?
Transactions are tend to lock your table(s) while doing their business, so there's no way that you can play transactions and non-transactional queries on a same instance at same time.
However, what you need to do is to process your data (xml) first (which might be time consuming) and later kick-in transaction once you've got the data ready.
P.S. I would personally suggest you to make use of transactions (in an efficient way) when it comes to insert multiple records because it creates a single Journal-file to handle all the insertion and speeds up SQLite operations a lot.
make your transactions smaller and don't forget (like I did) to still end transaction if exception thrown - use try, catch, finally...
Related
I have a lot of data that is stored in a CSV file (about 20,100 rows), which I need to insert into a sqlite database.
This insert is taking very long to complete. What is the fastest way to insert this data?
As you have suggested, number of rows are huge I will recommend not to use AsyncTask, as its not tied to your Activity lifecycle i.e if you activity which started it dies, it doesnt mean AsyncTask dies as well, so if you try initiate a AsyncTask and somehow if your activity dies e.g screen rotation or back key pressed, upon restarting another AsyncTask will get spawned rather then it getting linked to already executing AsyncTask. hence duplicating same operations.
So, all in all I would recommend following approach
(A)
Create a IntentService, it's handleIntent() api already executes in a worker thread so you don't have to worry about any thing, and once all messaged in its queue are finished it automatically dies, so no worry at all about leaking any resources.
write your logic for inserting rows in bulk, use content resolver bulkInsert() api for same. I will recommend inserting in 100 roes per batch, you can implement rollback and error checks to make sure insert goes normally.
Once all insert is finish, you can post back to your UI using Handler and Messengers.
with all this you will achieve two major challenge
Not to hang up your UI, escaping any possible ANR
Even if back key is pressed, ensured that db operation goes on smoothly as it was taken up in background task.
Using AsyncTask<>, insert 20,100 rows inserts in database. Using this asynctask whole work run in background. For more information follow this link
The best solution would be using services and executor because as OP described, process can take a lot time. Thanks that You will be able to close app or move it to background with no worried Your long process is destroyed.
Using AsyncTask is not a good idea because it was designed for short operations as it is described on http://developer.android.com/reference/android/os/AsyncTask.html You must also be careful with using it. Changing orientation screen cause recreating view and also task of asynctask.
AsyncTasks should ideally be used for short operations (a few seconds
at the most.) If you need to keep threads running for long periods of
time, it is highly recommended you use the various APIs provided by
the java.util.concurrent package such as Executor, ThreadPoolExecutor
and FutureTask.
I'm currently working on an Android application that requires reading from call history and text history. And for further analysis, I've extracted these the huge amount of entries from the corresponding content provider, created and inserted all of them to a SQLite database.
But the problem I've encountered is when this is running on a phone that has been used for years (meaning there's an enormous amount of data generated from calls and texts), the retrieval process and database building process takes too much time and may even cause the app to crash. Even if i tried to put these process in a AsyncTask, the problem still exists. So my question is:
Am i doing it in a good way to just put any time consuming operations away from Main UI, OR What's a better way, if any, to handle very very large amount of data in Android?
Use pagination logic. Fetch only the most recent and relevant data and load older data if the user requests it.
Call history on most android phones is limited to 500 entries CallLog.Calls, while SMS provider has not such limits, you can query the count of the tables and limit your queries to 50 at a time and pass it to a separate thread for processing. Also make sure you run this in a background service with low priority so as to not disturb any other operations ongoing in the device.
I have an application that is doing a LOT of sqllite transactions, I currently have a bit of a hang because I am doing the sqllite actions on the UI thread... yes bad...
so I made each item have a thread and execute on it assuming sqllite api was smart enough to FIFO them.. nope ... now I get database is locked exceptions
this says it should work
without completely rewriting my code, and having a list of transactions queue up and execute them all on the same thread (many different classes, would be kind of a pain)
is there a way for me to check, and not execute a thread unless there isnt a lock? a lock check per se, or something similar that would get this to work, is efficient and isn't a huge rewrite?
Thanks
My answer that you quoted seems to be confusing. You don't have to do anything special when you are accessing the same Android database using the same database object with multiple threads. Under the covers, Sqlite has it's own locking to guarantee that the database will not be corrupted. To quote my answer;
Sqlite under Android is single threaded. Even if multiple threads were using the same database connection, my understanding is that they would be blocked from running concurrently. There is no way to get around this limitation
It has it's own locking which serializes the requests. This means that adding multiple threads will not increase the performance of the database unfortunately.
As my other answer mentions, you cannot use multiple database objects to the same database from multiple threads since there is no locking and you will corrupt your database.
I need to store simultaneous readings from 10 sensors into a SQLite database at about 100Hz. Which means I need to store about 1000 readings a second into the database.
I've tried putting the database insert method in a Service running in its own process but I can only collect about 3.5 seconds worth of data. I think the Service is being killed at that point.
I am not using database transactions to insert the data. Just individual calls of INSERT INTO... Is it crucial to use transactions?
Is it necessary to run the database inserts in a separate process? Can I just run them in a new thread off the main process?
I can provide code if needed.
You need to use transactions, and "chunk" your requests by performing multiple inserts in the same transaction. You'll have to play with the number of inserts, but I would start at 1000 inserts per transaction, and tweak it from there.
Transactions are extremely slow, if you want to insert one record at a time. The reason for this is that SQLite waits for the data to be written to disk before completing the transaction, and so has to wait for the disk platters to spin completely around, during which time the disk is essentially inactive.
On a 7200 RPM hard drive, this essentially limits transactions to 60 per second.
http://www.sqlite.org/faq.html#q19
I need to store simultaneous readings from 10 sensors into a SQLite database at about 100Hz
If you are referring to Android's sensors, I am uncertain you will actually get data delivered to you that quickly.
I've tried putting the database insert method in a Service running in its own process
Do not use a separate process. You are adding overhead for no benefit.
Do your database I/O from whatever component is registered with your sensors. If that's a service, fine.
I can only collect about 3.5 seconds worth of data. I think the Service is being killed at that point.
Then you have bigger problems. Processes containing running services will not be killed 3.5 seconds after launch. Somehow, your service implementation is messed up.
I am not using database transactions to insert the data
As Mr. Harvey points out, transactions are essential for performance.
Is it crucial to use transactions?
Yes.
Is it necessary to run the database inserts in a separate process?
Not only is it not necessary, it borders on the ridiculous.
Can I just run them in a new thread off the main process?
Yes.
I am trying to understand the possible ways to work with SQLite when there can be multiple threads work on DB.
Based on various responses in stackoverflow and other sites, it appears that there will be locking issue when same sqlitehelper instance is used from multiple threads. In a typical java application, I would expect instance to mean single object of type sqlite helper to be used by different threads of application.In such cases, the locks ,I guess, are a matter of correctly using the synchronized blocks. [Correct me here as I am not comfortable with this way of looking at sqliethelper instance here]
My concern is with sharing same data base : when one instantiate sqlite helper in different threads [ie each thread has its own object instance] but working on same Database [this I guess is more inline with having same db instance].
In such cases I'm getting frequent database lock errors. This occurs even when the threads are working on different tables of database.
In my application database can be updated by user interaction through application or by getting data through server [periodic synchronization]. And some time when synchronization process and user activity overlaps, I get the lock issues. As this pattern of data processing seems to be common in application synchronizing with server, would like to know how do lock issue due the concurrency is to be handled.
I would like to understand this since if this is bound to happen always then probably need to make only one handler over database and implement queue over that to avoid lock. But that will mean the complete application needs to be aware that the database may not get updated immediately and they need to implement listener to know when the data is actually updated in database.
thanks
pradeep
As far as I know sqlite is intended for single process usage. No matter what you will always need to access the database from one thread at a time. You can do selects from multiple clients but can only write from one at a time. And other readers and writers will ahve to lock in the mean time.
As a side note - database access can hardly ever be considered instantaneous.