How to insert large amount of data in sqlite database in Android - android

I have a lot of data that is stored in a CSV file (about 20,100 rows), which I need to insert into a sqlite database.
This insert is taking very long to complete. What is the fastest way to insert this data?

As you have suggested, number of rows are huge I will recommend not to use AsyncTask, as its not tied to your Activity lifecycle i.e if you activity which started it dies, it doesnt mean AsyncTask dies as well, so if you try initiate a AsyncTask and somehow if your activity dies e.g screen rotation or back key pressed, upon restarting another AsyncTask will get spawned rather then it getting linked to already executing AsyncTask. hence duplicating same operations.
So, all in all I would recommend following approach
(A)
Create a IntentService, it's handleIntent() api already executes in a worker thread so you don't have to worry about any thing, and once all messaged in its queue are finished it automatically dies, so no worry at all about leaking any resources.
write your logic for inserting rows in bulk, use content resolver bulkInsert() api for same. I will recommend inserting in 100 roes per batch, you can implement rollback and error checks to make sure insert goes normally.
Once all insert is finish, you can post back to your UI using Handler and Messengers.
with all this you will achieve two major challenge
Not to hang up your UI, escaping any possible ANR
Even if back key is pressed, ensured that db operation goes on smoothly as it was taken up in background task.

Using AsyncTask<>, insert 20,100 rows inserts in database. Using this asynctask whole work run in background. For more information follow this link

The best solution would be using services and executor because as OP described, process can take a lot time. Thanks that You will be able to close app or move it to background with no worried Your long process is destroyed.
Using AsyncTask is not a good idea because it was designed for short operations as it is described on http://developer.android.com/reference/android/os/AsyncTask.html You must also be careful with using it. Changing orientation screen cause recreating view and also task of asynctask.
AsyncTasks should ideally be used for short operations (a few seconds
at the most.) If you need to keep threads running for long periods of
time, it is highly recommended you use the various APIs provided by
the java.util.concurrent package such as Executor, ThreadPoolExecutor
and FutureTask.

Related

Android loading string array into activity using background thread

I have an activity where the user enters a value in an EditText and I search a string array that I have defined in a xml file for a match. Each time the user changes the text I look for a match. When I start this activity I load the string array resource.
Should the loading of the array and the match finding occur in a background thread?
From what I understand I can use an AsyncTask which I am familiar with or a IntentService which I have no experience with. Would IntentService be overkill? What is ideal for this operation?
In some cases it is possible to accomplish the same task with either an AsyncTask or a Service however usually one is better suited to a task than the other.
AsyncTasks are designed for once-off time-consuming tasks that cannot be run of the UI thread. A common example is fetching/processing data when a button is pressed.
Services are designed to be continually running in the background. In the example above of fetching data when a button is pressed, you could start a service, let it fetch the data, and then stop it, but this is inefficient. It is far faster to use an AsyncTask that will run once, return the data, and be done.
If you need to be continually doing something in the background, though, a Service is your best bet. Examples of this include playing music, continually checking for new data, etc.
For the most part, Services are for when you want to run code even when your application's Activity isn't open. AsyncTasks are designed to make executing code off of the UI thread incredibly simple.
You should use AutoCompleteTextView and ContentProvider to do your implementation. Save your string array in database and access them by Cursor to popup and show in AutoCompleteTextView. There is an example available in the official document.

AsyncTask for longer than a few seconds? [duplicate]

This question already has answers here:
Android AsyncTask for long running operations
(4 answers)
Closed 9 years ago.
The API reference states,
AsyncTasks should ideally be used for short operations (a few seconds
at the most.)
Is the problem with a doInBackground that takes, say, 30 seconds that the thread pool might run out of threads? And if that's the reason, would it cease to be a problem if I ensure my app will never have more than one such long running doInBackground executing concurrently?
The answer given by #Walter Mundt is correct. Nevertheless, I would like to add a complement of information and give a pointer to a library that can be used for long running AsyncTask.
AsyncTasks have been designed for doing stuff in background. And, yes, it's right that if your AsyncTask lasts for two long, then you will face 2 different issues :
Activities are poorly tied to the activity life cycle and you won't get the result of your AsyncTask if your activity dies. Indeed, yes, you can but it will be the rough way.
AsyncTask are not very well documented. A naive, though intuitive, implementation and use of an asynctask can quickly lead to memory leaks.
RoboSpice, the library I would like to introduce, as proposed by #Walter Mundt, uses a background service to execute this kind of requests. It has been designed for network requests (potentially long running by nature), but it could be easily adapted to execute just long running tasks, unrelated to network. I would be glad to add a patch to it.
Here is the reason why AsyncTasks are bad for long running tasks. The following reasonning is an adaptation from exerpts of RoboSpice motivations : the app that explains why using RoboSpice is filling a need on the Android platform.
The AsyncTask and Activity life cycle
AsyncTasks don't follow Activity instances' life cycle. If you start an AsyncTask inside an Activity and you rotate the device, the Activity will be destroyed and a new instance will be created. But the AsyncTask will not die. It will go on living until it completes.
And when it completes, the AsyncTask won't update the UI of the new Activity. Indeed it updates the former instance of the activity that
is not displayed anymore. This can lead to an Exception of the type java.lang.IllegalArgumentException: View not attached to window manager if you
use, for instance, findViewById to retrieve a view inside the Activity.
Memory leak issue
It is very convenient to create AsyncTasks as inner classes of your Activities. As the AsyncTask will need to manipulate the views
of the Activity when the task is complete or in progress, using an inner class of the Activity seems convenient : inner classes can
access directly any field of the outer class.
Nevertheless, it means the inner class will hold an invisible reference on its outer class instance : the Activity.
On the long run, this produces a memory leak : if the AsyncTask lasts for long, it keeps the activity "alive"
whereas Android would like to get rid of it as it can no longer be displayed. The activity can't be garbage collected and that's a central
mechanism for Android to preserve resources on the device.
Progress of your task will be lost
You can use some workarounds to create a long running asynctask and manage its life cycle accordingly to the life cycle of the activity. You can either cancel the AsyncTask in the onStop method of your activity or you can let your async task finish, and not loose its progress and relink it to the next instance of your activity.
This is possible and we show how in RobopSpice motivations, but it becomes complicated and the code is not really generic. Moreover, you will still loose the progress of your task if the user leaves the activity and comes back. This same issue appears with Loaders, although it would be a simpler equivalent to the AsyncTask with relinking workaround mentionned above.
Using an Android service
The best option is to use a service to execute your long running background tasks. And that is exactly the solution proposed by RoboSpice. Again, it is designed for networking but could be extended to non-network related stuff. This library has a large number of features.
You can even get an idea of it in less than 30 seconds thanks to an infographics.
It is really a very very bad idea to use AsyncTasks for long running operations. Nevertheless, they are fine for short living ones such as updating a View after 1 or 2 seconds.
I encourage you to download the RoboSpice Motivations app, it really explains this in-depth and provides samples and demonstrations of the different ways to do some background operations.
If you are looking for an alternative to RoboSpice for non network related tasks (for instance without caching), you could also have a look at Tape.
I believe that AyncTasks are in general still tied to the foreground activity stack that spawned them, so that e.g. if an Activity spawns an AsyncTask, the user leaves the app, and then the OS is short of memory, it will kill the Activity's process (including the still-running AsyncTask), and just expect you to restore the state and start over if the user resumes/returns to your app.
For longer-running tasks, particularly the sort where there will only be only one or a few, you probably want a Service instead, because those can persist even when your app's UI is shut down to save memory.
Disclaimer: I haven't done Android coding in awhile, so this answer may be out of date or based on a flawed understanding of how things work. I will remove this caveat if someone with more recent experience can comment to confirm; high-rep folks are welcome to just edit this paragraph away instead if they know this is correct.

Android: How to retain AsyncTask instance when Activity gets destroyed?

In my app, I have a class that inherits from AsyncTask and which downloads huge amounts of data from the server. I am using a ProgressBar to indicate the progress of the download.
When the user hits the HOME key, the Activity to which this AsyncTask is attached, is destroyed but, download goes on.
How can I reattach this AsyncTask and show the progress to user? I tried using onRetainNonConfigurationInstance but Android 4.0 doesn't seem to invoke this method. My application does not use Fragments API.
What I did in this situation was as follows:
I created an IntentService to handle communication with the server. This has some of the benefits of AsyncTask (e.g., worker thread), but also some benefits of a Service (available any time, from anywhere).
The IntentService can be invoked either by a user action in my main Activity, or via an inexact repeating alarm.
The data is stored in an SQLite database, fronted by a ContentProvider. I dodge the issue of when/how to create my database and tables by using an SQLiteOpenHelper and calling getWritableDatabase() from the safety of my background IntentService.
When the task is done, it posts a Notification if my main Activity is not active.
One nice thing about this arrangement is, no progress bar is necessary. In fact, no UI is necessary. The user keeps using the application while the service is running, and the UI automatically refreshes itself as new data comes into the ContentProvider. Another nice aspect of it is it's less code to write than an AsyncTask. It automatically picks up where it last left off by reading the server-side metadata of the last entry from the database and asking the user to start after that point. Since it's not tied to any Activity instance, it doesn't care about onPostExecute() or configuration changes or any of that. And you don't have to worry about single-shot execution like AsyncTask.
If there is a need to download huge amount of data in background I would use service rather then AsyncTask. There is a good presentation from Google IO about using services.
Quote from AsyncTask documentation:
If you need to keep threads running for long periods of time, it is
highly recommended you use the various APIs provided by the
java.util.concurrent pacakge such as Executor, ThreadPoolExecutor and
FutureTask.
and
The task can be executed only once (an exception will be thrown if a second execution is attempted.)
As I understand, you cannot proceed with your last AsyncTask.
Still, you can load your data partially and save amount of data read and then start new AsyncTask which will start from last saved point. From my point of view this is not the best idea to pause loading when activity goes to background and it is better to use service to finish what was started.
Have you considered using a service to attach your AsyncTask to? Seeing as a permanently running service would probably be the best solution for your task at hand. All you'd have to do then will be to check if the service is running and if your download is running (easily done using static boolean variables) then you just create a progress dialog using some state saving variable in your service (maybe a percentage of the total file size downloaded etc.) in the onCreate method of your main activity.

Concurrent execution of two SQLite transactions in two threads

I have a certain update method in my Android App, which generates quite an amount of data - up to hundreds of database entries.
I also have a background service, aside from the UI thread. Both threads have to execute the update method, sometimes even at the same time - basically, this is about generating and caching data to display. Both the UI and background service need this data.
Currently, I have wrapped the method's execution in an ORMLite transaction, which maps to an ordinary SQLite transaction. However, I am afraid that this will bite me in the butt one day, when some race condition screws up the data cache.
The question: Do SQLite transactions protect me from concurrent execution, or should I rather implement some kind of worker thread which is spawned when the generator-method shall start, or blocking if the generator-method is already running?
UPDATE:
I have decided to not rely on SQLite logic for the protection of my high-level Java method. The solution was for me as follows:
Wrap the generating part of the method with synchronized
Introduce a variable which tracks the last time of executing the method (set at the end of the method, so it is the marker of execution END)
First thing in the synchronized section, check if the last execution is in a specific threshold (e.g. <= 100ms in the past)
If yes, skip generation
If no, perform generation
In this way, duplicate generation should not take place, since when the method is accessed from two threads at the same time, the first will generate, but the second will not. The most important part for me here is that it is still blocking, since both threads rely on the generation having taken place after they have called the method.
EDIT:
It seems I'm wrong in my below statement: The SQLite implementation is, according to many, thread safe. I have, however, bitterly experienced threading issues, especially when testing database access, but that must have been caused by other factors in my code then, I assume.
Sorry for the misleading answer.
ORIGIN:
Good question!
You should be very careful here because the standard Android database access objects (such as SQLiteDatabase, Cursor etc) are not thread-safe by default. Not even ContentProvider's seem to give you a complete protection unless you explicitly write them with multithreading in mind.
According to Android documentation on ContentProvider's and threading (almost at the end of the page):
"Because these methods [update() is one of the functions] might be called from any number of threads at the same time, they too must be implemented to be thread-safe."
I don't know if there is any explicit locking mechanism to SQLiteDatabases (as in locking the actual database file). I would assume that a transaction itself would lock, at least the very handle you access your database with. I don't know what is true for the case where you have multiple handles to your database.
Maybe you could try to implement some singleton object (A ContentProvider maybe?) to access your database with, but even then you'd have to manage some sort of "request queue" I suppose.
You should also consider not to make any calls to the file system (the database is on the file system) from the UI-thread, what-so-ever. There is no guarantee that the database will answer in time and you're likely to end up with an ANR (especially as you write "...which generates quite an amount of data").

When should I do certain SQLite operations on another thread(not the main thread)?

My Android application includes an SQLite database with an SQLiteOpenHelper class to help manage it. During application use, the user may perform some operations such as adding/deleting/updating etc on the database.
At some points the size of the operation will be known, like this:
user clicks button to save item
the SQLiteDatabase performs a single insert query
user continues using app
At other areas of the app, the operation may be large, like inserting 10+ items into the database all at once.
Questions:
should I thread simple operations like inserting/updating/deleting/viewing 1 item?
will it take longer to insert 1 item into a table which contains many items(like 30+) than it would take to insert into a table with no items?
if i don't need to thread such simple operations, at what point do you suggest i start threading them?
when i say thread i mean using a thread that is not the main UI thread.
edit: I realize that small operations do not take much time and i could very well get away with doing them on the main thread. I am just concerned that it would be bad practice to be executing them on the main thread and would like clarification!
General rule for everything: If it's fast enough, do it on the main thread. If not, use a worker thread.
Unless you have a ridiculously huge database, a single operation almost never warrants a separate thread. Databases in general are designed to scale well, but of course a very big database (10,000+ rows?) will be a bit slower than a small one. 30 rows, however, is nothing.
I would start threading stuff if you have a lot of operations going on, like a bunch of queries, or complicated queries that span several tables.
As with everything - profile your app, and if it's too slow, optimize. Don't write an awesome synchronized super-duper multi-core-ready database handler if none of your queries takes longer than 2ms.
Always measure before you optimize!
Make sure that DB operations you do affect user experience and than start looking for a solution.
If database stuff gets slow, then use AsyncTask, which was designed to perform tasks in the background, and then update the GUI on EDT.
There is absoulutely not reason to use a thread here. Just return the cursor, extract the information from the cursor and return it to the main activity.
Specifically speaking a thread is something ideally that is going to repeat until something happens or it times out. Since the database you are using i'm assuming is on the phone, it would take practically zero time to access it.
Also another thing you can do is create a Utility class to assist with your activity to database interaction. It would be what your activity calls to interact with the database. Specifically the flow of control would be like this:
Activity -> Utility -> Database
Its between the activity and the database to keep them isolated from each other and make it much easier to access whatever it needs since it doesn't have to go directly to the database itself.

Categories

Resources