Loading a large array of shorts in Android - android

Suppose I have an array of shorts of length 1,000,000 and that I need this to be generated or loaded into RAM (ideally within a few seconds) when my app starts and before I get an activity thread timeout.
I'll then have quick access to its entries during runtime.
How would you go about loading this to memory from file? Loading from txt file? From an SQLite file (with two integer columns in a single table, one for index and the other for value)?
Is it possible for an activity to request a longer idle time before it's deemed to have timed out?

The solution to your problem would be to spawn a thread to do the file reading. This is going to be a high level overview so you can track down the relevant items. It will address item 2 first, then item 1.
First, you need to get your long running processing off the UI thread. The 'idle' time responsiveness issue (application not responding) is because you are tying up the UI thread which is needed to do other actions. The thread cannot be reading a file and drawing the screen at the same time (at least not well).
I've found the easiest way to do this is using the Java ExecutorServices. What you want to do is package the logic for reading the file or database into a java Runnable or Callable and then run that using an ExecutorService. The executor will take care of starting a thread and reclaiming those resource when they are no longer in use.
One key issue to be aware of is that, once you create another thread, you need to be careful when updating the UI. Because ui objects are not thread safe, you can only update the ui from the UI thread (common sense, right?). If you are in an activity, you can do this by calling runOnUiThread() or you can create your own Handler in one of the ACtivity methods called by the UI thread in the first place.
Sam Dufel said:
Hmm... You could really pack that down if you could come with a format to replace all the dummy entries with just a count. Eg, 10,000 zeros get replaced by a pair of flag bytes followed by 10,000
Additionally, you may see increased speed by spawning multiple read threads. If you could break the file up into two files which would allow two threads to execute simultaneously you could see a significant speed up (depending on what else you need to do to the data). A sort of divide an conquer for loading up your data from file. A cursory google search should give you information on how to load shorts from a file. You should most definitely consider using a binary format given your specifications.

Related

How to insert large amount of data in sqlite database in Android

I have a lot of data that is stored in a CSV file (about 20,100 rows), which I need to insert into a sqlite database.
This insert is taking very long to complete. What is the fastest way to insert this data?
As you have suggested, number of rows are huge I will recommend not to use AsyncTask, as its not tied to your Activity lifecycle i.e if you activity which started it dies, it doesnt mean AsyncTask dies as well, so if you try initiate a AsyncTask and somehow if your activity dies e.g screen rotation or back key pressed, upon restarting another AsyncTask will get spawned rather then it getting linked to already executing AsyncTask. hence duplicating same operations.
So, all in all I would recommend following approach
(A)
Create a IntentService, it's handleIntent() api already executes in a worker thread so you don't have to worry about any thing, and once all messaged in its queue are finished it automatically dies, so no worry at all about leaking any resources.
write your logic for inserting rows in bulk, use content resolver bulkInsert() api for same. I will recommend inserting in 100 roes per batch, you can implement rollback and error checks to make sure insert goes normally.
Once all insert is finish, you can post back to your UI using Handler and Messengers.
with all this you will achieve two major challenge
Not to hang up your UI, escaping any possible ANR
Even if back key is pressed, ensured that db operation goes on smoothly as it was taken up in background task.
Using AsyncTask<>, insert 20,100 rows inserts in database. Using this asynctask whole work run in background. For more information follow this link
The best solution would be using services and executor because as OP described, process can take a lot time. Thanks that You will be able to close app or move it to background with no worried Your long process is destroyed.
Using AsyncTask is not a good idea because it was designed for short operations as it is described on http://developer.android.com/reference/android/os/AsyncTask.html You must also be careful with using it. Changing orientation screen cause recreating view and also task of asynctask.
AsyncTasks should ideally be used for short operations (a few seconds
at the most.) If you need to keep threads running for long periods of
time, it is highly recommended you use the various APIs provided by
the java.util.concurrent package such as Executor, ThreadPoolExecutor
and FutureTask.

Use AsyncTask or Thread for frequent read/write operations - Android

I have built an Android app that allows a user to create new levels for a game I am making. The level maker app allows the user to save new levels and preview all levels they have saved. When the user saves a level, my app generates a JSONObject that represents that level, then writes the JSONObject.soString() value to a local file (either a Shared Preferences file or a plain old .txt file - I don't know which is better yet). When the user wants to preview existing levels, the app reads in all the JSON Strings from my levels file, converts each JSON to a Bitmap (JSON -> ViewGroup -> Bitmap), then displays all the Bitmaps in a ListView.
Right now I am using 2 AsyncTasks: one for writing and one for reading. I don't know if I could potentially be spawning 5-10 new Threads per minute, though (one for each time the user saves a new level), and I don't know if that would be a bad thing as far as system resources goes.
Here's my question: because I may need to write to my levels file 5-10 times per minute, is it better to start a new AsyncTask each time the user saves a new level, or should I just create one background thread and execute Runnables on it each time the user saves or previews levels?
AsyncTask is controlled internally by a shared (static) ThreadPoolExecutor and a LinkedBlockingQueue. There are such parameters like core pool size and maximum pool size. Since Android 1.6, core pool size is 5, and the maximum pool size is 128. Queue size is 10 tasks.
So, maximum quantity of parallel AsyncTasks is 5, others will be queued. If your save operation is relatively slow, you may encounter situation when AsyncTask got killed by OS and nothing saved. If you need long-running save operation, it is better to use service.
For me, I prefer using background service with my own ThreadPoolExecutor - this gives me some extra control like cancelling tasks. If you don't want to write extra code, you may use IntentService - it also uses Executor and queues your tasks by itself.

Android Threading - Executing One thread before all others are executed

I'm new to Android programming and I have a threading issue.
I'm basically populating a GridView with images from 50 or so URLs, but those URLs will not be known until I retrieve a JSON object from an already known URL. I know that I have to fire off a bunch of threads for each URL download (using the AsyncTask class).
How can I effectively queue these threads so that the JSON thread is executed and finished first, so I can use data retrieved from that thread when I fire off those 50 image threads immediately after?
Do not start 50 threads. Use a single thread. While this should be separate from the UI thread for responsiveness, there is no need to spawn multiple threads, and certainly not one thread per URL.ce
Simply make your JSON network call, then parse the response, then (in the same thread) loop through the URLs requesting each one, and decoding the result into a Bitmap. You'd them add them within some model object to the Adapter for your GridView, which would automatically trigger the GridView to update on the UI thread.
There are better practices here, such as lazy loading, caching, and displaying a placeholder image while images are loading, but the exact implementation becomes too complex to describe here. Search for WeakReferenceMap and LruCache to find examples of the best practices for dynamic image loading into an AdapterView.
Well, you could just run the first fetch and when this is done, fire the 50 threads from this main thread. As Android 3.0 and later will kill your app when you do network communication from the UI thread, an AsyncTask could be a way for to fetch the JSON. When this returns it could fire the other threads.
One thing you should still consider is that Android is, as powerful as it is, still a device with limited capabilities. 50 Threads may use more resources that the target handset has and thus your app may be killed by the system (e.g. because of OOME). So wile on a desktop 50 threads don't sound much, they are much on a phone.
Also IIrc, there is a limit in the http spec, that one may only have 4 (?) simultaneous connections to one remote server. So consider queuing up the image loading requests and fetching them one after the other.

Android use Parcelable to send data between activities

In my application I want to pass an ArrayList of Parcelable objects (or a Parcelable[]) between two activities. On the first activity i call a restservice to get the json data, after that I use gson to get the List of Parcelable object and then i send the list to the second activity using putParcelableArrayListExtra(...). All works fine until the size of the list is about 1000, but over this size I get an ANR and application won't resume even if I wait for some minutes.
Is there a solution for this issue maintaining this approach? Is it due to a size limit for object put to an Intent?
I know I could pass the Json String and then get the list on the second activity, but I prefer to make first this kind of operations.
First off, ensure that the source of your ANR is indeed the large parcelable object, and not because you were performing a blocking operation (i.e. networking) on the main thread.
Now if the ANR issue really is due to these large objects, the right way to get this done is to deserialize the object and write it out to storage instead of passing it between activities. You're effectively doubling the amount of memory you incur by doing what you're doing.
Here are a few ways you can troubleshoot this:
StrictMode: StrictMode is most commonly used to catch accidental disk or network access on the application's main thread, where UI operations are received and animations take place. By keeping your application's main thread responsive, you also prevent ANR dialogs from being shown to users.
Traceview: Traceview is a graphical viewer to see logs created by an Android application. Via Traceview you can find errors in your application and measure its performance.
Memory Dump: You can create a memory snapshot and analyse it with the Eclipse Memory Analyzer.

Concurrent execution of two SQLite transactions in two threads

I have a certain update method in my Android App, which generates quite an amount of data - up to hundreds of database entries.
I also have a background service, aside from the UI thread. Both threads have to execute the update method, sometimes even at the same time - basically, this is about generating and caching data to display. Both the UI and background service need this data.
Currently, I have wrapped the method's execution in an ORMLite transaction, which maps to an ordinary SQLite transaction. However, I am afraid that this will bite me in the butt one day, when some race condition screws up the data cache.
The question: Do SQLite transactions protect me from concurrent execution, or should I rather implement some kind of worker thread which is spawned when the generator-method shall start, or blocking if the generator-method is already running?
UPDATE:
I have decided to not rely on SQLite logic for the protection of my high-level Java method. The solution was for me as follows:
Wrap the generating part of the method with synchronized
Introduce a variable which tracks the last time of executing the method (set at the end of the method, so it is the marker of execution END)
First thing in the synchronized section, check if the last execution is in a specific threshold (e.g. <= 100ms in the past)
If yes, skip generation
If no, perform generation
In this way, duplicate generation should not take place, since when the method is accessed from two threads at the same time, the first will generate, but the second will not. The most important part for me here is that it is still blocking, since both threads rely on the generation having taken place after they have called the method.
EDIT:
It seems I'm wrong in my below statement: The SQLite implementation is, according to many, thread safe. I have, however, bitterly experienced threading issues, especially when testing database access, but that must have been caused by other factors in my code then, I assume.
Sorry for the misleading answer.
ORIGIN:
Good question!
You should be very careful here because the standard Android database access objects (such as SQLiteDatabase, Cursor etc) are not thread-safe by default. Not even ContentProvider's seem to give you a complete protection unless you explicitly write them with multithreading in mind.
According to Android documentation on ContentProvider's and threading (almost at the end of the page):
"Because these methods [update() is one of the functions] might be called from any number of threads at the same time, they too must be implemented to be thread-safe."
I don't know if there is any explicit locking mechanism to SQLiteDatabases (as in locking the actual database file). I would assume that a transaction itself would lock, at least the very handle you access your database with. I don't know what is true for the case where you have multiple handles to your database.
Maybe you could try to implement some singleton object (A ContentProvider maybe?) to access your database with, but even then you'd have to manage some sort of "request queue" I suppose.
You should also consider not to make any calls to the file system (the database is on the file system) from the UI-thread, what-so-ever. There is no guarantee that the database will answer in time and you're likely to end up with an ANR (especially as you write "...which generates quite an amount of data").

Categories

Resources