I need to store simultaneous readings from 10 sensors into a SQLite database at about 100Hz. Which means I need to store about 1000 readings a second into the database.
I've tried putting the database insert method in a Service running in its own process but I can only collect about 3.5 seconds worth of data. I think the Service is being killed at that point.
I am not using database transactions to insert the data. Just individual calls of INSERT INTO... Is it crucial to use transactions?
Is it necessary to run the database inserts in a separate process? Can I just run them in a new thread off the main process?
I can provide code if needed.
You need to use transactions, and "chunk" your requests by performing multiple inserts in the same transaction. You'll have to play with the number of inserts, but I would start at 1000 inserts per transaction, and tweak it from there.
Transactions are extremely slow, if you want to insert one record at a time. The reason for this is that SQLite waits for the data to be written to disk before completing the transaction, and so has to wait for the disk platters to spin completely around, during which time the disk is essentially inactive.
On a 7200 RPM hard drive, this essentially limits transactions to 60 per second.
http://www.sqlite.org/faq.html#q19
I need to store simultaneous readings from 10 sensors into a SQLite database at about 100Hz
If you are referring to Android's sensors, I am uncertain you will actually get data delivered to you that quickly.
I've tried putting the database insert method in a Service running in its own process
Do not use a separate process. You are adding overhead for no benefit.
Do your database I/O from whatever component is registered with your sensors. If that's a service, fine.
I can only collect about 3.5 seconds worth of data. I think the Service is being killed at that point.
Then you have bigger problems. Processes containing running services will not be killed 3.5 seconds after launch. Somehow, your service implementation is messed up.
I am not using database transactions to insert the data
As Mr. Harvey points out, transactions are essential for performance.
Is it crucial to use transactions?
Yes.
Is it necessary to run the database inserts in a separate process?
Not only is it not necessary, it borders on the ridiculous.
Can I just run them in a new thread off the main process?
Yes.
Related
I'm currently working on an Android application that requires reading from call history and text history. And for further analysis, I've extracted these the huge amount of entries from the corresponding content provider, created and inserted all of them to a SQLite database.
But the problem I've encountered is when this is running on a phone that has been used for years (meaning there's an enormous amount of data generated from calls and texts), the retrieval process and database building process takes too much time and may even cause the app to crash. Even if i tried to put these process in a AsyncTask, the problem still exists. So my question is:
Am i doing it in a good way to just put any time consuming operations away from Main UI, OR What's a better way, if any, to handle very very large amount of data in Android?
Use pagination logic. Fetch only the most recent and relevant data and load older data if the user requests it.
Call history on most android phones is limited to 500 entries CallLog.Calls, while SMS provider has not such limits, you can query the count of the tables and limit your queries to 50 at a time and pass it to a separate thread for processing. Also make sure you run this in a background service with low priority so as to not disturb any other operations ongoing in the device.
I am creating an Android app for which I need to create a SQLite DB and pre-populate it with some values.
The Android documentation says this about what to do in "onCreate" of the SQLiteOpenHelper:
Called when the database is created for the first time. This is where the creation of tables and the initial population of the tables should happen.
Reference - http://developer.android.com/reference/android/database/sqlite/SQLiteOpenHelper.html#onCreate(android.database.sqlite.SQLiteDatabase)
I am doubtful about the following 2 things -
What is meant by "when database is created for the first time"? Is this done on the first launch of the app or only when the first DB request (read/write etc) is done.
If it is the latter, I fear that it may take quite some time to create DB, pre-populate it with values (I have about 60 rows to be inserted into 1 table) and then read the DB to show it. Is this the best practice?
I have been doing all my DB operations in AsyncTasks. But I am doing the table creations in onCreate using "db.execSQL" statements. Is this fine (in terms of convention/ performance) or should I go for an AsyncTask here as well?
Any help is appreciated.
1) The later. It is done on the first read or write to the DB.
Your fear might be correct, this is why you can ship your app with a database that's already populated. Or you can launch an AsyncTask with a simple SELECT 1 FROM anytable query. More about shipping with DB here. (60 rows is nothing to fear about tho, and you can safely just keep using AsyncTasks).
2) Yes it is fine. The onCreate logic will run when you first read/write the DB, so it if you always use AsyncTasks onCreate will run in an AsyncTask also.
What is meant by "when database is created for the first time"? Is this done on the first launch of the app or only when the first DB request (read/write etc) is done.
It happens when you first query from database in general term. After that only Upgrade method is called that too when you change the db version.
If it is the latter, I fear that it may take quite some time to create DB, pre-populate it with values (I have about 60 rows to be inserted into 1 table) and then read the DB to show it. Is this the best practice?
60 rows insertion is not a big task. More you can read about beginTransaction(),commitTransaction and endTransaction for insertion. It will make your insertion task lighting fast.
I have been doing all my DB operations in AsyncTasks. But I am doing the table creations in onCreate using "db.execSQL" statements. Is this fine (in terms of convention/ performance) or should I go for an AsyncTask here as well?
It good you are doing you Db operation in AsyncTask and its completely fine.
Speaking of DB operations:
Performing DB operations in AsyncTask is not a good approach, generally. As you might encounter a problem called "memory leak", and it might come as a silent assassin in the night.
There's lot written on this issue. Just google "asynctask leak context" and here you go.
So how to perform DB operations?
Using Loader API in conjunction with ContentProvider is considered good approach for querying database. Loader asynchronously queries your database and delivers the result to specified subscribers. Configuration changes or other sudden stuff does not bother it.
And it is really convenient to query your data using loader API once you know how to do it.
Single inserts/updates/deletes might be done directly from the main thread via ContentResolver. These calls will be blocking (synchronous), but I bet you user would never notice anything while the amount of data is not large.
If you're operating on a large dataset, and you fear you'll be significantly blocking UI thread, I'd suggest using IntentService or any custom Service capable of doing operations in background (note that by default Service operates on main UI thread and you have to specify background operation yourself or use IntentService)
Speaking of DB initialisation:
You might create a one-time IntentService, if you're initialising a large set of data. It will handle your request asynchronously and, for example, perform a broadcast that the application is set up and ready, so you might stop a "wait a sec, performing app initialisation" screen and show user your data.
There's also nothing wrong with shipping your database along with application, though it appears to be a bit hackish solution.
Either way, you choose what is more suitable for you.
I am planning on writing an application that saves a fair amount of data. Historically, I have simply written data directly to a server, and only used some simple key/value storage with shared preferences for local storage.
I am considering this time, instead, using SQLite to save the information at first, and sync the data to the server in the background later. This will benefit the user in a few ways: 1) can use the app offline 2) don't have to worry about data being saved right away, it happens when ever it can 3) more reliability.
My approach will be to get/set data from SQLite during UI usage, and use a background process to find new rows and put them on the server, flagging them as synced when it happens.
Does this sound reasonable?
You can use SQLIte for your scenario. But, while implementing, you can follow any one of this approach.
Approach #1: Use an Abstract Factory to Instantiate the SQLiteOpenHelper.
Approach #2: Wrap the SQLiteDatabase in a ContentProvider
Refer to this link for how to implement these 2 approaches. http://www.androiddesignpatterns.com/2012/05/correctly-managing-your-sqlite-database.html
Key points to be noted while using SQLite
Sqlite takes care of the file level locking.
Many threads can read,one can write. The locks prevent more than one
writing.
Android implements some java locking in SQLiteDatabase to help keep
things straight.
If we handle the database incorrectly from many threads and mess up the code, your
database will not be corrupted. Only few updates will be lost.
How "Multiple Threads - DB access" can be used for your scenario
The SqliteOpenHelper object holds on to one database connection.
If you try to write to the database from actual distinct connections (multiple threads) at the same time, one will fail. It will not wait till the first is done and then write. It will simply not write your change. Worse, if you don’t call the right version of insert/update on the SQLiteDatabase, you won’t get an exception. You’ll just get a message in your LogCat, and that will be it.
So recommended to write using single thread and read from multiple threads if necessary for faster access.
Does this sound reasonable?
Yes. Note that the synchronization process can get tricky (e.g., what happens if the server hiccups halfway through?), but that has mostly to do with synchronization and little to do with SQLite.
We implemented a solution that used a SQLite db on the device to sync data via a web service to the master database. We did this for a couple reasons: offline, poor connection, manual sync.
For our solution we had a flag on the table that determined if the data was pushed to the web service. Our web service also provided data back to our application to let us know if the data was received and processed correctly. This allowed us to clean up the data on the device, send notifications if there were failures, and resubmit the data if there were previous failures.
You can use push notifications as well if you have fixed the issues on the backend and have the device resend the data to the web service. This worked really well for us.
Why do we use the sqlite data base in android.I am developing an android application where the data is to be fetched from the server and do some data calculation and show it on the UI.
Is it good for me to fetch the data into the sqlite DB and update the UI on regular interval from the sqlite in evry 20 minutes or will it be good to sent the Http get request to the server and update the data from teh response on the UI.
I wanted to know which one will be better and why?Why to involve sqlite DB?
The data corresponds to some 40X40 table data on which some heavy mathematical processing is to be done and then displayed on the UI(similar to Stock market application) and data needs to be cleared after every 12 hours.
plz advice
Rgds,
Raul
It is good to use database in your case.
Pros:
If your application gets closed the in memory data will be lost, but after that you will be able to restore the state from the database if you have one
Especially for the case of complex calculations it is good to store the result once in the database and not recalculate it multiple times on demand
The database will untie your UI from the internet connection and thus you will be able to display results even if there is not internet connection
Using database you will be able to fetch the updated data from a background service, without impacting your UI
Organizing your data in database usually makes it a lot easier to manage all the application data.
Cons:
Adding database will require a bit of additional effort on your side
As you can see my list proves you SHOULD use database in your case. Maybe I am biased, but at least I provide you with things to consider.
It's really a design decision, SQLite offers a very robust way to organize and persist your data, you're only other options are to write to a file, or to save in SharedPrefs, both methods become a lot harder to manage once the size of your data begins to grow, as you must manually keep a list of objects and manage their names etc etc. 40 x 40 table data is large enough to justify using SQLite, even if you are dropping and recreating the table every 12 hours.
You might want to consider using an ORM library to make fetching and saving data from the DB simpler, ORMLite is good and compatible with Android
http://ormlite.com/
If your application relies heavily on an internet connection you don't need to buffer information in the database. However if you want to use the app where you have bad or no signal you might want to used cached values from the sqlite database.
With slow internet connection your application may be unresponsive so caching may be a good idea - but it doesn't necessarily be in the sqlite database. You should use the sqlite database for data that is required by the device frequently and that is irrelevant to your server component.
If the data is updated frequently but only while the application runs you might want to cache in the devices memory. I assume your app is not running all the time within the 12 hours but is called regularly instead to check something.
12hrs is a long time, so rather than leaving your data wander in RAM, i would suggest you to use database. Because you never know when you may need to read it again.
Otherwise, if your purpose is only to downloaded data, process it and display in activity, then you dont need to involve database because if your app is closed (due to user or low memory), in anyway your app will be downloading fresh data from server... am i right?
> update the UI on regular interval from the sqlite in evry 20 minutes
Dont expect your app to be open for such a long duration.
To precisely suggest to your case
Avoid DB
Fetch Data at app start or at appropriate time when app is opened
and save it in plain java objects.
Define Methods within it that perform operation in it.
Make map or list to save those POJO
Define Seprate Controller Classes within your project to update map of pojo at any
specific change to make fresh data available to UI.
I'm processing a lot of XML data that validates the local data storage within an AsyncTask object. First I tried to use transactions for these operations but while a transaction is in progress any other actions by the user will make the app freeze and wait for the transaction to finish, and sometimes even make the app stop responding.
The transactions are divided into several steps sometimes counting to a couple of hundreds per iteration. Because of the problems I went from using transactions to real-time queries which is very slow but solves the freezing - resulting in a very battery consuming application.
My question is; Is there a way to stop the transactions from locking the database? Or is my problem a result of poor preparation before the transactions?
Transactions are tend to lock your table(s) while doing their business, so there's no way that you can play transactions and non-transactional queries on a same instance at same time.
However, what you need to do is to process your data (xml) first (which might be time consuming) and later kick-in transaction once you've got the data ready.
P.S. I would personally suggest you to make use of transactions (in an efficient way) when it comes to insert multiple records because it creates a single Journal-file to handle all the insertion and speeds up SQLite operations a lot.
make your transactions smaller and don't forget (like I did) to still end transaction if exception thrown - use try, catch, finally...