Not a total show stopper for my project, but I'm a little concerned at the performance of SQLITE when I am deleting a specific row from the database.. Sometimes it's taking up to 5 seconds which seems too long and I can imagine users may thing the application has crashed.
My code is very straightforward, the delete call is simply :
boolean result;
result = mDb.delete(ACCIDENTS_MEDIA_TABLE, ACCIDENTS_MEDIA_KEY_ROWID + "=" + rowId, null) > 0;
So nothing odd there, the table is relative small in terms of fields (5) but 2 of those are blob fields so I'm wondering if that can be the reason.
If anyone has experience with this & suggestions how to improve my performance so it's not taking up to 5 seconds to delete a row, that would be appreciated.. Thanks.
Rgds,
Using an ASyncTask, or a seperate Runnable/Thread with a handler is, as suggested the best way to approach these type of issues.
Related
In my Android SQLite database, I have a column that is a datetime in the format YYYY-MM-DD HH:MM. I want a query (or a cursor, actually) that will give me all of the rows in, let's say May 2015. The sorting is where I'm getting stuck. I want my results broken down first by week, then for each week a sort based on other columns.
Right now I'm just sorting by the datetime column, followed by the other columns. Eg: ORDER BY t.TASKS_DATETIME ASC, s.STATUSES_RANK ASC, t.TASKS_CATEGORY. But basically every row has a unique value for datetime (even though I don't care about hours and minutes in this case) so no other sorting happens and the end result is not very user friendly. The sort order is really important for the user to look at and understand this data.
I thought about doing a separate select statement for each of the weeks, then doing a UNION ALL to get the month data. But you can't use the ORDER BY clause with a UNION, because it sorts the whole union, not just the individual select statements inside. Not to mention, it's pretty ugly.
I thought maybe I could do it on the Android/Java side and concatenate multiple Cursor objects. But I can't find any way to do that. Does anyone here know a way? I've been away from Java for a few years and was never that intimate with the language to start.
I thought maybe I could use the SQLite date functions [https://www.sqlite.org/lang_datefunc.html ] in some way. Maybe add a new column in my select that has the week of year. Then I could sort by that new column, followed by the other sort criteria that I want. Or maybe there is a way to put a range (or simplified datetime) into the ORDER BY clause. But this is pushing the boundaries of my SQLite skills. Does this seem possible? FYI, on the Android/Java side, I convert this datetime String to a GregorianCalendar.
I'm hoping someone smarter than me on here can point me in a good direction! None of these options seems great and I can't think of any others.
If I can't figure out anything else, I'll go with the UNION option. And add a "week_of_year" column to each select statement inside to identify that week, as suggested by other union/sorting questions here. Then sort by that "week_of_year" column, followed by my other sort criteria. I've started working on this. But I took a break to ask for better ideas here, cause my gut says there has to be a better way. But maybe not :)
As additional background, that I don't think is relevant, but just in case...
This scary query is being used in a StickyHeaderList [https://github.com/emilsjolander/StickyListHeaders ], where the weeks are the headers. If I don't sort first by datetime, the headers/weeks end up out of order. Sometimes they even duplicate.
I will probably at some point write my own alternative or alter this StickyHeaderList code, perhaps something that takes a cursor for each week/header rather than just one for the whole month. I can deal with dates easier in Java then SQL. But I don't have the resources for that today. The biggest headache, I imagine, will be having the multiple-select CAB menu allow me to select across different lists. If you happen to know an easy work around for that, maybe I will just do that and accept the major rework over figuring out this query.
UPDATE: It works!
Many thanks to #CL for giving me the answer. For posterity and clarity, here are some notes.
The correct answer is:
SELECT ...
FROM ...
ORDER BY strftime('%Y%W', t.TASKS_DATETIME),
s.STATUSES_RANK,
t.TASKS_CATEGORY
As noted, weeks are finicky things. I need to also account for users who have weeks that start on different days of the week. Note, the SQLite default is to start the ween on Monday (not Sunday). Also, the keyword is day not days for accounting for that offset. Also, the offset should be made to the right of the datetime.
This left me with:
"ORDER BY strftime('%Y %W', t." + TASKS_SCHED_DATETIME + weekOffset + ")," +
"s." + STATUSES_RANK + " ASC, " +
"t." + TASKS_CATEGORY + " DESC "
Which in the end becomes: ORDER BY strftime('%Y %W', t.task_sched_datetime, '+5 day' ), s.status_rank ASC, t.tasks_category DESC
weekOffset is defined in a little helper function:
private String calculateWeekOfset() {
int wkStart = App.getWeekStart();
int offset = 9 - wkStart; // Calendar.MONDAY == 2
return ", '+" + Integer.toString(offset)+ " day' ";
}
I think that's all. Thanks again!
To extract fields from a timestamp, use the strftime function.
In this case, you need the week of the year, and the year itself to make dates in different years unique:
SELECT ...
FROM ...
ORDER BY strftime('%Y%W', t.TASKS_DATETIME),
s.STATUSES_RANK,
t.TASKS_CATEGORY
If you do not want to start a week on a Monday, add or subtract the appropriate number of days:
... strftime('%Y%W', t.TASKS_DATETIME, '+1 days'), ...
I am currently building a database recording events on the phone, but as I don't want to make this a huge database, 100 events are more than enough.
This will keep my database light en efficient.
Unfortunately, I don't see a way to limit the number of rows other than
String sql = "DELETE FROM myTable WHERE _id <= "+limitId;
and I could run this code when the user launch/leaver the app, but I am expecting a better way to achieve this
Is there a more convenient way to achieve this?
If you are using a ContentProvider, you can implement your DELETE in onInsert, deleting a single row on every insert of a single row:
String sql = "DELETE FROM myTable WHERE _id IN (SELECT min(_id) FROM myTable)";
I guess you mean limit to the 100 newest events ?
if so, there is no better way to do it as you did: checking for entries on every insert and delete old entries if necessary.
It's just a matter of taste how or where you do your check, as flx mentioned you could do it in the ContentProvider or as you probably did in the BroadcastReceiver or Service where you actually add the new row. You could also set up a Trigger on your table, but the main idea remains the same. Here a link if you're interested in triggers:
http://www.tutorialspoint.com/sqlite/sqlite_triggers.htm
I have a record in my Android DB which is saved along with its creation time. I want to have a timer where after 2 hours this item will be deleted. I researched it and I came to two results:
one was to make an AsyncTask, but this solution seems to me like it would take too many resources without any need.
the second was to implement a Service. I know nothing of Services in Android and how they work. I made some reading and I don't see how I can have a service counting time and checking if it has exceeded its lifespan.
Halp plz.
I suggest that you could give a look into the AlarmManager Class
http://developer.android.com/reference/android/app/AlarmManager.html
Why not simply ignore Rows that are older than 2hours? Simply create a VIEW on your Table. Paired with an INSERT/UPDATE Trigger on the DB that delete old Records.
pseudo:
for the view select
select * from table where createdate > now()-2hours
create trigger on table.cleanuptrigger after insert/update
on table for each row when createdate < now()-2hours;
I have found that `"CountDownTimer" works well as long as your application is alive.
I maintain an application that is collecting a lot of information and is storing these information in an ArrayList.
In detail this ArrayList is defined as ArrayList<FileInformation> which has some member like:
private File mFile;
private Long mSize;
private int mCount;
private Long mFilteredSize;
private int mFilteredCount;
private int mNumberOfFilters;
etc.
This approach is working but is not very flexible when I would like to introduce some new functionality. It also has some limitations in terms of memory usage and scale-ability. Because of this I did some tests if a database is the better approach. From the flexibility there is no question, but somehow I'm not able to make it running fast enough to become a real alternative.
Right now the database has just one table like this:
CREATE TABLE ExtContent (
"path" TEXT not null,
"folderpath" TEXT not null,
"filename" TEXT,
"extention" TEXT,
"size" NUMERIC,
"filedate" NUMERIC,
"isfolder" INTEGER not null,
"firstfound" NUMERIC not null,
"lastfound" NUMERIC not null,
"filtered" INTEGER not null
);
The performance issue is immense. Collecting and writing ~14000 items takes ~3mins! when writing into the database and just 4-5secs if written into the ArrayList.
Creating the database in-memory does not make a big difference.
As my experience in terms of SQLITE is rather limited, I started by creating the entries via the android.database.sqlite.SQLiteDatabase.insert methode.
As there was no meaningful difference between a file based and a in-memory database, I guess using BEGIN TRANSACTION and COMMIT TRANSACTION will not make any difference.
Is there some way to optimize this behavior?
Just for clarification, putting BEGIN TRANSACTION and END TRANSACTION will increase the performance greatly. Quoted from http://www.sqlite.org/faq.html#q19 :
SQLite will easily do 50,000 or more INSERT statements per second on an average desktop computer. But it will only do a few dozen transactions per second. By default, each INSERT statement is its own transaction...
I had a similar issue on an app I was coding on the weekend.
Is the data in the database to be included in the app when it's released? If so, bulk inserts aren't they way to go, instead you want to look at creating the database and including it in the assets directory and copying it over to the device. Here's a great link.
Otherwise I'm not sure you can do much to improve performance, this link explains methods on bulk inserting into an SqlLite Database.
Edit: You may also want to post your insert code too.
This is opretty obvious. Assuming you already allocated object to insert into. ( This is the same workload for bot solutions ) Let's compare alternatives:
Inserting in ArrayList does:
- (optional) allocate new chinks of cells for pointers if necessary
- insert object pointer into array list on the end
... really fast
INserting into sqlite:
-prepare insertion query ( I hope you use prepared query, and do not construct it from strings)
-perform database table insertion with modifications of indexes etc.
... a lot of work
Only advantage of database is that you can:
- query it later
- it handles external storage transparently allowing you to have much more entities
But it comes at cost of performance.
Depending on what you are for, there could be better alternatives.
For example, in my android games I store highscore entries in JSON file and utilise
GSON Pull parser / databinding layer ( https://github.com/ko5tik/jsonserializer ) to create objects out of it. Typical load time for 2000 entries from external storage is about 2-3 seconds
We have about 7-8 tables in our Android application each having about 8 columns on an average. Both read and write operations are performed on the database and I am experimenting and trying to find ways to enhance the performance of the DataAccess layer. So, far I have tried the following:
Use positional arguments in where clauses (Reason: so that sqlite makes use of the same execution plan)
Enclose inserts and update with transactions(Reason: every db operation is enclosed within a transaction by default. Doing this will remove that overhead)
Indexing: I have not created any explicit index other than those created by default on the primary key and unique keys columns.(Reason: indexing will improve seek time)
I have mentioned my assumptions in paranthesis; please correct me if I am wrong.
Questions:
Can I add anything else to this list? I read somewhere that avoiding the use of db-journal can improve performance of updates? Is this a myth or fact? How can this be done, if recomended?
Are nested transactions allowed in SQLite3? How do they affect performance?
The thing is I have a function which runs an update in a loop, so, i have enclosed the loop within a transaction block. Sometimes this function is called from another loop inside some other function. The calling function also encloses the loop within a transaction block. How does such a nesting of transactions affect performance?
The where clauses on my queries use more than one columns to build the predicate. These columns might not necessarily by a primary key or unique columns. Should I create indices on these columns too? Is it a good idea to create multiple indices for such a table?
Pin down exactly which queries you need to optimize. Grab a copy of a typical database and use the REPL to time queries. Use this to benchmark any gains as you optimize.
Use ANALYZE to allow SQLite's query planner to work more efficiently.
For SELECTs and UPDATEs, indexes can things up, but only if the indexes you create can actually be used by the queries that you need speeding up. Use EXPLAIN QUERY PLAN on your queries to see which index would be used or if the query requires a full table scan. For large tables, a full table scan is bad and you probably want an index. Only one index will be used on any given query. If you have multiple predicates, then the index that will be used is the one that is expected to reduce the result set the most (based on ANALYZE). You can have indexes that contain multiple columns (to assist queries with multiple predicates). If you have indexes with multiple columns, they are usable only if the predicates fit the index from left to right with no gaps (but unused columns at the end are fine). If you use an ordering predicate (<, <=, > etc) then that needs to be in the last used column of the index. Using both WHERE predicates and ORDER BY both require an index and SQLite can only use one, so that can be a point where performance suffers. The more indexes you have, the slower your INSERTs will be, so you will have to work out the best trade-off for your situation.
If you have more complex queries that can't make use of any indexes that you might create, you can de-normalize your schema, structuring your data in such a way that the queries are simpler and can be answered using indexes.
If you are doing a large number of INSERTs, try dropping indexes and recreating them at the end. You will need to benchmark this.
SQLite does support nested transactions using savepoints, but I'm not sure that you'll gain anything there performance-wise.
You can gain lots of speed by compromising on data integrity. If you can recover from database corruption yourself, then this might work for you. You could perhaps only do this when you're doing intensive operations that you can recover from manually.
I'm not sure how much of this you can get to from an Android application. There is a more detailed guide for optimizing SQLite in general in the SQLite documentation.
Here's a bit of code to get EXPLAIN QUERY PLAN results into Android logcat from a running Android app. I'm starting with an SQLiteOpenHelper dbHelper and an SQLiteQueryBuilder qb.
String sql = qb.buildQuery(projection,selection,selectionArgs,groupBy,having,sortOrder,limit);
android.util.Log.d("EXPLAIN",sql + "; " + java.util.Arrays.toString(selectionArgs));
Cursor c = dbHelper.getReadableDatabase().rawQuery("EXPLAIN QUERY PLAN " + sql,selectionArgs);
if(c.moveToFirst()) {
do {
StringBuilder sb = new StringBuilder();
for(int i = 0; i < c.getColumnCount(); i++) {
sb.append(c.getColumnName(i)).append(":").append(c.getString(i)).append(", ");
}
android.util.Log.d("EXPLAIN",sb.toString());
} while(c.moveToNext());
}
c.close();
I dropped this into my ContentProvider.query() and now I can see exactly how all the queries are getting performed. (In my case it looks like the problem is too many queries rather than poor use of indexing; but maybe this will help someone else...)
I would add these :
Using of rawQuery() instead of building using ContentValues will fasten up in certain cases. off course it is a little tedious to write raw query.
If you have a lot of string / text type data, consider creating Virtual tables using full text search (FTS3), which can run faster query. you can search in google for the exact speed improvements.
A minor point to add to Robie's otherwise comprehensive answer: the VFS in SQLite (which is mostly concerned with locking) can be swapped out for alternatives. You may find one of the alternatives like unix-excl or unix-none to be faster but heed the warnings on the SQLite VFS page!
Normalization (of table structures) is also worth considering (if you haven't already) simply because it tends to provide the smallest representation of the data in the database; this is a trade-off, less I/O for more CPU, and one that is usually worthwhile in medium-scale enterprise databases (the sort I'm most familiar with), but I'm afraid I've no idea whether the trade-off works well on small-scale platforms like Android.