Develop Reference

SQLite selecting very slow when recent inserts have been made

I've got a table with about 7 million rows in it. I'm inserting on average about one row every second into the database. When I do this, I am noticing that it is taking an incredibly long time (as much as 15 seconds) to run a simple SELECT against the database, e.g. something like:
SELECT * FROM table WHERE rowid > 7100000
This select often returns no rows of data as sometimes no data has been inserted in this particular table. It is often happening even when the table I'm writing to isn't even actually inserting rows into the table I am reading.
The idea is that there are two separate processes, one is adding data, the other is trying to get all new data that has not yet been read. But the read side is connected to a UI and any noticable lag is intolerable, much less 15 seconds. This is being run under Android and the the UI thread doesn't like being blocked for that long either and it is wreaking havoc.
My initial thought was maybe the insert is requiring an update to the indicies as originally I had the index on a different field (a time field). This seems at least partially confirmed because if I use a database with only a few rows each select completes in a few milliseconds. But when I re-created the table to only have the rowid as primary key it actually got slower. I would expect inserting a new row at the end would always result in very fast reads when just comparing on the rowid as primary key.
I have tried enabling write ahead logging, but it appears that SQLCipher doesn't support this, at least not directly, as it doesn't adhere to the lastest API for android.database.sqlite.SQLiteDatabase. Even using "PRAGMA journal_mode = WAL" in the postKey hook hasn't made any difference.
What's going on here? How can I speed up my selects?
Update: I tried getting rid of sqlcipher and just using plain sqlite to see if that was a factor. I used sqlcipher_export to export to a plaintext database, and then used the default android.database.sqlite.SQLCipher. The delay time dropped from 10-20s to 1.8-2.8s. I then removed write-ahead and it dropped further to 1.3-2.7s. So the issue is still noticably there, although it did get a lot better.

SQLite is ultimately file-based, and there is no portable mechanism to communicate to another process which part of a file has changed. So when one process has written something, all other processes must drop their caches when they access the database file the next time.
If possible, modify your architecture that both parts of the code are in the same process and share the same database connection. (With multiple threads, this requires locking, but SQLite has not much concurrency anyway.)
Alternatively, write the new data into a separate database, and let the UI app move it to its own database.
I don't know why SQLCipher is so much slower (it's unlikely to be the CPU overhead of the decryption).

Parse.com equivalent to Group By

I have been reading through parse forums and I gather that as of 2 years ago there was no equivalent of SQL Group By.
I'm wondering if there have been any developments on this? I have thousands of records and I need to pull down all records in descending order of a value rating and then group them by name.
If this isn't available at present perhaps someone could suggest something I could work on instead.
Regards.

See https://parse.com/docs/android_guide#queries-basic , you can have secondary sorting key, which would be equivalent of "group by".
query.orderByDescending("rating");
query.addAscendingOrder("name");
could do the trick for you .

You have two options generally:
Refactor your schema to accommodate the model
Run a job to aggregate your data into "snapshots"
For option 1 it means creating "afterSave" handlers that update counters etc as data changes (expensive writes, cheap reads), here's a sample in the documentation that saves a "comments" count for a Post object:
https://parse.com/docs/cloud_code_guide#functions-aftersave
You would need to run a once-off job to set your initial counts since you already have data there (see option 2).
For option 2 you can learn more about background jobs by reading up here:
https://parse.com/docs/cloud_code_guide#jobs
Your job is allowed to run for up to 15 minutes (e.g. if it is processing insane amounts of data, or doing many things in sequence).

Why subsequent queries are faster than the very first one?

I have measured database queries and what I found is that the very first time it runs is slower than all subsequent queries. The very first time I run the code below I got: 11. For all subsequent calls it is always only 1. How does it work so it is faster after being first time queried?
long time = System.currentTimeMillis();
Cursor cursor = DBHelper.getInstance().getAllContacts();
Logger.log(TAG, "measured time is " + String.valueOf(System.currentTimeMillis() - time));
The `getAllContacts() method is:
(getReadableDatabase().rawQuery("SELECT _id, name, title FROM " + CONTACTS_TABLE_NAME, null));

There are a few effects that might be at play here.
First, you are also timing the getReadableDatabase() method: if that is doing significant work the first time (like opening the DB file!) and then caching what it does, you'll have a lot of overhead from that. Try just calling that method on its own before the timed zone so that you know that overhead isn't biting.
Secondly, SQLite works by compiling the SQL to an internal bytecode form and then executing that in its own little virtual machine. The compilation step can have non-trivial costs, and is specific to the SQL and the table(s) queried against. Luckily, you're sending the same SQL in every time (that's a constant table name, yes?) and SQLite is smart enough to implement an LRU cache of the compilations so that if you run the same query twice, it's faster. (I don't know the size of the cache; it's probably set at the time you build SQLite, with some sensible default.) You could test this in theory by just querying against a different table each time, but that would be silly. Closing the connection to the DB would also get rid of this cache, but would guarantee that you get poor performance; you're not supposed to measure just the start of things as then you're bearing lots of extra costs for no good reason.
Thirdly, it's quite possible that SQLite only reads some information about the DB and its tables when it actually needs to, so if this is the first query against that table, you may be bearing a number of extra costs. Try doing a different — otherwise never used — query against the table first, before the timing run. (Also, be aware of my caveats in the previous paragraph.)
Fourthly, the ORM layer might be caching the results, but frankly this is actually pretty unlikely as that's rather hard to get right (the hard part is working out when to flush the query results cache due to the DB being updated, and it's actually easier to just not bother with such complexity).

First time you call an SQL statement, SQLite engine has to prepare that typed SQL statement, ie, convert human readable text in machine executable code, which represents a slightly overhead.

Android - SQLite Trigger vs Java SQL insert, what is more efficient?

I want to insert a row in a table, and when I do the insertion I want to insert another row in another table at the same time.
What is the more efficient way to do this in an Android app?
Is it more efficient to do a trigger when I insert one row and SQL takes care of the insertion, or do two inserts with database.insert() in Java?
The information needed to insert the second row is already in the first insertion.
Thanks!

I want to insert a row in a table, and when I do the insertion I want
to insert another row in another table at the same time.
For sure use for this TRIGGER, because it working on database layer so you only need to create it and then it will be called itself when you do something(depends on how you implement trigger).
Also it's more faster and you do not call it explicit, just let database work. All what is database able to do, let do it on database.
Update:
So after a little discussion with #Graham Borland you can use also inserts in TRANSACTION.
Have look at Android Database Transaction.
And conclusion what is faster?
I exactly don't know it because i don't have performance tests but for sure TRANSACTIONS are the safest i think but whether faster than TRIGGERS this i don't know exactly but usage of transactions is very good and mainly safe approach to prevent database get to incorrect state that is very undesirable.

Is there any efficient way to make Android database query faster?

In my Android app, I need to get 50,000 database entries (text) and compare them with a value when the activity starts (in onCreate()). I am doing this with the simplest way: I get the whole table from db to a cursor. However this way is too laggy. Are there any other ways to do it more effectively ?
Edit: The app is "scrabble solver" that is why I am not using WHERE clause in my query (Take the whole data and compare it with combination of the input letters). At first I was using a big table which contains whole possible words. Now I am using 26 tables. This reduced the lag and I am making database calls on a thread - that solved a lot problems too. It is still little bit laggy but much better.

To summarize and add a bit more
Let the database do the work of searching for you, use a WHERE clause when you perform a query!
If your where clause does not operate on the primary key column or a unique column you should create an index for that column. Here is some info on how to do that: http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html#indexes
Google says: "Use question mark parameter markers such as 'phone=?' instead of explicit values in the selection parameter, so that queries that differ only by those values will be recognized as the same for caching purposes."
Run the query analysis using EXPLAIN QUERY PLAN http://www.sqlite.org/lang_explain.html and look for any scan operations, these are much slower than search operations. Uses indexes to avoid scan operations.
Don't perform any time consuming tasks in onCreate(), always use an AsyncTask, a Handler running on a background thread or some other non-main thread.
If you need to do full text search please read: http://www.sqlite.org/fts3.html

You should never read from the database in the UI thread. Use a background thread via AsyncTask or using regular threading. This will fix the UI lag issue your having.
Making the database read faster will help with bringing the data faster to the user but it's even more important that the fetching of the data does not block the user from using the app.
Check out the Following Links:
Painless Threading in Android
YouTube: Writing Zippy Android Apps

Use a WHERE clause in your SQL rather than reading the whole thing in. Then add an index on the columns in the WHERE clause.

At least you can put index on the field you compare and use WHERE clause. If you are comparing numerics Sqlite (db engine used by Android) supports functions such as MIN and MAX. Also if you are comparing partial strings you can use LIKE. For query optimization there are many resources such as this