I have a sqlite db that at the moment has few tables where the biggest one has over 10,000 rows. This table has four columns: id, term, definition, category. I have used a FTS3 module to speed up searching which helped a lot. However, now when I try to fetch 'next' or 'previous' row from table it takes longer than it was before I started using FTS3.
This is how I create virtual table:
CREATE VIRTUAL TABLE profanity USING fts3(_id integer primary key,name text,definition text,category text);
This is how I fetch next/previous rows:
SELECT * FROM dictionary WHERE _id < "+id + " ORDER BY _id DESC LIMIT 1
SELECT * FROM dictionary WHERE _id > "+id + " ORDER BY _id LIMIT 1
When I run these statements on the virtual table:
NEXT term is fetch within ~300ms,
PREVIOUS term is fetch within ~200ms
When I do it with normal table (the one created without FTS3):
NEXT term is fetch within ~3ms,
PREVIOUS term is fetch within ~2ms
Why there is such a big difference? Is there any way I can improve this speed?
EDITED:
I still can't get it to work!
Virtual table you've created is designed to provide full text queries. It's not aimed to fast processing standard queries using PK in where condition.
In this case there is no index on your _id column, so SQLite probably performs full table scan.
Next problem is your query - it's totally inefficient. Try something like this (untested):
SELECT * FROM dictionary WHERE _id = (select max(_id) from dictionary where _id < ?)
Next thing you can consider is redesign of your app. Instead of loading 1 row you, maybe you should get let's say 40, load them into memory and make background data loading when there is less than n to one of the ends. Long SQL operation will become invisible to user even if it'll last 3s instead of 0,3s
If you're running LIMIT 1 to begin with, you can remove the order by clause completely. This may help. I'm not familiar with FTS3, however.
You could also just flat out assign your id variable a ++ or -- and assert `WHERE _id = "+id+" LIMIT 1" which would make a single lookup instead of < or >.
Edit: and now that I look back at what I typed, if you do it that way, you can just remove LIMIT 1 completely, since your _id is your pk and must be unique.
hey look, a raw where clause!
Related
I have an Sqlite database on Android and if I put 2000 records in one table all the other tables get really slow.
With 2000 records in the one table I can run a 'SELECT COUNT(*)' on a table with 0 records and it can take anywhere from 5 to 30 seconds.
There can be many reasons for that. The usual reasons are:
Poorly written query
Not using a primary key, assuming one even exists on the table
Poorly designed data model (table structure)
Lack of indexes
IMO, your case may be lack of indexes. Check if you have indexed your tables/rows.
"SELECT COUNT( * )", the char ' * ' need database to extract every field when sql execute.
As an additional suggest, you can use primary key in the " count( < your primary key > ) ".
Is there a way to access the SQLiteLog or at least pipe out errors to do with SQLite?
I would like to automatically send any errors, like the below, so I can optimise the database by adding in indexes, or making other changes as need be.
E/SQLiteLog: (284) automatic index on messages(chat_id)
It could well be that I'd only want to catch code 284, rather than getting everything. Is there a way of doing this when building the app, as would be useful to pass to crashlytics to help with development going forward
A potential alternative, would be to run the queries preceded with EXPLAIN QUERY PLAN, and to then check for AUTOMATIC COVERING INDEX in the results.
e.g.
DROP TABLE IF EXISTS table1;
DROP TABLE IF EXISTS table2;
CREATE TABLE IF NOT EXISTS table1 (column1 TEXT, column2 TEXT);
CREATE TABLE IF NOT EXISTS table2 (column3, column4);
EXPLAIN QUERY PLAN
SELECT * FROM table1, table2 WHERE column1=column3;
Results in :-
As you can see this is based upon a predication of the number of rows rather than the actual number of rows, so works if there is no data (as above), which could well be beneficial, from a development perspective, as opposed to trying to trap on a as happens basis.
I need get the total SUM for each rows in my query, but I don't want go twice in the table.
I tried do this:
SELECT id, value, SUM(value) as total FROM product
But my result was this:
id value total
3 30 60
If I do the bellow query I get my wanted result, but I need go twice in the table:
SELECT id, value, (SELECT SUM(value) FROM product) as total FROM product
Or if I use 'WITH' clause, but this is not supported before Android 5:
WITH data AS (SELECT id, value FROM product)
SELECT id, value, (SELECT SUM(value) FROM data) as total FROM data
Wanted result:
id value total
1 10 60
2 20 60
3 30 60
Thank you!
It's not possible using your SQLite version. You'll have to use two selects.
Basically you have to use a subquery.
However, perhaps you may be less concerned about the 2nd table as I believe that the Query Planner will determine that it only needs to calculate the sum once and does away with the need for a variable as it stores the value in cache.
I believe that the results of using EXPLAIN QUERY PLAN your_query shows this. i.e. using
EXPLAIN QUERY PLAN SELECT id, value, (SELECT sum(value) FROM products) AS total FROM products;
results in :-
This being explained as (see bolded statements) :-
1.3. Subqueries
In all the examples above, the first column (column "selectid") is
always set to 0. If a query contains sub-selects, either as part of
the FROM clause or as part of SQL expressions, then the output of
EXPLAIN QUERY PLAN also includes a report for each sub-select. Each
sub-select is assigned a distinct, non-zero "selectid" value. The
top-level SELECT statement is always assigned the selectid value 0.
For example:
sqlite> EXPLAIN QUERY PLAN SELECT (SELECT b FROM t1 WHERE a=0), (SELECT a FROM t1 WHERE b=t2.c) FROM t2;
0|0|0|SCAN TABLE t2
0|0|0|EXECUTE SCALAR SUBQUERY 1
1|0|0|SEARCH TABLE t1 USING COVERING INDEX i2 (a=?)
0|0|0|EXECUTE CORRELATED SCALAR SUBQUERY 2
2|0|0|SEARCH TABLE t1 USING INDEX i3 (b=?)
The example above contains a pair of scalar subqueries assigned
selectid values 1 and 2. As well as a SCAN record, there are also 2
"EXECUTE" records associated with the top level subquery (selectid 0),
indicating that subqueries 1 and 2 are executed by the top level query
in a scalar context. The CORRELATED qualifier present in the EXECUTE
record associated with scalar subquery 2 indicates that the query must
be run separately for each row visited by the top level query. Its
absence in the record associated with subquery 1 means that the
subquery is only run once and the result cached. In other words,
subquery 2 may be more performance critical, as it may be run many
times whereas subquery 1 is only ever run once.
Unless the flattening optimization is applied, if a subquery appears
in the FROM clause of a SELECT statement, SQLite executes the subquery
and stores the results in a temporary table. It then uses the contents
of the temporary table in place of the subquery to execute the parent
query. This is shown in the output of EXPLAIN QUERY PLAN by
substituting a "SCAN SUBQUERY" record for the "SCAN TABLE" record that
normally appears for each element in the FROM clause. For example:
sqlite> EXPLAIN QUERY PLAN SELECT count(*) FROM (SELECT max(b) AS x FROM t1 GROUP BY a) GROUP BY x;
1|0|0|SCAN TABLE t1 USING COVERING INDEX i2
0|0|0|SCAN SUBQUERY 1
0|0|0|USE TEMP B-TREE FOR GROUP BY
If the flattening optimization is used on a subquery in the FROM
clause of a SELECT statement, then the output of EXPLAIN QUERY PLAN
reflects this. For example, in the following there is no "SCAN
SUBQUERY" record even though there is a subquery in the FROM clause of
the top level SELECT. Instead, since the flattening optimization does
apply in this case, the EXPLAIN QUERY PLAN report shows that the top
level query is implemented using a nested loop join of tables t1 and
t2.
sqlite> EXPLAIN QUERY PLAN SELECT * FROM (SELECT * FROM t2 WHERE c=1), t1;
0|0|0|SEARCH TABLE t2 USING INDEX i4 (c=?)
0|1|1|SCAN TABLE t1
EXPLAIN QUERY PLAN
End Note
Perhaps of relevance is this statement :-
The best feature of SQL (in all its implementations, not just SQLite)
is that it is a declarative language, not a procedural language. When
programming in SQL you tell the system what you want to compute, not
how to compute it. The task of figuring out the how is delegated to
the query planner subsystem within the SQL database engine.
Query Planning
You may also find this of interest he SQLite Query Optimizer Overview noting that as of release 3.8.0 The Next-Generation Query Planner is utilised.
I am developing dictionary application. It requires incremental search which means that SELECTING should be fast. There are 200000+ rows. Let me, first of all explain, table structure. I have this table:
CREATE TABLE meaning(
key TEXT,
value TEXT,
entries BLOB);
Some times ago I had this index:
CREATE INDEX index_key ON meaning (key)
This query was performed for around ~500ms which was very slow
SELECT value FROM meaning WHERE key LIKE 'boy%' LIMIT 100
Then I dropped this index, created incasesensitive index which helped to improve performance 2-3 times.
CREATE INDEX index_key ON meaning (key COLLATE NOCASE);
Now this query performing for 75ms(min) - 275(ms) which is quite slow for incremental search.
SELECT value FROM meaning WHERE key LIKE 'boy%' LIMIT 100
I have tried to optimize query according to this post.
SELECT value FROM meaning WHERE key >= 'boy' AND key<'boz' LIMIT 100
But this query is performed for 451ms.
EXPLAIN
SELECT value FROM meaning WHERE key LIKE 'boy%' LIMIT 100
This is returning following values:
EXPLAIN QUERY PLAN
SELECT value FROM meaning WHERE key LIKE 'boy%' LIMIT 100
This is returning this value(detail column):
SEARCH TABLE meaning USING INDEX index_key (key>? AND key<?) (~31250 rows)
Actually this values did not give me some sense or key what to optimize.
Is it possible to optimize SELECTion of words to be performed in ~10ms by optimization of this query or creating another table or changing some parameters of SQLite database? Could you suggest me the best way to do this?
PS. Please, do not suggest to use FTS table. In previous version of application I have used FTS. I agree that it is extremely fast. I left FTS table idea for 2 reasons:
It is not giving proper result(it contains the words which user do not need)
It takes more disk space
I want to add the results of two separate counting SQlite queries. Suppose I have 2 tables named entries and scores and have 2 queries:
SELECT COUNT(1) FROM entries WHERE
key NOT IN (SELECT key FROM scores)
SELECT COUNT(1) FROM scores WHERE
value <= threshold
Maybe I could do something like this to get the sum of their results:
SELECT COUNT(1) from (
SELECT key FROM entries WHERE
key NOT IN (SELECT key FROM scores)
UNION ALL
SELECT key FROM scores WHERE
value <= threshold
)
But is this a little too inefficient? This is called pretty often and may interfere with the UI's smoothness.
Thank you.
[EDIT] What I'm actually trying to do:
I'm making an app to help learning vocabulary. The entries table keeps 'static' data about word type, definition, etc. The scores table keeps information about how well you've learned the words (e.g. performance, scheduled next review time)
To check for the number of remaining words to learn/review, I count how many words do not exist in the scores table yet (i.e. never touched) or when the accumulated score is pretty low (i.e. needs reviewing).
The reason I don't merge those 2 tables into 1 (which would make my life much easier) is because sometimes I need to update the entries table either by inserting new words, deleting a few words, or updating their content, and I haven't found a simple way to do that. If I simply do INSERT OR REPLACE, I will lose the information about scores.
I think you're looking for a UNION. A union combines the results from two queries. Try this (sorry it isn't tested, I don't have access to SQLite):
SELECT COUNT(1) FROM
(
SELECT 1
FROM entries
WHERE key NOT IN (SELECT key FROM scores)
UNION ALL
SELECT 1
FROM scores
WHERE scores.value <= threshold
)
After reading the edit in your question explaining what you need to do, I think a JOIN would be more appropriate. This is a way of combining two tables into one query. Something like this:
SELECT COUNT(1)
FROM entries
LEFT JOIN score
ON score.key = entries.key
WHERE score.value <= threshold
OR score.key is null