I'm trying to get the lowest _id from my fts table with this query:
SELECT MIN(_id) FROM fts WHERE tbl_no=2 AND parent_id=6
The result I'm getting is 10. However the smallest _id is 9 and it fits the selection arguments.
If I instead use
SELECT _id FROM fts WHERE tbl_no=2 AND parent_id=6
and select the 1st row, I get the correct result: 9.
Does have something to do with the table being virtual (FTS)? I recently transfered from multiple tables to a single FTS and am experiencing this.
Am I guaranteed to get the results I want with the 2nd query, considering the table never updated and it's sorted by default.
Notes: I am running this on Android (tried rawQuery and query). I have the table in front of me and I know it's correct:
Is _id a numeric or a string?
With string comparison, '10' < '9'.
Try:
SELECT MIN(CAST(_id AS UNSIGNED)) FROM fts WHERE tbl_no=2 AND parent_id=6
To check. I would not use this in production however as it won't be able to use an index.
In FTS tables, all columns store string values, and the string '10' is lexicographically smaller than '9'.
Furthermore, MIN(SomeColumn) is not a full-text search query, and thus is not very efficient.
For a unique integer ID in FTS tables, you should use the internal docid column.
Related
I have a value from the epoch time like this 1549251913000, I save this value to SQLite. I create the table like the following:
CREATE TABLE TABLE_BOOKMARK (COLUMN_ID INTEGER PRIMARY KEY AUTOINCREMENT, COLUMN_TITLE TEXT, COLUMN_SOURCE TEXT, COLUMN_DATEANDTIME INTEGER, COLUMN_GUID TEXT);
that value is COLUMN_DATEANDTIME but with INTEGER type. but when I take the value, it doesn't match what I expected. it becomes like this -1231280856.
Please give me some advice, thanks
I have tried this solution but still mismatch when I get it from SQLite (seems it didn't work with my problem)
Create a column in as INTEGER datatype put LONG value into INTEGER in column
Make sure when you retrieve the value from cursor as LONG
Cursor cursor = db.rawQuery("SELECT * FROM " + TABLE_NAME, null);
long value = cursor.getLong(0);
For more : SQLite DataType Doc
Just to clarify the column type is, with one exception, largely irrelevant as any type of data can be stored in any type of column.
The exception is the rowid column or an alias of the rowid column, such a column MUST store an integer. By integer it is a 64bit signed integer and thus encompasses a java long.
The column type itself is also flexible for example CREATE TABLE mytable (mycolumn a_pretty_weird_column_type) is valid (as would be LONG). Such types are converted according to 5 rules to the column affinity.
Putting the above together, using :-
CREATE TABLE IF NOT EXISTS mytable (mycolumn a_pretty_weird_column_type);
DELETE FROM mytable;
INSERT INTO mytable VALUES
(1549251913000),
(999999999999999),('Fred'),(x'010203040506070809'),(0.234567),(null),
('999999999999999'), -- Note although specified as TEXT as mycolumn is effectively NUMERIC stored as INTEGER
('0.234567') -- As above but stored as REAL
;
SELECT
*,
typeof(mycolumn) AS coltype, -- The column type (note as per value not column definition)
hex(mycolumn) AS as_hex, -- Convert column to a hex representation of the data
CAST(mycolumn AS TEXT) AS as_text, -- follow rules
CAST(mycolumn AS INTEGER) AS as_integer,
CAST(mycolumn AS REAL) AS as_real,
CAST(mycolumn AS NUMERIC) AS as_numeric,
CAST(mycolumn AS BLOB) AS as_blob
FROM mytable;
results in :-
The bible as such is Datatypes In SQLite Version 3.
but when I take the value, it doesn't match what I expected. it
becomes like this -1231280856
The Issue
As such your issue is nothing to do with the column type, not SQLite as such, rather it's due to using the Cursor getInt method, instead of the getLong method.
One answer says,
Its always better to store date and time in form of Text in sqlite.
This is incorrect, from a space point of view and therefore the underlying efficiency the use of a numeric representation, i.e. 64bit signed integer as per SQL As Understood By SQLite - Date And Time Functions - Time Strings (maximum of 8 bytes), of the date and time will be more efficient than storing the 19 bytes (for an accuracy down to a second).
As per the docs sqlite does not have default storage class for date and time. Its always better to store date and time in form of Text in sqlite. You can then parse them into Date runtime whenever you want to use them
In my Android app, I create a FULLTEXT table like this:
CREATE VIRTUAL TABLE products USING fts3 (
_id integer PRIMARY KEY,
product_name text NOT NULL,
...
)
And I add this index:
CREATE INDEX product_name_index ON products (product_name)
The app populates the table with various products, each with a unique _id value.
However, when I then try to insert an already-existing product ID (using an _id value that is already in the table, but with a different product_name value) like this:
long rowId = db.insertOrThrow("products", null, contentValues);
a new row is added to the table (with a brand new rowId value returned)!
I expected the insertOrThrow command to fail, so where am I going wrong? Is it something to do with the fact that it's a FULLTEXT table or could the index I specified on the product_name column be messing things up somehow?
I read this section about INTEGER PRIMARY KEY, but unfortunately I'm none the wiser.
Update
When I try to perform the same operation on a standard (non-FULLTEXT) table, then the insertOrThrow command results in the expected SQLiteConstraintException.
I think the issue might be that an FTS table has the concept of a docid and a rowid column and specifying null for the docid results in that being given a value.
as per :-
There is one other subtle difference between "docid" and the normal
SQLite aliases for the rowid column.
Normally, if an INSERT or UPDATE
statement assigns discrete values to two or more aliases of the rowid
column, SQLite writes the rightmost of such values specified in the
INSERT or UPDATE statement to the database.
However, assigning a
non-NULL value to both the "docid" and one or more of the SQLite rowid
aliases when inserting or updating an FTS table is considered an
error. See below for an example.
1.3. Populating FTS Tables
I'm new in sqlite. I've built database but based on query I was trying to solve for it (which was over-complicated), I was suggested to look into normalising database, which I did, but can't seem to find examples on database that would be orientated around dates like a diary, with lots of daily entries. I'm working on app that would help log in everyday what I've eaten, what exercise did I do, what activities I've done, what was my well-being, how many hours I've slept. I will be able to go back to any day in the past and see what I was up to, so it will have to look up all entries for that particular date.
So I understand I need separate tables for food type, exercise type, activities types, event types and I need main table "diary" which will log each time date and id referencing another table. So I'm wondering if in that diary table I can have date column, id column and lets say type column (which will differentiate which table does id column references) or should I rather have date column and column for each of the other tables ids, even though I will be logging only one type at the time?
Also, would indexing the date column be a good idea?
Or maybe there is a better way to design that database? Any suggestions will be appreciated.
So I understand I need separate tables for food type, exercise type,
activities types, event types
If normalising then perhaps consider a single table for all types with a column to indicate the type.
So I'm wondering if in that diary table I can have date column, id
column and lets say type column (which will differentiate which table
does id column references) or should I rather have date column and
column for each of the other tables ids, even though I will be logging
only one type at the time?
If you are logging and assuming human input (as opposed to automated) then it is highly likely that a timestamp would be sufficient to uniquely identify a row.
As such there would be little need for an id column(in theory).
Saying that SQLite, unless you specify WITHOUT ROWID (which you might consider, this may be of use in deciding:-Clustered Indexes and the WITHOUT ROWID Optimization ), automatically creates a unique row identifier column i.e ROWID.
If you code a column as columnname INTEGER PRIMARY KEY or columnname INTEGER PRIMARY KEY AUTOINCREMENT then columnname becomes an alias for ROWID, in which case the unique value will be provided, if you do not provide a value when inserting.
However, if you were to specify timestamp INTEGER PRIMARY KEY and provide the current date/time as a value for the column when inserting, the current date/time would be stored and would also be indexed (it would have to be unique (current date/time would very likely be)).
So I'd suggest that a log entry need only be something like CREATE TABLE log (timestamp INTEGER PRIMARY KEY, eventref INTEGER);, where eventref is a reference to the event type.
As _id is required at times e.g. for a CursorAdapter then you could specify the columns to be extracted as *, timestemp as _id (3 columns timestamp, eventref and _id (timestamp and _id would be identical))or timstemp as _id, * (3 columns but _id, timestamp and eventref) or timestamp as _id, eventref (2 columns _id and eventref).
So using this model as the basis would minimise columns and be indexed automatically.
An example
You may have the events table as :-
Along with log table as :-
A query such as SELECT * FROM log JOIN events WHERE eventref = _id would give:-
Note! made up timestamps for illustration purposes
I'm having a hard time understanding how to use full text search (FTS) with Android. I've read the SQLite documentation on the FTS3 and FTS4 extensions. And I know it's possible to do on Android. However, I'm having a hard time finding any examples that I can comprehend.
The basic database model
A SQLite database table (named example_table) has 4 columns. However, there is only one column (named text_column) that needs to be indexed for a full text search. Every row of text_column contains text varying in length from 0 to 1000 words. The total number of rows is greater than 10,000.
How would you set up the table and/or the FTS virtual table?
How would you perform an FTS query on text_column?
Additional notes:
Because only one column needs to be indexed, only using an FTS table (and dropping example_table) would be inefficient for non-FTS queries.
For such a large table, storing duplicate entries of text_column in the FTS table would be undesirable. This post suggests using an external content table.
External content tables use FTS4, but FTS4 is not supported before Android API 11. An answer can assume an API >= 11, but commenting on options for supporting lower versions would be helpful.
Changing data in the original table does not automatically update the FTS table (and vice versa). Including triggers in your answer is not necessary for this basic example, but would be helpful nonetheless.
Most Basic Answer
I'm using the plain sql below so that everything is as clear and readable as possible. In your project you can use the Android convenience methods. The db object used below is an instance of SQLiteDatabase.
Create FTS Table
db.execSQL("CREATE VIRTUAL TABLE fts_table USING fts3 ( col_1, col_2, text_column )");
This could go in the onCreate() method of your extended SQLiteOpenHelper class.
Populate FTS Table
db.execSQL("INSERT INTO fts_table VALUES ('3', 'apple', 'Hello. How are you?')");
db.execSQL("INSERT INTO fts_table VALUES ('24', 'car', 'Fine. Thank you.')");
db.execSQL("INSERT INTO fts_table VALUES ('13', 'book', 'This is an example.')");
It would be better to use SQLiteDatabase#insert or prepared statements than execSQL.
Query FTS Table
String[] selectionArgs = { searchString };
Cursor cursor = db.rawQuery("SELECT * FROM fts_table WHERE fts_table MATCH ?", selectionArgs);
You could also use the SQLiteDatabase#query method. Note the MATCH keyword.
Fuller Answer
The virtual FTS table above has a problem with it. Every column is indexed, but this is a waste of space and resources if some columns don't need to be indexed. The only column that needs an FTS index is probably the text_column.
To solve this problem we will use a combination of a regular table and a virtual FTS table. The FTS table will contain the index but none of the actual data from the regular table. Instead it will have a link to the content of the regular table. This is called an external content table.
Create the Tables
db.execSQL("CREATE TABLE example_table (_id INTEGER PRIMARY KEY, col_1 INTEGER, col_2 TEXT, text_column TEXT)");
db.execSQL("CREATE VIRTUAL TABLE fts_example_table USING fts4 (content='example_table', text_column)");
Notice that we have to use FTS4 to do this rather than FTS3. FTS4 is not supported in Android before API version 11. You could either (1) only provide search functionality for API >= 11, or (2) use an FTS3 table (but this means the database will be larger because the full text column exists in both databases).
Populate the Tables
db.execSQL("INSERT INTO example_table (col_1, col_2, text_column) VALUES ('3', 'apple', 'Hello. How are you?')");
db.execSQL("INSERT INTO example_table (col_1, col_2, text_column) VALUES ('24', 'car', 'Fine. Thank you.')");
db.execSQL("INSERT INTO example_table (col_1, col_2, text_column) VALUES ('13', 'book', 'This is an example.')");
(Again, there are better ways in do inserts than with execSQL. I am just using it for its readability.)
If you tried to do an FTS query now on fts_example_table you would get no results. The reason is that changing one table does not automatically change the other table. You have to manually update the FTS table:
db.execSQL("INSERT INTO fts_example_table (docid, text_column) SELECT _id, text_column FROM example_table");
(The docid is like the rowid for a regular table.) You have to make sure to update the FTS table (so that it can update the index) every time you make a change (INSERT, DELETE, UPDATE) to the external content table. This can get cumbersome. If you are only making a prepopulated database, you can do
db.execSQL("INSERT INTO fts_example_table(fts_example_table) VALUES('rebuild')");
which will rebuild the whole table. This can be slow, though, so it is not something you want to do after every little change. You would do it after finishing all the inserts on the external content table. If you do need to keep the databases in sync automatically, you can use triggers. Go here and scroll down a little to find directions.
Query the Databases
String[] selectionArgs = { searchString };
Cursor cursor = db.rawQuery("SELECT * FROM fts_example_table WHERE fts_example_table MATCH ?", selectionArgs);
This is the same as before, except this time you only have access to text_column (and docid). What if you need to get data from other columns in the external content table? Since the docid of the FTS table matches the rowid (and in this case _id) of the external content table, you can use a join. (Thanks to this answer for help with that.)
String sql = "SELECT * FROM example_table WHERE _id IN " +
"(SELECT docid FROM fts_example_table WHERE fts_example_table MATCH ?)";
String[] selectionArgs = { searchString };
Cursor cursor = db.rawQuery(sql, selectionArgs);
Further Reading
Go through these documents carefully to see other ways of using FTS virtual tables:
SQLite FTS3 and FTS4 Extensions (SQLite docs)
Storing and Searching for Data (Android docs)
Additional Notes
Set operators (AND, OR, NOT) in SQLite FTS queries have Standard Query Syntax and Enhanced Query Syntax. Unfortunately, Android apparently does not support the Enhanced Query Syntax (see here, here, here, and here). That means mixing AND and OR becomes difficult (requiring the use of UNION or checking PRAGMA compile_options it seems). Very unfortunate. Please add a comment if there is an update in this area.
Don't forget when using content from to rebuild the fts table.
I do this with a trigger on update, insert, delete
I am making a dictionary of over 20,000 words in it. So, to make it work faster when search data, i am using fts3 table to do it.
my select query:
Cursor c=db.rawQuery("Select * from data where Word MATCH '"+word+"*'", null);
Using this query, it will show all the word that contain 'word' , but what i want is to get only the word that contain the beginning of the searching word.
Mean that i want it work like this query:
Cursor c=db.rawQuery("Select * from data where Word like '"+word+"%'", null);
Ex: I have : apple, app, and, book, bad, cat, car.
when I type 'a': i want it to show only: apple, app, and
What can i solve with this?
table(_id primary key not null autoincrement, word text)
FTS table does not use the above attributes. It ignores data type. It does not auto increment columns other than the hidden rowid column. "_id" will not act as a primary key here. Please verify that you are implementing an FTS table
https://www.sqlite.org/fts3.html
a datatype name may be optionally specified for each column. This is
pure syntactic sugar, the supplied typenames are not used by FTS or
the SQLite core for any purpose. The same applies to any constraints
specified along with an FTS column name - they are parsed but not used
or recorded by the system in any way.
As for your original question, match "abc*" already searches from the beginning of the word. For instance match "man*" will not match "woman".
FTS supports searching for the beginning of a string with ^:
SELECT * FROM FtsTable WHERE Word MATCH '^word*'
However, the full-text search index is designed to find words inside larger texts.
If your Word column contains only a single word, your query is more efficient if you use LIKE 'a%' and rely on a normal index.
To allow an index to be used with LIKE, the table column must have TEXT affinity, and the index must be declared as COLLATE NOCASE (because LIKE is not case sensitive):
CREATE TABLE data (
...
Word TEXT,
...
);
CREATE INDEX data_Word_index ON data(Word COLLATE NOCASE);
If you were to use GLOB instead, the index would have to be case sensitive (the default).
You can use EXPLAIN QUERY PLAN to check whether the query uses the index:
sqlite> EXPLAIN QUERY PLAN SELECT * FROM data WHERE Word LIKE 'a%';
0|0|0|SEARCH TABLE data USING INDEX data_Word_index (Word>? AND Word<?)