I have a link table event_user with a composite key based on an user_id and event_id.
I have an array of (user_id, event_id) pairs in Java where i want to ensure that there doesn't exists any other (user_id, event_id) pair in the table that doesn't exists in the array.
In other tables, i just created a string of ids, and then i created the following query
DELETE FROM tablename WHERE column NOT IN ( 1 , 2 , ... n)
However this behavior with NOT IN can not be achieved with multiple columns.
How can achieve that using Java and sqlite efficiently?
Thanks in advance.
In SQLite, IN works only with a single column.
So you have to write out all comparisons explicitly:
DELETE FROM TableName
WHERE (Col1 != 1 OR Col2 != 10)
AND (Col1 != 2 OR Col2 != 20)
AND ...;
SQLite can build an index for a large IN list, but this is not done for large expressions like this.
So if the query becomes too big, you can put the IDs into a temporary table, and use a subquery to check for matching rows:
DELETE FROM TableName
WHERE NOT EXISTS (SELECT 1
FROM IDsToKeep
WHERE IDsToKeep.Col1 = TableName.Col1
AND IDsToKeep.Col2 = TableName.Col2);
For efficiency, the temporary table should be indexed (or just be a WITHOUT ROWID table with a PK on both columns).
Related
In my Android app, I create a FULLTEXT table like this:
CREATE VIRTUAL TABLE products USING fts3 (
_id integer PRIMARY KEY,
product_name text NOT NULL,
...
)
And I add this index:
CREATE INDEX product_name_index ON products (product_name)
The app populates the table with various products, each with a unique _id value.
However, when I then try to insert an already-existing product ID (using an _id value that is already in the table, but with a different product_name value) like this:
long rowId = db.insertOrThrow("products", null, contentValues);
a new row is added to the table (with a brand new rowId value returned)!
I expected the insertOrThrow command to fail, so where am I going wrong? Is it something to do with the fact that it's a FULLTEXT table or could the index I specified on the product_name column be messing things up somehow?
I read this section about INTEGER PRIMARY KEY, but unfortunately I'm none the wiser.
Update
When I try to perform the same operation on a standard (non-FULLTEXT) table, then the insertOrThrow command results in the expected SQLiteConstraintException.
I think the issue might be that an FTS table has the concept of a docid and a rowid column and specifying null for the docid results in that being given a value.
as per :-
There is one other subtle difference between "docid" and the normal
SQLite aliases for the rowid column.
Normally, if an INSERT or UPDATE
statement assigns discrete values to two or more aliases of the rowid
column, SQLite writes the rightmost of such values specified in the
INSERT or UPDATE statement to the database.
However, assigning a
non-NULL value to both the "docid" and one or more of the SQLite rowid
aliases when inserting or updating an FTS table is considered an
error. See below for an example.
1.3. Populating FTS Tables
How can I find the tables having column Blob type in Sqlite. I need to get the table names from which I get the column blob type and then want to see the total no. of records where the blob is not empty.
If you wanted tables that have a column defined as a blob then you could use
SELECT * FROM sqlite_master WHERE sql LIKE '%blob%';
as the basis for determining the tables. e.g. this could return results such as :-
However, this does not necessarily find all values that are stored as blobs. This is because with the exception of the rowid column or an alias thereof, any type of value (blob included) can be stored in any column.
e.g. consider the following :-
DROP TABLE IF EXISTS not_a_blob_table;
CREATE TABLE IF NOT EXISTS not_a_blob_table (col1 TEXT, col2 INTEGER, col3 REAL, col4 something_or_other);
INSERT INTO not_a_blob_table VALUES
('test text',123,123.4567,'anything'), -- Insert using types as defined
(x'00',x'12',x'34',x'1234567890abcdefff00') -- Insert with all columns as blobs
;
SELECT typeof(col1),typeof(col2),typeof(col3),typeof(col4) FROM not_a_blob_table;
This results in :-
If you want to find all blobs then you would need to process all columns from all rows of all tables based upon a check for the column type. This could perhaps be based upon :-
SELECT typeof(col1),typeof(col2),typeof(col3),typeof(col4),* FROM not_a_blob_table
WHERE typeof(col1) = 'blob' OR typeof(col2) = 'blob' OR typeof(col3) = 'blob' OR typeof(col4) = 'blob';
Using the table above this would result (only the 2nd row has blobs) in :-
A further complication is what you mean by not empty, null obviously. However what about x'00'? or if you used a default of zeroblob(0) ?.
zeroblob(N)
The zeroblob(N) function returns a BLOB consisting of N bytes of 0x00. SQLite manages these zeroblobs very efficiently. Zeroblobs can
be used to reserve space for a BLOB that is later written using
incremental BLOB I/O. This SQL function is implemented using the
sqlite3_result_zeroblob() routine from the C/C++ interface.
If null though then this wouldn't have a type of blob, instead it's type would be null, which could complicate matters if checking for all values stored as blobs.
You may wish to consider having a look at the code from Are there any methods that assist with resolving common SQLite issues?
as this could well be the basis for what you want.
You also wish to have a look at typeof(X) and zeroblob(N).
I am using SQLite Database for my application. I have 4 columns- Student_Name,Student_Enroll, Student_Mob, Student_Address in my database. Now I can add new record if and only if one of four column value is different or all values are different. If all column values are same then no new record should be generated.
Can you please guide me to solve this issue?
To enforce that a set of columns must be unique, add a UNIQUE constraint:
create table Students (
/* ID INTEGER PRIMARY KEY, */
Student_Name TEXT,
Student_Enroll TEXT,
Student_Mob TEXT,
Student_Address TEXT,
UNIQUE (Student_Name, Student_Enroll, Student_Mob, Student_Address)
);
This allows new rows only if at least one of the four columns has a different value.
With a plain INSERT, attempting to insert a duplicate row will result in an error. If you simply want to ignore it instead, use INSERT OR IGNORE:
INSERT OR IGNORE INTO Students ...;
Despite of set your column as UNIQUE you also need to resolve the conflict created on each column when you try to insert new data.
To do so, define the behavior to solve the conflict:
"CREATE TABLE table (your columns here...(UNIQUE unique colums here...) ON CONFLICT REPLACE);"
During Create Database line insert UNIQUE ...for each column to insert only unique record.
Solution 1: (Simple)
Define all columns as unique:
create table TableName (id integer primary key autoincrement,
Student_Name text not null unique,
Student_Enroll text not null unique,
Student_Mob text not null unique);
You can add Student_Address as well, if you need to
Solution 2: (bit complex)
Use AND Operator with WHERE clause
INSERT INTO TableName (Student_Name, Student_Enroll, Student_Mob)
SELECT varStudentName, varStudentEnroll, varStudentMob
WHERE NOT EXISTS(SELECT 1 FROM TableName WHERE Student_Name = varStudentName OR Student_Enroll = varStudentEnroll OR Student_Mob = varStudentMob );
//If a record already contains a row, then the insert operation will be ignored.
You can find more information at the sqlite manual.
Live Example:
Open SQLite Online
Paste following code:
INSERT INTO demo (id,name,hint)
SELECT 4, 'jQuery', 'is a cross-platform JavaScript library designed to simplify the client-side scripting of HTML'
WHERE NOT EXISTS(SELECT 1 FROM demo WHERE name = 'jQuery' OR hint = 'is a cross-platform JavaScript library designed to simplify the client-side scripting of HTML' );
SELECT * from demo
Hit RUN
This won't insert 4th record and if you modify both values of WHERE clause then record will be inserted.
I'm having a hard time understanding how to use full text search (FTS) with Android. I've read the SQLite documentation on the FTS3 and FTS4 extensions. And I know it's possible to do on Android. However, I'm having a hard time finding any examples that I can comprehend.
The basic database model
A SQLite database table (named example_table) has 4 columns. However, there is only one column (named text_column) that needs to be indexed for a full text search. Every row of text_column contains text varying in length from 0 to 1000 words. The total number of rows is greater than 10,000.
How would you set up the table and/or the FTS virtual table?
How would you perform an FTS query on text_column?
Additional notes:
Because only one column needs to be indexed, only using an FTS table (and dropping example_table) would be inefficient for non-FTS queries.
For such a large table, storing duplicate entries of text_column in the FTS table would be undesirable. This post suggests using an external content table.
External content tables use FTS4, but FTS4 is not supported before Android API 11. An answer can assume an API >= 11, but commenting on options for supporting lower versions would be helpful.
Changing data in the original table does not automatically update the FTS table (and vice versa). Including triggers in your answer is not necessary for this basic example, but would be helpful nonetheless.
Most Basic Answer
I'm using the plain sql below so that everything is as clear and readable as possible. In your project you can use the Android convenience methods. The db object used below is an instance of SQLiteDatabase.
Create FTS Table
db.execSQL("CREATE VIRTUAL TABLE fts_table USING fts3 ( col_1, col_2, text_column )");
This could go in the onCreate() method of your extended SQLiteOpenHelper class.
Populate FTS Table
db.execSQL("INSERT INTO fts_table VALUES ('3', 'apple', 'Hello. How are you?')");
db.execSQL("INSERT INTO fts_table VALUES ('24', 'car', 'Fine. Thank you.')");
db.execSQL("INSERT INTO fts_table VALUES ('13', 'book', 'This is an example.')");
It would be better to use SQLiteDatabase#insert or prepared statements than execSQL.
Query FTS Table
String[] selectionArgs = { searchString };
Cursor cursor = db.rawQuery("SELECT * FROM fts_table WHERE fts_table MATCH ?", selectionArgs);
You could also use the SQLiteDatabase#query method. Note the MATCH keyword.
Fuller Answer
The virtual FTS table above has a problem with it. Every column is indexed, but this is a waste of space and resources if some columns don't need to be indexed. The only column that needs an FTS index is probably the text_column.
To solve this problem we will use a combination of a regular table and a virtual FTS table. The FTS table will contain the index but none of the actual data from the regular table. Instead it will have a link to the content of the regular table. This is called an external content table.
Create the Tables
db.execSQL("CREATE TABLE example_table (_id INTEGER PRIMARY KEY, col_1 INTEGER, col_2 TEXT, text_column TEXT)");
db.execSQL("CREATE VIRTUAL TABLE fts_example_table USING fts4 (content='example_table', text_column)");
Notice that we have to use FTS4 to do this rather than FTS3. FTS4 is not supported in Android before API version 11. You could either (1) only provide search functionality for API >= 11, or (2) use an FTS3 table (but this means the database will be larger because the full text column exists in both databases).
Populate the Tables
db.execSQL("INSERT INTO example_table (col_1, col_2, text_column) VALUES ('3', 'apple', 'Hello. How are you?')");
db.execSQL("INSERT INTO example_table (col_1, col_2, text_column) VALUES ('24', 'car', 'Fine. Thank you.')");
db.execSQL("INSERT INTO example_table (col_1, col_2, text_column) VALUES ('13', 'book', 'This is an example.')");
(Again, there are better ways in do inserts than with execSQL. I am just using it for its readability.)
If you tried to do an FTS query now on fts_example_table you would get no results. The reason is that changing one table does not automatically change the other table. You have to manually update the FTS table:
db.execSQL("INSERT INTO fts_example_table (docid, text_column) SELECT _id, text_column FROM example_table");
(The docid is like the rowid for a regular table.) You have to make sure to update the FTS table (so that it can update the index) every time you make a change (INSERT, DELETE, UPDATE) to the external content table. This can get cumbersome. If you are only making a prepopulated database, you can do
db.execSQL("INSERT INTO fts_example_table(fts_example_table) VALUES('rebuild')");
which will rebuild the whole table. This can be slow, though, so it is not something you want to do after every little change. You would do it after finishing all the inserts on the external content table. If you do need to keep the databases in sync automatically, you can use triggers. Go here and scroll down a little to find directions.
Query the Databases
String[] selectionArgs = { searchString };
Cursor cursor = db.rawQuery("SELECT * FROM fts_example_table WHERE fts_example_table MATCH ?", selectionArgs);
This is the same as before, except this time you only have access to text_column (and docid). What if you need to get data from other columns in the external content table? Since the docid of the FTS table matches the rowid (and in this case _id) of the external content table, you can use a join. (Thanks to this answer for help with that.)
String sql = "SELECT * FROM example_table WHERE _id IN " +
"(SELECT docid FROM fts_example_table WHERE fts_example_table MATCH ?)";
String[] selectionArgs = { searchString };
Cursor cursor = db.rawQuery(sql, selectionArgs);
Further Reading
Go through these documents carefully to see other ways of using FTS virtual tables:
SQLite FTS3 and FTS4 Extensions (SQLite docs)
Storing and Searching for Data (Android docs)
Additional Notes
Set operators (AND, OR, NOT) in SQLite FTS queries have Standard Query Syntax and Enhanced Query Syntax. Unfortunately, Android apparently does not support the Enhanced Query Syntax (see here, here, here, and here). That means mixing AND and OR becomes difficult (requiring the use of UNION or checking PRAGMA compile_options it seems). Very unfortunate. Please add a comment if there is an update in this area.
Don't forget when using content from to rebuild the fts table.
I do this with a trigger on update, insert, delete
I have a table with the following schema:
CREATE TABLE table (
msg_id TEXT,
thread_id TEXT,
.
.
.
date INTEGER
)
I need to retrieve the most recent n msg_id per unique value of thread_id. Is there a way to do it using a single query or will I need to query the database to get the most recent distinct thread_ids, then query the database again PER unique thread_id? I recall reading somewhere that multiple database queries can get expensive.
You could use a correlated subquery. For example, for N = 5 :
select *
from YourTable yt1
where 5 <
(
select count(*)
from YourTable yt2
where yt2.thread_id = yt1.thread_id
and yt2.msg_id < yt1.thread_id
)
This is not too fast, so you might be better of with multiple queries.