Develop Reference

Does SQLite `NOT IN` parameter have any size limit?

I have an SQLite DB where I perform a query like
Select * from table where col_name NOT IN ('val1','val2')
Basically I'm getting a huge list of values from server and I need to select the ones which is not present in the list given.
Currently its working fine, No issues. But the number of values from server becomes huge as the server DB is getting updated frequently.
So, I may get thousands of String values which I need to pass to the NOT IN
My question is, Will it cause any perfomance issue in the future? Does the NOT IN parameter have any size restriction? (like max 10000 values you can check)?
Will it cause any crash at some point?

This is an official reference about various limitation in sqlite. I think the Maximum Length Of An SQL Statement may related to your case. Default value is 1000000, and it is adjustable.
Except this I don't think any limitation existed for numbers of parameter of NOT IN clause.

With more than a few values to test for, you're better off putting them in a table that has an index on the column holding them. Then things like
SELECT *
FROM table
WHERE col_name NOT IN (SELECT value_col FROM value_table);
or
SELECT *
FROM table AS t
WHERE NOT EXISTS (SELECT 1 FROM value_table WHERE value_col = t.col_name);
will be reasonably efficient no matter how many records are in value_table because that index will be used to find entries.
Plus, of course, it makes it a lot easier to re-use prepared statements because you don't have to create a new one and re-bind every value (You are using prepared statements with placeholders for these values, right, and not trying to put their contents inline into a string?) every time you add a value to the ones you need to check. You just insert it into value_table instead.

Yes, there is a limit of 999 arguments as reported in the official documentation: https://www.sqlite.org/limits.html#max_variable_number

SQLite3 on Android not adding float numbers correctly

I am having some trouble running a few simple statements on SQLite3 on Android.
For example:
SELECT 1234 + 0.001
That should return 1234.001, right? SQLite, however, returns 1234 both on the Emulator (v21) and on a real device (v19).
Considering 1234 is stored on a column called Field1, type REAL, on Table1, I have tried all the options below:
SELECT Field1 + 0.001 FROM Table1
SELECT (Field1 * 1.000) + 0.001 FROM Table1
SELECT CAST(Field1 as FLOAT) + 0.001 FROM Table1
SELECT CAST(Field1 as REAL) + 0.001 FROM Table1
SELECT Field1 + 0.001 from Table1
SELECT CAST((Field1 + 0.001) as REAL) FROM Table1
Nothing seems to work, and in every single case I am getting 1234 instead of 1234.001. I need to get 1234.001 from 1234 in a query but SQLite3 isn't being helpful.
Another thing I just found out: if "Field1" <= 999, the SELECT works as expected and I get 999.001 in the result. Anything >= 1000 gives me an integer instead.
Can you please help me how to solve this?

Actually, I found out that the problem is not what it seems. The numbers are being treated as x.001, but not shown as such. Internally, the numbers were being properly stored in binary, but not all of the decimal part was being shown. After just casting them as text, I was able to see that the decimal part was all there.
Anyhow, thank you for your time and answers.

This is an issue of precision and how numbers are stored in a computer or for that matter in sqlite3. Numbers are stored in binary format, which means that by converting a decimal number to a binary number you loose some precision (in real numbers) read more here. Now, what are your options if you want the addition to yield 1234.001 as in your example?
A) Storing the number as a string.
You can always store the values '1234' and '0.001' as VARCHARs and in JAVA code parse this values to BigDecimals and perform your additions there. For more info on parsing, check out this link. The drawback of this method is that it will consume a lot of more storage space in your database and parsing operations aren't that fast either. Before using this method, think if this drawback will impact negatively the performance of your application.
B) Establish a MAXIMUM_PRECISION and store them as INTEGERs.
By default SQLITE3 stores INTEGERs using 8 bytes. This means that the largest integer that you can store is of the order of 10^19. Now, by storing integers you dont loose precision; so we can take advantage of this. Let's say that your MAXIMUM_PRECISION is 0.001 (as in your example) then you need to multiply this number by one thousand to get a 1. So what if instead of representing 1234.001 as a real, we represent it as 1234001 an int? that way we can store the int safely in sqlite3 and make sure that the operations work properly. In your code you can later on parse the number as a String and format it to display it to an user or you can parse it to a BigDecimal in order to keep precision. Of course, this will limit you to a maximum number of the order of 10^16; again check your requirements to see if this trick will work for your app. Please note that a similar trick is used to store currency without loosing precision in sqlit3, for more info see: this link

SQLite. How to SELECT values from TABLE fast?

I am developing dictionary application. It requires incremental search which means that SELECTING should be fast. There are 200000+ rows. Let me, first of all explain, table structure. I have this table:
CREATE TABLE meaning(
key TEXT,
value TEXT,
entries BLOB);
Some times ago I had this index:
CREATE INDEX index_key ON meaning (key)
This query was performed for around ~500ms which was very slow
SELECT value FROM meaning WHERE key LIKE 'boy%' LIMIT 100
Then I dropped this index, created incasesensitive index which helped to improve performance 2-3 times.
CREATE INDEX index_key ON meaning (key COLLATE NOCASE);
Now this query performing for 75ms(min) - 275(ms) which is quite slow for incremental search.
SELECT value FROM meaning WHERE key LIKE 'boy%' LIMIT 100
I have tried to optimize query according to this post.
SELECT value FROM meaning WHERE key >= 'boy' AND key<'boz' LIMIT 100
But this query is performed for 451ms.
EXPLAIN
SELECT value FROM meaning WHERE key LIKE 'boy%' LIMIT 100
This is returning following values:
EXPLAIN QUERY PLAN
SELECT value FROM meaning WHERE key LIKE 'boy%' LIMIT 100
This is returning this value(detail column):
SEARCH TABLE meaning USING INDEX index_key (key>? AND key<?) (~31250 rows)
Actually this values did not give me some sense or key what to optimize.
Is it possible to optimize SELECTion of words to be performed in ~10ms by optimization of this query or creating another table or changing some parameters of SQLite database? Could you suggest me the best way to do this?
PS. Please, do not suggest to use FTS table. In previous version of application I have used FTS. I agree that it is extremely fast. I left FTS table idea for 2 reasons:
It is not giving proper result(it contains the words which user do not need)
It takes more disk space

How should I implement a unique priority field in my database?

I'm creating a database for event objects, each event has a priority (long) which is unique. Objects with lower priority values are favored, newer object are initially assigned higher and higher priorities as the amount of object increases but the user can reassign priorities at will.
My question is should I use just _ID as the priority field? I figure that way when I select them and put them into my ArrayList in Java my events will already be sorted by priority high to low and save me the trouble of having to search. And depending on what algorithms SQLite actually uses selection by priority may be faster since it's also an index. I'm new to SQL and this all seems well and good to me, but there might be some drawback that is plain to someone more experienced.
Also, I'm a bit unclear on how insertion into the middle of a table would work, probably use the update function to increment the row_ids above it but how do I make sure it increments them in the proper order? (so as to not tread on each other).
Something like this?:
UPDATE event_table SET priority = priority + 1 WHERE priority > ?
P.S: I'm doing this on an Android 2.2 in SQLite 3

I think having a priority value of decimal datatype would make it easier to insert into the middle of the table. And not require you to do massive shifting when you're re-sorting.

Why is writing into an in-memory SQLITE database so much slower than using an ArrayList?

I maintain an application that is collecting a lot of information and is storing these information in an ArrayList.
In detail this ArrayList is defined as ArrayList<FileInformation> which has some member like:
private File mFile;
private Long mSize;
private int mCount;
private Long mFilteredSize;
private int mFilteredCount;
private int mNumberOfFilters;
etc.
This approach is working but is not very flexible when I would like to introduce some new functionality. It also has some limitations in terms of memory usage and scale-ability. Because of this I did some tests if a database is the better approach. From the flexibility there is no question, but somehow I'm not able to make it running fast enough to become a real alternative.
Right now the database has just one table like this:
CREATE TABLE ExtContent (
"path" TEXT not null,
"folderpath" TEXT not null,
"filename" TEXT,
"extention" TEXT,
"size" NUMERIC,
"filedate" NUMERIC,
"isfolder" INTEGER not null,
"firstfound" NUMERIC not null,
"lastfound" NUMERIC not null,
"filtered" INTEGER not null
);
The performance issue is immense. Collecting and writing ~14000 items takes ~3mins! when writing into the database and just 4-5secs if written into the ArrayList.
Creating the database in-memory does not make a big difference.
As my experience in terms of SQLITE is rather limited, I started by creating the entries via the android.database.sqlite.SQLiteDatabase.insert methode.
As there was no meaningful difference between a file based and a in-memory database, I guess using BEGIN TRANSACTION and COMMIT TRANSACTION will not make any difference.
Is there some way to optimize this behavior?

Just for clarification, putting BEGIN TRANSACTION and END TRANSACTION will increase the performance greatly. Quoted from http://www.sqlite.org/faq.html#q19 :
SQLite will easily do 50,000 or more INSERT statements per second on an average desktop computer. But it will only do a few dozen transactions per second. By default, each INSERT statement is its own transaction...

I had a similar issue on an app I was coding on the weekend.
Is the data in the database to be included in the app when it's released? If so, bulk inserts aren't they way to go, instead you want to look at creating the database and including it in the assets directory and copying it over to the device. Here's a great link.
Otherwise I'm not sure you can do much to improve performance, this link explains methods on bulk inserting into an SqlLite Database.
Edit: You may also want to post your insert code too.

This is opretty obvious. Assuming you already allocated object to insert into. ( This is the same workload for bot solutions ) Let's compare alternatives:
Inserting in ArrayList does:
- (optional) allocate new chinks of cells for pointers if necessary
- insert object pointer into array list on the end
... really fast
INserting into sqlite:
-prepare insertion query ( I hope you use prepared query, and do not construct it from strings)
-perform database table insertion with modifications of indexes etc.
... a lot of work
Only advantage of database is that you can:
- query it later
- it handles external storage transparently allowing you to have much more entities
But it comes at cost of performance.
Depending on what you are for, there could be better alternatives.
For example, in my android games I store highscore entries in JSON file and utilise
GSON Pull parser / databinding layer ( https://github.com/ko5tik/jsonserializer ) to create objects out of it. Typical load time for 2000 entries from external storage is about 2-3 seconds