Interesting issue while using SQLite in Android. I am seeing an inconsistency in the string length and quoting of a string between what is stored in the database and the materialized value seen in Java.
We are using an ORM called SugarORM to query the DB, but I've traced the offending code to the internal android.database.sqlite.SQLiteCursor class used within SugarORM, specifically the cursor.getString(columnIndex) method.
I have a string in the database that is an ISO data string 2019-03-25T19:19:39.664Z and is stored in a VARCHAR column . I have confirmed using DB Browser for SQLite that the length of the string as its stored in the database is indeed 24 characters. SELECT LENGTH(MyStringColumn) FROM MyTable WHERE ...
When I get the value of this string via cursor.getString(columnIndex), it is returning the string "2019-03-25T19:19:39.664Z". Notice the leading and trailing quotes. Java reports to me that the string is 26 characters long.
Any value that I store in this column that is not an ISO data does not have this behavior. I tried tracing the SQLiteCursor source back, but ultimately it ends up being a Native method and that's where my skill set stops.
Can anyone explain what might be going on here? I am probably just going to write a wrapper around my queries to get rid of the quotes, but its all very perplexing. The date string is being fed to a JavaScript interpreter and causing it to fail when creating a JavaScript Date object.
If it helps, I have replicated the behavior on both my S7 physical device and a Pixel 6 emulator.
As a quick get around you could use :-
SELECT length(replace(mystringcolumn,'"','')) FROM mytable;
or before using the original SELECT use :-
UPDATE mytable SET mystringcolumn = replace(mystringcolumn,'"','');
If this doesn't fix the issue, then for some reason it is the code that retrieves the data that is at fault.
e.g. consider :-
DROP TABLE IF EXISTS mytable;
CREATE TABLE IF NOT EXISTS mytable (mystringcolumn VARCHAR);
INSERT INTO mytable VALUES('2019-03-25T19:19:39.664Z'),('"2019-03-25T19:19:39.664Z"');
SELECT length(mystringcolumn), length(replace(mystringcolumn,'"','')) FROM mytable;
which results in :-
i.e. The 2nd row, 2nd column retrieves the appropriate value by using the replace function to strip of the quotes, if they exist.
As to why the quotes exist could depend upon either the way that the data is inserted (perhaps you have inadvertenly coded the quotes but the db being looked at isn't the actual database as copied from the App) or the way in which the data is being retrieved that for some reason adds them.
I don't believe it likely that the Cursor getString method has a bug in which the quotes are added, otherwise such an issue would likely be a recurring issue.
Related
I have an SQLite DB where I perform a query like
Select * from table where col_name NOT IN ('val1','val2')
Basically I'm getting a huge list of values from server and I need to select the ones which is not present in the list given.
Currently its working fine, No issues. But the number of values from server becomes huge as the server DB is getting updated frequently.
So, I may get thousands of String values which I need to pass to the NOT IN
My question is, Will it cause any perfomance issue in the future? Does the NOT IN parameter have any size restriction? (like max 10000 values you can check)?
Will it cause any crash at some point?
This is an official reference about various limitation in sqlite. I think the Maximum Length Of An SQL Statement may related to your case. Default value is 1000000, and it is adjustable.
Except this I don't think any limitation existed for numbers of parameter of NOT IN clause.
With more than a few values to test for, you're better off putting them in a table that has an index on the column holding them. Then things like
SELECT *
FROM table
WHERE col_name NOT IN (SELECT value_col FROM value_table);
or
SELECT *
FROM table AS t
WHERE NOT EXISTS (SELECT 1 FROM value_table WHERE value_col = t.col_name);
will be reasonably efficient no matter how many records are in value_table because that index will be used to find entries.
Plus, of course, it makes it a lot easier to re-use prepared statements because you don't have to create a new one and re-bind every value (You are using prepared statements with placeholders for these values, right, and not trying to put their contents inline into a string?) every time you add a value to the ones you need to check. You just insert it into value_table instead.
Yes, there is a limit of 999 arguments as reported in the official documentation: https://www.sqlite.org/limits.html#max_variable_number
I need to change the data type for my SQLite. I am worry that it might effect users who update the App. However, after reading the SQLite document in the following link
https://sqlite.org/datatype3.html
It would seem changing the data type when creating a table column shouldn't break the App. From what I read it seems unlike other SQL database engines, SQLite datatype is associated with the value itself and not the column data type that I initially assigned.
I was going to alter the column data type when user updates the App, but it doesn't seem necessary (nor possible without dropping and recreating table). Am I reading this correctly or am I making a mistake?
The App seems to work well when I test updating, but I want to make sure I am not missing anything. Any feedback is appreciated.
Changing the type name in the column definition can affect the affinity.
This might change the type of some values (for example, attempting to store the string '123' in an INTEGER column will result in the number 123), and might change how comparisons work (WHERE SomeColumn = ? will try to convert the value to the same type as the column's affinity).
So you should change the type name only if you are sure that your app handles the values in this column correctly.
I need to copy data from one table to another. Doing so, I'd like to set manually some static values that will override data from source table. Here is example:
INSERT INTO users (id_usr, name, description, mod_date, user_type)
SELECT id_usr, name, description, '2014-03-19 15:15:09', 'public'
FROM users_temp
WHERE user_type="" OR user_type IS NULL;
'Datetime' string and 'public' are those static values.
I'd like to know if it is possible, because on Android phone(SQLite from external native library) it doesn't copy any records, but on Windows it works fine.
I've found the problem. My users_temp table was actually a virtual table created from file. While creating that virtual table I have defined wrong charset. It caused that, at the end of every value there was added some kind of white character and condition:
WHERE user_type="" OR user_type IS NULL;
was never fulfilled.
After fixing the problem, statement is executing correctly.
There is a thing that I can not understand about Cursor.getType(), can anyone explain why do I get this stupid exception when I want to get columns type if cursor has no record but there are columns? I mean, if cursor has record there is no problem, I can use getType method to get columns type without any problem but if there is no any record it throws this exception out.
The question is why must I need records to get columns type? Why just knowing columns name is not enough to get types of it? Is this ridiculous?
It's because SQLite uses dynamic typing:
Most SQL database engines (every SQL database engine other than
SQLite, as far as we know) uses static, rigid typing. With static
typing, the datatype of a value is determined by its container - the
particular column in which the value is stored.
SQLite uses a more general dynamic type system. In SQLite, the
datatype of a value is associated with the value itself, not with its
container.
http://www.sqlite.org/datatype3.html
So no value, no data type.
I have a .csv file that I turned into an SQLite database with the first column as a date and time (as a TEXT datatype because it's in the format 7/20/2011 12:00:00 PM), the second column as the subsecond (a REAL, like 0.142857), and the rest of the columns as data about that time. I will be using it as a read-only db in my Android app, in which the user selects a start time and an end time and then the data is displayed graphically for that time period.
This is my first time using a database and I'm not sure how to work with this. I believe I need a unique NUMERIC primary key for each row but simply labeling them by row number is a waste of space. The first column is not unique and is a TEXT datatype anyway, though if the subsecond was to be somehow incorporated then it would be unique. But I really can't re-write the database because it has 65534 rows... How do you suggest I access specific times in this database?
In Android you need a column named _ID in your database (else you'll face some issues later on). You will use that as the primary key.
Dates and times can be stored in SQLite in the form of a text column in the following format (See http://www.sqlite.org/datatype3.html):
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS")
If your database is static, simply use a tool such as SQLite Database Browser to convert it to a format convenient for Android.
If your database is local and external(not remote), than you must have _id and another another table android_metadata which will hold the locale.
If your database was remote. Yes, you can it is only matter of speed if you are write, since you don't. Using WHERE clause will do the work.
Every date can be converted to numeric timestamp quite easy:
Date date = new Date();
long timestamp = date.getTime();
Date otherDate = new Date(timestamp);
Numbers are MUCH easier and faster to process than text fields. If you are completely sure, that you have unique data within column you can use it as primary key.
Importing csv file into table should be also easy:
//just open file in some known way and read it line by line
// we have declared String[] data, String line is line read from your csv somewhere earlier
// in code
data = line.split(",");
//here you have to process values, and execute insert statement
You have to create indexes on every column which will be used to search or order data. Please be also aware, that rows in table has no "default", "natural" or any other order. If you execute this same select statement twice you can get two totally different results in meaning of sorting. So simple select statement should look like that:
select
_id, other_colum_name, some_other_column
from
table_name
where
column_name = 5
and other_column_name = 3
order by
column_name;