I have a sqlite database where all rows have a UUID as the primary key, the db column is defined as a BLOB.
The keys are inserted as byte[] instead of Strings to avoid wasting storage and index spaces. I can insert, update and delete rows using SQLiteDatabase.compileStatement and using bindBlob on the SQLiteStatement but I can't find anyway to bind a blob to the parameters of a SELECT query.
Both SQLiteDatabase.query and .rawQuery expects my WHERE arguments to be Strings which will never match my byte array blobs. I can find my row if I construct my WHERE manually using a BLOB literal like this:
final Cursor query = db.query(getTableName(),
getColumns(),
"id = X'" + bytesToHex(getByteArrayFromUUID(id)) + "'" ,
null,
null,
null,
null);
But then I am vulnerable to SQL injections...
In every other language I have used SQLite in this is not a problem, is there really no way to get a standard prepared SELECT statement with android SQLite?
Most of the APIs in Android sqlite expect the parameter to be strings.
You can use compileStatement() to prepare your query and then use bindBlob() to bind a blob argument to it. Getting useful results out from the query is not easy though, SQLiteStatement has methods for only a few 1x1 result sets.
On the other hand, using a blob as a key doesn't seem like a good idea.
Related
How can I find the tables having column Blob type in Sqlite. I need to get the table names from which I get the column blob type and then want to see the total no. of records where the blob is not empty.
If you wanted tables that have a column defined as a blob then you could use
SELECT * FROM sqlite_master WHERE sql LIKE '%blob%';
as the basis for determining the tables. e.g. this could return results such as :-
However, this does not necessarily find all values that are stored as blobs. This is because with the exception of the rowid column or an alias thereof, any type of value (blob included) can be stored in any column.
e.g. consider the following :-
DROP TABLE IF EXISTS not_a_blob_table;
CREATE TABLE IF NOT EXISTS not_a_blob_table (col1 TEXT, col2 INTEGER, col3 REAL, col4 something_or_other);
INSERT INTO not_a_blob_table VALUES
('test text',123,123.4567,'anything'), -- Insert using types as defined
(x'00',x'12',x'34',x'1234567890abcdefff00') -- Insert with all columns as blobs
;
SELECT typeof(col1),typeof(col2),typeof(col3),typeof(col4) FROM not_a_blob_table;
This results in :-
If you want to find all blobs then you would need to process all columns from all rows of all tables based upon a check for the column type. This could perhaps be based upon :-
SELECT typeof(col1),typeof(col2),typeof(col3),typeof(col4),* FROM not_a_blob_table
WHERE typeof(col1) = 'blob' OR typeof(col2) = 'blob' OR typeof(col3) = 'blob' OR typeof(col4) = 'blob';
Using the table above this would result (only the 2nd row has blobs) in :-
A further complication is what you mean by not empty, null obviously. However what about x'00'? or if you used a default of zeroblob(0) ?.
zeroblob(N)
The zeroblob(N) function returns a BLOB consisting of N bytes of 0x00. SQLite manages these zeroblobs very efficiently. Zeroblobs can
be used to reserve space for a BLOB that is later written using
incremental BLOB I/O. This SQL function is implemented using the
sqlite3_result_zeroblob() routine from the C/C++ interface.
If null though then this wouldn't have a type of blob, instead it's type would be null, which could complicate matters if checking for all values stored as blobs.
You may wish to consider having a look at the code from Are there any methods that assist with resolving common SQLite issues?
as this could well be the basis for what you want.
You also wish to have a look at typeof(X) and zeroblob(N).
I am using SQLite in an Android app which stores, amongst other things, a table that records the mood of the user. The table schema is shown below
CREATE TABLE moods
(
dow INTEGER,
tsn INTEGER,
lato INTEGER,
agitation TEXT DEFAULT '{}',
preoccupancy TEXT DEFAULT '{}',
tensity TEXT DEFAULT '{}',
taps TEXT DEFAULT '{}'
);
Unlike, say, Postgres, SQLite does not have a dedicated jsonb storage type. To quote
Backwards compatibility constraints mean that SQLite is only able to store values that are NULL, integers, floating-point numbers, text, and BLOBs. It is not possible to add a sixth "JSON" type.
SQLite JSON1 extension documentation
On the SQLite commandline processor I can populate this table using a simple INSERT statement such as this one
INSERT INTO moods (dow,tsn,lato,agitation,preoccupancy,tensity,taps)
VALUES(1,0,20,'{"A2":1}','{"P4":2}','{"T4":3}','{"M10":4}');
since it accepts both single and double quotes for strings.
In order to accomplish the same thing in my Hybrid Android app I do the following
String ag = JOString("A" + DBManage.agitation,1);
String po = JOString("P" + (DBManage.preoccupancy + 15),1);
String te = JOString("T" + DBManage.tensity,1);
which returns strings such as {"P4":2} etc. My next step is as follows
ContentValues cv = new ContentValues();
cv.put("dow",0);
cv.put("tsn",1);
cv.put("lato",2);
cv.put("agitation",ag);
cv.put("preoccupancy",po);
cv.put("tensity",te);
long rowid = db.insert("moods",null,cv);
which works. However, upon examining the stored values I find that what has gone in is in fact
{"dow":0,"tsn":5,"lato":191,"tensity":"{\"T0\":1}","agitation":"
{\"A1\":1}","preoccupancy":"{\"P15\":1}","taps":"{}"}]
Unless I am misunderstanding something here the underlying ContentValues.put or the SQLiteDatabase.insert implementation has taken it upon itself to escape the quoted strings. Perhaps this is to be expected given that the benefit of going down the SQLiteDatabase.insert route with ContentValues is protection from SQL injection attacks. However, in the present instance it is being a hindrance not a help.
Short of executing raw SQL via execSQL is there another way to get the JSON into the SQLite database table here?
I should add that simply replacing the double quotes with single quotes
String ag = JOString("A" + DBManage.agitation,1);
ag = ag.replaceAll("\"","'");
will not work since a subsequent attempt to extract JSON from the table
SELECT ifnull((select json_extract(preoccupancy,'$.P4') from moods where dow = '1' AND tsn = '0'),0);
would result in the error malformed JSON being emitted by the SQLite JSON1 extension.
I have a ContentProvider that uses a custom CursorFacory in debug to print out the SQL queries (for debugging).
A certain query was returning 0 rows, while I knew there were rows that should have been included. So I copied the query from my logs, replaced the bind values and ran it in sqlite3 shell on the device and got the correct result.
The Query Code
cr.query (contentUri,
Projection.columns,
FeedColumns.FEED_TYPE + "=? AND " +
FeedColumns.SUB_TYPE + "=? AND " +
ProfileUpdateFeedItem.UPDATED_FIELD + "=? AND " +
FeedColumns.IS_NOTIFIED + "=?",
new String[] {FeedType.USER, // 2
WallPostData.WallPostType.PROFILE_UPDATE, // 1
ProfileUpdateData.ProfileField.STATUS, // 0
SQLBoolean.FALSE // 0
},
FeedColumns.CREATED + " ASC");
From the logs:
07-04 12:48:51.339 4067-4314/com.redacted.android D/DATABASE﹕ QUERY: SQLiteQuery: SELECT DISTINCT id, sender, data_1, data_2, photo, feed_type, sub_type, created, expiry, updated, comment_count, comment_unread, reaction_count, reaction_unread, sender_name, sender_photo, _id FROM wall WHERE feed_type=? AND sub_type=? AND data_1=? AND is_notified=? ORDER BY created ASC LIMIT 100
On device:
Enter SQL statements terminated with a ";"
sqlite> SELECT DISTINCT id, sender, data_1, data_2, photo, feed_type, sub_type, created, expiry, updated, comment_count, comment_unread, reaction_count, reaction_unread, sender_name, sender_photo, _id FROM wall WHERE feed_type=2 AND sub_type=1 AND data_1=0 AND is_notified=0 ORDER BY created ASC LIMIT 100;
53b702b827d7482062f52b03|a7e759d78abe4bfa97045ce49a24ab57|0|Educ||2|1|1404502712279|1404761912325|1404502712279|||||Luke Skywalker|pr/e5c2c0398b267f93683c80dc5009722e|49
The ContentProvider, however, doesn't agree and cursor.getCount() returns 0.
Any ideas why this is happening?
feed_type, sub_type, and is_notified are INTEGER columns.
data_1 is a BLOB that is storing an integer for any row that would qualify for this query, but stores strings for other types of data that could go in this table.
When you run in the shell i'm surprised you get any rows. The blob data type may not convert the keyed value properly for you. Typically the database API requires a special function to set the blob value as well as retrieve it.
So the problem here was the BLOB column. It was being evaluated properly in queries (The data in the table is used in a ListView and is displayed differently depending on the contents of the data_1 and data_2 columns).
Everything in the feed category gets parsed into a member of a class hierarchy rooted at an AnstractFeedObject.
Most fields that use both data_1 and data_2 store text in both, but some fields (those who correspond to a subset of the mentioned class hierarchy) use data_1 as a type enumeration that the UI uses to interpret the value stored in data_2. For example, a 0 type means that data_2 is a picture id (construct the url and download), while type 1 means it's actual text content.
What I ended up doing was that I replaced data_1 with an integer column called type_enumeration and renamed data_2 to data_1. Now that I know BLOB can cause those kinds of issues, I'll be changin data_2 also to a TEXT column.
If at some point in the future I need to store binary data in the DB, I'll add a bin_data to the column.
Now usually in a proper normalized schema you'd use linked tables to represent such hierarchy, but in a mobile environment, you want to minimize joins so a few extra columns are cheaper in terms of performance (at least that's been my experience).
I was curious if androids SQLiteDatabase insert method automatically handles type conversion.
Here is my example:
I have a csv file with a column name of age. Its type will be an INTEGER.
Lets say I have already created the database and table.
Now I am parsing the csv file with CSVReader, which parses each line and inserts each value into an index of a String[].
In order to insert each line of data into the database, I have to use a ContentValue object, which allows me to store values in it.
//Parse each line and store in line...
ContentValue values = new ContentValue();
values.put(KEY_AGE, line[1]); // Assume line[1] is the age
database.insert(table, null, values);
If I store the age value as a string (as seen above), and then insert it into the table, does Android handle the conversion to INTEGER before inserting it into the database?
I am asking this because I am trying to insert a bunch of tables into a database, and it looks much cleaner when I can just iterate through an array then explicitly state each put call, i.e:
Also if anyone has any design suggestions feel free to tell me.
CLEAN
int i = 0;
for(String s : TransitContract.Routes.COLUMN_ARRAY) {
values.put(s, line[i]);
i++;
}
UGLY
values.put(TransitContract.Routes.KEY_ROUTE_ID, line[0]);
values.put(TransitContract.Routes.KEY_AGENCY_ID, line[1]);
values.put(TransitContract.Routes.KEY_SHORT_NAME, line[2]);
values.put(TransitContract.Routes.KEY_LONG_NAME, line[3]);
values.put(TransitContract.Routes.KEY_DESCRIPTION, line[4]);
values.put(TransitContract.Routes.KEY_ROUTE_TYPE, Integer.parseInt(line[5]));
values.put(TransitContract.Routes.KEY_URL, line[6]);
values.put(TransitContract.Routes.KEY_COLOR, line[7]);
values.put(TransitContract.Routes.KEY_TEXT_COLOR, line[8]);
return mDatabase.insert(TransitContract.Routes.TABLE_NAME, null, values);
When you declare a column as INTEGER, SQLite will automatically convert strings to numbers, if possible.
See the documentation for Type Affinity.
If your ContentProvider doesn't restrict it (i.e. pass it directly to the SQLiteDatabase.insert() method), it should work. SQLite is not that picky about the types used in queries/inserts and the actual column type.
However, it would be best practice to parse and check the values before inserting. Otherwise you might actually insert a string which can't be parsed as integer and therefore retrieving the value might fail.
References:
Boolean datatype accepting string value and integer value
SQLite table with integer column stores string
I have a table containing a BLOB column (it's just 16 bytes). I want to run a query of the form SELECT * FROM myTable WHERE blobColumn = ? and bind a byte array to that column. Ideally I would be able to say
myDatabase.rawQuery("SELECT * FROM myTable WHERE blobColumn = ?", myByteArray)
or some variant thereof, but the rawQuery function only supports String arguments - although looking though the Android sources, it seems that the private methods do include bindBlob(int, byte[]).
I can, of course, just run the query SELECT * FROM myTable WHERE blobColumn = x'CAFE1234CAFE1234CAFE1234CAFE1234', but is there a way to do it that doesn't require converting the blob to a string?
Try this as your argument to your ? query:
new String [] { String.valueOf(mByteArray) }
I don't believe that you can compare the contents of a Blob field without first converting it. Could you create an additional column and load it with some type of key that could be unique and descriptive of it's associated Blob data?