How can I find the tables having column Blob type in Sqlite. I need to get the table names from which I get the column blob type and then want to see the total no. of records where the blob is not empty.
If you wanted tables that have a column defined as a blob then you could use
SELECT * FROM sqlite_master WHERE sql LIKE '%blob%';
as the basis for determining the tables. e.g. this could return results such as :-
However, this does not necessarily find all values that are stored as blobs. This is because with the exception of the rowid column or an alias thereof, any type of value (blob included) can be stored in any column.
e.g. consider the following :-
DROP TABLE IF EXISTS not_a_blob_table;
CREATE TABLE IF NOT EXISTS not_a_blob_table (col1 TEXT, col2 INTEGER, col3 REAL, col4 something_or_other);
INSERT INTO not_a_blob_table VALUES
('test text',123,123.4567,'anything'), -- Insert using types as defined
(x'00',x'12',x'34',x'1234567890abcdefff00') -- Insert with all columns as blobs
;
SELECT typeof(col1),typeof(col2),typeof(col3),typeof(col4) FROM not_a_blob_table;
This results in :-
If you want to find all blobs then you would need to process all columns from all rows of all tables based upon a check for the column type. This could perhaps be based upon :-
SELECT typeof(col1),typeof(col2),typeof(col3),typeof(col4),* FROM not_a_blob_table
WHERE typeof(col1) = 'blob' OR typeof(col2) = 'blob' OR typeof(col3) = 'blob' OR typeof(col4) = 'blob';
Using the table above this would result (only the 2nd row has blobs) in :-
A further complication is what you mean by not empty, null obviously. However what about x'00'? or if you used a default of zeroblob(0) ?.
zeroblob(N)
The zeroblob(N) function returns a BLOB consisting of N bytes of 0x00. SQLite manages these zeroblobs very efficiently. Zeroblobs can
be used to reserve space for a BLOB that is later written using
incremental BLOB I/O. This SQL function is implemented using the
sqlite3_result_zeroblob() routine from the C/C++ interface.
If null though then this wouldn't have a type of blob, instead it's type would be null, which could complicate matters if checking for all values stored as blobs.
You may wish to consider having a look at the code from Are there any methods that assist with resolving common SQLite issues?
as this could well be the basis for what you want.
You also wish to have a look at typeof(X) and zeroblob(N).
Question: Is it possible to use a variable as your table name without having to use string constructors to do so?
Info:
I'm working on a project right now that catalogs data from a star simulation of mine. To do so I'm loading all the data into a sqlite database. It's working pretty well, but I've decided to add a lot more flexibility, efficiency, and usability to my db. I plan on later adding planetoids to the simulation, and wanted to have a table for each star. This way I wouldn't have to query a table of 20m some planetoids for the 1-4k in each solar system.
I've been told using string constructors is bad because it leaves me vulnerable to a SQL injection attack. While that isn't a big deal here as I'm the only person with access to these dbs, I would like to follow best practices. And also this way if I do a project with a similar situation where it is open to the public, I know what to do.
Currently I'm doing this:
cursor.execute("CREATE TABLE StarFrame"+self.name+" (etc etc)")
This works, but I would like to do something more like:
cursor.execute("CREATE TABLE StarFrame(?) (etc etc)",self.name)
though I understand that this would probably be impossible. though I would settle for something like
cursor.execute("CREATE TABLE (?) (etc etc)",self.name)
If this is not at all possible, I'll accept that answer, but if anyone knows a way to do this, do tell. :)
I'm coding in python.
Unfortunately, tables can't be the target of parameter substitution (I didn't find any definitive source, but I have seen it on a few web forums).
If you are worried about injection (you probably should be), you can write a function that cleans the string before passing it. Since you are looking for just a table name, you should be safe just accepting alphanumerics, stripping out all punctuation, such as )(][;, and whitespace. Basically, just keep A-Z a-z 0-9.
def scrub(table_name):
return ''.join( chr for chr in table_name if chr.isalnum() )
scrub('); drop tables --') # returns 'droptables'
For people searching for a way to make the table as a variable, I got this from another reply to same question here:
It said the following and it works. It's all quoted from mhawke:
You can't use parameter substitution for the table name. You need to add the table name to the query string yourself. Something like this:
query = 'SELECT * FROM {}'.format(table)
c.execute(query)
One thing to be mindful of is the source of the value for the table name. If that comes from an untrusted source, e.g. a user, then you need to validate the table name to avoid potential SQL injection attacks. One way might be to construct a parameterised query that looks up the table name from the DB catalogue:
import sqlite3
def exists_table(db, name):
query = "SELECT 1 FROM sqlite_master WHERE type='table' and name = ?"
return db.execute(query, (name,)).fetchone() is not None
I wouldn't separate the data into more than one table. If you create an index on the star column, you won't have any problem efficiently accessing the data.
Try with string formatting:
sql_cmd = '''CREATE TABLE {}(id, column1, column2, column2)'''.format(
'table_name')
db.execute(sql_cmd)
Replace 'table_name' with your desire.
To avoid hard-coding table names, I've used:
table = "sometable"
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS {} (
importantdate DATE,
somename VARCHAR,
)'''.format(table))
c.execute('''INSERT INTO {} VALUES (?, ?)'''.format(table),
(datetime.strftime(datetime.today(), "%Y-%m-%d"),
myname))
As has been said in the other answers, "tables can't be the target of parameter substitution" but if you find yourself in a bind where you have no option, here is a method of testing if the table name supplied is valid.
Note: I have made the table name a real pig in an attempt to cover all of the bases.
import sys
import sqlite3
def delim(s):
delims="\"'`"
use_delim = []
for d in delims:
if d not in s:
use_delim.append(d)
return use_delim
db_name = "some.db"
db = sqlite3.connect(db_name)
mycursor = db.cursor()
table = 'so""m ][ `etable'
delimiters = delim(table)
if len(delimiters) < 1:
print "The name of the database will not allow this!"
sys.exit()
use_delimiter = delimiters[0]
print "Using delimiter ", use_delimiter
mycursor.execute('SELECT name FROM sqlite_master where (name = ?)', [table])
row = mycursor.fetchall()
valid_table = False
if row:
print (table,"table name verified")
valid_table = True
else:
print (table,"Table name not in database", db_name)
if valid_table:
try:
mycursor.execute('insert into ' +use_delimiter+ table +use_delimiter+ ' (my_data,my_column_name) values (?,?) ',(1,"Name"));
db.commit()
except Exception as e:
print "Error:", str(e)
try:
mycursor.execute('UPDATE ' +use_delimiter+ table +use_delimiter+ ' set my_column_name = ? where my_data = ?', ["ReNamed",1])
db.commit()
except Exception as e:
print "Error:", str(e)
db.close()
you can use something like this
conn = sqlite3.connect()
createTable = '''CREATE TABLE %s (# );''' %dateNow)
conn.execute(createTable)
basically, if we want to separate the data into several tables according to the date right now, for example, you want to monitor a system based on the date.
createTable = '''CREATE TABLE %s (# );''' %dateNow) means that you create a table with variable dateNow which according to your coding language, you can define dateNow as a variable to retrieve the current date from your coding language.
You can save your query in a .sql or txt file and use the open().replace() method to use variables in any part of your query. Long time reader but first time poster so I apologize if anything is off here.
```SQL in yoursql.sql```
Sel *
From yourdbschema.tablenm
```SQL to run```
tablenm = 'yourtablename'
cur = connect.cursor()
query = cur.execute(open(file = yoursql.sql).read().replace('tablenm',tablenm))
You can pass a string as the SQL command:
import sqlite3
conn = sqlite3.connect('db.db')
c = conn.cursor()
tablename, field_data = 'some_table','some_data'
query = 'SELECT * FROM '+tablename+' WHERE column1=\"'+field_data+"\""
c.execute(query)
I have a sqlite database where all rows have a UUID as the primary key, the db column is defined as a BLOB.
The keys are inserted as byte[] instead of Strings to avoid wasting storage and index spaces. I can insert, update and delete rows using SQLiteDatabase.compileStatement and using bindBlob on the SQLiteStatement but I can't find anyway to bind a blob to the parameters of a SELECT query.
Both SQLiteDatabase.query and .rawQuery expects my WHERE arguments to be Strings which will never match my byte array blobs. I can find my row if I construct my WHERE manually using a BLOB literal like this:
final Cursor query = db.query(getTableName(),
getColumns(),
"id = X'" + bytesToHex(getByteArrayFromUUID(id)) + "'" ,
null,
null,
null,
null);
But then I am vulnerable to SQL injections...
In every other language I have used SQLite in this is not a problem, is there really no way to get a standard prepared SELECT statement with android SQLite?
Most of the APIs in Android sqlite expect the parameter to be strings.
You can use compileStatement() to prepare your query and then use bindBlob() to bind a blob argument to it. Getting useful results out from the query is not easy though, SQLiteStatement has methods for only a few 1x1 result sets.
On the other hand, using a blob as a key doesn't seem like a good idea.
I have a Blob in an SQLite-Database.
When I type:
Select length(blobblob) FROM Database WHERE id=9387134648;
I get a size of 20519 bytes (quite big), but when I fetch the Blob with android:
Cursor cursor = database.rawQuery("Select ? FROM Database WHERE id=?;",
new String[]{"blobblob", "9387134648"});
if (cursor.moveToNext()) {
byte[] bytes = cursor.getBlob(0);
Log.v("Bloblength", String.valueOf(bytes.length));
}
I just get a size of 9 bytes.
What do I do wrong?
I'd guess that you're getting the string "blobblob" (as an array of bytes) out of your query, those eight characters plus a null terminator would be nine bytes. When you say this:
select ? from database where id = ?
Both placeholders will be replaced by values from your arguments and the arguments are Strings so the ? will be replaced with SQL strings. The result will be a query that is effectively this:
select 'blobblob' from database where id = '9387134648'
and all that query does is select the SQL string literal 'blobblob' from your table.
The underlying problem is that you can't use placeholders for SQL identifiers (table names, column names, ...) in a query, placeholders are only for values like strings and numbers. Your code should look more like this:
Cursor cursor = database.rawQuery("Select blobblob FROM Database WHERE id=?;",
new String[]{"9387134648"});
I'm not much of a Java or Android guy so hopefully I haven't butchered the code.
I'm passing the below statement as a rawQuery in Android:
SELECT DISTINCT ltUsers._id,ltUsers.NAME,ltUsers.GLOBAL_ID, ltGroups.GROUP_NAME
FROM ltUsers
JOIN ltGroups ON (ltUsers.GROUP_ID = ltGroups.GLOBAL_ID)
WHERE ltgroups.GLOBAL_ID = ? " +
ORDER BY ltUsers.NAME ASC,ltgroups.GLOBAL_ID ASC;
With the rawQuery as follows:
Cursor c = db.rawQuery(sql,args)
It works just fine if I pass a value to the parameter, e.g.
String[] args = new String[]{"2"}
However, I also want to be able to show all rows, unlimited by the GLOBAL_ID in the WHERE clause. Testing on a dump of my SQLite database outside of Android - as well as in Android by just writing the parameter directly into the statement - shows the following clause to be a valid way to do this:
WHERE ltGroups.GLOBAL_ID = ltGroups.GLOBAL_ID
Yet when I pass the field reference ltGroups.GLOBAL_ID or [ltGroups].[GLOBAL_ID] as a parameter it fails to return any rows in the rawQuery. Any ideas on why this might be happening? Happy to produce any extra information.
Parameters always replace specific values, not anything else.
When you put the string "ltGroups.GLOBAL_ID" into the parameters array, it is interpreted as exactly that, a string.
(To show all records, just omit the WHERE clause.)