Android SQLite insert JSON in TEXT column with escaping double quotes - android

I am using SQLite in an Android app which stores, amongst other things, a table that records the mood of the user. The table schema is shown below
CREATE TABLE moods
(
dow INTEGER,
tsn INTEGER,
lato INTEGER,
agitation TEXT DEFAULT '{}',
preoccupancy TEXT DEFAULT '{}',
tensity TEXT DEFAULT '{}',
taps TEXT DEFAULT '{}'
);
Unlike, say, Postgres, SQLite does not have a dedicated jsonb storage type. To quote
Backwards compatibility constraints mean that SQLite is only able to store values that are NULL, integers, floating-point numbers, text, and BLOBs. It is not possible to add a sixth "JSON" type.
SQLite JSON1 extension documentation
On the SQLite commandline processor I can populate this table using a simple INSERT statement such as this one
INSERT INTO moods (dow,tsn,lato,agitation,preoccupancy,tensity,taps)
VALUES(1,0,20,'{"A2":1}','{"P4":2}','{"T4":3}','{"M10":4}');
since it accepts both single and double quotes for strings.
In order to accomplish the same thing in my Hybrid Android app I do the following
String ag = JOString("A" + DBManage.agitation,1);
String po = JOString("P" + (DBManage.preoccupancy + 15),1);
String te = JOString("T" + DBManage.tensity,1);
which returns strings such as {"P4":2} etc. My next step is as follows
ContentValues cv = new ContentValues();
cv.put("dow",0);
cv.put("tsn",1);
cv.put("lato",2);
cv.put("agitation",ag);
cv.put("preoccupancy",po);
cv.put("tensity",te);
long rowid = db.insert("moods",null,cv);
which works. However, upon examining the stored values I find that what has gone in is in fact
{"dow":0,"tsn":5,"lato":191,"tensity":"{\"T0\":1}","agitation":"
{\"A1\":1}","preoccupancy":"{\"P15\":1}","taps":"{}"}]
Unless I am misunderstanding something here the underlying ContentValues.put or the SQLiteDatabase.insert implementation has taken it upon itself to escape the quoted strings. Perhaps this is to be expected given that the benefit of going down the SQLiteDatabase.insert route with ContentValues is protection from SQL injection attacks. However, in the present instance it is being a hindrance not a help.
Short of executing raw SQL via execSQL is there another way to get the JSON into the SQLite database table here?
I should add that simply replacing the double quotes with single quotes
String ag = JOString("A" + DBManage.agitation,1);
ag = ag.replaceAll("\"","'");
will not work since a subsequent attempt to extract JSON from the table
SELECT ifnull((select json_extract(preoccupancy,'$.P4') from moods where dow = '1' AND tsn = '0'),0);
would result in the error malformed JSON being emitted by the SQLite JSON1 extension.

Related

Error occurred while store image path to SQLite Database using Android?

It continues to occur error... what is the problem?
every variables are String.
String insertSQL = "INSERT INTO " + DBHelper.getTableName() + " VALUES (\''"+entry.getKey()+"\'', \''"+images+"\'')";
Error message
INSERT INTO LABELING RESULT VALUES (''Sky'',
''["/storage/emulated/0/DCIM/CandyCam/IMG_20171009_164101723.jpg","/storage/emulated/0/DCIM/Pictail/IMG_20180305_000218777.jpg","/storage/emulated/0/DCIM/Pictail/IMG_20180401_235850170.jpg","/storage/emulated/0/DCIM/Pictail/IMG_20180518_194252232.jpg"]''))
My table has three column : ID(Integer), LABEL(TEXT), IMAGES(TEXT)
Issue 1 you have specified a table name with a space i.e. LABELING RESULT, if this is the table name then you would need to enclose it e.g. [LABELING RESULT].
Issue 2 you have double single quotes when you should only have 1.
Issue 3 you have an extra closing parenthesis.
I believe the following is what you want :-
INSERT INTO [LABELING RESULT] VALUES ('Sky', '["/storage/emulated/0/DCIM/CandyCam/IMG_20171009_164101723.jpg","/storage/emulated/0/DCIM/Pictail/IMG_20180305_000218777.jpg","/storage/emulated/0/DCIM/Pictail/IMG_20180401_235850170.jpg","/storage/emulated/0/DCIM/Pictail/IMG_20180518_194252232.jpg"]'))
Which would (not checked though) be :-
String insertSQL = "INSERT INTO [" + DBHelper.getTableName() + "] VALUES ('"+entry.getKey()+"', '"+images+"')";
That assumes that Sky goes into the LABEL column and the 2 comma separated image paths go into the IMAGES column.
Additional re enclosing (the [ ] ):-
If you want to use a keyword as a name, you need to quote it. There
are four ways of quoting keywords in SQLite:
'keyword' A keyword in single quotes is a string literal.
"keyword" A keyword in double-quotes is an identifier.
[keyword] A keyword enclosed in square brackets is an identifier. This is not
standard SQL. This quoting mechanism is used by MS Access and SQL
Server and is included in SQLite for compatibility.
keyword A keyword enclosed in grave accents (ASCII code 96) is an identifier.
This is not standard SQL. This quoting mechanism is used by MySQL and
is included in SQLite for compatibility.
SQL As Understood By SQLite - SQLite Keywords

SQLite language independent query of numbers

I have written an android app for fun that have inner sqlite database after gathering many data using html parser I found my numbers that are saved as
text in database, are written in English language, so doing query in persian that people in my country try will return nothing on numbers
String q = "SELECT * FROM studentIDs WHERE field1 LIKE '%"+name+"%' OR field2 LIKE '%"+name+"%'";
while doing good on both field1 that is string ,it won't work on field2 that number stored as string, how should I perform language independent query on numbers?
I can't change characters from English to other I want support for both languages and I can't change it's type to integer because some records are English name
Sorry about my English and thanks in advance
Since your data type is String, it means you can store any character sequence to it(depends on your sqlite encoding config e.g utf8) and Sqlite doesn't care and shouldn't care about it.
You have a simple solution here:
Just write a simple mapper before any query to database:
String mapToEn(String query) {
return query
.replace('۰', '0')
.replace('۱', '1')
.replace('۲', '2')
.replace('۳', '3')
.replace('۴', '4')
.replace('۵', '5')
.replace('۶', '6')
.replace('۷', '7')
.replace('۸', '8')
.replace('۹', '9');
}
And use it on your query or query parameters before executing the query to database:
Result query(mapToEn(query));
Edit:
Since you said
I can't change it's type to integer cause some recored's are english
name
I thought your data in the field1 is a combination of numbers and characters, now that you clarified it only contains numeric or String data you have another solution.
Database Schema Migration
Since your database schema doesn't match your requirements anymore you have to make some changes to it. You need to differentiate the data type you have entered, simply adding two new column as field_str and field_num.
Basically you should write a database migration which is responsible for converting the field1 column's data from String to Integer if its an Integer without losing any data, Here are the steps you should do:
Add an Integer and a String column to your table respectively field_num and field_str.
Iterate through the table and parse all those Strings in field1
to Integer and insert them into 'field_numcolumn, and insert the unparseable ones intofield_str` column.
change your query accordingly.
Since sqlite does not support column drop, You either have to add a new column to your existing table and leave alone the old data to be there, or You can create a new table and migrate all of your data to the new table:
Here is some hypothetical situation:
sqlite> .schema
CREATE TABLE some_table(
id INTEGER PRIMARY KEY,
field1 TEXT,
field2 TEXT,
);
sqlite> select * from some_table;
id field1 field2
---------- ---------- ------
0 1234 name<br>
1 bahram name
Now create another table
sqlite> CREATE TABLE new_some_table(
...> id INTEGER PRIMARY KEY,
...> field_str TEXT,
...> field_num INTEGER,
...> field2 TEXT,
...> ) ;
Now copy your data from the old table
sqlite> INSERT INTO new_some_table(id, field_str, field2)
...> SELECT id, field1, field2, FROM some_table ;
sqlite> INSERT INTO new_some_table(id, field_num)
...> SELECT id, field1, FROM some_table WHERE typeof(field1) = "integer" ;
Now you can query your table based on what type of data you have.
Consider using an ORM which provides the migration tool, like Google's Room or dbflow.
As a simple knowledge all database store data in one file and there data get store as string. It depends on our Database Driver that interpret them and convert them into other data type as per Table schema.
So, different languages is also stored in database as string. No matter if it is number or character. For each language you have to put language converter.
So idea to make your database usable for your and all other country language is to Encode and Decode. convert any country number in English while save into database and vice-versa.
You have to make method for all languages for conversation Like this. Or make your own library class to use it universally. Hope you find this useful.
my idea is make semi-translator for numbers first check each char like this to see is it number in any language
static boolean isDigit(char ch)//from java doc
Determines if the specified character is a digit.
then use getNumericValue(char)
The nice thing about getNumericValue(char) is that it also works with strings like "٥" and "५" where ٥ and ५ are the digits 5 in Eastern Arabic and Hindi/Sanskrit respectively.

SQLiteException: unrecognized token: "\":

Here is the table where i want to insert the value:
" create table if not exists "+CipherCongfigTable +
" ( DATABSE_NAME **TEXT** PRIMARY KEY NOT NULL,DATABSE_KEY **TEXT** NOT NULL);";
when i want to insert
String **configDBPassword**= "**x\'2DD29CA89\'**"
through statement
"insert into "+CipherCongfigTable+ " values("+DataBaseName+","+**configDBPassword**+")"
I am getting exception:
unrecognized token: "\":
I need the password in the same format i.e. having escape charecter. Is there any way to do it????
Thanks
Don't manually build INSERT (or any other if you can avoid it) queries on Android (or any other database wrapper as long as there is a predefined API to get what you want). It opens up your application to quoting problems like the one from your question and --at the worst-- to SQL injection attacks from outside of your application.
For example, setting configDBPassword = "\"; DROP TABLE <tablename>; --" I could possibly wreak havoc on your database as long as configDBPassword can be entered by the user.
Also, SQLite uses double quotes ("), backticks (`, borrowed from MySQL), or square brackets ([], borrowed from MS SQL) to quote identifiers (e.g. column or table names with spaces in them), string literals are canonically quoted with single (') quotes. SQLite is a quite liberal in allowing to mix both quoting types, but it is significantly more readable to use the proper quoting style whereever appropriate. From the documentation:
Programmers are cautioned not to use the two exceptions described in the previous bullets. We emphasize that they exist only so that old and ill-formed SQL statements will run correctly. Future versions of SQLite might raise errors instead of accepting the malformed statements covered by the exceptions above.
As a matter of fact, you should avoid doing the quoting by yourself whenever possible. For inserting values, please instead use SQLiteDatabase.insert() which is the proper way of inserting values into an SQLiteDatabase on Android. It also does proper quoting of the arguments, too:
db.beginTransaction();
try {
final ContentValues values = new ContentValues();
values.put("DATABSE_NAME", DataBaseName);
values.put("DATABSE_KEY", configDBPassword);
db.insert(CipherCongfigTable, null, values);
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
Always properly quote your SQL parameters.
Try this .
"INSERT INTO "+CipherCongfigTable+ " values('DataBaseName','configDBPassword')”
You are not adding Single quotes(') in your insert query
Edit :
If you need to insert string with Single qotes means use like this before insert.
configDBPassword = configDBPassword.replace ("'", "''");
Also change your **
" create table if not exists "+CipherCongfigTable +
" ( DATABSE_NAME TEXT PRIMARY KEY NOT NULL,DATABSE_KEY TEXT NOT NULL);";

What primary key to use in my SQLite database?

I have a .csv file that I turned into an SQLite database with the first column as a date and time (as a TEXT datatype because it's in the format 7/20/2011 12:00:00 PM), the second column as the subsecond (a REAL, like 0.142857), and the rest of the columns as data about that time. I will be using it as a read-only db in my Android app, in which the user selects a start time and an end time and then the data is displayed graphically for that time period.
This is my first time using a database and I'm not sure how to work with this. I believe I need a unique NUMERIC primary key for each row but simply labeling them by row number is a waste of space. The first column is not unique and is a TEXT datatype anyway, though if the subsecond was to be somehow incorporated then it would be unique. But I really can't re-write the database because it has 65534 rows... How do you suggest I access specific times in this database?
In Android you need a column named _ID in your database (else you'll face some issues later on). You will use that as the primary key.
Dates and times can be stored in SQLite in the form of a text column in the following format (See http://www.sqlite.org/datatype3.html):
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS")
If your database is static, simply use a tool such as SQLite Database Browser to convert it to a format convenient for Android.
If your database is local and external(not remote), than you must have _id and another another table android_metadata which will hold the locale.
If your database was remote. Yes, you can it is only matter of speed if you are write, since you don't. Using WHERE clause will do the work.
Every date can be converted to numeric timestamp quite easy:
Date date = new Date();
long timestamp = date.getTime();
Date otherDate = new Date(timestamp);
Numbers are MUCH easier and faster to process than text fields. If you are completely sure, that you have unique data within column you can use it as primary key.
Importing csv file into table should be also easy:
//just open file in some known way and read it line by line
// we have declared String[] data, String line is line read from your csv somewhere earlier
// in code
data = line.split(",");
//here you have to process values, and execute insert statement
You have to create indexes on every column which will be used to search or order data. Please be also aware, that rows in table has no "default", "natural" or any other order. If you execute this same select statement twice you can get two totally different results in meaning of sorting. So simple select statement should look like that:
select
_id, other_colum_name, some_other_column
from
table_name
where
column_name = 5
and other_column_name = 3
order by
column_name;

Import MySQL DB to android SQLite DB

Can you please tell me how to import external SQLite or MySQL DB to android SQLite or how can i use the external db in the android application?
Take care that the external SQLite database you wish to import uses INTEGER datatypes for the primary key, not INT or any of the other variants of INT. Some flavors of SQLite from the various SQLite Consortium members treat INTEGER and INT primary keys the same whereas (flagship) SQLite treats them differently.
See section 2.0 here: http://www.sqlite.org/datatypes.html
and see section on RowId and Primary Key here: http://www.sqlite.org/lang_createtable.html#rowid
A PRIMARY KEY column only becomes an
integer primary key if the declared
type name is exactly "INTEGER". Other
integer type names like "INT" or
"BIGINT" or "SHORT INTEGER" or
"UNSIGNED INTEGER" causes the primary
key column to behave as an ordinary
table column with integer affinity and
a unique index, not as an alias for
the rowid. [emphasis added]
EDIT: If your external SQLite database has its integer primary keys defined as INT or any of the other variants of INT, rather than as "INTEGER", you can get erroneous results when attaching to the database file from a consortium member's implementation. Let's say the FK value in a child table is 110; a join that treats the integer value as an alias for the rowid will look to the parent table to grab the 110th physical row, which may or may not be the row whose PK value = 110 if, in the parent table, the PK was defined as INT or BIGINT any of the other variants! In SQLite only an INTEGER [verbatim] PK is treated as alias for rowid, but some implementations did not follow that rule and treat all INT types as aliases for the rowid. Thus, when attaching from one of those implementations to an external SQLite datafile that was created using flagship SQLite, it is imperative to have used "INTEGER" (verbatim) primary keys and not any of the other INT types.
Have a look here to find some converters tools that can convert an MySQL dumb to a SQLite database. You could then import the database in your application as usual.
It is impossible to use the MySQL db directly.
I know it can be a cheap way but I used a push message service (MQTT) to take a copy of my mysql dump file and push the message to the phone, where I run it through a parser to insert rows into my local db. The mqtt message service has a precompiled .jar library that you can use quite easily in an android app.
There are a variety of means in which to implement the mqtt on your web server (SAM->php, or what I did write a c++ listening service to forward messages [Using an open source mqtt library called mosquitto]). The code to parse in the data is only a few lines since the message I receive in my android service is just a byte array.
As an example, I convert the byte array into a string, tokenize the string, and then insert it into the database:
public void parseMysqlDump(byte[] thisPayload) {
ContentResolver cr = this.getContentResolver();
String mysqlDump = "";
for(int i = 0; i < thisPayload.length; i++) {
mysqlDump += (char) thisPayload[i];
}
String[] mysqlDumpNewLineTokens = mysqlDump.split("\\n");
//The first line is the table headers stuff so start with i=1 to get to the first
//data line
for(int i = 1; i < mysqlDumpNewLineTokens.length; i++) {
String[] mysqlDumpSpaceTokens = mysqlDumpNewLineTokens[i].split("\\s");
String dataTitle = mysqlDumpSpaceTokens[0];
ContentValues cv = new ContentValues();
cv.put(Vehicle_DataMetaData.DATA_VALUE, mysqlDumpSpaceTokens[1]);
cr.insert(Vehicle_DataMetaData.CONTENT_URI, cv);
}
TL;DR: Take a mysql dump, send it as a string to the phone, parse the string into the db.
If you are interested in the mqtt stuff or wanted any clarification, just comment and I will get back to you

Categories

Resources