I'm working in Android studio (java).
I'm working with a sqlite database and I want to be able to query a table. I would currently use the following code:
''
String strQuery = "select * from table where var1 = ?";
SQLiteStatement stmt = sqLiteDatabase.compileStatement(strQuery);
stmt.bindString(1, String.valueOf(strVarOne));
stmt.execute();
''
But I need to dynamically build the WHERE statement. I currently have a json array which I want to use to build the where statement that looks like this (for example).
[{"country":"South Africa"},{"province":"Gauteng"}]
In the above example I would want the query to look as follows:
"select * from table where country = 'South Africa' and province= 'Gauteng' "
Can anyone assist with how I could build and run that query dynamically depending on the json array?
Bear in mind that the json array may change - the values may differ and there may be more of them.
class Place {
#SerializedName("country")
String country = null;
#SerializedName("province")
String province = null;
}
Type token = new TypeToken<ArrayList<Place>>(){}.type
String yourJson = "[{\"country":\"South Africa\"},{\"province\":\"Gauteng\"}]";
Gson gson = new Gson();
ArrayList<Place> places = gson.fromJson(yourJson, token);
String selectFormat = "select * from table where %$1s = '%$2s' and %$3s = '%$4s' "
ArrayList<String> selectArguments = new ArrayList<String>();
for(int i=0; i<places.size(); i++){
if(places.country!=null){
selectArguments.add("country");
selectArguments.add(places.country);
}else if(places.province != null) {
selectArguments.add("province");
selectArguments.add(places.province);
}
}
I will not go as far as to tell you how to avoid SQL injection.
String fullSQLStatement = String.format(selectFormat,selectArguments.toArray(new String[selectArguments.size()]);
fullSQLStatement would become select * from table where country = 'South Africa' and province = 'Gauteng'
I might just be hitting native performance issues with SQLite, but there seems to be significant overhead associated with inserting in SQLite even using Transaction processing and pre-compiled statements. Or perhaps I am using them incorrectly.
I have a requirement to read a file line by line (via a URL) and create rows. The mechanism is generic, so any table can be created (within reason) and any number of rows added. I am seeing pretty decent performance for larger inserts, but with smaller ones it seems I have a minimum of +400ms. Since I may have over 100 of these to load, that small overhead is resulting in large load times.
For example some timings:
34 records Test #1 ----> 490 ms
36238 records Test #2 ----> 3021 ms
4 records Test #3 ----> 520 ms
Here is the code (I cut out all the try/catches and other code to boil it down to the actual insert code):
InputStream input = new BufferedInputStream(url.openStream());
// Create the file reader
BufferedReader br = new BufferedReader(new InputStreamReader(input, "UTF-8"));
StringBuffer insertSql = null;
// Build the SQL to bind
insertSql = new StringBuffer("INSERT INTO " + fileName + " (");
String sep = "";
insertSql.append("[" + getDbColumnNames().replaceAll(" ", "").replaceAll(",", "],[") + "]");
String[] columns = getDbColumnNames().split(",");
insertSql.append(") VALUES (");
for (#SuppressWarnings("unused") String col : columns) {
insertSql.append(sep.trim() + "?");
sep = ",";
}
insertSql.append(");");
this.open();
sqlDB.beginTransaction();
SQLiteStatement stmt = = sqlDB.compileStatement(insertSql.toString());
String line = "";
// Read the file line by line
while ((line = br.readLine()) != null) {
String[] tokens = line.split(",");
// Build the bindings and insert the data
int bindcnt = 1;
for (String token : tokens) {
stmt.bindString(bindcnt++, token.trim());
}
long entryID = stmt.executeInsert();
if (entryID < 0) {
success = false;
}
stmt.clearBindings();
}
sqlDB.setTransactionSuccessful();
sqlDB.endTransaction();
I have verified that the read buffer overhead is not significant for most of the reads.
I have a method which reads data from file line by line and takes value between coma, then puts this value into INSERT query. Data in file saved in this way:
–,08:10,–,20:20,08:15,08:16,20:26,20:27,08:20,08:21,20:31,20:32,08:30,08:31,20:40,20:41,08:37,08:38,20:46
20:47,08:48,08:50,20:56,20:57,09:00,09:01,21:07,21:08
08:53,–,17:43,09:01,09:03,09:13,09:15,18:02,18:04,–,–,09:19,09:25
Here is actual my code:
public void insertTime(SQLiteDatabase database, String table) throws FileNotFoundException {
BufferedReader br = null;
String line;
try {
int j = 0;
br = new BufferedReader(new InputStreamReader(context.getAssets().open("time.txt")));
database.beginTransaction();
while ((line = br.readLine()) != null) {
j++;
String query = "INSERT INTO "+table+""+j+" (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
// use comma as separator
String[] time = line.split(",");
for(int i = 1; i < time.length; i+=2) {
statement.bindString(1,time[i-1]);//arrival
statement.bindString(2,time[i]);//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
The problem is that data insert very slow, despite I use SQLiteStatement and transactions. For example, when I insert 69000 rows it takes about 65,929 seconds.
What have I to change in my code to improve speed of insertion ?
UPDATE
OK, I have simplified my code, I got rid of BufferedReader and now it looks like this
public void insertTime(SQLiteDatabase database) throws FileNotFoundException {
database.beginTransaction();
int r = 0;
while (r < 122) {
r++;
String query = "INSERT INTO table_1 (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
for(int i = 1; i < 1100; i++) {
statement.bindString(1,i+"");//arrival
statement.bindString(2,i+"");//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
}
But it still so long inserts data, more than 2 min. Do you have any ideas how to increase speed of my second example ?
Here is a very very detailed post on every method of increasing SQL insertion speed.
Move beginTransaction() and setTransactionSuccessful() outside of while loop and it will be way faster.
A new transaction is started for each item in the while() loop.
It might go a bit faster if you only have 1 transaction to do all your insertions.
Also, when your data is corrupt and String.split doesn't give you at least 2 items, then your transaction will not be ended properly due to an Exception being thrown.
Every time you insert a row in a table with indexes, the indexes have to be adjusted. That operation can be costly. Indexes are kept as b-trees and if you hit the rebalance point, you're bound to have a slowdown. One thing you can do to test this is to remove your indexes. You could also drop the indexes, insert, then re-create the indexes.
For those using JDBC (Java): to be sure, do you first set the autoCommit to FALSE?
I guess so, because you work with explicit transactions.
The performace gain I got by explicitly setting the autocommit off was over 1000 times!
So:
Class.forName("org.sqlite.JDBC");
String urlInput = "jdbc:sqlite:" + databaseFile;
databaseConnection = DriverManager.getConnection(urlInput);
databaseConnection.setAutoCommit( false);
And:
String sql = "INSERT INTO " + TABLE_NAME + " ( type, bi, ci, fvi, tvi, content_type) VALUES ('V',?,?,?,?,'rtf')";
PreparedStatement psi = databaseConnection.prepareStatement(sql);
for( Item item : items) {
psi.setInt(1, item.property1);
// ....
count = psi.executeUpdate();
}
databaseConnection.commit();
databaseConnection.setAutoCommit( true);
So, when somebody forgets this, this may have a huge effect.
I'm storing this data in a .dat file:
data = date + ": " + y + "L/100KM "+ " " + value1 + "dt "+ value2 + "KM\n";
Every line has different values of date,y,value1 and value2.
I want to retrieve variable value1 of every line. How to browse the file and extract this variable of all lines. I'm stucking in this problem in my project. Thanks for helping.
EDIT: Example:
I have this 3 datas stored in the file:
11/09: 5.8L/100KM 20dt 250KM
12/09: 6.4L/100KM 60dt 600KM
13/09: 7.5L/100KM 50dt 543KM
In that case, i want to retrieve 20dt, 60dt and 50dt.
Here's one suggestion using regular expressions:
String line = "12/09: 6.4L/100KM 60dt 600KM";
Pattern p = Pattern.compile("(\\d+)dt");
Matcher m = p.matcher(line);
if (m.find())
System.out.println(m.group(1)); // prints 60
If you have several lines to iterate over, you'd use for instance a new BufferedReader(new FileReader("youfile.dat")) and do something like
String line;
while ((line = br.nextLine()) != null) {
Matcher m = p.matcher(line);
if (m.find())
process(m.group(1));
}
You could also just use line.split(" ") and select the 3:rd element:
String line = "12/09: 6.4L/100KM 60dt 600KM";
String dtVal = line.split(" ")[2];
// Optional: Remove the "dt" part.
dtVal = dtVal.substring(0, dtVal.length() - 2);
System.out.println(dtVal);
I hope i explain the correctly...
What i want to do is be able to run a query and then get the first record of that query and put the characters into an character array, then i want to get the second query and put that into the same array depending on where it is suppose to be inserted into the array.
So for example...the first record from the query will have the string 'hello', i want this to be inserted into the array at position one. Then i want record two which will be 'you' to be inserted at position 6. so when the character array is out put it will display 'hello you'
How do i go about doing this?
SQLiteDatabase db = dbs.getReadableDatabase();
String SQL = "SELECT * FROM Table";
Cursor cursor = db.rawQuery(SQL, null);
startManagingCursor(cursor);
char charac_array[];
charac_array = new char[10];
while (cursor.moveToNext()) {
???
}
That's what you probably want:
StringBuilder sb = new StringBuilder();
while (cursor.moveToNext()) {
String s = cursor.getString(...);
if (sb.length() > 0) {
sb.append(" ");
}
sb.append(s);
}
String helloYou = sb.toString();
UPDATE
To fill a char array from String
StringBuilder sb = new StringBuilder();
char[] charArray = new char[10];
while (c.moveToNext()) {
String s = getFromCursor(c);
s.getChars(0, 10, charArray, 0); // <-- this copies 10 chars from s to charArray
sb.append(charArray);
}
You'll have to take care when s is less than 10 characters though.