I might just be hitting native performance issues with SQLite, but there seems to be significant overhead associated with inserting in SQLite even using Transaction processing and pre-compiled statements. Or perhaps I am using them incorrectly.
I have a requirement to read a file line by line (via a URL) and create rows. The mechanism is generic, so any table can be created (within reason) and any number of rows added. I am seeing pretty decent performance for larger inserts, but with smaller ones it seems I have a minimum of +400ms. Since I may have over 100 of these to load, that small overhead is resulting in large load times.
For example some timings:
34 records Test #1 ----> 490 ms
36238 records Test #2 ----> 3021 ms
4 records Test #3 ----> 520 ms
Here is the code (I cut out all the try/catches and other code to boil it down to the actual insert code):
InputStream input = new BufferedInputStream(url.openStream());
// Create the file reader
BufferedReader br = new BufferedReader(new InputStreamReader(input, "UTF-8"));
StringBuffer insertSql = null;
// Build the SQL to bind
insertSql = new StringBuffer("INSERT INTO " + fileName + " (");
String sep = "";
insertSql.append("[" + getDbColumnNames().replaceAll(" ", "").replaceAll(",", "],[") + "]");
String[] columns = getDbColumnNames().split(",");
insertSql.append(") VALUES (");
for (#SuppressWarnings("unused") String col : columns) {
insertSql.append(sep.trim() + "?");
sep = ",";
}
insertSql.append(");");
this.open();
sqlDB.beginTransaction();
SQLiteStatement stmt = = sqlDB.compileStatement(insertSql.toString());
String line = "";
// Read the file line by line
while ((line = br.readLine()) != null) {
String[] tokens = line.split(",");
// Build the bindings and insert the data
int bindcnt = 1;
for (String token : tokens) {
stmt.bindString(bindcnt++, token.trim());
}
long entryID = stmt.executeInsert();
if (entryID < 0) {
success = false;
}
stmt.clearBindings();
}
sqlDB.setTransactionSuccessful();
sqlDB.endTransaction();
I have verified that the read buffer overhead is not significant for most of the reads.
Related
Hi I have a csv that looks like this:
r1c1|r1c2|r1c3
r2c1|r2c2|r2c3
As you can see it is delimited by the character "|"
In my application, I am trying to explode this using input stream. Here is my code:
String line = "";
String cvsSplitBy = "|";
try {
File initialFile = new File(myfile.txt);
InputStream targetStream = new FileInputStream(initialFile);
BufferedReader reader = new BufferedReader(new InputStreamReader(targetStream));
while ((line = reader.readLine()) != null) {
String[] RowData = line.split(cvsSplitBy);
String c0 = RowData[0];
String c1 = RowData[1];
String c2 = RowData[2];
Toast.makeText(mainactivity.this, c2, Toast.LENGTH_LONG).show();
}
}catch (IOException ex) {
// handle exception
}
Unfortunately, this appears to return each character in the csv as a row. The toast example above returns 1 then 2.
Any ideas how to return the proper column, anyone?
split() splits string around matches of the given regular expression, therefore use of special character (and vertical bar is one of these) requires escaping to strip its "powers".
String cvsSplitBy = "\\|"
See docs: http://docs.oracle.com/javase/7/docs/api/java/lang/String.html
I have a method which reads data from file line by line and takes value between coma, then puts this value into INSERT query. Data in file saved in this way:
–,08:10,–,20:20,08:15,08:16,20:26,20:27,08:20,08:21,20:31,20:32,08:30,08:31,20:40,20:41,08:37,08:38,20:46
20:47,08:48,08:50,20:56,20:57,09:00,09:01,21:07,21:08
08:53,–,17:43,09:01,09:03,09:13,09:15,18:02,18:04,–,–,09:19,09:25
Here is actual my code:
public void insertTime(SQLiteDatabase database, String table) throws FileNotFoundException {
BufferedReader br = null;
String line;
try {
int j = 0;
br = new BufferedReader(new InputStreamReader(context.getAssets().open("time.txt")));
database.beginTransaction();
while ((line = br.readLine()) != null) {
j++;
String query = "INSERT INTO "+table+""+j+" (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
// use comma as separator
String[] time = line.split(",");
for(int i = 1; i < time.length; i+=2) {
statement.bindString(1,time[i-1]);//arrival
statement.bindString(2,time[i]);//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
The problem is that data insert very slow, despite I use SQLiteStatement and transactions. For example, when I insert 69000 rows it takes about 65,929 seconds.
What have I to change in my code to improve speed of insertion ?
UPDATE
OK, I have simplified my code, I got rid of BufferedReader and now it looks like this
public void insertTime(SQLiteDatabase database) throws FileNotFoundException {
database.beginTransaction();
int r = 0;
while (r < 122) {
r++;
String query = "INSERT INTO table_1 (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
for(int i = 1; i < 1100; i++) {
statement.bindString(1,i+"");//arrival
statement.bindString(2,i+"");//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
}
But it still so long inserts data, more than 2 min. Do you have any ideas how to increase speed of my second example ?
Here is a very very detailed post on every method of increasing SQL insertion speed.
Move beginTransaction() and setTransactionSuccessful() outside of while loop and it will be way faster.
A new transaction is started for each item in the while() loop.
It might go a bit faster if you only have 1 transaction to do all your insertions.
Also, when your data is corrupt and String.split doesn't give you at least 2 items, then your transaction will not be ended properly due to an Exception being thrown.
Every time you insert a row in a table with indexes, the indexes have to be adjusted. That operation can be costly. Indexes are kept as b-trees and if you hit the rebalance point, you're bound to have a slowdown. One thing you can do to test this is to remove your indexes. You could also drop the indexes, insert, then re-create the indexes.
For those using JDBC (Java): to be sure, do you first set the autoCommit to FALSE?
I guess so, because you work with explicit transactions.
The performace gain I got by explicitly setting the autocommit off was over 1000 times!
So:
Class.forName("org.sqlite.JDBC");
String urlInput = "jdbc:sqlite:" + databaseFile;
databaseConnection = DriverManager.getConnection(urlInput);
databaseConnection.setAutoCommit( false);
And:
String sql = "INSERT INTO " + TABLE_NAME + " ( type, bi, ci, fvi, tvi, content_type) VALUES ('V',?,?,?,?,'rtf')";
PreparedStatement psi = databaseConnection.prepareStatement(sql);
for( Item item : items) {
psi.setInt(1, item.property1);
// ....
count = psi.executeUpdate();
}
databaseConnection.commit();
databaseConnection.setAutoCommit( true);
So, when somebody forgets this, this may have a huge effect.
Hello I try to insert in sql
My code is:
archivo = new File("file name");
fr = new FileReader(archivo);
br = new BufferedReader(fr);
select = connection.createStatement();
statement = "INSERT INTO TABLE VALUES (";
cantidad = 0;
while(br.readLine() != null)
{
cantidad++;
}
br.close();
fr = new FileReader(archivo);
br = new BufferedReader(fr);
contador=0;
while((linea = br.readLine())!= null)
{
arrayString = linea.split("\\|");
for(int i = 0; i < arrayString.length; i++)
{
statement = statement + "'" + arrayString[i] + "',";
}
statement = statement.substring(0, statement.length() - 1) + ");";
select.executeUpdate(statement);
at the end before of select.executeUpdate(statement)
statement = INSERT INTO TABLE VALUES ('mx','mz','11','43','0','0','0','0','0','2015-01-19 09:24:20','0','10737','2015-01-19 09:24:20','20.71878','-103.45705','N','W','0.0','T','s','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','0.0','J1939','21','56','124','0','0','0');
then i receive the error SQL 22008
What is wrong?
Regards
The DateTime field format you have chosen is not acceptable.
2015-01-19 09:24:20
Try to convert to something like:
2015-01-19T09:24:20
Source on more formats:
http://msdn.microsoft.com/en-us/library/ms187819.aspx
It looks like your code is inserting some sort of tab or line break in the middle of your timestamps that you are trying to insert, that is making SQL Server throw an error that it didn't understand your format and tried to truncate/round the datetime value.
Try changing your code such that your datetime value are in the following format:
yyyy-mm-ddThh:mm:ss
For example:
2015-01-19T09:43:35
I am now trying to import csv files from a certain directory in sd card from an android device. Recently, I can successfully import a single csv files. However, I have no ideas on how to get the list of all csv files and then using a loop to import the csv file one by one.
This is the my code for importing single csv:
button_import_csv.setOnClickListener(new View.OnClickListener(){
public void onClick(View v){
DatabaseHelper helper = new DatabaseHelper(getApplicationContext());
SQLiteDatabase db = helper.getWritableDatabase();
try{
FileReader file = new FileReader("/sdcard/downloadedfolder/A1/adv_sales_order.csv");
BufferedReader buffer = new BufferedReader(file);
ContentValues contentValues=new ContentValues();
String line = "";
String tableName ="adv_sales_order";
db.beginTransaction();
while ((line = buffer.readLine()) != null) {
String[] str = line.split("\t");
contentValues.put("order_date", str[0]);
contentValues.put("cust_code", str[1]);
contentValues.put("customer_ref_no", str[2]);
contentValues.put("line_no", str[3]);
contentValues.put("item_code", str[4]);
contentValues.put("tran_code", str[5]);
contentValues.put("order_qty", str[6]);
db.insert(tableName, null, contentValues);
}
db.setTransactionSuccessful();
db.endTransaction();
}catch (IOException e){
}
}
});
The columns for different csv fileS are not the same.(For example,some may has 4 columns named A,B,C,D and the other one may has columns named as C,D,E,F) Besides hard coding all columns for each csv file, are there any possible ways?
Can anyone tell me any solution???Thank you.
There are two possibilities I can think of...
First: If you are in control of the filenames then give them names with a sequential numeric aspect, e.g., file1.csv, file2.csv etc You can then simply use a for loop to build the filenames and process them. Example...
// Lets say you have 5 files named file1.csv thru file5.csv
for(int i = 1; i < 6; i++) {
String filename = "file" + i + ".csv";
// Process the file which has the above filename
}
Second: Get all of the files in the directory using the listFiles() method. Example...
// This code assumes you have a File object for the directory called dir
File[] files = dir.listFiles();
for(int i = 0; i < files.length; i++) {
String filename = files[i].getAbsolutePath();
if (filename.endsWith(".csv")) {
// Process the file which has the above filename
}
}
I'm not sure if either of the code blocks above are perfect but basically they both simply use a for loop. There are other ways but those are the most straight-forward.
EDIT:
Some csv files use the first line to describe the column names. In some ways this is a bit like a schema of a dataset. Example (using comma-separated values)...
A,B,C,D
valueA,valueB,valueC,valueD
...
Using this approach means you can get access to the column names by reading the first line and splitting it to make an array. You can then use a for loop to put the ContentValues. Try the following...
// Read the first line separately and split to get the column names
line = buffer.readLine();
String[] cols = line.split("\t");
db.beginTransaction();
while ((line = buffer.readLine()) != null) {
String[] str = line.split("\t");
for (int i = 0; i < cols.length; i++) {
contentValues.put(cols[i], str[i]);
}
db.insert(tableName, null, contentValues);
}
db.setTransactionSuccessful();
db.endTransaction();
BTW I notice you're splitting on "\t" so make sure your column names on the first line are tab-delimited (obviously).
I'm storing this data in a .dat file:
data = date + ": " + y + "L/100KM "+ " " + value1 + "dt "+ value2 + "KM\n";
Every line has different values of date,y,value1 and value2.
I want to retrieve variable value1 of every line. How to browse the file and extract this variable of all lines. I'm stucking in this problem in my project. Thanks for helping.
EDIT: Example:
I have this 3 datas stored in the file:
11/09: 5.8L/100KM 20dt 250KM
12/09: 6.4L/100KM 60dt 600KM
13/09: 7.5L/100KM 50dt 543KM
In that case, i want to retrieve 20dt, 60dt and 50dt.
Here's one suggestion using regular expressions:
String line = "12/09: 6.4L/100KM 60dt 600KM";
Pattern p = Pattern.compile("(\\d+)dt");
Matcher m = p.matcher(line);
if (m.find())
System.out.println(m.group(1)); // prints 60
If you have several lines to iterate over, you'd use for instance a new BufferedReader(new FileReader("youfile.dat")) and do something like
String line;
while ((line = br.nextLine()) != null) {
Matcher m = p.matcher(line);
if (m.find())
process(m.group(1));
}
You could also just use line.split(" ") and select the 3:rd element:
String line = "12/09: 6.4L/100KM 60dt 600KM";
String dtVal = line.split(" ")[2];
// Optional: Remove the "dt" part.
dtVal = dtVal.substring(0, dtVal.length() - 2);
System.out.println(dtVal);