I currently have a CSV file that I parse and am trying to insert the data into the android database. The problem I am having is that it is taking way too long to insert all of the data. It's a good amount of data but I feel like it shouldn't take 20min or so to complete.
Basically, I create my database, then begin the parsing. While parsing through each individual CSV row, I grab the required data and insert it into the database. In total there are around 40000 rows.
Is there any way I can speed up this process? I have tried batch inserts but it never really helped (unless I did it wrong).
Code down below.
Thanks.
DatabaseHelper (i have two insert commands based on the amount of data in each csv row):
// add zipcode
public void add9Zipcode(String zip, String city, String state, String lat,
String longi, String decom) {
// get db and content values
SQLiteDatabase db = this.getWritableDatabase();
ContentValues values = new ContentValues();
db.beginTransaction();
try{
// add the values
values.put(KEY_ZIP, zip);
values.put(KEY_STATE, state);
values.put(KEY_CITY, city);
values.put(KEY_LAT, lat);
values.put(KEY_LONG, longi);
values.put(KEY_DECOM, decom);
// execute the statement
db.insert(TABLE_NAME, null, values);
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
db.close();
}
public void add12Zipcode(String zip, String city, String state, String lat,
String longi, String decom, String tax, String pop, String wages) {
// get db and content values
SQLiteDatabase db = this.getWritableDatabase();
ContentValues values = new ContentValues();
db.beginTransaction();
try{
// add the values
values.put(KEY_ZIP, zip);
values.put(KEY_STATE, state);
values.put(KEY_CITY, city);
values.put(KEY_LAT, lat);
values.put(KEY_LONG, longi);
values.put(KEY_DECOM, decom);
values.put(KEY_TAX, tax);
values.put(KEY_POP, pop);
values.put(KEY_WAGES, wages);
// execute the statement
db.insert(TABLE_NAME, null, values);
db.setTransactionSuccessful();
} finally{
db.endTransaction();
}
db.close();
}
Parse File:
public void parse(ArrayList<String> theArray, DatabaseHandler db) {
String[] data = null;
// while loop to get split the data into new lines
// for loop to split each string in the array list of zipcodes
for (int x = 0; x < theArray.size(); x++) {
if(x == 10000 || x == 20000 || x == 30000 || x == 40000){
Log.d(TAG, "x is 10k, 20k, 30k, 40k");
}
// split string first into an array
data = theArray.get(x).split(",");
// separate based on the size of the array: 9 or 12
if (data.length == 9) {
db.add9Zipcode(data[0], data[2], data[3], data[5], data[6],
data[8]);
} else if (data.length == 12) {
db.add12Zipcode(data[0], data[2], data[3], data[5], data[6],
data[8], data[9], data[10], data[11]);
/*
* theZip.zip = data[0]; theZip.city = data[2]; theZip.state =
* data[3]; theZip.lat = data[5]; theZip.longi = data[6];
* theZip.decom = data[8]; theZip. = data[9]; theZip.population
* = data[10]; theZip.wages = data[11];
*/
}
}
Refer to this answer I made previously: Inserting 1000000 rows in sqlite3 database
In short, use an InsertHelper and do more than one insert per transaction - unless you did something wonky, the speed increase should be noticeable.
Edit:
In short:
Your SQLiteOpenHelper should be a singleton used across your entire application.
Don't go around calling close() on your SQLiteDatabase instance - it's cached in the SQLiteOpenHelper and every time you close you force the helper to reopen it.
Batch your inserts, start a transaction outside the call to the addZipCode methods and mark it as successful after you've done all the inserts - then commit the transaction.
Use an InsertHelper - it will format the insert properly as a prepared statement and is nice and reusable.
Be mindful of synchronizing access to the database - unless you intend to do all your database work on the UI-thread (which is not recommended) - you either need to enable locking or guard access to the database to avoid concurrent access.
Related
I created a database with a table named flagTable, this table only has two fields, which are id(auto increment) and an integer field. Next, in my program, I have a button that will trigger a thread to start. When the thread is starting, it constantly retrieve data from database, and check for the for the value, if the value is equal to one then it will trigger another new Thread, something like this:
private class statusOfStrummingInAnotherDevice extends Thread {
int value;
public void run() {
try{
while(true){
try{
if(flagCursor == null){
flagCursor = cdb1.getFlagAll();
}
}catch(Exception e){break;}
try{
Log.i("MAIN3ACTIVITY","getting status");
int size = cdb1.getSize(flagCursor);
Log.i("MAIN3ACTIVITY","SIZE is" + String.valueOf(xyz));
for(int i = 0 ; i < size ; i++){
flagCursor.moveToPosition(i);
Log.i("MAIN3ACTIVITY","getting status jkasdfasdf");
value = cdb1.getFlag();
if(value == 1){
Log.i("FLAGCURSOR=====>>>>","Succesful");
releasingNotes = new ReleasingNotes(IntendedChord);
releasingNotes.start();
//break;
}
cdb1.updateFlag(0);
Log.i("FLAGCURSOR=====>>>>",String.valueOf(value));
}
flagCursor = null;
}catch(Exception e){break;}
Log.i("MAIN3ACTIVITY","thread is sleeping");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
break;
}
}
}catch(Exception e){
}
}
}
In the meantime, the data that were retrieved from the database is using this function:
public Cursor getFlagAll(){
return getReadableDatabase().rawQuery(
"SELECT _ID, flag from flagTable", null);
}
And, the data that were updated to the database through this method:
public int updateFlag(int i) {
SQLiteDatabase db = this.getWritableDatabase();
ContentValues contentValues = new ContentValues();
contentValues.put("flag",i);
return db.update("flagTable" , contentValues , "_ID" + "= ?",new String[]{String.valueOf(1)});
}
Now, above codes will give no error, however, the data that were retrieved from the database is always 1, it keeps trigger a new function. In my above codes, I stated if the value is equal to 1, then the current thread will trigger a new thread to start, When its finished, the program will update the current data to 0. So that, the next round of the infinite loop can stop triggering new thread until a the conditon is met. What is problem overhere? did my codes really updated the new value? or I need to referesh the database every time I updated a new value.
Use Listeners to your database.
use SQLiteTransactionListener and do your things in onCommit()
Some guide in details here :
https://developer.android.com/reference/android/database/sqlite/SQLiteTransactionListener.html and
http://www.programcreek.com/java-api-examples/index.php?api=android.database.sqlite.SQLiteTransactionListener
I have a method which reads data from file line by line and takes value between coma, then puts this value into INSERT query. Data in file saved in this way:
–,08:10,–,20:20,08:15,08:16,20:26,20:27,08:20,08:21,20:31,20:32,08:30,08:31,20:40,20:41,08:37,08:38,20:46
20:47,08:48,08:50,20:56,20:57,09:00,09:01,21:07,21:08
08:53,–,17:43,09:01,09:03,09:13,09:15,18:02,18:04,–,–,09:19,09:25
Here is actual my code:
public void insertTime(SQLiteDatabase database, String table) throws FileNotFoundException {
BufferedReader br = null;
String line;
try {
int j = 0;
br = new BufferedReader(new InputStreamReader(context.getAssets().open("time.txt")));
database.beginTransaction();
while ((line = br.readLine()) != null) {
j++;
String query = "INSERT INTO "+table+""+j+" (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
// use comma as separator
String[] time = line.split(",");
for(int i = 1; i < time.length; i+=2) {
statement.bindString(1,time[i-1]);//arrival
statement.bindString(2,time[i]);//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
The problem is that data insert very slow, despite I use SQLiteStatement and transactions. For example, when I insert 69000 rows it takes about 65,929 seconds.
What have I to change in my code to improve speed of insertion ?
UPDATE
OK, I have simplified my code, I got rid of BufferedReader and now it looks like this
public void insertTime(SQLiteDatabase database) throws FileNotFoundException {
database.beginTransaction();
int r = 0;
while (r < 122) {
r++;
String query = "INSERT INTO table_1 (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
for(int i = 1; i < 1100; i++) {
statement.bindString(1,i+"");//arrival
statement.bindString(2,i+"");//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
}
But it still so long inserts data, more than 2 min. Do you have any ideas how to increase speed of my second example ?
Here is a very very detailed post on every method of increasing SQL insertion speed.
Move beginTransaction() and setTransactionSuccessful() outside of while loop and it will be way faster.
A new transaction is started for each item in the while() loop.
It might go a bit faster if you only have 1 transaction to do all your insertions.
Also, when your data is corrupt and String.split doesn't give you at least 2 items, then your transaction will not be ended properly due to an Exception being thrown.
Every time you insert a row in a table with indexes, the indexes have to be adjusted. That operation can be costly. Indexes are kept as b-trees and if you hit the rebalance point, you're bound to have a slowdown. One thing you can do to test this is to remove your indexes. You could also drop the indexes, insert, then re-create the indexes.
For those using JDBC (Java): to be sure, do you first set the autoCommit to FALSE?
I guess so, because you work with explicit transactions.
The performace gain I got by explicitly setting the autocommit off was over 1000 times!
So:
Class.forName("org.sqlite.JDBC");
String urlInput = "jdbc:sqlite:" + databaseFile;
databaseConnection = DriverManager.getConnection(urlInput);
databaseConnection.setAutoCommit( false);
And:
String sql = "INSERT INTO " + TABLE_NAME + " ( type, bi, ci, fvi, tvi, content_type) VALUES ('V',?,?,?,?,'rtf')";
PreparedStatement psi = databaseConnection.prepareStatement(sql);
for( Item item : items) {
psi.setInt(1, item.property1);
// ....
count = psi.executeUpdate();
}
databaseConnection.commit();
databaseConnection.setAutoCommit( true);
So, when somebody forgets this, this may have a huge effect.
I am new to the android world and have a problem with an id. What i need is that when the user clicks on new match it will insert a new row into the db. This is working and i get the lastId but now i need this id in the next activities. How can i store that id so i can use it elsewhere?
This is how i insert the new match:
public void newMatch(WedstrijdenGeschiedenis wedstrijd){
// 1.
SQLiteDatabase db = this.getWritableDatabase();
// 2.
DateFormat dateFormat = new SimpleDateFormat("dd-MM-yyyy");
Date date = new Date();
ContentValues values = new ContentValues();
values.put(KEY_DATUM, dateFormat.format(date)); // get datum
// 3.
long lastId = db.insert(TABLE_WEDSTRIJD, // table
null, //nullColumnHack
values); // key/value -> keys = column names/ values = column values
Log.d("New Match","ID ="+lastId);
// 4. close
db.close();
}
so i see the lastId in LogCat but i don't know how to store it for further use. I tried void but offcourse that is not possible on void. Sorry for the dummy question
change void to long and add a return statement that returns the lastId
public long newMatch(WedstrijdenGeschiedenis wedstrijd){
// Your other code
return lastId;
}
Access it with:
long lastId= db.newMatch(new WedstrijdenGeschiedenis());
I have read several posts here on speed issues when looping through a cursor and tried the answers given in these posts such as e.g. do not use getcolumnindex in the loop call this once etc.
However with a database having around 2400 records it takes around 3 to 5 minutes to finish.
The loop is running in an async task method so that it does not hang up the device and the database is handled via a database adapter.
The loop code is as follows :
while (!exportrec.isAfterLast()) {
if ( exportrec.moveToNext() ) {
fulldate = exportnumberformatter(exportrec.getInt(daye))
+"/"+exportnumberformatter(exportrec.getInt(monthe))+"/"
+String.valueOf(exportrec.getInt(yeare));
fulltime = exportnumberformatter(exportrec.getInt(houre))+":"
+exportnumberformatter(exportrec.getInt(mine))+":"
+exportnumberformatter(exportrec.getInt(sece));
noiseid = exportrec.getInt(typee);
exportedinfo += exporttypes[id] +","+exportrec.getString(notee)+","+
fulldate+","+fulltime+" \n" ;
}
}
The exportnumberformatter does the following :
public String exportnumberformatter(int i) {
String result = Integer.toString(i);
if (result.length() >1 ) {
return Integer.toString(i);
}
String zeroprefix = "";
zeroprefix = "0"+result;
return zeroprefix ;
}
The cursor is called as follows before the loop to get the data :
exportrec = MD.GetAllLogs(2, "date_sort");
exportrec.moveToFirst();
The MD is the database adapter and the GetAllLogs Method (this has been played with to try and speed things up and so the date_sort that is used is really ignored here):
public Cursor GetAllLogs(Integer i,String sortfield)
{
String sorted = "";
if (i == 1 ) {
sorted = "DESC";
} else if (i == 2) {
sorted = "ASC";
}
return mDB.query(DB_TABLE, new String[] {COL_ID, COL_TYPE,COL_IMAGE, COL_INFO,COL_IMAGE,COL_HOUR,COL_SEC,COL_MIN,COL_DAY,COL_MON,COL_YEAR,COL_SORT_DATE},
null, null, null, null, COL_ID+" "+sorted);
}
When I created the table in the database it had no indexes so I created these via the upgrade method. However they did not error or appear to fail when I did this but what I do not know is A) does the database/table need rebuilding after an index is created and B) how to tell if they have been created ? the two indexes were based on the ID as the first and a field that holds the year month day hour minute second all in on Long Integer.
I am concerned that the loop appears to be taking this long to read through that many records.
Update:
rtsai2000's and the suggestion from CL answer has improved the speed from minutes to seconds
Your exportedInfo String is growing and growing. Save the results in an array and Stringify later (such as with StringBuilder).
You are not closing your cursor after reading the records.
List<String> exportedInfo = new ArrayList<String>();
Cursor exportrec = GetAllLogs();
try {
while (exportrec.moveToNext()) {
String info = String.format("%s, %s, %02d/%02d/%02d, %02d:%02d:%02d",
exporttypes[id],
exportrec.getString(notee),
exportrec.getInt(daye),
exportrec.getInt(monthe),
exportrec.getInt(yeare),
exportrec.getInt(houre),
exportrec.getInt(mine),
exportrec.getInt(sece));
exportedInfo.add(info);
}
} finally {
exportrec.close();
}
return exportedInfo;
I want to save weekdays in database, so i thought to store it by assigning int value to each day. i.e
1 -> Selected, 0 -> Not Selected.
Monday = 0/1
Tuesday = 0/1
.
.
.
.
.
Sunday = 0/1.
But this will make 7 columns in DB. So I was thinking if anyone can help me with this if I should store it in a single array and retrieve the values for further use. I was reading some examples over internet but didn't get it in a easy way.
To insert 7 values in one column you can use comma separator like this
where Total_Score_P1 is an string array
//string array
String[] Total_Score = new String[] { p1e1,p1e2,p1e3,p1e4,p1e5,p1e6 };
// Convderting it into a single string
String result_ScoreP1 = ("" + Arrays.asList(Total_Score_P1)).
replaceAll("(^.|.$)", " ").replace(", ", " , " );
result_ScoreP1 will be
// output of this
result_ScoreP1 = "p1e1,p1e2,p1e3,p1e4,p1e5,p1e6";
insert it as a single string in database and
when retrieve it in again break in parts like
// a string array list
// query fired
public ArrayList<String> rulTable(String id) {
// TODO Auto-generated method stub
ArrayList<String> Ruleob = new ArrayList<String>();
Cursor c_rule;
try
{
c_rule = db.query(NameTable, new String[]{
columns1
},
Rule_COurseID + "=" + id ,
null, null,
null, null, null);
c_rule.moveToFirst();
// if there is data available after the cursor's pointer, add
// it to the ArrayList that will be returned by the method.
if (!c_rule.isAfterLast())
{
do
{
Ruleob.add(c_rule.getString(0));
}
while (c_rule.moveToNext());
}
// let java know that you are through with the cursor.
c_rule.close();
}
catch(Exception e)
{
}
return Ruleob;
}
//list to get elements
ArrayList<String> ListOne = new ArrayList<String>();
ArrayList<String> row ;
try{
// received values
row = db.TheTable(id);
String r1 = row .get(0);
}
catch(Exception e)
{
}
StringTokenizer st2 = new StringTokenizer(r1, "||");
while(st2.hasMoreTokens()) {
String Desc = st2.nextToken();
System.out.println(Desc+ "\t" );
ListOne.add(Desc);
//
}
You can use a binary integer 1= selected 0 =Not Selected (1111111) (0000000)
total seven days so index 0=mon, 1=tues, 2=wed, 3=thurs, 4=friday, 5=sat, 6=sunday..and so on..
here 1111111 means all day selected, 0000000 all day not selected, 0001000 only thursday is selected.
I have also discovered a way i.e. convert your so called values to a JSON Array and then store the complete JSON String to an entity/field in Database.
It helps in serving the values easily and effectivly.
Create another table with a column for each day, boolean value. Make an association to this table by integer id (use a foreign key) This is the relational way of solving the problem.