I created a database with a table named flagTable, this table only has two fields, which are id(auto increment) and an integer field. Next, in my program, I have a button that will trigger a thread to start. When the thread is starting, it constantly retrieve data from database, and check for the for the value, if the value is equal to one then it will trigger another new Thread, something like this:
private class statusOfStrummingInAnotherDevice extends Thread {
int value;
public void run() {
try{
while(true){
try{
if(flagCursor == null){
flagCursor = cdb1.getFlagAll();
}
}catch(Exception e){break;}
try{
Log.i("MAIN3ACTIVITY","getting status");
int size = cdb1.getSize(flagCursor);
Log.i("MAIN3ACTIVITY","SIZE is" + String.valueOf(xyz));
for(int i = 0 ; i < size ; i++){
flagCursor.moveToPosition(i);
Log.i("MAIN3ACTIVITY","getting status jkasdfasdf");
value = cdb1.getFlag();
if(value == 1){
Log.i("FLAGCURSOR=====>>>>","Succesful");
releasingNotes = new ReleasingNotes(IntendedChord);
releasingNotes.start();
//break;
}
cdb1.updateFlag(0);
Log.i("FLAGCURSOR=====>>>>",String.valueOf(value));
}
flagCursor = null;
}catch(Exception e){break;}
Log.i("MAIN3ACTIVITY","thread is sleeping");
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
break;
}
}
}catch(Exception e){
}
}
}
In the meantime, the data that were retrieved from the database is using this function:
public Cursor getFlagAll(){
return getReadableDatabase().rawQuery(
"SELECT _ID, flag from flagTable", null);
}
And, the data that were updated to the database through this method:
public int updateFlag(int i) {
SQLiteDatabase db = this.getWritableDatabase();
ContentValues contentValues = new ContentValues();
contentValues.put("flag",i);
return db.update("flagTable" , contentValues , "_ID" + "= ?",new String[]{String.valueOf(1)});
}
Now, above codes will give no error, however, the data that were retrieved from the database is always 1, it keeps trigger a new function. In my above codes, I stated if the value is equal to 1, then the current thread will trigger a new thread to start, When its finished, the program will update the current data to 0. So that, the next round of the infinite loop can stop triggering new thread until a the conditon is met. What is problem overhere? did my codes really updated the new value? or I need to referesh the database every time I updated a new value.
Use Listeners to your database.
use SQLiteTransactionListener and do your things in onCommit()
Some guide in details here :
https://developer.android.com/reference/android/database/sqlite/SQLiteTransactionListener.html and
http://www.programcreek.com/java-api-examples/index.php?api=android.database.sqlite.SQLiteTransactionListener
Related
I have a method which reads data from file line by line and takes value between coma, then puts this value into INSERT query. Data in file saved in this way:
–,08:10,–,20:20,08:15,08:16,20:26,20:27,08:20,08:21,20:31,20:32,08:30,08:31,20:40,20:41,08:37,08:38,20:46
20:47,08:48,08:50,20:56,20:57,09:00,09:01,21:07,21:08
08:53,–,17:43,09:01,09:03,09:13,09:15,18:02,18:04,–,–,09:19,09:25
Here is actual my code:
public void insertTime(SQLiteDatabase database, String table) throws FileNotFoundException {
BufferedReader br = null;
String line;
try {
int j = 0;
br = new BufferedReader(new InputStreamReader(context.getAssets().open("time.txt")));
database.beginTransaction();
while ((line = br.readLine()) != null) {
j++;
String query = "INSERT INTO "+table+""+j+" (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
// use comma as separator
String[] time = line.split(",");
for(int i = 1; i < time.length; i+=2) {
statement.bindString(1,time[i-1]);//arrival
statement.bindString(2,time[i]);//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
} catch (IOException e) {
e.printStackTrace();
} finally {
if (br != null) {
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
The problem is that data insert very slow, despite I use SQLiteStatement and transactions. For example, when I insert 69000 rows it takes about 65,929 seconds.
What have I to change in my code to improve speed of insertion ?
UPDATE
OK, I have simplified my code, I got rid of BufferedReader and now it looks like this
public void insertTime(SQLiteDatabase database) throws FileNotFoundException {
database.beginTransaction();
int r = 0;
while (r < 122) {
r++;
String query = "INSERT INTO table_1 (arrival, departure) VALUES (?,?)";
SQLiteStatement statement = database.compileStatement(query);
for(int i = 1; i < 1100; i++) {
statement.bindString(1,i+"");//arrival
statement.bindString(2,i+"");//departure
statement.executeInsert();
statement.clearBindings();
}
}
database.setTransactionSuccessful();
database.endTransaction();
}
But it still so long inserts data, more than 2 min. Do you have any ideas how to increase speed of my second example ?
Here is a very very detailed post on every method of increasing SQL insertion speed.
Move beginTransaction() and setTransactionSuccessful() outside of while loop and it will be way faster.
A new transaction is started for each item in the while() loop.
It might go a bit faster if you only have 1 transaction to do all your insertions.
Also, when your data is corrupt and String.split doesn't give you at least 2 items, then your transaction will not be ended properly due to an Exception being thrown.
Every time you insert a row in a table with indexes, the indexes have to be adjusted. That operation can be costly. Indexes are kept as b-trees and if you hit the rebalance point, you're bound to have a slowdown. One thing you can do to test this is to remove your indexes. You could also drop the indexes, insert, then re-create the indexes.
For those using JDBC (Java): to be sure, do you first set the autoCommit to FALSE?
I guess so, because you work with explicit transactions.
The performace gain I got by explicitly setting the autocommit off was over 1000 times!
So:
Class.forName("org.sqlite.JDBC");
String urlInput = "jdbc:sqlite:" + databaseFile;
databaseConnection = DriverManager.getConnection(urlInput);
databaseConnection.setAutoCommit( false);
And:
String sql = "INSERT INTO " + TABLE_NAME + " ( type, bi, ci, fvi, tvi, content_type) VALUES ('V',?,?,?,?,'rtf')";
PreparedStatement psi = databaseConnection.prepareStatement(sql);
for( Item item : items) {
psi.setInt(1, item.property1);
// ....
count = psi.executeUpdate();
}
databaseConnection.commit();
databaseConnection.setAutoCommit( true);
So, when somebody forgets this, this may have a huge effect.
Many people said that SQLiteDatabase is thread-safe,but ,when i ran some simple tests with setup like this:
private class MyRunnable implements Runnable{
#Override
public void run() {
for(int i = 0 ;i < 100 ; i++){
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
Cursor cursor = database.query(true,"testdb",new String[]{"num"},"id=?",
new String[]{"1"},null,null,null,null);
cursor.moveToFirst();
int original = cursor.getInt(0);
ContentValues values = new ContentValues();
values.put("num",++original);
database.update("testdb",values,"id=?",new String[]{"1"});
}
}
}
MyRunnable runnable1 = new MyRunnable();
MyRunnable runnable2 = new MyRunnable();
new Thread(runnable1).start();
new Thread(runnable2).start();
In MyRunnable ,there is a loop that runs 100 times. Each time, the num field will add by 1 and the num's initial value is 0. we can see that the above code's expected result is 200,but i only get 197.
so does that mean SQLiteDatabase is not thread-safe?
No, the way you are handling the data is just not thread-safe. You could use an atomic SQL-query like:
UPDATE table SET num = num + 1
Instead of retrieving the data in your application code, modifying it and storing it back.
SQLite being threadsafe means that you can safely use all SQLite functions in multiple threads at once, it does not mean the database gets locked automatically depending on what you are doing. If your thread gets interrupted anywhere between the query() call and the update() call, returning to the thread at that point will mean the data you retrieved earlier is no longer up to date.
Ok, I've got this Retrofit Call that receives a list of objects and insert the into a local SQLite database. I want to display a message saying that the operation was successful with a Ok button that when pressed opens a new activity.
How do I check if my Query has finished so I can show the message?
final ContactApi contactApi = retrofit.create(ContactApi.class);
Call<List<Contact>> callContact = contactApi.getContact(token);
callContact.enqueue(new Callback<List<Contact>>() {
#Override
public void onResponse(Response<List<Contact>> response, Retrofit retrofit) {
List<Contact> contactList = response.body();
if (contactList != null) {
try {
DBHelper dbHelper = new DBHelper(TokenActivity.this, token);
SQLiteDatabase conn = dbHelper.getWritableDatabase();
RepoContact repocontact = new RepoContact(conn);
// Inserts each contact into the database
for (Contatc c : contactList) {
repositorioCadastro.inserirCadastro(c);
Log.i("ACTTOKEN", "Contact insert ID: " + c.getId());
}
} catch (SQLiteException e) {
Log.i("ACTTOKEN", "Faillure on insert: " + e.getMessage());
}
}
wrap your code in try{...}finally{...} blocks with a listener ( beginTransactionWithListener(SQLiteTransactionListener transactionListener)), and use the transactionListner to check whether everything went well within the transaction, in addition to everything within the try/finally.
what you have is good, just try adding finally block..
something like this..
db.beginTransaction();
try {
...
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
You can try a different loop, something like this:
for(int i = 0; i < contactList.size(); i++) {
Contact c = contactList.get(i);
repositorioCadastro.inserirCadastro(c);
Log.i("ACTTOKEN", "Contact insert ID: " + c.getId());
if(i == (contactList.size() - 1)) {
// DO SOMETHING HERE
}
}
You may check insert statement return a long when query successfully executed then long value.
db.insert()
returns the row ID of the newly inserted row, or -1 if an error occurred
This question already has answers here:
Android SQLite database: slow insertion
(5 answers)
Closed 8 years ago.
I want to inset more than 3000 records in android SQLite but the following code take to much time for the data inserion
Here is my code.
public boolean addSale(Sale objSale) {
ContentValues values = new ContentValues();
values.put(SALE_BRANCH, objSale.getBranch());
values.put(SALE_SUPPLIER, objSale.getSupplier());
values.put(SALE_BUYER, objSale.getBuyer());
values.put(SALE_CAT1, objSale.getCat1());
values.put(SALE_CAT2, objSale.getCat2());
values.put(SALE_CAT3, objSale.getCat3());
values.put(SALE_CAT4, objSale.getCat4());
values.put(SALE_CAT5, objSale.getCat5());
values.put(SALE_CAT6, objSale.getCat6());
values.put(SALE_DESIGNO, objSale.getDesigNo());
values.put(SALE_ITEMSIZE, objSale.getItemSize());
values.put(SALE_SALEQTY, objSale.getSaleQty());
values.put(SALE_STOCKQTY, objSale.getStockQty());
values.put(SALE_FinalProduct, objSale.getFinalProduct());
values.put(SALE_PriceRange, objSale.getPriceRange());
values.put(SALE_CoreNonCore, objSale.getCoreNonCore());
values.put(SALE_Color, objSale.getColor());
values.put(SALE_GSLCode, objSale.getGSLCode());
values.put(SALE_Wanted, objSale.getWanted());
values.put(SALE_Pqty, objSale.getPqty());
values.put(SALE_MRP, objSale.getMRP());
values.put(SALE_PRate, objSale.getPRate());
SQLiteDatabase db = this.getWritableDatabase();
db.insert(TABLE_SALE, null, values);
db.close();
return true;
}
And this one is my Asynk Task. Where i fetch data from webservice and insert into SQLite
protected void onPostExecute(String result)
{
try {
String l = result.replace("\\", "");
l = l.replace("''", "");
String sdsd = l.substring(1, l.length() - 1);
JSONArray jsonArray = new JSONArray(sdsd);
Log.i("JSON", "Number of surveys in feed: " +jsonArray.length());
/*if(db.delSaleData()){
}*/
for (int i = 0; i < jsonArray.length(); i++)
{
JSONObject jsonObject = jsonArray.getJSONObject(i);
sl.setBranch(jsonObject.getString("Branch"));
sl.setSupplier(jsonObject.getString("Supplier"));
sl.setBuyer(jsonObject.getString("Buyer"));
sl.setCat1(jsonObject.getString("Cat1"));
sl.setCat2(jsonObject.getString("Cat2"));
sl.setCat3(jsonObject.getString("Cat3"));
sl.setCat4(jsonObject.getString("Cat4"));
sl.setCat5(jsonObject.getString("Cat5"));
sl.setCat6(jsonObject.getString("Cat6"));
sl.setDesigNo(jsonObject.getString("DesigNo"));
sl.setItemSize(jsonObject.getString("ItemSize"));
sl.setSaleQty(jsonObject.getString("SaleQty"));
sl.setStockQty(jsonObject.getString("StockQty"));
sl.setFinalProduct(jsonObject.getString("FinalProduct"));
sl.setPriceRange(jsonObject.getString("PriceRange"));
sl.setCoreNonCore(jsonObject.getString("CoreNonCore"));
sl.setColor(jsonObject.getString("Color"));
sl.setGSLCode(jsonObject.getString("GSLCode"));
sl.setWanted(jsonObject.getString("Wanted"));
sl.setPqty(jsonObject.getString("Pqty"));
sl.setMRP(jsonObject.getString("MRP"));
sl.setPRate(jsonObject.getString("PRate"));
if(db.addSale(sl))
{
//Toast.makeText(getApplicationContext(), " Insert.." , Toast.LENGTH_SHORT).show();
}
}
setData();
} catch (Exception e) {
e.printStackTrace();
}
dialog.dismiss();
setTableData("All");
}
use the SQLiteStatement for example:
private void bulkInsertRecords(String[] records) {
String sql = "INSERT INTO "+ SAMPLE_TABLE_NAME +" VALUES (?,?,?);";
SQLiteStatement statement = sampleDB.compileStatement(sql); //Este é o prepare
sampleDB.beginTransaction();
for (int i = 0; i<records.length; i++) {
statement.clearBindings();
statement.bindString(1, records[0]);
statement.bindString(2, records[1]);
statement.bindString(3, records[2]);
statement.execute();
}
sampleDB.setTransactionSuccessful();
sampleDB.endTransaction();
}
Don't open and close the database every time you want to do an insert. Open it once and then close it when exiting the app.
Use the Asyncclass. Do the operation onBackground, not in the main UI thread.
UPDATE
On your code, you insert the SQLite via onPost, not onBackground. onPost do things on UI thread, while onBackground do it on the separate (background) thread so it doesnt effect the UI. So move the operation code to onBackground.
I currently have a CSV file that I parse and am trying to insert the data into the android database. The problem I am having is that it is taking way too long to insert all of the data. It's a good amount of data but I feel like it shouldn't take 20min or so to complete.
Basically, I create my database, then begin the parsing. While parsing through each individual CSV row, I grab the required data and insert it into the database. In total there are around 40000 rows.
Is there any way I can speed up this process? I have tried batch inserts but it never really helped (unless I did it wrong).
Code down below.
Thanks.
DatabaseHelper (i have two insert commands based on the amount of data in each csv row):
// add zipcode
public void add9Zipcode(String zip, String city, String state, String lat,
String longi, String decom) {
// get db and content values
SQLiteDatabase db = this.getWritableDatabase();
ContentValues values = new ContentValues();
db.beginTransaction();
try{
// add the values
values.put(KEY_ZIP, zip);
values.put(KEY_STATE, state);
values.put(KEY_CITY, city);
values.put(KEY_LAT, lat);
values.put(KEY_LONG, longi);
values.put(KEY_DECOM, decom);
// execute the statement
db.insert(TABLE_NAME, null, values);
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
db.close();
}
public void add12Zipcode(String zip, String city, String state, String lat,
String longi, String decom, String tax, String pop, String wages) {
// get db and content values
SQLiteDatabase db = this.getWritableDatabase();
ContentValues values = new ContentValues();
db.beginTransaction();
try{
// add the values
values.put(KEY_ZIP, zip);
values.put(KEY_STATE, state);
values.put(KEY_CITY, city);
values.put(KEY_LAT, lat);
values.put(KEY_LONG, longi);
values.put(KEY_DECOM, decom);
values.put(KEY_TAX, tax);
values.put(KEY_POP, pop);
values.put(KEY_WAGES, wages);
// execute the statement
db.insert(TABLE_NAME, null, values);
db.setTransactionSuccessful();
} finally{
db.endTransaction();
}
db.close();
}
Parse File:
public void parse(ArrayList<String> theArray, DatabaseHandler db) {
String[] data = null;
// while loop to get split the data into new lines
// for loop to split each string in the array list of zipcodes
for (int x = 0; x < theArray.size(); x++) {
if(x == 10000 || x == 20000 || x == 30000 || x == 40000){
Log.d(TAG, "x is 10k, 20k, 30k, 40k");
}
// split string first into an array
data = theArray.get(x).split(",");
// separate based on the size of the array: 9 or 12
if (data.length == 9) {
db.add9Zipcode(data[0], data[2], data[3], data[5], data[6],
data[8]);
} else if (data.length == 12) {
db.add12Zipcode(data[0], data[2], data[3], data[5], data[6],
data[8], data[9], data[10], data[11]);
/*
* theZip.zip = data[0]; theZip.city = data[2]; theZip.state =
* data[3]; theZip.lat = data[5]; theZip.longi = data[6];
* theZip.decom = data[8]; theZip. = data[9]; theZip.population
* = data[10]; theZip.wages = data[11];
*/
}
}
Refer to this answer I made previously: Inserting 1000000 rows in sqlite3 database
In short, use an InsertHelper and do more than one insert per transaction - unless you did something wonky, the speed increase should be noticeable.
Edit:
In short:
Your SQLiteOpenHelper should be a singleton used across your entire application.
Don't go around calling close() on your SQLiteDatabase instance - it's cached in the SQLiteOpenHelper and every time you close you force the helper to reopen it.
Batch your inserts, start a transaction outside the call to the addZipCode methods and mark it as successful after you've done all the inserts - then commit the transaction.
Use an InsertHelper - it will format the insert properly as a prepared statement and is nice and reusable.
Be mindful of synchronizing access to the database - unless you intend to do all your database work on the UI-thread (which is not recommended) - you either need to enable locking or guard access to the database to avoid concurrent access.