I am trying to insert around 2800 records into the sqlite database, it is taking 150 sec, which is way too much! Could anyone please tell how to optimize this insertion.
public void createVariantEntry(ArrayList<ArrayList<String>> str) {
InsertHelper ih = new InsertHelper(Database, VARIANT_TABLE_NAME);
final int varid = ih.getColumnIndex(VARIANT_ID);
final int varmakeid = ih.getColumnIndex(VARIANT_MAKE_ID);
final int varmodid = ih.getColumnIndex(VARIANT_MODEL_ID);
final int varname = ih.getColumnIndex(VARIANT_NAME);
final int varposteddate = ih.getColumnIndex(VARIANT_POSTED_DATE);
for(int i=0;i<1253;i++)
{
ih.prepareForInsert();
ih.bind(varid, str.get(i).get(0));
ih.bind(varmakeid, str.get(i).get(1));
ih.bind(varmodid, str.get(i).get(2));
ih.bind(varname, str.get(i).get(3));
ih.bind(varposteddate, str.get(i).get(4));
ih.execute();
}
for(int i=1255;i<str.size();i++)
{
ih.prepareForInsert();
ih.bind(varid, str.get(i).get(0));
ih.bind(varmakeid, str.get(i).get(1));
ih.bind(varmodid, str.get(i).get(2));
ih.bind(varname, str.get(i).get(3));
ih.bind(varposteddate, str.get(i).get(4));
ih.execute();
}
ih.close();
}
a great boost in performance will be gained when using transactions.
try {
SQLiteDatabase db = MySQLiteOpenHelper.getWritableDatabse();
ContentValues values = new ContentValues();
db.beginTransaction();
while ( more_data_to_insert ) {
// put the data in 'values'
values.put("col_1", data_1);
values.put("col_2", data_2);
// ...
values.put("col_n", data_n);
// Insert the row into the database.
db.insert("table_name", null, values);
}
db.setTransactionSuccessful();
} catch ( SQLiteException e ) {
// handle your sqlite errors
} finally {
db.endTransaction();
}
and don't use InsertHelper. its deprecated now.
Here are some general tips that might help you:
You can bulkInsert or applyBatch using ContentProviders to do a bunch of operations in one go:
How to use bulkInsert() function in android?
You can use transactions to speed things up as well:
Android Database Transaction
In some cases DatabaseUtils.InsertHelper has been known to provide faster inserts than the normal sqlite insert:
http://www.outofwhatbox.com/blog/2010/12/android-using-databaseutils-inserthelper-for-faster-insertions-into-sqlite-database/
After this, You'll have to do some benchmarking and optimize for your specific situation analyzing performance vs data integrity tradeoffs etc. Good luck.
Related
This is my first Application with database, I hope that someone can help me to understand this problem. I have this insert method:
public long insertData(String name, int password){
....
contentValues.put(KEY_NAME, name);
contentValues.put(KEY_PASSWORD, password);
return db.insert(DBHelper.TABle_NAME, null, contentValues);
}
I can insert few data with this method, but what about if I have thousands of rows? how can I insert all these data into database? where can I write all these data, in extra class or what?
As others have said, you'll need to do some sort of iteration.
Efficiency can be gained by performing a bulk transaction. Here's an example:
public int bulkInsert(#NonNull ContentValues[] values) {
int insertCount = 0;
SQLiteDatabase db = mSqlHelper.getWritableDatabase();
try {
db.beginTransaction();
for (ContentValues value : values) {
if (db.insertOrThrow(tableName, null, value) == -1) {
throw new Exception("Unknown error while inserting entry in database.");
}
insertCount++;
}
db.setTransactionSuccessful();
} catch (Exception e) {
Log.e(LOG_TAG, "An error occurred while bulk-inserting database entries.\n" + e.getMessage(), e);
} finally {
db.endTransaction();
}
return insertCount;
}
There is no 'bulk load' facility that I'm aware of.
You'd just have to spin through the list, and insert the items.
You might want to think about why you're potentially trying to insert thousands of items into a database on a hardware-limited device like a phone or a tablet.
Might it be better to put the data on a server, and create an API that you can use to load data (for display) by pages?
you can do it the same way, that you do with few data, you only need to catch the thousands rows to insert into your database using your method, you can use asyntask, or a service to do that
You can use the same method to insert any amount of records, whether it's 1 or 1,000. Use a loop to call your insert method and add your records to your database. Consider putting your database executions in an AsyncTask to prevent your UI thread from hanging.
Your data can come from anywhere, as long as it's formatted to fit your function parameters String, int
I am new to android and maybe its a silly question but i am not getting it. See i am designing a game in which we give scores to some persons. So i want to store the names of the persons in a database while installation and then their scores set to 0 initially which will be updated according to what the users select. Here i am not able to figure out that how should i enter the data as it will be around 100 names and their scores. Using INSERT INTO() statement will make it like 100 statements. So is there any short method like can we do it through strings or something. Just guessing though. Any help would be appreciated.
You don't hard-code names or scores into your SQL statements. Instead, you use parameters.
var command = new SQLiteCommand()
command.CommandText = "INSERT INTO Scores (name, score) VALUES(#name, #score)";
command.CommandType = CommandType.Text;
foreach (var item in data)
{
command.Parameters.Add(new SQLiteParameter("#name", item.Name));
command.Parameters.Add(new SQLiteParameter("#score", item.Score));
command.ExecuteNonQuery();
}
and then just loop through all of the names and scores.
I recommend you using a transaction.
You can archive this stating you want to use a transaction with beginTransaction(), do all the inserts on makeAllInserts() with a loop and if everything works then call setTransactionSuccessful() to do it in a batch operation. If something goes wrong, on the finally section you will call endTransaction() without setting the success, this will execute a rollback.
db.beginTransaction();
try {
makeAllInserts();
db.setTransactionSuccessful();
}catch {
//Error in between database transaction
}finally {
db.endTransaction();
}
For the makeAllInserts function, something like this could work out:
public void makeAllInserts() {
for(int i = 0; i < myData.size(); i++) {
myDataBase = openDatabase();
ContentValues values = new ContentValues();
values.put("name", myData.get(i).getName());
values.put("score", myData.get(i).getScore());
myDataBase.insert("MYTABLE", nullColumnHack, values);
}
}
If you also want to know about the nullColumnHack here you have a good link -> https://stackoverflow.com/a/2663620/709671
Hope it helps.
I have seen a lot of posts on optimizing SQLITE on android with bulk inserts
Currently its taking 90 seconds to do 900 inserts/updates. I added the Begin/End Transaction around them but only saw minor improvements.
So I want to add I believe SQLiteStatement
Here is my code currently
static ArrayList<ContentSystem> csList = new ArrayList<ContentSystem>();
..fill out csList..
_dh.BeginTransaction(Table);
for(int i = 0; i < csList.size(); ++i)
{
addItem(ma, csList.get(i), dh, Table, Key);
}
_dh.setTransactionSuccessful();
_dh.EndTransaction(Table);
public static void addItem(final MainActivity ma, ContentSystem cs,
final DatabaseHandler dh, String Table, String Key)
{
worked = dh.UpdateItem(cs.cv, Table, Key, cs.UUID);
if (worked == 0)
{
worked = dh.AddData(Table, cs.cv);
if (worked <= 0)
{
ErrorLogger.AddError("Error")
}
}
}
My problem is that if my csList contains ~1000 items and some are already in my list, some are not
so I am currently doing a update if the update returns 0 then I do an add
How could I get something like this to work in a bulk statement?
A bit more info
dh.Update
int wok = db.update(table, values, Key + " = ?", new String[] { Item });
dh.Add
int work = (int) db.insert(table, null, values);
ContentSystem is a list of ContentValues
Try INSERT OR REPLACE, instead of either just an update or a failed update followed by an insert.
Are you sure the instantiated SQLiteDatabase object in your DatabaseHandler class is the same as _dh?
You might have started a transaction for _dh, but that might not even be the instantiated object that is actually doing any work.
I am trying to build a sqlite database. It has 2800 entries in them. When i try to insert, it takes a minute or so and later gives the system error message. The respective codes are given below.
In the create database java file,
ContentValues cv4 = new ContentValues();
public long createVariantEntry(String varid, String makeid, String modelid, String varname, String posteddate) {
cv4.put(VARIANT_ID, varid);
cv4.put(VARIANT_MAKE_ID, makeid);
cv4.put(VARIANT_MODEL_ID, modelid);
cv4.put(VARIANT_NAME, varname);
cv4.put(VARIANT_POSTED_DATE, posteddate);
return Database.insert(VARIANT_TABLE_NAME, null, cv4);
}
In the mainActivity,
for(int i = 0; i<build_emp_database.size();i++)
{
md.createVariantEntry(build_emp_database.get(i).get(0), build_emp_database.get(i).get(1), build_emp_database.get(i).get(2), build_emp_database.get(i).get(3), build_emp_database.get(i).get(4));
}
Also, just for 2800 entries, it is taking more than a minute, is there any way to speed them up?? I have several small databases, and have loaded them successfully. This is the only big database and its creating an issue while inserting. Please help where am i going wrong.
Don't try to run the insert operation on the UI thread, for starters.
I suggest you investigate using a content provider as a wrapper around your database. The ContentResolver object provides methods for doing operations in batch, and is in general a more robust way of working with databases. Use an IntentService to run the insert operation in the background.
If you do a lot insert operation, you need use the ContentProviderOperation to optimize your db operation. like these:
ArrayList<ContentProviderOperation> ops = new ArrayList<ContentProviderOperation>();
for(int i = 0; i<build_emp_database.size();i++) {
ContentProviderOperation.Builder builder =
ContentProviderOperation.newInsert(yourUrl);
builder.withValue(VARIANT_ID, varid)
.withValue(VARIANT_MAKE_ID, makeid)
...
ops.add(builder.build());
}
yourContentResolver.applyBatch(yourauthority, ops);
I am working on a fuel use application which will run on Android 1.6 onwards. The bundled SQLite on v1.6 doesn't do foreign keys, so I've had to handle it manually. So far, I have done this using an Android transaction:
public static long addFuelUp(String registrationNumber, String date)
{
SQLiteDatabase db = uKMpgData.getReadableDatabase();
long result = -1;
ContentValues values = new ContentValues();
Cursor vehicleCursor = VehicleDataProvider.getVehicle(registrationNumber);
if(vehicleCursor.moveToNext())
{
Cursor fuelUpsCursor = getFuelUps(registrationNumber, date);
if(!fuelUpsCursor.moveToNext())
{
db.beginTransaction();
try
{
values.put(REGISTRATION_NO_COLUMN, registrationNumber.replace(" ", ""));
values.put(DATE_TIME_COLUMN, date);
result = db.insertOrThrow(FUEL_USE_TABLE_NAME, null, values);
db.setTransactionSuccessful();
}
catch(SQLException e)
{
Log.d("addFuelUp", e.getMessage());
}
finally
{
db.endTransaction();
vehicleCursor.close();
fuelUpsCursor.close();
}
}
}
return result;
}
I.e. fuel data cannot be entered unless there is a matching vehicle registration number in the database.
My question is, is there a better way to do this? I'm not a database expert, but I know you can set up triggers to enforce rules - are triggers more suited to handle constraints?
Cheers,
Barry
Triggers would be a good solution to this problem.
In fact there is an automated way to generate triggers for simulating foreign keys. SQLite for PC provides a utility called "genfkey" which can examine an existing database which uses foreign keys and outputs the corresponding triggers.