Related
How can i get the autoincrement value in thansaction body?
Code
public void insertAllStudents(List<Student> students) {
String sql = "INSERT INTO "+ StudentEntry.TABLE_NAME +" VALUES (?,?,?,?,?);";
SQLiteDatabase db = this.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
db.beginTransaction();
for (Student student: students) {
statement.clearBindings();
statement.bindString(2, student.getId());
statement.bindString(3, student.getFirstName());
statement.bindString(4, student.getLastName());
statement.bindLong(5, student.getBirthday());
statement.execute();
}
db.setTransactionSuccessful();
db.endTransaction();
}
The first column (_ID) is autoincrement field. Is it opportunity to get this value?
student.getId() -that's not id from database, that's different id.
If you change your code to use db.insert(), this method returns the id of the inserted row - see Get generated id after insert.
There is also a specialised SQLite function to get the last inserted row if you'd prefer to keep compiling statements, see Best way to get the ID of the last inserted row on SQLite
edit: example using db.insert(). This isn't tested but should be pretty close to functional.
db.beginTransaction();
boolean success = true;
final ContentValues values = new ContentValues();
for (final Student student: students) {
values.put("student_id", student.getId());
values.put("first_name", student.getFirstName());
values.put("last_name", student.getLastName());
values.put("birthday", student.getBirthday());
final long id = db.insert("my_table", null, values);
if (id == -1) {
success = false;
break;
}
// TODO do your thing with id here.
}
if (success) {
db.setTransactionSuccessful();
}
db.endTransaction();
Instead of statement.execute(), you can do statement.executeInsert(). This returns the row ID of the inserted row. Or, as #Tom suggested, you can use db.insert() instead, and it will also return the inserted row ID. Using a compiled statement like you are doing now is faster though.
If you want to try the db.insert() approach, it would look something like this:
ContentValues values = new ContentValues();
for (Student student: students) {
// use whatever constants you have for column names instead of these:
values.put(COLUMN_STUDENT_ID, student.getId());
values.put(COLUMN_STUDENT_FIRSTNAME, student.getFirstName());
values.put(COLUMN_STUDENT_LASTNAME, student.getLastName());
values.put(COLUMN_STUDENT_BIRTHDAY, student.getBirthday());
db.insert(StudentEntry.TABLE_NAME, null, values);
}
I need to parse a fairly large XML file (varying between about a hundred kilobytes and several hundred kilobytes), which I'm doing using Xml#parse(String, ContentHandler). I'm currently testing this with a 152KB file.
During parsing, I also insert the data in an SQLite database using calls similar to the following: getWritableDatabase().insert(TABLE_NAME, "_id", values). All of this together takes about 80 seconds for the 152KB test file (which comes down to inserting roughly 200 rows).
When I comment out all insert statements (but leave in everything else, such as creating ContentValues etc.) the same file takes only 23 seconds.
Is it normal for the database operations to have such a big overhead? Can I do anything about that?
You should do batch inserts.
Pseudocode:
db.beginTransaction();
for (entry : listOfEntries) {
db.insert(entry);
}
db.setTransactionSuccessful();
db.endTransaction();
That increased the speed of inserts in my apps extremely.
Update:
#Yuku provided a very interesting blog post: Android using inserthelper for faster insertions into sqlite database
Since the InsertHelper mentioned by Yuku and Brett is deprecated now (API level 17), it seems the right alternative recommended by Google is using SQLiteStatement.
I used the database insert method like this:
database.insert(table, null, values);
After I also experienced some serious performance issues, the following code speeded my 500 inserts up from 14.5 sec to only 270 ms, amazing!
Here is how I used SQLiteStatement:
private void insertTestData() {
String sql = "insert into producttable (name, description, price, stock_available) values (?, ?, ?, ?);";
dbHandler.getWritableDatabase();
database.beginTransaction();
SQLiteStatement stmt = database.compileStatement(sql);
for (int i = 0; i < NUMBER_OF_ROWS; i++) {
//generate some values
stmt.bindString(1, randomName);
stmt.bindString(2, randomDescription);
stmt.bindDouble(3, randomPrice);
stmt.bindLong(4, randomNumber);
long entryID = stmt.executeInsert();
stmt.clearBindings();
}
database.setTransactionSuccessful();
database.endTransaction();
dbHandler.close();
}
Compiling the sql insert statement helps speed things up. It can also require more effort to shore everything up and prevent possible injection since it's now all on your shoulders.
Another approach which can also speed things up is the under-documented android.database.DatabaseUtils.InsertHelper class. My understanding is that it actually wraps compiled insert statements. Going from non-compiled transacted inserts to compiled transacted inserts was about a 3x gain in speed (2ms per insert to .6ms per insert) for my large (200K+ entries) but simple SQLite inserts.
Sample code:
SQLiteDatabse db = getWriteableDatabase();
//use the db you would normally use for db.insert, and the "table_name"
//is the same one you would use in db.insert()
InsertHelper iHelp = new InsertHelper(db, "table_name");
//Get the indices you need to bind data to
//Similar to Cursor.getColumnIndex("col_name");
int first_index = iHelp.getColumnIndex("first");
int last_index = iHelp.getColumnIndex("last");
try
{
db.beginTransaction();
for(int i=0 ; i<num_things ; ++i)
{
//need to tell the helper you are inserting (rather than replacing)
iHelp.prepareForInsert();
//do the equivalent of ContentValues.put("field","value") here
iHelp.bind(first_index, thing_1);
iHelp.bind(last_index, thing_2);
//the db.insert() equilvalent
iHelp.execute();
}
db.setTransactionSuccessful();
}
finally
{
db.endTransaction();
}
db.close();
If the table has an index on it, consider dropping it prior to inserting the records and then adding it back after you've commited your records.
If using a ContentProvider:
#Override
public int bulkInsert(Uri uri, ContentValues[] bulkinsertvalues) {
int QueryType = sUriMatcher.match(uri);
int returnValue=0;
SQLiteDatabase db = mOpenHelper.getWritableDatabase();
switch (QueryType) {
case SOME_URI_IM_LOOKING_FOR: //replace this with your real URI
db.beginTransaction();
for (int i = 0; i < bulkinsertvalues.length; i++) {
//get an individual result from the array of ContentValues
ContentValues values = bulkinsertvalues[i];
//insert this record into the local SQLite database using a private function you create, "insertIndividualRecord" (replace with a better function name)
insertIndividualRecord(uri, values);
}
db.setTransactionSuccessful();
db.endTransaction();
break;
default:
throw new IllegalArgumentException("Unknown URI " + uri);
}
return returnValue;
}
Then the private function to perform the insert (still inside your content provider):
private Uri insertIndividualRecord(Uri uri, ContentValues values){
//see content provider documentation if this is confusing
if (sUriMatcher.match(uri) != THE_CONSTANT_IM_LOOKING_FOR) {
throw new IllegalArgumentException("Unknown URI " + uri);
}
//example validation if you have a field called "name" in your database
if (values.containsKey(YOUR_CONSTANT_FOR_NAME) == false) {
values.put(YOUR_CONSTANT_FOR_NAME, "");
}
//******add all your other validations
//**********
//time to insert records into your local SQLite database
SQLiteDatabase db = mOpenHelper.getWritableDatabase();
long rowId = db.insert(YOUR_TABLE_NAME, null, values);
if (rowId > 0) {
Uri myUri = ContentUris.withAppendedId(MY_INSERT_URI, rowId);
getContext().getContentResolver().notifyChange(myUri, null);
return myUri;
}
throw new SQLException("Failed to insert row into " + uri);
}
I have a cursor that contains 3 columns from my database. I need to iterate through, and add the IDs to another table. I tried using the code below, but it caused an "Application not responding" message, and took about 10 seconds to insert all the rows (about 1000).
I have changed the code so that I iterate through the cursor and insert into the DB separately, and this now takes about 500ms. Although it seems I have fixed my problem, I don't understand why.
Can someone please explain why the code below takes so long to execute?
public void setNowPlayingSongs(SQLiteDatabase db, Cursor cursor) {
// Clear all the current songs
db.delete(TABLE_NOW_PLAYING, null, null);
ContentValues cv = new ContentValues();
cursor.moveToFirst();
int index = cursor.getColumnIndex(COL_ID);
while (!cursor.isAfterLast()){
cv.put(COL_SONG_ID, cursor.getInt(index));
db.insert(TABLE_NOW_PLAYING, null, cv);
cursor.moveToNext();
}
}
And why this is so much faster? I don't understand why it's faster to loop through the cursor and then loop through a list. I thought the method above should be faster?
public void setNowPlayingSongs(SQLiteDatabase db, Cursor cursor) {
ContentValues cv;
List<Integer> ids;
int index;
// Clear all the current songs
db.delete(TABLE_NOW_PLAYING, null, null);
// Check which column holds the IDs
index = cursor.getColumnIndex(COL_ID);
// Add the ids to a list
ids = new ArrayList<Integer>();
cursor.moveToFirst();
while (!cursor.isAfterLast()) {
ids.add(cursor.getInt(index));
cursor.moveToNext();
}
// Insert the IDs into the now playing table.
cv = new ContentValues();
db.beginTransaction();
for (Integer id : ids) {
cv.put(COL_SONG_ID, id);
db.insert(TABLE_NOW_PLAYING, null, cv);
}
db.setTransactionSuccessful();
db.endTransaction();
}
When you do individual inserts, each one is actually wrapped in a SQLite transaction. If you instead perform all of your inserts in a single transaction, this should speed things up substantially.
db.beginTransaction();
try {
// do your inserts
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
Currently, I am using the following statement to create a table in an SQLite database on an Android device.
CREATE TABLE IF NOT EXISTS 'locations' (
'_id' INTEGER PRIMARY KEY AUTOINCREMENT, 'name' TEXT,
'latitude' REAL, 'longitude' REAL,
UNIQUE ( 'latitude', 'longitude' )
ON CONFLICT REPLACE );
The conflict-clause at the end causes that rows are dropped when new inserts are done that come with the same coordinates. The SQLite documentation contains further information about the conflict-clause.
Instead, I would like to keep the former rows and just update their columns. What is the most efficient way to do this in a Android/SQLite environment?
As a conflict-clause in the CREATE TABLE statement.
As an INSERT trigger.
As a conditional clause in the ContentProvider#insert method.
... any better you can think off
I would think it is more performant to handle such conflicts within the database. Also, I find it hard to rewrite the ContentProvider#insert method to consider the insert-update scenario. Here is code of the insert method:
public Uri insert(Uri uri, ContentValues values) {
final SQLiteDatabase db = mOpenHelper.getWritableDatabase();
long id = db.insert(DatabaseProperties.TABLE_NAME, null, values);
return ContentUris.withAppendedId(uri, id);
}
When data arrives from the backend all I do is inserting the data as follows.
getContentResolver.insert(CustomContract.Locations.CONTENT_URI, contentValues);
I have problems figuring out how to apply an alternative call to ContentProvider#update here. Additionally, this is not my favored solution anyways.
Edit:
#CommonsWare: I tried to implement your suggestion to use INSERT OR REPLACE. I came up with this ugly piece of code.
private static long insertOrReplace(SQLiteDatabase db, ContentValues values, String tableName) {
final String COMMA_SPACE = ", ";
StringBuilder columnsBuilder = new StringBuilder();
StringBuilder placeholdersBuilder = new StringBuilder();
List<Object> pureValues = new ArrayList<Object>(values.size());
Iterator<Entry<String, Object>> iterator = values.valueSet().iterator();
while (iterator.hasNext()) {
Entry<String, Object> pair = iterator.next();
String column = pair.getKey();
columnsBuilder.append(column).append(COMMA_SPACE);
placeholdersBuilder.append("?").append(COMMA_SPACE);
Object value = pair.getValue();
pureValues.add(value);
}
final String columns = columnsBuilder.substring(0, columnsBuilder.length() - COMMA_SPACE.length());
final String placeholders = placeholderBuilder.substring(0, placeholdersBuilder.length() - COMMA_SPACE.length());
db.execSQL("INSERT OR REPLACE INTO " + tableName + "(" + columns + ") VALUES (" + placeholders + ")", pureValues.toArray());
// The last insert id retrieved here is not safe. Some other inserts can happen inbetween.
Cursor cursor = db.rawQuery("SELECT * from SQLITE_SEQUENCE;", null);
long lastId = INVALID_LAST_ID;
if (cursor != null && cursor.getCount() > 0 && cursor.moveToFirst()) {
lastId = cursor.getLong(cursor.getColumnIndex("seq"));
}
cursor.close();
return lastId;
}
When I check the SQLite database, however, equal columns are still removed and inserted with new ids. I do not understand why this happens and thought the reason is my conflict-clause. But the documentation states the opposite.
The algorithm specified in the OR clause of an INSERT or UPDATE
overrides any algorithm specified in a CREATE TABLE. If no algorithm
is specified anywhere, the ABORT algorithm is used.
Another disadvantage of this attempt is that you loose the value of the id which is return by an insert statement. To compensate this, I finally found an option to ask for the last_insert_rowid. It is as explained in the posts of dtmilano and swiz. I am, however, not sure if this is safe since another insert can happen inbetween.
I can understand the perceived notion that it is best for performance to do all this logic in SQL, but perhaps the simplest (least code) solution is the best one in this case? Why not attempt the update first, and then use insertWithOnConflict() with CONFLICT_IGNORE to do the insert (if necessary) and get the row id you need:
public Uri insert(Uri uri, ContentValues values) {
final SQLiteDatabase db = mOpenHelper.getWritableDatabase();
String selection = "latitude=? AND longitude=?";
String[] selectionArgs = new String[] {values.getAsString("latitude"),
values.getAsString("longitude")};
//Do an update if the constraints match
db.update(DatabaseProperties.TABLE_NAME, values, selection, null);
//This will return the id of the newly inserted row if no conflict
//It will also return the offending row without modifying it if in conflict
long id = db.insertWithOnConflict(DatabaseProperties.TABLE_NAME, null, values, CONFLICT_IGNORE);
return ContentUris.withAppendedId(uri, id);
}
A simpler solution would be to check the return value of update() and only do the insert if the affected count was zero, but then there would be a case where you could not obtain the id of the existing row without an additional select. This form of insert will always return to you the correct id to pass back in the Uri, and won't modify the database more than necessary.
If you want to do a large number of these at once, you might look at the bulkInsert() method on your provider, where you can run multiple inserts inside a single transaction. In this case, since you don't need to return the id of the updated record, the "simpler" solution should work just fine:
public int bulkInsert(Uri uri, ContentValues[] values) {
final SQLiteDatabase db = mOpenHelper.getWritableDatabase();
String selection = "latitude=? AND longitude=?";
String[] selectionArgs = null;
int rowsAdded = 0;
long rowId;
db.beginTransaction();
try {
for (ContentValues cv : values) {
selectionArgs = new String[] {cv.getAsString("latitude"),
cv.getAsString("longitude")};
int affected = db.update(DatabaseProperties.TABLE_NAME,
cv, selection, selectionArgs);
if (affected == 0) {
rowId = db.insert(DatabaseProperties.TABLE_NAME, null, cv);
if (rowId > 0) rowsAdded++;
}
}
db.setTransactionSuccessful();
} catch (SQLException ex) {
Log.w(TAG, ex);
} finally {
db.endTransaction();
}
return rowsAdded;
}
In truth, the transaction code is what makes things faster by minimizing the number of times the database memory is written to the file, bulkInsert() just allows multiple ContentValues to be passed in with a single call to the provider.
One solution is to create a view for the locations table with a INSTEAD OF trigger on the view, then insert into the view. Here's what that would look like:
View:
CREATE VIEW locations_view AS SELECT * FROM locations;
Trigger:
CREATE TRIGGER update_location INSTEAD OF INSERT ON locations_view FOR EACH ROW
BEGIN
INSERT OR REPLACE INTO locations (_id, name, latitude, longitude) VALUES (
COALESCE(NEW._id,
(SELECT _id FROM locations WHERE latitude = NEW.latitude AND longitude = NEW.longitude)),
NEW.name,
NEW.latitude,
NEW.longitude
);
END;
Instead of inserting into the locations table, you insert into the locations_view view. The trigger will take care of providing the correct _id value by using the sub-select. If, for some reason, the insert already contains an _id the COALESCE will keep it and override an existing one in the table.
You'll probably want to check how much the sub-select affects performance and compare that to other possible changes you could make, but it does allow you keep this logic out of your code.
I tried some other solutions involving triggers on the table itself based on INSERT OR IGNORE, but it seems that BEFORE and AFTER triggers only trigger if it will actually insert into the table.
You might find this answer helpful, which is the basis for the trigger.
Edit: Due to BEFORE and AFTER triggers not firing when an insert is ignored (which could then have been updated instead), we need to rewrite the insert with an INSTEAD OF trigger. Unfortunately, those don't work with tables - we have to create a view to use it.
INSERT OR REPLACE works just like ON CONFLICT REPLACE. It will delete the row if the row with the unique column already exists and than it does an insert. It never does update.
I would recommend you stick with your current solution, you create table with ON CONFLICT clausule, but every time you insert a row and the constraint violation occurs, your new row will have new _id as origin row will be deleted.
Or you can create table without ON CONFLICT clausule and use INSERT OR REPLACE, you can use insertWithOnConflict() method for that, but it is available since API level 8, requires more coding and leads to the same solution as table with ON CONFLICT clausule.
If you still want to keep your origin row, it means you want to keep the same _id you will have to make two queries, first one for inserting a row, second to update a row if insertion failed (or vice versa). To preserve consistency, you have to execute queries in a transaction.
db.beginTransaction();
try {
long rowId = db.insert(table, null, values);
if (rowId == -1) {
// insertion failed
String whereClause = "latitude=? AND longitude=?";
String[] whereArgs = new String[] {values.getAsString("latitude"),
values.getAsString("longitude")};
db.update(table, values, whereClause, whereArgs);
// now you have to get rowId so you can return correct Uri from insert()
// method of your content provider, so another db.query() is required
}
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
Use insertWithOnConflict and set the last parameter (conflictAlgorithm) to CONFLICT_REPLACE.
Read more at the following links:
insertWithOnConflict documentation
CONFLICT_REPLACE flag
for me, none of the approaches are work if I don't have "_id"
you should first call update, if the affected rows are zero, then insert it with ignore:
String selection = MessageDetailTable.SMS_ID+" =?";
String[] selectionArgs = new String[] { String.valueOf(md.getSmsId())};
int affectedRows = db.update(MessageDetailTable.TABLE_NAME, values, selection,selectionArgs);
if(affectedRows<=0) {
long id = db.insertWithOnConflict(MessageDetailTable.TABLE_NAME, null, values, SQLiteDatabase.CONFLICT_IGNORE);
}
Use INSERT OR REPLACE.
This is the correct way to do it.
I need to parse a fairly large XML file (varying between about a hundred kilobytes and several hundred kilobytes), which I'm doing using Xml#parse(String, ContentHandler). I'm currently testing this with a 152KB file.
During parsing, I also insert the data in an SQLite database using calls similar to the following: getWritableDatabase().insert(TABLE_NAME, "_id", values). All of this together takes about 80 seconds for the 152KB test file (which comes down to inserting roughly 200 rows).
When I comment out all insert statements (but leave in everything else, such as creating ContentValues etc.) the same file takes only 23 seconds.
Is it normal for the database operations to have such a big overhead? Can I do anything about that?
You should do batch inserts.
Pseudocode:
db.beginTransaction();
for (entry : listOfEntries) {
db.insert(entry);
}
db.setTransactionSuccessful();
db.endTransaction();
That increased the speed of inserts in my apps extremely.
Update:
#Yuku provided a very interesting blog post: Android using inserthelper for faster insertions into sqlite database
Since the InsertHelper mentioned by Yuku and Brett is deprecated now (API level 17), it seems the right alternative recommended by Google is using SQLiteStatement.
I used the database insert method like this:
database.insert(table, null, values);
After I also experienced some serious performance issues, the following code speeded my 500 inserts up from 14.5 sec to only 270 ms, amazing!
Here is how I used SQLiteStatement:
private void insertTestData() {
String sql = "insert into producttable (name, description, price, stock_available) values (?, ?, ?, ?);";
dbHandler.getWritableDatabase();
database.beginTransaction();
SQLiteStatement stmt = database.compileStatement(sql);
for (int i = 0; i < NUMBER_OF_ROWS; i++) {
//generate some values
stmt.bindString(1, randomName);
stmt.bindString(2, randomDescription);
stmt.bindDouble(3, randomPrice);
stmt.bindLong(4, randomNumber);
long entryID = stmt.executeInsert();
stmt.clearBindings();
}
database.setTransactionSuccessful();
database.endTransaction();
dbHandler.close();
}
Compiling the sql insert statement helps speed things up. It can also require more effort to shore everything up and prevent possible injection since it's now all on your shoulders.
Another approach which can also speed things up is the under-documented android.database.DatabaseUtils.InsertHelper class. My understanding is that it actually wraps compiled insert statements. Going from non-compiled transacted inserts to compiled transacted inserts was about a 3x gain in speed (2ms per insert to .6ms per insert) for my large (200K+ entries) but simple SQLite inserts.
Sample code:
SQLiteDatabse db = getWriteableDatabase();
//use the db you would normally use for db.insert, and the "table_name"
//is the same one you would use in db.insert()
InsertHelper iHelp = new InsertHelper(db, "table_name");
//Get the indices you need to bind data to
//Similar to Cursor.getColumnIndex("col_name");
int first_index = iHelp.getColumnIndex("first");
int last_index = iHelp.getColumnIndex("last");
try
{
db.beginTransaction();
for(int i=0 ; i<num_things ; ++i)
{
//need to tell the helper you are inserting (rather than replacing)
iHelp.prepareForInsert();
//do the equivalent of ContentValues.put("field","value") here
iHelp.bind(first_index, thing_1);
iHelp.bind(last_index, thing_2);
//the db.insert() equilvalent
iHelp.execute();
}
db.setTransactionSuccessful();
}
finally
{
db.endTransaction();
}
db.close();
If the table has an index on it, consider dropping it prior to inserting the records and then adding it back after you've commited your records.
If using a ContentProvider:
#Override
public int bulkInsert(Uri uri, ContentValues[] bulkinsertvalues) {
int QueryType = sUriMatcher.match(uri);
int returnValue=0;
SQLiteDatabase db = mOpenHelper.getWritableDatabase();
switch (QueryType) {
case SOME_URI_IM_LOOKING_FOR: //replace this with your real URI
db.beginTransaction();
for (int i = 0; i < bulkinsertvalues.length; i++) {
//get an individual result from the array of ContentValues
ContentValues values = bulkinsertvalues[i];
//insert this record into the local SQLite database using a private function you create, "insertIndividualRecord" (replace with a better function name)
insertIndividualRecord(uri, values);
}
db.setTransactionSuccessful();
db.endTransaction();
break;
default:
throw new IllegalArgumentException("Unknown URI " + uri);
}
return returnValue;
}
Then the private function to perform the insert (still inside your content provider):
private Uri insertIndividualRecord(Uri uri, ContentValues values){
//see content provider documentation if this is confusing
if (sUriMatcher.match(uri) != THE_CONSTANT_IM_LOOKING_FOR) {
throw new IllegalArgumentException("Unknown URI " + uri);
}
//example validation if you have a field called "name" in your database
if (values.containsKey(YOUR_CONSTANT_FOR_NAME) == false) {
values.put(YOUR_CONSTANT_FOR_NAME, "");
}
//******add all your other validations
//**********
//time to insert records into your local SQLite database
SQLiteDatabase db = mOpenHelper.getWritableDatabase();
long rowId = db.insert(YOUR_TABLE_NAME, null, values);
if (rowId > 0) {
Uri myUri = ContentUris.withAppendedId(MY_INSERT_URI, rowId);
getContext().getContentResolver().notifyChange(myUri, null);
return myUri;
}
throw new SQLException("Failed to insert row into " + uri);
}