Develop Reference

Update SQLite table of Android application optimally

I am working on an android application using android:minSdkVersion="14". The application receives data as JSON from a server. The data received need to be added to an sqlite table. If a row exists, all fields except for two have to be updated. If a row does not already exist in the table, it has to be inserted. I am looking for the most efficient way as regards performance.
The function insertwithonCoflict() has been considered but it is not an option since in case of update, it updates all the fields including the two that should not be updated.
The function replace() is also not suitable.
I would opt for a SELECT to check if the row exists and then an INSERT or UPDATE but I was wondering if I could optimize the procedure somehow .

Two approaches:
Change the database structure so that the table has only the server data. Put local data (the two columns) in another table that references the server data table. When updating, just insert to the server data table with "replace" conflict resolution.
Do the select-insert/update logic.
For performance in any case, use database transactions to reduce I/O. That is, wrap the database update loop in a transaction and only commit it when you've done with everything. (In case the transaction becomes too large, split the loop into transaction chunks of maybe a few thousand rows.)

A nice solution I use is as follows:
long id = db.insertWithOnConflict(TABLE, null, contentValues, SQLiteDatabase.CONFLICT_IGNORE);
if(id!=-1) db.update(TABLE, contentValues, "_id=?", new String[]{String.valueOf(id)});
This ensures the row exists and has the latest values.

Accelerate the insertion into Sqlite datatbase Android?

I use this function to insert data into the SQLite Android data base:
public long insertAccount(String code,String name,int s3,int s4,String s5,String s6,int s7,
int s8,int s9,int s10,int s11,String s12,String s13,int s14,int s15,int s16) {
//container and place in it the information you want inserted, updated, etc.
ContentValues initialValues = new ContentValues();
initialValues.put(Code, code);
initialValues.put(Name,name);
initialValues.put(Type, s3);
initialValues.put(Level1, s4);
initialValues.put(Father, s5);
initialValues.put(ACCCurr,s6);
initialValues.put(AccNat, s7);
initialValues.put(LowLevel, s8);
initialValues.put(DefNum, s9);
initialValues.put(AccClass, s10);
initialValues.put(SubClass, s11);
initialValues.put(SSClass1, s12);
initialValues.put(SSClass2, s13);
initialValues.put(Stype1, s14);
initialValues.put(Stype2, s15);
initialValues.put(Stype3, s16);
return db.insert(DATABASE_TABLE, null, initialValues);
}
But this takes much time when inserting about 70,000+ rows! How can I accelerate the process of insertion into the data base, and after the insert is done, how can I apply Update on it?

Some options:
Prepopulate your database. See "Ship an application with a database"
Use transactions to reduce the time waiting for I/O. See e.g. "Android SQLite database: slow insertion". Likely you cannot wrap all 70k rows in a single transaction but something like 100..1000 inserts per transaction should be doable, cutting the cumulative I/O wait time by orders of magnitude.

Inserting into SQLlite android using PHP? how is it possible using php in android phone, I am sorry I didn't got this.
Anyways I believe you have written the java code up here and you have like 7k+ records that you want to insert in your db.
The style of inserting a bulk of records in any db is called "Bulk Inserts", the idea is to create as less number of transactions as possible and rather do all the inserts in one shot; In case of relational db's like sql server and oracle its done by specific api's as well, but in sqllite the plain old idea is to make a single transaction with a bunch of data
check out this article which uses the same technique http://www.techrepublic.com/blog/software-engineer/turbocharge-your-sqlite-inserts-on-android/ and also explains it quite well.

You have to use transaction to done insertion in 1 time. you can use this:
//before insertion
db.beginTransaction();
//====do insertion
//after insertion
db.setTransactionSuccessful()
db.endTransaction();

Populate an SQLiteDatabase in Android with static data

I have a database with multiple tables. One of these tables (sport) is where i have to put a static list of object, each one with an _id, name, logo and an int. The _id will be used by other tables to do some queries (eg. select from "table X" where sport_id = _id), so it shouldn't change overtime (is there a way to update all the reference to this _id if it will change?).
Where should i put the code (i think it will be a simple list of db.insertSport()) to make it add this row only one time (and check if the row number grow, to add the new ones)?
There won't be much row, 50 at the best.

I think I would make a method in the dbHelper to insert that data, then call that method immediately upon app start. I'm making a couple of assumptions here... first that you are shipping this static info with the app and when you want to add more info you will be shipping a new version.
You could store the data as a text file in your assets folder and then read the file in execute a batch insert in the method.
If you set it up right (use insertWithOnConflict and the CONFLICT_IGNORE flag in the method) it will only add the new rows (if any) each time so you can run it every time the app starts and not worry about duplicate data or crashes for constraint violations.
If you only want it to run the once and then again when there is additional info, put a version number in the text file and check that against the previous one (which you can store in SharedPreferences).
EDIT
Example of using insertWithOnConflict:
public long createItem(String yourdata) {
ContentValues initialValues = new ContentValues();
initialValues.put(YOUR_COLUMN, yourdata);
return mDb.insertWithOnConflict(YOUR_TABLE, null, initialValues,
SQLiteDatabase.CONFLICT_IGNORE);
}
You can read up on the SQLiteDatabase class (which has the constants and methods) here

How to speed up SQLite access on Android

I'm doing both an update and an insert to a review_schedule table. It seems as if the time for the both takes between 1 and 2 seconds, which is a bit slow. I've tried it with and without indexes set on the id field and the date_review field. Note that I'm using a precompiled statement for the insert, but not the update, because the compiled statement for update apparently isn't supported in Gingerbread. Here's the code:
public long insert(int id, String lastReviewDate, String nextReviewDate) {
this.insertStmt.bindLong(1, id);
this.insertStmt.bindString(2, lastReviewDate);
this.insertStmt.bindString(3, nextReviewDate);
return this.insertStmt.executeInsert();
}
public void updateSRSInfo(int id, String lastReviewDate,
String nextReviewDate) {
ContentValues contentValues = new ContentValues();
contentValues.put("last_review_date", lastReviewDate);
contentValues.put("next_review_date", nextReviewDate);
this.myDataBase.update(DataBaseHelper.WORD_REVIEW_SCHEDULE,
contentValues, "_id=?", new String[] { Integer.toString(id) });
}
Any suggestions appreciated.

If the length of the procedure making the App unresponsive, maybe you can move the operation into a different thread.

At least two options:
1) Read uncommitted transactions, since by default SQLite reads only committed transactions, which is slow. But you have to understands all risks which you migth have reading uncommitted records. Changing transaction isolation level can be done in this way:
database.execSQL("PRAGMA read_uncommitted = true;");
2) Use tables indexing

As you already have got an id,why not combine it with ContentUri.withAppendedId,then you can access this record directly.
i'm not sure whether this run faster or not as i haven't test this.

Bulk Insertion on Android device

I want to bulk insert about 700 records into the Android database on my next upgrade. What's the most efficient way to do this? From various posts, I know that if I use Insert statements, I should wrap them in a transaction. There's also a post about using your own database, but I need this data to go into my app's standard Android database. Note that this would only be done once per device.
Some ideas:
Put a bunch of SQL statements in a file, read them in a line at a time, and exec the SQL.
Put the data in a CSV file, or JSON, or YAML, or XML, or whatever. Read a line at a time and do db.insert().
Figure out how to do an import and do a single import of the entire file.
Make a sqlite database containing all the records, copy that onto the Android device, and somehow merge the two databases.
[EDIT] Put all the SQL statements in a single file in res/values as one big string. Then read them a line at a time and exec the SQL.
What's the best way? Are there other ways to load data? Are 3 and 4 even possible?

Normally, each time db.insert() is used, SQLite creates a transaction (and resulting journal file in the filesystem), which slows things down.
If you use db.beginTransaction() and db.endTransaction() SQLite creates only a single journal file on the filesystem and then commits all the inserts at the same time, dramatically speeding things up.
Here is some pseudo code from: Batch insert to SQLite database on Android
try
{
db.beginTransaction();
for each record in the list
{
do_some_processing();
if (line represent a valid entry)
{
db.insert(SOME_TABLE, null, SOME_VALUE);
}
some_other_processing();
}
db.setTransactionSuccessful();
}
catch (SQLException e) {}
finally
{
db.endTransaction();
}
If you wish to abort a transaction due to an unexpected error or something, simply db.endTransaction() without first setting the transaction as successful (db.setTransactionSuccessful()).
Another useful method is to use db.inTransaction() (returns true or false) to determine if you are currently in the middle of a transaction.
Documentation here

I've found that for bulk insertions, the (apparently little-used) DatabaseUtils.InsertHelper class is several times faster than using SQLiteDatabase.insert.
Two other optimizations also helped with my app's performance, though they may not be appropriate in all cases:
Don't bind values that are empty or null.
If you can be certain that it's safe to do it, temporarily turning off the database's internal locking can also help performance.
I have a blog post with more details.

This example below will work perfectly
String sql = "INSERT INTO " + DatabaseHelper.TABLE_PRODUCT_LIST
+ " VALUES (?,?,?,?,?,?,?,?,?);";
SQLiteDatabase db = this.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
db.beginTransaction();
for(int idx=0; idx < Produc_List.size(); idx++) {
statement.clearBindings();
statement.bindLong(1, Produc_List.get(idx).getProduct_id());
statement.bindLong(2, Produc_List.get(idx).getCategory_id());
statement.bindString(3, Produc_List.get(idx).getName());
// statement.bindString(4, Produc_List.get(idx).getBrand());
statement.bindString(5, Produc_List.get(idx).getPrice());
//statement.bindString(6, Produc_List.get(idx).getDiscPrice());
statement.bindString(7, Produc_List.get(idx).getImage());
statement.bindLong(8, Produc_List.get(idx).getLanguage_id());
statement.bindLong(9, Produc_List.get(idx).getPl_rank());
statement.execute();
}
db.setTransactionSuccessful();
db.endTransaction();

Well, my solution for this it kind of weird but works fine...
I compile a large sum of data and insert it in one go (bulk insert?)
I use the db.execSQL(Query) command and I build the "Query" with the following statement...
INSERT INTO yourtable SELECT * FROM (
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
.
.
.
SELECT 'data1','data2'....
)
The only problem is the building of the query which can be kind of messy.
I hope it helps

I don't believe there is any feasible way to accomplish #3 or #4 on your list.
Of the other solutions you list two that have the datafile contain direct SQL, and the other has the data in a non-SQL format.
All three would work just fine, but the latter suggestion of grabbing the data from a formatted file and building the SQL yourself seems the cleanest. If true batch update capability is added at a later date your datafile is still usable, or at least easily processable into a usable form. Also, creation of the datafile is more straightforward and less error prone. Finally, having the "raw" data would allow import into other data-store formats.
In any case, you should (as you mentioned) wrap the groups of inserts into transactions to avoid the per-row transaction journal creation.