Can we insert 1000 records in one go in Sqlite db in android. I know there is an approach of putting the loop and then calling the insert for 1000 times ?? Is there any other approach for this which is good in performance. And if this is the only approach, how much my performance would be impacted by this??
database.execSQL("PRAGMA synchronous=OFF");
Will enhance the speed database operations.
But it is risky in some conditions.
You should use transactions for such scenarios.
Pseudocode:
db.beginTransaction();
for (entry : listOfEntries) {
db.insert(entry);
}
db.setTransactionSuccessful();
db.endTransaction();
Check this post.
It has example of inserting values to database with and without transaction.
Also it has time measurment for this approaches(In this post difference is more than 100 times while inserting 100 records).
Related
I have a large file of json after the parsing using JACKSON, I must store it into a database, it takes a several minutes so my questions
1-there's any way to speed up the storage ?
2- there's another database ?
3- Using Object database could help me ?,
I use SQLITE Database, I heared about RealM but I'm not sure to use it, help me guys
If you're storing directly the json inside sqlite you're probably doing something "wrong".
A database is useful when you need RELATIONSHIPs beetween objects, and so you have tables, indexes, keys and stuff like that.
Anyway, it's really strange that it takes minutes, but I suggest you to rethink your architecture, and maybe just write it on a file.
Changes in SQLite are ACID (atomic, consistent, isolated, durable). This means that every update, insert and delete operation is ACID. Unfortunately this requires some overhead in the database processing therefore you should wrap updates in the SQLite database in an transaction and commit this transaction after several operations. This can significantly improve performance.
The following code demonstrates that performance optimization.
db.beginTransaction();
try {
for (int i= 0; i< values.lenght; i++){
// TODO prepare ContentValues object values
db.insert(your_table, null, values);
// In case you do larger updates
yieldIfContededSafely()
}
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
REfs:http://www.vogella.com/tutorials/AndroidSQLite/article.html
I use this function to insert data into the SQLite Android data base:
public long insertAccount(String code,String name,int s3,int s4,String s5,String s6,int s7,
int s8,int s9,int s10,int s11,String s12,String s13,int s14,int s15,int s16) {
//container and place in it the information you want inserted, updated, etc.
ContentValues initialValues = new ContentValues();
initialValues.put(Code, code);
initialValues.put(Name,name);
initialValues.put(Type, s3);
initialValues.put(Level1, s4);
initialValues.put(Father, s5);
initialValues.put(ACCCurr,s6);
initialValues.put(AccNat, s7);
initialValues.put(LowLevel, s8);
initialValues.put(DefNum, s9);
initialValues.put(AccClass, s10);
initialValues.put(SubClass, s11);
initialValues.put(SSClass1, s12);
initialValues.put(SSClass2, s13);
initialValues.put(Stype1, s14);
initialValues.put(Stype2, s15);
initialValues.put(Stype3, s16);
return db.insert(DATABASE_TABLE, null, initialValues);
}
But this takes much time when inserting about 70,000+ rows! How can I accelerate the process of insertion into the data base, and after the insert is done, how can I apply Update on it?
Some options:
Prepopulate your database. See "Ship an application with a database"
Use transactions to reduce the time waiting for I/O. See e.g. "Android SQLite database: slow insertion". Likely you cannot wrap all 70k rows in a single transaction but something like 100..1000 inserts per transaction should be doable, cutting the cumulative I/O wait time by orders of magnitude.
Inserting into SQLlite android using PHP? how is it possible using php in android phone, I am sorry I didn't got this.
Anyways I believe you have written the java code up here and you have like 7k+ records that you want to insert in your db.
The style of inserting a bulk of records in any db is called "Bulk Inserts", the idea is to create as less number of transactions as possible and rather do all the inserts in one shot; In case of relational db's like sql server and oracle its done by specific api's as well, but in sqllite the plain old idea is to make a single transaction with a bunch of data
check out this article which uses the same technique http://www.techrepublic.com/blog/software-engineer/turbocharge-your-sqlite-inserts-on-android/ and also explains it quite well.
You have to use transaction to done insertion in 1 time. you can use this:
//before insertion
db.beginTransaction();
//====do insertion
//after insertion
db.setTransactionSuccessful()
db.endTransaction();
I have been spending quite some time looking at some performance issues on our device, and noticed that we have quite a few apps all doing db reads/writes..
I started by using the Contacts API to insert new contacts & data rows, and it was painfully slow. 1 minute 18 seconds to insert about 1500 rows (250 raw contacts & 1250 data rows)..
I had used the insert helper in another app for performance inserts, and decided to write a test app which would write to separate db's w/ separate insert methods.
Each db has one table, each w/ 4 columns :
_ID, Name, Time, and Blob (all of type 'string') - just like contact provider defines the data columns.
_ID is auto increment pk, Name just inserts the same thing '1234567890', time is just the current system time in milis, and BLob is a string w/ length 6400 full of the letter 'A'...
I first checked the bulk insert, but all it does is loops through all the inserts you have defined, and is just as slow as doing the inserts individually (or negligible performance impact)..
I tested 3 different methods to do the inserts :
ContentValues w/ db.insert method :
SQLiteStatement w/ statement.execute() (done inside a transaction).
SqliteInsertHelper w/ transaction.
I can provide some code, but I got the best performance out of the InsertHelper, and wondering why it was deprecated :
Time to insert 100 records
ContentValues : 7.778 seconds ( 82 bytes written / ms )
SQLiteStatement : 1.311 seconds ( 489 bytes written / ms )
SqliteInsertHElper : 0.292 seconds (2197 bytes written / ms)
Any ideas?
It's hard to come by any information on why InsertHelper was deprecated without going to the actual commit that does deprecate it. The engineer that deprecated InsertHelper gave the following reason:
This class does not offer any advantages over SQLiteStatement and just makes code more complex and error-prone.
After refactoring from InsertHelper to SQLiteStatement I agree. One exception are for null-safe binding functions. Whereas InsertHelper automatically calls bindNull() for you, SQLiteStatement crashes if you pass, for example, a null String and you have to do your own null check before calling bindString().
See:
https://android.googlesource.com/platform/frameworks/base/+/b33eb4e%5E!/
InsertHelper allows users to do multiple inserts into a table using the same statement.But it is not a good way to insert as such as it is not thread-safe.
You should use transactions.. If you don't explicitly create a transaction for a database operation the framework creates one for each. Group your object together and insert them all at once. This will greatly increase performance.
i am developing android app, here i am having an huge no of data approximately 10000 records with 10 fields in the server, i need to get this data and store it in the my local db, so for this i tried to implement by getting the data in the form of json parsing it and inserting in db one by one, it is taking less time to download the data but more time to insert to the db, after some time i get to know that i am inserting to the db one by one, so insertion operations looping based on the total no of records which had been got. i tried to look for the alternatives i could not get the way for this, so i request you to give me suggestions and snippets to me achieve this.
Thanking you
use transactions to wrap multiple inserts into one operation, that's a lot faster: Improve INSERT-per-second performance of SQLite?
List<Item> list = getDataFromJson();
SQLiteDatabase db = getDatabase();
db.beginTransaction();
try {
// doing all the inserts (into memory) here
for(Item item : list) {
db.insert(table, null, item.getContentValues());
}
// nothing was actually inserted yet
db.setTransactionSuccessful();
} finally {
// all inserts happen now (if transaction was set to successful)
db.endTransaction();
}
I want to bulk insert about 700 records into the Android database on my next upgrade. What's the most efficient way to do this? From various posts, I know that if I use Insert statements, I should wrap them in a transaction. There's also a post about using your own database, but I need this data to go into my app's standard Android database. Note that this would only be done once per device.
Some ideas:
Put a bunch of SQL statements in a file, read them in a line at a time, and exec the SQL.
Put the data in a CSV file, or JSON, or YAML, or XML, or whatever. Read a line at a time and do db.insert().
Figure out how to do an import and do a single import of the entire file.
Make a sqlite database containing all the records, copy that onto the Android device, and somehow merge the two databases.
[EDIT] Put all the SQL statements in a single file in res/values as one big string. Then read them a line at a time and exec the SQL.
What's the best way? Are there other ways to load data? Are 3 and 4 even possible?
Normally, each time db.insert() is used, SQLite creates a transaction (and resulting journal file in the filesystem), which slows things down.
If you use db.beginTransaction() and db.endTransaction() SQLite creates only a single journal file on the filesystem and then commits all the inserts at the same time, dramatically speeding things up.
Here is some pseudo code from: Batch insert to SQLite database on Android
try
{
db.beginTransaction();
for each record in the list
{
do_some_processing();
if (line represent a valid entry)
{
db.insert(SOME_TABLE, null, SOME_VALUE);
}
some_other_processing();
}
db.setTransactionSuccessful();
}
catch (SQLException e) {}
finally
{
db.endTransaction();
}
If you wish to abort a transaction due to an unexpected error or something, simply db.endTransaction() without first setting the transaction as successful (db.setTransactionSuccessful()).
Another useful method is to use db.inTransaction() (returns true or false) to determine if you are currently in the middle of a transaction.
Documentation here
I've found that for bulk insertions, the (apparently little-used) DatabaseUtils.InsertHelper class is several times faster than using SQLiteDatabase.insert.
Two other optimizations also helped with my app's performance, though they may not be appropriate in all cases:
Don't bind values that are empty or null.
If you can be certain that it's safe to do it, temporarily turning off the database's internal locking can also help performance.
I have a blog post with more details.
This example below will work perfectly
String sql = "INSERT INTO " + DatabaseHelper.TABLE_PRODUCT_LIST
+ " VALUES (?,?,?,?,?,?,?,?,?);";
SQLiteDatabase db = this.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
db.beginTransaction();
for(int idx=0; idx < Produc_List.size(); idx++) {
statement.clearBindings();
statement.bindLong(1, Produc_List.get(idx).getProduct_id());
statement.bindLong(2, Produc_List.get(idx).getCategory_id());
statement.bindString(3, Produc_List.get(idx).getName());
// statement.bindString(4, Produc_List.get(idx).getBrand());
statement.bindString(5, Produc_List.get(idx).getPrice());
//statement.bindString(6, Produc_List.get(idx).getDiscPrice());
statement.bindString(7, Produc_List.get(idx).getImage());
statement.bindLong(8, Produc_List.get(idx).getLanguage_id());
statement.bindLong(9, Produc_List.get(idx).getPl_rank());
statement.execute();
}
db.setTransactionSuccessful();
db.endTransaction();
Well, my solution for this it kind of weird but works fine...
I compile a large sum of data and insert it in one go (bulk insert?)
I use the db.execSQL(Query) command and I build the "Query" with the following statement...
INSERT INTO yourtable SELECT * FROM (
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
.
.
.
SELECT 'data1','data2'....
)
The only problem is the building of the query which can be kind of messy.
I hope it helps
I don't believe there is any feasible way to accomplish #3 or #4 on your list.
Of the other solutions you list two that have the datafile contain direct SQL, and the other has the data in a non-SQL format.
All three would work just fine, but the latter suggestion of grabbing the data from a formatted file and building the SQL yourself seems the cleanest. If true batch update capability is added at a later date your datafile is still usable, or at least easily processable into a usable form. Also, creation of the datafile is more straightforward and less error prone. Finally, having the "raw" data would allow import into other data-store formats.
In any case, you should (as you mentioned) wrap the groups of inserts into transactions to avoid the per-row transaction journal creation.