Develop Reference

the best database for storing a large JSON

I have a large file of json after the parsing using JACKSON, I must store it into a database, it takes a several minutes so my questions
1-there's any way to speed up the storage ?
2- there's another database ?
3- Using Object database could help me ?,
I use SQLITE Database, I heared about RealM but I'm not sure to use it, help me guys

If you're storing directly the json inside sqlite you're probably doing something "wrong".
A database is useful when you need RELATIONSHIPs beetween objects, and so you have tables, indexes, keys and stuff like that.
Anyway, it's really strange that it takes minutes, but I suggest you to rethink your architecture, and maybe just write it on a file.

Changes in SQLite are ACID (atomic, consistent, isolated, durable). This means that every update, insert and delete operation is ACID. Unfortunately this requires some overhead in the database processing therefore you should wrap updates in the SQLite database in an transaction and commit this transaction after several operations. This can significantly improve performance.
The following code demonstrates that performance optimization.
db.beginTransaction();
try {
for (int i= 0; i< values.lenght; i++){
// TODO prepare ContentValues object values
db.insert(your_table, null, values);
// In case you do larger updates
yieldIfContededSafely()
}
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
REfs:http://www.vogella.com/tutorials/AndroidSQLite/article.html

Update SQLite table of Android application optimally

I am working on an android application using android:minSdkVersion="14". The application receives data as JSON from a server. The data received need to be added to an sqlite table. If a row exists, all fields except for two have to be updated. If a row does not already exist in the table, it has to be inserted. I am looking for the most efficient way as regards performance.
The function insertwithonCoflict() has been considered but it is not an option since in case of update, it updates all the fields including the two that should not be updated.
The function replace() is also not suitable.
I would opt for a SELECT to check if the row exists and then an INSERT or UPDATE but I was wondering if I could optimize the procedure somehow .

Two approaches:
Change the database structure so that the table has only the server data. Put local data (the two columns) in another table that references the server data table. When updating, just insert to the server data table with "replace" conflict resolution.
Do the select-insert/update logic.
For performance in any case, use database transactions to reduce I/O. That is, wrap the database update loop in a transaction and only commit it when you've done with everything. (In case the transaction becomes too large, split the loop into transaction chunks of maybe a few thousand rows.)

A nice solution I use is as follows:
long id = db.insertWithOnConflict(TABLE, null, contentValues, SQLiteDatabase.CONFLICT_IGNORE);
if(id!=-1) db.update(TABLE, contentValues, "_id=?", new String[]{String.valueOf(id)});
This ensures the row exists and has the latest values.

Use of Direct Queries in SQLite?

I am currently studying SQLite and I have found that it uses various classes like ContentValues for insertion and updation... I was wondering whether I have to follow the given way or can I write a normal SQL query and use db.execSQL() method to execute them?
Will it bring any inconsistency to my database because with these all "extra" steps doesnt it stop the flow of the query and I feel it would be faster if we use a query directly.

You can do any SQL command you want with db.execSQL except select command or any other SQL command that return data (you use db.rawQuery() for this). The classes used are helper classes that make it easy for you to manipulate DBs (try inserting 100 rows with 20 columns each using ContentValues and db.execSQL and you will get the point). For small tables it will not differ much (and you will not cause inconsistecies), however, for large tables with inputs that depend on user interface or use calculations, it might be useful to have a class like ContentValues with its helper methods.

Yes you can definitely use this way like using
myDB.execSQL("INSERT INTO MyTable VALUES ('fffff', 'numb', 20)");
to insert values but only when you are using database for small queries.
Also there are some flaws using direct methods which gets removed using ContentValues
For example,try to insert a blob into the database using this method ,you will get a null bitmap while converting the retrieved data to bitmap.But when you insert using ContentValues,you will get the correct data i.e you will be able to convert that into Bitmap.

Converting server data as Sqlite Database

i am developing android app, here i am having an huge no of data approximately 10000 records with 10 fields in the server, i need to get this data and store it in the my local db, so for this i tried to implement by getting the data in the form of json parsing it and inserting in db one by one, it is taking less time to download the data but more time to insert to the db, after some time i get to know that i am inserting to the db one by one, so insertion operations looping based on the total no of records which had been got. i tried to look for the alternatives i could not get the way for this, so i request you to give me suggestions and snippets to me achieve this.
Thanking you

use transactions to wrap multiple inserts into one operation, that's a lot faster: Improve INSERT-per-second performance of SQLite?
List<Item> list = getDataFromJson();
SQLiteDatabase db = getDatabase();
db.beginTransaction();
try {
// doing all the inserts (into memory) here
for(Item item : list) {
db.insert(table, null, item.getContentValues());
}
// nothing was actually inserted yet
db.setTransactionSuccessful();
} finally {
// all inserts happen now (if transaction was set to successful)
db.endTransaction();
}

Bulk Insertion on Android device

I want to bulk insert about 700 records into the Android database on my next upgrade. What's the most efficient way to do this? From various posts, I know that if I use Insert statements, I should wrap them in a transaction. There's also a post about using your own database, but I need this data to go into my app's standard Android database. Note that this would only be done once per device.
Some ideas:
Put a bunch of SQL statements in a file, read them in a line at a time, and exec the SQL.
Put the data in a CSV file, or JSON, or YAML, or XML, or whatever. Read a line at a time and do db.insert().
Figure out how to do an import and do a single import of the entire file.
Make a sqlite database containing all the records, copy that onto the Android device, and somehow merge the two databases.
[EDIT] Put all the SQL statements in a single file in res/values as one big string. Then read them a line at a time and exec the SQL.
What's the best way? Are there other ways to load data? Are 3 and 4 even possible?

Normally, each time db.insert() is used, SQLite creates a transaction (and resulting journal file in the filesystem), which slows things down.
If you use db.beginTransaction() and db.endTransaction() SQLite creates only a single journal file on the filesystem and then commits all the inserts at the same time, dramatically speeding things up.
Here is some pseudo code from: Batch insert to SQLite database on Android
try
{
db.beginTransaction();
for each record in the list
{
do_some_processing();
if (line represent a valid entry)
{
db.insert(SOME_TABLE, null, SOME_VALUE);
}
some_other_processing();
}
db.setTransactionSuccessful();
}
catch (SQLException e) {}
finally
{
db.endTransaction();
}
If you wish to abort a transaction due to an unexpected error or something, simply db.endTransaction() without first setting the transaction as successful (db.setTransactionSuccessful()).
Another useful method is to use db.inTransaction() (returns true or false) to determine if you are currently in the middle of a transaction.
Documentation here

I've found that for bulk insertions, the (apparently little-used) DatabaseUtils.InsertHelper class is several times faster than using SQLiteDatabase.insert.
Two other optimizations also helped with my app's performance, though they may not be appropriate in all cases:
Don't bind values that are empty or null.
If you can be certain that it's safe to do it, temporarily turning off the database's internal locking can also help performance.
I have a blog post with more details.

This example below will work perfectly
String sql = "INSERT INTO " + DatabaseHelper.TABLE_PRODUCT_LIST
+ " VALUES (?,?,?,?,?,?,?,?,?);";
SQLiteDatabase db = this.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
db.beginTransaction();
for(int idx=0; idx < Produc_List.size(); idx++) {
statement.clearBindings();
statement.bindLong(1, Produc_List.get(idx).getProduct_id());
statement.bindLong(2, Produc_List.get(idx).getCategory_id());
statement.bindString(3, Produc_List.get(idx).getName());
// statement.bindString(4, Produc_List.get(idx).getBrand());
statement.bindString(5, Produc_List.get(idx).getPrice());
//statement.bindString(6, Produc_List.get(idx).getDiscPrice());
statement.bindString(7, Produc_List.get(idx).getImage());
statement.bindLong(8, Produc_List.get(idx).getLanguage_id());
statement.bindLong(9, Produc_List.get(idx).getPl_rank());
statement.execute();
}
db.setTransactionSuccessful();
db.endTransaction();

Well, my solution for this it kind of weird but works fine...
I compile a large sum of data and insert it in one go (bulk insert?)
I use the db.execSQL(Query) command and I build the "Query" with the following statement...
INSERT INTO yourtable SELECT * FROM (
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
.
.
.
SELECT 'data1','data2'....
)
The only problem is the building of the query which can be kind of messy.
I hope it helps

I don't believe there is any feasible way to accomplish #3 or #4 on your list.
Of the other solutions you list two that have the datafile contain direct SQL, and the other has the data in a non-SQL format.
All three would work just fine, but the latter suggestion of grabbing the data from a formatted file and building the SQL yourself seems the cleanest. If true batch update capability is added at a later date your datafile is still usable, or at least easily processable into a usable form. Also, creation of the datafile is more straightforward and less error prone. Finally, having the "raw" data would allow import into other data-store formats.
In any case, you should (as you mentioned) wrap the groups of inserts into transactions to avoid the per-row transaction journal creation.