Working on database optimization, we split our database into two databases: db and db2. A low priority background thread is inserting into db2. Some of the queries on db are joined with db2, so we need to attach db2 to db. We enable WAL because want it all to be multithreaded.
SQLiteDatabase db = SQLiteDatabase.openDatabase(dbPath, ...);
db.enableWriteAheadLogging();
db.execSQL("attach " + db2path + " as db2");
To understand the problem, we run a simple two thread test. The first thread is inserting rows into db, and the second thread is selecting from db. Each thread prints the time delta from the previous loop and the time we were inside the database.
thread 1 loop: | thread 2 loop:
t1 = getTime() | t1 = getTime()
db2.execSQL("insert into ...."); | db2.execSQL("select ....");
t2 = t3 | t2 = t3
t3 = getTime() | t3 = getTime()
log("i: "+(t3-t1)+", delta: "+(t2-t1)) | log("s: "+(t3-t1)+", delta: "+(t2-t1))
What we see is that the selecting thread is blocking the inserting thread. This can be emphasized by doing a huge (and slow) select, and a tiny insert. You will see that the insert time and the delta increase approximately to the time of the select. If we don't run the slow threads, the insert thread speeds up considerably.
Digging into the source code of SQLiteDatabase I found the following lines in SQLiteDatabase#enableWriteAheadLogging():
// make sure this database has NO attached databases because sqlite's write-ahead-logging
// doesn't work for databases with attached databases
if (mHasAttachedDbsLocked) {
if (Log.isLoggable(TAG, Log.DEBUG)) {
Log.d(TAG, "this database: " + mConfigurationLocked.label
+ " has attached databases. can't enable WAL.");
}
return false;
}
Now to my questions:
What is the meaning of the comment? What exactly doesn't work? Is it some old code left behind? The documentation of ATTACH DATABASE (https://www.sqlite.org/lang_attach.html) explicitly indicates that ATTACH + WAL is OK (with a small caveat.)
Why is the Android binding code trying to protect us from SQLite internal issues? The way I see it, it's supposed to be a thin interface layer.
Edit: I reported this as a bug in AOSP issue tracker. Will update if an answer appears there.
WAL allows readers and a writer at the same time, but only from different connections. You should never use the same connection (the SQLiteDatabase object) from multiple threads.
The WAL setting is permanent; you do not need to execute it every time after opening the database.
The meaning of the comment is exactly what it says. (Nobody guarantees that this comment is correct.)
Sometimes, the Android framework tries to be clever. But you can just execute PRAGMA journal_mode = WAL manually.
Related
I am developing an android application in which I need to download an JSON string and save it in SQlite database in a specific format (In my perspective, I have no other option to choose any other data-storage). And this is my table-structure:
problem_table(pid INTEGER PRIMARY KEY,
num TEXT, title TEXT,
dacu INTEGER,
verdict_series TEXT)
And at launch I need almost 4200 rows to be entered into the database table. I am working on emulator and when I launch the app, it works perfectly. But the app seemed to be freeze for a while after database manipulation is begin. Eventually the app manages to insert all the row but take pretty much time. Even at a point it shows the following look:
So how can I reduce the time-memory complexity or how can I do this in more optimized way or avoid this temporary failure?
N.S. : I didn't check it in any real device yet for lack of my scope. My emulator is using 512 RAM and 48 heap size.
Don't do your database manipulations in UI thread but in an AsyncTask, Thread, Service or whatever, but not in the UI Thread.
I solved it by #Jakobud answer given here
Answer:
Normally, each time db.insert() is used, SQLite creates a transaction (and resulting journal file in the filesystem). If you use db.beginTransaction() and db.endTransaction() SQLite commits all the inserts at the same time, dramatically speeding things up.
Here is some pseudo code from: Batch insert to SQLite database on Android
try
{
db.beginTransaction();
for each record in the list
{
do_some_processing();
if (line represent a valid entry)
{
db.insert(SOME_TABLE, null, SOME_VALUE);
}
some_other_processing();
}
db.setTransactionSuccessful();
}
catch (SQLException e) {}
finally
{
db.endTransaction();
}
I have been spending quite some time looking at some performance issues on our device, and noticed that we have quite a few apps all doing db reads/writes..
I started by using the Contacts API to insert new contacts & data rows, and it was painfully slow. 1 minute 18 seconds to insert about 1500 rows (250 raw contacts & 1250 data rows)..
I had used the insert helper in another app for performance inserts, and decided to write a test app which would write to separate db's w/ separate insert methods.
Each db has one table, each w/ 4 columns :
_ID, Name, Time, and Blob (all of type 'string') - just like contact provider defines the data columns.
_ID is auto increment pk, Name just inserts the same thing '1234567890', time is just the current system time in milis, and BLob is a string w/ length 6400 full of the letter 'A'...
I first checked the bulk insert, but all it does is loops through all the inserts you have defined, and is just as slow as doing the inserts individually (or negligible performance impact)..
I tested 3 different methods to do the inserts :
ContentValues w/ db.insert method :
SQLiteStatement w/ statement.execute() (done inside a transaction).
SqliteInsertHelper w/ transaction.
I can provide some code, but I got the best performance out of the InsertHelper, and wondering why it was deprecated :
Time to insert 100 records
ContentValues : 7.778 seconds ( 82 bytes written / ms )
SQLiteStatement : 1.311 seconds ( 489 bytes written / ms )
SqliteInsertHElper : 0.292 seconds (2197 bytes written / ms)
Any ideas?
It's hard to come by any information on why InsertHelper was deprecated without going to the actual commit that does deprecate it. The engineer that deprecated InsertHelper gave the following reason:
This class does not offer any advantages over SQLiteStatement and just makes code more complex and error-prone.
After refactoring from InsertHelper to SQLiteStatement I agree. One exception are for null-safe binding functions. Whereas InsertHelper automatically calls bindNull() for you, SQLiteStatement crashes if you pass, for example, a null String and you have to do your own null check before calling bindString().
See:
https://android.googlesource.com/platform/frameworks/base/+/b33eb4e%5E!/
InsertHelper allows users to do multiple inserts into a table using the same statement.But it is not a good way to insert as such as it is not thread-safe.
You should use transactions.. If you don't explicitly create a transaction for a database operation the framework creates one for each. Group your object together and insert them all at once. This will greatly increase performance.
I'm writing on a small Android App with SQLite 3 support. ATM I'm implementing some DB-Functions and asking myself if it is better have one big joined query or if it is OK to use multiple queries in respect to App performance.
Let's say I have a 3 Tables "drivers", "driven_runs", "race_disciplines". Every driven_run has a driver and a race_discipline. Let's assume I want to get all race disciplines by a certain driver in a certain discipline.
Solution 1
I already coded a function getDriver(driverID) with a Driver-Object in return and a function getRaceDiscipline(disciplineID) with a Race Discipline in return. So I would just create a function
public ArrayList<DrivenRun> getDrivenRunsOnDiscipline(short driverID, short disciplineID) {
ArrayList<DrivenRun> drivenRuns = new ArrayList<DrivenRun>();
String sql = "SELECT * FROM " + DBHelper.TABLE_DRIVEN_RUNS + " WHERE "
+ DBHelper.DRIVEN_RUNS_COLUMN_DRIVER_ID + "=" + driverID + " AND "
+ DBHelper.DRIVEN_RUNS_COLUMN_RACE_DISCIPLINE_ID + "=" + disciplineID + ";";
Cursor result = rawQuery(sql);
if (result.moveToFirst()) {
do {
Driver driver = getDriver(driverID);
RaceDiscipline discipline = getRaceDiscipline(disciplineID);
DrivenRun run = new DrivenRun();
run.setDriver(driver);
run.setDiscipline(discipline);
run.setResult("WHATEVER");
drivenRuns.add(run);
} while(result.moveToNext());
}
return drivenRuns;
}
in this case there would be 3 queries executed on after another but the coding is much more simple.
Solution 2
I would create one big joined query like
String sql = "SELECT * FROM driven_runs CROSS JOIN drivers CROSS_JOIN race_disciplines WHERE driven_runs.driver_id=drivers.id AND driven_runs.race_discipline_id=race_disciplines.id"
Cursor result = rawQuery(sql);
and would manually create the Driver and DrivenRun Object.
This solution needs much more writing but only one query is executed (or does the DB executes 3 queries as well when joining 3 tables?)
Long story short, is it OK to go with solution 1 because in regards to performance there isn't much of a difference?
In general, go for the simpler code until there's a good performance reason not to. Given that this is SQLite anyway, I don't think there's likely to be much performance difference, since the overhead for queries is pretty low.
Premature optimization is the root of all evil.
You should use proper join syntax, then your query won't look so cumbersome:
SELECT THECOLUMNSYOUREALLYNEED
FROM driven_runs JOIN
drivers
on driven_runs.driver_id=drivers.id join
race_disciplines
on driven_runs.race_discipline_id=race_disciplines.id
where driver.id = YOURDRIVEIDHERE and race_discipline = YOURDISCIPLINEIDHERE
Also, only return the columns that you need. Second, insert the appropriate ids in the where clause. Your version is returning everything, which is totally unnecessary.
This is a pretty simple query and it does what SQL databases do best -- joining large tables together. Even with SQLite, you are probably better off letting the database do the work. It is, at the very least, going to save some round trip communication from the database layer back to the application layer.
In a more sophisticated environment, the database will take advantage of multiple processors, multiple disks, and intelligently cache results to further optimize query response.
I have to query three table, and display the data to my customerView.
My code is like this:
Log.v(TAG, System.CurrentTimeMillis())
int len = cursor.getCount();
Log.v(TAG, System.CurrentTimeMillis())
Product[] products = new Product[len];
int i = 0;
while(cursor.moveToNext()){
products[i] = new Product(cursor.getstring(0),.....);
}
Log.v(TAG, System.CurrentTimeMillis())
Sqlite query:
String sql = "SELECT T1.PRODUCT_ID, CODE, SHORT_DESCRIPTION, CATEGORY_CODE,
BRAND_CODE, FORM_CODE, DENOMINATOR, T1.PIECE_PRICE, T1.lowest_piece_price,
T2.sku_type, T1.master_sku " +
"FROM CUSTOMER_PROD_LIST_ITEMS T1 INNER JOIN PRODUCT T2 ON
T1.PRODUCT_ID = T2.ID INNER JOIN PRODUCT_UOMS ON T2.ID =
PRODUCT_UOMS.PRODUCT_ID"+
"WHERE T1.VALID = 1 AND PRODUCT_UOMS.VALID = 1 AND
CUSTOMER_PRODUCT_LIST_ID = " + customer_pdtlist_ID + "
ORDER BY T1.PRODUCT_ID ASC";
After my testing, if we have 1500rows in the cursor, we have to spend more than 30s to finish this line(cursor.getcount()) . If I delete this line, and use ArrayList to take place. i can find that we should spend more than 30s for Cursor.moveToNext().
So my question is why the first time cursor operation should take such long time? and how do we solve?
And this man have the same question Poor SQLite implementation? First time data access way too slow. but the answer is not working for me.
by the way, I find display same 1500rows in Iphone, just need amost 3s.
thanks in advance!!
This is an answer to why the first operation on your cursor is so slow. When a Cursor is backed by SQLite, Android uses the sqlite C library internally and creating a Cursor is analogous to creating a prepared statement in the C library. Creating a prepared statement is cheap and it does not perform any query. Taken from the C library's documentation:
sqlite3_prepare()
This routine converts SQL text into a prepared statement object and returns a pointer to that object. This interface requires a database connection pointer created by a prior call to sqlite3_open() and a text string containing the SQL statement to be prepared. This API does not actually evaluate the SQL statement. It merely prepares the SQL statement for evaluation.
When you call moveToNext() on the Cursor, that's when the query actually gets executed. moveToNext results in a call to the sqlite3_step() function in the C library. Again, taken from the documentation:
sqlite3_step()
This routine is used to evaluate a prepared statement that has been previously created by the sqlite3_prepare() interface. The statement is evaluated up to the point where the first row of results are available. To advance to the second row of results, invoke sqlite3_step() again. Continue invoking sqlite3_step() until the statement is complete. Statements that do not return results (ex: INSERT, UPDATE, or DELETE statements) run to completion on a single call to sqlite3_step().
So creating a Cursor is done lazily and the query is only evaluated when the cursor is first moved.
To find out why the query is taking such a long time, use EXPLAIN QUERY PLAN on your query and see where the bottleneck lies. Usually it's the lack of an appropriate index.
ok, guys, i have not been here for acouple days.And i found the solution that is you have to create index for your table which will improve the query speed. thanks all the same
I want to bulk insert about 700 records into the Android database on my next upgrade. What's the most efficient way to do this? From various posts, I know that if I use Insert statements, I should wrap them in a transaction. There's also a post about using your own database, but I need this data to go into my app's standard Android database. Note that this would only be done once per device.
Some ideas:
Put a bunch of SQL statements in a file, read them in a line at a time, and exec the SQL.
Put the data in a CSV file, or JSON, or YAML, or XML, or whatever. Read a line at a time and do db.insert().
Figure out how to do an import and do a single import of the entire file.
Make a sqlite database containing all the records, copy that onto the Android device, and somehow merge the two databases.
[EDIT] Put all the SQL statements in a single file in res/values as one big string. Then read them a line at a time and exec the SQL.
What's the best way? Are there other ways to load data? Are 3 and 4 even possible?
Normally, each time db.insert() is used, SQLite creates a transaction (and resulting journal file in the filesystem), which slows things down.
If you use db.beginTransaction() and db.endTransaction() SQLite creates only a single journal file on the filesystem and then commits all the inserts at the same time, dramatically speeding things up.
Here is some pseudo code from: Batch insert to SQLite database on Android
try
{
db.beginTransaction();
for each record in the list
{
do_some_processing();
if (line represent a valid entry)
{
db.insert(SOME_TABLE, null, SOME_VALUE);
}
some_other_processing();
}
db.setTransactionSuccessful();
}
catch (SQLException e) {}
finally
{
db.endTransaction();
}
If you wish to abort a transaction due to an unexpected error or something, simply db.endTransaction() without first setting the transaction as successful (db.setTransactionSuccessful()).
Another useful method is to use db.inTransaction() (returns true or false) to determine if you are currently in the middle of a transaction.
Documentation here
I've found that for bulk insertions, the (apparently little-used) DatabaseUtils.InsertHelper class is several times faster than using SQLiteDatabase.insert.
Two other optimizations also helped with my app's performance, though they may not be appropriate in all cases:
Don't bind values that are empty or null.
If you can be certain that it's safe to do it, temporarily turning off the database's internal locking can also help performance.
I have a blog post with more details.
This example below will work perfectly
String sql = "INSERT INTO " + DatabaseHelper.TABLE_PRODUCT_LIST
+ " VALUES (?,?,?,?,?,?,?,?,?);";
SQLiteDatabase db = this.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
db.beginTransaction();
for(int idx=0; idx < Produc_List.size(); idx++) {
statement.clearBindings();
statement.bindLong(1, Produc_List.get(idx).getProduct_id());
statement.bindLong(2, Produc_List.get(idx).getCategory_id());
statement.bindString(3, Produc_List.get(idx).getName());
// statement.bindString(4, Produc_List.get(idx).getBrand());
statement.bindString(5, Produc_List.get(idx).getPrice());
//statement.bindString(6, Produc_List.get(idx).getDiscPrice());
statement.bindString(7, Produc_List.get(idx).getImage());
statement.bindLong(8, Produc_List.get(idx).getLanguage_id());
statement.bindLong(9, Produc_List.get(idx).getPl_rank());
statement.execute();
}
db.setTransactionSuccessful();
db.endTransaction();
Well, my solution for this it kind of weird but works fine...
I compile a large sum of data and insert it in one go (bulk insert?)
I use the db.execSQL(Query) command and I build the "Query" with the following statement...
INSERT INTO yourtable SELECT * FROM (
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
SELECT 'data1','data2'.... UNION
.
.
.
SELECT 'data1','data2'....
)
The only problem is the building of the query which can be kind of messy.
I hope it helps
I don't believe there is any feasible way to accomplish #3 or #4 on your list.
Of the other solutions you list two that have the datafile contain direct SQL, and the other has the data in a non-SQL format.
All three would work just fine, but the latter suggestion of grabbing the data from a formatted file and building the SQL yourself seems the cleanest. If true batch update capability is added at a later date your datafile is still usable, or at least easily processable into a usable form. Also, creation of the datafile is more straightforward and less error prone. Finally, having the "raw" data would allow import into other data-store formats.
In any case, you should (as you mentioned) wrap the groups of inserts into transactions to avoid the per-row transaction journal creation.