I've modified my DatabaseHelper class to use the SQLCipher library.
To do that, I:
Copied the assets into my assets folder and the libraries (armeabi, x86, commons-codec, guava-r09, sqlcipher) into my libs folder.
Changed the imports in my DatabaseHelper class so that they point to import net.sqlcipher.database.* instead.
Call SQLiteDatabase.loadLibs(getApplicationContext()); when the app starts up.
Modified the lines where I call getReadableDatabase() and getWriteableDatabase() so that they include a passphrase as a parameter;
Everything seems to work fine as data is read/written properly. My issue is related to performance, as my app may execute DB operations with some frequency, causing it to become slow (after migrating to SQLCipher).
For my DatabaseHelper methods, I believe I'm following the standard approach, e.g.:
/*
* Getting all MyObjects
*/
public List<MyObject> getMyObjects() {
List<MyObject> objects = new ArrayList<MyObject>();
String selectQuery = "SELECT * FROM " + TABLE_NAME;
Log.v(LOG, selectQuery);
// Open
SQLiteDatabase db = this.getReadableDatabase("...the password...");
// I know this passphrase can be figured out by decompiling.
// Cursor with query
Cursor c = db.rawQuery(selectQuery, null);
// looping through all rows and adding to list
if (c.moveToFirst()) {
do {
MyObject object = createMyObjectFromCursor(c); // Method that builds MyObject from Cursor data
// adding to list
objects.add(object);
} while (c.moveToNext());
}
c.close();
db.close();
return objects;
}
I'm not entirely familiar with the internal mechanics of SQLCipher (e.g. does it decrypt the whole DB file when I call getReadableDatabase()?) but, while debugging, it seems that the overhead is in getReadableDatabase(password) and getWritableDatabase(password), which makes sense if my supposition above is true.
Would moving those calls to a DatabaseHelper.open() and DatabaseHelper.close() method which would be called by the Activities whenever they instantiate a DatabaseHelper, instead of calling them on each individual method, be a bad practice? Please share your knowledge on how to address this issue.
EDIT:
I've used DDMS to trace one of the methods and I can see that the overhead is indeed at the SQLiteOpenHelper.getReadableDatabase() (taking ~4 sec. each time). The queries seem to work fast and I don't think I need to worry about them.
If I drill down the calls, following the one with the longest duration every time, I end up with:
SQLiteDatabase.OpenOrCreateDatabase --> SqLiteDatabase.openDatabase --> SQLiteDatabase.openDatabase --> SQLiteDatabase.setLocale
So the SQLiteDatabase.setLocale(java.util.Locale) seems to be the culprit, as it is taking ~4 seconds everytime getReadableDatabase() is called. I've looked into the source for SQLiteDatabase and it just locks the DB, calls native_setLocale(locale.toString(), mFlags) (the 4 sec. overhead takes place here) and unlocks the DB.
Any idea on why this happens?
The performance issue you are seeing is most likely due to SQLCipher key derivation. SQLCipher's performance for opening a database is deliberately slow, using PBKDF2 to perform key derivation (i.e. thousands of SHA1 operations) to defend against brute force and dictionary attacks (you can read more about this at http://sqlcipher.net/design). This activity is deferred until the first use of the database, which happens to occur in setLocale, which is why you are seeing the performance issue there when profiling.
The best option is to cache the database connection so that it can be used multiple times without having to open and key the database repeatedly. If this is possible, opening the database once during startup is the preferred course of action. Subsequent access on the same database handle will not trigger key derivation, so performance will be much faster.
If this is not possible the other option is to disable or weaken key derivation. This will cause SQLCipher to use fewer rounds of PBKDF2 when deriving the key. While this will make the database open faster, it is significantly weaker from a security perspective. Thus it is not recommended except in exceptional cases. That said, here is the information on how to reduce the KDF iterations:
http://sqlcipher.net/sqlcipher-api/#kdf_iter
does it decrypt the whole DB file when I call getReadableDatabase()?
No. It decrypts pages (4KB??) as needed, on the fly.
it seems that the overhead is in getReadableDatabase(password) and getWritableDatabase(password), which makes sense if my supposition above is true
Only call those once for the lifetime of your process. Anything else is insecure, as it requires you to keep the password around, above and beyond any overhead issues.
Of course, you seem to be hard-coding a password, in which case all this encryption is pointless and a waste of time.
Please share your knowledge on how to address this issue.
Use Traceview to determine exactly where your time is being spent.
In one benchmark that I performed -- converting a SQLite benchmark to SQLCipher -- I could not detect any material overhead. Disk I/O swamped the encryption overhead, near as I can tell.
To the extent that a well-written SQLCipher for Android app adds overhead, it will make bad operations worse. So, for example, a query that needs to do a table scan sucks already; SQLCipher will make it suck incrementally harder. The solution there is to add the appropriate indexes (or FTS3) as needed to avoid the table scan.
Related
I've got a table with about 7 million rows in it. I'm inserting on average about one row every second into the database. When I do this, I am noticing that it is taking an incredibly long time (as much as 15 seconds) to run a simple SELECT against the database, e.g. something like:
SELECT * FROM table WHERE rowid > 7100000
This select often returns no rows of data as sometimes no data has been inserted in this particular table. It is often happening even when the table I'm writing to isn't even actually inserting rows into the table I am reading.
The idea is that there are two separate processes, one is adding data, the other is trying to get all new data that has not yet been read. But the read side is connected to a UI and any noticable lag is intolerable, much less 15 seconds. This is being run under Android and the the UI thread doesn't like being blocked for that long either and it is wreaking havoc.
My initial thought was maybe the insert is requiring an update to the indicies as originally I had the index on a different field (a time field). This seems at least partially confirmed because if I use a database with only a few rows each select completes in a few milliseconds. But when I re-created the table to only have the rowid as primary key it actually got slower. I would expect inserting a new row at the end would always result in very fast reads when just comparing on the rowid as primary key.
I have tried enabling write ahead logging, but it appears that SQLCipher doesn't support this, at least not directly, as it doesn't adhere to the lastest API for android.database.sqlite.SQLiteDatabase. Even using "PRAGMA journal_mode = WAL" in the postKey hook hasn't made any difference.
What's going on here? How can I speed up my selects?
Update: I tried getting rid of sqlcipher and just using plain sqlite to see if that was a factor. I used sqlcipher_export to export to a plaintext database, and then used the default android.database.sqlite.SQLCipher. The delay time dropped from 10-20s to 1.8-2.8s. I then removed write-ahead and it dropped further to 1.3-2.7s. So the issue is still noticably there, although it did get a lot better.
SQLite is ultimately file-based, and there is no portable mechanism to communicate to another process which part of a file has changed. So when one process has written something, all other processes must drop their caches when they access the database file the next time.
If possible, modify your architecture that both parts of the code are in the same process and share the same database connection. (With multiple threads, this requires locking, but SQLite has not much concurrency anyway.)
Alternatively, write the new data into a separate database, and let the UI app move it to its own database.
I don't know why SQLCipher is so much slower (it's unlikely to be the CPU overhead of the decryption).
In my content provider I create and maintain 3 SQLiteDatabase objects. They are created like this:
private ContentProviderHelper helper;
#Override
public boolean onCreate() { // that's the ContentProvider onCreate()
SQLiteDatabase dbLog = new DbLog(getContext()).getWritableDatabase();
SQLiteDatabase dbSession = new DbSession(getContext()).getWritableDatabase();
SQLiteDatabase dbLocation = new DbLocation(getContext()).getWritableDatabase();
helper = new ContentProviderHelper(UriManager.getAuthority(getContext()));
helper.addDb(dbLog, DbLog.TABLE_NAME, UriManager.LOG, SQLiteDatabase.CONFLICT_REPLACE);
helper.addDb(dbSession, DbSession.TABLE_NAME, UriManager.SESSION, SQLiteDatabase.CONFLICT_REPLACE);
helper.addDb(dbLocation, DbLocation.TABLE_NAME, UriManager.LOCATION, SQLiteDatabase.CONFLICT_REPLACE);
the ContentProviderHelper stores those SQLiteDatabase in an ArrayList indexed with the UriMatcher.
The <provider> is properly registered in the manifest and my app have SD-card permission.
It runs fine for most of our 500.000 user base, but every once in a while I get a SQLiteCantOpenDatabaseException: unable to open database file from the Google Play
the relevant stack track is:
Caused by: android.database.sqlite.SQLiteCantOpenDatabaseException: unable to open database file
at android.database.sqlite.SQLiteDatabase.dbopen(Native Method)
at android.database.sqlite.SQLiteDatabase.openDatabase(SQLiteDatabase.java:1013)
at android.database.sqlite.SQLiteDatabase.openDatabase(SQLiteDatabase.java:986)
at android.database.sqlite.SQLiteDatabase.openOrCreateDatabase(SQLiteDatabase.java:1051)
at android.app.ContextImpl.openOrCreateDatabase(ContextImpl.java:787)
at android.content.ContextWrapper.openOrCreateDatabase(ContextWrapper.java:221)
at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:157)
most often than not those error reports come from generic brandless poor quality devices.
Any help on how to properly avoid those errors will be much appreciated.
edit:
a different SQL exceptions that I'm getting on the same code.
Caused by: android.database.sqlite.SQLiteDiskIOException: disk I/O error: COMMIT;
at android.database.sqlite.SQLiteDatabase.native_execSQL(Native Method)
at android.database.sqlite.SQLiteDatabase.execSQL(SQLiteDatabase.java:1763)
at android.database.sqlite.SQLiteDatabase.endTransaction(SQLiteDatabase.java:583)
at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:137)
at ***.***.***.data.ContentManager.onCreate(ContentManager.java:26)
so is there any other way of handling this besides try{} catch(){} the hell out of it ?
edit:
Extra info regarding cursors usage:
In general cursors are kept opened for the minimum amount of time as possible. All 3 SQLiteDatabase have a Wrapper that do the actual Cursor calls, read the data from them and close them. There's only 1 instance where the cursor is passed back to the calling object, but it is an IntentService, again, single thread, single instance, single process (just like ContentProvider) that loops through the cursor and close it.
This doesn't look like a programming fault on your (Java) side, neither of the Android SQLite wrapping classes.
The provided stacktraces and code doesn't provide much information to proof my guess, but I think this should be caused by the unreliable nature of those SDCards (you already mentioned that the observed traces usually come from low quality phones).
The exception itself is generated in the native sqlite code, and while I havn't looked up the C/C++ part of the JNI interface, this should come directly from the underlying sqlite3_open call. For non obvious reasons the actuall errorcode is not included in the thrown exception, so you are basically out of luck here to find the root cause.
As this comes directly from the native layer it's some kind of filesystem/hardware problem. The card may be broken, the cardsocket may be broken/dirty/whatever or anything in between could be messed up (most likely physically).
To give you a possible solution for a problem you can't fix: Don't use the (sometimes unreliable) SDCard. If your databases are small enough (less than a few MBs) you should be fine storing them on the internal flash. This has to be reliable storage as if thouse writes or queries fail the user has more serious problems anyway.
If this is not an option, you could write "most recent data" (I don't know which kind of data you store) internally and migrate this data reguallary on the SDCard. This way you could at least try to store it without flodding your code with try/catch (if you try this in a seperate thread which catches any exception at the top level), but it doesn't solve the read part of the problem.
Another, while a little ugly, thing I can imagine: Just let the exceptions fly right through and catch them in the Application class (you can overload it), push it to disk (internal, obviously) and rethrow it to crash your app. On the next start you could then lookup if you've been crashed by an SQLite Exception and present a msg to the user which states something like "Sorry, your SDCard seems to be broken as we couldn't use it. Consider buying a new one of order a better device". ACRA may help you with this, it does what I tried to describe pretty well the last time I used it (1 year ago).
What I suppose is that you have some problem with concurrency and you are trying to get DB object more than once.
Moreover - having 3 different DB (probably) makes the application run 3 DB engines at a time (memory problems).
There is no single and simple solution while the problem is located in other place than exception is thrown, but some steps that you can do:
Put 3 databases into one file (if possible).
Make a simple singleton to keep DB object in memory and avoid concurrent attempts to the stored file. Remember that SQLite is claimed as "thread safe" not as "multithread" so the IO error can be just a natural behavior to protect DB corruption.
Make sure that cursors are closed as quick as possible (i.e. after query the object read the cursor and map data to some POJO objects.
Workaround - use try/catch while opening db, in case of exception try to do it after some time (1 second).
I have measured database queries and what I found is that the very first time it runs is slower than all subsequent queries. The very first time I run the code below I got: 11. For all subsequent calls it is always only 1. How does it work so it is faster after being first time queried?
long time = System.currentTimeMillis();
Cursor cursor = DBHelper.getInstance().getAllContacts();
Logger.log(TAG, "measured time is " + String.valueOf(System.currentTimeMillis() - time));
The `getAllContacts() method is:
(getReadableDatabase().rawQuery("SELECT _id, name, title FROM " + CONTACTS_TABLE_NAME, null));
There are a few effects that might be at play here.
First, you are also timing the getReadableDatabase() method: if that is doing significant work the first time (like opening the DB file!) and then caching what it does, you'll have a lot of overhead from that. Try just calling that method on its own before the timed zone so that you know that overhead isn't biting.
Secondly, SQLite works by compiling the SQL to an internal bytecode form and then executing that in its own little virtual machine. The compilation step can have non-trivial costs, and is specific to the SQL and the table(s) queried against. Luckily, you're sending the same SQL in every time (that's a constant table name, yes?) and SQLite is smart enough to implement an LRU cache of the compilations so that if you run the same query twice, it's faster. (I don't know the size of the cache; it's probably set at the time you build SQLite, with some sensible default.) You could test this in theory by just querying against a different table each time, but that would be silly. Closing the connection to the DB would also get rid of this cache, but would guarantee that you get poor performance; you're not supposed to measure just the start of things as then you're bearing lots of extra costs for no good reason.
Thirdly, it's quite possible that SQLite only reads some information about the DB and its tables when it actually needs to, so if this is the first query against that table, you may be bearing a number of extra costs. Try doing a different — otherwise never used — query against the table first, before the timing run. (Also, be aware of my caveats in the previous paragraph.)
Fourthly, the ORM layer might be caching the results, but frankly this is actually pretty unlikely as that's rather hard to get right (the hard part is working out when to flush the query results cache due to the DB being updated, and it's actually easier to just not bother with such complexity).
First time you call an SQL statement, SQLite engine has to prepare that typed SQL statement, ie, convert human readable text in machine executable code, which represents a slightly overhead.
I have an application that I can insert data into a SQLite database. There's a service thread running every sixty seconds checking if a condition is true and raising an alarm if necessary. Think "calendar" application.
I have lots of activities with a reference to their own SQLiteOpenHelper, and their own SQLiteDatabase object.
The application was working pretty well, but one of the main update applications I decided needed a progress dialog subject - that's another subject, but if you can tell me how I can get the spinner to spin that'd be great. It does however display.
But to get it to display, I needed to put it in a thread.
Creating this thread, I started to get loads of "database is locked" messages in the Log.
I figured the issue may have been related to the fact I had a task in the background running every sixty seconds - even though it visually only took a couple of seconds to do the update, so I placed the main write transactions for the updater function in a beginTransaction block. That's the only place I used one.
Everything seemed good, but then I started getting more spurious errors about the database being locked all over the place.
It's encouraged me to do some tidying up, but it seems after I do the first beginTransaction, any further database modifications I want to make fail with a report that the database is locked, despite that fact I do call db.close. I even close the database helper object and cursor just to be on the safe side.
I was able to clear up most of the warnings with regards to cursors still being open, eg
E/Database( 678): android.database.sqlite.DatabaseObjectNotClosedException: Application did not close the cursor or database object that was opened here
I couldn't figure out what was locking the database as everything was being closed.
I then decided to remove the "beginTransaction" which stabilised things - I could now navigate my program again. But Im still getting a few random locking issues.
E/Database( 1304): Failure 5 (database is locked) on 0x26e5d8 when executing 'INSERT INTO
Is it possible to figure out what is locking the database? I can see what is grumbling about the lock, but not what is locking it.
What is the best way to handle concurrent database updates/reads like this? I've read a lot about synchronized and ContentProviders but I must admit, it is a little bit big for me.
Any pointers on the "best" way of doing things?
And can I find out what is locking the database?
Thanks
Simon
class NAME extends SQLiteOpenHelper
add:
#Override
public void onOpen(SQLiteDatabase db) {
super.onOpen(db);
if (!db.isReadOnly()) {
db.execSQL("PRAGMA foreign_keys=ON;");
Cursor c = db.rawQuery("PRAGMA foreign_keys", null);
if (c.moveToFirst()) {
int result = c.getInt(0);
}
if (!c.isClosed()) {
c.close();
}
}
}
You may want to look at the latest version of Berkeley DB as an option. It provides a SQLite3-compatible SQL API (so that your application can stay the same) and it provides a finer granularity of locking and therefore better concurrency. Berkeley DB supports multiple concurrent read and write threads to actively access the database at the same time. You can find out additional information about Berkeley DB on our website.
I'm curious if anyone has done any performance testing on querying a ContentProvider via ContentResolver vs querying a SQLiteDatabase object in the same process. I'm guessing that a ContentResolver query passes back a Cursor that communicates with the database through a Binder (Android IPC). This means that if I read the contents of 100 records through the Cursor that would result 100 Binder method calls. Are my guesses correct and if so would that be significantly slower than accessing the database in the same process?
I have not done exactly that meassure. What I did was to meassure the performance of multiple inserts via a ContentProvider or directly via a SQLite database.
I inserted around 1000 items (one by one). It was much slower to insert via a ContentProvider. In my test almost 10% slower.
There's no definitive answer and results depends on what and how you do it.
For example, I want to share preferences between apps so ContentProvider seems the perfect answer. Yes if I don't mind a longer delay on first read as just connecting to a ContentProvider takes 120ms on a S10+ these days!
So if you have a UI depends on those settings, you'd better copy the preference file between apps (using a ContentProvider) and then read from the file directly otherwise the UI will be delayed before showing with appropriate theme. Fact is the onStart() will have already been called for the newly created activity.
On the opposite side doing DB operations (if done right) will not change much of the results, unless you need to re-connect frequently as it will add a non neglectable overhead.