TimeoutException: net.sqlcipher.database.SQLiteCompiledSql.finalize() timed out after 10 seconds (Android) - android

In my android app I am using Room with sqlcipher library for encrypt/decrypt. Often I see in Crashlytic the following crash:
java.util.concurrent.TimeoutException:
net.sqlcipher.database.SQLiteCompiledSql.finalize() timed out after 10
seconds at sun.misc.Unsafe.park(Native Method) at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:190) at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:868)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:902)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1227)
at
java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:231)
at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:294)
at
net.sqlcipher.database.SQLiteDatabase.lock(SQLiteDatabase.java:567)
at
net.sqlcipher.database.SQLiteCompiledSql.releaseSqlStatement(SQLiteCompiledSql.java:104)
at
net.sqlcipher.database.SQLiteCompiledSql.finalize(SQLiteCompiledSql.java:146)
at java.lang.Daemons$FinalizerDaemon.doFinalize(Daemons.java:289) at
java.lang.Daemons$FinalizerDaemon.runInternal(Daemons.java:276) at
java.lang.Daemons$Daemon.run(Daemons.java:137) at
java.lang.Thread.run(Thread.java:929)
The line where it crash is SQLiteDatabase.lock() line 567
Previously it was line 566, but in that method I inserted a check: if the database is not open -> return and don't proceed with locking, but it didn't help and the crash appeared again.
I think that this crash may be because the garbage collecting happens when the app is in the background (our app has a foreground service that works all the time). But not sure how to fix it.
As for Room: I don't close it, it is open all the time, because my app works all the time, so need it often. And close it after every query to the database is the bad practice.
I asked the developers of sqlcipher, but they don't know, what can cause this crash. Maybe anybody knows?

I've been seeing this in Crashlytics for quite a while, and seemingly has been able to fix this in the end completely, I'll post the full research here.
Issue
For queries like this one:
#Query("DELETE FROM table WHERE id NOT IN (:ids)")
abstract fun deleteNotInList(ids: List<String>): Int
Room generates code that doesn’t clean cursor and generated statements (on the image below, compare how the method one doesn’t use cursor, and doesn’t call release() methods, compared to the bottom one which calls cursor.close(); method names and queries are slightly different there, cause I simplified the snippets):
In this scenario statements are kept in memory un-released, and the releasing shifts to GC phase, inside finalize() method. And finalize() in SqlCipher (SQLiteCompiledSql) in turn needs to lock the Database in order to release the statement:
Problem is that the Database might be locked for a long transaction for more than 10 sec (or a batch of shorter transactions, awakening order using such locks isn’t guaranteed and isn’t fair).
GC watchdog crashes the thread when it reaches 10/20 seconds depending on the OS/JVM exact version.
Solution
Is to rewrite all DELETE / UPDATE using IN operator with manual raw queries like this:
#RawQuery
abstract fun deleteNotInListRaw(query: SimpleSQLiteQuery): Int
fun deleteNotInList(
ids: List<String>
) {
deleteNotInListRaw(
SimpleSQLiteQuery(
"DELETE FROM table WHERE id NOT IN (${ids.joinToString(prefix = "'", postfix = "'", separator = "','")})"
)
)
}
In this case the query uses cursor and closes it after deletion is done, still locking database but not on GC phase and on the dedicated thread.
P.S.
There’re more stable long term solutions possible but they needs to be implemented on the Room / SqlCipher sides.
In the given state, SqlCipher can be refactored to not lock the database on GC phase. There’s an open issue for that: https://github.com/sqlcipher/android-database-sqlcipher/issues/537
Room should probably fix the codegen and use the query builder and generate statement closure lines, there’s no open issues there, but I’ll double check this thought later and will raise this as an issue with them.
For us this has seemingly solved the issue completely.
Note that all DELETE / UPDATE queries using IN / NOT IN operators and probably some other operators that prevent Room from precompiling the query (due to runtime params) cause this. You can check the codegen to verify that generated code calls either cursor.close() or statement.release()

Related

How do I get Room database size

I want to monitor the size of my Room database in my android app. I have searched around for a while and found little info about exactly how to do that. Is there any Room (SQLite) command to determine the db size ?
After a while, searching around, I found some "old" (aka ancient) pragma calls which I tested, and it worked somehow. Not in the need to get the exact values, but a hint of the usage.
You could do a #Dao #Query in Room like this:
#Dao
interface SystemDao{
#Query("""
SELECT s.*, c.*
FROM pragma_page_size as s
JOIN pragma_page_count as c
""")
suspend fun getDatabaseSizeInfo():List<SystemDatabaseSizeInfo>
}
And in the data class "pojo" you define:
data class SystemDatabaseSizeInfo(
var page_size:Int?=null,
var page_count:Int?=null
)
Since the result from the pragma queries are page_size and page_count. Theese can later be multiplied, or presented separately.
You can read about SQLite pragma calls here:
https://www.sqlite.org/pragma.html
There are bunch of other pragma queries which may useful, but you should use them visely, since some of them may be devastating for the stability of the SQLite DB.
Android IDE/lint may render theese commands not valid (red), but if you have AS4.1Cx (C6+) you may be able to run some of them in the Database Inspector tool to see if they work. I have filed an issuetracker incident on that here: https://issuetracker.google.com/157374656
Thank you for all who answered, you led me to the correct path with intensive googling...
Haven't tried it, but it should be something alike this:
fun getSize(context: Context, fileName: String) : Long {
val file = context.getDatabasePath(fileName)
return file.length()
}
As #Commonsware suggested, unless the write-ahead log had been committed, it may be inaccurate (it's these *.wal files in the same directory). And one can't just add up the sizes, but would have to check if the database file is currently in use and if so, close it first. Another option would be .disableWriteAheadLogging(), which comes at the cost of some performance.
In Android Studio in debug mode in either a device or emulator, you can use the Device File Explorer which is at bottom right of Android Studio, goto data -> data -> your package name (eg. com.myapp) -> databases:

Updated RealmResults with time-based query

I'm starting to approach this wonderful world of Realm. I'm very happy of the results I'm getting and now I have one question to submit.
In my android app I've got a Fragment that displays data retrieved from Realm. The query condition is that the time this data refers to is in between the beginning and the end of today.
RealmResults<Appointment> results = realm
.where(MyObject.class)
.between("begin", rangeBegin, rangeEnd)
.between("end", rangeBegin, rangeEnd)
.findAllSorted("begin", Sort.ASCENDING);
This query is executed in the onStart() method helping me to exploit the live-update feature, which indeed works very well.
I've also added listeners for changes in order to optimize UI updates.
Now the question is: how does this live-update behave if the time conditions change? (Imagine I keep the app opened for more than one day without touching it or simply I keep the app opened for minutes around midnight)
From what I've seen it seems to do the same query done the very first time onStart() was executed.
Is there a way to have also live-updating query or should I re-run that query somewhere else outside onStart()?
Thank you in advance
Now the question is: how does this live-update behave if the time conditions change? (Imagine I keep the app opened for more than one day without touching it or simply I keep the app opened for minutes around midnight)
The query is pretty constant after you've set it up, so you'd need to execute a new query with different parameters for rangeStart and rangeEnd, and replace your other results.
This query is executed in the onStart() method helping me to exploit the live-update feature, which indeed works very well. I've also added listeners for changes in order to optimize UI updates.
Personally I'd advise to put the query in onCreateView() instead, and the Realm lifecycle management to onCreateView() and onDestroyView().
Also, you can avoid manually assigning RealmChangeListeners for displaying lists if you use RealmRecyclerViewAdapter (adapters 1.3.0 works with realm 1.2.0).
If you use RealmRecyclerViewAdapter, then just call adapter.updateData(newResults); and it'll update the view as needed.

Running JUnit test multiple times gives different results

I have a test case for my app which fills in the TextViews in an Activity and then simulates clicking the Save button which commits the data to a database. I repeat this several times with different data, call Instrumentation.waitForIdleSync(), and then check that the data inserted is in fact in the database. I recently ran this test three times in a row without changing or recompiling my code. The result each time was different: one test run passed and the other two test runs reported different data items missing from the database. What could cause this kind of behavior? Is it possibly due to some race condition between competing threads? How do I debug this when the outcome differs each time I run it?
Looks like a race condition.
remember in the world of threading there is no way to ensure runtime order.
I'm not an android dev so I'm only speculating but UI is only on one event thread generally so when you call the method from another thread (your test) you're probably breaking that as you're outside of the event thread.
You could try using a semaphore or more likely a lock on the resource.
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/Lock.html
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/Semaphore.html
I (finally!) found a solution to this problem. I now call finish() on the tested Activity to make sure that all of its connections to the database are closed. This seems to ensure consistency in the data when I run the assertions.
I would suggest making a probe for the database data rather than a straight assert on it. By this I mean make a piece of code that will keep checking the database for up to a certain amount of time for a condition rather than waiting for x seconds (or idle time) then check, I am not on a proper computer so the following is only pseudo code
public static void assertDatabaseHasData(String message, String dataExpected, long maxTimeToWaitFor){
long timeToWaitUntil = System.getCurrentTimeMillis() + maxTimeToWaitFor;
boolean expectationMatched = false;
do {
if(databaseCheck() == dataExpected){
expecttionMatched == true;
}
}while(!expectationMatched && System.getCurrentTimeMillis() < timeToWaituntil);
assertTrue(message, expectationMatched);
}
When i get to a computer i will try to relook into the above and make it better (I would actually of used hamcrest rather than asserts but that is personal preference)

Is closing a database opened with window.openDatabase necessary?

The code at the moment reads something in the order of...
DoAnything() {
OpenTheDatabase()
// ... Do all the things! ...
}
However, the database object is never closed. This is worrisome.
The database is opened as follows:
var db = window.openDatabase( ... paramters ... );
No .closeDatabase function exists, or the documentation is incomplete. I thought the following might suffice:
db=null;
I see that sqlite3_close(sqlite3*) and int sqlite3_close_v2(sqlite3*) exist, but I'm unsure how to apply them in this case.
How do I close the database, and is it necessary?
Generally you only have one database connection that you open on app startup, and there is no need to close it while the app is open. It's a single threaded, single user app, so a lot of the normal rules about database connections don't apply.
When the app shuts down, you can rely on the browser to close everything - given the average quality of code on the web, browsers have to be pretty good at cleanup.
Setting db to null and letting the garbage collector do its thing will probably also work, but it is better not to create the extra objects in the first place.

Using the same database in successive activities in Android without memory leak

I'll preface this question with the note that I have looked at this similar question, but I'm still encountering issues. Basically, I want to access the same database in two activities in my Android application. However, when I open it in the second activity, I'm getting two series of messages in my LogCat:
First:
"Uncaught exception thrown by finalizer (will be discarded):
Ljava/lang/IllegalStateException;: Finalizing cursor android.database.sqlite.SQLiteCursor#436053b8 on dogs that has not been deactivated or closed
at android.database.sqlite.SQLiteCursor.finalize(SQLiteCursor.java:596)"
(dogs is the name of a table in my database, dog_data)
Second:
"ERROR/Database(1316): Leak found
ERROR/Database(1316): java.lang.IllegalStateException: /data/data/com..../databases/dog_data SQLiteDatabase created and never closed"
As far as I can tell, I am closing my database upon exiting the first activity. Following the style of the notepad tutorial, I have a wrapper class "DbAdapter" around my SQLiteDatabase, and in the onPause() method of the first activity, I call the close method on that Adapter (which calls the close methods on my SQLiteDatabase and my SQLiteOpenHelper).
I think the issue is how I am trying to reopen the database in my second activity:
SQLiteDatabase db = openOrCreateDatabase("dog_data",
SQLiteDatabase.CREATE_IF_NECESSARY, null);
(I choose not to use a wrapper because I only needed to run one query on the database, perhaps this is an issue).
Can someone advise as to where my issue might be? I'll admit (as may be clear from my question) that I don't fully understand the implications of "closing" a database (the documentation for SQLiteDatabase.close() is not particularly specific), which is probably the main reason for my problem.
Thanks.
Just in case someone happens to encounter a similar issue (seems possible but probably unlikely), I recently stumbled onto the solution. In the insert method of my "DbAdapter", I was (stupidly) checking uniqueness via a query for a row with a given value for one of the fields, and seeing whether that query returned any rows. This was creating a cursor that I wasn't closing, which resulted in the "Finalizing cursor" error noted above.
I've received that error before and had to use cursor.close() to correct the issue. I'm not exactly sure why because there are times when I didn't use close() and received no error. Maybe it's a warning that only gets noticed when it is sitting next to a show stopping error?
I will say the proper procedure is open database connection -> create cursor by running db method -> iterate through cursor -> close cursor -> close database connection.

Categories

Resources