I have a android app with a local sqlite database that I access using Room Persistence Library.
I am able to access the file using this:
cAssetsDatabase cassetsDatabase = Room.databaseBuilder(context, cAssetsDatabase.class, "localDB.sqlite").fallbackToDestructiveMigration().build();
cAssets cassets = cassetsDatabase.daoAccesscAssetsDB().fetchAssetByTagID(TagID);
Everything works correctly, unfortunately after leaving this activity and returning, trying to access the database returns null.
When I pull the database off the device after accessing it, it seems that the table I used the select on has been truncated.
You will probably notice two other files the same name as the database file but one suffixed with -wal and the other suffixed with -shm. Copy all three files.
This is because WAL (Write-ahead logging) (which is likely being used), writes changes to the WAL file rather than the database file and only applies them to the database file when the database is checkpointed, otherwise the changes are accessed from the -wal file (-shm is a shared memory file a log file for the -wal file).
Related
I backed up the database by copying the db file, and then recovered the database by overwriting the db file. After recovery, the data of one of the tables was lost. This problem is occasional.
The recovery process is as follows:
Before starting to recover the database, close the currently running database and delete the corresponding database files, including shm and wal files
public void closeDb() {
if (db != null) {
if (db.isOpen()) {
db.getOpenHelper().close();
}
db = null;
}
}
Delete the currently running database files, including shm and wal files, and copy the backup db files (only db files) to the database directory. This step uses the Java File API. It has been confirmed that the file operation result is correct
Reopen the database using the method of initializing the database.
public static DBManager init() {
if (db == null) {
db = Room.databaseBuilder(application,
MyDb.class, dbPath).allowMainThreadQueries().addMigrations(MIGRATION_1_2, MIGRATION_2_3, MIGRATION_3_4, MIGRATION_4_5, MIGRATION_5_6).build();
}
return db;
}
There are no errors in the whole execution process. However, the data of a table in the database cannot be queried. After I try to delete the shm and wal files and restart the app, the data can be queried.
Seen from the problem phenomenon, it seems to be the problem of shm and wal files, but I can't find a solution. I tried to recover the db file, re-open the database, delete the shm and wal files, and then open the database again. There is still a problem that the data of a table cannot be queried.
Has anyone encountered similar problems? Please help me. Thank you
I don't have any ideas to solve this problem
Seen from the problem phenomenon, it seems to be the problem of shm and wal files, but I can't find a solution.
I believe that your issue is that you are not backing up the wal and shm files if they exist and are not empty. Such a backup would be an incomplete copy.
The wal file contains part of the database until it is fully checkpointed. If you do not cater for backing up a wal file that has any data, then you will lose data at best you may even corrupt the database.
You should always either copy the wal and shm file as well as the database if they exist or fully checkpoint, in which case the wal and shm files will be empty or deleted (should be the latter).
The easiest way to fully checkpoint the database is to close it and then backup the file (still best to be safe and see if the wal and shm file exists and back them up if there size is not 0 bytes).
You may wish to refer to https://www.sqlite.org/lang_vacuum.html
The restore should restore all 3 files if the 3 files were backed up (hence closing the database prior to backup is the way to go). The restore should also delete the wal and shm just in case they exist (shouldn't as you are closing the database before the restore).
Another backup option would be to to use VACUUM INTO, which I believe would be a wal/shm free copy of the file as the wal data is considered part of the database.
As for recovering you cannot, as there is no way to know what was in the wal file (the shm file isn't so important as it's a wal file for the wal file).
In my app I'm using Room with a prepopulated database myDatabase.db which is then accessed using this code:
private val database = Room
.databaseBuilder(context.applicationContext, AppDatabase::class.java, dbName)
.createFromAsset("myDatabase.db")
.build()
The file myDatabase.db is stored in the /assets/ folder. Then I had to add some new rows in one on my tables in such database. After that, I got an updated myDatabase.db file, which I copied to the /assets/ folder replacing the old file. And here comes the problem: after installing the app, it still uses the old data. I did some searching but I only found results related to migrations. However, in my case, there is no change in the schema of the table, only some new rows are added. Furthermore, I don't understand why the app doesn't pick the new myDatabase.db file and still uses the old one.
The only solution to get the app to use the new file myDatabase.db was to uninstall the app and then install it again, but that's something I'd like to avoid. So, how can I change the file myDatabase.db in assets folder and make the app pick the data in the new file through Room?
Furthermore, I don't understand why the app doesn't pick the new myDatabase.db file and still uses the old one.
A Room database is persistent and lasts until it is deleted. createFromAsset will only be invoked when the database does not exist and hence why, under normal situations it will not replace the existing database.
A solution assuming that the App only reads the pre-populated daatbase is to:-
add either of the fallbackToDestructiveMigration or fallbackToDestructiveMigrationFrom
Increase the database version number
DO NOT have a Migration that covers the old to new version
The migration instead of failing will delete the database and then attempt to create it. In doing so it will create the database from the new asset.
Currently, our app is using Room SQLite.
We need to let user to create application data backup, and export them as a single zip file.
Direct File Copy
I was wondering, is it safe, to perform direct File copy on application SQLite file to a temporary folder, for further zipping purpose? The reason I ask so is, I notice that sometimes the application database instead of appearing as single file like local-backup, it will have 2 additional files named local-backup-shm and local-backup-wal.
Read and Write to a temporary SQLite DB
Or, I should just create a temporary empty database, use Room to read application data and write to the temporary database? Then, zipping will be performed on the temporary database?
Copying the file is sufficient. That will copy the entire db.
Android Room has method void clearAllTables() and according to docs it makes the following:
Deletes all rows from all the tables that are registered to this database as entities().
This does NOT reset the auto-increment value generated by autoGenerate().
After deleting the rows, Room will set a WAL checkpoint and run VACUUM. This means that the data is completely erased. The space will be reclaimed by the system if the amount surpasses the threshold of database file size.
I checked it in my project and looks like db has no data after that call, but when I pulled
*.db file from my device and opened it in sqlite viewer I've seen that all the data exists, the tables are filled and nothing has been erased.
How can that be? I consider this as a potential flaw in my app. Please provide a reliable approach of cleaning room database
looks like db has no data after that call
It means the method worked.
when I pulled *.db file from my device and opened it in sqlite viewer
I've seen that all the data exists
Most probably the transactions are not moved to the original database from the WAL file yet.
Solution
You can force a checkpoint using the wal_checkpoint pragma. Query the following statement against the database.
pragma wal_checkpoint(full)
In some of my apps I do offer an in-app backup and restore of the Android database. The name of the database file is set in the SQLiteOpenHelper.
Now I heard that there might be additional files depending on the Android Version and/or the manufactor (e.g. HTC).
What are the names of all these files?
It depends.
Standard Android SQLite installations create just one single database file for each database. This file is located in "/data/data//databases" and is named as you instructed to do so for example in the "name" parameter in this constructor:
SQLiteOpenHelper(Context context, String name, SQLiteDatabase.CursorFactory factory, int version)
So a database file is called for example:
/data/data/<packagename>/databases/mydatabase.db
SQLite itself has a new feature called Write-Ahead-Logging (WAL). This WAL uses a different logging mechanism than those from traditional database makers. With WAL new commited data will be stored in an additional file that has "-wal" attached to your original filename. For example:
/data/data/<packagename>/databases/mydatabase.db-wal
This data is missing in the traditional database file until a configurable threadhold is reached - or the app developer instructs the database to move this data over.
However, a WAL-enabled database is not valid with the "*.db" file alone. All additional files are required to form the database. This means a WAL-enabled database will not work on an Android device with an SQLite version that does not support WAL. So a database backup from a WAL-enabled database will not work on older Android systems.
However, only few devices (from HTC for example) and some custom-ROMs enable WAL. Standard Android does not enable WAL.