I backed up the database by copying the db file, and then recovered the database by overwriting the db file. After recovery, the data of one of the tables was lost. This problem is occasional.
The recovery process is as follows:
Before starting to recover the database, close the currently running database and delete the corresponding database files, including shm and wal files
public void closeDb() {
if (db != null) {
if (db.isOpen()) {
db.getOpenHelper().close();
}
db = null;
}
}
Delete the currently running database files, including shm and wal files, and copy the backup db files (only db files) to the database directory. This step uses the Java File API. It has been confirmed that the file operation result is correct
Reopen the database using the method of initializing the database.
public static DBManager init() {
if (db == null) {
db = Room.databaseBuilder(application,
MyDb.class, dbPath).allowMainThreadQueries().addMigrations(MIGRATION_1_2, MIGRATION_2_3, MIGRATION_3_4, MIGRATION_4_5, MIGRATION_5_6).build();
}
return db;
}
There are no errors in the whole execution process. However, the data of a table in the database cannot be queried. After I try to delete the shm and wal files and restart the app, the data can be queried.
Seen from the problem phenomenon, it seems to be the problem of shm and wal files, but I can't find a solution. I tried to recover the db file, re-open the database, delete the shm and wal files, and then open the database again. There is still a problem that the data of a table cannot be queried.
Has anyone encountered similar problems? Please help me. Thank you
I don't have any ideas to solve this problem
Seen from the problem phenomenon, it seems to be the problem of shm and wal files, but I can't find a solution.
I believe that your issue is that you are not backing up the wal and shm files if they exist and are not empty. Such a backup would be an incomplete copy.
The wal file contains part of the database until it is fully checkpointed. If you do not cater for backing up a wal file that has any data, then you will lose data at best you may even corrupt the database.
You should always either copy the wal and shm file as well as the database if they exist or fully checkpoint, in which case the wal and shm files will be empty or deleted (should be the latter).
The easiest way to fully checkpoint the database is to close it and then backup the file (still best to be safe and see if the wal and shm file exists and back them up if there size is not 0 bytes).
You may wish to refer to https://www.sqlite.org/lang_vacuum.html
The restore should restore all 3 files if the 3 files were backed up (hence closing the database prior to backup is the way to go). The restore should also delete the wal and shm just in case they exist (shouldn't as you are closing the database before the restore).
Another backup option would be to to use VACUUM INTO, which I believe would be a wal/shm free copy of the file as the wal data is considered part of the database.
As for recovering you cannot, as there is no way to know what was in the wal file (the shm file isn't so important as it's a wal file for the wal file).
Related
I have a android app with a local sqlite database that I access using Room Persistence Library.
I am able to access the file using this:
cAssetsDatabase cassetsDatabase = Room.databaseBuilder(context, cAssetsDatabase.class, "localDB.sqlite").fallbackToDestructiveMigration().build();
cAssets cassets = cassetsDatabase.daoAccesscAssetsDB().fetchAssetByTagID(TagID);
Everything works correctly, unfortunately after leaving this activity and returning, trying to access the database returns null.
When I pull the database off the device after accessing it, it seems that the table I used the select on has been truncated.
You will probably notice two other files the same name as the database file but one suffixed with -wal and the other suffixed with -shm. Copy all three files.
This is because WAL (Write-ahead logging) (which is likely being used), writes changes to the WAL file rather than the database file and only applies them to the database file when the database is checkpointed, otherwise the changes are accessed from the -wal file (-shm is a shared memory file a log file for the -wal file).
How can I set the id (which is my #PrimaryKey(autogenerate = true)) back to 0 after I have deleted all data in Room Database Table? For the moment my delete works but the new inserted data increment where the last ID has been.
#Dao
interface MyDao {
#RawQuery
fun vacuumDb(supportSQLiteQuery: SupportSQLiteQuery): Int
}
When all of the content is deleted, execute this statement,
MyDao.vacuumDb(SimpleSQLiteQuery("VACUUM"))
VACUUM command does not change the content of your database but changes rowids. This will reset the rowids.
How VACUUM works?
The VACUUM command works by copying the contents of the database into a temporary database file and then overwriting the original with the contents of the temporary file. When overwriting the original, a rollback journal or write-ahead log WAL file is used just as it would be for any other database transaction. This means that when VACUUMing a database, as much as twice the size of the original database file is required in free disk space.
The VACUUM INTO command works the same way except that it uses the file named on the INTO clause in place of the temporary database and omits the step of copying the vacuumed database back over top of the original database.
The VACUUM command may change the ROWIDs of entries in any tables that do not have an explicit INTEGER PRIMARY KEY.
A VACUUM will fail if there is an open transaction on the database connection that is attempting to run the VACUUM. Unfinalized SQL statements typically hold a read transaction open, so the VACUUM might fail if there are unfinalized SQL statements on the same connection. VACUUM (but not VACUUM INTO) is a write operation and so if another database connection is holding a lock that prevents writes, then the VACUUM will fail.
An alternative to using the VACUUM command to reclaim space after data has been deleted is auto-vacuum mode, enabled using the auto_vacuum pragma. When auto_vacuum is enabled for a database free pages may be reclaimed after deleting data, causing the file to shrink, without rebuilding the entire database using VACUUM. However, using auto_vacuum can lead to extra database file fragmentation. And auto_vacuum does not compact partially filled pages of the database as VACUUM does.
More info: SQLite VACUUM
In Android Pie sqlite Write-Ahead logging (WAL) has been enabled by default. This is causing errors for my existing code only in Pie devices.
I have been unable to turn off WAL successfully using SQLiteDatabase.disableWriteAheadLogging() or PRAGMA journal_mode due to the way I access the database. I would like to disable WAL completely with an Android setting called db_compatibility_wal_supported :
Compatibility WAL (Write-Ahead Logging) for Apps
Does anyone know how to configure this? I don't know if this file can be altered programmatically at startup or if it is changed manually.
Further Details about the problem
I have a sqlite database (20mb+ / 250k records) in my app. This db is generated using plain java on my server. It contains a food database and the user of the app can add to the database (and the server is updated). This is stored in the assets folder in android.
During first installation the database is copied from assets to the app folder so that it can be written to, using effectively this method :
Copy SQLite database from assets folder
Unfortunately, once I start writing to the database using SqlDroid wal is enabled and the tables which were in the original db have vanished and only any newly created tables remain. The size of the database however is still 20mb+. All the database errors are due to the missing tables.
The table copying and writing method works perfectly in versions of Android prior to Pie.
The best and simplest way to disable WAL mode in your Database is as follows:
public class MyDbHelper extends SQLiteOpenHelper {
//...
#Override
public void onOpen(SQLiteDatabase db) {
db.disableWriteAheadLogging(); // Here the solution
super.onOpen(db);
}
//...
}
This way, all access to your database will be with WAL mode disabled. As much as you open and close multiple connections throughout the implementation of your App
If you are using Room, you won't have direct access to the database but you can instead set the journal mode when you are building/opening the database:
db = Room.databaseBuilder(context, Database.class, "Database")
.setJournalMode(JournalMode.TRUNCATE)
.build();
one cannot use SQLDroidDriver.ADDITONAL_DATABASE_FLAGS, simply because there is no constant available, which would negate flag ENABLE_WRITE_AHEAD_LOGGING.
WAL can still be disabled by creating either of these scenarios:
a) set flag OPEN_READONLY (applies to situations where R/O access does suffice).
b) run PRAGMA journal_mode=DELETE as the first query, in order to override PRAGMA journal_mode=WAL.
c) file an issue against SQLDroidConnection.java,
in order to have .enableWriteAheadLogging() and .disableWriteAheadLogging() supported on the driver-level.
#Rockvole please share error that you are facing, that help us to find appropriate solution.
Mean while, i understand that you want to close that WAL in android pie and you are using "SQLDroid" lib to create Sqlite DB.
This lib internally using "SQLiteDatabase" to store data locally, I think you need to call "SQLiteDatabase.disableWriteAheadLogging()" in "SQLiteDatabase" class where DB instance created the package name is "package org.sqldroid;"
or Get internal SQLiteDatabase instance and call disableWriteAheadLogging().
Second solution is create "config.xml" inside values folder and wirte "<bool name="db_compatibility_wal_supported">false</bool>" and run and check its work.
I finally found the answer. It seems that the database errors I was receiving is not directly related to WAL. It is because the widely used code to copy a database from assets has a bug in it where the database is left open during the copy operation. This only started causing a problem in Android P.
The solution is to close the database after you get the database file name.
SQLiteDatabase destinationDatabase = sqLiteOpenHelper.getWritableDatabase();
String dbFileName=destinationDatabase.getPath();
destinationDatabase.close();
// Now write the destinationDatabase using the dbFileName
This is detailed more here :
Android P - 'SQLite: No Such Table Error' after copying database from assets
Android Room has method void clearAllTables() and according to docs it makes the following:
Deletes all rows from all the tables that are registered to this database as entities().
This does NOT reset the auto-increment value generated by autoGenerate().
After deleting the rows, Room will set a WAL checkpoint and run VACUUM. This means that the data is completely erased. The space will be reclaimed by the system if the amount surpasses the threshold of database file size.
I checked it in my project and looks like db has no data after that call, but when I pulled
*.db file from my device and opened it in sqlite viewer I've seen that all the data exists, the tables are filled and nothing has been erased.
How can that be? I consider this as a potential flaw in my app. Please provide a reliable approach of cleaning room database
looks like db has no data after that call
It means the method worked.
when I pulled *.db file from my device and opened it in sqlite viewer
I've seen that all the data exists
Most probably the transactions are not moved to the original database from the WAL file yet.
Solution
You can force a checkpoint using the wal_checkpoint pragma. Query the following statement against the database.
pragma wal_checkpoint(full)
In my andoid application's database directory (/data/data/com.me.myApp/databases), for every sqlite database I create there is a corresponding database of the same name with "-journal" appended to it's name.
e.g: myDatabase, myDatabase-journal, myOtherDatabase.db, myOtherDatabase.db-journal
What is this?
and,
If I'm providing pre-filled databases for my app (as per: http://www.reigndesign.com/blog/using-your-own-sqlite-database-in-android-applications/) do I need to include these as well?'
Such -journal files do not need to (and should not) be distributed.
This is because the various journal files represent temporary data (ref. SQLite's Use Of Temporary Disk Files) used by SQLite. In particular a -journal file is a rollback journal.
A rollback journal is a temporary file used to implement atomic commit and rollback capabilities in SQLite. (For a detailed discussion of how this works, see the separate document titled Atomic Commit In SQLite.) The rollback journal is always located in the same directory as the database file and has the same name as the database file except with the 8 characters "-journal" appended.
The rollback journal is usually created when a transaction is first started and is usually deleted when a transaction commits or rolls back. The rollback journal file is essential for implementing the atomic commit and rollback capabilities of SQLite. Without a rollback journal, SQLite would be unable to rollback an incomplete transaction, and if a crash or power loss occurred in the middle of a transaction the entire database would likely go corrupt without a rollback journal.
In general these -journal files should only exist when there is an open SQLite database - or rather, a running transaction - but can be controlled via PRAGMA journal_mode. With default pragma settings the -journal files will be deleted.
The DELETE journaling mode is the normal behavior. In the DELETE mode, the rollback journal is deleted at the conclusion of each transaction. Indeed, the delete operation is the action that causes the transaction to commit.
Make sure to only copy the actual database files when the database is not opened and all journals have been deleted (or cleared) by SQLite itself; this implies all transactions have been completed and the database is in a consistent state.
A DB-JOURNAL file is a temporary database file created by SQLite database management systems during a transaction between an application and a database. It contains a rollback journal, which is a temporary database that stores the most recent state of the database.
Is the problem solved in this? I have a case where I can not copy the .db I want to copy on my local machine from the android device. I have to manually delete the -journal on the device to successfully copy the .db. This is my code written in C#
var devices = MediaDevice.GetDevices();
using (var device = devices.First(d => d.Description == "PM67"))
{
device.Connect();
var photoDir = device.GetDirectoryInfo(#"\Internal Shared Storage\Databases");
var files = photoDir.EnumerateFiles("database.db", SearchOption.AllDirectories);
foreach (var file in files)
{
MemoryStream memoryStream = new System.IO.MemoryStream();
device.DownloadFile(file.FullName, memoryStream);
memoryStream.Position = 0;
try
{
device.DeleteFile(#"\Internal Storage\Databases\database.db-journal");//this is the part where I'd like to delete the -journal but it's not working
}
catch
{
}
try
{
if (File.Exists(result) == true)
{
File.Delete(result);
}
}
catch
{
}
WriteSreamToDisk(result, memoryStream);
}
device.Disconnect();
}