In my andoid application's database directory (/data/data/com.me.myApp/databases), for every sqlite database I create there is a corresponding database of the same name with "-journal" appended to it's name.
e.g: myDatabase, myDatabase-journal, myOtherDatabase.db, myOtherDatabase.db-journal
What is this?
and,
If I'm providing pre-filled databases for my app (as per: http://www.reigndesign.com/blog/using-your-own-sqlite-database-in-android-applications/) do I need to include these as well?'
Such -journal files do not need to (and should not) be distributed.
This is because the various journal files represent temporary data (ref. SQLite's Use Of Temporary Disk Files) used by SQLite. In particular a -journal file is a rollback journal.
A rollback journal is a temporary file used to implement atomic commit and rollback capabilities in SQLite. (For a detailed discussion of how this works, see the separate document titled Atomic Commit In SQLite.) The rollback journal is always located in the same directory as the database file and has the same name as the database file except with the 8 characters "-journal" appended.
The rollback journal is usually created when a transaction is first started and is usually deleted when a transaction commits or rolls back. The rollback journal file is essential for implementing the atomic commit and rollback capabilities of SQLite. Without a rollback journal, SQLite would be unable to rollback an incomplete transaction, and if a crash or power loss occurred in the middle of a transaction the entire database would likely go corrupt without a rollback journal.
In general these -journal files should only exist when there is an open SQLite database - or rather, a running transaction - but can be controlled via PRAGMA journal_mode. With default pragma settings the -journal files will be deleted.
The DELETE journaling mode is the normal behavior. In the DELETE mode, the rollback journal is deleted at the conclusion of each transaction. Indeed, the delete operation is the action that causes the transaction to commit.
Make sure to only copy the actual database files when the database is not opened and all journals have been deleted (or cleared) by SQLite itself; this implies all transactions have been completed and the database is in a consistent state.
A DB-JOURNAL file is a temporary database file created by SQLite database management systems during a transaction between an application and a database. It contains a rollback journal, which is a temporary database that stores the most recent state of the database.
Is the problem solved in this? I have a case where I can not copy the .db I want to copy on my local machine from the android device. I have to manually delete the -journal on the device to successfully copy the .db. This is my code written in C#
var devices = MediaDevice.GetDevices();
using (var device = devices.First(d => d.Description == "PM67"))
{
device.Connect();
var photoDir = device.GetDirectoryInfo(#"\Internal Shared Storage\Databases");
var files = photoDir.EnumerateFiles("database.db", SearchOption.AllDirectories);
foreach (var file in files)
{
MemoryStream memoryStream = new System.IO.MemoryStream();
device.DownloadFile(file.FullName, memoryStream);
memoryStream.Position = 0;
try
{
device.DeleteFile(#"\Internal Storage\Databases\database.db-journal");//this is the part where I'd like to delete the -journal but it's not working
}
catch
{
}
try
{
if (File.Exists(result) == true)
{
File.Delete(result);
}
}
catch
{
}
WriteSreamToDisk(result, memoryStream);
}
device.Disconnect();
}
Related
I backed up the database by copying the db file, and then recovered the database by overwriting the db file. After recovery, the data of one of the tables was lost. This problem is occasional.
The recovery process is as follows:
Before starting to recover the database, close the currently running database and delete the corresponding database files, including shm and wal files
public void closeDb() {
if (db != null) {
if (db.isOpen()) {
db.getOpenHelper().close();
}
db = null;
}
}
Delete the currently running database files, including shm and wal files, and copy the backup db files (only db files) to the database directory. This step uses the Java File API. It has been confirmed that the file operation result is correct
Reopen the database using the method of initializing the database.
public static DBManager init() {
if (db == null) {
db = Room.databaseBuilder(application,
MyDb.class, dbPath).allowMainThreadQueries().addMigrations(MIGRATION_1_2, MIGRATION_2_3, MIGRATION_3_4, MIGRATION_4_5, MIGRATION_5_6).build();
}
return db;
}
There are no errors in the whole execution process. However, the data of a table in the database cannot be queried. After I try to delete the shm and wal files and restart the app, the data can be queried.
Seen from the problem phenomenon, it seems to be the problem of shm and wal files, but I can't find a solution. I tried to recover the db file, re-open the database, delete the shm and wal files, and then open the database again. There is still a problem that the data of a table cannot be queried.
Has anyone encountered similar problems? Please help me. Thank you
I don't have any ideas to solve this problem
Seen from the problem phenomenon, it seems to be the problem of shm and wal files, but I can't find a solution.
I believe that your issue is that you are not backing up the wal and shm files if they exist and are not empty. Such a backup would be an incomplete copy.
The wal file contains part of the database until it is fully checkpointed. If you do not cater for backing up a wal file that has any data, then you will lose data at best you may even corrupt the database.
You should always either copy the wal and shm file as well as the database if they exist or fully checkpoint, in which case the wal and shm files will be empty or deleted (should be the latter).
The easiest way to fully checkpoint the database is to close it and then backup the file (still best to be safe and see if the wal and shm file exists and back them up if there size is not 0 bytes).
You may wish to refer to https://www.sqlite.org/lang_vacuum.html
The restore should restore all 3 files if the 3 files were backed up (hence closing the database prior to backup is the way to go). The restore should also delete the wal and shm just in case they exist (shouldn't as you are closing the database before the restore).
Another backup option would be to to use VACUUM INTO, which I believe would be a wal/shm free copy of the file as the wal data is considered part of the database.
As for recovering you cannot, as there is no way to know what was in the wal file (the shm file isn't so important as it's a wal file for the wal file).
I have a android app with a local sqlite database that I access using Room Persistence Library.
I am able to access the file using this:
cAssetsDatabase cassetsDatabase = Room.databaseBuilder(context, cAssetsDatabase.class, "localDB.sqlite").fallbackToDestructiveMigration().build();
cAssets cassets = cassetsDatabase.daoAccesscAssetsDB().fetchAssetByTagID(TagID);
Everything works correctly, unfortunately after leaving this activity and returning, trying to access the database returns null.
When I pull the database off the device after accessing it, it seems that the table I used the select on has been truncated.
You will probably notice two other files the same name as the database file but one suffixed with -wal and the other suffixed with -shm. Copy all three files.
This is because WAL (Write-ahead logging) (which is likely being used), writes changes to the WAL file rather than the database file and only applies them to the database file when the database is checkpointed, otherwise the changes are accessed from the -wal file (-shm is a shared memory file a log file for the -wal file).
How can I set the id (which is my #PrimaryKey(autogenerate = true)) back to 0 after I have deleted all data in Room Database Table? For the moment my delete works but the new inserted data increment where the last ID has been.
#Dao
interface MyDao {
#RawQuery
fun vacuumDb(supportSQLiteQuery: SupportSQLiteQuery): Int
}
When all of the content is deleted, execute this statement,
MyDao.vacuumDb(SimpleSQLiteQuery("VACUUM"))
VACUUM command does not change the content of your database but changes rowids. This will reset the rowids.
How VACUUM works?
The VACUUM command works by copying the contents of the database into a temporary database file and then overwriting the original with the contents of the temporary file. When overwriting the original, a rollback journal or write-ahead log WAL file is used just as it would be for any other database transaction. This means that when VACUUMing a database, as much as twice the size of the original database file is required in free disk space.
The VACUUM INTO command works the same way except that it uses the file named on the INTO clause in place of the temporary database and omits the step of copying the vacuumed database back over top of the original database.
The VACUUM command may change the ROWIDs of entries in any tables that do not have an explicit INTEGER PRIMARY KEY.
A VACUUM will fail if there is an open transaction on the database connection that is attempting to run the VACUUM. Unfinalized SQL statements typically hold a read transaction open, so the VACUUM might fail if there are unfinalized SQL statements on the same connection. VACUUM (but not VACUUM INTO) is a write operation and so if another database connection is holding a lock that prevents writes, then the VACUUM will fail.
An alternative to using the VACUUM command to reclaim space after data has been deleted is auto-vacuum mode, enabled using the auto_vacuum pragma. When auto_vacuum is enabled for a database free pages may be reclaimed after deleting data, causing the file to shrink, without rebuilding the entire database using VACUUM. However, using auto_vacuum can lead to extra database file fragmentation. And auto_vacuum does not compact partially filled pages of the database as VACUUM does.
More info: SQLite VACUUM
In Android Pie sqlite Write-Ahead logging (WAL) has been enabled by default. This is causing errors for my existing code only in Pie devices.
I have been unable to turn off WAL successfully using SQLiteDatabase.disableWriteAheadLogging() or PRAGMA journal_mode due to the way I access the database. I would like to disable WAL completely with an Android setting called db_compatibility_wal_supported :
Compatibility WAL (Write-Ahead Logging) for Apps
Does anyone know how to configure this? I don't know if this file can be altered programmatically at startup or if it is changed manually.
Further Details about the problem
I have a sqlite database (20mb+ / 250k records) in my app. This db is generated using plain java on my server. It contains a food database and the user of the app can add to the database (and the server is updated). This is stored in the assets folder in android.
During first installation the database is copied from assets to the app folder so that it can be written to, using effectively this method :
Copy SQLite database from assets folder
Unfortunately, once I start writing to the database using SqlDroid wal is enabled and the tables which were in the original db have vanished and only any newly created tables remain. The size of the database however is still 20mb+. All the database errors are due to the missing tables.
The table copying and writing method works perfectly in versions of Android prior to Pie.
The best and simplest way to disable WAL mode in your Database is as follows:
public class MyDbHelper extends SQLiteOpenHelper {
//...
#Override
public void onOpen(SQLiteDatabase db) {
db.disableWriteAheadLogging(); // Here the solution
super.onOpen(db);
}
//...
}
This way, all access to your database will be with WAL mode disabled. As much as you open and close multiple connections throughout the implementation of your App
If you are using Room, you won't have direct access to the database but you can instead set the journal mode when you are building/opening the database:
db = Room.databaseBuilder(context, Database.class, "Database")
.setJournalMode(JournalMode.TRUNCATE)
.build();
one cannot use SQLDroidDriver.ADDITONAL_DATABASE_FLAGS, simply because there is no constant available, which would negate flag ENABLE_WRITE_AHEAD_LOGGING.
WAL can still be disabled by creating either of these scenarios:
a) set flag OPEN_READONLY (applies to situations where R/O access does suffice).
b) run PRAGMA journal_mode=DELETE as the first query, in order to override PRAGMA journal_mode=WAL.
c) file an issue against SQLDroidConnection.java,
in order to have .enableWriteAheadLogging() and .disableWriteAheadLogging() supported on the driver-level.
#Rockvole please share error that you are facing, that help us to find appropriate solution.
Mean while, i understand that you want to close that WAL in android pie and you are using "SQLDroid" lib to create Sqlite DB.
This lib internally using "SQLiteDatabase" to store data locally, I think you need to call "SQLiteDatabase.disableWriteAheadLogging()" in "SQLiteDatabase" class where DB instance created the package name is "package org.sqldroid;"
or Get internal SQLiteDatabase instance and call disableWriteAheadLogging().
Second solution is create "config.xml" inside values folder and wirte "<bool name="db_compatibility_wal_supported">false</bool>" and run and check its work.
I finally found the answer. It seems that the database errors I was receiving is not directly related to WAL. It is because the widely used code to copy a database from assets has a bug in it where the database is left open during the copy operation. This only started causing a problem in Android P.
The solution is to close the database after you get the database file name.
SQLiteDatabase destinationDatabase = sqLiteOpenHelper.getWritableDatabase();
String dbFileName=destinationDatabase.getPath();
destinationDatabase.close();
// Now write the destinationDatabase using the dbFileName
This is detailed more here :
Android P - 'SQLite: No Such Table Error' after copying database from assets
This question has been asked just out of curiosity!! I have implemented a database DB1 on android. But when I call method this.databaselist in main activity I am shown that two databases exist DB1 and DB-journal. Why is second database created by android automatically? Do we need to take care of these journal databse too while working on multiple databases??
The manner in which SQLite uses temporary files is not considered part of the contract that SQLite makes with applications.
A journal is a temporary file used to implement atomic commit and rollback capabilities in SQLite. The journal is always located in the same directory as the database file and has the same name as the database file except with the 8 characters "-journal" appended.
The journal is usually created when a transaction is first started and is usually deleted when a transaction commits or rolls back. The journal file is essential for implementing the atomic commit and rollback capabilities of SQLite. Without a journal, SQLite would be unable to rollback an incomplete transaction, and if a crash or power loss occurred in the middle of a transaction the entire database would likely go corrupt without a rollback journal.
The journal is usually created and destroyed at the start and end of a transaction, respectively. But there are exceptions to this rule.
If a crash or power loss occurs in the middle of a transaction, then the journal file is left on disk. The next time another application attempts to open the database file, it notices the presence of the abandoned rollback journal (we call it a "hot journal" in this circumstance) and uses the information in the journal to restore the database to its state prior to the start of the incomplete transaction. This is how SQLite implements atomic commit.
If an application puts SQLite in exclusive locking mode using the pragma:
PRAGMA locking_mode=EXCLUSIVE;
For more details Check
you can elect to keep the journal in memory so that no -journal file is created:
pragma journal_mode=memory;
Note that you need to run this each time you open the database.
credit goes here: How to disable Android SQLite Journal file?