I'm having a problem with a simple #insert operation in a Room database. These are my classes:
The model class
#Entity(tableName = "my_model")
data class MyModel(
#PrimaryKey #ColumnInfo(name = "id_model") var uid: Int,
#ColumnInfo(name = "name") var firstName: String,
#ColumnInfo(name = "last_name") var lastName: String
)
The DAO interface
interface MyModelDAO {
#Insert
fun createMyModel(myModel: MyModel)
}
The database
#Database(
entities = [(MyModel::class)],
version = 1,
exportSchema = false
)
abstract class MyDb : RoomDatabase() {
companion object {
private var INSTANCE: MyDb? = null
fun getInstance(context: Context): MyDb? {
if (INSTANCE == null) {
synchronized(MyDb::class) {
INSTANCE = Room.databaseBuilder(context.applicationContext,
MyDb::class.java, "mydb.db")
.allowMainThreadQueries()//for testing purposes only
.build()
}
}
return INSTANCE
}
fun destroyInstance() {
INSTANCE = null
}
}
abstract fun getMyModelDao(): MyModelDAO
}
And this is how I'm trying to insert an object.
val db = MinhaDb.getInstance(this)
db?.getMyModelDao()?.createMyModel(MyModel(111, "john", "doe"))
The thing is, the operation is not persisted in the db file. If I go into the databases folder, there is a mydb file, a wal and a shm file, and no tables are created in mydb.
However, if i call db?.close() after the insert operation, the operation happens as its supposed to (the table is created and populated) and the wal and shm files are not there.
What am I missing here? I'm pretty sure I shouldn't have to call close() on the database. I've tried surrounding the insert call with a beginTransaction() and a endTransaction() calls to see if it changed anything, but it didn't.
UPDATE:
As #musooff explained in the comments, apparently that's how sqlite dbs work. I queried the database after the insert calls and, indeed, the records where returned, even though the file itself seems empty.
TL;DR
Your code seems to be working fine. Don't get confused by the temporary files SQLite creates to operate.
The WAL and SHM files are temporary internal files that you shouldn't worry about.
If you are checking if the data is present examining the db file directly, the data might not be there yet. Wait until you close the connection.
Use a SQLiteBrowser to see if the data is present or not. You can check SQLiteBrowser or Android Debug Database
Instead of using a SQLiteBrowser, you can simple check if the data is present from your Android app using a SELECT query.
WAL and SHM files
As you have noticed, in your db directory you can find three generated files:
your-database-name
your-database-name-shm
your-database-name-wal
However, the only important for you, where the data really is, is your-database-name.
The wal file is used as a replacement of the Rollback Journal.
Beginning with version 3.7.0 (2010-07-21), SQLite supports a new transaction control mechanism called "write-ahead log" or "WAL". When a database is in WAL mode, all connections to that database must use the WAL. A particular database will use either a rollback journal or a WAL, but not both at the same time. The WAL is always located in the same directory as the database file and has the same name as the database file but with the string "-wal" appended.
What you mention about not being able to see the data in your database file while the wal file is still there, and then you close the connection and the wal file is gone and the data is finally persisted in the database, is the proper behavior while using the wal mechanism.
WAL dissapears
The WAL file exists for as long as any database connection has the database open. Usually, the WAL file is deleted automatically when the last connection to the database closes. (More here)
Transaction not being written immediately to database file
The traditional rollback journal works by writing a copy of the original unchanged database content into a separate rollback journal file and then writing changes directly into the database file. In the event of a crash or ROLLBACK, the original content contained in the rollback journal is played back into the database file to revert the database file to its original state. The COMMIT occurs when the rollback journal is deleted.
The WAL approach inverts this. The original content is preserved in
the database file and the changes are appended into a separate WAL
file. A COMMIT occurs when a special record indicating a commit is
appended to the WAL. Thus a COMMIT can happen without ever writing to
the original database, which allows readers to continue operating from
the original unaltered database while changes are simultaneously being
committed into the WAL. Multiple transactions can be appended to the
end of a single WAL file.
Of course, one wants to eventually transfer all the transactions that are appended in the WAL file back into the original database.
Moving the WAL file transactions back into the database is called a
"checkpoint".
By default, SQLite does a checkpoint automatically when the WAL file
reaches a threshold size of 1000 pages. (The
SQLITE_DEFAULT_WAL_AUTOCHECKPOINT compile-time option can be used to
specify a different default.) Applications using WAL do not have to do
anything in order to for these checkpoints to occur. But if they want
to, applications can adjust the automatic checkpoint threshold. Or
they can turn off the automatic checkpoints and run checkpoints during
idle moments or in a separate thread or process. (More here)
The SHM is just a temporary shared memory file, related to the WAL mechanism, whose only purpose is:
The shared-memory file contains no persistent content. The only purpose of the shared-memory file is to provide a block of shared memory for use by multiple processes all accessing the same database in WAL mode. (More here)
Related
I now finished the android codelab about room db with MVVM arch. But there is one part that i didn't exactly understand. This is a sentence from the codelab:
To delete all content and repopulate the database
whenever the app is created, you'll create a RoomDatabase.Callback and override onCreate().
and this is the code they provide:
private class WordDatabaseCallback(
private val scope: CoroutineScope
) : RoomDatabase.Callback() {
override fun onCreate(db: SupportSQLiteDatabase) {
super.onCreate(db)
INSTANCE?.let { database ->
scope.launch {
populateDatabase(database.wordDao())
}
}
}
suspend fun populateDatabase(wordDao: WordDao) {
// Delete all content here.
wordDao.deleteAll()
}
}
the part that I don't understand is "deleting all content". why do I need to delete all content when the app is created? and what do they mean by "whenever the app is created"? is it for the first time the app installed or everytime app is opened?
when I don't use this code, the app works fine too. can someone explain the purpose of deleting everything?
why do I need to delete all content when the app is created?
You don't need to, this is a specific scenario where for whatever reason you want the App to delete all content stored in the database. However there should not be any anyway as onCreate is only called when the database doesn't actually exist (unless manually invoked), so there will be no content to delete.
If you used a pre-packaged database using the .createFromAsset then onCreate isn't called.
I believe that they have just included a simple, unlikely to fail, introduction to using a Callback.
and what do they mean by "whenever the app is created"? is it for the first time the app installed or everytime app is opened?
The former. That is the whole purpose of a Database is to store data long term. As such the database is stored in the App's data space. When an App is installed then the data space will not have the database. So the App and thus Room has to know to create the database before it can be used.
So when you attempt to use the database, the processing of preparing to use the database checks to see if the database exists.
If the database does exist then it carries on without calling onCreate.
If the database does not exist then Room will try to create the database and the tables therein via the onCreate method. The Callback allows intervention at this stage by overriding the onCreate method.
If the App is stopped and rerun the database will still exist, and onCreate is not called.
If the App is uninstalled and then re-installed the the database will be deleted, so onCreate is called.
If a new version of the App is installed, then the database will still exist and onCreate is not called.
If a new version of the App is installed and it includes a new version for the database, onCreate is not called. Instead whatever migration path is specified will be taken.
I made a screen like the current image.
Data such as A, B, C.. are currently being set by getting from the strings.xml resource file.
I am now going to use Room DB instead of strings.xml and I want to get these data from Room.
To do this, we need to pre-populate the Room with data.
In the sample code I found, the method called addCallback() was usually used.
like this :
#Database(entities = arrayOf(Data::class), version = 1)
abstract class DataDatabase : RoomDatabase() {
abstract fun dataDao(): DataDao
companion object {
#Volatile private var INSTANCE: DataDatabase? = null
fun getInstance(context: Context): DataDatabase =
INSTANCE ?: synchronized(this) {
INSTANCE ?: buildDatabase(context).also { INSTANCE = it }
}
private fun buildDatabase(context: Context) =
Room.databaseBuilder(context.applicationContext,
DataDatabase::class.java, "Sample.db")
// prepopulate the database after onCreate was called
.addCallback(object : Callback() {
override fun onCreate(db: SupportSQLiteDatabase) {
super.onCreate(db)
// insert the data on the IO Thread
ioThread {
getInstance(context).dataDao().insertData(PREPOPULATE_DATA)
}
}
})
.build()
val PREPOPULATE_DATA = listOf(Data("1", "val"), Data("2", "val 2"))
}
}
However, as you can see from the code, in the end, data (here, val PREPOPULATE_DATA) is being created again within the code. (In another code, db.execSQL() is used)
In this way, there is no difference from fetching data from resource file in the end.
Is there any good way?
Developer documentation uses assets and files.
However, it is said that it is not supported within In-memory Room databases.
In this case, I do not know what In-memory means, so I am not using it.
In this case, I do not know what In-memory means, so I am not using it.
In-Memory will be a database that is not persistent, that is the database is created using in memory rather than as a file, at some time it will be deleted. You probably do not want an in-memory database.
However, as you can see from the code, in the end, data (here, val PREPOPULATE_DATA) is being created again within the code. (In another code, db.execSQL() is used)
This is a common misconception when writing Apps as the onCreate method of an activity is often repeated when an App is running. With an SQLite database the database is created once in it's lifetime, which would be from the very first time the App is run until the database file is deleted. The database will otherwise remain (even between App version changes).
Is there any good way?
You basically have two options for a pre-populated database. They are
to add the data when/after the database is created, as in your example code (which is not a good example as explained below), or
to utilise a pre-packaged database, that is a database that is created outside of the App (typically using an SQlite tool such as DBeaver, Navicat for SQlite, SQLiteStudio, DB Browser for SQLite).
Option 1 -Adding data
If the data should only be added once then using the overridden onCreate method via the CallBack can be used. However, using functions/methods from the #Dao annotated class(es) should not be used. Instead only SupportSQLiteDatabase functions/methods should be used e.g. execSQL (hence why the SupportSQLiteDatabase is passed to onCreate).
This is because at that stage the database has just been created and all the underlying processing has not been completed.
You could protect against duplicating data quite easily by using INSERT OR IGNORE .... rather than INSERT ..... This will skip insertion if there is an applicable constraint violation (rule being broken). As such it relies upon such rules being in force.
The two most commonly used constraints are NOT NULL and UNIQUE, the latter implicitly for a primary key.
In your case if a Data object has just the 2 fields (columns in Database terminology) then, as Room requires a primary key, an implicit UNIQUE constraint applies (could be either column or a composite primary key across both). As such adding Data(1,"val") a second time would result in a constraint violation which would result in either
The row being deleted and another inserted (if INSERT OR REPLACE)
This further complicated by the value of autogenerate.
An exception due to the violation.
The insert being skipped if INSERT OR IGNORE were used.
This option could be suitable for a small amount of data but if over used can start to bloat the code and result in it's maintainability being compromised.
If INSERT or IGNORE were utilised (or alternative checks) then this could, at some additional overhead, even be undertaken in the Callback's onOpen method. This being called every time the database is opened.
Pre-packaged Database
If you have lots of initial data, then creating the database externally, including it as an asset (so it is part of the package that is deployed) and then using Room's .createFromAsset (or the rarer used .createFromFile) would be the way to go.
However, the downfall with this, is that Room expects such a database to comply with the schema that it determines and those expectations are very strict. As such just putting together a database without understanding the nuances of Room then it can be a nightmare.
e.g. SQLite's flexibility allows column types to be virtually anything (see How flexible/restricive are SQLite column types?). Room only allows column types of INTEGER, TEXT, REAL or BLOB. Anything else and the result is an exception with the Expected .... Found ... message.
However, the easy way around this is to let Room tell you what the schema it expects is. To do so you create the #Entity annotated classes (the tables), create the #Database annotated class, including the respective entities in the entities parameter and then compile. In Android Studio's Android View java(generated) will then be visible in the explorer. Within that there will be a class that is the same name as the #Database annotated class but suffixed with _Impl. Within this class there is a function/method createAllTables and it includes execSQL statements for all the tables (the room_master_table should be ignored as Room will always create that itself).
The database, once created and saved, should be copied into the assets folder and using .createFromAsset(????) will then result in the pre-packaged data being from the package to the appropriate local storage location.
In the app I'm working on, we had a complex manual migration that required data parsing, manual SQL commands, etc. This was to convert a List<X> column into a new linked table of X. I've previously written about the approach, but the specific commands are not especially relevant for this question.
The issue I'm encountering is ~1% of users are experiencing a crash as part of this migration. This cannot be reproduced in testing, and due to our table's size, Crashlytics cannot show any useful error:
Losing customer data isn't catastrophic in this context, but being stuck in the current "try migrate, crash, reopen app and repeat" loop is. As such, I want to just give up on the migration and fall back to a destructive migration if we encounter an exception.
Any ideas how this can be done? My current solution is rerunning the DB changes (but not the presumably failing data migration) inside the catch, but this feels very hacky.
Our database is defined as:
Room.databaseBuilder(
context.applicationContext,
CreationDatabase::class.java,
"creation_database"
)
.addMigrations(MIGRATION_11_12, MIGRATION_14_15)
.fallbackToDestructiveMigration()
.build()
where MIGRATION_14_15 is:
private val MIGRATION_14_15 = object : Migration(14, 15) {
override fun migrate(database: SupportSQLiteDatabase) {
try {
// database.execSQL create table etc
} catch (e: Exception) {
e.printStackTrace()
// Here is where I want to give up, and start the DB from scratch
}
}
}
The problem you have is that you cannot (at least easily) invoke the fall-back as that is only invoked when there is no migration.
What you could do is to mimic what fall back does (well close to what it does). That is the fall-back will delete (I think) the database file and create the database from scratch and then invoke the databases _Impl (generated java) createAllTables method.
However, you would likely have issues if you deleted the file as the database connection has been passed to the migration.
So instead you could DROP all the app's tables using the code copied from the dropAllTables method from the generated java. You could then follow this with the code from the createAllTables method.
These methods are in the generated java as the class that is the same as the class that is annotated with #Database suffixed with _Impl.
The gotcha, is that the exception
(Expected .... Found ....) that you have shown is NOT within the migration but after the migration when Room is trying to build the database, so you have no control/place to do the above fall-back mimic unless this was done for all 14-15 migrations.
Perhaps what you could do is to trap the exception, present a dialog requesting the user to uninstall the app and to then re-install. This would then bypass the migration as it would be a fresh install.
I'm currently creating an Android application that uses a prepackaged database. The database is initially around 40MB in size and is stored in the assets/databases folder.
The Problem
When I look up how much space my application is using, the "App Info" page shows the following:
App Size: ~25mb
User Data: ~3mb
Cache: ~45mb
I believe that ROOM is causing this because of the following reasons:
I'm not caching anything in my application (Double checked using the Device File Explorer).
This problem started happening when I migrated from SQLiteOpenHelper to Room
The size of the database is much greater than the App Size, but it is around the same as the Cache size so whatever is stored as Cache must be related to the database.
Whenever I add/remove indices from my Entities, the Cache size changes.
When I clear the cache and open the app, the Cache size reverts back to ~45mb.
Is it normal for Android ROOM to be storing the prepackaged database as Cache? Any help would be much appreciated and let me know if you need more information.
Thank you
EDIT (Additional Information)
This is what I see in the Device File Explorer
And this is how I'm implementing the Room database
#Database(entities = {/* entities*/}, version = 1, exportSchema = false)
public abstract class MyDatabase extends RoomDatabase {
public abstract MyDao myDao();
private static volatile Database INSTANCE;
public static Database getDatabase(final Context context) {
if (INSTANCE == null) {
synchronized (MyDatabase.class) {
if (INSTANCE == null) {
INSTANCE = Room.databaseBuilder(context.getApplicationContext(),
MyDatabase.class, "database")
.createFromAsset("databases/database.db")
.build();
}
}
}
return INSTANCE;
}
}
I believe that what you are experiencing may be due to either:
you using an inMemoryDatabaseBuilder or
that you there is confusion due to Room defaulting to using WAL (Write Ahead Logging) whereas previously you were using Journal Mode.
Most likely the latter.
The difference being that in journal mode a log is kept of the changes made to the database, so the database grows as an when changes are applied and new pages are required. Whereas in WAL mode changes are applied to the -wal file and at points (CHECKPOINTs) the changes are then applied to the database file. So until a CHECKPOINT takes places all of the changes are stored in the -wal file.
e.g.
the -shm file is a WAL file for the WAL file
perhaps refer to https://sqlite.org/wal.html
Swapping to an inMemory database (with no other changes, so inserting exactly the same data) then :-
via Device File Explorer
via App Inspection (aka Database Inspector)
So I suspect that the -wal file is being considered as cache.
I am using Room Persistence Library 1.1.0. I could find the database file at /data/data/<package_name>/databases/ using Android Studio's Device File Explorer.
It contains multiple tables and I can access contents of that tables without any problem using room-DAOs. However when opening with sqlite-browser, is shows no table.
What might be the reason? Is it possible to resolve the issue without switching back to old SQLiteOpenHelper from room?
Solution
To open such databases* with sqlite-browser, you need to copy all three files. All must be in the same directory.
* Databases stored in multiple files as stated in the question.
Why three files?
As per docs, Starting from version 1.1.0, Room uses write-ahead logging as default journal mode for devices which has sufficient RAM and running on API Level 16 or higher. It was Truncate for all devices until this version. write-ahead logging has different internal structure compared to Truncate.
Take a look at the files temporary files used by SQLite now and then :
Until version 1.1.0
From version 1.1.0
If you want to change the journal mode explicitly to Truncate, you can do it this way. But, it is not recommended because WAL is much better compared to Truncate.
public static void initialize(Context context) {
sAppDatabase = Room.databaseBuilder(
context,
AppDatabase.class,
DATABASE_NAME)
.setJournalMode(JournalMode.TRUNCATE).build();
}
Is it possible to move it to single file without changing to Truncate ?
Yes, it is. Query the following statement against the database.
pragma wal_checkpoint(full)
It is discussed in detail here here.
Copy all three files from Device File Explorer in AndroidStudio to your PC directory and open the db file in Db Browser for SQLite (http://sqlitebrowser.org). Make sure all three files are in the same folder.
You can use the wal_checkpoint pragma to trigger a checkpoint which will move the WAL file transactions back into the database.
theRoomDb.query("pragma wal_checkpoint(full)", null)
or
// the result
// contains 1 row with 3 columns
// busy, log, checkpointed
Cursor cursor = theRoomDb.query("pragma wal_checkpoint(full)", null)
See PRAGMA Statements for more details about the pragma parameter values and results.
If the WAL is not enabled the pragma does nothing.
By the way, I tested with Room 1.1.1, and the WAL mode was not used by default, I had to enable it.
Room database Export and Import Solution
Im facing same problem in one of my project, i spend two days to resolve this issue.
Solution
Don't create multiple instance for Room library. Multiple instance creating all the problems.
MyApplication
class MyApplication: Application()
{
companion object {
lateinit var mInstanceDB: AppDatabase
}
override fun onCreate() {
super.onCreate()
mInstanceDB = AppDatabase.getInstance(this)
}
}
AppDatabase
fun getInstance(context: Context): AppDatabase
{
if (sInstance == null) {
sInstance = Room.databaseBuilder(context.applicationContext,AppDatabase::class.java, "database").allowMainThreadQueries().build()
return sInstance!!
}
}
Now use this instance in any number of activity or fragment just like that
{
var allcustomer = MyApplication.mInstanceDB.customerDao.getAll()
}
Export and Import use this library
implementation 'com.ajts.androidmads.sqliteimpex:library:1.0.0'
Github link