When updading DB is it acceptable to run large code to align the DB to my requirements.
For example, I need to alter the table and change column names. Then I need to get all my data in the DB and check if file is located than update the DB accordingly. I need it happen only once when user updates the app to this Room version.
val MIGRATION_8_9 = object : Migration(8, 9) {
override fun migrate(database: SupportSQLiteDatabase) {
database.execSQL("ALTER TABLE RideEntity RENAME videoPresent TO videoState")
GlobalScope.launch(Dispatchers.IO) {
val rides = DataBaseHelper.getAllPartsFromDB() //get all data
rides.forEach {
val path = MyApp.appContext.getExternalFilesDir(null)!!.path + "/" + it.name + "/"
val file = File(path + VIDEO_FILE).exists()
if (file) {
it.videoState = 1
DataBaseHelper.updateData(it) //set the data
}
}
}
}
}
Where:
suspend fun getAllPartsFromDB() = withContext(Dispatchers.IO) {
val parts = db.rideDao().getAllParts()
parts
}
Function:
#Query("SELECT * FROM rideentity ORDER BY time DESC")
fun getAllParts(): List<Parts>
So my question, despite this works, is this way acceptable? And if the migrate function called only once when the app DB updated from version X to Y
Is it acceptable to manage large DB manipulations inside Room migration?
Yes. However you may wish to put the update loop inside a transaction.
And if the migrate function called only once when the app DB updated from version X to Y
Yes it is only called the one time. The Migration(8,9) determines this that is the Migration will only be invoked when the version, as stored in the database header, is 8 and then the version number is set to 9.
Related
I am trying to insert data from a prepopulated database (EnglishVocabs.db) into table "vocab" of my Android app (app_database.db). I am using the following code to perform this operation:
val appDbFile = context.getDatabasePath("app_database.db")
val appdb = SQLiteDatabase.openDatabase(appDbFile.path, null, SQLiteDatabase.OPEN_READWRITE)
val insertUniqueVocabsSqlForAppDb = """
ATTACH '${preDbFile.path}' AS preDb;
INSERT INTO vocab(word, language_id, parts_json)
SELECT DISTINCT B.word, ${Language.ENGLISH.id}, B.parts_json
FROM preDb.EnglishVocabs AS B
WHERE B.word NOT IN (SELECT A.word FROM vocab A);
""".trimIndent()
appdb.beginTransactionWithListener(object : SQLiteTransactionListener {
override fun onBegin() {
Logger.d("on begin")
}
override fun onCommit() {
Logger.d("on commit")
}
override fun onRollback() {
Logger.d("on rollback")
}
})
try {
Logger.d("attached db = ${appdb.attachedDbs}")
val c = appdb.rawQuery(insertUniqueVocabsSqlForAppDb, arrayOf())
appdb.setTransactionSuccessful()
Logger.d("transaction success")
if(c.moveToFirst()){
Logger.d("response = ${c.getStringOrNull(0)}")
}
c.close()
}catch (e: Exception){
Logger.e(e.stackTraceToString())
}finally {
appdb.endTransaction()
appdb.close()
}
I am able to successfully run this code and the onCommit() method of the transaction listener is being called, indicating that the transaction has been committed.
However, when I go to check the app_database.db, the data has not been inserted.
Interestingly, when I copy both the prepopulated and app databases to my PC and run the SQL code using SQLite DB Browser, the data is inserted successfully (40k rows in 200ms). I am not sure what the issue could be in the Android environment. I've grant all necessary permissions.
Can anyone help me understand why this might be happening and how I can fix it?
UPDATE:
I use sqldelight as my app database. and I tried sqlDriver.execute()... too, nothing works
The method rawQuery() is used to return rows and not for INSERT statements.
Instead you should use execSQL() in 2 separate calls:
appdb.execSQL("ATTACH '${preDbFile.path}' AS preDb");
val insertUniqueVocabsSqlForAppDb = """
INSERT INTO vocab(word, language_id, parts_json)
SELECT DISTINCT B.word, ${Language.ENGLISH.id}, B.parts_json
FROM preDb.EnglishVocabs AS B
WHERE B.word NOT IN (SELECT A.word FROM vocab A);
""".trimIndent()
appdb.execSQL(insertUniqueVocabsSqlForAppDb)
I have 2 db files in asset. I have to prepopulate 2 tables
#Database(entities = [Quotes::class, Anime::class], version = 1, exportSchema = true)
Here I have tried something but it isn't working
#Provides
#Singleton
fun provideDatabase(
app: Application,
)= Room.databaseBuilder(app, QuotesDatabase::class.java, "quotes_database")
.createFromAsset("quotess.db")
.createFromAsset("animes.db")
.fallbackToDestructiveMigration()
.build()
You have a few options.
The simplest way would be to combine the two into a single asset. This could be done using one of the SQLite tools (SQliteStudio, DBeaver, Navicat for SQLite, DB Browser for SQLite).
You could, without using createFromAsset allow Room to build the database and the open each asset in turn and copy the data in the onCreate callback. When the onCreate callback is invoked, the database has been created along with the tables and it is passed to the callback as a SupportSQLiteDatabase. You could then copy each asset to a suitable location (databases directory), open the asset as an SQLiteDatabase, for each table, extract the data into a Cursor and then load the data from the Cursor into the SupportSQliteDatabase. You can the close the two SQLiteDatabase and then delete them.
A third option, would be to, prior to building the Room database, create the database (according to the SQL that can be found in the generated java), copying the two asset to a suitable location (databases directory) attaching both to the created database, copying the data from both to the respective tables detach the two asset databases and delete them. Then when building the Room database it will exist and be opened.
I don't believe that you need to, but you may have to set the user version of the created database.
Here's an in-principle (untested) example of the second option that you may find useful.
:-
#Database(entities = [Quotes::class,Anime::class], exportSchema = false, version = 1)
abstract class QuotesDatabase: RoomDatabase() {
abstract fun getAllDao(): AllDao
companion object {
#Volatile
private var instance: QuotesDatabase?=null
private var quotesAssetFileName = "quotess.db" /* CHANGE AS REQUIRED */
private var animeAssetFileName = "animes.db" /* CHANGE AS REQUIRED */
private var quotesTableName = "quotes_table" /* CHANGE AS REQUIRED */
private var animeTablename = "anime_table" /* CHANGE AS REQUIRED */
fun getInstance(context: Context): QuotesDatabase {
if (instance==null) {
instance = Room.databaseBuilder(context,QuotesDatabase::class.java,"quotes_database")
.addCallback(cb)
.build()
}
return instance as QuotesDatabase
}
val cb = object: Callback() {
override fun onCreate(db: SupportSQLiteDatabase) {
super.onCreate(db)
db.beginTransaction() /* MAY NOT BE ABLE TO BE USED - IF NOT REMOVE (as well as similar below)*/
copyAsset(Application().applicationContext, quotesAssetFileName)
val asset1db = SQLiteDatabase.openDatabase(Application().getDatabasePath(quotesAssetFileName).path,null,0)
populateFromCursor(asset1db.rawQuery("SELECT * FROM $quotesTableName",null), quotesTableName,db)
copyAsset(Application().applicationContext, animeAssetFileName)
val asset2db = SQLiteDatabase.openDatabase(Application().getDatabasePath(animeAssetFileName).path,null,0)
populateFromCursor(asset2db.rawQuery("SELECT * FROM $animeTablename",null), animeTablename,db)
db.setTransactionSuccessful() /* MAY NOT BE ABLE TO BE USED - IF NOT REMOVE (as well as similar below)*/
db.endTransaction() /* MAY NOT BE ABLE TO BE USED - IF NOT REMOVE (as well as similar below)*/
deleteAssetCopy(asset1db)
deleteAssetCopy(asset2db)
}
}
/* Populates the Room database using the extracted data (Cursor) from the asset copy database */
#SuppressLint("Range")
fun populateFromCursor(csr: Cursor, tableName: String, db: SupportSQLiteDatabase) {
val cv = ContentValues()
while (csr.moveToNext()) {
for (c in csr.columnNames) {
cv.put(c,"'" + csr.getString(csr.getColumnIndex(c))+"'")
}
db.insert(tableName,OnConflictStrategy.IGNORE,cv)
}
csr.close()
}
/* Copies the asset to the asset database */
fun copyAsset(context: Context, assetFileName: String) {
val asset = context.assets.open(assetFileName)
val db = context.getDatabasePath(assetFileName)
val os: OutputStream = db.outputStream()
asset.copyTo(os)
}
/* Deletes the copied assets database */
fun deleteAssetCopy(db: SQLiteDatabase) {
File(db.path).delete()
}
}
}
I have tried many ways to reset it.
allowBackupandfullBackupOnly had been set to false.
.fallbackToDestructiveMigration()
and delete database and cache files directly.
but it doesn't work.
Simplest way is to uninstall the app, this deletes the database file(s). So rerunning starts from a brand new database.
To use .fallbackToDestructiveMigration() you have to have to invoke a Migration by increasing the version number but NOT have a Migration for the particular path. You could argue that this doesn't reset the database as the newly created database will have the higher version number.
Using clearAllTables doesn't entirely reset the database as it will not delete the system tables. Most notably, sqlite_sequence, which is a table that hold the vale of the latest rowid on a per table basis. That is if you have autogenerate = true in the #PrimaryKey annotation for an field/column that resolves to a column type affinity of INTEGER then AUTOINCREMENT is coded then the sqlite_sequence table will be created (if not already in existence) and store the latest (and therefore highest) value of the said primary key. Thus if you have inserted 100 rows (for example) into a such a table, then after a clearAllTables the 100 will still be stored in the sqlite_sequnce table.
You could also, prior to building the database, delete the database. Here's an example that allows it to be deleted when building :-
#Database(entities = [Customer::class], version = 1)
abstract class CustomerDatabase: RoomDatabase() {
abstract fun customerDao(): CustomerDao
companion object {
private var instance: CustomerDatabase?=null
fun getDatabase(context: Context, resetDatabase: Boolean): CustomerDatabase {
if (resetDatabase && instance == null) {
(context.getDatabasePath("thedatabase.db")).delete()
}
if (instance == null) {
instance = Room.databaseBuilder(context,CustomerDatabase::class.java,"thedatabase.db")
.allowMainThreadQueries()
.build()
}
return instance as CustomerDatabase
}
fun getDatabase(context: Context): CustomerDatabase {
if (instance == null) {
instance = Room.databaseBuilder(context,CustomerDatabase::class.java,"thedatabase.db")
.allowMainThreadQueries()
.build()
}
return instance as CustomerDatabase
}
}
}
note that in addition to requesting the reset, a check is also made to ensure that an instance of the database hasn't been retrieved.
This would also be more efficient as clearAllTables still incurs processing of the underlying data and the ensuing VACUUM which can be quite resource hungry.
You can use this clear all tables
THis deletes all rows from all the tables that are registered to this database as Database.entities().
The Room database has a clearAllTables function that does clearing entities you defined with #Entity annotation. But there is a catch. It does not clear the system generated tables such as sqlite_sequence, which stores the autoincrement values.
But there are more factors to consider. Since clearAllTables itself run in a transaction, we cannot run combination of clearAllTables and clearing sqlite_sequence in a single transaction. If you try to run clearAllTables in a transaction, it will fail with an IllgalStateException.
The android SQLite database library creates an additional table called android_metadata which stores database locale, and the room database library creates another table called room_master_table, which keeps track of database integrity and helps database migrations. We should not delete or clear these two tables. Additionally, SQLite will create a sqlite_sequence table if you have defined autoincrement columns. Deleting this table is not allowed, but clearing this will reset the autoincrement values.
The room database compiler generates the clearAllTables function in the database class. Basically, it disables foreign key constraints, then starts a transaction and clears all rows in tables you have given in the database class, and after the end of the transaction, it re-enables foreign key constraints. See how this is done in the room database compiler source code room / room-compiler / src / main / kotlin / androidx / room / writer / DatabaseWriter.kt / createClearAllTables. The generated function differs based on one factor, whether you have defined foreign key constraints or not.
Based on the compiler source code, I wrote an extension function to reset the database. It will clear all tables you defined and will reset the autoincrement values.
fun RoomDatabase.resetDatabase(tables: List<String>? = null): Boolean {
val db = openHelper.writableDatabase
val tableNames = db.getTableNames()
val hasForeignKeys = db.hasForeignKeys(tables ?: tableNames.minus("sqlite_sequence"))
val supportsDeferForeignKeys = db.supportsDeferForeignKeys()
return try {
if (hasForeignKeys && !supportsDeferForeignKeys) {
// clear enforcement of foreign key constraints.
db.execSQL("PRAGMA foreign_keys = FALSE")
}
db.beginTransaction()
if (hasForeignKeys && supportsDeferForeignKeys) {
// enforce foreign key constraints after outermost transaction is committed.
db.execSQL("PRAGMA defer_foreign_keys = TRUE")
}
// clear all tables including sqlite_sequence table.
// deleting sqlite_sequence table is required to reset autoincrement value.
val tablesToClear = tables?.let {
if (tableNames.contains("sqlite_sequence")) {
it.plus("sqlite_sequence")
} else {
it
}
} ?: tableNames
for (tableName in tablesToClear) {
db.execSQL("DELETE FROM $tableName")
}
db.setTransactionSuccessful()
true
} catch (e: Exception) {
false
} finally {
db.endTransaction()
if (hasForeignKeys && !supportsDeferForeignKeys) {
// restore enforcement of foreign key constraints.
db.execSQL("PRAGMA foreign_keys = TRUE")
}
// blocks until there is no database writer and all are reading from the most recent database snapshot.
db.query("PRAGMA wal_checkpoint(FULL)").close()
if (!db.inTransaction()) {
db.execSQL("VACUUM")
}
}
}
fun SupportSQLiteDatabase.getTableNames(
exclude: List<String> = listOf("android_metadata", "room_master_table")
): List<String> {
val cursor = query("SELECT DISTINCT tbl_name FROM sqlite_master WHERE type='table'")
val tables = mutableListOf<String>()
while (cursor.moveToNext()) {
tables.add(cursor.getString(0))
}
cursor.close()
tables.removeAll(exclude)
return tables
}
fun SupportSQLiteDatabase.hasForeignKeys(tables: List<String>? = null): Boolean {
val tableNames = tables ?: getTableNames(exclude = listOf("android_metadata", "room_master_table", "sqlite_sequence"))
for (tableName in tableNames) {
val cursor = query("PRAGMA foreign_key_list($tableName)")
if (cursor.count > 0) {
cursor.close()
return true
}
cursor.close()
}
return false
}
fun SupportSQLiteDatabase.supportsDeferForeignKeys(): Boolean {
// defer_foreign_keys is only supported on API 21+
// Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP
val cursor = query("PRAGMA defer_foreign_keys")
return cursor.use { it.count > 0 }
}
I'm making an android app with Room database.
My plan is to prepopulate database with some initial data when it is installed on device,
and user can edit it and insert new row on each table.
New row id by users will start from, for example, 10000,
(the point of my question)
and later I want to add more data in the rows up to 9999.
Can I do this when users update the app?
or is there any other way?
Maybe should I try to import csv file to room database?
Thanks!!
my code to prepopulate from an app asset
Room.databaseBuilder(application, AppDatabase::class.java, DB_NAME)
.createFromAsset("database/appdatabase.db")
.build()
To make it so that the users start IF you have #PrimaryKey(autogenerate = true) then when preparing the original pre-populated data you can easily set the next userid to be used.
For example, if the Entity is :-
#Entity
data class User(
#PrimaryKey(autoGenerate = true)
val userId: Long=0,
val userName: String,
)
i.e. userid and userName are the columns and when first running you want the first App provided userid to be 10000 then you could use (as an example) the following in you SQLite Tool:-
CREATE TABLE IF NOT EXISTS `User` (`userId` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, `userName` TEXT);
INSERT INTO User (userName) VALUES('Fred'),('Mary'),('Sarah'); /* Add Users as required */
INSERT INTO User VALUES(10000 -1,'user to be dropped'); /* SETS the next userid value to be 10000 */
DELETE FROM user WHERE userid >= 10000 - 1; /* remove the row added */
Create the table according to the Entity (SQL was copied from the generated java #AppDatabase_Impl)
Loads some users
Add a user with a userId of 9999 (10000 - 1), this causes SQLite to record 9999 in the SQLite system table sqlite_sequnce for the user table.
Remove the user that was added to set the sequence number.
The following, if used after the above, demonstrates the result of doing the above :-
/* JUST TO DEMONSTRATE WHAT THE ABOVE DOES */
/* SHOULD NOT BE RUN as the first App user is added */
SELECT * FROM sqlite_sequence;
INSERT INTO user (username) VALUES('TEST USER FOR DEMO DO NOT ADD ME WHEN PREPARING DATA');
SELECT * FROM user;
The first query :-
i.e. SQLite has stored the value 9999 in the sqlite_sequence table for the table that is named user
The second query shows what happens when the first user is added :-
To recap running 1-4 prepares the pre-populated database so that the first App added user will have a userid of 10000.
Adding new data
You really have to decide how you are going to add the new data. Do you want a csv? Do you want to provide an updated AppDatabase? with all data or with just the new data? Do you need to preserve any existing User/App input data? What about a new installs? Th specifics will very likely matter.
Here's an example of how you could manage this. This uses an updated pre-populated data and assumes that existing data input by the App user is to be kept.
An important value is the 10000 demarcation between supplied userid's and those input via the App being used. As such the User Entity that has been used is:-
#Entity
data class User(
#PrimaryKey(autoGenerate = true)
val userId: Long=0,
val userName: String,
) {
companion object {
const val USER_DEMARCATION = 10000;
}
}
Some Dao's some that may be of use, others used in the class UserDao :-
#Dao
abstract class UserDao {
#Insert(onConflict = OnConflictStrategy.IGNORE)
abstract fun insert(user: User): Long
#Insert(onConflict = OnConflictStrategy.IGNORE)
abstract fun insert(users: List<User>): LongArray
#Query("SELECT * FROM user")
abstract fun getAllUsers(): List<User>
#Query("SELECT * FROM user WHERE userid < ${User.USER_DEMARCATION}")
abstract fun getOnlySuppliedUsers(): List<User>
#Query("SELECT * FROM user WHERE userid >= ${User.USER_DEMARCATION}")
abstract fun getOnlyUserInputUsers(): List<User>
#Query("SELECT count(*) > 0 AS count FROM user WHERE userid >= ${User.USER_DEMARCATION}")
abstract fun isAnyInputUsers(): Long
#Query("SELECT max(userid) + 1 FROM user WHERE userId < ${User.USER_DEMARCATION}")
abstract fun getNextSuppliedUserid(): Long
}
The #Database class AppDatabase :-
#Database(entities = [User::class],version = AppDatabase.DATABASE_VERSION, exportSchema = false)
abstract class AppDatabase: RoomDatabase() {
abstract fun getUserDao(): UserDao
companion object {
const val DATABASE_NAME = "appdatabase.db"
const val DATABASE_VERSION: Int = 2 /*<<<<<<<<<<*/
private var instance: AppDatabase? = null
private var contextPassed: Context? = null
fun getInstance(context: Context): AppDatabase {
contextPassed = context
if (instance == null) {
instance = Room.databaseBuilder(
context,
AppDatabase::class.java,
DATABASE_NAME
)
.allowMainThreadQueries()
.addMigrations(migration1_2)
.createFromAsset(DATABASE_NAME)
.build()
}
return instance as AppDatabase
}
val migration1_2 = object: Migration(1,2) {
val assetFileName = "appdatabase.db" /* NOTE appdatabase.db not used to cater for testing */
val tempDBName = "temp_" + assetFileName
val bufferSize = 1024 * 4
#SuppressLint("Range")
override fun migrate(database: SupportSQLiteDatabase) {
val asset = contextPassed?.assets?.open(assetFileName) /* Get the asset as an InputStream */
val tempDBPath = contextPassed?.getDatabasePath(tempDBName) /* Deduce the file name to copy the database to */
val os = tempDBPath?.outputStream() /* and get an OutputStream for the new version database */
/* Copy the asset to the respective file (OutputStream) */
val buffer = ByteArray(bufferSize)
while (asset!!.read(buffer,0,bufferSize) > 0) {
os!!.write(buffer)
}
/* Flush and close the newly created database file */
os!!.flush()
os.close()
/* Close the asset inputStream */
asset.close()
/* Open the new database */
val version2db = SQLiteDatabase.openDatabase(tempDBPath.path,null,SQLiteDatabase.OPEN_READONLY)
/* Grab all of the supplied rows */
val v2csr = version2db.rawQuery("SELECT * FROM user WHERE userId < ${User.USER_DEMARCATION}",null)
/* Insert into the actual database ignoring duplicates (by userId) */
while (v2csr.moveToNext()) {
database.execSQL("INSERT OR IGNORE INTO user VALUES(${v2csr.getLong(v2csr.getColumnIndex("userId"))},'${v2csr.getString(v2csr.getColumnIndex("userName"))}')",)
}
/* close cursor and the newly created database */
v2csr.close()
version2db.close()
tempDBPath.delete() /* Delete the temporary database file */
}
}
}
testing has been done on the main thread for convenience and brevity hence .allowMainThreadQueries
As can be seen a Migration from 1 to 2 is used this:-
takes the asset appdatabase.db 2nd version (another 3 "supplied" users have been added" using :-
CREATE TABLE IF NOT EXISTS `User` (`userId` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, `userName` TEXT NOT NULL);
INSERT INTO User (userName) VALUES('Fred'),('Mary'),('Sarah'); /* Add Users as required */
INSERT INTO User (userName) VALUES('Tom'),('Elaine'),('Jane'); /*+++++ Version 2 users +++++*/
INSERT INTO User VALUES(10000 -1,'user to be dropped'); /* SETS the next userid value to be 10000 */
DELETE FROM user WHERE userid >= 10000 - 1; /* remove the row added */```
So at first the asset appdatabase.db contains the original data (3 supplied users) and with the sequence number set to 9999.
If the App has database version 1 then this pre-populated database is copied.
Users of the App may add their own and userid's will be assigned 10000, 10001 ...
When the next version is released the asset appdatabase is changed accordingly maintaining the 9999 sequence number ignoring any App input userid's (they aren't known) and the database version is changed from 1 to 2.
The migration1_2 is invoked when the App is updated. If a new user installs the App then the database is created immediately from the asset by Room's createFromAsset.
Can I do this when users update the app? or is there any other way?
As above it can be done when the app is updated AND the database version is increased. It could be done other ways BUT detecting the changed data is what can get complicated.
Maybe should I try to import csv file to room database?
A CSV does not have the advantage of dealing with new installs and inherent version checking.
can I use migration without changing the database schema?
Yes, as the above shows.
Problem - Room DB getting wiped/cleared when doing force update play store update. I am working on a chat messenger application which uses Room DB as local database. Whenever I do a store update with increasing DB version, the local DB gets cleared and messages history are lost.
I'm Using Room DB. My Application is in the Play Store with the use of Room DB and the version is 4.
My Question is I'm changing the 9 tables schema, and now that I update the DB version, each table schema changes. Should I increase the DB version here? How can I accomplish this without losing the user data using Room DB for force update in Play Store? Ex. DB version is 4, I change the two tables’ elements like in the below query.
Do I need to increase DB version twice as two tables are changed or change to one number incremental will be fine? Example: Do I need DB to increase version to 6 OR keeping it 5 is enough?
private val mMigrationMessageStatus: Migration = object : Migration(4, 5) {
override fun migrate(database: SupportSQLiteDatabase) {
database.execSQL("ALTER TABLE message_status RENAME TO MessageStatus")
database.execSQL("ALTER TABLE MessageStatus ADD COLUMN userId TEXT NOT NULL default ''")
}
}
private val mMigrationGroupMember: Migration = object : Migration(4, 5) {
override fun migrate(database: SupportSQLiteDatabase) {
database.execSQL("ALTER TABLE group_member RENAME TO GroupMember")
database.execSQL("ALTER TABLE GroupMember ADD COLUMN userId TEXT NOT NULL default ''")
}
}
return Room.databaseBuilder(context, AppDatabase::class.java, dbName)
.allowMainThreadQueries()
.addMigrations(mMigrationMessageStatus,mMigrationGroupMember)
.build()
From room version 2.4.0, you can easily update using autoMigrations.
DATABASE CLASS
#Database(
version = 3,
autoMigrations = [
AutoMigration(from = 1, to = 2),
AutoMigration(from = 2, to = 3)
],
.....
)
DATA CLASS
#Entity(tableName = "user")
data class DataUser(
....
// I added this column, like this
#ColumnInfo(defaultValue = "")var test: String = ""
)
see reference below
android developer: room version2.4.0
android developer: autoMigration