Develop Reference

Update Local database without affecting the old database while uploading app to playstore

I have 2 fields in the local database(For eg. Name, Password). Now I uploaded the app to the Play Store. After that, I added one field in the database which is mobile number. So now the database has 3 fields(i.e Name, Password, Mobile Number). Now, what happens if I upload this app to the Play Store? Will it affect the database of the old users? How can I update that database without affecting the old local database of the users? I'm using Room Database

The update will be rolled out, via PlayStore, to old users unless it is a different App.
You MUST update the old users otherwise the App will crash. However, you can retain their data but you must cater for the new column.
As the schema has changed (a new column) and if there isn't a migration old users will experience a crash as Room checks to see if the schema, as per the #Entity annotated class (what is expected) against the database (what is found).
The crash would be along the lines of: java.lang.IllegalStateException: Room cannot verify the data integrity. Looks like you've changed schema but forgot to update the version number. You can simply fix this by increasing the version number. Expected identity hash: e843da3b4913dbc08880c558d759fe82, found: d5c32de20cfd495f9eae5463c1ec7433
hashes will differ (expected(1st) is as per the #Entity the found is as per the schema in the existing database)
What you need to do is
set the default value to a suitable value that indicates that no mobile number has been provided, and
add a migration that introduces the new column, and
increase the version number (which will invoke the migration, perform the migration and then processing continues to the check/open of the database).
if there is no Migration then a crash will ensue e.g. java.lang.IllegalStateException: A migration from 1 to 2 was required but not found. Please provide the necessary Migration path via RoomDatabase.Builder.addMigration(Migration ...) or allow for destructive migrations via one of the RoomDatabase.Builder.fallbackToDestructiveMigration* methods.
Demo
The following is a demo that will first create the App with the database at V1 without the Mobile field/column and then will migrate the existing database when the database is upgraded to V2. The existing users will have a value that indicates no mobile.
First the Database code for both versions with the V2 code commented out (The Migration doesn't need to be commented out but would obviously not be present for V1 (just saves having to repeat code)):-
const val DATABASE_VERSION = 1 /*<<<<<<<<<< WILL CHANGE to 2 FOR V2 */
const val USER_TABLE_NAME = "user"
const val USER_NAME_COLUMN = "name"
const val USER_PASSWORD_COLUMN = "password"
#Entity(tableName = USER_TABLE_NAME)
data class User(
#PrimaryKey
#ColumnInfo(name = USER_NAME_COLUMN)
val name: String, /* Original */
#ColumnInfo(name = USER_PASSWORD_COLUMN)
val password: String /* Original */
#Dao
interface UserDAOs {
#Insert(onConflict = OnConflictStrategy.IGNORE)
fun insert(user: User): Long
#Query("SELECT * FROM user")
fun getAllUsers(): List<User>
}
#Database(entities = [User::class], exportSchema = false, version = DATABASE_VERSION)
abstract class TheDatabase: RoomDatabase() {
abstract fun getUserDAOs(): UserDAOs
companion object {
private var instance: TheDatabase?=null
fun getInstance(context: Context): TheDatabase {
if (instance==null) {
instance=Room.databaseBuilder(context,TheDatabase::class.java,"the_database.db")
.allowMainThreadQueries() /* for brevity of the demo */
.build()
}
return instance as TheDatabase
}
}
}
Now some activity code to load some V1 data:-
class MainActivity : AppCompatActivity() {
lateinit var db: TheDatabase
lateinit var dao: UserDAOs
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
db = TheDatabase.getInstance(this)
dao = db.getUserDAOs()
dao.getAllUsers() /*<<<< force open the database in case no code runs (this when the version and schema checking and migration for V2 will take place ) */
if (DATABASE_VERSION == 1) {
dao.insert(User("Fred", "passwordFred")) /* Original */
dao.insert(User("Mary", "passwordMary")) /* Original */
}
/* commented out for V1 as mobile not a field in the User */
/*
if (DATABASE_VERSION == 2) {
dao.insert(User("Jane","passwordJane","1234567890"))
dao.insert(User("John","passwordJohn","0987654321"))
dao.insert(User("Pat","passwordPat"))
}
*/
}
}
When run for a fresh install (aka old user) then the database, via App Inspection:-
room_master_table is where the schema hash is stored and will be the found
as expected the two rows exist and have expected values.
Next the code is changed.
The database code becomes:-
The Database version is increased:-
const val DATABASE_VERSION = 2 /*<<<<<<<<<< WILL CHANGE to 2 FOR V2 */
2 new const vals are added:-
const val USER_MOBILE_COLUMN = "mobile" /*<<<<<<<<<< ADDED for V2 */
const val USER_MOBILE_DEFAULT_VALUE = "xxxxxxxxxx" /*<<<<<<<<<< ADDED for V2 */
The User class becomes:-
#Entity(tableName = USER_TABLE_NAME)
data class User(
#PrimaryKey
#ColumnInfo(name = USER_NAME_COLUMN)
val name: String, /* Original */
#ColumnInfo(name = USER_PASSWORD_COLUMN)
val password: String /* Original */ ,/*<<<<<<<<< ADDED comma FOR V2 */
/*<<<<<<<<<< SCHEMA CHANGES FOR V2 (see comma above) >>>>>>>>>>*/
#ColumnInfo(name = USER_MOBILE_COLUMN, defaultValue = USER_MOBILE_DEFAULT_VALUE) /*<<<<<<<<<< ADDED FOR V2 */
val mobile: String = "not provided" /*<<<<<<<<<< ADDED for V2 (default value allows mobile to not be given for V1 code in Main Activity)*/
)
The #Database annotated class TheDatabase has the migration added:-
#Database(entities = [User::class], exportSchema = false, version = DATABASE_VERSION)
abstract class TheDatabase: RoomDatabase() {
abstract fun getUserDAOs(): UserDAOs
companion object {
private var instance: TheDatabase?=null
fun getInstance(context: Context): TheDatabase {
if (instance==null) {
instance=Room.databaseBuilder(context,TheDatabase::class.java,"the_database.db")
.allowMainThreadQueries() /* for brevity of the demo */
.addMigrations(MIGRATE_1_to_2)
.build()
}
return instance as TheDatabase
}
val MIGRATE_1_to_2: Migration = object: Migration(1,2){
override fun migrate(db: SupportSQLiteDatabase) {
db.execSQL("ALTER TABLE $USER_TABLE_NAME ADD COLUMN $USER_MOBILE_COLUMN TEXT NOT NULL DEFAULT '$USER_MOBILE_DEFAULT_VALUE'")
/* So as to show Migration add a row when migrating (would not be done normally) */
val cv = ContentValues()
cv.put(USER_NAME_COLUMN,"Alice")
cv.put(USER_PASSWORD_COLUMN,"passwordAlice")
cv.put(USER_MOBILE_COLUMN,"1111111111")
db.insert(USER_TABLE_NAME,OnConflictStrategy.IGNORE,cv)
}
}
}
}
The commented out activity code is un-commented for V2:-
class MainActivity : AppCompatActivity() {
lateinit var db: TheDatabase
lateinit var dao: UserDAOs
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
db = TheDatabase.getInstance(this)
dao = db.getUserDAOs()
dao.getAllUsers() /*<<<< force open the database in case no code runs */
if (DATABASE_VERSION == 1) {
dao.insert(User("Fred", "passwordFred")) /* Original */
dao.insert(User("Mary", "passwordMary")) /* Original */
}
/* commented out for V1 as mobile not a field in the User */
if (DATABASE_VERSION == 2) {
dao.insert(User("Jane","passwordJane","1234567890"))
dao.insert(User("John","passwordJohn","0987654321"))
dao.insert(User("Pat","passwordPat"))
}
}
}
When the App is run then App Inspection now shows:-
As can be seen:-
Fred and Mary have have the recognisable indicator that the mobile wasn't provided i.e. it is xxxxxxxxxx
Alice has been added as part of the Migration (not that this would normally be included, it is just to show that the migration was performed)
Jane and John have been added with their provided mobile numbers
Pat has been added with the default value, as per the field default value (the database default value cannot be applied as mobile is not nullable)
Final Test
The remaining proof of concept, is when a new user installs the App i.e. a fresh/new install. In this scenario , for the demo, just the three V2 users will be inserted (Jane, John and Pat):-
Obviously the inserts are reflecting what the App user may do

How to export Room Database to .CSV

How can I export my Room Database to a .CSV file. I would like it to be saved to device storage. I searched everything and no answer was suitable. I hope there is a way for this.

You cannot just save a database as a CSV. However the database, if fully checkpointed, is just a file. If not fully checkpointed then it (unless write-ahead logging as been disabled) would be three files.
The database itself consists of various parts, a header (first 100 bytes of the file) and then blocks of data for the various components. Most of these dependant upon the schema (the tables), there are also system tables
sqlite_master is a table that holds the schema
if autogenerate = true is used for a integer type primary key then there is also the sqlite_sequence table
room itself has the room_master_table in which room stores a hash, this being compared against a compiled hash based upon the Room's expected schema.
To save all that data as a CSV, would be complex (and needless as you can just copy the database file(s)).
If what you want is a CSV of the app's data, then that would depend upon the tables. If you a single table then extracting the data as a CSV would be relatively simple but could be complicated if the data includes commas.
If there are multiple tables, then you would have to distinguish the data for the tables.
Again the simplest way, if just securing the data is to copy the file.
However as an example based upon :-
A database that has 3 tables (apart from the system tables)
PostDataLocal (see below for columns)
GroupDataLocal
AdminDataLocal
an existing answer has been adapted for the example
Then:-
The following in an #Dao annotated interface (namely AllDao) :-
#Query("SELECT postId||','||content FROM postDataLocal")
fun getPostDataLocalCSV(): List<String>
#Query("SELECT groupPostIdMap||','||groupId||','||groupName FROM groupDataLocal")
fun getGroupDataLocalCSV(): List<String>
#Query("SELECT adminGroupIdMap||','||userId||','||adminName||','||avatar FROM adminDataLocal")
fun getAdminDataLocalCSV(): List<String>
And the following function where dao is an AllDao instance previously instantiated :-
private fun createCSV() {
val sb = StringBuilder()
var afterFirst = false
sb.append("{POSTDATALOCAL}")
for (s in dao.getPostDataLocalCSV()) {
if(afterFirst) sb.append(",")
afterFirst = true
sb.append(s)
}
afterFirst = false
sb.append("{GROUPDATALOCAL}")
for (s in dao.getGroupDataLocalCSV()) {
if (afterFirst) sb.append(",")
afterFirst = true
sb.append(s)
}
afterFirst = false
sb.append("{ADMINDATALOCAL}")
for (s in dao.getAdminDataLocalCSV()) {
if ((afterFirst)) sb.append(",")
afterFirst = true
sb.append(s)
}
Log.d("CSV_DATA","CSV is :-\n\t$sb")
}
And then in an activity (where dao has been instantiated) the following:-
createCSV()
Then, when the database contains the following data (extracted via App Inspection) :-
PostDataLocal
GroupDataLocal
AdminDataLocal
The result written to the log (as could be written to a file rather than the log) is :-
D/CSV_DATA: CSV is :-
{POSTDATALOCAL}1,Post001,2,Post002,3,Post003{GROUPDATALOCAL}1,1,Group001 (Post001),1,2,Group002 (Post001),1,3,Group003 (Post001),2,4,Group004 (Post002),2,5,Group005 (Post002),3,6,Group006 (Post003){ADMINDATALOCAL}1,1,Admin001,admin001.gif,1,2,Admin002,admin002.gif,1,3,Admin003,admin003.gif,2,4,Admin004,admin004.gif,2,5,Admin005,admin005.gif,3,6,Admin006,admin006.gif,4,7,Admin007,admin007.gif,5,8,Admin008,admin008.gif,6,9,Admin009,admin009.gif,6,10,Admin010,admin010.gif
Note how headers have been included to distinguish between the tables
of course no consideration has been given to the inclusion of commas in the data (the above is intended to just show that in-principle you can generate a CSV representation of the data relatively easily)
Additional
Here's a more automated version in which you don't need to create the #Query annotated functions, rather it interrogates sqlite_master to extract the tables and the uses the table_info pragma to ascertain the columns, building the respective SQL.
As such it should cater for any Room database.
It also allows for the replacement of commas in the data with an indicator of a comma that could then be replaced when processing the CSV.
The supportive (secondary/invoked by the primary) function being :-
private fun getTableColumnNames(tableName: String, suppDB: SupportSQLiteDatabase): List<String> {
val rv = arrayListOf<String>()
val csr = suppDB.query("SELECT name FROM pragma_table_info('${tableName}')",null)
while (csr.moveToNext()) {
rv.add(csr.getString(0))
}
csr.close()
return rv.toList()
}
And the Primary function :-
private fun AutoCreateCSV(): String {
val replaceCommaInData = "{COMMA}" /* commas in the data will be replaced by this */
val rv = StringBuilder()
val sql = StringBuilder()
var afterFirstTable = false
var afterFirstColumn = false
var afterFirstRow = false
val suppDb = db.getOpenHelper().writableDatabase
var currentTableName: String = ""
val csr = db.query("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE('sqlite_%') AND name NOT LIKE('room_%') AND name NOT LIKE('android_%')", null)
while (csr.moveToNext()) {
sql.clear()
sql.append("SELECT ")
currentTableName = csr.getString(0)
if (afterFirstTable) rv.append(",")
afterFirstTable = true
afterFirstColumn = false
rv.append("{$currentTableName},")
for (columnName in getTableColumnNames(currentTableName,suppDb)) {
if (afterFirstColumn) sql.append("||','||")
afterFirstColumn = true
sql.append("replace(`$columnName`,',','$replaceCommaInData')")
}
sql.append(" FROM `${currentTableName}`")
val csr2 = db.query(sql.toString(),null)
afterFirstRow = false
while (csr2.moveToNext()) {
if (afterFirstRow) rv.append(",")
afterFirstRow = true
rv.append(csr2.getString(0))
}
csr2.close()
}
csr.close()
return rv.toString()
}
Using the same data and as the primary function returns a String the following code Log.d("CSV_DATA2",AutoCreateCSV()) results in :-
D/CSV_DATA2: {PostDataLocal},1,Post001,2,Post002,3,Post003,{GroupDataLocal},1,1,Group001 (Post001),1,2,Group002 (Post001),1,3,Group003 (Post001),2,4,Group004 (Post002),2,5,Group005 (Post002),3,6,Group006 (Post003),{AdminDataLocal},1,1,Admin001,admin001.gif,1,2,Admin002,admin002.gif,1,3,Admin003,admin003.gif,2,4,Admin004,admin004.gif,2,5,Admin005,admin005.gif,3,6,Admin006,admin006.gif,4,7,Admin007,admin007.gif,5,8,Admin008,admin008.gif,6,9,Admin009,admin009.gif,6,10,Admin010,admin010.gif
and if the data includes a comma e.g. Post001 is changed to be the value Post001, <<note the comma in the data>>
Then :-
D/CSV_DATA2: {PostDataLocal},1,Post001{COMMA} <<note the comma in the data>>,2,Post002,3 ....
this additional solution also fixes a little bug in the first where some separating commas were omitted between the header and the data.

Get all your data as a list from room and use this library
https://github.com/doyaaaaaken/kotlin-csv
It works well, here is my usage
private fun exportDatabaseToCSVFile(context: Context, list: List<AppModel>) {
val csvFile = generateFile(context, getFileName())
if (csvFile != null) {
exportDirectorsToCSVFile(csvFile, list)
} else {
//
}
}
private fun generateFile(context: Context, fileName: String): File? {
val csvFile = File(context.filesDir, fileName)
csvFile.createNewFile()
return if (csvFile.exists()) {
csvFile
} else {
null
}
}
private fun getFileName(): String = "temp.csv"
fun exportDirectorsToCSVFile(csvFile: File, list: List<AppModel>) {
csvWriter().open(csvFile, append = false) {
// Header
writeRow(listOf("row1", "row2", "row3"))
list.forEachIndexed { index, appModel ->
writeRow(listOf(getRow1, getRow2, getRow3))
}
shareCsvFile(csvFile)
}
}
private fun shareCsvFile(csvFile: File) {
// share your file, don't forget adding provider in your Manifest
}

What happens when room android db gets corrupted?

In a large scale app, there are chances of my db file (created through android room lib), getting corrupted. For such a user whats the fallback ?
Will room delete the db file and re-create from scratch in production mode or will I have to handle that myself ?

For such a user whats the fallback ?
Backups and restore, noting that backups should not be taken when there are any open transactions. It is best to ensure that if in WAL mode (which is the default mode with room) that the database is fully committed (i.e. the WAL file is empty or doesn't exist).
The backup could be a simple file copy or you can use the VACUUM INTO the latter having the advantage of potentially freeing space but the disadvantage is that it could be more resource intensive.
You could maintain additional databases where changes are applied to other databases as and when the main database is changed. Obviously there would be an overhead. This would have the advantage that if corruption were to occur that it would be less likely to occur in the other databases.
You could maintain a log that allowed roll back and roll forward of changes. The latter being used to roll forward from a backup to the point of corruption or close to the point of corruption.
Will room delete the db file and re-create from scratch in production mode or will I have to handle that myself ?
if detected (when opening) then yes :-
onCorruption
The method invoked when database corruption is detected. Default implementation will delete the database file.
https://developer.android.com/reference/androidx/sqlite/db/SupportSQLiteOpenHelper.Callback#onCorruption(androidx.sqlite.db.SupportSQLiteDatabase)
What happens when room android db gets corrupted?
here's an example of file corruption
2021-10-27 10:36:19.281 7930-7930/a.a.so69722729kotlinroomcorrupt E/SQLiteLog: (26) file is not a database
2021-10-27 10:36:19.285 7930-7930/a.a.so69722729kotlinroomcorrupt E/SupportSQLite: Corruption reported by sqlite on database: /data/user/0/a.a.so69722729kotlinroomcorrupt/databases/thedatabase
2021-10-27 10:36:19.286 7930-7930/a.a.so69722729kotlinroomcorrupt W/SupportSQLite: deleting the database file: /data/user/0/a.a.so69722729kotlinroomcorrupt/databases/thedatabase
2021-10-27 10:36:19.306 7930-7930/a.a.so69722729kotlinroomcorrupt D/TAG: onCreate Invoked.
2021-10-27 10:36:19.312 7930-7930/a.a.so69722729kotlinroomcorrupt D/TAG: onOpen Invoked.
As can be seen by logging from the callbacks that after the file is deleted that onCreate is invoked thus creating a new empty database.
The code used for the above, which you may wish to adapt for testing, is :-
A simple #Entity Something :-
#Entity
data class Something(
#PrimaryKey
val id: Long?=null,
val something: String
)
A Simple #Dao AllDao
#Dao
abstract class AllDao {
#Insert
abstract fun insert(something: Something)
#Query("SELECT * FROM something")
abstract fun getAllFromSomething(): List<Something>
}
The #Database TheDatabase
#Database(entities = [Something::class],version = 1)
abstract class TheDatabase: RoomDatabase() {
abstract fun getAllDao(): AllDao
companion object {
const val DATABASENAME = "thedatabase"
const val TAG = "DBINFO"
private var instance: TheDatabase? = null
var existed: Boolean = false
fun getInstance(context: Context): TheDatabase {
existed = exists(context)
if (exists(context)) {
Log.d(TAG,"Database exists so corrupting it before room opens the database.")
corruptDatabase(context)
}
if (instance == null) {
instance = Room.databaseBuilder(context,TheDatabase::class.java, DATABASENAME)
.allowMainThreadQueries()
.addCallback(cb)
.build()
}
instance!!.openHelper.writableDatabase // Force open
return instance as TheDatabase
}
object cb: Callback() {
override fun onCreate(db: SupportSQLiteDatabase) {
super.onCreate(db)
Log.d("TAG","onCreate Invoked.")
}
override fun onDestructiveMigration(db: SupportSQLiteDatabase) {
super.onDestructiveMigration(db)
Log.d("TAG","onDestructiveMigration Invoked.")
}
override fun onOpen(db: SupportSQLiteDatabase) {
super.onOpen(db)
Log.d("TAG","onOpen Invoked.")
}
}
fun exists(context: Context): Boolean {
val db = File(context.getDatabasePath(DATABASENAME).path)
return db.exists()
}
/**
* Corrupt the database by
* copying the file via buffered reads and write
* BUT only write part of a buffer
* AND skip the 3rd block
* Furthermore, delete the -wal file (possible that this )
* Note that often it is the -wal file deletion that corrupts
*/
fun corruptDatabase(context: Context) {
Log.d("TAG","corruptDatabase Invoked.")
var db: File = File(context.getDatabasePath(DATABASENAME).path)
logFileInfo(db,"Initial")
var tempdb = File(context.getDatabasePath("temp" + DATABASENAME).path)
logFileInfo(tempdb,"Initial")
val blksize = 128
var buffer = ByteArray(blksize)
var i: InputStream
var o: FileOutputStream
try {
i = FileInputStream(db)
o = FileOutputStream(tempdb)
var blocks = 0;
var writes = 0;
while (i.read(buffer) > 0) {
if(blocks++ % 2 == 1) {
writes++
o.write(buffer,buffer.size / 4,buffer.size / 4)
}
}
Log.d(TAG,"${blocks} Read ${writes} Written")
o.flush()
o.close()
i.close()
db = File(context.getDatabasePath(DATABASENAME).path)
logFileInfo(db,"After copy")
tempdb = File(context.getDatabasePath("temp${DATABASENAME}").path)
logFileInfo(tempdb,"After copy")
} catch (e: IOException) {
e.printStackTrace()
}
db.delete()
//(context.getDatabasePath(DATABASENAME+"-wal")).delete()
logFileInfo(db,"After delete")
tempdb.renameTo(context.getDatabasePath(DATABASENAME))
logFileInfo(tempdb,"After rename")
logFileInfo(context.getDatabasePath(DATABASENAME),"After rename/new file")
}
fun logFileInfo(file: File, prefix: String) {
Log.d(TAG,"${prefix} FileName is ${file.name}\n\tpath is ${file.path}\n\tsize is ${file.totalSpace} frespace is ${file.freeSpace}")
}
}
}
Finally an invoking Activity MainActivity
class MainActivity : AppCompatActivity() {
lateinit var db: TheDatabase
lateinit var dao: AllDao
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
db = TheDatabase.getInstance(this);
dao = db.getAllDao()
dao.insert(Something(something = "Something1 " + TheDatabase.existed))
dao.insert(Something(something = "Something2 " + TheDatabase.existed))
for(s: Something in dao.getAllFromSomething()) {
Log.d(TheDatabase.TAG,"Something with ID " + s.id + " is " + s.something)
}
}
}
Note it's the -wal deletion that corrupts for the database above. It is quite resilient in regards to blocks being deleted removed, at least for smaller databases where the WAL file is of sufficient size to recover as such by applying the changes stored within. However, testing based upon the above but with a second table and 100000 rows inserted, then the partial block writes and missed block writes do corrupt the database.

How to add new data to android room database when updating app?

I'm making an android app with Room database.
My plan is to prepopulate database with some initial data when it is installed on device,
and user can edit it and insert new row on each table.
New row id by users will start from, for example, 10000,
(the point of my question)
and later I want to add more data in the rows up to 9999.
Can I do this when users update the app?
or is there any other way?
Maybe should I try to import csv file to room database?
Thanks!!
my code to prepopulate from an app asset
Room.databaseBuilder(application, AppDatabase::class.java, DB_NAME)
.createFromAsset("database/appdatabase.db")
.build()

To make it so that the users start IF you have #PrimaryKey(autogenerate = true) then when preparing the original pre-populated data you can easily set the next userid to be used.
For example, if the Entity is :-
#Entity
data class User(
#PrimaryKey(autoGenerate = true)
val userId: Long=0,
val userName: String,
)
i.e. userid and userName are the columns and when first running you want the first App provided userid to be 10000 then you could use (as an example) the following in you SQLite Tool:-
CREATE TABLE IF NOT EXISTS `User` (`userId` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, `userName` TEXT);
INSERT INTO User (userName) VALUES('Fred'),('Mary'),('Sarah'); /* Add Users as required */
INSERT INTO User VALUES(10000 -1,'user to be dropped'); /* SETS the next userid value to be 10000 */
DELETE FROM user WHERE userid >= 10000 - 1; /* remove the row added */
Create the table according to the Entity (SQL was copied from the generated java #AppDatabase_Impl)
Loads some users
Add a user with a userId of 9999 (10000 - 1), this causes SQLite to record 9999 in the SQLite system table sqlite_sequnce for the user table.
Remove the user that was added to set the sequence number.
The following, if used after the above, demonstrates the result of doing the above :-
/* JUST TO DEMONSTRATE WHAT THE ABOVE DOES */
/* SHOULD NOT BE RUN as the first App user is added */
SELECT * FROM sqlite_sequence;
INSERT INTO user (username) VALUES('TEST USER FOR DEMO DO NOT ADD ME WHEN PREPARING DATA');
SELECT * FROM user;
The first query :-
i.e. SQLite has stored the value 9999 in the sqlite_sequence table for the table that is named user
The second query shows what happens when the first user is added :-
To recap running 1-4 prepares the pre-populated database so that the first App added user will have a userid of 10000.
Adding new data
You really have to decide how you are going to add the new data. Do you want a csv? Do you want to provide an updated AppDatabase? with all data or with just the new data? Do you need to preserve any existing User/App input data? What about a new installs? Th specifics will very likely matter.
Here's an example of how you could manage this. This uses an updated pre-populated data and assumes that existing data input by the App user is to be kept.
An important value is the 10000 demarcation between supplied userid's and those input via the App being used. As such the User Entity that has been used is:-
#Entity
data class User(
#PrimaryKey(autoGenerate = true)
val userId: Long=0,
val userName: String,
) {
companion object {
const val USER_DEMARCATION = 10000;
}
}
Some Dao's some that may be of use, others used in the class UserDao :-
#Dao
abstract class UserDao {
#Insert(onConflict = OnConflictStrategy.IGNORE)
abstract fun insert(user: User): Long
#Insert(onConflict = OnConflictStrategy.IGNORE)
abstract fun insert(users: List<User>): LongArray
#Query("SELECT * FROM user")
abstract fun getAllUsers(): List<User>
#Query("SELECT * FROM user WHERE userid < ${User.USER_DEMARCATION}")
abstract fun getOnlySuppliedUsers(): List<User>
#Query("SELECT * FROM user WHERE userid >= ${User.USER_DEMARCATION}")
abstract fun getOnlyUserInputUsers(): List<User>
#Query("SELECT count(*) > 0 AS count FROM user WHERE userid >= ${User.USER_DEMARCATION}")
abstract fun isAnyInputUsers(): Long
#Query("SELECT max(userid) + 1 FROM user WHERE userId < ${User.USER_DEMARCATION}")
abstract fun getNextSuppliedUserid(): Long
}
The #Database class AppDatabase :-
#Database(entities = [User::class],version = AppDatabase.DATABASE_VERSION, exportSchema = false)
abstract class AppDatabase: RoomDatabase() {
abstract fun getUserDao(): UserDao
companion object {
const val DATABASE_NAME = "appdatabase.db"
const val DATABASE_VERSION: Int = 2 /*<<<<<<<<<<*/
private var instance: AppDatabase? = null
private var contextPassed: Context? = null
fun getInstance(context: Context): AppDatabase {
contextPassed = context
if (instance == null) {
instance = Room.databaseBuilder(
context,
AppDatabase::class.java,
DATABASE_NAME
)
.allowMainThreadQueries()
.addMigrations(migration1_2)
.createFromAsset(DATABASE_NAME)
.build()
}
return instance as AppDatabase
}
val migration1_2 = object: Migration(1,2) {
val assetFileName = "appdatabase.db" /* NOTE appdatabase.db not used to cater for testing */
val tempDBName = "temp_" + assetFileName
val bufferSize = 1024 * 4
#SuppressLint("Range")
override fun migrate(database: SupportSQLiteDatabase) {
val asset = contextPassed?.assets?.open(assetFileName) /* Get the asset as an InputStream */
val tempDBPath = contextPassed?.getDatabasePath(tempDBName) /* Deduce the file name to copy the database to */
val os = tempDBPath?.outputStream() /* and get an OutputStream for the new version database */
/* Copy the asset to the respective file (OutputStream) */
val buffer = ByteArray(bufferSize)
while (asset!!.read(buffer,0,bufferSize) > 0) {
os!!.write(buffer)
}
/* Flush and close the newly created database file */
os!!.flush()
os.close()
/* Close the asset inputStream */
asset.close()
/* Open the new database */
val version2db = SQLiteDatabase.openDatabase(tempDBPath.path,null,SQLiteDatabase.OPEN_READONLY)
/* Grab all of the supplied rows */
val v2csr = version2db.rawQuery("SELECT * FROM user WHERE userId < ${User.USER_DEMARCATION}",null)
/* Insert into the actual database ignoring duplicates (by userId) */
while (v2csr.moveToNext()) {
database.execSQL("INSERT OR IGNORE INTO user VALUES(${v2csr.getLong(v2csr.getColumnIndex("userId"))},'${v2csr.getString(v2csr.getColumnIndex("userName"))}')",)
}
/* close cursor and the newly created database */
v2csr.close()
version2db.close()
tempDBPath.delete() /* Delete the temporary database file */
}
}
}
testing has been done on the main thread for convenience and brevity hence .allowMainThreadQueries
As can be seen a Migration from 1 to 2 is used this:-
takes the asset appdatabase.db 2nd version (another 3 "supplied" users have been added" using :-
CREATE TABLE IF NOT EXISTS `User` (`userId` INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, `userName` TEXT NOT NULL);
INSERT INTO User (userName) VALUES('Fred'),('Mary'),('Sarah'); /* Add Users as required */
INSERT INTO User (userName) VALUES('Tom'),('Elaine'),('Jane'); /*+++++ Version 2 users +++++*/
INSERT INTO User VALUES(10000 -1,'user to be dropped'); /* SETS the next userid value to be 10000 */
DELETE FROM user WHERE userid >= 10000 - 1; /* remove the row added */```
So at first the asset appdatabase.db contains the original data (3 supplied users) and with the sequence number set to 9999.
If the App has database version 1 then this pre-populated database is copied.
Users of the App may add their own and userid's will be assigned 10000, 10001 ...
When the next version is released the asset appdatabase is changed accordingly maintaining the 9999 sequence number ignoring any App input userid's (they aren't known) and the database version is changed from 1 to 2.
The migration1_2 is invoked when the App is updated. If a new user installs the App then the database is created immediately from the asset by Room's createFromAsset.
Can I do this when users update the app? or is there any other way?
As above it can be done when the app is updated AND the database version is increased. It could be done other ways BUT detecting the changed data is what can get complicated.
Maybe should I try to import csv file to room database?
A CSV does not have the advantage of dealing with new installs and inherent version checking.
can I use migration without changing the database schema?
Yes, as the above shows.

Room db migration fallbackToDestructiveMigration() not working

I am using Room with a prepopulated database in the assets folder. For an app update, I would like to alter this database by adding a new column and prepopulating this column with new data.
The database was auto-migrated from version 1 to 2 (a table was added). From version 2 to 3, I would now like to apply abovementioned changes by providing a different 'database.db' file in the assets folder and allowing for destructive migration.
#Database(entities = [Object1::class, Object2::class], version = 3, autoMigrations = [
AutoMigration (from = 1, to = 2)], exportSchema = true)
abstract class AppDatabase : RoomDatabase() {
abstract fun dao(): Dao
companion object {
private const val DB_NAME = "database.db"
#Volatile
private var instance: AppDatabase? = null
fun getInstance(context: Context): AppDatabase {
return instance ?: synchronized(this) {
instance ?: buildDatabase(context).also { instance = it }
}
}
private fun buildDatabase(context: Context): AppDatabase {
return Room.databaseBuilder(
context,
AppDatabase::class.java, "AppDB.db")
.fallbackToDestructiveMigration()
.createFromAsset(DB_NAME)
.build()
}
}
}
The problem is that I still get the following exception:
java.lang.IllegalStateException: A migration from 1 to 3 was required but not found. Please provide the necessary Migration path via RoomDatabase.Builder.addMigration(Migration ...) or allow for destructive migrations via one of the RoomDatabase.Builder.fallbackToDestructiveMigration* methods.
I am unsure why this would still happen. I thought it was either providing a migration script or allowing for destructive migration that makes the migration work.
Added Comment:-
I have tried an implemented migration, but the same exception as above happened again. When I try starting over with versionCode 1, I am getting "java.lang.IllegalStateException: Room cannot verify the data integrity. Looks like you've changed schema but forgot to update the version number. You can simply fix this by increasing the version number." I have also changed the database name and added android:allowBackup="false" in the manifest.
Any ideas?

I had problems using fallbackToDestructiveMigration and createFromAsset together. I would like to share my experience because it took me hours to find it. When you provide an asset db, you have to update the user version pragma of the default database file that you are providing with createFromAsset. If not, you always lose the data that you insert while the app is working.

I finally figured out what the problem was, it had nothing to do with the versioning or anything else related to room or the asset db file.
It was dependency injection.
I provided my database to Dagger in a DatabaseModule class as follows:
private const val DB_NAME = "database.db"
#InstallIn(SingletonComponent::class)
#Module
class DatabaseModule {
#Provides
fun provideDao(appDatabase: AppDatabase): Dao {
return appDatabase.dao()
}
#Provides
#Singleton
fun provideAppDatabase(#ApplicationContext appContext: Context): AppDatabase {
return Room.databaseBuilder(
appContext,
AppDatabase::class.java, "AppDB.db")
.createFromAsset(DB_NAME)
.build()
}
}
It was missing the fallBackToDestructiveMigration() call, so this messed up Room's internal onUpgrade call in RoomOpenHelper.java.
To fix it, I made my buildDatabase call in AppDatabase public and used it to provide the database to Dagger in the DatabaseModule class.

Digging through the room documentation doesn't turn much up, my hunch is that it has to do with the fact that you are using Automigrations instead of implemented migrations. Have you tried changing that Automigration from 1->2 to an implemented migration?
Also, since you are manually replacing it with a new database that has prepopulated data my solution would be to just get rid of the old migrations, change the name of the DB slightly and start over from version 1. There's no reason to maintain the old migrations if anyone going from older versions to the current version are having their DB deleted.

After extensive methodical testing, the only way that I can replicate your (1-3 required) failure is by excluding fallbackToDestructiveMigation. In which case the exception happens if the migration is from 1 to 3 or the migration is 3 to 1 (i.e. Asset Version at 3 but Room version at 1)
as per the spreadsheet screenshot below
1-3 exception when AssetDB Version =3 Database Version = 1 Room Version = 3
also 3-1 exception when AssetDB Version =3 Database Version = -1 Room Version = 1
-1 version means file does not exist (i.e. initial install)
I suspect that you have somehow inadvertently introduced one of the above two scanrios. What I haven't tested is alternative Room library versions. The above was tested with 2.4.0-alpha04 as per :-
implementation 'androidx.core:core-ktx:1.6.0'
implementation 'androidx.appcompat:appcompat:1.3.1'
implementation 'com.google.android.material:material:1.4.0'
implementation 'androidx.constraintlayout:constraintlayout:2.1.0'
implementation 'androidx.room:room-ktx:2.4.0-alpha04'
implementation 'androidx.room:room-runtime:2.4.0-alpha04'
testImplementation 'junit:junit:4.+'
androidTestImplementation 'androidx.test.ext:junit:1.1.3'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.4.0'
kapt 'androidx.room:room-compiler:2.4.0-alpha0
For the testing, I had two copies of the asset file, one at version 1 the other at version 2 (v1dbbase.db and v3dbbase.db), the data in a common column indicating the if the data was for version3. The actual asset file used was deleted before a test and the appropriate version copied and pasted to database.db
I had the two entities Object1 and Object2 and could comment in or out an extra column in either. e.g.:-
/*TESTING INCLUDE FOR V2+ >>>>>*///, #ColumnInfo(name = COL_EXTRAV2, defaultValue = "x") val object1_extra: String
- as above it is excluded
/*TESTING INCLUDE FOR V2+ >>>>>*/, #ColumnInfo(name = COL_EXTRAV2, defaultValue = "x") val object1_extra: String
- with the two //'s before the comma now included
both the extra columns commented out = Version 1
Object1's extra column included = Version 3
Object1's and Object2's extra column included = Version 3
Object2's extra column included but not Object1's was not considered.
A few constants were added to cater for logging.
Additionally to cater for logging a callback function was added (.addCallback) and onOpen, onCreate and onDestructiveMigration were all overridden to log the Room Version and Database Version.
To further enhance the logging, two functions were added, to get the version from the sqlite database header. One for the asset file, the other for the database. The functions being called/invoked BEFORE the database build.
To run a test it meant:-
Ensuring that the device had the App at the appropriate level.
Deleting the database.db asset
Copying and pasting the appropriate asset file as database.db (from either v1dbbase.db or v3dbbase.db)
Amending the Object1 class to include/exclude the extra column (as explained above)
Amending the Object2 class to include/exclude the extra columns (as explained above)
Amended the Room Version to the appropriate level.
The code used for testing:-
Object1
#Entity(tableName = TABLE_NAME)
data class Object1(
#PrimaryKey
#ColumnInfo(name = COL_ID)
val object1_id: Long,
#ColumnInfo(name = COL_NAME)
val object1_name: String
/*TESTING INCLUDE FOR V2+ >>>>>*///, #ColumnInfo(name = COL_EXTRAV2, defaultValue = "x") val object1_extra: String
) {
companion object {
const val TABLE_NAME = "object1"
const val COL_ID = TABLE_NAME + "_object1_id"
const val COL_NAME = TABLE_NAME + "_object1_name"
const val COL_EXTRAV2 = TABLE_NAME + "_object1_extrav2"
}
}
Object2
#Entity(tableName = TABLE_NAME)
data class Object2(
#PrimaryKey
#ColumnInfo(name = COL_ID)
val object2_id: Long,
#ColumnInfo(name = COL_NAME)
val object2_name: String
/*TESTING INCLUDE FOR V3>>>>>*///, #ColumnInfo(name = COL_EXTRAV3, defaultValue = "x") val object3_extrav3: String
) {
companion object {
const val TABLE_NAME = "object2"
const val COL_ID = TABLE_NAME + "_object2_id"
const val COL_NAME = TABLE_NAME + "_object2_name"
const val COL_EXTRAV3 = TABLE_NAME + "_object2_extrav3"
}
}
Dao
#Dao
abstract class Dao {
#Insert
abstract fun insert(object1: Object1): Long
#Insert
abstract fun insert(object2: Object2): Long
#Query("SELECT * FROM ${Object1.TABLE_NAME}")
abstract fun getAllFromObject1(): List<Object1>
#Query("SELECT * FROM ${Object2.TABLE_NAME}")
abstract fun getAllFromObject2(): List<Object2>
}
AppDatabase
#Database(
entities = [Object1::class, Object2::class],
version = AppDatabase.DBVERSION,
autoMigrations = [AutoMigration (from = 1, to = 2)],
exportSchema = true
)
abstract class AppDatabase : RoomDatabase() {
abstract fun dao(): Dao
companion object {
private const val DB_NAME = "database.db"
private const val DB_FILENAME = "AppDB.db" //<<<<< ADDED for getting header
const val TAG = "DBINFO" //<<<< ADDED for logging
const val DBVERSION = 1 //<<<<<ADDED for logging
#Volatile
private var instance: AppDatabase? = null
fun getInstance(context: Context): AppDatabase {
return instance ?: synchronized(this) {
//ADDED>>>>> to get database version from dbfile and assets before building the database
Log.d(TAG,
"AssetDB Version =${getAssetDBVersion(context, DB_NAME)} " +
"Database Version = ${getDBVersion(context, DB_FILENAME)} " +
"Room Version = ${DBVERSION}")
instance ?: buildDatabase(context).also { instance = it }
}
}
private fun buildDatabase(context: Context): AppDatabase {
return Room.databaseBuilder(
context,
AppDatabase::class.java, DB_FILENAME)
.fallbackToDestructiveMigration()
.createFromAsset(DB_NAME)
.allowMainThreadQueries()
.addCallback(rdc)
.build()
}
/* Call Backs for discovery */
object rdc: RoomDatabase.Callback(){
override fun onCreate(db: SupportSQLiteDatabase) {
super.onCreate(db)
Log.d(TAG,"onCreate called. DB Version = ${db.version}, Room Version is ${DBVERSION}")
}
override fun onOpen(db: SupportSQLiteDatabase) {
super.onOpen(db)
Log.d(TAG,"onOpen called. DB Version = ${db.version}, Room Version is ${DBVERSION}")
}
override fun onDestructiveMigration(db: SupportSQLiteDatabase) {
super.onDestructiveMigration(db)
Log.d(TAG,"onDestructiveMigration called. DB Version = ${db.version}, Room Version is ${DBVERSION}")
}
}
fun getAssetDBVersion(context: Context, assetFilePath: String): Int {
var assetFileHeader = ByteArray(100)
try {
var assetFileStream = context.assets.open(assetFilePath)
assetFileStream.read(assetFileHeader,0,100)
assetFileStream.close()
} catch (e: IOException) {
return -2 // Indicates file not found (no asset)
}
return ByteBuffer.wrap(assetFileHeader,60,4).getInt()
}
fun getDBVersion(context: Context, dbFileName: String): Int {
var SQLiteHeader = ByteArray(100)
val dbFile = context.getDatabasePath(dbFileName)
if(dbFile.exists()) {
var inputStream = dbFile.inputStream()
inputStream.read(SQLiteHeader, 0, 100)
inputStream.close()
return ByteBuffer.wrap(SQLiteHeader, 60, 4).getInt()
} else {
return -1 // Indicates no database file (e.g. new install)
}
}
}
}
you may wish to consider including the logging above, it could very easily detect issues with the version(s) being used.
MainActivity
class MainActivity : AppCompatActivity() {
lateinit var db: AppDatabase
lateinit var dao: Dao
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
db = AppDatabase.getInstance(this)
dao = db.dao()
for(o1: Object1 in dao.getAllFromObject1()) {
logObject1(o1)
}
for(o2: Object2 in dao.getAllFromObject2()) {
logObject2(o2)
}
}
fun logObject1(object1: Object1) {
Log.d(TAG,"ID is ${object1.object1_id}, Name is ${object1.object1_name}")
}
fun logObject2(object2: Object2) {
Log.d(TAG,"ID is ${object2.object2_id}, Name is ${object2.object2_name}")
}
companion object {
const val TAG = AppDatabase.TAG
}
}
In addition to utilising the above code and ensuring that the 6 tasks were undertaken I also kept a spreadsheet of the versions and the results e.g. :-
Previous answer (not the case after testing)
I believe that your issue may be with the pre-populated database, in that it's version number (user_version) hasn't been changed to 3.
you can change the version using the SQL (from an SQlite tool ) PRAGMA user_version = 3;
The documentation says :-
Here is what happens in this situation:
Because the database defined in your app is on version 3 and the database instance already installed on the device is on version 2, a migration is necessary.
Because there is no implemented migration plan from version 2 to version 3, the migration is a fallback migration.
Because the fallbackToDestructiveMigration() builder method is called, the fallback migration is destructive. Room drops the database instance that's installed on the device.
Because there is a prepackaged database file that is on version 3, Room recreates the database and populates it using the contents of the prepackaged database file.
If, on the other hand, you prepackaged database file were on version 2, then Room would note that it does not match the target version and would not use it as part of the fallback migration.
By note perhaps by the way of an exception?