Develop Reference

Android Room requires migration after data insertion by another app

I'm developing an Android app with an embedded SQLite database run with Android Room. I'm moving the app from desktop Java to Android and I need to bring data into the Android database from the old database used with the desktop app. I ran into a problem with Room when I tried using the altered database with my Android app.
I tried taking the database file created by Room and move the data from the old database into this one with a short Java app I coded for this purpose. I simply emptied the relevant tables, read data from the old database and inserted it into the new database.
When I tried moving the database file back to my Android devide and using it with the Android app I got this exception:
Room cannot verify the data integrity. Looks like you've changed schema but forgot to update the version number.
You can simply fix this by increasing the version number.
If I change the version of the database in my Room database code, Room would then require a piece of code for the migration even though the structure of the database has not been changed:
java.lang.IllegalStateException: A migration from 1 to 2 was required but not found. Please provide the necessary Migration path via RoomDatabase.Builder.addMigration(Migration ...) or allow for destructive migrations via one of the RoomDatabase.Builder.fallbackToDestructiveMigration* methods.
I tried clearing the app data cache and clearing all of its data and even uninstalling the app, then reinstalling it with the modified database file. Didn't make the problem go away. I also tried everything here Room cannot verify the data integrity including setting android:allowBackup="false" in my Manifest, and adding fallbackToDestructiveMigration to my database. After trying fallbackToDestructiveMigration, all of my new database's data was gone. But then I put the altered database file back in again and it worked. To be honest, I'm not entirely sure what step made it work. I think it was increasing the version number, running it with fallbackToDestructiveMigration and then moving the new database file back in again when fallbackToDestructiveMigration had wiped data from the database...
My question is how can Room know the database has been modified when there hadn't been any change to its structure that would make it incompatible with the Entities of my app? And how can know even though I've cleared the app's data from the device? And are there any steps I should take the next time that are not mentioned above? I quess I could just include migration code that does nothing. But that seems kind of silly, I'd like the version number to stay at 1 since the app is in development at the moment.

Add an empty Migration.
private val MIGRATION_1_2 = object : Migration(1, 2) {
override fun migrate(database: SupportSQLiteDatabase) {
// empty migration.
}
}
if (sInstance == null) {
if (!checkDataBase())
copyDataBase(context)
synchronized(AppDatabase::class) {
sInstance = Room.databaseBuilder(context.applicationContext, AppDatabase::class.java, DATABASE_NAME)
.addMigrations(MIGRATION_1_2)
.build()
}
}
return sInstance!!
You can check out this Migration Guide from Florina Muntenescu to learn more about Migrations

Upgrade database with Sqlite.net Pcl for Xamarin android

i have the problem that when i release new version of my application, if i add a new column to one of my db tables, the database doesn't update. Any one know how to create a script of upgrade versione in case there are new columns or new tables??
Thanks

You have to remember that CreateTable it's already doing the columns update for you, because internally it calls a method called MigrateTable.
However you could have to handle more advanced modification to your database, like adding triggers or something similar.
In that case i suggest you to perform modifications manually.
In Xamarin Forms i've ended up with this:
https://gist.github.com/matpag/b2545cc22c8e22449cd7eaf6b4910396
Could not be the best strategy ever but seems to work for me.
Summarizing :
You have to save the database version in an internal flag of the SQlite database called user_version accessible with PRAGMA keyword.
Every time you get the database connection, you have to perform a check and see if the current database version is the same as the app last database version.
If not you need to perform a database update and set the new current version.

It's not a matter of a script, as there isn't such a thing. You can release a version with a "patch" that will run once, extracting all your records to a temporary form -> deleting the table -> creating it again (will assure it's created with the new columns and so on) -> reinserting the records again. After a while, when you know that all your users (or whenever you set the limit) have moved to the newer version you can just eliminate the "patch" from your code.
Hope it helps.

The automatic migration feature is still not working in sqlite.net-pcl, but it looks like it does work in the other sqlite package: sqlite-net-pcl, which is actually the Xamarin recommend sqlite package.

onUpgrade method from SQLiteOpenHelper never called when replacing the database file

I know there are a lot of similar questions and I hope not being repeating but I tried different things and I am still unable to fix the issue. I am developing an Android app that uses SQLite to store some data internally. I have been using this app in my own phone for a month and I added lot of data I do not want to lose, but at the same time I have been improving the app in my computer, but the improvements made me change some column names, adding some columns and tables. So, right now I want to update the app in my phone.
The previous version had the database set to 2 and the new one is 3. In addition, my app has the ability to create backups of the database just copying the file to the SDcard, and the ability to restore it by replacing the internal database file for one that the user selects from the sd card. Now the problem is that I did a backup of the database from the phone and I moved it to my computer to import it in the new version of the app running in an emulator to ensure the app can upgrade the database as expected before I install the new version on my phone. The problem is that onUpgrade for the SQLiteOpenHelper is never called. I did different trials but nothing is working. I can ensure the database from my phone has version 2 (I used sqliteadmin "PRAGMA user_version;" SQL query and answers 2) and my app looks for version 3. The app crashes as it is unable to find the new tables.
Here you have the code executed when importing a backup:
private boolean importDatabase(String path) {
String dbPath = mDbHelper.getDbPath();
mDbHelper.close();
boolean result = copyFile(path, dbPath);
if (result) {
mDbHelper = new DbHelper(this);
mDbHelper.getWritableDatabase().close();
}
return result;
}
I have to mention as well that when the user reaches the place to import/export the database in the new version, a new empty database with version 3 has been created as the app needs an active database from the stating activity. Then, when importing, this empty database file is replaced by the imported one. I wonder if there is any cached data I am missing to clean.
Could you give me some clues to understand why onUpgrade is never called in this scenario?

Strategy for updating Android app (with database)

Please forgive me if this question has been answered - I searched and couldn't find it.
I have an Android app that I want to upgrade, and it uses a SQLite Database. I want to update some of the application logic in the app, but there will be no updates to the database schema or contents. I basically need to keep the database exactly as-is for the user.
Do I need to do anything in onUpgrade to ensure that the database is kept, or can I leave the DB stuff alone for this update?

The onUpgrade() method is used incases of version change. Which means the database stored in the phone needs to be altered or dropped or deleted and a new database to be created. As your application does not have any of these requirements you can leave the DB stuff for this update.

This related article may help you with your question.
The way that I understand it, is that you need to put your database changing code in onUpdate() if you WANT to update between versions. But since you don't intend to, and are probably keeping the database version the same, then you will most likely have no issues at all.

Upgrading will NOT interfere with SQLite. Changes to db structure will not be implemented unless you programmatically do so (in onUpgrade method) or you uninstall and reinstall your app.
As long as it is the SAME application that you are upgrading, your db will not be affected and your data will not be affected either. If you change the signing key used in building your apk, your db will be recreated.
Conversely, if you change database structure at any given point, your onUpgrade method will come into play. You will be forced to backup, drop, recreate and repopulate tables which have been changed between versions (oher tables remain untouched both in structure and in data).
NOTE: In debugging, i just uninstall and reinstall the app every time i make db changes, but in production you DONT want to do that.

Android backup/restore: how to backup an internal database?

I have implemented a BackupAgentHelper using the provided FileBackupHelper to backup and restore the native database I have. This is the database you typically use along with ContentProviders and which resides in /data/data/yourpackage/databases/.
One would think this is a common case. However the docs aren't clear on what to do: http://developer.android.com/guide/topics/data/backup.html. There is no BackupHelper specifically for these typical databases. Hence I used the FileBackupHelper, pointed it to my .db file in "/databases/", introduced locks around any db operation (such as db.insert) in my ContentProviders, and even tried creating the "/databases/" directory before onRestore() because it does not exist after install.
I have implemented a similar solution for the SharedPreferences successfully in a different app in the past. However when I test my new implementation in the emulator-2.2, I see a backup being performed to LocalTransport from the logs, as well as a restore being performed (and onRestore() called). Yet, the db file itself is never created.
Note that this is all after an install, and before first launch of the app, after the restore has been performed. Apart from that my test strategy was based on http://developer.android.com/guide/topics/data/backup.html#Testing.
Please also note I'm not talking about some sqlite database I manage myself, nor about backing up to SDcard, own server or elsewhere.
I did see a mention in the docs about databases advising to use a custom BackupAgent but it does not seem related:
However, you might want to extend
BackupAgent directly if you need to:
* Back up data in a database. If you have an SQLite database that you
want to restore when the user
re-installs your application, you need
to build a custom BackupAgent that
reads the appropriate data during a
backup operation, then create your
table and insert the data during a
restore operation.
Some clarity please.
If I really need to do it myself up to the SQL level, then I'm worried about the following topics:
Open databases and transactions. I have no idea how to close them from such a singleton class outside of my app's workflow.
How to notify the user that a backup is in progress and the database is locked. It might take a long time, so I might need to show a progress bar.
How to do the same on restore. As I understand, the restore might happen just when the user has already started using the app (and inputting data into the database). So you can't presume to just restore the backupped data in place (deleting the empty or old data). You'll have to somehow join it in, which for any non-trivial database is impossible due to the id's.
How to refresh the app after the restore is done without getting the user stuck at some - now - unreachable point.
Can I be sure the database has already been upgraded on backup or restore? Otherwise the expected schema might not match.

After revisiting my question, I was able to get it to work after looking at how ConnectBot does it. Thanks Kenny and Jeffrey!
It's actually as easy as adding:
FileBackupHelper hosts = new FileBackupHelper(this,
"../databases/" + HostDatabase.DB_NAME);
addHelper(HostDatabase.DB_NAME, hosts);
to your BackupAgentHelper.
The point I was missing was the fact that you'd have to use a relative path with "../databases/".
Still, this is by no means a perfect solution. The docs for FileBackupHelper mention for instance: "FileBackupHelper should be used only with small configuration files, not large binary files.", the latter being the case with SQLite databases.
I'd like to get more suggestions, insights into what is expected of us (what is the proper solution), and advice on how this might break.

Here's yet cleaner way to backup databases as files. No hardcoded paths.
class MyBackupAgent extends BackupAgentHelper{
private static final String DB_NAME = "my_db";
#Override
public void onCreate(){
FileBackupHelper dbs = new FileBackupHelper(this, DB_NAME);
addHelper("dbs", dbs);
}
#Override
public File getFilesDir(){
File path = getDatabasePath(DB_NAME);
return path.getParentFile();
}
}
Note: it overrides getFilesDir so that FileBackupHelper works in databases dir, not files dir.
Another hint: you may also use databaseList to get all your DB's and feed names from this list (without parent path) into FileBackupHelper. Then all app's DB's would be saved in backup.

A cleaner approach would be to create a custom BackupHelper:
public class DbBackupHelper extends FileBackupHelper {
public DbBackupHelper(Context ctx, String dbName) {
super(ctx, ctx.getDatabasePath(dbName).getAbsolutePath());
}
}
and then add it to BackupAgentHelper:
public void onCreate() {
addHelper(DATABASE, new DbBackupHelper(this, DB.FILE));
}

Using FileBackupHelper to backup/restore sqlite db raises some serious questions:
1. What happens if the app uses cursor retrieved from ContentProvider.query() and backup agent tries to override the whole file?
2. The link is a nice example of perfect (low entrophy ;) testing. You uninstall app, install it again and the backup is restored. However life can be brutal. Take a look at link. Let's imagine scenario when a user buys a new device. Since it doesn't have its own set, the backup agent uses other device's set. The app is installed and your backupHelper retrieves old file with db version schema lower than the current. SQLiteOpenHelper calls onDowngrade with the default implementation:
public void onDowngrade(SQLiteDatabase db, int oldVersion, int newVersion) {
throw new SQLiteException("Can't downgrade database from version " +
oldVersion + " to " + newVersion);
}
No matter what the user does he/she can't use your app on the new device.
I'd suggest using ContentResolver to get data -> serialize (without _ids) for backup and deserialize -> insert data for restore.
Note: get/insert data is done through ContentResolver thus avoiding cuncurrency issues. Serializing is done in your backupAgent. If you do your own cursor<->object mapping serializing an item can be as simple as implementing Serializable with transient field _id on the class representing your entity.
I'd also use bulk insert i.e. ContentProviderOperation example and CursorLoader.setUpdateThrottle so that the app is not stuck with restarting loader on data change during backup restore process.
If you happen do be in a situation of a downgrade, you can choose either to abort restore data or restore and update ContentResolver with fields relevant to the downgraded version.
I agree that the subject is not easy, not well explained in docs and some questions still remain like bulk data size etc.
Hope this helps.

As of Android M, there is now a full-data backup/restore API available to apps. This new API includes an XML-based specification in the app manifest that lets the developer describe which files to back up in a direct semantic way: 'back up the database called "mydata.db"'. This new API is much easier for developers to use -- you don't have to keep track of diffs or request a backup pass explicitly, and the XML description of which files to back up means you often don't need to write any code at all.
(You can get involved even in a full-data backup/restore operation to get a callback when the restore happens, for example. It's flexible that way.)
See the Configuring Auto Backup for Apps section on developer.android.com for a description of how to use the new API.

One option will be to build it in application logic above the database. It actually screams for such levell I think.
Not sure if you are doing it already but most people (despite android content manager cursor approach) will introduce some ORM mapping - either custom or some orm-lite approach. And what I would rather do in this case is:
to make sure your application works
fine when the app/data is added in
the background with new data
added/removed while the application
already started
to make some
Java->protobuf or even simply java
serialization mapping and write your
own BackupHelper to read the data
from the stream and simply add it to
database....
So in this case rather than doing it on db level do it on application level.