SQLite database migration appears to only partially apply in Espresso test - android

We have an SQLite database and a corresponding SQLiteOpenHelper subclass. This helper has an onDowngrade implementation that I would like to write an Espresso test for.
The full onDowngrade implementation is available here. This is a simplified version of it:
#Override
public void onDowngrade(SQLiteDatabase db, int oldVersion, int newVersion) {
db.execSQL("CREATE TABLE IF NOT EXISTS foo_tmp (_id integer primary key, bar text not null, baz text not null);");
db.execSQL("INSERT INTO foo_tmp(_id,bar,baz) SELECT _id,bar,baz FROM foo;");
db.execSQL("DROP TABLE IF EXISTS foo;");
db.execSQL("RENAME TABLE foo_tmp TO foo;");
}
The test loads a database dump with a very high version number and added or removed columns. It then gets a readable database and ensures that the version has been downgraded to the current expected version and that the column names are the expected column names. The full source is available here. This is what it looks like:
#Test
public void testMigration() throws IOException {
writeDatabaseFile("database" + File.separator + dbFilename);
InstancesDatabaseHelper databaseHelper = new InstancesDatabaseHelper();
SQLiteDatabase db = databaseHelper.getReadableDatabase();
assertThat(db.getVersion(), is(InstancesDatabaseHelper.DATABASE_VERSION));
List<String> newColumnNames = InstancesDatabaseHelper.getInstancesColumnNames(db);
assertThat(newColumnNames, contains(InstancesDatabaseHelper.CURRENT_VERSION_COLUMN_NAMES));
}
Everything works as intended if I manually load the same database dumps into the app. However, when I run this test, it looks like the last RENAME in the migration is not executed. If I comment out the last two SQL statements in the migration (dropping the original table and renaming the temporary table to the original table name), I can assert that the temporary table has the expected contents (here is a commit that shows this).
With some experimentation, we have found that adding databaseHelper.getReadableDatabase().close(); in the test after instantiating the SQLiteOpenHelper makes the tests pass. Given that the onDowngrade call is wrapped in a transaction, I don't understand how this is possible.
Could this point to a bug in our onDowngrade implementation? Is triggering migrations in Espresso tests different in some way?

There probably is a race condition, because SQLite is a shared resource.
eg. when the test runs before the last one COMMIT statement was issued.
Wrap it into a transaction (also see Isolation In SQLite):
if(! BuildConfig.DEBUG) {
db.beginTransaction();
} else {
db.beginTransactionWithListener(new SQLiteTransactionListener() {
#Override public void onBegin() {Log.d(LOG_TAG, "onBegin()");}
#Override public void onCommit() {Log.d(LOG_TAG, "onCommit()");}
#Override public void onRollback() {Log.d(LOG_TAG, "onRollback()");}
});
}
try {
db.execSQL("CREATE TABLE IF NOT EXISTS foo_tmp (_id integer primary key, bar text not null, baz text not null);");
db.execSQL("INSERT INTO foo_tmp(_id,bar,baz) SELECT _id,bar,baz FROM foo;");
db.execSQL("DROP TABLE IF EXISTS foo;");
db.execSQL("RENAME TABLE foo_tmp TO foo;");
db.setTransactionSuccessful();
} catch(SQLException e){
Log.d(LOG_TAG, "" + e.getMessage());
} finally {
db.endTransaction();
}
db.close();

Related

Android SQLite add new table on upgrade

I would like to update my database and add a new table. So following this Stackoverflow question, I have incremented my database from version 1 to version 2. Then in onUpgrade, I wrote the code to add new table and insert data:
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
if (oldVersion < 2) {
String CREATE_B_TABLE = "CREATE TABLE IF NOT EXISTS b ( " +
"_id INTEGER PRIMARY KEY AUTOINCREMENT, " +
"qset TEXT, "+
"highscore TEXT )";
db.execSQL(CREATE_B_TABLE);
String ADD_B = "INSERT INTO b ( qset ) VALUES ('1B'), ('2B'), ('3B'), ('4B'), ('5B'), ('6B'), ('7B'), ('8B'), ('9B'), ('10B')";
db.execSQL(ADD_B);
}
this.onCreate(db);
}
I also put the same code in onCreate to cater for new users.
However, something weird happened when the app is upgraded. The Add_B is inserted twice to the table, so I am having duplicated data. If I removed the code in onUpgrade, new table is added without duplication. It seem like onCreate is called when the app is upgraded, contradicting with the answer from another stackoverflow question.
If I read correctly the code you are intentionally calling the onCreate method from your onUpgrade.
Just before the method ends you call: this.onCreate(db)

App can't find sqlite table for specific user

I'm having an odd issue of a user of my companies internal app not running an sqlite query. It crashes with the following error:
android.database.sqlite.SQLiteException: no such column: intCol3 (code 1 SQLITE_ERROR): , while compiling: SELECT primaryKey, intCol1, textCol1, intCol2, textCol2, textCol3, intCol3 FROM table WHERE intCol4='1' LIMIT 100
intCol3 and intCol4 are new columns added to this table for the new version I'm trying to install. However, if I remove intCol3 it has an error with intCol4 and if I remove that as well, it has an issue with the primaryKey..
Initially I thought this was related to him being the only user who has android 9 on his phone but when trying it on a different android 9 device it works fine. It seems to be the second he links his google account to the device it breaks this query.
I've tried freshly installing the app and clearing all app data and anything cached to his phone and it still throws this exception.
I've also tried adding to the code to try creating the "Missing" columns each time he tries to log in but it doesn't make a difference.
Has anyone came across anything similar or be able to point me in the correct direction as to where to look?
This is my onCreate and onUpgrade
#Override
public void onCreate(SQLiteDatabase db) {
try {
createTables();
} catch (Exception e) {
Log.d("Error:","Failied to build tables:" + e);
}
}
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
Log.v("*! dbHelper upgrade", "Update contacts");
db.execSQL("DROP TABLE IF EXISTS contacts");
onCreate(db);
}
public void createTables() {
SQLiteDatabase db = this.getWritableDatabase();
Log.v("*! dbHelper create", "Create contacts");
db.execSQL(
"create table contacts " +
"(cnId integer primary key, cnName text, cnDate date)"
);
db.execSQL(
"create table parts " +
"(primaryKey integer primary key, intCol1 integer, textCol1 text, intCol2 integer, textCol2 text, textCol3 text, intCol3 integer, intCol4 integer)"
);
}
Two problems:
Your createTables() calls getWritableDatabase(). You cannot do that - it leads to a "called recursively" exception. Use the SQLiteDatabase passed as an argument to onCreate() instead.
Your onCreate() swallows the exception. Since onCreate() returns normally, the framework thinks database setup was successful. You need to let exceptions bubble up from SQLiteOpenHelper.

Why does SQLiteOpenHelper drop the table in "onUpgrade" method?

I learned a way which use SQLite3 on Android OS recently.
But it has a non acceptable part.
I can't understand why drop the table when called "onUpgrade" method of SQLiteOpenHelper.
Why need "onUpgarde" method?
If code executes "drop table", table data of old version DB will be removed, isn't it?
Why delete existing data of old DB?
How to restore existing DB data when drop the table?
[Here is learned code]
public class MySQLiteOpenHelper extends SQLiteOpenHelper {
final String CONFIRMED_SHEETS_TABLE = "confirmed_sheets";
public MySQLiteOpenHelper(Context context, String name, SQLiteDatabase.CursorFactory factory, int version) {
super(context, name, factory, version);
}
#Override
public void onCreate(SQLiteDatabase sqLiteDatabase) {
String sql = "create table" +
CONFIRMED_SHEETS_TABLE +
"_id integer primary key autoincrement, " +
"group text, " +
"num001 text, " +
"num002 text, " +
"num003 text, " +
"num004 text, " +
"num005 text, " +
"num006 text, " +
"date text)";
sqLiteDatabase.execSQL(sql);
}
#Override
public void onUpgrade(SQLiteDatabase sqLiteDatabase, int oldVersion, int newVersion) {
// non acceptable part //
String sql = "drop table if exists " + CONFIRMED_SHEETS_TABLE;
sqLiteDatabase.execSQL(sql);
onCreate(sqLiteDatabase);
}
}
If code executes "drop table", DB table data of old version DB will be
removed, isn't it?
Yup
Why need "onUpgrade" method?
If you are switching databases (for example because you added a new column), your app (usually) now depends on that change. Increasing the database version in your Helper class calls onUpgrade(), which allows you to take care of any migration to prepare the app to use your new schema.
Did you know why learned code executes "drop table"?
Convenience. It's not necessarily the right approach, but a database change can make it hard to take old data and merge it in the new table. Thus, it is easier logic-wise to simply start anew.
If you want to merge an existing and new table, have a look at this question.
You do not need to perform a DROP TABLE in onUpgrade(), as it is currently written in your code. The purpose of onUpgrade() is for your app to check if a new version of your app's database is being installed on a user's device, and if so, if there are any changes to your database, such as adding a new column, you can make those changes within onUpgrade(). If you never change your database schema, you never need to do anything in onUpgrade().
For more information and a introductory tutorial on Databases in Android, refer to the Notepad sample code here.
EDIT: also, here's an example of an onUpgrade() I wrote for one of my apps:
/**
* Handle upgrades to the database.
*/
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
if(DBG) Utils.log(TAG, "DBHelper: onUpgrade()...");
// Logs that the database is being upgraded
Utils.log(TAG, "w", "Upgrading database from version " + oldVersion + " to "
+ newVersion);
/*
* Every time you add new columns to the database in the phone, you will want
* to increment the Database version above and then add a condition in here for
* upgrading to it. Otherwise it will cause upgrading users to be nontrivial and
* lead to unnecessary crashes or upgrade instructions.
*/
if (newVersion > oldVersion && newVersion <= 4) {
db.execSQL("ALTER TABLE " + TrackerDb.SomeTable.TABLE_NAME + " ADD COLUMN "
+ TrackerDb.SomeTable.COLUMN_NAME_DATE_MODIFIED + " TEXT");
}
}
OnUpgrade() is called when db versions "old and new" does not match; which means that the user wants to change Database Structure (Dropping a table, Adding new Table, Modifying Table definition).
So the OnUpgrade() should contain the logic as:
Take Backup of the existing data (e.g. Generate insert statements for existing
data).
Modify Database Structure accordingly.
Restore the data from the backup.

Android SQLite Upgrade without losing data

I have created a SQLite database successfully and it works fine. However when the onUpgrade method is called, I'd like to do so without losing data. The app I'm developing is a quiz app. Simply, when the onCreate method is called I create and prepopulate a database with questions, answers etc. The last column is whether they have set the question as a favourite or not. What I would like to do is that when the onUpgrade method is called, I'd like to temporarily save that one column, drop the whole database, recreate it with any edits I've made to old questions and add any new questions then re-add back the questions that they set as favourites.
So one option I tried was the following:
db.execSQL("ALTER TABLE quiz RENAME TO temp_quiz");
onCreate(db);
db.execSQL("INSERT INTO quiz (favouries) SELECT favourites FROM temp_quiz");
db.execSQL("DROP TABLE IF EXISTS temp_quiz");
However this doesn't work owing to the fact INSERT INTO just adds new rows rather than replacing the existing rows. I have also tried REPLACE INTO, INSERT OR REPLACE INTO and
db.execSQL("INSERT INTO quiz (_id, favouries) SELECT _id, favourites FROM temp_quiz");
of which none work.
Currently I do have it set up to work by altering the name of the table, calling the onCreate(db) method and then setting up a cursor which reads each row and uses the db.update() method as shown below:
int place = 1;
int TOTAL_NUMBER_OF_ROWS = 500;
while (place < TOTAL_NUMBER_OF_ROWS) {
String[] columns = new String[] { "_id", ..........., "FAVOURITES" };
// not included all the middle columns
Cursor c = db.query("temp_quiz", columns, "_id=" + place, null, null, null, null);
c.moveToFirst();
String s = c.getString(10);
// gets the value from the FAVOURITES column
ContentValues values = new ContentValues();
values.put(KEY_FLAG, s);
String where = KEY_ROWID + "=" + place;
db.update(DATABASE_TABLE, values, where, null);
place++;
c.close();
}
However whilst this works it is extremely slow and will only get worse as my number of questions increases. Is there a quick way to do all this?
Thank you! P.S. Ideally it should only update the row if the row is present. So if in an upgrade I decide to remove a question, it should take this into account and not add a new row if the row doesn't contain any other data. It might be easier to get it to remove rows that don't have question data rather than prevent them being added.
changed it to:
db.execSQL("UPDATE new_quiz SET favourites = ( SELECT old_quiz.favourites
FROM old_quiz WHERE new_quiz._id = old_quiz._id) WHERE EXISTS
( SELECT old_quiz.favourites FROM old_quiz WHERE new_quiz._id = old_quiz._id)");
Which works :D
public class DataHelper extends SQLiteOpenHelper {
private static final String dbName="dbName";
private Context context;
private SQLiteDatabase db;
private final static int version = 1;
public static final String SurveyTbl = "CREATE TABLE SurveyTbl (SurveyId TEXT PRIMARY KEY, Idref TEXT, SurveyDate TEXT)";
public DataHelper(Context context) {
super(context, dbName, null, version);
this.db = getWritableDatabase();
this.context = context;
Log.i("", "********************DatabaseHelper(Context context)");
}
#Override
public void onCreate(SQLiteDatabase db) {
try {
db.execSQL(SurveyTbl);
} catch (Exception e) {
Log.i("", "*******************onCreate");
}
}
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
try {
db.execSQL("ALTER TABLE HandpumpSurveyTbl ADD COLUMN NalYozna TEXT");
} catch (Exception e) {
Log.i("", ""+e);
}
onCreate(db);
}
}
I didn't get to see your Quiz table schema, but I assume it has fields like "question", "answer", "favorites", and some kind of a unique primary key to identify each question, which I will just call rowId for now.
// after renaming the old table and adding the new table
db.execSQL("UPDATE new_quiz SET new_quiz.favorites = old_quiz.favorites where new_quiz.rowId = old_quiz.rowId");
That will update only the rows of the new quiz table that match the old quiz table, and set the favorites value from the old quiz table.
I assume you have some kind of a unique identifier to identify each question, so instead of the rowId above, you'll use that (question number or something).
For who don't know yet how to upgrade the version of the SQLite when upgrading the database schema for example, use the method needUpgrade(int newVersion)!
My code:
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion){
if(newVersion>oldVersion){
db.execSQL(scriptUpdate);
db.needUpgrade(newVersion);
}
}
ALTER TABLE mytable ADD COLUMN mycolumn TEXT
In your onUpgrade method, it would look something like this:
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
String upgradeQuery = "ALTER TABLE mytable ADD COLUMN mycolumn TEXT";
if (newVersion>oldVersion)
db.execSQL(upgradeQuery);
}
Example, how to drop a table and create a new table without losing data by using a temporary table:
db.execSQL("CREATE TEMPORARY TABLE temp_table (_id INTEGER PRIMARY KEY AUTOINCREMENT, col_1 TEXT, col_2 TEXT);");
db.execSQL("INSERT INTO temp_table SELECT _id, col_1, col_2 FROM old_table");
db.execSQL("CREATE TABLE new_table (_id INTEGER PRIMARY KEY AUTOINCREMENT, col_1 TEXT, col_2 TEXT, col_3 TEXT);");
db.execSQL("INSERT INTO new_table SELECT _id, col_1, col_2, null FROM temp_table");
db.execSQL("DROP TABLE old_table");
db.execSQL("DROP TABLE temp_table");

onCreate not being called after getWritableDatabase/getReadableDatabase

My app's got a database with three tables in it: one to store the names of the people it tracks, one to track an ongoing event, and one - for lack of a better term - for settings.
I load the first table when the app starts. I ask for a readable database to load in members to display, and later I write to the database when the list changes. I've had no problems here.
The other two tables, however, I can't get to work. The code in the helper classes is identical with the exception of class names and column names, and (at least until the point where I try to access the table) the code to use the table is nearly identical as well.
Here's the code for my helper class (I've got a separate helper for each table, and as I said, it's identical except for class names and columns):
public class db_MembersOpenHelper extends SQLiteOpenHelper
{
public static final String TABLE_NAME = "members_table";
public static final String[] COLUMN_NAMES = new String[] {
Constants.KEY_ID,
"name",
"score"
};
private static final String TABLE_CREATE = "CREATE TABLE " + TABLE_NAME + " ("
+ COLUMN_NAMES[0] + " INTEGER PRIMARY KEY autoincrement, "
+ COLUMN_NAMES[1] + " TEXT, "
+ COLUMN_NAMES[2] + " INTEGER);";
public db_MembersOpenHelper(Context context)
{
super(context, Constants.DATABASE_NAME, null, Constants.DATABASE_VERSION);
}
#Override
public void onCreate(SQLiteDatabase db) { db.execSQL(TABLE_CREATE); }
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion)
{
Log.w("TaskDBAdapter", "Upgrading from version " + oldVersion + " to " + newVersion + ".");
// Do nothing. We do not have any updated DB version
}
}
Here's how I use it successfully:
db_MembersOpenHelper membersDbHelper = new db_MembersOpenHelper(this);
SQLiteDatabase membersDb = membersDbHelper.getReadableDatabase();
Cursor membersResult = membersDb.query(TABLE_NAME, null, null, null, null, null, null);
members = new HashMap<String, Integer>();
membersResult.moveToFirst();
for(int r = 0; r < membersResult.getCount(); r++)
{
members.put(membersResult.getString(1), membersResult.getInt(2));
membersResult.moveToNext();
}
membersDb.close();
And here's where it fails:
db_PlayersOpenHelper playersDbHelper = new db_PlayersOpenHelper(this);
final SQLiteDatabase playersDb = playersDbHelper.getWritableDatabase();
if(newGame)
{
for(String name : players)
{
ContentValues row = new ContentValues();
row.put(COLUMN_NAMES[1], name);
row.put(COLUMN_NAMES[2], (Integer)null);
playersDb.insert(TABLE_NAME, null, row);
}
}
The first one works like a charm. The second results in ERROR/Database(6739): Error inserting achievement_id=null name=c
android.database.sqlite.SQLiteException: no such table: players_table: , while compiling: INSERT INTO players_table(achievement_id, name) VALUES(?, ?);
...
I did do some testing, and the onCreate method is not being called at all for the tables that aren't working. Which would explain why my phone thinks the table doesn't exist, but I don't know why the method isn't getting called.
I can't figure this out; what am I doing so wrong with the one table that I accidentally did right with the other?
I think the problem is that you are managing three tables with with three helpers, but only using one database. SQLiteOpenHelper manages on database, not one table. For example, it checks to see whether the database, not table, exists when it starts. It already does, so onCreate() does not fire.
I would manage all tables with one helper.
Let me see if I get this right. You are trying to create one database with three tables. But when you create the database, you create just one table; you are somehow instantiating the same database at a different place and wonder why its onCreate method doesn't get called. Is this a correct interpretation?
My strategy would be to try and create all three tables in the single onCreate() method.
If you are working with multiple tables, then you have to create all of the tables at once. If you have run your application first and later you update your database, then it will not upgrade your DB.
Now delete your application, then run it again.
There is one more solution but it is not proper. You can declare onOpen method in which you can call onCreate. And add IF NOT EXISTS before table name in your create table string. – Sourabh just now edit

Categories

Resources