How to change affinity?
Expected:
serial_number_prefix=Column{name='serial_number_prefix', type='TEXT', affinity='2', notNull=false, primaryKeyPosition=0, defaultValue='null'},
Actual:
serial_number_prefix=Column{name='serial_number_prefix', type='STRING', affinity='1', notNull=false, primaryKeyPosition=0, defaultValue='null'},
The only change is affinity = 1, but it should be affinity = 2.
How to change it?
This is my migration function
database.execSQL("ALTER TABLE values ADD COLUMN serial_number_prefix STRING DEFAULT null")
in data class
#SerializedName("serial_number_prefix")
#ColumnInfo(name = "serial_number_prefix")
val serialNumberPrefix: String?,
Room, is very specific about what column types can be and thus what Room expects.
Unlike SQLite which is very flexible with column Types, column types in Room must be one of TEXT, INTEGER, REAL or BLOB.
Room determines the TYPE used in the SQL
In SQLite STRING actually equates to the catch all type NUMERIC, which Room does not allow and thus it has an affinity of 1 which is UNDEFINED.
Without going into too much detail the easiest way to conform to Room's expectations is to
create the #Entity annotated classes
Create the #Database annotated class with the #Entity annotated classes specified in the entities parameter of the annotation.
Compile/Build the project (e.g. CTRL + F9).
In the Android View locate the generated Java.
Find the class that is the same name as the #Database annotated class BUT suffixed with _Impl.
Within the class find the createAllTables method.
Copy and paste the SQL for the table(s) and base the ALTER statement on the column definitions therein. This will be the exact definition that Room expects.
How to change it?
In your case you would use:-
ALTER TABLE values ADD COLUMN serial_number_prefix TEXT DEFAULT null
i.e. STRING has been changed to TEXT
Regarding #ColumnInfo(typeAffinity=2)
You could think that overriding the affinity using the typeAffinity parameter of the #ColumnInfo annotation would resolve the issue e.g. using #ColumnInfo(typeAffinity = 1).
However, 1 equates to UNDEFINED, and thus Room then uses the field's type (as per the quote below) to determine the type. Thus as the field types is a Stringm it uses TEXT when it builds the expected schema (and uses TEXT in the SQL to create the table). As TEXT equates to affinity=2, the same issue occurs.
Again use the column type from generated java in the ALTER SQL.
The type affinity for the column, which will be used when constructing the database.
If it is not specified, the value defaults to UNDEFINED and Room resolves it based on the field's type and available TypeConverters.
https://developer.android.com/reference/androidx/room/ColumnInfo#typeAffinity()
Related
I want my table names to be specific as I am migrating from Native Android Room database to Flutter using Drift library for database.
Room take a table name i.e. class name and creates a table:
e.g. the following will create table 'User' with field names as 'firstName' and lastName' in native Android using Room database support
#Entity
data class User(
#PrimaryKey val id: Int,
val firstName: String?,
val lastName: String?
)
When I try to replicate this in Flutter, I write:
#DataClassName('User') // To avoid drift changing it to 'user'
data class User(
TextColumn get firstName => text().named('firstName')();
TextColumn get lastName => text().named('lastName')();
// The above '.named()' makes sure drift does not change names to 'first_name' & 'last_name'
)
By doing above I am making sure the databases match exactly! (please correct me if I am wrong here)
Now when I try to replicate the migration statements in drift I am writing:
...
if (from < 4) await m.addColumn(Note, Note.path);
if (from < 5) await m.addColumn(Profile, Profile.squarePicture);
...
As you can see that I am writing note as a 'Note' because I have defined my table name as 'Note' and not 'note' same goes with profile! But the code is highlighting the following error:
The argument type 'Type' can't be assigned to the parameter type
'TableInfo<Table, dynamic>'
When I change table names to small alphabets i.e. 'n' and 'p' the error goes away but I do not understand WHY? My table names are explicitly with capital letters in the start.
I do not want to change anything in the database when my users will upgrade their app which was previously made in Native Android using Kotlin and Room database to the Flutter version. I want them to have a seamless experience with no loss of data!
The #DataClassName() annotation is used to change the name of the class which is generated by the drift. This generated class is what the drift will use when querying the database and getting the result.
By default, the generated class name will be the same as the class name except the drift will strip any last 's' character. For example, if your class name is "Users" the drift will generate a class with the name "User" with the 's' at the end stripped.
Coming to your question #DataClassName() has nothing to do with the table name in your database or any migration strategy. If you want to change the table name you can do something like this,
class User extends Table {
///This property will change the table name.
String get tableName => 'User';
...
}
Official Documentation - Dart table Names
Drift also generates a class that has an identical name to our table class. In your case, there will be a class generated $User which extends our table class, and a mixin TableInfo. This class's object is automatically created by the drift in the database generated code.
So drift will generate a variable named user which will be object of class $User. This variable needs to be used inside migration stregeies so that drift can successfully migrate a table.
I have a project that I wrote in kotlin. I want to insert data in different columns of the same table on different pages. I have specified these columns in the dataclass, but it gives a null data error.
In order to make this insert process more healthy, should I divide the table into two separate tables or send static 'null' data and update these fields?
In a database, such as SQLite (which Room is a wrapper around), the unit of insertion is a row.
A row will consist of the same number of columns. You cannot insert a column on it's own, other than if you ALTER the table to add or remove a column, when the change is reflected throughout the entire table.
if adding a column then a DEFAULT VALUE must be provided, this could be the default/implicit value of null or another specific value.
Room with Kotlin will apply a constraint (rule) of NOT NULL unless nulls are specifically allowed using for example ?
var theColumn: Int has the implicit NOT NULL
var theColumn: Int? does not have the implicit NOT NULL and nulls can be stored
var theColumn: Int=-1 will apply a default value of -1 in the absence of the field not being supplied a value when instantiating the object.
var theColumn: Int?=null will apply null in the absence of the field not being supplied a value when instantiating the object.
obviously fields may be altered before inserting the object, if var rather than val is used.
The data stored in the column can be interpreted to represent whatever you wish, often NULL will be used to represent a special situation such as no value.
If using an #Insert annotated function, then ALL columns are applied the values as obtained from the object or objects passed to the function. In Kotlin whether or not NULLs can be used is dependent upon the field definition or in some cases the #NonNull annotation.
#Insert indicates what is termed as a convenience method, it actually builds the underlying SQL along with binding the values using the SQLite API.
However, if you want flexibility, then an #Query annotation with suitable INSERT SQL statement can be used.
e.g. you could perhaps have a table that has 4 columns COL1, COL2, COL3 and COL4 and only apply some of the columns (the DEFAULT VALUE will be applied to the other column if specified, if not the NULL but if there is a NOT NULL constraint then a conflict would be raised).
So to insert when only two of the columns (COL2 and COL4) then you could use:-
#Query("INSERT INTO theTable (COL2,COL4) VALUES(:valueForCol2,:valueForCol4)")
fun insertCol2AndCol4Only(valueForCol2: Int, valueForCol4: Int?)
Note that valueForCol4 could be NULL. However, whether or not a NULL will result in a conflict depends upon how the field is defined in the #Entity annotated class.
Conflicts (breaking a rule) can be handled by SQLite, depending upon the type of the conflict. UNIQUE, PRIMARY KEY (which is really a UNIQUE conflict), CHECK (Room doesn't cater for CHECK constraints) and NOT NULL constraints can be handled in various ways at the SQLite level.
A common use of conflict handling is to IGNORE the conflict, in which case the action (INSERT or UPDATE) is ignored. In the case of INSERT the row is not inserted but SQLite ignores the conflict and doesn't issue an error.
So if for example COL4's field was var COL4: Int and not var COL4: Int? then the insert would fail and an SQlite Exception would occurr.
However if instead
#Query("INSERT OR IGNORE INTO theTable (COL2,COL4) VALUES(:valueForCol2,:valueForCol4)")
were used and the COL4 field were defined as var COL4: Int (implied NOT NULL constraint) then the conflict if NULL was passed as valueForCol4 then the row would not be inserted but no failure would occur as the NOT NULL conflict would be ignored.
With the #Insert annotation you can defined this conflict handling via the onConflictStrategy parameter e.g. #Insert(onConflictStrategy=OnConflict.IGNORE)
You may wish to consider reading the following:-
The On Conflict Clause
INSERT
In order to make this insert process more healthy, should I divide the table into two separate tables or send static 'null' data and update these fields?
Note the above is only a summary, INTEGER PRIMARY KEY aka #PrimaryKey var id: Long?=null or variations such as #PrimaryKey(autoGenerate=true) etc has specifically not been discussed.
The design of the database could be handled either way, from the very limited description of the scenario, a most likely suitable scenario cannot really be advised, although either could probably be an approach.
Additional
Based upon the comment:-
For example, I'm going to add the features of a car to the database, but it could be a different type at a time. So on the first page, the type of car will be chosen, like off road, sedan, 4x4, hatchback.
The perhaps consider having a feature table and a mapping table for a many-many relationship between car and it's features as per my response:-
I would suggest that features be a table and with a many-many relationship with the car. That is a car could have a 0-n features and a feature could be used by 0-n cars. The many-many relationship would require a third table known by many terms such as an associative table/reference table/ mapping table. Such a table has 2 core columns a column to map to the car and a column to map to the feature, the primary key being a composite of both these columns.
Here's a basic example of how this could work from an SQLite basis:-
DROP INDEX IF EXISTS carFeatureMap_idxon_feature;
DROP TABLE IF EXISTS carFeatureMap;
DROP TABLE IF EXISTS car;
DROP TABLE IF EXISTS feature;
CREATE TABLE IF NOT EXISTS car (
carId INTEGER PRIMARY KEY,
carname TEXT /* and other columns */
);
CREATE TABLE IF NOT EXISTS feature (
featureId INTEGER PRIMARY KEY,
featureDescription TEXT
);
CREATE TABLE IF NOT EXISTS carFeatureMap (
carIdMap INTEGER REFERENCES car(carId) ON DELETE CASCADE ON UPDATE CASCADE,
featureIdMap INTEGER REFERENCES feature(featureId) ON DELETE CASCADE ON UPDATE CASCADE,
PRIMARY KEY(carIdMap, featureIdMap)
);
/* Should improve efficiency of mapping from a feature */
CREATE INDEX IF NOT EXISTS carFeatureMap_idxon_feature ON carFeatureMap(featureIdMap);
/* Add some features */
INSERT OR IGNORE INTO feature VALUES(100,'4x4'),(101,'Sedan'),(106,'Convertable'),(null /*<<<< featureId generated by SQLite*/ ,'Hatchback');
/*Report1 Output the features */
SELECT * FROM feature;
/* Add some cars */
INSERT OR IGNORE INTO car VALUES(10,'Car1'),(20,'Car2'),(30,'Car3');
/*Report2 Output the cars */
SELECT * FROM car;
/* add the mappings/relationships/associations between cars and features */
INSERT OR IGNORE INTO carFeatureMap VALUES (10,101) /* Car 1 has 4x4*/,(10,106) /* Car 1 has Sedan */,(20,100);
/*Report3 Get the Cars with features cartesian product */
SELECT
car.carName,
featureDescription
FROM car
JOIN carFeatureMap ON car.carId=carFeatureMap.carIdMap
JOIN feature ON featureIdMap=featureId
;
/*Report4 Get the Cars with features with all the features concatendated, i.e. single output per car with features */
SELECT
car.carName,
group_concat(featureDescription,': ') AS allFeatures
FROM car
JOIN carFeatureMap ON car.carId=carFeatureMap.carIdMap
JOIN feature ON featureIdMap=featureId GROUP BY (carId)
;
/*Report5 Similar to the previous BUT if no features then output none so ALL cars are output */
SELECT
carName,
coalesce(
(
SELECT
group_concat(featureDescription)
FROM feature
JOIN carFeatureMap ON carFeatureMap.featureIdMap=featureId AND carFeatureMap.carIdMap=carId
),
'none'
) AS features
FROM car
;
/* Clean Up After Demo*/
DROP INDEX IF EXISTS carFeatureMap_idxon_feature;
DROP TABLE IF EXISTS carFeatureMap;
DROP TABLE IF EXISTS car;
DROP TABLE IF EXISTS feature;
Results from the demo code above
Report1 - The features
Report2 - The cars
**Report3 ** Cars and features
Report 4 Cars and features 2
Report 5 Cars and features 3
Is marking a primary key with #PrimaryKey(autoGenerate = true) exactly the same as if you had used PRIMARY KEY AUTOINCREMENT in an SQL statement?
Intuition tells me yes, but documentation seems to suggest no.
Room javadoc states:
Set to true to let SQLite generate the unique id.
as if setting it false will prevent SQLite from generating the key.
But SQLite documentation for AUTOINCREMENT states that SQLite always generates a currently-unique key if none is given when doing an INSERT, and that AUTOINCREMENT merely adds the additional behavior that SQLite will never allow an automatically generated key to overlap with any previously deleted row.
The SQLite documentation also recommends not using AUTOINCREMENT if it isn't needed (for performance reasons), and states that it is usually not needed. From the description, that seems to match my case. My table will be fine if a previously deleted row ID gets reused.
Is marking a primary key with #PrimaryKey(autoGenerate = true) exactly the same as if you had used PRIMARY KEY AUTOINCREMENT in an SQL statement?
Yes, as using autoGenerate=true adds the AUTOINCREMENT keyword.
But
as if setting it false will prevent SQLite from generating the key.
Is false.
If a class is:-
annotated with #Entity, and
the column/variable/member is annotated with #PrimaryKey, and
if the type resolves to an integer type
(byte .... double, primitive or Object (e.g. Double))
then the value can be generated (it is INTEGER PRIMARY KEY that makes the column a special column that can be generated as that column is then an alias of the rowid (a normally hidden column)).
AUTOINCREMENT is only applicable to aliases of the rowid (i.e. INTEGER PRIMARY KEY). It does not determine whether the value can be generated (in the absence of a value for the column or when the value is null).
What AUTOINCREMENT does is add an additional rule when generating the value. That rule being that the value MUST be higher than any ever used for that table.
There are subtle differences.
Without AUTOINCREMENT
deleting the row with the highest value, frees that value for subsequent use (and would be used to generate the value still higher than any other value that exists at that time), and
should the highest value (9223372036854775807) be reached SQLite will try to find a free lower value, and
lastly it is possible to double the range of values by using negative values.
With AUTOINCREMENT
deleting the row with the highest value does not free that value for subsequent use
should the highest value (9223372036854775807) be reached then subsequent attempts to insert with a generated value will fail with an SQLITE FULL error.
If you insert 1 row with a value of 9223372036854775807 then that's the only row that can be inserted.
negative values cannot be generated (can still be used)
an additional table is required (sqlite_sequence), which is automatically created by SQLite, that will have a row per table with AUTOINCREMENT. The highest used value is stored in the row. So whenever inserting when the value is to be generated requires the respective row to be retrieved and the value obtained, after insertion the value has to be updated. As such there are overheads associated with using AUTOINCREMENT.
Note the above is assuming that methods to circumvent SQLite's in-built handling are not circumvented (such as updating values in the sqlite_sequence table).
I would always advocate using (not using autoGenerate=true) e.g.
#PrimaryKey
Long id_column=null;
or
#PrimaryKey
var id_column: Long?=null
thus an #Insert (convenience insert) will autogenerate if no value is given for the id_column.
Demo
Consider the following two #Entity annotated classes (with and without autoGenerate=true) :-
AutoInc:-
#Entity
data class AutoInc(
#PrimaryKey(autoGenerate = true)
val id: Long?=null,
val other: String
)
NoAutoInc:-
#Entity
data class NoAutoInc(
#PrimaryKey
var id: Long?=null,
var other:String
)
Room (after compiling and looking at the generated java in the class that is the same name as the #Database annotated class) has the following in the createAllTables method/function:-
_db.execSQL("CREATE TABLE IF NOT EXISTS `AutoInc` (`id` INTEGER PRIMARY KEY AUTOINCREMENT, `other` TEXT NOT NULL)");
_db.execSQL("CREATE TABLE IF NOT EXISTS `NoAutoInc` (`id` INTEGER, `other` TEXT NOT NULL, PRIMARY KEY(`id`))");
i.e. the only difference is the AUTOINCREMENT keyword.
Then consider the following code :-
/* Typical where the id will be generated */
dao.insert(AutoInc(other = "A"))
dao.insert(AutoInc(null,other = "B"))
dao.insert(NoAutoInc(other ="A"))
dao.insert(NoAutoInc(null, other = "B"))
/* Beware */
/* Room interprets types different ways
here 0 is taken to be 0 as id is an Object
if long (Java) then 0 will be generated id
getters/setters are taken in to consideration when determining type
* */
dao.insert(AutoInc(0,other = "W"))
dao.insert(NoAutoInc(0,other ="W"))
/* Unusual */
dao.insert(AutoInc(-100,"X"))
dao.insert(NoAutoInc(-100,other ="X"))
dao.insert(AutoInc(9223372036854775807,"Y")) /* The maximum value for an id */
dao.insert(NoAutoInc(9223372036854775807,"Y")) /* The maximum value for an id */
When run then the tables (via Android Studio's App Inspection) are:-
AutInc:-
Note the Z row has not been added due to :-
E/SQLiteLog: (13) statement aborts at 4: [INSERT OR ABORT INTO `AutoInc` (`id`,`other`) VALUES (?,?)] database or disk is full
However, the disk isn't full as Disk Explorer shows:-
It's by no means full as Disk Explorer shows (and of course the subsequent step works inserting a row into the database):-
and
NoAutInc
Here the Z row has been added with a generated id based upon SQLite finding an unused value due to the highest allowable value for an id having been reached as opposed to the failure due to the disk/table full.
I have a migration where I am adding a new column
The migration is defined as
val MIGRATION_20_21: Migration = object : Migration(20, 21) {
override fun migrate(database: SupportSQLiteDatabase) {
database.execSQL("ALTER TABLE tests ADD COLUMN deleted INTEGER NOT NULL DEFAULT 0")
}
}
The field is dined as
#ColumnInfo(name = "deleted") var deleted: Boolean = false,
And I have a test case for this migration which looks like
#Test
#Throws(IOException::class)
fun migrate20to21() {
var db = helper.createDatabase(TEST_DB, 20).apply {
addTestsToDatabase(this, true)
close()
}
db = helper.runMigrationsAndValidate(TEST_DB, 21, true,
MIGRATION_20_21)
var cursor = db.query("SELECT * FROM tests WHERE id = ?", arrayOf("id0"))
MatcherAssert.assertThat(cursor, Is(notNullValue()))
MatcherAssert.assertThat(cursor.moveToFirst(), `is`(true))
MatcherAssert.assertThat(cursor.getColumnIndex("deleted"), `is`(38))
MatcherAssert.assertThat(cursor.getInt(cursor.getColumnIndex("deleted")), `is`(0))
I am getting the error Migration didn't properly handle: tests
I see a difference in expected and found table info
Expected: deleted=Column{name='deleted', type='INTEGER', affinity='3', notNull=true, primaryKeyPosition=0, defaultValue='null'}, local_created_at=Column{name='local_created_at', type='INTEGER', affinity='3', notNull=false, primaryKeyPosition=0, defaultValue='null'}
Found: deleted=Column{name='deleted', type='INTEGER', affinity='3', notNull=true, primaryKeyPosition=0, defaultValue='0'}
Why there is a difference in default value and how can it be null when I have defined it as not null.
Why there is a difference in default value and how can it be null when I have defined it as not null
The reason is that "Expected" is derived from the Entity (at compile time) and then compared to the schema of the actual database that is being/has been opened (what has been "Found"), so you need to change the Entity to also have a default value of 0, using the #ColumnInfo(defaultValue = "0").
Without then the default value will be null i.e. no default value specified.
The #ColumnInfo's defaultValue was introduced in Room 2.2.0, until that version the DEFAULT clause was (I believe) ignored and and expected/found conflict wouldn't arise.
So the Entity should contain
#ColumnInfo(defaultValue = "0")
val deleted: Int = 0
Or instead of Int = 0, Boolean = false
Amendment re update showing:-
#ColumnInfo(name = "deleted") var deleted: Boolean = false,
then you need to use :-
#ColumnInfo(name = "deleted", defaultValue = "0") var deleted: Boolean = false,
Re the comment
thank you. when I say var deleted: Boolean = false, is it not the same as saying defaultValue?
Correct it isn't the same.
Setting a value for var/val sets the value of the field when the value isn't provided when instantiating the object and effectively assigns the object a default value.
Whilst defaultValue = "0" says that the column definition includes the DEFAULT 0 in the table's column definition.
e.g.
`deleted` INTEGER NOT NULL DEFAULT 0
Without you get
`deleted` INTEGER NOT NULL
This is what Room EXPECTED (as it is without defaultValue =) in your case according to the Entity your ALTER column definition that was FOUND differs from the expected (no DEFAULT so Room's interpretation is defaultValue = null).
In your case the defaultValue = "0" is important as 0 (false) will be used as the value for the column for rows that existed before the ALTER, whilst null might result in run time issues.
using var deleted: Boolean = false, is also potentially important as it provides the default value if the value is omitted when instantiating the object.
Perhaps the best/easiest way to get things as Room expects is to make your Entity changes, then compile (e.g. Ctrl + F9) and to then inspect the generated java (easily visible from Android View in Android Studio) to then find the class that is named as per the #Database class suffixed with _Impl and then find the createAllTables method which includes the EXPECTED table definitions.
e.g.
Another Amendment
Regarding the comment
Does the migration uses the json files created by room?
No, UNLESS AutoMigration is used. AutoMigration builds the code for the migration (I believe) at compile time and hence why it needs schemas and requires exportSchema = true and the underlying gradle schema location directive.
Migrations actually take place at run time and do not need (nor even have access to the saved schemas as I believe that they are not included in the APK).
My current version is 24 but this field was added at version 21. SO even though I add default = 0, it only recreates 24.json. This test case for 20 to 21 fails as 21.json do not have default value.
I suspect that this could be due to different versions of Room. Prior to 2.2.0 defaultValue was not an option and thus it was ignored when comparing expected and found. Noting that this comparison is undertaken after migration(s) have been performed (I believe ALL migrations).
I have a SQLite DB which I cannot change or migrate and need to map a NUMERIC column to my Kotlin class using Room. What do I need to use as a data type of the field in Kotlin class? When using Boolean, Integer, Double, String, it fails with similar error message:
Expected:
TableInfo{name='calendars', columns={monday=Column{name='monday', type='REAL', affinity='4', ...}
Found:
TableInfo{name='calendars', columns={monday=Column{name='monday', type='NUMERIC', affinity='
EDIT: I suppose it's possible to update the database and change column type. But I still want to know if there is a way to map NUMERIC with Room.
If you have an existing database, open it with SQLite Browser, change the datatype there by selecting the table and editing it (changing the type to REAL). This preserves the data and afterwards you can use the database again.
Currently, Room seems to have problems mapping these datatypes, so you have to explicitly use REAL in your database.
Also it might be a good idea to annotate your field in your data object respectively.
#ColumnInfo(typeAffinity = ColumnInfo.REAL)
public double yourField;