Android: Migrating from legacy SQLite to Room database using attach - android

I'm working on an Android app where we need to migrate from some legacy SQLite code to a Room based implementation. But I've run into a couple of issues & I'm not sure how to solve them.
Environment:
Android 6.0+
SQLite implementation is in one database file
Room implementation is in a separate database file
Both will be in /data/user/0/.../databases/
What sorta works, but is slow:
We have a working test implementation to pull all the legacy SQLite tables' data into Kotlin objects, then do inserts into the Room based tables. But, it's fairly slow for our test cases (about 70 secs for 10k+ rows in at least one table). Real life cases could top 100k rows in at least one table. As far as I can tell, the select is one transaction and the inserts are wrapped in a transaction. So, I don't think it's transaction overhead slowing us down.
What we want to make work:
What I'm hoping to do is use SQLite's attach database and detach commands to load the legacy SQLite database into a Room DB connection. Then, I can do an insert into new_table.table_name select * from old_table.table_name. The actual SQL will be slightly different due to slight schema changes. But, the insert-select pattern will be the basic idea.
I'm using ContextWrapper to get the full database path for the legacy database file. That works fine.
What issues I'm seeing:
When I try to do the attach command, though, I was getting an IllegalStateException with something about not being able to enable/disable write-ahead logging while in a transaction or the db is open. But, the legacy DB is not open or in a transaction at that point.
So, I modified the approach to set the journal mode to TRUNCATE in the Room db connection setup. That allowed me to do the attach command. And the insert-select query seems to work as far as I can tell (I haven't fully tested that yet). BUT, when I tried the detach command, I got a SQLite 1 error about the database being locked.
For reference, my basic code for both above attach approaches is:
roomDb.execSQL("attach database '$fullLegacyDbPath' as old_db")
roomDb.execSQL($insertSelectQuery) // the "INSERT INTO ... SELECT ..." query
roomDb.execSQL("detach old_db")
Any ideas how I make this work (preferably without setting the journal mode to truncate)?

Related

Android room database is not inserting data in db immediately

I was using room in my android project, everything was working fine until one day I found Room doesn't insert data in db immediately.
How I reproduced it
When I run command appdatabase.getUserDao().insert(user) and after 2 seconds when i remove my battery from my phone and export db after that What I found is data is not inserted in db. that is 100% sure because my team has reproduced it multiple times. it seems instead of directly store into db it caches data for some time.
Is there any solution for it Where data will directly store in db instead it caches?
Yes, close the database before exporting:
RoomDatabase.close()
Make sure it is the
Room.databaseBuilder()
and not
Room.inMemoryDatabaseBuilder()
Check if the thread is executing properly and that you are not interrupting a #Transaction insert

Updating/Maintaining SQLite database after each App Release Xamarin Forms

This is my first time working on a Xamarin App and I am new to the app development world so I need some help figuring out this process.
Currently I run a php web service that generates some SQL files that I run in DB Browser and I get a database file which I then put into my Assets and Resources Folder. Using each platform's API I copy the database into a writable folder and use that to run my queries.
I followed this really helpful tutorial and it worked perfectly fine.
https://medium.com/#hameedkunkanoor/creating-a-sqlite-databse-and-storing-your-data-in-your-android-and-ios-application-in-xamarin-2ebaa79cdff0 .
After the "initial" setup I store a timestamp in a local table and and the next time the user opens the app I pass that timestamp and retrieve data that is older than that timestamp. The I update that timestamp and continue the process. That data is sent back in JSON format and make the updates to the tables.
My only concern is if a new version were to come out where I add a new table or a new column which is not present in the current version of my Database, how should I take care of those update Web Service calls? Is there a way of monitoring my DB version? I read somewhere where I could just ignore the new data that is not present already, like table or columns, but I'm not really sure how to do that.
I also saw that if I call CreateTable on my current tables I could potentially update them?
Also for future reference each time I develop a new app would I need to regenerate a new database file to store in the assets/resources folder? Is there a more automated process for this? Along with monitoring the version of my database?
Any Help/Tutorials/Suggestions would be greatly appreciated.
You have to remember that CreateTable it's already doing the columns update for you, because internally it calls a method called MigrateTable which you can see here for further clarification: https://github.com/praeclarum/sqlite-net/blob/master/src/SQLite.cs#L562.
However you could have to handle more advanced modification to your database, like adding triggers or something similar.
In that case i suggest you to perform modifications manually.
In Xamarin Forms i've ended up with this:
https://gist.github.com/matpag/b2545cc22c8e22449cd7eaf6b4910396
Could not be the best strategy ever but seems to work for me.
Summarizing :
You have to save the database version in an internal flag of the SQlite database called user_version accessible with PRAGMA keyword.
Every time you get the database connection, you have to perform a check and see if the current database version is the same as the app last database version.
If not you need to perform a database update and set the new current version.
Reference here.

Synchronize local SQLite database with central database

We are developing an Android App based on a product that already has a website. As a result, when launching the Android app it may happen that a lot of things differ between the local SQLite database and our central database.
What is the fastest way to synchronize the Android App?
Compare the data in our central database with the SQLite database and update/insert/delete accordingly (for each table)?
Or just drop the concerning tables and create them again - and execute a transaction with compiled statements?
Sounds like the first option may be very slow.
Any help would be appreciated!
In your case needs value the situation!! When I have this problems the first questions are...
The tables that I want to do 'drop table' have few data? If it's yes the best way is the 'drop table' command. If it's no then you need to use the 'insert' and 'update' commands. I need to remark that it's possible that the SQLite database grows with time, then the good programmer would second way because it's more secure and you think with future possibilities.
My app syncs any time or syncs a few times a day? If the answer is in any time then you need use the 'insert' and 'update' commands. Because you can't delete database tables every minutes or seconds.
Tell me if I helped you and good programming!

What is difference between SQLite and SQL

I know SQLite Data Base is used in mobile devices (Android, iPhone) and it is light, takes only Kb space. Is there any limitation in SQLite? I want to know how they are different.
Every SQL database uses its own implementation of the language that varies slightly. While basic queries are almost universal, there are notable nuances between MySQL, PostgreSQL, Microsoft SQL Server, Oracle Database, etc.
What's particularly notable about SQLite is that unlike all the others mentioned above, this database software doesn't come with a daemon that queries are passed through. This means that if multiple processes are using the database at once, they will be directly altering the data through the SQLite library and making the read / write data calls to the OS themselves. It also means that the locking mechanisms don't deal with contention very well.
This isn't a problem for most applications that where one would think of using SQLite -- the small overhead benefits and easy data retrieval are worth it. However, if you'll be accessing your database with more than one process or don't consider mapping all your requests through one thread, it could be slightly troublesome.
Sqlite is very light version of SQL supporting many features of SQL. Basically is been developed for small devices like mobile phones, tablets etc.
SQLite is a third party ,open-sourced and in-process database engine. SQL Server Compact is from Microsoft, and is a stripped-down version of SQL Server.They are two competing database engines.
SQL is query language. Sqlite is embeddable relational database management system.
Edit : ( Source from following comment on my answer )
Sqlite also doesn't require a special database server or anything. It's just a direct filesystem engine that uses SQL syntax. ( By : Adam Plocher )
Techinically, SQLite is not open-source software but rather public domain. There is no license. ( By : Larry Lustig )
SQL is a query language. SQLite is an embeddable relational database management system.
Unlike other databases (like SQL Server and MySQL) SQLite does not support stored procedures.
SQLite is file-based, unlike other databases, like SQL Server and MySQL which are server-based.
What is SQLite?
SQLite is an open-source, zero-configuration, self-contained, stand-alone, transaction relational database engine designed to be embedded into an application.
Python SQLite can be defined as a C Library developed using ANSI-C, light-weight disc based database; doesn't demand for an extra or any other separate server process.
What are some SQLite characteristic?
SQLite does NOT require a server to run (RDBMS such as MySQL, PostgreSQL, etc., requires a separate server process to operate).
Is self-contained, it requires minimal support from the operating system or external library. This makes SQLite usable in any environment especially in embedded devices like iPhones, Android phones, game consoles, handheld media players.
Does not use any configuration files.
Is ACID-compliant. It means all queries and changes are Atomic, Consistent, Isolated, and Durable, all changes within a transaction take place completely or not at all even when an unexpected situation like application crash, power failure, or operating system crash occurs.
Capable of creating in-memory databases that are very fast to work with.
Uses dynamic types for tables. It means you can store any value in any column, regardless of the data type.
Allows a single database connection to access multiple database files simultaneously.
SQLite & ACID
SQLite guarantees all the transactions are ACID compliant even if the transaction is interrupted by a program crash, operation system dump, or power failure to the computer.
Atomic: a transaction should be atomic. It means that a change cannot be broken down into smaller ones. When you commit a transaction, either the entire transaction is applied or not.
Consistent: a transaction must ensure to change the database from one valid state to another. When a transaction starts and executes a statement to modify data, the database becomes inconsistent. However, when the transaction is committed or rolled back, it is important that the transaction must keep the database consistent.
Isolation: a pending transaction performed by a session must be isolated from other sessions. When a session starts a transaction and executes the INSERT or UPDATE statement to change the data, these changes are only visible to the current session, not others. On the other hand, the changes committed by other sessions after the transaction started should not be visible to the current session.
Durable: if a transaction is successfully committed, the changes must be permanent in the database regardless of the condition such as power failure or program crash. On the contrary, if the program crashes before the transaction is committed, the change should not persist.
Credit Sqlitetutorial.net
The most basic difference between SQLite and SQL is :
SQL is a query language which is used by different SQL databases. It is not a database itself.
SQLite is a database management system itself which uses SQL.
SQL is a database querying language and SQLite is a database (RDBMS) which uses SQL specifications. SQLite can be said as competitor to Microsoft's SQL Server.
The name itself suggests that it is the light version of SQL RDBMS. It is used in most of the small and portable devices like Android and iOS devices.
SQLite: Database Management System (DBMS).
SQL: Structured Query Language is a computer language, used to Create, edit and get data from DBMS via queries.

Android database approach for future updates

What is the best way to work with the sqlite database in android?
The sqlite database file (copy it for the first time into the application environment)
OR
Creating the tables in code (in database helper's onCreate())
My database has 6 tables and it is empty for the first time. I ask this because I want to update my database in the future and would want to know the best approach for this.
Thank you!
You should create (in code) it the first time it is used. Android offers the SQLiteOpenHelper class that should be used for it. SQLiteOpenHelper defines the following methods:
onCreate(SQLiteDatabase db): invoked when the database is created, this is where you can create tables and columns to them, create views or triggers.
onUpgrade(SQLiteDatabse db, int oldVersion, int newVersion): Invoked if the used database is older than the current version. Handle here the upgrade stuff (data migration, table creation/deletion)
See here for a good tutorial: http://www.codeproject.com/KB/android/AndroidSQLite.aspx
If you don't expect to have your db updated by user interaction, the file might be the best option, especially in the case that you have a lot of data to insert (file copying vs a bunch of inserts).
On the other hand, if you expect to have some data altered or added by the user, the file approach will work only in the first release.
Any time you will need to update your schema or add new data (releasing an upgrade), you will need to consider that the existing data might be changed or enriched by some stuff that the users will expect to find AFTER the upgrade.
So replacing the file is not an option anymore.
In case you need to use the sqllite helper approach, I'd love to hear some feedbacks on my sqllite helper code generator that you can find here: github
Not specific to SQLLite or android, however I have worked on a Windows trading application where users could save down Xml 'documents' - ie: a custom view saving their reporting preferences and various other flags which could then be shared around the team. On startup a user's profile was loaded and their documents parsed to customize the UI.
The application was to have a release every 3 weeks and existing documents needed to work with the new application. This was a problem as occasionally the XML schema changed resulting in new or deleted fields.
The solution we came up with was to create an abstract type called Patcher. Each release could have one or more DerivedPatcher types with it which were run on the first load after an update. The Patcher would have an abstract method to patch the XMl documents. Meaning an XML document would be loaded in with the old schema and upgraded, saved back in-place using the new schema. The Patcher would also have a rollback method to allow unrolling if an error occurred.
The same approach could be applied to tables in a database. Basically if you can create a patcher or PatchManager to serialize key tables to XML in memory, then apply the DB changes and write the data back, you can achieve database migration in a generic, re-usable way.
A key benefit of this method is it can be tested on developer PCs before deployment, so long as you have access to some sample SQLLite data. Knowing how your users use your application is key here.
For large amounts of data you might want to consider this kind of solution: Create an empty database in code and provide an activity which responds to an intent with this action: android.intent.action.SEND. Respond by parsing the sent file and populate the database with the contents. Design a format which can be easily parsed (XML is not needed for everything ;-) so the code to parse the file and fill the database is small (my binary for this including an UI to show progress (which is the larger part of the activity) is less than 12 kB in size).
The file may be distributed separately (extra apk, download, whatever). The benefit of this approach is that you do not need to store your initial database content within the apk and thus the data is only stored once on the device (after the file has been deleted). Otherwise you have the data in the database plus the source code or asset in the apk.

Categories

Resources