Use my own SQLite database vs Import data from csv file - android

I'm currently trying to import ~300.000 lines saved in a file.sql.
After reading few forums, few posts stackoverflow, I found two main solutions :
Use my own database with android (https://blog.reigndesign.com/blog/using-your-own-sqlite-database-in-android-applications)
Use a csv file or sql file and import it with a transaction
How to populate a large sqlite database on first run
https://www.sqlite.org/cvstrac/wiki?p=ImportingFiles
Inserting large amount of data into android sqlite database?
What is the best solution, considering the performance aspect, the evolvment aspect (volumetry), the best practises on Android...?
In the perfect case, can I have a example ?
I'm using ORMlite libray, and classic DatabaseHelper methods like :
onCreate => to import my data
onUpgrade => to do my migrations scripts (https://riggaroo.co.za/android-sqlite-database-use-onupgrade-correctly/)
Edit : The database is used for the operations of read-write, and is synchronized with a primary server, in order to obtain and send the changes (using Flatbuffer).
The reason for which I need to import these data is because the synchronization dating of a timestamp 0 (all the lines contained in the base of data of the backend server) is too long at the level of the time.

What is the best solution, considering the performance aspect
Shipping a database in assets, and using SQLiteAssetHelper to install it as needed, should be much faster than anything involving executing database transactions.
volumetric evolution about data. the amount of my line (in file.sql), will grow.
Shipping a database in assets will be faster for deploying the app to a new user.
For existing users, neither of your approaches works, and you will need to craft your own code to blend:
Your new data that you want to ship with the app
The old data that you already shipped and have on the user's device
The data that is on the user's device that shipped with neither the original nor the updated app (e.g., data from a server, data entered by the user)

Related

App local database

Not sure, if this is important: app is written with react-native.
Question is about both iOS and android apps.
I have a static database, which contains currently ~9000 rows, each row contains 45 columns and about 280 letters total in it. So, basically, database is relatively small. I'll need to perform pattern search (equivalent of ILIKE in Postgres) and sorting based on misc columns with numeric values. No insertions, no modifications, no relations with other "Tables" required.
Should i write nodejs server, instantiate PG database, connect app through web-socket and start querying data from pg, or should i just somehow create local database in app and search through it right in app? The local db is way simpler but I'm worry about performance. If nothing wrong with performance, then what if the database will grow to 15000? 50000?
If the database can be local and there's no need for it to ever be persisted on servers, then you should probably explore Realm. I have not heard of how it great it performs with a lot of data but it acts as a database engine in your phone and has indexing as well.

How is android.database different from android.database.sqlite?

I am trying to develop this android application whose database is not stored in the device but on a server. To use sqlite database I need to use android.database.sqlite; and I haven't made much use of android.database package. I went through the documentation site but it did not state clearly if it's possible can store my sqlite database file on server and invoke methods to access it from there. What should I do?
SQLite is not an ideal database to use on a server. The andriod.database.sqlite package is going to have resources for dealing with sqlite specifically, where the android.database package is going to have generic database resources, not specific to sqlite.
For using sqlite on a server, first bear in mind that concurrent connections are a big no-no... sqlite is not designed to handle them.
If I still felt I had to do this, I'd probably look at creating an adapter on the server which would accept requests from the application, then use the android.database.sqlite package on the server, as it's local per the scope there (as sqlite is designed for).
https://www.sqlite.org/serverless.html

What is difference between SQLite and SQL

I know SQLite Data Base is used in mobile devices (Android, iPhone) and it is light, takes only Kb space. Is there any limitation in SQLite? I want to know how they are different.
Every SQL database uses its own implementation of the language that varies slightly. While basic queries are almost universal, there are notable nuances between MySQL, PostgreSQL, Microsoft SQL Server, Oracle Database, etc.
What's particularly notable about SQLite is that unlike all the others mentioned above, this database software doesn't come with a daemon that queries are passed through. This means that if multiple processes are using the database at once, they will be directly altering the data through the SQLite library and making the read / write data calls to the OS themselves. It also means that the locking mechanisms don't deal with contention very well.
This isn't a problem for most applications that where one would think of using SQLite -- the small overhead benefits and easy data retrieval are worth it. However, if you'll be accessing your database with more than one process or don't consider mapping all your requests through one thread, it could be slightly troublesome.
Sqlite is very light version of SQL supporting many features of SQL. Basically is been developed for small devices like mobile phones, tablets etc.
SQLite is a third party ,open-sourced and in-process database engine. SQL Server Compact is from Microsoft, and is a stripped-down version of SQL Server.They are two competing database engines.
SQL is query language. Sqlite is embeddable relational database management system.
Edit : ( Source from following comment on my answer )
Sqlite also doesn't require a special database server or anything. It's just a direct filesystem engine that uses SQL syntax. ( By : Adam Plocher )
Techinically, SQLite is not open-source software but rather public domain. There is no license. ( By : Larry Lustig )
SQL is a query language. SQLite is an embeddable relational database management system.
Unlike other databases (like SQL Server and MySQL) SQLite does not support stored procedures.
SQLite is file-based, unlike other databases, like SQL Server and MySQL which are server-based.
What is SQLite?
SQLite is an open-source, zero-configuration, self-contained, stand-alone, transaction relational database engine designed to be embedded into an application.
Python SQLite can be defined as a C Library developed using ANSI-C, light-weight disc based database; doesn't demand for an extra or any other separate server process.
What are some SQLite characteristic?
SQLite does NOT require a server to run (RDBMS such as MySQL, PostgreSQL, etc., requires a separate server process to operate).
Is self-contained, it requires minimal support from the operating system or external library. This makes SQLite usable in any environment especially in embedded devices like iPhones, Android phones, game consoles, handheld media players.
Does not use any configuration files.
Is ACID-compliant. It means all queries and changes are Atomic, Consistent, Isolated, and Durable, all changes within a transaction take place completely or not at all even when an unexpected situation like application crash, power failure, or operating system crash occurs.
Capable of creating in-memory databases that are very fast to work with.
Uses dynamic types for tables. It means you can store any value in any column, regardless of the data type.
Allows a single database connection to access multiple database files simultaneously.
SQLite & ACID
SQLite guarantees all the transactions are ACID compliant even if the transaction is interrupted by a program crash, operation system dump, or power failure to the computer.
Atomic: a transaction should be atomic. It means that a change cannot be broken down into smaller ones. When you commit a transaction, either the entire transaction is applied or not.
Consistent: a transaction must ensure to change the database from one valid state to another. When a transaction starts and executes a statement to modify data, the database becomes inconsistent. However, when the transaction is committed or rolled back, it is important that the transaction must keep the database consistent.
Isolation: a pending transaction performed by a session must be isolated from other sessions. When a session starts a transaction and executes the INSERT or UPDATE statement to change the data, these changes are only visible to the current session, not others. On the other hand, the changes committed by other sessions after the transaction started should not be visible to the current session.
Durable: if a transaction is successfully committed, the changes must be permanent in the database regardless of the condition such as power failure or program crash. On the contrary, if the program crashes before the transaction is committed, the change should not persist.
Credit Sqlitetutorial.net
The most basic difference between SQLite and SQL is :
SQL is a query language which is used by different SQL databases. It is not a database itself.
SQLite is a database management system itself which uses SQL.
SQL is a database querying language and SQLite is a database (RDBMS) which uses SQL specifications. SQLite can be said as competitor to Microsoft's SQL Server.
The name itself suggests that it is the light version of SQL RDBMS. It is used in most of the small and portable devices like Android and iOS devices.
SQLite: Database Management System (DBMS).
SQL: Structured Query Language is a computer language, used to Create, edit and get data from DBMS via queries.

Synchronise/update sqlite databases

We've got an android app and an iPhone app (same functionality) that use sqlite for local data storage. The apps initially come with no data, then on the first run they receive data from a remote server and store it in a sqlite database. The sqlite database is created by the server and the apps download it as one file, which is then used buy the apps. The database file is not very large by today's standards, but not a tiny one either - about 5-6 MB.
Now, once in a while, the apps need to refresh the data from the server. There a few approaches I can think of:
Download a new full database from the server and replace the existing one. This one sounds like the simplest way to deal with the problem were it not for a repeated 5-6 MB downloads. The apps do prompt the user whether they want to download the updates, so this may not be too much of a problem.
Download a delta database from the server, containing only the new/modified records and in some form information about what records to delete. This would lead to a much smaller download size, but the work on the client side is more complicated. I would need to read one database and, based on what is read, update another one. To the best of my knowledge, there's not way with sqlite to do something like insert into db1.table1 (select * from db2.table1) where db1 and db2 are two sqlite databases containing table1 of the same structure. (The full sqlite database contains about 10 tables with the largest one probably containing about 500 records or so.)
Download delta of the data in some other format (json, xml, etc.) and use this info to update the database in the app. Same as before: not to much problem on the server side, smaller download size than the full database, but quite a painful process to do the updates.
Which of the three approaches you recommend? Or maybe there's yet another way that I missed?
Many thanks in advance.
After much considerations and tries-and-errors, I went for a combination of options (2) and (3).
If no data is present at all, then the app downloads a full database file from the server.
If data is present and an update is required, the app downloads some database from the server. And checks the content of a particular value in a particular table. That value will state whether the new database is to replace the original or whether it contains deletions/updates/inserts
This turns out to be the fastest way (performance-wise) and leaves all the heavy lifting (determining whether to put everything into one database or just an update) to the server. Further, with this approach, if I need to modify the algorithm to, say, always download the full database, it would only be a change on the server without the need to re-compile and re-distribute the app.
Is there a way you can have a JSON field for each of the tables? For instance, if you got a table named users, have a column named "json" that stores the JSON for each of the users. In essence, it would contain the information the rest of the fields have.
So when you download the delta in JSON, all you got to do is insert the JSON's into the tables.
Of course with this method, you will need to do additional work in parsing the JSON and creating the model/object from it, but it's just an extra 3-4 small steps.
I will recommend approach 3, because app will download the json file more fast and local db will be updated more easily avoid overhead of more internet usages.
Just create a empty db initially according to server db and then regularly updated the same by fetching json

Android database approach for future updates

What is the best way to work with the sqlite database in android?
The sqlite database file (copy it for the first time into the application environment)
OR
Creating the tables in code (in database helper's onCreate())
My database has 6 tables and it is empty for the first time. I ask this because I want to update my database in the future and would want to know the best approach for this.
Thank you!
You should create (in code) it the first time it is used. Android offers the SQLiteOpenHelper class that should be used for it. SQLiteOpenHelper defines the following methods:
onCreate(SQLiteDatabase db): invoked when the database is created, this is where you can create tables and columns to them, create views or triggers.
onUpgrade(SQLiteDatabse db, int oldVersion, int newVersion): Invoked if the used database is older than the current version. Handle here the upgrade stuff (data migration, table creation/deletion)
See here for a good tutorial: http://www.codeproject.com/KB/android/AndroidSQLite.aspx
If you don't expect to have your db updated by user interaction, the file might be the best option, especially in the case that you have a lot of data to insert (file copying vs a bunch of inserts).
On the other hand, if you expect to have some data altered or added by the user, the file approach will work only in the first release.
Any time you will need to update your schema or add new data (releasing an upgrade), you will need to consider that the existing data might be changed or enriched by some stuff that the users will expect to find AFTER the upgrade.
So replacing the file is not an option anymore.
In case you need to use the sqllite helper approach, I'd love to hear some feedbacks on my sqllite helper code generator that you can find here: github
Not specific to SQLLite or android, however I have worked on a Windows trading application where users could save down Xml 'documents' - ie: a custom view saving their reporting preferences and various other flags which could then be shared around the team. On startup a user's profile was loaded and their documents parsed to customize the UI.
The application was to have a release every 3 weeks and existing documents needed to work with the new application. This was a problem as occasionally the XML schema changed resulting in new or deleted fields.
The solution we came up with was to create an abstract type called Patcher. Each release could have one or more DerivedPatcher types with it which were run on the first load after an update. The Patcher would have an abstract method to patch the XMl documents. Meaning an XML document would be loaded in with the old schema and upgraded, saved back in-place using the new schema. The Patcher would also have a rollback method to allow unrolling if an error occurred.
The same approach could be applied to tables in a database. Basically if you can create a patcher or PatchManager to serialize key tables to XML in memory, then apply the DB changes and write the data back, you can achieve database migration in a generic, re-usable way.
A key benefit of this method is it can be tested on developer PCs before deployment, so long as you have access to some sample SQLLite data. Knowing how your users use your application is key here.
For large amounts of data you might want to consider this kind of solution: Create an empty database in code and provide an activity which responds to an intent with this action: android.intent.action.SEND. Respond by parsing the sent file and populate the database with the contents. Design a format which can be easily parsed (XML is not needed for everything ;-) so the code to parse the file and fill the database is small (my binary for this including an UI to show progress (which is the larger part of the activity) is less than 12 kB in size).
The file may be distributed separately (extra apk, download, whatever). The benefit of this approach is that you do not need to store your initial database content within the apk and thus the data is only stored once on the device (after the file has been deleted). Otherwise you have the data in the database plus the source code or asset in the apk.

Categories

Resources