How does a CSV database compare to an SQLite database in Android?
Looking at other questions on StackOverflow, and reading Android Developer Documentation, I have seen SQLite databases being used far more often than reading data from a CSV file. There are also some questions where users have wanted to import a CSV file into an SQLite database (for example, this question or this one). Is there an advantage to using SQLite over CSV?
I have tried using both CSV and SQLite a little, and in terms of performance I don't see a huge difference, but correct me if I'm wrong here.
As I know there are different methods of reading a CSV file, I opened and read it using the BufferedReader like so:
BufferedReader reader =
new BufferedReader(new InputStreamReader(context.getAssets().open(fileName)));
And the SQLite database was opened in the usual way:
SQLiteDatabase db = helper.getReadableDatabase();
I'm not too sure about differences in functionality, although I am assuming SQLite is easier to manage and filter through, and that's why I'm asking this question.
So to summarise:
Are any of the two faster in terms of performance?
Does SQLite (or CSV) have any additional functionality that the other does not, especially in Android (as I am aware Android has its own SQLiteDatabase class?
Why does it seem that SQLite is used far more than CSV databases (i.e. reading and filtering a CSV file)?
EDIT:
Just to clarify, I know a CSV file is just a file with commas separating values, i.e. not a 'database' as a SQL database is. However I can still use a CSV file as a sort of database, where the comma separated values indicate different columns, which can also be filtered, by checking to see if a particular column matches a particular value. So I am asking which is better to read data from.
SQLite and CSV are not really the same thing, which makes the comparison rather difficult. SQLite is a (limited) SQL database, which means that you can use it to store structured and related data in a way that ensures consistency (related records not messing up when either is deleted) and simple way.
Even for flat data, SQLite makes it easier to query a subset of your data (say, all records where one of the columns is less than three), reorder the data depending on your needs and insert a record somewhere in the middle.
CSV on the other hand, is just a text file, that you can use for very simple data. It has little overhead, but it won't help you very much in the sense of data integrity and easy querying. If you don't need this, and always read all of it, sure, you can use CSV.
If none of this rings any bells to you (related records, and data integrity and such) I suggest reading up on relational database systems. As always, Wikipedia is an excellent source.
Related
Trying to make my first Android app. Excited to be asking my first question here! I'm no longer a young dog, so any encouragement is much appreciated.
I am rewriting a simple data driven webpage I wrote in the past. The data is currently stored in a SQL database (populated from a CSV) I select the data, apply some logic as I iterate the records, and basically create an HTML table. Let's just say the table displays some information about various cars.
My question is relating to the KISS approach to storing this data. There are only 75 records in the table. I just select them all, and iterate through the fetch array.
For the app, should I just parse the CSV using some kind of FileStreamReader and parsing classes, and populate some sort of collection of Car objects?
Should I translate this data into an XML and parse the XML?
Worth trying to use SQLite? Or best to avoid? (I really don’t like over-engineered approaches to anything)
I recomend create a database on SQLite. It offers some advantages for example the data types, because it only has 4 types. The control of the data is easy.
XML generates files which can overload the device depending how much they are
I ended up converting my flat file into an XML file. I don't see any advantage this offered except for complicating my code. I should have just parsed the flat file and created an array of objects directly from the flat file records.
In hindsight, SQL was clearly overkill for this situation.
the scenario is that -- in my application client would make dynamic edit text (of which i m not sure ), but i want to store all the information he creates. All other part have been done.
I just want to know which is the best way to store them in File OR in Database.
As if some says in Database, i want to know how?
and if in File, why?
I will encrypt and decry pt the data too.
Any help would be appreciable.
Thanks in advance.
Store the data in a database. If you're doing Android development, simply use SQLite - http://www.sqlite.org/.
There are many reasons not to store the data on the file system:
It is easier to query a database than a file system. You can quickly and easily answer questions like "How many records do I have?" or "How many items in a given category do I have?"
If you do this in the file system, you'll have to write your own code around querying.
SQLite has strong data types that enforce a schema. If you just write to a file, you'll have to ensure an ID is an integer etc. The database can do this for you.
From my experience when writing files to the file system, I've always ended up with orphaned files. You write them and then forget they are there and they never get deleted. It's annoying. With a database you can easily assess the state of the database and remove old/unused records.
I need to download onto my devices some data in multiple files.
Then this data will be copied to application's local db (this is SQLite db, however in future this may be Compact SQL on WInPhone).
What is the best format for such files?
I am considering such possibilities:
SQLited db file - possibly this will be easy to copy to my db. My current prefferance.
JSON format. Maybe not enough compact because column name will be repeating.
CSV - it allows to store only one table but I would prefer have few tables in one file
XML - I do not see any prefferaces over json.
JSON is the most popular, human-readable, easy to use format. There's tons of supporting libraries, native and not, for all OSs. It's fast and reliable. You can easily update the data you pass with it without updating the apps (which you cannot by passing an SQLite database and would be difficult with a CSV file). XML is being slowly deprecated for data communications... but if you see some special advantage with XML (parsing the XML directly, which is not as effective with JSON yet, for example), go for it. I'd choose JSON anyway, it's the current standard and will still be for a long time.
I want to know how exactly does sqllite works when you are dealing with database on android. I know that it writes everything on file with .db extension. But how does it read or write one particular table? Does it fetch the whole file or just the related part and how exactly does it do these operations? Can someone please suggest me some link? I tried google but the links I found just explain how to write queries.
For that you have to read basics of the databases .
all the db frameworks are almost same in terms of working so
you have to research on the basics of database (any).
here is some related information u can like
What does a database actually do to find out what matches a select statement?
To be blunt, it's a matter of brute force. Simply, it reads through each candidate record in the database and matches the expression to the fields. So, if you have "select * from table where name = 'fred'", it literally runs through each record, grabs the "name" field, and compares it to 'fred'.
Now, if the "table.name" field is indexed, then the database will (likely, but not necessarily) use the index first to locate the candidate records to apply the actual filter to.
This reduces the number of candidate records to apply the expression to, otherwise it will just do what we call a "table scan", i.e. read every row.
But fundamentally, however it locates the candidate records is separate from how it applies the actual filter expression, and, obviously, there are some clever optimizations that can be done.
How does a database interpret a join differently to a query with several "where key1 = key2" statements?
Well, a join is used to make a new "pseudo table", upon which the filter is applied. So, you have the filter criteria and the join criteria. The join criteria is used to build this "pseudo table" and then the filter is applied against that. Now, when interpreting the join, it's again the same issue as the filter -- brute force comparisons and index reads to build the subset for the "pseudo table".
How does the database store all its memory?
One of the keys to good database is how it manages its I/O buffers. But it basically matches RAM blocks to disk blocks. With the modern virtual memory managers, a simpler database can almost rely on the VM as its memory buffer manager. The high end DB'S do all this themselves.
How are indexes stored?
B+Trees typically, you should look it up. It's a straight forward technique that has been around for years. It's benefit is shared with most any balanced tree: consistent access to the nodes, plus all the leaf nodes are linked so you can easily traverse from node to node in key order. So, with an index, the rows can be considered "sorted" for specific fields in the database, and the database can leverage that information to it benefit for optimizations. This is distinct from, say, using a hash table for an index, which only lets you get to a specific record quickly. In a B-Tree you can quickly get not just to a specific record, but to a point within a sorted list.
The actual mechanics of storing and indexing rows in the database are really pretty straight forward and well understood. The game is managing buffers, and converting SQL in to efficient query paths to leverage these basic storage idioms.
Then, there's the whole multi-users, locking, logging, and transactions complexity on top of the storage idiom.
SQLite operation on Android is not any different from SQLite operation on any other platform.
Very short answer to your question: SQLite file is split into pages of fixed size.
Each database object (table, index, etc) occupies some number of pages. If objects needs to grow (like new rows are inserted into table) it may allocate more new pages either from free page list, or by growing database file in size. If rows are deleted or object dropped, reclaimed free space goes into free page list. During any operation, SQLite engine tries to NOT fetch whole file, however it maintains page cache for higher performance.
You can find much more detailed explanations on SQLite website in general, and about SQLite database file format in particular.
I'm creating my first android app that will make use of SQlite. I have zero experience with databases, except for creating a mysql database to use with wordpress...
Edit: After doing some research about rest, I'm still confused about how rest, sqlite, and android dev fit together. My goal is to access a rest-based web service through a url and access certain datasets, then store them in my SQlite database. Then I want to access the contents of the database through my java program, and use them accordingly.
The datasets can be downloaded individually in CSV format, but because I will be using so many of them, I don't want to go through every line individually and store them in the database. I'm hoping there's a more efficient way to store these datasets in the database.
My main questions are:
How can I copy the XML contents of a webpage from a url into my sqlite database? Can I do this with my java program, through the sqlite database, or a java library?
Do I only need to copy the contents of the webpages from the url into the sqlite database one time? If so, what can I do if any information is changed in the datasets?
You first need a schema for your sqllite DB. That schema should map to the objects behind the web service. For e.g, you need a Person table in your DB if there is a Person entity on the web. It depends on what all you want to capture.
When you are done designing the schema, you should start writing the code that help you create & manage DB on android. This is done with the help of SQLiteOpenHelper class:
http://developer.android.com/reference/android/database/sqlite/SQLiteOpenHelper.html
If you need to keep the DB synce'd with the data on the cloud (web services), you should implement sync. Android provides a very efficient sync framework.
Also, do watch this video from Android engineers explaining the best practices: http://www.youtube.com/watch?v=xHXn3Kg2IQE
Note, to actually fetch the data from the web service you would use UrlConnection API:
http://developer.android.com/reference/java/net/URLConnection.html
This sample probably captures most of it.
http://developer.android.com/resources/samples/SampleSyncAdapter/index.html
In terms of reading CSV files, there are some good resources here:
Can you recommend a Java library for reading (and possibly writing) CSV files?
Once you have read each CSV line into an object, then you can turn around and persist it to the database. I'm the author of ORMLite so I'll talk about using it. I don't believe there is a hibernate port for Android.
There are a number of Android examples to help you to get up to speed with ORMLite. Also some good tutorials. If you want to write a number of rows at once then I'd recommend using the batch tasks ORMLite feature. For information, see the discussion about creating lists of objects on the mailing list.
I can answer your first question about " I'm not sure how to add them efficiently"?
yes, SQlite is very powerful and intelligent, you can add thousand of records in one transaction, just like traditional database, It significantly improve performance.
about second question, as my understanding, because CVS file is very simple, so you can download and analyze it by yourself.