As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I really like Xml for saving data, but when does sqlite/database become the better option? eg, when the xml has more than x items or is greater than y MB?
I am coding an rss reader and I believe I made the wrong choice in using xml over a sqlite database to store a cache of all the feeds items. There are some feeds which have an xml file of ~1mb after a month, another has over 700 items, while most only have ~30 items and are ~50kb in size after a several months.
I currently have no plans to implement a cap because I like to be able to search through everything.
So, my questions are:
When is the overhead of sqlite/databases justified over using xml?
Are the few large xml files justification enough for the database when there are a lot of small ones, though even the small ones will grow over time? (a long long time)
updated (more info)
Every time a feed is selected in the GUI I reload all the items from that feeds xml file.
I also need to modify the read/unread status which seems really hacky when I loop through all nodes in the xml to find the item and then set it to read/unread.
Man do I have experience with this. I work on a project where we originally stored all of our data using XML, then moved to SQLite. There are many pros and cons to each technology, but it was performance that caused the switchover. Here is what we observed.
For small databases (a few meg or smaller), XML was much faster, and easier to deal with. Our data was naturally in a tree format, which made XML much more attractive, and XPath allowed us to do many queries in one simple line rather than having to walk down an ancestry tree.
We were programming in a Win32 environment, and used the standard Microsoft DOM library. We would load all the data into memory, parse it into a DOM tree and search, add, modify on the in memory copy. We would periodically save the data, and needed to rotate copies in case the machine crashed in the middle of a write.
We also needed to build up some "indexes" by hand using C++ tree maps. This, of course would be trivial to do with SQL.
Note that the size of the data on the filesystem was a factor of 2-4 smaller than the "in memory" DOM tree.
By the time the data got to 10M-100M size, we started to have real problems. Interestingly enough, at all data sizes, XML processing was much faster than SQLite turned out to be (because it was in memory, not on the hard drive)! The problem was actually twofold- first, loadup time really started to get long. We would need to wait a minute or so before the data was in memory and the maps were built. Of course once loaded the program was very fast. The second problem was that all of this memory was tied up all the time. Systems with only a few hundred meg would be unresponsive in other apps even though we ran very fast.
We actually looking into using a filesystem based XML database. There are a couple open sourced versions XML databases, we tried them. I have never tried to use a commercial XML database, so I can't comment on them. Unfortunately, we could never get the XML databases to work well at all. Even the act of populating the database with hundreds of meg of XML took hours.... Perhaps we were using it incorrectly. Another problem was that these databases were pretty heavyweight. They required Java and had full client server architecture. We gave up on this idea.
We found SQLite then. It solved our problems, but at a price. When we initially plugged SQLite in, the memory and load time problems were gone. Unfortunately, since all processing was now done on the harddrive, the background processing load went way up. While earlier we never even noticed the CPU load, now the processor usage was way up. We needed to optimize the code, and still needed to keep some data in memory. We also needed to rewrite many simple XPath queries as complicated multiquery algorithms.
So here is a summary of what we learned.
For tree data, XML is much easier to query and modify using XPath.
For small datasets (less than 10M), XML blew away SQLite in performance.
For large datasets (greater than 10M-100M), XML load time and memory usage became a big problem, to the point that some computers become unusable.
We couldn't get any opensource XML database to fix the problems associated with large datasets.
SQLite doesn't have the memory problems of XML DOM, but it is generally slower in processing the data (it is on the hard drive, not in memory). (note- SQLite tables can be stored in memory, perhaps this would make it as fast.... We didn't try this because we wanted to get the data out of memory.)
Storing and querying tree data in a table is not enjoyable. However, managing transactions and indexing partially makes up for it.
I basically agree with Mitchel, that this can be highly specific depending on what are you going to do with XML and SQLite. For your case (cache), it seems to me that using SQLite (or other embedded databases) makes more sense.
First I don't really think that SQLite will need more overhead than XML. And I mean both development time overhead and runtime overhead. Only problem is that you have a dependence on SQLite library. But since you would need some library for XML anyway it doesn't matter (I assume project is in C/C++).
Advantages of SQLite over XML:
everything in one file,
performance loss is lower than XML as cache gets bigger,
you can keep feed metadata separate from cache itself (other table), but accessible in the same way,
SQL is probably easier to work with than XPath for most people.
Disadvantages of SQLite:
can be problematic with multiple processes accessing same database (probably not your case),
you should know at least basic SQL. Unless there will be hundreds of thousands of items in cache, I don't think you will need to optimize it much,
maybe in some way it can be more dangerous from security standpoint (SQL injection). On the other hand, you are not coding web app, so this should not happen.
Other things are on par for both solutions probably.
To sum it up, answers to your questions respectively:
You will not know, unless you test your specific application with both back ends. Otherwise it's always just a guess. Basic support for both caches should not be a problem to code. Then benchmark and compare.
Because of the way XML files are organized, SQLite searches should always be faster (barring some corner cases where it doesn't matter anyway because it's blazingly fast). Speeding up searches in XML would require index database anyway, in your case that would mean having cache for cache, not a particularly good idea. But with SQLite you can have indexing as part of database.
Don't forget that you have a great database at your fingertips: the filesystem!
Lots of programmers forget that a decent directory-file structure is/has:
It's fast as hell
It's portable
It has a tiny runtime footprint
People are talking about splitting up XML files into multiple XML files... I would consider splitting your XML into multiple directories and multiple plaintext files.
Give it a go. It's refreshingly fast.
Use XML for data that the
application should know -
configuration, logging and what not.
Use databases(oracle, SQL server etc) for data that the user
interacts with directly or
indirectly - real data
Use SQLite if the user data is more
of a serialized collection - like
huge list of files and their content
or collection of email items etc.
SQLite is good at that.
Depends on the kind and the size of the data.
I wouldn't use XML for storing RSS items. A feed reader makes constant updates as it receives data.
With XML, you need to load the data from file first, parse it, then store it for easy search/retrieval/update. Sounds like a database...
Also, what happens if your application crashes? if you use XML, what state is the data in the XML file versus the data in memory. At least with SQLite you get atomicity, so you are assured that your application will start with the same state as when the last database write was made.
XML is best used as an interchange format when you need to move data from your application to somewhere else or share information between applications. A database should be the preferred method of storage for almost any size application.
When should XML be used for data persistence instead of a database? Almost never. XML is a data transport language. It is slow to parse and awkward to query. Parse the XML (don't shred it!) and convert the resulting data into domain objects. Then persist the domain objects. A major advantage of a database for persistence is SQL which means unstructured queries and access to common tools and optimization techniques.
I have made the switch to SQLite and I feel much better knowing it's in a database.
There are a lot of other benefits from this:
Adding new items is really simple
Sorting by multiple columns
Removing duplicates with a unique index
I've created 2 views, one for unread items and one for all items, not sure if this is the best use of views, but I really wanted to try using them.
I also benchmarked the xml vs sqlite using the StopWatch class, and the sqlite is faster, although it could just be that my way of parsing xml files wasn't the fastest method.
Small # items and size (25 items, 30kb)
~1.5 ms sqlite
~8.0 ms xml
Large # of items (700 items, 350kb)
~20 ms sqlite
~25 ms xml
Large file size (850 items, 1024kb)
~45 ms sqlite
~60 ms xml
To me it really depends on what you are doing with them, how many users/processes need access to them at the same time etc.
I work with large XML files all the time, but they are single process, import style items, that multi-user, or performance are not really needs.
SO really it is a balance.
If any time you will need to scale, use databases.
XML is good for storing data which is not completely structured and you typically want to exchange it with another application. I prefer to use a SQL database for data. XML is error prone as you can cause subtle errors due to typos or ommissions in the data itself. Some open source application frameworks use too many xml files for configuration, data, etc. I prefer to have it in SQL.
Since you ask for a rule of thumb, I would say that use XML based application data, configuration, etc if you are going to set it up once and not access/search it much. For active searches and updations, its best to go with SQL.
For example, a web server stores application data in a XML file and you dont really need to perform complex search, update the file. The web server starts, reads the xml file and thats that. So XML is perfect here. Suppose you use a framework like Struts. You need to use XML and the action configurations dont change much once the application is developed and deployed. So again, the XML file is a good way. Now if your Struts developed application allows extensive searches and updations, deletions, then SQL is the optimal way.
Offcourse, you will surely meet one or two developers in your organisation who will chant XML or SQL only and proclaim XML or SQL as the only way to go. Beware of such folks and do what 'feels' right for your application. Dont just follow a 'technology religion'.
Think of things like how often you need to update the data, how often you need to search the data. Then you will have your answer on what to use - XML or SQL.
I agree with #Bradley.
XML is very slow and not particularly useful as a storage format. Why bother? Will you be editing the data by hand using a text editor? If so, XML still isn't a very convenient format compared to something like YAML. With something like SQlite, queries are easier to write, and there's a well defined API for getting your data in and out.
XML is fine if you need to send data around between programs. But in the name of efficiency, you should probably produce the XML at sending time, and parse it into "real data" at receive time.
All the above means that your question about "when the overhead of a database is justified" is kind of moot. XML has a way higher overhead, all the time, than SQlite does. (Full-on databases like MSSQL are heavier, especially in administrative overhead, but that's a totally different question.)
XML can be stored as text and as a binary file format.
If your primary goal is to let a computer read / write a file format effeciently you should work with a binary file format.
Databases are an easy to use way of storing and maintaining data.
They are not the fastest way to store data that is a binary file format.
What can speed things up is using an in memory database / database type. Sqlite has this option.
And this sounds like the best way to do it for you.
My opinion is that you should use SQLite (or another appropriate embedded database) anytime you don't need a pure-text file format. Note, this is a pretty big exception. There are a lot of scenarios that require, or are benefited by, pure-text file formats.
As far as overhead goes, SQLite compiles to something like 250 k with normal flags. Many XML parsing libraries are larger than SQLite. You get no concurrency gains using XML. The SQLite binary file format is going to support much more efficient writes (largely because you can't append to the end of a well-formatted XML file). And even reading data, most of which I assume is fairly random access, is going to be faster using SQLite.
And to top it all off, you get access to the benefits of SQL like transactions and indexes.
Edit: Forgot to mention. One benefit of SQLite (as opposed to many databases) is that it allows any type in any row in any column. Basically, with SQLite you get the same freedom you have with XML in terms of datatypes. This also means that you don't have to worry about putting limits on text columns.
You should note that many large Relational DBs (Oracle and SQLServer) have XML datatypes to store data within a database and use XPath within the SQL statement to gain access to that data.
Also, there are native XML databases which work very much like SQLite in the sense they are one binary file holding a collection of documents (which could roughly be a table) then you can either XPath/XQuery on a single document or the whole collection. So with an XML database you can do things like store the days data as a separate XML document in the collection... so you just need to use that one document when your dealing with the data for today. But write an XQuery to figure out historical data on the collection of documents for that person. Slick.
I've used Berkeley XMLDB (now backed by Oracle). There are others if you search google for "Native XML Database". I've not seen a performance problem with storing/retrieving data in this manner.
XQuery is a different beast (but well worth learning), however you may be able to just use the XPaths you currently use with slight modifications.
A database is great as part of your program. If quering the data is part of your business logic.
XML is best as a file format, especially if you data format is:
1, Hierarchal
2, Likely to change in the future in ways you can't guess
3, The data is going to live longer than the program
I say it's not a matter of data size, but of data type. If your data is structured, use a relational database. If your data is semi-structured, use XML or - if the data amounts really grow too large - an XML database.
If your searching go with a db. You could split the xml files up into directories to ease seeking, but the managerial overhead easily gets quite heavy. You also get a lot more than just performance with a sql db...
Related
I have been looking at different ways to hold onto some predefined character data, however I am having a hard time nailing down which would be the best solution.
An example of data would be 10 strings, 5 int arrays (of size 10 each). There would be 10+ set's of this data. The application would load in the information and inflate generic "character" objects.
Possible solutions:
XML: Due to Android's structured XML requirements it can be hard to use without making a different XML file for every character, and even then it would have ID overlapping for similar named data values.
SQLite: Wouldn't be a huge database, but databases are ugly version controlled unless it is done with a create-database script which has its own downsides (such as making sure DB is up to date between builds).
Hard-coded Objects: By far my least favorite solution, using polymorphism to hardcode all of the objects. Too dirty, not nearly as dynamic as it should be.
I would like to consider things such as version controlling the files, ease of updating (due to them only being inflated, never changed by the app).
If this data is baked i would suggest to use harcoded data.
Reasons.
In those three solution you save the data in the application.
If you use XML-data, you have to consume the time while code parsing inside the code. And you have to write the code that parses your xml.
If you use SQLite, your data will be doubled because of you have to store this database in raw or assest directory, copied in the /data/data folder. Futhermore, if you use Strings and SQLite by default the data will be doubled again (due to UTF-16 encoding).
If think, if only you manage the data this is more usefull to store directly inside the code. Obviously, if you do not use tons of content:)
You might want to use the Realm framework, which is comparatively faster than SQLite and easy to implement inside in your current code.
It handles large data too and it feels like you're using only native android classes.
I have been asked to create a tiny android app.
In everyday work i code for .NET and I have no experience connected with Android, but as it is a really small app I guess it's going to be a good experience rather than something hard.
The core of the app would be a small database (probably XML, unless somebody suggest better solution) that would contain categories, names of the institutions assigned with each category and logo (not very high resolution I guess a single file would be <100kB) of the institution.
The database also would not be very big - I expect not more than 1000 records in total. The DB has to be totally offline and local, it cannot require Internet access when operating.
The model I assume would be to ship new version of the application when the database changes (which is not going to be very frequent).
What is the best way to deal with these requirements?
My first idea was to create an XML file that would contain the records and link to the image. The XML and all the images linked to it would be stored in single file (preferably zip) that would be stored in app resources. This is very good as it is going to be very easy to update the database.
The second idea that somebody suggested me would be to use SQLite and store images in BLOB. In general I have read that it isn't a good idea to store images in database directly, and I am afraid if it's going to be possible to meet all requirements mentioned above.
Mostly I have no idea how to update the database easily and attach it to new version of application.
Any suggestions?
I would be grateful for any response.
I wouldn't go about using XML to save your data and by no means zip anything.
I think your way of thinking is ok, but you're making things really complicated for yourself.
Seeing as you're used to .NET I suppose you're also pretty confident with SQL, so I'd suggest you have a look at how to use the built-in SQLite database in Android.
If you would go the XML route you'd have to serialize and de-serialize the XML file over and over again and then parse the XML. Ok you don't have a lot of data, but searching inside an XML file with at least 1000 nodes would be slow in comparison to the performance of a database.
Also upgrading an existing SQLite database is not that hard - Android has methods for that (onUpgrade coming from the SQLiteOpenHelper).
As to saving images I'm assuming that you won't fetch new pictures from the Internet, so it would be best just to store them in the drawable folder of your app (be mindful of different screensizes) and then reading them into an ImageView when needed. To figure out what image should go for what institution I would store either the image name of each image in the SQLite database or store the resource id for each image in the database - for instance R.drawable.myawesomepictureformyinstitution.
I know my answer is somewhat "superficial", but your question is also somewhat "broad" and hard to answer without me actually writing most of the code, and that's not my intention ;-)
Hope this helps - let me know if anything is unclear.
Long story short: I'm working on refactoring an old Android project of mine. Previously, it was using serialization, which was painfully slow and, from what I'm reading, a pretty lousy idea in general for Android apps. I'm looking for another way to persist both user-specific data as well and read-only data for the application.
There is going to be a good deal of data on both sides and I'm not sure if there's a "good" way to store it. Basically, the app is a small RPG. There are a number of "maps" that are represented as 2D arrays of Tiles. Each Tile will have a number of attributes, some simple primitives or enums, others additional objects, such as Events, which will also potentially contain various objects, etc. With 400 Tiles in a 20x20 map alone, there's a LOT of data to store. In addition to storing that data, it would need to store a lot of user-specific data, such as which Tiles have been visited, which Events have been successfully run, etc.
I've been examining methods of saving this data out and I just can't seem to settle on something. I guess it boils down to XML or JSON vs SQLite. XML or JSON would be more flexible in terms of future changes, which is good as I want flexibility in the data, ie, adding new attributes to existing objects, adding new objects as the need arises, etc. SQLite isn't as easily malleable as you have to change up the schema, perhaps adjust queries and indexes, etc, but I haven't really used SQLite in the past, so maybe there are some features that help to simplify that process. However, I would also like fast random access to data to avoid loading everything into memory at once if it can be helped. For example, when moving from one map to another, I'd much rather load the next map only when needed rather than having everything held in memory, which is where SQLite would shine as I'd be able to directly query the data rather than traversing a JSON/XML file to find potentially scattered data, ie, we load the map, but Events and objects contained in the Events may not be unique to that map and could easily lie elsewhere in the file or in another file entirely. However, normalizing the data in SQLite would mean a lot of tables and quite a bit of deconstructing/reconstructing of objects.
Writing user data would only occur when the user manually saves the game, so write performance is not a big issue.
I sometimes have a tendency to overanalyze and get hung up on stuff like this. Maybe neither case is necessarily "wrong" and I'm worrying about things that are infinitesimal. Maybe there are other cases that I haven't considered. I've used Hibernate and have considered something like ORMLite to handle a lot of the database nitty-gritty, but that would require a lot of retrofitting, likely much more than what I would need to do for other options.
I'd suggest you use SQLite. I think it makes the most sense considering the amount of data you're trying to store.
As far as your concerns with it not being as flexible, I would argue that point. Just use a ContentProvider. ContentProviders make it pretty easy to update the db schema and queries without affecting your existing functionality. If you use a ContentProvider, you could even swap out persistent data strategies in the future as well as use different ones simultaneously.
http://developer.android.com/guide/topics/providers/content-providers.html
I have three tables with 5 columns each. They can grow upto 15 rows each.
Does it make sense to have a SQLite database for this or should a file suffice? I am talking about pure performance basis.
my suggestion is to go for Sqlite, Sqlite having cursor which can navigate data, modify data very easily.
The same thing with File is quite complex, for that you have to do string function like indexOf(), subString(), replace() etc.
From a performance point of view this will matter very little, but from a maintenance point of view you should go with the sql approach, stay clear of any arcane home-brew approaches if there is a general well-known way of doing something :)
Also it should be a lot faster to write the code for the database rather that for the flat file. And, never optimize until you can measure and see that you have a performance issue.
Do you need any sort of concurrency support? If so, SQLite would make sense, rather than building that yourself.
I would side with using a database, since it would likely be easier to maintain the integrity of your data, with respect to relationships and the format of the data you're storing. Making an error when updating a table is easy to notice (exceptions are nice to prevent corrupt data), but writing out malformed data to your own format might not be easy to spot until it's too late.
Performance should probably come second to maintaining valid data.
I would also clearly vote for the DB solution. Additionally to the mentioned advantages, it's a good thing to learn anyway - and - honestly - at the end considering all issues it's not harder or more work than the file system solution.
By the way, back then I learned the DB with this tutorial: http://developer.android.com/resources/tutorials/notepad/index.html
It was alos very usefull for couple of other reasons...
If you don't already have a MySQL database and don't need any tricky joins, go with the filesystem.
I do web, not android, but I've had great success consuming JSON manifests out of the filesystem using JavaScript. It's so nice to cut the database overhead, and the filesystem is practically instant.
There is a web service that provides some data that my app makes use of. This data is fairly large and only changes VERY infrequently so I thought it would be nice if the app could cache it on the SD Card and only update it as needed.
Currently I'm grabbing the data (an XML file) and parsing it into an object tree using SAX. This process takes (at most) 2-3 seconds over my WIFI. However, serializing the resulting objects to the SDCard takes significantly longer (a minute or more) and deserializing it still takes longer than just download/parsing in the first place.
Does anyone have any recommendations for improving this or alternate ideas for persisting this data (other than just saving the XML file and reparsing every time)?
UPDATE: This is more than a trivial collection of records. The object-graph is actually ridiculously complex and storing it into a database would result in dozens of tables with only a single record in each one.
Android serialization is notoriously slow. I highly suggest switching to using XML or JSON (or whatever) and writing the file out that way. Since you've already got an XML parser, it may make the most sense just to cache the original XML file you downloaded and reparse it as necessary.
I have switched from Serializable to JSON file storage in an app before and the speed increase was incredible, at least one order of magnitude.
(I may be misunderstanding your question - I assume you are using Serializable for writing to the disc. If you are reproducing the XML, then I'm not sure why it is so much slower on the SD card. Also, I agree that the SQLite database makes the most sense typically, but as you've already stated it does not fit the needs of your application.)
Also unless your data is at least 100s of Kb, I would suggest just storing it in your private data storage instead of on the SD card. Keep in mind that you can't rely on the SD card being available.
I've just been writing an android application for the last week which basically does this. It fetches some (large) XML file online, and then displays part of the data in various views.
We do it by fetching and parsing the XML using SAX, and (while parsing) writing it all to a SQLite database. And then we are just querying the database each time we need to display some view of the dataset.
Works like a charm, and is fast enough for displaying a lot of data on a google map overlay, where we are querying the database on every single call to the draw method of our map overlay.
So I would definitely suggest going for a SQLite database, if the data in the XML document is easily represented in a database.
If the web service can give you just a specified number of results(something like: requestData between index 1 and 10 or give me first 25 results) try to use that (put a simple "Load more results" button or implement an auto-loading mechanism). If the web service not provide this feature then try to save your xml on sdcard and when you need the data try to parse just a specified number of results. Hope this help!
Why don't you use database? See Android Data Storage Guide