I'm working on a program that requires metadata information in order to populate some arrays. Let's say I have information like "Countries", "Districts" and a bunch of other metadata. That information is stored in a sqlite database. The program at some time need to load all the countries and iterate them in order to search for one. My question is: What is the best way to proceed: Keep the metadata in an array after query them, or every time I need them I should query the database?
Here's some more information so you can evaluate the performance:
Metadata tables (like countries): ~10
Estimated times I need to iterate the metadata: several (~100)
the arrays contains aprox. 5 fields (primitive types.)
If the amount of data is so large that if affects the amount of data available for your other data or for other app, you should keep it in the database and access it dynamically.
If the amount of data is rather small, and it's queried rather often, keeping it in memory is more efficient.
If the amount of data is rather small, and it's queried not very often, it will not make any noticeable difference what you do.
Your particular case is one of these three, but the only way to find out is to measure the performance yourself.
Related
If I want to use an AutoCompleteTextView as a search function to generate a list of strings (let's say about 1000-3000 strings) based on the text the user has currently inputted, would it be better to be loading these strings from an external database or having them stored in the internal SQLite database and loading the strings from there?
Is the amount of strings I plan on storing too big (each string will be about 10-20 characters long, they are constants and will never change) to be used in the SQLite database? How much would this slow down and/or bulk up my app? Should I just use the external database? Would the loading times be fast as the user is typing in a string?
I'm asking because if I can avoid using an external database I would prefer that since I wouldn't have to worry about the number of users accessing the database, maintaining it, and security issues.
would it be better to be loading these strings from an external database or having them stored in the internal SQLite database and loading the strings from there?
Depends on what you mean by 'better'. If you store them in a local database then the performance should be better, I recommend using indexes in this case. At the same time, this solution will require more memory although I think 3000 strings should be okay if they are just one-word strings.
Is the amount of strings I plan on storing too big to be used in the SQLite database?
Depends on what are those strings. If they are not too long then I think it should be fine. If each string contains a lot of text then I think this might cause problems while searching through them via SQL queries.
How much would this slow down and/or bulk up my app?
It shouldn't be noticeable, you can actually create a test for that, shouldn't take a lot of effort. Depends on your use case.
Should I just use the external database?
If those strings are like constants and they are not going to change, you can consider storing them in a local database, this should improve the performance versus the network-loading solution.
Would the loading times be fast as the user is typing in a string?
You can load the first set of data when they 'stop' typing. See my answer to similar question here: Best Practices for Handling Search
Once you load the list of data you can use this local cache for further searching, it should be fast enough so that user doesn't notice any delay.
I'm developing an android app that uses various Lists of Java Modelled entities and FileInputStream and FileOutputStream to store and load these.
The average complexity of my dataManager is 4*O(n^3) for update all entries of 4 different data sets at the same time, where every operation requires 3 nested for on an ArrayList because an Object Palace contains a List of Apartament and an Object Apartament contains a list of People
Could I have noticeable slow down in the app due to my approach?
what is the efficience If I use SQLite to perform the same operations?
In my opinion SQL would offer some advantages.
In terms of efficiency, I can't tell you which one is better (file vs database) without discussing the amount of data you're holding. 2 or 3 records is a very different story than 10,000 records (without even considering the nested levels).
If you're using a file, you have to load everything manually into memory to access your objects, and this means (most of the time) loading the file completely, and saving it completely again.
A database would allow you to fetch only the information that you currently need, and it also allows you to only save the data that has been modified.
There are many similar questions about this issue but I have clear points about my question to ask you.
I am new at Android development and before only I developed small applications which store small sized data. For example country List, calendar, birthday reminder etc. I stored my small data in single XML file and I parsed it with easy methods. This was enough for me. But for my Mobile Application Development Course I took a project which will store huge static data.
Specifications of my project will like these:
There are about 200 entities.
Each entity has about 20 sub categories which they stored in text format.
Each sub category has about 30-sub categories which they stored again in text format.
Also for each parent entity I will have 2-3 image
If I calculate simply, I have to store 200 X 20 X 30 = 120.000 static data for my application and data does not change. This is only install and use application. Online data interaction is not necessary. (If there are some changes for data I will relase major updates in long periods of time.)
My question is about storing method.
Which way should I choose? SQLite or XML parsing? For your answer can you explain advantages / disadvantages for your choice?
Interesting project, although not necessarily realistic.
To manage a large amount of "static" data, you'll want a database. XML parsing forces you to store the data in memory, which means that you have to read it into memory on a regular basis. Remember that you can't count the in-memory data being around when the user goes to your app; Android may have destroyed your app previously.
On the other hand, you can use an SQLite database on disk directly from your app. It's persistent, even if your app goes away. You'll have to load the database once, when you install the app.
Consider wrapping your SQLite database in a content provider. This will, among other things, allow you to do asynchronous queries using a CursorLoader.
I have a database, which has a table with fields like "title, album, artist..." and it also has many fields with html content for every record (up to 30).
Problem is, that this database has tens of thousands of records and is hundreds of megabytes large because of the html content. Because of the size of the sqlite file the search is very slow (also inserting new elements in a transaction is very slow ~10-30 second for 200 new rows). The very first LIKE query can take 10-15 seconds, other searches are fast enough (indices are created and work ok). When I removed the html content from the database the search was always instant.
So the question is, what is the best way to store that additional html content? Right now I play with the option to store it in separate files, but it can generate up to 600k files and more in the future, which is quiet slow to create. Storing the files in a zip archive will probably hit its file number limit. Other options are to zip files per table row, store the html in a separate table in the same database, or to create a separate database file for the html content.
What will give me the best performance? Or are there other better options? I need quick insert, update and serach.
There are a couple different things you could consider doing:
Split the data into separate tables. You could then have 1:1 mappings between the tables, and only join them in if necessary, speeding up queries without them.
Check your indexes. Just because you have them and you think they're working, doesn't mean they are. If I recall correctly, sqlite will use at most one index per query, so you need to make sure you have the best index possible available for the queries you're using. The ANALYZE command can help with that.
After some days of experimenting I came to this conclusion:
one database file with one table was the slowest (up to 10 seconds)
one database with two tables was twice as fast in the worst case scenario as one table
fastest was to have two separate database files. one with data needed for search and the other for the huge html data. this is almost instant in the worst case ~300ms and in normal usage it is instant
So I reccommend to use two separate database files in this scenario. If someone does not come with a faster/better solution I will accept this as the answer.
We're designing an Android app that has a lot of data ("customers", "products", "orders"...), and we don't want to query SQLite every time we need some record. We want to avoid to query the database as most as we can, so we decided to keep certain data always in memory.
Our initial idea is to create two simple classes:
"MemoryRecord": a class that will contain basically an array of objects (string, int, double, datetime, etc...), that are the data from a table record, and all methods to get those data in/out from this array.
"MemoryTable": a class that will contain basically a Map of [Key,MemoryRecord] and all methods to manipulate this Map and insert/update/delete record into/from database.
Those classes will be derived to every kind of table we have in the database. Of course there are other useful methods not listed above, but they are not important at this point.
So, when starting the app, we will load those tables from an SQLite database to memory using those classes, and every time we need to change some data, we will change in memory and post it into the database right after.
But, we want some help/advice from you. Can you suggest something more simple or efficient to implement such a thing? Or maybe some existing classes that already do it for us?
I understand what you guys are trying to show me, and I thank you for that.
But, let's say we have a table with 2000 records, and I will need to list those records. For each one, I have to query other 30 tables (some of them with 1000 records, others with 10 records) to add additional information in the list, and this while it's "flying" (and as you know, we must be very fast at this moment).
Now you'll be going to say: "just build your main query with all those 'joins', and bring all you need in one step. SQLite can be very fast, if your database is well designed, etc...".
OK, but this query will become very complicated and sure, even though SQLite is very fast, it will be "too" slow (2 a 4 seconds, as I confirmed, and this isn't an acceptable time for us).
Another complicator is that, depending on user interaction, we need to "re-query" all records, because the tables involved are not the same, and we have to "re-join" with another set of tables.
So, an alternative is bring only the main records (this will never change, no matter what user does or wants) with no join (this is very fast!) and query the other tables every time we want some data. Note that on the table with 10 records only, we will fetch the same records many and many times. In this case, it is a waste of time, because no matter fast SQLite is, it will always be more expensive to query, cursor, fetch, etc... than just grabbing the record from a kind of "memory cache". I want to make clear that we don't plan to keep all data in memory always, just some tables we query very often.
And we came to the original question: What is the best way to "cache" those records? I really like to focus the discussion on that and not "why do you need to cache data?"
The vast majority of the apps on the platform (contacts, Email, Gmail, calendar, etc.) do not do this. Some of these have extremely complicated database schemas with potentially a large amount of data and do not need to do this. What you are proposing to do is going to cause huge pain for you, with no clear gain.
You should first focus on designing your database and schema to be able to do efficient queries. There are two main reasons I can think of for database access to be slow:
You have really complicated data schemas.
You have a very large amount of data.
If you are going to have a lot of data, you can't afford to keep it all in memory anyway, so this is a dead end. If you have complicated structures, you would benefit in either case with optimizing them to improve performance. In both cases, your database schema is going to be key to good performance.
Actually optimizing the schema can be a bit a of a black art (and I am no expert on it), but some things to look out for are correctly creating indices on rows you will query, designing joins so they will take efficient paths, etc. I am sure there are lots of people who can help you with this area.
You could also try looking at the source of some of the platform's databases to get some ideas of how to design for good performance. For example the Contacts database (especially starting with 2.0) is extremely complicated and has a lot of optimizations to provide good performance on relatively large data and extensible data sets with lots of different kinds of queries.
Update:
Here's a good illustration of how important database optimization is. In Android's media provider database, a newer version of the platform changed the schema significantly to add some new features. The upgrade code to modify an existing media database to the new schema could take 8 minutes or more to execute.
An engineer made an optimization that reduced the upgrade time of a real test database from 8 minutes to 8 seconds. A 60x performance improvement.
What was this optimization?
It was to create a temporary index, at the point of upgrade, on an important column used in the upgrade operations. (And then delete it when done.) So this 60x performance improvement comes even though it also includes the time needed to build an index on one of the columns used during upgrading.
SQLite is one of those things where if you know what you are doing it can be remarkably efficient. And if you don't take care in how you use it, you can end up with wretched performance. It is a safe bet, though, if you are having performance issues with it that you can fix them by improving how you are using SQLite.
The problem with a memory cache is of course that you need to keep it in sync with the database. I've found that querying the database is actually quite fast, and you may be pre-optimizing here. I've done a lot of tests on queries with different data sets and they never take more than 10-20 ms.
It all depends on how you're using the data, of course. ListViews are quite well optimized to handle large numbers of rows (I've tested into the 5000 range with no real issues).
If you are going to stay with the memory cache, you may want have the database notify the cache when it's contents change and then you can update the cache. That way anyone can update the database without knowing about the caching. Also, if you build a ContentProvider over your database, you can use the ContentResolver to notify you of changes if you register using registerContentObserver.