In my application, I'm retrieving content from the database very often. So I'm likely to access data from database when it is required. For example, even within an activity I'll open and close the database 3 to 4 times. Is this enough to cause performance to suffer?
This would really depend on the amount of data you are reading and how complicated the queries are. Personally I wouldn't worry about unless you can notice the performance issue, why fix a problem that doesn't exist. That said if you are noticing performance problems and think the database may be the cause you can try the following:
Cache database reads so they only have to be done when something is updated.
Wrap database writes in transactions so the disk writes are done all in one go.
Only read the data you need. For instance if you have a list the displays 3 out of the 20 fields only read the 3 fields you need.
Related
I have a large SQLite database for my application (around 100MB). This database is read only and never written into. The performance of working with the database is decent, since I made sure that most of the queries have indexes backing them up. However I would like to speed it up further by caching the entire database into RAM to avoid disk access.
Is this possible and if so how can I do it?
A database file is a file, so you can just read it to put it into the OS's file cache. (And if the OS decides to throw the data out again, then it might have a reason for it …)
For an on-disk database, SQLite needs to check if some other process has made changes. To avoid that, you can use
PRAGMA locking_mode = EXCLUSIVE. However, the difference probably isn't measurable.
I have recently read inside documentation for Realm database that all of their queries are lazy and I am not sure, if I understand correctly the implications which this may cause.
Let me explain how I understand it and please feel free to correct me if I'm wrong. The way I see it is that, whenever I am making such command mRealm.where(Customer.class).equalTo(Customer.ID, "someId").findFirst(); I do not get Java object with all filled data, that is contained in db for this customer. Instead queries are made whenever I try to access some fields of this "fetched" object. Therefore I am wondering, if that is faster than OrmLite, if I want to access all of the fields of given class?
Another question related to it. Is it good idea to use Realm db for items which would be displayed in ListView or RecyclerView? If queries are constantly made during scrolling of the list, wouldn't it have serious impact on the performance?
I would be really glad, if someone could explain this to me in more detail.
RealmResults, which are created by our queries, are auto-updating views into the underlying data. You can think of them as type-safe cursors and they have some of the same trade-offs, but unlike Cursors we don't copy data into a CursorWindow so there is no pagination effect. You can access the entire object graph without being worried about reaching a CursorWindow limit.
Most ORM's copy all data into Java heap memory. This can be a potentially very costly operation both time- and memory-wise + you might have a lot of additional data you don't even need. The upside to this is that you do it once, then it is really fast to access the data.
Realm on the other hand, only load the data you actually need. This includes individual fields as well. So if you have a ListView showing 1 field from 10 items we will only load those 10 fields just like an CursorAdapter. It has to load that data from native memory though which is more expensive than reading it from the Java heap.
So to answer you question. Yes, Realm works perfectly fine with ListView.
If you want to know what is faster, try it ;). Probably Realm will be faster, because their propietary system that is more optimized than SQLite, even fine-tuned. But that's my supposition
No, it mustn't have impact in performance, because we are supposing that Realm is smart enough to fetch the real data when appropiate. When using another ORM, it's the same thing, data is fetched when a pagination occurs.
I am currently developing an Android app that works with an SQLite database locally. I am wondering what the best way is to load data from the database.
the database would contain about 3 tables that would total up to about 500 rows. What would be the best option in this case:
- Load all data on startup to fill the java model for use in the application.
- Load only data that is needed in each screen. This probably requires db calls on nearly each screen load but will need less memory. If this is the best solution, how do you handle this situation ? If you would open a view, you would have to query the db since not everything is pre-loaded. But if you would open it again afterwards, you would have to have some kind of 'caching' mechanism to detect if it's already pre-loaded ?
Sorry if my question is not very clear, I find this difficult to describe :-\ .
Thanks in advance for any tips.
Cheers
Wesley
What would you do if your database grew to 5,000 rows? 50,000?
Premature optimization is one of the leading causes of poor design. In my (relatively limited) experience, SQLite database queries are fast enough. Try loading your data on demand, as is fairly standard practice, and see if your program runs quickly enough.
When my app is first run, it creates 5 tables and inserts about 50 initial values. The user can delete any of these initial values if they want and they will add to them.
In this situation, what are the pros/cons between creating the db file and copying it over on first run and just putting a bunch of create/insert statements in onCreate?
It's crucial that user information doesn't get overwritten and because of that I'm leaning towards the create/insert statements, since those will fail/be minor if some bug triggers onCreate (if that's possible), whereas copying the db file would wipe the db.
There are no benefits of packaging the sqlite db rather than creating it IMO, just choose one and code accordingly, worrying that something it wrong with your code and deciding to use one method or another based on that is just an ice patch to bad programming.
No offense meant, I just think the reason why you are asking this question isnt right, the real difference would be between deciding something that just copies, but makes the APK bigger, but perhaps is faster than creating and populating.
I personally go with creating the DB from scratch, you will only do it "once", unless of course your update requires modifications to the DB or the user deleted the data. I would rather have the user wait a while, just once, and make the APK a considerable ammount of KBs lighter.
I think you've answered your own question as far as the cons of copying the whole db over. (Although I do like copying over databases for unit testing.) If you are looking for a less tedious way to populate those fifty values, you might try using .dump in sqlite3 and putting all of those insert statements into a single resource.
On the other hand, if onCreate gets called when it shouldn't, you probably have bigger problems to worry about.
We're designing an Android app that has a lot of data ("customers", "products", "orders"...), and we don't want to query SQLite every time we need some record. We want to avoid to query the database as most as we can, so we decided to keep certain data always in memory.
Our initial idea is to create two simple classes:
"MemoryRecord": a class that will contain basically an array of objects (string, int, double, datetime, etc...), that are the data from a table record, and all methods to get those data in/out from this array.
"MemoryTable": a class that will contain basically a Map of [Key,MemoryRecord] and all methods to manipulate this Map and insert/update/delete record into/from database.
Those classes will be derived to every kind of table we have in the database. Of course there are other useful methods not listed above, but they are not important at this point.
So, when starting the app, we will load those tables from an SQLite database to memory using those classes, and every time we need to change some data, we will change in memory and post it into the database right after.
But, we want some help/advice from you. Can you suggest something more simple or efficient to implement such a thing? Or maybe some existing classes that already do it for us?
I understand what you guys are trying to show me, and I thank you for that.
But, let's say we have a table with 2000 records, and I will need to list those records. For each one, I have to query other 30 tables (some of them with 1000 records, others with 10 records) to add additional information in the list, and this while it's "flying" (and as you know, we must be very fast at this moment).
Now you'll be going to say: "just build your main query with all those 'joins', and bring all you need in one step. SQLite can be very fast, if your database is well designed, etc...".
OK, but this query will become very complicated and sure, even though SQLite is very fast, it will be "too" slow (2 a 4 seconds, as I confirmed, and this isn't an acceptable time for us).
Another complicator is that, depending on user interaction, we need to "re-query" all records, because the tables involved are not the same, and we have to "re-join" with another set of tables.
So, an alternative is bring only the main records (this will never change, no matter what user does or wants) with no join (this is very fast!) and query the other tables every time we want some data. Note that on the table with 10 records only, we will fetch the same records many and many times. In this case, it is a waste of time, because no matter fast SQLite is, it will always be more expensive to query, cursor, fetch, etc... than just grabbing the record from a kind of "memory cache". I want to make clear that we don't plan to keep all data in memory always, just some tables we query very often.
And we came to the original question: What is the best way to "cache" those records? I really like to focus the discussion on that and not "why do you need to cache data?"
The vast majority of the apps on the platform (contacts, Email, Gmail, calendar, etc.) do not do this. Some of these have extremely complicated database schemas with potentially a large amount of data and do not need to do this. What you are proposing to do is going to cause huge pain for you, with no clear gain.
You should first focus on designing your database and schema to be able to do efficient queries. There are two main reasons I can think of for database access to be slow:
You have really complicated data schemas.
You have a very large amount of data.
If you are going to have a lot of data, you can't afford to keep it all in memory anyway, so this is a dead end. If you have complicated structures, you would benefit in either case with optimizing them to improve performance. In both cases, your database schema is going to be key to good performance.
Actually optimizing the schema can be a bit a of a black art (and I am no expert on it), but some things to look out for are correctly creating indices on rows you will query, designing joins so they will take efficient paths, etc. I am sure there are lots of people who can help you with this area.
You could also try looking at the source of some of the platform's databases to get some ideas of how to design for good performance. For example the Contacts database (especially starting with 2.0) is extremely complicated and has a lot of optimizations to provide good performance on relatively large data and extensible data sets with lots of different kinds of queries.
Update:
Here's a good illustration of how important database optimization is. In Android's media provider database, a newer version of the platform changed the schema significantly to add some new features. The upgrade code to modify an existing media database to the new schema could take 8 minutes or more to execute.
An engineer made an optimization that reduced the upgrade time of a real test database from 8 minutes to 8 seconds. A 60x performance improvement.
What was this optimization?
It was to create a temporary index, at the point of upgrade, on an important column used in the upgrade operations. (And then delete it when done.) So this 60x performance improvement comes even though it also includes the time needed to build an index on one of the columns used during upgrading.
SQLite is one of those things where if you know what you are doing it can be remarkably efficient. And if you don't take care in how you use it, you can end up with wretched performance. It is a safe bet, though, if you are having performance issues with it that you can fix them by improving how you are using SQLite.
The problem with a memory cache is of course that you need to keep it in sync with the database. I've found that querying the database is actually quite fast, and you may be pre-optimizing here. I've done a lot of tests on queries with different data sets and they never take more than 10-20 ms.
It all depends on how you're using the data, of course. ListViews are quite well optimized to handle large numbers of rows (I've tested into the 5000 range with no real issues).
If you are going to stay with the memory cache, you may want have the database notify the cache when it's contents change and then you can update the cache. That way anyone can update the database without knowing about the caching. Also, if you build a ContentProvider over your database, you can use the ContentResolver to notify you of changes if you register using registerContentObserver.