My Android app works by using a SQLite database that is generated on the user's PC and transferred to the device. It all works, but I had not anticipated the number of users who would have really huge amounts of data. In these cases, the UI is very sluggish as it waits for the data to be fetched.
I've tried a number of tricks that I was "sure" would speed things up, but nothing seems to have any noticeable effect. My queries are almost all very simple, being usually a single "col=val" for the WHERE clause, and INTEGER data in the column. So I can't do much with the queries.
The latest, and I am not an SQL expert by any means, was to use "CREATE INDEX" commands on the PC, believing that these indexes are used to speed up database searches. The indexes increased the size of the database file significantly, so I was then surprised that it seemed to have no effect whatsoever on the speed of my app! A screen that was taking 8 seconds to fill without indexes still takes about 8 seconds even with them. I was hoping to get things down to at least half that.
What I am wondering at this point is if the SQLite implementation on Android uses database indexes at all, or if I'm just wasting space by generating them. Can anyone answer this?
Also, any other things to try to speed up access?
(For what it's worth, on an absolute basis the users have nothing to complain about. My worst-case user so far has data that generates 630,000 records (15 tables), so there's only so much that's possible!)
Doug Gordon
GHCS Systems
SQLite will use the index if it is appropriate for the query. Use EXPLAIN
EXPLAIN QUERY PLAN ... your select statement ...
to see what indexes SQLite is using. The query plan is based on some assumptions about your database content. You may be able to improve the plan by using ANALYZE
I was finally able to achieve tremendous performance gains simply by querying the database in a much more efficient way. For example, in building up an array of information, I was previously querying the database for each row that I required with a "WHERE _id = n" type selector. But in doing it this way, I was issuing a dozen or more queries, one at a time.
Instead, I now build up a list of IDs that are required, then get them all with a single query of the form "WHERE _id IN (n1, n2, n3, ...)" and iterate through the returned cursor. Doing this and some other structure optimizations, the largest database is now almost as quick to view as the more average case.
Every time you're going to perform some kind of action (being database lookup, long-running calculation, web request etc.) taking more than a couple of hundreds of milliseconds, you should consider wrapping this inside an AsyncTask.
Painless Threading is a good article on this topic, so I recommend you take a close look at it.
This article discusses the threading
model used by Android applications and
how applications can ensure best UI
performance by spawning worker threads
to handle long-running operations,
rather than handling them in the main
thread.
Related
I'm trying to optimize the insertion speed for sqlite in android. I've tried numerous methods. Like setting the pragma settings, journal mode and etc.
Besides that, i've also tried individual inserts, prepared statement and bulk insert. Keep in mind that all of those has already been wrapped in a transaction.
The insertion speed in my smartphone(Samsung Galaxy SIII) is only up to about 250 rows per second for a 20 column table. According to Sqlite faq, sqlite can easily insert 50,000 inserts per second. Therefore, i'm really optimistic about the insert to have at least up to the thousands rather than hundreds. However, none of the above mentioned works. Or 250 rows of inserts is already optimized?
Please help me. Thanks.
Regards,
Dexter
One method I use on big company sized databases to increase bulk insert speed is to turn off indexing while doing bulk inserts. The drawback is you need to have exclusive access to the database and you have to turn indexing back on when your done.
In a mobile app you could easily get exclusive access to the database by putting up an "Updating the app" screen.
SQLITE you can drop any non unique indexes before performing the bulk insert. If your 20 column table has 20 non unique indexes (one for each column). Drop the 20 indexes. Perform the bulk inserts. Create the 20 indexes you previously dropped.
Also make sure your code is compact and doesn't do anything inside the insert loop that could be done before the loop. For example look up column indexes before your insert loop.
If this doesn't apply to your app just catalog it away as something you may need to do in the future on your back end database.
You can use Object/Relational Mapping (ORM) tool for better performance.
ORMLite, GreenDao etc.
Greendao is better one, have a look
http://greendao-orm.com/
Features :-
Maximum performance (probably the fastest ORM for Android)
Easy to use APIs
Highly optimized for Android
Minimal memory consumption
Small library size, focus on the essentials
Benefits of ORM tools :-
ORM can give you Relational Persistence, which results fastest performance.
Less development work
Since phones are not nearly as fast as computers, I can see having multiple joins in queries slow down the application.
With redundant data (or non-normalize) it seems like you will have a lot of duplicate data but faster queries.
So, in android/sqlite, is it better to normalize your database and therefore require more joins in queries or to simply have redundant data but use less joins?
Any suggestions would be great! Thanks!
I would go for normalizing the database, Processing power has improved significantly and running queries with joins would be easier than dealing with nightmare of de-normalized schema
Normalization doesn't always mean more joins. When the information you need is contained in the foreign key--which happens regularly with natural keys, but never happens with id numbers--you don't need a join to get it.
See, for example, this SO answer, which deals with stores and their addresses in 5NF with no joins.
I was browsing through the source code for Mendeley for Android by Martin Eve and saw that Temp tables are created along with the main tables by calling replace on the tables with _id. For example on the Collections table
db.execSQL(COLLECTIONS_CREATE.replace(" (_id", "_TEMP (_id"));
I guess it creates a new temporary table. Any more explanation on this would be great. Further on the data is first inserted to temp tables and later moved to main tables.
I searched through SO and came across What are the use cases for temporary tables in SQL database systems? and saw that temp tables are used for handling complex queries, sorting and for increasing performance. Can anyone explain how it helps in this situation?
Temp tables make things easier for the programmer by letting the programmer break up a single complex query into multiple relatively simpler queries, and also by letting the programmer store results temporarily so they can be consulted multiple times for different purposes during the course of the program without having to be reinstantiated each time. The latter also makes things easier for the computer. The disk subsystem and CPU can take a little rest, so to speak.
An example of the former: let say you wanted to get all records where:
the sale was in the eastern division
and involved one of the several new gizmos introduced last quarter
and occurred during the special 5-day bonanza sale
or
the sale was made by the boss's daughter
who floats from division to division
and the sale occurred at any time during the month of May
Your program will then generate an email praising the salesperson for the sale, with a cc to the division manager.
The single query that fetches records that satisfy either of those sets of conditions above might get a little unwieldy--just a little difficult for one's addled brain or weary eyes to handle after a long day of dealing with the sort of crap one has to deal with in most walks of life. It's a trivial example, of course; in a production system the conditions are often more complex than those above, involving calculations and tests for null values and all sorts of other tedious things that can cause a query statement to grow long and turn into a ball of tangled yarn.
So if you created a temp table, you could populate the temp table with the rows that satisfy the first set of conditions, and then write the second query that grabs the rows that satisfy the second set of conditions, and insert them into the temp table too, and voilà -- your temp table contains all the rows you need to work with, in two baby steps.
Temporary tables are just that, temporary. They go away once you close your connection, at least with sqlite. It's often more complicated than that with other DMBS', although this is usually the default behaviour.
So temporary tables are thus used whenever a temporary table is required! (Just kiddin'). Performance will be better as there is no I/O involved (which is often the cause of performance problems with databases).
Another use case is when you want to use what is in a permanent table without changing anything in the said table; you can then select the permanent table in a temp table, and keep your permanent table intact.
I'm just wondering what's the fastest way to look up if at least one row, sharing certain attributes with other rows, does exist.
I'm not quiet sure how I should use SQLite here. Other DBMs have optimizers etc.. Do I need to code in a special way for SQLite? And, has LIMIT any impact here?
Many thanks in advance.
HJW
The fact that SQLite is an embedded database does not imply that it cannot have a query optimizer ! see here for an overview of SQlite query optimizer
in short, SQLite will behave the same as any other DBMS regarding the way it treats a query...
now, you are running on android, so your processing power may be a bit restricted. you should take care to avoid some kind of queries, to avoid wasting processor time or memory. queries resulting in the sorting of rows are notable to be slow, especially when there is no index defined on the sort key.
We're designing an Android app that has a lot of data ("customers", "products", "orders"...), and we don't want to query SQLite every time we need some record. We want to avoid to query the database as most as we can, so we decided to keep certain data always in memory.
Our initial idea is to create two simple classes:
"MemoryRecord": a class that will contain basically an array of objects (string, int, double, datetime, etc...), that are the data from a table record, and all methods to get those data in/out from this array.
"MemoryTable": a class that will contain basically a Map of [Key,MemoryRecord] and all methods to manipulate this Map and insert/update/delete record into/from database.
Those classes will be derived to every kind of table we have in the database. Of course there are other useful methods not listed above, but they are not important at this point.
So, when starting the app, we will load those tables from an SQLite database to memory using those classes, and every time we need to change some data, we will change in memory and post it into the database right after.
But, we want some help/advice from you. Can you suggest something more simple or efficient to implement such a thing? Or maybe some existing classes that already do it for us?
I understand what you guys are trying to show me, and I thank you for that.
But, let's say we have a table with 2000 records, and I will need to list those records. For each one, I have to query other 30 tables (some of them with 1000 records, others with 10 records) to add additional information in the list, and this while it's "flying" (and as you know, we must be very fast at this moment).
Now you'll be going to say: "just build your main query with all those 'joins', and bring all you need in one step. SQLite can be very fast, if your database is well designed, etc...".
OK, but this query will become very complicated and sure, even though SQLite is very fast, it will be "too" slow (2 a 4 seconds, as I confirmed, and this isn't an acceptable time for us).
Another complicator is that, depending on user interaction, we need to "re-query" all records, because the tables involved are not the same, and we have to "re-join" with another set of tables.
So, an alternative is bring only the main records (this will never change, no matter what user does or wants) with no join (this is very fast!) and query the other tables every time we want some data. Note that on the table with 10 records only, we will fetch the same records many and many times. In this case, it is a waste of time, because no matter fast SQLite is, it will always be more expensive to query, cursor, fetch, etc... than just grabbing the record from a kind of "memory cache". I want to make clear that we don't plan to keep all data in memory always, just some tables we query very often.
And we came to the original question: What is the best way to "cache" those records? I really like to focus the discussion on that and not "why do you need to cache data?"
The vast majority of the apps on the platform (contacts, Email, Gmail, calendar, etc.) do not do this. Some of these have extremely complicated database schemas with potentially a large amount of data and do not need to do this. What you are proposing to do is going to cause huge pain for you, with no clear gain.
You should first focus on designing your database and schema to be able to do efficient queries. There are two main reasons I can think of for database access to be slow:
You have really complicated data schemas.
You have a very large amount of data.
If you are going to have a lot of data, you can't afford to keep it all in memory anyway, so this is a dead end. If you have complicated structures, you would benefit in either case with optimizing them to improve performance. In both cases, your database schema is going to be key to good performance.
Actually optimizing the schema can be a bit a of a black art (and I am no expert on it), but some things to look out for are correctly creating indices on rows you will query, designing joins so they will take efficient paths, etc. I am sure there are lots of people who can help you with this area.
You could also try looking at the source of some of the platform's databases to get some ideas of how to design for good performance. For example the Contacts database (especially starting with 2.0) is extremely complicated and has a lot of optimizations to provide good performance on relatively large data and extensible data sets with lots of different kinds of queries.
Update:
Here's a good illustration of how important database optimization is. In Android's media provider database, a newer version of the platform changed the schema significantly to add some new features. The upgrade code to modify an existing media database to the new schema could take 8 minutes or more to execute.
An engineer made an optimization that reduced the upgrade time of a real test database from 8 minutes to 8 seconds. A 60x performance improvement.
What was this optimization?
It was to create a temporary index, at the point of upgrade, on an important column used in the upgrade operations. (And then delete it when done.) So this 60x performance improvement comes even though it also includes the time needed to build an index on one of the columns used during upgrading.
SQLite is one of those things where if you know what you are doing it can be remarkably efficient. And if you don't take care in how you use it, you can end up with wretched performance. It is a safe bet, though, if you are having performance issues with it that you can fix them by improving how you are using SQLite.
The problem with a memory cache is of course that you need to keep it in sync with the database. I've found that querying the database is actually quite fast, and you may be pre-optimizing here. I've done a lot of tests on queries with different data sets and they never take more than 10-20 ms.
It all depends on how you're using the data, of course. ListViews are quite well optimized to handle large numbers of rows (I've tested into the 5000 range with no real issues).
If you are going to stay with the memory cache, you may want have the database notify the cache when it's contents change and then you can update the cache. That way anyone can update the database without knowing about the caching. Also, if you build a ContentProvider over your database, you can use the ContentResolver to notify you of changes if you register using registerContentObserver.