We integrate the SqlCiper sqlite db into our Android project. What we found out that there is some big difference of db query performance in unecrypted SqlCiper sqlite db and encrypted SqlCiper sqlite db.
We just did some basic timing logging at our code on same Android device:
unencrypted SqlCiper db: 100 db query, total time: 1-2 seconds
encrypted SqlCiper db: 100 db query, total time: 17 seconds.
As you could see that there is big increase in the running time when encryption is turned on in SqlCiper database.
Based on this post: SqlCiper Performance and SqlCiperSpeed, we won't see such big increase on iOS, however I didn't see any performance number on Android.
Do you guys see the same issues as we saw? Any suggestions to improve it?
First, ensure that you have a problem worth worrying about. A query that takes 170ms, instead of 10ms, is unlikely to be a material difference to the user, in isolation. In either case, you need to be using a background thread, as even 10ms is enough to cause you to drop a frame or two, depending on what else is going on. Hence, if you are doing this query in response to some discrete user request (e.g., tapping on an action item), the user is unlikely to notice the difference. Only if you are doing a lot of these queries in a short time frame are you likely to have speed issues that might cause problems for the user.
Second, ensure that you have tuned your database access in general, using things like the EXPLAIN keyword. SQLCipher for Android makes poor database I/O worse, such as queries that result in table scans (e.g., due to not having the right indices).
Third, use Traceview to determine precisely where your time is being spent.
You can optimized by the way:
1.In SQLCipher lib, used AES algorithm to encrypt data. --> You can change AES --> RC4. Performance increased 10-15%. (You can compared AES & RC4 before use it)
2.Important you shoud optimize data SQLite (create index, sql query,...);
http://www.sqlite.org/optoverview.html
Related
I'm asking for a piece of advice. Currently, we are developing android client for out service. Service produces like a lot of dynamic information, and it must be stored on users phone so it can be accessed without connection to the net. On iOS client we achieved this using restKit. On android I found that there is no tool like restKit. So there are 2 options - use sqlite db or cache last json response. I want to use sqlite db, but our android developer sad that it's not stable and slow. Is he right? What practice is better?
The second question is that I found a sqlite editor app, which allows to edit sqlite databases on phone. Is there any way to avoid this?
You can use SQLite because SQLite is capable of being extremely fast. If you are seeing speeds slower than other DB systems such as MySQL or PostGres, then you are not utilizing SQLite to its full potential. Optimizing for speed appears to be the second priority of D. Richard Hipp, the author and maintainer of SQLite. The first is data integrity and verifiability.
The first thing you should know is that most of the time spent by SQLite (and most other DB systems) is on disk access, which is slow compared to memory operations. So the key to optimizing SQLite is to minimize the amount of disk access required. This requires some understanding of when SQLite is accessing information on disk, and why. Because operating systems generally cache open disk files in memory, a carefully tuned use of SQLite can approach the speed of a pure in-memory database.
If you are new to optimization, you must know that the only reliable way to optimize for speed is to measure where time is being spent. Unless you are already familiar with doing this for SQLite, you will often guess wrong. Use quantitative tests on representative examples whenever possible. Unfortunately, reproducibly measuring performance on an application that does disk access isn't trivial.
However it is difficult to use cache no doubt cache is fast but its not for large data and data cannot stored on cache for long time. So if you want that user will use your app offline then you should place your data on SQLite in an optimized way.
Hope this will help you.
No according to experience SQLite is the most reliable database to use in android device itself. It doesn't have separate server process and it directly read/right to single disk file.
This link will provide more information
I'm trying to optimize the insertion speed for sqlite in android. I've tried numerous methods. Like setting the pragma settings, journal mode and etc.
Besides that, i've also tried individual inserts, prepared statement and bulk insert. Keep in mind that all of those has already been wrapped in a transaction.
The insertion speed in my smartphone(Samsung Galaxy SIII) is only up to about 250 rows per second for a 20 column table. According to Sqlite faq, sqlite can easily insert 50,000 inserts per second. Therefore, i'm really optimistic about the insert to have at least up to the thousands rather than hundreds. However, none of the above mentioned works. Or 250 rows of inserts is already optimized?
Please help me. Thanks.
Regards,
Dexter
One method I use on big company sized databases to increase bulk insert speed is to turn off indexing while doing bulk inserts. The drawback is you need to have exclusive access to the database and you have to turn indexing back on when your done.
In a mobile app you could easily get exclusive access to the database by putting up an "Updating the app" screen.
SQLITE you can drop any non unique indexes before performing the bulk insert. If your 20 column table has 20 non unique indexes (one for each column). Drop the 20 indexes. Perform the bulk inserts. Create the 20 indexes you previously dropped.
Also make sure your code is compact and doesn't do anything inside the insert loop that could be done before the loop. For example look up column indexes before your insert loop.
If this doesn't apply to your app just catalog it away as something you may need to do in the future on your back end database.
You can use Object/Relational Mapping (ORM) tool for better performance.
ORMLite, GreenDao etc.
Greendao is better one, have a look
http://greendao-orm.com/
Features :-
Maximum performance (probably the fastest ORM for Android)
Easy to use APIs
Highly optimized for Android
Minimal memory consumption
Small library size, focus on the essentials
Benefits of ORM tools :-
ORM can give you Relational Persistence, which results fastest performance.
Less development work
I have a huge database and I want my application to work with it as soon as possible. I'm using android so resources are more restricted. I know that its not a good idea to storage huge data in the sqlite database, but I need this.
Each database contain only ONE table and I use it READ only.
What advice can you give me to optimize databases as much as possible. I've already read this post, and except the PRAGMA commands what else can I use?
Maybe there are some special types of the tables which are restricted for read only queries, but principally faster then ordinary table types?
As long as your database fits on the device, there is no problem with that; you'll just have less space for other apps.
There is no special table type. However, if you have queries that use only a subset of a table's columns, and if you have enough space left, consider adding one or more covering indexes.
Being read-only allows the database to be optimized on the desktop, before you deploy it:
set page size, etc.;
create useful indexes;
ANALYZE
VACUUM
In your app, you might experiment with increasing the page cache size, but if your working set is larger than free memory, that won't help anyway. In any case, random reads from flash are fast, so that would not be much of a problem.
Huge is relative. But ultimately a device is constrained on storage and memory. So assuming that huge is beyond the typical constraints of a device, you have a few options.
The first option is to store your huge dataset in the cloud and the connected device can offer views into that data by offering cloud services with something like RESTful APIs from the coud to proffer the data to the device. If the device and app rely on always being connected, you don't need as much local storage unless you want to cache data.
Another approach is an occasionally connected device (sometimes offline) where you pull down a slice of the most relevant data to work on to the device. In that model, yo can work offline and push/pull back to the cloud. In this model, sqlite is the storage mechanism to hold that slice of relevant data.
EDIT based on comments:
Concerning optimizing what you have on the device, see the optimization FAQ here:
http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html
(in rough order of effectiveness)
Use an in-memory database
Use BEGIN TRANSACTION and END TRANSACTION
Use indexes Use PRAGMA cache_size
Use PRAGMA synchronous=OFF
Compact the database
Replace the memory allocation library
Use PRAGMA count_changes=OFF
Maybe I'm stating the obvious but you should probably just open it with the SQLITE_OPEN_READONLY flag to sqlite3_open: I think that SQLite will take advantage of this fact and optimize the behaviour of the engine.
Note that all normal SQL(ite) optimization tips still apply (e.g. VACUUMing to finalize the database, setting the correct page size at database creation, proper indexes and so on...)
In addition, if you have multiple threads accessing the database in your application, you may want to try out also the SQLITE_OPEN_NOMUTEX and SQLITE_OPEN_SHAREDCACHE flags (they require sqlite3_open_v2, though)
Also you need journalling switch off, because data not change http://www.sqlite.org/pragma.html#pragma_journal_mode
PRAGMA journal_mode=OFF
My Android app works by using a SQLite database that is generated on the user's PC and transferred to the device. It all works, but I had not anticipated the number of users who would have really huge amounts of data. In these cases, the UI is very sluggish as it waits for the data to be fetched.
I've tried a number of tricks that I was "sure" would speed things up, but nothing seems to have any noticeable effect. My queries are almost all very simple, being usually a single "col=val" for the WHERE clause, and INTEGER data in the column. So I can't do much with the queries.
The latest, and I am not an SQL expert by any means, was to use "CREATE INDEX" commands on the PC, believing that these indexes are used to speed up database searches. The indexes increased the size of the database file significantly, so I was then surprised that it seemed to have no effect whatsoever on the speed of my app! A screen that was taking 8 seconds to fill without indexes still takes about 8 seconds even with them. I was hoping to get things down to at least half that.
What I am wondering at this point is if the SQLite implementation on Android uses database indexes at all, or if I'm just wasting space by generating them. Can anyone answer this?
Also, any other things to try to speed up access?
(For what it's worth, on an absolute basis the users have nothing to complain about. My worst-case user so far has data that generates 630,000 records (15 tables), so there's only so much that's possible!)
Doug Gordon
GHCS Systems
SQLite will use the index if it is appropriate for the query. Use EXPLAIN
EXPLAIN QUERY PLAN ... your select statement ...
to see what indexes SQLite is using. The query plan is based on some assumptions about your database content. You may be able to improve the plan by using ANALYZE
I was finally able to achieve tremendous performance gains simply by querying the database in a much more efficient way. For example, in building up an array of information, I was previously querying the database for each row that I required with a "WHERE _id = n" type selector. But in doing it this way, I was issuing a dozen or more queries, one at a time.
Instead, I now build up a list of IDs that are required, then get them all with a single query of the form "WHERE _id IN (n1, n2, n3, ...)" and iterate through the returned cursor. Doing this and some other structure optimizations, the largest database is now almost as quick to view as the more average case.
Every time you're going to perform some kind of action (being database lookup, long-running calculation, web request etc.) taking more than a couple of hundreds of milliseconds, you should consider wrapping this inside an AsyncTask.
Painless Threading is a good article on this topic, so I recommend you take a close look at it.
This article discusses the threading
model used by Android applications and
how applications can ensure best UI
performance by spawning worker threads
to handle long-running operations,
rather than handling them in the main
thread.
We're designing an Android app that has a lot of data ("customers", "products", "orders"...), and we don't want to query SQLite every time we need some record. We want to avoid to query the database as most as we can, so we decided to keep certain data always in memory.
Our initial idea is to create two simple classes:
"MemoryRecord": a class that will contain basically an array of objects (string, int, double, datetime, etc...), that are the data from a table record, and all methods to get those data in/out from this array.
"MemoryTable": a class that will contain basically a Map of [Key,MemoryRecord] and all methods to manipulate this Map and insert/update/delete record into/from database.
Those classes will be derived to every kind of table we have in the database. Of course there are other useful methods not listed above, but they are not important at this point.
So, when starting the app, we will load those tables from an SQLite database to memory using those classes, and every time we need to change some data, we will change in memory and post it into the database right after.
But, we want some help/advice from you. Can you suggest something more simple or efficient to implement such a thing? Or maybe some existing classes that already do it for us?
I understand what you guys are trying to show me, and I thank you for that.
But, let's say we have a table with 2000 records, and I will need to list those records. For each one, I have to query other 30 tables (some of them with 1000 records, others with 10 records) to add additional information in the list, and this while it's "flying" (and as you know, we must be very fast at this moment).
Now you'll be going to say: "just build your main query with all those 'joins', and bring all you need in one step. SQLite can be very fast, if your database is well designed, etc...".
OK, but this query will become very complicated and sure, even though SQLite is very fast, it will be "too" slow (2 a 4 seconds, as I confirmed, and this isn't an acceptable time for us).
Another complicator is that, depending on user interaction, we need to "re-query" all records, because the tables involved are not the same, and we have to "re-join" with another set of tables.
So, an alternative is bring only the main records (this will never change, no matter what user does or wants) with no join (this is very fast!) and query the other tables every time we want some data. Note that on the table with 10 records only, we will fetch the same records many and many times. In this case, it is a waste of time, because no matter fast SQLite is, it will always be more expensive to query, cursor, fetch, etc... than just grabbing the record from a kind of "memory cache". I want to make clear that we don't plan to keep all data in memory always, just some tables we query very often.
And we came to the original question: What is the best way to "cache" those records? I really like to focus the discussion on that and not "why do you need to cache data?"
The vast majority of the apps on the platform (contacts, Email, Gmail, calendar, etc.) do not do this. Some of these have extremely complicated database schemas with potentially a large amount of data and do not need to do this. What you are proposing to do is going to cause huge pain for you, with no clear gain.
You should first focus on designing your database and schema to be able to do efficient queries. There are two main reasons I can think of for database access to be slow:
You have really complicated data schemas.
You have a very large amount of data.
If you are going to have a lot of data, you can't afford to keep it all in memory anyway, so this is a dead end. If you have complicated structures, you would benefit in either case with optimizing them to improve performance. In both cases, your database schema is going to be key to good performance.
Actually optimizing the schema can be a bit a of a black art (and I am no expert on it), but some things to look out for are correctly creating indices on rows you will query, designing joins so they will take efficient paths, etc. I am sure there are lots of people who can help you with this area.
You could also try looking at the source of some of the platform's databases to get some ideas of how to design for good performance. For example the Contacts database (especially starting with 2.0) is extremely complicated and has a lot of optimizations to provide good performance on relatively large data and extensible data sets with lots of different kinds of queries.
Update:
Here's a good illustration of how important database optimization is. In Android's media provider database, a newer version of the platform changed the schema significantly to add some new features. The upgrade code to modify an existing media database to the new schema could take 8 minutes or more to execute.
An engineer made an optimization that reduced the upgrade time of a real test database from 8 minutes to 8 seconds. A 60x performance improvement.
What was this optimization?
It was to create a temporary index, at the point of upgrade, on an important column used in the upgrade operations. (And then delete it when done.) So this 60x performance improvement comes even though it also includes the time needed to build an index on one of the columns used during upgrading.
SQLite is one of those things where if you know what you are doing it can be remarkably efficient. And if you don't take care in how you use it, you can end up with wretched performance. It is a safe bet, though, if you are having performance issues with it that you can fix them by improving how you are using SQLite.
The problem with a memory cache is of course that you need to keep it in sync with the database. I've found that querying the database is actually quite fast, and you may be pre-optimizing here. I've done a lot of tests on queries with different data sets and they never take more than 10-20 ms.
It all depends on how you're using the data, of course. ListViews are quite well optimized to handle large numbers of rows (I've tested into the 5000 range with no real issues).
If you are going to stay with the memory cache, you may want have the database notify the cache when it's contents change and then you can update the cache. That way anyone can update the database without knowing about the caching. Also, if you build a ContentProvider over your database, you can use the ContentResolver to notify you of changes if you register using registerContentObserver.