I am currently developing an Android app that works with an SQLite database locally. I am wondering what the best way is to load data from the database.
the database would contain about 3 tables that would total up to about 500 rows. What would be the best option in this case:
- Load all data on startup to fill the java model for use in the application.
- Load only data that is needed in each screen. This probably requires db calls on nearly each screen load but will need less memory. If this is the best solution, how do you handle this situation ? If you would open a view, you would have to query the db since not everything is pre-loaded. But if you would open it again afterwards, you would have to have some kind of 'caching' mechanism to detect if it's already pre-loaded ?
Sorry if my question is not very clear, I find this difficult to describe :-\ .
Thanks in advance for any tips.
Cheers
Wesley
What would you do if your database grew to 5,000 rows? 50,000?
Premature optimization is one of the leading causes of poor design. In my (relatively limited) experience, SQLite database queries are fast enough. Try loading your data on demand, as is fairly standard practice, and see if your program runs quickly enough.
Related
I'm asking for a piece of advice. Currently, we are developing android client for out service. Service produces like a lot of dynamic information, and it must be stored on users phone so it can be accessed without connection to the net. On iOS client we achieved this using restKit. On android I found that there is no tool like restKit. So there are 2 options - use sqlite db or cache last json response. I want to use sqlite db, but our android developer sad that it's not stable and slow. Is he right? What practice is better?
The second question is that I found a sqlite editor app, which allows to edit sqlite databases on phone. Is there any way to avoid this?
You can use SQLite because SQLite is capable of being extremely fast. If you are seeing speeds slower than other DB systems such as MySQL or PostGres, then you are not utilizing SQLite to its full potential. Optimizing for speed appears to be the second priority of D. Richard Hipp, the author and maintainer of SQLite. The first is data integrity and verifiability.
The first thing you should know is that most of the time spent by SQLite (and most other DB systems) is on disk access, which is slow compared to memory operations. So the key to optimizing SQLite is to minimize the amount of disk access required. This requires some understanding of when SQLite is accessing information on disk, and why. Because operating systems generally cache open disk files in memory, a carefully tuned use of SQLite can approach the speed of a pure in-memory database.
If you are new to optimization, you must know that the only reliable way to optimize for speed is to measure where time is being spent. Unless you are already familiar with doing this for SQLite, you will often guess wrong. Use quantitative tests on representative examples whenever possible. Unfortunately, reproducibly measuring performance on an application that does disk access isn't trivial.
However it is difficult to use cache no doubt cache is fast but its not for large data and data cannot stored on cache for long time. So if you want that user will use your app offline then you should place your data on SQLite in an optimized way.
Hope this will help you.
No according to experience SQLite is the most reliable database to use in android device itself. It doesn't have separate server process and it directly read/right to single disk file.
This link will provide more information
Let me explain how my application is supposed to work:
Application will ship with a sqlite database in its assets folder which will be copied into databases folder later and it has some content in it(categories, sub categories, products and news) which they all have image. Then after download user can update the content via internet and the application store the new content in database so the application can perform offline.
So my question is, after a while this content will increase in size, is it gonna cause my application to crash? Lets say I release the application with 1 MB database and after 2 years of work the database size goes up around 120 MB. Is it gonna make the application to crash?
Also the other concern is that currently I'm storing the images in database and I load'em from there. Is it a good approach? Because I don't want user to be able to clear the cache or delete the images because later on updating the content it has to download those deleted images again and it will consume traffic imo.
please remember that the Application should be able to load content offline
No, applications don't just crash because they have a large database.
Part of the point of a Cursor is that it gives you a view into a large set of data, without having to load it all into memory at the same time.
If you follow best practices I see no problem - you're using a database. Forget for a second that it's on Android - you should optimize your table structure, indexes, etc, as best you can.
Also, large database or not, don't make any queries to it on the main thread. Use the Loader API if you need to show the result of a query in your UI.
Last, potentially most importantly, rethink why you even need such a large database. Is it really that common that a user will need to access all data ever while offline? Or might it make more sense for you to only store data from the last week or month, etc, and tell them that they need to be online to access older data.
Regarding your 2nd question - please in the future separate that into a separate question. But, no, storing binary blobs (images in this case) in a sqlite database is not good approach. Also, if they clear data on the app, everything is gone, so there's no advantage to using a database to avoid that. I would suggest storing images in a folder named after your app in external storage of the device, potentially storing image URIs/names in the database.
Any problem with database will cause SQLiteException which you are able to handle in your app to prevent the abnormal termination.
Having said that, a database of 120 MB seems to be too much, are you sure your users will want all that?
i am coding an android app which needs a database. I won't use sqlite because i want a pure java core without any dependencies to androids library. To simplify the database access i'm using ORMLite.
So i've just compared the ORMLite android examples
HelloAndroid and HelloAndroidH2.
I've reduced the functionality of both examples to read operations.
The test tables(2 colums, primary_key;value) contains 2 datasets.
The Result:
SQLite: results appears immediately
H2: needs about two seconds to load results.
Where is the mistake or is it correct? Does h2 really need such a long time to load two small Datasets? Any other experiences?
Opening an H2 database using the default options is relatively slow on Android, as documented. There are a few ways to improve that, most of them are documented in the Android section of H2:
FILE_LOCK=FS (to use native file locking; saves at least 20 ms)
PAGE_SIZE=1024 (using a smaller page size seems to improve performance here)
CACHE_SIZE=8192 (to avoid using too much heap memory for the cache)
Also quite important is using an empty user name and password. If you don't, then the password is hashed, which is relatively slow on Android.
But in any case I think you will not be able to get the same opening speed as SQLite in the near future, sorry.
I am new to Android Application Development and a new member at stackoverflow. I am currently trying to design a recipe application. I have decided upon the features of the app and the scope it will cover. The scope is very vast for me in terms of covering all the recipes from all over the world. I am to deal with a lot of data in this process.
I am currently trying to figure a good and efficient way of handling the data in my app. So far, as per what I have read in different forums, I believe that I have two options in terms of a database choice : 1) SQLite 2) Database on remote server (MySql/Postgre)
Following are some of the thoughts that have been going on in my mind when it comes to taking a decision between the two :
1) SQLite : This could be a good option but would be slow as it would need to access the file system. I could eliminate the slowness by performing DB data fetch tasks in the AsyncTask. But then there could be a limitation of the storage on different phones. Also I believe using SQLite would be easier as compared to using a remote DB.
2) Remote Database : The issue that I can see here is the slowness with multiple DB requests coming at the same time. Can I use threads here in some way to queue multiple requests and handle them one by one ? Is there an efficient way to do this.
Also I have one more question in terms of the formatting of my data once I pull it out from the above DB's. Is there a way I could preserve the formatting of my data ?
I would be more than thankful if someone could share their knowledgeable and expert comments on the above scenario. Also this is not a homework for me and I am not looking for any ready made code solutions. I am just looking for hints/suggestions that would help me clear my thoughts and help me take a decision. I have been looking for this for sometime now but was not able to find concrete information. I hope I will get some good advice here from the experienced people who might have encountered similar situation.
Thanks for reading this long post.
What about combining both approaches?
A local SQLite database that has the least recently used receipes so you don't need network all the time. Network is way slower than accessing the filesystem.
Some remote database accessed via some HTTP interface where you can read / write the whole database. And if you want users to be able to add receipes for other users to see you'll need an external database anyways.
SQLite : This could be a good option but would be slow as it would need to access the file system.
Accessing a local database is pretty fast, 5ms or so if it's just a simple read only query on a small database.
But then there could be a limitation of the storage on different phones
Depends on your definition of huge database. It is okay if it is only 2MB which would be enough to store lots of text-only receipes.
Also I believe using SQLite would be easier as compared to using a remote DB.
Yes, Android has a nice built-in SQLite API but no remote database API. And you don't need to setup a database server & interface.
The issue that I can see here is the slowness with multiple DB requests coming at the same time.
A decent database server can handle thousands of requests. Depends on your server hardware & software. https://dba.stackexchange.com/ should have more info on that. Required performance depends on how much users you have / expect.
I'd suggest a simple REST interface to your database since it's pretty lightweight but does not expose your database directly to the web. There are tons of tutorials and books about creating such interfaces to databases. There are even hosted database services like nextDb that do most of the work for you.
Is there a way I could preserve the formatting of my data ?
You could store HTML formatted data in your database and display it in a WebView or a TextView (via Html#fromHtml()) - both can display formatted text.
Databases don't care what type of text you store, for transfer over the internets you may need to encode the text so it does not interfere with the transport formatting (XML, JSON, ...).
A simple way is to integrate Parse into your app. They have a nice framework that easily integrates into iOS and Android. Their plan is freemium, so you'll be able to use up to 1 million API request for no charge, and then its 7 cents for every request after that.
You'll have 1gb to store all your data sets / images, etc.
I don't use parse for everything, but I HIGHLY recommended it for large data schemes because they do all the scaling for you. Check out the API, I think it would be worth your time.
I just started to work on a few of my own projects, and I'm using Parse again. I have to say it's improved a lot over the last 6-8 months. Especially with the Twitter and Facebook integration.
The key issue here is the size of the data - any significant database of recipes would be too large to store on the phone imho,thus you seem stuck with the remote database solution.
As opposed to trying access the remote database from android I suggest you use a a go between web application that will process requests from the application and return JSON objects that you need.
It totally depends on your software requirements. If you need to deal with a small amount of data then you may choose SQLite, but for a huge amount to data better use a remote DB.
SQLite: It works fine with little amount of data & I experienced it response time is good.
Remote DB: I think you may use small server side app to submit the data to your client app. It will solve/reduce your thread related issues/complexities.
We're designing an Android app that has a lot of data ("customers", "products", "orders"...), and we don't want to query SQLite every time we need some record. We want to avoid to query the database as most as we can, so we decided to keep certain data always in memory.
Our initial idea is to create two simple classes:
"MemoryRecord": a class that will contain basically an array of objects (string, int, double, datetime, etc...), that are the data from a table record, and all methods to get those data in/out from this array.
"MemoryTable": a class that will contain basically a Map of [Key,MemoryRecord] and all methods to manipulate this Map and insert/update/delete record into/from database.
Those classes will be derived to every kind of table we have in the database. Of course there are other useful methods not listed above, but they are not important at this point.
So, when starting the app, we will load those tables from an SQLite database to memory using those classes, and every time we need to change some data, we will change in memory and post it into the database right after.
But, we want some help/advice from you. Can you suggest something more simple or efficient to implement such a thing? Or maybe some existing classes that already do it for us?
I understand what you guys are trying to show me, and I thank you for that.
But, let's say we have a table with 2000 records, and I will need to list those records. For each one, I have to query other 30 tables (some of them with 1000 records, others with 10 records) to add additional information in the list, and this while it's "flying" (and as you know, we must be very fast at this moment).
Now you'll be going to say: "just build your main query with all those 'joins', and bring all you need in one step. SQLite can be very fast, if your database is well designed, etc...".
OK, but this query will become very complicated and sure, even though SQLite is very fast, it will be "too" slow (2 a 4 seconds, as I confirmed, and this isn't an acceptable time for us).
Another complicator is that, depending on user interaction, we need to "re-query" all records, because the tables involved are not the same, and we have to "re-join" with another set of tables.
So, an alternative is bring only the main records (this will never change, no matter what user does or wants) with no join (this is very fast!) and query the other tables every time we want some data. Note that on the table with 10 records only, we will fetch the same records many and many times. In this case, it is a waste of time, because no matter fast SQLite is, it will always be more expensive to query, cursor, fetch, etc... than just grabbing the record from a kind of "memory cache". I want to make clear that we don't plan to keep all data in memory always, just some tables we query very often.
And we came to the original question: What is the best way to "cache" those records? I really like to focus the discussion on that and not "why do you need to cache data?"
The vast majority of the apps on the platform (contacts, Email, Gmail, calendar, etc.) do not do this. Some of these have extremely complicated database schemas with potentially a large amount of data and do not need to do this. What you are proposing to do is going to cause huge pain for you, with no clear gain.
You should first focus on designing your database and schema to be able to do efficient queries. There are two main reasons I can think of for database access to be slow:
You have really complicated data schemas.
You have a very large amount of data.
If you are going to have a lot of data, you can't afford to keep it all in memory anyway, so this is a dead end. If you have complicated structures, you would benefit in either case with optimizing them to improve performance. In both cases, your database schema is going to be key to good performance.
Actually optimizing the schema can be a bit a of a black art (and I am no expert on it), but some things to look out for are correctly creating indices on rows you will query, designing joins so they will take efficient paths, etc. I am sure there are lots of people who can help you with this area.
You could also try looking at the source of some of the platform's databases to get some ideas of how to design for good performance. For example the Contacts database (especially starting with 2.0) is extremely complicated and has a lot of optimizations to provide good performance on relatively large data and extensible data sets with lots of different kinds of queries.
Update:
Here's a good illustration of how important database optimization is. In Android's media provider database, a newer version of the platform changed the schema significantly to add some new features. The upgrade code to modify an existing media database to the new schema could take 8 minutes or more to execute.
An engineer made an optimization that reduced the upgrade time of a real test database from 8 minutes to 8 seconds. A 60x performance improvement.
What was this optimization?
It was to create a temporary index, at the point of upgrade, on an important column used in the upgrade operations. (And then delete it when done.) So this 60x performance improvement comes even though it also includes the time needed to build an index on one of the columns used during upgrading.
SQLite is one of those things where if you know what you are doing it can be remarkably efficient. And if you don't take care in how you use it, you can end up with wretched performance. It is a safe bet, though, if you are having performance issues with it that you can fix them by improving how you are using SQLite.
The problem with a memory cache is of course that you need to keep it in sync with the database. I've found that querying the database is actually quite fast, and you may be pre-optimizing here. I've done a lot of tests on queries with different data sets and they never take more than 10-20 ms.
It all depends on how you're using the data, of course. ListViews are quite well optimized to handle large numbers of rows (I've tested into the 5000 range with no real issues).
If you are going to stay with the memory cache, you may want have the database notify the cache when it's contents change and then you can update the cache. That way anyone can update the database without knowing about the caching. Also, if you build a ContentProvider over your database, you can use the ContentResolver to notify you of changes if you register using registerContentObserver.