Develop Reference

Sqlite insert performance for smartphone

I'm trying to optimize the insertion speed for sqlite in android. I've tried numerous methods. Like setting the pragma settings, journal mode and etc.
Besides that, i've also tried individual inserts, prepared statement and bulk insert. Keep in mind that all of those has already been wrapped in a transaction.
The insertion speed in my smartphone(Samsung Galaxy SIII) is only up to about 250 rows per second for a 20 column table. According to Sqlite faq, sqlite can easily insert 50,000 inserts per second. Therefore, i'm really optimistic about the insert to have at least up to the thousands rather than hundreds. However, none of the above mentioned works. Or 250 rows of inserts is already optimized?
Please help me. Thanks.
Regards,
Dexter

One method I use on big company sized databases to increase bulk insert speed is to turn off indexing while doing bulk inserts. The drawback is you need to have exclusive access to the database and you have to turn indexing back on when your done.
In a mobile app you could easily get exclusive access to the database by putting up an "Updating the app" screen.
SQLITE you can drop any non unique indexes before performing the bulk insert. If your 20 column table has 20 non unique indexes (one for each column). Drop the 20 indexes. Perform the bulk inserts. Create the 20 indexes you previously dropped.
Also make sure your code is compact and doesn't do anything inside the insert loop that could be done before the loop. For example look up column indexes before your insert loop.
If this doesn't apply to your app just catalog it away as something you may need to do in the future on your back end database.

You can use Object/Relational Mapping (ORM) tool for better performance.
ORMLite, GreenDao etc.
Greendao is better one, have a look
http://greendao-orm.com/
Features :-
Maximum performance (probably the fastest ORM for Android)
Easy to use APIs
Highly optimized for Android
Minimal memory consumption
Small library size, focus on the essentials
Benefits of ORM tools :-
ORM can give you Relational Persistence, which results fastest performance.
Less development work

sqlite insert vs update vs replace

I am a working on a project in which I retrieve data from facebook about friends of the user. Friends details vary some times while at the other times they are the same as the one stored in the db.
I can use the replace command to make sure that the db is consistent with whatever information I retrieve from the facebook.
My question is how efficient this technique will be? In other words, I can use two techniques:
One is to use the replace command and replace the complete record blindly
Second is to first check whether there is any difference from the record saved in the db and update only the fields that have changed
Which of these approaches is going to be more efficient?

I've found that queuing up a number of sqlite commands in a row is much more efficient than is doing anything else in between, even just comparing a few values.
I'd strongly recommend that you just do an update command. SQLite is fast.
My observation is that SQLite is always way faster than I am. So let it do the heavy lifting and just dump the data at it, and let it sort out your updates.
For example, I was searching through about 7,000 records. I pulled the records out into an array, did a quick check for one field, and separated it into two arrays. This was taking me about 5 seconds. I replaced it with two separate SQLite queries that each had to go through the entire data base. The revised dual query takes about a quarter second, near as I can tell, because its so crazy fast.
I've had similar speed luck with Updates in my big database.

How does sql lite work with files as database in android

I want to know how exactly does sqllite works when you are dealing with database on android. I know that it writes everything on file with .db extension. But how does it read or write one particular table? Does it fetch the whole file or just the related part and how exactly does it do these operations? Can someone please suggest me some link? I tried google but the links I found just explain how to write queries.

For that you have to read basics of the databases .
all the db frameworks are almost same in terms of working so
you have to research on the basics of database (any).
here is some related information u can like
What does a database actually do to find out what matches a select statement?
To be blunt, it's a matter of brute force. Simply, it reads through each candidate record in the database and matches the expression to the fields. So, if you have "select * from table where name = 'fred'", it literally runs through each record, grabs the "name" field, and compares it to 'fred'.
Now, if the "table.name" field is indexed, then the database will (likely, but not necessarily) use the index first to locate the candidate records to apply the actual filter to.
This reduces the number of candidate records to apply the expression to, otherwise it will just do what we call a "table scan", i.e. read every row.
But fundamentally, however it locates the candidate records is separate from how it applies the actual filter expression, and, obviously, there are some clever optimizations that can be done.
How does a database interpret a join differently to a query with several "where key1 = key2" statements?
Well, a join is used to make a new "pseudo table", upon which the filter is applied. So, you have the filter criteria and the join criteria. The join criteria is used to build this "pseudo table" and then the filter is applied against that. Now, when interpreting the join, it's again the same issue as the filter -- brute force comparisons and index reads to build the subset for the "pseudo table".
How does the database store all its memory?
One of the keys to good database is how it manages its I/O buffers. But it basically matches RAM blocks to disk blocks. With the modern virtual memory managers, a simpler database can almost rely on the VM as its memory buffer manager. The high end DB'S do all this themselves.
How are indexes stored?
B+Trees typically, you should look it up. It's a straight forward technique that has been around for years. It's benefit is shared with most any balanced tree: consistent access to the nodes, plus all the leaf nodes are linked so you can easily traverse from node to node in key order. So, with an index, the rows can be considered "sorted" for specific fields in the database, and the database can leverage that information to it benefit for optimizations. This is distinct from, say, using a hash table for an index, which only lets you get to a specific record quickly. In a B-Tree you can quickly get not just to a specific record, but to a point within a sorted list.
The actual mechanics of storing and indexing rows in the database are really pretty straight forward and well understood. The game is managing buffers, and converting SQL in to efficient query paths to leverage these basic storage idioms.
Then, there's the whole multi-users, locking, logging, and transactions complexity on top of the storage idiom.

SQLite operation on Android is not any different from SQLite operation on any other platform.
Very short answer to your question: SQLite file is split into pages of fixed size.
Each database object (table, index, etc) occupies some number of pages. If objects needs to grow (like new rows are inserted into table) it may allocate more new pages either from free page list, or by growing database file in size. If rows are deleted or object dropped, reclaimed free space goes into free page list. During any operation, SQLite engine tries to NOT fetch whole file, however it maintains page cache for higher performance.
You can find much more detailed explanations on SQLite website in general, and about SQLite database file format in particular.

Use of temp tables in SQLite databases

I was browsing through the source code for Mendeley for Android by Martin Eve and saw that Temp tables are created along with the main tables by calling replace on the tables with _id. For example on the Collections table
db.execSQL(COLLECTIONS_CREATE.replace(" (_id", "_TEMP (_id"));
I guess it creates a new temporary table. Any more explanation on this would be great. Further on the data is first inserted to temp tables and later moved to main tables.
I searched through SO and came across What are the use cases for temporary tables in SQL database systems? and saw that temp tables are used for handling complex queries, sorting and for increasing performance. Can anyone explain how it helps in this situation?

Temp tables make things easier for the programmer by letting the programmer break up a single complex query into multiple relatively simpler queries, and also by letting the programmer store results temporarily so they can be consulted multiple times for different purposes during the course of the program without having to be reinstantiated each time. The latter also makes things easier for the computer. The disk subsystem and CPU can take a little rest, so to speak.
An example of the former: let say you wanted to get all records where:
the sale was in the eastern division
and involved one of the several new gizmos introduced last quarter
and occurred during the special 5-day bonanza sale
or
the sale was made by the boss's daughter
who floats from division to division
and the sale occurred at any time during the month of May
Your program will then generate an email praising the salesperson for the sale, with a cc to the division manager.
The single query that fetches records that satisfy either of those sets of conditions above might get a little unwieldy--just a little difficult for one's addled brain or weary eyes to handle after a long day of dealing with the sort of crap one has to deal with in most walks of life. It's a trivial example, of course; in a production system the conditions are often more complex than those above, involving calculations and tests for null values and all sorts of other tedious things that can cause a query statement to grow long and turn into a ball of tangled yarn.
So if you created a temp table, you could populate the temp table with the rows that satisfy the first set of conditions, and then write the second query that grabs the rows that satisfy the second set of conditions, and insert them into the temp table too, and voilà -- your temp table contains all the rows you need to work with, in two baby steps.

Temporary tables are just that, temporary. They go away once you close your connection, at least with sqlite. It's often more complicated than that with other DMBS', although this is usually the default behaviour.
So temporary tables are thus used whenever a temporary table is required! (Just kiddin'). Performance will be better as there is no I/O involved (which is often the cause of performance problems with databases).
Another use case is when you want to use what is in a permanent table without changing anything in the said table; you can then select the permanent table in a temp table, and keep your permanent table intact.

Android SQLite Performance with Indexes

My Android app works by using a SQLite database that is generated on the user's PC and transferred to the device. It all works, but I had not anticipated the number of users who would have really huge amounts of data. In these cases, the UI is very sluggish as it waits for the data to be fetched.
I've tried a number of tricks that I was "sure" would speed things up, but nothing seems to have any noticeable effect. My queries are almost all very simple, being usually a single "col=val" for the WHERE clause, and INTEGER data in the column. So I can't do much with the queries.
The latest, and I am not an SQL expert by any means, was to use "CREATE INDEX" commands on the PC, believing that these indexes are used to speed up database searches. The indexes increased the size of the database file significantly, so I was then surprised that it seemed to have no effect whatsoever on the speed of my app! A screen that was taking 8 seconds to fill without indexes still takes about 8 seconds even with them. I was hoping to get things down to at least half that.
What I am wondering at this point is if the SQLite implementation on Android uses database indexes at all, or if I'm just wasting space by generating them. Can anyone answer this?
Also, any other things to try to speed up access?
(For what it's worth, on an absolute basis the users have nothing to complain about. My worst-case user so far has data that generates 630,000 records (15 tables), so there's only so much that's possible!)
Doug Gordon
GHCS Systems

SQLite will use the index if it is appropriate for the query. Use EXPLAIN
EXPLAIN QUERY PLAN ... your select statement ...
to see what indexes SQLite is using. The query plan is based on some assumptions about your database content. You may be able to improve the plan by using ANALYZE

I was finally able to achieve tremendous performance gains simply by querying the database in a much more efficient way. For example, in building up an array of information, I was previously querying the database for each row that I required with a "WHERE _id = n" type selector. But in doing it this way, I was issuing a dozen or more queries, one at a time.
Instead, I now build up a list of IDs that are required, then get them all with a single query of the form "WHERE _id IN (n1, n2, n3, ...)" and iterate through the returned cursor. Doing this and some other structure optimizations, the largest database is now almost as quick to view as the more average case.

Every time you're going to perform some kind of action (being database lookup, long-running calculation, web request etc.) taking more than a couple of hundreds of milliseconds, you should consider wrapping this inside an AsyncTask.
Painless Threading is a good article on this topic, so I recommend you take a close look at it.
This article discusses the threading
model used by Android applications and
how applications can ensure best UI
performance by spawning worker threads
to handle long-running operations,
rather than handling them in the main
thread.