In my Android app, I need to get 50,000 database entries (text) and compare them with a value when the activity starts (in onCreate()). I am doing this with the simplest way: I get the whole table from db to a cursor. However this way is too laggy. Are there any other ways to do it more effectively ?
Edit: The app is "scrabble solver" that is why I am not using WHERE clause in my query (Take the whole data and compare it with combination of the input letters). At first I was using a big table which contains whole possible words. Now I am using 26 tables. This reduced the lag and I am making database calls on a thread - that solved a lot problems too. It is still little bit laggy but much better.
To summarize and add a bit more
Let the database do the work of searching for you, use a WHERE clause when you perform a query!
If your where clause does not operate on the primary key column or a unique column you should create an index for that column. Here is some info on how to do that: http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html#indexes
Google says: "Use question mark parameter markers such as 'phone=?' instead of explicit values in the selection parameter, so that queries that differ only by those values will be recognized as the same for caching purposes."
Run the query analysis using EXPLAIN QUERY PLAN http://www.sqlite.org/lang_explain.html and look for any scan operations, these are much slower than search operations. Uses indexes to avoid scan operations.
Don't perform any time consuming tasks in onCreate(), always use an AsyncTask, a Handler running on a background thread or some other non-main thread.
If you need to do full text search please read: http://www.sqlite.org/fts3.html
You should never read from the database in the UI thread. Use a background thread via AsyncTask or using regular threading. This will fix the UI lag issue your having.
Making the database read faster will help with bringing the data faster to the user but it's even more important that the fetching of the data does not block the user from using the app.
Check out the Following Links:
Painless Threading in Android
YouTube: Writing Zippy Android Apps
Use a WHERE clause in your SQL rather than reading the whole thing in. Then add an index on the columns in the WHERE clause.
At least you can put index on the field you compare and use WHERE clause. If you are comparing numerics Sqlite (db engine used by Android) supports functions such as MIN and MAX. Also if you are comparing partial strings you can use LIKE. For query optimization there are many resources such as this
Related
I am working on android ROOM + Kotlin (Just started)
I want to make a query which update the row of my table, but I am not able to find a definition on how to access the value from the parameter inside the query
#Query("UPDATE note_table SET description = :description, title= :title, priority = :priority WHERE id =:id")
fun updateNote(note : Notes)
I want to access the description from note object. like note.description how to do that inside the query !! Any help will good!
That is not possible, as docs specify
#Query("SELECT * FROM user WHERE age > :minAge")
When this query is processed at compile time, Room matches
the :minAge bind parameter with the minAge method parameter. Room
performs the match using the parameter names. If there is a mismatch,
an error occurs as your app compiles.
Other option is using Raw query, but that is needlessly complected for the use case.
Understand why Room doesn't allow object references
Mapping relationships from a database to the respective object model is a common practice and works very well on the server side. Even when the program loads fields as they're accessed, the server still performs well.
However, on the client side, this type of lazy loading isn't feasible because it usually occurs on the UI thread, and querying information on disk in the UI thread creates significant performance problems. The UI thread typically has about 16 ms to calculate and draw an activity's updated layout, so even if a query takes only 5 ms, it's still likely that your app will run out of time to draw the frame, causing noticeable visual glitches. The query could take even more time to complete if there's a separate transaction running in parallel, or if the device is running other disk-intensive tasks. If you don't use lazy loading, however, your app fetches more data than it needs, creating memory consumption problems.
Object-relational mappings usually leave this decision to developers so that they can do whatever is best for their app's use cases. Developers usually decide to share the model between their app and the UI. This solution doesn't scale well, however, because as the UI changes over time, the shared model creates problems that are difficult for developers to anticipate and debug.
from docs : https://developer.android.com/training/data-storage/room/referencing-data
I've got a table with about 7 million rows in it. I'm inserting on average about one row every second into the database. When I do this, I am noticing that it is taking an incredibly long time (as much as 15 seconds) to run a simple SELECT against the database, e.g. something like:
SELECT * FROM table WHERE rowid > 7100000
This select often returns no rows of data as sometimes no data has been inserted in this particular table. It is often happening even when the table I'm writing to isn't even actually inserting rows into the table I am reading.
The idea is that there are two separate processes, one is adding data, the other is trying to get all new data that has not yet been read. But the read side is connected to a UI and any noticable lag is intolerable, much less 15 seconds. This is being run under Android and the the UI thread doesn't like being blocked for that long either and it is wreaking havoc.
My initial thought was maybe the insert is requiring an update to the indicies as originally I had the index on a different field (a time field). This seems at least partially confirmed because if I use a database with only a few rows each select completes in a few milliseconds. But when I re-created the table to only have the rowid as primary key it actually got slower. I would expect inserting a new row at the end would always result in very fast reads when just comparing on the rowid as primary key.
I have tried enabling write ahead logging, but it appears that SQLCipher doesn't support this, at least not directly, as it doesn't adhere to the lastest API for android.database.sqlite.SQLiteDatabase. Even using "PRAGMA journal_mode = WAL" in the postKey hook hasn't made any difference.
What's going on here? How can I speed up my selects?
Update: I tried getting rid of sqlcipher and just using plain sqlite to see if that was a factor. I used sqlcipher_export to export to a plaintext database, and then used the default android.database.sqlite.SQLCipher. The delay time dropped from 10-20s to 1.8-2.8s. I then removed write-ahead and it dropped further to 1.3-2.7s. So the issue is still noticably there, although it did get a lot better.
SQLite is ultimately file-based, and there is no portable mechanism to communicate to another process which part of a file has changed. So when one process has written something, all other processes must drop their caches when they access the database file the next time.
If possible, modify your architecture that both parts of the code are in the same process and share the same database connection. (With multiple threads, this requires locking, but SQLite has not much concurrency anyway.)
Alternatively, write the new data into a separate database, and let the UI app move it to its own database.
I don't know why SQLCipher is so much slower (it's unlikely to be the CPU overhead of the decryption).
I have an application that is doing a LOT of sqllite transactions, I currently have a bit of a hang because I am doing the sqllite actions on the UI thread... yes bad...
so I made each item have a thread and execute on it assuming sqllite api was smart enough to FIFO them.. nope ... now I get database is locked exceptions
this says it should work
without completely rewriting my code, and having a list of transactions queue up and execute them all on the same thread (many different classes, would be kind of a pain)
is there a way for me to check, and not execute a thread unless there isnt a lock? a lock check per se, or something similar that would get this to work, is efficient and isn't a huge rewrite?
Thanks
My answer that you quoted seems to be confusing. You don't have to do anything special when you are accessing the same Android database using the same database object with multiple threads. Under the covers, Sqlite has it's own locking to guarantee that the database will not be corrupted. To quote my answer;
Sqlite under Android is single threaded. Even if multiple threads were using the same database connection, my understanding is that they would be blocked from running concurrently. There is no way to get around this limitation
It has it's own locking which serializes the requests. This means that adding multiple threads will not increase the performance of the database unfortunately.
As my other answer mentions, you cannot use multiple database objects to the same database from multiple threads since there is no locking and you will corrupt your database.
I'm sort of lost on this. I have an application that is reading from a static SQLite database that has 439397 records (~32MB).
I am querying the database on a column that is indexed, but the it takes ~8-12 seconds to finish the query. The current query I am using is to do database.query(tableName, columnHeaders, "some_id=" + id) for a list of ids.
I tried doing the "WHERE some_id IN (id1, id2, id3)" approach, but that took over twice as long. I have a feeling that I might be doing it wrong.
The query is done in an AsyncTask, so I am at a lost at what other thing I could do to improve the performance.
UPDATE:
I resolved the problem by changing the behavior of the application.
You can use EXPLAIN QUERY PLAN to confirm that your index is indeed being properly used.
You can try running your query once with a COUNT(*) instead of the real column list, to see if the issue is the act of actually reading the row data off of flash storage (which is possible if there are lots of matches and lots of big columns).
You can try running your query to match on a single ID (rather than N of them), to start to try to get a handle on whether the issue is too many comparisons.
However, please bear in mind that AsyncTask does not somehow make things magically faster. It makes things magically not run on the main application thread.
Looks like you don't have an index on that field. Note that the index might need to cover several fields if you use them for filtering/sorting/grouping in your real query (can't tell in more details because I haven't seen it).
In my Android application user can fill database with imported data, then perform predefined, complex SQL query. With some data pattern, query will take very long (minutes and more on HTC Hero), while common usage will not exceed 10-15 seconds.
Is there any way to cancel pending SQLite query on Android? Timeout would be as good as well, partial results are not required. I'm performing query within AsyncTask.doInBackground()
I know best way is optimize query or change app logic, but in this case results depends on user knowledge anyway, so I'd like to help those who enter troublesome pattern.
You can't cancel pending sql query. It's done in SQLite engine. SQLite has native methods to interrupt query, but it's not accessible in java interface.
The best would be to improve query performance.
Read this: http://www.sqlite.org/cvstrac/wiki?p=PerformanceTuning
You can divide long running queries to smaller by limiting row number. Then it would be easier to cancel query that is performed in chunks.
ASyncTask supports cancellation and the flag should cancel all, including your SQL Query.
ASyncTask.cancel(boolean mayInterruptIfRunning)
SQLite provides a intrerupt method:
http://www.sqlite.org/c3ref/interrupt.html
I'm not sure how you would call that in Android though.
You can cancel running queries on sqlite by using android.os.CancellationSignal which can be passed to some of the methods of android.database.sqlite.SQLiteDatabase