I have a table (rowID, name, familyName) with at least 100.000 rows (SQLite). I am using this code to query my table when the text of an EditText changes (onTextChanged):
mySelectSql = "Select rowID from gamtable WHERE name GLOB ' " + searchEditText.getText().toString() + "*';";
Cursor c1 = db.rawQuery(mySelectSql, null);
rowID is primary key, and name is indexed! This code runs slowly (~6000ms) just on the first key that entered in the EditText, and works fast and very fast on second, third, etc.
I need to improve my code to result in first entered key as fast as second and third one...
Any help on this issue?
Assuming it's actually using an index (sounds like you've checked?), I would simply limit the number of results. Using the maximum number of displayable results in the UI preview.
SELECT rowID FROM gamtable WHERE name GLOB 'foo*' LIMIT 15;
Related
I have a value from the epoch time like this 1549251913000, I save this value to SQLite. I create the table like the following:
CREATE TABLE TABLE_BOOKMARK (COLUMN_ID INTEGER PRIMARY KEY AUTOINCREMENT, COLUMN_TITLE TEXT, COLUMN_SOURCE TEXT, COLUMN_DATEANDTIME INTEGER, COLUMN_GUID TEXT);
that value is COLUMN_DATEANDTIME but with INTEGER type. but when I take the value, it doesn't match what I expected. it becomes like this -1231280856.
Please give me some advice, thanks
I have tried this solution but still mismatch when I get it from SQLite (seems it didn't work with my problem)
Create a column in as INTEGER datatype put LONG value into INTEGER in column
Make sure when you retrieve the value from cursor as LONG
Cursor cursor = db.rawQuery("SELECT * FROM " + TABLE_NAME, null);
long value = cursor.getLong(0);
For more : SQLite DataType Doc
Just to clarify the column type is, with one exception, largely irrelevant as any type of data can be stored in any type of column.
The exception is the rowid column or an alias of the rowid column, such a column MUST store an integer. By integer it is a 64bit signed integer and thus encompasses a java long.
The column type itself is also flexible for example CREATE TABLE mytable (mycolumn a_pretty_weird_column_type) is valid (as would be LONG). Such types are converted according to 5 rules to the column affinity.
Putting the above together, using :-
CREATE TABLE IF NOT EXISTS mytable (mycolumn a_pretty_weird_column_type);
DELETE FROM mytable;
INSERT INTO mytable VALUES
(1549251913000),
(999999999999999),('Fred'),(x'010203040506070809'),(0.234567),(null),
('999999999999999'), -- Note although specified as TEXT as mycolumn is effectively NUMERIC stored as INTEGER
('0.234567') -- As above but stored as REAL
;
SELECT
*,
typeof(mycolumn) AS coltype, -- The column type (note as per value not column definition)
hex(mycolumn) AS as_hex, -- Convert column to a hex representation of the data
CAST(mycolumn AS TEXT) AS as_text, -- follow rules
CAST(mycolumn AS INTEGER) AS as_integer,
CAST(mycolumn AS REAL) AS as_real,
CAST(mycolumn AS NUMERIC) AS as_numeric,
CAST(mycolumn AS BLOB) AS as_blob
FROM mytable;
results in :-
The bible as such is Datatypes In SQLite Version 3.
but when I take the value, it doesn't match what I expected. it
becomes like this -1231280856
The Issue
As such your issue is nothing to do with the column type, not SQLite as such, rather it's due to using the Cursor getInt method, instead of the getLong method.
One answer says,
Its always better to store date and time in form of Text in sqlite.
This is incorrect, from a space point of view and therefore the underlying efficiency the use of a numeric representation, i.e. 64bit signed integer as per SQL As Understood By SQLite - Date And Time Functions - Time Strings (maximum of 8 bytes), of the date and time will be more efficient than storing the 19 bytes (for an accuracy down to a second).
As per the docs sqlite does not have default storage class for date and time. Its always better to store date and time in form of Text in sqlite. You can then parse them into Date runtime whenever you want to use them
I have a ContentProvider that uses a custom CursorFacory in debug to print out the SQL queries (for debugging).
A certain query was returning 0 rows, while I knew there were rows that should have been included. So I copied the query from my logs, replaced the bind values and ran it in sqlite3 shell on the device and got the correct result.
The Query Code
cr.query (contentUri,
Projection.columns,
FeedColumns.FEED_TYPE + "=? AND " +
FeedColumns.SUB_TYPE + "=? AND " +
ProfileUpdateFeedItem.UPDATED_FIELD + "=? AND " +
FeedColumns.IS_NOTIFIED + "=?",
new String[] {FeedType.USER, // 2
WallPostData.WallPostType.PROFILE_UPDATE, // 1
ProfileUpdateData.ProfileField.STATUS, // 0
SQLBoolean.FALSE // 0
},
FeedColumns.CREATED + " ASC");
From the logs:
07-04 12:48:51.339 4067-4314/com.redacted.android D/DATABASE﹕ QUERY: SQLiteQuery: SELECT DISTINCT id, sender, data_1, data_2, photo, feed_type, sub_type, created, expiry, updated, comment_count, comment_unread, reaction_count, reaction_unread, sender_name, sender_photo, _id FROM wall WHERE feed_type=? AND sub_type=? AND data_1=? AND is_notified=? ORDER BY created ASC LIMIT 100
On device:
Enter SQL statements terminated with a ";"
sqlite> SELECT DISTINCT id, sender, data_1, data_2, photo, feed_type, sub_type, created, expiry, updated, comment_count, comment_unread, reaction_count, reaction_unread, sender_name, sender_photo, _id FROM wall WHERE feed_type=2 AND sub_type=1 AND data_1=0 AND is_notified=0 ORDER BY created ASC LIMIT 100;
53b702b827d7482062f52b03|a7e759d78abe4bfa97045ce49a24ab57|0|Educ||2|1|1404502712279|1404761912325|1404502712279|||||Luke Skywalker|pr/e5c2c0398b267f93683c80dc5009722e|49
The ContentProvider, however, doesn't agree and cursor.getCount() returns 0.
Any ideas why this is happening?
feed_type, sub_type, and is_notified are INTEGER columns.
data_1 is a BLOB that is storing an integer for any row that would qualify for this query, but stores strings for other types of data that could go in this table.
When you run in the shell i'm surprised you get any rows. The blob data type may not convert the keyed value properly for you. Typically the database API requires a special function to set the blob value as well as retrieve it.
So the problem here was the BLOB column. It was being evaluated properly in queries (The data in the table is used in a ListView and is displayed differently depending on the contents of the data_1 and data_2 columns).
Everything in the feed category gets parsed into a member of a class hierarchy rooted at an AnstractFeedObject.
Most fields that use both data_1 and data_2 store text in both, but some fields (those who correspond to a subset of the mentioned class hierarchy) use data_1 as a type enumeration that the UI uses to interpret the value stored in data_2. For example, a 0 type means that data_2 is a picture id (construct the url and download), while type 1 means it's actual text content.
What I ended up doing was that I replaced data_1 with an integer column called type_enumeration and renamed data_2 to data_1. Now that I know BLOB can cause those kinds of issues, I'll be changin data_2 also to a TEXT column.
If at some point in the future I need to store binary data in the DB, I'll add a bin_data to the column.
Now usually in a proper normalized schema you'd use linked tables to represent such hierarchy, but in a mobile environment, you want to minimize joins so a few extra columns are cheaper in terms of performance (at least that's been my experience).
In android, SQLiteDatabase has a update function
update(String table, ContentValues values, String whereClause, String[] whereArgs)
new values in put in values
If I want to update a column A by adding one to it, how should I prepare the ContentValues values variable? I don't think the following would work.
cv.put("A", "A" + 1);
I can sure run execSQL with raw sql, but it does not return num of row updated
If you'd execute a raw query, something like this should work to increment the current value in the column:
UPDATE table_name SET column_a = column_a + 1 WHERE _id = 1
(where 1 is just an example to illustrate how to apply it to a specific row)
The same probably wouldn't work with ContentValues, since (as the name indicates) it takes the values to set the column to. That means it needs to have been evaluated before building the ContentValues, whereas with a raw query the value isn't evaluated until the query actually runs on the database.
You can of course retrieve the current value first and then increment that accordingly when issuing an update; that requires a select query first. Quite commonly though, you're working with objects in Java, where the column value for a row is bound up to a member field of the object. If you've got a setup like that, then you probably already have the current value at the moment you want to run an update query.
As such, it would just look somewhat like:
SomeObject object = ...;
cv.put("column_a", object.getSomeValue() + 1);
(where I'm assuming object.getSomeValue() will return an int)
// edit: here's some more examples for the raw query approach:
SQLite - increase value by a certain number
// edit2: You've edited your original question and added:
I can sure run execSQL with raw sql, but it does not return num of
row updated
If knowing how many rows the query changed is a must, then you can potentially leverage the changes() function. It still means you're going to have to run a second query though.
SELECT changes() FROM table_name
The docs say:
The changes() function returns the number of database rows that were
changed or inserted or deleted by the most recently completed INSERT,
DELETE, or UPDATE statement, exclusive of statements in lower-level
triggers. The changes() SQL function is a wrapper around the
sqlite3_changes() C/C++ function and hence follows the same rules for
counting changes.
Alternatively, you could look into the rawQuery() method that takes an SQL statement and returns the result as a Cursor. Not sure if it that even works for an update query, or whether the result would be anything sensible, but if you're really lucky, you may find that Cursor.getCount() gives you the number of affected rows.
To expand upon #MH's solution, there actually is a way to do a raw update AND get back the number of rows updated (because I'm doing the same thing in one of my projects). You have to use a compiled SQLiteStatement and then call the method executeUpdateDelete(). From the documentation:
public int executeUpdateDelete ()
Execute this SQL statement, if the the number of rows affected by execution of this SQL statement is of any importance to the caller - for example, UPDATE / DELETE SQL statements.
Returns
the number of rows affected by this SQL statement execution.
See the following sample code where I add a new column to my table and then update each column similarly to how you were asking:
db.beginTransaction();
try {
db.execSQL("ALTER TABLE " + TABLE_NAME + " ADD COLUMN "
+ COLUMN_NAME_LOCALTIME + " INTEGER");
String stmtString = "UPDATE " + TABLE_NAME + " SET "
+ COLUMN_NAME_LOCALTIME + "="
+ COLUMN_NAME_TIME + "+ (" + tzOffset + ")";
SQLiteStatement sqlStmt = db.compileStatement(stmtString);
int rows = sqlStmt.executeUpdateDelete();
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
I'm using a transaction here because in case I add the new column but CANNOT update the values, I want everything to rollback so I can attempt something else.
I'm using SQLite on Android using SQLiteDatabase (http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html)
I am developing a bible application, which has a single table with the following columns:
book : int
chapter : int
verse : int
wordIdx : int
strongId : string
word : string
each sentence is broken down in to a series of strongId/word pairs, so wordIdx is used to order the words, strongId is simply a index in to a concordance, and word is the word in the sentence.
so I have 300,000 rows
the bottleneck appears to be my query to get a list of words for each verse:
My SQL is effectively this:
SELECT strongId, word FROM ? WHERE book=? AND chapter=? AND verse=?
Here is the code:
Cursor cursor = mBible.database().rawQuery("SELECT " + KEY_STRONGID + "," + KEY_WORD + " FROM " + tableName() + " WHERE " + KEY_BOOK + "=? AND " + KEY_CHAPTER + "=? AND " + KEY_VERSE + "=?" , new String[] { String.valueOf(mChapter.mBook.index()), String.valueOf(mChapter.index()), String.valueOf(verse) });
cursor.moveToFirst();
mWordList = new ArrayList<Word>();
do {
mWordList.add(new Word(cursor.getString(1), cursor.getString(0)));
} while (cursor.moveToNext());
Now, I've tried putting each chapter in to its own temporary view (using CREATE TEMP VIEW) which cuts down the records to about 400 in my example how ever it is still taking far to long to query
Its taking of the order of 30 seconds to generate the text for two chapters to display to the user (using a temporary view and without using a temporary view). It takes about 5 seconds if I set up a dummy list of words to avoid the database query.
How can I improve the performance of this? It seems as if a temp view is having no impact on performance as I had hoped.
A view does not change the performance of a query; it just saves the query itself, not the results of the query.
If you open your database with the sqlite3 command-line tool on your desktop machine, you can use the EXPLAIN QUERY PLAN command to check how efficient your queries are.
Without any indexes, you query always scans the entire table:
> sqlite3 bible.db
SQLite version 3.7.15.2 2013-01-09 11:53:05
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> EXPLAIN QUERY PLAN SELECT strongId, word FROM MyTable WHERE book=1 AND chapter=2 AND verse=3;
0|0|0|SCAN TABLE MyTable (~1000 rows)
With an index on your three lookup fields, SQLite can do a fast search in the index and needs to read only the matching records from the table:
sqlite> CREATE INDEX b_c_v ON MyTable(book, chapter, verse);
sqlite> EXPLAIN QUERY PLAN SELECT strongId, word FROM MyTable WHERE book=1 AND chapter=2 AND verse=3;
0|0|0|SEARCH TABLE MyTable USING INDEX b_c_v (book=? AND chapter=? AND verse=?) (~8 rows)
If you create a covering index (with all fields used in the query, lookup fields first), SQLite does not need to read from the table at all. However, this does not give a big speedup over a normal index, and might not be worth the additional storage cost:
sqlite> CREATE INDEX cov ON MyTable(book, chapter, verse, strongId, word);
sqlite> EXPLAIN QUERY PLAN SELECT strongId, word FROM MyTable WHERE book=1 AND chapter=2 AND verse=3;
0|0|0|SEARCH TABLE MyTable USING COVERING INDEX cov (book=? AND chapter=? AND verse=?) (~8 rows)
Please note that SQLite can use at most one index per table in a query, so it does not always make sense to create multiple indexes.
Use EXPLAIN QUERY PLAN to check which indexes are actually used, and whether you can create a few indexes to optimize most of your queries.
Also see the Query Planning documentation.
I ended up creating temporary tables and performance is now acceptable
I have a sqlite db that at the moment has few tables where the biggest one has over 10,000 rows. This table has four columns: id, term, definition, category. I have used a FTS3 module to speed up searching which helped a lot. However, now when I try to fetch 'next' or 'previous' row from table it takes longer than it was before I started using FTS3.
This is how I create virtual table:
CREATE VIRTUAL TABLE profanity USING fts3(_id integer primary key,name text,definition text,category text);
This is how I fetch next/previous rows:
SELECT * FROM dictionary WHERE _id < "+id + " ORDER BY _id DESC LIMIT 1
SELECT * FROM dictionary WHERE _id > "+id + " ORDER BY _id LIMIT 1
When I run these statements on the virtual table:
NEXT term is fetch within ~300ms,
PREVIOUS term is fetch within ~200ms
When I do it with normal table (the one created without FTS3):
NEXT term is fetch within ~3ms,
PREVIOUS term is fetch within ~2ms
Why there is such a big difference? Is there any way I can improve this speed?
EDITED:
I still can't get it to work!
Virtual table you've created is designed to provide full text queries. It's not aimed to fast processing standard queries using PK in where condition.
In this case there is no index on your _id column, so SQLite probably performs full table scan.
Next problem is your query - it's totally inefficient. Try something like this (untested):
SELECT * FROM dictionary WHERE _id = (select max(_id) from dictionary where _id < ?)
Next thing you can consider is redesign of your app. Instead of loading 1 row you, maybe you should get let's say 40, load them into memory and make background data loading when there is less than n to one of the ends. Long SQL operation will become invisible to user even if it'll last 3s instead of 0,3s
If you're running LIMIT 1 to begin with, you can remove the order by clause completely. This may help. I'm not familiar with FTS3, however.
You could also just flat out assign your id variable a ++ or -- and assert `WHERE _id = "+id+" LIMIT 1" which would make a single lookup instead of < or >.
Edit: and now that I look back at what I typed, if you do it that way, you can just remove LIMIT 1 completely, since your _id is your pk and must be unique.
hey look, a raw where clause!