My app reads an XML file on the internet, takes note of the time and creates/writes an SQLite database. The next time data is required, if the time is >24hrs the database is updated (xml downloaded again).
The problem is that whenever I relaunch the app in AVD it has to re-download and so I notice that all the data in the database is written again (duplicated). So instead of 10 items, I have 20 (10+10 duplicates). If I relaunch again I get another 10 items duplicated.
I thought about how I could prevent the duplication of the database (or delete the old entries), so I decided to increment the database version every time the content is downloaded. I thought this would trigger the onUpgrade() method so the data would be cleared but nothing changes.
Now I am clueless. How should I go about this?
On your database create you'll want to use the UNIQUE constraint. You may not want the ON CONFLICT REPLACE that i use, but you should get the idea.
For Ex:
private static final String DATABASE_CREATE_NEWS= "create table news (_id integer primary key autoincrement, "title text not null, description text not null, date text not null, LastModified text not null, UNIQUE(title, date) ON CONFLICT REPLACE);";
Here is another solid thread that talks about it as well.
SQLite table constraint - unique on multiple columns
Here is some more info on the android sqlite: http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html
You should create an index on the columns that represent a unique identifier.
see this article on SQLite's website.
CREATE INDEX ix_tblexample ON TableName ( Column1, Column2, Column3 [, Column4, etc..])
Or (as per your comment) you can select the table into a cursor and check for each one.
String sql = "select * from " + tableName + "where column1 = " + param1 + "and column2 = " + param2;
Cursor cur = _db.rawQuery( sql, new String[0] );
if(cur.getCount() == 0)
{
//upload
}
Related
I am trying to create a view using the following source code:
SQLiteDatabase db = mManagerDbHelper.getWritableDatabase();
String sql = "SELECT * FROM users WHERE name = ?";
String[] selectionArgs = new String[] {"Bob"};
db.execSQL("CREATE VIEW bob_user AS " + sql, selectionArgs);
However, this code always returns this error:
android.database.sqlite.SQLiteException: parameters are not allowed in views (code 1)
How do I use the bindArgs parameter of execSQL(String, Object[]) method?
How do I use the bindArgs parameter of execSQL(String, Object[]) method?
In short you can't, a VIEW is intended to be stored and thus cannot be dynamically changed and hence why you can't bind parameters.
You could however get around it by using an expression for the right hand expression of the WHERE clause and if that expression were to select the value from a table then the value could be changed in that table (i.e. mimicbind in the following) e.g.
DROP VIEW IF EXISTS a_user;
DROP TABLE IF EXISTS users;
DROP TABLE IF EXISTS mimicbind;
CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT);
INSERT INTO users (name) VALUES('bob'),('mary'),('fred'),('sue');
CREATE TABLE IF NOT EXISTS mimicbind (mimicname TEXT PRIMARY KEY, value TEXT);
INSERT INTO mimicbind VALUES('a_user','bob');
CREATE VIEW a_user AS SELECT * FROM users WHERE name = (SELECT value FROM mimicbind WHERE mimicname = 'a_user');
SELECT * FROM a_user;
UPDATE mimicbind SET value = 'sue' WHERE mimicname = 'a_user';
SELECT * FROM a_user;
DROP VIEW IF EXISTS a_user;
DROP TABLE IF EXISTS users;
DROP TABLE IF EXISTS mimicbind;
The result from the queries being:-
and then after the value column in the mimicbind table is changed from bob to sue then:-
However, you may wish to consider what benefit there is to using a view as it would appear that for what you want it is an unnecessary complexity, when a straight forward select would likely be as, if not more efficient. That is a VIEW will, I believe. take up a minimum of 4k per VIEW so bob, alice, etc .... and that's going to be storage space largely wasted.
Even the solution via another table is probably not going to afford any benefit over just using a query. The end result of both would be a Cursor accessing exactly the same core data *(of course in the solution then mimicbind table is additionally accessed)(
I never found a solution here in stackoverflow. So I modified my sql statement and ended up with the following code:
SQLiteDatabase db = mManagerDbHelper.getWritableDatabase();
String sql = "SELECT * FROM users WHERE name = 'Bob'";
db.execSQL("CREATE VIEW bob_user AS " + sql);
Note that I used execSQL(String).
I am using an SQLite database that holds different messaging conversations in a thread id. To get the main conversation list I use the following code:
database.query(true, MessagesHelper.CONVERSATION_TABLE, inboxCols, null, null, MessagesHelper.THREAD_ID, null, "${MessagesHelper.DATE} DESC", null)
The issue is that I provide a function to load in older messages that are not showing, by which I use:
database.insert(MessagesHelper.CONVERSATION_TABLE, null, values)
The issue is that after inserting messages, which are older than the most recent one that the distinct list used to call, it now shows the older added messages in the inbox list, which makes the list all out of order and causes a lot of confusion.
Here is the create statement:
private const val CONVO_CREATE = "CREATE TABLE $CONVERSATION_TABLE($NAME VARCHAR(255), $THREAD_ID VARCHAR(255), $MESSAGE_ID VARCHAR(40), $ADDRESS VARCHAR(14), $BODY VARCHAR(500), $SUBJECT VARCHAR(100), $MMS VARCHAR(255), $MESSAGE_TYPE VARCHAR(7), $MMS_TYPE VARCHAR(20), $CONVERSATION_TYPE VARCHAR(20), $GROUP_ADDRESS VARCHAR(255), $GROUP_NAME VARCHAR(255), $READ VARCHAR(10), $WHO VARCHAR(3), $COUNT INTEGER, $DATE INTEGER);"
Inbox columns are:
val inboxCols = arrayOf(MessagesHelper.COUNT, MessagesHelper.GROUP_ADDRESS, MessagesHelper.NAME, MessagesHelper.ADDRESS, MessagesHelper.READ, MessagesHelper.GROUP_NAME, MessagesHelper.BODY, MessagesHelper.DATE, MessagesHelper.THREAD_ID, MessagesHelper.CONVERSATION_TYPE, MessagesHelper.MESSAGE_ID, MessagesHelper.WHO)
Is there any way to insert into the database but have the distinct query still sort by each thread_id by date as well?
Thanks!
I believe the issue is that DISTINCT considers the entire row being extracted, thus it is likely, based upon the column names, that both the COUNT column and the DATE column would or could likely be different when inserting a new message (e.g. perhaps count would initially be 0?) and thus cause them to be inserted as they make a new row DISTINCT (not a duplicate).
e.g. Consider this table :-
Then if DISTINCT is used just on the idbet column, the result is 2 rows 1 for where idbet is 3000 and another for where idbet is 1981 as per :-
However, if columns idbet and gamble are extracted then all 4 columns are extracted as there are now no duplicates, as per :-
If the row with _id 350 had WIN in the gamble column then 3 rows would be extracted as rows with _id's 349 and 350 would be a duplicate according to the idbet and gamble columns, as per :-
Perhaps rather than DISTINCT, or just DISTINCT you should use a WHERE condition or conditions (4th and 5th parameters of query). Perhaps "count < 1" as the 4th parameter (5th null) this does assume that count will initially be 0. An alternative would be to reduce the columns extracted but that may not be practical.
I have a ContentProvider that uses a custom CursorFacory in debug to print out the SQL queries (for debugging).
A certain query was returning 0 rows, while I knew there were rows that should have been included. So I copied the query from my logs, replaced the bind values and ran it in sqlite3 shell on the device and got the correct result.
The Query Code
cr.query (contentUri,
Projection.columns,
FeedColumns.FEED_TYPE + "=? AND " +
FeedColumns.SUB_TYPE + "=? AND " +
ProfileUpdateFeedItem.UPDATED_FIELD + "=? AND " +
FeedColumns.IS_NOTIFIED + "=?",
new String[] {FeedType.USER, // 2
WallPostData.WallPostType.PROFILE_UPDATE, // 1
ProfileUpdateData.ProfileField.STATUS, // 0
SQLBoolean.FALSE // 0
},
FeedColumns.CREATED + " ASC");
From the logs:
07-04 12:48:51.339 4067-4314/com.redacted.android D/DATABASE﹕ QUERY: SQLiteQuery: SELECT DISTINCT id, sender, data_1, data_2, photo, feed_type, sub_type, created, expiry, updated, comment_count, comment_unread, reaction_count, reaction_unread, sender_name, sender_photo, _id FROM wall WHERE feed_type=? AND sub_type=? AND data_1=? AND is_notified=? ORDER BY created ASC LIMIT 100
On device:
Enter SQL statements terminated with a ";"
sqlite> SELECT DISTINCT id, sender, data_1, data_2, photo, feed_type, sub_type, created, expiry, updated, comment_count, comment_unread, reaction_count, reaction_unread, sender_name, sender_photo, _id FROM wall WHERE feed_type=2 AND sub_type=1 AND data_1=0 AND is_notified=0 ORDER BY created ASC LIMIT 100;
53b702b827d7482062f52b03|a7e759d78abe4bfa97045ce49a24ab57|0|Educ||2|1|1404502712279|1404761912325|1404502712279|||||Luke Skywalker|pr/e5c2c0398b267f93683c80dc5009722e|49
The ContentProvider, however, doesn't agree and cursor.getCount() returns 0.
Any ideas why this is happening?
feed_type, sub_type, and is_notified are INTEGER columns.
data_1 is a BLOB that is storing an integer for any row that would qualify for this query, but stores strings for other types of data that could go in this table.
When you run in the shell i'm surprised you get any rows. The blob data type may not convert the keyed value properly for you. Typically the database API requires a special function to set the blob value as well as retrieve it.
So the problem here was the BLOB column. It was being evaluated properly in queries (The data in the table is used in a ListView and is displayed differently depending on the contents of the data_1 and data_2 columns).
Everything in the feed category gets parsed into a member of a class hierarchy rooted at an AnstractFeedObject.
Most fields that use both data_1 and data_2 store text in both, but some fields (those who correspond to a subset of the mentioned class hierarchy) use data_1 as a type enumeration that the UI uses to interpret the value stored in data_2. For example, a 0 type means that data_2 is a picture id (construct the url and download), while type 1 means it's actual text content.
What I ended up doing was that I replaced data_1 with an integer column called type_enumeration and renamed data_2 to data_1. Now that I know BLOB can cause those kinds of issues, I'll be changin data_2 also to a TEXT column.
If at some point in the future I need to store binary data in the DB, I'll add a bin_data to the column.
Now usually in a proper normalized schema you'd use linked tables to represent such hierarchy, but in a mobile environment, you want to minimize joins so a few extra columns are cheaper in terms of performance (at least that's been my experience).
In android, SQLiteDatabase has a update function
update(String table, ContentValues values, String whereClause, String[] whereArgs)
new values in put in values
If I want to update a column A by adding one to it, how should I prepare the ContentValues values variable? I don't think the following would work.
cv.put("A", "A" + 1);
I can sure run execSQL with raw sql, but it does not return num of row updated
If you'd execute a raw query, something like this should work to increment the current value in the column:
UPDATE table_name SET column_a = column_a + 1 WHERE _id = 1
(where 1 is just an example to illustrate how to apply it to a specific row)
The same probably wouldn't work with ContentValues, since (as the name indicates) it takes the values to set the column to. That means it needs to have been evaluated before building the ContentValues, whereas with a raw query the value isn't evaluated until the query actually runs on the database.
You can of course retrieve the current value first and then increment that accordingly when issuing an update; that requires a select query first. Quite commonly though, you're working with objects in Java, where the column value for a row is bound up to a member field of the object. If you've got a setup like that, then you probably already have the current value at the moment you want to run an update query.
As such, it would just look somewhat like:
SomeObject object = ...;
cv.put("column_a", object.getSomeValue() + 1);
(where I'm assuming object.getSomeValue() will return an int)
// edit: here's some more examples for the raw query approach:
SQLite - increase value by a certain number
// edit2: You've edited your original question and added:
I can sure run execSQL with raw sql, but it does not return num of
row updated
If knowing how many rows the query changed is a must, then you can potentially leverage the changes() function. It still means you're going to have to run a second query though.
SELECT changes() FROM table_name
The docs say:
The changes() function returns the number of database rows that were
changed or inserted or deleted by the most recently completed INSERT,
DELETE, or UPDATE statement, exclusive of statements in lower-level
triggers. The changes() SQL function is a wrapper around the
sqlite3_changes() C/C++ function and hence follows the same rules for
counting changes.
Alternatively, you could look into the rawQuery() method that takes an SQL statement and returns the result as a Cursor. Not sure if it that even works for an update query, or whether the result would be anything sensible, but if you're really lucky, you may find that Cursor.getCount() gives you the number of affected rows.
To expand upon #MH's solution, there actually is a way to do a raw update AND get back the number of rows updated (because I'm doing the same thing in one of my projects). You have to use a compiled SQLiteStatement and then call the method executeUpdateDelete(). From the documentation:
public int executeUpdateDelete ()
Execute this SQL statement, if the the number of rows affected by execution of this SQL statement is of any importance to the caller - for example, UPDATE / DELETE SQL statements.
Returns
the number of rows affected by this SQL statement execution.
See the following sample code where I add a new column to my table and then update each column similarly to how you were asking:
db.beginTransaction();
try {
db.execSQL("ALTER TABLE " + TABLE_NAME + " ADD COLUMN "
+ COLUMN_NAME_LOCALTIME + " INTEGER");
String stmtString = "UPDATE " + TABLE_NAME + " SET "
+ COLUMN_NAME_LOCALTIME + "="
+ COLUMN_NAME_TIME + "+ (" + tzOffset + ")";
SQLiteStatement sqlStmt = db.compileStatement(stmtString);
int rows = sqlStmt.executeUpdateDelete();
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
I'm using a transaction here because in case I add the new column but CANNOT update the values, I want everything to rollback so I can attempt something else.
I'm using SQLite on Android using SQLiteDatabase (http://developer.android.com/reference/android/database/sqlite/SQLiteDatabase.html)
I am developing a bible application, which has a single table with the following columns:
book : int
chapter : int
verse : int
wordIdx : int
strongId : string
word : string
each sentence is broken down in to a series of strongId/word pairs, so wordIdx is used to order the words, strongId is simply a index in to a concordance, and word is the word in the sentence.
so I have 300,000 rows
the bottleneck appears to be my query to get a list of words for each verse:
My SQL is effectively this:
SELECT strongId, word FROM ? WHERE book=? AND chapter=? AND verse=?
Here is the code:
Cursor cursor = mBible.database().rawQuery("SELECT " + KEY_STRONGID + "," + KEY_WORD + " FROM " + tableName() + " WHERE " + KEY_BOOK + "=? AND " + KEY_CHAPTER + "=? AND " + KEY_VERSE + "=?" , new String[] { String.valueOf(mChapter.mBook.index()), String.valueOf(mChapter.index()), String.valueOf(verse) });
cursor.moveToFirst();
mWordList = new ArrayList<Word>();
do {
mWordList.add(new Word(cursor.getString(1), cursor.getString(0)));
} while (cursor.moveToNext());
Now, I've tried putting each chapter in to its own temporary view (using CREATE TEMP VIEW) which cuts down the records to about 400 in my example how ever it is still taking far to long to query
Its taking of the order of 30 seconds to generate the text for two chapters to display to the user (using a temporary view and without using a temporary view). It takes about 5 seconds if I set up a dummy list of words to avoid the database query.
How can I improve the performance of this? It seems as if a temp view is having no impact on performance as I had hoped.
A view does not change the performance of a query; it just saves the query itself, not the results of the query.
If you open your database with the sqlite3 command-line tool on your desktop machine, you can use the EXPLAIN QUERY PLAN command to check how efficient your queries are.
Without any indexes, you query always scans the entire table:
> sqlite3 bible.db
SQLite version 3.7.15.2 2013-01-09 11:53:05
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> EXPLAIN QUERY PLAN SELECT strongId, word FROM MyTable WHERE book=1 AND chapter=2 AND verse=3;
0|0|0|SCAN TABLE MyTable (~1000 rows)
With an index on your three lookup fields, SQLite can do a fast search in the index and needs to read only the matching records from the table:
sqlite> CREATE INDEX b_c_v ON MyTable(book, chapter, verse);
sqlite> EXPLAIN QUERY PLAN SELECT strongId, word FROM MyTable WHERE book=1 AND chapter=2 AND verse=3;
0|0|0|SEARCH TABLE MyTable USING INDEX b_c_v (book=? AND chapter=? AND verse=?) (~8 rows)
If you create a covering index (with all fields used in the query, lookup fields first), SQLite does not need to read from the table at all. However, this does not give a big speedup over a normal index, and might not be worth the additional storage cost:
sqlite> CREATE INDEX cov ON MyTable(book, chapter, verse, strongId, word);
sqlite> EXPLAIN QUERY PLAN SELECT strongId, word FROM MyTable WHERE book=1 AND chapter=2 AND verse=3;
0|0|0|SEARCH TABLE MyTable USING COVERING INDEX cov (book=? AND chapter=? AND verse=?) (~8 rows)
Please note that SQLite can use at most one index per table in a query, so it does not always make sense to create multiple indexes.
Use EXPLAIN QUERY PLAN to check which indexes are actually used, and whether you can create a few indexes to optimize most of your queries.
Also see the Query Planning documentation.
I ended up creating temporary tables and performance is now acceptable