I successfully used the following BEFORE INSERT trigger to limit the number of rows stored in the SQLite database table locations. The database table acts as a cache in an Android application.
CREATE TRIGGER 'trigger_locations_insert'
BEFORE INSERT ON 'locations'
WHEN ( SELECT count(*) FROM 'locations' ) > '100'
BEGIN
DELETE FROM 'locations' WHERE '_id' NOT IN
(
SELECT '_id' FROM 'locations' ORDER BY 'modified_at' DESC LIMIT '100'
);
END
Meanwhile, I added a second trigger that allows me to INSERT OR UPDATE rows. - The discussion on that topic can be found in another thread. The second trigger requires a VIEW on which each INSERTis executed.
CREATE VIEW 'locations_view' AS SELECT * FROM 'locations';
Since an INSERT is no longer executed on the TABLE locations but on the VIEW locations_view, the above trigger does no longer work. If I apply the trigger on the VIEW the following error message is thrown.
Failure 1 (cannot create BEFORE trigger on view: main.locations_view)
Question:
How can I change the above trigger to observe each INSERT on the VIEW - or do you recommend another way to limit the number of rows? I would prefer to handle this kind of operation within the database, rather then running frontend code on my client.
Performance issues:
Although, the limiter (the above trigger) works in general - it performs less then optimal! Actually, the database actions take so long that an ANR is raised. As far as I can see, the reason is, that the limiter is called every time an INSERT happens. To optimize the setup, the bulk INSERT should be wrapped into a transaction and the limiter should perform right after. Is this possible? If you like to help, please place optimization comments concerning the bulk INSERT into the original question. Comments regarding the limiter are welcome here.
This type of trigger should work fine in conjunction with the other one. The problem appears to be that the SQL is unnecessarily quoting the _id field. It is selecting the literal string "_id" for every row and comparing that to the same literal string.
Removing the quotes around '_id' (both in the DELETE and in the sub-SELECT) should fix the problem.
Related
I have an SQLite DB where I perform a query like
Select * from table where col_name NOT IN ('val1','val2')
Basically I'm getting a huge list of values from server and I need to select the ones which is not present in the list given.
Currently its working fine, No issues. But the number of values from server becomes huge as the server DB is getting updated frequently.
So, I may get thousands of String values which I need to pass to the NOT IN
My question is, Will it cause any perfomance issue in the future? Does the NOT IN parameter have any size restriction? (like max 10000 values you can check)?
Will it cause any crash at some point?
This is an official reference about various limitation in sqlite. I think the Maximum Length Of An SQL Statement may related to your case. Default value is 1000000, and it is adjustable.
Except this I don't think any limitation existed for numbers of parameter of NOT IN clause.
With more than a few values to test for, you're better off putting them in a table that has an index on the column holding them. Then things like
SELECT *
FROM table
WHERE col_name NOT IN (SELECT value_col FROM value_table);
or
SELECT *
FROM table AS t
WHERE NOT EXISTS (SELECT 1 FROM value_table WHERE value_col = t.col_name);
will be reasonably efficient no matter how many records are in value_table because that index will be used to find entries.
Plus, of course, it makes it a lot easier to re-use prepared statements because you don't have to create a new one and re-bind every value (You are using prepared statements with placeholders for these values, right, and not trying to put their contents inline into a string?) every time you add a value to the ones you need to check. You just insert it into value_table instead.
Yes, there is a limit of 999 arguments as reported in the official documentation: https://www.sqlite.org/limits.html#max_variable_number
I am currently building a database recording events on the phone, but as I don't want to make this a huge database, 100 events are more than enough.
This will keep my database light en efficient.
Unfortunately, I don't see a way to limit the number of rows other than
String sql = "DELETE FROM myTable WHERE _id <= "+limitId;
and I could run this code when the user launch/leaver the app, but I am expecting a better way to achieve this
Is there a more convenient way to achieve this?
If you are using a ContentProvider, you can implement your DELETE in onInsert, deleting a single row on every insert of a single row:
String sql = "DELETE FROM myTable WHERE _id IN (SELECT min(_id) FROM myTable)";
I guess you mean limit to the 100 newest events ?
if so, there is no better way to do it as you did: checking for entries on every insert and delete old entries if necessary.
It's just a matter of taste how or where you do your check, as flx mentioned you could do it in the ContentProvider or as you probably did in the BroadcastReceiver or Service where you actually add the new row. You could also set up a Trigger on your table, but the main idea remains the same. Here a link if you're interested in triggers:
http://www.tutorialspoint.com/sqlite/sqlite_triggers.htm
I've got two SQLite databases, each with a table that I need to keep synchronized by merging rows that have the same key. The tables are laid out like this:
CREATE TABLE titles ( name TEXT PRIMARY KEY,
chapter TEXT ,
page INTEGER DEFAULT 1 ,
updated INTEGER DEFAULT 0 );
I want to be able to run the same commands on each of the two tables, with the result that for pairs of rows with the same name, whichever row has the greater value in updated will overwrite the other row completely, and rows which do not have a match are copied across, so both tables are identical when finished.
This is for an Android app, so I could feasibly do the comparisons in Java, but I'd prefer an SQLite solution if possible. I'm not very experienced with SQL, so the more explanation you can give, the more it'll help.
EDIT
To clarify: I need something I can execute at an arbitrary time, to be invoked by other code. One of the two databases is not always present, and may not be completely intact when operations on the other occur, so I don't think a trigger will work.
Assuming that you have attached the other database to your main database:
ATTACH '/some/where/.../the/other/db-file' AS other;
you can first delete all records that are to be overwritten because their updated field is smaller than the corresponding updated field in the other table:
DELETE FROM main.titles
WHERE updated < (SELECT updated
FROM other.titles
WHERE other.titles.name = main.titles.name);
and then copy all newer and missing records:
INSERT INTO main.titles
SELECT * FROM other.titles
WHERE name NOT IN (SELECT name
FROM main.titles);
To update in the other direction, exchange the main/other database names.
For this, you can use a trigger.
i.e.
CREATE TRIGGER sync_trigger
AFTER INSERT OR UPDATE OF updated ON titles
REFERENCING NEW AS n
FOR EACH ROW
DECLARE updated_match;
DECLARE prime_name;
DECLARE max_updated;
BEGIN
SET prime_name = n.name;
ATTACH database2name AS db2;
SELECT updated
INTO updated_match
FROM db2.titles t
WHERE t.name=prime_name)
IF updated_match is not null THEN
IF n.updated > updated_match THEN
SET max_updated=n.updated;
ELSE
SET max_updated=updated_match;
END IF;
UPDATE titles
SET updated=max_updated
WHERE name=prime_name;
UPDATE db2.titles
SET updated=max_updated
WHERE name=prime_name;
END IF;
END sync_trigger;
The syntax may be a little off. I don't use triggers all that often and this is a fairly complex one, but it should give you an idea of where to start at least. You will need to assign this to one database, exchanging "database2name" for the other database's name and then assign it again to the other database, swapping the "database2name" out for the other database.
Hope this helps.
What is the best way to maintain a "cumulative sum" of a particular data column in SQLite? I have found several examples online, but I am not 100% certain how I might integrate these approaches into my ContentProvider.
In previous applications, I have tried to maintain cumulative data myself, updating the data each time I insert new data into the table. For example, in the sample code below, every time I would add a new record with a value score, I would then manually update the value of cumulative_score based on its value in the previous row.
_id score cumulative_score
1 100 100
2 50 150
3 25 175
4 25 200
5 10 210
However, this is far from ideal and becomes very messy when handling tables with many columns. Is there a way to somehow automate the process of updating cumulative data each time I insert/update records in my table? How might I integrate this into my ContentProvider implementation?
I know there must be a way to do this... I just don't know how. Thanks!
Probably the easiest way is with a SQLite trigger. That is the closest I know
of to "automation". Just have an insert trigger that takes the previous
cumulative sum, adds the current score and stores it in the new row's cumulative
sum. Something like this (assuming _id is the column you are ordering on):
CREATE TRIGGER calc_cumulative_score AFTER INSERT ON tablename FOR EACH ROW
BEGIN
UPDATE tablename SET cumulative_score =
(SELECT cumulative_score
FROM tablename
WHERE _id = (SELECT MAX(_id) FROM tablename))
+ new.score
WHERE _id = new._id;
END
Making sure that the trigger and the original insert are in the same
transaction. For arbitrary updates of the score column, you would have to
have to implement a recursive trigger that somehow finds the next highest id (maybe by selecting by the min id
in the set of rows with an id greater than the current one) and updates its
cumulative sum.
If you are opposed to using triggers, you can do more or less the same thing in
the ContentProvider in the insert and update methods manually, though since
you're pretty much locked into SQLite on Android, I don't see much reason not to
use triggers.
I assume you are wanting to do this as an optimization, as otherwise you could just calculate the sum on demand (O(n) vs O(1), so you'd have to consider how big n might get, and how often you need the sums).
I am fetching my data with id which is Integer primary key or integer.
But after deleting any row...
After that if we make select query to show all.
But it will give force close because one id is missing.
I want that id can itself take auto increment & decrement.
when i delete a record at the end(i.g. id=7) after this i add a row then id must be 7 not 8. as same when i delete a row in middle(i.g. id=3) then all the row auto specify by acceding.
your idea can help me.
Most systems with auto-incrementing columns keep track of the last value inserted (or the next one to be inserted) and do not ever reissue a number (give the same number twice), even if the last number issued has been removed from the table.
Judging from what you are asking, SQLite is another such system.
If there is any concurrency in the system, then this is risky, but for a single-user, single-app-at-a-time system, you might get away with:
SELECT MAX(id_column) + 1 FROM YourTable
to find the next available value. Depending on how SQLite behaves, you might be able to embed that in the VALUES list of an INSERT statement:
INSERT INTO YourTable(id_column, ...)
VALUES((SELECT MAX(id_column) + 1 FROM YourTable), ...);
That may not work; you may have to do this as two operations. Note that if there is any concurrency, the two statement form is a bad ideaTM. The primary key unique constraint normally prevents disaster, but one of two concurrent statements fails because it tries to insert a value that the other just inserted - so it has to retry and hope for the best. Clearly, a cell phone has less concurrency than, say, a web server so the problem is correspondingly less severe. But be careful.
On the whole, though, it is best to let gaps appear in the sequence without worrying about it. It is usually not necessary to worry about them. If you must worry about gaps, don't let people make them in the first place. Or move an existing row to fill in the gap when you do a delete that creates one. That still leaves deletes at the end creating gaps when new rows are added, which is why it is best to get over the "it must be a contiguous sequence of numbers" mentality. Auto-increment guarantees uniqueness; it does not guarantee contiguity.