Develop Reference

Query that brings all the parents whose children meet a certain criteria

I need a query that brings all the parents whose children meet a certain criteria.
So far this is giving me something different than expected:
#Transaction
#Query("SELECT * FROM piece_unit_table put WHERE (SELECT et.date_of_creation FROM expense_table et WHERE et.piece_unit_parent_id = put.piece_unit_id) = :dateOfExpenseCreation AND project_id = :projectId")
public abstract LiveData<List<PieceUnit>> getPieceUnitsWhereDateOfExpenseCreation(long dateOfExpenseCreation, int projectId);
A more readable version of the same code above:
SELECT *
FROM piece_unit_table put
WHERE (
SELECT et.date_of_creation
FROM expense_table et
WHERE et.piece_unit_parent_id = put.piece_unit_id
) = :dateOfExpenseCreation
AND project_id = :projectId
At first I thought I got a jackpot with that one because it gave me responses on the first try (I usually spend a whole day just thinking about the structure of the query, I hate them), I never gave it too much thought, but the days passed by and it stopped giving me responses, so I'm guessing that maybe, the query was comparing some other dates inside the WHERE clause, more precisely, the dates on which the pieces where created.
Now that I reread it again it looks like a pretty bad query...
The way in which I'm storing dates (from the children side):
public void insertExpense(Expense expense) {
long now = Converters.dateToTimestamp(LocalDate.now());
expense .setDate_of_creation(now);
long edited = System.currentTimeMillis();
expense .setLast_edited(edited);
Log.println(Log.WARN, TAG, "insertExpense: date of creation is: " + expense.getDate_of_creation());
Log.println(Log.WARN, TAG, "insertExpense: expense project is: " + expense.getParent_project_id());
Log.println(Log.WARN, TAG, "insertExpense: expense piece parent is: " + expense.getPiece_unit_parent_id());
insert(expense);
}
The observer side:
Log.println(Log.ERROR, TAG, "getPieceUnitGroupedByExpenseProductsAtDateAndProject: specific date is: " + specificDate);
Log.d(TAG, "getPieceUnitGroupedByExpenseProductsAtDateAndProject: project id is: " + project);
MyObserver
.observeOnce(
pieceUnitRepository.getPieceUnitsWhereDateOfExpenseCreation(
specificDate,
project
),
pieceUnits ->
{
Log.d(TAG, "getPieceUnitGroupedByExpenseProductsAtDateAndProject: piece units are: " + pieceUnits);
if (pieceUnits != null && pieceUnits.size() > 0) {
PieceUnit p = pieceUnits.get(0);
Log.println(Log.WARN, TAG, "getPieceUnitGroupedByExpenseProductsAtDateAndProject: Hello there ;) : " + p.getBeginning_date());
}
The Logd's:
insertExpense: date of creation is: 18469
insertExpense: expense project is: 1
insertExpense: expense piece parent is: 4
getPieceUnitGroupedByExpenseProductsAtDateAndProject: specific date is: 18469
getPieceUnitGroupedByExpenseProductsAtDateAndProject: project id is: 1
getPieceUnitGroupedByExpenseProductsAtDateAndProject: piece units are: []
I'm not asking for a full solution, but any input could point me in the right direction, so thanks in advance.

Final answer (from second update):
SELECT
put.*
FROM piece_unit_table put
WHERE
EXISTS(
SELECT
1
FROM
expense_table et
WHERE
et.date_of_creation = :dateOfExpenseCreation AND et.piece_unit_parent_id = piece_unit_id
) AND
put.project_id = :projectId
EDITS
First answer :
In SQL there seems to be a function that helps on these cases:
all or not exist
https://stackoverflow.com/a/42439405/11214643
In the SQLite case there doesn't seem to be a direct equivalence, and every solution seems to be a case by case workaround depending on the type of clause.
In my case I LEFT JOIN-ed the children table, but group by-ing it before the join to avoid repeated rows on the left table.
As is expected, it was during the child table sub-query, that I filtered the table by their date condition
SELECT put.*
FROM piece_unit_table put
LEFT JOIN(
SELECT
piece_unit_parent_id
FROM
expense_table et
WHERE
et.date_of_creation = :dateOfExpenseCreation
GROUP BY
et.piece_unit_parent_id
) et1
ON
et1.piece_unit_parent_id = put.piece_unit_id
WHERE
put.project_id = :projectId
UPDATE:
It seems that behind curtains(either be SQLite or the ROOM interface), the LEFT JOIN is happening before the sub-query executes its own WHERE clause, as a result, the query is giving me parents, even when there are no children that meet the criteria, so the solution was to bring an additional column from the child temp table to the 'front', and check for it's non null-ability.
SELECT
put.*,
et1.puId AS expPuId
FROM piece_unit_table put
LEFT JOIN(
SELECT
piece_unit_parent_id AS puId
FROM
expense_table et
WHERE
et.date_of_creation = :dateOfExpenseCreation
GROUP BY
et.piece_unit_parent_id
) et1
ON
et1.puId = put.piece_unit_id
WHERE
put.project_id = :projectId AND expPuId NOT NULL
UPDATE 2:
Thanks to #forpas
The EXISTS and NOT EXISTS operators are in fact supported by SQLite, also, what was bringing me parents even when no children met the criteria was the second WHERE clause that was applied directly to the left table, but even if there was no clause applied to this table, it would've still gave me answers, because that's how LEFT JOIN works, if there are no matches it fills them with NULL's.
Here is a query that has the expected result, but does it better.
SELECT
put.*
FROM piece_unit_table put
WHERE
EXISTS(
SELECT
1
FROM
expense_table et
WHERE
et.date_of_creation = :dateOfExpenseCreation AND et.piece_unit_parent_id = piece_unit_id
) AND
put.project_id = :projectId

Missing records due to skipped frames

The app im working with is getting data from a .csv (20k-30k records) from a server and it needs to persist the data into an SQLiteDatabase.
It works but some records are missing and appeared that it have been skipped.
I/Choreographer( 2555): Skipped 46 frames! The application may be doing too much work on its main thread.
I know that this error says that the memory consumption is very high due to heavy load. Is there a more efficient way in persisting data in SQLiteDatabase rather than the classic accessing of CSV and processing it from there?
Code for writing in DB
String sql = "INSERT INTO " + tableName
+ " VALUES (?,?,?,?,?,?);";
SQLiteDatabase db = openHelper.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
try {
db.beginTransaction();
String[] sa = null;
for (final String csvline : arrCSV) {
statement.clearBindings();
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
}
statement.execute();
}
db.setTransactionSuccessful();
Log.d("Transaction", "Successful");
}catch(Exception e){
e.printStackTrace();
}finally {
statement.releaseReference();
statement.close();
db.endTransaction();
db.releaseMemory();
}
UPDATE
The missing records were not loaded in the Collection.
Is the skipping of frames the culprit here?
The loading in the collection is just a simple parsing of a csv file and
non replicable at times so Im assuming it is due to the skipping of frames.

I believe the issue is not linked to skipping frames and < 100 frames is considered a small/insignificant number. At least according to The application may be doing too much work on its main thread.
I frequently see it and has never been the cause of any issues. I've even seen it basically doing nothing other than returning a result from an activity to an activity that just starts the second activity.
As you have commented, the number of elements that result from the split is on occasion not 6. The issue is likely that the insert is not happening on such an occasion, perhaps due to constraints (without seeing how the columns are defined only guesses could be made).
However, you appear to consider that each line in csvline should be split into 6 elements. You should thus investigate as to why not?
To investigate I'd suggest getting details of the original data before the split and the resultant data after the split whenever the number of elements created by the split is not 6. e.g. by changing :-
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
}
statement.execute();
to
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
} else {
Log.d("NOT6SPLIT","CSVLINE WAS ===>" + csvline + "<===");
Log.d("NOT6SPLIT","CSVLINE WAS SPLIT INTO " + Integer.toString(sa.length) + " ELEMENTS :-");
for(String s: sa) {
Log.d("NOT6SPLIT","\tElement Value ===>" + s + "<===");
}
}
statement.execute();
Changing statement.execute() to :-
if (statement.executeInsert() < 1) {
Log.d("INSERTFAIL","Couldn't insert where CSVLINE was ===>" + csvline + "<===");
}
May also assist ('executeInsert' returns the rowid of the inserted record, else -1, not sure of the consequences of a table defined with WITHOUT ROWID).
It wouldn't surprise me at all if the issue boils down to your data containing characters that split considers special or metacharaceters:-
there are 12
characters with
special meanings:
the backslash \,
the caret ^,
the dollar sign $,
the period or dot .,
the vertical bar or pipe symbol |,
the question mark ?,
the asterisk or star *,
the plus sign +,
the opening parenthesis (,
the closing parenthesis ),
the opening square bracket [,
and the opening curly brace {,
These special characters are often called "metacharacters". Most
of them are errors when used alone.

Ormlite Join 3 Tables (Many-to-Many)

I'm trying to create search interface via ORMLite in Android. There are 3 tables created, that are making many-to-many relationship. (Recipe >> RecipesForTag << Tag)
tagQueryBuilder.where()
.like(Tag.COLUMN_NAME, "%" + text + "%");
recipesForTagQueryBuilder
.joinOr(tagQueryBuilder);
recipeQueryBuilder
.joinOr(recipesForTagQueryBuilder)
...other joins
.where()
.like(Recipe.COLUMN_TITLE, "%" + text + "%");
PreparedQuery<Recipe> preparedQuery = recipeQueryBuilder.prepare();
return recipeDao.query(preparedQuery);
When I'm not using line with .joinOr(recipesForTagQueryBuilder) everything is working fine. So what am I doing wrong?

OK my bad.
The only thing that was bad was using joinOr while I needed leftJoinOr... The question has one main problem: when middle table is empty no result will be find (also when ...other joins is nonempty). So result is simple:
recipesForTagQueryBuilder
.leftJoinOr(tagQueryBuilder);
recipeQueryBuilder
.leftJoinOr(recipesForTagQueryBuilder)

No such a column: R1.ReadingDate Error

I have created a view in sqlite using sqlite manager.
Create View vMeterReading
as
SELECT M._id as Meter_id, M.MeterNumber, R1.ReadingDate, R1.Reading AS CurrentReading, R2.ReadingDate AS PrevReadingDate, R2.Reading AS PrevMeterReading, R2.Rate as Rate, R2._id,R1.TenantMeter_id
FROM (Meters AS M INNER JOIN TenantMeters ON M._id = TenantMeters.Meter_id) INNER JOIN (MeterReading AS R1 INNER JOIN MeterReading AS R2 ON R1.TenantMeter_id = R2.TenantMeter_id) ON TenantMeters._id = R1.TenantMeter_id
WHERE (((R2.ReadingDate)=(SELECT Max(R3.ReadingDate)
FROM [MeterReading] AS R3
WHERE (R3.TenantMeter_id = R1.TenantMeter_id)
AND (R3.ReadingDate < R1.ReadingDate)
))) OR (((R2.TenantMeter_id) Is Null))
ORDER BY M.MeterNumber, R1.ReadingDate.
I have tried to use this view (select * from vMeterReading) in android and I get the error as titled.
I have also connected to the sqlite db from command prompt and tried to execute the same query. I get the same error.
What is wrong with the sql statement?
It works well in sql manager(firefox add-on).

SQLite makes no guarantees whether a table name prefix (like R1.1) is part of the result column name, or not.
You should explicitly give a name to all the view's columns with AS.

The problem was that the SQLite version used by android is not the same as the version used by sQLite manager. The android version does not support the join i was using.
The query below executes with out problems.
Got this advice from this site http://sqlite.1065341.n5.nabble.com/
SELECT M._id as Meter_id, M.MeterNumber, R1.ReadingDate as ReadingDate,
R1.Reading AS CurrentReading, R2.ReadingDate AS PrevReadingDate, R2.Reading
AS PrevMeterReading, R2.Rate as Rate, R2._id as _id,R1.TenantMeter_id as TenantMeterID
FROM Meters AS M INNER JOIN TenantMeters ON M._id = TenantMeters.Meter_id
INNER JOIN MeterReading AS R1 ON R1.TenantMeter_id = TenantMeters ._id
INNER JOIN MeterReading AS R2 ON
R1.TenantMeter_id = R2.TenantMeter_id
WHERE (((R2.ReadingDate)=(SELECT Max(R3.ReadingDate)
FROM [MeterReading] AS R3
WHERE (R3.TenantMeter_id = R1.TenantMeter_id)
AND (R3.ReadingDate < R1.ReadingDate)
))) OR (((R2.TenantMeter_id) Is Null))

select query with join is selecting data slowly from database in android

i have select query as follows
SELECT DISTINCT tt.firstname,
tt.lastname,
tc.caseid,
tt.courtcode AS courtid,
tcou.courtname,
(SELECT COUNT(*)
FROM tblcasetrafficticketlink
WHERE caseid = tc.caseid) AS ticketcount,
Max(tt.violationdate) AS violationdate,
( tt.address1
|| ','
|| tt.address2 ) AS address,
tt.city,
tt.state,
tt.zip,
tt.dob,
tt.sex
FROM tblcase tc
LEFT OUTER JOIN tblcasetrafficticketlink tcttl
ON tc.caseid = tcttl.caseid
LEFT OUTER JOIN tbltraffictickets tt
ON tcttl.courtid = tt.courtcode
AND tt.ticketnumber = tcttl.ticketnumber
AND ( tcttl.ticketextension = tt.ticketnumberex
OR tt.ticketnumberex IS NULL )
LEFT OUTER JOIN tblcourts tcou
ON tcou.courtid = tt.courtcode
WHERE tc.casetype = 'TRAFFIC'
AND tc.caseid<='"+recent_min_caseID+"'
GROUP BY tc.caseid,
tt.firstname,
tt.lastname,
tt.dob,
tt.sex,
tt.courtcode,
tcou.courtname,
tt.city,
tt.state,
tt.zip,
tc.casestatus,
tt.address1,
tt.address2
ORDER BY tc.caseid DESC
LIMIT 100;
this is taking much time to get data. can anybody help to improve performance.Here PRAGMA is useful? if so how? if not, tell me the way to fix this issue.

I've found that SQLite on Android seems to have some... unexpected quirks. In my case, it turned out that doing a straight query like select * from Foo where Bar is null was MUCH slower than selecting only the IDs and then fetching each row by ID individually. YMMV.