Develop Reference

Query that brings all the parents whose children meet a certain criteria

I need a query that brings all the parents whose children meet a certain criteria.
So far this is giving me something different than expected:
#Transaction
#Query("SELECT * FROM piece_unit_table put WHERE (SELECT et.date_of_creation FROM expense_table et WHERE et.piece_unit_parent_id = put.piece_unit_id) = :dateOfExpenseCreation AND project_id = :projectId")
public abstract LiveData<List<PieceUnit>> getPieceUnitsWhereDateOfExpenseCreation(long dateOfExpenseCreation, int projectId);
A more readable version of the same code above:
SELECT *
FROM piece_unit_table put
WHERE (
SELECT et.date_of_creation
FROM expense_table et
WHERE et.piece_unit_parent_id = put.piece_unit_id
) = :dateOfExpenseCreation
AND project_id = :projectId
At first I thought I got a jackpot with that one because it gave me responses on the first try (I usually spend a whole day just thinking about the structure of the query, I hate them), I never gave it too much thought, but the days passed by and it stopped giving me responses, so I'm guessing that maybe, the query was comparing some other dates inside the WHERE clause, more precisely, the dates on which the pieces where created.
Now that I reread it again it looks like a pretty bad query...
The way in which I'm storing dates (from the children side):
public void insertExpense(Expense expense) {
long now = Converters.dateToTimestamp(LocalDate.now());
expense .setDate_of_creation(now);
long edited = System.currentTimeMillis();
expense .setLast_edited(edited);
Log.println(Log.WARN, TAG, "insertExpense: date of creation is: " + expense.getDate_of_creation());
Log.println(Log.WARN, TAG, "insertExpense: expense project is: " + expense.getParent_project_id());
Log.println(Log.WARN, TAG, "insertExpense: expense piece parent is: " + expense.getPiece_unit_parent_id());
insert(expense);
}
The observer side:
Log.println(Log.ERROR, TAG, "getPieceUnitGroupedByExpenseProductsAtDateAndProject: specific date is: " + specificDate);
Log.d(TAG, "getPieceUnitGroupedByExpenseProductsAtDateAndProject: project id is: " + project);
MyObserver
.observeOnce(
pieceUnitRepository.getPieceUnitsWhereDateOfExpenseCreation(
specificDate,
project
),
pieceUnits ->
{
Log.d(TAG, "getPieceUnitGroupedByExpenseProductsAtDateAndProject: piece units are: " + pieceUnits);
if (pieceUnits != null && pieceUnits.size() > 0) {
PieceUnit p = pieceUnits.get(0);
Log.println(Log.WARN, TAG, "getPieceUnitGroupedByExpenseProductsAtDateAndProject: Hello there ;) : " + p.getBeginning_date());
}
The Logd's:
insertExpense: date of creation is: 18469
insertExpense: expense project is: 1
insertExpense: expense piece parent is: 4
getPieceUnitGroupedByExpenseProductsAtDateAndProject: specific date is: 18469
getPieceUnitGroupedByExpenseProductsAtDateAndProject: project id is: 1
getPieceUnitGroupedByExpenseProductsAtDateAndProject: piece units are: []
I'm not asking for a full solution, but any input could point me in the right direction, so thanks in advance.

Final answer (from second update):
SELECT
put.*
FROM piece_unit_table put
WHERE
EXISTS(
SELECT
1
FROM
expense_table et
WHERE
et.date_of_creation = :dateOfExpenseCreation AND et.piece_unit_parent_id = piece_unit_id
) AND
put.project_id = :projectId
EDITS
First answer :
In SQL there seems to be a function that helps on these cases:
all or not exist
https://stackoverflow.com/a/42439405/11214643
In the SQLite case there doesn't seem to be a direct equivalence, and every solution seems to be a case by case workaround depending on the type of clause.
In my case I LEFT JOIN-ed the children table, but group by-ing it before the join to avoid repeated rows on the left table.
As is expected, it was during the child table sub-query, that I filtered the table by their date condition
SELECT put.*
FROM piece_unit_table put
LEFT JOIN(
SELECT
piece_unit_parent_id
FROM
expense_table et
WHERE
et.date_of_creation = :dateOfExpenseCreation
GROUP BY
et.piece_unit_parent_id
) et1
ON
et1.piece_unit_parent_id = put.piece_unit_id
WHERE
put.project_id = :projectId
UPDATE:
It seems that behind curtains(either be SQLite or the ROOM interface), the LEFT JOIN is happening before the sub-query executes its own WHERE clause, as a result, the query is giving me parents, even when there are no children that meet the criteria, so the solution was to bring an additional column from the child temp table to the 'front', and check for it's non null-ability.
SELECT
put.*,
et1.puId AS expPuId
FROM piece_unit_table put
LEFT JOIN(
SELECT
piece_unit_parent_id AS puId
FROM
expense_table et
WHERE
et.date_of_creation = :dateOfExpenseCreation
GROUP BY
et.piece_unit_parent_id
) et1
ON
et1.puId = put.piece_unit_id
WHERE
put.project_id = :projectId AND expPuId NOT NULL
UPDATE 2:
Thanks to #forpas
The EXISTS and NOT EXISTS operators are in fact supported by SQLite, also, what was bringing me parents even when no children met the criteria was the second WHERE clause that was applied directly to the left table, but even if there was no clause applied to this table, it would've still gave me answers, because that's how LEFT JOIN works, if there are no matches it fills them with NULL's.
Here is a query that has the expected result, but does it better.
SELECT
put.*
FROM piece_unit_table put
WHERE
EXISTS(
SELECT
1
FROM
expense_table et
WHERE
et.date_of_creation = :dateOfExpenseCreation AND et.piece_unit_parent_id = piece_unit_id
) AND
put.project_id = :projectId

Missing records due to skipped frames

The app im working with is getting data from a .csv (20k-30k records) from a server and it needs to persist the data into an SQLiteDatabase.
It works but some records are missing and appeared that it have been skipped.
I/Choreographer( 2555): Skipped 46 frames! The application may be doing too much work on its main thread.
I know that this error says that the memory consumption is very high due to heavy load. Is there a more efficient way in persisting data in SQLiteDatabase rather than the classic accessing of CSV and processing it from there?
Code for writing in DB
String sql = "INSERT INTO " + tableName
+ " VALUES (?,?,?,?,?,?);";
SQLiteDatabase db = openHelper.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
try {
db.beginTransaction();
String[] sa = null;
for (final String csvline : arrCSV) {
statement.clearBindings();
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
}
statement.execute();
}
db.setTransactionSuccessful();
Log.d("Transaction", "Successful");
}catch(Exception e){
e.printStackTrace();
}finally {
statement.releaseReference();
statement.close();
db.endTransaction();
db.releaseMemory();
}
UPDATE
The missing records were not loaded in the Collection.
Is the skipping of frames the culprit here?
The loading in the collection is just a simple parsing of a csv file and
non replicable at times so Im assuming it is due to the skipping of frames.

I believe the issue is not linked to skipping frames and < 100 frames is considered a small/insignificant number. At least according to The application may be doing too much work on its main thread.
I frequently see it and has never been the cause of any issues. I've even seen it basically doing nothing other than returning a result from an activity to an activity that just starts the second activity.
As you have commented, the number of elements that result from the split is on occasion not 6. The issue is likely that the insert is not happening on such an occasion, perhaps due to constraints (without seeing how the columns are defined only guesses could be made).
However, you appear to consider that each line in csvline should be split into 6 elements. You should thus investigate as to why not?
To investigate I'd suggest getting details of the original data before the split and the resultant data after the split whenever the number of elements created by the split is not 6. e.g. by changing :-
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
}
statement.execute();
to
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
} else {
Log.d("NOT6SPLIT","CSVLINE WAS ===>" + csvline + "<===");
Log.d("NOT6SPLIT","CSVLINE WAS SPLIT INTO " + Integer.toString(sa.length) + " ELEMENTS :-");
for(String s: sa) {
Log.d("NOT6SPLIT","\tElement Value ===>" + s + "<===");
}
}
statement.execute();
Changing statement.execute() to :-
if (statement.executeInsert() < 1) {
Log.d("INSERTFAIL","Couldn't insert where CSVLINE was ===>" + csvline + "<===");
}
May also assist ('executeInsert' returns the rowid of the inserted record, else -1, not sure of the consequences of a table defined with WITHOUT ROWID).
It wouldn't surprise me at all if the issue boils down to your data containing characters that split considers special or metacharaceters:-
there are 12
characters with
special meanings:
the backslash \,
the caret ^,
the dollar sign $,
the period or dot .,
the vertical bar or pipe symbol |,
the question mark ?,
the asterisk or star *,
the plus sign +,
the opening parenthesis (,
the closing parenthesis ),
the opening square bracket [,
and the opening curly brace {,
These special characters are often called "metacharacters". Most
of them are errors when used alone.

I need to separate the text from a string based on column names

I am working on OCR based Android app, getting this text as string from the attached image dynamically (getting the text in Horizontal Direction from the image)
Text from Image:
"Part Name Part Cost Engine Oil and Oil Filter Replacement Rs 10K Alf Filter Rs 4500 Cabin AC Micro Filter Rs 4000 Pollen Filter Rs 1200 - 1500 AC Disinfectant Rs 3000 Fuel Filter Rs 6000 - 8000 Spark Plug Set Replacement (Applicable in TFSI / Petrol Car Range) Rs 10K Body Wash, Basic Clean 8. Engine Degrease Rs 3000 Body Wax Polish Detailed Rs 7000 - 8000 Car interior Dry Clean with Genn Clean Rs 8000 - 10000 Wheel Alignment \u0026 Balancing Rs 6000 - 7000 Brake Pads Replacernent (Pair) Rs 30K - 32K Brake Disc Replacernent (Pair) Rs 30K - 35K ..........".
I need to separate the Part Name and Part Cost(just 2 columns i.e Part Name, Part Cost) (ignore all extra text from the column heading). Separate the values from String and should store it in SQLIte Database Android. I am stuck how to get the values and separate them.

The text returned from the OCR isn't ideal. The first thing you should do is check if whatever OCR solution can be configured to provide a better output. Ideally, you want the lines to be separated by newline characters and the space between the columns to be interpreted as something more useful, such as a tab character.
If you have no way of changing the text you get, you'll have to find some way of parsing it. You may want to look into using a parser, such as ANTLR to make this easier.
The following observations may help you to come up with a parsing strategy:
Column 2 items all start with "Rs" or "Upto Rs".
Column 2 items end with:
A number (where a number is allowed to be a string of digits [0-9.], optionally followed by a "K"
"Lakh"
Column 1 items don't begin with a number or "Lakh"
So a basic algorithm could be:
List<String> column1 = new ArrayList<String>();
List<String> column2 = new ArrayList<String>();
String[] tokens = ocrString.split(" ");
List<String> column = column1;
String item = "";
for (int i = 0; i < tokens.length; i++) {
String token = tokens[i];
String nextToken = i == tokens.length - 1 ? "" : tokens[i+1];
if (column == column1) {
if (token == "Rs" || (token == "Upto" && nextToken == "Rs")) {
column = column2;
column.add(item); item = "";
i--; continue;
}
item += " " + token;
} else {
item += " " + token;
if (/*token is number or "Lakh" and nextToken is not*/) {
column.add(item); item = "";
column = column1;
}
}
}

Google Sheets API v4 - How to get the last row with value?

How to get the last row with value in the new Google Sheets API v4 ?
i use this to get a range from a sheet:
mService.spreadsheets().values().get("ID_SHEET", "Sheet1!A2:B50").execute();
how to detect the last row with value in the sheet ?

You can set the range to "A2:D" and this would fetch as far as the last data row in your sheet.

I managed to get it by counting the total rows from current Sheets.
Then append the data to the next row.
rowcount = this.mService.spreadsheets().values().get(spreadsheetId, range).execute().getValues().size()

Rather than retrieving all the rows into memory just to get the values in the last row, you can use the append API to append an empty table to the end of the sheet, and then parse the range that comes back in the response. You can then use the index of the last row to request just the data you want.
This example is in Python:
#empty table
table = {
'majorDimension': 'ROWS',
'values': []
}
# append the empty table
request = service.spreadsheets().values().append(
spreadsheetId=SPREADSHEET_ID,
range=RANGE_NAME,
valueInputOption='USER_ENTERED',
insertDataOption='INSERT_ROWS',
body=table)
result = request.execute()
# get last row index
p = re.compile('^.*![A-Z]+\d+:[A-Z]+(\d+)$')
match = p.match(result['tableRange'])
lastrow = match.group(1)
# lookup the data on the last row
result = service.spreadsheets().values().get(
spreadsheetId=SPREADSHEET_ID,
range=f'Sheetname!A{lastrow}:ZZ{lastrow}'
).execute()
print(result)

😢 Google Sheets API v4 does not have a response that help you to get the index of the last written row in a sheet (row that all cells below it are empty). Sadly, you'll have to workaround and fetch all sheet rows' into memory (I urge you to comment if I'm mistaken)
Example:
spreadsheet_id = '1TfWKWaWypbq7wc4gbe2eavRBjzuOcpAD028CH4esgKw'
range = 'Sheet1!A:Z'
rows = service.spreadsheets().values().get(spreadsheetId=spreadsheet_id, range=range).execute().get('values', [])
last_row = rows[-1] if rows else None
last_row_id = len(rows)
print(last_row_id, last_row)
Output:
13 ['this', 'is ', 'my', 'last', 'row']
💡 If you wish to append more rows to the last row, see this

You don't need to. Set a huge range (for example A2:D5000) to guarantee that all your rows will be located in it. I don't know if it has some further impact, may be increased memory consumption or something, but for now it's OK.
private List<String> getDataFromApi() throws IOException {
String spreadsheetId = "1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms";
String range = "A2:D5000";
List<String> results = new ArrayList<String>();
ValueRange response = this.mService.spreadsheets().values()
.get(spreadsheetId, range)
.execute();
List<List<Object>> values = response.getValues();
if (values != null) {
results.add("Name, Major");
for (List row : values) {
results.add(row.get(0) + ", " + row.get(3));
}
}
return results;
}
Look at the loop for (List row : values). If you have two rows in your table you will get two elements in values list.

Have a cell somewhere that doesn't interfere with your datarange with =COUNTA(A:A) formula and get that value.
In your case
=MAX(COUNTA(A:A50),COUNTA(B:B50))
?
If there could be empty cells inbetween the formula would be a little more tricky but I believe it saves you some memories.

2022 Update
I I don’t know if this will be relevant for someone in 2022, but now you can do it differently.
You can just set next value as range :
const column = "A"
const startIndex = 2
const range = column + startIndex + ":" + column
In resolve you get all data in column and range with last index.
I tested it on js and php

Following Mark B's answer, I created a function that performs a dummy append and then extracts the last row info from the dummy append's response.
def get_last_row_with_data(service, value_input_option="USER_ENTERED"):
last_row_with_data = '1'
try:
dummy_request_append = service.spreadsheets().values().append(
spreadsheetId='<spreadsheet id>',
range="{0}!A:{1}".format('Tab Name', 'ZZZ'),
valueInputOption='USER_ENTERED',
includeValuesInResponse=True,
responseValueRenderOption='UNFORMATTED_VALUE',
body={
"values": [['']]
}
).execute()
a1_range = dummy_request_append.get('updates', {}).get('updatedRange', 'dummy_tab!a1')
bottom_right_range = a1_range.split('!')[1]
number_chars = [i for i in list(bottom_right_range) if i.isdigit()]
last_row_with_data = ''.join(number_chars)
except Exception as e:
last_row_with_data = '1'
return last_row_with_data

How to insert a table in to group in Corona SDK (.Lua)?

I get error message when i try to insert a table into a group
My table code is containing images
Here is the code i am using for the table
local myJoints = {}
for i = 1,5 do
local link = {}
for j = 1,17 do
link[j] = display.newImage( "link.png" )
link[j].x = 121 + (i*34)
link[j].y = 55 + (j*17)
physics.addBody( link[j], { density=2.0, friction=0, bounce=0 } )
-- Create joints between links
if (j > 1) then
prevLink = link[j-1] -- each link is joined with the one above it
else
prevLink = wall -- top link is joined to overhanging beam
end
myJoints[#myJoints + 1] = physics.newJoint( "pivot", prevLink, link[j], 121 + (i*34), 46 + (j*17) )
end
end
and here is the code for group
GUI:insert(myJoints);
i have my background image in the GUI group and it is covering the table.
I don't know if it is actually possible to insert table into a group
Any help please
Thanks in Advance!

You can't insert a table into a group using the "insert" method because that method is looking for a display object. Try calling GUI.myJoints = myJoints. Also keep in mind that your table just references your display objects, which is different from having them in a group.