Develop Reference

Missing records due to skipped frames

The app im working with is getting data from a .csv (20k-30k records) from a server and it needs to persist the data into an SQLiteDatabase.
It works but some records are missing and appeared that it have been skipped.
I/Choreographer( 2555): Skipped 46 frames! The application may be doing too much work on its main thread.
I know that this error says that the memory consumption is very high due to heavy load. Is there a more efficient way in persisting data in SQLiteDatabase rather than the classic accessing of CSV and processing it from there?
Code for writing in DB
String sql = "INSERT INTO " + tableName
+ " VALUES (?,?,?,?,?,?);";
SQLiteDatabase db = openHelper.getWritableDatabase();
SQLiteStatement statement = db.compileStatement(sql);
try {
db.beginTransaction();
String[] sa = null;
for (final String csvline : arrCSV) {
statement.clearBindings();
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
}
statement.execute();
}
db.setTransactionSuccessful();
Log.d("Transaction", "Successful");
}catch(Exception e){
e.printStackTrace();
}finally {
statement.releaseReference();
statement.close();
db.endTransaction();
db.releaseMemory();
}
UPDATE
The missing records were not loaded in the Collection.
Is the skipping of frames the culprit here?
The loading in the collection is just a simple parsing of a csv file and
non replicable at times so Im assuming it is due to the skipping of frames.

I believe the issue is not linked to skipping frames and < 100 frames is considered a small/insignificant number. At least according to The application may be doing too much work on its main thread.
I frequently see it and has never been the cause of any issues. I've even seen it basically doing nothing other than returning a result from an activity to an activity that just starts the second activity.
As you have commented, the number of elements that result from the split is on occasion not 6. The issue is likely that the insert is not happening on such an occasion, perhaps due to constraints (without seeing how the columns are defined only guesses could be made).
However, you appear to consider that each line in csvline should be split into 6 elements. You should thus investigate as to why not?
To investigate I'd suggest getting details of the original data before the split and the resultant data after the split whenever the number of elements created by the split is not 6. e.g. by changing :-
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
}
statement.execute();
to
sa = csvline.split(",");
if(sa.length==6){
statement.bindString(1, sa[0]);
statement.bindString(2, sa[1]);
statement.bindString(3, sa[2]);
statement.bindString(4, sa[3]);
statement.bindString(5, sa[4]);
statement.bindString(6, sa[5]);
} else {
Log.d("NOT6SPLIT","CSVLINE WAS ===>" + csvline + "<===");
Log.d("NOT6SPLIT","CSVLINE WAS SPLIT INTO " + Integer.toString(sa.length) + " ELEMENTS :-");
for(String s: sa) {
Log.d("NOT6SPLIT","\tElement Value ===>" + s + "<===");
}
}
statement.execute();
Changing statement.execute() to :-
if (statement.executeInsert() < 1) {
Log.d("INSERTFAIL","Couldn't insert where CSVLINE was ===>" + csvline + "<===");
}
May also assist ('executeInsert' returns the rowid of the inserted record, else -1, not sure of the consequences of a table defined with WITHOUT ROWID).
It wouldn't surprise me at all if the issue boils down to your data containing characters that split considers special or metacharaceters:-
there are 12
characters with
special meanings:
the backslash \,
the caret ^,
the dollar sign $,
the period or dot .,
the vertical bar or pipe symbol |,
the question mark ?,
the asterisk or star *,
the plus sign +,
the opening parenthesis (,
the closing parenthesis ),
the opening square bracket [,
and the opening curly brace {,
These special characters are often called "metacharacters". Most
of them are errors when used alone.

Bulk update of more than 500 contacts

I am developing an app that needs to update many contacts and I am getting the following error.
android.content.OperationApplicationException: Too many content provider operations between yield points. The maximum number of operations per yield point is 500
I tried breaking the contacts up into smaller chunks to update, but I still get the same error. The good thing is that now, some contacts are updated (previously 0 contacts are updated). Any suggestions that can help me is greatly appreciated.
Uri uri = ContactsContract.Data.CONTENT_URI;
String selectionUpdate = ContactsContract.CommonDataKinds.Phone._ID + " = ? AND " + ContactsContract.Contacts.Data.MIMETYPE + " = ? ";
int i = 0;
int numRowsUpdated = 0;
int batchsize = 100;
for (EntityPhone ep : eps) {
if (ep.isUpdateNumber()) {
//update only when checkbox is ticked
ops.add(ContentProviderOperation.newUpdate(uri)
.withSelection(selectionUpdate, new String[]{ep.getPhoneId(), ContactsContract.CommonDataKinds.Phone.CONTENT_ITEM_TYPE})
.withValue(ContactsContract.CommonDataKinds.Phone.NUMBER, ep.getPhoneNumberNew())
.build());
i++;
if (i % batchsize == 0) {
i = 0;
ContentProviderResult[] count = contentResolver.applyBatch(ContactsContract.AUTHORITY, ops);
if (count != null) {
numRowsUpdated += count.length;
Log.i(TAG, "batch update success" + count.length);
} else {
Log.w(TAG, "batch update failed");
}
}
}
}
if (i != 0) {
ContentProviderResult[] count = contentResolver.applyBatch(ContactsContract.AUTHORITY, ops);
}
I have looked at the past questions, but they are mostly related to inserts, not updates.
Insertion of thousands of contact entries using applyBatch is slow
Whats the fastest way to create large numbers of contacts?
The reason why I want to update so many records at once is because my application is a 'contact number formatter' that allows the user to standardizes all the phone numbers in the phone easily. I do not have control of how many records the users want to update in a single batch. (https://play.google.com/store/apps/details?id=angel.phoneformat)

You're not creating a new object for ops. During subsequent calls to applyBatch, you're passing the previously applied operations back in as well. The first time ops contains 100 elements, then 200 and eventually it fails when it reaches 500. Change to
if (i % batchsize == 0) {
contentResolver.applyBatch(ContactsContract.AUTHORITY, ops);
ops = new ArrayList<ContentProviderOperation>(100);
}

Checking data enter by user with parameter set, sqlite android

i'm doing a checking from data enter by user with data that already set.
As example, data save in sqlite is 60-80. If user input is 88. i want to check the user input which is 88 with data 60-80, so that i can come out with appropriate advice
Help me.
i did like this
public String getRangeFromUserInput(int b1, int b2, int b3, int a1, int a2, int a3) {
if
(((70<b1<<100)||(70<b2<<100)||(70<b3<<100))&&((70<a1<<135)||(70<a2<<135)||(70<a3<<135)))
Toast.makeText(GlucoseAdd.this, "Your glucose level is at normal range", Toast.LENGTH_LONG).show();
else if
(((50<b1<<60)||(50<b2<<60)||(50<b3<<60))&&((135<a1<<180)||(135<a2<<180)||(135<a3<<180)))
Toast.makeText(GlucoseAdd.this, "You need to start diet. The diet should " +
"also be low in fat and high in dietary fibre. Regular exercise " +
"is important in maintaining balanced blood glucose levels.", Toast.LENGTH_LONG).show();
else if
(((b1<50)||(b2<50)||(b3<50))&&((180<a1<<200)||(180<a2<<200)||(180<a3<<200)))
Toast.makeText(GlucoseAdd.this, "You are in danger state.", Toast.LENGTH_LONG).show();
return null;
}
but when i enter data all prompt out the first one.. Why?

Since the data stored in your DB table is in the format of string i.e. 60-80
So you'll need to convert the user input value (for ex. 88) into i.e. 80-100 using the following code:
public String getRangeFromUserInput(String userInput) {
return getRangeFromUserInput(Integer.parseInt(userInput));
}
public String getRangeFromUserInput(int userInput) {
if (userInput >= 60 && userInput < 80)
return "60-80";
else if (userInput >= 80 && userInput < 100)
return "80-100";
// ...more range checks
return null;
}
Now, use the output of the method getRangeFromUserInput() which is a string (for ex. 80-100) in the where clause of your query as follows:
String where = "range = '" + getRangeFromUserInput("88") + "'";
Assuming the column name is range in your table. If the query returns a row then you can read the advice corresponding to the range using
cursor.getString(cursor.getColumnIndex("advice"))
Assuming the column name is advice in your table.

Android: Need Advice on SQLite, searching slow

I have to search a database that is 26024 entries and counting. It used to be fast with less records but now is taking like 10 seconds and slowing the app. I was wondering if i could get advice as to how speed up the process or if i'm doing anything wrong. Here is the code.
while (cursor.moveToNext()) {
String word = cursor.getString(0);
if (word.equals(input)) {
String nikus = cursor.getString(1);
String def = cursor.getString(2);
ret.append(" " + nikus + "\n"+ def + "\n");
g = null;
}
EDIT:
In my Database i have a definitions table and in the table there are 3 fields one is the words to be compared to, the sencond is the full word, and the third is the definition itself. Hopefully that helps you guys a little more.
CREATE TABLE [definitions] (
[word] TEXT,
[fullword] TEXT,
[definition] TEXT);
EDIT: here is the error im getting
01-04 00:47:54.678: E/CursorWindow(4722): need to grow: mSize = 1048576, size = 17, freeSpace() = 13, numRows = 15340

laalto's comment above is correct. You should be running a select with a where clause that only pulls back the rows where word is equal to input (don't forget about case sensitivity.) An index on the word column will help the query go even faster.

Insertion of thousands of contact entries using applyBatch is slow

I'm developing an application where I need to insert lots of Contact entries. At the current time approx 600 contacts with a total of 6000 phone numbers. The biggest contact has 1800 phone numbers.
Status as of today is that I have created a custom Account to hold the Contacts, so the user can select to see the contact in the Contacts view.
But the insertion of the contacts is painfully slow. I insert the contacts using ContentResolver.applyBatch. I've tried with different sizes of the ContentProviderOperation list(100, 200, 400), but the total running time is approx. the same. To insert all the contacts and numbers takes about 30 minutes!
Most issues I've found regarding slow insertion in SQlite brings up transactions. But since I use the ContentResolver.applyBatch-method I don't control this, and I would assume that the ContentResolver takes care of transaction management for me.
So, to my question: Am I doing something wrong, or is there anything I can do to speed this up?
Anders
Edit:
#jcwenger:
Oh, I see. Good explanation!
So then I will have to first insert into the raw_contacts table, and then the datatable with the name and numbers. What I'll lose is the back reference to the raw_id which I use in the applyBatch.
So I'll have to get all the id's of the newly inserted raw_contacts rows to use as foreign keys in the data table?

Use ContentResolver.bulkInsert (Uri url, ContentValues[] values) instead of ApplyBatch()
ApplyBatch (1) uses transactions and (2) it locks the ContentProvider once for the whole batch instead locking/unlocking once per operation. because of this, it is slightly faster than doing them one at a time (non-batched).
However, since each Operation in the Batch can have a different URI and so on, there's a huge amount of overhead. "Oh, a new operation! I wonder what table it goes in... Here, I'll insert a single row... Oh, a new operation! I wonder what table it goes in..." ad infinitium. Since most of the work of turning URIs into tables involves lots of string comparisons, it's obviously very slow.
By contrast, bulkInsert applies a whole pile of values to the same table. It goes, "Bulk insert... find the table, okay, insert! insert! insert! insert! insert!" Much faster.
It will, of course, require your ContentResolver to implement bulkInsert efficiently. Most do, unless you wrote it yourself, in which case it will take a bit of coding.

bulkInsert: For those interested, here is the code that I was able to experiment with. Pay attention to how we can avoid some allocations for int/long/floats :) this could save more time.
private int doBulkInsertOptimised(Uri uri, ContentValues values[]) {
long startTime = System.currentTimeMillis();
long endTime = 0;
//TimingInfo timingInfo = new TimingInfo(startTime);
SQLiteDatabase db = mOpenHelper.getWritableDatabase();
DatabaseUtils.InsertHelper inserter =
new DatabaseUtils.InsertHelper(db, Tables.GUYS);
// Get the numeric indexes for each of the columns that we're updating
final int guiStrColumn = inserter.getColumnIndex(Guys.STRINGCOLUMNTYPE);
final int guyDoubleColumn = inserter.getColumnIndex(Guys.DOUBLECOLUMNTYPE);
//...
final int guyIntColumn = inserter.getColumnIndex(Guys.INTEGERCOLUMUNTYPE);
db.beginTransaction();
int numInserted = 0;
try {
int len = values.length;
for (int i = 0; i < len; i++) {
inserter.prepareForInsert();
String guyID = (String)(values[i].get(Guys.GUY_ID));
inserter.bind(guiStrColumn, guyID);
// convert to double ourselves to save an allocation.
double d = ((Number)(values[i].get(Guys.DOUBLECOLUMNTYPE))).doubleValue();
inserter.bind(guyDoubleColumn, lat);
// getting the raw Object and converting it int ourselves saves
// an allocation (the alternative is ContentValues.getAsInt, which
// returns a Integer object)
int status = ((Number) values[i].get(Guys.INTEGERCOLUMUNTYPE)).intValue();
inserter.bind(guyIntColumn, status);
inserter.execute();
}
numInserted = len;
db.setTransactionSuccessful();
} finally {
db.endTransaction();
inserter.close();
endTime = System.currentTimeMillis();
if (LOGV) {
long timeTaken = (endTime - startTime);
Log.v(TAG, "Time taken to insert " + values.length + " records was " + timeTaken +
" milliseconds " + " or " + (timeTaken/1000) + "seconds");
}
}
getContext().getContentResolver().notifyChange(uri, null);
return numInserted;
}

An example of on how to override the bulkInsert(), in order to speed up multiples insert, can be found here

#jcwenger At first, after read your post, I think that's the reason of
bulkInsert is quicker than ApplyBatch, but after read the code of Contact Provider, I don't think so.
1.You said ApplyBatch use transactions, yes, but bulkInsert also use transactions. Here is the code of it:
public int bulkInsert(Uri uri, ContentValues[] values) {
int numValues = values.length;
mDb = mOpenHelper.getWritableDatabase();
mDb.beginTransactionWithListener(this);
try {
for (int i = 0; i < numValues; i++) {
Uri result = insertInTransaction(uri, values[i]);
if (result != null) {
mNotifyChange = true;
}
mDb.yieldIfContendedSafely();
}
mDb.setTransactionSuccessful();
} finally {
mDb.endTransaction();
}
onEndTransaction();
return numValues;
}
That is to say, bulkInsert also use transations.So I don't think that's the reason.
2.You said bulkInsert applies a whole pile of values to the same table.I'm sorry I can't find related code in the source code of froyo.And I want to know how could you find that?Could you tell me?
The reason I think is that:
bulkInsert use mDb.yieldIfContendedSafely() while applyBatch use
mDb.yieldIfContendedSafely(SLEEP_AFTER_YIELD_DELAY)/*SLEEP_AFTER_YIELD_DELAY = 4000*/
after reading the code of SQLiteDatabase.java, I find that, if set a time in yieldIfContendedSafely, it will do a sleep, but if you don't set the time, it will not sleep.You can refer to the code below which is a piece of code of SQLiteDatabase.java
private boolean yieldIfContendedHelper(boolean checkFullyYielded, long sleepAfterYieldDelay) {
if (mLock.getQueueLength() == 0) {
// Reset the lock acquire time since we know that the thread was willing to yield
// the lock at this time.
mLockAcquiredWallTime = SystemClock.elapsedRealtime();
mLockAcquiredThreadTime = Debug.threadCpuTimeNanos();
return false;
}
setTransactionSuccessful();
SQLiteTransactionListener transactionListener = mTransactionListener;
endTransaction();
if (checkFullyYielded) {
if (this.isDbLockedByCurrentThread()) {
throw new IllegalStateException(
"Db locked more than once. yielfIfContended cannot yield");
}
}
if (sleepAfterYieldDelay > 0) {
// Sleep for up to sleepAfterYieldDelay milliseconds, waking up periodically to
// check if anyone is using the database. If the database is not contended,
// retake the lock and return.
long remainingDelay = sleepAfterYieldDelay;
while (remainingDelay > 0) {
try {
Thread.sleep(remainingDelay < SLEEP_AFTER_YIELD_QUANTUM ?
remainingDelay : SLEEP_AFTER_YIELD_QUANTUM);
} catch (InterruptedException e) {
Thread.interrupted();
}
remainingDelay -= SLEEP_AFTER_YIELD_QUANTUM;
if (mLock.getQueueLength() == 0) {
break;
}
}
}
beginTransactionWithListener(transactionListener);
return true;
}
I think that's the reason of bulkInsert is quicker than applyBatch.
Any question please contact me.

I get the basic solution for you,
use "yield points" in batch operation.
The flip side of using batched operations is that a large batch may lock up the database for a long time preventing other applications from accessing data and potentially causing ANRs ("Application Not Responding" dialogs.)
To avoid such lockups of the database, make sure to insert "yield points" in the batch. A yield point indicates to the content provider that before executing the next operation it can commit the changes that have already been made, yield to other requests, open another transaction and continue processing operations.
A yield point will not automatically commit the transaction, but only if there is another request waiting on the database. Normally a sync adapter should insert a yield point at the beginning of each raw contact operation sequence in the batch. See withYieldAllowed(boolean).
I hope it's may be useful for you.

Here is am example of inserting same data amount within 30 seconds.
public void testBatchInsertion() throws RemoteException, OperationApplicationException {
final SimpleDateFormat FORMATTER = new SimpleDateFormat("mm:ss.SSS");
long startTime = System.currentTimeMillis();
Log.d("BatchInsertionTest", "Starting batch insertion on: " + new Date(startTime));
final int MAX_OPERATIONS_FOR_INSERTION = 200;
ArrayList<ContentProviderOperation> ops = new ArrayList<>();
for(int i = 0; i < 600; i++){
generateSampleProviderOperation(ops);
if(ops.size() >= MAX_OPERATIONS_FOR_INSERTION){
getContext().getContentResolver().applyBatch(ContactsContract.AUTHORITY,ops);
ops.clear();
}
}
if(ops.size() > 0)
getContext().getContentResolver().applyBatch(ContactsContract.AUTHORITY,ops);
Log.d("BatchInsertionTest", "End of batch insertion, elapsed: " + FORMATTER.format(new Date(System.currentTimeMillis() - startTime)));
}
private void generateSampleProviderOperation(ArrayList<ContentProviderOperation> ops){
int backReference = ops.size();
ops.add(ContentProviderOperation.newInsert(ContactsContract.RawContacts.CONTENT_URI)
.withValue(ContactsContract.RawContacts.ACCOUNT_NAME, null)
.withValue(ContactsContract.RawContacts.ACCOUNT_TYPE, null)
.withValue(ContactsContract.RawContacts.AGGREGATION_MODE, ContactsContract.RawContacts.AGGREGATION_MODE_DISABLED)
.build()
);
ops.add(ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, backReference)
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.StructuredName.CONTENT_ITEM_TYPE)
.withValue(ContactsContract.CommonDataKinds.StructuredName.GIVEN_NAME, "GIVEN_NAME " + (backReference + 1))
.withValue(ContactsContract.CommonDataKinds.StructuredName.FAMILY_NAME, "FAMILY_NAME")
.build()
);
for(int i = 0; i < 10; i++)
ops.add(ContentProviderOperation.newInsert(ContactsContract.Data.CONTENT_URI)
.withValueBackReference(ContactsContract.Data.RAW_CONTACT_ID, backReference)
.withValue(ContactsContract.Data.MIMETYPE, ContactsContract.CommonDataKinds.Phone.CONTENT_ITEM_TYPE)
.withValue(ContactsContract.CommonDataKinds.Phone.TYPE, ContactsContract.CommonDataKinds.Phone.TYPE_MAIN)
.withValue(ContactsContract.CommonDataKinds.Phone.NUMBER, Integer.toString((backReference + 1) * 10 + i))
.build()
);
}
The log:
02-17 12:48:45.496 2073-2090/com.vayosoft.mlab D/BatchInsertionTest﹕ Starting batch insertion on: Wed Feb 17 12:48:45 GMT+02:00 2016
02-17 12:49:16.446 2073-2090/com.vayosoft.mlab D/BatchInsertionTest﹕ End of batch insertion, elapsed: 00:30.951

Just for the information of the readers of this thread.
I was facing performance issue even if using applyBatch().
In my case there was database triggers written on one of the table.
I deleted the triggers of the table and its boom.
Now my app insert rows with blessing fast speed.