I am trying to build a sqlite database. It has 2800 entries in them. When i try to insert, it takes a minute or so and later gives the system error message. The respective codes are given below.
In the create database java file,
ContentValues cv4 = new ContentValues();
public long createVariantEntry(String varid, String makeid, String modelid, String varname, String posteddate) {
cv4.put(VARIANT_ID, varid);
cv4.put(VARIANT_MAKE_ID, makeid);
cv4.put(VARIANT_MODEL_ID, modelid);
cv4.put(VARIANT_NAME, varname);
cv4.put(VARIANT_POSTED_DATE, posteddate);
return Database.insert(VARIANT_TABLE_NAME, null, cv4);
}
In the mainActivity,
for(int i = 0; i<build_emp_database.size();i++)
{
md.createVariantEntry(build_emp_database.get(i).get(0), build_emp_database.get(i).get(1), build_emp_database.get(i).get(2), build_emp_database.get(i).get(3), build_emp_database.get(i).get(4));
}
Also, just for 2800 entries, it is taking more than a minute, is there any way to speed them up?? I have several small databases, and have loaded them successfully. This is the only big database and its creating an issue while inserting. Please help where am i going wrong.
Don't try to run the insert operation on the UI thread, for starters.
I suggest you investigate using a content provider as a wrapper around your database. The ContentResolver object provides methods for doing operations in batch, and is in general a more robust way of working with databases. Use an IntentService to run the insert operation in the background.
If you do a lot insert operation, you need use the ContentProviderOperation to optimize your db operation. like these:
ArrayList<ContentProviderOperation> ops = new ArrayList<ContentProviderOperation>();
for(int i = 0; i<build_emp_database.size();i++) {
ContentProviderOperation.Builder builder =
ContentProviderOperation.newInsert(yourUrl);
builder.withValue(VARIANT_ID, varid)
.withValue(VARIANT_MAKE_ID, makeid)
...
ops.add(builder.build());
}
yourContentResolver.applyBatch(yourauthority, ops);
Related
I am inserting a new book into my book table and after trying to assign it to a many-to-many relation table. Imo this should run in a transaction.
(Because if the m2m insertion fails, the information about the realtionship is lost). My code now looks as follows and fails as i cannot access the BookUserXRefDao.insert(bookUser); query due to static context errors.
Is there an easy way to fix this?
#Transaction
public void insertBook(Book theBook, List<Integer> userIds){
long newBookId= insert(theBook);
//Insert into the m2m relation
BookUserXRef[] bookUser = new BookUserXRef[userIds.size()];
for (int i = 0; i < userIds.size(); i++) {
BookUserXRef[i] = new BookUserXRef(newBookId,userIds.get(i));
}
BookUserXRefDao.insert(bookUser);
}
Just realized that i can access the Singleton Database Instance from within my transaction.
Therefore i could just use
AppDb.getAppDb().BookUserXRefDao().insert(bookUser);
That solved the problem.
I am new to android and maybe its a silly question but i am not getting it. See i am designing a game in which we give scores to some persons. So i want to store the names of the persons in a database while installation and then their scores set to 0 initially which will be updated according to what the users select. Here i am not able to figure out that how should i enter the data as it will be around 100 names and their scores. Using INSERT INTO() statement will make it like 100 statements. So is there any short method like can we do it through strings or something. Just guessing though. Any help would be appreciated.
You don't hard-code names or scores into your SQL statements. Instead, you use parameters.
var command = new SQLiteCommand()
command.CommandText = "INSERT INTO Scores (name, score) VALUES(#name, #score)";
command.CommandType = CommandType.Text;
foreach (var item in data)
{
command.Parameters.Add(new SQLiteParameter("#name", item.Name));
command.Parameters.Add(new SQLiteParameter("#score", item.Score));
command.ExecuteNonQuery();
}
and then just loop through all of the names and scores.
I recommend you using a transaction.
You can archive this stating you want to use a transaction with beginTransaction(), do all the inserts on makeAllInserts() with a loop and if everything works then call setTransactionSuccessful() to do it in a batch operation. If something goes wrong, on the finally section you will call endTransaction() without setting the success, this will execute a rollback.
db.beginTransaction();
try {
makeAllInserts();
db.setTransactionSuccessful();
}catch {
//Error in between database transaction
}finally {
db.endTransaction();
}
For the makeAllInserts function, something like this could work out:
public void makeAllInserts() {
for(int i = 0; i < myData.size(); i++) {
myDataBase = openDatabase();
ContentValues values = new ContentValues();
values.put("name", myData.get(i).getName());
values.put("score", myData.get(i).getScore());
myDataBase.insert("MYTABLE", nullColumnHack, values);
}
}
If you also want to know about the nullColumnHack here you have a good link -> https://stackoverflow.com/a/2663620/709671
Hope it helps.
Hello I am pretty new with SQLite and I am trying to deal with some database manipulation in my project.
I have a table with almost 4000 rows and this is the format of every row:
problem_id (string)
problem_no (string)
problem_title (string)
dacu (int)
I need to query a bunch of problem_no based on the problem_id. The quantity of query is almost 1000 at a time. So I wrote a query code like this:
Set<Integer> getProblemsTitle(HashSet<String> problemsIDs) {
SQLiteDatabase db = this.getReadableDatabase();
HashSet<Integer> problemNo = new HashSet<Integer>();
Cursor cursor = null;
for (Iterator<String> iterator = problemsIDs.iterator(); iterator.hasNext();) {
cursor = db.query(CommonUtils.PROBLEM_TABLE, new String[] {
CommonUtils.KEY_PROBLEM_NO },
CommonUtils.KEY_PROBLEM_ID + "=?",
new String[] { iterator.next() }, null, null, null, null);
if (cursor != null && cursor.moveToFirst()) {
problemNo.add(cursor.getInt(0));
}
cursor.close();
}
db.close();
Set<Integer> set = new TreeSet<Integer>(problemNo);
return set;
}
I know this is not a optimized snippet. And I need to optimize it a lot to reduce the execution time of the query. I did it inside AsyncTask but it is taking too much time.
How can I do this efficiently with faster performance?
You might want to consider taking this out of the database. If you just grabbed all the problems, you could add them all in code. Running one SELECT with 4000 results is still going to be much faster than a thousand SELECT statements.
The approach would be to grab them all, but sorted(ORDER BY problem_id). You could then just check each item in problemIDs against it, and add when you get a match.
You could also use the IN operator as Mathew suggests, but I don't know how efficient that will be with 1000 items in the set.
Don't iterate over a collection of IDs, but use the IN operator in a WHERE condition.
SELECT * FROM Table WHERE problem_id IN (1,2,3,4,5)
This will return all the records in the set. Whereas you are querying them one at a time.
You could try compiling a query, and maybe you can try to load the database into memory before reading.
Create an index on the problem_id column.
When the user downloads my app for the first time, the app downloads a CSV file which contains about 100,000 rows of data.
Then, I would like to populate my SQLite database with it.
Here is my code:
InputStream inputStream = activity.openFileInput(file);
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
String line = null;
dbHelper.beginTransaction();
while((line = bufferedReader.readLine()) != null) {
String[] values = line.replaceAll("'", "''").split(",");
dbHelper.insert(values);
}
dbHelper.setTransactionSuccessful();
dbHelper.endTransaction();
dbHelper.close();
In my DBHelper class, here is the method insert:
public void insert(String[] data) {
ContentValues values = new ContentValues();
values.put(DBHelper.SHAPE_ID, data[0]);
values.put(DBHelper.SHAPE_PT_LAT, data[1]);
values.put(DBHelper.SHAPE_PT_LON, data[2]);
values.put(DBHelper.SHAPE_PT_SEQUENCE, data[3]);
myDB.insert(DBHelper.TABLE_SHAPES, null, values);
}
I tried this code, it worked BUT it took 7 minutes to do it...
Am I doing something wrong ? Is there a better way (read faster) to populate a SQLite database?
You should download the SQLite database file itself.
If there is other data in the database that you want to keep, you could either
copy that other data into the downloaded database (since it is not as much data, this should be fast); or
keep the downloaded database and the other database separated (and ATTACH one to the other if you need to do joins between them).
Use a prepared statement, that should give a performance boost.
Something like:
SQLiteStatement statement = db.compileStatement("INSERT INTO " + TABLE_SHAPES + " VALUES(?, ?));
and change your insert() method to do something like:
statement.bindString(1, data[0]);
statement.bindString(2, data[1]);
statement.executeInsert();
According to this, SQLite3 has facilities to import a CSV file built-in:
.separator ','
.import '/path/to/csv/data' [table]
I'd imagine this can be done on Android using the DatabaseUtils.InsertHelper:
DatabaseUtils.InsertHelper helper = new DatabaseUtils.InsertHelper(dbFilePath, dbTablename);
helper.prepareForInsert();
// use the bind* methods to bind your columns of CSV data to the helper in a loop
helper.execute();
I also see the the class is now marked as deprecated (API level 17) in lieu of SQLiteStatement, which may or may not be relevant to your application. If you have further questions, please leave a comment.
I am trying to insert around 2800 records into the sqlite database, it is taking 150 sec, which is way too much! Could anyone please tell how to optimize this insertion.
public void createVariantEntry(ArrayList<ArrayList<String>> str) {
InsertHelper ih = new InsertHelper(Database, VARIANT_TABLE_NAME);
final int varid = ih.getColumnIndex(VARIANT_ID);
final int varmakeid = ih.getColumnIndex(VARIANT_MAKE_ID);
final int varmodid = ih.getColumnIndex(VARIANT_MODEL_ID);
final int varname = ih.getColumnIndex(VARIANT_NAME);
final int varposteddate = ih.getColumnIndex(VARIANT_POSTED_DATE);
for(int i=0;i<1253;i++)
{
ih.prepareForInsert();
ih.bind(varid, str.get(i).get(0));
ih.bind(varmakeid, str.get(i).get(1));
ih.bind(varmodid, str.get(i).get(2));
ih.bind(varname, str.get(i).get(3));
ih.bind(varposteddate, str.get(i).get(4));
ih.execute();
}
for(int i=1255;i<str.size();i++)
{
ih.prepareForInsert();
ih.bind(varid, str.get(i).get(0));
ih.bind(varmakeid, str.get(i).get(1));
ih.bind(varmodid, str.get(i).get(2));
ih.bind(varname, str.get(i).get(3));
ih.bind(varposteddate, str.get(i).get(4));
ih.execute();
}
ih.close();
}
a great boost in performance will be gained when using transactions.
try {
SQLiteDatabase db = MySQLiteOpenHelper.getWritableDatabse();
ContentValues values = new ContentValues();
db.beginTransaction();
while ( more_data_to_insert ) {
// put the data in 'values'
values.put("col_1", data_1);
values.put("col_2", data_2);
// ...
values.put("col_n", data_n);
// Insert the row into the database.
db.insert("table_name", null, values);
}
db.setTransactionSuccessful();
} catch ( SQLiteException e ) {
// handle your sqlite errors
} finally {
db.endTransaction();
}
and don't use InsertHelper. its deprecated now.
Here are some general tips that might help you:
You can bulkInsert or applyBatch using ContentProviders to do a bunch of operations in one go:
How to use bulkInsert() function in android?
You can use transactions to speed things up as well:
Android Database Transaction
In some cases DatabaseUtils.InsertHelper has been known to provide faster inserts than the normal sqlite insert:
http://www.outofwhatbox.com/blog/2010/12/android-using-databaseutils-inserthelper-for-faster-insertions-into-sqlite-database/
After this, You'll have to do some benchmarking and optimize for your specific situation analyzing performance vs data integrity tradeoffs etc. Good luck.