When the user downloads my app for the first time, the app downloads a CSV file which contains about 100,000 rows of data.
Then, I would like to populate my SQLite database with it.
Here is my code:
InputStream inputStream = activity.openFileInput(file);
InputStreamReader inputStreamReader = new InputStreamReader(inputStream);
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
String line = null;
dbHelper.beginTransaction();
while((line = bufferedReader.readLine()) != null) {
String[] values = line.replaceAll("'", "''").split(",");
dbHelper.insert(values);
}
dbHelper.setTransactionSuccessful();
dbHelper.endTransaction();
dbHelper.close();
In my DBHelper class, here is the method insert:
public void insert(String[] data) {
ContentValues values = new ContentValues();
values.put(DBHelper.SHAPE_ID, data[0]);
values.put(DBHelper.SHAPE_PT_LAT, data[1]);
values.put(DBHelper.SHAPE_PT_LON, data[2]);
values.put(DBHelper.SHAPE_PT_SEQUENCE, data[3]);
myDB.insert(DBHelper.TABLE_SHAPES, null, values);
}
I tried this code, it worked BUT it took 7 minutes to do it...
Am I doing something wrong ? Is there a better way (read faster) to populate a SQLite database?
You should download the SQLite database file itself.
If there is other data in the database that you want to keep, you could either
copy that other data into the downloaded database (since it is not as much data, this should be fast); or
keep the downloaded database and the other database separated (and ATTACH one to the other if you need to do joins between them).
Use a prepared statement, that should give a performance boost.
Something like:
SQLiteStatement statement = db.compileStatement("INSERT INTO " + TABLE_SHAPES + " VALUES(?, ?));
and change your insert() method to do something like:
statement.bindString(1, data[0]);
statement.bindString(2, data[1]);
statement.executeInsert();
According to this, SQLite3 has facilities to import a CSV file built-in:
.separator ','
.import '/path/to/csv/data' [table]
I'd imagine this can be done on Android using the DatabaseUtils.InsertHelper:
DatabaseUtils.InsertHelper helper = new DatabaseUtils.InsertHelper(dbFilePath, dbTablename);
helper.prepareForInsert();
// use the bind* methods to bind your columns of CSV data to the helper in a loop
helper.execute();
I also see the the class is now marked as deprecated (API level 17) in lieu of SQLiteStatement, which may or may not be relevant to your application. If you have further questions, please leave a comment.
Related
Pretty simple question, assuming I've got the model for a row in a table, I'd like to get the insert statement necessary to create that row.
List<MyModel> updatedRows = new Select()
.from(MyDatabase.getMyModels().get(table))
.where(Condition.column(NameAlias.builder("id").build())
.in(new Select(UpdatedRecord$Table.updated_record_id)
.from(UpdatedRecord.class)
.where(UpdatedRecord$Table.updated_table.eq(table))))
.queryList();
StringBuilder updateStatements = new StringBuilder();
for (MyModel tableModel : updatedRows) {
// this is the insert statement, but there's no way to get it as a string
tableModel.getModelAdapter().getInsertStatement();
updateStatements.append(insertSqlStatementString);
}
tableModel.getModelAdapter().getInsertStatement() properly returns the insert statement, however, it is in the form of a DatabaseStatement, and I can't find any documentation for that class. I want the insert statement as a string.
Looking at that line in the debugger shows that underneath it there is a Statement and inside of that, mSQL which holds the string.
Can anyone help me out or point me in the right direction?
Tried this to no avail:
String tableCreateStatement = tableModel.getModelAdapter().getCreationQuery();
DatabaseStatement dbSt = tableModel.getModelAdapter().getCompiledStatement();
tableModel.getModelAdapter().bindToInsertStatement(dbSt, tableModel);
AndroidDatabaseStatement a = (AndroidDatabaseStatement)dbSt;
String p = a.getStatement().toString();
Here's what I ended up using after the main contributor to DBFlow responded to my question here:
// BaseModel row;
ContentValues values = new ContentValues();
row.getModelAdapter().bindToInsertValues(values, row);
String query = SQLite.insert(row.getClass()).columnValues(values).getQuery();
Straight to point:
I am making a Quizz app in Android studio. I would like to get the question (and answers) that will fill the app from SQL table that i made in DB browser for SQLite.
My question is how do i connect those two (without rettyping all those question in android studio).
I have in somethin like a structure {ID, qusetion, answer} and I only point ID number i whant to fetch from the table and it fills my array in android studio.
I hope this make sense :)
Regards
You could put every question and answer in a file, line by line, things separated by a specific character.
ex:
What's the color of the sky?;Blue;Red;Yello;Black
Who is Harry potter?;A magician;A girl;A burrito;A car
And read the file, and insert everything them in the database on the first time you open the app.
Assuming you're able to export from DB Browser to a .sql file (including CREATE TABLE statements as you'll effectively be creating a new database from scratch), you can use this exported file to recreate your database in your app.
By overriding the SQLiteOpenHelper.onCreate() method, you could do something like this:
#Override
public void onCreate(SQLiteDatabase db) {
// Open the raw SQL file
InputStream inputStream = context.getResources().openRawResource(R.raw.mydb);
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
// Read the file
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = reader.readLine()) != null) {
sb.append(line);
}
reader.close();
// Pump the statements into the database
String queries = sb.toString()
for (String query : queries.split(";")) {
db.execSQL(query);
}
}
Replacing mydb with whatever the name of your sql file is in /res/raw
I am new to android and maybe its a silly question but i am not getting it. See i am designing a game in which we give scores to some persons. So i want to store the names of the persons in a database while installation and then their scores set to 0 initially which will be updated according to what the users select. Here i am not able to figure out that how should i enter the data as it will be around 100 names and their scores. Using INSERT INTO() statement will make it like 100 statements. So is there any short method like can we do it through strings or something. Just guessing though. Any help would be appreciated.
You don't hard-code names or scores into your SQL statements. Instead, you use parameters.
var command = new SQLiteCommand()
command.CommandText = "INSERT INTO Scores (name, score) VALUES(#name, #score)";
command.CommandType = CommandType.Text;
foreach (var item in data)
{
command.Parameters.Add(new SQLiteParameter("#name", item.Name));
command.Parameters.Add(new SQLiteParameter("#score", item.Score));
command.ExecuteNonQuery();
}
and then just loop through all of the names and scores.
I recommend you using a transaction.
You can archive this stating you want to use a transaction with beginTransaction(), do all the inserts on makeAllInserts() with a loop and if everything works then call setTransactionSuccessful() to do it in a batch operation. If something goes wrong, on the finally section you will call endTransaction() without setting the success, this will execute a rollback.
db.beginTransaction();
try {
makeAllInserts();
db.setTransactionSuccessful();
}catch {
//Error in between database transaction
}finally {
db.endTransaction();
}
For the makeAllInserts function, something like this could work out:
public void makeAllInserts() {
for(int i = 0; i < myData.size(); i++) {
myDataBase = openDatabase();
ContentValues values = new ContentValues();
values.put("name", myData.get(i).getName());
values.put("score", myData.get(i).getScore());
myDataBase.insert("MYTABLE", nullColumnHack, values);
}
}
If you also want to know about the nullColumnHack here you have a good link -> https://stackoverflow.com/a/2663620/709671
Hope it helps.
I am creating a database in Android, all was going nice, but when I was testing the queries retrieving the correct data I've got an error.
E/AndroidRuntime(14126): java.lang.IndexOutOfBoundsException: Invalid index 0, size is 0
I know that means that there is no data matching the query, but the fact is that I inserted the data by query and it actually has a match. And the same query works with all the codes that doesn't have accents.
These are the queries for inserting the rows.
INSERT INTO "codigo" VALUES('A','PEATÓN');
INSERT INTO "codigo" VALUES('B','PEATÓN');
INSERT INTO "codigo" VALUES('C','PEATÓN');
So I did a query that gets the values of the field that I was replacing, like this:
String selectCode = "select distinct c.tipo from codigo c";
Cursor cursor = database.rawQuery(selectCodigo, new String[] {});
cursor.moveToFirst();
while (!cursor.isAfterLast()) {
String codigo= cursor.getString(0);
codigos.add(codigo);
System.out.println(codigo);
cursor.moveToNext();
}
cursor.close();
return codigos;
And the result was this:
10-14 16:40:32.140: I/System.out(13716): PEAT�N
I have the text file in the /raw folder and I edited from the Eclipse so I make sure it wasn't the table export I did, but I have the same results.
This is the code that reads the file:
public int insertFromFile(SQLiteDatabase db,Context context, int resourceId) throws IOException {
// Reseting Counter
int result = 0;
// Open the resource
InputStream insertsStream = context.getResources().openRawResource(resourceId);
BufferedReader insertReader = new BufferedReader(new InputStreamReader(insertsStream));
// Iterate through lines (assuming each insert has its own line and theres no other stuff)
while (insertReader.ready()) {
String insertStmt = insertReader.readLine();
db.execSQL(insertStmt);
result++;
}
insertReader.close();
// returning number of inserted rows
return result;
}
How could I get that accent working?
I am using a lot of them, so, replacing the word is not a way out.
Please help, is the only thing I need to finish this project.
UPDATE:
Had the same problems two more times later. The first I fixed it by working with a .sql file coded in cp1525 and opened it in Eclipse with the default editor and find/replaced the wrong characters.
The last time I've got this error I made it the right way, and found that if you are working with SQLiteManager it imports files encoded in UTF-8 but it export files in ANSI, so I opened the file with Notepad++, change the enconding of the .sql file from ANSI to UTF-8 and it works fine, all the characters were shown fine.
The InputStreamReader constructor documentation says:
Constructs a new InputStreamReader on the InputStream in. This constructor sets the character converter to the encoding specified in the "file.encoding" property and falls back to ISO 8859_1 (ISO-Latin-1) if the property doesn't exist.
If the file is encoded in UTF-8, you have to tell the reader this.
in your SQLQueryString just add before
PRAGMA encoding = "UTF-8";
example
"PRAGMA encoding = \"UTF-8\"; INSERT INTO MYTABLE (mytablefield) VALUES ('value'); "
I am trying to build a sqlite database. It has 2800 entries in them. When i try to insert, it takes a minute or so and later gives the system error message. The respective codes are given below.
In the create database java file,
ContentValues cv4 = new ContentValues();
public long createVariantEntry(String varid, String makeid, String modelid, String varname, String posteddate) {
cv4.put(VARIANT_ID, varid);
cv4.put(VARIANT_MAKE_ID, makeid);
cv4.put(VARIANT_MODEL_ID, modelid);
cv4.put(VARIANT_NAME, varname);
cv4.put(VARIANT_POSTED_DATE, posteddate);
return Database.insert(VARIANT_TABLE_NAME, null, cv4);
}
In the mainActivity,
for(int i = 0; i<build_emp_database.size();i++)
{
md.createVariantEntry(build_emp_database.get(i).get(0), build_emp_database.get(i).get(1), build_emp_database.get(i).get(2), build_emp_database.get(i).get(3), build_emp_database.get(i).get(4));
}
Also, just for 2800 entries, it is taking more than a minute, is there any way to speed them up?? I have several small databases, and have loaded them successfully. This is the only big database and its creating an issue while inserting. Please help where am i going wrong.
Don't try to run the insert operation on the UI thread, for starters.
I suggest you investigate using a content provider as a wrapper around your database. The ContentResolver object provides methods for doing operations in batch, and is in general a more robust way of working with databases. Use an IntentService to run the insert operation in the background.
If you do a lot insert operation, you need use the ContentProviderOperation to optimize your db operation. like these:
ArrayList<ContentProviderOperation> ops = new ArrayList<ContentProviderOperation>();
for(int i = 0; i<build_emp_database.size();i++) {
ContentProviderOperation.Builder builder =
ContentProviderOperation.newInsert(yourUrl);
builder.withValue(VARIANT_ID, varid)
.withValue(VARIANT_MAKE_ID, makeid)
...
ops.add(builder.build());
}
yourContentResolver.applyBatch(yourauthority, ops);