To insert large amount of data in android SQLite Database - android

I am new in Android Development.
Currently, working with SQLite Database in Android.
My problem is that I have a large amount of data which I have to store in SQLite Database in Android.
There are 2 tables: One having 14927 rows and the other one has 9903 rows.
Currently the database in sql. And I have copy these data in excel sheet but don't understand how can I import these data in SQLite Database.
I go through the following link:
Inserting large amount of data into android sqlite database?
Here, the solution is posted regarding CSV File. But want to know other ways of doing this.
Please let me know what is the best way to import such a large data in Android.
Please help me. Thanks in advance.

Do Like This
SQLiteDatabase sd;
sd.beginTransaction();
for (int i = 0; i < data.size(); i++) {
ContentValues values = new ContentValues();
values.put(DBAdapter.Column1, "HP");
values.put(DBAdapter.Column2, "qw");
values.put(DBAdapter.Column3, "5280");
values.put(DBAdapter.Column4, "345, 546");
sd.insert(DBAdapter.TABLE, null, values);
sd.insertWithOnConflict(tableName, null, values, SQLiteDatabase.CONFLICT_IGNORE);
}
sd.setTransactionSuccessful();
sd.endTransaction();

Try this
SQLiteDatabase db = Your_DATABASE;
db.beginTransaction();
db.openDatabase();
for (int i = 0; i < array.size(); i++)
{
String sql = ( "INSERT INTO " + Table_NAME
+ "(" + COLUMN_1 + ","
+ COLUMN_2 + ","
+ COLUMN_3 + ","
+ COLUMN_4 + ","
+ ") values (?,?,?,?)");
SQLiteStatement insert = db.compileStatement(sql);
}
db.setTransactionSuccessful();
db.endTransaction();
db.closeDatabase();

Use DatabaseUtils.InsertHelper. In this article you will find an example of how to use it and other ways to speed up insertions. The example is as follows:
import android.database.DatabaseUtils.InsertHelper;
//...
private class DatabaseHelper extends SQLiteOpenHelper {
#Override
public void onCreate(SQLiteDatabase db) {
// Create a single InsertHelper to handle this set of insertions.
InsertHelper ih = new InsertHelper(db, "TableName");
// Get the numeric indexes for each of the columns that we're updating
final int greekColumn = ih.getColumnIndex("Greek");
final int ionicColumn = ih.getColumnIndex("Ionic");
//...
final int romanColumn = ih.getColumnIndex("Roman");
try {
while (moreRowsToInsert) {
// ... Create the data for this row (not shown) ...
// Get the InsertHelper ready to insert a single row
ih.prepareForInsert();
// Add the data for each column
ih.bind(greekColumn, greekData);
ih.bind(ionicColumn, ionicData);
//...
ih.bind(romanColumn, romanData);
// Insert the row into the database.
ih.execute();
}
}
finally {
ih.close();
}
}
//...
}

Related

How to insert to into many to many relationships. SQLite

In my android app I have an SQLite database. with this structure:
PrivateList(idList INTEGER PRIMARY KEY, name TEXT, creationDate TEXT, active INTEGER, deactivationDate TEXT);
PrivateProduct (idProduct INTEGER PRIMARY KEY, description TEXT, quantity INTEGER, active INTEGER, additionDate TEXT);
List_Product(idList INTEGER NOT NULL, idProduct INTEGER NOT NULL, PRIMARY KEY (idList, idProduct), FOREIGN KEY(idList) REFERENCES PrivateList(idList), FOREIGN KEY(idProduct) REFERENCES PrivateProduct(idProduct));
I have an autogenerator list and elements using for to try the app:
localDB = new LocalDB(this, "localBD", null, 1);
SQLiteDatabase sqLiteDatabase = localDB.getWritableDatabase();
if (sqLiteDatabase != null){
for (int i = 0; i < 10; i++){
String a = "List" + i;
String b = "Product" + i;
Log.i("execfor", "INSERT INTO PrivateList (name, creationDate, active, deactivationDate) " + " VALUES ('" + a + "', CURRENT_TIMESTAMP, 1, null);");
sqLiteDatabase.execSQL("INSERT INTO PrivateList (name, creationDate, active, deactivationDate) " + " VALUES ('" + a + "', CURRENT_TIMESTAMP, 1, null);");
sqLiteDatabase.execSQL("INSERT INTO PrivateProduct (description, quantity, active, additionDate) " + " VALUES ('" + b + "', 3, 1, CURRENT_TIMESTAMP);");
//sqLiteDatabase.execSQL("INSERT INTO List_Product (idList, idProduct) values ");//
}
}
But I can`t find the way to get rowIds from each list and product to insert both, idlist and idproduct, into List_Product.
Thank you in advance.
The main change to facilitate grabbing the id's would be to swap from using execSQL to using insert as insert returns the id of the inserted row, execsql does not.
A little more on this here Using execSQL for INSERT operation in Android SQLite.
However, I'm not sure if you can pass CURRENT_TIMESTAMP via a ContentValues and it would result getting the current timestamp as opposed to just setting the value to the literal CURRENT_TIMESTAMP. You could use DEFAULT CURRENT_TIMEPSTAMP in the respective column definitions (as I have in the code below).
I'd suggest that you would not want a link between every list/product permutation (that would be 100 rows for you 10 List rows and 10 Product rows) as in real life you would probably not have such a scenario rather you'd have some links between the two. So in the code below I've randomly created links.
First some code from the Database Helper (for my convenience named SO45449914) for performing the inserts:-
public long insertListRow(String name,
int active) {
ContentValues cv = new ContentValues();
cv.put(LISTNAME_COL,name);
cv.put(LISTACTIVE_COL,active);
cv.put(LISTDEACTIVATIONDATE_COL,"");
return this.getWritableDatabase().insert(PRIVATELISTTABLE,null,cv);
}
public long insertProductRow(String description,int quantity, int active) {
ContentValues cv = new ContentValues();
cv.put(PRODUCTDESCRIPTION_COL,description);
cv.put(PRODUCTQUANTITY_COL,quantity);
cv.put(PRODUCTACTIVE_COL,active);
return this.getWritableDatabase().insert(PRIVATEPRODUCTTABLE,null,cv);
}
public void insertListProductLink(long listid, long productid) {
ContentValues cv = new ContentValues();
cv.put(LISTPRODUCTLIST_COL,listid);
cv.put(LISTPRODUCTPRODUCT_COL,productid);
if (this.getWritableDatabase().insertOrThrow(LISTPRODUCTTABLE,null,cv) <0) {
//handle failed insert
}
}
Notes
- I've used class variables for all columns names.
- Columns that have the current time stamp get this via the default, so there is no need to have a cv.put for those columns.
In the activity is the following code :-
void doSO45449914() {
SO45449914 dbhelper = new SO45449914(this);
int loopcount = 10;
long[] listids = new long[loopcount];
long[] productids = new long [loopcount];
for (int i=0; i < 10; i++) {
listids[i] = dbhelper.insertListRow("a" + i,1);
productids[i] = dbhelper.insertProductRow("b" + i,3,1);
}
Cursor csra = dbhelper.getWritableDatabase().query(SO45449914.PRIVATELISTTABLE,
null,null,null,null,null,null
);
Cursor csrb = dbhelper.getWritableDatabase().query(SO45449914.PRIVATEPRODUCTTABLE,
null,null,null,null,null,null
);
Log.d("SO45449914","Number of rows in LIST TABLE = " + csra.getCount());
Log.d("SO45449914","Number of rows in PRODUCTS TABLE = " + csrb.getCount());
for (long aid: listids) {
Log.d("SO45449914","LIST ID from store = " + Long.toString(aid));
}
for (long bid: productids) {
Log.d("SO45449914","PRODUCT ID from store = " + Long.toString(bid));
}
for (long lid: listids) {
for (long prdid: productids) {
if ((Math.random() * 100) > 60) {
dbhelper.insertListProductLink(lid,prdid);
Log.d("SO45449914",
"Adding link between List id(" +
Long.toString(lid) +
") and product id(" +
Long.toString(prdid) +
")"
);
}
}
}
csra.close();
csrb.close();
}
Exlapnation
The first few lines prepare long arrays based upon the number of Lists and products to be created (same number of both). Integer loopcount determines how many.
The first loop, inserts Lists and Products which use the insert method storing the returned id in the respective array element.
Two Cursors are then created for obtaining row counts, which are then written to the log. The id's as stored in the arrays are output to the log.
Two nested loops are then invoked with Products being the inner (not that it matters) and randomly (about 40% of the time) a row will be inserted into the link table. I've assumed random but you always easily adjust the algorithm to follow a pattern. It's if ((Math.random() * 100) > 60) { that determines whether or not to insert a link.
The two Cursors are then closed.
Results
Here are screen shots of the resultant tables :-
PrivateList Table
PrivateProduct Table
List_Product Table
..... (44 rows in the List_Product table)
Well, this is what I did. Despite of the fact that there is a way do the same without so many rows in List_Product table; I'd like to understand the way. (Also I had problem in the for so it didnt do what I wanted exactly).
localDB = new LocalDB(this, "localBD", null, 1);
SQLiteDatabase sqLiteDatabase = localDB.getWritableDatabase();
if (sqLiteDatabase != null){
long idList;
long idProduct;
for (int i = 0; i < 10; i++){
String a = "List" + i;
String b = "Product" + i;
ContentValues contentValuesList = new ContentValues();
contentValuesList.put("name", a);
contentValuesList.put("active", 1);
contentValuesList.put("creationDate", "CreationDate");
contentValuesList.put("deactivationDate", "");
idList = sqLiteDatabase.insert("PrivateList", null, contentValuesList);
for (int j = 0; j < 10; j++){
ContentValues contentValuesProduct = new ContentValues();
contentValuesProduct.put("description", b);
contentValuesProduct.put("active", 1);
contentValuesProduct.put("quantity", 1);
contentValuesProduct.put("additionDate", "additionDdate");
idProduct = sqLiteDatabase.insert("PrivateProduct", null, contentValuesProduct);
ContentValues contentValuesListProduct = new ContentValues();
contentValuesListProduct.put("idList", idList);
contentValuesListProduct.put("idProduct", idProduct);
sqLiteDatabase.insert("List_Product", null, contentValuesListProduct);
}
I know it could be more efficient, but that it doesn't matter now.
This is the result in the database:
PrivateList:
with 10 rows
PrivateProduct:
with 100 rows.
List_Product:
The problem was that I didn't know the existance of sqlLiteDatabase.insert(...) method.
Thanks you all.

How to efficiently query sqlite database multiple times on Android

For my application, I need to query a sqlite database around 40-50 times. I am sure that the code I wrote is very inefficient. Unfortunately, I cannot find many examples online that involves querying the database many times.
String[] entryValArray = new String[indicesList.size()];
DBHelper dbHelper = new DBHelper(MainActivity.context);
SQLiteDatabase db = dbHelper.getReadableDatabase();
for (int i = 0; i < indicesList.size(); i++) {
int moddedIndex = Integer.parseInt(indicesList.get(i), 16) % DBHelper.numEntries;
String queryStr = "select * from " + DBHelper.TBL_NAME + " where " + DBHelper.IDStr +
" = " + Integer.toString(moddedIndex);
Cursor cursor = db.rawQuery(queryStr, null);
if (cursor.moveToFirst())
entryValArray[i] = cursor.getString(1);
cursor.close();
}
Basically, I am taking a list of strings, converting them to hex values, and then modding the value to get an index into a sqlite database. This is for a password generator application.
Is there a better way to do this, especially regarding creating a cursor and then closing it in every iteration.
First of all you have to change your query string as you need only one column value but you are using
Select *
instead of
Select yourColumn
. Secondly if your indices list size is not very large you can use
IN(values ) function of db instead of
" where " + DBHelper.IDStr +" = " + Integer.toString(moddedIndex);
this will return the result in only one query you don't have to run a whole loop.

Bulk Update Database

I've been looking on this site for a while but have not found the answer. I am trying to do a bulk update on data that I know is already in the table. I have one column that needs to be set when a certain condition comes back for the row ID. Here is the single method but I want to make this more efficient and do it as a bulk. Our database is not in a Provider so I just using a Helper class.
public void markUnavailable(int myId) {
SQLiteDatabase db = this.getWritableDatabase();
String sql = "UPDATE " + MYTABLE + " SET " + Col.IS_AVAILABLE + "= 0"+ " WHERE " + Col.MY_ID + "=" + myId;
db.execSQL(sql);
db.close();
}
I would like to pass in an array of myIds to do the bulk Update. I can't do a Insert or Replace statement because I don't have access to all the column data and don't want to pass this through due to too many codes changes.
public void markUnavailable(int[] myId) {
// ????
/// need some way to loop through and update in bulk
}
Try UPDATE tablename SET column=0 WHERE ID IN (...), where ... is a comma-delimited list of ID values.
I'm not an Android developer, but according to good database practices, you should:
public void markUnavailable(int[] myId) {
SQLiteDatabase db = this.getWritableDatabase();
db.beginTransaction();
SQLiteStatement upd=db.compileStatement("UPDATE "+MYTABLE+" SET "+Col.IS_AVAILABLE+"=0 WHERE "+Col.MY_ID+"=?";
for (int i = 0; i < myId.length; i++) {
upd.bindLong(1, myId[i]);
upd.execute();
}
db.endTransaction();
}
Android has SQLiteDatabase.update would be very usefull in this case, but String [] whereArgs would not deal well with your int[] myId.
The fastest way to do a bulk update would be to do it as a single transaction,by using begin and end transactions. Also if the size of the database is large it will be a good idea to make myID as the primary key of the table as it will significantly increase the speed of the speed in fetching the rows for update when the WHERE clause is used.[It is said that indexing can reduce the speed of update and insert but when the where clause is used,indexing has always increased my speed by huge margins.
public void markUnavailable(int[] myId) {
SQLiteDatabase db = this.getWritableDatabase();
db.beginTransaction();
SQLiteStatement upd=db.compileStatement("UPDATE "+MYTABLE+" SET "+Col.IS_AVAILABLE+"=0 WHERE "+Col.MY_ID+"=?");
for (int i = 0; i < myId.length; i++) {
upd.clearBindings();
upd.bindLong(1, myId[i]); // this matches the first "?"
upd.execute();
}
db.setTransactionSucessful();
db.endTransaction();
}

Retaining persistent information on different versions of the software

I am planning to write a free version and a full version of a software. I want the information stored by the free version of the software to be accessible by the full version also (I don't want to use Content Providers). And I also want to make sure this data is not lost when the software is updated. How do I achieve this?
You need to implement an intelligent way of onUpgrade for your sqlite helpers.
You should always have the new table creation query at hand, and use that for upgrade and transfer any existing data. Note: that the onUpgrade methods runs once for your sqlite helper object and you need to handle all the tables in it.
So what is recommended onUpgrade:
beginTransaction
run a table creation with if not exists (we are doing an upgrade, so the table might not exists yet, it will fail alter and drop)
put in a list the existing columns List<String> columns = DBUtils.GetColumns(db, TableName);
backup table (ALTER table " + TableName + " RENAME TO 'temp_" + TableName)
create new table (the newest table creation schema)
get the intersection with the new columns, this time columns taken from the upgraded table (columns.retainAll(DBUtils.GetColumns(db, TableName));)
restore data (String cols = StringUtils.join(columns, ",");
db.execSQL(String.format(
"INSERT INTO %s (%s) SELECT %s from temp_%s",
TableName, cols, cols, TableName));
)
remove backup table (DROP table 'temp_" + TableName)
setTransactionSuccessful
(This doesn't handle table downgrade, if you rename a column, you don't get the existing data transfered as the column names do not match).
.
public static List<String> GetColumns(SQLiteDatabase db, String tableName) {
List<String> ar = null;
Cursor c = null;
try {
c = db.rawQuery("select * from " + tableName + " limit 1", null);
if (c != null) {
ar = new ArrayList<String>(Arrays.asList(c.getColumnNames()));
}
} catch (Exception e) {
Log.v(tableName, e.getMessage(), e);
e.printStackTrace();
} finally {
if (c != null)
c.close();
}
return ar;
}
public static String join(List<String> list, String delim) {
StringBuilder buf = new StringBuilder();
int num = list.size();
for (int i = 0; i < num; i++) {
if (i != 0)
buf.append(delim);
buf.append((String) list.get(i));
}
return buf.toString();
}

SQLiteOpenHelper onUpgrade() Confusion Android

I am doing my first app with a database and I am having a little trouble understanding the onUpgrade function. My database has a table with an items and a favorite column so that the user can favorite an item. Most implementations I see simply drop the table and reconstruct it but I don't want to do this. I want to be able to add more items to the table.
When the app is upgraded through the android marketplace does the database know its version number? So could I increment the version number in the code and then export it to the marketplace and when the user boots up the upgraded version for the first time then onUpgrade will be called?
If this is the case my onUpgrade would simply pull from a file and add the database items in. Is this a standard way of doing things or is there a better way of handling this in Android. I am trying to stay as standard as possible.
Thanks
Ok, before you run into bigger problems you should know that SQLite is limited on the ALTER TABLE command, it allows add and rename only no remove/drop which is done with recreation of the table.
You should always have the new table creation query at hand, and use that for upgrade and transfer any existing data. Note: that the onUpgrade methods runs one for your sqlite helper object and you need to handle all the tables in it.
So what is recommended onUpgrade:
beginTransaction
run a table creation with if not exists (we are doing an upgrade, so the table might not exists yet, it will fail alter and drop)
put in a list the existing columns List<String> columns = DBUtils.GetColumns(db, TableName);
backup table (ALTER table " + TableName + " RENAME TO 'temp_" + TableName)
create new table (the newest table creation schema)
get the intersection with the new columns, this time columns taken from the upgraded table (columns.retainAll(DBUtils.GetColumns(db, TableName));)
restore data (String cols = StringUtils.join(columns, ",");
db.execSQL(String.format(
"INSERT INTO %s (%s) SELECT %s from temp_%s",
TableName, cols, cols, TableName));
)
remove backup table (DROP table 'temp_" + TableName)
setTransactionSuccessful
(This doesn't handle table downgrade, if you rename a column, you don't get the existing data transfered as the column names do not match).
.
public static List<String> GetColumns(SQLiteDatabase db, String tableName) {
List<String> ar = null;
Cursor c = null;
try {
c = db.rawQuery("select * from " + tableName + " limit 1", null);
if (c != null) {
ar = new ArrayList<String>(Arrays.asList(c.getColumnNames()));
}
} catch (Exception e) {
Log.v(tableName, e.getMessage(), e);
e.printStackTrace();
} finally {
if (c != null)
c.close();
}
return ar;
}
public static String join(List<String> list, String delim) {
StringBuilder buf = new StringBuilder();
int num = list.size();
for (int i = 0; i < num; i++) {
if (i != 0)
buf.append(delim);
buf.append((String) list.get(i));
}
return buf.toString();
}
Next to Pentium10's excellent answer, here are some good examples from living code:
Android AOSP: com.android.providers.calendar.CalendarDatabaseHelper.java
Android AOSP: com.android.browser.BrowserProvider.java
OpenIntents Notepad: org.openintents.notepad.NotePadProvider.java
Thank you for clarifying that onUpgrade() will not support Remove/Drop statements #Pentium 10
For those of you who would like to know the exact moment when onUpgrade() gets called, it is during a call to either getReadableDatabase() or getWriteableDatabase().
To those who are not clear how it ensure it gets triggered...the answer is: It is triggered when the database version provided to the constructor of SqLiteOpenHelper is updated. Here is a example
public class dbSchemaHelper extends SQLiteOpenHelper {
private String sql;
private final String D_TAG = "FundExpense";
//update this to get onUpgrade() method of sqliteopenhelper class called
static final int DB_VERSION = 2;
static final String DB_NAME = "fundExpenseManager";
public dbSchemaHelper(Context context) {
super(context, DB_NAME, null, DB_VERSION);
// TODO Auto-generated constructor stub
}
now to...onUpgrade()
#Override
public void onUpgrade(SQLiteDatabase arg0, int arg1, int arg2) {
sql = "ALTER TABLE " + fundExpenseSchema.Expense.TABLE_NAME + " ADD COLUMN " + fundExpenseSchema.Expense.FUNDID + " INTEGER";
arg0.execSQL(sql);
}
I've been using the solution proposed by #Pentium10 for a long time but today i had a problem, after doing alter table, getColumns from the original table still returns the same columns (in the new version of the db the table suffer mayor structure changes, some columns added some others), really i don't know why select statement does not reflect the structure changes, more over before creating my table again, select statement still returns the columns! When the table is not re-created yet!
So i manage solving this issue updating getColumns method using pragma table_info, like this:
/**
* Get a list of column base_dictionary for the selected table
*
* #param db
* Database that contains the table
* #param tableName
* Table name to be used
* #return A List of column name
*/
public static List<String> getColumns(SQLiteDatabase db, String tableName) {
List<String> ar = null;
Cursor c = null;
try {
c = db.rawQuery("pragma table_info(" + tableName + ")", null);
ar = new ArrayList<String>();
if (c != null && c.moveToFirst()) {
do {
ar.add(c.getString(c.getColumnIndexOrThrow("name")));
} while (c.moveToNext());
c.close();
}
} catch (Exception e) {
Log.v(tableName, e.getMessage(), e);
e.printStackTrace();
} finally {
if (c != null) c.close();
}
return ar;
}

Categories

Resources