currently, I have to insert more than 10 billions data into my android at a time. However, the problem of out of memory will crash the program. The sqlite insert test is quite simple. Just using the for loop to produce the sql insert command and wrqpped by "begin" and "commit".
private ArrayList<String> sqlInsertCmd = new ArrayList<String>();
int QUERIES_TIMES = 10000000;
private void CreateInsertQueries()
{
Random localRandom = new Random();
int i = 0;
while (i < QUERIES_TIMES)
{
int j = localRandom.nextInt(100000);
sqlInsertCmd.add("insert into " + TABLE + " (a,b,c) values (" + i + "," +
j + ",'" + String.valueOf(j) + "')");
++i;
}
}
Then..
mDB.beginTransaction();
for (int i=0; i<this.QUERIES_TIMES; i++)
{
mDB.execSQL(sqlInsertCmd.get(i));
}
mDB.setTransactionSuccessful();
mDB.endTransaction();
Are there any ideas to avoid the out of memory?
Thanks you all, but the code above is just a simple example. In my program, it is more complicated. I must store something in a container(such as hashMap) and build the sql statement dynamically. Could I create 10 services and each service handle 1/10 jobs?
A few things:
See my answer here for general tips when doing bulk INSERTs.
There is no need to have a temporary container for your INSERT statements (in this case, the ArrayList<>). Simply use beginTransaction() and endTransaction() in a try...finally.
Utilize pre-compiled statements via SQLiteStatement vs building each INSERT statement as in your example. This is needless thrashing.
Quick and dirty example:
// note: untested code used for illustration!
private boolean bulkInsertData(SQLiteDatabase db, final String tableName) {
final int NUM_ROWS = 10000000;
Random random = new Random();
try {
SQLiteStatement insStmt = insStmt = db.compileStatement("INSERT INTO " + tableName + " (a, b, c) VALUES (?, ?, ?);");
db.beginTransaction();
try {
for(int i = 0; i < NUM_ROWS; i++) {
insStmt.bindLong(1, i);
insStmt.bindLong(2, random.nextInt(100000));
insStmt.bindString(3, String.valueOf(i));
insStmt.executeInsert(); // should really check value here!
}
db.setTransactionSuccessful();
} finally {
db.endTransaction();
}
} catch(SQLException se) {
return false;
}
return true;
}
Instead of creating an ArrayList which contains 10000000 number of sql queries (and might be the potential reason for OutOfMemory), you may simply start inserting records in the same loop where you are creating query strings. For example:
private void CreateInsertQueries(SQLiteDatabase mDB)
{
Random localRandom = new Random();
int i = 0;
String query;
try{
mDB.beginTransaction();
while (i < QUERIES_TIMES)
{
int j = localRandom.nextInt(100000);
query = "insert into " + TABLE + " (a,b,c) values (" + i + "," +
j + ",'" + String.valueOf(j) + "')";
++i;
mDB.execSQL(query);
}
mDB.setTransactionSuccessful();
}
catch (SQLException e) { e.printStackTrace(); }
finally{ mDB.endTransaction(); }
}
Doing so will help you to skip ArrayList which (i think) is the reason for OutOfMemory due to holding so so many records.
And make sure you do use External Memory to host your database file, otherwise not all phones have bigger internal memory.
Related
In my android app I have an SQLite database. with this structure:
PrivateList(idList INTEGER PRIMARY KEY, name TEXT, creationDate TEXT, active INTEGER, deactivationDate TEXT);
PrivateProduct (idProduct INTEGER PRIMARY KEY, description TEXT, quantity INTEGER, active INTEGER, additionDate TEXT);
List_Product(idList INTEGER NOT NULL, idProduct INTEGER NOT NULL, PRIMARY KEY (idList, idProduct), FOREIGN KEY(idList) REFERENCES PrivateList(idList), FOREIGN KEY(idProduct) REFERENCES PrivateProduct(idProduct));
I have an autogenerator list and elements using for to try the app:
localDB = new LocalDB(this, "localBD", null, 1);
SQLiteDatabase sqLiteDatabase = localDB.getWritableDatabase();
if (sqLiteDatabase != null){
for (int i = 0; i < 10; i++){
String a = "List" + i;
String b = "Product" + i;
Log.i("execfor", "INSERT INTO PrivateList (name, creationDate, active, deactivationDate) " + " VALUES ('" + a + "', CURRENT_TIMESTAMP, 1, null);");
sqLiteDatabase.execSQL("INSERT INTO PrivateList (name, creationDate, active, deactivationDate) " + " VALUES ('" + a + "', CURRENT_TIMESTAMP, 1, null);");
sqLiteDatabase.execSQL("INSERT INTO PrivateProduct (description, quantity, active, additionDate) " + " VALUES ('" + b + "', 3, 1, CURRENT_TIMESTAMP);");
//sqLiteDatabase.execSQL("INSERT INTO List_Product (idList, idProduct) values ");//
}
}
But I can`t find the way to get rowIds from each list and product to insert both, idlist and idproduct, into List_Product.
Thank you in advance.
The main change to facilitate grabbing the id's would be to swap from using execSQL to using insert as insert returns the id of the inserted row, execsql does not.
A little more on this here Using execSQL for INSERT operation in Android SQLite.
However, I'm not sure if you can pass CURRENT_TIMESTAMP via a ContentValues and it would result getting the current timestamp as opposed to just setting the value to the literal CURRENT_TIMESTAMP. You could use DEFAULT CURRENT_TIMEPSTAMP in the respective column definitions (as I have in the code below).
I'd suggest that you would not want a link between every list/product permutation (that would be 100 rows for you 10 List rows and 10 Product rows) as in real life you would probably not have such a scenario rather you'd have some links between the two. So in the code below I've randomly created links.
First some code from the Database Helper (for my convenience named SO45449914) for performing the inserts:-
public long insertListRow(String name,
int active) {
ContentValues cv = new ContentValues();
cv.put(LISTNAME_COL,name);
cv.put(LISTACTIVE_COL,active);
cv.put(LISTDEACTIVATIONDATE_COL,"");
return this.getWritableDatabase().insert(PRIVATELISTTABLE,null,cv);
}
public long insertProductRow(String description,int quantity, int active) {
ContentValues cv = new ContentValues();
cv.put(PRODUCTDESCRIPTION_COL,description);
cv.put(PRODUCTQUANTITY_COL,quantity);
cv.put(PRODUCTACTIVE_COL,active);
return this.getWritableDatabase().insert(PRIVATEPRODUCTTABLE,null,cv);
}
public void insertListProductLink(long listid, long productid) {
ContentValues cv = new ContentValues();
cv.put(LISTPRODUCTLIST_COL,listid);
cv.put(LISTPRODUCTPRODUCT_COL,productid);
if (this.getWritableDatabase().insertOrThrow(LISTPRODUCTTABLE,null,cv) <0) {
//handle failed insert
}
}
Notes
- I've used class variables for all columns names.
- Columns that have the current time stamp get this via the default, so there is no need to have a cv.put for those columns.
In the activity is the following code :-
void doSO45449914() {
SO45449914 dbhelper = new SO45449914(this);
int loopcount = 10;
long[] listids = new long[loopcount];
long[] productids = new long [loopcount];
for (int i=0; i < 10; i++) {
listids[i] = dbhelper.insertListRow("a" + i,1);
productids[i] = dbhelper.insertProductRow("b" + i,3,1);
}
Cursor csra = dbhelper.getWritableDatabase().query(SO45449914.PRIVATELISTTABLE,
null,null,null,null,null,null
);
Cursor csrb = dbhelper.getWritableDatabase().query(SO45449914.PRIVATEPRODUCTTABLE,
null,null,null,null,null,null
);
Log.d("SO45449914","Number of rows in LIST TABLE = " + csra.getCount());
Log.d("SO45449914","Number of rows in PRODUCTS TABLE = " + csrb.getCount());
for (long aid: listids) {
Log.d("SO45449914","LIST ID from store = " + Long.toString(aid));
}
for (long bid: productids) {
Log.d("SO45449914","PRODUCT ID from store = " + Long.toString(bid));
}
for (long lid: listids) {
for (long prdid: productids) {
if ((Math.random() * 100) > 60) {
dbhelper.insertListProductLink(lid,prdid);
Log.d("SO45449914",
"Adding link between List id(" +
Long.toString(lid) +
") and product id(" +
Long.toString(prdid) +
")"
);
}
}
}
csra.close();
csrb.close();
}
Exlapnation
The first few lines prepare long arrays based upon the number of Lists and products to be created (same number of both). Integer loopcount determines how many.
The first loop, inserts Lists and Products which use the insert method storing the returned id in the respective array element.
Two Cursors are then created for obtaining row counts, which are then written to the log. The id's as stored in the arrays are output to the log.
Two nested loops are then invoked with Products being the inner (not that it matters) and randomly (about 40% of the time) a row will be inserted into the link table. I've assumed random but you always easily adjust the algorithm to follow a pattern. It's if ((Math.random() * 100) > 60) { that determines whether or not to insert a link.
The two Cursors are then closed.
Results
Here are screen shots of the resultant tables :-
PrivateList Table
PrivateProduct Table
List_Product Table
..... (44 rows in the List_Product table)
Well, this is what I did. Despite of the fact that there is a way do the same without so many rows in List_Product table; I'd like to understand the way. (Also I had problem in the for so it didnt do what I wanted exactly).
localDB = new LocalDB(this, "localBD", null, 1);
SQLiteDatabase sqLiteDatabase = localDB.getWritableDatabase();
if (sqLiteDatabase != null){
long idList;
long idProduct;
for (int i = 0; i < 10; i++){
String a = "List" + i;
String b = "Product" + i;
ContentValues contentValuesList = new ContentValues();
contentValuesList.put("name", a);
contentValuesList.put("active", 1);
contentValuesList.put("creationDate", "CreationDate");
contentValuesList.put("deactivationDate", "");
idList = sqLiteDatabase.insert("PrivateList", null, contentValuesList);
for (int j = 0; j < 10; j++){
ContentValues contentValuesProduct = new ContentValues();
contentValuesProduct.put("description", b);
contentValuesProduct.put("active", 1);
contentValuesProduct.put("quantity", 1);
contentValuesProduct.put("additionDate", "additionDdate");
idProduct = sqLiteDatabase.insert("PrivateProduct", null, contentValuesProduct);
ContentValues contentValuesListProduct = new ContentValues();
contentValuesListProduct.put("idList", idList);
contentValuesListProduct.put("idProduct", idProduct);
sqLiteDatabase.insert("List_Product", null, contentValuesListProduct);
}
I know it could be more efficient, but that it doesn't matter now.
This is the result in the database:
PrivateList:
with 10 rows
PrivateProduct:
with 100 rows.
List_Product:
The problem was that I didn't know the existance of sqlLiteDatabase.insert(...) method.
Thanks you all.
I created a simple chat application and now I want to store the chat messages in Sqlite to provide chat history to users. But I don't have idea how to create database and tables, and inserting data at runtime as well.
I want to store _sender, _receiver and _msg for now. I tried this post's answer but there was syntax error in
this.insertStmt = this.myDataBase.compileStatement(INSERT);
Even I do not know if it is right approach for my case as in chat history it seems the database is a bit complex.
I tried with this code:
public void createDynamicDatabase(Context context, String tableName, ArrayList<String> title)
{
Log.i("INSIDE createLoginDatabase() Method","*************creatLoginDatabase*********");
try
{
int i;
String queryString;
// Opens database in writable mode.
myDataBase = context.openOrCreateDatabase("Db",Context.MODE_WORLD_WRITEABLE, null);
//System.out.println("Table Name : "+tableName.get(0));
queryString = title.get(0)+" VARCHAR(30),";
Log.d("**createDynamicDatabase", "in oncreate");
for(i = 1;i < title.size() - 1; i++)
{
queryString += title.get(i);
queryString +=" VARCHAR(30)";
queryString +=",";
}
queryString+= title.get(i) +" VARCHAR(30)";
queryString = "CREATE TABLE IF NOT EXISTS " + tableName + "("+queryString+");";
System.out.println("Create Table Stmt : "+ queryString);
myDataBase.execSQL(queryString);
}
catch (SQLException ex)
{
Log.i("CreateDB Exception ",ex.getMessage());
}
}
public void insert(Context context, ArrayList<String> array_vals, ArrayList<String> title, String TABLE_NAME)
{
Log.d("Inside Insert","Insertion starts for table name: "+TABLE_NAME);
// Opens database in writable mode.
myDataBase = context.openOrCreateDatabase("Db",Context.MODE_WORLD_WRITEABLE, null);
String titleString = null;
String markString = null;
int i;
titleString = title.get(0)+",";
markString = "?,";
Log.d("**createDynamicDatabase", "in oncreate");
for(i = 1;i < title.size() - 1; i++)
{
titleString += title.get(i);
titleString +=",";
markString += "?,";
}
titleString+= title.get(i);
markString += "?";
//System.out.println("Title String: "+titleString);
//System.out.println("Mark String: "+markString);
INSERT="insert into "+ TABLE_NAME + "("+titleString+")"+ "values" +"("+markString+")";
System.out.println("Insert statement: "+INSERT);
//System.out.println("Array size iiiiii::: "+array_vals.size());
//this.insertStmt = this.myDataBase.compileStatement(INSERT);
int s=0;
while(s<array_vals.size())
{
System.out.println("Size of array1"+array_vals.size());
//System.out.println("Size of array"+title.size());
int j=1;
this.insertStmt = this.myDataBase.compileStatement(INSERT);
for(int k =0;k< title.size();k++)
{
//System.out.println("Value of column "+title+" is "+array_vals.get(k+s));
//System.out.println("PRINT S:"+array_vals.get(k+s));
System.out.println("BindString: insertStmt.bindString("+j+","+ array_vals.get(k+s)+")");
insertStmt.bindString(j, array_vals.get(k+s));
j++;
}
s+=title.size();
}
insertStmt.executeInsert();
}
I created database in onCreate() and than insert the data when user sends msg and receives.but getting this error.;
android.database.sqlite.SQLiteException: near "08104710280": syntax error (code 1): , while compiling: insert into 08104710280(sender,receiver,msg)values(?,?,?)
at this line
this.insertStmt = this.myDataBase.compileStatement(INSERT);
Look at SQLiteOpenHelper:
http://developer.android.com/reference/android/database/sqlite/SQLiteOpenHelper.html
Coding example:
http://www.vogella.com/tutorials/AndroidSQLite/article.html
The preferred and much easier way in android is to use Room library, which greatly reduces boilerplate and works smoothly with other architechture components like LiveData, handles background queries for you and provides compile time checks just for a few lines of code.
I have a simple insert SQLite statement :
private final String INSERT_FEED = "insert into "
+ TABLE_FEEDS
+ "([id],[first_name], [last_name], [image], [content]) "
+ "values (?,?,?,?,?)";
Which i used in a method :
public long insertFeed(FeedsModel feed) {
this.insertStmt = db.compileStatement(INSERT_FEED);
this.insertStmt.bindString(1, feed.getId() + "");
this.insertStmt.bindString(2, feed.getFirstName());
this.insertStmt.bindString(3, feed.getLastName());
this.insertStmt.bindString(4, feed.getProfileImage());
this.insertStmt.bindString(5, feed.getContentDetails());
long result = this.insertStmt.executeInsert();
return result;
}
Later, i used that method in my activity :
for(int i = 0; i < feedsAdapter.getCount() - 1; i++)
{
helper.insertFeed(feedsAdapter.getItem(i));
}
The problem is, the table always have only 1 row because the new data will overwrite the existing row, instead of inserted in a new row.
Your current for loop is inserting all the FeedsModels other than the last one. If you only have one model in the list it would explain why you are not seeing any new inserts into the table, instead do:
for (FeedsModel feedsModel : feedsAdapter) {
helper.insertFeed(feedsModel);
}
I am new in Android Development.
Currently, working with SQLite Database in Android.
My problem is that I have a large amount of data which I have to store in SQLite Database in Android.
There are 2 tables: One having 14927 rows and the other one has 9903 rows.
Currently the database in sql. And I have copy these data in excel sheet but don't understand how can I import these data in SQLite Database.
I go through the following link:
Inserting large amount of data into android sqlite database?
Here, the solution is posted regarding CSV File. But want to know other ways of doing this.
Please let me know what is the best way to import such a large data in Android.
Please help me. Thanks in advance.
Do Like This
SQLiteDatabase sd;
sd.beginTransaction();
for (int i = 0; i < data.size(); i++) {
ContentValues values = new ContentValues();
values.put(DBAdapter.Column1, "HP");
values.put(DBAdapter.Column2, "qw");
values.put(DBAdapter.Column3, "5280");
values.put(DBAdapter.Column4, "345, 546");
sd.insert(DBAdapter.TABLE, null, values);
sd.insertWithOnConflict(tableName, null, values, SQLiteDatabase.CONFLICT_IGNORE);
}
sd.setTransactionSuccessful();
sd.endTransaction();
Try this
SQLiteDatabase db = Your_DATABASE;
db.beginTransaction();
db.openDatabase();
for (int i = 0; i < array.size(); i++)
{
String sql = ( "INSERT INTO " + Table_NAME
+ "(" + COLUMN_1 + ","
+ COLUMN_2 + ","
+ COLUMN_3 + ","
+ COLUMN_4 + ","
+ ") values (?,?,?,?)");
SQLiteStatement insert = db.compileStatement(sql);
}
db.setTransactionSuccessful();
db.endTransaction();
db.closeDatabase();
Use DatabaseUtils.InsertHelper. In this article you will find an example of how to use it and other ways to speed up insertions. The example is as follows:
import android.database.DatabaseUtils.InsertHelper;
//...
private class DatabaseHelper extends SQLiteOpenHelper {
#Override
public void onCreate(SQLiteDatabase db) {
// Create a single InsertHelper to handle this set of insertions.
InsertHelper ih = new InsertHelper(db, "TableName");
// Get the numeric indexes for each of the columns that we're updating
final int greekColumn = ih.getColumnIndex("Greek");
final int ionicColumn = ih.getColumnIndex("Ionic");
//...
final int romanColumn = ih.getColumnIndex("Roman");
try {
while (moreRowsToInsert) {
// ... Create the data for this row (not shown) ...
// Get the InsertHelper ready to insert a single row
ih.prepareForInsert();
// Add the data for each column
ih.bind(greekColumn, greekData);
ih.bind(ionicColumn, ionicData);
//...
ih.bind(romanColumn, romanData);
// Insert the row into the database.
ih.execute();
}
}
finally {
ih.close();
}
}
//...
}
I am planning to write a free version and a full version of a software. I want the information stored by the free version of the software to be accessible by the full version also (I don't want to use Content Providers). And I also want to make sure this data is not lost when the software is updated. How do I achieve this?
You need to implement an intelligent way of onUpgrade for your sqlite helpers.
You should always have the new table creation query at hand, and use that for upgrade and transfer any existing data. Note: that the onUpgrade methods runs once for your sqlite helper object and you need to handle all the tables in it.
So what is recommended onUpgrade:
beginTransaction
run a table creation with if not exists (we are doing an upgrade, so the table might not exists yet, it will fail alter and drop)
put in a list the existing columns List<String> columns = DBUtils.GetColumns(db, TableName);
backup table (ALTER table " + TableName + " RENAME TO 'temp_" + TableName)
create new table (the newest table creation schema)
get the intersection with the new columns, this time columns taken from the upgraded table (columns.retainAll(DBUtils.GetColumns(db, TableName));)
restore data (String cols = StringUtils.join(columns, ",");
db.execSQL(String.format(
"INSERT INTO %s (%s) SELECT %s from temp_%s",
TableName, cols, cols, TableName));
)
remove backup table (DROP table 'temp_" + TableName)
setTransactionSuccessful
(This doesn't handle table downgrade, if you rename a column, you don't get the existing data transfered as the column names do not match).
.
public static List<String> GetColumns(SQLiteDatabase db, String tableName) {
List<String> ar = null;
Cursor c = null;
try {
c = db.rawQuery("select * from " + tableName + " limit 1", null);
if (c != null) {
ar = new ArrayList<String>(Arrays.asList(c.getColumnNames()));
}
} catch (Exception e) {
Log.v(tableName, e.getMessage(), e);
e.printStackTrace();
} finally {
if (c != null)
c.close();
}
return ar;
}
public static String join(List<String> list, String delim) {
StringBuilder buf = new StringBuilder();
int num = list.size();
for (int i = 0; i < num; i++) {
if (i != 0)
buf.append(delim);
buf.append((String) list.get(i));
}
return buf.toString();
}