Develop Reference

Android Pre Populated Database - Adding New Pre Populated Rows After Publication

I'm having trouble with a pre Populated database in android. Not the usual problems though. I've got the database working just fine.
My problem comes with adding new data after the app has been published.
I spent a lot of time with onupgrade method but then it dawned on me that my problem is elsewhere. Once I've added new lines to my database in my assets folder, how do I get these added to the database that was copied to my data/data folder.....
My database is where I store my level information for a game, the last column in the table is a flag to mark the level completed so I can't lose this information.

You could add some sql patch files, and then read them to upgrade your database.
I used it simply with the FileHelper static class I copied from Danny Remington's post on SO and then do :
try {
InputStream in = mgr.open(assetFile);
String[] statements = FileHelper.parseSqlFile(in);
dao.mDb.execSQL("BEGIN TRANSACTION;");
/*dao.mDb is a reference to the SQLiteDatabase from my dao*/
for (String statement : statements) {
dao.mDb.execSQL(statement);
}
dao.mDb.execSQL("COMMIT;");
in.close();
} catch (IOException e) {
//
}

SQLite file copied from Assets folder by Android SQLiteOpenHelper corrupted?

I used sqlite3 on my Mac to create a small database file (surnamed ".db") containing one table, plus the android_metadata table containing "en_US". I pasted the file into the assets folder of my Android Studio project. At runtime, the onCreate method of my SQLiteOpenHelper copies the file into /data/data/projectname/databases/filename.db on my Genymotion emulator (a Samsung Galaxy S5). But when I try to read the table in filename.db with the Cursor provided by the getReadableDatabase method of the SQLiteOpenHelper, I get the LogCat error "E/SQLiteLog﹕ (1) no such table: tablename". The same thing happens on my el-cheapo Azpen A727 Android tablet.
The filename.db file copied into the /data/data/projectname/databases folder on the emulator contains the same number of bytes (5120) as the original filename.db I created on my Mac with sqlite3. When I compare the two .db files with the Unix octal dump command "od -c", I found that they are identical for the first 16 bytes, and differ for the first time at the 17th byte. (The original file has octal 004 as its seventeenth byte, and the copy has 020.) Should the two files have had identical contents? When I try to examine /data/data/projectname/databases/filename.db with sqlite3 .dump, I get "ERROR: (11) database disk image is malformed".
The getReadableDatabase method of my SQLiteOpenHelper triggers a call to the onCreate method of the helper. In onCreate, I copy filename.db from the assets folder to the /data/data/projectname/databases directory. At the end of onCreate, I verified that the 17th byte in both copies of filename.db is 004.
I added an onOpen method to my SQLiteOpenHelper. It is called automatically by getReadableDatabase after onCreate, and does nothing except calling super.onOpen. After calling super.onOpen, my onOpen prints the 17th byte of both copies of filename.db. The byte in assets remains 004, but the byte in /data/data/projectname/databases has become 020. (The damage also occurs if onOpen does not call super.onOpen.)
What sinister force might be damaging /data/data/projectname/databases/filename.db between the end of the call to onCreate and the start of the call to onOpen? Or is the problem something else entirely? Thank you very much.
package edu.nyu.scps.offer;
import android.content.Context;
import android.content.res.AssetManager;
import android.database.Cursor;
import android.database.sqlite.SQLiteDatabase;
import android.database.sqlite.SQLiteOpenHelper;
import android.util.Log;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
public class Helper extends SQLiteOpenHelper {
private static final String DB_NAME = "offers.db";
private Context context;
public Helper(Context context) {
super(context, context.getDatabasePath(DB_NAME).getPath(), null, 1);
this.context = context;
}
#Override
public void onCreate(SQLiteDatabase db) {
try {
AssetManager assetManager = context.getAssets();
InputStream inputStream = assetManager.open(DB_NAME); //in assets folder
OutputStream outputStream = new FileOutputStream(getDatabaseName());
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) > 0) {
outputStream.write(buffer, 0, length);
}
outputStream.flush();
outputStream.close();
inputStream.close();
} catch (IOException exception) {
Log.e("myTag", "exception " + exception);
}
}
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
}
};
Here is a .dump of the file I created by hand using sqlite3:
sqlite3 offers.db .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE android_metadata (
"locale" text default "en_US"
);
INSERT INTO "android_metadata" VALUES('en_US');
CREATE TABLE offers (
_id integer primary key autoincrement,
title text,
description text
);
INSERT INTO "offers" VALUES(1,'Prewar Castle','It reminds me of the broken battlements of my own castle in Transylvania.');
INSERT INTO "offers" VALUES(2,'Shelter Island','heated saltwater pool, tennis, professional landscaping, finished lower level.');
INSERT INTO "offers" VALUES(3,'Amagansett','Heated pool, room for tennis, adjacent to nature conservancy, provate beach and pier rights.');
INSERT INTO "offers" VALUES(4,'Sagaponack Village','Insulate yourself from the barbarians at the hedgerow.');
INSERT INTO "offers" VALUES(5,'Amagansett Dunes','Close proximity to ocean beaches, oversized hot tub, 2 fireplaces. 4 bedrooms, 2.5 baths, .17 acres.');
INSERT INTO "offers" VALUES(6,'East Hampton Village','Outdoor shower, 2 car garage, fireplace, close to village and ocean beaches.');
INSERT INTO "offers" VALUES(7,'Yonkers','Batcave with stunning views of ice drifting down the Hudson. Hiking on Old Croton Aqueduct. Floods when Saw Mill River rises.');
INSERT INTO "offers" VALUES(8,'Tarrytown','Listen to the music of the New York State Thruway.');
INSERT INTO "offers" VALUES(9,'East Village','Pay extortion for a shoebox!');
INSERT INTO "offers" VALUES(10,'Poughkeepsie','Ranch-style living in chip plant recently vacated by a shrinking IBM.');
DELETE FROM sqlite_sequence;
INSERT INTO "sqlite_sequence" VALUES('offers',10);
COMMIT;
April 15, 2015: I discovered why my .db file was corrupted. When the getReadableDatabase method of class SQLiteOpenHelper called onCreate, the .db file had already been created and was even in the middle of a database transaction. The code in my onCreate overwrote this file completely. At some later time the SQLiteOpenHelper attempted to complete the transaction, but the .db file with which it started the transaction has been destroyed. Solution: the .db file must be copied from the project's assets before getReadableDatabase is called. The most straightforward approach would be to copy it in the constructor for my subclass of SQLiteOpenHelper. But Android's documentation says that SQLiteOpenHelper's constructor must "return very quickly" and defer the creation of the .db file to getReadableDatabase. So I ended up overriding getReadableDatabase with a new method that copies the .db file and then calls the getReadableDatabase of the superclass SQLiteOpenHelper. super.getReadableDatabase will not attempt to create a new .db file if it finds the file already in existence. Details omitted.
I debugged my original code only because I was curious to know why it corrupted the .db file. In real life I'm going to use SQLiteAssetHelper, and I am grateful for the suggestion.

Fastest and most efficient way to pre-populate database in Android

If you want to pre-populate a database (SQLite) in Android, this is not that easy as one might think.
So I found this tutorial which is often referenced here on Stack Overflow as well.
But I don't really like that way of pre-populating the database since you take the control from the database handler and create the files yourself. I would prefer to not touch the file system and let the database handler do everything on its own.
So what I thought one could do is create the database in the database handler's onCreate() as usual but then load a file (.sql) from /assets which contains the statements to fill in the values:
INSERT INTO testTable (name, pet) VALUES ('Mike', 'Tiger');
INSERT INTO testTable (name, pet) VALUES ('Tom', 'Cat');
...
But calling execSQL() in the handler's onCreate() doesn't really work. It seems that the /assets file must not have more than 1MB and the execSQL() only executes the first statement (Mike - Tiger).
What would you do do pre-populate the database?

I suggest the following:
Wrap all of your INSERT logic into a transaction (BEGIN... COMMIT, or via the beginTransaction()... endTransaction() APIs)
As already suggested, utilize the bind APIs and recycle objects.
Don't create any indexes until after this bulk insert is complete.
Additionally take a look at Faster bulk inserts in sqlite3?

Your question states, that you want the fastest way - but you don't like the way it's done in the article - you don't want to manually replace the DB file (even though, it may be actually faster than filling empty DB with queries).
I had exaclty the same thoughts - and I figured out, that populating via SQL statements and prepopulating can both be the best solution - but it depends on the way you will use the DB.
In my application I need to have about 2600 rows (with 4 columns) in DB at the very first run - it's the data for autocompletion and few other things. It will be modified quite rarely (users can add custom records, but most of the time - they don't need to) and is quite big. Populating it from SQL statements takes not only significantly more time, but more space in the APK (assuming I would store data inside it, alternatively I could download it from the internet).
This is the very simple case (the "Big" insert can take place only once and only at first startup) and I decided to go with copying prepopulated DB file. Sure, it may not be the nicest way - but it's faster. I want my users to be able to use the app as quickly as it's possible and treat speed as a priority - and they really like it. On the contrary, I doubt they would be glad when app would slow down because I thought that slower and nicer solution is actually better.
If instead of 2600 my table would have initially ~50 rows, I would go with SQL statements, since speed and size difference wouldn't be so big.
You have to decide which solution fits your case better. If you foresee any problems that may arise from using "prepopulated db" option - don't use it. If you are not sure about these problems - ask, providing more details on how you will use (and eventually, upgrade) contents of the DB. If you aren't quite sure which solution will be faster - benchmark it. And don't be afraid of that copying file method - it can work really well, if used wisely.

You can have your cake and eat it too. Here is a solution that can both respect the use of your db adapter and also use a simple (and much faster) copy process for a pre-populated database.
I'm using a db adapter based on one of Google's examples. It includes an internal class dbHelper() that extends Android's SQLiteOpenHelper() class. The trick is to override it's onCreate() method. This method is only called when the helper can't find the DB you are referencing and it has to create the DB for you. This should only happen the first time it is called on any given device installation, which is the only time you want to copy the DB. So override it like this -
#Override
public void onCreate(SQLiteDatabase db) {
mNeedToCopyDb = true;
}
Of course make sure you have first declared and initialized this flag in the DbHelper -
private Boolean mNeedToCopyDb = false;
Now, in your dbAdapter's open() method you can test to see if you need to copy the DB. If you do then close the helper, copy the DB and then finally open a new helper (see below code). All future attempts to open the db using the db adapter will find your (copied) DB and therefor the onCreate() method of the internal DbHelper class will not be called and the flag mNeedToCopyDb will remain false.
/**
* Open the database using the adapter. If it cannot be opened, try to
* create a new instance of the database. If it cannot be created,
* throw an exception to signal the failure.
*
* #return this (self reference, allowing this to be chained in an
* initialization call)
* #throws SQLException if the database could neither be opened nor created
*/
public MyDbAdapter open() throws SQLException {
mDbHelper = new DatabaseHelper(mCtx);
mDb = mDbHelper.getReadableDatabase();
if (mDbHelper.mNeedToCopyDb == true){
mDbHelper.close();
try {
copyDatabase();
} catch (IOException e) {
e.printStackTrace();
} finally {
mDbHelper = new DatabaseHelper(mCtx);
mDb = mDbHelper.getReadableDatabase();
}
}
return this;
}
Just place some code to do your database copy inside of your db adapter in a method named copyDatabase() as used above. You can use the value of mDb that was updated by the first instance of DbHelper (when it created the stub DB) to get the path to use for your output stream when you do the copy.
Construct your input stream like this
dbInputStream = mCtx.getResources().openRawResource(R.raw.mydatabase);
[note: If your DB file is too large to copy in one gulp then just break it up into a few pieces.]
This works very fast and puts all of the db access code (including the copying of the DB if needed) into your db adapter.

I wrote a DbUtils class similar to the previous answer. It is part of the ORM tool greenDAO and is available on github. The difference is that it will try to find statement boundaries using a simple regular expression, not just line endings. If you have to rely on a SQL file, I doubt that there's a faster way.
But, if you can supply the data in another format, it should be significantly faster than using a SQL script. The trick is to use a compiled statement. For each data row, you bind the parsed values to the statement and execute the statement. And, of course, you need to do this inside a transaction. I would recommend a simple delimiter separated file format (for example CSV) because it can be parsed faster than XML or JSON.
We did some performance tests for greenDAO. For our test data, we had insert rates of about 5000 rows per second. And for some reason, the rate dropped to half with Android 4.0.

ye, the assets maybe has size limit, so if bigger than the limit, you can cut to more files.
and exesql support more sql sentence, here give you a example:
BufferedReader br = null;
try {
br = new BufferedReader(new InputStreamReader(asManager.open(INIT_FILE)), 1024 * 4);
String line = null;
db.beginTransaction();
while ((line = br.readLine()) != null) {
db.execSQL(line);
}
db.setTransactionSuccessful();
} catch (IOException e) {
FLog.e(LOG_TAG, "read database init file error");
} finally {
db.endTransaction();
if (br != null) {
try {
br.close();
} catch (IOException e) {
FLog.e(LOG_TAG, "buffer reader close error");
}
}
}
above example require the INIT_FILE need every line is a sql sentence.
Also, if your sql sentences file is big, you can create the database out site of android(sqlite support for windows, linux, so you can create the database in your os, and copy the database file to your assets folder, if big, you can zip it)
when your application run, you can get the database file from assets, directed to save to your application's database folder (if you zip it, you can unzip to the application's database folder)
hope can help you -):

I used this method. First create your sqlite database there are a few programs you can use I like SqliteBrowser. Then copy your database file into your assets folder. Then you can use this code in the constructor of SQLiteOpenHelper.
final String outFileName = DB_PATH + NAME;
if(! new File(outFileName).exists()){
this.getWritableDatabase().close();
//Open your local db as the input stream
final InputStream myInput = ctx.getAssets().open(NAME, Context.MODE_PRIVATE);
//Open the empty db as the output stream
final OutputStream myOutput = new FileOutputStream(outFileName);
//final FileOutputStream myOutput = context.openFileOutput(outFileName, Context.MODE_PRIVATE);
//transfer bytes from the inputfile to the outputfile
final byte[] buffer = new byte[1024];
int length;
while ((length = myInput.read(buffer))>0){
myOutput.write(buffer, 0, length);
}
//Close the streams
myOutput.flush();
((FileOutputStream) myOutput).getFD().sync();
myOutput.close();
myInput.close();
}
} catch (final Exception e) {
// TODO: handle exception
}
DB_PATH is something like /data/data/com.mypackage.myapp/databases/
NAME is whatever database name you choose "mydatabase.db"
I know there are many improvements on this code but it worked so well and is VERY FAST. So I left it alone. Like this might be even better in the onCreate() method. Also checking if the file exists every time is probably not the best. Anyway like I said it works, it's fast and reliable.

If the data is not private then simply host it on your website then download it on first run. That way you can keep it up to date. So long as you remember to take app version into account when you upload it to your webserver.

Android does not update SQLite database

I am experiencing some trouble with an SQLIte database in my Android application.
The issue is that the database is never updated, not even on multiple restarts of the emulator, of Eclipse or after deletion from DDMS.
This is my onCreate method, located in a class that extends SQLiteOpenHelper:
public void onCreate(SQLiteDatabase database) {
try {
database.execSQL(ESSENCE_TABLE_CREATE);
database.execSQL(PACCO_TABLE_CREATE);
database.execSQL(TAVOLE_TABLE_CREATE);
database.rawQuery("insert into essenza values(1, 'Rovere')",
null); // added later
} catch (SQLException e) {
Log.e("DB", e.getMessage());
}
}
After instantiating the helper, I request a reference to the database:
helper = new DBHelper(context, dbpath + "/" + DATABASE_NAME);
database = helper.getWritableDatabase();
It seems that the rawQuery statement (which was added at a later time) is not executed and that the database in use is instead cached from a previous version. I also tried to change the version of the database, but it did not work. Am I missing something? Thanks in advance.

You have two options:
Use DDMs to delete the database file from your device (look in /data/data/). This will force Android to run onCreate again.
In your constructor, increment the database version you pass to SQLiteOpenHelper. Add your raw query to onUpgrade.
You probably want option 1. Option 2 is better if you have users of your app whose databases you want to update.

how to access path of database or file of different app in android

Is it possible to get path of database file in android at path : "/data/system/accounts.db"
in my app i want to use this database, but not getting its path.
if i do hardcoding and remove the file i'm able to do it. But i want to access it as database so that i can drop the table.
Any help would be appreciable.
code i tried:
private SQLiteDatabase db;
File dbFile = mContext.getDatabasePath("accounts.db");
if (dbFile.exists())
{
db = SQLiteDatabase.openOrCreateDatabase(dbFile, null);
db.execSQL("DROP TABLE IF EXISTS accounts");
db.setVersion(db.getVersion()+1);
}

You can use android.os.Environment.getDataDirectory() to get the path to the data directory.
Don't use the mContext.getDatabasePath if the db is not in the data dir of your app.
Also see
Is it possible to move the internal DB to the SDCard?
if you want to search, do a recursive file search via something like this:
private void searchFile(String startingPoint) {
Files[] files = File("/data").listFiles();
for (File f:files) {
if (f.isDirector) searchFile(f);
else if (f.getName().equals("accounts.db")) {
// delete db here...
// and exit the search here...
}
}
}
and check if it contains the accounds.db. If you reach another subdirectory, call your own function recursively with the subfolder as the starting point, and so on....
But make sure you have read access to the relevant folder you set as starting point to start the search.