Develop Reference

android file system persistance

Hi Iam having serious issues try to persist some serializable objects to a file on the local android file system. Iam getting a Bad file descriptor error and I think it is to do with my methods for creating the file. the file and checking if the file exists. i create a private file object in the class. Then, on write or read. I check file existance with the following code.
#Override
public boolean fileExists() {
File file = context.getFileStreamPath(filename);
return file.exists();
}
this doesnt instantiate my file object called "objectfile"!! but does check the "filename" exists.
to create the file I call this method if "filename" doesnt exist.
public void createFile()
{
objectfile = new File(context.getFilesDir(), filename);
objectfile.setReadable(true);
objectfile.setWritable(true);
}
Iam not sure if this will give me back my previously created file which would be ideally what I want to do. Is there a way i can just get the old file or create a new one and pass it to "objectfile" variable in the constructor??
Iam also wondering what the best way to do this is??
Or should i just use the mysqlite db? using object file persistance doesn't seem to be working out for me right now and iam working to a deadline. Also this method is mention in the gooogle docs so I thought it would be legit was to do it.
http://developer.android.com/training/basics/data-storage/files.html
here is my method for reading the serializable objects
public synchronized ArrayList<RoomItem> readObjects() {
final ArrayList<RoomItem> readlist = new ArrayList<>();
if(!fileExists())
return readlist;
if(objectfile == null)
createFile();
try {
finputstream = new FileInputStream(objectfile);
instream = new ObjectInputStream(finputstream);
readwritethread = new Thread(new Runnable() {
#Override
public void run() {
try {
final ArrayList<RoomItem> readitems = (ArrayList<RoomItem>) instream.readObject();
instream.close();
finputstream.close();
handler.post(new Runnable() {
#Override
public void run() {
listener.updateList(readitems);
}
});
} catch(IOException e)
{
e.printStackTrace();
}
catch(ClassNotFoundException e)
{
e.printStackTrace();
Log.d("read failed", "file read failed");
}
}
});
}
catch(Exception e)
{
e.printStackTrace();
}
timeOutReadWrite(readwritethread);
readwritethread.start();
try {
readwritethread.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
Log.d("read from file", "file read");
return readlist;
if anyone could suggest any improvements id really appreciate it. I use a handler to pass back to my activity and implement a listener interface on my activity thats call the activity when all the obj are read. Thanks again!

1#: Yes, it will return the original file you created.
2#: Depends on the thing you want to store, seems File is more flex from description
hope helpful.

We have used
FileOutputStream fos = context.openFileOutput("file.ser", Context.MODE_PRIVATE);
to write our serialized files.This will carete files in /data/data/app.package.name/files/. In fact, this path is returned by getFilesDir().
And while deserializing, use
//make sure you pass the same file that was passed to openFileOutput()..
FileInputStream fis = context.openFileInput("file.ser");
Also, to avoid confusing between file names you can use name of class that is being serialized.
Ex:
public static <T> void serialize(final Context context, final T objectToSerialize) {
....
....
Strin fileName = objectToSerialize.getClass().getSimpleName();
...
}
Do this and keep the method in util so it can be used for any type of objects (T type) to serialize.

Replacing database with another database at runtime.

I have two databases, one database is the primary. This primary DB is responsible for holding the current data which is up to date and my secondary DB is populated via a cron job, once the primary DB gets obsolete I want to replace it with the secondary DB via a file operation of just over writing the existing DB and refreshing my views. Is it possible to do this, is there a better way?
So far what I have done is:
public void writeToSD() throws IOException {
File f=new File("/mnt/sdcard/dump.db");
FileInputStream fis=null;
FileOutputStream fos=null;
try{
fis=new FileInputStream(f);
fos=new FileOutputStream("/data/data/com.one.two/databases/Bdr");
while(true){
int i=fis.read();
if(i!=-1){
fos.write(i);
}
else{
break;
}
}
fos.flush();
}
catch(Exception e){
e.printStackTrace();
}
finally{
try{
fos.close();
fis.close();
}
catch(IOException ioe){
System.out.println(ioe);
}
}

How about always using the same database files (let's say dbA, dbB) with two instances of SQLiteOpenHelper and using an utility class like this instead of using raw SQLiteOpenHelper:
class Db {
private SQLiteOpenHelper mPrimaryDb;
private SQLiteOpenHelper mSecondaryDb;
public Db(Context context) {
mPrimaryDb = new MyDbHelper(context, "db_a");
mSecondaryDb = new MyDbHelper(context, "db_b");
}
public SQLiteOpenHelper getPrimaryDb() {
return mPrimaryDb;
}
public SQLiteOpenHelper getSecondaryDb() {
return mSecondaryDb;
}
public void swapDb() {
SQLiteOpenHelper tmp = mPrimaryDb;
mPrimaryDb = mSecondaryDb;
mSecondaryDb = tmp;
// TODO: notify data users that data has changed, cleanup the old primary database, etc.
}
{

If you want to use file operations, renaming the data base files is faster. But during file operations all connections have to be closed before any action.
If insertion is too slow, I would not overwrite the database file. I would generate the new database with a temp name and the same table and view structure. After finishing writing to the temp file I would rename the file to the same name as the invariant part of the old database plus a version number or a timestamp . And in my application I would look periodically for a new version, if found I would close all connections to the old file and open the new database.

How to insert bulk data in android sqlite database using ormlite efficiently

I'm trying to insert 100000 records in android sqlite database at a time. I'm using following two different methods.
private void bulkInsertDataBySavePoint(final List<User> users) {
log.debug("bulkInsertDataBySavePoint()");
DatabaseConnection conn = null;
Savepoint savepoint = null;
try {
conn = userDao.startThreadConnection();
savepoint = conn.setSavePoint("bulk_insert");
for (User user : users) {
userDao.create(user);
}
} catch (SQLException e) {
log.error("Something went wrong in bulk Insert", e);
} finally {
if (conn != null) {
try {
conn.commit(savepoint);
userDao.endThreadConnection(conn);
} catch (SQLException e) {
log.error("Something went wrong in bulk Insert", e);
}
}
}
}
And
private void bulkInsertDataByCallBatchTasks(final List<User> users) {
log.debug("bulkInsertDataByCallBatchTasks()");
try {
userDao.callBatchTasks(new Callable<Void>() {
#Override
public Void call() throws Exception {
for (User user : users) {
userDao.create(user);
}
return null;
}
});
} catch (Exception e) {
e.printStackTrace();
}
}
Both methods work fine. On average they take 140 seconds and take 60-65% CPU which is not ok, I think.
The idea is, I have to consume an api which will provide json data. I have to parse that json data and then insert into sqlite database for offline usage.
I'm looking for an efficient way to solve this issue.
Any thought?

I'm trying to insert 100000 records in android sqlite database at a time... On average they take 140 seconds and take 60-65% CPU which is not ok in my opinion.
Unfortunately I don't have an easy answer for you. You may have to do this sort of insert directly using raw SQL to achieve faster performance on the limited Android CPU. Once you have the data inserted then you can turn to ORMLite to query or manipulate the data faster.

I've had the same problem, and found a reasonable workaround. This took insert time from 2 seconds to 150ms:
final OrmLiteSqliteOpenHelper myDbHelper = ...;
final SQLiteDatabase db = myDbHelper.getWritableDatabase();
db.beginTransaction();
try{
// do ormlite stuff as usual, no callBatchTasks() needed
db.setTransactionSuccessful();
}
finally {
db.endTransaction();
}

Hrm. Good idea #FarrukhNajmi. I've just added it to trunk. It will be in version 4.49.
#Gray Is it still unstable? when can we see it in maven?
And if com.j256.ormlite.dao.ForeignCollection#addAll make only one request it would be nice too.

Ormlite ObjectCache returning old data

I'm using Ormlite on Android and with the ObjectCache enabled, I get old data back after updating the table with an UpdateBuilder and a ColumnExpression. I have read through the doc and it does not warn against using the UpdateBuilder with the cache enabled.
The settings table should have just 1-5ish rows max. The updateColumnExpression seems like an easy way to allow only one of the rows to be true.
Is this the expected behavior?
public void setActiveSetting(String id)
{
try {
UpdateBuilder<Settings, Integer> updateBuilder2 = getHelper().getSettingsDao().updateBuilder();
updateBuilder2.updateColumnExpression("active", "id = " + id );
updateBuilder2.update();
} catch (SQLException e){
e.printStackTrace();
}
}
And this is the call that returns the outdated data:
public List<Settings> getSettings()
{
List<Settings> settings = null;
try {
settings = getHelper().getSettingsDao().queryForAll();
} catch (SQLException e) {
e.printStackTrace();
}
return settings;
}
And the settings DAO:
public Dao<Settings, Integer> getSettingsDao()
{
if (null == settingsDao) {
try {
settingsDao = getDao(Settings.class);
settingsDao.setObjectCache(true);
} catch (java.sql.SQLException e) {
e.printStackTrace();
}
}
return settingsDao;
}
Disabling the ObjectCache does return the correct data, but this data is fetched quite frequently, so I'd like to keep it.
Thanks

Is this the expected behavior?
Unfortunately, yes. If you had updated the object using dao.update(...); then the cache would know that the object needed to be refreshed. By using the UpdateBuilder to make mass changes to the table, there is no way for the cache to know which objects were affected.
You will need to clear the cache after your call to the UpdateBuilder finishes.

Reading SQL from file and SQLiteOpenHelper

I need to include an existing SQLiteDatabase in my Android app and I also want the ability to download and install a new db. I did some research and my first working solution came from here. I didn't like that solution, for one thing, it assumes that the database will always be in a fixed path and other oddities.
So, rather than putting the existing database file in the assets, I exported the database to a SQL file, read it in, and then in the onCreate() method of my SQLiteOpenHelper, I call a new method updateDatabase with an open DataInputStream with the file data.
I ran into a few issues, which I think I solved, but I'm sure I didn't think of all the issues, which are:
When SQLiteOpenHelper's onCreate method is called, the database has been created, it is open, and inTransaction() is true. As a result, if the imported sql file includes BEGIN TRANSACTION, an exception is thrown and if the sql String contains statements creating 'android_metadata' yet another exception. So, I added a simple search using String.contains() looking for these keywords, and set a boolean doExecute to false to avoid executing them. So a question is, is there a better SQL class or method to filter this, or even a better regexp method?
Similar issue with having unexpected line breaks in the SQL file. I read the file with readLine() and to look for line breaks, I simply use String.trim() on the line, then check for endsWith(";"). This puts some constraints on my input file, like not having multiple statements on a single line. So, is there a better way to pre-process SQL from a file?
Here's the code I use to create my db after I've gotten a DataInputStream from the assets resource or from a download:
public boolean updateDatabase(DataInputStream inStream, SQLiteDatabase db, boolean doClear) throws Error {
String sqlStatement = null;
boolean result = true;
boolean inOnCreate = true;
boolean wasInTransaction;
if(doClear) dropDatabase();
// if called from onCreate() db is open and inTransaction, else getWritableDatabase()
if(db == null) {
inOnCreate = false;
db = this.getWritableDatabase();
}
wasInTransaction = db.inTransaction(); // see NB below
boolean doExecute;
try {
while ((sqlStatement = inStream.readLine()) != null) {
// trim, so we can look for ';'
sqlStatement.trim();
if(!sqlStatement.endsWith(";")) {
continue; // line breaks in file, get whole statement
}
// NB - my file (exported from SQLite Database Browser starts with "BEGIN TRANSACTION;".
// executing this throws SQLiteException: cannot start a transaction within a transaction
// According to SQLiteDatabase doc for beginTransaction(), "Transactions can be nested"
// so this is a problem
// but... possibly it is an "exclusive transaction" ?
doExecute = true;
if(wasInTransaction) {
// don't execute BEGIN TRANSACTION; or COMMIT;
if((sqlStatement.contains("BEGIN" ) || sqlStatement.contains("begin" )) &&
(sqlStatement. contains("TRANSACTION") || sqlStatement.contains("transaction" ))) {
doExecute = false;
}
if(sqlStatement.contains("COMMIT") || sqlStatement.contains("commit")) {
doExecute = false;
}
} // inTransaction
// this statement could be in older databases, but this scheme doesn't need, can't have it
if(sqlStatement.contains("android_metadata")) {
doExecute = false;
}
if(doExecute) {
try {
db.execSQL(sqlStatement);
} catch (SQLException e) {
throw(new Error("Error executing SQL " + sqlStatement));
} // try/catch
} // doExecute
} // while()
} catch (IOException e) {
result = false; // which won't matter if we throw
throw(new Error("Error reading " + DB_SQL));
}
if(!inOnCreate) {
db.close();
}
return result;
}

Wouldn't want to have you cop out early on such ambitious and elegant implementations, but if you have your database already made and checked with a database browser and all, have you considered SQLite Asset Helper? If your main issue was being forced to use the asset folder, this method lib let's you use a file from any specified directory. Moreover, it allows for handling the raw .db file. Worth checking out.