I'm using an external database inside my android application and it directly embeds inside the apk package after compiling.
As I want to implement in app purchase in order to access some of its data I don't want to leave it without any encryption.
I used Sqlcipher library but it makes the app too big and slow.
Isn't there any other way to do this? For example an algorithm to encrypt the strings so I put the encrypted text in database and decrypt it inside application code?
The following is an example App that Encrypts part of the data that can then be turned on. It is based upon the code in my comment.
Stage 1 - The Master Database.
To start with you need the database that is to be the basis of the encrypted database (i.e. the MASTER database that IS NOT include in the App, it's use is to create the Encrypted database (or databases, perhaps a library, each database with a unique password/secret key if you wanted greater security)) in part consider this (as is used throughout the example) :-
As you can see this one will work by having a table called FreeData and another called PaidData. The tables definitions are the same EXCEPT that for the PaidData there is no ID column (the intention of this method is to decrypt the rows in the PaidData into the FreeData when/if the requested and the SecretKey (password) is valid.).
So The FreeData table looks like :-
The PaidData table looks like :-
So the only difference between the tables is the actual data contained within and that the id column is missing.
The id's will be generated when the encrypted data is extracted from the PaidData table, decrypted and the inserted into the FreeData table. Thus just one decryption is required to get access to the data.
Stage 2 - Generating the Encrypted Database for distribution with the App
This is done by a App just for this purpose using the EncryptDecrypt class very similar to the one at Encrypt data in SQLite
as per EncryptDecrypt.java
class EncryptDecrypt {
private Cipher cipher;
private static SecretKeySpec secretKeySpec;
private static IvParameterSpec ivParameterSpec;
private boolean do_encrypt = true;
/**
* Construct EncryptDecrypt instance that does not check user login-in
* mode, thus the assumption is that this user is NOT the special user
* NOUSER that doesn't require a password to login; this constructor
* is designed to ONLY be used when a user has been added by NOUSER,
* and to then encrypt the data using the enccryptForced method solely
* to encrypt any existing card data for the new user that has a password.
*
* #param context The context, required for database usage (user)
* #param skey The secret key to be used to encrypt/decrypt
*/
EncryptDecrypt(Context context, String skey) {
//DBUsersMethods users = new DBUsersMethods(context);
String saltasString = "there is no dark side of the moon it is all dark.";
String paddedskey = (skey + saltasString).substring(0,16);
secretKeySpec = new SecretKeySpec(paddedskey.getBytes(),"AES/CBC/PKCS5Padding");
ivParameterSpec = new IvParameterSpec((saltasString.substring(0,16)).getBytes());
try {
cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
} catch (Exception e){
//e.printStackTrace();
}
}
/**
* Normal encryption routine that will not encrypt data if the user is
* the special case NOUSER (i.e LOGIN mode is NOUSER), otherwise data
* is encrypted.
*
* #Param toEncrypt The string to be encrypted
* #return The encryted (or not if NOUSER) data as a string
*/
String encrypt(String toEncrypt) {
if (!do_encrypt) {
return toEncrypt;
}
byte[] encrypted;
try {
cipher.init(Cipher.ENCRYPT_MODE,secretKeySpec,ivParameterSpec);
encrypted = cipher.doFinal(toEncrypt.getBytes());
} catch (Exception e) {
//e.printStackTrace();
return null;
}
return Base64.encodeToString(encrypted, Base64.DEFAULT);
}
/**
* Encryption, irrespective of the USER type, noting that this should
* only be used in conjunction with an EncryptDecrypt instance created
* using the 2nd/extended constructor
*
* #param toEncrypt The string to be encrypted
* #return The encrypted data as a string
*/
String encryptForced(String toEncrypt) {
byte[] encrypted;
try {
cipher.init(Cipher.ENCRYPT_MODE,secretKeySpec,ivParameterSpec);
encrypted = cipher.doFinal(toEncrypt.getBytes());
} catch (Exception e) {
//e.printStackTrace();
return null;
}
return Base64.encodeToString(encrypted,Base64.DEFAULT);
}
/**
* Decrypt an encrypted string
* #param toDecrypt The encrypted string to be decrypted
* #return The decrypted string
*/
String decrypt(String toDecrypt) {
if (!do_encrypt) {
return toDecrypt;
}
byte[] decrypted;
try {
cipher.init(Cipher.DECRYPT_MODE,secretKeySpec,ivParameterSpec);
decrypted = cipher.doFinal(Base64.decode(toDecrypt,Base64.DEFAULT));
} catch (Exception e) {
//e.printStackTrace();
return null;
}
return new String(decrypted);
}
}
As this was designed for user login and multiple users where a salt was part of the database the salt has been hard coded using :- String saltasString = "there is no dark side of the moon it is all dark.";, The phrase can be changed as long as it is at least 16 characters in length (only the first 16 bytes are used).
A class is used to cater for potential flexibility/expansion where multiple tables can be specified or not for encryption and for multiple columns that can be encrypted, copied asis or skipped (e.g. id's would probably be skipped (in the example it's not even defined as a column).).
This class is TableColumnConvertList.java and is :-
public class TableColumnConvertList {
private ArrayList<TableEntry> tables;
public TableColumnConvertList() {
this.tables = new ArrayList<>();
}
public String[] getTables() {
String[] tableList = new String[tables.size()];
int ix = 0;
for (TableEntry te: this.tables) {
tableList[ix++] = te.getSourceTableName();
}
return tableList;
}
public String[] getTableColumnNamesToEncrypt(String tableName) {
String[] rv = null;
for(TableEntry te: this.tables) {
if (te.getSourceTableName().equals(tableName)) {
rv = new String[te.getColumnNamesToEncrypt().size()];
int ix=0;
for (String s: te.getColumnNamesToEncrypt()) {
rv[ix++] = s;
}
}
}
return rv;
}
public String[] getTableColumnNamesToCopyAsis(String tableName) {
String[] rv = null;
for (TableEntry te: this.tables) {
if (te.getSourceTableName().equals(tableName)) {
rv = new String[te.getColumnNamesToCopyAsis().size()];
int ix=0;
for (String s: te.getColumnNamesToCopyAsis()) {
rv[ix++] = s;
}
}
}
return rv;
}
public String[] getTableColumnNamesToSkip(String tableName) {
String[] rv = null;
for (TableEntry te: this.tables) {
if (te.sourceTableName.equals(tableName)) {
rv = new String[te.getColumnNamesToSkip().size()];
int ix =0;
for (String s: te.getColumnNamesToSkip()) {
rv[ix++] = s;
}
}
}
return rv;
}
public void addTable(
String sourceTableName,
String destinationTableName,
String[] columnNamesToEncrypt,
String[] columnNamesToCopyAsis,
String[] columnNamesToSkip
) {
tables.add(
new TableEntry(
sourceTableName,
destinationTableName,
columnNamesToEncrypt,
columnNamesToCopyAsis,
columnNamesToSkip
)
);
}
private class TableEntry {
private String sourceTableName;
private String destinationTableName;
private ArrayList<String> columnNamesToEncrypt;
private ArrayList<String> columnNamesToCopyAsis;
private ArrayList<String> columnNamesToSkip;
private TableEntry() {}
private TableEntry(String sourceTableName,
String destinationTableName,
String[] columnNamesToEncrypt,
String[] columnNamesToCopyAsis,
String[] columnNamesToSkip
) {
this.sourceTableName = sourceTableName;
this.destinationTableName = destinationTableName;
this.columnNamesToEncrypt = new ArrayList<>();
if (columnNamesToEncrypt != null && columnNamesToEncrypt.length > 0) {
for (String s: columnNamesToEncrypt) {
addColumn(s);
}
}
}
private void addColumn(String s) {
this.columnNamesToEncrypt.add(s);
}
private String getSourceTableName() {
return sourceTableName;
}
public String getDestinationTableName() {
return destinationTableName;
}
public void setSourceTableName(String sourceTableName) {
this.sourceTableName = sourceTableName;
}
public void setDestinationTableName(String destinationTableName) {
this.destinationTableName = destinationTableName;
}
private ArrayList<String> getColumnNamesToEncrypt() {
return columnNamesToEncrypt;
}
public void setColumnNamesToEncrypt(ArrayList<String> columnNamesToEncrypt) {
this.columnNamesToEncrypt = columnNamesToEncrypt;
}
private ArrayList<String> getColumnNamesToCopyAsis() {
return columnNamesToCopyAsis;
}
public void setColumnNamesToCopyAsis(ArrayList<String> columnNamesToCopyAsis) {
this.columnNamesToCopyAsis = columnNamesToCopyAsis;
}
public ArrayList<String> getColumnNamesToSkip() {
return columnNamesToSkip;
}
public void setColumnNamesToSkip(ArrayList<String> columnNamesToSkip) {
this.columnNamesToSkip = columnNamesToSkip;
}
}
}
The rest of this basic App, at present, is all in a single activity that uses two input's (EditTexts) :-
The secret key used to encrypt
The database name (file name) of the encrypted database.
Code in the App prevents using the same name as the base database, which needs to be copied into the assets folder.
and a Button, to initiate the Encryption if the input is good (to a fashion aka with limited validation).
This the layout xml activiy_main.xml is :-
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:orientation="vertical"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:padding="10dp"
tools:context=".MainActivity">
<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Database EncryptTool" />
<EditText
android:id="#+id/secretkey"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:hint="Secret Key to use to Encrypt the Database."
>
</EditText>
<EditText
android:id="#+id/databasename"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="MyDatabase"
android:hint="Database Name"
>
</EditText>
<Button
android:id="#+id/encrypt"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="ENCRYPT"
android:visibility="gone"
>
</Button>
</LinearLayout>
MainActivity.java where the work is done is :-
public class MainActivity extends AppCompatActivity {
public static final String ASSETDB_NAME = "basedb.db";
public static final int ASSETDB_NOT_FOUND = -10;
public static final int ASSETFILE_OPEN_ERROR = -11;
public static final int ASSETDB_OPEN_ERROR = -12;
public static final int ASSETDB_COPY_ERROR = -13;
public static final int ASSETDB_FLUSH_ERROR = -14;
public static final int ASSETDB_CLOSE_ERROR = -15;
public static final int ASSETFILE_CLOSE_ERROR = -16;
public static final int ASSETDB_CREATED_SUCCESSFULLY = 0;
public static final int BUFFERSIZE = 1024 * 4;
EditText mSecretKey, mDBName;
Button mEncryptButton;
TableColumnConvertList mTCCL = new TableColumnConvertList();
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
mDBName = this.findViewById(R.id.databasename);
mSecretKey = this.findViewById(R.id.secretkey);
mEncryptButton = this.findViewById(R.id.encrypt);
//<<<<<<<<< set what data to encrypt i.e. table(s) and the column(s) in the table >>>>>>>>>
mTCCL.addTable(
"PaidData",
"FreeData",
new String[]{"theData"},
new String[]{},
new String[]{"id"}
);
if (getDBFromAsset() >= 0) {
mEncryptButton.setVisibility(View.VISIBLE);
mEncryptButton.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View v) {
if (mDBName.getText().toString().length() < 1) {
Toast.makeText(
v.getContext(),
"The Database Name cannot be blank.",
Toast.LENGTH_LONG
).show();
mDBName.requestFocus();
return;
}
if (mDBName.getText().toString().equals(ASSETDB_NAME)) {
Toast.makeText(
v.getContext(),
"Database Name cannot be "
+ ASSETDB_NAME
+ ". Please change the name.",
Toast.LENGTH_LONG
).show();
mDBName.requestFocus();
return;
}
if (mSecretKey.getText().toString().length() < 1) {
Toast.makeText(
v.getContext(),
"The Secret Key cannot be blank.",
Toast.LENGTH_LONG
).show();
mSecretKey.requestFocus();
return;
}
if (createEncryptedDatabase(mTCCL,
mDBName.getText().toString(),
mSecretKey.getText().toString()
) == 0) {
Toast.makeText(v.getContext(),"Successfully Encrypted Database " + mDBName + " using Secret Key " + mSecretKey,Toast.LENGTH_LONG).show();
}
}
});
}
}
private boolean checkIfDataBaseExists(String databaseName) {
File dbFile = new File(this.getDatabasePath(databaseName).getPath());
if (dbFile.exists()) {
return true;
} else {
if (!dbFile.getParentFile().exists()) {
dbFile.getParentFile().mkdirs();
}
}
return false;
}
private boolean checkIfAssetDBExists() {
try {
InputStream is = this.getAssets().open(ASSETDB_NAME);
is.close();
return true;
} catch (IOException e) {
return false;
}
}
private int getDBFromAsset() {
int rv = ASSETDB_NOT_FOUND;
File dbFile = new File(this.getDatabasePath(ASSETDB_NAME).getPath());
InputStream is;
FileOutputStream os;
int read_length;
byte[] buffer = new byte[BUFFERSIZE];
if (!checkIfAssetDBExists()) {
return ASSETDB_NOT_FOUND;
}
if (checkIfDataBaseExists(ASSETDB_NAME)) {
dbFile.delete();
}
try {
rv = ASSETFILE_OPEN_ERROR;
is = this.getAssets().open(ASSETDB_NAME);
rv = ASSETDB_OPEN_ERROR;
os = new FileOutputStream(dbFile);
rv = ASSETDB_COPY_ERROR;
while ((read_length = is.read(buffer)) > 0) {
os.write(buffer,0,read_length);
}
rv = ASSETDB_FLUSH_ERROR;
os.flush();
rv = ASSETDB_CLOSE_ERROR;
os.close();
rv = ASSETFILE_CLOSE_ERROR;
is.close();
rv = ASSETDB_CREATED_SUCCESSFULLY;
} catch (IOException e) {
e.printStackTrace();
}
return rv;
}
private int createEncryptedDatabase(TableColumnConvertList tableColumnConvertList, String databaseName, String key) {
File copiedAssetDB = new File(this.getDatabasePath(ASSETDB_NAME).getPath());
File encryptedDB = new File(this.getDatabasePath(databaseName).getPath());
if (encryptedDB.exists()) {
encryptedDB.delete();
}
try {
byte[] buffer = new byte[BUFFERSIZE];
int read_length;
InputStream is = new FileInputStream(copiedAssetDB);
OutputStream os = new FileOutputStream(encryptedDB);
while ((read_length = is.read(buffer)) > 0) {
os.write(buffer,0,read_length);
}
os.flush();
os.close();
is.close();
} catch (IOException e) {
e.printStackTrace();
return -1;
}
SQLiteDatabase db = SQLiteDatabase.openDatabase(encryptedDB.getPath(),null,SQLiteDatabase.OPEN_READWRITE);
EncryptDecrypt ed = new EncryptDecrypt(this,key);
int errorcount = 0;
db.beginTransaction();
for (String t: tableColumnConvertList.getTables()) {
ContentValues cv = new ContentValues();
String[] columnsToEncrypt = tableColumnConvertList.getTableColumnNamesToEncrypt(t);
String[] columnOriginalValues = new String[columnsToEncrypt.length];
Cursor c = db.query(true,t,columnsToEncrypt,null,null,null,null,null, null);
int totalRows = c.getCount();
int updatedRows = 0;
while (c.moveToNext()) {
cv.clear();
int ovix=0;
StringBuilder whereClause = new StringBuilder();
for (String s: c.getColumnNames()) {
for (String ec: columnsToEncrypt ) {
if (s.equals(ec)) {
cv.put(s,ed.encrypt(c.getString(c.getColumnIndex(s))));
columnOriginalValues[ovix++] = c.getString(c.getColumnIndex(s));
if (whereClause.length() > 0) {
whereClause.append(" AND ");
}
whereClause.append(s).append("=?");
}
}
}
updatedRows += db.update(t,cv,whereClause.toString(),columnOriginalValues);
}
c.close();
Log.d("ENCRYPTRESULT","Read " + totalRows + " DISTINCT ROWS. Updated " + updatedRows);
errorcount += totalRows - updatedRows;
}
if (errorcount == 0) {
db.setTransactionSuccessful();
} else {
Toast.makeText(
this,
"Errors encountered Encrypting Database. Rolled back (not changed)",
Toast.LENGTH_LONG
).show();
}
db.endTransaction();
return errorcount;
}
}
Of importance is this line/code :-
TableColumnConvertList mTCCL = new TableColumnConvertList();
..........
//<<<<<<<<< set what data to encrypt i.e. table(s) and the column(s) in the table >>>>>>>>>
mTCCL.addTable(
"PaidData",
"FreeData",
new String[]{"theData"},
new String[]{},
new String[]{"id"}
);
This adds a table to the List of tables to be encrypted. It's parameters are :-
The name of the table that is to be included in Encryption.
the name of the table into which the encrypted data is to be stored.
Note this functionality is not present but could be added. As such the value is ignored.
The list of columns that are to be encrypted.
The list of columns that are to be copied asis.
This functionality is not present but could be added. As such the list is ignored.
The list of columns that are to be skipped (e.g. id columns).
Although coded, the functionality is not present. As such the list is ignored.
What the App does.
The final result is an database as per the database in the assets folder (named basedb.db) that has the data in the theData column of the PaidData table encrypted, but the the FreeData table is unchanged. This database could then be copied (e.g. using device explorer) and then included as an asset in the App that is to be distributed. That App could include a reversal of the Encryption using the secret key and the decryption part of the EncryptDecrypt class.
e.g.
The FreeData table :-
The PaidData table :-
When the App is started if copies the database (hard coded as basedb.db) from the assets folder it it exists and makes the Encrypt button visible.
If the Encrypt button isn't visible then the asset file was not located. So it's time to correct the issue (provide the correct database file).
Note as this is just a demo many checks/options that could/should be done or added are skipped for brevity.
If the Encrypt button appears then encryption is just a matter of hitting the button.
After hitting the button createEncryptedDatabase method is called.
This creates a copy, this will be the encrypted database, of the database copied from the assets folder by copying the file to it's new name (as per the given database name which MUST be different to the asset's file name).
Using the copied database it queries the table(s) defined in mTCCL (an instance of the TableColumnConvertList class) .
The query will extract data only for the columns that have been specified as those to be encrypted. The query only obtain distinct rows (i.e if multiple rows exist that has the same data in the columns then only one of the rows is extracted).
For each extracted row :-
The commonly used ContentValues instance is cleared.
The whereClause StringBuilder is cleared.
Each column in the Cursor is checked to see if it is a column, defined in the table being processed (it should be as only column t be encrypted are extracted).
if not then it is skipped.
The original value is saved in the appropriate element of the string array columnOriginalValues (this to be used as the bind parameters for the WHERE clause of the UPDATE)
An element of the ContentValues instance is added with the current column name and the encrypted data.
This is done as per cv.put(s,ed.encrypt(c.getString(c.getColumnIndex(s))));
If the length of the whereClause is greater than 0 then AND is added to the whereClause, then the column name suffixed with =? is added to the whereClause being built.
After all columns have been processed the SQLiteDatabase update method is called to update the columns setting the values to the encrypted values WHERE all the column match the original data.
After all rows have been processed the Cursor is closed and the next table processed.
If after all tables have been processed then error count is 0 then the transaction is set as successful, otherwise a message is Toasted Errors encountered Encrypting Database. Rolled back (not changed).
The transaction is then ended (if not set as successful then the data is not updated but rolled back).
The database will be in the data/data/package_name/databases folder e.g. :-
Example Decryption App for Answer 1
Note that the database (MyDatabase) was copied from data/data/package_name/databases of the App from the previous answer (i.e. the encrypted database) into the assets folder of this app
The following is a very basic App that initially only has the Free Data, but as an Edit Text and a Button that allows the Paid Data to be decrypted and retrieved. The available data (Free Data initially) is listed in a ListView; after decryption the PaidData having been copied to the FreeData is then avialable and listed.
Notes
the data can be decrypted numerous times and each such successful attempt will add and display more rows.
EncryptDEcrypt.java is identical to the one used in the Encrypt tool.
The Database Helper is :-
public class DBHelper extends SQLiteOpenHelper {
public static final String DBNAME = "MyDatabase";
public static final int DBVERSION = 1;
public static final String TBL_FREEDATA = "FreeData";
public static final String COL_FREEDATA_ID = "id";
public static final String COL_THEDATA = "theData";
SQLiteDatabase mDB;
public DBHelper(Context context) {
super(context, DBNAME, null, DBVERSION);
loadDBFromAssets(context);
mDB = this.getWritableDatabase();
}
#Override
public void onCreate(SQLiteDatabase db) {
}
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
}
public long insertFreeDataRow(String theData) {
ContentValues cv = new ContentValues();
cv.put(COL_THEDATA,theData);
return mDB.insert(TBL_FREEDATA,null,cv);
}
public Cursor getAllAvialableData() {
return mDB.query(TBL_FREEDATA,new
String[]{"*",COL_FREEDATA_ID + " AS " + BaseColumns._ID},
null,null,null,null,null
);
}
public void decryptAndLoadPaidData(Context context, String secretKey) {
EncryptDecrypt ed = new EncryptDecrypt(context,secretKey);
mDB.beginTransaction();
Cursor c = mDB.query("PaidData",null,null,null,null,null,null);
while (c.moveToNext()) {
String decrypted_data = ed.decrypt(c.getString(c.getColumnIndex(COL_THEDATA)));
if (decrypted_data != null) {
insertFreeDataRow(decrypted_data);
} else {
Toast.makeText(context,"Naughty, that's not the password.",Toast.LENGTH_LONG).show();
}
}
c.close();
mDB.setTransactionSuccessful();
mDB.endTransaction();
}
private boolean loadDBFromAssets(Context context) {
File dbFile = new File(context.getDatabasePath(DBNAME).getPath());
byte[] buffer = new byte[1024 * 4];
int read_length = 0;
if (dbFile.exists()) return true;
if (!dbFile.getParentFile().exists()) {
dbFile.getParentFile().mkdirs();
}
try {
InputStream assetdb = context.getAssets().open(DBNAME);
OutputStream realdb = new FileOutputStream(dbFile);
while ((read_length = assetdb.read(buffer)) > 0) {
realdb.write(buffer,0,read_length);
}
realdb.flush();
realdb.close();
assetdb.close();
} catch (IOException e) {
e.printStackTrace();
return false;
}
return true;
}
}
MainActivity.java is :-
public class MainActivity extends AppCompatActivity {
ListView mListView;
EditText mSecretKeyInput;
Button mDecrypt;
SimpleCursorAdapter mSCA;
Cursor mAllTheData;
DBHelper mDBhlpr;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
mListView = this.findViewById(R.id.list);
mSecretKeyInput = this.findViewById(R.id.secretKey);
mDecrypt = this.findViewById(R.id.decrypt);
mDBhlpr = new DBHelper(this);
manageListView();
manageDecryptButton();
}
private void manageListView() {
mAllTheData = mDBhlpr.getAllAvialableData();
if (mSCA == null) {
mSCA = new SimpleCursorAdapter(
this,android.R.layout.simple_list_item_1,mAllTheData,new String[]{DBHelper.COL_THEDATA},new int[]{android.R.id.text1},0);
mListView.setAdapter(mSCA);
} else {
mSCA.swapCursor(mAllTheData);
}
}
private void manageDecryptButton() {
mDecrypt.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View v) {
if (mSecretKeyInput.getText().toString().length() > 0) {
mDBhlpr.decryptAndLoadPaidData(v.getContext(),mSecretKeyInput.getText().toString());
manageListView();
}
}
});
}
}
Result
When first run the App only shows the Free Data as per :-
If the correct password/secret key is input and the Get Paid Data button is pressed then the extra data is added :-
If an incorrect password is provided, then the data is not loaded and a toast appears indicating the wrong password.
Related
I have kind of a quiz app and I have a database with all the questions in tables, for each question there is a column solved that I update if the answer was correct, so I can filter with SQL WHERE to only show unsolved questions.
Now every once in a while I have to correct typos in the questions or might want to add some new ones, so
How do I employ the corrected database (questions.db) from the assets to the saved one on the user device while keeping the solved columns?
I thought of and tried the following things without success:
Currently, i use a self-crafted solution to replace the database on the device (destructive) but between updates keep the solved info https://github.com/ueen/RoomAsset
Put solved info (question id solved y/n) in a separate table and LEFT JOIN to filter unsolved questions, this only complicated matters
Have an extra database for the solved questions, it seems there's no easy way to attach two Room Databases
So in essence, this may be inspiration for the Room dev team, I would like to have a proper migration strategy for createFromAsset with ability to specify certain columns/tables to be kept.
Thanks for your great work so far, I really enjoy using Android Jetpack and Room especially!
Also, I'm happy about any workaround I could employ to resolve this issue :)
I believe the following does what you want
#Database(version = DatabaseConstants.DBVERSION, entities = {Question.class})
public abstract class QuestionDatabase extends RoomDatabase {
static final String DBNAME = DatabaseConstants.DBNAME;
abstract QuestionDao questionsDao();
public static QuestionDatabase getInstance(Context context) {
copyFromAssets(context,false);
if (getDBVersion(context,DatabaseConstants.DBNAME) < DatabaseConstants.DBVERSION) {
copyFromAssets(context,true);
}
return Room.databaseBuilder(context,QuestionDatabase.class,DBNAME)
.addCallback(callback)
.allowMainThreadQueries()
.addMigrations(Migration_1_2)
.build();
}
private static RoomDatabase.Callback callback = new Callback() {
#Override
public void onCreate(#NonNull SupportSQLiteDatabase db) {
super.onCreate(db);
}
#Override
public void onOpen(#NonNull SupportSQLiteDatabase db) {
super.onOpen(db);
}
#Override
public void onDestructiveMigration(#NonNull SupportSQLiteDatabase db) {
super.onDestructiveMigration(db);
}
};
private static Migration Migration_1_2 = new Migration(1, 2) {
#Override
public void migrate(#NonNull SupportSQLiteDatabase database) {
}
};
private static boolean doesDatabaseExist(Context context) {
if (new File(context.getDatabasePath(DBNAME).getPath()).exists()) return true;
if (!(new File(context.getDatabasePath(DBNAME).getPath()).getParentFile()).exists()) {
new File(context.getDatabasePath(DBNAME).getPath()).getParentFile().mkdirs();
}
return false;
}
private static void copyFromAssets(Context context, boolean replaceExisting) {
boolean dbExists = doesDatabaseExist(context);
if (dbExists && !replaceExisting) return;
//First Copy
if (!replaceExisting) {
copyAssetFile(context);
return;
}
//Subsequent Copies
File originalDBPath = new File(context.getDatabasePath(DBNAME).getPath());
// Open and close the original DB so as to checkpoint the WAL file
SQLiteDatabase originalDB = SQLiteDatabase.openDatabase(originalDBPath.getPath(),null,SQLiteDatabase.OPEN_READWRITE);
originalDB.close();
//1. Rename original database
String preservedDBName = "preserved_" + DBNAME;
File preservedDBPath = new File (originalDBPath.getParentFile().getPath() + preservedDBName);
(new File(context.getDatabasePath(DBNAME).getPath()))
.renameTo(preservedDBPath);
//2. Copy the replacement database from the assets folder
copyAssetFile(context);
//3. Open the newly copied database
SQLiteDatabase copiedDB = SQLiteDatabase.openDatabase(originalDBPath.getPath(),null,SQLiteDatabase.OPEN_READWRITE);
SQLiteDatabase preservedDB = SQLiteDatabase.openDatabase(preservedDBPath.getPath(),null,SQLiteDatabase.OPEN_READONLY);
//4. get the orignal data to be preserved
Cursor csr = preservedDB.query(
DatabaseConstants.QUESTION_TABLENAME,DatabaseConstants.EXTRACT_COLUMNS,
null,null,null,null,null
);
//5. Apply preserved data to the newly copied data
copiedDB.beginTransaction();
ContentValues cv = new ContentValues();
while (csr.moveToNext()) {
cv.clear();
for (String s: DatabaseConstants.PRESERVED_COLUMNS) {
switch (csr.getType(csr.getColumnIndex(s))) {
case Cursor.FIELD_TYPE_INTEGER:
cv.put(s,csr.getLong(csr.getColumnIndex(s)));
break;
case Cursor.FIELD_TYPE_STRING:
cv.put(s,csr.getString(csr.getColumnIndex(s)));
break;
case Cursor.FIELD_TYPE_FLOAT:
cv.put(s,csr.getDouble(csr.getColumnIndex(s)));
break;
case Cursor.FIELD_TYPE_BLOB:
cv.put(s,csr.getBlob(csr.getColumnIndex(s)));
break;
}
}
copiedDB.update(
DatabaseConstants.QUESTION_TABLENAME,
cv,
DatabaseConstants.QUESTION_ID_COLUMN + "=?",
new String[]{
String.valueOf(
csr.getLong(
csr.getColumnIndex(DatabaseConstants.QUESTION_ID_COLUMN
)
)
)
}
);
}
copiedDB.setTransactionSuccessful();
copiedDB.endTransaction();
csr.close();
//6. Cleanup
copiedDB.close();
preservedDB.close();
preservedDBPath.delete();
}
private static void copyAssetFile(Context context) {
int buffer_size = 8192;
byte[] buffer = new byte[buffer_size];
int bytes_read = 0;
try {
InputStream fis = context.getAssets().open(DBNAME);
OutputStream os = new FileOutputStream(new File(context.getDatabasePath(DBNAME).getPath()));
while ((bytes_read = fis.read(buffer)) > 0) {
os.write(buffer,0,bytes_read);
}
os.flush();
os.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException("Unable to copy from assets");
}
}
private static int getDBVersion(Context context, String databaseName) {
SQLiteDatabase db = SQLiteDatabase.openDatabase( context.getDatabasePath(databaseName).getPath(),null,SQLiteDatabase.OPEN_READONLY);
int rv = db.getVersion();
db.close();
return rv;
}
}
This manages the Asset File copy (in this case directly from the assets folder) outside of Room and before the database is built doing it's own version and database existence checking. Although ATTACH could be used, the solution keeps the original and the new databases seperate when updating the new using a Cursor.
Some flexibility/adaptability has been included in that the columns to be preserved can be expanded upon. In the test runs DatabaseConstants includes :-
public static final String[] PRESERVED_COLUMNS = new String[]
{
QUESTION_SOLVED_COLUMN
};
public static final String[] EXTRACT_COLUMNS = new String[]
{
QUESTION_ID_COLUMN,
QUESTION_SOLVED_COLUMN
};
thus additional columns to be preserved can be added (of any type as per 5. in the copyFromAssets method).
The columns to be extracted can also be specified, in the case above, the ID column uniquely identifies the question so that is extracted in addition to the solved column for use by the WHERE clause.
Testing
The above has been tested to :-
Original
Copy the first version of the database from the assets when DBVERSION is 1.
Note that this originaly contains 3 questions as per
Part of the code (in the invoking activity checks to to see if all the solved values are 0 if so, then it chages the solved status of the question with an id of 2)
Not copy the database, but use the existing database when DBVERSION is 1 on a subseuent run(s).
ID 2 remains solved.
New
After renaming the original asset from to be prefixed with original_, editing the database to be as below and after copying it to the assets file :-
Without changing the DBVERSION (still 1) run and the original database is still in use.
After changing DBVERSION to 2 running copies the changed asset file and restores/preserves the solved status.
For subsequent runs the solved status for the new data remains.
For testing the invoking activity consisted of :-
public class MainActivity extends AppCompatActivity {
QuestionDatabase questionDatabase;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
questionDatabase = QuestionDatabase.getInstance(this);
int solvedCount = 0;
for (Question q: questionDatabase.questionsDao().getAll()) {
if (q.isSolved()) solvedCount++;
q.logQuestion();
}
if (solvedCount == 0) {
questionDatabase.questionsDao().setSolved(true,2);
}
for (Question q: questionDatabase.questionsDao().getAll()) {
q.logQuestion();
}
}
}
For each run it outputs all of the questions to the log twice. After the first if there are no solved questions it solves the question with an id of 2.
The output from the last run was :-
2020-01-08 09:14:37.689 D/QUESTIONINFO: ID is 1 Question is Editted What is x
Answers Are :-
a
b
x
Correct Answer is 3
Is Solved false
2020-01-08 09:14:37.689 D/QUESTIONINFO: ID is 2 Question is Edited What is a
Answers Are :-
a
b
c
Correct Answer is 1
Is Solved false
2020-01-08 09:14:37.689 D/QUESTIONINFO: ID is 3 Question is Edited What is b
Answers Are :-
a
b
c
Correct Answer is 2
Is Solved false
2020-01-08 09:14:37.689 D/QUESTIONINFO: ID is 4 Question is New Question What is d
Answers Are :-
e
f
d
Correct Answer is 3
Is Solved false
2020-01-08 09:14:37.692 D/QUESTIONINFO: ID is 1 Question is Editted What is x
Answers Are :-
a
b
x
Correct Answer is 3
Is Solved false
2020-01-08 09:14:37.692 D/QUESTIONINFO: ID is 2 Question is Edited What is a
Answers Are :-
a
b
c
Correct Answer is 1
Is Solved true
2020-01-08 09:14:37.692 D/QUESTIONINFO: ID is 3 Question is Edited What is b
Answers Are :-
a
b
c
Correct Answer is 2
Is Solved false
2020-01-08 09:14:37.693 D/QUESTIONINFO: ID is 4 Question is New Question What is d
Answers Are :-
e
f
d
Correct Answer is 3
Is Solved false
Additional - Improved Version
This is an approved version that caters for multiple tables and columns. To cater for tables a class TablePreserve has been added that allows a table, the columns to preserve, the columns to extract and the columns for the where clause. As per :-
public class TablePreserve {
String tableName;
String[] preserveColumns;
String[] extractColumns;
String[] whereColumns;
public TablePreserve(String table, String[] preserveColumns, String[] extractColumns, String[] whereColumns) {
this.tableName = table;
this.preserveColumns = preserveColumns;
this.extractColumns = extractColumns;
this.whereColumns = whereColumns;
}
public String getTableName() {
return tableName;
}
public String[] getPreserveColumns() {
return preserveColumns;
}
public String[] getExtractColumns() {
return extractColumns;
}
public String[] getWhereColumns() {
return whereColumns;
}
}
You create an Array of TablePreserve objects and they are looped through e.g.
public final class DatabaseConstants {
public static final String DBNAME = "question.db";
public static final int DBVERSION = 2;
public static final String QUESTION_TABLENAME = "question";
public static final String QUESTION_ID_COLUMN = "id";
public static final String QUESTION_QUESTION_COLUMN = QUESTION_TABLENAME;
public static final String QUESTION_ANSWER1_COLUMN = "answer1";
public static final String QUESTION_ANSWER2_COLUMN = "answer2";
public static final String QUESTION_ANSWER3_COLUMN = "answer3";
public static final String QUESTION_CORRECTANSWER_COLUMN = "correctAsnwer";
public static final String QUESTION_SOLVED_COLUMN = "solved";
public static final TablePreserve questionTablePreserve = new TablePreserve(
QUESTION_TABLENAME,
new String[]{QUESTION_SOLVED_COLUMN},
new String[]{QUESTION_ID_COLUMN,QUESTION_SOLVED_COLUMN},
new String[]{QUESTION_ID_COLUMN}
);
public static final TablePreserve[] TABLE_PRESERVELIST = new TablePreserve[] {
questionTablePreserve
};
}
Then QuestionsDatabase becomes :-
#Database(version = DatabaseConstants.DBVERSION, entities = {Question.class})
public abstract class QuestionDatabase extends RoomDatabase {
static final String DBNAME = DatabaseConstants.DBNAME;
abstract QuestionDao questionsDao();
public static QuestionDatabase getInstance(Context context) {
if (!doesDatabaseExist(context)) {
copyFromAssets(context,false);
}
if (getDBVersion(context, DatabaseConstants.DBNAME) < DatabaseConstants.DBVERSION) {
copyFromAssets(context, true);
}
return Room.databaseBuilder(context,QuestionDatabase.class,DBNAME)
.addCallback(callback)
.allowMainThreadQueries()
.addMigrations(Migration_1_2)
.build();
}
private static RoomDatabase.Callback callback = new Callback() {
#Override
public void onCreate(#NonNull SupportSQLiteDatabase db) {
super.onCreate(db);
}
#Override
public void onOpen(#NonNull SupportSQLiteDatabase db) {
super.onOpen(db);
}
#Override
public void onDestructiveMigration(#NonNull SupportSQLiteDatabase db) {
super.onDestructiveMigration(db);
}
};
private static Migration Migration_1_2 = new Migration(1, 2) {
#Override
public void migrate(#NonNull SupportSQLiteDatabase database) {
}
};
private static boolean doesDatabaseExist(Context context) {
if (new File(context.getDatabasePath(DBNAME).getPath()).exists()) return true;
if (!(new File(context.getDatabasePath(DBNAME).getPath()).getParentFile()).exists()) {
new File(context.getDatabasePath(DBNAME).getPath()).getParentFile().mkdirs();
}
return false;
}
private static void copyFromAssets(Context context, boolean replaceExisting) {
boolean dbExists = doesDatabaseExist(context);
if (dbExists && !replaceExisting) return;
//First Copy
if (!replaceExisting) {
copyAssetFile(context);
setDBVersion(context,DBNAME,DatabaseConstants.DBVERSION);
return;
}
//Subsequent Copies
File originalDBPath = new File(context.getDatabasePath(DBNAME).getPath());
// Open and close the original DB so as to checkpoint the WAL file
SQLiteDatabase originalDB = SQLiteDatabase.openDatabase(originalDBPath.getPath(),null,SQLiteDatabase.OPEN_READWRITE);
originalDB.close();
//1. Rename original database
String preservedDBName = "preserved_" + DBNAME;
File preservedDBPath = new File (originalDBPath.getParentFile().getPath() + File.separator + preservedDBName);
(new File(context.getDatabasePath(DBNAME).getPath()))
.renameTo(preservedDBPath);
//2. Copy the replacement database from the assets folder
copyAssetFile(context);
//3. Open the newly copied database
SQLiteDatabase copiedDB = SQLiteDatabase.openDatabase(originalDBPath.getPath(),null,SQLiteDatabase.OPEN_READWRITE);
SQLiteDatabase preservedDB = SQLiteDatabase.openDatabase(preservedDBPath.getPath(),null,SQLiteDatabase.OPEN_READONLY);
//4. Apply preserved data to the newly copied data
copiedDB.beginTransaction();
for (TablePreserve tp: DatabaseConstants.TABLE_PRESERVELIST) {
preserveTableColumns(
preservedDB,
copiedDB,
tp.getTableName(),
tp.getPreserveColumns(),
tp.getExtractColumns(),
tp.getWhereColumns(),
true
);
}
copiedDB.setVersion(DatabaseConstants.DBVERSION);
copiedDB.setTransactionSuccessful();
copiedDB.endTransaction();
//5. Cleanup
copiedDB.close();
preservedDB.close();
preservedDBPath.delete();
}
private static void copyAssetFile(Context context) {
int buffer_size = 8192;
byte[] buffer = new byte[buffer_size];
int bytes_read = 0;
try {
InputStream fis = context.getAssets().open(DBNAME);
OutputStream os = new FileOutputStream(new File(context.getDatabasePath(DBNAME).getPath()));
while ((bytes_read = fis.read(buffer)) > 0) {
os.write(buffer,0,bytes_read);
}
os.flush();
os.close();
fis.close();
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException("Unable to copy from assets");
}
}
private static int getDBVersion(Context context, String databaseName) {
SQLiteDatabase db = SQLiteDatabase.openDatabase( context.getDatabasePath(databaseName).getPath(),null,SQLiteDatabase.OPEN_READONLY);
int rv = db.getVersion();
db.close();
return rv;
}
private static void setDBVersion(Context context, String databaseName, int version) {
SQLiteDatabase db = SQLiteDatabase.openDatabase( context.getDatabasePath(databaseName).getPath(),null,SQLiteDatabase.OPEN_READWRITE);
db.setVersion(version);
db.close();
}
private static boolean preserveTableColumns(
SQLiteDatabase originalDatabase,
SQLiteDatabase newDatabase,
String tableName,
String[] columnsToPreserve,
String[] columnsToExtract,
String[] whereClauseColumns,
boolean failWithException) {
StringBuilder sb = new StringBuilder();
Cursor csr = originalDatabase.query("sqlite_master",new String[]{"name"},"name=? AND type=?",new String[]{tableName,"table"},null,null,null);
if (!csr.moveToFirst()) {
sb.append("\n\tTable ").append(tableName).append(" not found in database ").append(originalDatabase.getPath());
}
csr = newDatabase.query("sqlite_master",new String[]{"name"},"name=? AND type=?",new String[]{tableName,"table"},null,null,null);
if (!csr.moveToFirst()) {
sb.append("\n\tTable ").append(tableName).append(" not found in database ").append(originalDatabase.getPath());
}
if (sb.length() > 0) {
if (failWithException) {
throw new RuntimeException("Both databases are required to have a table named " + tableName + sb.toString());
}
return false;
}
for (String pc: columnsToPreserve) {
boolean preserveColumnInExtractedColumn = false;
for (String ec: columnsToExtract) {
if (pc.equals(ec)) preserveColumnInExtractedColumn = true;
}
if (!preserveColumnInExtractedColumn) {
if (failWithException) {
StringBuilder sbpc = new StringBuilder().append("Column in Columns to Preserve not found in Columns to Extract. Cannot continuue." +
"\n\tColumns to Preserve are :-");
}
throw new RuntimeException("Column " + pc + " is not int the Columns to Extract.");
}
return false;
}
sb = new StringBuilder();
for (String c: whereClauseColumns) {
sb.append(c).append("=? ");
}
String[] whereargs = new String[whereClauseColumns.length];
csr = originalDatabase.query(tableName,columnsToExtract,sb.toString(),whereClauseColumns,null,null,null);
ContentValues cv = new ContentValues();
while (csr.moveToNext()) {
cv.clear();
for (String pc: columnsToPreserve) {
switch (csr.getType(csr.getColumnIndex(pc))) {
case Cursor.FIELD_TYPE_INTEGER:
cv.put(pc,csr.getLong(csr.getColumnIndex(pc)));
break;
case Cursor.FIELD_TYPE_STRING:
cv.put(pc,csr.getString(csr.getColumnIndex(pc)));
break;
case Cursor.FIELD_TYPE_FLOAT:
cv.put(pc,csr.getDouble(csr.getColumnIndex(pc)));
break;
case Cursor.FIELD_TYPE_BLOB:
cv.put(pc,csr.getBlob(csr.getColumnIndex(pc)));
}
}
int waix = 0;
for (String wa: whereClauseColumns) {
whereargs[waix] = csr.getString(csr.getColumnIndex(wa));
}
newDatabase.update(tableName,cv,sb.toString(),whereargs);
}
csr.close();
return true;
}
}
I debugged and modified the code by MikeT a bit, now heres the final kotlin library with an easy databaseBuilder
https://github.com/ueen/RoomAssetHelper
Please read the documentation and report if you encounter any issue, enjoy :)
I'm using SQLCipher on Android to get an encrypted database. The DB gets an default password which is based on some hardcoded value. The user can also set a pincode on the app. When a user does that I also want change the DB password being used by SQLCipher. I already found a few post on StackerOverflow said that I should use rekey.
Currently I'm using this code like the posts suggested
final DatabaseHelper helper = new DatabaseHelper(this);
final SQLiteDatabase db = helper.getWritableDatabase(oldPassword);
final String PRAGMA_KEY = String.format("PRAGMA key = \"%s\";", oldPassword);
final String PRAGMA_REKEY = String.format("PRAGMA rekey = \"%s\";", newPassword);
db.rawExecSQL("BEGIN IMMEDIATE TRANSACTION;");
db.rawExecSQL(PRAGMA_KEY);
db.rawExecSQL(PRAGMA_REKEY);
db.close();
But when I try to insert into the DB after the password should have changed I get this error.
sqlite returned: error code = 26, msg = statement aborts at 1: [BEGIN EXCLUSIVE;] file is encrypted or is not a database
Failure 26 (file is encrypted or is not a database) on 0xb90048c0 when executing 'BEGIN EXCLUSIVE;'
FATAL EXCEPTION: IntentService[DBService]
Process: com.example, PID: 26502
net.sqlcipher.database.SQLiteException: file is encrypted or is not a database: BEGIN EXCLUSIVE;
at net.sqlcipher.database.SQLiteDatabase.native_execSQL(Native Method)
at net.sqlcipher.database.SQLiteDatabase.execSQL(SQLiteDatabase.java:1831)
at net.sqlcipher.database.SQLiteDatabase.beginTransactionWithListener(SQLiteDatabase.java:584)
at net.sqlcipher.database.SQLiteDatabase.beginTransaction(SQLiteDatabase.java:538)
at com.example.db.Provider.bulkInsert(OurProvider.java:196)
at android.content.ContentProvider$Transport.bulkInsert(ContentProvider.java:250)
at android.content.ContentResolver.bulkInsert(ContentResolver.java:1268)
at nl.qbusict.cupboard.ProviderCompartment.put(ProviderCompartment.java:158)
at com.example.db.DBUtils.saveObjects(DBUtils.java:32)
at com.example.services.DBService.getDataFromAPI(DBService.java:119)
at com.example.services.DBService.onHandleIntent(DBService.java:48)
at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:65)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:135)
at android.os.HandlerThread.run(HandlerThread.java:61)
I checked both the original password and the new password before and after the change they where all the same. I also tried adding db.rawExecSQL("END;"); and db.rawExecSQL("COMMIT;"); but that didn't help. Also I see a log msg come by error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt when I change the password. Don't know if that has anything to do with it?
public class OurProvider extends ContentProvider {
#Override
public int bulkInsert(Uri uri, ContentValues[] values) {
synchronized (LOCK) {
SQLiteDatabase db = getDatabase().getWritableDatabase(getPassword());
final String table = getTableString(uri);
db.beginTransaction();
int rowsInserted = 0;
try {
for (ContentValues value : values) {
db.insertWithOnConflict(table, null, value, SQLiteDatabase.CONFLICT_REPLACE);
rowsInserted++;
}
db.setTransactionSuccessful();
} catch (Exception e) {
Crashlytics.logException(e);
Log.d(TAG, Log.getStackTraceString(e));
rowsInserted = -1;
} finally {
db.endTransaction();
if (rowsInserted > 0) {
getContext().getContentResolver().notifyChange(uri, null);
}
}
return rowsInserted;
}
}
private String getPassword() {
final String password = Base64.encodeToString(OurApplication.getEncryptionKey().getEncoded(), Base64.DEFAULT);
Log.e("SQLCipher_OurProvider", "SQLCipher password: " + password);
return password;
}
private void initDb() {
SQLiteDatabase.loadLibs(getContext());
mDatabaseHelper = new DatabaseHelper(getContext());
}
public DatabaseHelper getDatabase() {
if (mDatabaseHelper == null) {
initDb();
}
return mDatabaseHelper;
}
}
public class DBService extends IntentService {
private void updatePasscode(Intent intent) {
final SecretKey oldKey = OurApplication.getEncryptionKey();
final String encryptedString = SecurePrefs.getEncryptedString(PrefKeys.ENC_CONFIRM);
if (!TextUtils.isEmpty(encryptedString)) {
String decryptedString = Crypto.decrypt(oldKey, encryptedString);
// check if the oldkey can decrypt the confirmation string.
if (BaseActivity.CONFIRM_STRING.equals(decryptedString)) {
String pin = intent.getStringExtra(KEY_PASSCODE);
if (pin.equals(PasscodeActivity.REMOVE_PIN)) {
pin = null;
}
final SecretKey newKey = SecurityUtil.generateKey(this, pin);
final String accessToken = getUserAccessToken();
final String refreshToken = SecurePrefs.getString(PrefKeys.USER_REFRESH_TOKEN);
final String email = SecurePrefs.getString(PrefKeys.USER_ID);
final String confirmEncrypted = SecurePrefs.getString(PrefKeys.ENC_CONFIRM);
// set the newly generated string in the application.
OurApplication.setEncryptionKey(newKey);
// clear the old encrypted prefs. save the values with the new encryption key.
SecurePrefs.clear();
SecurePrefs.putString(PrefKeys.USER_ACCESS_TOKEN, accessToken);
SecurePrefs.putString(PrefKeys.USER_REFRESH_TOKEN, refreshToken);
SecurePrefs.putString(PrefKeys.USER_ID, email);
SecurePrefs.putString(PrefKeys.ENC_CONFIRM, confirmEncrypted);
// update de encryption key in the database.
final String oldPassword = Base64
.encodeToString(oldKey.getEncoded(), Base64.DEFAULT);
final String newPassword = Base64
.encodeToString(newKey.getEncoded(), Base64.DEFAULT);
final String PRAGMA_KEY = String.format("PRAGMA key = \"%s\";", oldPassword);
final String PRAGMA_REKEY = String.format("PRAGMA rekey = \"%s\";", newPassword);
final DatabaseHelper helper = new DatabaseHelper(this);
final SQLiteDatabase db = helper.getWritableDatabase(oldPassword);
db.rawExecSQL("BEGIN IMMEDIATE TRANSACTION;");
db.rawExecSQL(PRAGMA_KEY);
db.rawExecSQL(PRAGMA_REKEY);
db.close();
sendBroadcast(IntentUtil.createBroadcastPasscodeUpdated());
}
}
}
}
You should not begin a transaction (this is handled internally and slightly different), nor do you need to perform the PRAGMA key='…'; portion of your code either. You can should just open the database with your call to getWritableDatabase(…); and then execute your PRAGMA rekey='…'; command.
I also had this problem and the reason turned out to be that the DB wasn't encrypted at the first place, so then PRAGMA key = <new_key> or PRAGMA rekey = <new_key> failed. So, the oldPassword CAN'T be null or empty.
I had a problem which I already solved but I still wants to know WHY
the solution solved it.
I wrote an android app that had a sqlite db after a couple of times I debugged it
The oncreate method in the db didnt got called (even though everything worked fine before)
After I changed the db version number from 1 to 2 everything worked fine again
Even though I uninstalled the app through the app manager and also removed the cache and
The local database information.
My question is as follows - does the local database data is saved somewhere else?
In case it doesn't - Why did it worked only after I upgraded the version number
not even when I erased all the app related data?
/**
* A class to handle sqlite reads/writes of user related data to be collected
*/
public class UserDataManager extends SQLiteOpenHelper {
// Class Variables
private final String TAG = UserDataManager.class.getSimpleName();
// Database Version
private static final int DATABASE_VERSION = 1;
// Database Name
public static final String DATABASE_NAME = "tmc";
// Tables
private static final String TABLE_USER = "user";
// Tables and table columns names
private String CREATE_USER_TABLE;
private static final String COLUMN_USER_ID = "user_id";
private static final String COLUMN_USER_MAIL = "email";
private static final String COLUMN_USER_ACTIVE = "user_active";
private static final String COLUMN_USER_NAME = "name";
private static final String COLUMN_USER_PASSWORD = "password";
private static final String COLUMN_USER_PHONE_NUMBER = "phone_number";
/**
* Class constructor
*
* #param context
* The context to run in
*/
public UserDataManager(Context context) {
super(context, DATABASE_NAME, null, DATABASE_VERSION);
}
// Creating Tables
#Override
public void onCreate(SQLiteDatabase db) {
CREATE_USER_TABLE = "CREATE TABLE IF NOT EXISTS " + TABLE_USER + " ("
+ COLUMN_USER_ID + " INTEGER PRIMARY KEY NOT NULL, "
+ COLUMN_USER_MAIL + " VARCHAR(64) NOT NULL, "
+ COLUMN_USER_NAME + " VARCHAR(64) NOT NULL, "
+ COLUMN_USER_PASSWORD + " VARCHAR(64) NOT NULL, "
+ COLUMN_USER_PHONE_NUMBER + " VARCHAR(64) NOT NULL, "
+ COLUMN_USER_ACTIVE + " INT NOT NULL);";
// create the tables
db.execSQL(CREATE_USER_TABLE);
}
// Upgrading database
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
// Drop older table if existed
db.execSQL("DROP TABLE IF EXISTS " + TABLE_USER);
// Create tables again
onCreate(db);
}
/**
* Adding a user to the database
*
* #param userId
* The created user id
* #param userName
* The user name
* #param userEmail
* The user email
* #param userPassword
* The user password
* #param userPhoneNumber
* The user phone number
* #param isActive
* Set to 1 if the user is active 0 otherwise
* #return True if the user added successfully false otherwise
*/
public boolean AddUser(int userId, String userName, String userEmail,
String userPassword, String userPhoneNumber, boolean isActive) {
// method variables
long rowId;
boolean pass = false;
int active = isActive ? 1 : 0;
SQLiteDatabase db = null;
ContentValues row = null;
// try to add the user to the db
try {
row = new ContentValues();
db = this.getWritableDatabase();
db.delete(TABLE_USER, null, null);
row.put(COLUMN_USER_ID, userId);
row.put(COLUMN_USER_NAME, userName);
row.put(COLUMN_USER_MAIL, userEmail);
row.put(COLUMN_USER_PASSWORD, userPassword);
row.put(COLUMN_USER_CAR_NUMBER, userPhoneNumber);
row.put(COLUMN_USER_ACTIVE, active);
rowId = db.insert(TABLE_USER, null, row);
if (rowId > -1) {
pass = true;
}
} catch (SQLException exception) {
Log.e(TAG, exception.getMessage());
} finally {
if (db != null) {
// close database connection
db.close();
}
}
return pass;
}
/**
* Get the current registered user
*
* #return The id of the column of the registered user
*/
public int GetRegisteredUserId() {
// method variables
int columnIndex = -1;
int userId = -1;
SQLiteDatabase db = null;
Cursor cursor = null;
// try to get the user from the database
try {
db = this.getReadableDatabase();
cursor = db.query(TABLE_USER, new String[] { COLUMN_USER_ID },
null, null, null, null, null);
if (cursor != null) {
boolean moved = cursor.moveToFirst();
if (moved) {
columnIndex = cursor.getColumnIndex(COLUMN_USER_ID);
if (columnIndex > -1) {
userId = cursor.getInt(columnIndex);
}
}
}
} catch (SQLException exception) {
Log.e(TAG, exception.getMessage());
} finally {
if (cursor != null)
// release cursor
cursor.close();
if (db != null)
// close database connection
db.close();
}
return userId;
}
/**
* Get the current user email
*
* #return The id of the column of the registered user
*/
public String GetRegisteredUserEmail() {
// method variables
int columnIndex = -1;
String userEmail = null;
SQLiteDatabase db = null;
Cursor cursor = null;
// try to get the user from the database
try {
db = this.getReadableDatabase();
cursor = db.query(TABLE_USER, new String[] { COLUMN_USER_MAIL },
null, null, null, null, null);
if (cursor != null) {
boolean moved = cursor.moveToFirst();
if (moved) {
columnIndex = cursor.getColumnIndex(COLUMN_USER_MAIL);
if (columnIndex > -1) {
userEmail = cursor.getString(columnIndex);
}
}
}
} catch (SQLException exception) {
Log.e(TAG, exception.getMessage());
} finally {
if (cursor != null)
// release cursor
cursor.close();
if (db != null)
// close database connection
db.close();
}
return userEmail;
}
/**
* Get the current user password
*
* #return The password of the current logged user
*/
public String GetRegisteredUserPassword() {
// method variables
int columnIndex = -1;
String userPassword = null;
SQLiteDatabase db = null;
Cursor cursor = null;
// try to get the user from the database
try {
db = this.getReadableDatabase();
cursor = db.query(TABLE_USER,
new String[] { COLUMN_USER_PASSWORD }, null, null, null,
null, null);
if (cursor != null) {
boolean moved = cursor.moveToFirst();
if (moved) {
columnIndex = cursor.getColumnIndex(COLUMN_USER_PASSWORD);
if (columnIndex > -1) {
userPassword = cursor.getString(columnIndex);
}
}
}
} catch (SQLException exception) {
Log.e(TAG, exception.getMessage());
} finally {
if (cursor != null)
// release cursor
cursor.close();
if (db != null)
// close database connection
db.close();
}
return userPassword;
}
/**
* Get number of rows in the user table
*
* #return the number of the rows in the user table (How many users are
* saved in the DB)
*/
public int GetRowCount() {
// method variables
int rowsCount = 0;
SQLiteDatabase db = null;
Cursor cursor = null;
// try to get the user from the database
try {
db = this.getReadableDatabase();
cursor = db.query(TABLE_USER, null, null, null, null, null, null);
if (cursor != null) {
boolean moved = cursor.moveToFirst();
if (moved) {
do {
rowsCount++;
} while (cursor.moveToNext());
}
}
} catch (SQLException exception) {
Log.e(TAG, exception.getMessage());
} finally {
if (cursor != null)
// release cursor
cursor.close();
if (db != null)
// close database connection
db.close();
}
return rowsCount;
}
/**
* Remove a user from the database
*
* #param userId
* The user id
*/
public void LogoutUser() {
// method variables
SQLiteDatabase db = null;
// try to remove a user from the database
try {
db = this.getWritableDatabase();
onUpgrade(db, DATABASE_VERSION, DATABASE_VERSION);
} catch (SQLException exception) {
Log.e(TAG, exception.getMessage());
} finally {
if (db != null) {
// close database connection
db.close();
}
}
}
/**
* Set a user to be active or not
*
* #param isActive
* 1 if the cigarette is active 0 otherwise
* #return True if the cigarette active field has changed false otherwise
*/
public boolean SetUserActive(boolean isActive) {
// method variables
int rowsAffected;
int active = isActive ? 1 : 0;
long userId;
String userIdString;
boolean pass = true;
SQLiteDatabase db = null;
ContentValues values = null;
// try to remove a device from the database
try {
userId = GetRegisteredUserId();
if (userId > -1) {
userIdString = String.valueOf(userId);
db = this.getWritableDatabase();
values = new ContentValues();
values.put(COLUMN_USER_ACTIVE, active);
rowsAffected = db.update(TABLE_USER, values, COLUMN_USER_ID
+ " = ?", new String[] { userIdString });
if (rowsAffected != 1) {
pass = false;
}
}
} catch (SQLException exception) {
Log.e(TAG, exception.getMessage());
} finally {
if (db != null) {
// close database connection
db.close();
}
}
return pass;
}
}
Notes -
1. Please note that my device is rooted and so after inserting the data to the db im changing the permissions on the db file for 777 so I can pull it from the phone to see whats in it (i.e. did the query pass or not)
2. The error that is being thrown is "android.database.sqlite.SQLiteException: no such table: user "
Chocolate chips cookies will be granted for any answer... =)
Why did it worked only after I upgraded the version number not even when I erased all the app related data?
As soon as you start working with either of getReadableDatabase() ,getWriteableDatabase() or any other SQLiteHelper class code. The first method calls is onCreate(SQLiteDatabase db) which creates Database under your application database path
/data/data/PACKAGE_NAME/databases/tmc (in your case).
If you modify your Database structure in SQliteHelper the first method get called is onUpgrage() which checks whether Database_Version get modified or not. If it's then it execute onUpgrade() with series of DROP TABLE IF EXIST followed by onCreate() which again create your database with new structure under your application path by replacing your previous database file.
Clearing Cached data using Application Manager indeed clear database and cached data of that application. But SQLiteHelper did check for Database_Version with old and new one. If new one is greater than old one. It does call onUpgrage() followed by onCreate().
When you intent to use Database with Android Application it get store under /data/data/PACKAGE_NAME/databases/tmc with application process security. Unable to access database file unless you have rooted Android device in which you already have.
One can create Developer Options or anything you like just to pull database from your application process to SD Card for unrooted devices.
Copy database file from application process path to SD Card for unrooted devices.
try {
File sd = Environment.getExternalStorageDirectory();
File data = Environment.getDataDirectory();
if (sd.canWrite()) {
String currentDBPath = "/data/data/" + getPackageName() + "/databases/ZnameDB"; //Your DATABASE_NAME
String backupDBPath = "ZnameDB_Dev.db"; //DATABASE_COPY_NAME UNDER SDCARD
File currentDB = new File(currentDBPath);
File backupDB = new File(sd, backupDBPath);
if (currentDB.exists()) {
FileChannel src = new FileInputStream(currentDB).getChannel();
FileChannel dst = new FileOutputStream(backupDB).getChannel();
dst.transferFrom(src, 0, src.size());
src.close();
dst.close();
Toast.makeText(SettingsActivity.this, "Database Transfered!", Toast.LENGTH_SHORT).show();
}
}
} catch (Exception e) {
Log.e(TAG, e.toString());
}
Answering your first question, all data is stored under YOUR_PACKAGE/databases/DATABASE.db only.
If you erase app through app manager all data is removed, just package remains. If you uninstall your app everything is cleared including package folder. Even if you set for your app install location to external SD card, database is stored internally anyway.
From documentation:
The .apk file is saved on the external storage, but all private user
data, databases, optimized .dex files, and extracted native code are
saved on the internal device memory.
SQLiteOpenHelper logic is simple:
checks if DB exists, if not new DB is created
retrieve DB version, initial value is 0, that's why in your app minimum value for DB version is 1 to call onCreate() method at least once
if version is equal to 0, onCreate() is called, or
if version is different than the one provided by your code onUpgrade() or onDowngrade() is called
So, whenever you upgrade your scheme, version number MUST be increased, there is no argue about that, in order to allow your app work properly.
Now, in your specific case I can only guess. I would say that erasing your package was not entirely successful and bits of data was left, especially if you mentioned that you did some manual modification on DB file. Maybe it has something to do with Android version running on your device but you didn't mentioned which one is it.
That's all. I hope my answer is satisfying.
Can you exec PRAGMA user_version; in your adb to get the db version? According to the source code of SQLiteOpenHelper, SQLite.getVersion() equals to SQLiteOpenHelper.mNewVersion, so onCreate() method won't be invoked. When you chmod 777 on db file, the user_version will be modified too.
Assuming the databases are not deleted while uninstalling app, this seems plausible. The databases are stored here DDMS/data/data/PACKAGE_NAME/databases/YOUR_DB_FILE. You can only see this if your phone is rooted.
Please check whether this assumption is true and correct me if I am wrong.
Thanks
Alright... I've had enough.
I'm thoroughly frustrated.
So I'd rather ask for help instead of a new monitor.
...And those are VERY expensive here.
Long story short... I have a database. And a table.
private String DEFINE_PROP_TYPES = "CREATE TABLE " + TABLE_PROP_TYPES + "("
+ TABLE_ID + " INTEGER PRIMARY KEY, "
+ TABLE_PROP_TYPE_NAME + " TEXT NOT NULL"
+ ")";
With an 'Adapter' class thrown in for good measure to manage it.
public abstract class DBAdapter
{
static public final String C_COLUMN_ID = "_id";
protected Context context;
protected DBHelper dbHelper;
protected SQLiteDatabase db;
protected String managedTable;
protected String[] columns;
public String getTableManaged()
{
return managedTable;
}
public void setTableManaged(String managedTable)
{
this.managedTable = managedTable;
}
public void setColumns(String[] columns)
{
this.columns = columns;
}
public DBAdapter(Context context)
{
this.context = context;
}
public void close()
{
dbHelper.close();
}
public DBAdapter open() throws SQLException
{
dbHelper = new DBHelper(context);
db = dbHelper.getWritableDatabase();
return this;
}
public Cursor getList()
{
Cursor c = db.query(true, managedTable, columns, null, null, null, null, null, null);
return c;
}
public long insert(ContentValues reg)
{
return 0;
}
}
public class PropTypesDBAdapter extends DBAdapter
{
static public final String C_TABLE_PROP_TYPES = "PROP_TYPES";
static public final String C_COLUMN_ID = "_id",
C_COLUMN_PROP_TYPES_NAME = "re_prop_type";
public PropTypesDBAdapter(Context context)
{
super(context);
this.setTableManaged(C_TABLE_PROP_TYPES);
this.setColumns(new String[] { C_COLUMN_ID,
C_COLUMN_PROP_TYPES_NAME });
}
public long insert(ContentValues reg)
{
if (db == null)
{
open();
}
return db.insert(C_TABLE_PROP_TYPES, null, reg);
}
}
And besides this pile of cute I have an activity class.
With spinners.
public class PropDetailActivity extends Activity implements LocationListener
{
// insert here some blah-blah constants not needed by spinners
private PropDBAdapter mHouses;
private RatingsDBAdapter mRatings;
private PropTypesDBAdapter mPropTypes;
private Cursor mCursorHouses,
mCursorRatings,
mCursorPropTypes;
long mPropType;
private long mPropId;
private Spinner spinnerRating, spinnerType;
AdapterView.OnItemSelectedListener spnLstPropType, spnLstRating;
protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_house_detail);
Intent intent = getIntent();
Bundle extra = intent.getExtras();
if (extra == null)
{
return;
}
// Figure all view widgets being retrieved here, including...
spinnerRating = (Spinner) findViewById(R.id.spinnerRating);
spinnerType = (Spinner) findViewById(R.id.spinnerType);
// Create adapter and cursor-y things here
mHouses = new PropDBAdapter(this);
mHouses.open();
// And now, for the juicy, deliciously irritating stuff:
String[] from = new String[] { PropTypesDBAdapter.C_COLUMN_PROP_TYPES_NAME };
int[] to = new int[] { android.R.id.text1 };
mPropTypes = new PropTypesDBAdapter(this);
mPropTypes.open();
mCursorPropTypes = mPropTypes.getList();
#SuppressWarnings("deprecation")
SimpleCursorAdapter adapterPropTypes = new SimpleCursorAdapter(this,
android.R.layout.simple_spinner_item,
mCursorPropTypes,
from, /*new String[] { RatingsDBAdapter.C_COLUMN_RATING_NAME }, */
to); /*new int[] { android.R.id.text1 } */
adapterPropTypes.setDropDownViewResource(android.R.layout.simple_spinner_dropdown_item);
spinnerType.setAdapter(adapterPropTypes);
spinnerRating.setSelection(pos);
spnLstPropType = new AdapterView.OnItemSelectedListener()
{
#Override
public void onItemSelected(AdapterView<?> parent, View view,
int pos, long id)
{
mPropType = id;
}
#Override
public void onNothingSelected(AdapterView<?> arg0) { }
};
spinnerType.setOnItemSelectedListener(spnLstPropType);
private int getItemPositionById(Cursor c, long id, DBAdapter adapter)
{
for (c.moveToFirst(); !c.isAfterLast(); c.moveToNext())
{
if (c.getLong(c.getColumnIndex(DBAdapter.C_COLUMN_ID)) == id)
{
return c.getPosition();
}
}
return 0;
}
private void query(long id)
{
mCursorHouses = mHouses.getRecord(id);
// Figure values being retrieved and set on their widgets instead of this comment... and now...
mPropType = mCursorHouses.getInt(mCursorHouses.getColumnIndex(PropDBAdapter.C_PROP_TYPE_ID));
spinnerType.setSelection(
getItemPositionById(
mCursorRatings,
mCursorHouses.getColumnIndex(PropDBAdapter.C_PROP_TYPE_ID),
mPropTypes
)
);
private void save()
{
ContentValues reg = new ContentValues();
// Read: values being put into 'reg'... eventually it should reach this:
reg.put(PropDBAdapter.C_PROP_TYPE_ID, mPropType);
try
{
if (mFormMode == PropListActivity.C_CREATE)
{
mHouses.insert(reg);
Toast.makeText(PropDetailActivity.this, R.string.house_create_notice, Toast.LENGTH_LONG).show();
}
else if (mFormMode == PropListActivity.C_EDIT)
{
Toast.makeText(PropDetailActivity.this, R.string.house_edit_notice, Toast.LENGTH_LONG).show();
reg.put(PropDBAdapter.C_COLUMN_ID, mPropId);
long resultCode = mHouses.update(reg);
Log.i(this.getClass().toString(), "Database operation result code: " + resultCode);
}
}
catch(SQLException e)
{
Log.i(this.getClass().toString(), e.getMessage());
}
setResult(RESULT_OK);
finish();
}
}
Spinners are being bad boys. Lazy bad boys on top of that.
They do load up the data -a list of real estate property types- they are meant to display.
After some spanking, that is.
But, hoping them to save THE VALUE YOU SELECT to SQLite? And to show THAT EXACT VALUE when fetching stuff back from the database?
Oh, no, no way no how.
They stubbornly stick to displaying always the same value upon activity startup.
So... please... I must draw upon your collective wisdom to save my sorry excuse for a project...
Pleasepleaseplease? :)
(IF you feel like diving into the whole uncut code, here's a GIT repository for you: https://github.com/CruxMDQ/Quoterv3)
Checking your code, I think I found the problem, change the following lines in your query method in PopDetailActivity.java.
For spinnerRating do:
spinnerRating.setSelection(
getItemPositionById(
mCursorRatings,
mCursorHouses.getInt(mCursorHouses.getColumnIndex(PropDBAdapter.C_PROP_RATING_ID)),
mRatings
)
);
and for spinnerType do:
spinnerType.setSelection(
getItemPositionById(
mCursorPropTypes,
mCursorHouses.getInt(mCursorHouses.getColumnIndex(PropDBAdapter.C_PROP_TYPE_ID)),
mPropTypes
)
);
EDIT:
In your query method, you initialize mPropTypeId, with the call to getItemPositionById() but in that call the first parameter should be mCursorPropTypes instead of mCursorHouses
A few things:
(1) I don't really see anywhere above where you actually create a SQLite database or use the SQLiteOpenHelper class to access that data. Take a look at this tutorial. It uses a simple single table set up to store data. Once you create the database it should be easy to read and write from it. Verify that you actually have a database created.
(2) Where are your SQL queries to return the data you're looking for? Even if data is being added you need to make sure you are getting the right data with your Cursor when you're done. If you're getting the same values each time is it possible that you are simply adding new data every time and retrieving the same value with your cursor - i.e. you're not telling the cursor to get the newly added data becuase you keep grabing the same index?
If you need to replace the data that's there you should be using update queries and not inserts.
I'm not pretty sure if this question is for here, but I want to ask all of you guys, who really can give me some advices of how to optimize better this piece of code, to run better in proper way and faster. The thing that I'm doing is that I'm downloading data over internet as JSON, parsing it and insert it in sqlite database. If the json string is not big, there is not a big problem for me, but when my json contains a lot of arrays and objects in some situations I'm waiting like 10-13 minutes to download/parse/insert all data in database, which is too much time.
The code that I'm showing is some kind of test code, because I was trying to implement InsertHelper to see if there will be a bit difference in speed, but the result is the same for now. Here is the code :
UserDatabaseHelper userDbHelper = RPCCommunicator.rpcUserDbHelper;
SQLiteDatabase db = userDbHelper.getWritableDatabase();
InsertHelper ih = new InsertHelper(db, "cards");
ih.prepareForInsert();
//ContentValues values = new ContentValues();
ContentValues valuess = new ContentValues();
try {
int objectid = ih.getColumnIndex("objectId");
ih.bind(objectid, objectId);
//values.put("objectId", objectId);
Log.d("", "ObjectId: " + objectId);
int objectoid = ih.getColumnIndex("objectOid");
ih.bind(objectoid, objectOid);
//values.put("objectOid", objectOid);
String jsonData = new String(cardBuffer, "UTF-8");
Log.d("JSONDATA", "JSONDATA VALID OR NOT : " + jsonData);
json = new JSONObject(jsonData);
JSONObject jsonObj = (JSONObject) new JSONTokener(jsonData).nextValue();
int collectionID = ih.getColumnIndex("collectionId");
int collectionId = Integer.parseInt(jsonObj.optString("collection_id","0"));
Log.d("Collection Id ", "Show Collection Id : " + collectionId);
if(collectionId!=0)
ih.bind(collectionID, collectionId);
//values.put("collectionId", collectionId);
int categoryID = ih.getColumnIndex("categoryId");
int categoryId = Integer.parseInt(jsonObj.optString("category_id", "0"));
Log.d("Category Id ", "Show Category Id : " + categoryId);
if(categoryId!=0)
ih.bind(categoryID, categoryId);
//values.put("categoryId", categoryId);
int dateCreated = ih.getColumnIndex("dateCreated");
String date = jsonObj.optString("date_created");
if(date!=null)
ih.bind(dateCreated, date);
//values.put("dateCreated", date);
int titlee = ih.getColumnIndex("title");
String title = jsonObj.optString("title");
Log.d("Title", "Show Title : " + title);
if(title!=null)
ih.bind(titlee, title);
//values.put("title", title);
// ... some other variables to get from JSON
JSONObject stats = jsonObj.optJSONObject("statistics");
if (jsonObj.has("statistics")) {
ContentValues values2 = new ContentValues();
InsertHelper ihr = new InsertHelper(db, "cardstats");
Iterator<Object> keys = stats.keys();
while (keys.hasNext()) {
ihr.prepareForInsert();
String key = (String) keys.next();
JSONObject obj = new JSONObject();
obj = stats.getJSONObject(key);
int paramId = Integer.parseInt(obj.optString("param_id"));
int cardIdTable = ihr.getColumnIndex("cardId");
ihr.bind(cardIdTable, objectId);
values2.put("cardId", objectId);
int statKey = ihr.getColumnIndex("statKeyId");
ihr.bind(statKey, paramId);
values2.put("statKeyId", paramId);
int catIdTable = ihr.getColumnIndex("catId");
int catId = Integer.parseInt(obj.optString("cat_id"));
ihr.bind(catIdTable, catId);
values2.put("catId", catId);
int paramtitle = ihr.getColumnIndex("title");
String paramTitle = obj.optString("param_title");
ihr.bind(paramtitle, paramTitle);
values2.put("title", paramTitle);
String cardstats = "SELECT cardId , statKeyId FROM cardstats WHERE cardId="+objectId+" AND statKeyId="+catId;
Cursor cardStats = userDbHelper.executeSQLQuery(cardstats);
if(cardStats.getCount()==0){
//userDbHelper.executeQuery("cardstats", values2);
ihr.execute();
} else {
for(cardStats.moveToFirst(); cardStats.moveToNext(); cardStats.isAfterLast()){
//int card = Integer.parseInt(cardStats.getString(cardStats.getColumnIndex("cardId")));
int statId = Integer.parseInt(cardStats.getString(cardStats.getColumnIndex("statKeyId")));
if(paramId != statId){
ihr.execute();
//userDbHelper.executeQuery("cardstats", values2);
} else {
userDbHelper.updateSQL("cardstats", values2, "cardId=?", new String[]{Integer.toString(objectId)});
}
}
}
cardStats.close();
//userDbHelper.executeQuery("cardstats", values2);
}
}// end if
String sql = "SELECT objectId FROM cards WHERE objectId = " + objectId;
Cursor cursor = userDbHelper.executeSQLQuery(sql);
if (cursor.getCount() == 0) {
ih.execute();
//userDbHelper.executeQuery("cards", values);
} else {
for (cursor.move(0); cursor.moveToNext(); cursor.isAfterLast()) {
int objectID = Integer.parseInt(cursor.getString(cursor.getColumnIndex("objectId")));
Log.d("","objectId : objectID - "+objectId+" "+objectID );
if (objectId != objectID) {
ih.execute();
//userDbHelper.executeQuery("cards", values);
} else if(objectId == objectID){
userDbHelper.updateSQL("cards", valuess, "objectId=?", new String[] {Integer.toString(objectId)});
}
}
}
cursor.close();
} catch (Exception e) {
e.printStackTrace();
Log.d("Error", ": " + e);
}
db.close();
return true;
}
*Edit: *
And here is how I save the binary data (images) which I get from internet :
public static void saveToExternalStorage(String servername, int userId, String filename, byte[] buffer){
try {
File myDir=new File("/sdcard/.Stampii/Users/"+servername+"/"+userId+"/Storage");
myDir.mkdirs();
File file = new File(myDir, filename);
FileOutputStream fos = new FileOutputStream(file);
fos.write(buffer);
fos.flush();
fos.close();
} catch (FileNotFoundException e){
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
So any kind of suggestions/advices are welcomed which will help me to improve this piece of code and make it run faster.
Thanks in advance!
Even if you have a lot of HTTP traffic (which you appear to have) you can still optimize your use of the database.
This naïve example that does 10000 inserts will show you the scale of improvement we're talking about here:
public class BombasticActivity extends Activity {
DBHelper mHelper;
SQLiteDatabase mDb;
InsertHelper mInsertHelper;
#Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.main);
mHelper = new DBHelper(this);
mDb = mHelper.getWritableDatabase();
mInsertHelper = new InsertHelper(mDb, "table1");
}
#Override
protected void onStart() {
super.onStart();
AsyncTask.SERIAL_EXECUTOR.execute(new MeasureTime(new Insert(10000, mInsertHelper)));
AsyncTask.SERIAL_EXECUTOR.execute(new MeasureTime(new DoInTransaction(mDb, new Insert(10000, mInsertHelper))));
}
#Override
protected void onDestroy() {
super.onDestroy();
mInsertHelper.close();
mDb.close();
mHelper.close();
}
static class MeasureTime implements Runnable {
final Runnable mAction;
MeasureTime(Runnable action) {
mAction = action;
}
public void run() {
final String name = mAction.getClass().getSimpleName();
System.out.println("Starting action (" + name + ")");
long t0 = System.currentTimeMillis();
try {
mAction.run();
} finally {
t0 = System.currentTimeMillis() - t0;
System.out.println("Time to complete action (" + name + "): " + t0 + "ms");
}
}
}
static class DoInTransaction implements Runnable {
final Runnable mAction;
final SQLiteDatabase mDb;
DoInTransaction(SQLiteDatabase db, Runnable action) {
mAction = action;
mDb = db;
}
public void run() {
mDb.beginTransaction();
try {
mAction.run();
mDb.setTransactionSuccessful();
} finally {
mDb.endTransaction();
}
}
}
static class Insert implements Runnable {
final int mNumberOfInserts;
final InsertHelper mInsertHelper;
Insert(int numberOfInserts, InsertHelper insertHelper) {
mNumberOfInserts = numberOfInserts;
mInsertHelper = insertHelper;
}
public void run() {
Random rnd = new Random(0xDEADBEEF);
ContentValues values = new ContentValues();
for (int i = 0; i < mNumberOfInserts; i++) {
values.put("text1", String.valueOf(rnd.nextDouble()));
values.put("text2", String.valueOf(rnd.nextFloat()));
values.put("text3", String.valueOf(rnd.nextLong()));
values.put("int1", rnd.nextInt());
mInsertHelper.insert(values);
if (i % 200 == 0) {
System.out.println("Done " + i + " inserts");
}
}
}
}
}
class DBHelper extends SQLiteOpenHelper {
DBHelper(Context context) {
super(context.getApplicationContext(), "bombastic", null, 1);
}
#Override
public void onCreate(SQLiteDatabase db) {
db.execSQL("CREATE TABLE table1 (_id INTEGER PRIMARY KEY AUTOINCREMENT, text1 TEXT, text2 TEXT, text3 TEXT, int1 INTEGER)");
}
#Override
public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {
}
}
On an ICS device (you can run it on Gingerbread if you start a thread or threadpool instead of abusing AsyncTask.SERIAL_EXECUTOR) the non-transaction version takes almost 4 minutes to complete (229484ms) while the version running in the transaction only takes about 3 seconds (2975ms).
So put it shortly, do a lot of updates - do it in a transaction.
To optimize your HTTP you should ensure that you are keeping the HTTP connection alive (keep-alive) and downloading larger chunks. Much larger than the ones you are doing now - if possible switch to a JSON parser that supports reading from a stream instead of loading the entire thing into a String before parsing it.
There are two time consuming activity involved in your case.
a. Downloading data in packets (assuming it to be HTTP). For a single packet it should take you about 1-3 sec depending on the network latency.
For 200 = 2X100 = 200 seconds ~ 3 mins
You can save lots of seconds, if you download entire data in say not more than 3-5 round-trip calls.
b. Database insert
You need to do file operation specifically write file operation which takes time. Honestly you cannot much optimization here
Check my other answer here