I have a database table with 2.5 million rows. I query this table by either one column (item_no) or two columns (item_no & subitem_no). In order to optimize my query I created two indexes. One for item_no and one for the pair of item_no and subitem_no.
Example:
CREATE INDEX Idx2 ON master(item_no);
CREATE INDEX Idx3 ON master(item_no, subitem_no);
Now when I run this query in SQLite browser:
SELECT component, hours, unit, price
FROM master
WHERE barcode = "234567"
AND item_no = "1234"
or this one:
SELECT component, hours, unit, price
FROM master
WHERE barcode = "234567"
AND item_no = "1234"
AND subitem_no = "34"
It executes extremely fast. Around 76ms - 186ms. This is what I want, the original select statement without the indexes took in between 4000 - 6000ms. So its the huge improvement I was looking for. So now I load the database onto my android device (Samsung Galaxy S6) and give it the same indexes. No improvement on query speed... at all, the select statement still takes 4000 - 6000ms to run.
Here's how I'm doing it. Please let me know if you see any errors or you can explain why I'm not seeing the expected performance increase.
db.execSQL("CREATE INDEX Idx2 ON master(item_no, subitem_no);");
db.execSQL("CREATE INDEX Idx3 ON master(item_no);");
public ArrayList<Data> getData(String barcode, String itemNo) {
ArrayList<Data> dataList = new ArrayList<>();
try {
Database db = Database.getInstance();
db.open();
String where = BARCODE_COLUMN + " = ? AND " + ITEM_NO_COLUMN + " = ?";
String[] columns = {COMPONENT_COLUMN, HOURS_COLUMN, UNIT_COLUMN, PRICE_COLUMN};
String[] args = {barcode, itemNo};
String sort = COMPONENT_COLUMN + " ASC";
Cursor cursor = db.getDb().query(true, MASTER_TABLE, columns, where, args, null, null, sort, null);
cursor.moveToFirst();
for (int i = 0; i < cursor.getCount(); i++) {
Data data = new Data();
data.setComponent(cursor.getString(cursor.getColumnIndex(COMPONENT_COLUMN)));
data.setHours(cursor.getString(cursor.getColumnIndex(HOURS_COLUMN)));
data.setUnit(cursor.getString(cursor.getColumnIndex(UNIT_COLUMN)));
data.setPrice(cursor.getString(cursor.getColumnIndex(PRICE_COLUMN)));
dataList.add(data);
if (!cursor.isLast()) {
cursor.moveToNext();
}
}
cursor.close();
db.close();
} catch (Exception e) {
e.printStackTrace();
}
return dataList;
}
It must of been a data cache issue. I restarted the device, uninstalled my app, then updated my index's per #Rotwang 's comments. Now when I run my app I get my desired 3ms query times.
Here were my fixes:
db.execSQL("CREATE INDEX Idx2 ON master(barcode, item_no, subitem_no);");
db.execSQL("CREATE INDEX Idx3 ON master(barcode, item_no);");
Related
I don't know what's wrong with my code I follow the rule but I get wrong result. I want to search db and find all rows data but I only get last row from sqlite. my code to search database is bellow:
public ArrayList<ArrayList<ContractSaveDataFromDB>> ActiveContractData(String phone, String numberId)
{
ArrayList<ContractSaveDataFromDB> UserData = new ArrayList<ContractSaveDataFromDB>();
ArrayList<ArrayList<ContractSaveDataFromDB>> SendUserData =
new ArrayList<ArrayList<ContractSaveDataFromDB>>();
SQLiteDatabase db = this.getReadableDatabase();
String whereClause = "phone = ? AND numberId = ?";
String[] whereArgs = new String[]{
phone,
numberId
};
String orderBy = "activeContract";
Cursor res2=db.query("usersAccount",null,whereClause,whereArgs,null,null,orderBy);
res2.moveToFirst();
do{
UserData.clear();
int index;
ContractSaveDataFromDB contractSaveDataFromDB=new ContractSaveDataFromDB();
index = res2.getColumnIndex("buyAmount");
String buyAmount = res2.getString(index);
contractSaveDataFromDB.setBuyAmount(buyAmount);
UserData.add(contractSaveDataFromDB);
SendUserData.add(UserData);
} while(res2.moveToNext());
res2.close();
db.close();
return SendUserData;
I don't know what's wrong. I appreciate if you help me to solve my problem.
you already added where clause so maybe it is filtering your results try to remove it by change this
Cursor res2=db.query("usersAccount",null,whereClause,whereArgs,null,null,orderBy);
to this
Cursor res2=db.query("usersAccount",null,null,null,null,null,orderBy);
I believe that your issues is that you are trying to use an ArrayList of ArrayList of ContractSaveDataFromDB objects.
I believe that an ArrayList of ContractSaveDataFromDB objects would suffice.
It would also help you if you learnt to do a bit of basic debugging, as an issue could be that you are not extracting multiple rows.
The following is an alternative method that :-
uses the ArrayList of ContractSaveDataFromDB objects,
introduces some debugging by the way of writing some potentially useful information to the log
and is more sound, as it will not crash if no rows are extracted
i.e. if you use moveToFirst and don't check the result (false means the move could not be accomplished) then you would get an error because you are trying to read row -1 (before the first row) as no rows exists in the cursor.
:-
public ArrayList<ContractSaveDataFromDB> ActiveContractData(String phone, String numberId) {
ArrayList<ContractSaveDataFromDB> SendUserData = new ArrayList<ContractSaveDataFromDB>();
SQLiteDatabase db = this.getReadableDatabase();
String whereClause = "phone = ? AND numberId = ?";
String[] whereArgs = new String[]{
phone,
numberId
};
String orderBy = "activeContract";
Cursor res2 = db.query("usersAccount", null, whereClause, whereArgs, null, null, orderBy);
Log.d("RES2 COUNT", "Number of rows in Res2 Cursor is " + String.valueOf(res2.getCount()));
while (res2.moveToNext()) {
ContractSaveDataFromDB current_user_data = new ContractSaveDataFromDB();
current_user_data.setBuyAmount(res2.getString(res2.getColumnIndex("buyAmount")));
Log.d("NEWROW", "Adding data from row " + String.valueOf(res2.getPosition()));
SendUserData.add(current_user_data);
}
res2.close();
db.close();
Log.d("EXTRACTED", "The number of rows from which data was extracted was " + String.valueOf(SendUserData.size()));
return SendUserData;
}
If after running you check the log you should see :-
A line detailing how many rows were extracted from the table
A line for each row (if any were extracted) saying Adding data from row ? (where ? will be the row 0 being the first)
A line saying The number of rows from which data was extracted was ? (? will be the number of elements in the array to be returned)
My app is using an external SQLite database. The database is created using DB Browser for SQLite software. I am using the following method to query my table with the column ENGLISH (same as en_word). However, problem is the query is slow when my database become large.
public static final String ENGLISH = "en_word";
public static final String TABLE_NAME = "words";
String sql = "SELECT * FROM " + TABLE_NAME +
" WHERE " + ENGLISH + " LIKE ? ORDER BY LENGTH(" + ENGLISH + ") LIMIT 100";
SQLiteDatabase db = initializer.getReadableDatabase();
Cursor cursor = null;
try {
cursor = db.rawQuery(sql, new String[]{"%" + englishWord.trim() + "%"});
List<Bean> wordList = new ArrayList<Bean>();
while(cursor.moveToNext()) {
String english = cursor.getString(1);
String mal = cursor.getString(2);
wordList.add(new Bean(english, mal));
}
return wordList;
} catch (SQLiteException exception) {
exception.printStackTrace();
return null;
} finally {
if (cursor != null)
cursor.close();
}
I tried to create index using DB Browser for SQLite.
CREATE INDEX `kindx` ON `words` ( `en_word` )
However, do I need to modify my code so that my app will query the database using this index? If so, how to do that?
The problem is that SQLite, like most relational databases, can use an index when the parameter to a 'like' clause ends with a wildcard, it cannot use an index when the parameter begins with a wildcard.
So, for this type of query, the index will not be used, and you wind up with a full table scan. This is why it is slower with a large number of rows.
You are actually attempting to do what is known as "full text search", which is not really possible to do efficiently without database features to support it directly.
I have not tried it, but I see that SQLite does have full-text search capabilities, and that it is supported on Android. See Full text search example in Android for an example.
For my application, I need to query a sqlite database around 40-50 times. I am sure that the code I wrote is very inefficient. Unfortunately, I cannot find many examples online that involves querying the database many times.
String[] entryValArray = new String[indicesList.size()];
DBHelper dbHelper = new DBHelper(MainActivity.context);
SQLiteDatabase db = dbHelper.getReadableDatabase();
for (int i = 0; i < indicesList.size(); i++) {
int moddedIndex = Integer.parseInt(indicesList.get(i), 16) % DBHelper.numEntries;
String queryStr = "select * from " + DBHelper.TBL_NAME + " where " + DBHelper.IDStr +
" = " + Integer.toString(moddedIndex);
Cursor cursor = db.rawQuery(queryStr, null);
if (cursor.moveToFirst())
entryValArray[i] = cursor.getString(1);
cursor.close();
}
Basically, I am taking a list of strings, converting them to hex values, and then modding the value to get an index into a sqlite database. This is for a password generator application.
Is there a better way to do this, especially regarding creating a cursor and then closing it in every iteration.
First of all you have to change your query string as you need only one column value but you are using
Select *
instead of
Select yourColumn
. Secondly if your indices list size is not very large you can use
IN(values ) function of db instead of
" where " + DBHelper.IDStr +" = " + Integer.toString(moddedIndex);
this will return the result in only one query you don't have to run a whole loop.
I'm creating a simple financial app where the user can input an income or expense. I cannot find anywhere how I can change the "total" amount by adding or subtracting numbers inside the database. The easiest way I can explain it is:
user enters an income of $10 : So I would add that 10 into the database.
user enters an expense of -$5 : so i would also add that into the database
the end result should be $5 as the total, but how do I do this?
I'm completely stuck as I've never use SQLite before. Thanks
You can do that simply by firing 2 commands on SQL
a) Use Select to get the value from the SQLite Database
b) In Android programming add them or subtract them
c) Update the new Total into the database
public void updateExpense(decimal Expense,String Condition) {
double current = 0;
db = this.getReadableDatabase();
String selectQuery = "select id, total from " + TABLE_YourTable ;
Cursor cursor = db.rawQuery(selectQuery, null);
int RowID=0;
if (cursor.moveToFirst()) {
current= Double.parseDouble(cursor.getString(1));
RowID= Integer.parseInt(cursor.getString(0));
}
/// Now we use condition --> if condition is positive it mean add ... if condition is negative it means
////subtract
if(Condition.equals("positive"){
current += Expense;
}else {
current =current - Expense;
}
cursor.close();
db.close();
//Your Update to SQLite
db = this.getReadableDatabase();
ContentValues values = new ContentValues();
values.put(total , current );
db.update(TABLE_YourTable , values, KEY_ID + " = ?", new String[] { String.valueOf(RowID) });
db.close();
}
I have large number of strings, approximately 15,000 that I stored in a SQLite database using the following code:
void addKey(String key, String value, String table) {
SQLiteDatabase db = this.getWritableDatabase();
ContentValues values = new ContentValues();
values.put(KEY_KEY, key); // Contact Name
values.put(KEY_VALUE, value); // Contact Phone
// Inserting Row
db.insert(table, null, values);
db.close(); // Closing database connection
}
And then i search through that database using the following method in order to pick out any strings that match the key im looking for:
public String searchKeyString(String key, String table){
String rtn = "";
Log.d("searchKeyString",table);
// Select All Query
String selectQuery = "SELECT * FROM " + table;
SQLiteDatabase db = this.getWritableDatabase();
Cursor cursor = db.rawQuery(selectQuery, null);
// looping through all rows and adding to list
if (cursor.moveToFirst()) {
do {
Log.d("searchKeyString","searching");
if(cursor.getString(1).equals(key))
rtn = rtn + "," + cursor.getString(2);
} while (cursor.moveToNext());
}
cursor.close();
db.close();
Log.d("searchKeyString","finish search");
return rtn;
}
The goal is to do this in real time as the user is typing on the keep board so response time is key and the way it stands now it takes over a second to run through the search.
I considered reading all of the items into an array list initially and sorting through that which might be faster, but i thought an array list of that size might cause memory issues. What is the best way to search through these entries in my database?
A couple of things you can do...
Change the return to a StringBuilder until the end.
Only use a readable version of the database (that's probably not making much difference though)
Do not get a new instance of the database every time, keep it opened until you don't need it anymore
Query for only what you need with the "WHERE" argument in the SQL query.
See the code below with some changes:
// move this somewhere else in your Activity or such
SQLiteDatabase db = this.getReadableDatabase();
public String searchKeyString(String key, String table){
StringBuilder rtn = new StringBuilder();
Log.d("searchKeyString",table);
// Select All Query
String selectQuery = "SELECT * FROM " + table + " WHERE KEY_KEY=?";
Cursor cursor = db.rawQuery(selectQuery, new String[] {key});
// you can change it to
// db.rawQuery("SELECT * FROM "+table+" WHERE KEY_KEY LIKE ?", new String[] {key+"%"});
// if you want to get everything starting with that key value
// looping through all rows and adding to list
if (cursor.moveToFirst()) {
do {
Log.d("searchKeyString","searching");
rtn.append(",").append(cursor.getString(2));
} while (cursor.moveToNext());
}
cursor.close();
Log.d("searchKeyString","finish search");
return rtn.toString();
}
Note even if you want this to happen in "real-time" for the user, you will still need to move this to a separate Thread or ASyncTask or you are going to run into problems....
You should consider using SELECT * FROM your-table LIMIT 50, for example. And you can put two buttons "Back", "Next" on your view. If every page has max 50 items, the user is at page 1, and he taps "Next", then you can use this query:
SELECT * FROM your-table LIMIT 50 OFFSET 50
If your table contains most of text-data, and you want to integrate search deeply into your app, consider using virtual table with FTS.
Let sqlite do the hard lifting.
First off, add an index to the field you're searching for, if you don't have one already. Secondly, don't do a SELECT all with manual table scan, but rather use a query in the form
SELECT column_value
FROM my_table
WHERE column_key LIKE "ABC%"
This returns the least amount of data, and the sql engine uses the index.
i dunno about better but maybe it'd be faster to make queries for the selected strings one by one.
public String searchKeyString(String key, String table){
String rtn = "";
Log.d("searchKeyString",table);
// Select All Query
String selectQuery = "SELECT * FROM " + table + "WHERE column_1 = " + key;
SQLiteDatabase db = this.getWritableDatabase();
Cursor cursor = db.rawQuery(selectQuery, null);
// looping through all rows and adding to list
if (cursor.moveToFirst()) {
rtn = rtn + "," + cursor.getString(2);
}
cursor.close();
db.close();
Log.d("searchKeyString","finish search");
return rtn;
}
EDIT:
Well i dunno how those custom keyboard apps do it, but those AutoCompleteTextViews are hooked up to adapters. you could just as easily make a cursorAdapter and hook your auto-complete view to it.
http://www.outofwhatbox.com/blog/2010/11/android-autocompletetextview-sqlite-and-dependent-fields/
http://www.opgenorth.net/blog/2011/09/06/using-autocompletetextview-and-simplecursoradapter-2/