I try to make a dictionary using Quick Search Box in Android. As shown in the SearchableDictionary tutorial, it loads all (999 definitions)data and uses them as matches to the input text to get the search suggestion. in my case, I have 26963 rows of data that need to be suggest while user input a word on QSB. therefore, I want to grab the char data one by one from the QSB, so that it will be efficiently load necessary suggestion. how can i do this?
here's the code i use...
bringit(200);
if (Intent.ACTION_VIEW.equals(intent.getAction())) {
// from click on search results
//Dictionary.getInstance().ensureLoaded(getResources());
String word = intent.getDataString();
//if(word.length() > 3){bringit(10);}
Dictionary.Word theWord = Dictionary.getMatches(word).get(0);
launchWord(theWord);
finish();
} else if (Intent.ACTION_SEARCH.equals(intent.getAction())) {
String query = intent.getStringExtra(SearchManager.QUERY);
//SearchManager.
//String bb =
mTextView.setText(getString(R.string.search_results, query));
WordAdapter wordAdapter = new WordAdapter(Dictionary.getMatches(query));
//letsCount(query);
mList.setAdapter(wordAdapter);
mList.setOnItemClickListener(wordAdapter);
}
Log.d("dict", intent.toString());
if (intent.getExtras() != null) {
Log.d("dict", intent.getExtras().keySet().toString());
}
}
private void letsCount(String query) {
// TODO Auto-generated method stub
for(int i=0; i<query.length(); i++){
definite[i] = query.charAt(i);
}
}
public void bringit(int sum) {
// TODO Auto-generated method stub
String[] ss = new String[10];
Log.d("dict", "loading words");
for(int i=1; i<=sum; i++){
KamusDbAdapter a = new KamusDbAdapter(getApplicationContext());
a.open();
Cursor x = a.quick(String.valueOf(i));startManagingCursor(x);
if(x.moveToFirst()){
ss[0] = x.getString(1);
ss[1] = x.getString(2);
}
Dictionary.addWord(ss[0].trim(), ss[1].trim());
Log.v("Debug",ss[0]+" "+ss[1]);
//onStop();
}
}
I use SQLite to collect data. and the other code is just same as the tutorial...
Retrieving a cursor is generally slow. You only want to retrieve one cursor which contains all the matching results.
You should perform the searching using SQL rather than fetching everything. A FULL_TEXT search is usually fastest for text matching, it is however slightly more complicated to implement than a simple LIKE, but I highly recommend you give it a try.
So you want to execute an SQL statement like:
SELECT * FROM my_table WHERE subject_column MATCH 'something'
See SQLite FTS Extension for more information. You can also use wild-cards to match part of a word.
In terms of search suggestions there is really no point returning more than around ~100 results since generally no users ever bother to scroll down that far, so you can further speed things up by adding a LIMIT 0, 100 to the end of your SQL statement.
If possible only start getting cursors once the user has entered more than X number of characters (usually 3 but in you're case this may not be appropriate). That way you're not performing searches that could potentially match thousands of items.
You seem to be leaving lots of cursors open until the application closes them even though you don't actually need them anymore: instead of calling startManagingCursor just make sure to call x.close() after your if (x.moveToFirst()) { ... } - this will free up memory faster.
On an unrelated note: please don't name your variables and methods things like ss or bringIt() as it makes code hard to read -- what is ss and what does bringIt() bring exactly?
You could have a look at the full text search extension in SQL Lite. Idea is to have a SQL query that fetches only the matching results, not all the results and then filter.
There is also a sample for the Android SDK: com/example/android/searchabledict/DictionaryDatabase
Related
I have been able to achieve a basic search capability in Android Room + FTS with the following query as an example in my Dao:
#Query("SELECT * FROM Conversation JOIN ConversationFts ON Conversation.id == ConversationFts.id WHERE ConversationFts.title LIKE :text GROUP BY Conversation.id")
public abstract DataSource.Factory<Integer, Conversation> search(String text);
Where the text is passed along between percentage characters, as such %lorem%.
This example works perfectly for the search of a single word and I want to expand this to be able to search for one or more words with the condition that they do not need to be in the order they are entered, but they must have a match for all the words. This means this has to be an AND and not OR situation.
I've found a few examples of SQL queries, but they require a specific query tailored for each case, for example: LIKE text AND LIKE another AND LIKE, etc... which is not a solution.
How can this be achieved? And to make it clear, I am not looking for a raw query solution, I want to stick to Room as much as possible, otherwise, I'd rather just use this as it is than to resort to raw queries.
EDIT: Adding an example per request
I search for do the results returned include all matches that contain do in the title, even if it is a partial match
I search for do and test the results returned include all matches that contain do and test, but in no specific order, even if they are partial matches.
However, if just one of them cannot be found in the text then it will not be returned in the results. For example, if do is found, but test is not then it will not be part of the results.
I think there is no way to do that except creating a raw query dynamically. You can write another method, something like this:
public abstract class ConversationDao {
public DataSource.Factory<Integer, Conversation> search(String text) {
StringBuilder builder = new StringBuilder();
String[] words = text.split("\\s+");
if (words.length > 0) {
builder.append(" WHERE ");
}
for (int i = 0; i < words.length; i++) {
builder.append("ConversationFts.title LIKE %").append(words[i]).append("%");
if (i < words.length - 1) {
builder.append(" AND ");
}
}
SupportSQLiteQuery query = new SimpleSQLiteQuery(
"SELECT * FROM Conversation JOIN ConversationFts ON Conversation.id == ConversationFts.id"
+ builder.toString()
+ " GROUP BY Conversation.id"
);
return search(query);
}
#RawQuery
public abstract DataSource.Factory<Integer, Conversation> search(SupportSQLiteQuery query);
}
Good day all, I have a list of Objects (Let's call them ContactObject for simplicity). This object contains 2 Strings, Name and Email.
This list of objects will number somewhere around 2000 in size. The goal here is to filter that list as the user types letters and display it on the screen (IE in a recyclerview) if they match. Ideally, It would filter where the objects with a not-null name would be above an object with a null name.
As of right now, the steps I am taking are:
1) Create 2 lists to start and get the String the user is typing to compare to
List<ContactObject> nameContactList = new ArrayList<>();
List<ContactObject> emailContactList = new ArrayList<>();
String compareTo; //Passed in as an argument
2) Loop though the master list of ContactObjects via an enhanced for loop
3) Get the name and email Strings
String name = contactObject.getName();
String email = contactObject.getEmail();
4) If the name matches, add it to the list. Intentionally skip this loop if the name is not null and it gets added to the list to prevent doubling.
if(name != null){
if(name.toLowerCase().contains(compareTo)){
nameContactList.add(contactObject);
continue;
}
}
if(email != null){
if(email.toLowerCase().contains(compareTo)){
emailContactList.add(contactObject);
}
}
5) Outside of the for loop now as the object lists are build, use a comparator to sort the ones with names (I do not care about sorting the ones with emails at the moment)
Collections.sort(nameContactList, new Comparator<ContactObject>() {
public int compare(ContactObject v1, ContactObject v2) {
String fName1, fName2;
try {
fName1 = v1.getName();
fName2 = v2.getName();
return fName1.compareTo(fName2);
} catch (Exception e) {
return -1;
}
}
});
6) Loop through the built lists (one sorted) and then add them to the master list that will be used to set into the adapter for the recyclerview:
for(ContactObject contactObject: nameContactList){
masterList.add(contactObject);
}
for(ContactObject contactObject: emailContactList){
masterList.add(contactObject);
}
7) And then we are all done.
Herein lies the problem, this code works just fine, but it is quite slow. When I am filtering through the list of 2000 in size, it can take 1-3 seconds each time the user types a letter.
My goal here is to emulate apps that allow you to search the contact list of the phone, but seem to always to it quicker than I am able to replicate.
Does anyone have any recommendations as to how I can speed this process up at all?
Is there some hidden Android secret I don't know of that only allows you to query a small section of the contacts in quicker succession?
can anyone explain why my inserts are taking so long in Ormlite? Doing 1,700 inserts in one sqlite transaction on the desktop takes less than a second. However, when using Ormlite for Android, it's taking about 70 seconds, and I can see each insert in the debugging messages.
When I try and wrap the inserts into one transaction it goes at exactly the same speed. I understand that there is overhead both for Android and for Ormlite, however, I wouldn't expect it to be that great. My code is below:
this.db = new DatabaseHelper(getApplicationContext());
dao = db.getAddressDao();
final BufferedReader reader = new BufferedReader(new InputStreamReader(getResources().openRawResource(R.raw.poi)));
try {
dao.callBatchTasks(new Callable<Void>() {
public Void call() throws Exception {
String line;
while ((line = reader.readLine()) != null) {
String[] columns = line.split(",");
Address address = new Address();
// setup Address
dao.create(address);
}
return null;
}
});
} catch (SQLException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
I've had the same problem, and found a reasonable workaround. This took insert time from 2 seconds to 150ms:
final OrmLiteSqliteOpenHelper myDbHelper = ...;
final SQLiteDatabase db = myDbHelper.getWritableDatabase();
db.beginTransaction();
try{
// do ormlite stuff as usual, no callBatchTasks() needed
db.setTransactionSuccessful();
}
finally {
db.endTransaction();
}
Update:
Just tested this on Xperia M2 Aqua (Android4.4/ARM) and callBatchTasks() is actually faster. 90ms vs 120ms. So I think more details are in order.
We have 3 tables/classes/DAOs: Parent, ChildWrapper, Child.
Relations: Parent to ChildWrapper - 1 to n, ChildWrapper to Child - n to 1.
Code goes like this:
void saveData(xml){
for (parents in xml){
parentDao.createOrUpdate(parent);
for (children in parentXml){
childDao.createOrUpdate(child);
childWrapperDao.createOrUpdate(generateWrapper(parent, child));
}
}
}
I've got original speed up on a specific Android4.2/MIPS set-top-box (STB).
callBatchTasks was the first option because that's what we use througout all the code and it works well.
parentDao.callBatchTasks(
// ...
saveData();
// ...
);
But inserts were slow, so we've tried to nest callBatchTasks for every used DAO, set autocommit off, startThreadConnection and probably something else - don't remember at the moment. To no avail.
From my own experience and other similar posts it seems the problem occurs when several tables/DAOs are involved and it has something to do with implemetation specifics of Android (or SQLite) for concrete devices.
Unfortunately, this may be "expected". I get similar performance when I do that number of inserts under my emulator as well. The batch-tasks and turning off auto-commit don't seem to help.
If you are looking to load a large amount of data into a database, you might consider replaying a database dump instead. See here:
Android OrmLite pre-populate database
My guess would be that you are slowing somewhat because you are doing two IO tasks at one time (at least in the code shown above). You are reading from a file and writing to a database (which is a file). Also, from what I understand transactions should be a reasonable size. 1600 seems like a very high number. I would start with 100 but play around with the size.
So essentially I suggest you "chunk" your reads and inserts.
Read 100 lines to a temp Array, then insert that 100. Then read the next 100, then insert, etc.
Dictioanry-based suggestions are displayed in a candidate view of IME when we start typing in the textview editor. For example if we type "th" in textview, then words like "this", "that", "the", "there" etc are displayed. I am trying to find out from Android repo source, as how these dictionary-based suggestions are fetched internally. Has anyone tried investigating this ?
I started creating an IME solution for android myself and I am using two things for my auto complete and suggestions. Not sure how this will help you or anyone. But this is what I have done...
I used the user_dict.db as a template database from the /data/data/com.android.providers.userdictionary/databases directory. I searched for the most common english words and imported them into the database. I did a simple database query for words like what the user typed in a new AsyncTask. Also when the user makes a "space" to complete the word, I used Jazzy, which is a spellchecker api for Java, and I took the input from the InputConnection and sent it to Jazzy to check. If the api came up with at least two results I would replace the user word with the first result from Jazzy.
Here is some of the code I used to create the suggestions list.
ArrayList<String> list = new ArrayList<String>();
.....AsyncTask.......
protected String doInBackground(String... str) {
list.clear();
list.add(totalString); // this is the string
//captured from InputConnection
Cursor c = db.getNameTitle(totalString); //this is my method in my Database
//adapter that queries the database and returns a limit of 30 results
if(c.moveToFirst()){
for(int i = 0; i < c.getCount(); i++){
list.add(c.getString(c.getColumnIndex(DBAdapter2.KEY_WORD)));
if(c.getCount() != i){
c.moveToNext();
}
}
}
c.close();
}
protected void onPostExecute(String result) {
mCandidateView.clear();
mCandidateView.setCandidatesViewShown(false);
Log.i("TAG", String.valueOf(list.size()));
if(list.size() > 0 && list.size() < 32){
mCandidateView.setSuggestions(list, true, true);//CandidateView similar to the SDK's example of SoftKeyboard or LatinIME
}
}
Hope this helps anyone. There may be another way to do this but this words great and it's fast. You may need to figure out the best query that suits your needs.
I currently have a statement which reads
if(Arrays.asList(results).contains("Word"));
and I want to add at least several more terms to the .contains parameter however I am under the impression that it is bad programming practice to have a large number of terms on one line..
My question is, is there a more suitable way to store all the values I want to have in the .contains parameters?
Thanks
You can use intersection of two lists:
String[] terms = {"Word", "Foo", "Bar"};
List<String> resultList = Arrays.asList(results);
resultList.retainAll(Arrays.asList(terms))
if(resultList.size() > 0)
{
/// Do something
}
To improve performance though, it's better to use the intersection of two HashSets:
String[] terms = {"Word", "Foo", "Bar"};
Set<String> termSet = new HashSet<String>(Arrays.asList(terms));
Set<String> resultsSet = new HashSet<String>(Arrays.asList(results));
resultsSet.retainAll(termSet);
if(resultsSet.size() > 0)
{
/// Do something
}
As a side note, the above code checks whether ANY of the terms appear in results. To check that ALL the terms appear in results, you simply make sure the intersection is the same size as your term list:
resultsSet.retainAll(termSet);
if(resultSet.size() == termSet.size())
You can utilize Android's java.util.Collections
class to help you with this. In particular, disjoint will be useful:
Returns whether the specified collections have no elements in common.
Here's a code sample that should get you started.
In your Activity or wherever you are checking to see if your results contain a word that you are looking for:
String[] results = {"dog", "cat"};
String[] wordsWeAreLookingFor = {"foo", "dog"};
boolean foundWordInResults = this.checkIfArrayContainsAnyStringsInAnotherArray(results, wordsWeAreLookingFor);
Log.d("MyActivity", "foundWordInResults:" + foundWordInResults);
Also in your the same class, or perhaps a utility class:
private boolean checkIfArrayContainsAnyStringsInAnotherArray(String[] results, String[] wordsWeAreLookingFor) {
List<String> resultsList = Arrays.asList(results);
List<String> wordsWeAreLookingForList = Arrays.asList(wordsWeAreLookingFor);
return !Collections.disjoint(resultsList, wordsWeAreLookingForList);
}
Note that this particular code sample will have contain true in foundWordInResults since "dog" is in both results and wordsWeAreLookingFor.
Why don't you just store your results in a HashSet? With a HashSet, you can benefit from hashing of the keys, and it will make your assertion much faster.
Arrays.asList(results).contains("Word") creates a temporary List object each time just to do linear search, it is not efficient use of memory and it's slow.
There's HashSet.containsAll(Collection collection) method you can use to do what you want, but again, it's not efficient use of memory if you want to create a temporary List of the parameters just to do an assertion.
I suggest the following:
HashSet hashSet = ....
public assertSomething(String[] params) {
for(String s : params) {
if(hashSet.contains(s)) {
// do something
break;
}
}
}