Sqlite Full Text Search Compression - android

I have implemented FTS in my app, but the size is too big. I would like to compress it somehow. I read this on the sqlite website:
-- Create an FTS4 table that stores data in compressed form. This
-- assumes that the scalar functions zip() and unzip() have been (or
-- will be) added to the database handle.
CREATE VIRTUAL TABLE papers USING fts4(author, document, compress=zip, uncompress=unzip);
But I am struggling to find an example of these scalar functions. Please if someone could provide an example of how I may implement such a function for compressing my text.
The text will be in english to start, but will eventually include arabic.
Update
Since it does not seem possible to compress the FTS tables. I would like to limit the actual tokens generated which would lead to smaller sizes.
For example the word "Book" will AFAIK be stored as Book Boo Bo B etc. However I only need it to be stored as complete words as that is all my users will search for.
How can I acheive that?

Even if you found a compression function example (which exit!), SQLite in Android doesn't support user defined functions.
Sample you found will not work in Android - at least in a standard Android device.

Related

Better way of using tables in android then innumerous if then statements (or switch/case)

I am developing a RPG character generator for a game using a purely random method (new Random().netInt(100)).
The system has 14 steps to create the character and some steps have between 1-13 dice rolls with table lookups. The problem is about half of the 'die rolls' have at least 5 rolls, each one being a lookup on a table, or another table.
Some steps are easy as there are very few options and coding in a simple switch/case of if/then loop is easy, if a little boring, but quite achievable.
In Excel I would use a vlookup to look up various tables, etc and get a simple result, but in Android, I'm wondering is there an easier way than manually coding several thousand if then statements for each "step".
Ive already simplified some table statements as there might be 5 options each using 1-20, 21-40, etc so simple maths will help, but other tables have been weighted due to RPG setting details, so changing them would give incorrect and inaccurate random results.
I thought of using SQLlite but numerous table lookups would result in very slow performance.
Currently the class if getting very large with the switch and if statements but its not slow, just cumbersome.
I've got the data in pdf, excel, etc and can cut n paste the table data easily enough..
Am I missing a functionality of android that can perform table lookups (i.e. Excel "VLookup" functionality?) or am I doomed to write thousand of if/then statements.
Found this
http://developer.android.com/training/search/search.html
Add the data into a flat file, load it into the schema, and load it into memory. Access should be faster, so might be a better option....
I will give it a go and see..

How do I optimize loading a lot of Strings in my android app?

I'm currently working on a project where I have a dictionary list of words of the english language. I use this list to figure out if a user has inputted a valid word. The whole list is rather lengthy, were talking about 100k of Strings, but I have partitioned the list into smaller bits. These files are stored as xml files and are accessed by the assetmanager. I use the parser as such to open the file:
document = documentBuilder.parse(xmlFile);
The document is a Document object using the library: org.w3c.dom.Document
So the problem is that this actually slows down the app A LOT, even with the smaller partitioned files (in fact initially I just had one file with all the 100k Strings and it crashed the app immediately). So should I use text files instead? Or instead of using the assetmanager should I store my data in resources? Or any other way to optimize speed for that matter? I'm open to any suggestions.
Why not store the word list in an sqlite table? You get indexed-based searches. If each entry is one word then a normal table is fine. If you are using multiple words per entry, then an fts table (full-text-search) solution would be best.
If speed is paramount (i.e. you don't care how much memory you use up -- which would be very unusual for an android app), then loading the words into a hashtable would probably give you the fastest per word lookup. It would take some time to load the list on startup, however.
So I was playing around with different approaches and found out that instead of using partitioned xml files, I used text files saved in the asset folder which greatly reduced execution time.
if you have a big traitement to deal with and you this my make your apps slow, i recommande you to use AsyncTask so that you can do what ever you want in backGround and meanwhile you can do another thing, take a look in this official documentation

how to insert images, links, carriage return into Searchable Dictionary for Android

I am developing a glossary using the sample code Searchable Dictionary. Thanks to searching here, I have figured out how to update the database, which is a .txt file, and then get it to load by changing the version number in Dictionary.java.
My question is, how to do the following:
I would like to be able to insert illustrative images into the definitions.
I would also like to insert links to other entries in the dictionary (e.g. 'inventory' should have a link to 'product flow' and other related terms).
I would also like to know how to insert a carriage return.
My original glossary in spreadsheet format has several fields: 'term' 'definition' 'example' 'related terms'. I want to be able to put in links and images inside these fields and have a couple of carriage returns in between each field to differentiate them.
The dictionary code seems to take in everything as a string, so even if I try to put 'image.jpg', or '\n' for a new line, it simply prints that as part of the string. Is there a way around this?
Searching stackoverflow gave a few links to using SQLite. I am honestly a newbie at all this; the last time I programmed anything significant was ten years ago. Rewriting the code to directly access a SQLite database would be nontrivial for me. So I would like to know if that is really the route I should be taking. If it is, then could you point me to the most simple tutorials for constructing a dictionary that way? I downloaded SQLite data browser, but haven't figured out how to use to construct a new database. I know it should not be so hard; I just don't know what I am doing. :(
If there is an easy way to just do it inline, still using the Searchable Dictionary sample code as a base, that would really make my day. Otherwise, any specific suggestions/directions would be really appreciated.
Thank you!!
Update:
For clarification, below is an example of one entry in my glossary, as desired. There are carriage returns between sections, and links and images are inline with text:
Heijunka, or Load Leveling - An approach to smooth production flow when a mix of products is to be produced, by identifying for a selected time period, the smallest batch size at which to produce each specific product in the mix, before switching over to make another product in the mix.
Example:
Keeping a steady work flow, even if much slower than the original max, reduces waste (<-this is a link to the entry 'waste' in the glossary):
[image of line of balance graph with load leveling, and without]
Related Terms: work structure, demand leveling (<-These are links to respective entries)
Not sure if you saw this already, but Android has some developer lessons for saving Key-Value sets for simple data, and saving to SQLlite for more complex structures.
It sounds like your app needs a database called "Invetory" with the following fields: "ProductImage", "ProductTitle", "ProductLink". And you want to store the image as a BLOB. There's a good SO post on how to take an image from a URL and convert it to a byte array for storage: how to store Image as blob in Sqlite & how to retrieve it?
For the carriage return, i'm assuming you're using "\n"? If that's not working have you tried unescaping your string for TextView:
String s = unescape(stringFromDatabase)
Or for SQLlite:
DatabaseUtils.sqlEscapeString()
Key-value data: http://developer.android.com/training/basics/data-storage/shared-preferences.html
SQLlite data: http://developer.android.com/training/basics/data-storage/databases.html
Additional SQLite resources:
http://www.youtube.com/watch?v=G8ZRXdztESU
http://www.vogella.com/articles/AndroidSQLite/article.html

Best method of finding random English words

I'm looking for the best way to find a random English word using an Android app. Is there a better way than saving a .txt file with all the words in your res/ folder, such as
http://www.stanford.edu/class/cs106l/assignments/dictionary.txt
then importing it into an ArrayList? The app I'm developing will search for words quite often, such as a random word starting with the letters 'A' or 'Be'. I'm worried about memory and runtime with this function.
Create a SQLite database containing all the words. You can then query that database using SQL to match your random word with various starting letters.
SQLite Details: http://developer.android.com/reference/android/database/sqlite/package-summary.html
Look for documentation on using LIKE in SQL for your text matching.
Example: http://www.techonthenet.com/sql/like.php
For randomness, find all words matching your criteria. Get the count of them (call it 'n'), then pick a random number 'x' between 1 and n, and take the xth record.
Use a prefix tree (or Trie) as described here.
It's extremly efficient, as all word-beginnings are only stored once.

How should I store data to make it easy and efficient to search in android

What I have is an app that displays some documents. In the string resources I have the documents divided into smaller pieces in anticipation of making them searchable. Think of them like newspapers with a number of articles where each article is a separate string resource. There will not be any storing of user input (unless I decide to store recent searches). In the search part of the android developer docs it mentions this but says it is not going to go into details of how to store and search data just how to use the search dialog and widget.
What kind of storage of my data should I be using. Is simple string resources good? should I look into a real databasing? which of these make it the most efficient and quickest to search? I'm new to android so any help would be appreciated.
answer:using android's built in sqlite database system and FTS3 tables.
I would definitely use a database for this.
Read all the documents and link each word to each document in a database.
A word search would then produce a list of documents containing this word quickly.
Make sure you reindex each time you add and remove a document.
By the way, you should see to improving your accept rate.
Also, this is problably not a Android question.

Categories

Resources