Using a humongous database in an android APP - android

I am trying to use a database for my application which needs a list of all the words in Arabic language, unfortunately this database is very large in size, more than 200 MB, I've seen that the only solution for such a problem is using a web service or having my database online and download it on first use which is not practical in my case since this is a game and the user can play it while he's disconnected, plus the download size will be large and it will use alot of space on his phone. I couldn't find a way to make the size of my DB reasonable.
My question is if there is a way to shrink the size of the database knowing that all the data stored in it is of the type text.
I've noticed that the keyboard in my phone has an auto-complete feature, where is it getting the list of valid words from? Can i use it for my application?

You'll want to store your words in a prefix tree (or trie). It is a space-efficient structure for this kind of data.
For more info, see: https://gamedev.stackexchange.com/questions/4142/best-way-to-store-a-word-list-in-java-android

your database might have so much extra information included, for example grammar, inflections, comments etc. If this is the case, then re-create your database with only the limited data/columns you need to be used inside phone.

Related

Android: Metadata about file

I am developing a notepad app which can store simple text files and checklists. Currently I maintain a separate file (say info.txt) that maintains information about whether a given file is a simple text file or a checklist and based on that I render my UI (for either listing all files or opening a file) to show that file in my app. However I am not very happy with this approach because is slow and does not appear to scale well.
Is there a better way to add "metadata" (e.g. if it is a simple text or cheklist, tags, etc) about a file in android?
Any help will be greatly appreciated
There are several ways of storing persistent data in Android.
The way you are currently doing it is through the device storage, and you are quite right it would probably not scale well in addition to being directly accessible to the user meaning they could edit or delete your metadata.
Using SharedPreferences would be one way of storing the metadata which has the advantage of being completely hidden from the user, as well as being relatively easy to set up. The main disadvantages I can see are that it may not scale well if a user has a large number of files, and it is much more difficult to retrieve files with certain criteria, a certain tag for instance, as you mention in the comments.
The best way to store data that will scale well, be persistent, and let you run queries on the data would be an on device SQLite database. SQLite will usually have more overhead in terms of setup time, but is far more robust and featured than any of the other options besides perhaps network based storage, which based on the information you have given is probably not something you are interested in. Based on your problem the SQLite database is probably the way to go and has the bonus of being expandable in case you ever decide to add more information, or even store the files in the SQLite database.

Android, storing big number of small images linked to database

I have been asked to create a tiny android app.
In everyday work i code for .NET and I have no experience connected with Android, but as it is a really small app I guess it's going to be a good experience rather than something hard.
The core of the app would be a small database (probably XML, unless somebody suggest better solution) that would contain categories, names of the institutions assigned with each category and logo (not very high resolution I guess a single file would be <100kB) of the institution.
The database also would not be very big - I expect not more than 1000 records in total. The DB has to be totally offline and local, it cannot require Internet access when operating.
The model I assume would be to ship new version of the application when the database changes (which is not going to be very frequent).
What is the best way to deal with these requirements?
My first idea was to create an XML file that would contain the records and link to the image. The XML and all the images linked to it would be stored in single file (preferably zip) that would be stored in app resources. This is very good as it is going to be very easy to update the database.
The second idea that somebody suggested me would be to use SQLite and store images in BLOB. In general I have read that it isn't a good idea to store images in database directly, and I am afraid if it's going to be possible to meet all requirements mentioned above.
Mostly I have no idea how to update the database easily and attach it to new version of application.
Any suggestions?
I would be grateful for any response.
I wouldn't go about using XML to save your data and by no means zip anything.
I think your way of thinking is ok, but you're making things really complicated for yourself.
Seeing as you're used to .NET I suppose you're also pretty confident with SQL, so I'd suggest you have a look at how to use the built-in SQLite database in Android.
If you would go the XML route you'd have to serialize and de-serialize the XML file over and over again and then parse the XML. Ok you don't have a lot of data, but searching inside an XML file with at least 1000 nodes would be slow in comparison to the performance of a database.
Also upgrading an existing SQLite database is not that hard - Android has methods for that (onUpgrade coming from the SQLiteOpenHelper).
As to saving images I'm assuming that you won't fetch new pictures from the Internet, so it would be best just to store them in the drawable folder of your app (be mindful of different screensizes) and then reading them into an ImageView when needed. To figure out what image should go for what institution I would store either the image name of each image in the SQLite database or store the resource id for each image in the database - for instance R.drawable.myawesomepictureformyinstitution.
I know my answer is somewhat "superficial", but your question is also somewhat "broad" and hard to answer without me actually writing most of the code, and that's not my intention ;-)
Hope this helps - let me know if anything is unclear.

Store large amounts of texts for Android app

I'm developing an Android app in which users will be able to write/save/modify potentially large pieces of text. I believe the amount of words will range from 10-1000. In the worst-case scenario, users will write a new piece of text everyday.
What is the best way to store these kinds of text data, holding in account the ability to easily modify saved pieces of text?
Store the data either as a file or in an sqlite database, if possible segment those pieces as separate records/files. For the loading part - there won't be any trouble of dealing with 1000 word within RAM, for example, if you load it inside a TextView. The limit to the size of text you allocate to your TextView is basically the amount of memory that you have.
I suggest testing your text editing view with ridiculously long texts at the end and if you see any issue (sluggish, runs out of memory etc), than you would have to take care of segmenting the document on your own. Hope this helps.
Best option Use sqlite database if you want to store data by day or by time. So, you can easily manage all your data.(if you looking for storage capacity then you can also manage your database in SDCARD(take backup in SDCARD, or create DB in SDCARD...etc.))
2nd Option store Data directly in SDCARD external storage(readable to user(He/She can Delete your data)).
you can use Encryption & Decryption in above both way.

Optimizing fast access to a readonly sqlite database?

I have a huge database and I want my application to work with it as soon as possible. I'm using android so resources are more restricted. I know that its not a good idea to storage huge data in the sqlite database, but I need this.
Each database contain only ONE table and I use it READ only.
What advice can you give me to optimize databases as much as possible. I've already read this post, and except the PRAGMA commands what else can I use?
Maybe there are some special types of the tables which are restricted for read only queries, but principally faster then ordinary table types?
As long as your database fits on the device, there is no problem with that; you'll just have less space for other apps.
There is no special table type. However, if you have queries that use only a subset of a table's columns, and if you have enough space left, consider adding one or more covering indexes.
Being read-only allows the database to be optimized on the desktop, before you deploy it:
set page size, etc.;
create useful indexes;
ANALYZE
VACUUM
In your app, you might experiment with increasing the page cache size, but if your working set is larger than free memory, that won't help anyway. In any case, random reads from flash are fast, so that would not be much of a problem.
Huge is relative. But ultimately a device is constrained on storage and memory. So assuming that huge is beyond the typical constraints of a device, you have a few options.
The first option is to store your huge dataset in the cloud and the connected device can offer views into that data by offering cloud services with something like RESTful APIs from the coud to proffer the data to the device. If the device and app rely on always being connected, you don't need as much local storage unless you want to cache data.
Another approach is an occasionally connected device (sometimes offline) where you pull down a slice of the most relevant data to work on to the device. In that model, yo can work offline and push/pull back to the cloud. In this model, sqlite is the storage mechanism to hold that slice of relevant data.
EDIT based on comments:
Concerning optimizing what you have on the device, see the optimization FAQ here:
http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html
(in rough order of effectiveness)
Use an in-memory database
Use BEGIN TRANSACTION and END TRANSACTION
Use indexes Use PRAGMA cache_size
Use PRAGMA synchronous=OFF
Compact the database
Replace the memory allocation library
Use PRAGMA count_changes=OFF
Maybe I'm stating the obvious but you should probably just open it with the SQLITE_OPEN_READONLY flag to sqlite3_open: I think that SQLite will take advantage of this fact and optimize the behaviour of the engine.
Note that all normal SQL(ite) optimization tips still apply (e.g. VACUUMing to finalize the database, setting the correct page size at database creation, proper indexes and so on...)
In addition, if you have multiple threads accessing the database in your application, you may want to try out also the SQLITE_OPEN_NOMUTEX and SQLITE_OPEN_SHAREDCACHE flags (they require sqlite3_open_v2, though)
Also you need journalling switch off, because data not change http://www.sqlite.org/pragma.html#pragma_journal_mode
PRAGMA journal_mode=OFF

Ways to save a lot of text and images in Android

Short version at the bottom
I'm working on an android app for a computer game, Heroes Of Newerth. A part of the apps functionality is to list all the heroes in the game. Each hero has:
a short description
a few stats(faction and primary attribute)
an icon
4 spells, which also has:
a short description
a few stats (mana cost, difference in ranks, etc.)
an icon.
There are approximately 110 heroes, which means I have about 500 sets of descriptions and stats.
I made a working version of the app. I downloaded all the images and put them in the drawable folder (note, this was 500 images), and created a Hero Enum which stored name, faction and primary attribute. Obviously, this was a bad idea, as it was horrible looking, and hard to extend to storing the rest of the data.
I have though about using a database, but as I don't have any experience with databases, I'm not really sure as how to do this, especially in Android. I looked it up, and it seems I need to initialize the database on the phone, which means I have to get that data from somewhere - which, again, means I'm back to square one.
I have never worked with this much data in a programming project, and have no idea for how to save it all. As if this is not enough, the game developer, S2 Games, releases new heroes with only weeks in between. As I wouldn't want to update one of my apps every other week, I want the app to be able to update itself with the new data. The best way I see this in my head is you download the app, either with a database of the current heroes, or without any, and the app checks each friday(patches are released on fridays) if the app is up to date. If not, update the database(with text and icons).
Short version
I want to save a few thousand strings, some formated in a special way(unless I can to this afterwards), and about 500 icons. How should I approach this?
Note: I know this was a really bad question, with a horrible structure, but I've been stuck here for weeks, and I couldn't get myself to ask someone, I really need help here!
well it's very recommended that you use sql and databases . you could go to w3schools for the basics .
if you don't want to use DB (or don't have time) , you can store all the data in xml files , and then parse them all . the images should never be part of the DB (or the xml files) , since they cause a bad performance while moving between items.put their names/paths instead .
if the images take a lot of space , consider using google expansion library .

Categories

Resources