I have an app that downloads many images that are about 50k to 100k in size. The full database can be anywhere from 50MB to 500MB.
We are currently using sqlite, but it has slow inserts. We did a test writing files and that was slow as well. It seems to be taking about 300 to 800ms per image to write - which comes out to about 250kb/sec (slower than the network speed). I did a benchmark with A1 benchmark and it says my write speed is 14MB/sec, so i'm not sure why my app is so slow?
Here are the primary requirements:
1) Fast writes
2) Fast reads/lookups via an custom integer key (this can be converted a file path for individual files)
3) Prevents user from easily accessing the images (if we save as images they can connect the external storage to their computer and browser, but we can write the files in reverse byte order)
hopefully we can still use the sqllite solution, as i like one file, but either way it seems like a storage issue?
As suggested in the comments, I think you are better off using the private memory storage for the actual images. This will have better speed then storing BLOBs in SQLite.
If you still need to keep a DB, for example for complex image searches or such, I suggest you just replace the BLOB field in your DB with a string with the actual location of the image file.
Another solution is to keep the images as app assets, but this assumes the images are always the same and can't change dynamically, and I doubt this is your use case.
Related
I have a rather large SQLite database (~20 mb) I need to access from my Android Xamarin-Forms app.
Everything online I've read says you can read the database by copying it to the filesystem first. For example, this question. However, won't that mean the large database is duplicated, wasting users' precious space (and nearly doubling the footprint of my app)?
There must be a way to read the SQLite database directly from the assets, or use some other method to bundle the database with my app that won't waste so much space. But how can I do this?
You don't want to use it from assets, even if you could, because assets is a compressed read only file, part of your installation. You can't write updates into it, which kills 90% of database use. And its inefficient for reading as its zipped up. So you really do need to copy it. If you're worried about disk space, consider downloading it from the web rather than keeping it in your apk.
im wondering if storing images as BLOB in the sqlite database would be a good idea?
Does anybody has performance - experience with storing images (blob).
My android app would be a small one, which will need to handle 20 to 100 images (100 kb up to 1MB per image). Worst case: I would say, that my database could reach a size of 100 MB. Does this has significant impact on the database performance?
Average case: I guess the average user of my app has 40 images with 200 kb per image, so the size of database would be around 8 MB.
Btw. of course the database stores also other "normal" data, so its not a image database only :)
Is storing the path to the image which is stored on the storage(internal or sd card) a better approach? I guess that retrieving the image file path from the database and open and load the image from file would be a little bit slower (but not really significant, since I need to load only two images at once).
A second question:
If I would use the second approach (store the path to image file in the database and load the image file):
Does a disk cache (DiskLruCache) is something useful in this scenario? Would it bring a significant performance boost? My understanding is that a Disk cache would store bitmaps (instead of encoded jpg or png) and therefore a disc cache would load a bitmap directly from storeage and my app would save the time to decode the image(jpg or png). Is that correct?
Btw. In the "database approach" i would store the image already decoded as bitmap. So it seems to me to be something similar as the disc cache, isnt it?
Edit:
I forgot to tell you, that i need to store the images persistent on the device. Im not talking about caching images that for example I have retrieved from a web service ...
My guess will be that the database will get significantly slower if you store that much of information in it, especially when you retrieve the images themselves.
On the other hand, as I get it, every user has the images associated with him downloaded after the application is installed. This is perfect place for using a library another SO user recommended to me in a question of mine I asked couple of days ago: Universal Image Downloader. Here is a link to the thread I speak about.
This library uses on disk caching, but abstracts away all the complexities for you (hopefully, I have not yet tried it, but it seems promising).
I have an android application that allows users to upload images to their account. Im storing the images as longblob files in a mysql db and pulling them from that but I have read on here and other places that its more efficient to store your images in a file system. I know it will work for my alpha to show but its already sucking up space in the db.
Ive seen plenty of people on here and other places mentioning file systems over using a db however....no one makes any references to specific file system software or set ups. Ideally I need a system that would allow for the fastest retrieval of images from it and it has to work with a query from php.
Any tips on the matter would be awesome :)
You could store the images on the file system, and use the database to keep a file-pointer, which is simply the path to the location of the image on your system. Then, use a query to fetch the location, and use that as you would for any image.
This thread on DaniWeb shows how uploads could be handled:
http://www.daniweb.com/web-development/php/threads/162230
Also, use relative paths in case you wish to move the location of the images in the future, as mentioned in the chosen answer here:
When storing Images in the File System, Use relative paths or absolute paths?
I am trying to build dictionary app(actually it is a modification of google SearchableDictionary sample), whose source of words and their definitions is very big, around 5MB. I tried many ways and using many formats and it still cant run properly on android. Sqlite database should be the best solution, I have built it and its size is 10MB(tried building it both before runtime and during runtime).
The main problem is the size of the definitions, but I have seen some other applications have managed to do this. It might be that there are some file size limits built into android system, but anyways if they werent it all takes so much to search and run queries in this sqlite database.
What am I doing wrong?
BTW: It HAS to be offline dictionary (download definitions max 1 time).
Problem in a nutshell:
word -definition
word2 -definition2...
Stored in a 10MB sqlite database (tried loading it from assets), not working.
With some hacks (loading it manually with eclipse DDMS tool) it is working but terribly slow.
Are you loading the database from the Assets folder? If yes, then that's your problem. There is a file size limit on what is in the assets folder (1mb I believe).
You have two options:
Split up your database into multiple 1mb files
Create a webservice. Have your application call the webservice which in turn downloads the database to your Android device. OR create a webservice API that your application uses to get data on as it needs it basis.
I have achieved by zipping the database and unzipping it into external caching direcotry (SD Card). you can look at the sample code here - http://www.android.manutech.mobi/2011/03/how-to-manage-sqlite-databases-with.html
Rename it to something like "databasename.mp3" or any media format. It won't be compressed by the package manager and therefore you can use it just like you need.
Have you tried compressing the file? If the data is just raw text I bet it'll compress to smaller than 1 MB.
1) Store a compressed version of only the words using the GZIP. (this will be very small only about 350k, and must be pre-sorted)
2) Load the list into a:
new ArrayList<String>();
3) Use a binarySearch to find the word
.binarySearch
4) When you need a definition, call an API like this with the word
http://services.aonaware.com//DictService/DictService.asmx/Define?word=aardvark
5) parse the resulting XML
10M file is just way too small to worry about. It didn't work, it might be because of the way of opening the database file. By default, Android read database file from ://data/data/PACKAGENAME/databases/DATABASENAME.
A simple solution could be: 1) compress the db file to res/raw/DATABASE.zip, 2) then unzip it to //data/data/PACKAGENAME/databases/.
You can get sqlite android demo code from here: http://www.cs.dartmouth.edu/~campbell/cs65/lecture15/lecture15.html
There is a web service that provides some data that my app makes use of. This data is fairly large and only changes VERY infrequently so I thought it would be nice if the app could cache it on the SD Card and only update it as needed.
Currently I'm grabbing the data (an XML file) and parsing it into an object tree using SAX. This process takes (at most) 2-3 seconds over my WIFI. However, serializing the resulting objects to the SDCard takes significantly longer (a minute or more) and deserializing it still takes longer than just download/parsing in the first place.
Does anyone have any recommendations for improving this or alternate ideas for persisting this data (other than just saving the XML file and reparsing every time)?
UPDATE: This is more than a trivial collection of records. The object-graph is actually ridiculously complex and storing it into a database would result in dozens of tables with only a single record in each one.
Android serialization is notoriously slow. I highly suggest switching to using XML or JSON (or whatever) and writing the file out that way. Since you've already got an XML parser, it may make the most sense just to cache the original XML file you downloaded and reparse it as necessary.
I have switched from Serializable to JSON file storage in an app before and the speed increase was incredible, at least one order of magnitude.
(I may be misunderstanding your question - I assume you are using Serializable for writing to the disc. If you are reproducing the XML, then I'm not sure why it is so much slower on the SD card. Also, I agree that the SQLite database makes the most sense typically, but as you've already stated it does not fit the needs of your application.)
Also unless your data is at least 100s of Kb, I would suggest just storing it in your private data storage instead of on the SD card. Keep in mind that you can't rely on the SD card being available.
I've just been writing an android application for the last week which basically does this. It fetches some (large) XML file online, and then displays part of the data in various views.
We do it by fetching and parsing the XML using SAX, and (while parsing) writing it all to a SQLite database. And then we are just querying the database each time we need to display some view of the dataset.
Works like a charm, and is fast enough for displaying a lot of data on a google map overlay, where we are querying the database on every single call to the draw method of our map overlay.
So I would definitely suggest going for a SQLite database, if the data in the XML document is easily represented in a database.
If the web service can give you just a specified number of results(something like: requestData between index 1 and 10 or give me first 25 results) try to use that (put a simple "Load more results" button or implement an auto-loading mechanism). If the web service not provide this feature then try to save your xml on sdcard and when you need the data try to parse just a specified number of results. Hope this help!
Why don't you use database? See Android Data Storage Guide