I am trying to download a operator specific db when the app is first time opening.
Since its operator specific, I cannot bundle this db as part of apk.
I have implemented a way to use DownloadManager to download the my_operator_specif_db.db and then reading it through SQLiteAssetHelper.
The size of the db is 23 mb. Since I am using "Accept-Encoding: gzip" in the headers, it will download zipped version of this db which is 8.3 mb. Still 8.3 mb transfer takes time to download on poor internet connection. Instead of this db, if I download a flat csv file with the same contents it comes around 13mb RAW and 3.8 mb zipped.
Effectively, I can make the download double times faster if I have a logic to load the flat csv file to an empty sqlite db.
Any suggestions on this approach?
How to load a csv file into db?
23 MB is not that much if what you're downloading is crucial for the app to function properly and you can always make it so that it is downloaded using wi-fi only(usually fast enough) That being said, you do want to keep the downloads to a minimum so what you can do is build the db from a raw text file(or .csv if that's what you have).
In my own opinion, if you're going to download something more than 1 MB, just get the gzipped db, it saves hassle. I'm saying this because downloading anything on a slow connection will be slow.
Related
I ship a read only database with my app including it in Assets/Database directory. All data is pre-populated beforehand.
After integrating with SQLCipher and encrypting the database I noticed that the APK size ballooned from 25MB to 150MB. Reason being is that the SQLite db file no longer gets compressed.
Android Studio normally will compress resources but after DB file is encrypted it seems zipping the file has no effect. I tried testing this outside with regular Zip & 7Zip and it makes no difference, zipped file is the same size as the original.
Un-encrypted database is around 130MB and when zipped takes only 18MB as most of it is text and strings zip very well. Zipping encrypted db file makes no difference in size.
Populating via server isn't a viable option as it would take forever due to amount of records. Downloading a full encrypted DB file from the server is the next option but it would still take too slow with a 140MB file.
Need to find an approach to ship encrypted DB with the app while still keeping the app size to a reasonable 20-30MB.
Any ideas?
Thanks.
Compression requires finding patterns in the data.
Encryption removes all such patterns.
You need to do the compression before the encryption.
This is not possible with SQLCipher; you'd have to ship a compressed and encrypted .sql file, and then execute it on a new, encrypted database.
I have an app that uses a sqlite db of size 30 mb (in zipped the size is 13 mb).
The updates are costly in size since the sqlite db is getting downloaded every time even though there is no change in data.
I have the following solutions in mind.
Approach 1:
Download the db on the launch of the app (if the db is not already present). So, the updates will not download. Problem with this approach is user needs to wait till the download is complete. It might make the user to uninstall the app since he has to wait.
Approach 2:
Put the sqlite db in the expansion file and it will downloaded in the play store before installation itself.
I am planning to take the approach 2.
Is it fine? In the approach 2,
Will the android allows to put an expansion file even though my app
size is less than 50 mb?
Will android show warning like "This app is going to do huge download.
Use it only on Wi-Fi kind of warning?
You can add the zipped DB file in your assests folder and first time when your app is launched unzip and copy the DB file from assests folder to the application memory.Now you can use it easily.
Note-One drawback of this approach is that you won't be able to delete the DB file from assests folder because it is readable only.
I have an app that includes a pre-populated sqlite database, encoded in UTF-8. DB file size is 170 MB. I am using SQLiteAssetHelper to manage the database
Considering most Android Apps are less than 50MB, it is quite large. My current app size according to the Settings is 238MB.
Is there any way to make the downloadable app size smaller when user downloads it from Store? I have read that you can zip the sqlite database, but I haven't tried that yet. Will that make the app's download size smaller? Are there any other better options?
I am concerned about the app's download size, not the storage space. I do need the app to work offline, so web service is not an option.
Is there any way to make the downloadable app size smaller when user downloads it from Store?
Have a smaller database.
Will that make the app's download size smaller?
Since the APK is already a ZIP file, no, as ZIPping an entry in a ZIP file will not reduce its size further.
Are there any other better options?
You could not use SQLiteAssetHelper, and download the database on the first run of your app. Or, you could not use SQLiteAssetHelper and look into using APK expansion files. Neither of those will change the number of bytes to be downloaded, but they will reduce the overall disk space your app takes up a bit. You can use tools like command-line unzip to see how much space your database file is taking up in the APK (my one test app's database is 89% compressed and so takes up 568 bytes instead of the uncompressed 5120 bytes). This space is non-recoverable, insofar as you have no way of deleting a database from assets/. Assuming a similar compression ratio, your 170MB database will take up an extra ~17MB on disk from the copy in assets/.
You could also not have the 170MB database on the device at all, and have your app work with some Web service to access a hosted copy of the database. This won't work offline, of course.
I ended up using text files for the major lookup tables in my database. I added them in the assets folder, along with the smaller sqlite db. I kept only the small tables that I needed to do inserts/adds in the sqlite db.
And I was also able to easily encrypt the data in the file (which is not easy to do in sqlite)
I had to create indexes to index the file data, and used binary search to find by key or value.
That shortened the apk size, and it's about 30 MB now. Before it was about 76MB. and the app size was > 200 MB.
I'm writing an android app that makes use of many images(dozens). These images, like most of the other data in the app, are updated from a remote database. For the data I am going to have a local database and sync it with the remote database every time the remote db is updated.
The remote database will store the images as URLs, and my app will download the images from these urls to display them in the app. So I could just have my local database sync with the remote one and I'll have the URL of all the image files I'm using and I can re download them from the remote server every time the app is run, but this is obviously slow and wastes a lot of data.
What I want to do is, everytime the database is updated and needs to be synced, the app will sync its local db and download the new images from their URLs as usual, but then it will save the image files somewhere on the device, so next time the app is run it can just grab the images from the device.
I can't seem to find an effective way of doing this, perhaps it's because it's a bad idea to do it this way in the first place? Sharepreferences probably won't have enough room, external storage isn't available all the time, and I hear it's a bad idea to just store a big chunk of binary data(such as an image) in the local database.
What are my options here?
Start with Android Storage Options.
Further:
external storage isn't available all the time
That's true in theory, but in practice, you'll have external storage 99% of the time, especially if you're developing for newer devices. Note that "external storage" and SD Card aren't the same thing -- the terminology here is confusing. For this, use getExternalFilesDir().
Regardless, if your image storage is a reasonable amount (<100MB is reasonable IMHO), then you can just use internal storage. As of Android 3.x, this won't cause a device to run out of space like early devices did. In practice, you'll be fine 99% of the time. Here you might want to use the cache dir, returned by getCacheDir().
In one of my apps I have a large number of files that are synced with a CDN. I use the sync process and database to retrieve download URLs and MD5 hashes of the files. If the MD5 hash has changed, then I download the file again in the background. I simply store the file using the hash as the file-name, so I can easily resolve duplicate files. Periodically, I also walk the cacheDir to see whether there are any files that are no longer referenced in the database.
I am working on a library app and have books stored as .sqlite files. Each book's sqlite database file is about 10MB in size. I first tried to put it in assets folder and then copying it to database/ folder but since the file is >1Mb this gives me an IOException. Then I tried to access it from raw folder but its still giving me IOException. So, what is the correct way to access such a file. Also, in future the app might need to download such files from server, so in that case where should I store such database files?
Thanks!!
Before Android 3.0 you are not allowed to open files larger than 1mb.
From the link: [P]rior to Android 2.3, any compressed asset file with an uncompressed size of over 1 MB cannot be read from the APK.
Here are some solutions:
Perhaps you want to look into shrinking the size of the databases, and that might be by making your own, or by removing some of the entries that you do not need/want.
Another solution would be to offload the databases to a server that you have access to and require the application to access the web to get the data that is necessary from this server. I can imagine a nice RESTful API to do this.
There is a workaround to this issue. The limitation of 1MB is only for SQLite files and not for other file types. Rename your SQLite file as something like "db.mp3" and then when your app starts, you can copy this file to your SD Card and use it as a normal SQLite file from SD card.
I have implemented this solution and it works perfectly fine on all Android versions.
Have the app download the database from a web server to the sdcard - this saves the waste of storing both a compressed version in the .apk and an uncompressed version outside. And it's not really any more insecure as an .apk is just a zip file anyone who really wants to can read.