I am working on a library app and have books stored as .sqlite files. Each book's sqlite database file is about 10MB in size. I first tried to put it in assets folder and then copying it to database/ folder but since the file is >1Mb this gives me an IOException. Then I tried to access it from raw folder but its still giving me IOException. So, what is the correct way to access such a file. Also, in future the app might need to download such files from server, so in that case where should I store such database files?
Thanks!!
Before Android 3.0 you are not allowed to open files larger than 1mb.
From the link: [P]rior to Android 2.3, any compressed asset file with an uncompressed size of over 1 MB cannot be read from the APK.
Here are some solutions:
Perhaps you want to look into shrinking the size of the databases, and that might be by making your own, or by removing some of the entries that you do not need/want.
Another solution would be to offload the databases to a server that you have access to and require the application to access the web to get the data that is necessary from this server. I can imagine a nice RESTful API to do this.
There is a workaround to this issue. The limitation of 1MB is only for SQLite files and not for other file types. Rename your SQLite file as something like "db.mp3" and then when your app starts, you can copy this file to your SD Card and use it as a normal SQLite file from SD card.
I have implemented this solution and it works perfectly fine on all Android versions.
Have the app download the database from a web server to the sdcard - this saves the waste of storing both a compressed version in the .apk and an uncompressed version outside. And it's not really any more insecure as an .apk is just a zip file anyone who really wants to can read.
Related
I ship a read only database with my app including it in Assets/Database directory. All data is pre-populated beforehand.
After integrating with SQLCipher and encrypting the database I noticed that the APK size ballooned from 25MB to 150MB. Reason being is that the SQLite db file no longer gets compressed.
Android Studio normally will compress resources but after DB file is encrypted it seems zipping the file has no effect. I tried testing this outside with regular Zip & 7Zip and it makes no difference, zipped file is the same size as the original.
Un-encrypted database is around 130MB and when zipped takes only 18MB as most of it is text and strings zip very well. Zipping encrypted db file makes no difference in size.
Populating via server isn't a viable option as it would take forever due to amount of records. Downloading a full encrypted DB file from the server is the next option but it would still take too slow with a 140MB file.
Need to find an approach to ship encrypted DB with the app while still keeping the app size to a reasonable 20-30MB.
Any ideas?
Thanks.
Compression requires finding patterns in the data.
Encryption removes all such patterns.
You need to do the compression before the encryption.
This is not possible with SQLCipher; you'd have to ship a compressed and encrypted .sql file, and then execute it on a new, encrypted database.
I'm looking to packing a large sqlite database with an android app, about 200-300MB. From what I've read, my options are putting it in the assets folder, and coping it out at runtime, resulting in duplicate data, or downloading the database from the web at runtime. I don't want to create a webserver for this app, so I was thinking of creating a seperate app that just installs the database to get around the data duplication problem.
All of those are pretty crummy, and I was wondering if there is a better solution?
First, I don't think you are allowed to put a 200-300MB database in your assets folder and distribute it as a single apk, since the hard limit for an apk file size is (as per google's documentation) 50 MB and I doubt that your db will compress that far.. Google introduced the "APK expansion files" for packaging large files with your apk. You can read all about those here: http://developer.android.com/google/play/expansion-files.html
I think (but I have no experience with expansion files myself) that this will not fix the issue you mentioned about having to deal with duplicate data because of the requirement to copy out your database file, but at least this will help you in the sense that you won't have to host your database file on your own webserver. I quickly scanned the documentation on the link I gave above and it clearly states that you should NOT throw away the expansion files when you are done with them, so no help on that part.
Of course, for the "setting up your own webserver part": I wouldn't even bother setting up my own webserver. There are plenty of parties out there that provide you some file hosting service. Probably you will have to pay for the bandwidth, but hey.. if people download your db file a lot, that also means that your app is doing well ;-)
I test of apk expansion files. I created a test app with a 100 MB sqlite database, uploaded the apk and database to the play store. I then downloaded the app onto the phone, the database was downloaded at the same time. The database was automatically renamed and placed in the publicly accessible directory:
/storage/sdcard0/Android/obb/com.example.app/main.1.com.example.app.obb
The app was able to open the database without any problems, but required external read and write permissions. The new and original database are bit for bit, the same. This appears to be the best solution for my use case.
A few issues, the new android developer console does not support uploading expansion files, so the old one needs to be used. Google only allows 2 expansion files, so that may be an issue for some use cases. Google say that you must not rename or delete the file, and that your app must be able to download the file from Google itself if it wasn't automatically installed.
I have an SQLite3 Database (created in a desktop sqlite application), with the android_metadata table for use in my android application.
What is the best way to create the android database using this database file?
I have tried including it in the assets folder, but got a size error when copying this to the application and wasn't sure if this was due to the asset files having a maximum size.
I have also read guides on storing the database on the sd card and accessing it.
Which function on androids sqlite helper is best to open a new database from an sqlite3 export?
I pushed the database file to data/data/com../databases/ and it created an "android.database.sqlite.SQLiteDatabaseCorruptException: disk malformed..
Not sure how to do this, appreciate any help!
You can attempt to use SQLiteAssetHelper for this, as it is designed to make it easy for developers to package a database with their app.
That being said, your database is much too big. There are plenty of 1.6 devices that will not have room for both your APK and the unpacked database.
The reason you get a size error is because android likes to compress files in the Assets folder but then can't decompress them if they are over 1 meg in size. Some file extensions do not get compressed like jpeg, gif, mp3, jet. I always name my db something like name.db.jet and then rename it to name.db when I copy it to the Database directory. The extension on the file really means nothing but naming it with an extension like .jet gets around the max size restrictions.
By the way, if you are going to include a DB that is 30mb in size, be sure to include it compressed. That will take it down to about 10mb in size.
I have been gathering information in this site and others about the best way to include a pre-populated sqlite database in Android. I just would like to confirm that I understood the reasons of why something that should be trivial… it is not.
So could someone please tell me if my following conclusions are correct or wrong?:
the main reason many people suggest copying a pre-populated database file from the assets folder to "/data/data/YOUR_PACKAGE/databases/" is because there is no way to access a database file in the assets folder or doing that would be overly complex (?). (Could someone clarify which of these two answers is the right one ?)
another important reason database files in the assets folder must be copied somewhere else is because files in that location cannot be updated. Then even if a database in the assets folder could be open, this would be useful only if such database does not have to be modified.
UPDATE: I launched a new thread focussing only on this issue: Opening a read only database directly in the assets folder.
files in the assets folder can be only 1 Mb size (unless they have certain file extensions such as mp3). Note that this restriction is not relevant if: your database is smaller that 1 Mb, or you do not mind dividing your database in 1 Mb chunks and putting them together at runtime, or you do not mind distributing a database file with a mp3 extension.
If the database file is copied from the assets folder to "/data/data/YOUR_PACKAGE/databases/", there is no way to delete the original database file at the assets folder to avoid having a duplicated file. This is also because files in the assets folder cannot be modified.
Making the puzzle a bit more complex: I found in the comments to the accepted answer of this question: Ship an application with a database
that copying the database from the assets folder to another location in fact does not work on some devices running 2.3+. Is that accurate ? If that is true, then the best alternative would be to download the database file from the web at first run ?
Thanks for any clarification.
You're essentially confusing apks with actual folders on your device.
Think of an apk as an install package - not unlike the msi of the Windows world. The whole goal of this install package is to securely authenticate and deliver code and resources to your device. In a naive implementation, you would then unpack said code to a read-only location, the resources somewhere read-writeable and be on your merry way.
To save space, Android is a bit smarter - the code and resources never leave the signed archive, so you always know it's the ones you put in and you don't waste space by storing the code twice. There's some real magic going on in the class loader that also allows it to unzip classes on the fly but that's besides the point.
So, essentially, everything in this compressed install package is read-only (by virtue of also being signed). It's your job to be the "installer" and move whatever resources you need to a read-writeable location. Of course, you can't touch the apk once it's in place since that would allow for malware and defeat the whole purpose of the signing.
Hope this clears the confusion.
I have a database of questions from an external SQlite manager. I would like to create a database on the first run of my app.
Is this possible instead of writing hundereds of lines of code?
Yes, this is possible. You can put the SQLite file in the assets folder of your Android project (or download it on demand) and copy it to the database folder of the installed app.
A tutorial explaining all the neccesary steps can be found here.
Note however that if you include the database file into your APK, it will consume twice the disk space than absolutely necessary (since it will exist twice, as asset and as usable database). Depending on the size of your database, it might be preferable to download it instead of embedding it into the APK.
Note furthermore, that prior to Android 2.3, any compressed asset file with an uncompressed size of over 1 MB cannot be read from the APK (and any asset will be compressed by default).