I have Android app with Assets. They contains 20,000+ files, most of them are simple text or png files separated into different folders and subfolders.
Maximal size of 1 single file is 500kb, 90% of them are around 2kb.
Sample structure
Data
... subfolders with 20k+ files
StartData
.... FolderA
........ 5 files *.txt
.... FolderB
........ 5 files *.txt
.... a.xml
.... b.xml
The problem is load time of app. During app loading, I need to open around 20 files from StartData directory and read settings from them. They are in 2 xml files and based on info in xml I open additional data from two subfolders FolderA / FoldeB.
The speed is, however, very poor and app starts at least 3-4x times longer than without assets.
In former version I have zip in raw directory, that was deocmpressed upon first run to documents directory. First run was very slow, but other were fast. The problem with this solution was the install spedd, since deocmpresing took up to 5 minutes on older phones and it cannot die during that process. If so, app was damaged and data only partially decompressed, so I dropped this solution.
With my former approach, if unzipping finished correctly, cold app start was about 5s. Now, with assets, it takes up to 20s. Any solution?
I have disabled assets compression (I have only *.txt, *.png and *.xml files) in gradle
aaptOptions{
ignoreAssetsPattern ''
noCompress 'txt', 'png', 'xml'
cruncherEnabled = false
}
but the improvement is only small.
Also, I am using NDK (JNI) and AssetManager to load my assets in C++. Solution for assets opening in C++ is taken from this: http://www.50ply.com/blog/2013/01/19/loading-compressed-android-assets-with-file-pointer/
I am not surprised that discovering offsets to thousands of files in the assets folder is slow, and that the file system is better for such task. I would say that maybe the data structure should be rewritten. For example, you could put this information in a database.
I am sceptical about the funopen() trick. I mean, it works reliably, but I am concerned about the overhead. Much more efficient approaches to deal with non-compressed assets are through AAsset_getBuffer() or AAsset_openFileDescriptor(). Check an example of using file descriptor. Note that in the end, there will be one file descriptor for all thousands of your 'files'.
By the way, I don't think that Java performance with these direct access methods is worse than C. You could use zlib to read your assets from the APK file (which is in ZIP format itself), but I doubt you will do it faster than the AssetManager.
Related
TL;DR: Is there a way to copy a zip file containing thousands of files from Android assets to internal storage that is more efficient than using ZipInputStream?
My Android app has several assets that need to be copied into device storage upon initial launch. These are stored as zip files in assets and are copied via ZipInputStream as described here (in the unzip method). There are 10 zip files totalling 36MB, and the unzip/copy process takes about 3 seconds.
The problem: I need to add a new asset that is 39MB, but it adds about 30 seconds to the process. My hunch is that this is because the asset consists of 5500 files (averaging about 7KB each). Since I need the assets at launch, running this in a background service is not an option, and 30+ seconds is a really long time to show a splash screen.
This post suggests using ZipFile instead of ZipInputStream but it does not seem to work properly in Android as noted here and other S/O posts, and I am experiencing the same ZipException described there (note this is after copying the zip file to internal storage - Android assets only provides a stream, not a file, so the zip must be copied from assets before the ZipFile method can be used).
Is there a more efficient way to go about this?
Unfortunately - no as writing each file consist of 3 main operations: create and open file for writing, write data in file, close file for writing. The fastest way to copy such amount of files - place them in one file, for example binary or file of sqlite database. Or you can find a way to read directly from archive. Keep in mind that you won't be able to delete this file from assets (at least I never heard of solution for that) so it seems useless to me.
I need to copy all files from assets folder to android cache to load this data faster. The main reason is loading it once during the start of the app. In the whole app lifecycle, I could access needed files from the cache which would be probably faster. I was searching on the Internet but found nothing. How to do that?
One Android "cache directory" is obtained by calling getCacheDir() on some Context, such as an IntentService or JobIntentService. Copying a bunch of content out of assets into files on the filesystem will take some time, and so a Service with a background thread may be appropriate. Doing the actual copying is a matter of:
Getting an InputStream on the desired asset from AssetManager
Getting a FileOutputStream on where you want to write the content to, such as a file inside of getCacheDir()
Using standard Java I/O to copy the bytes
However, please understand that "cache" is not some magic pixie dust that you spread over an app to make it faster. For example, getCacheDir() is not faster than getFilesDir(), or getExternalCacheDir(), or getExtenalFilesDir(), because they all point to the same hardware (on most devices). Files on the filesystem may be faster to access than are assets, since assets are stored in the APK and require a bit of work to read them out of the APK. So, this may help a bit.
However, since you have not used method tracing, or Log statements, or anything to determine where your time is being spent, it is entirely possible that you will go through this work and get no net improvement. For example, my main book is published as an APK, among other formats. That book has 200+ chapters, all stored as HTML in assets. I do not find that loading that HTML is especially slow. It is possible that using files rather than assets will help you more than it might help me, due to the nature of what you are doing in those pages.
But it is also possible that your performance issues come from:
JavaScript doing too much work
Forgetting that you have a bunch of things that you are downloading from the network, because the URLs to them are buried somewhere (e.g., images referenced in CSS files), and it takes a while for those images to download
Something else that you are doing in your app, while simultaneously you are trying to load this Web content, and so you are overloading the CPU of the device
And so on
I am programming a game in C++ that I intend to have running on both PC and Android. I have a few images and text files that I am using in my game. Where can I put these files so that they can be opened with ifstream or fopen? I feel inclined to mention that I have already experimented with storing them in the assets folder and can load them using the NDK AssetManager. My main problem with the assets folder is that ifstream can't actually open files stored there due to the fact that they're compressed or something of that nature. For the most part, that would be okay, but I have some libraries that are essential and they only accept the path to the file being used. Additionally, it would be nice to have (almost) the same code being used to load files on PC and android.
T.L.D.R.: Where can I put the resources for my game so that they can be opened with ifstream? I read somewhere that I could use external storage or something of that sort, but I would prefer another solution.
Where can I put these files so that they can be opened with ifstream or fopen?
You would need to package them in your app (e.g., assets/), then copy them to files on internal storage (e.g., getFilesDir()). Nothing that gets packaged with your app winds up as ordinary files that can be used with fopen() and kin otherwise.
I read on how to reduce SKMaps.zip file size by deleting some of the files. Regarding to the same I need more information.
I deleted grayscalestyle, outdoorstyle and nightstyle folders and all the contents of sound_files (Maps/Advisor/en) directory.
There are two folders .Common and .Routing, .Common is of 28.5MB!! Are all the contents of this directory required? I tried deleting these two folders completely which resulted in map crash. If can be deleted which all files in these two folders can be deleted?
PreInstalledMaps directory is >10MB, which files can be deleted from this? I don't have preinstalled maps feature at all.
I read this, I need much more clarifications WRT other directories. Please let me know which all directories/files can be deleted as I need to reduce final APK size as much as possible.
No Routing/directions in this app.
The common directory contains files shared by all styles - the biggest file in there is the fonts file.
The fonts file provides all the required glyphs for all the character sets that the SDK supports (latin, arabic, cyrillic, chinese, etc.) - for the time being you cannot delete anything in there. In a future update there will be the option to use system fonts (instead of a fonts file).
The only workaround (at this time) would be to separate this file from the bundle, host it on your servers and download it before starting the SDK - this will be a one time operation - it will reduces the initial app size but it will increase the startup time (the first time).
The PreInstalledMaps folder - you can delete the meta folder.
When I complete the Android app I am developing I will want to distribute it so that about 300 image files are stored on the sdcard. Is there any way to do this within the usual automatic installation system? I don't want to hog the internal memory by including the files in res/raw.
I believe the app can now be built with an instruction to install on the sdcard so I suppose I could do that and include the files in res/raw but is there any limit on the number of files in res/raw?
If you didn't want to include the images you could utilize the APK Expansion Files mechanism:
http://developer.android.com/google/play/expansion-files.html
It will even save it to the sdcard.
Otherwise, you will either have to include the images or download them programmatically to the sdcard. As for the limits of the res/raw/assets folder I couldn't find any hard figures. The upper limit though would be the max size of the apk. You could always write a little script that would place 5k/10k/20k 1byte files and see for yourself on that upper bound. If the actual amount of files became an issue you could always zip them in the assets/raw and unzip them to external storage.
I have personally shipped an application for a client that had upwards of 500 images in the assets folder and it worked very well.