I am working on an app and one of the features I am working on is to download some binary files. Some of them are really big (more than several mega-bytes). Downloads are completing fine as long as the file size is less than 2 GB.
I got stuck on a file that is 3.2GB in that I get progress updates (I am pooling the DownloadManager for progress updates), but when the download completes, the file is not present on the target file path. Interrogating the DownloadManager for that download id, I get STATUS_FAILED and reason ERROR_UNKNOWN - the favorite error details one will ever wish for!
What is weird is that this appears on most of the devices, but for some (like Samsung SG 4 Active OS 4.2.2 and LG Nexus 5 OS 4.4.2), it doesn't appear.
Doing some extra investigation, I found out that this seems to be a bug in Android DownloadManager implementation. It seems Android implementation stores the download count as an int, but when that count goes above Integer.MAX_VALUE the download ends as failed.
I am thinking to replace the DownloadManager usage with a foreground service, but I wouldn't give up yet ....
Did you guys face this and if so, how did you fix it?
Is there any work-around to use DownloadManager in pre-4.2.2 so I can download more than 2.1 GB per file?
To download such a large files, you need to download those in chunks. Either you can use any library that support HTTP range options to allow to pull down a single file in multiple pieces , supporting resume etc.
Or you can split your large file on your server then have a text file with MD5 hash of each file, when you first start to download then get the MD5 file once finish then check that hashes matches the downloaded pieces. If they do not then delete that piece and add it to queue of items to download.
Once all pieces downloaded and MD5 works, you can put the pieces back together as single file.
If you are thinking to download the file in the SD card then FAT32 is the default file system. There is a 4 GB per file limit with this file system.
From looking at the Android source code, it appears that this issue was resolved in JB-MR2.
It seems that the only way to work around this on older platform versions would be to modify the server such that it uses chunked transfer encoding[1] for these large resources. In that case, that Download Manager will ignore and not attempt to parse the Content-Length header.
[1] http://en.wikipedia.org/wiki/Chunked_transfer_encoding
There is one clear outcome of this:
You can not fix the DownloadManager. If it's a bug in it, it will be so. Therefore, in short, No, you can not workaround this issue using the DownloadManager. You could however workaround it using a server side approach that has been put into words in the other answers.
So, I think your simplest solution would be to force the minimum sdk level to JB-MR2 because #ksasq mentioned that this issue has been resolved.
If that is not plausible nor in your case possible, you can find the best file download library out there and create an interface similar to DownloadManager's for this library. Of course, this interface should be implemented to use the default DownloadManager for versions which do not have this bug and use the custom library for those which had this bug (and for files who cause the issue if possible).
Unfortunately, a search on google showed yingyixu's android-download-manager last updated in 2012.
Another unfortunate note about this topic by CommonsWare simply verifies that there is no DownloadManager in google's support libraries. Worse is that the guy gave up the idea of implementing his own port becuase it was way too complicated. You can only hope that yingyixu's library or some other library you hopefully find is good enough.
You can pass this issue by splitting file into smaller zip files. Next step is to join them on target, I've found ->this<- that might help you. If you will not compress file (split only option) you should have good performance. Other issue is that you will need twice as much storage space. You can download smaller files, about 100MB, write it to joined buffer and remove form file-system, that will preserve space wasting.
You could also take the fixed version of DownloadManager, change the package to your package structure and use this version instead of the system version. Eventually you need to import some classes from the original package android.app. Then register your implementation as a service.
Related
i just exported my game to use Play Asset delivery following this guide:
https://docs.unity3d.com/2021.1/Documentation/Manual/play-asset-delivery.html
in short
build app bundle enabled
split application binary also checked.
The exported aab size is 311mb and when uploading to google play it says:
base ----install_time ---30.1mb
UnityDataAssetPack ----install time ---264mb
But then when i download the app in any device, the file size is 846mb, more than double the expected size. Also it tries to download all at once, i thought that the dataassetpack was downloaded after the base one.
i'm using Unity 2021.1.16f1
Do you know any cause for this to happen, and how to fix it?
thank you very much!
You are comparing a "download size" to a "size on disk":
The size you see in the Play Console corresponds to the download size, which is compressed.
On the device, you see the size on disk where the data is stored uncompressed so it can be directly rendered without having to keep two copies of the same data (one compressed and one uncompressed), so that explains the discrepancy.
It seems that you have configured the modules to be install-time so they will be installed as part of the initial install. If you want them to be installed while the user is starting the game, you should configure them to be fast-follow. If you want to manually download them using the Play Core API, you should configure them to be on-demand. This is all explained in the page you linked in the Managing asset packs at runtime section.
That page also links to the Google documentation on the topic, which has a dedicated section for Unity -- which would be too long to copy here, so have a read and come back if you have more specific questions.
If you're only using Unity marking the AAB output type in the build settings, and not using Google's API to do your stuff via script, then it's correct. Unity has a bug when assembling bundles and will simply (at least) double the size of your app. Things will be doubled in your AAB, you'll see the stuff you had in streaming assets in split apks and also inside the base.apk. Unity's way is not reliable, it's quite wrong and Google's plain way for Unity also is not perfect, it will not work for every single project. I was able to get the hang of it, but in the end I wrote our own tool to create the asset packs, created our runtime implementation to use the API and also our build procedures to be called by our custom build class.
My suggestion is, squeeze the docs here https://developer.android.com/guide/playcore/asset-delivery/integrate-unity?language=api and build your custom solution.
With the fantastic help of Tim on this thread Controlling Android app with bluetooth mouse/ presenter I added bluetooth remote control of my application. After finishing this, I noticed my apk file went from 175k all the way to 1711k - 10 times larger! I have implemented the bluetooth changes on a test application without seeing any change in file size so I am not sure what caused this. I did have some kind of Eclipse problem in the middle of this and had to do a rebuild to get the source to run again so maybe that had something to do with it.
Searching for file size change here and on Google gives me questions of people who want to make larger files to hold more stuff, not my question. I can't seem to find relevant answers.
The application has no graphics, pdf's, or other files of any size at all. It just reads some GPS stuff, does some math, and displays the results as text.
Is there some way to get the file size back down? Is there an Eclipse option? This is my first real (non test) application and it is finished and working but I am concerned with the file size increase.
You can check the contents of apk file easily. An apk file is simply a zip file with different extension. Just replace .apk with .zip and unzip it. Then diff the contents with your previous .apk file and find out what file makes the difference.
I encountered a similar problem with the APK size. One way to check out which resources are the culprits is to 'Open Editor Log' in Console.
Some helpful explanations can be found here:
Unity build project game 2d to apk with huge size because textures
https://docs.unity3d.com/432/Documentation/Manual/ReducingFilesize.html
Has anyone tried using market apk expansion downloader library (available as an add-on) for downloading large files on android <= 2.2? The source code is a bit difficult to understand and I was not able to figure out how it can be used for downloading any large files in general. Are there any alternatives for downloading large files on lower android devices(apart from writing my own DownloadService which I wouldn't prefer doing if there is something already available).
I have not tried to use it but the docs do state the the expansion downloader library is compatibile with API Level 4, taken from the docs:
Note: By default, the Downloader Library requires API level 4, but the
APK Expansion Zip Library requires API level 5.
I completely agree that the source code is difficult to follow, however it was designed in such a way to be fault tolerant to network issues.
I think you can take the sample and roll with that, however I found it was very difficult to test.
We ended up going with our own in the end (using an Arbitrary server) to host the downloads.
I do not believe there are any alternatives, however the downloader library does have a few gems. For instance the Expansion Zip Library can be used independently of the Downloader Library and is a great way to read content directly from the zip.
The only challenge would be writing your own downloader service and performing a CRC check.
Seems like API 4 or >
The expansion API is used for "permanent" file extensions to your application, they don't want you to move them or delete them or even unpack them etc. but use them in place, it's unclear what would happen if you did but I suspect a lot of odd side effects.
You also don't have direct control of when this file is downloaded, it has to be associated in the market and then it happens automatically unless for some reason it "fails" in which case when your app starts you are expect to initiate the process manually and then you need to use the Download Libary, else it's automagic as my 3 year old says.
From your post you may be looking for a way to get large files (more than one) and do what you want with them, if so this is not a good method to do that since every time you add a new file via the market it would tell your users you have an updated version and overwrite the older file (though you can use the patch extension to avoid that if you are updating the original but that only moves it back one step)
All of that is here what expansions files are for and how to use them
If you just want a method to get large files using your own server take a look at the support package and specifically the loader and loader manager. They also say they are making source code to the extension library downloader and zip file manager available but I haven't bothered to check that yet.
I've been trying to use SVN Kit to gain access to an SVN repository from an Android app.
Our project is supposed to check for changes to files and download them if they have changed, but still with the ability to rollback to previous versions if nessecary.
I set up a regular Java project, and had no problem using SVN Kit, I logged into the SVN server and retrived a list of the files stored there.
But when I tryed to do this from an Android project all hell broke loose. The VM run out of heap space, just trying to build it and Eclipse went down in sreaming flames mith comments like 'GC overhead limit exceeded'.
I get the impression that this is due to the diffrent type of virtural machine that Android uses, and the SVNKit jars are compiled for a diffrent type of VM.
This Guy claims to have got it working, with what looks like an older version of Android.
Now I have suggested that we use some kind of ftp server approach, uploading new versions of the files, perhaps with seperate files containing versioning information, but I have to explore this path before I write it off as a dead end, or at least suggest that to my boss!
anyu help or suggestions would be greatly appricated.
My answer may not be directly related to SVNKit but would actually try to address the original problem.
As I understand your use-case, you want to download files if they have changed and are trying to use SVN to do this with its update command. I would rather create a simple web-service which should return me the list of files with their checksums or md5Hash. Now I am assuming that only the relevant files are present on the server. Upon receiving the hashes, you can compare them with the hashes on the device and decide to update based on any difference. This approach will work for rollback too as you need to just replace the file on the server with old file and this would be treated as a new one.
As far as heap overflow and out of memory is concerned, they are mostly because of the limited heap space around 10MB.
I created a SVN client based on SVNKit it is available for free here : https://play.google.com/store/apps/details?id=com.valleytg.oasvnlite.android
I am not sure if that will work for your purpose or you have to build your own. If you are looking to build your own, svnkit will not work directly on the android platform. Some of the libraries used by SVNkit are not available on the Android platform.
I'm porting a rather large game engine written in C++ from Windows/Mac to Android. There is a lot of pre-existing code to read assets for games. In addition, there is quite a bit of code doing file system calls (stat'ing the files to make sure they exist, looking up all of the files and directories inside of a directory, etc.)
Right now, I'm focusing on just getting something up and running as quickly as possible, so I'd prefer not to have to rewrite a lot of this. What would be a good way of getting our game assets onto the device and accessing them with minimal changes to our existing standard C++ file system API usage?
I've got some basic support implemented already using the Asset Manager API, but that doesn't support the file system calls and I'm concerned that the 1 MB asset size limit is going to bite me at some point.
I've also looked at OBB, but the tools for creating an OBB file don't look like they are part of the current SDK/NDK. Otherwise, that looks like it would be perfect.
Is it a horrible idea to package up all of the files and just extract them on the SD Card the first time the app is run? Or is there some better way of dealing with this?
Update: I'm also not very concerned on being able to run on a broad range of devices, I am specifically looking at newish tablets, probably the 10.1" Samsung Galaxy tab.
We ran into a similar problem in developing our (data-file-heavy) app, and we ended up deciding to keep the APK tiny and simply download our data files on first run; they're going to have to be downloaded either way, but a small APK works much better on older devices without a lot of internal storage. Plus, you can potentially rig up a way for people to copy over the files directly from their computer if they have a limited data plan or a slow internet connection on their phone.
The "Downloader" sample app in apps-for-android (confusingly buried under "Samples") is almost a fully-implemented solution for this - you can pretty much just plug in the particulars of your data files and let it do the rest.
I wrote an app that relies on putting a good amount of native code into the Android filesystem. I did this by packaging the files into the APK as 'resources'. Instead of pushing them to the SD card, you can put then into the application's private namespace, I.E. /data/data/com.yourdomain.yourapp/nativeFolder.
For details on how to accomplish this, you can see my answer to this question.
It's fairly simple to package to just unpack them on the first run and never worry about them again. Also, since they're under the application's namespace, they should be deleted if/when someone were to decide to delete your app.
EDIT:
This method can be used to put anything into the app's private area; /data/data/com.yourdomain.yourapp/
However, as far as I know, your application has to be the one to create all the folders and sub-folders in this area. Luckily this is fairly easy to do. For example to have your app make a folder:
Process mkdir = Runtime.getRuntime().exec("mkdir " +localPath);
That works as it would in most linux shells. I walked through the assets folder I packaged into my APK, made the corresponding directories and copied all the native files to those directories.
What you might be more concerned with is the limited Android shell. There are many commands that you might want that aren't present. stat for example isn't available, so all of this may be moot if your native code can't make it's system calls.