I'm going crazy, I created a file object, so it can be read with ObjectInputStream, and I placed the assets folder.
The method works with a file smaller than 1M, and give error with larger files.
I read that is a limit of Android platform, but I also know that can be "easily" avoided.
Those who have downloaded the game Reging Thunder, for example, can easily see that in their assets folder is a file 18.9M large.
This is my code for read 1 object from a ObjecInputStream
File f = File.createTempFile("mytempfile", "dat");
FileOutputStream fos = new FileOutputStream(f);
InputStream is = mc.getAssets().open(path,3);
ObjectInputStream ois=new ObjectInputStream(is);
byte[] data = (byte[]) ois.readObject();
fos.write(data);
fos.flush();
fos.close();
ois.close();
is.close();
now I have an uncompressed file and I can use it without worrying about the error "This file can not be opened as a file descriptor; it is probably compressed"
This function works well with files smaller than 1M, with bigger files return an
java.io.IOException on line "ObjectInputStream ois=new ObjectInputStream(is);"
why??
Faced the same issue. I've cut up my 4MB file into 1 MB chunks, and on the first run I join the chunks into a data folder on the phone. As an added bonus, the APK is properly compressed. The chunk files are called 1.db, 2.db, etc. The code goes like this:
File Path = Ctxt.getDir("Data", 0);
File DBFile = new File(Path, "database.db");
if(!DBFile.exists() || DatabaseNeedsUpgrade) //Need to copy...
CopyDatabase(Ctxt, DBFile);
static private void CopyDatabase(Context Ctxt, File DBFile) throws IOException
{
AssetManager assets = Ctxt.getAssets();
OutputStream outstream = new FileOutputStream(DBFile);
DBFile.createNewFile();
byte []b = new byte[1024];
int i, r;
String []assetfiles = assets.list("");
Arrays.sort(assetfiles);
for(i=1;i<10;i++) //I have definitely less than 10 files; you might have more
{
String partname = String.format("%d.db", i);
if(Arrays.binarySearch(assetfiles, partname) < 0) //No such file in assets - time to quit the loop
break;
InputStream instream = assets.open(partname);
while((r = instream.read(b)) != -1)
outstream.write(b, 0, r);
instream.close();
}
outstream.close();
}
The limitation is on compressed assets. If the asset is uncompressed, the system can memory-map the file data and use the Linux virtual memory paging system to pull in or discard 4K chunks as appropriate. (The "zipalign" tool ensures that uncompressed assets are word-aligned in the file, which means they'll also be aligned in memory when directly mapped.)
If the asset is compressed, the system has to uncompress the entire thing to memory. If you have a 20MB asset, that means 20MB of physical memory is tied up by your application.
Ideally the system would employ some sort of windowed compression, so that only parts need to be present, but that requires some fanciness in the asset API and a compression scheme that works well with random access. Right now APK == Zip with "deflate" compression, so that's not practical.
You can keep your assets uncompressed by giving them a suffix of a file type that doesn't get compressed (e.g. ".png" or ".mp3"). You may also be able to add them manually during the build process with "zip -0" instead of having them bundled up by aapt. This will likely increase the size of your APK.
Like Seva suggested you can split up your file in chunks. I used this to split up my 4MB file
public static void main(String[] args) throws Exception {
String base = "tracks";
String ext = ".dat";
int split = 1024 * 1024;
byte[] buf = new byte[1024];
int chunkNo = 1;
File inFile = new File(base + ext);
FileInputStream fis = new FileInputStream(inFile);
while (true) {
FileOutputStream fos = new FileOutputStream(new File(base + chunkNo + ext));
for (int i = 0; i < split / buf.length; i++) {
int read = fis.read(buf);
fos.write(buf, 0, read);
if (read < buf.length) {
fis.close();
fos.close();
return;
}
}
fos.close();
chunkNo++;
}
}
If you do not need to combine the files into a single file on the device again, just use this InputStream, which combines them into one on the fly.
import java.io.IOException;
import java.io.InputStream;
import android.content.res.AssetManager;
public class SplitFileInputStream extends InputStream {
private String baseName;
private String ext;
private AssetManager am;
private int numberOfChunks;
private int currentChunk = 1;
private InputStream currentIs = null;
public SplitFileInputStream(String baseName, String ext, int numberOfChunks, AssetManager am) throws IOException {
this.baseName = baseName;
this.am = am;
this.numberOfChunks = numberOfChunks;
this.ext = ext;
currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);
}
#Override
public int read() throws IOException {
int read = currentIs.read();
if (read == -1 && currentChunk < numberOfChunks) {
currentIs.close();
currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);
return read();
}
return read;
}
#Override
public int available() throws IOException {
return currentIs.available();
}
#Override
public void close() throws IOException {
currentIs.close();
}
#Override
public void mark(int readlimit) {
throw new UnsupportedOperationException();
}
#Override
public boolean markSupported() {
return false;
}
#Override
public int read(byte[] b, int offset, int length) throws IOException {
int read = currentIs.read(b, offset, length);
if (read < length && currentChunk < numberOfChunks) {
currentIs.close();
currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);
read += read(b, offset + read, length - read);
}
return read;
}
#Override
public int read(byte[] b) throws IOException {
return read(b, 0, b.length);
}
#Override
public synchronized void reset() throws IOException {
if (currentChunk == 1) {
currentIs.reset();
} else {
currentIs.close();
currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);
currentChunk = 1;
}
}
#Override
public long skip(long n) throws IOException {
long skipped = currentIs.skip(n);
if (skipped < n && currentChunk < numberOfChunks) {
currentIs.close();
currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);
skipped += skip(n - skipped);
}
return skipped;
}
}
Usage:
ObjectInputStream ois = new ObjectInputStream(new SplitFileInputStream("mytempfile", ".dat", 4, getAssets()));
Change the file extension to .mp3
I know this is an old question but I thought of a good solution.
Why not store the file already pre-zipped in the asset folder.
Then since it is already a zip file and therefore compressed it won't need to be compressed again. So if you wanted the file to be compressed to decrease the size of your apk but you don't want to deal with splitting up files I think this is easier.
When you need to read that file off the device just wrap the inputstream in a zipinputstream
http://developer.android.com/reference/java/util/zip/ZipInputStream.html
I found another solution, maybe you're interested in it.
In the root of your sources, where you have the build.xml file, you can overwrite the -package-resources target in the custom_rules.xml file, which is used for adding/modifying targets in ant without breaking anything in the standard android app build system.
Just create a file with this content:
<?xml version="1.0" encoding="UTF-8"?>
<project name="yourAppHere" default="help">
<target name="-package-resources" depends="-crunch">
<!-- only package resources if *not* a library project -->
<do-only-if-not-library elseText="Library project: do not package resources..." >
<aapt executable="${aapt}"
command="package"
versioncode="${version.code}"
versionname="${version.name}"
debug="${build.is.packaging.debug}"
manifest="${out.manifest.abs.file}"
assets="${asset.absolute.dir}"
androidjar="${project.target.android.jar}"
apkfolder="${out.absolute.dir}"
nocrunch="${build.packaging.nocrunch}"
resourcefilename="${resource.package.file.name}"
resourcefilter="${aapt.resource.filter}"
libraryResFolderPathRefid="project.library.res.folder.path"
libraryPackagesRefid="project.library.packages"
libraryRFileRefid="project.library.bin.r.file.path"
previousBuildType="${build.last.target}"
buildType="${build.target}"
ignoreAssets="${aapt.ignore.assets}">
<res path="${out.res.absolute.dir}" />
<res path="${resource.absolute.dir}" />
<nocompress /> <!-- forces no compression on any files in assets or res/raw -->
<!-- <nocompress extension="xml" /> forces no compression on specific file extensions in assets and res/raw -->
</aapt>
</do-only-if-not-library>
</target>
</project>
add file extension is mp3.I use mydb.mp3in assets folder and copy .this run without error.show check it.
Using GZIP would be another method. you only need to wrap InputStream inside GZIPInputStream. I used this for a database which size about 3.0 MB and output compress file was about 600KB.
For copying DB in firs run, I gzipped my source .db file using GZIP
tool.
Then renamed it to .jpg in order to avoid more compression
(these processes are done before compile APK FILE).
Then for reading compressed GZIP file from assetss
and copying it:
private void copydatabase() throws IOException {
// Open your local db as the input stream
InputStream myinput = mContext.getAssets().open(DB_NAME_ASSET);
BufferedInputStream buffStream = new BufferedInputStream(myinput);
GZIPInputStream zis = new GZIPInputStream(buffStream);
// Path to the just created empty db
String outfilename = DB_PATH + DB_NAME;
// Open the empty db as the output stream
OutputStream myoutput = new FileOutputStream(outfilename);
// transfer byte to inputfile to outputfile
byte[] buffer = new byte[1024];
int length;
while ((length = zis.read(buffer)) > 0) {
myoutput.write(buffer, 0, length);
}
// Close the streams
myoutput.flush();
myoutput.close();
zis.close();
buffStream.close();
myinput.close();
}
set database to multiple part with a program e.g "Win Hex", you can download from Link
and continue Load files bigger than 1M from assets folder
Instead of assets folder, I have put my large files in the raw folder. It works for me.
I use NetBeans to build the package and I not found how to change the settings of AAPT.
I have not tried the png, but the mp3 are compressed.
I can compile the package and then enter the assets folder with the parameter -0?
what would be the correct command to use?
Related
Hi I am uploading Large video files to server using Volley Multi-part Api but it takes much time for upload to server
Is it better to split my video files and send to server? If it is better please provide me code how can I do that, If not what is the best way to uploading big videos files to server quickly?
To split file into parts (chunks):
public static List<File> splitFile(File f) throws IOException {
int partCounter = 1;
List<File> result = new ArrayList<>();
int sizeOfFiles = 1024 * 1024;// 1MB
byte[] buffer = new byte[sizeOfFiles]; // create a buffer of bytes sized as the one chunk size
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(f));
String name = f.getName();
int tmp = 0;
while ((tmp = bis.read(buffer)) > 0) {
File newFile = new File(f.getParent(), name + "." + String.format("%03d", partCounter++)); // naming files as <inputFileName>.001, <inputFileName>.002, ...
FileOutputStream out = new FileOutputStream(newFile);
out.write(buffer, 0, tmp);//tmp is chunk size. Need it for the last chunk, which could be less then 1 mb.
result.add(newFile);
}
return result;
}
This method will split your file to chunks of size of 1MB (excluding the last chunk). After words you can send all these chunks too the server.
Also if you need to merge these files:
public static void mergeFiles(List<File> files, File into)
throws IOException {
BufferedOutputStream mergingStream = new BufferedOutputStream(new FileOutputStream(into))
for (File f : files) {
InputStream is = new FileInputStream(f);
Files.copy(is, mergingStream);
is.close();
}
mergingStream.close();
}
Just in case if your server is in Java also
I'm reading files from a ZIP file that's located in my Android assets folder using ZipInputStream: it works, but it's really slow, as it has to read it sequentially using getNextEntry(), and there are quite a lot of files.
If I copy the ZIP file onto the SD card, reading is really fast when using ZipFile.getEntry, but I didn't find a way to use ZipFile with the asset file!
Is there any way to access the ZIP in the asset folder in a speedy way? Or do I really have to copy the ZIP to the SD card?
(BTW, in case anybody wonders why I'm doing this: the app is larger than 50 MB, so in order to get it in the Play Store I have to use Expansion APKs; however, as this app should also be put into the Amazon App Store, I have to use another version for this, as Amazon doesn't support Expansion APKs, naturally... I thought that accessing a ZIP file at two different locations would be an easy way to handle this, but alas...)
This works for me:
private void loadzip(String folder, InputStream inputStream) throws IOException
{
ZipInputStream zipIs = new ZipInputStream(inputStream);
ZipEntry ze = null;
while ((ze = zipIs.getNextEntry()) != null) {
FileOutputStream fout = new FileOutputStream(folder +"/"+ ze.getName());
byte[] buffer = new byte[1024];
int length = 0;
while ((length = zipIs.read(buffer))>0) {
fout.write(buffer, 0, length);
}
zipIs.closeEntry();
fout.close();
}
zipIs.close();
}
You can store the uncompressed files directly in assets (i.e. unpack the zip into assets/ folder). This way, you can access the files directly and they will be compressed anyway when you build the APK.
You can create a ZipInputStream in the following way :
ZipInputStream zipIs = new ZipInputStream(context.getResources().openRawResource(your.package.com.R.raw.filename));
ZipEntry ze = null;
while ((ze = zipIs.getNextEntry()) != null) {
FileOutputStream fout = new FileOutputStream(FOLDER_NAME +"/"+ ze.getName());
byte[] buffer = new byte[1024];
int length = 0;
while ((length = zipIs.read(buffer))>0) {
fout.write(buffer, 0, length);
}
zipIs .closeEntry();
fout.close();
}
zipIs .close();
I need a list of italian words for a game I'm developing but I can't actually make it copy my database from assets. I tried quitea lot of solutions I found on the website, such as:
Using your own SQLite database in Android applications
how to copy large database which occupies much memory from assets folder to my application?
Load files bigger than 1M from assets folder
But I had no luck, it keeps on giving me this error on the line
os.write(buffer, 0, len);
but I can't understand why. Here's the function's code and the constants I'm using.
A strange thins is that my database stops copying after 11.45MB, just 1 MB away from the goal.
Can someone help me solve this? Many thanks :)
Use SQLiteAssetHelper, which has a debugged version of the package-the-database-with-the-app logic, so you do not need to mess with any of this yourself.
First of all by default asset folder supports max size for db file is 1mb.
You need to divide your database into parts.
Download HJSplit and divide your database into small parts
like 13MB = 13 parts each of 1MB.
demoDB.sqlitedb= 13MB
then
demodb..sqlitedb.001
demodb..sqlitedb.002
demodb..sqlitedb.003
demodb..sqlitedb.004
...
...
demodb..sqlitedb.013
Then use the following code to merge your database.
private void copyDataBase() throws IOException {
AssetManager am = mContext.getAssets();
OutputStream os = new FileOutputStream(DB_PATH + DB_NAME);
byte[] b = new byte[1024];
String[] files = am.list("");
Arrays.sort(files);
int r;
for (int i = 1; i <= 9; i++) {
InputStream is = am.open("demoDB.sqlitedb.00" + i);
while ((r = is.read(b)) != -1) {
os.write(b, 0, r);
}
Log.i("BABY_DATABASE_HELPER", "Copying the database (part " + i
+ " of 9)");
is.close();
}
os.close();
}
private void copyDataBase() throws IOException {
AssetManager am = getAssets();
OutputStream os = new FileOutputStream(DB_PATH + DB_NAME);
byte[] b = new byte[1024];
String[] files = am.list("");
Arrays.sort(files);
int r;
for (int i = 1; i <=21; i++) {
InputStream is;
if ( i < 10){
System.out.println("coping file demoDB.sqlitedb.00"+i );
is = am.open("demoDB.sqlitedb.00" + i);
}else{
System.out.println("coping file demoDB.sqlitedb.0"+i );
is = am.open("demoDB.sqlitedb.0" + i);
}
while ((r = is.read(b)) != -1) {
os.write(b, 0, r);
}
is.close();
}
os.close();
}
I know this was answered but I ran into something like this while creating tests where I wanted to store a particular database with faults in it to test bad data. Problem was test database was not found at all in the assets. To even see it I had to do this:
InputStream is = mContext.createPackageContext("com.activities.tests", Context.CONTEXT_IGNORE_SECURITY).getAssets().open("mydb.db");
By ignoring the security you can see it and then take the inputstream and save it to an external directory.
If I open a big zip file (250MB) via the ZipFile class and try to read the entries. This works fine on 2.x in the emulator and real hardware. If I use the exact some code on my tablet (Asus Transformer running 4.0.3) or the emulator (3.2), I can't read any entries. The size() function of the ZipFile class always returns zero and the ZipFile does not return any zip entries. Even the a zip app that comes with the ROM on my tablet can't read any entries. The zip file is not damaged. I checked it.
The code to read from ZipFile works fine on all version with smaller zip files. What has changed between 2.x and 3.x/4.x??
My Testfile is the C64Music.zip from the HighVoltage Sid Collection. It contains over 40.000 files and is around 250MB.
I have no clue where to look at.
This is a known issue with the android ZipFile implementation:
http://code.google.com/p/android/issues/detail?id=23207
Basically zip files only support up to 65k entries. There is an extended version of the zip file format called Zip64, which supports a larger number of entries. Unfortunately ZipFile on android cannot read Zip64. You will probably find that the C64Music.zip file is in the Zip64 format
A work around is to use the Apache Commons Compress library instead of the native implementation. Their version of ZipFile supports Zip64: http://commons.apache.org/compress/apidocs/org/apache/commons/compress/archivers/zip/ZipFile.html
public class Compress {
private static final int BUFFER = 2048;
private String[] _files;
private String _zipFile;
public Compress(String[] files, String zipFile) {
_files = files;
_zipFile = zipFile;
}
public void zip() {
try {
BufferedInputStream origin = null;
FileOutputStream dest = new FileOutputStream(_zipFile);
ZipOutputStream out = new ZipOutputStream(new BufferedOutputStream(dest));
byte data[] = new byte[BUFFER];
for(int i=0; i < _files.length; i++) {
Log.v("Compress", "Adding: " + _files[i]);
FileInputStream fi = new FileInputStream(_files[i]);
origin = new BufferedInputStream(fi, BUFFER);
ZipEntry entry = new ZipEntry(_files[i].substring(_files[i].lastIndexOf("/") + 1));
out.putNextEntry(entry);
int count;
while ((count = origin.read(data, 0, BUFFER)) != -1) {
out.write(data, 0, count);
}
origin.close();
}
out.close();
} catch(Exception e) {
e.printStackTrace();
}
}
}
Call Compress like given below where you want to zip a file :----
String zipFilePath = "fileName.zip";
File zipFile = new File(zipFilePath);
String[] files = new String[] {"/sdcard/fileName"};
if(!zipFile.exists()){
new Compress(files, zipFilePath).zip();
}
A lot of changes were made in 3.x/4.x to prevent abuse against the UI thread. Therefore, it is possible that your app is crashing because you are not offloading the expensive disk I/O operation to a separate Thread.
How can you read GZIP file in Android located in the "ASSETS" (or resources/raw) folder?
I have tried the following code, but my stream size is always 1.
GZIPInputStream fIn = new GZIPInputStream(mContext.getResources().openRawResource(R.raw.myfilegz));
int size = fIn.available();
for some reason the size is always 1. But if Idon't GZIP the file, it works fine.
NOTE:
Using Android 1.5
I met the same problem when reading a gz file from assets folder.
It's caused by the file name of the gz file. Just renaming yourfile.gz to other name like yourfile.bin. It seems Android build system would decompress a file automatically if it thought it's a gz.
public class ResLoader {
/**
* #param res
* #throws IOException
* #throws FileNotFoundException
* #throws IOException
*/
static void unpackResources() throws FileNotFoundException, IOException {
final int BUFFER = 8192;
android.content.res.Resources t = TestingE3d.mContext.getResources();
InputStream fis = t.openRawResource(R.raw.resources);
if (fis == null)
return;
ZipInputStream zin = new ZipInputStream(new BufferedInputStream(fis,
BUFFER));
ZipEntry entry;
while ((entry = zin.getNextEntry()) != null) {
int count;
FileOutputStream fos = TestingE3d.mContext.openFileOutput(entry
.getName(), 0);
BufferedOutputStream dest = new BufferedOutputStream(fos, BUFFER);
byte data[] = new byte[BUFFER];
while ((count = zin.read(data, 0, BUFFER)) != -1) {
dest.write(data, 0, count);
// Log.v("NOTAG", "writing "+count + " to "+entry.getName());
}
dest.flush();
dest.close();
}
zin.close();
}
}
R.raw.resources is a zip file - this class will decompress all files in that zip to your local folder.
I use this for NDK.
you can access your fils from ndk through:
/data/data//files/
package = package where ResLoader resides
filename = one of files that is in raw/resources.zip
this is the documented behavior of InflaterInputStream.available:
http://java.sun.com/javase/6/docs/api/java/util/zip/InflaterInputStream.html#available()
Returns 0 after EOF has been reached, otherwise always return 1.
abusing available is a common mistake --- in no case can you assume that it tells you the length of a file (though it sometimes happens to do so, as you've noticed). you want to keep calling read(byte[], int, int) until it returns 0. if you want the length to allocate a byte[] up front, you probably want to create a ByteArrayOutputStream and write to that each time you read, and then get a byte[] from that when you exit the loop. this works for all InputStreams in all cases.
It seems that the build system treats .gz files as a special case, even when it's included as a raw resource. Rename the .gz file to have a different extension, say .raw or .bin .
Valid at least for Android Studio 2.2 . I can't find any docs to confirm this is expected behaviour or, better, how to prevent it, but changing the extension at least works around the problem.
What happens if you use AssetManager instead of Resources? Example:
InputStream is = mContext.getAssets().open("myfilegz");
GZIPInputStream fIn = new GZIPINputStream(is);
Internally, Resources is just calling AssetManager; I wonder if somewhere along the way it musses things up.
Try looking at the source for Translate from apps-for-android open source project and see if that helps at all.
They use GZIPInputStream on a raw file in their selectRandomWord() function [line 326] (source pasted below)
public void selectRandomWord() {
BufferedReader fr = null;
try {
GZIPInputStream is =
new GZIPInputStream(getResources().openRawResource(R.raw.dictionary));