Android: read a GZIP file in the ASSETS folder - android

How can you read GZIP file in Android located in the "ASSETS" (or resources/raw) folder?
I have tried the following code, but my stream size is always 1.
GZIPInputStream fIn = new GZIPInputStream(mContext.getResources().openRawResource(R.raw.myfilegz));
int size = fIn.available();
for some reason the size is always 1. But if Idon't GZIP the file, it works fine.
NOTE:
Using Android 1.5

I met the same problem when reading a gz file from assets folder.
It's caused by the file name of the gz file. Just renaming yourfile.gz to other name like yourfile.bin. It seems Android build system would decompress a file automatically if it thought it's a gz.

public class ResLoader {
/**
* #param res
* #throws IOException
* #throws FileNotFoundException
* #throws IOException
*/
static void unpackResources() throws FileNotFoundException, IOException {
final int BUFFER = 8192;
android.content.res.Resources t = TestingE3d.mContext.getResources();
InputStream fis = t.openRawResource(R.raw.resources);
if (fis == null)
return;
ZipInputStream zin = new ZipInputStream(new BufferedInputStream(fis,
BUFFER));
ZipEntry entry;
while ((entry = zin.getNextEntry()) != null) {
int count;
FileOutputStream fos = TestingE3d.mContext.openFileOutput(entry
.getName(), 0);
BufferedOutputStream dest = new BufferedOutputStream(fos, BUFFER);
byte data[] = new byte[BUFFER];
while ((count = zin.read(data, 0, BUFFER)) != -1) {
dest.write(data, 0, count);
// Log.v("NOTAG", "writing "+count + " to "+entry.getName());
}
dest.flush();
dest.close();
}
zin.close();
}
}
R.raw.resources is a zip file - this class will decompress all files in that zip to your local folder.
I use this for NDK.
you can access your fils from ndk through:
/data/data//files/
package = package where ResLoader resides
filename = one of files that is in raw/resources.zip

this is the documented behavior of InflaterInputStream.available:
http://java.sun.com/javase/6/docs/api/java/util/zip/InflaterInputStream.html#available()
Returns 0 after EOF has been reached, otherwise always return 1.
abusing available is a common mistake --- in no case can you assume that it tells you the length of a file (though it sometimes happens to do so, as you've noticed). you want to keep calling read(byte[], int, int) until it returns 0. if you want the length to allocate a byte[] up front, you probably want to create a ByteArrayOutputStream and write to that each time you read, and then get a byte[] from that when you exit the loop. this works for all InputStreams in all cases.

It seems that the build system treats .gz files as a special case, even when it's included as a raw resource. Rename the .gz file to have a different extension, say .raw or .bin .
Valid at least for Android Studio 2.2 . I can't find any docs to confirm this is expected behaviour or, better, how to prevent it, but changing the extension at least works around the problem.

What happens if you use AssetManager instead of Resources? Example:
InputStream is = mContext.getAssets().open("myfilegz");
GZIPInputStream fIn = new GZIPINputStream(is);
Internally, Resources is just calling AssetManager; I wonder if somewhere along the way it musses things up.

Try looking at the source for Translate from apps-for-android open source project and see if that helps at all.
They use GZIPInputStream on a raw file in their selectRandomWord() function [line 326] (source pasted below)
public void selectRandomWord() {
BufferedReader fr = null;
try {
GZIPInputStream is =
new GZIPInputStream(getResources().openRawResource(R.raw.dictionary));

Related

Android assets, how can I read a file from a subfolder?

The code below works: it reads a file named "file.txt" which is located in the "assets" folder of the APK and stores it in a buffer. So far, so good:
String u = "content://com.example.app/file.txt:assets"
ContentResolver r = controls.activity.getContentResolver();
InputStream in = r.openInputStream(Uri.parse(u));
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];
int n = in.read(buffer);
while (n >= 0) {
out.write(buffer, 0, n);
n = in.read(buffer);
}
in.close();
return out.toByteArray();
If however the file I want to read is in a subfolder of assets, e.g. subfolder "sub", and I provide this Uri to the above code:
String u = "content://com.example.app/sub/file.txt:assets"
... then in this case I don't get anything. The file is there, as assets/sub/file.txt but the above code returns an empty buffer. The only change I made is to replace "file.txt" with "sub/file.txt" which points to where the file is stored.
What am I doing wrong? Is it wrong to create the uri string manually like that? I believe it's allowed to store files in assets subfolders... If that's allowed, how do I specify the path in the uri string?
Note that I'm not trying to give access to the file to another app, I just want to read my own file from my own APK's assets and put it in a buffer for internal use.
Any help is greatly appreciated!
Use AssetManager and its open() method. So, you would replace:
ContentResolver r = controls.activity.getContentResolver();
InputStream in = r.openInputStream(Uri.parse(u));
with:
AssetManager assets = controls.activity.getAssets();
InputStream in = assets.open("sub/file.txt");

How can I get file path on raw resources in Android Studio?

I have been using this tutorial to make some face detections on picture. The problem is when I getting the file path that used on java
String xmlFile = "E:/OpenCV/facedetect/lbpcascade_frontalface.xml";
CascadeClassifier classifier = new CascadeClassifier(xmlFile);
How can I translate on android studio. I try put my lbpcascade_frontalface.xml on raw resources. CascadeClassifier is a class that opencv library provided. The only problem is they only loaded string path (on xmlfile).
this is my code.
String pathtoRes = getRawPathAtempt2(context);
CascadeClassifier cascadeClassifier = new CascadeClassifier();
cascadeClassifier.load(pathtoRes);
I translated to some method like this.
public String getRawPathAtempt2(Context context) {
return "android.resource://" + context.getPackageName() + "/raw/" + "lbpcascade_frontalface.xml";
}
I get the asertions error by opencv that tell me the file is null. Thats mean I had been wrong when I used file path on my method. how can I get file path on raw resources? help me please I have been stuck for several days now
how can i get file path on raw resources?
You can't. There is no path. It is a file on your development machine. It is not a file on the device. Instead, it is an entry in the APK file that is your app on the device.
If your library supports an InputStream instead of a filesystem path, getResources().openRawResource() on a Context to get an InputStream on your raw resource.
This is how i solved the problem with #CommonWare Answer
i'm using input stream to get file
is = context.getResources().openRawResource(R.raw.lbpcascade_frontalface);
so make the file will be opened and i will make a point to the File class
File cascadeDir = context.getDir("cascade", Context.MODE_PRIVATE);
mCascadeFile = new File(cascadeDir, "lbpcascade_frontalface.xml");
and then i make a search to the path file using getAbsolutePath like this
cascadeClassifier.load(mCascadeFile.getAbsolutePath());
take a look at my full code
try {
is = context.getResources().openRawResource(R.raw.lbpcascade_frontalface);
File cascadeDir = context.getDir("cascade", Context.MODE_PRIVATE);
mCascadeFile = new File(cascadeDir, "lbpcascade_frontalface.xml");
os = new FileOutputStream(mCascadeFile);
byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = is.read(buffer)) != -1) {
os.write(buffer, 0, bytesRead);
}
is.close();
os.close();
cascadeClassifier.load(mCascadeFile.getAbsolutePath());
} catch (IOException e) {
Log.i(TAG, "face cascade not found");
}
Uri video = Uri.parse("android.resource://com.test.test/raw/filename");
Using this you can access the file in raw folder, if you want to access the file in asset folder use this URL...
file:///android_asset/filename
Make sure you don't add the extension to the filename. E.g. "/movie" not "/movie.mp4".
Refer to this link:
Raw folder url path?
We can use this Alternative method for finding the path of Files in Raw Folder.
InputStream inputStreamdat=getResources().openRawResource(R.raw.face_landmark_model);
File model=getDir("model", Context.MODE_PRIVATE);
File modelFile=new
File(model,"face_landmark_model.dat");
FileOutputStream os1 = new
FileOutputStream(modelFile);
byte[] buffer1 = new byte[4096];
int bytesRead1;
while ((bytesRead1=inputStreamdat.read(buffer1))!=-1) {
os1.write(buffer1, 0, bytesRead1);
}
inputStreamdat.close();
os1.close();
You Can Use
modelFile.getAbsolutePath()); For getting the Path

How can I retrieve an Android resource from its name (in Kotlin)? [duplicate]

I want to open a file from the folder res/raw/.
I am absolutely sure that the file exists.
To open the file I have tried
File ddd = new File("res/raw/example.png");
The command
ddd.exists();
yields FALSE. So this method does not work.
Trying
MyContext.getAssets().open("example.png");
ends up in an exception with getMessage() "null".
Simply using
R.raw.example
is not possible because the filename is only known during runtime as a string.
Why is it so difficult to access a file in the folder /res/raw/ ?
With the help of the given links I was able to solve the problem myself. The correct way is to get the resource ID with
getResources().getIdentifier("FILENAME_WITHOUT_EXTENSION",
"raw", getPackageName());
To get it as a InputStream
InputStream ins = getResources().openRawResource(
getResources().getIdentifier("FILENAME_WITHOUT_EXTENSION",
"raw", getPackageName()));
Here is example of taking XML file from raw folder:
InputStream XmlFileInputStream = getResources().openRawResource(R.raw.taskslists5items); // getting XML
Then you can:
String sxml = readTextFile(XmlFileInputStream);
when:
public String readTextFile(InputStream inputStream) {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
byte buf[] = new byte[1024];
int len;
try {
while ((len = inputStream.read(buf)) != -1) {
outputStream.write(buf, 0, len);
}
outputStream.close();
inputStream.close();
} catch (IOException e) {
}
return outputStream.toString();
}
You can read files in raw/res using getResources().openRawResource(R.raw.myfilename).
BUT there is an IDE limitation that the file name you use can only contain lower case alphanumeric characters and dot. So file names like XYZ.txt or my_data.bin will not be listed in R.
Here are two approaches you can read raw resources using Kotlin.
You can get it by getting the resource id. Or, you can use string identifier in which you can programmatically change the filename with incrementation.
Cheers mate 🎉
// R.raw.data_post
this.context.resources.openRawResource(R.raw.data_post)
this.context.resources.getIdentifier("data_post", "raw", this.context.packageName)

Android 3.x+ Java ZipFile Class - Can't read ZipEntries from big files

If I open a big zip file (250MB) via the ZipFile class and try to read the entries. This works fine on 2.x in the emulator and real hardware. If I use the exact some code on my tablet (Asus Transformer running 4.0.3) or the emulator (3.2), I can't read any entries. The size() function of the ZipFile class always returns zero and the ZipFile does not return any zip entries. Even the a zip app that comes with the ROM on my tablet can't read any entries. The zip file is not damaged. I checked it.
The code to read from ZipFile works fine on all version with smaller zip files. What has changed between 2.x and 3.x/4.x??
My Testfile is the C64Music.zip from the HighVoltage Sid Collection. It contains over 40.000 files and is around 250MB.
I have no clue where to look at.
This is a known issue with the android ZipFile implementation:
http://code.google.com/p/android/issues/detail?id=23207
Basically zip files only support up to 65k entries. There is an extended version of the zip file format called Zip64, which supports a larger number of entries. Unfortunately ZipFile on android cannot read Zip64. You will probably find that the C64Music.zip file is in the Zip64 format
A work around is to use the Apache Commons Compress library instead of the native implementation. Their version of ZipFile supports Zip64: http://commons.apache.org/compress/apidocs/org/apache/commons/compress/archivers/zip/ZipFile.html
public class Compress {
private static final int BUFFER = 2048;
private String[] _files;
private String _zipFile;
public Compress(String[] files, String zipFile) {
_files = files;
_zipFile = zipFile;
}
public void zip() {
try {
BufferedInputStream origin = null;
FileOutputStream dest = new FileOutputStream(_zipFile);
ZipOutputStream out = new ZipOutputStream(new BufferedOutputStream(dest));
byte data[] = new byte[BUFFER];
for(int i=0; i < _files.length; i++) {
Log.v("Compress", "Adding: " + _files[i]);
FileInputStream fi = new FileInputStream(_files[i]);
origin = new BufferedInputStream(fi, BUFFER);
ZipEntry entry = new ZipEntry(_files[i].substring(_files[i].lastIndexOf("/") + 1));
out.putNextEntry(entry);
int count;
while ((count = origin.read(data, 0, BUFFER)) != -1) {
out.write(data, 0, count);
}
origin.close();
}
out.close();
} catch(Exception e) {
e.printStackTrace();
}
}
}
Call Compress like given below where you want to zip a file :----
String zipFilePath = "fileName.zip";
File zipFile = new File(zipFilePath);
String[] files = new String[] {"/sdcard/fileName"};
if(!zipFile.exists()){
new Compress(files, zipFilePath).zip();
}
A lot of changes were made in 3.x/4.x to prevent abuse against the UI thread. Therefore, it is possible that your app is crashing because you are not offloading the expensive disk I/O operation to a separate Thread.

Load files bigger than 1M from assets folder

I'm going crazy, I created a file object, so it can be read with ObjectInputStream, and I placed the assets folder.
The method works with a file smaller than 1M, and give error with larger files.
I read that is a limit of Android platform, but I also know that can be "easily" avoided.
Those who have downloaded the game Reging Thunder, for example, can easily see that in their assets folder is a file 18.9M large.
This is my code for read 1 object from a ObjecInputStream
File f = File.createTempFile("mytempfile", "dat");
FileOutputStream fos = new FileOutputStream(f);
InputStream is = mc.getAssets().open(path,3);
ObjectInputStream ois=new ObjectInputStream(is);
byte[] data = (byte[]) ois.readObject();
fos.write(data);
fos.flush();
fos.close();
ois.close();
is.close();
now I have an uncompressed file and I can use it without worrying about the error "This file can not be opened as a file descriptor; it is probably compressed"
This function works well with files smaller than 1M, with bigger files return an
java.io.IOException on line "ObjectInputStream ois=new ObjectInputStream(is);"
why??
Faced the same issue. I've cut up my 4MB file into 1 MB chunks, and on the first run I join the chunks into a data folder on the phone. As an added bonus, the APK is properly compressed. The chunk files are called 1.db, 2.db, etc. The code goes like this:
File Path = Ctxt.getDir("Data", 0);
File DBFile = new File(Path, "database.db");
if(!DBFile.exists() || DatabaseNeedsUpgrade) //Need to copy...
CopyDatabase(Ctxt, DBFile);
static private void CopyDatabase(Context Ctxt, File DBFile) throws IOException
{
AssetManager assets = Ctxt.getAssets();
OutputStream outstream = new FileOutputStream(DBFile);
DBFile.createNewFile();
byte []b = new byte[1024];
int i, r;
String []assetfiles = assets.list("");
Arrays.sort(assetfiles);
for(i=1;i<10;i++) //I have definitely less than 10 files; you might have more
{
String partname = String.format("%d.db", i);
if(Arrays.binarySearch(assetfiles, partname) < 0) //No such file in assets - time to quit the loop
break;
InputStream instream = assets.open(partname);
while((r = instream.read(b)) != -1)
outstream.write(b, 0, r);
instream.close();
}
outstream.close();
}
The limitation is on compressed assets. If the asset is uncompressed, the system can memory-map the file data and use the Linux virtual memory paging system to pull in or discard 4K chunks as appropriate. (The "zipalign" tool ensures that uncompressed assets are word-aligned in the file, which means they'll also be aligned in memory when directly mapped.)
If the asset is compressed, the system has to uncompress the entire thing to memory. If you have a 20MB asset, that means 20MB of physical memory is tied up by your application.
Ideally the system would employ some sort of windowed compression, so that only parts need to be present, but that requires some fanciness in the asset API and a compression scheme that works well with random access. Right now APK == Zip with "deflate" compression, so that's not practical.
You can keep your assets uncompressed by giving them a suffix of a file type that doesn't get compressed (e.g. ".png" or ".mp3"). You may also be able to add them manually during the build process with "zip -0" instead of having them bundled up by aapt. This will likely increase the size of your APK.
Like Seva suggested you can split up your file in chunks. I used this to split up my 4MB file
public static void main(String[] args) throws Exception {
String base = "tracks";
String ext = ".dat";
int split = 1024 * 1024;
byte[] buf = new byte[1024];
int chunkNo = 1;
File inFile = new File(base + ext);
FileInputStream fis = new FileInputStream(inFile);
while (true) {
FileOutputStream fos = new FileOutputStream(new File(base + chunkNo + ext));
for (int i = 0; i < split / buf.length; i++) {
int read = fis.read(buf);
fos.write(buf, 0, read);
if (read < buf.length) {
fis.close();
fos.close();
return;
}
}
fos.close();
chunkNo++;
}
}
If you do not need to combine the files into a single file on the device again, just use this InputStream, which combines them into one on the fly.
import java.io.IOException;
import java.io.InputStream;
import android.content.res.AssetManager;
public class SplitFileInputStream extends InputStream {
private String baseName;
private String ext;
private AssetManager am;
private int numberOfChunks;
private int currentChunk = 1;
private InputStream currentIs = null;
public SplitFileInputStream(String baseName, String ext, int numberOfChunks, AssetManager am) throws IOException {
this.baseName = baseName;
this.am = am;
this.numberOfChunks = numberOfChunks;
this.ext = ext;
currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);
}
#Override
public int read() throws IOException {
int read = currentIs.read();
if (read == -1 && currentChunk < numberOfChunks) {
currentIs.close();
currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);
return read();
}
return read;
}
#Override
public int available() throws IOException {
return currentIs.available();
}
#Override
public void close() throws IOException {
currentIs.close();
}
#Override
public void mark(int readlimit) {
throw new UnsupportedOperationException();
}
#Override
public boolean markSupported() {
return false;
}
#Override
public int read(byte[] b, int offset, int length) throws IOException {
int read = currentIs.read(b, offset, length);
if (read < length && currentChunk < numberOfChunks) {
currentIs.close();
currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);
read += read(b, offset + read, length - read);
}
return read;
}
#Override
public int read(byte[] b) throws IOException {
return read(b, 0, b.length);
}
#Override
public synchronized void reset() throws IOException {
if (currentChunk == 1) {
currentIs.reset();
} else {
currentIs.close();
currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);
currentChunk = 1;
}
}
#Override
public long skip(long n) throws IOException {
long skipped = currentIs.skip(n);
if (skipped < n && currentChunk < numberOfChunks) {
currentIs.close();
currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);
skipped += skip(n - skipped);
}
return skipped;
}
}
Usage:
ObjectInputStream ois = new ObjectInputStream(new SplitFileInputStream("mytempfile", ".dat", 4, getAssets()));
Change the file extension to .mp3
I know this is an old question but I thought of a good solution.
Why not store the file already pre-zipped in the asset folder.
Then since it is already a zip file and therefore compressed it won't need to be compressed again. So if you wanted the file to be compressed to decrease the size of your apk but you don't want to deal with splitting up files I think this is easier.
When you need to read that file off the device just wrap the inputstream in a zipinputstream
http://developer.android.com/reference/java/util/zip/ZipInputStream.html
I found another solution, maybe you're interested in it.
In the root of your sources, where you have the build.xml file, you can overwrite the -package-resources target in the custom_rules.xml file, which is used for adding/modifying targets in ant without breaking anything in the standard android app build system.
Just create a file with this content:
<?xml version="1.0" encoding="UTF-8"?>
<project name="yourAppHere" default="help">
<target name="-package-resources" depends="-crunch">
<!-- only package resources if *not* a library project -->
<do-only-if-not-library elseText="Library project: do not package resources..." >
<aapt executable="${aapt}"
command="package"
versioncode="${version.code}"
versionname="${version.name}"
debug="${build.is.packaging.debug}"
manifest="${out.manifest.abs.file}"
assets="${asset.absolute.dir}"
androidjar="${project.target.android.jar}"
apkfolder="${out.absolute.dir}"
nocrunch="${build.packaging.nocrunch}"
resourcefilename="${resource.package.file.name}"
resourcefilter="${aapt.resource.filter}"
libraryResFolderPathRefid="project.library.res.folder.path"
libraryPackagesRefid="project.library.packages"
libraryRFileRefid="project.library.bin.r.file.path"
previousBuildType="${build.last.target}"
buildType="${build.target}"
ignoreAssets="${aapt.ignore.assets}">
<res path="${out.res.absolute.dir}" />
<res path="${resource.absolute.dir}" />
<nocompress /> <!-- forces no compression on any files in assets or res/raw -->
<!-- <nocompress extension="xml" /> forces no compression on specific file extensions in assets and res/raw -->
</aapt>
</do-only-if-not-library>
</target>
</project>
add file extension is mp3.I use mydb.mp3in assets folder and copy .this run without error.show check it.
Using GZIP would be another method. you only need to wrap InputStream inside GZIPInputStream. I used this for a database which size about 3.0 MB and output compress file was about 600KB.
For copying DB in firs run, I gzipped my source .db file using GZIP
tool.
Then renamed it to .jpg in order to avoid more compression
(these processes are done before compile APK FILE).
Then for reading compressed GZIP file from assetss
and copying it:
private void copydatabase() throws IOException {
// Open your local db as the input stream
InputStream myinput = mContext.getAssets().open(DB_NAME_ASSET);
BufferedInputStream buffStream = new BufferedInputStream(myinput);
GZIPInputStream zis = new GZIPInputStream(buffStream);
// Path to the just created empty db
String outfilename = DB_PATH + DB_NAME;
// Open the empty db as the output stream
OutputStream myoutput = new FileOutputStream(outfilename);
// transfer byte to inputfile to outputfile
byte[] buffer = new byte[1024];
int length;
while ((length = zis.read(buffer)) > 0) {
myoutput.write(buffer, 0, length);
}
// Close the streams
myoutput.flush();
myoutput.close();
zis.close();
buffStream.close();
myinput.close();
}
set database to multiple part with a program e.g "Win Hex", you can download from Link
and continue Load files bigger than 1M from assets folder
Instead of assets folder, I have put my large files in the raw folder. It works for me.
I use NetBeans to build the package and I not found how to change the settings of AAPT.
I have not tried the png, but the mp3 are compressed.
I can compile the package and then enter the assets folder with the parameter -0?
what would be the correct command to use?

Categories

Resources