I need a list of italian words for a game I'm developing but I can't actually make it copy my database from assets. I tried quitea lot of solutions I found on the website, such as:
Using your own SQLite database in Android applications
how to copy large database which occupies much memory from assets folder to my application?
Load files bigger than 1M from assets folder
But I had no luck, it keeps on giving me this error on the line
os.write(buffer, 0, len);
but I can't understand why. Here's the function's code and the constants I'm using.
A strange thins is that my database stops copying after 11.45MB, just 1 MB away from the goal.
Can someone help me solve this? Many thanks :)
Use SQLiteAssetHelper, which has a debugged version of the package-the-database-with-the-app logic, so you do not need to mess with any of this yourself.
First of all by default asset folder supports max size for db file is 1mb.
You need to divide your database into parts.
Download HJSplit and divide your database into small parts
like 13MB = 13 parts each of 1MB.
demoDB.sqlitedb= 13MB
then
demodb..sqlitedb.001
demodb..sqlitedb.002
demodb..sqlitedb.003
demodb..sqlitedb.004
...
...
demodb..sqlitedb.013
Then use the following code to merge your database.
private void copyDataBase() throws IOException {
AssetManager am = mContext.getAssets();
OutputStream os = new FileOutputStream(DB_PATH + DB_NAME);
byte[] b = new byte[1024];
String[] files = am.list("");
Arrays.sort(files);
int r;
for (int i = 1; i <= 9; i++) {
InputStream is = am.open("demoDB.sqlitedb.00" + i);
while ((r = is.read(b)) != -1) {
os.write(b, 0, r);
}
Log.i("BABY_DATABASE_HELPER", "Copying the database (part " + i
+ " of 9)");
is.close();
}
os.close();
}
private void copyDataBase() throws IOException {
AssetManager am = getAssets();
OutputStream os = new FileOutputStream(DB_PATH + DB_NAME);
byte[] b = new byte[1024];
String[] files = am.list("");
Arrays.sort(files);
int r;
for (int i = 1; i <=21; i++) {
InputStream is;
if ( i < 10){
System.out.println("coping file demoDB.sqlitedb.00"+i );
is = am.open("demoDB.sqlitedb.00" + i);
}else{
System.out.println("coping file demoDB.sqlitedb.0"+i );
is = am.open("demoDB.sqlitedb.0" + i);
}
while ((r = is.read(b)) != -1) {
os.write(b, 0, r);
}
is.close();
}
os.close();
}
I know this was answered but I ran into something like this while creating tests where I wanted to store a particular database with faults in it to test bad data. Problem was test database was not found at all in the assets. To even see it I had to do this:
InputStream is = mContext.createPackageContext("com.activities.tests", Context.CONTEXT_IGNORE_SECURITY).getAssets().open("mydb.db");
By ignoring the security you can see it and then take the inputstream and save it to an external directory.
Related
Hi I am uploading Large video files to server using Volley Multi-part Api but it takes much time for upload to server
Is it better to split my video files and send to server? If it is better please provide me code how can I do that, If not what is the best way to uploading big videos files to server quickly?
To split file into parts (chunks):
public static List<File> splitFile(File f) throws IOException {
int partCounter = 1;
List<File> result = new ArrayList<>();
int sizeOfFiles = 1024 * 1024;// 1MB
byte[] buffer = new byte[sizeOfFiles]; // create a buffer of bytes sized as the one chunk size
BufferedInputStream bis = new BufferedInputStream(new FileInputStream(f));
String name = f.getName();
int tmp = 0;
while ((tmp = bis.read(buffer)) > 0) {
File newFile = new File(f.getParent(), name + "." + String.format("%03d", partCounter++)); // naming files as <inputFileName>.001, <inputFileName>.002, ...
FileOutputStream out = new FileOutputStream(newFile);
out.write(buffer, 0, tmp);//tmp is chunk size. Need it for the last chunk, which could be less then 1 mb.
result.add(newFile);
}
return result;
}
This method will split your file to chunks of size of 1MB (excluding the last chunk). After words you can send all these chunks too the server.
Also if you need to merge these files:
public static void mergeFiles(List<File> files, File into)
throws IOException {
BufferedOutputStream mergingStream = new BufferedOutputStream(new FileOutputStream(into))
for (File f : files) {
InputStream is = new FileInputStream(f);
Files.copy(is, mergingStream);
is.close();
}
mergingStream.close();
}
Just in case if your server is in Java also
To unzip zip file I've used classes from package java.util.zip* by refering this and it works correctly however to unzip a file of 40MB it takes 59 seconds. When I tried same zip file on iPhone project (we are developing app for both platforms - Android & iPone & that have functionality to unzip zip file & save unzipped content to SDCARD-Android or document directory - iPhone ), it takes only 14 seconds. The iphone app uses ziparchive.
So my question are:
1.From above experiment it clears that unzipping & file write operation to SDCARD in Java consumes more time as compared to iPhone app, so I decided to use C/C++ level unzipping & file write operation using NDK.
Is this right choice ?
2.I have searched on google, stackoverflow & some suggested to use minizip but there no sufficient help on how to use minizip in android. Is anyboday have tried minizip for android ?
3.I also tried NDK development for libz to achieve my goal, as Libz is added in NDK but not getting how to use it. Is anybody tried libz in NDK?
4.Is there any other Framework in Java or C/C++ that unzip large zip file & write them to SDCARD in less time ?
Please help me.
Here is my Java Unzip Code :
public String unzip() {
String result;
try {
FileInputStream fin = new FileInputStream(this.filePath);
ZipInputStream zin = new ZipInputStream(fin);
ZipEntry ze = null;
while ((ze = zin.getNextEntry()) != null) {
Log.v("Unzip", "Unzipping " + ze.getName());
if (ze.isDirectory()) {
_dirChecker(ze.getName());
} else {
// Read 16 k at a time
byte[] buffer = new byte[16*1024];
int read;
FileOutputStream fout = new FileOutputStream(this.location+ "/" + ze.getName());
while ((read = zin.read(buffer)) != -1)
{
fout.write(buffer, 0, read);
}
zin.closeEntry();
fout.close();
}
}
zin.close();
result = "success";
} catch (Exception e) {
Log.e("unzip", "unzip", e);
result = "failure";
}
return result;
}
Why don't you try this code.It works awesome
String zipname = "data.zip";
FileInputStream fis = new FileInputStream(zipname);
ZipInputStream zis = new ZipInputStream(new BufferedInputStream(fis));
ZipEntry entry;
while ((entry = zis.getNextEntry()) != null) {
System.out.println("Unzipping: " + entry.getName());
int size;
byte[] buffer = new byte[2048];
FileOutputStream fos = new FileOutputStream(entry.getName());
BufferedOutputStream bos = new BufferedOutputStream(fos, buffer.length);
while ((size = zis.read(buffer, 0, buffer.length)) != -1) {
bos.write(buffer, 0, size);
}
bos.flush();
bos.close();
}
zis.close();
fis.close();
}
All of the unzip code eventually ends up in zlib. There is no Java implementation of "deflate" compression in Android core libs.
The only reason java.util.Zip should be slower than a "native" unzip is if the file I/O is done badly, e.g. something is using really small buffers. Looking at the code linked from the question, this is exactly what's happening -- it's operating on individual bytes.
One of the comments on the solution provided a patch that uses a 4K buffer. Drop that in and see what happens to your performance.
Try to just write 40Mb file to SDCard and measure time spent.
(Almost) all free (or even paid) implementations of zip archive support libraries are based on the same zlib code, which takes most processing speed during the unzipping. Java code should be much slower than native one, so I'd suggest to try NDK unzipping. Also, trying to unzip archive with zero compression level will give you a guess how much time unzipping code takes and how much time is spend on just data copying.
public boolean unzip(String zipfilepath, String destinationdir) {
try {
FileInputStream fis = new FileInputStream(zipfilepath);
ZipInputStream zis = new ZipInputStream(new BufferedInputStream(fis));
ZipEntry entry;
while ((entry = zis.getNextEntry()) != null) {
System.out.println("Unzipping: " + entry.getName());
int size;
byte[] buffer = new byte[2048];
FileOutputStream fos = new FileOutputStream(destinationdir+ "/" + entry.getName());
BufferedOutputStream bos = new BufferedOutputStream(fos, buffer.length);
while ((size = zis.read(buffer, 0, buffer.length)) != -1) {
bos.write(buffer, 0, size);
}
bos.flush();
bos.close();
}
zis.close();
fis.close();
return true;
}catch(Exception e){
return false;
}
}
String zipfilepath = context.getFilesDir().getPath()+"/"+myfile.zip;
String destinationdir = context.getFilesDir().getPath();
unzip(zipfilepath, destinationdir);
This question already has an answer here:
I need to be able to store sound files for my application on sdcard
(1 answer)
Closed 2 years ago.
I found this code which appears to be what I need in that it will copy byte for byte a file to the SDCard.
But how do I use it? say I have a text file called mytext.txt where do I put it in my application? and how would I reference it? I am using Eclipse
public static final void copyfile(String srFile, String dtFile){
try{
File f1 = new File(srFile);
File f2 = new File(dtFile);
InputStream in = new FileInputStream(f1);
OutputStream out = new FileOutputStream(f2);
byte[] buf = new byte[1024];
int len;
while ((len = in.read(buf)) > 0){
out.write(buf, 0, len);
}
in.close();
out.close();
System.out.println("File copied to " + f2.getAbsolutePath());
} catch(FileNotFoundException ex){
System.out.println(ex.getMessage() + " in the specified directory.");
System.exit(0);
} catch(IOException e){
System.out.println(e.getMessage());
}
}
I would make a FileUtilities class or somesuch. Have you looked at the examples here?
http://download.oracle.com/javase/tutorial/essential/io/copy.html
http://www.java2s.com/Code/Java/File-Input-Output/FileCopyinJava.htm
You don't want to blindly execute this code. It looks like it's meant for a java console app. System Printlines do not go anywhere that the user would see in an android application. I do not know what System.exit() does in an Android application, but you don't want to do this either. Depending on your application, you may want to add a toast notification that a copy fails. You want to at least log this.
Depending on the size of files you are copying, you may want to do this in a background thread as to not clog up your UI.
Well, at first glance that appears to be a sound method, except that you'd want to replace the System.out print statements with an android Log method... but besides that you could copy/paste that and include that method in a class.
To use it, however... you should have a look at the External Storage documentation.
http://developer.android.com/guide/topics/data/data-storage.html#filesExternal
You're going to need to use Android methods to get correct sdcard directories, etc...
You can add it as another method of your own Activity if your code is small, or you can create a utility class, let's suppose
class MyUtilities {
public static final void copyfile(String srFile, String dtFile) throws IOException, FileNotFoundException{
File f1 = new File(srFile);
File f2 = new File(dtFile);
InputStream in = new FileInputStream(f1);
OutputStream out = new FileOutputStream(f2);
byte[] buf = new byte[1024];
int len;
while ((len = in.read(buf)) > 0){
out.write(buf, 0, len);
}
in.close();
out.close();
Log.d("MyUtilities", "File copied to " + f2.getAbsolutePath());
}
}
and you will use it as:
TextEdit text1 = findViewById(R.id.text1);
TextEdit text2 = findViewById(R.id.text2);
String file1 = text1.getText();
String file2 = text2.getText();
if (text1 != null and text2 != null) {
try{
MyUtilities.copyfile (file1, file2);
} catch(FileNotFoundException ex){
Log.e("MyUtilities", ex.getMessage() + " in the specified directory.");
} catch(IOException e){
Log.e("MyUtilities", e.getMessage());
}
}
I added logs instead of the System.out and changed the Exception mechanism to better match android needs.
How can you read GZIP file in Android located in the "ASSETS" (or resources/raw) folder?
I have tried the following code, but my stream size is always 1.
GZIPInputStream fIn = new GZIPInputStream(mContext.getResources().openRawResource(R.raw.myfilegz));
int size = fIn.available();
for some reason the size is always 1. But if Idon't GZIP the file, it works fine.
NOTE:
Using Android 1.5
I met the same problem when reading a gz file from assets folder.
It's caused by the file name of the gz file. Just renaming yourfile.gz to other name like yourfile.bin. It seems Android build system would decompress a file automatically if it thought it's a gz.
public class ResLoader {
/**
* #param res
* #throws IOException
* #throws FileNotFoundException
* #throws IOException
*/
static void unpackResources() throws FileNotFoundException, IOException {
final int BUFFER = 8192;
android.content.res.Resources t = TestingE3d.mContext.getResources();
InputStream fis = t.openRawResource(R.raw.resources);
if (fis == null)
return;
ZipInputStream zin = new ZipInputStream(new BufferedInputStream(fis,
BUFFER));
ZipEntry entry;
while ((entry = zin.getNextEntry()) != null) {
int count;
FileOutputStream fos = TestingE3d.mContext.openFileOutput(entry
.getName(), 0);
BufferedOutputStream dest = new BufferedOutputStream(fos, BUFFER);
byte data[] = new byte[BUFFER];
while ((count = zin.read(data, 0, BUFFER)) != -1) {
dest.write(data, 0, count);
// Log.v("NOTAG", "writing "+count + " to "+entry.getName());
}
dest.flush();
dest.close();
}
zin.close();
}
}
R.raw.resources is a zip file - this class will decompress all files in that zip to your local folder.
I use this for NDK.
you can access your fils from ndk through:
/data/data//files/
package = package where ResLoader resides
filename = one of files that is in raw/resources.zip
this is the documented behavior of InflaterInputStream.available:
http://java.sun.com/javase/6/docs/api/java/util/zip/InflaterInputStream.html#available()
Returns 0 after EOF has been reached, otherwise always return 1.
abusing available is a common mistake --- in no case can you assume that it tells you the length of a file (though it sometimes happens to do so, as you've noticed). you want to keep calling read(byte[], int, int) until it returns 0. if you want the length to allocate a byte[] up front, you probably want to create a ByteArrayOutputStream and write to that each time you read, and then get a byte[] from that when you exit the loop. this works for all InputStreams in all cases.
It seems that the build system treats .gz files as a special case, even when it's included as a raw resource. Rename the .gz file to have a different extension, say .raw or .bin .
Valid at least for Android Studio 2.2 . I can't find any docs to confirm this is expected behaviour or, better, how to prevent it, but changing the extension at least works around the problem.
What happens if you use AssetManager instead of Resources? Example:
InputStream is = mContext.getAssets().open("myfilegz");
GZIPInputStream fIn = new GZIPINputStream(is);
Internally, Resources is just calling AssetManager; I wonder if somewhere along the way it musses things up.
Try looking at the source for Translate from apps-for-android open source project and see if that helps at all.
They use GZIPInputStream on a raw file in their selectRandomWord() function [line 326] (source pasted below)
public void selectRandomWord() {
BufferedReader fr = null;
try {
GZIPInputStream is =
new GZIPInputStream(getResources().openRawResource(R.raw.dictionary));
I'm going crazy, I created a file object, so it can be read with ObjectInputStream, and I placed the assets folder.
The method works with a file smaller than 1M, and give error with larger files.
I read that is a limit of Android platform, but I also know that can be "easily" avoided.
Those who have downloaded the game Reging Thunder, for example, can easily see that in their assets folder is a file 18.9M large.
This is my code for read 1 object from a ObjecInputStream
File f = File.createTempFile("mytempfile", "dat");
FileOutputStream fos = new FileOutputStream(f);
InputStream is = mc.getAssets().open(path,3);
ObjectInputStream ois=new ObjectInputStream(is);
byte[] data = (byte[]) ois.readObject();
fos.write(data);
fos.flush();
fos.close();
ois.close();
is.close();
now I have an uncompressed file and I can use it without worrying about the error "This file can not be opened as a file descriptor; it is probably compressed"
This function works well with files smaller than 1M, with bigger files return an
java.io.IOException on line "ObjectInputStream ois=new ObjectInputStream(is);"
why??
Faced the same issue. I've cut up my 4MB file into 1 MB chunks, and on the first run I join the chunks into a data folder on the phone. As an added bonus, the APK is properly compressed. The chunk files are called 1.db, 2.db, etc. The code goes like this:
File Path = Ctxt.getDir("Data", 0);
File DBFile = new File(Path, "database.db");
if(!DBFile.exists() || DatabaseNeedsUpgrade) //Need to copy...
CopyDatabase(Ctxt, DBFile);
static private void CopyDatabase(Context Ctxt, File DBFile) throws IOException
{
AssetManager assets = Ctxt.getAssets();
OutputStream outstream = new FileOutputStream(DBFile);
DBFile.createNewFile();
byte []b = new byte[1024];
int i, r;
String []assetfiles = assets.list("");
Arrays.sort(assetfiles);
for(i=1;i<10;i++) //I have definitely less than 10 files; you might have more
{
String partname = String.format("%d.db", i);
if(Arrays.binarySearch(assetfiles, partname) < 0) //No such file in assets - time to quit the loop
break;
InputStream instream = assets.open(partname);
while((r = instream.read(b)) != -1)
outstream.write(b, 0, r);
instream.close();
}
outstream.close();
}
The limitation is on compressed assets. If the asset is uncompressed, the system can memory-map the file data and use the Linux virtual memory paging system to pull in or discard 4K chunks as appropriate. (The "zipalign" tool ensures that uncompressed assets are word-aligned in the file, which means they'll also be aligned in memory when directly mapped.)
If the asset is compressed, the system has to uncompress the entire thing to memory. If you have a 20MB asset, that means 20MB of physical memory is tied up by your application.
Ideally the system would employ some sort of windowed compression, so that only parts need to be present, but that requires some fanciness in the asset API and a compression scheme that works well with random access. Right now APK == Zip with "deflate" compression, so that's not practical.
You can keep your assets uncompressed by giving them a suffix of a file type that doesn't get compressed (e.g. ".png" or ".mp3"). You may also be able to add them manually during the build process with "zip -0" instead of having them bundled up by aapt. This will likely increase the size of your APK.
Like Seva suggested you can split up your file in chunks. I used this to split up my 4MB file
public static void main(String[] args) throws Exception {
String base = "tracks";
String ext = ".dat";
int split = 1024 * 1024;
byte[] buf = new byte[1024];
int chunkNo = 1;
File inFile = new File(base + ext);
FileInputStream fis = new FileInputStream(inFile);
while (true) {
FileOutputStream fos = new FileOutputStream(new File(base + chunkNo + ext));
for (int i = 0; i < split / buf.length; i++) {
int read = fis.read(buf);
fos.write(buf, 0, read);
if (read < buf.length) {
fis.close();
fos.close();
return;
}
}
fos.close();
chunkNo++;
}
}
If you do not need to combine the files into a single file on the device again, just use this InputStream, which combines them into one on the fly.
import java.io.IOException;
import java.io.InputStream;
import android.content.res.AssetManager;
public class SplitFileInputStream extends InputStream {
private String baseName;
private String ext;
private AssetManager am;
private int numberOfChunks;
private int currentChunk = 1;
private InputStream currentIs = null;
public SplitFileInputStream(String baseName, String ext, int numberOfChunks, AssetManager am) throws IOException {
this.baseName = baseName;
this.am = am;
this.numberOfChunks = numberOfChunks;
this.ext = ext;
currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);
}
#Override
public int read() throws IOException {
int read = currentIs.read();
if (read == -1 && currentChunk < numberOfChunks) {
currentIs.close();
currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);
return read();
}
return read;
}
#Override
public int available() throws IOException {
return currentIs.available();
}
#Override
public void close() throws IOException {
currentIs.close();
}
#Override
public void mark(int readlimit) {
throw new UnsupportedOperationException();
}
#Override
public boolean markSupported() {
return false;
}
#Override
public int read(byte[] b, int offset, int length) throws IOException {
int read = currentIs.read(b, offset, length);
if (read < length && currentChunk < numberOfChunks) {
currentIs.close();
currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);
read += read(b, offset + read, length - read);
}
return read;
}
#Override
public int read(byte[] b) throws IOException {
return read(b, 0, b.length);
}
#Override
public synchronized void reset() throws IOException {
if (currentChunk == 1) {
currentIs.reset();
} else {
currentIs.close();
currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);
currentChunk = 1;
}
}
#Override
public long skip(long n) throws IOException {
long skipped = currentIs.skip(n);
if (skipped < n && currentChunk < numberOfChunks) {
currentIs.close();
currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);
skipped += skip(n - skipped);
}
return skipped;
}
}
Usage:
ObjectInputStream ois = new ObjectInputStream(new SplitFileInputStream("mytempfile", ".dat", 4, getAssets()));
Change the file extension to .mp3
I know this is an old question but I thought of a good solution.
Why not store the file already pre-zipped in the asset folder.
Then since it is already a zip file and therefore compressed it won't need to be compressed again. So if you wanted the file to be compressed to decrease the size of your apk but you don't want to deal with splitting up files I think this is easier.
When you need to read that file off the device just wrap the inputstream in a zipinputstream
http://developer.android.com/reference/java/util/zip/ZipInputStream.html
I found another solution, maybe you're interested in it.
In the root of your sources, where you have the build.xml file, you can overwrite the -package-resources target in the custom_rules.xml file, which is used for adding/modifying targets in ant without breaking anything in the standard android app build system.
Just create a file with this content:
<?xml version="1.0" encoding="UTF-8"?>
<project name="yourAppHere" default="help">
<target name="-package-resources" depends="-crunch">
<!-- only package resources if *not* a library project -->
<do-only-if-not-library elseText="Library project: do not package resources..." >
<aapt executable="${aapt}"
command="package"
versioncode="${version.code}"
versionname="${version.name}"
debug="${build.is.packaging.debug}"
manifest="${out.manifest.abs.file}"
assets="${asset.absolute.dir}"
androidjar="${project.target.android.jar}"
apkfolder="${out.absolute.dir}"
nocrunch="${build.packaging.nocrunch}"
resourcefilename="${resource.package.file.name}"
resourcefilter="${aapt.resource.filter}"
libraryResFolderPathRefid="project.library.res.folder.path"
libraryPackagesRefid="project.library.packages"
libraryRFileRefid="project.library.bin.r.file.path"
previousBuildType="${build.last.target}"
buildType="${build.target}"
ignoreAssets="${aapt.ignore.assets}">
<res path="${out.res.absolute.dir}" />
<res path="${resource.absolute.dir}" />
<nocompress /> <!-- forces no compression on any files in assets or res/raw -->
<!-- <nocompress extension="xml" /> forces no compression on specific file extensions in assets and res/raw -->
</aapt>
</do-only-if-not-library>
</target>
</project>
add file extension is mp3.I use mydb.mp3in assets folder and copy .this run without error.show check it.
Using GZIP would be another method. you only need to wrap InputStream inside GZIPInputStream. I used this for a database which size about 3.0 MB and output compress file was about 600KB.
For copying DB in firs run, I gzipped my source .db file using GZIP
tool.
Then renamed it to .jpg in order to avoid more compression
(these processes are done before compile APK FILE).
Then for reading compressed GZIP file from assetss
and copying it:
private void copydatabase() throws IOException {
// Open your local db as the input stream
InputStream myinput = mContext.getAssets().open(DB_NAME_ASSET);
BufferedInputStream buffStream = new BufferedInputStream(myinput);
GZIPInputStream zis = new GZIPInputStream(buffStream);
// Path to the just created empty db
String outfilename = DB_PATH + DB_NAME;
// Open the empty db as the output stream
OutputStream myoutput = new FileOutputStream(outfilename);
// transfer byte to inputfile to outputfile
byte[] buffer = new byte[1024];
int length;
while ((length = zis.read(buffer)) > 0) {
myoutput.write(buffer, 0, length);
}
// Close the streams
myoutput.flush();
myoutput.close();
zis.close();
buffStream.close();
myinput.close();
}
set database to multiple part with a program e.g "Win Hex", you can download from Link
and continue Load files bigger than 1M from assets folder
Instead of assets folder, I have put my large files in the raw folder. It works for me.
I use NetBeans to build the package and I not found how to change the settings of AAPT.
I have not tried the png, but the mp3 are compressed.
I can compile the package and then enter the assets folder with the parameter -0?
what would be the correct command to use?