I must be doing something really wrong. Running following code on Android it produces truncated file (_items_) without any exceptions or problems in the log. Running the same code with OpenJDK 7 it decompresses the file correctly.
try {
final InputStream fis = new GZIPInputStream(new FileInputStream("/storage/sdcard/_items"));
try {
final FileOutputStream fos = new FileOutputStream("/storage/sdcard/_items_");
try {
final byte[] buffer = new byte[1024];
int n;
while ((n = fis.read(buffer)) != -1) {
fos.write(buffer, 0, n);
}
} finally {
fos.close();
}
} finally {
fis.close();
}
} catch (final IOException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
I've tried this with Android emulator (API 18) and on Desire HD (Android 2.3.5) with the same buggy result.
Input file (_items): https://drive.google.com/file/d/0B6M72P2gzYmwaHg4SzRTYnRMOVk/edit?usp=sharing
Android truncated output file (_items_): https://drive.google.com/file/d/0B6M72P2gzYmwMUZIZ2FEaHNZUFk/edit?usp=sharing
The AOSP bug has been updated with analysis by an engineer from the Dalvik team.
Summary: the gzip stream has multiple members concatenated together, and the decompressor is expected to process them all, but Dalvik's implementation is stopping after the first.
Unless there's a way to convince the data source to compress its streams differently, you will need to find a replacement for GZIPInputStream.
Workaround is to use GZIPInputStream from JZlib (currently only in concatenated_gzip_streams branch). See https://github.com/ymnk/jzlib/issues/12 for more details.
Related
I currently work on an app where I use the phone camera and open CV to process the frames. Now I thought it would be cool to be able to send the frames to another Android client. I thought frame by frame with steamer could work, but don't know how to setup the host and if it's not efficient. Any suggestions?
If you just want to send each frame as a raw set of data you can use sockets.
This code below is old now but it worked fine when last tested - it sends an entire video but you can use the same to send whatever file you want:
//Send the video file to helper over a Socket connection so he helper can compress the video file
Socket helperSocket = null;
try {
Log.d("VideoChunkDistributeTask doInBackground","connecting to: " + helperIPAddress + ":" + helperPort);
helperSocket = new Socket(helperIPAddress, helperPort);
BufferedOutputStream helperSocketBOS = new BufferedOutputStream(helperSocket.getOutputStream());
byte[] buffer = new byte[4096];
//Write the video chunk to the output stream
//Open the file
File videoChunkFile = new File(videoChunkFileName);
BufferedInputStream chunkFileIS = new BufferedInputStream(new FileInputStream(videoChunkFile));
//First send a long with the file length - wrap the BufferedOutputStream in a DataOuputStream to
//allow us send a long directly
DataOutputStream helperSocketDOS = new DataOutputStream(
new BufferedOutputStream(helperSocket.getOutputStream()));
long chunkLength = videoChunkFile.length();
helperSocketDOS.writeLong(chunkLength);
Log.d("VideoChunkDistributeTask doInBackground","chunkLength: " + chunkLength);
//Now loop through the video chunk file sending it to the helper via the socket - note this will simply
//do nothing if the file is empty
int readCount = 0;
int totalReadCount = 0;
while(totalReadCount < chunkLength) {
//write the buffer to the output stream of the socket
readCount = chunkFileIS.read(buffer);
helperSocketDOS.write(buffer, 0, readCount);
totalReadCount += readCount;
}
Log.d("VideoChunkDistributeTask doInBackground","file sent");
chunkFileIS.close();
helperSocketDOS.flush();
} catch (UnknownHostException e) {
Log.d("VideoChunkDistributeTask doInBackground","unknown host");
e.printStackTrace();
return null;
} catch (IOException e) {
Log.d("VideoChunkDistributeTask doInBackground","IO exceptiont");
e.printStackTrace();
return null;
}
The full source code is at: https://github.com/mickod/ColabAndroid/tree/master/src/com/amodtech/colabandroid
You may also find there are more up to date socket libraries available which might be better for you to use, but the general principles should be similar.
If you want to stream your video so that the other app can play it like a regular video it streams from the web, then you would want to set up a web server on the 'sending' device. At this point it might be easier to send it to a server and stream from there instead.
I am using the write() method in order to write in a file of the external storage. This method only accepts byte[] as an input. I have tried passing a String and I get an error message ("The method write(int) in the type FileOutputStream is not applicable for the arguments String"). If I pass an int, I don't get error but in the file nothing is written. The value I get from calling getNumSentPackets() is an int and I need to convert it to byte[]. I have been looking at other questions already answered here and I have tried the ByteBuffer option but the result I get in the file is not what I want, this means, I don't get the number of sent packets. Can anybody help me, please?
This is my code:
public void createFile(String name) {
try {
String filename = name;
File myFile = new File(Environment.getExternalStorageDirectory(), filename);
if (!myFile.exists())
myFile.createNewFile();
String title = "FLOODING RESULTS FILE\n\n";
String sent = "Number of sent packets\n";
FileOutputStream fos;
byte[] data = title.getBytes();
byte[] intSent = sent.getBytes();
int numSent = mSender.getNumSentPackets();
byte[] numSentBytes = ByteBuffer.allocate(10).putInt(numSent).array();
try{
fos = new FileOutputStream(myFile);
fos.write(data);
fos.write(intSent);
fos.write(numSentBytes);
fos.flush();
fos.close();
} catch (Exception e) {
e.printStackTrace();
}
} catch (IOException e) {
e.printStackTrace();
}
}
public static int getNumSentPackets() {
return nSentPackets;
}
The expected output file would be as follows:
FLOODING RESULTS FILE
Number of sent packets 200
200 is only an example, meaning with this that I would like to see there a number which would correspond to the total number of sent packets.
Thank you in advance.
As I am a lazy developer, I like to use the existing facilities in my languages of choice, for example, for java, a PrintWriter.
public void createFile(String filename) {
try {
File myFile = new File(Environment.getExternalStorageDirectory(), filename);
PrintWriter out = new PrintWriter(myFile); // this will create the file if necessary
out.println("FLOODING RESULTS FILE");
out.println();
out.print("Number of sent packets ");
out.println(mSender.getNumSentPackets());
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
This is much easier to read and maintain than your current approach, and looks more idiomatic.
ByteBuffer.allocate(capacity).putInt(yourInt).array();
The text representation of "200" requires you to write 3 characters. All files are just a bunch of bytes in the end so there needs to be a mapping from character to some byte value. Assuming ASCII(*) the data to write into the file would be
// '2','0','0'
byte[] textVersion = { 50, 48, 48 }
int on the other hand is a 32bit numeric value, i.e. has 4 bytes and 200 is equivalent to
byte[] intVersion = { 0, 0, 0, 200 }
When using a ByteBuffer, you'll get this. If you write that into a file and a text viewer tries to display that it would display something like ◻◻◻Č if you're lucky. A 0 is actually a non printable control character and therefore often either skipped when printing or replaced with strange looking character like boxes. The 200 would be equivalent to Č in Windows-CP1250. It has no meaning on it's own when interpreted as UTF8 - it's the start of a 2 byte sequence and so the next 2 byte are required to determine which character to display.
You could have used
String.valueOf(200).getBytes( /* you should specify which encoding to use here */ );
which will create the "200" string first, then return you the required bytes for those 3 characters.
You should however use Java's character based IO facility: The numerous (and confusing) Reader & Writer implementations. They all(*^2) wrap an InputStream or OutputStream in the end and do the text to byte conversion for you.
PrintWriter is probably the most convenient to use but not without flaw: https://stackoverflow.com/a/15803472/995891
FileWriter should be avoided because you can't specify the encoding
The longer alternative route would be
BufferedWriter writer = new BufferedWriter(
new OutputStreamWriter(
new FileOutputStream(file), encoding));
writer.write("Hello ");
writer.write(String.valueOf(200));
writer.newLine();
(*) most encodings are ASCII compatible for the first 127 characters which basically covers normal english text.
(*^2) nothing forces a Writer to output the characters into a stream, e.g. StringWriter. But they are used mostly that way.
I am moving files(around 100-2000 files each of size 100-200KB) via thread from one folder to another. All goes well but on some Samsung & LG devices with sdcard, suddenly after copying, all or some of them go missing.
This does not happen every time, but approximately around once in every 20 times.
I have tried 3 techniques so far:
public void copyMethodA(File src, File dst){
if(!dst.exists()){
src.renameTo(dst);
}
}
copyMethodA(); resulted in loss of file in most of the times.
public void copyMethodB(File sourceFile, File destFile) throws IOException {
if (!destFile.exists()) {
destFile.createNewFile();
}
try {
FileChannel source = new FileInputStream(sourceFile).getChannel();
FileChannel destination = new FileOutputStream(destFile).getChannel();
destination.transferFrom(source, 0, source.size());
} finally {
source.close();
destination.close();
}
}
copyMethodB(); resulted in loss of file comparatively less number of times than A.
public void copyMethodC(File src, File dst) throws IOException {
InputStream in = new FileInputStream(src);
OutputStream out = new FileOutputStream(dst);
byte[] buf = new byte[10240];
int len;
while ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
}
in.close();
out.close();
}
copyMethodC(); very rarely resulted in loss of files. Hence currently using this one.
All 3 methods worked fine without a single loss of file on Xperia C & Nexus 5(both using internal storage)
But loss of files was observed on LG Optimus One(using sdcard) and some Samsung devices(using internal or sdcard)
Info about devices on which I have tested:
Nexus 5 - Android 4.4.2
Xperia C - Android 4.2.2
LG Optimus One - Android 2.3.3
Samsung devices - Android 4.0 and above
(I guess this problem isn't related to version of Android used)
I am avoiding to use huge 3rd party File IO API's as my Android app's size is just 300KB. Using API like Apache commons.io will bloat it to around 2.5MB
Is there any other safe, secure & better way to copy files ?
Thanks.
I think you should something like this :
you should check the return value of renameTo as, as the javadoc states it, there are many reasons for it to fail.
If the renameTo call failed, use the third way, with try / catch blocks to catch IOException when reading / writing the streams and make sure you only delete the source file if the copy was successful. You can then check the exception to understand why the copy failed and possibly retry it later.
I'm doing this (with a 1024 bytes buffer, like in #beni answer) to copy a few hundred files and I've never seen any loss.
I'm using this function and always works. I test this code on Nexus 5. in.read(buf) method returns -1 when is in the end of the file in other case return the number of the bytes readed. So, try this.
public static void copyFile(InputStream in, OutputStream out) throws IOException {
byte[] buffer = new byte[1024];
int read;
while ((read = in.read(buffer)) != -1) {
out.write(buffer, 0, read);
}
}
I have a problem with SHA-1 performance on Android. In C# I get calculated hash in about 3s, same calculation for Android takes about 75s. I think the problem is in reading operation from file, but I'm not sure how to improve performance.
Here's my hash generation method.
private static String getSHA1FromFileContent(String filename)
{
try
{
MessageDigest digest = MessageDigest.getInstance("SHA-1");
//byte[] buffer = new byte[65536]; //created at start.
InputStream fis = new FileInputStream(filename);
int n = 0;
while (n != -1)
{
n = fis.read(buffer);
if (n > 0)
{
digest.update(buffer, 0, n);
}
}
byte[] digestResult = digest.digest();
return asHex(digestResult);
}
catch (Exception e)
{
return null;
}
}
Any ideas how can I improve performance?
I tested it on my SGS (i9000) and it took 0.806s to generate the hash for a 10.1MB file.
Only difference is that in my code i am using BufferedInputStream in addition to the FileInputStream and the hex conversion library found at:
http://apachejava.blogspot.com/2011/02/hexconversions-convert-string-byte-byte.html
Also I would suggest that you close your file input stream in a finally clause
If I were you I would use the JNI like this guy did and get the speed up that way. This is exactly what the C interface was made for.
Well, I've been diving in the murky waters of low-level Android programming (native C/C++ using the CodeSourcery toolchain). I tried out the executable on an emulator and it worked. I'd like to try it out on a real device. So I plugged in my nexus and pushed the files on to the filesystem. Then I tried to execute the binary, and I got a permission error. It really doesn't matter how I mount it, or where I send it, I'm not root and it's not letting me execute it. Is there any way to run a program like this on a non-rooted phone?
Update: notice that apps targeting API level 29 (Android 10) and above will not be able to use the trick below, as the OS will restrict the execute permission. See Behavior changes: apps targeting API 29+.
After using the toolchain included in the Android NDK to compile your binaries, it is possible to package them with a typical Android app and have them spawn as subprocesses.
You'll have to include all the necessary files within the assets folder of your application. In order to run them, you have to have the program copy them from the assets folder to a runnable location like: /data/data/com.yourdomain.yourapp/nativeFolder
You can do this like so:
private static void copyFile(String assetPath, String localPath, Context context) {
try {
InputStream in = context.getAssets().open(assetPath);
FileOutputStream out = new FileOutputStream(localPath);
int read;
byte[] buffer = new byte[4096];
while ((read = in.read(buffer)) > 0) {
out.write(buffer, 0, read);
}
out.close();
in.close();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
Keep in mind that the assetPath is not absolute but in respect to assets/.
IE: "assets/nativeFolder" is just "nativeFolder"
To then run your application and read its output you could do something like this:
Process nativeApp = Runtime.getRuntime().exec("/data/data/com.yourdomain.yourapp/nativeFolder/application");
BufferedReader reader = new BufferedReader(new InputStreamReader(nativeApp.getInputStream()));
int read;
char[] buffer = new char[4096];
StringBuffer output = new StringBuffer();
while ((read = reader.read(buffer)) > 0) {
output.append(buffer, 0, read);
}
reader.close();
// Waits for the command to finish.
nativeApp.waitFor();
String nativeOutput = output.toString();