I have a problem related to Memory Leak. In my app I have to read 2MB data from a Video file, and the method related to same always called when the Activity's onCreate method called then the same statement which allocated 2MB byte array in the code, returns OutofMemory Exception frequently after 10 to 15 attempts because heap memory exceeds. The code is explained below (it is the part of my whole code):
//Reading DRM video from sdcard
File file = new File("/sdcard/TvAnyTime/watch/"+IDValue+".mp4");
try {
is = new FileInputStream(file);
} catch (FileNotFoundException e2) {
e2.printStackTrace();
}
//reading 2^21 bytes
fileData = new byte[2097152];
int read = 0;
while(read != fileData.length) {
try {
read += is.read(fileData, read, fileData.length - read);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
//The finalHashPattern is then xored across the video file 2^10 times
for(int i=0;i<2097152;i+=2048)
{
byte[] res = new byte[2048];
bytesafterXor = new byte[2048];
for(int j=0;j<2048;j++)
{
res[j] = fileData[i+j];
bytesafterXor[j] = (byte)(res[j]^finalhash[j]);
finaldatafile[i+j] = bytesafterXor[j];
}
finalHashafterXor.add(bytesafterXor);
}
The statement fileData = new byte[2097152]; is responsible for the OutOfMemory Exception because it is allocated every time when onCreate is called. Can we prevent the same by allocating each time a large memory? can we read it in chunks of data? Please suggest me the right solution regarding the same.
Thanks in advance.
Have you considered processing the input inside the loop where your reading the bytes in, rather than reading all of the bytes
fileData = new byte[2048];
int read = 0;
while(read != fileData.length) {
try {
read += is.read(fileData, read, fileData.length);
for(int i = 0; i < 2048; i++) {
// Processing here
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
You may put fileData as a static pointer an then in case onCreate alloc memory only once, when fileData==NULL;
Ok, since my edit to Kingamajick's answer "mysteriously" disappeared while waiting for peer review, here is how to do file (or streams in general) chunked reading right:
fileData = new byte[2048];
try {
int c = 0;
while( (c = is.read(fileData)) >= 0 ) {
// Process all bytes from fileData[0] through fileData[c-1], e.g.:
for(int i = 0; i < c; i++) {
// do something with fileData[i]
// ...
}
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Note that for files, the buffer (fileData above) will most likely always be completely filled by the read operation until there are not enough bytes left to read in the file. When reading from, for instance, network streams this is usually not the case as data is not always instantly available and when it becomes available it is likely just as much data as arrived in the last network packet. However, the above approach works for all streams.
Edit:
From your comment I take that you do not want to process the whole file but only the first 2mb. In this case you can modify the above approach a bit, like:
fileData = new byte[2048];
int leftToRead = 2048*1024; // Total amount of bytes you want to read.
try {
int c = 0;
// How many bytes may we read at once?
int maxRead = Math.min( fileData.length, leftToRead );
while( (leftToRead > 0) && (c = is.read(fileData, 0, maxRead)) >= 0 ) {
// Process all bytes from fileData[0] through fileData[c-1], e.g.:
for(int i = 0; i < c; i++) {
// do something with fileData[i]
// ...
}
// We read c bytes, so we may have some bytes left:
leftToRead -= c;
// How many bytes may we read at once?
maxRead = Math.min( fileData.length, leftToRead );
}
// Optionally:
if ( leftToRead > 0 ) {
System.out.println("Premature end of file.");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
When processing the file byte-by-byte, as you do in your code excerpt, you can basically freely choose the size of the fileData buffer. A smaller buffer does not provide any real benefit, because it may cause more read operations on the underlying file system. Something in the range from 1kb to 64kb is usually a good size.
You can exclude this critical file processing from Java code to Native code.
http://marakana.com/forums/android/examples/49.html
http://android.wooyd.org/JNIExample/files/JNIExample.pdf
Related
So, I used this Android sample as a guide to make a bluetooth connection without any kind of validation (This app will have a very restrict userbase and will not be available to download at the store).
I was able to transfer string just fine, and it works like a charm. My problem is when trying to transfer images.
I have one activity that sends the byte[] of the image to the bluetooth service and a handler on the other activity that recieves the message and do wharever with the said message.
The thing is, because of the size of the buffer the handler receives parts of the original byte[]. What I'm tryng to do is to merge all the parts in one byte and save it.
This is the loop that I do inside my handler:
byte[] result = new byte[originalByteSize];
byte[] readBuf = (byte[]) msg.obj;
if (cont < byteTimes){
if (result == null) {
result = appendData(readBuf,readBuf);
} else {
result = appendData(result,readBuf);
}
} else {
new SavePhotoTask(cont).execute(result);
}
This is the appendData function
protected byte[] appendData(byte[] firstObject,byte[] secondObject){
ByteArrayOutputStream outputStream = new ByteArrayOutputStream( );
try {
if (firstObject!=null && firstObject.length!=0)
outputStream.write(firstObject);
if (secondObject!=null && secondObject.length!=0)
outputStream.write(secondObject);
} catch (IOException e) {
e.printStackTrace();
}
return outputStream.toByteArray();
}
And here is where I write the file:
public class SavePhotoTask extends AsyncTask<byte[], String, String> {
int counter = 0;
public SavePhotoTask(int cont){
this.counter = cont;
}
#Override
protected String doInBackground(byte[]... jpeg) {
File photo = new File(Environment.getExternalStorageDirectory(), counter + "_photo.jpg");
if (photo.exists()) {
photo.delete();
}
try {
FileOutputStream fos = new FileOutputStream(photo.getPath());
fos.write(jpeg[0]);
fos.close();
} catch (java.io.IOException e) {
Log.e("PictureDemo", "Exception in photoCallback", e);
}
return (null);
}
What I needed is just a tip in the right direction, thanks.
I solved my problem with this answer
The problem was in the way I was writing and reading the stream.
public static void writeItem(OutputStream out, String s) throws IOException
{
// Get the array of bytes for the string item:
byte[] bs = s.getBytes(); // as bytes
// Encapsulate by sending first the total length on 4 bytes :
// - bits 7..0 of length
out.write(bs.length); // modulo 256 done by write method
// - bits 15..8 of length
out.write(bs.length>>>8); // modulo 256 done by write method
// - bits 23..16 of length
out.write(bs.length>>>16); // modulo 256 done by write method
// - bits 31..24 of length
out.write(bs.length>>>24); // modulo 256 done by write method
// Write the array content now:
out.write(bs); // Send the bytes
out.flush();
}
public static String readItem(InputStream in) throws IOException
{
// first, read the total length on 4 bytes
// - if first byte is missing, end of stream reached
int len = in.read(); // 1 byte
if (len<0) throw new IOException("end of stream");
// - the other 3 bytes of length are mandatory
for(int i=1;i<4;i++) // need 3 more bytes:
{
int n = in.read();
if (n<0) throw new IOException("partial data");
len |= n << (i<<3); // shift by 8,16,24
}
// Create the array to receive len bytes:
byte[] bs = new byte[len];
// Read the len bytes into the created array
int ofs = 0;
while (len>0) // while there is some byte to read
{
int n = in.read(bs, ofs, len); // number of bytes actually read
if (n<0) throw new IOException("partial data");
ofs += n; // update offset
len -= n; // update remaining number of bytes to read
}
// Transform bytes into String item:
return new String(bs);
}
I have an app for Android which downloads hundreds of files from the Internet. Some files turn out to be 0-byte after download. The app attempts to detect such cases and delete such files after download but sometimes it fails. The problem is more frequent on Android 4.x devices.
Here is the method which does the downloading. I gets the number of actually read bytes from inputStream.read(buffer).
public class Utils
{
public static class DownloadFileData
{
int nTotalSize;
int nDownloadedSize;
}
public interface ProgressCallback
{
void onProgress(long nCurrent, long nMax);
}
public static boolean downloadFile(String sFileURL, File whereToSave, DownloadFileData fileData, ProgressCallback progressCallback)
{
InputStream inputStream = null;
FileOutputStream fileOutput = null;
try
{
URL url = new URL(sFileURL);
URLConnection connection = url.openConnection();
//set up some things on the connection
connection.setDoOutput(true);
connection.connect();
fileOutput = new FileOutputStream(whereToSave);
inputStream = connection.getInputStream();
fileData.nTotalSize = connection.getContentLength();
fileData.nDownloadedSize = 0;
byte[] buffer = new byte[1024];
int bufferLength = 0; //used to store a temporary size of the buffer
// now, read through the input buffer and write the contents to the file
while ((bufferLength = inputStream.read(buffer)) > 0)
{
// if interrupted, don't download the file further and return
// also restore the interrupted flag so that the caller stopped also
if (Thread.interrupted())
{
Thread.currentThread().interrupt();
return false;
}
// add the data in the buffer to the file in the file output stream
fileOutput.write(buffer, 0, bufferLength);
// add up the size so we know how much is downloaded
fileData.nDownloadedSize += bufferLength;
if (null != progressCallback && fileData.nTotalSize > 0)
{
progressCallback.onProgress(fileData.nDownloadedSize, fileData.nTotalSize);
}
}
return true;
}
catch (FileNotFoundException e)
{
return false; // swallow a 404
}
catch (IOException e)
{
return false; // swallow a 404
}
catch (Throwable e)
{
return false;
}
finally
{
// in any case close input and output streams
if (null != inputStream)
{
try
{
inputStream.close();
inputStream = null;
}
catch (Exception e)
{
}
}
if (null != fileOutput)
{
try
{
fileOutput.close();
fileOutput = null;
}
catch (Exception e)
{
}
}
}
}
Here is the piece of code which processes the downloads. Since sometimes the number of read bytes is incorrect (it is > 0 and the real file has the size 0 bytes) I check the size of the downloaded file with outputFile.length(). But this again gives a value > 0 even though the file is really 0 byte. I tried to also just create a new file and read its size with recheckSizeFile.length(). Still the size is determined as > 0 while it's really 0 byte.
Utils.DownloadFileData fileData = new Utils.DownloadFileData();
boolean bDownloadedSuccessully = Utils.downloadFile(app.sCurrenltyDownloadedFile, outputFile, fileData, new Utils.ProgressCallback()
{
... // progress bar is updated here
});
if (bDownloadedSuccessully)
{
boolean bIsGarbage = false;
File recheckSizeFile = new File(sFullPath);
long nDownloadedFileSize = Math.min(recheckSizeFile.length(), Math.min(outputFile.length(), fileData.nDownloadedSize));
// if the file is 0bytes, it's garbage
if (0 == nDownloadedFileSize)
{
bIsGarbage = true;
}
// if this is a video and if of suspiciously small size, it's
// garbage, too
else if (Utils.isStringEndingWith(app.sCurrenltyDownloadedFile, App.VIDEO_FILE_EXTENSIONS) && nDownloadedFileSize < Constants.MIN_NON_GARBAGE_VIDEO_FILE_SIZE)
{
bIsGarbage = true;
}
if (bIsGarbage)
{
++app.nFilesGarbage;
app.updateLastMessageInDownloadLog("File is fake, deleting: " + app.sCurrenltyDownloadedFile);
// delete the garbage file
if (null != outputFile)
{
if (!outputFile.delete())
{
Log.e("MyService", "Failed to delete garbage file " + app.sCurrenltyDownloadedFile);
}
}
}
else
{
... // process the normally downloaded file
}
I am not sure but I think there is a bug in Android with reading file size. Has anyone seen a similar problem? Or am I maybe doing something wrong here?
Thanks!
EDIT: how i determine that the files are 0-byte:
all the files which get downloaded go thru the described routines. When I then later view the download folder with a file browser (Ghost Commander), some of the files (like maybe 10%) are 0-byte. They can't be played by a video player (shown as "broken file" icon).
It looks to me like your problem is that you only check for "garbage" files if the Utils.downloadFile call returns true. If the download fails in the getInputStream call or the first read, you will have created a file with zero length which will never be deleted.
You should call flush() on your FileOutputStream to ensure that all data is written to the file. This should make your issue with 0-byte files occur less often.
To check for 0 byte files using File.length() should work properly. Can you open a shell (adb shell) on the device and run ls -l to see the byte count displayed by it is 0 (maybe your file manager has some weird issues). Also please debug (or put some log statements) that sFullPath contains the correct file paths. I can't see where sFullPath gets set in your code above and why you don't just use outputFile but recreate another File object.
I'm writing an app which requires that images be downloaded from a Google Drive. I am currently doing this using the following code:
protected void downloadFromDrive(Context context) {
InputStream input = null;
FileOutputStream output = null;
try {
HttpRequest request = GoogleDriveWorker.get(context)
.getDrive()
.getRequestFactory()
.buildGetRequest(new GenericUrl(getUri()));
input = request.execute().getContent();
output = context.openFileOutput(getImageFilename(), Context.MODE_PRIVATE);
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize];
int len = 0;
while ((len = input.read(buffer)) != -1) {
output.write(buffer, 0, len);
}
} catch (UnrecoverableKeyException e) {
e.printStackTrace();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (CertificateException e) {
e.printStackTrace();
} catch (KeyStoreException e) {
e.printStackTrace();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if(output!=null)
output.close();
if(input!=null)
input.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
public String getUri() {
return mUri;
}
GoogleDriveWorker is just a class that gets a google drive with the credentials we're using. Anyway, most of the examples I can find use this basic structure to download a file from an InputStream and put it to an OutputStream, but the download rate is rather slow.
Firstly, can I speed it up by using a more sophisticated method than synchronously buffering the InputStream to the OutputStream a kilobyte at a time? It strikes me that I should try to read the InputStream on a different thread, and output to the OutputStream as kilobyte chunks become available using a queue of chunks. Tying the read code and the write code together seems clunky, and they will surely slow each other down.
Secondly, would changing the buffer size affect the data rate at all? A kilobyte seems small, but on a mobile connection maybe it's not that small. Then again the larger the chunk, the larger the wait from each section of the read/write loop. Is using a different-sized buffer worth considering?
I don't think there's a more sophisticated method than what you did.
You could probably make some experiments with larger chunks (for example few hundred KB) and measure the tiem. I think it was faster.
Also check the drive/java-api-client-library documentation about the chunk size. I think there was some explanation about it, but I'm not 100% sure about that.
In my android app, I am reading a file from internal storage every time a new game loads.
The first 4 times I do this, it works fine, but on the fifth time it force closes.
Here is my code
private String readFromInternalStorage(String filename) {
FileInputStream fis=null;
byte[] bytes = new byte[1000000];
try {
fis=startGame.openFileInput(filename);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
try {
fis.read(bytes);
} catch (IOException e) {
e.printStackTrace();
}
return new String(bytes);
}
While messing around with the code, I noticed that if I change the length of the byte array, it changes the amount of times I can read a file without it force closing. If I change the length to 2000000, it closes after the second time and if I change it to 100000 it closes after the eighth time. I'm pretty clueless as to why this would happen because I am creating a new byte array every time the method is called so I wouldn't think that the size would change anything.
Update:
After going back and doing some more testing it seems like file input has nothing to do with why my app is force closing. When this code is commented out, the app will load five levels in a row without force closing so I thought that it was the problem, but it still force closes after eight tries so clearly there's something else that's not working. Thanks for your help anyway.
I don't see a "close()" in your code:
http://docs.oracle.com/javase/6/docs/api/java/io/FileInputStream.html#close%28%29
You shouldn't hard-code the array size. Besides you should use finally, in order to make sure the FileInputStream is closed, even when failed.
Here's a code sample that shows how it should be done:
FileInputStream fis;
String info = "";
try {
fis = mContext.openFileInput(this.fileName);
byte[] dataArray = new byte[fis.available()];
if (dataArray.length > 0) {
while (fis.read(dataArray) != -1) {
info = new String(dataArray);
}
Log.i("File Reading" , "Success!");
isOk = true;
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e1) {
e1.printStackTrace();
} finally {
fis.close();
}
a safe version of what you do is e.g.:
private String readFromInternalStorage(String filename) {
FileInputStream fis = null;
File file = new File(startGame.getFilesDir(), filename);
long size = file.length();
// impossible to have more than that (= 2GB)
if (size > Integer.MAX_VALUE) {
Log.d("XXX", "File too big");
return null;
}
int iSize = (int) size;
try {
fis = new FileInputStream(file);
// part of Android since API level 1 - buffer can scale
ByteArrayBuffer bb = new ByteArrayBuffer(iSize);
// some rather small fixed buffer for actual reading
byte[] buffer = new byte[1024];
int read;
while ((read = fis.read(buffer)) != -1) {
// just append data as long as we can read more
bb.append(buffer, 0, read);
}
// return a new string based on the large buffer
return new String(bb.buffer(), 0, bb.length());
} catch (FileNotFoundException e) {
Log.w("XXX", e);
} catch (IOException e) {
Log.w("XXX", e);
} catch (OutOfMemoryError e) {
// this could be left out. Keep if you read several MB large files.
Log.w("XXX", e);
} finally {
// finally is executed even if you return in above code
// fis will be null if new FileInputStream(file) throws
if (fis != null) {
try {
fis.close();
} catch (IOException e) {
// ignored, nothing can be done if closing fails
}
}
}
return null;
}
I have implemented a bluetooth connection using the now-classic Google Bluetooth Chat code. However, I have a question which I just cannot seem to wrap my brain around.
The reading of the input stream goes something like this:
public void run() {
byte[] buffer = new byte[1024]; // buffer store for the stream
int bytes; // bytes returned from read()
// Keep listening to the InputStream until an exception occurs
while (true) {
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
// Send the obtained bytes to the UI Activity
mHandler.obtainMessage(MESSAGE_READ, bytes, -1, buffer)
.sendToTarget();
} catch (IOException e) {
break;
}
}
}
Now, that's fine if I was just printing out the characters I was receiving as in the original example. However, suppose I wanted to transfer an image file. I don't know the size of the file, so I cannot count the bytes received or anything like that. In my tests, I don't seem to be ever receiving a "-1" from the input stream, which appears to be the "norm" for reading from input streams. So how can I know that I have reached the end of the file that was being sent?
Thank you for your help and your time.
It seems Android bluetooth input streams never return -1.
I guess setup a simple protocol by sending file size in the first place and EOF signals at last will help.
No it does not. Android sends -1 only when the Socket is closed as far as I know. So a workaround could be to do a reconnect, but I was trying that for hours and did not get it working, since I do not understand this "special" Code here (copied from a Stackoverflow Thread) for setting up the socket:
BluetoothSocket tmp = null;
Log.d(TAG, "New Connection initialized");
Method m;
try {
m = device.getClass().getMethod("createRfcommSocket",
new Class[] { int.class });
tmp = (BluetoothSocket) m.invoke(device, 1);
} catch (Exception e) {
e.printStackTrace();
}
mmSocket = tmp;
This Socket only works, when my App is started for the first filetransfer. If I want to "Reconnect" with a completely new instantiated Object (and a new Socket created with that Code), the program freezes on the blocking method mmSocket.connect(). It seems like the Method never comes to an ending. This is driving me nuts...
Try
while ((bytes = mmInStream.read(buffer) != -1)
and see if that helps.
Try this:
public void run() {
byte[] buffer;
ArrayList<Integer> arr_byte = new ArrayList<Integer>();
while (true) {
try {
int data = mmInStream.read();
if(mmInStream.available()>0) {
arr_byte.add(data);
} else {
arr_byte.add(data);
buffer = new byte[arr_byte.size()];
for(int i = 0 ; i < arr_byte.size() ; i++) {
buffer[i] = arr_byte.get(i).byteValue();
}
Log.e("INPUT",new String(buffer));
mHandler.obtainMessage(MESSAGE_READ, bytes, -1, buffer)
.sendToTarget();
arr_byte = new ArrayList<Integer>();
}
} catch (IOException e) {
break;
}
}
}