I'm using the SpringFramework for Android to get my inputstreams.
#Override
public InputStream getImageStream(String url) {
InputStream is = new ByteArrayInputStream(template.getForObject(url, byte[].class));
return is;
}
For the first few inputstreams its going ok. No Problems at all, but then, I think it tries to get a very big inputstream. So then I get the outofmemory error.
I see a lot of posts using something like the following code:
public byte[] readBytes(InputStream inputStream) throws IOException {
ByteArrayOutputStream byteBuffer = new ByteArrayOutputStream();
int bufferSize = 1024;
byte[] buffer = new byte[bufferSize];
int len = 0;
while ((len = inputStream.read(buffer)) != -1) {
byteBuffer.write(buffer, 0, len);
}
byte[] byteArray= byteBuffer.toByteArray();
return byteArray;
}
The idea of this code is to read the inputstream in chunks right?
But the outofmemory error I'm getting is before I can even start the readBytes method. I tried putting resets everywhere...I thought maybe I should clear the memory somewhere after readbytes or something. But I do not know how and I don't know if that is the right way?
I think I'm having the basics wrong? I'm very new to android and java... Is there a way of getting the InputStream another way? I also read something about BufferedInputStream but I just can't think up of a way to fit it in.
My goal is to store a blob of the image in the database. And my input is the imgurl via oauth.
I can also call less quality versions of the inputstream through another url. And then everything works...
But I wanted to try it with the original image url, because maybe later I want to have the ability to download an original for printing the photo.
You are getting OutOfMemory error because you reading whole data in memory when using ByteArrayInputStream. Also notice that when you write in ByteArrayOutputStream it keeps data in memory too (it just simple wrapper for byte array). Probably you should use FileOutputStream instead as cache.
Related
In my application the user can choose a file using the chooser Intent, which will then be "imported" into the application and saved in internal storage for security reasons. This all worked fine and still does on some devices, but for example on the Google Pixel on Android 7.1.1 it only functions normally for the first 4-6 files and afterwards it acts very odd.
The performance was going down drastically so I checked my storage usage and found that it was continuously growing, although the file I was supposed to be saving was less than 1mb large. Importing a file would cause the amount of storage taken by my app to rise past 500mb and upward. I can't seem to find the cause for this.
The method I am using to save the files which is called in an async background task:
BufferedInputStream bis = null;
BufferedOutputStream bos = null;
OutputStream fos = new FileOutputStream(file);
int size = 0;
InputStream fis = getContentResolver().openInputStream(uri);
try{
bis = new BufferedInputStream(fis);
bos = new BufferedOutputStream(fos);
byte[] buf = new byte[1024];
int len = 1024;
while((len = bis.read(buf,0,len)) != -1){
bos.write(buf,0,len);
size = size+1024;
Log.v("Bytes written",""+size);
}
}catch (IOException e){
e.printStackTrace();
}finally {
try{
if(bis != null) bis.close();
if(bos != null) bos.close();
if(fis != null) fis.close();
if(fos != null) fos.close();
}catch(IOException e){
e.printStackTrace();
}
}
return Uri.fromFile(file);
The Uri which this function returns is then saved in an SQLite Database to be used later.
I appreciate all kinds of tips as to where this memory usage could be coming from.
Btw, this did not result from an update on the phone nor from any changes in my code, as it was working the last time I tested it and I haven't changed anything since.
I see a couple of things to correct:
1) The signature of the write method doesn't seem correct, if you write from a buffer you should use write(buff, offset, length).
2) You read into the buffer once, so it should be enugh to write out the buffer once too.
3) If you need to read the the buffer more than once, and write out the values more than once, use a while, not a do while. You have no garantee that the read operation was succesfull.
Ref to write method in Android Developer
I had an additional method which would append an index to a file added multiple times eg. "file.pdf", "file1.pdf","file2.pdf". This method wasn't correct, leading to an endless loop of creating a new file and appending an index. I managed to fix the problem by changing this method to avoid looping.
In retrospect I should have included that in my question.
I need to make API in Json to send the video. I don't want to send the path of video. What is the best way to send the video in JSON which will be used by android and iPhone guys. If I use the base64 or byte[] then I am getting the memory exception error.
File file = new File("video.mp4");
FileInputStream fis = new FileInputStream(file);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
byte[] buf = new byte[1024];
try {
for (int readNum; (readNum = fis.read(buf)) != -1;) {
bos.write(buf, 0, readNum); //no doubt here is 0
System.out.println("read " + readNum + " bytes,");
}
} catch (IOException ex) {
Logger.getLogger(genJpeg.class.getName()).log(Level.SEVERE, null, ex);
}
byte[] bytes = bos.toByteArray();
This is how you add a video byte by byte inside an byte array. You just then send the byte array as JSONOBject by following...
byte[] data; //array holding the video
String base64Encoded = DatatypeConverter.printBase64Binary(data); //You have encoded the array into String
now send that to server. (I am guessing you know how to)..
This is how you will decode your JSON to byteArray again.
byte[] base64Decoded = DatatypeConverter.parseBase64Binary(base64Encoded);
The typical way to send binary in json is to base64 encode it. Java provides different ways to Base64 encode and decode a byte[]. One of these is DatatypeConverter.
I hope it helps.
Cheers!
Edited:
You are getting OutOfMemoryException, because HeapMemory is 2Mb in size and your video is 2Mb, so when inserting into String, it's going out of memory. Even if you put it into an Object instances, you will EITHER have to re-initialize the heap or some way else. I will try to write an answer tomorrow. (Writing this half asleep, might be other way around_
Context
I believe I have searched enough about this topic but haven't found an answer yet. If any of you have any answer please help out.
I am trying to send an image as an output to a service call from an android app.
I have used BufferedImage, ImageIO and a ByteArrayOutputStream to convert the image to a Base64 encoded string (from many of the examples I could find on stackoverflow).
This Base64 encoded string is received by the android app. Now, this needs to be converted back to an image and shown on an activity.
FYI. The image I am trying to send over is a QR code generated using zxing libraries.
Problem
This is where I am struggling. I am not sure how to do it. I haven't posted any sample code, since the server side code is pretty self explanatory and I don't yet have any code on the server side.
Sample code on the Android side
private Bitmap getBitmap() {
String base64String="iVBORw0KGgoAAAANSUhEUgAAAZAAAAGQAQAAAACoxAthAAAGD0lEQVR42u1cTa7iaAw0YpFljpCbkIshESkXg5vkCCyzQHhcVQ7vC6/VmtmNrCA14iWpLCz/lMv+2vw/f+yAHJADckAOyL+ELGZ2drv08Wt8Xs3fdh26x/Pm6ynudv6MB6yf8FUREv/mJW6/bD09z3zD0t37efG7ARcQPOL+LglZrZ89THQGeBp05+ETbEdwPDfGW2HUopAlHpwGOMzQ3ePBuD3DWFeLa4bXlIb4ingJO8WdMVwnPjfE0MvwBvwqC2G8dO/+NUSAhMXeMuBrMIQPr8FhfoVYEQjzwwqb/O3rV4ItAskSiGxJO3XOTIEkcXpuBvxTea0BWVgII2hewxpREq6DdAHrdA9EzgnpwgF+VIR4x5wBWhDpIoIGdTFqR0QJ8gOpAuiDdZ94KQYJmwQk8uEp+BESB3Kk2Yps6fc+zRY+VRGCO8N6CV9ZURfhMIwM8MYzSAOZQyTPn3RRCUJzkC/g4gJjRc4I2zGFrGOwp/g1J7gaZEFWQC6YBliMb8G1sFhYETkjng7G3FKFSpD4SZsE7p4BAv85w39AkgLnCJ82wRaCgCOCEyM2nrwzIWW+kB4n0aVgSu4ttywEcTRI65hlIz5P9EZgxyM7p+yW9VUPwpYAPTJyxmRoFz0gV1P9fMNXUCv93heEuOSAuDOoLYKJPCMn2PHIPIKvprzWgcBh0BCGYZgVmDg82RPeQOlkVytLQVY8zdtKCA8KYogXaEOr+kO7tfFSCJLMEPwIOUOZkX2BeLJUIsRLw5MLQaD+OOsiqwNyBpxj1g2UjSHFQy8IYW+ECjlDCEZdFFlGu4hG2VIp24dYHQjVn851cSJViAbJroMiZ0SmIF1614RQGKBIFnlT3IDiyIcjQjH9UhXKQFgrHSYyJo4fxhwBstXKU7YO9SBoDYMWkC2iOaCHMEm4xiPhRF89cikIOTGlMfXIyBlbcZxEnMJ1zm15LQTxLI6uwhDcIEJFTTF5MuQjUOSl84qQRfIH/IICEQLEOA2jKIgXCtfGSyEIfSXFf7jJhzSwViqN3vuNQpWDdGgD0RlrCG7MkaYe+WoqoheJpRUhnAiyM7ZROvht4V4Ap6LskcESFt8LREUgrpoAlkA6zD8pGGePzDWBfm7LayXIihKBB31bCKGJZrZPg7zmvYuXUhDFyzYNkyLE5sC0EsC3QiVqBKI6EGneYIac/hkpcqTMmygUXojtIBvaNqEWhCMxCIDqC7I54GBIcxOyaPeCEAmj3adYrDkn2tonTgVYMU4VIdLGZY0Ln+FyVPRGeFD5VNODsSIEOfLGnEEdCEGDsqFGWTeknjUZphCEo2CWRGc11GgAw0AyZk8uebXGyJUglD4zIrD34RKIkC0pjmBH5v7FLetAnH8bNdHLRhr8Ey8/PLnJlpUgVISohXEWgPBZtA3F/hCLkZKJ331BCAaikIA4D6Y6oD0wk1y2LcrsFMVCEDe1hhgGcgriuQotnsxdODWOl74gRAswm0IsNQScaZuJI29qD+xdEiJB7CF+xBnJSyMxrgMxmWgbfldfykDoDa7ZIJc+tREhK4IgTOTJQ9e0CXUgnm4i9RutoXMvdFsBg+sgj9hue7ASRNIns4Jvs1BuwTzJj6gT3Fo9uRKEghjJsuaAjBLpwuDJXIr2585hCkFgiWnQ2PcEfoShSJcnPuZFE0G6UzNMKQShOZwRYaqVyJvIkWCLnAlhIc6rQhAqVIM1FHloCVwLYvIVTY4eFSHOhMBjb+iM8yjcqObgI5bmtkw9iLahuBXCzwqv4YCQvVGOjHkWoiRku40deOeQ4Ep9DJKw1iI0FmzmYqUgHIBgCdKa8y8aEFpqxbRdSYhOO3RgwqqLD22/3lzaOCgjZ+LtXKwOJM++QR9LVqRRIV0n90J1NrQmhMsQZEVMj2uORuEmksZuCy32dfCzCiSPvqQ4DBPxJAz34/JIEONltxFRDEJuwBO/NFYe+dLIJOIlD8WVhTA9khlSH8McUDvhOgVi2oitCCE14oMUCreq+Uo38U0a25fXMhBmS3nImOsxg/bj8uwb5mJmX7ujVSDHf/ZyQA7IATkg/w/IP8717dN+SEAgAAAAAElFTkSuQmCC";
ByteArrayInputStream imageArr=new ByteArrayInputStream(Base64.decode(base64String, Base64.DEFAULT));
Bitmap bitmap = BitmapFactory.decodeStream(imageArr);
return bitmap;
}
#haraldK, thanks for giving me the right pointer. I was also wondering why I didn't see any padding, but didn't put enough focus on it to understand the issue. But, based on your hint, I finally figured out the issue. The issue was that I was not closing the OutputStream given by Base64 Encoder properly. In my code I had a the Base64 encoder stream as a wrapper of a ByteArrayOutputStream as below:
final ByteArrayOutputStream os = new ByteArrayOutputStream();
String qString = "";
try {
MatrixToImageWriter.writeToStream(matrix, formatName, Base64.getEncoder().wrap(os));
qString = os.toString("UTF-8");
os.close();
return qString;
} catch (final IOException ioe) {
throw new UncheckedIOException(ioe);
}
In the above code, I was closing the ByteArrayOutputStream using os.close() which was not required. But what was required was closing of the OutputStream given out by the Base64 encoder. I changed the code as below to make it work:
final ByteArrayOutputStream os = new ByteArrayOutputStream();
OutputStream base64Stream = Base64.getEncoder().wrap(os);
String qString = "";
try {
MatrixToImageWriter.writeToStream(matrix, formatName, base64Stream);
base64Stream.close();
qString = os.toString("UTF-8");
os.close();
return qString;
} catch (final IOException ioe) {
throw new UncheckedIOException(ioe);
}
After the above change, the base64 encoded strings are properly padded and are properly getting decoded on the Android side. On the android side this is code I have to display the image:
byte[] imageArr=Base64.decode(base64String.getBytes(), Base64.NO_WRAP);
Bitmap bitmap=BitmapFactory.decodeByteArray(imageArr, 0, imageArr.length);
return bitmap;
I didn't know how to upvote your answer, so I have created this as a separate answer. Thanks for your help.
Can you please add your image Response string
As you are doing it in ByteArrayOutputStream you may missed the below item before decoding
String imageDataBytes = responseImageData.substring(responseImageData.indexOf(",")+1);
InputStream stream = new ByteArrayInputStream(Base64.decode(imageDataBytes.getBytes(), Base64.DEFAULT));
Bitmap bitmap = BitmapFactory.decodeStream(stream);
A good answer available for the same question here
Using Base64 from Apache commons
public byte[] encode(File file) throws FileNotFoundException, IOException {
byte[] encoded;
try (FileInputStream fin = new FileInputStream(file)) {
byte fileContent[] = new byte[(int) file.length()];
fin.read(fileContent);
encoded = Base64.encodeBase64(fileContent);
}
return encoded;
}
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:342)
at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:657)
at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:622)
at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:604)
I'm making small app for mobile device.
You cannot just load the whole file into memory, like here:
byte fileContent[] = new byte[(int) file.length()];
fin.read(fileContent);
Instead load the file chunk by chunk and encode it in parts. Base64 is a simple encoding, it is enough to load 3 bytes and encode them at a time (this will produce 4 bytes after encoding). For performance reasons consider loading multiples of 3 bytes, e.g. 3000 bytes - should be just fine. Also consider buffering input file.
An example:
byte fileContent[] = new byte[3000];
try (FileInputStream fin = new FileInputStream(file)) {
while(fin.read(fileContent) >= 0) {
Base64.encodeBase64(fileContent);
}
}
Note that you cannot simply append results of Base64.encodeBase64() to encoded bbyte array. Actually, it is not loading the file but encoding it to Base64 causing the out-of-memory problem. This is understandable because Base64 version is bigger (and you already have a file occupying a lot of memory).
Consider changing your method to:
public void encode(File file, OutputStream base64OutputStream)
and sending Base64-encoded data directly to the base64OutputStream rather than returning it.
UPDATE: Thanks to #StephenC I developed much easier version:
public void encode(File file, OutputStream base64OutputStream) {
InputStream is = new FileInputStream(file);
OutputStream out = new Base64OutputStream(base64OutputStream)
IOUtils.copy(is, out);
is.close();
out.close();
}
It uses Base64OutputStream that translates input to Base64 on-the-fly and IOUtils class from Apache Commons IO.
Note: you must close the FileInputStream and Base64OutputStream explicitly to print = if required but buffering is handled by IOUtils.copy().
Either the file is too big, or your heap is too small, or you've got a memory leak.
If this only happens with really big files, put something into your code to check the file size and reject files that are unreasonably big.
If this happens with small files, increase your heap size by using the -Xmx command line option when you launch the JVM. (If this is in a web container or some other framework, check the documentation on how to do it.)
If the file recurs, especially with small files, the chances are that you've got a memory leak.
The other point that should be made is that your current approach entails holding two complete copies of the file in memory. You should be able to reduce the memory usage, though you'll typically need a stream-based Base64 encoder to do this. (It depends on which flavor of the base64 encoding you are using ...)
This page describes a stream-based Base64 encoder / decoder library, and includes lnks to some alternatives.
Well, do not do it for the whole file at once.
Base64 works on 3 bytes at a time, so you can read your file in batches of "multiple of 3" bytes, encode them and repeat until you finish the file:
// the base64 encoding - acceptable estimation of encoded size
StringBuilder sb = new StringBuilder(file.length() / 3 * 4);
FileInputStream fin = null;
try {
fin = new FileInputStream("some.file");
// Max size of buffer
int bSize = 3 * 512;
// Buffer
byte[] buf = new byte[bSize];
// Actual size of buffer
int len = 0;
while((len = fin.read(buf)) != -1) {
byte[] encoded = Base64.encodeBase64(buf);
// Although you might want to write the encoded bytes to another
// stream, otherwise you'll run into the same problem again.
sb.append(new String(buf, 0, len));
}
} catch(IOException e) {
if(null != fin) {
fin.close();
}
}
String base64EncodedFile = sb.toString();
You are not reading the whole file, just the first few kb. The read method returns how many bytes were actually read. You should call read in a loop until it returns -1 to be sure that you have read everything.
The file is too big for both it and its base64 encoding to fit in memory. Either
process the file in smaller pieces or
increase the memory available to the JVM with the -Xmx switch, e.g.
java -Xmx1024M YourProgram
This is best code to upload image of more size
bitmap=Bitmap.createScaledBitmap(bitmap, 100, 100, true);
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 100, stream); //compress to which format you want.
byte [] byte_arr = stream.toByteArray();
String image_str = Base64.encodeBytes(byte_arr);
Well, looks like your file is too large to keep the multiple copies necessary for an in-memory Base64 encoding in the available heap memory at the same time. Given that this is for a mobile device, it's probably not possible to increase the heap, so you have two options:
make the file smaller (much smaller)
Do it in a stram-based way so that you're reading from an InputStream one small part of the file at a time, encode it and write it to an OutputStream, without ever keeping the enitre file in memory.
In Manifest in applcation tag write following
android:largeHeap="true"
It worked for me
Java 8 added Base64 methods, so Apache Commons is no longer needed to encode large files.
public static void encodeFileToBase64(String inputFile, String outputFile) {
try (OutputStream out = Base64.getEncoder().wrap(new FileOutputStream(outputFile))) {
Files.copy(Paths.get(inputFile), out);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
I have 5 Mb worth of images that I want to put into an sqlite db, in blob fields.
After the insertion, the db is around 50Mb.
This is how I get the byte[]:
private static byte[] getByteArrayFromFileName(String filename) {
int id = context.getResources().getIdentifier(filename, "raw", context.getPackageName());
ByteArrayOutputStream blob = new ByteArrayOutputStream();
BitmapFactory.decodeResource(context.getResources(), id).compress(CompressFormat.PNG, 0, blob);
return blob.toByteArray();
}
This is how I insert them into the db:
public void createImage(SQLiteDatabase database, Image image) throws DbException {
ContentValues values = new ContentValues();
values.put(ImageTable.DATA, image.data);
values.put(ImageTable.TYPE_ID, image.type.getValue());
values.put(ImageTable.LEVELID, image.levelId);
values.put(ImageTable.ID, image.id);
if (database.replace(ImageTable.TABLE_NAME, null, values) == -1) {
throw new DbException("createImage insertion error");
}
}
What am I screwing up? :)
edit: the problem was, I should not be putting bitmaps into the database, but just the raw (compressed in jpeg format)files. So for reference, here is a correct way of getting a byte[] from a bitmap file, so it's still small in size:
private static byte[] getByteArrayFromRawByFileName(Context context, String filename) throws IOException {
int id = context.getResources().getIdentifier(filename, "raw", context.getPackageName());
InputStream is = context.getResources().openRawResource(id);
ByteArrayOutputStream bos = new ByteArrayOutputStream();
int bytesRead;
byte[] b = new byte[1024];
while ((bytesRead = is.read(b)) != -1) {
bos.write(b, 0, bytesRead);
}
return bos.toByteArray();
}
Probably the problem is you are using JPEG images as source but before insertion you encoding them to PNG and that's the reason of 10x grow I believe. Your getByteArrayFromFileName() is also a fail cuz since you have a file you could read it as byteArray without BitmapFactory involving
Storing images as a blob in your database may not be such a good idea. A couple reasons are...
On Android, your database files are stored in the phone's internal storage, not the SD card. Since the phone's internal storage may be quite limited, it's always a good idea to store large stuff on the SD card. Although, this argument may be invalid on newer devices that do not allow for expanded storage.
If the user decided to uninstall your app, the images also go with it.
I would suggest saving the images to the SD card, and store the file path in the database.