Note:
Please bear with me, this (imho) is not a duplicate of the dozen questions asking about undoing in paint/draw scenarios.
Background:
I've been developing an image processing application, using Processing for Android and now I'm trying to implement a simple, one-step undo/redo functionality.
My initial (undo-friendly) idea was to apply the adjustments to the downsampled preview image only, keep an array of adjustment actions, and apply them at save-time to the original image. I had to sack this idea for two reasons:
some of the actions take a few seconds to finish, and if we have a few of these, it will make the already slow saving process tediously slower.
some actions (e.g. color-noise reduction) produce drastically different (wrong) results when applied to the downsampled image instead of the full-sized image. But anyways this is a less serious problem...
So I decided to go with storing the before/after images.
Problem:
Unfortunately buffering the images in memory is not an option because of memory limitations. So what I'm doing at the moment is saving the before/after images to internal storage.
But that creates a performance/quality dilemma:
jpeg is fast (i.e. ~500ms to save on my Xperia Arc S) but degrades the quality beyond acceptability after two/three iterations.
png is of course lossless, but is super slow (~7000ms to save) which makes it impractical.
bmp I guess would probably be fast, but android does not encode bmp (I think processing for android saves "file.bmp" as tiff).
tiff has somewhat acceptable performance (~1500ms to save), but android does not decode tiff.
I also tried writing the raw pixel array to a file using this function:
void writeData(String filename, int[] data) {
try {
DataOutputStream dos = new DataOutputStream(new BufferedOutputStream(openFileOutput(filename, Context.MODE_PRIVATE)));
for (int i = 0; i < data.length; i++) {
dos.writeInt(data[i]);
}
dos.close();
}
catch (IOException e) {
e.printStackTrace();
}
}
but it takes above 2000ms to finish, so I gave up on it for now.
Questions:
Is there a faster way of writing/reading the data for this purpose?
...or should I go back to the initial idea and try to solve its problems as much as possible?
Any other suggestions?
Update:
I came up with this method to write the raw data:
void saveRAW2(String filename) {
byte[] bytes = new byte[orig.pixels.length*3];
orig.loadPixels(); //orig = my original PImage, duh!
int index = 0;
for (int i = 0; i < bytes.length; i++) {
bytes[i++] = (byte)((orig.pixels[index] >> 16) & 0xff);
bytes[i++] = (byte)((orig.pixels[index] >> 8) & 0xff);
bytes[i] = (byte)((orig.pixels[index]) & 0xff);
index++;
}
saveBytes(filename, bytes);
}
...and it takes less than 1000ms to finish.
It runs 3 times faster than that if I write the file on my SD card, but I guess I can't count on that to be the same on every phone. right?
Anyways, I'm using this method to read the saved data back into orig.pixels:
void loadRAW(String filename) {
byte[] bytes = loadBytes(filename);
int index = 0;
int count = bytes.length/3;
for (int i = 0; i<count; i++) {
orig.pixels[i] =
0xFF000000 |
(bytes[index++] & 0xff) << 16 |
(bytes[index++] & 0xff) << 8 |
(bytes[index++] & 0xff);
}
orig.updatePixels();
}
This takes ~1500ms to finish. Any ideas for optimizing that?
I'd recommend finding out what the Android "scratch disk" area is, and processing your images as tiles, caching them on that scratch disk. This might be a bit slower than straight memory use, but it means you can do your image editing without running into memory limitations, and (provided Android's SDK has a sensible API), writing the tiles to a full file shouldn't take incredibly long. That said, you've kind of moved from "Processing" to plain Java, so the question isn't really about Processing anymore... and my answer is probably not as good as someone who's intimately familiar with the Android SDK
Related
I have implemented a small CNN in RenderScript and want to profile the performance on different hardware. On my Nexus 7 the times make sense, but on the NVIDIA Shield they do not.
The CNN (LeNet) is implemented in 9 layers residing in a queue, computation is performed in sequence. Each layer is timed individually.
Here is an example:
conv1 pool1 conv2 pool2 resh1 ip1 relu1 ip2 softmax
nexus7 11.177 7.813 13.357 8.367 8.097 2.1 0.326 1.557 2.667
shield 13.219 1.024 1.567 1.081 0.988 14.588 13.323 14.318 40.347
The distribution of the times are about right for the nexus, with conv1 and conv2 (convolution layers) taking most of the time. But on the shield, the times drop way beyond what's reasonable for layers 2-4 and seem to gather up towards the end. The softmax layer is a relatively small job, so 40ms is way too large. My timing method must be faulty, or something else is going on.
The code running the layers looks something like this:
double[] times = new double[layers.size()];
int layerindex = 0;
for (Layer a : layers) {
double t = SystemClock.elapsedRealtime();
//long t = System.currentTimeMillis(); // makes no difference
blob = a.forward(blob); // here we call renderscript forEach_(), invoke_() etc
//mRS.finish(); // makes no difference
t = SystemClock.elapsedRealtime() - t;
//t = System.currentTimeMillis() - t; // makes no difference
times[layerindex] += t; // later we take average etc
layerindex++;
}
It is my understanding that once forEach_() returns, the job is supposed to be finished. In any case, mRS.finish() should provide a final barrier. But looking at the times, the only reasonable explanation is that jobs are still processed in the background.
The app is very simple, I just run the test from MainActivity and print to logcat. Android Studio builds the app as a release and runs it on the device which is connected by USB.
(1) What is the correct way to time RenderScript processes?
(2) Is it true that when forEach_() returns, the threads spawned by the script are guaranteed to be done?
(3) In my test app, I simply run directly from the MainActivity. Is this a problem (other than blocking the UI thread and making the app unresponsive)? If this influences the timing or causes the weirdness, what is a proper way to set up a test app like this?
I've implemented CNNs in RenderScript myself, and as you explain, it does require chaining multiple processes and calling forEach_*() various times for each layer if you implement them each as a different kernel. As such, I can assure you that the forEach call returning does not really guarantee that the process has completed. In theory, this will only schedule the kernel and all queued up requests will actually run whenever the system determines it's best to, especially if they get processed in the tablet's GPU.
Usually, the only way to make absolutely sure you have some kind of control over a kernel truly running is by explicitly reading the output of the RS kernel in between layers, such as by using .copyTo() on the output allocation object of that kernel. This "forces" any queued up RS jobs that have not run yet (on which that layer's output allocation is dependent), to execute at that time. Granted, that may introduce data transfer overheads and your timing will not be fully accurate -- in fact, the execution time of the full network will quite surely be lower than the sum of the individual layers if timed in this manner. But as far as I know, it's the only reliable way to time individual kernels in a chain and it will give you some feedback to find out where bottlenecks are, and to better guide your optimization, if that's what you're after.
Maybe a little bit off topic: but for CNN, if you can structure your algorithm using matrix-matrix multiplication as basic computing blocks you can actually use RenderScript IntrinsicBLAS, especially BNNM and SGEMM.
Pros:
High performance implementation of 8bit Matrix Multiplication (BNNM), available in N Preview.
Back support back to Android 2.3 through RenderScript Support lib, when using Build-Tools 24.0.0 rc3 and above.
High performance GPU acceleration of SGEMM on Nexus5X and 6P with N Preview build NPC91K.
If you only use RenderScript Intrinsics, you can code everything in java.
Cons:
Your algorithm may need to be refactored, and need to be based on 2d matrix multiplication.
Though available in Android 6.0, but BNNM performance in 6.0 is not satisfactory. So it is better to use support lib for BNNM and set targetSdkVersion to be 24.
SGEMM GPU acceleration currently only available in Nexus5X and Nexus6P. And it currently requires the width and height of the Matrices to be multiples of 8.
It's worth trying if BLAS fits into your algorithm. And it is easy to use:
import android.support.v8.renderscript.*;
// if you are not using support lib:
// import android.renderscript.*;
private void runBNNM(int m, int n, int k, byte[] a_byte, byte[] b_byte, int c_offset, RenderScript mRS) {
Allocation A, B, C;
Type.Builder builder = new Type.Builder(mRS, Element.U8(mRS));
Type a_type = builder.setX(k).setY(m).create();
Type b_type = builder.setX(k).setY(n).create();
Type c_type = builder.setX(n).setY(m).create();
// If you are reusing the input Allocations, just create and cache them somewhere else.
A = Allocation.createTyped(mRS, a_type);
B = Allocation.createTyped(mRS, b_type);
C = Allocation.createTyped(mRS, c_type);
A.copyFrom(a_byte);
B.copyFrom(b_byte);
ScriptIntrinsicBLAS blas = ScriptIntrinsicBLAS.create(mRS);
// Computes: C = A * B.Transpose
int a_offset = 0;
int b_offset = 0;
int c_offset = 0;
int c_multiplier = 1;
blas.BNNM(A, a_offset, B, b_offset, C, c_offset, c_multiplier);
}
SGEMM is similar:
ScriptIntrinsicBLAS blas = ScriptIntrinsicBLAS.create(mRS);
// Construct the Allocations: A, B, C somewhere and make sure the dimensions match.
// Computes: C = 1.0f * A * B + 0.0f * C
float alpha = 1.0f;
float beta = 0.0f;
blas.SGEMM(ScriptIntrinsicBLAS.NO_TRANSPOSE, ScriptIntrinsicBLAS.NO_TRANSPOSE,
alpha, A, B, beta, C);
the title sounds maybe a bit like a "noob question" but I know quite well how to program for Android, I just to figure out what it is the best way to achieve what I want.
My use case is: the user takes a photo and sends it to our server which have a file-size limit (which could mean that we have to resize the photo directly on the device).
Seems easy, right? My problem are the following:
1) Better use intents which could crash because some camera apps are coded with the ass or build a basic view "take photo and confirm" with cawc camera libs ? (I did the two, I prefer intents but I'd like to have an opinion on that).
2) How do you handle the file size limit? I mean getting the size of the photo is quite easy with the File.length() (even if the returned value is not perfectly right) but if you goes over the limit, how can you say how big will be the resized picture? (you need to convert in bitmap to resize it and it's then a lot of problems with OOMException and you cannot calculate final size of a bitmap on the disk, you need to compress and write it to the disk and analyse the newly created file after).
Thanks for help :D
I did the same thing before.
1.I use intent to call the other camera app, and inside onActivityResult,
I get back the URI and process it as I need.
We do resize the pic, but I also keep the original ratio, and rotate it based on exif data.
Hopefully this resizing code block can give you some hints.
public static Bitmap DecodeImage(String path, int resolution) {
BitmapFactory.Options opts = new BitmapFactory.Options();
opts.inJustDecodeBounds = true;
BitmapFactory.decodeFile(path, opts);
opts.inSampleSize = computeSampleSize(opts, -1, resolution);
opts.inJustDecodeBounds = false;
return BitmapFactory.decodeFile(path, opts);
}
public static int computeSampleSize(BitmapFactory.Options options,
int minSideLength, int maxNumOfPixels) {
int initialSize = computeInitialSampleSize(options, minSideLength,
maxNumOfPixels);
int roundedSize;
if (initialSize <= 8) {
roundedSize = 1;
while (roundedSize < initialSize) {
roundedSize <<= 1;
}
} else {
roundedSize = (initialSize + 7) / 8 * 8;
}
return roundedSize;
}
private static int computeInitialSampleSize(BitmapFactory.Options options, int minSideLength, int maxNumOfPixels) {
double w = options.outWidth;
double h = options.outHeight;
int lowerBound = (maxNumOfPixels == -1) ? 1 :
(int) Math.ceil(Math.sqrt(w * h / maxNumOfPixels));
int upperBound = (minSideLength == -1) ? 128 :
(int) Math.min(Math.floor(w / minSideLength),
Math.floor(h / minSideLength));
if (upperBound < lowerBound) {
// return the larger one when there is no overlapping zone.
return lowerBound;
}
if ((maxNumOfPixels == -1) &&
(minSideLength == -1)) {
return 1;
} else if (minSideLength == -1) {
return lowerBound;
} else {
return upperBound;
}
}
The solution is not fancy but it is what I did it in the project, and we so far have no problems with it after release.
There are lots of question over this topic and hope you searched for it.
Q) Better use intents which could crash because some camera apps are coded with the ass or build a basic view "take photo and confirm" with cawc camera libs ? (I did the two, I prefer intents but I'd like to have an opinion on that).
A) Intents are best because they are build in, some device manufacturers intents for cropping and re-sizing images and I face similar problem of not having that on same manufacturer but on Older device. A stable/relible third party app would suffice for basic operations.
Q) How do you handle the file size limit? I mean getting the size of the photo is quite easy with the File.length() (even if the returned value is not perfectly right) but if you goes over the limit, how can you say how big will be the resized picture? (you need to convert in bitmap to resize it and it's then a lot of problems with OOMException and you cannot calculate final size of a bitmap on the disk, you need to compress and write it to the disk and analyse the newly created file after).
A)
How do you handle the file size limit?
Size limit depends on Camera, if you have 10MP camera then the resultant size would be greater than 5MP (hope you get the part).
You need to convert in bitmap to resize it and it's then a lot of problems with OOMException and you cannot calculate final size of a bitmap on the disk You can calculate the image size or sample it or crop it or resize it as far as you keep best practices of Android intact, recycle the bitmap as soon as you done with the bitmap operations. I have a app which has 100s of Images and most of them being send to server and some times it throws OOM, then I handle it accordingly.
You can gone through the link Loading Large Bitmap
They explained really well regarding bitmap scaling.
Please let me know if you don't get any point in this link.
Thanks
Basically I have 4 ImageView in the UI and I want to set image to them. Following is the related codes:
private void showPhoto() {
// (re)set the image button's image based on our photo
Photo p = mCrime.getPhoto();
BitmapDrawable b = null;
if (p != null) {
String path = getActivity()
.getFileStreamPath(p.getFilename()).getAbsolutePath();
Log.i(TAG,"shown picture name is: "+p.getFilename());
// b = PictureUtils.getScaledDrawable(getActivity(), path);
// Log.i(TAG,"entered！");
if (cachedImageNames.size()==0)
cachedImageNames.add(0,path);
else{
boolean isExisted = false;
for (String imagePath: cachedImageNames){
if (imagePath.equals(path)){
isExisted = true;
break;
//cachedImageNames.add(0,path);
}
}
if (!isExisted)
cachedImageNames.add(0,path);
}
}
mCrimes = CrimeLab.get(getActivity()).getCrimes();
// mPhotoView.setImageDrawable(b);
Log.i(TAG,"image names' list size is: "+cachedImageNames.size());
if(!(cachedImageNames.size()<imageViews.size())){
for (int i=0; i<imageViews.size(); i++){
b= PictureUtils.getScaledDrawable(getActivity(), cachedImageNames.get(i));
imageViews.get(i).setImageDrawable(b);
}
}
else{
for (int i=0; i<cachedImageNames.size(); i++){
b= PictureUtils.getScaledDrawable(getActivity(), cachedImageNames.get(i));
imageViews.get(i).setImageDrawable(b);
}
}
}
In the console, it is indicated that this line:
b= PictureUtils.getScaledDrawable(getActivity(),cachedImageNames.get(i));
results in such an error:
java.lang.OutOfMemoryError: Failed to allocate a 51916812 byte allocation with 16777216 free bytes and 36MB until OOM
I am really new to android... Any suggestion?
You are trying to allocate 50MB of ram on heap which is an too much for android-powered device. If this is really really necessary you can add:
android:largeHeap="true"
in the <application> part of your manifest. But this is typically a not recommended solution. Most apps should not need this and should instead focus on reducing their overall memory usage for improved performance. Enabling this also does not guarantee a fixed increase in available memory, because some devices are constrained by their total available memory.
Proper solution:
You need to look into how many images you are opening and how big they are. Note that a very common misconception is that people usually assume that the file size is what is being used in the memory. This is a very wrong assumption. PNGs, JPGs and etc are compressed format. When you load an image on your device each pixel is 4 bytes. If you have a 2000x2000 pixel image, when loaded, it consumes 16MB of RAM - which is typically a high limit for typical (non-game) apps.
If you don't need to load the full size image, you may want to look into options.inJustDecodeBounds technique.
If you are not using all images in your shown screen, you want to use caching techniques.
I'm developing a mobile application that runs on Android and IOS. It's capable of real-time-processing of a video stream. On Android I get the Preview-Videostream of the camera via android.hardware.Camera.PreviewCallback.onPreviewFrame. I decided to use the NV21-Format, since it should be supported by all Android-devices, whereas RGB isn't (or just RGB565).
For my algorithms, which mostly are for pattern recognition, I need grayscale images as well as color information. Grayscale is not a problem, but the color conversion from NV21 to BGR takes way too long.
As described, I use the following method to capture the images;
In the App, I override the onPreviewFrame-Handler of the Camera. This is done in CameraPreviewFrameHandler.java:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
{
try {
AvCore.getInstance().onFrame(data, _prevWidth, _prevHeight, AvStreamEncoding.NV21);
} catch (NativeException e)
{
e.printStackTrace();
}
}
The onFrame-Function then calls a native function which fetches data from the Java-Objects as local references. This is then converted to an unsigned char* bytestream and calls the following c++ function, which uses OpenCV to convert from NV21 to BGR:
void CoreManager::api_onFrame(unsigned char* rImageData, avStreamEncoding_t vImageFormat, int vWidth, int vHeight)
{
// rImageData is a local JNI-reference to the java-byte-array "data" from onPreviewFrame
Mat bgrMat; // Holds the converted image
Mat origImg; // Holds the original image (OpenCV-Wrapping around rImageData)
double ts; // for profiling
switch(vImageFormat)
{
// other formats
case NV21:
origImg = Mat(vHeight + vHeight/2, vWidth, CV_8UC1, rImageData); // fast, only creates header around rImageData
bgrMat = Mat(vHeight, vWidth, CV_8UC3); // Prepare Mat for target image
ts = avUtils::gettime(); // PROFILING START
cvtColor(origImg, bgrMat, CV_YUV2BGRA_NV21);
_onFrameBGRConversion.push_back(avUtils::gettime()-ts); // PROFILING END
break;
}
[...APPLICATION LOGIC...]
}
As one might conclude from comments in the code, I profiled the conversion already and it turned out that it takes ~30ms on my Nexus 4, which is unacceptable long for such a "trivial" pre-processing step. (My profiling methods are double-checked and working properly for real-time measurement)
Now I'm trying desperately to find a faster implementation of this color conversion from NV21 to BGR. This is what I've already done;
Adopted the code "convertYUV420_NV21toRGB8888" to C++ provided in this topic (multiple of the conversion-time)
Modified the code from 1 to use only integer operations (doubled conversion-time of openCV-Solution)
Browsed through a couple other implementations, all with similar conversion-times
Checked OpenCV-Implementation, they use a lot of bit-shifting to get performance. Guess I'm not able to do better on my own
Do you have suggestions / know good implementations or even have a completely different way to work around this Problem? I somehow need to capture RGB/BGR-Frames from the Android-Camera and it should work on as many Android-devices as possible.
Thanks for your replies!
Did you try libyuv? I used it in the past and if you compile it with NEON support, it uses an asm code optimized for ARM processors, you can start from there to further optimize for your special situation.
In order to minimize the memory usage of bitmaps, yet still try to maximize the quality of them, I would like to ask a simple question:
Is there a way for me to check if a given image file (.png file) has transparency using the API, without checking every pixel in it?
If the image doesn't have any transparency, it would be the best to use a different bitmap format that uses only the RGB values.
The problem is that Android also doesn't have a format for just the 3 colors. Only RGB_565, which they say that degrade the quality of the image and that should have dithering feature enabled.
Is there also a way to read only the RGB values and be able to show them?
For me bitmap.hasAlpha() works fine to check first if the bitmap has alpha values. Afterwards you have to run through the pixels and create a second bitmap with no alpha I would suggest.
Let's start a bit off-topic
the problem is that android also doesn't have a format for just the 3 colors . only RGB_565 , which they say that degrade the quality of the image and that should have dithering feature enabled.
The reason for that problem is not really Android specific. It's about performance while drawing images. You get the best performance if the pixeldata fits exactly in 1 32bit memory cell.
So the most obvious good pixel format is the ARGB_8888 format which uses exactly 32bit (24 for the color 8 for alpha). While drawing you don't need to do anything but to loop over the image data and each cell you read can be drawn directly. The only downside is the required memory to work with such images, both when they just sit in memory and while displaying them since the graphic hardware has to transfer more data.
The second best option is to use a format where several pixels fit into 1 cell. Using 2 pixels in 32bit you have 16bit per pixel left and one of the formats using 16bit is the 565 format. 5bit red, 6bit green, 5bit blue. While drawing this you can still work on memory cells separately and all you have to do is to split 1 cell in parts. Due to the smaller memory size required for images, drawing can sometimes be even faster than using 32bit colors. Since in the beginning of android memory was a much bigger problem they chose this format to be the default.
And the worst category of formats are those where pixels don't fit into those cells. If you take just the 3 colors you get 24 bit and those need to be distributed over 2 cells in 3 out of 4 cases. For example the second pixel would use the remaining 8 bit from the first cell & the first 16bit of the next cell. The extra work required to work with 24bit colors is so big that it is not used. And when drawing images you usually have alpha at some point anyways and if not you simply use 32bit but ignore the alpha bits.
So the 16bit approach looks ugly & the 24 bit approach does not make sense. And since the memory limitations of Android are not as tight as they were and the hardware got faster, Android has switched it's default to 32bit (explained in even more details in http://www.curious-creature.org/2010/12/08/bitmap-quality-banding-and-dithering/)
Back to your real question
is there a way for me to check if a given image file (png file) has transparency using the API , without checking every pixel in it?
I don't know. But JPEG images don't support alpha and PNG images usually have alpha. You could simply abuse the file extension to get it right in most cases.
But I would suggest you don't bother with all that and simply use ARGB_8888 and apply the nice image loading techniques detailed in the Android Training documentation about Displaying Bitmaps Efficiently.
The reason people run into memory problems is usually either that they have way more images loaded in memory than they currently display or they use giant images that can't be displayed on the small screen of a phone. And in my opinion it makes more sense to add good memory management than complicating your code to downgrade the image quality.
There is a way to check if a PNG file has transparency, or at least if it supports it:
public final static int COLOR_GREY = 0;
public final static int COLOR_TRUE = 2;
public final static int COLOR_INDEX = 3;
public final static int COLOR_GREY_ALPHA = 4;
public final static int COLOR_TRUE_ALPHA = 6;
private final static int DECODE_BUFFER_SIZE = 16 * 1024;
private final static int HEADER_DECODE_BUFFER_SIZE = 1024;
/** given an inputStream of a png file , returns true iff found that it has transparency (in its header) */
private static boolean isPngInputStreamContainTransparency(final InputStream pngInputStream) {
try {
// skip: png signature,header chunk declaration,width,height,bitDepth :
pngInputStream.skip(12 + 4 + 4 + 4 + 1);
final byte colorType = (byte) pngInputStream.read();
switch (colorType) {
case COLOR_GREY_ALPHA:
case COLOR_TRUE_ALPHA:
return true;
case COLOR_INDEX:
case COLOR_GREY:
case COLOR_TRUE:
return false;
}
return true;
} catch (final Exception e) {
}
return false;
}
Other than that, I don't know if such a thing is possible.
i've found the next links which could be helpful for checking if the png file has transparency . sadly, it's a solution only for png files . rest of the files (like webP , bmp, ...) need to have a different parser .
links:
http://www.java2s.com/Code/Java/2D-Graphics-GUI/PNGDecoder.htm
http://hg.l33tlabs.org/twl/file/tip/src/de/matthiasmann/twl/utils/PNGDecoder.java
http://www.java-gaming.org/index.php/topic,24202