Bitmap to big to use getPixels?

Bitmap to big to use getPixels? - android

If I make a picture with mine samsung galaxy s2, the picture is 3264 x 2448 px.
I want to use a color range check on that, but it doesn't work.
However, if I make the picture smaller for example, 2500 x 2500 (so less pixels), then it does work. But I want the use the picture size of the galaxy s2 (3264 x 2448).
I think it is a memory issue?
I don't exactly know anymore what the limit is.
But is their another way to "bypass" this issue?
This is a piece of code, how I do it now:
bmp = BitmapFactory.decodeResource(getResources(),
R.drawable.four_colors);
int width = bmp.getWidth();
int height = bmp.getHeight();
int[] pixels = new int[width * height];
bmp.getPixels(pixels, 0, width, 0, 0, width, height);
for (int y = 0; y < height; y++){
for (int x = 0; x < width; x++){
int index = y * width + x;
int R = (pixels[index] >> 16) & 0xff; //bitwise shifting
int G = (pixels[index] >> 8) & 0xff;
int B = pixels[index] & 0xff;
total++;
if ((G > R)&&(G > B)){
counter++;
}
}
}
It crashes because the picture is to big, smaller pics work.
So is their something to "bypass" this issue? instead of using smaller images :)
I tried some other things, without succes, I try to explain what I tried.
I tried to "cut" the image in two, and then scan it separate (didn't work).
and I tried to only scan the half of it (1632 x 1224) then rotate the image (180 degrees) and scan it again, but also this didn't work out.
Hit me :)

When playing around with huge images you really should be using the BitmapRegionDecoder to process it in chunks.
Edit - now with a simple example:
try {
// Processes file in res/raw/huge.jpg or png
BitmapRegionDecoder decoder = BitmapRegionDecoder.newInstance(getResources().openRawResource(R.raw.huge), false);
try {
final int width = decoder.getWidth();
final int height = decoder.getHeight();
// Divide the bitmap into 1024x768 sized chunks and process it.
int wSteps = (int) Math.ceil(width / 1024.0);
int hSteps = (int) Math.ceil(height / 768.0);
Rect rect = new Rect();
long total = 0L, counter = 0L;
for (int h = 0; h < hSteps; h++) {
for (int w = 0; w < wSteps; w++) {
int w2 = Math.min(width, (w + 1) * 1024);
int h2 = Math.min(height, (h + 1) * 768);
rect.set(w * 1024, h * 768, w2, h2);
Bitmap bitmap = decoder.decodeRegion(rect, null);
try {
int bWidth = bitmap.getWidth();
int bHeight = bitmap.getHeight();
int[] pixels = new int[bWidth * bHeight];
bitmap.getPixels(pixels, 0, bWidth, 0, 0, bWidth, bHeight);
for (int y = 0; y < bHeight; y++){
for (int x = 0; x < bWidth; x++){
int index = y * bWidth + x;
int R = (pixels[index] >> 16) & 0xff; //bitwise shifting
int G = (pixels[index] >> 8) & 0xff;
int B = pixels[index] & 0xff;
total++;
if ((G > R)&&(G > B)){
counter++;
}
}
}
} finally {
bitmap.recycle();
}
}
}
} finally {
decoder.recycle();
}

You can limit the amount of pixel data of the image that you are loading at once by using some of the parameters of the getPixels method. I did something very similar, and I would read them one row at a time, for example.
for(int y = 0; y < height; y++)
{
bitmap.getPixels(pixels, 0, width, 0, y, width, 1);

Related

Down scale issue on NV21 -> ARGB -> NV21 conversion

I have to provide a YUV(NV21) byte array to a recognition solution and I'd like, to reduce processing time, to down scale the preview frame.
From solutions gathered here and there on SO, I manage to convert on a 1:1 ratio and I get recognition hits. But if I'd like to scale the intermediate bitmap down, I get no result. Even if I scale it down to 95% only.
Any help would be appreciated.
Thus, every 400-ish ms I take the preview frame to convert it asynchronously. I convert it to ARGB using RenderScript, scale it down and then convert it back.
// Camera callback
#Override
public void onPreviewFrame(byte[] frame, Camera camera) {
if (camera != null) {
// Debounce
if ((System.currentTimeMillis() - mStart) > 400) {
mStart = System.currentTimeMillis();
Camera.Size size = camera.getParameters().getPreviewSize();
new FrameScaleAsyncTask(frame, size.width, size.height).execute();
}
}
if (mCamera != null) {
mCamera.addCallbackBuffer(mBuffer);
}
}
// In FrameScaleAsyncTask
#Override
protected Void doInBackground(Void... params) {
// Create YUV type for in-allocation
Type yuvType = new Type.Builder(mRenderScript, Element.U8(mRenderScript))
.setX(mFrame.length)
.create();
mAllocationIn = Allocation.createTyped(mRenderScript, yuvType, Allocation.USAGE_SCRIPT);
// Create ARGB-8888 type for out-allocation
Type rgbType = new Type.Builder(mRenderScript, Element.RGBA_8888(mRenderScript))
.setX(mWidth)
.setY(mHeight)
.create();
mAllocationOut = Allocation.createTyped(mRenderScript, rgbType, Allocation.USAGE_SCRIPT);
// Copy frame data into in-allocation
mAllocationIn.copyFrom(mFrame);
// Set script input and fire !
mScript.setInput(mAllocationIn);
mScript.forEach(mAllocationOut);
// Create a bitmap of camera preview size (see camera setup) and copy out-allocation to it
Bitmap bitmap = Bitmap.createBitmap(mWidth, mHeight, Bitmap.Config.ARGB_8888);
mAllocationOut.copyTo(bitmap);
// Scale bitmap down
double scaleRatio = 1;
Bitmap scaledBitmap = Bitmap.createScaledBitmap(
bitmap,
(int) (bitmap.getWidth() * scaleRatio),
(int) (bitmap.getHeight() * scaleRatio),
false
);
bitmap.recycle();
int size = scaledBitmap.getRowBytes() * scaledBitmap.getHeight();
int scaledWidth = scaledBitmap.getWidth();
int scaledHeight = scaledBitmap.getHeight();
int[] pixels = new int[scaledWidth * scaledHeight];
// Put bitmap pixels into an int array
scaledBitmap.getPixels(pixels, 0, scaledWidth, 0, 0, scaledWidth, scaledHeight);
mFrame = new byte[pixels.length * 3 / 2];
ImageHelper.encodeYUV420SPAlt(mFrame, pixels, scaledWidth, scaledHeight);
return null;
}
The RGB to YUV algorithm (see : this answer ):
public static void encodeYUV420SPAlt(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
// NV21 has a plane of Y and interleaved planes of VU each sampled by a factor of 2
// meaning for every 4 Y pixels there are 1 V and 1 U. Note the sampling is every other
// pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[uvIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U));
}
index++;
}
}
}

I finally end up resizing my image (as a OpenCV.Mat) directly in C++. This was way easier and faster.
Size size(correctedWidth, correctedHeight);
Mat dst;
resize(image, dst, size);

Android : Preview frame, converted from YCbCr_420_SP (NV21) format to RGB render correct picture but in green

I post this question as I didn’t find answers solving my issues in the following posts :
Converting preview frame to bitmap
Android decodeYUV420SP results in green images?
Displaying YUV Image in Android
I got my data and cameraResolution (after selecting BestPreviewSize) from :
public void onPreviewFrame(byte[] data, Camera camera) {
Point cameraResolution = configManager.getCameraResolution();
Handler thePreviewHandler = previewHandler;
if (cameraResolution != null && thePreviewHandler != null) {
Message message = thePreviewHandler.obtainMessage(previewMessage, cameraResolution.x,
cameraResolution.y, data);
message.sendToTarget();
previewHandler = null;
} else {
Log.d(TAG, "Got preview callback, but no handler or resolution available");
}
}
Later from conversion to RGB, I use :
int[] pixels=new int[yuvData.length];
// int size = dataWidth*dataHeight;
// int[] pixels=new int[size]; replacing yuvData.length by size is not working too
pixels=decodeYUV420SP(pixels,yuvData, dataWidth, dataHeight);
Bitmap bitmap = Bitmap.createBitmap(dataWidth, dataHeight, Bitmap.Config.RGB_565);
bitmap.setPixels(pixels, 0, dataWidth, 0, 0, dataWidth, dataHeight);
I tried 2 methods, one with RenderScript and one with decodeYUV420SP:
public Bitmap convertYUV420_NV21toRGB8888_RenderScript(byte [] data,int W, int H, CaptureActivityOCR fragment) {
// https://stackoverflow.com/questions/20358803/how-to-use-scriptintrinsicyuvtorgb-converting-byte-yuv-to-byte-rgba
RenderScript rs;
ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic;
rs = RenderScript.create(fragment.getActivity());
yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs)); //Create an intrinsic for converting YUV to RGB.
Type.Builder yuvType = new Type.Builder(rs, Element.U8(rs)).setX(data.length);
Allocation in = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT); //an Allocation will be populated with empty data when it is first created
Type.Builder rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs)).setX(W).setY(H);
Allocation out = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT); //an Allocation will be populated with empty data when it is first created
in.copyFrom(data);//Populate Allocations with data.
yuvToRgbIntrinsic.setInput(in); //Set the input yuv allocation, must be U8(RenderScript).
yuvToRgbIntrinsic.forEach(out); //Launch the appropriate kernels,Convert the image to RGB.
Bitmap bmpout = Bitmap.createBitmap(W, H, Bitmap.Config.ARGB_8888);
out.copyTo(bmpout); //Copy data out of Allocation objects.
return bmpout;
}
or
int[] decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width, int height) {
Log.e("camera", " decodeYUV420SP ");
// TODO Auto-generated method stub
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0) y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0) r = 0; else if (r > 262143) r = 262143;
if (g < 0) g = 0; else if (g > 262143) g = 262143;
if (b < 0) b = 0; else if (b > 262143) b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
return rgb;
}
But still I don't know why I got some good image but in green. (I have another similar method in black and white which is working)
In addition of the links above,i tried all posts linked to this subject on SO , so If someone could help for another tip please? I may be confused with the size to apply?
Somewhere after OnPreviewFrame method and before conversion to RGB method, I use another method with instruction below as I need to rotate the data received. I am wondering if this is not the origin of issue :
byte[] rotatedData = new byte[data.length];
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++)
rotatedData[x * height + height - y - 1] = data[x + y * width];
}
int tmp = width;
width = height;
height = tmp;
Please help me?

I think your rotation routine is wrong.
It works for Luma (Y), hence you get good results for black and white picture but not for chroma. If width and height are dimensions of your picture you do not rotate chroma values at all.
So you will need to add second loop for chroma and move pairs of bytes (V and U).
If you tried it without this rotation and you still have greenish picture the problem might be also in your decodeYUV420SP function.
There is not one universal formula for yuv to rgba conversion. It must match the opposite one.
Look here http://www.codeproject.com/Articles/402391/RGB-to-YUV-conversion-with-different-chroma-sampli and here http://www.fourcc.org/fccyvrgb.php
This one works for me
B = 1.164(Y - 16) + 2.018(U - 128)
G = 1.164(Y - 16) - 0.813(V - 128) - 0.391(U - 128)
R = 1.164(Y - 16) + 1.596(V - 128)
It seems to be the same as you are using only float version so you may try other.

I suggest to rotate the bitmap after RenderScript rotation:
if (bmpout == null) {
bmpout = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888);
}
rgbaAllocation.copyTo(bmpout);
if (matrixPostRotate90 == null) {
matrixPostRotate90 = new Matrix();
matrixPostRotate90.postRotate(90);
}
Bitmap rotatedBitmap = Bitmap.createBitmap(bmpout, 0, 0, w, h, matrixPostRotate90, true);

Android getPixels

Im currently working on a program which applies edge detection to an area of the preview frame. I have used previewcallback and got my cropped bitmap, have converted to grayscale using the following method.
int height1=120;
int width2=120;
final Bitmap resizedBitmap = Bitmap.createBitmap(bmp, 260, 15,
width2, height1);
try {
int bWidth = resizedBitmap.getWidth();
int bHeight = resizedBitmap.getHeight();
int[] pixels = new int[bWidth * bHeight];
resizedBitmap.getPixels(pixels, 0, bWidth, 0, 0, bWidth, bHeight);
for (int y = 0; y < bHeight; y++){
for (int x = 0; x < bWidth; x++){
int index = y * bWidth + x;
int R = (pixels[index] >> 16) & 0xff; //bitwise shifting
int G = (pixels[index] >> 8) & 0xff;
int B = pixels[index] & 0xff;
int gray = (int) (.299 * R + .587 * G + .114 * B);
}
}
I am very new to this, and would like to know whether gray is a 2D array of 120x120 pixels, or whether the value of gray is just being overwritten for each loop.
Apologies if this is very basic

Well, maybe I'm missing something, but as far as I can see gray is overwritten. You'd need something like
int[][] gray = new int[width][height];
// start loop
In the Loop:
gray[x][y] = ...;

How to convert YUV to RGB efficient

``I want to use RGB in the camera's preview.I used JNI to do the YUV to RGB conversion.I changed the data in RGB,then I show RGB on preview by using drawBitmap.But it shows the very slow,how could I improve it
public void onPreviewFrame(final byte[] data, Camera camera) {
Thread showPic = new Thread(new Runnable() {
#Override
public void run() {
// TODO Auto-generated method stub
Canvas c = mHolder.lockCanvas(null);
try {
synchronized (mHolder) {
// TODO Auto-generated method stub
int imageWidth = mCamera.getParameters()
.getPreviewSize().width;
int imageHeight = mCamera.getParameters()
.getPreviewSize().height;
int RGBData[] = new int[imageWidth * imageHeight];
int RGBDataa[] = new int[imageWidth * imageHeight];
int RGBDatab[] = new int[imageWidth * imageHeight];
int center = imageWidth * imageHeight / 2;
Jni.decodeYUV420SP(RGBData, data, imageWidth,
imageHeight); // decode
for (int i = 0; i < center; i++)
RGBDataa[i] = RGBData[i];
for (int i = center; i < imageWidth * imageHeight; i++)
RGBDatab[i - center] = RGBData[i];
for (int i = 0; i < center; i++)
RGBData[i] = RGBDatab[i];
for (int i = center; i < imageWidth * imageHeight; i++)
RGBData[i] = RGBDataa[i - center];
c.drawBitmap(RGBData, 0, imageWidth, 0, 0, imageWidth,
imageHeight, false, new Paint());
// Bitmap bm = Bitmap.createBitmap(RGBData, imageWidth,
// imageHeight, Config.ARGB_8888);
}
} finally {
if (data != null)
mHolder.unlockCanvasAndPost(c);
}
}
});
showPic.run();
}
The following code is Jni
public class Jni {
public native static void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width,
int height);
}
The method decodeYUV420SP is completed by C.

I came across this thread searching for YUV420sp to RGB565 conversion. Using decodeYUV420SP as suggested above works fine, but I want to add some runtime considerations.
My first test using decodeYUV420SP in Java took 74 seconds decoding and converting a 10 second .mp4 FullHD video (using an Nexus 6 Android device). The profiler reported 96% of the overall process has been spent in decodeYUV420SP.
Moving decodeYUV420SP to JNI / C made made overall time spent go down to 32 seconds. Turning on GCC code optimization, it fell to 12 seconds.
Although the OP suggested this already, I wanted to emphasizes it will not be an improvement to move the routine to Java. While it is correct a context change between Java and C comes at a cost, this is a good example an expensive operation performed in the native function will easily compensate this overhead. The statement on overhead costs is valid for very simple JNI functions and in case method and field lookups back and forth are required. Nothing of this applies here.
To improve the performance of the OP's code: get rid of the memory allocation done for each frame and store it globally, get rid of the arithmetic on RGBData* and move that stuff to JNI too. Arrays can be accessed on JNI level at no cost using the Critical functions.

It's not recommended to use jni because of slow speed for compiler to change java code to C++. I think you should use direct method like this
public void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width, int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0) y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0) r = 0; else if (r > 262143) r = 262143;
if (g < 0) g = 0; else if (g > 262143) g = 262143;
if (b < 0) b = 0; else if (b > 262143) b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
}
OR simply use this
Camera.Parameters.setPreviewFormat(ImageFormat.RGB_565);
to change the format output to rgb directly

Converting YUV->RGB(Image processing)->YUV during onPreviewFrame in android?

I am capturing image using SurfaceView and getting Yuv Raw preview data in public void onPreviewFrame4(byte[] data, Camera camera)
I have to perform some image preprocessing in onPreviewFrame so i need to convert Yuv preview data to RGB data than image preprocessing and back to Yuv data.
I have used both function for encoding and decoding Yuv data to RGB as following :
public void onPreviewFrame(byte[] data, Camera camera) {
Point cameraResolution = configManager.getCameraResolution();
if (data != null) {
Log.i("DEBUG", "data Not Null");
// Preprocessing
Log.i("DEBUG", "Try For Image Processing");
Camera.Parameters mParameters = camera.getParameters();
Size mSize = mParameters.getPreviewSize();
int mWidth = mSize.width;
int mHeight = mSize.height;
int[] mIntArray = new int[mWidth * mHeight];
// Decode Yuv data to integer array
decodeYUV420SP(mIntArray, data, mWidth, mHeight);
// Converting int mIntArray to Bitmap and
// than image preprocessing
// and back to mIntArray.
// Encode intArray to Yuv data
encodeYUV420SP(data, mIntArray, mWidth, mHeight);
}
}
static public void decodeYUV420SP(int[] rgba, byte[] yuv420sp, int width,
int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
// rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) &
// 0xff00) | ((b >> 10) & 0xff);
// rgba, divide 2^10 ( >> 10)
rgba[yp] = ((r << 14) & 0xff000000) | ((g << 6) & 0xff0000)
| ((b >> 2) | 0xff00);
}
}
}
static public void encodeYUV420SP_original(byte[] yuv420sp, int[] rgba,
int width, int height) {
final int frameSize = width * height;
int[] U, V;
U = new int[frameSize];
V = new int[frameSize];
final int uvwidth = width / 2;
int r, g, b, y, u, v;
for (int j = 0; j < height; j++) {
int index = width * j;
for (int i = 0; i < width; i++) {
r = (rgba[index] & 0xff000000) >> 24;
g = (rgba[index] & 0xff0000) >> 16;
b = (rgba[index] & 0xff00) >> 8;
// rgb to yuv
y = (66 * r + 129 * g + 25 * b + 128) >> 8 + 16;
u = (-38 * r - 74 * g + 112 * b + 128) >> 8 + 128;
v = (112 * r - 94 * g - 18 * b + 128) >> 8 + 128;
// clip y
yuv420sp[index++] = (byte) ((y < 0) ? 0 : ((y > 255) ? 255 : y));
U[index] = u;
V[index++] = v;
}
}
The problem is that encoding and decoding Yuv data might have some mistake because if i skip the preprocessing step than also encoded Yuv data are differ from original data of PreviewCallback.
Please help me to resolve this issue. I have to used this code in OCR scanning so i need to implement this type of logic.
If any other way of doing same thing than please provide me.
Thanks in advance. :)

Although the documentation suggests that you can set which format the image data should arrive from the camera in, in practice you often have a choice of one: NV21, a YUV format. For lots of information on this format see http://www.fourcc.org/yuv.php#NV21 and for information on the theory behind converting it to RGB see http://www.fourcc.org/fccyvrgb.php. There is a picture based explanation at Extract black and white image from android camera's NV21 format. There is an android specific section on a wikipedia page about the subject (thanks #AlexCohn): YUV#Y'UV420sp (NV21) to RGB conversion (Android).
However, once you've set up your onPreviewFrame routine, the mechanics of going from the byte array it sends you to useful data is somewhat, ummmm, unclear. From API 8 onwards, the following solution is available, to get to a ByteStream holiding a JPEG of the image (compressToJpeg is the only conversion option offered by YuvImage):
// pWidth and pHeight define the size of the preview Frame
ByteArrayOutputStream out = new ByteArrayOutputStream();
// Alter the second parameter of this to the actual format you are receiving
YuvImage yuv = new YuvImage(data, ImageFormat.NV21, pWidth, pHeight, null);
// bWidth and bHeight define the size of the bitmap you wish the fill with the preview image
yuv.compressToJpeg(new Rect(0, 0, bWidth, bHeight), 50, out);
This JPEG may then need to be converted into the format you want. If you want a Bitmap:
byte[] bytes = out.toByteArray();
Bitmap bitmap= BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
If, for whatever reason, you are unable to do this, you can do the conversion manually. Some problems to be overcome in doing this:
The data arrives in a byte array. By definition, bytes are signed numbers, meaning that they go from -128 to 127. However, the data is actually unsigned bytes (0 to 255). If this isn't dealt with, the outcome is doomed to have some odd clipping effects.
The data is in a very specific order (as per the previously mentioned web pages) and each pixel needs to be extracted carefully.
Each pixel needs to be put into the right place on a bitmap, say. This also requires a rather messy (in my view) approach of building a buffer of the data and then filling a bitmap from it.
In principle, the values should be stored [16..240], but it appears that they are stored [0..255] in the data sent to onPreviewFrame
Just about every web page on the matter proposes different coefficients, even allowing for [16..240] vs [0..255] options.
If you've actually got NV12 (another variant on YUV420), then you will need to swap the reads for U and V.
I present a solution (which seems to work), with requests for corrections, improvements and ways of making the whole thing less costly to run. I have set it out to hopefully make clear what is happening, rather than to optimise it for speed. It creates a bitmap the size of the preview image:
The data variable is coming from the call to onPreviewFrame
// Define whether expecting [16..240] or [0..255]
boolean dataIs16To240 = false;
// the bitmap we want to fill with the image
Bitmap bitmap = Bitmap.createBitmap(imageWidth, imageHeight, Bitmap.Config.ARGB_8888);
int numPixels = imageWidth*imageHeight;
// the buffer we fill up which we then fill the bitmap with
IntBuffer intBuffer = IntBuffer.allocate(imageWidth*imageHeight);
// If you're reusing a buffer, next line imperative to refill from the start,
// if not good practice
intBuffer.position(0);
// Set the alpha for the image: 0 is transparent, 255 fully opaque
final byte alpha = (byte) 255;
// Holding variables for the loop calculation
int R = 0;
int G = 0;
int B = 0;
// Get each pixel, one at a time
for (int y = 0; y < imageHeight; y++) {
for (int x = 0; x < imageWidth; x++) {
// Get the Y value, stored in the first block of data
// The logical "AND 0xff" is needed to deal with the signed issue
float Y = (float) (data[y*imageWidth + x] & 0xff);
// Get U and V values, stored after Y values, one per 2x2 block
// of pixels, interleaved. Prepare them as floats with correct range
// ready for calculation later.
int xby2 = x/2;
int yby2 = y/2;
// make this V for NV12/420SP
float U = (float)(data[numPixels + 2*xby2 + yby2*imageWidth] & 0xff) - 128.0f;
// make this U for NV12/420SP
float V = (float)(data[numPixels + 2*xby2 + 1 + yby2*imageWidth] & 0xff) - 128.0f;
if (dataIs16To240) {
// Correct Y to allow for the fact that it is [16..235] and not [0..255]
Y = 1.164*(Y - 16.0);
// Do the YUV -> RGB conversion
// These seem to work, but other variations are quoted
// out there.
R = (int)(Yf + 1.596f*V);
G = (int)(Yf - 0.813f*V - 0.391f*U);
B = (int)(Yf + 2.018f*U);
}
else {
// No need to correct Y
// These are the coefficients proposed by #AlexCohn
// for [0..255], as per the wikipedia page referenced
// above
R = (int)(Yf + 1.370705f*V);
G = (int)(Yf - 0.698001f*V - 0.337633f*U);
B = (int)(Yf + 1.732446f*U);
}
// Clip rgb values to 0-255
R = R < 0 ? 0 : R > 255 ? 255 : R;
G = G < 0 ? 0 : G > 255 ? 255 : G;
B = B < 0 ? 0 : B > 255 ? 255 : B;
// Put that pixel in the buffer
intBuffer.put(alpha*16777216 + R*65536 + G*256 + B);
}
}
// Get buffer ready to be read
intBuffer.flip();
// Push the pixel information from the buffer onto the bitmap.
bitmap.copyPixelsFromBuffer(intBuffer);
As #Timmmm points out below, you could do the conversion in int by multiplying the scaling factors by 1000 (ie. 1.164 becomes 1164) and then dividng the end results by 1000.

Why not specify that camera preview should provide RGB images?
i.e. Camera.Parameters.setPreviewFormat(ImageFormat.RGB_565);

You can use RenderScript -> ScriptIntrinsicYuvToRGB
Kotlin Sample
val rs = RenderScript.create(CONTEXT_HERE)
val yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs))
val yuvType = Type.Builder(rs, Element.U8(rs)).setX(byteArray.size)
val inData = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT)
val rgbaType = Type.Builder(rs, Element.RGBA_8888(rs)).setX(width).setY(height)
val outData = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT)
inData.copyFrom(byteArray)
yuvToRgbIntrinsic.setInput(inData)
yuvToRgbIntrinsic.forEach(outData)
val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
outData.copyTo(bitmap)

After some tests on Samsung S4 mini fastest code is (120% faster then Neil's [floats!] and 30% faster then original Hitesh's):
static public void decodeYUV420SP(int[] rgba, byte[] yuv420sp, int width,
int height) {
final int frameSize = width * height;
// define variables before loops (+ 20-30% faster algorithm o0`)
int r, g, b, y1192, y, i, uvp, u, v;
for (int j = 0, yp = 0; j < height; j++) {
uvp = frameSize + (j >> 1) * width;
u = 0;
v = 0;
for (i = 0; i < width; i++, yp++) {
y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
y1192 = 1192 * y;
r = (y1192 + 1634 * v);
g = (y1192 - 833 * v - 400 * u);
b = (y1192 + 2066 * u);
// Java's functions are faster then 'IFs'
r = Math.max(0, Math.min(r, 262143));
g = Math.max(0, Math.min(g, 262143));
b = Math.max(0, Math.min(b, 262143));
// rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) &
// 0xff00) | ((b >> 10) & 0xff);
// rgba, divide 2^10 ( >> 10)
rgba[yp] = ((r << 14) & 0xff000000) | ((g << 6) & 0xff0000)
| ((b >> 2) | 0xff00);
}
}
}
Speed is comparable to YuvImage.compressToJpeg() with ByteArrayOutputStream as output (30-50 ms for 640x480 image).
Result: Samsung S4 mini (2x1.7GHz) can't compress to JPEG/convert YUV to RGB in real time (640x480#30fps)

Java implementation is 10 times slow than the c version, I suggest you use GPUImage library or just move this part of code.
There is a android version of GPUImage:
https://github.com/CyberAgent/android-gpuimage
You can include this library if you use gradle, and call the method:
GPUImageNativeLibrary.YUVtoRBGA( inputArray, WIDTH, HEIGHT, outputArray);
I compare the time, for a NV21 image which is 960x540, use above java code, it cost 200ms+, with GPUImage version, just 10ms~20ms.

You can use ColorHelper library for this:
using ColorHelper;
YUV yuv = new YUV(0.1, 0.1, 0.2);
RGB rgb = ColorConverter.YuvToRgb(yuv);
Links:
Github
Nuget

Fixup the above code snippet
static public void decodeYUV420SP(int[] rgba, byte[] yuv420sp, int width,
int height) {
final int frameSize = width * height;
int r, g, b, y1192, y, i, uvp, u, v;
for (int j = 0, yp = 0; j < height; j++) {
uvp = frameSize + (j >> 1) * width;
u = 0;
v = 0;
for (i = 0; i < width; i++, yp++) {
y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
// above answer is wrong at the following lines. just swap ***u*** and ***v***
u = (0xff & yuv420sp[uvp++]) - 128;
v = (0xff & yuv420sp[uvp++]) - 128;
}
y1192 = 1192 * y;
r = (y1192 + 1634 * v);
g = (y1192 - 833 * v - 400 * u);
b = (y1192 + 2066 * u);
r = Math.max(0, Math.min(r, 262143));
g = Math.max(0, Math.min(g, 262143));
b = Math.max(0, Math.min(b, 262143));
// combine ARGB
rgba[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00)
| ((b >> 10) | 0xff);
}
}
}

Try RenderScript ScriptIntrinsicYuvToRGB, which comes with JellyBean 4.2 (Api 17+).
https://developer.android.com/reference/android/renderscript/ScriptIntrinsicYuvToRGB.html
On Nexus 7 (2013, JellyBean 4.3) a 1920x1080 image conversion (full HD camera preview) takes about 7 ms.

You can get the bitmap directly from the TextureView. Which is really fast.
Bitmap bitmap = textureview.getBitmap()

After reading many suggested links, articles, etc. I found the following great Android example app which captures the YUV Image from the camera and converts it into RGB Bitmap:
https://github.com/android/camera-samples/tree/main/CameraXTfLite
Nice things about this:
It uses the aforementioned RenderScript framework and the code can be easily reused - check out the YuvToRgbConverter.kt class
according to their documentation, this code achieves " ~30 FPS # 640x480 on a Pixel 3 phone"
After switching to this code (especially the YUV to RGB conversion part) my framerate doubled! I am not quite reaching 30 FPS overall since I am doing a bit more things after capturing the image, but the speed-up is remarkable!

Develop Reference

The Android operating system is a mobile operating system that was developed by Google (GOOGL?) to be primarily used for touchscreen devices, cell phones, and tablets.

Bitmap to big to use getPixels? - android

Related

Down scale issue on NV21 -> ARGB -> NV21 conversion

Android : Preview frame, converted from YCbCr_420_SP (NV21) format to RGB render correct picture but in green

Android getPixels

How to convert YUV to RGB efficient

Converting YUV->RGB(Image processing)->YUV during onPreviewFrame in android?

Categories

Resources