Develop Reference

Android ImageReader get NV21 format?

I do not have a background in imaging or graphics, so please bear with me :)
I am using JavaCV in one of my projects. In the examples, a Frame is constructed which has a buffer of a certain size.
When using the public void onPreviewFrame(byte[] data, Camera camera) function in Android, copying this data byte array is no problem if you declare the Frame as new Frame(frameWidth, frameHeight, Frame.DEPTH_UBYTE, 2); where frameWidth and frameHeight are declared as
Camera.Size previewSize = cameraParam.getPreviewSize();
int frameWidth = previewSize.width;
int frameHeight = previewSize.height;
Recently, Android added a method to capture your screen. Naturally, I wanted to grab those images and also covert them to Frames. I modified the example code from Google to use the ImageReader.
This ImageReader is constructed as ImageReader.newInstance(DISPLAY_WIDTH, DISPLAY_HEIGHT, PixelFormat.RGBA_8888, 2);. So currently it uses the RGBA_8888 pixel format. I use the following code to copy the bytes to the Frame, which is instantiated as new Frame(DISPLAY_WIDTH, DISPLAY_HEIGHT, Frame.DEPTH_UBYTE, 2);:
ByteBuffer buffer = mImage.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.remaining()];
buffer.get(bytes);
mImage.close();
((ByteBuffer) frame.image[0].position(0)).put(bytes);
But this gives me a java.nio.BufferOverflowException. I printed the sizes of both buffers and the Frame's buffer size is 691200 whereas the bytes array above is of size 1413056. Figuring out how this latter number is constructed failed because I ran into this native call. So clearly, this won't work out.
After quite a bit of digging I found out that the NV21 image format is "the default format for Camera preview images, when not otherwise set with setPreviewFormat(int)", but the ImageReader class does not support the NV21 format (see the format parameter). So that's tough luck. In the documentation it also reads that
"For the android.hardware.camera2 API, the YUV_420_888 format is recommended for YUV output instead."
So I tried creating an ImageReader like this ImageReader.newInstance(DISPLAY_WIDTH, DISPLAY_HEIGHT, ImageFormat.YUV_420_888, 2);, but this gives me java.lang.UnsupportedOperationException: The producer output buffer format 0x1 doesn't match the ImageReader's configured buffer format 0x23. so that won't work either.
As a last resort, I tried to convert RGBA_8888 to YUV myself using e.g. this post, but I fail to understand how I can obtain an int[] rgba as per the answer.
So, TL;DR how can I obtain NV21 image data like you get in Android's public void onPreviewFrame(byte[] data, Camera camera) camera function to instantiate my Frame and work with it using Android's ImageReader (and Media Projection)?
Edit (25-10-2016)
I have created the following conversion runnable to go from RGBA to NV21 format:
private class updateImage implements Runnable {
private final Image mImage;
public updateImage(Image image) {
mImage = image;
}
#Override
public void run() {
int mWidth = mImage.getWidth();
int mHeight = mImage.getHeight();
// Four bytes per pixel: width * height * 4.
byte[] rgbaBytes = new byte[mWidth * mHeight * 4];
// put the data into the rgbaBytes array.
mImage.getPlanes()[0].getBuffer().get(rgbaBytes);
mImage.close(); // Access to the image is no longer needed, release it.
// Create a yuv byte array: width * height * 1.5 ().
byte[] yuv = new byte[mWidth * mHeight * 3 / 2];
RGBtoNV21(yuv, rgbaBytes, mWidth, mHeight);
((ByteBuffer) yuvImage.image[0].position(0)).put(yuv);
}
void RGBtoNV21(byte[] yuv420sp, byte[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int A, R, G, B, Y, U, V;
int index = 0;
int rgbIndex = 0;
for (int i = 0; i < height; i++) {
for (int j = 0; j < width; j++) {
R = argb[rgbIndex++];
G = argb[rgbIndex++];
B = argb[rgbIndex++];
A = argb[rgbIndex++]; // Ignored right now.
// RGB to YUV conversion according to
// https://en.wikipedia.org/wiki/YUV#Y.E2.80.B2UV444_to_RGB888_conversion
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
// NV21 has a plane of Y and interleaved planes of VU each sampled by a factor
// of 2 meaning for every 4 Y pixels there are 1 V and 1 U.
// Note the sampling is every other pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (i % 2 == 0 && index % 2 == 0) {
yuv420sp[uvIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U));
}
index++;
}
}
}
}
The yuvImage object is initialized as yuvImage = new Frame(DISPLAY_WIDTH, DISPLAY_HEIGHT, Frame.DEPTH_UBYTE, 2);, the DISPLAY_WIDTH and DISPLAY_HEIGHT are just two integers specifying the display size.
This is the code where a background handler handles the onImageReady:
private final ImageReader.OnImageAvailableListener mOnImageAvailableListener
= new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(ImageReader reader) {
mBackgroundHandler.post(new updateImage(reader.acquireNextImage()));
}
};
...
mImageReader = ImageReader.newInstance(DISPLAY_WIDTH, DISPLAY_HEIGHT, PixelFormat.RGBA_8888, 2);
mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler);
The methods work and I at least don't get any errors, but the output image is malformed. What is going wrong in my conversion? An example image that is being created:
Edit (15-11-2016)
I have modified the RGBtoNV21 function to be the following:
void RGBtoNV21(byte[] yuv420sp, int width, int height) {
try {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int pixelStride = mImage.getPlanes()[0].getPixelStride();
int rowStride = mImage.getPlanes()[0].getRowStride();
int rowPadding = rowStride - pixelStride * width;
ByteBuffer buffer = mImage.getPlanes()[0].getBuffer();
Bitmap bitmap = Bitmap.createBitmap(getResources().getDisplayMetrics(), width, height, Bitmap.Config.ARGB_8888);
int A, R, G, B, Y, U, V;
int offset = 0;
for (int i = 0; i < height; i++) {
for (int j = 0; j < width; j++) {
// Useful link: https://stackoverflow.com/questions/26673127/android-imagereader-acquirelatestimage-returns-invalid-jpg
R = (buffer.get(offset) & 0xff) << 16; // R
G = (buffer.get(offset + 1) & 0xff) << 8; // G
B = (buffer.get(offset + 2) & 0xff); // B
A = (buffer.get(offset + 3) & 0xff) << 24; // A
offset += pixelStride;
int pixel = 0;
pixel |= R; // R
pixel |= G; // G
pixel |= B; // B
pixel |= A; // A
bitmap.setPixel(j, i, pixel);
// RGB to YUV conversion according to
// https://en.wikipedia.org/wiki/YUV#Y.E2.80.B2UV444_to_RGB888_conversion
// Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
// U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
// V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
Y = (int) Math.round(R * .299000 + G * .587000 + B * .114000);
U = (int) Math.round(R * -.168736 + G * -.331264 + B * .500000 + 128);
V = (int) Math.round(R * .500000 + G * -.418688 + B * -.081312 + 128);
// NV21 has a plane of Y and interleaved planes of VU each sampled by a factor
// of 2 meaning for every 4 Y pixels there are 1 V and 1 U.
// Note the sampling is every other pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (i % 2 == 0 && j % 2 == 0) {
yuv420sp[uvIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U));
}
}
offset += rowPadding;
}
File file = new File(Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES).getAbsolutePath(), "/Awesomebitmap.png");
FileOutputStream fos = new FileOutputStream(file);
bitmap.compress(Bitmap.CompressFormat.PNG, 100, fos);
} catch (Exception e) {
Timber.e(e, "Converting image to NV21 went wrong.");
}
}
Now the image is no longer malformed, but the chroma is off.
The right side is the bitmap that is being created in that loop, the left side is the NV21 saved to an image. So the RGB pixels are processed correctly. Clearly the chroma is off, but the RGB to YUV conversion should be the same one as depicted by wikipedia. What could be wrong here?

Generally speaking, the point of ImageReader is to give you raw access to the pixels sent to the Surface with minimal overhead, so attempting to have it perform color conversions doesn't make sense.
For the Camera you get to pick one of two output formats (NV21 or YV12), so pick YV12. That's your raw YUV data. For screen capture the output will always be RGB, so you need to pick RGBA_8888 (format 0x1) for your ImageReader, rather than YUV_420_888 (format 0x23). If you need YUV for that, you will have to do the conversion yourself. The ImageReader gives you a series of Plane objects, not a byte[], so you will need to adapt to that.

How Use Sobel matrix with one pixel

I wanna use Sobel operator in my android app. But I don't understand how use one pixel.
int sobel_x[][] = {{-1, 0, 1},
{-2, 0, 2},
{-1, 0, 1}};
int sobel_y[][] = {{-1, -2, -1},
{0, 0, 0},
{1, 2, 1}};
Bitmap source = ImageHelper.GetBitmapGromUri(Path);
int w = source.getWidth();
int h = source.getHeight();
int[] pixels;
pixels = new int[h * w];
source.getPixels(pixels, 0, w, 1, 1, w - 1, h - 1);
for(int i = 0;i < pixels.length;i++){
...
}
I try use get/setPixel. But is very very slow.

Good news and bad news. The following works but...
Android, for some peculiar reason, doesn't allow you to create an 8 bit grey scale image. This means that you have to create a greyscale in ARGB_8888 format. This is probably what was going wrong in your previous version, we read data as bytes when it wasn't.
The code below works and I've only run it on an emulator against your image where it is ridiculously slow (11 seconds). Of course your image is very big but this is still, I would guess, way to slow.
I would strongly suggest considering using OpenCV Java libraries as they are both fast and memory efficient unlike the Android Bitmap class!
public class Sobel {
private static Bitmap toGreyScale( Bitmap source ) {
Bitmap greyScaleBitmap = Bitmap.createBitmap(
source.getWidth(), source.getHeight(),
Bitmap.Config.ARGB_8888);
Canvas c = new Canvas(greyScaleBitmap);
Paint p = new Paint();
ColorMatrix cm = new ColorMatrix();
cm.setSaturation(0);
ColorMatrixColorFilter filter = new ColorMatrixColorFilter(cm);
p.setColorFilter(filter);
c.drawBitmap(source, 0, 0, p);
return greyScaleBitmap;
}
public static void doSobel( Bitmap source) {
Bitmap grey = toGreyScale(source);
int w = grey.getWidth();
int h = grey.getHeight();
// Allocate 4 times as much data as is necessary because Android.
int sz = w * h;
IntBuffer buffer = IntBuffer.allocate( sz );
grey.copyPixelsToBuffer( buffer );
final int[] bitmapData = buffer.array();
int[] output = new int[ w * h ];
for( int y=1; y<h-1; y++ ) {
for( int x=1; x<w-1; x++ ) {
int idx = (y * w + x );
// Apply Sobel filter
int tl = (bitmapData[idx - w - 1]) & 0xFF;
int tr = (bitmapData[idx - w + 1]) & 0xFF;
int l = (bitmapData[idx - 1]) & 0xFF;
int r = (bitmapData[idx + 1]) & 0xFF;
int bl = (bitmapData[idx + w - 1]) & 0xFF;
int br = (bitmapData[idx + w + 1]) & 0xFF;
int sx = (int) ( tr - tl + 2 * ( r - l ) + br - bl );
sx = sx & 0xFF;
// Put back into ARG and B bytes
output[toIdx] = (sx << 24) | ( sx << 16) | (sx << 8) | sx;
}
}
source.copyPixelsFromBuffer( IntBuffer.wrap(output));
}
}

Individual pixel access is definitely not the right way to go!
I assume that you have converted source to grayscale and so it's essentially a byte array. I suggest you use something like the following to extract the data:
int w = source.getWidth();
int h = source.getHeight();
ByteBuffer buffer = ByteBuffer.allocate( w * h );
source.copyPixelsToBuffer( buffer );
final byte[] bitmapData = buffer.array()
This gives you the source data. You can now apply your Sobel filters to it. Note that you want to write the resulting output bytes into a new byte array and then convert that back into an image.
byte[] output = new byte[ w * h ];
for( int y=1; y<h-1; y++ ) {
for( int x=1; x<w-1; x++ ) {
int idx = y * w + x;
// Apply Sobel filter
byte sx = (byte) ((-1 * bitmapData[idx-w-1]) + ( 1 * bitmapData[idx-w+1] ) + (-2 * bitmapData[idx-1]) + ( 2 * bitmapData[idx+1] ) + (-1 * bitmapData[idx+w-1]) + ( 1 * bitmapData[idx+w+1] ) );
output[idx] = sx;
}
}
You can so similarly for the horizontal filter.
Edited to correct type of output from int to byte

Android : Preview frame, converted from YCbCr_420_SP (NV21) format to RGB render correct picture but in green

I post this question as I didn’t find answers solving my issues in the following posts :
Converting preview frame to bitmap
Android decodeYUV420SP results in green images?
Displaying YUV Image in Android
I got my data and cameraResolution (after selecting BestPreviewSize) from :
public void onPreviewFrame(byte[] data, Camera camera) {
Point cameraResolution = configManager.getCameraResolution();
Handler thePreviewHandler = previewHandler;
if (cameraResolution != null && thePreviewHandler != null) {
Message message = thePreviewHandler.obtainMessage(previewMessage, cameraResolution.x,
cameraResolution.y, data);
message.sendToTarget();
previewHandler = null;
} else {
Log.d(TAG, "Got preview callback, but no handler or resolution available");
}
}
Later from conversion to RGB, I use :
int[] pixels=new int[yuvData.length];
// int size = dataWidth*dataHeight;
// int[] pixels=new int[size]; replacing yuvData.length by size is not working too
pixels=decodeYUV420SP(pixels,yuvData, dataWidth, dataHeight);
Bitmap bitmap = Bitmap.createBitmap(dataWidth, dataHeight, Bitmap.Config.RGB_565);
bitmap.setPixels(pixels, 0, dataWidth, 0, 0, dataWidth, dataHeight);
I tried 2 methods, one with RenderScript and one with decodeYUV420SP:
public Bitmap convertYUV420_NV21toRGB8888_RenderScript(byte [] data,int W, int H, CaptureActivityOCR fragment) {
// https://stackoverflow.com/questions/20358803/how-to-use-scriptintrinsicyuvtorgb-converting-byte-yuv-to-byte-rgba
RenderScript rs;
ScriptIntrinsicYuvToRGB yuvToRgbIntrinsic;
rs = RenderScript.create(fragment.getActivity());
yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs)); //Create an intrinsic for converting YUV to RGB.
Type.Builder yuvType = new Type.Builder(rs, Element.U8(rs)).setX(data.length);
Allocation in = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT); //an Allocation will be populated with empty data when it is first created
Type.Builder rgbaType = new Type.Builder(rs, Element.RGBA_8888(rs)).setX(W).setY(H);
Allocation out = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT); //an Allocation will be populated with empty data when it is first created
in.copyFrom(data);//Populate Allocations with data.
yuvToRgbIntrinsic.setInput(in); //Set the input yuv allocation, must be U8(RenderScript).
yuvToRgbIntrinsic.forEach(out); //Launch the appropriate kernels,Convert the image to RGB.
Bitmap bmpout = Bitmap.createBitmap(W, H, Bitmap.Config.ARGB_8888);
out.copyTo(bmpout); //Copy data out of Allocation objects.
return bmpout;
}
or
int[] decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width, int height) {
Log.e("camera", " decodeYUV420SP ");
// TODO Auto-generated method stub
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0) y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0) r = 0; else if (r > 262143) r = 262143;
if (g < 0) g = 0; else if (g > 262143) g = 262143;
if (b < 0) b = 0; else if (b > 262143) b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
return rgb;
}
But still I don't know why I got some good image but in green. (I have another similar method in black and white which is working)
In addition of the links above,i tried all posts linked to this subject on SO , so If someone could help for another tip please? I may be confused with the size to apply?
Somewhere after OnPreviewFrame method and before conversion to RGB method, I use another method with instruction below as I need to rotate the data received. I am wondering if this is not the origin of issue :
byte[] rotatedData = new byte[data.length];
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++)
rotatedData[x * height + height - y - 1] = data[x + y * width];
}
int tmp = width;
width = height;
height = tmp;
Please help me?

I think your rotation routine is wrong.
It works for Luma (Y), hence you get good results for black and white picture but not for chroma. If width and height are dimensions of your picture you do not rotate chroma values at all.
So you will need to add second loop for chroma and move pairs of bytes (V and U).
If you tried it without this rotation and you still have greenish picture the problem might be also in your decodeYUV420SP function.
There is not one universal formula for yuv to rgba conversion. It must match the opposite one.
Look here http://www.codeproject.com/Articles/402391/RGB-to-YUV-conversion-with-different-chroma-sampli and here http://www.fourcc.org/fccyvrgb.php
This one works for me
B = 1.164(Y - 16) + 2.018(U - 128)
G = 1.164(Y - 16) - 0.813(V - 128) - 0.391(U - 128)
R = 1.164(Y - 16) + 1.596(V - 128)
It seems to be the same as you are using only float version so you may try other.

I suggest to rotate the bitmap after RenderScript rotation:
if (bmpout == null) {
bmpout = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888);
}
rgbaAllocation.copyTo(bmpout);
if (matrixPostRotate90 == null) {
matrixPostRotate90 = new Matrix();
matrixPostRotate90.postRotate(90);
}
Bitmap rotatedBitmap = Bitmap.createBitmap(bmpout, 0, 0, w, h, matrixPostRotate90, true);

Converting camera YUV-data to ARGB with renderscript

My Problem is: I've set up a camera in Android and receive the preview data by using an onPreviewFrame-listener which passes me an byte[] array containing the image data in the default android YUV-format (device does not support R5G6B5-format). Each pixel consists of 12bits which makes the thing a little tricky. Now what I want to do is converting the YUV-data into ARGB-data in order to do image processing with it. This has to be done with renderscript, in order to maintain a high performance.
My idea was to pass two pixels in one element (which would be 24bits = 3 bytes) and then return two ARGB pixels. The problem is, that in Renderscript a u8_3 (a 3dimensional 8bit vector) is stored in 32bit, which means that the last 8 bits are unused. But when copying the image data into the allocation all of the 32bits are used, so the last 8bit get lost. Even if I used a 32bit input data, the last 8bit are useless, because they're only 2/3 of a pixel. When defining an element consisting a 3-byte-array it actually has a real size of 3 bytes. But then the Allocation.copyFrom()-method doesn't fill the in-Allocation with data, argueing it doesn't has the right data type to be filled with a byte[].
The renderscript documentation states, that there is a ScriptIntrinsicYuvToRGB which should do exactly that in API Level 17. But in fact the class doesn't exist. I've downloaded API Level 17 even though it seems not to be downloadable any more. Does anyone have any information about it? Does anyone have ever tried out a ScriptIntrinsic?
So in conclusion my question is: How to convert the camera data into ARGB data fast, hardwareaccelerated?
That's how to do it in Dalvik VM (found the code somewhere online, it works):
#SuppressWarnings("unused")
private void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width, int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
}

I'm sure you will find the LivePreview test application interesting ... it's part of the Android source code in the latest Jelly Bean (MR1). It implements a camera preview and uses ScriptIntrinsicYuvToRgb to convert the preview data with Renderscript. You can browse the source online here:
LivePreview

I was not able to get running ScriptInstrinsicYuvToRgb, so I decided to write my own RS solution.
Here's ready script (named yuv.rs):
#pragma version(1)
#pragma rs java_package_name(com.package.name)
rs_allocation gIn;
int width;
int height;
int frameSize;
void yuvToRgb(const uchar *v_in, uchar4 *v_out, const void *usrData, uint32_t x, uint32_t y) {
uchar yp = rsGetElementAtYuv_uchar_Y(gIn, x, y) & 0xFF;
int index = frameSize + (x & (~1)) + (( y>>1) * width );
int v = (int)( rsGetElementAt_uchar(gIn, index) & 0xFF ) -128;
int u = (int)( rsGetElementAt_uchar(gIn, index+1) & 0xFF ) -128;
int r = (int) (1.164f * yp + 1.596f * v );
int g = (int) (1.164f * yp - 0.813f * v - 0.391f * u);
int b = (int) (1.164f * yp + 2.018f * u );
r = r>255? 255 : r<0 ? 0 : r;
g = g>255? 255 : g<0 ? 0 : g;
b = b>255? 255 : b<0 ? 0 : b;
uchar4 res4;
res4.r = (uchar)r;
res4.g = (uchar)g;
res4.b = (uchar)b;
res4.a = 0xFF;
*v_out = res4;
}
Don't forget to set camera preview format to NV21:
Parameters cameraParameters = camera.getParameters();
cameraParameters.setPreviewFormat(ImageFormat.NV21);
// Other camera init stuff: preview size, framerate, etc.
camera.setParameters(cameraParameters);
Allocations initialization and script usage:
// Somewhere in initialization section
// w and h are variables for selected camera preview size
rs = RenderScript.create(this);
Type.Builder tbIn = new Type.Builder(rs, Element.U8(rs));
tbIn.setX(w);
tbIn.setY(h);
tbIn.setYuvFormat(ImageFormat.NV21);
Type.Builder tbOut = new Type.Builder(rs, Element.RGBA_8888(rs));
tbOut.setX(w);
tbOut.setY(h);
inData = Allocation.createTyped(rs, tbIn.create(), Allocation.MipmapControl.MIPMAP_NONE, Allocation.USAGE_SCRIPT & Allocation.USAGE_SHARED);
outData = Allocation.createTyped(rs, tbOut.create(), Allocation.MipmapControl.MIPMAP_NONE, Allocation.USAGE_SCRIPT & Allocation.USAGE_SHARED);
outputBitmap = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888);
yuvScript = new ScriptC_yuv(rs);
yuvScript.set_gIn(inData);
yuvScript.set_width(w);
yuvScript.set_height(h);
yuvScript.set_frameSize(previewSize);
//.....
Camera callback method:
public void onPreviewFrame(byte[] data, Camera camera) {
// In your camera callback, data
inData.copyFrom(data);
yuvScript.forEach_yuvToRgb(inData, outData);
outData.copyTo(outputBitmap);
// draw your bitmap where you want to
// .....
}

For anyone who didn't know, RenderScript is now in the Android Support Library, including intrinsics.
http://android-developers.blogspot.com.au/2013/09/renderscript-in-android-support-library.html
http://android-developers.blogspot.com.au/2013/08/renderscript-intrinsics.html

We now have the new renderscript-intrinsics-replacement-toolkit to do it. First, build and import the renderscript module to your project and add it as a dependency to your app module. Then, go to Toolkit.kt and add the following:
fun toNv21(image: Image): ByteArray? {
val nv21 = ByteArray((image.width * image.height * 1.5f).toInt())
return if (!nativeYuv420toNv21(
nativeHandle,
image.width,
image.height,
image.planes[0].buffer, // Y buffer
image.planes[1].buffer, // U buffer
image.planes[2].buffer, // V buffer
image.planes[0].pixelStride, // Y pixel stride
image.planes[1].pixelStride, // U/V pixel stride
image.planes[0].rowStride, // Y row stride
image.planes[1].rowStride, // U/V row stride
nv21
)
) {
null
} else nv21
}
private external fun nativeYuv420toNv21(
nativeHandle: Long,
imageWidth: Int,
imageHeight: Int,
yByteBuffer: ByteBuffer,
uByteBuffer: ByteBuffer,
vByteBuffer: ByteBuffer,
yPixelStride: Int,
uvPixelStride: Int,
yRowStride: Int,
uvRowStride: Int,
nv21Output: ByteArray
): Boolean
Now, go to JniEntryPoints.cpp and add the following:
extern "C" JNIEXPORT jboolean JNICALL Java_com_google_android_renderscript_Toolkit_nativeYuv420toNv21(
JNIEnv *env, jobject/*thiz*/, jlong native_handle,
jint image_width, jint image_height, jobject y_byte_buffer,
jobject u_byte_buffer, jobject v_byte_buffer, jint y_pixel_stride,
jint uv_pixel_stride, jint y_row_stride, jint uv_row_stride,
jbyteArray nv21_array) {
auto y_buffer = static_cast<jbyte*>(env->GetDirectBufferAddress(y_byte_buffer));
auto u_buffer = static_cast<jbyte*>(env->GetDirectBufferAddress(u_byte_buffer));
auto v_buffer = static_cast<jbyte*>(env->GetDirectBufferAddress(v_byte_buffer));
jbyte* nv21 = env->GetByteArrayElements(nv21_array, nullptr);
if (nv21 == nullptr || y_buffer == nullptr || u_buffer == nullptr
|| v_buffer == nullptr) {
// Log this.
return false;
}
RenderScriptToolkit* toolkit = reinterpret_cast<RenderScriptToolkit*>(native_handle);
toolkit->yuv420toNv21(image_width, image_height, y_buffer, u_buffer, v_buffer,
y_pixel_stride, uv_pixel_stride, y_row_stride, uv_row_stride,
nv21);
env->ReleaseByteArrayElements(nv21_array, nv21, 0);
return true;
}
Go to YuvToRgb.cpp and add the following:
void RenderScriptToolkit::yuv420toNv21(int image_width, int image_height, const int8_t* y_buffer,
const int8_t* u_buffer, const int8_t* v_buffer, int y_pixel_stride,
int uv_pixel_stride, int y_row_stride, int uv_row_stride,
int8_t *nv21) {
// Copy Y channel.
for(int y = 0; y < image_height; ++y) {
int destOffset = image_width * y;
int yOffset = y * y_row_stride;
memcpy(nv21 + destOffset, y_buffer + yOffset, image_width);
}
if (v_buffer - u_buffer == sizeof(int8_t)) {
// format = nv21
// TODO: If the format is VUVUVU & pixel stride == 1 we can simply the copy
// with memcpy. In Android Camera2 I have mostly come across UVUVUV packaging
// though.
}
// Copy UV Channel.
int idUV = image_width * image_height;
int uv_width = image_width / 2;
int uv_height = image_height / 2;
for(int y = 0; y < uv_height; ++y) {
int uvOffset = y * uv_row_stride;
for (int x = 0; x < uv_width; ++x) {
int bufferIndex = uvOffset + (x * uv_pixel_stride);
// V channel.
nv21[idUV++] = v_buffer[bufferIndex];
// U channel.
nv21[idUV++] = u_buffer[bufferIndex];
}
}
}
Finally, go to RenderscriptToolkit.h and add the following:
/**
* https://blog.minhazav.dev/how-to-use-renderscript-to-convert-YUV_420_888-yuv-image-to-bitmap/#tobitmapimage-image-method
* #param image_width width of the image you want to convert to byte array
* #param image_height height of the image you want to convert to byte array
* #param y_buffer Y buffer
* #param u_buffer U buffer
* #param v_buffer V buffer
* #param y_pixel_stride Y pixel stride
* #param uv_pixel_stride UV pixel stride
* #param y_row_stride Y row stride
* #param uv_row_stride UV row stride
* #param nv21 the output byte array
*/
void yuv420toNv21(int image_width, int image_height, const int8_t* y_buffer,
const int8_t* u_buffer, const int8_t* v_buffer, int y_pixel_stride,
int uv_pixel_stride, int y_row_stride, int uv_row_stride,
int8_t *nv21);
You are now ready to harness the full power of renderscript. Below, I am providing an example with the ARCore Camera Image object (replace the first line with whatever code gives you your camera image):
val cameraImage = arFrame.frame.acquireCameraImage()
val width = cameraImage.width
val height = cameraImage.height
val byteArray = Toolkit.toNv21(cameraImage)
byteArray?.let {
Toolkit.yuvToRgbBitmap(
byteArray,
width,
height,
YuvFormat.NV21
).let { bitmap ->
saveBitmapToDevice(
name,
session,
bitmap,
context
)}}

Extract black and white image from android camera's NV21 format

I have done some google-ing around and couldn't find enough information about this format. It is the default format for camera preview. Can anyone suggest good sources of information about it and how to extract data from a photo/preview image with that format? To be more specific, I need the black and white image extracted.
EDIT: Seems like that format is also called YCbCr 420 Semi Planar

I developed the following code to convert the NV21 to RGB, and it is working.
/**
* Converts YUV420 NV21 to RGB8888
*
* #param data byte array on YUV420 NV21 format.
* #param width pixels width
* #param height pixels height
* #return a RGB8888 pixels int array. Where each int is a pixels ARGB.
*/
public static int[] convertYUV420_NV21toRGB8888(byte [] data, int width, int height) {
int size = width*height;
int offset = size;
int[] pixels = new int[size];
int u, v, y1, y2, y3, y4;
// i percorre os Y and the final pixels
// k percorre os pixles U e V
for(int i=0, k=0; i < size; i+=2, k+=2) {
y1 = data[i ]&0xff;
y2 = data[i+1]&0xff;
y3 = data[width+i ]&0xff;
y4 = data[width+i+1]&0xff;
u = data[offset+k ]&0xff;
v = data[offset+k+1]&0xff;
u = u-128;
v = v-128;
pixels[i ] = convertYUVtoRGB(y1, u, v);
pixels[i+1] = convertYUVtoRGB(y2, u, v);
pixels[width+i ] = convertYUVtoRGB(y3, u, v);
pixels[width+i+1] = convertYUVtoRGB(y4, u, v);
if (i!=0 && (i+2)%width==0)
i+=width;
}
return pixels;
}
private static int convertYUVtoRGB(int y, int u, int v) {
int r,g,b;
r = y + (int)(1.402f*v);
g = y - (int)(0.344f*u +0.714f*v);
b = y + (int)(1.772f*u);
r = r>255? 255 : r<0 ? 0 : r;
g = g>255? 255 : g<0 ? 0 : g;
b = b>255? 255 : b<0 ? 0 : b;
return 0xff000000 | (b<<16) | (g<<8) | r;
}
This image helps to understand.
If you wanna just grayscale image is easer. You can discard all the U and V info, and take just the Y info. The code would can be like this:
/**
* Converts YUV420 NV21 to Y888 (RGB8888). The grayscale image still holds 3 bytes on the pixel.
*
* #param pixels output array with the converted array o grayscale pixels
* #param data byte array on YUV420 NV21 format.
* #param width pixels width
* #param height pixels height
*/
public static void applyGrayScale(int [] pixels, byte [] data, int width, int height) {
int p;
int size = width*height;
for(int i = 0; i < size; i++) {
p = data[i] & 0xFF;
pixels[i] = 0xff000000 | p<<16 | p<<8 | p;
}
}
To create your Bitmap just:
Bitmap bm = Bitmap.createBitmap(pixels, width, height, Bitmap.Config.ARGB_8888);
Where pixels is your int [] array.

NV21 is basically YUV420 but instead of planar format where Y, U and V have independent planes, NV21 has 1 plane for Luma and 2nd plane for Chroma. The format looks like
YYYYYYYYYYYYYYYYYYYYYYYYYYYYY
YYYYYYYYYYYYYYYYYYYYYYYYYYYYY
.
.
.
.
VUVUVUVUVUVUVUVUVUVUVUVUVUVUVU
VUVUVUVUVUVUVUVUVUVUVUVUVUVUVU
.
.
.
.
.

I also had lots of headache because of this preview format.
The best I could find are these:
http://www.fourcc.org/yuv.php#NV21
http://v4l2spec.bytesex.org/spec/r5470.htm
It seems that the Y component is the first width*height bytes int the array you get.
Some more informational links:
http://msdn.microsoft.com/en-us/library/ms867704.aspx#yuvformats_yuvsampling
http://msdn.microsoft.com/en-us/library/ff538197(v=vs.85).aspx
Hope this helps.

The data is in YUV420 format.
If you are only interested in the monochrome channel, i.e. "black and white", then this the first width x height bytes of the data buffer you already have.
The Y channel is the first image plane. It is exactly the grey/intensity/luminosity etc. channel.

Here's code to just extract the greyscale image data:
private int[] decodeGreyscale(byte[] nv21, int width, int height) {
int pixelCount = width * height;
int[] out = new int[pixelCount];
for (int i = 0; i < pixelCount; ++i) {
int luminance = nv21[i] & 0xFF;
out[i] = Color.argb(0xFF, luminance, luminance, luminance);
}
return out;
}

When you only need a grayscale camera preview, you could use a very simple
renderscript:
# pragma version(1)
# pragma rs java_package_name(com.example.name)
# pragma rs_fp_relaxed
rs_allocation gIn; // Allocation filled with camera preview data (byte[])
int previewwidth; // camera preview width (int)
// the parallel executed kernel
void root(uchar4 *v_out, uint32_t x,uint32_t y){
uchar c = rsGetElementAt_uchar(gIn,x+y*previewwidth);
*v_out = (uchar4){c,c,c,255};
}
Note : This is not faster than ScriptIntrinsicYuvToRGB (and a following ScriptIntrinsicColorMatrix to do the RGBA-> gray), but
it runs with API 11+ (where the Intrinsics need Api 17+).