Develop Reference

How to correctly pass YUV_420_888 Image Buffer from Java through JNI to OpenCV, accounting for stride/padding

I'm trying to convert the image data from an Android device from YUV_420_888 to an RGB matrix on the C++ side. On some devices, this is working flawlessly. On a Note 10, the image comes out looking like this:
My guess here is that the stride is causing this issue. How do I remove this extra data and then pass the correct buffer through JNI?
Here is the Java code:
IntBuffer rgb = image.getPlanes()[0].getBuffer().asIntBuffer();
NativeLib.passImageBuffer(rgb);
And here is the C++ code:
cv::Mat outputRGB;
cv::cvtColor(cv::Mat(height+height/2, width, CV_8UC1, inputRGB), outputRGB, CV_YUV2BGR_NV21);
I've tried some different image formats on the C++ side, but they all come back with the same band on the side of the screen.
I've implemented this answer, in order to remove the extra padding, but the image that is passed ends up being completely green. Do some corresponding edits need to be made to the C++ code? I've tried using a 3 channel format, but that crashes at runtime. I'm thinking that since passing the buffer works with the 1 channel matrix on phones that have 8 bits per pixel, that it should be possible to do that with the note 10?
Image.Plane Y = image.getPlanes()[0];
Image.Plane U = image.getPlanes()[1];
Image.Plane V = image.getPlanes()[2];
int[] rgbBytes = new int[image.getHeight()*image.getWidth()*4];
int idx = 0;
ByteBuffer yBuffer = Y.getBuffer();
int yPixelStride = Y.getPixelStride();
int yRowStride = Y.getRowStride();
ByteBuffer uBuffer = U.getBuffer();
int uPixelStride = U.getPixelStride();
int uRowStride = U.getRowStride();
ByteBuffer vBuffer = V.getBuffer();
int vPixelStride = V.getPixelStride();
int vRowStride = V.getRowStride();
ByteBuffer rgbBuffer = ByteBuffer.allocateDirect(rgb.limit());
for (int row = 0; row < image.getHeight(); row++) {
for (int col = 0; col < image.getWidth(); col++) {
int y = yBuffer.get(col*yPixelStride + row*yRowStride) & 0xff;
int u = uBuffer.get(col/2*uPixelStride + row/2*uRowStride) & 0xff;
int v = vBuffer.get(col/2*vPixelStride + row/2*vRowStride) & 0xff;
int y1 = ((19077 << 8) * y) >> 16;
int r = (y1 + (((26149 << 8) * v) >> 16) - 14234) >> 6;
int g = (y1 - (((6419 << 8) * u) >> 16) - (((13320 << 8) * v) >> 16) + 8708) >> 6;
int b = (y1 + (((33050 << 8) * u) >> 16) - 17685) >> 6;
if (r < 0) r = 0;
if (g < 0) g = 0;
if (b < 0) b = 0;
if (r > 255) r = 255;
if (g > 255) g = 255;
if (b > 255) b = 255;
byte pixel = (byte)(0xff000000 + b + 256 * (g + 256 * r));
rgbBuffer.put(pixel);
}
}

Look at this repo
https://github.com/quickbirdstudios/yuvToMat/
It supports different formats (YUV420, NV12) and variety of pixel and row strides.

MediaCodec encoder produces Black & White videos in some devices

I’m encoding a video from a set of Bitmaps by using the MediaCodec API.
As the minimum project API is 22, I’m using COLOR_FormatYUV420Flexible to obtain a suitable encoder.
The issue is that some users have reported that the output video result is in Black & White instead of in color.
The bitmaps being pushed into the encoder are converted to YUV by using the below algorithm.
My best guess is that in the affected devices the encoder may be expecting a specific YUV format.
The question is, how to find which YUV format does the encoder expect? And, isn’t there a common YUV format which will work for all devices?
Algorithm used to convert the bitmaps to YUV:
void toYuv(const uint32_t* argb, int8_t* yuv, const uint32_t width, const uint32_t height)
{
int32_t index_y = 0;
int32_t index_uv = width * height;
int32_t index = 0;
for (int j = 0; j < height; j++)
{
for (int i = 0; i < width; i++)
{
const int R = (argb[index] & 0x0000FF) >> 0;
const int G = (argb[index] & 0xFF00) >> 8;
const int B = (argb[index] & 0xFF0000) >> 16;
const int Y = ((((R * 77) + (G * 150)) + (B * 29)) + 128) >> 8;
const int U = (((((R * 127) - (G * 106)) - (B * 21)) + 128) >> 8) + 128;
const int V = (((((R * -43) - (G * 84)) + (B * 127)) + 128) >> 8) + 128;
yuv[index_y++] = static_cast<int8_t>((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (((j % 2) == 0) && ((index % 2) == 0))
{
yuv[index_uv++] = static_cast<int8_t>((V < 0) ? 0 : ((V > 255) ? 255 : V));
yuv[index_uv++] = static_cast<int8_t>((U < 0) ? 0 : ((U > 255) ? 255 : U));
}
index++;
}
}
}

How to convert YUV420SP to RGB and display it?

Im trying to render a video frame using android NDK.
Im using this sample of google Native-Codec NDK sample code and modified it so I can manually display each video frame (non-tunneled).
so I added this code to get the output buffer which is in YUV.
ANativeWindow_setBuffersGeometry(mWindow, bufferWidth, bufferHeight,
WINDOW_FORMAT_RGBA_8888
uint8_t *decodedBuff = AMediaCodec_getOutputBuffer(d->codec, status, &bufSize);
auto format = AMediaCodec_getOutputFormat(d->codec);
LOGV("VOUT: format %s", AMediaFormat_toString(format));
AMediaFormat *myFormat = format;
int32_t w,h;
AMediaFormat_getInt32(myFormat, AMEDIAFORMAT_KEY_HEIGHT, &h);
AMediaFormat_getInt32(myFormat, AMEDIAFORMAT_KEY_WIDTH, &w);
err = ANativeWindow_lock(mWindow, &buffer, nullptr);
and these codes to convert the YUV to RGB and display it using native window.
if (err == 0) {
LOGV("ANativeWindow_lock()");
int width =w;
int height=h;
int const frameSize = width * height;
int *line = reinterpret_cast<int *>(buffer.bits);
for (int y= 0; y < height; y++) {
for (int x = 0; x < width; x++) {
/*accessing YUV420SP elements*/
int indexY = y * width + x;
int indexU = (size + (y / 2) * (width ) + (x / 2) *2);
int indexV = (int) (size + (y / 2) * (width) + (x / 2) * 2 + 1);
/*todo; this conversion to int and then later back to int really isn't required.
There's room for better work here.*/
int Y = 0xFF & decodedBuff[indexY];
int U = 0xFF & decodedBuff[indexU];
int V = 0xFF & decodedBuff[indexV];
/*constants picked up from http://www.fourcc.org/fccyvrgb.php*/
int R = (int) (Y + 1.402f * (V - 128));
int G = (int) (Y - 0.344f * (U - 128) - 0.714f * (V - 128));
int B = (int) (Y + 1.772f * (U - 128));
/*clamping values*/
R = R < 0 ? 0 : R;
G = G < 0 ? 0 : G;
B = B < 0 ? 0 : B;
R = R > 255 ? 255 : R;
G = G > 255 ? 255 : G;
B = B > 255 ? 255 : B;
line[buffer.stride * y + x] = 0xff000000 + (B << 16) + (G << 8) + R;
}
}
ANativeWindow_unlockAndPost(mWindow);
Finally I was able to display a video on my device. Now my problem is the video does not scale to fit the surface view :(
Your thoughts are very much appreciated.

Down scale issue on NV21 -> ARGB -> NV21 conversion

I have to provide a YUV(NV21) byte array to a recognition solution and I'd like, to reduce processing time, to down scale the preview frame.
From solutions gathered here and there on SO, I manage to convert on a 1:1 ratio and I get recognition hits. But if I'd like to scale the intermediate bitmap down, I get no result. Even if I scale it down to 95% only.
Any help would be appreciated.
Thus, every 400-ish ms I take the preview frame to convert it asynchronously. I convert it to ARGB using RenderScript, scale it down and then convert it back.
// Camera callback
#Override
public void onPreviewFrame(byte[] frame, Camera camera) {
if (camera != null) {
// Debounce
if ((System.currentTimeMillis() - mStart) > 400) {
mStart = System.currentTimeMillis();
Camera.Size size = camera.getParameters().getPreviewSize();
new FrameScaleAsyncTask(frame, size.width, size.height).execute();
}
}
if (mCamera != null) {
mCamera.addCallbackBuffer(mBuffer);
}
}
// In FrameScaleAsyncTask
#Override
protected Void doInBackground(Void... params) {
// Create YUV type for in-allocation
Type yuvType = new Type.Builder(mRenderScript, Element.U8(mRenderScript))
.setX(mFrame.length)
.create();
mAllocationIn = Allocation.createTyped(mRenderScript, yuvType, Allocation.USAGE_SCRIPT);
// Create ARGB-8888 type for out-allocation
Type rgbType = new Type.Builder(mRenderScript, Element.RGBA_8888(mRenderScript))
.setX(mWidth)
.setY(mHeight)
.create();
mAllocationOut = Allocation.createTyped(mRenderScript, rgbType, Allocation.USAGE_SCRIPT);
// Copy frame data into in-allocation
mAllocationIn.copyFrom(mFrame);
// Set script input and fire !
mScript.setInput(mAllocationIn);
mScript.forEach(mAllocationOut);
// Create a bitmap of camera preview size (see camera setup) and copy out-allocation to it
Bitmap bitmap = Bitmap.createBitmap(mWidth, mHeight, Bitmap.Config.ARGB_8888);
mAllocationOut.copyTo(bitmap);
// Scale bitmap down
double scaleRatio = 1;
Bitmap scaledBitmap = Bitmap.createScaledBitmap(
bitmap,
(int) (bitmap.getWidth() * scaleRatio),
(int) (bitmap.getHeight() * scaleRatio),
false
);
bitmap.recycle();
int size = scaledBitmap.getRowBytes() * scaledBitmap.getHeight();
int scaledWidth = scaledBitmap.getWidth();
int scaledHeight = scaledBitmap.getHeight();
int[] pixels = new int[scaledWidth * scaledHeight];
// Put bitmap pixels into an int array
scaledBitmap.getPixels(pixels, 0, scaledWidth, 0, 0, scaledWidth, scaledHeight);
mFrame = new byte[pixels.length * 3 / 2];
ImageHelper.encodeYUV420SPAlt(mFrame, pixels, scaledWidth, scaledHeight);
return null;
}
The RGB to YUV algorithm (see : this answer ):
public static void encodeYUV420SPAlt(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
// NV21 has a plane of Y and interleaved planes of VU each sampled by a factor of 2
// meaning for every 4 Y pixels there are 1 V and 1 U. Note the sampling is every other
// pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[uvIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U));
}
index++;
}
}
}

I finally end up resizing my image (as a OpenCV.Mat) directly in C++. This was way easier and faster.
Size size(correctedWidth, correctedHeight);
Mat dst;
resize(image, dst, size);

Android YUV Image format

In our application, we need to transfer video, we are using Camera class to capture the buffer and send to destination,
I have set format is YV12 as a Camera parameter to receive the buffer,
for the 500X300 buffer, we receive buffer of 230400 bytes,
i want to know , is this expected buffer size ?
I believe the size would be
Y Plane = width * height = 500X300 = 150000
U Plane = width/2 * height/2 = = 37500
V Plane = width/2 * height/2 = = 37500
========
225000
========
Can anyone explain me, if i need to get stride values of each component, how can i get that
Is there any way to get it ?

I can show you how you can get int rgb[] from this:
public int[] decodeYUV420SP(byte[] yuv420sp, int width, int height) {
final int frameSize = width * height;
int rgb[] = new int[width * height];
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000)
| ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
return rgb;
}

I guess Android document is already explained it:
http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12

I think this is simple.
chekout YUVImage class from android. You can construct an YUV Image from byte[]data coming from camera preview.
You can write like this:
//width and height you get it from camera properties, image width and height of camera preview
YuvImage image=new YuvImage(data, ImageFormat.NV21, int width, int height, null);
byte[] newData = image.getYuvData();
//or if you want int format = image.getYuvFormat();

It's a quite old question, but I've struggled with the same issue for a few days. So I decided to write some comments to help others.
YV12 described in the Android developer site(here) seems not a kind of YV12 but IMC1. The page says that both of the y-stride and the uv-stride should be aligned in 16bytes.
And also this page says that:
For YV12, the image buffer that is received is not necessarily tightly
packed, as there may be padding at the end of each row of pixel data,
as described in YV12.
Based on the above comments, I calculated it using python command line:
>>> w = 500
>>> h = 300
>>> y_stride = (500 + 15) / 16 * 16.0
>>> y_stride
512.0
>>> y_size = y_stride * h
>>> y_size
153600.0
>>> uv_stride = (500 / 2 + 15) / 16 * 16.0
>>> uv_stride
256.0
>>> u_size = uv_stride * h / 2
>>> v_size = uv_stride * h / 2
>>> size = y_size + u_size + v_size
>>> size
230400.0