I’m encoding a video from a set of Bitmaps by using the MediaCodec API.
As the minimum project API is 22, I’m using COLOR_FormatYUV420Flexible to obtain a suitable encoder.
The issue is that some users have reported that the output video result is in Black & White instead of in color.
The bitmaps being pushed into the encoder are converted to YUV by using the below algorithm.
My best guess is that in the affected devices the encoder may be expecting a specific YUV format.
The question is, how to find which YUV format does the encoder expect? And, isn’t there a common YUV format which will work for all devices?
Algorithm used to convert the bitmaps to YUV:
void toYuv(const uint32_t* argb, int8_t* yuv, const uint32_t width, const uint32_t height)
{
int32_t index_y = 0;
int32_t index_uv = width * height;
int32_t index = 0;
for (int j = 0; j < height; j++)
{
for (int i = 0; i < width; i++)
{
const int R = (argb[index] & 0x0000FF) >> 0;
const int G = (argb[index] & 0xFF00) >> 8;
const int B = (argb[index] & 0xFF0000) >> 16;
const int Y = ((((R * 77) + (G * 150)) + (B * 29)) + 128) >> 8;
const int U = (((((R * 127) - (G * 106)) - (B * 21)) + 128) >> 8) + 128;
const int V = (((((R * -43) - (G * 84)) + (B * 127)) + 128) >> 8) + 128;
yuv[index_y++] = static_cast<int8_t>((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (((j % 2) == 0) && ((index % 2) == 0))
{
yuv[index_uv++] = static_cast<int8_t>((V < 0) ? 0 : ((V > 255) ? 255 : V));
yuv[index_uv++] = static_cast<int8_t>((U < 0) ? 0 : ((U > 255) ? 255 : U));
}
index++;
}
}
}
Related
I'm trying to convert the image data from an Android device from YUV_420_888 to an RGB matrix on the C++ side. On some devices, this is working flawlessly. On a Note 10, the image comes out looking like this:
My guess here is that the stride is causing this issue. How do I remove this extra data and then pass the correct buffer through JNI?
Here is the Java code:
IntBuffer rgb = image.getPlanes()[0].getBuffer().asIntBuffer();
NativeLib.passImageBuffer(rgb);
And here is the C++ code:
cv::Mat outputRGB;
cv::cvtColor(cv::Mat(height+height/2, width, CV_8UC1, inputRGB), outputRGB, CV_YUV2BGR_NV21);
I've tried some different image formats on the C++ side, but they all come back with the same band on the side of the screen.
I've implemented this answer, in order to remove the extra padding, but the image that is passed ends up being completely green. Do some corresponding edits need to be made to the C++ code? I've tried using a 3 channel format, but that crashes at runtime. I'm thinking that since passing the buffer works with the 1 channel matrix on phones that have 8 bits per pixel, that it should be possible to do that with the note 10?
Image.Plane Y = image.getPlanes()[0];
Image.Plane U = image.getPlanes()[1];
Image.Plane V = image.getPlanes()[2];
int[] rgbBytes = new int[image.getHeight()*image.getWidth()*4];
int idx = 0;
ByteBuffer yBuffer = Y.getBuffer();
int yPixelStride = Y.getPixelStride();
int yRowStride = Y.getRowStride();
ByteBuffer uBuffer = U.getBuffer();
int uPixelStride = U.getPixelStride();
int uRowStride = U.getRowStride();
ByteBuffer vBuffer = V.getBuffer();
int vPixelStride = V.getPixelStride();
int vRowStride = V.getRowStride();
ByteBuffer rgbBuffer = ByteBuffer.allocateDirect(rgb.limit());
for (int row = 0; row < image.getHeight(); row++) {
for (int col = 0; col < image.getWidth(); col++) {
int y = yBuffer.get(col*yPixelStride + row*yRowStride) & 0xff;
int u = uBuffer.get(col/2*uPixelStride + row/2*uRowStride) & 0xff;
int v = vBuffer.get(col/2*vPixelStride + row/2*vRowStride) & 0xff;
int y1 = ((19077 << 8) * y) >> 16;
int r = (y1 + (((26149 << 8) * v) >> 16) - 14234) >> 6;
int g = (y1 - (((6419 << 8) * u) >> 16) - (((13320 << 8) * v) >> 16) + 8708) >> 6;
int b = (y1 + (((33050 << 8) * u) >> 16) - 17685) >> 6;
if (r < 0) r = 0;
if (g < 0) g = 0;
if (b < 0) b = 0;
if (r > 255) r = 255;
if (g > 255) g = 255;
if (b > 255) b = 255;
byte pixel = (byte)(0xff000000 + b + 256 * (g + 256 * r));
rgbBuffer.put(pixel);
}
}
Look at this repo
https://github.com/quickbirdstudios/yuvToMat/
It supports different formats (YUV420, NV12) and variety of pixel and row strides.
Im trying to render a video frame using android NDK.
Im using this sample of google Native-Codec NDK sample code and modified it so I can manually display each video frame (non-tunneled).
so I added this code to get the output buffer which is in YUV.
ANativeWindow_setBuffersGeometry(mWindow, bufferWidth, bufferHeight,
WINDOW_FORMAT_RGBA_8888
uint8_t *decodedBuff = AMediaCodec_getOutputBuffer(d->codec, status, &bufSize);
auto format = AMediaCodec_getOutputFormat(d->codec);
LOGV("VOUT: format %s", AMediaFormat_toString(format));
AMediaFormat *myFormat = format;
int32_t w,h;
AMediaFormat_getInt32(myFormat, AMEDIAFORMAT_KEY_HEIGHT, &h);
AMediaFormat_getInt32(myFormat, AMEDIAFORMAT_KEY_WIDTH, &w);
err = ANativeWindow_lock(mWindow, &buffer, nullptr);
and these codes to convert the YUV to RGB and display it using native window.
if (err == 0) {
LOGV("ANativeWindow_lock()");
int width =w;
int height=h;
int const frameSize = width * height;
int *line = reinterpret_cast<int *>(buffer.bits);
for (int y= 0; y < height; y++) {
for (int x = 0; x < width; x++) {
/*accessing YUV420SP elements*/
int indexY = y * width + x;
int indexU = (size + (y / 2) * (width ) + (x / 2) *2);
int indexV = (int) (size + (y / 2) * (width) + (x / 2) * 2 + 1);
/*todo; this conversion to int and then later back to int really isn't required.
There's room for better work here.*/
int Y = 0xFF & decodedBuff[indexY];
int U = 0xFF & decodedBuff[indexU];
int V = 0xFF & decodedBuff[indexV];
/*constants picked up from http://www.fourcc.org/fccyvrgb.php*/
int R = (int) (Y + 1.402f * (V - 128));
int G = (int) (Y - 0.344f * (U - 128) - 0.714f * (V - 128));
int B = (int) (Y + 1.772f * (U - 128));
/*clamping values*/
R = R < 0 ? 0 : R;
G = G < 0 ? 0 : G;
B = B < 0 ? 0 : B;
R = R > 255 ? 255 : R;
G = G > 255 ? 255 : G;
B = B > 255 ? 255 : B;
line[buffer.stride * y + x] = 0xff000000 + (B << 16) + (G << 8) + R;
}
}
ANativeWindow_unlockAndPost(mWindow);
Finally I was able to display a video on my device. Now my problem is the video does not scale to fit the surface view :(
Your thoughts are very much appreciated.
I'm doing an app to record a video. What I would like is that when the app is recording the video, each frame is also saved in an arraylist of RGB values in order to extract particular information from it. I know that the two processes (video recording and extraction of frames) are asynchronous, but this is not a problem: the process of extraction can finish after the video recording.
Can someone tell me how I can extract the frames from the video?
Thanks very much.
Using the camera object you can override a function setPreviewCallback. from that function you would get an array of byte data. Which you can convert to bitmap or you can save them in an array of array.
Assuming that you were extends SurfaceView and Implements SurfaceHolder.Callback
Code looks like,
mCamera.setPreviewCallback(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera) {
//do some operations using data
}
});
If you want to get the RGB value you can use this function,
static public void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width,
int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000)
| ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
}
I think this is what you want as an answer. If i am wrong, please let me know whats is your actual requirement
For getting the green value, you can use the below code,
static public void calculateColor(int[] rgb, int[] color,
int width, int height, int component) {
for (int bin = 0; bin < 256; bin++) {
color[bin] = 0;
} // bin
int start = 0;
int end = width * height;
if (component == 0) // red
{
for (int pix = start; pix < end; pix += 3) {
int pixVal = (rgb[pix] >> 16) & 0xff;
color[pixVal]++;
} // pix
} else if (component == 1) // green
{
for (int pix = start; pix < end; pix += 3) {
int pixVal = (rgb[pix] >> 8) & 0xff;
color[pixVal]++;
} // pix
} else // blue
{
for (int pix = start; pix < end; pix += 3) {
int pixVal = rgb[pix] & 0xff;
color[pixVal]++;
} // pix
}
}
Calling this function would return the value of green for every pixel. If you wants the green value for a particular pixel, you have to modify this code.
Sample code for extracting frames from a .mp4 file can be found on the bigflake site. It uses MediaCodec, available on API 16+. If you need to run on older devices, your best bet is probably ffmpeg.
RGB data requires 3 bytes per pixel. 10 seconds of 720p video at 30 fps will require 1280*720*3*30*10 = 791 megabytes. Unless your video frames are fairly small, holding on to a large number of them may not be practical (or possible).
It seems to me you want to get the image while video recording is ongoing.
In that case you will need to look into Build your camera app. This page describes how to record a video using android camera class.
You can then get yuv images through Camera preview callback.
How can I convert Bitmap created using getDrawingCache() to YUV420planar format ?
The bitmap is created from a custom view like this
view.setDrawingCacheEnabled(true);
bitmap = Bitmap.createScaledBitmap(board.getDrawingCache(), 320, 240, true);
board.setDrawingCacheEnabled(false);
This question is similar but doesn't give the same format that I am looking for.
Convert bitmap array to YUV (YCbCr NV21).
Thanks in advance.
A bit of reading into YUV formats helped me figure it out. I just had to modify a tiny bit of the code linked above. For anyone who has the same doubt
void encodeYUV420SP(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uIndex = frameSize;
int vIndex = frameSize+((yuv420sp.length-frameSize)/2);
System.out.println(yuv420sp.length+" "+frameSize);
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ( ( -38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ( ( 112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
// NV21 has a plane of Y and interleaved planes of VU each sampled by a factor of 2
// meaning for every 4 Y pixels there are 1 V and 1 U. Note the sampling is every other
// pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[uIndex++] = (byte)((U<0) ? 0 : ((U > 255) ? 255 : U));
yuv420sp[vIndex++] = (byte)((V<0) ? 0 : ((V > 255) ? 255 : V));
}
index ++;
}
}
}
In our application, we need to transfer video, we are using Camera class to capture the buffer and send to destination,
I have set format is YV12 as a Camera parameter to receive the buffer,
for the 500X300 buffer, we receive buffer of 230400 bytes,
i want to know , is this expected buffer size ?
I believe the size would be
Y Plane = width * height = 500X300 = 150000
U Plane = width/2 * height/2 = = 37500
V Plane = width/2 * height/2 = = 37500
========
225000
========
Can anyone explain me, if i need to get stride values of each component, how can i get that
Is there any way to get it ?
I can show you how you can get int rgb[] from this:
public int[] decodeYUV420SP(byte[] yuv420sp, int width, int height) {
final int frameSize = width * height;
int rgb[] = new int[width * height];
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000)
| ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
return rgb;
}
I guess Android document is already explained it:
http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12
I think this is simple.
chekout YUVImage class from android. You can construct an YUV Image from byte[]data coming from camera preview.
You can write like this:
//width and height you get it from camera properties, image width and height of camera preview
YuvImage image=new YuvImage(data, ImageFormat.NV21, int width, int height, null);
byte[] newData = image.getYuvData();
//or if you want int format = image.getYuvFormat();
It's a quite old question, but I've struggled with the same issue for a few days. So I decided to write some comments to help others.
YV12 described in the Android developer site(here) seems not a kind of YV12 but IMC1. The page says that both of the y-stride and the uv-stride should be aligned in 16bytes.
And also this page says that:
For YV12, the image buffer that is received is not necessarily tightly
packed, as there may be padding at the end of each row of pixel data,
as described in YV12.
Based on the above comments, I calculated it using python command line:
>>> w = 500
>>> h = 300
>>> y_stride = (500 + 15) / 16 * 16.0
>>> y_stride
512.0
>>> y_size = y_stride * h
>>> y_size
153600.0
>>> uv_stride = (500 / 2 + 15) / 16 * 16.0
>>> uv_stride
256.0
>>> u_size = uv_stride * h / 2
>>> v_size = uv_stride * h / 2
>>> size = y_size + u_size + v_size
>>> size
230400.0