When fetching images:
assert(AIMAGE_FORMAT_JPEG == src_format, "Failed to get format");
AImage_getHeight(image, &height);
AImage_getWidth(image, &width);
AImage_getPlaneData(image, 0, &pixel, &y_len);
AImage_getPlaneRowStride(image, 0, &stride);
AImage_getPlanePixelStride(image, 0, &pixel_stride);
Note, that the last 2 commands return AMEDIA_ERROR_UNSUPPORTED for that format - see docs.
Now writing on the buffer:
uint8_t *out = buf.data;
for (int y = 0; y < height; y++) {
const uint8_t *pY = pixel + width*3*y;
for (int x = 0; x < width; x++) {
out[x*3 + 0] = pY[x*3 + 0];
out[x*3 + 1] = pY[x*3 + 1];
out[x*3 + 2] = pY[x*3 + 2];
}
out += width*3;
}
the app crashes after start up very quickly. The logcat does not output something meaningful to me - at least not readable. Any thoughts on how to approach this issue?
getPlaneData() for JPEG gives you the undecoded buffer. You can see that the returned size is much smaller than width*height*3.
Related
I have a YUV_420_888 image I got from the camera. I want to crop a rectangle out of grayscale of this image to feed to an image processing algorithm. This is what I have so far:
public static byte[] YUV_420_888toCroppedY(Image image, Rect cropRect) {
byte[] yData;
ByteBuffer yBuffer = image.getPlanes()[0].getBuffer();
int ySize = yBuffer.remaining();
yData = new byte[ySize];
yBuffer.get(yData, 0, ySize);
if (cropRect != null) {
int cropArea = (cropRect.right - cropRect.left) * (cropRect.bottom - cropRect.top);
byte[] croppedY = new byte[cropArea];
int cropIndex = 0;
// from the top of the rectangle, to the bottom, sequentially add rows to the output array, croppedY
for (int y = cropRect.top; y < cropRect.top + cropRect.height(); y++) {
// (2x+W) * y + x
int rowStart = (2*cropRect.left + cropRect.width()) * y + cropRect.left;
// (2x+W) * y + x + W
int rowEnd = (2*cropRect.left + cropRect.width()) * y + cropRect.left + cropRect.width();
for (int x = rowStart; x < rowEnd; x++) {
croppedY[cropIndex] = yData[x];
cropIndex++;
}
}
return croppedY;
}
return yData;
}
This function runs without error but the image I get out of it is garbage - it looks something like this:
I'm not sure how to solve this problem or what I'm doing wrong.
Your rowStart/end calculations are wrong.
You need to calculate the row start location based on the source image dimensions, not on your crop window dimensions. And I'm not sure where you get the factor of 2 from; there's 1 byte per pixel in the Y channel of the image.
They should be roughly:
int yRowStride = image.getPlanes()[0].getRowStride();
..
int rowStart = y * yRowStride + cropRect.left();
int rowEnd = y * yRowStride + cropRect.left() + cropRect.width();
I'm getting a strange glitch in a FFT graph for white noise:
I've checked with reference program and while noise file seems to be fine.
Is it a bug in implementation?
void four1(float data[], int nn, int isign) {
int n, mmax, m, j, istep, i;
float wtemp, wr, wpr, wpi, wi, theta;
float tempr, tempi;
n = nn << 1;
j = 1;
for (int i = 1; i < n; i += 2) {
if (j > i) {
tempr = data[j];
data[j] = data[i];
data[i] = tempr;
tempr = data[j + 1];
data[j + 1] = data[i + 1];
data[i + 1] = tempr;
}
m = n >> 1;
while (m >= 2 && j > m) {
j -= m;
m >>= 1;
}
j += m;
}
mmax = 2;
while (n > mmax) {
istep = 2 * mmax;
theta = TWOPI / (isign * mmax);
wtemp = sin(0.5 * theta);
wpr = -2.0 * wtemp * wtemp;
wpi = sin(theta);
wr = 1.0;
wi = 0.0;
for (m = 1; m < mmax; m += 2) {
for (i = m; i <= n; i += istep) {
j = i + mmax;
tempr = wr * data[j] - wi * data[j + 1];
tempi = wr * data[j + 1] + wi * data[j];
data[j] = data[i] - tempr;
data[j + 1] = data[i + 1] - tempi;
data[i] += tempr;
data[i + 1] += tempi;
}
wr = (wtemp = wr) * wpr - wi * wpi + wr;
wi = wi * wpr + wtemp * wpi + wi;
}
mmax = istep;
}
}
Apart from a few minor changes, this code appears to be taken out of the 2nd edition of Numerical Recipes in C. The documentation for this function (taken from the book) states:
Replaces data[1..2*nn] by its discrete Fourier transform, if isign is input as 1; or replaces data[1..2*nn] by nn times its inverse discrete Fourier transform, if isign is input as −1.
data is a complex array of length nn or, equivalently, a real array of length 2*nn. nn MUST be an integer power of 2 (this is not checked for!).
This implementation yields correct results, given an input array with 1-based indexing. You can choose to use the same indexing convention by allocating a C array of size 2*nn+1 and filling your array starting at index 1. Alternatively you could pass an array of size 2*nn which has been fill starting at index 0, but calling four1(data-1, nn, isign) (notice the -1 offset on the data array).
I am trying to capture the image data in the onFrameAvailable method from a Google Tango. I am using the Leibniz release. In the header file it is said that the buffer contains HAL_PIXEL_FORMAT_YV12 pixel data. In the release notes they say the buffer contains YUV420SP. But in the documentation it is said the pixels are RGBA8888 format (). I am a little confused and additionally. I don't really get image data but a lot of magenta and green. Right now I am trying to convert from YUV to RGB similar to this one. I guess there is something wrong with the stride, too. Here eís the code of the onFrameAvailable method:
int size = (int)(buffer->width * buffer->height);
for (int i = 0; i < buffer->height; ++i)
{
for (int j = 0; j < buffer->width; ++j)
{
float y = buffer->data[i * buffer->stride + j];
float v = buffer->data[(i / 2) * (buffer->stride / 2) + (j / 2) + size];
float u = buffer->data[(i / 2) * (buffer->stride / 2) + (j / 2) + size + (size / 4)];
const float Umax = 0.436f;
const float Vmax = 0.615f;
y = y / 255.0f;
u = (u / 255.0f - 0.5f) ;
v = (v / 255.0f - 0.5f) ;
TangoData::GetInstance().color_buffer[3*(i*width+j)]=y;
TangoData::GetInstance().color_buffer[3*(i*width+j)+1]=u;
TangoData::GetInstance().color_buffer[3*(i*width+j)+2]=v;
}
}
I am doing the yuv to rgb conversion in the fragment shader.
Has anyone ever obtained an RGB image for the Google Tango Leibniz release? Or had someone similar problems when converting from YUV to RGB?
YUV420SP (aka NV21) is correct for the time being. An explanation is here. In this format you have a width x height array where each element is a Y byte, followed by a width/2 x height/2 array where each element is a V byte and a U byte. Your code is implementing YV21, which has separate arrays for V and U instead of interleaving them in one array.
You mention that you are doing YUV to RGB conversion in a fragment shader. If all you want to do with the camera images is draw then you can use TangoService_connectTextureId() and TangoService_updateTexture() instead of TangoService_connectOnFrameAvailable(). This approach delivers the camera image to you already in an OpenGL texture that gives your fragment shader RGB values without bothering with the pixel format details. You will need to bind to GL_TEXTURE_EXTERNAL_OES (instead of GL_TEXTURE_2D), and your fragment shader would look something like this:
#extension GL_OES_EGL_image_external : require
precision mediump float;
varying vec4 v_t;
uniform samplerExternalOES colorTexture;
void main() {
gl_FragColor = texture2D(colorTexture, v_t.xy);
}
If you really do want to pass YUV data to a fragment shader for some reason, you can do so without preprocessing it into floats. In fact, you don't need to unpack it at all - for NV21 just define a 1-byte texture for Y and a 2-byte texture for VU, and load the data as-is. Your fragment shader will use the same texture coordinates for both.
By the way, if someone experienced problems with capturing the image data on the Leibniz release, too: One of the developers told me that there is a bug concerning the camera and that it should be fixed with the Nash release.
The bug caused my buffer to be null but when I used the Nash update I got data again. However, right now the problem is that the data I am using doesn't make sense. I guess/hope the cause is that the Tablet didn't get the OTA update yet (there can be a gap between the actual release date and the OTA software update).
Just try code following:
//C#
public bool YV12ToPhoto(byte[] data, int width, int height, out Texture2D photo)
{
photo = new Texture2D(width, height);
int uv_buffer_offset = width * height;
for (int i = 0; i < height; i++)
{
for (int j = 0; j < width; j++)
{
int x_index = j;
if (j % 2 != 0)
{
x_index = j - 1;
}
// Get the YUV color for this pixel.
int yValue = data[(i * width) + j];
int uValue = data[uv_buffer_offset + ((i / 2) * width) + x_index + 1];
int vValue = data[uv_buffer_offset + ((i / 2) * width) + x_index];
// Convert the YUV value to RGB.
float r = yValue + (1.370705f * (vValue - 128));
float g = yValue - (0.689001f * (vValue - 128)) - (0.337633f * (uValue - 128));
float b = yValue + (1.732446f * (uValue - 128));
Color co = new Color();
co.b = b < 0 ? 0 : (b > 255 ? 1 : b / 255.0f);
co.g = g < 0 ? 0 : (g > 255 ? 1 : g / 255.0f);
co.r = r < 0 ? 0 : (r > 255 ? 1 : r / 255.0f);
co.a = 1.0f;
photo.SetPixel(width - j - 1, height - i - 1, co);
}
}
return true;
}
I have succeeded.
I am trying to write a method to get the average color of the bottom 30% of an image. I'm trying to do this by going over each pixel individually, getting their colors, summing them op and dividing the result by the amount of pixels I checked. My code is:
int imageWidth = image.getWidth();
int imageHeight = image.getHeight();
int pixel;
int pixelSumRed = 0;
int pixelSumBlue = 0;
int pixelSumGreen = 0;
for (int i = 0; i < 100; i++) {
for (int j = 70; j < 100; j++) {
pixel = image.getPixel((int) Math.round((i/100)*imageWidth), (int) Math.round((j/100)*imageHeight));
pixelSumRed += Color.red(pixel);
pixelSumBlue += Color.blue(pixel);
pixelSumGreen += Color.green(pixel);
Log.d("Checks", "Pixel " + i + ", " + j + " red: " + Color.red(pixel) + ", green: " + Color.green(pixel) + ", blue: " + Color.blue(pixel));
}
}
averagePixelRed = pixelSumRed / 3000;
averagePixelBlue = pixelSumBlue / 3000;
averagePixelGreen = pixelSumGreen / 3000;
Now, I noticed that the same color is printed for each pixel, so for one image I'll get RGB = 210, 44, 70 for each pixel and for another image I'll get RGB = 12, 0, 90 for each pixel. Obviously something is wrong but I can't find what it is. I hope you guys can help me out.
The wonders of integer division! When you divide your i or j by 100, it is done as integer division, and it rounds down - you'll get 0 each time. You can re-order it to do:
(i * imageWidth) / 100
By referring Aegonis's work 1 and work 2, I also got the H.264 stream , but the color is not correct. I am using HTC Butterfly for development. Here is part of my code:
Camera:
parameters.setPreviewSize(width, height);
parameters.setPreviewFormat(ImageFormat.YV12);
parameters.setPreviewFrameRate(frameRate);
MediaCodec:
mediaCodec = MediaCodec.createEncoderByType("video/avc");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 320, 240);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 500000);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 15);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar);
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mediaCodec.start();
When using COLOR_FormatYUV420Planar the error shows "[OMX.qcom.video.encoder.avc] does not support color format 19," so I can only use "COLOR_FormatYUV420SemiPlanar". Does anyone know the reason why no support?
Got it, by using :
int colorFormat = 0;
MediaCodecInfo.CodecCapabilities capabilities = codecInfo.getCapabilitiesForType(mimeType);
for (int i = 0; i < capabilities.colorFormats.length && colorFormat == 0; i++) {
int format = capabilities.colorFormats[i];
Log.e(TAG, "Using color format " + format);
}
we can have color format 21 (COLOR_FormatYUV420SemiPlanar) and 2130708361 (no corresponding format), I think the format will change depends on device.
Then, I tried the color transform provided from the suggestions in work 1 and work 2:
public static byte[] YV12toYUV420PackedSemiPlanar(final byte[] input, final byte[] output, final int width, final int height) {
/*
* COLOR_TI_FormatYUV420PackedSemiPlanar is NV12
* We convert by putting the corresponding U and V bytes together (interleaved).
*/
final int frameSize = width * height;
final int qFrameSize = frameSize/4;
System.arraycopy(input, 0, output, 0, frameSize); // Y
for (int i = 0; i < qFrameSize; i++) {
output[frameSize + i*2] = input[frameSize + i + qFrameSize]; // Cb (U)
output[frameSize + i*2 + 1] = input[frameSize + i]; // Cr (V)
}
return output;
}
public static byte[] YV12toYUV420Planar(byte[] input, byte[] output, int width, int height) {
/*
* COLOR_FormatYUV420Planar is I420 which is like YV12, but with U and V reversed.
* So we just have to reverse U and V.
*/
final int frameSize = width * height;
final int qFrameSize = frameSize/4;
System.arraycopy(input, 0, output, 0, frameSize); // Y
System.arraycopy(input, frameSize, output, frameSize + qFrameSize, qFrameSize); // Cr (V)
System.arraycopy(input, frameSize + qFrameSize, output, frameSize, qFrameSize); // Cb (U)
return output;
}
public static byte[] swapYV12toI420(byte[] yv12bytes, int width, int height) {
byte[] i420bytes = new byte[yv12bytes.length];
for (int i = 0; i < width*height; i++)
i420bytes[i] = yv12bytes[i];
for (int i = width*height; i < width*height + (width/2*height/2); i++)
i420bytes[i] = yv12bytes[i + (width/2*height/2)];
for (int i = width*height + (width/2*height/2); i < width*height + 2*(width/2*height/2); i++)
i420bytes[i] = yv12bytes[i - (width/2*height/2)];
return i420bytes;
}
Obviously, the color transform of YV12toYUV420PackedSemiPlanar performs better than the other two. It is relatively better but still looks different in comparison with the real color. Is there something wrong with my code? Any comment will be appreciated.
Got it, now the color looks good, the test is based on HTC Butterfly.
When set the resolution to 320x240, your color transform should looks like:
System.arraycopy(input, 0, output, 0, frameSize);
for (int i = 0; i < (qFrameSize); i++) {
output[frameSize + i*2] = (input[frameSize + qFrameSize + i - 32 - 320]);
output[frameSize + i*2 + 1] = (input[frameSize + i - 32 - 320]);
}
for resolution 640x480 and above,
System.arraycopy(input, 0, output, 0, frameSize);
for (int i = 0; i < (qFrameSize); i++) {
output[frameSize + i*2] = (input[frameSize + qFrameSize + i]);
output[frameSize + i*2 + 1] = (input[frameSize + i]);
}
For the frame rate issue, we can use the getSupportedPreviewFpsRange() to check the supported frame rate range of our device as:
List<int[]> fpsRange = parameters.getSupportedPreviewFpsRange();
for (int[] temp3 : fpsRange) {
System.out.println(Arrays.toString(temp3));}
And the following setting works correct when play the encoded H.264 ES,
parameters.setPreviewFpsRange(29000, 30000);
//parameters.setPreviewFpsRange(4000,60000);//this one results fast playback when I use the FRONT CAMERA
After reading this discussion it turns out that more generalised way for encoding frames
of various resolutions is to align chroma plane by 2048 bytes before sending frame to the MediaCodec. This is actual for QualComm (OMX.qcom.video.encoder.avc) encoder which I believe HTC Butterfly has, but still does not works well for all resolutions. 720x480 and 176x144 are still have chroma plane misaligned according to the output video. Also, avoid resolutions which sizes can't be divided by 16.
The transformation is pretty simple:
int padding = 0;
if (mediaCodecInfo.getName().contains("OMX.qcom")) {
padding = (width * height) % 2048;
}
byte[] inputFrameBuffer = new byte[frame.length];
byte[] inputFrameBufferWithPadding = new byte[padding + frame.length];
ColorHelper.NV21toNV12(frame, inputFrameBuffer, width, height);
# copy Y plane
System.arraycopy(inputFrameBuffer, 0, inputFrameBufferWithPadding, 0, inputFrameBuffer.length);
int offset = width * height;
# copy U and V planes aligned by <padding> boundary
System.arraycopy(inputFrameBuffer, offset, inputFrameBufferWithPadding, offset + padding, inputFrameBuffer.length - offset);