I trained a model using tensorflow and then converted it to tensorflow-lite format.
Model inference worked absolutely fine on laptop using python.
Then I put the model in Android app and used tensorflowlite interpreter for inference and result was nothing but a full black image.
I ported the code in python to Java as is, still getting this garbage result.
Any idea where I might be going wrong here.
Python Code:
def preprocess(img):
return (img / 255. - 0.5) * 2
def deprocess(img):
return (img + 1) / 2
img_size = 256
frozen_model_filename = os.path.join('model/tflite', 'model.tflite')
image_1 = cv2.resize(imread(image_1), (img_size, img_size))
X_1 = np.expand_dims(preprocess(image_1), 0)
X_1 = X_1.astype(np.float32)
image_2 = cv2.resize(imread(image_2), (img_size, img_size))
X_2 = np.expand_dims(preprocess(image_2), 0)
X_2 = X_2.astype(np.float32)
interpreter = tf.lite.Interpreter(model_path=frozen_model_filename)
interpreter.allocate_tensors()
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
interpreter.set_tensor(input_details[0]['index'], X_1)
interpreter.set_tensor(input_details[1]['index'], X_2)
interpreter.invoke()
Output = interpreter.get_tensor(output_details[0]['index'])
Output = deprocess(Output)
imsave('result_tflite.jpg', Output[0])
Corresponding Java code for Android platform:
private ByteBuffer convertBitmapToByteBuffer(Bitmap bitmap) {
Bitmap resized = Bitmap.createScaledBitmap(bitmap, IMAGE_SIZE, IMAGE_SIZE, false);
ByteBuffer byteBuffer;
if(isQuant) {
byteBuffer = ByteBuffer.allocateDirect(BATCH_SIZE * IMAGE_SIZE * IMAGE_SIZE * PIXEL_SIZE);
} else {
byteBuffer = ByteBuffer.allocateDirect(4 * BATCH_SIZE * IMAGE_SIZE * IMAGE_SIZE * PIXEL_SIZE);
}
byteBuffer.order(ByteOrder.nativeOrder());
int[] intValues = new int[IMAGE_SIZE * IMAGE_SIZE];
resized.getPixels(intValues, 0, resized.getWidth(), 0, 0, resized.getWidth(), resized.getHeight());
int pixel = 0;
byteBuffer.rewind();
for (int i = 0; i < IMAGE_SIZE; ++i) {
for (int j = 0; j < IMAGE_SIZE; ++j) {
final int val = intValues[pixel++];
if(isQuant){
byteBuffer.put((byte) ((val >> 16) & 0xFF));
byteBuffer.put((byte) ((val >> 8) & 0xFF));
byteBuffer.put((byte) (val & 0xFF));
} else {
byteBuffer.putFloat((((val >> 16) & 0xFF) - 0.5f) * 2.0f);
byteBuffer.putFloat((((val >> 8) & 0xFF) - 0.5f) * 2.0f);
byteBuffer.putFloat((((val) & 0xFF ) - 0.5f) * 2.0f);
}
}
}
return byteBuffer;
}
private Bitmap getOutputImage(ByteBuffer output){
output.rewind();
int outputWidth = IMAGE_SIZE;
int outputHeight = IMAGE_SIZE;
Bitmap bitmap = Bitmap.createBitmap(outputWidth, outputHeight, Bitmap.Config.ARGB_8888);
int [] pixels = new int[outputWidth * outputHeight];
for (int i = 0; i < outputWidth * outputHeight; i++) {
int a = 0xFF;
float r = (output.getFloat() + 1) / 2.0f;
float g = (output.getFloat() + 1) / 2.0f;
float b = (output.getFloat() + 1) / 2.0f;
pixels[i] = a << 24 | ((int) r << 16) | ((int) g << 8) | (int) b;
}
bitmap.setPixels(pixels, 0, outputWidth, 0, 0, outputWidth, outputHeight);
return bitmap;
}
private void runInference(){
ByteBuffer byteBufferX1 = convertBitmapToByteBuffer(bitmap_x1);
ByteBuffer byteBufferX2 = convertBitmapToByteBuffer(bitmap_x2);
Object[] inputs = {byteBufferX1, byteBufferX2};
ByteBuffer byteBufferOutput;
if(isQuant) {
byteBufferOutput = ByteBuffer.allocateDirect(BATCH_SIZE * IMAGE_SIZE * IMAGE_SIZE * PIXEL_SIZE);
} else {
byteBufferOutput = ByteBuffer.allocateDirect(4 * BATCH_SIZE * IMAGE_SIZE * IMAGE_SIZE * PIXEL_SIZE);
}
byteBufferOutput.order(ByteOrder.nativeOrder());
byteBufferOutput.rewind();
Map<Integer, Object> outputs = new HashMap<>();
outputs.put(0, byteBufferOutput);
interpreter.runForMultipleInputsOutputs(inputs, outputs);
ByteBuffer out = (ByteBuffer) outputs.get(0);
Bitmap outputBitmap = getOutputImage(out);
// outputBitmap is just a full black image
}
Both Java and Python interpreter are base on C++ implementation so the results should be the same. The error should be in your JAVA code.
Here i think you forgot to multiply and divide to 255.
Related
I'm trying to convert the image data from an Android device from YUV_420_888 to an RGB matrix on the C++ side. On some devices, this is working flawlessly. On a Note 10, the image comes out looking like this:
My guess here is that the stride is causing this issue. How do I remove this extra data and then pass the correct buffer through JNI?
Here is the Java code:
IntBuffer rgb = image.getPlanes()[0].getBuffer().asIntBuffer();
NativeLib.passImageBuffer(rgb);
And here is the C++ code:
cv::Mat outputRGB;
cv::cvtColor(cv::Mat(height+height/2, width, CV_8UC1, inputRGB), outputRGB, CV_YUV2BGR_NV21);
I've tried some different image formats on the C++ side, but they all come back with the same band on the side of the screen.
I've implemented this answer, in order to remove the extra padding, but the image that is passed ends up being completely green. Do some corresponding edits need to be made to the C++ code? I've tried using a 3 channel format, but that crashes at runtime. I'm thinking that since passing the buffer works with the 1 channel matrix on phones that have 8 bits per pixel, that it should be possible to do that with the note 10?
Image.Plane Y = image.getPlanes()[0];
Image.Plane U = image.getPlanes()[1];
Image.Plane V = image.getPlanes()[2];
int[] rgbBytes = new int[image.getHeight()*image.getWidth()*4];
int idx = 0;
ByteBuffer yBuffer = Y.getBuffer();
int yPixelStride = Y.getPixelStride();
int yRowStride = Y.getRowStride();
ByteBuffer uBuffer = U.getBuffer();
int uPixelStride = U.getPixelStride();
int uRowStride = U.getRowStride();
ByteBuffer vBuffer = V.getBuffer();
int vPixelStride = V.getPixelStride();
int vRowStride = V.getRowStride();
ByteBuffer rgbBuffer = ByteBuffer.allocateDirect(rgb.limit());
for (int row = 0; row < image.getHeight(); row++) {
for (int col = 0; col < image.getWidth(); col++) {
int y = yBuffer.get(col*yPixelStride + row*yRowStride) & 0xff;
int u = uBuffer.get(col/2*uPixelStride + row/2*uRowStride) & 0xff;
int v = vBuffer.get(col/2*vPixelStride + row/2*vRowStride) & 0xff;
int y1 = ((19077 << 8) * y) >> 16;
int r = (y1 + (((26149 << 8) * v) >> 16) - 14234) >> 6;
int g = (y1 - (((6419 << 8) * u) >> 16) - (((13320 << 8) * v) >> 16) + 8708) >> 6;
int b = (y1 + (((33050 << 8) * u) >> 16) - 17685) >> 6;
if (r < 0) r = 0;
if (g < 0) g = 0;
if (b < 0) b = 0;
if (r > 255) r = 255;
if (g > 255) g = 255;
if (b > 255) b = 255;
byte pixel = (byte)(0xff000000 + b + 256 * (g + 256 * r));
rgbBuffer.put(pixel);
}
}
Look at this repo
https://github.com/quickbirdstudios/yuvToMat/
It supports different formats (YUV420, NV12) and variety of pixel and row strides.
I have to provide a YUV(NV21) byte array to a recognition solution and I'd like, to reduce processing time, to down scale the preview frame.
From solutions gathered here and there on SO, I manage to convert on a 1:1 ratio and I get recognition hits. But if I'd like to scale the intermediate bitmap down, I get no result. Even if I scale it down to 95% only.
Any help would be appreciated.
Thus, every 400-ish ms I take the preview frame to convert it asynchronously. I convert it to ARGB using RenderScript, scale it down and then convert it back.
// Camera callback
#Override
public void onPreviewFrame(byte[] frame, Camera camera) {
if (camera != null) {
// Debounce
if ((System.currentTimeMillis() - mStart) > 400) {
mStart = System.currentTimeMillis();
Camera.Size size = camera.getParameters().getPreviewSize();
new FrameScaleAsyncTask(frame, size.width, size.height).execute();
}
}
if (mCamera != null) {
mCamera.addCallbackBuffer(mBuffer);
}
}
// In FrameScaleAsyncTask
#Override
protected Void doInBackground(Void... params) {
// Create YUV type for in-allocation
Type yuvType = new Type.Builder(mRenderScript, Element.U8(mRenderScript))
.setX(mFrame.length)
.create();
mAllocationIn = Allocation.createTyped(mRenderScript, yuvType, Allocation.USAGE_SCRIPT);
// Create ARGB-8888 type for out-allocation
Type rgbType = new Type.Builder(mRenderScript, Element.RGBA_8888(mRenderScript))
.setX(mWidth)
.setY(mHeight)
.create();
mAllocationOut = Allocation.createTyped(mRenderScript, rgbType, Allocation.USAGE_SCRIPT);
// Copy frame data into in-allocation
mAllocationIn.copyFrom(mFrame);
// Set script input and fire !
mScript.setInput(mAllocationIn);
mScript.forEach(mAllocationOut);
// Create a bitmap of camera preview size (see camera setup) and copy out-allocation to it
Bitmap bitmap = Bitmap.createBitmap(mWidth, mHeight, Bitmap.Config.ARGB_8888);
mAllocationOut.copyTo(bitmap);
// Scale bitmap down
double scaleRatio = 1;
Bitmap scaledBitmap = Bitmap.createScaledBitmap(
bitmap,
(int) (bitmap.getWidth() * scaleRatio),
(int) (bitmap.getHeight() * scaleRatio),
false
);
bitmap.recycle();
int size = scaledBitmap.getRowBytes() * scaledBitmap.getHeight();
int scaledWidth = scaledBitmap.getWidth();
int scaledHeight = scaledBitmap.getHeight();
int[] pixels = new int[scaledWidth * scaledHeight];
// Put bitmap pixels into an int array
scaledBitmap.getPixels(pixels, 0, scaledWidth, 0, 0, scaledWidth, scaledHeight);
mFrame = new byte[pixels.length * 3 / 2];
ImageHelper.encodeYUV420SPAlt(mFrame, pixels, scaledWidth, scaledHeight);
return null;
}
The RGB to YUV algorithm (see : this answer ):
public static void encodeYUV420SPAlt(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
// NV21 has a plane of Y and interleaved planes of VU each sampled by a factor of 2
// meaning for every 4 Y pixels there are 1 V and 1 U. Note the sampling is every other
// pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[uvIndex++] = (byte) ((V < 0) ? 0 : ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte) ((U < 0) ? 0 : ((U > 255) ? 255 : U));
}
index++;
}
}
}
I finally end up resizing my image (as a OpenCV.Mat) directly in C++. This was way easier and faster.
Size size(correctedWidth, correctedHeight);
Mat dst;
resize(image, dst, size);
I wanna use Sobel operator in my android app. But I don't understand how use one pixel.
int sobel_x[][] = {{-1, 0, 1},
{-2, 0, 2},
{-1, 0, 1}};
int sobel_y[][] = {{-1, -2, -1},
{0, 0, 0},
{1, 2, 1}};
Bitmap source = ImageHelper.GetBitmapGromUri(Path);
int w = source.getWidth();
int h = source.getHeight();
int[] pixels;
pixels = new int[h * w];
source.getPixels(pixels, 0, w, 1, 1, w - 1, h - 1);
for(int i = 0;i < pixels.length;i++){
...
}
I try use get/setPixel. But is very very slow.
Good news and bad news. The following works but...
Android, for some peculiar reason, doesn't allow you to create an 8 bit grey scale image. This means that you have to create a greyscale in ARGB_8888 format. This is probably what was going wrong in your previous version, we read data as bytes when it wasn't.
The code below works and I've only run it on an emulator against your image where it is ridiculously slow (11 seconds). Of course your image is very big but this is still, I would guess, way to slow.
I would strongly suggest considering using OpenCV Java libraries as they are both fast and memory efficient unlike the Android Bitmap class!
public class Sobel {
private static Bitmap toGreyScale( Bitmap source ) {
Bitmap greyScaleBitmap = Bitmap.createBitmap(
source.getWidth(), source.getHeight(),
Bitmap.Config.ARGB_8888);
Canvas c = new Canvas(greyScaleBitmap);
Paint p = new Paint();
ColorMatrix cm = new ColorMatrix();
cm.setSaturation(0);
ColorMatrixColorFilter filter = new ColorMatrixColorFilter(cm);
p.setColorFilter(filter);
c.drawBitmap(source, 0, 0, p);
return greyScaleBitmap;
}
public static void doSobel( Bitmap source) {
Bitmap grey = toGreyScale(source);
int w = grey.getWidth();
int h = grey.getHeight();
// Allocate 4 times as much data as is necessary because Android.
int sz = w * h;
IntBuffer buffer = IntBuffer.allocate( sz );
grey.copyPixelsToBuffer( buffer );
final int[] bitmapData = buffer.array();
int[] output = new int[ w * h ];
for( int y=1; y<h-1; y++ ) {
for( int x=1; x<w-1; x++ ) {
int idx = (y * w + x );
// Apply Sobel filter
int tl = (bitmapData[idx - w - 1]) & 0xFF;
int tr = (bitmapData[idx - w + 1]) & 0xFF;
int l = (bitmapData[idx - 1]) & 0xFF;
int r = (bitmapData[idx + 1]) & 0xFF;
int bl = (bitmapData[idx + w - 1]) & 0xFF;
int br = (bitmapData[idx + w + 1]) & 0xFF;
int sx = (int) ( tr - tl + 2 * ( r - l ) + br - bl );
sx = sx & 0xFF;
// Put back into ARG and B bytes
output[toIdx] = (sx << 24) | ( sx << 16) | (sx << 8) | sx;
}
}
source.copyPixelsFromBuffer( IntBuffer.wrap(output));
}
}
Individual pixel access is definitely not the right way to go!
I assume that you have converted source to grayscale and so it's essentially a byte array. I suggest you use something like the following to extract the data:
int w = source.getWidth();
int h = source.getHeight();
ByteBuffer buffer = ByteBuffer.allocate( w * h );
source.copyPixelsToBuffer( buffer );
final byte[] bitmapData = buffer.array()
This gives you the source data. You can now apply your Sobel filters to it. Note that you want to write the resulting output bytes into a new byte array and then convert that back into an image.
byte[] output = new byte[ w * h ];
for( int y=1; y<h-1; y++ ) {
for( int x=1; x<w-1; x++ ) {
int idx = y * w + x;
// Apply Sobel filter
byte sx = (byte) ((-1 * bitmapData[idx-w-1]) + ( 1 * bitmapData[idx-w+1] ) + (-2 * bitmapData[idx-1]) + ( 2 * bitmapData[idx+1] ) + (-1 * bitmapData[idx+w-1]) + ( 1 * bitmapData[idx+w+1] ) );
output[idx] = sx;
}
}
You can so similarly for the horizontal filter.
Edited to correct type of output from int to byte
My Problem is: I've set up a camera in Android and receive the preview data by using an onPreviewFrame-listener which passes me an byte[] array containing the image data in the default android YUV-format (device does not support R5G6B5-format). Each pixel consists of 12bits which makes the thing a little tricky. Now what I want to do is converting the YUV-data into ARGB-data in order to do image processing with it. This has to be done with renderscript, in order to maintain a high performance.
My idea was to pass two pixels in one element (which would be 24bits = 3 bytes) and then return two ARGB pixels. The problem is, that in Renderscript a u8_3 (a 3dimensional 8bit vector) is stored in 32bit, which means that the last 8 bits are unused. But when copying the image data into the allocation all of the 32bits are used, so the last 8bit get lost. Even if I used a 32bit input data, the last 8bit are useless, because they're only 2/3 of a pixel. When defining an element consisting a 3-byte-array it actually has a real size of 3 bytes. But then the Allocation.copyFrom()-method doesn't fill the in-Allocation with data, argueing it doesn't has the right data type to be filled with a byte[].
The renderscript documentation states, that there is a ScriptIntrinsicYuvToRGB which should do exactly that in API Level 17. But in fact the class doesn't exist. I've downloaded API Level 17 even though it seems not to be downloadable any more. Does anyone have any information about it? Does anyone have ever tried out a ScriptIntrinsic?
So in conclusion my question is: How to convert the camera data into ARGB data fast, hardwareaccelerated?
That's how to do it in Dalvik VM (found the code somewhere online, it works):
#SuppressWarnings("unused")
private void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width, int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
}
I'm sure you will find the LivePreview test application interesting ... it's part of the Android source code in the latest Jelly Bean (MR1). It implements a camera preview and uses ScriptIntrinsicYuvToRgb to convert the preview data with Renderscript. You can browse the source online here:
LivePreview
I was not able to get running ScriptInstrinsicYuvToRgb, so I decided to write my own RS solution.
Here's ready script (named yuv.rs):
#pragma version(1)
#pragma rs java_package_name(com.package.name)
rs_allocation gIn;
int width;
int height;
int frameSize;
void yuvToRgb(const uchar *v_in, uchar4 *v_out, const void *usrData, uint32_t x, uint32_t y) {
uchar yp = rsGetElementAtYuv_uchar_Y(gIn, x, y) & 0xFF;
int index = frameSize + (x & (~1)) + (( y>>1) * width );
int v = (int)( rsGetElementAt_uchar(gIn, index) & 0xFF ) -128;
int u = (int)( rsGetElementAt_uchar(gIn, index+1) & 0xFF ) -128;
int r = (int) (1.164f * yp + 1.596f * v );
int g = (int) (1.164f * yp - 0.813f * v - 0.391f * u);
int b = (int) (1.164f * yp + 2.018f * u );
r = r>255? 255 : r<0 ? 0 : r;
g = g>255? 255 : g<0 ? 0 : g;
b = b>255? 255 : b<0 ? 0 : b;
uchar4 res4;
res4.r = (uchar)r;
res4.g = (uchar)g;
res4.b = (uchar)b;
res4.a = 0xFF;
*v_out = res4;
}
Don't forget to set camera preview format to NV21:
Parameters cameraParameters = camera.getParameters();
cameraParameters.setPreviewFormat(ImageFormat.NV21);
// Other camera init stuff: preview size, framerate, etc.
camera.setParameters(cameraParameters);
Allocations initialization and script usage:
// Somewhere in initialization section
// w and h are variables for selected camera preview size
rs = RenderScript.create(this);
Type.Builder tbIn = new Type.Builder(rs, Element.U8(rs));
tbIn.setX(w);
tbIn.setY(h);
tbIn.setYuvFormat(ImageFormat.NV21);
Type.Builder tbOut = new Type.Builder(rs, Element.RGBA_8888(rs));
tbOut.setX(w);
tbOut.setY(h);
inData = Allocation.createTyped(rs, tbIn.create(), Allocation.MipmapControl.MIPMAP_NONE, Allocation.USAGE_SCRIPT & Allocation.USAGE_SHARED);
outData = Allocation.createTyped(rs, tbOut.create(), Allocation.MipmapControl.MIPMAP_NONE, Allocation.USAGE_SCRIPT & Allocation.USAGE_SHARED);
outputBitmap = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888);
yuvScript = new ScriptC_yuv(rs);
yuvScript.set_gIn(inData);
yuvScript.set_width(w);
yuvScript.set_height(h);
yuvScript.set_frameSize(previewSize);
//.....
Camera callback method:
public void onPreviewFrame(byte[] data, Camera camera) {
// In your camera callback, data
inData.copyFrom(data);
yuvScript.forEach_yuvToRgb(inData, outData);
outData.copyTo(outputBitmap);
// draw your bitmap where you want to
// .....
}
For anyone who didn't know, RenderScript is now in the Android Support Library, including intrinsics.
http://android-developers.blogspot.com.au/2013/09/renderscript-in-android-support-library.html
http://android-developers.blogspot.com.au/2013/08/renderscript-intrinsics.html
We now have the new renderscript-intrinsics-replacement-toolkit to do it. First, build and import the renderscript module to your project and add it as a dependency to your app module. Then, go to Toolkit.kt and add the following:
fun toNv21(image: Image): ByteArray? {
val nv21 = ByteArray((image.width * image.height * 1.5f).toInt())
return if (!nativeYuv420toNv21(
nativeHandle,
image.width,
image.height,
image.planes[0].buffer, // Y buffer
image.planes[1].buffer, // U buffer
image.planes[2].buffer, // V buffer
image.planes[0].pixelStride, // Y pixel stride
image.planes[1].pixelStride, // U/V pixel stride
image.planes[0].rowStride, // Y row stride
image.planes[1].rowStride, // U/V row stride
nv21
)
) {
null
} else nv21
}
private external fun nativeYuv420toNv21(
nativeHandle: Long,
imageWidth: Int,
imageHeight: Int,
yByteBuffer: ByteBuffer,
uByteBuffer: ByteBuffer,
vByteBuffer: ByteBuffer,
yPixelStride: Int,
uvPixelStride: Int,
yRowStride: Int,
uvRowStride: Int,
nv21Output: ByteArray
): Boolean
Now, go to JniEntryPoints.cpp and add the following:
extern "C" JNIEXPORT jboolean JNICALL Java_com_google_android_renderscript_Toolkit_nativeYuv420toNv21(
JNIEnv *env, jobject/*thiz*/, jlong native_handle,
jint image_width, jint image_height, jobject y_byte_buffer,
jobject u_byte_buffer, jobject v_byte_buffer, jint y_pixel_stride,
jint uv_pixel_stride, jint y_row_stride, jint uv_row_stride,
jbyteArray nv21_array) {
auto y_buffer = static_cast<jbyte*>(env->GetDirectBufferAddress(y_byte_buffer));
auto u_buffer = static_cast<jbyte*>(env->GetDirectBufferAddress(u_byte_buffer));
auto v_buffer = static_cast<jbyte*>(env->GetDirectBufferAddress(v_byte_buffer));
jbyte* nv21 = env->GetByteArrayElements(nv21_array, nullptr);
if (nv21 == nullptr || y_buffer == nullptr || u_buffer == nullptr
|| v_buffer == nullptr) {
// Log this.
return false;
}
RenderScriptToolkit* toolkit = reinterpret_cast<RenderScriptToolkit*>(native_handle);
toolkit->yuv420toNv21(image_width, image_height, y_buffer, u_buffer, v_buffer,
y_pixel_stride, uv_pixel_stride, y_row_stride, uv_row_stride,
nv21);
env->ReleaseByteArrayElements(nv21_array, nv21, 0);
return true;
}
Go to YuvToRgb.cpp and add the following:
void RenderScriptToolkit::yuv420toNv21(int image_width, int image_height, const int8_t* y_buffer,
const int8_t* u_buffer, const int8_t* v_buffer, int y_pixel_stride,
int uv_pixel_stride, int y_row_stride, int uv_row_stride,
int8_t *nv21) {
// Copy Y channel.
for(int y = 0; y < image_height; ++y) {
int destOffset = image_width * y;
int yOffset = y * y_row_stride;
memcpy(nv21 + destOffset, y_buffer + yOffset, image_width);
}
if (v_buffer - u_buffer == sizeof(int8_t)) {
// format = nv21
// TODO: If the format is VUVUVU & pixel stride == 1 we can simply the copy
// with memcpy. In Android Camera2 I have mostly come across UVUVUV packaging
// though.
}
// Copy UV Channel.
int idUV = image_width * image_height;
int uv_width = image_width / 2;
int uv_height = image_height / 2;
for(int y = 0; y < uv_height; ++y) {
int uvOffset = y * uv_row_stride;
for (int x = 0; x < uv_width; ++x) {
int bufferIndex = uvOffset + (x * uv_pixel_stride);
// V channel.
nv21[idUV++] = v_buffer[bufferIndex];
// U channel.
nv21[idUV++] = u_buffer[bufferIndex];
}
}
}
Finally, go to RenderscriptToolkit.h and add the following:
/**
* https://blog.minhazav.dev/how-to-use-renderscript-to-convert-YUV_420_888-yuv-image-to-bitmap/#tobitmapimage-image-method
* #param image_width width of the image you want to convert to byte array
* #param image_height height of the image you want to convert to byte array
* #param y_buffer Y buffer
* #param u_buffer U buffer
* #param v_buffer V buffer
* #param y_pixel_stride Y pixel stride
* #param uv_pixel_stride UV pixel stride
* #param y_row_stride Y row stride
* #param uv_row_stride UV row stride
* #param nv21 the output byte array
*/
void yuv420toNv21(int image_width, int image_height, const int8_t* y_buffer,
const int8_t* u_buffer, const int8_t* v_buffer, int y_pixel_stride,
int uv_pixel_stride, int y_row_stride, int uv_row_stride,
int8_t *nv21);
You are now ready to harness the full power of renderscript. Below, I am providing an example with the ARCore Camera Image object (replace the first line with whatever code gives you your camera image):
val cameraImage = arFrame.frame.acquireCameraImage()
val width = cameraImage.width
val height = cameraImage.height
val byteArray = Toolkit.toNv21(cameraImage)
byteArray?.let {
Toolkit.yuvToRgbBitmap(
byteArray,
width,
height,
YuvFormat.NV21
).let { bitmap ->
saveBitmapToDevice(
name,
session,
bitmap,
context
)}}
In our application, we need to transfer video, we are using Camera class to capture the buffer and send to destination,
I have set format is YV12 as a Camera parameter to receive the buffer,
for the 500X300 buffer, we receive buffer of 230400 bytes,
i want to know , is this expected buffer size ?
I believe the size would be
Y Plane = width * height = 500X300 = 150000
U Plane = width/2 * height/2 = = 37500
V Plane = width/2 * height/2 = = 37500
========
225000
========
Can anyone explain me, if i need to get stride values of each component, how can i get that
Is there any way to get it ?
I can show you how you can get int rgb[] from this:
public int[] decodeYUV420SP(byte[] yuv420sp, int width, int height) {
final int frameSize = width * height;
int rgb[] = new int[width * height];
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000)
| ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
return rgb;
}
I guess Android document is already explained it:
http://developer.android.com/reference/android/graphics/ImageFormat.html#YV12
I think this is simple.
chekout YUVImage class from android. You can construct an YUV Image from byte[]data coming from camera preview.
You can write like this:
//width and height you get it from camera properties, image width and height of camera preview
YuvImage image=new YuvImage(data, ImageFormat.NV21, int width, int height, null);
byte[] newData = image.getYuvData();
//or if you want int format = image.getYuvFormat();
It's a quite old question, but I've struggled with the same issue for a few days. So I decided to write some comments to help others.
YV12 described in the Android developer site(here) seems not a kind of YV12 but IMC1. The page says that both of the y-stride and the uv-stride should be aligned in 16bytes.
And also this page says that:
For YV12, the image buffer that is received is not necessarily tightly
packed, as there may be padding at the end of each row of pixel data,
as described in YV12.
Based on the above comments, I calculated it using python command line:
>>> w = 500
>>> h = 300
>>> y_stride = (500 + 15) / 16 * 16.0
>>> y_stride
512.0
>>> y_size = y_stride * h
>>> y_size
153600.0
>>> uv_stride = (500 / 2 + 15) / 16 * 16.0
>>> uv_stride
256.0
>>> u_size = uv_stride * h / 2
>>> v_size = uv_stride * h / 2
>>> size = y_size + u_size + v_size
>>> size
230400.0