Crash Android Program On OpenCV Core.DCT() Method - android

I want to write an android program to compute dct of the input image, and I am using opencv android framework. first of all I convert input image to grayscale and then i want to compute dct using Core.dct() with 16*16 block. this is the dct computation part :
int rownum = 1;
for (int i = 1; i <= M - B + 1; i++) {
for (int j = 1; j <= N - B + 1; j++) {
Mat SubImg = GrayImage.submat(new Range(i, i+B-1), new Range(j, j+B-1));
Mat Block_DCT = Mat.zeros(16, 16, SubImg.type());
Mat Block_DCT_Quantized = Mat.zeros(16, 16, Block_DCT.type());
Core.dct(SubImg, Block_DCT);
Core.divide(Block_DCT, SQ16, Block_DCT_Quantized);
Mat row = Mat.zeros(1, B*B, Block_DCT_Quantized.type());
Block_DCT_Quantized.reshape(1, B*B);
rownum=rownum+1;
}
}
I debug the code and figure out that the application crash on the line Core.dct() !!!
i don't know what is the problem, but I suppose it's because of input type ...
can any body help me ? what should I do ?
Update :
I found the solution, the problem was because of size and type of Mat objects, actually Core.dct() method just work with this type Mat: cv_64fc1. I use SubImg.convertTo() function to change the type and correct the size of block, so it worked...
thanks to Rui Marques for answering...

dct needs float input.
you have to convert your image to CvType.CV_32F or CvType.CV_64F before applying dct / dft

Related

Run inference with an openCV image

I have an Android Project with OpenCV4.0.1 and TFLite installed.
And I want to make an inference with a pretrained MobileNetV2 of an cv::Mat which I extracted and cropped from a CameraBridgeViewBase (Android style).
But it's kinda difficult.
I followed this example.
That does the inference about a ByteBuffer variable called "imgData" (line 71, class: org.tensorflow.lite.examples.classification.tflite.Classifier)
That imgData looks been filled on the method called "convertBitmapToByteBuffer" from the same class (line 185), adding pixel by pixel form a bitmap that looks to be cropped little before.
private int[] intValues = new int[224 * 224];
Mat _croppedFace = new Mat() // Cropped image from CvCameraViewFrame.rgba() method.
float[][] outputVal = new float[1][1]; // Output value from my MobileNetV2 // trained model (i've changed the output on training, tested on python)
// Following: https://stackoverflow.com/questions/13134682/convert-mat-to-bitmap-opencv-for-android
Bitmap bitmap = Bitmap.createBitmap(_croppedFace.cols(), _croppedFace.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(_croppedFace, bitmap);
convertBitmapToByteBuffer(bitmap); // This call should be used as the example one.
// runInference();
_tflite.run(imgData, outputVal);
But, it looks that the input_shape of my NN is not correct, but I'm following the MobileNet example because my NN it's a MobileNetV2.
I've solved the error, but I'm sure that it isn't the best way to do it.
Keras MobilenetV2 input_shape is: (nBatches, 224, 224, nChannels).
I just want to predict a single image, so, nBaches == 1, and I'm working on RGB mode, so nChannels == 3
// Nasty nasty, but works. nBatches == 2? -- _cropped.shape() == (244, 244), 3 channels.
float [][][][] _inputValue = new float[2][_cropped.cols()][_cropped.rows()][3];
// Fill the _inputValue
for(int i = 0; i < _croppedFace.cols(); ++i)
for (int j = 0; j < _croppedFace.rows(); ++j)
for(int z = 0; z < 3; ++z)
_inputValue [0][i][j][z] = (float) _croppedFace.get(i, j)[z] / 255; // DL works better with 0:1 values.
/*
Output val, has this shape, but I don't really know why.
I'm sure that one's of that 2's is for nClasses (I'm working with 2 classes)
But I don't really know why it's using the other one.
*/
float[][] outputVal = new float[2][2];
// Tensorflow lite interpreter
_tflite.run(_inputValue , outputVal);
On python has the same shape:
Python prediction:
[[XXXXXX, YYYYY]] <- Sure for the last layer that I made, this is just a prototype NN.
Hope some one got help, and also that someone can improve the answer because this is not very optimized.

Unity native OpenGL texture displayed four times

I'm currently facing a problem I simply don't understand.
I employ ARCore for an inside out tracking task. Since I need to do some additional image processing I use Unitys capability to load a native c++ plugin. At the very end of each frame I pass the image in YUV_420_888 format as raw byte array to my native plugin.
A texture handle is created right at the beginning of the components initialization:
private void CreateTextureAndPassToPlugin()
{
Texture2D tex = new Texture2D(640, 480, TextureFormat.RGBA32, false);
tex.filterMode = FilterMode.Point;
tex.Apply();
debug_screen_.GetComponent<Renderer>().material.mainTexture = tex;
// Pass texture pointer to the plugin
SetTextureFromUnity(tex.GetNativeTexturePtr(), tex.width, tex.height);
}
Since I only need the grayscale image I basically ignore the UV part of the image and only use the y coordinates as displayed in the following:
uchar *p_out;
int channels = 4;
for (int r = 0; r < image_matrix->rows; r++) {
p_out = image_matrix->ptr<uchar>(r);
for (int c = 0; c < image_matrix->cols * channels; c++) {
unsigned int idx = r * y_row_stride + c;
p_out[c] = static_cast<uchar>(image_data[idx]);
p_out[c + 1] = static_cast<uchar>(image_data[idx]);
p_out[c + 2] = static_cast<uchar>(image_data[idx]);
p_out[c + 3] = static_cast<uchar>(255);
}
}
then each frame the image data is put into a GL texture:
GLuint gltex = (GLuint)(size_t)(g_TextureHandle);
glBindTexture(GL_TEXTURE_2D, gltex);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, 640, 480, GL_RGBA, GL_UNSIGNED_BYTE, current_image.data);
I know that I use way too much memory by creating and passing the texture as RGBA but since GL_R8 is not supported by OpenGL ES3 and GL_ALPHA always lead to internal OpenGL errors I just pass the greyscale value to each color component.
However in the end the texture is rendered as can be seen in the following image:
At first I thought, that the reason for this may lie in the other channels having the same values, however setting all other channels than the first one to any value does not have any impact.
Am I missing something OpenGL texture creation wise?
YUV_420_888 is a multiplane texture, where the luminance plane only contains a single channel per pixel.
for (int c = 0; c < image_matrix->cols * channels; c++) {
unsigned int idx = r * y_row_stride + c;
Your loop bounds assume c is in multiple of 4 channels, which is right for the output surface, but you then use it also when computing the input surface index. The input surface plane you are using only contains one channel, so idx is wrong.
In general you are also over writing the same memory multiple times - the loop increments c by one each iteration but you then write to c, c+1, c+2, and c+3 so overwrite three of the values you wrote last time.
Shorter answer - your OpenGL ES code is fine, but I think you're filling the texture with bad data.
Untested, but I think you need:
for (int c = 0; c < image_matrix->cols * channels; c += channels) {
unsigned int idx = (r * y_row_stride) + (c / channels);

Android YUV to grayscale performance optimization

I'm trying to convert an YUV image to grayscale, so basically I just need the Y values.
To do so I wrote this little piece of code (with frame being the YUV image):
imageConversionTime = System.currentTimeMillis();
size = frame.getSize();
byte nv21ByteArray[] = frame.getImage();
int lol;
for (int i = 0; i < size.width; i++) {
for (int j = 0; j < size.height; j++) {
lol = size.width*j + i;
yMatrix.put(j, i, nv21ByteArray[lol]);
}
}
bitmap = Bitmap.createBitmap(size.width, size.height, Bitmap.Config.ARGB_8888);
Utils.matToBitmap(yMatrix, bitmap);
imageConversionTime = System.currentTimeMillis() - imageConversionTime;
However, this takes about 13500 ms. I need it to be A LOT faster (on my computer it takes 8.5 ms in python) (I work on a Motorola Moto E 4G 2nd generation, not super powerful but it should be enough for converting images right?).
Any suggestions?
Thanks in advance.
First of all I would assign size.width and size.height to a variable. I don't think the compiler will optimize this by default, but I am not sure about this.
Furthermore Create a byte[] representing the result instead of using a Matrix.
Then you could do something like this:
int[] grayScalePixels = new int[size.width * size.height];
int cntPixels = 0;
In your inner loop set
grayScalePixels[cntPixels] = nv21ByteArray[lol];
cntPixels++;
To get your final image do the following:
Bitmap grayScaleBitmap = Bitmap.createBitmap(grayScalePixels, size.width, size.height, Bitmap.Config.ARGB_8888);
Hope it works properly (I have not tested it, however at least the shown principle should be applicable -> relying on a byte[] instead of Matrix)
Probably 2 years too late but anyways ;)
To convert to gray scale, all you need to do is set the u/v values to 128 and leave the y values as is. Note that this code is for YUY2 format. You can refer to this document for other formats.
private void convertToBW(byte[] ptrIn, String filePath) {
// change all u and v values to 127 (cause 128 will cause byte overflow)
byte[] ptrOut = Arrays.copyOf(ptrIn, ptrIn.length);
for (int i = 0, ptrInLength = ptrOut.length; i < ptrInLength; i++) {
if (i % 2 != 0) {
ptrOut[i] = (byte) 127;
}
}
convertToJpeg(ptrOut, filePath);
}
For NV21/NV12, I think the loop would change to:
for (int i = ptrOut.length/2, ptrInLength = ptrOut.length; i < ptrInLength; i++) {}
Note: (didn't try this myself)
Also I would suggest to profile your utils method and createBitmap functions separately.

Having problems using setPixel() method

Can someone explain me the way setPixel() method works in Android? I am trying to replace some pixels on a bitmap. I extract them by using getPixel() method and their individual colors, eg.green = Color.green(a[i][j]);, but I cannot set them new values back, so as to show the processed image onscreen.
Edit: This is where some of the processing occurs. I try to algorithmically convert to grayscale
for (int i = 0; i < grayWidth; i++) {
for (int j = 0; j < grayHeight; j++) {
a[i][j] = myImage.getPixel(i, j);
red = Color.red(a[i][j]);
green = Color.green(a[i][j]);
blue = Color.blue(a[i][j]);
gray = (red + green + blue) / 3;
a[i][j] = gray;
}
}
and then replace pixels:
for (int m = 0; m < grayHeight; m++) {
for (int n = 0; n < grayWidth; n++) {
grayScale.setPixel(m, n, a[m][n]);
}
}
and finally show it on-screen
imageView.setImageBitmap(grayScale);
Sorry for not explaining it thoroughly in the first place/
Why posts questions without any code? Put yourself in our position, how can we help you, if we do not know what you are trying to do? You're only presenting your problem in English language, which does not equal to programming language in most cases. We want to see the latter, supported with english description of what you are doing and what's going on.
Based on provided information and Android Dev Page for Bitmap, I can assume that your BitMap image might not be mutable. This would throw then IllegalStateException, but without seeing your LogCat / Code, I cannot be sure, whether this is the case.
If the BitMap indeed is immutable, then you can try and look at converting immutable bitmap to mutable and try again.

extract a vector of pixels from a frame using OpenCV?

I'am actually working in a project on android in which I want to extract some specific pixel values (according to a condition, using the threshold method , I have set some pixels to a certain values and all the others to zero) I want to extract all these values different from zero in a vector. I will use this vector of the chosen pixels to do some operation ( the mean value for exemple ) , Is there a method in OpenCV that can help me doing this ?
Thank you :)
I don't know such function but actually it's not hard to implement it (c++):
//'in' should be CV_8UC3
vector<int>& getNonZero(const Mat& in)
{
//get size of result vector
//this is all non-zero pixels:
int count = countNonZero(in);
vector<int>& result(count);
int k = 0;
for (int r = 0; r < in.rows; ++r)
{
for (int c = 0; c < in.cols; ++c)
{
if (in.at<int>(r, c))
{
result[k++] = in.at<int>(r, c);
}
}
}
return result;
}
And also many OpenCV functions have InputArray mask parameter so use it! For example:
void meanStdDev(InputArray src, OutputArray mean, OutputArray stddev, InputArray mask=noArray())

Categories

Resources