I'm trying to implement some computer vision in an android app.
I have opencv integrated and I'm writing native c++ code for it that's called using JNI. That all seems to be working. My problem is that when executing the computer vision code the following line crashes the app without any error.
detector->detectAndCompute(usr_img,usr_mask,usr_keypoints,usr_descriptors);
If I use the orb detector, instead of sift it does work. On my physical device it then crashes on knnMatch. Whereas on an emulated Pixel 5 it completes correctly. Maybe it has something to do with my opencv and android versions?
Here's the full computer vision code:
void process_image(char* in_filepath,char* out_filepath){
Mat usr_img = imread(in_filepath); //read images from the disk
Mat ref_img = imread("redacted");
Mat overlay_img = imread("redacted");
Mat out_img;//make a copy for output
usr_img.copyTo(out_img);
//Set up feature detector
Ptr<SIFT> detector = SIFT::create();
//Ptr<ORB> detector = ORB::create(); //detectAndCompute works if I use this instead
//Set up feature matcher
Ptr<BFMatcher> matcher = BFMatcher::create(NORM_HAMMING,true);
//generate mask for ref image (so features are not created from the background)
Mat ref_mask; //defines parts of the ref image that will be searched for features.
inRange(ref_img,Scalar(0.0,0.0,252.0),Scalar(2.0,2.0,255.0),ref_mask);
bitwise_not(ref_mask,ref_mask);//invert the mask
//and an all white mask for the usr image
Mat usr_mask = Mat(usr_img.cols,usr_img.rows, CV_8UC1, Scalar(255.0));
//detect keypoints
std::vector<KeyPoint> ref_keypoints, usr_keypoints;
Mat ref_descriptors, usr_descriptors;
detector->detectAndCompute(ref_img,ref_mask,ref_keypoints,ref_descriptors);
detector->detectAndCompute(usr_img,usr_mask,usr_keypoints,usr_descriptors);
//match descriptors between images, each match is a vector of matches by decreasing "distance"
std::vector<std::vector<DMatch>> matches;
matcher->knnMatch(usr_descriptors,ref_descriptors,matches,2);
//throw out bad matches
std::vector<DMatch> good_matches;
for(uint32_t i = 0; i < matches.size(); i++){
//consider it a good match if the next best match is 33% worse
if(matches[i][0].distance*1.33 < matches[i][1].distance){
good_matches.push_back(matches[i][0]);
}
}
//visualize the matches for debugging purposes
Mat draw_match_img;
drawMatches(usr_img,usr_keypoints,ref_img,ref_keypoints,good_matches,draw_match_img);
imwrite("redacted",draw_match_img);
}
My opencv version is 4.5.4
My android version is 9 on the physical phone, and 11, api 30 on the emulated pixel 5
I found the problem.
My images were 4000x3000px and approx 3000x1600. Scaling both of the images down by a factor of 2 causes everything to work properly.
I added a resize after each imread like this:
resize(x_img,x_img,Size(),0.5,0.5,INTER_CUBIC);
What this tells me is that SIFT in opencv 4.5.4 has a image size limit above which execution will crash without an error message. ..annoying.
It also explains why some of the detectors worked and some did not, and it even seemed to vary when I ran it on a real device vs an emulated one.
Related
it's the first time for me that I ask help here. I will try to be as precise as possible in my question.
I am trying to develop a shape detection app for Android.
I first identified the algorithm which works for my case playing with Python. Basically for each frame I do this:
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, lower_color, upper_color)
contours, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
for cnt in contours:
#here I filter my results
by this algorithm I am able to run the analysis realtime on videos having a frame rate of 120fps.
So I tryied to implement the same algorithm on Android Studio, doing the following for each Frame:
Imgproc.cvtColor(frameInput, tempFrame, Imgproc.COLOR_BGR2HSV);
Core.inRange(tempFrame,lowColorRoi,highColorRoi,tempFrame);
List<MatOfPoint> contours1 = new ArrayList<MatOfPoint>();
Imgproc.findContours(tempFrame /*.clone()*/, contours1, new Mat(), Imgproc.RETR_TREE, Imgproc.CHAIN_APPROX_SIMPLE);
for(MatOfPoint c : contours1){
//here I filter my results
}
and I see that only the findContour function takes 5-600ms to be performed at each iteration (I noticed that it takes also more using tempFrame.clone()), allowing more or less to run the analysis with only 2fps.
This speed is not acceptable at all of course. Do you have any suggestion about how to improve this speed? 30-40fps would be already a good target for me.
I will really appreciate any help from you all. Many thanks in advance.
I would suggest trying to do your shape analysis on a lower resolution version of the image, if that is acceptable. I often see directly proportional timing with number of pixels of the image and the number of channels of the image - so if you can halve the width and height it could be a 4 times performance improvement. If that works, likely the first thing to do is a resize, then all subsequent calls have a smaller burden.
Next, be careful using OpenCV in Java/Kotlin because there is a definite cost to marshalling over the JNI interface. You could write the majority of your code in native C++, and then make just a single call across JNI to a C++ function that handles all of the shape analysis at once.
I'm currently working on an app in C++ using the Android ndk, and I need to create a sampler to access the camera output image.
I have done this using the AIMAGE_FORMAT_YUV_420_888, and using the VkSamplerYcbcrConversion for accessing the image in the hardware buffer. I do the yuv -> rgb conversion in a shader, and it all looks good on my phone.
I have since discovered that this doesn't work on Samsung phones, in my case specifically the Samsung Galaxy S10/S10+.
The reason is that when I set up an image reader with the AIMAGE_FORMAT_YUV_420_888 I get a camera error using Samsung. On my OnePlus and on another phone I tried the pipeline worked entirely as expected. I created a very simple test setup to even try to open the camera with that image format in the ImageReader on Samsung S10 and got the error, but when I changed the ImageReader format to AIMAGE_FORMAT_JPEG the error went away and the camera seemed to start as expected.
AImageReader* SimpleCamera::CreateJpegReader()
{
AImageReader* reader = nullptr;
// media_status_t status = AImageReader_new(640, 480, AIMAGE_FORMAT_JPEG,
//AIMAGE_FORMAT_RGBA_8888
//media_status_t status = AImageReader_new(640, 480, AIMAGE_FORMAT_RGB_565,4, &reader);
media_status_t status = AImageReader_newWithUsage(640, 480,
//AIMAGE_FORMAT_RGBA_8888,
//AIMAGE_FORMAT_RGB_565,
//AIMAGE_FORMAT_RGB_888,
AIMAGE_FORMAT_JPEG,
AHARDWAREBUFFER_USAGE_GPU_SAMPLED_IMAGE | AHARDWAREBUFFER_USAGE_CPU_READ_RARELY,
4, &reader);
if (status != AMEDIA_OK) {
LOGE("Couldn't create new image reader");
return nullptr;
}
AImageReader_ImageListener listener{
.context = nullptr,
.onImageAvailable = imageCallback1,
};
AImageReader_setImageListener(reader, &listener);
return reader;
}
None of the other formats are guaranteed to be supported except AIMAGE_FORMAT_JPEG, but this format doesn't seem to work with the VkSamplerYcbcrConversion because the image layout is different.
Has anyone come up against this issue before? And if so how did you resolve it?
At a high level th goal is: In C++, get the image out of the camera2 api and onto a VkImage. If anyone knows an alternative way of doing that, I'm also all ears.
Try to use ImageFormat.PRIVATE with USAGE_GPU_SAMPLED_IMAGE flag. This used to work fine on the mentioned Samsung devices in particular.
Please make sure to read Vulkan specification, as there are quite a few android-specific and VkSamplerYcbcrConversion requirements.
I can also recommend to take a look at this great project which uses android camera2 api and vulkan.
i have made a 360 image viewer on unity and i am changing image texture dynamically using c# script , so its work fine on PC unity but when i run it on android device it say error below:
OPENGL NATIVE PLUG-IN ERROR: GL_INVALID_FRAMEBUFFER_OPERATION: Framebuffer is not complete or incompatible with command
My code below
IEnumerator registerFunc(WWW www)
{
yield return www;
if (www.error == null)
//if(true)
{
Debug.Log("OK - CountTime");
texturas = www.texture;
www.Dispose();
www = null;
sphereMS.material.mainTexture = texturas;// here i am getting error
}
else
{
Debug.Log("ERROR");
}
}
So any one can help me out in this how i can solve it.Thanks!
resolution is 9999x2429 and sphereMS is public variable to which i have
assigned sphere using drag and drop
That's the problem. 9999 is really big. The limit on most Android devices is about 2048. Anything above this is a problem unless you are running this on very high end and expensive Android device.
One thing to try is replace
texturas = www.texture;
with
texturas = new Texture2D(4, 4);
texturas.LoadImage(www.bytes);
The LoadImage function is used to load jpg/png image into a Texture. It might fail too due to the size of the image.
If LoadImage fails too then you should have different Texture resolution for each platform on your server. The Android platform should request for a lower resolution version of your Textures. Just keep reducing the Textures resolution until it stops crashing. 2048 resolution should be fine.
I suggest you read this post about texture size limit in Unity.
EDIT:
If you have changed the resolution to below 2048 and the problem is still there then this is a bug.
Install Unity 2017.2.0p1 or 2017.3.0b3 to get the fix. Few people experience this with the google-cardboard plugin.
Mostly it happens on Android devices using Mali GPU, even Asphalt 8 couldn't run on these devices (at least old ones).
I tried on Huawei Mate 10 and Samsung Galaxy A50s, as both are using Mali-G72 MP12 and MP3 respectively so getting the same error and the image gets black/blank (couldn't render).
While same works fine on other Android and iOS devices
use parametre
-disable-gpu
or
app.disableHardwareAcceleration()
look here
enter link description here
I am working on a project in android in which i am using OpenCV to detect faces from all the images which are in the gallery. The process of getting faces from the images is performing in the service. Service continuously working till all the images are processed. It is storing the detected faces in the internal storage and also showing in the grid view if activity is opened.
My code is:
CascadeClassifier mJavaDetector=null;
public void getFaces()
{
for (int i=0 ; i<size ; i++)
{
File file=new File(urls.get(i));
imagepath=urls.get(i);
defaultBitmap=BitmapFactory.decodeFile(file, bitmapFatoryOptions);
mJavaDetector = new CascadeClassifier(FaceDetector.class.getResource("lbpcascade_frontalface").getPath());
Mat image = new Mat (defaultBitmap.getWidth(), defaultBitmap.getHeight(), CvType.CV_8UC1);
Utils.bitmapToMat(defaultBitmap,image);
MatOfRect faceDetections = new MatOfRect();
try
{
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));
}
catch(Exception e)
{
e.printStackTrace();
}
if(faceDetections.toArray().length>0)
{
}
}
}
Everything is fine but it is detection faces very slow. The performance is very slow. When i debug the code then i found the line which is taking time is:
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));
I have checked multiple post for this problem but i didn't get any solution.
Please tell me what should i do to solve this problem.
Any help would be greatly appreciated. Thank you.
You should pay attention to the parameters of detectMultiScale():
scaleFactor – Parameter specifying how much the image size is reduced at each image scale. This parameter is used to create a scale pyramid. It is necessary because the model has a fixed size during training. Without pyramid the only size to detect would be this fix one (which can be read from the XML also). However the face detection can be scale-invariant by using multi-scale representation i.e., detecting large and small faces using the same detection window.
scaleFactor depends on the size of your trained detector, but in fact, you need to set it as high as possible while still getting "good" results, so this should be determined empirically.
Your 1.1 value can be a good value for this purpose. It means, a relative small step is used for resizing (reduce size by 10%), you increase the chance of a matching size with the model for detection is found. If your trained detector has the size 10x10 then you can detect faces with size 11x11, 12x12 and so on. But in fact a factor of 1.1 requires roughly double the # of layers in the pyramid (and 2x computation time) than 1.2 does.
minNeighbors – Parameter specifying how many neighbours each candidate rectangle should have to retain it.
Cascade classifier works with a sliding window approach. By applying this approach, you slide a window through over the image than you resize it and search again until you can not resize it further. In every iteration the true outputs (of cascade classifier) are stored but unfortunately it actually detects many false positives. And to eliminate false positives and get the proper face rectangle out of detections, neighbourhood approach is applied. 3-6 is a good value for it. If the value is too high then you can lose true positives too.
minSize – Regarding to the sliding window approach of minNeighbors, this is the smallest window that cascade can detect. Objects smaller than that are ignored. Usually cv::Size(20, 20) are enough for face detections.
maxSize – Maximum possible object size. Objects bigger than that are ignored.
Finally you can try different classifiers based on different features (such as Haar, LBP, HoG). Usually, LBP classifiers are a few times faster than Haar's, but also less accurate.
And it is also strongly recommended to look over these questions:
Recommended values for OpenCV detectMultiScale() parameters
OpenCV detectMultiScale() minNeighbors parameter
Instead reading images as Bitmap and then converting them to Mat via using Utils.bitmapToMat(defaultBitmap,image) you can directly use Mat image = Highgui.imread(imagepath); You can check here for imread() function.
Also, below line takes too much time because the detector is looking for faces with at least having Size(20, 20) which is pretty small. Check this video for visualization of face detection using OpenCV.
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));
My Android application uses javaCV and calls detectMultiScale() function with LBP cascade to detect faces. It works completely fine on my emulator. However, when I tried to test it on my HTC Incredible S, it returns 0, could not detect any face! Could anyone show me some hints why it does not work? Many thanks for your help!!!
Here is my code for face detection:
CASCADE_FILE = working_Dir.getAbsolutePath() + "/lbpcascade_frontalface.xml";
public static CvRect getFaceWithLBP(IplImage grayFaceImg)
{
CascadeClassifier cascade = new CascadeClassifier(CASCADE_FILE);
CvRect facesdetection = new CvRect(null);
cascade.detectMultiScale(grayFaceImg, facesdetection, 1.1, 2, CV_HAAR_FIND_BIGGEST_OBJECT | CV_HAAR_DO_ROUGH_SEARCH,
new CvSize(), new CvSize(grayFaceImg.width(), grayFaceImg.height()));
return facesdetection;
}
Just a note, as per the OpenCV documentation, the flags (such as CV_HAAR_FIND_BIGGEST_OBJECT | CV_HAAR_DO_ROUGH_SEARCH) can not be used with new cascades (like LBP ones).
void CascadeClassifier::detectMultiScale(const Mat& image, vector& objects, double scaleFactor=1.1, int minNeighbors=3, int flags=0, Size minSize=Size(), Size maxSize=Size())
Parameters:
cascade – Haar classifier cascade (OpenCV 1.x API only). It can be loaded from XML or YAML file using Load(). When the cascade is not needed anymore, release it using cvReleaseHaarClassifierCascade(&cascade).
image – Matrix of the type CV_8U containing an image where objects are detected.
objects – Vector of rectangles where each rectangle contains the detected object.
scaleFactor – Parameter specifying how much the image size is reduced at each image scale.
minNeighbors – Parameter specifying how many neighbors each candidate rectangle should have to retain it.
flags – Parameter with the same meaning for an old cascade as in the function cvHaarDetectObjects. It is not used for a new cascade.
minSize – Minimum possible object size. Objects smaller than that are ignored.
maxSize – Maximum possible object size. Objects larger than that are ignored.