I'm developing an Android application using Opencv to perform some heavy image processing including detecting the largest contour, crop the detected contour, apply segmentation logic, and match similarities of each segmented contour with a reference object array.
I'm done with the processing logic in real time with an fps of 3 and the treatment time is 0.4 second as an average which is good in my case.
The problem is that the project will be used in an industry, and I want to start processing the frame only when the product is in the camera visual field.
I've done some sort of motion detection to detect if there is some contour moving and then start the algorithm but the industry machine carpet is also moving so this approach won't work.
Here is the code for motion detection part :
#Override
public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
long e1 = Core.getTickCount();
contours.clear();
//gray frame because it requires less resource to process
mGray = inputFrame.gray();
//this function converts the gray frame into the correct RGB format for the BackgroundSubtractorMOG apply function
Imgproc.cvtColor(mGray, mRgb, Imgproc.COLOR_GRAY2RGB);
//apply detects objects moving and produces a foreground mask
//the lRate updates dynamically dependent upon seekbar changes
sub.apply(mRgb, mFGMask, lRate);
//erode and dilate are used to remove noise from the foreground mask
Imgproc.erode(mFGMask, mFGMask, new Mat());
Imgproc.dilate(mFGMask, mFGMask, new Mat());
//drawing contours around the objects by first called findContours and then calling drawContours
//RETR_EXTERNAL retrieves only external contours
//CHAIN_APPROX_NONE detects all pixels for each contour
Imgproc.findContours(mFGMask, contours, new Mat(), Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_NONE);
//draws all the contours in red with thickness of 2
Imgproc.drawContours(mRgb, contours, -1, new Scalar(255, 0, 0), 2);
long e2 = Core.getTickCount();
long e = e2 - e1;
double time = e / Core.getTickFrequency();
Log.d("timeTAG", "" + contours.size());
return mRgb;
}
What do you suggest as a solution for this problem ?
I am using a stencil mask on a 3d object using the hello ar java demo however i am running into some unexpected behaviour. My stencil mask correctly occludes the plane renderer but the 3d object (andy) does not seem to react expectedly. Instead he seems to get flipped as shown in the picture. I am not sure how to approach fixing this issue. Attached is the code snippet doing the stencil masking
Image of stencil correctly working on plane buffer but failing on 3d model
GLES20.glClear ( GLES20.GL_STENCIL_BUFFER_BIT );
GLES20.glEnable(GLES20.GL_STENCIL_TEST);
GLES20.glColorMask(false, false, false, false);
GLES20.glDepthMask(false);
GLES20.glStencilFunc(GLES20.GL_NEVER, 1, 0xFF);
GLES20.glStencilOp(GLES20.GL_REPLACE, GLES20.GL_KEEP, GLES20.GL_KEEP);
GLES20.glStencilMask(0xFF);
GLES20.glClear(GLES20.GL_STENCIL_BUFFER_BIT);
// controls how pixels are rendered in the stencil mask
quadDrawer.draw();
GLES20.glColorMask(true, true, true, true);
GLES20.glDepthMask(true);
GLES20.glStencilMask(0xFF);
GLES20.glStencilFunc(GLES20.GL_EQUAL, 0, 0xFF);
// Visualize planes.
// reacts correctly to the stencil mask
planeRenderer.drawPlanes(
session.getAllTrackables(Plane.class), camera.getDisplayOrientedPose(), projmtx);
// Visualize anchors created by touch.
float scaleFactor = 1.0f;
for (ColoredAnchor coloredAnchor : anchors) {
if (coloredAnchor.anchor.getTrackingState() != TrackingState.TRACKING) {
continue;
}
// Get the current pose of an Anchor in world space. The Anchor pose is updated
// during calls to session.update() as ARCore refines its estimate of the world.
coloredAnchor.anchor.getPose().toMatrix(anchorMatrix, 0);
// Update and draw the model and its shadow.
// does not react correctly to the
virtualObject.updateModelMatrix(anchorMatrix, scaleFactor);
virtualObjectShadow.updateModelMatrix(anchorMatrix, scaleFactor);
virtualObject.draw(viewmtx, projmtx, colorCorrectionRgba, coloredAnchor.color);
virtualObjectShadow.draw(viewmtx, projmtx, colorCorrectionRgba, coloredAnchor.color);
}
GLES20.glDisable(GLES20.GL_STENCIL_TEST);
Forgot to update the "solution" i found for this. Instead of applying the stencil mask over the 3d object, i render the background again but passing it through the stencil mask. This means that the background will overlay the 3d object hence achieving the desired "masking" effect.
I wanted to find the convex hull in order to even the edges of a hand-drawn triangle on paper. Smoothing using image processing was not enough because i needed to detect this triangle too and a hand drawn triangle tends to have more than three points if the approxPolyDP function is used. A convex hull of a triangle is correctly identified by the approxPolyDP function.
The problem is, i have other shapes in the image too on which a convex hull is created.
Before convex hull is used: Notice the contour labelled 3
After convex hull is used: the end points have been joined and the contour labelled 3 forms a triangle
Now i wanted to somehow exclude contour 3 from being detected as a triangle.
To do this my strategy was to remove this contour altogether from the ArrayList named hullMop. This is because my triangle detection function uses the contours from hullMop and so it wouldnt even check the contour labelled 3.
extcontours are the contours before convex hull is used.
This function checks if a point from hullMop is inside extcontours. If it isn't, then that must be removed from hullMop because they are the extra set of points generated because of the convex hull, or in other words, the red line in the second image.
Now at this point I feel there is a hole in my concept. The openCV documentation says that the convex Hull returns the subset of the points of the original array, in other words, subset of the points of extcontours.
My question is, how do i get the points of the red line created by the convexHull function. I dont want to use findContours because i feel there is a better way.
private void RemoveFalseHullTriangles(ArrayList<MatOfPoint> extcontours, ArrayList<MatOfPoint> hullMop, int width, int height) {
//if every single point of hullmop doesnt touch or isn't inside extcontours, then that point must be the red line
MatOfPoint2f Contours2f = new MatOfPoint2f();
double [] newA = new double[2];
int hullCounter = 0;
A: for(int i =0;i<extcontours.size();i++) {
MatOfPoint ExtCnt = extcontours.get(i);
MatOfPoint HullCnt = hullMop.get(hullCounter);
ExtCnt.convertTo(Contours2f, CvType.CV_32F);
B: for (int j = 0; j < HullCnt.rows(); j++) {
double[] pt = new double[2];
pt[0] = HullCnt.get(j,0)[0];
pt[1] = HullCnt.get(j,0)[1];
if (Math.abs(Imgproc.pointPolygonTest(Contours2f, new Point(pt), true)) > 40) {
//Remove index from HullMop
hullMop.remove(hullCounter);
hullCounter--;
break B;
}
}
hullCounter++;
}
}
Because the hullMop only has a subset of the points of extcontours, i may never know the points of the red line of the contour labelled 3 after convex hull is used.
Is there anyway to get coordinates of that red line generated by convex hull other than using findContours?
As referenced by Alexandar Reynolds, the problem really was detecting open contours first and excluding those contours before finding the convex hull.
The method to find open contours is explained here:
Recognize open and closed shapes opencv
Basically, if an outer contour has no child contour in the hierarchy, then it is an open contour and must be excluded before finding convex hull ( for my case).
I started by reading in this Mat.
Then I converted it to Greyscale and applied Imgproc.canny() to it, getting the following mask.
Then I used Imgproc.findContours() to find the contours, Imgproc.drawContours(), and Core.putText() to label the contours with numbers:
Then I did Rect boundingRect = Imgproc.boundingRect(contours.get(0));
Mat submatrix = new Mat();
submatrix = originalMat.submat(boundingRect); to get following submatrix:
So far so good. The Problem starts hereafter:
NOW I NEEDED A MASK OF THE submatrix. So I decided to use Imgproc.drawContours() to get the mask:
Mat mask = new Mat(submatrix.rows(), submatrix.cols(), CvType.CV_8UC1);
List<MatOfPoint> contourList = new ArrayList<>();
contourList.add(contours.get(0));
Imgproc.drawContours(mask, contourList, 0, new Scalar(255), -1);
I got the following mask:
WHAT I WAS EXPECTING was a filled (in white color) diamond shape on black background.
WHy am I getting this unexpected result?
EDIT:
When I replaced Mat mask = new Mat(submatrix.rows(),
submatrix.cols(), CvType.CV_8UC1); by Mat mask =
Mat.zeros(submatrix.rows(), submatrix.cols(), CvType.CV_8UC1);,
the last mask with white colored garbage was replaced by an empty
black mask withOUT any white color on it. I got the following submat
and mask:
I was getting the first contour in the list of contours (named
contours) by contours.get(0), and using this first contour to
calculate Imgproc.boundingRect() as well as in
contourList.add(contours.get(0)); later (where contourList is
the list of just one contour which will be used in the last
drawContours()).
Then I went ahead to change contours.get(0) to
contours.get(1) in Imgproc.boundingRect() as well as in contourList.add(); (just before Imgproc.drawContours()). That
resulted in this submat and mask:
Then I changed back to contours.get(0) in
Imgproc.boundingRect(); and let
contourList.add(contours.get(1)); be there. Got the following
submat and mask:
NOW I am completely Unable to Understand what is happening here.
I am not sure how this is handle in JAVA (I usually use OpenCV in c++ or python), but there is an error in your code...
The contours list will have a list of list of points. This points will refer to the original image. So, this mean that if the figure one is in lets say, x=300, y= 300, width= 100, height=100 then when you get your submatrix it will try to draw those points in a smaller image... so when it tries to draw point (300,300) in a 100 x 100 image, it will simply fail... probably throws an error or simply doesn't draw anything...
A solution for this is, do a for loop and substract to each point of the contour the initial point of the bounding rect (in my example (300,300)).
As, why there is some garbage drawn... well you never initialize the matrix. Not sure in JAVA, but in c++ you have to set them to 0.
I think it should be something like this:
Mat mask = new Mat(submatrix.rows(), submatrix.cols(), CvType.CV_8UC1, new Scalar(0));
I hope this helps :)
EDIT
I think I did not explain myself clearly before.
Your contours are an array of points (x,y). These are the coordinates of the points that represent each contour in the original image. This image has a size, and your submatrix has a smaller size. The points are outside of this small image boundaries....
you should do something like this to fix it:
for (int j = 0; j < contours[0].length; j++) {
contours[0][j].x -= boundingrect.x;
contours[0][j].y -= boundingrect.y;
}
and then you can draw the contours, since they will be in boundaries of the submat.
I think in java it is also possible to subtract the opencv points directly:
for (int j = 0; j < contours[0].length; j++) {
contours[0][j] -= boundingrect.tl();
}
but in this case I am not sure, since I have tried it in c++ only
boundingrect.tl() -> gives you the top left point of the rect
I am working on an app that is expected to remove image backgrounds using opencv, at first I tried using grabcut but it was too slow and the results were not always accurate, then I tried using threshold, although the results are not yet close th grabcut, its very fast and looks like a better, So my code is first looking at the image hue and analying which portion of it appears more, that portion is taken in as the background, the issue is at times its getting the foreground as background below is my code:
private Bitmap backGrndErase()
{
Bitmap bitmap = BitmapFactory.decodeResource(getResources(), R.drawable.skirt);
Log.d(TAG, "bitmap: " + bitmap.getWidth() + "x" + bitmap.getHeight());
bitmap = ResizeImage.getResizedBitmap(bitmap, calculatePercentage(40, bitmap.getWidth()), calculatePercentage(40, bitmap.getHeight()));
Mat frame = new Mat();
Utils.bitmapToMat(bitmap, frame);
Mat hsvImg = new Mat();
List<Mat> hsvPlanes = new ArrayList<>();
Mat thresholdImg = new Mat();
// int thresh_type = Imgproc.THRESH_BINARY_INV;
//if (this.inverse.isSelected())
int thresh_type = Imgproc.THRESH_BINARY;
// threshold the image with the average hue value
hsvImg.create(frame.size(), CvType.CV_8U);
Imgproc.cvtColor(frame, hsvImg, Imgproc.COLOR_BGR2HSV);
Core.split(hsvImg, hsvPlanes);
// get the average hue value of the image
double threshValue = this.getHistAverage(hsvImg, hsvPlanes.get(0));
Imgproc.threshold(hsvPlanes.get(0), thresholdImg, threshValue, mThresholdValue, thresh_type);
// Imgproc.adaptiveThreshold(hsvPlanes.get(0), thresholdImg, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C, Imgproc.THRESH_BINARY, 11, 2);
Imgproc.blur(thresholdImg, thresholdImg, new Size(5, 5));
// dilate to fill gaps, erode to smooth edges
Imgproc.dilate(thresholdImg, thresholdImg, new Mat(), new Point(-1, -1), 1);
Imgproc.erode(thresholdImg, thresholdImg, new Mat(), new Point(-1, -1), 3);
Imgproc.threshold(thresholdImg, thresholdImg, threshValue, mThresholdValue, Imgproc.THRESH_BINARY);
//Imgproc.adaptiveThreshold(thresholdImg, thresholdImg, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C, Imgproc.THRESH_BINARY, 11, 2);
// create the new image
Mat foreground = new Mat(frame.size(), CvType.CV_8UC3, new Scalar(255, 255, 255));
frame.copyTo(foreground, thresholdImg);
Utils.matToBitmap(foreground,bitmap);
//return foreground;
alreadyRun = true;
return bitmap;
}
the method responsible for Hue:
private double getHistAverage(Mat hsvImg, Mat hueValues)
{
// init
double average = 0.0;
Mat hist_hue = new Mat();
// 0-180: range of Hue values
MatOfInt histSize = new MatOfInt(180);
List<Mat> hue = new ArrayList<>();
hue.add(hueValues);
// compute the histogram
Imgproc.calcHist(hue, new MatOfInt(0), new Mat(), hist_hue, histSize, new MatOfFloat(0, 179));
// get the average Hue value of the image
// (sum(bin(h)*h))/(image-height*image-width)
// -----------------
// equivalent to get the hue of each pixel in the image, add them, and
// divide for the image size (height and width)
for (int h = 0; h < 180; h++)
{
// for each bin, get its value and multiply it for the corresponding
// hue
average += (hist_hue.get(h, 0)[0] * h);
}
// return the average hue of the image
average = average / hsvImg.size().height / hsvImg.size().width;
return average;
}
A sample of the input and output:[
Input Image 2 and Output:
Input Image 3 and Output:
Indeed, as others have said you are unlikely to get good results just with a threshold on hue. You can use something similar to GrabCut, but faster.
Under the hood, GrabCut calculates foreground and background histograms, then calculates the probability of each pixel being FG/BG based on these histograms, and then optimizes the resulting probability map using graph cut to obtain a segmentation.
Last step is most expensive, and it may be ignored depending on the application. Instead, you may apply the threshold to the probability map to obtain a segmentation. It may (and will) be worse than GrabCut, but will be better than your current approach.
There are some points to consider for this approach. The choice of histogram model would be very important here. You can either consider 2 channels in some space like YUV or HSV, consider 3 channels of RGB, or consider 2 channels of normalized RGB. You also have to select an appropriate bin size for those histograms. Too small bins would lead to 'overtraining', while too large will reduce the precision. The tradeoffs between those are a topic for a separate discussion, in brief - I would advice using RGB with 64 bins per channel for start and then see what changes are better for your data.
Also, you can get better results for coarse binning if you use interpolation to get values between bins. In past I have used trilinear interpolation and it was kind of good, compared to no interpolation at all.
But remember that there are no guarantees that your segmentation will be correct without prior knowledge on object shape, either with GrabCut, thresholding or this approach.
I would try again Grabcut, it is one of the best segmentation methods available. This is the result I get
cv::Mat bgModel,fgModel; // the models (internally used)
cv::grabCut(image,// input image
object_mask,// segmentation result
rectang,// rectangle containing foreground
bgModel,fgModel, // models
5,// number of iterations
cv::GC_INIT_WITH_RECT); // use rectangle
// Get the pixels marked as likely foreground
cv::compare(object_mask,cv::GC_PR_FGD,object_mask,cv::CMP_EQ);
cv::threshold(object_mask, object_mask, 0,255, CV_THRESH_BINARY); //ensure the mask is binary
The only problem of Grabcut is that you have to give as an input a rectangle containing the object you want to extract. Apart from that it works pretty well.
Your method of finding average hue is WRONG! As you most probably know, hue is expressed as angle and takes value in [0,360] range. Therefore, a pixel with hue 360 essentially has same colour as a pixel with hue 0 (both are pure red). In the same way, a pixel with hue 350 is actually closer to a pixel with hue 10 than a pixel with hue, say for example, 300.
As for opencv, cvtColor function actually divides calculated hue value by 2 to fit it in 8 bit integer. Thus, in opencv, hue values wrap after 180. Now, consider we have two red(ish) pixels with hues 10 and 170. If we take their average, we will get 90 — hue of pure cyan, the exact opposite of red — which is not our desired value.
Therefore, to correctly find the average hue, you need to first find average pixel value in RGB colour space, then calculate the hue from this RGB value. You can create 1x1 matrix with average RGB pixel and convert it to HSV/HSL.
Following the same reasoning, applying threshold to hue image doesn't work flawlessly. It does not consider wrapping of hue values.
If I understand correctly, you want to find pixels with similar hue as the background. Assuming we know the colour of background, I would do this segmentation in RGB space. I would introduce some tolerance variable. I would use the background pixel value as centre and this tolerance as radius and thus define a sphere in RGB colour space. Now, rest is inspecting each pixel value, if it falls inside this sphere, then classify as background; otherwise, regard it as foreground pixel.