I am new to OpenCV and am trying to count the number of objects in an image. I have done this before using MATLAB Image Processing Toolbox and adapted the same approach in OpenCV (Android) also.
The first step was to convert an image to gray scale. Then to threshold it and then counting the number of blobs. In Matlab there is a command - "bwlabel", which gives the number of blobs. I couldn't find such thing in OpenCV (again, I am a noob in OpenCV as well as Android).
Here is my code,
//JPG to Bitmap to MAT
Bitmap i = BitmapFactory.decodeFile(imgPath + "mms.jpg");
Bitmap bmpImg = i.copy(Bitmap.Config.ARGB_8888, false);
Mat srcMat = new Mat ( bmpImg.getHeight(), bmpImg.getWidth(), CvType.CV_8UC3);
Utils.bitmapToMat(bmpImg, srcMat);
//convert to gray scale and save image
Mat gray = new Mat(srcMat.size(), CvType.CV_8UC1);
Imgproc.cvtColor(srcMat, gray, Imgproc.COLOR_RGB2GRAY,4);
//write bitmap
Boolean bool = Highgui.imwrite(imgPath + "gray.jpg", gray);
//thresholding
Mat threshed = new Mat(bmpImg.getWidth(),bmpImg.getHeight(), CvType.CV_8UC1);
Imgproc.adaptiveThreshold(gray, threshed, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C, Imgproc.THRESH_BINARY, 75, 5);//15, 8 were original tests. Casey was 75,10
Core.bitwise_not(threshed, threshed);
Utils.matToBitmap(threshed, bmpImg);
//write bitmap
bool = Highgui.imwrite(imgPath + "threshed.jpg", threshed);
Toast.makeText(this, "Thresholded image saved!", Toast.LENGTH_SHORT).show();
In the next step, I tried to fill the holes and letters using dilation followed by an erosion but the blobs gets attached to each other which will ultimately give a wrong count. There is a tradeoff between filling holes and getting the blobs attached to each other on tuning the parameters for dilation and erosion.
Here is the code,
//morphological operations
//dilation
Mat dilated = new Mat(bmpImg.getWidth(),bmpImg.getHeight(), CvType.CV_8UC1);
Imgproc.dilate(threshed, dilated, Imgproc.getStructuringElement(Imgproc.MORPH_ELLIPSE, new org.opencv.core.Size (16, 16)));
Utils.matToBitmap(dilated, bmpImg);
//write bitmap
bool = Highgui.imwrite(imgPath + "dilated.jpg", dilated);
Toast.makeText(this, "Dilated image saved!", Toast.LENGTH_SHORT).show();
//erosion
Mat eroded = new Mat(bmpImg.getWidth(),bmpImg.getHeight(), CvType.CV_8UC1);
Imgproc.erode(dilated, eroded, Imgproc.getStructuringElement(Imgproc.MORPH_ELLIPSE, new org.opencv.core.Size(15, 15)));
Utils.matToBitmap(eroded, bmpImg);
//write bitmap
bool = Highgui.imwrite(imgPath + "eroded.jpg", eroded);
Toast.makeText(this, "Eroded image saved!", Toast.LENGTH_SHORT).show();
Because sometimes my M&Ms might be just next to each other! ;)
I also tried to use Hough Circles but the result is very unreliable (tested with coin images as well as real coins)
Here is the code,
//hough circles
Mat circles = new Mat();
// parameters
int iCannyUpperThreshold = 100;
int iMinRadius = 20;
int iMaxRadius = 400;
int iAccumulator = 100;
Imgproc.HoughCircles(gray, circles, Imgproc.CV_HOUGH_GRADIENT,
1.0, gray.rows() / 8, iCannyUpperThreshold, iAccumulator,
iMinRadius, iMaxRadius);
// draw
if (circles.cols() > 0)
{
Toast.makeText(this, "Coins : " +circles.cols() , Toast.LENGTH_LONG).show();
}
else
{
Toast.makeText(this, "No coins found", Toast.LENGTH_LONG).show();
}
The problem with this approach is that the algorithm is limited to perfect circles only (AFAIK). So, it doesn't work well when I try to scan and count M&Ms or coins lying on my desk (because angle of the device changes). With this approach, sometimes I get less no. of coins detected and sometimes more (I don't get it why more??).
On scanning this image the app sometimes shows 19 coins and sometimes 38 coins counted...I know there are other features which may be detected as circles but I totally don't get it why 38..?
So my questions...
Is there a better way to fill holes without joining adjacent blobs?
How do I count the number of objects accurately? I don't want to limit my app to counting only circles with HoughCircles approach.
FYI : OpenCV-2.4.9-android-sdk. Kindly keep in mind that I am a newbie in OpenCV and Android too.
Any help is much appreciated.
Thanks & Cheers!
Jainam
So to proceed we take your threshold image which you have generated as input and further modify it. The present code is in C++ but I guess you can easily convert it into android platform
Now instead of dilation or blurring you can try flood fill
which results in
Finally now applying the contour detection algorithm algorithm we get
The code for the above is
Mat dst = imread($path to the threshold image); // image should be single channel black and white image
imshow("dst",dst);
cv::Mat mask = cv::Mat::zeros(dst.rows + 2, dst.cols + 2, CV_8U);
// A image with size greater than the present object is created
cv::floodFill(dst, mask, cv::Point(0,0), 255, 0, cv::Scalar(), cv::Scalar(), 4 + (255 << 8) + cv::FLOODFILL_MASK_ONLY);
erode(mask,mask,Mat());
// Now to remove the outer boundary
rectangle(mask,Rect(0,0,mask.cols,mask.rows), Scalar(255,255,255),2,8,0);
imshow("Mask",mask);
Mat copy;
mask.copyTo(copy);
vector<vector<Point> > contours;
vector<Vec4i> hierarchy;
findContours( copy, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, Point(0, 0) );
vector<vector<Point> > contours_poly( contours.size() );
vector<Rect> boundRect( contours.size() );
vector<Point2f>center( contours.size() );
vector<float>Distance( contours.size() );
vector<float>radius( contours.size() );
Mat drawing = cv::Mat::zeros(mask.rows, mask.cols, CV_8U);
int num_object = 0;
for( int i = 0; i < contours.size(); i++ ){
approxPolyDP( Mat(contours[i]), contours_poly[i], 3, true );
// To get rid of the smaller object and the outer rectangle created
//because of the additional mask image we enforce a lower limit on area
//to remove noise and an upper limit to remove the outer border.
if (contourArea(contours_poly[i])>(mask.rows*mask.cols/10000) && contourArea(contours_poly[i])<mask.rows*mask.cols*0.9){
boundRect[i] = boundingRect( Mat(contours_poly[i]) );
minEnclosingCircle( (Mat)contours_poly[i], center[i], radius[i] );
circle(drawing,center[i], (int)radius[i], Scalar(255,255,255), 2, 8, 0);
rectangle(drawing,boundRect[i], Scalar(255,255,255),2,8,0);
num_object++;
}
}
cout <<"No. of object detected =" <<num_object<<endl;
imshow("drawing",drawing);
waitKey(2);
char key = (char) waitKey(20);
if(key == 32){
// You can save your images here using a space
}
I hope this helps you in solving your problem
Just check it out,
Blur source.
Threshold binary inverted on gray.
Find contours, note that you should use CV_RETR_EXTERNAL as contour retrieval mode.
You can take the contours size as your object count.
Code:
Mat tmp,thr;
Mat src=imread("img.jpg",1);
blur(src,src,Size(3,3));
cvtColor(src,tmp,CV_BGR2GRAY);
threshold(tmp,thr,220,255,THRESH_BINARY_INV);
imshow("thr",thr);
vector< vector <Point> > contours; // Vector for storing contour
vector< Vec4i > hierarchy;
findContours( thr, contours, hierarchy,CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE ); // Find the contours in the image
for( int i = 0; i< contours.size(); i=hierarchy[i][0] ) // iterate through each contour.
{
Rect r= boundingRect(contours[i]);
rectangle(src,r, Scalar(0,0,255),2,8,0);
}
cout<<"Numeber of contour = "<<contours.size()<<endl;
imshow("src",src);
waitKey();
Related
I'm trying to overlay on the live frame a little image previosuly selected by the user. I already have the image's path from a previous activity. My problem is that I am not able to show the image on the frame.
I'm trying to detect a rectangle on the frame, and over the rectangle display the image selected. I could detect the rectangle, but now i can't display the image on any part of the frame (I don't care about the rectangle right now).
I've been trying to do it with the explanations from Adding Image Overlay OpenCV for Android and add watermark small image to large image opencv4android but it didn't worked for me.
public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
Mat gray = inputFrame.gray();
Mat dst = inputFrame.rgba();
Imgproc.pyrDown(gray, dsIMG, new Size(gray.cols() / 2, gray.rows() / 2));
Imgproc.pyrUp(dsIMG, usIMG, gray.size());
Imgproc.Canny(usIMG, bwIMG, 0, threshold);
Imgproc.dilate(bwIMG, bwIMG, new Mat(), new Point(-1, 1), 1);
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
cIMG = bwIMG.clone();
Imgproc.findContours(cIMG, contours, hovIMG, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
for (MatOfPoint cnt : contours) {
MatOfPoint2f curve = new MatOfPoint2f(cnt.toArray());
Imgproc.approxPolyDP(curve, approxCurve, 0.02 * Imgproc.arcLength(curve, true), true);
int numberVertices = (int) approxCurve.total();
double contourArea = Imgproc.contourArea(cnt);
if (Math.abs(contourArea) < 100) {
continue;
}
//Rectangle detected
if (numberVertices >= 4 && numberVertices <= 6) {
List<Double> cos = new ArrayList<>();
for (int j = 2; j < numberVertices + 1; j++) {
cos.add(angle(approxCurve.toArray()[j % numberVertices], approxCurve.toArray()[j - 2], approxCurve.toArray()[j - 1]));
}
Collections.sort(cos);
double mincos = cos.get(0);
double maxcos = cos.get(cos.size() - 1);
if (numberVertices == 4 && mincos >= -0.3 && maxcos <= 0.5) {
//Small watermark image
Mat a = imread(img_path);
Mat bSubmat = dst.submat(0, dst.rows() -1 , 0, dst.cols()-1);
a.copyTo(bSubmat);
}
}
}
return dst;
}
NOTE: img_path is the path of the selected image I want to display over the frame. I got it from the previous activity.
By now, I just want to display the image over the frame. Later, I will try to display it on the same position where it found the rectangle.
Please, any suggestion or recommendation is welcome, as I am new with OpenCV. I'm sorry for my english, but feel free to ask me anything I didn't explain correctly. I'll do my best to explain it better.
Thanks a lot!
If you just want to display the image as an overlay, and not save it as part of the video, you may find it easier to simple display it in a separate view above the video view. This will likely use less processing, battery etc also.
If you want to draw onto the camera image bitmap then the following will allow you do that:
Bitmap cameraBitmap = BitmapFactory.decodeByteArray(bytes,0,bytes.length, opt);
Canvas camImgCanvas = new Canvas(cameraBitmap);
Drawable d = ContextCompat.getDrawable(getActivity(), R.drawable.myDrawable);
//Centre the drawing
int bitMapWidthCenter = cameraBitmap.getWidth()/2;
int bitMapheightCenter = cameraBitmap.getHeight()/2;
d.setBounds(bitMapWidthCenter, bitMapheightCenter, bitMapWidthCenter+d.getIntrinsicWidth(),
bitMapheightCenter+d.getIntrinsicHeight());
//And draw it...
d.draw(camImgCanvas);
I have tried 2 methodologies as follows:-
conversion of image to Mat
apply gaussian blur
then canny edge detection
find contours
The problem with this method is:
too many contours are detected
mostly open contours
doesn't detect what I want to detect
Then I changed my approach and tried adaptive thresholding after gaussian blur/median blur and it is much better and I am able to detect the corners in 50% cases
The current problem I am facing is that the page detection requires contrasting and plain background without any reflections. I think it's too idealistic for real world use.
This is where I would like some help. Even a direction towards the solution is highly appreciated especially in java. Thanks in anticipation
works absolutely fine with a significant contrasting background like this
Detected 4 corners
This picture gives troubles because the background isn't exactly the most contrasting
Initial largest contour found
Update: median blur did not help much so I traced the cause and found that the page boundary was detected in bits and pieces and not a single contour so it detected the biggest contour as a part of the page boundary Therefore performed some morphological operations to close relatively small gaps and the resultant largest contour is definitely improved but its its not optimum. Any ideas how I can improve the big gaps?
morphed original picture
largest contour found in the morphed image
PS morphing the image in ideal scenarios has led to detection of false contour boundaries. Any condition which can be checked before morphing an image is also a bonus. Thank you
If you use methods like that:
public static RotatedRect getBestRectByArea(List<RotatedRect> boundingRects) {
RotatedRect bestRect = null;
if (boundingRects.size() >= 1) {
RotatedRect boundingRect;
Point[] vertices = new Point[4];
Rect rect;
double maxArea;
int ixMaxArea = 0;
// find best rect by area
boundingRect = boundingRects.get(ixMaxArea);
boundingRect.points(vertices);
rect = Imgproc.boundingRect(new MatOfPoint(vertices));
maxArea = rect.area();
for (int ix = 1; ix < boundingRects.size(); ix++) {
boundingRect = boundingRects.get(ix);
boundingRect.points(vertices);
rect = Imgproc.boundingRect(new MatOfPoint(vertices));
if (rect.area() > maxArea) {
maxArea = rect.area();
ixMaxArea = ix;
}
}
bestRect = boundingRects.get(ixMaxArea);
}
return bestRect;
}
private static Bitmap findROI(Bitmap sourceBitmap) {
Bitmap roiBitmap = Bitmap.createBitmap(sourceBitmap.getWidth(), sourceBitmap.getHeight(), Bitmap.Config.ARGB_8888);
Mat sourceMat = new Mat(sourceBitmap.getWidth(), sourceBitmap.getHeight(), CV_8UC3);
Utils.bitmapToMat(sourceBitmap, sourceMat);
final Mat mat = new Mat();
sourceMat.copyTo(mat);
Imgproc.cvtColor(mat, mat, Imgproc.COLOR_RGB2GRAY);
Imgproc.threshold(mat, mat, 146, 250, Imgproc.THRESH_BINARY);
// find contours
List<MatOfPoint> contours = new ArrayList<>();
List<RotatedRect> boundingRects = new ArrayList<>();
Imgproc.findContours(mat, contours, new Mat(), Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
// find appropriate bounding rectangles
for (MatOfPoint contour : contours) {
MatOfPoint2f areaPoints = new MatOfPoint2f(contour.toArray());
RotatedRect boundingRect = Imgproc.minAreaRect(areaPoints);
boundingRects.add(boundingRect);
}
RotatedRect documentRect = getBestRectByArea(boundingRects);
if (documentRect != null) {
Point rect_points[] = new Point[4];
documentRect.points(rect_points);
for (int i = 0; i < 4; ++i) {
Imgproc.line(sourceMat, rect_points[i], rect_points[(i + 1) % 4], ROI_COLOR, ROI_WIDTH);
}
}
Utils.matToBitmap(sourceMat, roiBitmap);
return roiBitmap;
}
you can achieve for your source images results like this:
or that:
If you adjust threshold values and apply filters you can achieve even better results.
You can pick a single contour by using one or both of:
Use BoundingRect and ContourArea to evaluate the squareness of each contour. boundingRect() returns orthogonal rects., to handle arbitrary rotation better use minAreaRect() which returns optimally rotated ones.
Use Cv.ApproxPoly iteratively to reduce to a 4 sided shape
var approxIter = 1;
while (true)
{
var approxCurve = Cv.ApproxPoly(largestContour, 0, null, ApproxPolyMethod.DP, approxIter, true);
var approxCurvePointsTmp = new[] { approxCurve.Select(p => new CvPoint2D32f((int)p.Value.X, (int)p.Value.Y)).ToArray() }.ToArray();
if (approxCurvePointsTmp[0].Length == 4)
{
corners = approxCurvePointsTmp[0];
break;
}
else if (approxCurvePointsTmp[0].Length < 4) throw new InvalidOperationException("Failed to decimate corner points");
approxIter++;
}
However neither of these will help if the contour detection gives you two separate contours due to noise / contrast.
I think it would be possible to use the hough line transformation to help detect cases where a line has been split into two contours.
If so the search could be repeated for all combinations of joined contours to see if a bigger / more rectangular match is found.
Stop relying on edge detection, the worst methodology in the universe, and switch to some form of image segmentation.
The paper is white, the background is contrasted, this is the information that you should use.
The following is the code that I run to find the face region
private void detectRegion() {
//get the image from gallery and change it into bitmap
Bitmap bmpTemp = originalImg.copy(Bitmap.Config.ARGB_8888, true);
Utils.bitmapToMat(bmpTemp, mRgbMat);
Imgproc.cvtColor(mRgbMat, mHsvMat, Imgproc.COLOR_RGB2HSV, channelCount);
Scalar lowerThreshold = new Scalar(0, 0.23 * 255, 50); // Blue color – lower hsv values
Scalar upperThreshold = new Scalar(50, 0.68 * 255, 255); // Blue color – higher hsv values
Core.inRange(mHsvMat, lowerThreshold, upperThreshold, mMaskMat);
Imgproc.dilate(mMaskMat, mDilatedMat, new Mat());
Imgproc.findContours(mMaskMat, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
Imgproc.drawContours(mRgbMat, contours,counter, colorGreen, iLineThickness);
Log.d(TAG + " contours " , contours.get(counter).toString());
// convert to bitmap:
Bitmap bm = Bitmap.createBitmap(mHsvMat.cols(), mHsvMat.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mRgbMat, bm);
// find the imageview and draw it!
imageView.setImageBitmap(bm);
}
However, I get so many regions that I did't want.
This is my desired region ( the green one) that I find manually. I add the value when I click the button and draw different region when I press the button.
this is the message of show in logcat.
02-16 00:11:46.413 14234-14234/fyp.hkust.facet D/ColorizeFaceActivity contours: Mat [ 932*1*CV_32SC2, isCont=true, isSubmat=false, nativeObj=0xffffffff96aa78f8, dataAddr=0xffffffff96a57010 ]
I also tried to draw them all but there are so many extra regions that I don't want.
for (int contourIdx = 0; contourIdx < contours.size(); contourIdx++) {
if (contours.size() > 100) // Minimum size allowed for consideration
{
Imgproc.drawContours(mRgbMat, contours, contourIdx, colorGreen, iLineThickness);
Log.d(TAG + " contours " , contours.get(contourIdx).toString());
}
}
This is the whole result
How can I classify them to get the face region and also I need to extract the hsv value in that region. How can I do that? Please give me some help. Thank you very much.
This might have been answered but I desperately need an answer for this. I want to find the largest square or rectangle in an image using OpenCV in Android. All of the solutions that I found are C++ and I tried converting it but it doesn't work and I do not know where I'm wrong.
private Mat findLargestRectangle(Mat original_image) {
Mat imgSource = original_image;
Imgproc.cvtColor(imgSource, imgSource, Imgproc.COLOR_BGR2GRAY);
Imgproc.Canny(imgSource, imgSource, 100, 100);
//I don't know what to do in here
return imgSource;
}
What I am trying to accomplish in here is to create a new image that is based on the largest square found in the original image (return value Mat image).
This is what I want to happen:
1 http://img14.imageshack.us/img14/7855/s7zr.jpg
It's also okay that I just get the four points of the largest square and I think I can take it from there. But it would be better if I can just return the cropped image.
After canny
1- you need to reduce noises with gaussian blur and find all the contours
2- find and list all the contours' areas.
3- the largest contour will be nothing but the painting.
4- now use perpective transformation to transform your shape to a rectangle.
check sudoku solver examples to see the similar processing problem. (largest contour + perspective)
Took me a while to convert the C++ code to Java, but here it is :-)
Warning ! Raw code, totally not optimized and all.
I decline any liability in cases of injury or lethal accident
List<MatOfPoint> squares = new ArrayList<MatOfPoint>();
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
if (Math.random()>0.80) {
findSquares(inputFrame.rgba().clone(),squares);
}
Mat image = inputFrame.rgba();
Imgproc.drawContours(image, squares, -1, new Scalar(0,0,255));
return image;
}
int thresh = 50, N = 11;
// helper function:
// finds a cosine of angle between vectors
// from pt0->pt1 and from pt0->pt2
double angle( Point pt1, Point pt2, Point pt0 ) {
double dx1 = pt1.x - pt0.x;
double dy1 = pt1.y - pt0.y;
double dx2 = pt2.x - pt0.x;
double dy2 = pt2.y - pt0.y;
return (dx1*dx2 + dy1*dy2)/Math.sqrt((dx1*dx1 + dy1*dy1)*(dx2*dx2 + dy2*dy2) + 1e-10);
}
// returns sequence of squares detected on the image.
// the sequence is stored in the specified memory storage
void findSquares( Mat image, List<MatOfPoint> squares )
{
squares.clear();
Mat smallerImg=new Mat(new Size(image.width()/2, image.height()/2),image.type());
Mat gray=new Mat(image.size(),image.type());
Mat gray0=new Mat(image.size(),CvType.CV_8U);
// down-scale and upscale the image to filter out the noise
Imgproc.pyrDown(image, smallerImg, smallerImg.size());
Imgproc.pyrUp(smallerImg, image, image.size());
// find squares in every color plane of the image
for( int c = 0; c < 3; c++ )
{
extractChannel(image, gray, c);
// try several threshold levels
for( int l = 1; l < N; l++ )
{
//Cany removed... Didn't work so well
Imgproc.threshold(gray, gray0, (l+1)*255/N, 255, Imgproc.THRESH_BINARY);
List<MatOfPoint> contours=new ArrayList<MatOfPoint>();
// find contours and store them all as a list
Imgproc.findContours(gray0, contours, new Mat(), Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
MatOfPoint approx=new MatOfPoint();
// test each contour
for( int i = 0; i < contours.size(); i++ )
{
// approximate contour with accuracy proportional
// to the contour perimeter
approx = approxPolyDP(contours.get(i), Imgproc.arcLength(new MatOfPoint2f(contours.get(i).toArray()), true)*0.02, true);
// square contours should have 4 vertices after approximation
// relatively large area (to filter out noisy contours)
// and be convex.
// Note: absolute value of an area is used because
// area may be positive or negative - in accordance with the
// contour orientation
if( approx.toArray().length == 4 &&
Math.abs(Imgproc.contourArea(approx)) > 1000 &&
Imgproc.isContourConvex(approx) )
{
double maxCosine = 0;
for( int j = 2; j < 5; j++ )
{
// find the maximum cosine of the angle between joint edges
double cosine = Math.abs(angle(approx.toArray()[j%4], approx.toArray()[j-2], approx.toArray()[j-1]));
maxCosine = Math.max(maxCosine, cosine);
}
// if cosines of all angles are small
// (all angles are ~90 degree) then write quandrange
// vertices to resultant sequence
if( maxCosine < 0.3 )
squares.add(approx);
}
}
}
}
}
void extractChannel(Mat source, Mat out, int channelNum) {
List<Mat> sourceChannels=new ArrayList<Mat>();
List<Mat> outChannel=new ArrayList<Mat>();
Core.split(source, sourceChannels);
outChannel.add(new Mat(sourceChannels.get(0).size(),sourceChannels.get(0).type()));
Core.mixChannels(sourceChannels, outChannel, new MatOfInt(channelNum,0));
Core.merge(outChannel, out);
}
MatOfPoint approxPolyDP(MatOfPoint curve, double epsilon, boolean closed) {
MatOfPoint2f tempMat=new MatOfPoint2f();
Imgproc.approxPolyDP(new MatOfPoint2f(curve.toArray()), tempMat, epsilon, closed);
return new MatOfPoint(tempMat.toArray());
}
There are some related questions here in SO. Check them out:
OpenCV C++/Obj-C: Detecting a sheet of paper / Square Detection
How do I recognize squares in this image?
There is also an example shipped with OpenCV:
https://code.ros.org/trac/opencv/browser/trunk/opencv/samples/cpp/squares.cpp?rev=4079
Once you have the rectangle, you can align the picture by computing the homography with the rectangle corners and applying a perspective transform.
This might have been answered but I desperately need an answer for this. I want to find the largest square or rectangle in an image using OpenCV in Android. All of the solutions that I found are C++ and I tried converting it but it doesn't work and I do not know where I'm wrong.
private Mat findLargestRectangle(Mat original_image) {
Mat imgSource = original_image;
Imgproc.cvtColor(imgSource, imgSource, Imgproc.COLOR_BGR2GRAY);
Imgproc.Canny(imgSource, imgSource, 100, 100);
//I don't know what to do in here
return imgSource;
}
What I am trying to accomplish in here is to create a new image that is based on the largest square found in the original image (return value Mat image).
This is what I want to happen:
1 http://img14.imageshack.us/img14/7855/s7zr.jpg
It's also okay that I just get the four points of the largest square and I think I can take it from there. But it would be better if I can just return the cropped image.
After canny
1- you need to reduce noises with gaussian blur and find all the contours
2- find and list all the contours' areas.
3- the largest contour will be nothing but the painting.
4- now use perpective transformation to transform your shape to a rectangle.
check sudoku solver examples to see the similar processing problem. (largest contour + perspective)
Took me a while to convert the C++ code to Java, but here it is :-)
Warning ! Raw code, totally not optimized and all.
I decline any liability in cases of injury or lethal accident
List<MatOfPoint> squares = new ArrayList<MatOfPoint>();
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
if (Math.random()>0.80) {
findSquares(inputFrame.rgba().clone(),squares);
}
Mat image = inputFrame.rgba();
Imgproc.drawContours(image, squares, -1, new Scalar(0,0,255));
return image;
}
int thresh = 50, N = 11;
// helper function:
// finds a cosine of angle between vectors
// from pt0->pt1 and from pt0->pt2
double angle( Point pt1, Point pt2, Point pt0 ) {
double dx1 = pt1.x - pt0.x;
double dy1 = pt1.y - pt0.y;
double dx2 = pt2.x - pt0.x;
double dy2 = pt2.y - pt0.y;
return (dx1*dx2 + dy1*dy2)/Math.sqrt((dx1*dx1 + dy1*dy1)*(dx2*dx2 + dy2*dy2) + 1e-10);
}
// returns sequence of squares detected on the image.
// the sequence is stored in the specified memory storage
void findSquares( Mat image, List<MatOfPoint> squares )
{
squares.clear();
Mat smallerImg=new Mat(new Size(image.width()/2, image.height()/2),image.type());
Mat gray=new Mat(image.size(),image.type());
Mat gray0=new Mat(image.size(),CvType.CV_8U);
// down-scale and upscale the image to filter out the noise
Imgproc.pyrDown(image, smallerImg, smallerImg.size());
Imgproc.pyrUp(smallerImg, image, image.size());
// find squares in every color plane of the image
for( int c = 0; c < 3; c++ )
{
extractChannel(image, gray, c);
// try several threshold levels
for( int l = 1; l < N; l++ )
{
//Cany removed... Didn't work so well
Imgproc.threshold(gray, gray0, (l+1)*255/N, 255, Imgproc.THRESH_BINARY);
List<MatOfPoint> contours=new ArrayList<MatOfPoint>();
// find contours and store them all as a list
Imgproc.findContours(gray0, contours, new Mat(), Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
MatOfPoint approx=new MatOfPoint();
// test each contour
for( int i = 0; i < contours.size(); i++ )
{
// approximate contour with accuracy proportional
// to the contour perimeter
approx = approxPolyDP(contours.get(i), Imgproc.arcLength(new MatOfPoint2f(contours.get(i).toArray()), true)*0.02, true);
// square contours should have 4 vertices after approximation
// relatively large area (to filter out noisy contours)
// and be convex.
// Note: absolute value of an area is used because
// area may be positive or negative - in accordance with the
// contour orientation
if( approx.toArray().length == 4 &&
Math.abs(Imgproc.contourArea(approx)) > 1000 &&
Imgproc.isContourConvex(approx) )
{
double maxCosine = 0;
for( int j = 2; j < 5; j++ )
{
// find the maximum cosine of the angle between joint edges
double cosine = Math.abs(angle(approx.toArray()[j%4], approx.toArray()[j-2], approx.toArray()[j-1]));
maxCosine = Math.max(maxCosine, cosine);
}
// if cosines of all angles are small
// (all angles are ~90 degree) then write quandrange
// vertices to resultant sequence
if( maxCosine < 0.3 )
squares.add(approx);
}
}
}
}
}
void extractChannel(Mat source, Mat out, int channelNum) {
List<Mat> sourceChannels=new ArrayList<Mat>();
List<Mat> outChannel=new ArrayList<Mat>();
Core.split(source, sourceChannels);
outChannel.add(new Mat(sourceChannels.get(0).size(),sourceChannels.get(0).type()));
Core.mixChannels(sourceChannels, outChannel, new MatOfInt(channelNum,0));
Core.merge(outChannel, out);
}
MatOfPoint approxPolyDP(MatOfPoint curve, double epsilon, boolean closed) {
MatOfPoint2f tempMat=new MatOfPoint2f();
Imgproc.approxPolyDP(new MatOfPoint2f(curve.toArray()), tempMat, epsilon, closed);
return new MatOfPoint(tempMat.toArray());
}
There are some related questions here in SO. Check them out:
OpenCV C++/Obj-C: Detecting a sheet of paper / Square Detection
How do I recognize squares in this image?
There is also an example shipped with OpenCV:
https://code.ros.org/trac/opencv/browser/trunk/opencv/samples/cpp/squares.cpp?rev=4079
Once you have the rectangle, you can align the picture by computing the homography with the rectangle corners and applying a perspective transform.