Related
I am trying to develop an App that detects cards "master cards, visa, cutomer cards, etc" using Android Camera, for that purpose i used OpenCV4Android version 3.0.0. To achieve this task, i did the following:
1- converted the frame taken from the camera to gray scale using
Imgproc.cvtColor(this.mMatInputFrame, this.mMatGray, Imgproc.COLOR_BGR2GRAY);
2- blurring the frame using
Imgproc.blur(this.mMatGray, this.mMatEdges, new Size(7, 7));
3- apply Canny edge detector as follows
Imgproc.Canny(this.mMatEdges, this.mMatEdges, 2, 900, 7, true);
4- to show Canny's result on the real image, i did the following
this.mDest = new Mat(new Size(this.mMatInputFrame.width(), this.mMatInputFrame.height()), CvType.CV_8U, Scalar.all(0));
this.mMatInputFrame.copyTo(this.mDest, this.mMatEdges);
5- dialated the image using
dilated = new Mat();
Mat dilateElement = Imgproc.getStructuringElement(Imgproc.MORPH_DILATE, new Size(3, 3));
Imgproc.dilate(mMatEdges, dilated, dilateElement);
6- finding the contour of the card detected as follows:
ArrayList<MatOfPoint> contours = new ArrayList<>();
hierachy = new Mat();
Imgproc.findContours(dilated, contours, hierachy, Imgproc.RETR_CCOMP, Imgproc.CHAIN_APPROX_SIMPLE);
for (int i = 0; i < contours.size(); i++) {
if (Imgproc.contourArea(contours.get(i), true) > 90000) {
Rect rect = Imgproc.boundingRect(contours.get(i));
if (rect.height > 60) {
Imgproc.rectangle(mMatInputFrame, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), new Scalar(255, 0, 0));
}
}
}
When I run the App,
Case 1
if the card to be detected is of a homogenous color "the entire card is painted with the same color", Canny produces well defined edges which can easily detected as shown in the image "same-color-0" and "same-color-1".
moreover, when i place the card that of a homogenous color on a table and move the camera around it, the edges are getting detected properly despite i am moving the camera. or in other words, the red frame that surrounds the edges of
the card is always fixed around the edges and never disappears
case 2
if the card is not of a homogenous color "of a mixed colors", then the edge detection is bad as shown in image "mixed-color-0" and "mixed-color-1", and moreover, the red frame that surrounds the edges of the card disappears so often.
Another case extended from this case is, when the card is of two colors, one is light and one is dark, in this case, the edge detector detects only the dark part in the card because its edges are well defined as shown in image
"mixed-color-2"
Please let me know how to get well defined and card-sized edges of the cards regardless of the color?
is there any other more accurate way for edge detection?
same-color-0:
same-color-1
mixed-color-0
mixed-color-1
mixed-color-2
original images:
You can use Structured Edge Detection.
I got these results running the C++ code in this my other answer. This seems like a good and robust result to me.
To use this in Java, you should know that Structured Edge Detection is in contrib module ximgproc.
You probably need to recompile OpenCV to use it: Build OpenCV with contrib modules and Java wrapper
i am working on an Android app that will recognize a GO board and create a SGF file of it.
i made a version that is able to detect a board and warp the perspective to make it square ( code and example image below) unfortunately it gets a bit harder when adding stones.(image below)
Important things about a average go board:
round black and white stones
black lines on the board
board color ranges from white to light brown and sometimes with a wood grain
stones are placed on intersections of two lines
correct me if i am wrong but i think my current approach is not a good one.
Has somebody a general idea on how i can separate the stones and lines from the rest of the picture?
My code:
Mat input = inputFrame.rgba(); //original image
Mat gray = new Mat(); //grayscale image
//convert image to grayscale
Imgproc.cvtColor( input, gray, Imgproc.COLOR_RGB2GRAY);
//try to improve histogram (more contrast)
equalizeHist(gray, gray);
//blur image
Size s = new Size(5,5);
GaussianBlur(gray, gray, s, 0);
//apply adaptive treshold
adaptiveThreshold( gray, gray, 255, Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C, Imgproc.THRESH_BINARY,11,2);
//adding secondary treshold, removes a lot of noise
threshold(gray, gray, 0, 255, Imgproc.THRESH_BINARY + Imgproc.THRESH_OTSU);
Some images:
(source: eightytwo.axc.nl)
(source: eightytwo.axc.nl)
EDIT: 05-03-2016
Yay! managed to detect lines stones and color correctly. precondition the picture has to be only the board itself, without any other background visible.
I use houghLinesP (60lines) and houghCircles (17circles), duration on my phone(1th gen Moto G) about 5 seconds.
Detecting board and warp it turns out to be quite a challenge when it has to be working under different angles and lightning conditions.. still working on that
Suggestions for different approaches are still welcome!!
(source: eightytwo.axc.nl)
EDIT: 15-03-2016
i found a nice way to get line intersects with cross type morphological transformations, works amazing when the picture is taken directly above the board unfortunately not while at an angle (see below)
(source: eightytwo.axc.nl)
In my last update i showed line and stone detection with a picture taken from directly above since then i have been working on detecting the board and warping it in a way that my line and stone detection becomes useful.
harris corner detection
I struggled to get the right parameter settings and i am still not sure if they are optimal, can't find much information on how to optimize image before using harris corners. right now it detects to many corners to be useful. though it feels like it could work. (upper line with pictures in example)
Mat corners = new Mat();
Imgproc.cornerHarris(image, corners, 5, 3, 0.03);
Mat mask = new Mat(corners.size(), CvType.CV_8U, new Scalar(1));
Core.MinMaxLocResult maxVal = Core.minMaxLoc(corners);
Core.inRange(corners, new Scalar(maxVal.maxVal * 0.01), new Scalar(maxVal.maxVal), mask);
cross type morphological transformations
works great when picture is taken directly from above, used from an angle or with a rotated board does not work (middle line with pictures in example)
Imgproc.GaussianBlur(image, image, new Size(5, 5), 0);
Imgproc.adaptiveThreshold(image, image, 255, Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C, Imgproc.THRESH_BINARY_INV, 11, 2);
int morph_elem = 1; //0: Rect - 1: Cross - 2: Ellipse
int morph_size = 5;
int morph_operator = 0; //0: Opening - 1: Closing \n 2: Gradient - 3: Top Hat \n 4: Black Hat
Mat element = getStructuringElement( morph_elem, new Size(2 * morph_size + 1, 2 * morph_size + 1), new Point( morph_size, morph_size ));
morphologyEx(image, image, morph_operator + 2, element);
contour and houghlines
if there are no stones on the outer boardline and light conditions not to harsh it works pretty well. contours are only part of the board quite often(lower line with pictures in example)
Imgproc.GaussianBlur(image, image, new Size(5, 5), 0);
Imgproc.adaptiveThreshold(image, image, 255, Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C, Imgproc.THRESH_BINARY_INV, 11, 2);
Mat hierarchy = new Mat();
MatOfPoint biggest = null;
int contourId = 0;
double biggestArea = 0;
double minSize = 2000;
List<MatOfPoint> contours = new ArrayList<>();
findContours(InvertedImage, contours, hierarchy, Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE);
//find biggest
for( int x = 0; x < contours.size() ; x++ ){
double area = Imgproc.contourArea(contours.get(x));
if( area > minSize && area > biggestArea ){
biggestArea = area;
biggest = contours.get(x);
contourId = x;
}
}
providing the right picture all three the methods work but not good enough to be reliable. any thoughts on parameters, image pre-processing, different approaches or anything that might improve the detection are welcome=)
link to picture
EDIT: 31-03-2016
detecting lines and stones is pretty much solved so i will close this question. created a new one for detecting and warping accurately.
anybody interested in my progress: this is my GOSU Snap Alpha channel don't expect to much of it right now!
EDIT: 16-10-2016
Update: i saw some people are still following this question.
I tested some more stuff and started using Tensorflow, my neural network looks promising, you can have a look at it here.
A lot of work has to be done still, my current image dataset is awful and right now i am working on getting a big dataset.
the app works best using a square board with thick lines and decent lightning.
Assuming that you don't want to "force" your end user to take a cleanest pictures (like using an overlay like some of the QR code scanner for example)
Perhaps you could use some morphological transformations with differents kernels :
Opening and closing with a rectangular kernel for the lines
Opening and closing with an ellipse kernel to get the stones (it should be possible to invert the image at some point to get back the white or the black one)
Take a look at http://docs.opencv.org/2.4/doc/tutorials/imgproc/opening_closing_hats/opening_closing_hats.html (sorry this one is in C++ but I think this is almost the same in Java)
I had try these operations to remove a grid from a Sudoku to avoid noise in cell extraction and it worked like a charm.
Let me know of these informations were usefull for you (this is for sure a very interesting case)
I'm working on same program. I avoid finding lines at all.
First use perspective transform to get the board into a square as you have done. Find the edges of the 19x19 grid. Then assuming the board is 19x19 you can just compute the position of the lines. This works well for me. Then you compute the closest intersection of the center of the stone to determine which row and col line the stone is on. Works pretty well for me. Only probably is calibrating program for different lighting conditions and different color stones and boards.
I'm trying to recognize hand positions in OpenCV for Android. I'd like to reduce a detected hand shape to a set of simple lines (= point sequences). I'm using a thinning algorithm to find the skeleton lines of detected hand shapes. Here's an exemplary result (image of my left hand):
In this image I'd like to get the coordinates of the skeleton lines, i.e. "vectorize" the image. I've tried HoughLinesP but this only produces huge sets of very short lines, which is not what I want.
My second approach uses findContours:
// Get contours
Mat skeletonFrame; //image above
ArrayList<MatOfPoint> contours = new ArrayList<MatOfPoint>();
Imgproc.findContours(skeletonFrame, contours, new Mat(), Imgproc.RETR_CCOMP, Imgproc.CHAIN_APPROX_SIMPLE);
// Find longest contour
double maxLen = 0;
MatOfPoint max = null;
for (MatOfPoint c : contours) {
double len = Imgproc.arcLength(Util.convert(c), true); //Util.convert converts between MatOfPoint and MatOfPoint2f
if (len > maxLen) {
maxLen = len;
max = c;
}
}
// Simplify detected contour
MatOfPoint2f result = new MatOfPoint2f();
Imgproc.approxPolyDP(Util.convert(max), result, 5.0, false);
This basically works; however, the contours returned by findContours are always closed, which means that all the skeleton lines are represented twice.
Exemplary result: (gray lines = detected contours, not skeleton lines of first image)
So my question is: How can I avoid these closed contours and only get a collection of "single stroke" point sequences?
Did I miss something in the OpenCV docs? I'm not necessarily asking for code, a hint for an algorithm I could implement myself would also be great. Thanks!
I would start with real hand skeleton as kinematic
find fingers endpoints and hand/wrist base and perimeter boundary (red)
solve inverse kinematics
for example by CCD to match the fingers endpoints and not overlapping image. This way you should obtain anatomically correct answer
for simplification you can use kinematics like this
you should handle male/female/child differently (different finger lengths) or use some kind of calibration or measurement because of the different finger lengths. As you can see I skip the hand/wrist base bones they are not that important. The red outline can be found where perimeter has smaller curve radius.
How to solve your problem in your current implementation?
The first thinning approach is better so when you got the huge set of lines connect them to polylines after that compute angle of each line. If two joined lines have similar angle (up to treshold) then join them that should do what you want but do not expect you will get the lines similar to human bones especially for curves the result will be much different. In both count of lines and shape.
For better result you need to use geometrical thinning
But I have no Idea if it is present in OpenCV (I do not use this lib) the idea is to find the perimeter line and shift it perpendicular inwards by some small step similar to this. Stop if desired width is acquired
When shifted perimeter leads to too thin shape stop there and connect to thinned point from previous step (yellow line). This is all done on vectors (polylines) not on image pixels !!! Width can be computed as smallest perpendicular distance to any nearby line.
Main idea is to allow user to recolor to specific wall based user selection.
Currently i have implemented this feature using cvFloodFill (helps to prepare mask image) which can help me to change relative HSV value for wall so i can retain edges. but problem with this solution is that it works on color and all walls are repainted instead of single wall selected by user.
i have also tried canny edge detection but it just able to detect edge but not able to convert it to area.
Please find below code which i am currently using for repaint function
Prepare mask
cvFloodFill(mask, new CvPoint(295, 75), new CvScalar(255, 255, 255,0), cvScalarAll(1), cvScalarAll(1), null, 4, null);
split channel
cvSplit(hsvImage, hChannel, sChannel, vChannel, null);
change color
cvAddS(vChannel, new CvScalar(255*(0.76-0.40),0,0,0), vChannel, mask);
How can we detect edges and corresponding area from the image.
i am looking for solution which can be other than opencv but should be possible for iPhone and android
Edit
i am able to achieve somewhat result as below image using below steps
cvCvtColor(image, gray, CV_BGR2GRAY);
cvSmooth(gray,smooth,CV_GAUSSIAN,7,7,0,0);
cvCanny(smooth, canny, 10, 250, 5);
there are two problem with this output not sure how to resolve them
1. close near by edges
2. remove small edges
You could try something like :
Mat imageOut = Mat::zeros(imageIn.rows, imageIn.cols, CV_8UC3);
vector<vector<Point> > contours;
vector<Vec4i> hierarchy;
findContours( imageIn, contours, hierarchy, CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE );
for( int idx = 0; idx >= 0; idx = hierarchy[idx][0] )
{
Scalar color( rand()&255, rand()&255, rand()&255 );
drawContours( imageOut, contours, idx, color, CV_FILLED, 8, hierarchy );
}
It should draw the walls in different colors. If it works, that means that in "hierarchy" each wall is identified as a contour, you then will have to find out which one the user selected on his touch screen and do your color tuning processing.
You may have to change the different parameters in "findContours" link.
You will also need to smooth the input image before the contour detection to avoid being annoyed with the details or textures.
Hope that helps,
Thomas
I think I might have the solution for you!
There is a sample file called watershed.cpp in OpenCV, just run it and you'll get this result :
You can make your user draw on his screen to discriminate each wall.
Then if you want something more precise you can outline the areas (without touching other lines) like this :
And TADA! :
With a little work you can make it user-friendly (cancel last line, connect areas etc...)
Hope that helps!
I think you can use Canny Edge Detection algorithm to find edge difference. Some links
StackOverFlow
StackOverFlow
OpenCV QA
OpenCV
Native Tutorial
I hope this can help you out. Thanks.
Here is some OpenCV4Android code to find the largest contour in a Mat called image, which we'll assume is in the RGBA colour space. To find contours, it's first necessary to threshold or binarize the image (convert to black and white). Using a Gaussian Blur on the image before thresholding reduces the number of small contours that are produced. The size parameters to the blur and threshold must be odd numbers; you can play around to find which value gives the best results (here, I've used 7 for both).
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
Mat BW = new Mat();
Mat hierarchy = new Mat();
MatOfPoint largestContour;
Imgproc.cvtColor(image, image, Imgproc.COLOR_RGBA2GRAY); // convert to grayscale
Imgproc.GaussianBlur(image, BW, new Size(7,7), 0);
Imgproc.adaptiveThreshold(BW, BW, 255,
Imgproc.ADAPTIVE_THRESH_MEAN_C, Imgproc.THRESH_BINARY_INV, 7, 2.0);
Imgproc.findContours(BW, contours, hierarchy, Imgproc.RETR_EXTERNAL,
Imgproc.CHAIN_APPROX_SIMPLE);
double maxArea = 0;
for (MatOfPoint contour : contours) {
double area = Imgproc.contourArea(contour);
if (area > maxArea) {
maxArea = area;
largestContour = contour;
}
}
there are two problem with this output not sure how to resolve them 1. close near by edges 2. remove small edges
You can use morphologic operations to close the edges. Look for the dilation and closing operators.
You can remove small edges by doing labeling. Count the number of pixels in each region (connected white pixels). Remove any region with a number of pixels less than some threshold. I don't use opencv, but most libraries have a labeling function that will create an image where each set of touching pixels of a single color are assigned a unique color in the output image.
I am working on an Android Application in which a 3d scene is displayed and the user should be able to select an area by clicking/tapping the screen. The scene is pretty much a planar (game) board on which different objects are placed.
Now, the problem is how do I get the clicked area on the board from the actual screen-space coordinates?
I was planning on using gluUnProject(), as I have access to (almost) all the necessary parameters. Unfortunately I am missing the winZ param, and cannot get the current depth as the touch event is occurring in a different thread than the GL-thread.
My new plan is to still use gluUnProject, but with a winZ of 0, and then project the resulting point onto the board (the board stretches from 0,0,0 to 10,0,10 in model space), However, I can't seem to figure out how to do this?
I would be very happy if anyone could help me out with the maths needed to do this (matrices were never my strongest side), or perhaps find a better solution.
To clarify; here is an image of what I want to do:
The red rectangle represent the device screen, the green x is the touch event and the black square is the board (grey subdivisions represent a square of one unit). I need to figure out where on the board the touch has happened (in this case it is in square 1,1).
As you are working in 2D basically already, (I presume you mean your 3D board stretches from 0,0,0 to 10,10,0 (x,y,z).) you could translate and interpolate/extrapolate the 2D/3D space coordinates from your screen space coordinates without the gluUnProject(). You will need your screen resolution, and to pick the resolution of the 3D space grid you wish to convert to. If both the screen and 3D space origins are aligned (0,0 screen space is at 0,0,0 3D space), and your screen dimensions are 320x240, using your existing 10x10 3D grid, then 320/10 = 32, and 240/10 = 24, thus the screen space size of a single 1x1 area is 32x24. So if the user presses on 162, 40, then the user is pressing within ( 5, 1, 0) (162/32 >= 5 but < 6, 40/24 >= 1 but < 2 ) in the 3D space. If you need greater resolution than this you can select a higher 3D space grid resolution (i.e. using 20 instead of 10). You don't need to update the GL matrix to use this factor. Though it may make it simpler in some ways, I'm sure from a modeling perspective you would have additional work to do. Just be aware for a factor like 20, 1,3 would be at (.5, 1.5, 0). If your screen and 3D space origins are not already aligned will need to translate the screen space coord prior to this. If 0,0 screen space is 10,10,0, you will need to take your screen resolution and subtract the current point from it, making 0,0 into 320, 240 in this example, our example point of 162, 40, would be 158 (320-158 == 162), 200 (240-200 == 40).
If you'd like an overview of the projection matrix and how that all works, which could help you understand where to put the screen space dimensions in the unproject matrix, read this chapter of the OpenGL red book. http://www.glprogramming.com/red/chapter03.html
Hope this helps, and good luck!
So, I managed to solve this by doing the following:
float[] clipPoint = new float[4];
int[] viewport = new int[]{0, 0, width, height};
//screenY/screenX are the screen-coordinates, y should be flipped:
screenY = viewport[3] - screenY;
//Calculate a z-value appropriate for the far clip:
float dist = 1.0f;
float z = (1.0f/clip[0] - 1.0f/dist)/(1.0f/clip[0]-1.0f/clip[1]);
//Use gluUnProject to create a 3d point in the far clip plane:
GLU.gluUnProject(screenX, screenY, z, vMatrix, 0, pMatrix, 0, viewport, 0, clipPoint, 0);
//Get a point representing the 'camera':
float eyeX = lookat[0] + eyeOffset[0];
float eyeY = lookat[1] + eyeOffset[1];
float eyeZ = lookat[2] + eyeOffset[2];
//Do some magic to calculate where the line between clipPoint and eye/camera would intersect the y-plane:
float dX = eyeX - clipPoint[0];
float dY = eyeY - clipPoint[1];
float dZ = eyeZ - clipPoint[2];
float resX = glu[0] - (dX/dY)*glu[1];
float resZ = glu[2] - (dZ/dY)*glu[1];
//resX and resZ is the wanted result.