I am using Android OpenCV to detect features from the input frame of the camera. I am using the ORB feature detector and ORB descriptor extractor with BFMatcher. Now I got some matches Mat in the format of
matches = Mat [ 421*1*CV_32FC4, isCont=true, isSubmat=false, nativeObj=0x5fad7b30, dataAddr=0x5fab84f0 ]
I wonder what is the nativeObj and dataAddr represented for? I want to get the distance between two matched features in pixel, any idea?
I have found someone else had the same question and there was no reply. How to Access Points location on OpenCV Matcher?
Thanks in advance!
When you are performing match of descriptors of features you should get MatOfDMatch. Your code should be like this:
DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMING);
MatOfDMatch matches = new MatOfDMatch();
matcher.match(descriptors1,descriptors2 ,matches);
After you can transform MatOfDMatch to the List<DMatch> for easier manipulation. You can do it with :
List<DMatch> matchesList = matches.toList();
Then you can access to the matched points and obtain the Cartesian coordinates with:
Point pt1 = keypoints1.toList().get(matchesList.get(i).queryIdx).pt;
Point pt2 = keypoints2.toList().get(matchesList.get(i).trainIdx).pt;
After just calculate the distance between two points:
double dist_x_pow = Math.pow(Math.abs(pt1.x - pt2.x),2);
double dist_y_pow = Math.pow(Math.abs(pt1.y - pt2.y),2);
double DISTANCE = Math.sqrt(dist_x_pow + dist_y_pow);
About nativeObj and dataAddr I'm not sure, but I think it is related to the fact that OpenCV library is implemented in C and I think that this values represent address in memory of the Mat object.
Related
I'm trying to build a simple leaf recognition app with Android and OpenCV; my database consist in just 3 entries (3 pictures of 3 types of leaves) and I would like to be able to recognise if one of the pictures in the database is inside another picture captured by the smartphone.
I'm using the SURF method for extract keypoints from the database images and then compared them with the extracted keypoints of the captured image looking for a match.
My problem is that the result appears as a "color matching", more than a "feature matching": when I compare a picture from the database and the one captured, the number of matches is equal with all 3 entries and thus I get a wrong matching.
This one of the picture from the database (note that is without backgroud)
And this is the result that I get:
Image on top is the one captured from the smartphone and the image below is the result with matches highlighted.
Here is the code that I implemented:
Mat orig = Highgui.imread(photoPathwithoutFile);
Mat origBW = new Mat();
Imgproc.cvtColor(orig, origBW, Imgproc.COLOR_RGB2GRAY);
MatOfKeyPoint kpOrigin = createSURFdetector(origBW);
Mat descOrig = extractDescription(kpOrigin, origBW);
Leaf result = findMatches(descOrig);
Mat imageOut = orig.clone();
Features2d.drawMatches(orig, kpOrigin, maple, keypointsMaple, resultMaple, imageOut);
public MatOfKeyPoint createSURFdetector (Mat origBW) {
FeatureDetector surf = FeatureDetector.create(FeatureDetector.FAST);
MatOfKeyPoint keypointsOrig = new MatOfKeyPoint();
surf.detect(origBW, keypointsOrig);
return keypointsOrig;
}
public Mat extractDescription (MatOfKeyPoint kpOrig, Mat origBW) {
DescriptorExtractor surfExtractor = DescriptorExtractor.create(FeatureDetector.SURF);
Mat origDesc = new Mat();
surfExtractor.compute(origBW, kpOrig, origDesc);
return origDesc;
}
public Leaf findMatches (Mat descriptors) {
DescriptorMatcher m = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE);
MatOfDMatch max = new MatOfDMatch();
resultMaple = new MatOfDMatch();
resultChestnut = new MatOfDMatch();
resultSwedish = new MatOfDMatch();
Leaf match = null;
m.match(descriptors, mapleDescriptors, resultMaple);
Log.d("Origin", resultMaple.toList().size()+" matches with Maples");
if (resultMaple.toList().size() > max.toList().size()) { max = resultMaple; match = Leaf.MAPLE; }
m.match(descriptors, chestnutDescriptors, resultChestnut);
Log.d("Origin", resultChestnut.toList().size()+" matches with Chestnut");
if (resultChestnut.toList().size() > max.toList().size()) { max = resultChestnut; match = Leaf.CHESTNUT; }
m.match(descriptors, swedishDescriptors, resultSwedish);
Log.d("Origin", resultSwedish.toList().size()+" matches with Swedish");
if (resultSwedish.toList().size() > max.toList().size()) { max = resultSwedish; match = Leaf.SWEDISH; }
//return the match object with more matches
return match;
}
How can I get a more accurate matching not based on colours but on actual singularities of the picture?
Well, SURF is not the best candidate for this task. SURF descriptor basically encodes some gradient statistics in a small neighborhood of a corner. This gives you invariance to lot of transformations, but you lose the 'big picture' when doing this. This descriptor is used to narrow down a range of correspondences between points to be matched, and then some geometric contraints come into play.
In your case it seems that descriptors are not doing a great job at matching points, and since there are a LOT of them each point eventually gets a match (although it is strange that geometric testing didn't prevent that).
I can advice you to try different approach to matching, maybe HOG with descriptors trained to detect leaf types, or even something contour-based, since shape is what is really different between your images. You can for example detect leaf's outline, normalize it's length, find it's center and then in equal intervals calculate distance from each point to the center - that will be your descriptor. Than find the largest length and circularly shift this descriptor to start at the extrema and divide by this value - that will give you some basic invariance to choice of contour starting point, rotation and scale. But that will most likely fail under perspective and affine transformations.
If you would like to experiment further with feature points - try to detect less of them ,but more representative ones (filter by gradient strength, corner score or something). Maybe use SIFT instead of SURF - it should be a bit more precise. Check for amount of inliers after matching - best match should have higher ratio.
But honestly, this seems more like a machine learning task than computer vision.
Edit: I have checked your code and found out that you are not performing geometric checks on matches, hence why you are getting incorrect match. Try performing findHomography after matching and then consider only points that have been set to one in mask output argument. This will make you only consider points that can be warped to each other using homography and may improve matching a lot.
Edit2: added a code snippet (sorry, but I can't test Java at the moment, so it's in Python)
import cv2
import numpy as np
# read input
a = cv2.imread(r'C:\Temp\leaf1.jpg')
b = cv2.imread(r'C:\Temp\leaf2.jpg')
# convert to gray
agray = cv2.cvtColor(a, cv2.COLOR_BGR2GRAY)
bgray = cv2.cvtColor(b, cv2.COLOR_BGR2GRAY)
# detect features and compute descriptors
surf = cv2.SURF() # better use SIFT instead
kp1, d1 = surf.detectAndCompute(agray,None)
kp2, d2 = surf.detectAndCompute(bgray,None)
print 'numFeatures1 =', len(kp1)
print 'numFeatures2 =', len(kp2)
# use KNN matcher
bf = cv2.BFMatcher()
matches = bf.knnMatch(d1,d2, k=2)
# Apply Lowe ratio test
good = []
for m,n in matches:
if m.distance < 0.75*n.distance:
good.append(m)
print 'numMatches =', len(matches)
print 'numGoodMatches =', len(good)
# if have enough matches - try to calculare homography to discard matches
# that don't fit perspective transformation model
if len(good)>10:
# convert matches into correct format (python-specific)
src_pts = np.float32([ kp1[m.queryIdx].pt for m in good ]).reshape(-1,1,2)
dst_pts = np.float32([ kp2[m.trainIdx].pt for m in good ]).reshape(-1,1,2)
M, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC,5.0)
print 'numMatches =', sum(mask.ravel().tolist()) # calc number of 1s in mask
else:
print "not enough good matches are found"
It gives me following output for different leaves using SURF
numFeatures1 = 685
numFeatures2 = 1566
numMatches = 685
numGoodMatches = 52
numMatches = 11
You can see that the amount of 'real' matches is very small. But unfortunately numMatches is similar when we match different images of same leaf type. Maybe you can improve the result by tweaking parameters, but I think using keypoints here is just a not very good approach. Maybe it is due to the leaf variation even within a same class.
I want to calculate Standard Deviation of GRAYSCALE image. To do it I'm using this function:
MatOfDouble mean = new MatOfDouble();
MatOfDouble std = new MatOfDouble();
Core.meanStdDev(image, mean, std);
According to documentation:
http://docs.opencv.org/java/org/opencv/core/Core.html#meanStdDev(org.opencv.core.Mat,org.opencv.core.MatOfDouble,org.opencv.core.MatOfDouble)
I'm getting
mean - output parameter: calculated mean value
stddev - output parameter: calculateded standard deviation
But mean and stddev are both matrix. How can I get single double value from it?
In OpenCV for Android, the function org.opencv.Calib3d.findHomography(..) returns the homogeneous transformation matrix. For example, this only returns the homography:
Mat homography = Calib3d.findHomography(points1, points2, Calib3d.RANSAC, 0.5);
Is there a way to return the points that RANSAC actually uses from the Android OpenCV API?
Update
I am not sure whether it's a new addition to OpenCV or I've just missed it, but the findHomography() function actually can give you the inliers (OpenCV 2.4.2). The last parameter, mask, which is empty by default, will be filled with ones (or 255) at the indexes of the inliers foound by RANSAC.
Mat findHomography(InputArray srcPoints, InputArray dstPoints,
int method=0, double ransacReprojThreshold=3, OutputArray mask=noArray() )
// ^
// |
Old answer
The points used by RANSAC to estimate the homography (called inliers in technical docs) cannot be extracted directly. They are computed internally, but then the list is deleted.
A way to extract them is to modify the findHomography function (and the corresponding RANSAC functions). But this is ugly.
Another, cleaner way is to test what point pairs in the input match th homography:
use the projectPoints(points1, homography, points1_dest) (i hope this is the function name) to apply homography to points1.
The correct function name and input arguments order is:
void perspectiveTransform(InputArray src, OutputArray dst, InputArray m), in this case cv::perspectiveTransform(points1, points1_dest, homography)
OpenCV Perspective Transform
use cv::distance(points1_dest, points2)
The correct function name and input arguments order is:
double norm(InputArray src1, int normType=NORM_L2, InputArray mask=noArray())
possible implementation:
std::array<cv::Point2f, 1> pt1;
pt1[0] = points1_dest;
std::array<cv::Point2f, 1> pt2;
pt2[0] = points2;
distance = cv::norm(pt1, pt2));
OpenCV norm function
Distance between two points can be also calculated using Pythagorean theorem
to see which of them are close enough to their pair in points2. The distance should be smaller or equal to min_distance^2. In your case, 0.5*0.5 = 0.25.
I'm using openCV for android to implement a logo detection algorithm. my goal now is to find a predefined logo in a picture I've taken with the android camera.
I can't get ANY right matches.. I think this is very weird considering I'm almost only using openCV library functions.
First I detect keypoints using FAST detector, my images are 500x500 in size
afterwards I use SURF to describe these keypoints.
with knn I ask for the 2 best matches, and elliminate those who don't have A ratio smaller than 0.6 (first.distance/ second.distance).
I'm getting around 10 matches, but they are all wrong, when I draw every match (100+), they all seem to be wrong
I can't see what I'm doing wrong here, does anyone have the same problem, or know what I'm doing wrong?
FeatureDetector FAST = FeatureDetector.create(FeatureDetector.FAST);
// extract keypoints
FAST.detect(image1, keypoints);
FAST.detect(image2, logoKeypoints);
DescriptorExtractor SurfExtractor = DescriptorExtractor
.create(DescriptorExtractor.SURF);
Mat descriptors = new Mat();
Mat logoDescriptors = new Mat();
SurfExtractor.compute(image1, keypoints, descriptors);
SurfExtractor.compute(image2, logoKeypoints, logoDescriptors);
List<DMatch> matches = new ArrayList<DMatch>();
matches = knn(descriptors, logoDescriptors);
Scalar blue = new Scalar(0, 0, 255);
Scalar red = new Scalar(255, 0, 0);
Features2d.drawMatches(image2, logoKeypoints, image1, keypoints,
matches, rgbout, blue, red);
I think the problem is the matcher you are using. For floatbased such as (SURF)descriptors use FLANN as a matcher or BRUTEFORCE as a matcher. Also strive to use the same feature descriptor for both extraction and matching...i.e SURF features on SURF keypoints.
Read this post on stackoverflow,and the articles linked in it for better understanding.
How Does OpenCV ORB Feature Detector Work?
Is it normal? When I tried the brute force matcher, the result is consistent everytime, but flann is not.
A small amount of keypoints will match to different places. I am writing the code using the Android wrapper, the keypoint detector and descriptor is SURF, something like this:
Mat queryDescriptors = new Mat();
Mat trainDescriptors = new Mat();
DescriptorExtractor surfDE = DescriptorExtractor.create(DescriptorExtractor.SURF);
surfDE.compute(queryImage, queryKeyPoints, queryDescriptors);
surfDE.compute(trainImage, trainKeyPoints, trainDescriptors);
DescriptorMatcher dm = DescriptorMatcher.create(DescriptorMatcher.FLANNBASED);
List<DMatch> matches = new ArrayList<DMatch>();
dm.match(queryDescriptors, trainDescriptors, matches);
FLANN stands for Fast Library for Approximate Nearest Neighbors. The approximate nearest neighbor algorithms are non-deterministic, often randomized KD-trees.
According to Andrey, this is the reason! So yes, it is normal. To find more one would have to dissect the algorithms!