I have downloaded and successfully run the example provided in opencv4android sdk.
I am able to simply display the camera frames without any processing,
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
return inputFrame.rgba();
}
I want to process live frame with some predefined image template to recognize that template. I have taken reference from this post and implemented accordingly. But I get black screen only.
private Mat mCameraMat = new Mat();
private Mat mTemplateMat;
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
mCameraMat = inputFrame.rgba();
initialize();
int match_method = Imgproc.TM_SQDIFF;
// Create the result matrix
int result_cols = mCameraMat.cols() - mTemplateMat.cols() + 1;
int result_rows = mCameraMat.rows() - mTemplateMat.rows() + 1;
Log.d(TAG, " mCameraMat cols "+mCameraMat.cols());
Log.d(TAG, " mCameraMat rows "+mCameraMat.rows());
Log.d(TAG, " mTemplateMat cols "+mTemplateMat.cols());
Log.d(TAG, " mTemplateMat rows "+mTemplateMat.rows());
Mat result = new Mat(result_rows, result_cols, CvType.CV_32F);
// Do the Matching and Normalize
Imgproc.matchTemplate(mCameraMat, mTemplateMat, result, match_method);
Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());
// Localizing the best match with minMaxLoc
MinMaxLocResult mmr = Core.minMaxLoc(result);
Point matchLoc;
if (match_method == Imgproc.TM_SQDIFF || match_method == Imgproc.TM_SQDIFF_NORMED) {
matchLoc = mmr.minLoc;
} else {
matchLoc = mmr.maxLoc;
}
Rect roi = new Rect((int) matchLoc.x, (int) matchLoc.y, mTemplateMat.cols(), mTemplateMat.rows());
Core.rectangle(mCameraMat, new Point(roi.x, roi.y), new Point(roi.width - 2, roi.height - 2), new Scalar(255, 0, 0, 255), 2);
return result;
}
public void initialize(){
try {
if (mCameraMat.empty())
return;
if(mTemplateMat == null){
Mat temp = Utils.loadResource(Tutorial1Activity.this, R.drawable.icon);
mTemplateMat = new Mat(temp.size(), CvType.CV_32F);
Imgproc.cvtColor(temp, mTemplateMat, Imgproc.COLOR_BGR2RGBA);
Log.d(TAG, "initialize mTemplateMat cols "+mTemplateMat.cols());
Log.d(TAG, "initialize mTemplateMat rows "+mTemplateMat.rows());
}
} catch (IOException e) {
e.printStackTrace();
}
}
Note:
My ultimate goal is to recognize the playing cards from live camera. Kindly suggest best approach. Should I use image templates or any other thing to make things faster?
This is how I want to recognize multiple cards from live camera:
Result should be: ♠A ♠K ♠Q ♠J ♠10 when camera preview seems like below
Template matching is unlikely to be the best approach here.
Try aSIFT to do an affine invariant SIFT matching or a normal SIFT (OpenCV implementation exists). However,
since these are in C++, you may want to use JNI to make calls to it from Java on an Android device. This is probably the best way to detect the suit of the card from the 4 symbols.
Another option to detect and recognize the numbers/alphabets on the cards is to use a text detector like MSER and then a text recognizer on the regions of interest indicated by the MSER filter.
In any case, you are unlikely to be able produce the best of results from the kind of image you've shown in the image. You may be able to get acceptable performance for full frontal, upright images with the first method.
Related
The question comes 1st: I am looking for FAST approach to match images.
Now, the use case: I am developing a detector to detect orb on a 6x5 Match-3 game board for android platform. I have an array of the orb icon with transparent background, but the orb on the screen (screenshot) has different background color, probably different size too. I have to compare each orb on the screen with my array of icons (69 icons specifically) so it's a 69x30=2070 steps. I tried lazy implementation and group almost similar icon together to reduce the steps but still take a long time (10s at most) for computation. I also tried checking the channel and depth of image, resizing the images to have same size and tweaking the threshold value but still no luck.
I have tried Histogram Matching (seperate channel, grayscale), Template Matching (CCOEFF, SQDIFF, CCORR), AKAZE, ORB(unbounded, bounded), PHash all using OpenCV but histogram matching and PHash give me erroneous result (too much false positive), Template Matching consume 10s+ (considered too slow for user to wait) while AKAZE and ORB give better result than all other methods but still needs 6s+ per try. Is there any other method that can helps me cut down the computation time down to somewhere near 1s and can give better result considering the worst case scenario is 2070 steps?
Referrences that I have read that compares the performances of different feature matching algorithms:
A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK. It shows that ORB and BRISK should be averagely better than the other approach compared while AKAZE is moderately good for most cases. I deleted my Histogram comparison code as it is not really helpful but you may find the rest of it below.
Mat source = Utils.loadResource(this, R.drawable.orb_icon, Imgcodecs.CV_LOAD_IMAGE_UNCHANGED);
Mat tmp = new Mat();
Bitmap cropped_img = Bitmap.createBitmap(screenshot, x, y, width, height);
Utils.bitmapToMat(cropped_img, tmp);
//template matching code
int r_rows = source.rows() - tmp.rows() + 1;
int r_cols = source.cols() - tmp.cols() + 1;
Mat result = new Mat();
result.create(r_rows, r_cols, CvType.CV_32F);
Imgproc.matchTemplate(source, tmp, result, Imgproc.TM_CCOEFF_NORMED);
Core.MinMaxLocResult mmr = Core.minMaxLoc(result);
double maxVal = mmr.maxVal;
return maxVal;
//AKAZE
MatOfKeyPoint kp1 = new MatOfKeyPoint();
MatOfKeyPoint kp2 = new MatOfKeyPoint();
Mat desc1 = new Mat();
Mat desc2 = new Mat();
AKAZE akaze = AKAZE.create();
DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMING);
akaze.detectAndCompute(source, new Mat(), kp1, desc1);
akaze.detectAndCompute(tmp, new Mat(), kp2, desc2);
List<MatOfDMatch> knnMatches = new ArrayList<>();
matcher.knnMatch(desc1, desc2, knnMatches, 2);
float threshold = 0.7f;
int count = 0;
for(int i=0; i<knnMatches.size(); i++) {
if(knnMatches.get(i).rows() > 1) {
DMatch[] matches = knnMatches.get(i).toArray();
if(matches[0].distance < threshold * matches[1].distance) {
count++;
}
}
}
//ORB
ORB orb = ORB.create();
DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMING);
MatOfKeyPoint kp1 = new MatOfKeyPoint();
MatOfKeyPoint kp2 = new MatOfKeyPoint();
Mat desc1 = new Mat();
Mat desc2 = new Mat();
orb.detectAndCompute(source, new Mat(), kp1, desc1);
orb.detectAndCompute(tmp, new Mat(), kp2, desc2);
List<MatOfDMatch> knnMatches = new ArrayList<>();
matcher.knnMatch(desc1, desc2, knnMatches, 2);
float threshold = 0.8f;
int count = 0;
for(int i=0; i<knnMatches.size(); i++) {
if(knnMatches.get(i).rows() > 1) {
DMatch[] matches = knnMatches.get(i).toArray();
if(matches[0].distance < threshold * matches[1].distance) {
count++;
}
}
}
//PHash
Mat hash_source = new Mat();
Mat hash_tmp = new Mat();
Img_hash.pHash(tmp, hash_tmp);
Img_hash.pHash(source, hash_source);
Core.norm(source, tmp, Core.NORM_HAMMING);
Edit: As suggested, below is the game board, icon image, and orb screenshot sample.
ICON vs orb screenshot
Also, you may observe the simulation result of each approach by comparing the result(overlay smaller icon) on top of the orb on board:
Histogram Matching
,
Template Matching
and
AKAZE (similar to ORB)
After moving the variable initialization out of my comparison function to base class, detect keypoint and PHash of source icon images on class initialization, run detect and compute function in batch using List to reduce individual function call. It still takes up 4s+ for the image matching process. Time consumption is reduced but accuracy is still a major problem. You may observe my heap stack on below.
I am trying to make an automation android applicaton which finds an image(template/subimage) inside another image(main/bigger image).
Menu image is from onePlus 3T.
Whatsapp icon image is from motoG 3.
I have tried to find whatsapp image from oneplus 3T in its menu image and it was successfully found.
But, when i am trying to find some subimage from different device of different screensize, it is not working.
Can someone please help. Below is the code I am using.
class MatchingDemo {
public Mat run(Mat img, Mat templ, String outFile, int match_method) {
System.out.println("\nRunning Template Matching");
// / Create the result matrix
int result_cols = img.cols() - templ.cols() + 1;
int result_rows = img.rows() - templ.rows() + 1;
Mat result = new Mat(result_rows, result_cols, CvType.CV_32FC1);
// / Do the Matching and Normalize
Imgproc.matchTemplate(img, templ, result, match_method);
Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());
// / Localizing the best match with minMaxLoc
MinMaxLocResult mmr = Core.minMaxLoc(result);
Point matchLoc;
if (match_method == Imgproc.TM_SQDIFF || match_method == Imgproc.TM_SQDIFF_NORMED) {
matchLoc = mmr.minLoc;
} else {
matchLoc = mmr.maxLoc;
}
System.out.println("matchloc.x "+ matchLoc.x);
System.out.println("templ.cols "+ templ.cols());
System.out.println("matchloc.y "+ matchLoc.y);
System.out.println("templ.rows "+ templ.rows());
// / Show me what you got
Imgproc.rectangle(img, matchLoc, new Point(matchLoc.x + templ.cols(),
matchLoc.y + templ.rows()), new Scalar(0, 255, 0), 20);
// Save the visualized detection.
System.out.println("Writing "+ outFile);
Imgcodecs.imwrite(outFile, img);
return img;
}
}
I cropped the template image from your given snapshot and everything worked fine:
New template Image:
Code:
import cv2
import numpy as np
img_1 = cv2.imread("path/to/snapshot", 0)
img_rgb = cv2.imread("path/to/snapshot")
template_img = cv2.imread("path/to/template", 0)
h, w = template_img.shape
res = cv2.matchTemplate(img_1, template_img, cv2.TM_CCOEFF)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
cv2.rectangle(img_rgb, max_loc, (max_loc[0]+w, max_loc[1]+h), np.array([0, 0, 255]), 3)
cv2.imwrite("./debug.png", img_rgb)
Output:
Note: matchTemplate is a very basic implementation, to get more scale invariant results you may try with SIFT features
I'm using OpenCV4Android to try out template matching on an android phone. Essentially what I'm doing is taking a picture of some object that I wish to detect, cropping it and saving it as a template.
I then use my android camera and a surface view to constantly get images from the camera. I am then applying template matching on each image, converting the image to a Mat first. However, when applying template matching I only get around 3-4 fps.
What I am essentially doing is this:
mCameraMat = inputFrame.rgba();
int matchMethod = Imgproc.TM_CCOEFF_NORMED;
// mTemplateMat resized in terms of video size in prepareMediaRecorder.
// Very hacky solution so need to fix it!
int result_cols = mCameraMat.cols() - mTemplateMat.cols() + 1;
int result_rows = mCameraMat.rows() - mTemplateMat.rows() + 1;
mResult = new Mat(result_rows, result_cols, CvType.CV_32F);
// Move this to a new thread.
Imgproc.matchTemplate(mCameraMat, mTemplateMat, mResult, matchMethod);
Core.normalize(mResult, mResult, 0, 1, Core.NORM_MINMAX, -1, new Mat());
// Localizing the best match with minMaxLoc
MinMaxLocResult mmr = Core.minMaxLoc(mResult);
Point matchLoc;
if (matchMethod == Imgproc.TM_SQDIFF || matchMethod == Imgproc.TM_SQDIFF_NORMED) {
matchLoc = mmr.minLoc;
} else {
matchLoc = mmr.maxLoc;
}
// Draw a boundary around the detected object.
Imgproc.rectangle(mCameraMat, matchLoc, new Point(matchLoc.x + mTemplateMat.cols(),
matchLoc.y + mTemplateMat.rows()), new Scalar(TrackingActivity.r, TrackingActivity.g,
TrackingActivity.b, TrackingActivity.a), 2);
Where mTemplateMat is the template bitmap image converted into a Mat object.
The bottleneck is on the line
Imgproc.matchTemplate(mCameraMat, mTemplateMat, mResult, matchMethod);
If I remove that line, I get around 25 fps, which is much more acceptable. I'd be fine with anything above 13-14. I understand that template matching is a very expensive process and doing it every frame can be costly. I have tried to do it every 20 frames, but it still slows down the processing considerably, and the end video looks worse as there is a constant transition from a smooth fps display to a low fps display.
What are my options in optimising matchTemplate? Any tips are much appreciated.
I am new to OpenCV and am trying to count the number of objects in an image. I have done this before using MATLAB Image Processing Toolbox and adapted the same approach in OpenCV (Android) also.
The first step was to convert an image to gray scale. Then to threshold it and then counting the number of blobs. In Matlab there is a command - "bwlabel", which gives the number of blobs. I couldn't find such thing in OpenCV (again, I am a noob in OpenCV as well as Android).
Here is my code,
//JPG to Bitmap to MAT
Bitmap i = BitmapFactory.decodeFile(imgPath + "mms.jpg");
Bitmap bmpImg = i.copy(Bitmap.Config.ARGB_8888, false);
Mat srcMat = new Mat ( bmpImg.getHeight(), bmpImg.getWidth(), CvType.CV_8UC3);
Utils.bitmapToMat(bmpImg, srcMat);
//convert to gray scale and save image
Mat gray = new Mat(srcMat.size(), CvType.CV_8UC1);
Imgproc.cvtColor(srcMat, gray, Imgproc.COLOR_RGB2GRAY,4);
//write bitmap
Boolean bool = Highgui.imwrite(imgPath + "gray.jpg", gray);
//thresholding
Mat threshed = new Mat(bmpImg.getWidth(),bmpImg.getHeight(), CvType.CV_8UC1);
Imgproc.adaptiveThreshold(gray, threshed, 255, Imgproc.ADAPTIVE_THRESH_MEAN_C, Imgproc.THRESH_BINARY, 75, 5);//15, 8 were original tests. Casey was 75,10
Core.bitwise_not(threshed, threshed);
Utils.matToBitmap(threshed, bmpImg);
//write bitmap
bool = Highgui.imwrite(imgPath + "threshed.jpg", threshed);
Toast.makeText(this, "Thresholded image saved!", Toast.LENGTH_SHORT).show();
In the next step, I tried to fill the holes and letters using dilation followed by an erosion but the blobs gets attached to each other which will ultimately give a wrong count. There is a tradeoff between filling holes and getting the blobs attached to each other on tuning the parameters for dilation and erosion.
Here is the code,
//morphological operations
//dilation
Mat dilated = new Mat(bmpImg.getWidth(),bmpImg.getHeight(), CvType.CV_8UC1);
Imgproc.dilate(threshed, dilated, Imgproc.getStructuringElement(Imgproc.MORPH_ELLIPSE, new org.opencv.core.Size (16, 16)));
Utils.matToBitmap(dilated, bmpImg);
//write bitmap
bool = Highgui.imwrite(imgPath + "dilated.jpg", dilated);
Toast.makeText(this, "Dilated image saved!", Toast.LENGTH_SHORT).show();
//erosion
Mat eroded = new Mat(bmpImg.getWidth(),bmpImg.getHeight(), CvType.CV_8UC1);
Imgproc.erode(dilated, eroded, Imgproc.getStructuringElement(Imgproc.MORPH_ELLIPSE, new org.opencv.core.Size(15, 15)));
Utils.matToBitmap(eroded, bmpImg);
//write bitmap
bool = Highgui.imwrite(imgPath + "eroded.jpg", eroded);
Toast.makeText(this, "Eroded image saved!", Toast.LENGTH_SHORT).show();
Because sometimes my M&Ms might be just next to each other! ;)
I also tried to use Hough Circles but the result is very unreliable (tested with coin images as well as real coins)
Here is the code,
//hough circles
Mat circles = new Mat();
// parameters
int iCannyUpperThreshold = 100;
int iMinRadius = 20;
int iMaxRadius = 400;
int iAccumulator = 100;
Imgproc.HoughCircles(gray, circles, Imgproc.CV_HOUGH_GRADIENT,
1.0, gray.rows() / 8, iCannyUpperThreshold, iAccumulator,
iMinRadius, iMaxRadius);
// draw
if (circles.cols() > 0)
{
Toast.makeText(this, "Coins : " +circles.cols() , Toast.LENGTH_LONG).show();
}
else
{
Toast.makeText(this, "No coins found", Toast.LENGTH_LONG).show();
}
The problem with this approach is that the algorithm is limited to perfect circles only (AFAIK). So, it doesn't work well when I try to scan and count M&Ms or coins lying on my desk (because angle of the device changes). With this approach, sometimes I get less no. of coins detected and sometimes more (I don't get it why more??).
On scanning this image the app sometimes shows 19 coins and sometimes 38 coins counted...I know there are other features which may be detected as circles but I totally don't get it why 38..?
So my questions...
Is there a better way to fill holes without joining adjacent blobs?
How do I count the number of objects accurately? I don't want to limit my app to counting only circles with HoughCircles approach.
FYI : OpenCV-2.4.9-android-sdk. Kindly keep in mind that I am a newbie in OpenCV and Android too.
Any help is much appreciated.
Thanks & Cheers!
Jainam
So to proceed we take your threshold image which you have generated as input and further modify it. The present code is in C++ but I guess you can easily convert it into android platform
Now instead of dilation or blurring you can try flood fill
which results in
Finally now applying the contour detection algorithm algorithm we get
The code for the above is
Mat dst = imread($path to the threshold image); // image should be single channel black and white image
imshow("dst",dst);
cv::Mat mask = cv::Mat::zeros(dst.rows + 2, dst.cols + 2, CV_8U);
// A image with size greater than the present object is created
cv::floodFill(dst, mask, cv::Point(0,0), 255, 0, cv::Scalar(), cv::Scalar(), 4 + (255 << 8) + cv::FLOODFILL_MASK_ONLY);
erode(mask,mask,Mat());
// Now to remove the outer boundary
rectangle(mask,Rect(0,0,mask.cols,mask.rows), Scalar(255,255,255),2,8,0);
imshow("Mask",mask);
Mat copy;
mask.copyTo(copy);
vector<vector<Point> > contours;
vector<Vec4i> hierarchy;
findContours( copy, contours, hierarchy, CV_RETR_TREE, CV_CHAIN_APPROX_SIMPLE, Point(0, 0) );
vector<vector<Point> > contours_poly( contours.size() );
vector<Rect> boundRect( contours.size() );
vector<Point2f>center( contours.size() );
vector<float>Distance( contours.size() );
vector<float>radius( contours.size() );
Mat drawing = cv::Mat::zeros(mask.rows, mask.cols, CV_8U);
int num_object = 0;
for( int i = 0; i < contours.size(); i++ ){
approxPolyDP( Mat(contours[i]), contours_poly[i], 3, true );
// To get rid of the smaller object and the outer rectangle created
//because of the additional mask image we enforce a lower limit on area
//to remove noise and an upper limit to remove the outer border.
if (contourArea(contours_poly[i])>(mask.rows*mask.cols/10000) && contourArea(contours_poly[i])<mask.rows*mask.cols*0.9){
boundRect[i] = boundingRect( Mat(contours_poly[i]) );
minEnclosingCircle( (Mat)contours_poly[i], center[i], radius[i] );
circle(drawing,center[i], (int)radius[i], Scalar(255,255,255), 2, 8, 0);
rectangle(drawing,boundRect[i], Scalar(255,255,255),2,8,0);
num_object++;
}
}
cout <<"No. of object detected =" <<num_object<<endl;
imshow("drawing",drawing);
waitKey(2);
char key = (char) waitKey(20);
if(key == 32){
// You can save your images here using a space
}
I hope this helps you in solving your problem
Just check it out,
Blur source.
Threshold binary inverted on gray.
Find contours, note that you should use CV_RETR_EXTERNAL as contour retrieval mode.
You can take the contours size as your object count.
Code:
Mat tmp,thr;
Mat src=imread("img.jpg",1);
blur(src,src,Size(3,3));
cvtColor(src,tmp,CV_BGR2GRAY);
threshold(tmp,thr,220,255,THRESH_BINARY_INV);
imshow("thr",thr);
vector< vector <Point> > contours; // Vector for storing contour
vector< Vec4i > hierarchy;
findContours( thr, contours, hierarchy,CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE ); // Find the contours in the image
for( int i = 0; i< contours.size(); i=hierarchy[i][0] ) // iterate through each contour.
{
Rect r= boundingRect(contours[i]);
rectangle(src,r, Scalar(0,0,255),2,8,0);
}
cout<<"Numeber of contour = "<<contours.size()<<endl;
imshow("src",src);
waitKey();
I'm trying to match an image with the camera input in Android using template matching. When i try this with static 2 images like in here: OpenCV Template Matching example in Android, everything works just fine. But when I try to use the captured images from the camera, I do not get the correct result. Following is the code that I have written:
String baseDir = Environment.getExternalStorageDirectory().getAbsolutePath();
Mat img = Highgui.imread(baseDir + "/mediaAppPhotos/img2.png");
Mat templ = Highgui.imread(baseDir+ "/mediaAppPhotos/chars.png");
int result_cols = img.cols() - templ.cols() + 1;
int result_rows = img.rows() - templ.rows() + 1;
Mat result = new Mat(result_cols, result_rows, CvType.CV_32FC1);
// / Do the Matching and Normalize
Imgproc.matchTemplate(img, templ, result, Imgproc.TM_CCOEFF);
Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1,
new Mat());
// / Localizing the best match with minMaxLoc
MinMaxLocResult mmr = Core.minMaxLoc(result);
Point matchLoc;
if (Imgproc.TM_CCOEFF == Imgproc.TM_SQDIFF
|| Imgproc.TM_CCOEFF == Imgproc.TM_SQDIFF_NORMED) {
matchLoc = mmr.minLoc;
} else {
matchLoc = mmr.maxLoc;
}
// / Show me what you got
Core.rectangle(
img,
matchLoc,
new Point(matchLoc.x + templ.cols(), matchLoc.y
+ templ.rows()), new Scalar(0, 255, 0));
// Save the visualized detection.
System.out.println("Writing " + baseDir+ "/mediaAppPhotos/result.png");
Highgui.imwrite(baseDir + "/mediaAppPhotos/result.png", img);
I want to this template matching to work when the image is captured from the camera as well. Any help is greatly appreciated!
maybe is like this:
https://play.google.com/store/apps/details?id=in.mustafaak.imagematcher&hl=es_419
code available in github