Develop Reference

ObjectDetection Android using yolo in openCV

I am a beginner in using OpenCV in android. I want to detect the objects using Yolo. The thing is
I don't want to use OpenCV's camera listener onCameraFrame for getting frames.
I want to use onPreviewFrame listener. As you know we go the frames in byte[].
So I don't know how to connect frames in byte[] with Yolo using OpenCV.
Here is the code that I used and it is working well
#Override
public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
Mat frame = inputFrame.rgba();
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2RGB);
Mat imageBlob = Dnn.blobFromImage(frame, 0.00392, new Size(416, 416), new Scalar(0, 0, 0), false, false);
tinyYOLO.setInput(imageBlob);
List<Mat> result = new ArrayList<>(2);
List<String> outBlobNames = new ArrayList<>();
outBlobNames.add(0, "yolo_16");
outBlobNames.add(1, "yolo_23");
tinyYOLO.forward(result, outBlobNames);
.
.
.//SOME CODE HERE
.
And now I want to use onPreviewFrame:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
if (data == null)
throw new NullPointerException();
//---------------------HERE I Don't know what to do---------------------
Camera.Size size = camera.getParameters().getPreviewSize();
Mat frame = new Mat(size.width,size.width CvType.CV_8UC1);
frame.put(0, 0, data);
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2RGB);
//---------------------UP-------------------------------
Mat imageBlob = Dnn.blobFromImage(frame, 0.00392, new Size(416, 416), new Scalar(0, 0, 0),false, false);
tinyYOLO.setInput(imageBlob);
final List<Mat> result = new ArrayList<>(2);
final List<String> outBlobNames = new ArrayList<>();
outBlobNames.add(0, "yolo_16");
outBlobNames.add(1, "yolo_23");
tinyYOLO.forward(result, outBlobNames);
.
.
.//SOME CODE HERE
.
And with the second Code I got errors:
2020-01-23 15:26:55.279 30268-30268/com.thelonecoder.camera E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.thelonecoder.camera, PID: 30268
CvException [org.opencv.core.CvException: cv::Exception: OpenCV(3.4.5) /build/3_4_pack-android/opencv/modules/imgproc/src/color.hpp:255: error: (-2:Unspecified error) in function 'cv::CvtHelper::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::Set<3, 4>; VDcn = cv::Set<3, 4>; VDepth = cv::Set<0, 2, 5>; cv::SizePolicy sizePolicy = (cv::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
Invalid number of channels in input image:
'VScn::contains(scn)'
where
'scn' is 1
]

Finally, I found my answer:
I should have used 3 channels.
Mat frame = new Mat(size.height + size.height / 2, size.width, CvType.CV_8UC3);
frame.put(0, 0, data);
Imgproc.cvtColor(frame, frame, Imgproc.COLOR_RGBA2RGB);

converting a Mat into Bitmap causes an error

I am working with OpenCV4Android version 2.4.11, and I am trying to detect cornrs in the image. For that purpose, I am using Harris corner detector.the problem i am facing is, after detecting the corners in the image as shown below
in the code I want to display the image that contains the detected corners after converting it into Bitmap, but then i receive the below posted error.
step 1 and step 2 are executed without errors, but when i run step 3 i get the posted logcat error.
Please let mek know why i am receiving this error and how to solve it?
code:
//step 1
this.mMatGray = new Mat();
Imgproc.cvtColor(this.mMatInputFrame, this.mMatGray, Imgproc.COLOR_BGR2GRAY);
//step 2
Imgproc.cornerHarris(mMatGray,mMatGray,3,3,3,1);
//step 3
final Bitmap bitmap = Bitmap.createBitmap(mMatInputFrame.cols(), mMatInputFrame.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mMatGray, bitmap);
getActivity().runOnUiThread(new Runnable() {
#Override
public void run() {
mIVEdges.setImageBitmap(bitmap);
}
});
error:
OpenCV Error: Assertion failed (src.type() == CV_8UC1 || src.type() == CV_8UC3 || src.type() == CV_8UC4) in void Java_org_opencv_android_Utils_nMatToBitmap2(JNIEnv*, jclass, jlong, jobject, jboolean), file /hdd2/buildbot/slaves/slave_ardbeg1/50-SDK/opencv/modules/java/generator/src/cpp/utils.cpp, line 98
E/org.opencv.android.Utils: nMatToBitmap catched cv::Exception: /hdd2/buildbot/slaves/slave_ardbeg1/50-SDK/opencv/modules/java/generator/src/cpp/utils.cpp:98: error: (-215) src.type() == CV_8UC1 || src.type() == CV_8UC3 || src.type() == CV_8UC4 in function void Java_org_opencv_android_Utils_nMatToBitmap2(JNIEnv*, jclass, jlong, jobject, jboolean)
E/AndroidRuntime: FATAL EXCEPTION: Thread-2477
Process: com.example.cornerdetection_00, PID: 22407
CvException [org.opencv.core.CvException: /hdd2/buildbot/slaves/slave_ardbeg1/50-SDK/opencv/modules/java/generator/src/cpp/utils.cpp:98: error: (-215) src.type() == CV_8UC1 || src.type() == CV_8UC3 || src.type() == CV_8UC4 in function void Java_org_opencv_android_Utils_nMatToBitmap2(JNIEnv*, jclass, jlong, jobject, jboolean)
]
at org.opencv.android.Utils.nMatToBitmap2(Native Method)
at org.opencv.android.Utils.matToBitmap(Utils.java:123)
at org.opencv.android.Utils.matToBitmap(Utils.java:132)
at com.example.cornerdetection_00.FragOpenCVCam.cornerHarris(FragOpenCVCam.java:204)
at com.example.cornerdetection_00.FragOpenCVCam.onCameraFrame(FragOpenCVCam.java:159)
at org.opencv.android.CameraBridgeViewBase.deliverAndDrawFrame(CameraBridgeViewBase.java:387)
at org.opencv.android.JavaCameraView$CameraWorker.run(JavaCameraView.java:346)
update1:
now after testing the following code provided by "MIkka Marmik" i recieve the errors below
code:
Mat mMatGray = new Mat();
Imgproc.cvtColor(mMatGray, mMatGray, Imgproc.COLOR_BGR2GRAY);
//step 2
Imgproc.cornerHarris(mMatGray,mMatGray,3,3,3,1);
//step 3
final Bitmap bitmap = Bitmap.createBitmap(mMatGray.cols(), mMatGray.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mMatGray, bitmap);
error:
OpenCV Error: Assertion failed (scn == 3 || scn == 4) in void cv::cvtColor(cv::InputArray, cv::OutputArray, int, int), file /hdd2/buildbot/slaves/slave_ardbeg1/50-SDK/opencv/modules/imgproc/src/color.cpp, line 3739
E/org.opencv.imgproc: imgproc::cvtColor_11() caught cv::Exception: /hdd2/buildbot/slaves/slave_ardbeg1/50-SDK/opencv/modules/imgproc/src/color.cpp:3739: error: (-215) scn == 3 || scn == 4 in function void cv::cvtColor(cv::InputArray, cv::OutputArray, int, int)
E/AndroidRuntime: FATAL EXCEPTION: Thread-4693

The createBitmap method is expecting an image which is either 1,3 or 4 channels. You must convert your output mMatGray to the format required for the createBitmap method. However, after checking the OpenCV documentation for the Harris corner algorithm, there are a few more steps involved to set up your image for corner detection. Try this out:
Mat src_gray = new Mat();
Mat dst = new Mat();
Mat dst_norm = new Mat();
Mat dst_norm_scaled = new Mat();
// Detector parameters
int blockSize = 2;
int apertureSize = 3;
double k = 0.04;
// Filter params
int thresh = 200;
int max_thresh = 255;
// Detecting corner
Imgproc.cvtColor(src, src_gray, Imgproc.COLOR_BGR2GRAY);
Imgproc.cornerHarris(src_gray, dst, blockSize, apertureSize, k);
// Normalizing
Core.normalize(dst, dst_norm, 0, 255, Core.NORM_MINMAX);
Core.convertScaleAbs(dst_norm, dst_norm_scaled);
// Drawing a circle around corners
for (int j = 0; j < dst_norm.rows(); j++) {
for (int i = 0; i < dst_norm.cols(); i++) {
if (dst_norm.get(j, i)[0] > thresh) {
Core.circle(dst_norm_scaled, new Point(i, j), 5, new Scalar(255));
}
}
}
// Create bitmap
final Bitmap bitmap = Bitmap.createBitmap(dst_norm_scaled.cols(), dst_norm_scaled.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(dst_norm_scaled, bitmap);

try this
Mat mMatGray = new Mat();
mMatGray = Highgui.imread(path);
Imgproc.cvtColor(mMatGray, mMatGray, Imgproc.COLOR_RGB2BGR);
Imgproc.cvtColor(mMatGray, mMatGray, Imgproc.COLOR_BGR2GRAY);
//step 2
Imgproc.cornerHarris(mMatGray,mMatGray,3,3,3,1);
Core.normalize(mMatGray, mMatGray, 0, 255, Core.NORM_MINMAX, CvType.CV_32FC1, new Mat());
Core.convertScaleAbs(mMatGray, mMatGray);
//step 3
final Bitmap bitmap = Bitmap.createBitmap(mMatGray.cols(), mMatGray.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mMatGray, bitmap);

Error Converting a CtvColor on Android and OpenCV on Eclipse

i'm developing to Android using openCV in Eclipse. I'm trying do template matching Frame a Frame. I can't convert the template and doing the match template on it. I'm using that function:
public void initialize(){
if (src.empty())
return;
if(template == null){
Mat templ = Highgui.imread(getFileAbsPath("1.png",
Highgui.CV_LOAD_IMAGE_UNCHANGED);
template = new Mat(templ.size(), CvType.CV_32F);
Imgproc.cvtColor(templ, (Mat) template, Imgproc.COLOR_BGR2RGBA);
}
}
private String getFileAbsPath(String fileName) {
File f = new File(cacheDir, fileName);
return f.getAbsolutePath();
}
I get a error on:
Imgproc.cvtColor(templ, (Mat) template, Imgproc.COLOR_BGR2RGBA);
My image is that (it's number 1):
https://drive.google.com/file/d/0B5tH_Qo3-GvhaV9QSTUteXFiQmM/view?usp=sharing
Next, I've my method:
#Override
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
src = inputFrame.rgba();
initialize();
int match_method = Imgproc.TM_SQDIFF;
// Create the result matrix
int result_cols = src.cols() - ((Mat) template).cols() + 1;
int result_rows = src.rows() - ((Mat) template).rows() + 1;
Mat result = new Mat(result_rows, result_cols, CvType.CV_32F);
// Do the Matching and Normalize
Imgproc.matchTemplate(src, (Mat) template, result, match_method);
Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());
MinMaxLocResult mmr = Core.minMaxLoc(result);
Point matchLoc;
if (match_method == Imgproc.TM_SQDIFF || match_method == Imgproc.TM_SQDIFF_NORMED) {
matchLoc = mmr.minLoc;
} else {
matchLoc = mmr.maxLoc;
}
Rect roi = new Rect((int) matchLoc.x, (int) matchLoc.y, ((Mat) template).cols(), ((Mat) template).rows());
Core.rectangle(src, new Point(roi.x, roi.y), new Point(roi.width - 2, roi.height - 2), new Scalar(255, 0, 0, 255), 2);
return src;
}
I get a error on that line:
Imgproc.matchTemplate(src, (Mat) template, result, match_method);
I can't do the match, don't know why ... Cans someone help me ?

How to create and display histogram in OpenCV4Android?

Can anybody tell me what am I doing wrong or give me example how to create AND DISPLAY histogram from grayscale mat in OpenCV4 Android? There is my code...
Vector<Mat> bgr_planes = new Vector<Mat>();
Core.split(cleanFaceMatGRAY, bgr_planes);
MatOfInt histSize = new MatOfInt(256);
final MatOfFloat histRange = new MatOfFloat(0f, 256f);
boolean accumulate = false;
Mat b_hist = new Mat();
Imgproc.calcHist(bgr_planes, new MatOfInt(0),new Mat(), b_hist, histSize, histRange, accumulate);
And then I try to set it to Image view to see it:
ivPickedPhoto.setImageBitmap(AppTools.createBitmapFromMat(b_hist, Bitmap.Config.ARGB_8888));
Method for display was created on my own:
public class AppTools {
public static Bitmap createBitmapFromMat(Mat mat, Bitmap.Config config) {
Bitmap bmp2 = null;
bmp2 = Bitmap.createBitmap(mat.cols(), mat.rows(), config);
Utils.matToBitmap(mat, bmp2);
return bmp2;
}
This gives me an error in the ivPickedPhoto(...) lane:
12-10 21:12:59.910: E/AndroidRuntime(10301): Caused by: CvException [org.opencv.core.CvException: /home/reports/ci/slave_desktop/50-SDK/opencv/modules/java/generator/src/cpp/utils.cpp:98: error: (-215) src.type() == CV_8UC1 || src.type() == CV_8UC3 || src.type() == CV_8UC4 in function void Java_org_opencv_android_Utils_nMatToBitmap2(JNIEnv*, jclass, jlong, jobject, jboolean)
I guess my problem is with the way I want to display the histogram... prolly I cant convert it to bmp. What is a good way to show histogram?

The OpenCV Java Documentation states that the Mat object you are passing should be of type CV_8UC1, CV_8UC3 or CV_8UC4. The documentation of calcHist doesn't really specify which type you should expect your b_Hist to be, but since it is complaining about the Mat type:
error: (-215) src.type() == CV_8UC1 || src.type() == CV_8UC3 || src.type() == CV_8UC4
you should probably convert it to either of the three above formats.

Android OpenCV FeatureDetection -> runFeatureHomography giving messed up result image after matching

I'm pretty new to openCV, and at the moment im tryin to get an object detection running on an android device. What i basically do, is displaying a a camera preview in my app, and when i click on it it captures a picture. Then this picture is given to the runFeatureHomography - method, which first grabs the second image to which the taken image has to be compared. Then the method finds the keypoints in both pictures, computes them and matches them into one Mat called img_matches.
As basic as it can be i guess.
The object im trying to detect here at the moment is some kind of card, just like the format of a credit card. The card is blue and has lots of white and yellow text on it. I can only post one link, thats why i cant show pictures of them.
I dont know why, but when i display the result in the end / or save the result as a bitmap to my phone, it always looks kinda like this:
http://oi44.tinypic.com/oaqel0.jpg <-- result image after everything is done.
This shows me that the object i wanted to detect was indeed detected, but i dont know why there is a black background and not the pictures of the cards. Why doesn't it show my two images the way the are, just with all the lines on them?
In my code im using those three:
FeatureDetector detector = FeatureDetector.create(FeatureDetector.ORB);
DescriptorExtractor extractor = DescriptorExtractor.create(DescriptorExtractor.ORB);
DescriptorMatchermatcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE);
Here is some of my code:
private void runFeatureHomography(Bitmap image)
{
Mat img_object = getObjectImage();
Mat img_scene = newEmptyMat();
Bitmap myimg = image.copy(Bitmap.Config.ARGB_8888, true);
Utils.bitmapToMat(myimg, img_scene);
MatOfKeyPoint keyPoints_object = detectObjectKeyPoints();
MatOfKeyPoint keyPoints_scene = new MatOfKeyPoint();
this.detector.detect(img_scene, keyPoints_scene);
Mat descriptors_object = calculateObjectDescriptor();
Mat descriptors_scene = newEmptyMat();
this.extractor.compute(img_scene, keyPoints_scene, descriptors_scene);
MatOfDMatch matches = new MatOfDMatch();
matcher.match(descriptors_object, descriptors_scene, matches);
double min_dist = Double.MAX_VALUE;
for (int i = -1; ++i < descriptors_object.rows();)
{
double dist = matches.toArray()[i].distance;
if (dist < min_dist)
{
min_dist = dist;
}
}
List<DMatch> good_matches = new ArrayList<DMatch>();
for (int i = -1; ++i < descriptors_object.rows();)
{
if (matches.toArray()[i].distance <= 3 * min_dist)
{
good_matches.add(matches.toArray()[i]);
}
}
System.out.println("4");
Mat img_matches = newEmptyMat();
Features2d.drawMatches(
img_object,
keyPoints_object,
img_scene,
keyPoints_scene,
new MatOfDMatch(good_matches.toArray(new DMatch[good_matches
.size()])), img_matches, Scalar.all(-1),
Scalar.all(-1), new MatOfByte(),
Features2d.NOT_DRAW_SINGLE_POINTS);
List<Point> object = new ArrayList<Point>();
List<Point> scene = new ArrayList<Point>();
for (int i = -1; ++i < good_matches.size();)
{
object.add(keyPoints_object.toArray()[good_matches.get(i).queryIdx].pt);
scene.add(keyPoints_scene.toArray()[good_matches.get(i).trainIdx].pt);
}
Mat H = Calib3d.findHomography(
new MatOfPoint2f(object.toArray(new Point[object.size()])),
new MatOfPoint2f(scene.toArray(new Point[scene.size()])),
Calib3d.RANSAC, 3);
Point[] object_corners = new Point[4];
object_corners[0] = new Point(0, 0);
object_corners[1] = new Point(img_object.cols(), 0);
object_corners[2] = new Point(img_object.cols(), img_object.rows());
object_corners[3] = new Point(0, img_object.rows());
MatOfPoint2f scene_corners2f = new MatOfPoint2f();
Core.perspectiveTransform(new MatOfPoint2f(object_corners),
scene_corners2f, H);
Point[] scene_corners = scene_corners2f.toArray();
Point[] scene_corners_norm = new Point[4];
scene_corners_norm[0] = new Point(scene_corners[0].x
+ img_object.cols(), scene_corners[0].y);
scene_corners_norm[1] = new Point(scene_corners[1].x
+ img_object.cols(), scene_corners[1].y);
scene_corners_norm[2] = new Point(scene_corners[2].x
+ img_object.cols(), scene_corners[2].y);
scene_corners_norm[3] = new Point(scene_corners[3].x
+ img_object.cols(), scene_corners[3].y);
Core.line(img_matches, scene_corners_norm[0], scene_corners_norm[1],
new Scalar(0, 255, 0), 4);
Core.line(img_matches, scene_corners_norm[1], scene_corners_norm[2],
new Scalar(0, 255, 0), 4);
Core.line(img_matches, scene_corners_norm[2], scene_corners_norm[3],
new Scalar(0, 255, 0), 4);
Core.line(img_matches, scene_corners_norm[3], scene_corners_norm[0],
new Scalar(0, 255, 0), 4);
bmp = Bitmap.createBitmap(img_matches.cols(), img_matches.rows(),
Bitmap.Config.ARGB_8888);
Intent resultIntent = new Intent("com.example.capturetest.Result");
startActivity(resultIntent);
}
private volatile Mat cachedObjectDescriptor = null;
private volatile MatOfKeyPoint cachedObjectKeyPoints = null;
private volatile Mat cachedObjectImage = null;
private Mat calculateObjectDescriptor()
{
Mat objectDescriptor = this.cachedObjectDescriptor;
if (objectDescriptor == null)
{
Mat objectImage = getObjectImage();
MatOfKeyPoint objectKeyPoints = detectObjectKeyPoints();
objectDescriptor = newEmptyMat();
this.extractor.compute(objectImage, objectKeyPoints,
objectDescriptor);
this.cachedObjectDescriptor = objectDescriptor;
}
return objectDescriptor;
}
private MatOfKeyPoint detectObjectKeyPoints()
{
MatOfKeyPoint objectKeyPoints = this.cachedObjectKeyPoints;
if (objectKeyPoints == null)
{
Mat objectImage = getObjectImage();
objectKeyPoints = new MatOfKeyPoint();
this.detector.detect(objectImage, objectKeyPoints);
this.cachedObjectKeyPoints = objectKeyPoints;
}
return objectKeyPoints;
}
private Mat getObjectImage()
{
Mat objectImage = this.cachedObjectImage;
if (objectImage == null)
{
objectImage = newEmptyMat();
Bitmap bitmap = ((BitmapDrawable) iv.getDrawable()).getBitmap();
Bitmap img = bitmap.copy(Bitmap.Config.ARGB_8888, false);
Utils.bitmapToMat(img, objectImage);
this.cachedObjectImage = objectImage;
}
return objectImage;
}
private Mat newEmptyMat()
{
return new Mat();
}
After this line matcher.match(descriptors_object, descriptors_scene, matches); i tried to convert the three Mat img_object, img_scene and matches to bitmaps and saved them to my android device just for checking. They all look as they are supposed to look, so until this point everthing is fine.
But after this part...
Mat img_matches = newEmptyMat();
Features2d.drawMatches(
img_object,
keyPoints_object,
img_scene,
keyPoints_scene,
new MatOfDMatch(good_matches.toArray(new DMatch[good_matches
.size()])), img_matches, Scalar.all(-1),
Scalar.all(-1), new MatOfByte(),
Features2d.NOT_DRAW_SINGLE_POINTS);
... i tried to convert the Mat img_matches (which is supposed to have all the information of the two input pictures if i get it right), to a bitmap and save it on my android device, but the picture looks like the picture in the link above (black pictures with lines instead of card-pictures with lines).
Does any of you know what im doing wrong here? I seem to be stuck at the moment.
Thanks in advance guys.
Edit:
Just wanted to let you know that i got the same code running and WORKING as a normal java program on my desktop. The picture is taken from the webcam there. The result image is diplayed absolutely correct in the desktop program, with cards and lines instead of black and lines ;)

Alright, found a working way:
Imgproc.cvtColor(img_object, img_object, Imgproc.COLOR_RGBA2RGB);
Imgproc.cvtColor(img_scene, img_scene, Imgproc.COLOR_RGBA2RGB);
It seems that after i converted my Bitmaps to Mats i have to use the above two lines to convert them from RGBA to RGB. It also works with RGBA to GRAY if you prefer gray pictures.
It seems that RGBA format is not working in this case.
Hope this helps anybody coming here from google.