I am trying to create a character customization in unity for android. Now the scenario is that i have a dress model which is a texture2D, also a number of patterns and colors that the user can apply on this model of dress. Now when the user applies a pattern on to the dress i need to change the dress to be displayed in that pattern. For color i was able to change the rgb value to the desired color value. But for the pattern I will need to traverse through each pixel of the dress and apply the patterns corresponding color to the pixel of the dress. I achieved this by the following code.
IEnumerator Print() {
Texture2D tex = DressTexture;
Color32[] DressColor = tex.GetPixels32();
Color32[] PatternColor = PatternTexture.GetPixels32();
int j = 0;
Texture2D NewDressPattern = new Texture2D(tex.width, tex.height, TextureFormat.ARGB32, false);
for(int i = 0; i < DressColor.Length; i++) {
if(DressColor[i].a != 0) {
DressColor[i] = PatternColor[j];
j++;
if(j == PatternColor.Length - 1) {
j = 0;
}
}
else {
j = 0;
}
yield return null;
}
NewDressPattern.SetPixels32(DressColor);
NewDressPattern.Apply();
Debug.Log("texture created");
Sprite spr = Sprite.Create(NewDressPattern, new Rect(0, 0, tex.width, tex.height), Vector2.zero);
Debug.Log("sprite created");
sprite.sprite = spr;
}
Now the problem is this operation is too slow to complete. Any suggestions for a better way to achieve this would be really helpful. Also i am not aware of shaders much.
Texture operations always require some time consuming processing. There are workarounds you can do to minimize the impact on user. Like doing calculations multithreaded or nonblocking (I think that's what you are doing right now). But they will only minimize the problem, won't solve it.
You did not mention if the texture used in a 2D environment as a sprite or used on a 3D model.
For a 2D game:
What I would do is I would use a separate sprite for each object. Then I would overlap sprites and the result will exactly be like what you are trying to do above.
For a 3D model:
You can use some simple shaders to overlap multiple textures. Just Google it and you will have plenty of examples. Simple shaders like these does not require rocket science knowledge :)
Also, render-to-texture technique can be used to pre-render multiple texture passes. For example, the code below uses built-in "Diffuse Detail" shader to combine Texture2D objects and put result in RenderTexture:
public RenderTexture PreRenderCustomDressTexture(int customTextureWidth, int customTextureHeight, Texture2D pattern1, Texture2D pattern2)
{
RenderTexture customDressTexture = new RenderTexture( customTextureWidth, customTextureHeight, 32, RenderTextureFormat.ARGB32, RenderTextureReadWrite.Default );
Material blitMaterial = new Material( Shader.Find( "Diffuse Detail" ) );
blitMaterial.SetColor( "_Color", Color.white );
blitMaterial.SetTexture( "_MainTex", pattern1 );
blitMaterial.SetTexture( "_Detail", pattern2 );
Graphics.Blit( pattern1, customDressTexture, blitMaterial );
return customDressTexture;
}
Related
I have an Android Project with OpenCV4.0.1 and TFLite installed.
And I want to make an inference with a pretrained MobileNetV2 of an cv::Mat which I extracted and cropped from a CameraBridgeViewBase (Android style).
But it's kinda difficult.
I followed this example.
That does the inference about a ByteBuffer variable called "imgData" (line 71, class: org.tensorflow.lite.examples.classification.tflite.Classifier)
That imgData looks been filled on the method called "convertBitmapToByteBuffer" from the same class (line 185), adding pixel by pixel form a bitmap that looks to be cropped little before.
private int[] intValues = new int[224 * 224];
Mat _croppedFace = new Mat() // Cropped image from CvCameraViewFrame.rgba() method.
float[][] outputVal = new float[1][1]; // Output value from my MobileNetV2 // trained model (i've changed the output on training, tested on python)
// Following: https://stackoverflow.com/questions/13134682/convert-mat-to-bitmap-opencv-for-android
Bitmap bitmap = Bitmap.createBitmap(_croppedFace.cols(), _croppedFace.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(_croppedFace, bitmap);
convertBitmapToByteBuffer(bitmap); // This call should be used as the example one.
// runInference();
_tflite.run(imgData, outputVal);
But, it looks that the input_shape of my NN is not correct, but I'm following the MobileNet example because my NN it's a MobileNetV2.
I've solved the error, but I'm sure that it isn't the best way to do it.
Keras MobilenetV2 input_shape is: (nBatches, 224, 224, nChannels).
I just want to predict a single image, so, nBaches == 1, and I'm working on RGB mode, so nChannels == 3
// Nasty nasty, but works. nBatches == 2? -- _cropped.shape() == (244, 244), 3 channels.
float [][][][] _inputValue = new float[2][_cropped.cols()][_cropped.rows()][3];
// Fill the _inputValue
for(int i = 0; i < _croppedFace.cols(); ++i)
for (int j = 0; j < _croppedFace.rows(); ++j)
for(int z = 0; z < 3; ++z)
_inputValue [0][i][j][z] = (float) _croppedFace.get(i, j)[z] / 255; // DL works better with 0:1 values.
/*
Output val, has this shape, but I don't really know why.
I'm sure that one's of that 2's is for nClasses (I'm working with 2 classes)
But I don't really know why it's using the other one.
*/
float[][] outputVal = new float[2][2];
// Tensorflow lite interpreter
_tflite.run(_inputValue , outputVal);
On python has the same shape:
Python prediction:
[[XXXXXX, YYYYY]] <- Sure for the last layer that I made, this is just a prototype NN.
Hope some one got help, and also that someone can improve the answer because this is not very optimized.
I have tried 2 methodologies as follows:-
conversion of image to Mat
apply gaussian blur
then canny edge detection
find contours
The problem with this method is:
too many contours are detected
mostly open contours
doesn't detect what I want to detect
Then I changed my approach and tried adaptive thresholding after gaussian blur/median blur and it is much better and I am able to detect the corners in 50% cases
The current problem I am facing is that the page detection requires contrasting and plain background without any reflections. I think it's too idealistic for real world use.
This is where I would like some help. Even a direction towards the solution is highly appreciated especially in java. Thanks in anticipation
works absolutely fine with a significant contrasting background like this
Detected 4 corners
This picture gives troubles because the background isn't exactly the most contrasting
Initial largest contour found
Update: median blur did not help much so I traced the cause and found that the page boundary was detected in bits and pieces and not a single contour so it detected the biggest contour as a part of the page boundary Therefore performed some morphological operations to close relatively small gaps and the resultant largest contour is definitely improved but its its not optimum. Any ideas how I can improve the big gaps?
morphed original picture
largest contour found in the morphed image
PS morphing the image in ideal scenarios has led to detection of false contour boundaries. Any condition which can be checked before morphing an image is also a bonus. Thank you
If you use methods like that:
public static RotatedRect getBestRectByArea(List<RotatedRect> boundingRects) {
RotatedRect bestRect = null;
if (boundingRects.size() >= 1) {
RotatedRect boundingRect;
Point[] vertices = new Point[4];
Rect rect;
double maxArea;
int ixMaxArea = 0;
// find best rect by area
boundingRect = boundingRects.get(ixMaxArea);
boundingRect.points(vertices);
rect = Imgproc.boundingRect(new MatOfPoint(vertices));
maxArea = rect.area();
for (int ix = 1; ix < boundingRects.size(); ix++) {
boundingRect = boundingRects.get(ix);
boundingRect.points(vertices);
rect = Imgproc.boundingRect(new MatOfPoint(vertices));
if (rect.area() > maxArea) {
maxArea = rect.area();
ixMaxArea = ix;
}
}
bestRect = boundingRects.get(ixMaxArea);
}
return bestRect;
}
private static Bitmap findROI(Bitmap sourceBitmap) {
Bitmap roiBitmap = Bitmap.createBitmap(sourceBitmap.getWidth(), sourceBitmap.getHeight(), Bitmap.Config.ARGB_8888);
Mat sourceMat = new Mat(sourceBitmap.getWidth(), sourceBitmap.getHeight(), CV_8UC3);
Utils.bitmapToMat(sourceBitmap, sourceMat);
final Mat mat = new Mat();
sourceMat.copyTo(mat);
Imgproc.cvtColor(mat, mat, Imgproc.COLOR_RGB2GRAY);
Imgproc.threshold(mat, mat, 146, 250, Imgproc.THRESH_BINARY);
// find contours
List<MatOfPoint> contours = new ArrayList<>();
List<RotatedRect> boundingRects = new ArrayList<>();
Imgproc.findContours(mat, contours, new Mat(), Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
// find appropriate bounding rectangles
for (MatOfPoint contour : contours) {
MatOfPoint2f areaPoints = new MatOfPoint2f(contour.toArray());
RotatedRect boundingRect = Imgproc.minAreaRect(areaPoints);
boundingRects.add(boundingRect);
}
RotatedRect documentRect = getBestRectByArea(boundingRects);
if (documentRect != null) {
Point rect_points[] = new Point[4];
documentRect.points(rect_points);
for (int i = 0; i < 4; ++i) {
Imgproc.line(sourceMat, rect_points[i], rect_points[(i + 1) % 4], ROI_COLOR, ROI_WIDTH);
}
}
Utils.matToBitmap(sourceMat, roiBitmap);
return roiBitmap;
}
you can achieve for your source images results like this:
or that:
If you adjust threshold values and apply filters you can achieve even better results.
You can pick a single contour by using one or both of:
Use BoundingRect and ContourArea to evaluate the squareness of each contour. boundingRect() returns orthogonal rects., to handle arbitrary rotation better use minAreaRect() which returns optimally rotated ones.
Use Cv.ApproxPoly iteratively to reduce to a 4 sided shape
var approxIter = 1;
while (true)
{
var approxCurve = Cv.ApproxPoly(largestContour, 0, null, ApproxPolyMethod.DP, approxIter, true);
var approxCurvePointsTmp = new[] { approxCurve.Select(p => new CvPoint2D32f((int)p.Value.X, (int)p.Value.Y)).ToArray() }.ToArray();
if (approxCurvePointsTmp[0].Length == 4)
{
corners = approxCurvePointsTmp[0];
break;
}
else if (approxCurvePointsTmp[0].Length < 4) throw new InvalidOperationException("Failed to decimate corner points");
approxIter++;
}
However neither of these will help if the contour detection gives you two separate contours due to noise / contrast.
I think it would be possible to use the hough line transformation to help detect cases where a line has been split into two contours.
If so the search could be repeated for all combinations of joined contours to see if a bigger / more rectangular match is found.
Stop relying on edge detection, the worst methodology in the universe, and switch to some form of image segmentation.
The paper is white, the background is contrasted, this is the information that you should use.
I am fairly new to openCV libraries and I am trying to do real time object detection for a school project on an android app. followed this tutorial (https://www.youtube.com/watch?v=bSeFrPrqZ2A) and I am able to detect object by color on my android phone. Now I am trying to map out the trajectory of the object just like in this video (https://www.youtube.com/watch?v=QTYSRZD4vyI).
Following is some of the source code provided in the first youtube video.
void searchForMovement(int& x, int& y, Mat& mRgb1, Mat& threshold){
morphOps(threshold);
Mat temp;
threshold.copyTo(temp);
//these two vectors needed for output of findContours
vector< vector<Point> > contours;
vector<Vec4i> hierarchy;
//find contours of filtered image using openCV findContours function
//In OpenCV, finding contours is like finding white object from black background.
// So remember, object to be found should be white and background should be black.
//CV_CHAIN_APPROX_SIMPLE to draw 4 points of the contour
findContours(temp,contours,hierarchy,CV_RETR_CCOMP,CV_CHAIN_APPROX_SIMPLE );
double refArea = 0;
bool objectFound = false;
if (hierarchy.size() > 0) {
int numObjects = hierarchy.size();
//if number of objects greater than MAX_NUM_OBJECTS we have a noisy filter
if(numObjects<MAX_NUM_OBJECTS){
for (int index = 0; index >= 0; index = hierarchy[index][0]) {
Moments moment = moments((cv::Mat)contours[index]);
double area = moment.m00;
//if the area is less than 20 px by 20px then it is probably just noise
//if the area is the same as the 3/2 of the image size, probably just a bad filter
//we only want the object with the largest area so we safe a reference area each
//iteration and compare it to the area in the next iteration.
if(area>MIN_OBJECT_AREA && area<MAX_OBJECT_AREA && area>refArea){
x = moment.m10/area;
y = moment.m01/area;
objectFound = true;
refArea = area;
}else objectFound = false;
}
//let user know you found an object
if(objectFound ==true){
putText(mRgb1,"Tracking Object",Point(0,50),2,1,Scalar(0,255,0),2);
//draw object location on screen
drawObject(x,y,mRgb1);}
}else putText(mRgb1,"TOO MUCH NOISE! ADJUST FILTER",Point(0,50),1,2,Scalar(0,0,255),2);
}
}
void drawObject(int x, int y,Mat &frame){
Mat traj;
traj = frame;
//use some of the openCV drawing functions to draw crosshairs
//on your tracked image!
//UPDATE:JUNE 18TH, 2013
//added 'if' and 'else' statements to prevent
//memory errors from writing off the screen (ie. (-25,-25) is not within the window!)
circle(frame,Point(x,y),20,Scalar(0,255,0),2);
if(y-25>0)
line(frame,Point(x,y),Point(x,y-25),Scalar(0,255,0),2);
else line(traj,Point(x,y),Point(x,0),Scalar(0,255,0),2);
if(y+25<FRAME_HEIGHT)
line(frame,Point(x,y),Point(x,y+25),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(x,FRAME_HEIGHT),Scalar(0,255,0),2);
if(x-25>0)
line(traj,Point(x,y),Point(x-25,y),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(0,y),Scalar(0,255,0),2);
if(x+25<FRAME_WIDTH)
line(frame,Point(x,y),Point(x+25,y),Scalar(0,255,0),2);
else line(frame,Point(x,y),Point(FRAME_WIDTH,y),Scalar(0,255,0),2);
// add(traj, frame, frame);
putText(frame,intToString(x)+","+intToString(y),Point(x,y+30),1,1,Scalar(0,255,0),2);
}
How can I add onto this code to get the trajectory of an object showed in the 2nd video? Any suggestion would be much appreciated. Thank you.
http://opencv-srf.blogspot.co.uk/2010/09/object-detection-using-color-seperation.html
Found it. When doing it in android, need to make sure the lastX and lastY are updating as well.
I am trying to write human detector, it works now, but sometimes it reacts on cats/boxes etc., also i got like 5 fps. So the question is, how can i improve my algorithm for better fps and detection accuracy.
I have tried to use this one:
http://www.pyimagesearch.com/2015/11/09/pedestrian-detection-opencv/
But i couldnt find any way i could use this on android.
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
List<MatOfPoint> list = new ArrayList<>();
Mat frame = new Mat();
Mat gray = new Mat();
Mat hierarchy = new Mat();
Mat originalFrame = inputFrame.rgba();
Imgproc.medianBlur(originalFrame,originalFrame,3);
Imgproc.cvtColor(originalFrame, gray, Imgproc.COLOR_RGB2GRAY, 0);
HOGDescriptor hog = new HOGDescriptor();
//Получаем стандартный определитель людей и устанавливаем его нашему дескриптору
MatOfFloat descriptors = HOGDescriptor.getDefaultPeopleDetector();
hog.setSVMDetector(descriptors);
MatOfRect locations = new MatOfRect();
MatOfDouble weights = new MatOfDouble();
hog.detectMultiScale(gray, locations, weights);
Point rectPoint1 = new Point();
Point rectPoint2 = new Point();
Point fontPoint = new Point();
if (locations.rows() > 0) {
List<Rect> rectangles = locations.toList();
for (Rect rect : rectangles) {
rectPoint1.x = rect.x;
rectPoint1.y = rect.y;
fontPoint.x = rect.x;
fontPoint.y = rect.y - 4;
rectPoint2.x = rect.x + rect.width;
rectPoint2.y = rect.y + rect.height;
final Scalar rectColor = new Scalar( 0 , 0 , 0 );
// Добавляем на изображения найденную информацию
Imgproc.rectangle(originalFrame, rectPoint1, rectPoint2, rectColor, 2);
}
}
frame.release();
gray.release();
hierarchy.release();
list.clear();
return originalFrame;
}
You're using the HOG+SVM approach to detect people; it is inherently going to be quite slow. Never the less, you can use some of the suggestions in this question How to speed up svm.predict?
Depending on your problem, i.e. if the camera is static and the pedestrians are moving you could opt for a background subtraction approach this is probably the most efficient way but bear in mind that this will pick up any objects that are moving in the scene, so you could include thresholds to remove small objects. Some background subtraction algorithms include mixture of gaussian (MOG) or MOG2 or GMG. Also, an important thing to note is that these approaches rely on creating a background model of the scene, i.e. they assume static pixels over time to be part of the background, hence, when a pedestrian stands still for a while in the scene they get embedded into the background resulting in miss detection. There are many papers out there that provide potential solutions to that problem so you might want to have a look at them, here is one that produces decent results: Static and Moving Object Detection Using Flux Tensor with Split Gaussian Models
Additionally, you could opt for a data driven approach, either get a good pre-trained model and do your detection using that or train one yourself using TensorFlow, Caffe or Torch and use the dnn opencv_contrib module to do the detection.
I have a problem detecting speed traffic signs with opencv 2.4 for Android.
I do the following:
"capture frame -> convert it to HSV -> extract red areas -> detect signs with ellipse detection"
So far ellipse detection works perfect as long as picture is good quality.
But as you see in pictures bellow, that red extraction does not work OK, because of poor quality of picture frames, by my opinion.
Converting original image to HSV:
Imgproc.cvtColor(this.source, this.source, Imgproc.COLOR_RGB2HSV, 3);
Extracting red colors:
Core.inRange(this.source, new Scalar(this.h,this.s,this.v), new Scalar(230,180,180), this.source);
So my question is is there another way of detecting traffic sign like this or extracting red areas out of it, which by the way can be very faint like in last picture ?
This is the original image:
This is converted to HSV, as you can see red areas look the same color as nearby trees. Thats how I'm suppose to know it's red but I can't.
Converted to HSV:
This is with red colors extracted. If colors would be correct I should get almost perfect circle/ellipse around sign, but it is incomplet due to false colors.
Result after extraction:
Ellipse method:
private void findEllipses(Mat input){
Mat thresholdOutput = new Mat();
int thresh = 150;
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
MatOfInt4 hierarchy = new MatOfInt4();
Imgproc.threshold(source, thresholdOutput, thresh, 255, Imgproc.THRESH_BINARY);
//Imgproc.Canny(source, thresholdOutput, 50, 180);
Imgproc.findContours(source, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
RotatedRect minEllipse[] = new RotatedRect[contours.size()];
for(int i=0; i<contours.size();i++){
MatOfPoint2f temp=new MatOfPoint2f(contours.get(i).toArray());
if(temp.size().height > minEllipseSize && temp.size().height < maxEllipseSize){
double a = Imgproc.fitEllipse(temp).size.height;
double b = Imgproc.fitEllipse(temp).size.width;
if(Math.abs(a - b) < 10)
minEllipse[i] = Imgproc.fitEllipse(temp);
}
}
detectedObjects.clear();
for( int i = 0; i< contours.size(); i++ ){
Scalar color = new Scalar(180, 255, 180);
if(minEllipse[i] != null){
detectedObjects.add(new DetectedObject(minEllipse[i].center));
DetectedObject detectedObj = new DetectedObject(minEllipse[i].center);
Core.ellipse(source, minEllipse[i], color, 2, 8);
}
}
}
Problematic sign:
You can find a review of traffic signs detection methods here and here.
You'll see that there are 2 ways you can achieve this:
Color-based (like what you're doing now)
Shape-based
In my experience, I found that shape-based methods works pretty good, because the color may change a lot under different lighting conditions, camera quality, etc.
Since you need to detect speed traffic signs, which I assume are always circular, you can use an ellipse detector to find all circular objects in your image, and then apply some validation to determine if it's a traffic sign or not.
Why ellipse detection?
Well, since you're looking for perspective distorted circles, you are in fact looking for ellipses. Real-time ellipse detection is an interesting (although limited) research topic. I'll point you out to 2 papers with C++ source code available (which you can use in you app through native JNI calls):
L. Libuda, I. Grothues, K.-F. Kraiss, Ellipse detection in digital image
data using geometric features, in: J. Braz, A. Ranchordas, H. Arajo,
J. Jorge (Eds.), Advances in Computer Graphics and Computer Vision,
volume 4 of Communications in Computer and Information Science,
Springer Berlin Heidelberg, 2007, pp. 229-239. link, code
M. Fornaciari, A. Prati, R. Cucchiara,
"A fast and effective ellipse detector for embedded vision applications", Pattern Recognition, 2014 link, code
UPDATE
I tried the method 2) without any preprocessing. You can see that at least the sign with the red border is detected very good:
Referencing to your text:
This is converted to HSV, as you can see red areas look the same color
as nearby trees. Thats how I'm suppose to know it's red but I can't.
I want to show you my result of basically what you did (simple operations should be easily transferable to android openCV):
// convert to HSV
cv::Mat hsv;
cv::cvtColor(input,hsv,CV_BGR2HSV);
std::vector<cv::Mat> channels;
cv::split(hsv,channels);
// opencv = hue values are divided by 2 to fit 8 bit range
float red1 = 25/2.0f;
// red has one part at the beginning and one part at the end of the range (I assume 0° to 25° and 335° to 360°)
float red2 = (360-25)/2.0f;
// compute both thresholds
cv::Mat thres1 = channels[0] < red1;
cv::Mat thres2 = channels[0] > red2;
// choose some minimum saturation
cv::Mat saturationThres = channels[1] > 50;
// combine the results
cv::Mat redMask = (thres1 | thres2) & saturationThres;
// display result
cv::imshow("red", redMask);
These are my results:
From your result, please mind that findContours alters the input image, so maybe you extracted the ellipse but just don't see it in the image anymore, if you saved the image AFTER findContours.
private void findEllipses(Mat input){
Mat thresholdOutput = new Mat();
int thresh = 150;
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
MatOfInt4 hierarchy = new MatOfInt4();
Imgproc.threshold(source, thresholdOutput, thresh, 255, Imgproc.THRESH_BINARY);
//Imgproc.Canny(source, thresholdOutput, 50, 180);
Imgproc.findContours(source, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
// source = thresholdOutput;
RotatedRect minEllipse[] = new RotatedRect[contours.size()];
for(int i=0; i<contours.size();i++){
MatOfPoint2f temp=new MatOfPoint2f(contours.get(i).toArray());
if(temp.size().height > minEllipseSize && temp.size().height < maxEllipseSize){
double a = Imgproc.fitEllipse(temp).size.height;
double b = Imgproc.fitEllipse(temp).size.width;
if(Math.abs(a - b) < 10)
minEllipse[i] = Imgproc.fitEllipse(temp);
}
}
detectedObjects.clear();
for( int i = 0; i< contours.size(); i++ ){
Scalar color = new Scalar(180, 255, 180);
if(minEllipse[i] != null){
detectedObjects.add(new DetectedObject(minEllipse[i].center));
DetectedObject detectedObj = new DetectedObject(minEllipse[i].center);
Core.ellipse(source, minEllipse[i], color, 2, 8);
}
}
}
have you tried using opencv ORB? it works really well.
I created a haar cascade for a traffic sign (roundabout in my case) and used opencv ORB to match features and remove any false positives.
For image recognition used Google's tensorflow and results were spectacular.