Develop Reference

OpenCV speed traffic sign detection

I have a problem detecting speed traffic signs with opencv 2.4 for Android.
I do the following:
"capture frame -> convert it to HSV -> extract red areas -> detect signs with ellipse detection"
So far ellipse detection works perfect as long as picture is good quality.
But as you see in pictures bellow, that red extraction does not work OK, because of poor quality of picture frames, by my opinion.
Converting original image to HSV:
Imgproc.cvtColor(this.source, this.source, Imgproc.COLOR_RGB2HSV, 3);
Extracting red colors:
Core.inRange(this.source, new Scalar(this.h,this.s,this.v), new Scalar(230,180,180), this.source);
So my question is is there another way of detecting traffic sign like this or extracting red areas out of it, which by the way can be very faint like in last picture ?
This is the original image:
This is converted to HSV, as you can see red areas look the same color as nearby trees. Thats how I'm suppose to know it's red but I can't.
Converted to HSV:
This is with red colors extracted. If colors would be correct I should get almost perfect circle/ellipse around sign, but it is incomplet due to false colors.
Result after extraction:
Ellipse method:
private void findEllipses(Mat input){
Mat thresholdOutput = new Mat();
int thresh = 150;
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
MatOfInt4 hierarchy = new MatOfInt4();
Imgproc.threshold(source, thresholdOutput, thresh, 255, Imgproc.THRESH_BINARY);
//Imgproc.Canny(source, thresholdOutput, 50, 180);
Imgproc.findContours(source, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
RotatedRect minEllipse[] = new RotatedRect[contours.size()];
for(int i=0; i<contours.size();i++){
MatOfPoint2f temp=new MatOfPoint2f(contours.get(i).toArray());
if(temp.size().height > minEllipseSize && temp.size().height < maxEllipseSize){
double a = Imgproc.fitEllipse(temp).size.height;
double b = Imgproc.fitEllipse(temp).size.width;
if(Math.abs(a - b) < 10)
minEllipse[i] = Imgproc.fitEllipse(temp);
}
}
detectedObjects.clear();
for( int i = 0; i< contours.size(); i++ ){
Scalar color = new Scalar(180, 255, 180);
if(minEllipse[i] != null){
detectedObjects.add(new DetectedObject(minEllipse[i].center));
DetectedObject detectedObj = new DetectedObject(minEllipse[i].center);
Core.ellipse(source, minEllipse[i], color, 2, 8);
}
}
}
Problematic sign:

You can find a review of traffic signs detection methods here and here.
You'll see that there are 2 ways you can achieve this:
Color-based (like what you're doing now)
Shape-based
In my experience, I found that shape-based methods works pretty good, because the color may change a lot under different lighting conditions, camera quality, etc.
Since you need to detect speed traffic signs, which I assume are always circular, you can use an ellipse detector to find all circular objects in your image, and then apply some validation to determine if it's a traffic sign or not.
Why ellipse detection?
Well, since you're looking for perspective distorted circles, you are in fact looking for ellipses. Real-time ellipse detection is an interesting (although limited) research topic. I'll point you out to 2 papers with C++ source code available (which you can use in you app through native JNI calls):
L. Libuda, I. Grothues, K.-F. Kraiss, Ellipse detection in digital image
data using geometric features, in: J. Braz, A. Ranchordas, H. Arajo,
J. Jorge (Eds.), Advances in Computer Graphics and Computer Vision,
volume 4 of Communications in Computer and Information Science,
Springer Berlin Heidelberg, 2007, pp. 229-239. link, code
M. Fornaciari, A. Prati, R. Cucchiara,
"A fast and effective ellipse detector for embedded vision applications", Pattern Recognition, 2014 link, code
UPDATE
I tried the method 2) without any preprocessing. You can see that at least the sign with the red border is detected very good:

Referencing to your text:
This is converted to HSV, as you can see red areas look the same color
as nearby trees. Thats how I'm suppose to know it's red but I can't.
I want to show you my result of basically what you did (simple operations should be easily transferable to android openCV):
// convert to HSV
cv::Mat hsv;
cv::cvtColor(input,hsv,CV_BGR2HSV);
std::vector<cv::Mat> channels;
cv::split(hsv,channels);
// opencv = hue values are divided by 2 to fit 8 bit range
float red1 = 25/2.0f;
// red has one part at the beginning and one part at the end of the range (I assume 0° to 25° and 335° to 360°)
float red2 = (360-25)/2.0f;
// compute both thresholds
cv::Mat thres1 = channels[0] < red1;
cv::Mat thres2 = channels[0] > red2;
// choose some minimum saturation
cv::Mat saturationThres = channels[1] > 50;
// combine the results
cv::Mat redMask = (thres1 | thres2) & saturationThres;
// display result
cv::imshow("red", redMask);
These are my results:
From your result, please mind that findContours alters the input image, so maybe you extracted the ellipse but just don't see it in the image anymore, if you saved the image AFTER findContours.

private void findEllipses(Mat input){
Mat thresholdOutput = new Mat();
int thresh = 150;
List<MatOfPoint> contours = new ArrayList<MatOfPoint>();
MatOfInt4 hierarchy = new MatOfInt4();
Imgproc.threshold(source, thresholdOutput, thresh, 255, Imgproc.THRESH_BINARY);
//Imgproc.Canny(source, thresholdOutput, 50, 180);
Imgproc.findContours(source, contours, hierarchy, Imgproc.RETR_LIST, Imgproc.CHAIN_APPROX_SIMPLE);
// source = thresholdOutput;
RotatedRect minEllipse[] = new RotatedRect[contours.size()];
for(int i=0; i<contours.size();i++){
MatOfPoint2f temp=new MatOfPoint2f(contours.get(i).toArray());
if(temp.size().height > minEllipseSize && temp.size().height < maxEllipseSize){
double a = Imgproc.fitEllipse(temp).size.height;
double b = Imgproc.fitEllipse(temp).size.width;
if(Math.abs(a - b) < 10)
minEllipse[i] = Imgproc.fitEllipse(temp);
}
}
detectedObjects.clear();
for( int i = 0; i< contours.size(); i++ ){
Scalar color = new Scalar(180, 255, 180);
if(minEllipse[i] != null){
detectedObjects.add(new DetectedObject(minEllipse[i].center));
DetectedObject detectedObj = new DetectedObject(minEllipse[i].center);
Core.ellipse(source, minEllipse[i], color, 2, 8);
}
}
}

have you tried using opencv ORB? it works really well.
I created a haar cascade for a traffic sign (roundabout in my case) and used opencv ORB to match features and remove any false positives.
For image recognition used Google's tensorflow and results were spectacular.

Dealing with Android's texture size limit

I have a requirement to display somewhat big images on an Android app.
Right now I'm using an ImageView with a source Bitmap.
I understand openGL has a certain device-independent limitation as to
how big the image dimensions can be in order for it to process it.
Is there ANY way to display these images (with fixed width, without cropping) regardless of this limit,
other than splitting the image into multiple ImageView elements?
Thank you.
UPDATE 01 Apr 2013
Still no luck so far all suggestions were to reduce image quality. One suggested it might be possible to bypass this limitation by using the CPU to do the processing instead of using the GPU (though might take more time to process).
I don't understand, is there really no way to display long images with a fixed width without reducing image quality? I bet there is, I'd love it if anyone would at least point me to the right direction.
Thanks everyone.

You can use BitmapRegionDecoder to break apart larger bitmaps (requires API level 10). I've wrote a method that will utilize this class and return a single Drawable that can be placed inside an ImageView:
private static final int MAX_SIZE = 1024;
private Drawable createLargeDrawable(int resId) throws IOException {
InputStream is = getResources().openRawResource(resId);
BitmapRegionDecoder brd = BitmapRegionDecoder.newInstance(is, true);
try {
if (brd.getWidth() <= MAX_SIZE && brd.getHeight() <= MAX_SIZE) {
return new BitmapDrawable(getResources(), is);
}
int rowCount = (int) Math.ceil((float) brd.getHeight() / (float) MAX_SIZE);
int colCount = (int) Math.ceil((float) brd.getWidth() / (float) MAX_SIZE);
BitmapDrawable[] drawables = new BitmapDrawable[rowCount * colCount];
for (int i = 0; i < rowCount; i++) {
int top = MAX_SIZE * i;
int bottom = i == rowCount - 1 ? brd.getHeight() : top + MAX_SIZE;
for (int j = 0; j < colCount; j++) {
int left = MAX_SIZE * j;
int right = j == colCount - 1 ? brd.getWidth() : left + MAX_SIZE;
Bitmap b = brd.decodeRegion(new Rect(left, top, right, bottom), null);
BitmapDrawable bd = new BitmapDrawable(getResources(), b);
bd.setGravity(Gravity.TOP | Gravity.LEFT);
drawables[i * colCount + j] = bd;
}
}
LayerDrawable ld = new LayerDrawable(drawables);
for (int i = 0; i < rowCount; i++) {
for (int j = 0; j < colCount; j++) {
ld.setLayerInset(i * colCount + j, MAX_SIZE * j, MAX_SIZE * i, 0, 0);
}
}
return ld;
}
finally {
brd.recycle();
}
}
The method will check to see if the drawable resource is smaller than MAX_SIZE (1024) in both axes. If it is, it just returns the drawable. If it's not, it will break the image apart and decode chunks of the image and place them in a LayerDrawable.
I chose 1024 because I believe most available phones will support images at least that large. If you want to find the actual texture size limit for a phone, you have to do some funky stuff through OpenGL, and it's not something I wanted to dive into.
I wasn't sure how you were accessing your images, so I assumed they were in your drawable folder. If that's not the case, it should be fairly easy to refactor the method to take in whatever parameter you need.

You can use BitmapFactoryOptions to reduce size of picture.You can use somthing like that :
BitmapFactory.Options options = new BitmapFactory.Options();
options.inSampleSize = 3; //reduce size 3 times

Have you seen how your maps working? I had made a renderer for maps once. You can use same trick to display your image.
Divide your image into square tiles (e.g. of 128x128 pixels). Create custom imageView supporting rendering from tiles. Your imageView knows which part of bitmap it should show now and displays only required tiles loading them from your sd card. Using such tile map you can display endless images.

It would help if you gave us the dimensions of your bitmap.
Please understand that OpenGL runs against natural mathematical limits.
For instance, there is a very good reason a texture in OpenGL must be 2 to the power of x. This is really the only way the math of any downscaling can be done cleanly without any remainder.
So if you give us the exact dimensions of the smallest bitmap that's giving you trouble, some of us may be able to tell you what kind of actual limit you're running up against.

Deskew and filter an image for OCR

I have made a lot of research and have not found a suitable solution for this. I am programming an Android OCR app. I have already successfully loaded Tesseract and Leptonica and I am successfully taking and processing images as well as converting them to text using OCR. However, the recognition accuracy is not so good.
After much tweaking we found out that we were not filtering, cleaning and/or deskewing the image enough to aid Tesseract in the OCR processing. Therefore, I looked on the internet for any library or code that would work for me and use it on Android to no avail.
Does anyone know of a library or can provide me some code to aid me on accomplishing this? All I want is to take a bitmap, convert it to black and white, deskew and/or execute some filtering tasks and give it to Tesseract for it to convert it to text using OCR.

ImageMagick can do that. Commandline:
convert \
input.{png,pdf,tif,jpeg,gif,...} \
-colorspace grayscale \
-threshold 50% \
-deskew \
output.{png,pdf,tif,jpeg,gif,...}
-colorspace grayscale : helps to also process colored input.
-threshold 50% : play with the percentage value -- but basically it converts to black + white only.
-deskews : deskews
However, I'm not sure how easy or difficult it is to build ImageMagick for the Android platform. It's available for Linux, Mac OS X, Windows, Solaris, HP-UX,... so: it multi-platform by design anyway.

I used the below approach to convert my image to B&W and that helped me to increase my accuracy very much.
private Bitmap GetBinaryBitmap(Bitmap bitmap_src) {
Bitmap bitmap_new = bitmap_src.copy(bitmap_src.getConfig(), true);
for (int x = 0; x < bitmap_new.getWidth(); x++) {
for (int y = 0; y < bitmap_new.getHeight(); y++) {
int color = bitmap_new.getPixel(x, y);
color = GetNewColor(color);
bitmap_new.setPixel(x, y, color);
}
}
return bitmap_new;
}
private double GetColorDistance(int c1, int c2) {
int db = Color.blue(c1) - Color.blue(c2);
int dg = Color.green(c1) - Color.green(c2);
int dr = Color.red(c1) - Color.red(c2);
double d = Math.sqrt(Math.pow(db, 2) + Math.pow(dg, 2) + Math.pow(dr, 2));
return d;
}
private int GetNewColor(int c) {
double dwhite = GetColorDistance(c, Color.WHITE);
double dblack = GetColorDistance(c, Color.BLACK);
if (dwhite <= dblack) {
return Color.WHITE;
} else {
return Color.BLACK;
}
}
Hope it helps.

Creating a 1bpp (bit per pixel) Bitmap in Android

Ok so been racking my brain on this one all day. Trying to figure out how I can convert a Bitmap from canvas to a 1bpp (bit per pixel) Bitmap file in Android and physically save it as such.
So far I've iterated through the bitmap and created an int[] of the resulting pixel values as 1s or 0s. However, my next question is what do I do with that?
What I tried to do was something like
int[] bits = // populated earlier
byte[] bmp = new byte[bits.length / 8];
int byteindex = 0;
int bitindex = 0;
for (int i=0; i<bits.length; i++) {
if (bits[i] == 1)
// set to 1
else
// set to 0
if (bitindex++ == 8) {
bitindex = 0;
byteindex++;
}
}
OutputStream out = new FileOutputStream("/mnt/sdcard/dynbmp.bmp");
out.write(bmp);
out.close();
I get a file out of it but it's obviously not a valid bmp file. Who knows what it is. You'll have to forgive me for my lack of bit-byte and imaging knowledge, but where am I screwing up? Do I the idea completely wrong? Am I missing some header info or something?

Yes, you are missing several things. It's a little bit more complicated... Look here:
http://en.wikipedia.org/wiki/BMP_file_format

Access to raw data in ARGB_8888 Android Bitmap

I am trying to access the raw data of a Bitmap in ARGB_8888 format on Android, using the copyPixelsToBuffer and copyPixelsFromBuffer methods. However, invocation of those calls seems to always apply the alpha channel to the rgb channels. I need the raw data in a byte[] or similar (to pass through JNI; yes, I know about bitmap.h in Android 2.2, cannot use that).
Here is a sample:
// Create 1x1 Bitmap with alpha channel, 8 bits per channel
Bitmap one = Bitmap.createBitmap(1,1,Bitmap.Config.ARGB_8888);
one.setPixel(0,0,0xef234567);
Log.v("?","hasAlpha() = "+Boolean.toString(one.hasAlpha()));
Log.v("?","pixel before = "+Integer.toHexString(one.getPixel(0,0)));
// Copy Bitmap to buffer
byte[] store = new byte[4];
ByteBuffer buffer = ByteBuffer.wrap(store);
one.copyPixelsToBuffer(buffer);
// Change value of the pixel
int value=buffer.getInt(0);
Log.v("?", "value before = "+Integer.toHexString(value));
value = (value >> 8) | 0xffffff00;
buffer.putInt(0, value);
value=buffer.getInt(0);
Log.v("?", "value after = "+Integer.toHexString(value));
// Copy buffer back to Bitmap
buffer.position(0);
one.copyPixelsFromBuffer(buffer);
Log.v("?","pixel after = "+Integer.toHexString(one.getPixel(0,0)));
The log then shows
hasAlpha() = true
pixel before = ef234567
value before = 214161ef
value after = ffffff61
pixel after = 619e9e9e
I understand that the order of the argb channels is different; that's fine. But I don't
want the alpha channel to be applied upon every copy (which is what it seems to be doing).
Is this how copyPixelsToBuffer and copyPixelsFromBuffer are supposed to work? Is there any way to get the raw data in a byte[]?
Added in response to answer below:
Putting in buffer.order(ByteOrder.nativeOrder()); before the copyPixelsToBuffer does change the result, but still not in the way I want it:
pixel before = ef234567
value before = ef614121
value after = ffffff41
pixel after = ff41ffff
Seems to suffer from essentially the same problem (alpha being applied upon each copyPixelsFrom/ToBuffer).

One way to access data in Bitmap is to use getPixels() method. Below you can find an example I used to get grayscale image from argb data and then back from byte array to Bitmap (of course if you need rgb you reserve 3x bytes and save them all...):
/*Free to use licence by Sami Varjo (but nice if you retain this line)*/
public final class BitmapConverter {
private BitmapConverter(){};
/**
* Get grayscale data from argb image to byte array
*/
public static byte[] ARGB2Gray(Bitmap img)
{
int width = img.getWidth();
int height = img.getHeight();
int[] pixels = new int[height*width];
byte grayIm[] = new byte[height*width];
img.getPixels(pixels,0,width,0,0,width,height);
int pixel=0;
int count=width*height;
while(count-->0){
int inVal = pixels[pixel];
//Get the pixel channel values from int
double r = (double)( (inVal & 0x00ff0000)>>16 );
double g = (double)( (inVal & 0x0000ff00)>>8 );
double b = (double)( inVal & 0x000000ff) ;
grayIm[pixel++] = (byte)( 0.2989*r + 0.5870*g + 0.1140*b );
}
return grayIm;
}
/**
* Create a gray scale bitmap from byte array
*/
public static Bitmap gray2ARGB(byte[] data, int width, int height)
{
int count = height*width;
int[] outPix = new int[count];
int pixel=0;
while(count-->0){
int val = data[pixel] & 0xff; //convert byte to unsigned
outPix[pixel++] = 0xff000000 | val << 16 | val << 8 | val ;
}
Bitmap out = Bitmap.createBitmap(outPix,0,width,width, height, Bitmap.Config.ARGB_8888);
return out;
}
}

My guess is that this might have to do with the byte order of the ByteBuffer you are using. ByteBuffer uses big endian by default.
Set endianess on the buffer with
buffer.order(ByteOrder.nativeOrder());
See if it helps.
Moreover, copyPixelsFromBuffer/copyPixelsToBuffer does not change the pixel data in any way. They are copied raw.

I realize this is very stale and probably won't help you now, but I came across this recently in trying to get copyPixelsFromBuffer to work in my app. (Thank you for asking this question, btw! You saved me tons of time in debugging.) I'm adding this answer in the hopes it helps others like me going forward...
Although I haven't used this yet to ensure that it works, it looks like that, as of API Level 19, we'll finally have a way to specify not to "apply the alpha" (a.k.a. premultiply) within Bitmap. They're adding a setPremultiplied(boolean) method that should help in situations like this going forward by allowing us to specify false.
I hope this helps!

This is an old question, but i got to the same issue, and just figured out that the bitmap byte are pre-multiplied, you can set the bitmap (as of API 19) to not pre-multiply the buffer, but in the API they make no guarantee.
From the docs:
public final void setPremultiplied(boolean premultiplied)
Sets whether the bitmap should treat its data as pre-multiplied.
Bitmaps are always treated as pre-multiplied by the view system and Canvas for performance reasons. Storing un-pre-multiplied data in a Bitmap (through setPixel, setPixels, or BitmapFactory.Options.inPremultiplied) can lead to incorrect blending if drawn by the framework.
This method will not affect the behaviour of a bitmap without an alpha channel, or if hasAlpha() returns false.
Calling createBitmap or createScaledBitmap with a source Bitmap whose colors are not pre-multiplied may result in a RuntimeException, since those functions require drawing the source, which is not supported for un-pre-multiplied Bitmaps.