Develop Reference

How to improve OpenCV face detection performance in android?

I am working on a project in android in which i am using OpenCV to detect faces from all the images which are in the gallery. The process of getting faces from the images is performing in the service. Service continuously working till all the images are processed. It is storing the detected faces in the internal storage and also showing in the grid view if activity is opened.
My code is:
CascadeClassifier mJavaDetector=null;
public void getFaces()
{
for (int i=0 ; i<size ; i++)
{
File file=new File(urls.get(i));
imagepath=urls.get(i);
defaultBitmap=BitmapFactory.decodeFile(file, bitmapFatoryOptions);
mJavaDetector = new CascadeClassifier(FaceDetector.class.getResource("lbpcascade_frontalface").getPath());
Mat image = new Mat (defaultBitmap.getWidth(), defaultBitmap.getHeight(), CvType.CV_8UC1);
Utils.bitmapToMat(defaultBitmap,image);
MatOfRect faceDetections = new MatOfRect();
try
{
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));
}
catch(Exception e)
{
e.printStackTrace();
}
if(faceDetections.toArray().length>0)
{
}
}
}
Everything is fine but it is detection faces very slow. The performance is very slow. When i debug the code then i found the line which is taking time is:
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));
I have checked multiple post for this problem but i didn't get any solution.
Please tell me what should i do to solve this problem.
Any help would be greatly appreciated. Thank you.

You should pay attention to the parameters of detectMultiScale():
scaleFactor – Parameter specifying how much the image size is reduced at each image scale. This parameter is used to create a scale pyramid. It is necessary because the model has a fixed size during training. Without pyramid the only size to detect would be this fix one (which can be read from the XML also). However the face detection can be scale-invariant by using multi-scale representation i.e., detecting large and small faces using the same detection window.
scaleFactor depends on the size of your trained detector, but in fact, you need to set it as high as possible while still getting "good" results, so this should be determined empirically.
Your 1.1 value can be a good value for this purpose. It means, a relative small step is used for resizing (reduce size by 10%), you increase the chance of a matching size with the model for detection is found. If your trained detector has the size 10x10 then you can detect faces with size 11x11, 12x12 and so on. But in fact a factor of 1.1 requires roughly double the # of layers in the pyramid (and 2x computation time) than 1.2 does.
minNeighbors – Parameter specifying how many neighbours each candidate rectangle should have to retain it.
Cascade classifier works with a sliding window approach. By applying this approach, you slide a window through over the image than you resize it and search again until you can not resize it further. In every iteration the true outputs (of cascade classifier) are stored but unfortunately it actually detects many false positives. And to eliminate false positives and get the proper face rectangle out of detections, neighbourhood approach is applied. 3-6 is a good value for it. If the value is too high then you can lose true positives too.
minSize – Regarding to the sliding window approach of minNeighbors, this is the smallest window that cascade can detect. Objects smaller than that are ignored. Usually cv::Size(20, 20) are enough for face detections.
maxSize – Maximum possible object size. Objects bigger than that are ignored.
Finally you can try different classifiers based on different features (such as Haar, LBP, HoG). Usually, LBP classifiers are a few times faster than Haar's, but also less accurate.
And it is also strongly recommended to look over these questions:
Recommended values for OpenCV detectMultiScale() parameters
OpenCV detectMultiScale() minNeighbors parameter

Instead reading images as Bitmap and then converting them to Mat via using Utils.bitmapToMat(defaultBitmap,image) you can directly use Mat image = Highgui.imread(imagepath); You can check here for imread() function.
Also, below line takes too much time because the detector is looking for faces with at least having Size(20, 20) which is pretty small. Check this video for visualization of face detection using OpenCV.
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));

What is the limiting factor in loading images in Android?

So I'm getting really confused here. The designer I work with wants high-quality images (png files) for Android tablets, but the game also has smaller images for less-powerful devices. I figured that the amount of memory on the heap would be the metric to determine which set of images to use, by using Runtime.getRuntime().maxMemory() - Runtime.getRuntime().totalMemory(). That doesn't seem to be the case though. On BlueStacks it can load the high-quality images just fine, and it has around 40,000,000 bytes. The designer's Galaxy Nexus has black boxes for some of the larger images (which I understand is due to a lack of memory for loading the image), but his Galaxy Nexus has about 50,000,000 available bytes, which is even more than BlueStacks.
So what is the limiting factor? And on a related matter, how is it that there are mobile games that have impressive quality visuals, yet I can't manage to load a few images? What am I doing wrong?
To note, I am using AndEngine, and below is an example of how I'm loading the images.
BuildableBitmapTextureAtlas resetTA = new BuildableBitmapTextureAtlas(this.getTextureManager(), 310 / d, 190 / d,
TextureOptions.BILINEAR);
resetTR = BitmapTextureAtlasTextureRegionFactory.createTiledFromAsset(resetTA, this, "gfx/" + lowres + "reset.png", 1, 1);
try
{
resetTA.build(new BlackPawnTextureAtlasBuilder<IBitmapTextureAtlasSource, BitmapTextureAtlas>(0, 0, 0));
resetTA.load();
}
catch (TextureAtlasBuilderException e)
{
Debug.e(e);
}
One of the images that isn't loading in the Galaxy Nexus is a sprite sheet png file that's 2320x464.

There are two limiting factors here. First it's the heap memory. To find the available heap for your app you can use the method with the Runtime class, but that will tell you the maximum memory your app can use before it completely crashes. A limit that your app should respect in Android can be found this way:
ActivityManager am = (ActivityManager) getSystemService(ACTIVITY_SERVICE);
int memoryClass = am.getMemoryClass();
Log.d("MyTag", "Heap: + Integer.toString(memoryClass));
The second is GL_MAX_TEXTURE_SIZE value that limits the maximum dimension of a square texture for given device. This can vary, but the minimum these days seems to be 2048, therefore your textures can be as much as 2048x2048 pixels large. However the only recommendation is that the dimension must be larger than the screen dimensions and the real size is up to the manufacturer of the phone.
I think you can use the following code to find out the size:
int[] maxTextureSize = new int[1];
GLES20.glGetIntegerv(GLES20.GL_MAX_TEXTURE_SIZE, maxTextureSize, 0);
Log.d("MyTag", "GL_MAX_TEXTURE_SIZE: " + Integer.toString(int[0]));
The games can have impressive graphics before they split the big textures to smaller, load only what is needed and reuse as much as possible. I've made a game where some levels have 50000px wide ground by assembling it from 256x256 pieces making the game sharp even on full HD tablets. The pieces were distributed over several 2048x2048 textures.

HW accelerated activity - how to get OpenGL texture size limit?

I'm trying to enable hw acceleration in Honeycomb, and display some Bitmaps on Canvas.
All works fine, but for large bitmaps (>2048 in one dimension), I get error in log:
OpenGLRenderer: Bitmap too large to be uploaded into a texture
I know this is because of hw limitation, and can work-around it by reducing max bitmap size to be displayed if hw acceleration is enabled (checking by View.isHardwareAccelerated()).
My question is: how to easily determine max texture size available for Bitmap drawing by hardware.
2048 seems to be limit on my device, but it may be different on different ones.
Edit: I'm not creating OpenGL app, just normal app, which can utilize hw acceleration. Thus I'm not familiar with OpenGL at all, I just see OpenGL related error in log, and look to solve it.

Currently the minimum limit is 2048px (i.e. the hardware must support textures at least 2048x2048.) In ICS we will introduce a new API on the Canvas class that will give you this information:
Canvas.getMaximumBitmapWidth() and Canvas.getMaximumBitmapHeight().

Another way of getting the maximum allowed size would be to loop through all EGL10 configurations and keep track of the largest size.
public static int getMaxTextureSize() {
// Safe minimum default size
final int IMAGE_MAX_BITMAP_DIMENSION = 2048;
// Get EGL Display
EGL10 egl = (EGL10) EGLContext.getEGL();
EGLDisplay display = egl.eglGetDisplay(EGL10.EGL_DEFAULT_DISPLAY);
// Initialise
int[] version = new int[2];
egl.eglInitialize(display, version);
// Query total number of configurations
int[] totalConfigurations = new int[1];
egl.eglGetConfigs(display, null, 0, totalConfigurations);
// Query actual list configurations
EGLConfig[] configurationsList = new EGLConfig[totalConfigurations[0]];
egl.eglGetConfigs(display, configurationsList, totalConfigurations[0], totalConfigurations);
int[] textureSize = new int[1];
int maximumTextureSize = 0;
// Iterate through all the configurations to located the maximum texture size
for (int i = 0; i < totalConfigurations[0]; i++) {
// Only need to check for width since opengl textures are always squared
egl.eglGetConfigAttrib(display, configurationsList[i], EGL10.EGL_MAX_PBUFFER_WIDTH, textureSize);
// Keep track of the maximum texture size
if (maximumTextureSize < textureSize[0])
maximumTextureSize = textureSize[0];
}
// Release
egl.eglTerminate(display);
// Return largest texture size found, or default
return Math.max(maximumTextureSize, IMAGE_MAX_BITMAP_DIMENSION);
}
From my testing, this is pretty reliable and doesn't require you to create an instance.
Performance-wise, this took 18 milliseconds to execute on my Note 2 and only 4 milliseconds on my G3.

If you want to know dynamically the texture size limit of your device (because it's change depending on the device), you have to call this method:
int[] maxTextureSize = new int[1];
gl.glGetIntegerv(GL10.GL_MAX_TEXTURE_SIZE, maxTextureSize, 0);
And don't forget that for some device (the Nexus One for example), the texture size must be a power of 2 !
I know my answer comes a long time after the last update of this topic...sorry

According to the specification, calling glGetIntegerv with GL_MAX_TEXTURE_SIZE.
GL_MAX_TEXTURE_SIZE params returns one value. The value gives a rough estimate of the largest texture that the GL can handle. The value must be at least 64.
http://www.khronos.org/opengles/sdk/docs/man/xhtml/glGet.xml

Opengl es 1.1, texture compression ETC1 and Mipmaping (complete set of mipmaps error)

When I activate the mipmaping on uncompressed texture, all is working perfectly.
When I do it on ETC1 texture, the texture is blank, certainly because le complete set of mipmaps was not given.
The code is very simple and works on iPhone (with PVR compression, of course).
It doesn't work on Android. The mipmap was build with an external tool, and past together.
I stop making mipmap at the size of 4, because glCompressedTexImage2D return an opengl error if try using mipmap lower.
for(u32 i=0; i<=levels; i++)
{
size = KC_TexByte(pagex, pagey, tex_type);
glCompressedTexImage2D(GL_TEXTURE_2D, i, type, pagex, pagey, 0, size, ptr);
pagex = MAX(pagex/2, 4);
pagey = MAX(pagey/2, 4);
ptr += size;
KC_Error(); // test openGL error
}

The reason your texture is blank is because it is required that the mipmap go all the way to 1x1.
I would imagine that the error you're getting with small compressed textures is because the texture format you're attempting to use (etc1?) doesn't support those sizes. You'd have to use non-compressed images at those small sizes...

Thanks, but your solution is not the right one; I found another solution.
you're right when you explain that all the mipmap is requiered, until size 1x1
you're wrong, we can't have different format between mipmap
The right way is:
using size to 1x1
keep in mind it's compressed data with bloc, so the size in BYTE doesn't divide by 4 each step. after 8x8, the size stay at the same value.
sx = size in X
sy = size in Y
byte = ((sx+3)/4)*((sy+3)/4) * 8 * 2; // 8 = bit per pixel

for(u32 i=0; i<=levels; i++)
Seems you'd want i < levels instead of <=.

How can I find the maximum texture size for different phones?

I'm trying to find out the maximum texture size for the original Motorola Droid. I believe the G1 has a maximum texture size of 512, but it would be nice if there was a more official way I could find out so I can build a proper tile system.

You can request the max texture size using glGetIntegerv:
int[] maxTextureSize = new int[1];
gl.glGetIntegerv(GL10.GL_MAX_TEXTURE_SIZE, maxTextureSize, 0);
Log.i("glinfo", "Max texture size = " + maxTextureSize[0]);

Also check out http://www.glbenchmark.com/ - it has an extensive database of OpenGL environment and performance details for mobile devices