Develop Reference

How to improve OpenCV face detection performance in android?

I am working on a project in android in which i am using OpenCV to detect faces from all the images which are in the gallery. The process of getting faces from the images is performing in the service. Service continuously working till all the images are processed. It is storing the detected faces in the internal storage and also showing in the grid view if activity is opened.
My code is:
CascadeClassifier mJavaDetector=null;
public void getFaces()
{
for (int i=0 ; i<size ; i++)
{
File file=new File(urls.get(i));
imagepath=urls.get(i);
defaultBitmap=BitmapFactory.decodeFile(file, bitmapFatoryOptions);
mJavaDetector = new CascadeClassifier(FaceDetector.class.getResource("lbpcascade_frontalface").getPath());
Mat image = new Mat (defaultBitmap.getWidth(), defaultBitmap.getHeight(), CvType.CV_8UC1);
Utils.bitmapToMat(defaultBitmap,image);
MatOfRect faceDetections = new MatOfRect();
try
{
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));
}
catch(Exception e)
{
e.printStackTrace();
}
if(faceDetections.toArray().length>0)
{
}
}
}
Everything is fine but it is detection faces very slow. The performance is very slow. When i debug the code then i found the line which is taking time is:
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));
I have checked multiple post for this problem but i didn't get any solution.
Please tell me what should i do to solve this problem.
Any help would be greatly appreciated. Thank you.

You should pay attention to the parameters of detectMultiScale():
scaleFactor – Parameter specifying how much the image size is reduced at each image scale. This parameter is used to create a scale pyramid. It is necessary because the model has a fixed size during training. Without pyramid the only size to detect would be this fix one (which can be read from the XML also). However the face detection can be scale-invariant by using multi-scale representation i.e., detecting large and small faces using the same detection window.
scaleFactor depends on the size of your trained detector, but in fact, you need to set it as high as possible while still getting "good" results, so this should be determined empirically.
Your 1.1 value can be a good value for this purpose. It means, a relative small step is used for resizing (reduce size by 10%), you increase the chance of a matching size with the model for detection is found. If your trained detector has the size 10x10 then you can detect faces with size 11x11, 12x12 and so on. But in fact a factor of 1.1 requires roughly double the # of layers in the pyramid (and 2x computation time) than 1.2 does.
minNeighbors – Parameter specifying how many neighbours each candidate rectangle should have to retain it.
Cascade classifier works with a sliding window approach. By applying this approach, you slide a window through over the image than you resize it and search again until you can not resize it further. In every iteration the true outputs (of cascade classifier) are stored but unfortunately it actually detects many false positives. And to eliminate false positives and get the proper face rectangle out of detections, neighbourhood approach is applied. 3-6 is a good value for it. If the value is too high then you can lose true positives too.
minSize – Regarding to the sliding window approach of minNeighbors, this is the smallest window that cascade can detect. Objects smaller than that are ignored. Usually cv::Size(20, 20) are enough for face detections.
maxSize – Maximum possible object size. Objects bigger than that are ignored.
Finally you can try different classifiers based on different features (such as Haar, LBP, HoG). Usually, LBP classifiers are a few times faster than Haar's, but also less accurate.
And it is also strongly recommended to look over these questions:
Recommended values for OpenCV detectMultiScale() parameters
OpenCV detectMultiScale() minNeighbors parameter

Instead reading images as Bitmap and then converting them to Mat via using Utils.bitmapToMat(defaultBitmap,image) you can directly use Mat image = Highgui.imread(imagepath); You can check here for imread() function.
Also, below line takes too much time because the detector is looking for faces with at least having Size(20, 20) which is pretty small. Check this video for visualization of face detection using OpenCV.
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));

How to create a circular loop of animations in AndEngine?

I want to do an "chain" or circular loop of animations as can be described below:
LABEL start
do Anim1->Anim2->Anim3->Anim4
GOTO start
The above will do a circular loop: Anim1->Anim2->Anim3->Anim4 and back to Anim1 and so on.
I am not able to merge all the PNGs in one Texture because Andengine/Android is limited in loading the resources. However, when I split my initial large tile into 4 smaller tiles, everything works fine.
I tried to use an AnimationListener inside Anim1. When onAnimationFinished() is called, I detach Anim1, and run Anim2 and do this in chain of inner functions. However, when I am in Anim4, I do not know how to go back to the start and attach Anim1.
Note: All this problem could be solved if you know how I can pack a set of 150 PNGs that individually quite large but fit in a tile of 4096x4096 px.
Thank you for your help!
EDIT (following JiMMaR's proposed solution):
I am using Texture Packer and the overall Texture exceeds 4096*4096, causing an OutOfMemory error on Android.
At the moment, I have split the Textures into four tiles and I four PNG tilemaps.

You can use 'Texture Packer' to see if all your images can fit in 4096 x 4096.
You can download Texture Packer from here. (NOTE: Texture Packer supports AndEngine data output too)
You can use "BuildableBitmapTextureAtlas" and "BlackPawnTextureAtlasBuilder" classes to pack your PNGs into one texture atlas.

You should post some code so we can see the implementation.
Try to use several AnimatedSprites with animation listeners in each one. This way you can start the animation of the next sprite in the onAnimationFinished() call.
private class AnimationLooperListener implements IAnimationListener{
private AnimatedSprite nextSpriteToAnimate;
public AnimationLooperListener(AnimatedSprite sprite){
nextSpriteToAnimate = sprite;
}
/** other methods are hidden */
public void onAnimationFinished(AnimatedSprite sprite){
sprite.setVisible(false);
nextSpriteToAnimate.setVisible(true);
nextSpriteToAnimate.animate(100, new AnimationLooperListener(sprite);
}
}
AnimatedSprite sprite1 = new AnimatedSprite(0, 0, tiledTextureRegion, vertex);
AnimatedSprite sprite2 = new AnimatedSprite(0, 0, tiledTextureRegion, vertex);
AnimatedSprite sprite2 = new AnimatedSprite(0, 0, tiledTextureRegion, vertex);
AnimationLooperListener listener1 = new AnimationLooperListener(sprite2);
AnimationLooperListener listener2 = new AnimationLooperListener(sprite3);
AnimationLooperListener listener3 = new AnimationLooperListener(sprite1);
sprite1.animate(100, listener1);
sprite2.animate(100, listener2);
sprite3.animate(100, listener3);
This way, you have an animation loop, between several sprites that can be created using several TiledTextureRegions.

How load a lot of textures in andengine

I am trying to load textures as follows:
private Texture mTexture;
...
public Textures(final BaseGameActivity activity, final Engine engine) {
this.mTexture = new Texture(2048, 1024,
TextureOptions.BILINEAR_PREMULTIPLYALPHA);
this.mBackgroundTextureRegion = TextureRegionFactory.createFromAsset(
this.mTexture, activity, "img/back.png", 0, 0);
this.mSwingBackTextureRegion = TextureRegionFactory.createFromAsset(
this.mTexture, activity, "img/player.png", 836, 0);
...
I want to load more than 200 textures. However, the current method that I am using is too long.
Are there faster methods to complete it?
I am working in GLES1.

The easiest way to do it is with Texture Packer, found here
This allows you to add multiple image files in to one easy to load spritesheet. The engine loads this spritesheet in to a texture and creates a class that lets you easily reference each image from that spreadsheet. Turn 200 TextureRegions in to 1 TexturePack.
I'm using GLES2 and I'm not sure where the source files are for GLES1. Poke around the forums and you should be able to find out how to use them. There has been plenty of talk about it.

There is a texture packer built in AndEngine which does this automagically. Try searching the AndEngine forum.
http://www.andengine.org/forums/

AndEngine getting error: Supplied pTextureAtlasSource must not exceed bounds of Texture

im new to android gaming and started andengine and facing problem while using createTiledFromAsset
the code where im getting problem is
#Override
public void onLoadResources() {
mBitmapTextureAtlas = new BitmapTextureAtlas(128, 128,
TextureOptions.BILINEAR);
BitmapTextureAtlasTextureRegionFactory.setAssetBasePath("gfx/");
mPlayerTextureRegion = BitmapTextureAtlasTextureRegionFactory
.createTiledFromAsset(this.mBitmapTextureAtlas, this,
"move.png", 0, 0, 10, 1);
mEngine.getTextureManager().loadTexture(mBitmapTextureAtlas);
}
#Override
public Scene onLoadScene() {
mEngine.registerUpdateHandler(new FPSLogger());
mMainScene = new Scene();
mMainScene
.setBackground(new ColorBackground(0.09804f, 0.6274f, 0.8784f));
player = new AnimatedSprite(0, 0, mPlayerTextureRegion);
mMainScene.attachChild(player);
return mMainScene;
}
im not getting the error as my BitmapTextureAtlas is of 128*128 and each tiled part coming from createTiledFromAsset should be of 78*85 as the passed arguments to it are 1 row and 10 columns and my source image is of 779*85 which means when the width is tiled to 10 parts then 779/10=78 approx which will be width of each tiled part and as row is 1 so 85/1=85 hence the width*height of each tiled part which is to be placed on the BitmapTextureAtlas is 78*85 and the BitmapTextureAtlas itself has size 128*128 then why the error saying Supplied pTextureAtlasSource must not exceed bounds of Texture
what is happening here ...? or im not understanding the actual functions ...? if im wrong then how the process of createTiledFromAsset is working........?

I found the same problem too. I am using a large sprite sheet, and try to use it. after searching, I found some clue in this tutorial: getting-started-working-with-images and i realize that the value of width and height used in:
BitmapTextureAtlas(WIDTH, HEIGHT ,TextureOptions.BILINEAR);
must be higher than the image size. for example if I use 1200*100 sprite sheet, I must use the width and height higher than 1200*100.
BitmapTextureAtlas(2047, 128, TextureOptions.BILINEAR);

If I understand BitmapTextureAtlas correctly, you are trying to put 779*85 image into the small space of 128*128. TextureAtlas is a large canvas on which you are supposed to place many images. These images are later accessed using object called TextureRegion, which basically specifies the size and coordinates of the smaller picture on the canvas. The method createTiledFromAsset probably copies the original image 1:1 onto the TextureAtlas and saves the coordinates of the tiles.
Please note that TextureRegion has nothing to do with the image itself, it is merely a "pointer" to the place on TextureAtlas where the image is stored.
To get the idea of what a TextureAtlas actually is, look at the awesome pictures at the bottom of this page:
http://www.blackpawn.com/texts/lightmaps/default.html

Spritesheet programmatically cutting: best practices

I have a big spritesheet (3808x1632) composed by 42 frames.
I would present an animation with these frames and I use a thread to load a bitmap array with all the frames, with a splash screen waiting for its end.
I'm not using a SurfaceView (and a draw function of a canvas), I just load frame by frame in an ImageView in my main layout.
My approach is similar to Loading a large number of images from a spritesheet
The completion actually takes almost 15 seconds, not acceptable.
I use this kind of function:
for (int i=0; i<TotalFramesTeapotBG; i++) {
xStartTeapotBG = (i % framesInRowsTeapotBG) * frameWidthTeapotBG;
yStartTeapotBG = (i / framesInRowsTeapotBG) * frameHeightTeapotBG;
mVectorTeapotBG.add(Bitmap.createBitmap(framesBitmapTeapotBG, xStartTeapotBG, yStartTeapotBG, frameWidthTeapotBG, frameHeightTeapotBG));
}
framesBitmapTeapotBG is the big spritesheet.
Looking more deeply, I've read in the logcat that the createBitmap function takes a lot of time, maybe because the spritesheet is too big.
I found somewhere that I could make a window on the big spritesheet, using the rect function and canvas, creating small bitmaps to be loaded in the array, but it was not really clear. I'm talking about that post: cut the portion of bitmap
My question is: how can I speed the spritesheet cut?
Edit:
I'm trying to use this approach but I cannot see the final animation:
for (int i=0; i<TotalFramesTeapotBG; i++) {
xStartTeapotBG = (i % framesInRowsTeapotBG) * frameWidthTeapotBG;
yStartTeapotBG = (i / framesInRowsTeapotBG) * frameHeightTeapotBG;
Bitmap bmFrame = Bitmap.createBitmap(frameWidthTeapotBG, frameHeightTeapotBG, Bitmap.Config.ARGB_8888);
Canvas c = new Canvas(bmFrame);
Rect src = new Rect(xStartTeapotBG, yStartTeapotBG, frameWidthTeapotBG, frameHeightTeapotBG);
Rect dst = new Rect(0, 0, frameWidthTeapotBG, frameHeightTeapotBG);
c.drawBitmap(framesBitmapTeapotBG, src, dst, null);
mVectorTeapotBG.add(bmFrame);
}
Probably, the Bitmap bmFrame is not correctly managed.

The short answer is better memory management.
The sprite sheet you're loading is huge, and then you're making a copy of it into a bunch of little bitmaps. Supposing the sprite sheet can't be any smaller, I'd suggest taking one of two approaches:
Use individual bitmaps. This will reduce the memory copies as well as the number of times Dalvik will have to grow the heap. However, these benefits may be limited by the need to load many images off the filesystem instead of just one. This would be the case in a normal computer, but Android systems may get different results since they're run off flash memory.
Blit directly from your sprite sheet. When drawing, just draw straight from sprite sheet using something like Canvas.drawBitmap(Bitmap bitmap, Rect src, Rect dst, Paint paint). This will reduce your file loads to one large allocation that probably only needs to happen once in the lifetime of your activity.
I think the second option is probably the better of the two since it will be easier on the memory system and be less work for the GC.

Thanks to stevehb for the suggestion, I finally got it:
for (int i = 0; i < TotalFramesTeapotBG; i++) {
xStartTeapotBG = (i % framesInRowsTeapotBG) * frameWidthTeapotBG;
yStartTeapotBG = (i / framesInRowsTeapotBG) * frameHeightTeapotBG;
Bitmap bmFrame = Bitmap.createBitmap(frameWidthTeapotBG, frameHeightTeapotBG, Bitmap.Config.ARGB_8888);
Canvas c = new Canvas(bmFrame);
Rect src = new Rect(xStartTeapotBG, yStartTeapotBG, xStartTeapotBG+frameWidthTeapotBG, yStartTeapotBG+frameHeightTeapotBG);
Rect dst = new Rect(0, 0, frameWidthTeapotBG, frameHeightTeapotBG);
c.drawBitmap(framesBitmapTeapotBG, src, dst, null);
mVectorTeapotBG.add(bmFrame);
}
The computation time falls incredibly! :)

Use a LevelListDrawable. Cut the sprites into individual frames and drop them in your drawable resource directory. Either programmatically or through an xml based level-list drawable create your drawable. Then use ImageView.setImageLevel() to pick your frame.

I use a method of slicing based on rows and columns. However your sprite sheet is rather huge. You have to think you are putting that whole sheet into memory. 3808x1632x4 is the size of the image in memory.
Anyway, what I do is I take an image (lets say a 128x128) and then tell it there are 4 columns and 2 rows in the Sprite(bitmap, 4, 2) constructor. Then you can slice and dice based on that. bitmap.getWidth() / 4 etc... pretty simple stuff. However if you want to do some real stuff use OpenGL and use textures.
Oh I also forgot to mention there are some onDraw stuff that needs to happen. Basically you keep an index counter and slice a rectangle from the bitmap and draw that from a source rectangle to a destination rectangle on the canvas.