I come from the Qt world and i am porting an application to Android. I am bit confused, i am banging my head for a few days now on something that must be so trivial that i cannot find why it's not working.
Some background: i have a C++ engine which i use trough NDK and JNI. This engine creates some bitmaps and passes them to the Java side, the Java side must display them on a View and let the user interact with them (drag and such).
The engine works properly, because i use it under Qt with full success. This is the workflow:
1- Java loads a big Bitmap from a custom data file (the C++ engine expects it to be in ARGB format, but it's compressed JPG data)
Bitmap.Config fmt = Bitmap.Config.ARGB_8888;
Bitmap bitmap = BitmapFactory.decodeByteArray(buffer, 0, size).copy( fmt , false);
2- initialize the C++ engine passing the bitmap. The C++ engine "breaks" the bitmap in smaller tiles. For tile it builds a rather complex alpha mask and stores it into the first byte of the bitmap (the "a" byte). This alpha mask only uses two values: 0xFF for opaque and 0x00 for transparent.
init_C_engine( this.fullImage );
3- The Java side then allocates all the tiles bitmaps, i do in two steps because before init i dont know which size will the tiles be. The engine will populate the tile_width and tile_height arrays:
Bitmap.Config fmt = Bitmap.Config.ARGB_8888;
for (int t = 0; t < this.puzzle_size; t++ ){
tile_data[ t ] = Bitmap.createBitmap( tile_width[t], tile_height[t], fmt);
4- Last step,inside the C++ engine, all the tiles bitmaps are filled:
for ( int n = 0; n < nBitmaps; n++ )
{
jobject bitmap = env->GetObjectArrayElement( bitmaps, n );
AndroidBitmap_getInfo(env, bitmap, &info);
AndroidBitmap_lockPixels(env, bitmap, reinterpret_cast<void **>(&pixels));
game->getTileBitmap( n, (unsigned char*)pixels );
AndroidBitmap_unlockPixels(env, bitmap);
env->SetObjectArrayElement( bitmaps, n, bitmap );
}
}
Now, in my custom View:
protected void onDraw(Canvas canvas) {
super.onDraw(canvas);
canvas.drawColor(Color.BLACK);
for ( int tile = 0; tile < board.nTiles; tile++ ){
canvas.drawBitmap( tile_data[tile],
tile_x[tile],
tile_y[tile], paint);
}
}
What i expect is that on my View i see my tiles with transparent areas. what i get instead is a weird behaviour so that on the black background i see the ENTIRE tile like the alpha bytes are all set to opaque, but when i move the tiles one of top of the other, the "transparent" areas get combined in some strange way, like colors are "xor"ed or multiplied in some way! When i move one tile on the other i can see the areas where the alpha bytes are set to transparent but colors gets mangled instead of being transparend!
Basically i expect that pixels having alpha set to 0 are drawn as transparent... i looked on internet but i could not find anything usefull to help me out....
Does somebody have ideas? Anything will be appreciated!
thanks.
Shouldn't you use the index t instead of tile inside the for loop inside onDraw? Like this:
canvas.drawBitmap(tile_data[t], tile_x[t], tile_y[t], paint);
Related
I am kind of stuck with this problem, and I know there are so many questions about it on stack overflow but in my case. Nothing gives the expected result.
The Context:
Am using Android OpenCV along with Tesseract so I can read the MRZ area in the passport. When the camera is started I pass the input frame to an AsyncTask, the frame is processed, the MRZ area is extracted succesfully, I pass the extracted MRZ area to a function prepareForOCR(inputImage) that takes the MRZ area as gray Mat and Will output a bitmap with the thresholded image that I will pass to Tesseract.
The problem:
The problem is while thresholding the Image, I use adaptive thresholding with blockSize = 13 and C = 15, but the result given is not always the same depending on the lighting of the image and the conditions in general from which the frame is taken.
What I have tried:
First I am resizing the image to a specific size (871,108) so the input image is always the same and not dependant on which phone is used.
After resizing, I try with different BlockSize and C values
//toOcr contains the extracted MRZ area
Bitmap toOCRBitmap = Bitmap.createBitmap(bitmap);
Mat inputFrame = new Mat();
Mat toOcr = new Mat();
Utils.bitmapToMat(toOCRBitmap, inputFrame);
Imgproc.cvtColor(inputFrame, inputFrame, Imgproc.COLOR_BGR2GRAY);
TesseractResult lastResult = null;
for (int B = 11; B < 70; B++) {
for (int C = 11; C < 70; C++){
if (IsPrime(B) && IsPrime(C)){
Imgproc.adaptiveThreshold(inputFrame, toOcr, 255, Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C, Imgproc.THRESH_BINARY, B ,C);
Bitmap toOcrBitmap = OpenCVHelper.getBitmap(toOcr);
TesseractResult result = TesseractInstance.extractFrame(toOcrBitmap, "ocrba");
if (result.getMeanConfidence()> 70) {
if (MrzParser.tryParse(result.getText())){
Log.d("Main2Activity", "Best result with " + B + " : " + C);
return result;
}
}
}
}
}
Using the code below, the thresholded result image is a black on white image which gives a confidence greater than 70, I can't really post the whole image for privacy reasons, but here's a clipped one and a dummy password one.
Using the MrzParser.tryParse function which adds checks for the character position and its validity within the MRZ, am able to correct some occurences like a name containing a 8 instead of B, and get a good result but it takes so much time, which is normal because am thresholding almost 255 images in the loop, adding to that the Tesseract call.
I already tried getting a list of C and B values which occurs the most but the results are different.
The question:
Is there a way to define a C and blocksize value so that it s always giving the same result, maybe adding more OpenCV calls so The input image like increasing contrast and so on, I searched the web for 2 weeks now I can't find a viable solution, this is the only one that is giving accurate results
You can use a clustering algorithm to cluster the pixels based on color. The characters are dark and there is a good contrast in the MRZ region, so a clustering method will most probably give you a good segmentation if you apply it to the MRZ region.
Here I demonstrate it with MRZ regions obtained from sample images that can be found on the internet.
I use color images, apply some smoothing, convert to Lab color space, then cluster the a, b channel data using kmeans (k=2). The code is in python but you can easily adapt it to java. Due to the randomized nature of the kmeans algorithm, the segmented characters will have label 0 or 1. You can easily sort it out by inspecting cluster centers. The cluster-center corresponding to characters should have a dark value in the color space you are using.
I just used the Lab color space here. You can use RGB, HSV or even GRAY and see which one is better for you.
After segmenting like this, I think you can even find good values for B and C of your adaptive-threshold using the properties of the stroke width of the characters (if you think the adaptive-threshold gives a better quality output).
import cv2
import numpy as np
im = cv2.imread('mrz1.png')
# convert to Lab
lab = cv2.cvtColor(cv2.GaussianBlur(im, (3, 3), 1), cv2.COLOR_BGR2Lab)
im32f = np.array(im[:, :, 1:3], dtype=np.float32)
k = 2 # 2 clusters
term_crit = (cv2.TERM_CRITERIA_EPS, 30, 0.1)
ret, labels, centers = cv2.kmeans(im32f.reshape([im.shape[0]*im.shape[1], -1]),
k, None, term_crit, 10, 0)
# segmented image
labels = labels.reshape([im.shape[0], im.shape[1]]) * 255
Some results:
I started by reading in this Mat.
Then I converted it to Greyscale and applied Imgproc.canny() to it, getting the following mask.
Then I used Imgproc.findContours() to find the contours, Imgproc.drawContours(), and Core.putText() to label the contours with numbers:
Then I did Rect boundingRect = Imgproc.boundingRect(contours.get(0));
Mat submatrix = new Mat();
submatrix = originalMat.submat(boundingRect); to get following submatrix:
So far so good. The Problem starts hereafter:
NOW I NEEDED A MASK OF THE submatrix. So I decided to use Imgproc.drawContours() to get the mask:
Mat mask = new Mat(submatrix.rows(), submatrix.cols(), CvType.CV_8UC1);
List<MatOfPoint> contourList = new ArrayList<>();
contourList.add(contours.get(0));
Imgproc.drawContours(mask, contourList, 0, new Scalar(255), -1);
I got the following mask:
WHAT I WAS EXPECTING was a filled (in white color) diamond shape on black background.
WHy am I getting this unexpected result?
EDIT:
When I replaced Mat mask = new Mat(submatrix.rows(),
submatrix.cols(), CvType.CV_8UC1); by Mat mask =
Mat.zeros(submatrix.rows(), submatrix.cols(), CvType.CV_8UC1);,
the last mask with white colored garbage was replaced by an empty
black mask withOUT any white color on it. I got the following submat
and mask:
I was getting the first contour in the list of contours (named
contours) by contours.get(0), and using this first contour to
calculate Imgproc.boundingRect() as well as in
contourList.add(contours.get(0)); later (where contourList is
the list of just one contour which will be used in the last
drawContours()).
Then I went ahead to change contours.get(0) to
contours.get(1) in Imgproc.boundingRect() as well as in contourList.add(); (just before Imgproc.drawContours()). That
resulted in this submat and mask:
Then I changed back to contours.get(0) in
Imgproc.boundingRect(); and let
contourList.add(contours.get(1)); be there. Got the following
submat and mask:
NOW I am completely Unable to Understand what is happening here.
I am not sure how this is handle in JAVA (I usually use OpenCV in c++ or python), but there is an error in your code...
The contours list will have a list of list of points. This points will refer to the original image. So, this mean that if the figure one is in lets say, x=300, y= 300, width= 100, height=100 then when you get your submatrix it will try to draw those points in a smaller image... so when it tries to draw point (300,300) in a 100 x 100 image, it will simply fail... probably throws an error or simply doesn't draw anything...
A solution for this is, do a for loop and substract to each point of the contour the initial point of the bounding rect (in my example (300,300)).
As, why there is some garbage drawn... well you never initialize the matrix. Not sure in JAVA, but in c++ you have to set them to 0.
I think it should be something like this:
Mat mask = new Mat(submatrix.rows(), submatrix.cols(), CvType.CV_8UC1, new Scalar(0));
I hope this helps :)
EDIT
I think I did not explain myself clearly before.
Your contours are an array of points (x,y). These are the coordinates of the points that represent each contour in the original image. This image has a size, and your submatrix has a smaller size. The points are outside of this small image boundaries....
you should do something like this to fix it:
for (int j = 0; j < contours[0].length; j++) {
contours[0][j].x -= boundingrect.x;
contours[0][j].y -= boundingrect.y;
}
and then you can draw the contours, since they will be in boundaries of the submat.
I think in java it is also possible to subtract the opencv points directly:
for (int j = 0; j < contours[0].length; j++) {
contours[0][j] -= boundingrect.tl();
}
but in this case I am not sure, since I have tried it in c++ only
boundingrect.tl() -> gives you the top left point of the rect
I am new at this - asking questions, android developement and NDK. I hope I am clear enough.
I need to be able to create multiple surfaces/bitmaps.
e.g.
Surface s = new Surface (width, height)
they can copy between each other
s->copy (s2) copy surface s to s2 (including format conversion between RGBA and alpha-text surface and resizing/scaling)
use fill (x, u, w, h, color) - fill rectangle with color (something like glClear)
As far as I understand you have only one ANativeWindow which is supplied to you by android_app->window variable and if I use EGL I can create upto 1 EGLSurface. I need to be able to create many surfaces (~ 100 for instance). How is this possible? And then blit all of them to the window framebuffer
There is also android/bitmap.h But I am not getting it exactly how to work with it. But it does not offer me API to create surface, just to get already created or something like this?
You can create bitmap through JNI calls:
// setup bitmap class
jclass bitmap_class = (jclass)env->FindClass ("android/graphics/Bitmap");
// setup create method
jmethodID bitmap_create_method = env->GetStaticMethodID (bitmap_class, "createBitmap", "(IILandroid/graphics/Bitmap$Config;)Landroid/graphics/Bitmap;");
// get_enum_value return jobject corresponding in our case to Bitmap.Config.ARGB_8888. (the implentation is irrelevant here)
jobject bitmap_config_ARGB = get_enum_value ("android/graphics/Bitmap$Config", "ARGB_8888");
// Do not forget to call DeleteLocalRef where appropriate
// create the bitmap by calling the CreateBitmap method
// Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
jobject bitmap = env->CallStaticObjectMethod (bitmap_class, bitmap_create_method, width, height, bconfig);
// at the end of course clean-up must be done
env->DeleteLocalRef (bitmap);
You can access some bitmap properties and the raw pixels through the API in android/bitmap.h
AndroidBitmap_getInfo gives information about format (ARGB_8888 or alpha-only), dimensions, stride or pitch.
AndroidBitmap_lockPixels give the raw pixels. After finished manipulating the pixels one MUST call AndroidBitmap_unlockPixels
To make fill (color, dimension)
JNI can help. This can be written through JNI calls (I will use java because it is easier for me to write and clearer to read).
canvas.save ();
canvas.setBitmap (bitmap);
canvas.clipRect (left, top, right, bottom, Region.Op.REPLACE);
canvas.drawColor (color, PorterDuff.Mode.SRC);
canvas.restore ();
To copy one bitmap over another one - copy (src_bitmap, src_rect, dest_rect)
canvas.save ();
canvas.setBitmap (dest_bitmap);
canvas.clipRect (left, top, right, bottom, Region.Op.REPLACE);
canvas.drawBitmap (src_bitmap, src_rect, dest_rect, null);
canvas.restore ();
You can create Bitmaps and use the jnigraphics library (android/bitmap.h) or you can use multiple EGL textures.
Using Bitmaps you'll have to implement fill yourself, because Bitmap does only have pixel-based getters and setters (see setPixels(..))
I'm trying to port an emulator that i have written in java to android. Things have been going nicely, I was able to port most of my codes with minor changes however due to how emulation works, I need to render image at pixel level.
As for desktop java I use
int[] pixelsA = ((DataBufferInt) src.getRaster().getDataBuffer()).getData();
which allow me to get the reference to the pixel buffer and update it on the fly(minimize object creations)
Currently this is what my emulator for android does for every frame
#Override
public void onDraw(Canvas canvas)
{
buffer = Bitmap.createBitmap(pixelsA, 256, 192, Bitmap.Config.RGB_565);
canvas.drawBitmap(buffer, 0, 0, null);
}
pixelsA is an array int[], pixelsA contains all the colour informations, so every frame it will have to create a bitmap object by doing
buffer = Bitmap.createBitmap(pixelsA, 256, 192, Bitmap.Config.RGB_565);
which I believe is quite expensive and slow.
Is there any way to draw pixels efficiently with canvas?
One quite low-level method, but working fine for me (with native code):
Create Bitmap object, as big as your visible screen.
Also create a View object and implement onDraw method.
Then in native code you'd load libjnigraphics.so native library, lookup functions AndroidBitmap_lockPixels and AndroidBitmap_unlockPixels.
These functions are defined in Android source in bitmap.h.
Then you'd call lock/unlock on a bitmap, receiving address to raw pixels. You must interpret RGB format of pixels accordingly to what it really is (16-bit 565 or 32-bit 8888).
After changing content of the bitmap, you want to present this on screen.
Call View.invalidate() on your View. In its onDraw, blit your bitmap into given Canvas.
This method is very low level and dependent on actual implementation of Android, however it's very fast, you may get 60fps no problem.
bitmap.h is part of Android NDK since platform version 8, so this IS official way to do this from Android 2.2.
You can use the drawBitmap method that avoids creating a Bitmap each time, or even as a last resort, draw the pixels one by one with drawPoint.
Don't recreate the bitmap every single time. Try something like this:
Bitmap buffer = null;
#Override
public void onDraw(Canvas canvas)
{
if(buffer == null) buffer = Bitmap.createBitmap(256, 192, Bitmap.Config.RGB_565);
buffer.copyPixelsFromBuffer(pixelsA);
canvas.drawBitmap(buffer, 0, 0, null);
}
EDIT: as pointed out, you need to update the pixel buffer. And the bitmap must be mutable for that to happen.
if pixelsA is already an array of pixels (which is what I would infer from your statement about containing colors) then you can just render them directly without converting with:
canvas.drawBitmap(pixelsA, 0, 256, 0, 0, 256, 192, false, null);
I have a big spritesheet (3808x1632) composed by 42 frames.
I would present an animation with these frames and I use a thread to load a bitmap array with all the frames, with a splash screen waiting for its end.
I'm not using a SurfaceView (and a draw function of a canvas), I just load frame by frame in an ImageView in my main layout.
My approach is similar to Loading a large number of images from a spritesheet
The completion actually takes almost 15 seconds, not acceptable.
I use this kind of function:
for (int i=0; i<TotalFramesTeapotBG; i++) {
xStartTeapotBG = (i % framesInRowsTeapotBG) * frameWidthTeapotBG;
yStartTeapotBG = (i / framesInRowsTeapotBG) * frameHeightTeapotBG;
mVectorTeapotBG.add(Bitmap.createBitmap(framesBitmapTeapotBG, xStartTeapotBG, yStartTeapotBG, frameWidthTeapotBG, frameHeightTeapotBG));
}
framesBitmapTeapotBG is the big spritesheet.
Looking more deeply, I've read in the logcat that the createBitmap function takes a lot of time, maybe because the spritesheet is too big.
I found somewhere that I could make a window on the big spritesheet, using the rect function and canvas, creating small bitmaps to be loaded in the array, but it was not really clear. I'm talking about that post: cut the portion of bitmap
My question is: how can I speed the spritesheet cut?
Edit:
I'm trying to use this approach but I cannot see the final animation:
for (int i=0; i<TotalFramesTeapotBG; i++) {
xStartTeapotBG = (i % framesInRowsTeapotBG) * frameWidthTeapotBG;
yStartTeapotBG = (i / framesInRowsTeapotBG) * frameHeightTeapotBG;
Bitmap bmFrame = Bitmap.createBitmap(frameWidthTeapotBG, frameHeightTeapotBG, Bitmap.Config.ARGB_8888);
Canvas c = new Canvas(bmFrame);
Rect src = new Rect(xStartTeapotBG, yStartTeapotBG, frameWidthTeapotBG, frameHeightTeapotBG);
Rect dst = new Rect(0, 0, frameWidthTeapotBG, frameHeightTeapotBG);
c.drawBitmap(framesBitmapTeapotBG, src, dst, null);
mVectorTeapotBG.add(bmFrame);
}
Probably, the Bitmap bmFrame is not correctly managed.
The short answer is better memory management.
The sprite sheet you're loading is huge, and then you're making a copy of it into a bunch of little bitmaps. Supposing the sprite sheet can't be any smaller, I'd suggest taking one of two approaches:
Use individual bitmaps. This will reduce the memory copies as well as the number of times Dalvik will have to grow the heap. However, these benefits may be limited by the need to load many images off the filesystem instead of just one. This would be the case in a normal computer, but Android systems may get different results since they're run off flash memory.
Blit directly from your sprite sheet. When drawing, just draw straight from sprite sheet using something like Canvas.drawBitmap(Bitmap bitmap, Rect src, Rect dst, Paint paint). This will reduce your file loads to one large allocation that probably only needs to happen once in the lifetime of your activity.
I think the second option is probably the better of the two since it will be easier on the memory system and be less work for the GC.
Thanks to stevehb for the suggestion, I finally got it:
for (int i = 0; i < TotalFramesTeapotBG; i++) {
xStartTeapotBG = (i % framesInRowsTeapotBG) * frameWidthTeapotBG;
yStartTeapotBG = (i / framesInRowsTeapotBG) * frameHeightTeapotBG;
Bitmap bmFrame = Bitmap.createBitmap(frameWidthTeapotBG, frameHeightTeapotBG, Bitmap.Config.ARGB_8888);
Canvas c = new Canvas(bmFrame);
Rect src = new Rect(xStartTeapotBG, yStartTeapotBG, xStartTeapotBG+frameWidthTeapotBG, yStartTeapotBG+frameHeightTeapotBG);
Rect dst = new Rect(0, 0, frameWidthTeapotBG, frameHeightTeapotBG);
c.drawBitmap(framesBitmapTeapotBG, src, dst, null);
mVectorTeapotBG.add(bmFrame);
}
The computation time falls incredibly! :)
Use a LevelListDrawable. Cut the sprites into individual frames and drop them in your drawable resource directory. Either programmatically or through an xml based level-list drawable create your drawable. Then use ImageView.setImageLevel() to pick your frame.
I use a method of slicing based on rows and columns. However your sprite sheet is rather huge. You have to think you are putting that whole sheet into memory. 3808x1632x4 is the size of the image in memory.
Anyway, what I do is I take an image (lets say a 128x128) and then tell it there are 4 columns and 2 rows in the Sprite(bitmap, 4, 2) constructor. Then you can slice and dice based on that. bitmap.getWidth() / 4 etc... pretty simple stuff. However if you want to do some real stuff use OpenGL and use textures.
Oh I also forgot to mention there are some onDraw stuff that needs to happen. Basically you keep an index counter and slice a rectangle from the bitmap and draw that from a source rectangle to a destination rectangle on the canvas.