Develop Reference

Extract a person from an image in Google ML Kit with Selfie Segmentation in Kotlin

I would like to remove a background from an image with a person in order to use this image in some other part of my android app. I have applied Google ML Kit - Selfie Segmentation and receive segmentation Mask as a result. What I want to do now is - save the image without a background to a device so I can then use it in other parts of my app.
Original image:
Image after Applied Selfie Segmentation:
My problem is I need to somehow apply values from segmentationMask (256 * 256) to the Bitmap where I will remove background and save new Bitmap (with values from segmentation mask on only those pixels that are currently blue) with pixels from segmentation mask. Would anyone be so king and point me in the direction? Should I try to achieve this with Matrix ?? Currently I am drawind this image on jetpack compose Canvas with drawPoints comand in DrawScope.

There is an enableRawSizeMask API on the SelfieSegmenterOptions.Builder. If you don't set it, the returned mask should have the same dimension as the input image.
Then a naive approach would be: Iterating through the result mask to determine whether a certain pixel should be kept, if not, call setPixel method on the original Bitmap to set the background pixel to transparent (or whatever color you want).
This is for sure not the most performant way, but maybe a good start.

Maybe you can try it,this is my way
private void initUI(){
ImageView imageView=findViewById(R.id.iv);
Paint paint=new Paint();
paint.setAntiAlias(true);
SelfieSegmenterOptions options=new SelfieSegmenterOptions.Builder()
.setDetectorMode(SelfieSegmenterOptions.STREAM_MODE)
//.enableRawSizeMask()
.build();
Segmenter segmenter = Segmentation.getClient(options);
Bitmap bitmap= BitmapFactory.decodeResource(getResources(), R.drawable.mv1);
InputImage image = InputImage.fromBitmap(bitmap, 0);
segmenter.process(image).addOnSuccessListener(segmentationMask -> {
ByteBuffer mask = segmentationMask.getBuffer();
int maskWidth = segmentationMask.getWidth();
int maskHeight = segmentationMask.getHeight();
Bitmap background=Bitmap.createBitmap(maskColorsFromByteBuffer(mask,maskWidth,maskHeight),maskWidth,maskHeight, Bitmap.Config.ARGB_8888);
//创建一个bitmap
Bitmap foreground=Bitmap.createBitmap(bitmap.getWidth(),bitmap.getHeight(), Bitmap.Config.ARGB_8888); //裁剪后的图像
Canvas canvas=new Canvas(foreground);
//绘制扣出来的背景
canvas.drawBitmap(background,0,0,paint);
//设置混合模式
paint.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.SRC_OUT));
//绘制原图
canvas.drawBitmap(bitmap,0,0,paint);
imageView.setImageBitmap(foreground);
}).addOnFailureListener(e -> {
// Task failed with an exception
// ...
});
}
private int[] maskColorsFromByteBuffer(ByteBuffer byteBuffer,int maskWidth,int maskHeight) {
#ColorInt int[] colors = new int[maskWidth * maskHeight];
for (int i = 0; i < maskWidth * maskHeight; i++) {
float backgroundLikelihood = 1 - byteBuffer.getFloat();
if (backgroundLikelihood > 0.9) { //128
colors[i] = Color.argb(255, 255, 0, 255);
//colors[i] = Color.argb(0, 255, 0, 255);
} else if (backgroundLikelihood > 0.2) {
// Linear interpolation to make sure when backgroundLikelihood is 0.2, the alpha is 0 and
// when backgroundLikelihood is 0.9, the alpha is 128.
// +0.5 to round the float value to the nearest int.
int alpha = (int) (182.9 * backgroundLikelihood - 36.6 + 0.5);
colors[i] = Color.argb(255, 255, 0, 255);
}
}
return colors;
}

The transparent pixels in my Bitmap shown as white

I'm using c code to get frame from a gif file and it's working fine using ffmpeg library av_read_frame, then I convert the returned image from this format BGRA to ARGB format using this method libyuv::ABGRToARGB.
In the java part I receive the bitmap and when I put it in the ImageView the transparent pixels drawn white.
Bitmap bitmap= Bitmap.createBitmap(150, 150, Bitmap.Config.ARGB_8888);
getGifFrame(gifFile,bitmap); //native method which get image frame from gif file.
imageTest.setImageBitmap(backgroundBitmap);//this bitmap in debug mode I can see that it has transparent pixels, but in drawn it appears white!
even when I loop on the returned bitmap pixels and check for each pixel I find them transparent!
for (int x = 0; x < b.getWidth(); x++)
{
for (int y = 0; y < b.getHeight(); y++)
{
int color = b.getPixel(x, y);
if (color == Color.WHITE)//This condition never occurred
{
b.setPixel(x, y, Color.TRANSPARENT);
}
}
}
Nothing happened and it's still white. Then I did convert all transparent pixels to TRANSPARENT !! and guess what, It's working!
else if (color == Color.TRANSPARENT)
{
b.setPixel(x, y, Color.TRANSPARENT);
}
I don't understand why that happen. Any help would be appreciated.
EDIT 1:
I get all pixels from the bitmap and set them again without doing anything and it worked also!?
bitmap.getPixels(pixels, 0, width, 0, 0, width, height);
bitmap.setPixels(pixels, 0, width, 0, 0, width,height);

Detecting circles of a color gives "image must be 8-bit single-channel in HoughCircle" error in OpenCV

I am building an Android Application and I want to be able to detect black circles. I am using OpenCV3 for Android and I am able to filter out the black colours from my camera feed using the code below. Kindly note that I used the Color-blob-detection example given in the Android SDK and tweaked the code as such:
public Mat onCameraFrame(CameraBridgeViewBase.CvCameraViewFrame inputFrame) {
mRgba = inputFrame.rgba();
if (mIsColorSelected) {
Bitmap resultBitmap;
resultBitmap = Bitmap.createBitmap(mRgba.cols(), mRgba.rows(),Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mRgba, resultBitmap);
//TODO - look for circles
Mat mat = new Mat(resultBitmap.getWidth(), resultBitmap.getHeight(), CvType.CV_8UC1);
Utils.bitmapToMat(resultBitmap, mat);
final Bitmap bitmap;
//TODO - filter out the black only
Mat mHSV = mat;
Mat mHSVThreshed = mat;
Mat mRgba2 = mat;
Imgproc.cvtColor(mat, mHSV, Imgproc.COLOR_BGR2HSV, 0);
Core.inRange(mHSV, new Scalar(0, 0, 0), new Scalar(130, 130, 130), mHSVThreshed);
Imgproc.cvtColor(mHSVThreshed, mat, Imgproc.COLOR_GRAY2BGR, 0);
Imgproc.cvtColor(mat, mRgba2, Imgproc.COLOR_BGR2RGBA, 0);
Imgproc.GaussianBlur(mRgba2, mRgba2, new Size(9, 9), 2, 2);
//this is for displaying purposes only.
//At this point, the image would be black and white, where the white spots are the black detected blobs
// Bitmap bmp = Bitmap.createBitmap(mRgba2.cols(), mRgba2.rows(), Bitmap.Config.ARGB_8888);
//Utils.matToBitmap(mRgba2, bmp);
//bitmap = bmp; //resultBitmap;
//TODO - new circle detection code: this uses the colour filtered Mat
mat = mRgba2;
Imgproc.HoughCircles(mat, circles,
Imgproc.CV_HOUGH_GRADIENT, 1, minDist, 100,
20, 0, 0);
/* get the number of circles detected */
int numberOfCircles = (circles.rows() == 0) ? 0 : circles.cols();
/* draw the circles found on the image */
for (int i = 0; i < numberOfCircles; i++) {
/* get the circle details, circleCoordinates[0, 1, 2] = (x,y,r)
* (x,y) are the coordinates of the circle's center
*/
double[] circleCoordinates = circles.get(0, i);
int x = (int) circleCoordinates[0], y = (int) circleCoordinates[1];
Point center = new Point(x, y);
int radius = (int) circleCoordinates[2];
/* circle's outline */
Imgproc.circle(mRgba2, center, radius, new Scalar(0,
200, 255), 4);
/* circle's center outline */
Imgproc.rectangle(mRgba2, new Point(x - 5, y - 5),
new Point(x + 5, y + 5),
new Scalar(0, 200, 255), -1);
}
Utils.matToBitmap(mRgba2, resultBitmap);
bitmap = resultBitmap;
runOnUiThread(new Runnable() {
#Override
public void run() {
mOpenCvCameraView.disableView();
mOpenCvCameraView.setVisibility(SurfaceView.GONE);
imageView.setVisibility(View.VISIBLE);
imageView.setImageBitmap(bitmap);
}
});
}
return mRgba;
}
What my code does is that it takes a snapshot of the camera feed as a Mat, and then using that Mat, I do some image processing on it and to black out everything except the black colour. The resulting Mat is the mRgba2 variable and I converted to Bitmap and displayed on an ImageView. I displayed this Bitmap to confirm that I am getting the result I want.
After I know that I'm able to filter out the colour I wanted, I then run a GaussianBlur on it then proceed to run HoughCircles. However, when I run the
Imgproc.HoughCircles(mat, circles,
Imgproc.CV_HOUGH_GRADIENT, 1, minDist, 100,
20, 0, 0);
line, I get an
The source image must be 8-bit, single-channel in function CvSeq* cvHoughCircles(CvArr*, void*, int, double, double, double, double, int, int)
Error.
I know that the code wherein I run HoughCircles on a given mat variable works, because I tested it before. Now, changing the mat variable I feed onto it does not, and I wonder what I did differently for the code to not work.

HoughCircles runs only on grayscale (CV8U) images.
Replacing
Imgproc.cvtColor(mat, mRgba2, Imgproc.COLOR_BGR2RGBA, 0);
with
Imgproc.cvtColor(mat, mRgba2, Imgproc.COLOR_BGR2GRAY, 0);
should remove the error.
Then the problem is to detect circles of a given color. This won't pose much of a problem in your case, because you are detecting black circles anyway. Instead of black circles, this will detect dark circles.
If you have dark colors other than black in some of the circles, you can filter them out separately by looking at the pixel values in a different color space.

Android 4.3 PBO not working

I am using PBO to take screenshot. However, the result image is all black. It works perfectly fine without PBO. Is there any thing that I need to take care before doing this ?
I even tried by rendering to a FBO and then use GLES30.glReadBuffer(GLES30.GL_COLOR_ATTACHMENT0), no hope
public void SetupPBO(){
GLES30.glGenBuffers(1, pbuffers, 0);
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, pbuffers[0]);
int size = (int)this.mScreenHeight * (int)this.mScreenWidth * 4;
GLES30.glBufferData(GLES30.GL_PIXEL_PACK_BUFFER, size, null, GLES30.GL_DYNAMIC_READ);
checkGlError("glReadBuffer");
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, 0);
}
private void Render(float[] m) {
.......//Normal render logic
exportBitmap();
}
private void exportBitmap() {
int screenshotSize = (int)this.mScreenWidth * (int)this.mScreenHeight;
ByteBuffer bb = ByteBuffer.allocateDirect(screenshotSize * 4);
bb.order(ByteOrder.nativeOrder());
// set the target framebuffer to read
GLES30.glReadBuffer(GLES30.GL_FRONT);
checkGlError("glReadBuffer");
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, pbuffers[0]);
GLES30.glReadPixels(0, 0, (int)mScreenWidth, (int)mScreenHeight, GL10.GL_RGBA, GL10.GL_UNSIGNED_BYTE, bb); //<------ not working ?????
int pixelsBuffer[] = new int[screenshotSize];
bb.asIntBuffer().get(pixelsBuffer);
bb = null;
for (int i = 0; i < screenshotSize; ++i) {
// The alpha and green channels' positions are preserved while the
// red and blue are swapped
pixelsBuffer[i] = ((pixelsBuffer[i] & 0xff00ff00))
| ((pixelsBuffer[i] & 0x000000ff) << 16)
| ((pixelsBuffer[i] & 0x00ff0000) >> 16);
}
Bitmap bitmap = Bitmap.createBitmap((int)mScreenWidth, (int)mScreenHeight, Bitmap.Config.ARGB_8888);
bitmap.setPixels(pixelsBuffer, screenshotSize - (int)mScreenWidth, -(int)mScreenWidth, 0, 0, (int)mScreenWidth, (int)mScreenHeight);
SaveBitmap(bitmap);
}

GLES30.glReadPixels(0, 0, (int)mScreenWidth, (int)mScreenHeight, GL10.GL_RGBA, GL10.GL_UNSIGNED_BYTE, bb);
bb is interpret as an offset in your PBO. Thus you're writing out of buffer (On some drivers this code cause crash). You should pass 0 instead of bb. To retrive the data from PBO use glMapBuffer.

Create nine-patch from bitamp - and getting new adjust bitmap

I'm tring to download an image from the net, and to convert it to ninepatch format, how I select the stretchable pixels is out of scope for this thread.
I'm trying to use gist solution but I couldn't get it to work.
This is the source code:
NinePatchDrawable np = createNinePathWithCapInsets(res, bitmap, top,left, bitmap.getWidth() - right, bitmap.getHeight() - bottom, null);
and to get the resulted bitmap:
Bitmap outputBitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
Canvas canvas = new Canvas(outputBitmap);
//np.setBounds(0, 0, width, height);
np.draw(canvas);
canvas.drawBitmap(outputBitmap, width, height, new Paint());
The returned bitmap doesn't contain the source bitmap and looks just like 4 segmentations(rect seg').
I could find any other to achive that goal, even not using OpenGL
The resulted image is:

try this:
public static byte[] getChunk(int xs, int xe, int ys, int ye) {
ByteBuffer b = ByteBuffer.allocate(84).order(ByteOrder.nativeOrder());
b.put((byte) 1); // wasDeserialized
b.put((byte) 2); // numXDivs
b.put((byte) 2); // numYDivs
b.put((byte) 9); // numColors
b.putInt(0); // UNKNOWN
b.putInt(0); // UNKNOWN
b.putInt(0); // paddingLeft
b.putInt(0); // paddingRight
b.putInt(0); // paddingTop
b.putInt(0); // paddingBottom
b.putInt(0); // UNKNOWN
b.putInt(xs); // segX start
b.putInt(xe); // segX end
b.putInt(ys); // segY start
b.putInt(ye); // segY end
for (int i = 0; i < 9; i++) {
b.putInt(1); // NO_COLOR
}
return b.array();
}
you can use it in NinePatchDrawable constructor (Bitmap, byte[], Rect, String)