Develop Reference

How to most efficiently use and apply Android CameraX Image Analysis setTargetRotation

Similar to what is laid out in the tutorial I am initializing CameraX's Image Analysis use case with code:
ImageAnalysis imageAnalysis =
new ImageAnalysis.Builder()
.setTargetResolution(new Size(1280, 720))
.setTargetRotation(Surface.ROTATION_90)
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.build();
imageAnalysis.setAnalyzer(executor, new ImageAnalysis.Analyzer() {
#Override
public void analyze(#NonNull ImageProxy image) {
int rotationDegrees = image.getImageInfo().getRotationDegrees();
// insert your code here.
}
});
cameraProvider.bindToLifecycle((LifecycleOwner) this, cameraSelector, imageAnalysis, preview);
I am trying to use setTargetRotation method but I am not clear as to how I am supposed to apply this rotation to the output image as vaguely described in the docs:
The rotation value of ImageInfo will be the rotation, which if applied to the output image, will make the image match target rotation specified here.
If I set a breakpoint in the analyze() method shown above, the image object does not get rotated when changing the setTargetRotation value, so I assume the docs are telling me to grab the orientation with getTargerRotation() in the sense that these two pieces of code (builder vs analyzer) are coded separately and this information can be passed between the two without actually applying any rotation. Did I understand this correctly? This really doesn't make sense to me as the setTargetResolution method actually changes the size sent via the ImageProxy. I'd think setTargetRotation should also apply said rotation, but it appears not.
If my understanding is correct, is there an optimal efficient way to rotate these ImageProxy objects after entering the analyze method? Right now I'm doing it after converting to Bitmap via
Bitmap myImg = BitmapFactory.decodeResource(someInutStream);
Matrix matrix = new Matrix();
matrix.postRotate(30);
Bitmap rotated = Bitmap.createBitmap(myImg, 0, 0, myImg.getWidth(), myImg.getHeight(),
matrix, true);
Above idea came from here, but I'd think this is not the most efficient way to do this. I'm sure I could also come up with a way to transpose the arrays, but this could get tedious and messy quickly. Isn't there any way to setup the ImageAnalysis Builder to send the rotated ImageProxy directly rather than having to make a bitmap of everything?

The rotation value of ImageInfo will be the rotation, which if applied
to the output image, will make the image match target rotation
specified here.
An example to understand the definition would be to assume the target rotation matches the device's orientation. Applying the returned rotation to the output image will result in the image being upright, i.e matching the device's orientation. So if the device is in its natural portrait position, the rotated output image will also be in that orientation.
Output image + Rotated by rotation value --> Upright output image
CameraX's documentation includes a section about rotations, since it can be a confusing topic. You can check it out here.
Going back to your question about setTargetRotation and the ImageAnalysis use case, it isn't meant to rotate the images passed to the Analyzer, but it should affect the rotation information of the images, i.e ImageProxy.getImageInfo().getRotationDegrees(). Transforming images (rotating, cropping, etc) can be an expensive operation, so CameraX does not perform any modification to the analysis frames, but it provides the required metadata to make sense of the output images, metadata that can then be used with image processors that analyze the frames.
If you need to rotate each analysis frame, using the Bitmap approach is one way, but it can be costly. Another more performant way may be to do it in native code.

Best usage of CameraX for MLKit Text Recognition on Android

I need to implement text recognition using MLKit on Android and I have decided to use the new CameraX api as the camera lib. I am struggling with the correct "pipeline" of classes or data flow of the image because CameraX is quite new and not many resources is out there. The use case is that I take the picture, crop it in the middle by some bounds that are visible in the UI and then pass this cropped image to the MLKit that will process the image.
Given that, is there some place for ImageAnalysis.Analyzer
api? From my understanding this analyzer is used only for previews and not the captured image.
My first idea was to use takePicture method that accepts OnImageCapturedCallback but when I've tried access eg. ImageProxy.height the app crashed with an exception java.lang.IllegalStateException: Image is already closed and I could not find any fix for that.
Then I've decided to use another overload of takePicture method and now I save image to the file, then read it to Bitmap, crop this image and now I have an image that can be passed to MLKit. But when I take a look at FirebaseVisionImage that is passed to FirebaseVisionTextRecognizer it has a factory method to which I can pass the Image that I get from OnImageCapturedCallback which seems that I am doing some unnecessary steps.
So my questions are:
Is there some class (CaptureProcessor?) that will take care of the cropping of taken image? I suppose that then I could use OnImageCapturedCallback where I would receive already cropped image.
Should I even use ImageAnalysis.Analyzer if I am not doing realtime processing and I am doing post processing?
I suppose that I can achieve what I want with my current approach but I am feeling that I could use much more of CameraX than I currently am.
Thanks!

Is there some class (CaptureProcessor?) that will take care of the
cropping of taken image?
You can set the crop aspect ratio after you build the ImageCapture use case by using the setCropAspectRatio(Rational) method. This method crops from the center of the rotated output image. So basically what you'd get back after calling takePicture() is what I think you're asking for.
Should I even use ImageAnalysis.Analyzer if I am not doing realtime
processing and I am doing post processing?
No, it wouldn't make sense in your scenario. As you mentioned, only when doing real-time image processing would you want to use ImageAnalysis.Analyzer.
ps: I'd be interested in seeing the code you use for takePicture() that caused the IllegalStateException.
[Edit]
Taking a look at your code
imageCapture?.takePicture(executor, object : ImageCapture.OnImageCapturedCallback() {
override fun onCaptureSuccess(image: ImageProxy) {
// 1
super.onCaptureSuccess(image)
// 2
Log.d("MainActivity", "Image captured: ${image.width}x${image.height}")
}
})
At (1), if you take a look at super.onCaptureSuccess(imageProxy)'s implementation, it actually closes the imageProxy passed to the method. Accessing the image's width and height in (2) throws an exception, which is normal -since the image has been closed-. The documentation states:
The application is responsible for calling ImageProxy.close() to close
the image.
So when using this callback, you should probably not call super..., just use the imageProxy, then before returning from the method, manually close it (ImageProxy.close()).

Why doesn't bitmap data from TextureView.getBitmap update in an ImageView?

I have a piece of code that displays camera preview in a TextureView. Once in a while (let's say the user presses a button) I obtain a bitmap from the TextureView and set it for display in an ImageView, like this:
private void freezeFrame() {
textureView.getBitmap(freezeBitmap);
imageView.setImageBitmap(freezeBitmap);
}
The freezeBitmap is pre-allocated and reused on multiple calls to the above method.
The problem is that the ImageView keeps showing the state of the bitmap from the first call to freezeFrame(). Calling imageView.invalidate() doesn't help.
The problem disappears when I do one of the following:
set imageView to use a software layer (this leads me to believe the bitmap is correctly obtained from the textureView but not uploaded to the GPU in hardware mode),
create a new bitmap on each call to getBitmap(),
modify the bitmap, e.g. with freezeBitmap.setPixel(0, 0, 0).
So I wonder: why doesn't ImageView "notice" that the bitmap has been updated by TextureView.getBitmap yet it does notice the change performed by setPixel()?
Is there a good way to force the update?

Get detected face bitmap

I'm experimenting with the following Google sample: https://github.com/googlesamples/android-vision/tree/master/visionSamples/FaceTracker
The sample is using the Play Service new Face detection APIs, and draws a square on detected faces on the camera video stream.
I'm trying to figure out if it is possible to save the frames that has detected faces in them, from following the code it seems that the face detector's processor is a good place to perform the 'saving' but it only supplies the detection meta data and not the actual frame.
Your guidance will be appreciated.

You can get it in the following way:
Bitmap source = ((BitmapDrawable) yourImageView.getDrawable()).getBitmap();
// detect faces
Bitmap faceBitmap = createBitmap(source,
face.getPosition().x,
face.getPosition().y,
face.getWidth(),
face.getHeight());

Yes it is possible. I answered to question about getting frames from CameraSource here. Most trickiest parts are to access CameraSource frames and to convert Frame datatype to Bitmap. Then having frames as Bitmaps you can pass them to you FaceGraphic class and in method draw() save those Bitmaps, because draw() is called only when faces are detected.

createBitmap() leads me into a java.lang.OutOfMemoryError

i try to rotate 3 imageViews (or better the Bitmaps behind them) every 10-100ms.
i do the rotation like this:
ImageView ivLoad;
Bitmap bMapLoad;
....
Matrix mat=new Matrix();
mat.reset();
mat.postScale(1.55f, 1.55f);
mat.postRotate((float)currentLoadDegree+(float)LoadDegree);
bMapLoad = Bitmap.createBitmap(bMapLoadgr, 0, 0, bMapLoadgr.getWidth(), bMapLoadgr.getHeight(), mat, true);
ivLoad.setImageBitmap(bMapLoad);
ivLoad.setScaleType(ScaleType.CENTER);
....
the first time i start the app everthing works fine.
second time also works
but the 3rd time i start the app it crashs with the following error:
03-27 10:01:09.234: E/AndroidRuntime(3603): java.lang.OutOfMemoryError
03-27 10:01:09.234: E/AndroidRuntime(3603): at android.graphics.Bitmap.nativeCreate(Native Method)
03-27 10:01:09.234: E/AndroidRuntime(3603): at android.graphics.Bitmap.createBitmap(Bitmap.java:605)
03-27 10:01:09.234: E/AndroidRuntime(3603): at android.graphics.Bitmap.createBitmap(Bitmap.java:551)
after trying around a long time i found out that when i call System.exit(0) in the onDestroy methode everthing works.
now i don't know if there would be a better way because on google a lot of peaople mean that System.exit(0) is unsafe.
so will i get problems with this?

Instead of rotating the Bitmap, you could rotate the canvas you are drawing on.
canvas.save();
canvas.translate(-canvasWidth/2, -canvasHeight/2);
canvas.rotate(degrees)
canvas.drawBitmap( ... )
canvas.translate(-canvasWidth/2, -canvasHeight/2);
canvas.restore();
Now you only get a new bitmap, when the image itself is updated, even though you can rotate it as frequent as you like. But if you get a new Bitmap, you still need to call Bitmap.recycle() on the old one.

You shouldn't recreate the bitmap on every step of the rotation, instead you should just try to draw it rotated. That's also possible with a Matrix (what you already use) and will avoid the excessive memory usage.
Android: How to rotate a bitmap on a center point

You get OutOfMemoryError because you load the Bitmap every time you rotate ImageView. You should consider reusing already loaded bitmap. Also call Bitmap.recycle() method when you do not need the bitmap any more.