UPDATE : question is about Camera2
I'm trying to figure out who applies the rotation transform when a camera preview is drawn on a SurfaceTexture.
When requesting the preview sizes from the camera you always get pairs where the width is larger than the height (because landscape is the usual orientation when taking a picture).
When using the device on portrait mode and setting the preview size (for ex. 1600 x1200) the frames from the camera are correctly rotated but I can't find out where it's done, is it something CameraDevice does automatically based on Surface type ? or is it the SurfaceTexture who rotates the image?
The answer depends a bit on which camera API you're using; since you mention CameraDevice, I assume you mean camera2.
In camera2, one of the goals was to simplify the rotation handling where possible; so if you draw preview into a SurfaceView, the camera service and the hardware compositor / GPU cooperate and handle all the rotation for you. You always request landscape resolutions from the sensor (which is aligned with the long edge of the sensor matching the long edge of the device), and you just need to make sure your SurfaceView's aspect ratio matches that of the requested resolution (so in portrait, you need 9:16 View to display 1080p correctly; in landscape you need a 16:9 View).
For TextureView, and completely custom GL drawing with SurfaceTexture, the API can't do it all for you. The SurfaceTexture's transform is set by the camera API to rotate the camera output to the device's native orientation (generally portrait for phones, landscape for some tablets), but it can't handle the additional rotation if your app's UI is not in the native orientation.
In that case, you'll need to get the transform matrix from the SurfaceTexure, apply the extra rotation to match your app UI, and set that as the TextureView matrix.
It's been about a year since I worked on Camera stuff, so I apologize for a lack of specifics. But I do remember that the key when displaying the Camera preview was calculating the correct transform matrix to apply to the SurfaceView (or TextureView) to apply any scaling or rotation that is needed. So regarding "who is doing the rotation" I guess you can say it is the view, as instructed by the Transform that you supply. So the matrix will be based on the relationship of the preview resolution you request (from the list of what's capable for your device) compared to the actual laid out dimensions of your view, plus the physical orientation of the device sensor as compared to the current orientation of the device screen.
In the old camera API, Camera.CameraInfo has an orientation field that supplies the physical orientation with which the sensor is mounted. It's typically 90 for a landscape mounted sensor on most phones, but you'll also find some 270s out there (effectively "upside down"). 0 and 180 may exist but I've never seen them.
Related
I have a proprietary device with a rotated camera, that doesn't report its rotation to the OS.
We use CameraX with PreviewView, and I get the image 90 deg rotated, and mirrored.
Is there an efficient way to rotate the preview?
We tried:
Setting the setTargetOrientation of camera, of analyzer, of Preview. None works.
Rotating the PreviewView of course doesn't work, it just modifies the ViewPort because it won't rotate the surface.
ViewFinder library - outputs bitmaps but doesn't connect to Preview (or does anybody have a solution for that?). Without setSurfaceProvider, nothing will empty the frames pipeline, and it will display nothing.
Ugly "Solution":
We currently use bitmap (via Analyzer), then we display the bitmap. That's ugly, and CPU expensive.
Is Camera2 the only way to rotate previews nicely? Camera2 does bitmaps anyway, but it may do the YUV conversion on hardware.
NOTE
Solutions like here don't work. Setting target orientation of the Preview only modifies the scale to compensate for aspect ratio. It doesn't actually rotate. Also that doesn't work because it's obsolete and not valid anymore.
The Preview#setTargetRotation API should work: https://developer.android.com/reference/androidx/camera/core/Preview#setTargetRotation(int)
For this API to work, your PreviewView has to be in the COMPATIBLE mode:
PreviewView#setImplementationMode(ImplementationMode.COMPATIBLE);
You can find a complete sample on GitHub: https://github.com/androidx/androidx/blob/androidx-main/camera/integration-tests/viewtestapp/src/main/java/androidx/camera/integration/view/PreviewViewFragment.java
I uploaded a screen record of the preview rotating. Please take look and let me know if this is not your desired result: https://github.com/xizhang/public-files/blob/main/stackoverflow74798791/preview_rotation.mp4
You can also download the APK and test it yourself: https://github.com/xizhang/public-files/blob/main/stackoverflow74798791/camera-testapp-view-debug.apk
The precondition is we have to let Activity rotate follow the system orientation, so i add following code to the AndroidManifest.xml：
android:configChanges="keyboard|keyboardHidden|navigation|orientation|screenLayout|screenSize|uiMode"
android:screenOrientation="sensor"
And set the rotationAnimation to ROTATION_ANIMATION_SEAMLESS.
WindowManager.LayoutParams windowAttributes = getWindow().getAttributes(); windowAttributes.rotationAnimation = WindowManager.LayoutParams.ROTATION_ANIMATION_SEAMLESS; getWindow().setAttributes(windowAttributes);
I reset the correct glViewport, projection matrix and model matrix in surfaceChanged()，the proplem is that serveral frames looks weired while change the device orientation, but after rotation is complete, all frames look good.
This image shows what's going on:
It tried to reset viewport and matrix while onConfigurationChanged() is invoked, but it doesn't work.
I want the frames look seamless while change the device orientation, just as the Pixels' camera APP did.
It's simple: it's not possibile to avoid this completly.
You could let the SurfaceView invisible while rotation is not completed, but the Surface cannot renders not-yet-existent pixels for fill the new space that is created when the "landscape mode" is just started: Surface/View should adapts itself and it takes few milliseconds which are few frames as you reported (more the Device is slow/cheap and more frames are affected by this behavious/issue) expecially for the Camera to get new pixels from a different orientation.
I want to horizontally invert frames coming to Surface object made from TextureView. What I can do is to set transformation matrix to this TextureView instance, whereas I postScale by -1 for x and 1 for y (leave unchanged), and than postTranslate dx on the full width of the view and leave dy unchanged (0F).
But the problem appears when I rotate my device by 90 degrees (horizontally) with screen rotation off: the image is rotated by 180 degrees, and it, of course, makes perfect sense, because x and y axises did not change.
How it can be solved? Is it possible to play with the translation matrix in such a way to resolve this problem? Or may be with OpenGL ES tools?
P.S.: strangely, but rotation is done two times two times more comparing to the device rotation itself, e.g. when I rotate the device by 90 degrees - preview is rotated by 180.
P.S.S.: I tried to invert the preview using Matrix's setPolyToPoly method... and have got exactly the same result.
P.S.S.S.: Also, played with open gl to achieve the goal using simple scale and rotation transformations for the model and projection matricies, and have got exact the same result!
Update:
These are screenshots that describe default behavior of the front camera - frame inversion is applied by HAL by default and I can't read the text; still, whether I rotate the device or not - "frame orientation" does not change:
And these are screenshots when I apply, e.g., Matrix.scaleM(modelMatrix, 0, -1F, 1F, 1F); and then apply this matrix to every coordinate that comes into vertex shader, so I can now read the text because I applied the inversion myself so HAL's inversion with my custom inversion will result into inversion absence, but when I rotate my device (with device orientation change on rotation disabled, of course, and that's the point) - I'll see myself flipped upside down, and that's, of course, make perfect sense, because device's coordinate system won't change. Still, I want to be able to avoid image rotation somehow on device rotation itself (like in the default mode - whether the device is rotated or not - the preview image just "does not care"), and to still be able to read the text (I mean, like in portrait mode).
Do you run your tests on front or back camera? Usually only the front camera frame needs to be flipped horizontally, not the back camera.
Anyway if you need different transformations based on device orientation, you need to detect orientation changes in your Activity. Camera doesn't do anything by itself when device is rotated, you will get the same results whatever transformation method you choose.
Update #2
I just realised that this problem can be solved if transformation is applied at the right moment in the pipeline. I will show a working example, it uses OpenGL extensively but I hope you can find something of use. See this code fragment, it uses a vertex shader to transform the coordinates. In drawFrame() there's a call to getTransformMatrix(). Instead of performing this call, get a matrix from this method I wrote when performing a similar task.
Although Camera API is deprecated, it's simpler to get started than Camera2. But the document seems not to be that easy to understand(with my reading ability). I've searched a lot trying to figure it out. But there're still bunch of questions.
1.What is camera's orientation? Is there any definition that I miss? There're 4 dimensions(left, top, right, bottom) of camera. When we say orientation, which one is used and what is the relative object?
2.Why it's different between camera's orientation and device's?
3.Could you please tell me how to understand the algorithm given in the document of Camera.setDisplayOrientation() and Camera.parameters.setRotation()? There're sample code given by document for these two methods. These two algorithm both use sum and difference of orientation/rotation. I may use it directly in my project. But what does that mean exactly?
Besides, taking back-facing camera as example, the former uses (info.orientation - degrees + 360) % 360, while the latter uses (info.orientation + orientation) % 360. Confusing:/
Thanks if you can answer any of these questions above or just share some of your thoughts and experiences.
1. it is the how the camera is orientated in your device
2. Device orientation is the current orientation/rotation of your device (landscape/portrait )
3. Camera.setDisplayOrientation()
affects the preview
Camera.parameters.setRotation()
affects the result pictures (jpeg etc.)
Documentation:
Camera.setDisplayOrientation()
Set the clockwise rotation of preview display in degrees. This affects
the preview frames and the picture displayed after snapshot. This
method is useful for portrait mode applications. Note that preview
display of front-facing cameras is flipped horizontally before the
rotation, that is, the image is reflected along the central vertical
axis of the camera sensor. So the users can see themselves as looking
into a mirror.
Camera.parameters.setRotation()
Sets the clockwise rotation angle in degrees relative to the orientation of the camera. This affects the pictures returned from JPEG Camera.PictureCallback. The camera driver may set orientation in the EXIF header without rotating the picture. Or the driver may rotate the picture and the EXIF thumbnail. If the Jpeg picture is rotated, the orientation in the EXIF header will be missing or 1 (row #0 is top and column #0 is left side).
Why in the portrait screen orientation the camera surface is rotated by -90 degrees "90 degrees anticlockwise". I want the camera preview to be normal regardless of the screen orientation. How can I achieve that.
Did you tried this:
mCam.setDisplayOrientation(90); use it inside surfaceChanged(...)
setDisplayOrientation
Set the clockwise rotation of preview display in degrees. This affects the preview frames and the picture displayed after snapshot. This method is useful for portrait mode applications. Note that preview display of front-facing cameras is flipped horizontally before the rotation, that is, the image is reflected along the central vertical axis of the camera sensor. So the users can see themselves as looking into a mirror.
This does not affect the order of byte array passed in onPreviewFrame(byte[], Camera), JPEG pictures, or recorded videos. This method is not allowed to be called during preview.