I'm upgrading my Android OpenGL app to run on my new 4K phone and am intending to have a fullscreen activity whose dimensions are both logically and physically 4K. However, without special-casing for 4K, I'm running into the evidently expected scenario that my activity is only 1080p in size and I wish to bump it up to 4K.
I have followed the available documentation involving querying the display modes, which thankfully does enumerate both 1080p and 4K as available display modes. The pseudo code getBestDisplayModeId() below represents this step, which I consider to be functioning accurately. This function can be assumed to correctly return the display mode id of the 4K display mode. I then set the preferredDisplayModeId in the LayoutParams for the window to that optimal display mode id and use setAttributes() to hopefully apply it to the window.
This is executed during the onCreate() for my fullscreen Activity. Note that the activity was created via the fullscreen activity wizard in Android Studio and it essentially contains just a GLSurfaceView and associated Renderer (the typical OpenGL setup).
My (GL) Renderer class was temporarily modified (for debugging purposes) to display a Toast (via IPC), showing its dimensions whenever its onSurfaceChanged() overridden method is called. The Toast is always reporting that the dimensions are 1080p, rather than the intended 4K.
How can I correctly apply the preferredDisplayModeId and then get an Activity/View/GLSurfaceView that is truly the intended 4K size?
int getBestDisplayModeId()
{
// 1. query available display modes for display 0 via DisplayManager
// 2. return display mode corresponding to the highest resolution
}
FullscreenActivity.onCreate()
{
Window w = getWindow();
WindowManager.LayoutParams p = w.getAttributes();
p.preferredDisplayModeId = getBestDisplayModeId(); // get 4K id
w.setAttributes(p); // set 4K resolution
// when does the switch to 4K occur?
mGLView = new MyGLSurfaceView();
setContentView(mGLView);
}
MyRenderer.onSurfaceChanged(int w, int h)
{
// only shows 1080p resolution, never 4K
}
Related
*Note: I figured out why the image was too dark at max SENSOR_SENSITIVITY: it was just a matter of EXPOSURE_TIME being too short, had to pump it up to 8 digits. Now all I have to do is save the values for SENSOR_SENSITIVITY and EXPOSURE_TIME. So, this question is answered, unless there's a better way?
Using Camera X (Camera2Interop) I want to turn off auto-exposure and white balance. Because I want to manually adjust the brightness, and ideally have it locked at that value until I adjust it again.
I'm not capturing an image but just working with the preview, and image analysis.
When turning CONTROL_MODE off or AE off the preview & analysis are black (unless pointed directly at light bulb), and I can get an image by adjusting SENSOR_SENSITIVITY, but the image is still too dark at maximum and has noise in it. When I modify EXPOSURE_TIME the image goes 100% black no matter what.*
Using AE_LOCK + AE_EXPOSURE_COMPENSATION instead fixes the issue of the black preview and prevents auto-correction, but isn't consistent, for I don't have a way to reset the brightness to what was set previously. For example: if I open the main camera app and go back to mine the brightness is now locked at a different value.
Camera2Interop.Extender extender = new Camera2Interop.Extender(builder);
//extender.setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CameraMetadata.CONTROL_AF_MODE_OFF);
//extender.setCaptureRequestOption(CaptureRequest.CONTROL_AE_EXPOSURE_COMPENSATION,-1);
extender.setCaptureRequestOption(CaptureRequest.CONTROL_AF_MODE, CameraMetadata.CONTROL_AF_MODE_MACRO);
//extender.setCaptureRequestOption(CaptureRequest.SENSOR_EXPOSURE_TIME,10);
//extender.setCaptureRequestOption(CaptureRequest.CONTROL_MODE, CaptureRequest.CONTROL_MODE_OFF);
//extender.setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF);
extender.setCaptureRequestOption(CaptureRequest.CONTROL_AWB_MODE, CaptureRequest.CONTROL_AWB_MODE_OFF);
//extender.setCaptureRequestOption(CaptureRequest.SENSOR_SENSITIVITY,99999);
extender.setCaptureRequestOption(CaptureRequest.CONTROL_AWB_LOCK,true);
extender.setCaptureRequestOption(CaptureRequest.CONTROL_AE_LOCK,true);
//extender.setCaptureRequestOption(CaptureRequest.BLACK_LEVEL_LOCK,true);
Another issue is When adjusting brightness via EXPOSURE_COMPENSATION it seems to compound. For example, if it's set to -3 the image gets a little darker with every start of the application, or probably every time startCameraX is run. Perhaps this is an easy fix by moving it out of startcamera, but I was attempting to start or reset to it's previously set value.
ExposureState exposureState = camera.getCameraInfo().getExposureState();
if (!exposureState.isExposureCompensationSupported()) return;
Range<Integer> range = exposureState.getExposureCompensationRange();
int index = exposureState.getExposureCompensationIndex();
if (range.contains(1) && index != -3) {
camera.getCameraControl().setExposureCompensationIndex(-3);
}
Also, all the warning that come along with the camera2 interop extender makes it seem odd that's the official solution.
Other failed attempts: Canceling focus and metering, and set the metering point to 0x0 pixels.
camera.getCameraControl().cancelFocusAndMetering();//.setExposureCompensationIndex(12);
MeteringPointFactory meteringPointFactory = previewView.getMeteringPointFactory();
MeteringPoint meteringPoint = meteringPointFactory.createPoint(0,0, 0);
FocusMeteringAction action = new FocusMeteringAction.Builder(meteringPoint).build();
//.setAutoCancelDuration(1, TimeUnit.MICROSECONDS).build();
cameraControl.startFocusAndMetering(action);
In my application I have a SurfaceView in my Activity layout so its surface is not affected by any Fragment lifecycles. I use the surface to draw to it from JNI using following code (some of the varialbes are global):
void onDraw() {
int height=0, width=0;
ANativeWindow_Buffer nativeBuffer;
ARect redrawRect;
if (nativeWindow != NULL) {
height = ANativeWindow_getHeight(nativeWindow);
width = ANativeWindow_getWidth(nativeWindow);
redrawRect.left = 0;
redrawRect.top = 0;
redrawRect.right = width;
redrawRect.bottom = height;
if (ANativeWindow_lock(nativeWindow, &nativeBuffer, &redrawRect) == 0) {
height = (redrawRect.bottom - redrawRect.top);
width = (redrawRect.right - redrawRect.left);
bufferSize= height * width * 4;
memcpy(nativeBuffer.bits, pbuf, bufferSize);
ANativeWindow_unlockAndPost(nativeWindow);
}
}
}
As long as I do not change the size and position of the SurfaceView this works great. However, when I start drawing, adjust the SurfaceView in size and position (content is adjusted fine), stop the drawing and then start it again I get the following error:
E/BufferQueue: [SurfaceView] connect: already connected (cur=2, req=2)
What I understand from the documentation so far is that my SurfaceView is the consumer and the ANativeWindow the producer. The SurfaceView holds a BufferQueue which send the display data (copied by memcpy) from the producer to the consumer side. So after the size and location change one of them (I'm not sure which) is trying to connect to that BufferQueue again. Since in my code I'm not calling queue/acquire etc. specifically I'm not sure how to prevent this.
What I do not know: Is connect called from ANativeWindow_lock or ANativeWindow_fromSurface on the NDK side? Could the buffer received by the lock method leak somehow to prevent the connect next time? I also added log messages to the lock and unlock results and found that after the size and location change ANativeWindow_lock fails. This seems odd to me because my unlock is only performed when the lock was successful and I'm not locking the surface from the Java side at all. If I try to unlock it then from JNI I get:
Surface::unlockAndPost failed, no locked buffer
I also thought about other memory leaks and analysed it with LeakCanary. This does not find any leaks, even when I do a heap dump manually tho I'm not sure how good it works with native code.
If I do not draw at all this error does not appear. Do you have any ideas for this?
If it matters I'm developing on Android KitKat (4.4.2). Any help is highly appreciated.
In my Camera2 API project for Android, I want to set a region for my Exposure Calculation. Unfortunately it doesn't work. On the other side the Focus region works without any problems.
Device: Samsung S7 / Nexus 5
1.) Initial values for CONTROL_AF_MODE & CONTROL_AE_MODE
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_AUTO);
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_ON);
2.) Create the MeteringRectangle List
meteringFocusRectangleList = new MeteringRectangle[]{new MeteringRectangle(0,0,500,500,1000)};
3.) Check if it is supported by the device and set the CONTROL_AE_REGIONS (same for CONTROL_AF_REGIONS)
if (camera2SupportHandler.cameraCharacteristics.get(CameraCharacteristics.CONTROL_MAX_REGIONS_AE) > 0) {
camera2SupportHandler.mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AE_REGIONS, meteringFocusRectangleList);
}
4.) Tell the camera to start Exposure control
camera2SupportHandler.mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER, CameraMetadata.CONTROL_AE_PRECAPTURE_TRIGGER_START);
The CONTROL_AE_STATE is always in CONTROL_AE_STATE_SEARCHING, but doesn't use the configured regions...
After long testing & development I've found an answer.
The coordinate system - Camera 1 API VS Camera 2 API
RED = CAM1; GREEN = CAM2; As shown in the image below, the blue rect are the coordinates for a possible focus/exposure area for the Cam1. By using the Cam2 API, there must be firstly queried the max of the height and the width. Please find more info here.
Initial values for CONTROL_AF_MODE & CONTROL_AE_MODE: See in the question above.
Set the CONTROL_AE_REGIONS: See in the question above.
Set the CONTROL_AE_PRECAPTURE_TRIGGER.
// This is how to tell the camera to start AE control
CaptureRequest captureRequest = camera2SupportHandler.mPreviewRequestBuilder.build();
camera2SupportHandler.mCaptureSession.setRepeatingRequest(captureRequest, captureCallbackListener, camera2SupportHandler.mBackgroundHandler);
camera2SupportHandler.mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER, CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER_START);
camera2SupportHandler.mCaptureSession.capture(captureRequest, captureCallbackListener, camera2SupportHandler.mBackgroundHandler);
The ''captureCallbackListener'' gives feedback of the AE control (of course also for AF control)
So this configuration works for the most Android phones. Unfortunately it doesn't work for the Samsung S6/7. For this reason I've tested their Camera SDK, which can be found here.
After deep investigations I've found the config field ''SCaptureRequest.METERING_MODE''. By setting this to the value of ''SCaptureRequest.METERING_MODE_MANUAL'', the AE area works also the Samsung phones.
I'll add an example to github asap.
Recently I had the same problem and finally found a solution that helped me.
All I needed to do was to step 1 pixel from the edges of the active sensor rectangle. In your example instead of this rectangle:
meteringRectangleList = new MeteringRectangle[]{new MeteringRectangle(0,0,500,500,1000)};
I would use this:
meteringRectangleList = new MeteringRectangle[]{new MeteringRectangle(1,1,500,500,1000)};
and it started working as magic on both Samsung and Nexus 5!
(note that you should also step 1 pixel from right/bottom edges if you use maximum values there)
It seems that many vendors have poorly implemented this part of documentation
If the metering region is outside the used android.scaler.cropRegion returned in capture result metadata, the camera device will ignore the sections outside the crop region and output only the intersection rectangle as the metering region in the result metadata. If the region is entirely outside the crop region, it will be ignored and not reported in the result metadata.
After implementing the camera2 API for the inApp camera I noticed that on Samsung devices the images appear blurry. After searching about that I found the Sasmung Camera SDK (http://developer.samsung.com/galaxy#camera). So after implementing the SDK on Samsung Galaxy S7 the images are fine now, but on Galaxy S6 they are still blurry. Someone experienced those kind of issues with Samsung devices?
EDIT:
To complement #rcsumners comment. I am setting autofocus by using
mPreviewBuilder.set(SCaptureRequest.CONTROL_AF_TRIGGER, SCaptureRequest.CONTROL_AF_TRIGGER_START);
mSCameraSession.capture(mPreviewBuilder.build(), new SCameraCaptureSession.CaptureCallback() {
#Override
public void onCaptureCompleted(SCameraCaptureSession session, SCaptureRequest request, STotalCaptureResult result) {
isAFTriggered = true;
}
}, mBackgroundHandler);
It is a long exposure image where the use has to take an image of a static non moving object. For this I am using the CONTROL_AF_MODE_MACRO
mCaptureBuilder.set(SCaptureRequest.CONTROL_AF_MODE, SCaptureRequest.CONTROL_AF_MODE_MACRO);
and also I am enabling auto flash if it is available
requestBuilder.set(SCaptureRequest.CONTROL_AE_MODE,
SCaptureRequest.CONTROL_AE_MODE_ON_AUTO_FLASH);
I am not really an expert in this API, I mostly followed the SDK example app.
There could be a number of issues causing this problem. One prominent one is the dimensions of your output image
I ran Camera2 API and the preview is clear, but the output was quite blurry
val characteristics: CameraCharacteristics? = cameraManager.getCameraCharacteristics(cameraId)
val size = characteristics?.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)?.getOutputSizes(ImageFormat.JPEG) // The issue
val width = imageDimension.width
val height = imageDimension.height
if (size != null) {
width = size[0].width; height = size[0].height
}
val imageReader = ImageReader.newInstance(width, height, ImageFormat.JPEG, 5)
The code below was returning a dimension about 245*144 which was way to small to be sent to the image reader. Some how the output was stretching the image making it end up been blurry. Therefore I removed this line below.
val size = characteristics?.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)?.getOutputSizes(ImageFormat.JPEG) // this was returning a small
Setting the width and height manually resolved the issue.
You're setting the AF trigger for one frame, but then are you waiting for AF to complete? For AF_MODE_MACRO (are you verifying the device lists support for this AF mode?) you need to wait for AF_STATE_FOCUSED_LOCKED before the image is guaranteed to be stable and sharp. (You may also receive NOT_FOCUSED_LOCKED if the AF algorithm can't reach sharp focus, which could be because the object is just too close for the lens, or the scene is too confusing)
On most modern devices, it's recommended to use CONTINUOUS_PICTURE and not worry about AF triggering unless you really want to lock focus for some time period. In that mode, the device will continuously try to focus to the best of its ability. I'm not sure all that many devices support MACRO, to begin with.
I spent a lot of time debugging different problems that were reproducible only on a specific devices.
For instance I left my attempts to take a picture from a camera using an Intent. Because only a limited set of the devices behave as expected.
Another example is when I use a byte array from the onPictureTakenCallback:
public void onPictureTaken(byte[] data, Camera camera) {
byte[] tempData = new byte[data.length];
System.arraycopy(data, 0, dataTemp, 0, data.length);
///...
}
So if I don't make a copy, but use original "data" array some time later then I fall into troubles because some devices clean this array up after a time. But other devices don't do such cleaning so it works perfectly without doing a copy.
One more example:
Some devices return null when:
Camera.Parameters params = camera.getParameters();
List<Camera.Size> sizes = params.getSupportedPreviewSizes();
// sizes is null
But most of devices (I think) return a list of supported sizes.
So I wonder if is there any kind of knowledge base / FAQ assembled of such problems? If not, let's post here issues with which we faced?
I'm unaware of it. But byte array you are receiving is mmapped, and in control of another (native) application (and thus data may go at camera application discretion, if it reuses this buffer)
Best way is to copy it away to safe location ASAP
As for preview sizes - they are a mess. Even if you get this list, not all resolutions are supported actually ( I got segfaults on bigger resolutions - somehow preview buffer did not fit ). Only way is to probe whether this preview size is actually supported by activating them in turn and waiting for exc eption