I am trying to capture a high resolution frame (1280x720) from the camera in a pair of Google Glass using OpenCV 2.4.10 for Android. I have implemented the CameraBridgeViewBase.CvCameraViewListener2 in my Activity and try to grab the frame in the onCameraFrame method. So far everything works well, and i get a 512x288 Mat object.
My problem is that the 512x288 resolution is not high enough for what I need. So I tried to setup my project the same way as they do in Sample 3 that follows with OpenCV: http://goo.gl/iDyqQj. The problem is that it only works for resolutions below 512x288, as soon as I increase the resolution above this level it defaults back to to being 512x288 (without any notice).
I found some suggestions, http://goo.gl/X2wtM4, that OpenCV is restricting the frame size to a maximum of the screen resolution. But the Google Glass screen should have a 640x360 resolution? I tried to do as described in the answer, but when I override calculateCameraFrameSize and return a Size-object larger than 512x288, I get a distorted frame (but with the larger dimensions, see below).
Does anyone have a suggestion on how capture a higher captured resolution on the Google Glass using OpenCV?
So I found a solution. It seem to be two separate problems. As I thought in my question you need to override calculateCameraFrameSize in JavaCameraView to be able to fetch higher resolutions than the device's screen in onCameraFrame. This is apparently a design choice by OpenCV and have been since version 2.4.5. So this is why I could not get a frame with higher resolution.
Even though I now can get a frame with higher resolution, it still is distorted for most preview sizes. This is a bug in the GDK that seem to have been known for quite some time (since XE10 if I understood correctly), but still is not fixed. Fortunately there is a workaround! The issue is avoided by manually setting the FPS of the preview using setPreviewFpsRange after you acquire the Camera.
Camera.Parameters params = camera.getParameters();
params.setPreviewFpsRange(30000, 30000);
camera.setParameters(params);
Related
In my app I'm trying to use ArCore as sort of a "camera assistant" in a custom camera view.
To be clear - I want to display images for the user in his camera and have him capture images that don't contain the AR models.
From what I understand, in order to capture an image with ArCore I'll have to use the Camera2 API which is enabled by configuring the session to use the "shared Camera".
However, I can't seem to configure the camera to use any high-end resolutions (I'm using pixel 3 so I should be able to go as high as 12MP).
In the "shared camera example", they toggle between Camera2 and ArCore (a shame there's no API for CameraX) and it has several problems:
In the ArCore mode the image is blurry (I assume that's because the depth sensor is disabled as stated in their documentation)
In the Camera2 mode I can't enhance the resolution at all.
I can't use the Camera2 API to capture an image while displaying models from ArCore.
Is this requirement at all possible at the moment?
I have not worked yet with shared camera with ARCore, but I can say a few things regarding the main point of your question.
In ARCore you can configure both CPU image size and GPU image size. You can do that by checking all available camera configurations (available through Session.getSupportedCameraConfigs(CameraConfigFilter cameraConfigFilter)) and selecting your preferred one by passing it back to the ARCore Session. On each CameraConfig you can check which CPU image size and GPU texture size you will get.
Probably you are currently using (maybe by default?) a CameraConfig with the lowest CPU image, 640x480 pixels if I remember correctly, so yes it definitely looks blurry when rendered (but nothing to do with depth sensor in this regard).
Sounds like you could just select a higher CPU image and you're good to go... but unfortunately that's not the case because that configuration applies to every frame. Getting higher resolution CPU images will result in much lower performance. When I tested this I got about 3-4 frames per second on my test device, definitely not ideal.
So now what? I think you have 2 options:
Pause the ARCore session, switch to a higher CPU image for 1 frame, get the image and switch back to the "normal" configuration.
Probably you are already getting a nice GPU image, maybe not the best due to camera Preview, but hopefully good enough? Not sure how you are rendering it, but with some OpenGL skills you can copy that texture. Not directly, of course, because of the whole GL_TEXTURE_EXTERNAL_OES thing... but rendering it onto another framebuffer and then reading the texture attached to it could work. Of course you might need to deal with texture coordinates yourself (full image vs visible area) but that's another topic.
Regarding CameraX, note that it is wrapping Camera2 API in order to provide some camera use cases so that app developers don't have to worry about the camera lifecycle. As I understand it would not be suitable for ARCore to use CameraX as I imagine they need full control of the camera.
I hope that helps a bit!
Update: This looks like it's related to this: Image data from Android camera2 API flipped & squished on Galaxy S5 - I consider this as a bug since Nexus 5/6 works correctly and it makes no sense to need to obtain full sensor size and then cropping manually to reach the desired aspect ratio, might as well not using "supported" output sizes as well!
Problem:
Get characteristics of a camera using Camera2 API, and extract output sizes suitable for a MediaCodec.class
Create a MediaCodec input surface with one of the suitable camera output sizes. Feed the output to some MediaMuxer or whatever, to see the output.
Start camera capture requests using the codec's created surface as the target.
Codec output has the correct size. But the result differs by device:
Nexus 5/6: everything ok on Android 5/6.
Samsung tablet with Android 5.1: for some resolutions, the image is obviously stretched, indicating that the camera output resolution does not match the surface size. Becomes very obvious when starting to rotate the camera - image becomes more and more skewed since it's not aligned with the X/Y axes. For some other resolutions the output is OK. There is no pattern here related to either the size or the aspect ratio.
No problem, one would say. Maybe the surface is not created exactly at the specified width and height, or whatever (even if the output sizes were extracted specifically for a MediaCodec.class target).
So, I created an OpenGL context, generated a texture, a SurfaceTexture for it, set its default buffer size to the camera output size, and created a Surface using the texture. I won't go into the gory details of drawing that to a TextureView or back to the MediaCodec's EGL surface. The result is the same - the Camera2 capture requests outputs a distorted image only for some resolutions.
Digging deeper: calling getTransformMatrix on the SurfaceTexture immediately after updateTexImage - the matrix is always the identity matrix, as expected.
So, the real problem here is that the camera is NOT capturing at the size of the provided target surface. The solution would thereby be to get the actual size the camera is capturing, and the rest is pure GL matrix transforms to draw correctly. But - HOW DO I GET THAT?
Note: using the old Camera API, with exactly the same "preview size" and the same surface as the target (either MediaCodec's or the custom one) - ALL IS FINE! But I can't use the old camera API, since it's both deprecated and also seems to have a max capture size of 1080p, while the Camera2 API goes beyond that, and I need to support 4k recording.
I encounter similar issue, model SM-A7009 with api level 21, legacy camera2 device.
The preview is stretched, surfaceTexture.setDefaultBufferSize not working, the framework will override these value when preview started.
The preview sizes reported from StreamConfigurationMap.getOutputSizes(SurfaceTexture.class) are not all supported.
Only three of them are supported.
$ adb shell dumpsys media.camera |grep preview-size
preferred-preview-size-for-video: 1920x1080
preview-size: 1440x1080
preview-size-values: 1920x1080,1440x1080,1280x720,1056x864,960x720,880x720,800x480,720x480,640x480,528x432,352x288,320x240,176x144
The system dump info list many of the preview sizes, after check all of them, I found only 1440x1080, 640x480, 320x240 are supported.
The supported preview sizes all have 1.33333 ratio. They have the same ratio reported from CameraCharacteristics.SENSOR_INFO_ACTIVE_ARRAY_SIZE.
So I thought it's a bug in some samsung devices with legacy camera2 api in API 21.
The solution is making these devices using deprecated camera API.
Hope it would be helpful for anyone reach here.
So yes, this is a bug on those Samsung devices.
Generally this happens when you ask for multiple different aspect ratios on output, and the device-specific camera code trips over itself on cropping and scaling all of them correctly. You may be able to avoid it by ensuring all requested sizes have the same aspect ratio.
The resolution is probably actually what you asked for - but it's been incorrectly scaled (you could test this with an ImageReader at the problematic size, where you get an explicit buffer you can poke at.)
We are adding additional testing to the Android compliance tests to try to ensure these kinds of stretched outputs don't continue to happen.
I have an app that I am working on that makes use of the front-facing camera on the device. I am trying to set the preview size by getting the list of supported preview sizes and looping through them looking for one that is pretty close. The method that I wrote to do so is basically the same as the one from the OS's own camera app. The method works fine, exactly how I would like it to, that's not why I am here.
I was having problems with the camera preview looking obviously skewed on some devices; either squishing or stretching the preview of the image. I couldn't figure out why it was doing this so I stepped through it and looked at all of the supported preview sizes available to my front-facing camera and found that there were only 2 and neither of them were the correct aspect to be usable. My "surfaceChanged" method in my SurfaceHolder.Callback class is reporting a width and height of 762x480 for the front-facing camera, but of the two supported preview sizes (acquired with cam.getParameters().getSupportedPreviewSizes()) both were in the opposite aspect: 480x800, 320x640.
With these as the only options, it seems impossible to have a preview for my front-facing camera that is not skewed. I know that in versions 2.3 or less, arbitrary values can be used for width and height without regard to supported sizes, but I am trying to make my app work for newer versions of the OS as well. How can I make the preview look correct?
My initial question remains technically unsolved (and I still believe it to be impossible) however I did figure out the trick to a non-skewed preview.
Given the supported preview sizes all being in the incorrect orientation compared to my device's screen it does indeed seem impossible to have a normal preview that fills the entire screen. After looking at several native camera apps on different devices (all of which that display the front-facing preview with no skewing at all) I noticed that very few of the devices (only one, in fact, and that one had a non-standard screen size) had previews that covered the entire screen as I was attempting to do in my app.
I reworked my app to get the size with the closest width and height to the device screen, but without going over in either dimension or in aspect ratio, and then manually set the height and width of the SurfaceView to match the selected size. True there is a small amount of black space around the preview on some devices now, but it definitely looks much better than being skewed. I had a panel with buttons for snapping a picture and manipulating flash settings, etc. on one side anyways, so it really is less noticeable than one might think.
I'm trying to develop an app for Android, and I would need to get uncompressed pictures with a resolution as high as possible from the camera. I tried takePicture's rawCallback and postviewCallback, but they are not working.
Right now I'm trying with OpenCV (version 2.4) using VideoCapture, but I'm stuck in the default 960x720, which is poor for what I need; and my phone, a Samsung Galaxy S3, is able to provide, theoretically, up to 8Mpx (3,264×2,448 for pictures, and 1,920×1,080 for video, according to Wikipedia). VideoCapture.set(Highgui.CV_CAP_PROP_FRAME_WIDTH/HEIGHT, some number) makes the camera return a black image as far as I've found.
Is there any way to obtain a higher resolution, either through OpenCV or with the Android API, without compressing?
I'm really sorry if this has been asked before; I have been looking for days and I have found nothing.
Thank you for your time!
EDIT: Although it is not exactly what I was asking, I found that there is a way to do something very similar: if you set an OnPreviewCallback for the Camera, using setPreviewCallback, you do get the raw picture data from the camera (at least in the S3 I'm working with). I leave it here in case somebody finds it useful in the future.
EDIT: A partial solution is explained in an answer below. To sum up,
vc.set(Highgui.CV_CAP_PROP_FRAME_WIDTH, desiredFrameWidth);
vc.set(Highgui.CV_CAP_PROP_FRAME_HEIGHT, desiredFrameHeight);
works under some conditions; please see below for further detail.
You have to get supported camera preview resoultions by calling getSupportedPreviewSizes.
After this you can set any resolution with method setPreviewSize. And don't forget to setParameters in the end. Actally many OpenCV Android examples contain this information (look at sample3).
In case anybody ever finds this useful, I found a (partial) solution: If your VideoCapture variable is called vc, this should work:
vc.set(Highgui.CV_CAP_PROP_FRAME_WIDTH, desiredFrameWidth);
vc.set(Highgui.CV_CAP_PROP_FRAME_HEIGHT, desiredFrameHeight);
Mind that the combination of width and height must be one of the supported picture formats for your camera, otherwise it will just get a black image. You can get those through Camera.Parameters.getSupportedPictureSizes().
However, setting a high resolution appears to exceed the YUV conversion buffer's capacity, so I'm still struggling with that. I'm going to make a new separate question for that, to keep everything clearer: new thread
setPreviewSize does not set picture resolution. setPictureSize does.
I have a camera app, in it's simplest state it's nothing more then the cameraPreview example with some 'takePicture' code. The link to the example online is for 2.0, and i'm developping against 1.5 (API lvl 3), but still, here it is: http://developer.android.com/resources/samples/ApiDemos/src/com/example/android/apis/graphics/CameraPreview.html
The biggest difference with the old version is the whole "getOptimalPreviewSize" thing.
Everything is done landscape.
Now the problem is: i have a preview, but when i take the picture there is more information on that picture then there is on the preview. The top and bottom show stuff that wasn't visible on the preview.
Now i am going to put an overlay on top of the preview, to align the object in the picture with something. If the picture is taken, the whole thing gets squeezed a bit, and i'm all out of alignment :(.
The camera app on the system doesn't have this problem, so it must be possible to fix this. Any thoughts?
If i must manually set the preview and/or picture size, i'll have trouble with different handsets i guess, and because there are a lot of function only since API lvl 5 (e.g.: getOptimalPreviewSize), I can't use these.
Having built a custom camera app for Android, I know exactly what you are going through. Android 1.5 makes up only 1.1% of the Android users as of 10/29/2011. You will be better off jumping up to at least API level 5. If you want to support portrait and landscape previews consistently on all devices, I recommend you go even higher.
Make use of the getSupportedPreviewSizes() and getSupportedPicturesSizes(). These functions tell you exactly what the camera supports (varies by phone/manufacturer). Run through the enumerations and find values that match from both. Use the one that suits you best.
Word of warning: Failure to set preview and picture sizes that are actually supported can cause your app to crash on certain devices. I've seen this first hand.
reference to 1.5 Android users
The largest preview size returned by getSupportedPicturSizes represents the native resolution of your camera. If the aspect ratio of that size differs from your preview size or the picture size that you set your Camera object to then then cropping will occur. You can compare the aspect ratios to determine how much will be cropped and in which direction.(top/bottom vs left/right)