I am using Camera2 API to create a Camera component that can scan barcodes and has ability to take pictures during scanning. It is kinda working but the preview is flickering - it seems like previous frames and sometimes green frames are interrupting realtime preview.
My code is based on Google's Camera2Basic. I'm just adding one more ImageReader and its surface as a new output and target for CaptureRequest.Builder. One of the readers uses JPEG and the other YUV. Flickering disappears when I remove the JPEG reader's surface from outputs (not passing this into createCaptureSession).
There's quite a lot of code so I created a gist: click - Tried to get rid of completely irrelevant code.
Is the device you're testing on a LEGACY-level device?
If so, any captures targeting a JPEG output may be much slower since they can run a precapture sequence, and may briefly pause preview as well.
But it should not cause green frames, unless there's a device-level bug.
If anyone ever struggles with this. There is table in the docs showing that if there are 3 targets specified, the YUV ImageReader can use images with maximum size equal to the preview size (maximum 1920x1080). Reducing this helped!
Yes you can. Assuming that you configure your preview to feed the ImageReader with YUV frames (because you could also put JPEG there, check it out), like so:
mImageReaderPreview = ImageReader.newInstance(mPreviewSize.getWidth(), mPreviewSize.getHeight(), ImageFormat.YUV_420_888, 1);
You can process those frames inside your OnImageAvailable listener:
#Override
public void onImageAvailable(ImageReader reader) {
Image mImage = reader.acquireNextImage();
if (mImage == null) {
return;
}
try {
// Do some custom processing like YUV to RGB conversion, cropping, etc.
mFrameProcessor.setNextFrame(mImage));
mImage.close();
} catch (IllegalStateException e) {
Log.e("TAG", e.getMessage());
}
Related
In my project, I need to capture the frames of the camera streams continuously. Here is the current code snippet I used.
To set up the ImageReader, I set the maxImages to 20. Let is every time when callback is triggered, there would have 20 frames in the ImageReader Queue.
imageReader = ImageReader.newInstance(
optimumSize.getWidth(),
optimumSize.getHeight(),
ImageFormat.YUV_420_888,
20
);
Then to access each image of these 20 frames. I used the following snippet.
imageReader.setOnImageAvailableListener(new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(ImageReader reader) {
Image image = reader.acquireNextImage();
while (image != null) {
// some processing here.....
image.close();
image = reader.acquireNextImage();
}
if (image != null) {
image.close();
}
}
}, processingHandler);
The key obstacle here is to be able to access each of 20 frames in a callback, for further image processing. However the aforementioned code seems have some problems (I can only access the latest image in the underlying queue). In fact, I only need to access a small patch (50 x 50 pixels) in each frames, specified by users.
The reason for doing this is that I need to get the 20 continuous frames data with sampling frequency being ~60Hz. This seems really hard to achieve if we can only access single frame in each callback, which can only achieve up to 30fps.
Any suggestions would be super welcome! Thanks!
Setting maxImages to 20 just means the queue will allow you to acquire 20 Images at the same time; it does not mean the onImageAvailable callback will fire only once 20 images are queued. The callback fires as soon as a single image is present.
Most camera devices run at 30fps max, so it's not surprising that's the speed you're seeing. Some cameras do have 60fps modes, but you have to explicitly switch to a CONTROL_AE_TARGET_FPS_RANGE of (60,60) to get that, and that's only if the device's CONTROL_AE_AVAILABLE_TARGET_FPS_RANGE values include that range.
60fps may also be resolution-limited (check the StreamConfigurationMap for minimum frame durations to find what resolutions can support 60fps, if you want to double-check).
I followed the Android Studio tutorial to get the CameraPreview to work (Camera API Android Developer Guide). This works fine for me and i can view the camera stream in my FrameLayout.
But I would like to get the RGB values from a specific Pixel in the Preview everytime it changes. I did not find a method which gives me the previewImage as a bitmap and was not able to understand the usage of the onPreviewFrame method
#Override
public void onPreviewFrame(byte[] data, Camera camera) {}
How can I get the RGB values from a Camerapreview Pixel?
If you are using the Camera2 API, you can implement the ImageReader.OnImageAvailableListener class in your application. After that, you override the onImageAvailable function , which gets an ImageReader as argument. Then you can access the image just recorded with imageReader.acquireNextImage().
With either API, you need to handle processing YUV data yourself, unfortunately.
Camera devices natively produce YUV data, not RGB, so the API doesn't spend extra resources to auto-convert the data. The main easy exception is piping data to the GPU, where the GPU driver auto-converts YUV to RGB for you within your pixel shader.
But if you're just in regular app code, you need to parse the data.
For the deprecated android.hardware.Camera API, the output is NV21 by default, and you can usually select YV12 as another option.
The wikipedia article on YUV is relatively helpful: https://en.wikipedia.org/wiki/YUV
But it does have the wrong conversion coefficients for YUV->RGB conversion; they should be:
R = Y + 1.402 (Cr-128)
G = Y - 0.34414 (Cb-128) - 0.71414 (Cr-128)
B = Y + 1.772 (Cb-128)
(Cb = U, Cr = V)
You can also take a look at this stackoverflow post:
Extract black and white image from android camera's NV21 format
which has code that looks to be correct for the conversion.
Question
I want to do something similar to what's done on the Camera2Basic sample, that is:
Previewing images from the camera using a TextureView
Processing images from the camera using a ImageReader
With a few differences regarding 2:
I'm only interested on the gray channel (brightness) from the images to be processed. Their dimensions should be around 1000 x 1000 pixels (and not the highest resolution available)
When a image to be processed is available, a generic process(Image) method will be called instead of saving images to disk. What this method does is out of the scope of this question, but it takes around 50 ms to return
The image data should be processed periodically (around 10 FPS, but speed is not critical) instead of eventually
How can I accomplish this using the Camera2 API?
Observations
I've changed the way I'm creating the ImageReader instance, selecting smaller dimensions and a different format (YUV_420_888 instead of JPEG). The Y plane will be accessed in order to get the brightness data. Is there a more efficient format (since I'm simply ignoring the U and V planes)?
Both TextureView and ImageReader surfaces should be filled periodically, but at different rates. Since there can be only one repeating CameraRequest on a CameraCaptureSession (which can be set by calling setRepeatingRequest()), am I supposed to manually call capture() periodically (e.g. call setRepeatingRequest() with the preview request and call capture() periodically with the process request)?
Can the performance be improved by sending reprocessed requests to obtain the images to be processed from the preview images? If so, how can I do it?
I don't know how to help you with the gray channel, I suggest you to try to study the planes of the YUV format image and try to get it from there.
Also try to check all the values that you can set in the CaptureBuilder, maybe you can achieve your objetive using SENSOR_TEST_PATTERN_MODE, COLOR_CORRECTION_MODE, or BLACK_LEVEL_LOCK. You can check all the info in android documentation
About process just one of every 10 frames, just discard the frames in your process() method using a simple:
if (result.getFrameNumber() % 10 != 0) return;
Finally remember to close all the images that you recieve in your ImageReader OnImageAvailableListener to avoid memory leaks and improve your performance :P
#Override
public void onImageAvailable(ImageReader imageReader) {
Image image = null;
try {
image = imageReader.acquireNextImage();
//Do whatever you want with your Image
if (image != null) {
image.close();
}
} catch (IllegalStateException iae) {
if (image != null) {
image.close();
}
}
}
hope that it will help you, let me know if I can help you in something else!
I've bumped into the issue with slow focusing on Nexus 6.
I develop camera application and now I'm using camera2 API.
For application needs we create preview request with 2 surfaces
- SurfaceView (viewfinder)
- YUV ImageReader surface (to use data in hstogram calculation)
And there is a critical point! If just add only viewfinder surface, focusing occurs as normal. But with 2 those surfaces focusing occurs very slow with visual steps of lens moving!
Code is quite standard, written according google documentations:
mImageReaderPreviewYUV = ImageReader.newInstance(previewWidth, previewHeight, ImageFormat.YUV_420_888, 2);
previewRequestBuilder = camDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
previewRequestBuilder.addTarget(getCameraSurface()); //Add surface of SurfaceView
previewRequestBuilder.addTarget(mImageReaderPreviewYUV); //Add ImageReader
mCaptureSession.setRepeatingRequest(previewRequestBuilder.build(), captureCallback null);
Does the system logcat show any warnings about buffers not being available?
Is the preview frame rate slow, or is smooth (~30fps) but focusing just works oddly?
If the former, you may not be returning Image objects to the ImageReader (by closing them once done with them) at 30 fps, so the camera device is starved for buffers to fill, and cannot maintain 30fps preview.
To test this, implement the minimal ImageReaderListener.onImageAvailable(ImageReader reader) method that just returns the image immediately:
public class TestImageListener extends ImageReaderListener {
public void onImageAvailable(ImageReader reader) {
Image img = reader.acquireNextImage();
img.close();
}
}
...
mImageReaderPreviewYUV.setOnImageAvailableListener(new TestImageListener());
If this lets you get fluid preview, then your image processing is too slow.
As a solution, you should increase the number of buffers in your ImageReader, and the nuse the reader.acquireLatestImage() to drop older buffers and only process the newest Image each time you calculate your histogram.
I had the same issues on the N6 and I think it works smoother now - add the ImageReader surface before the camera surface:
previewRequestBuilder = camDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
previewRequestBuilder.addTarget(mImageReaderPreviewYUV); //Add ImageReader
previewRequestBuilder.addTarget(getCameraSurface()); //Add surface of SurfaceView
I also tested my camera app with a N4/5.0.1 and both ways work perfectly there.
I am creating an Android app to do some image processing techniques with the camera and it needs to be fast. This is the pseudo-code of how the entire system works:
1. loop while not finished
1.1 get image frame
1.2 process image for object detection
2. end loop
I actually have questions on the basics of the Camera class:
Is previewing the perceived image from the camera faster than no previews at all? The former means using SurfaceView to preview the image.
Let's say from the takePicture() method, can the image data array be obtained without the previews?
My real question is, what is the best way to obtain the image data (say, byte[] array) quickly and iteratively after processing the image (as stated on top)?
I planned to use takePicture() method to get the image data, but I need your opinion if this is the only way or if there other better ways.
You can setup a SurfaceView as the Camera's preview display and get the data of every preview frame using the PreviewCallback. This would be better than using takePicture if you don't need the high resolution that takePicture captures. In other words, if you want to capture images of lower quality at a faster rate, use PreviewCallback... if you want to capture images of higher quality at a very slow rate, use takePicture.
As for your questions, I don't think you can take pictures without using a preview display, but i could be wrong.
class MainActivity extends Activity implements Camera.PreviewCallback, SurfaceHolder.Callback {
...
public void surfaceCreated(SurfaceHolder holder) {
camera = Camera.open();
camera.setPreviewCallback(this);
...
}
public void onPreviewFrame(byte[] data, Camera camera) {
// image data contained in data... do as you wish
}
}