Develop Reference

Android Camera2 pipeline for preview and MediaCodec input Surface

I'm trying to send Android Camera2 output to both a preview Surface and a surface obtained from MediaCodec.createInputSurface(). However, when I pass those surfaces to a call to CameraDevice.createCaptureSession and then try to build a CaptureRequest, I get:
android.hardware.camera2.CameraAccessException: CAMERA_ERROR (3): submitRequestList - cannot use a surface that wasn't configured.
The CaptureRequest building logic (see below) is from an offical Flutter camera plugin and works fine when you use MediaRecorder.getSurface() instead of MediaCodec.createInputSurface(). Which suggests that the MediaCodec surface hasn't been configured. I'm using a VideoEncoder class from well-tried open source RTMP code (https://github.com/pedroSG94/rtmp-rtsp-stream-client-java) that works with the old Camera API (i.e. not Camera2). That class initialises the codec thus:
String type = CodecUtil.H264_MIME;
MediaCodecInfo encoder = chooseEncoder(type);
try {
if (encoder != null) {
codec = MediaCodec.createByCodecName(encoder.getName());
} else {
Log.e(TAG, "Valid encoder not found");
return false;
}
MediaFormat videoFormat;
//if you dont use mediacodec rotation you need swap width and height in rotation 90 or 270
// for correct encoding resolution
String resolution;
if (!hardwareRotation && (rotation == 90 || rotation == 270)) {
resolution = height + "x" + width;
videoFormat = MediaFormat.createVideoFormat(type, height, width);
} else {
resolution = width + "x" + height;
videoFormat = MediaFormat.createVideoFormat(type, width, height);
}
Log.i(TAG, "Prepare video info: " + this.formatVideoEncoder.name() + ", " + resolution);
videoFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT,
this.formatVideoEncoder.getFormatCodec());
videoFormat.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 0);
videoFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitRate);
videoFormat.setInteger(MediaFormat.KEY_FRAME_RATE, fps);
videoFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, iFrameInterval);
if (hardwareRotation) {
videoFormat.setInteger("rotation-degrees", rotation);
}
if (this.avcProfile > 0 && this.avcProfileLevel > 0) {
// MediaFormat.KEY_PROFILE, API > 21
videoFormat.setInteger("profile", this.avcProfile);
// MediaFormat.KEY_LEVEL, API > 23
videoFormat.setInteger("level", this.avcProfileLevel);
}
codec.configure(videoFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
inputSurface = codec.createInputSurface();
The code that fails when I try to build a capture request is in CameraCaptureSession.StateCallback.onConfigured, where the call to build() raises the exception:
createCaptureSession( CameraDevice.TEMPLATE_RECORD, successCallback, surfaceFromMediaCodec );
private void createCaptureSession(
int templateType, Runnable onSuccessCallback, Surface... surfaces)
throws CameraAccessException {
// Close any existing capture session.
closeCaptureSession();
// Create a new capture builder.
captureRequestBuilder = cameraDevice.createCaptureRequest(templateType);
// Build Flutter surface to render to
SurfaceTexture surfaceTexture = flutterTexture.surfaceTexture();
surfaceTexture.setDefaultBufferSize(previewSize.getWidth(), previewSize.getHeight());
Surface flutterSurface = new Surface(surfaceTexture);
captureRequestBuilder.addTarget(flutterSurface);
List<Surface> remainingSurfaces = Arrays.asList(surfaces);
if (templateType != CameraDevice.TEMPLATE_PREVIEW) {
// If it is not preview mode, add all surfaces as targets.
for (Surface surface : remainingSurfaces) {
captureRequestBuilder.addTarget(surface);
}
}
// Prepare the callback
CameraCaptureSession.StateCallback callback =
new CameraCaptureSession.StateCallback() {
#Override
public void onConfigured(#NonNull CameraCaptureSession session) {
try {
if (cameraDevice == null) {
dartMessenger.send(
DartMessenger.EventType.ERROR, "The camera was closed during configuration.");
return;
}
cameraCaptureSession = session;
captureRequestBuilder.set(
CaptureRequest.CONTROL_MODE, CameraMetadata.CONTROL_MODE_AUTO);
cameraCaptureSession.setRepeatingRequest(captureRequestBuilder.build(), null, null);
if (onSuccessCallback != null) {
onSuccessCallback.run();
}
} catch (CameraAccessException | IllegalStateException | IllegalArgumentException e) {
Log.i( TAG, "exception building capture session " + e );
dartMessenger.send(DartMessenger.EventType.ERROR, e.getMessage());
}
}
#Override
public void onConfigureFailed(#NonNull CameraCaptureSession cameraCaptureSession) {
dartMessenger.send(
DartMessenger.EventType.ERROR, "Failed to configure camera session.");
}
};
// Collect all surfaces we want to render to.
List<Surface> surfaceList = new ArrayList<>();
surfaceList.add(flutterSurface);
surfaceList.addAll(remainingSurfaces);
// Start the session
cameraDevice.createCaptureSession(surfaceList, callback, null);
}
If I remove the MediaCodec inputSurface as a build target, it works (but I don't capture anything into the MediaCodec). What am I missing? BTW, there are bits of Flutter code in the second code extract but there's no evidence that the embedding in Flutter is relevant.

Answering my own question. I was thrown off the scent by the misleading Exception message "cannot use a surface that wasn't configured". The surface was configured. And I thought I'd checked the sizes, but one was 720x480, the other was 480x720. It worked after I swapped.

Android Camera2 configure crashes without reporting proper error

I am trying some preview frame processing on top of camera2basic example from android. createCaptureSession call is failing with no so useful error reporting. Can someone help me understand whats going wrong?
Following works and still image capturing works as expected
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface()),
Following also works with my custom frame processing happening
mPreviewRequestBuilder.addTarget(mFrameReader.getSurface());
mCameraDevice.createCaptureSession(Arrays.asList(surface, mFrameReader.getSurface()), ....
However, its fails without proper error reporting when try both the things together
mPreviewRequestBuilder.addTarget(mFrameReader.getSurface());
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface(), mFrameReader.getSurface()),
I see following error in logs and my app crashes
06-20 06:37:56.430 28839-29176/com.example.android.camera2basic
E/CameraCaptureSession: Session 0: Failed to create capture session;
configuration failed
For reference heres the whole method
/**
* Creates a new {#link CameraCaptureSession} for camera preview.
*/
private void createCameraPreviewSession() {
try {
SurfaceTexture texture = mTextureView.getSurfaceTexture();
assert texture != null;
// We configure the size of default buffer to be the size of camera preview we want.
texture.setDefaultBufferSize(mPreviewSize.getWidth(), mPreviewSize.getHeight());
// This is the output Surface we need to start preview.
Surface surface = new Surface(texture);
// We set up a CaptureRequest.Builder with the output Surface.
mPreviewRequestBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
mPreviewRequestBuilder.addTarget(surface);
mPreviewRequestBuilder.addTarget(mFrameReader.getSurface());
// Here, we create a CameraCaptureSession for camera preview.
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface(), mFrameReader.getSurface()),
new CameraCaptureSession.StateCallback() {
#Override
public void onConfigured(#NonNull CameraCaptureSession cameraCaptureSession) {
// The camera is already closed
if (null == mCameraDevice) {
return;
}
// When the session is ready, we start displaying the preview.
mCaptureSession = cameraCaptureSession;
try {
// Auto focus should be continuous for camera preview.
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AF_MODE,
CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
// Flash is automatically enabled when necessary.
setAutoFlash(mPreviewRequestBuilder);
// Finally, we start displaying the camera preview.
mPreviewRequest = mPreviewRequestBuilder.build();
mCaptureSession.setRepeatingRequest(mPreviewRequest,
mCaptureCallback, mBackgroundHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}
#Override
public void onConfigureFailed(
#NonNull CameraCaptureSession cameraCaptureSession) {
showToast("Failed");
}
}, null
);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}

Android Camera2, take picture continuously

I need to take pictures continuously with Camera2 API. It works fine on high end devices (for instance a Nexus 5X), but on slower ones (for instance a Samsung Galaxy A3), the preview freezes.
The code is a bit long, so I post only the most relevant parts:
Method called to start my preview:
private void startPreview() {
SurfaceTexture texture = mTextureView.getSurfaceTexture();
if(texture != null) {
try {
// We configure the size of default buffer to be the size of camera preview we want.
texture.setDefaultBufferSize(mPreviewSize.getWidth(), mPreviewSize.getHeight());
// This is the output Surface we need to start preview.
Surface surface = new Surface(texture);
// We set up a CaptureRequest.Builder with the output Surface.
mPreviewRequestBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
mPreviewRequestBuilder.addTarget(surface);
// Here, we create a CameraCaptureSession for camera preview.
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface()), new CameraCaptureSession.StateCallback() {
#Override
public void onConfigured(#NonNull CameraCaptureSession cameraCaptureSession) {
// If the camera is already closed, return:
if (mCameraDevice == null) { return; }
// When the session is ready, we start displaying the preview.
mCaptureSession = cameraCaptureSession;
// Auto focus should be continuous for camera preview.
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
mPreviewRequest = mPreviewRequestBuilder.build();
// Start the preview
try { mCaptureSession.setRepeatingRequest(mPreviewRequest, null, mPreviewBackgroundHandler); }
catch (CameraAccessException e) { e.printStackTrace(); }
}
#Override
public void onConfigureFailed(#NonNull CameraCaptureSession cameraCaptureSession) {
Log.e(TAG, "Configure failed");
}
}, null
);
}
catch (CameraAccessException e) { e.printStackTrace(); }
}
}
Method called to take a picture:
private void takePicture() {
try {
CaptureRequest.Builder captureBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);
captureBuilder.addTarget(mImageReader.getSurface());
mCaptureSession.capture(captureBuilder.build(), null, mCaptureBackgroundHandler);
}
catch (CameraAccessException e) { e.printStackTrace(); }
}
And here is my ImageReader:
private final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(final ImageReader reader) {
mSaveBackgroundHandler.post(new Runnable() {
#Override
public void run() {
// Set the destination file:
File destination = new File(getExternalFilesDir(null), "image_" + mNumberOfImages + ".jpg");
mNumberOfImages++;
// Acquire the latest image:
Image image = reader.acquireLatestImage();
// Save the image:
ByteBuffer buffer = image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.remaining()];
buffer.get(bytes);
FileOutputStream output = null;
try {
output = new FileOutputStream(destination);
output.write(bytes);
}
catch (IOException e) { e.printStackTrace(); }
finally {
image.close();
if (null != output) {
try { output.close(); }
catch (IOException e) { e.printStackTrace(); }
}
}
// Take a new picture if needed:
if(mIsTakingPictures) {
takePicture();
}
}
});
}
};
I have a button that toggle the mIsTakingPictures boolean, and makes the first takePicture call.
To recap, I'm using 3 threads:
one for the preview
one for the capture
one for the image saving
What can be the cause of this freeze?

It's impossible to avoid framing lost in your preview when you are taking images all time on weak devices. The only way to avoid this is on devices which support TEMPLATE_ZERO_SHUTTER_LAG and using a reprocessableCaptureSession. The documentation about this is pretty horrible and find a way to implement it can be a odyssey. I have this problem a few months ago and finally I found the way to implement it:
How to use a reprocessCaptureRequest with camera2 API
In that answer you can also find some Google CTS test's which also implements ReprocessableCaptureSession and shoot some burst captures with ZSL template.
Finally, you can also use a CaptureBuilder with your preview surface and the image reader surface attached, in that case your preview will continue working all time and also you will save each frame as a new picture. But you will still having the freeze problem.
I also tried implement a burst capture using a handler which dispatch a new capture call each 100 milliseconds, this second option was pretty good in performance and avoiding frame rate lost, but you will not get as many captures per second like the two ImageReader option.
Hope that my answer will help you a bit, API 2 still being a bit complex and there's not so many examples or information about it.

One thing I noticed on low end devices: the preview stops after a capture, even when using camera 1 api, so it has to be restarted manually, thus producing a small preview freeze when capturing a high resolution picture.
But the camera 2 api provides the possibility to get raw image when taking a still capture (that wasn't possible on the devices I have when using camera 1 (Huawei P7, Sony Xperia E5, wiko UFeel)). Using this feature is much faster than capturing a JPEG (maybe due to JPEG compression), so the preview can be restarted earlier, and the preview freeze is shorter. Of course using this solution you'll have to convert the picture from YUV to JPEG in a background task..

How to get each frame data using camera2 API in Android5.0 in realtime

I am working with camera2Basic now and trying to get each frame data to do some image processing. I am using camera2 API in Android5.0, everything is fine when only doing the camera preview and it is fluid. But the preview stuck when I use the ImageReader.OnImageAvailableListener callback to get each frame data, this cause a bad User Experience.
The following is my related codes:
This is the setup for camera and ImageReader, I set the format of image is YUV_420_888
public<T> Size setUpCameraOutputs(CameraManager cameraManager,Class<T> kClass, int width, int height) {
boolean flagSuccess = true;
try {
for (String cameraId : cameraManager.getCameraIdList()) {
CameraCharacteristics characteristics = cameraManager.getCameraCharacteristics(cameraId);
// choose the front or back camera
if (FLAG_CAMERA.BACK_CAMERA == mChosenCamera &&
CameraCharacteristics.LENS_FACING_BACK != characteristics.get(CameraCharacteristics.LENS_FACING)) {
continue;
}
if (FLAG_CAMERA.FRONT_CAMERA == mChosenCamera &&
CameraCharacteristics.LENS_FACING_FRONT != characteristics.get(CameraCharacteristics.LENS_FACING)) {
continue;
}
StreamConfigurationMap map = characteristics.get(
CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
Size largestSize = Collections.max(
Arrays.asList(map.getOutputSizes(ImageFormat.YUV_420_888)),
new CompareSizesByArea());
mImageReader = ImageReader.newInstance(largestSize.getWidth(), largestSize.getHeight(),
ImageFormat.YUV_420_888, 3);
mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler);
...
mCameraId = cameraId;
}
} catch (CameraAccessException e) {
e.printStackTrace();
} catch (NullPointerException e) {
}
......
}
When the camera opened successfully, I Create a CameraCaptureSession for camera preview
private void createCameraPreviewSession() {
if (null == mTexture) {
return;
}
// We configure the size of default buffer to be the size of camera preview we want.
mTexture.setDefaultBufferSize(mPreviewSize.getWidth(), mPreviewSize.getHeight());
// This is the output Surface we need to start preview
Surface surface = new Surface(mTexture);
// We set up a CaptureRequest.Builder with the output Surface.
try {
mPreviewRequestBuilder =
mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
mPreviewRequestBuilder.addTarget(mImageReader.getSurface());
mPreviewRequestBuilder.addTarget(surface);
// We create a CameraCaptureSession for camera preview
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface()),
new CameraCaptureSession.StateCallback() {
#Override
public void onConfigured(CameraCaptureSession session) {
if (null == mCameraDevice) {
return;
}
// when the session is ready, we start displaying the preview
mCaptureSession = session;
// Finally, we start displaying the camera preview
mPreviewRequest = mPreviewRequestBuilder.build();
try {
mCaptureSession.setRepeatingRequest(mPreviewRequest,
mCaptureCallback, mBackgroundHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}
#Override
public void onConfigureFailed(CameraCaptureSession session) {
}
}, null);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}
The last is the ImageReader.OnImageAvailableListener callback
private final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(ImageReader reader) {
Log.d(TAG, "The onImageAvailable thread id: " + Thread.currentThread().getId());
Image readImage = reader.acquireLatestImage();
readImage.close();
}
};
Maybe I do the wrong setup but I try several times and it doesn't work. Maybe there is another way to get frame data rather than ImageReader but I don't know.
Anybody knows how to get each frame data in realtime?

I do not believe that Chen is correct. The image format has almost 0 effect on the speed on the devices I have tested. Instead, the problem seems to be with the image size. On an Xperia Z3 Compact with the image format YUV_420_888, I am offered a bunch of different options in the StreamConfigurationMap's getOutputSizes method:
[1600x1200, 1280x720, 960x720, 720x480, 640x480, 480x320, 320x240, 176x144]
For these respective sizes, the maximum fps I get when setting mImageReader.getSurface() as a target for the mPreviewRequestBuilder are:
[13, 18, 25, 28, 30, 30, 30, 30 ]
So one solution is to use a lower resolution to achieve the rate you want. For the curious... note that these timings do not seem to be affected by the line
mPreviewRequestBuilder.addTarget(surface);
...
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface()),
I was worried that adding the surface on the screen might be adding overhead, but if I remove that first line and change the second to
mCameraDevice.createCaptureSession(Arrays.asList(mImageReader.getSurface()),
then I see the timings change by less than 1 fps. So it doesn't seem to matter whether you are also displaying the image on the screen.
I think there is simply some overhead in the camera2 API or ImageReader's framework that makes it impossible to get the full rate that the TextureView is clearly getting.
One of the most disappointing things of all is that, if you switch back to the deprecated Camera API, you can easily get 30 fps by setting up a PreviewCallback via the Camera.setPreviewCallbackWithBuffer method. With that method, I am able to get 30fps regardless of the resolution. Specifically, although it does not offer me 1600x1200 directly, it does offer 1920x1080, and even that is 30fps.

I'm trying the same things, I think you may change the Format like
mImageReader = ImageReader.newInstance(largestSize.getWidth(),
largestSize.getHeight(),
ImageFormat.FLEX_RGB_888, 3);
Because using the YUV may cause CPU to compress the data and it may cost some time. RGB can be directly showed on the device. And detect face from image should put in other Thread you must know it.

ImageReader Android can't read any image

I'm trying to implement an ImageReader on my application, but i don't know why, he doesn't read anything.
List<Surface> surfaces = new ArrayList<Surface>();
Surface previewSurface = new Surface(texture);
previewRequestBuilder.addTarget(previewSurface);
recordRequestBuilder.addTarget(previewSurface);
surfaces.add(previewSurface);
Surface recorderSurface = mediaRecorder.getSurface();
surfaces.add(recorderSurface);
ImageReader mImageReader = ImageReader.newInstance(previewSize.getWidth(),previewSize.getHeight(), ImageFormat.JPEG,5);
Surface processSurface = mImageReader.getSurface();
surfaces.add(processSurface);
mImageReader.setOnImageAvailableListener(new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(ImageReader reader) {
Log.v("ImageReader ","An Image");
}
},null);
cameraDevice.createCaptureSession(surfaces, new CameraCaptureSession.StateCallback() {
#Override
public void onConfigured(CameraCaptureSession cameraCaptureSession) {
captureSession = cameraCaptureSession;
updateRequest(PREVIEW_REQUEST);
}
#Override
public void onConfigureFailed(CameraCaptureSession cameraCaptureSession) {
Activity activity = getActivity();
if (null != activity) {
Toast.makeText(activity, "Failed", Toast.LENGTH_SHORT).show();
}
}
}, null);
} catch (CameraAccessException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
So i got 3 Surface : previewSurface for the display, recordSurface for recording the video and processSurface to get the images (with ImageReader) and process it.
But I don't even see my Log.v once !
Thanks by advance for your answers.

There are at least 2 reasons why your code might not work:
In your implementation of the OnImageAvailableListener, in the method onImageAvailable(ImageReader reader) you do not read and close the image. In my experience if you don't read/close the image from the reader the camera freezes. If this is the case then you should see the log message at least once (or even more times). I would suggest that you add reading and closing the image to the method:
#Override
public void onImageAvailable(ImageReader reader) {
Log.v("ImageReader ","An Image");
Image img = reader.acquireNextImage();
img.close();
}
Taking 3 streams (for 3 surfaces) of a certain size and type might not be supported on your device. You should verify what support you have on your device (LEGACY/LIMITED/FULL). For instance, your device may not support 3 simultaneous streams of maximum size. Check the documentation. There are nice tables showing what is possible and carefully check if your sizes/types are ok.