Develop Reference

MediaCodec's Persistent Input Surface unsupported by Camera2 Session?

I am writing an Android app that supports saving RAW/JPEG and records videos at the same time. I tried passing 4 surfaces when creating CameraCaptureSession: preview, 2x ImageSaver and 1x PersistentInputSurface created by MediaCodec#createPersistentInputSurface. By using persistent input surface, I intend to avoid a stoppage between 2 captures.
When creating the session it fails with:
W/CameraDevice-JV-0: Stream configuration failed due to: endConfigure:380: Camera 0: Unsupported set of inputs/outputs provided
Session 0: Failed to create capture session; configuration failed
I have tried taking out all other surfaces, leaving only the PersistentInputSurface, still fails.
#Override
public void onResume() {
super.onResume();
//Some other setups...
if (persistentRecorderSurface == null) {
persistentRecorderSurface = MediaCodec.createPersistentInputSurface();
}
startBackgroundThread();
startCamera();
if (mPreviewView.isAvailable()) {
configureTransform(mPreviewView.getWidth(), mPreviewView.getHeight());
} else {
mPreviewView.setSurfaceTextureListener(mSurfaceTextureListener);
}
if (mOrientationListener != null && mOrientationListener.canDetectOrientation()) {
mOrientationListener.enable();
}
}
private void createCameraPreviewSessionLocked() {
try {
SurfaceTexture texture = mPreviewView.getSurfaceTexture();
texture.setDefaultBufferSize(mPreviewSize.getWidth(), mPreviewSize.getHeight());
Surface surface = new Surface(texture);
mPreviewRequestBuilder = mBackCameraDevice.createCaptureRequest(
CameraDevice.TEMPLATE_PREVIEW);
mPreviewRequestBuilder.addTarget(surface);
mBackCameraDevice.createCaptureSession(Arrays.asList(
surface,
mJpegImageReader.get().getSurface(),
mRAWImageReader.get().getSurface(),
persistentRecorderSurface
), new CameraCaptureSession.StateCallback() {
#Override
public void onConfigured(CameraCaptureSession session) {
synchronized (mCameraStateLock) {
if (mBackCameraDevice == null) {
return;
}
try {
setup3AControlsLocked(mPreviewRequestBuilder);
session.setRepeatingRequest(mPreviewRequestBuilder.build(),
mPreCaptureCallback, mBackgroundHandler);
mState = CameraStates.PREVIEW;
} catch (CameraAccessException | IllegalStateException e) {
e.printStackTrace();
return;
}
mSession = session;
}
}
#Override
public void onConfigureFailed(CameraCaptureSession session) {
showToast("Failed to configure camera.");
}
}, mBackgroundHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}

It'd be helpful to see the system log lines right before that error line to confirm, but most likely:
You need to actually tie the persistentRecorderSurface to a MediaRecorder or MediaCodec, and call prepare() on those, before you create the camera capture session.
Otherwise, there's nothing actually at the other end of the persistent surface, and the camera can't tell what resolution or other settings are required.
Also keep in mind that there are limits on how many concurrent outputs you can have from the camera, depending on its supported hardware level and capabilities. There is currently no requirement that a device must support your combination of outputs (preview, record, JPEG, RAW), unfortunately, so it's very likely many or all devices will still give you an error.

Android Camera2, take picture continuously

I need to take pictures continuously with Camera2 API. It works fine on high end devices (for instance a Nexus 5X), but on slower ones (for instance a Samsung Galaxy A3), the preview freezes.
The code is a bit long, so I post only the most relevant parts:
Method called to start my preview:
private void startPreview() {
SurfaceTexture texture = mTextureView.getSurfaceTexture();
if(texture != null) {
try {
// We configure the size of default buffer to be the size of camera preview we want.
texture.setDefaultBufferSize(mPreviewSize.getWidth(), mPreviewSize.getHeight());
// This is the output Surface we need to start preview.
Surface surface = new Surface(texture);
// We set up a CaptureRequest.Builder with the output Surface.
mPreviewRequestBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
mPreviewRequestBuilder.addTarget(surface);
// Here, we create a CameraCaptureSession for camera preview.
mCameraDevice.createCaptureSession(Arrays.asList(surface, mImageReader.getSurface()), new CameraCaptureSession.StateCallback() {
#Override
public void onConfigured(#NonNull CameraCaptureSession cameraCaptureSession) {
// If the camera is already closed, return:
if (mCameraDevice == null) { return; }
// When the session is ready, we start displaying the preview.
mCaptureSession = cameraCaptureSession;
// Auto focus should be continuous for camera preview.
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE);
mPreviewRequest = mPreviewRequestBuilder.build();
// Start the preview
try { mCaptureSession.setRepeatingRequest(mPreviewRequest, null, mPreviewBackgroundHandler); }
catch (CameraAccessException e) { e.printStackTrace(); }
}
#Override
public void onConfigureFailed(#NonNull CameraCaptureSession cameraCaptureSession) {
Log.e(TAG, "Configure failed");
}
}, null
);
}
catch (CameraAccessException e) { e.printStackTrace(); }
}
}
Method called to take a picture:
private void takePicture() {
try {
CaptureRequest.Builder captureBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);
captureBuilder.addTarget(mImageReader.getSurface());
mCaptureSession.capture(captureBuilder.build(), null, mCaptureBackgroundHandler);
}
catch (CameraAccessException e) { e.printStackTrace(); }
}
And here is my ImageReader:
private final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(final ImageReader reader) {
mSaveBackgroundHandler.post(new Runnable() {
#Override
public void run() {
// Set the destination file:
File destination = new File(getExternalFilesDir(null), "image_" + mNumberOfImages + ".jpg");
mNumberOfImages++;
// Acquire the latest image:
Image image = reader.acquireLatestImage();
// Save the image:
ByteBuffer buffer = image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.remaining()];
buffer.get(bytes);
FileOutputStream output = null;
try {
output = new FileOutputStream(destination);
output.write(bytes);
}
catch (IOException e) { e.printStackTrace(); }
finally {
image.close();
if (null != output) {
try { output.close(); }
catch (IOException e) { e.printStackTrace(); }
}
}
// Take a new picture if needed:
if(mIsTakingPictures) {
takePicture();
}
}
});
}
};
I have a button that toggle the mIsTakingPictures boolean, and makes the first takePicture call.
To recap, I'm using 3 threads:
one for the preview
one for the capture
one for the image saving
What can be the cause of this freeze?

It's impossible to avoid framing lost in your preview when you are taking images all time on weak devices. The only way to avoid this is on devices which support TEMPLATE_ZERO_SHUTTER_LAG and using a reprocessableCaptureSession. The documentation about this is pretty horrible and find a way to implement it can be a odyssey. I have this problem a few months ago and finally I found the way to implement it:
How to use a reprocessCaptureRequest with camera2 API
In that answer you can also find some Google CTS test's which also implements ReprocessableCaptureSession and shoot some burst captures with ZSL template.
Finally, you can also use a CaptureBuilder with your preview surface and the image reader surface attached, in that case your preview will continue working all time and also you will save each frame as a new picture. But you will still having the freeze problem.
I also tried implement a burst capture using a handler which dispatch a new capture call each 100 milliseconds, this second option was pretty good in performance and avoiding frame rate lost, but you will not get as many captures per second like the two ImageReader option.
Hope that my answer will help you a bit, API 2 still being a bit complex and there's not so many examples or information about it.

One thing I noticed on low end devices: the preview stops after a capture, even when using camera 1 api, so it has to be restarted manually, thus producing a small preview freeze when capturing a high resolution picture.
But the camera 2 api provides the possibility to get raw image when taking a still capture (that wasn't possible on the devices I have when using camera 1 (Huawei P7, Sony Xperia E5, wiko UFeel)). Using this feature is much faster than capturing a JPEG (maybe due to JPEG compression), so the preview can be restarted earlier, and the preview freeze is shorter. Of course using this solution you'll have to convert the picture from YUV to JPEG in a background task..

Affdex Android SDK - Save and use CameraDetector

Mostly I want to know if there is a fundamental conflict that I can't share the same resource with the library, if so, I will need to take a different approach.
My goal is to have low quality video with the detector's meta data saved at the same time, so that I can do some post processing and slicing without much of a delay.
Based on the CameraDetectorDemo - camera detector
I have been initializing a MediaRecorder, but it saves a black screen if I start it before the detector, and it crashes on start (with code -19) if I start it after the detector. The detector is attaching the preview, maybe it is to do with that.
I added some buttons to control these functions:
protected void cameraInit() {
String state = Environment.getExternalStorageState();
if (!Environment.MEDIA_MOUNTED.equals(state)) {
Log.d(LOG_TAG, "Drive not mounted - cannot write video");
return;
}
File file = new File(getExternalFilesDir(Environment.DIRECTORY_MOVIES), "demo.gp3");
Log.d(LOG_TAG, String.format("Camera Initializing. Setting output to: %s", file.getAbsolutePath()));
// Set sources
recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
recorder.setVideoSource(MediaRecorder.VideoSource.CAMERA);
// Set profile
recorder.setProfile(CamcorderProfile.get(CamcorderProfile.QUALITY_LOW));
// Set output profile
recorder.setOutputFile(file.getAbsolutePath());
// Set preview output
recorder.setPreviewDisplay(cameraPreview.getHolder().getSurface());
try {
this.recorder.prepare();
} catch (IOException e) {
Log.e(LOG_TAG, "IO exception on camera Initialization");
e.printStackTrace();
} catch (IllegalStateException e) {
// This is thrown if the previous calls are not called with the
// proper order
Log.e(LOG_TAG, "Failed to initialize things properly :( ");
e.printStackTrace();
}
}
protected void cameraStart() {
Log.d(LOG_TAG, "Camera Start");
this.recorder.start();
}
protected void cameraStop() {
Log.d(LOG_TAG, "Camera Stop");
this.recorder.stop();
}

The Affdex SDK's CameraDetector needs access to the camera to get its preview frames and process them, so that's not going to work if the MediaRecorder has control of the camera.
Probably your best bet is to take preview frames from the camera, feed them to an Affdex FrameDetector for processing, and also save them to a video file via a MediaCodec and MediaMuxer, although I haven't tried that.

onFaceDetection(Camera.Face[] faces, Camera camera) keeps on executing continuously

I am creating an application capture image as it detect the face and I am able to achieve that, but only one issue the OnFaceDetection function of FaceDetectionListener is keep on executing even if there is no face in-front of camera. I am pasting my code.
mCamera.setFaceDetectionListener(new Camera.FaceDetectionListener() {
#Override
public void onFaceDetection(Camera.Face[] faces, Camera camera) {
try {
if (lastCaptureTiume + 10000 <= System.currentTimeMillis() || !faceCaptured) {
mCamera.takePicture(null, null, jpegCallback);
lastCaptureTiume = System.currentTimeMillis();
faceCaptured = true;
}
}
catch (Exception e) {
}
}
});
The issue is it is keep on takeing picture although there is no face in front of camera.

This behavior is different for different devices For my Note 3 onFaceDetection keeps on executing even without face and for nexus phone its performing perfectly.
Well I didn't got any other solution so I put a face check condition.
if (faces != null && faces.length > 0) {
//Do code here
}

how to add another byte array into onpreviewframe's byte array in surfaceview in android

Actually i need to do simple task
add logo into video created by surface View
First i did that convert logo image to byte array and successfully able to do so
in next phase need to add logo byte array to OnPreviewFrame method's parameter Byte[] xxx
how can i add these two byte array
is there any way for video recording using surface view with my static application logo on corner of video.
following my code:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
Camera.Parameters parameters = camera.getParameters();
if (yuvIplimage != null && recording) {
if (nextFrametime == -1) {
nextFrametime = startTime;
}
if (System.currentTimeMillis() >= nextFrametime) {
videoTimestamp = 1000 * (System.currentTimeMillis() - startTime);
// Put the camera preview frame right into the yuvIplimage
// object
yuvIplimage.getByteBuffer().put(data);
try {
// Get the correct time
recorder.setTimestamp(videoTimestamp);
// Record the image into FFmpegFrameRecorder
recorder.record(yuvIplimage);
recordError = false;
} catch (FFmpegFrameRecorder.Exception e) {
recordError = true;
nextFrametime = -1;
frameCount = 0;
Log.v(LOG_TAG, e.getMessage());
e.printStackTrace();
}
nextFrametime = startTime
+ (long) ((1000.0 / 15) * (++frameCount));
}
}
}

You probably want to add an overlay on top of the preview frames, kind of a watermark for the video you produce. You will get best results if you handle this with ffmpeg on native level, but for JavaCV, see the tutorial.