Develop Reference

Android Camera2 API - Video always saved in landscape

I am working on a video recording app in which I want to record videos in portrait. Everything seems fine except for the video which is saved in landscape mode. I tried the implementation using this project: https://github.com/HofmaDresu/AndroidCamera2Sample as an example, but still, the video is being saved in landscape mode.
void PrepareMediaRecorder()
{
if (mediaRecorder == null)
{
mediaRecorder = new MediaRecorder();
}
else
{
mediaRecorder.Reset();
}
var map = (StreamConfigurationMap)characteristics.Get(CameraCharacteristics.ScalerStreamConfigurationMap);
if (map == null)
{
return;
}
videoFileName = GetVideoFilePath();
mediaRecorder.SetAudioSource(AudioSource.Mic);
mediaRecorder.SetVideoSource(VideoSource.Surface);
mediaRecorder.SetOutputFormat(OutputFormat.Mpeg4);
mediaRecorder.SetOutputFile(videoFileName);
mediaRecorder.SetVideoEncodingBitRate(10000000);
mediaRecorder.SetVideoFrameRate(30);
var videoSize = ChooseVideoSize(map.GetOutputSizes(Java.Lang.Class.FromType(typeof(MediaRecorder))));
mediaRecorder.SetVideoEncoder(VideoEncoder.H264);
mediaRecorder.SetAudioEncoder(AudioEncoder.Aac);
mediaRecorder.SetVideoSize(videoSize.Width, videoSize.Height);
int rotation = (int)Activity.WindowManager.DefaultDisplay.Rotation;
mediaRecorder.SetOrientationHint(GetOrientation(rotation));
mediaRecorder.Prepare();
}

Assuming a high-quality video player shows you the video in portrait (if not, your GetOrientation method probably has an error in it), but other players you still care about are stuck on landscape:
You'll have to rotate the frames yourself. Unfortunately, this is messy, since there's no automatic control for this on the media encoder APIs that I know of.
Options are receiving frames via an ImageReader from the camera, and then doing the rotation in Java or via JNI, in native code, and then sending the frame to the encoder either via an ImageWriter to a MediaRecorder or MediaCodec Surface, or writing frames via MediaCodec's ByteBuffer interface.
Or you could send the frames to the GPU via a SurfaceTexture, rotate in a fragment shader, and then write out to a Surface tied to a MediaRecorder/MediaCodec again.
Both of these require a lot of boilerplate code and understanding of lower-level details, unfortunately.

Android Camera2 session creation logic

I am new to Camera2 framework and trying to understand the logic of creation of capture sessions.
I need a simple thing - preview and record video. I also want to set the correct orientation hint at the time I start recording the video. But I came to a chicken/egg problem.
Here is my logic:
In order to start recording, I am doing this:
val recordRequest = session.device.createCaptureRequest(CameraDevice.TEMPLATE_RECORD).apply {
// Add the preview and recording surface targets
addTarget(viewFinder.holder.surface)
addTarget(recorder.surface)
}.build()
session.setRepeatingRequest(recordRequest, null, cameraHandler)
recorder.setOrientationHint(it) // NOT allowed after getSurface()!
recorder.prepare() // NOT allowed after getSurface()!
recorder.start()
However, I already called recorder.surface (or getSurface()) when I added targets above. One can think that I can prepare and then add targets, however, the documentation for addTarget() says, that the surface The Surface added must be one of the surfaces included in the most recent call to CameraDevice#createCaptureSession
That leads to an interesting problem. Whenever I open the app, I need to create the capture session to start previewing camera image. However, at the point of creation the createCaptureSession() needs to include all the surfaces that will come in future capture requests. Which means that I also need to include the recording surface, even if I simply open camera without recording yet. How do I get this Surface for recording? Well, the documentation says I can get it from MediaRecorder or I can get it from MediaCodec. I want to get it from MediaRecorder since I want to use CamcorderProfiles. However, as I showed in the above code, once I get the surface from the recorder at the point of session creation - I cannot do any changes there at the point of starting recording, like setting orientation hint.
The official Camera2Video sample app does a trick - it uses createPersistentInputSurface however in their example the camera is fixed, which allows them to allocate enough memory for it and use that surface throughout the app lifecycle.
How can this be solved? Am I misunderstanding the concepts here? How can I create the recorder at a later point, when I start recording, but still have the surface for it created earlier, when I open the camera for preview?

Using a persistent input surface is the right approach. Create a new MediaRecorder once you know the orientation for recording, and set its Surface using the persistent input surface.
That's exactly what the Camera2Video sample does, as well:
// React to user touching the capture button
capture_button.setOnTouchListener { view, event ->
when (event.action) {
MotionEvent.ACTION_DOWN -> lifecycleScope.launch(Dispatchers.IO) {
// Prevents screen rotation during the video recording
requireActivity().requestedOrientation =
ActivityInfo.SCREEN_ORIENTATION_LOCKED
// Start recording repeating requests, which will stop the ongoing preview
// repeating requests without having to explicitly call `session.stopRepeating`
session.setRepeatingRequest(recordRequest, null, cameraHandler)
// Finalizes recorder setup and starts recording
recorder.apply {
// Sets output orientation based on current sensor value at start time
relativeOrientation.value?.let { setOrientationHint(it) }
prepare()
start()
}
and recorder is created with an earlier-created persistent surface:
/** Saves the video recording */
private val recorder: MediaRecorder by lazy { createRecorder(recorderSurface)
}
When you say the camera is fixed, do you mean the app orientation being fixed, or that the sample doesn't support switching front/back cameras? None of that should particularly matter for persistent surfaces; you can create a new one when you switch cameras or change orientations, if you need to.

Android MediaRecorder saving empty file and weird camera behaviours

I am having a headache over the Camera API 1 for android. After reading all of the Internet content, I made some sample app that works OK. It creates a service, which then is used to operate with the camera in the background, so there is no preview or activity enabled. To achieve this I use a dummy SurfaceHolder, like this:
protected class MySurfaceHolder implements SurfaceHolder {
private final Surface surface;
private final SurfaceTexture surfaceTexture;
public MySurfaceHolder () {
int[] textures = new int[1];
GLES20.glGenTextures(1, textures, 0);
if (textures.length > 0) {
this.surfaceTexture = new SurfaceTexture(textures[0]);
this.surface = new Surface(this.surfaceTexture);
} else {
this.surface = null;
this.surfaceTexture = null;
}
}
[...]
}
and then I use it like this
// simplified version of my code
try {
initializeCamera(); // open camera and set Camera.Parameters
camera.setPreviewDisplay(new MySurfaceHolder());
camera.startPreview();
camera.unlock();
initializeMediaRecorder(); // create MediaRecorder, set video/audio parameters
mediaRecorder.prepare();
mediaRecorder.start();
// wait until recording finish and exit
} finally {
stopRecording();
}
the Camera and MediaRecorder initialization methods are just like the documentation states they should be (and they work).
Everything works and operates as it should. Almost everything - sometimes, under unknown circumstances the MediaRecorder creates empty files, like 32kB containing only headers and info about the video - no frames. The longer I record like this, the bigger is the file (few kB every few seconds). After 1 minute, the file weights about 80kB. Funny thing is I know that the camera is working and capturing frames (I debugged it a little showing preview frames), but the frames are not written into the output file.
Also when it happens I am not able to record in FHD (1920x1080) - I get the "start failed" message - at this time camera is not capturing frames. The same thing could happen when I use wrong (not supported) video size. I suppose in this case the message is thrown at the mediaRecorder.start(); line, and stopRecording(); is invoked but I am not sure.
After some time or after unknown action the problem is suddenly gone (I don't know when, I don't know how). It happens for sure on Android 5.1, but may happen on other versions as well.
Could this bug be related to my custom surface code?
What could cause the MediaRecorder to not write frames into a file?
Why I am not able to record in FHD, but in the same time I am able to record in HD (1280x720)?
Is there any alternative for MediaRecorder, so I can avoid these bugs?
May it happen when another app is trying to get Camera object, thus distrupting current recording? If so, how to regain access to the Camera object (I apparently am not able to do this now on some devices).
EDIT:
I think I might have a clue. I am calling
camera.setOneShotPreviewCallback(new Camera.PreviewCallback() {
// ... get current frame
}
camera.startPreview();
to get preview frame of current recording. It appears that the bug occurs when I am using this method to get preview frame (at random times). It seems flawed, because not all devices react to this thing properly (sometimes there is no preview frame...). Is there any other, better method of handling current preview frame without the real surface?

AudioRecord in Android live wallpaper

I am working on a live wallpaper that uses a AudioRecord object to listen to the phone's mic. This works fine until I start using the wallpaper and then also view the preview wallpaper in the wallpaper selection menu. When this happens, I start having problems starting and stopping the AudioRecord object. Currently this is handled by onVisibilityChanged:
#Override
public void onVisibilityChanged(boolean visible) {
if (this.visible == visible)
return;
this.visible = visible;
if (visible) {
recorder.startRecording();
handler.post(drawRunner);
} else {
recorder.stop();
handler.removeCallbacks(drawRunner);
}
}
Is there some solution to this? I have tried several things, but nothing has worked out. This solution does (rarely) work, and as far as I can tell this is because the two instances of the wallpaper engine update their visibility in an unpredictable order. Sometimes the main wallpaper sets itself to not be visible before the preview sets itself to be visible, and everything is fine. Sometimes this order is reversed and everything breaks.

So, I found an answer and it was stupidly simple. I swear I tried this already and it didn't work, but everything is working fine now. All I had to do was move the AudioRecord object outside of the engine class and into the service class. I then create the AudioRecord object in onCreateEngine only if it isn't already instantiated. That way, all engine instances use the same AudioRecord object.

SurfaceView getting black when calling Camera.stopPreview()

I am doing some image processing of previews captured by the camera. It is cpu consuming task and I have to stop preview to make it faster. Before new frame is being processed I am calling Camera.stopPreview() and after Camera.startPreview().
However, I would like to have last captured frame displayed on SurfaceView after stopping the preview. It works 'out of the box' on 2.3 devices, however, SurfaceView gets black after calling Camera.stopPreview() on older versions of SDK. Does anyone know what has changed and what to do?

Yes, this was an improvement of the 2.3.
I had this problem in 2.2 as well, there was no way to work on a preview image despite the fact this was theoretically possible according to the API. To solve this I had to actually take a picture using Camera.takePicture(null, null, Camera.PictureCallback myCallback) (see info here) and then to implement a callback to handle the taken picture. The instance of the class that implements this callback is actually the parameter to pass to Camera.takePicture() and the callback method itself looks like this:
public void onPictureTaken(byte[] JPEGData, Camera camera) {
final Bitmap bitmap = createBitmapFromView(JPEGData);
// do something with the Bitmap
}
Doing that way prevents the picture to be saved on the external storage with the regular pictures taken with the camera application. Should you need to serialize the Bitmap you'll have to do it explicitely. But it doesn't prevent the camera's trigger sound from being emitted.
Camera.takePicture() has to be called wile the preview is running. stopPreview() can be called right after.
One thing to be careful with /!\:
Camera.takePicture() is not reentrant (at all). The callback must have returned before any subsequent call of Camera.takePicture(). This was freezing my phone, I had to shutdown and restart it before it to be usable again. As the action was triggered by a button, on my side, I had to shield it with a boolean:
if (!mPictureTaken) {
mPictureTaken = true; // absolutely NOT reentrant. Any double click sticks the phone otherwise.
mCameraView.takePicture(callback);
}