I am working on a video recording app in which I want to record videos in portrait. Everything seems fine except for the video which is saved in landscape mode. I tried the implementation using this project: https://github.com/HofmaDresu/AndroidCamera2Sample as an example, but still, the video is being saved in landscape mode.
void PrepareMediaRecorder()
{
if (mediaRecorder == null)
{
mediaRecorder = new MediaRecorder();
}
else
{
mediaRecorder.Reset();
}
var map = (StreamConfigurationMap)characteristics.Get(CameraCharacteristics.ScalerStreamConfigurationMap);
if (map == null)
{
return;
}
videoFileName = GetVideoFilePath();
mediaRecorder.SetAudioSource(AudioSource.Mic);
mediaRecorder.SetVideoSource(VideoSource.Surface);
mediaRecorder.SetOutputFormat(OutputFormat.Mpeg4);
mediaRecorder.SetOutputFile(videoFileName);
mediaRecorder.SetVideoEncodingBitRate(10000000);
mediaRecorder.SetVideoFrameRate(30);
var videoSize = ChooseVideoSize(map.GetOutputSizes(Java.Lang.Class.FromType(typeof(MediaRecorder))));
mediaRecorder.SetVideoEncoder(VideoEncoder.H264);
mediaRecorder.SetAudioEncoder(AudioEncoder.Aac);
mediaRecorder.SetVideoSize(videoSize.Width, videoSize.Height);
int rotation = (int)Activity.WindowManager.DefaultDisplay.Rotation;
mediaRecorder.SetOrientationHint(GetOrientation(rotation));
mediaRecorder.Prepare();
}
Assuming a high-quality video player shows you the video in portrait (if not, your GetOrientation method probably has an error in it), but other players you still care about are stuck on landscape:
You'll have to rotate the frames yourself. Unfortunately, this is messy, since there's no automatic control for this on the media encoder APIs that I know of.
Options are receiving frames via an ImageReader from the camera, and then doing the rotation in Java or via JNI, in native code, and then sending the frame to the encoder either via an ImageWriter to a MediaRecorder or MediaCodec Surface, or writing frames via MediaCodec's ByteBuffer interface.
Or you could send the frames to the GPU via a SurfaceTexture, rotate in a fragment shader, and then write out to a Surface tied to a MediaRecorder/MediaCodec again.
Both of these require a lot of boilerplate code and understanding of lower-level details, unfortunately.
Related
I am new to Camera2 framework and trying to understand the logic of creation of capture sessions.
I need a simple thing - preview and record video. I also want to set the correct orientation hint at the time I start recording the video. But I came to a chicken/egg problem.
Here is my logic:
In order to start recording, I am doing this:
val recordRequest = session.device.createCaptureRequest(CameraDevice.TEMPLATE_RECORD).apply {
// Add the preview and recording surface targets
addTarget(viewFinder.holder.surface)
addTarget(recorder.surface)
}.build()
session.setRepeatingRequest(recordRequest, null, cameraHandler)
recorder.setOrientationHint(it) // NOT allowed after getSurface()!
recorder.prepare() // NOT allowed after getSurface()!
recorder.start()
However, I already called recorder.surface (or getSurface()) when I added targets above. One can think that I can prepare and then add targets, however, the documentation for addTarget() says, that the surface The Surface added must be one of the surfaces included in the most recent call to CameraDevice#createCaptureSession
That leads to an interesting problem. Whenever I open the app, I need to create the capture session to start previewing camera image. However, at the point of creation the createCaptureSession() needs to include all the surfaces that will come in future capture requests. Which means that I also need to include the recording surface, even if I simply open camera without recording yet. How do I get this Surface for recording? Well, the documentation says I can get it from MediaRecorder or I can get it from MediaCodec. I want to get it from MediaRecorder since I want to use CamcorderProfiles. However, as I showed in the above code, once I get the surface from the recorder at the point of session creation - I cannot do any changes there at the point of starting recording, like setting orientation hint.
The official Camera2Video sample app does a trick - it uses createPersistentInputSurface however in their example the camera is fixed, which allows them to allocate enough memory for it and use that surface throughout the app lifecycle.
How can this be solved? Am I misunderstanding the concepts here? How can I create the recorder at a later point, when I start recording, but still have the surface for it created earlier, when I open the camera for preview?
Using a persistent input surface is the right approach. Create a new MediaRecorder once you know the orientation for recording, and set its Surface using the persistent input surface.
That's exactly what the Camera2Video sample does, as well:
// React to user touching the capture button
capture_button.setOnTouchListener { view, event ->
when (event.action) {
MotionEvent.ACTION_DOWN -> lifecycleScope.launch(Dispatchers.IO) {
// Prevents screen rotation during the video recording
requireActivity().requestedOrientation =
ActivityInfo.SCREEN_ORIENTATION_LOCKED
// Start recording repeating requests, which will stop the ongoing preview
// repeating requests without having to explicitly call `session.stopRepeating`
session.setRepeatingRequest(recordRequest, null, cameraHandler)
// Finalizes recorder setup and starts recording
recorder.apply {
// Sets output orientation based on current sensor value at start time
relativeOrientation.value?.let { setOrientationHint(it) }
prepare()
start()
}
and recorder is created with an earlier-created persistent surface:
/** Saves the video recording */
private val recorder: MediaRecorder by lazy { createRecorder(recorderSurface)
}
When you say the camera is fixed, do you mean the app orientation being fixed, or that the sample doesn't support switching front/back cameras? None of that should particularly matter for persistent surfaces; you can create a new one when you switch cameras or change orientations, if you need to.
My first experience with background video recording on Android was with JavaCV FFMpegRecoder. It's easy to implement, just create camera instance in activity, set PreviewCallback listener in the service, on onPreviewFrame just send byte to FFMpegRecorder and don't destroy (disconnect) camera in onPause or onStop of course
But FFMpegRecorder isn't that good (cpu, memory usage)
So I found INDExOS m4m library (by Intel): https://github.com/INDExOS/media-for-mobile
It has CameraCapturerActivity.java - https://github.com/INDExOS/media-for-mobile/blob/master/samples/src/main/java/org/m4m/samples/CameraCapturerActivity.java
Seems it really doesn't eat many resources
I decided to try recording in background mode, I just simply commented its onPause method where stop recording and preview methods are executed, but it just doesn't record anything (freezes on the last frame) until I return to activity
When I set PreviewCallback listener to this class, onPreviewFrame sends byte in background ok, seems onFrameAvailable of SurfaceTexture related to delivering frames in m4m library are stopped when onPause is called from Activity
library has two onFrameAvailable listeners:
first in PreviewRender.java - https://github.com/INDExOS/media-for-mobile/blob/master/android/src/main/java/org/m4m/android/PreviewRender.java#L241, seems class contains everything that related to displaying frames in view class (so should not be important for recording video)
second in CameraSource.java - https://github.com/INDExOS/media-for-mobile/blob/master/android/src/main/java/org/m4m/android/CameraSource.java#L222
seems this is the main class that gets frames, and I guess it is used for video recording
But also it seems those classes are still related in quite things
For example if I comment createPreview method in CameraCapturerActivity.java
private void createPreview() {
surfaceView = new GLSurfaceView(getApplicationContext());
surfaceView.setDebugFlags(GLSurfaceView.DEBUG_CHECK_GL_ERROR);
((RelativeLayout) findViewById(R.id.camera_layout)).addView(surfaceView, 0);
preview = capture.createPreview(surfaceView, camera);
preview.setFillMode(fillMode);
if (getRequestedOrientation() == ActivityInfo.SCREEN_ORIENTATION_PORTRAIT) {
capture.setOrientation(90);
} else if (getRequestedOrientation() == ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE) {
capture.setOrientation(0);
}
preview.start();
}
app will run ok, of course I won't see frames, but recording won't be working when trying to press the record button, so recording doesn't work if preview wasn't created
So I need help to understand how all of this works, how could I pause preview but continue recording in background when I leave activity and resume preview when returning to activity again. I didn't work with SurfaceTexture, GLSurfaceView, only worked with ordinary SurfaceView and its holder callbacks onSurfaceCreated, onSurfaceChanged,..
I just don't see in the project something similar to onSurfaceDestroy which would stop recording when user leaves activity
I see OpenGl API, textures are also used in Grafika project https://github.com/google/grafika
So I believe there are people who worked with something like this and could know how SurfaceTexture and its callbacks (onFrameAvailable,..) works
Of course many things are related to m4m library code design itself, but still hard to understand something when you didn't work with all of this (opengl, surfacetexture,...)
UPDATE
Now I know a little about EGLContext, that we have to set it to a specific source (for preview or for recording - encoder)
I succeeded to make Grafika recording example working in background https://github.com/google/grafika/blob/master/src/com/android/grafika/ContinuousCaptureActivity.java
In that sample class I commented everything in onPause, removed mDisplaySurface and did other things
And onFrameAvailable looks like this now:
#Override // SurfaceTexture.OnFrameAvailableListener; runs on arbitrary thread
public void onFrameAvailable(SurfaceTexture surfaceTexture) {
Log.d(TAG, "frame available");
if (mEglCore == null) {
return;
}
mEncoderSurface.makeCurrent();
mCameraTexture.updateTexImage();
mCameraTexture.getTransformMatrix(mTmpMatrix);
if (!mFileSaveInProgress) {
GLES20.glViewport(0, 0, VIDEO_WIDTH, VIDEO_HEIGHT);
mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
drawExtra(mFrameNum, VIDEO_WIDTH, VIDEO_HEIGHT);
mCircEncoder.frameAvailableSoon();
mEncoderSurface.setPresentationTime(mCameraTexture.getTimestamp());
mEncoderSurface.swapBuffers();
}
mFrameNum++;
//mHandler.sendEmptyMessage(MainHandler.MSG_FRAME_AVAILABLE);
}
So now when I press home button, it still records frames to a file I can see it later
Now I need to get back to m4m library, cause it record audio and has utils for frame processing
In Grafika, all the videos are recorded in internal storage.
Try to change the output path of video to see it in other player. Something like that:
File outputFile = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS);
boolean isPresent = true;
if (!outputFile.exists()) {
isPresent = outputFile.mkdir();
}
if (isPresent) {
outputFile = new File(outputFile.getAbsolutePath(),"camera-test.mp4");
} else {
// Failure
}
You can change CameraCaptureActivity (Show + camera capture) example to perform what you want. Comment OnPause method and change openCamera like that:
private void openCamera(int desiredWidth, int desiredHeight) {
if (mCamera != null) {
return;
}
This is coded in NativeScript, so I'll try my best to adapt the scenario to Java. I have created an in-app video view with support to record the video.
This is done as follows:
First I create a SurfaceView that will hold the preview of the camera:
this.mSurfaceView = new android.view.SurfaceView(this._context);
this.mHolder = this.mSurfaceView.getHolder();
this.mHolder.setType(android.view.SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
Then I create an instance of the Camera, and sets the video surface:
var mCamera = android.hardware.Camera;
var camera = mCamera.open(1);
this.camera = camera;
this.camera.setDisplayOrientation(90);
var parameters = camera.getParameters();
parameters.setRecordingHint(true);
if( parameters.isVideoStabilizationSupported() ){
parameters.setVideoStabilization(true);
}
camera.setParameters(parameters);
this.camera.setPreviewDisplay(_this.mHolder);
this.camera.startPreview();
this.camera.startFaceDetection();
Now, all is good. I have the camera preview in the view that I want it to be. The color is good and I think the image aspect ratio is good too.
However, when I initiate the recording, as I do with the following code:
this.mediarecorder = new android.media.MediaRecorder();
// Step 1: Unlock and set camera to MediaRecorder
this.camera.unlock();
this.mediarecorder.setCamera(this.camera);
// Step 2: Set sources
this.mediarecorder.setAudioSource(android.media.MediaRecorder.AudioSource.CAMCORDER);
this.mediarecorder.setVideoSource(android.media.MediaRecorder.VideoSource.CAMERA);
//this.mediarecorder.setOutputFormat(android.media.MediaRecorder.OutputFormat.MPEG_4);
// Step 3: Set a CamcorderProfile (requires API Level 8 or higher)
this.mediarecorder.setProfile(android.media.CamcorderProfile.get(android.media.CamcorderProfile.QUALITY_HIGH));
// platform.screen.mainScreen.widthDIPs
// platform.screen.mainScreen.heightDIPs
// Step 4: Set output file
var fileName = "videoCapture_" + new Date() + ".mp4";
var path = android.os.Environment.getExternalStoragePublicDirectory(android.os.Environment.DIRECTORY_DCIM).getAbsolutePath() + "/Camera/" + fileName;
this.file = new java.io.File(path);
this.mediarecorder.setOutputFile(this.file.toString());
this.mediarecorder.setOrientationHint(270);
try {
this.mediarecorder.prepare();
this.mediarecorder.start();
} catch( ex ) {
console.log(ex);
}
Then, the image suddenly becomes darker, and my face (its what's in focus when I'm trying it out) gets wider. So the aspect ratio changes, and so does the lighting somehow.
I have tried setting setPictureSize on the camera parameters, and setVideoSize on the MediaRecorder with no luck. And for the lighting change, I have simply no clue as to whats going on. Now I've been googling myself half way to heaven, and still found nothing, so I hope someone here has got any tip on what to pursue next?
Video recording generally tries to run at a steady frame rate, such as 30fps. Camera preview often slows itself down to 10-15fps to maintain brightness, so if you're in a darker location, video recording will be darker (since it can't expose for longer than 1/30s instead of 1/10s that camera preview can).
Did you call setVideoSize before or after calling setProfile? The setProfile call changes many parameters, including preview size; most video recording sizes are 16:9, and the default camera preview resolution is likely a 4:3 size. So when you start the recording, the aspect ratio switches.
Most video recording apps use 16:9 preview sizes even before starting recording so that they're consistent. You can also record 4:3 video, but that's generally not what people want to see.
I am having a headache over the Camera API 1 for android. After reading all of the Internet content, I made some sample app that works OK. It creates a service, which then is used to operate with the camera in the background, so there is no preview or activity enabled. To achieve this I use a dummy SurfaceHolder, like this:
protected class MySurfaceHolder implements SurfaceHolder {
private final Surface surface;
private final SurfaceTexture surfaceTexture;
public MySurfaceHolder () {
int[] textures = new int[1];
GLES20.glGenTextures(1, textures, 0);
if (textures.length > 0) {
this.surfaceTexture = new SurfaceTexture(textures[0]);
this.surface = new Surface(this.surfaceTexture);
} else {
this.surface = null;
this.surfaceTexture = null;
}
}
[...]
}
and then I use it like this
// simplified version of my code
try {
initializeCamera(); // open camera and set Camera.Parameters
camera.setPreviewDisplay(new MySurfaceHolder());
camera.startPreview();
camera.unlock();
initializeMediaRecorder(); // create MediaRecorder, set video/audio parameters
mediaRecorder.prepare();
mediaRecorder.start();
// wait until recording finish and exit
} finally {
stopRecording();
}
the Camera and MediaRecorder initialization methods are just like the documentation states they should be (and they work).
Everything works and operates as it should. Almost everything - sometimes, under unknown circumstances the MediaRecorder creates empty files, like 32kB containing only headers and info about the video - no frames. The longer I record like this, the bigger is the file (few kB every few seconds). After 1 minute, the file weights about 80kB. Funny thing is I know that the camera is working and capturing frames (I debugged it a little showing preview frames), but the frames are not written into the output file.
Also when it happens I am not able to record in FHD (1920x1080) - I get the "start failed" message - at this time camera is not capturing frames. The same thing could happen when I use wrong (not supported) video size. I suppose in this case the message is thrown at the mediaRecorder.start(); line, and stopRecording(); is invoked but I am not sure.
After some time or after unknown action the problem is suddenly gone (I don't know when, I don't know how). It happens for sure on Android 5.1, but may happen on other versions as well.
Could this bug be related to my custom surface code?
What could cause the MediaRecorder to not write frames into a file?
Why I am not able to record in FHD, but in the same time I am able to record in HD (1280x720)?
Is there any alternative for MediaRecorder, so I can avoid these bugs?
May it happen when another app is trying to get Camera object, thus distrupting current recording? If so, how to regain access to the Camera object (I apparently am not able to do this now on some devices).
EDIT:
I think I might have a clue. I am calling
camera.setOneShotPreviewCallback(new Camera.PreviewCallback() {
// ... get current frame
}
camera.startPreview();
to get preview frame of current recording. It appears that the bug occurs when I am using this method to get preview frame (at random times). It seems flawed, because not all devices react to this thing properly (sometimes there is no preview frame...). Is there any other, better method of handling current preview frame without the real surface?
I'm using the MediaStore.ACTION_VIDEO_CAPTURE intent to capture video and later play it back using a VideoView. I would like to know the orientation of the video that was captured.
I don't want to use the orientation at the time of the intent call because the user may rotate the device prior to hitting the shutter button. I also don't want to implement my own custom video capture.
Is the orientation of the video stored in the saved file and/or returned in the intent result?
Is the orientation of the video stored in the saved file and/or returned in the intent result?
The AOSP VideoCamera activity does supply the rotation value of the camera device to the MediaRecorder.setOrientationHint() method. Here's an excerpt of the code in the VideoCamera.initializeRecorder() code related to this:
// See android.hardware.Camera.Parameters.setRotation for
// documentation.
// Note that mOrientation here is the device orientation, which is
// the opposite of what getWindowManager().getDefaultDisplay().getRotation()
// would return, which is the orientation the graphics need to rotate
// in order to render correctly.
int rotation = 0;
if (mOrientation != OrientationEventListener.ORIENTATION_UNKNOWN) {
CameraInfo info = CameraHolder.instance().getCameraInfo()[mCameraId];
if (info.facing == CameraInfo.CAMERA_FACING_FRONT) {
rotation = (info.orientation - mOrientation + 360) % 360;
} else { // back-facing camera
rotation = (info.orientation + mOrientation) % 360;
}
}
mMediaRecorder.setOrientationHint(rotation);
Here's the documentation for setOrientationHint() explaining the storage of the information:
Sets the orientation hint for output video playback. This method
should be called before prepare(). This method will not trigger the
source video frame to rotate during video recording, but to add a
composition matrix containing the rotation angle in the output video
if the output format is OutputFormat.THREE_GPP or OutputFormat.MPEG_4
so that a video player can choose the proper orientation for playback.
Note that some video players may choose to ignore the compostion
matrix in a video during playback.
Since you are using the framework's VideoView widget to playback the video, it should already handle the information in the composition matrix correctly and you should only need to compare the width and height of the video to decide whether to set a landscape or portrait orientation for your playback activity. One simple way to do this would be to just call ThumbnailUtils.createVideoThumbnail() (which internally uses MediaMetaDataRetriever) and check the resolution of the returned bitmap.
Similar to how al. suggested in his answer, I use the MediaMetadataRetriever but with the METADATA_KEY_VIDEO_ROTATION key:
MediaMetadataRetriever mediaMetadataRetriever = new MediaMetadataRetriever();
mediaMetadataRetriever.setDataSource( absoluteVideoFilePath );
String orientation = mediaMetadataRetriever.extractMetadata(
MediaMetadataRetriever.METADATA_KEY_VIDEO_ROTATION );
I checked capturing video with all 4 different orientations of the front camera, and the metadata is always different.
Note: METADATA_KEY_VIDEO_ROTATION needs API Level 17 or above.
You can use MediaMetadataRetriever to get the width and height of your video and figure out the orientation prior to loading it.