Develop Reference

Continue recording video in background - SurfaceTexture, GLSurfaceView

My first experience with background video recording on Android was with JavaCV FFMpegRecoder. It's easy to implement, just create camera instance in activity, set PreviewCallback listener in the service, on onPreviewFrame just send byte to FFMpegRecorder and don't destroy (disconnect) camera in onPause or onStop of course
But FFMpegRecorder isn't that good (cpu, memory usage)
So I found INDExOS m4m library (by Intel): https://github.com/INDExOS/media-for-mobile
It has CameraCapturerActivity.java - https://github.com/INDExOS/media-for-mobile/blob/master/samples/src/main/java/org/m4m/samples/CameraCapturerActivity.java
Seems it really doesn't eat many resources
I decided to try recording in background mode, I just simply commented its onPause method where stop recording and preview methods are executed, but it just doesn't record anything (freezes on the last frame) until I return to activity
When I set PreviewCallback listener to this class, onPreviewFrame sends byte in background ok, seems onFrameAvailable of SurfaceTexture related to delivering frames in m4m library are stopped when onPause is called from Activity
library has two onFrameAvailable listeners:
first in PreviewRender.java - https://github.com/INDExOS/media-for-mobile/blob/master/android/src/main/java/org/m4m/android/PreviewRender.java#L241, seems class contains everything that related to displaying frames in view class (so should not be important for recording video)
second in CameraSource.java - https://github.com/INDExOS/media-for-mobile/blob/master/android/src/main/java/org/m4m/android/CameraSource.java#L222
seems this is the main class that gets frames, and I guess it is used for video recording
But also it seems those classes are still related in quite things
For example if I comment createPreview method in CameraCapturerActivity.java
private void createPreview() {
surfaceView = new GLSurfaceView(getApplicationContext());
surfaceView.setDebugFlags(GLSurfaceView.DEBUG_CHECK_GL_ERROR);
((RelativeLayout) findViewById(R.id.camera_layout)).addView(surfaceView, 0);
preview = capture.createPreview(surfaceView, camera);
preview.setFillMode(fillMode);
if (getRequestedOrientation() == ActivityInfo.SCREEN_ORIENTATION_PORTRAIT) {
capture.setOrientation(90);
} else if (getRequestedOrientation() == ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE) {
capture.setOrientation(0);
}
preview.start();
}
app will run ok, of course I won't see frames, but recording won't be working when trying to press the record button, so recording doesn't work if preview wasn't created
So I need help to understand how all of this works, how could I pause preview but continue recording in background when I leave activity and resume preview when returning to activity again. I didn't work with SurfaceTexture, GLSurfaceView, only worked with ordinary SurfaceView and its holder callbacks onSurfaceCreated, onSurfaceChanged,..
I just don't see in the project something similar to onSurfaceDestroy which would stop recording when user leaves activity
I see OpenGl API, textures are also used in Grafika project https://github.com/google/grafika
So I believe there are people who worked with something like this and could know how SurfaceTexture and its callbacks (onFrameAvailable,..) works
Of course many things are related to m4m library code design itself, but still hard to understand something when you didn't work with all of this (opengl, surfacetexture,...)
UPDATE
Now I know a little about EGLContext, that we have to set it to a specific source (for preview or for recording - encoder)
I succeeded to make Grafika recording example working in background https://github.com/google/grafika/blob/master/src/com/android/grafika/ContinuousCaptureActivity.java
In that sample class I commented everything in onPause, removed mDisplaySurface and did other things
And onFrameAvailable looks like this now:
#Override // SurfaceTexture.OnFrameAvailableListener; runs on arbitrary thread
public void onFrameAvailable(SurfaceTexture surfaceTexture) {
Log.d(TAG, "frame available");
if (mEglCore == null) {
return;
}
mEncoderSurface.makeCurrent();
mCameraTexture.updateTexImage();
mCameraTexture.getTransformMatrix(mTmpMatrix);
if (!mFileSaveInProgress) {
GLES20.glViewport(0, 0, VIDEO_WIDTH, VIDEO_HEIGHT);
mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
drawExtra(mFrameNum, VIDEO_WIDTH, VIDEO_HEIGHT);
mCircEncoder.frameAvailableSoon();
mEncoderSurface.setPresentationTime(mCameraTexture.getTimestamp());
mEncoderSurface.swapBuffers();
}
mFrameNum++;
//mHandler.sendEmptyMessage(MainHandler.MSG_FRAME_AVAILABLE);
}
So now when I press home button, it still records frames to a file I can see it later
Now I need to get back to m4m library, cause it record audio and has utils for frame processing

In Grafika, all the videos are recorded in internal storage.
Try to change the output path of video to see it in other player. Something like that:
File outputFile = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS);
boolean isPresent = true;
if (!outputFile.exists()) {
isPresent = outputFile.mkdir();
}
if (isPresent) {
outputFile = new File(outputFile.getAbsolutePath(),"camera-test.mp4");
} else {
// Failure
}
You can change CameraCaptureActivity (Show + camera capture) example to perform what you want. Comment OnPause method and change openCamera like that:
private void openCamera(int desiredWidth, int desiredHeight) {
if (mCamera != null) {
return;
}

Android Camera2 take picture while processing frames

I am using Camera2 API to create a Camera component that can scan barcodes and has ability to take pictures during scanning. It is kinda working but the preview is flickering - it seems like previous frames and sometimes green frames are interrupting realtime preview.
My code is based on Google's Camera2Basic. I'm just adding one more ImageReader and its surface as a new output and target for CaptureRequest.Builder. One of the readers uses JPEG and the other YUV. Flickering disappears when I remove the JPEG reader's surface from outputs (not passing this into createCaptureSession).
There's quite a lot of code so I created a gist: click - Tried to get rid of completely irrelevant code.

Is the device you're testing on a LEGACY-level device?
If so, any captures targeting a JPEG output may be much slower since they can run a precapture sequence, and may briefly pause preview as well.
But it should not cause green frames, unless there's a device-level bug.

If anyone ever struggles with this. There is table in the docs showing that if there are 3 targets specified, the YUV ImageReader can use images with maximum size equal to the preview size (maximum 1920x1080). Reducing this helped!

Yes you can. Assuming that you configure your preview to feed the ImageReader with YUV frames (because you could also put JPEG there, check it out), like so:
mImageReaderPreview = ImageReader.newInstance(mPreviewSize.getWidth(), mPreviewSize.getHeight(), ImageFormat.YUV_420_888, 1);
You can process those frames inside your OnImageAvailable listener:
#Override
public void onImageAvailable(ImageReader reader) {
Image mImage = reader.acquireNextImage();
if (mImage == null) {
return;
}
try {
// Do some custom processing like YUV to RGB conversion, cropping, etc.
mFrameProcessor.setNextFrame(mImage));
mImage.close();
} catch (IllegalStateException e) {
Log.e("TAG", e.getMessage());
}

Android MediaRecorder saving empty file and weird camera behaviours

I am having a headache over the Camera API 1 for android. After reading all of the Internet content, I made some sample app that works OK. It creates a service, which then is used to operate with the camera in the background, so there is no preview or activity enabled. To achieve this I use a dummy SurfaceHolder, like this:
protected class MySurfaceHolder implements SurfaceHolder {
private final Surface surface;
private final SurfaceTexture surfaceTexture;
public MySurfaceHolder () {
int[] textures = new int[1];
GLES20.glGenTextures(1, textures, 0);
if (textures.length > 0) {
this.surfaceTexture = new SurfaceTexture(textures[0]);
this.surface = new Surface(this.surfaceTexture);
} else {
this.surface = null;
this.surfaceTexture = null;
}
}
[...]
}
and then I use it like this
// simplified version of my code
try {
initializeCamera(); // open camera and set Camera.Parameters
camera.setPreviewDisplay(new MySurfaceHolder());
camera.startPreview();
camera.unlock();
initializeMediaRecorder(); // create MediaRecorder, set video/audio parameters
mediaRecorder.prepare();
mediaRecorder.start();
// wait until recording finish and exit
} finally {
stopRecording();
}
the Camera and MediaRecorder initialization methods are just like the documentation states they should be (and they work).
Everything works and operates as it should. Almost everything - sometimes, under unknown circumstances the MediaRecorder creates empty files, like 32kB containing only headers and info about the video - no frames. The longer I record like this, the bigger is the file (few kB every few seconds). After 1 minute, the file weights about 80kB. Funny thing is I know that the camera is working and capturing frames (I debugged it a little showing preview frames), but the frames are not written into the output file.
Also when it happens I am not able to record in FHD (1920x1080) - I get the "start failed" message - at this time camera is not capturing frames. The same thing could happen when I use wrong (not supported) video size. I suppose in this case the message is thrown at the mediaRecorder.start(); line, and stopRecording(); is invoked but I am not sure.
After some time or after unknown action the problem is suddenly gone (I don't know when, I don't know how). It happens for sure on Android 5.1, but may happen on other versions as well.
Could this bug be related to my custom surface code?
What could cause the MediaRecorder to not write frames into a file?
Why I am not able to record in FHD, but in the same time I am able to record in HD (1280x720)?
Is there any alternative for MediaRecorder, so I can avoid these bugs?
May it happen when another app is trying to get Camera object, thus distrupting current recording? If so, how to regain access to the Camera object (I apparently am not able to do this now on some devices).
EDIT:
I think I might have a clue. I am calling
camera.setOneShotPreviewCallback(new Camera.PreviewCallback() {
// ... get current frame
}
camera.startPreview();
to get preview frame of current recording. It appears that the bug occurs when I am using this method to get preview frame (at random times). It seems flawed, because not all devices react to this thing properly (sometimes there is no preview frame...). Is there any other, better method of handling current preview frame without the real surface?

Android camera: very low preview FPS

A week ago I've started researching the Android camera API. I have successfully inited the camera and started preview, and it worked fine. Then I've found out I wasn't initializing and releasing the camera properly, so I overhauled the code somewhat, and now I have a problem that didn't occur initially: extremely low FPS. About 0.5, that's 2 seconds per frame. Interestingly enough, I get 1 frame with a delay, and then a second frame immediately after (1-15 ms), followed again by 2 seconds delay before the next frame.
This is my camera initialization code:
m_openedCamera = Camera.open(id);
m_surfaceHolder = new SurfaceView(MyApplication.instance().getApplicationContext()).getHolder();
Assert.assertNotNull(m_openedCamera);
// This is required on A500 for some reason
Camera.Parameters params = m_openedCamera.getParameters();
params.setPreviewFormat(ImageFormat.NV21);
params.setPreviewSize(320, 240);
if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.ICE_CREAM_SANDWICH)
{
params.setRecordingHint(true);
params.setAutoExposureLock(true);
params.setAutoWhiteBalanceLock(true);
}
m_openedCamera.setParameters(params);
int bitsPerPx = ImageFormat.getBitsPerPixel(ImageFormat.NV21);
int width = params.getPreviewSize().width;
int height = params.getPreviewSize().height;
int size = (int)(width * height * bitsPerPx / 8.0);
m_openedCamera.addCallbackBuffer( new byte[size] );
m_openedCamera.addCallbackBuffer( new byte[size] );
m_openedCamera.addCallbackBuffer( new byte[size] );
m_openedCamera.addCallbackBuffer( new byte[size] );
m_openedCamera.setErrorCallback(this);
m_openedCamera.setPreviewDisplay(m_surfaceHolder);
m_openedCameraFacing = facing;
m_openedCamera.setPreviewCallback(this);
m_openedCamera.startPreview();
I have just added callback buffers - hasn't changed anything. In my initial code from a week ago I had no surface view, but removing it now has no effect either.
This occurs on my second, much newer tablet as well, even though FPS is higher there (8-10), and there's no double frame there, the frames are spaced evenly. The FPS used to be at least 20. Light conditions haven't changed between now and then, btw.
Update: tried opening the camera in a separate thread as described here - no change.

params.setRecordingHint(true); is used to start video record starts.
Please check it again with "params.setRecordingHint(false)"

Please make your own previewcallbak function and then
private PreviewCallback mPreviewCallback = new PreviewCallback();
....
setPreviewCallback(mPreviewCallback);
Regarding PreviewCallback(), you can refer to CameraTest.Java in cts directory : cts/tests/tests/hawdware/src/android/hardware/cts

From your comment below
"I have successfully inited the camera and started preview, and it worked fine. Then I've found out I wasn't initializing and releasing the camera properly"
I guess you was using the default camera parameters defined in camera HAL at first. Although it is not clear what "initializing" means, I think they is likely camera parameters setting.
So, I'd like to suggest you remove the code for params and then test it again.
Or, you can test it one by one
Remove only "params.setPreviewSize(320, 240);"
Remove only "params.setPreviewFormat(ImageFormat.NV21);"
Remove both 1 and 2.

Android camera pixel manipulation

I am trying to build a camera app that takes in the camera preview, manipulates the pixels, and then displays the new image. I need the manipulation to happen in real time.
From what I have read online, and from questions here, you need to make a custom surface view and manipulate the pixel array from the onPreviewFrame method. I have built a custom surface view and have this method running. I have converted the YUV to RGB.
Now, my question is, how do I display this new pixel array on the screen in real time? Do I somehow return it in the onPreviewFrame method? Do I have to change the byte[] array? Do I take my new pixel array and display it using a Bitmap? Is there a way to get the byte[] array from the camera preview without even displaying the preview?
If someone could answer these questions, with code examples that would be great! I am kind of new to Android, so I need the answers explained well enough for me to understand. Here is part of the code I have that runs the camera preview:
public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
// If your preview can change or rotate, take care of those events here.
// Make sure to stop the preview before resizing or reformatting it.
if (mHolder.getSurface() == null){
// preview surface does not exist
return;
}
// stop preview before making changes
try {
mCamera.stopPreview();
} catch (Exception e){
// ignore: tried to stop a non-existent preview
}
// start preview with new settings
try {
//parameters.setPreviewSize(w, h);
mCamera.setParameters(parameters);
mCamera.setPreviewDisplay(mHolder);
mCamera.setPreviewCallback(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera)
{
System.out.println("onPreviewFrame");
//transforms NV21 pixel data into RGB pixels
decodeYUV420SP(pixels, data, previewSize.width, previewSize.height);
//Outuput the value of the top left pixel in the preview to LogCat
Log.i("Pixels", "The top right pixel has the following RGB (hexadecimal) values:"
+Integer.toHexString(pixels[0]));
}
});
mCamera.startPreview();
} catch (Exception e){
Log.d(null, "Error starting camera preview: " + e.getMessage());
}
}
This gives me the rgb pixel array I want to display instead of the preview. How do I do this?

You cannot manipulate the surface that you connected to camera preview. The byte array you receive in onPreviewFrame() is just a copy of what the framework displays on the screen. Moreover, you will find that the two streams are asynchronous: you can slow down the callbacks (e.g. by adding some sleep() into your callback), but the preview surface will be updated nevertheless.
You can hide the preview SurfaceView by placing other views on top of it, or you can get rid of this view altogether by using setPreviewTexture() instead of setPreviewDisplay() (note: Added in API level 11). Hiding the surface is not as easy as it may seem: the framework may pop it up to the top, it requires careful synchronization of camera start or restart with layout.
Anyway, after you have the surface hidden, you can use the byte array received in onPreviewFrame() to generate an image and display it. You are free to manipulate the pixels to your liking. I believe that the optimal technique is to send the pixel data to OpenGL: you can use a shader to offload YCrCb (NV21) to RGB conversion to GPU.