I want to render the camera output into a view and once in a while save the camera output frame to a file, with the constraint being - the saved frame should be the same resolution as the camera is configured, while the view is smaller than the camera output (maintaining the aspect ratio).
Based on the ContinuousCaptureActivity example in grafika, I thought the best approach would be to send the camera to a SurfaceTexture and generally rendering the output and downscaling it into a SurfaceView, and when needed, render the full frame into a different Surface that has no view, in order to retrieve a byte buffer from it in parallel to the regular SurfaceView rendering.
The example is very similar to my situation - the preview is rendered to a view of smaller size and can be recorded and saved at the full resolution via a VideoEncoder.
I replaced the VideoEncoder logic with my own and got stuck trying to provide a Surface, like the encoder does, for the full resolution rendering. How do I create such a Surface? Am I approaching this correctly?
Some code ideas based on the example:
Inside the surfaceCreated(SurfaceHolder holder) method (line 350):
#Override // SurfaceHolder.Callback
public void surfaceCreated(SurfaceHolder holder) {
Log.d(TAG, "surfaceCreated holder=" + holder);
mEglCore = new EglCore(null, EglCore.FLAG_RECORDABLE);
mDisplaySurface = new WindowSurface(mEglCore, holder.getSurface(), false);
mDisplaySurface.makeCurrent();
mFullFrameBlit = new FullFrameRect(
new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_EXT));
mTextureId = mFullFrameBlit.createTextureObject();
mCameraTexture = new SurfaceTexture(mTextureId);
mCameraTexture.setOnFrameAvailableListener(this);
Log.d(TAG, "starting camera preview");
try {
mCamera.setPreviewTexture(mCameraTexture);
} catch (IOException ioe) {
throw new RuntimeException(ioe);
}
mCamera.startPreview();
// *** MY EDIT START ***
// Encoder creation no longer needed
// try {
// mCircEncoder = new CircularEncoder(VIDEO_WIDTH, VIDEO_HEIGHT, 6000000,
// mCameraPreviewThousandFps / 1000, 7, mHandler);
// } catch (IOException ioe) {
// throw new RuntimeException(ioe);
// }
mEncoderSurface = new WindowSurface(mEglCore, mCameraTexture); // <-- Crashes with EGL error 0x3003
// *** MY EDIT END ***
updateControls();
}
The drawFrame() method (line 420):
private void drawFrame() {
//Log.d(TAG, "drawFrame");
if (mEglCore == null) {
Log.d(TAG, "Skipping drawFrame after shutdown");
return;
}
// Latch the next frame from the camera.
mDisplaySurface.makeCurrent();
mCameraTexture.updateTexImage();
mCameraTexture.getTransformMatrix(mTmpMatrix);
// Fill the SurfaceView with it.
SurfaceView sv = (SurfaceView) findViewById(R.id.continuousCapture_surfaceView);
int viewWidth = sv.getWidth();
int viewHeight = sv.getHeight();
GLES20.glViewport(0, 0, viewWidth, viewHeight);
mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
mDisplaySurface.swapBuffers();
// *** MY EDIT START ***
// Send it to the video encoder.
if (someCondition) {
mEncoderSurface.makeCurrent();
GLES20.glViewport(0, 0, VIDEO_WIDTH, VIDEO_HEIGHT);
mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
mEncoderSurface.swapBuffers();
try {
mEncoderSurface.saveFrame(new File(getExternalFilesDir(null), String.valueOf(System.currentTimeMillis()) + ".png"));
} catch (IOException e) {
e.printStackTrace();
}
}
// *** MY EDIT END ***
}
You're on the right track. The SurfaceTexture just does a quick bit of wrapping around the original YUV frame from the camera, so the "external" texture is the original image, with no changes. You can't read the pixels straight out of an external texture, so you have to render it somewhere first.
The easiest way to do this is to create an off-screen pbuffer surface. Grafika's gles/OffscreenSurface class does exactly this (with a call to eglCreatePbufferSurface()). Make that EGLSurface current, render the texture onto a FullFrameRect, then read the framebuffer with glReadPixels() (see EglSurfaceBase#saveFrame() for code). Don't call eglSwapBuffers().
Note that you're not creating an Android Surface for the output, just an EGLSurface. (They're different.)
Related
When i have tried to render texture and transformation matrix to the EGLSurface, no display is seen in the view.
As a follow up of this issue , slightly i have modified slightly the code by following grafika/fadden sample code continuous capture
Here is my code:
Here is a draw method which runs on RenderThread.
This draw method is getting invoked properly whevener the data is produced at the producer end from Native Code.
public void drawFrame() {
mOffScreenSurface.makeCurrent();
mCameraTexture.updateTexImage();
mCameraTexture.getTransformMatrix(mTmpMatrix);
mSurfaceWindowUser.makeCurrent();
mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
mSurfaceWindowUser.swapBuffers();
}
run method of RenderThread ->
public void run() {
Looper.prepare();
mHandler = new RenderHandler(this);
mEglCore = new EglCore(null, EglCore.FLAG_RECORDABLE);
mOffScreenSurface = new OffscreenSurface(mEglCore, 640, 480);
mOffScreenSurface.makeCurrent();
mFullFrameBlit = new FullFrameRect(
new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_EXT));
mTextureId = mFullFrameBlit.createTextureObject();
mCameraTexture = new SurfaceTexture(mTextureId);
mCameraSurface = new Surface (mCameraTexture); // This surface i am sending to Native Code where i use ANativeWindow reference and copy the data using post method. {producer}
mCameraTexture.setOnFrameAvailableListener(new SurfaceTexture.OnFrameAvailableListener() {
#Override
public void onFrameAvailable(SurfaceTexture surfaceTexture) {
Log.d (TAG, "Long breath.. data is pumbed by Native Layer producer..");
mHandler.frameReceivedFromProducer();
}
});
mSurfaceWindowUser = new WindowSurface(mEglCore, mSurfaceUser, false); // this mSurfaceUser is a surface received from MainActivity TextureView.
}
To confirm if the produce at the native side producing the data, if i pass directly the user surface Without any EGL configurations, the frames are rendered into the screen.
At the native Level,
geometryResult = ANativeWindow_setBuffersGeometry(userNaiveWindow,640, 480, WINDOW_FORMAT_RGBA_8888);
To Render the frame i use
ANativeWindow_lock and ANativeWindow_unlockAndPost() to render directly frame into buffer.
I could not able to think what could be wrong and where i have to dig more ?
Thanks fadden for your help.
I'm trying to play the same video at the same time in two different textureviews. I've used code from grafika (MoviePlayer and ContinuousCaptureActivity) to try to get it to work (thanks fadden). To make the problem simpler, I'm trying to do it with just one TextureView first.
At the moment I've created a TextureView, and once it get a SurfaceTexture, I create a WindowSurface and make it current. Then I generate a TextureID generated using a FullFrameRect object.
#Override
public void onSurfaceTextureAvailable(SurfaceTexture surface, int
width, int height) {
mSurfaceTexture = surface;
mEGLCore = new EglCore(null, EglCore.FLAG_TRY_GLES3);
Log.d("EglCore", "EGL core made");
mDisplaySurface = new WindowSurface(mEGLCore, mSurfaceTexture);
mDisplaySurface.makeCurrent();
Log.d("DisplaySurface", "mDisplaySurface made");
mFullFrameBlit = new FullFrameRect(new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_EXT));
mTextureID = mFullFrameBlit.createTextureObject();
//mSurfaceTexture.attachToGLContext(mTextureID);
clickPlayStop(null);
}
Then I get an off-screen SurfaceTexture, link it with the TextureID that I got above and create a surface to pass to a MoviePlayer thus:
public void clickPlayStop(#SuppressWarnings("unused") View unused) {
if (mShowStopLabel) {
Log.d(TAG, "stopping movie");
stopPlayback();
// Don't update the controls here -- let the task thread do it after the movie has
// actually stopped.
//mShowStopLabel = false;
//updateControls();
} else {
if (mPlayTask != null) {
Log.w(TAG, "movie already playing");
return;
}
Log.d(TAG, "starting movie");
SpeedControlCallback callback = new SpeedControlCallback();
callback.setFixedPlaybackRate(24);
MoviePlayer player = null;
MovieTexture = new SurfaceTexture(mTextureID);
MovieTexture.setOnFrameAvailableListener(this);
Surface surface = new Surface(MovieTexture);
try {
player = new MoviePlayer(surface, callback, this);//TODO
} catch (IOException ioe) {
Log.e(TAG, "Unable to play movie", ioe);
return;
}
adjustAspectRatio(player.getVideoWidth(), player.getVideoHeight());
mPlayTask = new MoviePlayer.PlayTask(player, this);
mPlayTask.setLoopMode(true);
mShowStopLabel = true;
mPlayTask.execute();
}
}
The idea is that the SurfaceTexture gets a raw frame which I can use as an OES_external texture to sample from with OpenGL. Then I can call DrawFrame() from my EGLContext after setting my WindowSurface as current.
private void drawFrame() {
Log.d(TAG, "drawFrame");
if (mEGLCore == null) {
Log.d(TAG, "Skipping drawFrame after shutdown");
return;
}
// Latch the next frame from the camera.
mDisplaySurface.makeCurrent();
MovieTexture.updateTexImage();
MovieTexture.getTransformMatrix(mTransformMatrix);
// Fill the WindowSurface with it.
int viewWidth = mTextureView.getWidth();
int viewHeight = mTextureView.getHeight();
GLES20.glViewport(0, 0, viewWidth, viewHeight);
mFullFrameBlit.drawFrame(mTextureID, mTransformMatrix);
mDisplaySurface.swapBuffers();
}
If I wanted to do it with 2 TextureViews, the idea would be to call makeCurrent() and draw into each buffer for each view, then call swapBuffers() after the drawing is done.
This is what I want to do, but I am pretty sure this is not what my code is actually doing. Could somebody help me understand what I need to change to make it work?
#Fadden
Update: This is interesting. I changed the code in onSurfaceTextureAvailable to this:
#Override
public void onSurfaceTextureAvailable(SurfaceTexture surface, int
width, int height) {
mSurfaceTexture = surface;
TextureHeight = height;
TextureWidth = width;
//mEGLCore = new EglCore(null, EglCore.FLAG_TRY_GLES3);
Log.d("EglCore", "EGL core made");
//mDisplaySurface = new WindowSurface(mEGLCore, mSurfaceTexture);
//mDisplaySurface.makeCurrent();
Log.d("DisplaySurface", "mDisplaySurface made");
//mFullFrameBlit = new FullFrameRect(new Texture2dProgram(Texture2dProgram.ProgramType.OPENGL_TEST));
//mTextureID = mFullFrameBlit.createTextureObject();
//clickPlayStop(null);
// Fill the SurfaceView with it.
//int viewWidth = width;
//int viewHeight = height;
//GLES20.glViewport(0, 0, viewWidth, viewHeight);
//mFullFrameBlit.drawFrame(mTextureID, mTransformMatrix);
//mFullFrameBlit.openGLTest();
//mFullFrameBlit.testDraw(mDisplaySurface.getHeight(),mDisplaySurface.getWidth());
//mDisplaySurface.swapBuffers();
}
So, it shouldn't call anything else, just show the empty TextureView - and this is what I see...
Thanks to Fadden for the help.
So there seemed to be some unknown issue that was resolved when I used a new thread to decode and produce the frames. I haven't found out what caused the original problem, but I have found a way around it.
This question already has answers here:
How to get raw preview data from Camera object at least 15 frames per second in Android?
(6 answers)
Closed 9 years ago.
My application currently has a preview screen and I want it to capture many frames a second
for processing. At the moment, my preview is only storing an image every second, however I require a much larger fps capture rate. Any help would be appreciated.
Another problem (if you can) is that my images are rotated 90 degrees when they appear on my sd card. No internet solutions so far have helped me for these problems :(
Thanks :)
public class MyCameraSurfaceView extends SurfaceView implements SurfaceHolder.Callback{
private SurfaceHolder mHolder;
private Camera mCamera;
public MyCameraSurfaceView(Context context, Camera camera) {
super(context);
mCamera = camera;
// Install a SurfaceHolder.Callback so we get notified when the
// underlying surface is created and destroyed.
mHolder = getHolder();
mHolder.addCallback(this);
// deprecated setting, but required on Android versions prior to 3.0
mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
}
#Override
public void surfaceChanged(SurfaceHolder holder, int format, int weight,
int height) {
// If your preview can change or rotate, take care of those events here.
// Make sure to stop the preview before resizing or reformatting it.
mCamera.setDisplayOrientation(90);
if (mHolder.getSurface() == null){
// preview surface does not exist
return;
}
// stop preview before making changes
try {
mCamera.stopPreview();
} catch (Exception e){
// ignore: tried to stop a non-existent preview
}
// make any resize, rotate or reformatting changes here
// start preview with new settings
try {
mCamera.setPreviewDisplay(mHolder);
Camera.Parameters parameters = mCamera.getParameters();
List<Camera.Size> size = parameters.getSupportedPreviewSizes();
parameters.setPreviewSize(size.get(0).width, size.get(0).height);
mCamera.setParameters(parameters);
mCamera.startPreview();
} catch (Exception e){}
mCamera.setPreviewCallback(new Camera.PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera) {
Camera.Parameters parameters = camera.getParameters();
int format = parameters.getPreviewFormat();
//YUV formats require more conversion
if (format == ImageFormat.NV21 || format == ImageFormat.YUY2 || format == ImageFormat.NV16) {
int w = parameters.getPreviewSize().width;
int h = parameters.getPreviewSize().height;
// Get the YuV image
YuvImage yuv_image = new YuvImage(data, format, w, h, null);
// Convert YuV to Jpeg
Rect rect = new Rect(0, 0, w, h);
ByteArrayOutputStream output_stream = new ByteArrayOutputStream();
yuv_image.compressToJpeg(rect, 10, output_stream);
byte[] byt = output_stream.toByteArray();
FileOutputStream outStream = null;
try {
outStream = new FileOutputStream(String.format(
"/sdcard/bb%d.jpg", System.currentTimeMillis() / 1000));
outStream.write(byt);
outStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
}
}
}
});
}
#Override
public void surfaceCreated(SurfaceHolder holder) {
// TODO Auto-generated method stub
// The Surface has been created, now tell the camera where to draw the preview.
try {
mCamera.setDisplayOrientation(90);
mCamera.setPreviewDisplay(holder);
mCamera.startPreview();
} catch (IOException e) {
}
}
#Override
public void surfaceDestroyed(SurfaceHolder holder) {
// TODO Auto-generated method stub
}
}
First, note that Camera.Parameters allows you to set the frame rate with setPreviewFrameRate. The value must be in the range described by getPreviewFpsRange.
Second, handling preview frames in a byte[] is going to restrict your frame rate severely because of the amount of data that has to be copied around. If you want to write unmodified full-frame YUV data to disk then you don't currently have a choice. If you can cope with compression artifacts, and you have Android 4.3 or later, you can just save the data as an MPEG video and read the frames back later. See the CameraToMpegTest.java sample on this page for a code example.
Rotating an image by 90 or 180 degrees is straightforward to code. The Bitmap class can do it if you don't want to write it yourself.
setPreviewCallback() is an easy, but less efficient way to request preview frames. The main problem is that the framework may be very busy allocating the byte[] chunks to fill, and the garbage collector may take a heavy price. The preferred method is to use setPreviewCallbackWithBuffer(), but even this does not guarantee desired frame rate, as can be seen in How to get raw preview data from Camera object at least 15 frames per second in Android?
I'm puzzled by OpenCV's Android camera sample code. They make a custom class which implements SurfaceHolder.Callback and put the following line inside the method surfaceChanged:
mCamera.setPreviewDisplay(null);
The Android documentation for setPreviewDisplay explains:
This method must be called before startPreview(). The one exception is
that if the preview surface is not set (or set to null) before
startPreview() is called, then this method may be called once with a
non-null parameter to set the preview surface. (This allows camera
setup and surface creation to happen in parallel, saving time.) The
preview surface may not otherwise change while preview is running.
Unusually, OpenCV's code never calls setPreviewDisplay with a non-null SurfaceHolder. It works fine, but changing the rotation of the image using setDisplayOrientation doesn't work. This line also doesn't appear to do anything, since I get the same results without it.
If I call setPreviewDisplay with the SurfaceHolder supplied to surfaceChanged instead of null, the image rotates but does not include the results of the image processing. I also get an IllegalArgumentException when calling lockCanvas later on.
What's going on?
Here are the (possibly) most relevant parts of their code, slightly simplified and with methods inlined. Here is the full version.
Class definition
public abstract class SampleViewBase extends SurfaceView
implements SurfaceHolder.Callback, Runnable {
When the camera is opened
mCamera.setPreviewCallbackWithBuffer(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera) {
synchronized (SampleViewBase.this) {
System.arraycopy(data, 0, mFrame, 0, data.length);
SampleViewBase.this.notify();
}
camera.addCallbackBuffer(mBuffer);
}
});
When the surface changes
/* Now allocate the buffer */
mBuffer = new byte[size];
/* The buffer where the current frame will be copied */
mFrame = new byte [size];
mCamera.addCallbackBuffer(mBuffer);
try {
mCamera.setPreviewDisplay(null);
} catch (IOException e) {
Log.e(TAG, "mCamera.setPreviewDisplay/setPreviewTexture fails: " + e);
}
[...]
/* Now we can start a preview */
mCamera.startPreview();
The run method
public void run() {
mThreadRun = true;
Log.i(TAG, "Starting processing thread");
while (mThreadRun) {
Bitmap bmp = null;
synchronized (this) {
try {
this.wait();
bmp = processFrame(mFrame);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
if (bmp != null) {
Canvas canvas = mHolder.lockCanvas();
if (canvas != null) {
canvas.drawBitmap(bmp, (canvas.getWidth() - getFrameWidth()) / 2,
(canvas.getHeight() - getFrameHeight()) / 2, null);
mHolder.unlockCanvasAndPost(canvas);
}
}
}
Log.i(TAG, "Finishing processing thread");
}
I ran into this same problem. Instead of using a SurfaceView.Callback, I subclassed their class JavaCameraView. See my live face detection and drawing sample here. It was then trivial to rotate the matrix coming out of the camera according to the device's orientation, prior to processing. Relevant excerpt of linked code:
#Override
public Mat onCameraFrame(Mat inputFrame) {
int flipFlags = 1;
if(display.getRotation() == Surface.ROTATION_270) {
flipFlags = -1;
Log.i(VIEW_LOG_TAG, "Orientation is" + getRotation());
}
Core.flip(inputFrame, mRgba, flipFlags);
inputFrame.release();
Imgproc.cvtColor(mRgba, mGray, Imgproc.COLOR_RGBA2GRAY);
if (mAbsoluteFaceSize == 0) {
int height = mGray.rows();
if (Math.round(height * mRelativeFaceSize) > 0) {
mAbsoluteFaceSize = Math.round(height * mRelativeFaceSize);
}
}
}
I solved the rotation issue using OpenCV itself: after finding out how much the screen rotation needs to be corrected using this code, I apply a rotation matrix to the raw camera image (after converting from YUV to RGB):
Point center = new Point(mFrameWidth/2, mFrameHeight/2);
Mat rotationMatrix = Imgproc.getRotationMatrix2D(center, totalRotation, 1);
[...]
Imgproc.cvtColor(mYuv, mIntermediate, Imgproc.COLOR_YUV420sp2RGBA, 4);
Imgproc.warpAffine(mIntermediate, mRgba, rotationMatrix,
new Size(mFrameHeight, mFrameWidth));
A separate issue is that setPreviewDisplay(null) gives a blank screen on some phones. The solution, which I got from here and draws on this bugreport and this SO question, passes a hidden, "fake" SurfaceView to the preview display to get it to start, but actually displays the output on an overlaid custom view, which I call CameraView. So, after calling setContentView() in the activity's onCreate(), stick in this code:
if (VERSION.SDK_INT < VERSION_CODES.HONEYCOMB) {
final SurfaceView fakeView = new SurfaceView(this);
fakeView.setLayoutParams(new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT));
fakeView.setZOrderMediaOverlay(false);
final CameraView cameraView = (CameraView) this.findViewById(R.id.cameraview);
cameraView.setZOrderMediaOverlay(true);
cameraView.fakeView = fakeView;
}
Then, when setting the preview display, use this code:
try {
if (VERSION.SDK_INT >= VERSION_CODES.HONEYCOMB)
mCamera.setPreviewTexture(new SurfaceTexture(10));
else
mCamera.setPreviewDisplay(fakeView.getHolder());
} catch (IOException e) {
Log.e(TAG, "mCamera.setPreviewDisplay fails: "+ e);
}
If you are only developing for Honeycomb and above, just replace setPreviewDisplay(null) with mCamera.setPreviewTexture(new SurfaceTexture(10)); and be done with it. setDisplayOrientation() still doesn't work if you do this, though, so you'll still have to use the rotation matrix solution.
I want to do some image processing with images from camera and display it on a SurfaceView but I don't know how to modify the camera frame. I tried to use setPreviewCallbackWithBuffer and onPreviewFrame but they do not work as expected, the frame is not modified.
/** A basic Camera preview class */
public class CameraPreview extends SurfaceView implements
SurfaceHolder.Callback, Camera.PreviewCallback {
private SurfaceHolder mHolder;
private Camera mCamera;
private byte[] mData;
private long prevFrameTick = System.currentTimeMillis();
Canvas mCanvas;
public CameraPreview(Context context, Camera camera) {
super(context);
mCamera = camera;
// Install a SurfaceHolder.Callback so we get notified when the
// underlying surface is created and destroyed.
mHolder = getHolder();
mHolder.addCallback(this);
Size previewSize = mCamera.getParameters().getPreviewSize();
mData = new byte[(int) (previewSize.height * previewSize.width * 1.5)];
initBuffer();
// deprecated setting, but required on Android versions prior to 3.0
mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
}
private void initBuffer() {
mCamera.addCallbackBuffer(mData);
mCamera.addCallbackBuffer(mData);
mCamera.addCallbackBuffer(mData);
mCamera.setPreviewCallbackWithBuffer(this);
}
public void setCamera(Camera cam) {
mCamera = cam;
initBuffer();
}
public void surfaceCreated(SurfaceHolder holder) {
// The Surface has been created, now tell the camera where to draw the
// preview.
try {
mCamera.setPreviewDisplay(holder);
initBuffer();
mCamera.startPreview();
} catch (IOException e) {
Log.d("APP",
"Error setting camera preview: " + e.getMessage());
}
}
public void surfaceDestroyed(SurfaceHolder holder) {
// empty. Take care of releasing the Camera preview in your activity.
}
public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
// If your preview can change or rotate, take care of those events here.
// Make sure to stop the preview before resizing or reformatting it.
if (mHolder.getSurface() == null) {
// preview surface does not exist
return;
}
// stop preview before making changes
try {
mCamera.stopPreview();
} catch (Exception e) {
// ignore: tried to stop a non-existent preview
}
// set preview size and make any resize, rotate or
// reformatting changes here
// start preview with new settings
try {
mCamera.setPreviewDisplay(mHolder);
initBuffer();
mCamera.startPreview();
} catch (Exception e) {
Log.d("APP",
"Error starting camera preview: " + e.getMessage());
}
}
public void onPreviewFrame(byte[] data, Camera camera) {
// System.arraycopy(data, 0, mData, 0, data.length);
Log.e("onPreviewFrame", data.length + " "
+ (System.currentTimeMillis() - prevFrameTick));
prevFrameTick = System.currentTimeMillis();
mData = new byte[data.length];
mCamera.addCallbackBuffer(mData);
}
}
You cannot modify the preview data sent to a SurfaceView, if you're using the setPreviewDisplay() call. The preview video stream is managed entirely outside of your application and isn't accessible to it.
There are a few options you can take:
You can place a second view on top of the SurfaceView, such as an ImageView or another SurfaceView, and draw the data received by the onPreviewFrame callback into this view. You'll have to do some color/pixel format conversion from the preview callback format (usually NV21) for display, and obviously you have to run your image processing on that data first as well. This isn't very efficient, unless you're willing to write some JNI code.
On Android 3.0 or newer, you can use the Camera.setPreviewTexture() method, and pipe the camera preview stream into an OpenGL texture by using a SurfaceTexture object, which you can then manipulate in OpenGL before displaying. Then you don't need the preview callbacks at all. This is more efficient if GPU processing is sufficient. You can also use the OpenGL readPixels call to get the processed preview data back to your application, if you want to display it/process it some other way.
Maybe it will be helpfull to someone.
I have solved this problem by using the OpenCV library for retrieving Frames from a Camera.
In the OpenCv 3 there is a method onCameraFrame(CvCameraViewFrame inputFrame):
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
// here you can do something with inputFrame before it appears on the preview
return inputFrame.rgba();
}
You can just try the Camera Preview project from the samples folder.
Or you can do this in the ndk https://vec.io/posts/how-to-render-image-buffer-in-android-ndk-native-code
But I haven't tryed this jet.
Or here you can find decoding from YUV to RGB in C/C++ with NDK https://github.com/youten/YUV420SP