I want to do some image processing with images from camera and display it on a SurfaceView but I don't know how to modify the camera frame. I tried to use setPreviewCallbackWithBuffer and onPreviewFrame but they do not work as expected, the frame is not modified.
/** A basic Camera preview class */
public class CameraPreview extends SurfaceView implements
SurfaceHolder.Callback, Camera.PreviewCallback {
private SurfaceHolder mHolder;
private Camera mCamera;
private byte[] mData;
private long prevFrameTick = System.currentTimeMillis();
Canvas mCanvas;
public CameraPreview(Context context, Camera camera) {
super(context);
mCamera = camera;
// Install a SurfaceHolder.Callback so we get notified when the
// underlying surface is created and destroyed.
mHolder = getHolder();
mHolder.addCallback(this);
Size previewSize = mCamera.getParameters().getPreviewSize();
mData = new byte[(int) (previewSize.height * previewSize.width * 1.5)];
initBuffer();
// deprecated setting, but required on Android versions prior to 3.0
mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
}
private void initBuffer() {
mCamera.addCallbackBuffer(mData);
mCamera.addCallbackBuffer(mData);
mCamera.addCallbackBuffer(mData);
mCamera.setPreviewCallbackWithBuffer(this);
}
public void setCamera(Camera cam) {
mCamera = cam;
initBuffer();
}
public void surfaceCreated(SurfaceHolder holder) {
// The Surface has been created, now tell the camera where to draw the
// preview.
try {
mCamera.setPreviewDisplay(holder);
initBuffer();
mCamera.startPreview();
} catch (IOException e) {
Log.d("APP",
"Error setting camera preview: " + e.getMessage());
}
}
public void surfaceDestroyed(SurfaceHolder holder) {
// empty. Take care of releasing the Camera preview in your activity.
}
public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
// If your preview can change or rotate, take care of those events here.
// Make sure to stop the preview before resizing or reformatting it.
if (mHolder.getSurface() == null) {
// preview surface does not exist
return;
}
// stop preview before making changes
try {
mCamera.stopPreview();
} catch (Exception e) {
// ignore: tried to stop a non-existent preview
}
// set preview size and make any resize, rotate or
// reformatting changes here
// start preview with new settings
try {
mCamera.setPreviewDisplay(mHolder);
initBuffer();
mCamera.startPreview();
} catch (Exception e) {
Log.d("APP",
"Error starting camera preview: " + e.getMessage());
}
}
public void onPreviewFrame(byte[] data, Camera camera) {
// System.arraycopy(data, 0, mData, 0, data.length);
Log.e("onPreviewFrame", data.length + " "
+ (System.currentTimeMillis() - prevFrameTick));
prevFrameTick = System.currentTimeMillis();
mData = new byte[data.length];
mCamera.addCallbackBuffer(mData);
}
}
You cannot modify the preview data sent to a SurfaceView, if you're using the setPreviewDisplay() call. The preview video stream is managed entirely outside of your application and isn't accessible to it.
There are a few options you can take:
You can place a second view on top of the SurfaceView, such as an ImageView or another SurfaceView, and draw the data received by the onPreviewFrame callback into this view. You'll have to do some color/pixel format conversion from the preview callback format (usually NV21) for display, and obviously you have to run your image processing on that data first as well. This isn't very efficient, unless you're willing to write some JNI code.
On Android 3.0 or newer, you can use the Camera.setPreviewTexture() method, and pipe the camera preview stream into an OpenGL texture by using a SurfaceTexture object, which you can then manipulate in OpenGL before displaying. Then you don't need the preview callbacks at all. This is more efficient if GPU processing is sufficient. You can also use the OpenGL readPixels call to get the processed preview data back to your application, if you want to display it/process it some other way.
Maybe it will be helpfull to someone.
I have solved this problem by using the OpenCV library for retrieving Frames from a Camera.
In the OpenCv 3 there is a method onCameraFrame(CvCameraViewFrame inputFrame):
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
// here you can do something with inputFrame before it appears on the preview
return inputFrame.rgba();
}
You can just try the Camera Preview project from the samples folder.
Or you can do this in the ndk https://vec.io/posts/how-to-render-image-buffer-in-android-ndk-native-code
But I haven't tryed this jet.
Or here you can find decoding from YUV to RGB in C/C++ with NDK https://github.com/youten/YUV420SP
Related
I want to render the camera output into a view and once in a while save the camera output frame to a file, with the constraint being - the saved frame should be the same resolution as the camera is configured, while the view is smaller than the camera output (maintaining the aspect ratio).
Based on the ContinuousCaptureActivity example in grafika, I thought the best approach would be to send the camera to a SurfaceTexture and generally rendering the output and downscaling it into a SurfaceView, and when needed, render the full frame into a different Surface that has no view, in order to retrieve a byte buffer from it in parallel to the regular SurfaceView rendering.
The example is very similar to my situation - the preview is rendered to a view of smaller size and can be recorded and saved at the full resolution via a VideoEncoder.
I replaced the VideoEncoder logic with my own and got stuck trying to provide a Surface, like the encoder does, for the full resolution rendering. How do I create such a Surface? Am I approaching this correctly?
Some code ideas based on the example:
Inside the surfaceCreated(SurfaceHolder holder) method (line 350):
#Override // SurfaceHolder.Callback
public void surfaceCreated(SurfaceHolder holder) {
Log.d(TAG, "surfaceCreated holder=" + holder);
mEglCore = new EglCore(null, EglCore.FLAG_RECORDABLE);
mDisplaySurface = new WindowSurface(mEglCore, holder.getSurface(), false);
mDisplaySurface.makeCurrent();
mFullFrameBlit = new FullFrameRect(
new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_EXT));
mTextureId = mFullFrameBlit.createTextureObject();
mCameraTexture = new SurfaceTexture(mTextureId);
mCameraTexture.setOnFrameAvailableListener(this);
Log.d(TAG, "starting camera preview");
try {
mCamera.setPreviewTexture(mCameraTexture);
} catch (IOException ioe) {
throw new RuntimeException(ioe);
}
mCamera.startPreview();
// *** MY EDIT START ***
// Encoder creation no longer needed
// try {
// mCircEncoder = new CircularEncoder(VIDEO_WIDTH, VIDEO_HEIGHT, 6000000,
// mCameraPreviewThousandFps / 1000, 7, mHandler);
// } catch (IOException ioe) {
// throw new RuntimeException(ioe);
// }
mEncoderSurface = new WindowSurface(mEglCore, mCameraTexture); // <-- Crashes with EGL error 0x3003
// *** MY EDIT END ***
updateControls();
}
The drawFrame() method (line 420):
private void drawFrame() {
//Log.d(TAG, "drawFrame");
if (mEglCore == null) {
Log.d(TAG, "Skipping drawFrame after shutdown");
return;
}
// Latch the next frame from the camera.
mDisplaySurface.makeCurrent();
mCameraTexture.updateTexImage();
mCameraTexture.getTransformMatrix(mTmpMatrix);
// Fill the SurfaceView with it.
SurfaceView sv = (SurfaceView) findViewById(R.id.continuousCapture_surfaceView);
int viewWidth = sv.getWidth();
int viewHeight = sv.getHeight();
GLES20.glViewport(0, 0, viewWidth, viewHeight);
mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
mDisplaySurface.swapBuffers();
// *** MY EDIT START ***
// Send it to the video encoder.
if (someCondition) {
mEncoderSurface.makeCurrent();
GLES20.glViewport(0, 0, VIDEO_WIDTH, VIDEO_HEIGHT);
mFullFrameBlit.drawFrame(mTextureId, mTmpMatrix);
mEncoderSurface.swapBuffers();
try {
mEncoderSurface.saveFrame(new File(getExternalFilesDir(null), String.valueOf(System.currentTimeMillis()) + ".png"));
} catch (IOException e) {
e.printStackTrace();
}
}
// *** MY EDIT END ***
}
You're on the right track. The SurfaceTexture just does a quick bit of wrapping around the original YUV frame from the camera, so the "external" texture is the original image, with no changes. You can't read the pixels straight out of an external texture, so you have to render it somewhere first.
The easiest way to do this is to create an off-screen pbuffer surface. Grafika's gles/OffscreenSurface class does exactly this (with a call to eglCreatePbufferSurface()). Make that EGLSurface current, render the texture onto a FullFrameRect, then read the framebuffer with glReadPixels() (see EglSurfaceBase#saveFrame() for code). Don't call eglSwapBuffers().
Note that you're not creating an Android Surface for the output, just an EGLSurface. (They're different.)
I'm working on an android app that is processing the input image from the camera and displays it to the user. This is fairly simple, I register a PreviewCallback on the camera object with the setPreviewCallbackWithBuffer.
This is easy and works smoothly with the old camera API
public void onPreviewFrame(byte[] data, Camera cam) {
// custom image data processing
}
I'm trying to port my app to take advantage of the new Camera2 API and I'm not sure how exactly shall I do that. I followed the Camera2Video in L Preview samples that allows to record a video. However, there is no direct image data transfer in the sample, so I don't understand where exactly shall I get the image pixel data and how to process it.
Could anybody help me or suggest the way how one can get the the functionality of PreviewCallback in android L, or how it's possible to process preview data from the camera before displaying it to the screen? (there is no preview callback on the camera object)
Thank you!
Combining a few answers into a more digestible one because #VP's answer, while technically clear, is difficult to understand if it's your first time moving from Camera to Camera2:
Using https://github.com/googlesamples/android-Camera2Basic as a starting point, modify the following:
In createCameraPreviewSession() init a new Surface from mImageReader
Surface mImageSurface = mImageReader.getSurface();
Add that new surface as a output target of your CaptureRequest.Builder variable. Using the Camera2Basic sample, the variable will be mPreviewRequestBuilder
mPreviewRequestBuilder.addTarget(mImageSurface);
Here's the snippet with the new lines (see my #AngeloS comments):
private void createCameraPreviewSession() {
try {
SurfaceTexture texture = mTextureView.getSurfaceTexture();
assert texture != null;
// We configure the size of default buffer to be the size of camera preview we want.
texture.setDefaultBufferSize(mPreviewSize.getWidth(), mPreviewSize.getHeight());
// This is the output Surface we need to start preview.
Surface surface = new Surface(texture);
//#AngeloS - Our new output surface for preview frame data
Surface mImageSurface = mImageReader.getSurface();
// We set up a CaptureRequest.Builder with the output Surface.
mPreviewRequestBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
//#AngeloS - Add the new target to our CaptureRequest.Builder
mPreviewRequestBuilder.addTarget(mImageSurface);
mPreviewRequestBuilder.addTarget(surface);
...
Next, in setUpCameraOutputs(), change the format from ImageFormat.JPEG to ImageFormat.YUV_420_888 when you init your ImageReader. (PS, I also recommend dropping your preview size for smoother operation - one nice feature of Camera2)
mImageReader = ImageReader.newInstance(largest.getWidth() / 16, largest.getHeight() / 16, ImageFormat.YUV_420_888, 2);
Finally, in your onImageAvailable() method of ImageReader.OnImageAvailableListener, be sure to use #Kamala's suggestion because the preview will stop after a few frames if you don't close it
#Override
public void onImageAvailable(ImageReader reader) {
Log.d(TAG, "I'm an image frame!");
Image image = reader.acquireNextImage();
...
if (image != null)
image.close();
}
Since the Camera2 API is very different from the current Camera API, it might help to go through the documentation.
A good starting point is camera2basic example. It demonstrates how to use Camera2 API and configure ImageReader to get JPEG images and register ImageReader.OnImageAvailableListener to receive those images
To receive preview frames, you need to add your ImageReader's surface to setRepeatingRequest's CaptureRequest.Builder.
Also, you should set ImageReader's format to YUV_420_888, which will give you 30fps at 8MP (The documentation guarantees 30fps at 8MP for Nexus 5).
In the ImageReader.OnImageAvailableListener class, close the image after reading as shown below (this will release the buffer for next capture). You will have to handle exception on close
Image image = imageReader.acquireNextImage();
ByteBuffer buffer = image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.remaining()];
buffer.get(bytes);
image.close();
I needed the same thing, so I used their example and added a call to a new function when the camera is in preview state.
private CameraCaptureSession.CaptureCallback mCaptureCallback
= new CameraCaptureSession.CaptureCallback()
private void process(CaptureResult result) {
switch (mState) {
case STATE_PREVIEW: {
if (buttonPressed){
savePreviewShot();
}
break;
}
The savePreviewShot() is simply a recycled version of the original captureStillPicture() adapted to use the preview template.
private void savePreviewShot(){
try {
final Activity activity = getActivity();
if (null == activity || null == mCameraDevice) {
return;
}
// This is the CaptureRequest.Builder that we use to take a picture.
final CaptureRequest.Builder captureBuilder =
mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
captureBuilder.addTarget(mImageReader.getSurface());
// Orientation
int rotation = activity.getWindowManager().getDefaultDisplay().getRotation();
captureBuilder.set(CaptureRequest.JPEG_ORIENTATION, ORIENTATIONS.get(rotation));
CameraCaptureSession.CaptureCallback CaptureCallback
= new CameraCaptureSession.CaptureCallback() {
#Override
public void onCaptureCompleted(CameraCaptureSession session, CaptureRequest request,
TotalCaptureResult result) {
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd_HH:mm:ss:SSS");
Date resultdate = new Date(System.currentTimeMillis());
String mFileName = sdf.format(resultdate);
mFile = new File(getActivity().getExternalFilesDir(null), "pic "+mFileName+" preview.jpg");
Log.i("Saved file", ""+mFile.toString());
unlockFocus();
}
};
mCaptureSession.stopRepeating();
mCaptureSession.capture(captureBuilder.build(), CaptureCallback, null);
} catch (Exception e) {
e.printStackTrace();
}
};
It's better to init ImageReader with max image buffer is 2 then use reader.acquireLatestImage() inside onImageAvailable().
Because acquireLatestImage() will acquire the latest Image from the ImageReader's queue, dropping older one. This function is recommended to use over acquireNextImage() for most use-cases, as it's more suited for real-time processing. Note that max image buffer should be at least 2.
And remember to close() your image after processing.
I am using Zbar to scan the QRCodes in landscape mode. I have a FrameLayout in xml which is nested in RelativeLayout and FrameLayout size is 300dp X 200dp. I have been googling for two says and almost read all the stackoverflow threads regarding this issue. but nothing seems to work for me.
Here is my code.
public class CameraPreview extends SurfaceView implements SurfaceHolder.Callback {
private SurfaceHolder mHolder;
private Camera mCamera;
private PreviewCallback previewCallback;
private AutoFocusCallback autoFocusCallback;
public CameraPreview(Context context, Camera camera,
PreviewCallback previewCb,
AutoFocusCallback autoFocusCb) {
super(context);
mCamera = camera;
previewCallback = previewCb;
autoFocusCallback = autoFocusCb;
/*
* Set camera to continuous focus if supported, otherwise use
* software auto-focus. Only works for API level >=9.
*/
/*
Camera.Parameters parameters = camera.getParameters();
for (String f : parameters.getSupportedFocusModes()) {
if (f == Parameters.FOCUS_MODE_CONTINUOUS_PICTURE) {
mCamera.setFocusMode(Parameters.FOCUS_MODE_CONTINUOUS_PICTURE);
autoFocusCallback = null;
break;
}
}
*/
// Install a SurfaceHolder.Callback so we get notified when the
// underlying surface is created and destroyed.
mHolder = getHolder();
mHolder.addCallback(this);
// deprecated setting, but required on Android versions prior to 3.0
mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
}
public void surfaceCreated(SurfaceHolder holder) {
// The Surface has been created, now tell the camera where to draw the preview.
try {
if(holder != null)
{
mCamera.setPreviewDisplay(holder);
Camera.Parameters parameters = mCamera.getParameters();
List<Size> sizes = parameters.getSupportedPictureSizes();
parameters.setPictureSize(sizes.get(0).width, sizes.get(0).height); // mac dinh solution 0
parameters.set("orientation","landscape");
//parameters.setPreviewSize(viewWidth, viewHeight);
List<Size> size = parameters.getSupportedPreviewSizes();
parameters.setPreviewSize(size.get(0).width, size.get(0).height);
mCamera.setParameters(parameters);
}
} catch (IOException e) {
Log.d("DBG", "Error setting camera preview: " + e.getMessage());
}
}
public void surfaceDestroyed(SurfaceHolder holder) {
// Camera preview released in activity
}
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
/*
* If your preview can change or rotate, take care of those events here.
* Make sure to stop the preview before resizing or reformatting it.
*/
if (mHolder.getSurface() == null){
// preview surface does not exist
return;
}
// stop preview before making changes
try {
mCamera.stopPreview();
} catch (Exception e){
// ignore: tried to stop a non-existent preview
}
try {
// Hard code camera surface rotation 90 degs to match Activity view in portrait
mCamera.setDisplayOrientation(0);
mCamera.setPreviewDisplay(mHolder);
mCamera.setPreviewCallback(previewCallback);
mCamera.startPreview();
mCamera.autoFocus(autoFocusCallback);
} catch (Exception e){
Log.d("DBG", "Error starting camera preview: " + e.getMessage());
}
}
}
Any help will be highly appreciated.
Thanks
The aspect ratio of your surface (300dp X 200dp) seems not to match that one of your camera previewSize. First determine the aspect ratio of your previewSize (parameters.getPreviewSize()) and then set the size of your surface to the same aspect ratio. When the aspect ratios are nearly the same, then the streching should be gone. Hope that helps!
ali's right. You can find my code here which is based on Ali's solution.
This question already has answers here:
How to get raw preview data from Camera object at least 15 frames per second in Android?
(6 answers)
Closed 9 years ago.
My application currently has a preview screen and I want it to capture many frames a second
for processing. At the moment, my preview is only storing an image every second, however I require a much larger fps capture rate. Any help would be appreciated.
Another problem (if you can) is that my images are rotated 90 degrees when they appear on my sd card. No internet solutions so far have helped me for these problems :(
Thanks :)
public class MyCameraSurfaceView extends SurfaceView implements SurfaceHolder.Callback{
private SurfaceHolder mHolder;
private Camera mCamera;
public MyCameraSurfaceView(Context context, Camera camera) {
super(context);
mCamera = camera;
// Install a SurfaceHolder.Callback so we get notified when the
// underlying surface is created and destroyed.
mHolder = getHolder();
mHolder.addCallback(this);
// deprecated setting, but required on Android versions prior to 3.0
mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
}
#Override
public void surfaceChanged(SurfaceHolder holder, int format, int weight,
int height) {
// If your preview can change or rotate, take care of those events here.
// Make sure to stop the preview before resizing or reformatting it.
mCamera.setDisplayOrientation(90);
if (mHolder.getSurface() == null){
// preview surface does not exist
return;
}
// stop preview before making changes
try {
mCamera.stopPreview();
} catch (Exception e){
// ignore: tried to stop a non-existent preview
}
// make any resize, rotate or reformatting changes here
// start preview with new settings
try {
mCamera.setPreviewDisplay(mHolder);
Camera.Parameters parameters = mCamera.getParameters();
List<Camera.Size> size = parameters.getSupportedPreviewSizes();
parameters.setPreviewSize(size.get(0).width, size.get(0).height);
mCamera.setParameters(parameters);
mCamera.startPreview();
} catch (Exception e){}
mCamera.setPreviewCallback(new Camera.PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera) {
Camera.Parameters parameters = camera.getParameters();
int format = parameters.getPreviewFormat();
//YUV formats require more conversion
if (format == ImageFormat.NV21 || format == ImageFormat.YUY2 || format == ImageFormat.NV16) {
int w = parameters.getPreviewSize().width;
int h = parameters.getPreviewSize().height;
// Get the YuV image
YuvImage yuv_image = new YuvImage(data, format, w, h, null);
// Convert YuV to Jpeg
Rect rect = new Rect(0, 0, w, h);
ByteArrayOutputStream output_stream = new ByteArrayOutputStream();
yuv_image.compressToJpeg(rect, 10, output_stream);
byte[] byt = output_stream.toByteArray();
FileOutputStream outStream = null;
try {
outStream = new FileOutputStream(String.format(
"/sdcard/bb%d.jpg", System.currentTimeMillis() / 1000));
outStream.write(byt);
outStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
}
}
}
});
}
#Override
public void surfaceCreated(SurfaceHolder holder) {
// TODO Auto-generated method stub
// The Surface has been created, now tell the camera where to draw the preview.
try {
mCamera.setDisplayOrientation(90);
mCamera.setPreviewDisplay(holder);
mCamera.startPreview();
} catch (IOException e) {
}
}
#Override
public void surfaceDestroyed(SurfaceHolder holder) {
// TODO Auto-generated method stub
}
}
First, note that Camera.Parameters allows you to set the frame rate with setPreviewFrameRate. The value must be in the range described by getPreviewFpsRange.
Second, handling preview frames in a byte[] is going to restrict your frame rate severely because of the amount of data that has to be copied around. If you want to write unmodified full-frame YUV data to disk then you don't currently have a choice. If you can cope with compression artifacts, and you have Android 4.3 or later, you can just save the data as an MPEG video and read the frames back later. See the CameraToMpegTest.java sample on this page for a code example.
Rotating an image by 90 or 180 degrees is straightforward to code. The Bitmap class can do it if you don't want to write it yourself.
setPreviewCallback() is an easy, but less efficient way to request preview frames. The main problem is that the framework may be very busy allocating the byte[] chunks to fill, and the garbage collector may take a heavy price. The preferred method is to use setPreviewCallbackWithBuffer(), but even this does not guarantee desired frame rate, as can be seen in How to get raw preview data from Camera object at least 15 frames per second in Android?
I'm puzzled by OpenCV's Android camera sample code. They make a custom class which implements SurfaceHolder.Callback and put the following line inside the method surfaceChanged:
mCamera.setPreviewDisplay(null);
The Android documentation for setPreviewDisplay explains:
This method must be called before startPreview(). The one exception is
that if the preview surface is not set (or set to null) before
startPreview() is called, then this method may be called once with a
non-null parameter to set the preview surface. (This allows camera
setup and surface creation to happen in parallel, saving time.) The
preview surface may not otherwise change while preview is running.
Unusually, OpenCV's code never calls setPreviewDisplay with a non-null SurfaceHolder. It works fine, but changing the rotation of the image using setDisplayOrientation doesn't work. This line also doesn't appear to do anything, since I get the same results without it.
If I call setPreviewDisplay with the SurfaceHolder supplied to surfaceChanged instead of null, the image rotates but does not include the results of the image processing. I also get an IllegalArgumentException when calling lockCanvas later on.
What's going on?
Here are the (possibly) most relevant parts of their code, slightly simplified and with methods inlined. Here is the full version.
Class definition
public abstract class SampleViewBase extends SurfaceView
implements SurfaceHolder.Callback, Runnable {
When the camera is opened
mCamera.setPreviewCallbackWithBuffer(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera) {
synchronized (SampleViewBase.this) {
System.arraycopy(data, 0, mFrame, 0, data.length);
SampleViewBase.this.notify();
}
camera.addCallbackBuffer(mBuffer);
}
});
When the surface changes
/* Now allocate the buffer */
mBuffer = new byte[size];
/* The buffer where the current frame will be copied */
mFrame = new byte [size];
mCamera.addCallbackBuffer(mBuffer);
try {
mCamera.setPreviewDisplay(null);
} catch (IOException e) {
Log.e(TAG, "mCamera.setPreviewDisplay/setPreviewTexture fails: " + e);
}
[...]
/* Now we can start a preview */
mCamera.startPreview();
The run method
public void run() {
mThreadRun = true;
Log.i(TAG, "Starting processing thread");
while (mThreadRun) {
Bitmap bmp = null;
synchronized (this) {
try {
this.wait();
bmp = processFrame(mFrame);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
if (bmp != null) {
Canvas canvas = mHolder.lockCanvas();
if (canvas != null) {
canvas.drawBitmap(bmp, (canvas.getWidth() - getFrameWidth()) / 2,
(canvas.getHeight() - getFrameHeight()) / 2, null);
mHolder.unlockCanvasAndPost(canvas);
}
}
}
Log.i(TAG, "Finishing processing thread");
}
I ran into this same problem. Instead of using a SurfaceView.Callback, I subclassed their class JavaCameraView. See my live face detection and drawing sample here. It was then trivial to rotate the matrix coming out of the camera according to the device's orientation, prior to processing. Relevant excerpt of linked code:
#Override
public Mat onCameraFrame(Mat inputFrame) {
int flipFlags = 1;
if(display.getRotation() == Surface.ROTATION_270) {
flipFlags = -1;
Log.i(VIEW_LOG_TAG, "Orientation is" + getRotation());
}
Core.flip(inputFrame, mRgba, flipFlags);
inputFrame.release();
Imgproc.cvtColor(mRgba, mGray, Imgproc.COLOR_RGBA2GRAY);
if (mAbsoluteFaceSize == 0) {
int height = mGray.rows();
if (Math.round(height * mRelativeFaceSize) > 0) {
mAbsoluteFaceSize = Math.round(height * mRelativeFaceSize);
}
}
}
I solved the rotation issue using OpenCV itself: after finding out how much the screen rotation needs to be corrected using this code, I apply a rotation matrix to the raw camera image (after converting from YUV to RGB):
Point center = new Point(mFrameWidth/2, mFrameHeight/2);
Mat rotationMatrix = Imgproc.getRotationMatrix2D(center, totalRotation, 1);
[...]
Imgproc.cvtColor(mYuv, mIntermediate, Imgproc.COLOR_YUV420sp2RGBA, 4);
Imgproc.warpAffine(mIntermediate, mRgba, rotationMatrix,
new Size(mFrameHeight, mFrameWidth));
A separate issue is that setPreviewDisplay(null) gives a blank screen on some phones. The solution, which I got from here and draws on this bugreport and this SO question, passes a hidden, "fake" SurfaceView to the preview display to get it to start, but actually displays the output on an overlaid custom view, which I call CameraView. So, after calling setContentView() in the activity's onCreate(), stick in this code:
if (VERSION.SDK_INT < VERSION_CODES.HONEYCOMB) {
final SurfaceView fakeView = new SurfaceView(this);
fakeView.setLayoutParams(new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT));
fakeView.setZOrderMediaOverlay(false);
final CameraView cameraView = (CameraView) this.findViewById(R.id.cameraview);
cameraView.setZOrderMediaOverlay(true);
cameraView.fakeView = fakeView;
}
Then, when setting the preview display, use this code:
try {
if (VERSION.SDK_INT >= VERSION_CODES.HONEYCOMB)
mCamera.setPreviewTexture(new SurfaceTexture(10));
else
mCamera.setPreviewDisplay(fakeView.getHolder());
} catch (IOException e) {
Log.e(TAG, "mCamera.setPreviewDisplay fails: "+ e);
}
If you are only developing for Honeycomb and above, just replace setPreviewDisplay(null) with mCamera.setPreviewTexture(new SurfaceTexture(10)); and be done with it. setDisplayOrientation() still doesn't work if you do this, though, so you'll still have to use the rotation matrix solution.