I'm fighting with an IllegalArgumentException when using lockCanvas() in onPreviewFrame.
Basically what I want to achieve is to grab the frame, process it, and draw it directly to the surface of SurfacePreview.
here is my code of onPreviewFrame
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
Canvas canvas = null;
if (mHolder == null) {
return;
}
int mImgFormat = mCamera.getParameters().getPreviewFormat();
try {
canvas = mHolder.lockCanvas();
canvas.drawColor(0, android.graphics.PorterDuff.Mode.CLEAR);
mHolder.unlockCanvasAndPost(canvas);
} catch (Exception e) {
e.printStackTrace();
} finally {
if (canvas != null)
{
mHolder.unlockCanvasAndPost(canvas);
}
}
}
I have read a lot of documentation and topics about camera in android, and I suppose that locking the surface that frames are drawn onto isn't possible, because it's beyond scope of the application control. Is it true?
One of possible solutions is to draw on top of the surface view with another view, but I want to keep my processed frames (there will be some canny edge detection and color corrections, which may be time consuming) drawn in sync with preview. When camera will keep spitting frames with good fps my calculations will have a hard time catching up, and in worst case scenario draw an overlay that is dozens of frames behind.
I studied openCV source and for me it looks like they've managed to do this in CameraBridgeViewBase.java:
protected void deliverAndDrawFrame(CvCameraViewFrame frame) {
Mat modified;
if (mListener != null) {
modified = mListener.onCameraFrame(frame);
} else {
modified = frame.rgba();
}
boolean bmpValid = true;
if (modified != null) {
try {
Utils.matToBitmap(modified, mCacheBitmap);
} catch(Exception e) {
Log.e(TAG, "Mat type: " + modified);
Log.e(TAG, "Bitmap type: " + mCacheBitmap.getWidth() + "*" + mCacheBitmap.getHeight());
Log.e(TAG, "Utils.matToBitmap() throws an exception: " + e.getMessage());
bmpValid = false;
}
}
if (bmpValid && mCacheBitmap != null) {
Canvas canvas = getHolder().lockCanvas();
if (canvas != null) {
canvas.drawColor(0, android.graphics.PorterDuff.Mode.CLEAR);
canvas.drawBitmap(mCacheBitmap, (canvas.getWidth() - mCacheBitmap.getWidth()) / 2, (canvas.getHeight() - mCacheBitmap.getHeight()) / 2, null);
if (mFpsMeter != null) {
mFpsMeter.measure();
mFpsMeter.draw(canvas, 20, 30);
}
getHolder().unlockCanvasAndPost(canvas);
}
}
}
I'm either missing something or I'm just too old for this :)
Also, it's my first post here so hello everyone :)
Actually, openCV's camera plays a little trick. In the CameraBridgeViewBase.java and JavaCameraView.java file. I found that a SurfaceTexture class(from api 11) is used to receive frames from the camera. The advantage of the SurfaceTexture is you do not need to always draw it on the screen(it can be contained in the memory). Then a SurfaceView class is responsible to draw the processed frame from the onPreviewFrame function. Note this SurfaceView is not bound to the camera. Hope this can be help.
Related
I have a SurfaceView object with a draw method that gets called by 1 thread about 600 times from inside a for loop. No other thread calls draw during this time.
The draw method:
public void draw() {
if(this.surface_created){
long now = System.nanoTime();
Canvas canvas = holder.lockCanvas();
Log.d("GAME_SURFACE", "Time to lock:" + (System.nanoTime()-now));
if (canvas == null)
{
System.out.println("[GameSurface] Cannot draw onto canvas as it's null");
}
else
{
canvas.drawBitmap(this.bitmap, null, this.dest, null);
}
holder.unlockCanvasAndPost(canvas);
}
else{
System.out.println("[GameSurface] SURFACE NOT CREATED YET.");
}
}
The problem happens when i use holder.lockCanvas , which takes around 14 million nanoseconds. So to draw all bitmaps it ends up taking 13 seconds to do so. Why is it taking so long? Is there a workaround?
I am having difficulty when I am trying to live stream from android device to RTMP. First it was streaming good however, it was tilted so got a code to rotate the image.
public IplImage rotateImage(IplImage img) {
IplImage img_rotate = IplImage.create(img.height(), img.width(),
img.depth(), img.nChannels());
//acctually, I don't know how to use these two methods properly
cvTranspose(img, img_rotate);
cvFlip(img_rotate, img_rotate, 1); //?????90?
return img_rotate;
}
My code where I call the rotate from onPreviewFrame():
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
if (yuvIplimage != null && recording) {
videoTimestamp = 1000 * (System.currentTimeMillis() - startTime);
// Put the camera preview frame right into the yuvIplimage object
yuvIplimage.getByteBuffer().put(data);
try {
// Get the correct time
recorder.setTimestamp(videoTimestamp);
// Record the image into FFmpegFrameRecorder
recorder.record(rotateImage(yuvIplimage));
} catch (FFmpegFrameRecorder.Exception e) {
Log.v(LOG_TAG,e.getMessage());
e.printStackTrace();
}
}
}
As a result of this, I am getting the below image
In some other post, I came to know that we have to use
rgbimage = IplImage.create(imageWidth, imageHeight, IPL_DEPTH_8U, 3);
opencv_imgproc.cvCvtColor(yuvimage, rgbimage, opencv_imgproc.CV_YUV2BGR_NV21);
Then we can rotate however when ever I use this, I get exception BufferOverflow sometimes and sometimes RuntimeException
I'm currently developing an Android face detection app, and I'm wondering how I can slow down the capture frame rate purposefully. Currently it is around 30fps, and all I really require is 5-10fps. That way I don't need to use up additional processing which can be used on other tasks.
I'm wondering if a Thread.sleep() is all that is needed to do the trick, or should I look into setting it via cvSetCaptureProperty(CvCapture* capture, int property_id, double value)? I read that it only works on certain cameras though, and is, for the most part, useless...
I have also read about setting maximum frame size (e.g. mOpenCvCameraView.setMaxFrameSize(640, 480);) but... it doesn't make sense to me to do that?..
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
mRgba = inputFrame.rgba();
mGray = inputFrame.gray();
if (mAbsoluteFaceSize == 0) {
int height = mGray.rows();
if (Math.round(height * mRelativeFaceSize) > 0) {
mAbsoluteFaceSize = Math.round(height * mRelativeFaceSize);
}
mNativeDetector.setMinFaceSize(mAbsoluteFaceSize);
}
MatOfRect faces = new MatOfRect();
if (mDetectorType == JAVA_DETECTOR) {
if (mJavaDetector != null)
mJavaDetector.detectMultiScale(mGray, faces, SCALE_FACTOR, MIN_NEIGHBOURS, 2,
new Size(mAbsoluteFaceSize, mAbsoluteFaceSize), new Size());
}
else if (mDetectorType == NATIVE_DETECTOR) {
if (mNativeDetector != null)
mNativeDetector.detect(mGray, faces);
}
else {
Log.e(TAG, "Detection method is not selected!");
}
//put thread to sleep to slow capture?
try {
Thread.sleep(50);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return mRgba;
}
Any advice is greatly appreciated! Thanks.
I don't recommend to you to use cvSetCaptureProperty() because its behaviour is very rhapsodic.
You should rather register the timestamp of last frame arrived (and processed) to onCameraFrame() and return from the event handler if the difference between last timestamp and now is less then ~100 ms.
You can use a counter. Let's say fps=30, and you want to process only 5fs, then we have:
class YourClass{
int counter = 0;
// Your code
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
counter++;
if (counter < 6)
return null;
counter = 0;
//Your processing code here
}
}
Running:
SystemClock.sleep(...);
in onCameraFrame works well for me and reduces power usage of the app.
I'm puzzled by OpenCV's Android camera sample code. They make a custom class which implements SurfaceHolder.Callback and put the following line inside the method surfaceChanged:
mCamera.setPreviewDisplay(null);
The Android documentation for setPreviewDisplay explains:
This method must be called before startPreview(). The one exception is
that if the preview surface is not set (or set to null) before
startPreview() is called, then this method may be called once with a
non-null parameter to set the preview surface. (This allows camera
setup and surface creation to happen in parallel, saving time.) The
preview surface may not otherwise change while preview is running.
Unusually, OpenCV's code never calls setPreviewDisplay with a non-null SurfaceHolder. It works fine, but changing the rotation of the image using setDisplayOrientation doesn't work. This line also doesn't appear to do anything, since I get the same results without it.
If I call setPreviewDisplay with the SurfaceHolder supplied to surfaceChanged instead of null, the image rotates but does not include the results of the image processing. I also get an IllegalArgumentException when calling lockCanvas later on.
What's going on?
Here are the (possibly) most relevant parts of their code, slightly simplified and with methods inlined. Here is the full version.
Class definition
public abstract class SampleViewBase extends SurfaceView
implements SurfaceHolder.Callback, Runnable {
When the camera is opened
mCamera.setPreviewCallbackWithBuffer(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera) {
synchronized (SampleViewBase.this) {
System.arraycopy(data, 0, mFrame, 0, data.length);
SampleViewBase.this.notify();
}
camera.addCallbackBuffer(mBuffer);
}
});
When the surface changes
/* Now allocate the buffer */
mBuffer = new byte[size];
/* The buffer where the current frame will be copied */
mFrame = new byte [size];
mCamera.addCallbackBuffer(mBuffer);
try {
mCamera.setPreviewDisplay(null);
} catch (IOException e) {
Log.e(TAG, "mCamera.setPreviewDisplay/setPreviewTexture fails: " + e);
}
[...]
/* Now we can start a preview */
mCamera.startPreview();
The run method
public void run() {
mThreadRun = true;
Log.i(TAG, "Starting processing thread");
while (mThreadRun) {
Bitmap bmp = null;
synchronized (this) {
try {
this.wait();
bmp = processFrame(mFrame);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
if (bmp != null) {
Canvas canvas = mHolder.lockCanvas();
if (canvas != null) {
canvas.drawBitmap(bmp, (canvas.getWidth() - getFrameWidth()) / 2,
(canvas.getHeight() - getFrameHeight()) / 2, null);
mHolder.unlockCanvasAndPost(canvas);
}
}
}
Log.i(TAG, "Finishing processing thread");
}
I ran into this same problem. Instead of using a SurfaceView.Callback, I subclassed their class JavaCameraView. See my live face detection and drawing sample here. It was then trivial to rotate the matrix coming out of the camera according to the device's orientation, prior to processing. Relevant excerpt of linked code:
#Override
public Mat onCameraFrame(Mat inputFrame) {
int flipFlags = 1;
if(display.getRotation() == Surface.ROTATION_270) {
flipFlags = -1;
Log.i(VIEW_LOG_TAG, "Orientation is" + getRotation());
}
Core.flip(inputFrame, mRgba, flipFlags);
inputFrame.release();
Imgproc.cvtColor(mRgba, mGray, Imgproc.COLOR_RGBA2GRAY);
if (mAbsoluteFaceSize == 0) {
int height = mGray.rows();
if (Math.round(height * mRelativeFaceSize) > 0) {
mAbsoluteFaceSize = Math.round(height * mRelativeFaceSize);
}
}
}
I solved the rotation issue using OpenCV itself: after finding out how much the screen rotation needs to be corrected using this code, I apply a rotation matrix to the raw camera image (after converting from YUV to RGB):
Point center = new Point(mFrameWidth/2, mFrameHeight/2);
Mat rotationMatrix = Imgproc.getRotationMatrix2D(center, totalRotation, 1);
[...]
Imgproc.cvtColor(mYuv, mIntermediate, Imgproc.COLOR_YUV420sp2RGBA, 4);
Imgproc.warpAffine(mIntermediate, mRgba, rotationMatrix,
new Size(mFrameHeight, mFrameWidth));
A separate issue is that setPreviewDisplay(null) gives a blank screen on some phones. The solution, which I got from here and draws on this bugreport and this SO question, passes a hidden, "fake" SurfaceView to the preview display to get it to start, but actually displays the output on an overlaid custom view, which I call CameraView. So, after calling setContentView() in the activity's onCreate(), stick in this code:
if (VERSION.SDK_INT < VERSION_CODES.HONEYCOMB) {
final SurfaceView fakeView = new SurfaceView(this);
fakeView.setLayoutParams(new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT));
fakeView.setZOrderMediaOverlay(false);
final CameraView cameraView = (CameraView) this.findViewById(R.id.cameraview);
cameraView.setZOrderMediaOverlay(true);
cameraView.fakeView = fakeView;
}
Then, when setting the preview display, use this code:
try {
if (VERSION.SDK_INT >= VERSION_CODES.HONEYCOMB)
mCamera.setPreviewTexture(new SurfaceTexture(10));
else
mCamera.setPreviewDisplay(fakeView.getHolder());
} catch (IOException e) {
Log.e(TAG, "mCamera.setPreviewDisplay fails: "+ e);
}
If you are only developing for Honeycomb and above, just replace setPreviewDisplay(null) with mCamera.setPreviewTexture(new SurfaceTexture(10)); and be done with it. setDisplayOrientation() still doesn't work if you do this, though, so you'll still have to use the rotation matrix solution.
Hi I am not getting a smooth move, in my game train moves front and back it is nice, but train movement is not smooth(like not train move fixed speed and smooth). Please help............
public void run() {
Canvas canvas = null;
while (mRun) {
long beginTimeMillis, timeTakenMillis, timeLeftMillis;
canvas = mHolder.lockCanvas();
if (canvas != null) {
beginTimeMillis = System.currentTimeMillis();
gp.doDraw(canvas);
gp.animate();
timeTakenMillis = System.currentTimeMillis() - beginTimeMillis;
timeLeftMillis = (1000L / 30) - timeTakenMillis;
Log.i("timeLeftMillis"+timeLeftMillis,"");
mHolder.unlockCanvasAndPost(canvas);
if (timeLeftMillis < 5) {
timeLeftMillis = 5;
}
try {
TimeUnit.MILLISECONDS.sleep(timeLeftMillis);
} catch (InterruptedException ie) {
}
}
}
}
Edit: I don't know how to use the thread while getting a smooth move.
are you using View or SurfaceView. If your using View, use surfaceview. It will make your application much smoother.