Im trying to do some image processing using openCV on the array returned from onPreviewFrame(byte[] rawData), i have converted the byte array into int array.
my question is, how to convert this int array into Iplimage or Mat ?
will converting to the new Camera2 be any good?
i have tried this way to convert the array.
void ProcImage(int Width, int Height, int imageArray[]) {
cv::Mat mat = cv::Mat(Width, Height, CV_32SC1 , imageArray);
}
but mat.data contains values like -14545898.
any solutions ?
thanks in advance
This is most likely due to the Mat type CV_32SC1 that you used.
You don't even have to convert the int array, and use the byte array as is
cv::Mat mat = new cv::Mat(Width, Height, CvType.CV_8UC3, byteArray);
where byteArray is your byte array.
If you still want to use the int array trying changing the type to one of Mat: CV_32SC1，CV_32SC2，CV_32SC3，CV_32SC4 as specified in http://ninghang.blogspot.com/2012/11/list-of-mat-type-in-opencv.html
Directly in Java you could do something like this (it is based on the implementation of JavaCameraView):
...
Mat frameData = new Mat(frameHeight + (frameHeight/2), frameWidth, CvType.CV_8UC1);
#Override
public void onPreviewFrame(byte[] frame, Camera arg1) {
Log.d(TAG, "Preview Frame received. Frame size: " + frame.length);
synchronized (this) {
frameData.put(0, 0, frame);
...
this.notify();
}
}
public Mat getRgbaFrame() {
Mat rgba = new Mat();
Imgproc.cvtColor(frameData, rgba, Imgproc.COLOR_YUV2RGBA_NV21, 4);
return mRgba;
}
public Mat getGrayFrame() {
return frameData.submat(0, frameHeight, 0, frameWidth);
}
Related
I am trying face detection and adding mask(graphic overlay) using google vision api ,the problem is i could not get the ouptut from camera after detecting and adding mask.so far I have tried this solution from github , https://github.com/googlesamples/android-vision/issues/24 ,based on this issue i have added a custom detector class,
Mobile Vision API - concatenate new detector object to continue frame processing . and added this on mydetector class How to create Bitmap from grayscaled byte buffer image? .
MyDetectorClass
class MyFaceDetector extends Detector<Face>
{
private Detector<Face> mDelegate;
MyFaceDetector(Detector<Face> delegate) {
mDelegate = delegate;
}
public SparseArray<Face> detect(Frame frame) {
// *** add your custom frame processing code here
ByteBuffer byteBuffer = frame.getGrayscaleImageData();
byte[] bytes = byteBuffer.array();
int w = frame.getMetadata().getWidth();
int h = frame.getMetadata().getHeight();
YuvImage yuvimage=new YuvImage(bytes, ImageFormat.NV21, w, h, null);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
yuvimage.compressToJpeg(new Rect(0, 0, w, h), 100, baos); // Where 100 is the quality of the generated jpeg
byte[] jpegArray = baos.toByteArray();
Bitmap bitmap = BitmapFactory.decodeByteArray(jpegArray, 0, jpegArray.length);
Log.e("got bitmap","bitmap val " + bitmap);
return mDelegate.detect(frame);
}
public boolean isOperational() {
return mDelegate.isOperational();
}
public boolean setFocus(int id) {
return mDelegate.setFocus(id);
}
}
frame processing
public SparseArray<Face> detect(Frame frame)
{
// *** add your custom frame processing code here
ByteBuffer byteBuffer = frame.getGrayscaleImageData();
byte[] bytes = byteBuffer.array();
int w = frame.getMetadata().getWidth();
int h = frame.getMetadata().getHeight();
YuvImage yuvimage=new YuvImage(bytes, ImageFormat.NV21, w, h, null);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
yuvimage.compressToJpeg(new Rect(0, 0, w, h), 100, baos); // Where 100 is the quality of the generated jpeg
byte[] jpegArray = baos.toByteArray();
Bitmap bitmap = BitmapFactory.decodeByteArray(jpegArray, 0, jpegArray.length);
Log.e("got bitmap","bitmap val " + bitmap);
return mDelegate.detect(frame);
}
i am getting a rotated bitmap ,that is without the mask (graphic overlay) i have added .How can i get the camera output with mask .
Thanks in advance.
The simple answer is: You can't.
Why? Android camera output frames in NV21 ByteBuffer. And you must generate your masks based on the landmarks points in a separated Bitmap, then join them.
Sorry but, that's how the Android Camera API work. Nothing can be done. You must do it manually.
Also, I wouldn't get the camera preview then convert it to YuvImage then to Bitmap. That process consumes a lot of resources and makes preview very very slow. Instead I would use this method which will be a lot faster and rotates your preview internally so you don't loose time doing it:
outputFrame = new Frame.Builder().setImageData(mPendingFrameData, mPreviewSize.getWidth(), mPreviewSize.getHeight(), ImageFormat.NV21)
.setId(mPendingFrameId)
.setTimestampMillis(mPendingTimeMillis)
.setRotation(mRotation)
.build();
mDetector.receiveFrame(outputFrame);
All the code can be found in CameraSource.java
In my android app, I am trying to take frames in "onPreviewFrame" and convert them to Mat object so i can do on them some image processing, but i don't know how to convert them back to byte[] type and make them be shown on camera preview.
Does any one know how to do that?
Here is my onPreviewFrame function, i am trying to change frame color to grey:
public void onPreviewFrame(byte[] data, Camera arg1)
{
Mat mRgba = new Mat(PreviewSizeWidth, PreviewSizeHeight, CvType.CV_8UC3);
mRgba.put(0, 0, data);
Imgproc.cvtColor(mRgba, mRgba, Imgproc.COLOR_BGR2GRAY);
mRgba.put(0, 0, data);
mCamera.addCallbackBuffer(data);
}
Thanks in advance!
From an older implementation of the OpenCV camera viewer pipeline:
private Mat mYuv, mGraySubmat;
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
if(mYuv == null) {
mYuv = new Mat(height + height / 2, width, CvType.CV_8UC1);
mGraySubmat = mYuv.submat(0, height, 0, width);
}
mYuv.put(0, 0, data);
Imgproc.cvtColor(mGraySubmat, mRgba, Imgproc.COLOR_GRAY2RGBA, 4);
}
I think you can just call the addCallbackBuffer method in the end and it'd work fine!
My goal is to take bytearray at onPreviewFrame convert it into openCv MAT for some processing and later display processed frame in imageview.
My implementation of onPreviewFrame is as follows:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
Mat cvMat=Byte_to_Mat(data);
Imgproc.cvtColor(cvMat, cvMat, Imgproc.COLOR_YUV2BGR_NV12, 4);
Imgproc.cvtColor(cvMat, cvMat, Imgproc.COLOR_BGR2GRAY, 4);
Bitmap bm = Bitmap.createBitmap(cvMat.cols(), cvMat.rows(), Bitmap.Config.ARGB_8888);
public Mat Byte_to_Mat(byte[] data) {
Mat jpegData = new Mat(1, data.length, CvType.CV_8UC1);
jpegData.put(0, 0, data);
Mat bgrMat = new Mat();
bgrMat = Imgcodecs.imdecode(jpegData, Imgcodecs.IMREAD_COLOR);
return bgrMat;
}
However Bitmap.createBitmap throws following exception
java.lang.IllegalArgumentException: width and height must be > 0
When capturing frames with the camera in my Android app the images contain a lot of noise.
Here are 3 frames i have captured with my device if you look at the on the right of these frames you see that every frame has this dark gray noise. I need the frames to be the same if that is possible.
Does anyone know where this noise is coming from and more important how to get rid of it?
#Override
public void onFocus(byte[] data) {
frame = data;
processFrame();
}
private void processFrame() {
int height = mCameraPreview.getFrameHeight();
int width = mCameraPreview.getFrameWidth();
Mat mYuv = new Mat( height + height/2, width, CvType.CV_8UC1 );
mYuv.put( 0, 0, frame );
Mat mRgba = new Mat();
Imgproc.cvtColor( mYuv, mRgba, Imgproc.COLOR_YUV2RGBA_NV21, 4 );
filter.apply(mRgba);
}
In my filter method I make use of OpenCV template matching to find this specific part of the image. The objectDetected method saves the result as an image.
#Override
public void apply(Mat src) {
Mat match = templateMatch(src, template1, 0.07);
if(match != null){
Mat match2 = templateMatch(match, template2, 0.04);
if(match2 != null){
Mat gray = new Mat(match2.size(), CvType.CV_8U);
Imgproc.cvtColor(match2, gray, Imgproc.COLOR_BGR2GRAY);
objectDetectedListener.objectDetected(gray, "frame");
I get a runtime error if I try to convert camera preview YUV byte array
to a RGB(A) byte array with Imgproc.cvtColor( mYUV_Mat, mRgba_Mat, Imgproc.COLOR_YUV420sp2RGBA, 4 )
in onPreviewFrame(byte[] data, Camera camera):
Preview.java:
mCamera.setPreviewCallback(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera)
{
// Pass YUV data to draw-on-top companion
System.arraycopy(data, 0, mDrawOnTop.mYUVData, 0, data.length);
mDrawOnTop.invalidate();
}
});
DrawOnTop.java:
public class DrawOnTop extends View {
Bitmap mBitmap;
Mat mYUV_Mat;
protected void onDraw(Canvas canvas) {
if (mBitmap != null)
{
canvasWidth = canvas.getWidth();
canvasHeight = canvas.getHeight();
int newImageWidth = 640;
int newImageHeight = 480;
marginWidth = (canvasWidth - newImageWidth)/2;
if( mYUV_Mat != null ) mYUV_Mat.release();
//mYUV_Mat = new Mat( newImageWidth, newImageHeight, CvType.CV_8UC1 );
mYUV_Mat = new Mat( newImageWidth, newImageHeight, CvType.CV_8UC4 );
mYUV_Mat.put( 0, 0, mYUVData );
//Mat mRgba_Mat = new Mat();
Mat mRgba_Mat = new Mat(newImageWidth,newImageHeight,CvType.CV_8UC4);
//Mat mRgba_Mat = mYUV_Mat;
//Imgproc.cvtColor( mYUV_Mat, mRgba_Mat, Imgproc.COLOR_YUV2RGBA_NV21, 4 );
//Imgproc.cvtColor( mYUV_Mat, mRgba_Mat, Imgproc.COLOR_YUV420sp2RGB, 4 );
Imgproc.cvtColor( mYUV_Mat, mRgba_Mat, Imgproc.COLOR_YUV420sp2RGBA, 4 );
// Draw Bitmap New:
Bitmap mBitmap = Bitmap.createBitmap( newImageWidth, newImageHeight, Bitmap.Config.ARGB_8888 );
Utils.matToBitmap( mRgba_Mat, mBitmap );
mRgba_Mat.release();
}
}
The conversion mYUV_Mat.put( 0, 0, mYUVData ) runs correctly.
But the attempts to convert mYUV_Mat to mRgb_Mat using Imgproc.cvtColor
lead all to runtime errors ("Source not found." with Eclipse).
What is the correct Imgproc.cvtColor command for my goal?
(I don't want to use a Java YUV2RGB(A) decode method because it's to slow
for image processing. I want to use the OpenCV Imgproc.cvtColor method
because I can do native calls)
Maybe the Imgproc library isn't properly included in your project, but other OpenCV libraries are? The line that crashes is the first line where you use a method from Imgproc, which would explain why earlier parts of the code run correctly.
Your code looks fine, except you can use the no-argument constructor for mRgba_Mat (since most OpenCV4Android functions, including cvtColor, can infer the required size of the destination matrix), and you're allocating a lot of wasted space for mYUV_Mat. You don't need a full 4 channels if you just allocate YUV matrices 50% more space than their RGB counterparts:
mYUV_Mat = new Mat( newImageHeight + newImageHeight / 2, newImageWidth, CvType.CV_8UC1 );