Develop Reference

Android opencv byte[] to mat to byte[]

My goal is to add an overlay on the camera preview that will find book edges. For that, I override the onPreviewFrame where I do the following:
public void onPreviewFrame(byte[] data, Camera camera) {
Camera.Parameters parameters = camera.getParameters();
int width = parameters.getPreviewSize().width;
int height = parameters.getPreviewSize().height;
Mat mat = new Mat((int) (height*1.5), width, CvType.CV_8UC1);
mat.put(0,0,data);
byte[] bytes = new byte[(int) (height*width*1.5)];
mat.get(0,0,bytes);
if (!test) { //to only do once
File pictureFile = getOutputMediaFile();
try {
FileOutputStream fos = new FileOutputStream(pictureFile);
fos.write(bytes);
fos.close();
Uri picUri = Uri.fromFile(pictureFile);
updateGallery(picUri);
test = true;
} catch (IOException e) {
e.printStackTrace();
}
}
}
For now I simply want to take one of the previews and save it after the conversion to mat.
After spending countless hours getting the above to look right, the saved picture cannot be seen on my testing phone (LG Leon). I can't seem to find the issue. Am I mixing the height/width because I'm taking pictures in portrait mode? I tried switching them and still doesn't work. Where is the problem?

The fastest method I managed to find is described HERE in my recently asked question. You can find the method to extract the image in the answer I wrote in my question below. The thing is that the image you get through onPreviewFrame() is NV21. After receiving this image it may be that you need to convert it to RGB (depends on what do you want to achieve; this is also done in the answer I gave you previously).

Seems quite inefficient but it works for me (for now):
//get the camera parameters
Camera.Parameters parameters = camera.getParameters();
int width = parameters.getPreviewSize().width;
int height = parameters.getPreviewSize().height;
//convert the byte[] to Bitmap through YuvImage;
//make sure the previewFormat is NV21 (I set it so somewhere before)
YuvImage yuv = new YuvImage(data, parameters.getPreviewFormat(), width, height, null);
ByteArrayOutputStream out = new ByteArrayOutputStream();
yuv.compressToJpeg(new Rect(0, 0, width, height), 70, out);
Bitmap bmp = BitmapFactory.decodeByteArray(out.toByteArray(), 0, out.size());
//convert Bitmap to Mat; note the bitmap config ARGB_8888 conversion that
//allows you to use other image processing methods and still save at the end
Mat orig = new Mat();
bmp = bmp.copy(Bitmap.Config.ARGB_8888, true);
Utils.bitmapToMat(bmp, orig);
//here you do whatever you want with the Mat
//Mat to Bitmap to OutputStream to byte[] to File
Utils.matToBitmap(orig, bmp);
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bmp.compress(Bitmap.CompressFormat.JPEG, 70, stream);
byte[] bytes = stream.toByteArray();
File pictureFile = getOutputMediaFile();
try {
FileOutputStream fos = new FileOutputStream(pictureFile);
fos.write(bytes);
fos.close();
} catch (IOException e) {
e.printStackTrace();
}

Camera.PreviewCallback equivalent in Camera2 API

Is there any equivalent for Camera.PreviewCallback in Camera2 from API 21,better than mapping to a SurfaceTexture and pulling a Bitmap ? I need to be able to pull preview data off of the camera as YUV?

You can start from the Camera2Basic sample code from Google.
You need to add the surface of the ImageReader as a target to the preview capture request:
//set up a CaptureRequest.Builder with the output Surface
mPreviewRequestBuilder = mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
mPreviewRequestBuilder.addTarget(surface);
mPreviewRequestBuilder.addTarget(mImageReader.getSurface());
After that, you can retrieve the image in the ImageReader.OnImageAvailableListener:
private final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(ImageReader reader) {
Image image = null;
try {
image = reader.acquireLatestImage();
if (image != null) {
ByteBuffer buffer = image.getPlanes()[0].getBuffer();
Bitmap bitmap = fromByteBuffer(buffer);
image.close();
}
} catch (Exception e) {
Log.w(LOG_TAG, e.getMessage());
}
}
};
To get a Bitmap from the ByteBuffer:
Bitmap fromByteBuffer(ByteBuffer buffer) {
byte[] bytes = new byte[buffer.capacity()];
buffer.get(bytes, 0, bytes.length);
return BitmapFactory.decodeByteArray(bytes, 0, bytes.length);
}

Yes, use the ImageReader class.
Create an ImageReader using the format ImageFormat.YUV_420_888 and your desired size (make sure you select a size that's supported by the camera device you're using).
Then use ImageReader.getSurface() for a Surface to provide to CameraDevice.createCaptureSession(), along with your other preview outputs, if any.
Finally, in your repeating capture request, add the ImageReader provided surface as a target before setting it as the repeating request in your capture session.

Capturing Preview Frame in Portrait

Is there any way to acquire the preview frame directly in portrait inside onPreviewFrame method?
I've tried:
camera.setDisplayOrientation(90);
but this seems to work only for the display. The doc reports:
Set the clockwise rotation of preview display in degrees. This affects
the preview frames and the picture displayed after snapshot. This
method is useful for portrait mode applications.
This does not affect the order of byte array passed in onPreviewFrame(byte[], Camera), JPEG pictures, or recorded videos.
This method is not allowed to be called during preview.
I'm targetting API level >= 8, and I've a portrait locked app. I want to avoid manually rotating byte array of data passed as frame.
Many thanks in advance.

Try this it will work
public void takeSnapPhoto() {
camera.setOneShotPreviewCallback(new Camera.PreviewCallback() {
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
Camera.Parameters parameters = camera.getParameters();
int format = parameters.getPreviewFormat();
//YUV formats require more conversion
if (format == ImageFormat.NV21 || format == ImageFormat.YUY2 || format == ImageFormat.NV16) {
int w = parameters.getPreviewSize().width;
int h = parameters.getPreviewSize().height;
// Get the YuV image
YuvImage yuv_image = new YuvImage(data, format, w, h, null);
// Convert YuV to Jpeg
Rect rect = new Rect(0, 0, w, h);
ByteArrayOutputStream output_stream = new ByteArrayOutputStream();
yuv_image.compressToJpeg(rect, 100, output_stream);
byte[] byt = output_stream.toByteArray();
FileOutputStream outStream = null;
try {
// Write to SD Card
File file = createFileInSDCard(FOLDER_PATH, "Image_"+System.currentTimeMillis()+".jpg");
//Uri uriSavedImage = Uri.fromFile(file);
outStream = new FileOutputStream(file);
outStream.write(byt);
outStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
}
}
}
});}

Android byte[] to image in Camera.onPreviewFrame

When trying to convert the byte[] of Camera.onPreviewFrame to Bitamp using BitmapFactory.decodeByteArray gives me an error SkImageDecoder::Factory returned null
Following is my code:
public void onPreviewFrame(byte[] data, Camera camera) {
Bitmap bmp=BitmapFactory.decodeByteArray(data, 0, data.length);
}

This has been hard to find! But since API 8, there is a YuvImage class in android.graphics. It's not an Image descendent, so all you can do with it is save it to Jpeg, but you could save it to memory stream and then load into Bitmap Image if that's what you need.
import android.graphics.YuvImage;
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
try {
Camera.Parameters parameters = camera.getParameters();
Size size = parameters.getPreviewSize();
YuvImage image = new YuvImage(data, parameters.getPreviewFormat(),
size.width, size.height, null);
File file = new File(Environment.getExternalStorageDirectory()
.getPath() + "/out.jpg");
FileOutputStream filecon = new FileOutputStream(file);
image.compressToJpeg(
new Rect(0, 0, image.getWidth(), image.getHeight()), 90,
filecon);
} catch (FileNotFoundException e) {
Toast toast = Toast
.makeText(getBaseContext(), e.getMessage(), 1000);
toast.show();
}
}

Since Android 3.0 you can use a TextureView and TextureSurface to display the camera, and then use mTextureView.getBitmap() to retrieve a friendly RGB preview frame.
A very skeletal example of how to do this is given in the TextureView docs. Note that you'll have to set your application or activity to be hardware accelerated by putting android:hardwareAccelerated="true" in the manifest.

I found the answer after a long time. Here it is...
Instead of using BitmapFactory, I used my custom method to decode this byte[] data to a valid image format. To decode the image to a valid image format, one need to know what picture format is being used by the camera by calling camera.getParameters().getPictureFormat(). This returns a constant defined by ImageFormat. After knowing the format, use the appropriate encoder to encode the image.
In my case, the byte[] data was in the YUV format, so I looked for YUV to BMP conversion and that solved my problem.

you can try this:
This example send camera frames to server
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
try {
byte[] baos = convertYuvToJpeg(data, camera);
StringBuilder dataBuilder = new StringBuilder();
dataBuilder.append("data:image/jpeg;base64,").append(Base64.encodeToString(baos, Base64.DEFAULT));
mSocket.emit("newFrame", dataBuilder.toString());
} catch (Exception e) {
Log.d("########", "ERROR");
}
}
};
public byte[] convertYuvToJpeg(byte[] data, Camera camera) {
YuvImage image = new YuvImage(data, ImageFormat.NV21,
camera.getParameters().getPreviewSize().width, camera.getParameters().getPreviewSize().height, null);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
int quality = 20; //set quality
image.compressToJpeg(new Rect(0, 0, camera.getParameters().getPreviewSize().width, camera.getParameters().getPreviewSize().height), quality, baos);//this line decreases the image quality
return baos.toByteArray();
}

Best way to scale size of camera picture before saving to SD

The code below is executed as the jpeg picture callback after TakePicture is called. If I save data to disk, it is a 1280x960 jpeg. I've tried to change the picture size but that's not possible as no smaller size is supported. JPEG is the only available picture format.
PictureCallback jpegCallback = new PictureCallback() {
public void onPictureTaken(byte[] data, Camera camera) {
FileOutputStream out = null;
Bitmap bm = BitmapFactory.decodeByteArray(data, 0, data.length);
Bitmap sbm = Bitmap.createScaledBitmap(bm, 640, 480, false);
data.Length is something like 500k as expected. After executing BitmapFactory.decodeByteArray(), bm has a height and width of -1 so it appears the operation is failing.
It's unclear to me if Bitmap can handle jpeg data. I would think not but I have seem some code examples that seem to indicate it is.
Does data need to be in bitmap format before decoding and scaling?
If so, how to do this?
Thanks!

On your surfaceCreated, you code set the camara's Picture Size, as shown the code below:
public void surfaceCreated(SurfaceHolder holder) {
camera = Camera.open();
try {
camera.setPreviewDisplay(holder);
Camera.Parameters p = camera.getParameters();
p.set("jpeg-quality", 70);
p.setPictureFormat(PixelFormat.JPEG);
p.setPictureSize(640, 480);
camera.setParameters(p);
} catch (IOException e) {
e.printStackTrace();
}
}