I'm making a line follower for my robot on Android (to learn Java/Android programming), currently I'm facing the image processing problem: the camera preview returns an image format called YUV which I want to convert to a threshold in order to know where the line is, how would one do that?
As of now I've succeeded getting something, that is I definitely can read data from the camera preview and by some miracle even know if the light intensity is over or under a certain value at a certain area on the screen. My goal is to draw the robot's path on an overlay over the camera's preview, that too works to some extent, but the problem is the YUV management.
As you can see not only the dark area is drawn sideways, but it also repeats itself 4 times and the preview image is stretched, I cannot figure out how to fix these problems.
Here's the relevant part of code:
public void surfaceCreated(SurfaceHolder arg0) {
// TODO Auto-generated method stub
// camera setup
mCamera = Camera.open();
Camera.Parameters parameters = mCamera.getParameters();
List<Camera.Size> sizes = parameters.getSupportedPreviewSizes();
for(int i=0; i<sizes.size(); i++)
{
Log.i("CS", i+" - width: "+sizes.get(i).width+" height: "+sizes.get(i).height+" size: "+(sizes.get(i).width*sizes.get(i).height));
}
// change preview size
final Camera.Size cs = sizes.get(8);
parameters.setPreviewSize(cs.width, cs.height);
// initialize image data array
imgData = new int[cs.width*cs.height];
// make picture gray scale
parameters.setColorEffect(Camera.Parameters.EFFECT_MONO);
parameters.setFocusMode(Camera.Parameters.FOCUS_MODE_AUTO);
mCamera.setParameters(parameters);
// change display size
LayoutParams params = (LayoutParams) mSurfaceView.getLayoutParams();
params.height = (int) (mSurfaceView.getWidth()*cs.height/cs.width);
mSurfaceView.setLayoutParams(params);
LayoutParams overlayParams = (LayoutParams) swOverlay.getLayoutParams();
overlayParams.width = mSurfaceView.getWidth();
overlayParams.height = mSurfaceView.getHeight();
swOverlay.setLayoutParams(overlayParams);
try
{
mCamera.setPreviewDisplay(mSurfaceHolder);
mCamera.setDisplayOrientation(90);
mCamera.startPreview();
}
catch (IOException e)
{
e.printStackTrace();
mCamera.stopPreview();
mCamera.release();
}
// callback every time a new frame is available
mCamera.setPreviewCallback(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera)
{
// create bitmap from camera preview
int pixel, pixVal, frameSize = cs.width*cs.height;
for(int i=0; i<frameSize; i++)
{
pixel = (0xff & ((int) data[i])) - 16;
if(pixel < threshold)
{
pixVal = 0;
}
else
{
pixVal = 1;
}
imgData[i] = pixVal;
}
int cp = imgData[(int) (cs.width*(0.5+(cs.height/2)))];
//Log.i("CAMERA", "Center pixel RGB: "+cp);
debug.setText("Center pixel: "+cp);
// process preview image data
Paint paint = new Paint();
paint.setColor(Color.YELLOW);
int start, finish, last;
start = finish = last = -1;
float x_ratio = mSurfaceView.getWidth()/cs.width;
float y_ratio = mSurfaceView.getHeight()/cs.height;
// display calculated path on overlay using canvas
Canvas overlayCanvas = overlayHolder.lockCanvas();
overlayCanvas.drawColor(0, Mode.CLEAR);
// start by finding the tape from bottom of the screen
for(int y=cs.height; y>0; y--)
{
for(int x=0; x<cs.width; x++)
{
pixel = imgData[y*cs.height+x];
if(pixel == 1 && last == 0 && start == -1)
{
start = x;
}
else if(pixel == 0 && last == 1 && finish == -1)
{
finish = x;
break;
}
last = pixel;
}
//overlayCanvas.drawLine(start*x_ratio, y*y_ratio, finish*x_ratio, y*y_ratio, paint);
//start = finish = last = -1;
}
overlayHolder.unlockCanvasAndPost(overlayCanvas);
}
});
}
This code generates an error sometimes when quitting the application due to some method being called after release, which is the least of my problems.
UPDATE:
Now that the orientation problem is fixed (CCD sensor orientation) I'm still facing the repetition problem, this is probably related to my YUV data management...
Your surface and camera management looks correct, but I would doublecheck that camera actually accepted preview size settings ( some camera implementations reject some settings silently)
As you are working with portrait mode, you have to keep in mind that camera does not give a fart about prhone orientation - its coordinate origin isdetermined by CCD chip and is always to right corner and scan direction is from top to bottom and right to left - quite different from your overlay canvas. ( But if you are in landscape mode, everything is correct ;) ) - this is certaily source of odd drawing result
Your threshloding is bit naive and not very usefull in real life - I would suggest adaptive threshloding. In our javaocr project ( pure java, also has android demos ) we implemented efficient sauvola binarisation (see demos):
http://sourceforge.net/projects/javaocr/
Performance binarisation can be improved to work only on single image rows (patches welcome)
Issue with UV part of image is easy - default forman is NV21, luminance comes first
and this is just byte stream, and you do not need UV part of image at all (look into demos above)
Related
I am trying to use Camera2 to allow an app to take a simple picture. I managed to get a working sample using android-Camera2Basic sample code, the problem is that the camera preview is very dark (same problem as this other question), following some answers i did get a proper FPS range [15, 15], setting this in the lockFocus() method allows the app to great a clear picture with correct brightness and fixes the preview from the camera:
private void lockFocus() {
try {
// This is how to tell the camera to lock focus.
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AF_TRIGGER, CameraMetadata.CONTROL_AF_TRIGGER_START);
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, Range.create(15, 15));
// Tell #mCaptureCallback to wait for the lock.
mState = STATE_WAITING_LOCK;
mCaptureSession.capture(mPreviewRequestBuilder.build(), mCaptureCallback, mBackgroundHandler);
} catch (CameraAccessException e) {
e.printStackTrace();
}
}
But the preview before taking the pic is still very dark. I tried to set up the same line of code in other parts of the sample but is not working. How can i fix it in order to ge the same results in the preview? I am working with a Samsung SM-P355M tablet.
Using an FPS range with equal lower and upper bounds, such as [15,15], [30,30], [etc...], will put a constrain to the AE algorithm about how much it can adjust to light changes, which therefore may produce dark results. Such type of ranges are meant for video recording to maintain a constant FPS. For photos you need to find a range with a wide spread between the lower and upper bound, such as [7,30], [15,25], [etc...]
The next method can help you to find the optimal FPS range. Take into account that it is meant for photos and not video recording as it discards FPS ranges with equal lower and upper bounds.
(Adjust MIN_FPS_RANGE and MAX_FPS_RANGE to your requirements)
#Nullable
public static Range<Integer> getOptimalFpsRange(#NonNull final CameraCharacteristics characteristics) {
final int MIN_FPS_RANGE = 0;
final int MAX_FPS_RANGE = 30;
final Range<Integer>[] rangeList = characteristics.get(CameraCharacteristics.CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES);
if ((rangeList == null) || (rangeList.length == 0)) {
Log.e(TAG, "Failed to get FPS ranges.");
return null;
}
Range<Integer> result = null;
for (final Range<Integer> entry : rangeList) {
int candidateLower = entry.getLower();
int candidateUpper = entry.getUpper();
if (candidateUpper > 1000) {
Log.w(TAG,"Device reports FPS ranges in a 1000 scale. Normalizing.");
candidateLower /= 1000;
candidateUpper /= 1000;
}
// Discard candidates with equal or out of range bounds
final boolean discard = (candidateLower == candidateUpper)
|| (candidateLower < MIN_FPS_RANGE)
|| (candidateUpper > MAX_FPS_RANGE);
if (discard == false) {
// Update if none resolved yet, or the candidate
// has a >= upper bound and spread than the current result
final boolean update = (result == null)
|| ((candidateUpper >= result.getUpper()) && ((candidateUpper - candidateLower) >= (result.getUpper() - result.getLower())));
if (update == true) {
result = Range.create(candidateLower, candidateUpper);
}
}
}
return result;
}
After lots of reseach it seams there is no easy fix for this (at least not with our same hardware), so we implemented a new version of the camera activities this time using the deprecated Camera Api and everything works as espected. Not really a clean solution but so far works for me.
I have a Module to put cartoon face on eyes or anywhere else on live Camera Preview. I am using Moodme Sdk. I have implemented camera preview. I m getting landmark x and y axis value. But I don't know where do i add those landmark and how to put that image on eyes using landmark. This is code for while getting person face on live camera.
#Override
public void onImageAvailable(ImageReader reader) {
Image image = imageReader.acquireLatestImage();
if (image == null) {
return;
}
ByteBuffer yBuffer = image.getPlanes()[0].getBuffer();
byte[] yBytes = new byte[yBuffer.remaining()];
yBuffer.get(yBytes);
ByteBuffer uBuffer = image.getPlanes()[1].getBuffer();
byte[] uBytes = new byte[uBuffer.remaining()];
uBuffer.get(uBytes);
ByteBuffer vBuffer = image.getPlanes()[2].getBuffer();
byte[] vBytes = new byte[vBuffer.remaining()];
vBuffer.get(vBytes);
tracker.processImageBuffer(yBytes, WIDTH, HEIGHT, WIDTH, MDMTrackerManager.FrameFormat.GRAY);
//renderer.updateTextureImage(yBytes, uBytes, vBytes, image.getPlanes()[1].getPixelStride());
image.close();
if (tracker.isFaceTracked()) {
// renderer.updateVertices();
}
if (tracker.isFaceTracked()) {
// translate to opengl coordinates
float[] landmarks = new float[66*2];
for (int i = 0; i < 66; ++i) {
if(i >=17 && i <27 || i >=36 && i <48 ) {
landmarks[2 * i] = 1.0f - tracker.getLandmarks()[2 * i] / (HEIGHT / 2);
landmarks[2 * i + 1] = 1.0f - tracker.getLandmarks()[2 * i + 1] / (WIDTH / 2);
}
}
// renderer.updateLandmarks(landmarks);
} else {
// renderer.updateLandmarks(null);
}
long currentTime = System.currentTimeMillis();
double fps = 1000.0 / (currentTime - lastFrameTime);
updater.update(fps);
lastFrameTime = currentTime;
}
I have also used Face Detection Library But that is not giving me accurate result.Is There any good Library For Face detection and put image or Mask on Camera Preview. Any help will be appreciated.
There are many libraries available which add a face mask on camera preview. Almost all of them use OpenCV. Check out these libraries.
FaceFilter
Face Replace
FaceTracker
Android GPUimage
The Android GPUimage seems to add image on Camera Preview. A similar question used this library to add face mask on camera preview. You can take a look into the answer posted on the question.
The FaceFilter library does the same work, but on a captured image. However you can see the tutorial for the library posted by the author and integrate it with face detection. There are several tutorials for face detection. This tutorial explains how to implement face detection, while also overlaying graphics on it. Although there is not much on the overlaid graphics in the tutorial, it might solve your question.
I'm building an app for Android but The device do have a square screen. The screen is 320x320 and the camera app use the SurfaceView to show the preview as below :
mCameraView = (SurfaceView) findViewById(R.id.surface);
LayoutParams params = mCameraView.getLayoutParams();
int camera_dimension = getResources().getInteger(R.integer.camera_dimension);
params.height = camera_dimension; //320px
params.width = camera_dimension;
mCameraView.setLayoutParams(params);
mCameraViewHolder = mCameraView.getHolder();
mCameraViewHolder.addCallback(this);
mCameraViewHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
On Surface changed, I'm doing this and it works but the supporter preview is w480 x h320
public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
int mWidth = w;
int mHeight = h;
int mFormat = format;
try {
mCameraAccess.mCamera.stopPreview();
} catch (Exception e){
// ignore: tried to stop a non-existent preview
}
Camera.Parameters param = mCameraAccess.mCamera.getParameters();
List<Camera.Size> SupportedPreview = param.getSupportedPreviewSizes();
int ScreenSize = getResources().getInteger(R.integer.camera_dimension);
for (int i = 0; i < SupportedPreview.size(); i++) {
if (SupportedPreview.get(i).height == ScreenSize) {
param.setPreviewSize(SupportedPreview.get(i).width, SupportedPreview.get(i).height);
break;
}
}
mCameraAccess.mCamera.setParameters(param);
mCameraAccess.mCamera.startPreview();
}
How can make sure that my preview inside the viewholder is not compressed but a kind of centercrop. As the preview is a rectangle, I just need the square centered on the image. Usually I'm using scaleType but it's not supported in Surface view
Any idea ?
The solution which i have figured out is:
1)-Keep surface view full screen,so that is doesn't stretch.
2)-Put a view over surfaceview with full opacity.So that is looks like camera is already squared.
3)-After capturing image or video,you will have to crop them.
For video you have to use some video processing library like javacv.Using this library you can extract video frames,convert them to bitmap,crop bitmap in square and then re-encode into video.
To get accurate results you will need to play around with different techniques like zooming camera during capture etc. according to your needs.
Original Image:
Squared Image:
I am trying to set the best possible output picture size in my camera object. So that, i can get a perfect downscaled sample image and display it.
During debugging i observed i am setting output picture size exactly the size of my screen dimensions. But when i DecodeBounds of the returned image by camera. I get some larger number!
Also i am not setting my display dimensions as expected output picture size. Code used to calculate and set the output picture size is given below.
I am using this code for devices having API level < 21, so using camera shouldn't be a problem.
I don't have any idea of why i am getting this behavior. Thanks in advance for help!
Defining Camera parameter
Camera.Parameters parameters = mCamera.getParameters();
setOutputPictureSize(parameters.getSupportedPictureSizes(), parameters); //update paramters in this function.
//set the modified parameters back to mCamera
mCamera.setParameters(parameters);
Optimal picture size calculation
private void setOutputPictureSize(List<Camera.Size> availablePicSize, Camera.Parameters parameters)
{
if (availablePicSize != null) {
int bestScore = (1<<30); //set an impossible value.
Camera.Size bestPictureSize = null;
for (Camera.Size pictureSize : availablePicSize) {
int curScore = calcOutputScore(pictureSize); //calculate sore of the current picture size
if (curScore < bestScore) { //update best picture size
bestScore = curScore;
bestPictureSize = pictureSize;
}
}
if (bestPictureSize != null) {
parameters.setPictureSize(bestPictureSize.width, bestPictureSize.height);
}
}
}
//calculates score of a target picture size compared to screen dimensions.
//scores are non-negative where 0 is the best score.
private int calcOutputScore(Camera.Size pictureSize)
{
Point displaySize = AppData.getDiaplaySize();
int score = (1<<30);//set an impossible value.
if (pictureSize.height < displaySize.x || pictureSize.width < displaySize.y) {
return score; //return the worst possible score.
}
for (int i = 1; ; ++i) {
if (displaySize.x * i > pictureSize.height || displaySize.y * i > pictureSize.width) {
break;
}
score = Math.min(score, Math.max(pictureSize.height-displaySize.x*i, pictureSize.width-displaySize.y*i));
}
return score;
}
Finally i resolved the issue after many attempts! Below are my findings:
Step 1. If we are already previewing, call mCamera.stopPreview()
Step 2. Set modified parameters by calling mCamera.setParameters(...)
Step 3. Start previewing again, call mCamera.startPreview()
If i call mCamera.setParameters without stopping preview (Assuming camera is previewing). Camera seems to ignore the updated parameters.
I came up with this solution after several trail and errors. Anyone know better way to update parameters during preview please share.
I have found out that when using a textureview instead of a surfaceview as a camera preview (both hooked up to the camera via a mediarecorder) then the preview is much more fuzzy.
What I mean by fuzzy is that in a texture view you can see the pixels, especially when zooming. That is not the case when using a surfaceview. Why is that the case?
UPD:
Sorry,but after I re-write my shit code, the key is the preview size too small that caused "fuzziness", so you should set a reasonable preview Size,not the reason strikeout below, but auto-focus is suggested ...
Size size = getBestSupportSize(parameters.getSupportedPreviewSizes(), width, height);
parameters.setPreviewSize(size.width, size.height);
As to the method getBestSupportSize(), how to get the bestSize for your project needs, in this case, it is as large as the screen width andhe ratio is 4/3 your's may be some other, I calculate the ration dividing width/height.
private Size getBestSupportSize(List<Size> sizes, int width, int height) {
Size bestsize = sizes.get(0);
int screenWidth = getResources().getDisplayMetrics().widthPixels;
int dt = Integer.MAX_VALUE;
for (int i = sizes.size() - 1; i >= 0; i--) {
Log.d(TAG, "-index : " + i);
Size s = sizes.get(i);
if (s.width * 3.0f / 4 == s.height) {
int newDT = Math.abs(screenWidth - s.width);
if (newDT < dt && screenWidth < s.width) {
dt = newDT;
bestsize = s;
}
}
}
return bestsize;//note that if no "4/3" size supported,default return size[0]
}
So this "fuzziness" was caused by a small previewSize calcualate a best size for the camera using this getSupportedPreviewSizes() method
And I will keep the autoFocus snippet below, strikeout though, FYR if is needed.
Well i got the solution for this "fuzzy" problem,and my case is just using TextureView andsurfaceTexture to take a pic instead of old surfaceView withsurfaceHolderway.
The key is set this mCamera.autofocus(), why the pic is"fuzzy" is bacause we lack of this autoFocus setting.
like below :
mCamera.setPreviewTexture(surface);
//enable autoFocus if moving
mCamera.setAutoFocusMoveCallback(new AutoFocusMoveCallback() {
#Override
public void onAutoFocusMoving(boolean start, Camera camera) {
if (start) { //true means you are moving the camera
mCamera.autoFocus(myAutoFocus);
}
}
});
mCamera.startPreview();
The autoFocusCallback like this:
AutoFocusCallback myAutoFocus = new AutoFocusCallback() {
#Override
public void onAutoFocus(boolean success, Camera camera) {
}
};