So I have an application that uses CameraX ImageCapture use case to take a selfie picture than then is passed to an AI algorithm to do some stuff with it.
Now, I have an user with a Samsung Galaxy S21 who when taking pictures in one specific place with some specific light conditions is producing an image that doesn't work as expected with the AI algorithm. I have examined these images myself and noticed that the problem seems to be that ImageCapture applies strong noise reduction, so strong that even for the human eye it looks wrong, like if it was a painting instead of a photograph.
I sent a modified version of such app to this user for trying which captures the image from the Analysis use case instead and the produced image does not have that problem, so it seems whatever it is, it's some post-processing done by the ImageCapture use case that's not done in the Analysis use case.
Now, I don't seem to find a way to tweak this post-processing in CameraX, in fact I haven't even found how to do it with Camera2 neither. At first I thought it may be HDR, and I found there are some extensions to enable HDR, Night Mode and such in CameraX, but all these are disabled by default according to the documentation, and as far as you use the DEFAULT_FRONT_CAMERA none should be applied, and that's what I'm using.
CameraSelector.DEFAULT_FRONT_CAMERA
In any case it's clear that some heavy post-processing is being done to these images in the ImageCapture use case, so I'm wondering how could I disable these.
BTW, I tried initialising the ImageCapture use case with CAPTURE_MODE_MINIMIZE_LATENCY in the hope that such flag would reduce the post-processing and hopefully remove noise reduction, but that didn't work.
imageCapture = new ImageCapture.Builder()
.setTargetResolution(resolution)
.setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
.build();
Any ideas on how to go beyond this to get that noise reduction filter disabled?
Thanks,
Fran
I found a way using Camera2Interop.Extender:
private void initImageCapture(Size resolution) {
Log.d(TAG, "initCameraCapture: ");
ImageCapture.Builder imageCaptureBuilder = new ImageCapture.Builder();
imageCaptureBuilder
.setTargetResolution(resolution)
.setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY);
Camera2Interop.Extender extender = new Camera2Interop.Extender(imageCaptureBuilder);
extender.setCaptureRequestOption(CaptureRequest.NOISE_REDUCTION_MODE, CaptureRequest.NOISE_REDUCTION_MODE_OFF);
imageCapture = imageCaptureBuilder.build();
}
Related
So I have an issue with a device (Alcatel 5033D 1) that's always giving me a resolution of 480x640 for CameraX analysis. This is the last version of the code in which I init it:
private void initCameraAnalysis(Size resolution) {
Log.d(TAG, "initCameraAnalysis: ");
cameraAnalysis = new ImageAnalysis.Builder()
.setTargetResolution(resolution)
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setImageQueueDepth(0)
.build();
cameraAnalysis.setAnalyzer(
cameraExecutor,
new QualityAnalyzer()
);
}
I just added this part to see if it would help, initially these calls weren't there and they actually don't help with this issue:
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setImageQueueDepth(0)
The resolution is passed when calling that method as follows, and so I'm requesting 720x1280 that's what I need (using portrait orientation):
initCameraAnalysis(new Size(720, 1280));
Now, this is a low end device and always fallback to 480x640 but I have another app made by a colleague that still uses Camera2 and that one can make 720x1280 with this same device without issue, so I know for sure the device is actually capable of doing 720x1280.
Moreover, I have added an ImageCapture use case and that one can do 720x1280 without issue too, but I need it to do 720x1280 during the analysis.
So I wonder if there'd be any way to force 720x1280 in the analysis use case despite the stubbornness of CameraX not taking it, even if it means using some Camera2 extension or whatever.
Otherwise is likely I'd have to rewrite the whole app using Camera2 instead, which seems like going backwards in time...
Thanks a lot in advance!
I don't know If you have seen it but just in case you did not I would like to point what stated here:
Express the resolution Size in the coordinate frame after rotating the supported sizes by the target rotation. For example, a device with portrait natural orientation in natural target rotation requesting a portrait image may specify 480x640, and the same device, rotated 90 degrees and targeting landscape orientation may specify 640x480.
Maybe also give it a shot with:
val imageAnalysis = ImageAnalysis.Builder()
.setTargetResolution(Size(1280, 720))
.build()
I used the latest Camera2Basic sample program as a source for my trials:
https://github.com/android/camera-samples.git
Basically I configured the CaptureRequest before I call the capture() function in the takePhoto() function like this:
private fun prepareCaptureRequest(captureRequest: CaptureRequest.Builder) {
//set all needed camera settings here
captureRequest.set(CaptureRequest.CONTROL_MODE, CaptureRequest.CONTROL_MODE_OFF)
captureRequest.set(CaptureRequest.CONTROL_AF_MODE, CaptureRequest.CONTROL_AF_MODE_OFF);
//captureRequest.set(CaptureRequest.CONTROL_AF_TRIGGER, CaptureRequest.CONTROL_AF_TRIGGER_CANCEL);
//captureRequest.set(CaptureRequest.CONTROL_AWB_LOCK, true);
captureRequest.set(CaptureRequest.CONTROL_AWB_MODE, CaptureRequest.CONTROL_AWB_MODE_OFF);
captureRequest.set(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF);
//captureRequest.set(CaptureRequest.CONTROL_AE_LOCK, true);
//captureRequest.set(CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER, CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER_CANCEL);
//captureRequest.set(CaptureRequest.NOISE_REDUCTION_MODE, CaptureRequest.NOISE_REDUCTION_MODE_FAST);
//flash
if (mState == CaptureState.PRECAPTURE){
//captureRequest.set(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF);
captureRequest.set(CaptureRequest.FLASH_MODE, CaptureRequest.FLASH_MODE_OFF)
}
if (mState == CaptureState.TAKEPICTURE) {
//captureRequest.set(CaptureRequest.FLASH_MODE, CaptureRequest.FLASH_MODE_SINGLE)
//captureRequest.set(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_ON_ALWAYS_FLASH);
captureRequest.set(CaptureRequest.FLASH_MODE, CaptureRequest.FLASH_MODE_SINGLE)
}
val iso = 100
captureRequest.set(CaptureRequest.SENSOR_SENSITIVITY, iso)
val fractionOfASecond = 750.toLong()
captureRequest.set(CaptureRequest.SENSOR_EXPOSURE_TIME, 1000.toLong() * 1000.toLong() * 1000.toLong() / fractionOfASecond)
//val exposureTime = 133333.toLong()
//captureRequest.set(CaptureRequest.SENSOR_EXPOSURE_TIME, exposureTime)
//val characteristics = cameraManager.getCameraCharacteristics(cameraId)
//val configs: StreamConfigurationMap? = characteristics[CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP]
//val frameDuration = 33333333.toLong()
//captureRequest.set(CaptureRequest.SENSOR_FRAME_DURATION, frameDuration)
val focusDistanceCm = 20.0.toFloat() //20cm
captureRequest.set(CaptureRequest.LENS_FOCUS_DISTANCE, 100.0f / focusDistanceCm)
//captureRequest.set(CaptureRequest.COLOR_CORRECTION_MODE, CameraMetadata.COLOR_CORRECTION_MODE_FAST)
captureRequest.set(CaptureRequest.COLOR_CORRECTION_MODE, CaptureRequest.COLOR_CORRECTION_MODE_TRANSFORM_MATRIX)
val colorTemp = 8000.toFloat();
val rggb = colorTemperature(colorTemp)
//captureRequest.set(CaptureRequest.COLOR_CORRECTION_TRANSFORM, colorTransform);
captureRequest.set(CaptureRequest.COLOR_CORRECTION_GAINS, rggb);
}
but the picture that is returned never is the picture where the flash is at its brightest. This is on a Google Pixel 2 device.
As I only take one picture I am also not sure how to check some CaptureResult states to find the correct one as there is only one.
I already looked at the other solutions to similar problems here but they were either never really solved or somehow took the picture during capture preview which I don't want.
Other strange observations are that on different devices the images are taken (also not always at the right moment), but then the manual values I set are not observed in the JPEG metadata of the image.
If needed I can put my git fork on github.
Long exposure time in combination with flash seems to be the basic issue and when the results are not that good, this means that the timing of your preset isn't that good. You'd have to optimize the exposure time's duration, in relation to the flash's timing (just check the EXIF of some photos for example values). You could measure the luminosity with an ImageAnalysis.Analyzer (this had been removed from the sample application, but elder revisions still have an example). And I've tried with the default Motorola camera app; there the photo also seems to be taken shortly after the flash, when the brightness is already decaying (in order to avoid the dazzling bright). That's the CaptureState.PRECAPTURE, where you switch the flash off. Flashing in two stages is rather the default and this might yield better results.
If you want it to be dazzlingly bright (even if this is generally not desired), you could as well first switch on the torch, that the image, switch off the torch again (I use something alike this, but only for barcode scanning). This would at least prevent any expose/flash timing issues.
When changed values are not represented in EXIF, you'd need to use ExifInterface, in order to update them (there's an example which updates the orientation, but one can update any value).
So I migrated from using legacy camera api to CameraX and even though it was quite simple to setup, I've noticed one issue. Now camera seems to take almost twice if not longer to start showing preview than it had before.
I'm testing on galaxy s7.
My code looks like this:
val previewConfig = PreviewConfig.Builder().apply {
setTargetAspectRatio(Rational(1, 1))
setTargetResolution(Size(binding.codeScannerView.width, binding.codeScannerView.height))
}.build()
val preview = Preview(previewConfig)
preview.setOnPreviewOutputUpdateListener { preview ->
val parent = binding.codeScannerView.parent as ViewGroup
parent.removeView(binding.codeScannerView)
parent.addView(binding.codeScannerView, 0)
binding.codeScannerView.surfaceTexture = preview.surfaceTexture
}
val analyzerConfig = ImageAnalysisConfig.Builder().apply {
val analyzerThread = HandlerThread(
"QrCodeReader").apply { start() }
setCallbackHandler(Handler(analyzerThread.looper))
setImageReaderMode(
ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
}.build()
val analyzerUseCase = ImageAnalysis(analyzerConfig).apply {
analyzer = QrCodeAnalyzer(requireContext(), Handler(), { qrCode ->
if (activity == null) {
return#QrCodeAnalyzer
}
presenter.disableCameraPreview()
presenter.updateTable(qrCode.toLowerCase().parseTableId(), isFromOrder, Screens.MENU_SCREEN)
})
}
CameraX.bindToLifecycle(this, preview, analyzerUseCase)
Any ideas on how to make it appear faster?
P. S. I can also see tearing in preview once in a while
So I've spent quite some time trying to find the solution, to no avail.
I have even encountered multiple issues (with alpha04) like:
Random SIGSEGV crashes when turning camera on/off
I tried sample projects and codelabs from google which also were not working 100% of the time on tested devices
At some point I got notification that
camera was being used in background, even though It was bound to
lifecycle and window closed, which is the last thing I want my users
to see.
Camera was indeed loading slower and I was getting horrible FPS even with analyzer off.
Resolution would drop down to lowest possible and preview would be pixelated on some devices
Every once in a while preview would start tearing vertically
Analyzer frame was different size than preview and there were some aspect ratio issues which took quite some time to resolve.
There's still quite some boilerplate required for it to work
Documentation for edge cases is pretty much non existent, so most of the stuff is trial and error.
In the end I just started looking for other libraries and came upon https://github.com/natario1/CameraView This is by far the easiest to use library I have ever seen for camera. Way simplier than camerax, it seems to just work, loads way faster, renders preview at 2x-3x higher FPS even with analyzer step running in the background. So far I had no issues with it.
Even though I strongly believe, that I was missing something, when using CameraX and there's probably a way to make it work, in the end it just doesn't seem worth it for now and I'll probably wait till there's a production ready version until I try again.
After implementing the camera2 API for the inApp camera I noticed that on Samsung devices the images appear blurry. After searching about that I found the Sasmung Camera SDK (http://developer.samsung.com/galaxy#camera). So after implementing the SDK on Samsung Galaxy S7 the images are fine now, but on Galaxy S6 they are still blurry. Someone experienced those kind of issues with Samsung devices?
EDIT:
To complement #rcsumners comment. I am setting autofocus by using
mPreviewBuilder.set(SCaptureRequest.CONTROL_AF_TRIGGER, SCaptureRequest.CONTROL_AF_TRIGGER_START);
mSCameraSession.capture(mPreviewBuilder.build(), new SCameraCaptureSession.CaptureCallback() {
#Override
public void onCaptureCompleted(SCameraCaptureSession session, SCaptureRequest request, STotalCaptureResult result) {
isAFTriggered = true;
}
}, mBackgroundHandler);
It is a long exposure image where the use has to take an image of a static non moving object. For this I am using the CONTROL_AF_MODE_MACRO
mCaptureBuilder.set(SCaptureRequest.CONTROL_AF_MODE, SCaptureRequest.CONTROL_AF_MODE_MACRO);
and also I am enabling auto flash if it is available
requestBuilder.set(SCaptureRequest.CONTROL_AE_MODE,
SCaptureRequest.CONTROL_AE_MODE_ON_AUTO_FLASH);
I am not really an expert in this API, I mostly followed the SDK example app.
There could be a number of issues causing this problem. One prominent one is the dimensions of your output image
I ran Camera2 API and the preview is clear, but the output was quite blurry
val characteristics: CameraCharacteristics? = cameraManager.getCameraCharacteristics(cameraId)
val size = characteristics?.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)?.getOutputSizes(ImageFormat.JPEG) // The issue
val width = imageDimension.width
val height = imageDimension.height
if (size != null) {
width = size[0].width; height = size[0].height
}
val imageReader = ImageReader.newInstance(width, height, ImageFormat.JPEG, 5)
The code below was returning a dimension about 245*144 which was way to small to be sent to the image reader. Some how the output was stretching the image making it end up been blurry. Therefore I removed this line below.
val size = characteristics?.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP)?.getOutputSizes(ImageFormat.JPEG) // this was returning a small
Setting the width and height manually resolved the issue.
You're setting the AF trigger for one frame, but then are you waiting for AF to complete? For AF_MODE_MACRO (are you verifying the device lists support for this AF mode?) you need to wait for AF_STATE_FOCUSED_LOCKED before the image is guaranteed to be stable and sharp. (You may also receive NOT_FOCUSED_LOCKED if the AF algorithm can't reach sharp focus, which could be because the object is just too close for the lens, or the scene is too confusing)
On most modern devices, it's recommended to use CONTINUOUS_PICTURE and not worry about AF triggering unless you really want to lock focus for some time period. In that mode, the device will continuously try to focus to the best of its ability. I'm not sure all that many devices support MACRO, to begin with.
I am working on an app that will allow a user to take quick click and forget snapshots. Most of the app is done except for the camera working that way I would like. Right now I have the camera working but I can't seem to find a way to disable the shutter sound and I cant find a way to disable displaying the preview. I was able to cover the preview up with a control but I would rather just not have it displayed if possible.
To sum things up, these are the items that I would like to disable while utilizing the built in Camera controls.
Shutter sound
Camera screen display
Image preview onPictureTaken
Does anyone know of a resource that could point me in the right direction, I would greatly appreciate it. I have been following CommonsWare's example from this sample fairly closely.
Thank you.
This is actually a property in the build.prop of a phone. I'm unsure if its possible to change this. Unless you completely override it and use your own camera code. Using what you can that is available in the SDK.
Take a look at this:
CameraService.cpp
. . .
CameraService::Client::Client(const sp<CameraService>& cameraService,
const sp<ICameraClient>& cameraClient,
const sp<CameraHardwareInterface>& hardware,
int cameraId, int cameraFacing, int clientPid) {
mPreviewCallbackFlag = FRAME_CALLBACK_FLAG_NOOP;
mOrientation = getOrientation(0, mCameraFacing == CAMERA_FACING_FRONT);
mOrientationChanged = false;
cameraService->setCameraBusy(cameraId);
cameraService->loadSound();
LOG1("Client::Client X (pid %d)", callingPid)
}
void CameraService::loadSound() {
Mutex::Autolock lock(mSoundLock);
LOG1("CameraService::loadSound ref=%d", mSoundRef);
if (mSoundRef++) return;
mSoundPlayer[SOUND_SHUTTER] = newMediaPlayer("/system/media/audio/ui/camera_click.ogg");
mSoundPlayer[SOUND_RECORDING] = newMediaPlayer("/system/media/audio/ui/VideoRecord.ogg");
}
As can be noted, the click sound is started without your interaction.
This is the service used in the Gingerbread Source code.
The reason they DON'T allow this is because it is illegal is some countries. Only way to achieve what you want is to have a custom ROM.
Update
If what being said here: http://androidforums.com/t-mobile-g1/6371-camera-shutter-sound-effect-off.html
still applies, then you could write a timer that turns off the sound (Silent Mode) for a couple of seconds and then turn it back on each time you take a picture.
You may use the data from the preview callback using a function to save it at a picture on some type of trigger such as a button, using onclick listener. you could compress the image to jpeg or png. In this way, there no shutterCallback to be implemented. and therefore you can play any sound you want or none when taking a picture.
You can effectively hide the preview surface by giving it dimensions of 1p in the xml file (I found an example the said 0p but for some reason that was giving me errors).
It may be illegal to have a silent shutter in some places, but it doesn't appear that the US is such a place, as my HTC One gives me an option to silence it, and in fact, since Android 4.2 you can do this:
Camera.CameraInfo info=new Camera.CameraInfo();
if (info.canDisableShutterSound) {
camera.enableShutterSound(false);
}