I was working with CameraX and had hard time converting captured ImageProxy to Bitmap. After searching and trying, I formulated a solution. Later I found that it was not optimum so I changed the design. That forced me to drop hours of work.
Since I (or someone else) might need it in a future, I decided to post here as a question and post and answer to it for reference and scrutiny. Feel free to add better answer if you have one.
The relevant code is:
class ImagePickerActivity : AppCompatActivity() {
private var width = 325
private var height = 205
#RequiresApi(Build.VERSION_CODES.LOLLIPOP)
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_image_picker)
view_finder.post { startCamera() }
}
#RequiresApi(Build.VERSION_CODES.LOLLIPOP)
private fun startCamera() {
// Create configuration object for the viewfinder use case
val previewConfig = PreviewConfig.Builder().apply {
setTargetAspectRatio(Rational(1, 1))
//setTargetResolution(Size(width, height))
setLensFacing(CameraX.LensFacing.BACK)
setTargetAspectRatio(Rational(width, height))
}.build()
}
// Create configuration object for the image capture use case
val imageCaptureConfig = ImageCaptureConfig.Builder()
.apply {
setTargetAspectRatio(Rational(1, 1))
// We don't set a resolution for image capture instead, we
// select a capture mode which will infer the appropriate
// resolution based on aspect ration and requested mode
setCaptureMode(ImageCapture.CaptureMode.MIN_LATENCY)
}.build()
// Build the image capture use case and attach button click listener
val imageCapture = ImageCapture(imageCaptureConfig)
capture_button.setOnClickListener {
imageCapture.takePicture(object : ImageCapture.OnImageCapturedListener() {
override fun onCaptureSuccess(image: ImageProxy?, rotationDegrees: Int) {
//How do I get the bitmap here?
//imageView.setImageBitmap(someBitmap)
}
override fun onError(useCaseError: ImageCapture.UseCaseError?, message: String?, cause: Throwable?) {
val msg = "Photo capture failed: $message"
Toast.makeText(baseContext, msg, Toast.LENGTH_SHORT).show()
Log.e(localClassName, msg)
cause?.printStackTrace()
}
})
}
CameraX.bindToLifecycle(this, preview, imageCapture)
}
}
So the solution was to add extension method to Image and here is the code
class ImagePickerActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_image_picker)
}
private fun startCamera() {
val imageCapture = ImageCapture(imageCaptureConfig)
capture_button.setOnClickListener {
imageCapture.takePicture(object : ImageCapture.OnImageCapturedListener() {
override fun onCaptureSuccess(image: ImageProxy?, rotationDegrees: Int) {
imageView.setImageBitmap(image.image?.toBitmap())
}
//.....
})
}
}
}
fun Image.toBitmap(): Bitmap {
val buffer = planes[0].buffer
buffer.rewind()
val bytes = ByteArray(buffer.capacity())
buffer.get(bytes)
return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
}
Slightly modified version. Using the inline function use on the Closable ImageProxy
imageCapture.takePicture(
object : ImageCapture.OnImageCapturedListener() {
override fun onCaptureSuccess(image: ImageProxy?, rotationDegrees: Int) {
image.use { image ->
val bitmap: Bitmap? = image?.let {
imageProxyToBitmap(it)
} ?: return
}
}
})
private fun imageProxyToBitmap(image: ImageProxy): Bitmap {
val buffer: ByteBuffer = image.planes[0].buffer
val bytes = ByteArray(buffer.remaining())
buffer.get(bytes)
return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
}
Here is the safest approach, using MLKit's own implementation.
Tested and working on MLKit version 1.0.1
import com.google.mlkit.vision.common.internal.ImageConvertUtils;
Image mediaImage = imageProxy.getImage();
InputImage image = InputImage.fromMediaImage(mediaImage, imageProxy.getImageInfo().getRotationDegrees());
Bitmap bitmap = ImageConvertUtils.getInstance().getUpRightBitmap(image)
Java Implementation of Backbelt's Answer.
private Bitmap imageProxyToBitmap(ImageProxy image) {
ByteBuffer buffer = image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.remaining()];
buffer.get(bytes);
return BitmapFactory.decodeByteArray(bytes,0,bytes.length,null);
}
There is second version of takePicture method at the moment (CameraX version 1.0.0-beta03). It provides several ways to persist image (OutputStream or maybe File can be useful in your case).
If you still want to convert ImageProxy to Bitmap here is my answer to similar question, which gives the correct implemetation of this conversion.
Please kindly take a look at this answer. All you need to apply it to your question is to get Image out of your ImageProxy
Image img = imaget.getImage();
Related
I have created an app with QR code scanner, however I can't scan a QR code with a logo using it, any solution for this?
This is the code of my QR code scanner app:
class QRCode (val onQrCodeScanned: (String) -> Unit) : ImageAnalysis.Analyzer {
#RequiresApi(Build.VERSION_CODES.M)
private val supportedImageFormats = listOf(
ImageFormat.YUV_420_888 ,
ImageFormat.YUV_422_888 ,
ImageFormat.YUV_444_888)
#RequiresApi(Build.VERSION_CODES.M)
override fun analyze(image: ImageProxy) {
if (image.format in supportedImageFormats) {
val bytes = image.planes.first().buffer.toByteArray()
val source = PlanarYUVLuminanceSource(
bytes,
image.width,
image.height,
0,
0,
image.width,
image.height,
false)
val binaryBmp = BinaryBitmap(HybridBinarizer(source))
try {
val result = MultiFormatReader().apply {
setHints(
mapOf(
DecodeHintType.POSSIBLE_FORMATS to arrayListOf(
BarcodeFormat.QR_CODE
)
)
)
}.decode(binaryBmp)
onQrCodeScanned(result.text)
} catch (e: Exception) {
e.printStackTrace()
} finally {
image.close()
}
}
}
private fun ByteBuffer.toByteArray(): ByteArray {
rewind()
return ByteArray(remaining()).also {
get(it)
}
}
}
It can scan other QR codes but not QR codes with logo.
This is my QR code:
I fixed it... The problem is in-app browsers cannot process websites that are built using Javascript.
This is what my code for launching the browser looks like before and after:
Before:
#Composable
fun LoadWebUrl(url: String) {
val context = LocalContext.current
AndroidView(factory = {
WebView(context).apply {
webViewClient = WebViewClient()
loadUrl(url)
}
})
}
After:
#Composable
fun LoadWebUrl(url: String) {
val context = LocalContext.current
IntentUtils.launchBrowser(context, url)
}
I am creating a simple qr scanner using Camerax and google ML Kit. I am opening an intent after the string value is extracted from the QR code. The problem I'm facing is , the intent is opening multiple times. How do I resolve this?
The following is the setup for image analysis.DisplayQR intent will open after receiving string value inside QR code.
val imageAnalysis = ImageAnalysis.Builder()
.setTargetResolution(Size(640, 480))
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.build()
imageAnalysis.setAnalyzer(
ContextCompat.getMainExecutor(this),
CodeAnalyzer(this, object : CallBackInterface {
override fun onSuccess(qrString: String?) {
imageAnalysis.clearAnalyzer()
Toast.makeText(this#ActivityQR,qrString,Toast.LENGTH_SHORT).show()
Log.d("rty",qrString.toString())
//the following intent is opening multiple times
val visitordetails =
Intent(this#ActivityQR, DisplayQR::class.java)
visitordetails.putExtra("VISITOR_QR", qrString)
startActivity(visitordetails)
}
override fun onFailed() {
}
})
)
cameraProvider.bindToLifecycle(this, selectedCamera, imageAnalysis, cameraPreview)
Code for analyzing the image
class CodeAnalyzer(context: Context, callBackInterface: CallBackInterface):imageAnalysis.Analyzer {
private val context: Context = context
private val callback: CallBackInterface = callBackInterface
#SuppressLint("UnsafeOptInUsageError")
override fun analyze(image: ImageProxy) {
var scanner: BarcodeScanner = BarcodeScanning.getClient()
val scannedIMage = image.image
if (scannedIMage != null) {
var scannedInputImage = InputImage.fromMediaImage(
scannedIMage,
image.imageInfo.rotationDegrees
)
scanner.process(scannedInputImage).addOnSuccessListener { barCodes ->
for (qrCode in barCodes) {
when (qrCode.valueType) {
Barcode.TYPE_TEXT -> {
val qrString: String? = qrCode.rawValue
if (qrString != null) {
callback.onSuccess(qrString) //Here I am calling the callback
}
}
}
}
}.addOnFailureListener {
}.addOnCompleteListener {
image.close()
}
}
}
}
Edit: Corrected activity name
I use the new CameraX API for taking a picture like this :
imageButton.setOnClickListener{
val file = File(externalMediaDirs.first(),
"${System.currentTimeMillis()}.jpg")
imageCapture.takePicture(executor, object :
ImageCapture.OnImageCapturedListener() {
override fun onCaptureSuccess(
image: ImageProxy,
rotationDegrees: Int)
{
// DO I NEED TO USE 'image' TO ACCESS THE IMAGE DATA FOR MANIPULATION (e.g. image.planes)
}
override fun onError(
imageCaptureError: ImageCapture.ImageCaptureError,
message: String,
cause: Throwable?
) {
val msg = "Photo capture failed: $message"
Log.e("CameraXApp", msg, cause)
}
})
imageCapture.takePicture(file, executor,
object : ImageCapture.OnImageSavedListener {
override fun onError(
imageCaptureError: ImageCapture.ImageCaptureError,
message: String,
exc: Throwable?
) {
val msg = "Photo capture failed: $message"
Log.e("CameraXApp", msg, exc)
}
override fun onImageSaved(file: File) {
val msg = "Photo capture succeeded: ${file.absolutePath}"
Log.d("CameraXApp", msg)
}
}
}
I want to apply some image processing with Renderscript when the image is captured. But I dont know how to access the pixels of the image that is captured.
Can someone provide a solution ?
I tried it with image.planes within the onCaptureSuccess() callback (see comment)
I must admit that I am new to this and do not know really what planes are. Before this, I only worked with Bitmaps ( doing some image processing on Bitmaps with Renderscript). Is there a way to turn a frame/image into a Bitmap and apply some sort image processing on it "on the fly" before saving it as a file ? If yes, how ? The official CameraX guidelines are not very helpful when it comes to this.
This is probably very late answer but the answer is in ImageCapture callback.
You get an Image with
val image = imageProxy.image
Notice this will only work when you have images in YUV format, in order to configure your camera with that you can do it with
imageCapture = ImageCapture.Builder()
.setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
.setBufferFormat(ImageFormat.YUV_420_888)
...
.build()
Now can get image.planes
The default image format is JPEG
and you can access the buffer with
buffer: ByteBuffer = imageProxy.image.planes[0].getBuffer()
and if you plan on setting it to YUV
val image = imageProxy.image
val yBuffer = image.planes[0].buffer // Y
val uBuffer = image.planes[1].buffer // U
val vBuffer = image.planes[2].buffer // V
I try to use new CameraX api and I got this error: When I capture the image this image stored with wrong rotation. For example I capture in portrait orientation but result image in landscape orientation.
There is my code:
private fun startCamera() {
val previewConfig = PreviewConfig.Builder().apply {
setTargetResolution(Size(textureView.width, textureView.height))
setTargetRotation(textureView.display.rotation)
}.build()
val imageCaptureConfig = ImageCaptureConfig.Builder().apply {
setCaptureMode(CaptureMode.MIN_LATENCY)
setTargetAspectRatio(RATIO_4_3)
setTargetRotation(textureView.display.rotation)
}.build()
imageCapture = ImageCapture(imageCaptureConfig)
val preview = Preview(previewConfig)
preview.setOnPreviewOutputUpdateListener { previewOutput ->
removeView(textureView)
addViewMatchParent(textureView, position = 0)
textureView.surfaceTexture = previewOutput.surfaceTexture
textureView.updateTransformForCameraFinderView()
}
(context as? LifecycleOwner)?.let { lifecycleOwner ->
CameraX.bindToLifecycle(lifecycleOwner, preview, imageCapture)
}
}
private fun capturePhoto() {
tempImageFile = generateTmpFile(false)
val executor = Executor { it.run() }
imageCapture.takePicture(tempImageFile!!, executor, object : OnImageSavedListener {
override fun onError(error: ImageCaptureError, message: String, exc: Throwable?) {
exc?.printStackTrace()
}
override fun onImageSaved(photoFile: File) {
post {
// load image into ImageView by Glide
showCapturedPhotoPreview(photoFile)
}
}
})
}
Please give me advise how can I fix it?
P.S. I tried to find solution so don't copy-paste first looking like something similar)
Update: I tried to do my CameraView like in this sample but in their case it works, in my - no)
Try it:
val imageCaptureConfig = ImageCaptureConfig.Builder().apply {
setCaptureMode(CaptureMode.MIN_LATENCY)
setTargetAspectRatio(RATIO_4_3)
// play with this line!
setTargetRotation(Surface.ROTATION_0)
setTargetRotation(textureView.display.rotation)
}.build()
I select line that fixes my problem again:
setTargetRotation(Surface.ROTATION_0)
I'm using Google Play Vision API to detect qr codes. I'm not interested in other formats, so I'm trying to use that API call to speed up detection. It works fine if I'm calling it as setBarcodeFormats(Barcode.ALL_FORMATS), it detects qr codes with format 256 (QR_CODE), but if I'm initializing it as setBarcodeFormats(Barcode.QR_CODE), it does not detect anything. Here's some code snippets:
override fun onCreate(savedInstanceState: Bundle?) {
...
barcodeDetector = BarcodeDetector.Builder(applicationContext).setBarcodeFormats(Barcode.ALL_FORMATS).build()
...
private inner class ImageReaderOnImageAvailableListenerImpl : ImageReader.OnImageAvailableListener {
override fun onImageAvailable(reader: ImageReader) {
val image = reader.acquireNextImage()
val buffer = image.planes[0].buffer;
val bitmap: Bitmap? = if (buffer.hasArray()) {
BitmapFactory.decodeByteArray(buffer.array(), buffer.arrayOffset(), buffer.remaining(), null);
} else {
val byteArray = ByteArray(buffer.remaining())
buffer.get(byteArray)
BitmapFactory.decodeByteArray(byteArray, 0, byteArray.size, null);
}
val barcodes = barcodeDetector.detect(Frame.Builder().setBitmap(bitmap).build())
image.close()
this#QrCodeCaptureActivity.imageView.setImageBitmap(bitmap)
if (barcodes.size() > 0) {
for (index in 0 until barcodes.size()) {
val barcode = barcodes.valueAt(index)
logd("barcode $index: ${barcode.format} ${barcode.valueFormat} ${barcode.rawValue}")
}
} else {
logd("no barcodes (${image.width}x${image.height})")
}
}
}