Develop Reference

Kotlin Jetpack Compose: Display ByteArray or FileStream as Image in Android

Heyho,
I'm quiet new to Kotlin, but I came across this problem:
I have a library function, that generates an image(QR-Code). Now I'd like to display that image... But I've no idea how. The documentation only explains how to save the image locally. But I'm not really interested in saving it. So I can either get the image as a FileStream or as a ByteArray. Any possibility to display any of these as an Image in the UI?
An Example:
#Composable
fun QrCode(stand: String) {
Text(text = "QR-Code:", fontSize = 16.sp)
//? Image(QRCode(stand).render().getBytes()) // this obviously won't work
}
Any ideas?

In order to display an Image, you can refer this.
#Composable
fun BitmapImage(bitmap: Bitmap) {
Image(
bitmap = bitmap.asImageBitmap(),
contentDescription = "some useful description",
)
}
So the remaining is to find a way to convert your target input into a Bitmap.
If you have ByteArray of the image file, you can refer to this.
fun convertImageByteArrayToBitmap(imageData: ByteArray): Bitmap {
return BitmapFactory.decodeByteArray(imageData, 0, imageData.size)
}
If you just possess the QR String, you can refer this to convert QR String to a Bitmap.
fun encodeAsBitmap(source: String, width: Int, height: Int): Bitmap? {
val result: BitMatrix = try {
MultiFormatWriter().encode(source, BarcodeFormat.QR_CODE, width, height, null)
} catch (e: Exception) {
return null
}
val w = result.width
val h = result.height
val pixels = IntArray(w * h)
for (y in 0 until h) {
val offset = y * w
for (x in 0 until w) {
pixels[offset + x] = if (result[x, y]) Color.BLACK else Color.WHITE
}
}
val bitmap = Bitmap.createBitmap(w, h, Bitmap.Config.ARGB_8888)
bitmap.setPixels(pixels, 0, width, 0, 0, w, h)
return bitmap
}
And you need to add this dependency implementation 'com.google.zxing:core:3.3.1' when you decide to use the above code.

Tensorflowlite on android buffer size error

Im trying to build a image classifier android app. I've built my model using keras.
The model is as follows:
model.add(MobileNetV2(include_top=False, weights='imagenet',input_shape=(224, 224, 3)))
model.add(GlobalAveragePooling2D())
model.add(Dropout(0.5))
model.add(Dense(3, activation='softmax'))
model.layers[0].trainable = False
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()
Output:
Model: "sequential_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
mobilenetv2_1.00_224 (Functi (None, 7, 7, 1280) 2257984
_________________________________________________________________
global_average_pooling2d_2 ( (None, 1280) 0
_________________________________________________________________
dropout_2 (Dropout) (None, 1280) 0
_________________________________________________________________
dense_1 (Dense) (None, 3) 3843
=================================================================
Total params: 2,261,827
Trainable params: 3,843
Non-trainable params: 2,257,984
After training Im converting the model using
model = tf.keras.models.load_model('model.h5')
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
open(f"myModel.tflite", "wb").write(tflite_model)
for android the code is as:
make_prediction.setOnClickListener(View.OnClickListener {
var resized = Bitmap.createScaledBitmap(bitmap, 224, 224, true)
val model = MyModel.newInstance(this)
var tbuffer = TensorImage.fromBitmap(resized)
var byteBuffer = tbuffer.buffer
// Creates inputs for reference.
val inputFeature0 = TensorBuffer.createFixedSize(intArrayOf(1, 224, 224, 3), DataType.FLOAT32)
inputFeature0.loadBuffer(byteBuffer)
// Runs model inference and gets result.
val outputs = model.process(inputFeature0)
val outputFeature0 = outputs.outputFeature0AsTensorBuffer
var max = getMax(outputFeature0.floatArray)
text_view.setText(labels[max])
// Releases model resources if no longer used.
model.close()
})
but whenever i try to run my app it closes and i get this error in the logcat.
java.lang.IllegalArgumentException: The size of byte buffer and the shape do not match.
if I change the input shape of my image to 300 from 224 and train my model on 300 input shape and plug in to android I get anthor error.
java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite buffer with 1080000 bytes and a Java Buffer with 150528 bytes
Any kind of help will be really appreciated.

Use it like:
make_prediction.setOnClickListener(View.OnClickListener {
var resized = Bitmap.createScaledBitmap(bitmap, 224, 224, true)
val model = MyModel.newInstance(this)
var tImage = TensorImage(DataType.FLOAT32)
var tensorImage = tImage.load(resized)
var byteBuffer = tensorImage.buffer
// Creates inputs for reference.
//val inputFeature0 = TensorBuffer.createFixedSize(intArrayOf(1, 224, 224, 3), DataType.FLOAT32)
//inputFeature0.loadBuffer(byteBuffer)
// Runs model inference and gets result.
val outputs = model.process(byteBuffer)
val outputFeature0 = outputs.outputFeature0AsTensorBuffer
var max = getMax(outputFeature0.floatArray)
text_view.setText(labels[max])
// Releases model resources if no longer used.
model.close()
})
Then check with debugger if the problem persists or
val outputFeature0 = outputs.outputFeature0AsTensorBuffer
causes another one.
Ping me if you need more help

BitmapFactory.decodeByteArray() works incorrect after using ExifInterface

I try to add simple functionality with getting picture from gallery or from camera in my Android app. And all works fine, I successfully obtain uri of picture and after that want to show this picture to user. And this code works well:
private fun showImageToUser(uri: Uri) {
val inputStream = contentResolver?.openInputStream(uri)
val bytes = inputStream?.readBytes() ?: return
val bitmapOriginal = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
// show bitmap in ImageView...
}
Then, I want to rotate image if need (for example, after camera all images have 90-degrees rotation). For this I use ExifInterface (from androidx.exifinterface:exifinterface:1.3.1). But there is something strange. In this code:
private fun showImageToUser(uri: Uri) {
val inputStream = contentResolver?.openInputStream(uri)
val exifInterface = ExifInterface(inputStream ?: return)
val rotation = exifInterface.getAttributeInt(ExifInterface.TAG_ORIENTATION, ExifInterface.ORIENTATION_NORMAL)
val bytes = inputStream.readBytes()
val bitmapOriginal = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
val bitmapRotated = bitmapOriginal.rotateImageIfRequired(rotation) // simple logic for rotating in rotateImageIfRequired method...
}
the bitmapOriginal is always null. As you can see, I create exifInterface object before inputStream.readBytes(). If I swap them, and try to run the following code:
private fun showImageToUser(uri: Uri) {
val inputStream = contentResolver?.openInputStream(uri)
val bytes = inputStream?.readBytes() ?: return
val exifInterface = ExifInterface(inputStream)
val rotation = exifInterface.getAttributeInt(ExifInterface.TAG_ORIENTATION, ExifInterface.ORIENTATION_NORMAL)
val bitmapOriginal = BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
val bitmapRotated = bitmapOriginal.rotateImageIfRequired(rotation)
}
then bitmapOriginal will not be null, but the rotation value will be always ORIENTATION_UNDEFINED.
So, what am I doing wrong? How to get correctly bytes and image orientation from uri and create after that bitmap with correct orientation?

Is there a way to crop Image/ImageProxy (before passing to MLKit's analyzer)?

I'm using CameraX's Analyzer use case with the MLKit's BarcodeScanner. I would like to crop portion of the image received from the camera, before passing it to the scanner.
What I'm doing right now is I convert ImageProxy (that I recieve in the Analyzer) to a Bitmap, crop it and then pass it to the BarcodeScanner. The downside is that it's not a very fast and efficient process.
I've also noticed the warning I get in the Logcat when running this code:
ML Kit has detected that you seem to pass camera frames to the
detector as a Bitmap object. This is inefficient. Please use
YUV_420_888 format for camera2 API or NV21 format for (legacy) camera
API and directly pass down the byte array to ML Kit.
It would be nice to not to do ImageProxy conversion, but how do I crop the rectangle I want to analyze?
What I've already tried is to set a cropRect field of the Image (imageProxy.image.cropRect) class, but it doesn't seem to affect the end result.

Yes, it's true that if you use ViewPort and set viewport to yours UseCases(imageCapture or imageAnalysis as here https://developer.android.com/training/camerax/configuration) you can get only information about crop rectangle especially if you use ImageAnalysis(because if you use imageCapture, for on-disk the image is cropped before saving and it doesn't work for ImageAnalysis and if you use imageCapture without saving on disk) and here solution how I solved this problem:
First of all set view port for use cases as here: https://developer.android.com/training/camerax/configuration
Get cropped bitmap to analyze
override fun analyze(imageProxy: ImageProxy) {
val mediaImage = imageProxy.image
if (mediaImage != null && mediaImage.format == ImageFormat.YUV_420_888) {
croppedBitmap(mediaImage, imageProxy.cropRect).let { bitmap ->
requestDetectInImage(InputImage.fromBitmap(bitmap, rotation))
.addOnCompleteListener { imageProxy.close() }
}
} else {
imageProxy.close()
}
}
private fun croppedBitmap(mediaImage: Image, cropRect: Rect): Bitmap {
val yBuffer = mediaImage.planes[0].buffer // Y
val vuBuffer = mediaImage.planes[2].buffer // VU
val ySize = yBuffer.remaining()
val vuSize = vuBuffer.remaining()
val nv21 = ByteArray(ySize + vuSize)
yBuffer.get(nv21, 0, ySize)
vuBuffer.get(nv21, ySize, vuSize)
val yuvImage = YuvImage(nv21, ImageFormat.NV21, mediaImage.width, mediaImage.height, null)
val outputStream = ByteArrayOutputStream()
yuvImage.compressToJpeg(cropRect, 100, outputStream)
val imageBytes = outputStream.toByteArray()
return BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.size)
}
Possibly there is a loss in conversion speed, but on my devices I did not notice the difference. I set 100 quality in method compressToJpeg, but mb if set less quality it can improve speed, it need test.
upd: May 02 '21 :
I found another way without convert to jpeg and then to bitmap. This should be a faster way.
Set viewport as previous.
Convert YUV_420_888 to NV21, then crop and analyze.
override fun analyze(imageProxy: ImageProxy) {
val mediaImage = imageProxy.image
if (mediaImage != null && mediaImage.format == ImageFormat.YUV_420_888) {
croppedNV21(mediaImage, imageProxy.cropRect).let { byteArray ->
requestDetectInImage(
InputImage.fromByteArray(
byteArray,
imageProxy.cropRect.width(),
imageProxy.cropRect.height(),
rotation,
IMAGE_FORMAT_NV21,
)
)
.addOnCompleteListener { imageProxy.close() }
}
} else {
imageProxy.close()
}
}
private fun croppedNV21(mediaImage: Image, cropRect: Rect): ByteArray {
val yBuffer = mediaImage.planes[0].buffer // Y
val vuBuffer = mediaImage.planes[2].buffer // VU
val ySize = yBuffer.remaining()
val vuSize = vuBuffer.remaining()
val nv21 = ByteArray(ySize + vuSize)
yBuffer.get(nv21, 0, ySize)
vuBuffer.get(nv21, ySize, vuSize)
return cropByteArray(nv21, mediaImage.width, cropRect)
}
private fun cropByteArray(array: ByteArray, imageWidth: Int, cropRect: Rect): ByteArray {
val croppedArray = ByteArray(cropRect.width() * cropRect.height())
var i = 0
array.forEachIndexed { index, byte ->
val x = index % imageWidth
val y = index / imageWidth
if (cropRect.left <= x && x < cropRect.right && cropRect.top <= y && y < cropRect.bottom) {
croppedArray[i] = byte
i++
}
}
return croppedArray
}
First crop fun I took from here: Android: How to crop images using CameraX?
And I found also another crop fun, it seems that it is more complicated:
private fun cropByteArray(src: ByteArray, width: Int, height: Int, cropRect: Rect, ): ByteArray {
val x = cropRect.left * 2 / 2
val y = cropRect.top * 2 / 2
val w = cropRect.width() * 2 / 2
val h = cropRect.height() * 2 / 2
val yUnit = w * h
val uv = yUnit / 2
val nData = ByteArray(yUnit + uv)
val uvIndexDst = w * h - y / 2 * w
val uvIndexSrc = width * height + x
var srcPos0 = y * width
var destPos0 = 0
var uvSrcPos0 = uvIndexSrc
var uvDestPos0 = uvIndexDst
for (i in y until y + h) {
System.arraycopy(src, srcPos0 + x, nData, destPos0, w) //y memory block copy
srcPos0 += width
destPos0 += w
if (i and 1 == 0) {
System.arraycopy(src, uvSrcPos0, nData, uvDestPos0, w) //uv memory block copy
uvSrcPos0 += width
uvDestPos0 += w
}
}
return nData
}
Second crop fun I took from here:
https://www.programmersought.com/article/75461140907/
I would be glad if someone can help improve the code.

I'm still improving the way to do it. But this will work for me now
CameraX crop image before sending to analyze
<androidx.constraintlayout.widget.ConstraintLayout
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:paddingBottom="#dimen/_40sdp">
<androidx.camera.view.PreviewView
android:id="#+id/previewView"
android:layout_width="match_parent"
android:layout_height="0dp"
app:layout_constraintDimensionRatio="1:1"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintTop_toTopOf="parent" /></androidx.constraintlayout.widget.ConstraintLayout>
Cropping an image into 1:1 before passing it to analyze
override fun onCaptureSuccess(image: ImageProxy) {
super.onCaptureSuccess(image)
var bitmap: Bitmap = imageProxyToBitmap(image)
val dimension: Int = min(bitmap.width, bitmap.height)
bitmap = ThumbnailUtils.extractThumbnail(bitmap, dimension, dimension)
imageView.setImageBitmap(bitmap) //Here you can pass the crop[from the center] image to analyze
image.close()
}
**Function for converting into bitmap **
private fun imageProxyToBitmap(image: ImageProxy): Bitmap {
val buffer: ByteBuffer = image.planes[0].buffer
val bytes = ByteArray(buffer.remaining())
buffer.get(bytes)
return BitmapFactory.decodeByteArray(bytes, 0, bytes.size)
}

You would use ImageProxy.SetCroprect to get the rect and then use CropRect to set it.
For example if you had imageProxy, you would do : ImageProxy.setCropRect(Rect) and then you would do ImageProxy.CropRect.

Is it possible to process data of camerax preview before displaying it?

I would like to process the image that is displayed in preview and display display the processed version.
I've tried modifying it using the imageProxy in analyzer, but that does not seem to be doing anything.
I know that older camera apis are able to do it, but CameraX does not seem to have the apis.

No, this flow is not supported by cameraX. Actually, this use case has never
been supported by any Android camera API. What you have to do, is create your own renderer (preferably, OpenGL), hide the native preview surface, and send the modified frames to your renderer.

Hope this will help:
val mediaImage = imageProxy.image ?: return
var bitmap = ImageUtils.convertYuv420888ImageToBitmap(mediaImage)
val rotationDegrees = imageProxy.imageInfo.rotationDegrees
val matrix = Matrix()
matrix.postRotate(rotationDegrees.toFloat())
bitmap =
Bitmap.createBitmap(bitmap, 0, 0, mediaImage.width, mediaImage.height, matrix, true)
val cropHeight = if (bitmap.width < previewView.width) {
// if preview area larger than analysing image
val koeff = bitmap.width.toFloat() / previewView.width.toFloat()
previewView.height.toFloat() * koeff
} else {
// if preview area smaller than analysing image
val prc = 100 - (previewView.width.toFloat() / (bitmap.width.toFloat() / 100f))
previewView.height + ((previewView.height.toFloat() / 100f) * prc)
}
val cropTop = (bitmap.height / 2) - ((cropHeight) / 2)
if (cropTop > 0) {
Bitmap.createBitmap(bitmap, 0, cropTop.toInt(), bitmap.width, cropHeight.toInt())
.also { process(it, imageProxy) }
} else {
imageProxy.image?.let { process(it, imageProxy) }
}