I use the cameraX API in Android to analyze multiple frames in a period of 5 up to 60 seconds. There are multiple conditional tasks I want to do with the images depending on what tasks the user selected. These include:
scan for barcodes/qr codes (using google mlkit)
scan for text (using google mlkit)
custom edge detection using openCV in C++ with JNI
save image as png file (losless)
show frames in app (PreviewView or ImageView)
These tasks heavily vary in workload and time to finish, so instead of waiting for each task to finish until getting a new frame, I want to receive constant frames and let each task only start with the newest frame when it's finished with it's last workload.
while MLKit takes YUV images as input, openCV uses RGBA (or BGRA), so no matter which output format I choose, I will need to convert it some way. My choice was to use RGBA_8888 as output format and convert it into a bitmap since bitmap is supported from both MLKit and OpenCV and the conversion from RGBA to bitmap is much quicker than from YUV to bitmap. But using bitmaps I get huge problems with memory to the extend of the app just getting closed by Android. Using the Android Studio Profiler, I noticed the native part of ram usage going up constantly, staying that high even after workload is done and the camera is unbound.
I read online that it is heavily suggested to recycle bitmaps after use to free up their memory space. Problem here is that all these tasks run and finish independently and I couldn't come up with a good solution for recycling the bitmap as soon as possible without heavily increasing memory usage by keeping them in memory for a certain time (like 10 seconds).
I thought about using jobs for each task and to recycle when all jobs are done, but this doesn't work for the MLKit analyses because they return using a listener, resulting in the jobs ending before the task is actually done.
I appreciate any input for how to efficiently recycle the bitmaps, using something different than bitmaps, reducing memory consumption or any code improvements in general!
Here are code samples for the image analysis and for the barcode scanner. They should suffice for giving a general idea of the running code.
val imageAnalysisBuilder =
ImageAnalysis
.Builder()
.setTargetResolution(android.util.Size(720, 1280))
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setOutputImageFormat(OUTPUT_IMAGE_FORMAT_RGBA_8888)
val imageAnalysis = imageAnalysisBuilder.build()
imageAnalysis.setAnalyzer(Executors.newSingleThreadExecutor()) { imageProxy ->
//bitmap conversion from https://github.com/android/camera-samples
var bitmap = Bitmap.createBitmap(imageProxy.width, imageProxy.height, Bitmap.Config.ARGB_8888)
imageProxy.use { bitmap.copyPixelsFromBuffer(it.planes[0].buffer) }
val rotationDegrees = imageProxy.imageInfo.rotationDegrees
imageProxy.close()
if (!barcodeScannerBusy) {
CoroutineScope.launch { startMlKitBarcodeScanner(bitmap, rotationDegrees) }
}
if (!textRecognitionBusy) {
CoroutineScope.launch { startMlKitTextRecognition(bitmap, rotationDegrees) }
}
//more tasks with same pattern
//when to recycle bitmap???
}
private fun startMlKitBarcodeScanner(bitmap: Bitmap, rotationDegrees: Int) {
barcodeScannerBusy = true
val inputImage = InputImage.fromBitmap(bitmap, rotationDegrees)
barcodeScanner?.process(inputImage)
?.addOnSuccessListener { barcodes ->
//do stuff with barcodes
}
?.addOnFailureListener {
//failure handling
}
?.addOnCompleteListener {
barcodeScannerBusy = false
//can't recycle bitmap here since other tasks might still use it
}
}
I solved the issue by now. Mainly by using a bitmap buffer variable for each task working with the image. Downside is that in the worst case, I create the same bitmap multiple times in a row. Upside is that each task can use its own bitmap independently of any other task.
Also since the device I use is not the most powerful (quite the contrary in fact), I decided to split up some of the tasks into multiple analyzers and assign a new analyzer to the camera when needing it.
Also if copying the planes of the imageProxy multiple times the way I do it here, you need to use the rewind() method before creating a new bitmap with it.
lateinit var barcodeScannerBitmapBuffer: Bitmap
lateinit var textRecognitionBitmapBuffer: Bitmap
val imageAnalysisBuilder =
ImageAnalysis
.Builder()
.setTargetResolution(android.util.Size(720, 1280))
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setOutputImageFormat(OUTPUT_IMAGE_FORMAT_RGBA_8888)
val imageAnalysis = imageAnalysisBuilder.build()
imageAnalysis.setAnalyzer(Executors.newSingleThreadExecutor()) { imageProxy ->
if (barcodeScannerBusy && textRecognitionBusy) {
imageProxy.close()
return#Analyzer
}
if (!::barcodeScannerBitmapBuffer.isInitialized) {
barcodeScannerBitmapBuffer = Bitmap.createBitmap(
imageProxy.width,
imageProxy.height,
Bitmap.Config.ARGB_8888
)
}
if (!::textRecognitionBitmapBuffer.isInitialized) {
textRecognitionBitmapBuffer = Bitmap.createBitmap(
imageProxy.width,
imageProxy.height,
Bitmap.Config.ARGB_8888
)
}
if (!barcodeScannerBusy) {
imageProxy.use {
//bitmap conversion from https://github.com/android/camera-samples
barcodeScannerBitmapBuffer.copyPixelsFromBuffer(it.planes[0].buffer)
it.planes[0].buffer.rewind()
}
}
if (!textRecognitionBusy) {
imageProxy.use { textRecognitionBitmapBuffer.copyPixelsFromBuffer(it.planes[0].buffer) }
}
val rotationDegrees = imageProxy.imageInfo.rotationDegrees
imageProxy.close()
if (::barcodeScannerBitmapBuffer.isInitialized &&!barcodeScannerBusy) {
startMlKitBarcodeScanner(barcodeScannerBitmapBuffer, rotationDegrees)
}
if (::textRecognitionBitmapBuffer.isInitialized && !textRecognitionBusy) {
startMlKitTextRecognition(textRecognitionBitmapBuffer, rotationDegrees)
}
}
Related
I want to know if there is any efficient way to make app to app communication using IPC. I went to the guide of services that uses AIDL from the docs. But what I really want is to have an image to transfer between them.(Images are high quality) The only issue is android only let us use 1024 kB or even less to pass data through bundle. if you do more than that you'll end up with TransactionTooLargeException. Right now I'm compressing the image and passing base64 string between the two apps and it works fine. But sometimes some images can not be compressed at all. How can I do something like that. I'm compressing image using
bitmap.compress(imageformat=webp,quality=90,compress.nowrap)
the quality will reduce by 10 if I get TransactionTooLargeException. Any ideas on how to do something like that working on android?
But the thing is I don't want to open any other app. The image that I'm receiving will be processed and send a status to the image that was sent from the application. Like 'very cool image' 'very bad image'.in a string status.
Aidl link
Thanks.
var quality = 100
private fun sendImageToDevice(quality: Int, icon: Bitmap) {
Log.e(TAG, "quality of image is $quality ")
runOnUiThread {
Toast.makeText(
this#MainActivity,
"Image Quality $quality",
Toast.LENGTH_SHORT
)
.show()
}
if (quality > 0)
try {
mProcessImageService?.sendImageFormat(icon.toBase64(quality))
} catch (e: TransactionTooLargeException) {
e.message
this.quality = quality - 20
sendImageToDevice(this.quality, icon)
}
else {
}
}
fun Bitmap.toBase64(quality: Int): String {
val outputStream = ByteArrayOutputStream()
this.compress(Bitmap.CompressFormat.JPEG, quality, outputStream)
val base64String: String = Base64.encodeToString(outputStream.toByteArray(), Base64.NO_WRAP)
Log.d("MainActivity", "outputstream size is ${outputStream.size()}")
return base64String
}
I'm developing a Face Detection feature with Camera2 and MLKit.
In the Developer Guide, in the Performance Tips part, they say to capture images in ImageFormat.YUV_420_888 format if using the Camera2 API, which is my case.
Then, in the Face Detector part, they recommend to use an image with dimensions of at least 480x360 pixels for faces recognition in real time, which is again my case.
Ok, let's go ! Here is my code, working well
private fun initializeCamera() = lifecycleScope.launch(Dispatchers.Main) {
// Open the selected camera
cameraDevice = openCamera(cameraManager, getCameraId(), cameraHandler)
val previewSize = if (isPortrait) {
Size(RECOMMANDED_CAPTURE_SIZE.width, RECOMMANDED_CAPTURE_SIZE.height)
} else {
Size(RECOMMANDED_CAPTURE_SIZE.height, RECOMMANDED_CAPTURE_SIZE.width)
}
// Initialize an image reader which will be used to display a preview
imageReader = ImageReader.newInstance(
previewSize.width, previewSize.height, ImageFormat.YUV_420_888, IMAGE_BUFFER_SIZE)
// Retrieve preview's frame and run detector
imageReader.setOnImageAvailableListener({ reader ->
lifecycleScope.launch(Dispatchers.Main) {
val image = reader.acquireNextImage()
logD { "Image available: ${image.timestamp}" }
faceDetector.runFaceDetection(image, getRotationCompensation())
image.close()
}
}, imageReaderHandler)
// Creates list of Surfaces where the camera will output frames
val targets = listOf(viewfinder.holder.surface, imageReader.surface)
// Start a capture session using our open camera and list of Surfaces where frames will go
session = createCaptureSession(cameraDevice, targets, cameraHandler)
val captureRequest = cameraDevice.createCaptureRequest(
CameraDevice.TEMPLATE_PREVIEW).apply {
addTarget(viewfinder.holder.surface)
addTarget(imageReader.surface)
}
// This will keep sending the capture request as frequently as possible until the
// session is torn down or session.stopRepeating() is called
session.setRepeatingRequest(captureRequest.build(), null, cameraHandler)
}
Now, I want to capture a still image...and this is my problem because, ideally, I want:
a full resolution image or, as least, bigger than 480x360
in JPEG format to be able to save it
The Camera2Basic sample demonstrates how to capture an image (samples for Video and SlowMotion are crashing) and MLKit sample uses the so old Camera API !! Fortunately, I've succeeded is mixing these samples to develop my feature but I'm failed to capture a still image with a different resolution.
I think I have to stop the preview session to recreate one for image capture but I'm not sure...
What I have done is the following but it's capturing images in 480x360:
session.stopRepeating()
// Unset the image reader listener
imageReader.setOnImageAvailableListener(null, null)
// Initialize an new image reader which will be used to capture still photos
// imageReader = ImageReader.newInstance(768, 1024, ImageFormat.JPEG, IMAGE_BUFFER_SIZE)
// Start a new image queue
val imageQueue = ArrayBlockingQueue<Image>(IMAGE_BUFFER_SIZE)
imageReader.setOnImageAvailableListener({ reader - >
val image = reader.acquireNextImage()
logD {"[Still] Image available in queue: ${image.timestamp}"}
if (imageQueue.size >= IMAGE_BUFFER_SIZE - 1) {
imageQueue.take().close()
}
imageQueue.add(image)
}, imageReaderHandler)
// Creates list of Surfaces where the camera will output frames
val targets = listOf(viewfinder.holder.surface, imageReader.surface)
val captureRequest = createStillCaptureRequest(cameraDevice, targets)
session.capture(captureRequest, object: CameraCaptureSession.CaptureCallback() {
override fun onCaptureCompleted(
session: CameraCaptureSession,
request: CaptureRequest,
result: TotalCaptureResult) {
super.onCaptureCompleted(session, request, result)
val resultTimestamp = result.get(CaptureResult.SENSOR_TIMESTAMP)
logD {"Capture result received: $resultTimestamp"}
// Set a timeout in case image captured is dropped from the pipeline
val exc = TimeoutException("Image dequeuing took too long")
val timeoutRunnable = Runnable {
continuation.resumeWithException(exc)
}
imageReaderHandler.postDelayed(timeoutRunnable, IMAGE_CAPTURE_TIMEOUT_MILLIS)
// Loop in the coroutine's context until an image with matching timestamp comes
// We need to launch the coroutine context again because the callback is done in
// the handler provided to the `capture` method, not in our coroutine context
# Suppress("BlockingMethodInNonBlockingContext")
lifecycleScope.launch(continuation.context) {
while (true) {
// Dequeue images while timestamps don't match
val image = imageQueue.take()
if (image.timestamp != resultTimestamp)
continue
logD {"Matching image dequeued: ${image.timestamp}"}
// Unset the image reader listener
imageReaderHandler.removeCallbacks(timeoutRunnable)
imageReader.setOnImageAvailableListener(null, null)
// Clear the queue of images, if there are left
while (imageQueue.size > 0) {
imageQueue.take()
.close()
}
// Compute EXIF orientation metadata
val rotation = getRotationCompensation()
val mirrored = cameraFacing == CameraCharacteristics.LENS_FACING_FRONT
val exifOrientation = computeExifOrientation(rotation, mirrored)
logE {"captured image size (w/h): ${image.width} / ${image.height}"}
// Build the result and resume progress
continuation.resume(CombinedCaptureResult(
image, result, exifOrientation, imageReader.imageFormat))
// There is no need to break out of the loop, this coroutine will suspend
}
}
}
}, cameraHandler)
}
If I uncomment the new ImageReader instanciation, I have this exception:
java.lang.IllegalArgumentException: CaptureRequest contains
unconfigured Input/Output Surface!
Can anyone help me ?
This IllegalArgumentException:
java.lang.IllegalArgumentException: CaptureRequest contains unconfigured Input/Output Surface!
... obviously refers to imageReader.surface.
Meanhile (with CameraX) this works different, see CameraFragment.kt ...
Issue #197: Firebase Face Detection Api issue while using cameraX API;
there might soon be a sample application matching your use case.
ImageReader is sensitive to the choice of format and/or combination of usage flags. The documentation points certain combinations of format may be unsupported. With some Android devices (perhaps some older phone models) you might find the IllegalArgumentException is not thrown using the JPEG format. But it doesn't help much - you want something versatile.
What I have done in the past is to use ImageFormat.YUV_420_888 format (this will be backed by the hardware and ImageReader implementation). This format does not contain pre-optimizations that prevent the application accessing the image via the internal array of planes. I notice you have used it successfully already in your initializeCamera() method.
You may then extract the image data from the frame you want
Image.Plane[] planes = img.getPlanes();
byte[] data = planes[0].getBuffer().array();
and then via a Bitmap create the still image using JPEG compression, PNG, or whichever encoding you choose.
ByteArrayOutputStream out = new ByteArrayOutputStream();
YuvImage yuvImage = new YuvImage(data, ImageFormat.NV21, width, height, null);
yuvImage.compressToJpeg(new Rect(0, 0, width, height), 100, out);
byte[] imageBytes = out.toByteArray();
Bitmap bitmap= BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.length);
ByteArrayOutputStream out2 = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.JPEG, 75, out2);
I am working on an app that uses a Recyclerview to display mp3 files, providing its cover art image along with other info. It works but is slow once it starts dealing with a dozen or more cover arts to retrieve, as I am currently doing this from the id3 on the main thread, which I know is not a good idea.
Ideally, I would work with placeholders so that the images can be added as they become available. I've been looking into moving the retrieval to a background thread and have looked at different options: AsyncTask, Service, WorkManager. AsyncTask seems not to be the way to go as I face memory leaks (I need context to retrieve the cover art through MetadataRetriever). So I am leaning away from that. Yet I am struggling to figure out which approach is best in my case.
From what I understand I need to find an approach that allows multithreading and also a means to cancel the retrieval in case the user has already moved on (scrolling or navigating away). I am already using Glide, which I understand should help with the caching.
I know I could rework the whole approach and provide the cover art as images separately, but that seems a last resort to me, as I would rather not weigh down the app with even more data.
The current version of the app is here (please note it will not run as I cannot openly divulge certain aspects). I am retrieving the cover art as follows (on the main thread):
static public Bitmap getCoverArt(Uri medUri, Context ctxt) {
MediaMetadataRetriever mmr = new MediaMetadataRetriever();
mmr.setDataSource(ctxt, medUri);
byte[] data = mmr.getEmbeddedPicture();
if (data != null) {
return BitmapFactory.decodeByteArray(data, 0, data.length);
} else {
return null;
}
}
I've found many examples with AsyncTask or just keeping the MetaDataRetriever on the main thread, but have yet to find an example that enables a dozen or more cover arts to be retrieved without slowing down the main thread. I would appreciate any help and pointers.
It turns out it does work with AsyncTask, as long as it is not a class onto itself but setup and called from a class with context. Here is a whittled down version of my approach (I am calling this from within my Adapter.):
//set up titles and placeholder image so we needn't wait on the image to load
titleTv.setText(selectedMed.getTitle());
subtitleTv.setText(selectedMed.getSubtitle());
imageIv.setImageResource(R.drawable.ic_launcher_foreground);
imageIv.setAlpha((float) 0.2);
final long[] duration = new long[1];
//a Caching system that helps reduce the amount of loading needed. See: https://github.com/cbonan/BitmapFun?files=1
if (lruCacheManager.getBitmapFromMemCache(selectedMed.getId() + position) != null) {
//is there an earlier cached image to reuse? imageIv.setImageBitmap(lruCacheManager.getBitmapFromMemCache(selectedMed.getId() + position));
imageIv.setAlpha((float) 1.0);
titleTv.setVisibility(View.GONE);
subtitleTv.setVisibility(View.GONE);
} else {
//time to load and show the image. For good measure, the duration is also queried, as this also needs the setDataSource which causes slow down
new AsyncTask<Uri, Void, Bitmap>() {
#Override
protected Bitmap doInBackground(Uri... uris) {
MediaMetadataRetriever mmr = new MediaMetadataRetriever();
mmr.setDataSource(ctxt, medUri);
byte[] data = mmr.getEmbeddedPicture();
Log.v(TAG, "async data: " + Arrays.toString(data));
String durationStr = mmr.extractMetadata(MediaMetadataRetriever.METADATA_KEY_DURATION);
duration[0] = Long.parseLong(durationStr);
if (data != null) {
InputStream is = new ByteArrayInputStream(mmr.getEmbeddedPicture());
return BitmapFactory.decodeStream(is);
} else {
return null;
}
}
#Override
protected void onPostExecute(Bitmap bitmap) {
super.onPostExecute(bitmap);
durationTv.setVisibility(View.VISIBLE);
durationTv.setText(getDisplayTime(duration[0], false));
if (bitmap != null) {
imageIv.setImageBitmap(bitmap);
imageIv.setAlpha((float) 1.0);
titleTv.setVisibility(View.GONE);
subtitleTv.setVisibility(View.GONE);
} else {
titleTv.setVisibility(View.VISIBLE);
subtitleTv.setVisibility(View.VISIBLE);
}
lruCacheManager.addBitmapToMemCache(bitmap, selectedMed.getId() + position);
}
}.execute(medUri);
}
I have tried working with Glide for the caching, but I haven't been able to link the showing/hiding of the TextViews to whether there is a bitmap. In a way though, this is sleeker as I don't need to load the bulk of the Glide-library. So I am happy with this for now.
I'm trying to convert image taken from resources to ByteArray which
will later be send through Socket. I've been measuring time of each of this conversion.
I've done it on both Flutter and native Android (Kotlin). All of the test were done on the same image which was about 1-2MB.
Flutter code :
sendMessage() async {
if (socket != null) {
Stopwatch start = Stopwatch()..start();
final imageBytes = await rootBundle.load('assets/images/stars.jpg');
final image = base64Encode(imageBytes.buffer.asUint8List(imageBytes.offsetInBytes, imageBytes.lengthInBytes));
print('Converting took ${start.elapsedMilliseconds}');
socket.emit("message", [image]);
}
}
Kotlin code:
private fun sendMessage() {
var message = ""
val thread = Thread(Runnable {
val start = SystemClock.elapsedRealtime()
val bitmap = BitmapFactory.decodeResource(resources, R.drawable.stars)
message = Base64.encodeToString(getBytesFromBitmap(bitmap), Base64.DEFAULT)
Log.d("Tag", "Converting time was : ${SystemClock.elapsedRealtime() - start}")
})
thread.start()
thread.join()
socket.emit("message", message)
}
private fun getBytesFromBitmap(bitmap: Bitmap): ByteArray? {
val stream = ByteArrayOutputStream()
bitmap.compress(Bitmap.CompressFormat.JPEG, 100, stream)
return stream.toByteArray()
}
I've been actually expecting native code to be much much faster than Flutter's but thats not the case.. Conversion for Flutter takes about 50ms and its around 2000-3000ms for native.
I thought that Threading may be the case, so I've tried to run this conversion on background thread for native code but it didn't help.
Can you please tell me why is there such a different in time, and how I can implement it better in native code? Is there a way to omit casting to Bitmap etc.? Maybe this makes it so long.
EDIT. Added getBytesFromBitmap function
the difference you see is that in flutter code you just read your data without any image decoding, while in kotlin you are first decoding to Bitmap and then you are compress()ing it back - if you want to speed it up simply get an InputStream by calling Resources#openRawResource and read your image resource without any decoding
It have something to do with the way you convert it to bytes... Can you please post your
getBytesFromBitmap func? Plus, the conversion in native code really should be done in background thread, please upload the your results in this case.
I have a camera2 implementation. The current setup is, it uses a texture view surface to display the actual camera view and an ImageReader surface for capturing images.
Now I want to capture preview frames as well. So I tried adding a new ImageReader surface for capturing frames. But when I add that surface to createCaptureSession request, the screen goes blank. What could possibly be wrong? Below is the code that I use to add surfaces to createCaptureSession
val surface = preview.surface
?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)
val previewIRSurface = previewImageReader?.surface
?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)
val captureSurface = captureImageReader?.surface
?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)
try {
val template = if (zsl) CameraDevice.TEMPLATE_ZERO_SHUTTER_LAG else CameraDevice.TEMPLATE_PREVIEW
previewRequestBuilder = camera?.createCaptureRequest(template)
?.apply { addTarget(surface) }
?: throw CameraAccessException(CameraAccessException.CAMERA_ERROR)
val surfaces: ArrayList<Surface> = arrayListOf(surface, previewIRSurface, captureSurface)
camera?.createCaptureSession(surfaces, sessionCallback, backgroundHandler)
} catch (e: CameraAccessException) {
throw RuntimeException("Failed to start camera session")
}
The initialization of ImageReaders is like this.
private fun prepareImageReaders() {
val largestPreview = previewSizes.sizes(aspectRatio).last()
previewImageReader?.close()
previewImageReader = ImageReader.newInstance(
largestPreview.width,
largestPreview.height,
internalOutputFormat,
4 // maxImages
).apply { setOnImageAvailableListener(onPreviewImageAvailableListener, backgroundHandler) }
val largestPicture = pictureSizes.sizes(aspectRatio).last()
captureImageReader?.close()
captureImageReader = ImageReader.newInstance(
largestPicture.width,
largestPicture.height,
internalOutputFormat,
2 // maxImages
).apply { setOnImageAvailableListener(onCaptureImageAvailableListener, backgroundHandler) }
}
More clarifications about the parameters used above:
internalOutput format is either ImageFormat.JPEG or ImageFormat.YUV_420_888.
Image sizes are based on best possible sizes
It works good with either of the image readers individually but as soon as I add both together, blank screen!
Testing on Samsung Galaxy S8 with Android Oreo (8.0)
The original code is here https://github.com/pvasa/cameraview-ex/blob/development/cameraViewEx/src/main/api21/com/priyankvasa/android/cameraviewex/Camera2.kt
maxImages == 4 may be too much and exhaust your RAM. Also, it's not clear what internalOutputFormat you use, and whether it is compatible with the largestPreview size.
The bottom line is, study the long list of tables for supported surface list parameter of createCaptureSession(). Depending on your camera capabilities, the three surfaces that you use, could be too much.
From the comments below, a working solution: "The error itself doesn't say much [...] but upon searching, it is found that multiple surfaces are not supported for JPEG format. Upon changing it to YUV_420_888 it works flawlessly."