Develop Reference

Need to capture a still image during face detection with MLKit and Camera2

I'm developing a Face Detection feature with Camera2 and MLKit.
In the Developer Guide, in the Performance Tips part, they say to capture images in ImageFormat.YUV_420_888 format if using the Camera2 API, which is my case.
Then, in the Face Detector part, they recommend to use an image with dimensions of at least 480x360 pixels for faces recognition in real time, which is again my case.
Ok, let's go ! Here is my code, working well
private fun initializeCamera() = lifecycleScope.launch(Dispatchers.Main) {
// Open the selected camera
cameraDevice = openCamera(cameraManager, getCameraId(), cameraHandler)
val previewSize = if (isPortrait) {
Size(RECOMMANDED_CAPTURE_SIZE.width, RECOMMANDED_CAPTURE_SIZE.height)
} else {
Size(RECOMMANDED_CAPTURE_SIZE.height, RECOMMANDED_CAPTURE_SIZE.width)
}
// Initialize an image reader which will be used to display a preview
imageReader = ImageReader.newInstance(
previewSize.width, previewSize.height, ImageFormat.YUV_420_888, IMAGE_BUFFER_SIZE)
// Retrieve preview's frame and run detector
imageReader.setOnImageAvailableListener({ reader ->
lifecycleScope.launch(Dispatchers.Main) {
val image = reader.acquireNextImage()
logD { "Image available: ${image.timestamp}" }
faceDetector.runFaceDetection(image, getRotationCompensation())
image.close()
}
}, imageReaderHandler)
// Creates list of Surfaces where the camera will output frames
val targets = listOf(viewfinder.holder.surface, imageReader.surface)
// Start a capture session using our open camera and list of Surfaces where frames will go
session = createCaptureSession(cameraDevice, targets, cameraHandler)
val captureRequest = cameraDevice.createCaptureRequest(
CameraDevice.TEMPLATE_PREVIEW).apply {
addTarget(viewfinder.holder.surface)
addTarget(imageReader.surface)
}
// This will keep sending the capture request as frequently as possible until the
// session is torn down or session.stopRepeating() is called
session.setRepeatingRequest(captureRequest.build(), null, cameraHandler)
}
Now, I want to capture a still image...and this is my problem because, ideally, I want:
a full resolution image or, as least, bigger than 480x360
in JPEG format to be able to save it
The Camera2Basic sample demonstrates how to capture an image (samples for Video and SlowMotion are crashing) and MLKit sample uses the so old Camera API !! Fortunately, I've succeeded is mixing these samples to develop my feature but I'm failed to capture a still image with a different resolution.
I think I have to stop the preview session to recreate one for image capture but I'm not sure...
What I have done is the following but it's capturing images in 480x360:
session.stopRepeating()
// Unset the image reader listener
imageReader.setOnImageAvailableListener(null, null)
// Initialize an new image reader which will be used to capture still photos
// imageReader = ImageReader.newInstance(768, 1024, ImageFormat.JPEG, IMAGE_BUFFER_SIZE)
// Start a new image queue
val imageQueue = ArrayBlockingQueue<Image>(IMAGE_BUFFER_SIZE)
imageReader.setOnImageAvailableListener({ reader - >
val image = reader.acquireNextImage()
logD {"[Still] Image available in queue: ${image.timestamp}"}
if (imageQueue.size >= IMAGE_BUFFER_SIZE - 1) {
imageQueue.take().close()
}
imageQueue.add(image)
}, imageReaderHandler)
// Creates list of Surfaces where the camera will output frames
val targets = listOf(viewfinder.holder.surface, imageReader.surface)
val captureRequest = createStillCaptureRequest(cameraDevice, targets)
session.capture(captureRequest, object: CameraCaptureSession.CaptureCallback() {
override fun onCaptureCompleted(
session: CameraCaptureSession,
request: CaptureRequest,
result: TotalCaptureResult) {
super.onCaptureCompleted(session, request, result)
val resultTimestamp = result.get(CaptureResult.SENSOR_TIMESTAMP)
logD {"Capture result received: $resultTimestamp"}
// Set a timeout in case image captured is dropped from the pipeline
val exc = TimeoutException("Image dequeuing took too long")
val timeoutRunnable = Runnable {
continuation.resumeWithException(exc)
}
imageReaderHandler.postDelayed(timeoutRunnable, IMAGE_CAPTURE_TIMEOUT_MILLIS)
// Loop in the coroutine's context until an image with matching timestamp comes
// We need to launch the coroutine context again because the callback is done in
// the handler provided to the `capture` method, not in our coroutine context
# Suppress("BlockingMethodInNonBlockingContext")
lifecycleScope.launch(continuation.context) {
while (true) {
// Dequeue images while timestamps don't match
val image = imageQueue.take()
if (image.timestamp != resultTimestamp)
continue
logD {"Matching image dequeued: ${image.timestamp}"}
// Unset the image reader listener
imageReaderHandler.removeCallbacks(timeoutRunnable)
imageReader.setOnImageAvailableListener(null, null)
// Clear the queue of images, if there are left
while (imageQueue.size > 0) {
imageQueue.take()
.close()
}
// Compute EXIF orientation metadata
val rotation = getRotationCompensation()
val mirrored = cameraFacing == CameraCharacteristics.LENS_FACING_FRONT
val exifOrientation = computeExifOrientation(rotation, mirrored)
logE {"captured image size (w/h): ${image.width} / ${image.height}"}
// Build the result and resume progress
continuation.resume(CombinedCaptureResult(
image, result, exifOrientation, imageReader.imageFormat))
// There is no need to break out of the loop, this coroutine will suspend
}
}
}
}, cameraHandler)
}
If I uncomment the new ImageReader instanciation, I have this exception:
java.lang.IllegalArgumentException: CaptureRequest contains
unconfigured Input/Output Surface!
Can anyone help me ?

This IllegalArgumentException:
java.lang.IllegalArgumentException: CaptureRequest contains unconfigured Input/Output Surface!
... obviously refers to imageReader.surface.
Meanhile (with CameraX) this works different, see CameraFragment.kt ...
Issue #197: Firebase Face Detection Api issue while using cameraX API;
there might soon be a sample application matching your use case.

ImageReader is sensitive to the choice of format and/or combination of usage flags. The documentation points certain combinations of format may be unsupported. With some Android devices (perhaps some older phone models) you might find the IllegalArgumentException is not thrown using the JPEG format. But it doesn't help much - you want something versatile.
What I have done in the past is to use ImageFormat.YUV_420_888 format (this will be backed by the hardware and ImageReader implementation). This format does not contain pre-optimizations that prevent the application accessing the image via the internal array of planes. I notice you have used it successfully already in your initializeCamera() method.
You may then extract the image data from the frame you want
Image.Plane[] planes = img.getPlanes();
byte[] data = planes[0].getBuffer().array();
and then via a Bitmap create the still image using JPEG compression, PNG, or whichever encoding you choose.
ByteArrayOutputStream out = new ByteArrayOutputStream();
YuvImage yuvImage = new YuvImage(data, ImageFormat.NV21, width, height, null);
yuvImage.compressToJpeg(new Rect(0, 0, width, height), 100, out);
byte[] imageBytes = out.toByteArray();
Bitmap bitmap= BitmapFactory.decodeByteArray(imageBytes, 0, imageBytes.length);
ByteArrayOutputStream out2 = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.JPEG, 75, out2);

How to get URL from PlacePhotoMetadata

I can't seem to get the url, from the PlacePhotoMetadata object. Debugger shows that there is an URL there but I can't seem to access it.
How do you access the URL in the object?
val placeId = "ChIJa147K9HX3IAR-lwiGIQv9i4"
val photoMetadataResponse = mGeoDataClient.getPlacePhotos(placeId)
photoMetadataResponse.addOnCompleteListener { task ->
// Get the list of photos
val photos = task.result
// Get the PlacePhotoMetadataBuffer (metadata for all of the photos)
val photoMetadataBuffer = photos.photoMetadata
// Get the first photo in the list
for (photo in photoMetadataBuffer) {
// Get the attribution text
val attribution = photo.attributions
}
}

You can't. Take a look at the documentation for PlacePhotoMetadata. There are methods to download a bitmap of the image, but no methods that return the URL.
To get the photo you should do something like this:
// this is your for-loop:
photoMetadataBuffer.forEach { photo ->
photo.getPhoto(client).setResultCallback({ result ->
// do whatever you want here:
showPhotoWithAttribution(photo.attributions, result.getBitmap())
})
}
Note that replacing a for-loop with a forEach call has no real advantage, it just makes your code look cleaner.

save image to custom folder titanium

I have developed an app to take a photo from the phone camera. But now I need to store the image to the phone memory into a folder created by me.
I have tried this:
var filename = Titanium.Filesystem.resourcesDirectory + "/newImageFile.jpg";
var imageFile = Titanium.Filesystem.getFile(filename);
imageFile.write(capturedImg);
But it does not apear in the gallery. How can I store the image to the phone memory and how can I create a costume folder in the phone memory to store the image?

- (void)imagePickerController:(UIImagePickerController *)picker didFinishPickingMediaWithInfo:(NSDictionary<NSString *,id> *)info
{
NSData *imageData = [info objectForKey:UIImagePickerControllerOriginalImage];
if(picker.sourceType==UIImagePickerControllerSourceTypeCamera)
{
//to save camera roll
[[PHPhotoLibrary sharedPhotoLibrary] performChanges:^{
[PHAssetChangeRequest creationRequestForAssetFromImage:image];
} completionHandler:nil];
}
}
This will save image in camera roll which is taken by you

This Will Create Photo Album and save into this album.
I came up with this singleton class to handle it:
import Photos
class CustomPhotoAlbum {
static let albumName = "Titanium"
static let sharedInstance = CustomPhotoAlbum()
var assetCollection: PHAssetCollection!
init() {
func fetchAssetCollectionForAlbum() -> PHAssetCollection! {
let fetchOptions = PHFetchOptions()
fetchOptions.predicate = NSPredicate(format: "title = %#", CustomPhotoAlbum.albumName)
let collection = PHAssetCollection.fetchAssetCollectionsWithType(.Album, subtype: .Any, options: fetchOptions)
if let firstObject: AnyObject = collection.firstObject {
return collection.firstObject as! PHAssetCollection
}
return nil
}
if let assetCollection = fetchAssetCollectionForAlbum() {
self.assetCollection = assetCollection
return
}
PHPhotoLibrary.sharedPhotoLibrary().performChanges({
PHAssetCollectionChangeRequest.creationRequestForAssetCollectionWithTitle(CustomPhotoAlbum.albumName)
}) { success, _ in
if success {
self.assetCollection = fetchAssetCollectionForAlbum()
}
}
}
func saveImage(image: UIImage) {
if assetCollection == nil {
return // If there was an error upstream, skip the save.
}
PHPhotoLibrary.sharedPhotoLibrary().performChanges({
let assetChangeRequest = PHAssetChangeRequest.creationRequestForAssetFromImage(image)
let assetPlaceholder = assetChangeRequest.placeholderForCreatedAsset
let albumChangeRequest = PHAssetCollectionChangeRequest(forAssetCollection: self.assetCollection)
albumChangeRequest.addAssets([assetPlaceholder])
}, completionHandler: nil)
}
}
When you first instantiate the class, the custom album will be created if it doesn't already exist. You can save an image like this:
CustomPhotoAlbum.sharedInstance.saveImage(image)
NOTE: The CustomPhotoAlbum class assumes the app already has permission to access the Photo Library. Dealing with the permissions is a bit outside the scope of this question/answer. So make sure PHPhotoLibrary.authorizationStatus() == .Authorize before you use it. And request authorization if necessary.

ListView reusing old images

I created a plugin using Picasso and it uses the android.widget.ImageView to load the cached image into.
The plugin works fine if using a Repeater but whenever i try using it with a ListView after scrolling past about the 7th item the ListView begins to reuse old images even if the image source is different

The reason why is because list views reuse the entire fragment; so what happens is that your img being reused gets the old image shown unless you clear it.
I actually use Picasso myself; and this is my current picasso library.
So if you look in my code below, when I set the new .url, I clear the existing image. (I made a comment on the specific line) -- This way the image now show blank, and then picasso loads it from either memory, disk or a remote url (in my case a remote url) and it will assign the proper image.
"use strict";
var Img = require('ui/image').Image;
var application = require("application");
var PT = com.squareup.picasso.Target.extend("Target",{
_owner: null,
_url: null,
onBitmapLoaded: function(bitmap, from) {
// Since the actual image / target is cached; it is possible that the
// target will not match so we don't replace the image already seen
if (this._url !== this._owner._url) {
return;
}
this._owner.src = bitmap;
},
onBitmapFailed: function(ed) {
console.log("Failed File", this._url);
},
onPrepareLoad: function(ed) {
}
});
Object.defineProperty(Img.prototype, "url", {
get: function () {
return this._url;
},
set: function(src) {
if (src == null || src === "") {
this._url = "";
this.src = null;
return;
}
var dest = src;
this._url = dest;
this.src = null; // -- THIS IS THE LINE TO CLEAR THE IMAGE
try {
var target = new PT();
target._owner = this;
target._url = dest;
var x = com.squareup.picasso.Picasso.with(application.android.context).load(dest).into(target);
} catch (e) {
console.log("Exception",e);
}
},
enumerable: true,
configurable: true
});
Please note you only need to require this class once, then it attaches itself to the <Image> component and adds the new .url property; this allows me to use this in the Declarative XML in all the rest of the screens and when I need picasso, I just use the .url property to have picasso take over the loading of that image.

Using muPDF with curl/flip effect

I'm using muPDF for reading PDFs in my application. I don't like its default animation (Switching horizontally). In other side i found this brilliant library for curl effect on images, and this project for flip-flap effect on layouts.
In curl sample project, in CurlActivity, all of data are images and set in PageProvider like this:
private class PageProvider implements CurlView.PageProvider {
// Bitmap resources.
private int[] mBitmapIds = { R.drawable.image1, R.drawable.image2,
R.drawable.image3, R.drawable.image4};
And use it like this:
private CurlView mCurlView;
mCurlView = (CurlView) findViewById(R.id.curl);
mCurlView.setPageProvider(new PageProvider());
And CurlView extends from GLSurfaceView and implements View.OnTouchListener, CurlRenderer.Observer
But in muPDF if i'm not mistaken, data are in core object. core is instance of MuPDFCore. And using it like this:
MuPDFReaderView mDocView;
MuPDFView pageView = (MuPDFView) mDocView.getDisplayedView();
mDocView.setAdapter(new MuPDFPageAdapter(this, this, core));
MuPDFReaderView extends ReaderView and ReaderView extends AdapterView<Adapter> and implements GestureDetector.OnGestureListener, ScaleGestureDetector.OnScaleGestureListener, Runnable.
My question is where how can I using curl effect in muPDF? Where should I get pages one by one and converting them to bitmaps? and then changing aspects of the Adapter in muPDF to CurlView.
In flip-flap sample project, in FlipHorizontalLayoutActivity (I like this effect too), we have these:
private FlipViewController flipView;
flipView = new FlipViewController(this, FlipViewController.HORIZONTAL);
flipView.setAdapter(new TravelAdapter(this));
setContentView(flipView);
And FlipViewController extends AdapterView<Adapter>, and data set in TravelAdapter that extends BaseAdapter.
No one has done this before? Or can help me to do that?!
EDIT:
I found another good open source PDF reader with curl effect called fbreaderJ. its developer says "An additional module that allows to open PDF files in FBReader. Based on radaee pdf library."
I got confused! cause radaeepdf is closed source and downloadable project is just for demo and inserted username and password is for this package.
People want to change whole fbreader project such as package name.
Another issue for make me confused is where is this additional module source code?!
Anyway, if someone wants to help me, fbreader has done it very well.
EDIT:
I talked to Robin Watts, who developed muPDF (or one of developers), and he said:
Have you read platform/android/ClassStructure.txt ? MuPDF is
primarily a C library. The standard api is therefore a C one. Rather
than exposing that api exactly as is to Java (which would be the
nicest solution, and something that I've done some work on, but have
not completed due to lack of time), we've implemented MuPDFCore to
wrap up just the bits we needed. MuPDFCore handles opening a PDF file,
and getting bitmaps from it to be used in views. or rather, MuPDFCore
returns 'views', not 'bitmaps'. If you need bitmaps, then you're going
to need to make changes in MuPDFCore.
There are too many errors when changing a little part of MuPDFReaderView class. I get confused! These are related to each other.
Please answer more precisely.
EDIT:
And bounty has expired.

If the muPDF does not support rendering to a bitmap, you have no other choice than rendering to a regular view and take a screen dump to a bitmap like this:
View content = findViewById(R.id.yourPdfView);
Bitmap bitmap = content.getDrawingCache();
Then use this bitmap as input to your other library.

Where should i get pages one by one and converting them to bitmaps?
In our application (newspaper app) we use MuPDF to render PDFs.
The workflow goes like this:
Download PDF file (we have one PDF per newspaper page)
Render it with MuPDF
Save the bitmap to the filesystem
Load the Bitmap from filesystem as background image to a view
So, finally, what we use is MuPDFCore.java and its methods drawPage(...) and onDestroy()
Is this what you want to know or do i miss the point?
EDIT
1.) I think it is not necessary to post code how to download a file. But after downloading i add a RenderTask (extends from Runnable) to a Renderqueue and trigger that queue. The RenderTask needs some information for rendering:
/**
* constructs a new RenderTask instance
* #param context: you need Context for MuPdfCore instance
* #param pageNumber
* #param pathToPdf
* #param renderCallback: callback to set bitmap to the view after
* rendering
* #param heightOfRenderedBitmap: this is the target height
* #param widthOfRenderedBitmap: this is the target width
*/
public RenderTask (Context context, Integer pageNumber, String pathToPdf, IRenderCallback,
renderCallback, int heightOfRenderedBitmap,
int widthOfRenderedBitmap) {
//store things in fields
}
2.) + 3.) The Renderqueue wraps the RenderTask in a new Thread and starts it. So the run-method of the RenderTask will be invoked:
#Override
public void run () {
//do not render it if file exists
if (exists () == true) {
finish();
return;
}
Bitmap bitmap = render();
//if something went wrong, we can't store the bitmap
if (bitmap == null) {
finish();
return;
}
//now save the bitmap
// in my case i save the destination path in a String field
imagePath = save(bitmap, new File("path/to/your/destination/folder/" + pageNumber + ".jpg"));
bitmap.recycle();
finish();
}
/**
* let's trigger the callback
*/
private void finish () {
if (renderCallback != null) {
// i send the whole Rendertask to callback
// maybe in your case it is enough to send the pageNumber or path to
// renderend bitmap
renderCallback.finished(this);
}
}
/**
* renders a bitmap
* #return
*/
private Bitmap render() {
MuPDFCore core = null;
try {
core = new MuPDFCore(context, pathToPdf);
} catch (Exception e) {
return null;
}
Bitmap bm = Bitmap.createBitmap(widthOfRenderedBitmap, heightOfRenderedBitmap, Config.ARGB_8888);
// here you render the WHOLE pdf cause patch-x/-y == 0
core.drawPage(bm, 0, widthOfRenderedBitmap, heightOfRenderedBitmap, 0, 0, widthOfRenderedBitmap, heightOfRenderedBitmap, core.new Cookie());
core.onDestroy();
core = null;
return bm;
}
/**
* saves bitmap to filesystem
* #param bitmap
* #param image
* #return
*/
private String save(Bitmap bitmap, File image) {
FileOutputStream out = null;
try {
out = new FileOutputStream(image.getAbsolutePath());
bitmap.compress(Bitmap.CompressFormat.JPEG, 80, out);
return image.getAbsolutePath();
} catch (Exception e) {
return null;
}
finally {
try {
if (out != null) {
out.close();
}
} catch(Throwable ignore) {}
}
}
}
4.) I think it is not necessary to post code how to set a bitmap as background of a view