Hi I'm trying to implement dlib facial landmark detection in android. I know, but I need as many fps I can get. There are 2 issues that I face,
Conversion Chain
Resizing
Currently, I am getting the data from a preview callback set to a camera. It outputs a byte[] of a NV21 Image. Since dlib dont know image and only know array2d<dlib::rgb_pixel>, I need to conform the data to it. The implementation that I get uses bitmap, and when I try to use there code, I have a chain of conversion byte[]->bmp->array2d, I want to implement a byte[]->array2d conversion.
Now, I need to leverage the performance of dlib by manipulating the size of the image fed in to it. My use-case though doesn't involve small faces so I can down-scale the input image to boost performance, but lets say I am successful on making the byte[]->array2d conversion, how can I resize the image? Resizing in bitmap though have many fast implementations but I need to cut the bitmap involvement to extract more fps. I have an option on resizing the byte[] or the converted one array2d, but again... how? Im guessing its good to do the resizing after the conversion because it will now be operating on native and not on java.
Edit
The down-scaling should take the byte[](not the dlib::arrray2d) form as input since I need to do something on the down-scaled byte[].
So my final problem is to implement this on jni
byte[] resize(ByteArray img, Size targetSize);
and
dlib::array2d<rgb_pixel> convert(ByteArray img);
This question helped me a lot, and made me understand the nv21 structure. Using the code in the question I was able to develop a converter from nv21 byte[] to array2d<rgb>.
What's left unsloved now is the resize.
Performing any resizing in Java is probably bad because of poor compiler optimization. Your most performant option CPU-wise would probably be to write a specialized NV12 resize in C++, then convert to RGB. Swapping the order may only be slightly slower though, and much easier to write.
Your other option is to do all this on the GPU using shaders. GPU are way faster at this sort of thing, but they are finnicky. You might need a CPU fallback anyway (if the GPU isn't available for whatever reason, not familiar with Android).
Related
To start with the question: what is the most efficient way to initialize and use ImageReader with the camera2 api, knowing that I am always going to convert the capture into a Bitmap?
I'm playing around with the Android camera2 samples, and everything is working quite nicely. However, for my purposes I always need to perform some post processing on captured still images, for which I require a Bitmap object. Presently I am using BitmapFactory.decodeByteArray(...) using the bytes coming from the ImageReader.acquireNextImage().getPlanes()[0].getBuffer() (I'm paraphrasing). While this works acceptably, I still feel like there should be a way to improve performance. The captures are encoded in ImageFormat.Jpeg and need to be decoded again to get the Bitmap, which seems redundant. Ideally I'd obtain them in PixelFormat.RGB_888 and just copy that to a Bitmap using Bitmap.copyPixelsFromBuffer(...), but it doesn't seem like initializing an ImageReader with that format has reliable device support. YUV_420_888 could be another option, but looking around SO it seems that it requires jumping through some hoops to decode into a Bitmap. Is there a recommended way to do this?
The question is what you are optimizing for.
Jpeg is without doubt the easiest format supported by all devices. Decoding it to bitmap is not redundant as it seems because encoding the picture into jpeg he is usually performed by kind of hardware. This means that uses minimal bandwidth to transmit the image from the sensor to your application. on some devices this is the only way to get maximum resolution. BitmapFactory.decodeByteArray(...) is often performed by special hardware decoder too. The major problem with this call is that may cause out of memory exception, because the output bitmap is too big. So you will find many examples the do subsampled decoding, tuned for the use case where the bitmap must be displayed on the phone screen.
If your device supports required resolution with RGB_8888, go for it: this needs minimal post-processing. But scaling such image down may be more CPU intensive then dealing with Jpeg, and memory consumption may be huge. Anyways, only few devices support this format for camera capture.
As for YUV_420_888 and other YUV formats,
the advantages over Jpeg are even smaller than for RGB.
If you need the best quality image and don't have memory limitations, you should go for RAW images which are supported on most high-end devices these days. You will need your own conversion algorithm, and probably make different adaptations for different devices, but at least you will have full command of the picture acquisition.
After a while I now sort of have an answer to my own question, albeit not a very satisfying one. After much consideration I attempted the following:
Setup a ScriptIntrinsicYuvToRGB RenderScript of the desired output size
Take the Surface of the used input allocation, and set this as the target surface for the still capture
Run this RenderScript when a new allocation is available and convert the resulting bytes to a Bitmap
This actually worked like a charm, and was super fast. Then I started noticing weird behavior from the camera, which happened on other devices as well. As it would turn out, the camera HAL doesn't really recognize this as a still capture. This means that (a) the flash / exposure routines don't fire in this case when they need to and (b) if you have initiated a precapture sequence before your capture auto-exposure will remain locked unless you manage to unlock it using AE_PRECAPTURE_TRIGGER_CANCEL (API >= 23) or some other lock / unlock magic which I couldn't get to work on either device. Unless you're fine with this only working in optimal lighting conditions where no exposure adjustment is necessary, this approach is entirely useless.
I have one more idea, which is to setup an ImageReader with a YUV_420_888 output and incorporating the conversion routine from this answer to get RGB pixels from it. However, I'm actually working with Xamarin.Android, and RenderScript user scripts are not supported there. I may be able to hack around that, but it's far from trivial.
For my particular use case I have managed to speed up JPEG decoding to acceptable levels by carefully arranging background tasks with subsampled decodes of the versions I need at multiple stages of my processing, so implementing this likely won't be worth my time any time soon. If anyone is looking for ideas on how to approach something similar though; that's what you could do.
Change the Imagereader instance using a different ImageFormat like this:
ImageReader.newInstance(width, height, ImageFormat.JPEG, 1)
I know the topic has already appeared multiple times but none of the solutions actually seems to be working, ranging from answers suitable for the old camera API (where the YUV data comes in a neat byte[] array), via corrupted images, to saving JPEG's all in "green scale" using RenderScript (which by the way is the best I am currently able to do).
The way the Camera2Basic example does it is that it simply sets the Type.Builder's format to JPEG. The problem with this (as discussed in multiple posts as well) is that it slows down the camera pipeline. YUV_420_888 works much much faster.
So, did anyone manage to perform a proper YUV_420_888 -> JPEG conversion?
I tried searching in a ton of places about doing this, with no results. I did read that the only (as far as I know) way to obtain image frames was to use a ImageReader, which gives me a Image to work with. However, a lot of work must be done before I have a nice enough image (converting Image to byte array, then converting between formats - YUV_420_888 to ARGB_8888 - using RenderScript, then turning it into a Bitmap and rotating it manually - or running the application on landscape mode). By this point a lot of processing is made, and I haven't even started the actual processing yet (I plan on running some native code on it). Additionally, I tried to lower the resolution, with no success, and there is a significant delay when drawing on the surface.
Is there a better approach to this? Any help would be greatly appreciated.
Im not sure what exactly you are doing with the images, but a lot of times only a grayscale image is actually needed (again depending on your exact goal) If your camera outputs YUV, the grayscale information is in the Y channel. The nice thing,is you don't need to convert to numerous colorspaces and working with only one layer (as opposed to three) decreases the size of your data set greatly.
If you need color images then this wouldn't help
I need to make some operations on bitmaps in android, such as chanding pixel colors, transparency, painting another bitmap on the first one with transparency, zooming, croping, applying masks and so on. Now I make it with java and it is really slow. It take about 5 seconds to compute a 600x600 pixels image on nexus 4.
What technology will be suitable for this operations?
I tried library GPUImage that uses OpenGL ES 2.0, but I faced a problem when trying to apply a mask to an image, mask size is 600x600 as the original image, so I should pass an array of 1440000 color values, and the GLSL language doesnt support such big arrays. 360000 of vec4 values is too large for it too.
May be I can pass a whole image as a texture2d object, but I cant find how to do it. Cold someone please give me a tutorial?
The second option is renderscript. It is not so suitable because I want phones with android 2.3 has fast bitmap operations, but all in all improving speed on 3.0+ devices will be a good thing. The problem with it was how to pass it data of second bitmap - 360000 of float4 values or a bitmap itself. I coudnt find information about it too, so I would be thankful if someone give me an advice or tutorial.
The third option is using NDK. But I dont know will it be fast enough and again cant find good information to start with ndk in general and image processing with ndk in particular. So I again need an advice or tutorial.
So, all in all, please tell me what other ways of fast image processing are there and please give me some advices or tutorials on this 3 methods.
And what is the best way to make operations on images really fast?
If you find GLES suitable, divide the image into small tiles, and operate on one tile at a time. (Some drivers work this way.)
Using an Android (2.3.3) phone, I can use the camera to retrieve a preview with the onPreviewFrame(byte[] data, Camera camera) method to get the YUV image.
For some image processing, I need to convert this data to an RGB image and show it on the device. Using the basic java / android method, this runs at a horrible rate of less then 5 fps...
Now, using the NDK, I want to speed things up. The problem is: How do I convert the YUV array to an RGB array in C? And is there a way to display it (using OpenGL perhaps?) in the native code? Real-time should be possible (the Qualcomm AR demos showed us that).
I cannot use the setTargetDisplay and put an overlay on it!
I know Java, recently started with the Android SDK and have zero experience in C
Have you considered using OpenCV's Android port? It can do a lot more than just color conversion, and it's quite fast.
A Google search returned this page for a C implementation of YUV->RGB565. The author even included the JNI wrapper for it.
You can also succeed by staying with Java. I did this for the imagedetectíon of the androangelo-app.
I used the sample code which you find here by searching "decodeYUV".
For processing the frames, the essential part to consider is the image-size.
Depending on the device you may get quite large images. i.e. for the Galaxy S2
the smallest supported previewsize is 640*480. This is a big amount of pixels.
What I did, is to use only every second row and every second column, after yuvtorgb decoding. So processing a 320*240 image works quite well and allowed me to get frame-rates of 20fps. (including some noise-reduction, a color-conversion from rgb to hsv and a circledetection)
In addition You should carefully check the size of the image-buffer provided to the setPreview function. If it is too small, the garbage-collection will spoil everything.
For the result you can check the calibration-screen of the androangelo-app. There I had an overlay of the detected image over the camera-preview.