I've got an Android application which does motion detection and video recording. It supports both the Camera and Camera2 APIs in order to provide backwards compatibility. I'm using an ImageReader with the Camera2 API in order to do motion detection. I'm currently requesting JPEG format images, which are very slow. I understand that requesting YUV images would be faster, but is it true that the YUV format varies depending on which device is being used? I just wanted to check before I give up on optimizing this.
All devices will support NV21 and YV12 formats for the old camera API (since API 12), and for camera2, all devices will support YUV_420_888.
YUV_420_888 is a flexible YUV format, so it can represent multiple underlying formats (including NV21 and YV12). So you'll need to check the pixel and row strides in the Images from the ImageReader to ensure you're reading through the 3 planes of data correctly.
If you need full frame rate, you need to work in YUV - JPEG has a lot of encoding overhead and generally won't run faster than 2-10fps, while YUV will run at 30fps at least at preview resolutions.
I solved this problem by using the luminance (Y) values only, the format for which doesn't vary between devices. For the purposes of motion detection, a black and white image is fine. This also gets around the problem on API Level 21 where some of the U and V data is missing when using the ImageReader.
Related
I'm trying to do real time hand detection app . In android devices not all devices support RGB output from camera devices. Mediapipe seems to work on RGB frames by default. As Mediapipe might be trained on RGB images so I just can't give YUV frames as input without conversion.
Conversion using loops is too costly , Using library like opencv is overkill just for coversion (as it increases app size), Neon implementation is hard for me as beginner programmer .
Is there is any MediaPipe way to convert to RGB format?
(Answers on android way are also welcome)
(I'm using camera2 api , I can't use cameraX as my use case doesn't support it)
Starting from version 1.1.0-alpha08 you can safely request RGBA format.
Documentation states that:
Add setOutputImageFormat API for image analysis config. User can select ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888 or ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888. By default, ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888 will be selected.
Say that i set up 3 output surfaces.
SurfaceView with 480p
MediaCodec's InputSurface with 1080p
ImageReader with YUV format with 720p
How did android managed to generate different resolution with requested data format?
How camera sensor driver involved in this?
Or the camera sensor can only output one certain resolution & format, then android camera framework did the rest of the job.
What technique the camera framework been used regarding to scaling and format translating.
Any Hardware acceleration tech involved?
--
I just learned that if the underlying was camera 1,
it use opengles to scale and render to multi surfaces.
https://cs.android.com/android/platform/superproject/+/master:frameworks/base/core/java/android/hardware/camera2/legacy/RequestThreadManager.java
Or implemented by OEM's Camera HAL?
In general, the ISP (image signal processor) that receives the raw camera data has several scaler units, which can crop and scale each frame to multiple different sizes.
So on Android, the generation of multiple outputs is handled by the camera HAL of the device, typically heavily leveraging the ISP scaling capability. Most high-end devices run the image sensor at full resolution (in photo mode at least) and hardware downscale for preview or other outputs.
The legacy code you point to was for implementing the new camera2 API on top of the deprecated camera1 API, which required doing scaling in the Android framework, where the GPU was leveraged for this since there's no generic access to ISP scaling units outside the camera HAL. That code isn't used for modern devices that have an up-to-date camera HAL.
I’m encoding a set of JPEGs into mp4 by using the MediaCodec API. The photos could have any resolution, but I adjust all photos to be multiple of 16 to ensure they have a compatible size with MediaCodec, and make sure they are within the supported sizes returned by the codec video capabilities.
I’ve found out that in some old devices using the OMX.qcom.video.encoder.avc codec, some resolutions produce garbled videos, as seen in the next samples with different aspect ratios. The problem does not happen when using standard aspect ratios such as 16:9, 4:3, etc, only when using custom ones.
Original
Result
Original
Result
Investigating the issue, I discovered through another user’s question that this could be related to the fact old Qualcomm devices require the Y plane of the YUV data to be aligned at a 2K boundary. But, I’m not working with YUV data at all directly, instead I’m using an Input Surface and rendering through OpenGL.
My guess is that maybe the Codec underlaying system for the Input Surface works anyway with YUV buffers and the Qualcomm codec handles all the conversion, is just a guess. But if so, then, is there any formula I could use to adjust the resolution and align it to such boundary requirement, even if it will produce some cropping? Or if I am misled about my guess, then what could be causing such issue?
See the next accepted answer for the statement about the 2K boundary alignment.
How to get stride and Y plane alignment values for MediaCodec encoder
I'm very new to android. I'm trying to use the new Android Camera2 api to build a real time image processing application. My application requires to maintain a good FPS rate as well. Following some examples i managed to do the image processing inside the onImageAvailable(ImageReader reader) method available with ImageReader class. However by doing so, i can only manage to get a frame rate around 5-7 FPS.
I've seen that it is advised to use RenderScript for YUV processing with Android camera2 api. Will using RenderScript gain me higher FPS rates?
If so please can someone guide me on how to implement that, as i'm new to android i'm having a hard time grasping concepts of Allocation and RenderScript. Thanks in advance.
I don't know what type of image processing you want to perform. But in case that you are interested only in the intensity of the image (i.e. grayvalue information) you don't need any conversion of the YUV data array (e.g. into jpeg). For an image consisting of n pixels the intensity information is given by the first n bytes of the YUV data array. So, just cut those bytes out of the YUV data array:
byte[] intensity = new byte[width*height];
intensity = Arrays.copyOfRange(data, 0, width*height);
In theory, you can get the available fps ranges with this call:
characteristics.get(CameraCharacteristics.CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES);
and set the desired fps range here:
mPreviewRequestBuilder.set(CaptureRequest.CONTROL_AE_TARGET_FPS_RANGE, bestFPSRange);
So in principle, you should choose a range with the same lower and upper bound, and that should keep your frame rate constant.
HOWEVER, on devices with a LEGACY profile, none of the devices I have tested have been able to achieve 30fps at 1080p (S5, Z3 Compact, Huawei Mate S, and HTC One M9). The only way I was able to achieve that was by using a device (LG G4) that turned out to have a FULL profile.
Renderscript will not buy you anything here if you are going to use it inside the onImageAvailable callback. It appears that getting the image at that point is the bottleneck on LEGACY devices since the new camera2 API simply wraps the old one, and is presumably creating so much overhead that the callback does not occur at 30fps anymore. So if Renderscript is to work, you would need to need to create a Surface and find another way of grabbing the frames off of it.
Here is the kicker though... if you move back the deprecated API, I would almost guarantee 30fps at whatever resolution you want. At least that is what I found on all of the devices I tested....
I'm grabbing images from an Android camera using onPreviewFrame(). I only really need the grayscale portion of the image, which I'm passing as a byte array to some native code.
On the newer OS versions that I'm supporting, both YV12 and NV21 preview formats are supported, with NV21 as the default.
Is one of these likely to be faster (i.e., matching the hardware format and requiring no extra processing by the OS) on most devices? Is it something entirely dependent on the device manufacturer?
The manufacturer might matter depending on camera type and hardware. The NV21 format is android default. As far as speed, Im doubting you'll notice a difference regardless of format. Personally i would use NV21 YCrCb. Here is a link on different formats - Types of Android Image Formatting
I believe any processing will be orders of magnitude longer than hardware conversion of frame formats. If it isn't, it will be completely device-dependent.