Is it possible to render camera in openles 2.0 with forced delay? for example delay is 5 frames? For now i'm rendering output with surfacetexture and opengles-2.0.
If you're receiving 30 frames per second, and you want to introduce a constant delay of 5 frames, you need to do something with those 5 frames.
SurfaceTexture doesn't do (much) buffering. Because the producer and consumer endpoints are in the same process, it would be very easy to cause a deadlock by overrunning the consumer. So SurfaceTexture uses a BufferQueue in "async" mode, which means it drops frames if the consumer isn't ready for the next one.
So you'll need a way to do your own buffering, which means copying the data out of the "external" texture. One way to do this is to render the textures to a series of FBOs, from which you can render later. Be sure to keep an eye on the memory usage here -- a 1920x1080 32-bit ARGB image occupies about 8MB, so keeping 5 around increases your memory footprint by 40MB.
Some loss of color fidelity may result, as a YUV-to-RGB conversion will be involved, but if your eventual target is RGB then this shouldn't matter.
So it's possible, but it's not easy. What are you trying to accomplish?
Related
I try to run a high frame rate opengl program on Android. It was no more than 60 FPS.
But it was 400+ FPS running in GLTextureView. https://gist.github.com/eleventigers/9545428
As I know, SurfaceView's efficiency is much higher than TextureView. How can I set GLSurfaceView's max fps. Or write a custom SurfaceView with OpenGL.
On Android, you cannot disable screen vertical sync so anything that you display on screen is capped to 60FPS (because screen has a 60Hz refresh rate, maybe some android phones exist with a higher rate but they would be the exception).
It may be possible to disable Vsync by using custom ROMs but you cannot force your users to use those and you would probably get screen tearing which would not look good anyway.
When you report 400 FPS with GLTextureView it seems what is measured is the time needed to render into an offscreen GL texture ... so of course you can render at a very high rate in an offscreen texture but you cannot display it at more than 60FPS on screen (meaning that the extra frames are skipped and never displayed)
I want to fix the frame rate of camera preview in Android, i.e., 20fps, or 30 fps. However, we find the frame rate is unstable.
In the android document, it is said that the frame rate is fluctuated between the minimum frame rate and the maximum one which are defined in getSupportedPreviewFpsRange.
https://developer.android.com/reference/android/hardware/Camera.Parameters.html#getSupportedPreviewFpsRange%28%29
My questions are:
1) Which factors influence the frame rate? exposure time, white balance, frame resolution, background CPU loading, and etc.?
2) Is there any method to fix the frame rate by customised above factors?
3) In my project, higher frame rate is better. If the frame rate is unstable in the end. Can I increase the minimum frame rate? or fix the minimum frame rate?
4) It seems that the video taking is somewhat different with preview model, Can I fix the frame rate or minimum frame rate of video taking in Android?
Finally, we found that IOS can fix the frame rate using videoMinFrameDuration and
videoMaxFrameDuration.
Thanks.
First of all, please note that the camera API that you ask about was deprecated more than 3 years ago. The new camera2 API provides much more control over all aspects of capture, including frame rate.
Especially, if your goal is smooth video recording. Actually, the MediaRecorder performs its job decently on older devices, but I understand that this knowledge has little practical value if for some reason you cannot use the MediaRecorder.
Usually, the list of supported FPS ranges includes fixed ranges, e.g. 30 fps, intended exactly for video recording. Note that you are expected to choose a compliant (recommended) preview (video) resolution.
Two major factors cause frame rate variations within the declared range: exposure adjustments and focus adjustments. To achieve uniform rate, you should disable autofocus. If your camera supports exposure control, you should lock it, too. Refrain from using exotic "scenes" and "effects". SCENE_MODE_BARCODE and EFFECT_MONO don't seem to cause problems with frame rate. Whitebalance is OK, too.
There exist other factors that cause frame rate fluctuations that are completely under your control.
Make sure that your camera callbacks do not interfere with, and are not delayed by the Main (UI) thread. To achieve that, you must open the camera on a secondary HandlerThread. The new camera2 API makes thread management for camera callbacks easier.
Don't use setPreviewCallback() which automatically allocates pixel buffers for each frame. This is a significant burden for garbage collector, which may lock all threads once in a while for major cleanup. Instead, use setPreviewCallbackWithBuffer() and preallocate just enough pixel buffers to keep it always busy.
Don't perform heavy calculations in the context of your onPreviewFrame() callback. Pass all work to a different thread. Do your best to release the pixel buffer as early as possible.
Even for the old camera API, if the device lists a supported FPS range of (30, 30), then you should be able to select this range and get consistent, fixed video recording.
Unfortunately, some devices disregard your frame rate request once the scene conditions get too dark, and increase exposure times past 1/30s. For many applications, this is the preferable option, but such applications should simply be selecting a wider frame rate range like (15, 30).
I have a graphics engine which for every frame writes approximately 300 bytes of data
with approximately six glTexSubImage2D calls to a texture (including writing to mipmaps).
The texture is a texture of unigned bytes using GL_RED and GL_R8 as format\internal format.
The uploading is performed via a pixel buffer, which in itself has been written to one frame ahead.
This quite small operation have a deep impact on the framerate regardless of what else gles is drawing.
Anyone have any idea of what is causing this frame rate drop, and how I can get past it?
Update:
Solidpixels answer indicates it's a synchronization issue. In this case, I would be perfectly fine even if the texture modification is not synchronized at all with the rendering. Can I just tell gles to just write the data straight into the texture memory without caring at all if just half the new data is there?
Note:
It is only the glTexSubImage2D which affects the frame rate, the glSubBufferData upload does not affect at all.
On some devices it seems to mather when I actually perform the glTexSubImage2D in respect to when I draw using the texture, not all devices though.
I'd guess on the devices which slow down you still have outstanding references to the texture in flight, so you are forcing the drivers to create a ghost allocation. In depth explanation here:
https://community.arm.com/graphics/b/blog/posts/mali-performance-6-efficiently-updating-dynamic-resources
My application needs to do some processing on live camera frames on the CPU, before rendering them on the GPU. There's also some other stuff being rendered on the GPU which is dependent on the results of the CPU processing; therefore it's important to keep everything synchronised so we don't render the frame itself on the GPU until the results of the CPU processing for that frame are also available.
The question is what's the lowest overhead approach for this on android?
The CPU processing in my case just needs a greyscale image, so a YUV format where the Y plane is packed is ideal (and tends to be a good match to the native format of the camera devices too). NV12, NV21 or fully planar YUV would all provide ideal low-overhead access to greyscale, so that would be preferred on the CPU side.
In the original camera API the setPreviewCallbackWithBuffer() was the only sensible way to get data onto the CPU for processing. This had the Y plane separate so was ideal for the CPU processing. Getting this frame available to OpenGL for rendering in a low overhead way was the more challenging aspect. In the end I wrote a NEON color conversion routine to output RGB565 and just use glTexSubImage2d to get this available on the GPU. This was first implemented in the Nexus 1 timeframe, where even a 320x240 glTexSubImage2d call took 50ms of CPU time (poor drivers trying to do texture swizzling I presume - this was significantly improved in a system update later on).
Back in the day I looked into things like eglImage extensions, but they don't seem to be available or well documented enough for user apps. I had a little look into the internal android GraphicsBuffer classes but ideally want to stay in the world of supported public APIs.
The android.hardware.camera2 API had promise with being able to attach both an ImageReader and a SurfaceTexture to a capture session. Unfortunately I can't see any way of ensuring the right sequential pipeline here - holding off calling updateTexImage() until the CPU has processed is easy enough, but if another frame has arrived during that processing then updateTexImage() will skip straight to the latest frame. It also seems with multiple outputs there will be independent copies of the frames in each of the queues that ideally I'd like to avoid.
Ideally this is what I'd like:
Camera driver fills some memory with the latest frame
CPU obtains pointer to the data in memory, can read Y data without a copy being made
CPU processes data and sets a flag in my code when frame is ready
When beginning to render a frame, check if a new frame is ready
Call some API to bind the same memory as a GL texture
When a newer frame is ready, release the buffer holding the previous frame back into the pool
I can't see a way of doing exactly that zero-copy style with public API on android, but what's the closest that it's possible to get?
One crazy thing I tried that seems to work, but is not documented: The ANativeWindow NDK API can accept data NV12 format, even though the appropriate format constant is not one of the ones in the public headers. That allows a SurfaceTexture to be filled with NV12 data by memcpy() to avoid CPU-side colour conversion and any swizzling that happens driver side in glTexImage2d. That is still an extra copy of the data though that feels like it should be unnecessary, and again as it's undocumented might not work on all devices. A supported sequential zero-copy Camera -> ImageReader -> SurfaceTexture or equivalent would be perfect.
The most efficient way to process video is to avoid the CPU altogether, but it sounds like that's not an option for you. The public APIs are generally geared toward doing everything in hardware, since that's what the framework itself needs, though there are some paths for RenderScript. (I'm assuming you've seen the Grafika filter demo that uses fragment shaders.)
Accessing the data on the CPU used to mean slow Camera APIs or working with GraphicBuffer and relatively obscure EGL functions (e.g. this question). The point of ImageReader was to provide zero-copy access to YUV data from the camera.
You can't really serialize Camera -> ImageReader -> SurfaceTexture as ImageReader doesn't have a "forward the buffer" API. Which is unfortunate, as that would make this trivial. You could try to replicate what SurfaceTexture does, using EGL functions to package the buffer as an external texture, but again you're into non-public GraphicBuffer-land, and I worry about ownership/lifetime issues of the buffer.
I'm not sure how the parallel paths help you (Camera2 -> ImageReader, Camera2 -> SurfaceTexture), as what's being sent to the SurfaceTexture wouldn't have your modifications. FWIW, it doesn't involve an extra copy -- in Lollipop or thereabouts, BufferQueue was updated to allow individual buffers to move through multiple queues.
It's entirely possible there's some fancy new APIs I haven't seen yet, but from what I know your ANativeWindow approach is probably the winner. I suspect you'd be better off with one of the Camera formats (YV12 or NV21) than NV12, but I don't know for sure.
FWIW, you will drop frames if your processing takes too long, but unless your processing is uneven (some frames take much longer than others) you'll have to drop frames no matter what. Getting into the realm of non-public APIs again, you could switch the SurfaceTexture to "synchronous" mode, but if your buffers fill up you're still dropping frames.
I'd like to display decoded video frames from MediaCodec out of order, or omit frames, or show frames multiple times.
I considered configuring MediaCodec to use a Surface, call MediaCodec.dequeueOutputBuffer() repeatedly, save the resulting buffer indices, and then later call MediaCodec.releaseOutputBuffer(desired_index, true), but there doesn't seem to be a way to increase the number of output buffers, so I might run out of output buffers if I'm dealing with a lot of frames to be rearranged.
One idea I'm considering is to use glReadPixels() to read the pixel data into a frame buffer, convert the color format appropriately, then copy it to a SurfaceView when I need the frame displayed. But this seems like a lot of copying (and color format conversion) overhead, especially when I don't inherently need to modify the pixel data.
So I'm wondering if there is a better, more performant way. Perhaps there is a way to configure a different Surface/Texture/Buffer for each decoded frame, and then a way to tell the SurfaceView to display a specific Surface/Texture/Buffer (without having to do a memory copy). It seems like there must be a way to accomplish this with OpenGL, but I'm very new to OpenGL and could use recommendations on areas to investigate. I'll even go NDK if I have to.
So far I've been reviewing the Android docs, and fadden's bigflake and Grafika. Thanks.
Saving copies of lots of frames could pose a problem when working with higher-resolution videos and higher frame counts. A 1280x720 frame, saved in RGBA, will be 1280x720x4 = 3.5MB. If you're trying to save 100 frames, that's 1/3rd of the memory on a 1GB device.
If you do want to go this approach, I think what you want to do is attach a series of textures to an FBO and render to them to store the pixels. Then you can just render from the texture when it's time to draw. Sample code for FBO rendering exists in Grafika (it's one of the approaches used in the screen recording activity).
Another approach is to seek around in the decoded stream. You need to seek to the nearest sync frame before the frame of interest (either by asking MediaExtractor to do it, or by saving off encoded data with the BufferInfo flags) and decode until you reach the target frame. How fast this is depends on how many frames you need to traverse, the resolution of the frames, and the speed of the decoder on your device. (As you might expect, stepping forward is easier than stepping backward. You may have noticed a similar phenomena in other video players you've used.)
Don't bother with glReadPixels(). Generally speaking, if decoded data is directly accessible from your app, you're going to take a speed hit (more so on some devices than others). Also, the number of buffers used by the MediaCodec decoder is somewhat device-dependent, so I wouldn't count on having more than 4 or 5.