Im Making an android Game with OpenGl ES, I Want to capture what is rendered onto screen By FloatBuffer and save it for later use, For Example If this is the OutPut:
I want this for Result (as PNG Image) :
How can I do this?
What is on screen won't be a floating point buffer - it's typically RGBA8 unorm 32-bit per pixel.
Capture via glReadPixels to fetch the raw RGBA data - you'll have to supply the raw to PNG save functionality, that's not part of OpenGL ES itself.
Note that this is a relatively expensive operation, especially at high screen resolutions, so don't expect to do this at interactive frame rates.
Related
I'm trying to understand graphics memory usage/flow in Android and specifically with respect to encoding frames from the camera using MediaCodec. In order to do that I'm having to understand a bunch of graphics, OpenGL, and Android terminology/concepts that are unclear to me. I've read the Android graphics architecture material, a bunch of SO questions, and a bunch of source but I'm still confused primarily because it seems that terms have different meanings in different contexts.
I've looked at CameraToMpegTest from fadden's site here. My specific question is how MediaCodec::createInputSurface() works in conjunction with Camera::setPreviewTexture(). It seems that an OpenGL texture is created and then this is used to create an Android SurfaceTexture which can then be passed to setPreviewTexture(). My specific questions:
What does calling setPreviewTexture() actually do in terms of what memory buffer the frames go to from the camera?
From my understanding an OpenGL texture is a chunk of memory that is accessible by the GPU. On Android this has to be allocated using gralloc with the correct usage flags. The Android description of SurfaceTexture mentions that it allows you to "stream images to a given OpenGL texture": https://developer.android.com/reference/android/graphics/SurfaceTexture.html#SurfaceTexture(int). What does a SurfaceTexture do on top of an OpenGL texture?
MediaCodec::createInputSurface() returns an Android Surface. As I understand it an Android Surface represents the producer side of a buffer queue so it may be multiple buffers. The API reference mentions that "the Surface must be rendered with a hardware-accelerated API, such as OpenGL ES". How do the frames captured by the camera get from the SurfaceTexture to this Surface that is input to the encoder? I see CameraToMpegTest creates an EGLSurface using this Surface somehow but not knowing much about EGL I don't get this part.
Can someone clarify the usage of "render"? I see things such as "render to a surface", "render to the screen" among other usages that seem to maybe mean different things.
Edit: Follow-up to mstorsjo's responses:
I dug into the code for SurfaceTexture and CameraClient::setPreviewTarget() in CameraService some more to try and understand the inner workings of Camera::setPreviewTexture() better and have some more questions. To my original question of understanding the memory allocation it seems like SurfaceTexture creates a BufferQueue and CameraService passes the associated IGraphicBufferProducer to the platform camera HAL implementation. The camera HAL can then set the gralloc usage flags appropriately (e.g. GRALLOC_USAGE_SW_READ_RARELY | GRALLOC_USAGE_SW_WRITE_NEVER | GRALLOC_USAGE_HW_TEXTURE) and also dequeue buffers from this BufferQueue. So the buffers that the camera captures frames into are gralloc allocated buffers with some special usage flags like GRALLOC_USAGE_HW_TEXTURE. I work on ARM platforms with unified memory architectures so the GPU and CPU can access the same memory so what kind of impact would the GRALLOC_USAGE_HW_TEXTURE flag have on how the buffer is allocated?
The OpenGL (ES) part of SurfaceTexture seems to mainly be implemented as part of GLConsumer and the magic seems to be in updateTexImage(). Are there additional buffers being allocated for the OpenGL (ES) texture or is the same gralloc buffer that was filled by the camera able to be used? Is there some memory copying that has to happen here to get the camera pixel data from the gralloc buffer into the OpenGL (ES) texture? I guess I don't understand what calling updateTexImage() does.
It means that the camera provides the output frames via an opaque handle instead of in a user-provided buffer within the application's address space (if using setPreviewCallback or setPreviewCallbackWithBuffer). This opaque handle, the texture, can be used within OpenGL drawing.
Almost. In this case, the OpenGL texture is not a physical chunk of memory, but a handle to a variable chunk of memory within an EGL context. In this case, the sample code itself doesn't actually allocate or size the texture, it only creates a "name"/handle for a texture using glGenTextures - it's basically just an integer. Within normal OpenGL (ES), you'd use OpenGL functions to allocate the actual storage for the texture and fill it with content. In this setup, SurfaceTexture provides an Android level API/abstraction to populate the texture with data (i.e. allocate storage for it with the right flags, provide it with a size and content) - allowing you to pass the SurfaceTexture to other classes that can fill it with data (either Camera that takes a SurfaceTexture directly, or wrap in the Surface class to be able to use it in other contexts). This allows filling the OpenGL texture with content efficiently, without having to pass a buffer of raw data to your application's process and having your app upload it to OpenGL.
(Answering points 3 and 4 in reverse order.) OpenGL (ES) is a generic API for drawing. In the normal/original setup, consider a game, you'd have a number of textures for different parts of the game content (backgrounds, props, actors, etc), and then with OpenGL APIs draw this to the screen. The textures could either be more or less just copied as such to the screen, or be wrapped around a 3D object built out of triangles. This is the process called "rendering", taking the input textures and set of triangles and drawing it. In the simplest cases, you would render content straight to the screen. The GPU usually can do the same rendering into any other output buffer as well. In games, it is common to render some scene into a texture, and use that prerendered texture as part of the final render which actually ends up displayed on the screen.
An EGL context is created for passing the output from the camera into the encoder input. An EGL context is basically a context for doing OpenGL rendering. The target for the rendering is the Surface from the encoder. That is, whatever graphics is drawn using OpenGL ends up in the encoder input buffer instead of on the screen. Now the scene that is drawn using OpenGL could be any sequence of OpenGL function calls, rendering a game scene into the encoder. (This is what the Android Breakout game recorder example does.) Within the context, an texture handle is created. Instead of filling the texture with content by loading a picure from disk, as in normal game graphics rendering, this is made into a SurfaceTexture, to allow Camera to fill it with the camera picture. The SurfaceTexture class provides a callback, giving a signal when the Camera has updated the content. When this callback is received, the EGL context is activated and one frame is rendered into the EGL context output target (which is the encoder input). The rendering itself doesn't do anything fancy, but more or else copies the input texture as-is straight into the output.
This might all sound quite roundabout, but it does give a few benefits:
The actual raw bits of the camera frames never need to be handled directly within the application code (and potentially never within the application's process and address space at all). For low resolutions, this isn't much of an issue, but the setPreviewCallback API is a bottleneck when it comes to higher resolutions.
You can do color adjustments and anything else you can do within OpenGL, almost for free with GPU acceleration.
I am doing an Android project about dealing with video frame, I need to handle every frame before display it. The process includes scaling up frames from 1920x1080 to 2560x1440 resolution, color space conversion and some necessary image processing based on RGB, and all these works should be finished within 33ms~40ms.
I have optimized the yuv->rgb and other processing with arm neon, they worked well. But I have to scale up frame firstly from 1080p to 2k resolution, it's the bottleneck of performance now.
My question is how to efficiently scale up image from 1080p to 2k resolution within 20ms, I don't have much experience about scaling algorithm, so any suggestions are helpful.
Could I use arm neon to optimize the existing algorithm?
The hardware environment:
CPU: Samsung Exynos 5420
Memory: 3GB
Display: 2560X1600 px
Update:
I will describe my decoding process, I use MediaCodec to decode the normal video(H.264) to YUV(NV12), the default decoder is hardware, it's very fast. Then I use arm neon to convert NV12 to RGBW, and then send RGBW frame to surfaceflinger to display. I just use normal SurfaceView rahter than GLSurfaceView.
The bottleneck is how to scale up YUV from 1080p to 2K fast.
I find that examples work well, so allow me to lead with this example program that uses OpenGL shaders to convert from YUV -> RGB: http://www.fourcc.org/source/YUV420P-OpenGL-GLSLang.c
What I envision for your program is:
Hardware video decodes H.264 stream -> YUV array
Upload that YUV array as a texture to OpenGL; actually, you will upload 3 different textures-- Y, U, and V
Run a fragment shader that converts those Y, U, and V textures into an RGB(W) image; this will produce a new texture in video memory
Run a new fragment shader against the texture generated in previous step in order to scale the image
There might be a bit of a learning curve involved here, but I think it's workable, given your problem description. Take it one step at a time: get the OpenGL framework in place, try uploading just the Y texture and writing a naive fragment shader that just emits a grayscale pixel based on the Y sample, then move onto correctly converting the image, then get a really naive upsampler working, then put a more sophisticated upsampler into service.
I'd also recommend opengl es too, mainly because of the project I'm currently working on, also playing video. For me, the display is 1920 x 1080, so the texture I'm using is 2048 x 1024. I get approx 35 fps on a quad core arm7.
Use a GLSurfaceView and your own custom renderer. If you're using ffmpeg then once you've decoded your video frames, use sws_scale to scale your frame and then just upload it into the opengl texture. The larger your texture/display, the less fps you will get because it a lot of time taken uploading large images to the gpu every frame.
Depending on your needs for decoding your video input is what you will have to research. For me, I had to compile ffmpeg for android and start from there.
my apologies for putting this in an answer. i dont have enough points to make a comment.
I'd like to add that you might run into OGL texture limitations. I have tried to use OGL for the opposite problem; scaling down from the camera in real time. the problem is that the max OGL texture is 2048x2048. Not sure if this is true for all devices. this limit was true on newer kit like N72013 and LG2. in the end, i had to write in in the NDK without OGL by optimising the hell out of it by hand.
good luck, though.
The example DecodeEditEncodeTest.java in bigflake.com, demonstrates an example of simple editing (swap the color channels using OpenGL FRAGMENT_SHADER).
Here, I want to do some complicated image processing (such as adding something in it) of each frame.
In this way, does it mean I cannot use surface. Instead, I need to use buffer?
But from EncodeDecodeTest.java, it says:
(1) Buffer-to-buffer. Buffers are software-generated YUV frames in ByteBuffer objects, and decoded to the same. This is the slowest (and least portable) approach, but it allows the application to examine and modify the YUV data.
(2) Buffer-to-surface. Encoding is again done from software-generated YUV data in ByteBuffers, but this time decoding is done to a Surface. Output is checked with OpenGL ES, using glReadPixels().
(3) Surface-to-surface. Frames are generated with OpenGL ES onto an input Surface, and decoded onto a Surface. This is the fastest approach, but may involve conversions between YUV and RGB.
If I use Buffer-to-buffer, from what the above says, it is the slowest and least portable. How slow would it be?
Or I use surface-to-surface, and read pixels out from the surface.
Which way is more feasible?
Any example available?
In OpenGL ES 1.1, I would like to take multiple texture Ids and combine them into a single textureId. Then I would be able to use this resulting texture multiple times in the future. My texture sources could be transparent PNGs that I want to stack together. This would be a huge optimization since I wouldn't have to render multiple textures every frame.
I have seen examples like the wiki Texture_Combiners, but it doesn't seem like the results are reusable.
Also, if there is a way to mask an image with another into a reusable texture, that would be extremely helpful too.
What you want to do is render to texture. If you're writing for iOS you're guaranteed that the OES framebuffer extension will be available, so you can use that. If you're writing for Android or another platform then the extension may be available but isn't guaranteed. If it isn't available you can fall back on glCopyTexImage2D.
So in the first case you'd create a frame buffer which has a texture as its colour buffer. Render to that then switch to another frame buffer and you can henceforth draw from the texture.
In the second you'd draw into whatever frame buffer you have, then use glCopyTexImage2D to copy from the current colour buffer into a texture. This will be a little slower because it's a copy, but it'll still probably be a lot faster than reading back the rendered content and then uploading it yourself.
ES 2.0 makes the functions contained in the framebuffer extension mandatory, so ES 2.0 capable GPUs are very likely to support the extension.
I would like to know if there is any kind of limitation on the texture size that can be used in any Android Opengl Es 2.0 projects. I understand that having a huge texture of size 4096x4096 is a bit meaning less as it is rendered on a small screen. But What if the requirement is to switch between many textures at run time? And If I want to have a texture atlas to do a quick single upload instead of multiple smaller texture upload. Please let me know your ideas in this regards.
Also I am sure there has to be a limitation on the size of image that can be processed by a device, as the memory on the device is limited. But I would like to know if it is resolution based or is it size based. I mean if a device has a limitation of 1024x1024 image size can it handle a compressed texture of size 2048x2048 that would be of same size approx as uncompressed 1024x1024.
Also please let me know on an general basis usually how much the limitation on texture size or resolution normal devices running android 2.2 and above would be.
Also please let me know if there are any best practices when handling high resolution images in opengles 2.0 to get best performance in both load time and also run time.
There is a hardware limitation on the texture sizes. To manually look them up, you can go to a site such as glbenchmark.com (Here displaying details about google galaxy nexus).
To automatically find the maximum size from your code, you can use something like:
int[] max = new int[1];
gl.glGetIntegerv(GL10.GL_MAX_TEXTURE_SIZE, max, 0); //put the maximum texture size in the array.
(For GL10, but the same method exists for GLES20)
When it comes to the processing or editing of an image you usually use an instance of Bitmap when working in android. This holds the uncompressed values of your image and is thus resolution dependant. However, it is recommended that you use compressed textures for your openGL applications as this improves the memory-use efficiency (note that you cannot modify these compressed textures).
From the previous link:
Texture compression can significantly increase the performance of your
OpenGL application by reducing memory requirements and making more
efficient use of memory bandwidth. The Android framework provides
support for the ETC1 compression format as a standard feature [...]
You should take a look at this document which contains many good practices and hints about texture loading and usage. The author explicitly writes:
Best practice: Use ETC for texture compression.
Best practice: Make sure your geometry and texture resolutions are
appropriate for the size they're displayed at. Don't use a 1k x 1k
texture for something that's at most 500 pixels wide on screen. The
same for geometry.