I am trying to read pixels/data from an OpenGL texture which is bound to GL_TEXTURE_EXTERNAL_OES.
The reason for binding the texture to that target is because in order get live camera feed on android a SurfaceTexture needs to be created from an OpenGL texture which is bound to GL_TEXTURE_EXTERNAL_OES.
Since android uses OpenGL ES I can't use glGetTexImage() to read the image data.
Therefore I am binding the target to an FBO and then reading it using readPixels(). This is my code:
GLuint framebuffer;
glGenFramebuffers(1, &framebuffer);
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
//Attach 2D texture to this FBO
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_EXTERNAL_OES, cameraTexture, 0);
status("glFramebufferTexture2D() returned error %d", glGetError());
However I am getting error 1282 (GL_INVALID_OPERATION) for some reason.
I think this might be the problem:
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
GL_TEXTURE_EXTERNAL_OES, cameraTexture, 0);
You should not attach the cameraTexture to the framebuffer, instead you should generate a new texture in the format of GL_TEXTURE_2D
glGenTextures(1, mTextureHandle, 0);
glBindTexture(GL_TEXTURE_2D, mTextureHandle[0]);
...
cameraTexture is the one you get from SurfaceTexture, and it is the source used for rendering. This new texture is the one you should render to (which could be used later in the rendering pipeline). Then do this:
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
GL_TEXTURE_2D, mTextureHandle[0], 0);
cameraTexture is drawn to the framebuffer's attached texture, using a simple shader program to draw, bind the cameraTexture when the shader program is in use:
glBindTexture(GL_TEXTURE_EXTERNAL_OES, cameraTexture);
The GL_TEXTURE_EXTERNAL_OES texture target is usually in a YUV color space and glReadPixels() requires the target to be RGB. It probably does not do the color space conversion automatically. However, you could do the conversion in your own fragment shader which renders RGB into another texture and then use glReadPixels() to read that.
texture for YUV420 to RGB conversion in OpenGL ES
Related
I can't get the texture tied to a SurfaceTexture to display in Unity.
Update 4: Based on the pipeline in update 1 (surface->external texture via surface texture -> fbo -> texture 2d) I know the SurfaceTexture isn't properly converting its surface to a texture. I can get correctly drawn pictures from its surface via pixelcopy and I can confirm my FBO drawing to texture2d pipeline works with some test colors. So the question is, why can't the SurfaceTexture convert its surface to a texture?
I generate a Texture in Java and pass its pointer back to Unity:
public void initGLTexture()
{
Log.d("Unity", "initGLTexture");
int textures[] = new int[1];
GLES20.glGenTextures(1, textures, 0);
mTextureId = textures[0];
GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mTextureId);
GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}
I create a SurfaceTexture from the id (in Java):
mSurfaceTexture = new SurfaceTexture(mTextureId);
mSurfaceTexture.setDefaultBufferSize(512, 512);
I use a third-party library, GeckoView, to render onto the Surface of the SurfaceTexture. I call the following method from Unity's OnRenderObject() to keep all GL rendering on the same thread:
mSurfaceTexture.updateTexImage();
I know the above code allows proper drawing onto the surface.
I call the following in Unity to load the texture:
_imageTexture2D = Texture2D.CreateExternalTexture(
512,512,TextureFormat.RGBA32,false,true,(IntPtr) mTextureId);
_rawImage.texture = _imageTexture2D;
Why does the RawImage with the texture applied show only this sprite-looking thing, which should be a webpage?
Update 1: So I've been working on the hypothesis of: use Gecko to draw to the Surface, and use a SurfaceTexture to render this surface to a GL_TEXTURE_EXTERNAL_OES. Since I can't display this on Unity (not sure why) I am drawing this texture to a frame buffer and copying the pixels in the framebuffer to a GL_TEXTURE_2D. I am getting a web page in the texture_2d (in the emulator with an imageview and glReadPixels). However, when I import the work into Unity to test if the pipeline is okay thus far I just get a black screen. I CAN get images of the surface via the PixelCopy api.
Here is my FBO overview code - my rendering code comes from grafika's texture2D program:
// bind display buffer
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
GlUtil.checkGlError("glbindframebuffer");
// unbind external texture to make sure it's fresh
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GlUtil.checkGlError("glunbindexternaltex");
// bind source texture (done in drawFrame as well )
GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mOffscreenTextureId);
GlUtil.checkGlError("glBindFramebuffer");
// draw to frame buffer
GLES20.glClearColor(0.0f, 0.0f, 0.0f, 1.0f); // again, only really need to
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT); // clear pixels outside rect
mFullScreen.drawFrame(mOffscreenTextureId, mIdentityMatrix);
// unbind source texture
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
GlUtil.checkGlError("glBindTexture2d");
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GlUtil.checkGlError("glunbindexternaltex");
// make sure we're still bound to fbo
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
GlUtil.checkGlError("glBindTexture2d");
// copy pixels from frame buffer to display texture
GLES20.glCopyTexImage2D(GLES20.GL_TEXTURE_2D,0,GLES20.GL_RGBA,0,0,512,512,0);
// read pixels from the display buffer to imageview for debugging
BitmapDisplay.mBitmap = SavePixels(0,0,512,512);
// unbind texture
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
Here's my player settings > other:
Update 2: Possible pipeline to try: call the draw function of the external texture to FBO (attached to Unity's texture_2d) in C++ via this interface.
Update 3: Calling the Java functions from native code that are responsible for drawing the texture from the SurfaceTexture to the FBO to Unity's texture via the GL.IssuePluginEvent produce a black texture as in the first update. It will show images in the emulator but not in Unity.
I had to do a similar task a couple of months ago and found out that the correct pipeline is creating a texture in Unity, obtaining a native pointer in C and finally updating it in the Java layer.
Please take a look at this sample project, it should give you a different perspective.
https://github.com/robsondepaula/unity-android-native-camera
Regards
How is it possible to have a multisampled texture as part of an FBO in OpenGL ES 3.0 (Android)?
The method glTexImage2DMultisample does not seem to exist.
I also want to call glReadPixels on this texture later on in this code,
so the multisampled texture should also be readable.
Is there some kind of extension or utility I would need to use for this?
You want glTexStorage2DMultisample. In general writing multisampled data back to memory is expensive, and needs a resolve using glBlitFramebuffer to consolidate to a single sample.
Consider using this extension to get a "free" resolve on most tile-based architectures.
https://www.khronos.org/registry/OpenGL/extensions/EXT/EXT_multisampled_render_to_texture.txt
Actually, opengl es not support for texture multisampling, glTexStorage2DMultisample not work for texture multisampling. opengl es only support renderbuffer for multisampling, in my case, I resolved multisampling by create a renderbuffer, works charm.
how I did this:
glGenFramebuffers(1, &fbo);
glBindFramebuffer(GL_FRAMEBUFFER, fbo);
glGenRenderbuffers(1, &rbo);
glBindRenderbuffer(GL_RENDERBUFFER, rbo);
glRenderbufferStorageMultisample(GL_RENDERBUFFER, 4, GL_RGBA8, width, height);
glBindRenderbuffer(GL_RENDERBUFFER, 0);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, rbo);
if (glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) {
LOGI("ERROR::FRAMEBUFFER:: Framebuffer is not complete!");
}
glBindFramebuffer(GL_FRAMEBUFFER, 0);
then render:
glBindFramebuffer(GL_READ_FRAMEBUFFER, fbo);
glBindFramebuffer(GL_DRAW_FRAMEBUFFER, 0);
glBlitFramebuffer(0, 0, mScreenWidth, mScreenHeight, 0, 0, mScreenWidth, mScreenHeight,
GL_COLOR_BUFFER_BIT, GL_NEAREST);
this works in opengl es 3.2, Android platform.
I want to create a framebuffer in OpenGL ES 2 with a color texture and depth and stencil renderbuffers. However, OpenGL ES doesn't seem to have GL_DEPTH24_STENCIL8 or GL_DEPTH_STENCIL_ATTACHMENT. Using two separate renderbuffers gives the error "Stencil and z buffer surfaces have different formats! Returning GL_FRAMEBUFFER_UNSUPPORTED!"
Is this not possible in OpenGL ES?
My FBO creation code:
private int width, height;
private int framebufferID,
colorTextureID,
depthRenderBufferID,
stencilRenderBufferID;
public FBO(int w, int h) {
width = w;
height = h;
int[] array = new int[1];
//Create the FrameBuffer and bind it
glGenFramebuffers(1, array, 0);
framebufferID = array[0];
glBindFramebuffer(GL_FRAMEBUFFER, framebufferID);
//Create the texture for color, so it can be rendered to the screen
glGenTextures(1, array, 0);
colorTextureID = array[0];
glBindTexture(GL_TEXTURE_2D, colorTextureID);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width, height, 0, GL_RGBA, GL_UNSIGNED_BYTE, (java.nio.ByteBuffer) null);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
// attach the texture to the framebuffer
glFramebufferTexture2D( GL_FRAMEBUFFER, // must be GL_FRAMEBUFFER
GL_COLOR_ATTACHMENT0, // color attachment point
GL_TEXTURE_2D, // texture type
colorTextureID, // texture ID
0); // mipmap level
glBindTexture(GL_TEXTURE_2D, 0);
// is the color texture okay? hang in there buddy
FBOUtils.checkCompleteness(framebufferID);
//Create the depth RenderBuffer
glGenRenderbuffers(1, array, 0);
depthRenderBufferID = array[0];
glBindRenderbuffer(GL_RENDERBUFFER, depthRenderBufferID);
glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT16, width, height);
//Create stencil RenderBuffer
glGenRenderbuffers(1, array, 0);
stencilRenderBufferID = array[0];
glBindRenderbuffer(GL_RENDERBUFFER, stencilRenderBufferID);
glRenderbufferStorage(GL_RENDERBUFFER, GL_STENCIL_INDEX8, width, height);
// bind renderbuffers to framebuffer object
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, depthRenderBufferID);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_STENCIL_ATTACHMENT, GL_RENDERBUFFER, stencilRenderBufferID);
// make sure nothing screwy happened
FBOUtils.checkCompleteness(framebufferID);
glBindFramebuffer(GL_FRAMEBUFFER, 0);
}
Packed depth/stencil surfaces are not a standard part of OpenGL ES 2.0, but are added via this extension:
https://www.khronos.org/registry/gles/extensions/OES/OES_packed_depth_stencil.txt
If the extension is supported on your platform (it usually is), the token names from OpenGL will generally work, but note that most have an _OES postfix because it is an OES extension, e.g. the internal format token is GL_DEPTH24_STENCIL8_OES.
The extension doesn't define a single combined attachment point such as GL_DEPTH_STENCIL_ATTACHMENT (that is added in OpenGL ES 3.0), but you can attach the same renderbuffer to one or both of the single attachment points. Note that it is not allowed to attach two different depth or stencil surfaces to the depth and stencil attachment points if you have attached a packed depth/stencil surface to the other (i.e. if you attach a packed depth/stencil to one attachment point, the other can either be attached to the same packed surface or unused).
In short, this is implementation dependent. What you try in the posted code with using a separate renderbuffer for depth and stencil is basically legal in ES 2.0. But then there's this paragraph in the spec:
[..] some implementations may not support rendering to particular combinations of internal formats. If the combination of formats of the images attached to a framebuffer object are not supported by the implementation, then the framebuffer is not complete under the clause labeled FRAMEBUFFER_UNSUPPORTED.
That's exactly the GL_FRAMEBUFFER_UNSUPPORTED error you are seeing. Your implementation apparently does not like the combination of depth and stencil buffer, and is at liberty to refuse supporting it while still being spec compliant.
There's one other aspect that makes your code device dependent. The combination of format and type you're using for your texture:
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width, height, 0,
GL_RGBA, GL_UNSIGNED_BYTE, (java.nio.ByteBuffer) null);
basically corresponds to an RGBA8 internal format (even though that naming is not used in the ES 2.0 spec). In base ES 2.0, this is not a color-renderable format. If you want something that is supported across the board, you'll have to use GL_UNSIGNED_SHORT_5_6_5, GL_UNSIGNED_SHORT_4_4_4_4, or GL_UNSIGNED_SHORT_5_5_5_1 for the type. Well, theoretically a device can refuse to support almost any format. The only strict requirement is that it supports at least one format combination.
Rendering to RGBA8 formats is available on many devices as part of the OES_rgb8_rgba8 extension.
As already pointed out in another answer, combined depth/stencil formats are not part of base ES 2.0, and only available with the OES_packed_depth_stencil extension.
In my android application I have texture with mipmap array
final int[] textureHandle = new int[1];
GLES20.glGenTextures(1, textureHandle, 0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureHandle);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, filter);
GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D);
Initial size of texture is 1024x1024. I need get textures:
512x512 (level 1)
256x256 (level 2)
128x128 (level 3)
Has OpenGL any api to access textures from mipmap array?
Full OpenGL has a glGetTexImage() call for this purpose. But it's not available in any version of OpenGL ES, up to the current ES 3.1.
Without having this call, you can set the texture as an FBO attachment, and then read the content with glReadPixels(). To read level lev:
int[] fboIds = new int[1];
GLES20.glGenFramebuffers(1, fboIds, 0);
int fboId = fboIds[0];
glBindFramebuffer(GLES20.GL_FRAMEBUFFER, fboId);
glFramebufferTexture2D(GLES20.GL_FRAMEBUFFER, GLES20.GL_COLOR_ATTACHMENT0,
GLES20.GL_TEXTURE_2D, mTextureHandle, lev);
glReadPixels(...);
glBindFramebuffer(GLES20.GL_FRAMEBUFFER, 0);
This approach is somewhat restricted because it will only work for texture formats that are color-renderable. Other formats will not be valid as FBO attachments. The only formats guaranteed to be color-renderable in ES 2.0 are RGBA4444, RGB5_A1 and RGB565. Many devices have extensions that add support for formats like RGB8 and RGBA8.
The next best approach is to render a quad with the texture, and then read the result with glReadPixels(). But you would have the same restrictions on the format of the render target. For example if your texture is RGBA8, and your device does not support rendering to RGBA8 surfaces, you would lose precision with this approach.
Anyone know how to render to specific mip-level texture?
Currently I am binding the mip-level texture by:
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
GL_TEXTURE_2D, textID, mip-level);
Then later in my code, I will do something like this:
glBindFramebuffer(GL_FRAMEBUFFER, FBO_ID);
drawArrays(...);
But I the shader is not executed!!!
If textID is anything other than 0, this should be generating a GL_INVALID_VALUE error.
GL_INVALID_VALUE is generated if level is not 0 and texture is not 0.
I suggest you take a look at glFramebufferTexture2D for OpenGL ES. It is valid to do what you want in normal OpenGL but not in OpenGL ES :-\