Develop Reference

SurfaceTexture in Android plugin doesn't work in Unity

I can't get the texture tied to a SurfaceTexture to display in Unity.
Update 4: Based on the pipeline in update 1 (surface->external texture via surface texture -> fbo -> texture 2d) I know the SurfaceTexture isn't properly converting its surface to a texture. I can get correctly drawn pictures from its surface via pixelcopy and I can confirm my FBO drawing to texture2d pipeline works with some test colors. So the question is, why can't the SurfaceTexture convert its surface to a texture?
I generate a Texture in Java and pass its pointer back to Unity:
public void initGLTexture()
{
Log.d("Unity", "initGLTexture");
int textures[] = new int[1];
GLES20.glGenTextures(1, textures, 0);
mTextureId = textures[0];
GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mTextureId);
GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}
I create a SurfaceTexture from the id (in Java):
mSurfaceTexture = new SurfaceTexture(mTextureId);
mSurfaceTexture.setDefaultBufferSize(512, 512);
I use a third-party library, GeckoView, to render onto the Surface of the SurfaceTexture. I call the following method from Unity's OnRenderObject() to keep all GL rendering on the same thread:
mSurfaceTexture.updateTexImage();
I know the above code allows proper drawing onto the surface.
I call the following in Unity to load the texture:
_imageTexture2D = Texture2D.CreateExternalTexture(
512,512,TextureFormat.RGBA32,false,true,(IntPtr) mTextureId);
_rawImage.texture = _imageTexture2D;
Why does the RawImage with the texture applied show only this sprite-looking thing, which should be a webpage?
Update 1: So I've been working on the hypothesis of: use Gecko to draw to the Surface, and use a SurfaceTexture to render this surface to a GL_TEXTURE_EXTERNAL_OES. Since I can't display this on Unity (not sure why) I am drawing this texture to a frame buffer and copying the pixels in the framebuffer to a GL_TEXTURE_2D. I am getting a web page in the texture_2d (in the emulator with an imageview and glReadPixels). However, when I import the work into Unity to test if the pipeline is okay thus far I just get a black screen. I CAN get images of the surface via the PixelCopy api.
Here is my FBO overview code - my rendering code comes from grafika's texture2D program:
// bind display buffer
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
GlUtil.checkGlError("glbindframebuffer");
// unbind external texture to make sure it's fresh
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GlUtil.checkGlError("glunbindexternaltex");
// bind source texture (done in drawFrame as well )
GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mOffscreenTextureId);
GlUtil.checkGlError("glBindFramebuffer");
// draw to frame buffer
GLES20.glClearColor(0.0f, 0.0f, 0.0f, 1.0f); // again, only really need to
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT); // clear pixels outside rect
mFullScreen.drawFrame(mOffscreenTextureId, mIdentityMatrix);
// unbind source texture
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
GlUtil.checkGlError("glBindTexture2d");
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GlUtil.checkGlError("glunbindexternaltex");
// make sure we're still bound to fbo
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
GlUtil.checkGlError("glBindTexture2d");
// copy pixels from frame buffer to display texture
GLES20.glCopyTexImage2D(GLES20.GL_TEXTURE_2D,0,GLES20.GL_RGBA,0,0,512,512,0);
// read pixels from the display buffer to imageview for debugging
BitmapDisplay.mBitmap = SavePixels(0,0,512,512);
// unbind texture
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
Here's my player settings > other:
Update 2: Possible pipeline to try: call the draw function of the external texture to FBO (attached to Unity's texture_2d) in C++ via this interface.
Update 3: Calling the Java functions from native code that are responsible for drawing the texture from the SurfaceTexture to the FBO to Unity's texture via the GL.IssuePluginEvent produce a black texture as in the first update. It will show images in the emulator but not in Unity.

I had to do a similar task a couple of months ago and found out that the correct pipeline is creating a texture in Unity, obtaining a native pointer in C and finally updating it in the Java layer.
Please take a look at this sample project, it should give you a different perspective.
https://github.com/robsondepaula/unity-android-native-camera
Regards

how to assign one texture to another efficiently in OpenGL ES for Android

I want to copy texture1 to texture2. The background is that I generate 15 empty texture id and bind them to GLES11Ext.GL_TEXTURE_EXTERNAL_OES, the following is my code:
int[] textures = new int[15];
GLES20.glGenTextures(15, textures, 0);
GlUtil.checkGlError("glGenTextures");
for(int i=0;i<15;i++)
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textures[i]);
then I always transfer camera preview to textures[0], I want copy the texture from textures[0] to textures[1] in order to keep the frame content of timestamp 1, and copy the texture from textures[0] to textures[2] in order to keep the frame content of timestamp 2... it looks like buffer some texture data in GPU and render some of that in the future. So I want to know is there anyway to do this? And can I just use textures[2]=textures[0] to copy texture data?

There isn't a very direct way to copy texture data in ES 2.0. The easiest way is probably using glCopyTexImage2D(). To use this, you have to create an FBO, and attach the source texture to it. Say if srcTexId is the id of the source texture, and dstTexId the id of the destination texture:
GLuint fboId = 0;
glGenFramebuffers(1, &fboId);
glBindFramebuffer(GL_FRAMEBUFFER, fboId);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, srcTexId, 0);
glBindTexture(GL_TEXTURE_2D, dstTexId);
glCopyTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, 0, 0, width, height, 0);
That being said, from your description, I don't believe that this is really what you should be doing, for the following reasons:
I don't think copying texture data as shown above will work for the external textures you are using.
Copying texture data will always be expensive, and sounds completely unnecessary to solve your problem.
It sounds like you want to keep the 15 most recent camera images. To do this, you can simply track which of your 15 textures contains the most recent image, and treat the list of the 15 texture ids as a circular buffer.
Say initially you create your texture ids:
int[] textures = new int[15];
GLES20.glGenTextures(15, textures, 0);
int newestIdx = 0;
Then every time you receive a new frame, you write it to the next entry in your list of texture ids, wrapping around at 15:
newestIdx = (newestIdx + 1) % 15;
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textures[newestIdx]);
// Capture new frame into currently bound texture.
Then, every time you want to use the ith frame, with 0 referring to the most recent, 1 to the frame before that, etc, you bind it with:
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textures[(newestIdx + i) % 15]);
So the textures never get copied. You just keep track of which texture contains which frame, and access them accordingly.

Antialiasing and texture filtering in native Android application using OpenGL ES

I am mapping a 360 degree photospheric image on a sphere with camera at centre of sphere. When we build the application using native Android SDK and OpenGL ES 2.0, we see jaggy edges like the one shown in below image. The jags are visible on the arms of sofa, edges of floor etc. when I see this view using cardboard device
On the other hand, the same image ( 4096X2048 resolution) is rendered perfectly in unity3d application for Android.
The magnification and minification filter for the texture that we are using is GL_LINEAR.
Source Code used for setting filtering paramenters for texture and generating Texture and mipmap:
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mSphereTextureIds[0]);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR_MIPMAP_LINEAR);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
// Load the bitmap into the bound texture.
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bitmap, 0);
GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D);
Code for Multisample Antialiasing ( 4X MSAA)
cardboardView.setEGLConfigChooser(new MyConfigChooser());
cardboardView.setRenderer(renderer);
I have also turned on 4x Multisample anti aliasing. But even after doing this, there are jags inside texture. I used highp precision for both float and int in both of my shaders.
class MyConfigChooser implements CardboardView.EGLConfigChooser {
#Override
public EGLConfig chooseConfig(EGL10 egl, EGLDisplay display) {
int attribs[] = {
EGL10.EGL_LEVEL, 0,
EGL10.EGL_RENDERABLE_TYPE, 4, // EGL_OPENGL_ES2_BIT
EGL10.EGL_COLOR_BUFFER_TYPE, EGL10.EGL_RGB_BUFFER,
EGL10.EGL_RED_SIZE, 8,
EGL10.EGL_GREEN_SIZE, 8,
EGL10.EGL_BLUE_SIZE, 8,
EGL10.EGL_DEPTH_SIZE, 16,
EGL10.EGL_SAMPLE_BUFFERS, 1,
EGL10.EGL_SAMPLES, 2, // This is for 4x MSAA.
EGL10.EGL_NONE
};
EGLConfig[] configs = new EGLConfig[1];
int[] configCounts = new int[1];
egl.eglChooseConfig(display, attribs, configs, 1, configCounts);
if (configCounts[0] == 0) {
// Failed! Error handling.
Log.d("test","MSAA Failed............");
return null;
} else {
return configs[0];
}
}
Is there some post processing that Unity does on the textures for filtering during magnification that gives it a better look without aliasing effect ?
Can we do anisotropic filtering to smoothen the texture on magnification without blurring on Android ? If yes can we get a clue on how to do that ?
Is 4X the maximum MSAA that is currently supported on Android ? (I am using Nexus 5).
Thanks in advance
Apurv Nigam

Can we do anisotropic filtering to smoothen the texture on
magnification without blurring on Android?
Not in general - it's "really expensive" so not much mobile hardware supports it.
If yes can we get a clue on how to do that?
There is no support for it in "official" OpenGL ES; check the vendor extensions to see if one exists. For desktop GL it is supported via the GL_EXT_texture_filter_anisotropic extension - I've not seen a mobile GPU supporting it.
Is 4X the maximum MSAA that is currently supported on Android? (I am
using Nexus 5).
Depends on vendor - some devices support 8x (Mali-T760), some older devices don't support it at all (Tegra-1/2/3) - so YMMV.

I found the answer to my problem. I figured out that the image was being scaled down by Universal Image Loader library being used by me. So the bitmap that was being given to texImage2D call was already scaled down causing a pixelated texture. So I decoded my image using BitmapFactory using isScaled=false option.

Common reasons for OpenGL Textures not being rendered

I just heard from a user who says that my (Android OpenGL ES 2.0) app (a game) won't run on his HTC 1X+ handset. All he gets is the music and the banner ad at the top of the screen and nothing else.
Unfortunately I don't have an HTC 1X+ to test this on.
Some notes:
Although my textures are not power of 2, I'm only using GLES20.GL_CLAMP_TO_EDGE
From what I've read, the HTC 1X+ has a max Texture Size of 2048 x 2048 and it gets it's resources from the XHDPI folder (annoyingly), even so, I have only 1 texure that exceeds that size, all other objects displayed on my app's opening page use textures much smaller than this max amount, so something should be displayed.
I'm not using texture compression of any kind
My app runs quite happily on the 15 (aprox) other devices I, and others have tested it on - just the 1x (so far) is giving problems.
Can anyone point out some common issues with OpenGL ES 2.0 that could be causing these textures not to be rendered? Are there any quirks with certain Android versions or devices?
I haven't yet posted any code simply because the app works on most devices, and I'm not sure which parts of the code would be helpful, but if any code is required, please just ask.
Edit - including texture loading code
public static int LoadTexture(GLSurfaceView view, Bitmap imgTex){
//Array for texture
int textures[] = new int[1];
try {
//texture name
GLES20.glGenTextures(1, textures, 0);
//Bind textures
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textures[0]);
//Set parameters for texture
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
//Apply texture
GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, imgTex, 0);
//clamp texture
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T,GLES20.GL_CLAMP_TO_EDGE);
} catch (Exception e){
}
//Increase texture count
textureCount++;
//Return texture
return textures[0];
}

Have you checked the sampler properties?
The settings look correct, but why not also specify the GL_TEXTURE_WRAP_S setting? Use GLES20.glTexParameteri for integer values.
What type of bitmap are you using?
Try to force the internal format to GL_RGBA, GL_BGR, GL_RGB
Do you properly unbind or bind to a different texture?
Other texture settings may be causing havoc in other parts of the code..
Do you specify the correct texture unit in your shader?
Print out the shader's sampler attribute position so you know this is correct, and make sure to bind the texture to it explicitly during rendering.

android read pixels from GL_TEXTURE_EXTERNAL_OES

I am trying to read pixels/data from an OpenGL texture which is bound to GL_TEXTURE_EXTERNAL_OES.
The reason for binding the texture to that target is because in order get live camera feed on android a SurfaceTexture needs to be created from an OpenGL texture which is bound to GL_TEXTURE_EXTERNAL_OES.
Since android uses OpenGL ES I can't use glGetTexImage() to read the image data.
Therefore I am binding the target to an FBO and then reading it using readPixels(). This is my code:
GLuint framebuffer;
glGenFramebuffers(1, &framebuffer);
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
//Attach 2D texture to this FBO
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_EXTERNAL_OES, cameraTexture, 0);
status("glFramebufferTexture2D() returned error %d", glGetError());
However I am getting error 1282 (GL_INVALID_OPERATION) for some reason.

I think this might be the problem:
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
GL_TEXTURE_EXTERNAL_OES, cameraTexture, 0);
You should not attach the cameraTexture to the framebuffer, instead you should generate a new texture in the format of GL_TEXTURE_2D
glGenTextures(1, mTextureHandle, 0);
glBindTexture(GL_TEXTURE_2D, mTextureHandle[0]);
...
cameraTexture is the one you get from SurfaceTexture, and it is the source used for rendering. This new texture is the one you should render to (which could be used later in the rendering pipeline). Then do this:
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0,
GL_TEXTURE_2D, mTextureHandle[0], 0);
cameraTexture is drawn to the framebuffer's attached texture, using a simple shader program to draw, bind the cameraTexture when the shader program is in use:
glBindTexture(GL_TEXTURE_EXTERNAL_OES, cameraTexture);

The GL_TEXTURE_EXTERNAL_OES texture target is usually in a YUV color space and glReadPixels() requires the target to be RGB. It probably does not do the color space conversion automatically. However, you could do the conversion in your own fragment shader which renders RGB into another texture and then use glReadPixels() to read that.
texture for YUV420 to RGB conversion in OpenGL ES