Develop Reference

SurfaceTexture in Android plugin doesn't work in Unity

I can't get the texture tied to a SurfaceTexture to display in Unity.
Update 4: Based on the pipeline in update 1 (surface->external texture via surface texture -> fbo -> texture 2d) I know the SurfaceTexture isn't properly converting its surface to a texture. I can get correctly drawn pictures from its surface via pixelcopy and I can confirm my FBO drawing to texture2d pipeline works with some test colors. So the question is, why can't the SurfaceTexture convert its surface to a texture?
I generate a Texture in Java and pass its pointer back to Unity:
public void initGLTexture()
{
Log.d("Unity", "initGLTexture");
int textures[] = new int[1];
GLES20.glGenTextures(1, textures, 0);
mTextureId = textures[0];
GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mTextureId);
GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}
I create a SurfaceTexture from the id (in Java):
mSurfaceTexture = new SurfaceTexture(mTextureId);
mSurfaceTexture.setDefaultBufferSize(512, 512);
I use a third-party library, GeckoView, to render onto the Surface of the SurfaceTexture. I call the following method from Unity's OnRenderObject() to keep all GL rendering on the same thread:
mSurfaceTexture.updateTexImage();
I know the above code allows proper drawing onto the surface.
I call the following in Unity to load the texture:
_imageTexture2D = Texture2D.CreateExternalTexture(
512,512,TextureFormat.RGBA32,false,true,(IntPtr) mTextureId);
_rawImage.texture = _imageTexture2D;
Why does the RawImage with the texture applied show only this sprite-looking thing, which should be a webpage?
Update 1: So I've been working on the hypothesis of: use Gecko to draw to the Surface, and use a SurfaceTexture to render this surface to a GL_TEXTURE_EXTERNAL_OES. Since I can't display this on Unity (not sure why) I am drawing this texture to a frame buffer and copying the pixels in the framebuffer to a GL_TEXTURE_2D. I am getting a web page in the texture_2d (in the emulator with an imageview and glReadPixels). However, when I import the work into Unity to test if the pipeline is okay thus far I just get a black screen. I CAN get images of the surface via the PixelCopy api.
Here is my FBO overview code - my rendering code comes from grafika's texture2D program:
// bind display buffer
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
GlUtil.checkGlError("glbindframebuffer");
// unbind external texture to make sure it's fresh
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GlUtil.checkGlError("glunbindexternaltex");
// bind source texture (done in drawFrame as well )
GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mOffscreenTextureId);
GlUtil.checkGlError("glBindFramebuffer");
// draw to frame buffer
GLES20.glClearColor(0.0f, 0.0f, 0.0f, 1.0f); // again, only really need to
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT); // clear pixels outside rect
mFullScreen.drawFrame(mOffscreenTextureId, mIdentityMatrix);
// unbind source texture
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
GlUtil.checkGlError("glBindTexture2d");
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GlUtil.checkGlError("glunbindexternaltex");
// make sure we're still bound to fbo
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
GlUtil.checkGlError("glBindTexture2d");
// copy pixels from frame buffer to display texture
GLES20.glCopyTexImage2D(GLES20.GL_TEXTURE_2D,0,GLES20.GL_RGBA,0,0,512,512,0);
// read pixels from the display buffer to imageview for debugging
BitmapDisplay.mBitmap = SavePixels(0,0,512,512);
// unbind texture
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
Here's my player settings > other:
Update 2: Possible pipeline to try: call the draw function of the external texture to FBO (attached to Unity's texture_2d) in C++ via this interface.
Update 3: Calling the Java functions from native code that are responsible for drawing the texture from the SurfaceTexture to the FBO to Unity's texture via the GL.IssuePluginEvent produce a black texture as in the first update. It will show images in the emulator but not in Unity.

I had to do a similar task a couple of months ago and found out that the correct pipeline is creating a texture in Unity, obtaining a native pointer in C and finally updating it in the Java layer.
Please take a look at this sample project, it should give you a different perspective.
https://github.com/robsondepaula/unity-android-native-camera
Regards

OpenGL ES 2.0 FrameBuffer gives back a blackscreen

Alright I made an application that uses a FBO to render to a texture and later uses this texture to display it on a quad.
That's the code I am using to initialize the FBO and the texture:
IntBuffer intBuffer = IntBuffer.allocate(1);
GLES20.glGenFramebuffers(1, intBuffer);
fbo = intBuffer.get(0);
GLES20.glGenTextures(1, intBuffer);
tex = intBuffer.get(0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, tex);
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGBA, width, height, 0, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, null);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_NEAREST);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_NEAREST);
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, fbo);
GLES20.glFramebufferTexture2D(GLES20.GL_FRAMEBUFFER, GLES20.GL_COLOR_ATTACHMENT0, GLES20.GL_TEXTURE_2D, tex, 0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, 0);
(1. code in onDrawFrame())
The problem is that I don't know where to put the code, so at first I put the code in the onDrawFrame() method. When I do that the texturing works fine and I get the result that I want. But since I am creating a new FBO and texture every frame, after some time The application starts to lag and at the end the emulator crashes.
(2. code in onSurfaceChanged())
After noticing that behaviour I decided to put the code in the onSurfaceChanged() method. This time the application didn't crash but the entire texture is black. I tried to clear the texture or draw a white quad over it but nothing worked. And when I try to get an error from GLES20.glCheckFramebufferStatus(GLES20.GL_FRAMEBUFFER) or GLES20.glGetError() it says that everything is fine.
I also tried to put the code in the onDrawFrame() method surrounded with a if-clause so that it's only executed once. The result is also the black texture.
(3. code in onDrawFrame() and onSurfaceChanged())
The last thing I tried was to put the code in both methods. I wanted to reduce the code in the onDrawFrame() method until I find the codeline that makes the texture work fine. But when I started the application the quad flickered, changing from the black texture to the right texture.
There is no problem with the texture displaying since it works with the one method. I also tried to add a depth-buffer to the code but the result is the same. So since I don't need it in my application I commented it out.
Is there anyone who can say what's going on? If someone has a application that works but doesn't know what's the problem with mine it would be nice if that person could upload the project because I didn't find a tutorial that has a android project up for download.

Alright thanks to Rabbid76 for giving me a checklist to find my problem. I tried to use the fbo as a texture. This is the right binding: GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, tex). In the tutorial I have read the FBO was also bound. I still don't know why it works when I initialize the FBO in the onDrawFrame() method and use it then as texture.

Why POT textures work slower than non-pot?

I use following function to load textures
public static int loadTexture(Bitmap bmp)
{
final int[] textureHandle = new int[1];
GLES20.glGenTextures(1, textureHandle, 0);
if (textureHandle[0] != 0)
{
// Bind to the texture in OpenGL
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureHandle[0]);
// Set filtering
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
//GLES20.glGenerateMipmap(textureHandle[0]);
//adapt texture to POT
int adaptedWidth= (int) Math.pow(2,Math.ceil(Math.log(bmp.getWidth())/Math.log(2d)));
int adaptedHeight= (int) Math.pow(2,Math.ceil(Math.log(bmp.getHeight())/Math.log(2d)));
Log.d("texture",adaptedWidth+","+adaptedHeight);
Bitmap tmp = Bitmap.createScaledBitmap(bmp, adaptedWidth, adaptedHeight, false);
Log.d("asize",tmp.getWidth()+","+tmp.getHeight());
// Load the bitmap into the bound texture.
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, tmp, 0);
//GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bmp, 0);
tmp.recycle();
// Recycle the bitmap, since its data has been loaded into OpenGL.
//bmp.recycle();
}
if (textureHandle[0] == 0)
{
throw new RuntimeException("Error loading texture.");
}
return textureHandle[0];
}
I got 14-17 fps with this code.Hovever if I load my bitmap(which is non POT) directly without adaptation to POT.FPS jumps to 28-30.I thought POT textures should work faster then non-POT.Is there explanation for this?
UPD:Rendering code:
#Override
public void onDrawFrame(GL10 gl) {
//curScale=modelMatrix[SCALE_X];
TimeMeasurer.reset();
long curTS= SystemClock.uptimeMillis();
long frameRenderTime=curTS-ts;
//Log.d("renderer","FPS:"+1000.0/frameRenderTime);
Log.d("renderer","frame render time:"+frameRenderTime);
ts=curTS;
GLES20.glClear(GLES20.GL_DEPTH_BUFFER_BIT | GLES20.GL_COLOR_BUFFER_BIT);
if (piecesMesh!=null) {
Matrix.setIdentityM(MVPMatrix,0);
Matrix.multiplyMM(MVPMatrix,0,projMatrix,0,modelMatrix,0);
drawPassivePieces();
drawActivePieces();
if (helper!=null) {
drawHelper();
}
}
TimeMeasurer.measure("onDrawFrame execution time:");
}
private void drawPassivePieces() {
//shadows
shadowProgram.useProgram();
shadowProgram.setUniforms(MVPMatrix,textureMaskId);
shadowMesh.bindPieceData(shadowProgram,false);
shadowMesh.drawPieces(false);
shadowMesh.disableAttributes(shadowProgram);
//pieces
piecesProgram.useProgram();
piecesProgram.setUniforms(MVPMatrix, textureImageId, textureMaskId);
piecesMesh.bindPieceData(piecesProgram,false);
piecesMesh.drawPieces(false);
piecesMesh.disableAttributes(piecesProgram);
}
private void drawActivePieces() {
//shadows
shadowProgram.useProgram();
shadowProgram.setUniforms(MVPMatrix,textureMaskId);
shadowMesh.bindPieceData(shadowProgram,true);
shadowMesh.drawPieces(true);
shadowMesh.disableAttributes(shadowProgram);
//pieces
piecesProgram.useProgram();
piecesProgram.setUniforms(MVPMatrix, textureImageId, textureMaskId);
piecesMesh.bindPieceData(piecesProgram,true);
piecesMesh.drawPieces(true);
piecesMesh.disableAttributes(piecesProgram);
}
public void drawHelper() {
helperProgram.useProgram();
helper.bindData(helperProgram);
helper.draw();
helper.disableAttributes(helperProgram);
}

Without a detailed performance analysis, it's not really possible to do more than speculate.
One likely cause is that your rendering is limited by memory bandwidth of texture sampling. If you make the texture larger, the total amount of memory accessed is larger, causing the slowdown.
Or, very related to the above, your cache hit rate for texture sampling drops if the sampled texels are spread out farther in memory, which happens when you upscale the texture. Lower cache hit rate means slower performance.
You really shouldn't artificially make the texture larger than necessary, unless it's needed due to the limited NPOT support in ES 2.0, and the hardware you use does not advertise the OES_texture_npot extension. I doubt that anybody has made hardware that prefers POT textures in a long time.

There are big advantages to using POT textures. In OpenGLES 2.0 they allow you to use mipmaps and useful texture addressing modes like repeat. You also can utilize memory more efficiently because lots of implementations allocate memory as if your texture is POT anyway.
However, in this case where you just take a non-POT texture and scale it up, I'd expect performance to be slightly worse as a result. You're missing out on the big potential win because you're not using mipmaps. By using a larger texture you're just asking more of the GPU's texture cache because the useful parts of the image are now more spread out in memory than they were previously.
Another way to look at it is that in the absence of mipmapping, big textures are going to perform worse than little textures, and your rescaling process is just making your texture bigger.
I'm surprised the difference is so noticeable though - are you sure that your rescaling codepath isn't doing anything unexpected like resizing too large, or picking a different texture format or filter mode?

Common reasons for OpenGL Textures not being rendered

I just heard from a user who says that my (Android OpenGL ES 2.0) app (a game) won't run on his HTC 1X+ handset. All he gets is the music and the banner ad at the top of the screen and nothing else.
Unfortunately I don't have an HTC 1X+ to test this on.
Some notes:
Although my textures are not power of 2, I'm only using GLES20.GL_CLAMP_TO_EDGE
From what I've read, the HTC 1X+ has a max Texture Size of 2048 x 2048 and it gets it's resources from the XHDPI folder (annoyingly), even so, I have only 1 texure that exceeds that size, all other objects displayed on my app's opening page use textures much smaller than this max amount, so something should be displayed.
I'm not using texture compression of any kind
My app runs quite happily on the 15 (aprox) other devices I, and others have tested it on - just the 1x (so far) is giving problems.
Can anyone point out some common issues with OpenGL ES 2.0 that could be causing these textures not to be rendered? Are there any quirks with certain Android versions or devices?
I haven't yet posted any code simply because the app works on most devices, and I'm not sure which parts of the code would be helpful, but if any code is required, please just ask.
Edit - including texture loading code
public static int LoadTexture(GLSurfaceView view, Bitmap imgTex){
//Array for texture
int textures[] = new int[1];
try {
//texture name
GLES20.glGenTextures(1, textures, 0);
//Bind textures
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textures[0]);
//Set parameters for texture
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
//Apply texture
GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, imgTex, 0);
//clamp texture
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T,GLES20.GL_CLAMP_TO_EDGE);
} catch (Exception e){
}
//Increase texture count
textureCount++;
//Return texture
return textures[0];
}

Have you checked the sampler properties?
The settings look correct, but why not also specify the GL_TEXTURE_WRAP_S setting? Use GLES20.glTexParameteri for integer values.
What type of bitmap are you using?
Try to force the internal format to GL_RGBA, GL_BGR, GL_RGB
Do you properly unbind or bind to a different texture?
Other texture settings may be causing havoc in other parts of the code..
Do you specify the correct texture unit in your shader?
Print out the shader's sampler attribute position so you know this is correct, and make sure to bind the texture to it explicitly during rendering.

Loading Textures fast into OpenGL 2.0

I am building a simple live wallpaper for Android. I am uploading the required texture into OpenGL ES 2.0 using the below code. I have loaded all my images into a single file of size 2048x2048. This below code takes about 900 to 1200 ms to load the texture. Is this a normal time or am I doing something wrong to make it slow?
I also try to clear the list of textures in Opengl every time the onSurfaceCreated is called in my renderer. Is this right to be done, or is there a way to simple check if the previously loaded texture is already in memory and if so avoid clearing and reloading? Please let me know your comments on this. Thank you.
Also on screen orientation change the OnSurfaceCreated is called. So the texture upload happens again. This is not a good idea. What is the work around?
public int addTexture(Bitmap texture) {
int bitmapFormat = texture.getConfig() == Config.ARGB_8888 ? GLES20.GL_RGBA : GLES20.GL_RGB;
int[] textures = new int[1];
GLES20.glGenTextures(1, textures, 0);
int textureId = textures[0];
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureId);
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bitmapFormat, texture, 0);
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR_MIPMAP_LINEAR);
GLES20.glTexParameterf(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_REPEAT);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_REPEAT);
GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D);
return textureId;
}

A few ways you can improve performance.
Do not load the texture every time onSurfaceChanged is called. Initialize your textureId to -1 (in the constructor/surfaceCreated of your renderer) and check at the beginning of onSurfaceChanged if you have a different Id. When you call glGenTextures, you will get a positive number.
Do you need the Mipmaps? That might be the key point of your method here. Try without the line GLES20.glGenerateMipMap(GLES20.GL_TEXTURE_2D);
2048x2048 is huge. Especially for textures. Do you really need that much detail? Maybe 1024x1024 is enough.
Avoid RGB_888, use RGB_565 instead: you'll get almost the same visual quality for half the size.