Develop Reference

SurfaceTexture in Android plugin doesn't work in Unity

I can't get the texture tied to a SurfaceTexture to display in Unity.
Update 4: Based on the pipeline in update 1 (surface->external texture via surface texture -> fbo -> texture 2d) I know the SurfaceTexture isn't properly converting its surface to a texture. I can get correctly drawn pictures from its surface via pixelcopy and I can confirm my FBO drawing to texture2d pipeline works with some test colors. So the question is, why can't the SurfaceTexture convert its surface to a texture?
I generate a Texture in Java and pass its pointer back to Unity:
public void initGLTexture()
{
Log.d("Unity", "initGLTexture");
int textures[] = new int[1];
GLES20.glGenTextures(1, textures, 0);
mTextureId = textures[0];
GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mTextureId);
GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameterf(GL_TEXTURE_EXTERNAL_OES, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
GLES20.glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}
I create a SurfaceTexture from the id (in Java):
mSurfaceTexture = new SurfaceTexture(mTextureId);
mSurfaceTexture.setDefaultBufferSize(512, 512);
I use a third-party library, GeckoView, to render onto the Surface of the SurfaceTexture. I call the following method from Unity's OnRenderObject() to keep all GL rendering on the same thread:
mSurfaceTexture.updateTexImage();
I know the above code allows proper drawing onto the surface.
I call the following in Unity to load the texture:
_imageTexture2D = Texture2D.CreateExternalTexture(
512,512,TextureFormat.RGBA32,false,true,(IntPtr) mTextureId);
_rawImage.texture = _imageTexture2D;
Why does the RawImage with the texture applied show only this sprite-looking thing, which should be a webpage?
Update 1: So I've been working on the hypothesis of: use Gecko to draw to the Surface, and use a SurfaceTexture to render this surface to a GL_TEXTURE_EXTERNAL_OES. Since I can't display this on Unity (not sure why) I am drawing this texture to a frame buffer and copying the pixels in the framebuffer to a GL_TEXTURE_2D. I am getting a web page in the texture_2d (in the emulator with an imageview and glReadPixels). However, when I import the work into Unity to test if the pipeline is okay thus far I just get a black screen. I CAN get images of the surface via the PixelCopy api.
Here is my FBO overview code - my rendering code comes from grafika's texture2D program:
// bind display buffer
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
GlUtil.checkGlError("glbindframebuffer");
// unbind external texture to make sure it's fresh
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GlUtil.checkGlError("glunbindexternaltex");
// bind source texture (done in drawFrame as well )
GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, mOffscreenTextureId);
GlUtil.checkGlError("glBindFramebuffer");
// draw to frame buffer
GLES20.glClearColor(0.0f, 0.0f, 0.0f, 1.0f); // again, only really need to
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT); // clear pixels outside rect
mFullScreen.drawFrame(mOffscreenTextureId, mIdentityMatrix);
// unbind source texture
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
GlUtil.checkGlError("glBindTexture2d");
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GlUtil.checkGlError("glunbindexternaltex");
// make sure we're still bound to fbo
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, mFrameBufferId);
GlUtil.checkGlError("glBindTexture2d");
// copy pixels from frame buffer to display texture
GLES20.glCopyTexImage2D(GLES20.GL_TEXTURE_2D,0,GLES20.GL_RGBA,0,0,512,512,0);
// read pixels from the display buffer to imageview for debugging
BitmapDisplay.mBitmap = SavePixels(0,0,512,512);
// unbind texture
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D,0);
Here's my player settings > other:
Update 2: Possible pipeline to try: call the draw function of the external texture to FBO (attached to Unity's texture_2d) in C++ via this interface.
Update 3: Calling the Java functions from native code that are responsible for drawing the texture from the SurfaceTexture to the FBO to Unity's texture via the GL.IssuePluginEvent produce a black texture as in the first update. It will show images in the emulator but not in Unity.

I had to do a similar task a couple of months ago and found out that the correct pipeline is creating a texture in Unity, obtaining a native pointer in C and finally updating it in the Java layer.
Please take a look at this sample project, it should give you a different perspective.
https://github.com/robsondepaula/unity-android-native-camera
Regards

Is there a better way to get the Camera pixels than onPreviewFrame on Android?

onPreviewFrame only gets called when the preview frames from the camera get displayed. I'm processing the image as an open gl texture using the same technique here:
http://nhenze.net/?p=172&cpage=1#comment-8424
But it seems like a waste to render the preview to the screen just so I can draw over it with my textured image. Is there a better way to get the pixels from the camera than during the call to onPreviewFrame?

yes, you can use a SurfaceTexture. There's some mild trickiness here as it needs to be an external texture not the normal 2D texture.
This means that if you render the texture with ES2 you need some like
#extension GL_OES_EGL_image_external : require
uniform samplerExternalOES s_texture;
in the fragment shader.
Example code:
int[] textures = new int[1];
// generate one texture pointer and bind it as an external texture.
GLES20.glGenTextures(1, textures, 0);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textures[0]);
// No mip-mapping with camera source.
GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GL10.GL_TEXTURE_MIN_FILTER,
GL10.GL_LINEAR);
GLES20.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR);
// Clamp to edge is only option.
GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GL10.GL_TEXTURE_WRAP_S, GL10.GL_CLAMP_TO_EDGE);
GLES20.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
GL10.GL_TEXTURE_WRAP_T, GL10.GL_CLAMP_TO_EDGE);
int texture_id = textures[0];
SurfaceTexture mTexture = new SurfaceTexture(texture_id);
mTexture.setOnFrameAvailableListener(this);
Camera cam = Camera.open();
cam.setPreviewTexture(mTexture);

Opengl es loads texture but now everything is that texture

I am now able to load a texture to the project I have made but it gave me results that I was not expecting. Instead of loading my 91 tiles and having them have crate textures, and having a picture of a puppy be at 10x and 10 y, everything is the puppy.
This is my draw code
#Override
public void onDrawFrame(GL10 gl) {
// TODO Auto-generated method stub
gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_DEPTH_BUFFER_BIT);
gl.glLoadIdentity();
for(int i =0;i<91;i++)
{
myfloortiles[i].draw(gl);
}
gl.glLoadIdentity();
gl.glOrthof(0, width, height, 0, 0, 1);
gl.glDisable(GL10.GL_DEPTH_TEST);
// gl.glClear(GL10.GL_COLOR_BUFFER_BIT);
gl.glTranslatef(10, 10, 1);
bitmap=BitmapFactory.decodeResource(context.getResources(), R.drawable.puppy);
int textureID;
int[] temp = new int[1];
gl.glGenTextures(1, temp, 0);
textureID = temp[0];
gl.glBindTexture(GL10.GL_TEXTURE_2D, textureID);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MIN_FILTER,
GL10.GL_NEAREST);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MAG_FILTER,
GL10.GL_NEAREST); // GL_LINEAR for quality.
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_S,
GL10.GL_CLAMP_TO_EDGE);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_T,
GL10.GL_CLAMP_TO_EDGE);
GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, bitmap, 0);
bitmap.recycle();
}
Of course I have to be doing something wrong. Is it that I have to make a 3d object and then render the texture to it? Or is there another way? Also how would I have an animation if i chose the 3d object with texture path? Would I swap in new images when I want it to draw a different image for the interface? Is my place the image in a specific location logic correct? Or is that also terribly incorrect also?

Technically not a mistake, but a problem is generating texture names and creating a texture object in the drawing function. Texture creating is a one-time thing:
You generate a texture name, which is a handle by which you refer to the texture later (glGenTextures)
You bind that name (glBindTexture)
You upload the image (glTexImage and/or glTexSubImage)
For drawing the actual textured meshes you
enable texturing (*glEnable(GL_TEXTURE_…)*)
bind the texture(s) (glBindTexture)
draw the mesh(es)
In your code you're going through the whole texture creation phase for every redraw. The last created texture is also the bound texture and thus will be subsequently applied to the following drawing calls. Which are about to happen in the next call of onDrawFrame
On a side note: You clearly have not understood how the matrix stack works. You mixed calls of glOrtho, glLoadIdentity, glTranslate and so on without any sensible structure. You need to get the understanding of those straigt, too. I'd say even before you tackle texturing.

Create Bitmap from byte array using OpenGL in Android

I have a ByteArray and want to create a bitmap out of it using OpenGL. In Android, there is decodeByteArray() method which returns a Bitmap object that can be drawn on ImageView.
What is the equivalent method available in OpenGL?

There is no equivalent function in OpenGL (ES) since it is a pure API and not made for decoding byte arrays.
However, if you want to apply some texture onto your model you might use an approach like this:
gl.glGenTextures(1, textures, 0);
gl.glBindTexture(GL10.GL_TEXTURE_2D, textures[0]);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MIN_FILTER,GL10.GL_NEAREST);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MAG_FILTER,GL10.GL_LINEAR);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_S,GL10.GL_REPEAT);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_T,GL10.GL_REPEAT);
GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, texture, 0);
texture.recycle();

Android Drawing text with opengl es crashes after a few minutes

I've created an opengl surface and everything works fine, however when I try to draw text onto it using the following method:
public void loadFPSTexture(GL10 gl){
Bitmap bitmap = Bitmap.createBitmap(256, 256, Bitmap.Config.RGB_565);
bitmap.eraseColor(Color.BLACK);
Canvas canvas = new Canvas(bitmap);
Paint textPaint = new Paint();
textPaint.setTextSize(35);
textPaint.setFakeBoldText(true);
textPaint.setAntiAlias(true);
textPaint.setARGB(255, 255, 255, 255);
canvas.drawText("FPS "+reportedFramerate, 10,35, textPaint);
gl.glGenTextures(1, texturesFPS, 0);
gl.glBindTexture(GL10.GL_TEXTURE_2D, texturesFPS[0]);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MIN_FILTER, GL10.GL_LINEAR);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_MAG_FILTER, GL10.GL_LINEAR);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_S, GL10.GL_CLAMP_TO_EDGE);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_T, GL10.GL_REPEAT);
GLUtils.texImage2D(GL10.GL_TEXTURE_2D, 0, bitmap, 0);
bitmap.recycle();
}
Then used in my onDraw function with:
gl.glPushMatrix();
gl.glEnable(GL10.GL_TEXTURE_2D);
gl.glBindTexture(GL10.GL_TEXTURE_2D, texturesFPS[0]);
gl.glTranslatef(-surfaceSize.x/1.5f, surfaceSize.y/1.5f, 0.0f);
gl.glScalef(10, 10, 1.0f);
gl.glColor4f(1.0f, 1.0f, 1.0f, saturation_head);
gl.glEnableClientState(GL10.GL_VERTEX_ARRAY);
gl.glEnableClientState(GL10.GL_TEXTURE_COORD_ARRAY);
gl.glVertexPointer(3, GL10.GL_FLOAT, 0,vertexBuffer);
gl.glTexCoordPointer(2, GL10.GL_FLOAT, 0, textureFPSBuffer);
gl.glDrawElements(GL10.GL_TRIANGLES, indices.length,GL10.GL_UNSIGNED_SHORT, indexBuffer);
gl.glDisableClientState(GL10.GL_TEXTURE_COORD_ARRAY);
gl.glDisableClientState(GL10.GL_VERTEX_ARRAY);
gl.glDisable(GL10.GL_TEXTURE_2D);
gl.glPopMatrix();
I get a weird crash and the surface is unable to be drawn after this point. Logcat shows a constant stream of the following:
E/Adreno200-EGL( 2578): eglLockWindowSurface: failed to map the memory for fd=32 offs=1536000
E/SurfaceFlinger( 2578): GL error 0x0505
E/Adreno200-EGL( 2578): eglLockWindowSurface: failed to map the memory for fd=32 offs=1536000
E/Adreno200-EGL( 2578): egliSwapWindowSurface: oglSwapBuffer failed
E/SurfaceFlinger( 2578): eglSwapBuffers: EGL error 0x3003 (EGL_BAD_ALLOC)
I'm not sure why this is happening? any help would be much appreciated!

The trouble is that I was generating textures repeatedly without ever deleting them. Simply adding one line before generation is enough to prevent a memory leak (it seems there is no need to check that a texture has already been generated):
gl.glDeleteTextures(1, texturesFPS, 1);
gl.glGenTextures(1, texturesFPS, 1);
Simple as that :)

Looks like you're creating a new texture each time you call loadFPSTexture(), and never release it. After some time this will cause you to run out of memory, which could explain the EGL_BAD_ALLOC in your logs.
It would be better to create your bitmap, canvas and texturesFPS variables just once, and reuse them in the loadFPSTexture() function. In that case, you should probably use GLUtils.texSubImage2D() instead of GLUtils.texImage2D(), to upload new bitmap data to the existing texture.

I don't have comment access yet (frequent other 'help' sites, just getting up on SO), but wanted to echo (and upvoted) svdree's note, and add more details -- because other people starting with GLES will certainly hit similar issues.
Your bitmap, canvas, and texture (and paint!) should be created once, wherever you set up your gles resources to begin with. They should be deleted when you clean up resources for the app. Unless you are resizing the bitmap/texture, recreating is thrashing memory (cpu and gpu) unnecessarily.
The initial creation of the texture, you'd use the GLUtils.texImage2D function to prepare the texture (can just upload the bitmap as-is, don't care about the data). That ensures the texture is allocated by the driver, proper width/height buffer ready for later updates.
Rendering the fps might then look more like:
s_bitmap.eraseColor(Color.BLACK);
s_canvas.drawText("FPS "+reportedFramerate, 10, 35, s_textPaint);
gl.glBindTexture(GL10.GL_TEXTURE_2D, s_texturesFPS[0]);
GLUtils.texSubImage2D(GL10.GL_TEXTURE_2D, 0, 0, 0, s_bitmap);
... and that's it. An order of magnitude faster, and obviously much cleaner. :-)
You can make it even faster beyond that, for example only erase/fill the bitmap rectangle where the fps is being drawn, and then use gl.glTexSubImage2D directly to upload only those rows the text is rendered to (saving you clearing, and uploading, say 220 extra rows of data that isn't changing...).
Hope that helps!