Develop Reference

treating image from glReadPixels with OpenCV and return it as a texture

I am trying to start with some basic operations with OpenCV and GLES20 on Android using C++.
I use CameraGLSurfaceView and its callback onCameraTexture(...) which calls I pass into my native library.
Calls are flowing well, I can read frame buffer to vector and pass it to texture without changing and it works as expected.
But when I try to work with pixels I get image broken.
My C++ code:
cv::Mat in(w,h,CV_8UC4);
cv::Mat out(w,h,CV_8UC4);
glReadPixels(0, 0, w, h, GL_RGBA, GL_UNSIGNED_BYTE, in.data);
// following operations break image >>
cv::cvtColor(in, out, CV_RGBA2BGRA);
cv::flip(out, in, 0);
cv::cvtColor(in, out, CV_BGRA2RGBA);
// << prev operations break image
glBindTexture(GL_TEXTURE_2D, (GLuint) tex2);
glTexImage2D(GL_TEXTURE_2D,
0,
GL_RGBA,
w,
h,
0,
GL_RGBA,
GL_UNSIGNED_BYTE,
out.ptr());
glBindTexture(GL_TEXTURE_2D, 0);
in.release();
out.release();
Without signed operations picture goes to texture and is displayed well.
I understand that my mistake is in converting formats between OpenGL and OpenCV.
How to convert formats properly?

It's my mistake with sizes of Mat:
cv::Mat in(w,h,CV_8UC4) should be cv::Mat in(h,w,CV_8UC4)

Creating OpenCL memory object from OpenGL ES mipmaps

My android app passes in an OpenGL texture2D to my OpenCL kernel, however when I use clCreateFromGLTexture2D() with mipmap level 1, I get CL_INVALID_MIPLEVEL.
I am creating my OpenGL texture and generating mipmaps like this:
GLES20.glGenTextures ( 2, targetTex, 0 );
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, targetTex[0]);
GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D,GLES20.GL_TEXTURE_MIN_FILTER,GLES20.GL_LINEAR);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D,GLES20.GL_TEXTURE_MAG_FILTER,GLES20.GL_LINEAR);
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGBA, image_width, image_height, 0, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, null);
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 1, GLES20.GL_RGBA, image_width, image_height, 0, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, null);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);
The texture is then rendered to by binding it with a FBO:
targetFramebuffer = IntBuffer.allocate(1);
GLES20.glGenFramebuffers(1, targetFramebuffer);
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, targetFramebuffer.get(0));
GLES20.glFramebufferTexture2D(GLES20.GL_FRAMEBUFFER, GLES20.GL_COLOR_ATTACHMENT0, GLES20.GL_TEXTURE_2D, targetTex[0], 0);
GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, 0);
I then create a cl memory object like so:
mem_images[0] = clCreateFromGLTexture2D(m_clContext, CL_MEM_READ_ONLY, GL_TEXTURE_2D, 1, in_tex, &err);
The memory object is created successfully when the miplevel is set to 0 and all the kernels work fine. It's only when the miplevel is set to that the above function call gives me CL_INVALID_MIPLEVEL error.
OpenCL specification docs (http://www.khronos.org/registry/cl/sdk/1.1/docs/man/xhtml/clCreateFromGLTexture2D.html) say that a possible reason could be that the OpenGL implementation does not support creating from non-zero mipmap levels. However I don't know if this is definitely the case and am not sure how to find out.
EDIT:
After Andon's reply I changed my OpenGL texture generation to following:
GLES20.glGenTextures ( 2, targetTex, 0 );
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, targetTex[0]);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D,GLES20.GL_TEXTURE_MIN_FILTER,GLES20.GL_LINEAR_MIPMAP_NEAREST);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D,GLES20.GL_TEXTURE_MAG_FILTER,GLES20.GL_LINEAR);
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGBA, image_width, image_height, 0, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, null);
GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);
and after drawing into the FBO which is bound with the texture, this is how I generate mipmaps:
GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, targetTex[0]);
GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);
GLES20.glFinish();
I still get the same error when trying to create a CL memory object from GL texture with mipmap level not equal to 0.

There is something very unusual about the dimensions of your second-level mipmap in this code. The dimensions of mipmap LOD N+1 should be LOD N{w,h}/2. You can have LODs with the same resolution in GL, but those LODs will not be mipmaps. To make your 2D texture mipmap complete, you need log2 (max (width,height))-1 LODs in addition to the base image 0, and each should be 1/4 the resolution (1/2 each dimension) of the last. I have to agree with CL on this one, LOD 1 is not really a mipmap.
Moreover, you are calling glGenerateMipmap (...) on a newly created texture that has no data store. That is a meaningless thing to do, glGenerateMipmap (...) will build your mip-chain for you starting with the base LOD image; at the time you called this, there was no base LOD.
Your call to GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D); really ought to be moved to a point after you draw into your FBO. Done properly, you do not have to manually allocate storage for lower detail mipmap LODs. This function will take care of that for you. Thus, you should actually remove the code that allocates LOD 1 in the first place.

Black artifacts with open gl es 2.0 texture rendering on Certain devices

While rendering a texture on some devices (only galaxy s3 mini confirmed) i got dark area flickering on the texture as described in this thread:
Black Artifacts on Android in OpenGL ES 2
I'm not allowed to comment this thread (not enough credit) but I would like clarification from the author who solved this issue:
Could you explain a little more how you use glTexImage2D() and glTexSubImage2D() to solve this?
In code I got these lines to load the bitmaps:
(As you can see I'm using texImage2D to load the bitmap, the android documentation about gltexImage2D only provides attribute types but no explaination)
...
final BitmapFactory.Options options = new BitmapFactory.Options();
options.inScaled = false;
final Bitmap bitmap = BitmapFactory.decodeResource(
context.getResources(), resourceId, options);
if (bitmap == null) {
if (LoggerConfig.ON) {
Log.w(TAG, "Resource ID " + resourceId + " could not be decoded.");
}
glDeleteTextures(1, textureObjectIds, 0);
return 0;
}
glBindTexture(GL_TEXTURE_2D, textureObjectIds[0]);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR_MIPMAP_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
texImage2D(GL_TEXTURE_2D, 0, bitmap, 0);
glGenerateMipmap(GL_TEXTURE_2D);
...
edit:
tried to implement the solution according to the link in top but no luck, same flickering effect,
new code to load bitmap:
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(bitmap.getWidth() * bitmap.getHeight() * 4);
byteBuffer.order(ByteOrder.BIG_ENDIAN);
IntBuffer ib = byteBuffer.asIntBuffer();
int[] pixels = new int[bitmap.getWidth() * bitmap.getHeight()];
bitmap.getPixels(pixels, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());
for(int i=0; i<pixels.length; i++){
ib.put(pixels[i] << 8 | pixels[i] >>> 24);
}
bitmap.recycle();
byteBuffer.position(0);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, bitmap.getWidth(), bitmap.getHeight(), 0, GL_RGBA, GL_UNSIGNED_BYTE, null);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, bitmap.getWidth(), bitmap.getHeight(), GL_RGBA, GL_UNSIGNED_BYTE, byteBuffer);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE );
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE );
Illustration of odd behavior, see the black area to the middle right in the image:
(I need 10 reputations just to post an image?!?!?)
https://dl.dropboxusercontent.com/u/61092317/blackflickering.jpg

The issue in the other question that you've referenced appears to be this: the author specified MipMaps when originally loading up his textures, and the MipMaps didn't work properly. There's a possibility that he wasn't using glTexImage2d properly.
The artifact that he got was: when the texture unit tried to move from one MipMap level to the next, there was no information there, and it rendered a blank (apparently black) texture instead. So, at least one level of his MipMap did get loaded.
I'm unsure if that author actually resolved his issue properly. OpenGL can be incredibly specific about how you load textures into it, and even the smallest error in your code can cause a problem. The problem may only occur on some platforms, so you get the impression that something is wrong with the device, but it still may be the code.
The best place to start is the Red Book: http://www.glprogramming.com/red/chapter09.html
And then: http://www.opengl.org/sdk/docs/man3/xhtml/glGenerateMipmap.xml
All of the functions that you want to know about are outlined there.
There's a few details that aren't answered in your question. Are you creating your own MipMaps by hand? Are you wanting OpenGL to generate MipMaps automatically?
I would suggest starting with a simple texture, that has no MipMaps, just one level. See if you can make that work. If that fixes the problem, then start moving towards using MipMaps.
As for the original question, there isn't exactly a difference between loading your texture data with glTexImage2D or updating texture data with glTexSubImage2D. glTexImage2D is used to specify a texture (ie: the width / height of the texture, which is different for each MipMap level), whereas glTexSubImage2D is used to update all or part of a texture that has already been specified by glTexImage2D. In the second instance, you're just updating it with new texels.

how to convert AVFrame to texture used by glTexImage2D?

as you know that AVFrame has 2 property:pFrame->data, pFrame->linesize. After i read frame from video /sdcard/test.mp4 (android platform), and convert to RGB AVFrame vice:
img_convert_ctx = sws_getContext(pCodecCtx->width, pCodecCtx->height,
pCodecCtx->pix_fmt,
target_width, target_height, PIX_FMT_RGB24, SWS_BICUBIC,
NULL, NULL, NULL);
if(img_convert_ctx == NULL) {
LOGE("could not initialize conversion context\n");
return;
}
sws_scale(img_convert_ctx, (const uint8_t* const*)pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
I get pFrameRGB after converted. I need to texture it in opengl by using glTextImage2D:
// Create The Texture
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 256, 256, 0, GL_RGB, GL_UNSIGNED_BYTE, data);
// i don't know data in here is pFrameRGB->data or not,if not, how to convert it to approriate format for glTextImage so i can display video in opengl, using AVFrame and gltextImage2D. All helps from u are very appreciated.

The actual pixel data for the pFrameRGB is in pFrameRGB->data[0]... In your glTexImage2D call use GL_RGBA instead of GL_RGB. Thats what I did to get it working anyway. Assuming you have your texture and vertex coordinates created correctly then that should get it working. I can post some code examples if that doesn't help.

Two things: (1) is target_width and target_height powers of 2? If not, you'll get a blank screen. You need to scale the picture to a power of 2. (2) where it says "data" in glTexImage, you need to use pFrameRGB->data[0] as Kieran pointed out.
One more thing, you don't need to use RGBA (unless it's faster. I haven't tested it out). If you do, you change the ffmpeg target format to PIX_FMT_RGBA and then change both instances of GL_RGB to GL_RGBA.
Other than that, I don't see anything wrong with your code. I have almost the same exact code and it runs just fine.

Android opengl texture problem - display has messed up with lot of colors

I am trying to load a texture on to my android app display, where I am using code from this github.
I get my pixels messed up completely on the screen, And I have no idea, what's going on. The only thing I change in that code is I have memcpy, which copied uint8_t buffer into s_pixels instead of render_pixels in glbuffer.c file. My frame pixels are in rgb565 format.
Is it somekind of configuration problem or any problem with the way I copy pixels?
EDIT
Below is the code :
pictureQ is as below
pictureQ {
uint8_t *data;
int size;
}
memcpy(s_pixels,&(pictureQ[qFirst].data[0]) , 307200);
//render_pixels(s_pixels);
glClear(GL_COLOR_BUFFER_BIT);
// glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, 480, 320, GL_RGB, GL_UNSIGNED_SHORT_5_6_5, &(pictureRGBQ[qFirst].data[0]));
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 480, 320, 0, GL_RGB, GL_UNSIGNED_SHORT_5_6_5, s_pixels);
check_gl_error("glTexSubImage2D");
glDrawTexiOES(0, 0, 0, s_w, s_h);
check_gl_error("glDrawTexiOES");
memset(s_pixels, 0, 307200);

Ok, It was my mistake, I was passing the data for the pixels instead of the data. Thanks for your response Reuben Scatton.