I have a FloatBuffer as an output from the neural network, where the RGB channels are encoded with [-1 .. +1] values. I would like to render them on-screen, using GLSurfaceView. What is the best way to handle it?
I can dump the buffer into SSBO and write a compute shader, which maps it to ByteBuffer of [0 .. 255] range, then somehow bind it to regular texture. Or maybe I can set up my compute shader to output directly to some texture buffer? Or maybe I am supposed to read my SSBO directly from the fragment shader (and implement my own linear interpolation)?
So, which is the best way to render stuff via OpenGL ES? Please, help.
You can try to load it with but it depends how many update you need per seconde. That is to test with you machine.
First Bind your texture (you must create one) then when your input buffer is ready use
GLES20.glTexSubImage2D(GLES20.GL_TEXTURE_2D, 0, 0, 0, width, height, GLES30.GL_RGB, GLES20.GL_FLOAT, InputFloatBuffer);
It work well with ByteBuffer and i did not try with Float but there is no Signed_float format.
Use a kernel to change the signed float to byte.
Related
this is how i am made an array of triangle
float[] tableVerticesWithTriangle = {
// triangle 1
0f, 0f, 9f, 14f, 0f, 14f,
// triangle 2
0f, 0f, 9f, 0f, 9f, 14f
};
and this is how i have allocated the block in native environment
vertexData = ByteBuffer
.allocateDirect(
tableVerticesWithTriangle.length * BYTES_PER_FLOAT)
.order(ByteOrder.nativeOrder()).asFloatBuffer();
vertexData.put(tableVerticesWithTriangle);
The reason people use ByteBuffer.allocateDirect() is that other buffer classes, like FloatBuffer, do not have an allocateDirect() method. Only ByteBuffer can be allocated as a direct buffer. So allocating a ByteBuffer, and then using the memory as a FloatBuffer, is the only way to get a directly allocated FloatBuffer.
What is a direct buffer?
The documentation of isDirect() of the FloatBuffer class explains it like this:
Indicates whether this buffer is direct. A direct buffer will try its best to take advantage of native memory APIs and it may not stay in the Java heap, so it is not affected by garbage collection.
A float buffer is direct if it is based on a byte buffer and the byte buffer is direct.
In other (less formal) words, a native buffer is a native memory allocation that Java is not messing with.
When are direct buffers required?
Strangely enough, I have never been able to find clear documentation for this. So the following is a hypothesis that I confirmed with experiments, without finding any counter-examples so far.
Direct buffers have to be used when a buffer is passed to an OpenGL API where the memory is used by the OpenGL implementation after the call returns.
There is only one example of this I could find: client side vertex arrays (which BTW are marked as a legacy feature in ES 3.0, but still supported). This is the glVertexAttribPointer() call with the following signature, which supports vertex arrays without the use of VBOs:
glVertexAttribPointer(int indx, int size, int type, boolean normalized,
int stride, Buffer ptr)
In this case, OpenGL will pull vertex data from the buffer in later draw calls, so the buffer content has to remain accessible to OpenGL after the call returns, and will potentially be read directly by the GPU.
In all other cases (again according to my hypothesis), it is not necessary to use direct buffers. You can for example do the following:
float[] vertexData = {...};
GLES20.glBufferData(GL_ARRAY_BUFFER, vertexData.length * 4,
FloatBuffer.wrap(vertexData), GLES20.GL_STATIC_DRAW);
The glBufferData() call consumes the data during the call, and the original buffer can not be accessed by OpenGL after the call returns. Therefore, it is not necessary to use a direct buffer.
On Android using OpenGL ES 2.0 I try to perform certain performance tests using different internal texture formats.
Initially I have a lot of RGBA textures (png) which I want to load and store internally in a different format with OpenGL (for example RGB and LUMINANCE). I load my textures using glTexImage2D like this:
Bitmap bitmap = BitmapFactory.decodeResource(context.getResources(),resourceId);
...
int size = bitmap.getRowBytes() * bitmap.getHeight();
ByteBuffer b = ByteBuffer.allocate(size);
bitmap.copyPixelsToBuffer(b);
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGBA, bitmap.getWidth(),
bitmap.getHeight(), 0, GLES20.GL_RGBA,
GLES20.GL_UNSIGNED_BYTE, b);
This works fine, however if I change the first GLES20.GL_RGBA (The internalFormat parameter) to anything else (GLES20.GL_RGB or GLES20.GL_LUMINANCE) my texture appears all black. Changing the second GLES20.GL_RGBA to the same value will display something - but obviously not correctly as the original data is RGBA.
I thought maybe it has something to with the shader code - that maybe texture2D(..) returns a different value because the internal format of the texture is different. My shader code is simply:
gl_FragColor = texture2D(texture, fragment_texture_coordinate);
I tried changing this around too, but no luck yet. So I thought maybe glTex2DImage is not at all working as I think it does (I am not an expert on this area whatsoever).
What am I doing wrong?
Edit:
I overlooked this little detail on texImage2D. It appears that:
internalformat must match format. No conversion between formats is supported during texture image processing. type may be used as a hint to specify how much precision is desired, but a GL implementation may choose to store the texture array at any internal resolution it chooses.
What I gather from this, is that if you want to store your textures different from their original format you'll have to convert it yourself.
Your fragment shader must be written to agree with the format you are giving to glTexImage2D(). For GL_RGB, it should force the alpha to 1.0, like this:
vec3 Color_RGB = texture2D(sampler2d, texCoordinate);
gl_FragColor = vec4(Color_RGB, 1.0);
But, for GL_RGBA, it should look like this:
vec4 Color_RGBA = texture2D(sampler2d, texCoordinate);
gl_FragColor = Color_RGBA;
And, as has been discussed, you can only use the Android Bitmap class for textures if your PNG files have no transparency. This article explains that:
http://software.intel.com/en-us/articles/porting-opengl-games-to-android-on-intel-atom-processors-part-1
On Android, I'm trying to perform some OpenGL processing on camera frames, show those frames in the camera preview and then encode the frames in a video file. I'm trying to do this with OpenGL, using the GLSurfaceView and GLSurfaceView.Renderer and with FFMPEG for video encoding.
I've successfully processed the image frames using a shader. Now I need to encode the processed frames to video. The GLSurfaceView.Renderer provides the onDrawFrame(GL10 ..) method. It's in this method that I'm attempting to read the image frames using just glReadPixels() and then place the frames on a queue for encoding to video. On it's own, glReadPixels() is much too slow - my frame rate is in the single digits. I'm attempting to speed this up using Pixel Buffer Objects. This is not working. After plugging in the pbo, the frame rate is unchanged. This is my first time using OpenGL and I do not know where to begin looking for the problem. Am I doing this right? Can anyone give me some direction? Thanks in advance.
public class MainRenderer implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener {
.
.
public void onDrawFrame ( GL10 gl10 ) {
//Create a buffer to hold the image frame
ByteBuffer byte_buffer = ByteBuffer.allocateDirect(this.width * this.height * 4);
byte_buffer.order(ByteOrder.nativeOrder());
//Generate a pointer to the frame buffers
IntBuffer image_buffers = IntBuffer.allocate(1);
GLES20.glGenBuffers(1, image_buffers);
//Create the buffer
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, image_buffers.get(0));
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, byte_buffer.limit(), byte_buffer, GLES20.GL_STATIC_DRAW);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, image_buffers.get(0));
//Read the pixel data into the buffer
gl10.glReadPixels(0, 0, this.width, this.height, GL10.GL_RGBA, GL10.GL_UNSIGNED_BYTE, byte_buffer);
//encode the frame to video
enQueueForEncoding(byte_buffer);
//unbind the buffer
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
}
.
.
}
I have never tried something like that before (opengl+video enconding) but I can tell you that reading from device memory is SLOW. Try double buffering, this may help since the GPU can keep rendering to the second buffer while the DMA controller reads back stuff.
Load a profiler (check your devices' GPU vendor), this may give you some idea. Another thing that may help is setting internal pbuffer format to something else, try lower numbers and dropping a channel (alpha).
EDIT: If you feel like that, you can encode the video at the GPU, that's going to boost, memory and processing wise, your application.
As I remember glBufferData() is not mapping your internal buffer onto GPU memory, it just copies data from your memory into the buffer (initializes).
To get access to the memory, which is allocated by glBufferData(), you should use glMapBufferRange(). That function returns a Java Buffer object which you can read.
I'm trying to draw two objects using two different textures with one shader program in OpenGL ES 2.0 for Android. The first object should have texture0 and the second sould have texture1.
In fragment shader I have:
uniform sampler2D tex;
and in java code:
int tiu0 = 0;
int tiu1 = 1;
int texLoc = glGetUniformLocation(program, "tex");
glUseProgram(program);
// bind texture0 to texture image unit 0
glActiveTexture(GL_TEXTURE0 + tiu0);
glBindTexture(GL_TEXTURE_2D, texture0);
// bind texture1 to texture image unit 1
glActiveTexture(GL_TEXTURE0 + tiu1);
glBindTexture(GL_TEXTURE_2D, texture1);
glUniform1i(texLoc, tiu0);
// success: glGetError returns GL_NO_ERROR, glGetUniformiv returns 0 for texLoc
drawFirstObject(); // should have texture0
glUniform1i(texLoc, tiu1);
// success: glGetError returns GL_NO_ERROR, glGetUniformiv returns 1 for texLoc
drawSecondObject(); // should have texture1
Running on Samsung Galaxy Ace with Android 2.3.3 both objects have the same texture0. Similar code runs correctly in OpenGL 2.0 on my desktop computer.
If I remove drawFirstObject, the second object will have texture1.
If I remove drawSecondObject, the first object will have texture0.
If somewhere between drawFirstObject and drawSecondObject I change the program for a while:
glUseProgram(0); // can be also any valid program other than the program from the next call
glUseProgram(program);
then both objects will have texture1.
Values of uniforms different from sampler2D are always set correctly.
I know I can draw the two objects with different textures using only one texture image unit and binding appropriate texture to that texture image unit before drawing the object, but I also want to know what's going on here.
Is something wrong with my code? Is it possible in OpenGL ES 2.0 to draw the objects with different textures by only switching between texture image units as I shown in the code? If it's impossible, is that difference between OpenGL 2.0 (where it's possible) and OpenGL ES 2.0 documented anywhere? I can't find it.
After hours of further research I've found out that this problem is specific to Adreno 200 GPU that is utilized in my Samsung Galaxy Ace (GT-S5830). It seems like Adreno 200 driver assigns the texture to the sampler in the first call to a drawing function and after that it ignores any changes to the sampler value (glUniform1i(samplerLocation, textureImageUnit)) until one of the two occurs:
glUseProgram is called with a different shader program,
a different texture is bound to any texture image unit used by the shader program.
There's a thread in the forums of the manufacturer of Adreno 200 GPU describing the very same problem.
So if you call drawing functions several times with the same shader program and with different textures bound before, there are two workarounds to the described problem:
Call glUseProgram(0); glUseProgram(yourDrawingProgram); before every drawing function.
Before every drawing call, bind different texture to at least one texture image unit used by your shader program. This solution can be difficult to maintain, because if you bind the same texture that is already bound to the texture image unit, the problem will remain. So in this case the easiest solution is to simply not change sampler values and bind textures of all texture image units used by the shader program before every drawing call.
How to load a Mesh (3dmax) in Android OpenGL into a ByteBuffer ?
example:
float triangleCoords[] = {
-0.5f, -0.25f , 0, // 0,
0.5f, -0.25f , 0, // 1,
0.0f, 0.559016994f, 0 // 2,
};
ByteBuffer vbb = ByteBuffer.allocateDirect(triangleCoords.length * _4_FLOAT_LENGTH);
vbb.order(ByteOrder.nativeOrder());
triangleVB = vbb.asFloatBuffer();
triangleVB.put(triangleCoords);
triangleVB.position(0);
how to load a mesh like an array or Coords or ineed a library for this job
It basically boils down to loading the data from the file into your data structures (like the vertex array from your code snippet) and providing these to OpenGL for rendering (in your case glDrawArrays, for example). This is generally the same procedure, no matter if working with Android, Windows, Plan9, OpenGL or Direct3D. Only the implementation details matter, therefore this question is a bit broad.
You can look at this description of the 3DS file format (hinted in your question), which also has links to tutorials for loading these files. Although the 3DS format is a quite easy to read binary format, for a start you might also look into the Wavefront OBJ format, a simple ASCII file format that nearly every modeling software can export to (but this should also hold for 3DS). Loading these formats into a bunch of simple vertex arrays should boil down to some few lines of code.
These should get you started. If you encounter any specific problems implementing these mesh loading functionalities on your particular platform, feel free to ask a more specific question.