Develop Reference

why do we allocate blocks in bytes rather than floats in OpenGL (ES) android , though we work with float most of the times

this is how i am made an array of triangle
float[] tableVerticesWithTriangle = {
// triangle 1
0f, 0f, 9f, 14f, 0f, 14f,
// triangle 2
0f, 0f, 9f, 0f, 9f, 14f
};
and this is how i have allocated the block in native environment
vertexData = ByteBuffer
.allocateDirect(
tableVerticesWithTriangle.length * BYTES_PER_FLOAT)
.order(ByteOrder.nativeOrder()).asFloatBuffer();
vertexData.put(tableVerticesWithTriangle);

The reason people use ByteBuffer.allocateDirect() is that other buffer classes, like FloatBuffer, do not have an allocateDirect() method. Only ByteBuffer can be allocated as a direct buffer. So allocating a ByteBuffer, and then using the memory as a FloatBuffer, is the only way to get a directly allocated FloatBuffer.
What is a direct buffer?
The documentation of isDirect() of the FloatBuffer class explains it like this:
Indicates whether this buffer is direct. A direct buffer will try its best to take advantage of native memory APIs and it may not stay in the Java heap, so it is not affected by garbage collection.
A float buffer is direct if it is based on a byte buffer and the byte buffer is direct.
In other (less formal) words, a native buffer is a native memory allocation that Java is not messing with.
When are direct buffers required?
Strangely enough, I have never been able to find clear documentation for this. So the following is a hypothesis that I confirmed with experiments, without finding any counter-examples so far.
Direct buffers have to be used when a buffer is passed to an OpenGL API where the memory is used by the OpenGL implementation after the call returns.
There is only one example of this I could find: client side vertex arrays (which BTW are marked as a legacy feature in ES 3.0, but still supported). This is the glVertexAttribPointer() call with the following signature, which supports vertex arrays without the use of VBOs:
glVertexAttribPointer(int indx, int size, int type, boolean normalized,
int stride, Buffer ptr)
In this case, OpenGL will pull vertex data from the buffer in later draw calls, so the buffer content has to remain accessible to OpenGL after the call returns, and will potentially be read directly by the GPU.
In all other cases (again according to my hypothesis), it is not necessary to use direct buffers. You can for example do the following:
float[] vertexData = {...};
GLES20.glBufferData(GL_ARRAY_BUFFER, vertexData.length * 4,
FloatBuffer.wrap(vertexData), GLES20.GL_STATIC_DRAW);
The glBufferData() call consumes the data during the call, and the original buffer can not be accessed by OpenGL after the call returns. Therefore, it is not necessary to use a direct buffer.

SurfaceTexture/Surface mapping with ANativeWindow

Given a GraphicBufferProducer I create a Surface and then retrieve ANativeWindow. Using ANativeWindow_lock I get a pointer to the buffer. Using the buffer, I do a memcpy into the buffer. The problem is that whatever I draw on this buffer is restricted to less than 25% of the screen. Keep in mind that the dimensions of buffer.width and buffer.height are very close to the resolution of the screen itself.
My question is, why does the buffer only cover a small portion of the screen? And how do I make sure it covers most if not all of the screen? For reference here's the code:
ANativeWindow_Buffer buffer;
// window is created from a "new Surface(sp<IGraphicBufferProducer>)"
if (ANativeWindow_lock(window, &buffer, NULL) == 0) {
// For testing purposes just put grey in the buffer
memcpy(buffer.bits, 0x99, buffer.width * buffer.height);
ANativeWindow_unlockAndPost(window);
}

I have a guess, although I've never used an ANativeWindow_Buffer. memcpy copies a certain number of bytes. How many bits per pixel is your image? If the value is greater than 8, you aren't transfering the full buffer. Since its probably 4 bytes per pixel (AARRGGBB), you probably need to multiply that by 4.

Using Pixel Buffer Objects (PBO's) on Android

On Android, I'm trying to perform some OpenGL processing on camera frames, show those frames in the camera preview and then encode the frames in a video file. I'm trying to do this with OpenGL, using the GLSurfaceView and GLSurfaceView.Renderer and with FFMPEG for video encoding.
I've successfully processed the image frames using a shader. Now I need to encode the processed frames to video. The GLSurfaceView.Renderer provides the onDrawFrame(GL10 ..) method. It's in this method that I'm attempting to read the image frames using just glReadPixels() and then place the frames on a queue for encoding to video. On it's own, glReadPixels() is much too slow - my frame rate is in the single digits. I'm attempting to speed this up using Pixel Buffer Objects. This is not working. After plugging in the pbo, the frame rate is unchanged. This is my first time using OpenGL and I do not know where to begin looking for the problem. Am I doing this right? Can anyone give me some direction? Thanks in advance.
public class MainRenderer implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener {
.
.
public void onDrawFrame ( GL10 gl10 ) {
//Create a buffer to hold the image frame
ByteBuffer byte_buffer = ByteBuffer.allocateDirect(this.width * this.height * 4);
byte_buffer.order(ByteOrder.nativeOrder());
//Generate a pointer to the frame buffers
IntBuffer image_buffers = IntBuffer.allocate(1);
GLES20.glGenBuffers(1, image_buffers);
//Create the buffer
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, image_buffers.get(0));
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, byte_buffer.limit(), byte_buffer, GLES20.GL_STATIC_DRAW);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, image_buffers.get(0));
//Read the pixel data into the buffer
gl10.glReadPixels(0, 0, this.width, this.height, GL10.GL_RGBA, GL10.GL_UNSIGNED_BYTE, byte_buffer);
//encode the frame to video
enQueueForEncoding(byte_buffer);
//unbind the buffer
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
}
.
.
}

I have never tried something like that before (opengl+video enconding) but I can tell you that reading from device memory is SLOW. Try double buffering, this may help since the GPU can keep rendering to the second buffer while the DMA controller reads back stuff.
Load a profiler (check your devices' GPU vendor), this may give you some idea. Another thing that may help is setting internal pbuffer format to something else, try lower numbers and dropping a channel (alpha).
EDIT: If you feel like that, you can encode the video at the GPU, that's going to boost, memory and processing wise, your application.

As I remember glBufferData() is not mapping your internal buffer onto GPU memory, it just copies data from your memory into the buffer (initializes).
To get access to the memory, which is allocated by glBufferData(), you should use glMapBufferRange(). That function returns a Java Buffer object which you can read.

Fast dynamic vertices in OpenGL ES 2.0 on Android

I am trying to batch draw a bunch of lines on Android using OpenGL ES 2.0 and I need to know the best way to do this.
Right now I made a class called LineEngine which builds up a FloatBuffer of all the vertices to draw and then draws all the lines at once. The problem is that apparently FloatBuffer.put() is very slow and is gobbling up CPU time like crazy.
Here is my class
public class LineEngine {
private static final float[] IDENTIY = new float[16];
private FloatBuffer mLinePoints;
private FloatBuffer mLineColors;
private int mCount;
public LineEngine(int maxLines) {
Matrix.setIdentityM(IDENTIY, 0);
ByteBuffer byteBuf = ByteBuffer.allocateDirect(maxLines * 2 * 4 * 4);
byteBuf.order(ByteOrder.nativeOrder());
mLinePoints = byteBuf.asFloatBuffer();
byteBuf = ByteBuffer.allocateDirect(maxLines * 2 * 4 * 4);
byteBuf.order(ByteOrder.nativeOrder());
mLineColors = byteBuf.asFloatBuffer();
reset();
}
public void addLine(float[] position, float[] color){
mLinePoints.put(position, 0, 8); //These lines
mLineColors.put(color, 0, 4); // are taking
mLineColors.put(color, 0, 4); // the longest!
mCount++;
}
public void reset(){
mLinePoints.position(0);
mLineColors.position(0);
mCount = 0;
}
public void draw(){
mLinePoints.position(0);
mLineColors.position(0);
GraphicsEngine.setMMatrix(IDENTIY);
GraphicsEngine.setColors(mLineColors);
GraphicsEngine.setVertices4d(mLinePoints);
GraphicsEngine.disableTexture();
GLES20.glDrawArrays(GLES20.GL_LINES, 0, mCount * 2);
GraphicsEngine.disableColors();
reset();
}
}
Is there a better way to batch all these lines together?

What you are trying to do is called SpriteBatching. If you want your program to be robust in openGL es you have to check the following( a list of what makes your program slow ) :
-Changing to many opengl ES states each frame.
-Enable//Dissable (textures etc) again and again. Once you enable something you dont have to do it again it will be applied automaticly its frame.
-Using to many assets instead of spriteSheets. Yes thats make a huge performance impact. For example if you have 10 images you have to load for each image a different texture and that is SLOW. A better way to do this is to create a spriteSheet.(You can check google for free spriteSheet creators).
-Your images are high quality. Yes! check your resolution and file extension. Prefer .png files.
-Care for the api 8 float buffer bug(you have to use int buffer instead to fix the bug).
-Using java containers. This is the biggest pitfall, if you use string concatenation(that's not a container but it returns a new string each time) or a List or any other container that RETURNS A NEW CLASS INSTANCE chances are your program will be slow due to garbage collection. For input handling i would suggest you to search a teqnique called the Pool Class. Its use is to recycle objects instead of creating new ones. Garbage collector is the big enemy, try to make him happy and avoid any programming technique that might call him.
-Never load things on the fly, instead make a loader class and load all the necessary assets in the begining of the app.
If you do implement this list then your chances are your program will be robust.
One last adition. What is a sprite batcher? A spriteBatcher is a class that uses a single texture to render multiple objects. It will create automaticly vertices, indices, colors, u - v coords and it will render them as a batch. This pattern will save GPU power as well as cpu power. From your posted code i can't be sure what causes the cpu to slow down but from my experience is due to one(or more) things of the list i previously mention. Check the list, follow it, search google for spriteBatching and i am sure your program will run fast. Hope i helped!
Edit: I think i found what causes your program to slow down, you dont flip the buffer dude! You just reset the position. You just add add add more objects and you cause buffer overload. In the reset method just flip the buffer. mLineColors.flip mLinePaints.flip will do the job. Make sure you call them each frame if you send new verices each frame.

FloatBuffer.put(float[]) with a relatively large float array should be considerably faster. The single put(float) calls have plenty of overhead.

Just go for a very simple native function which will be called from your class. You can put float[] to OpenGL directly, no need to kill CPU time with a silly buffer interface.

Spritesheet programmatically cutting: best practices

I have a big spritesheet (3808x1632) composed by 42 frames.
I would present an animation with these frames and I use a thread to load a bitmap array with all the frames, with a splash screen waiting for its end.
I'm not using a SurfaceView (and a draw function of a canvas), I just load frame by frame in an ImageView in my main layout.
My approach is similar to Loading a large number of images from a spritesheet
The completion actually takes almost 15 seconds, not acceptable.
I use this kind of function:
for (int i=0; i<TotalFramesTeapotBG; i++) {
xStartTeapotBG = (i % framesInRowsTeapotBG) * frameWidthTeapotBG;
yStartTeapotBG = (i / framesInRowsTeapotBG) * frameHeightTeapotBG;
mVectorTeapotBG.add(Bitmap.createBitmap(framesBitmapTeapotBG, xStartTeapotBG, yStartTeapotBG, frameWidthTeapotBG, frameHeightTeapotBG));
}
framesBitmapTeapotBG is the big spritesheet.
Looking more deeply, I've read in the logcat that the createBitmap function takes a lot of time, maybe because the spritesheet is too big.
I found somewhere that I could make a window on the big spritesheet, using the rect function and canvas, creating small bitmaps to be loaded in the array, but it was not really clear. I'm talking about that post: cut the portion of bitmap
My question is: how can I speed the spritesheet cut?
Edit:
I'm trying to use this approach but I cannot see the final animation:
for (int i=0; i<TotalFramesTeapotBG; i++) {
xStartTeapotBG = (i % framesInRowsTeapotBG) * frameWidthTeapotBG;
yStartTeapotBG = (i / framesInRowsTeapotBG) * frameHeightTeapotBG;
Bitmap bmFrame = Bitmap.createBitmap(frameWidthTeapotBG, frameHeightTeapotBG, Bitmap.Config.ARGB_8888);
Canvas c = new Canvas(bmFrame);
Rect src = new Rect(xStartTeapotBG, yStartTeapotBG, frameWidthTeapotBG, frameHeightTeapotBG);
Rect dst = new Rect(0, 0, frameWidthTeapotBG, frameHeightTeapotBG);
c.drawBitmap(framesBitmapTeapotBG, src, dst, null);
mVectorTeapotBG.add(bmFrame);
}
Probably, the Bitmap bmFrame is not correctly managed.

The short answer is better memory management.
The sprite sheet you're loading is huge, and then you're making a copy of it into a bunch of little bitmaps. Supposing the sprite sheet can't be any smaller, I'd suggest taking one of two approaches:
Use individual bitmaps. This will reduce the memory copies as well as the number of times Dalvik will have to grow the heap. However, these benefits may be limited by the need to load many images off the filesystem instead of just one. This would be the case in a normal computer, but Android systems may get different results since they're run off flash memory.
Blit directly from your sprite sheet. When drawing, just draw straight from sprite sheet using something like Canvas.drawBitmap(Bitmap bitmap, Rect src, Rect dst, Paint paint). This will reduce your file loads to one large allocation that probably only needs to happen once in the lifetime of your activity.
I think the second option is probably the better of the two since it will be easier on the memory system and be less work for the GC.

Thanks to stevehb for the suggestion, I finally got it:
for (int i = 0; i < TotalFramesTeapotBG; i++) {
xStartTeapotBG = (i % framesInRowsTeapotBG) * frameWidthTeapotBG;
yStartTeapotBG = (i / framesInRowsTeapotBG) * frameHeightTeapotBG;
Bitmap bmFrame = Bitmap.createBitmap(frameWidthTeapotBG, frameHeightTeapotBG, Bitmap.Config.ARGB_8888);
Canvas c = new Canvas(bmFrame);
Rect src = new Rect(xStartTeapotBG, yStartTeapotBG, xStartTeapotBG+frameWidthTeapotBG, yStartTeapotBG+frameHeightTeapotBG);
Rect dst = new Rect(0, 0, frameWidthTeapotBG, frameHeightTeapotBG);
c.drawBitmap(framesBitmapTeapotBG, src, dst, null);
mVectorTeapotBG.add(bmFrame);
}
The computation time falls incredibly! :)

Use a LevelListDrawable. Cut the sprites into individual frames and drop them in your drawable resource directory. Either programmatically or through an xml based level-list drawable create your drawable. Then use ImageView.setImageLevel() to pick your frame.

I use a method of slicing based on rows and columns. However your sprite sheet is rather huge. You have to think you are putting that whole sheet into memory. 3808x1632x4 is the size of the image in memory.
Anyway, what I do is I take an image (lets say a 128x128) and then tell it there are 4 columns and 2 rows in the Sprite(bitmap, 4, 2) constructor. Then you can slice and dice based on that. bitmap.getWidth() / 4 etc... pretty simple stuff. However if you want to do some real stuff use OpenGL and use textures.
Oh I also forgot to mention there are some onDraw stuff that needs to happen. Basically you keep an index counter and slice a rectangle from the bitmap and draw that from a source rectangle to a destination rectangle on the canvas.