So I'm making an Android 2.2 app that uses GLSurfaceView. My question is, since OpenGL tends to buffer commands, does that mean it requires associated memory (e.g. the bitmap in a call to glTexSubImage2D() ) to stick around until it is done? Or does it make itself a copy of any memory needed for buffered commands?
I ask as this code tends to cause a long stall and an eventual crash on hardware (HTC Desrire) but not on the emulator:
//bm is a Bitmap stored in a vector from previous commands
//pt is a Point stored at the same time
GLUtils.texSubImage2D(GL10.GL_TEXTURE_2D, 0, pt.x, pt.y, bm);
bm.recycle();
Now if I add glFlush() like so:
//bm is a Bitmap stored in a vector from previous commands
//pt is a Point stored at the same time
GLUtils.texSubImage2D(GL10.GL_TEXTURE_2D, 0, pt.x, pt.y, bm);
**AGL.glFlush();** //or glFinish
bm.recycle();
It appears to work great. Now is this an actual functionality for glFlush/glFinish, to prevent memory from being cleared out from underneath OpenGL?
Good question. With Textures, and Vertex Buffer Objects incidentally, once you have called the methods that actually load the image data for the Texture, you do not need to hold onto the buffer/array you have in client memory.
Don't mistake the rendering buffer with memory associated with Textures. The Texture graphics are saved in memory for the GPU, they are not part of the buffer that is being flushed to finally draw.
Related
I want to take screenshot of current frame in OpenGL for further processing and I'm trying to improve the performance of glReadPixels by using PBO to asynchronously read framebuffers.
I'm under the impression that glReadPixels after GL_PIXEL_PACK_BUFFER is bound to buffer should return immediately, but it actually takes similar or even more time than not using PBO.
Here are samples of my codes:
// Setup PBO
GLES30.glGenBuffers(nPbo, pboIndex, 0);
for(int i=0;i<nPbo; i++){
GLES30.glBindBuffer (GL_PIXEL_PACK_BUFFER, pboIndex[i]);
GLES30.glBufferData(GL_PIXEL_PACK_BUFFER, size, null,GL_STREAM_READ);
}
GLES30.glBindBuffer(GL_PIXEL_PACK_BUFFER, 0);
......
// For each frame, trigger async transfer of framebuffer to PBO.
// Note that I don't even map the PBO to memory yet
GLES30.glBindBuffer (GL_PIXEL_PACK_BUFFER, pboIndex[index]);
// The following is a JNI method to overload glReadPixels in GLES20.glReadPixels,
// to allow passing int offset to the last param in order to use PBO,
// and slowdown (around 500ms on my device) happens here
GLES3PBOReadPixelsFix.glReadPixelsPBO(0, 0, mWidth, mHeight, GLES30.GL_RGBA, GLES30.GL_UNSIGNED_BYTE, 0);
GLES30.glBindBuffer(GL_PIXEL_PACK_BUFFER, 0);
Based on this article, the cause of the slowdown could be due to conversion between internal format, which may be GL_BGRA, and pixel transfer format, which is GL_RGBA in my code. Changing the transfer format to GL_RGB will reduce the latency of glReadPixels to around 100ms, but when I map the buffer with GLES30.glMapBufferRange the output frame doesn't look rendered correctly. I also tried the GL_BGRA format in GLES11Ext but it will cause GL_INVALID_OPERATION in glReadPixel.
Is there any other way to make glReadPixels on Android return immediately so that PBO can improve performance?
As Reto has suggested, it turns out to be an implementation specific issue. The GPU that I was originally testing with is Adreno 306. When I test the same codes on Samsung Note 4 (Adreno 420), it works as expected. So it's always worthwhile to test on different devices and GPUs for such types of issues.
I have some code which is loading an image into an OpenGL texture. In the process, I end up loading 3 bitmaps, since I need to load the original bitmap (sized appropriately for the display) and reorient the bitmap based on EXIF data. I'm very quickly calling .recycle() on each bitmap, but I'm noticing that my memory doesn't seem to change.
Here's what the Memory Monitor shows:
As you can see, after loading the image I'm using about 60MB of memory. When I rotate the device that drops off a bit then comes back up. That leads me to think there is no leak, since the memory never goes above that.
When I click the GC button in the memory analyzer, my memory footprint drops dramatically to around 8 MB. This makes sense as the three bitmaps created during the process were recycled, so can be garbage collected. Then you can see that when I rotate again and the activity is rebuilt, the memory jumps right back up.
Here's my code to show you why so many bitmaps are created and when they're recycled.
void layoutImage() {
...
Bitmap bitmap = loadOrientedConstrainedBitmapWithBackouts(...);
imageTexture = new GLTexture(bitmap);
bitmap.recycle(); // recycle bitmap 2
}
Bitmap loadOrientedConstrainedBitmapWithBackouts(Context context, Uri uri, int maxSize) {
...
Bitmap bitmap = loadBitmapWithBackouts(context, uri, sampleSize); // create bitmap 1
...
Bitmap out = orientBitmap(bitmap, orientation); // create bitmap 2
bitmap.recycle(); // recycle bitmap 1
return out;
}
Bitmap orientBitmap(Bitmap source, int orientation) {
...
return Bitmap.createBitmap(bitmap, 0, 0, bitmap.getWidth(), bitmap.getHeight, matrix, true); // create bitmap 3
}
I'm not really sure that this is a problem, so to speak, since the memory isn't climbing (so no leak), but I'm curious when it stays so high. Since forcing a garbage collection clears it just fine, should I assume that if the system needs that memory it will be collected on the next GC pass? It's been running the whole time I've been writing this and is still sitting comfortably at 60 MB.
Question 1: Can I just trust that the garbage collector will take that memory back if needed?
Also, if we're supposed to be so judiciously recycling our bitmaps, why do so many of the Bitmap methods say things like "The new bitmap may be the same object as source, or a copy may have been made." Do I really have to check the equality every time I use those methods to recycle the bitmap if it's a different object?
Question 2: When using Bitmap creation methods, that may or may not return the same bitmap or a copy, do I need to check source and output equality to recycle the source if it's a copy?
Edit:
I have tried analyzing this with MAT, using a heap dump at peak usage (should be 60 MB), but it only reports 18.2 MB of usage and nothing unusual looking. Could they be reading things differently?
Question 1: Can I just trust that the garbage collector will take that memory back if needed?
Yes. If the incoming references are cleared, the garbage collector will take the memory when it is needed (typically for a new allocation). Calling recycle() doesn't help this process along or make it happen any faster.
The recycle() method exists because Bitmap objects were not counted against the heap until Android 3.0; so the method was helpful to assist the GC since it didn't otherwise have a record of that memory counted against its heap. In 3.0+, the memory is tracked against the heap so this extra bookkeeping isn't necessary anymore.
Question 2: When using Bitmap creation methods, that may or may not return the same bitmap or a copy, do I need to check source and output equality to recycle the source if it's a copy?
The createBitmap() method will return the same object if:
The source is immutable
x and y are both zero
width and height match the source width and height
No transformation matrices have been applied
Since it looks like you are passing in a transformation matrix, you will always get a copy unless the matrix is identity for some reason. But again, no real need to recycle() unless you are still supporting 2.x versions.
(This is due to the limitations of the server software I will be using, if I could change it, I would).
I am receiving a sequence of 720x480 JPEG files (about 6kb in size), over a socket. I have benchmarked the network, and have found that I am capable of receiving those JPEGs smoothly, at 60FPS.
My current drawing operation is on a Nexus 10 display of 2560x1600, and here's my decoding method, once I have received the byte array from the socket:
public static void decode(byte[] tmp, Long time) {
try {
BitmapFactory.Options options = new BitmapFactory.Options();
options.inPreferQualityOverSpeed = false;
options.inDither = false;
Bitmap bitmap = BitmapFactory.decodeByteArray(tmp, 0, tmp.length, options);
Bitmap background = Bitmap.createScaledBitmap
(bitmap, MainActivity.screenwidth, MainActivity.screenheight, false);
background.setHasAlpha(false);
Canvas canvas = MainActivity.surface.getHolder().lockCanvas();
canvas.drawColor(Color.BLACK);
canvas.drawBitmap(background, 0, 0, new Paint());
MainActivity.surface.getHolder().unlockCanvasAndPost(canvas);
} catch (Exception e) {
e.printStackTrace();
}
}
As you can see, I am clearing the canvas from a SurfaceView, and then drawing the Bitmap to the SurfaceView. My issue is that it is very, very, slow.
Some tests based on adding System.currentTimeMillis() before and after the lock operation result in approximately a 30ms difference between getting the canvas, drawing the bitmap, and then pushing the canvas back. The displayed SurfaceView is very laggy, sometimes it jumps back and forth, and the frame rate is terrible.
Is there a referred method for drawing like this? Again, I can't modify what I'm getting from the server, but I'd like the bitmaps to be displayed at 60FPS when possible.
(I've tried setting the contents of an ImageView, and am receiving similar results). I have no other code in the SurfaceView that could impact this. I have set the holder to the RGBA_8888 format:
getHolder().setFormat(PixelFormat.RGBA_8888);
Is it possible to convert this stream of Bitmaps into a VideoView? Would that be faster?
Thanks.
Whenever you run into performance questions, use Traceview to figure out exactly where your problem lies. Using System.currentTimeMillis() is like attempting to trim a steak with a hammer.
The #1 thing her is to get the bitmap decoding off the main application thread. Do that in a background thread. Your main application thread should just be drawing the bitmaps, pulling them off of a queue populated by that background thread. Android has the main application thread set to render on a 60fps basis as of Android 4.1 (a.k.a., "Project Butter"), so as long as you can draw your Bitmap in a couple of milliseconds, and assuming that your network and decoding can keep your queue current, you should get 60fps results.
Also, always use inBitmap with BitmapFactory.Options on Android 3.0+ when you have images of consistent size, as part of your problem will be GC stealing CPU time. Work off a pool of Bitmap objects that you rotate through, so that you generate less garbage and do not fragment your heap so much.
I suspect that you are better served letting Android scale the image for you in an ImageView (or just by drawing to a View canvas) than you are in having BitmapFactory scale the image, as Android can take advantage of hardware graphics acceleration for rendering, which BitmapFactory cannot. Again, Traceview is your friend here.
With regards to:
and have found that I am capable of receiving those JPEGs smoothly, at 60FPS.
that will only be true sometimes. Mobile devices tend to be mobile. Assuming that by "6kb" you mean 6KB (six kilobytes), you are assuming a ~3Mbps (three megabits per second) connection, and that's far from certain.
With regards to:
Is it possible to convert this stream of Bitmaps into a VideoView?
VideoView is a widget that plays videos, and you do not have a video.
Push come to shove, you might need to drop down to the NDK and do this in native code, though I would hope not.
I'm using very large bitmaps and I store data in a big int[]. The images can be really large and I can't downsample them (I'm getting the bitmaps over the wire and rendering them).
The problem I'm hitting is on very large bitmaps (bitmap size = 64MB), where I try to allocate int array with size 16384000. I'm testing this on Samsung Galaxy SII, which should have enough memory to handle this, but it seems there is a "cap" on heap size. The method Runtime.getRuntime().maxMemory() returns 64MB, so this is the max heap size for this particular device.
The API level is set to 10, so I can't use android:largeHeap attribute suggested elsewhere (and I don't even know if that would help).
Is there any way to allocate more than 64MB? I tried allocating the array in native (using JNI NewIntArray function), but that fails as well. It seems as though it is bound by the same limit as jvm.
I could however allocate memory on the native side using NewDirectByteBuffer, but since this byte buffer is not backed by an array, I can not get access to int[] (using asIntBuffer().array() in java which I need in order to display the image using setPixels or createBitmap. I guess OpenGL would be a way to go, but I have (so far) 0 experience with OpenGL.
Is there a way to somehow access allocated memory as int[] that I am missing?
So, the only way I've found so far is to allocate image using NDK. Furthermore, since Bitmap does not use existing Buffer as pixel "storage" (the method copyPixelsFromBuffer is also bound to memory limits; and judging by the method name, it copies the data anyway).
The solution (I've only prototyped it roughly) is to malloc whatever the size of the image is, and fill it using c/c++ and then use ByteBuffer in Java with OpenGLES.
The current prototype creates a simple plane and applies this image as a texture to it (luckily, OpenGLES methods take Buffer as input, which seems to work as expected). I'm using glTexImage2D to apply this buffer as a texture to a plane. Here is a sample, where mImageData is ByteBuffer allocated (and filled) on the native side.
int[] textures = new int[1];
gl.glGenTextures(1, textures, 0);
mTextureId = textures[0];
gl.glBindTexture(GL10.GL_TEXTURE_2D, mTextureId);
gl.glTexParameterf(GL10.GL_TEXTURE_2D, GL10.GL_TEXTURE_WRAP_T, GL10.GL_REPEAT);
gl.glTexImage2D(GL10.GL_TEXTURE_2D, 0, GL10.GL_RGBA, 4000, 4096, 0, GL10.GL_RGBA, GL10.GL_UNSIGNED_BYTE, mImageData);
I assume OP already solved this, but if you have a Stream, you can stream the bitmap into a file and read that file using inSampleSize
I am streaming a video in android and I decode frames in native code and then copy the pixels to a bitmap, then display the bitmap in Java using canvas.unlockandpost with a while loop for all the bitmaps.
Everything is fine, but the streaming of bitmaps is very slow and causes a crash. I only see a message on logcat saying that "low memory no more background processes".
I see on the allocation table from eclipse, that the bitmaps that I created are not getting deleted from memory, even though, I am overwritng the pixels everytime. Is there any way I can clean up the memory it is keeping.
My code is as follows.
C Code :
AndroidBitmapInfo info;
void* pixels;
int ret;
if ((ret =AndroidBitmap_lockPixels(env, bitmap, &pixels)) < 0) {
}
memcpy(pixels, pictureRGB, 480*320);
AndroidBitmap_unlockPixels(env, bitmap);
Java Code
Bitmap mBitmap = Bitmap.createBitmap(480, 320, Bitmap.Config.RGB_565);
renderbitmap(mBitmap, 0);
canvas.drawBitmap(mBitmap, 0, 0, null);
The code shown in your question is missing some critical parts to fully understand your problem, but it sounds like you're creating a new bitmap for every frame. Since Android only allows for about 16MB of allocations for each Java VM, your app will get killed after about 52 frames. You can create a bitmap once and re-use it many times. To be more precise, you are creating a bitmap (Bitmap.CreateBitmap), but not destroying it (Bitmap.recycle). That would solve your memory leak, but still would not be the best way to handle it. Since the bitmap size doesn't change, create it once when your activity starts and re-use it throughout the life of your activity.