I am loading textures into OpenGLES on Android and maintaining a reference to the generated id in a HashMap.
At a given point in time not all textures that have been loaded will be in use, but may be used at a later point in time, therefore if the device has enough free memory I'd like to keep the textures loaded.
However, if the device starts running low on memory I would like to delete any textures that are not in use since they can always be reloaded later if they're required.
I've tried a few methodologies for handling this scenario.
1. Respond to system memory warnings
If the application receives a memory warning, then it will identify which textures are not in use and schedule for those textures to be deleted.
This method did work reasonably well.
2. Use Soft References
In this approach the application would maintain a List<SoftReference<Texture>> where the Texture class is a wrapper around the id that was returned when a given texture was loaded into OpenGLES.
If a given Texture is not in use at a given point in time then only a SoftReference would exist to this Texture and thus if the garbage collector deemed it necessary it could reclaim this memory and the finalize method on the Texture class would schedule for this texture to be deleted.
This approach seemed ideal based upon the description of SoftReference in the Java documentation, since they would only be reclaimed when more memory was required.
Soft reference objects, which are cleared at the discretion of the garbage collector in response to memory demand. Soft references are most often used to implement memory-sensitive caches.
However, the Android implementation of SoftReference does not work like this: since Android 9 as the garbage collector is more aggressive and the soft references are reclaimed almost immediately regardless of whether the device is low on memory.
3. Use LruCache
The Android documentation advises against using SoftReference in a cache implementation and to use a LruCache instead. However, the LruCache poses some drawbacks.
Firstly, you have to specifying the maximum size of the cache. It isn’t obvious what to set the cache size to: ideally it’d just automatically be set as high as possible while still being a good citizen. If it’s set too small, it might be constantly reloading textures unnecessarily.
Secondly, a Texture may be removed from the cache which is currently in use and thus may be deleted from OpenGLES and then display as a missing texture to the user.
Is there a better way to maintain a cache of textures in OpenGL and be responsive to low memory scenarios (besides just deleting textures upon memory warnings)?
Talking about best practices, large caches (that can take all the memory) with OpenGL ids for textures are rarely used. Actually it's worth to have only a few textures to render in the scene at one moment. More textures you have in the scene, more texture switches you need to do in frame. It costs. Texture atlases were developed many years ago to reduce number of texture switches. However, atlases still can take a lot of memory.
With development of hardware capabilities and user expectations, texture atlases evolved into virtual/mega/sparse textures + texture data streaming for the cases of high memory usage. The idea is to use virtual memory approach and load/unload blocks of one very large texture in real-time. It has some drawbacks, good discussion about it can be found here. LRU cache can be built on top of it to tell what blocks are required at the moment.
Of course, engines can preload many textures (sparse or usual ones) and unload it after usage, e.g. as a part of dynamic loading of open world in a game. These textures aren't used for rendering simultaneously, and no one waits until they take all the memory to start unloading. The eviction algorithm here is highly dependant on particular app, even though maximum cache size in Mb is widely used practice here.
Related
I'm working on an Android app built in Unity3D that needs to create new textures at runtime every so often based off different images pixel data.
Since Unity for Android uses OpenGL ES and my app is a graphical one that needs to run at ideally a solid 60 frames per second, I've created a C++ plugin operating on OpenGL code instead of just using Unity's Texture2D slow texture construction. The plugin allows me to upload the pixel data to a new OpenGL texture, then let Unity know about it through their Texture2D's CreateExternalTexture() function.
Since the version of OpenGL ES running in this setup is unfortunately single-threaded, in order to keep things running in frame I do a glTexImage2D() call with an already gen'd TextureID but with null data in the first frame. And then call glTexSubImage2D() with a section of my buffer of pixel data, over multiple subsequent frames to fill out the whole texture, essentially doing the texture creation synchronously but chunking the operation up over multiple frames!
Now, the problem I'm having is that every time I create a new texture with large dimensions, that very first glTexImage2D() call will still lead to a frame-out, even though I'm putting null data into it. I'm guessing that the reason for this is that there is still a pretty large memory allocation going on in the background with that first glTexImage2D() call, even though I'm not filling in the image until later.
Unfortunately, these images that I'm creating textures for are of varying sizes that I don't know of beforehand and so I can't just create a bunch of textures up front on load, I need to specify a new width and height with each new texture every time. =(
Is there anyway I can avoid this memory allocation, maybe allocating a huge block of memory at the start and using it as a pool for new textures? I've read around and people seem to suggest using FBO's instead? I may have misunderstood but it seemed to me like you still need to do a glTexImage2D() call to allocate the texture before attaching it to the FBO?
Any and all advice is welcome, thanks in advance! =)
PS: I don't come from a Graphics background, so I'm not aware of best practices with OpenGL or other graphics libraries, I'm just trying to create new textures at runtime without framing out!
I haven't dealt with the specific problem you've faced, but I've found texture pools to be immensely useful in OpenGL in terms of getting efficient results without having to put much thought into it.
In my case the problem was that I can't use the same texture for an input to a deferred shader as the texture used to output the results. Yet I often wanted to do just that:
// Make the texture blurry.
blur(texture);
Yet instead I was having to create 11 different textures with varying resolutions and having to swap between them as inputs and outputs for horizontal/vertical blur shaders with FBOs to get a decent-looking blur. I never liked GPU programming very much because some of the most complex state management I've ever encountered was often there. It felt incredibly wrong that I needed to go to the drawing board just to figure out how to minimize the number of textures allocated due to this fundamental requirement that texture inputs for shaders cannot also be used as texture outputs.
So I created a texture pool and OMG, it simplified things so much! It made it so I could just create temporary texture objects left and right and not worry about it because the destroying the texture object doesn't actually call glDeleteTextures, it simply returns them to the pool. So I was able to finally be able to just do:
blur(texture);
... as I wanted all along. And for some funny reason, when I started using the pool more and more, it sped up frame rates. I guess even with all the thought I put into minimizing the number of textures being allocated, I was still allocating more than I needed in ways the pool eliminated (note that the actual real-world example does a whole lot more than blurs including DOF, bloom, hipass, lowpass, CMAA, etc, and the GLSL code is actually generated on the fly based on a visual programming language the users can use to create new shaders on the fly).
So I really recommend starting with exploring that idea. It sounds like it would be helpful for your problem. In my case I used this:
struct GlTextureDesc
{
...
};
... and it's a pretty hefty structure given how many texture parameters we can specify (pixel format, number of color components, LOD level, width, height, etc. etc.).
Yet the structure is comparable and hashable and ends up being used as a key in a hash table (like unordered_multimap) along with the actual texture handle as the value associated.
That allows us to then do this:
// Provides a pool of textures. This allows us to conveniently rapidly create,
// and destroy texture objects without allocating and freeing an excessive number
// of textures.
class GlTexturePool
{
public:
// Creates an empty pool.
GlTexturePool();
// Cleans up any textures which haven't been accessed in a while.
void cleanup();
// Allocates a texture with the specified properties, retrieving an existing
// one from the pool if available. The function returns a handle to the
// allocated texture.
GLuint allocate(const GlTextureDesc& desc);
// Returns the texture with the specified key and handle to the pool.
void free(const GlTextureDesc& desc, GLuint texture);
private:
...
};
At which point we can create temporary texture objects left and right without worrying about excessive calls to glTexImage2D and glDeleteTextures.I found it enormously helpful.
Finally of note is that cleanup function above. When I store textures in the hash table, I put a time stamp on them (using system real time). Periodically I call this cleanup function which then scans through the textures in the hash table and checks the time stamp. If a certain period of time has passed while they're just sitting there idling in the pool (say, 8 seconds), I call glDeleteTextures and remove them from the pool. I use a separate thread along with a condition variable to build up a list of textures to remove the next time a valid context is available by periodically scanning the hash table, but if your application is all single-threaded, you might just invoke this cleanup function every few seconds in your main loop.
That said, I work in VFX which doesn't have quite as tight realtime requirements as, say, AAA games. There's more of a focus on offline rendering in my field and I'm far from a GPU wizard. There might be better ways to tackle this problem. However, I found it enormously helpful to start with this texture pool and I think it might be helpful in your case as well. And it's fairly trivial to implement (just took me half an hour or so).
This could still end up allocating and deleting lots and lots of textures if the texture sizes and formats and parameters you request to allocate/free are all over the place. There it might help to unify things a bit, like at least using POT (power of two) sizes and so forth and deciding on a minimum number of pixel formats to use. In my case that wasn't that much of a problem since I only use one pixel format and the majority of the texture temporaries I wanted to create are exactly the size of a viewport scaled up to the ceiling POT.
As for FBOs, I'm not sure how they help your immediate problem with excessive texture allocation/freeing either. I use them primarily for deferred shading to do post-processing for effects like DOF after rendering geometry in multiple passes in a compositing-style way applied to the 2D textures that result. I use FBOs for that naturally but I can't think of how FBOs immediately reduce the number of textures you have to allocate/deallocate, unless you can just use one big texture with an FBO and render multiple input textures to it to an offscreen output texture. In that case it wouldn't be the FBO helping directly so much as just being able to create one huge texture whose sections you can use as input/output instead of many smaller ones.
I'm writing an image gallery app and I keep running into out of memory errors. I cache all my images but the problem occurs when I try switching between images really fast. I'm assuming the app is allocating memory faster than the GC has time to free them up (because the crash doesn't happen when I switch images slowly).
After banging my head against this problem for days, I finally decided to give largeHeap setting in the manifest file a try. After this setting, my app no longer crashes no matter how fast I switch between images.
Now, I want to know if there is any convention or general guideline to using largeHeap setting because it probably wouldn't make much sense if, say, a note taking app used largeHeap. Generally speaking, what apps are a good candidate for largeHeap setting?
Thanks
Generally speaking, what apps are a good candidate for largeHeap setting?
Ones where you can justify to the user why you're forcing all their other apps out of memory, to give you an outsized amount of heap space.
Personally, I would not consider "an image gallery app" to qualify. AutoCAD, video editors, and the like would qualify.
With respect to your memory management issues, make sure that you are using inBitmap on BitmapOptions when running on API Level 11+, so you recycle existing buffers rather than go through garbage collection. Particularly for an image gallery, where you probably have a lot of fairly consistent thumbnail sizes, recycling existing buffers will be a huge benefit. This can help both overall memory consumption (i.e., you are truly out of memory) and memory fragmentation (i.e., you get an OutOfMemoryError with plenty of heap space, but no single block big enough for your allocation, due to Android's frakkin' non-compacting garbage collector).
You might also consider looking at existing image cache implementations, such as the one that Picasso has, to see if there are some tips you could learn (or possibly just reuse).
First, make sure you aren't loading larger bitmaps than necessary:
Load a Scaled Down Version into Memory.
Then, before trying largeHeap, try to free the memory quickly yourself:
If you call bitmap.recycle(); as soon as you are SURE you will not use a bitmap again, then the bulk of that bitmap's memory will be immediately freed. (When the GC gets around to it, all that remains is a tiny object.)
On newer Android versions, there are alternatives (instead of recycle) that may be more effective:
Managing Bitmap Memory
Personally, I still use recycle often, especially if I might be loading a different size image, so can't reuse the existing one. Also, I find it easier to code "unloading" of old media separately from "loading" of new media, when changing to a different fragment or activity:
As leave the old fragment, all old bitmaps I recycle (then, if reachable from a static field, set to null).
The rule of thumb for whether to use largeHeap, is to consider it after you've tried alternative ways to reduce memory usage.
Code your app so you could turn it off again, and still run.
For example, monitor your memory usage, and load "scaled down" bitmaps if memory is tight. Will the user really notice if a given image is not at their device's "retina" resolution?
Or if it is an older, slower, device, will largeHeap make your app feel unresponsive / jerky? If so, can you drop resolution even further, or show fewer bitmaps at one time?
Get your app to work in all circumstances, without largeHeap [by techniques mentioned above]. NOTE: you can "force-test" running on tight memory, by allocating some "dummy" bitmaps, and hold references to them in global fields, so they don't get freed.
NOW you are able to evaluate the trade-off, as it affects YOUR app:
When you do turn largeHeap on, use your app heavily - are there places where it is now "more sluggish", or animations "stutter" or otherwise seem less smooth? BE SURE TO TEST ON AT LEAST ONE OLDER DEVICE, AND ON ONE HIGH_RESOLUTION DEVICE.
You might be seeing long GC times, due to the larger heap.
OR you might conclude that largeHeap is working well for you, and now you can confidently say that it is the best choice in your circumstance.
I am implementing a game engine using OpenGL and wonder if it's a good idea to manually unload textures that are not used within a certain radius of the camera. It has been suggested to me for Android and OpenGL ES that it might be a good idea to load and unload textures on the fly as needed to save memory. Is this recommended for OpenGL ES and OpenGL? I am personally not convinced by the approach but I am curious as to when this will have benefits, if any, in the setting of OpenGL on a desktop PC and with OpenGL ES on a mobile phone.
It will depend on how many textures/the size of the textures you use in your game.
I believe the magic number is minimum 10MB of texture memory, so if your textures use more than that people tend to use texture compression (phone compatibility issues here, some compression formats are proprietary, there's a whole topic on this). You could unload textures when they aren't being used but there will be a heavy cost when you reload them, so be aware of that.
Generally speaking, you shouldn't need to resort to this method though.
If you don't need that memory, then it does no harm being there. And if you do need that memory, then you already know that you need to do it. So the answer is... do it if you need to.
Please note I do NOT have a memory leak. My question is about a subtler issue.
I recently wrote an android app which does image processing. The image is loaded as a Bitmap, then copied out in pixels, processed in a way that uses lots of memory (think Fourier transforms in floating point representations and stuff), then converted back into a bitmap and saved.
The problem is, through at least android OS 2.3, the total memory limitation (typically 16MB) is combined java and (externally stored) Bitmaps, and the java high water mark doesn't go down (that I can discern) even when the memory is free (successfully GC'd), which means when I go to allocate the final Bitmap, I am often "out of memory" even though by that point I have freed (and GC'd) most of the space. I.e., I never need the full 16MB at once, but the space left for Bitmaps appears to be 16MB minus the MAX historical java heap usage (as opposed to current usage).
I watched a tech talk by one of the android developers about memory issues and he implied this problem has been fixed in subsequent versions of the OS (they moved Bitmap memory into the java heap space), but in the meantime most of the people wanting to use my app are running 2.2 or 2.3.
Long story short, I am wondering if the java heap is ever compacted (de-fragmented, in effect) so that the high-water mark shrinks (and if so, how to make it happen)?
If not, then does anybody have another suggestion how to deal with this problem?
Long story short, I am wondering if the java heap is ever compacted (de-fragmented, in effect) so that the high-water mark shrinks (and if so, how to make it happen)?
Whatever its behavior is, it most certainly is not under your control.
If not, then does anybody have another suggestion how to deal with this problem?
Ideally, reuse your own Bitmaps. You don't indicate what "processed in a way that uses lots of memory" really is. However, if it does not change the dimensions or bit depth of the image, copy the data back out to the original Bitmap rather than allocating a fresh one, if you can.
Image processing on Android 2.x is one of the few places where I can see justifying using multiple processes. You will add overhead for schlepping the image data between processes, but the other process has its own heap (Java and native), so this may give you more "elbow room".
So far, no indication that there is any way to compact the memory.
Here is my workaround, which is suboptimal but much better than the behavior before:
I now intentionally hold on to the original Bitmap while I am doing my processing, and then recycle() and null it, and GC(), but not until just before allocating my output Bitmap.
What this does is reserve external (Bitmap) space, and cause my application to run out of java heap (during processing, before calling recycle()), which I can at least catch and handle by retrying on a smaller image. (Before, everything seemed to be fine until I tried to save, but by then it was too late and with no way to recover.)
Technically this limits my max image size to less than I should be able to do with the allotted memory (because I need to reserve space in the heap and external at the same time when in truth I never need both together), but at least I can still handle a reasonable image size.
What was happening before is I would free and recycle the Bitmap early which allowed the high water mark on the java heap to use up essentially all of my memory allotment, meaning from that point forward I couldn't open or create any more Bitmaps at all (other than tiny thumbnail sizes sometimes).
Imo, this is a major bug in the way android handles Bitmap memory, but I believe it is fixed in newer versions of the OS so hopefully I can disable this workaround conditional on the OS release.
I'm assuming that you already call Bitmap.recycle() but it's the only thing I remembered and you didn't talk about.
In my game i have 12 different animations and each animation consist of 10 bitmaps and each bitmap is 200x150 pixel. Every time user touches to the screen(virtual gamepad) one of the animations will play. Normally this is an easy job but performance wise i am having trouble. Loading all bitmaps on an array from the beginning of the game cause to crash game because of VM memory budget problem. It seems
my other option is to recreate bitmaps everytime i need and then recyle them, but this might cause lag because of creating time and garbage collection. I can't think of third option so i am asking you. Is there any memory efficient and not laggy option ?
That's 14MB of memory for your animations which might be a lot for the VM heap it should fit nicely into the graphics card memory. I am assuming that you are using OpenGL (if not you will almost have to). In which case you can build your frames one-by-one, bind them to a texture and release them from the VM's memtory. This will be fairly fast but I wouldn't do it often or offer some sort of loading feature if the lag is greater than 1/2 second or so.
I would consider using a SpriteSheet for this. SpriteSheet would contain all the frames of the animation on one bitmap, and you can then programatically load the correct frame coordinates to create your animation.