Can anyone explain the below Logcat message :
D/dalvikvm(4440): GC_EXTERNAL_ALLOC freed 338K, 47% free 6427K/11911K, external 20418K/22446K, paused 53ms
E/dalvikvm-heap(4440): 2519424-byte external allocation too large for this process.
D/dalvikvm(4440): GC_FOR_MALLOC freed <1K, 47% free 6427K/11911K, external 20398K/22446K, paused 40ms
E/GraphicsJNI(4440): VM won't let us allocate 2519424 bytes
In older versions of Android, certain bits of native framework code would tell the VM about native allocations. This "external allocation" mechanism was an ugly hack introduced so that native allocations would cause the Dalvik VM to do a garbage collection pass.
The basic problem was that the Java-language Bitmap object used native memory for the pixel storage. Because the managed-heap objects were tiny, and the native-heap objects were large, you could allocate tons of Bitmaps without causing a GC. This was causing apps to bloat up and the system to slow down.
So, "external allocations" were introduced. Whenever the pixel storage for a Bitmap was allocated on the native heap, an equal amount of memory was deducted from the managed heap. The idea is that, if your heap is filling up with no-longer-reference bitmaps, you'll run out of managed heap space and the GC will fire.
(Unfortunately the GC can't actually release the native storage -- you need to run a finalizer to do that, and finalizers run in a separate pass after the GC completes. For a while the native objects were also holding on to some additional managed-heap objects, so you'd have to GC + finalize + GC to actually clean everything up.)
My "favorite" part about external allocations is that the API was a simple "increase by N" / "decrease by N", which meant there was no way to associate the native heap with a managed heap object, or check for leaks. Because all of the information about the Bitmap was kept in the native object, you couldn't even guess at how much native storage was needed, so it was impossible to look at an hprof dump and figure out how much memory a Bitmap was actually using.
In Android 3.0 ("Honeycomb") the pixel storage was moved onto the managed heap, and the external allocation mechanism was removed.
So, what the log message in your question means is: some code, probably Bitmap, wanted to allocate 2.5MB of native heap, but that would exceed the VM's external allocation heap limit. You need to figure out what's eating up 20MB of external allocation storage and release some of it.
The only way to get information about external allocations is by watching the event log. A few years back I threw a script together (gclog.py -- was in AOSP dalvik/tools for a while). I have no idea if it will still do anything useful. I talk about how to use it in this old thread.
Related
I have an application which is performing some memory intensive tasks. I am trying to figure out what the total available memory is and what the available free memory is. I am doing so by using Runtime.getRuntime().freeMemory() and Runtime.getRuntime().totalMemory()
I am curious if the result I am getting is the total amount of free memory and total memory on the device or if it is the total amount of memory available to that instance of the Dalvik VM that the application is running on top of? I would appreciate some feedback. Thanks!
According to the JavaDocs, those should be in terms of the heap for your process. That being said:
getMemoryClass() on ActivityManager is a more Android-y way to determine the heap size for your process.
Because Dalvik's garbage collector is non-compacting, memory reported by methods like freeMemory() will overstate how much you can allocate. By "non-compacting", I mean that if you free up two blocks of memory that happen to be adjacent, the garbage collector leaves them as two blocks of memory. Contrast this to the Java VM, which will recognize that the two blocks are adjacent and update the heap to reflect one larger free block rather than two smaller free blocks. The non-compacting garbage collector means that your heap is more prone to fragmentation than complete exhaustion. You will try to allocate some large block and get an OutOfMemoryError, not because the heap lacks free memory, but because there is no single free block big enough for your request.
Can you please tell me how does Dalvik memory fragmentation handling?
For example, if I have a an app allocation some small memory. but free some of them, and then later on trying to allocate a big chunk. How can Dalvik handle such request, with all the 'holes' in the heap?
I need help in understanding this from logcat:
11-15 02:50:21.837 D/dalvikvm( 990): GC_BEFORE_OOM freed 16K, 40% free 9830K/16384K, paused 90ms, total 90ms
11-15 02:50:21.837 E/dalvikvm-heap( 990): Out of memory on a 5032864-byte allocation.
It said my application has use 9830K out of 16384K. And I understand 16M is the maximum heap size, it can't grow any further.
But that should leave 6554K free (16384k - 9830K).
Why dalvik says 'out of memory on a 5032864 byte allocation. That is 4914K which is smaller that I have left.
Thank you.
Take a Look at this, It might help you to understand how to handle dalvik memory fragmentation
Link1
Like most implementations of Garbage Collector (GC), the GC of Dalvik/Java uses double references: reference variables in memory don't point directly toward their objects but toward a second reference which finally point toward the individual objects. Therefore, the GC is free to move around the memory allocations and fuses together the holes when the application need a big memory allocation without impacting in any way the various reference variables allocated by Java.
Update -- Pardon my verbosity below. As concisely as possible, this behaves itself:
try {
mArrowBitmap = BitmapFactory.decodeFile(nam);
} catch(OutOfMemoryError e) {
Log.e(TAG, "OUT OF MEMORY ERROR!!! ["+e+"]");
}
meaning that it throws an OOME when the external heap is exhausted. And I can catch that, and use it in unit testing.
This, however, misbehaves:
try {
mArrowDrawable = new BitmapDrawable(mContext.getResources(), mArrowBitmap);
} catch(OutOfMemoryError e) {
Log.e(TAG, "OUT OF MEMORY ERROR!!! ["+e+"]");
}
meaning that it performs a really impressive suppuku. All I can do then is go through its pockets looking for loose change.
And for those with more time on their hands . . .
Problem: I have lots of bitmaps, some from Resources, some loaded externally. I do some unavoidable manipulation on the device (i.e. stuff I can't do ahead of time, or do server-side). Someplace during loading, I have at least one memory leak. I'm chasing that, and pretending to be competent (and spending lots of time staring at DDMS/MAT). Thanks to lots of comments on SO, I have done at least the basic research on bitmaps, especially on where they live pre-Honeycomb. My app runs fine Honeycomb and forward. It runs less fine on Gingerbread. I can either
(a) ignore the memory leak since Honeycomb and forward will eventually reclaim the heap memory and, instead, try to get everybody on the planet to stop using Gingerbread, or
(b) isolate and squish the memory leak(s).
Option (b) seems better.
On Gingerbread, I understand that the native "backing" memory for my bitmaps (including the ones that the system mysteriously creates on my behalf when I load .xml or Drawables), all lives outside the Dalvik heap. I further apprehend that the only thing on the Dalvik heap is a (relatively) small pointer to that native backing memory. When I attempt to release resources, I must locate all the bitmaps I have created, and do a synchronous recycle on them. My scheme for this, fwiw, is to "register" all my bitmaps into a simple vector when I create them, and then iterate the vector to recycle anything resembling a bitmap in native backing memory.
For this reason, I am untroubled by the various reports from the Dalvik heap that there is plenty of memory on hand, yet my decode calls fail for lack of memory. In fact, I have dedicated some of my unit-test activities to pushing the boundary right to the edge to verify I'm not misbehaving (and, hopefully, do not do so in the future). I do these unit tests using the "standard" Android unit test classes, since Robolectric can't "see" the heap to which I refer above. Confession: I really like Robolectric much better, because it's about a zillion times faster.
That is a long way of getting to here:
When I intentionally run the native 'backing' memory off the end, by creating too many bitmaps, most of the time, I get what I expect: OutOfMemoryError. I catch that Error and act responsibly (for very small values of 'responsibly').
When I prevail upon the Drawable calls (instead of the BitmapFactory calls, but sometimes both do this behavior), instead of an Error or an Exception, I get this in the logcat:
11-17 16:22:39.372: I/dalvikvm-heap(15069): Clamp target GC heap from 33.916MB to 32.000MB
11-17 16:22:39.372: D/dalvikvm(15069): GC_CONCURRENT freed 5K, 39% free 3701K/6023K, external 26854K/27965K, paused 2ms+2ms
11-17 16:22:39.402: D/dalvikvm(15069): GC_EXTERNAL_ALLOC freed 5K, 39% free 3696K/6023K, external 26854K/27965K, paused 36ms
11-17 16:22:39.442: E/dalvikvm-heap(15069): 81796-byte external allocation too large for this process.
11-17 16:22:39.442: E/dalvikvm(15069): Out of memory: Heap Size=6023KB, Allocated=3696KB, Bitmap Size=26854KB, Limit=32768KB
11-17 16:22:39.442: E/dalvikvm(15069): Trim info: Footprint=6023KB, Allowed Footprint=6023KB, Trimmed=404KB
11-17 16:22:39.442: E/GraphicsJNI(15069): VM won't let us allocate 81796 bytes
11-17 16:22:39.462: I/dalvikvm-heap(15069): Clamp target GC heap from 33.912MB to 32.000MB
11-17 16:22:39.462: D/dalvikvm(15069): GC_CONCURRENT freed <1K, 39% free 3696K/6023K, external 26854K/27965K, paused 3ms+2ms
I grok the clamping parts. What I can't seem to untangle is how to detect the part in the middle:
11-17 16:22:39.442: E/dalvikvm(15069): Out of memory: Heap Size=6023KB, Allocated=3696KB, Bitmap Size=26854KB, Limit=32768KB
It does not appear to be either an Error or an Exception. I can, of course, see this bad behavior manually by watching DDMS. But I can't figure out how to isolate it programmatically, so that I can incorporate it into my unit tests. It would, of course, also be nice to catch this behavior in my app, so that if a mistake does slip past my gauntlet of unit tests (it happens(way too often)) I might avoid the app unceremoniously barfing on its shoes.
Many thanks for the dedicated folks hereabouts who have explained the mysteries of Android memory with aplomb. For anybody still reading, and puzzled by as I was by Android's entertaining memory structure, I heartily recommend starting here, with Mr. Dubroy's excellent presentation -- Well worth watching, especially to the end of the audience questions. Right at the end is a person off-camera who asks some exceedingly well-informed questions about these same baffling memory bits.
Any guidance here would be gratefully accepted.
If you see the Android logs, you may see a lot of those things.
What do they mean, knowing those may help us doing better memory allocations.
Example:
28470 dalvikvm D GC_FOR_MALLOC freed 665 objects / 239992 bytes in 71ms
28470 dalvikvm D GC_FOR_MALLOC freed 673 objects / 240288 bytes in 87ms
21940 dalvikvm D GC_EXPLICIT freed 4802 objects / 185320 bytes in 78ms
28470 dalvikvm D GC_FOR_MALLOC freed 666 objects / 240536 bytes in 63ms
GC_FOR_MALLOC means that the GC was triggered because there wasn't enough memory left on the heap to perform an allocation. Might be triggered when new objects are being created.
GC_EXPLICIT means that the garbage collector has been explicitly asked to collect, instead of being triggered by high water marks in the heap. Happens all over the place, but most likely when a thread is being killed or when a binder communication is taken down.
There are a few others as well:
GC_CONCURRENT Triggered when the heap has reached a certain amount of objects to collect.
GC_EXTERNAL_ALLOC means that the the VM is trying to reduce the amount of memory used for collectable objects, to make room for more non-collectable.
Update: There has been a name-change of the first event in later versions of Android. It's now called "GC_FOR_ALLOC".
There is also a new event available, although very rare in modern phones:
GC_BEFORE_OOM means that the system is running really low on memory, and that there is a final GC performed, in order to avoid calling the low memory killer.
Another place where the Dalvik garbage collector messages are explained is in this video: Google I/O 2011: Memory management for Android Apps
At about 14 minutes into the presentation, he breaks down the message format. (BTW, that video has really good info on debugging memory leaks)
Roughly speaking, the format is [Reason] [Amount Freed], [Heap Statistics], [External Memory Statistics], [Pause Time]
Reason
Robert/yuku already gave info on the meaning of these.
Amount Freed
E.g. freed 2125K
Self explanatory
Heap Statistics
E.g. 47% free 6214K/11719K
These numbers reflect conditions after the GC ran. The "47% free" and 6214K reflect the current heap usage. The 11719K represents the total heap size. From what I can tell, the heap can grow/shrink, so you will not necessarily have an OutOfMemoryError if you hit this limit.
External Memory Statistics
E.g external 7142K/8400K
Note: This might only exist in pre-Honeycomb versions of Android (pre 3.0).
Before Honeycomb, bitmaps are allocated external to your VM (e.g. Bitmap.createBitmap() allocates the bitmap externally and only allocates a few dozen bytes on your local heap). Other examples of external allocations are for java.nio.ByteBuffers.
Pause Time
If it's a concurrent GC event, there will be two times listed. One is for a pause before the GC, one is for a pause when the GC is mostly done.
E.g. paused 3ms+5ms
For non-concurrent GC events, there is only one pause time and it's typically much bigger.
E.g. paused 87ms
I also found this in the Android sources, dalvik/vm/alloc/Heap.h. May this be useful.
typedef enum {
/* Not enough space for an "ordinary" Object to be allocated. */
GC_FOR_MALLOC,
/* Automatic GC triggered by exceeding a heap occupancy threshold. */
GC_CONCURRENT,
/* Explicit GC via Runtime.gc(), VMRuntime.gc(), or SIGUSR1. */
GC_EXPLICIT,
/* GC to try to reduce heap footprint to allow more non-GC'ed memory. */
GC_EXTERNAL_ALLOC,
/* GC to dump heap contents to a file, only used under WITH_HPROF */
GC_HPROF_DUMP_HEAP
} GcReason;
I have an application which creates a series of Bitmaps. I am holding these Bitmaps in SoftReferences, to avoid running out of memory. I would like the heap size to grow to the maximum 16MBs before it starts collecting my SoftReferences. However, the SoftReferences are collected very eagerly, prior to growing the heap to it's max. Is there anyway to force the references to be collected less eagerly? or to manually grow the heap to the max? Possibly a way to just start the application with 16mb allocated?
There is currently no way to modify the SoftReference collection behavior.
I'm not sure what you mean by "manually growing the heap to the max".
If these are instances of the Bitmap class (as opposed to some custom bitmap implementation), the pixel data is actually stored on the native heap, but uses an annoying "external allocation" accounting trick, which complicates the situation further. (Notably, the storage is freed by a finalizer, not the GC, and finalizers have to execute in a separate thread after the GC completes. It's possible to allocate Bitmaps until you run out of memory, and the GC has no way to release the memory before throwing an OOM error.)
It's a known issue in Andriod. When VM collects soft references, it either collects all or none, although it should collect 'some' and should not collect anything when there is plenty of free heap.
Check: http://code-gotcha.blogspot.com/2011/09/softreference.html