Can you please tell me how does Dalvik memory fragmentation handling?
For example, if I have a an app allocation some small memory. but free some of them, and then later on trying to allocate a big chunk. How can Dalvik handle such request, with all the 'holes' in the heap?
I need help in understanding this from logcat:
11-15 02:50:21.837 D/dalvikvm( 990): GC_BEFORE_OOM freed 16K, 40% free 9830K/16384K, paused 90ms, total 90ms
11-15 02:50:21.837 E/dalvikvm-heap( 990): Out of memory on a 5032864-byte allocation.
It said my application has use 9830K out of 16384K. And I understand 16M is the maximum heap size, it can't grow any further.
But that should leave 6554K free (16384k - 9830K).
Why dalvik says 'out of memory on a 5032864 byte allocation. That is 4914K which is smaller that I have left.
Thank you.
Take a Look at this, It might help you to understand how to handle dalvik memory fragmentation
Link1
Like most implementations of Garbage Collector (GC), the GC of Dalvik/Java uses double references: reference variables in memory don't point directly toward their objects but toward a second reference which finally point toward the individual objects. Therefore, the GC is free to move around the memory allocations and fuses together the holes when the application need a big memory allocation without impacting in any way the various reference variables allocated by Java.
Related
Can anyone explain the below Logcat message :
D/dalvikvm(4440): GC_EXTERNAL_ALLOC freed 338K, 47% free 6427K/11911K, external 20418K/22446K, paused 53ms
E/dalvikvm-heap(4440): 2519424-byte external allocation too large for this process.
D/dalvikvm(4440): GC_FOR_MALLOC freed <1K, 47% free 6427K/11911K, external 20398K/22446K, paused 40ms
E/GraphicsJNI(4440): VM won't let us allocate 2519424 bytes
In older versions of Android, certain bits of native framework code would tell the VM about native allocations. This "external allocation" mechanism was an ugly hack introduced so that native allocations would cause the Dalvik VM to do a garbage collection pass.
The basic problem was that the Java-language Bitmap object used native memory for the pixel storage. Because the managed-heap objects were tiny, and the native-heap objects were large, you could allocate tons of Bitmaps without causing a GC. This was causing apps to bloat up and the system to slow down.
So, "external allocations" were introduced. Whenever the pixel storage for a Bitmap was allocated on the native heap, an equal amount of memory was deducted from the managed heap. The idea is that, if your heap is filling up with no-longer-reference bitmaps, you'll run out of managed heap space and the GC will fire.
(Unfortunately the GC can't actually release the native storage -- you need to run a finalizer to do that, and finalizers run in a separate pass after the GC completes. For a while the native objects were also holding on to some additional managed-heap objects, so you'd have to GC + finalize + GC to actually clean everything up.)
My "favorite" part about external allocations is that the API was a simple "increase by N" / "decrease by N", which meant there was no way to associate the native heap with a managed heap object, or check for leaks. Because all of the information about the Bitmap was kept in the native object, you couldn't even guess at how much native storage was needed, so it was impossible to look at an hprof dump and figure out how much memory a Bitmap was actually using.
In Android 3.0 ("Honeycomb") the pixel storage was moved onto the managed heap, and the external allocation mechanism was removed.
So, what the log message in your question means is: some code, probably Bitmap, wanted to allocate 2.5MB of native heap, but that would exceed the VM's external allocation heap limit. You need to figure out what's eating up 20MB of external allocation storage and release some of it.
The only way to get information about external allocations is by watching the event log. A few years back I threw a script together (gclog.py -- was in AOSP dalvik/tools for a while). I have no idea if it will still do anything useful. I talk about how to use it in this old thread.
I have an application which is performing some memory intensive tasks. I am trying to figure out what the total available memory is and what the available free memory is. I am doing so by using Runtime.getRuntime().freeMemory() and Runtime.getRuntime().totalMemory()
I am curious if the result I am getting is the total amount of free memory and total memory on the device or if it is the total amount of memory available to that instance of the Dalvik VM that the application is running on top of? I would appreciate some feedback. Thanks!
According to the JavaDocs, those should be in terms of the heap for your process. That being said:
getMemoryClass() on ActivityManager is a more Android-y way to determine the heap size for your process.
Because Dalvik's garbage collector is non-compacting, memory reported by methods like freeMemory() will overstate how much you can allocate. By "non-compacting", I mean that if you free up two blocks of memory that happen to be adjacent, the garbage collector leaves them as two blocks of memory. Contrast this to the Java VM, which will recognize that the two blocks are adjacent and update the heap to reflect one larger free block rather than two smaller free blocks. The non-compacting garbage collector means that your heap is more prone to fragmentation than complete exhaustion. You will try to allocate some large block and get an OutOfMemoryError, not because the heap lacks free memory, but because there is no single free block big enough for your request.
If you see the Android logs, you may see a lot of those things.
What do they mean, knowing those may help us doing better memory allocations.
Example:
28470 dalvikvm D GC_FOR_MALLOC freed 665 objects / 239992 bytes in 71ms
28470 dalvikvm D GC_FOR_MALLOC freed 673 objects / 240288 bytes in 87ms
21940 dalvikvm D GC_EXPLICIT freed 4802 objects / 185320 bytes in 78ms
28470 dalvikvm D GC_FOR_MALLOC freed 666 objects / 240536 bytes in 63ms
GC_FOR_MALLOC means that the GC was triggered because there wasn't enough memory left on the heap to perform an allocation. Might be triggered when new objects are being created.
GC_EXPLICIT means that the garbage collector has been explicitly asked to collect, instead of being triggered by high water marks in the heap. Happens all over the place, but most likely when a thread is being killed or when a binder communication is taken down.
There are a few others as well:
GC_CONCURRENT Triggered when the heap has reached a certain amount of objects to collect.
GC_EXTERNAL_ALLOC means that the the VM is trying to reduce the amount of memory used for collectable objects, to make room for more non-collectable.
Update: There has been a name-change of the first event in later versions of Android. It's now called "GC_FOR_ALLOC".
There is also a new event available, although very rare in modern phones:
GC_BEFORE_OOM means that the system is running really low on memory, and that there is a final GC performed, in order to avoid calling the low memory killer.
Another place where the Dalvik garbage collector messages are explained is in this video: Google I/O 2011: Memory management for Android Apps
At about 14 minutes into the presentation, he breaks down the message format. (BTW, that video has really good info on debugging memory leaks)
Roughly speaking, the format is [Reason] [Amount Freed], [Heap Statistics], [External Memory Statistics], [Pause Time]
Reason
Robert/yuku already gave info on the meaning of these.
Amount Freed
E.g. freed 2125K
Self explanatory
Heap Statistics
E.g. 47% free 6214K/11719K
These numbers reflect conditions after the GC ran. The "47% free" and 6214K reflect the current heap usage. The 11719K represents the total heap size. From what I can tell, the heap can grow/shrink, so you will not necessarily have an OutOfMemoryError if you hit this limit.
External Memory Statistics
E.g external 7142K/8400K
Note: This might only exist in pre-Honeycomb versions of Android (pre 3.0).
Before Honeycomb, bitmaps are allocated external to your VM (e.g. Bitmap.createBitmap() allocates the bitmap externally and only allocates a few dozen bytes on your local heap). Other examples of external allocations are for java.nio.ByteBuffers.
Pause Time
If it's a concurrent GC event, there will be two times listed. One is for a pause before the GC, one is for a pause when the GC is mostly done.
E.g. paused 3ms+5ms
For non-concurrent GC events, there is only one pause time and it's typically much bigger.
E.g. paused 87ms
I also found this in the Android sources, dalvik/vm/alloc/Heap.h. May this be useful.
typedef enum {
/* Not enough space for an "ordinary" Object to be allocated. */
GC_FOR_MALLOC,
/* Automatic GC triggered by exceeding a heap occupancy threshold. */
GC_CONCURRENT,
/* Explicit GC via Runtime.gc(), VMRuntime.gc(), or SIGUSR1. */
GC_EXPLICIT,
/* GC to try to reduce heap footprint to allow more non-GC'ed memory. */
GC_EXTERNAL_ALLOC,
/* GC to dump heap contents to a file, only used under WITH_HPROF */
GC_HPROF_DUMP_HEAP
} GcReason;
I have an application which creates a series of Bitmaps. I am holding these Bitmaps in SoftReferences, to avoid running out of memory. I would like the heap size to grow to the maximum 16MBs before it starts collecting my SoftReferences. However, the SoftReferences are collected very eagerly, prior to growing the heap to it's max. Is there anyway to force the references to be collected less eagerly? or to manually grow the heap to the max? Possibly a way to just start the application with 16mb allocated?
There is currently no way to modify the SoftReference collection behavior.
I'm not sure what you mean by "manually growing the heap to the max".
If these are instances of the Bitmap class (as opposed to some custom bitmap implementation), the pixel data is actually stored on the native heap, but uses an annoying "external allocation" accounting trick, which complicates the situation further. (Notably, the storage is freed by a finalizer, not the GC, and finalizers have to execute in a separate thread after the GC completes. It's possible to allocate Bitmaps until you run out of memory, and the GC has no way to release the memory before throwing an OOM error.)
It's a known issue in Andriod. When VM collects soft references, it either collects all or none, although it should collect 'some' and should not collect anything when there is plenty of free heap.
Check: http://code-gotcha.blogspot.com/2011/09/softreference.html
While running my apps, I have got this kind of logs:
GC_EXTERNAL_ALLOC freed 2K, 38% free 8772K/14087K, external 17480K/17998K, paused 87ms
GC_FOR_MALLOC freed 0K, 38% free 8772K/14087K, external 17480K/17998K, paused 67ms
GC_CONCURRENT freed 2125K, 47% free 6214K/11719K, external 7142K/8400K, paused 3ms+5ms
Does anyone know what these logs mean? Thanks in advance!
What is the difference between 'GC_EXTERNAL_ALLOC', 'GC_FOR_MALLOC' and 'GC_CONCURRENT'. Are there some other 'GC' events?
What does '38% free 8772K/14087K' mean? What is '8772K' and what is '14087K'?
What does 'external 17480K/17998K' mean? What is '17480K' and what is '17998K'?
Thanks!
Another place where the Dalvik garbage collector messages are explained is in this video: Google I/O 2011: Memory management for Android Apps
At about 14 minutes into the presentation, he breaks down the message format. (BTW, that video has really good info on debugging memory leaks)
Roughly speaking, the format is [Reason] [Amount Freed], [Heap Statistics], [External Memory Statistics], [Pause Time]
Reason
Viktor/Robert already explained GC_CONCURRENT, GC_FOR_MALLOC, GC_EXTERNAL_ALLOC.
There is also:
GC_HPROF_DUMP_HEAP - If you dump heap by clicking the "dump heap" button from DDMS or programatically
GC_EXPLICIT - If you call System.gc()
Amount Freed
E.g. freed 2125K
Self explanatory
Heap Statistics
E.g. 47% free 6214K/11719K
These numbers reflect conditions after the GC ran. The "47% free" and 6214K reflect the current heap usage. The 11719K represents the total heap size. From what I can tell, the heap can grow/shrink, so you will not necessarily have an OutOfMemoryError if you hit this limit.
External Memory Statistics
E.g external 7142K/8400K
Note: This might only exist in pre-Honeycomb versions of Android (pre 3.0).
Before Honeycomb, bitmaps are allocated external to your VM (e.g. Bitmap.createBitmap() allocates the bitmap externally and only allocates a few dozen bytes on your local heap). Other examples of external allocations are for java.nio.ByteBuffers.
Pause Time
If it's a concurrent GC event, there will be two times listed. One is for a pause before the GC, one is for a pause when the GC is mostly done.
E.g. paused 3ms+5ms
For non-concurrent GC events, there is only one pause time and it's typically much bigger.
E.g. paused 87ms
I was also looking for this information.
GC stands for garbage-collector, which collects unused objects during runtime of your app.
GC_EXTERNAL_ALLOC: Means that the VM is trying to reduce the amount of memory used collectable objects, to make room for non-collectable objects.
GC_FOR_MALLOC: Means that the GC was triggered because there wasn't enough memory left on the heap to perform an allocation. Might be triggered when new objects are being created.
GC_CONCURRENT: Triggered when the heap has reached a certain amount of objects to collect.
I am guessing that the 38% free means that 38% of the heap is unused. 8772K would be the size of the used memory on the heap and 14087K would be the size of the heap.
I don´t know, sorry.
Please note that all of this information is based on a post from Robert on a similiar question posted here on stackoverflow. All credit (as long as this is correct) goes to Robert.