Develop Reference

QNetworkAccessManager - first GET very slow

I have a problem with using the QNetworkAccessManager in Qt 5.5 on android. Downloading a simple, small graphic file via http GET results in a lot of garbage collection calls and a lockup of the UI during that time. Subsequent GETs work flawlessly and without these GC calls. The code is as follows:
void DownloadManager::downloadFile(QUrl fromUrl, QString toFilePath) {
_currentFilePath = toFilePath;
QNetworkRequest request;
request.setUrl(fromUrl);
qDebug() << "before";
_currentReply = _mgr.get(request);
qDebug() << "after";
connect(_currentReply, SIGNAL(error(QNetworkReply::NetworkError)), this, SLOT(downloadError(QNetworkReply::NetworkError)));
connect(_currentReply, SIGNAL(downloadProgress(qint64, qint64)), this, SLOT(downloadProgress(qint64,qint64)));
connect(_currentReply, SIGNAL(finished()), this, SLOT(downloadFinished()));
}
DownloadManager is a custom QObject-derived class without any special features that are relevant to the get request. _mgr is a QNetworkAccessManager Object that's allocated during DownloadManagers cTor.
As you can see, this is just a textbook example of a get request, nothing too fancy about it. And as I said: it works, for the most part. Only the first get request ends up like this:
D/ .../downloadmanager.cpp:61 (void DownloadManager::downloadFile(QUrl, QString)): before
D/dalvikvm(13298): GC_CONCURRENT freed 2290K, 25% free 10911K/14407K, paused 2ms+3ms, total 29ms
D/dalvikvm(13298): GC_CONCURRENT freed 501K, 25% free 10884K/14407K, paused 13ms+2ms, total 35ms
D/dalvikvm(13298): GC_CONCURRENT freed 524K, 25% free 10892K/14407K, paused 12ms+3ms, total 36ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 6ms
D/dalvikvm(13298): GC_CONCURRENT freed 537K, 25% free 10887K/14407K, paused 2ms+9ms, total 32ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 14ms
D/dalvikvm(13298): GC_CONCURRENT freed 840K, 25% free 10899K/14407K, paused 12ms+3ms, total 38ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 11ms
D/dalvikvm(13298): GC_CONCURRENT freed 1294K, 25% free 10901K/14407K, paused 2ms+2ms, total 27ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 11ms
D/dalvikvm(13298): GC_CONCURRENT freed 1187K, 22% free 11330K/14407K, paused 2ms+2ms, total 30ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 15ms
D/dalvikvm(13298): GC_CONCURRENT freed 1459K, 19% free 11919K/14535K, paused 13ms+4ms, total 64ms
D/dalvikvm(13298): WAIT_FOR_CONCURRENT_GC blocked 18ms
D/ .../downloadmanager.cpp:65 (void DownloadManager::downloadFile(QUrl, QString)): after
I simply don't understand what causes that much GC to happen - it takes about a full to one and a half seconds for everything to work out (for a download that should take a split-second and, moreover, be asynchronus and not locking up the UI).
Additional Information:
It is always only the first download that triggers this. Subsequent downloads, even for the exact same file, work flawlessly
It doesn't matter if there's a file at the exact location with the exact name or not. Downloading the file, deleting it, getting back into the application and redownloading it provides the same results - the first get is slow and has the GC, the second works perfectly fine.
I call all that from a QML File, that causes a singleton c++ object to call DownloadManager::downloadFile.
Other than the QML UI, nothing else is running within the application. No heavy data exchanges, no background loading on other threads, nothing.
I'd be thankful for any pointers towards solving this.

I didn't try on Android, but I had the same issue on Windows. Because those are the same symptoms, I would say this is likely the same reason, which is that the implementation is lazily loading some shared library on the first get() call. This is particularly true when using an encrypted connection; in my case I can see in Visual Studio that 19 DLLs are loaded on the first get() call.
One way to work around that is to pre-connect to the server using connectToHost or connectToHostEncrypted, depending on whether you are using an encrypted connection (e.g. HTTPS) or not. I am calling that at application startup, but anytime the UI is idle should be fine. Then subsequent get() calls will all have the same performance, including the first one, since the libraries have been loaded and the connection is already established. I am assuming that connecting to any server will load the libraries.
See https://forum.qt.io/topic/65201/qnetworkaccessmanager-first-get-very-slow/14 for details on the generic error (not specific to Android).

Why so many GC_FOR_ALLOC in a simple app?

I'm getting way too many GC_FOR_ALLOC from the dalvikvm.
I'm getting XML from a REST service: in one activity I parse about 100 lines programatically(me) and in the other activity I use the SimpleXML to parse about 200 lines.
In the first one I get 50 GC_FOR_ALLOC.
In the second one I get like 300!! (I can't even post it all, the body makes 29579 characters and it's allowed only 30k)
I've searched and almost everyone complains about gc_for_"M"alloc and not gc_for_"A"lloc.
Is the SimpleXML the problem because the instances created?
I'll post the logcat dump by dalvikvm, maybe the values have some information.
Thank you very much for your help.
12-11 06:13:49.564: D/dalvikvm(6759): GC_FOR_ALLOC freed 362K, 13% free 4116K/4688K, paused 181ms, total 182ms
12-11 06:13:50.074: D/dalvikvm(6759): GC_FOR_ALLOC freed 303K, 13% free 4134K/4708K, paused 142ms, total 142ms
.... repeated many times .....
12-11 06:14:06.254: D/dalvikvm(6759): GC_FOR_ALLOC freed 73K, 13% free 4159K/4768K, paused 53ms, total 53ms
12-11 06:14:06.314: D/dalvikvm(6759): GC_FOR_ALLOC freed 103K, 13% free 4159K/4768K, paused 56ms, total 57ms
12-11 06:14:06.374: D/dalvikvm(6759): GC_FOR_ALLOC freed 29K, 12% free 4203K/4768K, paused 54ms, total 54ms
12-11 06:14:06.424: D/dalvikvm(6759): GC_FOR_ALLOC freed 73K, 13% fre

You can see the most-recently-allocated objects using the DDMS Allocation Tracker (memory debugging docs, old blog post, ddms docs). This will show you what's being allocated and give you a stack trace for the place where the allocation is being performed.
Another blog post describes MAT and other relevant tools, though heap-dump analysis is less useful for this sort of problem because it generally shows you the objects that haven't been freed, and you're more interested in the objects that are being freed.

In Android Dalvik VM, GC_FOR_ALLOC is inovked in object alloc step when dlmalloc footprint space is NOT enough for new or heap->bytesAllocated + n > hs->softLimit. You can set dalvik.system.setTargetHeapUtilization lower for more free heap space.

you can use MAT MAT tutorial
to find how many object are creating and garbage collected. so that youcan optimize your code

If you get that multiple GC_FOR_ALLOC while your app is lagging, there is a big possibility that the bug is in a loop. Check where the line of code starts to trigger the GC then start tracing the code from there. In my experience, I mistyped my inner loop iterator which causes the program to make an infinite loop. I created a bug like this:
for(int i=0; i<list.size(); i++) {
for(int j=i+1 j<list.size(); i++) {
// I mistyped the iterator of integer j with i
// making an infinite loop which triggered the GC.
//appears many times
}
}

I encounter the same problem today.
I find a not ended loop in my code such as while(i < xx), but I not implement the i++ statement in the while body.
So the messages like you meet appeared.
Check your code firstly please.

My log:
D/dalvikvm: GC_FOR_ALLOC freed 549K, 9% free 7878K/8596K, paused 30ms, total 34ms
...freed 539K, 9% free 7888K/8596K, paused 30ms, total 30ms
...freed 1856K, 21% free 8083K/10108K, paused 51ms, total 51ms
...freed 582K, 9% free 7845K/8596K, paused 38ms, total 38ms
Explain:
When your app get memory more limit per app. Dalvik/Ant call garbage collector.
What limits memory for your App decide Dalvik/Ant. As you see for my app Dalvik decide 8596K(double case) and 8083K(one case).
And limits change in runtime.
And you can not be sure when this happens. But you can reduce the likelihood. Decreasing the amount of memory that your application consumes.
PS: Decide when call GC teakes Dalvik/Ant. And you can not be sure when this happens. But you can reduce the likelihood. Decreasing the amount of memory that your application consumes.
PS: In "Monitor android" see tab "Monitors", graphics "Memory". And use buttons: "pause(enabled)", Initiate GC, "Dump Java Heap" "Start Alocation Tracking(very useful)". And use official guide for this:
https://developer.android.com/studio/profile/am-memory.html?utm_source=android-studio.
As far as I understand App must not stop/pause working or crashes when VM call GC.

other packages also run on my android app

I have to run my android application which means am getting the following messages on my logcat window:
12:05:36.056: System_process I/ActivityManager(59): Displayed activity com.xmlparsing/.MainActivity: 581 ms (total 581 ms)
12:05:41.136: com.android.defcon D/dalvikvm(253): GC_EXPLICIT freed 152 objects / 11024 bytes in 60ms
12:05:46.126: com.svox.pico D/dalvikvm(264): GC_EXPLICIT freed 31 objects / 1512 bytes in 50ms
12:05:51.156: com.android.setting D/dalvikvm(125): GC_EXPLICIT freed 975 objects / 42392 bytes in 78ms
12:06:40.666: com.xmlparsing D/dalvikvm(1131): GC_FOR_MALLOC freed 3884 objects / 321368 bytes in 50ms
12:06:40.836: com.xmlparsing D/dalvikvm(1131): GC_FOR_MALLOC freed 3654 objects / 427296 bytes in 48ms
12:06:41.026: com.xmlparsing D/dalvikvm(1131): GC_FOR_MALLOC freed 4007 objects / 418288 bytes in 45ms
12:06:41.245: com.xmlparsing D/dalvikvm(1131): GC_FOR_MALLOC freed 4516 objects / 414608 bytes in 54ms
12:06:41.456: com.xmlparsing D/dalvikvm(1131): GC_FOR_MALLOC freed 4194 objects / 399504 bytes in 53ms
12:06:41.616: com.xmlparsing D/dalvikvm(1131): GC_FOR_MALLOC freed 3365 objects / 412104 bytes in 43ms
12:06:41.666: com.xmlparsing I/System.out(1131): category size6
12:06:41.666: com.xmlparsing I/System.out(1131): category name is---------------->Photos
Here my application package is com.xmlparsing.
But here other packages(com.android.defcon,com.svox.pico,com.android.settings) is running on starting.this packages only taking more time to reading(loading) the data.please check the time.my xml file is read within second.why that package is loading.how can i control it.please give me solution for these

Those packages are not running "on your app". Android supports multitasking, so it is possible to have more than one app alive at a time.
In any case, the LogCat you posted shows that it is simply a GC call, which means Android is trying to free up memory. There is nothing you can do to control this.
If you want to view LogCay messages from only your app, try filtering by process id.

What you're seeing are log messages from the different processes that are going through Garbage Collection (GC).
On any system, Android or otherwise, there will likely be multiple processes running at the same time. On Android, the processes will periodically go through Garbage Collection to free up memory and keep the system running well and within its constrained resources. This is expected behavior.

Android Bitmap Memory Use

I am running a stack of image filters and seem to be hitting some memory issues.
At the beginning of the image processing I am using this much memory:
GC_FOR_MALLOC freed 3K, 45% free 3237K/5831K, external 47586K/49634K, paused 17ms
At the end I am using this much (after all processing is finished):
GC_EXTERNAL_ALLOC freed 5K, 16% free 16056K/18951K, external 51430K/52196K, paused 23ms
After I am finished with each bitmap I set it to recycle and to null:
someBitmap.recycle();
someBitmap = null;
Is there anything else I should be doing to them? Is there any cleanup I should do to the Canvas being used?
Also my filters are objects instantiated like:
BoxBlurFilter blurFilter = new BoxBlurFilter();
Is there anything I should do to release them? In iOS memory allocated with "new" I am responsible to free.
Sorry for the trivial memory management questions, but I am quite new to Android dev and things are quite different than iOS.
Thank you!
EDIT 2, I removed my full filter code.

So after setting the filter instances to null and all the byte arrays to null (in the code above and in the filter objects) I now have approximately the same size heap as I did before I run the filter:
GC_EXPLICIT freed 5126K, 77% free 3243K/13703K, external 51430K/53478K, paused 18ms
That means it went from 16mb being used to 3.2mb. Much better!
So I guess the answer to my question is make sure you set everything to null if you want it to be freed.

OutOfMemory error though free memory is available

I'm seeing a pretty odd problem. Essentially sometimes large bitmap memory allocations will fail even though there's apparently tons of memory. There are a number of posts that appear to ask a similar question but they are all related to pre-honeycomb android. My understanding is that images are allocated on heap now, instead of some outside memory. Anyway, please look at this log below:
10-14 13:43:53.020: INFO/dalvikvm-heap(31533): Grow heap (frag case) to 40.637MB for 942134-byte allocation
10-14 13:43:53.070: DEBUG/dalvikvm(31533): GC_FOR_ALLOC freed 126K, 11% free 41399K/46343K, paused 31ms
10-14 13:43:53.130: DEBUG/dalvikvm(31533): GC_FOR_ALLOC freed 920K, 13% free 40478K/46343K, paused 30ms
10-14 13:43:53.180: DEBUG/dalvikvm(31533): GC_FOR_ALLOC freed 1026K, 13% free 40479K/46343K, paused 30ms
10-14 13:43:53.250: DEBUG/dalvikvm(31533): GC_FOR_ALLOC freed 931K, 12% free 41193K/46343K, paused 31ms
10-14 13:43:53.250: INFO/dalvikvm-heap(31533): Grow heap (frag case) to 41.313MB for 1048592-byte allocation
10-14 13:43:53.280: DEBUG/dalvikvm(31533): GC_FOR_ALLOC freed <1K, 11% free 42217K/47431K, paused 31ms
10-14 13:44:01.520: DEBUG/dalvikvm(31533): GC_CONCURRENT freed 3493K, 15% free 40646K/47431K, paused 3ms+9ms
10-14 13:44:08.130: DEBUG/dalvikvm(31533): GC_EXPLICIT freed 16461K, 47% free 25527K/47431K, paused 3ms+6ms
10-14 13:44:09.150: DEBUG/dalvikvm(31533): GC_FOR_ALLOC freed 1007K, 45% free 26191K/47431K, paused 35ms
10-14 13:44:09.160: INFO/dalvikvm-heap(31533): Grow heap (frag case) to 29.334MB for 3850256-byte allocation
10-14 13:44:09.200: DEBUG/dalvikvm(31533): GC_CONCURRENT freed 0K, 37% free 29951K/47431K, paused 2ms+4ms
10-14 13:44:11.970: DEBUG/dalvikvm(31533): GC_FOR_ALLOC freed 1878K, 38% free 29784K/47431K, paused 37ms
10-14 13:44:12.410: DEBUG/dalvikvm(31533): GC_FOR_ALLOC freed 62K, 36% free 30441K/47431K, paused 32ms
10-14 13:44:12.440: DEBUG/dalvikvm(31533): GC_FOR_ALLOC freed <1K, 32% free 32325K/47431K, paused 32ms
10-14 13:44:12.440: INFO/dalvikvm-heap(31533): Forcing collection of SoftReferences for 3850256-byte allocation
10-14 13:44:12.480: DEBUG/dalvikvm(31533): GC_BEFORE_OOM freed 124K, 33% free 32200K/47431K, paused 37ms
10-14 13:44:12.480: ERROR/dalvikvm-heap(31533): Out of memory on a 3850256-byte allocation.
I apologise for including so much logging, I hope it's relevant. The way I read it is that the system continuously readjusts heap size until it eventually reaches heap max. Then, we request an especially large allocation that fails. Clearly there is more than enough memory available (about 15 megs). Does this mean that heap is internally fragmented and there are no contiguous memory segments large enough to handle our allocation? If that's the case what should I do? If that's not it, then what?
Thanks in advance.

The weird behavior is because bitmaps are allocated on the native heap and not on the garbage collected, but android can only track objects on the garbage collected heap. From Android 2.2 (or 2.3 maybe) this has changed and allocated bitmaps are visible too if you make a heap dump.
Back to the question, your problem is most probably that the bitmaps you loaded manually are not freed appropriately. One typical problem is that some callback remains set or the view is still referring the bitmap.
The other common problem is that if you load big bitmaps manually (not as a resource), you will need to call recycle() on them when you don't need it anymore, which will free the bitmap from the native memory so the garbage collector will be able to its work as it should. (The GC only sees objects on the GC heap, and doesn't no which object to free to free memory from the native heap, and actually doesn't even care about it.)
I have this little baby at hand all the time:
public static void stripImageView(ImageView view) {
if ( view.getDrawable() instanceof BitmapDrawable ) {
((BitmapDrawable)view.getDrawable()).getBitmap().recycle();
}
view.getDrawable().setCallback(null);
view.setImageDrawable(null);
view.getResources().flushLayoutCache();
view.destroyDrawingCache();
}

The images are fetched from the Web, each ranging from 300K to 500K in
size, and stored in an arrayList of Drawables.
The kb file size of the image you're loading from the web isn't directly relevant. Since they're converted into bitmaps you need to calculate width * height * 4 bytes per image for regular ARGB images. (width and height in px).
The bitmaps consume native heap, which usually doesn't show in a hprof. The hprof should only show you the number of objects, i.e. BitmapDrawables or Bitmaps that are left.
I use this code in my app to output the current used memory used by the app and native heap:
public static void logHeap(Class clazz) {
Double allocated = new Double(Debug.getNativeHeapAllocatedSize())/new Double((1048576));
Double available = new Double(Debug.getNativeHeapSize())/1048576.0);
Double free = new Double(Debug.getNativeHeapFreeSize())/1048576.0);
DecimalFormat df = new DecimalFormat();
df.setMaximumFractionDigits(2);
df.setMinimumFractionDigits(2);
Log.d(APP, "debug. =================================");
Log.d(APP, "debug.heap native: allocated " + df.format(allocated) + "MB of " + df.format(available) + "MB (" + df.format(free) + "MB free) in [" + clazz.getName().replaceAll("com.myapp.android.","") + "]");
Log.d(APP, "debug.memory: allocated: " + df.format(new Double(Runtime.getRuntime().totalMemory()/1048576)) + "MB of " + df.format(new Double(Runtime.getRuntime().maxMemory()/1048576))+ "MB (" + df.format(new Double(Runtime.getRuntime().freeMemory()/1048576)) +"MB free)");
System.gc();
System.gc();
// don't need to add the following lines, it's just an app specific handling in my app
if (allocated>=(new Double(Runtime.getRuntime().maxMemory())/new Double((1048576))-MEMORY_BUFFER_LIMIT_FOR_RESTART)) {
android.os.Process.killProcess(android.os.Process.myPid());
}
}
which I call when starting or finishing an activity during development.
logHeap(this.getClass());
Here are some informative links - generally there are lots of threads about this topic on here.
Bitmaps in Android
Android: Eclipse MAT does not appear to show all my app's objects
Here's also a useful slide by Romain Guy (Android Framework engineer) about soft references, weak references, simple caches, image handling: http://docs.huihoo.com/google/io/2009/Th_0230_TurboChargeYourUI-HowtomakeyourAndroidUIfastandefficient.pdf

The system doesn't readjust heap sizes. Your app has a (semi-) predefined heap space. Heap sizes can be adjusted in some custom ROMs. This isn't really germane to your problem though.
The order of the garbage collection you're seeing seems to suggest that at the time of the GC_BEFORE_OOM happens your app's running out of its heap space. The stats reported by the Dalvik logger could be system-wide.

yes the error "out of Memory" occur when we are working with big size Bitmap. for that i have an solution, with Decoding of the image we can easily avoid this error.
BitmapFactory.Options o = new BitmapFactory.Options();
o.inJustDecodeBounds = true;
FileInputStream fis = new FileInputStream(files.getAbsolutePath());
BitmapFactory.decodeStream(fis, null, o);
fis.close();
final int REQUIRED_SIZE=70;
int width_tmp=o.outWidth, height_tmp=o.outHeight;
int scale=1;
while(true){
if(width_tmp/2<REQUIRED_SIZE || height_tmp/2<REQUIRED_SIZE)
break;
width_tmp/=2;
height_tmp/=2;
scale*=2;
}
BitmapFactory.Options op = new BitmapFactory.Options();
op.inSampleSize = scale;
fis = new FileInputStream(files.getAbsolutePath());
bitmap[i] = BitmapFactory.decodeStream(fis, null, op);
fis.close();