My Android project gets built fine on a Windows dev machine with lots of RAM. But in a simple Docker container (FROM ubuntu:xenial + Android SDK) gradle build (./gradlew assembleTrunkDebug) fails on 58% while executing the task:
transformClassesWithPreJackPackagedLibrariesForTrunkDebug
Even with --stacktrace --debug the most I get from the error is:
Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)
There are several strange thing that I have noticed:
It doesn't matter what memory arguments I'm passing to the docker container instantiation. It always shows me the same memory stats.
Tasks: 2 total, 1 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1019776 total, 860604 free, 88840 used, 70332 buff/cache
KiB Swap: 1168688 total, 853640 free, 315048 used. 824484 avail Mem
I'm usually doing something like:
docker run -it --volume=/Users/MyUser/code/localDebugRepo:/localDebugRepo --workdir="/localDebugRepo" --memory="2048m" --memory-swap="2048m" 66b48030ee34 /bin/bash
But I have also tried to pass less memory and the same memory-swap or more total memory, but it always shows me Mem: 1019776 and Swap: 1168688.
I also noticed in the task manager that VBoxHeadless.exe uses only 41MB of RAM. While the misfortunate task is running (which takes a long time before crashing), the RAM usage doesn't change, but the Disk transfer is huge and only consumes more and more (since it's on SSD drive).
Android suggests the new build tool Jack and Jill used for the latest sdks has to be allowed to use at least 1536m memory. But the Docker image should run with max of 2048m and having in mind that I wasn't able to shrink the swap, I have the feeling something goes wrong there.
So any thoughts what might have caused this consistent error?
Related
I am using an Azure DevOps Pipeline for CI/CD for a ReactNative Android app.
It has been working great for a while now, but in my latest release the Gradle build is running into the following error:
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:signProductionReleaseBundle'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.FinalizeBundleTask$BundleToolRunnable
> Java heap space
The relevant part of my azure-pipelines.yml file looks like this:
pool:
vmImage: "macos-latest" # I am using this image because I am also doing iOS builds (not sure if it's relevant to the problem)
jobs:
- job: DeployAndroid
steps:
- task: Gradle#2
inputs:
gradleWrapperFile: "MyApp/android/gradlew"
cwd: "MyApp/android"
tasks: "bundleProductionRelease"
publishJUnitResults: false
javaHomeOption: "JDKVersion"
sonarQubeRunAnalysis: false
Java heap space isn't a very descriptive error, but it seems reasonable to assume it is a memory issue. I attempted to increase the max JVM memory by adding the gradleOptions argument:
- task: Gradle#2
inputs:
gradleOptions: "-Xmx3072m"
The default value is -Xmx1024m, so I thought tripling the memory (-Xmx3072m) might work. Unfortunately I am still getting the same error.
Does anyone have any other ideas on how I can fix this error?
I discovered there were several problems that needed to solved to get this to work.
For some reason the gradleOptions argument in azure-pipelines.yml wasn't increasing the memory, even though the documentation shows it being used specifically for the -Xmx flag. (I imagine something about my setup is just overriding it)
Instead I added the following line to my gradle.properties file:
org.gradle.jvmargs=-Xmx10240m -XX:MaxPermSize=4096m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
I found that the MacOS Azure images have the following specs:
Mac pros with a 3 core CPU, 14 GB of RAM, and 14 GB of SSD disk space
So, I set the Xmx flag (max JVM heap memory size) to a relatively high 10GB (10240m).
I don't actually know what XX:MaxPermSize does, but it is also related to memory. The other flags are just related to debugging.
This ended up only being half of my problem, but the other half isn't particularly relevant to my original question.
I originally placed all the blame on Azure, but I realized that I had only run the debug bundle locally, and the production bundle was also failing with the same error on my local machine.
My other problem was specific to React Native. It resulted in the same error though, so I imagine both are somehow related.
The issue was that I had upgraded to a more recent version of React Native, but missed some of the changes that had been made to android/app/build.gradle.
I doubt the specifics of what I happened to be missing would be very helpful to anyone, but if you run into this error, I suggest you check the React Native Upgrade Helper to see if you're missing anything in your build.gradle file.
I'm, currently trying to optimise my build time of an android application I'm developing. Currently it builds for about a minute and a half initial and about a minute for incremental build. I've tried all the recommendation from this page : https://developer.android.com/studio/build/optimize-your-build#optimize
We just managed to get rid of the annotation processors we previously used, but this does not decrease the initial or incremental build times , just gives us the opportunity to use Instant run - with which we previously had a lot of issues , ex. not hot swapping at all.
We made some profiling and found that more than half of the time is taken from the :app:packageProductionDebug task.
Here is a profiler sample of one of my incremental build :
total: 58s
:app:packageProductionDebug 38.933s
:app:transformDexArchiveWithDexMergerForProductionDebug 6.697s
:app:transformClassesWithDexBuilderForProductionDebug 3.833s
:app:compileProductionDebugJavaWithJavac 2.891s
:app:transformClassesWithFirebasePerformancePluginForProductionDebug 1.530s
:app:processProductionDebugResources 1.500s
:app:compileProductionDebugKotlin 1.478s
What is this task doing ? I imagine it is only packaging the previously compiled code into apk. If I'm not wrong, why this task takes 80% of the time ? Can I make something in order to improve this ?
So I found what was causing the package task to run so much time . I was having those properties in gradle.properties file
org.gradle.daemon=true
org.gradle.jvmargs=-Xms1024m -Xmx5000m -Xcheck:jni -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
After removing those properties the package task runs for a second and have overall incremental build time for about 15 seconds. I have no idea why those properties caused such a drastic decrease in the performance of the build , but I don't care, as far as I have 15 seconds build
Nearby I noticed that my Android app often crash because of OOM.
I use Android Studio 3.2.3 Profiler, so I find that native heap are up to 300M.
I want to find the memeory use by every C++ so files, BUT I can not find a tool to accomplish my work.
Any advice is appreciated
adb shell dumpsys mypackage
App Summary
Pss(KB)
------
Java Heap: 11568
Native Heap: 202176
Code: 18576
Stack: 2716
Graphics: 84772
Private Other: 11300
System: 43414
TOTAL USS: 331108
TOTAL: 343727 TOTAL SWAP PSS: 30795
I just updated to Android Studio version 2.0 Preview 8 and gradle plugin 2.0.0-alpha8. I noticed that now there's a warning in case the heap space is not big enough for dexing.
As they say in the proper page (http://tools.android.com/recent)
In 2.0.0-alpha8 we've added some automatic diagnostics for this: if
the build process is too small, we switch back to out-of-memory dexing
and emit a build warning explaining how to bump up the Gradle daemon
size. (We're also working on some additional related improvements for
the next build after alpha8.)
When I went building my project I got the following error:
To run dex in process, the Gradle daemon needs a larger heap. It
currently has 3641 MB. For faster builds, increase the maximum heap
size for the Gradle daemon to more than 4g as specified in
dexOptions.javaMaxHeapSize. To do this, set org.gradle.jvmargs=-Xmx4g
as specified in dexOptions.javaMaxHeapSize in the project
gradle.properties. For more information see
https://docs.gradle.org/current/userguide/build_environment.html
actually, I get a lot of these in a row. Which surprises me because that's what I see in my gradle.properties file (in my project home dir)
ANDROID_BUILD_TARGET_SDK_VERSION=23
ANDROID_BUILD_TOOLS_VERSION=23
ANDROID_BUILD_SDK_VERSION=23
ANDROID_BUILD_MIN_SDK_VERSION=16
org.gradle.jvmargs=-Xmx4g -XX:MaxPermSize=4096m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
also, in the build.gradle of my main module, I put
android {
dexOptions {
incremental true
javaMaxHeapSize "4g"
}
}
so I don't understand where the builder/dexer/compiler is getting that 3641MB value.
Builds are extremely slow
I Disabled instant run and now things are faster. 48-50 seconds for a build. I still get the same error but just once and not many times in a row.
I'm on Linux, my ulimit -a is the following:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31483
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31483
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Also, this is my studio.vmoptions file: https://gist.github.com/MarKco/1ae7918daf867a378a2f I guess it's unedited. I could have a custom one, but I don't know where it could be.
I got it solved, or so it seems.
The last change I had made before I posted the question was the following:
I had
ANDROID_BUILD_TARGET_SDK_VERSION=22
ANDROID_BUILD_TOOLS_VERSION=22
ANDROID_BUILD_SDK_VERSION=22
ANDROID_BUILD_MIN_SDK_VERSION=16
org.gradle.jvmargs=-Xmx4g -XX:MaxPermSize=4096m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
and I changed it to
ANDROID_BUILD_TARGET_SDK_VERSION=23
ANDROID_BUILD_TOOLS_VERSION=23
ANDROID_BUILD_SDK_VERSION=23
ANDROID_BUILD_MIN_SDK_VERSION=16
org.gradle.jvmargs=-Xmx4g -XX:MaxPermSize=4096m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
I don't know why it wasn't changed automatically, since I've been using 23 as target version from a long time. Anyways, after I changed from 22 to 23 I didn't have good builds yet, but that seemed to be due to a gradle sync still running in background. I killed all the gradle projects, kept the version 23 in place of 22, disabled the instant run and performed a new synchronization. At first I kept seing the "To run dex in process, the Gradle daemon needs a larger heap." error, just one time and not repeated as it happened before. But build after build I saw build times getting lower, until I got a 10 seconds build. I guess it's solved. I hope, at least.
P.S. In the end I changed the MaxPermSize accordingly to what Henry said, now my configuration is
ANDROID_BUILD_TARGET_SDK_VERSION=23
ANDROID_BUILD_TOOLS_VERSION=23
ANDROID_BUILD_SDK_VERSION=23
ANDROID_BUILD_MIN_SDK_VERSION=16
org.gradle.jvmargs=-Xmx4g -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
I would like to know how I can troubleshoot high memory usage problems of my app on Android. I've search the internet and found out that the DDMS plugin is useful in taking a memory dump of the heap for my app. This however has been useless.
The app "Usage Timelines Free" is showing 94 MBs of memory used, while the DDMS heap dump shows me a total of 8.4 MBs, with the suspected leaks being the resource files.
When I generate a dump from adb (dumpsys meminfo), I get:
Shared Private Heap Heap Heap
Pss Dirty Dirty Size Alloc Free
------ ------ ------ ------ ------ ------
Native 20 8 20 54588 39431 1892
Dalvik 6732 9952 6396 10756 10028 728
Cursor 0 0 0
Ashmem 184 0 184
Other dev 11462 1172 11384
.so mmap 2467 2072 1156
.jar mmap 0 0 0
.apk mmap 48 0 0
.ttf mmap 2 0 0
.dex mmap 1037 0 0
Other mmap 41 16 32
Unknown 46352 292 46348
TOTAL 68345 13512 65520 65344 49459 2620
How can I know what is behind this huge memory consumption. My app is a foreground monitor service which runs forever, with a few activities which are accessed a few times per day.
Thanks.
Minimize your android data usage. Check the link :http://engineroots.games4punk.com/minimize-your-android-data-usage/
DDMS and got a heap memory dump. It only showed 8.4 MBs being used!
Then you are only using 8.4MB of heap space after a complete garbage collection (a net effect of creating the heap dump).
Note that if you ran this on Android 1.x/2.x, MAT will not report space consumed by bitmaps. Always try to dump the heap from an Android 3.0+ device or emulator.
When I generate a dump from adb (dumpsys meminfo), I get:
My guess is that you are running this on an Android 1.x/2.x device or emulator, and you have a lot of bitmap memory. Try running your heap dump and other tests on an Android 3.0+ environment.
Also, bear in mind that adb dumpsys meminfo does not perform a complete garbage collection, whereas dumping the HPROF file does. The Android garbage collector is optimized to minimize CPU utilization and therefore does not attempt to perform a complete garbage collection. Hence, at runtime, the heap is usually filled with garbage that will be reclaimed, as needed, by the GC engine.
You might also wish to read Dianne Hackborn's essay on this subject.