Develop Reference

Error in building android image, fails with jack server

I am trying to build an android image, the build system by default uses jack server which keeps on failing again and again.
Can any body tell me how can i fix it.
I get the following error during build
FAILED: setup-jack-server /bin/bash -c
"(prebuilts/sdk/tools/jack-admin install-server
prebuilts/sdk/tools/jack-launcher.jar
prebuilts/sdk/tools/jack-server-4.11.ALPHA.jar 2>&1 || (exit 0) ) &&
(JACK_SERVER_VM_ARGUMENTS="-Dfile.encoding=UTF-8
-XX:+TieredCompilation" prebuilts/sdk/tools/jack-admin start-server 2>&1 || exit 0 ) && (prebuilts/sdk/tools/jack-admin update server
prebuilts/sdk/tools/jack-server-4.11.ALPHA.jar 4.11.ALPHA 2>&1 || exit
0 ) && (prebuilts/sdk/tools/jack-admin update jack
prebuilts/sdk/tools/jacks/jack-4.32.CANDIDATE.jar 4.32.CANDIDATE ||
exit 47 )" Writing client settings in /home/test/.jack-settings
Installing jack server in "/home/test/.jack-server"
Warning: The JKS keystore uses a proprietary format. It is recommended
to migrate to PKCS12 which is an industry standard format using
"keytool -importkeystore -srckeystore
/home/test/.jack-server/server.jks -destkeystore
/home/test/.jack-server/server.jks -deststoretype pkcs12".
Warning: The JKS keystore uses a proprietary format. It is recommended
to migrate to PKCS12 which is an industry standard format using
"keytool -importkeystore -srckeystore
/home/test/.jack-server/client.jks -destkeystore
/home/test/.jack-server/client.jks -deststoretype pkcs12".
Communication error with Jack server (58), try 'jack-diagnose' or see
Jack server log Failed to contact Jack server: Problem reading
/home/test/.jack-server/client.pem. Try 'jack-diagnose' Failed to
contact Jack server: Problem reading
/home/test/.jack-server/client.pem. Try 'jack-diagnose'
Jack server diagnose also dosen't give any usefull information
./prebuilts/sdk/tools/jack-diagnose error: process ID list syntax
error
Usage: ps [options]
Try 'ps --help <simple|list|output|threads|misc|all>' or 'ps --help
<s|l|o|t|m|a>' for additional help text.
For more details see ps(1). Port 8077 is used by another process
(pid=), please ensure to free the port or change port configuration in
'/home/test/.jack-settings' and
'/home/test/.jack-server/config.properties' error: process ID list
syntax error
Usage: ps [options]
Try 'ps --help <simple|list|output|threads|misc|all>' or 'ps --help
<s|l|o|t|m|a>' for additional help text.
For more details see ps(1). Port 8076 is used by another process
(pid=), please ensure to free the port or change port configuration in
'/home/test/.jack-settings' and
'/home/test/.jack-server/config.properties'
and jack program throws an error for open SSL connect.
SSL error when connecting to the Jack server, retrying with debug
Trying 127.0.0.1...
TCP_NODELAY set
Connected to localhost (127.0.0.1) port 8076 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /home/test/.jack-server/server.pem CApath: /etc/ssl/certs } [5 bytes data]
TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data]
OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8076
Closing connection 0
Can anybody please help me out in resolving this. I am struggling to get a clean build.

When Jack diagnose gives error 35 (SSL error) the following steps worked for me
Edit java.security file under /etc/java-8-openjdk/security/
Remove TLSv1, TLSv1.1 from jdk.tls.disabledAlgorithms
Restart the jack server:
cd <aosp-home-dir>/prebuilts/sdk/tools/
./jack-admin kill-server
./jack-admin start-server

I have accidentally uninstalled jack server while building Android AOSP

I am building the Android code for Android Go on my ubuntu 14.04 machine.
While building I got some problems with jack server and I ended up uninstalling the jack server (accidentally).
How do I install the Jack Server again ?
I have tried to install the jack server using the following command:
jack-admin install-server jack-launcher.jar jack-server-4.11.ALPHA.jar
However, I get an error:
Jack server installation not found
Kindly help how do I install Jack Server again.
I have also followed guides from Google Search Result but it has not helped so far.
Update -1
After trying the answer from #gaoc I get the following error each time I try to make the build. :
[ 0% 19/82490] Ensuring Jack server is installed and started
FAILED: setup-jack-server
/bin/bash -c "(prebuilts/sdk/tools/jack-admin install-server prebuilts/sdk/tools/jack-launcher.jar prebuilts/sdk/tools/jack-server-4.11.ALPHA.jar 2>&1 || (exit 0) ) && (JACK_SERVER_VM_ARGUMENTS=\"-Dfile.encoding=UTF-8 -XX:+TieredCompilation\" prebuilts/sdk/tools/jack-admin start-server 2>&1 || exit 0 ) && (prebuilts/sdk/tools/jack-admin update server prebuilts/sdk/tools/jack-server-4.11.ALPHA.jar 4.11.ALPHA 2>&1 || exit 0 ) && (prebuilts/sdk/tools/jack-admin update jack prebuilts/sdk/tools/jacks/jack-4.32.CANDIDATE.jar 4.32.CANDIDATE || exit 47 )"
Jack server already installed in "/home/c_sganig/.jack-server"
Communication error with Jack server (3), try 'jack-diagnose' or see Jack server log
Communication error with Jack server 3. Try 'jack-diagnose'
Communication error with Jack server 3. Try 'jack-diagnose'
[ 0% 34/82490] build out/target/product/msm8909go/emmc_appsboot.mbn
make: Entering directory `/home/c_sganig/code/LA.UM.6.7.r1/bootable/bootloader/lk'
including app/aboot dev/fbcon dev/gcdb/display dev/keys dev/pmic/pm8x41 dev/pmic/pmi8994 dev/qpnp_haptic dev/vib lib/debug lib/heap lib/libc lib/libfdt lib/openssl lib/ptable
including lib/openssl/crypto lib/zlib_inflate
make[1]: Entering directory `/home/c_sganig/code/LA.UM.6.7.r1/bootable/bootloader/lk'
generating ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/config.h
generating ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/system-onesegment.ld
linking ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk
../../../prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/bin/arm-eabi-ld: warning: /home/c_sganig/code/LA.UM.6.7.r1/prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/bin/../lib/gcc/arm-eabi/4.8/libgcc.a(bpabi.o) uses 32-bit enums yet the output is to use variable-size enums; use of enum values across objects may fail
../../../prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/bin/arm-eabi-ld: warning: /home/c_sganig/code/LA.UM.6.7.r1/prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/bin/../lib/gcc/arm-eabi/4.8/libgcc.a(_divdi3.o) uses 32-bit enums yet the output is to use variable-size enums; use of enum values across objects may fail
../../../prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/bin/arm-eabi-ld: warning: /home/c_sganig/code/LA.UM.6.7.r1/prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/bin/../lib/gcc/arm-eabi/4.8/libgcc.a(_udivdi3.o) uses 32-bit enums yet the output is to use variable-size enums; use of enum values across objects may fail
generating image: ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk.bin
generating listing: ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk.lst
generating symbols: ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk.sym
generating listing: ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk.debug.lst
text data bss dec hex filename
367924 200272 202520 770716 bc29c ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk
../../../prebuilts/gcc/linux-x86/arm/arm-eabi-4.8/bin/arm-eabi-objcopy -O binary ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk.bin
generating size map: ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk.size
generating stripped elf: ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk_s.elf
cp -f ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/build-msm8909/lk_s.elf ../../../out/target/product/msm8909go/obj/EMMC_BOOTLOADER_OBJ/../../emmc_appsboot.mbn
make[1]: Leaving directory `/home/c_sganig/code/LA.UM.6.7.r1/bootable/bootloader/lk'
make: Leaving directory `/home/c_sganig/code/LA.UM.6.7.r1/bootable/bootloader/lk'
ninja: build stopped: subcommand failed.
11:47:18 ninja failed with: exit status 1
#### failed to build some targets (45 seconds) ####

Show the list of removed packages:
$ awk '$3 == "remove" { print $1, $2, $4 }' /var/log/dpkg.log | tee list
If you can find the removed package in the list, then reinstall:
$ sudo apt-get install --reinstall nameofpackage

Following is the solution that helped me with jack server issue:
Delete ~/.jack-server directory and ~/.jack.settings file. This will be recreated when we run make command, so nothing to worry about that.
Sync repo again using this command (takes about 30-40 minutes):
repo sync -c --no-clone-bundle --no-tags -j4
This -c option with sync command will tell the git about branch you really need instead of all repositories. -jn (-j4 in my case) , here n refers to the number of threads you want.
Now build environment , lunch and make as usual.
I will also share another link to Google Group which has some significant information regarding this issue. : https://issuetracker.google.com/issues/37070263

AOSP Build error: EXECUTABLES/iw_intermediates/version.c

I've made a AOSP build 7.1.1 couple of days ago and it works well. Today I decided to make a clean build (make clean) and start working on it. After that when I'm trying to compile by make otapackage after some time it gives me below error.
[ 44% 7552/17136] build out/target/product/hikari/gen/EXECUTABLES/iw_intermediates/version.c
FAILED: /bin/bash -c "external/iw/version.sh out/target/product/hikari/gen/EXECUTABLES/iw_intermediates/version.c"
fatal: No names found, cannot describe anything.
ninja: build stopped: subcommand failed.
build/core/ninja.mk:148: recipe for target 'ninja_wrapper' failed
make: *** [ninja_wrapper] Error 1
#### make failed to build some targets (01:39:34 (hh:mm:ss)) ####
Everytime, its the same. Any solution ?

Its failing when it is trying to check the git version number.
Inside external/iw/version.sh, it is failing when it gets to:
descr=$(git describe --match=v*):
if [ -d .git ] && head=`git rev-parse --verify HEAD 2>/dev/null`; then
git update-index --refresh --unmerged > /dev/null
descr=$(git describe --match=v*)
# on git builds check that the version number above
# is correct...
[ "${descr%%-*}" = "v$VERSION" ] || exit 2
v="${descr#v}"
if git diff-index --name-only HEAD | read dummy ; then
v="$v"-dirty
fi
else
v="$VERSION"
fi
echo '#include "iw.h"' > "$OUT"
echo "const char iw_version[] = \"$v\";" >> "$OUT"
I had the same problem and there is two ways to go about it:
You can comment out everything except v="$VERSION" and the last two
echo lines.But this is a temporary fix.
Another way to solve this is by doing a repo sync before you execute make. I fixed it by simply doing a repo sync and that updated and
fixed everything for me.

Can't compile CyanogenMod 13

I'm trying to compile CyanogenMod from scratch with the help of this guide. I've done most of the steps without any errors, but at the brunch angler step, I keep getting this error:
Building with Jack: /home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/android-support-v13-ics-mr1_intermediates/classes.jack
host Executable: llvm-rs-cc (/home/hexafluoride/android/system/out/host/linux-x86/obj/EXECUTABLES/llvm-rs-cc_intermediates/llvm-rs-cc)
Building with Jack: /home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/com.android.gallery3d.common2_intermediates/classes.jack
target Symbolic: libssl (/home/hexafluoride/android/system/out/target/product/angler/symbols/system/lib64/libssl.so)
target StaticLib: libLLVMAArch64CodeGen (/home/hexafluoride/android/system/out/target/product/angler/obj/STATIC_LIBRARIES/libLLVMAArch64CodeGen_intermediates/libLLVMAArch64CodeGen.a)
Launching background server java -Dfile.encoding=UTF-8 -Xms2560m -XX:+TieredCompilation -jar /home/hexafluoride/android/system/out/host/linux-x86/framework/jack-launcher.jar -cp /home/hexafluoride/android/system/out/host/linux-x86/framework/jack.jar com.android.jack.server.JackSimpleServer
target StaticLib: libLLVMAArch64Info (/home/hexafluoride/android/system/out/target/product/angler/obj/STATIC_LIBRARIES/libLLVMAArch64Info_intermediates/libLLVMAArch64Info.a)
Launching background server java -Dfile.encoding=UTF-8 -Xms2560m -XX:+TieredCompilation -jar /home/hexafluoride/android/system/out/host/linux-x86/framework/jack-launcher.jar -cp /home/hexafluoride/android/system/out/host/linux-x86/framework/jack.jar com.android.jack.server.JackSimpleServer
target StaticLib: libLLVMAArch64Desc (/home/hexafluoride/android/system/out/target/product/angler/obj/STATIC_LIBRARIES/libLLVMAArch64Desc_intermediates/libLLVMAArch64Desc.a)
target StaticLib: libLLVMAArch64AsmParser (/home/hexafluoride/android/system/out/target/product/angler/obj/STATIC_LIBRARIES/libLLVMAArch64AsmParser_intermediates/libLLVMAArch64AsmParser.a)
target StaticLib: libLLVMAArch64AsmPrinter (/home/hexafluoride/android/system/out/target/product/angler/obj/STATIC_LIBRARIES/libLLVMAArch64AsmPrinter_intermediates/libLLVMAArch64AsmPrinter.a)
ERROR: /home/hexafluoride/android/system/packages/apps/Gallery2/gallerycommon/src/com/android/gallery3d/exif/ExifData.java:1: The type java.lang.Object cannot be found in source files, imported jack libs or the classpath
ERROR: /home/hexafluoride/android/system/packages/apps/Gallery2/gallerycommon/src/com/android/gallery3d/exif/ExifData.java:19: The import android cannot be resolved
* lots of errors saying that basic types can't be resolved after this *
And then make fails. I've tried doing make showcommands, and this is the output:
if [ -s /home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/android-support-v4-jellybean_intermediates/jack-rsc/java-source-list-uniq ] ; then export tmpEcjArg="#/home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/android-support-v4-jellybean_intermediates/jack-rsc/java-source-list-uniq"; else export tmpEcjArg=""; fi; JACK_VM_COMMAND="java -Dfile.encoding=UTF-8 -Xms2560m -XX:+TieredCompilation -jar /home/hexafluoride/android/system/out/host/linux-x86/framework/jack-launcher.jar " JACK_JAR="/home/hexafluoride/android/system/out/host/linux-x86/framework/jack.jar" /home/hexafluoride/android/system/out/host/linux-x86/bin/jack #build/core/jack-default.args --verbose error -g --classpath /home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/sdk_v16_intermediates/classes.jack:/home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/sdk_v16_intermediates/classes.jack:/home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/android-support-v4-ics-mr1_intermediates/classes.jack --import /home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/android-support-v4-ics-mr1_intermediates/classes.jack -D jack.import.resource.policy=keep-first -D jack.import.type.policy=keep-first -D jack.java.source.version=1.7 --output-jack /home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/android-support-v4-jellybean_intermediates/classes.jack $tmpEcjArg || ( rm -f /home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/android-support-v4-jellybean_intermediates/classes.jack ; exit 41 )
ERROR: /home/hexafluoride/android/system/frameworks/support/v4/jellybean/android/support/v4/app/NavUtilsJB.java:0: The type android.app.Notification cannot be found in source files, imported jack libs or the classpath
build/core/java.mk:636: recipe for target '/home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/android-support-v4-jellybean_intermediates/classes.jack' failed
make: *** [/home/hexafluoride/android/system/out/target/common/obj/JAVA_LIBRARIES/android-support-v4-jellybean_intermediates/classes.jack] Error 41
What am I doing wrong? I'm on Ubuntu Server 16.04.

Alright, the problem probably was because I abruptly interrupted the build process because of a hang, and then tried to continue building from there.
I did a make clobber && brunch angler and it built successfully.

Android source code compile error: "Try increasing heap size with java option '-Xmx<size>'"

Error happens when I try to compile Android source code(Sourcecode-version: 6.0.1; RAM: 6G; host system: ubuntu 14.04),log is below:
including ./system/netd/Android.mk ...
including ./system/security/keystore-engine/Android.mk ...
including ./system/security/keystore/Android.mk ...
including ./system/security/softkeymaster/Android.mk ...
including ./system/tools/aidl/Android.mk ...
including ./system/update_engine/Android.mk ...
including ./system/vold/Android.mk ...
including ./system/weaved/Android.mk ...
including ./system/webservd/Android.mk ...
including ./tools/external/fat32lib/Android.mk ...
Starting build with ninja
ninja: Entering directory `.'
[ 0% 1/21275] Ensure Jack server is installed and started
Jack server already installed in "/home/eddy/.jack-server"
Launching Jack server java -Djava.io.tmpdir=/tmp -Dfile.encoding=UTF-8 -XX:+TieredCompilation -cp /home/eddy/.jack-server/launcher.jar com.android.jack.launcher.ServerLauncher
[ 0% 17/21275] host Java: conscrypt-host (out/host/common/obj/JAVA_LIBRARIES/conscrypt-host_intermediates/classes)
warning: [options] bootstrap class path not set in conjunction with -source 1.7
external/conscrypt/src/openjdk/java/org/conscrypt/Platform.java:39: warning: AlgorithmId is internal proprietary API and may be removed in a future release
import sun.security.x509.AlgorithmId;
^
external/conscrypt/src/openjdk/java/org/conscrypt/Platform.java:243: warning: AlgorithmId is internal proprietary API and may be removed in a future release
return AlgorithmId.get(oid).getName();
^
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
3 warnings
[ 0% 18/21275] host Java: signapk (out/host/common/obj/JAVA_LIBRARIES/signapk_intermediates/classes)
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
[ 0% 73/21275] Building with Jack: out/target/common/obj/JAVA_LIBRARIES/framework_intermediates/with-local/classes.dex
FAILED: /bin/bash out/target/common/obj/JAVA_LIBRARIES/framework_intermediates/with-local/classes.dex.rsp
GC overhead limit exceeded
Try increasing heap size with java option '-Xmx<size>'
Warning: This may have produced partial or corrupted output.
ninja: build stopped: subcommand failed.
make: *** [ninja_wrapper] Error 1
#### make failed to build some targets (14:09 (mm:ss)) ####
eddy#eddy-OptiPlex-390:~/WORKING_DIRECTORY$
Anyone can tell me why the compiling failed?

I had the same problem. So I've tried to set JACK_SERVER_VM_ARGUMENTS to include -Xmx=4g, but when building again the log output showed that this was not included in the startup. I don't know why, it seems like the env vars do not get passed to the build script correctly.
Solution
Before starting a clean android build set the JACK_SERVER_VM_ARGUMENTS to include -Xmx=4g, then stop and start the jack server manually. Given you're in the main source tree of AOSP run the following:
export JACK_SERVER_VM_ARGUMENTS="-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx4g"
./prebuilts/sdk/tools/jack-admin kill-server
./prebuilts/sdk/tools/jack-admin start-server
for cm you can use
export JACK_SERVER_VM_ARGUMENTS="-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx4g"
jack-admin kill-server && jack-admin start-server
This resolved the issue for me.

The current way to set Xmx for jack is:
export JACK_SERVER_VM_ARGUMENTS="-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx4096m"
out/host/linux-x86/bin/jack-admin kill-server
out/host/linux-x86/bin/jack-admin start-server

I also had this problem after updating to 6.0 My computer is an i7 laptop with 8GB of ram. It worked fine with v5.x and below.
The reason is not enough memory as the error message states. In the v6.x build, more Jack is used. In my case, reducing the number of Jacks to 1 resolved the issue.
I can now continue compiling with 8GB of ram.
$HOME/.jack-server/config.properties
jack.server.max-service=1
[100% 19740/19740]
#### make completed successfully (11:55:27 (hh:mm:ss)) ####

Android build (tested with 7.1.2) uses its own, private version of Jack (there could be a Jack toolchain installed elsewhere on the system), so you have to use Android-specific variables.
From https://android.googlesource.com/platform/prebuilts/sdk/+/master/tools/README-jack-server.md:
(search for "If you experience Jack compilations failing on Out of memory error.:")
To summarize:
$ # Stop the Jack server
$ jack-admin stop-server
$ # apply the new setting
$ export ANDROID_JACK_VM_ARGS="-Xmx2g -Dfile.encoding=UTF-8 -XX:+TieredCompilation"
$ Finally, restart the build (do not manually start the Jack server!)
$ m # or any other build command
The build system should pick up the change and output the line:
Environment variable ANDROID_JACK_VM_ARGS was set, regenerating...
or (after a subsequent change):
Environment variable ANDROID_JACK_VM_ARGS was modified (-Xmx2g -Dfile.encoding=UTF-8 -XX:+TieredCompilation => -Xmx4g -Dfile.encoding=UTF-8 -XX:+TieredCompilation), regenerating...

I found this article:
Building AOSP 7.x Nougat with only 8 GiB RAM
on an 8 GiB machine, Max Heap size is less than 2 GiB:
$ java -XshowSettings 2>&1 | grep Heap
Max. Heap Size (Estimated): 1.71G
The solution
Once I understood the problem, it was just a question of changing the default heap size when jack-server is launched. To fix it, you need to edit
~/.jack-settings, and add this line
JACK_SERVER_VM_ARGUMENTS="-Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx4096m"
Then, restart jack-server – otherwise if it is running in the background it will continue to use the old Xmx value:
$ prebuilts/sdk/tools/jack-admin kill-server
$ prebuilts/sdk/tools/jack-admin start-server
Launching Jack server java -XX:MaxJavaStackTraceDepth=-1 -Djava.io.tmpdir=/tmp -Dfile.encoding=UTF-8 -XX:+TieredCompilation -Xmx4096m
-cp /home/chris/.jack-server/launcher.jar com.android.jack.launcher.ServerLauncher

This was also my problem: the java default Xmx setting was already close to 4GB (precisely: 4011MB) so the issue was rather the number of concurrent jack servers running. My machine had 8 CPUs but only 16GB of RAM, but would have required 32GB (8*4=32).
My (slightly more dynamic) solution:
MAX=$(($(free -g | awk '/^Mem:/{print $NF}') / 4))
if [ $MAX -le 0 ]; then
MAX=1
fi
echo "SERVER_NB_COMPILE=$MAX" >> ~/.jack
mkdir -p ~/.jack-server
echo "jack.server.max-service=$MAX" >> ~/.jack-server/config.properties
It adds the MAX number of jack server to both the old-location and new-location based on the android official doc:
Jack Troubleshooting: If your computer becomes unresponsive during compilation or if you experience Jack compilations failing on “Out of memory error”, you can improve the situation by reducing the number of Jack simultaneous compilations by editing your $HOME/.jack and changing SERVER_NB_COMPILE to a lower value. https://source.android.com/source/jack.html