Develop Reference

Building Qt 5.12 LTS for Android with SSL and SQL support

With google deprecating GCC, Qt 5.12 for Android makes a transition away from it in favor of Clang. Which renders the existing building guides obsolete, as they are all GCC based.
The stock Qt build for Android is lacking some important functionality, and furthermore, the related documentation seems to be fairly outdated.
Additionally, 5.12 launches with a critical Android related bug, which pretty much mandates a custom build to incorporate the fix for the time being.
I myself haven't previously used Clang at all, and I am one of those people who really prefer to just make applications rather than to go through the often excruciatingly frustrating experience of trying to build the requisite tools.
Qt on Clang is here to stay, and 5.12 being a long term support release, I think a detailed step by step guide how to produce a working Qt build will be of benefit to a lot of people. It would definitely save me days of headaches and setbacks, and so I am willing to offer a generous bounty to the first reproducible answer, in addition to any bounties that may be required to promote the question visibility.
My personal requirements are SSL, MySQL and PostgreSQL support, although additional functionality is welcome. It would seem that a Linux based guide will be the most beneficial format, as it is also applicable to windows via MSYS.

I can't reply for all your requests, but I can say something about openssl:
Firstoff, even with Qt 5.12, if you are using the official android builds downloaded from Qt itself (via the Maintenancetool), then you still have to use the gcc toolchain and openssl 1.0.2. The Qt builds require 1.0.* and clang support has only been added to openssl since 1.1.1. Support for this version of openssl will hopefully come with Qt 5.13. See QTBUG-71391 for more details.
That beeing said, if you cross-compile Qt for Android yourself (or visit this post in the future, when Qt supports this), you can use openssl 1.1 and use clang to compile it.
The steps are documented on their github in NOTES.ANDROID. The steps are relatively straight forward and boil down to a few changes to PATH in order to build the library. A basic script, with switches for all android architectures provided by Qt, would be:
TOOLCHAIN_VERSION=4.9
HOST_ARCH=linux-x86_64
case "$ANDROID_TARGET_ARCH" in
arm64-v8a)
API_VERSION=21
ARCH_ID=android-arm64
TOOLCHAIN=aarch64-linux-android-$TOOLCHAIN_VERSION
;;
armeabi-v7a)
API_VERSION=16
ARCH_ID=android-arm
TOOLCHAIN=arm-linux-android-$TOOLCHAIN_VERSION
;;
x86)
API_VERSION=16
ARCH_ID=android-x86
TOOLCHAIN=x86-$TOOLCHAIN_VERSION
;;
*)
echo "Unsupported ANDROID_TARGET_ARCH: $ANDROID_TARGET_ARCH"
exit 1
;;
esac
export ANDROID_NDK=/path/to/ndk
export PATH=$ANDROID_NDK/toolchains/llvm/prebuilt/$HOST_ARCH/bin/:$ANDROID_NDK/toolchains/$TOOLCHAIN/prebuilt/$HOST_ARCH/bin:$PATH
./Configure $ARCH_ID shared no-ssl3 -D__ANDROID_API__=$API_VERSION
make SHLIB_VERSION_NUMBER= SHLIB_EXT=.so build_libs
In this script, ANDROID_TARGET_ARCH is simply the value of the qmake variable with the same name, so this script could be invoked by qmake. The steps that need to be done in detail are:
Prepare some variables:
TOOLCHAIN_VERSION: Simply the version of the gcc toolchain to be used (yes, this is still needed, as some tools, linke ranlib etc. are still used from there). As of NDK v18 the toolchain version is still 4.9
HOST_ARCH: The architecture of the host system. The sample sets this to linux. If you are on window/macos adjust this accordingly.
API_VERSION: The Android SDK version openssl should build for. I set the values to the version that Qt uses for builds for these plattforms, but other versions should be fine as well
ARCH_ID: The name of the android architecture as used by openssl
TOOLCHAIN: The name of the gcc toolchain to be used
Ensure the ANDROID_NDK environment variable is set to wherever you installed the NDK
Update the path to contain both, the clang/llvm toolchain and the gcc toolchain for your specific plattform. In the script, the locations of the toolchains are derived from the previous variables
Explicitly run the Configure script - and not config. Pass the target architecture and additional flags. (I for example prefer to disable ssl3 for security reasons)
Run make to build the library. The SHLIB_VERSION_NUMBER= SHLIB_EXT=.so part is needed to ensure that the binaries created do not have a version number as part of their name, as android does not support that.
And thats it!

Android NDK build-host-executable.mk: No such file or directory

Short question:
I'm trying to build an app that has a JNI dependency, using Eclipse ADT and NDK r10e (the current version). The JNI dependency (apparently) builds a binary to run on the build host, using build-host-executable.mk from the NDK. However, that script seems to have been removed in recent NDK versions. What do I do?
Long question:
The JNI dependency is platform/external/srec from AOSP, which comes with an Android.mk file (actually, several of them).
I have set up Eclipse to build the native parts according to the instructions here, and copied the JNI code tree into the jni directory of my app's source tree. Additionally, I had to edit jni/Android.mk, adding the following line:
export TARGET_BUILD_TYPE := debug
which, as I understand it, would be set by the AOSP toolchain but is not set when building from Eclipse.
However, I get the following error:
android-ndk-r10e/build/core/build-host-executable.mk: No such file or directory
This is caused by a line in one of my Android.mk files:
include $(BUILD_HOST_EXECUTABLE)
The NDK defines BUILD_HOST_EXECUTABLE in build/core/build-all.mk:
BUILD_HOST_EXECUTABLE := $(BUILD_SYSTEM)/build-host-executable.mk
However, build_host_executable.mk is absent from the build/core directory (and so are the other build-host-*.mk scripts declared in the surrounding lines).
Googling around a bit, I find that this script seems to have been present in earlier NDK versions (up to at least NDK r7 r7b) but seems to be absent from later versions (possibly since r10b). The JNI lib was used by the (now abandoned) Voice Dial app. It had its last commit in December 2014, though the last real code change might have been in June 2014. AOSP had stopped building it by June 2015. Comparing this with the NDK release history, it might never have been built against NDK versions later than r9d or maybe r10.
The NDK r7b I found appears to be a patched version to which host target support added, see also https://github.com/flyskywhy/android-ndk-host. Looks like support for this was planned in NDK and some stubs added, but it was never fully implemented. That gives me the choice of either building the host stuff with the local toolchain, or patching NDK r10e to add host target support.
On the other hand, this package has been part of AOSP for a while (Cyanogenmod included it until KitKat and dropped it in Lollipop), therefore the AOSP toolchain must have been capable of building it. Any pointers to that toolchain? Maybe it's possible to extract the relevant parts from the AOSP toolchain and add them to the package.

The JNI dependency in question is taken from the AOSP code tree. The toolchain used by AOSP is different from NDK, though both are built around a set of GNU make macros. The format for Android.mk (their makefile equivalent) is intentionally similar between both, so that in many cases a project can be built with either toolchain.
One case for which this will not work is host targets, i.e. building binaries (executables or libraries) intended to run on the build host rather than on the Android device. The AOSP toolchain has this kind of support but the NDK doesn't. Confusingly, the NDK does have some stubs for that functionality (including definitions for BUILD_HOST_EXECUTABLE and similar) but the implementation is missing.
There are three options. Be aware that the package you are building may depend on other packages from the AOSP source code, which you would need to build as well.
Build the native code (or the host targets) using the AOSP toolchain, as described in this answer. This approach is most likely to be successful, as this is the toolchain for which the package was designed. It requires at least the packages containing the build tools, which are several gigabytes in size. You can get the entire AOSP source code as described here, which should give you all the dependencies you need, but this will download some 50 gigabytes (!) of data.
There is a patched version of NDK which adds support for host targets here (or just the make macros here). However, it is based on the now-outdated NDK r7b and had its last commit sometime in 2012 – you'll be working with a toolchain that is somewhat exotic and no longer maintained. Also, you'll need to determine your dependencies manually by examining the package.
Ditch the Android.mk files that build the host targets and build these with a regular GNU toolchain. An introduction to that can be found here. You can amend a higher-level Android.mk to invoke the "other" toolchain for your host targets, so you can automate the whole build. All of this may be a lot of work if the Android.mk content is very complex, but it is a one-time effort, after which your code will build with standard toolchains. As with the previous approach, you'll need to determine your dependencies manually. Dependencies which are required for the host target may be easier, though, if they are standard libraries which were ported to Android, as there may be an upstream version designed to be built with the GNU toolchain.

How to build linux command for Android on Linux machine

Since there are a few commands missing on Android, I want to build them by myself.
Take one as an example:
There is no dig command on Android.
I've got the source code from ftp://ftp.isc.org/isc/bind9/9.2.3/bind-9.2.3.tar.gz and then I want to build it. But I got stuck.
How can I build the executable on my linux machine?

You can do it in two ways.
To build any C Source code for Android Platform you can use Android NDK Toolchian, It lets you build Native code for another platform.
It can be used as a stand-alone compiler without an Android APK.
Documentation and Sample Code.
To build it as an executable (without APK) and not shared object, replace include $(BUILD_SHARED_LIBRARY) with include $(BUILD_EXECUTABLE) in Android.mk.
For Linux commands on Android you can use Busybox.
It supports most of the commands.
However dig is not supported in busybox, you can add it by following this article.
Hopefully this gives you a start.

cmake vs waf: mainly for c++ windows/linux and android

After searching a lot and reading a lot of information, I cannot decide which tool I should use for compiling my code. My codebase is mainly c++. I use primarily linux as my development machine.
Based on opinions I read before, my final candidates are waf and cmake, but I cannot decide myself which one should be more appropiate.
My primary requirements are:
Must be able to compile software in windows/linux and android.
Must be ready to run tests.
Must be able to play nicely with other libraries that must be compiled with another build system but most likely will have to be compiled from source.
Must be able to add custom steps, like for example, generating some data from some files (mainly graphics) before compiling, all integrated in the build system.
Some strong preferences are:
Being ready to support MAC compilation.
Being able to cross-compile from linux as many platforms as I can (maybe windows/linux/android but cannot MAC?)
Being able to add support for iOS compilation if the need arises.
Would be nice if the invocation interface was similar to that of autotools, since it is the one many people know and it is well documented.
Some questions:
If I have some rare requirement, which build system would be more ready to be extended?
Are both currently well maintained? (I wonder about waf mainly).
Community: if I find a problem, both communities are big enough to support me, in your experience?
For now my feeling is that I favour waf a bit as a tool, but cmake seems to have been quite successful for whatever reason.

Don't know much about waf, but CMake fits your requirements pretty well. I do know waf is written in Python, my personal favourite programming language ATM.
My primary requirements are:
Must be able to compile software in windows/linux and android.
CMake does Windows and Linux very well but so does any other build system worth its salt.
Someone wrote some Android scripts for CMake. Can't find anything similar for waf (my Google-fu turns up nothing.)
Must be ready to run tests.
CMake has a sibling testing framework.
Must be able to play nicely with other libraries that must be compiled with another build system but most likely will have to be compiled from source.
CMake has good integration with pkg-config, and can link against arbitrary shared libraries.
Must be able to add custom steps, like for example, generating some data from some files (mainly graphics) before compiling, all integrated in the build system.
CMake can generate custom rules.
Some strong preferences are:
Being ready to support MAC compilation.
CMake supports Mac quite well. It will even make you an Xcode project if you want, but it can also do command line builds.
Being able to cross-compile from linux as many platforms as I can (maybe windows/linux/android but cannot MAC?)
Cross-compiling is supported in CMake. CMake will not be the primary source of pain with cross-compiling - literally everything else will.
Especially with regards to cross-compiling for Mac. It's possible, but not worth it to cross-compile for that platform, considering you need access to a Mac anyways to get the libraries and header files, you need to patch GCC and clang and LLVM, etc. The only sound reason I've heard for going through this much pain is running an automated build server. Anyways, if you get a working Linux -> Mac toolchain, you should be able to cross-compile with CMake as if it were any other Unix platform.
Being able to add support for iOS compilation if the need arises.
iOS cross-compilation can be done, but you need a Mac.
Would be nice if the invocation interface was similar to that of autotools, since it is the one many people know and it is well documented.
Write a configure script that just calls CMake (cmake .). Then your users can do a ./configure && make && make install on platforms where that makes sense. There's also CPack which lets you generate DEB, RPM, NSIS (Windows) and DMG (Mac) installers/packages.
Some questions:
If I have some rare requirement, which build system would be more ready to be extended?
CMake is very extensible. It can be extended to support new languages and target platforms. (Given that waf is written in Python, it's going to be pretty hackable too.)
Are both currently well maintained? (I wonder about waf mainly).
CMake is mature and well-maintained.
Community: if I find a problem, both communities are big enough to support me, in your experience?
The community and extensions available are what keeps me coming back to CMake, from things like bakefile, honestly.

WAF
is pure Python
becomes part of your project, i.e. no external dependency
supports many build tools
can be used to do all kind of automations, not just building
It works perfectly for Linux, Mac or Windows.
On Android, gradle is the chosen build tool of Google. To use that is
wise, because it is set up to work by Google. You can call waf from
gradle and vice-versa, though.
If you want to learn all the low level Android
SDK tools, you could also use
WAF directly.
The SDK has
javac for Android Runtime (formerly Dalvik), Android\'s JVM, and produces a .class file
jar can also be used for Android
d8 (formerly dx) produces .dex files, with Dalvik executable code
aapt2 can then produce the .apk
javac and jar are known to WAF. For dx and aapt2 you would need
to create your own tasks, which is very
easy.
You would best make a WAF tool and
share it. Tools are either part of WAF or there is
waftools.
There are also these Steinwurf
tools.
If you make Android native code using
NDK:
you use CLANG, which is known to WAF
Further on you mentioned requirements:
WAF has waf_unit_test
WAF can do gnu_cross compilation. The Gnu toolchain knowns many
targets. But for Android you would need to set things up yourself
using the SDK or NDK. For NDK you could use the Gnu toolchain.
You would do waf configure, waf build instead of configure,
make, but you could wrap a Configure or Makefile around waf to
have the same commands.
WAF is very easily extendible with Python
WAF is now on gitlab and
constantly worked on.
The community is surely smaller than for CMake. But it is Python.
You can look into it and find out for yourself. You can also
contribute and become part of the community.

How to compile GCC for android?

Is there any way to compile GCC for android? Basically, have the GCC compiler accessible in an android terminal emulator and able to compile a binary that will run on android from a C or C++ source file.
My intention is to use this and eventually install make.

The Android NDK already includes the complete GNU toolchain which runs on your computer. You should be able to use it to compile native versions of whatever program you want.
In the NDK, see docs/STANDALONE-TOOLCHAIN.html for setting up the cross-compiler. You probably want to follow the steps for "Invoking the compiler (the easy way)" to set up a copy of the toolchain which you should be able to use with configure scripts commonly included with GNU applications (like gcc).
This really is a non-trivial task and I recommend you read the docs very carefully. It also contains various values for CFLAGS, LDFLAGS etc. that you will need to customize and use when configuring and compiling your programs.
As for compiling gcc, make, etc., themselves, it's probably not necessary or desirable to do so. gcc, for instance, is going to be huge and you may not have enough storage on the phone to install it. If your ultimate intention is to compile some program to run natively on the Android device, then I would recommend you just use the NDK and point the app's configure script at it.