Develop Reference

bz2 module fails when building Python 3.7

I'm trying to cross compile Python 3.7 for Android. I see in my output that bz2 if failing with the following error
building '_bz2' extension
/home/dematic/SPE/python3-android/sdk/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -isystem /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/include -isystem /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/include/openssl -no-integrated-as -I. -I./Include -target aarch64-none-linux-androideabi22 -target aarch64-none-linux-androideabi22 -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -I./Include -I. -I/home/dematic/SPE/python3-android/src/Python-3.7.3/Include -I/home/dematic/SPE/python3-android/src/Python-3.7.3 -c /home/dematic/SPE/python3-android/src/Python-3.7.3/Modules/_bz2module.c -o build/temp.linux-aarch64-3.7/home/dematic/SPE/python3-android/src/Python-3.7.3/Modules/_bz2module.o
/home/dematic/SPE/python3-android/sdk/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -isystem /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/include -isystem /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/include/openssl -no-integrated-as -shared -target aarch64-none-linux-androideabi22 -fuse-ld=lld -L /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib -target aarch64-none-linux-androideabi22 -fuse-ld=lld -L /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib -target aarch64-none-linux-androideabi22 -fuse-ld=lld -L /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib -fPIC -target aarch64-none-linux-androideabi22 build/temp.linux-aarch64-3.7/home/dematic/SPE/python3-android/src/Python-3.7.3/Modules/_bz2module.o -L. -L/home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib -lbz2 -lpython3.7m -o build/lib.linux-aarch64-3.7/_bz2.cpython-37m.so
ld.lld: error: /home/dematic/SPE/python3-android/build/19c-22-aarch64-linux-androideabi-4.9/lib/libbz2.a(bzlib.o) is incompatible with aarch64linux
clang: error: linker command failed with exit code 1 (use -v to see invocation)
I am building bzip2 1.0.6 without any issues, but I assume I'm not linking to it correctly or some other issue. Is there some sort of other architecture I'm supposed to be building?
This is the project I'm trying to build with
https://github.com/GRRedWings/python3-android

I'm trying to cross compile Python 3.7 for Android. I see in my output that bz2 if failing with the following error
The Bzip2 makefiles are not written for cross-compiles. They effectively ignore a user's flags like CFLAGS and LDFLAGS. The makefiles actually blows away a user's CFLAGS and sets it to CFLAGS=-Wall -Winline -O2 -g $(BIGFILES). Your flags like -target aarch64-none-linux-androideabi22 are not used.
There are two Makefiles in play. One is called Makefile and it builds the static library, if I recall correctly. The second is Makefile-libbz2_so, and it build the shared object. You need to fix the omissions and apply the fixes to both makefiles.
You should probably use a patched Bzip like bzip2-noloader. It honors a user's CFLAGS, CXXFLAGS, LDFLAGS, etc. The check-in of interest is Commit 34d170f31106.
The makefile recipes in bzip2-noloader look similar to the following. They preserve Seward's original settings in BZIP_CFLAGS. But they also utilize CPPFLAGS and allow a user override in CFLAGS. The override will pickup your flags like -target aarch64-none-linux-androideabi22.
blocksort.o: blocksort.c
$(CC) $(CPPFLAGS) $(BZIP_CFLAGS) $(CFLAGS) -c blocksort.c
Programs use LDFLAGS as expected:
bzip2: libbz2.a bzip2.o
$(CC) $(CPPFLAGS) $(BZIP_CFLAGS) $(CFLAGS) $(LDFLAGS) -o bzip2 bzip2.o -L. -lbz2
Finally, the bzip2-noloader fork also honor's PREFIX, DESTDIR, etc. So you can perform staged installs, too.
I am building bzip2 1.0.6 without any issues ...
You are probably building for i686 or x86_64, and not Aarch64. The problem does not surface until link time. You can use objdump to inspect the object files, if interested.
Also note the makefile does this:
CC=gcc
AR=ar
RANLIB=ranlib
LDFLAGS=
You may need to tweak those variable assignments, too. Sometimes ar and ranlib use unusual names, like ranlib-5.0. And also be sure the tools are on-path.
The way to write makefiles to avoid these sorts of problems is detailed at 7.2.3 Variables for Specifying Commands in the GNU Coding Standards. The short of it is, (1) leave CFLAGS (and friends) for the user; and (2) if a flag is needed, then always supply it.
The GNU Coding Standards uses this as an example:
CFLAGS = -g
ALL_CFLAGS = -I. $(CFLAGS)
.c.o:
$(CC) -c $(CPPFLAGS) $(ALL_CFLAGS) $<
Users can override the default CFLAGS of -g, and -I is always added because it is needed for the compile.

cmake cannot exclude /usr/include when building for Android

I'm building parquet-cpp project on Ubuntu for Android using cmake 3.10 (by following tutorials "Cross Compiling for Android with a Standalone Toolchain" in this link). I got this error:
In file included from /usr/include/errno.h:28:
/usr/include/features.h:391:10: fatal error: 'gnu/stubs.h' file not found
#include <gnu/stubs.h>
I checked compiling command and found an argument -isystem /usr/include. I tried to exclude /usr/include from search paths but I failed. This is the full command:
<standalone_toolchain>/bin/clang++ --target=armv7-none-linux-androideabi --gcc-toolchain=<standalone_toolchain> --sysroot=<standalone_toolchain>/sysroot -DBOOST_ALL_DYN_LINK -DBOOST_TEST_DYN_LINK -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Dthriftz_EXPORTS -I<source_dir>/parquet-cpp/build/thrift_ep-prefix/src/thrift_ep-build/lib/cpp -I<source_dir>/parquet-cpp/build/thrift_ep-prefix/src/thrift_ep/lib/cpp -I<source_dir>/parquet-cpp/build/thrift_ep-prefix/src/thrift_ep-build -I<source_dir>/parquet-cpp/build/thrift_ep-prefix/src/thrift_ep/lib/cpp/src -isystem /usr/include -isystem <standalone_toolchain>/sysroot/usr/include -isystem <standalone_toolchain>/include/c++/4.9.x -isystem <standalone_toolchain>/include/c++/4.9.x/arm-linux-androideabi/bits -march=armv7-a -marm -mfpu=neon -mfloat-abi=softfp -funwind-tables -no-canonical-prefixes --sysroot <standalone_toolchain>/sysroot -Qunused-arguments -ggdb -O0 -g -fPIC -Wno-variadic-macros -Wno-long-long -Wno-c++11-long-long -Wno-deprecated-register -pthread -g -fPIC -std=c++11 -o CMakeFiles/thriftz.dir/src/thrift/transport/TZlibTransport.cpp.o -c <source_dir>/parquet-cpp/build/thrift_ep-prefix/src/thrift_ep/lib/cpp/src/thrift/transport/TZlibTransport.cpp
I've tried lots of ways I found out when googling (i.e. here and here) but nothing works.
Can anyone here help me? Note that I'm a newbie to cross compiling :(

if you mean by exclude is to delete:
then type this in the terminal --> sudo gksu nautilus a new file manager window will open were you can access all files as root, now reach you file then delete it.

"qmake" and "make" QtRemoteObjects (from git) for Qt 5.7 Android on Windows 7 does not work

I have cloned the QtRemoteObjects repository from: http://code.qt.io/cgit/playground/qtremoteobjects.git/
on my Windows 7 using GitBash.exe.
The project files are download to "D:\Daten\Android\qtremoteobjects\". Based on the article on this website: https://www.kdab.com/qt-android-create-android-service-using-qt/ I run this command in GitBash:
$ cd d:/Daten/Android/qtremoteobjects/
$ git c:/Qt5.7.0/5.7/android_armv7/bin/qmake -r && make && make install
QMAKE PART
For the qmake part these errors show up:
Reading D:/Daten/Android/qtremoteobjects/tests/auto/cmake/cmake.pro
Das System kann den angegebenen Pfad nicht finden.
meaning the system could not find the given path. And the next two errors are:
WARNING: Failure to find: rep_server_source.h
Project MESSAGE: cmake executable not found. Not running CMake unit tests
First question: Why doesnt the system find the path for cmake.pro and rep_server_source.h and why isnt the cmake executable not found even though I have separately installed CMake.
MAKE PART
The next error that shows up is:
bash: make: command not found
As a solution to this I separated the qmake and make process. I first just ran qmake
$ git c:/Qt5.7.0/5.7/android_armv7/bin/qmake -r
and when qmake is finished (and the same errors as above still show up) i run mingw32-make.exe from Qt5.7->Tools in the qtremoteobjects folder:
$ c:/Qt5.7.0/Tools/mingw530_32/bin/mingw32-make.exe
and this is the result:
cd src/ && ( test -e Makefile || C:/Qt5.7.0/5.7/android_x86/bin/qmake.exe
D:/Daten/Android/qtremoteobjects/src/src.pro -o Makefile ) && C:/Qt5.7.0/Tools/mingw530_32/bin/mingw32-make -f Makefile
mingw32-make[1]: Entering directory 'D:/Daten/Android/qtremoteobjects/src'
cd remoteobjects/ && ( test -e Makefile || C:/Qt5.7.0/5.7/android_x86/bin/qmake.exe D:/Daten/Android/qtremoteobjects/src/remoteobjects/remoteobjects.pro -o Makefile ) && C:/Qt5.7.0/Tools/mingw530_32/bin/mingw32-make -f Makefile
mingw32-make[2]: Entering directory 'D:/Daten/Android/qtremoteobjects/src/remoteobjects'
D:\Daten\Android\android-ndk-r10e/toolchains/x86-4.9/prebuilt/windows/bin/i686-linux-android-g++ -c -ffunction-sections -funwind-tables -O2 -fomit-frame-pointer -fstrict-aliasing -funswitch-loops -finline-limit=300 -DANDROID -Wa,--noexecstack -std=c++11 -mfpmath=sse -O2 -O2 -fvisibility=hidden -fvisibility-inlines-hidden -fno-exceptions -Wall -Wno-psabi -W -Wvla -D_REENTRANT -fPIC -DQT_NO_XKBCOMMON -DQT_BUILD_REMOTEOBJECTS_LIB -DQT_BUILDING_QT -DQT_NO_CAST_TO_ASCII -DQT_ASCII_CAST_WARNINGS -DQT_MOC_COMPAT -DQT_USE_QSTRINGBUILDER -DQT_DEPRECATED_WARNINGS -DQT_DISABLE_DEPRECATED_BEFORE=0x050000 -DQT_NO_CAST_TO_ASCII -DQT_NO_CAST_FROM_ASCII -DQT_NO_CAST_FROM_BYTEARRAY -DQT_NO_URL_CAST_FROM_STRING -DQT_BUILD_REMOTEOBJECTS_LIB -DQT_NO_EXCEPTIONS -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -I. -Ic:/utils/openssl-android-master/include -I../../include -I../../include/QtRemoteObjects -I../../include/QtRemoteObjects/5.6.1 -I../../include/QtRemoteObjects/5.6.1/QtRemoteObjects -I. -IC:/Qt5.7.0/5.7/android_x86/include -IC:/Qt5.7.0/5.7/android_x86/include/QtNetwork -IC:/Qt5.7.0/5.7/android_x86/include/QtCore/5.7.0 -IC:/Qt5.7.0/5.7/android_x86/include/QtCore/5.7.0/QtCore -IC:/Qt5.7.0/5.7/android_x86/include/QtCore -I.moc -isystem D:/Daten/Android/android-ndk-r10e/sources/cxx-stl/gnu-libstdc++/4.9/include -isystem D:/Daten/Android/android-ndk-r10e/sources/cxx-stl/gnu-libstdc++/4.9/libs/x86/include -isystem D:/Daten/Android/android-ndk-r10e/platforms/android-9/arch-x86/usr/include -IC:/Qt5.7.0/5.7/android_x86/mkspecs/android-g++ -o .obj/qconnection_local_backend.obj qconnection_local_backend.cpp
Makefile:1720: recipe for target '.obj/qconnection_local_backend.obj' failed
process_begin: CreateProcess(NULL, D:\Daten\Android\android-ndk-r10e/toolchains/x86-4.9/prebuilt/windows/bin/i686-linux-android-g++ -c -ffunction-sections -funwind-tables -O2 -fomit-frame-pointer -fstrict-aliasing -funswitch-loops -finline-limit=300 -DANDROID -Wa,--noexecstack -std=c++11 -mfpmath=sse -O2 -O2 -fvisibility=hidden -fvisibility-inlines-hidden -fno-exceptions -Wall -Wno-psabi -W -Wvla -D_REENTRANT -fPIC -DQT_NO_XKBCOMMON -DQT_BUILD_REMOTEOBJECTS_LIB -DQT_BUILDING_QT -DQT_NO_CAST_TO_ASCII -DQT_ASCII_CAST_WARNINGS -DQT_MOC_COMPAT -DQT_USE_QSTRINGBUILDER -DQT_DEPRECATED_WARNINGS -DQT_DISABLE_DEPRECATED_BEFORE=0x050000 -DQT_NO_CAST_TO_ASCII -DQT_NO_CAST_FROM_ASCII -DQT_NO_CAST_FROM_BYTEARRAY -DQT_NO_URL_CAST_FROM_STRING -DQT_BUILD_REMOTEOBJECTS_LIB -DQT_NO_EXCEPTIONS -DQT_NO_DEBUG -DQT_NETWORK_LIB -DQT_CORE_LIB -I. -Ic:/utils/openssl-android-master/include -I../../include -I../../include/QtRemoteObjects -I../../include/QtRemoteObjects/5.6.1 -I../../include/QtRemoteObjects/5.6.1/QtRemoteObjects -I. -IC:/Qt5.7.0/5.7/android_x86/include -IC:/Qt5.7.0/5.7/android_x86/include/QtNetwork -IC:/Qt5.7.0/5.7/android_x86/include/QtCore/5.7.0 -IC:/Qt5.7.0/5.7/android_x86/include/QtCore/5.7.0/QtCore -IC:/Qt5.7.0/5.7/android_x86/include/QtCore -I.moc -isystem D:/Daten/Android/android-ndk-r10e/sources/cxx-stl/gnu-libstdc++/4.9/include -isystem D:/Daten/Android/android-ndk-r10e/sources/cxx-stl/gnu-libstdc++/4.9/libs/x86/include -isystem D:/Daten/Android/android-ndk-r10e/platforms/android-9/arch-x86/usr/include -IC:/Qt5.7.0/5.7/android_x86/mkspecs/android-g++ -o .obj/qconnection_local_backend.obj qconnection_local_backend.cpp, ...) failed.
make (e=2): Das System kann die angegebene Datei nicht finden.
mingw32-make[2]: *** [.obj/qconnection_local_backend.obj] Error 2
mingw32-make[2]: Leaving directory 'D:/Daten/Android/qtremoteobjects/src/remoteobjects'
Makefile:44: recipe for target 'sub-remoteobjects-make_first-ordered' failed
mingw32-make[1]: *** [sub-remoteobjects-make_first-ordered] Error 2
mingw32-make[1]: Leaving directory 'D:/Daten/Android/qtremoteobjects/src'
Makefile:46: recipe for target 'sub-src-make_first' failed
mingw32-make: *** [sub-src-make_first] Error 2
Second Question Why is make not successful here? what do these errors mean? What is wrong with '.obj/qconnection_local_backend.obj'
Is it because I am running make on Windows?
Thanks,

CMake (Android Toolchain) - Error running link command: No such file or directory

I am using CMake for building the Android project.
“android.toolchain.cmake” toolchain is used for building the project.
All the environment variables are set as described in the usage part of the android cmake.
Following Option variables are set as environment variables
ANDTOOLCHAIN
ANDROID_STANDALONE_TOOLCHAIN
ANDROID_ABI
ANDROID_NATIVE_API
ANDROID_STL
CC and CXX to point to the androideabi-gcc & androideabi-g++ respectively
CMAKE is successfully generating the Make files but when I “make” the project its throwing “Error running link command: No such file or directory”.
The details of the link.txt are as under
*cd /Extra/svnroot/2014_Project/build/project123 && /usr/bin/cmake -E cmake_link_script CMakeFiles/project123dir/link.txt --verbose=1
CMAKE_C_COMPILER -fPIC -Wno-psabi --sysroot=/path_to_sysroot_dir/sysroot -fpic -funwind-tables -finline-limit=64 -fsigned-char -no-canonical-prefixes -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -fdata-sections -ffunction-sections -Wa,--noexecstack -Wno-missing-field-initializers -Wno-strict-aliasing -Wno-deprecated -Wno-unused -Wno-unused-variable -Wsign-compare -Wsequence-point -Wunused-but-set-variable -Wall -Wextra -Wno-unused-parameter -fno-strict-aliasing -pthread -Wl,--fix-cortex-a8 -Wl,--no-undefined -Wl,-allow-shlib-undefined -Wl,--gc-sections -fuse-ld=gold -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -shared -Wl,-soname,libproject123.so -o ../libproject123.so CMakeFiles/project123_Sources.dir/MyAdapter.cpp.o
CMakeFiles/ project123_Sources.dir /SystemInfo.cpp.o CMakeFiles/
project123_Sources.dir /ServiceControl.cpp.o CMakeFiles/
project123_Sources.dir /Wrapper.cpp.o CMakeFiles/
project123_Sources.dir /Service.cpp.o CMakeFiles/
project123_Sources.dir /MemMap.cpp.o CMakeFiles/
project123_Sources.dir.dir/Trace.cpp.o
-L/path_to_ toolchain_dir/android-toolchain/bin/armeabi-v7a
-L/Extra/svnroot/2014_Q4_Mobile/build
-L/path_to_thirdparty_lib/zlib-1.2.7/lib
-L/path_to_thirdparty_lib /libzip-0.11.1
-L/path_to_thirdparty_lib /boost_1_53_0/lib-android
-L/path_to_thirdparty_lib/curl-7.26.0-patch/lib
-L/path_to_thirdparty_lib /fips-openssl-1.0.1g/lib /path_to_thirdparty_lib
/boost_1_53_0/lib-android/libboost_date_time-gcc-mt-1_53.a
/path_to_thirdparty_lib/boost_1_53_0/lib-android/libboost_filesystem-gcc-mt-1_53.a
/path_to_thirdparty_lib/boost_1_53_0/lib-android/libboost_system-gcc-mt-1_53.a /path_to_thirdparty_lib
/boost_1_53_0/lib-android/libboost_thread-gcc-mt-1_53.a
/path_to_thirdparty_lib
/boost_1_53_0/lib-android/libboost_program_options-gcc-mt-1_53.a
/path_to_thirdparty_lib
/boost_1_53_0/lib-android/libboost_serialization-gcc-mt-1_53.a
/path_to_thirdparty_lib /curl-7.26.0-patch/lib/libcurl.a
/path_to_thirdparty_lib /fips-openssl-1.0.1g/lib/libssl.a
/path_to_thirdparty_lib /fips-openssl-1.0.1g/lib/libcrypto.a
/path_to_thirdparty_lib /zlib-1.2.7/lib/libz.a
/path_to_thirdparty_lib /libzip-0.11.1/libzip.a -ldl -lrt
"/path_to_toolchain_dir/android-toolchain/arm-linux-androideabi/lib/armv7-a/thumb/libstdc++.a"
"/path_to_toolchain_dir/android-toolchain/arm-linux-androideabi/lib/armv7-a/thumb/libsupc++.a"*
All files in the link.txt file are pointing to the right directory/path and there is no problem with any of the file paths. In the link.txt file instead of using the actual path to the compiler it is using CMAKE_C_COMPILER. I presume this is creating the problem since “make” command is not able to find the actual path to the compiler.
Any help in fixing this issue would be greatly appreciated.

I encountered the same issue recently, and couldn't figure out why it failed to replace CMAKE_C_COMPILER with the right path. So what I ended up doing is to modify the android.toolchain.cmake, where the CMAKE_CXX_CREATE_YYY is being set. From
set( CMAKE_CXX_CREATE_SHARED_LIBRARY "<CMAKE_C_COMPILER> ...")
set( CMAKE_CXX_CREATE_SHARED_MODULE "<CMAKE_C_COMPILER> ...")
set( CMAKE_CXX_LINK_EXECUTABLE "<CMAKE_C_COMPILER> ...")
to
set( CMAKE_CXX_CREATE_SHARED_LIBRARY "<CMAKE_CXX_COMPILER> ...")
set( CMAKE_CXX_CREATE_SHARED_MODULE "<CMAKE_CXX_COMPILER> ...")
set( CMAKE_CXX_LINK_EXECUTABLE "<CMAKE_CXX_COMPILER> ...")
And CMake was able to generate the right link.txt. Hope this helps.

how to force "./configure" to stop using -pthread during compilation?

I am trying to cross-compile something for Android, which has pthread support but does NOT use -pthread when compiling and linking. If you try to use -pthread, compilation using the cross-compiler will fail.
When trying to cross-compile something by first using the 'configure' tool, it keeps trying to use -pthread to do things like "check for gethostbyname" and therefore my configure fails:
configure:21229: checking for gethostbyname
configure:21229: /mnt/hgfs/Documents/thesis/android-wmon/core/android-wireshark/agcc \
-o conftest -D_U_="__attribute__((unused))" -g -O2 -Wall -W -Wextra \
-Wdeclaration-after-statement -Wendif-labels -Wpointer-arith \
-Wno-pointer-sign -Warray-bounds -Wcast-align -Wformat-security \
-fexcess-precision=fast -I/usr/local/include -pthread \
-I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include \
-I/usr/local/include -Wl,--as-needed -L/usr/local/lib conftest.c >&5
arm-eabi-gcc: error: unrecognized option '-pthread'
I cannot find where it picks up this flag from, and how I can get it to stop doing that. I do not see it in configure anywhere, so it must be dynamically generating it. I just can't seem to find where it does this so I can remove it from whatever variable it is stored within configure.

Well, it's a hack but it seems as though you can manipulate CFLAGS before it tries the link test:
ac_fn_c_try_link ()
{
CFLAGS="${CFLAGS/-pthread/}"
... that is in 'configure'