I'm building an Android library based on Tensorflow with Bazel.
Here the BUILD file
cc_binary(
name = "libfoo.so",
srcs = glob([
"jni/**/*.cc",
"jni/**/*.h",
]),
copts = [ "-fexceptions", "-DEIGEN_AVOID_STL_ARRAY",
"-mfpu=neon", "-std=c++11",
"-DMIN_LOG_LEVEL=0", "-DTF_LEAN_BINARY",
"-O2", ],
linkopts = [
"-llog",
"-lm",
],
linkshared = 1,
deps = [
"#org_tensorflow//tensorflow/core:android_tensorflow_lib",
"#boringssl//:crypto",
],
)
The linker complains about not finding -lpthread, while I didn't add this flag to linkopts.
I've checked the executed command, and in fact there is extra flags on it: -lz -lpthread ...
Where did they came from ? Is there a workaround for this ?
I got the answer from the tensorflow's issue tracker.
#jart
Since copts and linkopts are viral and propagate to dependencies, -lpthread is most likely being inherited from #boringssl//:crypto
#andrewharp
-lpthread is not necessary or possible on Android, so it sounds like the solution would be to add another condition for the select statement as in the linked commit google/protobuf#1386:
...
The only other workaround I know of that doesn't require editing the other repository is to create a dummy libpthread.so target, but that's pretty hacky.
Related
I have a binary in Android that links to a static library A. Static library libA depends on multiple shared libraries.
The binary does not do anything except it imports a class from the static library and executes a simple function.
However, the binary fails to build except I link against the same shared libraries to which the static library A is linked because the compiler tries to recompile libA with the build config of the binary.
Here is my Android.bp of the static library:
cc_library_static {
name: "libA",
relative_install_path: "hw",
vendor: true,
rtti: true,
cflags: [
"-Wall",
"-Wextra",
"-g",
"-DUNIT_TEST",
"-fexceptions"
],
srcs: [
"libA.cpp",
],
shared_libs: [
"libhidlbase",
"libhidltransport",
"libutils",
"liblog"
],
header_libs: [
"lib_a_stub_headers",
],
whole_static_libs: [
"lib_a_stub",
],
export_include_dirs: ["."]
}
Here is my Android.bp for the binary:
cc_binary{
name: "simplebinary",
relative_install_path: "hw",
vendor: true,
cflags: [
"-fexceptions"
],
whole_static_libs: [
"libA"
],
shared_libs: [
"vendor.test.hal#1.0",
],
srcs: [
"simplebinary.cpp",
],
}
The build of the binary fails with:
libA.hpp:4:10: fatal error: 'lib/lib.hpp' file not found
I'm building using the command mm
According to the error message, the compiler cannot find a header file in its header search path. Header includes are resolved during the preprocessor stage, therefore this is not a linking problem. The preprocessor runs at the beginning of compilation, the linking is done at the end.
From your description, I understand that the code for simplebinary includes the header libA.hpp, provided by libA. I understand that libA.hpp is contained in the same directory as the Android.bp that defines the libA module. Because of the export_include_dirs: ["."], this directory is added to the header search path for the compilation of simplebinary. Therefore, the compiler can find libA.hpp when compiling simplebinary.
Now libA.hpp includes CommonAPI/CommonAPI.hpp. I do not know to which library this header belongs. I assume the header belongs to some library libB, and libA links against libB. I further assume that libB has export_include_dirs set to point to the folder containing CommonAPI/CommonAPI.hpp. You can then make libA re-export this header by adding export_shared_lib_headers: ["libB"] to the module declaration of libA. If libB is not a shared library, you would have to use export_static_lib_headers or export_header_lib_headers instead (reference).
I put react-native.jar to tensorflow android's directory and add configuration to tensorflow/examples/android/BUILD file like this:
java_import(
name = "react-native",
jars = [
"react-native-0.39.2-sources.jar",
],
)
Now run bazel build //tensorflow/examples/android:tensorflow_demo work well. But then use import com.facebook.react.*; in java file and run bazel build ... again, it throws a nonexistent error, can't find com.facebook.react.* package.
In order to expose the classes in the JAR to the Java code in the Android build, you need to add a dependency in the android_binary that you are building on the java_import you have created.
For example:
# tensorflow/examples/android/BUILD
java_import(
name = "react-native",
jars = [
"react-native-0.39.2-sources.jar",
],
)
android_binary(
name = "tensorflow_demo",
srcs = glob(["src/**/*.java"]),
deps = [
":tensorflow_native_libs",
"//tensorflow/contrib/android:android_tensorflow_inference_java",
":react-native",
],
)
I'm currently trying to compile the target //tensorflow:libtensorflow_cc.so of TensorFlow with bazel for Android. I need this library in order to get the javacpp-presets for TensorFlow working with Android.
I tried the following statement:
bazel build -c opt //tensorflow:libtensorflow_cc.so --crosstool_top=//external:android/crosstool --cpu=armeabi-v7a --host_crosstool_top=#bazel_tools//tools/cpp:toolchain --verbose_failures
Which however results in the error that S_IREAD, S_IWRITE cannot be found:
external/gif_archive/giflib-5.1.4/lib/egif_lib.c:62:6: error: 'S_IREAD' undeclared (first use in this function)
S_IREAD | S_IWRITE);
^
external/gif_archive/giflib-5.1.4/lib/egif_lib.c:62:6: note: each undeclared identifier is reported only once for each function it appears in
external/gif_archive/giflib-5.1.4/lib/egif_lib.c:62:16: error: 'S_IWRITE' undeclared (first use in this function)
S_IREAD | S_IWRITE);
^
Target //tensorflow:libtensorflow_cc.so failed to build
Inspired by the Android build in the Android Demo build, I also tried to change the cc_binary definition to the following code, but still got the same error.
cc_binary(
name = "libtensorflow_cc.so",
copts = tf_copts(),
linkopts = [
"-landroid",
"-ljnigraphics",
"-llog",
"-lm",
"-z defs",
"-s",
"-Wl,--icf=all", # Identical Code Folding
],
linkshared = 1,
linkstatic = 1,
deps = [
"//tensorflow/c:c_api",
"//tensorflow/cc:cc_ops",
"//tensorflow/core:tensorflow",
],
)
From googling, I found out that the S_IWRITE flags are deprecated and therefore have never been implemented in Android. However, I have no idea how to get around this problem.
To sum it up: Do you know how I can build the libtensorflow_cc.so target for Android? The library build in the Android example is not enough for me because I also need the cc_ops included.
From googling, I found out that the S_IWRITE flags are deprecated and therefore have never been implemented in Android.
It looks like we've changed our minds on that for the sake of compatibility: https://android.googlesource.com/platform/bionic/+/1f1a51aecd7c825418bfedcb66772e92de790149%5E%21/#F2
#if defined(__USE_BSD) || defined(__USE_GNU)
#define S_IREAD S_IRUSR
#define S_IWRITE S_IWUSR
#define S_IEXEC S_IXUSR
#endif
That's the system's sys/stat.h; it hasn't been shipped in the NDK yet. Unfortunately most of the NDK headers are very out of date. This is https://github.com/android-ndk/ndk/issues/120.
We'll get a fix for this into NDK r14 (I just filed https://github.com/android-ndk/ndk/issues/211 to fix up the old headers in case #120 doesn't get fixed by then).
Until then, you could add those defines to your cflags. Looks like the way to do this in bazel would be:
cc_binary(
name = "libtensorflow_cc.so",
defines = [
"S_IREAD=S_IRUSR",
"S_IWRITE=S_IWUSR",
"S_IEXEC=S_IXUSR",
],
...
)
https://www.bazel.io/versions/master/docs/be/c-cpp.html#cc_binary.defines
I've been benchmarking tensorflow models on Exynos 7420 with benchmark_model. I'd like to speed test Quantization per Pete Warden's blog but have not been able to compile benchmark_model with quantization deps yet as they break a number of things.
I've followed the guidelines listed in this stack overflow thread:
//tensorflow/tools/benchmark/BUILD cc_binary
deps = [":benchmark_model_lib",
"//tensorflow/contrib/quantization/kernels:quantized_ops",
],
//tensorflow/contrib/quantization/kernels/BUILD:
deps = [
"//tensorflow/contrib/quantization:cc_array_ops",
"//tensorflow/contrib/quantization:cc_math_ops",
"//tensorflow/contrib/quantization:cc_nn_ops",
#"//tensorflow/core",
#"//tensorflow/core:framework",
#"//tensorflow/core:lib",
#"//tensorflow/core/kernels:concat_lib_hdrs",
#"//tensorflow/core/kernels:conv_ops",
#"//tensorflow/core/kernels:eigen_helpers",
#"//tensorflow/core/kernels:ops_util",
#"//tensorflow/core/kernels:pooling_ops",
"//third_party/eigen3",
"#gemmlowp//:eight_bit_int_gemm",
],
Then run:
bazel build -c opt --cxxopt='-std=gnu++11'--crosstool_top=//external:android/crosstool --cpu=armeabi-v7a --host_crosstool_top=#bazel_tools//tools/cpp:toolchain tensorflow/tools/benchmark:benchmark_model --verbose_failures
Which (with following all other instructions in linked post) succeeds with the exception that it fails to link against pthread.
I've tried removing -lpthread in tensorflow/tensorflow.bzl tfcopts(), and similarly in tensorflow/tools/proto_text/BUILD, and tensorflow/cc/BUILD.
def tf_copts():
return (["-fno-exceptions", "-DEIGEN_AVOID_STL_ARRAY"] +
if_cuda(["-DGOOGLE_CUDA=1"]) +
if_android_arm(["-mfpu=neon"]) +
select({"//tensorflow:android": [
"-std=c++11",
"-DMIN_LOG_LEVEL=0",
"-DTF_LEAN_BINARY",
"-O2",
],
"//tensorflow:darwin": [],
"//tensorflow:ios": ["-std=c++11",],
#"//conditions:default": ["-lpthread"]}))
"//conditions:default": []}))
Still getting getting the link error.
external/androidndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lpthread
collect2: error: ld returned 1 exit status
Any help much appreciated, I'm pretty stuck.
Env:
Ubuntu 14.04
tensorflow commit #4462
android_ndk_r11c
android-sdk-linux r24.4.1
Python 2.7.12 :: Continuum Analytics, Inc.
./configure without support for GCP, HDFS, or GPU
Transcribing GitHub answer from Andrew Harp on TF team. Thanks!!!
The above changes were all unnecessary. You can get quantization working for benchmark_model (or any target dependent on android_tensorflow_lib) with the following:
git pull --recurse-submodules (to get #gemmlowp libs, also can git clone --recursive)
The following edit to //tensorflow/core/BUILD
`
diff --git a/tensorflow/core/BUILD b/tensorflow/core/BUILD
--- a/tensorflow/core/BUILD
+++ b/tensorflow/core/BUILD
## -713,8 +713,11 ## cc_library(
# binary size (by packaging a reduced operator set) is a concern.
cc_library(
name = "android_tensorflow_lib",
- srcs = if_android([":android_op_registrations_and_gradients"]),
- copts = tf_copts(),
+ srcs = if_android([":android_op_registrations_and_gradients",
+ "//tensorflow/contrib/quantization:android_ops",
+ "//tensorflow/contrib/quantization/kernels:android_ops",
+ "#gemmlowp//:eight_bit_int_gemm_sources"]),
+ copts = tf_copts() + ["-Iexternal/gemmlowp"],
linkopts = ["-lz"],
tags = [
"manual",
Just tested, works great. Interestingly, quantization produces graphs 1/4 the size but inference execution 4-5x as slow as unquantized graphs - seems like the quantized ops are still being optimized.
i discovered strange behavior of android_cmake and boost. i start cmake using
cmake ../ -DCMAKE_TOOLCHAIN_FILE=/opt/android/android-cmake/android.toolchain.cmake -DBOOST_ROOT=/opt/android/android-cmake/common-libs/boost -DBOOST_INCLUDEDIR=/opt/android/android-cmake/common-libs/boost/boost-trunk/boost -DBOOST_LIBRARYDIR=/opt/android/android-cmake/common-libs/boost/libs/armeabi-v7a -DBoost_DEBUG=1
as you can see all 3 boost variables BOOST_ROOT, BOOST_INCLUDEDIR and BOOST_LIBRARYDIR are set. the boost is found by "find_host_package" function. below is a part of an output:
-- [ /opt/sw/cmake-2.8.10.2/share/cmake-2.8/Modules/FindBoost.cmake:646 ] BOOST_ROOT = /opt/android/android-cmake/common-libs/boost
-- [ /opt/sw/cmake-2.8.10.2/share/cmake-2.8/Modules/FindBoost.cmake:648 ] BOOST_INCLUDEDIR = /opt/android/android-cmake/common-libs/boost/boost-trunk/boost
-- [ /opt/sw/cmake-2.8.10.2/share/cmake-2.8/Modules/FindBoost.cmake:650 ] BOOST_LIBRARYDIR = /opt/android/android-cmake/common-libs/boost/libs/armeabi-v7a
-- [ /opt/sw/cmake-2.8.10.2/share/cmake-2.8/Modules/FindBoost.cmake:705 ] Include debugging info:
-- [ /opt/sw/cmake-2.8.10.2/share/cmake-2.8/Modules/FindBoost.cmake:707 ] _boost_INCLUDE_SEARCH_DIRS = /opt/android/android-cmake/common-libs/boost/boost-trunk/boost;/opt/android/android-cmake/common-libs/boost/include;/opt/android/android-cmake/common-libs/boost;C:/boost/include;C:/boost;/boost/include;/boost;/sw/local/include
-- [ /opt/sw/cmake-2.8.10.2/share/cmake-2.8/Modules/FindBoost.cmake:734 ] location of version.hpp: /usr/local/include/boost/version.hpp
as you can see, boost take version.hpp not from set location, but from
/usr/local/include/boost/version.hpp
even though my specified location does contain version.hpp:
/opt/android/android-cmake/common-libs/boost/boost-trunk/boost/version.hpp
now i cant be sure what is going on and where cmake takes the boost.. any suggestions?
You may need to forcefully exclude the default search,
Boost_NO_SYSTEM_PATHS - Set to ON to disable searching in locations
not specified by these hint variables. Default is OFF.
http://www.cmake.org/cmake/help/git-master/module/FindBoost.html
Try,
cmake ../ -DCMAKE_TOOLCHAIN_FILE=/opt/android/android-cmake/android.toolchain.cmake -DBOOST_ROOT=/opt/android/android-cmake/common-libs/boost -DBOOST_INCLUDEDIR=/opt/android/android-cmake/common-libs/boost/boost-trunk/boost -DBOOST_LIBRARYDIR=/opt/android/android-cmake/common-libs/boost/libs/armeabi-v7a -DBoost_DEBUG=1 -DBoost_NO_SYSTEM_PATHS=ON