Develop Reference

File not Found when compiling android resources (inconsistent across projects, consistent within projects)

I have a project that uses bazel and includes a demo Android app to show some functionality and widgets. I wanted to split this out as the base for a new demo/experimentation app to try out other widget combinations. So I copied the WORKSPACE file and the code from the demo app package into a new project (the package names changed but I modified the code and BUILD files to account for this). The original project builds just fine, but the new one consistently fails with the following error:
ERROR: D:/_bazel_out/s6wwoqrr/external/maven_android_everything/BUILD:568:11: Compiling Android resources for #maven_android_everything//:com
_google_android_material_material failed: (Exit 1): ResourceProcessorBusyBox.exe failed: error executing command bazel-out\x64_windows-opt-ex
ec-2B5CBBC6\bin\external\bazel_tools\src\tools\android\java\com\google\devtools\build\android\ResourceProcessorBusyBox.exe ... (remaining 1 a
rgument skipped)
Jul 29, 2022 3:04:29 PM java.util.stream.ForEachOps$ForEachOp$OfRef accept
SEVERE: Error during Compiling bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_goog
le_android_material_material\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml:
Command: external\androidsdk\build-tools\33.0.0\aapt2.exe\
compile\
-v\
--legacy\
-o\
C:\Users\bdlei\AppData\Local\Temp\android_resources_tmp3839340977225638696\compiled\bazel-out\android-armeabi-v7a-fastbuild\bin\exter
nal\maven_android_everything\_aar\unzipped\resources\com_google_android_material_material\res\
bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_mat
erial\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml
Output:
bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_material\re
s\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml: error: file not found.
Exception in thread "main" com.google.devtools.build.android.aapt2.ResourceCompiler$CompileError
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompileError.of(ResourceCompiler.java:111)
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompilingVisitor.getCompiledArtifacts(ResourceCompiler.java:431)
at com.google.devtools.build.android.aapt2.ResourceCompiler.getCompiledArtifacts(ResourceCompiler.java:465)
at com.google.devtools.build.android.UnvalidatedAndroidData.compile(UnvalidatedAndroidData.java:103)
at com.google.devtools.build.android.CompileLibraryResourcesAction.main(CompileLibraryResourcesAction.java:143)
at com.google.devtools.build.android.ResourceProcessorBusyBox$Tool$6.call(ResourceProcessorBusyBox.java:99)
at com.google.devtools.build.android.ResourceProcessorBusyBox.processRequest(ResourceProcessorBusyBox.java:234)
at com.google.devtools.build.android.ResourceProcessorBusyBox.main(ResourceProcessorBusyBox.java:177)
Suppressed: java.lang.RuntimeException: Error during Compiling bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_eve
rything\_aar\unzipped\resources\com_google_android_material_material\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml:
Command: external\androidsdk\build-tools\33.0.0\aapt2.exe\
compile\
-v\
--legacy\
-o\
C:\Users\bdlei\AppData\Local\Temp\android_resources_tmp3839340977225638696\compiled\bazel-out\android-armeabi-v7a-fastbuild\bin\exter
nal\maven_android_everything\_aar\unzipped\resources\com_google_android_material_material\res\
bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_mat
erial\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml
Output:
bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_material\re
s\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml: error: file not found.
at com.google.devtools.build.android.CommandHelper.execute(CommandHelper.java:42)
at com.google.devtools.build.android.AaptCommandBuilder.execute(AaptCommandBuilder.java:297)
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompileTask.compile(ResourceCompiler.java:234)
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompileTask.call(ResourceCompiler.java:178)
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompileTask.call(ResourceCompiler.java:125)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListe
nableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
INFO: Elapsed time: 335.194s, Critical Path: 145.75s
INFO: 1568 processes: 74 internal, 1308 local, 186 worker.
FAILED: Build did NOT complete successfully
The referenced file: bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_mat erial\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml does exist under the bazel-out directory of my project, but after the build failure, the temp directory C:\Users\bdlei\AppData\Local\Temp\android_resources_tmp3839340977225638696 does not exist. I had an explorer window open to my AppData\Local\Temp directory last time I tried building and many such android_resources_temp... came and went during the build (which I expected). I removed older ones before trying the build again and while I wasn't able to track if the one named in the error ever existed, it does not exist afterwards (exactly one of these was left after the build failure).
The behavior is consistent: the original project consistently succeeds and the branched out fails. This persists through bazel clean, bazel sync, and bazel shutdown calls.
Both projects are depending on android via a call into a 3rd "dependencies" project that creates a maven repo and builds a repo out of that:
def android():
if 'maven_android_everything' not in native.existing_rules():
maven_install(
name = "maven_android_everything",
artifacts = [
"androidx.activity:activity:1.2.3", # using 1.4.0 causes indexing issues in dex without force-jumbo, even without dagger-views.
"androidx.appcompat:appcompat:1.3.1",
"androidx.appcompat:appcompat-resources:1.3.1",
"androidx.constraintlayout:constraintlayout:2.1.1",
"androidx.core:core:1.7.0",
"androidx.fragment:fragment:1.3.6",
"androidx.lifecycle:lifecycle-viewmodel:2.3.1",
"androidx.viewpager2:viewpager2:1.0.0",
"androidx.recyclerview:recyclerview:1.0.0",
"androidx.tracing:tracing:1.0.0",
"com.google.android.material:material:1.4.0",
],
excluded_artifacts = [
maven.exclusion(
group = "com.google.guava",
artifact = "guava",
),
],
repositories = [
"https://maven.google.com",
"https://repo1.maven.org/maven2/",
],
)
new_simple_repo( # Simple custom repo rule to make an #android repo for use elsewhere
name = "android",
build_file_content = """
package(default_visibility = ["//visibility:public"])
android_library(
name = "android",
exports = [
"#guava//:concurrent",
"#maven_android_everything//:androidx_activity_activity",
"#maven_android_everything//:androidx_appcompat_appcompat",
"#maven_android_everything//:androidx_appcompat_appcompat_resources",
"#maven_android_everything//:androidx_constraintlayout_constraintlayout",
"#maven_android_everything//:androidx_core_core",
"#maven_android_everything//:androidx_fragment_fragment",
"#maven_android_everything//:androidx_lifecycle_lifecycle_viewmodel",
"#maven_android_everything//:androidx_viewpager2_viewpager2",
"#maven_android_everything//:androidx_recyclerview_recyclerview",
"#maven_android_everything//:com_google_android_material_material",
],
)
""")
I've tried bumping versions to see if maybe that might help, but after slogging through dependency conflicts and finally getting bazel sync to succeed, I hit this bug.
I'm trying to pare my "playgroung" project down and see if I can isolate anything, but even removing a dependency on the original project doesn't help.

I had some time to come back to this today and after much experimenting, including trying to set up a minimal example to use when filing a bug with bazel, I believe I found the problem. It appears to relate to the lengths of resulting file names. By trying various lengths of the name of the maven repo used in the maven_install command, I was able to get the build to fail or pass consistently based on the length. Any name 21 characters or shorter and the build succeeded, while any name longer than 21 characters would result in the file-not-found error.

How to resolve #bazel_tools//tools/android:databinding_exec when switching from bazel 3.6.0 to 3.7.0

I’ve been using bazel to build an android project, initially starting with a 0.24 version, then using the 2.0 version for a longer time, and now I am trying to switch to the latest 3.7.0 / 3.7.1 versions.
Currently I am using the following important .bazelrc settings for android:
build:androidbuild --crosstool_top=#androidndk//:default_crosstool
build:androidbuild --host_crosstool_top=#bazel_tools//tools/cpp:toolchain
build:android-32 --config=androidbuild --cpu=x86 --fat_apk_cpu=x86
All cc_binary / cc_library targets can still be built the same way we used to, but the android_binary targets no longer builds and I am getting this error:
ERROR: While resolving toolchains for target
#bazel_tools//tools/android:databinding_exec: No matching toolchains
found for types #bazel_tools//tools/cpp:toolchain_type. Maybe
--incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See
https://github.com/bazelbuild/bazel/issues/10134 for details and
migration instructions.
I tried to find which was the bazel version that started to throw this error, and I found that I can still build my project with 3.6.0, but I start getting the above error if I switch to 3.7.0 or 3.7.1.
Looking at the 3.6.0 vs 3.7.0 changes, I am wondering if the new create_dummy_sdk_toolchain call in tools/android/android_sdk_repository_template.bzl would cause this failure or there is something else I am missing (like changes in the cc_flags_supplier.bzl / compiler_flag.bzl sources to set incompatible_use_toolchain_transition to True).
I tried the approach of using --platforms instead, but I am still getting errors even in that case:
build:androidbuild --extra_toolchains=#androidndk//:all
build:android-32 --config=androidbuild --cpu=x86 --platforms=//config:android_x86
With:
platform(
name = "android_x86",
constraint_values = [
"#bazel_tools//platforms:x86_32",
"#bazel_tools//platforms:android",
],
cpu_constraints = [
],
os_constraints = [
],
target_platform = True,
)
Error:
ERROR: While resolving toolchains for target
#bazel_tools//tools/android:instrumentation_test_check: No matching
toolchains found for types #bazel_tools//tools/cpp:toolchain_type.
Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped
and there is no default C++ toolchain added in the WORKSPACE file? See
https://github.com/bazelbuild/bazel/issues/10134 for details and
migration instructions.
Can you please advise me on what is missing from the project that is required to solve the problem caused by the 3.6.0 vs 3.7.0 diffs, do we now need to define a proper toolchain for Android (can't we rely anymore on bazel to figure out how to build android targets)?
UPDATE Dec/16/2020
Looks like the problem was caused by this code:
WORKSPACE:
register_execution_platforms(
"//tools/config:host_platform",
)
In the BUILD file:
platform(
name = "host_platform",
constraint_values = [
],
cpu_constraints = [
"#bazel_tools//platforms:x86_32",
"#bazel_tools//platforms:x86_64",
],
host_platform = True,
os_constraints = [
"#bazel_tools//platforms:windows",
],
)
For some reason this works in bazel 3.6.0 even when the host is Linux, but it's no longer working in 3.7.0.

The problem was caused by this code:
WORKSPACE:
register_execution_platforms(
"//tools/config:host_platform",
)
In the BUILD file:
platform(
name = "host_platform",
constraint_values = [
],
cpu_constraints = [
"#bazel_tools//platforms:x86_32",
"#bazel_tools//platforms:x86_64",
],
host_platform = True,
os_constraints = [
"#bazel_tools//platforms:windows",
],
)
For some reason this works in bazel 3.6.0 even when the host is Linux, but it's no longer working in 3.7.0.

when I do bazel build --platforms=//:linux_arm64 it gave me the same error:
hile resolving toolchains for target #io_bazel_rules_go//:cgo_context_data: No matching toolchains found for types #bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.
The follow command works
bazel build --platforms=#io_bazel_rules_go//go/toolchain:linux_arm64

Are linker flags (linkopts) propagats in Bazel?

I'm building an Android library based on Tensorflow with Bazel.
Here the BUILD file
cc_binary(
name = "libfoo.so",
srcs = glob([
"jni/**/*.cc",
"jni/**/*.h",
]),
copts = [ "-fexceptions", "-DEIGEN_AVOID_STL_ARRAY",
"-mfpu=neon", "-std=c++11",
"-DMIN_LOG_LEVEL=0", "-DTF_LEAN_BINARY",
"-O2", ],
linkopts = [
"-llog",
"-lm",
],
linkshared = 1,
deps = [
"#org_tensorflow//tensorflow/core:android_tensorflow_lib",
"#boringssl//:crypto",
],
)
The linker complains about not finding -lpthread, while I didn't add this flag to linkopts.
I've checked the executed command, and in fact there is extra flags on it: -lz -lpthread ...
Where did they came from ? Is there a workaround for this ?

I got the answer from the tensorflow's issue tracker.
#jart
Since copts and linkopts are viral and propagate to dependencies, -lpthread is most likely being inherited from #boringssl//:crypto
#andrewharp
-lpthread is not necessary or possible on Android, so it sounds like the solution would be to add another condition for the select statement as in the linked commit google/protobuf#1386:
...
The only other workaround I know of that doesn't require editing the other repository is to create a dummy libpthread.so target, but that's pretty hacky.

Tensorflow: How to compile libtensorflow_cc.so for Android

I'm currently trying to compile the target //tensorflow:libtensorflow_cc.so of TensorFlow with bazel for Android. I need this library in order to get the javacpp-presets for TensorFlow working with Android.
I tried the following statement:
bazel build -c opt //tensorflow:libtensorflow_cc.so --crosstool_top=//external:android/crosstool --cpu=armeabi-v7a --host_crosstool_top=#bazel_tools//tools/cpp:toolchain --verbose_failures
Which however results in the error that S_IREAD, S_IWRITE cannot be found:
external/gif_archive/giflib-5.1.4/lib/egif_lib.c:62:6: error: 'S_IREAD' undeclared (first use in this function)
S_IREAD | S_IWRITE);
^
external/gif_archive/giflib-5.1.4/lib/egif_lib.c:62:6: note: each undeclared identifier is reported only once for each function it appears in
external/gif_archive/giflib-5.1.4/lib/egif_lib.c:62:16: error: 'S_IWRITE' undeclared (first use in this function)
S_IREAD | S_IWRITE);
^
Target //tensorflow:libtensorflow_cc.so failed to build
Inspired by the Android build in the Android Demo build, I also tried to change the cc_binary definition to the following code, but still got the same error.
cc_binary(
name = "libtensorflow_cc.so",
copts = tf_copts(),
linkopts = [
"-landroid",
"-ljnigraphics",
"-llog",
"-lm",
"-z defs",
"-s",
"-Wl,--icf=all", # Identical Code Folding
],
linkshared = 1,
linkstatic = 1,
deps = [
"//tensorflow/c:c_api",
"//tensorflow/cc:cc_ops",
"//tensorflow/core:tensorflow",
],
)
From googling, I found out that the S_IWRITE flags are deprecated and therefore have never been implemented in Android. However, I have no idea how to get around this problem.
To sum it up: Do you know how I can build the libtensorflow_cc.so target for Android? The library build in the Android example is not enough for me because I also need the cc_ops included.

From googling, I found out that the S_IWRITE flags are deprecated and therefore have never been implemented in Android.
It looks like we've changed our minds on that for the sake of compatibility: https://android.googlesource.com/platform/bionic/+/1f1a51aecd7c825418bfedcb66772e92de790149%5E%21/#F2
#if defined(__USE_BSD) || defined(__USE_GNU)
#define S_IREAD S_IRUSR
#define S_IWRITE S_IWUSR
#define S_IEXEC S_IXUSR
#endif
That's the system's sys/stat.h; it hasn't been shipped in the NDK yet. Unfortunately most of the NDK headers are very out of date. This is https://github.com/android-ndk/ndk/issues/120.
We'll get a fix for this into NDK r14 (I just filed https://github.com/android-ndk/ndk/issues/211 to fix up the old headers in case #120 doesn't get fixed by then).
Until then, you could add those defines to your cflags. Looks like the way to do this in bazel would be:
cc_binary(
name = "libtensorflow_cc.so",
defines = [
"S_IREAD=S_IRUSR",
"S_IWRITE=S_IWUSR",
"S_IEXEC=S_IXUSR",
],
...
)
https://www.bazel.io/versions/master/docs/be/c-cpp.html#cc_binary.defines

Benchmarking Quantization on Android

I've been benchmarking tensorflow models on Exynos 7420 with benchmark_model. I'd like to speed test Quantization per Pete Warden's blog but have not been able to compile benchmark_model with quantization deps yet as they break a number of things.
I've followed the guidelines listed in this stack overflow thread:
//tensorflow/tools/benchmark/BUILD cc_binary
deps = [":benchmark_model_lib",
"//tensorflow/contrib/quantization/kernels:quantized_ops",
],
//tensorflow/contrib/quantization/kernels/BUILD:
deps = [
"//tensorflow/contrib/quantization:cc_array_ops",
"//tensorflow/contrib/quantization:cc_math_ops",
"//tensorflow/contrib/quantization:cc_nn_ops",
#"//tensorflow/core",
#"//tensorflow/core:framework",
#"//tensorflow/core:lib",
#"//tensorflow/core/kernels:concat_lib_hdrs",
#"//tensorflow/core/kernels:conv_ops",
#"//tensorflow/core/kernels:eigen_helpers",
#"//tensorflow/core/kernels:ops_util",
#"//tensorflow/core/kernels:pooling_ops",
"//third_party/eigen3",
"#gemmlowp//:eight_bit_int_gemm",
],
Then run:
bazel build -c opt --cxxopt='-std=gnu++11'--crosstool_top=//external:android/crosstool --cpu=armeabi-v7a --host_crosstool_top=#bazel_tools//tools/cpp:toolchain tensorflow/tools/benchmark:benchmark_model --verbose_failures
Which (with following all other instructions in linked post) succeeds with the exception that it fails to link against pthread.
I've tried removing -lpthread in tensorflow/tensorflow.bzl tfcopts(), and similarly in tensorflow/tools/proto_text/BUILD, and tensorflow/cc/BUILD.
def tf_copts():
return (["-fno-exceptions", "-DEIGEN_AVOID_STL_ARRAY"] +
if_cuda(["-DGOOGLE_CUDA=1"]) +
if_android_arm(["-mfpu=neon"]) +
select({"//tensorflow:android": [
"-std=c++11",
"-DMIN_LOG_LEVEL=0",
"-DTF_LEAN_BINARY",
"-O2",
],
"//tensorflow:darwin": [],
"//tensorflow:ios": ["-std=c++11",],
#"//conditions:default": ["-lpthread"]}))
"//conditions:default": []}))
Still getting getting the link error.
external/androidndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lpthread
collect2: error: ld returned 1 exit status
Any help much appreciated, I'm pretty stuck.
Env:
Ubuntu 14.04
tensorflow commit #4462
android_ndk_r11c
android-sdk-linux r24.4.1
Python 2.7.12 :: Continuum Analytics, Inc.
./configure without support for GCP, HDFS, or GPU

Transcribing GitHub answer from Andrew Harp on TF team. Thanks!!!
The above changes were all unnecessary. You can get quantization working for benchmark_model (or any target dependent on android_tensorflow_lib) with the following:
git pull --recurse-submodules (to get #gemmlowp libs, also can git clone --recursive)
The following edit to //tensorflow/core/BUILD
`
diff --git a/tensorflow/core/BUILD b/tensorflow/core/BUILD
--- a/tensorflow/core/BUILD
+++ b/tensorflow/core/BUILD
## -713,8 +713,11 ## cc_library(
# binary size (by packaging a reduced operator set) is a concern.
cc_library(
name = "android_tensorflow_lib",
- srcs = if_android([":android_op_registrations_and_gradients"]),
- copts = tf_copts(),
+ srcs = if_android([":android_op_registrations_and_gradients",
+ "//tensorflow/contrib/quantization:android_ops",
+ "//tensorflow/contrib/quantization/kernels:android_ops",
+ "#gemmlowp//:eight_bit_int_gemm_sources"]),
+ copts = tf_copts() + ["-Iexternal/gemmlowp"],
linkopts = ["-lz"],
tags = [
"manual",
Just tested, works great. Interestingly, quantization produces graphs 1/4 the size but inference execution 4-5x as slow as unquantized graphs - seems like the quantized ops are still being optimized.