Develop Reference

File not Found when compiling android resources (inconsistent across projects, consistent within projects)

I have a project that uses bazel and includes a demo Android app to show some functionality and widgets. I wanted to split this out as the base for a new demo/experimentation app to try out other widget combinations. So I copied the WORKSPACE file and the code from the demo app package into a new project (the package names changed but I modified the code and BUILD files to account for this). The original project builds just fine, but the new one consistently fails with the following error:
ERROR: D:/_bazel_out/s6wwoqrr/external/maven_android_everything/BUILD:568:11: Compiling Android resources for #maven_android_everything//:com
_google_android_material_material failed: (Exit 1): ResourceProcessorBusyBox.exe failed: error executing command bazel-out\x64_windows-opt-ex
ec-2B5CBBC6\bin\external\bazel_tools\src\tools\android\java\com\google\devtools\build\android\ResourceProcessorBusyBox.exe ... (remaining 1 a
rgument skipped)
Jul 29, 2022 3:04:29 PM java.util.stream.ForEachOps$ForEachOp$OfRef accept
SEVERE: Error during Compiling bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_goog
le_android_material_material\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml:
Command: external\androidsdk\build-tools\33.0.0\aapt2.exe\
compile\
-v\
--legacy\
-o\
C:\Users\bdlei\AppData\Local\Temp\android_resources_tmp3839340977225638696\compiled\bazel-out\android-armeabi-v7a-fastbuild\bin\exter
nal\maven_android_everything\_aar\unzipped\resources\com_google_android_material_material\res\
bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_mat
erial\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml
Output:
bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_material\re
s\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml: error: file not found.
Exception in thread "main" com.google.devtools.build.android.aapt2.ResourceCompiler$CompileError
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompileError.of(ResourceCompiler.java:111)
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompilingVisitor.getCompiledArtifacts(ResourceCompiler.java:431)
at com.google.devtools.build.android.aapt2.ResourceCompiler.getCompiledArtifacts(ResourceCompiler.java:465)
at com.google.devtools.build.android.UnvalidatedAndroidData.compile(UnvalidatedAndroidData.java:103)
at com.google.devtools.build.android.CompileLibraryResourcesAction.main(CompileLibraryResourcesAction.java:143)
at com.google.devtools.build.android.ResourceProcessorBusyBox$Tool$6.call(ResourceProcessorBusyBox.java:99)
at com.google.devtools.build.android.ResourceProcessorBusyBox.processRequest(ResourceProcessorBusyBox.java:234)
at com.google.devtools.build.android.ResourceProcessorBusyBox.main(ResourceProcessorBusyBox.java:177)
Suppressed: java.lang.RuntimeException: Error during Compiling bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_eve
rything\_aar\unzipped\resources\com_google_android_material_material\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml:
Command: external\androidsdk\build-tools\33.0.0\aapt2.exe\
compile\
-v\
--legacy\
-o\
C:\Users\bdlei\AppData\Local\Temp\android_resources_tmp3839340977225638696\compiled\bazel-out\android-armeabi-v7a-fastbuild\bin\exter
nal\maven_android_everything\_aar\unzipped\resources\com_google_android_material_material\res\
bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_mat
erial\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml
Output:
bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_material\re
s\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml: error: file not found.
at com.google.devtools.build.android.CommandHelper.execute(CommandHelper.java:42)
at com.google.devtools.build.android.AaptCommandBuilder.execute(AaptCommandBuilder.java:297)
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompileTask.compile(ResourceCompiler.java:234)
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompileTask.call(ResourceCompiler.java:178)
at com.google.devtools.build.android.aapt2.ResourceCompiler$CompileTask.call(ResourceCompiler.java:125)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListe
nableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
INFO: Elapsed time: 335.194s, Critical Path: 145.75s
INFO: 1568 processes: 74 internal, 1308 local, 186 worker.
FAILED: Build did NOT complete successfully
The referenced file: bazel-out\android-armeabi-v7a-fastbuild\bin\external\maven_android_everything\_aar\unzipped\resources\com_google_android_material_mat erial\res\animator\mtrl_extended_fab_change_size_collapse_motion_spec.xml does exist under the bazel-out directory of my project, but after the build failure, the temp directory C:\Users\bdlei\AppData\Local\Temp\android_resources_tmp3839340977225638696 does not exist. I had an explorer window open to my AppData\Local\Temp directory last time I tried building and many such android_resources_temp... came and went during the build (which I expected). I removed older ones before trying the build again and while I wasn't able to track if the one named in the error ever existed, it does not exist afterwards (exactly one of these was left after the build failure).
The behavior is consistent: the original project consistently succeeds and the branched out fails. This persists through bazel clean, bazel sync, and bazel shutdown calls.
Both projects are depending on android via a call into a 3rd "dependencies" project that creates a maven repo and builds a repo out of that:
def android():
if 'maven_android_everything' not in native.existing_rules():
maven_install(
name = "maven_android_everything",
artifacts = [
"androidx.activity:activity:1.2.3", # using 1.4.0 causes indexing issues in dex without force-jumbo, even without dagger-views.
"androidx.appcompat:appcompat:1.3.1",
"androidx.appcompat:appcompat-resources:1.3.1",
"androidx.constraintlayout:constraintlayout:2.1.1",
"androidx.core:core:1.7.0",
"androidx.fragment:fragment:1.3.6",
"androidx.lifecycle:lifecycle-viewmodel:2.3.1",
"androidx.viewpager2:viewpager2:1.0.0",
"androidx.recyclerview:recyclerview:1.0.0",
"androidx.tracing:tracing:1.0.0",
"com.google.android.material:material:1.4.0",
],
excluded_artifacts = [
maven.exclusion(
group = "com.google.guava",
artifact = "guava",
),
],
repositories = [
"https://maven.google.com",
"https://repo1.maven.org/maven2/",
],
)
new_simple_repo( # Simple custom repo rule to make an #android repo for use elsewhere
name = "android",
build_file_content = """
package(default_visibility = ["//visibility:public"])
android_library(
name = "android",
exports = [
"#guava//:concurrent",
"#maven_android_everything//:androidx_activity_activity",
"#maven_android_everything//:androidx_appcompat_appcompat",
"#maven_android_everything//:androidx_appcompat_appcompat_resources",
"#maven_android_everything//:androidx_constraintlayout_constraintlayout",
"#maven_android_everything//:androidx_core_core",
"#maven_android_everything//:androidx_fragment_fragment",
"#maven_android_everything//:androidx_lifecycle_lifecycle_viewmodel",
"#maven_android_everything//:androidx_viewpager2_viewpager2",
"#maven_android_everything//:androidx_recyclerview_recyclerview",
"#maven_android_everything//:com_google_android_material_material",
],
)
""")
I've tried bumping versions to see if maybe that might help, but after slogging through dependency conflicts and finally getting bazel sync to succeed, I hit this bug.
I'm trying to pare my "playgroung" project down and see if I can isolate anything, but even removing a dependency on the original project doesn't help.

I had some time to come back to this today and after much experimenting, including trying to set up a minimal example to use when filing a bug with bazel, I believe I found the problem. It appears to relate to the lengths of resulting file names. By trying various lengths of the name of the maven repo used in the maven_install command, I was able to get the build to fail or pass consistently based on the length. Any name 21 characters or shorter and the build succeeded, while any name longer than 21 characters would result in the file-not-found error.

How to resolve #bazel_tools//tools/android:databinding_exec when switching from bazel 3.6.0 to 3.7.0

I’ve been using bazel to build an android project, initially starting with a 0.24 version, then using the 2.0 version for a longer time, and now I am trying to switch to the latest 3.7.0 / 3.7.1 versions.
Currently I am using the following important .bazelrc settings for android:
build:androidbuild --crosstool_top=#androidndk//:default_crosstool
build:androidbuild --host_crosstool_top=#bazel_tools//tools/cpp:toolchain
build:android-32 --config=androidbuild --cpu=x86 --fat_apk_cpu=x86
All cc_binary / cc_library targets can still be built the same way we used to, but the android_binary targets no longer builds and I am getting this error:
ERROR: While resolving toolchains for target
#bazel_tools//tools/android:databinding_exec: No matching toolchains
found for types #bazel_tools//tools/cpp:toolchain_type. Maybe
--incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See
https://github.com/bazelbuild/bazel/issues/10134 for details and
migration instructions.
I tried to find which was the bazel version that started to throw this error, and I found that I can still build my project with 3.6.0, but I start getting the above error if I switch to 3.7.0 or 3.7.1.
Looking at the 3.6.0 vs 3.7.0 changes, I am wondering if the new create_dummy_sdk_toolchain call in tools/android/android_sdk_repository_template.bzl would cause this failure or there is something else I am missing (like changes in the cc_flags_supplier.bzl / compiler_flag.bzl sources to set incompatible_use_toolchain_transition to True).
I tried the approach of using --platforms instead, but I am still getting errors even in that case:
build:androidbuild --extra_toolchains=#androidndk//:all
build:android-32 --config=androidbuild --cpu=x86 --platforms=//config:android_x86
With:
platform(
name = "android_x86",
constraint_values = [
"#bazel_tools//platforms:x86_32",
"#bazel_tools//platforms:android",
],
cpu_constraints = [
],
os_constraints = [
],
target_platform = True,
)
Error:
ERROR: While resolving toolchains for target
#bazel_tools//tools/android:instrumentation_test_check: No matching
toolchains found for types #bazel_tools//tools/cpp:toolchain_type.
Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped
and there is no default C++ toolchain added in the WORKSPACE file? See
https://github.com/bazelbuild/bazel/issues/10134 for details and
migration instructions.
Can you please advise me on what is missing from the project that is required to solve the problem caused by the 3.6.0 vs 3.7.0 diffs, do we now need to define a proper toolchain for Android (can't we rely anymore on bazel to figure out how to build android targets)?
UPDATE Dec/16/2020
Looks like the problem was caused by this code:
WORKSPACE:
register_execution_platforms(
"//tools/config:host_platform",
)
In the BUILD file:
platform(
name = "host_platform",
constraint_values = [
],
cpu_constraints = [
"#bazel_tools//platforms:x86_32",
"#bazel_tools//platforms:x86_64",
],
host_platform = True,
os_constraints = [
"#bazel_tools//platforms:windows",
],
)
For some reason this works in bazel 3.6.0 even when the host is Linux, but it's no longer working in 3.7.0.

The problem was caused by this code:
WORKSPACE:
register_execution_platforms(
"//tools/config:host_platform",
)
In the BUILD file:
platform(
name = "host_platform",
constraint_values = [
],
cpu_constraints = [
"#bazel_tools//platforms:x86_32",
"#bazel_tools//platforms:x86_64",
],
host_platform = True,
os_constraints = [
"#bazel_tools//platforms:windows",
],
)
For some reason this works in bazel 3.6.0 even when the host is Linux, but it's no longer working in 3.7.0.

when I do bazel build --platforms=//:linux_arm64 it gave me the same error:
hile resolving toolchains for target #io_bazel_rules_go//:cgo_context_data: No matching toolchains found for types #bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.
The follow command works
bazel build --platforms=#io_bazel_rules_go//go/toolchain:linux_arm64

Android NDK: unused import statement with header file

I'm trying to add tracing to my C++ project in Android Studio, I'm quite simply following the example in the docs to create a small profiling library in my app: https://developer.android.com/ndk/reference/group/tracing
I get an 'Unused import statement' message on the "#include " line in Android Studio as well as error: use of undeclared identifier 'ATrace_beginSection' compile errors. My CMakeLists.txt file for the library is:
project(profiling)
if(ANDROID)
include_directories(${ANDROID_SYSROOT}/usr/include)
message(STATUS "Including ${ANDROID_SYSROOT}/usr/include")
endif()
set(profiling_SRCS
profiling.cpp
)
set(profiling_HEADERS
profiling.h
)
add_library(profiling STATIC ${profiling_SRCS} ${profiling_HEADERS})
I've checked the cmake log for the "message" entry above, ANDROID_SYSROOT is indeed pointing to the correct location. The library shows up in the app->cpp section in Android Studio, the CMakeLists.txt file shows up in the External Build Files section, I've tried resyncing Gradle, I've tried cleaning and rebuilding, no dice. My targetSdkVersion is set to 27 in my build.gradle.
What am I missing?

Gaaaah. The problem was the minSdkVersion setting in build.gradle - trace.h has this at the top #if __ANDROID_API__ >= 23. For whatever it's worth, I also don't need to include ${ANDROID_SYSROOT}/usr/include in the CmakeLists.txt for this library... Hopefully this will help someone else in the future.

Your cmake file looks not complete. Try to add the path reference point using ${CMAKE_CURRENT_SOURCE_DIR}. E.g.
cmake_minimum_required(VERSION 3.4.1)
project(profiling)
if(ANDROID)
include_directories(${ANDROID_SYSROOT}/usr/include)
message(STATUS "Including ${ANDROID_SYSROOT}/usr/include")
endif()
set(profiling_SRCS
${CMAKE_CURRENT_SOURCE_DIR}/src/main/cpp/profiling.cpp
)
set(profiling_HEADERS
${CMAKE_CURRENT_SOURCE_DIR}/src/main/cpp/profiling.h
)
add_library(profiling STATIC ${profiling_SRCS} ${profiling_HEADERS})

Benchmarking Quantization on Android

I've been benchmarking tensorflow models on Exynos 7420 with benchmark_model. I'd like to speed test Quantization per Pete Warden's blog but have not been able to compile benchmark_model with quantization deps yet as they break a number of things.
I've followed the guidelines listed in this stack overflow thread:
//tensorflow/tools/benchmark/BUILD cc_binary
deps = [":benchmark_model_lib",
"//tensorflow/contrib/quantization/kernels:quantized_ops",
],
//tensorflow/contrib/quantization/kernels/BUILD:
deps = [
"//tensorflow/contrib/quantization:cc_array_ops",
"//tensorflow/contrib/quantization:cc_math_ops",
"//tensorflow/contrib/quantization:cc_nn_ops",
#"//tensorflow/core",
#"//tensorflow/core:framework",
#"//tensorflow/core:lib",
#"//tensorflow/core/kernels:concat_lib_hdrs",
#"//tensorflow/core/kernels:conv_ops",
#"//tensorflow/core/kernels:eigen_helpers",
#"//tensorflow/core/kernels:ops_util",
#"//tensorflow/core/kernels:pooling_ops",
"//third_party/eigen3",
"#gemmlowp//:eight_bit_int_gemm",
],
Then run:
bazel build -c opt --cxxopt='-std=gnu++11'--crosstool_top=//external:android/crosstool --cpu=armeabi-v7a --host_crosstool_top=#bazel_tools//tools/cpp:toolchain tensorflow/tools/benchmark:benchmark_model --verbose_failures
Which (with following all other instructions in linked post) succeeds with the exception that it fails to link against pthread.
I've tried removing -lpthread in tensorflow/tensorflow.bzl tfcopts(), and similarly in tensorflow/tools/proto_text/BUILD, and tensorflow/cc/BUILD.
def tf_copts():
return (["-fno-exceptions", "-DEIGEN_AVOID_STL_ARRAY"] +
if_cuda(["-DGOOGLE_CUDA=1"]) +
if_android_arm(["-mfpu=neon"]) +
select({"//tensorflow:android": [
"-std=c++11",
"-DMIN_LOG_LEVEL=0",
"-DTF_LEAN_BINARY",
"-O2",
],
"//tensorflow:darwin": [],
"//tensorflow:ios": ["-std=c++11",],
#"//conditions:default": ["-lpthread"]}))
"//conditions:default": []}))
Still getting getting the link error.
external/androidndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/../lib/gcc/arm-linux-androideabi/4.9/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lpthread
collect2: error: ld returned 1 exit status
Any help much appreciated, I'm pretty stuck.
Env:
Ubuntu 14.04
tensorflow commit #4462
android_ndk_r11c
android-sdk-linux r24.4.1
Python 2.7.12 :: Continuum Analytics, Inc.
./configure without support for GCP, HDFS, or GPU

Transcribing GitHub answer from Andrew Harp on TF team. Thanks!!!
The above changes were all unnecessary. You can get quantization working for benchmark_model (or any target dependent on android_tensorflow_lib) with the following:
git pull --recurse-submodules (to get #gemmlowp libs, also can git clone --recursive)
The following edit to //tensorflow/core/BUILD
`
diff --git a/tensorflow/core/BUILD b/tensorflow/core/BUILD
--- a/tensorflow/core/BUILD
+++ b/tensorflow/core/BUILD
## -713,8 +713,11 ## cc_library(
# binary size (by packaging a reduced operator set) is a concern.
cc_library(
name = "android_tensorflow_lib",
- srcs = if_android([":android_op_registrations_and_gradients"]),
- copts = tf_copts(),
+ srcs = if_android([":android_op_registrations_and_gradients",
+ "//tensorflow/contrib/quantization:android_ops",
+ "//tensorflow/contrib/quantization/kernels:android_ops",
+ "#gemmlowp//:eight_bit_int_gemm_sources"]),
+ copts = tf_copts() + ["-Iexternal/gemmlowp"],
linkopts = ["-lz"],
tags = [
"manual",
Just tested, works great. Interestingly, quantization produces graphs 1/4 the size but inference execution 4-5x as slow as unquantized graphs - seems like the quantized ops are still being optimized.

Tensorflow Quantized Graph for Android

I'm trying to load a quantized graph into an Android app.
My BUILD file contains
deps = ["//tensorflow/core:android_tensorflow_lib",
"//tensorflow/contrib/quantization:cc_array_ops",
"//tensorflow/contrib/quantization:cc_math_ops",
"//tensorflow/contrib/quantization:cc_nn_ops",
"//tensorflow/contrib/quantization/kernels:quantized_ops"]
The additional quantization deps work for standalone C++ builds.
I can't compile with Bazel, due to a large number of errors in GEMMLOWP. What's the proper way to include gemmlowp and the quantization ops in Android?
Here is an example error:
external/gemmlowp/eight_bit_int_gemm/eight_bit_int_gemm.cc:125:13: error: 'int32_t' is not a member of 'std'
MatrixMap<std::int32_t, ResultOrder> result(c, m, n, ldc);
This is on Ubuntu 16.04 with Bazel 0.3.0.
Here's a gist that has the output of two sequential attempts to build the package - it fails on highwayhash the first time and gemmlowp the second.
https://gist.github.com/ericdanz/81b799f2e0bbb3cc462aa3c90468c71b
Ultimately got it to compile and run with liberal addition of "-std=c++11" in BUILD files for gemmlowp and highwayhash, and substitution of the android framework for the framework dependencies in the quantized ops. It produces fairly different results though, and runs about 4x slower (26-3200ms vs 6-800 ms). I'll try to do a little deeper investigation.

Here's what worked for me - it is basically a combination of all the comments from Eric D above, but I wanted to put it all in one place for someone new who comes across this problem:
Add quantized_ops in deps to libtensorflow_demo.so in the BUILD file for the Android app:
deps = ["//tensorflow/core:android_tensorflow_lib",
"//tensorflow/contrib/quantization/kernels:quantized_ops",]
Modify the deps for quantized_ops in tensorflow/contrib/quantization/kernels/BUILD:
deps = [
"//tensorflow/contrib/quantization:cc_array_ops",
"//tensorflow/contrib/quantization:cc_math_ops",
"//tensorflow/contrib/quantization:cc_nn_ops",
"//tensorflow/core:android_tensorflow_lib",
#"//tensorflow/core",
#"//tensorflow/core:framework",
#"//tensorflow/core:lib",
#"//tensorflow/core/kernels:concat_lib_hdrs",
#"//tensorflow/core/kernels:conv_ops",
#"//tensorflow/core/kernels:ops_util_hdrs",
#"//tensorflow/core/kernels:pooling_ops_hdrs",
#"//tensorflow/core/kernels:eigen_helpers",
#"//tensorflow/core/kernels:ops_util",
#"//tensorflow/core/kernels:pooling_ops",
"//third_party/eigen3",
"#gemmlowp//:eight_bit_int_gemm",
],
Remove/comment out the .Doc() parts in tensorflow/contrib/quantization/ops/array_ops.cc, math_ops.cc and nn_ops.cc
Modify the deps for cc_array_ops, cc_math_ops and cc_nn_ops in tensorflow/contrib/quantization/BUILD:
deps = [
#"//tensorflow/core:framework",
"//tensorflow/core:android_tensorflow_lib",
],
Run bazel build command for Android app with --cxxopt="-std=c++11" flag.