I am building an Android library that contains a native part written using STL that is hidden under Java wrapper. In NDK documentation in a section Static runtimes it is stated that I should use shared runtime to avoid problems with multiple static runtimes, but I want to try using gnustl_static with a flag -fvisibility=hidden set.
The question is, will it solve possible problems or a shared library is the only way out?
No, building with -fvisibility=hidden will not help with the problems created by linking multiple libraries to a static C++ runtime.
Unless you are building gnustl_static yourself, the -fvisibility=hidden flag only applies to your native code which depends on gnustl_static. It won't change how the symbols are exported from gnustl_static.
In addition, if you link two shared libraries–or a shared library and
an executable– against the same static runtime, the final binary image
of each shared library includes a copy of the runtime's code. Having
multiple instances of runtime code is problematic because of
duplication of certain global variables that the runtime uses or
provides internally.
However, linking against a static C++ runtime should work perfectly fine as long as you are only creating one shared library.
This problem does not apply to a project comprising a single shared
library. For example, you can link against stlport_static, and expect
your app to behave correctly. If your project requires several shared
library modules, we recommend that you use the shared library variant
of your C++ runtime.
Related
We have a project that uses a library that is built on top of Google's Mediapipe, which is built using the Bazel build system.
The project itself is an Android Native Library, built using Gradle with CMake
externalNativeBuild {
cmake {
cppFlags "-std=c++17 -fopenmp -static-openmp -fexceptions -frtti -Wl,-s -Wno-unused-command-line-argument"
arguments "-DANDROID_STL=c++_shared", "-DOpenCV_DIR=${opencvDir}", "-DANDROID_ARM_NEON=TRUE"
}
}
So we end up with 2 (or more later, also dependent on OpenCV for example) shared object libraries - the actual SDK & the Mediapipe project.
We're seeing issues that are similar to this, which lead me to look into the runtime part of our project.
E/libc++abi: terminating with uncaught exception of type std::bad_cast: std::bad_cast
I saw this comment on that issue thread, and adding
System.loadLibrary("c++_shared");
Solved the crash.
However, this is not a practical solution as the project we're building would provide a native SDK in the form of multiple .so files and I wouldn't want to force our clients to have to explicitly load the shared runtime library before using our library.
The gradle library has "-DANDROID_STL=c++_shared" flag, so this is using the shared one, but I couldn't find any way to compile Mediapipe (with Bazel) using c++_shared. I couldn't find any reference to using shared runtime when compiling Bazel projects (except for this, which isn't exactly relevant and the solution didn't help me)
We might be able to work around this by setting -DANDROID_STL=c++_static, but this has other issues, mainly, it violates Android's guidelines for using multiple shared libraries, though it might be possible for for middleware vendors
So the question is,
Is it possible to build Mediapipe (or any other Bazel based) using c++_shared Android STL
If not, are there any other options to solve the runtime conflicts
Is it even a runtime conflict or something else?
I managed to get it working as suggested by using c++_static on all of our shared objects (SDK, Mediapipe, OpenCV and others)
I'm working on an Android app that uses some 3rd party native libraries. We need to add some functionality to these libraries. So I wrote a set of C++ classes with the code that I need and also cross-compile openssl, as we are using openssl crypto functions.
I made a demo app to test my code and everything is working fine.
Now I need to send this code to the people that are developing the 3rd party libraries, and they will add our code and wrap all the native code in a JAR for us to use it in our app.
I used this example NDK hello libs as a base. As I want to send my code as a shared library (.so file), I did this in the 'CMakeList.txt':
cmake_minimum_required(VERSION 3.4.1)
set(distribution_DIR ${CMAKE_SOURCE_DIR}/../distribution)
add_library(lib_crypto STATIC IMPORTED)
set_target_properties(lib_crypto PROPERTIES IMPORTED_LOCATION
${distribution_DIR}/openssl/lib/${ANDROID_ABI}/libcrypto.a)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=gnu++11")
### This code just calls my my-lib ###
add_library(native-lib SHARED
src/main/cpp/native-lib.cpp)
### This is the lib I want to build ###
add_library(my-lib SHARED
src/main/cpp/*****.cpp
# more files...
)
target_include_directories(my-lib PRIVATE ${distribution_DIR}/openssl/include)
target_link_libraries(
native-lib
my-lib
)
target_link_libraries(
neclic
lib_crypto
)
I built the APK and took 'my-lib.so' from it. Then I made a new project using this lib (and including the .h headers) and everything is working as expected.
My questions is:
Is there a better way of doing this? May I expect any problem when I send this library to be integrated with the original code? For example, do they need to use the same NDK version? Or 'my-lib.so' can be used in any android app if the arch is supported?
Thanks!
Generally speaking, you don't need to send your libneclic.so to be packed in a JAR. You can use it in your APK as is, unless the 3rd party libraries that your collegues develop outside, need your library as dependency.
Actually, the supported way to disribute compiled Java libraries with native components are not JAR, but rather AAR files. But this is a technicality that is not important for this discussion.
If you use your libneclic.so directly in your APK, that is, you have Java class(es) that uses JNI to call functions in your library, you can keep the same CMakeList.txt in your project, or you can put the the built library in JniLibs folder, and Android Studio will pick it from there when it packages the APK.
On the other hand, if the library you built has exported functions that will be called from some 3rd party library, make sure that both use same shared STL (luckily, with the latest NDK release 18, there is only one shared STL implementation available, c++_shared.
It is highly recommended that the libraries are built with same NDK version, even though, most likely, moderate difference between these versions will not cause negative results. Each NDK release brings significant improvements, not only in performance, but also fixes bugs, including some security ones.
Still, it is safer to agree on a common older release of NDK, than to keep track of all possible inconsistencies.
Note that by default, CMake uses c++_static STL to build your libneclic.so, and you must override that, by setting ANDROID_STL.
I'm compiling a 3rd-party Java library for Android that uses JNI. I read the relevant pages on adding C++ support on developer.android but I'm still confused about a couple of issues regarding C++ STL runtime that I was hoping I could clear up here:
1- My library has no control over the app it will be embedded in, so I don't know if there will be other libraries that might use a static/shared STLs. If I use a static C++ runtime with ANDROID_STL=c++_static, is it safe, or should I have to worry about another library that could be using something like gnustl_static which might conflict with mine?
2- If I use a shared C++ runtime with ANDROID_STL=c++_shared, is it a guarantee that a specific element in the STL will use the libc++ runtime or could it be possible to use gnustl if it doesn't exist? For example, If I was using std::string with a shared c++ runtime (c++_shared) in an app that has another library of gnustl_static, will my std::string implementation be taken from libc++ or gnustl?
Ideally, I'd like to have a very stripped down version of a static c++ runtime with (c++_static) that only includes std::vector, std::string and std::map. I was actually planning to use something like -ffunction-sections as described here and #768.
Please advise and thank you.
Environment Details
Pkg.Desc = Android NDK
Pkg.Revision = r15c
Android Studio = 3.1.2
system: cmake Host OS: Arch Linux ($ uname -r % 4.18.5-arch1-1-ARCH)
Compiler: Clang++
STL: c++_static/c++_shared
Your concern is a very real one. But if handled properly, you can find a robust way out.
The warnings about using single C++ runtime across all libraries in the app (and the whole idea to define C++ support in NDK as APP_STL vs. most other flags such as LOCAL_CFLAGS or LOCAL_SHARED_LIBRARIES, are relevant for the native libraries that are connected. JNI libraries that never communicate directly (except through their corresponding Java layers) can use different C++ runtimes. Another point is that normal build will only package one C++ runtime shared lib into the APK. Note that versioning is also a potential hazard: if a developer who adds your library uses a different NDK release, there might be collisions or unexpected side effects when his version of STL runtime works with your code.
Therefore, to achieve maximum flexibility, your library should use a static C++ runtime. This may effect the size of the binary, but if, as you say, you use only a limited subset of STL, this extra will be rather small.
The bottom line, you will have minimum to worry about if build your shared library with libc++_static.
I am using a number of static pre-built static libraries in my native android application and everything works fine. Now I want to switch one of my static libraries to be .so. I was successfully able to build .so library by replacing BUILD_STATIC_LIBRARY with BUILD_SHARED_LIBRARY in its android.mk and adding required dependencies.
I was also able to build my application by replacing corresponding PREBUILT_STATIC_LIBRARY with PREBUILT_SHARED_LIBRARY in its android.mk. The resulting application now fails to start. I cannot even get to point where debugger attaches to the application.
Besides that what I do not understand is how the build system knows that the function should be imported from the library. My so library should export one function, but I did not declare it as dllexport/import or something. Still there are no unresolved symbols in my application (when I remove my prebuilt library from the list, the unresolved symbol appears as expected).
The other question is that I see there are two .so files generated. One big file in obj/local/$(TARGET_ARCH_ABI) folder and another small one in libs/$(TARGET_ARCH_ABI). When declaring my prebuilt library I reference the second one in libs folder.
I did try to search stackoverflow for answers and found quite a few related posts:
loading library (.so file) in android
NDK - How to use a generated .so library in another project
How to use .so file in Android code to use the native methods
How to use libffmpeg.so in Android project?
but I do not see how these posts related to my problem since I can successfully build and even link my application.
You need to load the libraries in reverse dependency order in the java code. You previously probably have something like this:
System.loadLibrary("mylib");
Now if your prebuilt library (that was previously a static library, now a shared library) is named dependencylib, you need to change the code for loading the libraries into this:
System.loadLibrary("dependencylib");
System.loadLibrary("mylib");
As for your question how the linker can figure it out; when linking libmylib.so, it looks for all undefined symbols in all the other libraries you specified (i.e. in libdependencylib.so, and in libc.so and other system libraries). As long as all undefined symbols are found somewhere, the linker is ok. Then at runtime, when libmylib.so is loaded, it does the same routine again; all undefined symbols are looked up in the list of symbols loaded in the current process. On linux, you normally don't need to manually mark symbols as dllexport as you do on windows - all non-static symbols are exported by default.
There may be two reasons why the app fails to start after the change of STATIC -> SHARED.
The prebuilt library is not installed. With your device connected, run adb ls -l /data/your.package.name/lib/. Do you see the library there?
The prebuilt library is not loaded. In your main Java class, try
static {
System.loadLibrary("prebuiltname");
System.loadLibrary("yourlib");
}
This is a static constructor, the safest place to load JNI library.
If you are on linux you will see exported symbols using nm -D. example nm -D libzip.so:
...
0000000000009dc0 T zip_unchange
0000000000009dd0 T zip_unchange_all
0000000000009e30 T zip_unchange_archive
0000000000009e60 T _zip_unchange_data
If you want to control visibility of your functions use __attribute__ ((visibility ("default"))) and command line -fvisibility=hidden. More information here.
Now I want to switch one of my static libraries to be .so. I was successfully able to build .so library by replacing BUILD_STATIC_LIBRARY with BUILD_SHARED_LIBRARY in its android.mk and adding required dependencies.
I don't think you can do it if its a C++ library. From <doc>/CPLUSPLUS-SUPPORT.html:
Please keep in mind that the static library variant of a given C++
runtime SHALL ONLY BE LINKED INTO A SINGLE BINARY for optimal
conditions.
What this means is that if your project consists of a
single shared library, you can link against, e.g., stlport_static, and
everything will work correctly.
On the other hand, if you have two
shared libraries in your project (e.g. libfoo.so and libbar.so) which
both link against the same static runtime, each one of them will
include a copy of the runtime's code in its final binary image. This
is problematic because certain global variables used/provided
internally by the runtime are duplicated.
This is likely to result in code that doesn't work correctly, for example:
* memory allocated in one library, and freed in the other would leak
or even corrupt the heap.
* exceptions raised in libfoo.so cannot be caught in libbar.so (and may
simply crash the program).
* the buffering of cout not working properly
This problem also happens if you want to link an executable and a shared
library to the same static library.
In other words, if your project requires several shared library modules,
then use the shared library variant of your C++ runtime.
From above, it means everything needs to link against the same C++ standard runtime shared object.
I'd like to compile a library (static but also could be shared) written in C++ so that is statically contains all the C++ runtime functionality that I use in the library, i.e. another project can simply include my .a or .so file without having to worry about further dependencies. How can I do this using the NDK/Android build system?
So far, I have
APP_STL := c++_static
in my Application.mk and build my library using
include $(BUILD_STATIC_LIBRARY)
in my Android.mk after setting up the project. However, if I then pull in the resulting library into a second project (say, a dynamic library) that does not specify an STL (e.g. pure C) I get a load of symbol not found linker errors. Only when I specify to use the c++_static STL in the second project do these errors go away indicating that c++_static was not pulled into the first library. (I know this isn't surprising but it's not the behaviour I want).
Using good ol' make files and specifying build flags myself allows me to combine static libraries. My problem is coaxing the NDK build system to do it :)
if you compile a C++ runtime statically to your lib, unfortunately another project have to worry about this dependency:
If several .so files are using static C++ runtimes, this can lead to corruption and crashes because of the sharing of global variables used by the runtimes.
If you compile everything (your lib and the final project) into a single .so file, you can use one static C++ runtime, and it's better then if it's declared inside the end project. Because forcing the static inclusion of the C++ runtime in your lib would prevent the end project of using a C++ runtime.