Lately, we're experiencing native crashes spike in production. All those crashes are related to rendering UI and happened mostly on Android 12 and vendors like Samsung. Some of them from Play Console are:
[libhwui.so] SkCanvas::restore()
SIGSEGV
backtrace:
#00 pc 0x000000000026c9d8 /system/lib64/libhwui.so (SkCanvas::restore()+344)
#01 pc 0x000000000026c73c /system/lib64/libhwui.so
(android::SkiaCanvas::restore()+52)
#02 pc 0x00000000002008e0 /system/lib64/libhwui.so
(android::CanvasJNI::restore(long)+56)
#03 pc 0x0000000000bef14c /data/misc/apexdata/com.android.art/dalvik-
cache/arm64/boot.oat (android.graphics.Canvas.restore+60)
or
[[vdso]] __kernel_rt_sigreturn
SIGSEGV
#00 pc 0x00000000001ac258 /apex/com.android.art/lib64/libart.so
(art::ArtMethod::GetOatQuickMethodHeader(unsigned long)+40)
#01 pc 0x0000000000597864 /apex/com.android.art/lib64/libart.so (void
art::StackVisitor::WalkStack<(art::StackVisitor::CountTransitions)0>(bool)+516)
#02 pc 0x000000000059a514 /apex/com.android.art/lib64/libart.so
(art::StackVisitor::GetVRegFromDebuggerShadowFrame(unsigned short, art::VRegKind,
unsigned int*) const+284)
#03 pc 0x000000000059a314 /apex/com.android.art/lib64/libart.so
(art::StackVisitor::GetVReg(art::ArtMethod*, unsigned short, art::VRegKind, unsigned
int*, std::__1::optional<art::DexRegisterLocation>) const+76)
But we cannot understand their cause because neither Firebase Crashlytics nor other third parties can catch native crashes. We've googled everything and found nothing about how to get more data for the native crashes to find the cause. Maybe somebody has an experience with it and can share.
Related
Note: Symbols are showing up in crashlytics for our c++ library, the problem is that they aren't showing for system libraries like libc, libart, libbase, and libandroid_runtime.
We have some tricky crashes that happen entirely in the Android runtime and these are hard to debug without symbols. In firebase crashlytics we see the following stack trace:
Crashed: Thread : SIGABRT 0x0000000000000000
#00 pc 0x4e574 libc.so
#01 pc 0x4e540 libc.so
#02 pc 0x5677d8 libart.so
#03 pc 0x13ab0 libbase.so
#04 pc 0x13090 libbase.so
#05 pc 0x38cb6c libart.so
#06 pc 0x39f7d8 libart.so
#07 pc 0x1260e0 libandroid_runtime.so
#08 pc 0x124ef4 libandroid_runtime.so
#09 pc 0x124dc4 libandroid_runtime.so
#10 pc 0x115468 libandroid_runtime.so
When I force a test crash in our C++ library by dereferencing a null pointer, I see the following backtrace in my local Android Studio console:
...snip...
#06 pc 00000000002d7644 /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+148) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#07 pc 00000000002cdd64 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#08 pc 00000000002f23d0 /apex/com.android.art/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+312) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#09 pc 00000000003839f4 /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+800) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#10 pc 00000000003813f4 /apex/com.android.art/lib64/libart.so (MterpInvokeVirtualRange+1368) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#11 pc 00000000002c8714 /apex/com.android.art/lib64/libart.so (mterp_op_invoke_virtual_range+20) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
...snip...
However, the same crash in crashlytics looks like this:
...snip...
#07 pc 0x222244 libart.so
#08 pc 0x218964 libart.so
#09 pc 0x284208 libart.so
#10 pc 0x3e34ac libart.so
#11 pc 0x800ba4 libart.so
...snip...
How can we get crashlytics to include the information that is clearly in the crashdump?
Some notes on our build setup:
We have already followed https://firebase.google.com/docs/crashlytics/ndk-reports
Gradle Project 1 builds the native library, ensuring that debug is on and symbols are not stripped.
Gradle Project 2 builds the app and links in the library and tells crashlytics to upload native symbols
firebaseCrashlytics {
nativeSymbolUploadEnabled true
unstrippedNativeLibsDir file("PATH/TO/UNSTRIPPED/DIRECTORY")
}
This behavior is expected. When Crashlytics gets NDK crashes, these need to be symbolicated. In order to do this, the respective symbol files should be uploaded to Crashlytics.
With the configuration you mentioned, the symbols available in your app will be uploaded to Crashlytics.
nativeSymbolUploadEnabled true
But in the case of system libraries, the respective symbol files are not available (they are not public as far as I know). So Crashlytics won't have access to the required symbol files for symbolicating frames from system libraries.
We're investigating on an app developed by another team a native Crash on Android related to HereMaps (HERE SDK Navigation edition, navigate-4.10.2.0.7878) with this stacktrace:
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> com.mydomain.myapp <<<
backtrace:
#00 pc 0000000000051010 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164)
#00 pc 0000000000e0143c /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 0000000000e0158c /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 0000000000e014f4 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 0000000000e01478 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 0000000000debfe4 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 000000000144cc84 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 00000000014035bc /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 00000000013fa668 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 000000000155cff0 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 00000000011273f8 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so (Java_com_here_sdk_navigation_VisualNavigator_disposeNativeHandle+480)
#00 pc 0000000000042020 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.odex (art_jni_trampoline+96)
#00 pc 000000000020988c /apex/com.android.art/lib64/libart.so (nterp_helper+1948)
#00 pc 0000000000b6999c /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.vdex (com.here.sdk.navigation.VisualNavigator.access$000)
#00 pc 0000000000209124 /apex/com.android.art/lib64/libart.so (nterp_helper+52)
#00 pc 0000000000b69974 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.vdex (com.here.sdk.navigation.VisualNavigator$1.disposeNative)
#00 pc 0000000000073c6c /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.odex (com.here.NativeBase$DisposableReference.dispose+156)
#00 pc 000000000020a0a0 /apex/com.android.art/lib64/libart.so (nterp_helper+4016)
#00 pc 00000000009a3834 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.vdex (com.here.NativeBase.cleanUpQueue+26)
#00 pc 0000000000209124 /apex/com.android.art/lib64/libart.so (nterp_helper+52)
#00 pc 00000000009a380e /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.vdex (com.here.NativeBase.access$100)
#00 pc 0000000000209124 /apex/com.android.art/lib64/libart.so (nterp_helper+52)
#00 pc 00000000009a37be /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.vdex (com.here.NativeBase$DisposableReference.<init>+22)
#00 pc 000000000020a044 /apex/com.android.art/lib64/libart.so (nterp_helper+3924)
#00 pc 00000000009a37ca /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.vdex (com.here.NativeBase$DisposableReference.<init>)
#00 pc 000000000020a748 /apex/com.android.art/lib64/libart.so (nterp_helper+5720)
#00 pc 00000000009a37fc /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.vdex (com.here.NativeBase.<init>+28)
#00 pc 000000000020a044 /apex/com.android.art/lib64/libart.so (nterp_helper+3924)
#00 pc 00000000009ac7f8 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/oat/arm64/base.vdex (com.here.sdk.core.threading.RunnableImpl.<init>+10)
#00 pc 00000000002cdd64 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548)
#00 pc 00000000003d5660 /apex/com.android.art/lib64/libart.so (art::JNI<false>::CallNonvirtualVoidMethodV(_JNIEnv*, _jobject*, _jclass*, _jmethodID*, std::__va_list)+492)
#00 pc 00000000003d4ab4 /apex/com.android.art/lib64/libart.so (art::JNI<false>::NewObjectV(_JNIEnv*, _jclass*, _jmethodID*, std::__va_list)+736)
#00 pc 0000000000e0e858 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 0000000000e37fac /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 0000000000e37a5c /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 000000000140dad0 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 000000000144c5e8 /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 000000000144d2bc /data/app/~~yFKezl3wu05hiNZ1aIFJIQ==/com.mydomain.myapp-sfkAGrmiHUTbEivXLD8EKQ==/base.apk!libheresdk.so
#00 pc 00000000000b2fd0 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+264)
#00 pc 0000000000052834 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)
EDIT:
After some refactoring and cleaning operations in our code we've reached a clean condition in which we are confident that there aren't any leak; we've used LeakCanary to investigate and remove all these ones, but the native crash it's still here.
So we've tried to return back to basis and we've cloned HEREMaps Navigate Samples from github and we've found that in Navigation Sample there isn't any native crash, but there is also an only activity with heremaps instance references which die within the entire application.
To replicate a similar use case then our we've added an activity before the MainActivity of the Sample and we've tried to launch the activity and return back to the first one to show what is the behaviour of the library in terms of resources releasing.
Only opening and closing MainActivity from a starting one doesn't have any native crash also because VisualNavigator (which seems to be the class with native crash in the backtrace) is retained by it's delegates (aka Listeners) => setupListeners() in NavigationExample. So we've also removed all listeners when MainActivity calls onDestroy and in this way we see always the same native crash as in our app, going back from MainActivity to starting one.
Note: to be able to always reproduce the native crash we use LeakCanary because the library force GC when MainActivity has been destroyed; if we remove it the GC operation is done randomly by the system.
Links:
our version of HEREMaps Navigation SDK sample modified to reproduce native crash on github
video of crash of the sample: https://youtube.com/shorts/edY-TRvWh3I
So the big question here is: have we implemented wrong HEREMaps SDK integration (and managing instance releasing) or this native crash is inside the library and has to be fixed from the library owner?
Try find a way to make the VisualNavigator persistent:
Using StopRendering() is fine but do not null the VisualNavigator.
I had to go through the same issue when cancelling the route too quickly, although it got way worse over the recent updates and just crashes when just stopping navigation and this fixes it.
In the HERE SDK developer's guide there is a chapter that shows how to stop navigation. It is recommended to call stopRendering(). In your example app, it seems you do not call visualNavigator.stopRendering() when the intermediate activity gets destroyed. Try to call it.
In the developer options menu of the device you can turn on "Don't keep activities" to simulate such scenarios.
My Android app is suffering from an'abort' error after recent update.
But I don't know why this problem happens and how can I debug it.
This 'abort' error occurred especially on Android 10 and 11.
It is not recorded on Firebase Crashlytics, I can see it only on Google play console.
The log is as below.
backtrace:
#00 pc 000000000004ef24 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164)
#00 pc 000000000053b078 /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+2340)
#00 pc 000000000001394c /system/lib64/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_3::__invoke(char const*)+76)
#00 pc 00000000000130cc /system/lib64/libbase.so (android::base::LogMessage::~LogMessage()+312)
#00 pc 000000000058e1d4 /apex/com.android.art/lib64/libart.so (art::Thread::AssertNoPendingException() const+1836)
#00 pc 00000000001c15cc /apex/com.android.art/lib64/libart.so (art::ClassLinker::FindClass(art::Thread*, char const*, art::Handle<art::mirror::ClassLoader>)+64)
#00 pc 0000000000380c34 /apex/com.android.art/lib64/libart.so (art::JNI<false>::FindClass(_JNIEnv*, char const*)+1144)
#00 pc 00000000000200d4 /data/app/~~0HXm4UzxwERXczUAACEC9g==/com.****.****-L0QwKgab-Tg3REmBFsu7iA==/lib/arm64/libmm.so
#00 pc 0000000000020b90 /data/app/~~0HXm4UzxwERXczUAACEC9g==/com.****.****-L0QwKgab-Tg3REmBFsu7iA==/lib/arm64/libmm.so
#00 pc 00000000000b6234 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+64)
#00 pc 0000000000050e64 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)
I applied 'View Binding' to this update.
Could it be an cause?
Also I wonder if an Android app can be alive when 'abort' error occurred.
Please help me...
libmm is a file of an obfuscation solution which we applied to the app. It is a binary obfuscation solution and it is run before application runs on memory as far as I know.
I get reports of these crashes from users of my app on Android 6.0 devices. The crash stack doesn't contain any keywords of my source code. I can't determined what caused the problem and don't know what to do. What's wrong with it? :(
SIGSEGV(SEGV_MAPERR):
#00 pc 0018b7fc /system/lib/libart.so (art::gc::collector::ConcurrentCopying::FillWithDummyObject(art::mirror::Object*, unsigned int)+1195) [armeabi-v7a::bd556bcd01d8c215f5cd133343e0a883]
#01 pc 0018ba89 /system/lib/libart.so (art::gc::collector::MarkSweep::ProcessMarkStack(bool)+100) [armeabi-v7a::bd556bcd01d8c215f5cd133343e0a883]
java:
java.lang.Daemons$HeapTaskDaemon.run(Daemons.java:386)
java.lang.Thread.run(Thread.java:818)
SIGSEGV(SEGV_MAPERR):
#00 pc 001976fc /system/lib/libart.so (art::gc::collector::ConcurrentCopying::Copy(art::mirror::Object*)+923) [armeabi-v7a::44966e008bb1c7f64243a7db7f321398]
#01 pc 001979a1 /system/lib/libart.so (art::gc::collector::MarkSweep::ProcessMarkStack(bool)+112) [armeabi-v7a::44966e008bb1c7f64243a7db7f321398]
……
#55 pc 001979a1 /system/lib/libart.so (art::gc::collector::MarkSweep::ProcessMarkStack(bool)+112) [armeabi-v7a::44966e008bb1c7f64243a7db7f321398]
#56 pc 001979a1 /system/lib/libart.so (_ZN3art2gc9collec
There are lots of native crashes in my app, only in 5.0 or 5.1 android system, the crash trace are bellow:
SIGBUS(BUS_ADRERR):
#00 pc 0022dece /system/lib/libart.so (art::StackVisitor::WalkStack(bool) +209)
#01 pc 00237915 /system/lib/libart.so (_jobject* art::Thread::CreateInternalStackTrace<false>(art::ScopedObjectAccessAlreadyRunnable const&) const +56)
#02 pc 002081b3 /system/lib/libart.so (art::Throwable_nativeFillInStackTrace(_JNIEnv*, _jclass*) +18)
#03 pc 00000c1d /data/dalvik-cache/arm/system#framework#boot.oat
Does anyone else have met this problem before ? what is /data/dalvik-cache/arm/system#framework#boot.oat ? Any help on how to proceed would be appreciated ! Thanks