Develop Reference

Dealing with android native crashes

Lately, we're experiencing native crashes spike in production. All those crashes are related to rendering UI and happened mostly on Android 12 and vendors like Samsung. Some of them from Play Console are:
[libhwui.so] SkCanvas::restore()
SIGSEGV
backtrace:
#00 pc 0x000000000026c9d8 /system/lib64/libhwui.so (SkCanvas::restore()+344)
#01 pc 0x000000000026c73c /system/lib64/libhwui.so
(android::SkiaCanvas::restore()+52)
#02 pc 0x00000000002008e0 /system/lib64/libhwui.so
(android::CanvasJNI::restore(long)+56)
#03 pc 0x0000000000bef14c /data/misc/apexdata/com.android.art/dalvik-
cache/arm64/boot.oat (android.graphics.Canvas.restore+60)
or
[[vdso]] __kernel_rt_sigreturn
SIGSEGV
#00 pc 0x00000000001ac258 /apex/com.android.art/lib64/libart.so
(art::ArtMethod::GetOatQuickMethodHeader(unsigned long)+40)
#01 pc 0x0000000000597864 /apex/com.android.art/lib64/libart.so (void
art::StackVisitor::WalkStack<(art::StackVisitor::CountTransitions)0>(bool)+516)
#02 pc 0x000000000059a514 /apex/com.android.art/lib64/libart.so
(art::StackVisitor::GetVRegFromDebuggerShadowFrame(unsigned short, art::VRegKind,
unsigned int*) const+284)
#03 pc 0x000000000059a314 /apex/com.android.art/lib64/libart.so
(art::StackVisitor::GetVReg(art::ArtMethod*, unsigned short, art::VRegKind, unsigned
int*, std::__1::optional<art::DexRegisterLocation>) const+76)
But we cannot understand their cause because neither Firebase Crashlytics nor other third parties can catch native crashes. We've googled everything and found nothing about how to get more data for the native crashes to find the cause. Maybe somebody has an experience with it and can share.

How to symbolicate libart.so or libc.so stacktraces in Crashlytics Android NDK?

Note: Symbols are showing up in crashlytics for our c++ library, the problem is that they aren't showing for system libraries like libc, libart, libbase, and libandroid_runtime.
We have some tricky crashes that happen entirely in the Android runtime and these are hard to debug without symbols. In firebase crashlytics we see the following stack trace:
Crashed: Thread : SIGABRT 0x0000000000000000
#00 pc 0x4e574 libc.so
#01 pc 0x4e540 libc.so
#02 pc 0x5677d8 libart.so
#03 pc 0x13ab0 libbase.so
#04 pc 0x13090 libbase.so
#05 pc 0x38cb6c libart.so
#06 pc 0x39f7d8 libart.so
#07 pc 0x1260e0 libandroid_runtime.so
#08 pc 0x124ef4 libandroid_runtime.so
#09 pc 0x124dc4 libandroid_runtime.so
#10 pc 0x115468 libandroid_runtime.so
When I force a test crash in our C++ library by dereferencing a null pointer, I see the following backtrace in my local Android Studio console:
...snip...
#06 pc 00000000002d7644 /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+148) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#07 pc 00000000002cdd64 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+548) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#08 pc 00000000002f23d0 /apex/com.android.art/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+312) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#09 pc 00000000003839f4 /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+800) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#10 pc 00000000003813f4 /apex/com.android.art/lib64/libart.so (MterpInvokeVirtualRange+1368) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
#11 pc 00000000002c8714 /apex/com.android.art/lib64/libart.so (mterp_op_invoke_virtual_range+20) (BuildId: adb75d6f792faa24b1bc8cf512fb112c)
...snip...
However, the same crash in crashlytics looks like this:
...snip...
#07 pc 0x222244 libart.so
#08 pc 0x218964 libart.so
#09 pc 0x284208 libart.so
#10 pc 0x3e34ac libart.so
#11 pc 0x800ba4 libart.so
...snip...
How can we get crashlytics to include the information that is clearly in the crashdump?
Some notes on our build setup:
We have already followed https://firebase.google.com/docs/crashlytics/ndk-reports
Gradle Project 1 builds the native library, ensuring that debug is on and symbols are not stripped.
Gradle Project 2 builds the app and links in the library and tells crashlytics to upload native symbols
firebaseCrashlytics {
nativeSymbolUploadEnabled true
unstrippedNativeLibsDir file("PATH/TO/UNSTRIPPED/DIRECTORY")
}

This behavior is expected. When Crashlytics gets NDK crashes, these need to be symbolicated. In order to do this, the respective symbol files should be uploaded to Crashlytics.
With the configuration you mentioned, the symbols available in your app will be uploaded to Crashlytics.
nativeSymbolUploadEnabled true
But in the case of system libraries, the respective symbol files are not available (they are not public as far as I know). So Crashlytics won't have access to the required symbol files for symbolicating frames from system libraries.

What is the Android 'abort' error and why Firebase Crashlytics does not catch it?

My Android app is suffering from an'abort' error after recent update.
But I don't know why this problem happens and how can I debug it.
This 'abort' error occurred especially on Android 10 and 11.
It is not recorded on Firebase Crashlytics, I can see it only on Google play console.
The log is as below.
backtrace:
#00 pc 000000000004ef24 /apex/com.android.runtime/lib64/bionic/libc.so (abort+164)
#00 pc 000000000053b078 /apex/com.android.art/lib64/libart.so (art::Runtime::Abort(char const*)+2340)
#00 pc 000000000001394c /system/lib64/libbase.so (android::base::SetAborter(std::__1::function<void (char const*)>&&)::$_3::__invoke(char const*)+76)
#00 pc 00000000000130cc /system/lib64/libbase.so (android::base::LogMessage::~LogMessage()+312)
#00 pc 000000000058e1d4 /apex/com.android.art/lib64/libart.so (art::Thread::AssertNoPendingException() const+1836)
#00 pc 00000000001c15cc /apex/com.android.art/lib64/libart.so (art::ClassLinker::FindClass(art::Thread*, char const*, art::Handle<art::mirror::ClassLoader>)+64)
#00 pc 0000000000380c34 /apex/com.android.art/lib64/libart.so (art::JNI<false>::FindClass(_JNIEnv*, char const*)+1144)
#00 pc 00000000000200d4 /data/app/~~0HXm4UzxwERXczUAACEC9g==/com.****.****-L0QwKgab-Tg3REmBFsu7iA==/lib/arm64/libmm.so
#00 pc 0000000000020b90 /data/app/~~0HXm4UzxwERXczUAACEC9g==/com.****.****-L0QwKgab-Tg3REmBFsu7iA==/lib/arm64/libmm.so
#00 pc 00000000000b6234 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+64)
#00 pc 0000000000050e64 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64)
I applied 'View Binding' to this update.
Could it be an cause?
Also I wonder if an Android app can be alive when 'abort' error occurred.
Please help me...
libmm is a file of an obfuscation solution which we applied to the app. It is a binary obfuscation solution and it is run before application runs on memory as far as I know.

Android mystery crash in libicuuc.so (utext_close)

Our Android game started to log weird crashes in Google Play ANRs & crashes section after the latest update.
backtrace:
#00 pc 000000000011be50 /system/lib64/libicuuc.so (utext_close_60+52)
#01 pc 00000000001aa4c4 /system/lib64/libicui18n.so (icu_60::RegexPattern::zap()+212)
#02 pc 00000000001aa560 /system/lib64/libicui18n.so (icu_60::RegexPattern::~RegexPattern()+36)
#03 pc 0000000000094120 /system/framework/arm64/boot-core-libart.oat (offset 0x90000) (java.math.NativeBN.BN_copy [DEDUPED]+160)
#04 pc 000000000018340c /system/framework/arm64/boot-core-libart.oat (offset 0x90000) (libcore.util.NativeAllocationRegistry$CleanerThunk.run+76)
#05 pc 000000000040c754 /system/framework/arm64/boot.oat (offset 0x13b000) (sun.misc.Cleaner.clean+164)
#06 pc 00000000001daa0c /system/framework/arm64/boot.oat (offset 0x13b000) (java.lang.ref.ReferenceQueue.enqueueLocked+236)
#07 pc 00000000001dab2c /system/framework/arm64/boot.oat (offset 0x13b000) (java.lang.ref.ReferenceQueue.enqueuePending+172)
#08 pc 00000000001d7b04 /system/framework/arm64/boot-core-libart.oat (offset 0x90000) (java.lang.Daemons$ReferenceQueueDaemon.runInternal+244)
#09 pc 000000000015978c /system/framework/arm64/boot-core-libart.oat (offset 0x90000) (java.lang.Daemons$Daemon.run+76)
#10 pc 00000000002c1038 /system/framework/arm64/boot.oat (offset 0x13b000) (java.lang.Thread.run+72)
#11 pc 000000000056ef88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#12 pc 00000000000d4204 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#13 pc 0000000000472fd4 /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
#14 pc 0000000000474090 /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*)+424)
#15 pc 000000000049f684 /system/lib64/libart.so (art::Thread::CreateCallback(void*)+1120)
#16 pc 0000000000083588 /system/lib64/libc.so (__pthread_start(void*)+36)
#17 pc 00000000000241dc /system/lib64/libc.so (__start_thread+68)
The only changed code in the update that is somewhat relevant to this was in our analytics module where we are doing some string replacements:
private static String formatStringForAnalytics(String str) {
if(str == null) {
return "";
}
return str.replaceAll(" ", "_").replaceAll(":", "");
}
We haven't been able to reproduce the crash ourselves. So we have to wait a bit to make and release a new update where we disable this piece of code and see if the crash seems to disappear.
I just find weird that standard string replacements would cause something like this. Anyone else encountered similar crashes?

The problem was actually wrong string replacement function.
The API documentation says that replaceAll() method
Replaces each substring of this string that matches the given regular
expression with the given replacement.
I changed the code to use replace() method
Replaces each substring of this string that matches the literal target
sequence with the specified literal replacement sequence.
This is actually what the code intended to do in first place. With this change the crashes disappeared.
It seems that when certain strings where logged into analytics the replaceAll() managed to somehow break the string and caused it to crash. This answer for question about differences between replace() and replaceAll() seems indeed to be very true in this case: Using the wrong function can lead to subtle bugs.

How to check crash log using android ndk in cocos2d-x

I use cocos2d-x to develop a game, after it runs perfectly in iOs, I turn around to android platform.
But it run somewhere in android, and I only receive some error log like this:
08-26 10:49:23.823: A/libc(2884): Fatal signal 11 (SIGSEGV) at 0x0000000c (code=1), thread 2917 (Thread-285)
With this, I can't fix the crash log.
So my question is how can I check the stack of crash log?
And in SO, there are some similar questions, but with none useful answer to me.
Using ndk-stack to read crash logs
Unable to get line no from stack trace in android ndk
How to get Useful crashlog information Android Cocos2dx C++

I find the perfect answer in http://www.cocos2d-x.org/boards/6/topics/20437.
The solution is:
When you get errors like this on the LogCat, they are saved on $PROJECT_PATH/obj/local/armeabi where $PROJECT_PATH is the path to your cocos2d-x android project. To symbolicate the messages to something understandable, you can use the ndk-stack tool.
Open up the Terminal (or Cygwin, not sure though) and type in
cd $ANDROID_NDK
adb logcat | ./ndk-stack -sym $PROJECT_PATH/obj/local/armeabi
where:
$ANDROID_NDK is the path to your android NDK
$PROJECT_PATH is the path to your cocos2d-x android project
And I try it, and get the following stack-trace:
********** Crash dump: **********
Build fingerprint: 'google/soju/crespo:4.1.2/JZO54K/485486:user/release-keys'
pid: 2884, tid: 2917, name: Thread-285 >>> com.gumichina.allen <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0000000c
Stack frame #00 pc 0062bfb4 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCObject::release()+20): Routine release in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/cocoa/CCObject.cpp:82
Stack frame #01 pc 006d8094 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCTouchHandler::~CCTouchHandler()+128): Routine ~CCTouchHandler in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/touch_dispatcher/CCTouchHandler.cpp:108
Stack frame #02 pc 006d8524 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCTargetedTouchHandler::~CCTargetedTouchHandler()+84): Routine ~CCTargetedTouchHandler in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/touch_dispatcher/CCTouchHandler.cpp:193
Stack frame #03 pc 006d8570 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCTargetedTouchHandler::~CCTargetedTouchHandler()+20): Routine ~CCTargetedTouchHandler in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/touch_dispatcher/CCTouchHandler.cpp:193
Stack frame #04 pc 0062bff4 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCObject::release()+84): Routine release in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/cocoa/CCObject.cpp:86
Stack frame #05 pc 006aade4 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::ccArrayRemoveObjectAtIndex(cocos2d::_ccArray*, unsigned int, bool)+100): Routine ccArrayRemoveObjectAtIndex in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/support/data_support/ccCArray.cpp:197
Stack frame #06 pc 006aafa0 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::ccArrayRemoveObject(cocos2d::_ccArray*, cocos2d::CCObject*, bool)+72): Routine ccArrayRemoveObject in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/support/data_support/ccCArray.cpp:235
Stack frame #07 pc 0062fed4 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCArray::removeObject(cocos2d::CCObject*, bool)+52): Routine removeObject in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/cocoa/CCArray.cpp:295
Stack frame #08 pc 006d6c90 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCTouchDispatcher::forceRemoveDelegate(cocos2d::CCTouchDelegate*)+500): Routine forceRemoveDelegate in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/touch_dispatcher/CCTouchDispatcher.cpp:200
Stack frame #09 pc 006d7888 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCTouchDispatcher::touches(cocos2d::CCSet*, cocos2d::CCEvent*, unsigned int)+1620): Routine touches in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/touch_dispatcher/CCTouchDispatcher.cpp:449
Stack frame #10 pc 006d7b74 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCTouchDispatcher::touchesEnded(cocos2d::CCSet*, cocos2d::CCEvent*)+56): Routine touchesEnded in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/touch_dispatcher/CCTouchDispatcher.cpp:507
Stack frame #11 pc 0068acb8 /mnt/asec/com.gumichina.allen-2/lib/liballen.so (cocos2d::CCEGLViewProtocol::handleTouchesEnd(int, int*, float*, float*)+124): Routine handleTouchesEnd in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/platform/CCEGLViewProtocol.cpp:333
Stack frame #12 pc 0069340c /mnt/asec/com.gumichina.allen-2/lib/liballen.so (Java_org_cocos2dx_lib_Cocos2dxRenderer_nativeTouchesEnd+88): Routine Java_org_cocos2dx_lib_Cocos2dxRenderer_nativeTouchesEnd in /Users/tangyue/project/crosskaiser-native/application/proj.android/../../cocos2dx/platform/android/jni/TouchesJni.cpp:42
Stack frame #13 pc 0001de30 /system/lib/libdvm.so (dvmPlatformInvoke+112)
Stack frame #14 pc 0004ce73 /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+390)
Stack frame #15 pc 0004ef8f /system/lib/libdvm.so (dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*)+174)
Stack frame #16 pc 00027260 /system/lib/libdvm.so
Stack frame #17 pc 0002bb38 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+180)
Stack frame #18 pc 0005f5b1 /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, std::__va_list)+272)
Stack frame #19 pc 0005f5db /system/lib/libdvm.so (dvmCallMethod(Thread*, Method const*, Object*, JValue*, ...)+20)
Stack frame #20 pc 00054193 /system/lib/libdvm.so
Stack frame #21 pc 00012b90 /system/lib/libc.so (__thread_entry+48): Unable to open symbol file obj/local/armeabi//libc.so. Error (9): Bad file descriptor
Stack frame #22 pc 000122f4 /system/lib/libc.so (pthread_create+172): Unable to open symbol file obj/local/armeabi//libc.so. Error (9): Bad file descriptor