I am looking for a native crash in my application but i cant find a tool that help me to translate the address to the function names in my library i saw this post : how to use addr2line but it seems that only works when you have a trace like this :
I/DEBUG ( 31): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG ( 31): Build fingerprint: 'generic/sdk/generic:2.3/GRH55/79397:eng/test-keys'
I/DEBUG ( 31): pid: 378, tid: 386 >>> com.example.gltest <<<
I/DEBUG ( 31): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000000
I/DEBUG ( 31): r0 001dbdc0 r1 00000001 r2 00000000 r3 00000000
I/DEBUG ( 31): r4 00000000 r5 40a40000 r6 4051a480 r7 42ddbee8
I/DEBUG ( 31): r8 43661b24 r9 42ddbed0 10 42ddbebc fp 41e462d8
I/DEBUG ( 31): ip 00000001 sp 436619d0 lr 83a12f5d pc 8383deb4 cpsr 20000010
I/DEBUG ( 31): #00 pc 0003deb4 /data/data/com.example.gltest/lib/libnativemaprender.so
I/DEBUG ( 31): #01 pc 00039b76 /data/data/com.example.gltest/lib/libnativemaprender.so
I/DEBUG ( 31): #02 pc 00017d34 /system/lib/libdvm.so
as you can see in the last 3 lines the library of your application is there : ibnativemaprender.so
and only if your library appear there the addr2line app will work
but in my trace i have this :
I/DEBUG (16251): #00 pc 000161c0 /system/lib/libc.so
I/DEBUG (16251): #01 lr afd196f1 /system/lib/libc.so
and my library is not show there until later :
/DEBUG (16251): stack:
I/DEBUG (16251): 46f0c818 afd426a4
I/DEBUG (16251): 46f0c81c 000b7290
I/DEBUG (16251): 46f0c820 00000afa
I/DEBUG (16251): 46f0c824 afd187b9 /system/lib/libc.so
I/DEBUG (16251): 46f0c828 afd42644
I/DEBUG (16251): 46f0c82c afd467c4
I/DEBUG (16251): 46f0c830 00000000
I/DEBUG (16251): 46f0c834 afd196f1 /system/lib/libc.so
I/DEBUG (16251): 46f0c838 00000001
I/DEBUG (16251): 46f0c83c 46f0c86c
I/DEBUG (16251): 46f0c840 81b6a4a0 /data/data/com.android.ANMP/lib/mylib.so
so is there a tool that can help me? or a different way to use addr2line?
No, not really. To do that you need to perform the search manually.
Manually, there are a three options:
Add debug prints to find out where and when it happens
Use IDA pro, (which is very expensive tool), load your library into it and look for the address 81b6a4a0, try to figure out which function it is. If you compile with debug symbols it should be easy enough.
Find the base address of your library, use objdump (from the NDK toolchain) to find what's in the address 81b6a4a0 (after removing base address from it).
In both last ways, if you don't find anything helpful, pull libc.so and libdvm.so from the device using adb and try to find what are the functions that appears in the stack trace, then check where do you call those functions in your code.
take a look at this. I have not yet found the time to try, so tell if if worked for you
You can use GDB to get stack traces. If you have a rooted phone or use the emulator you can enable core dumps. Then you can use GDB to examine the core dumps.
See this question for setting up GDB in Eclipse:
android how to use ndk gdb with a pure native executable
Related
I am debugging a huge NDK project in Android Studio, the app crashes at semingly random times with message similar to
A/libc: Fatal Signal 11 (SIGSEGV), code 1, fault addr 0x30 in fid 22993 (Thread-99505)
The numbers are different at times but the error is similar, and sometimes the Run tab in Android studio doesn't even show any error at all and the program just crashes with the message Application terminated
By looking around I have found that android should generate some kind of crash log similar to ones shown here
https://source.android.com/devices/tech/debug/native-crash
But I do not understand how to get them, or whether there is some setting in Android Studio that allows me to view them
How do I get crash reports like these from Android Studio?
Android Studio version 3.0.1 in Apple macbook
Instead of the Run tab, look at the Logcat. Don't set the filter to your app only. There is a good chance that you will see a stack trace, like
I/DEBUG ( 31): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG ( 31): Build fingerprint: 'generic/google_sdk/generic/:2.2/FRF91/43546:eng/test-keys'
I/DEBUG ( 31): pid: 351, tid: 351 >>> /data/local/ndk-tests/crasher <<<
I/DEBUG ( 31): signal 11 (SIGSEGV), fault addr 0d9f00d8
I/DEBUG ( 31): r0 0000af88 r1 0000a008 r2 baadf00d r3 0d9f00d8
I/DEBUG ( 31): r4 00000004 r5 0000a008 r6 0000af88 r7 00013c44
I/DEBUG ( 31): r8 00000000 r9 00000000 10 00000000 fp 00000000
I/DEBUG ( 31): ip 0000959c sp be956cc8 lr 00008403 pc 0000841e cpsr 60000030
I/DEBUG ( 31): #00 pc 0000841e /data/local/ndk-tests/crasher
I/DEBUG ( 31): #01 pc 000083fe /data/local/ndk-tests/crasher
I/DEBUG ( 31): #02 pc 000083f6 /data/local/ndk-tests/crasher
I/DEBUG ( 31): #03 pc 000191ac /system/lib/libc.so
I/DEBUG ( 31): #04 pc 000083ea /data/local/ndk-tests/crasher
I/DEBUG ( 31): #05 pc 00008458 /data/local/ndk-tests/crasher
I/DEBUG ( 31): #06 pc 0000d362 /system/lib/libc.so
I/DEBUG ( 31):
Android NDK includes a command ndk-stack that can help match the addresses with actual source files of your code, which is based on the addr2line utility in NDK toolchain.
Further instructions for understanding the stack trace: https://developer.android.com/studio/debug/stacktraces
From comments:
You may find the 'unstripped' libraries under app/build/intermediates/transforms/mergeJniLibs
I have a hello world C++ application that links with many 3rdpart and home-made libraries. It crashes immediately when trying to load it, with Segmentation Fault.
The same code, with the same dependencies, all compiled for Linux - runs successfully.
How can I start to debug it?
What could be the reason?
Edit: This is what logcat prints:
F/libc ( 8129): Fatal signal 11 (SIGSEGV), code 128, fault addr 0x0 in tid 8129 (HelloWorldApp) I/DEBUG ( 2876): *** *** *** *** *** ***
*** *** *** *** *** *** *** *** *** *** I/DEBUG ( 2876): Build fingerprint: 'Intel/cht_hr/cht_hr:5.1.1/LMY47Z/LS0000037:userdebug/test-keys' I/DEBUG ( 2876): Revision: '0' W/NativeCrashListener( 3194): Couldn't find ProcessRecord for pid 8129 I/DEBUG ( 2876): ABI: 'x86' I/DEBUG ( 2876): pid: 8129, tid: 8129, name: HelloWorldApp >>> ./StaticImageOR <<< E/DEBUG ( 2876): AM write failure (32 / Broken pipe) I/DEBUG ( 2876): signal 11 (SIGSEGV), code 128 (SI_KERNEL), fault addr 0x0 I/DEBUG ( 2876): eax ff84cadc ebx f32b7c6c ecx 00000010 edx 00000000 I/DEBUG ( 2876): esi f32c6610 edi 00000000 I/DEBUG ( 2876): xcs 00000023 xds 0000002b xes 0000002b xfs 00000000 xss 0000002b I/DEBUG ( 2876): eip f2600cfb ebp ff84c6dc esp ff84c5f4 flags 00010246 I/DEBUG ( 2876): I/DEBUG ( 2876): backtrace: I/DEBUG ( 2876): #00 pc 0062fcfb /system/lib/libcommander.so I/DEBUG ( 2876): #01 pc 0017651f /system/lib/libcommander.so I/DEBUG ( 2876): #02 pc 00001fcb /system/bin/linker (__dl__ZN6soinfo16CallConstructorsEv.part.23+1275) I/DEBUG ( 2876):
#03 pc 00001c26 /system/bin/linker (__dl__ZN6soinfo16CallConstructorsEv.part.23+342) I/DEBUG ( 2876):
#04 pc 00001c26 /system/bin/linker (__dl__ZN6soinfo16CallConstructorsEv.part.23+342) I/DEBUG ( 2876):
#05 pc 00008706 /system/bin/linker (__dl___linker_init+4998) I/DEBUG ( 2876): #06 pc 00009e0e /system/bin/linker (__dl__start+30) I/DEBUG ( 2876): I/DEBUG ( 2876): Tombstone written to: /data/tombstones/tombstone_02
So I undersatnd the problem is in libcommander loading.
How to continue with this?
Thanks!
The easiest way is to run your application under debugger, then most recent android studio is really good at debugging c++ code. For instructions how to debug jni code, find Create Hello-JNI with Android Studio in google code labs.
Other solution is to look into logcat, you might find there reason for the crash. For example you can find lines like
I/DEBUG ( 8704): signal 11 (SIGSEGV), fault addr deadbaad
...
...
which will contain call stack for the location where crash happend. With that information you can use ndk-stack - tool (see here ndk-stack), to turn frame addresses into symbolic names.
Android will put into logcat other warnings/jni error that might help you find a location of the problem.
I have a strange problem that I don't know how to face.
My application uses a WebView to load a HTML app. It is a long and complex app, that also involves JavaScriptInterfaces.
Lately I am getting Core Dumps in the native webcore module that kill the whole activity.
It happens sometimes, but frequently, let's say 1 of every 5 times.
It always happens while WebView is loading the app, but on different places. It looks like a call to an exported Java function is involved, but not a specific one. I've refactored a zillion times everything and, while sometimes it seems resolved, it's just a matter of luck. Suddenly it fails again.
If WebView finishes loading and initing the app, it NEVER crashes. It only happens during load/init/start execution.
The only pattern that nearly always appears is, it looks like it always crashes in the same place:
I/DEBUG (30186): #00 pc 003a37a4 /system/lib/libwebcore.so (_ZNK5Layer8getChildEi)
I/DEBUG (30186): #01 pc 00285bae /system/lib/libwebcore.so
I/DEBUG (30186): #02 pc 0001edb0 /system/lib/libdvm.so (dvmPlatformInvoke)
I/DEBUG (30186): #03 pc 000594ba /system/lib/libdvm.so (_Z16dvmCallJNIMethodPKjP6JValuePK6MethodP6Thread)
Of course, I know this is an incomplete question, I'll post anything on request, but I'd like to answer if somebody has an idea on how can I guess what's more or less going on, I mean, can I look somewhere with this stack trace?? I am completely lost when it comes to the Android Native Layer.
If only I knew any relation to the crash I could refactor Java or JavaScript, but I'm completely blind, and as much as I have moved everything nothing seems to improve....
Thanks in advance !!!
PD- I am on ICS v.4.04 with a Samsung Galaxy Tab 10.1
/libc ( 575): Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1)
I/DEBUG (30186): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG (30186): Build fingerprint: 'samsung/GT-P7500/GT-P7500:4.0.4/IMM76D/XXLQ8:user/release-keys'
I/DEBUG (30186): pid: 575, tid: 591 >>> com.regaliz.libneo <<<
I/DEBUG (30186): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000000
I/DEBUG (30186): r0 0117f608 r1 00000000 r2 00feb880 r3 00000000
I/DEBUG (30186): r4 00feb880 r5 00ffb568 r6 00000007 r7 50fbfe0c
I/DEBUG (30186): r8 5bf04c10 r9 50fbfe04 10 00000000 fp 5bf04c24
I/DEBUG (30186): ip 5a229ba5 sp 5bf04c08 lr 5a229bb3 pc 5a3477a4 cpsr 800f3430
I/DEBUG (30186): d0 0000000042c80000 d1 0000000043ae0000
I/DEBUG (30186): d2 3f00000043ae0000 d3 43ae000041c80000
I/DEBUG (30186): d4 00000000432e0000 d5 411efaf800000000
I/DEBUG (30186): d6 3fe03e1c9b413986 d7 41d4634c06a07c39
I/DEBUG (30186): d8 0000000000000000 d9 0000000000000000
I/DEBUG (30186): d10 0000000000000000 d11 0000000000000000
I/DEBUG (30186): d12 0000000000000000 d13 0000000000000000
I/DEBUG (30186): d14 0000000000000000 d15 0000000000000000
I/DEBUG (30186): scr 20000011
I/DEBUG (30186):
I/DEBUG (30186): #00 pc 003a37a4 /system/lib/libwebcore.so (_ZNK5Layer8getChildEi)
I/DEBUG (30186): #01 pc 00285bae /system/lib/libwebcore.so
I/DEBUG (30186): #02 pc 0001edb0 /system/lib/libdvm.so (dvmPlatformInvoke)
I/DEBUG (30186): #03 pc 000594ba /system/lib/libdvm.so (_Z16dvmCallJNIMethodPKjP6JValuePK6MethodP6Thread)
I/DEBUG (30186): #04 pc 00030c0c /system/lib/libdvm.so
I/DEBUG (30186): #05 pc 000343e4 /system/lib/libdvm.so (_Z12dvmInterpretP6ThreadPK6MethodP6JValue)
I/DEBUG (30186): #06 pc 0006ce76 /system/lib/libdvm.so (_Z14dvmCallMethodVP6ThreadPK6MethodP6ObjectbP6JValueSt9__va_list)
I/DEBUG (30186): #07 pc 0006ce98 /system/lib/libdvm.so (_Z13dvmCallMethodP6ThreadPK6MethodP6ObjectP6JValuez)
I/DEBUG (30186): #08 pc 0005ffe0 /system/lib/libdvm.so
I/DEBUG (30186): #09 pc 00013048 /system/lib/libc.so (__thread_entry)
I/DEBUG (30186): #10 pc 00012b74 /system/lib/libc.so (pthread_create)
I/DEBUG (30186):
I/DEBUG (30186): code around pc:
I/DEBUG (30186): 5a347784 47a868e1 f57f69a0 4620e95c bf00bd7c .h.G.i..\. F|...
I/DEBUG (30186): 5a347794 0036eb46 3088f8d0 bf3a4299 3080f8d0 F.6....0.B:....0
I/DEBUG (30186): 5a3477a4 0021f853 47702000 e005b911 d0034288 S.!.. pG.....B..
I/DEBUG (30186): 5a3477b4 280068c0 e000d1fa 47702001 41f0e92d .h.(..... pG-..A
I/DEBUG (30186): 5a3477c4 f8c54605 460e10a0 f5d62400 4607fa1b .F.....F.$.....F
I/DEBUG (30186):
I/DEBUG (30186): code around lr:
I/DEBUG (30186): 5a229b90 b9084603 e0042001 f8d34620 f6a611a4 .F... .. F......
I/DEBUG (30186): 5a229ba0 bd10fd01 4614b510 2100b14b f11d4618 .......FK..!.F..
I/DEBUG (30186): 5a229bb0 4601fdf3 4620b118 ffe2f7ff 2001e000 ...F.. F.......
I/DEBUG (30186): 5a229bc0 bf00bd10 4ff0e92d 6d464607 460cb0a7 ....-..O.FFm...F
I/DEBUG (30186): 5a229bd0 0b90f10d 0a88f10d 52f4f8d6 f6fa4628 ...........R(F..
I/DEBUG (30186):
I/DEBUG (30186): stack:
I/DEBUG (30186): 5bf04bc8 5bf04c24
I/DEBUG (30186): 5bf04bcc 400828c0 /system/lib/libc.so
I/DEBUG (30186): 5bf04bd0 56d045f8 /dev/ashmem/dalvik-LinearAlloc (deleted)
I/DEBUG (30186): 5bf04bd4 00ffb568 [heap]
I/DEBUG (30186): 5bf04bd8 00000003
I/DEBUG (30186): 5bf04bdc 50fbff00
I/DEBUG (30186): 5bf04be0 5bf04c10
I/DEBUG (30186): 5bf04be4 400827f0 /system/lib/libc.so
I/DEBUG (30186): 5bf04be8 5bf04bf0
I/DEBUG (30186): 5bf04bec 40129393 /system/lib/libutils.so
I/DEBUG (30186): 5bf04bf0 000607dd
I/DEBUG (30186): 5bf04bf4 00000000
I/DEBUG (30186): 5bf04bf8 40eb4a68 /dev/ashmem/dalvik-heap (deleted)
I/DEBUG (30186): 5bf04bfc 40eb4a68 /dev/ashmem/dalvik-heap (deleted)
I/DEBUG (30186): 5bf04c00 df0027ad
I/DEBUG (30186): 5bf04c04 00000000
I/DEBUG (30186): #01 5bf04c08 56e6d840 /dev/ashmem/dalvik-LinearAlloc (deleted)
I/DEBUG (30186): 5bf04c0c 40822db4 /system/lib/libdvm.so
D/PlasmaWebView[1]( 575): Invalidate triggered
I/BootReceiver( 7248): Copying /data/tombstones/tombstone_05 to DropBox (SYSTEM_TOMBSTONE)
I/DEBUG (30186): debuggerd committing suicide to free the zombie!
I/WindowManager( 7248): WIN DEATH: Window{415980d8 com.regaliz.libneo/com.regaliz.libneo.NativeStory paused=false}
I/ActivityManager( 7248): Process com.regaliz.libneo (pid 575) has died.
W/WindowManager( 7248): Force-removing child win Window{416c4f88 SurfaceView paused=false} from container Window{415980d8 com.regaliz.libneo/com.regaliz.libneo.NativeStory paused=false}
I/flint (28979): jni: deleting text normalizer ...
I/flint (28979): jni: deleting synthesizer ...
I/flint (28979): jni: deleting project file ...
I/flint (28979): jni: deleting project resource holder ...
I/DEBUG ( 631): debuggerd: Oct 7 2012 16:35:04
D/Zygote ( 7206): Process 575 terminated by signal (11)
W/ActivityManager( 7248): Force removing ActivityRecord{41303a20 com.regaliz.libneo/.NativeStory}: app died, no saved state
Edit: looks like i have hit a critical bug of webkit: https://bugs.webkit.org/show_bug.cgi?id=93447 . Unfortunately its marked as WONTFIX because Google is moving away from webkit into the new rendering engine Blink... im scared :(
Edit 2: This is already reported on Android Bug Tool with 9 stars, but nobody seems to know why it happens :( https://code.google.com/p/android/issues/detail?id=24232
Edit 3: Too early to open champagne, but it looks like WebView doesn't like to be initialized with a heavy page load when inside a ViewSwitcher and the active View is not him(her/it)self ...
It's a little temerary to say, because other times the error appeared sooner or later, but after 5 days of intensive debugging I'm pretty sure I have found the recipe for disaster. I can conclude:
Putting a WebView inside a ViewFlipper, and loading a complex page in
it, while showing another ViewFlipper page (in my case a simple
throbber) would kill the Activity triggering the core dump
I still keep the WebView in the ViewFlipper, but as long as during Init (page load, page parse, body.onload) its viewflipper page is shown, I have no problems. As soon as I retry the throbber thing ... bang!
After the page is inited, and webview has stabilized, I can successfully change the ViewFlipper views.
A few modes of DEBUGing
Have you Asynced Yet?
Are you running the content in a seperate thread?
Can you reproduce this on another device/emulator?
This might be due to unsupported/changed libwebcore or other file versions.
Try the JS on a Desktop Browser?
If the javascript runs fine on a Standard Browser we have a degree of confidence. You can take this a level further and use Desktop tools to poke and analyze your JS code to see if there's problem there.
I have written an Android app entirely in C using the new native_app_glue code that comes with the latest Android NDK. It's all working fine except one strange thing. When I close down my app and start it again, I get a crash in my .so shared object. I used addr2line to find out the function that crashes, but it just doesn't make any sense because it's always a different function that causes the crash and of course, the first time it worked fine, too. So it must be something else.
Does anybody have an idea what could cause this behaviour? As I said, the behaviour is like this:
1) First time start --> app always works fine
2) Second time start --> app crashes
3) Third time start --> app works fine again
4) Fourth time start --> app crashes again
5) Fifth time start --> works fine again
6) Sixth time --> crash
and so on and so forth.
I don't know what's wrong there. Maybe my app doesn't get shut down correctly? But I'm getting and handling both term messages (APP_CMD_TERM_WINDOW and APP_CMD_DESTROY). I've really got no clue what's wrong there...
Thanks for help!
Edit: Here's the crash log:
I/DEBUG ( 31): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG ( 31): Build fingerprint: 'generic/sdk/generic:2.3.1/GSI11/93351:eng/test-keys'
I/DEBUG ( 31): pid: 1105, tid: 1114 >>> com.example.testapp <<<
I/DEBUG ( 31): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000058
I/DEBUG ( 31): r0 8382746d r1 00000000 r2 00000058 r3 00000058
I/DEBUG ( 31): r4 00000000 r5 8382746d r6 00242920 r7 00000554
I/DEBUG ( 31): r8 00245150 r9 001ec8e0 10 839651dc fp 00000000
I/DEBUG ( 31): ip 839654e0 sp 435fc2b0 lr 8381c78f pc 83861858 cpsr 00000030
I/DEBUG ( 31): #00 pc 00061858 /data/data/com.example.testapp/lib/libtestapp.so
I/DEBUG ( 31): #01 pc 0001c78a /data/data/com.example.testapp/lib/libtestapp.so
I/DEBUG ( 31): #02 pc 0001d7d4 /data/data/com.example.testapp/lib/libtestapp.so
I/DEBUG ( 31): #03 pc 000ab1fa /data/data/com.example.testapp/lib/libtestapp.so
I/DEBUG ( 31): #04 pc 000fc9e2 /data/data/com.example.testapp/lib/libtestapp.so
I/DEBUG ( 31): #05 pc 00011a7c /system/lib/libc.so
I/DEBUG ( 31): #06 pc 00011640 /system/lib/libc.so
It's possible that your library is still loaded when you try to restart your app. I know that loading the same lib twice will cause a crash.
the solution that I go with is in the same activity that you load the lib, override OnDestroy with a second same loadlibrary. that will crash the native side when the activity finishes (probably best to load your library on the launcher activity). that way you'll have a clean slate when you restart your app.
From your dump it seems that the problem is not in any native_app_glue call but in your source code.
Initially, if it's crashing calling a third party function, most probably, you would have, at least, one libc.so reference in your back trace but, instead, the first reference is to your own library. Can you, please, post the stack information? without seen the source code, I would bet for some kind of NULL pointer reference.
I am stuck with my app, as I am unable to debug as it is a multithreaded one and crashes with error SIGSEGV. I get lot of information from LogCat, which gives me addresses in my native library. It would be helpful if I could convert these addresses into my code.
Does anybody have any idea how to use addr2line, which is provided with android-ndk?
Let's say that logcat show you the following crash log (this is from one of my projects):
I/DEBUG ( 31): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
I/DEBUG ( 31): Build fingerprint: 'generic/sdk/generic:2.3/GRH55/79397:eng/test-keys'
I/DEBUG ( 31): pid: 378, tid: 386 >>> com.example.gltest <<<
I/DEBUG ( 31): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000000
I/DEBUG ( 31): r0 001dbdc0 r1 00000001 r2 00000000 r3 00000000
I/DEBUG ( 31): r4 00000000 r5 40a40000 r6 4051a480 r7 42ddbee8
I/DEBUG ( 31): r8 43661b24 r9 42ddbed0 10 42ddbebc fp 41e462d8
I/DEBUG ( 31): ip 00000001 sp 436619d0 lr 83a12f5d pc 8383deb4 cpsr 20000010
I/DEBUG ( 31): #00 pc 0003deb4 /data/data/com.example.gltest/lib/libnativemaprender.so
I/DEBUG ( 31): #01 pc 00039b76 /data/data/com.example.gltest/lib/libnativemaprender.so
I/DEBUG ( 31): #02 pc 00017d34 /system/lib/libdvm.so
Look at the last 3 lines; this is your callstack. 'pc' is the program counter, and the pc for stack frame #00 gives you the address where the crash occurred. This is the number to pass to addr2line.
I'm using NDK r5, so the executable I'm using is located at $NDK/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin; make sure that is in your $PATH. The command to use looks like
arm-linux-androideabi-addr2line -C -f -e obj/local/armeabi/libXXX.so <address>
Or, for the case above:
arm-linux-androideabi-addr2line -C -f -e obj/local/armeabi/libnativemaprender.so 0003deb4
Which gives you the location of the crash.
Note:
The -C flag is to demangle C++ code
Use the .so file under
obj/local/armeabi, since this is the
non-stripped version
Also, when using NDK r5 with a 2.3 AVD, it is actually possible to debug multithreaded code.
There's an easier way to do this now (ndk-r7). Check out the ndk-stack command. The docs are in you_android_ndk_path/docs/NDK-STACK.html