Develop Reference

Access to raw sockets in Android instrumented tests

I've trawled through many of the answers here regarding socket access via native code on Android, and others regarding socket access for "androidTest" instrumented tests, but none are able to fully explain some odd behaviour I'm seeing.
Android Studio 2.3.3 on Windows 10, NDK 15.1.x, build tools 25.0.3. Building with Cmake. I am porting some native code over to Android intended to be distributed as a library that is wrapped in a Java JNI-based API. This part seems to be working fine; at least I can debug and log into the native code and see where things go wrong.
I've created a few instrumented tests to exercise the JNI, but it looks like the native side does not have access to sockets, even if the test app that Android Studio wraps up for your intrumented tests does (i.e., it has the INTERNET permission applied, and I can see that it is part of the pushed manifest of the test app. I also applied the ACCESS_NETWORK_STATE perm in a fit of pique.)
That is, parts of the library know how to set up and use TCP sockets (datagram and stream; in this case stream) and resolve DNS, which fails (This might be a bug in my port, since the device itself still seems to resolve DNS based on the adb shell output below). If I test with an IP address it retries until it fails. Each call to socket() returns an ERRNO of 11 EAGAIN ("Try again").
If I use the adb shell to login to the device under test over USB, I can ping and use curl, etc. But, as soon as I run-as as the test app I am no longer allowed to use any network device.
shell#venice:/data/data/org.clvrmnky.library.test $ ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
^C
--- www.example.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2004ms
1|shell#venice:/data/data/org.clvrmnky.library.test $ ping -I wlan0 www.example.com
ping: SO_BINDTODEVICE: Operation not permitted
2|shell#venice:/data/data/org.clvrmnky.library.test $ curl -4 --verbose www.example.com/
* Trying 93.184.216.34...
* connect to 93.184.216.34 port 80 failed: Connection timed out
* Failed to connect to www.example.com port 80: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to www.example.com port 80: Connection timed out
7|shell#venice:/data/data/org.clvrmnky.library.test $
I won't paste it here, but wlan0 (and others) exists and is UP with a valid, reachable IP address. It is "Link encap:UNSPEC" which I admit I don't fully grok.
I'm making an assumption that the app failing on the successive socket calls and the inability to bind to a network device in and use it in an adb shell are related in some manner, but if someone has a reason why this may not be so, please let me know.
I've tried:
Invoking the test methods in a background thread, just in case the instrumented tests were running in the test app main thread.
Setting the StrictMode thread policy to permitAll() in the #Before clause of the test class.
Running with and without the INTERNET permission set in the test manifest. Without this permission it reaches the native code, but fails much earlier in the initialization. I'd have to dig into my notes to fetch more details about that. I assume that I need this permission. (Side note: if I enable INTERNET then the remote debugger can no longer attach to the device, so I can debug via logs only.)

It looks like instrumented tests, at least as of this writing, do not support access to raw sockets through the NDK. The .test app that is auto-created seemed to have everything it needed in the manifest (I checked base.apk right from the device).
Once a DemoApp was created with identical manifest settings I was able to open sockets, connect to them, do name lookups, etc. Furthermore, I was able to access the network via adb shell when running as ("run-as") the DemoApp.
There is still something strange going on, as I created a separate JNI app that opens sockets, and hacked an instrumented test against that. At least briefly adb shell running as ext.other.app.test did have internet access.

Debugging a native application with GDB on Android

I have a native C++ app which is supposed to run on an Android device. The application crashes at the startup, most likely when calling a function from a linked library. I am trying to debug it with gdb, but I can't succeed.
I am starting gdbserver on the Android device on some arbitrarily picked port (2000):
shell#msm8996: gdbserver :2000 my_app
Process my_app created; pid = 3420
Listening on port 2000
Now I am trying to attach to this process on my Host system (Windows 7) with gdb that was provided in Android-NDK.
C:\> gdb
(gdb) attach 3420
Can't attach to process.
(gdb) target remote :2000
:2000: The system tried to join a drive to a directory on a joined drive.
What is the problem?
EDIT:
Prior to running gdb I forwarded the port 2000 using adb:
adb forward tcp:2000 tcp:2000
This at least helped me to establish some communication, but:
(gdb) target remote :2000
Remote debugging using :2000
warning: Architecture rejected target-supplied description
Remote 'g' packet reply is too long: 00000000000000000000000000000...
On the device side:
Listening on port 2000
Remote debugging from host 127.0.0.1
readchar: Got EOF
Remote side has terminated connection. GDBserver will reopen the connection.
Listening on port 2000

You are most probably using different architecture / version of gdb. When you start the gdb, it displays a line like (I am showing what my GDB shows):
This GDB was configured as "--host=x86_64-linux-gnu
--target=arm-Linux-android"
Check if this matches with your phone's architecture.
Downloading the correct GDB version may solve your problem.

Build time configuration of Android TCP/IP MTU

I know that it is possible to read and set the MTU during runtime e.g. with:
adb shell ip -d -s l l dev rmnet0
adb shell ifconfig rmnet0 mtu <MTU>
This has the problem that the MTU set this way does not hold after restarting the phone.
So how to configure the default MTU for an interface when building your own build with Android Open Source Project sources?
There seems to be no mention about this anywhere and grepping code does not seem to help. Also the default Linux kernel configuration files where this is done do not seem to exist in Android.
There sure must be a way to do this, right?

Apparently PMTUD is a better solution to this (RFC 4821).
It can be enabled in AOSP builds in init.rc (e.g. system/core/rootdir/init.rc) by adding following lines at the end of section "on boot":
on boot
# Other configurations here...
# Set TCP MTU Probing to automatic:
write /proc/sys/net/ipv4/tcp_mtu_probing 1

What is the ADB?

I keep reading tutorials where I'm supposed to enter something into the ADB command line and that it's in my Android sdk/platform-tools. So I find it, click on it, and a black screen comes up for about 2 seconds and while it's up, it scrolls through a bunch of text. So how am I supposed to use this "adb"?

It is called the Android Debug Bridge, and the Android Developers Site documentation does a better job of explaining it than I can:
http://developer.android.com/guide/developing/tools/adb.html
If you are looking for the adb command line, navigate to <sdk>/platform-tools/ and run
adb.exe shell
from the command line.

Pretty sure that is well documented since day 1 on the Android Debug Bridge
Android Debug Bridge (adb) is a versatile command line tool that lets
you communicate with an emulator instance or connected Android-powered
device. It is a client-server program that includes three components:
A client, which runs on your development machine. You can invoke a
client from a shell by issuing an adb command. Other Android tools
such as the ADT plugin and DDMS also create adb clients. A server,
which runs as a background process on your development machine. The
server manages communication between the client and the adb daemon
running on an emulator or device. A daemon, which runs as a background
process on each emulator or device instance.
So plain old English, ADB can be found on %ANDROID_HOME%/platform-toos/, and it's this magical command line that allows you to comunicate with your mobile device, either a physical or a Virtual device (AVD), so whenever you deploy you are passing the application through the device thanks to the ADB on a specific client port on your computer to the daemon port on the device.
Interesting things you can do with it?
Logcat: ./adb logcat allows you to see the log trace of each proces.
Install: ./adb install allows you to install apk to the device.
Killing:./adb kill-sever
Starting:./adb stat-server
Enter SQLite3: adb -s your_device shell
Use the monkey: adb shell monkey -v -p your.app.package 500 to
generate random events
And a lot more! Read the documentation it's beatiful and self-explanatory.

Where does Android store shutdown logs?

I know that the boot up log can be obtained by pulling out contents of kmsg or dmesg through ADB.
But I'm not aware of how to retrieve the shutdown logs in Android as there's no /var folder in Android (place where most desktop linux distros generally store their shutdown logs).
So how can I obtain the shutdown logs in Android?

Look in some locations such as these:
/proc/last_kmsg
/data/tombstones/
/data/dontpanic/
/data/system/dropbox/
(This list isn't strictly kernel logs, including framework and application logs too, which are also sometimes of interest)

One work around I found for collecting shutdown logs in Android is to run adb pull /proc/kmsg C:\Logs.txt on the host PC and then switch off the device. You will get the logs till the USB communication between the host and the device snaps! I know this is only one case out of the numerous shutdown scenarios but I haven't found satisfactory answers for other cases!

TL;DR:
Run command through adb that copies logcat and proc/kmsg to a file and keep it running even when adb disconnects with nohup, disown or setsid. Probably needs busybox, needs root and adb root, too.
setsid cat proc/kmsg > /sdcard/kmsg.txt &
and
logcat -v long -f /sdcard/logcat.txt (somehow only works without setsid)
Or add normal copy commands to some startup script.
/TL;DR
You can constantly copy proc/kmsg and logcat to a file on your android device or a microSD card to get the logs even after adb disconnects.
You need root access and adb root access for this to work. For the latter, use the setting in the developer options if you have a custom rom or the adbd insecure app.
After using adb shell to get your android shell, type su to get superuser access.
Then you not only need to put an ampersand (&) after the command but also make sure that the command keeps running after adb disconnects. That is done by nohup, disown or setsid (see here for usage).
If it doesn't work because you don't have these commands, you need to install busybox.
See my question here.
See here for how to get logcat and kernel logs and print it to some file or merge it.
See developer.android.com/tools/help/logcat.html for parameters for the logcat command.
In the end you could have a command like setsid cat proc/kmsg > /sdcard/kmsg.txt & for the kernel messages.
For logcat you could have one of the following commands: logcat -v long -f /sdcard/logcat.txt or logcat -v long > /sdcard/logcat.txt
I don't know why, but sometimes it didn't work with setsid and just didn't copy continuously but stopped shortly after executing the command. In these situations, it also showed up when entering jobs, which it didn't otherwise. Then it just worked without setsid, it stayed alive after disconnecting and reconnecting. I guess you must just try when the file does keep getting larger. If someone figured out why it is behaving like it is... let me know and I'll edit the answer.
Probably adding the commands to a startup script could be a solution for some, too.
Hope this helps.
fightcookie

Newer phones do NOT use any of these locations so if you're reading this article then as of now
The kernel crash logs are now in /sys/fs/pstore instead of /proc/last_kmsg

I was looking for the same thing, and finally, I found the answer!
In android 8 all logs are located in \data\log\android_logs\... including apps and kernel logs. Kernel logs are called kmsgcat-log_timestamp_.gz
edit: Although this is a very old thread, I think the answer might be helpful.