Develop Reference

How do root apps access the ui of other apps?

I am looking to build an app that (with root privileges) can access the ui on another app and click buttons on the screen. I have seen this done with many of the macro apps and some "bot" apps. I know they use the 'device administrator' and i have seen some that need the 'overlay over other apps' permission. I'm not looking for a comprehensive guide, and i know i would have to use some kind of ocr software to process the ui elements. Can someone give me some pointers? Example: https://forum.xda-developers.com/showthread.php?t=2241770
Thanks for any help!
Taine

I don't know how THEY do it, but one possible way is through the input shell command. I expect that normal programs don't have permission to use it, but adb shell does, and rooted programs presumably would. Usage is printed as follows:
Usage: input [<source>] <command> [<arg>...]
The sources are:
mouse
keyboard
joystick
touchnavigation
touchpad
trackball
stylus
dpad
touchscreen
gamepad
The commands and default sources are:
text <string> (Default: touchscreen) [delay]
keyevent [--longpress] <key code number or name> ... (Default: keyboard)
tap <x> <y> (Default: touchscreen)
swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
press (Default: trackball)
roll <dx> <dy> (Default: trackball)
You'd do something like
List<String> commandLine = Arrays.asList("input touchscreen swipe 500 10 500 500 100".split(" "));
Process process = Runtime.getRuntime().exec(commandLine);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
as mentioned in Any way to run shell commands on android programmatically? . (I've adapted it a bit, and haven't actually tested it, so it might take some fiddling.)
Note that it doesn't let you, e.g, see where buttons are, so you'll have to know what you're doing beforehand.
You might also try Instrumentation, as mentioned here: Android simulate key press

Log the Key Event when using Adb input

I use the command adb shell su -- getevent -lt /dev/input/event5 > Test.txt to log the touch event. It works when I touch the screen, i.e, it writes all actions (key down, key move and key up with the coordinates).
But, when I use the command adb shell input tap x y to simulate the touch, even if the device get the event (for example, the app is opened) but in the log file, there is no log lines about this tap.
How can I log the simulate the key event touch with adb?

The reason for you not being to see your input tap events in the getevent output is that sendevent and getevent work with Linux kernel input events and input command injects the events directly into the Android input event queue.
This (old but still useful) article has some nice diagrams illustrating the input event propagation in Android.
Unfortunately there is no easy (ready to use) way to do what you asked for. The closest thing I could think of is using dumpsys input command - it shows last 10 input events (including ones injected by the input command) in the RecentQueue: section.

What are the commands needed to write events in /dev/input/eventX?

I'm developing an android app that needs to send touch events to /dev/input/eventX. I know the C code structure to do such a thing is the following:
struct input_event {
struct timeval time;
unsigned short type;
unsigned short code;
unsigned int value;
};
To use such a code, I need to set NDK. Instead, I want to run equivalent linux commands using Runtime.getRunTime.exec () in android without the need to use NDK. Is there any way to do that?
If no, what are the rest of C code needed to send an event? For example, how can I send a touch event at x=200 and y=300 to event0? I searched and I didn't find a clear solution.
Thanks.

I'm not clear about why you need to send event to /dev/input/eventX directly. But if it can send via adb, you can inject many type events to device.
Try this on your computer:
adb shell input tap 200 300
Or this on your android device shell:
input tap 200 300
But it has a high delay because of outside injection.
More info about input command see here.
Usage: input [<source>] <command> [<arg>...]
The sources are:
mouse
keyboard
joystick
touchnavigation
touchpad
trackball
stylus
dpad
touchscreen
gamepad
The commands and default sources are:
text <string> (Default: touchscreen)
keyevent [--longpress] <key code number or name> ... (Default: keyboard)
tap <x> <y> (Default: touchscreen)
swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
press (Default: trackball)
roll <dx> <dy> (Default: trackball)

Does "adb shell input text" simulate software keyboard input?

adb shell input text "sometext"
OR
adb shell input keyevent eventid
do these simulate actual input from a virtual/ hardware keyboard respectively ?
I did not find any documentation for these commands on developer.android.com/
Is there any trusted documentation for these commands?

adb shell input help produces (after a long list of input devices):
The commands and default sources are:
text <string> (Default: touchscreen)
keyevent [--longpress] <key code number or name> ... (Default: keyboard)
tap <x> <y> (Default: touchscreen)
swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
press (Default: trackball)
roll <dx> <dy> (Default: trackball)
So it seems "text" is indeed for virtual (Default: touchscreen) and keyevent for physical (Default: keyboard).
You can override the text input device - adb shell input keyboard text "foo" works just fine. You can't send raw keycodes from the screen though.

Related to
adb shell input text "some\ text"
I only find the source code:
E. g. for Android 8.1 here.
The method private void sendText(int source, String text) { is relevant for the encoding.
And searching for KeyCharacterMap.VIRTUAL_KEYBOARD that is used in previous method I found a description here for heading "Virtual Key Character Map File".
From that info I looked into an old SM-G900F. I found under /system/usr/keychars/Virtual.kcm.
Inside of that file it maps the key events to the most common characters (mostly only ASCII).
So to your question:
do these simulate actual input from a virtual/ hardware keyboard
respectively ?
Yes, the code takes the text and tries to map it back via the KeyCharacterMap.VIRTUAL_KEYBOARD to key events and sends them.
In my experience all unknown characters lead to a cancellation of the whole text.
And you have to escape some characters - sometimes space with %s, or with '\ ', other special characters like & have to be escaped too.
And on some devices I experienced that a long text (ca. 200 characters) written with adb shell input text "<longText>" was partly lost - even the bigger part at the end!
It looked to me depending on the manufacturer.

There is a good description available for adb shell input * commands, just type
adb shell input help
For adb shell keyevent commands, different keycodes are available.

I am developing a work that evaluate the performance among different keyboards and tries to simulate real-user keyboard typing. The tool that I am using (android view client) uses input text to send text to the device. However, when using this tool to simulate text input, I observed differences in the behaviour of the keyboards, caused by the usage of this input method.
Using input text, the pointer location doesn't change and the keyboard doesn't show any keypress animations. Contrariwise, when using input tap X Y to press a key, the visual behaviour is the same as a real user taping the key. Also, the behaviour of the GBoard is different for both input methods. When using input text and then tapping a suggested word, the keyboard doesn't add a trailing space. The same doesn´t happen when using input tap. This helps to conclude that indeed there are differences between these two input methods.

"adb shell input keyevent eventid" for sure will not simulate real keyevent as the device id == 0.
what about "adb shell input text "sometext"" it is anyway not from pysical ... so I guess it will do is as clikcing on softkeyboard ?

How to use ADB to send touch events to device using sendevent command?

I am trying to send touch events to a device using AndroidDebugBridge, so that I can do some basic automation for UI tests. I have followed the discussion in LINK. I am able to use sendevent to simulate touch on emulators, but unable to do the same on a device.
Like in above link the emulator seems to send out 6 events for each touch ( xcoord, ycoord, 2 for press,2 for release) and it was easy to use this information to sendevents, but a getevent for the touchscreen for a device seems to generate far too many events.
Has somebody managed to send touch from ADB to a device? Could you please share the solution.

Android comes with an input command-line tool that can simulate miscellaneous input events. To simulate tapping, it's:
input tap x y
You can use the adb shell ( > 2.3.5) to run the command remotely:
adb shell input tap x y

In order to do a particular action (for example to open the web browser), you need to first figure out where to tap. To do that, you can first run:
adb shell getevent -l
Once you press on the device, at the location that you want, you will see this output:
<...>
/dev/input/event3: EV_KEY BTN_TOUCH DOWN
/dev/input/event3: EV_ABS ABS_MT_POSITION_X 000002f5
/dev/input/event3: EV_ABS ABS_MT_POSITION_Y 0000069e
adb is telling you that a key was pressed (button down) at position 2f5, 69e in hex which is 757 and 1694 in decimal.
If you now want to generate the same event, you can use the input tap command at the same position:
adb shell input tap 757 1694
More info can be found at:
https://source.android.com/devices/input/touch-devices.html
http://source.android.com/devices/input/getevent.html

2.3.5 did not have input tap, just input keyevent and input text
You can use the monkeyrunner for it: (this is a copy of the answer at https://stackoverflow.com/a/18959385/1587329):
You might want to use monkeyrunner like this:
$ monkeyrunner
>>> from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice
>>> device = MonkeyRunner.waitForConnection()
>>> device.touch(200, 400, MonkeyDevice.DOWN_AND_UP)
You can also do a drag, start activies etc.
Have a look at the api for MonkeyDevice.

You don't need to use
adb shell getevent -l
command, you just need to enable in Developer Options on the device [Show Touch data] to get X and Y.
Some more information can be found in my article here: https://mobileqablog.wordpress.com/2016/08/20/android-automatic-touchscreen-taps-adb-shell-input-touchscreen-tap/

Building on top of Tomas's answer, this is the best approach of finding the location tap position as an integer I found:
adb shell getevent -l | grep ABS_MT_POSITION --line-buffered | awk '{a = substr($0,54,8); sub(/^0+/, "", a); b = sprintf("0x%s",a); printf("%d\n",strtonum(b))}'
Use adb shell getevent -l to get a list of events, the using grep for ABS_MT_POSITION (gets the line with touch events in hex) and finally use awk to get the relevant hex values, strip them of zeros and convert hex to integer. This continuously prints the x and y coordinates in the terminal only when you press on the device.
You can then use this adb shell command to send the command:
adb shell input tap x y

Consider using Android's uiautomator, with adb shell uiautomator [...] or directly using the .jar that comes with the SDK.