I am looking to build an app that (with root privileges) can access the ui on another app and click buttons on the screen. I have seen this done with many of the macro apps and some "bot" apps. I know they use the 'device administrator' and i have seen some that need the 'overlay over other apps' permission. I'm not looking for a comprehensive guide, and i know i would have to use some kind of ocr software to process the ui elements. Can someone give me some pointers? Example: https://forum.xda-developers.com/showthread.php?t=2241770
Thanks for any help!
Taine
I don't know how THEY do it, but one possible way is through the input shell command. I expect that normal programs don't have permission to use it, but adb shell does, and rooted programs presumably would. Usage is printed as follows:
Usage: input [<source>] <command> [<arg>...]
The sources are:
mouse
keyboard
joystick
touchnavigation
touchpad
trackball
stylus
dpad
touchscreen
gamepad
The commands and default sources are:
text <string> (Default: touchscreen) [delay]
keyevent [--longpress] <key code number or name> ... (Default: keyboard)
tap <x> <y> (Default: touchscreen)
swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
press (Default: trackball)
roll <dx> <dy> (Default: trackball)
You'd do something like
List<String> commandLine = Arrays.asList("input touchscreen swipe 500 10 500 500 100".split(" "));
Process process = Runtime.getRuntime().exec(commandLine);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
as mentioned in Any way to run shell commands on android programmatically? . (I've adapted it a bit, and haven't actually tested it, so it might take some fiddling.)
Note that it doesn't let you, e.g, see where buttons are, so you'll have to know what you're doing beforehand.
You might also try Instrumentation, as mentioned here: Android simulate key press
Related
Setup
I'm on an Android 10 device and have an app (henceforth generator) that generates data it can copy to the device's clipboard (this involves swipes, taps, etc.). I need to do that repeatedly, as many times as possible (hundreds, thousands, whatever), and retrieve that data programmatically (presumably through a bash or perl script wrapped around adb commands).
Question
What would be some effective ways of achieving this?
Elaboration on what I've tried
I actually have a solution, I am just not sure it's the "best" one can do (whatever that means). generator is closed-source so I can't alter its behavior in any way, but I can repeatedly
start generator and/or switch focus to it with
adb shell monkey -p <generator-app> 1
mimic the tapping actions that produce the data and copy it to the clipboard with various
adb shell input tap x y
commands;
open/switch focus to the Termux terminal emulator (on the Android device):
adb shell monkey -p com.termux 1
access the clipboard via its Termux:API add-on, by running the termux-clipboard-get command in the termux terminal (again, on the Android device). I control this through adb with
adb shell input text 'termux-clipboard-get\>\>file-on-android-gadget.txt' && adb shell input keyevent 66
(keyevent 66 means ENTER, and the \>\> means append, >>, with escaped characters).
repeat the whole shebang.
This gives me many instances of the desired output in file-on-android-gadget.txt, saved locally on the phone (I'm OK retrieving it later, etc.).
Alternative routes?
What bothers me is the need to switch back and forth between two apps. I don't know that I have much choice over how I interact with generator, as it's closed-source and does not expose its internals, but perhaps one can do better at "the other end", i.e. when retrieving the clipboard data.
Newer Android APIs have locked down access to the clipboard, rendering it difficult to interact with programmatically, through adb. This SO thread points to a repo for a (Kotlin + gradle) app that sets the clipboard via a service runnable through adb, but my attempts to get the clipboard have been unsuccessful (i.e. getPrimaryClip() returns empty). I believe this is because, as per the docs:
getPrimaryClip
public ClipData getPrimaryClip ()
Returns the current primary clip on the clipboard. If the application
is not the default IME or does not have input focus this return null.
So there might just be no way around having to focus on a second app before the clipboard can be retrieved. But I have no experience programming apps, hence the question.
A further reference
During my peregrinations I found another repo for an Android app (Java + maven) that both gets and sets the clipboard. Same caveat though: as of Android 10, it needs to be in focus for the broadcast receiver to retrieve the clipboard data (other than that it works fine; I have it on my Android 10 device as we speak). So this would fare no better than my currently-working termux-based solution detailed above.
I use the command adb shell su -- getevent -lt /dev/input/event5 > Test.txt to log the touch event. It works when I touch the screen, i.e, it writes all actions (key down, key move and key up with the coordinates).
But, when I use the command adb shell input tap x y to simulate the touch, even if the device get the event (for example, the app is opened) but in the log file, there is no log lines about this tap.
How can I log the simulate the key event touch with adb?
The reason for you not being to see your input tap events in the getevent output is that sendevent and getevent work with Linux kernel input events and input command injects the events directly into the Android input event queue.
This (old but still useful) article has some nice diagrams illustrating the input event propagation in Android.
Unfortunately there is no easy (ready to use) way to do what you asked for. The closest thing I could think of is using dumpsys input command - it shows last 10 input events (including ones injected by the input command) in the RecentQueue: section.
I'm developing an android app that needs to send touch events to /dev/input/eventX. I know the C code structure to do such a thing is the following:
struct input_event {
struct timeval time;
unsigned short type;
unsigned short code;
unsigned int value;
};
To use such a code, I need to set NDK. Instead, I want to run equivalent linux commands using Runtime.getRunTime.exec () in android without the need to use NDK. Is there any way to do that?
If no, what are the rest of C code needed to send an event? For example, how can I send a touch event at x=200 and y=300 to event0? I searched and I didn't find a clear solution.
Thanks.
I'm not clear about why you need to send event to /dev/input/eventX directly. But if it can send via adb, you can inject many type events to device.
Try this on your computer:
adb shell input tap 200 300
Or this on your android device shell:
input tap 200 300
But it has a high delay because of outside injection.
More info about input command see here.
Usage: input [<source>] <command> [<arg>...]
The sources are:
mouse
keyboard
joystick
touchnavigation
touchpad
trackball
stylus
dpad
touchscreen
gamepad
The commands and default sources are:
text <string> (Default: touchscreen)
keyevent [--longpress] <key code number or name> ... (Default: keyboard)
tap <x> <y> (Default: touchscreen)
swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
press (Default: trackball)
roll <dx> <dy> (Default: trackball)
adb shell input text "sometext"
OR
adb shell input keyevent eventid
do these simulate actual input from a virtual/ hardware keyboard respectively ?
I did not find any documentation for these commands on developer.android.com/
Is there any trusted documentation for these commands?
adb shell input help produces (after a long list of input devices):
The commands and default sources are:
text <string> (Default: touchscreen)
keyevent [--longpress] <key code number or name> ... (Default: keyboard)
tap <x> <y> (Default: touchscreen)
swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
press (Default: trackball)
roll <dx> <dy> (Default: trackball)
So it seems "text" is indeed for virtual (Default: touchscreen) and keyevent for physical (Default: keyboard).
You can override the text input device - adb shell input keyboard text "foo" works just fine. You can't send raw keycodes from the screen though.
Related to
adb shell input text "some\ text"
I only find the source code:
E. g. for Android 8.1 here.
The method private void sendText(int source, String text) { is relevant for the encoding.
And searching for KeyCharacterMap.VIRTUAL_KEYBOARD that is used in previous method I found a description here for heading "Virtual Key Character Map File".
From that info I looked into an old SM-G900F. I found under /system/usr/keychars/Virtual.kcm.
Inside of that file it maps the key events to the most common characters (mostly only ASCII).
So to your question:
do these simulate actual input from a virtual/ hardware keyboard
respectively ?
Yes, the code takes the text and tries to map it back via the KeyCharacterMap.VIRTUAL_KEYBOARD to key events and sends them.
In my experience all unknown characters lead to a cancellation of the whole text.
And you have to escape some characters - sometimes space with %s, or with '\ ', other special characters like & have to be escaped too.
And on some devices I experienced that a long text (ca. 200 characters) written with adb shell input text "<longText>" was partly lost - even the bigger part at the end!
It looked to me depending on the manufacturer.
There is a good description available for adb shell input * commands, just type
adb shell input help
For adb shell keyevent commands, different keycodes are available.
I am developing a work that evaluate the performance among different keyboards and tries to simulate real-user keyboard typing. The tool that I am using (android view client) uses input text to send text to the device. However, when using this tool to simulate text input, I observed differences in the behaviour of the keyboards, caused by the usage of this input method.
Using input text, the pointer location doesn't change and the keyboard doesn't show any keypress animations. Contrariwise, when using input tap X Y to press a key, the visual behaviour is the same as a real user taping the key. Also, the behaviour of the GBoard is different for both input methods. When using input text and then tapping a suggested word, the keyboard doesn't add a trailing space. The same doesn´t happen when using input tap. This helps to conclude that indeed there are differences between these two input methods.
"adb shell input keyevent eventid" for sure will not simulate real keyevent as the device id == 0.
what about "adb shell input text "sometext"" it is anyway not from pysical ... so I guess it will do is as clikcing on softkeyboard ?
I need to test an Android application that takes multi-touch input, such as pinch and rotate. I'd like to test this using the emulator if possible. Is there a way to send Linux input events, via ADB or otherwise, without modifying the Android source?
https://source.android.com/devices/tech/input/touch-devices.html
Events such as ABS_MT_POSITION_X, ABS_MT_POSITION_Y, and ABS_MT_TRACKING_ID.
Old question but still open.
Indeed there are options:
1.) Send commands via telnet (https://developer.android.com/studio/run/emulator-console)
2.) Use adb like (input tap X Y), e.g.
adb shell input tap 100 100