Develop Reference

How to detect or monitor touching edittext in another android app?

Is it possible to monitor touching edit text behavior in another android app? For example, when touching edit text in specific android app, paste some words automatically which have been set up before.
PS. if it is not possible, any way to receive soft keyboard pop up broadcast ?

It's possible to detect when an EditText is touched, even when the EditText belongs to another app. However, this can only be done through Accessibility Services.
First, you need to create a serviceconfig.xml file in your xml folder:
<?xml version="1.0" encoding="utf-8"?>
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
android:description="#string/accessibility_permission_desc"
android:accessibilityEventTypes="typeAllMask"
android:accessibilityFeedbackType="feedbackGeneric"
android:notificationTimeout="100"
android:canRetrieveWindowContent="true"
/>
Second, you need to create an AccessibilityService, like so:
public class MyAccessibilityService extends AccessibilityService {
...
#Override
public void onAccessibilityEvent(AccessibilityEvent accessibilityEvent) {
switch(accessibilityEvent.getEventType()) {
case AccessibilityEvent.TYPE_VIEW_CLICKED:
case AccessibilityEvent.TYPE_VIEW_FOCUSED:
try {
Class className = Class.forName(accessibilityEvent.getClassName().toString());
if (EditText.class.isAssignableFrom(className)) {
// An EditText was Clicked or Focused
// Use other methods from the accessibilityEvent to do what
// you need to do
}
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
break;
}
}
...
}
Third, you need to add the service to the AndroidManifest.xml:
<service android:name=".MyAccessibilityService"
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
<intent-filter>
<action android:name="android.accessibilityservice.AccessibilityService" />
</intent-filter>
<meta-data
android:name="android.accessibilityservice"
android:resource="#xml/serviceconfig" />
</service>
Finally, you need to enable the AccessibilityService through the Accessibility Settings. You can go directly to there through:
startActivity(new Intent(android.provider.Settings.ACTION_ACCESSIBILITY_SETTINGS));
Edit:
You can't get the EditText through an AccessibilityEvent. Instead, you need to get an AccessibilityNodeInfo and perform typical EditText features through this AccessibilityNodeInfo.
To get the AccessibilityNodeInfo, you'll need to call this code:
AccessibilityNodeInfo nodeInfo = accessibilityEvent.getSource()==null ? null : accessibilityEvent.getSource();
Then, you can perform specific actions like setText() like so:
if (nodeInfo != null) {
nodeInfo.refresh();
Bundle bundle = new Bundle();
bundle.putString(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, newString);
nodeInfo.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, bundle);
}
Notice how I'm using performAction() onto the AccessibilityNodeInfo and using the action AccessibilityNodeInfo.ACTION_SET_TEXT and passing in a bundle with the key AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE.
This is almost the same as calling setText() on the EditText View.
If you don't want to setText() and want to just directly paste text into the EditText, you can use:
if (nodeInfo != null) {
nodeInfo.refresh();
nodeInfo.performAction(AccessibilityNodeInfo.ACTION_PASTE);
}

Android Clickable On screen text

I was curious, I seen this app the other day that allowed it to open other apps and set certain functions up for you automatically. I have came to realize that it must be using an on screen click function of some sort, but I can't seem to find any documentation for something like this. For example if we know the on screen text from the other app is "Ready", is there a way to read that text and maybe do something like:
protected void processText(String text)
{
if (text.contains("Ready"))
// click the ready text
}

I have done this using AccessibilityService. It will only work fine on API level >= 16 though.
You need to extend AccessibilityService. For instance this class will get text of USSD responses and dismiss the dialog.
// ....
public class UssdAccessibilityService extends AccessibilityService {
public UssdAccessibilityService() {
}
#TargetApi(16)
#Override
public void onAccessibilityEvent(AccessibilityEvent event) {
if (!"com.android.phone".equalsIgnoreCase((String)event.getPackageName())){
// In this example we are only interested in events comming
// from "com.android.phone" package
event.recycle();
return;
}
String className = (String)event.getClassName();
if (className == null || (!className.contains("AlertDialog") && !className.contains("AlertDialog"))){
// Class is not an USSD dialog
event.recycle();
return;
}
AccessibilityNodeInfo source = event.getSource();
if (source == null) {
// getSource() is annotated #Nullable, so we do this to be
// safe just in case
event.recycle();
return;
}
AccessibilityNodeInfo acceptButton = null;
String ussdText = null;
int childCount = source.getChildCount();
for (int i = 0; i < childCount; i++){
AccessibilityNodeInfo current = source.getChild(i);
if (current == null)
continue;
String currentText = (String)current.getText();
if (current.isClickable()){
// In the case of USSD dialogs, there is only one clickable.
// May be necessary to do more robust search in other scenarios
acceptButton = current;
continue;
}
ussdText = currentText;
current.recycle();
}
if (ussdText!= null) {
if (acceptButton != null)
acceptButton.performAction(AccessibilityNodeInfo.ACTION_CLICK);
}
source.recycle();
event.recycle();
}
// ....
}
You must declare the accessibility service in the manifest under <application>
<service
android:name=".UssdAccessibilityService"
android:enabled="true"
android:label="Read USSD codes and dismiss"
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
<intent-filter>
<action android:name="android.accessibilityservice.AccessibilityService" />
</intent-filter>
<meta-data
android:name="android.accessibilityservice"
android:resource="#xml/accessibility_service_config" />
</service>
Under res/xml create accessibility_service_config.xml
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
android:description="#string/accessibility_service_description"
android:packageNames="com.android.phone,com.ats.android.activationcodebot"
android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged"
android:accessibilityFlags="flagDefault"
android:accessibilityFeedbackType="feedbackSpoken"
android:notificationTimeout="100"
android:canRetrieveWindowContent="true"
/>
Of course you have to adapt this code to your own needs.
Finally you will have to enable the accessibility service manually on Settings > Accessibility in Android (or ask the user to do it).
Read more ... Developing an Accessibility Service

AccessibilityService not returning view ids

I'm writing an acessibility service in Android which relies on getting the view id of the currently selected view, however on some devices (Nexus 6P 6.0.1, Samsung Galaxy S6 edge+ 5 + 6.0.1) I get no view id through and on others (HTC One M8 5.0.1) it comes through fine. Because it works fine on some devices, I'm sure there's not a problem with my code, however I've posted a minimal test case below.
Can anyone can help me get my service reporting ids across all devices?
The A11y service
public class ViewIdLoggingAccessibilityService extends AccessibilityService {
#Override
public void onAccessibilityEvent(AccessibilityEvent event) {
AccessibilityNodeInfo source = event.getSource();
if (source == null) {
Log.d("onAccessibilityEvent", "source was null for: " + event);
} else {
String viewIdResourceName = source.getViewIdResourceName();
Log.d("onAccessibilityEvent", "viewid: " + viewIdResourceName);
}
}
#Override
public void onInterrupt() {
Log.d("!# onInterrupt", "called");
}
}
The AndroidManifest.xml
<manifest package="com.example.a11yservice"
xmlns:android="http://schemas.android.com/apk/res/android"
>
<application
android:allowBackup="true"
android:icon="#drawable/ic_launcher"
android:label="#string/app_name"
android:theme="#style/AppTheme"
>
<service
android:name=".ViewIdLoggingAccessibilityService"
android:label="#string/view_id_logging_service_label"
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"
>
<intent-filter>
<action android:name="android.accessibilityservice.AccessibilityService" />
</intent-filter>
<meta-data
android:name="android.accessibilityservice"
android:resource="#xml/serviceconfig"
/>
</service>
</application>
</manifest>
serviceconfig.xml
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
android:accessibilityEventTypes="typeViewClicked|typeViewFocused"
android:accessibilityFeedbackType="feedbackGeneric"
android:accessibilityFlags="flagReportViewIds"
android:canRetrieveWindowContent="true"
android:notificationTimeout="100" />

It's known issue and was posted to AOSP issue tracker some time ago. You can check status of issue here: "Accessibility Issue: flagReportViewIds has no effect on real devices in Android M".
Unfortunately issue exists even on latest Android 7.0 but I found simple workaround. You can call refresh() method on AccessibilityNodeInfo object to refresh it and collect all missed data including id.
Something like that:
public void onAccessibilityEvent(AccessibilityEvent event) {
AccessibilityNodeInfo ani = event.getSource();
if (ani != null) {
ani.refresh(); // to fix issue with viewIdResName = null on Android 6+
}
}

I found that if the viewId isn't available, querying the focused element seems to work:
AccessibilityNodeInfo source = findFocus(AccessibilityNodeInfo.FOCUS_INPUT);
viewIdResourceName = source.getViewIdResourceName();
returns the correct view id

How to interact with USSD dialog programmatically in android

I want to use USSD dialog which comes after dialing any USSD code say *123# which asks user to enter option number to perform specific task(s) depending upon sim card vendors. I need to interact with that dialog to provide input in the text box given into it programmatically.
However, I am able to read the USSD response that comes in Alert Dialog after dialing any USSD code, using AccessibilityService and I'm showing the response in a Toast as shown in the code below. I haven't found any solution to interact with USSD dialog yet.
public class UssdService extends AccessibilityService{
public static String TAG = "USSD";
#Override
public void onAccessibilityEvent(AccessibilityEvent event) {
Log.d(TAG, "onAccessibilityEvent");
String text = event.getText().toString();
if (event.getClassName().equals("android.app.AlertDialog")) {
Log.d(TAG, text);
Toast.makeText(this, text, Toast.LENGTH_LONG).show();
}
}
#Override
public void onInterrupt() {
}
#Override
protected void onServiceConnected() {
super.onServiceConnected();
Log.d(TAG, "onServiceConnected");
AccessibilityServiceInfo info = new AccessibilityServiceInfo();
info.flags = AccessibilityServiceInfo.DEFAULT;
info.packageNames = new String[]{"com.android.phone"};
info.eventTypes = AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED;
info.feedbackType = AccessibilityServiceInfo.FEEDBACK_GENERIC;
setServiceInfo(info);
}
}
Here is the service declaration in Manifest:
<service android:name=".UssdService"
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
<intent-filter>
<action android:name="android.accessibilityservice.AccessibilityService" />
</intent-filter>
<meta-data android:name="android.accessibilityservice"
android:resource="#xml/config_service" />
</service>

For interacting with USSD dialog, I used below code.
I used the below code for click event:
List<AccessibilityNodeInfo> list = nodeInfo.findAccessibilityNodeInfosByText("Send");
for (AccessibilityNodeInfo node : list) {
node.performAction(AccessibilityNodeInfo.ACTION_CLICK);
}
I used the below code for setText in EditText. This is setText where the current focus is.
AccessibilityNodeInfo nodeInput = nodeInfo.findFocus(AccessibilityNodeInfo.FOCUS_INPUT);
Bundle bundle = new Bundle();
bundle.putCharSequence(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE,pMPIN);
nodeInput.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT,bundle);
nodeInput.refresh();

My company Hover has developed an Android SDK which uses accessibility services to run multi-step USSD sessions and have it appear to happen inside your app. The underlying method is similar to what is outlined in the accepted answer, but there is more of an abstraction layer, support for lots of devices that behave differently, and it hides the session from the user.
You create configurations for USSD services, trigger the session to run from your app and pass in any runtime variables you need. When the response is returned your app is notified and you can parse it as you need. It works on Android 4.3 and above.
The SDK is free to integrate and use until you hit large scale. Please see our docs to get started.
(Disclosure: I am the CTO of Hover)

Glass voice command nearest match from given list

With Glass you can launch an app via the 'OK, Glass' menu and it seems to pick the nearest match unless a command is miles off, and you can obviously see the list of commands.
Is there anyway from within the app, or from the voice prompt (after the initial app trigger) to have a similar list given and return the nearest match.
Random (non-real world) example, an app that shows you a colour, "OK Glass, show the colour red"
'show the colour' could be your voice trigger and seems to be matched by glass on a 'nearest neighbor' method, however 'red' is just read in as free text and could be easily misheard as 'dread' or 'head', or even 'read' as there is no way of differentiating 'read' from 'red'.
Is there a way to pass a list of pre-approved option (red, green, blue, orange*, etc.) to this stage, or to another voice prompt within the app so the user can see the list and get more accurate results when there is a finite set of expected responses (like the main ok glass screen)?
*ok well nothing rhymes with orange, we're probably safe there

The Google GDK doesn't support this feature yet. However, the necessary features are already available in some libraries and you can use them as long as the GDK doesn't support this natively.
What you have to do:
Pull the GlassVoice.apk from your Glass: adb pull /system/app/GlassVoice.apk
Use dex2jar to convert this apk into a jar file.
Add the jar file to your build path
Now you can use this library like this:
public class VoiceActivity extends Activity {
private VoiceInputHelper mVoiceInputHelper;
private VoiceConfig mVoiceConfig;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.voice_activity);
String[] items = {"red", "green", "blue", "orange"};
mVoiceConfig = new VoiceConfig("MyVoiceConfig", items);
mVoiceInputHelper = new VoiceInputHelper(this, new MyVoiceListener(mVoiceConfig),
VoiceInputHelper.newUserActivityObserver(this));
}
#Override
protected void onResume() {
super.onResume();
mVoiceInputHelper.addVoiceServiceListener();
}
#Override
protected void onPause() {
super.onPause();
mVoiceInputHelper.removeVoiceServiceListener();
}
public class MyVoiceListener implements VoiceListener {
protected final VoiceConfig voiceConfig;
public MyVoiceListener(VoiceConfig voiceConfig) {
this.voiceConfig = voiceConfig;
}
#Override
public void onVoiceServiceConnected() {
mVoiceInputHelper.setVoiceConfig(mVoiceConfig, false);
}
#Override
public void onVoiceServiceDisconnected() {
}
#Override
public VoiceConfig onVoiceCommand(VoiceCommand vc) {
String recognizedStr = vc.getLiteral();
Log.i("VoiceActivity", "Recognized text: "+recognizedStr);
return voiceConfig;
}
#Override
public FormattingLogger getLogger() {
return FormattingLoggers.getContextLogger();
}
#Override
public boolean isRunning() {
return true;
}
#Override
public boolean onResampledAudioData(byte[] arg0, int arg1, int arg2) {
return false;
}
#Override
public boolean onVoiceAmplitudeChanged(double arg0) {
return false;
}
#Override
public void onVoiceConfigChanged(VoiceConfig arg0, boolean arg1) {
}
}
}

You can take advantage of the disambiguation step that occurs when multiple Activities or Services support the same Voice Trigger: simply have multiple Activities or Services in your application support "show me the color" as the voice trigger and label them with the color options.
Your manifest would look something like:
<application
android:allowBackup="true"
android:label="#string/app_name"
android:icon="#drawable/icon_50"
>
<activity
android:name="com.mycompany.RedActivity"
android:label="#string/red"
android:icon="#drawable/icon_red"
>
<intent-filter>
<action android:name="com.google.android.glass.action.VOICE_TRIGGER"/>
</intent-filter>
<meta-data
android:name="com.google.android.glass.VoiceTrigger"
android:resource="#xml/activity_start"
/>
</activity>
<activity
android:name="com.mycompany.BlueActivity"
android:label="#string/blue"
android:icon="#drawable/icon_blue"
>
<intent-filter>
<action android:name="com.google.android.glass.action.VOICE_TRIGGER"/>
</intent-filter>
<meta-data
android:name="com.google.android.glass.VoiceTrigger"
android:resource="#xml/activity_start"
/>
</activity>
<!-- ... -->
</application>
Those Activities or Services would only be used as a "trampoline" to launch the main logic of your app with the color selection.

If you haven't already, you should take a look at contextual voice menus that were added just a few weeks ago to the GDK. I had your exact same problem just the day before it was released, looking at it the next day and finding this helped me a lot! :)