I am trying to implement an AccessibilityService that records the user's actions (only click events at this point) and stores them, such that they can be replayed at a later point in time.
For this I register for AccessibilityEvent.TYPE_VIEW_CLICKED events and check if I could somehow recover them later on (when all I have is a reference to the window / root node of the activity) by using two strategies:
Get the clicked view's id and look for this id in the root node's tree
Get the clicked view's text and look for this text in the root node's tree
I have tested this in various applications and different parts of the Android system and the results have been very confusing. About half of the views were not recoverable by any of the two strategies, and some views were sometimes reported as being recoverable and sometimes not. I found out that the latter was due to a race condition, since the accessibility service runs in a different process than the application to which the clicked views belong.
My question now is whether there is a better way to get a handle to a view in an accessibility and find this view again in a later execution of the application.
Below you find the code of my AccessiblityService class:
public class RecorderService extends AccessibilityService {
private static final String TAG = "RecorderService";
#Override
public void onAccessibilityEvent(AccessibilityEvent event) {
switch (event.getEventType()) {
case AccessibilityEvent.TYPE_VIEW_CLICKED:
AccessibilityNodeInfo node = event.getSource();
if (node == null) {
Log.i(TAG, "node is null");
return;
}
AccessibilityNodeInfo root = getRootInActiveWindow();
if (root == null) {
Log.i(TAG, "root is null");
return;
}
// Strategy #1: locate node via its id
String id = node.getViewIdResourceName();
if (id == null) {
Log.i(TAG, "id is null");
} else {
List<AccessibilityNodeInfo> rootNodes = root.findAccessibilityNodeInfosByViewId(id);
if (rootNodes.size() == 1) {
Log.i(TAG, "success (via id)");
return;
} else {
Log.i(TAG, "multiple nodes with that id");
}
}
// Strategy #2: locate node via its text
CharSequence text = node.getText();
if (text == null) {
Log.i(TAG, "text is null");
} else {
List<AccessibilityNodeInfo> rootNodes = root.findAccessibilityNodeInfosByText(text.toString());
if (rootNodes.size() == 1) {
Log.i(TAG, "success (via text)");
return;
}
}
Log.i(TAG, "failed, node was not recoverable");
}
}
#Override
protected boolean onKeyEvent(KeyEvent event) {
Log.i("Key", event.getKeyCode() + "");
return true;
// return super.onKeyEvent(event);
}
#Override
public void onInterrupt() {
}
}
I am developing this on SDK Version 21 (Lollipop) and testing it on a HTC Nexus M8 and a Samsung Galaxy Note2, both showing similar results.
Related
I have an AR Screen where I placed some 3d objects. I am using google AR Core with android SDK.
I need to remove those objects from the scene on clicking of a clear button.
I was able to remove the last placed object. But the other objects were not removing.
This is the code i am using.
if (newAnchor != null) {
arFragment.getArSceneView().getScene().removeChild(newAnchor);
newAnchor.getAnchor().detach();
you can detach android model from scene form using below code
List<Node> children = new ArrayList<>(arFragment.getArSceneView().getScene().getChildren());
for (Node node : children) {
if (node instanceof AnchorNode) {
if (((AnchorNode) node).getAnchor() != null) {
((AnchorNode) node).getAnchor().detach();
}
}
if (!(node instanceof Camera) && !(node instanceof Sun)) {
node.setParent(null);
}
}
You can detect if a user has tapped on a particular node like this - in this example the check is to see if the hit node is an 'Andy' renderable, i.e. the default example renderable with Sceneform:
private void handleOnTouch(HitTestResult hitTestResult, MotionEvent motionEvent) {
Log.d(TAG,"handleOnTouch");
// First call ArFragment's listener to handle TransformableNodes.
arFragment.onPeekTouch(hitTestResult, motionEvent);
// Check for touching a Sceneform node
if (hitTestResult.getNode() != null) {
Log.d(TAG,"handleOnTouch hitTestResult.getNode() != null");
Node hitNode = hitTestResult.getNode();
if (hitNode.getRenderable() == andyRenderable) {
//DO whatever you need to here...
}
return;
And you can delete a node like this:
private void removeAnchorNode(AnchorNode nodeToremove) {
//Remove an anchor node
if (nodeToremove != null) {
arFragment.getArSceneView().getScene().removeChild(nodeToremove);
anchorNodeList.remove(nodeToremove);
nodeToremove.getAnchor().detach();
nodeToremove.setParent(null);
nodeToremove = null;
} else {
//Handle error case here
}
}
Full code available here: https://github.com/mickod/LineView - the above are edited excerpts.
If you need to cleanup the scene, you can do the following:
arSceneView.getScene().callOnHierarchy(node -> {
node.setParent(null);
if (node instanceof AnchorNode) {
((AnchorNode) node).getAnchor().detach();
}
}
I am developing a floating window, which will be shown on apps I am observing using AccessibilityService.
But I don't want the window shown on other apps.
So I firstly configure the android:packageNames property of <accessibility-service>. Then find some solutions to check whether user is using apps I configured, like getRunningTasks, getRunningAppProcesses and UsageStatsManager. But they all have some shortcomings.
The question is:
Is there any way to check whether apps I configured using Accessibility is foreground currently?
OR:
Is there any way to make the window shown on apps I configured using Accessiblity and not shown on other apps?
Thanks a lot!
Thanks to idea of #ataulm !
I find a perfect way of checking whether my apps are in foreground:
set accessibility's packageName to all
set accessibility's EventTypes to typeAllMask
Java:
public void onAccessibilityEvent(AccessibilityEvent event) {
if (event == null || event.getSource() == null) {
return;
}
AccessibilityNodeInfo source = event.getSource();
String className = source.getClassName().toString();
switch (event.getEventType()) {
case AccessibilityEvent.TYPE_VIEW_SCROLLED:
Log.i(TAG, "onAccessibilityEvent: scrolled");
break;
case AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED:
setForegroundApp(event.getPackageName());
break;
}
}
public static String sForegroundPackageName;
static final List<String> supportedPackagesName = blablablabla;
public static void setForegroundApp(CharSequence packageName) {
if (packageName != null) {
sForegroundPackageName = packageName.toString();
}
}
// use this to check.
public static boolean isForeground() {
return supportedPackagesName.contains(sForegroundPackageName);
}
I am using accessibility service for reading top package. But now I need to stop app and stop it's all running services through accessibility.
I have been searching on same found clean master doing same.
Please suggest how can I do same?
Using accessibility service you can force stop application.Ref below code
public class MyAccessibilityService extends AccessibilityService {
#Override
public void onAccessibilityEvent(AccessibilityEvent event) {
//TYPE_WINDOW_STATE_CHANGED == 32
if (AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED == event
.getEventType()) {
AccessibilityNodeInfo nodeInfo = event.getSource();
if (nodeInfo == null) {
return;
}
List<AccessibilityNodeInfo> list = nodeInfo
.findAccessibilityNodeInfosByViewId("com.android.settings:id/left_button");
//We can find button using button name or button id
for (AccessibilityNodeInfo node : list) {
node.performAction(AccessibilityNodeInfo.ACTION_CLICK);
}
list = nodeInfo
.findAccessibilityNodeInfosByViewId("android:id/button1");
for (AccessibilityNodeInfo node : list) {
node.performAction(AccessibilityNodeInfo.ACTION_CLICK);
}
}
}
#Override
public void onInterrupt() {
// TODO Auto-generated method stub
}
}
Open Setting screen click application which u want to force stop from installed application.
https://developer.android.com/guide/topics/ui/accessibility/services.html#event-details
I'm working in Accessibility Service. i need to get the app name using Accessibility Service. i have studied the documentation of the Accessibility Service in developer's of Android website. but there is no mention about getting the app name using Accessibility.
I also want to extract text from "TextViews" of the other apps(Activities) running in background. How i can do this..
I'm assuming you know how to implement an AccessibilityService.
Retrieving window content:
First register for TYPE_WINDOW_CONTENT_CHANGED events.
#Override
public void onAccessibilityEvent(AccessibilityEvent accessibilityEvent){
int eventType = accessibilityEvent.getEventType();
switch (eventType) {
case AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED:
AccessibilityNodeInfo rootNode = getRootInActiveWindow();
ArrayList<AccessibilityNodeInfo> textViewNodes = new ArrayList<AccessibilityNodeInfo>();
findChildViews(rootNode);
for(AccessibilityNodeInfo mNode : textViewNodes){
if(mNode.getText()==null){
return;
}
String tv1Text = mNode.getText().toString();
//do whatever you want with the text content...
}
break;
}
}
Method findChildViews() :
private void findChildViews(AccessibilityNodeInfo parentView) {
if (parentView == null || parentView.getClassName() == null || ) {
return;
}
if (childCount == 0 && (parentView.getClassName().toString().contentEquals("android.widget.TextView"))) {
textViewNodes.add(parentView);
} else {
for (int i = 0; i < childCount; i++) {
findChildViews(parentView.getChild(i));
}
}
}
}
As far as i know, there's no assured way to get the app name, but you can try fetching the text content from the event object you get from TYPE_WINDOW_STATE_CHANGED events.
Try dumping accessibilityEvent.toString() & you'll know what i'm talking about.
Also, accessibilityEvent.getPackageName() is a simple way to get package name of that app; in case you find it useful!
previous answer is missing definition of childCount
int childCount = parentView.getChildCount();
I am struggling with one very strange bug in my app.
I have added TTS to it, and I am using the build one. The user can choose the language from the spinner which is filled in during AsyncTask started in onResume().
The AsyncTask looks like this:
private class AsyncTTSDownload extends AsyncTask<Void, Integer, String> {
#Override
protected String doInBackground(Void... params) {
try {
languagesTTS = tts.testLang();
} catch (Exception ex) {
if (D)
Log.e(TAG, ex.toString());
}
return null;
}
#Override
protected void onPostExecute(String result) {
ttsUpdate.dismiss();
TTSSpinnerAdapter adapterTTS = new TTSSpinnerAdapter(
MyTTS.this, android.R.layout.simple_spinner_item,
languagesTTS);
int savedLangTTS = ttsLang.getInt("savedTTS", -1);
langTTS.setAdapter(adapterTTS);
if (savedLangTTS == -1)
{
try {
int langObject = languagesTTS.indexOf(tts.getLanguage());
langTTS.setSelection(langObject);
} catch (IndexOutOfBoundsException ie) {
langTTS.setSelection(0);
}
} else {
langTTS.setSelection(savedLangTTS);
}
Locale langChoosen = (Locale) langTTS.getItemAtPosition(langTTS
.getSelectedItemPosition());
tts.setTTSLanguage(langChoosen);
}
#Override
protected void onPreExecute() {
ttsUpdate = ProgressDialog.show(MyTTS.this, "Wait",
"Loading TTS...");
ttsUpdate.setCancelable(false);
}
}
the thing is, that I am from time to time getting different number of languages supported. This is on this same device, during this same run. Just I open and close Activity with TTS. This bug is causing IndexOutOfBoundsException. This is my way of getting TTS languages:
public List<Locale> testLang() {
Locale[] AvalLoc = Locale.getAvailableLocales();
List<Locale> listaOK = new ArrayList<Locale>();
String tester = "";
for (Locale l : AvalLoc) {
if(tester.contains(l.getLanguage()))
{
continue;
}
int buf = tts.isLanguageAvailable(l);
if (buf == TextToSpeech.LANG_MISSING_DATA
|| buf == TextToSpeech.LANG_NOT_SUPPORTED) {
//TODO maybe
} else {
listaOK.add(l);
tester += l.getLanguage() + ";";
}
}
tts.setLanguage(Locale.ENGLISH);
return listaOK;
}
For now I've only find out a small hack for not showing this error, just save in shared preferences number of languages and compare it with what tts received, but it is not working well at all. Each time I am getting different number.
For me it seems, that something is not finished or started when I am starting again this same activity after return, because this is tts.isAvaliableLanguage(l) who is deciding whether language is supported or not and from time to time, one language is not supported and after reload it is.
EDIT:
As there appeared new comment about my question I need to add one important thing about TTS engine itself.
testLang() is a method inside my class Called TTSClass, that is implementing TextToSpeech.OnInitListener. tts object is created in onCreate of MyTTS activity and this constructor looks like this in TTSClass:
public TTSClass(Context context, Locale language) {
contextTTS = context;
languageTTS = language;
tts = new TextToSpeech(contextTTS, this);
}
and call in activity:
tts = new TTSClass(getApplicationContext(), Locale.ENGLISH);
Because TTSClass implements TextToSpeech.OnInitListener there is also onInit() method which looks like this:
#Override
public void onInit(int status) {
if (status == TextToSpeech.SUCCESS) {
int result = 0;
result = tts.setLanguage(languageTTS);
if (result == TextToSpeech.LANG_MISSING_DATA
|| result == TextToSpeech.LANG_NOT_SUPPORTED) {
if(D) Log.e(TAG, "This Language is not supported");
}
if(D) Log.d(TAG,"Initialized");
} else {
if(D) Log.e(TAG, "Initilization Failed!");
}
}
So, this is everything connecting to this class and problem I think. If anything is missing, let me now.
EDIT2:
Suggested by shoe rat comment I've run few more tests, and the outcome is just amazing, or extraordinary, I think it is better word.
So what I've done was adding 3 Log from different places in code informing me about list size on different stages.
First was added in onInit() in if status == TextToSpeech.SUCCESS. This one is just simple call of testLang().size(). The outcome is 5 languages - that is the correct number and it is always like this, no matter if there is or isn't an exception.
Second was added there:
protected String doInBackground(Void... params) {
try {
Log.w(TAG,"before: "+tts.testLang().size());
languagesTTS = tts.testLang();
}
and this one is starting to act quite weird. It is sometimes, or even quite often, showing number lower than 5. But this is not the strangest thing.
The third one is just at the beginning of onPostExecute checking the size of languagesTTS. And believe or not, the number is quite often totally different from the second log. However, it is never smaller. It can be equal or bigger.
Does anyone know, what is going one?
I've found solution. It came out that indeed it was initialization problem.
I'm not sure if documentation is saying anything about it, but it seem like the TTS engine initialization is done asynchronously, so it can finish at any time.
My solution was to change the doInBackground() method like this:
#Override
protected String doInBackground(Void... params) {
try {
while(!TTSClass.isInit){}
languagesTTS = tts.testLang();
} catch (Exception ex) {
if (D)
Log.e(TAG, ex.toString());
}
return null;
}
and in onInit() method I've added isInit public static boolean variable:
#Override
public void onInit(int status) {
if (status == TextToSpeech.SUCCESS) {
int result = 0;
result = tts.setLanguage(languageTTS);
if (result == TextToSpeech.LANG_MISSING_DATA
|| result == TextToSpeech.LANG_NOT_SUPPORTED) {
if(D) Log.e(TAG, "This Language is not supported");
}
if(D) Log.d(TAG,"initialized");
isInit = true;
} else {
if(D) Log.e(TAG, "Initilization Failed!");
}
}
Hope, that someone will find it helpful.