It is common knowledge that we can observe or query the view hierarchy of any app with an AccessibilityService:
Create your own accessibility service.
It is also possible to perform actions on behalf of the user:
Developing an Accessibility Service for Android.
My question is, can we modify the view hierarchy of a foreground app with an AccessibilityService?
I have already referred the following questions:
How do I add and remove a layout programmatically from an accessibility service?
Get view of AccessibilityNodeInfo to create overlay.
What they're doing is using the WindowManager and the SYSTEM_ALERT_WINDOW permission to overlay a view on top of the app in the foreground. The problem with this approach is that if the user presses BACK or HOME, the app is dismissed, but the view remains visible on the screen, even after the app is gone. The view is on TOP of the view hierarchy, and not a part of it.
Is there a way to add / modify the AccessibilityNodeInfo objects?
Are these the same as a View or ViewGroup?
Are we allowed to add views or modify existing views with an AccessibilityService?
My requirement is to display a small view within the app itself.
It has to be part of the view hierarchy of the app, so that it stays
or goes with the app. For this I need to modify the view hierarchy of
the app, namely the AccessibilityNodeInfo objects retrieved from the
service.
What I want is something similar to addView(), but add the View to the view hierarchy of the app itself, not on top of it.
How can we do this? Is this possible?
UPDATE:
Apps that support Custom Views for Accessibility
No, you can't modify the view hierarchy of another app because it exists in a separate process.
This is similar to not being able to modify accessibility nodes from within an accessibility service.
1) You can exploit draw over other apps permission. That solution will allow you only to draw overlays over another apps and not to change another apps behavior.
2) You can exploit instrumentation test mechanism. If you have enough information about the app(app id, activity name) and enough privileges (Run an instrument test from within app and wait for result), or root privileges. Here is an example:
#RunWith(AndroidJUnit4::class)
class InjectView {
#get:Rule
val activityRule = ActivityTestRule<MainActivity>(MainActivity::class.java)
#Test
fun injectView() {
val rootLayout = activityRule.activity.findViewById<ViewGroup>(android.R.id.content)
activityRule.runOnUiThread {
rootLayout.addView(TextView(activityRule.activity).apply {
text = "Injected View"
})
}
Thread.sleep(10_000)
}
}
You can still go with the approach of drawing on top of the app, not as part of its view hierarchy (which is impossible) - using SYSTEM_ALERT_WINDOW permission.
In order to know when the app is dismissed and dismiss your own overlay - listen to the accessibility event TYPE_WINDOW_STATE_CHANGED and check that there's a package change.
You can also go further with listening to TYPE_WINDOW_CONTENT_CHANGED and determining that there was some layout update.
More accessibility events might come in handy or fine-tuning your overlay's accuracy.
In short - as long as you have the appropriate information to know about the underlying app's layout and when things happen, you can draw on top as if it's part of the app.
Might be tricky and require some calculations since you're not able to just push views into the hierarchy, but totally doable.
Related
I'm implementing a screen for Android TV, which has a screen title and a button on the left side. And a list of custom views/rows(selectable/clickable), arranged vertically, on the right side of the page.
We want the button on the left to be in focus when the user sees that screen. For that, I'm calling button.requestFocus() in the onResume() of the fragment.
This breaks the accessibility. When talkback is enabled, the first thing announced is the button's label. What I want is to announce the title first and then the button's label.
I tried to announce a custom text(could be title) by
rootView.announceForAccessibility(accessibilityText)
where rootView is the root of the xml layout and accessibilityText a text which needs to be announced.
But it doesn't help, and the button's label gets the priority.
How can I solve the issue?
I would ask you to consider WCAG Guideline 3.2.1:
The intent of this Success Criterion is to ensure that functionality is predictable as visitors navigate their way through a document. Any component that is able to trigger an event when it receives focus must not change the context. Examples of changing context when a component receives focus include, but are not limited to:
forms submitted automatically when a component receives focus;
new windows launched when a component receives focus;
focus is changed to another component when that component receives focus; <-- emphasis here
Also a quote from the Android Accessibility Team:
So something similar that people like to do is manage accessibility focus themselves. And again, this is a bad idea. accessibility focus has to be determined by the accessibility service, and just like announcements this creates an inconsistency in experience. And actually, that one of the biggest issues that accessibility users face, inconsistency, across applications and over time.
With that said, you may want to consider looking at ensuring the focus order / priority of the component using the following attributes:
android:nextFocusUp
android:nextFocusDown
android:nextFocusLeft
android:nextFocusRight
And also ensure that any group component that may get highlighted has the importantForAccessibility attribute set.
I'd like to try help some more, but without an example XML file, it's difficult to get to your particular use case. Have you tried testing the view layout with accessibility users?
I took a cue from this article by ATAUL MUNIM. I added a check if talkback is enabled, before requesting the focus explicitly.
protected fun isTalkBackEnabled(): Boolean {
val a11yServices = context?.getSystemService(ACCESSIBILITY_SERVICE) as? AccessibilityManager
return a11yServices?.isTouchExplorationEnabled?:false
}
and
if(isTalkBackEnabled().not()) {
button.requestFocus()
}
This solution pretty much bailed me out from the problem I was facing. It was also the only way forward for me because my app's min API level is 21 which eliminates the option to use android:screenReaderFocusable
I tried to create an Android Accessibility Service to detect all Elements in an App and show Accessibility Information like Content Description or Labeled by.
At the moment I can, using the AccessibilityService Class, log the Node Hierarchy when opening an App:
switch (evt.getEventType()) {
case AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED: {
logNodes(getRootInActiveWindow(), 0);
But how can I get the information only of a View Element that was tapped/clicked? Is there an easy way, using Accessibility Events to do so?
If I use for example AccessibilityEvent.TYPE_VIEW_CLICKED I can detect clickable View Elements like Buttons or Checkboxes by clicking on them. But is there an Accessibility Event to get non clickable View Elements like Text-Views?
I know that Google's talback speaks whatever you tap on, using AccessibilityServices, and it doesn’t matter if it is a button or any other view Element.
I am trying to achieve a floating draggable view that will be displayed across all the activities of a single app.
Meaning there will be a view of 30% height and 50% width of the screen that the user will be able to drag on the screen and it will be drawn above any activity in the app. When the user will switch to another activity within the same app the floating view should remain in the same place.
I know it is possible using SYSTEM_ALERT_WINDOW permission, which I want to avoid because this view is only required to be displayed within the app and I don't want the users to be asked to approve such permission.
Another important point:
This view is going to be part of an SDK.
Meaning that I am not responsible for the activities/fragments/layouts of the app.
I can have/assume:
Base Application that the hosting app will subclass
All the root layouts have some identifier and are of some specific type (e.g. all the root layouts are RelativeLayout and have an id root_layout)
Similar questions that was asked 4-6 years ago with 0 satisfying answers:
Floating view over the whole application
Android floating view across activities
i can suggest you using a service running in the background holding that specific view. The use of a service will answer your need in terms where you want that specific view to remain operative until the user should choose to disable or close it manually. You need to use the asynchronous service which is the IntentService so it will not affect you ui main thread (don't forget the activitiy's life cycle and hierarchy).
see the following links please.
1 - link to google developers
https://developer.android.com/training/run-background-service/create-service
2 - link to an example of such implementation
https://medium.com/exploring-code/create-chat-heads-like-facebook-messenger-32f7f1a62064
Is it possible to access the view (or any other reference) of the recent application on Android? That is the window that appears on top when you long press the home button on some devices.
I have not found it in the view hierarchy of my Activity, nor in the Window. The only notification I get is a loss of focus.
Help appreciated!
No, sorry. This is not being displayed by your process, but rather by an OS process, so it is not in your foreground activity's view hierarchy. The same holds true for things like Toasts.
Already asked a similar question, yet without much luck.
Suppose I have a service and I need a view to pop up above it. In the same time, they both should be intractable, i.e. the user should be able to both click buttons within the view, as well as ones on the service in the background.
Is this in theory possible? If yes, how should I initialize that view?
Thanks!
Yes it's possible, what you need to do is call the WindowManager service and add your view via the same.
WindowManager windowManager=(WindowManager)getSystemService(WINDOW_SERVICE);
LayoutInflater inflater=(LayoutInflater)getSystemService(LAYOUT_INFLATER_SERVICE);
RelativeLayout layout=(RelativeLayout) inflater.inflate(R.layout.box,null);
You need a WindowManager.LayoutParams object which should contain the parameters for the layout
windowManager.addView(layout,params);
Well, adds the view
What you want is to add a view from your running service instance. This way you can persist the view across all activities - and from anywhere else. See this great example:
http://www.piwai.info/chatheads-basics/
Services most definitely can have a user interface: Input methods are an obvious example. See http://developer.android.com/resources/samples/SoftKeyboard/index.html for an example.
I guess you are misusing the word "Service".
Service is invisible, Activities are visible.
There are no buttons in an Service!
So you have no choice! You should put both views in one Activity, and I would use a RelativeLayout and set the visibility of your chidren to GONE/Visible.
http://developer.android.com/reference/android/widget/RelativeLayout.html
Also using a popup and making the layout under it clickable will disturb the user. You are completely changing User experience. I strongly suggest too make your popup appear at the top/bottom of your initial layout
Services run in the background and do not have an UI. So you can not show something over a Service.
If you need a Service to notify user about something, then use Notification.
Ayou could use a Toast, but I advise against it, as it can confuse users since it can pop-out over another app's activity.
What you want is an Activity instead of a Service and a Dialog instead View. I suggest you read this document by google: http://developer.android.com/guide/topics/fundamentals.html
However to answer your question about both being interactable. This isn't possible. At any given time 1 and only 1 activity is on the top of the activity stack. The user can only interact with that activity. If you want something like a floating window then will have to create it yourself. Although keep in mind that goes against the android design principles.