Develop Reference

How to extract information about UI elements in an android app at runtime?

Hierarchy Viewer lets you extract information about the UI elements by connecting your phone to the laptop. It works by polling the Window Manager/View Manager for information via adb.
I was wondering if it is possible to extract UI information of the foreground screen on the phone at runtime using a background service?
I am trying to build an accessibility service (similar to Talkback), and was trying to find a way to collect information about all the UI elements (their class (e.g. button) and location) on the active foreground screen. I have been able to make an accessibility service, but I have not been able to find a way to get the UI info.

I wrote some code and works fine, maybe it's what you want.
Firstly, get the content view:
ViewGroup parent = (ViewGroup) findViewById(android.R.id.content);
Then use a method to check every element of the parent view:
private void showViewInfo(View view){
if(view instanceof ViewGroup){
//do something when find view group:
System.out.println(view.getClass().toString()+"\tx:"+view.getX()+"\ty:"+view.getY());
for(int i=0;i<((ViewGroup) view).getChildCount();i++){
showViewInfo(((ViewGroup) view).getChildAt(i));
}
}else{
//do something when find children
System.out.println(view.getClass().toString()+"\tx:"+view.getX()+"\ty:"+view.getY());
}
}
Finally, call the method with the parent view as parameter:
showViewInfo(parent);
and you can get something like this:
System.out: class android.support.v7.widget.ContentFrameLayout x:0.0 y:0.0
Here is whole code:
public class MainActivity extends AppCompatActivity {
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
ViewGroup parent = (ViewGroup) findViewById(android.R.id.content);
showViewInfo(parent);
}
private void showViewInfo(View view){
if(view instanceof ViewGroup){
//do something when find view group:
System.out.println(view.getClass().toString()+"\tx:"+view.getX()+"\ty:"+view.getY());
//find the children
for(int i=0;i<((ViewGroup) view).getChildCount();i++){
showViewInfo(((ViewGroup) view).getChildAt(i));
}
}else{
//do something when find children
System.out.println(view.getClass().toString()+"\tx:"+view.getX()+"\ty:"+view.getY());
}
}
}

I was able to find an answer to my problem.
The Android accessibility API provides this functionality. I was able to implement an Accessibility Service and for each accessibility event, I was able to use the Accessibility Node Info functions, which is essentially a tree representation of all the UI elements on the foreground screen. For example A linear layout consists of a button and an image, then the button and the image would be children for the LL.
The root of the tree can be found by:
AccessibilityNodeInfo rootNode = getRootInActiveWindow();
Then I was able to traverse the tree (Breadth first search) using the getChild() function which gives me a list of all the UI elements on the active screen.
Hope it helps someone else too!

Instagram force touch menu on Android

Instagram on Android recently add new feature, when user long click to items at the Browse section, a popup menu shows and allow user to pre-view photo/video instead of going to its details.
It is really cool like iOS force touch feature.
Does anyone know any idea how we can do the same on Android app?
Can we just use Context Menu or Overlay Window to do that?
Thanks

Now I can do quite the same on UI with this library
https://github.com/tvbarthel/BlurDialogFragment
But the thing is:
When I long press the button, I have to RELEASE my finger to continue touching the dialog fragment. The touch event is still sent to activity not DialogFragment.
Do you know how to pass touch event / focus to dialog fragment right after it is showed?

It's the closest library I've found on the internet,
https://github.com/nantaphop/HoverTouchView
it might not be exactly like IG's, but for sure it gives the idea

I Have found 3DTouch as well as in Instagrame.
This library is hosted on Jitpack.io, which means to use it you will have to add the following to your root build.gradle file.
allprojects {
repositories {
...
maven { url "https://jitpack.io" }
}
}
And then you will need to add the following dependency to your applications build.gradle file.
dependencies {
compile 'com.github.shalskar:PeekAndPop:v1.0.1'
}
Usage:
Basic usage is easy, simply provide an activity instance, a layout resource for the peek and pop and 1 or more views that will show the peek and pop when long clicked.
PeekAndPop peekAndPop = new PeekAndPop.Builder(this)
.peekLayout(R.layout.peek_view)
.longClickViews(view)
.build();
You can get the peek view by calling getPeekView() on the PeekAndPop object, and use findViewById() to get access any views in the peek layout.
View peekView = peekAndPop.getPeekView();
ImageView imageView = peekView.findViewById(R.id.image_view);
TextView textView = peekView.findViewById(R.id.text_view);
Often you will want to have the peek and pop shown when an item in a list (or other scrollable view) is clicked on, to ensure the peek and pop works correctly, you will have to add this line of code:
.parentViewGroupToDisallowTouchEvents(viewGroup)

AppGyver Supersonic navigating between Views and creating duplicates

I'm using AppGyver Steroids and Supersonic to build an app and I'm having some issues navigating between views programmatically.
Based on the docs, you navigate between views like this:
var view_obj = new supersonic.ui.View("main#index");
supersonic.ui.layers.push(view_obj);
However, when I inspect things via the Chrome DevTools, it appears that a second duplicate view is created i.e. If I navigate away from the index page and then navigate back, I now have two index pages, instead of what [I think] should be one. It also doesn't close the previous view I was on.
How can I prevent this from happening and simply move to the existing view, instead of duplicating views? How do I close a view after I have navigated away from it?
Thanks.

The problem you're encountering is that you're creating a new supersonic.ui.View("main#index") every time you navigate. On top of this, I think you want to return to the same view when you navigate back to a view for the second time, i.e. you want the view to remain in memory even if it has been removed from the navigation stack with pop() (rather than pushing a new instance of that view). For this, you need to preload or "start()" the view, as described in the docs here.
I implemented my own helper function to make this easier; here is my code:
start = function(dest, isModal) {
var viewId=dest,
view=new supersonic.ui.View({
location: dest,
id: viewId
});
view.isStarted().then(function(started) {
if (started) {
if (isModal) {supersonic.ui.modal.show(view);}
else {supersonic.ui.layers.push(view);}
} else {
// Start Spinner
supersonic.ui.views.start(view).then(function() {
if (isModal) {supersonic.ui.modal.show(view);}
else {supersonic.ui.layers.push(view);}
// Stop Spinner
}, function(error) {
// Stop Spinner
A.error(error);
});
}
});
};
Use it like start('module#view');. As a bonus, you can pass true as the second argument and it gets pushed as a modal instead.
It checks if you've already started a view - if so, it just pushes that view back onto the stack. If not, it start()s (i.e. preloads) it, then pushes it. This ensures that the view stays in memory (with any user input that has been modified) even when you pop() it from the stack.
You have to imagine that the layer stack is actually a stack in the Computer Science sense. You can only add and remove views at the top of the stack. The consequence of this is that complex navigations such as A > B > C > D > B are difficult/hacky to do (in this case, you'd have to pop() D and C in succession to get back to B).
Views will close if you pop() them, as long as you didn't start() them. If you did, and you pop() them, they remain in memory. To kill that view, you have to call stop() on it, as described in the docs I linked above.

try
var view_obj = new supersonic.ui.View("main#index");
supersonic.ui.layers.replace(view_obj);
And take a look at supersonic.ui.layers.pop();

Thanks to LeedsEbooks for helping me get my head around this challenge. I was able to find a solution. Here is the code:
var start = function(route_str, isModal) {
var regex = /(.*?)#(.*)/g;
var match_obj = regex.exec(route_str);
var view_id_str = match_obj[2],
view_location_str = route_str,
view = new supersonic.ui.View({
location: view_location_str,
id: view_id_str
});
view.isStarted().then(function(started) {
if (started)
{
if (isModal)
{
supersonic.ui.modal.show(view);
}
else {
supersonic.ui.layers.push(view);
}
}
else
{
// Start Spinner
supersonic.ui.views.start(view).then(function() {
if (isModal)
{
supersonic.ui.modal.show(view);
}
else
{
supersonic.ui.layers.push(view);
}
// Stop Spinner
}, function(error) {
// Stop Spinner
A.error(error);
});
}
});
};
You must ensure that your route has the format module#view as defined in the documentation.
PLEASE NOTE
There seems to some problem with the supersonic ui method for starting views. If you run the following code:
supersonic.ui.views.start("myapp#first-view");
supersonic.ui.views.find("first-view").then( function(startedView) {
console.log(startedView);
});
You'll notice that your view id and location are identical. This seems to be wrong as the id should be first-view and location should be myapp#first-view.
So I decided to not use the AppGyver methods and create my own preload method instead, which I run from the controller attached to my home view (this ensures that all the views I want to preload are handled when the app loads). Here is the function to do this:
var preload = function(route_str)
{
var regex = /(.*?)#(.*)/g;
var match_obj = regex.exec(route_str);
var view = new supersonic.ui.View({
location: route_str,
id: match_obj[2]
});
view.start();
};
By doing this, I'm sure that the view will get loaded with the right location and id, and that when I use my start() function later, I won't have any problems.
You'll want to make sure that your structure.coffee file doesn't have any preload instructions so as not to create duplicate views that you'll have problems with later.
Finally, I have a view that is 2 levels in that is a form that posts data via AJAX operation. I wanted the view to go back to the previous view when the AJAX operation was complete. Using my earlier function resulted in the push() being rejected. It would be nice if AppGyver Supersonic could intelligently detect that pushing to a previous view should default to a layers.pop operation, but you don't always get what you want. Anyway, I managed to solve this using supersonic.ui.layers.pop(), which simply does what the Back button would have done.
Everything working as intended now.

How to make a ViewFlipper behave like a Scroller?

Good day everyone.
I am creating a calendar component, and I'm working in the month view. I have created a view named MonthView, and I am adding a couple instances of this to a ViewFlipper:
viewFlipper = new ViewFlipper(getContext());
viewFlipper.addView(new MonthView(viewFlipper.getContext()));
viewFlipper.addView(new MonthView(viewFlipper.getContext()));
I have implemented the fling gesture so that I change views when sliding my finger left or right. This will cyclically update and display the months.
Now, I need to give the fling gesture a smoothly effect when touching and slowly sliding my finger. The same we get when we use a Slider instead a ViewFlipper.
The problem with Scroller is that the effect is not cyclic. Once I get to the last view, I have to slide in the other direction.
I need someone help me find how to give a scroll-like effect to the ViewFlipper, or how to make a Scroller cyclic.
Thanks in advance.
Extra comment:
I have already implemented a ViewFlipper with 2 views. I update the views by using the SimpleOnGestureListener.onFling(...) method, and the behavior I got is something like this:
Imagine I always slide from rigth to left, like flipping a book's page to read the next one, and also imagine there is a caption in the header of the view that is displayed after flipping.
View # 0 --> Caption: January 2011
View # 1 --> Caption: Febrary 2011
View # 0 --> Caption: March 2011
View # 1 --> Caption: April 2011
View # 0 --> Caption: May 2011
If at this point I slide from left to right, the result will be something like:
View # 1 --> Caption: April 2011
View # 0 --> Caption: March 2011
The ability to cyclically move forward or backward, giving the user the idea of having infinite views, but using only a couple is characteristic of ViewFlipper, and that's what I can't loose.
That's why I need a way to add the cool scroll effect without loosing what I've got.
Thanks.

Then you can use ViewFlinger!
viewflinger this an android widget (extends ViewGroup) that allows to group a set of views that can be swiped horizontally. It offers a smoother transition that cannot be accomplished using ViewFlipper. This is based on the Workspace class on the iosched project, which is based in the class with the same name of the Launcher app.
Download: 1.0.2 | Sources | JavaDoc
If you use Maven, you can use it as an artifact from this repository: http://mvn.egoclean.com/. Also, you would want to look this video where I show how it looks like: http://www.youtube.com/watch?v=gqIXq5x7iLs (sorry for my accent which sucks)

I think what you wanted to do is create an apparently infinite list of layouts being flinged by either the ViewFlipper or Christian's ViewFlinger. And also you want to keep reusing views / layouts inside the Flinger / Flipper. Right ?
If yes, probably the following is what you wanted to do. I've done this based on Christian's ViewFlinger,
Here you go,
First add three layouts to the ViewFlinger:
<com.egoclean.android.widget.flinger.ViewFlinger
android:id="#+id/calendarViewFlipper"
android:layout_width="fill_parent"
android:layout_height="fill_parent" >
<ScrollView
android:id="#+id/calendarViewLayout0"
android:layout_width="fill_parent"
android:layout_height="fill_parent" >
</ScrollView>
<ScrollView
android:id="#+id/calendarViewLayout1"
android:layout_width="fill_parent"
android:layout_height="fill_parent" >
</ScrollView>
<ScrollView
android:id="#+id/calendarViewLayout2"
android:layout_width="fill_parent"
android:layout_height="fill_parent" >
</ScrollView>
</com.egoclean.android.widget.flinger.ViewFlinger>
Then inside your activity, you take an array of three views so that you can access them directly through the array instead of searching every time inside the flinger,
private ViewFlinger viewFlinger;
private ViewGroup layouts[] = new ViewGroup[3];
private boolean userEvent = false;
#Override
public final void onCreateSub(Bundle savedInstanceState)
{
setContentView(R.layout.your_layout);
viewFlinger = (ViewFlinger) findViewById(R.id.calendarViewFlipper);
layouts[0] = (ViewGroup) findViewById(R.id.calendarViewLayout0);
layouts[1] = (ViewGroup) findViewById(R.id.calendarViewLayout1);
layouts[2] = (ViewGroup) findViewById(R.id.calendarViewLayout2);
viewFlinger.setOnScreenChangeListener(new ViewFlinger.OnScreenChangeListener()
{
#Override
public void onScreenChanging(View newScreen, int newScreenIndex)
{
}
#Override
public void onScreenChanged(View newScreen, int newScreenIndex)
{
if (userEvent)
{
ViewGroup tempLayout = null;
if (newScreenIndex != 1)
{
// We don't want our actions to raise events and create a cyclic event chain
userEvent = false;
if (newScreenIndex == 2) // Scrolling towards right
{
tempLayout = layouts[0];
viewFlinger.removeViewFromFront();
viewFlinger.addViewToBack(tempLayout);
layouts[0] = layouts[1];
layouts[1] = layouts[2];
layouts[2] = tempLayout;
// Any other logic comes here...
}
else if (newScreenIndex == 0) // Scrolling towards left
{
tempLayout = layouts[2];
viewFlinger.removeViewFromBack();
viewFlinger.addViewToFront(tempLayout);
layouts[2] = layouts[1];
layouts[1] = layouts[0];
layouts[0] = tempLayout;
// Any other logic comes here...
}
// We switch the screen index back to 1 since the current screen index would change back to 1
viewFlinger.setCurrentScreenNow(1, false);
userEvent = true;
// And any other logic that you'd like to put when the swapping is complete May be fill the swapped view with the correct values based on its new location etc...
View result = refreshView(tempLayout.getChildAt(0));
if (result.getParent() != tempLayout)
{
((ViewGroup) result.getParent()).removeView(result);
tempLayout.removeAllViews();
tempLayout.addView(result);
}
}
}
}
});
}
I hope this is clear to you and helps you with your problem. It is working very fine for me! Should work fine for you too.
P.S. Thanks # Christian for the ViewFlinger, it is awesome. However it lacks some good onConfigurationChanged logic, if you get time do put something in :). The rest is the best !

Finding if first image is subset of second image

I am writing Instrumentation tests for my android app. One thing I want to do is to find whether all the UI components are on the screen or not. For that I have taken the screen shot of the complete screen and then I am looking for a particular widget in that image.
This code need to be running on the device only, not on desktop.
E.g. the full screen shot (image-1) have various android components like textview, button, listview and a image. Now I have a subset of this image (image-2), suppose the image of the button.
How can I find that whether image-2 is part of image-1?

Assuming that this code is happening from within the application, it doesn't seem like image comparison is the easiest way to determine whether a view is visible.
If you are writing an external instrumentation application of some sort, and this answer doesn't apply, please let me know.
Here's what I would do to test for the presence of UI elements from within the app:
From the Android API docs on the View object: you can find a view by its ID that was set up in the XML file:
<Button
android:id="#+id/my_button"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="#string/my_button_text"/>
In the App:
Button myButton = (Button) findViewById(R.id.my_button);
Then, check the getVisibility() and getGlobalVisibleRect (Rect r, Point globalOffset), both documented on the View doc page.
Pseudocode:
int[] viewIds = {<known ids from xml>};
foreach(int viewId in viewIds) {
View v = findViewById(viewId);
if (v!=null) {
bool isVisible = (v.getVisibility()==VISIBLE) && getGlobalVisibleRect(new Rect(), new Point());
// do something with the visible/invisible info
}
}