How does Appium/UiAutomator actually see composables? - android

I have been rewriting my app to Jetpack Compose, and I already have existing Appium tests. After rewriting (using testTag modifiers to replace what used to be resource IDs) all of my tests work fine. So I am not asking for help with getting things working, but I am trying to understand how it works. When I look at the App Source tab in Appium, I see this:
But this screen is just an Activity with a ComposeView created with the setContent extension. How does Appium see a Column with a verticalScroll modifier as a ScrollView, and a TextField as an EditText, etc? I know Compose is not just creating Views under the hood, so where is this mapping being done? I tried searching the UiAutomator and Appium source code and could not find the right code.
Anyone out there have any expertise here to help me understand?

Appium uses UiAutomatorViewer, which you can find in Android\Sdk\tools\bin.
We can look at the source to find ScreenshotAction, which uses DumpCommand, which dumps the view as XML using AccessibilityNodeInfoDumper.
Every composable has AccessibilityNodeInfo, where the className property is the same as the XML equivalent class name.
You can find at least some of the "mapping" from compose to xml class names in AndroidComposeViewAccessibilityDelegateCompat.
If the composable has vertical scrolling, the className will be android.widget.ScrollView, like this:
val yScrollState = semanticsNode.config.getOrNull(SemanticsProperties.VerticalScrollAxisRange)
val scrollAction = semanticsNode.config.getOrNull(SemanticsActions.ScrollBy)
if (yScrollState != null && scrollAction != null) {
info.className = "android.widget.ScrollView"
}
A lot of the mapping is also based on Role:
semanticsNode.config.getOrNull(SemanticsProperties.Role)?.let {
when (it) {
Role.Button -> info.className = "android.widget.Button"
Role.Checkbox -> info.className = "android.widget.CheckBox"
Role.Switch -> info.className = "android.widget.Switch"
Role.RadioButton -> info.className = "android.widget.RadioButton"
Role.Tab -> info.roleDescription = AccessibilityRoleDescriptions.Tab
Role.Image -> info.className = "android.widget.ImageView"
}
}
That's one of the reasons it's important to set a Role for you custom composables. Like:
#Composable
fun CustomButton() {
Box(
modifier = Modifier.semantics {
role = Role.Button
}
)
}

Related

Jetpack Compose: Mimicking spinner.setSelection() inside of a DropDownMenu

The use case is that you have 10s or 100s of items inside of a dropdown menu, the dropdown options have some ordering - as with number values or alphabetical listing of words - and selections are made in succession.
When the user reopens the menu, you'd like for it to open in the same region as their last selection, so that for instance you don't jump from "car" to "apple" but rather from "car" to "cat". Or if they just opted to view order number 358, they can quickly view order number 359.
Using views, you could create a Spinner and put all of your items in an ArrayAdapter and then call spinner.setSelection() to scroll directly to the index you want.
DropdownMenu doesn't have anything like HorizontalPager's scrollToPage(). So what solutions might exist to achieve this?
So far, I've tried adding verticalScroll() to the DropdownMenu's modifier and trying to do arithmetic with the scrollState. But it crashes at runtime with an error saying the component has infinite height, the same error you get if you try to nest scrollable components like a LazyColumn inside of a Column with verticalScroll.
It's a known issue.
DropdownMenu has its own vertical scroll modifier inside, and there is no API to work with it.
Until this problem is fixed by providing a suitable API, the only workaround I can think of is to create your own view - you can take the source code of DropdownMenu as reference.
I'll post a more detailed answer here because I don't want to mislead anyone with my comment above.
If you're in Android Studio, click the three dots on the mouse-hover quick documentation box and select "edit source" to open the source for DropdownMenu in AndroidMenu.android.kt. Then observe that it uses a composable called DropdownMenuItemContent. Edit source again and you're in Menu.kt.
You'll see this:
#Composable
internal fun DropdownMenuContent(
...
...
...
{
Column(
modifier = modifier
.padding(vertical = DropdownMenuVerticalPadding)
.width(IntrinsicSize.Max)
.verticalScroll(rememberScrollState()),//<-We want this
content = content
)
}
So in your custom composable just replace that rememberScrollState() with your favorite variable name for a ScrollState.
And then chain that reference all the way back up to your original view.
Getting Access to the ScrollState
#Composable
fun MyCustomDropdownMenu(
expanded:Boolean,
scrollStateProvidedByTopParent:ScrollState,
...
...
)
{...}
#Composable
fun MyCustomDropdownMenuContent(
scrollStateProvidedByTopParent:ScrollState,
...
...
)
{...}
//And now for your actual content
#Composable
fun TopParent(){
val scrollStateProvidedByTopParent=rememberScrollState()
val spinnerExpanded by remember{mutableStateOf(false)}
...
...
Box{
Row(modifier=Modifier.clickable(onClick={spinnerExpanded=!spinnerExpanded}))//<-More about this line in the sequel
{
Text("Options")
Icon(imageVector = Icons.Filled.ArrowDropDown, contentDescription = "")
MyCustomDropdownMenu(
expanded = spinnerExpanded,
scrollStateProvidedByTopParent=scrollStateProvidedByTopParent,
onDismissRequest = { spinnerExpanded = false })
{//your spinner content}
}
}
}
The above only specifies how to access the ScrollState of the DropdownMenu. But once you have the ScrollState, you'll have to do some arithmetic to get the scroll position right when it opens. Here's one way that seems alright.
Calculating the scroll distance
Even after setting the contents of the menu items explicitly, the distance was never quite right if I relied on those values. So I used an onTextLayout callback inside the Text of my menu items in order to get the true Text height at the time of rendering. Then I use that value for the arithmetic. It looks like this:
#Composable
fun TopParent(){
val scrollStateProvidedByTopParent=rememberScrollState()
val spinnerExpanded by remember{mutableStateOf(false)}
val chosenText:String by remember{mutableStateOf(myListOfSpinnerOptions[0])
val height by remember{mutableStateOf(0)}
val heightHasBeenChecked by remember{mutableStateOf(false)}
val coroutineScope=rememberCoroutineScope()
...
...
Box{
Row(modifier=Modifier.clickable(onClick={spinnerExpanded=!spinnerExpanded
coroutineScope.launch{scrollStateProvidedByTopParent.scrollTo(height*myListOfSpinnerOptions.indexOf[chosenText])}}))//<-This gets some arithmetic for scrolling distance
{
Text("Options")
Icon(imageVector = Icons.Filled.ArrowDropDown, contentDescription = "")
MyCustomDropdownMenu(
expanded = spinnerExpanded,
scrollStateProvidedByTopParent=scrollStateProvidedByTopParent,
onDismissRequest = { spinnerExpanded = false }) {
myListOfSpinnerOptions.forEach{option->
DropdownMenuItem(onClick={
chosenText=option
spinnerExpanded=false
}){
Text(option,onTextLayout={layoutResult->
if (!heightHasBeenChecked){
height=layoutResults.size.height
heightHasBeenChecked=true
}
}
)
}
}
}
}
}

Jetpack Compose State Hoisting, Previews, and ViewModels best practices

So it seems like the recommended thing in Jetpack Compose is to hoist state out of your composables, to make them stateless, reusable, and testable, and allow using them in previews easily.
So instead of having something like
#Composable
fun MyInputField() {
var text by remember { mutableStateOf("") }
TextField(value = text, onValueChange = { text = it })
}
You'd hoist the state, like this
#Composable
fun MyInputField(text: String, onTextChange: (String) -> Unit) {
TextField(value = text, onValueChange = onTextChange)
}
This is fine, however what of some more complex uses?
Let's pretend I have a screen represented by a composable, with multiple interactions between the View and the ViewModel. This screen is split into multiple inner composable (think for instance one for a header, one for the body, which in turn is split into several smaller composables)
You can't create a ViewModel (with viewModel() at least, you can instantiate one manually) inside a composable and use this composable in a Preview (previews don't support creating viewmodel like this)
Using a ViewModel inside the inner composables would make them stateful, wouldn't it ?
So the "cleanest" solution I see, would be to instantiate my viewmodel only at the highest composable level, and then pass to the children composables only vals representing the state, and callbacks to the ViewModel functions.
But that's wild, I'm not passing down all my ViewModel state and functions through individual parameters to all composables needing them.
Grouping them in a data class for example could be a solution
data class UiState(
val textInput: String,
val numberPicked: Int,
……
and maybe create another one for callbacks ?
But that's still creating a whole new class just to mimic what the viewmodel already has.
I don't actually see what the best way of doing this could be, and I find nothing about that anywhere
A good way to manage complex states is to encapsulate required complex behavior into a class and use remember function while having stateless widgets as most as you can and change any properties of state whenever it's required.
SearchTextField is a component that uses only state hoisting, SearchBar has back arrow and SearchTextField and also itself is a stateless composable. Communication between these two and parent of Searchbar is handled via callback functions only which makes both SearchTextField re-suable and easy to preview with a default state in preview. HomeScreen contains this state and where you manage changes.
Full implementation is posted here.
#Composable
fun <R, S> rememberSearchState(
query: TextFieldValue = TextFieldValue(""),
focused: Boolean = false,
searching: Boolean = false,
suggestions: List<S> = emptyList(),
searchResults: List<R> = emptyList()
): SearchState<R, S> {
return remember {
SearchState(
query = query,
focused = focused,
searching = searching,
suggestions = suggestions,
searchResults = searchResults
)
}
}
remember function to keep state for this only to be evaluated during the composition.
class SearchState<R, S>(
query: TextFieldValue,
focused: Boolean,
searching: Boolean,
suggestions: List<S>,
searchResults: List<R>
) {
var query by mutableStateOf(query)
var focused by mutableStateOf(focused)
var searching by mutableStateOf(searching)
var suggestions by mutableStateOf(suggestions)
var searchResults by mutableStateOf(searchResults)
val searchDisplay: SearchDisplay
get() = when {
!focused && query.text.isEmpty() -> SearchDisplay.InitialResults
focused && query.text.isEmpty() -> SearchDisplay.Suggestions
searchResults.isEmpty() -> SearchDisplay.NoResults
else -> SearchDisplay.Results
}
}
And change state in any part of UI by passing state to other composable or by ViewModel as
fun HomeScreen(
modifier: Modifier = Modifier,
viewModel: HomeViewModel,
navigateToTutorial: (String) -> Unit,
state: SearchState<TutorialSectionModel, SuggestionModel> = rememberSearchState()
) {
Column(
modifier = modifier.fillMaxSize()
) {
SearchBar(
query = state.query,
onQueryChange = { state.query = it },
onSearchFocusChange = { state.focused = it },
onClearQuery = { state.query = TextFieldValue("") },
onBack = { state.query = TextFieldValue("") },
searching = state.searching,
focused = state.focused,
modifier = modifier
)
LaunchedEffect(state.query.text) {
state.searching = true
delay(100)
state.searchResults = viewModel.getTutorials(state.query.text)
state.searching = false
}
when (state.searchDisplay) {
SearchDisplay.InitialResults -> {
}
SearchDisplay.NoResults -> {
}
SearchDisplay.Suggestions -> {
}
SearchDisplay.Results -> {
}
}
}
}
Jetmagic is an open source framework that deals exactly with this issue while also solving other major issues that Google neglected when developing Compose. Concerning your request, you don't pass in viewmodels at all as parameters. Jetmagic follows the "hoisted state" pattern, but it manages the viewmodels for you and keeps them associated with your composables. It treats composables as resources in a way that is similar to how the older view system treats xml layouts. Instead of directly calling a composable function, you ask Jetmagic's framework to provide you with an "instance" of the composable that best matches the device's configuration. Keep in mind, under the older xml-based system, you could effectively have multiple layouts for the same screen (such as one for portrait mode and another for landscape mode). Jetmagic picks the correct one for you. When it does this, it provides you with an object that it uses to manage the state of the composable and it's related viewmodel.
You can easily access the viewmodel anywhere within your screen's hierarchy without the need to pass the viewmodel down the hierarchy as parameters. This is done in part using CompositionLocalProvider.
Jetmagic is designed to handle the top-level composables that make up your screen. Within your composable hierarchy, you still call composables as you normally do but using state hoisting where it makes sense.
The best thing is to download Jetmagic and try it out. It has a great demo that illustrates the solution you are looking for:
https://github.com/JohannBlake/Jetmagic

Why need the author to add the keyword remember in this #Composable?

The Code A is from the project ThemingCodelab, you can see full code here.
I think that the keyword remember is not necessary in Code A.
I have tested the Code B, it seems that I can get the same result just like Code A.
Why need the author to add the keyword remember in this #Composable ?
Code A
#Composable
fun Home() {
val featured = remember { PostRepo.getFeaturedPost() }
val posts = remember { PostRepo.getPosts() }
MaterialTheme {
Scaffold(
topBar = { AppBar() }
) { innerPadding ->
LazyColumn(contentPadding = innerPadding) {
item {
Header(stringResource(R.string.top))
}
item {
FeaturedPost(
post = featured,
modifier = Modifier.padding(16.dp)
)
}
item {
Header(stringResource(R.string.popular))
}
items(posts) { post ->
PostItem(post = post)
Divider(startIndent = 72.dp)
}
}
}
}
}
Code B
#Composable
fun Home() {
val featured =PostRepo.getFeaturedPost()
val posts = PostRepo.getPosts()
...//It's the same with the above code
}
You need to use remember to prevent recomputation during recomposition.
Your example works without remember because this view will not recompose while you scroll through it.
But if you use animations, add state variables or use a view model, your view can be recomposed many times(when animating up to once a frame), in which case getting data from the repository will be repeated many times, so you need to use remember to save the result of the computation between recompositions.
So always use remember inside a view builder if the calculations are at least a little heavy, even if right now it looks like the view is not gonna be recomposed.
You can read more about the state in compose in documentation, including this youtube video, which explains the basic principles.

How to use android accessibility for global texts?

I have some metric units like "kg" and "ml". Talkback reads the letters, and I implemented the contentDescription on every case:
when (someText) {
"kg" -> someText.contentDescription = "kilograms"
"ml" -> someText.contentDescription = "milliliters"
}
This works, but I need to make this global: every kg or ml in the app, needs to be read by Talkback as "kilograms" or "milliliters".
Which is the best approach for this?
It cannot be set for Talkback on an app level. You could create a utils function and use it throughout the codebase.
Something like:
fun TextView.setUnit(unit: String) {
text = unit
when (unit) {
"kg" -> contentDescription = "kilograms"
"ml" -> contentDescription = "milliliters"
}
}
And then to use it:
textView.setUnit("kg")
Another option could be to make a custom view for rendering units that sets content description.

Jetpack Compose Crossfade broken in Alpha

My crossfade animations are no longer working since the release of Compose Alpha and I would really appreciate some help getting them working again. I am fairly new to Android/Compose. I understand that Crossfade is looking for a state change in its targetState to trigger the crossfade animation, but I am confused how to incorporate this. I am trying to wrap certain composables in the Crossfade animation.
Here are the official docs and helpful playground example, but I still cannot get it to work since the release of Alpha
https://developer.android.com/reference/kotlin/androidx/compose/animation/package-summary#crossfade
https://foso.github.io/Jetpack-Compose-Playground/animation/crossfade/
Here is my code, in this instance I was hoping to use the String current route itself as the targetState as a mutableStateOf object. I'm willing to use whatever will work though.
#Composable
fun ExampleComposable() {
val navBackStackEntry by navController.currentBackStackEntryAsState()
val currentRoute: String? = navBackStackEntry?.arguments?.getString(KEY_ROUTE)
val exampleRouteTargetState = remember { mutableStateOf(currentRoute)}
Scaffold(
...
NavHost(navController, startDestination = "Courses") {
composable("Route") {
Crossfade(targetState = exampleRouteTargetState, animationSpec = tween(2000)) {
ExampleComposable1()
}
}
composable("Other Route")
ExampleComposable2()
}
)
...
}
Shouldn't navigation trigger a state change of the "exampleRouteTargetState" variable and then trigger crossfade? I could also wrap the composable elsewhere if you think wrapping it inside the NavHost may create an issue. Thanks so much for the help!!
Lately Google Accompanist has added a library which provides Compose Animation support for Jetpack Navigation Compose.. Do check it out. 👍🏻
https://github.com/google/accompanist/tree/main/navigation-animation
Still haven't gotten Crossfade working again, but I was able to implement some transitions inside NavHost. Hope this helps someone. Here are the docs if you want to fine tune these high level animations:
https://developer.android.com/jetpack/compose/animation#animatedvisibility
#ExperimentalAnimationApi
#Composable
fun ExampleAnimation(content: #Composable () -> Unit) {
AnimatedVisibility(
visible = true,
enter = fadeIn(initialAlpha = 0.3f),
exit = fadeOut(),
content = content,
initiallyVisible = false
)
}
And then simply wrap your NavHost composable declarations with your animation like so
NavHost(navController, startDestination = "A Route") {
composable(Screen.YourObject.Route) {
ExampleAnimation {
YourComposable()
}
}

Categories

Resources