Develop Reference

Detect when the user lifts their finger (off the screen)

We have pointerInput for detecting tap, drag and pan events, and it also provides a handy awaitPointerEventScope, the pointer being the finger, for mobile devices here. Now, we do have a awaitFirstDown() for detecting when the finger first makes contact with the screen, but I can't seem to find an upper equivalent to this method.
I have a little widget that I wish to detect taps on, but the thing is that the app is made for such a use-case that the user might be in weird positions during its use, and so I wished to have it trigger the required action on just a touch and lift of the finger. The paranoia is that the user might accidentally 'drag' their finger (even by a millimeter, android still picks it up), and I do not want that to be the case. I could implement a tap as well as a drag listener, but none of them offer a finger-lift detection, as far as I know.
What solution, if there is one as of now, is suitable for the use-case while adhering to and leveraging the declarative nature of Compose while keeping the codebase to a minimum?

Better way, and what is suggested by Android code if you are not using interoperability with existing View code is Modifier.pointerInput()
A special PointerInputModifier that provides access to the underlying
MotionEvents originally dispatched to Compose. Prefer pointerInput and
use this only for interoperation with existing code that consumes
MotionEvents. While the main intent of this Modifier is to allow
arbitrary code to access the original MotionEvent dispatched to
Compose, for completeness, analogs are provided to allow arbitrary
code to interact with the system as if it were an Android View.
val pointerModifier = Modifier
.pointerInput(Unit) {
forEachGesture {
awaitPointerEventScope {
awaitFirstDown()
// ACTION_DOWN here
do {
//This PointerEvent contains details including
// event, id, position and more
val event: PointerEvent = awaitPointerEvent()
// ACTION_MOVE loop
// Consuming event prevents other gestures or scroll to intercept
event.changes.forEach { pointerInputChange: PointerInputChange ->
pointerInputChange.consumePositionChange()
}
} while (event.changes.any { it.pressed })
// ACTION_UP is here
}
}
}
This answer explains in detail how it works, internals and key points to consider when creating your own gestures.
Also this is a gesture library you can check out for onTouchEvent counterpart and 2 for detectTransformGestures with onGestureEnd callback and returns number of pointers down or list of PointerInputChange in onGesture event. Which can be used as
Modifier.pointerMotionEvents(
onDown = {
// When down is consumed
it.consumeDownChange()
},
onMove = {
// Consuming move prevents scroll other events to not get this move event
it.consumePositionChange()
},
onUp= {}
delayAfterDownInMillis = 20
)
Edit
As of 1.2.0-beta01, partial consumes like
PointerInputChange.consemePositionChange(),
PointerInputChange.consumeDownChange(), and one for consuming all changes PointerInputChange.consumeAllChanges() are deprecated
PointerInputChange.consume()
is the only one to be used preventing other gestures/event.

pointerInteropFilter is the way to go
Item(
Modifier.pointerInteropFilter {
if (it.action == MotionEvent.ACTION_UP) {
triggerAction()
}
true // Consume touch, return false if consumption is not required here
}
)

How to orient GLTF in Android Sceneform level with gravity

I've been getting my butt kicked trying to get a vertically placed 3d model GLB format placed properly on a vertical surface.
Just to be clear, I am not referring to the difficulty of identifying vertical surface, that is a whole other problem in itself.
Removing common boilerplate of setup to minimize this post.
I am using a fragment that extends ARFragment.
class SceneFormARFragment: ArFragment() {
Then of course I have supplied the config with a few tweaks.
override fun getSessionConfiguration(session: Session?): Config {
val config = super.getSessionConfiguration(session)
// By default we are not tracking and tracking is driven by startTracking()
config.planeFindingMode = Config.PlaneFindingMode.DISABLED
config.focusMode = Config.FocusMode.AUTO
return config
}
And to start and stop my AR experience I wrote a couple of methods inside the fragment as follows.
private fun startTracking() = viewScope.launchWhenResumed {
try {
arSceneView.session?.apply {
val changedConfig = config
changedConfig.planeFindingMode = Config.PlaneFindingMode.HORIZONTAL_AND_VERTICAL
configure(changedConfig)
}
logv("startTracking")
planeDiscoveryController.show()
arSceneView.planeRenderer.isVisible = true
arSceneView.cameraStreamRenderPriority = 7
} catch (ex: Exception) {
loge("error starting ar session: ${ex.message}")
}
}
private fun stopTracking() = viewScope.launchWhenResumed {
try {
arSceneView.session?.apply {
val changedConfig = config
changedConfig.planeFindingMode = Config.PlaneFindingMode.DISABLED
configure(changedConfig)
}
logv("stopTracking")
planeDiscoveryController.hide()
arSceneView.planeRenderer.isVisible = false
arSceneView.cameraStreamRenderPriority = 0
} catch (ex: Exception) {
loge("error stopping ar session: ${ex.message}")
}
}
In case you are wondering the reason for "starting and stopping" the AR experience is to maximize the GPU cycles for other UX interactions that are heavy on this overlaid screen, so we wait to start or stop based on current live data state of other things that are happening.
Ok moving on.
Let's review the HitResult handling:
In this method I do a few things:
Load two variations of TV 3d models from the cloud (wall mount and stand mount)
I remove any active models if they have tapped a new area
Create an anchor node from the hitresult and assign it a name to remove it later
Add a TVTransformableNode to it and assign it a name to retrieve and manipulate it later
Determine the look direction of the horizontal stand mount 3D Model TV and set the worldRotation of the anchorNode to the new lookRotation. (NOTE*, I feel like the rotation should be applied to the TVNode, but it only seems to work when I apply it to the AnchorNode for whatever reason.) This camera position math also seems to help the vertical wall mount TV face outwards and anchor correctly. (I have reviewed the GLB models and I know they are properly anchored from the back on the wall model and from the bottom on the floor model)
I then limit the plane movement of the node to it's own respective plane type so that a floor model doesn't slide up to a wall and so that a wall model doesn't slide down to the floor.
That's about it. The horizontal placement works great, but the vertical placement is always randomized.
OnTapArPlane Code below:
private fun onARSurfaceTapped() {
setOnTapArPlaneListener { hitResult, plane, _ ->
var isHorizontal = false
val renderable = when (plane.type) {
Plane.Type.HORIZONTAL_UPWARD_FACING -> {
isHorizontal = true
standmountTVRenderable
}
Plane.Type.VERTICAL -> wallmountTVRenderable
else -> {
activity?.toast("Do you want it to fall on your head really?")
return#setOnTapArPlaneListener
}
}
lastSelectedPlaneOrientation = plane.type
removeActive3DTVModel()
val anchorNode = AnchorNode(hitResult.createAnchor())
anchorNode.name = TV_ANCHOR_NAME
anchorNode.setParent(arSceneView.scene)
val tvNode = TransformableNode(this.transformationSystem)
tvNode.scaleController.isEnabled = false
tvNode.setParent(anchorNode)
tvNode.name = TV_NODE_NAME
tvNode.select()
// Set orientation towards camera
// Ref: https://github.com/google-ar/sceneform-android-sdk/issues/379
val cameraPosition = arSceneView.scene.camera.worldPosition
val tvPosition = anchorNode.worldPosition
val direction = Vector3.subtract(cameraPosition, tvPosition)
if(isHorizontal) {
tvNode.translationController.allowedPlaneTypes.clear()
tvNode.translationController.allowedPlaneTypes.add(Plane.Type.HORIZONTAL_UPWARD_FACING)
} else {
tvNode.translationController.allowedPlaneTypes.clear()
tvNode.translationController.allowedPlaneTypes.add(Plane.Type.VERTICAL)
}
val lookRotation = Quaternion.lookRotation(direction, Vector3.up())
anchorNode.worldRotation = lookRotation
tvNode.renderable = renderable
addVideoTo3DModel(renderable)
}
}
Ignore the addvideoTo3dModel call, as that works fine, and I commented it
out just to ensure it doesn't play a role.
Things I've tried.
Extracting Translation without Rotation like described here interestingly enough, it does cause the TV to appear level with the floor each time, but then the TV is always mounted as if the anchor is at the base instead of the center back. So it's bad.
I've tried reviewing various posts and translating Unity or ARCore stuff directly into Sceneform, but failed to get anything to affect the outcome. example
I've tried creating the anchor from the plane and the pose as indicated in this answer with no luck
I've reviewed this link but never found anything useful
I've tracked this issue and tried solutions recommended by people in the thread, but no luck
The last thing I tried, and this is a bit embarrassing lol. I opened all 256 tagged with "SceneForm" in Stack Overflow and reviewed EVERY SINGLE one of them for anything that would help.
So I've exhausted the internet. All I have left is to ask the community and of course send help to SceneForm team at Android which I'm also going to do.
My best guess is that I need to do the Quaternion.axisRotation(Vector3, Float), but everything I have guessed at or trialed and errored has not worked. I assume I need to set the localRotation using worldPostion values for xyz of the phone maybe to help identify gravity. I really just don't know anymore lol.
I know Sceneform is pretty new and the documentation is HORRIBLE and may as well not exist with the lack of content or doc headers on it. The developers must really not want people to use it yet I'm guessing :(.
Last thing I'll say, is everything is working perfectly in my current implementation with the exception of the rotated vertical placement. Just to avoid rabbit trails on this discussion, I'm not having any other issues.
Oh and one last clue that I've noticed.
The TV almost seems to pivot around the center of the vertical plane, based on where I tap, the bottom almost seems to point towards the arbitrary center of the plane, if that helps anyone figure it out.
Oh and yes, I know my textures are missing from the GLBs, I packaged them incorrectly and intend to fix it later.
Screenshots attached.

Well I finally got it. Took awhile and some serious trial and error of rotating every node, axis, angle, and rotation before I finally got it to place nicely. So I'll share my results in case anyone else needs this as well.
End Result looked like:
Of course it is mildly subjective to how you held the phone and it's understanding of the surroundings, but it's always pretty darn close to level now without fail in both landscape and portrait testing that I have done.
So here's what I've learned.
Setting the worldRotation on the anchorNode will help keep the 3DModel facing towards the cameraview using a little subtraction.
val cameraPosition = arSceneView.scene.camera.worldPosition
val tvPosition = anchorNode.worldPosition
val direction = Vector3.subtract(cameraPosition, tvPosition)
val lookRotation = Quaternion.lookRotation(direction, Vector3.up())
anchorNode.worldRotation = lookRotation
However, this did not fix the orientation issue on the vertical placement. I found that if i did an X Rotation of 90 degress on the look rotation it worked everytime. It may differ based on your 3d model, but my anchor is center middle back, so I'm not sure how it determine which way was up. However, I noticed whenever I would set a worldRotation on the tvNode it would place the TV level, but would be leaning forward 90 degress. So after playing with the various rotations, I finally got the answer.
val tvRotation = Quaternion.axisAngle(Vector3(1f, 0f, 0f), 90f)
tvNode.worldRotation = tvRotation
That fixed up my problem. So The end Result of the onSurfaceTap and placement was this:
setOnTapArPlaneListener { hitResult, plane, _ ->
var isHorizontal = false
val renderable = when (plane.type) {
Plane.Type.HORIZONTAL_UPWARD_FACING -> {
isHorizontal = true
standmountTVRenderable
}
Plane.Type.VERTICAL -> wallmountTVRenderable
else -> {
activity?.toast("Do you want it to fall on your head really?")
return#setOnTapArPlaneListener
}
}
lastSelectedPlaneOrientation = plane.type
removeActive3DTVModel()
val anchorNode = AnchorNode(hitResult.createAnchor())
anchorNode.name = TV_ANCHOR_NAME
anchorNode.setParent(arSceneView.scene)
val tvNode = TransformableNode(this.transformationSystem)
tvNode.scaleController.isEnabled = false //disable scaling
tvNode.setParent(anchorNode)
tvNode.name = TV_NODE_NAME
tvNode.select()
val cameraPosition = arSceneView.scene.camera.worldPosition
val tvPosition = anchorNode.worldPosition
val direction = Vector3.subtract(cameraPosition, tvPosition)
//restrict moving node to active surface orientation
if (isHorizontal) {
tvNode.translationController.allowedPlaneTypes.clear()
tvNode.translationController.allowedPlaneTypes.add(Plane.Type.HORIZONTAL_UPWARD_FACING)
} else {
tvNode.translationController.allowedPlaneTypes.clear()
tvNode.translationController.allowedPlaneTypes.add(Plane.Type.VERTICAL)
//x 90 degree rotation to flat mount TV vertical with gravity
val tvRotation = Quaternion.axisAngle(Vector3(1f, 0f, 0f), 90f)
tvNode.worldRotation = tvRotation
}
//set anchor nodes world rotation to face the camera view and up
val lookRotation = Quaternion.lookRotation(direction, Vector3.up())
anchorNode.worldRotation = lookRotation
tvNode.renderable = renderable
viewModel.updateStateTo(AriaMainViewModel.ARFlowState.REPOSITIONING)
}
This has been tested pretty thoroughly without issues so far in portrait and landscape. I still have other issues with Sceneform, such as the dots only showing up about half the time even when there is a valid surface, and of course vertical detection on a mono color wall is not possible with the current SDK without a picture on the wall or something to distinguish the wall.
Also performing screenshots is not good as it doesn't include the 3D Model so that required custom Pixel Copy work and my screenshots are a bit slow, but at least they work, no thanks to the SDK.
So they have a long ways to go and it's frustrating to blaze the trail with their product and lack of documentation and definitely lack of responsiveness to customer serivce as well as GitHub logged issues, but hey at least I got it, and I hope this helps someone else.
Happy Coding!

Sceneform Collisions With Camera

I'm stretching my very limited ARCore knowledge.
My question is similar (but different) to this question
I want to work out if my device camera node intersects/overlaps with my other nodes, but I've not been having any luck so far
I'm trying something like this (the camera is another node):
scene.setOnUpdateListener(frameTime -> {
Node x = scene.overlapTest(scene.getCamera());
if (x != null) {
Log.i(TAG, "setUpArComponents: CAMERA HIT DETECTED at: " + x.getName());
logNodeStatus(x);
}
});
Firstly, does this make sense?
I can detect all node collisions in my scene using:
for (Node node : nodes) {
...
ArrayList<Node> results = scene.overlapTestAll(node);
...
}
Assuming that there isn't a renderable for the Camera node (so no default collision shape), I tried to set my own collision shape, but this was actually catching all the tap events I was trying to perform, so I figured I must be doing this wrong.
I'm thinking about things like fixing a deactivated node in front of the camera.
I may be asking for too much of ARCore, but has anyone found a way to detect a collision between the "user" (i.e. camera node) and another node? Or should I be doing this "collision detection" via indoor positioning instead?
Thanks in advance :)
UPDATE: it's really hacky and performance-heavy, but you can actually compare the camera's and node's world space positions from within onUpdate inside a node, you'll probably have to manage some tolerance and other things to smooth out interactions.

One idea to do the same thing is to use a raycast to hit the objects and if they are close do something. You could use something like this in the onUpdateListener:
Camera camera = arSceneView.getScene().getCamera();
Ray ray = new Ray(camera.getWorldPosition(), camera.getForward());
HitTestResult result = arSceneView.getScene().hitTest(ray);
if (result.getNode() != null && result.getDistance() <= SOME_THRESHOLD) {
// Hit something
doSomething (result.getNode());
}

Google Maps Android, detect user interation in a location aware map

I've an Android Application that displays a map.
The map is by default centered on the user position following its movements (the center is updated according to the position updates).
However I want the user to be able to use gestures to navigate throw the map. When the user starts the navigation I want the "following" to stop, and a button is displayed so that it can start again.
How can I know when the user has moved the map center?
On the GoogleMap.OnCameraChangeListener, I don't know if the change is due to a location changed or a user interaction.
I've a a kind of working solution using the OnCameraChangeListener, but its a bit "dirty" and I don't find it very nice:
map.setOnCameraChangeListener(new GoogleMap.OnCameraChangeListener() {
#Override
public void onCameraChange(CameraPosition position) {
// gpsPosition contains the last position obtained
if(isFollowing && gpsPosition!=null && (Math.abs(position.target.latitude - gpsPosition.latitude)>0.000001 || Math.abs(position.target.longitude - gpsPosition.longitude)>0.000001) ){
isFollowing = false;
map.getUiSettings().setMyLocationButtonEnabled(true);
}
}
});
Is there a nice Solution to the problem?

If you want to detect user interacting with the map OnCameraChangeListener is surely too late to do that. It may be called seconds after user started interacting with the map.
What you need to do, is add 2 other listeners:
OnMyLocationButtonClickListener - to know when to start tracking (set your isFollowing to true)
OnMyLocationChangeListener - when your isFollowing is true, call GoogleMap.animateCamera to the new position
So the only thing left is when you set isFollowing to false:
Create an invisible overlay in your layout (like here, but with normal View instead of custom) and assign this overlay (the View) an OnTouchListener (ok, it is 3rd, I lied that you need only 2).
In this listener always return false, but also set isFollowing to false. This when when user starts interacting with the map and you should stop automated camera movements.
I also see you are showing and hiding my location button, so do this where you change the value of isFollowing. Btw.: good idea to do that. I'll try it too.

I have never tried this, but here are some ideas off the top of my (balding) head:
Option #1: Handle the location change updates yourself, by recentering the map yourself when the location change comes in. If a camera-change event occurs that does not appear to be tied to the location change (e.g., 5+ milliseconds later), that was presumably a user modifying the map via gestures.
Option #2: Subclass the Maps V2 MapView and override touch-related methods like onTouchEvent(). In addition to chaining to the superclass, you would know that a camera change that happens very soon from now probably is from the user modifying the map via gestures.
Option #3: Do both of the above. That way, all changes to the map should touch your code, which should increase the reliability of your determining the source of the camera change.
BTW, I filed a feature request for a better solution.

The simpliest way should be saving the center Longitude and Latitude of your Map.
Now create an EventListener and check it in an if-statement:
if(oldLongLat != newLongLat){
//Something has moved
}else {
//Nothing has moved
}

Dragging Camera based on Touch (Unity)

In my game, I want the camera to move based on the user swiping their finger. My issue is that when someone removes their finger from the screen, and places it down again, the camera jumps to a new position... I am guessing it has to do with the coordinates, I say this because if I click somewhere far from where I removed my finger from the screen, the camera jumps. Here is the code:
var speed : int = 1;
var lastPoint : Vector2; //make this private if you want to.
function Update()
{
var offset : float; //offset of the touch from last frame
if(Input.touches.Length 0)//make sure we have a touch in the first place
{
var evt : Touch = Input.touches[0]; //setting up touch events so we can get fancy.
if(evt.phase == TouchPhase.Began) //this is the first frame the screen has been touched for, simply save the point.
{
lastPoint == evt.position;
}
else if(evt.phase == TouchPhase.Moved
{
offset = evt.position.x - lastPoint.x ;//take the difference
//I'm going to use Transform.Rotate because it's handy dandy
transform.Rotate(0,offset * speed,0);
//save the new "lastPoint"
lastPoint = evt.position;
}
else if(evt.phase == TouchPhase.Ended)
{
//If you want the object to drift after you spin it you can make a function to go here.
//To do this, take the speed of the rotation and continue to rotate all while subtracting off of the speed.
//I would use the Transform.Rotate function on this too.
//If you need me to I could write this function too.
}
}
}
What is the solution to have the camera resume, and not jump once someone puts their finger down again?
I am also willing to redo it, if there is a better solution/more efficient method of solving the issue..
Thanks a lot!

The supplied code seems fine, all you need to do is when the user is not pressing, you update the lastPoint every frame. Or if you know when the user start to press, just set the lastPoint to the current pos before calculating the offset. This way the offset is zero the first frame the user touch the screen.