Develop Reference

AR Android Sceneform SDK display model only on the floor

I'm using Sceneform SDK for Android version
implementation 'com.google.ar.sceneform.ux:sceneform-ux:1.15.0'
I need my 3D model to be displayed only on the floor. For example I have a 3D cage (simple transparent cuboid) and I need to place this 3D model over real object. For now if my real object has enough big surface the model will be placed at the top of it instead of go over it and I need to avoid behavior.
Here is some code here.
Logic to init ArFragment and display model at the center of the camera. I'm making a HitTest at the center of the my device camera any time when frame is changed.
private fun initArFragment() {
arFragment.arSceneView.scene.addOnUpdateListener {
arFragment.arSceneView?.let { sceneView ->
sceneView.arFrame?.let { frame ->
if (frame.camera.trackingState == TrackingState.TRACKING) {
val hitTest =
frame.hitTest(sceneView.width / 2f, sceneView.height / 2f)
val hitTestIterator = hitTest.iterator()
if (hitTestIterator.hasNext()) {
val hitResult = hitTestIterator.next()
val anchor = hitResult.createAnchor()
if (anchorNode == null) {
anchorNode = AnchorNode()
anchorNode?.setParent(sceneView.scene)
transformableNode =
DragTransformableNode(arFragment.transformationSystem)
transformableNode?.setParent(anchorNode)
boxNode = createBoxNode(.4f, .6f, .4f) // Creating cuboid
boxNode?.setParent(transformableNode)
}
anchorNode?.anchor?.detach()
anchorNode?.anchor = anchor
}
}
}
}
}
}
I think it's expected behavior because HitTest hits on the surface of real object as well. But Don't know how to avoid this behavior.
Is there a way to ignore real objects and place 3D model always at the floor?
UPDATE
I tried to follow #Mick suggestions. I'm trying to group all HitTestResult. When HitTest is done I get list a of all HitResults for all visible planes. I'm grouping them by its rounded Y axis.
Example
{1.35 -> [Y is 1.36776767, Y is 1.35434343, Y is 1.35999999, Y is 1.37723278]}
{1.40 -> [Y is 1.4121212, Y is 1.403232323, Y is 1.44454545, Y is 1.40001011]}
Then for X and Z anchor points I'm using FIRST HitResult with from array sorted keys from MIN to MAX key in the example it's 1.35.
For Y anchor point I'm getting array of a MIN group elements and get it's average value.
val hitResultList = hitTestIterator.asSequence().toList().groupBy { round(it.hitPose.ty() * 20) / 20 }.minBy { it.key }?.value
val hitResult = hitResultList?.first()!!
val averageValueOfY = hitResultList?.map { it.hitPose.ty() }?.average()
createModel(hitResult, averageValueOfY)
Method to create Model
private fun createModel(newHitResult: HitResult, averageValueOfY: Double) {
try {
val newAnchorPose = newHitResult.createAnchor().pose
anchorNode?.anchor?.detach()
anchorNode?.anchor = arFragment.arSceneView.session?.createAnchor(Pose(floatArrayOf(newAnchorPose.tx(),
averageValueOfY.toFloat(), newAnchorPose.tz()), floatArrayOf(0f, 0f, 0f, 0f)))
isArBagModelRendered = true
transformableNode?.select()
} catch (exception: Exception) {
Timber.d(exception)
}
}
This code update helped to get the behaviour which I tried to achieve, but I noticed that sometimes my Y anchor point is underground it looks like MIN plane was detected under the floor :( and I don't know how to fix this issue for now.

Actually I think you possibly have two separate problems you will face for your use case:
object being placed on the top surface as you have highlighted in the question
Occlusion, or not showing the part of the model that should be hidden behind the object, when the model is actually put in the correct place.
A simple solution to the first problem so you can check the second, or maybe more accurately a workaround, might be to simply get the user to place the object in front of the real object, i.e. the case in your example above, and then move it back until it is exactly where they want it.
If you leave the plane highlighting on, i.e. the grid lines which show where a plane is detected, it may be more intuitive for a user to 'hit' the floor also rather than the top of the object.
This would allow you test quickly if the occlusion issue is actually the more serious issue, before you go too much further.
A more complex solution would be to iterate through the planes and experiment with comparing the 'pose' at the centre of each plane to see if you can find reliable way to decide which is the floor - the method is part of the Plane class:
public Pose getCenterPose ()
Returns the pose of the center of the detected plane. The pose's transformed +Y axis will be point normal out of the plane, with the +X and +Z axes orienting the extents of the bounding rectangle.
There are also methods to get the size of the width or depth of the plane if you were sure the floor will always be the biggest plane:
public float getExtentX ()
Returns the length of this plane's bounding rectangle measured along the local X-axis of the coordinate space centered on the plane.
public float getExtentZ ()
Returns the length of this plane's bounding rectangle measured along the local Z-axis of the coordinate frame centered on the plane.
Unfortunately, I don't think there is any existing handy help function like 'get lowest plane', or get 'largest plane' etc.
Note on the occlusion issue, there are frameworks and libraries that do provide some forms of software based occlusion, i.e. without requiring the device to have extra depth sensors, so it may be worth exploring these a little also.

Why does the child of anchorNode move parallel with the camera in ARCore-Location?

I implement an app which place AR labels at specific gps location with ARCore-Location(https://github.com/appoly/ARCore-Location).The logic is that create anchor Node based on the gps location and the location of camera:
`
double rotationRadian = Math.toRadians(rotation); //rotation is the angle between the bearing of the specific gps location and camera's location and the azimuth of the camera.
float zRotated = (float) (z * Math.cos(rotationRadian));
float xRotated = (float) -(z * Math.sin(rotationRadian));
float y = frame.getCamera().getDisplayOrientedPose().ty() + (float) heightAdjustment;
Pose translation = Pose.makeTranslation(xRotated, y, zRotated);
Anchor newAnchor = mSession.createAnchor(
frame.getCamera()
.getDisplayOrientedPose()
.compose(translation)
.extractTranslation()
);
marker.anchorNode = new LocationNode(newAnchor, marker, this);
marker.anchorNode.setScalingMode(LocationMarker.ScalingMode.GRADUAL_FIXED_SIZE);
marker.anchorNode.setParent(mArSceneView.getScene());
marker.anchorNode.addChild(mLocationMarkers.get(i).node); //add the node presenting the AR labels.
marker.node.setLocalPosition(Vector3.zero());
`
then in the Update() of the anchorNode,scale and rotate the node presenting the ar label:
`
final Vector3 cameraPosition = getScene().getCamera().getWorldPosition();
Vector3 direction = Vector3.subtract(cameraPosition, n.getWorldPosition());
n.setWorldPosition(new Vector3(n.getWorldPosition().x, getHeight(), n.getWorldPosition().z));
Quaternion lookRotation = Quaternion.lookRotation(direction, Vector3.up());
n.setWorldRotation(lookRotation);
n.setWorldScale(new Vector3(scale, scale, scale));
`
The code set an interval(which defaults as 5 seconds) after which the session will refresh the anchorNode(detach the old and create the new).
I log out the world positions of anchorNode and node, and find that during the interval,these values do not change,but when i move the phone(do not change the pose and just hold the phone and walk straight on the path) the AR labels move parallel with the phone(it seems that the labels walk as well but are not sticked on the screen).
While the world position of the anchorNode change(maybe after the interval),it is obvious that the labels pop on the screen(not move parallel with the phone).
I want to achieve the performance that when I hold the phone and walk,the ar label does not move parallel with the camera but stays where it should be and looks like it is right here in the real world(it does not change the position relative to the real object in the real world).
I guess the reason why the ar label moves parallel with the camera when the world positions of the anchorNode do not change is that the anchor is created based on the camera,and when the AR session understands the surrounding environment,the node moves parallel with the camera to keep the relationship between the anchor and the camera.But when i try to create the anchorNode at an absolute world position,it has the same situation.
Then I find a description like this:
Adding ARAnchor objects to a session is effectively about "labeling" a point in real-world space so that you can refer to it later. The point (1,1,1) (for example) is always the point (1,1,1) — you can't move it someplace else because then it's not the point (1,1,1) anymore.in How do I programmatically move an ARAnchor?
So I understand the anchorNode as it is put at the location just as other real object in the real world,
then I use the setLocationPosition to add the node to the anchor(the anchor stay static,then the node shold stay static too), or if I use the anchorNode instead of node attached to anchorNode to present the AR labels,maybe i can get what i want.But I try these test,it fails.
I have the question about what's on earth the anchor mean and how I can get what i want to implement.
Thanks for any help.

I have tried more tests and found that during the interval,the anchors are not updated,while the children move parallel with the camera.While for those anchors created associated with the trackable objects such as plane,the child nodes are static.Also I find some description like this:
placing an anchor that is not associated with a Trackable object is usually not a good experience. The trackable object (planes, augmented images, oriented points) are update by ARCore to represent the connections between the real world image and the augmented, virtual images. If you place an anchor in the "air", it will drift and move relative to the real world. (https://github.com/google-ar/sceneform-android-sdk/issues/185)
I try to place ar object based on the anchors associated with the plane and created by hitTest,the rendering node works fine.
I guess this is just the reason,I need more experiments and if it is right,I want to know how to solve the problem.

Is possible to retrieve the physical (device) camera height over the physical floor?

I'm developing an app using ARCore. In this app I need:
1) to place an object always staying at the same pose in world space. Following the "Working with Anchors" article recommendations (https://developers.google.com/ar/develop/developer-guides/anchors) I'm attaching an anchor to the ARCore Session. That is, I'm not using Trackables at all.
2) as a secondary requisite the object must be placed automatically, that is, without tapping on the screen.
I've managed to solve the two requisites, having now the object "floating" in front of me, this way (very common code):
private void onSceneUpdate(FrameTime frameTime) {
...
if (_renderable!=null && _anchorNode==null) {
float[] position = {0f,0f,-10f};
float[] rotation = {0,0,0,1};
//
Anchor anchor=_arFragment.getArSceneView().getSession().createAnchor(new Pose(position,rotation));
//
_anchorNode = new AnchorNode(anchor);
_anchorNode.setRenderable(_renderable);
_anchorNode.setParent(_arFragment.getArSceneView().getScene());
_anchorNode.setLocalScale(new Vector3(0.01f,0.01f,0.01f)); //cm -> m
...
}
As i want the object to be on the floor, I need to find out what the height of my physical (device) camera above the floor is, in order to subtract that number from the current object's Y coordinate:
float[] position = {0f,HERE_THE_VALUE_TO_SUBTRACT_FROM_CAMERA_HEIGHT,-10f};
Certainly, it's an easy implementation when plane Trackables are used but here I have the requisites above-named.
I've managed to solve the two requisites, having now the object "floating" in front of me.
As i want the object to be on the floor, I need to find out what the height of my physical (device) camera above the floor is, in order to subtract that number from the current object's Y coordinate.
Trying with different camera/device Pose retrieval APIs, namely: frame.getAndroidSensorPose(), frame.getCamera().getPose() and frame.getCamera().getDisplayOrientedPose() are showing not valid values.
Thanks for your advice.
P.S.:Certainly, it's an easy implementation when plane Trackables are used but here I have other requisites, as above-named.
EDIT after Michael Dougan comments.
Well I think we have then two ways to achieve the requisites:
1) leave the code w/o changes, keeping on using the Session Anchor, asking the user to launch the app and the to follow a "calibration process" which the device on the floor. As this is a professional use app, and not a consumer one, we think it is perfectly suitable;
2) go ahead with the good-and-old Trackables, by means of the usual floor as an anchor, including the pose of that anchor in the calculation of the position of the 3D model.

Sceneform Collisions With Camera

I'm stretching my very limited ARCore knowledge.
My question is similar (but different) to this question
I want to work out if my device camera node intersects/overlaps with my other nodes, but I've not been having any luck so far
I'm trying something like this (the camera is another node):
scene.setOnUpdateListener(frameTime -> {
Node x = scene.overlapTest(scene.getCamera());
if (x != null) {
Log.i(TAG, "setUpArComponents: CAMERA HIT DETECTED at: " + x.getName());
logNodeStatus(x);
}
});
Firstly, does this make sense?
I can detect all node collisions in my scene using:
for (Node node : nodes) {
...
ArrayList<Node> results = scene.overlapTestAll(node);
...
}
Assuming that there isn't a renderable for the Camera node (so no default collision shape), I tried to set my own collision shape, but this was actually catching all the tap events I was trying to perform, so I figured I must be doing this wrong.
I'm thinking about things like fixing a deactivated node in front of the camera.
I may be asking for too much of ARCore, but has anyone found a way to detect a collision between the "user" (i.e. camera node) and another node? Or should I be doing this "collision detection" via indoor positioning instead?
Thanks in advance :)
UPDATE: it's really hacky and performance-heavy, but you can actually compare the camera's and node's world space positions from within onUpdate inside a node, you'll probably have to manage some tolerance and other things to smooth out interactions.

One idea to do the same thing is to use a raycast to hit the objects and if they are close do something. You could use something like this in the onUpdateListener:
Camera camera = arSceneView.getScene().getCamera();
Ray ray = new Ray(camera.getWorldPosition(), camera.getForward());
HitTestResult result = arSceneView.getScene().hitTest(ray);
if (result.getNode() != null && result.getDistance() <= SOME_THRESHOLD) {
// Hit something
doSomething (result.getNode());
}

Panning the view of a gameObject instead of the camera in Unity3d?

I'm having a hard time to pan a view of a gameObject in Unity3d. I'm new to scripting and I'm trying to develop an AR (Augmented Reality) application for Android.
I need to have a gameObject (e.g. a model of a floor), from the normal top down view, rendered to a "pseudo" iso view, inclined to 45 degrees. As the gameObject is inclined, I need to have a panning function on its view, utilizing four (4) buttons (for left, right, forward(or up), backward(or down)).
The problem is that, I cannot use any of the known panning script snippets around the forum, as the AR camera has to be static in the scene.
Need to mention that, I need the panning function to be active only at the isometric view, (which I already compute with another script), not on top down view. So there must be no problem with the inclination of the axes of the gameObject, right?
Following, are two mockup images of the states, the gameObject (model floor) is rendered and the script code (from Unity reference), that I'm currently using, which is not very much functional for my needs.
Here is the code snippet, for left movement of the gameObject. I use the same with a change in -, +speed values, for the other movements, but I get it only move up, down, not forth, backwards:
#pragma strict
// The target gameObject.
var target: Transform;
// Speed in units per sec.
var speedLeft: float = -10;
private static var isPanLeft = false;
function FixedUpdate()
{
if(isPanLeft == true)
{
// The step size is equal to speed times frame time.
var step = speedLeft * Time.deltaTime;
// Move model position a step closer to the target.
transform.position = Vector3.MoveTowards(transform.position, target.position, step);
}
}
static function doPanLeft()
{
isPanLeft = !isPanLeft;
}
It would be great, if someone be kind enough to take a look at this post, and make a suggestion on how this functionality can be coded the easiest way, as I'm a newbie?
Furthermore, if a sample code or a tutorial can be provided, it will be appreciated, as I can learn from this, a lot. Thank you all in advance for your time and answers.

If i understand correctly you have a camera with some fixed rotation and position and you have a object you want to move up/down/left/right from the cameras perspective
To rotated an object to a set of angles you simply do
transform.rotation = Quaternion.Euler(45, 45, 45);
Then to move it you use the cameras up/right/forward in worldspace like this to move it up and left
transform.position += camera.transform.up;
transform.position -= camera.transform.right;
If you only have one camera in your scene you can access its transform by Camera.main.transform
An example of how to move it when someone presses the left arrow
if(Input.GetKeyDown(KeyCode.LeftArrow))
{
transform.position -= camera.transform.right;
}