Develop Reference

Why are drawables being pushed passed the edges of layout?

I am working on a pin entry screen and the pin digit orbs are being scaled once their total width becomes greater than the content space available. But the first and final orb are pushing out of the content space and I can't figure out why.
This is an exmaple of the pin orbs pushing out the edges:
Code:
padViewHolder.addView(
padView,
ViewGroup.LayoutParams.MATCH_PARENT,
ViewGroup.LayoutParams.MATCH_PARENT)
padView.setPadActionListener(object : PadView.PadActionListener {
override fun symbolPressed() {
val digit = ImageView(pinPadDigitContainer.context)
digit.setImageResource(R.drawable.ic_pin_digit_filled)
if (noSpace) {
noSpace = false
pinPadDigitContainer.addView(digit)
pinPadDigitContainer.resizeChildrenToFit()
} else {
val space = Space(context)
pinPadDigitContainer.addView(space)
pinPadDigitContainer.addView(digit)
pinPadDigitContainer.resizeChildrenToFit()
pinPadDigitContainer.invalidate()
}
}
private fun ViewGroup.resizeChildrenToFit() {
val numOfOrbs = (childCount - 1) / 2
val displayWidth = resources.displayMetrics.run { widthPixels / density }
val singleSpace = calculatePercentageSpace(displayWidth).toInt() // 7% of screen width
val widthOfOrb = resources.getDimensionPixelSize(R.dimen.ic_pin_digit_width) // 29dp
val dpInt = resources.getDimensionPixelSize(R.dimen.ic_pin_orb_space) // 3dp
val contentSpace = width - paddingLeft - paddingRight
var childrenWidth = numOfOrbs * widthOfOrb + (singleSpace * (numOfOrbs - 1))
if (childrenWidth <= contentSpace) {
for (i in 0 until childCount) {
calculateOrbAndSpaceWidth(i, widthOfOrb, singleSpace)
}
} else {
childrenWidth = remain
val smallOrb = contentSpace / numOfOrbs - 3
for (i in 0 until childCount) {
calculateOrbAndSpaceWidth(i, smallOrb, dpInt)
}
}
}
This is the part of the view i'm working with:
<LinearLayout
android:id="#+id/meta__display2_container"
android:layout_width="0dp"
android:minWidth="0dp"
android:layout_height="0dp"
android:minHeight="0dp"
android:gravity="center"
android:orientation="horizontal"
android:clipChildren="true"
app:layout_constraintDimensionRatio="5.5:1"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintTop_toBottomOf="#id/meta__display1_s"
app:layout_constraintWidth_percent="0.56" />

You haven't posted the actual code where you calculate the scaled widths (calculateOrbAndSpaceWidth) but it looks like that's not being worked out correctly. Your code's a little hard for me to follow, so I can't really rework it, but can't you do something like this?
val orbWidth = // get default orb width
val spaceWidth = // get default space between orbs
val totalWidth = (orbWidth * orbCount) + (spaceWidth * (orbCount-1))
val availableWidth = // get width of view - padding
// Work out what scaling factor would make all the orbs fit the available space exactly.
// If they need to be scaled down (scale: less than 1.0) use that value, otherwise keep them
// at the current size (scale: 1.0)
val scaleFactor = minOf(1f, availableWidth / totalWidth.toFloat())
And then draw all your orbs with the scale factor applied to the orb dimensions, space widths, and any coordinate offsets. (I don't know how you're actually drawing them, but you're scaling them somehow!) When scaleFactor is 1.0 it'll draw everything as normal, when it's smaller it'll adjust all your measurements to make them fit.
Also make sure you're not mixing up dp and px measurements - you seem to be working with both. Personally I'd convert any of your dp values to pixels (which you're mostly doing) and just work with that. This part here:
val displayWidth = resources.displayMetrics.run { widthPixels / density }
Is taking the width of the window in pixels and converting it to dp. You only use it in this function call:
val singleSpace = calculatePercentageSpace(displayWidth).toInt()
so I can't see what you're doing with it - but you either that result to calculateOrbAndSpaceWidth, or dpInt (which is a px value), and you're getting expected results with one and unexpected with the other. So just make sure you're being consistent in there - might be fine, just pointing it out!

Off-center camera projection for 3d illusion using LibGDX

I'm trying to achieve this 3d pop out of the screen kind of effect using LibGDX on Android (the camera process described at the given link):
https://www.anxious-bored.com/blog/
but the result I get is stretched objects when the eye moves around.
I simulate the eye position using a gyroscope to get device rotation and assume a fixed distance from the device screen. I've also incorporated ARCore for eye tracking, but that also yields the same result.
Here is a GIF of what I'm seeing:
https://i.stack.imgur.com/YOf6n.gif
Anyone have any idea on what I'm doing wrong?
Here is the relevant code I'm using in Kotlin.
private fun updateCamera() {
val camera = // The Perspective camera instance
// Move the camera to the eye position, but keep the same camera direction
camera.view.setToLookAt(camera.direction, camera.up).mul(GDXHelperInstances.matrix4_1.setToTranslation(GDXHelperInstances.vector3_1.set(camera.position).scl(-1f)))
// Get the size of the screen in meters
val deviceHalfHeight = camera.viewportHeight / Gdx.graphics.ppcY * 0.01f
val deviceHalfWidth = camera.viewportWidth / Gdx.graphics.ppcX * 0.01f
// Get the device position and plane relative to the camera
val pos = Vector3(Vector3.Zero).mul(camera.view)
val plane = Plane(GDXHelperInstances.vector3_1.set(camera.direction).scl(-1f), Vector3.Zero)
// Calculate the bounds of the viewport in virtual space (the device screen dimensions in meters with center at Vector3.Zero) relative to the camera (view space)
val left = pos.x - deviceHalfWidth
val right = pos.x + deviceHalfWidth
val bottom = pos.y - deviceHalfHeight
val top = pos.y + deviceHalfHeight
val nearScale = camera.near / plane.distance(camera.position).absoluteValue
// Off-axis projection
camera.projection.setToProjection(left*nearScale, right*nearScale, bottom*nearScale, top*nearScale, camera.near, camera.far)
// Calculate new asymmetrical frustum
camera.combined.set(camera.projection)
Matrix4.mul(camera.combined.`val`, camera.view.`val`)
camera.invProjectionView.set(camera.combined)
Matrix4.inv(camera.invProjectionView.`val`)
camera.frustum.update(camera.invProjectionView)
}

How to completely fit an AR ViewRenderable on target image?

I want an AR ViewRenderable to be placed over the real world target picture and completely fit over its boundaries.
Although ViewSizer is used to change the size of AR objects, but it does it globally(sets dpPerMeter for every situation). However, I want to scale it wrt target's size,
I think something can be done by setLocalScale methods along with getExtentX and getExtentZ, but not sure how to set the parameters for my cause.
Have a look at the current code snippet :
setAnchor(image.createAnchor(image.getCenterPose()));
Node cornerNode = new Node();
cornerNode.setParent(this);
cornerNode.setLocalRotation(new Quaternion(new Vector3(1,0,0), -90));
cornerNode.setLocalPosition(new Vector3(0.0f, 0.0f, 0f));
cornerNode.setRenderable(targetImage.getNow(null));

By default, 250dp equals to 1 meters, but you can change it by ViewRenderable.builder().setSizer(DpToMetersViewSizer(you_size))
you have to calculate the scale by the image size and AR core estimated size, so add these code like
// in this example, the image is 100 cm x 66 cm
val imageWidth = 1f // = 1m
val imageHeight = 0.66 // = 66 cm
val scaledWidth = imageWidth / image.extentX
val scaledHeight = imageHeight / image.extentZ
// scale the Node
node.localScale = Vector3(scaledWidth, scaledHeight, scaledWidth)
// also, my view_wall.xml is 250dp x 166dp and the VerticalAlignment is center, like
val wall = ViewRenderable.builder().setView(this, R.layout.view_wall)
.setVerticalAlignment(ViewRenderable.VerticalAlignment.CENTER)
.build()
It works for me, help it works for you.
by the way this is my practice project https://github.com/swarmnyc/arcore-augmented-image-swarm

Combine image with video stream on Android

I am investigating Augmented Reality on Android.
I am using ARCore and Sceneform within an Android application.
I have tried out the sample projects and now would like to develop my own application.
One effect I would like to achieve is to combine/overlay an image (say .jpeg or .png) with a live feed from the devices onboard camera.
The image will have a transparent background that allows the user to see the live feed and image simultaneously
However I do not want the overlayed image to be a fixed/static watermark, When the user zooms in, out or pans the overlayed image must also zoom in, out and pan etc.
I do not wish the overplayed image to become 3d or anything of that nature.
Is this effect possible with Sceneform? or will I need to use other 3rd party libraries and/or tools to achieve the desired results.
UPDATE
The user is drawing on a blank sheet of white paper. The sheet of paper is orientated so that the user is comfortably drawing (either left or right handed). The user is free to move the sheet of paper while they complete their image.
An Android device is held above the sheet of paper filming the user drawing their selected image.
The live camera feed is being cast to a large TV or monitor screen.
To aid the user they have selected a static image to "trace" or "Copy".
This image is chosen on the Android device and is being combined with the live camera stream within the Android application.
The user can zoom in and out on their drawing and the combined live stream and selected static image will also zoom in and out, this will enable the user to make an accurate copy of the selected static image by drawing "Free Hand".
When the user looks directly at the sheet of paper, they only see their drawing.
When the user views the cast live stream of them drawing on the TV or monitor they see their drawing and the chosen static image superimposed. The user can control the transparency of the static image to assist them in making an accurate copy of it.

I think what you are looking for is to use AR to display an image so that the image stays in place, for example over a sheet of paper in order to act as a guide for drawing a copy of the image on the paper.
There are 2 parts to this. First is to locate the sheet of paper, the second is to place the image over the paper and keep it there as the phone moves around.
Locating the sheet of paper can be done just by detecting the plane with the paper (having some contrast, or pattern or something vs. a plain white sheet of paper will help), then tap on where the center of the page should be. This is done in the HelloSceneform sample.
If you want to have a more accurate bounding of the paper, you could tap the 4 corners of the paper, and then create anchors there. To do this register a plane tapped listener in onCreate()
arFragment.setOnTapArPlaneListener(this::onPlaneTapped);
Then in onPlaneTapped, create the 4 anchorNodes. Once you have 4, initialize the drawing to be displayed.
private void onPlaneTapped(HitResult hitResult, Plane plane, MotionEvent event) {
if (cornerAnchors.size() != 4) {
AnchorNode corner = createCornerNode(hitResult.createAnchor());
arFragment.getArSceneView().getScene().addChild(corner);
cornerAnchors.add(corner);
}
if (cornerAnchors.size() == 4 && drawingNode == null) {
initializeDrawing();
}
}
To initialize the drawing, create a Sceneform Texture from the bitmap or drawable. This can be from a resource or a file URL. You want the texture to show the whole image, and scale as the model holding it is resized.
private void initializeDrawing() {
Texture.Sampler sampler = Texture.Sampler.builder()
.setWrapMode(Texture.Sampler.WrapMode.CLAMP_TO_EDGE)
.setMagFilter(Texture.Sampler.MagFilter.NEAREST)
.setMinFilter(Texture.Sampler.MinFilter.LINEAR_MIPMAP_LINEAR)
.build();
Texture.builder()
.setSource(this, R.drawable.logo_google_developers)
.setSampler(sampler)
.build()
.thenAccept(texture -> {
MaterialFactory.makeTransparentWithTexture(this, texture)
.thenAccept(this::buildDrawingRenderable);
});
}
The model to hold the texture is just a flat quad sized to the smallest dimension between the corners. This is the same logic as laying out a quad using OpenGL.
private void buildDrawingRenderable(Material material) {
Integer[] indices = {
0, 1, 3, 3, 1, 2
};
//Calculate the center of the corners.
float min_x = Float.MAX_VALUE;
float max_x = Float.MIN_VALUE;
float min_z = Float.MAX_VALUE;
float max_z = Float.MIN_VALUE;
for (AnchorNode node : cornerAnchors) {
float x = node.getWorldPosition().x;
float z = node.getWorldPosition().z;
min_x = Float.min(min_x, x);
max_x = Float.max(max_x, x);
min_z = Float.min(min_z, z);
max_z = Float.max(max_z, z);
}
float width = Math.abs(max_x - min_x);
float height = Math.abs(max_z - min_z);
float extent = Math.min(width / 2, height / 2);
Vertex[] vertices = {
Vertex.builder()
.setPosition(new Vector3(-extent, 0, extent))
.setUvCoordinate(new Vertex.UvCoordinate(0, 1)) // top left
.build(),
Vertex.builder()
.setPosition(new Vector3(extent, 0, extent))
.setUvCoordinate(new Vertex.UvCoordinate(1, 1)) // top right
.build(),
Vertex.builder()
.setPosition(new Vector3(extent, 0, -extent))
.setUvCoordinate(new Vertex.UvCoordinate(1, 0)) // bottom right
.build(),
Vertex.builder()
.setPosition(new Vector3(-extent, 0, -extent))
.setUvCoordinate(new Vertex.UvCoordinate(0, 0)) // bottom left
.build()
};
RenderableDefinition.Submesh[] submeshes = {
RenderableDefinition.Submesh.builder().
setMaterial(material)
.setTriangleIndices(Arrays.asList(indices))
.build()
};
RenderableDefinition def = RenderableDefinition.builder()
.setSubmeshes(Arrays.asList(submeshes))
.setVertices(Arrays.asList(vertices)).build();
ModelRenderable.builder().setSource(def)
.setRegistryId("drawing").build()
.thenAccept(this::positionDrawing);
}
The last part is to position the quad in the center of the corners, and create a Transformable node so the image can be nudged into position, rotated, or scaled to be the perfect size.
private void positionDrawing(ModelRenderable drawingRenderable) {
//Calculate the center of the corners.
float min_x = Float.MAX_VALUE;
float max_x = Float.MIN_VALUE;
float min_z = Float.MAX_VALUE;
float max_z = Float.MIN_VALUE;
for (AnchorNode node : cornerAnchors) {
float x = node.getWorldPosition().x;
float z = node.getWorldPosition().z;
min_x = Float.min(min_x, x);
max_x = Float.max(max_x, x);
min_z = Float.min(min_z, z);
max_z = Float.max(max_z, z);
}
Vector3 center = new Vector3((min_x + max_x) / 2f,
cornerAnchors.get(0).getWorldPosition().y, (min_z + max_z) / 2f);
Anchor centerAnchor = null;
Vector3 screenPt = arFragment.getArSceneView().getScene().getCamera().worldToScreenPoint(center);
List<HitResult> hits = arFragment.getArSceneView().getArFrame().hitTest(screenPt.x, screenPt.y);
for (HitResult hit : hits) {
if (hit.getTrackable() instanceof Plane) {
centerAnchor = hit.createAnchor();
break;
}
}
AnchorNode centerNode = new AnchorNode(centerAnchor);
centerNode.setParent(arFragment.getArSceneView().getScene());
drawingNode = new TransformableNode(arFragment.getTransformationSystem());
drawingNode.setParent(centerNode);
drawingNode.setRenderable(drawingRenderable);
}

The intended AR reference image can be scaled with ARobjects as points for the sizing of the template for the user.
The more complex AR images will not work easily, since the AR image is overlaid on top of the users tracing, and this will obstruct the tip of their pen/pencil.
My solution is to chromakey the white paper. This will replace the white paper with the chosen image or live feed. Moving the paper around as you specified would be an issue, unless you have a means of tracking the paper position.
As you can see in this example, AR objects are in front, while chromakey is background. Tracing surface (paper) would be in the center.
Reference to this example is on the link below.
RJ
YouTube - AR tracked environment

Converting Camera Coordinates to Custom View Coordinates

I am trying to make a simple face detection app consisting of a SurfaceView (essentially a camera preview) and a custom View (for drawing purposes) stacked on top. The two views are essentially the same size, stacked on one another in a RelativeLayout. When a person's face is detected, I want to draw a white rectangle on the custom View around their face.
The Camera.Face.rect object returns the face bound coordinates using the coordinate system explained here and the custom View uses the coordinate system described in the answer to this question. Some sort of conversion is needed before I can use it to draw on the canvas.
Therefore, I wrote an additional method ScaleFacetoView() in my custom view class (below) I redraw the custom view every time a face is detected by overriding the OnFaceDetection() method. The result is the white box appears correctly when a face is in the center. The problem I noticed is that it does not correct track my face when it moves to other parts of the screen.
Namely, if I move my face:
Up - the box goes left
Down - the box goes right
Right - the box goes upwards
Left - the box goes down
I seem to have incorrectly mapped the values when scaling the coordinates. Android docs provide this method of converting using a matrix, but it is rather confusing and I have no idea what it is doing. Can anyone provide some code on the correct way of converting Camera.Face coordinates to View coordinates?
Here's the code for my ScaleFacetoView() method.
public void ScaleFacetoView(Face[] data, int width, int height, TextView a){
//Extract data from the face object and accounts for the 1000 value offset
mLeft = data[0].rect.left + 1000;
mRight = data[0].rect.right + 1000;
mTop = data[0].rect.top + 1000;
mBottom = data[0].rect.bottom + 1000;
//Compute the scale factors
float xScaleFactor = 1;
float yScaleFactor = 1;
if (height > width){
xScaleFactor = (float) width/2000.0f;
yScaleFactor = (float) height/2000.0f;
}
else if (height < width){
xScaleFactor = (float) height/2000.0f;
yScaleFactor = (float) width/2000.0f;
}
//Scale the face parameters
mLeft = mLeft * xScaleFactor; //X-coordinate
mRight = mRight * xScaleFactor; //X-coordinate
mTop = mTop * yScaleFactor; //Y-coordinate
mBottom = mBottom * yScaleFactor; //Y-coordinate
}
As mentioned above, I call the custom view like so:
#Override
public void onFaceDetection(Face[] arg0, Camera arg1) {
if(arg0.length == 1){
//Get aspect ratio of the screen
View parent = (View) mRectangleView.getParent();
int width = parent.getWidth();
int height = parent.getHeight();
//Modify xy values in the view object
mRectangleView.ScaleFacetoView(arg0, width, height);
mRectangleView.setInvalidate();
//Toast.makeText( cc ,"Redrew the face.", Toast.LENGTH_SHORT).show();
mRectangleView.setVisibility(View.VISIBLE);
//rest of code

Using the explanation Kenny gave I manage to do the following.
This example works using the front facing camera.
RectF rectF = new RectF(face.rect);
Matrix matrix = new Matrix();
matrix.setScale(1, 1);
matrix.postScale(view.getWidth() / 2000f, view.getHeight() / 2000f);
matrix.postTranslate(view.getWidth() / 2f, view.getHeight() / 2f);
matrix.mapRect(rectF);
The returned Rectangle by the matrix has all the right coordinates to draw into the canvas.
If you are using the back camera I think is just a matter of changing the scale to:
matrix.setScale(-1, 1);
But I haven't tried that.

The Camera.Face class returns the face bound coordinates using the image frame that the phone would save into its internal storage, rather than using the image displayed in the Camera Preview. In my case, the images were saved in a different manner from the camera, resulting in a incorrect mapping. I had to manually account for the discrepancy by taking the coordinates, rotating it counter clockwise 90 degrees and flipping it on the y-axis prior to scaling it to the canvas used for the custom view.
EDIT:
It would also appear that you can't change the way the face bound coordinates are returned by modifying the camera capture orientation using the Camera.Parameters.setRotation(int) method either.