I am investigating Augmented Reality on Android.
I am using ARCore and Sceneform within an Android application.
I have tried out the sample projects and now would like to develop my own application.
One effect I would like to achieve is to combine/overlay an image (say .jpeg or .png) with a live feed from the devices onboard camera.
The image will have a transparent background that allows the user to see the live feed and image simultaneously
However I do not want the overlayed image to be a fixed/static watermark, When the user zooms in, out or pans the overlayed image must also zoom in, out and pan etc.
I do not wish the overplayed image to become 3d or anything of that nature.
Is this effect possible with Sceneform? or will I need to use other 3rd party libraries and/or tools to achieve the desired results.
UPDATE
The user is drawing on a blank sheet of white paper. The sheet of paper is orientated so that the user is comfortably drawing (either left or right handed). The user is free to move the sheet of paper while they complete their image.
An Android device is held above the sheet of paper filming the user drawing their selected image.
The live camera feed is being cast to a large TV or monitor screen.
To aid the user they have selected a static image to "trace" or "Copy".
This image is chosen on the Android device and is being combined with the live camera stream within the Android application.
The user can zoom in and out on their drawing and the combined live stream and selected static image will also zoom in and out, this will enable the user to make an accurate copy of the selected static image by drawing "Free Hand".
When the user looks directly at the sheet of paper, they only see their drawing.
When the user views the cast live stream of them drawing on the TV or monitor they see their drawing and the chosen static image superimposed. The user can control the transparency of the static image to assist them in making an accurate copy of it.
I think what you are looking for is to use AR to display an image so that the image stays in place, for example over a sheet of paper in order to act as a guide for drawing a copy of the image on the paper.
There are 2 parts to this. First is to locate the sheet of paper, the second is to place the image over the paper and keep it there as the phone moves around.
Locating the sheet of paper can be done just by detecting the plane with the paper (having some contrast, or pattern or something vs. a plain white sheet of paper will help), then tap on where the center of the page should be. This is done in the HelloSceneform sample.
If you want to have a more accurate bounding of the paper, you could tap the 4 corners of the paper, and then create anchors there. To do this register a plane tapped listener in onCreate()
arFragment.setOnTapArPlaneListener(this::onPlaneTapped);
Then in onPlaneTapped, create the 4 anchorNodes. Once you have 4, initialize the drawing to be displayed.
private void onPlaneTapped(HitResult hitResult, Plane plane, MotionEvent event) {
if (cornerAnchors.size() != 4) {
AnchorNode corner = createCornerNode(hitResult.createAnchor());
arFragment.getArSceneView().getScene().addChild(corner);
cornerAnchors.add(corner);
}
if (cornerAnchors.size() == 4 && drawingNode == null) {
initializeDrawing();
}
}
To initialize the drawing, create a Sceneform Texture from the bitmap or drawable. This can be from a resource or a file URL. You want the texture to show the whole image, and scale as the model holding it is resized.
private void initializeDrawing() {
Texture.Sampler sampler = Texture.Sampler.builder()
.setWrapMode(Texture.Sampler.WrapMode.CLAMP_TO_EDGE)
.setMagFilter(Texture.Sampler.MagFilter.NEAREST)
.setMinFilter(Texture.Sampler.MinFilter.LINEAR_MIPMAP_LINEAR)
.build();
Texture.builder()
.setSource(this, R.drawable.logo_google_developers)
.setSampler(sampler)
.build()
.thenAccept(texture -> {
MaterialFactory.makeTransparentWithTexture(this, texture)
.thenAccept(this::buildDrawingRenderable);
});
}
The model to hold the texture is just a flat quad sized to the smallest dimension between the corners. This is the same logic as laying out a quad using OpenGL.
private void buildDrawingRenderable(Material material) {
Integer[] indices = {
0, 1, 3, 3, 1, 2
};
//Calculate the center of the corners.
float min_x = Float.MAX_VALUE;
float max_x = Float.MIN_VALUE;
float min_z = Float.MAX_VALUE;
float max_z = Float.MIN_VALUE;
for (AnchorNode node : cornerAnchors) {
float x = node.getWorldPosition().x;
float z = node.getWorldPosition().z;
min_x = Float.min(min_x, x);
max_x = Float.max(max_x, x);
min_z = Float.min(min_z, z);
max_z = Float.max(max_z, z);
}
float width = Math.abs(max_x - min_x);
float height = Math.abs(max_z - min_z);
float extent = Math.min(width / 2, height / 2);
Vertex[] vertices = {
Vertex.builder()
.setPosition(new Vector3(-extent, 0, extent))
.setUvCoordinate(new Vertex.UvCoordinate(0, 1)) // top left
.build(),
Vertex.builder()
.setPosition(new Vector3(extent, 0, extent))
.setUvCoordinate(new Vertex.UvCoordinate(1, 1)) // top right
.build(),
Vertex.builder()
.setPosition(new Vector3(extent, 0, -extent))
.setUvCoordinate(new Vertex.UvCoordinate(1, 0)) // bottom right
.build(),
Vertex.builder()
.setPosition(new Vector3(-extent, 0, -extent))
.setUvCoordinate(new Vertex.UvCoordinate(0, 0)) // bottom left
.build()
};
RenderableDefinition.Submesh[] submeshes = {
RenderableDefinition.Submesh.builder().
setMaterial(material)
.setTriangleIndices(Arrays.asList(indices))
.build()
};
RenderableDefinition def = RenderableDefinition.builder()
.setSubmeshes(Arrays.asList(submeshes))
.setVertices(Arrays.asList(vertices)).build();
ModelRenderable.builder().setSource(def)
.setRegistryId("drawing").build()
.thenAccept(this::positionDrawing);
}
The last part is to position the quad in the center of the corners, and create a Transformable node so the image can be nudged into position, rotated, or scaled to be the perfect size.
private void positionDrawing(ModelRenderable drawingRenderable) {
//Calculate the center of the corners.
float min_x = Float.MAX_VALUE;
float max_x = Float.MIN_VALUE;
float min_z = Float.MAX_VALUE;
float max_z = Float.MIN_VALUE;
for (AnchorNode node : cornerAnchors) {
float x = node.getWorldPosition().x;
float z = node.getWorldPosition().z;
min_x = Float.min(min_x, x);
max_x = Float.max(max_x, x);
min_z = Float.min(min_z, z);
max_z = Float.max(max_z, z);
}
Vector3 center = new Vector3((min_x + max_x) / 2f,
cornerAnchors.get(0).getWorldPosition().y, (min_z + max_z) / 2f);
Anchor centerAnchor = null;
Vector3 screenPt = arFragment.getArSceneView().getScene().getCamera().worldToScreenPoint(center);
List<HitResult> hits = arFragment.getArSceneView().getArFrame().hitTest(screenPt.x, screenPt.y);
for (HitResult hit : hits) {
if (hit.getTrackable() instanceof Plane) {
centerAnchor = hit.createAnchor();
break;
}
}
AnchorNode centerNode = new AnchorNode(centerAnchor);
centerNode.setParent(arFragment.getArSceneView().getScene());
drawingNode = new TransformableNode(arFragment.getTransformationSystem());
drawingNode.setParent(centerNode);
drawingNode.setRenderable(drawingRenderable);
}
The intended AR reference image can be scaled with ARobjects as points for the sizing of the template for the user.
The more complex AR images will not work easily, since the AR image is overlaid on top of the users tracing, and this will obstruct the tip of their pen/pencil.
My solution is to chromakey the white paper. This will replace the white paper with the chosen image or live feed. Moving the paper around as you specified would be an issue, unless you have a means of tracking the paper position.
As you can see in this example, AR objects are in front, while chromakey is background. Tracing surface (paper) would be in the center.
Reference to this example is on the link below.
RJ
YouTube - AR tracked environment
Related
I'm trying to achieve this 3d pop out of the screen kind of effect using LibGDX on Android (the camera process described at the given link):
https://www.anxious-bored.com/blog/
but the result I get is stretched objects when the eye moves around.
I simulate the eye position using a gyroscope to get device rotation and assume a fixed distance from the device screen. I've also incorporated ARCore for eye tracking, but that also yields the same result.
Here is a GIF of what I'm seeing:
https://i.stack.imgur.com/YOf6n.gif
Anyone have any idea on what I'm doing wrong?
Here is the relevant code I'm using in Kotlin.
private fun updateCamera() {
val camera = // The Perspective camera instance
// Move the camera to the eye position, but keep the same camera direction
camera.view.setToLookAt(camera.direction, camera.up).mul(GDXHelperInstances.matrix4_1.setToTranslation(GDXHelperInstances.vector3_1.set(camera.position).scl(-1f)))
// Get the size of the screen in meters
val deviceHalfHeight = camera.viewportHeight / Gdx.graphics.ppcY * 0.01f
val deviceHalfWidth = camera.viewportWidth / Gdx.graphics.ppcX * 0.01f
// Get the device position and plane relative to the camera
val pos = Vector3(Vector3.Zero).mul(camera.view)
val plane = Plane(GDXHelperInstances.vector3_1.set(camera.direction).scl(-1f), Vector3.Zero)
// Calculate the bounds of the viewport in virtual space (the device screen dimensions in meters with center at Vector3.Zero) relative to the camera (view space)
val left = pos.x - deviceHalfWidth
val right = pos.x + deviceHalfWidth
val bottom = pos.y - deviceHalfHeight
val top = pos.y + deviceHalfHeight
val nearScale = camera.near / plane.distance(camera.position).absoluteValue
// Off-axis projection
camera.projection.setToProjection(left*nearScale, right*nearScale, bottom*nearScale, top*nearScale, camera.near, camera.far)
// Calculate new asymmetrical frustum
camera.combined.set(camera.projection)
Matrix4.mul(camera.combined.`val`, camera.view.`val`)
camera.invProjectionView.set(camera.combined)
Matrix4.inv(camera.invProjectionView.`val`)
camera.frustum.update(camera.invProjectionView)
}
I would like to reduce the reduce bar code tracking window when using the google vision api. There are some answers here but they feel a bit outdated.
I'm using google's sample: https://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart
Currently, I try to figure out if a barcode is inside my overlay box inside BarcodeScannerProcessor onSuccess callback:
override fun onSuccess(barcodes: List<Barcode>, graphicOverlay: GraphicOverlay) {
if(barcodes.isEmpty())
return;
for(barcode in barcodes) {
val center = Point(graphicOverlay.imageWidth / 2, graphicOverlay.imageHeight / 2)
val rectWidth = graphicOverlay.imageWidth * Settings.OverlayWidthFactor
val rectHeight = graphicOverlay.imageHeight * Settings.OverlayHeightFactor
val left = center.x - rectWidth / 2
val top = center.y - rectHeight / 2
val right = center.x + rectWidth / 2
val bottom = center.y + rectHeight / 2
val rect = Rect(left.toInt(), top.toInt(), right.toInt(), bottom.toInt())
val contains = rect.contains(barcode.boundingBox!!)
val color = if(contains) Color.GREEN else Color.RED
graphicOverlay.add(BarcodeGraphic(graphicOverlay, barcode, "left: ${barcode.boundingBox!!.left}", color))
}
}
Y-wise it works perfectly, but the X values from barcode.boundingBox e.g. barcode.boundingBox.left seems to have an offset. Is it based on what's being calculated in GraphicOverlay?
I'm expecting the value below to be close to 0, but the offset is about 90 here:
Or perhaps it's more efficient to crop the image according to the box?
Actually the bounding box is correct. The trick is that the image aspect ratio doesn't match the viewport aspect ratio so the image is cropped horizontally. Try to open settings (a gear in the top right corner) and choose an appropriate resolution.
For example take a look at these two screenshots. On the first one the selected resolution (1080x1920) matches my phone resolution so the padding looks good (17px). On the second screenshot the aspect ratio is different (1.0 for 720x720 resolution) therefore the image is cropped and the padding looks incorrect.
So the offset should be transformed from image coordinates to the screen coordinates. Under the hood GraphicOverlay uses a matrix for this transformation. You can use the same matrix:
for(barcode in barcodes) {
barcode.boundingBox?.let { bbox ->
val offset = floatArrayOf(bbox.left.toFloat(), bbox.top.toFloat())
graphicOverlay.transformationMatrix.mapPoints(offset)
val leftOffset = offset[0]
val topOffset = offset[1]
...
}
}
The only thing is that the transformationMatrix is private, so you should add a getter to access it.
As you know, the preview size of the camera is configurable at the settings menu. This configurable size specifies the graphicOverlay dimensions.
On the other hand, the aspect ratio of the CameraSourcePreview (i.e. preview_view in activity_vision_live_preview.xml) which is shown on the screen, does not necessarily equal to the ratio of the graphicOverlay. Because depends on the size of the phone's screen and the height that the parent ConstraintLayout allows occupying.
So, in the preview, based on the difference between the aspect ratio of graphicOverlay and preview_view, some part of the graphicOverlay might not be shown horizontally or vertically.
There are some parameters inside GraphicOverlay that can help us to adjust the left and top of the barcode's boundingBox in such a way that they start from 0 in the visible area.
First of all, they should be accessible out of the GraphicOverlay class. So, it's just enough to write a getter method for them:
GraphicOverlay.java
public class GraphicOverlay extends View {
...
/**
* The factor of overlay View size to image size. Anything in the image coordinates need to be
* scaled by this amount to fit with the area of overlay View.
*/
public float getScaleFactor() {
return scaleFactor;
}
/**
* The number of vertical pixels needed to be cropped on each side to fit the image with the
* area of overlay View after scaling.
*/
public float getPostScaleHeightOffset() {
return postScaleHeightOffset;
}
/**
* The number of horizontal pixels needed to be cropped on each side to fit the image with the
* area of overlay View after scaling.
*/
public float getPostScaleWidthOffset() {
return postScaleWidthOffset;
}
}
Now, it is possible to calculate the left and top difference gap using these parameters like the following:
BarcodeScannerProcessor.kt
class BarcodeScannerProcessor(
context: Context
) : VisionProcessorBase<List<Barcode>>(context) {
...
override fun onSuccess(barcodes: List<Barcode>, graphicOverlay: GraphicOverlay) {
if (barcodes.isEmpty()) {
Log.v(MANUAL_TESTING_LOG, "No barcode has been detected")
}
val leftDiff = graphicOverlay.run { postScaleWidthOffset / scaleFactor }.toInt()
val topDiff = graphicOverlay.run { postScaleHeightOffset / scaleFactor }.toInt()
for (i in barcodes.indices) {
val barcode = barcodes[i]
val color = Color.RED
val text = "left: ${barcode.boundingBox!!.left - leftDiff} top: ${barcode.boundingBox!!.top - topDiff}"
graphicOverlay.add(MyBarcodeGraphic(graphicOverlay, barcode, text, color))
logExtrasForTesting(barcode)
}
}
...
}
Visual Result:
Here is the visual result of the output. As it's obvious in the pictures, the gap between both left & top of the barcode and the left and top of the visible area is started from 0. In the case of the left picture, the graphicOverlay is set to the size of 480x640 (aspect ratio ≈ 1.3334) and for the right one 360x640 (aspect ratio ≈ 1.7778). In both cases, on my phone, the CameraSourcePreview has a steady size of 1440x2056 pixels (aspect ratio ≈ 1.4278), so it means that the calculation truly reflected the position of the barcode in the visible area.
(note that the aspect ratio of the visible area in one experiment is lower than that of graphicOverlay, and in another experiment, greater: 1.3334 < 1.4278 < 1.7778. So, the left values and top values are adjusted respectively.)
i am new to unity and need some help regarding creating a background that will look something like this (A bit jittery because its a gif), i want it to be like fill every screen size and have size 1/8th of the screen (the black box):
You can use the following setup:
First the image should have borders like this one and set its Wrap mode to reapeat in the import settings
Your background should be a ScreenSpace Overlay Canvas (depends on your setup ofcourse)
Within that Canvas have a RawImage object, use your image as Texture and add this component to it
[RequireComponent(typeof(RawImage))]
public class BackgroundController : MonoBehaviour
{
[Header("References")]
[SerializeField] private RectTransform _rectTransform;
[SerializeField] private RectTransform _parentRectTransform;
[SerializeField] private RawImage _image;
[Header("Settings")]
[SerializeField] private Vector2 repeatCount;
[SerializeField] private Vector2 scroll;
[SerializeField] private Vector2 offset;
private void Awake()
{
if (!_image) _image = GetComponent<RawImage>();
_image.uvRect = new Rect(offset, repeatCount);
}
// Start is called before the first frame update
private void Start()
{
if (!_rectTransform) _rectTransform = GetComponent<RectTransform>();
if (!_parentRectTransform) _parentRectTransform = GetComponentInParent<RectTransform>();
SetScale();
}
// Update is called once per frame
private void Update()
{
#if UNITY_EDITOR
// Only done in the Unity editor since later it is unlikely that your screensize changes
SetScale();
#endif
offset += scroll * Time.deltaTime;
_image.uvRect = new Rect(offset, repeatCount);
}
private void SetScale()
{
// get the diagonal size of the screen since the parent is the Canvas with
// ScreenSpace overlay it is always fiting the screensize
var parentCorners = new Vector3[4];
_parentRectTransform.GetLocalCorners(parentCorners);
var diagonal = Vector3.Distance(parentCorners[0], parentCorners[2]);
// set width and height to at least the diagonal
_rectTransform.sizeDelta = new Vector2(diagonal, diagonal);
}
}
This first scales the RawImage to fit the diagonal size of the parent. Since it is already fitting the screen this gets us the screen sizes => always fills the entire screen, no matter what the scales or rotation are (as long as your RawImage is on the center of the screen ofcourse).
Using the repeatCount you define how often the texture should be on the background.
Then using the scroll you can define how fast and in which direction the background should scroll. The script basically simply updates the RawImage.uvRect every frame.
Finally you simply rotate the RawImage so the scroll goes in the final direction you want
So you want some kind of infinity scrolling background?
Here is some simple way to create it.
1) You will need tilable image (that can be connected one side to other seamlessly). You can use one frame from your gif. Add it to your assets (just drag and drop it there).
2) In your Unity scene create new Quad object (GameObject->3d Object->Quad)
3) Drag and drop your image from your assets window right onto your Quad. That will apply texture to it.
4) Create simple script on your Quad object. I called mine RollerScript
using System.Collections;
using UnityEngine;
public class RollerScript : MonoBehaviour
{
public float speed = 2f;
public MeshRenderer renderer;
void Update()
{
Vector2 offset = new Vector2(Time.time * speed, Time.time * speed);
renderer.material.mainTextureOffset = offset;
}
}
5) Go back to Editor and assign renderer field (drag your Quad object from Hierarchy to that field)
6) Hit Play and adjust speed parameter in your script editor window. Your texture will scroll diagonally to right-top (as on your gif). If you want another direction you can change this line:
Vector2 offset = new Vector2(Time.time * speed, Time.time * speed);
Set x or y value of Vector2 to zero if you want NO scrolling horizontaly/vertically. Change x or y value to -x or -y if you want to scroll in opposite direction.
I need to draw something like this:
I was hoping that this guy posted some code of how he drew his segmented circle to begin with, but alas he didn't.
I also need to know which segment is where after interaction with the wheel - for instance if the wheel is rotated, I need to know where the original segments are after the rotation action.
Two questions:
Do I draw this segmented circle (with varying colours and content placed on the segment) with OpenGL or using Android Canvas?
Using either of the options, how do I register which segment is where?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
EDIT:
Ok, so I've figured out how to draw the segmented circle using Canvas (I'll post the code as an answer). And I'm sure I'll figure out how to rotate the circle soon. But I'm still unsure how I'll recognize a separate segment of the drawn wheel after the rotation action.
Because, what I'm thinking of doing is drawing the segmented circle with these wedges, and the sort of handling the entire Canvas as an ImageView when I want to rotate it as if it's spinning. But when the spinning stops, how do I differentiate between the original segments drawn on the Canvas?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I've read about how to draw a segment on its own (here also), OpenGL, Canvas and even drawing shapes and layering them, but I've yet to see someone explaining how to recognize the separate segments.
Can drawBitmap() or createBitmap() perhaps be used?
If I go with OpenGL, I'll probably be able to rotate the segmented wheel using OpenGL's rotation, right?
I've also read that OpenGL might be too powerful for what I'd like to do, so should I rather consider "the graphic components of a game library built on top of OpenGL"?
This kind of answers my first question above - how to draw the segmented circle using Android Canvas:
Using the code found here, I do this in the onDraw function:
// Starting values
private int startAngle = 0;
private int numberOfSegments = 11;
private int sweepAngle = 360 / numberOfSegments;
#Override
protected void onDraw(Canvas canvas) {
setUpPaint();
setUpDrawingArea();
colours = getColours();
Log.d(TAG, "Draw the segmented circle");
for (int i = 0; i < numberOfSegments; i++) {
// pick a colour that is not the previous colour
paint.setColor(colours.get(pickRandomColour()));
// Draw arc
canvas.drawArc(rectF, startAngle, sweepAngle, true, paint);
// Set variable values
startAngle -= sweepAngle;
}
}
This is how I set up the drawing area based on the device's screen size:
private void setUpDrawingArea() {
Log.d(TAG, "Set up drawing area.");
// First get the screen dimensions
Point size = new Point();
Display display = DrawArcActivity.this.getWindowManager().getDefaultDisplay();
display.getSize(size);
int width = size.x;
int height = size.y;
Log.d(TAG, "Screen size = "+width+" x "+height);
// Set up the padding
int paddingLeft = (int) DrawArcActivity.this.getResources().getDimension(R.dimen.padding_large);
int paddingTop = (int) DrawArcActivity.this.getResources().getDimension(R.dimen.padding_large);
int paddingRight = (int) DrawArcActivity.this.getResources().getDimension(R.dimen.padding_large);
int paddingBottom = (int) DrawArcActivity.this.getResources().getDimension(R.dimen.padding_large);
// Then get the left, top, right and bottom Xs and Ys for the rectangle we're going to draw in
int left = 0 + paddingLeft;
int top = 0 + paddingTop;
int right = width - paddingRight;
int bottom = width - paddingBottom;
Log.d(TAG, "Rectangle placement -> left = "+left+", top = "+top+", right = "+right+", bottom = "+bottom);
rectF = new RectF(left, top, right, bottom);
}
That (and the other functions which are pretty straight forward, so I'm not going to paste the code here) draws this:
The segments are different colours with every run.
I am trying to make a simple face detection app consisting of a SurfaceView (essentially a camera preview) and a custom View (for drawing purposes) stacked on top. The two views are essentially the same size, stacked on one another in a RelativeLayout. When a person's face is detected, I want to draw a white rectangle on the custom View around their face.
The Camera.Face.rect object returns the face bound coordinates using the coordinate system explained here and the custom View uses the coordinate system described in the answer to this question. Some sort of conversion is needed before I can use it to draw on the canvas.
Therefore, I wrote an additional method ScaleFacetoView() in my custom view class (below) I redraw the custom view every time a face is detected by overriding the OnFaceDetection() method. The result is the white box appears correctly when a face is in the center. The problem I noticed is that it does not correct track my face when it moves to other parts of the screen.
Namely, if I move my face:
Up - the box goes left
Down - the box goes right
Right - the box goes upwards
Left - the box goes down
I seem to have incorrectly mapped the values when scaling the coordinates. Android docs provide this method of converting using a matrix, but it is rather confusing and I have no idea what it is doing. Can anyone provide some code on the correct way of converting Camera.Face coordinates to View coordinates?
Here's the code for my ScaleFacetoView() method.
public void ScaleFacetoView(Face[] data, int width, int height, TextView a){
//Extract data from the face object and accounts for the 1000 value offset
mLeft = data[0].rect.left + 1000;
mRight = data[0].rect.right + 1000;
mTop = data[0].rect.top + 1000;
mBottom = data[0].rect.bottom + 1000;
//Compute the scale factors
float xScaleFactor = 1;
float yScaleFactor = 1;
if (height > width){
xScaleFactor = (float) width/2000.0f;
yScaleFactor = (float) height/2000.0f;
}
else if (height < width){
xScaleFactor = (float) height/2000.0f;
yScaleFactor = (float) width/2000.0f;
}
//Scale the face parameters
mLeft = mLeft * xScaleFactor; //X-coordinate
mRight = mRight * xScaleFactor; //X-coordinate
mTop = mTop * yScaleFactor; //Y-coordinate
mBottom = mBottom * yScaleFactor; //Y-coordinate
}
As mentioned above, I call the custom view like so:
#Override
public void onFaceDetection(Face[] arg0, Camera arg1) {
if(arg0.length == 1){
//Get aspect ratio of the screen
View parent = (View) mRectangleView.getParent();
int width = parent.getWidth();
int height = parent.getHeight();
//Modify xy values in the view object
mRectangleView.ScaleFacetoView(arg0, width, height);
mRectangleView.setInvalidate();
//Toast.makeText( cc ,"Redrew the face.", Toast.LENGTH_SHORT).show();
mRectangleView.setVisibility(View.VISIBLE);
//rest of code
Using the explanation Kenny gave I manage to do the following.
This example works using the front facing camera.
RectF rectF = new RectF(face.rect);
Matrix matrix = new Matrix();
matrix.setScale(1, 1);
matrix.postScale(view.getWidth() / 2000f, view.getHeight() / 2000f);
matrix.postTranslate(view.getWidth() / 2f, view.getHeight() / 2f);
matrix.mapRect(rectF);
The returned Rectangle by the matrix has all the right coordinates to draw into the canvas.
If you are using the back camera I think is just a matter of changing the scale to:
matrix.setScale(-1, 1);
But I haven't tried that.
The Camera.Face class returns the face bound coordinates using the image frame that the phone would save into its internal storage, rather than using the image displayed in the Camera Preview. In my case, the images were saved in a different manner from the camera, resulting in a incorrect mapping. I had to manually account for the discrepancy by taking the coordinates, rotating it counter clockwise 90 degrees and flipping it on the y-axis prior to scaling it to the canvas used for the custom view.
EDIT:
It would also appear that you can't change the way the face bound coordinates are returned by modifying the camera capture orientation using the Camera.Parameters.setRotation(int) method either.