I am working on an Android Application in which a 3d scene is displayed and the user should be able to select an area by clicking/tapping the screen. The scene is pretty much a planar (game) board on which different objects are placed.
Now, the problem is how do I get the clicked area on the board from the actual screen-space coordinates?
I was planning on using gluUnProject(), as I have access to (almost) all the necessary parameters. Unfortunately I am missing the winZ param, and cannot get the current depth as the touch event is occurring in a different thread than the GL-thread.
My new plan is to still use gluUnProject, but with a winZ of 0, and then project the resulting point onto the board (the board stretches from 0,0,0 to 10,0,10 in model space), However, I can't seem to figure out how to do this?
I would be very happy if anyone could help me out with the maths needed to do this (matrices were never my strongest side), or perhaps find a better solution.
To clarify; here is an image of what I want to do:
The red rectangle represent the device screen, the green x is the touch event and the black square is the board (grey subdivisions represent a square of one unit). I need to figure out where on the board the touch has happened (in this case it is in square 1,1).
As you are working in 2D basically already, (I presume you mean your 3D board stretches from 0,0,0 to 10,10,0 (x,y,z).) you could translate and interpolate/extrapolate the 2D/3D space coordinates from your screen space coordinates without the gluUnProject(). You will need your screen resolution, and to pick the resolution of the 3D space grid you wish to convert to. If both the screen and 3D space origins are aligned (0,0 screen space is at 0,0,0 3D space), and your screen dimensions are 320x240, using your existing 10x10 3D grid, then 320/10 = 32, and 240/10 = 24, thus the screen space size of a single 1x1 area is 32x24. So if the user presses on 162, 40, then the user is pressing within ( 5, 1, 0) (162/32 >= 5 but < 6, 40/24 >= 1 but < 2 ) in the 3D space. If you need greater resolution than this you can select a higher 3D space grid resolution (i.e. using 20 instead of 10). You don't need to update the GL matrix to use this factor. Though it may make it simpler in some ways, I'm sure from a modeling perspective you would have additional work to do. Just be aware for a factor like 20, 1,3 would be at (.5, 1.5, 0). If your screen and 3D space origins are not already aligned will need to translate the screen space coord prior to this. If 0,0 screen space is 10,10,0, you will need to take your screen resolution and subtract the current point from it, making 0,0 into 320, 240 in this example, our example point of 162, 40, would be 158 (320-158 == 162), 200 (240-200 == 40).
If you'd like an overview of the projection matrix and how that all works, which could help you understand where to put the screen space dimensions in the unproject matrix, read this chapter of the OpenGL red book. http://www.glprogramming.com/red/chapter03.html
Hope this helps, and good luck!
So, I managed to solve this by doing the following:
float[] clipPoint = new float[4];
int[] viewport = new int[]{0, 0, width, height};
//screenY/screenX are the screen-coordinates, y should be flipped:
screenY = viewport[3] - screenY;
//Calculate a z-value appropriate for the far clip:
float dist = 1.0f;
float z = (1.0f/clip[0] - 1.0f/dist)/(1.0f/clip[0]-1.0f/clip[1]);
//Use gluUnProject to create a 3d point in the far clip plane:
GLU.gluUnProject(screenX, screenY, z, vMatrix, 0, pMatrix, 0, viewport, 0, clipPoint, 0);
//Get a point representing the 'camera':
float eyeX = lookat[0] + eyeOffset[0];
float eyeY = lookat[1] + eyeOffset[1];
float eyeZ = lookat[2] + eyeOffset[2];
//Do some magic to calculate where the line between clipPoint and eye/camera would intersect the y-plane:
float dX = eyeX - clipPoint[0];
float dY = eyeY - clipPoint[1];
float dZ = eyeZ - clipPoint[2];
float resX = glu[0] - (dX/dY)*glu[1];
float resZ = glu[2] - (dZ/dY)*glu[1];
//resX and resZ is the wanted result.
Related
I have a question regarding transformations in OpenGL ES 2. I'm currently drawing a rectangle using triangle fans as depicted in the image below. The origin is located in its center, while its width and height are 0.6 and 2 respectively. I assume that these sizes are related to the model space. However, in order to maintain the ratio of height and width on a tablet or phone one has to do a projection that considers the proportion of the device lengths (again width and height). This is why I call orthoM(projectionMatrix, 0, -aspectRatio, aspectRatio, -1f, 1f, -1f, 1f);and the aspectRatio is given by float aspectRatio = (float) width / (float) height. This finally leads to the rectangle shown in the image below. Now, I would like to move the rectangle along the x-axis to the border of the screen. However, I was not able to come up with the correct calculation to do so, either I moved it too little or too much. So how would the calculation look like? Furtermore, I'm a little bit confused about the sizes given in the model space. What are the max and min values that can be achieved there?
Thanks a lot!
Vertex position of the rectangle are in world space. A way to do this it could be get the screen coordinates you want to move to and then transform them into world space.
For example:
If the screen is 300 x 200 and you are in the center 0,0 in world space (or 150, 100) in screen space). You want to translate to 300.
So the transformation should be screen_position to normalized device coordiantes and then multiply by inverseOf(projection matrix * view matrix) and divided by the w component.
Here it is explained for mouse that it is finally the same, just that you know the z because it is the one you used for your rectangle already (if it is on the plane x,y): OpenGL Math - Projecting Screen space to World space coords.
I wonder if there is an easy way to flip the y-coordinates when using perspective projection? The threads about the issue seem to focused on orthographic projection. I am translating my game based on Canvas to OpenGL ES 2.0 and have relatively complex collision detection. And a lot of syntax is based on the y-axis starts from top of the screen with 0 and ends on the bottom of thes screen for instance 2560
#Override
public void onSurfaceChanged(GL10 unused, int width, int height) {
game_width = width;
game_height = height;
GLES20.glViewport(0, 0, width, height);
// while the width will vary as per aspect ratio.
final float ratio = (float) width / height;
final float left = -ratio;
final float right = ratio;
final float bottom = -1.0f;
final float top = 1.0f;
final float near = 1f;
final float far = 40.0f;
Matrix.frustumM(mProjectionMatrix, 0, left, right, bottom, top, near, far);
}
There is very little difference using orthogonal or frustum matrix so the most simple answer would to simply swap the bottom and top parameters or even set them to whatever you need.
But to look into frustum a bit more:
What this method does is it creates a matrix that will scale the objects depending on the distance from near. It is designed so that an object at near is scaled by 1.0. So for instance if you put a rectangle with coordinates left, right, top, bottom as x and y then near as z and using no other matrix but the frustum the result will be exactly a full screen rectangle.
Objects that are closer to near will usually not be drawn and those further will be scaled linearly depending on all parameters but far. The far parameter effects nothing but where your objects will stop being drawn. So in most cases there is no difference if you put a very large far value but one very important; Effect of having a large far value will be precision of depth test. So when using depth buffer ensure that this value is as small as possible but still large enough to see all your objects.
In most cases we define frustum with a field of view as angle. You define constant near, far and fov from which the border parameters are then computed like right = tan(fov)*near*0.5 and top = tan(fov)*near*0.5*(viewHeight/viewWidth). These are just some examples though as there are many ways to define it.
In your case there is no reason not to define these values as you please. So having something like left = 0.0, right = width, bottom = height and top = 0.0. But then you still need to define near and far values which must be positive. Then if your objects are at 0.0 distance then they will all be clipped.
To avoid this it is best if you use a lookAt procedure which will generate another matrix that may define "camera" position in your scene. By simply putting it to z=-near you should see the objects exactly as with using orthographic projection. The problem now is that if you want to "zoom in" by putting the camera closer to the objects those objects will again not be drawn.
To achieve something like that you need to define some maximum scale for instance maxZoom = 10.0. What you would do then is divide all of the border parameters (top, left...) with that value. You would also apply this scale to the z value in your lookAt matrix to see the scene as not being zoomed.
So in general to flip the coordinates you may modify the border values or you may play with look at matrix. There are other ways as well but these are pretty standard. I hope this clears up a few things for you.
So here's the problem overview: I render a number of models using OpenGL ES in GvrView using the GvrView.StereoRenderer and I want to determine which exact model I'm looking at.
My idea is to reproject the screen coordinates back to the model space (discarding Z) and check if the point (lets call it Point) is in the range:
(ModelsMinX < Point.x < ModelsMaxX) and (ModelsMinY < Point.y < ModelsMaxY).
I was trying to use GLU.gluUnproject to get the initial coordinates. This function requires the current viewport and that's where the problems begin:
GvrView.StereoRenderer has a method .onDrawEye, which is called whenever there's something specific to one eye that should be setup before rendering (aka the view and the projection matrices should be acquired from the Eye instance). An Eye also has a method .getViewport which is supposed to return a viewport for the current eye, however the returned result is completely clear to me. More specifically, I'm developing on Nexus 6 (1440x2560 pixels) and .getViewport returns:
x = 0, y 0, width = 1280, height = 1440 // for the first eye
x = 1280, y 0, width = 1280, height = 1440 // for the second eye.
Now this is interesting.. Somehow I assumed two things about the current viewport:
width = 1440, height = 1280 (we are in the landscape mode after all);
the viewport size for each eye will be half the size of the whole viewport.
Hence, calling .gluUnproject on the middle point of the viewport:
GLU.gluUnProject(viewport.width / 2, viewport.height / 2, 0, mEyeViewMatrix, 0, mEyeProjectionMatrix, 0, new int[] {viewport.x, viewport.y, viewport.width, viewport.height}, 0, center, 0);
does not yield expected results, in fact, it gives me all 0s. I found this question (Determining exact eye view size), but the guy gets even stranger viewport values and it doesn't contain an answer..
So the question is - how to I get from the 'eye'-space coordinates to the model? And what those coordinates even are?
Here's the project's github for reference: https://github.com/bowlingforsoap/CardboardDataVisualizationJava.
Some other approaches I'm aware of:
In the treasurehunt demo they use an opposite way of doing things - they go from a model coordinate (0, 0, 0, 1) into the head view space using the HeadTransform to get the headView matrix (look for method .isLookingAtObject in https://github.com/googlevr/gvr-android-sdk/blob/master/samples/sdk-treasurehunt/src/main/java/com/google/vr/sdk/samples/treasurehunt/TreasureHuntActivity.java).
Using raycasting. I'm not sure this is going to help my cause, because after I get an observed object I would like to create an 'floating' activity which is going to contain information about it (I certainly don't wanna render that data through shaders).
Wheh! Boy, that's a lot of text. But yeah, it seems like a generic problem, yet I haven't found an easy/elegant/working solution to that.. Would appreciate any feedback.
I'm implementing 3d card flip animation for android (api > 14) and have an issue with big screen tablets (> 2048 dpi). During problem investigation i've come to the following basic block:
Tried to just transform a view (simple ImageView) using matrix and rotateY of camera by some angle and it works ok for angle < 60 and angle > 120 (transformed and displayed) but image disappears (just not displayed) when angle is between 60 and 120. Here is the code I use:
private void applyTransform(float degree)
{
float [] values = {1.0f, 0.0f, 0.0f, 0.0f, 1.0f, 0.0f, 0.0f, 0.0f, 1.0f};
float centerX = image1.getMeasuredWidth() / 2.0f;
float centerY = image1.getMeasuredHeight() / 2.0f;
Matrix m = new Matrix();
m.setValues(values);
Camera camera = new Camera();
camera.save();
camera.rotateY(degree);
camera.getMatrix(m);
camera.restore();
m.preTranslate(-centerX, -centerY); // 1 draws fine without these 2 lines
m.postTranslate(centerX, centerY); // 2
image1.setImageMatrix(m);
}
And here is my layout XML
<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent">
<ImageView
android:id="#+id/ImageView01"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_gravity="center"
android:src="#drawable/naponer"
android:clickable="true"
android:scaleType="matrix">
</ImageView>
</FrameLayout>
So I have the following cases:
works fine for any angle, any center point if running on small screens 800X480, 1024x720, etc...
works ok for angle < 60 and > 120 when running on big screen devices 2048x1536, 2560x1600...
works ok for any angle on any device if rotation not centered (matrix pre and post translations commented out )
fails (image disappears) when running on big screen device, rotation centered and angle is between 60 and 120 degrees.
Please tell what I'm doing wrong and advise some workaround... thank you!!!
This problem is caused by the camera distance used to calculate the transformation. While the Camera class itself doesn't say much about the subject, it is better explained in the documentation for the View.setCameraDistance() method (emphasis mine):
Sets the distance along the Z axis (orthogonal to the X/Y plane on
which views are drawn) from the camera to this view. The camera's
distance affects 3D transformations, for instance rotations around the
X and Y axis. (...)
The distance of the camera from the view plane can have an affect on
the perspective distortion of the view when it is rotated around the x
or y axis. For example, a large distance will result in a large
viewing angle, and there will not be much perspective distortion of
the view as it rotates. A short distance may cause much more
perspective distortion upon rotation, and can also result in some
drawing artifacts if the rotated view ends up partially behind the
camera (which is why the recommendation is to use a distance at
least as far as the size of the view, if the view is to be rotated.)
To be honest, I hadn't seen this particular effect (not drawing at all) before, but I suspected it could be related to this question related to perspective distortion I'd encountered in the past. :)
Therefore, the solution is to use the Camera.setLocation() method to ensure this doesn't happen.
An important distinction with the View.setCameraDistance() method is that the units are not the same, since setLocation() doesn't use pixels. While setCameraDistance() adjusts for density, setLocation() does not. Therefore, if you wanted to calculate an appropriate z-distance based on the view's dimensions, remember to adjust for density. For example:
float cameraDistance = Math.max(image1.getMeasuredHeight(), image1.getMeasuredWidth()) * 5;
float densityDpi = getResources().getDisplayMetrics().densityDpi;
camera.setLocation(0, 0, -cameraDistance / densityDpi);
Instead of using 12 lines to create rotation matrix, you could just implement this one in first line http://en.wikipedia.org/wiki/Rotation_matrix
Depending of effect you want, you might want to center image to axis you want to rotate around.
http://en.wikipedia.org/wiki/Transformation_matrix
Hmm for image disappearing, I would guess it has something to do with either memory (out of memory - although this would bring exception) or rounding problems. Maybe you could try increasing precision to double precision?
One thing that comes to mind is that cos(alpha) goes toward 0 when alpha goes toward PI/2. Other than that I don's see any correlation between angles and why it doesn't work for big images.
You need to adjust your Translate coordinates. When calculating the translation for your image you need to take image size into account too. When you perform matrix calculations you set android:scaleType="matrix" for your ImageView. This aligns your image at the top left corner by default. Then, when you apply your pre/post translation, your image may get off the bounds of your ImageView (especially if the ImageView is relatively large and your image is relatively small, like in case of beeg screen tablets).
The following translation results in the image being rotated around its center Y axis and keeps the image aligned to the top left corner:
m.preTranslate(-imageWidth/2, 0);
m.postTranslate(imageWidth/2, 0);
The following alternative results in the image being rotated around its center Y/X axises and aligns the image to the center of the ImageView:
m.preTranslate(-imageWidth/2, -imageHeight/2);
m.postTranslate(centerX, centerY);
If your image is a bitmap you can use intrinsic width/height:
Drawable drawable = image1.getDrawable();
imageHeight = drawable.getIntrinsicHeight();
imageWidth = drawable.getIntrinsicWidth();
I'm hoping someone can help me out. I'm making an image manipulation app, and I found I needed a better way to load in large images.
My plan, is to iterate through "hypothetical" pixels of an image (a "for loop" that covers width/height of the base image, so each iteration represents a pixel), scale/translate/rotate that pixels position relative to the view, then use this information to determine which pixels are being displayed in the view itself, then use a combination of BitmapRegionDecoder and BitmapFactory.Options to load in only the section of image that the output actually needs rather than a full (even if scaled) image.
So far I seem to have covered scale of the image and translation properly, but I can't seem to figure out how to calculate rotation. Since it's not a real Bitmap pixel I can't use Matrix.rotate =( Here is the image translations in the onDraw of the view, imgPosX and imgPosY hold the center point of the image:
m.setTranslate(-userImage.getWidth() / 2.0f, -userImage.getHeight() / 2.0f);
m.postScale(curScale, curScale);
m.postRotate(angle);
m.postTranslate(imgPosX, imgPosY);
mCanvas.drawBitmap(userImage.get(), m, paint);
and here is the math so far of how I'm trying to determine if an images pixel is on the screen:
for(int j = 0;j < imageHeight;j++) {
for(int i = 0;i < imageWidth;i++) {
//image starts completely center in view, assume image is original size for simplicity
//this is the original starting position for each pixel
int x = Math.round(((float) viewSizeWidth / 2.0f) - ((float) newImageWidth / 2.0f) + i);
int y = Math.round(((float) viewSizeHeight / 2.0f) - ((float) newImageHeight / 2.0f) + j);
//first we scale the pixel here, easy operation
x = Math.round(x * imageScale);
y = Math.round(y * imageScale);
//now we translate, we do this by determining how many pixels
//our images x/y coordinates have differed from it's original
//starting point, imgPosX and imgPosY in the view start in center
//of view
x = x + Math.round((imgPosX - ((float) viewSizeWidth / 2.0f)));
y = y + Math.round((imgPosY - ((float) viewSizeHeight / 2.0f)));
//TODO need rotation here
}
}
so, assuming my math up until rotation is correct (probably not but it appears to be working so far), how would I then calculate the rotation from that pixels position? I've tried other similar questions like:
Link 1
Link 2
Link 3
without using rotation the pixels I expect to actually be on the screen are represented (I made text file that outputs the results in 1's and 0's so I can have a visual representation of whats on the screen), but with the formula found in those questions the information isn't what is expected. (Scenario: I've rotated an image so only the top left corner is visible in the view. Using the info from Here to rotate the pixel, I should expect to see a triangular set of 1's in the upper left corner of the output file, but that's not the case)
So, how would I calculate a a pixels position after rotation without using the Android matrix? But still get the same results.
And if I've just messed it up entirely my apologies =( Any help would be appreciated, this project has gone on for so long and I want to finally be done lol
If you need any more information I will provide as much as I possibly can =) Thank you for your time
I realize this question is particularly difficult so I will be posting a bounty as soon as SO allows.
You do not need to create your own Matrix, use the existing one.
http://developer.android.com/reference/android/graphics/Matrix.html
You can map bitmap coordinates to screen coordinates by using
float[] coords = {x, y};
m.mapPoints(coords);
float sx = coords[0];
float sy = coords[1];
If you want to map screen to bitmap coordinates, you can create the inverse matrix
Matrix inverse = new Matrix(m);
inverse.inverse();
inverse.mapPoints(...)
I think your overall approach is going to be slow, as doing the pixel manipulation on the CU from Java has a lot of overhead. When drawing bitmaps normally, the pixel manipulation is done on the GPU.