Develop Reference

Changing camera's forward vector has strange results in Android openGL

I'm trying to create a 3D Android game using the OpenGL library. From what I can tell, the Android platform (or OpenGL) doesn't provide a camera object, so I made my own. I managed to create a square that is drawn on the screen. I also managed to get the camera to move in all 3 directions without issue. The problem I'm having is when I turn the camera either in the x or y axis, the square is displayed in a very strange way. It seems highly distorted its perspective. Here's some pictures
Camera at origin looking forward: Position [0,0,0.55f], Forward [0,0,-1], Up [0,1,0]
Camera moved to the right: Position [3,0,0.55f], Forward [0,0,-1], Up [0,1,0]
Camera turned slightly to the right: Position [0,0,0.55f], Forward [0.05f,0,-1], Up [0,1,0]
I'm not sure where the problem is getting generated. Here are the vertices of the square:
static float vertices[] = {
// Front face (CC order)
-1.0f, 1.0f, 1.0f, // top left
-1.0f, -1.0f, 1.0f, // bottom left
1.0f, -1.0f, 1.0f, // bottom right
1.0f, 1.0f, 1.0f, // top right
};
From what I've read, my matrix multiplications are in the correct order. I pass the Matrices to the vertex shader and do the multiplications there:
private final String vertexShaderCode =
"uniform mat4 uVMatrix;" +
"uniform mat4 uMMatrix;" +
"uniform mat4 uPMatrix;" +
"attribute vec4 aVertexPosition;" + // passed in
"attribute vec4 aVertexColor;" +
"varying vec4 vColor;" +
"void main() {" +
" gl_Position = uPMatrix * uVMatrix * uMMatrix * aVertexPosition;" +
" vColor = aVertexColor;" + // pass the vertex's color to the pixel shader
"}";
The Model matrix just moves it to the origin and makes it scale 1:
Matrix.setIdentityM(ModelMatrix, 0);
Matrix.translateM(ModelMatrix, 0, 0, 0, 0);
Matrix.scaleM(ModelMatrix, 0, 1, 1, 1);
I use my Camera object to update the View Matrix:
Matrix.setLookAtM(ViewMatrix, 0,
position.x, position.y, position.z,
position.x + forward.x, position.y + forward.y, position.z + forward.z,
up.x, up.y, up.z);
Here is my ProjectionMatrix:
float ratio = width / (float) height;
Matrix.frustumM(ProjectionMatrix, 0, -ratio, ratio, -1, 1, 0.01f, 10000f);
What am I missing?

With the parameters you pass to frustrumM():
Matrix.frustumM(ProjectionMatrix, 0, -ratio, ratio, -1, 1, 0.01f, 10000f);
you have an extremely strong perspective. This is the reason why the geometry looks very distorted as soon as you rotate it just slightly.
The left, right, bottom, and top values are distances measured at the depth of the near clip plane. So for example for your top value of 1.0, with the near value at 0.01, the top plane of the view volume will move at distance of 1.0 away from the viewing direction at a forward distance of 0.01.
Doing the math, you get atan(top / near) = atan(1.0 / 0.01) = atan(100.0) = 89.42 degrees for half the vertical view angle, or 178.85 degrees for the whole view angle, which corresponds to an extreme fish eye lense on a camera, covering almost the whole space in front of the camera.
To use a more sane level of perspective, you can calculate the values based on the desired view angle. With alpha being the vertical view angle:
float near = 0.01f;
float top = tan(0.5f * alpha) * near;
float right = top * ratio;
Matrix.frustumM(ProjectionMatrix, 0, -right, right, -top, top, near, 10000f);
Start with view angles in the range of 45 to 60 degrees for a generally pleasing amount of perspective. And remember that the tan() function used above takes angles in radian, so you'll have to convert it first if your original angle is in degrees.
Or, if you're scared of math, you can always use perspectiveM() instead. ;)

Depending on your setup, your vertex buffer may be wrong . Your vertices array is a vec3. And the attribute position in your shader is a vec4.
Also you projection matrix is strange , depending on the aspect ratio you want you'll get bizarre results.
You should use perspectiveM instead.

How to display a scaled down GL ES texture without clamp or repeat on android

I am currently rendering a camera preview using GL ES 2.0 on android to a SurfaceTexture, rendering it with opengl, then transferring it to a media codec's input surface to for recording. It is displayed to the user in a surface view and by setting that surface view's aspect ratio the camera preview is not distorted based on screen size.
The recording is in portrait, but at some point the incoming texture will start coming in landscape, at which point I'd like to zoom out and display it as a "movie" stretched wide to fit to the edge of the screen horizonatally with black bars on the top and bottom to maintain the aspect ratio of the texture.
The drawing code in onDrawFrame is pretty simple. The link has the rest of the setup code for shaders and the like but it's just setting up a triangle strip to draw.
private final float[] mTriangleVerticesData = {
// X, Y, Z, U, V
-1.f, -1.f, 0, 0.f, 0.f,
1.f, -1.f, 0, 1.f, 0.f,
-1.f, 1.f, 0, 0.f, 1.f,
1.f, 1.f, 0, 1.f, 1.f,
};
public static final String VERTEX_SHADER =
"uniform mat4 uMVPMatrix;\n" +
"uniform mat4 uSTMatrix;\n" +
"attribute vec4 aPosition;\n" +
"attribute vec4 aTextureCoord;\n" +
"varying vec2 vTextureCoord;\n" +
"void main() {\n" +
" gl_Position = uMVPMatrix * aPosition;\n" +
" vTextureCoord = (uSTMatrix * aTextureCoord).xy;\n" +
"}\n";
private float[] mMVPMatrix = new float[16];
private float[] mSTMatrix = new float[16];
public TextureManager() {
mTriangleVertices = ByteBuffer.allocateDirect(
mTriangleVerticesData.length * FLOAT_SIZE_BYTES)
.order(ByteOrder.nativeOrder()).asFloatBuffer();
mTriangleVertices.put(mTriangleVerticesData).position(0);
mTriangleHalfVertices = ByteBuffer.allocateDirect(
mTriangleVerticesHalfData.length * FLOAT_SIZE_BYTES)
.order(ByteOrder.nativeOrder()).asFloatBuffer();
mTriangleHalfVertices.put(mTriangleVerticesHalfData).position(0);
Matrix.setIdentityM(mSTMatrix, 0);
}
onDrawFrame(){
mSurfaceTexture.getTransformMatrix(mSTMatrix);
GLES20.glUseProgram(mProgram);
checkGlError("glUseProgram");
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, 0);
GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, mTextureID);
mTriangleVertices.position(TRIANGLE_VERTICES_DATA_POS_OFFSET);
GLES20.glVertexAttribPointer(maPositionHandle, 3, GLES20.GL_FLOAT, false,
TRIANGLE_VERTICES_DATA_STRIDE_BYTES, mTriangleVertices);
checkGlError("glVertexAttribPointer maPosition");
GLES20.glEnableVertexAttribArray(maPositionHandle);
checkGlError("glEnableVertexAttribArray maPositionHandle");
mTriangleVertices.position(TRIANGLE_VERTICES_DATA_UV_OFFSET);
GLES20.glVertexAttribPointer(maTextureHandle, 2, GLES20.GL_FLOAT, false,
TRIANGLE_VERTICES_DATA_STRIDE_BYTES, mTriangleVertices);
checkGlError("glVertexAttribPointer maTextureHandle");
GLES20.glEnableVertexAttribArray(maTextureHandle);
checkGlError("glEnableVertexAttribArray maTextureHandle");
Matrix.setIdentityM(mMVPMatrix, 0);
GLES20.glUniformMatrix4fv(muMVPMatrixHandle, 1, false, mMVPMatrix, 0);
GLES20.glUniformMatrix4fv(muSTMatrixHandle, 1, false, mSTMatrix, 0);
GLES20.glDrawArrays(GLES20.GL_TRIANGLE_STRIP, 0, 4);
checkGlError("glDrawArrays");
GLES20.glFinish();`
}
Things I've tried that haven't quite worked: Scaling mMVPMatrix or mSTMatrix to zoom in. I can zoom in, so I can get the "center slice" of the landscape video to display without distortion, but this cuts off a huge 40% portion of the video, so it isn't a great solution. Zooming out by scaling these matricies causes the texture to repeat the pixel on the edge because of the clamp to edge behavior.
Halving the x,y,z parts of mTriangleVerticesData gives some of the desired behavior as seen in the screenshot below, exact aspect ratio aside. The center part of the picture is halved and centered, as expected. However, the texture is repeated to the left, right, and bottom, and there is distortion to the top left. What I want is the center to be as it is, with black/nothing surrounding it.
I could scale out then translate mMVPMatrix or mSTMatrix and then change my shader to display black for anything outside (0,1) but eventually I want to overlay multiple textures on top of one another, like a full size background and partial size foreground texture. To do this I must eventually figure out how to only display a texture in a portion of the available space, not just manipulate the texture so it looks like that's what's happening.
Thanks for reading all that. Any help, suggestions, or wild guesses are appreciated.

The repeated image chunks look like GPU tiling artifacts, not texture repeating. Add a glClear() call to erase the background.

It would seem what you are looking for is a "fit" system to get the correct frame of your element. It would mean for instance you are having a 100x200 image and you want to display it in a frame of 50x50. The result should then be seeing the whole image in rectangle (25, 0, 25, 50). So the resulting frame must respect the image ratio (25/50 = 100/200) and original frame boundaries must be respected.
To achieve this generally you need to compare the image ratio and the target frame ratio: imageRatio = imageWidth/imageHeight and frameRatio = frameWidth/frameHeight. Then if image ratio is larger then the frame ratio it means you need black borders on top and bottom while if the frame ratio is larger then you will see black borders on left and right side.
So to compute the target frame:
imageRatio = imageWidth/imageHeight
frameRatio = frameWidth/frameHeight
if(imageRatio > frameRatio) {
targetFrame = {0, (frameHeight-frameWidth/imageRatio)*.5, frameWidth, frameWidth/imageRatio} // frame as: {x, y, width, height}
}
else {
targetFrame = {(frameWidth-frameHeight*imageRatio)*.5, 0, frameHeight*imageRatio, frameHeight} // frame as: {x, y, width, height}
}
In your case the image width and height are the ones received from the stream; frame width and height are from your target frame which seems to be a result from matrices but for full screen case that would simply be values from glOrtho if you use it. The target frame should then be used to construct the vertices positions so you get exactly correct vertex data to display the full texture.
I see you use matrices to do all the computation in your case and the same algorithm may be used to be converted to matrix but I discourage you to do so. You seem to be over-abusing matrices which makes your code completely unmaintainable. I suggest in your case you keep to "ortho" projection matrix, use frames to draw textures and only use matrix scale and translations where it makes sense to do so.

Doesn't rotate after applying transformation matrix [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Is Google’s Android OpenGL tutorial teaching incorrect linear algebra?
Learning OpenGL ES 2.0 on Android. Using Emulator, running Android 4.1.
Copied and pasted snippets from Android Developer Site / OpenGL
Updated onDrawFrame method. Pasted below.
Added Matrix.setIdentityM(mRotationMatrix, 0) since it's was a Null Matrix.
Changed mAngle to angle (line 16).
public void onDrawFrame(GL10 unused) {
// Redraw background color
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
// Set the camera position (View matrix)
Matrix.setLookAtM(mVMatrix, 0, 0, 0, -3, 0f, 0f, 0f, 0f, 1.0f, 0.0f);
// Calculate the projection and view transformation
Matrix.multiplyMM(mMVPMatrix, 0, mProjMatrix, 0, mVMatrix, 0);
// Create a rotation transformation for the triangle
long time = SystemClock.uptimeMillis() % 4000L;
float angle = 0.090f * ((int) time);
Matrix.setIdentityM(mRotationMatrix, 0); //added
Matrix.setRotateM(mRotationMatrix, 0, angle, 0, 0, 1.0f); //changed
// Combine the rotation matrix with the projection and camera view
Matrix.multiplyMM(mMVPMatrix, 0, mRotationMatrix, 0, mMVPMatrix, 0);
// Draw shape
mTriangle.draw(mMVPMatrix);
}
And Commented out setRenderMode(RENDERMODE_WHEN_DIRTY);
Yet the Triangle drawn did not rotated. Where did I go wrong?

Thanks to this question here. I learnt to solve this.
Just edit the Vertex Shader Code. uMVPMatrix is important without this the projection is not applied.
private final String vertexShaderCode = "attribute vec4 vPosition;"
+"uniform mat4 uMVPMatrix;"
+ "void main() {" + " gl_Position = uMVPMatrix * vPosition;" + "}";

Android OpenGL weirdness with the setLookAtM method

As a beginner to android and openGL 2.0 es, I'm testing simple things and see how it goes.
I downloaded the sample at http://developer.android.com/training/graphics/opengl/touch.html .
I changed the code to check if I could animate a rotation of the camera around the (0,0,0) point, the center of the square.
So i did this:
public void onDrawFrame(GL10 unused) {
// Draw background color
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
// Set the camera position (View matrix)
long time = SystemClock.uptimeMillis() % 4000L;
float angle = ((float) (2*Math.PI)/ (float) 4000) * ((int) time);
Matrix.setLookAtM(mVMatrix, 0, (float) (3*Math.sin(angle)), 0, (float) (3.0f*Math.cos(angle)), 0 ,0, 0, 0f, 1.0f, 0.0f);
// Calculate the projection and view transformation
Matrix.multiplyMM(mMVPMatrix, 0, mProjMatrix, 0, mVMatrix, 0);
// Draw square
mSquare.draw(mMVPMatrix);
}
I expected the camera to look always to the center of the square (the (0,0,0) point) but that's not what happens. The camera is indeed rotating around the square but the square does not stay in the center of the screen.. instead it is moving along the X axis...:
I also expected that if we gave the eyeX and eyeY the same values as centerX and centerY,like this:
Matrix.setLookAtM(mVMatrix, 0, 1, 1, -3, 1 ,1, 0, 0f, 1.0f, 0.0f);
the square would keep it's shape (I mean, your field of vision would be dragged but along a plane which would be paralel to the square), but that's also not what happens:
This is my projection matrix:
float ratio = (float) width / height;
// this projection matrix is applied to object coordinates
// in the onDrawFrame() method
Matrix.frustumM(mProjMatrix, 0, -ratio, ratio, -1, 1, 2, 7);
What is going on here?

Looking at the source code to the example you downloaded, I can see why you're having that problem, it has to do with the order of the matrix multiplication.
Typically in OpenGL source you see matrices set up such that
transformed vertex = projMatrix * viewMatrix * modelMatrix * input vertex
However in the source example program that you downloaded, their shader is setup like this:
" gl_Position = vPosition * uMVPMatrix;"
With the position on the other side of the matrix. You can work with OpenGL in this way, but it requires that you reverse the lhs/rhs of your matrix multiplications.
Long story short, in your case, you should change your shader to read:
" gl_Position = uMVPMatrix * vPosition;"
and then I believe you will get the expected behavior.

Is Google's Android OpenGL tutorial teaching incorrect linear algebra?

After helping another user with a question regarding the Responding to Touch Events Android tutorial, I downloaded the source code, and was quite baffled by what I saw. The tutorial seems to not be able to decide whether it wants to use row vectors or column vectors, and it looks all mixed up to me.
On the Android Matrix page, they claim that their convention is column-vector/column-major, which is typical of OpenGL.
Am I right, or is there something I am missing? Here are the relevant bits of it:
Start out by creating a MVPMatrix by multiplying mProjMatrix * mVMatrix. So far so good.
// Set the camera position (View matrix)
Matrix.setLookAtM(mVMatrix, 0, 0, 0, -3, 0f, 0f, 0f, 0f, 1.0f, 0.0f);
// Calculate the projection and view transformation
Matrix.multiplyMM(mMVPMatrix, 0, mProjMatrix, 0, mVMatrix, 0)
Next they are appending a rotation to the left hand side of the MVPMatrix? This seems a little weird.
// Create a rotation for the triangle
Matrix.setRotateM(mRotationMatrix, 0, mAngle, 0, 0, -1.0f);
// Combine the rotation matrix with the projection and camera view
Matrix.multiplyMM(mMVPMatrix, 0, mRotationMatrix, 0, mMVPMatrix, 0)
Uploading in non-transposed order.
GLES20.glUniformMatrix4fv(mMVPMatrixHandle, 1, false, mvpMatrix, 0);
Finally in their shader, a vector*matrix multiplication?
// the matrix must be included as a modifier of gl_Position
" gl_Position = vPosition * uMVPMatrix;"
Adding this all together, we get:
gl_Position = vPosition * mRotation * mProjection * mView;
Which is not correct by any stretch of my imagination. Is there any explanation that I'm not seeing as to what's going on here?

As the guy who wrote that OpenGL tutorial, I can confirm that the example code is incorrect. Specifically, the order of the factors in the shader code should be reversed:
" gl_Position = uMVPMatrix * vPosition;"
As to the application of the rotation matrix, the order of the factors should also be reversed so that the rotation is the last factor. The rule of thumb is that matrices are applied in right-to-left order, and the rotation is applied first (it's the the "M" part of "MVP"), so it needs to be the rightmost operand. Furthermore, you should use a scratch matrix for this calculation, as recommended by Ian Ni-Lewis (see his more complete answer, below):
float[] scratch = new float[16];
// Combine the rotation matrix with the projection and camera view
Matrix.multiplyMM(scratch, 0, mMVPMatrix, 0, mRotationMatrix, 0);
Thanks for calling attention to this problem. I'll get the training class and sample code fixed as soon as I can.
Edit: This issue has now been corrected in the downloadable sample code and the OpenGL ES training class, including comments on the correct order of the factors. Thanks for the feedback, folks!

The tutorial is incorrect, but many of the mistakes either cancel each other out or are not obvious in this very limited context (fixed camera centered at (0,0), rotation around Z only). The rotation is backwards, but otherwise it kind of looks right. (To see why it's wrong, try a less trivial camera: set the eye and lookAt to y=1, for instance.)
One of the things that made this very hard to debug is that the Matrix methods don't do any alias detection on their inputs. The tutorial code makes it seem like you can call Matrix.multiplyMM with the same matrix used as both an input and the result. This isn't true. But because the implementation multiplies a column at a time, it's far less obvious that something is wrong if the right hand side is reused (as in the current code, where mMVPMatrix is the rhs and the result) than if the left hand side is reused. Each column on the left is read before the corresponding column in the result is written, so the output will be correct even if the LHS is overwritten. But if the right-hand side is the same as the result, then its first column will be overwritten before it's finished being read.
So the tutorial code is at a sort of local maximum: it seems like it works, and if you change any one thing, it breaks spectacularly. Which leads one to believe that wrong as it looks, it might just be correct. ;-)
Anyway, here's some replacement code that gets what I think is the intended result.
Java code:
#Override
public void onDrawFrame(GL10 unused) {
float[] scratch = new float[16];
// Draw background color
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
// Set the camera position (View matrix)
Matrix.setLookAtM(mVMatrix, 0, 0, 0, -3, 0f, 0f, 0f, 0f, 1.0f, 0.0f);
// Calculate the projection and view transformation
Matrix.multiplyMM(mMVPMatrix, 0, mProjMatrix, 0, mVMatrix, 0);
// Draw square
mSquare.draw(mMVPMatrix);
// Create a rotation for the triangle
Matrix.setRotateM(mRotationMatrix, 0, mAngle, 0, 0, 1.0f);
// Combine the rotation matrix with the projection and camera view
Matrix.multiplyMM(scratch, 0, mMVPMatrix, 0, mRotationMatrix, 0);
// Draw triangle
mTriangle.draw(scratch);
}
Shader code:
gl_Position = uMVPMatrix * vPosition;
NB: these fixes make the projection correct, but they also reverse the direction of rotation. That's because the original code applied the transformations in the wrong order. Think of it this way: instead of rotating the object clockwise, it was rotating the camera counterclockwise. When you fix the order of operations so that the rotation is applied to the object instead of the camera, then the object starts going counterclockwise. It's not the matrix that's wrong; it's the angle that was used to create the matrix.
So to get the 'correct' result, you also need to flip the sign of mAngle.

I solved this problem as follows:
#Override
public void onDrawFrame(GL10 unused) {
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
Matrix.setLookAtM(mViewMatrix, 0, 0, 0, -1f, 0f, 0f, 0f, 0f, 1.0f, 0.0f);
Matrix.setRotateM(mModelMatrix, 0, mAngle, 0, 0, 1.0f);
Matrix.translateM(mModelMatrix, 0, 0.4f, 0.0f, 0);
mSquare.draw(mProjMatrix,mViewMatrix,mModelMatrix);
}
#Override
public void onSurfaceChanged(GL10 unused, int width, int height) {
...
Matrix.frustumM(mProjMatrix, 0, -ratio, ratio, -1, 1, 1, 99);
}
class Square {
private final String vertexShaderCode =
"uniform mat4 uPMatrix; \n" +
"uniform mat4 uVMatrix; \n" +
"uniform mat4 uMMatrix; \n" +
"attribute vec4 vPosition; \n" +
"void main() { \n" +
" gl_Position = uPMatrix * uVMatrix * uMMatrix * vPosition; \n" +
"} \n";
...
public void draw(float[] mpMatrix,float[] mvMatrix,float[]mmMatrix) {
...
mPMatrixHandle = GLES20.glGetUniformLocation(mProgram, "uPMatrix");
mVMatrixHandle = GLES20.glGetUniformLocation(mProgram, "uVMatrix");
mMMatrixHandle = GLES20.glGetUniformLocation(mProgram, "uMMatrix");
GLES20.glUniformMatrix4fv(mPMatrixHandle, 1, false, mpMatrix, 0);
GLES20.glUniformMatrix4fv(mVMatrixHandle, 1, false, mvMatrix, 0);
GLES20.glUniformMatrix4fv(mMMatrixHandle, 1, false, mmMatrix, 0);
...
}
}

I’m working on the same issue and that’s what I found:
I believe that Joe’s sample is CORRECT,
including
the order of the factors in the shader code:
gl_Position = vPosition * uMVPMatrix;
To verify it just try to rotate the triangle with reversed factors order,
it will stretch the triangle to vanishing point at 90 degrees.
The real problem seems to be in setLookAtM function.
In Joe’s sample parameters are:
Matrix.setLookAtM(mVMatrix, 0,
0f, 0f,-3f, 0f, 0f, 0f, 0f, 1f, 0f );
which is perfectly logical as well.
However, the resulting view matrix looks weird to me:
-1 0 0 0
0 1 0 0
0 0 -1 0
0 0 -3 1
As we can see, this matrix will invert X coordinate,
since the first member is –1,
which will lead to left/right flip on the screen.
It will also reverse Z-order, but let's focus on X coordinate here.
I think that setLookAtM function is also working correctly.
However, since Matrix class is NOT a part of OpenGL,
it can use some other coordinates system,
for example - regular screen coordinates with Y axis pointing down.
This is just a guess, I didn’t really verify that.
Possible solutions:
We can build desirable view matrix manually,
the code is:
Matrix.setIdentityM(mVMatrix,0);
mVMatrix[14] = -3f;
OR
we can try to trick setLookAtM function by giving it
reversed camera coordinates:
0, 0, +3 (instead of –3).
Matrix.setLookAtM(mVMatrix, 0,
0f, 0f, 3f, 0f, 0f, 0f, 0f, 1f, 0f );
The resulting view matrix will be:
1 0 0 0
0 1 0 0
0 0 1 0
0 0 -3 1
That’s exactly what we need.
Now camera behaves as expected,
and sample works correctly.

No other suggestions worked for me using the current updated Android example code except for the following when trying to move the triangle.
The following link contains the answer. Took over a day to locate it. Posting here to help others as I seen this post many times. OpenGL ES Android Matrix Transformations