Develop Reference

what do texture2D().r and texture2D().a mean?

I am using OpenGL ES in Android programming, when I transform YUV(NV21) to RGB in shader, like:
vec3 yuv = vec3(
(texture2D(u_TextureY, vTextureCoord).r - 0.0625),
texture2D(u_TextureUV, vTextureCoord).a - 0.5,
texture2D(u_TextureUV, vTextureCoord).r - 0.5
);
then I'll get YUV data that seperating from u_TextureY and u_TextureUV.
I know that NV21 format is like: YYYYYY...UVUV... BUT how can I transform YUYV422 to RGB?
So, my problem is what do "r" and "a" mean in texture2D(u_TextureY, vTextureCoord).r and .a ? then I can find the way to do YUYV422->RGB.

The return type of texture2D is vec4. In GLSL the components of the vector can be separately accessed:
See the The OpenGL Shading Language specification:
5.5 Vector and Scalar Components and Length
The names of the components of a vector or scalar are denoted by a single letter. As a notational convenience, several letters are associated with each component based on common usage of position, color or texture coordinate vectors. The individual components can be selected by following the variable
name with period ( . ) and then the component name.
The component names supported are:
{x, y, z, w} Useful when accessing vectors that represent points or normals
{r, g, b, a} Useful when accessing vectors that represent colors
{s, t, p, q} Useful when accessing vectors that represent texture coordinates
The order of the components can be different to swizzle them, or replicated:
vec4 pos = vec4(1.0, 2.0, 3.0, 4.0);
vec4 swiz= pos.wzyx; // swiz = (4.0, 3.0, 2.0, 1.0)
vec4 dup = pos.xxyy; // dup = (1.0, 1.0, 2.0, 2.0)
float f = 1.2;
vec4 dup = f.xxxx; // dup = (1.2, 1.2, 1.2, 1.2)
This means, that .r gives the 1st component of the vec4 and .a gives the 4th component of the vec4.

OK, I have solved it.
The answer from #Rabbid76 is great, so we know .r and .a are from {r, g, b, a}.
My question is not just what is .r.a, it's why is .r.a .
I solved the problem starting from this answer enter link description here
glTexImage2D(GL20.GL_TEXTURE_2D, 0, GL20.GL_LUMINANCE,
width, height, 0, GL20.GL_LUMINANCE, GL20.GL_UNSIGNED_BYTE, buffer_y);
by setting GL_LUMINANCE, OpenGL puts this byte into R,G and B components of the texture. So we can use .r to get Y values, also .g or .b.
glTexImage2D(GL20.GL_TEXTURE_2D, 0, GL20.GL_LUMINANCE_ALPHA,
width/2, height/2, 0, GL20.GL_LUMINANCE_ALPHA, GL20.GL_UNSIGNED_BYTE,
buffer_uv);
by setting GL_LUMINANCE_ALPHA, OpenGL puts the first byte 'V' into r/g/b components, and puts the second byte 'U' into a component. So we can use .a to get U, and .r to get V(.g or .b is ok).
As we know that NV21 format is like YYYY...UVUV... It seems like 'U' is the first byte in UV texture.
buffer_uv = ByteBuffer.allocateDirect(width * height * 4)
.order(ByteOrder.nativeOrder());
buffer_uv is order with ByteOrder.nativeOrder(), retrieves the native byte order of the underlying platform. From enter link description here
That means the UV values(...UVUV...) are read from native buffer by native byte order like ...VUVU... , it's reverse.

how to check ray intersection with object in ARCore

Is there a way to check if I touched the object on the screen ? As I understand the HitResult class allows me to check if I touched the recognized and maped surface. But I want to check this I touched the object that is set on that surface.

ARCore doesn't really have a concept of an object, so we can't directly provide that. I suggest looking at ray-sphere tests for a starting point.
However, I can help with getting the ray itself (to be added to HelloArActivity):
/**
* Returns a world coordinate frame ray for a screen point. The ray is
* defined using a 6-element float array containing the head location
* followed by a normalized direction vector.
*/
float[] screenPointToWorldRay(float xPx, float yPx, Frame frame) {
float[] points = new float[12]; // {clip query, camera query, camera origin}
// Set up the clip-space coordinates of our query point
// +x is right:
points[0] = 2.0f * xPx / mSurfaceView.getMeasuredWidth() - 1.0f;
// +y is up (android UI Y is down):
points[1] = 1.0f - 2.0f * yPx / mSurfaceView.getMeasuredHeight();
points[2] = 1.0f; // +z is forwards (remember clip, not camera)
points[3] = 1.0f; // w (homogenous coordinates)
float[] matrices = new float[32]; // {proj, inverse proj}
// If you'll be calling this several times per frame factor out
// the next two lines to run when Frame.isDisplayRotationChanged().
mSession.getProjectionMatrix(matrices, 0, 1.0f, 100.0f);
Matrix.invertM(matrices, 16, matrices, 0);
// Transform clip-space point to camera-space.
Matrix.multiplyMV(points, 4, matrices, 16, points, 0);
// points[4,5,6] is now a camera-space vector. Transform to world space to get a point
// along the ray.
float[] out = new float[6];
frame.getPose().transformPoint(points, 4, out, 3);
// use points[8,9,10] as a zero vector to get the ray head position in world space.
frame.getPose().transformPoint(points, 8, out, 0);
// normalize the direction vector:
float dx = out[3] - out[0];
float dy = out[4] - out[1];
float dz = out[5] - out[2];
float scale = 1.0f / (float) Math.sqrt(dx*dx + dy*dy + dz*dz);
out[3] = dx * scale;
out[4] = dy * scale;
out[5] = dz * scale;
return out;
}
If you're calling this several times per frame see the comment about the getProjectionMatrix and invertM calls.

Apart from Mouse Picking with Ray Casting, cf. Ian's answer, the other commonly used technique is a picking buffer, explained in detail (with C++ code) here
The trick behind 3D picking is very simple. We will attach a running
index to each triangle and have the FS output the index of the
triangle that the pixel belongs to. The end result is that we get a
"color" buffer that doesn't really contain colors. Instead, for each
pixel which is covered by some primitive we get the index of this
primitive. When the mouse is clicked on the window we will read back
that index (according to the location of the mouse) and render the
select triangle red. By combining a depth buffer in the process we
guarantee that when several primitives are overlapping the same pixel
we get the index of the top-most primitive (closest to the camera).
So in a nutshell:
Every object's draw method needs an ongoing index and a boolean for whether this draw renders the pixel buffer or not.
The render method converts the index into a grayscale color and the scene is rendered
After the whole rendering is done, retrieve the pixel color at the touch position GL11.glReadPixels(x, y, /*the x and y of the pixel you want the colour of*/). Then translate the color back to an index and the index back to an object. Voilà, you have your clicked object.
To be fair, for a mobile usecase you should probably read a 10x10 rectangle, iterate trough it and pick the first found non-background color - because touches are never that precise.
This approach works independently of the complexity of your objects

Converting pixel co-ordinates to normalized co-ordinates at draw time in OpenGL 3.0

I am drawing a triangle in OpenGL like:
MyGLRenderer( )
{
fSampleVertices = ByteBuffer.allocateDirect( fSampleVerticesData.length * 4 )
.order ( ByteOrder.nativeOrder( ) ).asFloatBuffer( );
fSampleVertices.put( fSampleVerticesData ).position ( 0 );
Log.d( TAG, "MyGLRender( )" );
}
private FloatBuffer fSampleVertices;
private final float[] fSampleVerticesData =
{ .8f, .8f, 0.0f, -.8f, .8f, 0.0f, -.8f, -.8f, 0.0f };
public void onDrawFrame( GL10 unused )
{
GLES30.glViewport ( 0, 0, mWidth, mHeight );
GLES30.glClear ( GLES30.GL_COLOR_BUFFER_BIT );
GLES30.glUseProgram ( dProgramObject1 );
GLES30.glVertexAttribPointer ( 0, 3, GLES30.GL_FLOAT, false, 0, fSampleVertices );
GLES30.glEnableVertexAttribArray ( 0 );
GLES30.glDrawArrays( GLES30.GL_TRIANGLES, 0, 3 );
//Log.d( TAG, "onDrawFrame( )" );
}
So since I have experimented with the co-ordinates it doesn't take long to figure out that the visible area of the screen
is between -1,1. So then the triangle takes up 80% of the screen. As well I have determined that the pixel dimensions of my
GLSurfaceView are 2560 in width and 1600 in height.
So then given a triangle with these pixel based co-ordinates (fBoardOuter):
1112.0f
800.0f
0.0f
-1280.0f
800.0f
0.0f
-1280.0f
-800.0f
0.0f
I have to either convert those pixel co-ordinates to something between -1,1 or find out a way to have gl convert those co-ordinates
at the time they are drawn? Since I am very new to OpenGL I am looking for some guidance to do this?
My vertex shader is like:
String sVertexShader1 =
"#version 300 es \n"
+ "in vec4 vPosition; \n"
+ "void main() \n"
+ "{ \n"
+ " gl_Position = vPosition; \n"
+ "} \n";
Would I be correct then in saying that a pixels based system would be called world co-ordinates? What I am trying to do right now is just some 2D drawing for a board game.
I've discovered that Android has this function:
orthoM(float[] m, int mOffset, float left, float right, float bottom, float top, float near, float far)
However there is nothing in the documentation I've read so far that explain the usage of the matrix of how a float[] with pixel co-ordinates can be transformed to normalized co-ordinates with that matrix in GLES30.
I've also found the documentation here:
http://developer.android.com/guide/topics/graphics/opengl.html
Based off the documentation I have tried to create an example:
http://pastebin.com/5PTsfSdz
In the pastebin example fSampleVertices I thought would be much smaller and at the center of the screen but it isn't it's still almost the entire screen and fBoardOuter just shows me a black screen if I try to put it into glDrawArray.

You will probably need to find a book or some good tutorials to get a strong grasp on some of these concepts. But since there some specific items in your question, I'll try and explain them as well as I can within this format.
The coordinate system you discovered, where the range is [-1.0, 1.0] in the x- and y coordinate directions, is officially called Normalized Device Coordinates, often abbreviated as NDC. Which is very similar to the name you came up with, so some of the OpenGL terminology is actually very logical. :)
At least as long as you're dealing with 2D coordinates, this is the coordinate range your vertex shader needs to produce. I.e. the coordinates you assign to the built-in gl_Position variable need to be within this range to be visible in the output. Things gets slightly more complicated if you're dealing with 3D coordinates and are applying perspective projections, but we'll skip over that part for now.
Now, as you already guessed, you have two main options if you want to specify your coordinates in a different coordinate system:
You transform them to NDC in your code before you pass them to OpenGL.
You have OpenGL apply transformations to your input coordinates.
Option 2 is clearly the better one, since GPUs are very efficient at performing this job.
On a very simple level, this means that you modify the coordinates in your vertex shader. If you look at your very simple first vertex shader:
in vec4 vPosition;
void main()
{
gl_Position = vPosition;
}
you get the coordinates provided by your app code in the vPosition input variable, and you assign exactly he same coordinates to the vertex shader output gl_Position.
If you want to use a different coordinate system, you process the input coordinates in the vertex shader code, and assign those processed coordinates to the output instead.
Modern versions of OpenGL don't really have a name for those coordinate systems anymore. There used to be "model coordinates" and "world coordinates" when some of this stuff was still hardwired into a fixed pipeline. Now that this is done with programmable shader code, those concepts are not relevant anymore from the OpenGL point of view. All it cares about are the coordinates that come out of the vertex shader. Everything that happens before that is your own business.
The canonical way of applying linear transformations, which includes the translations and scaling you need for your intended use, is by multiplying the coordinates with a transformation matrix. You already discovered the android.opengl.Matrix package that contains some utility functions for building transformation matrices if you don't want to write the (simple) code yourself.
Once you have a transformation matrix, you pass it into the vertex shader as a uniform variable, and apply the matrix in your shader code. The way this looks in the shader code is for example:
in vec4 vPosition;
uniform mat4 TransformMat;
void main()
{
gl_Position = TransformMat * vPosition;
}
To set the value of this matrix, you need to get the location of the uniform variable once after linking the shader, with prog your shader program:
GLint transformLoc = GLES20.glGetUniformLocation(prog, "TransformMat");
Then, at least once, and every time you want to change the matrix, you call:
GLES20.glUniformMatrix4fv(transformLoc, 1, GL_FALSE, mat, 0);
where mat is the matrix you either built yourself, or got from one of the utility functions in android.opengl.Matrix. Note that this call needs to be after you make the program current with glUseProgram().

corrupt data loaded by GLES20.glTexImage2D using GL_LUMINANCE_ALPHA

I am using opengl ES2.0 to render 16bit greyscale data on Android. (I'm handling 16bit to 8bit dynamic range scaling in the shader, the 16bit input is mandatory because it is incoming format of the data.) The resolution is 640x512.
At the moment I have this working by pushing pixels 2 at a time into a 320x512 32bit RGBA texture. i.e:
texture2D(thData, vTextureCoord)[0] is pixel i lower byte,
texture2D(thData, vTextureCoord)[1] is pixel i upper byte,
texture2D(thData, vTextureCoord)[2] is pixel i+1 lower byte,
texture2D(thData, vTextureCoord)[3] is pixel i+1 upper byte.
I am able to reconstruct the data from this but I have reconstruct the original resolution by rendering to a 640x512 buffer and using gl_FragCoord.x and a conditional statement to determine whether to draw the pixel from channels 0,1 or 2,3. The code looks like this:
private final String mFragmentShader =
"precision highp float;\n" +
"varying vec2 vTextureCoord;\n" +
"uniform sampler2D thData;\n" +
"void main() {\n" +
//This works out whether the pixel is in an odd or even column:
"int odd = 0;\n" +
"if (fract(gl_FragCoord.x/2.) >= 0.5) odd = 2;\n"+
//This chooses the channels based on the column number
"float data =((texture2D(thData, vTextureCoord)[odd]) + (texture2D(thData, vTextureCoord)[odd+1]*256.))*256.;\n" +
This would be fine in principle but for some reason I get seem to be getting some columns swapped, my guess is that it could be rounding errors causing an incorrect result from the conditional statement.
What I'd like to do is avoid the down sampling by using a 16bit full resolution texture in the first place, this seems really simple; I have just changed the glTexImage2D code from:
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGBA, 320, 512, 0, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, sBuffer);
To
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE_ALPHA, 640, 512, 0, GLES20.GL_LUMINANCE_ALPHA, GLES20.GL_UNSIGNED_BYTE, sBuffer);
However, when I do this I appear to be getting strange data in the texture. texture2D(thData, vTextureCoord)[0] appears to be correct but texture2D(thData, vTextureCoord)[1] has the same data as element 0, the upper byte of the pixel is nowhere to be seen.
Am I missing anything here?
Or can anyone suggest an alternative approach?

GL_LUMINANCE_ALPHA textures replicate the first component across all x, y, and z components. The second component is in w, as the "ALPHA" part of the enum name implies. See the definition of GL_LUMINANCE_ALPHA format from: http://www.khronos.org/opengles/sdk/docs/man/xhtml/glTexImage2D.xml

Did you set all pixel unpack parameters? See glPixelStorei. Most likely you must set GL_UNPACK_ALIGNMENT to 1.

android OpenGL ES coordinates mapping

I did a lot of search and nothing solved my problem. I'm both new to android and to 3d programming. I'm working on an Android project where I need to draw a 3d object on the android device using opengl es. For each pixel I have Distance value between 200 and 9000, which needs to be mapped as a Z coordinate. The object is 320x240.
The questions are:
How do I map from (x,y,z) to opengl es coordinate system? I have created a vertex array whose values are {50f, 50f, 400f, 50f, 51f, 290f, ...}. Each pixel is represented as 3 floats (x,y,z).
How can this vertex array be drawn using opengl on an android?
Is it possible to draw 320*240 pixels using OpenGl ES?

OpenGL doesn't really work well with large numbers (like anything over 10.0f, just the way it is designed). It would be better to convert your coordinates to be between -1 and 1 (i.e. normalize) than to try and make openGL use coordinates of 50f or 290f.
The reason the coordinates are normalized to between -1 and 1 is because model coordinates are only supposed to be relative to each other and not indicative of their actual dimensions in a specific game/app. The model could be used in many different games/apps with different coordinate systems, so you want all the model coordinates to be in some normalized standard form, so the programmer can then interpret in their own way.
To normalize, you loop through all your coordinates and find the value furthest from 0 i.e.
float maxValueX = 0;
float maxValueY = 0;
float maxValueZ = 0;
// find the max value of x, y and z
for(int i=0;i<coordinates.length'i++){
maxValueX = Math.max(Math.abs(coordinates[i].getX()), maxValueX);
maxValueY = Math.max(Math.abs(coordinates[i].getY()), maxValueY);
maxValueZ = Math.max(Math.abs(coordinates[i].getZ()), maxValueZ);
}
// convert all the coordinates to be between -1 and 1
for(int i=0;i<coordinates.length'i++){
Vector3f coordinate = coordinates[i];
coordinate.setX(coordinate.getX() / maxValueX);
coordinate.setY(coordinate.getY() / maxValueY);
coordinate.setZ(coordinate.getZ() / maxValueZ);
}
You only need to do this once. Assuming you are storing your data in a file, you can write a little utility program that does the above to the file and save it, rather than doing it every time you load the data into your app

Checkout the GLSurfaceView Activity in the APIDemos that ship with the Android SDK. That will give you a basic primer on how Android handles rendering through OpenGL ES. This is located in android-sdk/samples/android-10/ApiDemos. Make sure you have downloaded the 'Samples for SDK' under the given API level.
Here's a couple of resources to get you started as well:
Android Dev Blog on GLSurfaceView
Instructions on OpenGLES
Android Development Documentation on OpenGL
Hope that helps.

Adding to what James had mentioned about normalizing to [-1,1].
A little bit of code :
FIll in data in a flat array as x,y,z assuming you are using a vertex shader similar to :
"attribute vec3 coord3d;" +
"uniform mat4 transform;" +
"void main(void) {" +
" gl_Position = transform * vec4(coord3d.xyz, 1.0f);" + // size of 3 with a=1.0f for all points
" gl_PointSize = 10.0;"+
"}"
Get the attribute :
attribute_coord3d = glGetAttribLocation(program, "coord3d");
Create VBO:
glGenBuffers(1, vbo,0);
Bind
glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
Put data in:
glBufferData(GL_ARRAY_BUFFER, size:SIZE_OF_ARRAY, makeFloatBuffer(FlatArray), GL_STATIC_DRAW);
where makeFloatBuffer is a function that creates a buffer:
private FloatBuffer makeFloatBuffer(float[] arr) {
ByteBuffer bb = ByteBuffer.allocateDirect(arr.length*4);
bb.order(ByteOrder.nativeOrder());
FloatBuffer fb = bb.asFloatBuffer();
fb.put(arr);
fb.position(0);
return fb;
}
Bind and Point to buffer:
glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
glEnableVertexAttribArray(attribute_coord3d);
glVertexAttribPointer(attribute_coord3d,
size:3,GL_FLOAT,false,stride:vertexStride, 0);
where vertexStride = num_components*Float.BYTES; in our case num_components = 3 // x,y,z.
Draw:
glDrawArrays(GL_POINTS, 0, NUM_OF_POINTS);
Disable VBO:
glDisableVertexAttribArray(attribute_coord2d);