How to use experimental meshing API with Project Tango - android

I apologize in advance for my long post.
My purpose is to create a meshing app for a Project Tango Yellowstone device to create a 3D map of building interiors. I intend to make use of the experimental meshing API added in recent versions of the tango-examples-c code.
I'm using point-cloud-jni-example (turing) as a starting point and so far have done the following:
Set config_experimental_enable_scene_reconstruction tango config parameter in point_cloud_app.cc (see docs)
// Enable scene reconstruction
ret = TangoConfig_setBool(tango_config_,
config_experimental_enable_scene_reconstruction", true);
if (ret != TANGO_SUCCESS) {
LOGE("PointCloudApp: config_experimental_enable_scene_reconstruction() failed"
"with error code: %d", ret);
return ret;
}
Added extractMesh native method in TangoJNINative.java
// Extracts the full mesh from the scene reconstruction.
public static native float extractMesh();
Added matching extractMesh function to the jni_interface.cc
JNIEXPORT void JNICALL
Java_com_projecttango_experiments_nativepointcloud_TangoJNINative_extractMesh(
JNIEnv*, jobject) {
app.ExtractMesh();
}
Added ExtractMesh method in point_cloud_app.cc
void PointCloudApp::ExtractMesh() {
// see line 1245 of tango_client_api.h
mesh_ptr = new TangoMesh_Experimental();
TangoService_Experimental_extractMesh(mesh_ptr);
mesh = *mesh_ptr;
LOGE("PointCloudApp: num_vertices: %d", mesh.num_vertices);
float float1, float2, float3;
float1 = mesh.vertices[1][0];
float2 = mesh.vertices[1][1];
float3 = float1 + float2; // these lines show I can use the vertex data
LOGE("PointCloudApp: First vertex, x: %f", mesh.vertices[1][0]); // this line causes app to crash; printing the vertex data seems to be the problem
}
Added TangoMesh_Experimental declaration to point_cloud_app.h
// see line 1131 of tango_client_api.h
TangoMesh_Experimental* mesh_ptr;
TangoMesh_Experimental mesh;
Added an additional button to call the extractMesh native method. (not showing this one as it is pretty straightforward)
For reference, here is the TangoMesh_Experimental Struct from the API:
// A mesh, described by vertices and face indices, with optional per-vertex
// normals and colors.
typedef struct TangoMesh_Experimental {
// Index into a three-dimensional fixed grid.
int32_t index[3];
// Array of vertices. Each vertex is an {x, y, z} coordinate triplet, in
// meters.
float (*vertices)[3];
// Array of faces. Each face is an index triplet into the vertices array.
uint32_t (*faces)[3];
// Array of per-vertex normals. Each normal is a normalized {x, y, z} vector.
float (*normals)[3];
// Array of per-vertex colors. Each color is a 4-tuple of 8-bit {R, G, B, A}
// values.
uint8_t (*colors)[4];
// Number of vertices, describing the size of the vertices array.
uint32_t num_vertices;
// Number of faces, describing the size of the faces array.
uint32_t num_faces;
// If true, each vertex will have an associated normal. In that case, the
// size of the normals array will be equal to num_vertices. Otherwise, the
// size of the normals array will be 0.
bool has_normals;
// If true, each vertex will have an associated color. In that case, the size
// of the colors array will be equal to num_vertices. Otherwise, the size of
// the colors array will be 0.
bool has_colors;
} TangoMesh_Experimental;
My current understanding of this struct is:
The three pointers in float (*vertices)[3]; point to the addresses at the start of three chuncks of memory for the x, y, and z coordinates for the vertices for the mesh (the same is true for normals and colors colors). A specific vertex is composed of an x, y, and z component found at a specific index in the three arrays.
Similarly, the uint32_t (*faces)[3] array has three pointers to the beginning of three chunks of memory, but a specific set of three elements here instead contains index numbers that indicate which three vertices (from the vertices array (each with three coordinates)) make up that face.
The current status is I am able to extract the mesh, and print some of it to the console, then crashes without errors
PointCloudApp: PointCloudApp: num_vertices: 8044
If I omit the last line I added in point_cloud_app.cc (#4, above), the app doesn't crash. I am able to access the vertex data and do something with it, but printing it using LOGE causes a crash 9 times out of 10. Occasionally, it does print the value correctly without crashing. Could the vertex data have holes or invalid values?
I have tried returning test_float from JNI back to java, but it crashes again when I try to do so.
Suggestions?

vertices is a dynamic array of points, where each point is a float[3]. Try this example:
for (int i = 0; i < mesh.num_vertices; ++i) {
printf("%d: x=%f y=%f z=%f\n", i, mesh.vertices[i][0],
mesh.vertices[i][1], mesh.vertices[i][2]);
}
If you look at the memory layout, it would be x0 y0 z0 x1 y1 z1 etc, each of those a float.

Related

Texture coordinates inconsistent

I tried tiling texture coordinates on a quad in unity for some debugging.
On desktop it had no problems but on android(old tablet) I see inconsistency in texture coordinates(Image links below show the results).
As we go from 0.0,0.0 to 1.0,1.0 the texture coordinates keeps becoming inconsistent.
the inconsistency increases as the tiling values go up, Results in images are for 60,160.
my device's android version is 5.0 and GPU supports opengl es 2.0 not 3.0.
So I would like to know why is this happening on my tablet but not on my desktop ,would appreciate information about what is happening in the gpu that is causing it.
Shader I used-
Shader "Unlit/uvdisplay"
{
Properties
{
}
SubShader
{
Tags { "RenderType"="Opaque" }
Pass
{
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
struct appdata
{
float4 vertex : POSITION;
float2 uv : TEXCOORD0;
};
struct v2f
{
float2 uv : TEXCOORD0;
float4 vertex : SV_POSITION;
};
v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
o.uv = v.uv;
return o;
}
fixed4 frag (v2f i) : SV_Target
{
// sample the texture
fixed4 col = fixed4(frac(i.uv*fixed2(60.0,160.0)),0.0,1.0);
return col;
}
ENDCG
}
}
}
texcoord at origin
texcoord at the opposit end
Not all Android devices support highp in the fragment shader, some older devices are limited to mediump.
In Unity terms, based on these definitions, it's as if you ask for float precision in your fragment shader, but you get half precision.
Not too much you can do about it except try to keep values near the origin to minimize error.
Also, on some of the devices that only support half precision, as long as you do nothing except sample the texture (i.e. don't modify or swizzle the texture coordinates before using it), the driver/GPU is able to use the full precision for the texture sampling.

Swap out MainTex pixels with other textures' via surface Shader (Unity)

The main texture of my surface shader is a Google Maps image tile, similar to this:
.
I want to replace pixels that are close to a specified color with that from a separate texture. What is working now is the following:
Shader "MyShader"
{
Properties
{
_MainTex("Base (RGB) Trans (A)", 2D) = "white" {}
_GrassTexture("Grass Texture", 2D) = "white" {}
_RoadTexture("Road Texture", 2D) = "white" {}
_WaterTexture("Water Texture", 2D) = "white" {}
}
SubShader
{
Tags{ "Queue" = "Transparent-1" "IgnoreProjector" = "True" "ForceNoShadowCasting" = "True" "RenderType" = "Opaque" }
LOD 200
CGPROGRAM
#pragma surface surf Lambert alpha approxview halfasview noforwardadd nometa
uniform sampler2D _MainTex;
uniform sampler2D _GrassTexture;
uniform sampler2D _RoadTexture;
uniform sampler2D _WaterTexture;
struct Input
{
float2 uv_MainTex;
};
void surf(Input IN, inout SurfaceOutput o)
{
fixed4 ct = tex2D(_MainTex, IN.uv_MainTex);
// if the red (or blue) channel of the pixel is within a
// specific range, get either a 1 or a 0 (true/false).
int grassCond = int(ct.r >= 0.45) * int(0.46 >= ct.r);
int waterCond = int(ct.r >= 0.14) * int(0.15 >= ct.r);
int roadCond = int(ct.b >= 0.23) * int(0.24 >= ct.b);
// if none of the above conditions is a 1, then we want to keep our
// current pixel's color:
half defaultCond = 1 - grassCond - waterCond - roadCond;
// get the pixel from each texture, multiple by their check condition
// to get:
// fixed4(0,0,0,0) if this isn't the right texture for this pixel
// or fixed4(r,g,b,1) from the texture if it is the right pixel
fixed4 grass = grassCond * tex2D(_GrassTexture, IN.uv_MainTex);
fixed4 water = waterCond * tex2D(_WaterTexture, IN.uv_MainTex);
fixed4 road = roadCond * tex2D(_RoadTexture, IN.uv_MainTex);
fixed4 def = defaultCond * ct; // just used the MainTex pixel
// then use the found pixels as the Albedo
o.Albedo = (grass + road + water + def).rgb;
o.Alpha = 1;
}
ENDCG
}
Fallback "None"
}
This is the first shader I've ever written, and it probably isn't very performant. It seems counter intuitive to me to call tex2D on each texture for every pixel to just throw that data away, but I couldn't think of a better way to do this without if/else (which I read were bad for GPUs).
This is a Unity Surface Shader, and not a fragment/vertex shader. I know there is a step that happens behind the scenes that will generate the fragment/vertex shader for me (adding in the scene's lighting, fog, etc.). This shader is applied to 100 256x256px map tiles (2560x2560 pixels in total). The grass/road/water textures are all 256x256 pixels as well.
My question is: is there a better, more performant way of accomplishing what I'm doing here? The game runs on Android and iOS.
I'm not a specialist in Shader performance, but assuming you have a relatively small number of source tiles that you wish to render in the same frame it might make more sense to store the result of the pixel replacement and reuse it.
As you are stating that the resulting image is going to be the same size as your source tile, just render the source tile using your surface shader (without any lighting though, you may want to consider using a simple, flat pixel shader!) into a RenderTexture once and then use that RenderTexture as source for your world rendering. That way you are doing the expensive work only once per source tile and thus it isn't even important anymore whether your shader is well optimized.
If all textures are static, you might even consider not doing this at runtime, but just translate them once in the Editor.

Android fragment shader having problems

Sorry for the broad question, but I didn't know quite how to word it. I am creating an app that manipulates the camera's pixels. I am new to OpenGl and my problem could be in how I link textures to the shaders, or somewhere in my actual shader code.
I have an RGB look up table that I turn into a texture and pass into the shader to use as the manipulation table. I believe my texture is of the proper size and setting, but I am not 100% sure. In my shader I have this:
uniform sampler2D data_Texture // The RGB look up table texture
uniform samplerExternalOES u_Texture; // The camera's texture
And this is in my main loop in the shader:
// Color changing Algorithm
vec3 texel = texture2D(u_Texture, v_TexCoordinate).rgb;
gl_FragColor = vec4(texel.x, texel.y, texel.z, 1.0);
float rr = gl_FragColor.r * 255.0;
float gg = gl_FragColor.g * 255.0;
float bb = gl_FragColor.b * 255.0;
int r = int(rr);
int g = int(gg);
int b = int(bb);
int index = ((r/4) * 4096) + ((g/4) * 64) + (b/4);
int x = int(mod(float(index), 512.0));
int y = index / 512;
vec4 data = texture2D(data_Texture, vec2(float(x)/512.0, float(y)/512.0));
We take the camera's RGB pixels to get an index for the look up table. Then we try to get the rgb data out of the look up table to replace the camera's pixel with. This is where the problem occurs. As you can probably tell from the code above, we don't actually change the FragColor with our data. This is because we were testing and found an interesting occurrence. When we comment out the last line in the main loop,
//vec4 data = texture2D(data_Texture, vec2(float(x)/512.0, float(y)/512.0));
the camera just displays like normal, because we don't do any manipulations on the actual FragColor. But when we leave the last line in, the pixels turn green for dark colors and pink/orange for light colors.
Why does filling this data variable, and not explicitly changing the FragColor, change the camera's pixels??

OpenCV for Android - Access elements of Mat

What is the standard way to access and modify individual elements of a Mat in OpenCV4Android? Also, what is the format of the data for BGR (which is the default, I think) and grayscale?
edit: Let's make this more specific. mat.get(row, col) returns a double array. What is in this array?
If you just want to access some pixels do it by using double[] get(int row, int col) and writing using put(int row, int col, double... data). If you are thinking in accessing the whole image or iterate the data of the image in a loop the best thing you should do is copy the Mat data into a Java primitive data type. When you're done with the data operations just copy the data back into a Mat structure.
Images use CV_8U, if you have a grayscale image it will use CV_8UC1 if you have a RGB image it will use a Mat of CV_8UC3 (3 channels of CV_8U). CV_8U is the equivalent of byte in java. :)
I can give you and example of a method I use in Java (Android platform) to binarize a grayscale image:
private Mat featuresVectorBinarization(Mat fv){
int size = (int) fv.total() * fv.channels();
double[] buff = new double[size];
fv.get(0, 0, buff);
for(int i = 0; i < size; i++)
{
buff[i] = (buff[i] >= 0) ? 1 : 0;
}
Mat bv = new Mat(fv.size(), CvType.CV_8U);
bv.put(0, 0, buff);
return bv;
}
Hope that helps.
What is the standard way to access and modify individual elements of a Mat in OpenCV4Android?
A Mat is the thing we call a "matrix" in Mathematics - a rectangular array of quantities set out by rows and columns. Those "quantities" represent pixels in the case of an image Mat, (e.g. every element of a matrix can be the color of each pixel in an image Mat). From this tutorial:
in the above image you can see that the mirror of the car is nothing
more than a matrix containing all the intensity values of the pixel
points.
So how would you go about iterating through a matrix? How about this:
for (int row=0; row<mat.rows(); row++) {
for (int col=0; col<mat.cols(); col++ ) {
//...do what you want..
//e.g. get the value of the 3rd element of 2nd row
//by mat.get(2,3);
}
}
What is the standard way to access and modify individual elements of a Mat in OpenCV4Android?
You get the value of an element of Mat by using its function get(x)(y), where x is the first coordinate (row number) and y is the second coordinate (column number) of the element. For example to get the 4th element of the 7th row of a BGR image Mat named bgrImageMat, use the get method of Mat to get an array of type double, which will have a size of 3, each array element representing each of the Blue, Green, and Red channels of the BGR image format.
double [] bgrColor = bgrImageMat.get();
Also, what is the format of the data for BGR (which is the default, I think) and grayscale? edit: Let's make this more specific. mat.get(row, col) returns a double array. What is in this array?
You can read about BGR color format and grayscale from the web. e.g. BGR and Grayscale.
In short, BGR color format has 3 channels: Blue, Green and Red. So the double array returned by mat.get(row, col), when mat is a BGR image, is an array of size 3, and each of its elements contains the values of each of the Blue, green and red channels respectively.
Likewise, Grayscale format is a 1 channel color format, so the double returned will have a size of 1.
What I could understand from learning about OpenCV Mat object is that Mat is object that can represent any image of W x H pixels.
Now lets say you want to access center pixel of your image then
X = W/2
Y = H/2
Then you can access pixel data as follows
double[] data = matObject.get(x,y);
Now what does data represent and what is size of data array.That depends on image type.
if image is grayscale then data.length = 1 , since there is one channel only , and data[0] represents color value of that pixel ie. 0(black) - 255(white)
if image is color image then data.length = 4(rgba) , since there are four channels , and data[0-n] represent color value of that pixel

Optical flow in Android

We have been dealing with OpenCV for two weeks to make it work on Android.
Do you know where can we find an Android implementation of optical flow? It would be nice if it's implemented using OpenCV.
Openframeworks has openCV baked in, as well as many other interesting libraries. It has a very elegant strucutre, and I have used it with android to make a virtual mouse of the phone using motion estimation from the camera.
See the ports to android here http://openframeworks.cc/setup/android-studio/
Seems they recently added support for android studio, otherwise eclipse works great.
Try this
#Override
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
mRgba = inputFrame.rgba();
if (mMOP2fptsPrev.rows() == 0) {
//Log.d("Baz", "First time opflow");
// first time through the loop so we need prev and this mats
// plus prev points
// get this mat
Imgproc.cvtColor(mRgba, matOpFlowThis, Imgproc.COLOR_RGBA2GRAY);
// copy that to prev mat
matOpFlowThis.copyTo(matOpFlowPrev);
// get prev corners
Imgproc.goodFeaturesToTrack(matOpFlowPrev, MOPcorners, iGFFTMax, 0.05, 20);
mMOP2fptsPrev.fromArray(MOPcorners.toArray());
// get safe copy of this corners
mMOP2fptsPrev.copyTo(mMOP2fptsSafe);
}
else
{
//Log.d("Baz", "Opflow");
// we've been through before so
// this mat is valid. Copy it to prev mat
matOpFlowThis.copyTo(matOpFlowPrev);
// get this mat
Imgproc.cvtColor(mRgba, matOpFlowThis, Imgproc.COLOR_RGBA2GRAY);
// get the corners for this mat
Imgproc.goodFeaturesToTrack(matOpFlowThis, MOPcorners, iGFFTMax, 0.05, 20);
mMOP2fptsThis.fromArray(MOPcorners.toArray());
// retrieve the corners from the prev mat
// (saves calculating them again)
mMOP2fptsSafe.copyTo(mMOP2fptsPrev);
// and save this corners for next time through
mMOP2fptsThis.copyTo(mMOP2fptsSafe);
}
/*
Parameters:
prevImg first 8-bit input image
nextImg second input image
prevPts vector of 2D points for which the flow needs to be found; point coordinates must be single-precision floating-point numbers.
nextPts output vector of 2D points (with single-precision floating-point coordinates) containing the calculated new positions of input features in the second image; when OPTFLOW_USE_INITIAL_FLOW flag is passed, the vector must have the same size as in the input.
status output status vector (of unsigned chars); each element of the vector is set to 1 if the flow for the corresponding features has been found, otherwise, it is set to 0.
err output vector of errors; each element of the vector is set to an error for the corresponding feature, type of the error measure can be set in flags parameter; if the flow wasn't found then the error is not defined (use the status parameter to find such cases).
*/
Video.calcOpticalFlowPyrLK(matOpFlowPrev, matOpFlowThis, mMOP2fptsPrev, mMOP2fptsThis, mMOBStatus, mMOFerr);
cornersPrev = mMOP2fptsPrev.toList();
cornersThis = mMOP2fptsThis.toList();
byteStatus = mMOBStatus.toList();
y = byteStatus.size() - 1;
for (x = 0; x < y; x++) {
if (byteStatus.get(x) == 1) {
pt = cornersThis.get(x);
pt2 = cornersPrev.get(x);
Core.circle(mRgba, pt, 5, colorRed, iLineThickness - 1);
Core.line(mRgba, pt, pt2, colorRed, iLineThickness);
}
}
return mRgba;
}

Categories

Resources