Develop Reference

Fragment shader very slow when call texture2D many time

I have a fragment shader for blur skin. Input is a YUV texture (it include y texture, u texture, v texture). With frame resolution is 1280x720 run on galaxy A6, it take 80ms-120ms/frame.
I found the bottle neck at greenValue() function call. It take most of time if I call it like the following code. If I only call "sampleColor = greenValue(blurCoordinates[0]);\n" ... (not "sampleColor += greenValue(blurCoordinates[0]);\n" ... it will very fast.
+ "float greenValue(vec2 coord)\n"
+ "{\n"
+ " return texture2D(y_tex, coord).r - 0.344 * (texture2D(u_tex, coord).r - 0.5) - 0.714 * (texture2D(v_tex, coord).r - 0.5);\n"
+ "}\n";
+ "// some code .... if (current pixel is skin color)"
+ "vec2 blurCoordinates[20];\n"
+ "blurCoordinates[0] = interp_tc.xy + singleStepOffset * vec2(0.0, -10.0);\n" +
+ "blurCoordinates[1] = interp_tc.xy + singleStepOffset * vec2(0.0, 10.0);\n" +
+ "blurCoordinates[2] = interp_tc.xy + singleStepOffset * vec2(-10.0, 0.0);\n" +
+ "blurCoordinates[3] = interp_tc.xy + singleStepOffset * vec2(10.0, 0.0);\n" +
+ "blurCoordinates[4] = interp_tc.xy + singleStepOffset * vec2(5.0, -8.0);\n" +
+ "blurCoordinates[5] = interp_tc.xy + singleStepOffset * vec2(5.0, 8.0);\n" +
+ "blurCoordinates[6] = interp_tc.xy + singleStepOffset * vec2(-5.0, 8.0);\n" +
+ "blurCoordinates[7] = interp_tc.xy + singleStepOffset * vec2(-5.0, -8.0);\n" +
+ "blurCoordinates[8] = interp_tc.xy + singleStepOffset * vec2(8.0, -5.0);\n" +
+ "blurCoordinates[9] = interp_tc.xy + singleStepOffset * vec2(8.0, 5.0);\n" +
+ "blurCoordinates[10] = interp_tc.xy + singleStepOffset * vec2(-8.0, 5.0);\n" +
+ "blurCoordinates[11] = interp_tc.xy + singleStepOffset * vec2(-8.0, -5.0);\n" +
+ "blurCoordinates[12] = interp_tc.xy + singleStepOffset * vec2(0.0, -6.0);\n" +
+ "blurCoordinates[13] = interp_tc.xy + singleStepOffset * vec2(0.0, 6.0);\n" +
+ "blurCoordinates[14] = interp_tc.xy + singleStepOffset * vec2(6.0, 0.0);\n" +
+ "blurCoordinates[15] = interp_tc.xy + singleStepOffset * vec2(-6.0, 0.0);\n" +
+ "blurCoordinates[16] = interp_tc.xy + singleStepOffset * vec2(-4.0, -4.0);\n" +
+ "blurCoordinates[17] = interp_tc.xy + singleStepOffset * vec2(-4.0, 4.0);\n" +
+ "blurCoordinates[18] = interp_tc.xy + singleStepOffset * vec2(4.0, -4.0);\n" +
+ "blurCoordinates[19] = interp_tc.xy + singleStepOffset * vec2(4.0, 4.0);\n";
+ "// some code ...."
+ "float sampleColor = centralColor.g * 20.0;\n"
+ "sampleColor += greenValue(blurCoordinates[0]);\n"
+ "sampleColor += greenValue(blurCoordinates[1]);\n"
+ "sampleColor += greenValue(blurCoordinates[2]);\n"
+ "sampleColor += greenValue(blurCoordinates[3]);\n"
+ "sampleColor += greenValue(blurCoordinates[4]);\n"
+ "sampleColor += greenValue(blurCoordinates[5]);\n"
+ "sampleColor += greenValue(blurCoordinates[6]);\n"
+ "sampleColor += greenValue(blurCoordinates[7]);\n"
+ "sampleColor += greenValue(blurCoordinates[8]);\n"
+ "sampleColor += greenValue(blurCoordinates[9]);\n"
+ "sampleColor += greenValue(blurCoordinates[10]);\n"
+ "sampleColor += greenValue(blurCoordinates[11]);\n"
+ "sampleColor += greenValue(blurCoordinates[12]) * 2.0;\n"
+ "sampleColor += greenValue(blurCoordinates[13]) * 2.0;\n"
+ "sampleColor += greenValue(blurCoordinates[14]) * 2.0;\n"
+ "sampleColor += greenValue(blurCoordinates[15]) * 2.0;\n"
+ "sampleColor += greenValue(blurCoordinates[16]) * 2.0;\n"
+ "sampleColor += greenValue(blurCoordinates[17]) * 2.0;\n"
+ "sampleColor += greenValue(blurCoordinates[18]) * 2.0;\n"
+ "sampleColor += greenValue(blurCoordinates[19]) * 2.0;\n";
Are there a better solution to this problem, or any way to optimize this shader?
Update
If frame resoluton is 1280x720, number of texture2D() calling = 1280x720x20x3 = 55,296,000 (calling) / frame
If I replace (u_tex and v_tex) by y_tex (for test), it's very fast. So, maybe access 3 difference texture make bottle neck.

You're doing 60 texture samples per pixel on an entry level phone! Yes, it's going to be slow.
Not sure which device variant you have, but assuming it's the Mali-T830 MP1 one then you can do 1 texture sample per clock cycle. Assuming it's around 550Mhz for sake of argument and easy maths, then:
550M / 55M samples = 100ms / frame
... which is about what you are seeing. This shader is just far too complex for this device.
Flipping this around you can set a budget based on what you want to achieve. I.e. at 60 FPS
550M / 60 = 9.16M cycles/frame.
9.16M / (1280 * 720) = 9.95 samples per pixel.
For YUV data try to pull it directly from an external sampler. For this you can sample directly from the original YUV data including color conversion, on a newer device this will be much faster than hand rolling your own sampling code as most GPUs can sample YUV natively (Note - it won't help much on a Mali-T830 though).
Are there a better solution to this problem, or any way to optimize this shader?
Learn to use a profiler - helps avoid guess work.
For Mali GPUs, see the Streamline profiler in Arm Mobile Studio (https://developer.arm.com/mobile-studio).
If I only call "sampleColor = greenValue(blurCoordinates[0]);\n" ... (not "sampleColor += greenValue(blurCoordinates[0]);\n" ... it will very fast.
Yes, the compiler will optimize out 19 out of the 20 samples, so you're doing 5% of the work, so would expect it to be 20x faster.
If I replace (u_tex and v_tex) by y_tex (for test), it's very fast.
This makes the three texture operations identical, so you remove 2 out of 3 texture samples, and the compiler can merge a lot of the maths.

Crop YUV byte[] based on a rectangle without any convertion

I have tried to use the logic and pictorial representation from this SO. I am though confused with the images since one of them follow 4:1:1 whereas the later one does 4:2:2 nomenclature for YUV image (NV21).
Right now the issue is that i get an image (converted to Bitmap/PNG) with YUV component all over, essentially an unusable image.
Any recommendation to fix this?
private byte[] cropImage(byte[] data, Rect cropRect) {
int dataHeight = 480;
int dataWidth = 640;
int totalWH = dataWidth * dataHeight;
// make rect points even, currently the width & height is even number
// adjust x coordinates to make them
if (cropRect.left % 2 != 0 || cropRect.right % 2 != 0) {
cropRect.left -= 1;
cropRect.right -= 1;
}
// adjust y coordinates to make them even
if (cropRect.top % 2 != 0 || cropRect.bottom % 2 != 0) {
cropRect.top -= 1;
cropRect.bottom -= 1;
}
int area = cropRect.width() * cropRect.height() * 3/2;
Logger.getLogger().d("Size of byte array " + data.length + " Size of alloc area " + area);
byte[] pixels = new byte[area];//the size of the array is the dimensions of the sub-photo
// size.total = size.width * size.height;
// y = yuv[position.y * size.width + position.x];
// u = yuv[(position.y / 2) * (size.width / 2) + (position.x / 2) + size.total];
// v = yuv[(position.y / 2) * (size.width / 2) + (position.x / 2) + size.total + (size.total / 4)];
try {
// copy Y plane first
int srcOffset = cropRect.top * dataWidth;
int destOffset = 0;
int lengthToCopy = cropRect.width();
int y = 0;
for (; y < cropRect.height(); y++, srcOffset += dataWidth, destOffset += cropRect.width()) {
// Logger.getLogger().d("IO " + srcOffset + cropRect.left + " oO " + destOffset + " LTC " + lengthToCopy);
System.arraycopy(data, srcOffset + cropRect.left, pixels, destOffset, lengthToCopy);
}
Logger.getLogger().d("Completed Y copy");
// U and V components are not-interleaved, hence their size is just 1/4th the original size
// copy U plane
int nonYPlanerHeight = dataHeight / 4;
int nonYPlanerWidth = dataWidth / 4;
srcOffset = totalWH + (cropRect.top / 4 * nonYPlanerWidth);
for (y = 0; y < cropRect.height();
y++, srcOffset += nonYPlanerWidth, destOffset += cropRect.width() / 4) {
System.arraycopy(data, srcOffset + cropRect.left / 4, pixels, destOffset, cropRect.width() / 4);
}
Logger.getLogger().d("Completed U copy " + y + " destOffset=" + destOffset);
// copy V plane
srcOffset = totalWH + totalWH / 4 + (cropRect.top / 4 * nonYPlanerWidth);
for (y = 0; y < cropRect.height();
y++, srcOffset += nonYPlanerWidth, destOffset += cropRect.width() / 4) {
System.arraycopy(data, srcOffset + cropRect.left / 4, pixels, destOffset, cropRect.width() / 4);
}
Logger.getLogger().d("Completed V copy " + y + " destOffset=" + destOffset);
} catch (ArrayIndexOutOfBoundsException ae) {
// do nothing
Logger.getLogger().e("Exception " + ae.getLocalizedMessage());
}
return pixels;
}

Custom Onlayout not working

Can anybody tell me how to change data dynamically in onLayout().i want to display first 8 values and after rotation complete another 8 values until it display array limit.here is my code-
if (position < conversion.length) {
if (!childRotate) {
child.setAngle(angle);
child.setPosition(i);
position = i;
child.setImageResource(conversion[position]);
Float an = angle;
Log.i("prj,=angle", an.toString() + ",i=" + i.toString()
+ ",onlayout,position=" + position.toString());
position++;
} else {
child.setAngle(angle);
child.setPosition(i);
child.setImageResource(conversion[position]);
Float an = angle;
Log.i("prj,angle=", an.toString() + ",i=" + i.toString()
+ ",onlayout,position=" + position.toString());
position++;
// childRotate = false;
}
}
left = Math
.round((float) (((layoutWidth / 2) - childWidth / 2) + radius
* Math.cos(Math.toRadians(angle))));
top = Math
.round((float) (((layoutHeight / 2) - childHeight / 2) + radius
* Math.sin(Math.toRadians(angle))));
child.layout(left, top, left + childWidth, top + childHeight);
angle += angleDelay;

Android OpenGL 2.0 : Object Picking (seeing if you hit a object with a ray)

So I am trying to get object picking working in OpenGL 2, in OpenGK 1 I used the glpixelColor which was pretty straight forward.
I have the following code to give me the start and end point of my ray??
Log.i("My POSITION", "x:" + mRenderer.eye.x + " y:" + mRenderer.eye.y + " z:" + mRenderer.eye.z);
float xyzw[] = unproject(x, mRenderer.screenHeight - y, -1.0f);
Log.i("Start of ray", "x:" + xyzw[0] + " y:" + xyzw[1] + " z:" + xyzw[2]);
xyzw = unproject(x, mRenderer.screenHeight - y, 1.0f);
Log.i("End of Ray", "x:" + xyzw[0] + " y:" + xyzw[1] + " z:" + xyzw[2]);
I use the following function
public float[] unproject(float rx, float ry, float rz) {
float[] xyzw = {0, 0, 0, 0};
int[] viewport = { 0, 0, mRenderer.screenWidth, mRenderer.screenHeight};
android.opengl.GLU.gluUnProject(rx, ry, rz, mRenderer.mViewMatrix, 0, mRenderer.mProjectionMatrix, 0, viewport, 0, xyzw, 0);
xyzw[0] /= xyzw[3];
xyzw[1] /= xyzw[3];
xyzw[2] /= xyzw[3];
xyzw[3] = 1;
return xyzw;
}
I get the following output
My POSITION: x:-1.857801 y:0.0 z:-8.655011
04-21
Start of ray: x:-1.8198236 y:-0.005848203 z:-8.688532
04-21
End of Ray: x:758.43915 y:-117.07846 z:-679.7136
Question
How do I work out if I have tapped on a cube at a position? Say the cube is at 1,1,1

I have tried this and it kinda works
float xDif = (xyzw2[0] - xyzw[0]) / 1000;
float yDif = (xyzw2[1] - xyzw[1]) / 1000;
float zDif = (xyzw2[2] - xyzw[2]) / 1000;
for (int i = 0; i < 1000; i ++)
{
if ((xyzw[0] + (xDif * i)) > mRenderer.cube.position.x - 1.0 && (xyzw[0] + (xDif * i)) < mRenderer.cube.position.x + 1.0 &&
(xyzw[1] + (yDif * i)) > mRenderer.cube.position.y - 1.0 && (xyzw[1] + (yDif * i)) < mRenderer.cube.position.y + 1.0 &&
(xyzw[2] + (zDif * i)) > mRenderer.cube.position.z - 1.0 && (xyzw[2] + (zDif * i)) < mRenderer.cube.position.z + 1.0)
{
Log.i("Hit cube", "HIT");
break;
}
}

Limit Scrolling on offline maps, in Android

I got these piece of codes or patches from osmdroid, and I decided to ask for your help guys because i don't have the enough knowledge to combine these codes to come up with on a solution on my problem, Scrolling limit on an offline map. I searched across the web, and modified tutorials. Honestly I tried to modify these codes but i have not found any progress. Basically I have an offline map from mapnik, and a few overlays. I don't know where to properly place these set of codes. Your ideas and modification will be a great help and also helps me keep going with my project and I guess your answers will definitely help others with the same problem as mine in the future. I know this is to much. Thank you sirs for your time, and God Bless.
public void onCreate(Bundle savedInstanceState) {
...
...
m_mapView = (MapView) findViewById(R.id.mapview);
m_mapView.setTileSource(TileSourceFactory.MAPNIK);
}
First: BoundingBox
BoundingBoxE6 bbox = new BoundingBoxE6(9.37398, 123.33761, 9.23948, 123.25035);
this.setScrollableAreaLimit(bbox);
Second: LimitScrollToGeographicArea.patch
Index: MapView.java
===================================================================
--- MapView.java (revision 944)
+++ MapView.java (working copy)
## -103,6 +103,8 ##
protected MapListener mListener;
+ protected Rect mScrollableAreaLimit;
+
// for speed (avoiding allocations)
private final Matrix mMatrix = new Matrix();
private final MapTileProviderBase mTileProvider;
## -505,6 +507,36 ##
mMapOverlay.setUseDataConnection(aMode);
}
+ /**
+ * Set the map to limit it's scrollable view to the specified BoundingBoxE6. Note that, like
+ * North/South bounds limiting, this allows an overscroll of half the screen size. This means
+ * each border can be scrolled to the center of the screen.
+ *
+ * #param boundingBox
+ * A lat/long bounding box to limit scrolling to, or null to remove any scrolling
+ * limitations
+ */
+ public void setScrollableAreaLimit(BoundingBoxE6 boundingBox) {
+ final int worldSize_2 = TileSystem.MapSize(MapViewConstants.MAXIMUM_ZOOMLEVEL) / 2;
+
+ // Clear scrollable area limit if null passed.
+ if (boundingBox == null) {
+ mScrollableAreaLimit = null;
+ return;
+ }
+
+ // Get NW/upper-left
+ final Point upperLeft = TileSystem.LatLongToPixelXY(boundingBox.getLatNorthE6() / 1E6,
+ boundingBox.getLonWestE6() / 1E6, MapViewConstants.MAXIMUM_ZOOMLEVEL, null);
+ upperLeft.offset(-worldSize_2, -worldSize_2);
+
+ // Get SE/lower-right
+ final Point lowerRight = TileSystem.LatLongToPixelXY(boundingBox.getLatSouthE6() / 1E6,
+ boundingBox.getLonEastE6() / 1E6, MapViewConstants.MAXIMUM_ZOOMLEVEL, null);
+ lowerRight.offset(-worldSize_2, -worldSize_2);
+ mScrollableAreaLimit = new Rect(upperLeft.x, upperLeft.y, lowerRight.x, lowerRight.y);
+ }
+
// ===========================================================
// Methods from SuperClass/Interfaces
// ===========================================================
## -772,10 +804,26 ##
//I am confused with these codes below, where should I declare it? Int x, y in the onCreate method?
x += (worldSize_2 * 2);
while (x > worldSize_2)
x -= (worldSize_2 * 2);
- while (y < -worldSize_2)
- y += (worldSize_2 * 2);
- while (y > worldSize_2)
- y -= (worldSize_2 * 2);
+ if (y < -worldSize_2)
+ y = -worldSize_2;
+ if (y > worldSize_2)
+ y = worldSize_2;
+
+ if (mScrollableAreaLimit != null) {
+ final int zoomDiff = MapViewConstants.MAXIMUM_ZOOMLEVEL - getZoomLevel();
+ final int minX = mScrollableAreaLimit.left >> zoomDiff;
+ final int minY = mScrollableAreaLimit.top >> zoomDiff;
+ final int maxX = mScrollableAreaLimit.right >> zoomDiff;
+ final int maxY = mScrollableAreaLimit.bottom >> zoomDiff;
+ if (x < minX)
+ x = minX;
+ else if (x > maxX)
+ x = maxX;
+ if (y < minY)
+ y = minY;
+ else if (y > maxY)
+ y = maxY;
+ }
super.scrollTo(x, y);
// do callback on listener
Another one:
scrollToMethod
public void scrollTo(int x, int y) {
int curZoomLevel = mZoomLevel;
final int worldSize_2 = TileSystem.MapSize(curZoomLevel) / 2;
Log.v("HELP", "Scrolling to X=" + x + " Y=" + y + " ZL=" + curZoomLevel + " - WW="+worldSize_2);
while (x < -worldSize_2)
x += (worldSize_2 * 2);
while (x > worldSize_2)
x -= (worldSize_2 * 2);
if (y < -worldSize_2)
y = -worldSize_2;
if (y > worldSize_2)
y = worldSize_2;
if (mScrollableAreaLimit != null) {
int targetZoomLevel = getZoomLevel();
final int zoomDiff = MapViewConstants.MAXIMUM_ZOOMLEVEL - targetZoomLevel;
//final int zoomDiff = MapViewConstants.MAXIMUM_ZOOMLEVEL - mZoomLevel;
final int minX = mScrollableAreaLimit.left >> zoomDiff;
final int minY = mScrollableAreaLimit.top >> zoomDiff;
final int maxX = mScrollableAreaLimit.right >> zoomDiff;
final int maxY = mScrollableAreaLimit.bottom >> zoomDiff;
Log.v("HELP", "Limit: minX=" + minX + " maxX=" + maxX + " minY=" + minY + " maxY=" + maxY + " ZL=" + curZoomLevel + " ZLTarget="+ targetZoomLevel + " ZD="+zoomDiff);
if (x < minX) {
Log.v("HELP", "!!! X=" + x + " minX=" + minX + " CORRECTION:" + (minX-x));
x = minX;
} else if (x > maxX) {
Log.v("HELP", "!!! X=" + x + " maxX=" + maxX + " CORRECTION:" + (maxX-x));
x = maxX;
}
if (y < minY) {
Log.v("HELP", "!!! Y=" + y + " minY=" + minY + " CORRECTION:" + (minY-y));
y = minY;
} else if (y > maxY) {
Log.v("HELP", "!!! Y=" + y + " maxY=" + maxY + " CORRECTION:" + (maxY-y));
y = maxY;
}
}
super.scrollTo(x, y);
// do callback on listener
if (mListener != null) {
final ScrollEvent event = new ScrollEvent(this, x, y);
mListener.onScroll(event);
}
}

First of all use this command on your terminal:
svn checkout http://osmdroid.googlecode.com/svn/branches/release_3_0_5
It will download a stable version
Then follow this to import contents of the folder you downloaded:
In Eclipse, right-click on the Package area, and select the following:
click on Import...
select General -> Existing Projects into Workspace
click Next
click Browse...
select the checked out projects' directories
osmdroid-android (import as a java project)
OSMMapTilePackager (import as a java project)
OpenStreetMapViewer (mport as an android project)
click OK
click Finish
Now open this java file-->
osmdroid-android/src/org/osmdroid/view/MapView.java
Now as stated in this patch file, modify MapView.java ( add
code wherever + , remove code wherever -)
Also modify computeScroll() in MapView.java as stated here
Now, to use this modified .java file, you need to create a new jar
file that you can include in your project
Here is a step by step process to create jar
Add this newly created jar file to your project's build path and you
are ready to use your modified jar
Now use this in your activity class:
BoundingBoxE6 bbox = new BoundingBoxE6(limit north, limit east, limit south, limit west);
mapView.setScrollableAreaLimit(bbox);