I am following this blog post and GitHub almost exactly:
Blog
Github
But when I run, take a picture and call this line:
var outputs = new float[tfLabels.Count];
tfInterface.Feed("Placeholder", floatValues, 1, 227, 227, 3);
tfInterface.Run(new[] { "loss" });
tfInterface.Fetch("loss", outputs);
The app actually crashes and generates the error below on the .Run line.
I get this error in the output window (and the app crashes):
04-04 17:39:12.575 E/TensorFlowInferenceInterface( 8017): Failed to
run TensorFlow inference with inputs:[Placeholder], outputs:[loss]
Unhandled Exception:
Java.Lang.IllegalArgumentException: Input to reshape is a tensor with
97556 values, but the requested shape requires a multiple of 90944
[[Node: block0_0_reshape0 = Reshape[T=DT_FLOAT, Tshape=DT_INT32,
_device="/job:localhost/replica:0/task:0/device:CPU:0"](block0_0_concat,
block0_0_reshape0/shape)]]
According to the posts I am reading from the searching I am doing on this error, I sort of understand this is due to the image not fitting the expected size exactly but in the example I am following, this is resized to fit 227x227 everytime and converted to float like in these lines:
var resizedBitmap = Bitmap.CreateScaledBitmap(bitmap, 227, 227, false).Copy(Bitmap.Config.Argb8888, false);
var floatValues = new float[227 * 227 * 3];
var intValues = new int[227 * 227];
resizedBitmap.GetPixels(intValues, 0, 227, 0, 0, 227, 227);
for(int i = 0; i < intValues.Length; i++)
{
var val = intValues[i];
floatValues[i * 3 + 0] = ((val & 0xFF) - 104);
floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - 117);
floatValues[i * 3 + 2] = (((val >> 16) & 0xFF) - 123);
}
So, I don't understand what is causing this or how to fix it. Please help!
UPDATE: I found out the issue is with my model or my labels. I found this out by simply swapping in the model and label file from the sample/github above while leaving all my code the same. When I did this, I no longer get the error. HOWEVER, this still doesn't tell me much. The error is not very explanatory to point me in a direction of what could be wrong with my model. I assume it is the model because the labels file is simply just a text file with labels on each line. I used Custom Vision Service on Azure to create my model. It trained fine and tests just fine on the web portal. I then exported it as TensorFlow. So, I am not sure what I could have done wrong or how to fix it.
Thanks!
After no answers here and several days of searching and trial and error, I have found the issue. In general, I guess this reshape error I was getting you can get if you are feeding the model with an image size other that it is expecting or setup to receive.
The issue is that, everything I have read says that typically you must feed the model with a 227 x 227 x 3 image. Then, I started noticing that size varies on some posts. Some people say 225 x 225 x 3, others say 250 x 250 x 3 and so on. I had tried those sizes as well with no luck.
As you can see in my edit in the question, I did have a clue. When using somebody else's pretrained model, my code works fine. However, when I use my custom model which I created on the Microsoft Azure CustomVision.ai site, I was getting this error.
So, I decided I would try to inspect the models to see what was different. I followed this post: Inspect a pre trained model
When I inspected the model that works using TensorBoard, I see that the input is 227 x 227 x 3 which is what I expected. However, when I viewed my model, I noticed that it was 224 x 224 x 3! I changed my code to resize the image to that size and it works! Problem went away.
So, to summarize, for some reason Microsoft Custom Vision service model generated a model to expect an image size of 224 x 224 x 3. I didn't see any documentation or setting for this. I also don't know if that number will change with each model. If you get a similar shape error, the first place I would check is the size of the image you are feeding your model and what it expects as an input. The good news is you can check your model, even if pre-trained, using TensorBoard and the post I linked above. Look at the input section, it should look something like this:
Hope this helps!
Related
I trained a quantized semantic segmentation model with my own dataset using the python scripts available on Deeplab's official Github page. I used the mobilenetv2_coco_voc_trainaug backbone. I checked the result model in Netron and this how the input an output looks:
As you can see, the output is an array of int64 with size of 257x257. From my understanding this array should contain the index of label with the highest probability at every array index, or am I missing something?
But when when I try to read this in Android, I got just zeros and ones, indiferent of what is in picture, people, cow, etc.
for (y in 0 until imageHeight) {
for (x in 0 until imageWidth) {
// resultBuffer is a ByteBuffer of size imageSize * imageSize * 8
val value = resultBuffer.getLong((y * imageWidth + x) * 8)
}
}
The result is not that accurate either, since I'm getting segmentation values where I shouldn't.
Any help would be appreciated!
Cant comment yet, lets try guess.
You are trying to use quantized model with int64 output. Output should be 8bit type
And yes, accuracy will drop with quantized model
EDIT: obviously I've made ann error in atan2 after many different approaches, but still curious to find formulas.
I've read multiple topics and articles, yet do not understand why atan and atan2 give different results after convertion into degrees. Here is my example (from CIElab colour space) x -79.7751, y 2.677374209. Calculation without any changes to code will give incorrect value in both cases:
atan(-79.7751 / 2.677374209) = -1,537 / or in degrees -88,07778762
atan2(-79.7751 , 2.677374209) = 3,1258 / or in degrees 179,0965
However, that example is from an article so we can check results. Basically answer should be 271.9222. I've found over the internet correction for atan, now I've got same results, yet I still have indeterminate value for x=y=0 in atan, hence some corrections should be made for atan2. Unfortunately I did not found any that would yield same result and and would be proven (I mean what could be found in literature). Most sources for CIElab say to use atan2 without any corrections and as you can below it is not correct in my particular case. Here is what I got on this:
atan(-79.7751 / 2.677374209) = -1,537 / for atan < 0 : atan + 2pi
hence atan = 4,7459 / or in degrees 271,922 ( BINGO! )
atan2(-79.7751 , 2.677374209) = ̶3̶,̶1̶2̶5̶8̶ 3,108 / or in degrees ̶1̶7̶9̶,̶0̶9̶6̶5̶ 178,0777876
now convert, for deg > 90 : 450 - deg
atan2 result 271,922
̶A̶s̶ ̶y̶o̶u̶ ̶c̶a̶n̶ ̶s̶e̶e̶,̶ ̶̶c̶l̶o̶s̶e̶ ̶b̶u̶t̶ ̶n̶o̶ ̶c̶i̶g̶a̶r̶̶.̶ ̶A̶l̶s̶o̶ ̶i̶t̶ ̶d̶i̶f̶f̶e̶r̶s̶ ̶b̶y̶ ̶1̶ ̶d̶e̶g̶r̶e̶e̶ ̶a̶n̶d̶ ̶s̶o̶m̶e̶ ̶c̶h̶a̶n̶g̶e̶.̶ My question is how exactly I can make a correction of atan2 to get proper results?
Your formulas may be wrong, depending on the library you use. See the C++ std for atan and atan2:
atan(y/x) returns in range {-pi/2, pi/2}
atan2(y,x) returns in range {0, 2pi}
You have exchanged x,y in both cases. This has the effect of this figure:
I'm an experienced native iOS developer making my first foray into Android through Unity. I'm trying to set up a custom shader, but I'm having some trouble with the Normal maps. I've got them working perfectly in the Unity simulator on my computer, but when I build to an actual device (Samsung Galaxy S8+), the Normal maps don't work at all.
I'm using Mars as my test case. Here's the model running in the simulator on my computer:
And here's a screenshot from my device, running exactly the same code.
I've done a LOT of research, and apparently using Normal maps on Android with Unity is not an easy thing. There are a lot of people asking about it, but almost every answer I've found has said the trick is to override the texture import settings, and force it to be "Truecolor" which seems to be "RGBA 32 Bit" according to Unity's documentation. This hasn't helped me, though.
Another thread suggested reducing the Asino Level to zero, and another suggested turning off Mip Maps. I don't know what either of those are, but neither helped.
Here's my shader code, simplified but containing all references to Normal mapping:
void surf (Input IN, inout SurfaceOutputStandard o) {
half4 d = tex2D (_MainTex , IN.uv_MainTex);
half4 n = tex2D (_BumpMap , IN.uv_BumpMap);
o.Albedo = d.rgb;
o.Normal = UnpackNormal(n);
o.Metallic = 0.0;
o.Smoothness = 0.0;
}
I've seen some threads suggesting replacements for the "UnpackNormal()" function in the shader code, indicating that it might not be the thing to do on Android or mobile in general, but none of the suggested replacements have changed anything for better or worse: the normal maps continue to work in the simulator, but not on the device.
I've even tried making my own normal maps programmatically from a grayscale heightmap, to try to circumvent any import settings I may have done wrong. Here's the code I used, and again it works in the simulator but not on the device.
public Texture2D NormalMap(Texture2D source, float strength = 10.0f) {
Texture2D normalTexture;
float xLeft;
float xRight;
float yUp;
float yDown;
float yDelta;
float xDelta;
normalTexture = new Texture2D (source.width, source.height, TextureFormat.RGBA32, false, true);
for (int y=0; y<source.height; y++) {
for (int x=0; x<source.width; x++) {
xLeft = source.GetPixel (x - 1, y).grayscale * strength;
xRight = source.GetPixel (x + 1, y).grayscale * strength;
yUp = source.GetPixel (x, y - 1).grayscale * strength;
yDown = source.GetPixel (x, y + 1).grayscale * strength;
xDelta = ((xLeft - xRight) + 1) * 0.5f;
yDelta = ((yUp - yDown) + 1) * 0.5f;
normalTexture.SetPixel(x,y,new Color(xDelta,yDelta,1.0f,yDelta));
}
}
normalTexture.Apply();
return normalTexture;
}
Lastly, in the Build Settings, I've got the Platform set to Android and I've tried it using Texture Compression set to both "Don't Override" and "ETC (default)". The former was the original setting and the latter seemed to be Unity's suggestion both by the name and in the documentation.
I'm sure there's just some flag I haven't checked or some switch I haven't flipped, but I can't for the life of me figure out what I'm doing wrong here, or why there would be such a stubborn difference between the simulator and the device.
Can anyone help a Unity newbie out, and show me how these damn Normal maps are supposed to work on Android?
Check under:
Edit -> Project Settings -> Quality
Android is usually set to Fastest.
I want to load .png file via asset manager which is provided by android sdk. AssetManager manager; /........./ BitmapFactory.decodeStream(manager.open(path));
It returns BGR format data but opengl es 2.0 uses RGB format so , Blue seems red , red seems blue, how odd.
Is there any solution for it?
I use Nvıdia Tegra 2 (Android 2.2) device for test the application along with c++ via JNI.
You must know the number of bits for colors, let's say n bit is for a color, so the first n bit represents BLUE, the second n bits represent GREEN and the final n bits represent RED in the input. You need to swap these bit groups into the correct order, like this:
output = (input << (2 * n)) + (input << n >> n) + (input >> (2 * n));
To be able to use this solution you need to find out how much is n.
Recent versions of OpenGL, also provide BGR input formats; OpenGL-ES not, unfortunatly. Since you're on Android you have to deal with OpenGL-ES.
If you're using a fragment shader it is also trivial to apply a rgb→bgr swizzle, which if probably the easiest way to overcome this problem.
I was surfing the net looking for a nice effect for turning pages on Android and there just doesn't seem to be one. Since I'm learning the platform it seemed like a nice thing to be able to do is this.
I managed to find a page here: http://wdnuon.blogspot.com/2010/05/implementing-ibooks-page-curling-using.html
- (void)deform
{
Vertex2f vi; // Current input vertex
Vertex3f v1; // First stage of the deformation
Vertex3f *vo; // Pointer to the finished vertex
CGFloat R, r, beta;
for (ushort ii = 0; ii < numVertices_; ii++)
{
// Get the current input vertex.
vi = inputMesh_[ii];
// Radius of the circle circumscribed by vertex (vi.x, vi.y) around A on the x-y plane
R = sqrt(vi.x * vi.x + pow(vi.y - A, 2));
// Now get the radius of the cone cross section intersected by our vertex in 3D space.
r = R * sin(theta);
// Angle subtended by arc |ST| on the cone cross section.
beta = asin(vi.x / R) / sin(theta);
// *** MAGIC!!! ***
v1.x = r * sin(beta);
v1.y = R + A - r * (1 - cos(beta)) * sin(theta);
v1.z = r * (1 - cos(beta)) * cos(theta);
// Apply a basic rotation transform around the y axis to rotate the curled page.
// These two steps could be combined through simple substitution, but are left
// separate to keep the math simple for debugging and illustrative purposes.
vo = &outputMesh_[ii];
vo->x = (v1.x * cos(rho) - v1.z * sin(rho));
vo->y = v1.y;
vo->z = (v1.x * sin(rho) + v1.z * cos(rho));
}
}
that gives an example (above) code for iPhone but I have no idea how I would go about implementing this on android. Could any of the Math gods out there please help me out with how I would go about implementing this in Android Java.
Is it possible using the native draw APIs, would I have to use openGL? Could I mimik the behaviour somehow?
Any help would be appreciated. Thanks.
****************EDIT**********************************************
I found a Bitmap Mesh example in the Android API demos: http://developer.android.com/resources/samples/ApiDemos/src/com/example/android/apis/graphics/BitmapMesh.html
Maybe someone could help me out on an equation to simply fold the top right corner inward diagnally across the page to create a similar effect that I can later apply shadows to to gie it more depth?
I'm doing some experimenting on page curl effect on Android using OpenGL ES at the moment. It's quite a sketch actually but maybe gives some idea how to implement page curl for your needs. If you're interested in 3D page flip implementation that is.
As for the formula you're referring to - I tried it out and didn't like the result too much. I'd say it simply doesn't fit small screen very well and started to hack a more simple solution.
Code can be found here:
https://github.com/harism/android_page_curl/
While writing this I'm in the midst of deciding how to implement 'fake' soft shadows - and whether to create a proper application to show off this page curl effect. Also this is pretty much one of the very few OpenGL implementations I've ever done and shouldn't be taken too much as a proper example.
I just created a open source project which features a page curl simulation in 2D using the native canvas: https://github.com/moritz-wundke/android-page-curl
I'm still working on it to add adapters and such to make it usable as a standalone view.
EDIT: Links updated.
EDIT: Missing files has been pushed to repo.
I'm pretty sure, that you'd have to use OpenGL for a nice effect. The basic UI framework's capabilities are quite limited, you can only do basic transformations (alpha, translate, rotate) on Views using animations.
Tho it might be possible to mimic something like that in 2D using a FrameLayout, and a custom View in it.