TLDR: How to use Variables from frozen tensorflow graphs on Android?
1. What I want to do
I have a Tensorflow model that keeps an internal state in multiple variables, created with: state_var = tf.Variable(tf.zeros(shape, dtype=tf.float32), name='state', trainable=False).
This state is modified during inference:
tf.assign(state_var, new_value)
I now want to deploy the model on Android. I was able to make the Tensorflow example App run. There, a frozen model is loaded, which works fine.
2. Restoring variables from frozen graph does not work
However, when you freeze a graph using the freeze_graph script, all Variables are converted to constants. This is fine for weights of the network, but not for the internal state. The inference fails with the following message. I interpret this as "assign does not work on constant tensors"
java.lang.RuntimeException: Failed to load model from 'file:///android_asset/model.pb'
at org.tensorflow.contrib.android.TensorFlowInferenceInterface.<init>(TensorFlowInferenceInterface.java:113)
...
Caused by: java.io.IOException: Not a valid TensorFlow Graph serialization: Input 0 of node layer_1/Assign was passed float from layer_1/state:0 incompatible with expected float_ref.
Luckily, you can blacklist Variables from being converted to constants. However, this also doesn't work because the frozen graph now contains uninitialized variables.
java.lang.IllegalStateException: Attempting to use uninitialized value layer_7/state
3. Restoring SavedModel does not work on Android
One last version I have tried is to use the SavedModel format which should contain both, a frozen graph and the variables. Unfortunately, calling the restore method does not work on Android.
SavedModelBundle bundle = SavedModelBundle.load(modelFilename, modelTag);
// produces error:
E/AndroidRuntime: FATAL EXCEPTION: main
Process: org.tensorflow.demo, PID: 27451
java.lang.UnsupportedOperationException: Loading a SavedModel is not supported in Android. File a bug at https://github.com/tensorflow/tensorflow/issues if this feature is important to you at org.tensorflow.SavedModelBundle.load(Native Method)
4. How can I make this work?
I don't know what else I can try. Here's what I would imagine, but I don't know how to make it work:
Figure out a way to initialize variables on Android
Figure out a different way to freeze the model, so that maybe the initializer op is also part of the frozen graph and can be run from Android
Find out if/how RNNs/LSTMs are implemented internally, because these should also have the same requirement of using variables during inference (and I assume LSTMs to be able to be deployed on Android).
???
I have solved this myself by going down a different route. To the best of my knowledge, the "variable" concept cannot be used in the same way on Android as I was used to in Python (e.g. you cannot initialize variables and then have an internal state of the network be updated during inference).
Instead, you can use placehlder and output nodes to preserve the state inside your Java code and feed it to the network on every inference call.
replace all tf.Variable occurances with tf.placeholder. The shape stays the same.
I also defined an additional node used to read the output. (Maybe you can simply read the placeholder itself, I haven't tried that.) tf.identity(inputs, name='state_output')
During inference on Android, you then feed the initial state into the network.
float[] values = {0, 0, 0, ...}; // zeros of the correct shape
inferenceInterface.feed('state', values, ...);
After inference, you read the resulting internal state of the network
float[] values = new float[output_shape];
inferenceInterface.fetch('state_output', values);
You then remember this output in Java to pass it into the 'state' placeholder for the next inference call.
Related
In Android, we can use HIDL data types in lieu of normal C++ data types. Their functionality remains mostly the same, and this proxy code is for behind-the-scenes serialization and IPC which HIDL takes care of.
Now when I try to check if a HIDL vector is empty, I get the error that empty is not a member of hidl_vec. How then can I check this? Some other functions that work on a normal vector work with hidl_vec such as foo.size(), foo.begin(), foo.end() but I prefer to not use the size check for this reason.
#include "gmock/gmock.h"
using ::android::hardware::hidl_vec;
using ::android::hardware::hidl_string;
hidl_vec<hidl_string> foo;
ASSERT_FALSE(foo.empty());
Throws the error:
error: no member named 'empty' in 'android::hardware::hidl_vecandroid::hardware::hidl_string'
ASSERT_FALSE(foo.empty());
I have created a small test app for this issue here: (https://github.com/Winghin2517/EpoxyExample2).
I would like to pass a list of objects into the epoxy controller so that I can generate a graph. I have however encountered this error when building the app:
error: Epoxy Processor Exception: Type in Iterable does not implement
hashCode. Type: kwaai.com.exampleepoxy_hashcodeequals.GraphData (View
Prop {view='HeaderView', name='setLineChart',
type=java.util.LinkedList})
Epoxy requires every model attribute to implement equals and hashCode
so that changes in the model can be tracked. If you want the attribute
to be excluded, use the option 'DoNotHash'. If you want to ignore this
warning use the option 'IgnoreRequireHashCode'
I think it is because I using the #ModelProp on a List of Objects (LinkedList of GraphDataFeed) and not on a primitive type as per the example app from Epoxy.
#ModelProp
public void setLineChart(LinkedList<GraphData> graphDataFeed) { }
So I folllowed the options and modified my #ModelProp to reflect this:
#ModelProp(options = ModelProp.Option.IgnoreRequireHashCode)
After the change the app builds and runs correctly. You can see the graph below.
However, I do not want to ignore the attribute as I understand Epoxy uses Diffing to update the models in the recyclerview: https://github.com/airbnb/epoxy/wiki/Diffing
Ignoring the attribute might mean that my models do not get updated correctly in the recyclerview.
In the guidance material here (https://github.com/airbnb/epoxy/wiki/Epoxy-Models#annotations), I see it says:
A model's state is determined by its equals and hashCode
implementations, which is based on the value of all of the model's
properties.
This state is used in diffing to determine when a model has changed so
Epoxy can update the view.
These methods are generated so you don't have to created them
manually.
Why are these methods not generated for me then and if they are not generated, how do I generate these methods myself to get rid of the error?
Your GraphData class needs to implement equals and hashcode. It says this right in the error message you copied
Type in Iterable does not implement hashCode. Type: kwaai.com.exampleepoxy_hashcodeequals.GraphData
I'm using OpenGL ES 2 via jni.
When going over the OpenGL Trace view of my frame rendering, I saw this error in the console:
error applying transformations for glUniform1f(location = 6, x = 1.000000)
java.lang.RuntimeException: No such property: PROGRAM_STATE/PROGRAMS/${program}/ACTIVE_UNIFORMS/6/UNIFORM_VALUE
It seems like I'm calling uniform for a non existing variable, so I went over all of the usages I have for this call, and also tried checking with 'glError', but I found nothing.
How can I know what is causing this?
I suspect you are trying to upload to a non-existent unifom in the current program.
Did you call glGetUniformLocation() on all the symbol names in this linked version of the program?
You can't safely make assumptions that, e.g. they increment, so you really do need to call glGetUniformLocation() for all of them (or set binding locations directly, but that's not available until OpenGL ES 3.x).
I'm currently searching for a bug which is based on a OpenGL program being invalid. But it is difficult to find the source of the problem without knowing where it might come from.
When I create the program it is valid. Furthermore I don't use glDeleteProgram().
To determine wether my program is valid or not I use glIsProgram().
Generally in OpenGL, objects are not created until they are first bound.
glGenTextures (...) for instance, reserves and returns one or more names for texture objects but those names do not become actual textures until bound to something like glBindTexture (GL_TEXUTRE_2D, ...). In other words, the names are reserved but what they reference is not instantiated/initialized yet.
What glIs* (...) actually returns is whether the name you pass it is the name of a created object. Thus, if you never bind an object it is never created, and this function will return GL_FALSE.
glUseProgram (...) is the function that OpenGL uses to bind GLSL program objects. Even though GLSL program and shader objects work differently from all other types of OpenGL objects it is very likely that glIsProgram (...) is not going to return GL_TRUE until sometime after you have called glUseProgram (...) on it at least once.
Incidentally, to validate a program I would suggest you use glValidateProgram (...) instead.
Here is some C pseudo-code that demonstrates how to use glValidateProgram:
GLint valid = GL_FALSE;
glValidateProgram (program);
glGetProgramiv (program, GL_VALIDATE_STATUS, &valid);
If valid is GL_TRUE after this, your program is valid (e.g. it was successfully linked).
I have got some strange and unexpected results from my program in OpenGL ES for android for example in the code below:
matrix = ThisRot.get();
gl.glMultMatrixf(matrix, 0);
currentRotation.toMatrix(matrix);
temp.set(matrix);
I set the matrix value before I use it as an argument for gl.glMultMatrixf and after that I change the value of matrix and use it for another purpose, but it has effect an the way the object rotate so it should have effect on gl.glMultMatrixf(). and that's not the only one, some other places in my code I had this unexpected results. so I have thought maybe these happen due to mutual exclusion and multitreading and those kind of things.
am I right? should we worry about multithreading when we code in Opengl ES for android? How can I avoid these kind of problems.
Of course you should worry about multithreading. In particular, Android creates its own GLThread for rendering, when you attach a GLRenderer-derived class to a GLSurfaceView using the setRenderer() function.
In fact, multithreading can cause crashes (not only unexpected behavior) in your programs especially when you loop through arrays adding/removing objects and such.
Check if you are modifying the same data inside the onDrawFrame function of your GLRenderer and your own thread. If you are, try adding the following around the modified code (in both threads):
synchronize(variable) {
modify(variable);
}
This will lock the variable throughout the modify() function until it ends the block. Try not to overuse it, though, only in places where you need it. One thread will block the other one until it's finished!