Develop Reference

Use Variables from frozen tensorflow graph on Android

TLDR: How to use Variables from frozen tensorflow graphs on Android?
1. What I want to do
I have a Tensorflow model that keeps an internal state in multiple variables, created with: state_var = tf.Variable(tf.zeros(shape, dtype=tf.float32), name='state', trainable=False).
This state is modified during inference:
tf.assign(state_var, new_value)
I now want to deploy the model on Android. I was able to make the Tensorflow example App run. There, a frozen model is loaded, which works fine.
2. Restoring variables from frozen graph does not work
However, when you freeze a graph using the freeze_graph script, all Variables are converted to constants. This is fine for weights of the network, but not for the internal state. The inference fails with the following message. I interpret this as "assign does not work on constant tensors"
java.lang.RuntimeException: Failed to load model from 'file:///android_asset/model.pb'
at org.tensorflow.contrib.android.TensorFlowInferenceInterface.<init>(TensorFlowInferenceInterface.java:113)
...
Caused by: java.io.IOException: Not a valid TensorFlow Graph serialization: Input 0 of node layer_1/Assign was passed float from layer_1/state:0 incompatible with expected float_ref.
Luckily, you can blacklist Variables from being converted to constants. However, this also doesn't work because the frozen graph now contains uninitialized variables.
java.lang.IllegalStateException: Attempting to use uninitialized value layer_7/state
3. Restoring SavedModel does not work on Android
One last version I have tried is to use the SavedModel format which should contain both, a frozen graph and the variables. Unfortunately, calling the restore method does not work on Android.
SavedModelBundle bundle = SavedModelBundle.load(modelFilename, modelTag);
// produces error:
E/AndroidRuntime: FATAL EXCEPTION: main
Process: org.tensorflow.demo, PID: 27451
java.lang.UnsupportedOperationException: Loading a SavedModel is not supported in Android. File a bug at https://github.com/tensorflow/tensorflow/issues if this feature is important to you at org.tensorflow.SavedModelBundle.load(Native Method)
4. How can I make this work?
I don't know what else I can try. Here's what I would imagine, but I don't know how to make it work:
Figure out a way to initialize variables on Android
Figure out a different way to freeze the model, so that maybe the initializer op is also part of the frozen graph and can be run from Android
Find out if/how RNNs/LSTMs are implemented internally, because these should also have the same requirement of using variables during inference (and I assume LSTMs to be able to be deployed on Android).
???

I have solved this myself by going down a different route. To the best of my knowledge, the "variable" concept cannot be used in the same way on Android as I was used to in Python (e.g. you cannot initialize variables and then have an internal state of the network be updated during inference).
Instead, you can use placehlder and output nodes to preserve the state inside your Java code and feed it to the network on every inference call.
replace all tf.Variable occurances with tf.placeholder. The shape stays the same.
I also defined an additional node used to read the output. (Maybe you can simply read the placeholder itself, I haven't tried that.) tf.identity(inputs, name='state_output')
During inference on Android, you then feed the initial state into the network.
float[] values = {0, 0, 0, ...}; // zeros of the correct shape
inferenceInterface.feed('state', values, ...);
After inference, you read the resulting internal state of the network
float[] values = new float[output_shape];
inferenceInterface.fetch('state_output', values);
You then remember this output in Java to pass it into the 'state' placeholder for the next inference call.

Android, import external library to NDK

i am making an Android Shapefile reader app on Android with the use of NDK. I want to use C++ to parse shape files. I found library "Shapefile C Library". Github: https://github.com/sw897/shapelib.
I am using Android studio and have no idea how to import it to my Android project so i could use functions described in: http://shapelib.maptools.org/shp_api.html
Any tips?

First, start hacking away at the Hello-JNI example from Google:
https://github.com/googlesamples/android-ndk/tree/master/hello-jni
Use that as a test bed.
Then, create a Java Class with public native methods that let you interact with your library. Something of a high level API, probably to pass a file name or buffer to SHPOpenLL and return a pointer to the ShapeFile context (ShapeHandle). Looks like your library is written in C, so you should be able to write a similar interface to query the shapefile passing a (jint) c-style cast of your ShapeHandle pointer.
Next, you need to play around with javah to generate the header for your shapefile interface. Once the header is generated, you can use it to implement your interface in a .cc file. From there you will basically make Java calls to your C interface and it will return jobjects. (jint, jbool, jstring, etc...)
I'm looking at the ShapeLib that you want to use and while it's easy enough, there will be some gotchas.
You will need to implement SAHooks for file I/O. I suggest looking at NVFile for an example how to access APK files (or downloaded files). You will need to init it using activity.context.assetmanager. Then use nv_file to wrap FRead/FSeek/etc...
Passing info back to java is going to be tough. Personally, I would build a Java class to hold the shape information and fill it out on the C side. However, you might be tempted to query these parameters from the SHPObject one at a time. Will be slow, but less error prone.
E.g.
// Java
MyJavaShapeObject obj = new MyJavaShapeObject();
_c_retrieveShapeObj((jint)pShapeFile, obj);
// C
java_blah_blah_c_retrieveShapeObj(JNIEnv* env, jclass activity, jint theShapeFile, jobject theObj){
ShapeHandle pShapeFileHandle = (ShapeHandle)theShapeFile; // c style conversion
// http://stackoverflow.com/questions/11647646/how-to-use-the-jni-to-change-the-fields-of-a-java-class
// http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/functions.html#wp16613
// http://stackoverflow.com/a/36759159/7949696
}

Making sense of symbol table entry (when header function is not present)

I have a JNI function in a C++ library.
When I create the library using cmake (but forget to put function declaration) in the header file, the library is created successfully.
When I look for function name in library symbol table I get following output
nm libuserlibrary.so | grep printxx
00506e60 T _Z60Java_com_example_user_myapplication_userlibrary_printxxP7JNIEnv_P8_jobject
But when I give function declaration in the header file, I get following output
nm libuserlibrary.so | grep printxx
00506e50 T Java_com_example_user_myapplication_userlibrary_printxx
Why is there this difference between these two symbol table entries? What is the purpose of _Z60 and P7JNIEnv_P8_jobject around the function name?
I also noticed that in the first case, I cannot call the JNI funciton from Android java code (it says unsatisfied-linker-error, implementation not found).

C++ allows function overloads and namespaces like Java does. So, it annotated the function name with parameter information so the linker can bind to the correct overload.
JNI was designed for C which does not allow function overloads or namespaces. So it invented it's own annotation system and provides the javah tool to help you use it. The header can be used in C++ too. C++ was designed to allow some functions to be called as if they written in C. The header has code that indicates that to the compiler. So, put it all together and you can write Java-callable functions in C++.

getReference() vs. getChild()

I was wondering what the difference between database.getReference("foo/bar/123") and database.getReference("foo").child("bar").child("123") is?
I'm assuming that the later one will load the complete "foo" object whereas database.getReference("foo/bar/123") just loads the "123" object?
Is my assumption correct or what is the correct / most efficient way to only load data of "123"?

The two are equivalent. You can inspect this manually this by printing the toString() format for both References.
References are cheap - there's nothing inefficient about either solution. Neither one has yet loaded any data. A Reference is just a pointer to a location in the database.

It should not make a difference, a reference is not actually accessed when instantiated. This is the most relevant document I can find,
https://firebase.google.com/docs/reference/node/firebase.database.Reference
The docs don't say it explicitly, but requests are only performed when using the .set() or .on() methods

Does not GLES20 Have GL_HALF_FLOAT_OES?

My question is simple. Why isn't there any GL_HALF_FLOAT_OES constant on Java side in GLES20.java class? However, "GLES2/gl2ext.h" on native side, has it... what happens if i specify value of this C constant on Java side manually? Do you think it is not valid and this is why they had not included it within GLES20.java class?

Float point textures are not a part of OpenGL ES 2.0 specs that's why they are not included in gl2ext.h. And please note that they are available only if GPU supports GL_EXT_color_buffer_half_float extension so you should check presence of this extension first.
Basically, you won't find constants for a lot of vendor-specific stuff too. This doesn't mean these values are invalid and it is absolutely OK to define these constants in code (my code has a lot of such constants, to name a few - constants for DXT/PVRTC compression and Tegra's CSAA).