Develop Reference

Use Variables from frozen tensorflow graph on Android

TLDR: How to use Variables from frozen tensorflow graphs on Android?
1. What I want to do
I have a Tensorflow model that keeps an internal state in multiple variables, created with: state_var = tf.Variable(tf.zeros(shape, dtype=tf.float32), name='state', trainable=False).
This state is modified during inference:
tf.assign(state_var, new_value)
I now want to deploy the model on Android. I was able to make the Tensorflow example App run. There, a frozen model is loaded, which works fine.
2. Restoring variables from frozen graph does not work
However, when you freeze a graph using the freeze_graph script, all Variables are converted to constants. This is fine for weights of the network, but not for the internal state. The inference fails with the following message. I interpret this as "assign does not work on constant tensors"
java.lang.RuntimeException: Failed to load model from 'file:///android_asset/model.pb'
at org.tensorflow.contrib.android.TensorFlowInferenceInterface.<init>(TensorFlowInferenceInterface.java:113)
...
Caused by: java.io.IOException: Not a valid TensorFlow Graph serialization: Input 0 of node layer_1/Assign was passed float from layer_1/state:0 incompatible with expected float_ref.
Luckily, you can blacklist Variables from being converted to constants. However, this also doesn't work because the frozen graph now contains uninitialized variables.
java.lang.IllegalStateException: Attempting to use uninitialized value layer_7/state
3. Restoring SavedModel does not work on Android
One last version I have tried is to use the SavedModel format which should contain both, a frozen graph and the variables. Unfortunately, calling the restore method does not work on Android.
SavedModelBundle bundle = SavedModelBundle.load(modelFilename, modelTag);
// produces error:
E/AndroidRuntime: FATAL EXCEPTION: main
Process: org.tensorflow.demo, PID: 27451
java.lang.UnsupportedOperationException: Loading a SavedModel is not supported in Android. File a bug at https://github.com/tensorflow/tensorflow/issues if this feature is important to you at org.tensorflow.SavedModelBundle.load(Native Method)
4. How can I make this work?
I don't know what else I can try. Here's what I would imagine, but I don't know how to make it work:
Figure out a way to initialize variables on Android
Figure out a different way to freeze the model, so that maybe the initializer op is also part of the frozen graph and can be run from Android
Find out if/how RNNs/LSTMs are implemented internally, because these should also have the same requirement of using variables during inference (and I assume LSTMs to be able to be deployed on Android).
???

I have solved this myself by going down a different route. To the best of my knowledge, the "variable" concept cannot be used in the same way on Android as I was used to in Python (e.g. you cannot initialize variables and then have an internal state of the network be updated during inference).
Instead, you can use placehlder and output nodes to preserve the state inside your Java code and feed it to the network on every inference call.
replace all tf.Variable occurances with tf.placeholder. The shape stays the same.
I also defined an additional node used to read the output. (Maybe you can simply read the placeholder itself, I haven't tried that.) tf.identity(inputs, name='state_output')
During inference on Android, you then feed the initial state into the network.
float[] values = {0, 0, 0, ...}; // zeros of the correct shape
inferenceInterface.feed('state', values, ...);
After inference, you read the resulting internal state of the network
float[] values = new float[output_shape];
inferenceInterface.fetch('state_output', values);
You then remember this output in Java to pass it into the 'state' placeholder for the next inference call.

Android Opengl ES 3.0 Pixel Buffer Object And glReadPixel

I want to improve glReadPixels() performance using PBO(for Android 4.3+), As glReadPixels takes about 30~50ms(1280*960) for each frame in my test device.But I can't use glReadPixels directly in Java, since the last param cannot be zero or "null". I also try to use ndk as mentioned here. But the last param needs to be (void*) either in my ndk's GLES3/gl3.h. So the question is how can I use glReadPixels to make PBO work? Is this the right solution for better performance?

You will need to cast the last parameter from int to void* explicitly in your c++ codes.

MEDIA_PROJECTION_SERVICE not valid in call to getSystemService()

I am trying to develop an application that requires the ability to capture screen content. I'm targeting lollipop to avoid the requirement for root. When trying to get an instance of the MediaProjectionManager via a call to getSystemService() I am getting the following error reported in Android Studio:
Must be one of: Context.POWER_SERVICE, Context.WINDOW_SERVICE, Context.LAYOUT_INFLATER_SERVICE, Context.ACCOUNT_SERVICE, Context.ACTIVITY_SERVICE, Context.ALARM_SERVICE, Context.NOTIFICATION_SERVICE, Context.ACCESSIBILITY_SERVICE, Context.CAPTIONING_SERVICE, Context.KEYGUARD_SERVICE, Context.LOCATION_SERVICE, Context.SEARCH_SERVICE, Context.SENSOR_SERVICE, Context.STORAGE_SERVICE, Context.WALLPAPER_SERVICE, Context.VIBRATOR_SERVICE, Context.CONNECTIVITY_SERVICE, Context.WIFI_SERVICE, Context.WIFI_P2P_SERVICE, Context.NSD_SERVICE, Context.AUDIO_SERVICE, Context.MEDIA_ROUTER_SERVICE, Context.TELEPHONY_SERVICE, Context.CLIPBOARD_SERVICE, Context.INPUT_METHOD_SERVICE, Context.TEXT_SERVICES_MANAGER_SERVICE, Context.DROPBOX_SERVICE, Context.DEVICE_POLICY_SERVICE, Context.UI_MODE_SERVICE, Context.DOWNLOAD_SERVICE, Context.NFC_SERVICE, Context.BLUETOOTH_SERVICE, Context.USB_SERVICE, Context.INPUT_SERVICE, Context.DISPLAY_SERVICE, Context.USER_SERVICE, Context.PRINT_SERVICE less... (Ctrl+F1)
Reports two types of problems:
* Supplying the wrong type of resource identifier. For example, when calling Resources.getString(int id), you should be passing R.string.something, not R.drawable.something.
* Passing the wrong constant to a method which expects one of a specific set of constants. For example, when calling View#setLayoutDirection, the parameter must be android.view.View.LAYOUT_DIRECTION_LTR or android.view.View.LAYOUT_DIRECTION_RTL.
I am currently at a loss as to why this constant is not considered valid, it's there as an autocomplete option, so it's present, and it's shown in all sample code I have seen for screen capture in lollipop. I have verified that the project setup specifies Android SDK 21 as min and target. Is there something else obvious/stupid I might be missing that would cause this error?
UPDATE: Took the exact same code to Eclipse and it works without issue. So this is related to something in Android Studio specifically it seems.

I get this error while getting Context.BLUETOOTH_SERVICEand the error doc (Must be one of..) contains Context.BLUETOOTH_SERVICE though.
This is not the way Android Studio "1.2" should work. :#
Anyway, its an Inspection bug, Constant and Resource Type mismatch (How in the hell Bluetooth is a resource in android context).
You can suppress this for Class/Method/Statement, for statement, add #SuppressWarnings("ResourceType") above or before the statement.
Another approach:
Goto Settings>>Editor>>Inspection>>Android>>Constant and Resource Type Mismatches and make the severity to anything but Error, probably Warning or Weak Warning.
(Though it fixes the error issue, but I want this mismatch to be an error when it really happens.)

Run into the same problem, it is so strange, there are no any other threads talking about this problem.
Well, actually you can just ignore this error and still run the program, even with
the red marks on it

What are reasons for a OpenGL program to become invalid?

I'm currently searching for a bug which is based on a OpenGL program being invalid. But it is difficult to find the source of the problem without knowing where it might come from.
When I create the program it is valid. Furthermore I don't use glDeleteProgram().
To determine wether my program is valid or not I use glIsProgram().

Generally in OpenGL, objects are not created until they are first bound.
glGenTextures (...) for instance, reserves and returns one or more names for texture objects but those names do not become actual textures until bound to something like glBindTexture (GL_TEXUTRE_2D, ...). In other words, the names are reserved but what they reference is not instantiated/initialized yet.
What glIs* (...) actually returns is whether the name you pass it is the name of a created object. Thus, if you never bind an object it is never created, and this function will return GL_FALSE.
glUseProgram (...) is the function that OpenGL uses to bind GLSL program objects. Even though GLSL program and shader objects work differently from all other types of OpenGL objects it is very likely that glIsProgram (...) is not going to return GL_TRUE until sometime after you have called glUseProgram (...) on it at least once.
Incidentally, to validate a program I would suggest you use glValidateProgram (...) instead.
Here is some C pseudo-code that demonstrates how to use glValidateProgram:
GLint valid = GL_FALSE;
glValidateProgram (program);
glGetProgramiv (program, GL_VALIDATE_STATUS, &valid);
If valid is GL_TRUE after this, your program is valid (e.g. it was successfully linked).

Does multithreading affect programs in OpenGL ES for android?

I have got some strange and unexpected results from my program in OpenGL ES for android for example in the code below:
matrix = ThisRot.get();
gl.glMultMatrixf(matrix, 0);
currentRotation.toMatrix(matrix);
temp.set(matrix);
I set the matrix value before I use it as an argument for gl.glMultMatrixf and after that I change the value of matrix and use it for another purpose, but it has effect an the way the object rotate so it should have effect on gl.glMultMatrixf(). and that's not the only one, some other places in my code I had this unexpected results. so I have thought maybe these happen due to mutual exclusion and multitreading and those kind of things.
am I right? should we worry about multithreading when we code in Opengl ES for android? How can I avoid these kind of problems.

Of course you should worry about multithreading. In particular, Android creates its own GLThread for rendering, when you attach a GLRenderer-derived class to a GLSurfaceView using the setRenderer() function.
In fact, multithreading can cause crashes (not only unexpected behavior) in your programs especially when you loop through arrays adding/removing objects and such.
Check if you are modifying the same data inside the onDrawFrame function of your GLRenderer and your own thread. If you are, try adding the following around the modified code (in both threads):
synchronize(variable) {
modify(variable);
}
This will lock the variable throughout the modify() function until it ends the block. Try not to overuse it, though, only in places where you need it. One thread will block the other one until it's finished!