Develop Reference

Fastest way to run recurrent neural network (inference) on mobile device

What I have: A trained recurrent neural network in Tensorflow.
What I want: A mobile application that can run this network as fast as possible (inference mode only, no training).
I believe there are multiple ways how I can accomplish my goal, but I would like you feedback/corrections and additions because I have never done this before.
Tensorflow Lite. Pro: Straight forward, available on Android and iOS. Contra: Probably not the fastest method, right?
TensorRT. Pro: Very fast + I can write custom C code to make it faster. Contra: Used for Nvidia devices so no easy way to run on Android and iOS, right?
Custom Code + Libraries like openBLAS. Pro: Probably very fast and possibility to link to it on Android on iOS (if I am not mistaken). Contra: Is there much use for recurrent neural networks? Does it really work well on Android + iOS?
Re-implement Everything. I could also rewrite the whole computation in C/C++ which shouldn't be too hard with recurrent neural networks. Pro: Probably the fastest method because I can optimize everything. Contra: Will take a long time and if the network changes I have to update my code as well (although I am willing to do it this way if it really is the fastest). Also, how fast can I make calls to libraries (C/C++) on Android? Am I limited by the Java interfaces?
Some details about the mobile application. The application will take a sound recording of the user, do some processing (like Speech2Text) and output the text. I do not want to find a solution that is "fast enough", but the fastest option because this will happen over very large sound files. So almost every speed improvement counts. Do you have any advice, how I should approach this problem?
Last question: If I try to hire somebody to help me out, should I look for an Android/iOS-, Embedded- or Tensorflow- type of person?

1. TensorflowLite
Pro: it uses GPU optimizations on Android; fairly easy to incorporate into Swift/Objective-C app, and very easy into Java/Android (just adding one line in gradle.build); You can transform TF model to CoreML
Cons: if you use C++ library - you will have some issues adding TFLite as a library to your Android/Java-JNI (there is no native way to build such library without JNI); No GPU support on iOS (community works on MPS integration tho)
Also here is reference to TFLite speech-to-text demo app, it could be useful.
2. TensorRT
It uses TensorRT uses cuDNN which uses CUDA library. There is CUDA for Android, not sure if it supports the whole functionality.
3. Custom code + Libraries
I would recommend you to use Android NNet library and CoreML; in case you need to go deeper - you can use Eigen library for linear algebra. However, writing your own custom code is not beneficial in the long term, you would need to support/test/improve it - which is a huge deal, more important than performance.
Re-implement Everything
This option is very similar to the previous one, implementing your own RNN(LSTM) should be fine, as soon as you know what you are doing, just use one of the linear algebra libraries (e.g. Eigen).
The overall recommendation would be to:**
try to do it server side: use some lossy compression and serverside
speech2text;
try using Tensorflow Lite; measure performance, find bottlenecks, try to optimize
if some parts of TFLite would be too slow - reimplement them in custom operations; (and make PR to the Tensorflow)
if bottlenecks are on the hardware level - goto 1st suggestion

Maybe you should try this lib, it can run on android and ios devices.
https://github.com/Tencent/TNN

Why do Android services not run in separate threads?

Why does a service not run in it's own thread under Android? And what were the design considerations?

It is all about Flexibility: I assume you are familiar with so many Frameworks like for example e.g. Volley, Retrofit etc. These Frameworks has their own Thread implementation behind scene so if Service by default is executed in a separate Thread then it is overkill. Why? because the threading is handled by the framework already so why the need to have another thread?
See also Why is creating a Thread said to be expensive?. And put also into consideration that you are doing Threading in an Android device and not into a powerful Desktop.

Take a look at IntentService I think this is that you are looking for.

It gives you more flexibility. For example you could run each request in a seperate thread with special priorities or you could store the requests in a queue and only start the next thread if the previous one has finished. A IntentService provides a default configuration that in the most cases fits your needs from scratch.(http://developer.android.com/guide/components/services.html). Such decisions depend on your use case.

Updating UI from background thread using native code entirely (no JNI)?

I am writing an Android application that does much of a processing in the background thread. The calculations are performed in the native code using C++. I want to update UI with the partial results during calculations.
I am able to do so through JNI, namely set reference to UI object in java and then call a method of this object from native code trough JNI. I am looking for a more efficient solution without any JNI calls in the rendering part (the basic setup and an activation entry point will have to be in java).
Is it possible to change execution thread from background to UI entirely in native code without using JNI (only in the rendering part)?
Thanks for your answers. Much appreciated.
EDIT:
I am considering using OpenGL to render calculated content (sort of video frames) in my view. In that case I would probably want to use eglSwapBuffers() method that is available in EGL library from 2.3 onwards. The biggest problem is, I think, how to easily switch from background "calculation" thread to UI "open GL" thread in native code without JNI overhead. What would you recommend? Thanks for your help!
PSEUDOCODE:
Here is some pseudocode that helps to illustrate what I want to achieve here. It is more like a threading issue but Android framework comes into play as well.
// background thread
void Java_com_example_hellojni_HelloJni_stringFromJNI( JNIEnv* env,
jobject thiz)
{
//long calculation
struct result * res = calculate();
//want to update ui using opengl on ui thread
updateGL(res);
}
//want to run it on ui thread as opengl can be run only on ui thread
void updateGL(struct result * res)
{
//some opengl code with "res"
someGLcodeWithResult(res);
//render surface
eglSwapBuffers();
}
EDIT 2:
As the bounty slowly draws to close, one more clarification. There are couple of ways to call the updateGL method above. The most typical one is to use GLSurfaceView in java code. It would require setting a renderer (setRenderer()) and then overriding onDrawFrame with some code to call JNI/native layer. This way for every frame being rendered one JNI call gets executed.
I would like to do the rendering a little differently. I want to call updateGL without using java/JNI code at all and use only the native layer (as presented in the excerpt above). The biggest problem for me now is how to run the updateGL on UI thread (required by OpenGL) without the java callback. Is it at all feasible?

I suggest taking a look at the Mozilla Fennec source. There is a minimal Java/JNI shim that allows rendering on Android, but all the work is done via native code (regular ol' Mozilla code)

Great thoughts, but try and understand this thing
You have code in C++ in native
UI in java runs in VM
Now what you are asking is C++ code to interact with UI in Java VM which is another program to communicate. The only way to do it would be some interprocess communication. Having said that, now go figure that out which suits you best. JNI is fast and reliable, since u dont want to go that way, figure out any other IPC that suits you.

I'm facing the same problem right now. Perhaps this might help:
When you take a look at the STABLE-APIS.html in the latest NDK it says:
> V. Android-9 Stable Native APIs:
All the APIs listed below are available for developing native code that runs
on Android > 2.3 system images and above.
The EGL graphics library:
EGL provides a native platform interface to allocate and manage OpenGLES
surfaces. For more information about its features, please see:
http://www.khronos.org/egl
In a nutshell, this will allow you to do the following directly from
native code:
- List supported EGL configurations
- Allocate and release OpenGLES surfaces
- Swap/Flip surfaces for display (eglSwapBuffers)
So if you're targeting Android-9 then you could try to use the EGL lib in native code.
(Let me know if this works :))

Is it possible to change execution thread from background to UI entirely in native code without using JNI?
You cannot modify widgets from native code.

Is SQL or general file access appropriate in the Android main UI thread?

I'm trying to follow Android best practices, so in debug mode I turn all the following on:
StrictMode.setThreadPolicy(new StrictMode.ThreadPolicy.Builder().detectAll().penaltyLog().build()); //detect and log all thread violations
StrictMode.setVmPolicy(new StrictMode.VmPolicy.Builder().detectAll().penaltyLog().build()); //detect and log all virtual machine violations
Android now yells at me when I try to use any sort of file access or SQL in the main (UI) thread. But I see so many recommendations to use file access and/or SQL in the main thread. For example, the main activity should load default preference values inside onCreate() in case they haven't been set yet:
PreferenceManager.setDefaultValues(context, resId, readAgain);
Oops---that results in a file access on the first application execution, because onCreate() is called on the UI thread. The only way around it I can see is to start a separate thread---which introduces a race condition with other UI code that might read the preferences and expect the default values to already be set.
Think also of services such as the DownloadManager. (Actually, it's so buggy that it's useless in real life, but let's pretend it works for a second.) If you queue up a download, you get an event (on the main thread) telling you a download has finished. To actually get information about that download (it only gives you a download ID), you have to query the DownloadManager---which involves a cursor, giving you an error if you have a strict policy turned on.
So what's the story---is it fine to access cursors in the main thread? Or is it a bad thing, and half the Android development team and Android book authors forgot about that?

The only way around it I can see is to start a separate thread---which introduces a race condition with other UI code that might read the preferences and expect the default values to already be set.
Then use an AsyncTask, putting the setDefaultValues() call in doInBackground() and the "other UI code that might read the preferences" in onPostExecute().
To actually get information about that download (it only gives you a download ID), you have to query the DownloadManager---which involves a cursor, giving you an error if you have a strict policy turned on.
So query the DownloadManager in a background thread.
So what's the story---is it fine to access cursors in the main thread?
That depends on your definition of "fine".
On Android 1.x and most 2.x devices, the filesystem used is YAFFS2, which basically serializes all disk access across all processes. The net effect is that while your code may appear sufficiently performant in isolation, it appears sluggish at times in production because of other things going on in the background (e.g., downloading new email).
While this is a bit less of an issue in Android 3.x and above (they switched to ext4), there's no question that flash I/O is still relatively slow -- it will just be a bit more predictably slow.
StrictMode is designed to point out where sluggishness may occur. It is up to you to determine which are benign and which are not. In an ideal world, you'd clean up them all; in an ideal world, I'd have hair.
Or is it a bad thing, and half the Android development team and Android book authors forgot about that?
It's always been a "bad thing".
I cannot speak for "half the Android development team". I presume that, early on, they expected developers to apply their existing development expertise to detect sluggish behavior -- this is not significantly different than performance issues in any other platform. Over time, they have been offering more patterns to steer developers in a positive path (e.g., the Loader framework), in addition to system-level changes (e.g., YAFFS2->ext4) to make this less of a problem. In part, they are trying to address places where Android introduces distinct performance-related challenges, such as the single-threaded UI.
Similarly, I cannot speak for all Android book authors. I certainly didn't focus on performance issues in early editions of my books, as I was focusing on Android features and functions. Over time, I have added more advice in these areas. I have also contributed open source code related to these topics. In 2012, I'll be making massive revisions to my books, and creating more open source projects, to continue addressing these issues. I suspect, given your tone, that I (and probably others) are complete failures in your eyes in this regard, and you are certainly welcome to your opinion.

How to measure Phonegap UI performance?

I want to make performance improvements to my Phonegap application, currently targeting Android, but I want to base my actions on some measureable evidence.
Are there any tools I can use to measure the performance of Phonegap UI (time from clicking to finishing the action) on device? And mainly to measure the improvement of these times after I do some performance tuning?

Yes, you can use Traceview - it's an excellent tool:
http://developer.android.com/guide/developing/debugging/debugging-tracing.html

I'm currently using this
and i strongly recommend it
https://github.com/GameMaker/PhoneGap-Performance-Test--PGPT-

JavaScript is GC based language. It means, it has unpredictable GC time which makes main thread stops. This makes UI struggles. On native implementation, you have control to use GC or not.
This wouldn't be a problem on Android. Android always had those struggling because of GC on Java. Considering this fact, most of Android users doesn't care about UX a lot. But on iOS, your HTML5 based app never be better experience than competitor's native app.
There are many workarounds for this GC time issue. Such as incremental-GC, realtime-GC and so on. But actually, there's no real solution. Because the primitive problem is you don't have control.
Regards
Rajeev