I'm developing an Android app that will hold a tensorflow-lite model for offline inference.
I know that it is impossible to completely avoid someone stealing my model, but I would like to make a hard time for someone trying it.
I thought to keep my .tflite model inside the .apk but without the weights of the top layer. Then, at execution time I could download the weights of the last layer and load it in memory.
So, if someone try to steal my model he would get a useless model because it couldn't be used due to the missing weights of the last layer.
It is possible to generate a tflite model without the weights of the last layer?
Is it possible load those weights in a already loaded model in memory?
This is how I loading my .tflite model:
tflite = new Interpreter(loadModelFile(), tfliteOptions);
// loads tflite grapg from file
private MappedByteBuffer loadModelFile() throws IOException {
AssetFileDescriptor fileDescriptor = mAssetManager.openFd(chosen);
FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());
FileChannel fileChannel = inputStream.getChannel();
long startOffset = fileDescriptor.getStartOffset();
long declaredLength = fileDescriptor.getDeclaredLength();
return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
}
Are there other approaches to make my model safer? I really need to make inference locally.
If we are talking about Keras models ( or any other model in TF ), we can easily remove the last layer and then convert it to a TF Lite model with tf.lite.TFLiteConverter. That should not be a problem.
Now, in Python, get the last layer's weights and convert it to a nice JSON file. This JSON file could be hosted on cloud ( like Firebase Cloud Storage ) and can be downloaded by the app.
The weights could be parsed as an array() object. The actiavtions from the TF Lite model could be dot multiplied with the weights parsed from the JSON. Lastly, we apply an activation to provide predictions, which we need indeed!
The model is so precisely trained that it could be rarely used for any other use case. So, I think we do not need to worry about that.
Also, it will be better if we use some cloud hosting platforms, which use requests and APIs instead of directly loading a raw model.
Related
I've made an Image classification model and converted it to tflite format.
Then I've verified tflite model in Python using tf.lite.Interpreter — it produces same results for my test image as the original model. Here's a colab link to verify.
Then I embedded it to a sample Android app, using Android Studio ML Model Binding and exact example code from Android studio.
Here's the main activity code, you can also use this link to navigate to the full android project.
val assetManager = this.assets
val istr = assetManager.open("test_image.JPG") //The same image
val b = BitmapFactory.decodeStream(istr)
val model = Model2.newInstance(this) //Model definition generated by Android Studio
// Creates inputs for reference.
val image = TensorImage.fromBitmap(b)
// Runs model inference and gets result.
val outputs = model.process(image)
val probability = outputs.probabilityAsCategoryList
probability.sortByDescending { it.score }
val top9 = probability.take(9)
this.findViewById<TextView>(R.id.results_text).text = top9.toString()
And then I'm getting completely different results on Android for the same model and the same input image.
Here are results matching my initial model in Python:
Here are wrong results I'm getting in Android app:
Links to the model and the test image are there in both examples, but I'll post them into the question once again:
tflite model
test image
I guess it has something to do with input/output formats of the model. Or the image is interpreted differently in python and in android. Or the metadata I added to the model is somehow wrong. Anyways, I've tried everything to localize the issue and now I'm stuck.
How do I fix my model or Android code so it produces the same results as my python code?
I've managed to find and fix the issue:
My model from this tutorial included a built-in image normalization layer. Image normalization is when you transform standard 0-255 image color values to 0.0-1.0 float values, suitable for machine learning.
But the metadata I used for the tflite model included 2 parameters for external normalization: mean and std.
Formula for each value being: normalized_value = (value - mean) / std
Since my model handles its own normalization, I need to turn off external normalization by setting mean = 0 and std = 1.
This way I'll get normalized_value = value.
So, setting the tflite metadata parameters to these:
image_min=0,
image_max=255.0,
mean=[0.0],
std=[1.0]
fixed the double normalization issue and my model now produces correct results in Android app.
I have been trying to graph data in Android using matplotlib through Chaquopy. So far, creating the plots themselves in Chaquopy has been very easy. However, I am unclear on exactly how to save the plot and load it into the ImageView. I looked at this example (How to display python matplotlib graphs (png) with Chaquopy in Android Studio), and that seems like a promising method, but it involves plotting in Python, storing the image as a bytes object, and then loading the bytes object into the ImageView in Java. Ideally, I would like to be able to do all of this, including updating the ImageView, from the python script. Is this possible?
Thanks for any help and advice!
Edit:
I found a solution that avoids the bytes object entirely:
root = Environment.getExternalStorageDirectory()
plt.savefig(root.getAbsolutePath() + "/fig1.png")
bitmap = BitmapFactory.decodeFile(root.getAbsolutePath() + "/fig1.png")
self.findViewById(R.id.imageView).setImageBitmap(bitmap)
If anyone has a solution using the bytes object though I would welcome any suggestions.
We have an Android app that uses Protocol Buffers to store application data. The data format (roughly) is a single protobuf ("container") that contains a list of protobufs ("items") as a repeated field:
message Container {
repeated Item item = 1;
}
When we want to save a change to an item, we must recreate the protobuf container, add all the items to it, then serialize it and write it to a file.
The problem with this a approach is it potentially triples the memory used when saving because the data has to first be copied from the model class to the protobuf builder and then to a byte array when the protobuf is serialized, all before writing it out to a file stream.
What we would like is a way to create our protobuf container and lazily serialize it to a stream, then simply add each protobuf item (created from our model data) to the container which serializes and writes it to the stream, rather than keeping all the items in memory until we've created the entire container in memory.
Is there a way to build a protobuf and serialize it lazily to a stream?
If there's not a way to do this officially, are there any libraries that can help? Does anyone have any suggestions or ideas how to solve this in other ways? Alternative data formats or technologies (e.g. JSON or XML containing protobufs) that would make this possible?
For serialization:
protobuf is an appendable format, with individual items being merged, and repeated items being appended
Therefore, to write a sequence as a lazy stream, all you need to do is repeatedly write the same structure with only one item in the list: serializing a sequence of 200 x "Container with 1 Item" is 100% identical to serializing 1 x "Container with 200 Items".
So: just do that!
For deserialization:
That is technically very easy to read as a stream - it all, however, comes down to which library you are using. For example, I expose this in protobuf-net (a .NET / C# implementation) as Serializer.DeserializeItems<T>, which reads (fully lazy/streaming) a sequence of messages of type T, based on the assumption that they are in the form you describe in the question (so Serializer.DeserializeItems<Item> would be the streaming way that replaces Serializer.Deserialize<Container> - the outermost object kinda doesn't really exist in protobuf)
If this isn't available, but you have access to a raw reader API, what you need to do is:
read one varint for the header - this will be the value 10 (0x0A), i.e. "(1 << 3) | 2" for the field-number (1) and wire-type (2) respectively - so this could also be phrased: "read a single byte from the stream , and check the value is 10"
read one varint for the length of the following item
now:
if the reader API allows you to restrict the maximum number of bytes to process, use this length to specify the length that follows
or wrap the stream API with a length-limiting stream, limited to that length
or just manually read that many bytes, and construct an in-memory stream from the payload
rinse, repeat
There is no such thing. A protobuf is a packed structure. In order to do this effectively it would need all the data. You will have to add the "streaming protocol" yourself. Maybe send a protobuf msg every N items.
In the normal java version of Protocol buffers there is Delimited files where you write Protocol-Buffers one at a time. I am not sure if it is in the Android version
aLocation.writeDelimitedTo(out);
As Marc has indicated it easily implemented; just write a length followed
the serialised bytes. In normal (non android) java version of prortocol-buffers you can also do (you have to serialise to a byte array or something similar)
private CodedOutputStream codedStream = null;
public void write(byte[] bytes) throws IOException {
if (bytes != ConstClass.EMPTY_BYTE_ARRAY) {
codedStream.writeRawVarint32(bytes.length);
codedStream.writeRawBytes(bytes);
codedStream.flush();
}
}
and
private CodedInputStream coded;
public byte[] read() throws IOException {
if (coded == null) {
throw new IOException("Reader has not been opened !!!");
}
if (coded.isAtEnd()) {
return null;
}
return coded.readBytes().toByteArray();
Something may be possible in other Protocol-Buffers versions
I am trying to use gson to do my object mapping on the android emulator.
It has been ridiculously slow when processing json data around 208 kb. I do not have any hierarchies in my json.
After the object mapping is done, i can see it that gson created around 500 records.
It is taking it over 3 minutes on the android emulator to map the input json.
I have annotated my entity which comprises of strings and couple of floats.
An I missing something?
Any ideas, best practices would greatly help.
Are there any ways of quickly object mapping the json data?
URL myURL = new URL(url);
/* Open a connection to that URL. */
URLConnection ucon = myURL.openConnection();
/*
* Define InputStreams to read from the URLConnection.
*/
InputStream is = ucon.getInputStream();
InputStreamReader reader = new InputStreamReader(is);
long tickCount = System.currentTimeMillis();
Policy[] policies = new Gson().fromJson(reader, Policy[].class);
long endCount = System.currentTimeMillis() - tickCount;
Log.d("Time to pull policies in milliseconds", "" + endCount);
I've seen questions like this come up before, and the general consensus is that Jackson is much faster than Gson. See the following links for more information:
Jackson Vs. Gson
Replace standard Android JSON parser for better performance?
http://www.cowtowncoder.com/blog/archives/2009/12/entry_345.html
https://stackoverflow.com/questions/338586/a-better-java-json-library
Here is one which specifically discusses Android: http://ubikapps.net/?p=525
Have you tried the mixing the GSON streaming parser with the Gson object? http://sites.google.com/site/gson/streaming (look for the Mixed read example).
This approach may help since Gson reads in an entire parse tree and then acts on it. With a large array list, reading in all elements and attempting to parse may cause lot of memory swaps (or thrashing). This approach will read in one element at a time.
Hope this helps.
You'd probably get better performance if you wrapped that InputStream in a BufferedInputStream with a nice big buffer...
3 minutes is insane. I seldom run the emulator but I have an app with a ~1.1MB JSON asset and that takes around 5 seconds to load and process on hardware.
(Which is still far too long, but still).
I've found that I can speed up gson.fromJSON quite considerably by not modelling all the elements in the JSON that I won't need. GSON will happily fill in only what is specified in your response classes.
I have found that CREATING a Gson instance is a very expensive operation, both in terms of CPU used and memory allocated.
Since Gson instances are thread-safe, constructing and reusing a single static instance pays off, especially if you are serializing / deserializing often.
Is it possible to create a simple 3D model (for example in 3DS MAX) and then import it to Android?
That's where I got to:
I've used Google's APIDemos as a starting point - there are rotating cubes in there, each specified by two arrays: vertices and indices.
I've build my model using Blender and exported it as OFF file - it's a text file that lists all the vertices and then faces in terms of these vertices (indexed geometry)
Then I've created a simple C++ app that takes that OFF and writes it as two XMLs containing arrays (one for vertices and one for indices)
These XML files are then copied to res/values and this way I can assign the data they contain to arrays like this:
int vertices[] = context.getResources().getIntArray(R.array.vertices);
I also need to manually change the number of faces to be drawn in here: gl.glDrawElements(GL10.GL_TRIANGLES, 212*6, GL10.GL_UNSIGNED_SHORT, mIndexBuffer); - you can find that number (212 in this case) on top of the OFF file
Here you can find my project page, which uses this solution: Github project > vsiogap3d
you may export it to ASE format.
from ASE, you can convert it to your code manually or programatically.
You will need vertex for vertices array and faces for indices in Android.
don't forget you have to set
gl.glFrontFace(GL10.GL_CCW);
because 3ds max default is counter clockwise.
It should be possible. You can have the file as a data file with your program (and as such it will be pushed onto the emulator and packaged for installation onto an actual device). Then you can write a model loader and viewer in java using the Android and GLES libraries to display the model.
Specific resources on this are probably limited though. 3ds is a proprietry format so 3rd party loaders are in shortish supply and mostly reverse engineered. Other formats (such as blender or milkshape) are more open and you should be able to find details on writing a loader for them in java fairly easily.
Have you tried min3d for android? It supports 3ds max,obj and md2 models.
Not sure about Android specifically, but generally speaking you need a script in 3DS Max that manually writes out the formatting you need from the model.
As to whether one exists for Android or not, I do not know.
You can also convert 3DS MAX model with the 3D Object Converter
http://web.t-online.hu/karpo/
This tool can convert 3ds object to text\xml format or c code.
Please note that the tool is not free. You can try for a 30-day trial period. 'C' code and XML converters are available.
'c' OpenGL output example:
glDisable(GL_TEXTURE_2D);
glEnable(GL_LIGHTING);
glEnable(GL_NORMALIZE);
GLfloat Material_1[] = { 0.498039f, 0.498039f, 0.498039f, 1.000000f };
glBegin(GL_TRIANGLES);
glMaterialfv(GL_FRONT,GL_DIFFUSE,Material_1
glNormal3d(0.452267,0.000000,0.891883);
glVertex3d(5.108326,1.737655,2.650969);
glVertex3d(9.124107,-0.002484,0.614596);
glVertex3d(9.124107,4.039649,0.614596);
glEnd();
Or direct 'c' output:
Point3 Object1_vertex[] = {
{5.108326,1.737655,2.650969},
{9.124107,-0.002484,0.614596},
{9.124107,4.039649,0.614596}};
long Object1_face[] = {
3,0,1,2,
3,3,4,5
3,6,3,5};
You can migrate than those collections of objects to your Java code.