I have a basic question about how to determine the image pre-processing parameters like - "IMAGE_MEAN", "IMAGE_STD" for various tensorflow pre-trained models. The Android sample applications for TensorFlow provides these parameters for a certain inception_v3 model in the ClassifierActivity.java (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/ClassifierActivity.java) as shown below -
"If you want to use a model that's been produced from the TensorFlow for Poets codelab, you'll need to set IMAGE_SIZE = 299, IMAGE_MEAN = 128, IMAGE_STD = 128"
How do I determine these parameters for other TF models
Also, while converting the TF model to CoreML model, to be used on iOS, there are additional image pre-processing parameters that need to be specified (like - red_bias, green_bias, blue_bias and image_scale) as shown in the code segment below. The below parameters are for inception_v1_2016.pb model. If I want to use another pre-trained model like - ResNet50, MobileNet, etc how do I determine these parameters
tf_converter.convert(tf_model_path = 'inception_v1_2016_08_28_frozen.pb',
mlmodel_path = 'InceptionV1.mlmodel',
output_feature_names = ['InceptionV1/Logits/Predictions/Softmax:0'],
image_input_names = 'input:0',
class_labels = 'imagenet_slim_labels.txt',
red_bias = -1,
green_bias = -1,
blue_bias = -1,
image_scale = 2.0/255.0
)
Any help will be greatly appreciated
Unfortunately, the preprocessing requirements of various ImageNet models are still under documented. ResNet and VGG models both use the same preprocessing parameters. You can find biases for each of the color channels here:
https://github.com/fchollet/deep-learning-models/blob/master/imagenet_utils.py#L11
The preprocessing for Inception_V3, MobileNet, and other models can be found in the individual model files of this repo: https://github.com/fchollet/deep-learning-models
When converting to Core ML you always need to specify preprocessing biases on a per channel basis. So in the case of a VGG-type preprocessing, you can just copy each channel's biases directly from the code linked to above. It's super important to note that the biases are applied (added) BEFORE scaling. You can read more about setting the proper values here: http://machinethink.net/blog/help-core-ml-gives-wrong-output/
The conversion code you posted looks good for MobileNet or Inception_V3 models, but would not work for VGG or ResNet. For those you'd need:
tf_converter.convert(...
red_bias=-123.68,
green_bias=-116.78,
blue_bias=-103.94
)
No scaling is required.
Related
I've adapted Tensorflow Lite's Salad Detector Colab and am able to train my own models and get them working on Android but I'm trying to count Objects and I need more than the 25 limit that is the default.
The models have a method for increasing detections so, in the above Colab, I inserted the following code:
spec = model_spec.get('efficientdet_lite4')
spec.tflite_max_detections=50
And on the Android side of things
val options = ObjectDetector.ObjectDetectorOptions.builder()
.setMaxResults(50)
.setScoreThreshold(10)
.build()
The models are training fine but I'm still only able to detect 25 Objects in a single image.
Is there a problem with my models? Or are there any other settings I can change in my Android code that will increase the number of detections?
Solved this myself after Googling a different SOF question on efficientdet_lite4, I stumbled on an AHA moment.
My problem was here:
spec = model_spec.get('efficientdet_lite4')
spec.tflite_max_detections=50
I needed to change the whole spec of the model:
spec = object_detector.EfficientDetLite4Spec(
model_name='efficientdet-lite4',
uri='https://tfhub.dev/tensorflow/efficientdet/lite4/feature-vector/2',
hparams='',
model_dir=None,
epochs=50,
batch_size=64,
steps_per_execution=1,
moving_average_decay=0,
var_freeze_expr='(efficientnet|fpn_cells|resample_p6)',
**tflite_max_detections=50**,
strategy=None,
tpu=None,
gcp_project=None,
tpu_zone=None,
use_xla=False,
profile=False,
debug=False,
tf_random_seed=111111,
verbose=0
)
From there I was able to train the model and things worked on the Android side of things.
This has been bugging me for a few weeks!
I am developing an Android application which requires an ML model integration.For it I am using TensorFlow lite for deployment.I am using Custom Model based Siamese Network for output and the output shape is [1 128].When I infer the tf lite model in python on Google Colab the output [1 128] numbers are different from the one being produced on my Android device.THe input image is same on both inferences and also the input and output shapes but still I am getting different output vectors on my Android Phone and Python TFlite model.I am using Firebase Machine Learning.
Android Code
val interpreter=Interpreter(model)
val imageBitmap= Bitmap.createScaledBitmap(BitmapFactory.decodeFileDescriptor(contentResolver.openFileDescriptor(fileUri,"r")?.fileDescriptor),256,256,true)
val inputImage=ByteBuffer.allocateDirect(256*256*3*4).order(ByteOrder.nativeOrder())
for(ycord in 0 until 256){
for(xcord in 0 until 256){
val pixel=imageBitmap.getPixel(xcord,ycord)
inputImage.putFloat(Color.red(pixel)/1.0f)
inputImage.putFloat(Color.green(pixel)/1.0f)
inputImage.putFloat(Color.blue(pixel)/1.0f)
}
}
imageBitmap.recycle()
val modelOutput=ByteBuffer.allocateDirect(outputSize).order(ByteOrder.nativeOrder())
interpreter.run(inputImage,modelOutput)
modelOutput.rewind()
val probs=modelOutput.asFloatBuffer()
success(ImageProcessResult.Success(probs))
Kindly help me.I need it soon.Any help is appreciated
You are resizing the bitmap to [256,256] in the Android platform.
Even the slightest change in input vectors would change the output vector. When you resize the bitmaps, you change the input vector. However, if the model is general enough the final result which would be argmax of the output vector (in classification) would be the same.
In the case of Siamese, I believe it won't affect the final result (similarity score) in a meaningful way if the model is not overfitted.
I created a Tensorflow model which takes a single 700x700 48-dimension "image" as an input (input shape is {1, 700, 700, 48}).
To do so, I used Numpy's numpy.concatenate([array_of_images], -1), when array_of_images is an array of 16 700x700 JPEG images.
I converted the model to Tensorflow Lite and I'm running it on Android.
No conversion errors or anything - all ops are valid and supported.
My question is - where in Android (or how) can I create an N-dimensional object (or container) and use it as an input to the model?
I think you have 16 RGB images,
on android you load your bitmaps into image tensors like this :
var bitmap1 = Bitmap.load( from anywhere )
var tImage1 = TensorImage(DataType.FLOAT32)
tImage1.load(bitmap1)
for each image,
then
input = arrayof(tImage1.buffer, tImage2.buffer,........tImage16.buffer)
interpreteur.runForMultipleInputsOutputs(arrayOf(input), output)
i'm not sure but this can give you an idea
I want to test a learnt model under Python on android, I'm on Tensorflow 0.9.
To do so, I freezed my graph to have a single pb file with the graph and weight. I used Queues to manage my learning batches.
When running my session on Android, I specify the input tensor by its name "input_node", which is the data layer as input in my network.
X = tf.reshape(X, [-1, W, H, 1], name="input_node")
and call the "output_node" layer :
output = tf.reshape(h_fc11, shape=[-1, 8], name="output_node")
Here is the call in tensorflow_jni.cc:
std::vector<std::pair<std::string, tensorflow::Tensor> > input_tensors({{"input_node", input_tensor}});
s = session->Run(input_tensors, output_names, {}, &output_tensors);
The batch generation is done before, so it should not been used when testing.
But I have the following error:
tensorflow_jni.cc:312 Error during inference: Invalid argument: No OpKernel was registered to support Op 'RandomShuffleQueue' with these attrs
[[Node: shuffle_batch/random_shuffle_queue = RandomShuffleQueuecapacity=10750, component_types=[DT_FLOAT, DT_FLOAT], container="", min_after_dequeue=10000, seed=0, seed2=0, shapes=[[10000], [8]], shared_name=""]]
It seems that the batch generation layer is called (my images are 100x100 and I have 8 outputs), but I don't know why.
When testing the same model with the same input/output layers though the image_labelling.cc directly on Mac (building with Bazel), I don't have the error.
I do not understand why the RandomShuffleQueue is needed when testing. Am I missing something to specify the part of the graph I want to use? Are all the layers of the graph verified even if not used?
Thanks.
I'm still working on the documentation for this, but I think the optimize_for_inference script should help you here:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/optimize_for_inference.py
You pass in a frozen graph, the input nodes, and output nodes, and it removes all of the other ops that aren't needed.
My Android application uses javaCV and calls detectMultiScale() function with LBP cascade to detect faces. It works completely fine on my emulator. However, when I tried to test it on my HTC Incredible S, it returns 0, could not detect any face! Could anyone show me some hints why it does not work? Many thanks for your help!!!
Here is my code for face detection:
CASCADE_FILE = working_Dir.getAbsolutePath() + "/lbpcascade_frontalface.xml";
public static CvRect getFaceWithLBP(IplImage grayFaceImg)
{
CascadeClassifier cascade = new CascadeClassifier(CASCADE_FILE);
CvRect facesdetection = new CvRect(null);
cascade.detectMultiScale(grayFaceImg, facesdetection, 1.1, 2, CV_HAAR_FIND_BIGGEST_OBJECT | CV_HAAR_DO_ROUGH_SEARCH,
new CvSize(), new CvSize(grayFaceImg.width(), grayFaceImg.height()));
return facesdetection;
}
Just a note, as per the OpenCV documentation, the flags (such as CV_HAAR_FIND_BIGGEST_OBJECT | CV_HAAR_DO_ROUGH_SEARCH) can not be used with new cascades (like LBP ones).
void CascadeClassifier::detectMultiScale(const Mat& image, vector& objects, double scaleFactor=1.1, int minNeighbors=3, int flags=0, Size minSize=Size(), Size maxSize=Size())
Parameters:
cascade – Haar classifier cascade (OpenCV 1.x API only). It can be loaded from XML or YAML file using Load(). When the cascade is not needed anymore, release it using cvReleaseHaarClassifierCascade(&cascade).
image – Matrix of the type CV_8U containing an image where objects are detected.
objects – Vector of rectangles where each rectangle contains the detected object.
scaleFactor – Parameter specifying how much the image size is reduced at each image scale.
minNeighbors – Parameter specifying how many neighbors each candidate rectangle should have to retain it.
flags – Parameter with the same meaning for an old cascade as in the function cvHaarDetectObjects. It is not used for a new cascade.
minSize – Minimum possible object size. Objects smaller than that are ignored.
maxSize – Maximum possible object size. Objects larger than that are ignored.