I have created a model in Keras using Transfer Learning.
I used InceptionV3 model as base and made initial 52 layers as non-trainable.
Then I add my custom layers on top of it.
Trained it and saved to hdf5 file.
The model predicted A.png on my laptop. Correct
I converted it to pb file using a Github tool.
Then I predicted A.png by pb model on laptop. It was correct.
Then I moved pb file to android asset folder.
I add Tensorflow Mobile dependency on it (not TensorflowLite).
I also added the same A.png to asset.
I loaded the model and pass A.png to pb model.
Output was wrong class.
I tried with other images. Always it points to the same wrong class.
Output never changed.
So I feel the hdf5 , pb model are correct but there is some mistake in my code which is passing the A.png to pb model. Please help me!
resized_image -> Bitmap of A.png
inferenceInterface -> tensorflow model interface
INPUT_NODE -> name of input node
OUTPUT_NODE -> name of output node
1,128,128,3 -> image is 128x128 and 3 channels
imageValuesFloat = normalizeBitmap(resized_image,128,127.5f,1.0f);
inferenceInterface.feed(INPUT_NODE,imageValuesFloat,1,128,128,3);
inferenceInterface.run(OUTPUT_NODES);
//declare array to hold results obtained from model
float[] result = new float[OUTPUT_SIZE];
//copy the output into the result array
inferenceInterface.fetch(OUTPUT_NODE,result);
Here is the normalizeBitmap function
public float[] normalizeBitmap(Bitmap source,int size,float mean,float std){
float[] output = new float[size * size * 3];
int[] intValues = new int[source.getHeight() * source.getWidth()];
source.getPixels(intValues, 0, source.getWidth(), 0, 0, source.getWidth(), source.getHeight());
for (int i = 0; i < intValues.length; ++i) {
final int val = intValues[i];
output[i * 3] = (((val >> 16) & 0xFF) - mean)/std;
output[i * 3 + 1] = (((val >> 8) & 0xFF) - mean)/std;
output[i * 3 + 2] = ((val & 0xFF) - mean)/std;
}
return output;
}
Related
I have created an image classifier in Keras, later I saved the model in pb format to use it in android.
However, in the python code, it can classify the image properly. But in android whatever image I gave as input the output is always the same .
This is how I have trained my model
rom keras.models import Sequential
from keras.layers import Conv2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
# Initialising the CNN
classifier = Sequential()
# Step 1 - Convolution
classifier.add(Conv2D(32, (3, 3), input_shape = (64, 64, 3), activation = 'relu'))
# Step 2 - Pooling
classifier.add(MaxPooling2D(pool_size = (2, 2)))
# Adding a second convolutional layer
classifier.add(Conv2D(32, (3, 3), activation = 'relu'))
classifier.add(MaxPooling2D(pool_size = (2, 2)))
# Step 3 - Flattening
classifier.add(Flatten())
# Step 4 - Full connection
classifier.add(Dense(units = 128, activation = 'relu'))
classifier.add(Dense(units = 1, activation = 'sigmoid'))
# Compiling the CNN
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
# Part 2 - Fitting the CNN to the images
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
test_datagen = ImageDataGenerator(rescale = 1./255)
training_set = train_datagen.flow_from_directory('dataset/training_set',
target_size = (64, 64),
batch_size = 32,
class_mode = 'binary')
test_set = test_datagen.flow_from_directory('dataset/test_set',
target_size = (64, 64),
batch_size = 32,
class_mode = 'binary')
classifier.fit_generator(training_set,
steps_per_epoch = 8000,
epochs = 25,
validation_data = test_set,
validation_steps = 2000)
classifier.summary()
classifier.save('saved_model.h5')
Later I convert that keras model(saved_model.h5) to tensorflow model by using this
This is how I have converted my bitmap float array
public static float[] getPixels(Bitmap bitmap) {
final int IMAGE_SIZE = 64;
int[] intValues = new int[IMAGE_SIZE * IMAGE_SIZE];
float[] floatValues = new float[IMAGE_SIZE * IMAGE_SIZE * 3];
if (bitmap.getWidth() != IMAGE_SIZE || bitmap.getHeight() != IMAGE_SIZE) {
// rescale the bitmap if needed
bitmap = ThumbnailUtils.extractThumbnail(bitmap, IMAGE_SIZE, IMAGE_SIZE);
}
bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());
for (int i = 0; i < intValues.length; ++i) {
final int val = intValues[i];
// bitwise shifting - without our image is shaped [1, 64, 64, 1] but we need [1, 168, 168, 3]
floatValues[i * 3 + 2] = Color.red(val) / 255.0f;
floatValues[i * 3 + 1] = Color.green(val) / 255.0f;
floatValues[i * 3] = Color.blue(val) / 255.0f;
}
return floatValues;
}
Later, I tried to classify image using tensorflow in android , like following .
TensorFlowInferenceInterface tensorFlowInferenceInterface;
tensorFlowInferenceInterface = new TensorFlowInferenceInterface(getAssets(),"model.pb");
float[] output = new float[2];
tensorFlowInferenceInterface.feed("conv2d_11_input",
getPixels(bitmap), 1,64,64,3);
tensorFlowInferenceInterface.run(new String[]{"dense_12/Sigmoid"});
tensorFlowInferenceInterface.fetch("dense_12/Sigmoid",output);
Whatever image I gave the value of the output is [1,0]
Is there anything I have missed?
The color components returned by Color.red(int), Color.blue(int) and Color.green(int) are integers in the range [0, 255] (see doc). The same thing holds when reading images using ImageDataGenerator of Keras. However, as I stated in comments section, in prediction phase you need to do the same preprocessing steps as done in training phase. You are scaling the image pixels by 1./255 in training (using rescale = 1./255 in ImageDataGenerator) and therefore, according to the first point I mentioned, this must also be done in prediction:
floatValues[i * 3 + 2] = Color.red(val) / 255.0;
floatValues[i * 3 + 1] = Color.green(val) / 255.0;
floatValues[i * 3] = Color.blue(val) / 255.0;
I'm trying to build a classification model with keras and deploy the model to my Android phone. I use the code from this website to deploy my own converted model, which is a .pb file, to my Android phone. I load a image from my phone and everything worked fine, but the prediction result is totally different from the result I got from my PC.
The procedure of testing on my PC are:
load the image with cv2, and convert to np.float32
use the keras resnet50 'preprocess_input' python function to preprocess the image
expand the image dimension for batching (batch size is 1)
forward the image to model and get the result
Relevant code:
img = cv2.imread('./my_test_image.jpg')
x = preprocess_input(img.astype(np.float32))
x = np.expand_dims(x, axis=0)
net = load_model('./my_model.h5')
prediction_result = net.predict(x)
And I noticed that the image preprocessing part of Android is different from the method I used in keras, which mode is caffe(convert the images from RGB to BGR, then zero-center each color channel with respect to the ImageNet dataset). It seems that the original code is for mode tf(will scale pixels between -1 to 1).
So I modified the following code of 'preprocessBitmap' to what I think it should be, and use a 3 channel RGB image with pixel value [127,127,127] to test it. The code predicted the same result as .h5 model did. But when I load a image to classify, the prediction result is different from .h5 model.
Does anyone has any idea? Thank you very much.
I have tried the following:
Load a 3 channel RGB image in my Phone with pixel value [127,127,127], and use the modified code below, and it will give me a prediction result that is same as prediction result using .h5 model on PC.
Test the converted .pb model on PC using tensorflow gfile module with a image, and it give me a correct prediction result (compare to .h5 model). So I think the converted .pb file does not have any problem.
Entire section of preprocessBitmap
// code of 'preprocessBitmap' section in TensorflowImageClassifier.java
TraceCompat.beginSection("preprocessBitmap");
// Preprocess the image data from 0-255 int to normalized float based
// on the provided parameters.
bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());
for (int i = 0; i < intValues.length; ++i) {
// this is a ARGB format, so we need to mask the least significant 8 bits to get blue, and next 8 bits to get green and next 8 bits to get red. Since we have an opaque image, alpha can be ignored.
final int val = intValues[i];
// original
/*
floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - imageMean) / imageStd;
floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - imageMean) / imageStd;
floatValues[i * 3 + 2] = ((val & 0xFF) - imageMean) / imageStd;
*/
// what I think it should be to do the same thing in mode caffe when using keras
floatValues[i * 3 + 0] = (((val >> 16) & 0xFF) - (float)123.68);
floatValues[i * 3 + 1] = (((val >> 8) & 0xFF) - (float)116.779);
floatValues[i * 3 + 2] = (((val & 0xFF)) - (float)103.939);
}
TraceCompat.endSection();
This question is old, but remains the top Google result for preprocess_input for ResNet50 on Android. I could not find an answer for implementing preprocess_input for Java/Android, so I came up with the following based on the original python/keras code:
/*
Preprocesses RGB bitmap IAW keras/imagenet
Port of https://github.com/tensorflow/tensorflow/blob/v2.3.1/tensorflow/python/keras/applications/imagenet_utils.py#L169
with data_format='channels_last', mode='caffe'
Convert the images from RGB to BGR, then will zero-center each color channel with respect to the ImageNet dataset, without scaling.
Returns 3D float array
*/
static float[][][] imagenet_preprocess_input_caffe( Bitmap bitmap ) {
// https://github.com/tensorflow/tensorflow/blob/v2.3.1/tensorflow/python/keras/applications/imagenet_utils.py#L210
final float[] imagenet_means_caffe = new float[]{103.939f, 116.779f, 123.68f};
float[][][] result = new float[bitmap.getHeight()][bitmap.getWidth()][3]; // assuming rgb
for (int y = 0; y < bitmap.getHeight(); y++) {
for (int x = 0; x < bitmap.getWidth(); x++) {
final int px = bitmap.getPixel(x, y);
// rgb-->bgr, then subtract means. no scaling
result[y][x][0] = (Color.blue(px) - imagenet_means_caffe[0] );
result[y][x][1] = (Color.green(px) - imagenet_means_caffe[1] );
result[y][x][2] = (Color.red(px) - imagenet_means_caffe[2] );
}
}
return result;
}
Usage with a 3D tensorflow-lite input with shape (1,224,224,3):
Bitmap bitmap = <your bitmap of size 224x224x3>;
float[][][][] imgValues = new float[1][bitmap.getHeight()][bitmap.getWidth()][3];
imgValues[0]=imagenet_preprocess_input_caffe(bitmap);
... <prep tfInput, tfOutput> ...
tfLite.run(tfInput, tfOutput);
I have an Android Tablet where I installed the TensorFlow-Lite DetectorActivity in the examples that were available. It works well on an Android Tablet. However, when I tried to deploy it on a RaspberryPi 3 Model B that ran Android Things, it didn't run. There seemed to be an issue with configuring the camera properly in terms of enabling a live camera preview and running an analysis.
My original goal is to make an object detection app run on Android Things. It is also essential to draw the bounding rectangle on the detected objects.
I was looking for an example of an Android App that used TensorFlow-Lite and ran on Android Things. I quickly found this example from hackster.io that uses Image Classification to dispense candy. I ran it on my RaspberryPi Board, and it ran. It gives the results, the name of the object, the confidence level, as well as the ID. I was okay to build upon this sample code. Instead of a live camera feed, I could just make the app take a photo, analyze, and give the result. After which, it takes another photo and the cycle continues.
However, it did not specify the location in terms of a RectF Object.
What I tried to do was to adapt the recognizeFunction in the TFLite Android Example, it's in the TFLiteObjectDetectionAPIModel class. I adapted it to the doIdentification function of the Candy Dispenser Android App. My function now looks like this:
// outputLocations: array of shape [Batchsize, NUM_DETECTIONS,4]
// contains the location of detected boxes
private float[][][] outputLocations;
// outputClasses: array of shape [Batchsize, NUM_DETECTIONS]
// contains the classes of detected boxes
private float[][] outputClasses;
// outputScores: array of shape [Batchsize, NUM_DETECTIONS]
// contains the scores of detected boxes
private float[][] outputScores;
// numDetections: array of shape [Batchsize]
// contains the number of detected boxes
private float[] numDetections;
private static final int NUM_DETECTIONS = 10;
private static final float IMAGE_MEAN = 128.0f;
private static final float IMAGE_STD = 128.0f;
private void doIdentification(Bitmap image) {
Log.e(TAG, "doing identification!");
Trace.beginSection("recognizeImage");
int numBytesPerChannel;
if (TF_OD_API_IS_QUANTIZED) {
Log.e(TAG, "model is quantized");
numBytesPerChannel = 1; // Quantized
} else {
Log.e(TAG, "model is NOT quantized");
numBytesPerChannel = 4; // Floating point
}
ByteBuffer imgData = ByteBuffer.allocateDirect(1 * TF_INPUT_IMAGE_HEIGHT * TF_INPUT_IMAGE_HEIGHT
* 3 * numBytesPerChannel);
Trace.beginSection("preprocessBitmap");
// Preprocess the image data from 0-255 int to normalized float based
// on the provided parameters.
int[] intValues = new int[TF_INPUT_IMAGE_HEIGHT * TF_INPUT_IMAGE_HEIGHT];
image.getPixels(intValues, 0, image.getWidth(), 0, 0, image.getWidth(), image.getHeight());
imgData.rewind();
for (int i = 0; i < TF_INPUT_IMAGE_HEIGHT; ++i) {
for (int j = 0; j < TF_INPUT_IMAGE_HEIGHT; ++j) {
int pixelValue = intValues[i * TF_INPUT_IMAGE_HEIGHT + j];
if (TF_OD_API_IS_QUANTIZED) {
imgData.put((byte) ((pixelValue >> 16) & 0xFF));
imgData.put((byte) ((pixelValue >> 8) & 0xFF));
imgData.put((byte) (pixelValue & 0xFF));
} else {
imgData.putFloat((((pixelValue >> 16) & 0xFF) - IMAGE_MEAN) / IMAGE_STD);
imgData.putFloat((((pixelValue >> 8) & 0xFF) - IMAGE_MEAN) / IMAGE_STD);
imgData.putFloat(((pixelValue & 0xFF) - IMAGE_MEAN) / IMAGE_STD);
}
}
}
Trace.endSection(); // preprocessBitmap
// Allocate space for the inference results
byte[][] confidencePerLabel = new byte[1][mLabels.size()];
//for box detections
// Copy the input data into TensorFlow.
Trace.beginSection("feed");
outputLocations = new float[1][NUM_DETECTIONS][4];
outputClasses = new float[1][NUM_DETECTIONS];
outputScores = new float[1][NUM_DETECTIONS];
numDetections = new float[1];
Object[] inputArray = {imgData};
Map<Integer, Object> outputMap = new HashMap<>();
outputMap.put(0, outputLocations);
outputMap.put(1, outputClasses);
outputMap.put(2, outputScores);
outputMap.put(3, numDetections);
Trace.endSection();
// Read image data into buffer formatted for the TensorFlow model
TensorFlowHelper.convertBitmapToByteBuffer(image, intValues, imgData);
// Run inference on the network with the image bytes in imgData as input,
// storing results on the confidencePerLabel array.
Trace.beginSection("run");
mTensorFlowLite.runForMultipleInputsOutputs(inputArray, outputMap);
Trace.endSection();
// TODO - we try and fetch our rectF's here
final ArrayList<Recognition> recognitions = new ArrayList<>(NUM_DETECTIONS);
for (int i = 0; i < NUM_DETECTIONS; ++i) {
final RectF detection =
new RectF(
outputLocations[0][i][1] * TF_OD_API_INPUT_SIZE,
outputLocations[0][i][0] * TF_OD_API_INPUT_SIZE,
outputLocations[0][i][3] * TF_OD_API_INPUT_SIZE,
outputLocations[0][i][2] * TF_OD_API_INPUT_SIZE);
// SSD Mobilenet V1 Model assumes class 0 is background class
// in label file and class labels start from 1 to number_of_classes+1,
// while outputClasses correspond to class index from 0 to number_of_classes
int labelOffset = 1;
Log.e(TAG, "adding the following to our results: ");
Log.e(TAG, "recognition id: " + i);
Log.e(TAG, "recognition label: " + mLabels.get((int) outputClasses[0][i] + labelOffset));
Log.e(TAG, "recognition confidence: " + outputScores[0][i]);
recognitions.add(
new Recognition(
"" + i,
mLabels.get((int) outputClasses[0][i] + labelOffset),
outputScores[0][i],
detection));
}
Trace.endSection(); // "recognizeImage"
// TODO -- This is the old working code
// Get the results with the highest confidence and map them to their labels
Collection<Recognition> results = TensorFlowHelper.getBestResults(confidencePerLabel, mLabels);
Log.e(TAG, "results count is = " + results.size());
// Report the results with the highest confidence
onClassificationComplete(results);
}
I set the Quantized constant to true, and ran the code. However, I was greeted by the following error:
java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite tensor with type UINT8 and a Java object of type [[[F (which is compatible with the TensorFlowLite type FLOAT32).
and the line responsible is:
mTensorFlowLite.runForMultipleInputsOutputs(inputArray, outputMap);
I tried changing to to just mTensorFlowLite.run, but that resulted in a different error.
Has anyone implemented Object Detection (drawing a RectF on the detected objects) on Android Things?
The issue here is likely you are mixing models. There is a difference between image classification and object detection models. Classification simply reports the confidence of a certain type of object within the image and detection adds on identifying the object's location. The candy dispenser sample you are starting from uses an image classification model (mobilenet_quant_v1_224.tflite) whereas the TFLite sample you mentioned runs an object detection model (mobilenet_ssd.tflite).
I would recommend starting from the sample that does object detection and work through the camera issues rather than solving the problem the other way around. The candy dispenser sample (as well as the official image classifier sample) provide a good reference for getting the camera on the RPi3 to capture an image and converting it for use with the model.
I am trying to run a Tensorflow model on my Android application, but the same trained model gives different results (wrong inference) compared to when it is run on Python on desktop.
The model is a simple sequential CNN to recognize characters, much like this number plate recognition network, minus the windowing, as my model has the characters already cropped into place.
I have:
Model saved in protobuf (.pb) file - modeled and trained in Keras on Python/Linux + GPU
The inference was tested on a different computer on pure Tensorflow, to make sure Keras was not the culprit. Here, the results were as expected.
Tensorflow 1.3.0 is being used on Python and Android. Installed from PIP on Python and jcenter on Android.
The results on Android do not resemble the expected outcome.
The input is a 129*45 RGB image, so a 129*45*3 array, and the output is a 4*36 array (representing 4 characters from 0-9 and a-z).
I used this code to save the Keras model as a .pb file.
Python code, this works as expected:
test_image = [ndimage.imread("test_image.png", mode="RGB").astype(float)/255]
imTensor = np.asarray(test_image)
def load_graph(model_file):
graph = tf.Graph()
graph_def = tf.GraphDef()
with open(model_file, "rb") as f:
graph_def.ParseFromString(f.read())
with graph.as_default():
tf.import_graph_def(graph_def)
return graph
graph=load_graph("model.pb")
with tf.Session(graph=graph) as sess:
input_operation = graph.get_operation_by_name("import/conv2d_1_input")
output_operation = graph.get_operation_by_name("import/output_node0")
results = sess.run(output_operation.outputs[0],
{input_operation.outputs[0]: imTensor})
Android code, based on this example; this gives seemingly random results:
Bitmap bitmap;
try {
InputStream stream = getAssets().open("test_image.png");
bitmap = BitmapFactory.decodeStream(stream);
} catch (IOException e) {
e.printStackTrace();
}
inferenceInterface = new TensorFlowInferenceInterface(context.getAssets(), "model.pb");
int[] intValues = new int[129*45];
float[] floatValues = new float[129*45*3];
String outputName = "output_node0";
String[] outputNodes = new String[]{outputName};
float[] outputs = new float[4*36];
bitmap.getPixels(intValues, 0, bitmap.getWidth(), 0, 0, bitmap.getWidth(), bitmap.getHeight());
for (int i = 0; i < intValues.length; ++i) {
final int val = intValues[i];
floatValues[i * 3 + 0] = ((val >> 16) & 0xFF) / 255;
floatValues[i * 3 + 1] = ((val >> 8) & 0xFF) / 255;
floatValues[i * 3 + 2] = (val & 0xFF) / 255;
}
inferenceInterface.feed("conv2d_1_input", floatValues, 1, 45, 129, 3);
inferenceInterface.run(outputNodes, false);
inferenceInterface.fetch(outputName, outputs);
Any help is greatly appreciated!
One Problem is in the lines:
floatValues[i * 3 + 0] = ((val >> 16) & 0xFF) / 255;
floatValues[i * 3 + 1] = ((val >> 8) & 0xFF) / 255;
floatValues[i * 3 + 2] = (val & 0xFF) / 255;
where the RGB values are divided by an integer, thus yielding an integer result (namely 0 every time).
Moreover, the division, even if executed with a 255.0 yielding a float between 0 and 1.0 may pose a problem, as the values aren't distributed in the projection space (0..1) like they were in Natura. To explain this: a value of 255 in the sensor domain (i.e. the R value for example) means that the natural value of the measured signal fell somewhere in the "255" bucket which is a whole range of energies/intensities/etc. Mapping this value to 1.0 will most likely cut half of its range, as subsequent calculations could saturate at a maximum multiplicator of 1.0 which really is only the midpoint of a +- 1/256 bucket. So maybe the transformation would be more correctly a mapping to the midpoints of a 256-bucket division of the 0..1 range:
((val & 0xff) / 256.0) + (0.5/256.0)
but this is just a guess from my side.
I have been looking at converting the NV21 byte[] that I get from onPreviewFrame(). I have searched the forums and google for various solutions. I have tried RenderScripts and some other code examples. Some of them give me an image with a yellow tint, some give me an image with red and blue flipped (after I flip it back in the code, I get yellow tint back), some give me strange color features all throughout the image (almost like a negative), some give me a grayscale image, some give me an image so dark you can't really make anything out.
Since I am the one typing the question, I realize I must be the idiot in the room so we will start with this post. This particular solution gives me a very dark image, but I am not cool enough to be able to comment yet. Has anyone tried this solution or has one that produces an image with the same quality as the original NV21 format?
I need either a valid ARGB byte[] or a valid Bitmap, I can modify my project to deal with either. Just for reference I have tried these (and a few others that are really just carbon copies of these):
One solution I tried
Another solution I tried
If you are trying to convert YUV from camera to Bitmap, here is something you can try:
// import android.renderscript.*
// RenderScript mRS;
// ScriptIntrinsicYuvToRGB mYuvToRGB;
// Allocation yuvPreviewAlloc;
// Allocation rgbOutputAlloc;
// Create RenderScript context, ScriptIntrinsicYuvToRGB and Allocations and keep reusing them.
if (NotInitialized) {
mRS = RenderScript.create(this).
mYuvToRGB = ScriptIntrinsicYuvToRGB.create(mRS, Element.YUV(mRS));
// Create a RS Allocation to hold NV21 data.
Type.Builder tYuv = new Type.Builder(mRS, Element.YUV(mRS));
tYuv.setX(width).setY(height).setYuvFormat(android.graphics.ImageFormat.NV21);
yuvPreviewAlloc = Allocation.createTyped(mRS, tYuv.create(), Allocation.USAGE_SCRIPT | Allocation.USAGE_IO_INPUT);
// Create a RS Allocation to hold RGBA data.
Type.Builder tRgb = new Type.Builder(mRS, Element.RGBA_8888(mRS));
tRgb.setX(width).tRgb(height);
rgbOutputAlloc = Allocation.createTyped(mRS, tRgb.create(), Allocation.USAGE_SCRIPT);
// Set input of ScriptIntrinsicYuvToRGB
mYuvToRGB.setInput(yuvPreviewAlloc);
}
// Use rsPreviewSurface as one of the output surface from Camera API.
// You can refer to https://github.com/googlesamples/android-HdrViewfinder/blob/master/Application/src/main/java/com/example/android/hdrviewfinder/HdrViewfinderActivity.java#L504
Surface rsPreviewSurface = yuvPreviewAlloc.getSurface();
...
// Whenever a new frame is available
// Update the yuv Allocation with a new Camera buffer without any copy.
// You can refer to https://github.com/googlesamples/android-HdrViewfinder/blob/master/Application/src/main/java/com/example/android/hdrviewfinder/ViewfinderProcessor.java#L109
yuvPreviewAlloc.ioReceive();
// The actual Yuv to Rgb conversion.
mYuvToRGB.forEach(rgbOutputAlloc);
// Copy the rgb Allocation to a Bitmap.
rgbOutputAlloc.copyTo(mBitmap);
// continue processing mBitmap.
...
When using ScriptIntrinsics I highly recommend to update to at least JellyBean 4.3 or higher (API18). Things are much easier to use than in JB 4.2 (API 17).
ScriptIntrinsicYuvToRGB is not as complicated as it seems.
Especially you don´t need Type.Builder objects.
Camera preview format must be NV21 !
in the onCreate()... method create the RenderScript object and the Intrinsic:
mRS = RenderScript.create(this);
mYuvToRGB = ScriptIntrinsicYuvToRGB.create(mRS, Element.U8_4(mRS));
With your cameraPreviewWidth and cameraPreviewHeight calculate the
length of the camera data byte array:
int yuvDatalength = cameraPreviewWidth*cameraPreviewHeight*3/2 ; // this is 12 bit per pixel
You need a bitmap for output:
mBitmap = Bitmap.createBitmap(cameraPreviewWidth, cameraPreviewHeight, Bitmap.Config.ARGB_8888);
Then you create the input and output allocations (here are the changes in API18+)
yuvPreviewAlloc = Allocation.createSized(mRS, Element.U8(mRS), yuvDatalength);
rgbOutputAlloc = Allocation.createFromBitmap(mRS, mBitmap); // this simple !
and set the script-input to the input allocation
mYuvToRGB.setInput(yuvPreviewAlloc); // this has to be done only once !
In the camera loop (whenever a new frame is avaliable), copy the NV21 byte-array (data[]) to the yuvPreviewAlloc, execute the script and copy result to bitmap:
yuvPreviewAlloc.copyFrom(data); // or yuvPreviewAlloc.copyFromUnchecked(data);
mYuvToRGB.forEach(rgbOutputAlloc);
rgbOutputAlloc.copyTo(mBitmap);
For example: on Nexus 7 (2013, JellyBean 4.3) a full HD (1920x1080) camera preview conversion takes about 7 ms.
I was able to get a different method working (one that was previously linked) by using the code here. But that was giving the Red/Blue color flip. So, I just rearranged the U and V lines and all was ok. This is not as fast as a RenderScript though. It would be good to have a RenderScript that functioned properly. Here is the code:
static public void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width, int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0) y = 0;
if ((i & 1) == 0) {
u = (0xff & yuv420sp[uvp++]) - 128; //Just changed the order
v = (0xff & yuv420sp[uvp++]) - 128; //It was originally v then u
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0) r = 0; else if (r > 262143) r = 262143;
if (g < 0) g = 0; else if (g > 262143) g = 262143;
if (b < 0) b = 0; else if (b > 262143) b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
}
Any one have a RenderScript that doesn't have color tint and or flip problems?