Develop Reference

Tensorflow Lite - Input shape must be 5 dimensional error

I am trying to port a tensorflow model to tensorflow lite to use it in an android application. The conversion is successful and everything runs except for Internal error: Failed to run on the given Interpreter: input must be 5-dimensional. The input in the original model was input_shape=(20, 320, 240, 1), which is 20 320 x 240 grayscale images (therefore ...,1). Here is the important code:
List<Mat> preprocessedFrames = preprocFrames(buf);
//has length of 20 -> no problem there (shouldn't affect dimensionality either...)
int[] output = new int[2];
float[][][] inputMatrices = new float[preprocessedFrames.toArray().length][320][240];
for(int i = 0; i < preprocessedFrames.toArray().length; i++) {
Mat inpRaw = preprocessedFrames.get(i);
Bitmap data = Bitmap.createBitmap(inpRaw.cols(), inpRaw.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(inpRaw, data);
int[][] pixels = pixelsFromBitmap(data);
float[][] inputMatrix = inputMatrixFromIntPixels(pixels);
// returns float[][] with floats from 0 to 1
inputMatrices[i] = inputMatrix;
}
try{
detector.run(inputMatrices, output);
Debug("results: " + output.toString());
}
The model gives me an output of 2 neurons translating into 2 labels.
The model code is the following:
model = tf.keras.Sequential(name='detector')
model.add(tf.keras.layers.Conv3D(filters=(56), input_shape=(20, 320, 240, 1), strides=(2,2,2), kernel_size=(3,11,11), padding='same', activation="relu"))
model.add(tf.keras.layers.AveragePooling3D(pool_size=(1,4,4)))
model.add(tf.keras.layers.Conv3D(filters=(72), kernel_size=(4,7,7), strides=(1,2,2), padding='same'))
model.add(tf.keras.layers.Conv3D(filters=(81), kernel_size=(2,4,4), strides=(2,2,2), padding='same'))
model.add(tf.keras.layers.Conv3D(filters=(100), kernel_size=(1,2,2), strides=(3,2,2), padding='same'))
model.add(tf.keras.layers.Conv3D(filters=(128), kernel_size=(1,2,2), padding='same'))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(768, activation='tanh', kernel_regularizer=tf.keras.regularizers.l2(0.011)))
model.add(tf.keras.layers.Dropout(rate=0.1))
model.add(tf.keras.layers.Dense(256, activation='sigmoid', kernel_regularizer=tf.keras.regularizers.l2(0.012)))
model.add(tf.keras.layers.Dense(2, activation='softmax'))
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.00001), loss=tf.keras.losses.CategoricalCrossentropy(),
metrics=['accuracy'])
EDIT: I printed out the first input tensor as follows:
int[] shape = detector.getInputTensor(0).shape();
for(int r = 0; r < shape.length; r++){
Log.d("********" + r, "*******: " + r + " : " + shape[r]);
}
With that I first get the output [1,20,320,240,1]and after that I only get [20,320,240]. I am really quite desperate now...

So, I figured it out by myself and it seems like I really only had to make the input 5 dimensional by putting the content into a first dimension and every single pixel into a fifth dimension. I don't know why, but I will accept that xD.
float[][] output = new float[1][2];
float[][][][][] inputMatrices = new float[1][preprocessedFrames.toArray().length][320][240][1];
for(int i = 0; i < preprocessedFrames.toArray().length; i++) {
Mat inpRaw = preprocessedFrames.get(i);
Bitmap data = Bitmap.createBitmap(inpRaw.cols(), inpRaw.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(inpRaw, data);
int[][] pixels = pixelsFromBitmap(data);
float[][] inputMatrix = inputMatrixFromIntPixels(pixels);
for (int j = 0; j < inputMatrix.length - 1; j++) {
for(int k = 0; k < inputMatrix[0].length - 1; k++) {
inputMatrices[0][i][k][j][0] = inputMatrix[j][k];
}
}
}

Watershed algorithm with touch on image to add marker OpenCV android

I am trying to save the watershed segments as image to sdcard in android.
Code tried in c++ and working fine
for (int m = 0; m < images.size(); m++) {
//wshed = wshed*0.5 + imgGray*0.5;
cv::Mat input_bgra;
cv::cvtColor(images[m], input_bgra, CV_BGR2BGRA);
// find all white pixel and set alpha value to zero:
for (int y = 0; y < input_bgra.rows; ++y)
for (int x = 0; x < input_bgra.cols; ++x)
{
cv::Vec4b & pixel = input_bgra.at<cv::Vec4b>(y, x);
// if pixel is black
if (pixel[0] == 0 && pixel[1] == 0 && pixel[2] == 0)
{
// set alpha to zero:
pixel[3] = 0;
}
}
std::ostringstream name;
name << "D:/Sathiya/res/intlayer" << m << ".png";
imwrite(name.str(), input_bgra);
}
Not sure how to achieve this in android.

I am trying to save watershed segments as png in java , is it straightforward to do.

Histogram Matching in Renderscript

In order to align the intensity values of two grayscale Images (as a first step for further processing) I wrote a Java method that:
converts the bitmaps of the two images into two int[] arrays containing the bitmap's intensities (I just take the red component here, since it's grayscale, i.e. r=g=b ).
public static int[] bmpToData(Bitmap bmp){
int width = bmp.getWidth();
int height = bmp.getHeight();
int anzpixel = width*height;
int [] pixels = new int[anzpixel];
int [] data = new int[anzpixel];
bmp.getPixels(pixels, 0, width, 0, 0, width, height);
for (int i = 0 ; i < anzpixel ; i++) {
int p = pixels[i];
int r = (p & 0xff0000) >> 16;
//int g = (p & 0xff00) >> 8;
//int b = p & 0xff;
data[i] = r;
}
return data;
}
aligns the cumulated intensity distributions of Bitmap 2 to that of Bitmap 1
//aligns the intensity distribution of a grayscale picture moving (given by int[] //data2) the the intensity distribution of a reference picture fixed (given by // int[] data1)
public static int[] histMatch(int[] data1, int[] data2){
int anzpixel = data1.length;
int[] histogram_fixed = new int[256];
int[] histogram_moving = new int[256];
int[] cumhist_fixed = new int[256];
int[] cumhist_moving = new int[256];
int i=0;
int j=0;
//read intensities of fixed und moving in histogram
for (int n = 0; n < anzpixel; n++) {
histogram_fixed[data1[n]]++;
histogram_moving[data2[n]]++;
}
// calc cumulated distributions
cumhist_fixed[0]=histogram_fixed[0];
cumhist_moving[0]=histogram_moving[0];
for ( i=1; i < 256; ++i ) {
cumhist_fixed[i] = cumhist_fixed[i-1]+histogram_fixed[i];
cumhist_moving[i] = cumhist_moving[i-1]+histogram_moving [i];
}
// look-up-table lut[]. For each quantile i of the moving picture search the
// value j of the fixed picture where the quantile is the same as that of moving
int[] lut = new int[anzpixel];
j=0;
for ( i=0; i < 256; ++i ){
while(cumhist_fixed[j]< cumhist_moving[i]){
j++;
}
// check, whether the distance to the next-lower intensity is even lower, and if so, take this value
if ((j!=0) && ((cumhist_fixed[j-1]- cumhist_fixed[i]) < (cumhist_fixed[j]- cumhist_fixed[i]))){
lut[i]= (j-1);
}
else {
lut[i]= (j);
}
}
// apply the lut[] to moving picture.
i=0;
for (int n = 0; n < anzpixel; n++) {
data2[n]=(int) lut[data2[n]];
}
return data2;
}
converts the int[] arrays back to Bitmap.
public static Bitmap dataToBitmap(int[] data, int width, int heigth) {
int index=0;
Bitmap bmp = Bitmap.createBitmap(width, heigth, Bitmap.Config.ARGB_8888);
for (int x = 0; x < width; x++) {
for (int y = 0; y < heigth; y++) {
index=y*width+x;
int c = data[index];
bmp.setPixel(x,y,Color.rgb(c, c, c));
}
}
return bmp;
}
While the core procedure 2) is straightforward and fast, the conversion steps 1) and 3) are rather inefficient. It would be more than cool to do the whole thing in Renderscript. But, honestly, I am completely lost in doing so because of missing documentation and, while there are many impressing examples on what Renderscript COULD perform, I don't see a way to benefit from these possibilities (no books, no docu). Any advice is highly appreciated!

As a starting point, use Android Studio to "Import Sample..." and select Basic Render Script. This will give you a working project that we will now modify.
First, let's add more Allocation references to MainActivity. We will use them to communicate image data, histograms and the LUT between Java and Renderscript.
private Allocation mInAllocation;
private Allocation mInAllocation2;
private Allocation[] mOutAllocations;
private Allocation mHistogramAllocation;
private Allocation mHistogramAllocation2;
private Allocation mLUTAllocation;
Then in onCreate() load another image, which you will also need to add to /res/drawables/.
mBitmapIn2 = loadBitmap(R.drawable.cat_480x400);
In createScript() create additional allocations:
mInAllocation2 = Allocation.createFromBitmap(mRS, mBitmapIn2);
mHistogramAllocation = Allocation.createSized(mRS, Element.U32(mRS), 256);
mHistogramAllocation2 = Allocation.createSized(mRS, Element.U32(mRS), 256);
mLUTAllocation = Allocation.createSized(mRS, Element.U32(mRS), 256);
And now the main part (in RenderScriptTask):
/*
* Invoke histogram kernel for both images
*/
mScript.bind_histogram(mHistogramAllocation);
mScript.forEach_compute_histogram(mInAllocation);
mScript.bind_histogram(mHistogramAllocation2);
mScript.forEach_compute_histogram(mInAllocation2);
/*
* Variables copied verbatim from your code.
*/
int []histogram_fixed = new int[256];
int []histogram_moving = new int[256];
int[] cumhist_fixed = new int[256];
int[] cumhist_moving = new int[256];
int i=0;
int j=0;
// copy computed histograms to Java side
mHistogramAllocation.copyTo(histogram_fixed);
mHistogramAllocation2.copyTo(histogram_moving);
// your code again...
// calc cumulated distributions
cumhist_fixed[0]=histogram_fixed[0];
cumhist_moving[0]=histogram_moving[0];
for ( i=1; i < 256; ++i ) {
cumhist_fixed[i] = cumhist_fixed[i-1]+histogram_fixed[i];
cumhist_moving[i] = cumhist_moving[i-1]+histogram_moving [i];
}
// look-up-table lut[]. For each quantile i of the moving picture search the
// value j of the fixed picture where the quantile is the same as that of moving
int[] lut = new int[256];
j=0;
for ( i=0; i < 256; ++i ){
while(cumhist_fixed[j]< cumhist_moving[i]){
j++;
}
// check, whether the distance to the next-lower intensity is even lower, and if so, take this value
if ((j!=0) && ((cumhist_fixed[j-1]- cumhist_fixed[i]) < (cumhist_fixed[j]- cumhist_fixed[i]))){
lut[i]= (j-1);
}
else {
lut[i]= (j);
}
}
// copy the LUT to Renderscript side
mLUTAllocation.copyFrom(lut);
mScript.bind_LUT(mLUTAllocation);
// Apply LUT to the destination image
mScript.forEach_apply_histogram(mInAllocation2, mInAllocation2);
/*
* Copy to bitmap and invalidate image view
*/
//mOutAllocations[index].copyTo(mBitmapsOut[index]);
// copy back to Bitmap in preparation for viewing the results
mInAllocation2.copyTo((mBitmapsOut[index]));
Couple notes:
In your part of the code I also fixed LUT allocation size - only 256 locations are needed,
As you can see, I left the computation of cumulative histogram and LUT on Java side. These are rather difficult to efficiently parallelize due to data dependencies and small scale of the calculations, but considering the latter I don't think it's a problem.
Finally, the Renderscript code. The only non-obvious part is the use of rsAtomicInc() to increase values in histogram bins - this is necessary due to potentially many threads attempting to increase the same bin concurrently.
#pragma version(1)
#pragma rs java_package_name(com.example.android.basicrenderscript)
#pragma rs_fp_relaxed
int32_t *histogram;
int32_t *LUT;
void __attribute__((kernel)) compute_histogram(uchar4 in)
{
volatile int32_t *addr = &histogram[in.r];
rsAtomicInc(addr);
}
uchar4 __attribute__((kernel)) apply_histogram(uchar4 in)
{
uchar val = LUT[in.r];
uchar4 result;
result.r = result.g = result.b = val;
result.a = in.a;
return(result);
}

OpenCV convert color per pixel

Hello i want to convert the color in image, i'm using per-pixel methods but it seems very slow
src.getPixels(pixels, 0, width, 0, 0, width, height);
// RGB values
int R;
for (int i = 0; i < pixels.length; i++) {
// Get RGB values as ints
// Set pixel color
pixels[i] = color;
}
// Set pixels
src.setPixels(pixels, 0, width, 0, 0, width, height);
my question, is there any way i can do it using openCV? change pixel to the color i want ?

I recommend this excellent article on how to access/modify an opencv image buffer. I recommend
"the efficient way":
int i,j;
uchar* p;
for( i = 0; i < nRows; ++i)
{
p = I.ptr<uchar>(i);
for ( j = 0; j < nCols; ++j)
{
p[j] = table[p[j]];
}
Or "the iterator-safe method":
MatIterator_<Vec3b> it, end;
for( it = I.begin<Vec3b>(), end = I.end<Vec3b>(); it != end; ++it)
{
(*it)[0] = table[(*it)[0]];
(*it)[1] = table[(*it)[1]];
(*it)[2] = table[(*it)[2]];
}
For further optimizations, using cv::LUT() (where possible) can give huge speedups, but it is more intensive to design/code.

You can access Pixels by using:
img.at<Type>(y, x);
So to change an RGB Value you can use:
// read color
Vec3b intensity = img.at<Vec3b>(y, x);
// compute new color using intensity.val[0] etc. to access color values
// write new color
img.at<Vec3b>(y, x) = intensity;
#Boyko mentioned an Article from OpenCV concerning fast access to the image pixels if you want to iterate over all Pixel. The Method I would prefer from this Article is the iterator Method, as it is only slightly slower than direct pointer access but safer to use.
Example Code:
Mat& AssignNewColors(Mat& img)
{
// accept only char type matrices
CV_Assert(img.depth() != sizeof(uchar));
const int channels = img.channels();
switch(channels)
{
// case 1: skipped here
case 3:
{
// Read RGG Pixels
Mat_<Vec3b> _img = img;
for( int i = 0; i < img.rows; ++i)
for( int j = 0; j < img.cols; ++j )
{
_img(i,j)[0] = computeNewColor(_img(i,j)[0]);
_img(i,j)[1] = computeNewColor(_img(i,j)[1]);
_img(i,j)[2] = computeNewColor(_img(i,j)[2]);
}
img = _img;
break;
}
}
return img;
}

Compare RGB color values of two images in android

To get RGB values of one image i used the follwing code snippet
int[] pix = new int[picw * pich];
bitmap.getPixels(pix, 0, picw, 0, 0, picw, pich);
int R, G, B,Y;
for (int y = 0; y < pich; y++){
for (int x = 0; x < picw; x++)
{
int index = y * picw + x;
int R = (pix[index] >> 16) & 0xff; //bitwise shifting
int G = (pix[index] >> 8) & 0xff;
int B = pix[index] & 0xff;
//R,G.B - Red, Green, Blue
//to restore the values after RGB modification, use
//next statement
pix[index] = 0xff000000 | (R << 16) | (G << 8) | B;
}}
I want to compare two images,i know that comparing pixel values would be more expensive.I also analysed OpenCV library but i won't
get into my requirement.
Is there any algorithm to compare images using RGB values in android?
or
Is any other method to compare RGB values?
Thanks,

I'm not sure what your requirements are, but if all you want to do is compare the (RGB) color palettes of two images, you might want to use the PaletteFactory methods from Apache Commons Imaging (fka "Sanselan"):
The PaletteFactory methods build up collections (int[] and List<>) which can then be iterated over. I'm not sure just what kind of comparison you need to do, but a fairly simple case, using e.g. makeExactRgbPaletteSimple(), would be:
final File img1 = new File("path/to/image_1.ext")
final File img2 = new File("path/to/image_2.ext")
final PaletteFactory pf;
final int MAX_COLORS = 256;
final Palette p1 = pf.makeExactRgbPaletteSimple(img1, MAX_COLORS);
final Palette p2 = pf.makeExactRgbPaletteSimple(img2, MAX_COLORS);
final ArrayList<Int> matches = new ArrayList<Int>(Math.max(p1.length(), p2.length()));
int matchPercent;
// Palette objects are pre-sorted, afaik
if ( (p1 != null) && (p2 != null) ) {
if (p1.length() > p2.length()) {
for (int i = 0; i < p1.length(); i++) {
final int c1 = p1.getEntry(i);
final int c2 = p2.getPaletteIndex(c1);
if (c2 != -1) {
matches.add(c1);
}
}
matchPercent = ( (int)( (float)matches.size()) / ((float)p1.length) * 100 ) )
} else if (p2.length() >= p1.length()) {
for (int i = 0; i < p1.length(); i++) {
final int c1 = p2.getEntry(i);
final int c2 = p1.getPaletteIndex(c1);
if (c2 != -1) {
matches.add(c1);
}
}
matchPercent = ( (int)( (float)matches.size()) / ((float)p2.length) * 100 ) )
}
}
This is just a minimal example which may or may not compile and is almost certainly not what you're looking for in terms of comparison logic.
Basically what it does is check if each member of p1 is also a member of p2, and if so, adds it to matches. Hopefully the logic is correct, no guarantees. matchPercent is the percentage of colors which exist in both Palettes.
This is probably not the comparison method you want. It is just a simple example.
You will definitely need to play around with the 2nd parameter to makeExactRgbPaletteSimple(), int max, as I chose 256 arbitrarily - remember, the method will (annoyingly, imo) return null if max is too small.
I would suggest building from source as the repos have not been updated for quite some time. The project is definitely not mature, but it is fairly small, reasonably fast for medium-sized images, and pure Java.
Hope this helps.