Develop Reference

Tensorflow Lite model run does not change the output buffer

The Problem
I have been working on implementing a super resolution model with Tensorflow Lite. I have an empty bitmap 4x the size of the input bitmap (which is bmp):
Bitmap out = Bitmap.createBitmap(bmp.getWidth() * 4, bmp.getHeight() * 4, Bitmap.Config.ARGB_8888);
And I converted both bitmaps to TensorImages
TensorImage originalImage = TensorImage.fromBitmap(bmp);
TensorImage superImage = TensorImage.fromBitmap(out);
However, when I run the model (InterpreterApi tflite):
tflite.run(originalImage.getBuffer(), superImage.getBuffer());
The bitmap from superImage has not changed, and it holds the blank bitmap I made at the start.
superImage.getBitmap();
Things I've tried
I looked at basic examples and documentation, most are geared toward classification but they all seemed to do it this way.
I fed the input bitmap to the output, and my app showed the input, so I know that the file picking and preview works.
I tested with different datatypes to store the output, and they either left it blank or weren't compatible with Tensorflow.
What I think
I suspect the problem has something to do with tflite.run() changing a separate instance of superImage, and I get left with the old one. I may also need a different data format that I haven't tried yet.
Thank you for your time.

Template Matching for android-opencv application

I have developed an algorithm for android using OpenCV. I need to find overlap between the previous image and current frame. So, I have produced the template from previous image to match with current frame to make a photograph. It is the procedure to complete photographing. (Taking more than 10 picture)
Here is the code that I have developed to find the overlap.
public void overlapFinder(Mat inputFrame , Mat inputTemplate )
{
Mat mResult;
int resultWidth = inputFrame.width() - inputTemplate.width() + 1;
int resultHeight = inputFrame.height() - inputTemplate.height() + 1;
mResult = new Mat(resultHeight, resultWidth, CvType.CV_8U);
Imgproc.matchTemplate(inputFrame, inputTemplate, mResult,Imgproc.TM_CCORR_NORMED) ;
Core.MinMaxLocResult result = Core.minMaxLoc(mResult);
#SuppressWarnings("unused")
double maxVal = result.maxVal;
}
The problem is that when the "overlap function" is called after generating template from previous image, the application is crashed.
Would anyone please help me with that?
Thanks

Maybe you really need to do some debugging first, but in any case, I can see from your code that it would be worthwhile checking the sizes of your images - it seems that your code assumes the template is always smaller than the input frame.
If that's not true, you will get negative resultWidth and/or resultHeight, which will make it crash.
One other thing - the documentation suggests that the result type should be CV_32FC1.
PS - Try initialising your result like this:
mResult.create(resultHeight, resultWidth, CvType.CV_32FC1);

Fast dynamic vertices in OpenGL ES 2.0 on Android

I am trying to batch draw a bunch of lines on Android using OpenGL ES 2.0 and I need to know the best way to do this.
Right now I made a class called LineEngine which builds up a FloatBuffer of all the vertices to draw and then draws all the lines at once. The problem is that apparently FloatBuffer.put() is very slow and is gobbling up CPU time like crazy.
Here is my class
public class LineEngine {
private static final float[] IDENTIY = new float[16];
private FloatBuffer mLinePoints;
private FloatBuffer mLineColors;
private int mCount;
public LineEngine(int maxLines) {
Matrix.setIdentityM(IDENTIY, 0);
ByteBuffer byteBuf = ByteBuffer.allocateDirect(maxLines * 2 * 4 * 4);
byteBuf.order(ByteOrder.nativeOrder());
mLinePoints = byteBuf.asFloatBuffer();
byteBuf = ByteBuffer.allocateDirect(maxLines * 2 * 4 * 4);
byteBuf.order(ByteOrder.nativeOrder());
mLineColors = byteBuf.asFloatBuffer();
reset();
}
public void addLine(float[] position, float[] color){
mLinePoints.put(position, 0, 8); //These lines
mLineColors.put(color, 0, 4); // are taking
mLineColors.put(color, 0, 4); // the longest!
mCount++;
}
public void reset(){
mLinePoints.position(0);
mLineColors.position(0);
mCount = 0;
}
public void draw(){
mLinePoints.position(0);
mLineColors.position(0);
GraphicsEngine.setMMatrix(IDENTIY);
GraphicsEngine.setColors(mLineColors);
GraphicsEngine.setVertices4d(mLinePoints);
GraphicsEngine.disableTexture();
GLES20.glDrawArrays(GLES20.GL_LINES, 0, mCount * 2);
GraphicsEngine.disableColors();
reset();
}
}
Is there a better way to batch all these lines together?

What you are trying to do is called SpriteBatching. If you want your program to be robust in openGL es you have to check the following( a list of what makes your program slow ) :
-Changing to many opengl ES states each frame.
-Enable//Dissable (textures etc) again and again. Once you enable something you dont have to do it again it will be applied automaticly its frame.
-Using to many assets instead of spriteSheets. Yes thats make a huge performance impact. For example if you have 10 images you have to load for each image a different texture and that is SLOW. A better way to do this is to create a spriteSheet.(You can check google for free spriteSheet creators).
-Your images are high quality. Yes! check your resolution and file extension. Prefer .png files.
-Care for the api 8 float buffer bug(you have to use int buffer instead to fix the bug).
-Using java containers. This is the biggest pitfall, if you use string concatenation(that's not a container but it returns a new string each time) or a List or any other container that RETURNS A NEW CLASS INSTANCE chances are your program will be slow due to garbage collection. For input handling i would suggest you to search a teqnique called the Pool Class. Its use is to recycle objects instead of creating new ones. Garbage collector is the big enemy, try to make him happy and avoid any programming technique that might call him.
-Never load things on the fly, instead make a loader class and load all the necessary assets in the begining of the app.
If you do implement this list then your chances are your program will be robust.
One last adition. What is a sprite batcher? A spriteBatcher is a class that uses a single texture to render multiple objects. It will create automaticly vertices, indices, colors, u - v coords and it will render them as a batch. This pattern will save GPU power as well as cpu power. From your posted code i can't be sure what causes the cpu to slow down but from my experience is due to one(or more) things of the list i previously mention. Check the list, follow it, search google for spriteBatching and i am sure your program will run fast. Hope i helped!
Edit: I think i found what causes your program to slow down, you dont flip the buffer dude! You just reset the position. You just add add add more objects and you cause buffer overload. In the reset method just flip the buffer. mLineColors.flip mLinePaints.flip will do the job. Make sure you call them each frame if you send new verices each frame.

FloatBuffer.put(float[]) with a relatively large float array should be considerably faster. The single put(float) calls have plenty of overhead.

Just go for a very simple native function which will be called from your class. You can put float[] to OpenGL directly, no need to kill CPU time with a silly buffer interface.

Android, using Bitmap (NDK + alpha bytes): i miss something?

I come from the Qt world and i am porting an application to Android. I am bit confused, i am banging my head for a few days now on something that must be so trivial that i cannot find why it's not working.
Some background: i have a C++ engine which i use trough NDK and JNI. This engine creates some bitmaps and passes them to the Java side, the Java side must display them on a View and let the user interact with them (drag and such).
The engine works properly, because i use it under Qt with full success. This is the workflow:
1- Java loads a big Bitmap from a custom data file (the C++ engine expects it to be in ARGB format, but it's compressed JPG data)
Bitmap.Config fmt = Bitmap.Config.ARGB_8888;
Bitmap bitmap = BitmapFactory.decodeByteArray(buffer, 0, size).copy( fmt , false);
2- initialize the C++ engine passing the bitmap. The C++ engine "breaks" the bitmap in smaller tiles. For tile it builds a rather complex alpha mask and stores it into the first byte of the bitmap (the "a" byte). This alpha mask only uses two values: 0xFF for opaque and 0x00 for transparent.
init_C_engine( this.fullImage );
3- The Java side then allocates all the tiles bitmaps, i do in two steps because before init i dont know which size will the tiles be. The engine will populate the tile_width and tile_height arrays:
Bitmap.Config fmt = Bitmap.Config.ARGB_8888;
for (int t = 0; t < this.puzzle_size; t++ ){
tile_data[ t ] = Bitmap.createBitmap( tile_width[t], tile_height[t], fmt);
4- Last step,inside the C++ engine, all the tiles bitmaps are filled:
for ( int n = 0; n < nBitmaps; n++ )
{
jobject bitmap = env->GetObjectArrayElement( bitmaps, n );
AndroidBitmap_getInfo(env, bitmap, &info);
AndroidBitmap_lockPixels(env, bitmap, reinterpret_cast<void **>(&pixels));
game->getTileBitmap( n, (unsigned char*)pixels );
AndroidBitmap_unlockPixels(env, bitmap);
env->SetObjectArrayElement( bitmaps, n, bitmap );
}
}
Now, in my custom View:
protected void onDraw(Canvas canvas) {
super.onDraw(canvas);
canvas.drawColor(Color.BLACK);
for ( int tile = 0; tile < board.nTiles; tile++ ){
canvas.drawBitmap( tile_data[tile],
tile_x[tile],
tile_y[tile], paint);
}
}
What i expect is that on my View i see my tiles with transparent areas. what i get instead is a weird behaviour so that on the black background i see the ENTIRE tile like the alpha bytes are all set to opaque, but when i move the tiles one of top of the other, the "transparent" areas get combined in some strange way, like colors are "xor"ed or multiplied in some way! When i move one tile on the other i can see the areas where the alpha bytes are set to transparent but colors gets mangled instead of being transparend!
Basically i expect that pixels having alpha set to 0 are drawn as transparent... i looked on internet but i could not find anything usefull to help me out....
Does somebody have ideas? Anything will be appreciated!
thanks.

Shouldn't you use the index t instead of tile inside the for loop inside onDraw? Like this:
canvas.drawBitmap(tile_data[t], tile_x[t], tile_y[t], paint);

android fast pixel access and manipulation

I'm trying to port an emulator that i have written in java to android. Things have been going nicely, I was able to port most of my codes with minor changes however due to how emulation works, I need to render image at pixel level.
As for desktop java I use
int[] pixelsA = ((DataBufferInt) src.getRaster().getDataBuffer()).getData();
which allow me to get the reference to the pixel buffer and update it on the fly(minimize object creations)
Currently this is what my emulator for android does for every frame
#Override
public void onDraw(Canvas canvas)
{
buffer = Bitmap.createBitmap(pixelsA, 256, 192, Bitmap.Config.RGB_565);
canvas.drawBitmap(buffer, 0, 0, null);
}
pixelsA is an array int[], pixelsA contains all the colour informations, so every frame it will have to create a bitmap object by doing
buffer = Bitmap.createBitmap(pixelsA, 256, 192, Bitmap.Config.RGB_565);
which I believe is quite expensive and slow.
Is there any way to draw pixels efficiently with canvas?

One quite low-level method, but working fine for me (with native code):
Create Bitmap object, as big as your visible screen.
Also create a View object and implement onDraw method.
Then in native code you'd load libjnigraphics.so native library, lookup functions AndroidBitmap_lockPixels and AndroidBitmap_unlockPixels.
These functions are defined in Android source in bitmap.h.
Then you'd call lock/unlock on a bitmap, receiving address to raw pixels. You must interpret RGB format of pixels accordingly to what it really is (16-bit 565 or 32-bit 8888).
After changing content of the bitmap, you want to present this on screen.
Call View.invalidate() on your View. In its onDraw, blit your bitmap into given Canvas.
This method is very low level and dependent on actual implementation of Android, however it's very fast, you may get 60fps no problem.
bitmap.h is part of Android NDK since platform version 8, so this IS official way to do this from Android 2.2.

You can use the drawBitmap method that avoids creating a Bitmap each time, or even as a last resort, draw the pixels one by one with drawPoint.

Don't recreate the bitmap every single time. Try something like this:
Bitmap buffer = null;
#Override
public void onDraw(Canvas canvas)
{
if(buffer == null) buffer = Bitmap.createBitmap(256, 192, Bitmap.Config.RGB_565);
buffer.copyPixelsFromBuffer(pixelsA);
canvas.drawBitmap(buffer, 0, 0, null);
}
EDIT: as pointed out, you need to update the pixel buffer. And the bitmap must be mutable for that to happen.

if pixelsA is already an array of pixels (which is what I would infer from your statement about containing colors) then you can just render them directly without converting with:
canvas.drawBitmap(pixelsA, 0, 256, 0, 0, 256, 192, false, null);