Develop Reference

Tensorflow Lite model run does not change the output buffer

The Problem
I have been working on implementing a super resolution model with Tensorflow Lite. I have an empty bitmap 4x the size of the input bitmap (which is bmp):
Bitmap out = Bitmap.createBitmap(bmp.getWidth() * 4, bmp.getHeight() * 4, Bitmap.Config.ARGB_8888);
And I converted both bitmaps to TensorImages
TensorImage originalImage = TensorImage.fromBitmap(bmp);
TensorImage superImage = TensorImage.fromBitmap(out);
However, when I run the model (InterpreterApi tflite):
tflite.run(originalImage.getBuffer(), superImage.getBuffer());
The bitmap from superImage has not changed, and it holds the blank bitmap I made at the start.
superImage.getBitmap();
Things I've tried
I looked at basic examples and documentation, most are geared toward classification but they all seemed to do it this way.
I fed the input bitmap to the output, and my app showed the input, so I know that the file picking and preview works.
I tested with different datatypes to store the output, and they either left it blank or weren't compatible with Tensorflow.
What I think
I suspect the problem has something to do with tflite.run() changing a separate instance of superImage, and I get left with the old one. I may also need a different data format that I haven't tried yet.
Thank you for your time.

Creating Graphics on Android Tiles

I am creating an Android Tile that is meant to display custom and dynamically created graphics, i.e. a chart.
However, due to several limitations I have yet to find a way to do so. Tiles seem to work fundamentally different than Activities do and the Tiles' API only allows for several, predefined UI elements to be created. The only usable one for me seems to be the Image LayoutElement.
The Image can be created by either passing a resource or a ByteArray. Former is not possible when dealing with dynamically created graphs.
Thus, my only hope (I think) is to create an Image in the form of a ByteArray myself.
How can I do this? Is there any Java framework to draw graphics directly?
I have considered the following:
Using the provided UI elements: wouldn't work since the placement is way to imprecise and the exact position of an element cannot be controlled. Also, these elements are not meant for drawing.
Using AWT: doesn't work on Android. Thus, almost any drawing and/or charting library is out of the game.
JavaFX: would probably work but there seems to be now way to draw directly on ByteArrays/BufferedImages as the application needs to be rendered first. Rendering JavaFX doesn't seem possible for Tiles.
Using Android's Canvas: again, an Activity is needed.

Turns out I was wrong: you can very well use the Canvas within a Tile. Converting it to a resource is, however, a little tricky, so here's some code:
final Bitmap bitmap = Bitmap.createBitmap(chart.getWidth(), chart.getHeight(),
Bitmap.Config.RGB_565);
final Canvas canvas = new Canvas(bitmap);
// Sets the background color
final Color background = Color.valueOf(chart.getBackgroundColor());
canvas.drawRGB(
Math.round(background.red() * 255),
Math.round(background.green() * 255),
Math.round(background.blue() * 255)
);
// YOUR DRAWING OPERATIONS: e.g. canvas.drawRect
final ByteBuffer byteBuffer = ByteBuffer.allocate(bitmap.getByteCount());
bitmap.copyPixelsToBuffer(byteBuffer);
final byte[] bytes = byteBuffer.array();
return new ResourceBuilders.ImageResource.Builder()
.setInlineResource(
new ResourceBuilders.InlineImageResource.Builder()
.setData(bytes)
.setWidthPx(chart.getWidth())
.setHeightPx(chart.getHeight())
.setFormat(ResourceBuilders.IMAGE_FORMAT_RGB_565)
.build()
)
.build();

This example shows using Compose Canvas to render charts for Tiles.
https://github.com/google/horologist/pull/249
Also you can encode to PDF
Remove
setFormat(ResourceBuilders.IMAGE_FORMAT_RGB_565)
and use
val bytes = ByteArrayOutputStream().apply {
compress(Bitmap.CompressFormat.PNG, 100, this)
}.toByteArray()

Algorithm to floodfill bitmap using android Renderscript

I an trying to make an app which fill color to images. It is working fine using Java, but due to some performance issue I want to fill bitmaps using renderscript. I googled lots of things about renderscript but I haven't got anything suitable. Can you please guys guide me how to fill bitmaps using renderscript. Any help will be appreciated. Thanks

The basic thing you'll have to do is create an Allocation for the input Bitmap then a mutable Bitmap and Allocation for the output. Assuming you have an input bitmap called inputBitmap it could look something like this:
private RenderScript mRsCtx; // RenderScript context, created during init
private ScriptC_bitmapFill mFill; // RS kernel instance, created during init
.
.
.
public Bitmap doFill(Bitmap inputBitmap) {
// Ensure your input bitmap is also in ARGB8888
Bitmap output = Bitmap.createBitmap(inputBitmap.getWidth(),
inputBitmap.getHeight(),
Bitmap.Config.ARGB_8888);
Allocation outAlloc = Allocation.createFromBitmap(mRsCtx, output);
Allocation inAlloc = Allocation.createFromBitmap(mRsCtx, inputBitmap);
// Now call your kernel then copy back the results
mFill.forEach_root(inAlloc, outAlloc);
outAlloc.copyTo(outBitmap);
return outBitmap;
}
If you are just filling the entire image or even a region, you'll then have a RS kernel which will change the pixel value at specific locations when the kernel is called for it. Here's a very simple RS kernel which just fills the entire image with a solid color:
#pragma version(1)
#pragma rs java_package_name(com.example.bitmapfill)
void root(const uchar4 *v_in, uchar4 *v_out) {
v_out->r = 0x12;
v_out->g = 0x34;
v_out->b = 0x56;
}
Note that since you're not really doing anything with the input allocation/bitmap in this case (just filling the entire thing), you could just leave out the input allocation and use the dimensions. But, if you are only going to manipulate a portion of the input (a small subsection), then you'll have to copy the other pixels from input to output instead of filling.
For additional information about RS and some of its internals, performance, etc. you may find this talk useful: https://youtu.be/3ynA92x8WQo

Has anyone managed to obtain a YUV_420_888 frame using RenderScript and the new Camera API?

I'm using RenderScript and Allocation to obtain YUV_420_888 frames from the Android Camera2 API, but once I copy the byte[] from the Allocation I receive only the Y plane from the 3 planes which compose the frame, while the U and V planes values are set to 0 in the byte[]. I'm trying to mimic the onPreviewframe from the previos camera API in order to perform in app processing of the camera frames. My Allocation is created like:
Type.Builder yuvTypeBuilderIn = new Type.Builder(rs, Element.YUV(rs));
yuvTypeBuilderIn.setX(dimensions.getWidth());
yuvTypeBuilderIn.setY(dimensions.getHeight());
yuvTypeBuilderIn.setYuvFormat(ImageFormat.YUV_420_888);
allocation = Allocation.createTyped(rs, yuvTypeBuilderIn.create(),
Allocation.USAGE_IO_INPUT | Allocation.USAGE_SCRIPT);
while my script looks like:
#pragma version(1)
#pragma rs java_package_name(my_package)
#pragma rs_fp_relaxed
rs_allocation my_frame;

The Android sample app HdrViewfinderDemo uses RenderScript to process YUV data from camera2.
https://github.com/googlesamples/android-HdrViewfinder
Specifically, the ViewfinderProcessor sets up the Allocations, and hdr_merge.rs reads from them.

Yes I did it, since I couldn't find anything useful. But I didn't go the proposed way of defining an allocation to the surface. Instead I just converted the output of the three image planes to RGB. The reason for this approach is that I use the YUV420_888 data twofold. First on a high frequency basis just the intensity values (Y). Second, I need to make some color Bitmaps too. Thus, the following solution. The script takes about 80ms for a 1280x720 YUV_420_888 image, maybe not ultra fast, but ok for my purpose.
UPDATE: I deleted the code here, since I wrote a more general solution here YUV_420_888 -> Bitmap conversion that takes into account pixelStride and rowStride too.

I think that you can use an ImageReader to get the frames of you camera into YUV_420_888
reader = ImageReader.newInstance(previewSize.getWidth(), previewSize.getHeight(), ImageFormat.YUV_420_888, 2);
Then you set an OnImageAvailableListener to the reader :
reader.setOnImageAvailableListener(new ImageReader.OnImageAvailableListener() {
#Override
public void onImageAvailable(ImageReader reader) {
int jump = 4; //Le nombre d'image à sauter avant d'en traiter une, pour liberer de la mémoire
Image readImage = reader.acquireNextImage();
readImage.getPlane[0] // The Y plane
readImage.getPlane[1] //The U plane
readImage.getPlane[2] //The V plane
readImage.close();
}
}, null);
Hope that will help you

I'm using almost the same method as widea in their answer.
The exception you keep getting after ~50 frames might be due to the fact that you're processing all the frames by using acquireNextImage. The documentation suggest to:
Warning: Consider using acquireLatestImage() instead, as it will automatically release older images, and allow slower-running processing routines to catch up to the newest frame. [..]
So in case your exception is a IllegalStateException, switching to acquireLatestImage might help.
And make sure you call close() on all images retrieved from ImageReader.

Resize bitmap using Renderscript

I'm currently working on a project where I have to use RenderScript, so i started learning about it, and it's a great technology, because, just like openGL, it lets you use computational code that goes to a native level, and doesn't have to use the dalvik vm. This part of the code, being processed much faster than if you would use normal android code.
I started working with image processing and what i was wondering is:
Is it possible to resize a bitmap using RenderScript? this should be much faster then resizing an bitmap using android code. Plus, renderscript can process information that is bigger than 48mB (limit on some phones for each process).

While you could use Rendscript to do the bitmap resize, I'm not sure if that's the best choice. A quick look at the Android code base shows that Java API does go into native code to do a bitmap resize, although if the resize algorithm isn't to your needs, you'll have to implement your own.
There are a number of answers on SO for getting the bitmap scaled efficiently. My recommendation is to try those, and if they still aren't doing what your want, either as quickly or how the results appear visually to then investigate into writing your own. If you still want to write your own, do use the performance tools available to see if you really are faster or just reinventing the wheel.

You can use the below function to resize the image.
private Bitmap resize(Bitmap inBmp) {
RenderScript mRs = RenderScript.create(getApplication());
Bitmap outBmp = Bitmap.createBitmap(OUTPUT_IMAGE_WIDTH, inBmp.getHeight() * OUTPUT_IMAGE_WIDTH /inBmp.getWidth(), inBmp.getConfig());
ScriptIntrinsicResize siResize = ScriptIntrinsicResize.create(mRs);
Allocation inAlloc = Allocation.createFromBitmap(mRs, inBmp);
Allocation outAlloc = Allocation.createFromBitmap(mRs, outBmp);
siResize.setInput(inAlloc);
siResize.forEach_bicubic(outAlloc);
outAlloc.copyTo(outBmp);
inAlloc.destroy();
outAlloc.destroy();
siResize.destroy();
return outBmp;
}
OUTPUT_IMAGE is the integer value specifying the width of the output image.
NOTE: While using the RenderScript Allocation you have to be very careful as they lead to memory leakages.