Develop Reference

Creating Graphics on Android Tiles

I am creating an Android Tile that is meant to display custom and dynamically created graphics, i.e. a chart.
However, due to several limitations I have yet to find a way to do so. Tiles seem to work fundamentally different than Activities do and the Tiles' API only allows for several, predefined UI elements to be created. The only usable one for me seems to be the Image LayoutElement.
The Image can be created by either passing a resource or a ByteArray. Former is not possible when dealing with dynamically created graphs.
Thus, my only hope (I think) is to create an Image in the form of a ByteArray myself.
How can I do this? Is there any Java framework to draw graphics directly?
I have considered the following:
Using the provided UI elements: wouldn't work since the placement is way to imprecise and the exact position of an element cannot be controlled. Also, these elements are not meant for drawing.
Using AWT: doesn't work on Android. Thus, almost any drawing and/or charting library is out of the game.
JavaFX: would probably work but there seems to be now way to draw directly on ByteArrays/BufferedImages as the application needs to be rendered first. Rendering JavaFX doesn't seem possible for Tiles.
Using Android's Canvas: again, an Activity is needed.

Turns out I was wrong: you can very well use the Canvas within a Tile. Converting it to a resource is, however, a little tricky, so here's some code:
final Bitmap bitmap = Bitmap.createBitmap(chart.getWidth(), chart.getHeight(),
Bitmap.Config.RGB_565);
final Canvas canvas = new Canvas(bitmap);
// Sets the background color
final Color background = Color.valueOf(chart.getBackgroundColor());
canvas.drawRGB(
Math.round(background.red() * 255),
Math.round(background.green() * 255),
Math.round(background.blue() * 255)
);
// YOUR DRAWING OPERATIONS: e.g. canvas.drawRect
final ByteBuffer byteBuffer = ByteBuffer.allocate(bitmap.getByteCount());
bitmap.copyPixelsToBuffer(byteBuffer);
final byte[] bytes = byteBuffer.array();
return new ResourceBuilders.ImageResource.Builder()
.setInlineResource(
new ResourceBuilders.InlineImageResource.Builder()
.setData(bytes)
.setWidthPx(chart.getWidth())
.setHeightPx(chart.getHeight())
.setFormat(ResourceBuilders.IMAGE_FORMAT_RGB_565)
.build()
)
.build();

This example shows using Compose Canvas to render charts for Tiles.
https://github.com/google/horologist/pull/249
Also you can encode to PDF
Remove
setFormat(ResourceBuilders.IMAGE_FORMAT_RGB_565)
and use
val bytes = ByteArrayOutputStream().apply {
compress(Bitmap.CompressFormat.PNG, 100, this)
}.toByteArray()

Android Resources GC Issue

I am using
Drawable drawable = res.getDrawable(id);
Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_4444);
Canvas canvas = new Canvas(bitmap);
bitmap.eraseColor(0);
drawable.setBounds(0,0, width, height);
drawable.draw(canvas);
return load(bitmap, linear);
to load a drawable from a resource id into OpenGL with a given width, and height. (Using
android.opengl.GLUtils.texImage2D(GL_TEXTURE_2D, 0, bitmap, 0);
)
The load function does the GL-calls, and calls also bitmap.recycle().
I specify width and height myself, because Android would match the resolution to the screen size, which I don't want.
Now my problem (this part is all working fine):
if I start my app for the first time, from Android Studio, everything works; HOWEVER if I want to restart it, it crashes because of OutOfMemoryError. I am doing the exactly same calls in both cases.
I located the issue to be in the resource management of Android, as you can see in the heap analysis:
my most expensive allocations
My images are way smaller than 9 MB each in raw (512x512, RGBA, so 1 MB).
How can I prevent Android from storing these large byte arrays, which probably are meant as some kind of cache; which however doesn't run on first start after app installation?
I am testing on Android 6.0.1, API Version 23, Galaxy S5.

Implementation of texImage2D looks like this:
public static void texImage2D(int target, int level, int internalformat,
Bitmap bitmap, int border) {
if (bitmap == null) {
throw new NullPointerException("texImage2D can't be used with a null Bitmap");
}
if (bitmap.isRecycled()) {
throw new IllegalArgumentException("bitmap is recycled");
}
if (native_texImage2D(target, level, internalformat, bitmap, -1, border)!=0) {
throw new IllegalArgumentException("invalid Bitmap format");
}
}
It doesn't look like it's recycling anything. Are you sure you are not loading a huge bitmap into memory? Two calls of those are more than enough to guarantee a huge explosion in your app, if not just one (I've seen it happen many times in my app). Remember, restarting your activity does not mean restarting your proccess.
Run the Android Profiler before the first load and check how much memory it takes.
Also, you can cache and reuse bitmaps yourself.

I solved it (myself) by putting the files into the raw folder of the resource directory, and loading them using
fun loadBitmap(res: Resources, rawId: Int): Bitmap {
val inputStream = BufferedInputStream(res.openRawResource(rawId))
return BitmapFactory.decodeStream(inputStream)
}
and then calling
load(bitmap, linear);
and
bitmap.recycle()
like before.
Luckily those all were png/jpeg files, so I didn't need the additional features of the drawables folder. Using this, they'll automatically use their right resolution.
My Java RAM allocation is now back on 25 MB to 35 MB instead of the 110 MB when using the old way :).

JavaCV Convert color HSV2RGB very slow

I have recently started with JavaCv using Android for camera preview image processing.
Basically, I take the camera preview, do some processing, convert it to HSV to modify some colors, and then I want to convert it to RGBA to fill a bitmap.
Everything works normally, but quite slow. In order to find the slowest part I made some measurements, and to my surprise found this line:
cvCvtColor( hsvimage, imageBitmap, CV_HSV2RGB); //<-- 50msecs
where hsvimage is a 3-channel IplImage, and imageBitmap is 4 channel.image. (The conversion is good and leaves the alpha channel to 255, giving an opaque bitmap as expected)
Just for comparison, the following two lines only take 3msec
cvCvtColor(yuvimage, bgrimage, CV_YUV2BGR_NV21);
cvCvtColor(bgrimage, hsvimage, CV_BGR2HSV);
(yuvimage is 1 channel IplImage, bgrimage and hsvimage are 3 channel IplImages)
It seems as if the first conversion (HSV2RGB) isn't so much optimized as others. Also tested it with a 3-channel destination image, just in case, but with the same results.
I would like to find a way to make it as fast as BGR2HSV. Possible ways:
Find if there is another "equivalent" constant to CV_HSV2RGB which is
faster
Get direct access to the H-S-V byte arrays and make my own "fast" conversion
in C.
Any idea to solve this issue will be welcome
--EDIT--
All this is happening with a small 320*240 image and running on a Xiaomi Redmi Note 4. Most of the operations such as converting color from RGB to HSV take less than 1 msec. Canny takes 5msec, Floodfill takes about 5 or 6 msec. It is only this conversion HSV2RGB which gives such strange results.
Will try to use OpenCV directly (not JavaCV) to see if this behaviour disappears.
I was using an old JavaCV version (0.11) Now I have updated to 1.3 and results are nearly the same
...
long startTime=System.currentTimeMillis();
cvCvtColor(hsvimage, imageBitmap, CV_HSV2RGB);
Log.w(LOG_TAG, "Time:" + String.valueOf(System.currentTimeMillis() - startTime)); //<-- From 45 to 50msec
Log.w(LOG_TAG,"Channels:"+imageBitmap.nChannels()); // <-- returns 4
I can fill a 32bit/pixel android bitmap with the result
Mat mim4C= new Mat(imageBitmap);
Mat mhsvimage = new Mat(hsvimage);
long startTime**strong text**=System.currentTimeMillis();
CvtColor(mhsvimage, mim4C, CV_HSV2RGB);
Log.w(LOG_TAG, "Time:" + String.valueOf(System.currentTimeMillis() - startTime)); //<-- From 45 to 50mse
IplImage iim4C=new IplImage(mim4C);
Log.w(LOG_TAG,"Channels:"+iim4C.nChannels()); // <-- returns 3!!!
In this second case, if I try to fill a 32bits/pixel android bitmap (after converting back mim4C to IplImage), it crashes since it has 3 channels

Image data from Android camera2 API flipped & squished on Galaxy S5

I am implementing an app that uses real-time image processing on live images from the camera. It was working, with limitations, using the now deprecated android.hardware.Camera; for improved flexibility & performance I'd like to use the new android.hardware.camera2 API. I'm having trouble getting the raw image data for processing however. This is on a Samsung Galaxy S5. (Unfortunately, I don't have another Lollipop device handy to test on other hardware).
I got the overall framework (with inspiration from the 'HdrViewFinder' and 'Camera2Basic' samples) working, and the live image is drawn on the screen via a SurfaceTexture and a GLSurfaceView. However, I also need to access the image data (grayscale only is fine, at least for now) for custom image processing. According to the documentation to StreamConfigurationMap.isOutputSupportedFor(class), the recommended surface to obtain image data directly would be ImageReader (correct?).
So I've set up my capture requests as:
mSurfaceTexture.setDefaultBufferSize(640, 480);
mSurface = new Surface(surfaceTexture);
...
mImageReader = ImageReader.newInstance(640, 480, format, 2);
...
List<Surface> surfaces = new ArrayList<Surface>();
surfaces.add(mSurface);
surfaces.add(mImageReader.getSurface());
...
mCameraDevice.createCaptureSession(surfaces, mCameraSessionListener, mCameraHandler);
and in the onImageAvailable callback for the ImageReader, I'm accessing the data as follows:
Image img = reader.acquireLatestImage();
ByteBuffer grayscalePixelsDirectByteBuffer = img.getPlanes()[0].getBuffer();
...but while (as said) the live image preview is working, there's something wrong with the data I get here (or with the way I get it). According to
mCameraInfo.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP).getOutputFormats();
...the following ImageFormats should be supported: NV21, JPEG, YV12, YUV_420_888. I've tried all (plugged in for 'format' above), all support the set resolution according to getOutputSizes(format), but none of them give the desired result:
NV21: ImageReader.newInstance throws java.lang.IllegalArgumentException: NV21 format is not supported
JPEG: This does work, but it doesn't seem to make sense for a real-time application to go through JPEG encode and decode for each frame...
YV12 and YUV_420_888: this is the weirdest result -- I can see get the grayscale image, but it is flipped vertically (yes, flipped, not rotated!) and significantly squished (scaled significantly horizontally, but not vertically).
What am I missing here? What causes the image to be flipped and squished? How can I get a geometrically correct grayscale buffer? Should I be using a different type of surface (instead of ImageReader)?
Any hints appreciated.

I found an explanation (though not necessarily a satisfactory solution): it turns out that the sensor array's aspect ratio is 16:9 (found via mCameraInfo.get(CameraCharacteristics.SENSOR_INFO_ACTIVE_ARRAY_SIZE);).
At least when requesting YV12/YUV_420_888, the streamer appears to not crop the image in any way, but instead scale it non-uniformly, to reach the requested frame size. The images have the correct proportions when requesting a 16:9 format (of which there are only two higher-res ones, unfortunately). Seems a bit odd to me -- it doesn't appear to happen when requesting JPEG, or with the equivalent old camera API functions, or for stills; and I'm not sure what the non-uniformly scaled frames would be good for.
I feel that it's not a really satisfactory solution, because it means that you can't rely on the list of output formats, but instead have to find the sensor size first, find formats with the same aspect ratio, then downsample the image yourself (as needed)...
I don't know if this is the expected outcome here or a 'feature' of the S5. Comments or suggestions still welcome.

I had the same problem and found a solution.
The first part of the problem is setting the size of the surface buffer:
// We configure the size of default buffer to be the size of camera preview we want.
//texture.setDefaultBufferSize(width, height);
This is where the image gets skewed, not in the camera. You should comment it out, and then set an up-scaling of the image when displaying it.
int[] rgba = new int[width*height];
//getImage(rgba);
nativeLoader.convertImage(width, height, data, rgba);
Bitmap bmp = mBitmap;
bmp.setPixels(rgba, 0, width, 0, 0, width, height);
Canvas canvas = mTextureView.lockCanvas();
if (canvas != null) {
//canvas.drawBitmap(bmp, 0, 0, null );//configureTransform(width, height), null);
//canvas.drawBitmap(bmp, configureTransform(width, height), null);
canvas.drawBitmap(bmp, new Rect(0,0,320,240), new Rect(0,0, 640*2,480*2), null );
//canvas.drawBitmap(bmp, (canvas.getWidth() - 320) / 2, (canvas.getHeight() - 240) / 2, null);
mTextureView.unlockCanvasAndPost(canvas);
}
image.close();
You can play around with the values to fine tune the solution for your problem.

How to improve OpenCV face detection performance in android?

I am working on a project in android in which i am using OpenCV to detect faces from all the images which are in the gallery. The process of getting faces from the images is performing in the service. Service continuously working till all the images are processed. It is storing the detected faces in the internal storage and also showing in the grid view if activity is opened.
My code is:
CascadeClassifier mJavaDetector=null;
public void getFaces()
{
for (int i=0 ; i<size ; i++)
{
File file=new File(urls.get(i));
imagepath=urls.get(i);
defaultBitmap=BitmapFactory.decodeFile(file, bitmapFatoryOptions);
mJavaDetector = new CascadeClassifier(FaceDetector.class.getResource("lbpcascade_frontalface").getPath());
Mat image = new Mat (defaultBitmap.getWidth(), defaultBitmap.getHeight(), CvType.CV_8UC1);
Utils.bitmapToMat(defaultBitmap,image);
MatOfRect faceDetections = new MatOfRect();
try
{
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));
}
catch(Exception e)
{
e.printStackTrace();
}
if(faceDetections.toArray().length>0)
{
}
}
}
Everything is fine but it is detection faces very slow. The performance is very slow. When i debug the code then i found the line which is taking time is:
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));
I have checked multiple post for this problem but i didn't get any solution.
Please tell me what should i do to solve this problem.
Any help would be greatly appreciated. Thank you.

You should pay attention to the parameters of detectMultiScale():
scaleFactor – Parameter specifying how much the image size is reduced at each image scale. This parameter is used to create a scale pyramid. It is necessary because the model has a fixed size during training. Without pyramid the only size to detect would be this fix one (which can be read from the XML also). However the face detection can be scale-invariant by using multi-scale representation i.e., detecting large and small faces using the same detection window.
scaleFactor depends on the size of your trained detector, but in fact, you need to set it as high as possible while still getting "good" results, so this should be determined empirically.
Your 1.1 value can be a good value for this purpose. It means, a relative small step is used for resizing (reduce size by 10%), you increase the chance of a matching size with the model for detection is found. If your trained detector has the size 10x10 then you can detect faces with size 11x11, 12x12 and so on. But in fact a factor of 1.1 requires roughly double the # of layers in the pyramid (and 2x computation time) than 1.2 does.
minNeighbors – Parameter specifying how many neighbours each candidate rectangle should have to retain it.
Cascade classifier works with a sliding window approach. By applying this approach, you slide a window through over the image than you resize it and search again until you can not resize it further. In every iteration the true outputs (of cascade classifier) are stored but unfortunately it actually detects many false positives. And to eliminate false positives and get the proper face rectangle out of detections, neighbourhood approach is applied. 3-6 is a good value for it. If the value is too high then you can lose true positives too.
minSize – Regarding to the sliding window approach of minNeighbors, this is the smallest window that cascade can detect. Objects smaller than that are ignored. Usually cv::Size(20, 20) are enough for face detections.
maxSize – Maximum possible object size. Objects bigger than that are ignored.
Finally you can try different classifiers based on different features (such as Haar, LBP, HoG). Usually, LBP classifiers are a few times faster than Haar's, but also less accurate.
And it is also strongly recommended to look over these questions:
Recommended values for OpenCV detectMultiScale() parameters
OpenCV detectMultiScale() minNeighbors parameter

Instead reading images as Bitmap and then converting them to Mat via using Utils.bitmapToMat(defaultBitmap,image) you can directly use Mat image = Highgui.imread(imagepath); You can check here for imread() function.
Also, below line takes too much time because the detector is looking for faces with at least having Size(20, 20) which is pretty small. Check this video for visualization of face detection using OpenCV.
mJavaDetector.detectMultiScale(image,faceDetections,1.1, 10, 0, new Size(20,20), new Size(image.width(), image.height()));