Develop Reference

Does mirroring the (front) camera affect MLKit with CameraX?

You might consider this question a sequel to this other one I've recently asked. I have a custom device with a front camera that is mirrored by default and I would like it to always behave like it was flipped horizontally. Note that the end goal of my app is to perform face, barcode and facemask detection (the last one with a custom .tflite model), and I'm trying to understand if it's possible with the current state of CameraX and the quirks of the device I'm using.
For the preview, I can force PreviewView to use a TextureView by calling PreviewView#implementationMode = PreviewView.ImplementationMode.COMPATIBLE so I can just call PreviewView#setScaleX(-1) and it gets displayed like I want.
For taking pictures, I haven't tried yet but I know I can pass setReversedHorizontal(true) to the ImageCapture.OutputFileOptions used in the ImageCapture.Builder, so the image should be saved in reverse.
For image analysis I really don't know. If the input image is taken directly from the preview stream, then it should still be in its default state because PreviewView#setScaleX(-1) only flips the View on screen, right? So in my Analyzer's analyze() I would need to convert the ImageProxy#image to bitmap, then flip it and pass the result to InputImage.fromBitmap().
Is this everything I need? Is there a better way to do this?

empty view when displaying WebView (WebGL an Video) inside 3D app

I inserted a webview in my 3D Game. I can manage to retrieve URL and display it on a canvas and texture it in my 3D rendering.
Everything work fine except from the Video the WebGL and Threeje rendering.
For this 3 case I cannot see anything except the 2 canvas from threeJS (the frame canvas information) and ( the check box option)
In all these 3 case the view where WebGL or video are empty should be rendered are empty.
For the video I found out that I can see the image only when I touch the seekbar under the play button and audio button.
So it looks like the rendering view seems to be hidden or not visible or something else.
So I tried to get the view rendered by WebView to check the status. But WebView only got One View and it is not possible to get the elements view inside to check it.
So I am not sure if the problem come from the view status or from conflict with the 3D environment.
So I would like to know if someone have an idea about the problem. Or if someone could tell me how to retrieve the detailed view from webview, if it is possible.
NEW INFORMATION
I think that the problem could come from:
running WEBGL(opengl 1.4) from WebView inside GvrActivity(OpenGL es 2)
may cause a conflict when rendering both Android OpenGL at same time.
Concerning the media and audio. I am also running voice recognition
(Context.AUDIO_SERVICE) and mediaplayer to watch loader video.
LAST TESTING:
So if I run WebView in another activity everything is fine. The problem is that I would like to retrieve an access to this activity to get the Webview VIEW displayed in the main activity layout.
Is that possible?
LAST TESTING:
Changing Context Activity starting WebView does not resolve the problem.
So I found out that if I attach the WebView to the getWindow().addContentView
there is no problem But is if I add the view to MyGLSurfaceView extends GvrView
i got the problem. An I think it is because I already use the Media player to render video To 3D mesh and OpenGL to draw The 3D scene. But I am not sure of that.
LAST TESTING : (je rame (french expression))
I try everything. I think I did. And I think that after 3 weeks I start to be out of resources.
Concerning the audio from the webview I am sure a get in conflict with the voice recognition mAudioManager = (AudioManager) context.getSystemService(Context.AUDIO_SERVICE); Which I think is use by Webview because I can use it displaying video inside canvas in 3D OpenglES.
Concerning the video from Webview in need to touch The progression bar to see something. But as Webview is a black box. No way to get the progress bar view and act on it.
Concerning WebGL and threeJS. I can display web text and image but nothing related to opengl, white display but not transparent, because a set transparency to check. Can only be displayed outside of the OPenGL environment and outside of Surface and surfacetexture.
Concerning Webview: Only one view output. So all the view rendering seems to be done internally using tag parser for position and canvas construction, so no view may be used. But is I consider that the ourput view of Webview is a complete bitmap of all the tag canvas (that I my one interpretation it is easy to use tag tree for rendering fast access and easy structured design information structure). But I am wondering in this case why WebGL and ThreeJs cannot be copied using surfacetexture.updateTexImage() if webview output is a bitmap canvas.
Because everything are 2D canvas when they are displayed inside view.
So 3 week trying to find answer. I hope that someone will find it.
Because I was planing to do an ART gallery in VR where anyone could watch video or 3D ART. For the video and 360 video I could make it but not for WebGL and ThreeJS.
3D inside 3D is the top in ART technology. Imagine that you could go in any 3D shop or Accessing Web URL to any Video of artiste or whatever. The last thing is possible but not the WEGgl and TrheeJS. Watching YouTube 360 VR is easy a can make it, it works very well with voice command, just Hack to get the download address, easy. Why to block when the best is to be open and let the imagination create the tomorrow tools?
So I give up and go back to my OPenCL recognition application which is nearly finished. Form recognition could be great new tools.
By the way. Do not hesitate to ask me for my APK if you want to check by yourself. Stack Overflow are allowed to contact me as they got my email address.
LAST TEST (07/07/2020)
So i think I understood the problem concerning the non accessibility to the Video and webgL display. The first intention concerning Video is surely the fact that they wanted to avoid the ability to copy the video and the problem of optimization of the rendering. Concerning WebGL their is no need to avoid the copying. But it need to be optimized to avoid to redraw the all webview picture when only one part is modified. So it look like the rendering of video and webGL is done in another thread and that the rendering is independent of the other HTML TAG.
Some said that because it is done in other thread it is not possible to access it. Which is false, because the canvas where the rendering is done must be accessible if we want it to be accessible. It is just a bitmap for the video and a frame buffer(viewport) for webGL. I agree that is done by GPU card and not by software but the resulting canvas(rectangle) could be accessible because at the end it is displayed. The problem is that it is not part anymore of the WebView final view, they just send the position where the display must be done to the GPU as viewport coordinate.
So when I try to SurfaceTexture.updateTexImage() I just get some part of the web page, no video and no webgl.
Here is an example to anderstand:
if you load this URL "https://webglfundamentals.org/webgl/webgl-2d-triangle-with-position-for-color.html" and look at the code in you browser.
I can see and manipulate the slider without problem but the GL result is not visible. Because the canvas is an openGL context and it is process by the GPU and directly copy to the final frame buffer at the right place but it is not part of the Webview final view, which is unique. For the video if a can get the slider i can have some picture but i have to get it and check is modification, hard work and i am not really interested by video, found other way to watch it in OpenGL .
So I think that Android could make an effort to try to give the access to the Canvas for webgl rendering. And find a way to give an access to it inside WebView. 3D inside 3D is already 4 dimension. Could be good. But not available at the moment. But it will.
I will look at it, just by curiosity. But no chance I think.

here is the only answer i have found :
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/3wrULcul8lw
today (24/09/2020) i read this :
DRM-protected video can be presented only on an overlay plane. Video players that support protected content must be implemented with SurfaceView. Software running on unprotected hardware can't read or write the buffer; hardware-protected paths must appear on the Hardware Composer overlay (that is, protected videos disappear from the display if Hardware Composer switches to OpenGL ES composition).
30/09/2020 :
i am able to see and manipulate WebGl display using Surface.lockHardwareCanvas but a looze the focus during manipulation and sometimes it does not work, little probleme ;)).
But using lockHardwareCanvas when a display youtube web page with video or without, everything is frezzing, cannot udtate the view neither scroll it.
Mais ça avance ;))

Here is my solution.
WebGL and Jthree can be display using Surface.lockHardwareCanvas
the problem is using Surface.lockHardwareCanvas with htlm other than accelerated View. In this case there is no frame to listen so no refreshing of the view. so must be done in other way.
concerning the OpenGL,WebGL and Jthree. It depends on the application somme are not weel displayed for many reason (no GL_TEXTURE_EXTERNAL_OES, canvas not facteur of 2, and bad view size). And the problem is that there is no new onframeavailable so need to find other way to refresh the view.
It took quite a long time to anderstand but it is working even in GVRActivity ;))
Regards.

How to render in Unity without the editor/scene/player being visible?

In my Android-Unity app I have some data such as text, text size, text coordinates etc that I will use to create the output and store it as a screenshot I will later use in my app.
But I need to make this happen when the user isn't seeing the Unity player/scene. So my question is that is it possible to render the contents and then take a screenshot of the same without the user seeing the Unity editor/player/scene whatever one may call it? Is there a way to do it in the background? Thanks!

I'm not sure exactly what you're trying to accomplish but I can clarify the use of a culling mask here. Here is what the scene looks like. I have embedded the view from Camera 2 into the bottom left of the game. You can see that Camera 2 is not displaying the floor. This is because I set it's culling to only respond to objects tagged with a custom layer here. Where as the Main Camera's culling mask is set to everything. Now, anything I tag with the "Custom" layer will be visible in the second camera and everything else would not be. I'm assuming what you want to do from here is to tag the things you want visible when you take a screenshot with a specific layer, then set the Culling Mask of your "Screenshot Camera" to that layer and take a screenshot with that Camera. That way you can specify what Objects/UI want visible in that camera.

Using OCR mobile vision to anchor image to detected text

I am using the Text Recognition (mobile vision/ML) by Google to detect text on Camera feed. Once I detect text and ensure it is equal to "HERE WE GO", I draw a heart shape beside the detected text using the passed boundries.
The problem I am facing that the shape is jumping and lagging behind. I want it more like Anchored to the detected text. Is there something I can do to improve that?
I heard about ArCore library but it seems it is based on existing images to determine the anchor however in my case it can be any text that matches "HERE WE GO".
Any suggestions ?

I believe you are trying to overlay text on the camera preview in realtime. There will be small delay between the camera input and detection. Since the API is async by the time the output returns you would be showing another frame.
To alleviate that you can either make the processing part sync with using some lock/mutex or overlay another image that only refreshes after the processing is done.
We have some examples here: https://github.com/firebase/quickstart-android/tree/master/mlkit
and also I fixed a similar problem on iOS by using DispatchGroup https://github.com/googlecodelabs/mlkit-ios/blob/master/translate/TranslateDemo/CameraViewController.swift#L245

Option 1: Refer tensor flow android sample here
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/android
especially these classes:
1. Object tracker: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/tracking/ObjectTracker.java
2.Overlay
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/OverlayView.java
3.Camera Activity and Camera Fragment https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/android/src/org/tensorflow/demo/CameraActivity.java
Option 2: A sample code can be found in below code lab. They are doing something similar for barcode.
https://codelabs.developers.google.com/codelabs/barcodes/index.html?index=..%2F..index#0

Android blur surfaceview used in camera

I have a camera preview in my android app. As you all probably know it is implemented by a surfaceview in android.
In my photo app, which allows users to take pictures, I want to blur the camera preview (the surface view) if the user has not logged in yet if the user is logged in, I will display the normal preview (without blur)
Blur as something like
But there seems to be no way to do it
Couple things come to mind but I am not sure how to achieve it
use a blur overlay and place it on top of the surface view, but how do u create such blur overlay?
another approach is to change the attribute of the window, but I can't do it to surfaceview,
getWindow().addFlags(WindowManager.LayoutParams.FLAG_BLUR_BEHIND);
So can we create a window overlay the surfaceview and set the flag like that? I don't think so
can someone tell me how to blur a camera preview, which is a surface view
note: I am trying to blur an area which is the output from camera preview, so it is not like I am blurring a static image, the blur area will change depending on where you point your phone camera

The best way to do this is to take a screen shot of the control, apply a blur to it, and then show that image over the top of the original control. This is how the yahoo weather app does it and its how google suggest you do things like this.
Render script does bluring fast. I've also got some code, but it's not currently at hand right now.
These might help:
http://blog.neteril.org/blog/2013/08/12/blurring-images-on-android
http://docs.xamarin.com/recipes/android/other_ux/drawing/blur_an_image_with_renderscript/
I've also read that there are methods built into Android that do this, but the API isn't public so we cannot use it... which sucks.

Although your 2nd option is losing support in later versions of the Android OS, it may be a good option. I would think to bring a window IN FRONT of your surfaceview, and use the blur_behind property of your now new IN FRONT window.
Android API documentation
"
This constant was deprecated in API level 14.
Blurring is no longer supported.
Window flag: blur everything behind this window.
Constant Value: 4 (0x00000004)"
I do have to admit, Im not 100% sure this would work, but its definitely worth a try.

You have to change the camera parameters; in this case with Camera.Parameters.setPictureSize().
The basic workflow here is :
Camera.Parameters cp = mCamera.getParameters(); // get the current params
cp.set...(); // change something
mCamera.setParameters(cp); // write the params back
Make sure that every resolution that you set via this function is supported. You can get a list of the resolutions that are supported on a device via Camera.Parameters.getSupportedPictureSizes(). and check this Camera documentation.