I have a working bit of Android Native code which process video frames on the fly via the CameraX API. The basic workflow is this:
User points phone at things
Code takes a single frame and processes it then returns a result. While this happens, all other frames are dropped (but the user doesn't notice - they just see smooth video)
When available, the result is displayed for the user as on screen text
Rinse and repeat
My app will be in React-Native but I need to keep this bit of code on Native for performance reasons.
So I'm wondering if I can use a Native Module as a sort of black box that takes in a camera stream and returns results. If so, can I keep the flow as above? I particularly want to avoid the user having to do anything other than point the camera.
Related
Background
I am building an Optical Character Recognition (OCR) tool that makes sense of photographed Forms.
Arguably the most complicated part of the pipeline is to get the target Document into perspective; basically what is attempted in this Tutorial.
Due to the fact that the data is acquired often in very poor conditions, i.e.:
Uncontrolled Brightness
Covered or removed corners
Background containing more texture than the Target Document
Shades
Overlapped Documents
I have "solved" the Problem using Instance + Semantic Segmentation.
Situation
The images are uploaded by Users via an App that captures images as is. There are versions for the App in both Android and IOS.
Question
Would it be possible to force the App to use the Users' mobile phone Documents mode (if present) prior to acquiring the photo?
The objective is to simplify the pipeline.
In end effect, at a description level, the App would have to do three things:
1 - Activate the Documents mode
2 - Outline the Target Document; if possible even showing the yellow frame.
3 - Upload the processed file to the server. Orientation and extension are not important.
iOS
This isn't a "mode" for the native camera app.
Android
There isn't a way to have the the "documents mode" automatically selected. This isn't available for all Android devices, either. So even if you could, it wouldn't be reliable.
Best bet is following the documentation for Building a camera app, rather than using the native camera if special document scanning is essential. This won't come out of the box on either platform for you.
My app uses the camera stream to perform some live image classification on the incoming images. For debugging purposes, I'd like to use some prerecorded videos instead.
Does someone know how to use a video in the similar way as the camera, meaning that the video should play continuously while the classifier runs in parallel and always takes the most recent frame (dropping intermediary frames as long as the classification is running).
I just need some high level information how to do it and can figure out the coding myself.
It seems to me that I would need to run the classifier on a background thread that never dies, but rather waits for an input Image -> processes the image -> then posts the results back to the UI-thread for displaying them -> then take the most recent image from the video stream and classify it.
In the camera based version of the app, the above mentioned infinite classification loop is triggered by the cameras OnImageAvailableListener and it's OnImageAvailable method.
The question is: What is the best way to imitate this behaviour? Using threads, Handlers, Loopers? Or is there some way to put attach a Listener to the Classifier to notify the UI-thread that the image processing has been completed?
I want to configure front and back both cameras into Android camera2 API, to take pictures and videos from both cameras simultaneously, I have created 2 texture views , when ever I am opening one camera (front or back) my code is working fine but whenever I am trying to open both cameras simultaneously , code is breaking upon creating session, I am getting cameraAccessException :configure stream : method not implimented.
I want to save both front and back camera captured images as one image and both video as one video.
Guys it will be very much helpful if you can put some sample code or some sample link.
i am using one plus 6, i recently downloaded an app "Dual camera fron back Camera", by using this i am able to capture image from front and back both camera on the same time, so if somebody want to suggest for no hardware support, i think it may be valid for other phones but for my case i think i am missing something in coding, till now from google search it looks like there is some problem with session creation for second camera, i debugged my code, during creation of second camera session it fails so if you have any idea about that, please share.
Thanks
Rakesh
The camera API is fine with it, but most Android devices do not have sufficient hardware resources to run both cameras at once, so you will generally get an error trying to open the second camera.
Both image sensors are typically connected to the same image signal processor (ISP), and that ISP can only operate one camera at a time. Some high-end devices have ISPs with multiple processing pipelines which can in theory run more than one camera at a time, but they often require using multiple pipelines to handle advanced functionality or very high resolutions for the main (back) camera.
So on those devices, multiple cameras may be usable at once, but not at maximum resolution or other similar restrictions.
Some manufacturers include multi-camera features in their own camera app, since they know exactly what the limitations are and can write application code to work within them. They may not make multi-camera available to normal apps, due to concerns about performance, thermal limits, or just lack of time to verify more than the exact use case they implement in their own app.
The Android camera API does not currently have a way to query if multiple cameras can be used at once, or if they can be, what the restrictions are. So the only thing you can do is try, and handle the errors in case that isn't feasible.
I wrote a small game. It has a front facing cam preview implemented. I want to be able to record the entire screen to a mp4. How would I do that? Anyone know a nice tutorial for recording the entire screen to mp4 (in code, so not just screenshots.. I want to enable the user to make a recording, while playing the game).
Indeed, as #Michael stated, directly accessing the screenshot and framebuffer mechanisms is not possible. That doesn't mean you're completely out of luck, though, if you want to record gaming sessions. It's just more work.
If you generate a perfect record of the various actions in your game, save that record. You can then build a service (web service, local, whatever) that accepts the record as an input. In effect, it would replay the game. You'd then reconstruct the game in such a manner that you could generate a video.
You could also replay in the Android emulator and create a video from that. Not as easily deployable, but it depends on what you want to do with it.
Apps aren't allowed to grab the screen contents (see this discussion).
[L]etting applications grab the screen's content liberally is a serious security risk, which is why the platform prevents it (taking a snapshot requires either direct physical manipulation from the user, or using a debugging tool).
And as #CommonsWare mentioned in his comment you can't use the getDrawingCache method either to grab a Bitmap of the View if you're using a SurfaceView or one of its children (e.g. VideoView). More details about that here.
I have doing a basic object detection on the camera preview screen in Android (greater than 3.2). For the devices which do not support processing on preview screen, I am buffering the preview screen, processing it and clearing the buffer. This part is working as desired.
What I now want is this app to run in the background while any other app is running in the foreground. I am using android service and am able to run a small test app in the background. However my concern is with the camera preview app.
I don't want to display the preview screen but use the preview screen information for processing. This might be too much to ask, but I wanted to know if this is even possible. I came across this link which shows some hope. Basically I want to process the video (preview) stream without displaying it on the screen. If this is doable, then I can think of putting this app in the background and some other app in the foreground.
Unfortunately I won't be able to share the code, however it is the standard logic of creating a surface view and starting the preview.
I would really appreciate any insight into this.
Check comments here.Basically he opens camera hardware, set preview callback and do startpreview without setting the previewDisplay (this might not work on every device). You can try this from your background service. All this will work if your foreground doesn't access the camera app. Please update this if it works. I am interested to know.