this is regarding Android's Camera2 APIs. Since capture result and output frame are produced asynchronously, one could get capture result much before the actual frame. Is there a good way to associate produced frame with the corresponding capture result ?
Assuming you are talking about a frame that is sent to an ImageReader or SurfaceTexture upon capture (as in the ubiquitous camera2basic example), the trick is to compare unique timestamps identifying the images.
Save the TotalCaptureResult somewhere accessible when it is available in your CameraCaptureSession.CaptureCallback's onCaptureComplete(...) call.
Then, when the actual image is available via your ImageReader.OnAvailableListener or SurfaceTexture.OnFrameAvailableListener, get the image's timestamp:
Long imageTimestamp = Long.valueOf(reader.acquireNextImage().getTimestamp()); or
Long imageTimestamp = Long.valueOf(surfaceTexture.getTimestamp()), respectively.
Compare timestamps with: imageTimestamp.equals(totalCaptureResult.get(CaptureResult.SENSOR_TIMESTAMP));
Notes:
The timestamp may not be an actual true system timestamp for your device, but it is guaranteed to be unique and monotonically increasing, so it works as an ID.
If you are sending the image to a SurfaceHolder or something else instead, you're out of luck as only the pixel information gets sent, not the timestamp present in the Image object. I'm not sure about the other places you can send a frame, e.g., MediaRecorder or Allocation, but I think not.
You probably need to add each new TotalCaptureResult to a growing set as they are generated, and then compare an incoming image's timestamp against all of these, because of the asynchronous nature you noted. I'll let you figure out how to do that as you see fit.
I had to solve a similar situation (sync frames across surfaces); Sumner's solution (.getTimestamp() of the respective received Image object) did the trick for me for SurfaceTexture and ImageReader.
Just a quick note on other surfaces (which, as pointed out, don't give you an Image object): at least for MediaCodec, the BufferInfo object received by the onOutputBufferAvailable callback has a presentationTimeUs, which is "derived from the presentation timestamp passed in with the corresponding input buffer" and, at least for me, appears to match the timestamps from other surfaces. (Note the different unit though.)
Related
API level 21 introduced camera2, with it setRepeatingRequest and setRepeatingBurst. I have read the doc here, but still cannot catch the difference between the two. Any idea?
Well, you'll notice that the constructors for these two methods are slightly different. setRepeatingBurst's first argument is List<CaptureRequest>, and setRepeatingRequests's is just a CaptureRequest.
According to the docs,
setRepeatingBurst
With this method, the camera device will continually capture images, cycling through the settings in the provided list of CaptureRequests, at the maximum rate possible.
setRepeatingRequest
With this method, the camera device will continually capture images using the settings in the provided CaptureRequest, at the maximum rate possible.
So, setRepeatingBurst can be used to capture images with a list of different settings.
That's my best understanding, hope it helps!
Think of setRepeatingRequest as ONE CaptureRequest with one set of settings to continually capture images.
Where as in setRepeatingBurst there is a list CaptureRequest and each "CaptureRequest" has its own setting to continually capture images.
Conclusion: setRepeatingBurst call is like making multiple setRepeatingRequest calls in one call.
I am getting the TotalCaptureResults object from the camera, using the Camera2 API in Android. I am using a preview, not a single image. Is there a way to get bytes[] from TotalCaptureResults?
Thank you.
Short answer: no.
All CaptureResults objects contain only metadata about a frame capture, no actual pixel information. The associated pixel data are sent to wherever you designated as the target Surface in your CaptureRequest.Builder. So you need to check with whatever Surface you set up, such as an ImageReader which will give you access to an Image output from the camera, which will give you access to the bytes[].
I have an infinite loop in my Render Thread. I tried measuring assuming that every call to eglSwapBuffers draws a new frame, but that is giving me results like 200 fps, which is not possible, right? The refresh rate cannot exceed 60?
Now I am doing the same thing but also using surfaceTexture.getTimeStamp() of the surfaceTexture of the SurfaceView. I consider a frame as having been drawn only if the timestamp returned in the previous iteration is not the same as in the current. Is the an acceptable way to measure? This is showing 50-55fps when I do no drawing. ie the loop has only eglSwapBuffers() and the getTimeStamp calls.
The surfaceTexture.getTimeStamp() seems to be giving the correct result. I tested it by adding up all the differences between results returned by consecutive getTimeStamp() calls and it is equal to the total time the code ran for. This indicates that no frames are being left unconsidered etc.
Another solution I found is this Android app. I do not know how it works but it is giving approximately the same results as the above method.
If I use 'setPreviewDisplay' in the Camera, like the api below
public final void setPreviewDisplay (SurfaceHolder holder)
What kind of format will be passed to SurfaceHolder for display? Is it NY21 or YV12?
Thank you.
The exact format is platform-dependent, since the image frames sent to a Surface are not accessible to applications.
In fact, it may not be any of the formats listed in android.graphics.ImageFormat, since for efficiency, an entirely device-specific format may be in use, with different padding and stride requirements.
Do you have some use case in mind where knowing the format is important? Since the data is not accessible, I'm curious as to what that is.
I'd like to use MediaCodec to encode the data coming from the camera (reason: it's more low-level so hopefully faster than using MediaRecorder). Using Camera.PreviewCallBack, I capture the data from the camera into a byte-buffer, in order to pass it on to a MediaCodec object.
To do this, I need to fill in a MediaFormat-object, which would be fairly easy if I knew the MIME-code of the data coming from the camera. I can pick this format using setPreviewFormat() choosing one of the constants declared in te ImageFormat-class.
Hence my question: given the different options provided by the ImageFormat-class to set the camera preview-format, what are the corresponding MIME-type codes?
Thanks a lot in advance.
See example at https://gist.github.com/3990442. You should set MIME type of what you want to get out of encoder, i.e. "video/avc".