I am doing a project on image processing stuff. I receive a raw h264 video stream in real time and decode it using MediaCodec. I have successfully displayed the decoded video on a TextureView or SurfaceView. Now I want to process each frame, do something to it using OpenCV4Android and show the updated video frame on the screen. I know OpenCV has a sample project that demonstrates how to process video frames from the phone camera, but I wonder how to do it if I have another video source.
Also I have some questions on TextureView:
What does the onSurfaceTextureUpdated() from SurfaceTextureListener do? If I call getBitmap() in this function, then does that mean I get each frame of the video? And what about SurfaceTexture.onFrameAvailableListener?
Is it possible to use a hidden TextureView as an intermediate, extract its frames for processing and render it back to another surface, say, OpenGL ES texture for displaying?
The various examples in Grafika that use Camera as input can also work with input from a video stream. Either way you send the video frame to a Surface.
If you want to work with a frame of video in software, rather than on the GPU, things get more difficult. You either have to receive the frame on a Surface and copy it to a memory buffer, probably performing an RGB-to-YUV color conversion in the process, or you have to get the YUV buffer output from MediaCodec. The latter is tricky because a few different formats are possible, including Qualcomm's proprietary tiled format.
With regard to TextureView:
onSurfaceTextureUpdated() is called whenever TextureView receives a new frame. You can use getBitmap() to get every frame of the video, but you need to pace the video playback to match your filter speed -- TextureView will drop frames if you fall behind.
You could create a "hidden TextureView" by putting other View elements on top of it, but that would be silly. TextureView uses a SurfaceTexture to convert the video frames to OpenGL ES textures, then renders them as part of drawing the View UI. The bitmap data is retrieved with glReadPixels(). You can just use these elements directly. The bigflake ExtractMpegFramesTest demonstrates this.
Related
I need to record video with timestamp on each video frame. I see a example in cts which use InputSurace.java and OutputSurface.java to connect Decoder and Encoder to transcode video files. Is it possible to reuse these two android java class to implement a timestamp video recorder?
I try to use OutputSurface as Camera preview output and use InputSurface as MediaCodec Encoder input but sounds like only record 2 or 3 frames then it hang there forever!
Take your time and explore this link to get an idea how to feed the Camera preview into a video file. Once you are confident about the mechanism, you should be able to feed the MediaCodec input surface with some kind of OpenGL magic to put extra graphics on the top of the Camera's preview. I would recommend to tweak the example code's drawExtra() as a start.
I have seen question Record video from camera parallel to OpenCV processing on android about similar problem but it not answered.
I see two posibilities:
to record video from camera, using standard android tool MediaRecorder and simultaneously process byte[] array with OpenCV in previewCallback.
record video with MediaCodec like in Gragika project https://github.com/google/grafika. While recording use glReadPixels from texture and pas it to OpenCV Mat for processing.
What better or maybe exists another approach?
I have worked it out.
For the first way you mentioned:
Lock the camera before preparing a MediaRecorder.
Relock the camera calling reconnect() immediately after mediaRecorder's start() method.
However, since API level 14 the camera can be automatically relocked after calling start() so actually you don't need to reconnect().
Set preview callback. Use setPreviewCallbackWithBuffer() and set more than one callback buffer if you want to achieve higher frame rate.
But from my experiments, frames from preview callback and frames recorded by MediaRecorder are asynchronous. So it is hard to guarantee frame synchronization if you need to.
For the second way: You have full control of frames.
Thanks to fadden's project, I succeeded in recording while doing image processing by using MediaCodec to encode video frames from preview callback and passing the preview data to native layer via JNI for image processing with OpenCV.
It should be noted that for real-time efficiency you must do the processing in another thread and ensure synchronization with the encoding thread.
currently, I am playing a video on a GLSurfaceView using OpenGL ES 2.0. Now, I am searching for a way to encode this video played on the surface view into a MP4 video using MediaCodec.
Therefore, I found the bigflake example, which seems to solve my issue perfectly (http://bigflake.com/mediacodec/EncodeAndMuxTest.java.txt).
However, it seems that I am too stupid to set the input source right. This example uses mEncoder.createInputSurface() to create the input source, however I have a GLSurfaceView where the video is actually played. So how do I set my own surface as input source for the encoder?
Since you are using GLSurfaceView, you need to insert intercepting code in onDrawFrame(), while allocating surface in onSurfaceCreated().
Input surface can be created as usual after setting up encoder parameters.
Interceptor can be done in a form of copying egl scene into frame buffer via copying shader. And then do swapbuffer to encode frame.
Try look at tutorial for arbitrary elg scene capturing at
https://software.intel.com/en-us/articles/intel-inde-media-pack-for-android-tutorials-video-capturing-for-opengl-applications
I want to use glSurfaceView to render video in GB(API 10), so that i can use glReadPixels call to get the frame data. But setSurface in MediaPlayer requires API 14. Now any way can i get my work done, any Support libraries, or native code?
Ultimately i want to capture frames of video to JPEG images while the video is rendered in fullscreen. The cartoon video is in BGRA format(hope so!)
So, In my application, I am able to show effects(like blur filter, gaussian) to video that comes from Camera using GPUImage library.
Basically, I (library) will take the input from the Camera, get's the raw byte data, converts it into RGBA format from YUV format, then applies effects to this image and displays on the Surface of GLSurfaceView using OpenGL. finally, to the user, it looks like a video with effects applied.
Now I want to record the frames of Surface as a video using MediaCodec API.
but this discussion says that we can not pass a predefined Surface to the MediaCodec.
I have seen some samples at bigflake where he is creating Surface using MediaCodec.createInputSurface() but for me, Surface comes from the GLSurfaceView.
So, how can I record a frames of a Surface as a video?
I will record the audio parallelly, merge that video and audio using FFMPEG and present to the user as a Video with effects applied.
You can see a complete example of this in Grafika.
In particular, the "Show + capture camera" activity records camera output to .mp4. It also demonstrates applying some simple image processing techniques in the GL shader. It uses a GLSurfaceView and a convoluted dance to keep the recording going across orientation changes.
Also possibly of interest, the "Record GL app with FBO" activity records OpenGL ES rendering a couple different ways. It uses plain SurfaceView and is much more straightforward.