I took the code from grafika for Double Decoder and changed it so that one textureview is outputting from mediaplayer and the other textureview outputting the camera preview. It would be working fine except that in the textureview with the media player output will get frames of the camera preview interlaced. It's really funky behavior and I am not doing anything dynamic to reconfigure the output viewport.
Both Textureviews & SurfaceTextures are separated and are independently assigned to their respective output -> mediaplayer output and camera preview output:
mediaPlayer.setSurface(surface);
camera.setPreviewTexture(surfaceTexture);
Running the mediaPlayer code and camera code on a separate threads does not help. It seems like the SurfaceTexture frame buffers are being somehow being shared and frames are being dropped or interfering with each other? Could this be be a performance issue? Even when the original example of the code was to be able to decode movies in parallel?
Related
I have a camera app that analyzes video when recording (detect camera movement).
I use camera2 API and I get frames (from camera) when recording video (with MediaRecorder). But the video file should be analyzed once more on a server.
The problem is that I analyze non-compressed frames on phone. But server analyzes video frames after compression and decompression. And those frames are definitelly different.
My MotionDetector takes frames from camera, calculates shift between current frame and previous frame using Phase Correlate.
After video file uploaded to the server, it is analyzed with the same MotionDetector.
If I compare shift-between-frames on uncomperred frames to shift-between-frames on compressed frames -- they are obviously different.
Is there a way to get frames after compression?
Currently data flow looks like:
1. Frames from camera
2.1 frames from camera displayed on a screen
2.2 frames from camera are fed to MediaRecorder.
2.3 frames from camera are fed to my MotionDetector
3. MediaRecorder encodes camera frames into H264 format.
4 MediaRecorder output stored to file
I need the data workflow to look like:
1. Frames from camera
2.1 frames from camera displayed on a screen
2.2 frames from camera are fed into MediaRecorder.
3. MediaRecorder encodes camera frames into H264 format.
4.1 MediaRecorder output stored to file
4.2 MediaRecorder output goes to MediaCodec
5. MediaCodec decodes frames.
6. frames from MediaCoded are fed to MotionDetector.
Is there a way to do that?
If you stop using MediaRecorder, and just use MediaCodec directly for encoding video, you should be able to inspect the encoded video buffers before writing them to a MediaMuxer for storage to disk.
That's a lot more complicated than just using MediaRecorder directly, but a lot more flexible for cases like this.
I need to record video with timestamp on each video frame. I see a example in cts which use InputSurace.java and OutputSurface.java to connect Decoder and Encoder to transcode video files. Is it possible to reuse these two android java class to implement a timestamp video recorder?
I try to use OutputSurface as Camera preview output and use InputSurface as MediaCodec Encoder input but sounds like only record 2 or 3 frames then it hang there forever!
Take your time and explore this link to get an idea how to feed the Camera preview into a video file. Once you are confident about the mechanism, you should be able to feed the MediaCodec input surface with some kind of OpenGL magic to put extra graphics on the top of the Camera's preview. I would recommend to tweak the example code's drawExtra() as a start.
I am doing a project on image processing stuff. I receive a raw h264 video stream in real time and decode it using MediaCodec. I have successfully displayed the decoded video on a TextureView or SurfaceView. Now I want to process each frame, do something to it using OpenCV4Android and show the updated video frame on the screen. I know OpenCV has a sample project that demonstrates how to process video frames from the phone camera, but I wonder how to do it if I have another video source.
Also I have some questions on TextureView:
What does the onSurfaceTextureUpdated() from SurfaceTextureListener do? If I call getBitmap() in this function, then does that mean I get each frame of the video? And what about SurfaceTexture.onFrameAvailableListener?
Is it possible to use a hidden TextureView as an intermediate, extract its frames for processing and render it back to another surface, say, OpenGL ES texture for displaying?
The various examples in Grafika that use Camera as input can also work with input from a video stream. Either way you send the video frame to a Surface.
If you want to work with a frame of video in software, rather than on the GPU, things get more difficult. You either have to receive the frame on a Surface and copy it to a memory buffer, probably performing an RGB-to-YUV color conversion in the process, or you have to get the YUV buffer output from MediaCodec. The latter is tricky because a few different formats are possible, including Qualcomm's proprietary tiled format.
With regard to TextureView:
onSurfaceTextureUpdated() is called whenever TextureView receives a new frame. You can use getBitmap() to get every frame of the video, but you need to pace the video playback to match your filter speed -- TextureView will drop frames if you fall behind.
You could create a "hidden TextureView" by putting other View elements on top of it, but that would be silly. TextureView uses a SurfaceTexture to convert the video frames to OpenGL ES textures, then renders them as part of drawing the View UI. The bitmap data is retrieved with glReadPixels(). You can just use these elements directly. The bigflake ExtractMpegFramesTest demonstrates this.
I am trying to re-encode existing video frames to create a video which will have all frames as keyframes(I-frames). The current grafika project has a CameraCaptureActivity which in turn uses VideoEncoderCore to encode videos. I have modified this to record a video from camera so that i'll have all frames as Keyframes but this module has lot of moving parts and that is making it hard to uncouple it from renderer and camera to have a smooth pipeline of frames going into encoded video with all supplied frames as keyframes. Any Ideas?
So, In my application, I am able to show effects(like blur filter, gaussian) to video that comes from Camera using GPUImage library.
Basically, I (library) will take the input from the Camera, get's the raw byte data, converts it into RGBA format from YUV format, then applies effects to this image and displays on the Surface of GLSurfaceView using OpenGL. finally, to the user, it looks like a video with effects applied.
Now I want to record the frames of Surface as a video using MediaCodec API.
but this discussion says that we can not pass a predefined Surface to the MediaCodec.
I have seen some samples at bigflake where he is creating Surface using MediaCodec.createInputSurface() but for me, Surface comes from the GLSurfaceView.
So, how can I record a frames of a Surface as a video?
I will record the audio parallelly, merge that video and audio using FFMPEG and present to the user as a Video with effects applied.
You can see a complete example of this in Grafika.
In particular, the "Show + capture camera" activity records camera output to .mp4. It also demonstrates applying some simple image processing techniques in the GL shader. It uses a GLSurfaceView and a convoluted dance to keep the recording going across orientation changes.
Also possibly of interest, the "Record GL app with FBO" activity records OpenGL ES rendering a couple different ways. It uses plain SurfaceView and is much more straightforward.