I'm using MediaCodec to decode a H.264 video # 30FPS that I receive from an RTSP live stream, the decoder runs on an android device.
However, I see a latency in the output of the MediaCodec's decoder.
It looks like the decoder waits until it receives about 15 frames before providing the decoded frames, resulting in ~500ms latency in the rendered video.
The latency is not accepted for my project, as the user expects to see the live video immediately when it arrives to his device.
Is there a way to configure the MediaCodec, so it doesn't buffer the incoming frames and outputs the decoded frames as soon as they are ready to be displayed?
Thanks for the help.
If possible, try to change the encoding of the videos.
Related
I have an audio and video stream and would like to generate an mp4. However, all the samples I've seen tend to generate an mp4 when it has the entire stream. What I want to accomplish is to encode the stream to mp4 while receiving the audio/video stream from the web.
It isn't clear whether an mp4 can be created on-the-fly or needs to wait until the final input data has been received. I believe that this has to be possible because when you record a video on your device using the camera, as soon as you stop recording, the video is already available. This means that it was encoding it on-the-fly. So it really shouldn't matter whether the frames come from the camera or from a video stream over the web. Or am I possibly missing something?
Without knowing how mp4 is actually generated, I'm guessing that even if the encoder receives a few seconds of frames and audio, it still can generate that portion of the mp4 and save it to disk. At any point in time, my app should be able to stop the streaming and the encoder just saves the final portion of the stream. What would be nice is that if the app were to crash during encoding, that at least the part that was encoded still gets saved to disk as a valid mp4. Not sure if that is even possible.
Ultimately I would like to do this using the MediaCodec, OpenGL ES and a Surface.
I’m trying to show live preview from one android device to another.
Here is what I did,
Sender : 1.Camera frame(YUV) -> 2.Mediacodec (encode h264 byte[])-> 3.MediaMuxer-> 4.mp4
I'm sending output of media encoder via socket connection.
Receiver : 5.byte[] via socket ->6. Mediacodec (decoder) -> 7.Play.
Till step 5 everything works fine.
However I'm not able to decode the byte[]. What is missing here? I guess, I'm not able to send(don't know how to send) sps and pps properly! also how to test that what I'm sending is correct data?
Thanks.
You are muxing to mp4 (container format) the encoded h264 video. However, on the decoding side, you are not using demuxer (or parser/splitter). Media codec decoder can take elementary video not any container format.
Use mediaextractor to demux in android. (https://developer.android.com/reference/android/media/MediaExtractor.html)
I am wrapping H264 video with RTP for network streaming. In order to do this, I am using Android's MediaCodec configured as an encoder to generate the H264. The RTP code is my own.
When I read the stream (using my local network to the streaming server), I can tell when the keyframe is received because the video sharpens up to what I would expect. However, as I move my hand across the video view, I see a significant amount of pixelation until the next keyframe comes in. My video is 960x720, 30fps and I am sending a keyframe every 2 seconds.
I can pump the raw h264 (not wrapped as RTP) from the MediaCodec encoder via a Datagram socket and play it back using ffplay. There are no such effects when I do this. It has to be something with the RTP encapsulation. I verified that my stop/stop bits of the FU-A fragmentation packets are correct, as is the RTP header marker bit. I'm sort of at a loss for what else could be the problem?
I'm making an app which is using MediaCodec APIs.
The app runs on two phones. The first phone reads the video from the sdcard and then uses the MediaCodec encoder to encode the frames in avc format and then streams the frames to another device. The second device has a MediaCodec decoder running. The decoder decodes the frames and render them on a Surface.
The code is running fine but after sometime when the size of the frames gets more, the first device is sometime not able to stream the video and the encoder stops reporting the following log :
E/OMX-VENC-720p( 212): Poll timedout, pipeline stalled due to client/firmware ETB: 496, EBD: 491, FTB: 492, FBD: 492
So I want to implement frame skipping on the encoder side.
What's the best way to skip the frames and not stream them to the other device. ?
PS. On a separate note if anyone can suggest me of any other way of streaming a video to other device it'll be really nice.
Please try Intel INDE Media Pack with tutorials on https://software.intel.com/en-us/articles/intel-inde-media-pack-for-android-tutorials. It has Camera, File and Game streaming components, which make streaming with help of Wowza and a set of samples demonstrating how to use it as a server and as a client
Background:
I've connected Android's MediaCodec to FFmpeg for muxing a variety of formats not supported by MediaMuxer, including rtmp:// output via a .flv container. Such streaming muxers require longer, unpredictable ownership of MediaCodec's output buffers, as they may perform networking I/O on any packet processing step. For my video stream, I'm using MediaCodec configured for Surface input. To decouple muxing from encoding, I queue MediaCodec's ByteBuffer output buffers to my muxer via a Handler.
All works splendidly if I mux the .flv output to file, rather than rtmp endpoint.
Problem:
When muxing to rtmp://... endpoint I notice my streaming application begins to block on calls to eglSwapBuffers(mEGLDisplay, mEncodingEGLSurface) at dequeueOutputBuffer() once I'm retaining even a few MediaCodec output buffers in my muxing queue as MediaCodec seems to be locked to only 4 output buffers.
Any tricks to avoid copying all encoder output returned by MediaCodec#dequeueOutputBuffers and immediately calling releaseOutputBuffer(...)?
The full source of my project is available on Github. Specifically, see:
AndroidEncoder.java: Abstract Encoder class with shared behavior between Audio and Video encoders: mainly drainEncoder(). Writes data to a Muxer instance.
FFmpegMuxer.java: Implements Muxer
CameraEncoder.java. Sends camera frames to an AndroidEncoder subclass configured for Video encoding.
Systrace
Here's some systrace output streaming 720p # 2Mbps video to Zencoder.
Solved
Copying then releasing the MediaCodec encoder output ByteBuffers as soon as they're available solves the issue without significantly affecting performance. I recycle the ByteBuffer copies in an ArrayDeque<ByteBuffer> for each muxer track, which limits the number of allocations.
Unfortunately this kind of use case is not supported by most Android phones. MediaCodec is just an abstraction of the OMX IL API used by codec vendors on the devices. The vendor codecs require a certain number of input and output buffers for a given configuration.
While theoretically, having one output buffer queued with the encoder should be enough, a lot of the times this is not supported by vendor codecs because it results in lower encoding performance; hence the codec stalls. This is even more prevalent with input buffers.
There is no application control on the number of input/output buffers that are allocated for a MediaCodec instance - Android tries to allocate the minimum number of buffers required, to save memory. Therefore, your only option is to copy the output buffers. While this is not ideal, the encoded buffers tend to be reasonably small.