Background:
I've connected Android's MediaCodec to FFmpeg for muxing a variety of formats not supported by MediaMuxer, including rtmp:// output via a .flv container. Such streaming muxers require longer, unpredictable ownership of MediaCodec's output buffers, as they may perform networking I/O on any packet processing step. For my video stream, I'm using MediaCodec configured for Surface input. To decouple muxing from encoding, I queue MediaCodec's ByteBuffer output buffers to my muxer via a Handler.
All works splendidly if I mux the .flv output to file, rather than rtmp endpoint.
Problem:
When muxing to rtmp://... endpoint I notice my streaming application begins to block on calls to eglSwapBuffers(mEGLDisplay, mEncodingEGLSurface) at dequeueOutputBuffer() once I'm retaining even a few MediaCodec output buffers in my muxing queue as MediaCodec seems to be locked to only 4 output buffers.
Any tricks to avoid copying all encoder output returned by MediaCodec#dequeueOutputBuffers and immediately calling releaseOutputBuffer(...)?
The full source of my project is available on Github. Specifically, see:
AndroidEncoder.java: Abstract Encoder class with shared behavior between Audio and Video encoders: mainly drainEncoder(). Writes data to a Muxer instance.
FFmpegMuxer.java: Implements Muxer
CameraEncoder.java. Sends camera frames to an AndroidEncoder subclass configured for Video encoding.
Systrace
Here's some systrace output streaming 720p # 2Mbps video to Zencoder.
Solved
Copying then releasing the MediaCodec encoder output ByteBuffers as soon as they're available solves the issue without significantly affecting performance. I recycle the ByteBuffer copies in an ArrayDeque<ByteBuffer> for each muxer track, which limits the number of allocations.
Unfortunately this kind of use case is not supported by most Android phones. MediaCodec is just an abstraction of the OMX IL API used by codec vendors on the devices. The vendor codecs require a certain number of input and output buffers for a given configuration.
While theoretically, having one output buffer queued with the encoder should be enough, a lot of the times this is not supported by vendor codecs because it results in lower encoding performance; hence the codec stalls. This is even more prevalent with input buffers.
There is no application control on the number of input/output buffers that are allocated for a MediaCodec instance - Android tries to allocate the minimum number of buffers required, to save memory. Therefore, your only option is to copy the output buffers. While this is not ideal, the encoded buffers tend to be reasonably small.
Related
I'm using MediaCodec to decode a H.264 video # 30FPS that I receive from an RTSP live stream, the decoder runs on an android device.
However, I see a latency in the output of the MediaCodec's decoder.
It looks like the decoder waits until it receives about 15 frames before providing the decoded frames, resulting in ~500ms latency in the rendered video.
The latency is not accepted for my project, as the user expects to see the live video immediately when it arrives to his device.
Is there a way to configure the MediaCodec, so it doesn't buffer the incoming frames and outputs the decoded frames as soon as they are ready to be displayed?
Thanks for the help.
If possible, try to change the encoding of the videos.
I am using mediacodec to decodec a h264 stream on samsung S6, android 5.1.1, found the input buffer to mediacodec must start with "0001"(and don't need to set pps, sps), or the ACodec will report error.
I also tried to use mediaextractor to play a mp4 file, it works fine, but the buffer to mediacodec is not start with "0001".
I don't know why decodec a h264 stream has such limitation, currently i need to analyze the stream from socket, and cut the data into small packages(each package start with 0001) and then give them to mediacodec, but it is inefficient.
MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 1024, 1024);
Some specific decoders may also be able to decode H264 NAL units in the "mp4" format (with a different kind of startcode), but that's not guaranteed across all devices.
It may be that Samsung's version of MediaExtractor returns it in this format, if they know that their own decoder can handle it. There is at least earlier precedent that Samsung did the same, nonstandard thing with timestamps with their version of MediaExtractor, see e.g. https://code.google.com/p/android/issues/detail?id=74356.
(Having MediaExtractor return data that only the current device's decoder can handle is wrong IMO, though, since one may want to use MediaExtractor to read a file but send the compressed data over the network to another device for decoding, and in these cases, returning data in a nonstandard format is wrong.)
As fadden wrote, MediaCodec operates on full NAL units though, so you need to provide data in this format (even if you think it feels inefficient). If you receive data over a socket in a format where this information (about frame boundaries) isn't easily available, then that's an issue with your protocol format (e.g., implementing RTP reception is not easy!), not with MediaCodec itself - it's a quite common limitation to need to have full frames before decoding, instead of being able to feed random chunks until you have a full frame. This shouldn't be inefficient unless your own implementation of it is inefficient.
In general android will expect nal units for each input. For some devices i have found that setting the csd-0/1 on the media-format for h264 to not work consistently. But if you feed each of the parameters sets as input buffers the media-codec will pick it up as a format-change.
int outputBufferIndex = NativeDecoder.DequeueOutputBuffer (info, 1000);
if (outputBufferIndex == (int)MediaCodec.InfoOutputFormatChanged) {
Console.WriteLine ("Format changed: {0}", NativeDecoder.OutputFormat);
} else if (outputBufferIndex >= 0) {
CodecOutputBufferAvailable (NativeDecoder, outputBufferIndex, info);
}
Also note it is mandatory for Nexus and some other samsung devices to set:
formatDescription.SetInteger(MediaFormat.KeyWidth, SelectedPalette.Value.Width);
formatDescription.SetInteger(MediaFormat.KeyHeight, SelectedPalette.Value.Height);
formatDescription.SetInteger(MediaFormat.KeyMaxInputSize, SelectedPalette.Value.Width * SelectedPalette.Value.Height);
I am lucky in my situation i can query these resolutions. But you can parse the resolution manually from SPS and PPS nal units.
// NOTE i am using Xamarin here. But the calls and things are pretty much the same. I am fairly certain there are bugs in the iOS VideoToolbox Xamarin Wrapper so yeah.. Keep that in mind if your ever considering Xamarin for video decoding. Its great for everything but anything thats slightly more custom or low-level.
I'm decoding a H.264 stream on Android 4.2 using Mediacodec. Unfortunately, the decoder always buffers 6-10 frames, which lead to annoying latency, and Android does not provide any API to adjust buffer size. So my question is, how to modify the Android source code (or the OMX driver) in order to reduce the buffer size for realtime video streaming?
Generally speaking, you don't. The number of buffers in the queue is determined by the codec. Different devices, and different codecs on the same device, can behave differently.
Unless you're using the software AVC codec, the codec implementation is provided as a binary by the hardware OEM, so there's no way to modify it (short of hex-editing).
Is there a reasonable way to play a memory buffer containing AAC (in an MP4 container) in Android? I cannot write this buffer to disk... it must be played from memory.
Seems that my only option is to decode to PCM manually and use AudioTrack since Android doesn't expose any API that plays non-PCM memory buffers. OpenSL ES has support for memory sources, but Android doesn't implement this.
The only way I can find that does this is (as indicated in my question) to manually decode the AAC. There are a few packages for Android that do this, perhaps ffmpeg is the most well-tested and well-known. ffmpeg can be distilled down to just an AAC decoder so the memory needs are quite small (although it's frustrating to introduce a decoder to a device that already has one natively).
Once AAC is decoded into PCM, it can be played in Java (AudioTrack), or natively (OpenSL ES). The playback latency is notably lower for the native option, if such matters to you.
I have a project where I have been asked to display a video stream in android, the stream is raw H.264 and I am connecting to a server and will receive a byte stream from the server.
Basically I'm wondering is there a way to send raw bytes to a decoder in android and display it on a surface?
I have been successful in decoding H264 wrapped in an mp4 container using the new MediaCodec and MediaExtractor API in android 4.1, unfortunately I have not found a way to decode a raw H264 file or stream using these API's.
I understand that one way is to compile and use FFmpeg but I'd rather use a built in method that can use HW acceleration. I also understand RTSP streaming is supported in android but this is not an option. Android version is not an issue.
I can't provide any code for this unfortunately, but I'll do my best to explain it based on how I got it to work.
So here is my overview of how I got raw H.264 encoded video to work using the MediaCodec class.
Using the link above there is an example of getting the decoder setup and how to use it, you will need to set it up for decoding H264 AVC.
The format of H.264 is that it’s made up of NAL Units, each starting with a start prefix of three bytes with the values 0x00, 0x00, 0x01 and each unit has a different type depending on the value of the 4th byte right after these 3 starting bytes. One NAL Unit IS NOT one frame in the video, each frame is made up of a number of NAL Units.
Basically I wrote a method that finds each individual unit and passes it to the decoder (one NAL Unit being the starting prefix and any bytes there after up until the next starting prefix).
Now if you have the decoder setup for decoding H.264 AVC and have an InputBuffer from the decoder then you are ready to go. You need to fill this InputBuffer with a NAL Unit and pass it back to the decoder and continue doing this for the length of the stream.
But, to make this work I had to pass the decoder a SPS (Sequence Parameter Set) NAL Unit first. This unit has a byte value of 0x67 after the starting prefix (the 4th byte), on some devices the decoder would crash unless it received this Unit first.
Basically until you find this unit, ignore all other NAL Units and keep parsing the stream until you get this unit, then you can pass all other units to the decoder.
Some devices didn't need the SPS first and some did, but you are better of passing it in first.
Now if you had a surface that you passed to the decoder when you configured it then once it gets enough NAL units for a frame it should display it on the surface.
You can download the raw H.264 from the server, then offer it via a local HTTP server running on the phone and then let VLC for Android do playback from that HTTP server. You should use VLC's http/h264:// scheme to force the demuxer to raw H.264 (if you don't force the demuxer VLC may not be able to recognize the stream, even when the MIME type returned by the HTTP server is set correctly). See
https://github.com/rauljim/tgs-android/blob/integrate_record/src/com/tudelft/triblerdroid/first/VideoPlayerActivity.java#L211
for an example on how to create an Intent that will launch VLC.
Note: raw H.264 apparently has no timing info, so VLC will play as fast as possible.
First embedding it in MPEGTS will be better. Haven't found a Android lib that will do that yet.
Here are the resources I've found helpful in a similar project:
This video has been super insightful in understanding how MediaCodec handles raw h.264 streams on a high level.
This thread goes into a bit more detail as to handling the SPS/PPS NALUs specifically. As was mentioned above, you need to separate individual NAL Units using the start prefix, and then hand the remaining data to the MediaCodec.
This repo (libstreaming) is a great example of decoding an H264 stream in Android using RTSP/RTP for transmission.