I made the observation that for devices in the same price range, Snapdragon-based decoders can have a much higher decoding latency than Exynos-based decoders. This is most noticable when decoding h264 streams with the value "pic_order_cnt_type" inside the SPS set to 0 instead of 2.
I am wondering if you have also observed this behaviour and if there is any fix for that ( I have already opened an issue here but no response so far)
Some technical details:
I have built a simple example app that uses AMediaCodec to decode h264 streams. It uploads the "decoding latency", as a test result into a Firestore database. code
Here is a comparison of the decoding latency for different h264 streams on a Pixel 4 (using snapdragon decoder) and a Samsung Galaxy Note 9 (using exynos decoder):
Pixel 4
Galaxy Note 9
As you can see, for the video called jetson/h264/2/test.h264, the decoding time on the snapdragon device is ~21 times higher than for the Samsung device. pic_order_cnt_type==0 for this stream. However, on other streams the difference in decoding time is insignificant. (they all use pic_order_cnt_type==2)
The main parameter determining if the snapdragon decoder enters a "low-latency decoding path" seems to be the value pic_order_cnt_type mentioned above.
If I understand the h264 specs correctly, if this value is set to 2 picture re-ordering is impossible (no buffered frames). If it is set to 0, picture re-ordering is possible, but not neccessarily used by the encoder. However, the snapdragon decoder doesn't differentiate between "possible" and "actually used by the encoder", leading to the big decoding latency difference.
I was able to reduce the decoding latency on snapdragon by manipulating the bitstream before sending it to the decoder (adding VUI with num_reorder_frames=0 and max_dec_frame_buffering=0) but it never results in 0 buffered frames, only less buffered frames.
Related
The three Android devices that I am testing out have three different AVC packets. Here below are the samples for Samsung, Motorola, and Doffe.
Samsung
1700000000014d001effe10012674d001eda0280bfe5948283030368509a8001000468ee0
Motorola
1700000000014d001effe10012674d001ee901405ff2ca41418181b4284d4001000468ee06e2
Doofe
170000000001640029ffe1001067640029ac1b1a80a02ff9601e1108a701000468ea43cb
This causes huge problems when I interleave the videos. The Video player gets obviously confused and does not play.
How do I ensure that the video headers are the same? Should I use a software encoder and bypass the hardware encoders?
How is carried out encoding on the clients?
PPS SPS describe your video stream parameters like frame sizes, profile, etc. and almost always are generated by encoder.
How do I ensure that the video headers are the same?
Therefore, you have to ensure all devices use the same video encoder and publish video with the same format (frame size, bitrate, fps, profile, key frame count, etc)
Should I use a software encoder and bypass the hardware encoders?
In your case, preferentially to use soft encoders. But you first just can try to make encoders configurations the same. Most likely it resolve your issue
I am using mediacodec to decodec a h264 stream on samsung S6, android 5.1.1, found the input buffer to mediacodec must start with "0001"(and don't need to set pps, sps), or the ACodec will report error.
I also tried to use mediaextractor to play a mp4 file, it works fine, but the buffer to mediacodec is not start with "0001".
I don't know why decodec a h264 stream has such limitation, currently i need to analyze the stream from socket, and cut the data into small packages(each package start with 0001) and then give them to mediacodec, but it is inefficient.
MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 1024, 1024);
Some specific decoders may also be able to decode H264 NAL units in the "mp4" format (with a different kind of startcode), but that's not guaranteed across all devices.
It may be that Samsung's version of MediaExtractor returns it in this format, if they know that their own decoder can handle it. There is at least earlier precedent that Samsung did the same, nonstandard thing with timestamps with their version of MediaExtractor, see e.g. https://code.google.com/p/android/issues/detail?id=74356.
(Having MediaExtractor return data that only the current device's decoder can handle is wrong IMO, though, since one may want to use MediaExtractor to read a file but send the compressed data over the network to another device for decoding, and in these cases, returning data in a nonstandard format is wrong.)
As fadden wrote, MediaCodec operates on full NAL units though, so you need to provide data in this format (even if you think it feels inefficient). If you receive data over a socket in a format where this information (about frame boundaries) isn't easily available, then that's an issue with your protocol format (e.g., implementing RTP reception is not easy!), not with MediaCodec itself - it's a quite common limitation to need to have full frames before decoding, instead of being able to feed random chunks until you have a full frame. This shouldn't be inefficient unless your own implementation of it is inefficient.
In general android will expect nal units for each input. For some devices i have found that setting the csd-0/1 on the media-format for h264 to not work consistently. But if you feed each of the parameters sets as input buffers the media-codec will pick it up as a format-change.
int outputBufferIndex = NativeDecoder.DequeueOutputBuffer (info, 1000);
if (outputBufferIndex == (int)MediaCodec.InfoOutputFormatChanged) {
Console.WriteLine ("Format changed: {0}", NativeDecoder.OutputFormat);
} else if (outputBufferIndex >= 0) {
CodecOutputBufferAvailable (NativeDecoder, outputBufferIndex, info);
}
Also note it is mandatory for Nexus and some other samsung devices to set:
formatDescription.SetInteger(MediaFormat.KeyWidth, SelectedPalette.Value.Width);
formatDescription.SetInteger(MediaFormat.KeyHeight, SelectedPalette.Value.Height);
formatDescription.SetInteger(MediaFormat.KeyMaxInputSize, SelectedPalette.Value.Width * SelectedPalette.Value.Height);
I am lucky in my situation i can query these resolutions. But you can parse the resolution manually from SPS and PPS nal units.
// NOTE i am using Xamarin here. But the calls and things are pretty much the same. I am fairly certain there are bugs in the iOS VideoToolbox Xamarin Wrapper so yeah.. Keep that in mind if your ever considering Xamarin for video decoding. Its great for everything but anything thats slightly more custom or low-level.
I am attempting to record an mp4 through the use of OpenGL SurfaceView and using a MediaCodec encoder to encode the frames of the video in one thread and in another thread I am using AudioRecord to record audio samples and using a seperate encoder to encode the audio data and finally a MediaMuxer to do the actualy muxing.
On some devices (mostly higher end devices) the method that I am using works completely fine and the output video is to be expected with no error or anything like that. However, on some other devices that isnt the case.
What happens is on some devices, (ie Moto Droid Turbo 2) the audio encoder (which is running in its own thread) will process a few audio samples then return MediaCodec.INFO_BUFFER_UNAVAILABLE when attempting to dequeue an input buffer for the encoder, this start happening after about 5 samples that are successfully encoded and the video encoder runs completely fine. But on another device (ie the Samsung Galaxy Alpha) just opposite happens where the video encoder begins to return the MediaCodec.INFO_BUFFER_UNAVAILABLE status when dequeue an input buffer for the video encoder and the audio encoder runs fine.
I guess my question is, and I have looked all over for a completely clear explanation of this, is what is the cause of a buffer being unavailable? Other then not releasing the buffer before next use, what can cause this? I am 98% certain that I am releasing the buffers when the need to be released because the exact same code will work on most devices that I have tested on. (Nexus 5x, Nexus 6P, Samsung S6, Samsung Edge/Edge+, Moto X (2014), Moto X (2015), HTC One M9, HTC One M8, etc).
Does anyone have any ideas? I didnt think posting code examples was necessary because I know it works on most devices I have tested but my question is what can cause MediaCodec.INFO_BUFFER_UNAVAILABLE to be returned aside from not release the buffer, if anything?
EDIT:
Further investigating the issue and it become a little strange even. I was starting to think that this had something to do with processing power of the device, which would explain why the higher end devices worked fine but the lower end did not. HOWEVER, the Samsung Note Edge (which works fine) has the same CPU, GPU, chipset AND amount of RAM as the Moto Droid Turbo, which encounters the error with dequeing audio buffers from the audio encoder.
Edit 2:
Yup, this was entirely something that I was doing that caused this. The issue was happening because I missed a release call to the buffer in the instance that the muxer hadn't yet started yet. So instead of releasing the buffer in that instance, I just ignore it and moved it which caused the buffer to be hung up. problem solved
I'm decoding a H.264 stream on Android 4.2 using Mediacodec. Unfortunately, the decoder always buffers 6-10 frames, which lead to annoying latency, and Android does not provide any API to adjust buffer size. So my question is, how to modify the Android source code (or the OMX driver) in order to reduce the buffer size for realtime video streaming?
Generally speaking, you don't. The number of buffers in the queue is determined by the codec. Different devices, and different codecs on the same device, can behave differently.
Unless you're using the software AVC codec, the codec implementation is provided as a binary by the hardware OEM, so there's no way to modify it (short of hex-editing).
Hardware:Nexus 7
OS Version: Android 4.1.2
Problem:
I use OpenMAX IL to decode H.264 video stream(640x480). Between I give a raw H.264 frame and I receive a YUV frame, it is about 2~4 seconds. The YUV frame is correct.
Question:
Is it really need so much time? Or I have made wrong configurations?