I am using mediacodec to decodec a h264 stream on samsung S6, android 5.1.1, found the input buffer to mediacodec must start with "0001"(and don't need to set pps, sps), or the ACodec will report error.
I also tried to use mediaextractor to play a mp4 file, it works fine, but the buffer to mediacodec is not start with "0001".
I don't know why decodec a h264 stream has such limitation, currently i need to analyze the stream from socket, and cut the data into small packages(each package start with 0001) and then give them to mediacodec, but it is inefficient.
MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 1024, 1024);
Some specific decoders may also be able to decode H264 NAL units in the "mp4" format (with a different kind of startcode), but that's not guaranteed across all devices.
It may be that Samsung's version of MediaExtractor returns it in this format, if they know that their own decoder can handle it. There is at least earlier precedent that Samsung did the same, nonstandard thing with timestamps with their version of MediaExtractor, see e.g. https://code.google.com/p/android/issues/detail?id=74356.
(Having MediaExtractor return data that only the current device's decoder can handle is wrong IMO, though, since one may want to use MediaExtractor to read a file but send the compressed data over the network to another device for decoding, and in these cases, returning data in a nonstandard format is wrong.)
As fadden wrote, MediaCodec operates on full NAL units though, so you need to provide data in this format (even if you think it feels inefficient). If you receive data over a socket in a format where this information (about frame boundaries) isn't easily available, then that's an issue with your protocol format (e.g., implementing RTP reception is not easy!), not with MediaCodec itself - it's a quite common limitation to need to have full frames before decoding, instead of being able to feed random chunks until you have a full frame. This shouldn't be inefficient unless your own implementation of it is inefficient.
In general android will expect nal units for each input. For some devices i have found that setting the csd-0/1 on the media-format for h264 to not work consistently. But if you feed each of the parameters sets as input buffers the media-codec will pick it up as a format-change.
int outputBufferIndex = NativeDecoder.DequeueOutputBuffer (info, 1000);
if (outputBufferIndex == (int)MediaCodec.InfoOutputFormatChanged) {
Console.WriteLine ("Format changed: {0}", NativeDecoder.OutputFormat);
} else if (outputBufferIndex >= 0) {
CodecOutputBufferAvailable (NativeDecoder, outputBufferIndex, info);
}
Also note it is mandatory for Nexus and some other samsung devices to set:
formatDescription.SetInteger(MediaFormat.KeyWidth, SelectedPalette.Value.Width);
formatDescription.SetInteger(MediaFormat.KeyHeight, SelectedPalette.Value.Height);
formatDescription.SetInteger(MediaFormat.KeyMaxInputSize, SelectedPalette.Value.Width * SelectedPalette.Value.Height);
I am lucky in my situation i can query these resolutions. But you can parse the resolution manually from SPS and PPS nal units.
// NOTE i am using Xamarin here. But the calls and things are pretty much the same. I am fairly certain there are bugs in the iOS VideoToolbox Xamarin Wrapper so yeah.. Keep that in mind if your ever considering Xamarin for video decoding. Its great for everything but anything thats slightly more custom or low-level.
Related
I'm trying to decode h.264 stream using Android's MediaCodec interface. Everything is working just fine on my test devices, but on one customer device that I dont' have access to (Samsung Tab S) there are strange issues.
When I decode the stream I don't send any SPS/PPS NALs or an initial frame. I just start pushing the data from the live stream, chopped into blocks ending with a 0x09 NAL and the decoder will synchronize itself nicely without problems quite quickly.
The issue with at least this one device is that when I get a BufferInfo from the decoder it will claim it decoded 1413120 bytes of data, but the buffer size is only 1382400! So of course if I even try to get that much data out of the buffer it will crash.
The video is 1280x720 and is decoded into NV12, so the buffer size is just fine. The reported decoded output size isn't. If I force the size as 1382400 and convert the NV12 into RGB I get almost correct picture. The first 32 lines have a strong green color and the blue channel is shifted by quite a lot. This means that the UV block is decoded partially wrong on this device.
Has anyone run into this kind of an issue before? I have recorded the raw h264 stream from that specific device and it plays just fine with no green blocks or color shifts.
Should I really set up SPS/PPS and an initial frame before starting the streaming? The stream seems to contain everything needed since the decoder realizes the resolution to be correct, sets up buffers and decodes on every other device I've tested except this one. So I'm just wondering if Samsung has something special going on.
Another application decoding the same stream shows it without issues, but as far as I know they use ffmpeg internally, not MediaCodec. I would rather use built-in system codecs if possible.
Here is an example of the result. Don't have an image of just the stream, do note that the frame is rotated. The Y component in the green area is just fine and on the white block on the right you can clearly see the blue shift.
Edit: Even if I start the decoder with the SPS/PPS blocks in csd-0 the color problems persist. So it isn't due to that.
Also managed to test the exact stream with another device. No green bar, no color shifts. So it's a problem with the codec in that particular device/model.
I had similar issues in the past (specifically on Samsung devices) and if I remember correctly it was due to the missing SPS/PPS data. You must feed in the SPS/PPS data if you want consistent results.
Not a direct solution to your issue but a possible workaround is to use an alternative decoder (if present) when running on that particular device.
I'm not sure how you are instantiating your decoder but often people use the mime type like so:
decoder = MediaCodec.createDecoderByType("video/avc");
The device will then choose the preferred decoder (probably hardware).
You can alternatively instantiate a specific decoder like so:
decoder = MediaCodec.createByCodecName("OMX.google.h264.decoder");
// OR
decoder = MediaCodec.createByCodecName("OMX.qcom.video.decoder.avc");
In my experience, most devices have at least 2 different H264 decoders available and you may find that an alternative decoder on this device performs without faults.
You can list all the available codecs using the following code:
static MediaCodecInfo[] getCodecs() {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
MediaCodecList mediaCodecList = new MediaCodecList(MediaCodecList.ALL_CODECS);
return mediaCodecList.getCodecInfos();
} else {
int numCodecs = MediaCodecList.getCodecCount();
MediaCodecInfo[] mediaCodecInfo = new MediaCodecInfo[numCodecs];
for (int i = 0; i < numCodecs; i++) {
MediaCodecInfo codecInfo = MediaCodecList.getCodecInfoAt(i);
mediaCodecInfo[i] = codecInfo;
}
return mediaCodecInfo;
}
}
We faced a similar issue of Out of memory when trying to play on Amazon Fire stick (AFTMM) using h265 decoder at 480p and 1440p layer. The issue was coming for both SDR and HDR playback. While this definitely is a device specific issue, there could be some workarounds like reducing the ref value while encoding and/or reducing CRF as well. This works for us.
I searched for hours ..
I just want a working decode/encode of a recorded movie.
Is this even possible on android 4.1?
Now i writes only a few kb's to my mp4 file. No errors.
After this will work, i will use KEY_FRAME_RATE and KEY_I_FRAME_INTERVAL to put it in slow motion.
I used a mediaExtractor to configure the MediaCodec.
I see 3 steps (see gist for complete code):
1./
encoder.dequeueInputBuffer(5000);
extractor.readSampleData(inputBuf, offset);
ptsUsec2 = extractor.getSampleTime();
encoder.queueInputBuffer(inputBufIndex, ...);
2./
encoder.dequeueOutputBuffer(info, 5000);
ByteBuffer encodedData = encoderOutputBuffers[encoderStatus];
//i write encodedData to a FileOutputStream (to save the MP4);
decoder.queueInputBuffer(inputBufIndex, ...);
3./
decoder.dequeueOutputBuffer(info, 5000);
decoder.releaseOutputBuffer(decoderStatus, ...);
Here is the complete function i modified from google's EncodeDecodeTest file:
gist
Thanks for help,
Felix
Some additional information is available on bigflake. In particular, FAQ item #9.
The format of frames coming out of the MediaCodec decoder is not guaranteed to be useful. Many popular devices decode data into a proprietary YUV format, which is why the checkFrame() function in the buffer-to-buffer test can't always verify the results. You'd expect the MediaCodec encoder to be able to accept the frames output by the decoder, but that's not guaranteed.
Coding against API 18+ is generally much easier because you can work with a Surface rather than a ByteBuffer.
Of course, if all you want is slow-motion video, you don't need to decode and re-encode the H.264 stream. All you need to do is alter the presentation time stamps, which are in the .mp4 wrapper. On API 18+, you can extract with MediaExtractor and immediately encode with MediaMuxer, without involving MediaCodec at all. On API 16, MediaMuxer doesn't exist, so you'd need some other way to wrap H.264 as .mp4.
Unless, of course, you have some aversion to variable-frame-rate video, in which case you'll need to re-encode it with the "slow motion" frames repeated (and timestamps adjusted appropriately). The KEY_FRAME_RATE and KEY_I_FRAME_INTERVAL values will not help you -- they're set when the encoder is configured, and have no affect on frame timing.
I'm having a very difficult time with MediaCodc. I've used it previously to decode a raw h.264 stream and learned a significant amount. At least I thought I had.
My stream is h.264 in Annex B format. Looking at the raw data, the structure of my NAL packet code types are as follows:
[0x09][0x09][0x06] and then 8 packets of [0x21].
[0x09][0x09][0x27][0x28][0x06] and then 8 packets of [0x21].
This is not how I am receiving them. I am attempting to build a complete Access Unit from these raw NAL unit types.
First thing that is strange to me is the double [0x09] which is the Access Unit Delimiter packet. I am pretty sure the h.264 document specifies only 1 AUD per Access Unit. BTW, I am able to record the raw data and play it using ffmpeg with, and without, the extra AUD. For now, I am detecting this message and stripping the first one off before sending the entire Access Unit to the MediaCodec.
Second thing is, I have hardcoded the SPS/PPS byte array messages [0x27/0x28] and I am setting these in the MediaFormat used to initialize the MediaCodec, similar to:
format.setByteBuffer("csd-0", ByteBuffer.wrap( mySPS ));
format.setByteBuffer("csd-1", ByteBuffer.wrap( myPPS ));
My video stream provider vendor tells me the video is 1280 x 720, however, when I convert it to an mp4 file, the metadata says its 960 x 720. Another oddity.
Changing these different parameters around, I am still unable to get a valid buffer index in my Thread that processes the decoder output (dequeueOutputBuffer returns -1). I have also varied the timeout for this to no avail. If I manually set the SPS/PPS as the first packet NOT using the example above, I do get the -3 "output buffers have changed" which is meaningless since I am using API 20. But everything else I get returned is -1.
I've read about the Emulation Prevention Byte encoding of h.264. I am able to strip this byte out and send to the MediaCodec. Doesn't seem to make a difference. Also, the documentation for MediaCodec doesn't explicitly say if the code is expecting the EPB to be stripped out or not ... ?
Other than the video frame resolution, the only other thing that is different than my previous success is the existence of the SEI packet type [0x06]. I'm not sure if I should be doing something special with this or not?
I know there are a number of folks who have used MediaCodec have had issues with it, mostly because the documentation is not very good. Can anyone offer any other advice as to what I could be doing wrong?
I'm decoding a H.264 stream on Android 4.2 using Mediacodec. Unfortunately, the decoder always buffers 6-10 frames, which lead to annoying latency, and Android does not provide any API to adjust buffer size. So my question is, how to modify the Android source code (or the OMX driver) in order to reduce the buffer size for realtime video streaming?
Generally speaking, you don't. The number of buffers in the queue is determined by the codec. Different devices, and different codecs on the same device, can behave differently.
Unless you're using the software AVC codec, the codec implementation is provided as a binary by the hardware OEM, so there's no way to modify it (short of hex-editing).
I have a project where I have been asked to display a video stream in android, the stream is raw H.264 and I am connecting to a server and will receive a byte stream from the server.
Basically I'm wondering is there a way to send raw bytes to a decoder in android and display it on a surface?
I have been successful in decoding H264 wrapped in an mp4 container using the new MediaCodec and MediaExtractor API in android 4.1, unfortunately I have not found a way to decode a raw H264 file or stream using these API's.
I understand that one way is to compile and use FFmpeg but I'd rather use a built in method that can use HW acceleration. I also understand RTSP streaming is supported in android but this is not an option. Android version is not an issue.
I can't provide any code for this unfortunately, but I'll do my best to explain it based on how I got it to work.
So here is my overview of how I got raw H.264 encoded video to work using the MediaCodec class.
Using the link above there is an example of getting the decoder setup and how to use it, you will need to set it up for decoding H264 AVC.
The format of H.264 is that it’s made up of NAL Units, each starting with a start prefix of three bytes with the values 0x00, 0x00, 0x01 and each unit has a different type depending on the value of the 4th byte right after these 3 starting bytes. One NAL Unit IS NOT one frame in the video, each frame is made up of a number of NAL Units.
Basically I wrote a method that finds each individual unit and passes it to the decoder (one NAL Unit being the starting prefix and any bytes there after up until the next starting prefix).
Now if you have the decoder setup for decoding H.264 AVC and have an InputBuffer from the decoder then you are ready to go. You need to fill this InputBuffer with a NAL Unit and pass it back to the decoder and continue doing this for the length of the stream.
But, to make this work I had to pass the decoder a SPS (Sequence Parameter Set) NAL Unit first. This unit has a byte value of 0x67 after the starting prefix (the 4th byte), on some devices the decoder would crash unless it received this Unit first.
Basically until you find this unit, ignore all other NAL Units and keep parsing the stream until you get this unit, then you can pass all other units to the decoder.
Some devices didn't need the SPS first and some did, but you are better of passing it in first.
Now if you had a surface that you passed to the decoder when you configured it then once it gets enough NAL units for a frame it should display it on the surface.
You can download the raw H.264 from the server, then offer it via a local HTTP server running on the phone and then let VLC for Android do playback from that HTTP server. You should use VLC's http/h264:// scheme to force the demuxer to raw H.264 (if you don't force the demuxer VLC may not be able to recognize the stream, even when the MIME type returned by the HTTP server is set correctly). See
https://github.com/rauljim/tgs-android/blob/integrate_record/src/com/tudelft/triblerdroid/first/VideoPlayerActivity.java#L211
for an example on how to create an Intent that will launch VLC.
Note: raw H.264 apparently has no timing info, so VLC will play as fast as possible.
First embedding it in MPEGTS will be better. Haven't found a Android lib that will do that yet.
Here are the resources I've found helpful in a similar project:
This video has been super insightful in understanding how MediaCodec handles raw h.264 streams on a high level.
This thread goes into a bit more detail as to handling the SPS/PPS NALUs specifically. As was mentioned above, you need to separate individual NAL Units using the start prefix, and then hand the remaining data to the MediaCodec.
This repo (libstreaming) is a great example of decoding an H264 stream in Android using RTSP/RTP for transmission.