I've got an H264 stream and want to use MediaCodec to decode it to surface to display.
Because I'm only decoding, and want to target as many devices as possible, I'm supporting API 16 and used the ExtractMpegFrames test as reference.
http://bigflake.com/mediacodec/ExtractMpegFramesTest.java.txt
When calling dequeOutputBuffers(info,timeout); the info.size is always zero. If I hardcode releaseOutputBuffer(index,true); i get the display (depending on how i set up surface... creating a surface in main activity onCreate and passing it to configure works, using the surface concept from the extractmpegframes does not...might debug that).
Regardless if i use a boolean based on info.size or if i hardcode the "true"...i get a lot of
"SEC_VIDEO_DEC(1840): output buffer is smaller than decoded data size Out Length" (on samsung galaxy note 10.1) which after googling led me to:
https://gitorious.org/replicant/device_samsung_aries-common/source/9ff4c660a554dc2816db67004fccb10f6ad0e0fa:aries/sec_mm/sec_omx/sec_omx_component/video/dec/SEC_OMX_Vdec.c#L849
line 849
and on this device i have a lot of stuttering due to what appears to be a large amount of allocations and subsequent garbage collection.
surface is definitely not null...Samsung tablet is 4.1.2..
have also tried on a new Lenovo Yoga 10 HD (4.2 pretty sure, tablet is out right now) and also have stuttering video, size is not zero, but is "small" compared to phone
questions:
1. why is size always zero on the note tablet but not on galaxyS3 phone?
2. is there a way around the zero size for the tablet?
3. is there something i should check to determine if a device won't support what i'm doing? (even if it is at a supposedly supported sdk level)
4. do i need to catch a changing sps/pps midstream and stop the decoder, reconfigure it and restart it?
decoder setup is now parsing sps/pps from first AU:
private final String MIME_TYPE = "video/avc"; // H.264 Advanced Video Coding
//set width and height
//cropping parameters should be used only when [frame_cropping_flag] is enabled in SPS.
if (h.sps.frame_cropping_flag == 1)
{
mWidth = ((h.sps.pic_width_in_mbs_minus1 + 1) * 16) - h.sps.frame_crop_left_offset * 2 - h.sps.frame_crop_right_offset * 2;
mHeight= ((2 - h.sps.frame_mbs_only_flag)* (h.sps.pic_height_in_map_units_minus1 +1) * 16) - (h.sps.frame_crop_top_offset * 2) - (h.sps.frame_crop_bottom_offset * 2);
}
else
{
mWidth = ((h.sps.pic_width_in_mbs_minus1 + 1) * 16);
mHeight= ((2 - h.sps.frame_mbs_only_flag)* (h.sps.pic_height_in_map_units_minus1 +1) * 16);
}
MediaFormat format = MediaFormat.createVideoFormat(MIME_TYPE, mWidth, mHeight);
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1920 * 1080);
format.setInteger(MediaFormat.KEY_MAX_WIDTH,1920);
format.setInteger(MediaFormat.KEY_MAX_HEIGHT,1080);
format.setInteger(MediaFormat.KEY_PUSH_BLANK_BUFFERS_ON_STOP,1);
//we know we have Codec-Sepcific Data if we got here
format.setByteBuffer("csd-0", sps);
if (ppsOffset != -1)
{
format.setByteBuffer("csd-1", pps);
}
else
{
//can this happen?
Log.d(s_logTag, "UpdateInput - did not find PPS data");
}
// Create a MediaCodec for the desired codec using the format variable we just created along with the surface passed to us in the constructor
mDecoder = MediaCodec.createDecoderByType(MIME_TYPE);
mDecoder.configure(format, mSurface, null, 0); //not an encoder so pass 0 instead of flag: MediaCodec.CONFIGURE_FLAG_ENCODE);
mDecoder.start();
Related
I'm working with Android MediaCodec and use it for a realtime H264 encoding and decoding frames from camera. I use MediaCodec in synchronous manner and render the output to the Surface of decoder and everething works fine except that I have a long latency from a realtime, it takes 1.5-2 seconds and I'm very confused why is it so.
I measured a total time of encoding and decoding processes and it keeps around 50-65 milliseconds so I think the problem isn't in them.
I tried to change the configuration of the encoder but it didn't help and currently it configured like this:
val formatEncoder = MediaFormat.createVideoFormat("video/avc", 1920, 1080)
formatEncoder.setInteger(MediaFormat.KEY_FRAME_RATE, 30)
formatEncoder.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5)
formatEncoder.setInteger(MediaFormat.KEY_BIT_RATE, 1920 * 1080)
formatEncoder.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface)
val encoder = MediaCodec.createEncoderByType("video/avc")
encoder.configure(formatEncoder, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
val inputSurface = encoder.createInputSurface() // I use it to send frames from camera to encoder
encoder.start()
Changing the configuration of the decoder also didn't help me at all and currently I configured it like this:
val formatDecoder = MediaFormat.createVideoFormat("video/avc", 1920, 1080)
val decoder = MediaCodec.createDecoderByType("video/avc")
decoder.configure(formatDecoder , outputSurface, null, 0) // I use outputSurface to render decoded frames into it
decoder.start()
I use the following timeouts for waiting for available encoder/decoder buffers I tried to reduce their values but it didn't help me and I left them like this:
var TIMEOUT_IN_BUFFER = 10000L // microseconds
var TIMEOUT_OUT_BUFFER = 10000L // microseconds
Also I measured the time of consuming the inputSurface a frame and this time takes 0.03-0.05 milliseconds so it isn't a bottleneck. Actually I measured all the places where a bottleneck could be, but I wasn't found anything and I think the problem is in the encoder or decoder itself or in their configurations, or maybe I should use some special routine for sending frames to encoding/decoding..
I also tried to use HW accelerated codec and it's the only thing that helped me, when I use it the latency reduces to ~ 500-800 milliseconds but it still doesn't fit me for a realtime streaming.
It seems to me that the encoder or decoder buffers several frames before start displaying them on the surface and eventually it leads to the latency and if it really so then how can I disable bufferization or reduce the time of it?
Please help me I'm stucking on this problem for about half a year and have no idea how to reduce the latency, I'm sure that it's possible because popular apps like Telegram, Viber, WhatsApp etc. work fine and without latency so what's the secret here?
UPD 07.07.2021:
I still haven't found a solution to get rid of the latency. I've tried to change h264 profiles, increase and decrease I-frame inteval, bitrate, framerate, but result the same, the only thing that hepls a little to reduce the latency - downgrade the resolution from 1920x1080 to e.g. 640x480, but this "solution" doesn't suit me because I want to encode/decode a realtime video with 1920x1080 resolution.
UPD 08.07.2021:
I found out that if I change the values of TIMEOUT_IN_BUFFER and TIMEOUT_OUT_BUFFER from 10_000L to 100_000L it decreases the latency a bit but increases the delay of showing the first frame quite a lot after start encoding/decoding process.
It's possible your encoder is producing B frames -- bilinear interpolation frames. They increase quality and latency, and are great for movies. But no good for low-latency applications.
Key frames = I (interframes)
Predicted frames = P (difference from previous frames)
Interpolated frames = B
A sequence of frames including B frames might look like this:
IBBBPBBBPBBBPBBBI
11111111
12345678901234567
The encoder must encode each P frame, and the decoder must decode it, before the preceding B frames make any sense. So in this example the frames get encoded out of order like this:
1 5 2 3 4 9 6 7 8 13 10 11 12 17 17 13 14 15
In this example the decoder can't handle frame 2 until the encoder has sent frame 5.
On the other hand, this sequence without B frames allows coding and decoding the frames in order.
IPPPPPPPPPPIPPPPPPPPP
Try using the Constrained Baseline Profile setting. It's designed for low latency and low power use. It suppresses B frames. I think this works.
mediaFormat.setInteger(
"profile",
CodecProfileLevel.AVCProfileConstrainedBaseline);
I believe android h264 decoder have latency (at-least in most cases i've tried). Probably that's why android developers added PARAMETER_KEY_LOW_LATENCY from API level 30.
However I could decrease the delay some frames by querying for the output some more times.
Reason: no idea. It's just result of boring trial and errors
int inputIndex = m_codec.dequeueInputBuffer(-1);// Pass in -1 here bc we don't have a playback time reference
if (inputIndex >= 0) {
ByteBuffer buffer;
if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.LOLLIPOP) {
buffer = m_codec.getInputBuffer(inputIndex);
} else {
ByteBuffer[] bbuf = m_codec.getInputBuffers();
buffer = bbuf[inputIndex];
}
buffer.put(frame);
// tell the decoder to process the frame
m_codec.queueInputBuffer(inputIndex, 0, frame.length, 0, 0);
}
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
int outputIndex = m_codec.dequeueOutputBuffer(info, 0);
if (outputIndex >= 0) {
m_codec.releaseOutputBuffer(outputIndex, true);
}
outputIndex = m_codec.dequeueOutputBuffer(info, 0);
if (outputIndex >= 0) {
m_codec.releaseOutputBuffer(outputIndex, true);
}
outputIndex = m_codec.dequeueOutputBuffer(info, 0);
if (outputIndex >= 0) {
m_codec.releaseOutputBuffer(outputIndex, true);
}
You need to configure customized(or KEY_LOW_LATENCY if it is supported) low latency parameters for different cpu venders. It is a common problem for android phone.
Check this code https://github.com/moonlight-stream/moonlight-android/blob/master/app/src/main/java/com/limelight/binding/video/MediaCodecHelper.java
I searched online but there is very little information about this.
I have a live broadcasting app where I send encoded H264 video frames and AAC audio chunks resulting from camera & mic using the Android MediaCodec SDK, over a RTMP stack.
My live streams are 720p and I aim for great quality with 2500Kbps. This obviously requires a very good network connection which means 4G if you use a data plan.
Problem is even with the greatest connection there will be low peaks and congestion, so there will be moments where the network cant hold such heavy stream. Because I want to offer high reliability, I want to include automatic adaptive bitrate on my app so that the image quality is dropped in favor or reliability.
The thing is -- how to achieve this automatic adaptation to the network conditions without losing frames? Is it even possible? I've used professional encoding devices like Cerevo and they dont ever lose frames -- however with my app I always get some awful dragging due to p-frames being lost in the network.
This is what I currently have:
private long adaptBitrate(long idleNanos, Frame frame) {
int bytes = frame.getSize();
long nowNanos = System.nanoTime();
if (nowNanos - mLastNanos > 1000L * 1000 * 1000) {
double idle = (double) idleNanos / (double) (nowNanos - mLastNanos);
float actualBitrate = newBitrate;
int size = mBuffer.size();
String s = "Bitrate: " + actualBitrate / 1000
+ " kbps In-Flight:" + bytes
+ " idle: " + idle;
if (size > MAX_BUF_SIZE && size > mLastSize) {
Log.i(TAG, "adaptBitrate: Dropping bitrate");
newBitrate = (int) ((double) actualBitrate * BITRATE_DROP_MULTIPLIER);
if (newBitrate < MIN_BITRATE) {
newBitrate = MIN_BITRATE;
}
s += " late => " + newBitrate;
mRtmpHandler.requestBitrate(newBitrate);
} else if (size <= 2 && idle > IDLE_THRESHOLD) {
mIdleFrames++;
if(mIdleFrames >= MIN_IDLE_FRAMES){
Log.i(TAG, "adaptBitrate: Raising bitrate");
newBitrate = (int) ((double) newBitrate * BITRATE_RAISE_MULTIPLIER);
if (newBitrate > MAX_BITRATE) {
newBitrate = MAX_BITRATE;
}
s += " idle => " + newBitrate;
mRtmpHandler.requestBitrate(newBitrate);
mIdleFrames = 0;
}
}
debugThread(Log.VERBOSE, s);
mLastNanos = System.nanoTime();
mLastSize = size;
idleNanos = 0;
}
return idleNanos;
}
So if my buffer is exceeding a threshold, I lower the bitrate. If my app is spending too much time waiting for a new frame, for a number of consecutive frames, then I raise the bitrate.
No matter how cautious I am with the threshold values, I am always losing important information and my stream breaks until the next keyframe arrives (2 secs). Sometimes it seems that the network can hold a certain bitrate (stable at 1500kbps, for instance) but the image will still have some dragging as though a frame was lost in the way. With good network conditions, everything is smooth.
How do these streaming devices handle these situations? It always looks great with them, no dragging or skipped frames at all...
There is indeed no information online about adaptive bitrate from the broadcaster side, surprisingly. When I had to implement something like this with RTSP and two rtp sockets, I took a similar approach, creating a polling class that would modestly increase the bitrate of the mediacodec when the packet buffer was >$GOOD_PCT free, aggressively halve it when the queue was less than $BAD_PCT free, and do nothing if it was in between. Partially seen here. I'm not sure I have a complete picture of your solution based on the posted code, but you are adjusting the mediacodec bitrate directly, correct? The only time I had corruption is when I requested a sync frame from the mediacodec, so avoid that if it's in your code. Hope this helps.
I am trying to decode AAC encoded files in my application and to initialise the MediaFormatobject used to initialise my MediaCodec object, This is the code for setting up the variables for the MediaFormat object
MediaExtractor mediaExtractor = new MediaExtractor();
try {
mediaExtractor.setDataSource(audioFilePath);
} catch (IOException e) {
return false;
}
Log.d(TAG, "Number of tracks in the file are:" + mediaExtractor.getTrackCount());
MediaFormat mediaFormat = mediaExtractor.getTrackFormat(0);
Log.d(TAG, "mediaFormat:" + mediaFormat.toString());
mSampleRate = mediaFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE);
Log.d(TAG, "mSampleRate: " + mSampleRate);
mChannels = mediaFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
Log.d(TAG, "mChannels number of channels: " + mChannels);
// Reading the duration from the file and converting from micro seconds to milliseconds.
mDuration = (int) (mediaFormat.getLong(MediaFormat.KEY_DURATION) / 1000);
Log.d(TAG, "duration: " + mDuration);
// Getting the csd-0 info from the file ..
mCSDBuffer = mediaFormat.getByteBuffer("csd-0");
The problem I am facing is that the statement mCSDBuffer = mediaFormat.getByteBuffer("csd-0") fetches me null for the same file on some devices. The application is in production and I see this error on armabi-v7a/armabiprocessors devices with android API level of 17, 18 and 19 and most of these errors are on Samsung devices. Any direction on this?
If the csd-0 buffer is null, then I would expect it still to decode correctly when passed into MediaCodec. Does it, if you just choose not to set the csd-0 data as input to MediaCodec, if it is null? In general, you should be able to decode the MediaExtractor output if you just pipe it straight to MediaCodec.
The actual format of the data output from MediaExtractor is not very strictly specified though, so in practice it is known that some manufacturers (mainly Samsung) change this in a way that only their own decoder handles. See e.g. https://code.google.com/p/android/issues/detail?id=74356 for another case of the same.
Ideally, the Android CTS tests would be made stricter to make sure that MediaExtractor behaves consistently, allowing its use in a more generic context, or use another decoder than MediaCodec. (E.g. with the current Samsung issues, you can't use MediaExtractor on one device, send the extracted data over a network to another device and decode it there.)
I'm using ffmpeg in Android project via JNI to decode real-time H264 video stream. On the Java side I'm only sending the the byte arrays into native module. Native code is running a loop and checking data buffers for new data to decode. Each data chunk is processed with:
int bytesLeft = data->GetSize();
int paserLength = 0;
int decodeDataLength = 0;
int gotPicture = 0;
const uint8_t* buffer = data->GetData();
while (bytesLeft > 0) {
AVPacket packet;
av_init_packet(&packet);
paserLength = av_parser_parse2(_codecPaser, _codecCtx, &packet.data, &packet.size, buffer, bytesLeft, AV_NOPTS_VALUE, AV_NOPTS_VALUE, AV_NOPTS_VALUE);
bytesLeft -= paserLength;
buffer += paserLength;
if (packet.size > 0) {
decodeDataLength = avcodec_decode_video2(_codecCtx, _frame, &gotPicture, &packet);
}
else {
break;
}
av_free_packet(&packet);
}
if (gotPicture) {
// pass the frame to rendering
}
The system works pretty well until incoming video's resolution changes. I need to handle transition between 4:3 and 16:9 aspect ratios. While having AVCodecContext configured as follows:
_codecCtx->flags2|=CODEC_FLAG2_FAST;
_codecCtx->thread_count = 2;
_codecCtx->thread_type = FF_THREAD_FRAME;
if(_codec->capabilities&CODEC_FLAG_LOW_DELAY){
_codecCtx->flags|=CODEC_FLAG_LOW_DELAY;
}
I wasn't able to continue decoding new frames after video resolution change. The got_picture_ptr flag that avcodec_decode_video2 enables when whole frame is available was never true after that.
This ticket made me wonder if the issue isn't connected with multithreading. Only useful thing I've noticed is that when I change thread_type to FF_THREAD_SLICE the decoder is not always blocked after resolution change, about half of my attempts were successfull. Switching to single-threaded processing is not possible, I need more computing power. Setting up the context to one thread does not solve the problem and makes the decoder not keeping up with processing incoming data.
Everything work well after app restart.
I can only think of one workoround (it doesn't really solve the problem): unloading and loading the whole library after stream resolution change (e.g as mentioned in here). I don't think it's good tho, it will propably introduce other bugs and take a lot of time (from user's viewpoint).
Is it possible to fix this issue?
EDIT:
I've dumped the stream data that is passed to decoding pipeline. I've changed the resolution few times while stream was being captured. Playing it with ffplay showed that in moment when resolution changed and preview in application froze, ffplay managed to continue, but preview is glitchy for a second or so. You can see full ffplay log here. In this case video preview stopped when I changed resolution to 960x720 for the second time. (Reinit context to 960x720, pix_fmt: yuv420p in log).
I'm trying to implement HW-accelrated H264 video encoding on Android ICS 4.0.4. Since MediaCodec class is not available I have to use stagefright API. But when I put HardwareCodecsOnly flag, OMXCodec::Create always returns NULL.
If I call OMXCodec::findMatchingCodecs() with flag kHardwareCodecsOnly, I got following list:
- OMX.TI.DUCATI1.VIDEO.H264E
- OMX.qcom.7x30.video.encoder.avc
- OMX.qcom.video.encoder.avc
- OMX.TI.Video.encoder
- OMX.Nvidia.h264.encoder
- OMX.SEC.AVC.Encoder
so I guess it means that HW-encoding supported by hardware.
When I put no flags in OMXCodec::Create - codec created well, but I guess it is in software mode
(btw, how can I check- which codec exactly was created?)
Browsing OMXCodec sources I've found interesting lines:
if (createEncoder) {
sp<MediaSource> softwareCodec =
InstantiateSoftwareEncoder(componentName, source, meta);
if (softwareCodec != NULL) {
LOGV("Successfully allocated software codec '%s'", componentName);
return softwareCodec;
}
}
it looks like for Encoder it always tries to instance Software codec first.
What am I doing wrong? Any help wil be greatly appreciated. Thanks
Here's a code of OMXCodec creation:
mClient = new OMXClient();
mClient->connect();
logger->log("mClient.connect();");
enc_meta = new MetaData;
// frame size of target video file
int width = 640; //720;
int height = 480;
int kFramerate = 15;
int kVideoBitRate = 500000;
int kIFramesIntervalSec = 5;
int32_t colorFormat = OMX_COLOR_FormatYUV420SemiPlanar;
enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_AVC); //MEDIA_MIMETYPE_VIDEO_MPEG4); //MEDIA_MIMETYPE_VIDEO_H263);//MEDIA_MIMETYPE_VIDEO_AVC);
enc_meta->setInt32(kKeyWidth, width);
enc_meta->setInt32(kKeyHeight, height);
enc_meta->setInt32(kKeyFrameRate, kFramerate);
enc_meta->setInt32(kKeySampleRate, 44100);
enc_meta->setInt32(kKeyBitRate, kVideoBitRate);
enc_meta->setInt32(kKeyStride, width);
enc_meta->setInt32(kKeySliceHeight, height);
enc_meta->setInt32(kKeyIFramesInterval, kIFramesIntervalSec);
enc_meta->setInt32(kKeyColorFormat, colorFormat);
mVideoSource = OMXCodec::Create(
mClient->interface(),
enc_meta,
true,
mSrc,
NULL,
OMXCodec::kHardwareCodecsOnly );
logger->log("OMXCodec_CREATED result: %d", (mVideoSource!=NULL) ? 1 : 0);
In Android ICS 4.0.4, the codec registration was static i.e. all codecs were registered as part of an array KEncoderInfo as can be found here.
The methodology to differentiate between hardware and software codecs is pretty simple. If the component name doesn't start with OMX, then it is construed to be a software codec as shown in theIsSoftwareCodec method.
Since you are trying an AVC encoder, the software codec if created would be AVCEncoder as can be found from it's Factory reference.
To check which codec was created, you can enable logs in OMXCodec.cpp file by removing the comment as #define LOG_NDEBUG 0 in this line, save and recompile to build libstagefright.so which could be used to generate the logs on logcat screen.
EDIT:
In case of rtsp streaming, one needs to enable the logs in ACodec.cpp .
One needs to ascertain if libstagefrighthw.so is present in /system/lib which will register the OMX core with the Stagefright framework.