I am decoding a h264 video stream with the following code (original guide):
public void configure(Surface surface, int width, int height, ByteBuffer csd0) {
String VIDEO_FORMAT = "video/avc";
if (mConfigured) {
throw new IllegalStateException("Decoder is already configured");
}
MediaFormat format = MediaFormat.createVideoFormat(VIDEO_FORMAT, width, height);
// little tricky here, csd-0 is required in order to configure the codec properly
// it is basically the first sample from encoder with flag: BUFFER_FLAG_CODEC_CONFIG
format.setByteBuffer("csd-0", csd0);
try {
mCodec = MediaCodec.createDecoderByType(VIDEO_FORMAT);
} catch (IOException e) {
throw new RuntimeException("Failed to create codec", e);
}
mCodec.configure(format, surface, null, 0);
mCodec.start();
mConfigured = true;
}
#SuppressWarnings("deprecation")
public void decodeSample(byte[] data, int offset, int size, long presentationTimeUs, int flags) {
if (mConfigured && mRunning) {
int index = mCodec.dequeueInputBuffer(mTimeoutUs);
if (index >= 0) {
ByteBuffer buffer;
// since API 21 we have new API to use
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.LOLLIPOP) {
buffer = mCodec.getInputBuffers()[index];
buffer.clear();
} else {
buffer = mCodec.getInputBuffer(index);
}
if (buffer != null) {
buffer.put(data, offset, size);
mCodec.queueInputBuffer(index, 0, size, presentationTimeUs, flags);
}
}
}
}
#Override
public void run() {
try {
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
while (mRunning) {
if (mConfigured) {
int index = mCodec.dequeueOutputBuffer(info, mTimeoutUs);
if (index >= 0) {
// setting true is telling system to render frame onto Surface
mCodec.releaseOutputBuffer(index, true);
if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
break;
}
}
} else {
// just waiting to be configured, then decode and render
try {
Thread.sleep(10);
} catch (InterruptedException ignore) {
}
}
}
} finally {
if (mConfigured) {
mCodec.stop();
mCodec.release();
}
}
}
I can run this on both my Nexus 6 (api 22) and Samsung galaxy core (api 16) on low and medium quality. However when I switch to high quality (720p) it crashes on the Samsung after about 30 frames (but nothing is rendered to the screen).
E/ACodec﹕ [OMX.qcom.video.decoder.avc] ERROR(0x8000100a)
E/MediaCodec﹕ Codec reported an error. (omx error 0x8000100a, internalError -2147483648)
[...]
W/System.err﹕ java.lang.IllegalStateException
W/System.err﹕ at android.media.MediaCodec.dequeueInputBuffer(Native Method)
W/System.err﹕ at com.test.stream.VideoDecoder$Worker.decodeSample(VideoDecoder.java:95)
W/System.err﹕ at com.test.stream.VideoDecoder.decodeSample(VideoDecoder.java:24)
W/System.err﹕ at com.test.stream.VideoThread.run(VideoThread.java:160)
The error above is the first error that appears, the IllegalStateException is afterwards thrown on each frame.
My question is, is this a device specific problem (because of: older api/device, less powerful, etc.) or is something actually wrong?
and how should I deal with this?
For my Android h.264 decoder i do it slightly different to your setup. I think your using more modern api level. But for me it looks more like this:
public void startDecoder() {
// Initilize codec
mediaCodec = MediaCodec.createDecoderByType("video/avc");
mediaFormat = MediaFormat.createVideoFormat("video/avc", 0, 0);
bufferInfo = new MediaCodec.BufferInfo();
// STOPS unit-tests from crashing here from mocked out android
if (mediaCodec != null) {
mediaCodec.configure(mediaFormat, targetSurface, null, 0);
mediaCodec.start();
decoderThread = new Thread(this);
decoderThread.start();
}
}
// Decoder Thread refers to this class which does the decoder/render loop:
public void run() {
//mediaCodec input + output dequeue timeouts
long kInputBufferTimeoutMs = 50;
long kOutputBufferTimeoutMs = 50;
while (running && mediaCodec != null) {
synchronized (mediaCodec) {
// stop if not running.
if (!running || mediaCodec == null)
break;
// Only push in new data if there is data available in the queue
if (naluSegmentQueue.size() > 0) {
int inputBufferIndex = mediaCodec.dequeueInputBuffer(kInputBufferTimeoutMs);
if (inputBufferIndex >= 0) {
NaluSegment segment = naluSegmentQueue.poll();
codecInputBufferAvailable(segment, mediaCodec, inputBufferIndex);
}
}
// always check if output is available.
int outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, kOutputBufferTimeoutMs);
if (outputBufferIndex >= 0) {
// Try and render first
codecOuputBufferAvailable(mediaCodec, outputBufferIndex, bufferInfo);
} else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// Subsequent data will conform to new format.
// Can ignore if using getOutputFormat(outputBufferId)
mediaFormat = mediaCodec.getOutputFormat();
}
}
}
}
To put data into the decoder including the parameters. I don't bother with trying to use the csd-0/1 network streams can have changing format descriptions and its easier to just let it be picked up dynamically.
private void codecInputBufferAvailable(NaluSegment segment, MediaCodec codec, int index) {
int flags = (segment.getType() == NaluType.SPS
|| segment.getType() == NaluType.PPS
|| segment.getType() == NaluType.SUPP_ENHANCEMENT) ?
MediaCodec.BUFFER_FLAG_CODEC_CONFIG : MediaCodec.BUFFER_FLAG_SYNC_FRAME;
ByteBuffer[] buffers = codec.getInputBuffers();
ByteBuffer buffer = buffers[index];
// Can throw buffer overflow exception when buffer sizes are too small.
try {
buffer.put(segment.getBuffer());
codec.queueInputBuffer(index, 0, segment.getBufferSize(), 0, flags);
} catch(Exception e) {
Log.e(TAG, "Failed to push buffer to decoder");
}
}
IMPORTANT: buffer.put(segment.getBuffer());
getBuffer() here always returns a 4 byte annexb buffer. The android decoders do not understand 3 byte nal units. So if you have a 3 byte nal unit turn it into 4 bytes magic sequence with length + 1 and 0x00, 0x00, 0x00, 0x01 as the start magic sequence the rest of the buffer should be &buffer[headerLength].
Notice the try-catch here this doesn't give a compiler warning but it can throw a buffer overflow exception here if your have a very big payload and the byte-buffer is too small.
So long as your parse out your NAL units correctly this should work for you. But for my case i noticed that the NAL units can be 3 or 4 bytes for the magic header.
/**
* H264 is comprised of NALU segments.
*
* XXXX Y ZZZZZZZZ -> XXXX Y ZZZZZZZZ -> XXXX Y ZZZZZZZZ
*
* Each segment is comprised of:
*
* XXXX -> Magic byte header (0x00, 0x00, 0x00, 0x01) NOTE: this can be either 3 of 4 bytes
* Y -> The Nalu Type
* ZZZ... -> The Payload
*
* Notice there is no nalu length specified. To parse an nalu, you must
* read until the next magic-byte-sequence AKA the next segment to figure
* out the full nalu length
**/
public static List<NaluSegment> parseNaluSegments(byte[] buffer) throws NaluBufferException {
List<NaluSegment> segmentList = new ArrayList<>();
if (buffer.length < 6) {
return segmentList;
}
int lastStartingOffset = -1;
for (int i = 0; i < buffer.length - 10; ++i) {
**if (buffer[i] == 0x00 && buffer[i+1] == 0x00 && buffer[i+2] == 0x01)** {
int naluType = (buffer[i+3] & 0x1F);
NaluSegment segment = new NaluSegment(naluType, 3, i);
**if (i > 0 && buffer[i-1] == 0x00)** {
// This is actually a 4 byte segment
int currentSegmentOffset = segment.getOffset();
segment.setHeaderSize(4);
segment.setOffset(currentSegmentOffset - 1);
}
...
Create your own nalu-segment objects and don't forget the trailing NAL.
I hope this helps.
Related
I'm doing a transcoder using MediaCodec.
I created two mediacodec instance, one is for decoding and another is for encoding. I'm trying to send decoders outputBuffer directly into encoders inputBuffer.
It seems has no problem while compiling and executing.And it runs quickly.
But the output video file has something wrong.I checked the metadata of the output video and they are all right : bitrate, framerate, resolution ...Only the images in the video is wrong like this:screen shot
I thought it has somethings wrong,but I cannot figure it out...
I searched libraries and documents, and I found some sample codes using Texture surface to render the decoder output data and tranfer the data into the encoder. But I thought it should not be neccessary for me. Because I dont need to edit images of the video.What I only need to do is changing the bitrate and resolution to make the file's size smaller.
here is the core code in my project:
private void decodeCore() {
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
int frameCount = 0;
while (mDecodeRunning) {
int inputBufferId = mDecoder.dequeueInputBuffer(50);
if (inputBufferId >= 0) {
// fill inputBuffers[inputBufferId] with valid data
int sampleSize = mExtractor.readSampleData(mDecodeInputBuffers[inputBufferId], 0);
if (sampleSize >= 0) {
long time = mExtractor.getSampleTime();
mDecoder.queueInputBuffer(inputBufferId, 0, sampleSize, time, 0);
} else {
mDecoder.queueInputBuffer(inputBufferId, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
}
mExtractor.advance();
}
int outputBufferId = mDecoder.dequeueOutputBuffer(bufferInfo, 50);
if (outputBufferId >= 0) {
FrameData data = mFrameDataQueue.obtain();
//wait until queue has space to push data
while (data == null) {
try {
Thread.sleep(20);
} catch (InterruptedException e) {
e.printStackTrace();
}
data = mFrameDataQueue.obtain();
}
data.data.clear();
data.size = 0;
data.offset = 0;
data.flag = 0;
data.frameTimeInUs = bufferInfo.presentationTimeUs;
// outputBuffers[outputBufferId] is ready to be processed or rendered.
if (bufferInfo.size > 0) {
ByteBuffer buffer = mDecodeOutputBuffers[outputBufferId];
buffer.position(bufferInfo.offset);
buffer.limit(bufferInfo.offset + bufferInfo.size);
data.data.put(buffer);
data.data.flip();
data.size = bufferInfo.size;
data.frameIndex = frameCount++;
}
data.flag = bufferInfo.flags;
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
Log.d("bingbing_transcode", "decode over! frames:" + (frameCount - 1));
mDecodeRunning = false;
}
mFrameDataQueue.pushToQueue(data);
mDecoder.releaseOutputBuffer(outputBufferId, false);
Log.d("bingbing_transcode", "decode output:\n frame:" + (frameCount - 1) + "\n" + "size:" + bufferInfo.size);
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
mDecodeOutputBuffers = mDecoder.getOutputBuffers();
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// Subsequent data will conform to new format.
mDecodeOutputVideoFormat = mDecoder.getOutputFormat();
configureAndStartEncoder();
}
}
mDecoder.stop();
mDecoder.release();
}
private void encodeCore() {
int trackIndex = 0;
boolean muxerStarted = false;
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
int frameCount = 0;
while (mEncodeRunning) {
int inputBufferId = mEncoder.dequeueInputBuffer(50);
if (inputBufferId >= 0) {
FrameData data = mFrameDataQueue.pollFromQueue();
//wait until queue has space to push data
while (data == null) {
try {
Thread.sleep(20);
} catch (InterruptedException e) {
e.printStackTrace();
}
data = mFrameDataQueue.obtain();
}
if (data.size > 0) {
ByteBuffer inputBuffer = mEncodeInputBuffers[inputBufferId];
inputBuffer.clear();
inputBuffer.put(data.data);
inputBuffer.flip();
}
mEncoder.queueInputBuffer(inputBufferId, 0, data.size, data.frameTimeInUs, data.flag);
mFrameDataQueue.recycle(data);
}
int outputBufferId = mEncoder.dequeueOutputBuffer(bufferInfo, 50);
if (outputBufferId >= 0) {
// outputBuffers[outputBufferId] is ready to be processed or rendered.
ByteBuffer encodedData = mEncodeOutputBuffers[outputBufferId];
if (bufferInfo.size > 0) {
if (encodedData == null) {
throw new RuntimeException("encoderOutputBuffer " + outputBufferId + " was null");
}
if (!muxerStarted) {
throw new RuntimeException("muxer hasn't started");
}
frameCount++;
}
// adjust the ByteBuffer values to match BufferInfo (not needed?)
encodedData.position(bufferInfo.offset);
encodedData.limit(bufferInfo.offset + bufferInfo.size);
mMuxer.writeSampleData(trackIndex, encodedData, bufferInfo);
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
Log.d("bingbing_transcode", "encode over! frames:" + (frameCount - 1));
mEncodeRunning = false;
}
mEncoder.releaseOutputBuffer(outputBufferId, false);
Log.d("bingbing_transcode", "encode output:\n frame:" + (frameCount - 1) + "\n" + "size:" + bufferInfo.size);
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
mEncodeOutputBuffers = mEncoder.getOutputBuffers();
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// should happen before receiving buffers, and should only happen once
if (muxerStarted) {
throw new RuntimeException("format changed twice");
}
MediaFormat newFormat = mEncoder.getOutputFormat();
Log.d("bingbing_transcode", "encoder output format changed: " + newFormat);
// now that we have the Magic Goodies, start the muxer
trackIndex = mMuxer.addTrack(newFormat);
mMuxer.start();
muxerStarted = true;
mEncodeOutputVideoFormat = newFormat;
}
}
mEncoder.stop();
mEncoder.release();
if (muxerStarted) {
mMuxer.stop();
mMuxer.release();
}
}
these two functions run in two different threads.
FrameData is a simple storage of frame bytebuffer and frame present time and something needed
When using bytebuffer input, there are a few details that are undefined about the input data layout. When the width isn't a multiple of 16, some encoders want to have the input data row length padded to a multiple of 16, while others will assume a line length equal to the width, with no extra padding.
The Android CTS tests (which define what behaviour one can expect across all devices) for encoding from bytebuffer inputs intentionally only test resolutions that are a multiple of 16, since they know different hardware vendors do this differently, and they didn't want to enforce any particular handling.
You can't generally assume that the decoder output would use a similar row size as what the encoder consumes either. The decoder is free to (and some actually do) return a significantly larger width than the actual content size, and use the crop_left/crop_right fields for indicating what parts of it actually are intended to be visible. So in case the decoder did that, you can't pass the data straight from the decoder to the encoder unless you copy it line by line, taking into account the actual line sizes used by the decoder and encoder.
Additionally, you can't even assume that the decoder uses a similar pixel format as the encoder. Many qualcomm devices use a special tiled pixel format for the decoder output, while the encoder input is normal planar data. In these cases, you'd have to implement a pretty complex logic for unshuffling the data before you can feed it into the encoder.
Using a texture surface as intermediate hides all of these details. It might not sound completely necessary for your use case, but it does hide all the variation in buffer formats between decoder and encoder.
I am currently attempting to parse H264 data from an RTP stream and then send it to the MediaCodec to render on a SurfaceView for Android.
However, I'm not certain how to:
build the H264 slices properly from the RTP packets
send the H264 slices to the media codec once they are assembled into slices
I have not seen any examples of this implemented in a clear and concise way and I haven't found the MediaCodec docs to be at all helpful.
Anyone have any experience in this domain?
void videoCodec(ByteBuffer input, int flags) {
bufferInfo.set(0, 0, 0, flags);
int inputBufferId = codec.dequeueInputBuffer(10000);
if (inputBufferId >= 0) {
//put data
ByteBuffer inputData = inputBuffers[inputBufferId];
inputData.clear();
inputData.put(input);
//queue it up
codec.queueInputBuffer(inputBufferId, 0, input.limit(), 0, flags);
}
int outputBufferId = codec.dequeueOutputBuffer(bufferInfo, 10000);
if (outputBufferId >= 0) {
// outputBuffers[outputBufferId] is ready to be processed or rendered.
Timber.e("Rendering Data with Index of: %s", outputBufferId);
codec.releaseOutputBuffer(outputBufferId, true);
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
outputBuffers = codec.getOutputBuffers();
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// Subsequent data will conform to new format.
//format = codec.getOutputFormat();
}
}
MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, 1920, 1080);
codec = MediaCodec.createDecoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
codec.configure(format, surfaceVideo.getHolder().getSurface(), null, 0);
codec.start();
inputBuffers = codec.getInputBuffers();
outputBuffers = codec.getOutputBuffers();
while (streaming) {
//receive RTP Packet
h264Parser(rtpPacket.getPayload());
}
And the the h264Parser looks something like this:
void h264Parser(byte[] payload) {
int packetType = (byte) payload[0] & (byte) 0x1F;
boolean startBit = (payload[1] & 0x80) != 0;
boolean endBit = (payload[1] & 0x40) != 0;
int flags = 0;
switch (packetType) {
case 7:
pps = new ByteArrayOutputStream();
pps.write(prefix);
pps.write(payload);
break;
case 8:
if (pps.size() > 0) {
pps.write(payload);
hasPps = true;
flags = MediaCodec.BUFFER_FLAG_CODEC_CONFIG;
payload = pps.toByteArray();
//Send packet to decoder
videoCodec(ByteBuffer.wrap(payload), flags);
break;
case 28:
if (hasPps) {
if (startBit) {
baos = new ByteArrayOutputStream();
baos.write(prefix);
baos.write(payload);
} else if (endBit) {
if(baos != null) {
baos.write(payload);
flags = MediaCodec.BUFFER_FLAG_KEY_FRAME;
payload = baos.toByteArray();
//Send packet to decoder
videoCodec(ByteBuffer.wrap(payload), flags);
hasPps = false;
} else {
if(baos != null ) {
baos.write(payload);
}
}
}
break;
case 1:
break;
default:
}
As far as i remember MediaCodec uses full acess units, not only slices (someone correct me of i'm wrong)
So you have to build a complete acess unit form RTP and feed it to the decoder (sadly i have no experience with RTP and cannot help you with building one).
You send the access units to the decoder as follows:
Dequeue a inputbuffer
int inputBufferIndex = decoder.dequeueInputBuffer(TIMEOUT_USEC);
ByteBuffer inputBuffer = videoDecoderInputBuffers[videoInputBufIndex];
Fill it with your access unit
inputBuffer.put(acessUnit);
inputBuffer.flip();
Queue the buffer for decoding
decoder.queueInputBuffer(inputBufferIndex,0,inputBuffer.limit(), 0, FLAGS);
I hope this helps a little
I have application (Qt + Android), that creates live stream from Android's Camera (AVC) + AudioRecorder (AAC) and then sends encoded data to RTMP server using librtmp library (v 2.4).
AVC MediaCodec main func.:
public void videoEncode(byte[] data) {
// Video buffers
videoCodecInputBuffers = videoMediaCodec.getInputBuffers();
videoCodecOutputBuffers = videoMediaCodec.getOutputBuffers();
int inputBufferIndex = videoMediaCodec.dequeueInputBuffer(-1);
if (inputBufferIndex >= 0) {
videoInputBuffer = videoCodecInputBuffers[inputBufferIndex];
videoCodecInputData = YV12toYUV420Planar(data, encWidth * encHeight);
videoInputBuffer.clear();
videoInputBuffer.put(videoCodecInputData);
videoMediaCodec.queueInputBuffer(inputBufferIndex, 0, videoCodecInputData.length, 0, 0);
}
// Get AVC/H.264 frame
int outputBufferIndex = videoMediaCodec.dequeueOutputBuffer(videoBufferInfo, 0);
while(outputBufferIndex >= 0) {
videoOutputBuffer = videoCodecOutputBuffers[outputBufferIndex];
videoOutputBuffer.get(videoCodecOutputData, 0, videoBufferInfo.size);
// H.264 / AVC header
if(videoCodecOutputData[0] == 0x00 && videoCodecOutputData[1] == 0x00 && videoCodecOutputData[2] == 0x00 && videoCodecOutputData[3] == 0x01) {
// I-frame
boolean keyFrame = false;
if((videoBufferInfo.flags & MediaCodec.BUFFER_FLAG_SYNC_FRAME) == MediaCodec.BUFFER_FLAG_SYNC_FRAME) {
resetTimestamp();
keyFrame = true;
}
int currentTimestamp = cameraAndroid.calcTimestamp();
if(prevTimestamp == currentTimestamp) currentTimestamp++;
sendVideoData(videoCodecOutputData, videoBufferInfo.size, currentTimestamp, cameraAndroid.calcTimestamp()); // Native C func
prevTimestamp = currentTimestamp;
// SPS / PPS sent
spsPpsFrame = true;
}
videoMediaCodec.releaseOutputBuffer(outputBufferIndex, false);
outputBufferIndex = videoMediaCodec.dequeueOutputBuffer(videoBufferInfo, 0);
}
}
AAC MediaCodec main func.:
public void audioEncode(byte[] data) {
// Audio buffers
audioCodecInputBuffers = audioMediaCodec.getInputBuffers();
audioCodecOutputBuffers = audioMediaCodec.getOutputBuffers();
// Add raw chunk into buffer
int inputBufferIndex = audioMediaCodec.dequeueInputBuffer(-1);
if (inputBufferIndex >= 0) {
audioInputBuffer = audioCodecInputBuffers[inputBufferIndex];
audioInputBuffer.clear();
audioInputBuffer.put(data);
audioMediaCodec.queueInputBuffer(inputBufferIndex, 0, data.length, 0, 0);
}
// Encode AAC
int outputBufferIndex = audioMediaCodec.dequeueOutputBuffer(audioBufferInfo, 0),
audioOutputBufferSize = 0;
while(outputBufferIndex >= 0) {
audioOutputBuffer = audioCodecOutputBuffers[outputBufferIndex];
audioOutputBuffer.get(audioCodecOutputData, 0, audioBufferInfo.size);
if(spsPpsFrame || esdsChunk) {
int currentTimestamp = cameraAndroid.calcTimestamp();
if(prevTimestamp == currentTimestamp) currentTimestamp++;
sendAudioData(audioCodecOutputData, audioBufferInfo.size, currentTimestamp); // Native C func
prevTimestamp = currentTimestamp;
esdsChunk = false;
}
// Next chunk
audioMediaCodec.releaseOutputBuffer(outputBufferIndex, false);
outputBufferIndex = audioMediaCodec.dequeueOutputBuffer(audioBufferInfo, 0);
}
}
Camera frames encoded in setPreviewCallbackWithBuffer and AudioRecorder's chunks in other thread:
audioThread = new Thread(new Runnable() {
public void run() {
audioBufferSize = AudioRecord.getMinBufferSize(44100, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT);
while(!audioThread.interrupted()) {
int ret = mic.read(audioCodecInputData, 0, audioBufferSize);
if(ret >= 0)
cameraAndroid.audioEncode(audioCodecInputData);
}
}
});
sendVideoData and sendAudioData are native C functions (librtmp func-s + JNI):
public synchronized native void sendVideoData(byte[] buf, int size, int timestamp, boolean keyFrame);
public synchronized native void sendAudioData(byte[] buf, int size, int timestamp);
The main thing, that I can't understood is: why live stream is absolutely unstable, when I playing them from Adobe Flash Player?
First 1-2 seconds of stream is absolutely correct, but then I always see I-frames every 2 seconds (videoMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 2)) and very bad sound stream, that I can hear for milliseconds during I-frame interval and then it interrupts.
Can someone show to me correct way for creating stable live stream, please? Where I'm wrong?
Also, I post here AVC/AAC MediaCodec settings (may be something wrong here?):
// H.264/AVC (advanced video coding) format
MediaFormat videoMediaFormat = MediaFormat.createVideoFormat("video/avc", encWidth, encHeight);
videoMediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar);
videoMediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, encWidth * encHeight * 4); // бит в секунду
videoMediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, fps); // FPS
videoMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, iFrameInterval); // interval секунд между I-frames
videoMediaCodec = MediaCodec.createEncoderByType("video/avc");
videoMediaCodec.configure(videoMediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
// AAC (advanced audio coding) format
MediaFormat audioMediaFormat = MediaFormat.createAudioFormat("audio/mp4a-latm", 44100, 1); // mime-type, sample rate, channel count
audioMediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 64 * 1000); // kbps
audioMediaFormat.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC);
audioMediaFormat.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, audioBufferSize); // 4096 (default) / 4736 * 1 (min audio buffer size)
audioMediaCodec = MediaCodec.createEncoderByType("audio/mp4a-latm");
audioMediaCodec.configure(audioMediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
Update:
I tried to play stream with ffmpeg (thanks #Robert Rowntree) and what I see in console constantly:
Non-monotonous DTS in output stream 0:1; previous: 95054, current:
46136; changing to 95056. This may result in incorrect timestamps in
the output file.
So, I check output from android app, but I can't see wrong lines (a - encoded AAC chunk, v - encoded AVC frame, integer value - timestamp in milliseconds): output.txt
Is that correct timestamps?
I'm trying to use the Android MediaCodec for decoding NAL units. The problem is that the video is played with a delay of 2-3 seconds, as the first I-Frame is played only after receiving the next one (the time difference between I-Frames is 2-3 seconds). So if I put the same I-frame 2 times in a row then it will play without any delay. But in this case I can't play the rest frames.
I do not understand what the problem could be. I have tried to fix it on my own, but I can't.
public void play(MediaCodec decoder, PESPacket currentPES){
byte[] data = currentPES.data.toByteArray();
pts = currentPES.getPts();
try {
if (data[4] == 0x67) {
Log.d(TAG, "found sps/pps!");
int ibs = decoder.dequeueInputBuffer(0);
if (ibs >= 0) {
ByteBuffer buffer = decoder.getInputBuffers()[ibs];
buffer.clear();
buffer.put(data);
decoder.queueInputBuffer(ibs, 0, data.length, pts, MediaCodec.BUFFER_FLAG_CODEC_CONFIG);
}
} else {
if (data[4] == 0x41) { // not I-frame
int ibs = decoder.dequeueInputBuffer(0);
if (ibs >= 0) {
System.out.println("queueInputBuffer little");
ByteBuffer buffer = decoder.getInputBuffers()[ibs];
buffer.clear();
buffer.put(data);
decoder.queueInputBuffer(ibs, 0, data.length, pts, 0);
}
}
if (data[4] == 0x65) { // I-frame
int ibs = decoder.dequeueInputBuffer(0);
if (ibs >= 0) {
System.out.println("queueInputBuffer");
ByteBuffer buffer = decoder.getInputBuffers()[ibs];
buffer.clear();
buffer.put(data);
decoder.queueInputBuffer(ibs, 0, data.length, pts, 0);
}
}
}
int outputBufferIndex = decoder.dequeueOutputBuffer(bufferInfo, 10000);
if (outputBufferIndex >= 0) {
System.out.println("releaseOutputBuffer " + outputBufferIndex);
decoder.releaseOutputBuffer(outputBufferIndex, true);
} else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
System.out.println("INFO_OUTPUT_BUFFERS_CHANGED ");
} else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat format = decoder.getOutputFormat();
System.out.println("INFO_OUTPUT_FORMAT_CHANGED " + format);
}
} catch (Throwable t) {
t.printStackTrace();
}
}
It is maybe too late, but wanted to share some info.
If only send IDR frames to video decoder, you can try set "android._num-input-buffers" and "android._num-output-buffers" to 1 in the MediaFormat when configure the decoder. This will try to allocate single buffer in video decoder, thus output buffer immediately. However, the behaviour is not guaranteed, will depends on decoder's implementation.
I'm guessing, Google's software H264 decoder, "omx.google.h264.decoder" should support this behaviour, but I have not tested.
Refer to Android source code FrameDecoder.cpp as following for how to set the parameters. The FrameDecoder.cpp is the framework layer implementation of Android API MetaDataExtractor.getFrameAt().
http://androidxref.com/9.0.0_r3/xref/frameworks/av/media/libstagefright/FrameDecoder.cpp#411
451 // For the thumbnail extraction case, try to allocate single buffer in both
452 // input and output ports, if seeking to a sync frame. NOTE: This request may
453 // fail if component requires more than that for decoding.
454 bool isSeekingClosest = (mSeekMode == MediaSource::ReadOptions::SEEK_CLOSEST)
455 || (mSeekMode == MediaSource::ReadOptions::SEEK_FRAME_INDEX);
456 if (!isSeekingClosest) {
457 videoFormat->setInt32("android._num-input-buffers", 1);
458 videoFormat->setInt32("android._num-output-buffers", 1);
459 }
I've a problem with this library
https://github.com/fyhertz/libstreaming
it allows to send via wireless the streaming of photocamera, it use 3 methods: two with mediacodec and one with mediarecorder.
I would like to modify it, and I have to use only the mediacodec;however first of all I tried the code of the example 2 of the library, but I've always found the same error:
the log tell me that the device can use the mediacodec, it set the encoder and when it test the decoder it fall and the buffer is filled with -1.
This is the method in the EncoderDebugger class where the exception occurs, some kind soul can help me please?
private long decode(boolean withPrefix) {
int n =3, i = 0, j = 0;
long elapsed = 0, now = timestamp();
int decInputIndex = 0, decOutputIndex = 0;
ByteBuffer[] decInputBuffers = mDecoder.getInputBuffers();
ByteBuffer[] decOutputBuffers = mDecoder.getOutputBuffers();
BufferInfo info = new BufferInfo();
while (elapsed<3000000) {
// Feeds the decoder with a NAL unit
if (i<NB_ENCODED) {
decInputIndex = mDecoder.dequeueInputBuffer(1000000/FRAMERATE);
if (decInputIndex>=0) {
int l1 = decInputBuffers[decInputIndex].capacity();
int l2 = mVideo[i].length;
decInputBuffers[decInputIndex].clear();
if ((withPrefix && hasPrefix(mVideo[i])) || (!withPrefix && !hasPrefix(mVideo[i]))) {
check(l1>=l2, "The decoder input buffer is not big enough (nal="+l2+", capacity="+l1+").");
decInputBuffers[decInputIndex].put(mVideo[i],0,mVideo[i].length);
} else if (withPrefix && !hasPrefix(mVideo[i])) {
check(l1>=l2+4, "The decoder input buffer is not big enough (nal="+(l2+4)+", capacity="+l1+").");
decInputBuffers[decInputIndex].put(new byte[] {0,0,0,1});
decInputBuffers[decInputIndex].put(mVideo[i],0,mVideo[i].length);
} else if (!withPrefix && hasPrefix(mVideo[i])) {
check(l1>=l2-4, "The decoder input buffer is not big enough (nal="+(l2-4)+", capacity="+l1+").");
decInputBuffers[decInputIndex].put(mVideo[i],4,mVideo[i].length-4);
}
mDecoder.queueInputBuffer(decInputIndex, 0, l2, timestamp(), 0);
i++;
} else {
if (VERBOSE) Log.d(TAG,"No buffer available !7");
}
}
// Tries to get a decoded image
decOutputIndex = mDecoder.dequeueOutputBuffer(info, 1000000/FRAMERATE);
if (decOutputIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
decOutputBuffers = mDecoder.getOutputBuffers();
} else if (decOutputIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
mDecOutputFormat = mDecoder.getOutputFormat();
} else if (decOutputIndex>=0) {
if (n>2) {
// We have successfully encoded and decoded an image !
int length = info.size;
mDecodedVideo[j] = new byte[length];
decOutputBuffers[decOutputIndex].clear();
decOutputBuffers[decOutputIndex].get(mDecodedVideo[j], 0, length);
// Converts the decoded frame to NV21
convertToNV21(j);
if (j>=NB_DECODED-1) {
flushMediaCodec(mDecoder);
if (VERBOSE) Log.v(TAG, "Decoding "+n+" frames took "+elapsed/1000+" ms");
return elapsed;
}
j++;
}
mDecoder.releaseOutputBuffer(decOutputIndex, false);
n++;
}
elapsed = timestamp() - now;
}
throw new RuntimeException("The decoder did not decode anything.");
}
Here's my suggestions:
(1) check the settings of encoder and decoder, and make sure that they match. For example, revolution and color format are the same.
(2) make sure the very first packet generated by the encoder has been sent and pushed into the decoder. This packet defines the basic settings of the video stream.
(3) the decoder usually buffers 5-10 frames. So data in the buffer is invalid for a few hundred ms.
(4) while initiating the decoder, set the surface as null. Otherwise the output buffer will be read by the surface and probably released automatically.