Develop Reference

Decode H.264 stream using MediaCodec API JNI

I am developing a H.264 decoder using MediaCodec API. I am trying to call MediaCodec java API in JNI layer inside a function like:
void Decompress(const unsigned char *encodedInputdata, unsigned int inputLength, unsigned char **outputDecodedData, int &width, int &height) {
// encodedInputdata is encoded H.264 remote stream
// .....
// outputDecodedData = call JNI function of MediaCodec Java API to decode
// .....
}
Later I will send the outputDecodedData to my existing video rendering pipeline and render on Surface.
I hope I will be able to write a Java function to decode the input stream, but these would be challenge -
This resource states that -
...you can't do anything with the decoded video frame but render them
to surface
Here a Surface has been passed decoder.configure(format, surface, null, 0) to render the output ByteBuffer on the surface and claimed We can't use this buffer but render it due to the API limit.
So, will I able to send the output ByteBuffer to native layer to cast as unsigned char* and pass to my rendering pipeline instead of passing a Surface ot configure()?

I see two fundamental problems with your proposed function definition.
First, MediaCodec operates on access units (NAL units for H.264), not arbitrary chunks of data from a stream, so you need to pass in one NAL unit at a time. Once the chunk is received, the codec may want to wait for additional frames to arrive before producing any output. You cannot in general pass in one frame of input and wait to receive one frame of output.
Second, as you noted, the ByteBuffer output is YUV-encoded in one of several color formats. The format varies from device to device; Qualcomm devices notably use their own proprietary format. (It has been reverse-engineered, though, so if you search around you can find some code to unravel it.)
The common workaround is to send the video frames to a SurfaceTexture, which converts them to GLES "external" textures. These can be manipulated in various ways, or rendered to a pbuffer and extracted with glReadPixels().

MediaCodec is giving a storeMetaDataInBuffers trace error

I'm getting on the logcat the next error while encoding via the MediaCodec in Android.
The actual encoding works fine and the output is produced correctly, so I can't really understand why I get this trace. Is it a harmless error trace, or is there something I'm missing?
E/ACodec(6438): [OMX.qcom.video.encoder.h263] storeMetaDataInBuffers (output) failed w/ err -1010
Next is the code where I get the trace
final int BIT_RATE = 4000000;
final int FRAME_RATE = 30;
final int IFRAME_INTERVAL = 5;
final String MIME_TYPE = "video/avc";
final MediaFormat format = MediaFormat.createVideoFormat(MIME_TYPE, width, height);
format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
format.setInteger(MediaFormat.KEY_BIT_RATE, BIT_RATE);
format.setInteger(MediaFormat.KEY_FRAME_RATE, FRAME_RATE);
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, IFRAME_INTERVAL);
MediaCodec encoder = MediaCodec.createEncoderByType(MIME_TYPE);
//---------------------------------
// NEXT LINE PRODUCES THE TRACE
encoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
//---------------------------------

It's harmless, most devices show this. See Q12 at http://bigflake.com/mediacodec/.
This only tells that the first way of signaling surface encoding wasn't supported by the encoder, so it used some other ways of setting it up. (There are multiple ways for the MediaCodec/ACodec layer to tell the individual encoder about it.)

The previous answer has indicated that the warning is quite harmless. Some additional information on the log and reasons behind the same
This trace in the log is indicating that the encoder is not supporting storeMetadataInBuffers on the output port. For an encoder, this mode could be supported on both input and output ports.
This mode is employed for input port to pass raw image data in metadata format i.e. pass only a reference to the gralloc handle which can employed by the encoder to access the data. This is employed by the camera and/or other screen recording applications to pass a reference to YUV data to the encoder.
The metadata mode was supported for output port also for potential encapsulation of output bitstream data. For example, when a Miracast or WiFi-Display session is active and the data being encoded is secure like a premium content, it becomes necessary to protect data between the encoder and HDCP encryption module, during which metadata format becomes handy. Not many encoders support this mode and hence, you observe this warning.

Problems of using MediaCodec.getOutputFormat() for an encoder in Android 4.1/4.2 devices

I'm trying to use MediaCodec to encode frames (either by camera or decoder) into a video.
When processing the encoder output by dequeueOutputBuffer(), I expect to receive the return index = MediaCodec.INFO_OUTPUT_FORMAT_CHANGED, so I can call getOutputFormat() to get the encoder output format as the input of the currently used ffmpeg muxer.
I have tested some pad/phone devices with Android version 4.1~4.3. All of them have at least one hardware video AVC encoder and is used in the test. On the devices with version 4.3, the encoder gives MediaCodec.INFO_OUTPUT_FORMAT_CHANGED before writing the encoded data as expected, and the output format returned from getOutputFormat() can be used by the muxer correctly. On the devices with 4.2.2 or lower, the encoder never gives MediaCodec.INFO_OUTPUT_FORMAT_CHANGED while it can still output the encoded elementary stream, but the muxer cannot know the exact output format.
I want to ask the following questions:
Does the behavior of encoder (gives MediaCodec.INFO_OUTPUT_FORMAT_CHANGED or not before outputing encoded data) depend on the Android API Level or the chips on individual devices?
If the encoder writes data before MediaCodec.INFO_OUTPUT_FORMAT_CHANGED appears, is there any way to get the output format of the encoded data?
The encoder still output the codec config data (with flag MediaCodec.BUFFER_FLAG_CODEC_CONFIG) on the devices before the encoded data. It is mostly used to config a decoder, but can I derive the output format by the codec config data?
I have tried these solutions to get the output format but failed:
Call getOutputFormat() frequently during the whole encode process. However, all of them throw IllegalStateException without the appearance of MediaCodec.INFO_OUTPUT_FORMAT_CHANGED.
Use the initial MediaFormat use to config the encoder at the beginning, like the example:
m_init_encode_format = MediaFormat.createVideoFormat(m_encode_video_mime, m_frame_width, m_frame_height);
int encode_bit_rate = 3000000;
int encode_frame_rate = 15;
int encode_iframe_interval = 2;
m_init_encode_format.setInteger(MediaFormat.KEY_COLOR_FORMAT, m_encode_color_format);
m_init_encode_format.setInteger(MediaFormat.KEY_BIT_RATE, encode_bit_rate);
m_init_encode_format.setInteger(MediaFormat.KEY_FRAME_RATE, encode_frame_rate);
m_init_encode_format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, encode_iframe_interval);
m_encoder = MediaCodec.createByCodecName(m_video_encoder_codec_info.getName());
m_encoder.configure(m_init_encode_format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
// Assume m_init_encode_format is the output format of the encoder
However it fails since the output format of the encoder is still "changed" from the initial one.
Please help me to realize the behavior of an encoder, and if there is any solution to query the output format if the required MediaCodec.INFO_OUTPUT_FORMAT_CHANGED is missing.
By comparing the output format and the codec config data, the missing fields are csd-0, csd-1, and a "what" field with value = 1869968451.
(I do not understand the "what" field. It seems to be a constant and is not required. Can anyone tell me about its meaning?)
If I parse the codec config data as the csd-1 field (last 8 bytes) and csd-0 field (remaining bytes), it seems that the muxer can work correctly and output a video playable on all of the testing devices.
(But I want to ask: is this 8-byte assumption correct, or there is more reliable way to parse the data?)
However, I got another problem that If I decode the video by Android MediaCodec again, the BufferInfo.presentationTimeUs value get by dequeueOutputBuffer() is 0 for most of the decoded frames. Only the last few frames has correct time. The sample time get by MediaExtractor.getSampleTime() is correct and exactly the value I set to the encoder/muxer, but the decoded frame time is not. This issue only happen on 4.2.2 or lower device.
It is strange that the frame time is incorrect but the video can be playback in correct speed on the device. (Most of the devices with 4.2.2 or lower I've tested has only 1 Video AVC decoder.) Do I need to set other fields that may affect the presentation time?

The behavior of MediaCodec encoders was changed in Android 4.3 to accommodate the introduction of the MediaMuxer class. In Android 4.3, you will always receive INFO_OUTPUT_FORMAT_CHANGED from the encoder. In previous releases, you will not. (I've updated the relevant FAQ entry.)
There is no way to query the encoder for the MediaFormat.
I haven't used an ffmpeg-based muxer, so I'm not sure what information it needs. If it's looking for the csd-0 / csd-1 keys, you can extract those from the CODEC_CONFIG packet (I think you have to parse the SPS / PPS values out and place them in the separate keys). Examining the contents of the MediaFormat on a 4.3 device will show you which fields you're lacking.

To init ffmpeg muxer for video correctly i use next:
int outputBufferIndex = videoCodec.dequeueOutputBuffer(bufferInfo, -1);
if (MediaCodec.BUFFER_FLAG_CODEC_CONFIG == bufferInfo.flags) {
ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
headerData = new byte[bufferInfo.size];
outputBuffer.get(headerData);
// jni call
WriteVideoHeader(headerData, headerData.length);
videoCodec.releaseOutputBuffer(outputBufferIndex, false);
}
In jni I use something like this:
jint Java_com_an_FileWriterEx_WriteVideoHeader(JNIEnv * env, jobject this, jbyteArray data, jint datasize)
{
jboolean isCopy;
jbyte* rawjBytes = (*env)->GetByteArrayElements(env, data, &isCopy);
WriteVideoHeaderInternal(env, m_pFormatCtx, m_pVideoStream, rawjBytes, datasize);
(*env)->ReleaseByteArrayElements(env, data, rawjBytes, 0);
return 0;
}
jint WriteVideoHeaderInternal(JNIEnv * env, AVFormatContext* pFormatCtx, AVStream* pVideoStream, jbyte* data, jint datasize)
{
jboolean bNoError = JNI_TRUE;
jbyte* pExtDataBuffer = av_malloc(datasize);
if(!pExtDataBuffer)
{
LOGI("av alloc error\n");
bNoError = JNI_FALSE;
}
if (bNoError)
{
memcpy(pExtDataBuffer, data, datasize * sizeof(jbyte));
pVideoStream->codec->extradata = pExtDataBuffer;
pVideoStream->codec->extradata_size = datasize;
}
}

For the parsing of codec config data, it is wrong that assuming the last 8 bytes are the PPS data. The data must be parsed according to the start code and nal_unit_type.

How to initialize MediaFormat to configure a MediaCodec to decode raw AAC data?

I have a strange issue with my StreamPlayer and I need any help I can get.
The main goal I need to achieve is StreamPlayer which is able to play back MPEG-2 Transport Streams with smallest possible latency. For this I am following this approach:
The stream is parsed by a Java based TS Parser. I have implemented a TSExtractor which is similar to the MediaExtractor and which works fine. I can receive all the media samples for a selected track the same way it is possible using the MediaExtractor with
extractor.readSampleData(...);
extractor.advance();
To decode the AAC data I want to create and configure an instance of MediaCodec. Using the MediaExtractor class this is usually done by
MediaFormat mediaFormat = extractor.getTrackFormat(i);
decoder = MediaCodec.createDecoderByType(mediaFormat.getString(MediaFormat.KEY_MIME));
decoder.configure(mediaFormat, null, null, 0);
As I have to initialize the MediaFormat in the TSExtractor.getTrackFormat(int track) method I use
MediaFormat mf = MediaFormat.createAudioFormat ("audio/mp4a-latm", getSampleRate(), getChannelCount());
and because all my AAC samples include an ADTS I do
mediaFormat.setInteger(MediaFormat.KEY_IS_ADTS, 1);
After reading this post I finally add an ESDS frame using the "csd-0" key
mediaFormat.setByteBuffer("csd-0", ByteBuffer.allocate(2).put(new byte[]{(byte) 0x11, (byte)0x90}));
where the values 0x11 and 0x90 are extracted from the ADTS.
When I now want to decode the AAC samples the decoder posts
AAC decoder returned error 4097, substituting silence
to the Log.
To verify that my TSExtractor extracts the samples correctly I recorded the same stream using VLC remuxing it to an mp4 file without transcoding so the raw stream is unchanged. Now I can initialize the MediaExtractor with the recorded mp4 file and compare the samples created by my TSExtractor and the MediaExtractor. Using trail and error I found a very strange behavior:
When I configure the MediaCodec using the mediaFormat created by the MediaExtractor the MediaCodec decodes the AAC samples returned by my TSExtractor without any problems. Comparing the MediaFormat, which basically wraps a HashMap, created by my TSExtractor and the one created by the MediaExtractor gives this differences:
Created by MediaExtractor:
mediaFormat: {max-input-size=1212, durationUs=77428875, is-adts=1,
channel-count=2, mime=audio/mp4a-latm,
csd-0=java.nio.ByteArrayBuffer[position=0,limit=2,capacity=2],
sample-rate=48000}
Created by TSExtractor:
mediaFormat: {is-adts=1, channel-count=2, mime=audio/mp4a-latm,
csd-0=java.nio.ByteArrayBuffer[position=2,limit=2,capacity=2],
sample-rate=48000}
Even when I adopt the MediaFormat created by the TSExtractor to be similar to the one created by the MediaExtractor the decoder gives the same error using the self created and decodes without any problems using the other one.
Any help would be really helpful.

I really don't know why, but it turns out that initializing the "csd-0" ByteBuffer this way
mediaFormat.setByteBuffer("csd-0", ByteBuffer.allocate(2).put(new byte[]{(byte) 0x11, (byte)0x90}));
doesn't work, but initializing it this way
byte[] bytes = new byte[]{(byte) 0x11, (byte)0x90};
ByteBuffer bb = ByteBuffer.wrap(bytes);
mediaFormat.setByteBuffer("csd-0", bb);
does.
BTW, comparing these two byteBuffers using
bb1.equals(bb2);
returns true.
Very strange!

Values in csd-0 depends on ADTS header.
ADTS header length is up to 9 bytes. To generate csd-0 you need the second and the third byte of header.
int profile = (header[2] & 0xC0) >> 6;
int srate = (header[2] & 0x3C) >> 2;
int channel = ((header[2] & 0x01) << 2) | ((header.[3] & 0xC0) >> 6)
ByteBuffer csd = ByteBuffer.allocate(2);
csd.put(0, (byte)( ((profile + 1) << 3) | srate >> 1 ) );
csd.put(1, (byte)( ((srate << 7) & 0x80) | channel << 3 ) );
Now you got valid csd-0 for this aac audio stream.

In the failing case u may need to call ByteBuffer's rewind method first. If u look carefully u'll see the position is different between the MediaExtractor and the TSExtractor:
csd-0=java.nio.ByteArrayBuffer[position=0,limit=2,capacity=2]
vs
csd-0=java.nio.ByteArrayBuffer[position=2,limit=2,capacity=2]
ByteBuffer's equals only compares the bytes after position until a mismatch; in ur case one buffer is already positioned at the end hence no mismatch.

Thanks for the above code for calculating CSD. Unfortunately this was not working for me. My decoder was failing with the above csd setting. Finally I found the issue. According to the documentation first "5 bits " of CSD is the object type ( Profile). Above code profile is added to only 4 bits. So changing the code as below works fine for me
int profile = (header[2] & 0xC0) >> 6;
int srate = (header[2] & 0x3C) >> 2;
int channel = ((header[2] & 0x01) << 2) | ((header.[3] & 0xC0) >> 6)
ByteBuffer csd = ByteBuffer.allocate(2);
csd.put(0, (byte)(profile << 3 | srate >> 1));
csd.put(1, (byte)((srate & 0x01) << 7 | channel << 3));

How to set encoder buffer size created by MediaCodec

I am trying to use Nexus to test encoding with Mediacodec APIs. I can see the inputBuffers provided by the encoder is 119040 (by logging inputBuffers.capacity). But the size of the frame, i.e. input, is 460800. I got error message at inputBuffer.put with buffer overflow. So I was about to set the input buffer to 460800. The API I could find is BufferInfo.set. However, I cannot find a way to attach this setting to the encoder. Could someone help? Thanks!!!
encoder = MediaCodec.createByCodecName(codecInfo.getName());
ByteBuffer[] inputBuffers = encoder.getInputBuffers();
if (inputBufferIndex >= 0) {
ByteBuffer inputBuffer = inputBuffers[inputBufferIndex];
inputBuffer.clear();
inputBuffer.put(input);
encoder.queueInputBuffer(inputBufferIndex, 0, input.length, 0, 0);}

I'm late to the party, but based on: Android MediaCodec Documentation I think the correct way to change the buffer will be to adjust the MAX_INPUT_SIZE, something like:
int width=800;
int height=480;
encoder = MediaCodec.createByCodecName(codecInfo.getName());
format = MediaFormat.createVideoFormat ("video/avc", width, height);
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE,655360); // 0.5MB but adjust it as you need.

You don't set the size of the input buffer. The size is determined by the MediaFormat, specifically the width, height, and color format. If your input data has a different size, you will need to convert it to the format that the codec is expecting.
This isn't entirely trivial but is doable. For examples, see the buffer-to-buffer tests in the CTS EncodeDecodeTest. The test queries the codec to see what color format is supported, generates frames in that format, submits them to the encoder, then decodes the video and confirms that what comes out is the same as what went in.
The test generally requires API 18 (Android 4.3), but the buffer-to-buffer code will work in API 16. Whether or not it works on any given device is a different question -- since the CTS test didn't exist until API 18, it's possible for pre-4.3 devices to do this wrong.