Develop Reference

How can I record, encrypt (in memory), and mux audio and video on android without the files getting out of sync?

We’re attempting to save video and audio from an Android device into an encrypted file. Our current implementation pipes the outputs from microphone and camera through the MediaEncoder class. As the data is output from MediaEncoder, we are encrypting and writing the contents of the byte buffer to disk. This approach works, however, when attempting to stitch the files back together with FFMPEG, we notice that the two streams seem to get out of sync somewhere mid stream. It appears that a lot of important metadata is lost with this method, specifically presentation timestamps and frame rate data as ffmpeg has to do some guess work to mux the files.
Are there techniques for keeping these streams in sync without using MediaMuxer? The video is encoded with H.264 and the audio with AAC.
Other Approaches:
We attempted to use the MediaMuxer to mux the output data to a file, but our use case requires that we encrypt the bytes of data before they are saved to disk which eliminates the possibility of using the default constructor.
Additionally, we have attempted to use the newly added (API 26) constructor that takes a FileDescriptor instead and have that pointed to a ParcelFileDescriptor that wrapped an Encrypted Document (https://android.googlesource.com/platform/development/+/master/samples/Vault/src/com/example/android/vault/EncryptedDocument.java). However, this approach led to crashes at the native layer and we believe it may have to do with this comment from the source code (https://android.googlesource.com/platform/frameworks/base.git/+/master/media/java/android/media/MediaMuxer.java#353) about the native writer trying to memory map the output file.
import android.graphics.YuvImage
import android.media.MediaCodec
import android.media.MediaCodecInfo
import android.media.MediaFormat
import android.media.MediaMuxer
import com.callyo.video_10_21.Utils.YuvImageUtils.convertNV21toYUV420Planar
import java.io.FileDescriptor
import java.util.*
import java.util.concurrent.atomic.AtomicReference
import kotlin.properties.Delegates
class VideoEncoderProcessor(
private val fileDescriptor: FileDescriptor,
private val width: Int,
private val height: Int,
private val frameRate: Int
): MediaCodec.Callback() {
private lateinit var videoFormat: MediaFormat
private var trackIndex by Delegates.notNull<Int>()
private var mediaMuxer: MediaMuxer
private val mediaCodec = createEncoder()
private val pendingVideoEncoderInputBufferIndices = AtomicReference<LinkedList<Int>>(LinkedList())
companion object {
private const val VIDEO_FORMAT = "video/avc"
}
init {
mediaMuxer = MediaMuxer(fileDescriptor, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)
mediaCodec.setCallback(this)
mediaCodec.start()
}
private fun createEncoder(): MediaCodec {
videoFormat = MediaFormat.createVideoFormat(VIDEO_FORMAT, width, height).apply {
setInteger(MediaFormat.KEY_FRAME_RATE, frameRate)
setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible)
setInteger(MediaFormat.KEY_BIT_RATE, width * height * 5)
setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)
}
return MediaCodec.createEncoderByType(VIDEO_FORMAT).apply {
configure(videoFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
}
}
override fun onInputBufferAvailable(codec: MediaCodec, index: Int) {
// logic for handling stream end omitted for clarity
/* Video frames come in asynchronously from input buffer availability
* so we need to keep track of available buffers in queue */
pendingVideoEncoderInputBufferIndices.get().add(index)
}
override fun onError(codec: MediaCodec, e: MediaCodec.CodecException) {}
override fun onOutputFormatChanged(codec: MediaCodec, format: MediaFormat) {
trackIndex = mediaMuxer.addTrack(format)
mediaMuxer.start()
}
override fun onOutputBufferAvailable(codec: MediaCodec, index: Int, bufferInfo: MediaCodec.BufferInfo) {
val buffer = mediaCodec.getOutputBuffer(index)
buffer?.apply {
if (bufferInfo.size != 0) {
limit(bufferInfo.offset + bufferInfo.size)
rewind()
mediaMuxer.writeSampleData(trackIndex, this, bufferInfo)
}
}
mediaCodec.releaseOutputBuffer(index, false)
if (bufferInfo.flags == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
mediaCodec.stop()
mediaCodec.release()
mediaMuxer.stop()
mediaMuxer.release()
}
}
// Public method that receives raw unencoded video data
fun encode(yuvImage: YuvImage) {
// logic for handling stream end omitted for clarity
pendingVideoEncoderInputBufferIndices.get().poll()?.let { index ->
val buffer = mediaCodec.getInputBuffer(index)
buffer?.clear()
// converting frame to correct color format
val input =
yuvImage.convertNV21toYUV420Planar(ByteArray(yuvImage.yuvData.size), yuvImage.width, yuvImage.height)
buffer?.put(input)
buffer?.let {
mediaCodec.queueInputBuffer(index, 0, input.size, System.nanoTime() / 1000, 0)
}
}
}
}
Additional Info:
I’m using MediaCodec.Callback() (https://developer.android.com/reference/kotlin/android/media/MediaCodec.Callback?hl=en) to handle the encoding asynchronously.

Introduction
I'm going to reference the following Q/A: sync audio and video with mediacodec and mediamuxer
Since the information is lost:
in order to sync audio and video you have to "calculate the number of audio samples that should play for each frame of video"
The author continued and provided an example, e.g.
It depends on the sample rate and the frame rate:
at 24fps and 48000Hz every frame is long (48000hz/24fps)= 2000 sample
at 25 fps and 48000Hz: (48000hz/25fps)= 1920 sample
Examples
Have a look at the following example that muxes a video and audio file, where they set the samples sizes and combine video and audio (from: https://github.com/Docile-Alligator/Infinity-For-Reddit/blob/61c5682b06fb3739a9f980700e6602ae0f39d5a2/app/src/main/java/ml/docilealligator/infinityforreddit/services/DownloadRedditVideoService.java#L506 )
private boolean muxVideoAndAudio(String videoFilePath, String audioFilePath, String outputFilePath) {
try {
File file = new File(outputFilePath);
file.createNewFile();
MediaExtractor videoExtractor = new MediaExtractor();
videoExtractor.setDataSource(videoFilePath);
MediaExtractor audioExtractor = new MediaExtractor();
audioExtractor.setDataSource(audioFilePath);
MediaMuxer muxer = new MediaMuxer(outputFilePath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
videoExtractor.selectTrack(0);
MediaFormat videoFormat = videoExtractor.getTrackFormat(0);
int videoTrack = muxer.addTrack(videoFormat);
audioExtractor.selectTrack(0);
MediaFormat audioFormat = audioExtractor.getTrackFormat(0);
int audioTrack = muxer.addTrack(audioFormat);
boolean sawEOS = false;
int offset = 100;
int sampleSize = 2048 * 1024;
ByteBuffer videoBuf = ByteBuffer.allocate(sampleSize);
ByteBuffer audioBuf = ByteBuffer.allocate(sampleSize);
MediaCodec.BufferInfo videoBufferInfo = new MediaCodec.BufferInfo();
MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();
videoExtractor.seekTo(0, MediaExtractor.SEEK_TO_CLOSEST_SYNC);
audioExtractor.seekTo(0, MediaExtractor.SEEK_TO_CLOSEST_SYNC);
muxer.start();
while (!sawEOS) {
videoBufferInfo.offset = offset;
videoBufferInfo.size = videoExtractor.readSampleData(videoBuf, offset);
if (videoBufferInfo.size < 0 || audioBufferInfo.size < 0) {
sawEOS = true;
videoBufferInfo.size = 0;
} else {
videoBufferInfo.presentationTimeUs = videoExtractor.getSampleTime();
videoBufferInfo.flags = videoExtractor.getSampleFlags();
muxer.writeSampleData(videoTrack, videoBuf, videoBufferInfo);
videoExtractor.advance();
}
}
boolean sawEOS2 = false;
while (!sawEOS2) {
audioBufferInfo.offset = offset;
audioBufferInfo.size = audioExtractor.readSampleData(audioBuf, offset);
if (videoBufferInfo.size < 0 || audioBufferInfo.size < 0) {
sawEOS2 = true;
audioBufferInfo.size = 0;
} else {
audioBufferInfo.presentationTimeUs = audioExtractor.getSampleTime();
audioBufferInfo.flags = audioExtractor.getSampleFlags();
muxer.writeSampleData(audioTrack, audioBuf, audioBufferInfo);
audioExtractor.advance();
}
}
try {
muxer.stop();
muxer.release();
} catch (IllegalStateException ignore) {}
} catch (IOException e) {
e.printStackTrace();
return false;
}
return true;
}
Have a look # the following page: https://sisik.eu/blog/android/media/mix-audio-into-video
From there, they have a wonderful example in the section: Muxing Frames Into MP4 With MediaMuxer that you could use to stitch the file back together.
From there:
In my case, I want to get input from MPEG-4 video and from AAC/M4A audio file, > and mux both inputs into one MPEG-4 output video file. To accomplish that, I
created the following mux() method
fun mux(audioFile: String, videoFile: String, outFile: String) {
// Init extractors which will get encoded frames
val videoExtractor = MediaExtractor()
videoExtractor.setDataSource(videoFile)
videoExtractor.selectTrack(0) // Assuming only one track per file. Adjust code if this is not the case.
val videoFormat = videoExtractor.getTrackFormat(0)
val audioExtractor = MediaExtractor()
audioExtractor.setDataSource(audioFile)
audioExtractor.selectTrack(0) // Assuming only one track per file. Adjust code if this is not the case.
val audioFormat = audioExtractor.getTrackFormat(0)
// Init muxer
val muxer = MediaMuxer(outFile, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4)
val videoIndex = muxer.addTrack(videoFormat)
val audioIndex = muxer.addTrack(audioFormat)
muxer.start()
// Prepare buffer for copying
val maxChunkSize = 1024 * 1024
val buffer = ByteBuffer.allocate(maxChunkSize)
val bufferInfo = MediaCodec.BufferInfo()
// Copy Video
while (true) {
val chunkSize = videoExtractor.readSampleData(buffer, 0)
if (chunkSize > 0) {
bufferInfo.presentationTimeUs = videoExtractor.sampleTime
bufferInfo.flags = videoExtractor.sampleFlags
bufferInfo.size = chunkSize
muxer.writeSampleData(videoIndex, buffer, bufferInfo)
videoExtractor.advance()
} else {
break
}
}
// Copy audio
while (true) {
val chunkSize = audioExtractor.readSampleData(buffer, 0)
if (chunkSize >= 0) {
bufferInfo.presentationTimeUs = audioExtractor.sampleTime
bufferInfo.flags = audioExtractor.sampleFlags
bufferInfo.size = chunkSize
muxer.writeSampleData(audioIndex, buffer, bufferInfo)
audioExtractor.advance()
} else {
break
}
}
// Cleanup
muxer.stop()
muxer.release()
videoExtractor.release()
audioExtractor.release()
}
Update
Based on your comments, I think the main issue is the fileDescriptor. Specifically, they are only using the RandomAccessFile for the file descriptor but the native interface is the one doing the reading.
I have a suggestion then, that maybe you should considering using a FileDescriptor that is in-memory and not based on a file.
So, read the encrypted file and decrypt it in memory, then convert those bytes into a new in-memory fileDescriptor. Feed that in-memory fileDescriptor to the muxor and see what happens.
There's a great answer about this where they use secure private app-only sockets to create a file descriptor, see: Create an in-memory FileDescriptor
Check specifically the second part of that answer starting from:
A better, but more complicated solution is to create a socket in the
filesystem namespace.
Reference: https://stackoverflow.com/a/62651005/1688441
So in more detail:
Read the encrypted file and decrypt it into the bytes but keep in memory
Create a localSocket in your app's private data area, and a Server.
Start listening on your server and accepting the unencrypted bytes.
Create a localSocket Client and pump the unencrypted bytes to the server.
Also pass the fileDescriptor of the client to the muxor.
As the answer states:
This does create a file on the filesystem, but none of the data that passes through the socket is ever written to disk, it is entirely
in-memory. The file is just a name that represents the socket, similar
to the files in /dev that represent devices. Because the socket is
accessed through the filesystem, it is subject to the usual filesystem
permissions, so it is easy to restrict access to the socket by placing
the socket in your app's private data area.
Since this technique creates a file on the filesystem, it would be a
good idea to delete the file after you're done, and also perhaps to
check for and clean up old sockets every so often, in case your app
crashes and leaves old files laying around.

How to play raw NAL units in Android MediaCodec

I initially tried How to play raw NAL units in Andoid exoplayer? but I noticed I'm gonna have to do things in low level.
I've found this simple MediaCodec example. As you can see, it's a thread that plays a file on a surface passed to it.
Notice the lines
mExtractor = new MediaExtractor();
mExtractor.setDataSource(filePath);
It looks like that I should create my own MediaExtractor which, instead of extracting the video units from a file, it'll use the h264 NAL units from a buffer I'll provide.
I can then call mExtractor.setDataSource(MediaDataSource dataSource), see MediaDataSource
It has readAt(long position, byte[] buffer, int offset, int size)
This is where it reads the NAL units. However, how should I pass them? I have no information on the structure of the buffer that needs to be read.
Should I pass a byte[] buffer with the NAL units in it, and if so, in which format? What is the offset for? If it's a buffer, shouldn't I just erase the lines that were read and thus have no offset or size?
By the way, the h264 NAL units are streaming ones, they come from RTP packets, not files. I'm gonna get them through C++ and store them on a buffer an try to pass to the mediaExtractor.
UPDATE:
I've been reading a lot about MediaCodec and I think I understand it better. According to https://developer.android.com/reference/android/media/MediaCodec, everything relies on something of this type:
MediaCodec codec = MediaCodec.createByCodecName(name);
MediaFormat mOutputFormat; // member variable
codec.setCallback(new MediaCodec.Callback() {
#Override
void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {
ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);
// fill inputBuffer with valid data
…
codec.queueInputBuffer(inputBufferId, …);
}
#Override
void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, …) {
ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
// bufferFormat is equivalent to mOutputFormat
// outputBuffer is ready to be processed or rendered.
…
codec.releaseOutputBuffer(outputBufferId, …);
}
#Override
void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {
// Subsequent data will conform to new format.
// Can ignore if using getOutputFormat(outputBufferId)
mOutputFormat = format; // option B
}
#Override
void onError(…) {
…
}
});
codec.configure(format, …);
mOutputFormat = codec.getOutputFormat(); // option B
codec.start();
// wait for processing to complete
codec.stop();
codec.release();
As you can see, I can pass input buffers and get decoded output buffers. The exact byte formats are still a mystery, but I think that's how it works. Also according to the same article, the usage of ByteBuffers is slow, and Surfaces are preferred. They consume the output buffers automatically. Although there's no tutorial on how to do it, there's a section in the article that says it's almost identical, so I guess I just need to add the additional lines
codec.setInputSurface(Surface inputSurface)
codec.setOutputSurface(Surface outputSurface)
Where inputSurface and outputSurface are Surfaces which I pass to a MediaPlayer which I use (how) to display the video in an activity. And the output buffers will simply not come on onOutputBufferAvailable (because the surface consumes them first), neither onInputBufferAvailable.
So the questions now are: how exactly do I construct a Surface which contains the video buffer, and how do I display a MediaPlayer into an activity
For output I can simply create a Surface and pass to a MediaPlayer and MediaCodec, but what about input? Do I need ByteBuffer for the input anyways, and Surface would just be for using other outputs as inputs?

you first need to remove the NAL units , and feed the raw H264 bytes into this method, how ever in your case your reading from the file , so no need to remove any thing since your not using packets , just feed the data bytes to this method:
rivate void initDecoder(){
try {
writeHeader = true;
if(mDecodeMediaCodec != null){
try{
mDecodeMediaCodec.stop();
}catch (Exception e){}
try{
mDecodeMediaCodec.release();
}catch (Exception e){}
}
mDecodeMediaCodec = MediaCodec.createDecoderByType(MIME_TYPE);
//MIME_TYPE = video/avc in your case
mDecodeMediaCodec.configure(format,mSurfaceView.getHolder().getSurface(),
null,
0);
mDecodeMediaCodec.start();
mDecodeInputBuffers = mDecodeMediaCodec.getInputBuffers();
} catch (IOException e) {
e.printStackTrace();
mLatch.trigger();
}
}
private void decode(byte[] data){
try {
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
int inputBufferIndex = mDecodeMediaCodec.dequeueInputBuffer(1000);//
if (inputBufferIndex >= 0) {
ByteBuffer buffer = mDecodeInputBuffers[inputBufferIndex];
buffer.clear();
buffer.put(data);
mDecodeMediaCodec.queueInputBuffer(inputBufferIndex,
0,
data.length,
packet.sequence / 1000,
0);
data = null;
//decodeDataBuffer.clear();
//decodeDataBuffer = null;
}
int outputBufferIndex = mDecodeMediaCodec.dequeueOutputBuffer(info,
1000);
do {
if (outputBufferIndex == MediaCodec.INFO_TRY_AGAIN_LATER) {
//no output available yet
} else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
//encodeOutputBuffers = mDecodeMediaCodec.getOutputBuffers();
} else if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
format = mDecodeMediaCodec.getOutputFormat();
//mediaformat changed
} else if (outputBufferIndex < 0) {
//unexpected result from encoder.dequeueOutputBuffer
} else {
mDecodeMediaCodec.releaseOutputBuffer(outputBufferIndex,
true);
outputBufferIndex = mDecodeMediaCodec.dequeueOutputBuffer(info,
0);
}
} while (outputBufferIndex > 0);
}
please dont forget that iFrame (the first frame bytes) contains sensitive data and MUST be fed to the decoder first

Replacing audio track of .mp4 file

I currently want to replace the audio of .mp4 video file, with another .mp3 audio file.if replacing the audio track of original video is not possible,Please give me solution for how to keep both the audio tracks and let the user to select the desired audio track while playing.
I tried using MediaMuxer and mediaExtractor still i couldnt find out the correct solution.Can anyone please help me.
In media muxer sample program https://developer.android.com/reference/android/media/MediaMuxer.html
MediaMuxer muxer = new MediaMuxer("temp.mp4", OutputFormat.MUXER_OUTPUT_MPEG_4);
// More often, the MediaFormat will be retrieved from MediaCodec.getOutputFormat()
// or MediaExtractor.getTrackFormat().
MediaFormat audioFormat = new MediaFormat(...);
MediaFormat videoFormat = new MediaFormat(...);
int audioTrackIndex = muxer.addTrack(audioFormat);
int videoTrackIndex = muxer.addTrack(videoFormat);
ByteBuffer inputBuffer = ByteBuffer.allocate(bufferSize);
boolean finished = false;
BufferInfo bufferInfo = new BufferInfo();
muxer.start();
while(!finished) {
// getInputBuffer() will fill the inputBuffer with one frame of encoded
// sample from either MediaCodec or MediaExtractor, set isAudioSample to
// true when the sample is audio data, set up all the fields of bufferInfo,
// and return true if there are no more samples.
finished = getInputBuffer(inputBuffer, isAudioSample, bufferInfo);
if (!finished) {
int currentTrackIndex = isAudioSample ? audioTrackIndex : videoTrackIndex;
muxer.writeSampleData(currentTrackIndex, inputBuffer, bufferInfo);
}
};
muxer.stop();
muxer.release();
i am using android API 23,i am getting error saying getInputBuffer and isAudioSample cannot be resolved.
MediaFormat audioFormat=new MediaFormat(...);
What should i write inside the paranthesis.Where should i mention my video and audio file.I searched a lot Please give me some solution to this problem

Currently you can't write anything within the parenthesis. You have to use MediaFormatstatic methods:
MediaFormat audioFormat = MediaFormat.createAudioFormat(MediaFormat.MIMETYPE_AUDIO_AAC, 160000, 1);
MediaFormat videoFormat = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_MPEG4, 1280, 720);
The values that I added here are random. You have to specify:
For the audio: the myme type of the resulting file, the bitrate and amount of channels of the resulting audio
For the video: the myme type of the resulting file, the heigth and width of the resulting video.

What does it mean if Android MediaCodec dequeueOutputBuffer returns -1?

I followed the official Android documentation to setup an encoder for my audio input using a MediaCodec object. But the method dequeueOutputBuffer in the code below always returns -1. What does this return value mean?
My code:
/*configuarion of MediaCodec object*/
codec = MediaCodec.createEncoderByType("audio/mp4a-latm");
MediaFormat format = new MediaFormat();
format.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
format.setInteger(MediaFormat.KEY_SAMPLE_RATE, 44100);
format.setInteger(MediaFormat.KEY_BIT_RATE, 64 * 1024);
format.setInteger(MediaFormat.KEY_AAC_PROFILE,MediaCodecInfo.CodecProfileLevel.AACObjectHE);
codec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
codec.start();
ByteBuffer[] inputBuffers = codec.getInputBuffers();
ByteBuffer[] outputBuffers = codec.getOutputBuffers();
/*main loop to encode audio data*/
for (;;) {
int inputBufferIndex = codec.dequeueInputBuffer(-1); -->this is never zero so data should be written correctly!
if (inputBufferIndex >= 0) {
// write data from audiorecord in queue
inputBuffer = inputBuffers[inputBufferIndex];
inputBuffer.clear();
inputBuffer.put(buf);-->data from audiorecord object
encoder.queueInputBuffer(inputBufferIndex, 0, buf.length, 0, 0);
codec.queueInputBuffer(inputBufferIndex, ...);
}
int outputBufferIndex = codec.dequeueOutputBuffer(bufferInfo, 0);-------->>> "-1"
if (outputBufferIndex >= 0) {
//do something but never called because outputbufferindex is always -1??

Getting -1 as return value when you have passed non negative timeoutUS indicates that call to dequeueOutputBuffer has timed out. According to Android Sample, it returns following integer
public static final int INFO_TRY_AGAIN_LATER = -1;

As far as my knowledge goes, you have to increment the timestamp field in "encoder.queueInputBuffer" everytime because if the same timestamp is given everytime, buffer may be dropped by encoder.

What does the error "OMX_GetExtensionIndex failed" mean in the context of a MediaCodec and how can I resolve it?

I am trying to use the low-level media APIs to decode files in mp3 and other formats so I can process them. I am following the tutorial here to take an encoded file and play it back with an AudioTrack and my code is largely the same but I am getting an error.
Here is my code:
public void playSongMono(View view)
{
//create extractor and point it to file
AssetFileDescriptor fd = getResources().openRawResourceFd(R.raw.song);
MediaExtractor extractor = new MediaExtractor();
extractor.setDataSource(fd.getFileDescriptor(),fd.getStartOffset(),fd.getLength());
//get MIME type of file
MediaFormat inputFormat = extractor.getTrackFormat(0);
Log.d("MonoPlayer","TRACKS #: "+extractor.getTrackCount());
String mime = inputFormat.getString(MediaFormat.KEY_MIME);
Log.d("MonoPlayer","FORMAT: "+mime);
extractor.selectTrack(0);
//create codec
MediaCodec codec = MediaCodec.createDecoderByType(mime);
Log.d("MonoPlayer","1");
codec.configure(inputFormat, null, null, 0);
Log.d("MonoPlayer","2");
codec.start();
Log.d("MonoPlayer","3");
ByteBuffer[] inputBuffers = codec.getInputBuffers();
ByteBuffer[] outputBuffers = codec.getOutputBuffers();
//get a buffer and fill it
int inputBufferIndex = codec.dequeueInputBuffer(1000000);
if(inputBufferIndex >= 0)
{
Log.d("MonoPlayer","4");
//fill the buffer
int sampleSize = extractor.readSampleData(inputBuffers[inputBufferIndex], 0);
long presentationTimeUs = 0;
boolean sawInputEOS= false;
if (sampleSize < 0) {
sawInputEOS = true;
sampleSize = 0;
} else {
presentationTimeUs = extractor.getSampleTime();
}
codec.queueInputBuffer(inputBufferIndex,
0,
sampleSize,
presentationTimeUs,
sawInputEOS ? MediaCodec.BUFFER_FLAG_END_OF_STREAM : 0);
if (!sawInputEOS) {
extractor.advance();
}
}
BufferInfo info = new BufferInfo(); //this will be populated with data by dequeueOutputBuffer
//get the index of a finished buffer. since we only enqueued one we should only wait for one
int resultBufferIndex = codec.dequeueOutputBuffer(info, 1000000);
if (resultBufferIndex >= 0 )
{
Log.d("MonoPlayer","5");
//we now have a buffer of pcm data
byte[] chunk = new byte[info.size];
outputBuffers[resultBufferIndex].get(chunk);
outputBuffers[resultBufferIndex].clear();
codec.releaseOutputBuffer(resultBufferIndex, false);
//create audiotrack to play sound
audiotrack = new AudioTrack(AudioManager.STREAM_MUSIC,
44100,
AudioFormat.CHANNEL_OUT_STEREO,
AudioFormat.ENCODING_PCM_16BIT,
chunk.length,
AudioTrack.MODE_STATIC);
audiotrack.play();
audiotrack.write(chunk, 0, chunk.length);
}
extractor.release();
extractor = null;
codec.stop();
codec.release();
codec = null;
}
And upon execution of this code I get the following logcat output
D MonoPlayer TRACKS #: 1
D MonoPlayer FORMAT: audio/mpeg
I OMXClient Using client-side OMX mux.
D MonoPlayer 1
E OMXNodeInstance OMX_GetExtensionIndex failed
D MonoPlayer 2
D MonoPlayer 3
D MonoPlayer 4
Above is the error I mentioned. I'm not exactly sure why this error is occurring or what it means. However, I have tried to gather some information. The use of log output suggests that the error happens with the line codec.configure(inputFormat, null, null, 0);. I have tried removing the line, which predictably results in an illegal state error but removes the error in question. Additionally, in the code I posted, the Log.d("MonoPlayer","5"); is never reached, even if the time out on the dequeue call is set to indefinite, so I assume the decoder is not properly configured.
If anyone has any information on why I might be getting this error and what I could do to fix it, that would be great. Thanks in advance.

That message is probably harmless. I see it in tests that succeed.
It appears to be coming from line 288 of OMXNodeInstance.cpp. The OMX.google.android.index.enableAndroidNativeBuffers extension lookup fails, which just means the extension wasn't defined on that device.