Using the MediaExtractor class, I am able to get encoded audio sample data from an saved mp4 video with the below:
ByteBuffer byteBuffer = ByteBuffer.allocate(1024 * 256);
MediaExtractor audioExtractor = new MediaExtractor();
try {
int trackIndex = -1;
audioExtractor.setDataSource(originalMediaItem.getFilePath());
for (int i = 0; i < audioExtractor.getTrackCount(); i++) {
MediaFormat format = audioExtractor.getTrackFormat(i);
String mime = format.getString(MediaFormat.KEY_MIME);
if (mime.startsWith("audio/")) {
trackIndex = i;
break;
}
}
audioExtractor.selectTrack(trackIndex);
mAudioFormatMedia = audioExtractor.getTrackFormat(trackIndex);
mAudioTrackIndex = mMediaMuxer.addTrack(mAudioFormatMedia);
int size = audioExtractor.readSampleData(byteBuffer, 0);
do {
if (audioExtractor.getSampleTrackIndex() == 1) {
long presentationTime = audioExtractor.getSampleTime();
mInputBufferHashMap.put(presentationTime, byteBuffer);
audioExtractor.advance();
size = audioExtractor.readSampleData(byteBuffer, 0);
}
} while (size >= 0);
audioExtractor.release();
audioExtractor = null;
} catch (IOException e) {
e.printStackTrace();
}
I have a video source coming from a GlSurface and then want to use a MediaMuxer to mux this video with the audio extraction mentioned previously. Audio is interleaved into the muxer using the hashmap as video is being processed. I am successful in muxing both the Video and Audio and creating a playable mp4 video, however the audio does not sound anything like the original audio of the original mp4.
I do see the expected bufferinfo.size and bufferInfo.presentationTimeUs when I write to the muxer:
mMediaMuxer.writeSampleData(mAudioTrackIndex, buffer, mAudioBufferInfo);
Log.d(TAG, String.format("Wrote %d audio bytes at %d", mAudioBufferInfo.size, mAudioBufferInfo.presentationTimeUs));
I've tried to use the standard inputBuffer, outputBuffer with MediaCodec, like this https://gist.github.com/a-m-s/1991ab18fbcb0fcc2cf9, but this produces the same audio, and from my understanding, MediaExtractor should already be encoded audio data, so data should be able to be piped directly.
What is also interesting is that when i check for the flags when initially extracting:
if( (audioExtractor.getSampleFlags() & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0)
Log.d(TAG, "BUFFER_FLAG_END_OF_STREAM")
Neither of the above get printed for the original mp4 video. I am now questioning the original mp4 video and whether if it is possible to have a non-extractable audiotrack for an mp4 and how I can possibly confirm this.
I believe I've looked at most if not all the MediaExtractor questions on stackoverflow and a lot of the singleton solutions for MediaExtractor on github. Does anyone know of a way to extract audio another way, i.e. using ExoPlayer (preferrably not ffmpeg because it adds a ton of overhead on the android project). Any insights would help if there are any errors in my current implementation!
EDIT 1: This is what the format is audioExtractor.getTrackFormat(trackIndex):
{max-bitrate=512000, sample-rate=48000, track-id=2, durationUs=22373187, mime=audio/mp4a-latm, profile=2, channel-count=4, language=```, aac-profile=2, bitrate=512000, max-input-size=1764, csd-0=java.nio.HeapByteBuffer[pos=0 lim=2 cap=2]}
Problem was attempting to create a Map for the audio data. The AudioData was not correct. I was able to solve this by batching audio sample data while writing videoData using a method like the below:
private void writeAudioSampleData(
MediaExtractor audioExtractor, MediaMuxer muxer, int filterStart, int filterEnd) {
mFilterStart = filterEnd;
MediaCodec.BufferInfo audioBufferInfo = new MediaCodec.BufferInfo();
boolean audioExtractorDone = false;
audioExtractor.seekTo(filterStart, MediaExtractor.SEEK_TO_CLOSEST_SYNC);
synchronized (mAudioLockObject) {
while (!audioExtractorDone) {
try {
audioBufferInfo.size =
audioExtractor.readSampleData(audioInputBuffer, 0);
} catch (Exception e) {
e.printStackTrace();
}
if (DEBUG) {
Log.d(TAG, "audioBufferInfo.size: " + audioBufferInfo.size);
}
if (audioBufferInfo.size < 0) {
audioBufferInfo.size = 0;
audioExtractorDone = true;
} else {
audioBufferInfo.presentationTimeUs = audioExtractor.getSampleTime();
if (audioBufferInfo.presentationTimeUs > filterEnd) {
break; //out of while
}
if (audioBufferInfo.presentationTimeUs >= filterStart &&
audioBufferInfo.presentationTimeUs <= filterEnd) {
audioBufferInfo.presentationTimeUs -= mOriginalMediaItem.mRecordingStartTs;
audioBufferInfo.flags = audioExtractor.getSampleFlags();
try {
muxer.writeSampleData(mAudioTrackIndex, audioInputBuffer,
audioBufferInfo);
if (DEBUG)Log.d(TAG, String.format("Wrote %d audio bytes at %d",
audioBufferInfo.size, audioBufferInfo.presentationTimeUs));
} catch(IllegalArgumentException | IllegalStateException |
NullPointerException ignore) {}
}
audioExtractor.advance();
}
}
}
Related
I'm doing a transcoder using MediaCodec.
I created two mediacodec instance, one is for decoding and another is for encoding. I'm trying to send decoders outputBuffer directly into encoders inputBuffer.
It seems has no problem while compiling and executing.And it runs quickly.
But the output video file has something wrong.I checked the metadata of the output video and they are all right : bitrate, framerate, resolution ...Only the images in the video is wrong like this:screen shot
I thought it has somethings wrong,but I cannot figure it out...
I searched libraries and documents, and I found some sample codes using Texture surface to render the decoder output data and tranfer the data into the encoder. But I thought it should not be neccessary for me. Because I dont need to edit images of the video.What I only need to do is changing the bitrate and resolution to make the file's size smaller.
here is the core code in my project:
private void decodeCore() {
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
int frameCount = 0;
while (mDecodeRunning) {
int inputBufferId = mDecoder.dequeueInputBuffer(50);
if (inputBufferId >= 0) {
// fill inputBuffers[inputBufferId] with valid data
int sampleSize = mExtractor.readSampleData(mDecodeInputBuffers[inputBufferId], 0);
if (sampleSize >= 0) {
long time = mExtractor.getSampleTime();
mDecoder.queueInputBuffer(inputBufferId, 0, sampleSize, time, 0);
} else {
mDecoder.queueInputBuffer(inputBufferId, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
}
mExtractor.advance();
}
int outputBufferId = mDecoder.dequeueOutputBuffer(bufferInfo, 50);
if (outputBufferId >= 0) {
FrameData data = mFrameDataQueue.obtain();
//wait until queue has space to push data
while (data == null) {
try {
Thread.sleep(20);
} catch (InterruptedException e) {
e.printStackTrace();
}
data = mFrameDataQueue.obtain();
}
data.data.clear();
data.size = 0;
data.offset = 0;
data.flag = 0;
data.frameTimeInUs = bufferInfo.presentationTimeUs;
// outputBuffers[outputBufferId] is ready to be processed or rendered.
if (bufferInfo.size > 0) {
ByteBuffer buffer = mDecodeOutputBuffers[outputBufferId];
buffer.position(bufferInfo.offset);
buffer.limit(bufferInfo.offset + bufferInfo.size);
data.data.put(buffer);
data.data.flip();
data.size = bufferInfo.size;
data.frameIndex = frameCount++;
}
data.flag = bufferInfo.flags;
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
Log.d("bingbing_transcode", "decode over! frames:" + (frameCount - 1));
mDecodeRunning = false;
}
mFrameDataQueue.pushToQueue(data);
mDecoder.releaseOutputBuffer(outputBufferId, false);
Log.d("bingbing_transcode", "decode output:\n frame:" + (frameCount - 1) + "\n" + "size:" + bufferInfo.size);
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
mDecodeOutputBuffers = mDecoder.getOutputBuffers();
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// Subsequent data will conform to new format.
mDecodeOutputVideoFormat = mDecoder.getOutputFormat();
configureAndStartEncoder();
}
}
mDecoder.stop();
mDecoder.release();
}
private void encodeCore() {
int trackIndex = 0;
boolean muxerStarted = false;
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
int frameCount = 0;
while (mEncodeRunning) {
int inputBufferId = mEncoder.dequeueInputBuffer(50);
if (inputBufferId >= 0) {
FrameData data = mFrameDataQueue.pollFromQueue();
//wait until queue has space to push data
while (data == null) {
try {
Thread.sleep(20);
} catch (InterruptedException e) {
e.printStackTrace();
}
data = mFrameDataQueue.obtain();
}
if (data.size > 0) {
ByteBuffer inputBuffer = mEncodeInputBuffers[inputBufferId];
inputBuffer.clear();
inputBuffer.put(data.data);
inputBuffer.flip();
}
mEncoder.queueInputBuffer(inputBufferId, 0, data.size, data.frameTimeInUs, data.flag);
mFrameDataQueue.recycle(data);
}
int outputBufferId = mEncoder.dequeueOutputBuffer(bufferInfo, 50);
if (outputBufferId >= 0) {
// outputBuffers[outputBufferId] is ready to be processed or rendered.
ByteBuffer encodedData = mEncodeOutputBuffers[outputBufferId];
if (bufferInfo.size > 0) {
if (encodedData == null) {
throw new RuntimeException("encoderOutputBuffer " + outputBufferId + " was null");
}
if (!muxerStarted) {
throw new RuntimeException("muxer hasn't started");
}
frameCount++;
}
// adjust the ByteBuffer values to match BufferInfo (not needed?)
encodedData.position(bufferInfo.offset);
encodedData.limit(bufferInfo.offset + bufferInfo.size);
mMuxer.writeSampleData(trackIndex, encodedData, bufferInfo);
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
Log.d("bingbing_transcode", "encode over! frames:" + (frameCount - 1));
mEncodeRunning = false;
}
mEncoder.releaseOutputBuffer(outputBufferId, false);
Log.d("bingbing_transcode", "encode output:\n frame:" + (frameCount - 1) + "\n" + "size:" + bufferInfo.size);
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
mEncodeOutputBuffers = mEncoder.getOutputBuffers();
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// should happen before receiving buffers, and should only happen once
if (muxerStarted) {
throw new RuntimeException("format changed twice");
}
MediaFormat newFormat = mEncoder.getOutputFormat();
Log.d("bingbing_transcode", "encoder output format changed: " + newFormat);
// now that we have the Magic Goodies, start the muxer
trackIndex = mMuxer.addTrack(newFormat);
mMuxer.start();
muxerStarted = true;
mEncodeOutputVideoFormat = newFormat;
}
}
mEncoder.stop();
mEncoder.release();
if (muxerStarted) {
mMuxer.stop();
mMuxer.release();
}
}
these two functions run in two different threads.
FrameData is a simple storage of frame bytebuffer and frame present time and something needed
When using bytebuffer input, there are a few details that are undefined about the input data layout. When the width isn't a multiple of 16, some encoders want to have the input data row length padded to a multiple of 16, while others will assume a line length equal to the width, with no extra padding.
The Android CTS tests (which define what behaviour one can expect across all devices) for encoding from bytebuffer inputs intentionally only test resolutions that are a multiple of 16, since they know different hardware vendors do this differently, and they didn't want to enforce any particular handling.
You can't generally assume that the decoder output would use a similar row size as what the encoder consumes either. The decoder is free to (and some actually do) return a significantly larger width than the actual content size, and use the crop_left/crop_right fields for indicating what parts of it actually are intended to be visible. So in case the decoder did that, you can't pass the data straight from the decoder to the encoder unless you copy it line by line, taking into account the actual line sizes used by the decoder and encoder.
Additionally, you can't even assume that the decoder uses a similar pixel format as the encoder. Many qualcomm devices use a special tiled pixel format for the decoder output, while the encoder input is normal planar data. In these cases, you'd have to implement a pretty complex logic for unshuffling the data before you can feed it into the encoder.
Using a texture surface as intermediate hides all of these details. It might not sound completely necessary for your use case, but it does hide all the variation in buffer formats between decoder and encoder.
I would like to produce mp4 file by multiplexing audio from mic (overwrite didGetAudioData) and video from camera (overwrite onpreviewframe).However, I encountered the sound and video synchronization problem, video will appear faster than audio. I wondered if the problem related to incompatible configurations or presentationTimeUs, could someone guide me how to fix the problem. Below were my software.
Video configuration
formatVideo = MediaFormat.createVideoFormat(MIME_TYPE_VIDEO, 640, 360);
formatVideo.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar);
formatVideo.setInteger(MediaFormat.KEY_BIT_RATE, 2000000);
formatVideo.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
formatVideo.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
got video presentationPTS as below,
if(generateIndex == 0) {
videoAbsolutePtsUs = 132;
StartVideoAbsolutePtsUs = System.nanoTime() / 1000L;
}else {
CurrentVideoAbsolutePtsUs = System.nanoTime() / 1000L;
videoAbsolutePtsUs =132+ CurrentVideoAbsolutePtsUs-StartVideoAbsolutePtsUs;
}
generateIndex++;
audio configuration
format = MediaFormat.createAudioFormat(MIME_TYPE, 48000/*sample rate*/, AudioFormat.CHANNEL_IN_MONO /*Channel config*/);
format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC);
format.setInteger(MediaFormat.KEY_SAMPLE_RATE,48000);
format.setInteger(MediaFormat.KEY_CHANNEL_COUNT,1);
format.setInteger(MediaFormat.KEY_BIT_RATE,64000);
got audio presentationPTS as below,
if(generateIndex == 0) {
audioAbsolutePtsUs = 132;
StartAudioAbsolutePtsUs = System.nanoTime() / 1000L;
}else {
CurrentAudioAbsolutePtsUs = System.nanoTime() / 1000L;
audioAbsolutePtsUs =CurrentAudioAbsolutePtsUs - StartAudioAbsolutePtsUs;
}
generateIndex++;
audioAbsolutePtsUs = getJitterFreePTS(audioAbsolutePtsUs, audioInputLength / 2);
long startPTS = 0;
long totalSamplesNum = 0;
private long getJitterFreePTS(long bufferPts, long bufferSamplesNum) {
long correctedPts = 0;
long bufferDuration = (1000000 * bufferSamplesNum) / 48000;
bufferPts -= bufferDuration; // accounts for the delay of acquiring the audio buffer
if (totalSamplesNum == 0) {
// reset
startPTS = bufferPts;
totalSamplesNum = 0;
}
correctedPts = startPTS + (1000000 * totalSamplesNum) / 48000;
if(bufferPts - correctedPts >= 2*bufferDuration) {
// reset
startPTS = bufferPts;
totalSamplesNum = 0;
correctedPts = startPTS;
}
totalSamplesNum += bufferSamplesNum;
return correctedPts;
}
Was my issue caused by applying jitter function for audio only? If yes, how could I apply jitter function for video? I also tried to find correct audio and video presentationPTS by https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/EncodeDecodeTest.java. But encodedecodeTest only provided video PTS. That's the reason my implementation used system nanotime for both audio and video. If I want to use video presentationPTS in encodedecodetest, how to construct the compatible audio presentationPTS? Thanks for help!
below are how i queue yuv frame to video mediacodec for reference. For audio part, it is identical except for different presentationPTS.
int videoInputBufferIndex;
int videoInputLength;
long videoAbsolutePtsUs;
long StartVideoAbsolutePtsUs, CurrentVideoAbsolutePtsUs;
int put_v =0;
int get_v =0;
int generateIndex = 0;
public void setByteBufferVideo(byte[] buffer, boolean isUsingFrontCamera, boolean Input_endOfStream){
if(Build.VERSION.SDK_INT >=18){
try{
endOfStream = Input_endOfStream;
if(!Input_endOfStream){
ByteBuffer[] inputBuffers = mVideoCodec.getInputBuffers();
videoInputBufferIndex = mVideoCodec.dequeueInputBuffer(-1);
if (VERBOSE) {
Log.w(TAG,"[put_v]:"+(put_v)+"; videoInputBufferIndex = "+videoInputBufferIndex+"; endOfStream = "+endOfStream);
}
if(videoInputBufferIndex>=0) {
ByteBuffer inputBuffer = inputBuffers[videoInputBufferIndex];
inputBuffer.clear();
inputBuffer.put(mNV21Convertor.convert(buffer));
videoInputLength = buffer.length;
if(generateIndex == 0) {
videoAbsolutePtsUs = 132;
StartVideoAbsolutePtsUs = System.nanoTime() / 1000L;
}else {
CurrentVideoAbsolutePtsUs = System.nanoTime() / 1000L;
videoAbsolutePtsUs =132+ CurrentVideoAbsolutePtsUs - StartVideoAbsolutePtsUs;
}
generateIndex++;
if (VERBOSE) {
Log.w(TAG, "[put_v]:"+(put_v)+"; videoAbsolutePtsUs = " + videoAbsolutePtsUs + "; CurrentVideoAbsolutePtsUs = "+CurrentVideoAbsolutePtsUs);
}
if (videoInputLength == AudioRecord.ERROR_INVALID_OPERATION) {
Log.w(TAG, "[put_v]ERROR_INVALID_OPERATION");
} else if (videoInputLength == AudioRecord.ERROR_BAD_VALUE) {
Log.w(TAG, "[put_v]ERROR_ERROR_BAD_VALUE");
}
if (endOfStream) {
Log.w(TAG, "[put_v]:"+(put_v++)+"; [get] receive endOfStream");
mVideoCodec.queueInputBuffer(videoInputBufferIndex, 0, videoInputLength, videoAbsolutePtsUs, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
} else {
Log.w(TAG, "[put_v]:"+(put_v++)+"; receive videoInputLength :" + videoInputLength);
mVideoCodec.queueInputBuffer(videoInputBufferIndex, 0, videoInputLength, videoAbsolutePtsUs, 0);
}
}
}
}catch (Exception x) {
x.printStackTrace();
}
}
}
How I solved this in my application was by setting the PTS of all video and audio frames against a shared "sync clock" (note the sync also means it's thread-safe) that starts when the first video frame (having a PTS 0 on its own) is available. So if audio recording starts sooner than video, audio data is dismissed (doesn't go into encoder) until video starts, and if it starts later, then the first audio PTS will be relative to the start of the entire video.
Ofcourse you are free to allow audio to start first, but players will usually skip or wait for the first video frame anyway. Also be careful that encoded audio frames will arrive "out of order" and MediaMuxer will fail with an error sooner or later. My solution was to queue them all like this: sort them by pts when a new one comes in, then write everything that is older than 500 ms (relative to the newest one) to MediaMuxer, but only those with a PTS higher than the latest written frame. Ideally this means data is smoothly written to MediaMuxer, with a 500 ms delay. Worst case, you will lose a few audio frames.
I have a mp4 video file in my sd card. I would like to extract the audio from the video and then save the extracted audio as a separate file on the sd card using MediaExtractor Api. Here is the code I've tried:
MediaExtractor extractor = new MediaExtractor();
extractor.setDataSource(MEDIA_PATH_To_File_On_SDCARD);
for (i = 0; i < extractor.getTrackCount(); i++) {
MediaFormat format = extractor.getTrackFormat(i);
String mime = format.getString(MediaFormat.KEY_MIME);
if (mime.startsWith("audio/")) {
extractor.selectTrack(i);
decoder = MediaCodec.createDecoderByType(mime);
if(decoder != null)
{
decoder.configure(format, null, null, 0);
}
break;
}
}
Am stuck here I have no idea of how to take the selected audio track and save it to the sd card.
Late to the party, this can be done by using MediaExtractor and MediaMuxer Apis together, Check out the working URL from below,
/**
* #param srcPath the path of source video file.
* #param dstPath the path of destination video file.
* #param startMs starting time in milliseconds for trimming. Set to
* negative if starting from beginning.
* #param endMs end time for trimming in milliseconds. Set to negative if
* no trimming at the end.
* #param useAudio true if keep the audio track from the source.
* #param useVideo true if keep the video track from the source.
* #throws IOException
*/
#SuppressLint("NewApi")
public void genVideoUsingMuxer(String srcPath, String dstPath, int startMs, int endMs, boolean useAudio, boolean useVideo) throws IOException {
// Set up MediaExtractor to read from the source.
MediaExtractor extractor = new MediaExtractor();
extractor.setDataSource(srcPath);
int trackCount = extractor.getTrackCount();
// Set up MediaMuxer for the destination.
MediaMuxer muxer;
muxer = new MediaMuxer(dstPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
// Set up the tracks and retrieve the max buffer size for selected
// tracks.
HashMap<Integer, Integer> indexMap = new HashMap<Integer, Integer>(trackCount);
int bufferSize = -1;
for (int i = 0; i < trackCount; i++) {
MediaFormat format = extractor.getTrackFormat(i);
String mime = format.getString(MediaFormat.KEY_MIME);
boolean selectCurrentTrack = false;
if (mime.startsWith("audio/") && useAudio) {
selectCurrentTrack = true;
} else if (mime.startsWith("video/") && useVideo) {
selectCurrentTrack = true;
}
if (selectCurrentTrack) {
extractor.selectTrack(i);
int dstIndex = muxer.addTrack(format);
indexMap.put(i, dstIndex);
if (format.containsKey(MediaFormat.KEY_MAX_INPUT_SIZE)) {
int newSize = format.getInteger(MediaFormat.KEY_MAX_INPUT_SIZE);
bufferSize = newSize > bufferSize ? newSize : bufferSize;
}
}
}
if (bufferSize < 0) {
bufferSize = DEFAULT_BUFFER_SIZE;
}
// Set up the orientation and starting time for extractor.
MediaMetadataRetriever retrieverSrc = new MediaMetadataRetriever();
retrieverSrc.setDataSource(srcPath);
String degreesString = retrieverSrc.extractMetadata(MediaMetadataRetriever.METADATA_KEY_VIDEO_ROTATION);
if (degreesString != null) {
int degrees = Integer.parseInt(degreesString);
if (degrees >= 0) {
muxer.setOrientationHint(degrees);
}
}
if (startMs > 0) {
extractor.seekTo(startMs * 1000, MediaExtractor.SEEK_TO_CLOSEST_SYNC);
}
// Copy the samples from MediaExtractor to MediaMuxer. We will loop
// for copying each sample and stop when we get to the end of the source
// file or exceed the end time of the trimming.
int offset = 0;
int trackIndex = -1;
ByteBuffer dstBuf = ByteBuffer.allocate(bufferSize);
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
muxer.start();
while (true) {
bufferInfo.offset = offset;
bufferInfo.size = extractor.readSampleData(dstBuf, offset);
if (bufferInfo.size < 0) {
Log.d(TAG, "Saw input EOS.");
bufferInfo.size = 0;
break;
} else {
bufferInfo.presentationTimeUs = extractor.getSampleTime();
if (endMs > 0 && bufferInfo.presentationTimeUs > (endMs * 1000)) {
Log.d(TAG, "The current sample is over the trim end time.");
break;
} else {
bufferInfo.flags = extractor.getSampleFlags();
trackIndex = extractor.getSampleTrackIndex();
muxer.writeSampleData(indexMap.get(trackIndex), dstBuf, bufferInfo);
extractor.advance();
}
}
}
muxer.stop();
muxer.release();
return;
}
You can use the above-mentioned method by using a single line:
genVideoUsingMuxer(videoFile, originalAudio, -1, -1, true, false)
Also, read the comments to use this method more efficiently.
GIST: https://gist.github.com/ArsalRaza/132a6e99d59aa80b9861ae368bc786d0
take a look at my post Decoding Video and Encoding again by Mediacodec gets a corrupted file where there is an example (just take care about the answer too).
you have to use a MediaMuxer, call AddTrack for the video track and write the data to this track to the muxer after encoding each frame. You have to add track for audio too. If you just want only audio, ignore the video part, and just save the data to the muxer related to the audio. You can see some examples in grafika page, one of them could be this: https://github.com/google/grafika/
Also you can find more examples here: http://www.bigflake.com/mediacodec/
Thanks
I want to convert the file from .mp3 to AAC and I reference the source code of the Spydroid APP.
It convert the audio data from Microphone audio source to AAC.
I try to modify the code and change the audio source from Microphone audio source to local mp3 file.
But I don't know how to pushing raw audio to the decoder...
Does somebody can help me and give me some suggestion ?
The code of encode is like the following:
protected void encodeWithMediaCodec() throws IOException {
final int bufferSize = AudioRecord.getMinBufferSize(mQuality.samplingRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT)*2;
((AACLATMPacketizer)mPacketizer).setSamplingRate(mQuality.samplingRate);
//The music file in my phone , and I want to push it to the
long file_size = 0;
string path = "/storage/sdcard1/music/test.mp3"
File audio = new File(path );
mMediaCodec = MediaCodec.createEncoderByType("audio/mp4a-latm");
MediaFormat format = new MediaFormat();
format.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
format.setInteger(MediaFormat.KEY_BIT_RATE, mQuality.bitRate);
format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
format.setInteger(MediaFormat.KEY_SAMPLE_RATE, mQuality.samplingRate);
format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC);
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, mBuffer_Size);
mMediaCodec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mMediaCodec.start();
final MediaCodecInputStream inputStream = new MediaCodecInputStream(mMediaCodec);
final ByteBuffer[] inputBuffers = mMediaCodec.getInputBuffers();
mThread = new Thread(new Runnable() {
#Override
public void run() {
int len = 0, bufferIndex = 0;
try {
while (!Thread.interrupted()) {
bufferIndex = mMediaCodec.dequeueInputBuffer(10000);
if (bufferIndex>=0) {
inputBuffers[bufferIndex].clear();
len = audio.length();
if (len == AudioRecord.ERROR_INVALID_OPERATION || len == AudioRecord.ERROR_BAD_VALUE) {
Log.e(TAG,"An error occured with the AudioRecord API !");
} else {
Log.v(TAG,"Pushing raw audio to the decoder: len="+len+" bs: "+inputBuffers[bufferIndex].capacity());
mMediaCodec.queueInputBuffer(bufferIndex, 0, len, System.nanoTime()/1000, 0);
}
}
}
} catch (RuntimeException e) {
e.printStackTrace();
}
}
});
mThread.start();
// The packetizer encapsulates this stream in an RTP stream and send it over the network
mPacketizer.setDestination(mDestination, mRtpPort, mRtcpPort);
mPacketizer.setInputStream(inputStream);
mPacketizer.start();
mStreaming = true;
}
It has error log like the following:
V/AACStream(25015): Pushing raw audio to the decoder: len=11952883 bs: 4096
W/System.err(25015): java.lang.IllegalStateException
W/System.err(25015): at android.media.MediaCodec.queueInputBuffer(Native Method)
W/System.err(25015): at net.majorkernelpanic.streaming.audio.AACStream$1.run(AACStream.java:258)
W/System.err(25015): at java.lang.Thread.run(Thread.java:856)
W/InputMethodManagerService( 574): Window already focused, ignoring focus gain of: com.android.internal.view.IInputMethodClient$Stub$Proxy#42afeb98 attribute=null
And the error of the code is at the
mMediaCodec.queueInputBuffer(bufferIndex, 0, len, System.nanoTime()/1000, 0);
I think the error is at the len , but I don't know how to fix it...
And it seems does not put the music file to the encoder...And I have two question
Question 1:
How to pushing the audio file to the MediaCodec?
Did I missing something at the code ?
Question 2:
How to make sure the data has been encode to the AAC ??
I've a problem with this library
https://github.com/fyhertz/libstreaming
it allows to send via wireless the streaming of photocamera, it use 3 methods: two with mediacodec and one with mediarecorder.
I would like to modify it, and I have to use only the mediacodec;however first of all I tried the code of the example 2 of the library, but I've always found the same error:
the log tell me that the device can use the mediacodec, it set the encoder and when it test the decoder it fall and the buffer is filled with -1.
This is the method in the EncoderDebugger class where the exception occurs, some kind soul can help me please?
private long decode(boolean withPrefix) {
int n =3, i = 0, j = 0;
long elapsed = 0, now = timestamp();
int decInputIndex = 0, decOutputIndex = 0;
ByteBuffer[] decInputBuffers = mDecoder.getInputBuffers();
ByteBuffer[] decOutputBuffers = mDecoder.getOutputBuffers();
BufferInfo info = new BufferInfo();
while (elapsed<3000000) {
// Feeds the decoder with a NAL unit
if (i<NB_ENCODED) {
decInputIndex = mDecoder.dequeueInputBuffer(1000000/FRAMERATE);
if (decInputIndex>=0) {
int l1 = decInputBuffers[decInputIndex].capacity();
int l2 = mVideo[i].length;
decInputBuffers[decInputIndex].clear();
if ((withPrefix && hasPrefix(mVideo[i])) || (!withPrefix && !hasPrefix(mVideo[i]))) {
check(l1>=l2, "The decoder input buffer is not big enough (nal="+l2+", capacity="+l1+").");
decInputBuffers[decInputIndex].put(mVideo[i],0,mVideo[i].length);
} else if (withPrefix && !hasPrefix(mVideo[i])) {
check(l1>=l2+4, "The decoder input buffer is not big enough (nal="+(l2+4)+", capacity="+l1+").");
decInputBuffers[decInputIndex].put(new byte[] {0,0,0,1});
decInputBuffers[decInputIndex].put(mVideo[i],0,mVideo[i].length);
} else if (!withPrefix && hasPrefix(mVideo[i])) {
check(l1>=l2-4, "The decoder input buffer is not big enough (nal="+(l2-4)+", capacity="+l1+").");
decInputBuffers[decInputIndex].put(mVideo[i],4,mVideo[i].length-4);
}
mDecoder.queueInputBuffer(decInputIndex, 0, l2, timestamp(), 0);
i++;
} else {
if (VERBOSE) Log.d(TAG,"No buffer available !7");
}
}
// Tries to get a decoded image
decOutputIndex = mDecoder.dequeueOutputBuffer(info, 1000000/FRAMERATE);
if (decOutputIndex == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
decOutputBuffers = mDecoder.getOutputBuffers();
} else if (decOutputIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
mDecOutputFormat = mDecoder.getOutputFormat();
} else if (decOutputIndex>=0) {
if (n>2) {
// We have successfully encoded and decoded an image !
int length = info.size;
mDecodedVideo[j] = new byte[length];
decOutputBuffers[decOutputIndex].clear();
decOutputBuffers[decOutputIndex].get(mDecodedVideo[j], 0, length);
// Converts the decoded frame to NV21
convertToNV21(j);
if (j>=NB_DECODED-1) {
flushMediaCodec(mDecoder);
if (VERBOSE) Log.v(TAG, "Decoding "+n+" frames took "+elapsed/1000+" ms");
return elapsed;
}
j++;
}
mDecoder.releaseOutputBuffer(decOutputIndex, false);
n++;
}
elapsed = timestamp() - now;
}
throw new RuntimeException("The decoder did not decode anything.");
}
Here's my suggestions:
(1) check the settings of encoder and decoder, and make sure that they match. For example, revolution and color format are the same.
(2) make sure the very first packet generated by the encoder has been sent and pushed into the decoder. This packet defines the basic settings of the video stream.
(3) the decoder usually buffers 5-10 frames. So data in the buffer is invalid for a few hundred ms.
(4) while initiating the decoder, set the surface as null. Otherwise the output buffer will be read by the surface and probably released automatically.