I am trying to write a video player using the MediaDecoder class, i came across a problem that its blocking my development,
here is a code snippet
extractor = new MediaExtractor();
extractor.setDataSource(filename);
MediaFormat format = extractor.getTrackFormat(i);
extractor.selectTrack(0);
MediaCodec decoder = MediaCodec.createDecoderByType(mime);
decoder.configure(format, null, null, 0);
decoder.start();
ByteBuffer[] inputBuffers = decoder.getInputBuffers();
ByteBuffer[] outputBuffers = decoder.getOutputBuffers();
Log.d(TAG, " "+decoder.getOutputFormat());
The problem is the output format printed changes with each device making it impossible to print it out to an Open GL texture.
Is there a way to force the decoder to output always the same format?
If no, does anyone knows of any libraries available to make the conversion?
Thanks a lot for any insights
The decoder's output format can't be set. In fact, you probably shouldn't even examine it until you receive an updated MediaFormat from dequeueOutputBuffer() -- before the first chunk of data is returned, you'll get a MediaCodec.INFO_OUTPUT_FORMAT_CHANGED result.
If you want to access the decoded frame from OpenGL ES, you should create a Surface from a SurfaceTexture and pass that into the decoder configure() call. It's more efficient, and doesn't require you to know anything about the data format.
For an example, see the DecodeEditEncodeTest.java CTS test, and note in particular the OutputSurface class and how it's used in checkVideoFile().
Related
I have to decode an H.264 stream with the Android MediaCodec API. Problem is I don't know at this point the size of the output stream (I mean resolution of frames : 640x360? 1280x720? 1920x1080? ...). I use my own access/demuxer code to get the input stream as a ByteBuffer object. First step to create my MediaCodec is:
MediaCodec mediaCodec = MediaCodec.createDecoderByType("video/avc");
Then, to configure it, I would like to have the proper MediaFormat information to configure my MediaCodec object fine. Currently, I do:
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 1280, 720);
But the size is not always 1280x720 and my MediaFormat needs more information (on some test devices, decoder produces weird results but works fine on others). I read the doc and I saw that we can get the proper MediaFormat using a MediaExtractor and getTrackFormat(/* idx */);. But how can I use MediaExtractor with my raw ByteBuffer (setDataSource(...) can only handle file descriptors or uris)? Is there an other way to process?
Thanks!
EDIT: I extracted SPS and PPS from my stream and provided it to the codec at the configuration step (passing by the MediaFormat) doing:
mediaFormat.setByteBuffer("csd-0", bufferSPS);
mediaFormat.setByteBuffer("csd-1", bufferPPS);
/*...*/
mediaCodec.configure(mediaFormat, null, null, 0);
I am working on a project that needs to process a video using OpenGL on Android. I decided to use MediaCodec and I managed to get it works with the help from ExtractDecodeEditEncodeMuxTest. The result is quite good, I have it receives a video, extracts the tracks, decodes the videotrack, edits with OpenGL, and encodes to a video file.
The problem is that the result video can be play well on Android, but when it comes to iOS, two-thirds of the screen is green.
I tried to solve with the suggestions from here, here, and here, experiment different formats for the encoder, but the problem is still the same.
Could someone suggest me the reasons that can cause this problem and how to fix it?
This is the video when it's played on iOS
This is the configuration for the encoder
MediaCodec mediaCodec = MediaCodec.createEncoderByType("video/avc");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 540, 960);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 2000000);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, CodecCapabilities.COLOR_FormatSurface);
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
Update
I wonder if i had any mistake with the video orientation, because the working partial of the output video has the same ratio as the desired output resolution, but in horizontal orientation. The input is vertical recorded, so does the desired output.
Here is the code of the decoder configuration:
inputFormat.setInteger(MediaFormat.KEY_WIDTH, 540);
inputFormat.setInteger(MediaFormat.KEY_HEIGHT, 960);
inputFormat.setInteger("rotation-degrees", 90);
String mime = inputFormat.getString(MediaFormat.KEY_MIME);
MediaCodec decoder = MediaCodec.createDecoderByType(mime);
decoder.configure(inputFormat, surface, null, 0);
Update Dec 25: I've tried different resolutions and orientations when configuring both encoder and decoder to check if the video's orientation is the problem or not, but the output video just got rotated, the green problem is still there.
I also tried "video/mp4v-es" for the encoder, the result video is viewable on Mac, but the iPhone cannot even play it.
I've just solved it. The reason turns out to be the MediaMuxer, it wraps the h264 raw stream in some sort of container that iOS cant understand. So instead of using MediaMuxer, I write the raw h264 stream from the encoder to a file, and use mp4parser to mux it into a mp4 file.
I know the answer now: It has to do with the fps range. I changed the fps rate on my camera params and on the media codec and suddenly it worked!
I have seen the below example for encode/decode using MediaCodec API.
https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/EncodeDecodeTest.java
In which there is a comparsion of the guessed presentation time and the presentation time received from decoded info.
assertEquals("Wrong time stamp", computePresentationTime(checkIndex),
info.presentationTimeUs);
Because the decoder just decode the data in encoded buffer, I think there is any timestamp info could be parsed in this encoder's output H.264 stream.
I am writing an Android application which mux a H264 stream (.h264) encoded by MediaCodec to mp4 container by using ffmpeg (libavformat).
I don't want to use MediaMuxer because it require version 4.3 which is too high.
However, ffmpeg seems not recognize the presentation timestamp in a packet encoded by MediaCodec, so I always get NO_PTS value when try to read a frame from the stream.
Anyone know how to get the correct presentation timestamp in this situation?
to send timestamps from MediaCodec encoder to ffmpeg you need to convert like that:
jint Java_com_classclass_WriteVideoFrame(JNIEnv * env, jobject this, jbyteArray data, jint datasize, jlong timestamp) {
....
AVPacket pkt;
av_init_packet(&pkt);
AVCodecContext *c = m_pVideoStream->codec;
pkt.pts = (long)((double)timestamp * (double)c->time_base.den / 1000.0);
pkt.stream_index = m_pVideoStream->index;
pkt.data = rawjBytes;
pkt.size = datasize;
where time_base depends on framerate
upd re timestamps flow in pipline:
neither decoder nor encoder knows time-stamps by their own. timestamps are set to these components via
decoder.queueInputBuffer(inputBufIndex, 0, info.size, info.presentationTimeUs, info.flags);
or
encoder.queueInputBuffer(inputBufIndex, 0, 0, ptsUsec, info.flags);
these timestamps could be taken from extractor, from camera or generated by app, but decoder\encoder just passes through these time-stamps without changing them. as a result time-stamps go unchanged from source to sink (muxer).
for sure there are some exclusions: if frames frequency in changed - frame rate conversion for example. or if encoder makes encoding with B-frames and reordering happens.
or encoder can add time-stamps to the encoder frame header - optional, not mandatory by standard. i think all of this is not applied to current android version, codecs or your usage scenario.
I am encoding a video using MediaCodec using Camera's setPreviewCallback.
(I follow this example Encoding H.264 from camera with Android MediaCodec).
I use the follow setting for the MediaCodec:
mediaCodec = MediaCodec.createEncoderByType("video/avc");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 1280, 720);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 8000000);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar);
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mediaCodec.start();
My Camera setting are:
Camera.Parameters parameters = mCamera.getParameters();
parameters.setPreviewFormat(ImageFormat.NV21);
parameters.setPictureSize(previewWidth, 1280);
parameters.setPreviewSize(previewWidth, 720);
parameters.setPreviewFrameRate(30);
parameters.setPreviewFpsRange(5000,30000);
mCamera.setParameters(parameters);
I got a video but there are two problems:
its colors are wrong.
its playing too fast.
Here is an example video
http://www.youtube.com/watch?v=I1Eg2bvrHLM&feature=youtu.be
Does any of you guys know what are the causes of this two problem? And may be tell me some ways to solve this problem because I am totally lost/confused now.
thanks for reading and would appreciate any comments and opinions.
The YUV formats used by Camera output and MediaCodec input have their U/V planes swapped.
If you are able to move the data through a Surface you can avoid this issue; however, you lose the ability to examine the YUV data. An example of recording to a .mp4 file from a camera can be found on bigflake.
Some details about the colorspaces and how to swap them is in this answer.
There is no timestamp information in the raw H.264 elementary stream. You need to pass the timestamp through the decoder to MediaMuxer or whatever you're using to create your final output. If you don't, the player will just pick a rate, or possibly play the frames as fast as it can.
I am reading the Android documents about MediaCodec and other online tutorials/examples. As I understand it, the way to use the MediaCodec is like this (decoder example in pseudo code):
//-------- prepare audio decoder, format, buffers, and files --------
MediaExtractor extractor;
MediaCodec codec;
ByteBuffer[] codecInputBuffers;
ByteBuffer[] codecOutputBuffers;
extractor = new MediaExtractor();
extractor.setDataSource();
MediaFormat format = extractor.getTrackFormat(0);
//---------------- start decoding ----------------
codec = MediaCodec.createDecoderByType(mime);
codec.configure(format, null /* surface */, null /* crypto */, 0 /* flags */);
codec.start();
codecInputBuffers = codec.getInputBuffers();
codecOutputBuffers = codec.getOutputBuffers();
extractor.selectTrack(0);
//---------------- decoder loop ----------------
while (MP3_file_not_EOS) {
//-------- grasp control of input buffer from codec --------
codec.dequeueInputBuffer();
//---- fill input buffer with data from MP3 file ----
extractor.readSampleData();
//-------- release input buffer so codec can have it --------
codec.queueInputBuffer();
//-------- grasp control of output buffer from codec --------
codec.dequeueOutputBuffer();
//-- copy PCM samples from output buffer into another buffer --
short[] PCMoutBuffer = copy_of(OutputBuffer);
//-------- release output buffer so codec can have it --------
codec.releaseOutputBuffer();
//-------- write PCMoutBuffer into a file, or play it -------
}
//---------------- stop decoding ----------------
codec.stop();
codec.release();
Is this the right way to use the MediaCodec? If not, please enlighten me with the right approach. If this is the right way, how do I measure the performance of the MediaCodec? Is it the time difference between when codec.dequeueOutputBuffer() returns and when codec.queueInputBuffer() returns? I'd like an accuracy/precision of microseconds. Your ideas and thoughts are appreciated.
(merging comments and expanding slightly)
You can't simply time how long a single buffer submission takes, because the codec might want to queue up more than one buffer before doing anything. You will need to measure it in aggregate, timing the duration of the entire file decode with System.nanoTime(). If you turn the copy_of operation into a no-op and just discard the decoded data, you'll keep the output side (writing the decoded data to disk) out of the calculation.
Excluding the I/O from the input side is more difficult. As noted in the MediaCodec docs, the encoded input/output "is not a stream of bytes, it's a stream of access units". So you'd have to populate any necessary codec-specific-data keys in MediaFormat, and then identify individual frames of input so you can properly feed the codec.
An easier but less accurate approach would be to conduct a separate pass in which you time how long it takes to read the input data, and then subtract that from the total time. In your sample code, you would keep the operations on extractor (like readSampleData), but do nothing with codec (maybe dequeue one buffer and just re-use it every time). That way you only measure the MediaExtractor overhead. The trick here is to run it twice, immediately before the full test, and ignore the results from the first -- the first pass "warms up" the disk cache.
If you're interested in performance differences between devices, it may be the case that the difference in input I/O time, especially from a "warm" cache, is similar enough and small enough that you can just disregard it and not go through all the extra gymnastics.