So I am trying to decode a stream of raw h264 data and render it to a surface on Android. Here are the steps:
Receive a packet of h264 stream
Accumulate it and try to extract NAL units (byte sequences starting with 00 00 00 01 (NAL header) and up until the next NAL header.
For every extracted NAL unit, call feedFrame(data) where data is a byte[] that starts with NAL header and contains the extracted unit.
See the video rendered on the surface I provided.
The following code does utilizes the AVC decoder:
public StreamReceiver(DashCamActivity activity, Surface surface, int width, int height, byte[] sps, byte[] pps) {
this.activity = activity;
decoder = MediaCodec.createDecoderByType("video/avc");
format.setByteBuffer("csd-0", ByteBuffer.wrap(sps));
format.setByteBuffer("csd-1", ByteBuffer.wrap(pps));
decoder.configure(format, surface, null, 0);
decoder.start();
}
public void shutdown()
{
decoder.stop();
decoder.release();
}
public void feedFrame(byte[] data)
{
BufferInfo info = new BufferInfo();
int inputIndex = decoder.dequeueInputBuffer(1000);
if(inputIndex == -1)
return;
ByteBuffer inputBuffer = decoder.getInputBuffers()[inputIndex];
if (inputIndex >= 0) {
inputBuffer.clear();
inputBuffer.put(data, 0, data.length);
inputBuffer.clear();
decoder.queueInputBuffer(inputIndex, 0, data.length, 0, 0);
}
int outIndex = decoder.dequeueOutputBuffer(info, 1000);
switch (outIndex) {
case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
break;
case MediaCodec.INFO_TRY_AGAIN_LATER:
break;
case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
break;
default:
decoder.releaseOutputBuffer(outIndex, true);
break;
}
}
For smaller resolutions (1024x768, 1280x800) everything works perfectly. However with larger resolutions (1920x1080, 1900x600), where the length of the byte array I provide is above 65535 (64k), the video starts having stutters and artifacts and Logcat reports strange decoder errors (e.g. IOCTL_MFC_DEC_EXE failed(ret : -2001) on Galaxy S3).
This also happens on a relatively new device that can play 4k with twice the framerate I provide. So I must be doing something wrong, and I don't know if my 64k theory has any truth in it, it's merely an observation.
So to recap:
I am providing individual NAL units to the decoder, starting with the
header.
The h264 stream is of baseline profile, level 4.0.
Writing the contents of the NAL units to a file in the order they arrive produces a video file that is fully playable using the basic media players
How do I get it to play at high resolutions?
Related
I am currently working on a project which requires the contents(the camera image along with the AR objects for each frame) of the SceneView in a byte array format to stream the data.
I have tried to Mirror the SceneView to a MediaCodec encoder's input surface and use the MediaCodec's output buffer in an asynchronous manner based on what I understood from the MediaRecorder sample.
I have been unable to get it to work as I expect it to. The output buffer from the MediaCodec's callback either displays a black screen (when converted to bitmap) or it has way too little buffer contents (<100 bytes).
I have a feeling that the mirroring to a surface is not occurring as I expect it to, It would be much appreciated if someone could provide a sample that showcases the proper way to use MediaCodec with SceneForm
The code I am currently using for accessing the ByteBuffer from MediaCodec :
MediaFormat format = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC,
arFragment.getArSceneView().getWidth(),
arFragment.getArSceneView().getHeight());
// Set some properties to prevent configure() from throwing an unhelpful exception.
format.setInteger(MediaFormat.KEY_COLOR_FORMAT,
MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface);
format.setInteger(MediaFormat.KEY_BIT_RATE, 50000);
format.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 0); //All key frame stream
MediaCodec mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
mediaCodec.setCallback(new MediaCodec.Callback() {
#Override
public void onInputBufferAvailable(#NonNull MediaCodec codec, int index) {
}
#Override
public void onOutputBufferAvailable(#NonNull MediaCodec codec, int index, #NonNull MediaCodec.BufferInfo info) {
ByteBuffer outputBuffer = mediaCodec.getOutputBuffer(index);
outputBuffer.position(info.offset);
outputBuffer.limit(info.offset + info.size);
byte[] data = new byte[outputBuffer.remaining()];
outputBuffer.get(data);
Bitmap bitmap = BitmapFactory.decodeByteArray(
NV21toJPEG(data, videoWidth, videoHeight, 100),
0, data.length);
Log.d(TAG, "onOutputBufferAvailable: "+bitmap);
mediaCodec.releaseOutputBuffer(index, false);
}
#Override
public void onError(#NonNull MediaCodec codec, #NonNull MediaCodec.CodecException e) {
Log.e(TAG, "onError: ");
}
#Override
public void onOutputFormatChanged(#NonNull MediaCodec codec, #NonNull MediaFormat format)
{
Log.d(TAG, "onOutputFormatChanged: " + format);
}
});
mediaCodec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
Surface surface = mediaCodec.createInputSurface();
mediaCodec.start();
arFragment.getArSceneView().startMirroringToSurface(surface, 0, 0, arFragment.getArSceneView().getWidth(), arFragment.getArSceneView().getHeight());
The generated bitmap :
The input to the MediaCodec encoder is a Surface (in your case, could be a byte buffer as well) with raw YUV data (not directly accessible). The output from the encoder is an encoded H264/AVC bitstream.
In your case, it seems like you are trying to read the encoded bitstream and interpret it as raw YUV data.
Do you really want to encode the input data into a video format (for streaming?), or are you just trying to use MediaCodec to convert data from a Surface into YUV data accessible in a byte buffer? You could get a hand on the YUV data creating a MediaCodec decoder and promptly feeding data from the encoder to the decoder, but that's all very roundabout.
The direct way of receiving data through a Surface and accessing the pixel data is via the ImageReader class. See Using Camera2 API with ImageReader for a bigger example of using this.
Playing video from RTSP stream on Android surface using solution from this repo
https://github.com/alexandruc/android-ffmpeg-player/blob/master/android-ffmpeg-player/jni/ffmpeg-player.c
Video is playng but have a lot of glitches especially when something moving.
Have not enought expirience using libav.
Will be happy if someone can help or give links on some tutorials or community.
Here is the function to display video on surface.
void* decodeAndRender(void *voidArgs) {
auto *args = (decode_args*)voidArgs;
CamCon* cc = getCamCon(args->name);
ANativeWindow_Buffer windowBuffer;
AVPacket packet;
int i=0;
int frameFinished;
int lineCnt;
int counter = 0;
while(av_read_frame(cc->formatCtx, &packet)>=0 && cc->isConnect) {
counter = 1;
// Is this a packet from the video stream?
if(packet.stream_index==cc->videoStreamIdx) {
// Decode video frame
avcodec_decode_video2(cc->codecCtx, cc->decodedFrame, &frameFinished, &packet);
// Did we get a video frame?
if(frameFinished) {
// RECORD video
if(cc->isRecord)
recordMP4(packet, cc);
// DISPLAY video
// Convert the image from its native format to RGBA
sws_scale (
cc->sws_ctx,
(uint8_t const * const *)cc->decodedFrame->data,
cc->decodedFrame->linesize,
0,
cc->codecCtx->height,
cc->frameRGBA->data,
cc->frameRGBA->linesize
);
// lock the window buffer
if (ANativeWindow_lock(cc->window, &windowBuffer, NULL) < 0) {
LOGE("cannot lock window");
} else {
// draw the frame on buffer
LOGI("copy buffer %d:%d:%d", cc->displayWidth, cc->displayHeight, cc->displayWidth * cc->displayHeight*4);
LOGI("window buffer: %d:%d:%d", windowBuffer.width,
windowBuffer.height, windowBuffer.stride);
memcpy(windowBuffer.bits, cc->buffer, cc->displayWidth * cc->displayHeight * 4);
// unlock the window buffer and post it to display
ANativeWindow_unlockAndPost(cc->window);
// count number of frames
++i;
}
}
}
// Free the packet that was allocated by av_read_frame
av_free_packet(&packet);
}
LOGI("total No. of frames decoded and rendered %d", i);
finish(args->env, args->name);
}
I'm writing a plugin for Unity that decodes and takes the frames from a video file using the Media Extractor and re-encodes to a new video file. However the frames are being decoded into an array of the wrong size (on android 4.2.2) because the codec thinks the height is 736 when it is actually 720.
for (int i = 0; i < numTracks; ++i)
{
MediaFormat format = extractor.getTrackFormat(i);
String mime = format .getString(MediaFormat.KEY_MIME);
if(mime.startsWith("video/"))
{
extractor.selectTrack(i);
//Decoder
decoder = MediaCodec.createDecoderByType(mime);
decoder.configure(format, null, null, 0);
break;
}
}
The output buffer index returns INFO_OUTPUT_BUFFERS_CHANGED and then INFO_OUTPUT_FORMAT_CHANGED. Logging this informs me that decoder thinks there is a height of 736 instead of the correct 720.
decoder.queueInputBuffer(inputBufIndex, 0, sampleSize, extractor.getSampleTime(), 0);
//Get Outputbuffer Index
int outIndex = decoder.dequeueOutputBuffer(info, 10000);
This works fine on a device running 4.4, the problem is only present on an older 4.2 device. Anyone have any thoughts?
Keep in mind that you need to check the crop fields in MediaFormat as well, the height field is the full height of the output buffer including potential padding. See e.g. the checkFrame function in https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/EncodeDecodeTest.java - you'll get the actual content height as format.getInteger("crop-bottom") - format.getInteger("crop-top") + 1.
I have successfully implemented encoding/decoding raw AAC-LC using MediaCodec. I'm using the same technique described here to encode the data. However, I store the raw AAC data (without headers) and then attach headers on the fly as I pass the data through a MediaCodec decoder. This all works absolutely perfectly on the Nexus 4 and Nexus 5 both running Android 4.4. However, on the Galaxy Nexus (running Android 4.3) I get:
W/SoftAAC2(1234): AAC decoder returned error 16388, substituting silence
Error 16388 means a decode frame error.
I've tried with and without an initial MediaCodec.BUFFER_FLAG_CODEC_CONFIG but that doesn't make a difference.
Here is the simplest case (using config packet) to reproduce the error:
MediaFormat format = new MediaFormat();
format.setInteger(MediaFormat.KEY_SAMPLE_RATE, 44100);
format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
format.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC);
format.setInteger(MediaFormat.KEY_IS_ADTS, 1);
byte[] bytes = new byte[]{(byte) 0x11, (byte)0x90};
ByteBuffer bb = ByteBuffer.wrap(bytes);
format.setByteBuffer("csd-0", bb);
MediaCodec codec = MediaCodec.createDecoderByType("audio/mp4a-latm");
codec.configure(format, null /* surface */, null /* crypto */, 0 /* flags */);
codec.start();
ByteBuffer[] codecInputBuffers = codec.getInputBuffers();
int inputBufIndex = codec.dequeueInputBuffer(TIMEOUT_US);
if (inputBufIndex >= 0) {
ByteBuffer dstBuf = codecInputBuffers[inputBufIndex];
byte[] data = {-1, -7, 80, 0, 1, 63, -4, 18, 8}; // values taken from SO answer linked above (chanCfg = 1, packetLen = 9)
dstBuf.clear();
dstBuf.put(data);
codec.queueInputBuffer(inputBufIndex, 0, data.length, 0, MediaCodec.BUFFER_FLAG_CODEC_CONFIG);
}
Obviously there is a lot more to the code than this, but this includes all the executed code up to the point of the error message.
The solution is to not include the ADTS headers. Both 4.3 and 4.4 support packets without ADTS headers.
I'm trying to get this to work on Android 4.1 (using an upgraded Asus Transformer tablet). Thanks to Alex's response to my previous question, I already was able to write some raw H.264 data to a file, but this file is only playable with ffplay -f h264, and it seems like it's lost all information regarding the framerate (extremely fast playback). Also the color-space looks incorrect (atm using the camera's default on encoder's side).
public class AvcEncoder {
private MediaCodec mediaCodec;
private BufferedOutputStream outputStream;
public AvcEncoder() {
File f = new File(Environment.getExternalStorageDirectory(), "Download/video_encoded.264");
touch (f);
try {
outputStream = new BufferedOutputStream(new FileOutputStream(f));
Log.i("AvcEncoder", "outputStream initialized");
} catch (Exception e){
e.printStackTrace();
}
mediaCodec = MediaCodec.createEncoderByType("video/avc");
MediaFormat mediaFormat = MediaFormat.createVideoFormat("video/avc", 320, 240);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 125000);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 15);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar);
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mediaCodec.start();
}
public void close() {
try {
mediaCodec.stop();
mediaCodec.release();
outputStream.flush();
outputStream.close();
} catch (Exception e){
e.printStackTrace();
}
}
// called from Camera.setPreviewCallbackWithBuffer(...) in other class
public void offerEncoder(byte[] input) {
try {
ByteBuffer[] inputBuffers = mediaCodec.getInputBuffers();
ByteBuffer[] outputBuffers = mediaCodec.getOutputBuffers();
int inputBufferIndex = mediaCodec.dequeueInputBuffer(-1);
if (inputBufferIndex >= 0) {
ByteBuffer inputBuffer = inputBuffers[inputBufferIndex];
inputBuffer.clear();
inputBuffer.put(input);
mediaCodec.queueInputBuffer(inputBufferIndex, 0, input.length, 0, 0);
}
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
int outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo,0);
while (outputBufferIndex >= 0) {
ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
byte[] outData = new byte[bufferInfo.size];
outputBuffer.get(outData);
outputStream.write(outData, 0, outData.length);
Log.i("AvcEncoder", outData.length + " bytes written");
mediaCodec.releaseOutputBuffer(outputBufferIndex, false);
outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, 0);
}
} catch (Throwable t) {
t.printStackTrace();
}
}
Changing the encoder type to "video/mp4" apparently solves the framerate-problem, but since the main goal is to make a streaming service, this is not a good solution.
I'm aware that I dropped some of Alex' code considering the SPS and PPS NALU's, but I was hoping this would not be necessary since that information was also coming from outData and I assumed the encoder would format this correctly. If this is not the case, how should I arrange the different types of NALU's in my file/stream?
So, what am I missing here in order to make a valid, working H.264 stream? And which settings should I use to make a match between the camera's colorspace and the encoder's colorspace?
I have a feeling this is more of a H.264-related question than a Android/MediaCodec topic. Or am I still not using the MediaCodec API correctly?
Thanks in advance.
For your fast playback - frame rate issue, there is nothing you have to do here. Since it is a streaming solution the other side has to be told the frame rate in advance or timestamps with each frame. Both of these are not part of elementary stream. Either pre-determined framerate is chosen or you pass on some sdp or something like that or you use existing protocols like rtsp. In the second case the timestamps are part of the stream sent in form of something like rtp. Then the client has to depay the rtp stream and play it bacl. This is how elementary streaming works. [either fix your frame rate if you have a fixed rate encoder or give timestamps]
Local PC playback will be fast because it will not know the fps. By giving the fps parameter before the input e.g
ffplay -fps 30 in.264
you can control the playback on the PC.
As for the file not being playable: Does it have a SPS and PPS. Also you should have NAL headers enabled - annex b format. I don't know much about android, but this is requirement for any h.264 elementary stream to be playable when they are not in any containers and need to be dumped and played later.
If android default is mp4, but default annexb headers will be switched off, so perhaps there is a switch to enable it. Or if you are getting data frame by frame, just add it yourself.
As for color format: I would guess the default should work. So try not setting it.
If not try 422 Planar or UVYV / VYUY interleaved formats. usually cameras are one of those. (but not necessary, these may be the ones I have encountered more often).
Android 4.3 (API 18) provides an easy solution. The MediaCodec class now accepts input from Surfaces, which means you can connect the camera's Surface preview to the encoder and bypass all the weird YUV format issues.
There is also a new MediaMuxer class that will convert your raw H.264 stream to a .mp4 file (optionally blending in an audio stream).
See the CameraToMpegTest source for an example of doing exactly this. (It also demonstrates the use of an OpenGL ES fragment shader to perform a trivial edit on the video as it's being recorded.)
You can convert color spaces like this, if you have set the preview color space to YV12:
public static byte[] YV12toYUV420PackedSemiPlanar(final byte[] input, final byte[] output, final int width, final int height) {
/*
* COLOR_TI_FormatYUV420PackedSemiPlanar is NV12
* We convert by putting the corresponding U and V bytes together (interleaved).
*/
final int frameSize = width * height;
final int qFrameSize = frameSize/4;
System.arraycopy(input, 0, output, 0, frameSize); // Y
for (int i = 0; i < qFrameSize; i++) {
output[frameSize + i*2] = input[frameSize + i + qFrameSize]; // Cb (U)
output[frameSize + i*2 + 1] = input[frameSize + i]; // Cr (V)
}
return output;
}
Or
public static byte[] YV12toYUV420Planar(byte[] input, byte[] output, int width, int height) {
/*
* COLOR_FormatYUV420Planar is I420 which is like YV12, but with U and V reversed.
* So we just have to reverse U and V.
*/
final int frameSize = width * height;
final int qFrameSize = frameSize/4;
System.arraycopy(input, 0, output, 0, frameSize); // Y
System.arraycopy(input, frameSize, output, frameSize + qFrameSize, qFrameSize); // Cr (V)
System.arraycopy(input, frameSize + qFrameSize, output, frameSize, qFrameSize); // Cb (U)
return output;
}
You can query the MediaCodec for it's supported bitmap format and query your preview.
Problem is, some MediaCodecs only support proprietary packed YUV formats that you can't get from the preview.
Particularly 2130706688 = 0x7F000100 = COLOR_TI_FormatYUV420PackedSemiPlanar .
Default format for the preview is 17 = NV21 = MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV411Planar = YCbCr 420 Semi Planar
If you did not explicitly request another pixel format, the camera preview buffers will arrive in a YUV 420 format known as NV21, for which COLOR_FormatYCrYCb is the MediaCodec equivalent.
Unfortunately, as other answers on this page mention, there is no guarantee that on your device, the AVC encoder supports this format. Note that there exist some strange devices that do not support NV21, but I don't know any that can be upgraded to API 16 (hence, have MediaCodec).
Google documentation also claims that YV12 planar YUV must be supported as camera preview format for all devices with API >= 12. Therefore, it may be useful to try it (the MediaCodec equivalent is COLOR_FormatYUV420Planar which you use in your code snippet).
Update: as Andrew Cottrell reminded me, YV12 still needs chroma swapping to become COLOR_FormatYUV420Planar.