Develop Reference

Android - Decoding h264 raw stream manually

So I am trying to decode a stream of raw h264 data and render it to a surface on Android. Here are the steps:
Receive a packet of h264 stream
Accumulate it and try to extract NAL units (byte sequences starting with 00 00 00 01 (NAL header) and up until the next NAL header.
For every extracted NAL unit, call feedFrame(data) where data is a byte[] that starts with NAL header and contains the extracted unit.
See the video rendered on the surface I provided.
The following code does utilizes the AVC decoder:
public StreamReceiver(DashCamActivity activity, Surface surface, int width, int height, byte[] sps, byte[] pps) {
this.activity = activity;
decoder = MediaCodec.createDecoderByType("video/avc");
format.setByteBuffer("csd-0", ByteBuffer.wrap(sps));
format.setByteBuffer("csd-1", ByteBuffer.wrap(pps));
decoder.configure(format, surface, null, 0);
decoder.start();
}
public void shutdown()
{
decoder.stop();
decoder.release();
}
public void feedFrame(byte[] data)
{
BufferInfo info = new BufferInfo();
int inputIndex = decoder.dequeueInputBuffer(1000);
if(inputIndex == -1)
return;
ByteBuffer inputBuffer = decoder.getInputBuffers()[inputIndex];
if (inputIndex >= 0) {
inputBuffer.clear();
inputBuffer.put(data, 0, data.length);
inputBuffer.clear();
decoder.queueInputBuffer(inputIndex, 0, data.length, 0, 0);
}
int outIndex = decoder.dequeueOutputBuffer(info, 1000);
switch (outIndex) {
case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
break;
case MediaCodec.INFO_TRY_AGAIN_LATER:
break;
case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:
break;
default:
decoder.releaseOutputBuffer(outIndex, true);
break;
}
}
For smaller resolutions (1024x768, 1280x800) everything works perfectly. However with larger resolutions (1920x1080, 1900x600), where the length of the byte array I provide is above 65535 (64k), the video starts having stutters and artifacts and Logcat reports strange decoder errors (e.g. IOCTL_MFC_DEC_EXE failed(ret : -2001) on Galaxy S3).
This also happens on a relatively new device that can play 4k with twice the framerate I provide. So I must be doing something wrong, and I don't know if my 64k theory has any truth in it, it's merely an observation.
So to recap:
I am providing individual NAL units to the decoder, starting with the
header.
The h264 stream is of baseline profile, level 4.0.
Writing the contents of the NAL units to a file in the order they arrive produces a video file that is fully playable using the basic media players
How do I get it to play at high resolutions?

How to achive better quality with libav on android surface?

Playing video from RTSP stream on Android surface using solution from this repo
https://github.com/alexandruc/android-ffmpeg-player/blob/master/android-ffmpeg-player/jni/ffmpeg-player.c
Video is playng but have a lot of glitches especially when something moving.
Have not enought expirience using libav.
Will be happy if someone can help or give links on some tutorials or community.
Here is the function to display video on surface.
void* decodeAndRender(void *voidArgs) {
auto *args = (decode_args*)voidArgs;
CamCon* cc = getCamCon(args->name);
ANativeWindow_Buffer windowBuffer;
AVPacket packet;
int i=0;
int frameFinished;
int lineCnt;
int counter = 0;
while(av_read_frame(cc->formatCtx, &packet)>=0 && cc->isConnect) {
counter = 1;
// Is this a packet from the video stream?
if(packet.stream_index==cc->videoStreamIdx) {
// Decode video frame
avcodec_decode_video2(cc->codecCtx, cc->decodedFrame, &frameFinished, &packet);
// Did we get a video frame?
if(frameFinished) {
// RECORD video
if(cc->isRecord)
recordMP4(packet, cc);
// DISPLAY video
// Convert the image from its native format to RGBA
sws_scale (
cc->sws_ctx,
(uint8_t const * const *)cc->decodedFrame->data,
cc->decodedFrame->linesize,
0,
cc->codecCtx->height,
cc->frameRGBA->data,
cc->frameRGBA->linesize
);
// lock the window buffer
if (ANativeWindow_lock(cc->window, &windowBuffer, NULL) < 0) {
LOGE("cannot lock window");
} else {
// draw the frame on buffer
LOGI("copy buffer %d:%d:%d", cc->displayWidth, cc->displayHeight, cc->displayWidth * cc->displayHeight*4);
LOGI("window buffer: %d:%d:%d", windowBuffer.width,
windowBuffer.height, windowBuffer.stride);
memcpy(windowBuffer.bits, cc->buffer, cc->displayWidth * cc->displayHeight * 4);
// unlock the window buffer and post it to display
ANativeWindow_unlockAndPost(cc->window);
// count number of frames
++i;
}
}
}
// Free the packet that was allocated by av_read_frame
av_free_packet(&packet);
}
LOGI("total No. of frames decoded and rendered %d", i);
finish(args->env, args->name);
}

FFMPEG sws_scale Crash on Android

I have an app that convert images to video, in Google Play I see the following crash (which the only details I get is the name of the function and I don't understand the rest):
backtrace:
#00 pc 0000cc78 /data/app-lib/com.myapp-1/libswscale.so (sws_scale+204)
#01 pc 000012af /data/app-lib/com.myapp-1/libffmpeg.so (OpenImage+322)
code around pc:
79065c58 e58d8068 e58d2070 e58d3074 059d00b0
The code point to the function sws_scale, the code works almost all the time on my device (Nexus 5) but I see a lot of reports even with the same device with that issue. Any idea why this could happen?
AVFrame* OpenImage(const char* imageFileName, int W_VIDEO, int H_VIDEO, int* numBytes)
{
AVFormatContext *pFormatCtx;
AVCodecContext *pCodecCtx;
AVCodec *pCodec;
AVFrame *pFrame;
int frameFinished;
uint8_t *buffer;
AVPacket packet;
int srcBytes;
AVFrame* frame2 = NULL;// scaled frame
uint8_t* frame2_buffer;
struct SwsContext *resize;
if(av_open_input_file(&pFormatCtx, imageFileName, NULL, 0, NULL)!=0)
{
LOGI("Can't open image file '%s'\n", imageFileName);
return NULL;
}
//dump_format(pFormatCtx, 0, imageFileName, 0);
if (av_find_stream_info(pFormatCtx) < 0)
{
LOGI("Can't find stream info.");
return NULL;
}
pCodecCtx = pFormatCtx->streams[0]->codec;
pCodecCtx->pix_fmt = PIX_FMT_YUV420P;
// Find the decoder for the video stream
pCodec = avcodec_find_decoder(pCodecCtx->codec_id);
if (!pCodec)
{
LOGI("Codec not found\n");
return NULL;
}
// Open codec
if(avcodec_open(pCodecCtx, pCodec)<0)
{
LOGI("Could not open codec\n");
return NULL;
}
pFrame = avcodec_alloc_frame();
if (!pFrame)
{
LOGI("Can't allocate memory for AVFrame\n");
return NULL;
}
// Determine required buffer size and allocate buffer
srcBytes = avpicture_get_size(PIX_FMT_YUV420P, pCodecCtx->width, pCodecCtx->height);
buffer = (uint8_t *) av_malloc(srcBytes * sizeof(uint8_t));
avpicture_fill((AVPicture *) pFrame, buffer, PIX_FMT_YUV420P, pCodecCtx->width, pCodecCtx->height);
// Read frame
if (av_read_frame(pFormatCtx, &packet) >= 0)
{
int ret;
// if(packet.stream_index != 0)
// continue;
ret = avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
if (ret > 0)
{
//LOGI("Frame is decoded, size %d\n", ret);
pFrame->quality = 4;
// Create another frame for resized result
frame2 = avcodec_alloc_frame();
*numBytes = avpicture_get_size(PIX_FMT_YUV420P, W_VIDEO, H_VIDEO);
frame2_buffer = (uint8_t *)av_malloc(*numBytes * sizeof(uint8_t));
avpicture_fill((AVPicture*)frame2, frame2_buffer, PIX_FMT_YUV420P, W_VIDEO, H_VIDEO);
// Get resize context
resize = sws_getContext(pCodecCtx->width, pCodecCtx->height, PIX_FMT_YUV420P, W_VIDEO, H_VIDEO, PIX_FMT_YUV420P, SWS_BICUBIC, NULL, NULL, NULL);
// frame2 should be filled with resized samples
ret = sws_scale(resize, (const uint8_t* const*)pFrame->data, pFrame->linesize, 0, pCodecCtx->height, frame2->data, frame2->linesize);
sws_freeContext(resize);
}
else
LOGI("Error [%d] while decoding frame: %s\n", ret, strerror(AVERROR(ret)));
}
av_free(pFrame);
av_free_packet(&packet);
avcodec_close(pCodecCtx);
//av_free(pCodecCtx);
av_close_input_file(pFormatCtx);
return frame2;
}

After your avcodec_decode_video2, do not check ret only. You need to check frameFinished too. if frameFinished == 0, your frame should not be used (because not filled). I do not know for images, but when you decode video, it happens very often. You need to read the next packet and give to the next call of avcodec_decode_video2.
On a side note: why are you forcing pCodecCtx->pix_fmt = PIX_FMT_YUV420P? It is automatically set to the correct format by av_find_stream_info, and you should use it as sws_getContext parameter.
Last thing: no need to fill your pFramewith avpicture_fill. You only need to av_frame_alloc() it, and avcodec_decode_video2 will take care of filling it.

After this line
*numBytes = avpicture_get_size(PIX_FMT_YUV420P, W_VIDEO, H_VIDEO);
You have not checked the return value.
In the avpicture_get_size describe:
"Returns:
the computed picture buffer size or a negative error code in case of error"
When you check the *numBytes (and srcBytes, buffer, frame2_buffer) value(s), maybe it will be better...

How to display a JPEG on ANativeWindow?

I am trying to show a JPEG to a ANativeWindow with the Android NDK.
I am getting the ANativeWindow* by doing:
_window = ANativeWindow_fromSurface(env, surface)
I am loading the jpeg, with libjpeg-turbo, by doing:
if (tjDecompressHeader2(tj, jpeg, jpegSize, &width, &height, &subsamp) == 0) {
int format = TJPF_ARGB;
int pitch = _windowWidth * tjPixelSize[format];
_bufferSize = pitch * _windowHeight;
_buffer = realloc(_buffer, _bufferSize);
tjDecompress2(tj, jpeg, jpegSize, _buffer, _windowWidth, pitch, _windowHeight, format, 0);
}
My Question is, how to show the decoded jpeg on the surface ? I am currently doing this:
ANativeWindow_Buffer surface_buffer;
if (ANativeWindow_lock(_window, &surface_buffer, NULL) == 0) {
memcpy(surface_buffer.bits, _buffer, _bufferSize);
ANativeWindow_unlockAndPost(_window);
}
But the result (see below) is not what I was expecting. What should I do before sending the buffer to the surface ?
Thanks

Just need to set ANativeWindow's format with ANativeWindow_setBuffersGeometry(_window, 0, 0, WINDOW_FORMAT_RGBA_8888). And then use TJPF_RGBA format instead of TJPF_ARGB.

Converting FFmpeg frame to OpenGL ES texture

I'm trying to convert from a video using FFmpeg to an OpenGL ES texture in jni, but all that I get is a black texture. I have output the OpenGL with the glGetError(), but there is no error.
Here is my code:
void* pixels;
int err;
int i;
int frameFinished = 0;
AVPacket packet;
static struct SwsContext *img_convert_ctx;
static struct SwsContext *scale_context = NULL;
int64_t seek_target;
int target_width = 320;
int target_height = 240;
GLenum error = GL_NO_ERROR;
sws_freeContext(img_convert_ctx);
i = 0;
while((i==0) && (av_read_frame(pFormatCtx, &packet)>=0)) {
if(packet.stream_index==videoStream) {
avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished, &packet);
if(frameFinished) {
LOGI("packet pts %llu", packet.pts);
img_convert_ctx = sws_getContext(pCodecCtx->width, pCodecCtx->height,
pCodecCtx->pix_fmt,
target_width, target_height, PIX_FMT_RGB24, SWS_BICUBIC,
NULL, NULL, NULL);
if(img_convert_ctx == NULL) {
LOGE("could not initialize conversion context\n");
return;
}
sws_scale(img_convert_ctx, (const uint8_t* const*)pFrame->data, pFrame->linesize, 0, pCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize);
LOGI("sws_scale");
videoTextures = new Texture*[1];
videoTextures[0]->mWidth = 256; //(unsigned)pCodecCtx->width;
videoTextures[0]->mHeight = 256; //(unsigned)pCodecCtx->height;
videoTextures[0]->mData = pFrameRGB->data[0];
glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
glGenTextures(1, &(videoTextures[0]->mTextureID));
glBindTexture(GL_TEXTURE_2D, videoTextures[0]->mTextureID);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
if(0 == got_texture)
{
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, videoTextures[0]->mWidth, videoTextures[0]->mHeight, 0, GL_RGBA, GL_UNSIGNED_BYTE, (GLvoid *)videoTextures[0]->mData);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0,0, videoTextures[0]->mWidth, videoTextures[0]->mHeight, GL_RGBA, GL_UNSIGNED_BYTE, (GLvoid *)videoTextures[0]->mData);
}else
{
glTexSubImage2D(GL_TEXTURE_2D, 0, 0,0, videoTextures[0]->mWidth, videoTextures[0]->mHeight, GL_RGBA, GL_UNSIGNED_BYTE, (GLvoid *)videoTextures[0]->mData);
}
i = 1;
error = glGetError();
if( error != GL_NO_ERROR ) {
LOGE("couldn't create texture!!");
switch (error) {
case GL_INVALID_ENUM:
LOGE("GL Error: Enum argument is out of range");
break;
case GL_INVALID_VALUE:
LOGE("GL Error: Numeric value is out of range");
break;
case GL_INVALID_OPERATION:
LOGE("GL Error: Operation illegal in current state");
break;
case GL_OUT_OF_MEMORY:
LOGE("GL Error: Not enough memory to execute command");
break;
default:
break;
}
}
}
}
av_free_packet(&packet);
}
I have succeeded in changing pFrameRGB to a java bitmap， but I just want to change it to a texture in the c code.
Edit1 I have output the Texture ID, it is 0; could texture ID be zero? I changed my code
but it always be zero.
Edit2
the Texture display, but it is a mess.

Not used to GLES, but GL. In there 320, 240 are not valied 512,256 as to power of 2. Else you need to use texture_rectangle extension which texcoords are not from 0-1 but 0-w/h. As for uploading texture data, glTexImage(...) is needed to be used the first time (even with data 0), then glTexSubImage is enough, i think sizing etc is initialized with the first, the second just sends the meat.
Regarding ffmpeg usage, perhaps a version issue but img_context is more near to be renamed to sws_getContext, and initialized only once, if cpu usage is an issue use SWS_LINEAR instead of SWS_CUBIC, also i assume pFrameRGB has been correctly avcodec_alloc_frame()'ed, if you are going to use GL_RGBA you should use PIX_FMT_RGBA, PIX_FMT_RGB24 would be for GL_RGB texture piping, finally you lack a packet stack so you can go reading in advance to keep display good in sync and not late.
I've read some comments about unpack alignment, i didn't need that (and seeing the success in the area doubt of it) to implement an ffmpeg to OpenGL/OpenAL media library (http://code.google.com/p/openmedialibrary), nicely the audio bits have also been extracted to an ffmpeg to OpenAL loader (http://code.google.com/p/openalextensions). Haves some nice features and currently i'm trying to work with texture compression to see if it can perform still better. Consider that tutorials or even ready to use gpl code.
Hope to give some enlightenment on the obscure (by lack) art of ffmpeg to OpenGL/AL integration.

Try to append 16 zero bytes to each packet before passing to decoder.
Some comments from the avcodec.h:
/*
* #warning The input buffer must be FF_INPUT_BUFFER_PADDING_SIZE larger than
* the actual read bytes because some optimized bitstream readers read 32 or 64
* bits at once and could read over the end.
* #warning The end of the input buffer buf should be set to 0 to ensure that
* no overreading happens for damaged MPEG streams.
*/