Develop Reference

MediaExtractor returns wrong number of tracks?

I'm trying to build a player that plays mpeg2-ts stream with 2 audio tracks using MediaCodec and MediaExtractor. When I set the URL to the extractor: extractor.setDataSource(URL) int the Logcat I can see that the framework has found the 2 audio tracks:
But afterwards I call:
int trackCount = extractor.getTrackCount();
for (int i = 0; i < track_count; i++){
format = extractor.getTrackFormat(i);
String mime = format.getString(MediaFormat.KEY_MIME);
if (mime.startsWith("video/")) ...
if (mime.startsWith("audio/")) ...
}
trackCount aways equals 2(1 audio track & 1 video track). What am I doing wrong?

You're not doing anything wrong - it just seems that the MPEG2TSExtractor class (the actual implementation behind MediaExtractor for mpeg2 ts files) only supports one audio stream and one video stream.
See e.g. the init method in https://android.googlesource.com/platform/frameworks/av/+/1a9c3954a/media/libstagefright/mpeg2ts/MPEG2TSExtractor.cpp (lines 156-193). So if you need to demux any mpeg2 ts streams with multiple audio streams, you basically need to bundle a demuxer of your own.

extract all video frames in Android

I recorded a Video for limited time. Now i want to fetch all frames of video. I am using the below code and by using it i am able to get frames but i am not getting all video frames. 3 to 4 frames are repeated then i got a different frame. But as we all know we can get 25- 30 frames in a second to display smooth video. How to get all frames.
for (int i = 0; i < 30; i++) {
Bitmap bArray = mediaMetadataRetriever.getFrameAtTime(
1000000 * i,
MediaMetadataRetriever.OPTION_CLOSEST);
savebitmap(bArray, 33333 * i);
}
I don't want to use NDK. I got this link don't know what should be the value for "argb8888". I am getting error here. Can anyone explain how to do it.
Getting frames from Video Image in Android

I faced the same problem before and the Android's MediaMetadataRetriever seems not appropriated for this task since it doesn't have a good precision.
I used a library called "FFmpegMediaMetadataRetriever" in android studio:
Add this line in your build.graddle under module app:
compile 'com.github.wseemann:FFmpegMediaMetadataRetriever:1.0.14'
Rebuild your project.
Use the FFmpegMediaMetadataRetriever class to grab frames with higher
precision:
FFmpegMediaMetadataRetriever med = new FFmpegMediaMetadataRetriever();
med.setDataSource("your data source");
and in your loop you can grab frame using:
Bitmap bmp = med.getFrameAtTime(i*1000000, FFmpegMediaMetadataRetriever.OPTION_CLOSEST);

To get image frames from video we can use ffmpeg.For integrating FFmpeg in android we can use precompiled libraries like ffmpeg-android.
To extract image frames from a video we can use below command
String[] complexCommand = {"-y", "-i", inputFileAbsolutePath, "-an",
"-r", "1/2", "-ss", "" + startMs / 1000, "-t", "" + (endMs - startMs)
/ 1000, outputFileAbsolutePath};
Here,
-y
Overwrite output files
-i
ffmpeg reads from an arbitrary number of input “files” specified by the -i option
-an
Disable audio recording.
-r
Set frame rate
-ss
seeks to position
-t
limit the duration of data read from the input file
Here in place of inputFileAbsolutePath you have to specify the absolute path of video file from which you want to extract images.
For complete code check out this on my repository .Inside extractImagesVideo() method I am running command for extracting images from video.
For complete tutorial regarding integration of ffmpeg library and using ffmpeg commands to edit videos, check out this post which I have written on my blog.

You need to do :
Decode the video.
Present the decoded images at least as fast as 24 images / second. I
suppose you can skip this step.
Save the decoded images.
It appears that decoding the video would be the most challenging step. People and companies have spent years developing codecs (encoder / decoder) for various video formats.
Use this library JMF for FFMPEG.

how to read video file and split it into frames

I have this question: how can I load, in Android, a video file stored in my device, and how can I split it into frames?
I'm using IntelliJ and I want to split the video into frames in order to process them with some image processing techniques (with OpenCv for Android library).

You don't strictly need to use OpenCV for this. You can use the MediaMetaDataRetreiver class provided by the SDK. It provides methods to extract metadata from all kinds of media files. You can try something like:
MediaMetadataRetriever retriever = new MediaMetadataRetriever();
retriever.setDataSource(file.getAbsolutePath());
imgView.setImageBitmap(retriever.getFrameAtTime(TIME_OFFSET,MediaMetadataRetriever.OPTION_CLOSEST));
where TIME_OFFSET is in microseconds.

Grabbing a video frame in OpenCV is pretty easy. There are lots of examples on OpenCV site. However crucial thing is to set-up opencv on andriod. You can follow this link on getting started with Opencv on andriod.
http://opencv.org/android
Once you have opencv installed on andriod. You can easily load video file and grab frame in Mat structure and then do some processing on it.
Here is the sample one. It will need some modification to run it on andriod. I think you will need to used NDK on andriod for this.
int main(int argc, char*argv[])
{
char *my_file = "C:\\vid_an2\\desp_me.avi";
std::cout<<"Video File "<<my_file<<std::endl;
cv::VideoCapture input_video;
if(input_video.open(my_file))
{
std::cout<<"Video file open "<<std::endl;
}
else
{
std::cout<<"Not able to Video file open "<<std::endl;
}
namedWindow("My_Win",1);
namedWindow("Segemented", 1);
Mat cap_img;
for(;;)
{
input_video >> cap_img;
imshow("My_Win", cap_img);
waitKey(0);
}
return 0;
}

Retrieve album art using FFmpeg

I'm developing an Android application that relies on FFmpeg to retrieve audio metadata. I know it's possible to retrieve album art programmatically using FFMpeg. However, once you have decoded the art (a video frame within an MP3) how do generate an image file (a PNG) for use within an application? I've search all over but can't seem to find a working example.
Edit, here is the solution:
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
void retrieve_album_art(const char *path, const char *album_art_file) {
int i, ret = 0;
if (!path) {
printf("Path is NULL\n");
return;
}
AVFormatContext *pFormatCtx = avformat_alloc_context();
printf("Opening %s\n", path);
// open the specified path
if (avformat_open_input(&pFormatCtx, path, NULL, NULL) != 0) {
printf("avformat_open_input() failed");
goto fail;
}
// read the format headers
if (pFormatCtx->iformat->read_header(pFormatCtx) < 0) {
printf("could not read the format header\n");
goto fail;
}
// find the first attached picture, if available
for (i = 0; i < pFormatCtx->nb_streams; i++)
if (pFormatCtx->streams[i]->disposition & AV_DISPOSITION_ATTACHED_PIC) {
AVPacket pkt = pFormatCtx->streams[i]->attached_pic;
FILE* album_art = fopen(album_art_file, "wb");
ret = fwrite(pkt.data, pkt.size, 1, album_art);
fclose(album_art);
av_free_packet(&pkt);
break;
}
if (ret) {
printf("Wrote album art to %s\n", album_art_file);
}
fail:
av_free(pFormatCtx);
// this line crashes for some reason...
//avformat_free_context(pFormatCtx);
}
int main() {
avformat_network_init();
av_register_all();
const char *path = "some url";
const char *album_art_file = "some path";
retrieve_album_art(path, album_art_file);
return 0;
}

To use ffmpeg programmatically, I think you would have to call read_apic() in libavformat (which is part of ffmpeg).
From the commandline, you can apparently do this:
ffmpeg -i input.mp3 -an -vcodec copy cover.jpg
The commandline behaviour implies that the cover art image is seen as just another video stream (containing just one frame), so using libavformat in the usual way you would to demux the video part of a stream should produce that image.
Sample code for demuxing: ffmpeg/docs/examples/demuxing.c The first (and only) AVPacket that would be obtained from demuxing the video stream in an mp3 would contain the JPEG file (still encoded as JPEG, not decoded).
AVFormatContext* fmt_ctx;
// set up fmt_ctx to read first video stream
AVPacket pkt;
av_read_frame(fmt_ctx, &pkt);
FILE* image_file = fopen("image.jpg", "wb");
int result = fwrite(pkt.data, pkt.size, 1, image_file);
fclose(image_file);
If there are multiple images, I think they would be seen as separate video streams, rather than as separate packets in the same stream. The first stream would be the one with the largest resolution.
All this is probably implemented internally in terms of read_apic().
The ID3v2 spec allows for any image format, but recommends JPEG or PNG. In practice all images in ID3 are JPEG.
EDIT: Moved some of the less useful bits to postscript:
P.S. ffmpeg -i input.mp3 -f ffmetadata metadata.txt will produce an ini-like file containing the metadata, but the image is not even referred to in there, so that is not a useful approach.
P.S. There may be multiple images in an ID3v2 tag. You may have to handle the case when there is more than one image or more than one type of image present.
P.S. ffmpeg is probably not the best software for this. Use id3lib, TagLib, or one of the other implementations of ID3. These can be used either as libraries (callable from the language of your choice) or as commandline utilities. There is sample C++ code for TagLib here: How do I use TagLib to read/write coverart in different audio formats? and for id3lib here: How to get album art from audio files using id3lib.

As an addition the answer above, I needed a way to resize the output image as well so I found the below command while experimenting with the command in the current answer:
ffmpeg -i input.mp3 -filter:v scale=-2:250 -an output.jpeg
So this basically scales the output image to whatever ratio or value you want.

Android - Include native StageFright features in my own project

I am currently developing an application that needs to record audio, encode it as AAC, stream it, and do the same in reverse - receiving stream, decoding AAC and playing audio.
I successfully recorded AAC (wrapped in a MP4 container) using the MediaRecorder, and successfully up-streamed audio using the AudioRecord class. But, I need to be able to encode the audio as I stream it, but none of these classes seem to help me do that.
I researched a bit, and found that most people that have this problem end up using a native library like ffmpeg.
But I was wondering, since Android already includes StageFright, that has native code that can do encoding and decoding (for example, AAC encoding and AAC decoding), is there a way to use this native code on my application? How can I do that?
It would be great if I only needed to implement some JNI classes with their native code. Plus, since it is an Android library, it would be no licensing problems whatever (correct me if I'm wrong).

yes, you can use libstagefright, it's very powerful.
Since stagefright is not exposed to NDK, so you will have to do extra work.
There are two ways:
(1) build your project using android full source tree. This way takes a few days to setup, once ready, it's very easy, and you can take full advantage of stagefright.
(2) you can just copy include file to your project, it's inside this folder:
android-4.0.4_r1.1/frameworks/base/include/media/stagefright
then you will have export the library function by dynamically loading libstagefright.so, and you can link with your jni project.
To encode/decode using statgefright, it's very straightforward, a few hundred of lines can will do.
I used stagefright to capture screenshots to create a video, which will be available in our Android VNC server, to be released soon.
the following is a snippet, I think it's better than using ffmpeg to encode a movie. You can add audio source as well.
class ImageSource : public MediaSource {
ImageSource(int width, int height, int colorFormat)
: mWidth(width),
mHeight(height),
mColorFormat(colorFormat)
{
}
virtual status_t read(
MediaBuffer **buffer, const MediaSource::ReadOptions *options) {
// here you can fill the buffer with your pixels
}
...
};
int width = 720;
int height = 480;
sp<MediaSource> img_source = new ImageSource(width, height, colorFormat);
sp<MetaData> enc_meta = new MetaData;
// enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_H263);
// enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_MPEG4);
enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_AVC);
enc_meta->setInt32(kKeyWidth, width);
enc_meta->setInt32(kKeyHeight, height);
enc_meta->setInt32(kKeySampleRate, kFramerate);
enc_meta->setInt32(kKeyBitRate, kVideoBitRate);
enc_meta->setInt32(kKeyStride, width);
enc_meta->setInt32(kKeySliceHeight, height);
enc_meta->setInt32(kKeyIFramesInterval, kIFramesIntervalSec);
enc_meta->setInt32(kKeyColorFormat, colorFormat);
sp<MediaSource> encoder =
OMXCodec::Create(
client.interface(), enc_meta, true, image_source);
sp<MPEG4Writer> writer = new MPEG4Writer("/sdcard/screenshot.mp4");
writer->addSource(encoder);
// you can add an audio source here if you want to encode audio as well
//
//sp<MediaSource> audioEncoder =
// OMXCodec::Create(client.interface(), encMetaAudio, true, audioSource);
//writer->addSource(audioEncoder);
writer->setMaxFileDuration(kDurationUs);
CHECK_EQ(OK, writer->start());
while (!writer->reachedEOS()) {
fprintf(stderr, ".");
usleep(100000);
}
err = writer->stop();