I'm developing an Android application that relies on FFmpeg to retrieve audio metadata. I know it's possible to retrieve album art programmatically using FFMpeg. However, once you have decoded the art (a video frame within an MP3) how do generate an image file (a PNG) for use within an application? I've search all over but can't seem to find a working example.
Edit, here is the solution:
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
void retrieve_album_art(const char *path, const char *album_art_file) {
int i, ret = 0;
if (!path) {
printf("Path is NULL\n");
return;
}
AVFormatContext *pFormatCtx = avformat_alloc_context();
printf("Opening %s\n", path);
// open the specified path
if (avformat_open_input(&pFormatCtx, path, NULL, NULL) != 0) {
printf("avformat_open_input() failed");
goto fail;
}
// read the format headers
if (pFormatCtx->iformat->read_header(pFormatCtx) < 0) {
printf("could not read the format header\n");
goto fail;
}
// find the first attached picture, if available
for (i = 0; i < pFormatCtx->nb_streams; i++)
if (pFormatCtx->streams[i]->disposition & AV_DISPOSITION_ATTACHED_PIC) {
AVPacket pkt = pFormatCtx->streams[i]->attached_pic;
FILE* album_art = fopen(album_art_file, "wb");
ret = fwrite(pkt.data, pkt.size, 1, album_art);
fclose(album_art);
av_free_packet(&pkt);
break;
}
if (ret) {
printf("Wrote album art to %s\n", album_art_file);
}
fail:
av_free(pFormatCtx);
// this line crashes for some reason...
//avformat_free_context(pFormatCtx);
}
int main() {
avformat_network_init();
av_register_all();
const char *path = "some url";
const char *album_art_file = "some path";
retrieve_album_art(path, album_art_file);
return 0;
}
To use ffmpeg programmatically, I think you would have to call read_apic() in libavformat (which is part of ffmpeg).
From the commandline, you can apparently do this:
ffmpeg -i input.mp3 -an -vcodec copy cover.jpg
The commandline behaviour implies that the cover art image is seen as just another video stream (containing just one frame), so using libavformat in the usual way you would to demux the video part of a stream should produce that image.
Sample code for demuxing: ffmpeg/docs/examples/demuxing.c The first (and only) AVPacket that would be obtained from demuxing the video stream in an mp3 would contain the JPEG file (still encoded as JPEG, not decoded).
AVFormatContext* fmt_ctx;
// set up fmt_ctx to read first video stream
AVPacket pkt;
av_read_frame(fmt_ctx, &pkt);
FILE* image_file = fopen("image.jpg", "wb");
int result = fwrite(pkt.data, pkt.size, 1, image_file);
fclose(image_file);
If there are multiple images, I think they would be seen as separate video streams, rather than as separate packets in the same stream. The first stream would be the one with the largest resolution.
All this is probably implemented internally in terms of read_apic().
The ID3v2 spec allows for any image format, but recommends JPEG or PNG. In practice all images in ID3 are JPEG.
EDIT: Moved some of the less useful bits to postscript:
P.S. ffmpeg -i input.mp3 -f ffmetadata metadata.txt will produce an ini-like file containing the metadata, but the image is not even referred to in there, so that is not a useful approach.
P.S. There may be multiple images in an ID3v2 tag. You may have to handle the case when there is more than one image or more than one type of image present.
P.S. ffmpeg is probably not the best software for this. Use id3lib, TagLib, or one of the other implementations of ID3. These can be used either as libraries (callable from the language of your choice) or as commandline utilities. There is sample C++ code for TagLib here: How do I use TagLib to read/write coverart in different audio formats? and for id3lib here: How to get album art from audio files using id3lib.
As an addition the answer above, I needed a way to resize the output image as well so I found the below command while experimenting with the command in the current answer:
ffmpeg -i input.mp3 -filter:v scale=-2:250 -an output.jpeg
So this basically scales the output image to whatever ratio or value you want.
Related
I have some trouble when converting from JPG to HEIC format (using HEVC Encoding) on Android
I'm using Nokia HEIF's library (https://github.com/nokiatech/heif/) for parsing data which I need to put some data includes:
Decoder config data (as a byte array) following the name called by Nokia, I think that data can be profile data, header data, ... that store the VPS/PPS, Exif, metadata ...
Image data following the name called by Nokia -> This may is a bitstream of HEVC Encoding.
So now, I got the image data bitstream (2) based FFMPEG tool by command as bellow:
ffmpeg -i oktest.png -crf 12 -preset medium -pix_fmt yuv420p -f hevc.bin
This is a referring link from Nokia that they guide how to do feed data and encode to HEIF format.
https://github.com/nokiatech/heif/issues/44
But, I didn't understand how I can get the decoder config data from the FFMPEG tool because I'm developing on Android platforms instead. I mean some definitions are determined different from the mind so I can't find any solutions for this. I already request a ticket to Nokia support, but no response yet.
val heif = HEIF()
val decoderConfig = context.assets.open(bitStreamDecoderConfig).readBytes()
val decoderConfig = decoderBuffer?.array() ?: ByteArray(0)
val imageData = FileInputStream(bitStreamData).readBytes()
val imageItem =
HEVCImageItem(heif, Size(imageWidth, imageHeight), decoderConfig, imageData)
heif.primaryImage = imageItem
heif.majorBrand = HEIF.BRAND_MIF1
heif.addCompatibleBrand(HEIF.BRAND_HEIC)
heif.save("$targetOutputFolder/$fileNameNoneExtension.${destinationType()}")
There is a page at http://jpgtoheif.com/ which suggest use ffmpeg itself instead the nokia tool.
there is another solution libheif with
a windows build is at https://github.com/pphh77/libheif-Windowsbinary/releases
I recorded a Video for limited time. Now i want to fetch all frames of video. I am using the below code and by using it i am able to get frames but i am not getting all video frames. 3 to 4 frames are repeated then i got a different frame. But as we all know we can get 25- 30 frames in a second to display smooth video. How to get all frames.
for (int i = 0; i < 30; i++) {
Bitmap bArray = mediaMetadataRetriever.getFrameAtTime(
1000000 * i,
MediaMetadataRetriever.OPTION_CLOSEST);
savebitmap(bArray, 33333 * i);
}
I don't want to use NDK. I got this link don't know what should be the value for "argb8888". I am getting error here. Can anyone explain how to do it.
Getting frames from Video Image in Android
I faced the same problem before and the Android's MediaMetadataRetriever seems not appropriated for this task since it doesn't have a good precision.
I used a library called "FFmpegMediaMetadataRetriever" in android studio:
Add this line in your build.graddle under module app:
compile 'com.github.wseemann:FFmpegMediaMetadataRetriever:1.0.14'
Rebuild your project.
Use the FFmpegMediaMetadataRetriever class to grab frames with higher
precision:
FFmpegMediaMetadataRetriever med = new FFmpegMediaMetadataRetriever();
med.setDataSource("your data source");
and in your loop you can grab frame using:
Bitmap bmp = med.getFrameAtTime(i*1000000, FFmpegMediaMetadataRetriever.OPTION_CLOSEST);
To get image frames from video we can use ffmpeg.For integrating FFmpeg in android we can use precompiled libraries like ffmpeg-android.
To extract image frames from a video we can use below command
String[] complexCommand = {"-y", "-i", inputFileAbsolutePath, "-an",
"-r", "1/2", "-ss", "" + startMs / 1000, "-t", "" + (endMs - startMs)
/ 1000, outputFileAbsolutePath};
Here,
-y
Overwrite output files
-i
ffmpeg reads from an arbitrary number of input “files” specified by the -i option
-an
Disable audio recording.
-r
Set frame rate
-ss
seeks to position
-t
limit the duration of data read from the input file
Here in place of inputFileAbsolutePath you have to specify the absolute path of video file from which you want to extract images.
For complete code check out this on my repository .Inside extractImagesVideo() method I am running command for extracting images from video.
For complete tutorial regarding integration of ffmpeg library and using ffmpeg commands to edit videos, check out this post which I have written on my blog.
You need to do :
Decode the video.
Present the decoded images at least as fast as 24 images / second. I
suppose you can skip this step.
Save the decoded images.
It appears that decoding the video would be the most challenging step. People and companies have spent years developing codecs (encoder / decoder) for various video formats.
Use this library JMF for FFMPEG.
I have integrated FFMPEG into my application and I want to convert videos to audio files,
But I want to do it using native implementation , (JNI) I don't want to use ffmpeg scripts ,
I have already tried this
You can't convert video to audio. You can however extract and only store the audio sub-streams of an AVFormatContext. Pseudocode:
// look for the first auid substream, and save its index:
for (size_t i = 0; i < AvFormatContextInstance->nb_streams; ++i)
if (AvFormatContextInstance->streams[i]->codec->codec_type == AVMEDIA_TYPE_AUDIO)
streamindex = i;
Now all you need to do is discard all other streams on other indexes and save AVPackets from the recognized audio stream.
I have this question: how can I load, in Android, a video file stored in my device, and how can I split it into frames?
I'm using IntelliJ and I want to split the video into frames in order to process them with some image processing techniques (with OpenCv for Android library).
You don't strictly need to use OpenCV for this. You can use the MediaMetaDataRetreiver class provided by the SDK. It provides methods to extract metadata from all kinds of media files. You can try something like:
MediaMetadataRetriever retriever = new MediaMetadataRetriever();
retriever.setDataSource(file.getAbsolutePath());
imgView.setImageBitmap(retriever.getFrameAtTime(TIME_OFFSET,MediaMetadataRetriever.OPTION_CLOSEST));
where TIME_OFFSET is in microseconds.
Grabbing a video frame in OpenCV is pretty easy. There are lots of examples on OpenCV site. However crucial thing is to set-up opencv on andriod. You can follow this link on getting started with Opencv on andriod.
http://opencv.org/android
Once you have opencv installed on andriod. You can easily load video file and grab frame in Mat structure and then do some processing on it.
Here is the sample one. It will need some modification to run it on andriod. I think you will need to used NDK on andriod for this.
int main(int argc, char*argv[])
{
char *my_file = "C:\\vid_an2\\desp_me.avi";
std::cout<<"Video File "<<my_file<<std::endl;
cv::VideoCapture input_video;
if(input_video.open(my_file))
{
std::cout<<"Video file open "<<std::endl;
}
else
{
std::cout<<"Not able to Video file open "<<std::endl;
}
namedWindow("My_Win",1);
namedWindow("Segemented", 1);
Mat cap_img;
for(;;)
{
input_video >> cap_img;
imshow("My_Win", cap_img);
waitKey(0);
}
return 0;
}
I am currently developing an application that needs to record audio, encode it as AAC, stream it, and do the same in reverse - receiving stream, decoding AAC and playing audio.
I successfully recorded AAC (wrapped in a MP4 container) using the MediaRecorder, and successfully up-streamed audio using the AudioRecord class. But, I need to be able to encode the audio as I stream it, but none of these classes seem to help me do that.
I researched a bit, and found that most people that have this problem end up using a native library like ffmpeg.
But I was wondering, since Android already includes StageFright, that has native code that can do encoding and decoding (for example, AAC encoding and AAC decoding), is there a way to use this native code on my application? How can I do that?
It would be great if I only needed to implement some JNI classes with their native code. Plus, since it is an Android library, it would be no licensing problems whatever (correct me if I'm wrong).
yes, you can use libstagefright, it's very powerful.
Since stagefright is not exposed to NDK, so you will have to do extra work.
There are two ways:
(1) build your project using android full source tree. This way takes a few days to setup, once ready, it's very easy, and you can take full advantage of stagefright.
(2) you can just copy include file to your project, it's inside this folder:
android-4.0.4_r1.1/frameworks/base/include/media/stagefright
then you will have export the library function by dynamically loading libstagefright.so, and you can link with your jni project.
To encode/decode using statgefright, it's very straightforward, a few hundred of lines can will do.
I used stagefright to capture screenshots to create a video, which will be available in our Android VNC server, to be released soon.
the following is a snippet, I think it's better than using ffmpeg to encode a movie. You can add audio source as well.
class ImageSource : public MediaSource {
ImageSource(int width, int height, int colorFormat)
: mWidth(width),
mHeight(height),
mColorFormat(colorFormat)
{
}
virtual status_t read(
MediaBuffer **buffer, const MediaSource::ReadOptions *options) {
// here you can fill the buffer with your pixels
}
...
};
int width = 720;
int height = 480;
sp<MediaSource> img_source = new ImageSource(width, height, colorFormat);
sp<MetaData> enc_meta = new MetaData;
// enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_H263);
// enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_MPEG4);
enc_meta->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_AVC);
enc_meta->setInt32(kKeyWidth, width);
enc_meta->setInt32(kKeyHeight, height);
enc_meta->setInt32(kKeySampleRate, kFramerate);
enc_meta->setInt32(kKeyBitRate, kVideoBitRate);
enc_meta->setInt32(kKeyStride, width);
enc_meta->setInt32(kKeySliceHeight, height);
enc_meta->setInt32(kKeyIFramesInterval, kIFramesIntervalSec);
enc_meta->setInt32(kKeyColorFormat, colorFormat);
sp<MediaSource> encoder =
OMXCodec::Create(
client.interface(), enc_meta, true, image_source);
sp<MPEG4Writer> writer = new MPEG4Writer("/sdcard/screenshot.mp4");
writer->addSource(encoder);
// you can add an audio source here if you want to encode audio as well
//
//sp<MediaSource> audioEncoder =
// OMXCodec::Create(client.interface(), encMetaAudio, true, audioSource);
//writer->addSource(audioEncoder);
writer->setMaxFileDuration(kDurationUs);
CHECK_EQ(OK, writer->start());
while (!writer->reachedEOS()) {
fprintf(stderr, ".");
usleep(100000);
}
err = writer->stop();