Develop Reference

FFmpeg .avi to .mp4 video transcoding - Android

Using OpenCV 4.5.2 + FFMPEG on an android app
I'm trying to convert an .avi video file into a .mp4 file using x264, by running
ffmpeg -i input.avi -c:v libx264 output.mp4
The transcoding is processed correctly but, when I play the video, the colors are a bit... saturated?
This transcoding is part of the following flow:
Grab a .mov video file
Use OpenCV VideoCapture and VideoWriter to write text on the video frames (output is .avi)
Then I need to convert .avi file into .mp4 so it's reproducible on exoplayer.
In step 2. I'm looping all video frames and writing them to a new file, writing a text on them.
val videoWriter = VideoWriter(
outputFilePath,
VideoWriter.fourcc('M', 'J', 'P', 'G'),
15.0,
Size(1920.0, 1088.0),
true
)
val frame = Mat()
videoCapture.read(frame)
Imgproc.putText(
frame,
"This is a text",
Point(200.0, 200.0),
3,
5.0,
Scalar(
255.0,
124.0,
124.0,
255.0
),
1
)
videoWriter.write(frame)
I know that step 2. is probably not corrupting the frames because in my sample app, I'm displaying all frames in an ImageView, and they all match the original .mov video. So, my guess is that the issue is occurring on 3.
I'm using 'com.arthenica:mobile-ffmpeg-min-gpl:4.4' for android to execute the FFMPEG command as follows:
FFmpeg.executeAsync("-i $outputFilePath -c:v libx264 -y ${mp4File.path}")
where outputFilePath is the path for the .avi file and mp4File is an existing empty .mp4 file.
So I guess what I'm looking for is a way to have a lossless video color transcoding between .avi and .mp4 files.
Here's a screenshot of my sample app. The image on top is the last frame of the .avi video. The image on the bottom is the last frame played on a video player for the .mp4 transcoded video. This frame color difference is noticeable throughout the whole video.
EDIT: After some digging, I found out that the issue is that the VideoWritter is messing with the RGB colors. I still don't know the reason why this is happeninng.

Figured it out myself with some debug assitance from #llogan.
So, it looks like VideoCapture exports frames with BGR format, thus the Red and Blue colors being switched out. In order to fix my issue all I had to do was to convert the frame from BGR to RGB using the OpenCV utility method:
val frame = Mat()
val frame1 = Mat()
videoCapture.read(frame)
Imgproc.cvtColor(frame, frame1, Imgproc.COLOR_BGR2RGB)
videoWriter.write(frame1)

FFmpeg adding image watermark to video process is very slow

I am adding image watermark to video with help of FFmpeg but FFmpeg takes an inordinate amount of time with the below command-
String[] cmd = {"-i",videoPath, "-i", waterMark.toString(),"-filter_complex","overlay=5:5","-codec:a", "copy", outputPath};
so i tried another command which was little bit faster but increase output file size(which i do not want)
String[] cmd = {"-y","-i", videoPath, "-i", waterMark.toString(), "-filter_complex", "overlay=5:5", "-c:v","libx264","-preset", "ultrafast", outputPath};
Some one please explain to me how to increase the speed of FFmpeg watermarking speed without increasing the size of output.
Thanks.

You mentioned that a 7MB video takes between 30-60 seconds.
There is always a trade off when choosing between speed and quality.
I tested on my phone using a 7MB file and it took 13 seconds, still slow, but we can't expect much better then that.
Ways to increase speed:
Lowering the frame rate, using the -r command
Changing the bitrate, using the -b:v and -b:a commands
Change the Constant Rate Factor, using -crf. The default value is 21
The range of the quantizer scale is 0-51: where 0 is lossless, 23 is default, and 51 is worst possible. A lower value is a higher quality and a subjectively sane range is 18-28. Consider 18 to be visually lossless or nearly so: it should look the same or nearly the same as the input but it isn't technically lossless.
This is what I have found works the best on most android devices:
String[] s = {"-i", VideoPath, "-i", ImagePath, "-filter_complex", "[0:v]pad=iw:if(lte(ih\\,iw)\\,ih\\,2*trunc(iw*16/9/2)):(ow-iw)/2:(oh-ih)/2[v0];[1:v][v0]scale2ref[v1][v0];[v0][v1]overlay=x=(W-w)/2:y=(H-h)/2[v]", "-map", "[v]", "-map", "0:a", "-c:v", "libx264", "-preset", "ultrafast", "-r", myFrameRate, directoryToStore[0] + "/" + SavedVideoName};
I reduced my framerate slightly, you can experiment what works best for you. I'm using mp4parser to retrieve the frame rate.
I have to give credit to #Gyan that provided me with a way to perfectly scale images that is being placed on top of a video, you can look at the question I asked here.
If you are unsure about the frame rate, you can remove it from the command and first test if your speed is reduced.
Try it, if you have any questions, please ask.
OP opted to go with the following command:
String[] cmd = {"-y","-i", videoPath, "-i", waterMark.toString(), "-filter_complex", "overlay=(main_w-overlay_w-10):5", "-map", "0:a","-c:v", "libx264", "-crf", "28","-preset", "ultrafast" ,outputPath};
Edit
Just to add on the command I mentioned and provide a detailed explanation of how to use it etc:
String[] cmd = {"-i", videoPath, "-i", waterMark.toString(), "-filter_complex", "[0:v]pad=iw:if(lte(ih\\,iw)\\,ih\\,2*trunc(iw*16/9/2)):(ow-iw)/2:(oh-ih)/2[v0];[1:v][v0]scale2ref[v1][v0];[v0][v1]overlay=x=(W-w)/2:y=(H-h)/2[v]", "-map", "[v]", "-map", "0:a", "-c:v", "libx264", "-preset", "ultrafast", "-r", myFrameRate, outputPath};
This is for device's that has a display aspect ratio of 16:9. If you want this filter to work on all device's you will have to get the aspect ratio of the device and change the filter 16/9/2 respectively.
You can get the device aspect ratio by creating this methods:
int gcd(int p, int q) {
if (q == 0) return p;
else return gcd(q, p % q);
}
void ratio(int a, int b) {
final int gcd = gcd(a,b);
if(a > b) {
setAspectRatio(a/gcd, b/gcd);
} else {
setAspectRatio(b/gcd, a/gcd);
}
}
void setAspectRatio(int a, int b) {
System.out.println("aspect ratio = "+a + " " + b);
//This is the string that will be used in the filter (instead of hardcoding 16/9/2
filterAspectRatio = a + "/" + b + "/" + "2";
}
Now you have the correct aspect ratio and you can change the filter accordingly.
Next, create a watermark and add it to a view, make that view the size of the device (match_parent) and scale/place the watermark where you would like it to be. You can then get the bitmap by calling:
Bitmap waterMarkBitmap = watermarkView.getDrawingCache();
and create a file from the Bitmap, like this:
String outputFilename = "myCoolWatermark.png"; //provide a name for you saved watermark
File path = Environment.getExternalStorageDirectory(); //this can be changed to where you want to store the bitmap
File waterMark = new File(path, outputFilename);
try (FileOutputStream out = new FileOutputStream(waterMark)) {
waterMarkBitmap.compress(Bitmap.CompressFormat.PNG, 100, out); // PNG is a lossless format, the compression factor (100) is ignored
} catch (IOException e) {
e.printStackTrace();
}
The watermark is created and can be reused, or you can delete it when you are done with it.
Now you can call the command mentioned above.

This is a very common question here. The simple answer is that you can't increase the encoding speed of ffmpeg on Android. You're encoding on a phone, so you don't expect desktop/server performance using an encoder and no hardware acceleration support.
There are a few things users can do:
Stream copy the audio with -c:a copy (you're already doing that).
Use -preset ultrafast to give up encoding efficiency for encoding speed (you're also already doing that).
Make the output width x height smaller with the scale filter (probably not an acceptable option for you).
Make sure your x264 was not compiled with --disable-asm so you can take advantage of the various ARM and NEON optimizations in x264 for a significant increase in encoding speed. However, I don't know which Android devices support that, but it's something to look into. For a quick check to see if you are using any optimizations refer to the console output from ffmpeg and search for using cpu capabilities. If none! then it is not using any optimizations, otherwise it may say ARMv7 NEON or something like that.
Offload the encoding to a server. Saves your users' battery life too.
All this for an annoying watermark? Avoid re-encoding and use a player to overlay the watermark.
Apparently FFmpeg has MediaCodec decoding support on Android, but encoding is the bottleneck here. However maybe it will save a few fps.
Send a patch to FFmpeg that enables MediaCodec encoding support or wait a few years for someone else to do so.
Forget ffmpeg and use MediaCodec directly. I am clueless about this and too lazy to look it up, but I assume it uses hardware to encode and I'll guess you can use it to make an overlay. Someone correct me if I am wrong.

Reverse video in android

I have recorded a video from camera in my app and saved in device storage.Now I want to reverse the video such that it plays from backwards.i.e. if video is of 10 seconds then the last frame at 10th second will become first frame and it starts playing from there to 1st second first frame.I want to save the reversed video in a file.How should i proceed in that?

If you are prepared to use ffmpeg you can use this approach - it essentially breaks the video into frames and then builds it again in reverse order:
https://stackoverflow.com/a/8137637/334402
There are several ways to use ffmpeg in Android but the 'wrapper' approach is one which I have found a reasonable blend of performance and ease of use. Some example Android ffmpeg wrapper:
http://hiteshsondhi88.github.io/ffmpeg-android-java/
https://github.com/guardianproject/android-ffmpeg
It's worth being aware that this will be time-consuming on a Mobile - if you have the luxury of being able to upload to a server and doing the reversal there it might be quicker.

Thanks to Mick for giving me an idea to use ffmpeg for reversing video.
I have posted code at github for reversing video along with performing other video editing operation using ffmpeg and complete tutorial in my blog post here.
As written in my blog post,
For reversing video,first we need to divide video into segments with
duration of 10 seconds or less because reverse video command for
ffmpeg will not work for long duration videos unless your device has
32 GB of RAM.
Hence,to reverse a video-
1.Divide video into segments with duration of 10 seconds or less.
2.Reverse the segmented videos
3.Concatenate reversed segmented videos in reverse order.
For dividing video into segments with duration of 6 seconds we can use
the below command-
String[] complexCommand = {"-i", inputFileAbsolutePath, "-c:v",
"libx264", "-crf", "22", "-map", "0", "-segment_time", "6", "-g", "9",
"-sc_threshold", "0", "-force_key_frames", "expr:gte(t,n_forced*6)",
"-f", "segment", outputFileAbsolutePath};
Here,
-c:v libx264
encodes all video streams with libx264
-crf
Set the quality for constant quality mode.
-segment_time
time for each segment of video
-g
GOP size
-sc_threshold
set scene change threshold.
-force_key_frames expr:gte(t,n_forced*n)
Forcing a keyframe every n seconds
After segmenting video,we need to reverse the segmented videos.For
that we need to run a loop where each segmented video file will be
reversed.
To reverse a video with audio(without removing its audio) we can use
the below command-
String command[] = {"-i", inputFileAbsolutePath, "-vf", "reverse",
"-af", "areverse", outputFileAbsolutePath};
To reverse a video with audio removing its audio we can use the below
command-
String command[] = {"-i", inputFileAbsolutePath, "-an", "-vf",
"reverse", outputFileAbsolutePath};
To reverse a video without audio we can use the below command-
String command[] = {"-i",inputFileAbsolutePath, "-vf", "reverse",
outputFileAbsolutePath};
After reversing segmented videos,we need to concatenate reversed
segmented videos in reverse order.For that we sort videos on the basis
of last modified file using Arrays.sort(files,
LastModifiedFileComparator.LASTMODIFIED_REVERSE).
Then, to concatenate reversed segmented videos(with audio) we can use the below
command-
String command[] =
{"-i",inputFile1AbsolutePath,"-i",inputFile2AbsolutePath
.....,"-i",inputFileNAbsolutePath,"-filter_complex","[0:v0] [0:a0]
[1:v1] [1:a1]...[N:vN] concat=n=N:v=1:a=1 [v]
[a],"-map","[v]","-map","[a]", outputFileAbsolutePath};
To concatenate reversed segmented videos(without audio) we can use the below
command-
String command[] =
{"-i",inputFile1AbsolutePath,"-i",inputFile2AbsolutePath
.....,"-i",inputFileNAbsolutePath,"-filter_complex","[0:0] [1:0]
[2:0]...[N:0] concat=n=N:v=1:a=0",outputFileAbsolutePath};
Here,
-filter_complex [0:v0] [0:a0] [1:v1] [1:a1]…[N:vN] tells ffmpeg what streams to send to the concat filter.In the above case, video stream 0
[0:v0] and audio stream 0 [0:a0] from input 0,video stream 1 [1:v1]
and audio stream 1 [1:v1] from input 1 and so on.
concat filter is used to concatenate audio and video streams, joining
them together one after the other.The filter accepts the following
options:
n
Set the number of segments. Default is 2.
v
Set the number of output video streams, that is also the number of
video streams in each segment. Default is 1.
a
Set the number of output audio streams, that is also the number of
audio streams in each segment. Default is 0.

Android: Jcodec: encoding image files to .mp4 video

I am using JCODEC to create a video of my screen activities. I don’t want to use android NDK as I want to do it in JAVA. I am running a for loop to encode images using SequenceEncoder. The problem is the loop is taking too long to run and log cat gives messages of GC_FOR_ALLOC freed. For even 5 iterations the loop takes many seconds. So I am unable to take proper video of my activities. I tried to make changes in the code but its not helping. Please help me with this. Suggest other options if available. Thanks in advance.
File file = new File(Environment.getExternalStorageDirectory()+"/a.mp4");
SequenceEncoder encoder = new SequenceEncoder(file);
mview.setDrawingCacheEnabled(true);
// only 5 frames in total
for (int i = 1; i <= 5; i++) {
// getting bitmap from drawable path
mview.postInvalidate();
encoder.encodeNativeFrame(this.fromBitmap(mview.getDrawingCache()));
}
encoder.finish();

Retrieve album art using FFmpeg

I'm developing an Android application that relies on FFmpeg to retrieve audio metadata. I know it's possible to retrieve album art programmatically using FFMpeg. However, once you have decoded the art (a video frame within an MP3) how do generate an image file (a PNG) for use within an application? I've search all over but can't seem to find a working example.
Edit, here is the solution:
#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
void retrieve_album_art(const char *path, const char *album_art_file) {
int i, ret = 0;
if (!path) {
printf("Path is NULL\n");
return;
}
AVFormatContext *pFormatCtx = avformat_alloc_context();
printf("Opening %s\n", path);
// open the specified path
if (avformat_open_input(&pFormatCtx, path, NULL, NULL) != 0) {
printf("avformat_open_input() failed");
goto fail;
}
// read the format headers
if (pFormatCtx->iformat->read_header(pFormatCtx) < 0) {
printf("could not read the format header\n");
goto fail;
}
// find the first attached picture, if available
for (i = 0; i < pFormatCtx->nb_streams; i++)
if (pFormatCtx->streams[i]->disposition & AV_DISPOSITION_ATTACHED_PIC) {
AVPacket pkt = pFormatCtx->streams[i]->attached_pic;
FILE* album_art = fopen(album_art_file, "wb");
ret = fwrite(pkt.data, pkt.size, 1, album_art);
fclose(album_art);
av_free_packet(&pkt);
break;
}
if (ret) {
printf("Wrote album art to %s\n", album_art_file);
}
fail:
av_free(pFormatCtx);
// this line crashes for some reason...
//avformat_free_context(pFormatCtx);
}
int main() {
avformat_network_init();
av_register_all();
const char *path = "some url";
const char *album_art_file = "some path";
retrieve_album_art(path, album_art_file);
return 0;
}

To use ffmpeg programmatically, I think you would have to call read_apic() in libavformat (which is part of ffmpeg).
From the commandline, you can apparently do this:
ffmpeg -i input.mp3 -an -vcodec copy cover.jpg
The commandline behaviour implies that the cover art image is seen as just another video stream (containing just one frame), so using libavformat in the usual way you would to demux the video part of a stream should produce that image.
Sample code for demuxing: ffmpeg/docs/examples/demuxing.c The first (and only) AVPacket that would be obtained from demuxing the video stream in an mp3 would contain the JPEG file (still encoded as JPEG, not decoded).
AVFormatContext* fmt_ctx;
// set up fmt_ctx to read first video stream
AVPacket pkt;
av_read_frame(fmt_ctx, &pkt);
FILE* image_file = fopen("image.jpg", "wb");
int result = fwrite(pkt.data, pkt.size, 1, image_file);
fclose(image_file);
If there are multiple images, I think they would be seen as separate video streams, rather than as separate packets in the same stream. The first stream would be the one with the largest resolution.
All this is probably implemented internally in terms of read_apic().
The ID3v2 spec allows for any image format, but recommends JPEG or PNG. In practice all images in ID3 are JPEG.
EDIT: Moved some of the less useful bits to postscript:
P.S. ffmpeg -i input.mp3 -f ffmetadata metadata.txt will produce an ini-like file containing the metadata, but the image is not even referred to in there, so that is not a useful approach.
P.S. There may be multiple images in an ID3v2 tag. You may have to handle the case when there is more than one image or more than one type of image present.
P.S. ffmpeg is probably not the best software for this. Use id3lib, TagLib, or one of the other implementations of ID3. These can be used either as libraries (callable from the language of your choice) or as commandline utilities. There is sample C++ code for TagLib here: How do I use TagLib to read/write coverart in different audio formats? and for id3lib here: How to get album art from audio files using id3lib.

As an addition the answer above, I needed a way to resize the output image as well so I found the below command while experimenting with the command in the current answer:
ffmpeg -i input.mp3 -filter:v scale=-2:250 -an output.jpeg
So this basically scales the output image to whatever ratio or value you want.