Using OpenCV 4.5.2 + FFMPEG on an android app
I'm trying to convert an .avi video file into a .mp4 file using x264, by running
ffmpeg -i input.avi -c:v libx264 output.mp4
The transcoding is processed correctly but, when I play the video, the colors are a bit... saturated?
This transcoding is part of the following flow:
Grab a .mov video file
Use OpenCV VideoCapture and VideoWriter to write text on the video frames (output is .avi)
Then I need to convert .avi file into .mp4 so it's reproducible on exoplayer.
In step 2. I'm looping all video frames and writing them to a new file, writing a text on them.
val videoWriter = VideoWriter(
outputFilePath,
VideoWriter.fourcc('M', 'J', 'P', 'G'),
15.0,
Size(1920.0, 1088.0),
true
)
val frame = Mat()
videoCapture.read(frame)
Imgproc.putText(
frame,
"This is a text",
Point(200.0, 200.0),
3,
5.0,
Scalar(
255.0,
124.0,
124.0,
255.0
),
1
)
videoWriter.write(frame)
I know that step 2. is probably not corrupting the frames because in my sample app, I'm displaying all frames in an ImageView, and they all match the original .mov video. So, my guess is that the issue is occurring on 3.
I'm using 'com.arthenica:mobile-ffmpeg-min-gpl:4.4' for android to execute the FFMPEG command as follows:
FFmpeg.executeAsync("-i $outputFilePath -c:v libx264 -y ${mp4File.path}")
where outputFilePath is the path for the .avi file and mp4File is an existing empty .mp4 file.
So I guess what I'm looking for is a way to have a lossless video color transcoding between .avi and .mp4 files.
Here's a screenshot of my sample app. The image on top is the last frame of the .avi video. The image on the bottom is the last frame played on a video player for the .mp4 transcoded video. This frame color difference is noticeable throughout the whole video.
EDIT: After some digging, I found out that the issue is that the VideoWritter is messing with the RGB colors. I still don't know the reason why this is happeninng.
Figured it out myself with some debug assitance from #llogan.
So, it looks like VideoCapture exports frames with BGR format, thus the Red and Blue colors being switched out. In order to fix my issue all I had to do was to convert the frame from BGR to RGB using the OpenCV utility method:
val frame = Mat()
val frame1 = Mat()
videoCapture.read(frame)
Imgproc.cvtColor(frame, frame1, Imgproc.COLOR_BGR2RGB)
videoWriter.write(frame1)
Used concat demuxer in FFMPEG code to concat videos :-
cmd = arrayOf("-y", "-f", "concat", "-safe", "0", "-i", textFile!!.path, "-c:v", "libx264", "-crf", "23", "-preset", "ultrafast", "-c:a", "copy", outputFile.path)
After concatenate finished, the video one played correctly has both audio & video but video two - video is missing has only audio. Output file has size & time duration including two videos.
The textFile contains :-
file '/path/to/file1'
file '/path/to/file2'
Used absolutePath for two mp4 video files.
Tryed this FFMPEG concat video finished but the video missing. But no solution to my problem.
Anything I am missing in the command.
I am adding image watermark to video with help of FFmpeg but FFmpeg takes an inordinate amount of time with the below command-
String[] cmd = {"-i",videoPath, "-i", waterMark.toString(),"-filter_complex","overlay=5:5","-codec:a", "copy", outputPath};
so i tried another command which was little bit faster but increase output file size(which i do not want)
String[] cmd = {"-y","-i", videoPath, "-i", waterMark.toString(), "-filter_complex", "overlay=5:5", "-c:v","libx264","-preset", "ultrafast", outputPath};
Some one please explain to me how to increase the speed of FFmpeg watermarking speed without increasing the size of output.
Thanks.
You mentioned that a 7MB video takes between 30-60 seconds.
There is always a trade off when choosing between speed and quality.
I tested on my phone using a 7MB file and it took 13 seconds, still slow, but we can't expect much better then that.
Ways to increase speed:
Lowering the frame rate, using the -r command
Changing the bitrate, using the -b:v and -b:a commands
Change the Constant Rate Factor, using -crf. The default value is 21
The range of the quantizer scale is 0-51: where 0 is lossless, 23 is default, and 51 is worst possible. A lower value is a higher quality and a subjectively sane range is 18-28. Consider 18 to be visually lossless or nearly so: it should look the same or nearly the same as the input but it isn't technically lossless.
This is what I have found works the best on most android devices:
String[] s = {"-i", VideoPath, "-i", ImagePath, "-filter_complex", "[0:v]pad=iw:if(lte(ih\\,iw)\\,ih\\,2*trunc(iw*16/9/2)):(ow-iw)/2:(oh-ih)/2[v0];[1:v][v0]scale2ref[v1][v0];[v0][v1]overlay=x=(W-w)/2:y=(H-h)/2[v]", "-map", "[v]", "-map", "0:a", "-c:v", "libx264", "-preset", "ultrafast", "-r", myFrameRate, directoryToStore[0] + "/" + SavedVideoName};
I reduced my framerate slightly, you can experiment what works best for you. I'm using mp4parser to retrieve the frame rate.
I have to give credit to #Gyan that provided me with a way to perfectly scale images that is being placed on top of a video, you can look at the question I asked here.
If you are unsure about the frame rate, you can remove it from the command and first test if your speed is reduced.
Try it, if you have any questions, please ask.
OP opted to go with the following command:
String[] cmd = {"-y","-i", videoPath, "-i", waterMark.toString(), "-filter_complex", "overlay=(main_w-overlay_w-10):5", "-map", "0:a","-c:v", "libx264", "-crf", "28","-preset", "ultrafast" ,outputPath};
Edit
Just to add on the command I mentioned and provide a detailed explanation of how to use it etc:
String[] cmd = {"-i", videoPath, "-i", waterMark.toString(), "-filter_complex", "[0:v]pad=iw:if(lte(ih\\,iw)\\,ih\\,2*trunc(iw*16/9/2)):(ow-iw)/2:(oh-ih)/2[v0];[1:v][v0]scale2ref[v1][v0];[v0][v1]overlay=x=(W-w)/2:y=(H-h)/2[v]", "-map", "[v]", "-map", "0:a", "-c:v", "libx264", "-preset", "ultrafast", "-r", myFrameRate, outputPath};
This is for device's that has a display aspect ratio of 16:9. If you want this filter to work on all device's you will have to get the aspect ratio of the device and change the filter 16/9/2 respectively.
You can get the device aspect ratio by creating this methods:
int gcd(int p, int q) {
if (q == 0) return p;
else return gcd(q, p % q);
}
void ratio(int a, int b) {
final int gcd = gcd(a,b);
if(a > b) {
setAspectRatio(a/gcd, b/gcd);
} else {
setAspectRatio(b/gcd, a/gcd);
}
}
void setAspectRatio(int a, int b) {
System.out.println("aspect ratio = "+a + " " + b);
//This is the string that will be used in the filter (instead of hardcoding 16/9/2
filterAspectRatio = a + "/" + b + "/" + "2";
}
Now you have the correct aspect ratio and you can change the filter accordingly.
Next, create a watermark and add it to a view, make that view the size of the device (match_parent) and scale/place the watermark where you would like it to be. You can then get the bitmap by calling:
Bitmap waterMarkBitmap = watermarkView.getDrawingCache();
and create a file from the Bitmap, like this:
String outputFilename = "myCoolWatermark.png"; //provide a name for you saved watermark
File path = Environment.getExternalStorageDirectory(); //this can be changed to where you want to store the bitmap
File waterMark = new File(path, outputFilename);
try (FileOutputStream out = new FileOutputStream(waterMark)) {
waterMarkBitmap.compress(Bitmap.CompressFormat.PNG, 100, out); // PNG is a lossless format, the compression factor (100) is ignored
} catch (IOException e) {
e.printStackTrace();
}
The watermark is created and can be reused, or you can delete it when you are done with it.
Now you can call the command mentioned above.
This is a very common question here. The simple answer is that you can't increase the encoding speed of ffmpeg on Android. You're encoding on a phone, so you don't expect desktop/server performance using an encoder and no hardware acceleration support.
There are a few things users can do:
Stream copy the audio with -c:a copy (you're already doing that).
Use -preset ultrafast to give up encoding efficiency for encoding speed (you're also already doing that).
Make the output width x height smaller with the scale filter (probably not an acceptable option for you).
Make sure your x264 was not compiled with --disable-asm so you can take advantage of the various ARM and NEON optimizations in x264 for a significant increase in encoding speed. However, I don't know which Android devices support that, but it's something to look into. For a quick check to see if you are using any optimizations refer to the console output from ffmpeg and search for using cpu capabilities. If none! then it is not using any optimizations, otherwise it may say ARMv7 NEON or something like that.
Offload the encoding to a server. Saves your users' battery life too.
All this for an annoying watermark? Avoid re-encoding and use a player to overlay the watermark.
Apparently FFmpeg has MediaCodec decoding support on Android, but encoding is the bottleneck here. However maybe it will save a few fps.
Send a patch to FFmpeg that enables MediaCodec encoding support or wait a few years for someone else to do so.
Forget ffmpeg and use MediaCodec directly. I am clueless about this and too lazy to look it up, but I assume it uses hardware to encode and I'll guess you can use it to make an overlay. Someone correct me if I am wrong.
I am using MediaMuxer and MediaExtractor to transcode video. Now i want to add text on video at certain position. How can i achieve this? Basically i am recording video and transcode it in lower resolution now i want to add text overlay without using ffmpeg. I tried to add watermark by using ffmpeg but it takes too much time. I am using this library.This is my command:
String[] complexCommand = {"-y" ,"-i", "/storage/emulated/0/Download/VID.mp4","-strict",
"experimental", "-vf", "movie="
+"/storage/emulated/0/Download/iqm_logo.png [watermark]; [in][watermark] overlay=main_w/2-overlay_w/2:450 [out]",
"-b:v", "1540k", "-c:v", "libx264","-threads","30","-preset","ultrafast","-r", "30","-s", "1080x1920","-c:a", "copy",
"/storage/emulated/0/Download/watermark.mp4"};
I recorded a Video for limited time. Now i want to fetch all frames of video. I am using the below code and by using it i am able to get frames but i am not getting all video frames. 3 to 4 frames are repeated then i got a different frame. But as we all know we can get 25- 30 frames in a second to display smooth video. How to get all frames.
for (int i = 0; i < 30; i++) {
Bitmap bArray = mediaMetadataRetriever.getFrameAtTime(
1000000 * i,
MediaMetadataRetriever.OPTION_CLOSEST);
savebitmap(bArray, 33333 * i);
}
I don't want to use NDK. I got this link don't know what should be the value for "argb8888". I am getting error here. Can anyone explain how to do it.
Getting frames from Video Image in Android
I faced the same problem before and the Android's MediaMetadataRetriever seems not appropriated for this task since it doesn't have a good precision.
I used a library called "FFmpegMediaMetadataRetriever" in android studio:
Add this line in your build.graddle under module app:
compile 'com.github.wseemann:FFmpegMediaMetadataRetriever:1.0.14'
Rebuild your project.
Use the FFmpegMediaMetadataRetriever class to grab frames with higher
precision:
FFmpegMediaMetadataRetriever med = new FFmpegMediaMetadataRetriever();
med.setDataSource("your data source");
and in your loop you can grab frame using:
Bitmap bmp = med.getFrameAtTime(i*1000000, FFmpegMediaMetadataRetriever.OPTION_CLOSEST);
To get image frames from video we can use ffmpeg.For integrating FFmpeg in android we can use precompiled libraries like ffmpeg-android.
To extract image frames from a video we can use below command
String[] complexCommand = {"-y", "-i", inputFileAbsolutePath, "-an",
"-r", "1/2", "-ss", "" + startMs / 1000, "-t", "" + (endMs - startMs)
/ 1000, outputFileAbsolutePath};
Here,
-y
Overwrite output files
-i
ffmpeg reads from an arbitrary number of input “files” specified by the -i option
-an
Disable audio recording.
-r
Set frame rate
-ss
seeks to position
-t
limit the duration of data read from the input file
Here in place of inputFileAbsolutePath you have to specify the absolute path of video file from which you want to extract images.
For complete code check out this on my repository .Inside extractImagesVideo() method I am running command for extracting images from video.
For complete tutorial regarding integration of ffmpeg library and using ffmpeg commands to edit videos, check out this post which I have written on my blog.
You need to do :
Decode the video.
Present the decoded images at least as fast as 24 images / second. I
suppose you can skip this step.
Save the decoded images.
It appears that decoding the video would be the most challenging step. People and companies have spent years developing codecs (encoder / decoder) for various video formats.
Use this library JMF for FFMPEG.