I have troubles in my current project which requires video processing. Basically crop function (video should be squared), trimming (video shouldn't be longer than 30 seconds) and quality reduction (bitrate should be equal 713K).
I've succesfully embedded FFmpeg into application, all functions are working quite fine except one major detail - processing as per my boss is taking too long time. For video that have around 52 MB for 36 seconds it's taking 50 seconds to perforn all the operations (I'm trimming video to 30 seconds before any other operation obviously). The problem is that on parallel project on iOS video processing takes like 10-15 seconds for greater files. I assume that it's related to fact that they're using Apple QuickTime format which obviusly was developed by Apple so it's not surprising that it's working quite fast.
So well, it was introduction, now my question: is there any way for Android to process any video in any quality (for now we can assume that all videos are in h264) in time of 10-15 seconds (not more then in 30 seconds, as my boss said)? Some alternative to FFmpeg, that can perform operations faster? I'nm pretty sure that there is no possibility to perform such work in a such short time, since I already feel like I searched thought while Internet, but I want to make sure that there is really no possibility to do such work. If anyone can provide me links to solution more faster than FFmpeg or confirm that there is no such solution, I will be very gratefull.
Update
Thanks to Alex Cohn I've resolved this with MediaCodec. After a while, I got 20 seconds processing on 52MB video with cropping to square and lowering bitrate. For any future Googlers out of here I can suggest to take a look at this respository:
Many stuff about MediaCodec
and more precisely at this file: Extract, edit and encode again, video and audio
If the video has been recorded on the same device, you have a very good chance that MediaCodec and native Android media APIs will be much faster (running both decoder and encoder in HW). Otherwise, you can try to decode the video with MediaCodec, and fall back to FFmpeg software decoder if it fails. Even then, if you can use MediaCodec for compression, this alone may deliver performance that will satisfy your boss.
There exists a (deprecated) project called libstagefright that builds FFmpeg with support for the hardware codec, and it was designed to work on API 10+.
Don't forget to compare the CPU characteristics of your Android device and the iOS platform (if that's a 6S, they have a significant fore). Consider multithreaded encoding and decoding.
BTW: Note that FFmpeg does not come with H264 encoder, and the typical bundle of FFmpeg+x264 is GPL, and requires all your app to be open sourced (or pay a hefty license fee for x264, but still be forced to comply with LGPL of FFmpeg).
Note that you can make square video by manipulating the MP4 headers only, without transcoding!
Related
My android app plays videos in Exoplayer 2, and now I'd like to play a video backwards.
I searched around a lot and found only the idea to convert it to a gif and this from WeiChungChang.
Is there any more straight-forward solution? Another player or a library that implements this for me is probably too much to ask, but converting it to a reverse gif gave me a lot of memory problems and I don't know what to do with the WeiChungChang idea. Playing only mp4 in reverse would be enough tho.
Videos are frequently encoded such that the encoding for a given frame is dependent on one or more frames before it, and also sometimes dependent on one or more frames after it also.
In other words to create the frame correctly you may need to refer to one or more previous and one or more subsequent frames.
This allows a video encoder reduce file or transmission size by encoding fully the information for every reference frame, sometimes called I frames, but for the frames before and/or after the reference frames only storing the delta to the reference frames.
Playing a video backwards is not a common player function and the player would typically have to decode the video as usual (i.e. forwards) to get the frames and then play them in the reverse order.
You could extend ExoPlayer to do this yourself but it may be easier to manipulate the video on the server side if possible first - there exist tools which will reverse a video and then your players will be able to play it as normal, for example https://www.videoreverser.com, https://www.kapwing.com/tools/reverse-video etc
If you need to reverse it on the device for your use case, then you could use ffmpeg on the device to achieve this - see an example ffmpeg command to do this here:
https://video.stackexchange.com/a/17739
If you are using ffmpeg it is generally easiest to use via a wrapper on Android such as this one, which will also allow you test the command before you add it to your app:
https://github.com/WritingMinds/ffmpeg-android-java
Note that video manipulation is time and processor hungry so this may be slow and consume more battery than you want on your mobile device if the video is long.
How streaming apps like Youtube, Hotstar or any other video player app, programmatically detects if network is getting slow over run-time and based on that they change video quality based on changes in network speed?
Many streaming services nowadays use HTTP-based streaming protocols. But there are exceptions; especially with low-latency streaming; e.g. WebRTC or Websocket-based solutions.
Assuming that you're using a HTTP-based protocol like HLS or MPEG-DASH, the "stream" is a long chain of video segments that are downloaded one after another. A video segment is a file in "TS" or "MP4" format (in some MP4 cases, video and audio are splitted into separate files); typically a segment has 2 or 6 or 10 seconds of audio and/or video.
Based on the playlist or manifest (or sometimes simply from decoding the segment), the player knows how many seconds of a single segment contains. It also knows how long it took to download that segment. You can measure the available bandwidth by diving the (average) size of a video segment file by the (average) time it took to download.
At the moment that it takes more time to download a segment than to play it, you know that the player will stall as soon as the buffer is empty; stalling is generally referred to as "buffering". Adaptive Bitrate (aka. ABR) is a technique that tries to prevent buffering; see https://en.wikipedia.org/wiki/Adaptive_bitrate_streaming (or Google for the expression) - when the player notices that the available bandwidth is lower than the bit rate of the video stream, it can switch to another version of the same stream that has a lower bit rate (typically achieved by higher compression and/or lower resolution - which results in less quality, but that's better than buffering)
PS #1: WebRTC and Websocket-based streaming solutions cannot use this measuring trick and must implement other solutions
PS #2: New/upcoming variants of HLS (eg. LL-HLS and LHLS) and MPEG-DASH use other HTTP technologies (like chunked-transfer or HTTP PUSH) to achieve lower latency - these typically do not work well with the mentioned measuring technique and use different techniques which I consider outside scope here.
You have to use a streaming server in order to do that. Wowza server is one of them (not free). The client and server will exchange information about the connexion and distribute chuncks of the video, depending on the network speed.
I'm trying to decode (and render) live H.264 over RTSP in an Android app.
Assuming, there are no network latency issues, the latency should not exceed several seconds.
The first try was to use the MediaPlayer which was fine but the internal buffering of the infrastructure causes delays of 10-15 seconds.
Right now the main dilemma is between using the new MediaCodec APIs or with FFMPeg.
I know there are many tutorials/samples out there talking about FFMPeg but I didn't see any comparison.
I think I understand most of the pros/cons for each but before spending ages on making one of them working I would like to be sure.
I haven't seen too much info on mediacodec api, I do know that ffmpeg gives you considerably better quality and latency than the built in rtsp functionality in android
I'm making an app that takes a video and does some computation on the video. I need to carry out this computation on individual frames of the video. So, I have two questions:
Are videos in Android stored as a sequence of pictures? (I've seen a lot of Android devices that advertise having 25-30 fps cameras) If yes, can I, as a developer, get access to these frames that make up a video and how so?
If not, is there any way for me to generate at least 15-20 distinct frames per second from a video taken on an android device? (and of course, do the computation on those frames generated)
Videos are stored as videos. To manipulate frames one can use FFMPEG library. There are FFMPEG ports to Android such as in Dolphin opensource player. This would require C/C++ programming with NDK though.
I'm currently working on an app that lets the user choose an MP3 audio file. The file is then processed by my app.
For this processing, the application would need to decode audio files to get the raw PCM output.
To decode MP3, I have two options:
Use the Android system to decode MP3 and get the PCM data.
Decode the MP3 myself on the phone, WITHOUT paying MP3 licensing fees.
My question is whether #1 is technically possible? And for #2, whether the MP3 license on the phone covers an app as well?
To my knowledge, there is no Android-provided way to decode MP3s.
I've used JLayer in the past, and can recommend it for MP3 processing. Using the NDK with a c++ library might be faster, but if you're looking to keep it Java, that's what I'd use. It's still faster than real-time, roughly 30 seconds to decode all frames in an average bitrate 3 minute MP3. That's with an Galaxy S(1GHz), so any newer phones are faster.
As far as licensing goes, I can't help you there. JLayer itself is LGPL, but the world of MP3 licensing is murkier than used motor oil. After a few days of searching for a concrete answer, I just gave up and did it. The world at large seems divided on who even holds the license in the first place.
the Android system can decode mp3 file now, see here it describes the media codec, container, and network protocol support provided by the Android platform.
The MedieCodec is a very powful framework to encode and decode media file.
Option 1 is definitely not possible (unless you want to target ICS+ devices and are willing to write native C code to decode MP3s with OpenSL). Geobits recommendation of jLayer is a good one. For the most part, dealing with jLayer is a breeze. Here's a good blog post that will help: http://mindtherobot.com/blog/624/android-audio-play-an-mp3-file-on-an-audiotrack/