I'm making an app that takes a video and does some computation on the video. I need to carry out this computation on individual frames of the video. So, I have two questions:
Are videos in Android stored as a sequence of pictures? (I've seen a lot of Android devices that advertise having 25-30 fps cameras) If yes, can I, as a developer, get access to these frames that make up a video and how so?
If not, is there any way for me to generate at least 15-20 distinct frames per second from a video taken on an android device? (and of course, do the computation on those frames generated)
Videos are stored as videos. To manipulate frames one can use FFMPEG library. There are FFMPEG ports to Android such as in Dolphin opensource player. This would require C/C++ programming with NDK though.
Related
My android app plays videos in Exoplayer 2, and now I'd like to play a video backwards.
I searched around a lot and found only the idea to convert it to a gif and this from WeiChungChang.
Is there any more straight-forward solution? Another player or a library that implements this for me is probably too much to ask, but converting it to a reverse gif gave me a lot of memory problems and I don't know what to do with the WeiChungChang idea. Playing only mp4 in reverse would be enough tho.
Videos are frequently encoded such that the encoding for a given frame is dependent on one or more frames before it, and also sometimes dependent on one or more frames after it also.
In other words to create the frame correctly you may need to refer to one or more previous and one or more subsequent frames.
This allows a video encoder reduce file or transmission size by encoding fully the information for every reference frame, sometimes called I frames, but for the frames before and/or after the reference frames only storing the delta to the reference frames.
Playing a video backwards is not a common player function and the player would typically have to decode the video as usual (i.e. forwards) to get the frames and then play them in the reverse order.
You could extend ExoPlayer to do this yourself but it may be easier to manipulate the video on the server side if possible first - there exist tools which will reverse a video and then your players will be able to play it as normal, for example https://www.videoreverser.com, https://www.kapwing.com/tools/reverse-video etc
If you need to reverse it on the device for your use case, then you could use ffmpeg on the device to achieve this - see an example ffmpeg command to do this here:
https://video.stackexchange.com/a/17739
If you are using ffmpeg it is generally easiest to use via a wrapper on Android such as this one, which will also allow you test the command before you add it to your app:
https://github.com/WritingMinds/ffmpeg-android-java
Note that video manipulation is time and processor hungry so this may be slow and consume more battery than you want on your mobile device if the video is long.
I have troubles in my current project which requires video processing. Basically crop function (video should be squared), trimming (video shouldn't be longer than 30 seconds) and quality reduction (bitrate should be equal 713K).
I've succesfully embedded FFmpeg into application, all functions are working quite fine except one major detail - processing as per my boss is taking too long time. For video that have around 52 MB for 36 seconds it's taking 50 seconds to perforn all the operations (I'm trimming video to 30 seconds before any other operation obviously). The problem is that on parallel project on iOS video processing takes like 10-15 seconds for greater files. I assume that it's related to fact that they're using Apple QuickTime format which obviusly was developed by Apple so it's not surprising that it's working quite fast.
So well, it was introduction, now my question: is there any way for Android to process any video in any quality (for now we can assume that all videos are in h264) in time of 10-15 seconds (not more then in 30 seconds, as my boss said)? Some alternative to FFmpeg, that can perform operations faster? I'nm pretty sure that there is no possibility to perform such work in a such short time, since I already feel like I searched thought while Internet, but I want to make sure that there is really no possibility to do such work. If anyone can provide me links to solution more faster than FFmpeg or confirm that there is no such solution, I will be very gratefull.
Update
Thanks to Alex Cohn I've resolved this with MediaCodec. After a while, I got 20 seconds processing on 52MB video with cropping to square and lowering bitrate. For any future Googlers out of here I can suggest to take a look at this respository:
Many stuff about MediaCodec
and more precisely at this file: Extract, edit and encode again, video and audio
If the video has been recorded on the same device, you have a very good chance that MediaCodec and native Android media APIs will be much faster (running both decoder and encoder in HW). Otherwise, you can try to decode the video with MediaCodec, and fall back to FFmpeg software decoder if it fails. Even then, if you can use MediaCodec for compression, this alone may deliver performance that will satisfy your boss.
There exists a (deprecated) project called libstagefright that builds FFmpeg with support for the hardware codec, and it was designed to work on API 10+.
Don't forget to compare the CPU characteristics of your Android device and the iOS platform (if that's a 6S, they have a significant fore). Consider multithreaded encoding and decoding.
BTW: Note that FFmpeg does not come with H264 encoder, and the typical bundle of FFmpeg+x264 is GPL, and requires all your app to be open sourced (or pay a hefty license fee for x264, but still be forced to comply with LGPL of FFmpeg).
Note that you can make square video by manipulating the MP4 headers only, without transcoding!
Is there a way to record square (640x640) videos and concat them in Android? I looked up in the Internet and found some solutions. The solution seems to be "ffmpeg". However, to use ffmpeg I need to dive into NDK and build ffmpeg from its sources. Is there a solution by only using the Android SDK?
My basic needs are:
Record multiple videos (square format)
Resize captured videos (i.e. 480x480 to 640x640)
Concat captured videos
Rotate final video (clockwise 90)
Final output will be in mp4 or mpg format
Is there a solution by only using the Android SDK?
Not really.
Your primary video recording option is MediaRecorder, and it supports exactly nothing of what you list. For example, there is no requirement for any Android device to support taking square videos.
You are also welcome to use the camera preview stuff to assemble your own videos from individual frames. Vine does this, AFAIK. There, you could perhaps use existing Bitmap facilities to handle the cropping, resizing, and rotating. However, this will be slow, and doing this work in a way that can keep up with a reasonable frame rate will be difficult. Also, I do not know if there is a library that can stitch those frames together into a video, or blend in any sort of audio (camera previews are pure images).
We are working on Android 3D Animation App.
We need to identify images, then save and encode the same to video using FFmpeg (Since Android API is not supporting). Once the video is generated, then audio is appended to the same.
We are facing 2 problems on this.
First is the memory leakage issue at the time of saving identified images for encoding. CPU of emulator is getting overloaded. Whether FFpmeg is called every time when an image is selected? How to resolve this issue?
Second (in case if we get through the first one) we are not able to encode the selected images, since this is generating green color video. What could be reason for this?
Whether is there any tool other than FFmpeg for video encoding from images to H264?
Whether images version (Rastar or Vector) will impact this video encoding?
Whether Android OS version is considered?
Any valuable inputs on this will be greatly appreciated.
Thanks
I played also with that idea using ffmpeg on an android phone, but I would suggest to do that on a server which has much more power. On a server you don't need to think about the cpu load of a smartphone.
In general for improving your ffmpeg run you need to publish the ffmpeg calls. ffmpeg is quiet complex where the order of the parameters directly correlates with the efficience.
I don't know which container format you preferer but maybe a simple mjpeg codec could work for you. AFIK there a just the jpeg frames connected to each other which should be much simple then encoding a video to h264/x264 (ffmpeg uses the last one).
A combination of both may be to generate a mjpeg stream which will be converted on the server side to a h264 video which may be downloaded to the client. but that really depends on the length of the video if you don't want to waste the traffic of your customers.
I'm currently working on an app that lets the user choose an MP3 audio file. The file is then processed by my app.
For this processing, the application would need to decode audio files to get the raw PCM output.
To decode MP3, I have two options:
Use the Android system to decode MP3 and get the PCM data.
Decode the MP3 myself on the phone, WITHOUT paying MP3 licensing fees.
My question is whether #1 is technically possible? And for #2, whether the MP3 license on the phone covers an app as well?
To my knowledge, there is no Android-provided way to decode MP3s.
I've used JLayer in the past, and can recommend it for MP3 processing. Using the NDK with a c++ library might be faster, but if you're looking to keep it Java, that's what I'd use. It's still faster than real-time, roughly 30 seconds to decode all frames in an average bitrate 3 minute MP3. That's with an Galaxy S(1GHz), so any newer phones are faster.
As far as licensing goes, I can't help you there. JLayer itself is LGPL, but the world of MP3 licensing is murkier than used motor oil. After a few days of searching for a concrete answer, I just gave up and did it. The world at large seems divided on who even holds the license in the first place.
the Android system can decode mp3 file now, see here it describes the media codec, container, and network protocol support provided by the Android platform.
The MedieCodec is a very powful framework to encode and decode media file.
Option 1 is definitely not possible (unless you want to target ICS+ devices and are willing to write native C code to decode MP3s with OpenSL). Geobits recommendation of jLayer is a good one. For the most part, dealing with jLayer is a breeze. Here's a good blog post that will help: http://mindtherobot.com/blog/624/android-audio-play-an-mp3-file-on-an-audiotrack/