I am trying to implement my own media player. What is the best way to render video and audio? At this point I am thinking to use SurfaceView and AudioTrack classes, but not sure if it is the best option. I am interested in SDK and NDK solutions.
File output on regular desktop is non-blocking, that is OS takes care of buffering and actual disk writes are asynchronous to the thread that initiates the output. Does the same principle apply to video and audio output? If not, I would need to run a separate thread to handle output asynchronously from decoding/demuxing.
What free software decoders are available for android? I am thinking to use ffmpeg. Can relatively recent (say, top 30% in terms of CPU power) tablet handle 1,280×720 and 1,920×1,080 formats in software mode?
Rock Player is an open source player for android (it's official site). You can get the source from it's source code download page. They use the ffmpeg which is a LGPL library. Pock Player developers do extra efforts to write some asm making decoding faster.
Related
Assuming we have a Surface in Android that displays a video (e.g. h264) with a MediaPlayer:
1) Is it possible to change the displayed saturation, contrast & brightness of the displayed on the surface video? In real time? E.g. Images can use setColorFilter is there anything similar in Android to process the video frames?
Alternative question (if no. 1 is too difficult):
2) If we would like to export this video with e.g. an increased saturation, we should use a Codec, e.g. MediaCodec. What technology (method, class, library, etc...) should we use before the codec/save action to apply the saturation change?
For display only, one easy approach is to use a GLSurfaceView, a SurfaceTexture to render the video frames, and a MediaPlayer. Prokash's answer links to an open source library that shows how to accomplish that. There are a number of other examples around if you search those terms together. Taking that route, you draw video frames to an OpenGL texture and create OpenGL shaders to manipulate how the texture is rendered. (I would suggest asking Prokash for further details and accepting his answer if this is enough to fill your requirements.)
Similarly, you could use the OpenGL tools with MediaCodec and MediaExtractor to decode video frames. The MediaCodec would be configured to output to a SurfaceTexture, so you would not need to do much more than code some boilerplate to get the output buffers rendered. The filtering process would be the same as with a MediaPlayer. There are a number of examples using MediaCodec as a decoder available, e.g. here and here. It should be fairly straightforward to substitute the TextureView or SurfaceView used in those examples with the GLSurfaceView of Prokash's example.
The advantage of this approach is that you have access to all the separate tracks in the media file. Because of that, you should be able to filter the video track with OpenGL and straight copy other tracks for export. You would use a MediaCodec in encode mode with the Surface from the GLSurfaceView as input and a MediaMuxer to put it all back together. You can see several relevant examples at BigFlake.
You can use a MediaCodec without a Surface to access decoded byte data directly and manipulate it that way. This example illustrates that approach. You can manipulate the data and send it to an encoder for export or render it as you see fit. There is some extra complexity in dealing with the raw byte data. Note that I like this example because it illustrates dealing with the audio and video tracks separately.
You can also use FFMpeg, either in native code or via one of the Java wrappers out there. This option is more geared towards export than immediate playback. See here or here for some libraries that attempt to make FFMpeg available to Java. They are basically wrappers around the command line interface. You would need to do some extra work to manage playback via FFMpeg, but it is definitely doable.
If you have questions, feel free to ask, and I will try to expound upon whatever option makes the most sense for your use case.
If you are using a player that support video filters then you can do that.
Example of such a player is VLC, which is built around FFMPEG [1].
VLC is pretty easy to compile for Android. Then all you need is the libvlc (aar file) and you can build your own app. See compile instructions here.
You will also need to write your own module. Just duplicate an existing one and modify it. Needless to say that VLC offers strong transcoding and streaming capabilities.
As powerful VLC for Android is, it has one huge drawback - video filters cannot work with hardware decoding (Android only). This means that the entire video processing is on the CPU.
Your other options are to use GLSL / OpenGL over surfaces like GLSurfaceView and TextureView. This guaranty GPU power.
I want to create an Android app that plays multiple mp3s simultaneously, with precise sync (less than 1/10 of a second off) and independent volume control. Size of each mp3 could be over 1MB, run time up to several minutes. My understanding is that MediaPlayer will not do the precise sync, and SoundPool can't handle files over 1MB or 5 seconds run time. I am experimenting with superpowered and may end up using that, but I'm wondering if there's anything simpler, given that I don't need any processing (reverb, flange, etc.), which is superpowered's focus.
Also ran across the YouTube video on Android high-performance audio, from Google I/O 2016. Wondering if anyone has any experience with this.
https://www.youtube.com/watch?v=F2ZDp-eNrh4
Superpowered was originally made for my DJ app (DJ Player in the App Store), where precisely syncing multiple tracks is a requirement.
Therefore, syncing multiple mp3s and independent volume control is definitely possible and core to Superpowered. All you need is the SuperpoweredAdvancedAudioPlayer class for this.
The CrossExample project in the SDK has two players playing in sync.
The built-in audio features in Android are highly device and/or build dependent. You can't get a consistent feature set with those. In general, the audio features of Android are not stable. That's why you need a specialized audio library which does everything "inside" your application (so is not a "wrapper" around Android's audio features).
When you are playing compressed files (AAC, MP3, etc) on Android in most situations they are decoded in hardware to save power, except when the output goes to a USB audio interface. The hardware codec accepts data in big chunks (again, to save power). Since it's not possible to issue a command to start playing multiple streams at once, what will often be happening is that one stream will already send a chunk of compressed audio to hardware codec, and it will start playing, while others haven't yet sent their data.
You really need to decode these files in your app and mix the output to produce a single audio stream. Then you will guarantee the desired synchronization. The built-in mixing facilities are mostly intended to allow multiple apps to use the same sound output, they are not designed for multitrack mixing.
I'm trying to make a H.264 sniffer for my android distribution.
Basically, what I want is to just dump any H.264 stream that is passed from a lambda android application by intercepting it on its way to the hardware video decoder.
I have an odroid-c1 board (AmLogic S805 SoC) and my android build setup is ready. Now, what I need to know is where is the code called when a new H.264 frame is ready to be sent to decoding. Surely there must be a common place ?
When searching the build tree for files referencing H.264 or OMX, I have various results including libstagefright/omx/, ffmpeg/libavcodec/, LibPlayer/amffmpeg/ (Amlogic's own fork of FFmpeg) [..].
If you have any idea or name of functions that are part of the video decoding path, I'll take them :). Thanks !
As far as I know the Amlogic CPU contains two DSPs for Audio and Video decoding. Data is delivered to them through the amports driver in kernel space.
The userspace part of this driver is the libamcodec which provides a thin layer to this driver.
I do not know what kind of layers in Android are invloded but most probably they use Amlogic ffmpeg which is using libamplayer as middle layer which calls in the end libamcodec....
I want to stream video recording from my android phone to network media server.
The first problem is that when setting MediaRecorder output to socket, the stream is missing some mdat size headers. This can be fixed by preprocessing that stream locally and adding missing data to stream in order to produce valid output stream.
The question is how to proceed from there.
How can I go about output that stream as an RTMP stream?
First, let's unwind your question. As you've surmised, RTMP isn't currently supported by Android. You can use a few side libraries to add support, but these may not be full implementations or have other undesirable side effects and bugs that cause them to fail to meet your needs.
The common alternative in this case is to use RTSP. It provides a comparable session format that has its own RFC, and its packet structure when combined with RTP is very similar (sans some details) to your desired protocol. You could perform the necessary fixups here to transmute RTP/RTSP into RTMP, but as mentioned, such effort is currently outside the development scope of your application.
So, let's assume you would like to use RTMP (invalidating this thread) and that the above-linked library does not meet your needs.
You could, for example, follow this tutorial for recording and playback using Livu, Wowza, and Adobe Flash Player, talking with the Livu developer(s) about licensing their client. Or, you could use this client library and its full Android recorder example to build your client.
To summarize:
RTSP
This thread, using Darwin Media Server, Windows Media Services, or VLC
RTMP
This library,
This thread and this tutorial, using Livu, Wowza, and Adobe Flash Player
This client library and this example recorder
Best of luck with your application. I admit that I have a less than comprehensive understanding of all of these libraries, but these appear to be the standard solutions in this space at the time of this writing.
Edit:
According to the OP, walking the RTMP library set:
This library: He couldn't make the library demos work. More importantly, RTMP functionality is incomplete.
This thread and this tutorial, using Livu, Wowza, and Adobe Flash Player: This has a long tutorial on how to consume video, but its tutorial on publication is potentially terse and insufficient.
This client library and this example recorder: The given example only covers audio publication. More work is needed to make this complete.
In short: more work is needed. Other answers, and improvements upon these examples, are what's needed here.
If you are using a web-browser on Android device, you can use WebRTC for video capturing and server-side recording, i.e with Web Call Server 4
Thus the full path would be:
Android Chrome [WebRTC] > WCS4 > recording
So you don't need RTMP protocol here.
If you are using a standalone RTMP app, you can use any RTMP server for video recording. As i know Wowza supports H.264+Speex recording.
I've been experimenting with making an android app for the past week or two and have reached a point where I need to decide on a general plan of action concerning the code I'm writing.
started with SoundPool, easy to use, not very flexible. went on to AudioTrack, seems good but slow.
So now I'm looking at the ndk..
Does the ndk have direct access to AudioTrack? or something else?
What is the general concensus on this kind of thing?
A guess is to make the UI in java and the 'sound engine' in C++
I'd like to get started on the right track before I write too much.
edit:
It is a music studio that plays and manipulates wav files from the sdcard as well as realtime sound synthesis.
The Android SDK doesn’t offer a fully featured audio processing solution like Core Audio on iOS. The available features are exposed through OpenSL ES interface. Any audio processing feature's availability is dependent on the device manufacturer’s Android flavor and configuration, so you can not rely on them.
To wit, the infamous Android fragmentation problem is even bigger in audio.
Even worse, if a device reports an audio feature available, it may still not work properly. For example: audio playback rate on a Samsung Galaxy Note III.
Realtime sound synthesis? None of the Android SDK’s features are designed for that.
The best way is doing the UI in Java and dealing with sound in C++.
There are some 'audio engine' offers on the web, but many of them are just “wrappers” around the same thing.
As cofounder of Superpowered, allow me to recommend the use the Superpowered Audio SDK, which is a total audio processing solution designed for real-time and highest performance, without any dependency on Android’s audio offerings, and runs on all.
http://superpowered.com/
There are a ton of Audio Latency issues in Android. There's really not anything that can be done about it. It seems like ICS (4.0) may have done some improvements on it, from what I've read.
You could subscribe to Andraudio and you'd actually be better off directing Android Audio questions through their emailing list than through Stackoverflow:
http://music.columbia.edu/mailman/listinfo/andraudio