I'm currently implementing a sound effect mixing on Android via OpenSL. I have an initial implementation going, but I've encountered some issues.
My implementation is as follows:
1) For each sound effect I create several AudioPlayer objects (one for each simultaneous sound) that uses an SLDataLocator_AndroidFD data source that in turn refers to an OGG file. For example, if I have a gun firing sound (lets call it gun.ogg) that is played in rapid succession, I use around 5 AudioPlayer objects that refer to the same gun.ogg audio source and also the same outputmix object.
2) When I need to play that sound effect, I search through all the AudioPlayer objects I created and find one that isn't currently in the SL_PLAYSTATE_PLAYING state and use it to play the effect.
3) Before playing a clip, I seek to the start of it using SLPlayItf::SetPosition.
This is working out alright so far, but there is some crackling noise that occurs when playing sounds in rapid succession. I read on the Android NDK newsgroup that OpenSL on Android has problems with switching data sources. Has anyone come across this issue?
I'm also wondering if anyone else seen or come up with a sound mixing approach for OpenSL on Android. If so, does your approach differ from mine? Any advice on the crackling noise?
I've scoured the internet for OpenSL documentation and example code, but there isn't much out there with regards to mixing (only loading which I've figured out already). Any help would be greatly appreciated.
This is probably not the best approach (creating many instances of audio players). Unfortunately the Android version (2.3) of OpenSL ES doesn't support SLDynamicSourceItf. Which would be similar to OpenAL's source binding interface. One approach would be to create multiple stream players. You would then search for a stream player that isn't currently playing and start streaming your sound effect to it from memory. It's not ideal but it's doable.
You should probably not use the ogg format for sound effects either. You're better off with WAV (PCM) as it won't need to be decoded.
Ogg is fine for streaming background music.
Related
Let me refraise my question, I wrote it in a hurry.
Current situation:
I have set up a digital video recorder to record broadcasts provided via DVB-C. It is running on a raspberry 3B using TVHeadend and jetty/cling to provide UPnP and other possibilities to access media files. For watching recordings, I wrote an android player app using IJKPlayer, which runs on smartphones, FireTV and AndroidTV.
One hassle when playing media files which are currently beeing recorded is, that IJKPlayer doesn not support timeshifting. Means, when I start playing a currently recording file, I can only watch the length which is known by the player at that moment. Anything which is recorded afterwards can not be played. I need to exit the player activity and start it again. I have resolved that issue by "simulating" a completed recoding using a custom servlet implementation. Since the complete length of the recording is already known, I can use ffmpeg to accomplish this.
Future situation:
I plan to move away from IJKPlayer to ExoPlayer, because it supports hardware playback and is much faster when playing h.264 media. I can of course use the same solution like above, but as far as I have found out yet, ExoPlayer can support media files which are currently being recorded by using the Timeline class. However, I don't seem to find neither a usefull documentation nor any good example. Hence, I would appreciate any help with the timeline object.
Regards
Harry
Looks like my approach won't work. At least, I didn't find a solution. Problem is, that the server returns the stream size as it is during player-start-time. I didn't find a method to update the media duration for "regular" files.
However, I can solve the problem by changing the server side. Instead of accessing a regular file, I convert the file to m3u8 in realtime, using ffmpeg. I then throw the m3u8 URI onto the player and it updates the duration of the stream (while playing) without the need to create any additional code on the client side.
How can I play background audio, in Android, without interrupting the MediaPlayer playback, by either using MediaPlayer (preferred) or OpenSL ES?
I know SoundPool is able to play sound effects without interrupting any MediaPlayer playback, but the size is limited to 1M per effect, which is way to less. Not requesting audio focus, via AudioManager doesn't seem to work either, audio doesn't play at all in this case.
And in the case of OpenSL ES, all audio generally stops when I start to play a longer asset file. It's similar to the behaviour of SoundPool described above.
Edit from the comments:
I don't want to interrupt other music players, it's the background
audio of a game, which shall play without interrupting the, for
example, music of a player. Games like Subway Surfer, Clash Royale and
such seem to have this achieved somehow, but I could not achieve it
via OpenSL ES, or MediaPlayer.
In order to play sound in background you can use SoundPool, AudioTracks and OpenSlES.
Soundpool: Use small files and make a sequence. In my last project i use 148 sound files (all small) in different scenarios and played using soundpool. Make a list and play one by one or in parallel. Also in games usually you have a small set of sound for particular scenario and it loops. Best is soundpool for that. You can also perform some effects like rate change. Also ogg files are way small, so use them.
AudioTrack: You can use RAW PCM data in audio track. If you want you can extract pcm data using MediaExtractor from almost all formats and use it. But it will be a little work for you in your case, but it is really good (supports huge data, even live streaming).
OpenSLES: Apparently android uses opensles in background for all its purpose. So using it will help you a lot. But it's not easy to get everything done on it. You need to learn more for lesser work.
I have been deeply working on OpenSlES for about 20 days and still i will say for small purpose use soundpool, medium to high level implementation use AudioTracks and for kickass implementation use OpenSLES.
PS: It will be bad effect on your user if you play game sound in background while they are playing their music or their call. Personal experience.
I am looking for a way to mix an audio into an already-playing mixed audio stream. For example, when a sound is being played half way through, I want to add in another sound to play together without interrupting the first sound to continue. I would also like to have an ability to withdraw a playing sound stream from the mixed playing stream. Going through Android's relevant document, I think that the only possible solution is to use native OpenSL ES via JNI to develop my own library where I can programmatically mix in/take out an audio stream from mixed audio streams. I would like to hear if anyone has a way to achieve it with less effort.
Thank you
Chris
Have you considered SoundPool?
http://developer.android.com/reference/android/media/SoundPool.html
I need to be able to play two or more (let's say, up to 5) short ogg files simultaneously. And by simultaneously I mean in perfect synchrony. I am able to load them to SoundPool and play, but this sometimes creates a noticeable difference in playback start time, which I want to get rid of.
From my understanding this can be avoided if mixing PCMs into one buffer and playing. But OGG's are not PCMs and need to be somehow efficiently decoded before playing and latency must be very low, ideally as soon as user presses the button. So I figured I need a way to stream OGG into PCM and as I receive buffers I would mix them and feed to AudioTrack. My requirement is Android 2.3.3+, so I cannot use any new codecs provided in Jelly Bean.
Also although OGGs themselves are small, there is a lot of them. So keeping them all decoded in memory (SoundPool or some pre-decoding) may case problems too.
Can someone give me a tip where to dig? Can OpenSL ES do that for me? Or should I think about integrating ffmpeg? And is it even possible to stream simultaneus files with low latency?
Thanks
You can play sounds using AssetPlayers, but this sometimes creates a noticeable difference in playback start time, yeh...
So, i recomend to decode ogg using Ogg Vorbis (like here) and then using this PCM buffer for BufferPlayer.
Btw, check this OpenSL ES wrappers
https://github.com/Suvitruf/Android-ndk/tree/master/OpenSLES
I am develping a media player for my learning purpose and I want to have crossfading feature in media player app. But I don't have clue where to start from. i tried searching on inernet but no luck. I am using Android MediaPlayer class for all media player related operation. Anyone know any workaround to achieve the same.
thanks for your support
Try to use the AudioTrack instead of MediaPlayer. Generally, I'd suggest the following plan:
Learn some sources for an app that uses AudioTrack. A good player can be found here
This is an aac audio player that uses JNI for aac-audio decoding.
Find a MP3 decoding library. The library should be a Java one (look at this for example, or it is possible there are another libraries (I did not use such java libraries)) or a C/C++ library (in this case you also will use it through JNI).
When you will get simple working MP3 player, add manual crossfading (this should be easy, if you aware of basics of digital audio).
Try to use two objects of MediaPlayer one after another with crossfade, like in this class https://github.com/psaravan/JamsMusicPlayer/blob/f165057dd664727ed06b9fac2c27557e5fb7e7ee/jamsMusicPlayer/src/main/java/com/jams/music/player/Services/AudioPlaybackService.java
When second mediaPlayer stars first pauses for while (On some devices mostly on samsung )so transition is not smooth. MediaPlayer has this issue and its been reported to google since long ago but still not resolved(https://issuetracker.google.com/issues/36931073) so we can do nothing about it. So I used Exoplayer for playing audio and it works very smoothly without any pause.