I was looking at Android's SoundPool as a mechanism to implement sound effects in my generic game development library. It seemed ideal.
But a little bit of research indicates that there all kinds of bugs in SoundPool. Are the bugs in SoundPool still relevant?
Because I'm developing a library, any bugs in SoundPool become bugs in my library, and I want to insulate my users from that.
So my question is basically: what API should I use for audio?
Using AudioTrack and writing my own mixer is not out of the question. But obviously it would be preferable to avoid doing that. And is there any API to provide decoding for me?
I need to be able to play a reasonable number of simultaneous sound effects (at least 16, let's say), and have even more open. Sounds need to start playing with low latency. WAV files need to be supported (MP3/Ogg is unimportant). Sound effects need to support seamless looping and dynamic, individual volume adjustment. The Android app lifecycle needs to be properly supported.
I have heard there is a 1MB limit somewhere for SoundPool, this is probably acceptable for each individual sound effect but not for all buffers/sounds. Can someone tell me exactly what the limit is on?
Finally, I need to be able to play background music as well, in compressed formats, with low CPU load. I assume MediaPlayer is ideal for this. Can it be used in parallel with another API?
I know a few people have been using MediaPlayer to fill in for SoundPool. But does it support the features that I need?
Are there any other audio APIs I've missed?
Just to add some more recent feedback on this issue. I've been using SoundPool for some time in an app with a fairly large user base for key press sounds. Our use case:
Must be played immediately
Up to 3+ sounds in parallel
We make use of the setRate across it's full range [0.5f-2.0f]
I've now experienced two major device specific issue and have decided to cut my losses and switch away from SoundPool
A large number of 4.4 LG devices (mostly the LG G2/G3 line) were having a native crash with their implementation of SoundPool. This was fixed in an update (eventually) but we still have a lot of users with un-upgraded devices
Sony Xperia devices currently have all sorts of issue with SoundPool as reported by others. In my case, I've discovered that if you use setRate with rate > 1.0f the SoundPool with start throwing exceptions until your app quits (and burn through a bunch of battery in the process).
TL;DR; I no longer think it's worth the danger/hassle of debugging SoundPool
Stick with OGG files and SoundPool will do you just fine. It's the nature of the multi-platform beast that is Android that there WILL be hardware configurations that will not work with every significant program, no matter how diligently the programmers try.
If this is a large and well-funded project, add to the funding one of each major phone for testing. It's actually much cheaper than the programmer time spent researching and trying to guess what their performance is.
Sorry. Seems as if this isn't the answer that you were looking for. Good luck!
DISCLAIMER: I have a small amount of experience with MediaPlayer, and no successful experience with the other APIs I mention, and the following information is based on what I've read in the DOCs and what I've read from google searches.
You could use mediaplayer (for the background music) with other audio APIs, since MediaPlayer automatically runs on it's own thread, but I believe it has a high-ish cpu load, and I don't think it would take compressed bits very well, but I'm not too sure.
There's also JetPlayer http://developer.android.com/reference/android/media/JetPlayer.html which seems like a lot of work to use effectively, but it would work very well with playing background music, then playing other sounds as needed in the game. From what I read of the DOCs, it takes a MIDI file (I think?) and you mute and unmute tracks to make it work how you want it to.
I like AudioTrack because it gives you the ability to edit sounds at runtime by changing the frequencies of the sound, and SoundPool can do the same.
Though for your situation, AudioTrack doesn't seem like it would work well, since playing two sounds would require two threads because AudioTrack is blocking (I'm pretty sure).
And with SoundPool, I'm thinking that since you have 16 sounds, maybe take two threads with one SoundPool in each thread and apply 8 sounds to each SoundPool. I don't really know though, as I've never even tried using SoundPool.
And again, my information is not based on experience, just what it appears from what I've read, so I may be completely or maybe just slightly wrong, or heck, who knows.
And I don't really know anything about the SoundPool bugs, since I haven't researched it.
Related
Whenever I play a sound effect in my LibGDX game on an Android device, the game stutters. I have tried the game on three Samsung devices:
On Galaxy S7 Edge (2016, Android 8) and Galaxy Tab S 10.5 (2014, Android 6.0.1) the game is still playable, but not running smoothly whenever there are multiple sound effects being played (looping a sound effects are not a problem).
However on Galaxy S20 Ultra (2020, Android 10) the game is unplayable: Every call to Sound.play() takes 2...4 ms and causes "AUDIO_OUTPUT_FLAG_FAST denied by server; frameCount 0 -> 54276" error. This error does not appear with the other devices, but Sound.play() still takes 1...2 ms which of course is a considerable portion of a 16 ms frame.
So what I think is pretty clear is that the problem is in the Sound.play() method, not for example the number of concurrent sounds playing (which I have limited to 8 but have tried 4 as well), or that the Android device would be too slow to process the sounds (in which case a 6 year old GT should not outperform this year's high-end S20), or that the sound effect files would be too large (the one I'm using for testing was originally a 3.8 kB WAV). And yes, I am using AssetManager to load the sounds in advance.
I have now spent two long days doing research, found about 15-20 topics on different forums about what I believe is the same or related issue, and tried out all the suggested fixes without any success:
Changing audio format from WAV to OGG
Different sample rates: 44.1k, 48k, 96k on both formats (with 96k, there is no stutter and no error, but no audible sound either)
Adding silence of 1 or 2 seconds to the end of the sound effect (which itself is 41 ms long), with all the combinations of the above formats and sample rates.
Some say that looping a silent sound clip "in background" has solved the problem, but I anyway have another sound (car engine) looping constantly in the game and that seems to have no effect.
I have also seen suggestions to use Music class instead of Sound, but it's not suitable for collision sound effects with Box2D because pitch cannot be adjusted.
The only workaround that I found but have not tried yet is playing the sounds on a different thread. I have not tried it because I'm not familiar with multithreading and have not been able to find a comprehensive enough guide on how to do it (properly) in LibGDX. I also assume that this approach would be problematic for any sounds which may have to be paused, stopped or adjusted during playback by some actor from the main thread. Furthermore, according to https://github.com/libgdx/libgdx/wiki/Threading, "You should never perform multi-threaded operations on anything that is graphics or audio related".
Therefore, before I even start familiarizing my self on that topic (multithreading), I just wanted to ask once more: Is there really no other solution? It just doesn't feel right that a high-end Android device from this year cannot start the playback of a small WAV sound any faster than in 4 ms. There are lot of games in Play store with working sound effects and smooth gameplay, are they all really using multithreading?
I don't have a complete answer, but I'll share some ideas here.
My own anecdotal experience is that sound operations such as starting sound playback tend to be too time-consuming for a typical render thread on Android. I've tried a few different approaches (AudioTrack, SoundPool, etc.), and as best I can remember have gotten similar results in each case.
Putting the audio on a different thread seems like the most practical solution. I understand the hesitance if you're not familiar with multithreading, and I think you're right to be cautious, especially when using a third-party library. However, for simple tasks, Android supplies some fairly straightforward tools, like HandlerThread and Handler, that could perhaps be leveraged.
As for the LibGDX documentation saying not to perform multi-threaded operations on anything audio related, it's not clear to me whether that means don't do anything audio-related on a thread other than the render thread, or if it just means to keep all audio on a single thread, but that that thread doesn't have to be the render thread. If it's the latter, then putting audio on a separate thread might be an option.
I took a quick look at the LibGDX source code. I'd have to spend more time to better understand what's going on, but I see use of both AudioTrack and SoundPool, and I'm pretty sure I've run into this issue with both.
But, I also see some signs of asynchronous sound functionality. There are some classes with 'asynchronous' in the name that use a dedicated handler thread. I don't know if this functionality is documented (I couldn't find the documentation immediately) or otherwise supported, but it does seem to be present in the source code. The comments say there are some limitations, but it's not immediately clear to me what they are.
As for communication between the render thread and an audio thread, it would add some complexity, but you should be able to do it fairly straightforwardly using handlers or other similar tools. In fact, that's what the LibGDX code I looked at does - it creates a HandlerThread and uses a Handler (naturally) to post to it. It can still be difficult, especially when using a third-party library where you don't control where all audio operations occur. For example, LibGDX may always set up the audio objects on a specific thread (e.g. the render thread), which means if you use another thread, you'll be using the objects on a thread other than that on which they were created. I doubt that would be an issue, but it depends on the technology. (For example, the documentation for ExoPlayer says that instances should only be used from a single thread.)
In my own code I'm doing all audio myself, so I control it and can put everything on the same thread. That might be difficult or impossible with LibGDX, but the presence of the 'asynchronous' audio classes may be a hint that playing audio on a different thread is safe to do. (And maybe you can make use of those classes, assuming they're a supported part of the API.)
In case someone else has this issue. In your AndroidLauncher, override this.
#Override
public AndroidAudio createAudio(Context context, AndroidApplicationConfiguration config) {
return new AsynchronousAndroidAudio(context, config);
}
You MUST make sure you don't have any SoundId actions (eg. some_sound.Stop(sound_id)). As those will not work with AsynchronousAndroidAudio and will crash the game. So check that before you publish your game.
In my android application I need to capture the user's speech from the microphone and then pass it to the server. Currently, I use the MediaRecorder class. However, it doesn't satisfy my needs, because I want to make glowing effect, based on the current volume of input sound, so I need an AudioStream, or something like that, I guess. Currently, I use the following:
this.recorder = new MediaRecorder();
this.recorder.setAudioSource(MediaRecorder.AudioSource.MIC);
this.recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
this.recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);
this.recorder.setOutputFile(FILENAME);
I am writing using API level 7, so I don't see any other AudioEncoders, but AMR Narrow Band. Maybe that's the reason of awful noise which I hear in my recordings.
The second problem I am facing is poor sound quality, noise, so I want to reduct (cancel, suppress) it, because it is really awful, especially on my noname chinese tablet. This should be server-side, because, as far as I know, requiers a lot of resources, and not all of the modern gadgets (especially noname chinese tablets) can do that as fast as possible. I am free to choose, which platform to use on the server, so it can be ASP.NET, PHP, JSP, or whatever helps me to make the sound better. Speaking about ASP.NET, I have come across a library, called NAudio, may be it can help me in some way. I know, that there is no any noise reduction solution built in the library, but I have found some examples on FFT and auto-corellation using it, so it may help.
To be honest, I have never worked with sound this close before and I have no idea where to start. I have googled a lot about noise reduction techniques, code examples and found nothing. You guys are my last hope.
Thanks in advance.
Have a look at this article.
Long story short, it uses MediaRecorder.AudioSource.VOICE_RECOGNITION instead of AudioSource.MIC, which gave me really good results and noise in the background did reduce very much.
The great thing about this solution is, it can be used with both AudioRecord and MediaRecorder class.
For audio capture you can use the AudioRecord class. This lets you record raw audio, i.e. you are not restricted to "narrow band" and you can also measure the volume.
Many smartphones have two microphones, one is the MIC you are using, the other one is near camera for video shooting, called CAMCORDER. You can get data from both of them to do noise reduction. There are many papers talking about audio noise reduction with multiple microphones.
Ref: http://developer.android.com/reference/android/media/MediaRecorder.AudioSource.html
https://www.google.com/search?q=noise+reduction+algorithm+with+two+mic
I have a song playing continuously in background using an Android MediaPlayer, by continuously I mean it loops forever.
I also have 25-30 short sounds that I play at discrete duration's using SoundPool Class, and they also have to play again and again with the main song.
The sounds are 0.5 second duration each and some of them are 0.7Mb in size, while the average size of a song would be approximately 0.4Mb.
I am able to do this entire thing smoothly on the device I made this app on, then I came to know that SoundPool has problems playing on Dual Core Phones. So I withdrew support for them.
Lately I tested the app on an older device running 2.3.3 and the new problem is that the sounds seem to play properly for first 4-5 cycles, i.e. till the main song plays 4-5 times, then at 6th time, there is a noticeable delay in the playing of sounds, I logged the values of SoundPool.play() and found out that delay is ~(1-2 sec), and this issue is only on lower end phones, why is this happening, I suppose that the problem is caused by the SoundPool class and is not at all an hardware issue since the device I tested it on had ample RAM i.e. 500Mb and a decent 600Mhz processor.
I really need to fix this as I expect my app to run on most devices, except dual core phone for time being.
Also I cannot use MediaPlayer to play my sounds as I do not want to increase CPU overhead and make my app lag more and also I do not know how many sounds I want to play before hand.
Any clue on how to fix this issue and reduce the lag, is most welcome.
Audio / Video playback is usually problematic since there are a lot of hardware / lowlevel software factors involved (there might be specialized decoding hardware on separate chips and/or embedded into the cpu and all of that in combination with software decoders, switching between bluetooth, headphonejack and the 2 speakers you have, a huge software stack from kernel to android system, etc). It is probably the most fragile subsystem of Android devices. A lot of devices have smaller quirks in that area and it isn't directly Mhz / RAM / Dualcore related. And unfortunately you have to live with some of them :(
Your best bet is to experiment with different ways to play / compress your audio. Things like restarting the playback after 5 cycles etc
I have been scratching my head for the past week to do this effect on the text. http://www.youtube.com/watch?v=gB2PL33DMFs&feature=related
Would be great if someone can give me some tips or guidance or tutorial on how to do this.
thankz for reading and answering =D
If all you want is to display a movie with video and sound, a MediaPlayer can do that easily.
So I assume that you're actually talking about synchronizing some sort of animated display with a sound file being played separately. We did this using a MediaPlayer and polling getCurrentPosition from within an animation loop. This more or less works, but there are serious problems that need to be overcome. (All this deals with playing mp3 files; we didn't try any other audio formats).
First, your mp3 must be recorded at 44,100 Hz sampling rate. Otherwise the value returned by getCurrentPosition is way off. (We think it's scaled by the ratio of the actual sampling rate to 44,100, but we didn't verify this hypothesis.) A bit rate of 128,000 seems to work best.
Second, and more serious, is that the values returned by getCurrentPosition seem to drift away over time from the sound coming out of the device. After about 45 seconds, this starts to be quite noticeable. What's worse is that this drift is significantly different (but always present) in different OS levels, and perhaps from device to device. (We tested this in 2.1 and 2.2 on both emulators and real devices, and 3.0 on an emulator.) We suspected some sort of buffering problem, but couldn't really diagnose it. Our work-around was to break up longer mp3 files into short segments and chain their playback. Lots of bookkeeping aggravation. This is still under test, but so far it seems to have worked.
Ted Hopp: time drifting on MP3 files is likely caused by those MP3 files being VBR. I've been developing Karaoke apps for a while, and pretty much every toolkit - from Qt Phonon to ffmpeg - had problems reporting correct audio position on variable MP3 files. I assume this is because they all try to calculate the current audio position by using the number of decoded frames, which makes it unreliable for VBR MP3s. I described it in a user-friendly way in the Karaoke Lyrics Editor FAQ
Unfortunately the only solution I found is to recode MP3s to CBR. Another was to ditch the current position completely, and rely only on system clocks. That actually produced a better result for VBR MP3s, but still not as good as recoding them into CBR.
I have an app that is streaming audio content and sometimes it just
stops all of the suddent.
the logcat windows shows --
AudioHardware pcm playback is going to standby
and that's it.
I saw on another thread (pun intended) that someone was saying it was
because he was using too many threads. Could that really be causing
this? Could i give the audio thread higher priority?
Anyway to prevent the audio hardware pcm from going to standby?
I too am having this problem, and have nearly exhausted all my resources trying to find a workaround. I found an article on how to prevent this with AudioRecord, and maybe the same principles he talks about there will apply to you as well:
http://hificorder.blogspot.com/
In my case, I am having lockups using SoundPool. Increasing the number of max streams allowed prevented this from happening in most cases, but it still happens in other situations. I really wish there was better information about how to handled this correctly, because as it is, this is a show stopping bug in my app which may force me to abandon it all together.
Be sure to post back if you have any findings to add.