Develop Reference

LibGDX / Android: Playing sound effects makes the game stutter - Sound.play() takes 4 ms on a high-end device

Whenever I play a sound effect in my LibGDX game on an Android device, the game stutters. I have tried the game on three Samsung devices:
On Galaxy S7 Edge (2016, Android 8) and Galaxy Tab S 10.5 (2014, Android 6.0.1) the game is still playable, but not running smoothly whenever there are multiple sound effects being played (looping a sound effects are not a problem).
However on Galaxy S20 Ultra (2020, Android 10) the game is unplayable: Every call to Sound.play() takes 2...4 ms and causes "AUDIO_OUTPUT_FLAG_FAST denied by server; frameCount 0 -> 54276" error. This error does not appear with the other devices, but Sound.play() still takes 1...2 ms which of course is a considerable portion of a 16 ms frame.
So what I think is pretty clear is that the problem is in the Sound.play() method, not for example the number of concurrent sounds playing (which I have limited to 8 but have tried 4 as well), or that the Android device would be too slow to process the sounds (in which case a 6 year old GT should not outperform this year's high-end S20), or that the sound effect files would be too large (the one I'm using for testing was originally a 3.8 kB WAV). And yes, I am using AssetManager to load the sounds in advance.
I have now spent two long days doing research, found about 15-20 topics on different forums about what I believe is the same or related issue, and tried out all the suggested fixes without any success:
Changing audio format from WAV to OGG
Different sample rates: 44.1k, 48k, 96k on both formats (with 96k, there is no stutter and no error, but no audible sound either)
Adding silence of 1 or 2 seconds to the end of the sound effect (which itself is 41 ms long), with all the combinations of the above formats and sample rates.
Some say that looping a silent sound clip "in background" has solved the problem, but I anyway have another sound (car engine) looping constantly in the game and that seems to have no effect.
I have also seen suggestions to use Music class instead of Sound, but it's not suitable for collision sound effects with Box2D because pitch cannot be adjusted.
The only workaround that I found but have not tried yet is playing the sounds on a different thread. I have not tried it because I'm not familiar with multithreading and have not been able to find a comprehensive enough guide on how to do it (properly) in LibGDX. I also assume that this approach would be problematic for any sounds which may have to be paused, stopped or adjusted during playback by some actor from the main thread. Furthermore, according to https://github.com/libgdx/libgdx/wiki/Threading, "You should never perform multi-threaded operations on anything that is graphics or audio related".
Therefore, before I even start familiarizing my self on that topic (multithreading), I just wanted to ask once more: Is there really no other solution? It just doesn't feel right that a high-end Android device from this year cannot start the playback of a small WAV sound any faster than in 4 ms. There are lot of games in Play store with working sound effects and smooth gameplay, are they all really using multithreading?

I don't have a complete answer, but I'll share some ideas here.
My own anecdotal experience is that sound operations such as starting sound playback tend to be too time-consuming for a typical render thread on Android. I've tried a few different approaches (AudioTrack, SoundPool, etc.), and as best I can remember have gotten similar results in each case.
Putting the audio on a different thread seems like the most practical solution. I understand the hesitance if you're not familiar with multithreading, and I think you're right to be cautious, especially when using a third-party library. However, for simple tasks, Android supplies some fairly straightforward tools, like HandlerThread and Handler, that could perhaps be leveraged.
As for the LibGDX documentation saying not to perform multi-threaded operations on anything audio related, it's not clear to me whether that means don't do anything audio-related on a thread other than the render thread, or if it just means to keep all audio on a single thread, but that that thread doesn't have to be the render thread. If it's the latter, then putting audio on a separate thread might be an option.
I took a quick look at the LibGDX source code. I'd have to spend more time to better understand what's going on, but I see use of both AudioTrack and SoundPool, and I'm pretty sure I've run into this issue with both.
But, I also see some signs of asynchronous sound functionality. There are some classes with 'asynchronous' in the name that use a dedicated handler thread. I don't know if this functionality is documented (I couldn't find the documentation immediately) or otherwise supported, but it does seem to be present in the source code. The comments say there are some limitations, but it's not immediately clear to me what they are.
As for communication between the render thread and an audio thread, it would add some complexity, but you should be able to do it fairly straightforwardly using handlers or other similar tools. In fact, that's what the LibGDX code I looked at does - it creates a HandlerThread and uses a Handler (naturally) to post to it. It can still be difficult, especially when using a third-party library where you don't control where all audio operations occur. For example, LibGDX may always set up the audio objects on a specific thread (e.g. the render thread), which means if you use another thread, you'll be using the objects on a thread other than that on which they were created. I doubt that would be an issue, but it depends on the technology. (For example, the documentation for ExoPlayer says that instances should only be used from a single thread.)
In my own code I'm doing all audio myself, so I control it and can put everything on the same thread. That might be difficult or impossible with LibGDX, but the presence of the 'asynchronous' audio classes may be a hint that playing audio on a different thread is safe to do. (And maybe you can make use of those classes, assuming they're a supported part of the API.)

In case someone else has this issue. In your AndroidLauncher, override this.
#Override
public AndroidAudio createAudio(Context context, AndroidApplicationConfiguration config) {
return new AsynchronousAndroidAudio(context, config);
}
You MUST make sure you don't have any SoundId actions (eg. some_sound.Stop(sound_id)). As those will not work with AsynchronousAndroidAudio and will crash the game. So check that before you publish your game.

How does android deal with playing video given its memory constraints?

I'm curious if someone can point me to (or just describe) how, given that an Android application has an extremely limited memory space to play with (I think it's around 20 megs), a video player can load and play videos that are an order of magnitude or more larger in size. Is the app loading the video in some sort of chunks?
I ask because I have an app that has some video assets embedded and the app has grown to be about 80 megs and is just a total monster for compiling and debugging (without the assets the app would only be about 2 megs), and I was thinking that I should just remove the assets and have them download on the side and sit on the sdcard, rather than within the apk, but I'm worried that loading them and playing them at run time will bust through the app's memory allotment, and was hoping someone can shed some light on what my options are.
TIA

They buffer the video in manageable chunks, yes. Even if your videos are GBs in size, you won't hit a memory wall using standard video playing APIs. You can use the standard calls to setVideoURI or setVideoPath to point to the file and it will handle it from there. The same works for MediaPlayer in general if you're not using a VideoView
Downloading the videos outside the apk is still a good idea, though. If I had to download a new 80MB file for each incremental upgrade, I'd probably just uninstall instead. If you don't want to host them yourself, look into the supplemental apk option.

You can take your game into the native layer where you can bypass this limit. Take a look at a blog I wrote http://jnispot.blogspot.com/2012/10/jnindk-arrays-bitmaps-and-games-oh-my.html

I have no idea if this is what video apps do, but you can actually increase the memory beyond the normal limit by using malloc statements in native code. This bypasses the normal max of around 20 MB.
I have only used this for encryption on android devices where AES encryption needs rather karger amounts of memory when dealing with large encrypted files (even around 10MB files).

LibGDX Sounds Issue

I am developing a game for Android and the Desktop with LibGDX. I am having a problem with playing sounds. The game is a labyrinth style game, there are balls that roll around on the device using the accelerometer. When balls hit the border, or one another a sound is played. The volume is set based on the linear velocity of the collision. The problem is, when the balls get really close to the border, they bounce many times in a small period of time. This ends up bogging down the main thread, and the UI starts to stutter. In log-cat it says "reducing sample rate" or something like that, because it can't handle the load. Also, when there are a bunch of collisions, the sounds keep playing after there aren't anymore collisions.
I need each of the sounds to be played independently of the other sounds. I was thinking, maybe creating a separate thread for the sounds. Any help would be greatly appreciated.

I working now with the sounds of my game. The last LibGDX version works fine playing a lot of sounds simultaneously. All you need to do is, if you plan to play them on the same time is control the number of maximum sounds played (more sounds requires more resources of the device) and reduce the sample rate and quality of the most played. You can resample your sound with Audacity. Try to save it as a OGG file with less quality and try again. Also, you can create your sound as static and play it many times from the same sound without create a new one.
Hope this helps you.

Android Karaoke Text?

I have been scratching my head for the past week to do this effect on the text. http://www.youtube.com/watch?v=gB2PL33DMFs&feature=related
Would be great if someone can give me some tips or guidance or tutorial on how to do this.
thankz for reading and answering =D

If all you want is to display a movie with video and sound, a MediaPlayer can do that easily.
So I assume that you're actually talking about synchronizing some sort of animated display with a sound file being played separately. We did this using a MediaPlayer and polling getCurrentPosition from within an animation loop. This more or less works, but there are serious problems that need to be overcome. (All this deals with playing mp3 files; we didn't try any other audio formats).
First, your mp3 must be recorded at 44,100 Hz sampling rate. Otherwise the value returned by getCurrentPosition is way off. (We think it's scaled by the ratio of the actual sampling rate to 44,100, but we didn't verify this hypothesis.) A bit rate of 128,000 seems to work best.
Second, and more serious, is that the values returned by getCurrentPosition seem to drift away over time from the sound coming out of the device. After about 45 seconds, this starts to be quite noticeable. What's worse is that this drift is significantly different (but always present) in different OS levels, and perhaps from device to device. (We tested this in 2.1 and 2.2 on both emulators and real devices, and 3.0 on an emulator.) We suspected some sort of buffering problem, but couldn't really diagnose it. Our work-around was to break up longer mp3 files into short segments and chain their playback. Lots of bookkeeping aggravation. This is still under test, but so far it seems to have worked.

Ted Hopp: time drifting on MP3 files is likely caused by those MP3 files being VBR. I've been developing Karaoke apps for a while, and pretty much every toolkit - from Qt Phonon to ffmpeg - had problems reporting correct audio position on variable MP3 files. I assume this is because they all try to calculate the current audio position by using the number of decoded frames, which makes it unreliable for VBR MP3s. I described it in a user-friendly way in the Karaoke Lyrics Editor FAQ
Unfortunately the only solution I found is to recode MP3s to CBR. Another was to ditch the current position completely, and rely only on system clocks. That actually produced a better result for VBR MP3s, but still not as good as recoding them into CBR.

and again about memory issues with SoundPool

I am aware that SoundPool was intended to handle small fx like sounds and I made sure my 4 sound clips which I want to play one by one in some sequence are small enough.
I used ogg quality 0 and clips are 35kb, 14kb, 21kb and 23kb totaling 92kb of compressed audio. I have no idea how to estimate what the uncompressed size would be, but it should not be a lot, right?
So when I play 4 sounds in sequence, it works well for first 9 times (9 sequences x 4 sounds) but starts to cause memory issues on the nines sequence for one of the sounds. It is always sequence 9 when I start to see error.
What is the best way to handle that? I have a few ideas:
1) compress sounds even more (ogg quality -1 and mono instead of stereo)
2) unload and load sounds constantly using SoundPool.load and SoundPool.unload
3) release and recreate soundPool instance time from time
Is there anything else I can do? It is embarrassing that android api cannot handle so small clips. I wonder how people create games with a lot of sound effects...
Errors look like that:
ERROR/AudioFlinger(35): not enough memory for AudioTrack size=1048640
DEBUG/MemoryDealer(35): AudioTrack (0x25018,size=1048576)

It seems I was able to resolve my issue after:
1) downsampled my sound clips from 44 to 22khz. Uncompressed size was cut in half (I figured how to estimate uncompressed sound size - export your clip to uncompressed wav). I used nice open source tool Audacity
2) trim the sounds more to reduce duration
3) put try/catch around play() just in case (it does catch errors when it try to play the sound but cannot)

That seems odd. SoundPool should be expanding your audio clips into memory when it loads them, and once loaded I wouldn't expect you to run into memory issues later. When I've run into memory issues it was right at the beginning, not later on. You sure you're not loading more sounds later on?
To answer your other question, some games use JET Player and MIDI sounds instead of SoundPool. The JetBoy sample program that comes in the Android SDK is a great example of how to do this type of sound work in an app.