I am developing a musical application that plays several sounds all together, like a sort of multitrack program. A timer triggers when it's the moment for all the sounds to play. Sounds have to be played exactly on the beat and perfectly synchronized and mixed.
The easiest solution i found is to have one MediaPlayer for each sound, initialize all of them at the very beginning, with MediaPlayer.create() inside my app's OnCreate().
To make it more reliable and quick I made TWO mediaplayers for each sound.
Then I have a timer that calls simple loop, similar to this:
for (int w=0; w<SOUNDS; w++)
if (must_play(w)) {
if (mp[w].isPlaying()) {
if (mp2[w].isPlaying())
mp[w].seekTo(0);
else mp2[w].start();
} else mp[w].start();
}
I used .seekTo(0) because I found it slightly faster than making .stop() and .start().
But the sounds are not always perfectly synchronized. A 1/10 of second of delay between the .start() of two mediaplayers is very annoying if those two sounds are two drums that are supposed to play perfectly in line.
Is there a way to force all mediaplayers that are instructed to .start() to effectly play all at once?
Please note that the sounds may be very short, like drum sounds: the problem is not to keep the multiple media players synchronized over time, but to make them start exactly all together, without delay.
The question is rather tricky because in my opinion involves two problems:
Prioritization, above other processes that may slow down the timer or the app itself.
How to create a single command/object/method (or whatever) that runs all the sounds atomically.
Thank you.
I made it, just used the SoundPool class instead of MediaPlayer. It's much more responsive, and handles very well multitracks, even cutting the oldest sound played if the streams are not enough. Although the questions weren't answered, the problem is fixed.
Related
Whenever I play a sound effect in my LibGDX game on an Android device, the game stutters. I have tried the game on three Samsung devices:
On Galaxy S7 Edge (2016, Android 8) and Galaxy Tab S 10.5 (2014, Android 6.0.1) the game is still playable, but not running smoothly whenever there are multiple sound effects being played (looping a sound effects are not a problem).
However on Galaxy S20 Ultra (2020, Android 10) the game is unplayable: Every call to Sound.play() takes 2...4 ms and causes "AUDIO_OUTPUT_FLAG_FAST denied by server; frameCount 0 -> 54276" error. This error does not appear with the other devices, but Sound.play() still takes 1...2 ms which of course is a considerable portion of a 16 ms frame.
So what I think is pretty clear is that the problem is in the Sound.play() method, not for example the number of concurrent sounds playing (which I have limited to 8 but have tried 4 as well), or that the Android device would be too slow to process the sounds (in which case a 6 year old GT should not outperform this year's high-end S20), or that the sound effect files would be too large (the one I'm using for testing was originally a 3.8 kB WAV). And yes, I am using AssetManager to load the sounds in advance.
I have now spent two long days doing research, found about 15-20 topics on different forums about what I believe is the same or related issue, and tried out all the suggested fixes without any success:
Changing audio format from WAV to OGG
Different sample rates: 44.1k, 48k, 96k on both formats (with 96k, there is no stutter and no error, but no audible sound either)
Adding silence of 1 or 2 seconds to the end of the sound effect (which itself is 41 ms long), with all the combinations of the above formats and sample rates.
Some say that looping a silent sound clip "in background" has solved the problem, but I anyway have another sound (car engine) looping constantly in the game and that seems to have no effect.
I have also seen suggestions to use Music class instead of Sound, but it's not suitable for collision sound effects with Box2D because pitch cannot be adjusted.
The only workaround that I found but have not tried yet is playing the sounds on a different thread. I have not tried it because I'm not familiar with multithreading and have not been able to find a comprehensive enough guide on how to do it (properly) in LibGDX. I also assume that this approach would be problematic for any sounds which may have to be paused, stopped or adjusted during playback by some actor from the main thread. Furthermore, according to https://github.com/libgdx/libgdx/wiki/Threading, "You should never perform multi-threaded operations on anything that is graphics or audio related".
Therefore, before I even start familiarizing my self on that topic (multithreading), I just wanted to ask once more: Is there really no other solution? It just doesn't feel right that a high-end Android device from this year cannot start the playback of a small WAV sound any faster than in 4 ms. There are lot of games in Play store with working sound effects and smooth gameplay, are they all really using multithreading?
I don't have a complete answer, but I'll share some ideas here.
My own anecdotal experience is that sound operations such as starting sound playback tend to be too time-consuming for a typical render thread on Android. I've tried a few different approaches (AudioTrack, SoundPool, etc.), and as best I can remember have gotten similar results in each case.
Putting the audio on a different thread seems like the most practical solution. I understand the hesitance if you're not familiar with multithreading, and I think you're right to be cautious, especially when using a third-party library. However, for simple tasks, Android supplies some fairly straightforward tools, like HandlerThread and Handler, that could perhaps be leveraged.
As for the LibGDX documentation saying not to perform multi-threaded operations on anything audio related, it's not clear to me whether that means don't do anything audio-related on a thread other than the render thread, or if it just means to keep all audio on a single thread, but that that thread doesn't have to be the render thread. If it's the latter, then putting audio on a separate thread might be an option.
I took a quick look at the LibGDX source code. I'd have to spend more time to better understand what's going on, but I see use of both AudioTrack and SoundPool, and I'm pretty sure I've run into this issue with both.
But, I also see some signs of asynchronous sound functionality. There are some classes with 'asynchronous' in the name that use a dedicated handler thread. I don't know if this functionality is documented (I couldn't find the documentation immediately) or otherwise supported, but it does seem to be present in the source code. The comments say there are some limitations, but it's not immediately clear to me what they are.
As for communication between the render thread and an audio thread, it would add some complexity, but you should be able to do it fairly straightforwardly using handlers or other similar tools. In fact, that's what the LibGDX code I looked at does - it creates a HandlerThread and uses a Handler (naturally) to post to it. It can still be difficult, especially when using a third-party library where you don't control where all audio operations occur. For example, LibGDX may always set up the audio objects on a specific thread (e.g. the render thread), which means if you use another thread, you'll be using the objects on a thread other than that on which they were created. I doubt that would be an issue, but it depends on the technology. (For example, the documentation for ExoPlayer says that instances should only be used from a single thread.)
In my own code I'm doing all audio myself, so I control it and can put everything on the same thread. That might be difficult or impossible with LibGDX, but the presence of the 'asynchronous' audio classes may be a hint that playing audio on a different thread is safe to do. (And maybe you can make use of those classes, assuming they're a supported part of the API.)
In case someone else has this issue. In your AndroidLauncher, override this.
#Override
public AndroidAudio createAudio(Context context, AndroidApplicationConfiguration config) {
return new AsynchronousAndroidAudio(context, config);
}
You MUST make sure you don't have any SoundId actions (eg. some_sound.Stop(sound_id)). As those will not work with AsynchronousAndroidAudio and will crash the game. So check that before you publish your game.
I'm looking to create a drum machine in ActionScript 3 (as an Adobe AIR Android app), which will keep to a user defined tempo (BPM).
I am struggling to find a way to keep the project in time, I have, at the moment, made it so that 5 different sounds are represented in rows of 8 squares, and the user can click each square to choose when to play that sound (hope this makes sense).
At the moment I am using Timer to keep the project in time, which is very laggy and inconsistent.
using timer is a bad idea for this, there I said it...
The issue is that the timer has a drift and fires several milliseconds later.
Try a simple test where you have a timer that executes every 500ms, and then compare the getTimer() count. What I have found in my experiments that the timer is continually off and it looks like it doesn't self correct. I've tried using a self-correcting timer, that changes the firing time based on the getTimer() difference since last run, but it's still not reliable. and anytime your processor's load picks up, the timer will be off anyway.
The correct way of dealing with this is to use byteArray data as a source for the sound. Based on the calculation of sampling resolution you can populate the stream with the data in advance, and the sound will play on time, pretty much guaranteed. I haven't gone as far as to create something that does this myself. But there are several libraries that you can utilize that can help you with this.
My top two decremented libraries are SiON and tonfall
you can see a sample of SiON here http://wonderfl.net/c/qf4b
and tonfall example at http://tonematrix.audiotool.com/
While I haven't tried them on android, I think either should work
I am developing a game for Android and the Desktop with LibGDX. I am having a problem with playing sounds. The game is a labyrinth style game, there are balls that roll around on the device using the accelerometer. When balls hit the border, or one another a sound is played. The volume is set based on the linear velocity of the collision. The problem is, when the balls get really close to the border, they bounce many times in a small period of time. This ends up bogging down the main thread, and the UI starts to stutter. In log-cat it says "reducing sample rate" or something like that, because it can't handle the load. Also, when there are a bunch of collisions, the sounds keep playing after there aren't anymore collisions.
I need each of the sounds to be played independently of the other sounds. I was thinking, maybe creating a separate thread for the sounds. Any help would be greatly appreciated.
I working now with the sounds of my game. The last LibGDX version works fine playing a lot of sounds simultaneously. All you need to do is, if you plan to play them on the same time is control the number of maximum sounds played (more sounds requires more resources of the device) and reduce the sample rate and quality of the most played. You can resample your sound with Audacity. Try to save it as a OGG file with less quality and try again. Also, you can create your sound as static and play it many times from the same sound without create a new one.
Hope this helps you.
I have searched for this online, but am still a bit confused (as I'm sure others will be if they think of something like this). I'd like to preface by saying that this is not for homework and/or profit.
I wanted to create an app that could listen to your microwave as you prepare popcorn. It would work by sounding an alarm when there's a certain time interval between pops (say 5-6 seconds). Again, this is simply a project to keep me occupied - not for a class.
Either way, I'm having trouble trying to figure out how to analyze the audio intake in real-time. That is, I need a way to log the time when a "pop" occurs. So that you guys don't think I didn't do any research into the matter, I've checked out this SO question and have extensively searched the AudioRecord function list.
I'm thinking that I will probably have to do something with one of the versions of read() and then compare the recorded audio every 2 seconds or so to the recorded audio of a "pop" (i.e. if 70% or more of the byte[] audioData array is the same as that of a popping sound, then log the time). Can anyone with Android audio input experience let me know if I'm at least on the right track? This is not a question of me wanting you to code anything for me, but a question as to whether I'm on the correct track, and, if not, which direction I should head instead.
I think I have an easier way.
You could use the MediaRecorder 's getMaxAmplitude method.
Anytime your recorder detects a big jump in amplitude, you have detected a corn pop!
Check out this code (ignore the playback part): Playing back sound coming from microphone in real-time
Basically the idea is that you will have to take the value of each 16-bit sample (which corresponds to the value of the wave at that time). Using the sampling rate, you can calculate the time between peaks in volume. I think that might accomplish what you want.
this may be a bit overkill, but there is a framework from MIT media labs called funf: http://code.google.com/p/funf-open-sensing-framework/
They already created classes for audio input and some analysis (FFT and the like), also saving to files or uploading is implemented as far as I've seen, and they handle most of the sensors available on the phone.
You can also get inspired from the code they wrote, which I think is pretty good.
I was looking at Android's SoundPool as a mechanism to implement sound effects in my generic game development library. It seemed ideal.
But a little bit of research indicates that there all kinds of bugs in SoundPool. Are the bugs in SoundPool still relevant?
Because I'm developing a library, any bugs in SoundPool become bugs in my library, and I want to insulate my users from that.
So my question is basically: what API should I use for audio?
Using AudioTrack and writing my own mixer is not out of the question. But obviously it would be preferable to avoid doing that. And is there any API to provide decoding for me?
I need to be able to play a reasonable number of simultaneous sound effects (at least 16, let's say), and have even more open. Sounds need to start playing with low latency. WAV files need to be supported (MP3/Ogg is unimportant). Sound effects need to support seamless looping and dynamic, individual volume adjustment. The Android app lifecycle needs to be properly supported.
I have heard there is a 1MB limit somewhere for SoundPool, this is probably acceptable for each individual sound effect but not for all buffers/sounds. Can someone tell me exactly what the limit is on?
Finally, I need to be able to play background music as well, in compressed formats, with low CPU load. I assume MediaPlayer is ideal for this. Can it be used in parallel with another API?
I know a few people have been using MediaPlayer to fill in for SoundPool. But does it support the features that I need?
Are there any other audio APIs I've missed?
Just to add some more recent feedback on this issue. I've been using SoundPool for some time in an app with a fairly large user base for key press sounds. Our use case:
Must be played immediately
Up to 3+ sounds in parallel
We make use of the setRate across it's full range [0.5f-2.0f]
I've now experienced two major device specific issue and have decided to cut my losses and switch away from SoundPool
A large number of 4.4 LG devices (mostly the LG G2/G3 line) were having a native crash with their implementation of SoundPool. This was fixed in an update (eventually) but we still have a lot of users with un-upgraded devices
Sony Xperia devices currently have all sorts of issue with SoundPool as reported by others. In my case, I've discovered that if you use setRate with rate > 1.0f the SoundPool with start throwing exceptions until your app quits (and burn through a bunch of battery in the process).
TL;DR; I no longer think it's worth the danger/hassle of debugging SoundPool
Stick with OGG files and SoundPool will do you just fine. It's the nature of the multi-platform beast that is Android that there WILL be hardware configurations that will not work with every significant program, no matter how diligently the programmers try.
If this is a large and well-funded project, add to the funding one of each major phone for testing. It's actually much cheaper than the programmer time spent researching and trying to guess what their performance is.
Sorry. Seems as if this isn't the answer that you were looking for. Good luck!
DISCLAIMER: I have a small amount of experience with MediaPlayer, and no successful experience with the other APIs I mention, and the following information is based on what I've read in the DOCs and what I've read from google searches.
You could use mediaplayer (for the background music) with other audio APIs, since MediaPlayer automatically runs on it's own thread, but I believe it has a high-ish cpu load, and I don't think it would take compressed bits very well, but I'm not too sure.
There's also JetPlayer http://developer.android.com/reference/android/media/JetPlayer.html which seems like a lot of work to use effectively, but it would work very well with playing background music, then playing other sounds as needed in the game. From what I read of the DOCs, it takes a MIDI file (I think?) and you mute and unmute tracks to make it work how you want it to.
I like AudioTrack because it gives you the ability to edit sounds at runtime by changing the frequencies of the sound, and SoundPool can do the same.
Though for your situation, AudioTrack doesn't seem like it would work well, since playing two sounds would require two threads because AudioTrack is blocking (I'm pretty sure).
And with SoundPool, I'm thinking that since you have 16 sounds, maybe take two threads with one SoundPool in each thread and apply 8 sounds to each SoundPool. I don't really know though, as I've never even tried using SoundPool.
And again, my information is not based on experience, just what it appears from what I've read, so I may be completely or maybe just slightly wrong, or heck, who knows.
And I don't really know anything about the SoundPool bugs, since I haven't researched it.