Develop Reference

Soundpool doesn't play maximum number of streams

I'm using soundpool in the game I'm developing. It plays piano notes most of the time playing several notes simultaneously. On some of the phones, the notes play very well. On the other hand when I tested on another phone some of the notes don't play. To investigate the problem, I've written the following code to understand the origin of the problem. There are 85 notes that play sequentially. When the delay between the notes is about 1 seconds, the notes play well without any skip or delay. This show all the samples are loaded without any error. But when I reduce the delay to 100ms, the notes start to overlap and after playing several notes sequentially in a proper way, the soundpool stops and waits the samples to finish before playing the remaining notes. On the other hand on another phone, even when the delay is 100ms all the notes play properly. Since the maximum number of streams is 100, the soundpool should play all the notes even if they are played simultaneously. I also checked the memory while running the following code and there weren't any overloading.
I have been working on this problem for a couple of days, so if you help me I'd appreciate it greatly.
spl=new SoundPool(100, AudioManager.STREAM_MUSIC, 0));
Handler handler = new Handler();
Runnable runnable = new Runnable() {
int cnt1=0,cnt2=0;
#Override
public void run() {
cnt1=cnt1+1;
if (cnt1>10){
stream=spl.play(sndIds[cnt2],volumesToBePlayed.get(i), volumesToBePlayed.get(i),1,0,1f);
Log.i("playing", String.valueOf(cnt2) + " " + String.valueOf(stream));
cnt1=0;
cnt2=cnt2+1;
}
if(cnt2>= sndIds.lenght){
handler.removeCallbacks(this);
} else {
handler.postDelayed(this, 100);
}
}
};
runnable.run();
EDIT: After investigating the problem a bit more, I've noticed that some phones are able to play more simultaneous samples than the others. In the piano applications, like mine, if there would be many simultaneous or overlapping play actions, I think, it is not a good idea to make the max number of streams large since the soundpool skips the notes played when the full capacity is reached. When the maximum is below the capacity of the phone, soundpool dosn't skip the new sound but stop the already playing and the oldest streams first if their priorities are the same. This sounds less disturbing to the user. I solved the problem by reducing the max number of streams to 12.

Why is my MediaPlayer audio distorted/clipping?

When I play a sound in my app it comes off as clippy and distorted. Here is a recording: recording.
Here is the sound file as it was uploaded to android studio: success sound cue
Here is the function that calls the sound from the .raw
public void playCorrectAnswerSound() {
final MediaPlayer mp = MediaPlayer.create(this, R.raw.correct);
mp.start();
}
Heres how I call it:
Thread t = new Thread(){
public void run(){
playCorrectAnswerSound();
}
};
t.start()
This is my first time debugging a sound related issue. I don't know what else to include in this post, so if you need more info please say so.
EDIT: I was asked to record more of the distortion. Here it is. Also I should say that after more testing, my physical device lacks the sound distortion while the sound distortion is present on 3 different emulators.

I'm going to say this is stuttering due to underrun (starvation) of the MediaPlayer's internal playback buffer. That is, the MediaPlayer can't supply data fast enough to keep up with the sound hardware (which suggests a severe lack of processing power). If the buffer starves, it'll start to play back old data (because it's a circular buffer). This causes a sharp phase transition, which sounds like a "click". Presumably the MediaPlayer recovers quickly enough that the "correct" sound resumes playing shortly thereafter.
Here is a picture of the spectrum from Audacity. 0-4KHz. The first row is the clean .mp3; the next four rows are the distorted recordings (in no particular order). All rows have been aligned in time, and are roughly the same amplitude. The large vertical stripes in the last four rows represent the distortion/clicks that you hear.

audio latency issues

In the application which I want to create, I face some technical obstacles. I have two music tracks in the application. For example, a user imports the music background as a first track. The second path is a voice recorded by the user to the rhythm of the first track played by the speaker device (or headphones). At this moment we face latency. After recording and playing back in the app, the user hears the loss of synchronisation between tracks, which occurs because of the microphone and speaker latencies.
Firstly, I try to detect the delay by filtering the input sound. I use android’s AudioRecord class, and the method read(). This method fills my short array with audio data.
I found that the initial values of this array are zeros so I decided to cut them out before I will start to write them into the output stream.
So I consider those zeros as a „warmup” latency of the microphone. Is this approach correct? This operation gives some results, but it doesn’t resolve the problem, and at this stage, I’m far away from that.
But the worse case is with the delay between starting the speakers and playing the music. This delay I cannot filter or detect. I tried to create some calibration feature which counts the delay. I play a „beep” sound through the speakers, and when I start to play it, I also begin to measure time. Then, I start recording and listen for this sound being detected by the microphone. When I recognise this sound in the app, I stop measuring time. I repeat this process several times, and the final value is the average from those results. That is how I try to measure the latency of the device. Now, when I have this value, I can simply shift the second track backwards to achieve synchronisation of both records (I will lose some initial milliseconds of the recording, but I skip this case, for now, there are some possibilities to fix it).
I thought that this approach would resolve the problem, but it turned out this is not as simple as I thought. I found two issues here:
1. Delay while playing two tracks simultaneously
2. Random in device audio latency.
The first: I play two tracks using AudioTrack class and I run method play() like this:
val firstTrack = //creating a track
val secondTrack = //creating a track
firstTrack.play()
secondTrack.play()
This code causes delays at the stage of playing tracks. Now, I don’t even have to think about latency while recording; I cannot play two tracks simultaneously without delays. I tested this with some external audio file (not recorded in my app) - I’m starting the same audio file using the code above, and I can see a delay. I also tried it with MediaPlayer class, and I have the same results. In this case, I even try to play tracks when callback OnPreparedListener invoke:
val firstTrack = //AudioPlayer
val secondTrack = //AudioPlayer
second.setOnPreparedListener {
first.start()
second.start()
}
And it doesn’t help.
I know that there is one more class provided by Android called SoundPool. According to the documentation, it can be better with playing tracks simultaneously, but I can’t use it because it supports only small audio files and that can't limit me.
How can I resolve this problem? How can I start playing two tracks precisely at the same time?
The second: Audio latency is not deterministic - sometimes it is smaller, and sometimes it’s huge, and it’s out of my hands. So measuring device latency can help but again - it cannot resolve the problem.
To sum up: is there any solution, which can give me exact latency per device (or app session?) or other triggers which detect actual delay, to provide the best synchronisation while playback two tracks at the same time?
Thank you in advance!

Synchronising audio for karaoke apps is tough. The main issue you seem to be facing is variable latency in the output stream.
This is almost certainly caused by "warm up" latency: the time it takes from hitting "play" on your backing track to the first frame of audio data being rendered by the audio device (e.g. headphones). This can have large variance and is difficult to measure.
The first (and easiest) thing to try is to use MODE_STREAM when constructing your AudioTrack and prime it with bufferSizeInBytes of data prior to calling play (more here). This should result in lower, more consistent "warm up" latency.
A better way is to use the Android NDK to have a continuously running audio stream which is just outputting silence until the moment you hit play, then start sending audio frames immediately. The only latency you have here is the continuous output latency.
If you decide to go down this route I recommend taking a look at the Oboe library (full disclosure: I am one of the authors).
To answer one of your specific questions...
Is there a way to calculate the latency of the audio output stream programatically?
Yes. The easiest way to explain this is with a code sample (this is C++ for the AAudio API but the principle is the same using Java AudioTrack):
// Get the index and time that a known audio frame was presented for playing
int64_t existingFrameIndex;
int64_t existingFramePresentationTime;
AAudioStream_getTimestamp(stream, CLOCK_MONOTONIC, &existingFrameIndex, &existingFramePresentationTime);
// Get the write index for the next audio frame
int64_t writeIndex = AAudioStream_getFramesWritten(stream);
// Calculate the number of frames between our known frame and the write index
int64_t frameIndexDelta = writeIndex - existingFrameIndex;
// Calculate the time which the next frame will be presented
int64_t frameTimeDelta = (frameIndexDelta * NANOS_PER_SECOND) / sampleRate_;
int64_t nextFramePresentationTime = existingFramePresentationTime + frameTimeDelta;
// Assume that the next frame will be written into the stream at the current time
int64_t nextFrameWriteTime = get_time_nanoseconds(CLOCK_MONOTONIC);
// Calculate the latency
*latencyMillis = (double) (nextFramePresentationTime - nextFrameWriteTime) / NANOS_PER_MILLISECOND;
A caveat: This method relies on accurate timestamps being reported by the audio hardware. I know this works on Google Pixel devices but have heard reports that it isn't so accurate on other devices so YMMV.

Following the answer of donturner, here's a Java version (that also uses other methods depending on the SDK version)
/** The audio latency has not been estimated yet */
private static long AUDIO_LATENCY_NOT_ESTIMATED = Long.MIN_VALUE+1;
/** The audio latency default value if we cannot estimate it */
private static long DEFAULT_AUDIO_LATENCY = 100L * 1000L * 1000L; // 100ms
/**
* Estimate the audio latency
*
* Not accurate at all, depends on SDK version, etc. But that's the best
* we can do.
*/
private static void estimateAudioLatency(AudioTrack track, long audioFramesWritten) {
long estimatedAudioLatency = AUDIO_LATENCY_NOT_ESTIMATED;
// First method. SDK >= 19.
if (Build.VERSION.SDK_INT >= 19 && track != null) {
AudioTimestamp audioTimestamp = new AudioTimestamp();
if (track.getTimestamp(audioTimestamp)) {
// Calculate the number of frames between our known frame and the write index
long frameIndexDelta = audioFramesWritten - audioTimestamp.framePosition;
// Calculate the time which the next frame will be presented
long frameTimeDelta = _framesToNanoSeconds(frameIndexDelta);
long nextFramePresentationTime = audioTimestamp.nanoTime + frameTimeDelta;
// Assume that the next frame will be written at the current time
long nextFrameWriteTime = System.nanoTime();
// Calculate the latency
estimatedAudioLatency = nextFramePresentationTime - nextFrameWriteTime;
}
}
// Second method. SDK >= 18.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED && Build.VERSION.SDK_INT >= 18) {
Method getLatencyMethod;
try {
getLatencyMethod = AudioTrack.class.getMethod("getLatency", (Class<?>[]) null);
estimatedAudioLatency = (Integer) getLatencyMethod.invoke(track, (Object[]) null) * 1000000L;
} catch (Exception ignored) {}
}
// If no method has successfully gave us a value, let's try a third method
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
AudioManager audioManager = (AudioManager) CRT.getInstance().getSystemService(Context.AUDIO_SERVICE);
try {
Method getOutputLatencyMethod = audioManager.getClass().getMethod("getOutputLatency", int.class);
estimatedAudioLatency = (Integer) getOutputLatencyMethod.invoke(audioManager, AudioManager.STREAM_MUSIC) * 1000000L;
} catch (Exception ignored) {}
}
// No method gave us a value. Let's use a default value. Better than nothing.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
estimatedAudioLatency = DEFAULT_AUDIO_LATENCY;
}
return estimatedAudioLatency
}
private static long _framesToNanoSeconds(long frames) {
return frames * 1000000000L / SAMPLE_RATE;
}

The android MediaPlayer class is notoriously slow to begin audio playback, I experienced an issue in an app I was creating where there was a greater than one second delay to begin playing an audio clip. I resolved it by switching to ExoPlayer which resulted in the playback starting within 100ms. I've also read that ffmpeg has even faster start audio startup time than ExoPlayer but I haven't used it so I can't make any promises.

Python Kivy: Play sound multiple times at once

I am using Python Kivy for an Android Game. I want to play a sound on an event
sound = SoundLoader.load("sound.wav")
def onEvent():
sound.play()
, and It works. But now the problem: Of course an event can, and in my case will happen again before the sound is done playing from the last event. And as the sounds are based on a play/pause idea I am getting a problem playing multiple sounds of the same object at once. That can be solved like this for first:
onEvent():
SoundLoader.load("sound.wav").play()
As this creates a new object all the time, and so is able to play it the same time another event plays the sound. But the problem using this method is quite obvious, because the sound must be loaded everytime the event occurs, and so causes a delay until it's played.
Is there a more useful way to do this?
{ if you don't understand what I am talking about, or just don't see the problem, feel free to ask }

You can workaround this by loading multiple instances of the sound.
For instance
sounds = [SoundLoader.load("sound.wav") for _ in range(10)]
index = 0
and then
def play():
sounds[index].play()
index = (index + 1) % len(sounds)
The more sounds you load, the more instances you can have playing at the same time (in this example 10).

How to play sounds at accurate periods of time across different devices in Android

I'm developing a game in Android and I came across a very annoying, hard-to-find bug. The issue is that when you are using SoundPool to play your sounds, you can actually loop whatever sound you are playing. In this case, the issue is the "running steps" sound; this sound gets executed quite fast and continually (around every 400ms) when the main character is running.
Now when playing the sound on a regular (not so powerful) device e.g. Samsung SII, the sound is played every 500ms - however, if I run the very same code on another device (let's say, Samsung SIV, Samsung SIII), the sound plays twice or even three times faster.
It seems like the more powerful the device hardware specs are, the faster it plays. On some devices, it plays so fast that you almost hear one solid continuous sound. I've been looking for techniques to set a specific ratio on the time period between sound plays, but it doesn't work properly and the issue remains. Does anyone know how to fix it, either using SoundPool, MediaPlayer, or any other sound-controlling API on Android?

You could use an AudioTrack to play a continuous stream of PCM data, since you would pass a stream you could be sure about the interval between sounds. the downside could be a little delay when first starting the sound but it depends on the minimum buffer size, and it depends, I think, on android version and device. On my galaxy s2 android 4.1 it was about 20ms.if you think this could be an option I can post some code

The problem with just looping or using a regular interval for something like footsteps is that you have a possible decoupling of sound and visuals. If your sound gets delays or sped up, or your visuals get delayed or sped up, you would have to adjust for that delay dynamically and automatically. You already have that issue right here
A better solution would be to place a trigger on the exact event which should trigger the sound (in this case, the foot being placed down), which then plays the sound. This also means that if you have multiple sources of the sound (like multiple footsteps), you don't have to manually start the sound with the right interval.

I can't seem to replicate the issue on Galaxy Nexus and Nexus S, does that mean I fixed it? Or maybe you could show what you're doing differently from this:
SoundPool soundPool = new SoundPool(4, AudioManager.STREAM_MUSIC, 100);
Integer sound1 = soundPool.load(this, R.raw.file1, 1);
Integer sound2 = soundPool.load(this, R.raw.file2, 1);
playSound(sound1);
public void playSound(int sound) {
AudioManager mgr = (AudioManager)getSystemService(Context.AUDIO_SERVICE);
float volume = mgr.getStreamVolume(AudioManager.STREAM_MUSIC)
/ mgr.getStreamMaxVolume(AudioManager.STREAM_MUSIC);
soundPool.play(sound, volume, volume, 1, -1, 1.0f);
}

If the problem is that you want to control the interval between the discrete sounds, The easiest way to do this is with a handler.
Basically you start a sound playing which is an asynchronous process. Then you use a handler to schedule a message to play the next sound sometime in the future. It will take some trial and error to get it right, but you will be guaranteed that the sound will start at the same interval after the previous sound on every device.
Here is some code to illustrate what I am talking about.
Here is a handler implementation you could use:
handler = new Handler() {
/* (non-Javadoc)
* #see android.os.Handler#handleMessage(android.os.Message)
*/
#Override
public void handleMessage(Message msg) {
if (msg.what == NEXT_ITEM_MSG) {
playNextSound();
}
else if (msg.what == SEQUENCE_COMPLETE_MSG) {
// notify a listener
listener.onSoundComplete()
}
}
};
Then you could write playNextSound like this:
private void playNextSound() {
if (mRunning) {
// Get the first item
SoundSequenceItem item = currentSequence.getNextSequenceItem();
if (item == null) {
Message msg = handler.obtainMessage(SEQUENCE_COMPLETE_MSG);
handler.sendMessage(msg);
return;
}
// Play the sound
int iSoundResId = item.getSoundResId();
if (iSoundResId != -1) {
player.playSoundNow(soundResId);
}
// schedule a message to advance to next item after duration
Message msg = handler.obtainMessage(NEXT_ITEM_MSG);
handler.sendMessageDelayed(msg, item.getDuration());
}
}
and your SoundSequenceItem could just be a simple class that has a sound file resource id and a duration. If you want to keep playing the sound while the character is moving you could do something like this:
public void onSoundComplete() {
if (character.isRunning()) {
currentSequence.addSequenceItem(new SoundSequenceItem(R.id.footsteps,500);
playNextSound();
}
}
Or you could modify playNextSound to continually play the same sound. Mine is written this way to be able to play different sounds in sequence.

I have had a lot of problems developing apps which used sounds and stuff like that. I would not suggest you to use SoundPool since it is bug-affected, and also be aware that looping sounds with SoundPool won't work on devices which are 4.3 and higher, see this open issue, at AOSP - Issue tracker.
I think that the solution is to go native and use OpenSL ES o similar libraries.