Develop Reference

audio latency issues

In the application which I want to create, I face some technical obstacles. I have two music tracks in the application. For example, a user imports the music background as a first track. The second path is a voice recorded by the user to the rhythm of the first track played by the speaker device (or headphones). At this moment we face latency. After recording and playing back in the app, the user hears the loss of synchronisation between tracks, which occurs because of the microphone and speaker latencies.
Firstly, I try to detect the delay by filtering the input sound. I use android’s AudioRecord class, and the method read(). This method fills my short array with audio data.
I found that the initial values of this array are zeros so I decided to cut them out before I will start to write them into the output stream.
So I consider those zeros as a „warmup” latency of the microphone. Is this approach correct? This operation gives some results, but it doesn’t resolve the problem, and at this stage, I’m far away from that.
But the worse case is with the delay between starting the speakers and playing the music. This delay I cannot filter or detect. I tried to create some calibration feature which counts the delay. I play a „beep” sound through the speakers, and when I start to play it, I also begin to measure time. Then, I start recording and listen for this sound being detected by the microphone. When I recognise this sound in the app, I stop measuring time. I repeat this process several times, and the final value is the average from those results. That is how I try to measure the latency of the device. Now, when I have this value, I can simply shift the second track backwards to achieve synchronisation of both records (I will lose some initial milliseconds of the recording, but I skip this case, for now, there are some possibilities to fix it).
I thought that this approach would resolve the problem, but it turned out this is not as simple as I thought. I found two issues here:
1. Delay while playing two tracks simultaneously
2. Random in device audio latency.
The first: I play two tracks using AudioTrack class and I run method play() like this:
val firstTrack = //creating a track
val secondTrack = //creating a track
firstTrack.play()
secondTrack.play()
This code causes delays at the stage of playing tracks. Now, I don’t even have to think about latency while recording; I cannot play two tracks simultaneously without delays. I tested this with some external audio file (not recorded in my app) - I’m starting the same audio file using the code above, and I can see a delay. I also tried it with MediaPlayer class, and I have the same results. In this case, I even try to play tracks when callback OnPreparedListener invoke:
val firstTrack = //AudioPlayer
val secondTrack = //AudioPlayer
second.setOnPreparedListener {
first.start()
second.start()
}
And it doesn’t help.
I know that there is one more class provided by Android called SoundPool. According to the documentation, it can be better with playing tracks simultaneously, but I can’t use it because it supports only small audio files and that can't limit me.
How can I resolve this problem? How can I start playing two tracks precisely at the same time?
The second: Audio latency is not deterministic - sometimes it is smaller, and sometimes it’s huge, and it’s out of my hands. So measuring device latency can help but again - it cannot resolve the problem.
To sum up: is there any solution, which can give me exact latency per device (or app session?) or other triggers which detect actual delay, to provide the best synchronisation while playback two tracks at the same time?
Thank you in advance!

Synchronising audio for karaoke apps is tough. The main issue you seem to be facing is variable latency in the output stream.
This is almost certainly caused by "warm up" latency: the time it takes from hitting "play" on your backing track to the first frame of audio data being rendered by the audio device (e.g. headphones). This can have large variance and is difficult to measure.
The first (and easiest) thing to try is to use MODE_STREAM when constructing your AudioTrack and prime it with bufferSizeInBytes of data prior to calling play (more here). This should result in lower, more consistent "warm up" latency.
A better way is to use the Android NDK to have a continuously running audio stream which is just outputting silence until the moment you hit play, then start sending audio frames immediately. The only latency you have here is the continuous output latency.
If you decide to go down this route I recommend taking a look at the Oboe library (full disclosure: I am one of the authors).
To answer one of your specific questions...
Is there a way to calculate the latency of the audio output stream programatically?
Yes. The easiest way to explain this is with a code sample (this is C++ for the AAudio API but the principle is the same using Java AudioTrack):
// Get the index and time that a known audio frame was presented for playing
int64_t existingFrameIndex;
int64_t existingFramePresentationTime;
AAudioStream_getTimestamp(stream, CLOCK_MONOTONIC, &existingFrameIndex, &existingFramePresentationTime);
// Get the write index for the next audio frame
int64_t writeIndex = AAudioStream_getFramesWritten(stream);
// Calculate the number of frames between our known frame and the write index
int64_t frameIndexDelta = writeIndex - existingFrameIndex;
// Calculate the time which the next frame will be presented
int64_t frameTimeDelta = (frameIndexDelta * NANOS_PER_SECOND) / sampleRate_;
int64_t nextFramePresentationTime = existingFramePresentationTime + frameTimeDelta;
// Assume that the next frame will be written into the stream at the current time
int64_t nextFrameWriteTime = get_time_nanoseconds(CLOCK_MONOTONIC);
// Calculate the latency
*latencyMillis = (double) (nextFramePresentationTime - nextFrameWriteTime) / NANOS_PER_MILLISECOND;
A caveat: This method relies on accurate timestamps being reported by the audio hardware. I know this works on Google Pixel devices but have heard reports that it isn't so accurate on other devices so YMMV.

Following the answer of donturner, here's a Java version (that also uses other methods depending on the SDK version)
/** The audio latency has not been estimated yet */
private static long AUDIO_LATENCY_NOT_ESTIMATED = Long.MIN_VALUE+1;
/** The audio latency default value if we cannot estimate it */
private static long DEFAULT_AUDIO_LATENCY = 100L * 1000L * 1000L; // 100ms
/**
* Estimate the audio latency
*
* Not accurate at all, depends on SDK version, etc. But that's the best
* we can do.
*/
private static void estimateAudioLatency(AudioTrack track, long audioFramesWritten) {
long estimatedAudioLatency = AUDIO_LATENCY_NOT_ESTIMATED;
// First method. SDK >= 19.
if (Build.VERSION.SDK_INT >= 19 && track != null) {
AudioTimestamp audioTimestamp = new AudioTimestamp();
if (track.getTimestamp(audioTimestamp)) {
// Calculate the number of frames between our known frame and the write index
long frameIndexDelta = audioFramesWritten - audioTimestamp.framePosition;
// Calculate the time which the next frame will be presented
long frameTimeDelta = _framesToNanoSeconds(frameIndexDelta);
long nextFramePresentationTime = audioTimestamp.nanoTime + frameTimeDelta;
// Assume that the next frame will be written at the current time
long nextFrameWriteTime = System.nanoTime();
// Calculate the latency
estimatedAudioLatency = nextFramePresentationTime - nextFrameWriteTime;
}
}
// Second method. SDK >= 18.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED && Build.VERSION.SDK_INT >= 18) {
Method getLatencyMethod;
try {
getLatencyMethod = AudioTrack.class.getMethod("getLatency", (Class<?>[]) null);
estimatedAudioLatency = (Integer) getLatencyMethod.invoke(track, (Object[]) null) * 1000000L;
} catch (Exception ignored) {}
}
// If no method has successfully gave us a value, let's try a third method
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
AudioManager audioManager = (AudioManager) CRT.getInstance().getSystemService(Context.AUDIO_SERVICE);
try {
Method getOutputLatencyMethod = audioManager.getClass().getMethod("getOutputLatency", int.class);
estimatedAudioLatency = (Integer) getOutputLatencyMethod.invoke(audioManager, AudioManager.STREAM_MUSIC) * 1000000L;
} catch (Exception ignored) {}
}
// No method gave us a value. Let's use a default value. Better than nothing.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
estimatedAudioLatency = DEFAULT_AUDIO_LATENCY;
}
return estimatedAudioLatency
}
private static long _framesToNanoSeconds(long frames) {
return frames * 1000000000L / SAMPLE_RATE;
}

The android MediaPlayer class is notoriously slow to begin audio playback, I experienced an issue in an app I was creating where there was a greater than one second delay to begin playing an audio clip. I resolved it by switching to ExoPlayer which resulted in the playback starting within 100ms. I've also read that ffmpeg has even faster start audio startup time than ExoPlayer but I haven't used it so I can't make any promises.

Android AudioTrack stream cuts out early

I'm trying to play some looping sound in Android, and I have that going pretty well for me. All good things must come to an end, though, and I would like for that to include my audio loop. However, if I call AudioTrack.release() after this loop, as I should, the end of my audio stream gets cut off - there is extra data that I know I'm supposed to hear, but don't.
I've verified this by putting in a Thread.sleep(2000) before the release - the sound plays correctly with that in there. My code looks something like this:
// Initialize Audiotrack
int minBufferSize = AudioTrack.getMinBufferSize(SAMPLE_RATE, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT);
mAudioTrack = new AudioTrack(AudioManager.STREAM_MUSIC, SAMPLE_RATE, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT, 2 * minBufferSize, AudioTrack.MODE_STREAM);
mAudioTrack.play();
// Play looping sound
while (stuff) {
mAudioTrack.write(stuff);
}
// Play one last bit of sound before returning
mAudioTrack.write(lastSound);
// Block until the AudioTrack has played everything we've given it
Thread.sleep(2000);
// Get rid of the Audiotrack
mAudioTrack.release();
I suppose I could leave the Thread.sleep(2000) in there and call it a day, but that sounds messy and irresponsible to me. I'd like to either have a while() loop block for the most appropriate amount of time, or use AudioTrack.setPlaybackPositionUpdateListener() and put the release() in there.
If I go the first route, I need something to pend on, and AudioTrack.getPlayState() appears to always report the track as playing. So I'm stuck there.
If I go the second route, I need a way of getting the position in the AudioTrack buffer that was written to last, so I can tell the AudioTrack what position I'm waiting for it to play up to. I don't have any ideas as to how to get that information, though.
I guess I don't really care which way I do it, so any help towards solving the problem one way or the other would be much appreciated.

The problem is related to the buffer size in the AudioTrack.
Imagine the minBufferSize is 8k. This means that the AudioTrack will play sound when the buffer is full.
mAudioTrack.write(stuff);
If stuff is only 4K, the AudioTrack will wait until the next call to write until it has enough data to play.
Conclusion: You need to keep track on how much data you have written, and at the end of your playback feed the AudioTrack with some dummy bytes to complete minBufferSize. To make thing easier you could just feed a whole minBufferSize amount of silence bytes.
By the way, to feed dummy or silence just fill the data with zeroes.

Android active noise cancellation

I'm working a somewhat ambitious project to get active noise-reduction achieved on Android with earbuds or headphones on.
My objective is to record ambient noise with the android phone mic, invert the phase (a simple *-1 on the short-value pulled from the Audio Record?), and playback that inverted waveform through the headphones. If the latency and amplitude are close to correct, it should nullify a good amount of mechanical structured noise in the environment.
Here's what I've got so far:
#Override
public void run()
{
Log.i("Audio", "Running Audio Thread");
AudioRecord recorder = null;
AudioTrack track = null;
short[][] buffers = new short[256][160];
int ix = 0;
/*
* Initialize buffer to hold continuously recorded audio data, start recording, and start
* playback.
*/
try
{
int N = AudioRecord.getMinBufferSize(8000,AudioFormat.CHANNEL_IN_MONO,AudioFormat.ENCODING_PCM_16BIT);
recorder = new AudioRecord(MediaRecorder.AudioSource.MIC, 8000, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, N*10);
//NoiseSuppressor ns = NoiseSuppressor.create(recorder.getAudioSessionId());
//ns.setEnabled(true);
track = new AudioTrack(AudioManager.STREAM_MUSIC, 8000,
AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT, N*10, AudioTrack.MODE_STREAM);
recorder.startRecording();
track.play();
/*
* Loops until something outside of this thread stops it.
* Reads the data from the recorder and writes it to the audio track for playback.
*/
while(!stopped)
{
short[] buffer = buffers[ix++ % buffers.length];
N = recorder.read(buffer,0,buffer.length);
for(int iii = 0;iii<buffer.length;iii++){
//Log.i("Data","Value: "+buffer[iii]);
buffer[iii] = buffer[iii] *= -1;
}
track.write(buffer, 0, buffer.length);
}
}
catch(Throwable x)
{
Log.w("Audio", "Error reading voice audio", x);
}
/*
* Frees the thread's resources after the loop completes so that it can be run again
*/
finally
{
recorder.stop();
recorder.release();
track.stop();
track.release();
}
}
I was momentarily excited to find the Android API actually already has a NoiseSuppression algorithm (you'll see it commented out above). I tested with it and found NoiseSuppressor wasn't doing much to null out constant tones which leads me to believe it's actually just performing a band-pass filter at non-vocal frequencies.
So, my questions:
1) The above code takes about 250-500ms from mic record through playback in headphones. This latency sucks and it would be great to reduce it. Any suggestions there would be appreciated.
2) Regardless of how tight the latency is, my understanding is that the playback waveform WILL have phase offset from the actual ambient noise waveform. This suggests I need to execute some kind of waveform matching to calculate this offset and compensate. Thoughts on how that gets calculated?
3) When it comes to compensating for latency, what would that look like? I've got an array of shorts coming in every cycle, so what would a 30ms or 250ms latency look like?
I'm aware of fundamental problems with this approach being that the location of the phone being not next to the head is likely to introduce some error, but I'm hopeful with some either dynamic or fixed latency correction it maybe be possible to overcome it.
Thanks for any suggestions.

Even if you were able to do something about the latency, it's a difficult problem as you don't know the distance of the phone from the ear, plus there's the fact that distance is not fixed (as the user will move the phone), plus the fact that you don't have a microphone for each ear (so you can't know what the wave will be at one ear until after it's got there, even if you have zero latency)
Having said that, you might be able to do something that could cancel highly periodic waveforms. All you could do though is allow the user to manually adjust the time delay for each ear - as you have no microphones near the ears themselves, you can have no way in your code to know if you're making the problem better or worse.

AudioFlinger could not create track. status: -12

I am programming for android 2.2 and am trying to using the
SoundPool class to play several sounds simultaneously but at what feel like random times sound will stop coming out of the speakers.
for each sound that would have been played this is printed in the logcat:
AudioFlinger could not create track. status: -12
Error creating AudioTrack
Audio track delete
No exception is thrown and the program continues to execute without any changes except for the lack of volume. I've had a really hard time tracking down what conditions cause the error or recreating it after it happens. I can't find the error in the documentation anywhere and am pretty much at a loss.
Any help would be greatly appreciated!
Edit: I forgot to mention that I am loading mp3 files, not ogg.

i had almost this exact same problem with some sounds i was attempting to load and play recently.
i even broke it down to loading a single mp3 that was causing this error.
one thing i noted: when i loaded with a loop of -1, it would fail with the "status 12" error, but when i loaded it to loop 0 times, it would succeed. even attempting to load 1 time failed.
the final solution was to open the mp3 in an audio editor and re-edit it with slightly lesser quality so that the file is now smaller, and doesn't seem to take up quite as many resources in the system.
finally, there is this discussion that encourages performing a release on the objects you are using, because there is indeed a hard limit on the resources that can be used, and it is system-wide, so if you use several of the resources, other apps will not be able to use them.
https://groups.google.com/forum/#!topic/android-platform/tyITQ09vV3s/discussion%5B1-25%5D
For audio, there's a hard limit of 32 active AudioTrack objects per
device (not per app: you need to share those 32 with rest of the system), and AudioTrack is used internally beneath SoundPool,
ToneGenerator, MediaPlayer, native audio based on OpenSL ES, etc. But
the actual AudioTrack limit is < 32; it depends more on soft factors
such as memory, CPU load, etc. Also note that the limiter in the
Android audio mixer does not currently have dynamic range compression,
so it is possible to clip if you have a large number of active sounds
and they're all loud.
For video players the limit is much much lower due to the intense load
that video puts on the device.
I'll use this as an opportunity to remind media developers: please
remember to call release() for media objects when your app is paused.
This frees up the underlying resources that other apps will need.
Don't rely on the media objects being cleaned up in finalize by the
garbage collector, as that has unpredictable timing.

I had a similar issue where the music tracker within my Android game would drop notes and I got the Audioflinger error (although my status was -22). I got it working however so this might help some people.
The problem occurred when a single sample was being output multiple times simultaneously. So in my case it was a single sample being played on two or more tracks. This seemed to occasionally deadlock or something and one of the two notes would be dropped. The solution was to have two copies of the sample (two actual ogg files - identical but both in the assets). Then on each track even although I was playing the same sample, it was coming from a different file. This totally fixed the issue for me.
Not sure why it works as I cache the samples into memory, but even loading the same file into two different sounds didn't fix it. Only when the samples came out of two different files did the errors go away.
I'm sure this won't help everyone and it's not the prettiest fix but it might help someone.

john.k.doe is right. You must reduce the size of your mp3 file. You should keep the size under 100kb per file. I had to reduce my 200kb file to 72kb using a constante bit rate(CBR) of 32kbps instead of the usual 128kbps. That worked for me!

Try
final ToneGenerator tg = new ToneGenerator(AudioManager.STREAM_NOTIFICATION, 50);
tg.startTone(ToneGenerator.TONE_PROP_BEEP, 200);
tg.release();
Releasing should keep your resources.

I was with this problem. In order to solve it i run the method .release() of SoundPool object after finish playing the sound.
Here's my code:
SoundPool pool = new SoundPool(10, AudioManager.STREAM_MUSIC, 50);
final int teste = pool.load(this.ctx,this.soundS,1);
pool.setOnLoadCompleteListener(new OnLoadCompleteListener(){
#Override
public void onLoadComplete(SoundPool sound,int sampleId,int status){
pool.play(teste, 20,20, 1, 0, 1);
new Thread(new Runnable(){
#Override
public void run(){
try {
Thread.sleep(2000);
pool.release();
} catch (InterruptedException e) { e.printStackTrace(); }
}
}).start();
}
});
Note that in my case my sounds had length 1-2 seconds max, so i put the value of 2000 miliseconds in Thread.sleep(), in order to only release the resources after the player have had finished.

Like said above, there is a problem with looping: when I set repeat to -1 I get this error, but with 0 everything is working properly.
I've noticed that some sounds give this error when I'm trying to play them one by one. For example:
mSoundPool.stop(mStreamID);
mStreamID = mSoundPool.play(mRandID, mVolume, mVolume, 1, -1, 1f);
In such case, first track is played ok, but when I switch sounds, next track gives this error. It seems that using looping, a buffer is somehow overloaded, and mSoundPool.stop cannot release resources immediately.
Solution:
final Handler handler = new Handler();
handler.postDelayed(new Runnable() {
#Override
public void run() {
mStreamID = mSoundPool.play(mRandID, mVolume, mVolume, 1, -1, 1f);
}, 350);
And it's working, but delay is different for different devices.

In my case, reducing the quality and thereby the file sizes of the MP3's to under 100kb wasn't sufficient, as some 51kb files worked while some longer duration 41kb files still did not.
What helped us was reducing the sample rate from 44100 to 22050 or shortening the duration to less than 5 seconds.

I see too many overcomplicated answer. Error -12 means that you did not release the variables.
I had the same problem after I played an OGG audio file 8 times.
This worked for me:
SoundPoolPlayer onBeep; //Global variable
if(onBeep!=null){
onBeep.release();
}
onBeep = SoundPoolPlayer.create(getContext(), R.raw.micon);
onBeep.setOnCompletionListener(
new MediaPlayer.OnCompletionListener() {
#Override
public void onCompletion(MediaPlayer mp) { //mp will be null here
loge("ON Beep! END");
startGoogleASR_API_inner();
}
}
);
onBeep.play();
Releasing the variable right after .play() would mess things up, and it is not possible to release the variable inside onCompletion, so notice how I release the variable before using it(and checking for null to avoid nullpointer exceptions).
It works like charm!

A single soundPool has an internal memory limitation of 1 (one) Mb. You might be hitting this if your sound is very high quality. If you have many sounds and are hitting this limit, just create more soundpools, and distribute your sounds across them.
You may not even be able to reach the hard track limit if you are running out of memory before you get there.
That error not only appears when the stream or track limit has been reached, but also the memory limit. Soundpool will stop playing old and/or de-prioritized sounds in order to play a new sound.

AudioTrack restarting even after it is stopped

I created a simple application that generates a square wave of given frequency and plays it using AudioTrack in STREAM mode (STREAM_MUSIC). Everything seems to be working fine and the sound plays okay, however when the stream is finished I get messages in the log:
W/AudioTrack( 7579): obtainBuffer() track 0x14c228 disabled, restarting ...
Even after calling the stop() function I still get these.
I believe I properly set the AudioTrack buffer size, based on minimal size required by AudioTrack (in my case 6x1024). I feed it with smaller buffers of 1024 shorts.
Is it okay that I'm getting these and should I leave it like that?

Ok, I think the problem is solved. The error is generated when the buffer is not completely filled with data on time (buffer underrun) . I have no idea what the timeout is but if you experience this make sure that:
You don't call the play method until you have some data in the buffer.
You can generate the data fast enough to beat the timeout.
After you are finished feeding the buffer with data, before you call stop() method, make sure that the "last" buffer was completely filled with data before timeout.
I dealt with the last issue by always waiting a little (until timeout) then sending 1 buffer full of zeroes and finally calling the stop() function.
Keep in mind that you must always send the buffer in smaller chunks, even if you have the big chunk ready. It still bothers me a bit that I'm not 100% sure if that is the right way but the errors are gone so I guess I can live with that :)

I've found that even when the buffer is technically long enough, and filled with bytes, if they aren't properly formatted (audio shorts converted to a byte array) it will still throw you that error.

I was getting that warning when I instantiated the Audiotrack, called audioTrack.play() and there was a slight delay between the play() call and the audioTrack.write(). If I called play() right before write() the warning disappeared.

I've solved by this
if (mAudioTrack.getPlayState()!=AudioTrack.PLAYSTATE_PLAYING)
mAudioTrack.play();
mAudioTrack.write(b, 0, sz * 2);
mAudioTrack.stop();
mAudioTrack.flush();