In the application which I want to create, I face some technical obstacles. I have two music tracks in the application. For example, a user imports the music background as a first track. The second path is a voice recorded by the user to the rhythm of the first track played by the speaker device (or headphones). At this moment we face latency. After recording and playing back in the app, the user hears the loss of synchronisation between tracks, which occurs because of the microphone and speaker latencies.
Firstly, I try to detect the delay by filtering the input sound. I use android’s AudioRecord class, and the method read(). This method fills my short array with audio data.
I found that the initial values of this array are zeros so I decided to cut them out before I will start to write them into the output stream.
So I consider those zeros as a „warmup” latency of the microphone. Is this approach correct? This operation gives some results, but it doesn’t resolve the problem, and at this stage, I’m far away from that.
But the worse case is with the delay between starting the speakers and playing the music. This delay I cannot filter or detect. I tried to create some calibration feature which counts the delay. I play a „beep” sound through the speakers, and when I start to play it, I also begin to measure time. Then, I start recording and listen for this sound being detected by the microphone. When I recognise this sound in the app, I stop measuring time. I repeat this process several times, and the final value is the average from those results. That is how I try to measure the latency of the device. Now, when I have this value, I can simply shift the second track backwards to achieve synchronisation of both records (I will lose some initial milliseconds of the recording, but I skip this case, for now, there are some possibilities to fix it).
I thought that this approach would resolve the problem, but it turned out this is not as simple as I thought. I found two issues here:
1. Delay while playing two tracks simultaneously
2. Random in device audio latency.
The first: I play two tracks using AudioTrack class and I run method play() like this:
val firstTrack = //creating a track
val secondTrack = //creating a track
firstTrack.play()
secondTrack.play()
This code causes delays at the stage of playing tracks. Now, I don’t even have to think about latency while recording; I cannot play two tracks simultaneously without delays. I tested this with some external audio file (not recorded in my app) - I’m starting the same audio file using the code above, and I can see a delay. I also tried it with MediaPlayer class, and I have the same results. In this case, I even try to play tracks when callback OnPreparedListener invoke:
val firstTrack = //AudioPlayer
val secondTrack = //AudioPlayer
second.setOnPreparedListener {
first.start()
second.start()
}
And it doesn’t help.
I know that there is one more class provided by Android called SoundPool. According to the documentation, it can be better with playing tracks simultaneously, but I can’t use it because it supports only small audio files and that can't limit me.
How can I resolve this problem? How can I start playing two tracks precisely at the same time?
The second: Audio latency is not deterministic - sometimes it is smaller, and sometimes it’s huge, and it’s out of my hands. So measuring device latency can help but again - it cannot resolve the problem.
To sum up: is there any solution, which can give me exact latency per device (or app session?) or other triggers which detect actual delay, to provide the best synchronisation while playback two tracks at the same time?
Thank you in advance!
Synchronising audio for karaoke apps is tough. The main issue you seem to be facing is variable latency in the output stream.
This is almost certainly caused by "warm up" latency: the time it takes from hitting "play" on your backing track to the first frame of audio data being rendered by the audio device (e.g. headphones). This can have large variance and is difficult to measure.
The first (and easiest) thing to try is to use MODE_STREAM when constructing your AudioTrack and prime it with bufferSizeInBytes of data prior to calling play (more here). This should result in lower, more consistent "warm up" latency.
A better way is to use the Android NDK to have a continuously running audio stream which is just outputting silence until the moment you hit play, then start sending audio frames immediately. The only latency you have here is the continuous output latency.
If you decide to go down this route I recommend taking a look at the Oboe library (full disclosure: I am one of the authors).
To answer one of your specific questions...
Is there a way to calculate the latency of the audio output stream programatically?
Yes. The easiest way to explain this is with a code sample (this is C++ for the AAudio API but the principle is the same using Java AudioTrack):
// Get the index and time that a known audio frame was presented for playing
int64_t existingFrameIndex;
int64_t existingFramePresentationTime;
AAudioStream_getTimestamp(stream, CLOCK_MONOTONIC, &existingFrameIndex, &existingFramePresentationTime);
// Get the write index for the next audio frame
int64_t writeIndex = AAudioStream_getFramesWritten(stream);
// Calculate the number of frames between our known frame and the write index
int64_t frameIndexDelta = writeIndex - existingFrameIndex;
// Calculate the time which the next frame will be presented
int64_t frameTimeDelta = (frameIndexDelta * NANOS_PER_SECOND) / sampleRate_;
int64_t nextFramePresentationTime = existingFramePresentationTime + frameTimeDelta;
// Assume that the next frame will be written into the stream at the current time
int64_t nextFrameWriteTime = get_time_nanoseconds(CLOCK_MONOTONIC);
// Calculate the latency
*latencyMillis = (double) (nextFramePresentationTime - nextFrameWriteTime) / NANOS_PER_MILLISECOND;
A caveat: This method relies on accurate timestamps being reported by the audio hardware. I know this works on Google Pixel devices but have heard reports that it isn't so accurate on other devices so YMMV.
Following the answer of donturner, here's a Java version (that also uses other methods depending on the SDK version)
/** The audio latency has not been estimated yet */
private static long AUDIO_LATENCY_NOT_ESTIMATED = Long.MIN_VALUE+1;
/** The audio latency default value if we cannot estimate it */
private static long DEFAULT_AUDIO_LATENCY = 100L * 1000L * 1000L; // 100ms
/**
* Estimate the audio latency
*
* Not accurate at all, depends on SDK version, etc. But that's the best
* we can do.
*/
private static void estimateAudioLatency(AudioTrack track, long audioFramesWritten) {
long estimatedAudioLatency = AUDIO_LATENCY_NOT_ESTIMATED;
// First method. SDK >= 19.
if (Build.VERSION.SDK_INT >= 19 && track != null) {
AudioTimestamp audioTimestamp = new AudioTimestamp();
if (track.getTimestamp(audioTimestamp)) {
// Calculate the number of frames between our known frame and the write index
long frameIndexDelta = audioFramesWritten - audioTimestamp.framePosition;
// Calculate the time which the next frame will be presented
long frameTimeDelta = _framesToNanoSeconds(frameIndexDelta);
long nextFramePresentationTime = audioTimestamp.nanoTime + frameTimeDelta;
// Assume that the next frame will be written at the current time
long nextFrameWriteTime = System.nanoTime();
// Calculate the latency
estimatedAudioLatency = nextFramePresentationTime - nextFrameWriteTime;
}
}
// Second method. SDK >= 18.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED && Build.VERSION.SDK_INT >= 18) {
Method getLatencyMethod;
try {
getLatencyMethod = AudioTrack.class.getMethod("getLatency", (Class<?>[]) null);
estimatedAudioLatency = (Integer) getLatencyMethod.invoke(track, (Object[]) null) * 1000000L;
} catch (Exception ignored) {}
}
// If no method has successfully gave us a value, let's try a third method
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
AudioManager audioManager = (AudioManager) CRT.getInstance().getSystemService(Context.AUDIO_SERVICE);
try {
Method getOutputLatencyMethod = audioManager.getClass().getMethod("getOutputLatency", int.class);
estimatedAudioLatency = (Integer) getOutputLatencyMethod.invoke(audioManager, AudioManager.STREAM_MUSIC) * 1000000L;
} catch (Exception ignored) {}
}
// No method gave us a value. Let's use a default value. Better than nothing.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
estimatedAudioLatency = DEFAULT_AUDIO_LATENCY;
}
return estimatedAudioLatency
}
private static long _framesToNanoSeconds(long frames) {
return frames * 1000000000L / SAMPLE_RATE;
}
The android MediaPlayer class is notoriously slow to begin audio playback, I experienced an issue in an app I was creating where there was a greater than one second delay to begin playing an audio clip. I resolved it by switching to ExoPlayer which resulted in the playback starting within 100ms. I've also read that ffmpeg has even faster start audio startup time than ExoPlayer but I haven't used it so I can't make any promises.
Related
I'm using Exoplayer to play a video in my application, what I want to do is change the speed of playback, for which Exoplayer provides a straightforward solution:
val playbackParameters = PlaybackParameters(whateverSpeedFloat)
exoPlayer.setPlaybackParameters(playbackParameters)
Now this works, but the problem I have is that the effect is not immediate, when you change the speed it takes a few frames for the actual speed to change. I guess it's because some of the frames are preloaded or buffered and the set playback parameters only affect the frames after this.
If I stop the video, and change speed from say 0.5x to 2x, then press play, it's very obvious that there is a delay in playback speed change. But, if I press stop, change the speed from 0.5x to 2x AND seek a different point in the video, and press play, it works great, there is no delay. I guess it reloads/buffers the new frames with the right playback parameters. I tried doing
exoPlayer.clearVideoDecoderOutputBufferRenderer()
after changing speeds to try and rebuffer the frames after setting the playback parameters but it doesn't seem to change anything.
Any ideas on how to fix this? Or other video player libraries that wouldn't have this problem?
UPDATE:
I still haven't found a solution, I took the problem to ExoPlayer and an issue was raised and "fixed"(https://github.com/google/ExoPlayer/issues/7982), however I'm still having the same problem, so atm I'll just get back to them and wait.
However, what was mentioned is that the delay is a know issue, and that right now there is no solution,
Correct. We looked at options to address the delay a while ago but couldn't find a clean/general/easy to implement approach (there is no API we can use to process the audio just before the mixer, so we have to do it upstream of the audio track buffer, which introduces latency).
Instead, they suggested initialising Exoplayer with a DefaultRenderersFactory with AudioTrackPlaybackParams set to true:
val defaultRenderersFactory =
DefaultRenderersFactory(this).setEnableAudioTrackPlaybackParams(true)
exoPlayer = SimpleExoPlayer.Builder(this, defaultRenderersFactory).build()
And this does in fact get rid of the delay (not 100% but I'd say around 80% which is good enough), but then the video speed gets all clunky and starts freezing and changing speeds every time it is paused/played or seeked to a different point.
I also tried modifying the buffering configuration #GensaGames suggested but even though I tested different configurations for a while, I never saw any change in the behaviour so discarded the solution and went to the exoPlayer repo.
I'll update this question when I finally have a working video speed changer.
I think you can decrease time of the buffering within configuration setup during ExpPlayer initializing. Below ex. on configuration, you can go throw documentation and check possible values.
/* Instantiate a DefaultLoadControl.Builder. */
DefaultLoadControl.Builder builder = new
DefaultLoadControl.Builder();
/* Maximum amount of media data to buffer (in milliseconds). */
final long loadControlMaxBufferMs = 60000;
/*Configure the DefaultLoadControl to use our setting for how many
Milliseconds of media data to buffer. */
builder.setBufferDurationsMs(
DefaultLoadcontrol.DEFAULT MIN BUFFER MS,
loadControlMaxBufferMs,
/* To reduce the startup time, also change the line below */
DefaultLoadControl.DEFAULT_BUFFER_FOR_PLAYBACK_MS,
DefaultLoadControl.DEFAULT_BUFFER_FOR_PLAYBACK_AFTER_REBUFFER_MS);
/* Build the actual DefaultLoadControl instance */
DefaultLoadControl loadControl = builder.createDefaultLoadControl();
/* Instantiate ExoPlayer with our configured DefaultLoadControl */
ExoPlayer player = ExoPlayerFactory.newSimpleInstance(
new DefaultRenderersFactory(this),
new DefaultTrackSelector(),
loadControl);
There is also good article about changing buffer time options.
I am transcoding videos based on the example given by Google (https://android.googlesource.com/platform/cts/+/master/tests/tests/media/src/android/media/cts/ExtractDecodeEditEncodeMuxTest.java)
Basically, transocding of MP4 files works, but on some phones I get some weird results. If for example I transcode a video with audio on an HTC One, the code won't give any errors but the file cannot play afterward on the phone. If I have a 10 seconds video it jumps to almost the last second and you only here some crackling noise. If you play the video with VLC the audio track is completely muted.
I did not alter the code in terms of encoding/decoding and the same code gives correct results on a Nexus 5 or MotoX for example.
Anybody having an idea why it might fail on that specific device?
Best regard and thank you,
Florian
I made it work in Android 4.4.2 devices by following changes:
Set AAC profile to AACObjectLC instead of AACObjectHE
private static final int OUTPUT_AUDIO_AAC_PROFILE = MediaCodecInfo.CodecProfileLevel.AACObjectLC;
During creation of output audio format, use sample rate and channel count of input format instead of fixed values
MediaFormat outputAudioFormat = MediaFormat.createAudioFormat(OUTPUT_AUDIO_MIME_TYPE,
inputFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE),
inputFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT));
Put a check just before audio muxing audio track to control presentation timestamps. (To avoid timestampUs X < lastTimestampUs X for Audio track error)
if (audioPresentationTimeUsLast == 0) { // Defined in the begining of method
audioPresentationTimeUsLast = audioEncoderOutputBufferInfo.presentationTimeUs;
} else {
if (audioPresentationTimeUsLast > audioEncoderOutputBufferInfo.presentationTimeUs) {
audioEncoderOutputBufferInfo.presentationTimeUs = audioPresentationTimeUsLast + 1;
}
audioPresentationTimeUsLast = audioEncoderOutputBufferInfo.presentationTimeUs;
}
// Write data
if (audioEncoderOutputBufferInfo.size != 0) {
muxer.writeSampleData(outputAudioTrack, encoderOutputBuffer, audioEncoderOutputBufferInfo);
}
Hope this helps...
If original CTS tests fail you need to go to device vendors and ask for fixes
I'm developing a game in Android and I came across a very annoying, hard-to-find bug. The issue is that when you are using SoundPool to play your sounds, you can actually loop whatever sound you are playing. In this case, the issue is the "running steps" sound; this sound gets executed quite fast and continually (around every 400ms) when the main character is running.
Now when playing the sound on a regular (not so powerful) device e.g. Samsung SII, the sound is played every 500ms - however, if I run the very same code on another device (let's say, Samsung SIV, Samsung SIII), the sound plays twice or even three times faster.
It seems like the more powerful the device hardware specs are, the faster it plays. On some devices, it plays so fast that you almost hear one solid continuous sound. I've been looking for techniques to set a specific ratio on the time period between sound plays, but it doesn't work properly and the issue remains. Does anyone know how to fix it, either using SoundPool, MediaPlayer, or any other sound-controlling API on Android?
You could use an AudioTrack to play a continuous stream of PCM data, since you would pass a stream you could be sure about the interval between sounds. the downside could be a little delay when first starting the sound but it depends on the minimum buffer size, and it depends, I think, on android version and device. On my galaxy s2 android 4.1 it was about 20ms.if you think this could be an option I can post some code
The problem with just looping or using a regular interval for something like footsteps is that you have a possible decoupling of sound and visuals. If your sound gets delays or sped up, or your visuals get delayed or sped up, you would have to adjust for that delay dynamically and automatically. You already have that issue right here
A better solution would be to place a trigger on the exact event which should trigger the sound (in this case, the foot being placed down), which then plays the sound. This also means that if you have multiple sources of the sound (like multiple footsteps), you don't have to manually start the sound with the right interval.
I can't seem to replicate the issue on Galaxy Nexus and Nexus S, does that mean I fixed it? Or maybe you could show what you're doing differently from this:
SoundPool soundPool = new SoundPool(4, AudioManager.STREAM_MUSIC, 100);
Integer sound1 = soundPool.load(this, R.raw.file1, 1);
Integer sound2 = soundPool.load(this, R.raw.file2, 1);
playSound(sound1);
public void playSound(int sound) {
AudioManager mgr = (AudioManager)getSystemService(Context.AUDIO_SERVICE);
float volume = mgr.getStreamVolume(AudioManager.STREAM_MUSIC)
/ mgr.getStreamMaxVolume(AudioManager.STREAM_MUSIC);
soundPool.play(sound, volume, volume, 1, -1, 1.0f);
}
If the problem is that you want to control the interval between the discrete sounds, The easiest way to do this is with a handler.
Basically you start a sound playing which is an asynchronous process. Then you use a handler to schedule a message to play the next sound sometime in the future. It will take some trial and error to get it right, but you will be guaranteed that the sound will start at the same interval after the previous sound on every device.
Here is some code to illustrate what I am talking about.
Here is a handler implementation you could use:
handler = new Handler() {
/* (non-Javadoc)
* #see android.os.Handler#handleMessage(android.os.Message)
*/
#Override
public void handleMessage(Message msg) {
if (msg.what == NEXT_ITEM_MSG) {
playNextSound();
}
else if (msg.what == SEQUENCE_COMPLETE_MSG) {
// notify a listener
listener.onSoundComplete()
}
}
};
Then you could write playNextSound like this:
private void playNextSound() {
if (mRunning) {
// Get the first item
SoundSequenceItem item = currentSequence.getNextSequenceItem();
if (item == null) {
Message msg = handler.obtainMessage(SEQUENCE_COMPLETE_MSG);
handler.sendMessage(msg);
return;
}
// Play the sound
int iSoundResId = item.getSoundResId();
if (iSoundResId != -1) {
player.playSoundNow(soundResId);
}
// schedule a message to advance to next item after duration
Message msg = handler.obtainMessage(NEXT_ITEM_MSG);
handler.sendMessageDelayed(msg, item.getDuration());
}
}
and your SoundSequenceItem could just be a simple class that has a sound file resource id and a duration. If you want to keep playing the sound while the character is moving you could do something like this:
public void onSoundComplete() {
if (character.isRunning()) {
currentSequence.addSequenceItem(new SoundSequenceItem(R.id.footsteps,500);
playNextSound();
}
}
Or you could modify playNextSound to continually play the same sound. Mine is written this way to be able to play different sounds in sequence.
I have had a lot of problems developing apps which used sounds and stuff like that. I would not suggest you to use SoundPool since it is bug-affected, and also be aware that looping sounds with SoundPool won't work on devices which are 4.3 and higher, see this open issue, at AOSP - Issue tracker.
I think that the solution is to go native and use OpenSL ES o similar libraries.
I have created a metronome-type application with a specified swing interval of 750 milliseconds on the pendulum and playing a single audio file at the maximum swing arc... repeating the swinging of the pendulum and playing of the sound indefinitely. However, I am finding that the actual timing of execution of the code varies dramatically from device-to-device and even performs with variance on a single device. My intent is to swing the pendulum at the rate of 80 beats per minute and play the audio file with each "beat". I adjusted the 750 millisecond setting to accommodate the time required to play the audio file. This slightly reduced the millisecond setting from 750 down to about 680. I tested using various devices and found that the results of a one minute run of the metronome performed dramatically differently for timing as I tested with various Android devices even though I am defining my timing elements based on milliseconds.
I am using Android SoundPool to access a .wav file to play the sound.
I found quite a few references to Soundpool timing issues and concerns but have not yet found a viable and reliable solution to deliver consistent timing for an application like this.
It seems that the swing of the pendulum is pretty consistent based on the specified delay so I believe the variation is due to variable timing during execution of the SoundPool code playing the audio. Is there a reliable way to execute code to play sounds on a consistent and "exact" timing interval with Android?
One way to do this is a handler. This allows you to start the audio clip at exactly the same time regardless of how long the clip actually takes to play. You don't need SoundPool for this, just SoundPlayer.
A handler allows you to schedule a message to be delivered to your code some time in the future. Since playing a sound with SoundPlayer is asynchronous, you can use this simple mechanism to play a sound on a regular interval.
Here is some code to show how it might work.
handler = new Handler() {
/* (non-Javadoc)
* #see android.os.Handler#handleMessage(android.os.Message)
*/
#Override
public void handleMessage(Message msg) {
if (msg.what == NEXT_SOUND_MSG) {
playNextSound()
}
}
};
// Set up media player for sounds
player = new SoundPlayer(context);
player.start();
private void playNextSound() {
if (mRunning) {
// Play the sound
int iSoundResId = item.getSoundResId();
if (iSoundResId != -1) {
playSound(iSoundResId);
}
// schedule a message to advance to next item after duration
Message msg = handler.obtainMessage(NEXT_SOUND_MSG);
handler.sendMessageDelayed(msg, interval);
}
}
When using MediaPlayer, I noticed that whenever my phone stucks, the MediaPlayer glitches and then continues playing from the position in the audio it glitched.
This is bad for my implementation since I want the audio to be played at a specific time.
If I have a song of 1000 millisecond length, I want is the ability to set MediaPlayer to start playing at some specific time t, and then exactly stop at at time t+1000.
This means that I actually need two things:
1) Start MediaPlayer at a specific time with a very small delay.
2) Making MediaPlayer glitches ignore the audio they glitched on and continue playing in order to finish the song on time.
The delay of the functions is very important to me and I need the audio to be played exactly(~) at the time it was supposed to be played.
Thanks!
You will need to use possibly mp.getDuration(); and/or mp.getCurrentPosition(); although it's impossible to know exactly what you mean by "I need the audio to be played exactly(~) at the time it was supposed to be played."
Something like this should get you started:
int a = (mp.getCurrentPosition() + b);
Thanks for the answer Mike. but unfortunately this won't help me. Let's say that I asked MediaPlayer to start playing a song of length 3:45 at 00:00. At 01:00 I started using the phone's resources, due to the heavy usage my phone glitched making MediaPlayer pause for 2 seconds.
Time:
00:00-01:00 - I heard the audio at 00:00-01:00
01:00-01:02 - I heard silence because the phone glitched
01:02-03:47 - I heard the audio at 01:00-03:45 with 2 second time skew
Now from what I understood MediaPlayer is a bad choice of usage on this problem domain, since MediaPlayer provides a high level API.I am currently experimenting with the
AudioTrack class which should provide me with what I need:
//Creating a new audio track
AudioTrack audioTrack = new AudioTrack(...)
//Get start time
long start = System.currentTimeMillis();
// loop until finished
for (...) {
// Get time in song
long now = System.currentTimeMillis();
long nowInSong = now - start;
// get a buffer from the song at time nowInSong with a length of 1 second
byte[] b = getAudioBuffer(nowInSong);
// play 1 second of music
audioTrack.write(b, 0, b.length);
// remove any unplayed data
audioTrack.flush();
}
Now if I glitch I only glitch for 1 second and then I correct myself by playing the right audio at the right time!
NOTE
I haven't tested this code but it seems like the right way to do it. If it will actually work I will update this post again.
P.S. seeking in MediaPlayer is:
1. A heavy operation that will surely delay my music (every millisecond counts here)
2. Is not thread safe and cannot be used from multiple threads (seeks, starts etc...)