I have an app that uses OpenSL ES. When I try to use it on a Nexus9 6.0.1, I hear a noise like I have the wrong sampling rate. On other devices all is OK.
My SLDataFormat_PCM structure:
SLDataFormat_PCM format_pcm = {
SL_DATAFORMAT_PCM,
aChannels,
48000 * 1000,
SL_PCMSAMPLEFORMAT_FIXED_16,
SL_PCMSAMPLEFORMAT_FIXED_16,
aChannels == 2 ? SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT
: SL_SPEAKER_FRONT_CENTER,
SL_BYTEORDER_LITTLEENDIAN
};
When I change the sample rate (+/- 1Hz) in this structure, the output sounds OK, but I receive an AudioTrack debug message:
W/AudioTrack: AUDIO_OUTPUT_FLAG_FAST denied by client; transfer 1, track 47999 Hz, output 48000 Hz
Why do I have a problem in FAST mode, if the Nexus9 has 48000Hz?
I checked it using this method:
jclass clazz = env.getEnv()->FindClass("android/media/AudioSystem");
jmethodID mid = env.getEnv()->GetStaticMethodID(clazz, "getPrimaryOutputSamplingRate", "()I");
int nSampleRate = env.getEnv()->CallStaticIntMethod(clazz, mid);
LOGDEBUG << "Sample Rate: " << nSampleRate;
[ DBG:c894860f] 11:16:14.902: Sample Rate: 48000
Is there a better method to get device's sample rate?
Yes there's a method to find the preferred sample rate for a device though it'll work for API level > 16. You can have a look at my answer here.
And about your SLDataFormat_PCM structure. You've initialized with sample rate 48k*1k! If you want to sample your PCM data in 48k, try using the code below.
// configure audio source
SLDataFormat_PCM format_pcm = {
SL_DATAFORMAT_PCM,
aChannels,
SL_SAMPLINGRATE_48,
SL_PCMSAMPLEFORMAT_FIXED_16,
SL_PCMSAMPLEFORMAT_FIXED_16,
aChannels == 2 ? SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT
: SL_SPEAKER_FRONT_CENTER,
SL_BYTEORDER_LITTLEENDIAN
};
I didn't work with Nexus 9 before, so I don't know if it supports 48k sampling rate. But, anyway, you can check if it supports.
The problem was with mutex in callback function.
UPD:
OpenSLES Readme
Known issues
At 48000Hz, Galaxy Nexus and Nexus 10 produce glitchy output. At
44100Hz, Galaxy Nexus tends to glitch when switching activities or
bringing up large dialogs. Touch sounds occasionally cause OpenSL to
glitch. It's probably a good idea to disable touch sounds in audio
apps. These problems are not specific to opensl_stream and have been
reproduced in other settings.
Related
I'm using Oboe 1.2 in an audio android application. When I call getFramesPerBurst(), which gives the endpoint buffer size, I get expected results (240 frames) if the number of output channels is set to 2. However when I set 4 output channels, the value returned by getFramesPerBurst() is around 960 (!). Is that normal ? Is that a limitation of the hardware (I tested on 4 different devices though, with different os version) ? A limitation of Oboe ? I notice also that this value is different than the value given by the property PROPERTY_OUTPUT_FRAMES_PER_BUFFER of AudioManager from the AudioService.
oboe::AudioStreamBuilder builder;
if (!oboe::AudioStreamBuilder::isAAudioRecommended()){
builder.setAudioApi(oboe::AudioApi::OpenSLES);
}
builder.setSharingMode(oboe::SharingMode::Exclusive);
builder.setFormat(oboe::AudioFormat::Float);
builder.setChannelCount(4);
builder.setCallback(&_oboeCallback);
builder.setPerformanceMode(oboe::PerformanceMode::LowLatency);
oboe::Result result = builder.openStream(&_stream);
if (result == oboe::Result::OK) {
int framePerBurst = _stream->getFramesPerBurst(); // gives value around 960 for 4 channels, 240 for 2 channels
_stream->setBufferSizeInFrames(2*framePerBurst);
}
Unless you are connecting to an audio device which actually has 4 independent channels (e.g. a USB audio interface or DJ controller like this one) then your 4 channel stream will need to be mixed into an N channel stream where N is the number of channels in your audio device. This could be 2 (stereo) for headphones or 1 (mono) for a built-in speaker.
The mixer introduces latency and larger buffer sizes. This is the difference in buffer sizes you see when you request a channel count of 2 vs 4.
For the lowest latency always leave the channel count unspecified when creating the stream, then do any channel count conversion inside your own app. There's an example of this here.
I'm working on a DSP project on Android which requires low latency audio I/O. For this reason, I'm using Oboe library. In the LiveEffect example, the synchronous recording and playback is demonstrated. However, for acoustic feedback neutralization, I need the other way around, that is to generate White Noise signal through a built-in speaker first, then record it using a mic. I tried to modify LiveEffect example using this asked question, i.e setting the recording stream as Master (callback) and using non-blocking write method for the playback stream. But I got the following error when I run my code on Pixel XL (Android 9.0):
D/AudioStreamInternalCapture_Client: processDataNow() wait for valid timestamps
D/AudioStreamInternalCapture_Client: advanceClientToMatchServerPosition() readN = 0, writeN = 384, offset = -384
--------- beginning of crash
A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x5800003f666c66 in tid 2852 (AAudio_1), pid 2796 (ac.oiinitialize)
Here is my callback:
oboe::DataCallbackResult
AudioEngine::onAudioReady(oboe::AudioStream *oboeStream, void *audioData, int32_t numFrames) {
assert(oboeStream == mRecordingStream);
int32_t framesToWrite = mPlayStream->getFramesPerBurst();
oscillator_->whiteNoise(framesToWrite); // write white noise into buffer;
oboe::ResultWithValue<int32_t> result = mPlayStream->write(oscillator_->write(), framesToWrite, 0);
// oscillator_->write() returns const void* buffer;
if (result != oboe::Result::OK) {
LOGE("input stream read error: %s", oboe::convertToText(result.error()));
return oboe::DataCallbackResult ::Stop;
}
// add Adaptive Feedback Neutralization Algorithm here....
return oboe::DataCallbackResult::Continue;
}
Is my approach correct for generating a signal and then capturing it through a mic? If so, can anyone help me with this error? Thank you in advance.
However, for acoustic feedback neutralization, I need the other way around, that is to generate White Noise signal through a built-in speaker first, then record it using a mic
You can still do this using an output stream callback and a non-blocking read on the input stream. This is the more common (and tested) way of doing synchronous I/O. A Larsen effect will work fine this way.
Your approach should still work, however, I'd stick to the LiveEffect way of setting up the streams since it works.
In terms of your error SIGSEGV usually means a null pointer dereference - are you starting your input stream before the output stream? This could meant you're attempting to write to the output stream which hasn't yet been opened.
I'm using the Android oboe library for high performance audio in a music game.
In the assets folder I have 2 .raw files (both 48000Hz 16 bit PCM wavs and about 60kB)
std_kit_sn.raw
std_kit_ht.raw
These are loaded into memory as SoundRecordings and added to a Mixer. kSampleRateHz is 48000:
stdSN= SoundRecording::loadFromAssets(mAssetManager, "std_kit_sn.raw");
stdHT= SoundRecording::loadFromAssets(mAssetManager, "std_kit_ht.raw");
mMixer.addTrack(stdSN);
mMixer.addTrack(stdFT);
// Create a builder
AudioStreamBuilder builder;
builder.setFormat(AudioFormat::I16);
builder.setChannelCount(1);
builder.setSampleRate(kSampleRateHz);
builder.setCallback(this);
builder.setPerformanceMode(PerformanceMode::LowLatency);
builder.setSharingMode(SharingMode::Exclusive);
LOGD("After creating a builder");
// Open stream
Result result = builder.openStream(&mAudioStream);
if (result != Result::OK){
LOGE("Failed to open stream. Error: %s", convertToText(result));
}
LOGD("After openstream");
// Reduce stream latency by setting the buffer size to a multiple of the burst size
mAudioStream->setBufferSizeInFrames(mAudioStream->getFramesPerBurst() * 2);
// Start the stream
result = mAudioStream->requestStart();
if (result != Result::OK){
LOGE("Failed to start stream. Error: %s", convertToText(result));
}
LOGD("After starting stream");
They are called appropriately to play with standard code (as per Google tutorials) at required times:
stdSN->setPlaying(true);
stdHT->setPlaying(true); //Nasty Sound
The audio callback is standard (as per Google tutorials):
DataCallbackResult SoundFunctions::onAudioReady(AudioStream *mAudioStream, void *audioData, int32_t numFrames) {
// Play the stream
mMixer.renderAudio(static_cast<int16_t*>(audioData), numFrames);
return DataCallbackResult::Continue;
}
The std_kit_sn.raw plays fine. But std_kit_ht.raw has a nasty distortion. Both play with low latency. Why is one playing fine and the other has a nasty distortion?
I loaded your sample project and I believe the distortion you hear is caused by clipping/wraparound during mixing of sounds.
The Mixer object from the sample is a summing mixer. It just adds the values of each track together and outputs the sum.
You need to add some code to reduce the volume of each track to avoid exceeding the limits of an int16_t (although you're welcome to file a bug on the oboe project and I'll try to add this in an upcoming version). If you exceed this limit you'll get wraparound which is causing the distortion.
Additionally, your app is hardcoded to run at 22050 frames/sec. This will result in sub-optimal latency across most mobile devices because the stream is forced to upsample to the audio device's native frame rate. A better approach would be to leave the sample rate undefined when opening the stream - this will give you the optimal frame rate for the current audio device - then use a resampler on your source files to supply audio at this frame rate.
I'm developing a VoIP application that runs at the sampling rate of 48 kHz. Since it uses Opus, which uses 48 kHz internally, as its codec, and most current Android hardware natively runs at 48 kHz, AEC is the only piece of the puzzle I'm missing now. I've already found the WebRTC implementation but I can't seem to figure out how to make it work. It looks like it corrupts the memory randomly and crashes the whole thing sooner or later. When it doesn't crash, the sound is kinda chunky as if it's quieter for the half of the frame. Here's my code that processes a 20 ms frame:
webrtc::SplittingFilter* splittingFilter;
webrtc::IFChannelBuffer* bufferIn;
webrtc::IFChannelBuffer* bufferOut;
webrtc::IFChannelBuffer* bufferOut2;
// ...
splittingFilter=new webrtc::SplittingFilter(1, 3, 960);
bufferIn=new webrtc::IFChannelBuffer(960, 1, 1);
bufferOut=new webrtc::IFChannelBuffer(960, 1, 3);
bufferOut2=new webrtc::IFChannelBuffer(960, 1, 3);
// ...
int16_t* samples=(int16_t*)data;
float* fsamples[3];
float* foutput[3];
int i;
float* fbuf=bufferIn->fbuf()->bands(0)[0];
// convert the data from 16-bit PCM into float
for(i=0;i<960;i++){
fbuf[i]=samples[i]/(float)32767;
}
// split it into three "bands" that the AEC needs and for some reason can't do itself
splittingFilter->Analysis(bufferIn, bufferOut);
// split the frame into 6 consecutive 160-sample blocks and perform AEC on them
for(i=0;i<6;i++){
fsamples[0]=&bufferOut->fbuf()->bands(0)[0][160*i];
fsamples[1]=&bufferOut->fbuf()->bands(0)[1][160*i];
fsamples[2]=&bufferOut->fbuf()->bands(0)[2][160*i];
foutput[0]=&bufferOut2->fbuf()->bands(0)[0][160*i];
foutput[1]=&bufferOut2->fbuf()->bands(0)[1][160*i];
foutput[2]=&bufferOut2->fbuf()->bands(0)[2][160*i];
int32_t res=WebRtcAec_Process(aecState, (const float* const*) fsamples, 3, foutput, 160, 20, 0);
}
// put the "bands" back together
splittingFilter->Synthesis(bufferOut2, bufferIn);
// convert the processed data back into 16-bit PCM
for(i=0;i<960;i++){
samples[i]=(int16_t) (CLAMP(fbuf[i], -1, 1)*32767);
}
If I comment out the actual echo cancellation and just do the float conversion and band splitting back and forth, it doesn't corrupt the memory, doesn't sound weird and runs indefinitely. (I do pass the farend/speaker signal into AEC, I just didn't want to make the mess of my code by including it in the question)
I've also tried Android's built-in AEC. While it does work, it upsamples the captured signal from 16 kHz.
Unfortunately, there is no free AEC package that support 48khz. So, either move to 32khz or use a commercial AEC package at 48khz.
The aac decoder is initialized as below:
MediaFormat outfmt = new MediaFormat();
outfmt.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
outfmt.setInteger(MediaFormat.KEY_AAC_PROFILE, mAudioProfile);
mSampleRate = format.getInteger(MediaFormat.KEY_SAMPLE_RATE);
outfmt.setInteger(MediaFormat.KEY_SAMPLE_RATE, mSampleRate);
mChannels = format.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
outfmt.setInteger(MediaFormat.KEY_CHANNEL_COUNT, mChannels);
outfmt.setInteger(MediaFormat.KEY_BIT_RATE, 64000);
audioEncoder.configure(outfmt, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
audioEncoder.start();
But the encoder behaviors different on two devices.
One outputs normal presentation:
64000 128000 192000 256000 320000
Another outputs as two channels:
64000 64000 128000 128000 192000 192000 256000 256000 320000 320000
And the format extracted using MediaExtractor is different on two devices:
the normal one is
{max-input-size=1572864, aac-profile=2,
csd-0=java.nio.ByteArrayBuffer[position=0,limit=2,capacity=2], sample-rate=16000,
durationUs=8640000, channel-count=1, mime=audio/mp4a-latm, isDMCMMExtractor=1}
The other is
{max-input-size=798, durationUs=8640000, channel-count=1, mime=audio/mp4a-latm,
csd-0=java.nio.ByteArrayBuffer[position=0,limit=2,capacity=2], sample-rate=16000}
So the original audio has one channel and the encoder is configured with one channel too.But the encoder outputs as in two channel way.
Does it matter with isDMCMMExtractor flag?
Help!Help!
#fadden
First off, the question is very hard to understand - both of the listed MediaFormat contents show channel-count=1, so there's very little actual explanation of the issue itself, only an explanation of other surrounding details.
However - the software AAC decoder in some android versions (4.1 if I remember correctly, possibly 4.2 as well) will decode mono AAC into stereo - not sure if some of the hardware AAC decoders do the same. You can argue whether this is a bug or just unexpected behaviour, but it's something you have to live with. In the case that the decoder returns stereo data even though the input was mono, both stereo channels will have the same (mono) content.
So basically, you have to be prepared to handle this - either pass the actual format information from the decoder (not from MediaExtractor) to whoever is using the data (e.g. reconfigure the audio output to stereo), or be prepared to mix down stereo back into mono if you really need to have the output in mono format.