I want to amplify the audioData that is recorded by microphone using Oboe Library.
I created AudioEngine.cpp like this: https://github.com/google/oboe/blob/master/samples/LiveEffect/src/main/cpp/LiveEffectEngine.cpp
Here's is the class that has audioData:
DataCallbackResult
AudioEngine::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {
/* some code */
// add your audio processing here
return DataCallbackResult::Continue;
}
In the LiveEffect sample both the recording and playback streams are AudioFormat::I16 i.e. 16 bit integers. On this line you're casting to float:
auto *outputData = static_cast<float *>(audioData);
This is going to cause the distortion you hear so instead just cast to int16_t and multiply by a constant amplitude.
Make sure to check that the scaled up sample value isn't above INT16_MAX otherwise you'll get wraparound and distortion.
Related
I'm trying to use oboe in my audio/video communication app, and I'm trying the onAudioReady round-trip callback as in the oboe guide: https://github.com/google/oboe/blob/main/docs/FullGuide.md
Now I'm frustrating:
If the read directly write into the *audioData, the sound quality is perfect, i.e.:
auto result = recordingStream->read(audioData, numFrames, 0);
But if I add a buffer between them, there is significant noise/glitch which is very undesirable:
auto result = recordingStream->read(buffer, numFrames, 0);
std::copy(buffer, buffer + numFrames, static_cast<int16_t *>(audioData));
By inspecting log, this buffering action is done within 1ms, suppose won't hurt?
Both 1 and 2 also use PCM_I16 audio format, buffer is int16_t * with size of numFrames.
Hopefully someone can point out what's wrong to cause this? Sorry I'm lack of audio processing and c++ knowledge.
I've figured it out because the channel is stereo, samples per frames are 2, i.e.:
auto result = recordingStream->read(buffer, numFrames, 0);
std::copy(buffer, buffer + numFrames * 2, static_cast<int16_t *>(audioData));
I'm using the Android oboe library for high performance audio in a music game.
In the assets folder I have 2 .raw files (both 48000Hz 16 bit PCM wavs and about 60kB)
std_kit_sn.raw
std_kit_ht.raw
These are loaded into memory as SoundRecordings and added to a Mixer. kSampleRateHz is 48000:
stdSN= SoundRecording::loadFromAssets(mAssetManager, "std_kit_sn.raw");
stdHT= SoundRecording::loadFromAssets(mAssetManager, "std_kit_ht.raw");
mMixer.addTrack(stdSN);
mMixer.addTrack(stdFT);
// Create a builder
AudioStreamBuilder builder;
builder.setFormat(AudioFormat::I16);
builder.setChannelCount(1);
builder.setSampleRate(kSampleRateHz);
builder.setCallback(this);
builder.setPerformanceMode(PerformanceMode::LowLatency);
builder.setSharingMode(SharingMode::Exclusive);
LOGD("After creating a builder");
// Open stream
Result result = builder.openStream(&mAudioStream);
if (result != Result::OK){
LOGE("Failed to open stream. Error: %s", convertToText(result));
}
LOGD("After openstream");
// Reduce stream latency by setting the buffer size to a multiple of the burst size
mAudioStream->setBufferSizeInFrames(mAudioStream->getFramesPerBurst() * 2);
// Start the stream
result = mAudioStream->requestStart();
if (result != Result::OK){
LOGE("Failed to start stream. Error: %s", convertToText(result));
}
LOGD("After starting stream");
They are called appropriately to play with standard code (as per Google tutorials) at required times:
stdSN->setPlaying(true);
stdHT->setPlaying(true); //Nasty Sound
The audio callback is standard (as per Google tutorials):
DataCallbackResult SoundFunctions::onAudioReady(AudioStream *mAudioStream, void *audioData, int32_t numFrames) {
// Play the stream
mMixer.renderAudio(static_cast<int16_t*>(audioData), numFrames);
return DataCallbackResult::Continue;
}
The std_kit_sn.raw plays fine. But std_kit_ht.raw has a nasty distortion. Both play with low latency. Why is one playing fine and the other has a nasty distortion?
I loaded your sample project and I believe the distortion you hear is caused by clipping/wraparound during mixing of sounds.
The Mixer object from the sample is a summing mixer. It just adds the values of each track together and outputs the sum.
You need to add some code to reduce the volume of each track to avoid exceeding the limits of an int16_t (although you're welcome to file a bug on the oboe project and I'll try to add this in an upcoming version). If you exceed this limit you'll get wraparound which is causing the distortion.
Additionally, your app is hardcoded to run at 22050 frames/sec. This will result in sub-optimal latency across most mobile devices because the stream is forced to upsample to the audio device's native frame rate. A better approach would be to leave the sample rate undefined when opening the stream - this will give you the optimal frame rate for the current audio device - then use a resampler on your source files to supply audio at this frame rate.
My goal is to play local file while recording device's microphone input with low-latency.
I've come to Superpowered library, because from the documentation it provides low-latency feature.
I've created the player using SuperpoweredAdvancedAudioPlayer and SuperpoweredAndroidAudioIO and it plays fine.
SuperpoweredAndroidAudioIO has the construcor with parameters boolean enableInput, boolean enableOutput. Currently I'm using enableInput == false and enableOutput == true. When I put these parameters to true - no effect.
I wonder if it is possible to record file and play other file simultaneously?
Also there is SuperpoweredRecorder class in library but it says not for direct writing to disk. And need to use createWAV, fwrite, closeWAV methods.
I've tried implement Recorder separately but the quality is not good (it is two-three times faster than real recording + sound is distored).
Here is the simplest piece of code for recording I used:
void SuperpoweredFileRecorder::start(const char *destinationPath) {
file = createWAV(destinationPath, sampleRate, 2);
audioIO = new SuperpoweredAndroidAudioIO(sampleRate, bufferSize, true, false, audioProcessing, NULL, bufferSize); // Start audio input/output.
}
void SuperpoweredFileRecorder::stop() {
closeWAV(file);
audioIO->stop();
}
static bool audioProcessing(void *clientdata, short int *audioInputOutput, int numberOfSamples, int samplerate) {
fwrite(audioInputOutput, sizeof(short int), numberOfSamples, file);
return false;
}
Probably I cannot use Superpowered for that purpose and need to just make recording with OpenSL ES directly.
Thanks in advance!
After experiments I found the solution.
SuperpoweredRecorder works fine for recording tracks;
I've created to separate SuperpoweredAndroidAudioIO sources - one for playback and another for recorder. After some synchronization manipulation it works well (I minimized latency to very low level, so it suits my needs).
I post some code snippet with the idea I implemented:
https://bitbucket.org/snippets/kasurd/Mynnp/nativesuperpoweredrecorder-with
Hope it helps somebody!
You can do this with one instance of the SuperpoweredAndroidAudioIO with enableInput and enableOutput set to true.
The audio processing callback (audioProcessing() in your case) receives audio (microphone) in the audioInputOutput parameter. Just pass that to your SuperpoweredRecorder, and it will write it onto disk.
After that, do your SuperpoweredAdvancedAudioPlayer processing, and convert the result into audioInputOutput. That will go to the audio output.
So it's like, in pseudo-code:
audioProcessing(audioInputOutput) {
recorder->process(audioInputOutput)
player->process(some_buffer)
float_to_short_int(some_buffer, audioInputOutput)
}
Never do any fwrite in the audio processing callback, as it must complete within a very short time, and disk operations may be too slow.
For me this works when I double the numberOfSamples
fwrite(audioInputOutput, sizeof(short int), numberOfSamples * 2, file);
This will lead to a clear stereo output
I use the following code in a Thread to capture raw audio samples from the microphone and play it back through the speaker.
public void run(){
short[] lin = new short[SIZE_OF_RECORD_ARRAY];
int num = 0;
// am = (AudioManager) this.getSystemService(Context.AUDIO_SERVICE); // -> MOVED THESE TO init()
// am.setMode(AudioManager.MODE_IN_COMMUNICATION);
record.startRecording();
track.play();
while (passThroughMode) {
// while (!isInterrupted()) {
num = record.read(lin, 0, SIZE_OF_RECORD_ARRAY);
for(i=0;i<lin.length;i++)
lin[i] *= WAV_SAMPLE_MULTIPLICATION_FACTOR;
track.write(lin, 0, num);
}
// /*
record.stop();
track.stop();
record.release();
track.release();
// */
}
where record is an AudioRecord and track is an Audiotrack. I need to know in detail (and in a simplified way if possible) how the AudioRecord stores PCM data and AudioTrack plays PCM data. This is how I have understood it so far:
As the while() loop is continuously running, record obtains SIZE_OF_RECORD_ARRAY number of samples (which is 1024 for now) as shown in the figure. The samples get saved contiguously in the lin[] array of shorts (16 bit shorts, as I am using 16 bit PCM encoding). This is done by record.read(). Then track.write() places these samples in the speaker which is played by the hardware. Is this correct or am I missing something here?
As for how the samples are laid out in memory; they're just arrays of linear approximations to a sound wave, taken at discrete times (like your figure shows). In the case of stereo, the samples will be interleaved (LRLRLRLR...).
When it comes to the path the audio takes, you're essentially right, although there are a few more steps involved:
Writing data to your Java AudioTrack causes it to make a JNI (Java Native Interface) call to a native helper class, which in turn calls the native AudioTrack class.
The AudioTracks are owned by the AudioFlinger, which periodically takes data from all the AudioTracks on a given output thread (which have been mixed by the AudioMixer) and writes it to the audio HAL output stream class.
From there the data goes to the user-space ALSA library, and through a couple of intermediate steps to the kernel-space PCM driver. Then further on from there; typically going through some kind of DSP that applies various acoustic compensation filters, and eventually making it's way to the hardware codec, which controls the speaker DAC and amplifiers.
When recording from the internal microphone(s) you'd have more or less the same steps, except that they'd be done in the opposite order.
Note that some of these steps (essentially everything from the audio HAL and below) are platform-specific, and therefore might differ between platforms from different vendors (and even different platforms from the same vendor).
How can I implement
setRate(float f)
for my Android MediaPlayer, and secondly is it posible?
I believe this is the function you are looking for.
This sets the sampling rate at which the audio data will be consumed and played back, not the original sampling rate of the content. Setting it to half the sample rate of the content will cause the playback to last twice as long, but will also result in a negative pitch shift. The valid sample rate range is from 1Hz to twice the value returned by getNativeOutputSampleRate(int).
If you want to play mp3 directly using AudioTrack, you can either have a look at this example or convert your mp3 file to wav format, which enables AudioTrack to use it without hassle. This is the tradeoff you should account for if you want to adjust the playback rate easily.
Android 6.0 adds PlaybackParams for MediaPlayer, so you can now do this:
String recordingPath = recordingDirectory + File.separator + "music.mp3";
MediaPlayer audioPlayer = MediaPlayer.create(getApplicationContext(), Uri.parse(recordingPath));
audioPlayer.setAudioStreamType(AudioManager.STREAM_MUSIC);
PlaybackParams params = new PlaybackParams();
params.setSpeed(0.75f);
audioPlayer.setPlaybackParams(params);
audioPlayer.start();
I don't have an Android 6 device yet, but this works for me in the emulator.
Based on the Android developer documentation, you may have to use SoundPool instead.
Android Developer: Media SoundPool-setRate
public final void setRate (int streamID, float rate)
Change playback rate. The playback rate allows the application to vary
the playback rate (pitch) of the sound. A value of 1.0 means playback
at the original frequency. A value of 2.0 means playback twice as
fast, and a value of 0.5 means playback at half speed. If the stream
does not exist, it will have no effect.
Parameters
streamID: a streamID returned by the play() function
rate: playback rate (1.0 = normal playback, range 0.5 to 2.0)