How can I do audio recordings using android.media.AudioRecord without any smartphone-manufacturer-dependent fancy signal processing like automatic gain control (AGC) and/or equalization, noise suppression, echo cancellation, ... just the pure microphone signal?
Background
MediaRecorder.AudioSource provides nine constants,
DEFAULT and MIC initially being there,
VOICE_UPLINK, VOICE_DOWNLINK, and VOICE_CALL added in API level 4,
CAMCORDER and VOICE_RECOGNITION added in API 7,
VOICE_COMMUNICATION added in API 11,
REMOTE_SUBMIX added in API 19 but not available to third-party applications.
But none of them does a clean job across all smartphones. Rather, I have to find out myself it seems, which device uses which combinations of signal processing blocks for which MediaRecorder.AudioSource constant.
Would be nice to have a tenth constant like PURE_MIC added in API level 20.
But as long as this is not available, what can I do instead?
Short answer is "Nothing".
The AudioSources correspond to various logical audio input devices depending on the accessories that you have connected to the phone and the current use-case, which in turn corresponds to physical devices (primary built-in mic, secondary mic, wired headset mic, etc) with different tunings.
Each such combination of physical device and tuning is trimmed by the OEM to meet both external requirements (e.g. CTS, operator requirements, etc) and internal acoustic requirements set by the OEM itself. This process may cause the introduction of various filters - such as AGC, noise suppression, equalization, etc - into the audio input path at the hardware codec or multimedia DSP level.
While a PURE_MIC source might be useful in for some applications, it's not something that's available today.
On many devices you can control things like microphone gain, and possibly even the filter chain, by using amixer to write to the hardware codec's ALSA controls. However, this would obviously be a very platform-specific approach, and I also suspect that you have to be running as either the root or audio user to be allowed to do this.
Some devices add AGC effect to the sound input tract by default. Therefore, you need to obtain reference to corresponding AudioEffect object and force it to disable.
First, obtain AutomaticGainControl object linked to the AudioRecord audio session, and then just set it disabled:
if (AutomaticGainControl.isAvailable()) {
AutomaticGainControl agc = AutomaticGainControl.create(
myAudioRecord.getAudioSessionId()
);
agc.setEnabled(false);
}
Note: Most of the audio sources (including DEFAULT) apply processing to the audio signal. To record raw audio select UNPROCESSED. Some devices do not support unprocessed input. Call AudioManager.getProperty("PROPERTY_SUPPORT_AUDIO_SOURCE_UNPROCESSED") first to verify it's available. If it is not, try using VOICE_RECOGNITION instead, which does not employ AGC or noise suppression. You can use UNPROCESSED as an audio source even when the property is not supported, but there is no guarantee whether the signal will be unprocessed or not in that case.
Android documentation Link https://developer.android.com/guide/topics/media/mediarecorder.html#example
AudioManager audioManager = (AudioManager)getSystemService(Context.AUDIO_SERVICE);
if(audioManager.getProperty(AudioManager.PROPERTY_SUPPORT_AUDIO_SOURCE_UNPROCESSED) !=null)
mRecorder.setAudioSource(MediaRecorder.AudioSource.UNPROCESSED);
else
mRecorder.setAudioSource(MediaRecorder.AudioSource.VOICE_RECOGNITION);
MIC should be fine, and for the rest you need to know if they are supported.
I've made a class for this:
enum class AudioSource(val audioSourceValue: Int, val minApi: Int) {
VOICE_CALL(MediaRecorder.AudioSource.VOICE_CALL, 4), DEFAULT(MediaRecorder.AudioSource.DEFAULT, 1), MIC(MediaRecorder.AudioSource.MIC, 1),
VOICE_COMMUNICATION(MediaRecorder.AudioSource.VOICE_COMMUNICATION, 11), CAMCORDER(MediaRecorder.AudioSource.CAMCORDER, 7),
VOICE_RECOGNITION(MediaRecorder.AudioSource.VOICE_RECOGNITION, 7),
VOICE_UPLINK(MediaRecorder.AudioSource.VOICE_UPLINK, 4), VOICE_DOWNLINK(MediaRecorder.AudioSource.VOICE_DOWNLINK, 4),
#TargetApi(Build.VERSION_CODES.KITKAT)
REMOTE_SUBMIX(MediaRecorder.AudioSource.REMOTE_SUBMIX, 19),
#TargetApi(Build.VERSION_CODES.N)
UNPROCESSED(MediaRecorder.AudioSource.UNPROCESSED, 24);
fun isSupported(context: Context): Boolean =
when {
Build.VERSION.SDK_INT < minApi -> false
this != UNPROCESSED -> true
else -> {
val audioManager: AudioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager
Build.VERSION.SDK_INT >= Build.VERSION_CODES.N && "true" == audioManager.getProperty(AudioManager.PROPERTY_SUPPORT_AUDIO_SOURCE_UNPROCESSED)
}
}
companion object {
fun getAllSupportedValues(context: Context): ArrayList<AudioSource> {
val values = AudioSource.values()
val result = ArrayList<AudioSource>(values.size)
for (value in values)
if (value.isSupported(context))
result.add(value)
return result
}
}
}
Related
I build out the hdrvivid-debug.apk and installed on a mobile phone with Android 12. when playing one HDR Vivid test stream, the display is very darker compared to the default video player on the phone.
the vivid stream' file name is "hdr_vivid_selftest_dmsync_pq.mp4". it is used to do hdrvivid player test, it simply display a white rectanglular on the middle of screen. I can provide this stream if you want.
Thank you!
Houxiang
Update:
You can check whether the device has the HDR Vivid video decoding capability according the following method, docs link:
// Check the support for MediaCodec on the device.
MediaCodecList mcList = new MediaCodecList(MediaCodecList.ALL_CODECS);
MediaCodecInfo[] mcInfos = mcList.getCodecInfos();
for (MediaCodecInfo mci : mcInfos) {
// Filter out the encoder.
if (mci.isEncoder()) {
continue;
}
String[] types = mci.getSupportedTypes();
String typesArr = Arrays.toString(types);
// Filter out the non-HEVC decoder.
if (!typesArr.contains("hevc")) {
continue;
}
for (String type : types) {
// Check whether 10-bit HEVC decoding is supported.
MediaCodecInfo.CodecCapabilities codecCapabilities = mci.getCapabilitiesForType(type);
for (MediaCodecInfo.CodecProfileLevel codecProfileLevel : codecCapabilities.profileLevels) {
if (codecProfileLevel.profile == HEVCProfileMain10 || codecProfileLevel.profile == HEVCProfileMain10HDR10
|| codecProfileLevel.profile == HEVCProfileMain10HDR10Plus) {
// true means supported.
return true;
}
}
}
}
// false means unsupported.
return false;
If the device has the HDR Vivid video decoding capability, you can adjust the brightness of the video by adjusting the brightness bar of the device..
If the HDR capability is not supported, you need to call the setBrightness() interface to set the video brightness, and this interface takes effect only on devices that do not support HDR.
It seems that the HarmonyOS 2.0.0.268 system of Honor V30 Pro may not support HDR Vivid. The sample code invokes the setBrightness interface, the set value takes effect. But, the phone's default player may have done other processing, so it looks like the brightness may not be the same.
If HDR is not supported, you can set setBrightness() to a moderate brightness value.
What interface is used when you use the HDR Vivid capability? Native interface or Java interface?
About the display is too dark, Have you adjusted the output brightness?
https://developer.huawei.com/consumer/en/doc/development/Media-Guides/android-hdr-0000001276893212
If it's convenient, please provide a comparison video and HDR Vivid resource file for us to check.
I have a use case to change stream volume programmatically, but on newer android volume, raising the volume above a certain limit (60% as per my observations which corresponds to step 9 on most phones) results in a warning dialog:
Listening at high volume for a long time may damage your hearing. Tap OK to allow the volume
to be increased above safe levels
Cancel OK
I couldn't find any documentation about this in the the android developer portal, all I could find are some random articles citing the European regulations like this one:
According to regulations set by the European Committee for Electrotechnical Standarisation (CENELEC), all electronic devices capable of media playback sold after February 2013 must have a default output volume level of a maximum 85 dB. Users can choose to override the warning to increase the volume to a maximum of 100 dB, but in doing so the warning must re-appear after 20 hours of music playback.
So I need to figure out reliably what that number is, so I don't ever result in a volume change that would show this dialog, but I also don't want to just use step 9 as the max volume and, then find out that it's not the right value for another phone. Does the android API expose the max safe stream volume anywhere? If not, then do they at least document the step number that corresponds to it for different phone?
Thanks!
There's a resource which holds the safe volume step: config_safe_media_volume_index
// .../overlay/frameworks/base/core/res/res/values/config.xml
<integer name="config_safe_media_volume_index">7</integer>
It is defined HERE
And it is used HERE
You can get it dinamically via:
int safeVolumeStep;
int safeVolumeStepResourceId =
getResources().getIdentifier("config_safe_media_volume_index", "integer", "android");
if(safeVolumeStepResourceId != 0) {
safeVolumeStep = getResources().getInteger(safeVolumeStepResourceId);
} else {
Log.w("TESTS", "Resource config_safe_media_volume_index not found. Setting a hardcoded value");
// We probably won't fall here because config_safe_media_volume_index is defined in the AOSP
// It not a vendor specific resource...
// For any case, try to set the safe step manually to 60% of the max volume.
AudioManager audioManager = (AudioManager) getSystemService(Context.AUDIO_SERVICE);
int maxVolume = audioManager.getStreamMaxVolume(AudioManager.STREAM_MUSIC);
safeVolumeStep = (int) (maxVolume * 0.6f);
}
Log.d("TESTS", "Safe Volume Step: " + safeVolumeStep +
" Safe volume step resourceID: " + Integer.toHexString(safeVolumeStepResourceId) );
I tested here in a Galaxy S10 and I'm getting 9.
In the application which I want to create, I face some technical obstacles. I have two music tracks in the application. For example, a user imports the music background as a first track. The second path is a voice recorded by the user to the rhythm of the first track played by the speaker device (or headphones). At this moment we face latency. After recording and playing back in the app, the user hears the loss of synchronisation between tracks, which occurs because of the microphone and speaker latencies.
Firstly, I try to detect the delay by filtering the input sound. I use android’s AudioRecord class, and the method read(). This method fills my short array with audio data.
I found that the initial values of this array are zeros so I decided to cut them out before I will start to write them into the output stream.
So I consider those zeros as a „warmup” latency of the microphone. Is this approach correct? This operation gives some results, but it doesn’t resolve the problem, and at this stage, I’m far away from that.
But the worse case is with the delay between starting the speakers and playing the music. This delay I cannot filter or detect. I tried to create some calibration feature which counts the delay. I play a „beep” sound through the speakers, and when I start to play it, I also begin to measure time. Then, I start recording and listen for this sound being detected by the microphone. When I recognise this sound in the app, I stop measuring time. I repeat this process several times, and the final value is the average from those results. That is how I try to measure the latency of the device. Now, when I have this value, I can simply shift the second track backwards to achieve synchronisation of both records (I will lose some initial milliseconds of the recording, but I skip this case, for now, there are some possibilities to fix it).
I thought that this approach would resolve the problem, but it turned out this is not as simple as I thought. I found two issues here:
1. Delay while playing two tracks simultaneously
2. Random in device audio latency.
The first: I play two tracks using AudioTrack class and I run method play() like this:
val firstTrack = //creating a track
val secondTrack = //creating a track
firstTrack.play()
secondTrack.play()
This code causes delays at the stage of playing tracks. Now, I don’t even have to think about latency while recording; I cannot play two tracks simultaneously without delays. I tested this with some external audio file (not recorded in my app) - I’m starting the same audio file using the code above, and I can see a delay. I also tried it with MediaPlayer class, and I have the same results. In this case, I even try to play tracks when callback OnPreparedListener invoke:
val firstTrack = //AudioPlayer
val secondTrack = //AudioPlayer
second.setOnPreparedListener {
first.start()
second.start()
}
And it doesn’t help.
I know that there is one more class provided by Android called SoundPool. According to the documentation, it can be better with playing tracks simultaneously, but I can’t use it because it supports only small audio files and that can't limit me.
How can I resolve this problem? How can I start playing two tracks precisely at the same time?
The second: Audio latency is not deterministic - sometimes it is smaller, and sometimes it’s huge, and it’s out of my hands. So measuring device latency can help but again - it cannot resolve the problem.
To sum up: is there any solution, which can give me exact latency per device (or app session?) or other triggers which detect actual delay, to provide the best synchronisation while playback two tracks at the same time?
Thank you in advance!
Synchronising audio for karaoke apps is tough. The main issue you seem to be facing is variable latency in the output stream.
This is almost certainly caused by "warm up" latency: the time it takes from hitting "play" on your backing track to the first frame of audio data being rendered by the audio device (e.g. headphones). This can have large variance and is difficult to measure.
The first (and easiest) thing to try is to use MODE_STREAM when constructing your AudioTrack and prime it with bufferSizeInBytes of data prior to calling play (more here). This should result in lower, more consistent "warm up" latency.
A better way is to use the Android NDK to have a continuously running audio stream which is just outputting silence until the moment you hit play, then start sending audio frames immediately. The only latency you have here is the continuous output latency.
If you decide to go down this route I recommend taking a look at the Oboe library (full disclosure: I am one of the authors).
To answer one of your specific questions...
Is there a way to calculate the latency of the audio output stream programatically?
Yes. The easiest way to explain this is with a code sample (this is C++ for the AAudio API but the principle is the same using Java AudioTrack):
// Get the index and time that a known audio frame was presented for playing
int64_t existingFrameIndex;
int64_t existingFramePresentationTime;
AAudioStream_getTimestamp(stream, CLOCK_MONOTONIC, &existingFrameIndex, &existingFramePresentationTime);
// Get the write index for the next audio frame
int64_t writeIndex = AAudioStream_getFramesWritten(stream);
// Calculate the number of frames between our known frame and the write index
int64_t frameIndexDelta = writeIndex - existingFrameIndex;
// Calculate the time which the next frame will be presented
int64_t frameTimeDelta = (frameIndexDelta * NANOS_PER_SECOND) / sampleRate_;
int64_t nextFramePresentationTime = existingFramePresentationTime + frameTimeDelta;
// Assume that the next frame will be written into the stream at the current time
int64_t nextFrameWriteTime = get_time_nanoseconds(CLOCK_MONOTONIC);
// Calculate the latency
*latencyMillis = (double) (nextFramePresentationTime - nextFrameWriteTime) / NANOS_PER_MILLISECOND;
A caveat: This method relies on accurate timestamps being reported by the audio hardware. I know this works on Google Pixel devices but have heard reports that it isn't so accurate on other devices so YMMV.
Following the answer of donturner, here's a Java version (that also uses other methods depending on the SDK version)
/** The audio latency has not been estimated yet */
private static long AUDIO_LATENCY_NOT_ESTIMATED = Long.MIN_VALUE+1;
/** The audio latency default value if we cannot estimate it */
private static long DEFAULT_AUDIO_LATENCY = 100L * 1000L * 1000L; // 100ms
/**
* Estimate the audio latency
*
* Not accurate at all, depends on SDK version, etc. But that's the best
* we can do.
*/
private static void estimateAudioLatency(AudioTrack track, long audioFramesWritten) {
long estimatedAudioLatency = AUDIO_LATENCY_NOT_ESTIMATED;
// First method. SDK >= 19.
if (Build.VERSION.SDK_INT >= 19 && track != null) {
AudioTimestamp audioTimestamp = new AudioTimestamp();
if (track.getTimestamp(audioTimestamp)) {
// Calculate the number of frames between our known frame and the write index
long frameIndexDelta = audioFramesWritten - audioTimestamp.framePosition;
// Calculate the time which the next frame will be presented
long frameTimeDelta = _framesToNanoSeconds(frameIndexDelta);
long nextFramePresentationTime = audioTimestamp.nanoTime + frameTimeDelta;
// Assume that the next frame will be written at the current time
long nextFrameWriteTime = System.nanoTime();
// Calculate the latency
estimatedAudioLatency = nextFramePresentationTime - nextFrameWriteTime;
}
}
// Second method. SDK >= 18.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED && Build.VERSION.SDK_INT >= 18) {
Method getLatencyMethod;
try {
getLatencyMethod = AudioTrack.class.getMethod("getLatency", (Class<?>[]) null);
estimatedAudioLatency = (Integer) getLatencyMethod.invoke(track, (Object[]) null) * 1000000L;
} catch (Exception ignored) {}
}
// If no method has successfully gave us a value, let's try a third method
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
AudioManager audioManager = (AudioManager) CRT.getInstance().getSystemService(Context.AUDIO_SERVICE);
try {
Method getOutputLatencyMethod = audioManager.getClass().getMethod("getOutputLatency", int.class);
estimatedAudioLatency = (Integer) getOutputLatencyMethod.invoke(audioManager, AudioManager.STREAM_MUSIC) * 1000000L;
} catch (Exception ignored) {}
}
// No method gave us a value. Let's use a default value. Better than nothing.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
estimatedAudioLatency = DEFAULT_AUDIO_LATENCY;
}
return estimatedAudioLatency
}
private static long _framesToNanoSeconds(long frames) {
return frames * 1000000000L / SAMPLE_RATE;
}
The android MediaPlayer class is notoriously slow to begin audio playback, I experienced an issue in an app I was creating where there was a greater than one second delay to begin playing an audio clip. I resolved it by switching to ExoPlayer which resulted in the playback starting within 100ms. I've also read that ffmpeg has even faster start audio startup time than ExoPlayer but I haven't used it so I can't make any promises.
Is there a way to find out if the decoder that received using MediaCodec.createDecoderByType(type) is a hardware decoder or a software decoder?
There is no real formal flag for indicating whether a codec is a hardware or software codec. In practice, you can do this, though:
MediaCodec codec = MediaCodec.createDecoderByType(type);
if (codec.getName().startsWith("OMX.google.")) {
// Is a software codec
}
(The MediaCodec.getName() method is available since API level 18. For lower API levels, you instead need to iterate over the entries in MediaCodecList and manually pick the right codec that fits your needs instead.)
Putting it here for anyone it might help. According to code for libstagefright, any codec which starts with OMX.google. or c2.android. or does not start with (OMX. and c2.) are all software codecs.
//static
bool MediaCodecList::isSoftwareCodec(const AString &componentName) {
return componentName.startsWithIgnoreCase("OMX.google.")
|| componentName.startsWithIgnoreCase("c2.android.")
|| (!componentName.startsWithIgnoreCase("OMX.")
&& !componentName.startsWithIgnoreCase("c2."));
}
Source:
https://android.googlesource.com/platform/frameworks/av/+/master/media/libstagefright/MediaCodecList.cpp#320
I am transcoding videos based on the example given by Google (https://android.googlesource.com/platform/cts/+/master/tests/tests/media/src/android/media/cts/ExtractDecodeEditEncodeMuxTest.java)
Basically, transocding of MP4 files works, but on some phones I get some weird results. If for example I transcode a video with audio on an HTC One, the code won't give any errors but the file cannot play afterward on the phone. If I have a 10 seconds video it jumps to almost the last second and you only here some crackling noise. If you play the video with VLC the audio track is completely muted.
I did not alter the code in terms of encoding/decoding and the same code gives correct results on a Nexus 5 or MotoX for example.
Anybody having an idea why it might fail on that specific device?
Best regard and thank you,
Florian
I made it work in Android 4.4.2 devices by following changes:
Set AAC profile to AACObjectLC instead of AACObjectHE
private static final int OUTPUT_AUDIO_AAC_PROFILE = MediaCodecInfo.CodecProfileLevel.AACObjectLC;
During creation of output audio format, use sample rate and channel count of input format instead of fixed values
MediaFormat outputAudioFormat = MediaFormat.createAudioFormat(OUTPUT_AUDIO_MIME_TYPE,
inputFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE),
inputFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT));
Put a check just before audio muxing audio track to control presentation timestamps. (To avoid timestampUs X < lastTimestampUs X for Audio track error)
if (audioPresentationTimeUsLast == 0) { // Defined in the begining of method
audioPresentationTimeUsLast = audioEncoderOutputBufferInfo.presentationTimeUs;
} else {
if (audioPresentationTimeUsLast > audioEncoderOutputBufferInfo.presentationTimeUs) {
audioEncoderOutputBufferInfo.presentationTimeUs = audioPresentationTimeUsLast + 1;
}
audioPresentationTimeUsLast = audioEncoderOutputBufferInfo.presentationTimeUs;
}
// Write data
if (audioEncoderOutputBufferInfo.size != 0) {
muxer.writeSampleData(outputAudioTrack, encoderOutputBuffer, audioEncoderOutputBufferInfo);
}
Hope this helps...
If original CTS tests fail you need to go to device vendors and ask for fixes