Develop Reference

Android oboe c++ Some sounds distorted on playback

I'm using the Android oboe library for high performance audio in a music game.
In the assets folder I have 2 .raw files (both 48000Hz 16 bit PCM wavs and about 60kB)
std_kit_sn.raw
std_kit_ht.raw
These are loaded into memory as SoundRecordings and added to a Mixer. kSampleRateHz is 48000:
stdSN= SoundRecording::loadFromAssets(mAssetManager, "std_kit_sn.raw");
stdHT= SoundRecording::loadFromAssets(mAssetManager, "std_kit_ht.raw");
mMixer.addTrack(stdSN);
mMixer.addTrack(stdFT);
// Create a builder
AudioStreamBuilder builder;
builder.setFormat(AudioFormat::I16);
builder.setChannelCount(1);
builder.setSampleRate(kSampleRateHz);
builder.setCallback(this);
builder.setPerformanceMode(PerformanceMode::LowLatency);
builder.setSharingMode(SharingMode::Exclusive);
LOGD("After creating a builder");
// Open stream
Result result = builder.openStream(&mAudioStream);
if (result != Result::OK){
LOGE("Failed to open stream. Error: %s", convertToText(result));
}
LOGD("After openstream");
// Reduce stream latency by setting the buffer size to a multiple of the burst size
mAudioStream->setBufferSizeInFrames(mAudioStream->getFramesPerBurst() * 2);
// Start the stream
result = mAudioStream->requestStart();
if (result != Result::OK){
LOGE("Failed to start stream. Error: %s", convertToText(result));
}
LOGD("After starting stream");
They are called appropriately to play with standard code (as per Google tutorials) at required times:
stdSN->setPlaying(true);
stdHT->setPlaying(true); //Nasty Sound
The audio callback is standard (as per Google tutorials):
DataCallbackResult SoundFunctions::onAudioReady(AudioStream *mAudioStream, void *audioData, int32_t numFrames) {
// Play the stream
mMixer.renderAudio(static_cast<int16_t*>(audioData), numFrames);
return DataCallbackResult::Continue;
}
The std_kit_sn.raw plays fine. But std_kit_ht.raw has a nasty distortion. Both play with low latency. Why is one playing fine and the other has a nasty distortion?

I loaded your sample project and I believe the distortion you hear is caused by clipping/wraparound during mixing of sounds.
The Mixer object from the sample is a summing mixer. It just adds the values of each track together and outputs the sum.
You need to add some code to reduce the volume of each track to avoid exceeding the limits of an int16_t (although you're welcome to file a bug on the oboe project and I'll try to add this in an upcoming version). If you exceed this limit you'll get wraparound which is causing the distortion.
Additionally, your app is hardcoded to run at 22050 frames/sec. This will result in sub-optimal latency across most mobile devices because the stream is forced to upsample to the audio device's native frame rate. A better approach would be to leave the sample rate undefined when opening the stream - this will give you the optimal frame rate for the current audio device - then use a resampler on your source files to supply audio at this frame rate.

android - MediaCodec video only decoder framerate control

Refer to this link, I just add a simple delay when output buffer available:
ByteBuffer buffer = outputBuffers[outIndex];
Log.v("DecodeActivity", "We can't use this buffer but render it due to the API limit, " + buffer);
// We use a very simple clock to keep the video FPS, or the video
// playback will be too fast
while (info.presentationTimeUs / 1000 > System.currentTimeMillis() - startMs) {
try {
sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
break;
}
}
decoder.releaseOutputBuffer(outIndex, true);
But when I feed a 25fps video only frames, the decoded video looks like only 10fps (many frames looks like dropped).
But if I add a frameconut to check the fps, it's really 25fps, and if add MediaMuxer to mux the frames in input buffer, it playbacks fine, which means frames actually not been dropped.
So it's wired why fames there but not show on screen, but if I remove the delay, the playback will be very quick (almost 50fps).

Just found issue caused by TextureView, after change TextureView to SurfaceView, it works fine now.
But still not clear why TextureView performance is so bad.

MediaRecorder.setVideoFrameRate() not having any effect

I'm trying to get my Android app to record video with a lower frame rate (to reduce file size).
Here's my MediaRecorder configuration code:
m_mediaRecorder = new MediaRecorder();
m_mediaRecorder.setCamera(mCamera);
m_mediaRecorder.setVideoSource(MediaRecorder.VideoSource.DEFAULT);
CamcorderProfile profile = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH);
m_mediaRecorder.setOutputFormat(profile.fileFormat);
m_mediaRecorder.setVideoEncodingBitRate(profile.videoBitRate);
m_mediaRecorder.setVideoEncoder(profile.videoCodec);
m_mediaRecorder.setVideoSize(640, 480);
m_mediaRecorder.setVideoFrameRate(10);
m_mediaRecorder.setOrientationHint((int) orientationListener.getPreviewRotation(null));
m_mediaRecorder.setOutputFile(videoDirectory + "/" + uuid + ".mp4");
try {
m_mediaRecorder.prepare();
m_mediaRecorder.start();
} catch (Exception e) {
e.printStackTrace();
}
The video does record successfully, but no matter what I try, the frame rate seems to be fixed at 30 fps. m_mediaRecorder.setVideoFrameRate(10); has no effect.
(If I set the videoBitRate to a lower value, this reduces the file size but also reduces the quality of each individual frame - something we do not want to do.)
(For the record - Android 6.0.1; SDK Version 21.)
What am I missing?
Thanks,
Reuven

mediaRecorder.setCaptureRate(20);
mediaRecorder.setVideoFrameRate(20);
I am successfull with this, please try it. Thanks!

May be it makes sense to limit frame rate on Camera as well?
Anyway, do not expect significant gain from lowering frame rate, because the higher interval between frames, the more bits are required to encode the same picture.
Much more bitrate saving can be achieved by reducing picture resolution.

MediaCodec audio/video muxing issues ond Android

I am transcoding videos based on the example given by Google (https://android.googlesource.com/platform/cts/+/master/tests/tests/media/src/android/media/cts/ExtractDecodeEditEncodeMuxTest.java)
Basically, transocding of MP4 files works, but on some phones I get some weird results. If for example I transcode a video with audio on an HTC One, the code won't give any errors but the file cannot play afterward on the phone. If I have a 10 seconds video it jumps to almost the last second and you only here some crackling noise. If you play the video with VLC the audio track is completely muted.
I did not alter the code in terms of encoding/decoding and the same code gives correct results on a Nexus 5 or MotoX for example.
Anybody having an idea why it might fail on that specific device?
Best regard and thank you,
Florian

I made it work in Android 4.4.2 devices by following changes:
Set AAC profile to AACObjectLC instead of AACObjectHE
private static final int OUTPUT_AUDIO_AAC_PROFILE = MediaCodecInfo.CodecProfileLevel.AACObjectLC;
During creation of output audio format, use sample rate and channel count of input format instead of fixed values
MediaFormat outputAudioFormat = MediaFormat.createAudioFormat(OUTPUT_AUDIO_MIME_TYPE,
inputFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE),
inputFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT));
Put a check just before audio muxing audio track to control presentation timestamps. (To avoid timestampUs X < lastTimestampUs X for Audio track error)
if (audioPresentationTimeUsLast == 0) { // Defined in the begining of method
audioPresentationTimeUsLast = audioEncoderOutputBufferInfo.presentationTimeUs;
} else {
if (audioPresentationTimeUsLast > audioEncoderOutputBufferInfo.presentationTimeUs) {
audioEncoderOutputBufferInfo.presentationTimeUs = audioPresentationTimeUsLast + 1;
}
audioPresentationTimeUsLast = audioEncoderOutputBufferInfo.presentationTimeUs;
}
// Write data
if (audioEncoderOutputBufferInfo.size != 0) {
muxer.writeSampleData(outputAudioTrack, encoderOutputBuffer, audioEncoderOutputBufferInfo);
}
Hope this helps...

If original CTS tests fail you need to go to device vendors and ask for fixes

How to get CamcorderProfile.videoBitRate for an Android device?

My app uses HLS to stream video from a server, but when I request the HLS stream from the server I need to pass it the max video bitrate the device can handle. In the Android API guides it says that "a device's available video recording profiles can be used as a proxy for media playback capabilities," but when I try to retrieve the videoBitRate for the devices back-facing camera it always comes back as 12Mb/s regardless of the device (Galaxy Nexus, Galaxy Tab Plus 7", Galaxy Tab 8.9), despite the fact that they have 3 different GPUs (PowerVR SGX540, Mali-400 MP, Tegra 250 T20). Here's my code, am I doing something wrong?
CamcorderProfile camcorderProfile = CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH);
targetVideoBitRate = camcorderProfile.videoBitRate;
If I try this on the Galaxy Tab Plus:
boolean hasProfile = CamcorderProfile.hasProfile(CamcorderProfile.QUALITY_HIGH);
it returns True, despite the fact that QUALITY_HIGH is for 1080p recording and the specs say it can only record at 720p.

It looks like I've found the answer to my own question.
I didn't read the documentation closely enough, QUALITY_HIGH is not equivalent to 1080p it is simply a way of specifying the highest quality profile the device supports. Therefore, by definition, CamcorderProfile.hasProfile( CamcorderProfile.QUALITY_HIGH ) is always true. I should have written something like this:
public enum mVideoQuality {
FullHD, HD, SD
}
mVideoQuality mMaxVideoQuality;
int mTargetVideoBitRate;
private void initVideoQuality {
if ( CamcorderProfile.hasProfile( CamcorderProfile.QUALITY_1080P ) ) {
mMaxVideoQuality = mVideoQuality.FullHD;
} else if ( CamcorderProfile.hasProfile( CamcorderProfile.QUALITY_720P ) ) {
mMaxVideoQuality = mVideoQuality.HD;
} else {
mMaxVideoQuality = mVideoQuality.SD;
}
CamcorderProfile cProfile = CamcorderProfile.get( CamcorderProfile.QUALITY_HIGH );
mTargetVideoBitRate = cProfile.videoBitRate;
}
Most of my devices are still reporting support for 1080p encoding, which I'm skeptical of, however I ran this code on a Sony Experia Tipo ( my low end test device ) and it reported a max encode quality of 480p with a videoBitRate of 720Kb/s.
As I said, I'm not sure if every device can be trusted, but I have seen a range of video bitrates from 720Kb/s to 17Mb/s and Profile qualities from 480p - 1080p. Hopefully other people will find this information to be useful.