Develop Reference

Android: Read audio data from file uri

I want to analyse an audio file (mp3 in particular) which the user can select and determine what notes are played, when they're player and with what frequency.
I already have some working code for my computer, but I want to be able to use this on my phone as well.
In order to do this however, I need access to the bytes of the audio file. On my PC I could just open a stream and use AudioFormat to decode it and then read() the bytes frame by frame.
Looking at the Android Developer Forums I can only find classes and examples for playing a file (without access to the bytes) or recording to a file (I want to read from a file).
I'm pretty confident that I can set up a file chooser, but once I have the Uri from that, I don't know how to get a stream or the bytes.
Any help would be much appreciated :)
Edit: Is a similar solution to this possible? Android - Read a File
I don't know if I could decode the audio file that way or if there would be any problems with the Android API...

So I solved it in the following way:
Get an InputStream with
final InputStream inputStream = getContentResolver().openInputStream(selectedUri);
Then pass it in this function and decode it using classes from JLayer:
private synchronized void decode(InputStream in)
throws BitstreamException, DecoderException {
ArrayList<Short> output = new ArrayList<>(1024);
Bitstream bitstream = new Bitstream(in);
Decoder decoder = new Decoder();
float total_ms = 0f;
float nextNotify = -1f;
boolean done = false;
while (! done) {
Header frameHeader = bitstream.readFrame();
if (total_ms > nextNotify) {
mListener.OnDecodeUpdate((int) total_ms);
nextNotify += 500f;
}
if (frameHeader == null) {
done = true;
} else {
total_ms += frameHeader.ms_per_frame();
SampleBuffer buffer = (SampleBuffer) decoder.decodeFrame(frameHeader, bitstream); // CPU intense
if (buffer.getSampleFrequency() != 44100 || buffer.getChannelCount() != 2) {
throw new DecoderException("mono or non-44100 MP3 not supported", null);
}
short[] pcm = buffer.getBuffer();
for (int i = 0; i < pcm.length-1; i += 2) {
short l = pcm[i];
short r = pcm[i+1];
short mono = (short) ((l + r) / 2f);
output.add(mono); // RAM intense
}
}
bitstream.closeFrame();
}
bitstream.close();
mListener.OnDecodeComplete(output);
}
The full project (in case you want to look up the particulars) can be found here:
https://github.com/S7uXN37/MusicInterpreterStudio/

audio and video track synchronization issue when using mediacodec and mediamuxer for mp4 files

I would like to produce mp4 file by multiplexing audio from mic (overwrite didGetAudioData) and video from camera (overwrite onpreviewframe).However, I encountered the sound and video synchronization problem, video will appear faster than audio. I wondered if the problem related to incompatible configurations or presentationTimeUs, could someone guide me how to fix the problem. Below were my software.
Video configuration
formatVideo = MediaFormat.createVideoFormat(MIME_TYPE_VIDEO, 640, 360);
formatVideo.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar);
formatVideo.setInteger(MediaFormat.KEY_BIT_RATE, 2000000);
formatVideo.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
formatVideo.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
got video presentationPTS as below,
if(generateIndex == 0) {
videoAbsolutePtsUs = 132;
StartVideoAbsolutePtsUs = System.nanoTime() / 1000L;
}else {
CurrentVideoAbsolutePtsUs = System.nanoTime() / 1000L;
videoAbsolutePtsUs =132+ CurrentVideoAbsolutePtsUs-StartVideoAbsolutePtsUs;
}
generateIndex++;
audio configuration
format = MediaFormat.createAudioFormat(MIME_TYPE, 48000/*sample rate*/, AudioFormat.CHANNEL_IN_MONO /*Channel config*/);
format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC);
format.setInteger(MediaFormat.KEY_SAMPLE_RATE,48000);
format.setInteger(MediaFormat.KEY_CHANNEL_COUNT,1);
format.setInteger(MediaFormat.KEY_BIT_RATE,64000);
got audio presentationPTS as below,
if(generateIndex == 0) {
audioAbsolutePtsUs = 132;
StartAudioAbsolutePtsUs = System.nanoTime() / 1000L;
}else {
CurrentAudioAbsolutePtsUs = System.nanoTime() / 1000L;
audioAbsolutePtsUs =CurrentAudioAbsolutePtsUs - StartAudioAbsolutePtsUs;
}
generateIndex++;
audioAbsolutePtsUs = getJitterFreePTS(audioAbsolutePtsUs, audioInputLength / 2);
long startPTS = 0;
long totalSamplesNum = 0;
private long getJitterFreePTS(long bufferPts, long bufferSamplesNum) {
long correctedPts = 0;
long bufferDuration = (1000000 * bufferSamplesNum) / 48000;
bufferPts -= bufferDuration; // accounts for the delay of acquiring the audio buffer
if (totalSamplesNum == 0) {
// reset
startPTS = bufferPts;
totalSamplesNum = 0;
}
correctedPts = startPTS + (1000000 * totalSamplesNum) / 48000;
if(bufferPts - correctedPts >= 2*bufferDuration) {
// reset
startPTS = bufferPts;
totalSamplesNum = 0;
correctedPts = startPTS;
}
totalSamplesNum += bufferSamplesNum;
return correctedPts;
}
Was my issue caused by applying jitter function for audio only? If yes, how could I apply jitter function for video? I also tried to find correct audio and video presentationPTS by https://android.googlesource.com/platform/cts/+/jb-mr2-release/tests/tests/media/src/android/media/cts/EncodeDecodeTest.java. But encodedecodeTest only provided video PTS. That's the reason my implementation used system nanotime for both audio and video. If I want to use video presentationPTS in encodedecodetest, how to construct the compatible audio presentationPTS? Thanks for help!
below are how i queue yuv frame to video mediacodec for reference. For audio part, it is identical except for different presentationPTS.
int videoInputBufferIndex;
int videoInputLength;
long videoAbsolutePtsUs;
long StartVideoAbsolutePtsUs, CurrentVideoAbsolutePtsUs;
int put_v =0;
int get_v =0;
int generateIndex = 0;
public void setByteBufferVideo(byte[] buffer, boolean isUsingFrontCamera, boolean Input_endOfStream){
if(Build.VERSION.SDK_INT >=18){
try{
endOfStream = Input_endOfStream;
if(!Input_endOfStream){
ByteBuffer[] inputBuffers = mVideoCodec.getInputBuffers();
videoInputBufferIndex = mVideoCodec.dequeueInputBuffer(-1);
if (VERBOSE) {
Log.w(TAG,"[put_v]:"+(put_v)+"; videoInputBufferIndex = "+videoInputBufferIndex+"; endOfStream = "+endOfStream);
}
if(videoInputBufferIndex>=0) {
ByteBuffer inputBuffer = inputBuffers[videoInputBufferIndex];
inputBuffer.clear();
inputBuffer.put(mNV21Convertor.convert(buffer));
videoInputLength = buffer.length;
if(generateIndex == 0) {
videoAbsolutePtsUs = 132;
StartVideoAbsolutePtsUs = System.nanoTime() / 1000L;
}else {
CurrentVideoAbsolutePtsUs = System.nanoTime() / 1000L;
videoAbsolutePtsUs =132+ CurrentVideoAbsolutePtsUs - StartVideoAbsolutePtsUs;
}
generateIndex++;
if (VERBOSE) {
Log.w(TAG, "[put_v]:"+(put_v)+"; videoAbsolutePtsUs = " + videoAbsolutePtsUs + "; CurrentVideoAbsolutePtsUs = "+CurrentVideoAbsolutePtsUs);
}
if (videoInputLength == AudioRecord.ERROR_INVALID_OPERATION) {
Log.w(TAG, "[put_v]ERROR_INVALID_OPERATION");
} else if (videoInputLength == AudioRecord.ERROR_BAD_VALUE) {
Log.w(TAG, "[put_v]ERROR_ERROR_BAD_VALUE");
}
if (endOfStream) {
Log.w(TAG, "[put_v]:"+(put_v++)+"; [get] receive endOfStream");
mVideoCodec.queueInputBuffer(videoInputBufferIndex, 0, videoInputLength, videoAbsolutePtsUs, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
} else {
Log.w(TAG, "[put_v]:"+(put_v++)+"; receive videoInputLength :" + videoInputLength);
mVideoCodec.queueInputBuffer(videoInputBufferIndex, 0, videoInputLength, videoAbsolutePtsUs, 0);
}
}
}
}catch (Exception x) {
x.printStackTrace();
}
}
}

How I solved this in my application was by setting the PTS of all video and audio frames against a shared "sync clock" (note the sync also means it's thread-safe) that starts when the first video frame (having a PTS 0 on its own) is available. So if audio recording starts sooner than video, audio data is dismissed (doesn't go into encoder) until video starts, and if it starts later, then the first audio PTS will be relative to the start of the entire video.
Ofcourse you are free to allow audio to start first, but players will usually skip or wait for the first video frame anyway. Also be careful that encoded audio frames will arrive "out of order" and MediaMuxer will fail with an error sooner or later. My solution was to queue them all like this: sort them by pts when a new one comes in, then write everything that is older than 500 ms (relative to the newest one) to MediaMuxer, but only those with a PTS higher than the latest written frame. Ideally this means data is smoothly written to MediaMuxer, with a 500 ms delay. Worst case, you will lose a few audio frames.

Increase volume output of recorded audio

I am trying to make a call recording app in Android. I am using loudspeaker to record both uplink and downlink audio. The only problem I am facing is the volume is too low. I've increased the volume of device using AudioManager to max and it can't go beyond that.
I've first used MediaRecorder, but since it had limited functions and provides compressed audio, I've tried with AudioRecorder. Still I havn't figured out how to increase the audio. I've checked on projects on Github too, but it's of no use. I've searched on stackoverflow for last two weeks, but couldn't find anything at all.
I am quite sure that it's possible, since many other apps are doing it. For instance Automatic Call recorder does that.
I understand that I have to do something with the audio buffer, but I am not quite sure what needs to be done on that. Can you guide me on that.
Update:-
I am sorry that I forgot to mention that I am already using Gain. My code is almost similar to RehearsalAssistant (in fact I derived it from there). The gain doesn't work for more than 10dB and that doesn't increase the audio volume too much. What I wanted is I should be able to listen to the audio without putting my ear on the speaker which is what lacking in my code.
I've asked a similar question on functioning of the volume/loudness at SoundDesign SE here. It mentions that the Gain and loudness is related but it doesn't set the actual loudness level. I am not sure how things work, but I am determined to get the loud volume output.

You obviously have the AudioRecord stuff running, so I skip the decision for sampleRate and inputSource. The main point is that you need to appropriately manipulate each sample of your recorded data in your recording loop to increase the volume. Like so:
int minRecBufBytes = AudioRecord.getMinBufferSize( sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT );
// ...
audioRecord = new AudioRecord( inputSource, sampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, minRecBufBytes );
// Setup the recording buffer, size, and pointer (in this case quadruple buffering)
int recBufferByteSize = minRecBufBytes*2;
byte[] recBuffer = new byte[recBufferByteSize];
int frameByteSize = minRecBufBytes/2;
int sampleBytes = frameByteSize;
int recBufferBytePtr = 0;
audioRecord.startRecording();
// Do the following in the loop you prefer, e.g.
while ( continueRecording ) {
int reallySampledBytes = audioRecord.read( recBuffer, recBufferBytePtr, sampleBytes );
int i = 0;
while ( i < reallySampledBytes ) {
float sample = (float)( recBuffer[recBufferBytePtr+i ] & 0xFF
| recBuffer[recBufferBytePtr+i+1] << 8 );
// THIS is the point were the work is done:
// Increase level by about 6dB:
sample *= 2;
// Or increase level by 20dB:
// sample *= 10;
// Or if you prefer any dB value, then calculate the gain factor outside the loop
// float gainFactor = (float)Math.pow( 10., dB / 20. ); // dB to gain factor
// sample *= gainFactor;
// Avoid 16-bit-integer overflow when writing back the manipulated data:
if ( sample >= 32767f ) {
recBuffer[recBufferBytePtr+i ] = (byte)0xFF;
recBuffer[recBufferBytePtr+i+1] = 0x7F;
} else if ( sample <= -32768f ) {
recBuffer[recBufferBytePtr+i ] = 0x00;
recBuffer[recBufferBytePtr+i+1] = (byte)0x80;
} else {
int s = (int)( 0.5f + sample ); // Here, dithering would be more appropriate
recBuffer[recBufferBytePtr+i ] = (byte)(s & 0xFF);
recBuffer[recBufferBytePtr+i+1] = (byte)(s >> 8 & 0xFF);
}
i += 2;
}
// Do other stuff like saving the part of buffer to a file
// if ( reallySampledBytes > 0 ) { ... save recBuffer+recBufferBytePtr, length: reallySampledBytes
// Then move the recording pointer to the next position in the recording buffer
recBufferBytePtr += reallySampledBytes;
// Wrap around at the end of the recording buffer, e.g. like so:
if ( recBufferBytePtr >= recBufferByteSize ) {
recBufferBytePtr = 0;
sampleBytes = frameByteSize;
} else {
sampleBytes = recBufferByteSize - recBufferBytePtr;
if ( sampleBytes > frameByteSize )
sampleBytes = frameByteSize;
}
}

Thanks to Hartmut and beworker for the solution. Hartmut's code did worked at near 12-14 dB. I did merged the code from the sonic library too to increase volume, but that increase too much noise and distortion, so I kept the volume at 1.5-2.0 and instead tried to increase gain. I got decent sound volume which doesn't sound too loud in phone, but when listened on a PC sounds loud enough. Looks like that's the farthest I could go.
I am posting my final code to increase the loudness. Be aware that using increasing mVolume increases too much noise. Try to increase gain instead.
private AudioRecord.OnRecordPositionUpdateListener updateListener = new AudioRecord.OnRecordPositionUpdateListener() {
#Override
public void onPeriodicNotification(AudioRecord recorder) {
aRecorder.read(bBuffer, bBuffer.capacity()); // Fill buffer
if (getState() != State.RECORDING)
return;
try {
if (bSamples == 16) {
shBuffer.rewind();
int bLength = shBuffer.capacity(); // Faster than accessing buffer.capacity each time
for (int i = 0; i < bLength; i++) { // 16bit sample size
short curSample = (short) (shBuffer.get(i) * gain);
if (curSample > cAmplitude) { // Check amplitude
cAmplitude = curSample;
}
if(mVolume != 1.0f) {
// Adjust output volume.
int fixedPointVolume = (int)(mVolume*4096.0f);
int value = (curSample*fixedPointVolume) >> 12;
if(value > 32767) {
value = 32767;
} else if(value < -32767) {
value = -32767;
}
curSample = (short)value;
/*scaleSamples(outputBuffer, originalNumOutputSamples, numOutputSamples - originalNumOutputSamples,
mVolume, nChannels);*/
}
shBuffer.put(curSample);
}
} else { // 8bit sample size
int bLength = bBuffer.capacity(); // Faster than accessing buffer.capacity each time
bBuffer.rewind();
for (int i = 0; i < bLength; i++) {
byte curSample = (byte) (bBuffer.get(i) * gain);
if (curSample > cAmplitude) { // Check amplitude
cAmplitude = curSample;
}
bBuffer.put(curSample);
}
}
bBuffer.rewind();
fChannel.write(bBuffer); // Write buffer to file
payloadSize += bBuffer.capacity();
} catch (IOException e) {
e.printStackTrace();
Log.e(NoobAudioRecorder.class.getName(), "Error occured in updateListener, recording is aborted");
stop();
}
}
#Override
public void onMarkerReached(AudioRecord recorder) {
// NOT USED
}
};

simple use MPEG_4 format
To increase the call recording volume use AudioManager as follows:
int deviceCallVol;
AudioManager audioManager;
Start Recording:
audioManager = (AudioManager)context.getSystemService(Context.AUDIO_SERVICE);
//get the current volume set
deviceCallVol = audioManager.getStreamVolume(AudioManager.STREAM_VOICE_CALL);
//set volume to maximum
audioManager.setStreamVolume(AudioManager.STREAM_VOICE_CALL, audioManager.getStreamMaxVolume(AudioManager.STREAM_VOICE_CALL), 0);
recorder.setAudioSource(MediaRecorder.AudioSource.VOICE_CALL);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setAudioEncoder(MediaRecorder.AudioEncoder.AAC);
recorder.setAudioEncodingBitRate(32);
recorder.setAudioSamplingRate(44100);
Stop Recording:
//revert volume to initial state
audioManager.setStreamVolume(AudioManager.STREAM_VOICE_CALL, deviceCallVol, 0);

In my app I use an open source sonic library. Its main purpose is to speed up / slow down speech, but besides this it allows to increase loudness too. I apply it to playback, but it must work for recording similarly. Just pass your samples through it before compressing them. It has a Java interface too. Hope this helps.

Why AVCodecContext extradata is NULL?

I am trying to decode h264 video using ffmpeg and stagefright library. I'm using this example.
The example shows how to decode mp4 files, but i want to decode only h264 video.
Here is piece of my code..
AVFormatSource::AVFormatSource(const char *videoPath)
{
av_register_all();
mDataSource = avformat_alloc_context();
avformat_open_input(&mDataSource, videoPath, NULL, NULL);
for (int i = 0; i < mDataSource->nb_streams; i++)
{
if (mDataSource->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO)
{
mVideoIndex = i;
break;
}
}
mVideoTrack = mDataSource->streams[mVideoIndex]->codec;
size_t bufferSize = (mVideoTrack->width * mVideoTrack->height * 3) / 2;
mGroup.add_buffer(new MediaBuffer(bufferSize));
mFormat = new MetaData;
switch (mVideoTrack->codec_id == CODEC_ID_H264)
{
mConverter = av_bitstream_filter_init("h264_mp4toannexb");
mFormat->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_AVC);
if (mVideoTrack->extradata[0] == 1) //SIGSEGV Here
{
mFormat->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_AVC);
mFormat->setData(kKeyAVCC, kTypeAVCC, mVideoTrack->extradata,
mVideoTrack->extradata_size);
}
}
mFormat->setInt32(kKeyWidth, mVideoTrack->width);
mFormat->setInt32(kKeyHeight, mVideoTrack->height);
}
mVideoTrack->extradata is NULL. What i'm doing wrong?? My question is, what should be in mVideoTrack->extradata for kKeyAVCC ??
Please help me, I need Your help.
Thanks in advance.

If your input is a raw h.264 file, It is already in annex B format. So you do not need to do the "h264_mp4toannexb" conversion. In addition, in annex B, the SPS/PPS are sent inline with the first (or every) IDR frame. So no extra data is needed. Read more here: Possible Locations for Sequence/Picture Parameter Set(s) for H.264 Stream

MediaCodec.INFO_OUTPUT_FORMAT_CHANGED not called when decoding h.264 in some devices

I'm using the following code to prepare the hardware decoder. I expect outputBufferIndex to be -1 and then followed by MediaCodec.INFO_OUTPUT_FORMAT_CHANGED. It shouldn't >=0 before notifying format changed.
I tested the code in 25 different devices, and 7 of them never return INFO_OUTPUT_FORMAT_CHANGED. mediaCodec.getOutputFormat() returned IllegalStateException when I got outputBufferIndex >= 0. I have no idea if it was a coincidence that all devices did't work were android 4.2.2 with OMX.qcom.video.decoder.avc decoder.
for (int i = 0; i < videoExtractor.getTrackCount(); i++) {
MediaFormat mediaFormat = videoExtractor.getTrackFormat(i);
String mime = mediaFormat.getString(MediaFormat.KEY_MIME);
if (mime.startsWith("video/")) {
videoExtractor.selectTrack(i);
videoCodec = MediaCodec.createDecoderByType(mediaFormat.getString(MediaFormat.KEY_MIME));
videoCodec.configure(mediaFormat, null, null, 0);
videoCodec.start();
}
}
ByteBuffer[] videoInputBuffers = videoCodec.getInputBuffers();
while (true) {
int sampleTrackIndex = videoExtractor.getSampleTrackIndex();
if (sampleTrackIndex == -1) {
break;
} else { // decode video
int inputBufferIndex = videoCodec.dequeueInputBuffer(0);
if (inputBufferIndex >= 0) {
int bytesRead = videoExtractor.readSampleData(videoInputBuffers[inputBufferIndex], 0);
if (bytesRead >= 0) {
videoCodec.queueInputBuffer(inputBufferIndex, 0, bytesRead,
videoExtractor.getSampleTime(), 0);
videoExtractor.advance();
}
}
MediaCodec.BufferInfo videoBufferInfo = new MediaCodec.BufferInfo();
int outputBufferIndex = videoCodec.dequeueOutputBuffer(videoBufferInfo, 0);
if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat format = videoCodec.getOutputFormat();
Log.w("video format changed: " + videoCodec.getOutputFormat());
//do something...
break;
} else if (outputBufferIndex >= 0) {
//not supposed to happen!
}
}
}
Thank you very much for the clues and helps!

In Android 4.3, a collection of MediaCodec tests were added to CTS. If you look at the way doEncodeDecodeVideoFromBuffer() works in EncodeDecodeTest, you can see that it expects the INFO_OUTPUT_FORMAT_CHANGED result before any data. If it doesn't get it, the call to checkFrame() will fail when it tries to get the color format. Prior to Android 4.3, there were no tests, and any behavior is possible.
Having said that, I don't recall seeing this behavior on the (Qualcomm-based) Nexus 4.
At any rate, I'm not sure how much this will actually hold you back, unless you're able to decode the proprietary buffer layout Qualcomm uses. You can see in that same checkFrame() function that it punts when it sees OMX_QCOM_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka. Sending the output to a Surface may be a viable alternative depending on what you're up to.
Most of the MediaCodec code on bigflake and in Grafika targets API 18 (Android 4.3), because that's when the behavior became more predictable. (The availability of surface input and MediaMuxer is also of tremendous value.)