Develop Reference

Android - WebSocketClient Byte[] into audio

I have an android application that is receiving bytes[] data through a WebSocketClient. Those are the bytes i'm receiving:
#Override
public void onBinaryReceived(byte[] data) {
//System.out.println("onBinaryReceived");
Log.i("Binary: ", data.toString());
}
[B#91e31b9
[B#317b9fe
Those bytes are 3LAS mp3 voice data that are being streamed from an embedded system (Intel Edison). I'd like to decode those bytes into audio in my Android application. I've done some research and figured I should be using AudioTrack to do this. Sadly i'm pretty new to android so I have no idea where to start or how I should code that. If I understand the protocol correctly, I should concatenate thoses bytes continuously into a bigger byte that is being sent to the AudioTrack class.
First things first, should I be using AudioTrack to transform those bytes into audio ? Should I use the android MediaPlayer to have the audio played in my application ?
Honestly i'm pretty lost so any help / advice would be greatly appreciated. Don't hesitate to ask for more informations.
EDIT :
I finally went for AudioTrack and tried using it like such :
This is the initialization of my webSocket and my audio track.
public void createWebSocketClient() {
final URI uri;
int bufferSize = AudioTrack.getMinBufferSize(8000, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT);
final AudioTrack mAudioTrack = new AudioTrack(AudioManager.STREAM_MUSIC, 8000, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT, bufferSize, AudioTrack.MODE_STREAM);
mAudioTrack.play();
try {
uri = new URI("ws://10.0.0.1:9601/");
Log.i("Uri:", "Good");
} catch (URISyntaxException e) {
e.printStackTrace();
Log.i("Uri:", "Bad");
return;
}
webSocketClient = new WebSocketClient(uri) {
public void onBinaryReceived(byte[] data) {
//System.out.println("onBinaryReceived");
Log.i("Binary: ", String.valueOf(data));
mAudioTrack.write(data, 0, data.length);
}
}
With this i'm hearing something but it's not voice, more like a high pitch continuous noise. I'm supposed to be receiving MP3 and i'm using AudioFormat.ENCODING_PCM_16BIT, I guess the problem is here. Problem is, in the documentation there is nothing for ENCODING_MP3_*.
I just arrived to the conclusion that I had to decode MP3 into PCM. I'm trying to do so with JLayer but i'm having a lot of trouble. For a weird reason using JLayer causes an exception to my WebSocket and disconnects it :
11-15 15:18:27.506 11099-11712/com.example.jneb.myapplication I/System.out: Exceptionlength=433; index=433
Here is the code i'm using to decode the byte into PCM inside the onMessage webSocket function :
#Override
public void onBinaryReceived(byte[] data) {
System.out.println("onBinaryReceived");
// Should be decoding shit here //
try {
ByteArrayInputStream bis = new ByteArrayInputStream(data);
Bitstream bits = new Bitstream(bis);
Header frameHeader = bits.readFrame();
SampleBuffer buff = (SampleBuffer) decoder.decodeFrame(frameHeader, bits);
Log.i("LengthBuff: ", String.valueOf(buff.getBuffer().length));
mAudioTrack.write(buff.getBuffer(), 0, buff.getBuffer().length);
bits.closeFrame();
} catch (BitstreamException e) {
Log.i("Exception", " : Shit");
e.printStackTrace();
} catch (DecoderException e) {
Log.i("Exception", " : Shit shit shit");
e.printStackTrace();
}
}

Android MediaCodec flush after MediaExtractor seekTo and AudioTrack flush behaviour unexpectedly bad

I am using the combination of AudioTrack, MediaCodec and MediaExtractor to decode and play music.
As per the document,
In order to start decoding data that's not adjacent to previously submitted data (i.e. after a seek) it is necessary to flush() the decoder. Any input or output buffers the client may own at the point of the flush are immediately revoked, i.e. after a call to flush() the client does not own any buffers anymore.
I am calling flush after seek, so that i should also call mAudioTrack.flush();
Though flush is called, audioTrack plays some part of previously written data and continues from newly written data.
Without calling decoder.flush, possibly hearing noticeable glitches in audio playback. So how to achieve this instantaneous flush and continue playback of newly written data?
Code Snippet:
Updated code
do {
int codedbufferIndex = decoder.dequeueInputBuffer(1000);
if (codedbufferIndex >= 0) {
ByteBuffer codecInput = inputBuffers[codedbufferIndex];
synchronized (playerState) {
if (seek) {
extractor.seekTo(seekTo,
MediaExtractor.SEEK_TO_CLOSEST_SYNC);
// audioTrack.pause();
// audioTrack.stop();
// audioTrack.flush();
decoder.flush();
seek = false;
// audioTrack.play();
continue;
}
}
read = extractor.readSampleData(codecInput, offset);
if (read < 0) {
if (extractor.hasCacheReachedEndOfStream())
Log.e(TAG, "extractor.hasCacheReachedEndOfStream()");
break;
}
presentationTimeUs = extractor.getSampleTime();
decoder.queueInputBuffer(codedbufferIndex, offset, read,
presentationTimeUs, (read > 0) ? 0
: MediaCodec.BUFFER_FLAG_END_OF_STREAM);
int decodedDataBufIndex = decoder.dequeueOutputBuffer(info,
2000);
if (decodedDataBufIndex >= 0) {
ByteBuffer codecOutput = outputBuffers[decodedDataBufIndex];
byte[] atInput = new byte[info.size];
codecOutput.get(atInput);
codecOutput.clear();
decoder.releaseOutputBuffer(decodedDataBufIndex, false);
if(info.offset != 0){
Log.e(TAG,"info.offset = "+String.valueOf(info.offset));
}
audioTrack.write(atInput, /*0*/info.offset, info.size);
extractor.advance();
} else if (decodedDataBufIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat newFormat = decoder.getOutputFormat();
Log.e(TAG,
"newFormat: "
+ newFormat
.getString(MediaFormat.KEY_MIME));
Log.e(TAG,
"newFormat: "
+ String.valueOf(newFormat
.getInteger(MediaFormat.KEY_SAMPLE_RATE)));
Log.e(TAG, "Inside INFO_OUTPUT_FORMAT_CHANGED");
audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC,
sampleRate, AudioFormat.CHANNEL_OUT_STEREO,
AudioFormat.ENCODING_PCM_16BIT, 32768,
AudioTrack.MODE_STREAM);
audioTrack.play();
}
} else {
Log.e(TAG,"codedbufferIndex is "+String.valueOf(codedbufferIndex));
}
} while (read >= 0);

AudioTrack.flush() is no-op if not paused or stopped.
No-op if not stopped or paused, or if the track's creation mode is not
MODE_STREAM.
One solution would be to lower the buffer of the AudioTrack, so the playing of the previously submitted bytes is not perceptible. In my app, I use a rather large size of 32K and it isn't bothersome.

Android sound recorder bad performance

//constructor
android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO);
/////////////
//thread run() method
int N = AudioRecord.getMinBufferSize(8000,AudioFormat.CHANNEL_IN_MONO,AudioFormat.ENCODING_PCM_16BIT);
AudioRecord recorder = new AudioRecord(AudioSource.MIC, 8000, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, N*10);
recorder.startRecording();
while(!stopped)
{
try {
//if not paused upload audio
if (uploadAudio == true) {
short[][] buffers = new short[256][160];
int ix = 0;
//allocate buffer for audio data
short[] buffer = buffers[ix++ % buffers.length];
//write audio data to track
N = recorder.read(buffer,0,buffer.length);
//create bytes big enough to hold audio data
byte[] bytes2 = new byte[buffer.length * 2];
//convert audio data from short[][] to byte[]
ByteBuffer.wrap(bytes2).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().put(buffer);
//encode audio data for ulaw
read(bytes2, 0, bytes2.length);
See here for ulaw encoder code. Im using the read, maxAbsPcm and encode methods
//send audio data
//os.write(bytes2,0,bytes2.length);
}
} finally {
}
}
os.close();
}
catch(Throwable x)
{
Log.w("AudioWorker", "Error reading voice AudioWorker", x);
}
finally
{
recorder.stop();
recorder.release();
}
///////////
So this works ok. The audio is sent in the proper format to the server and played at the opposite end. However the audio skips often. Example: saying 1,2,3,4 will play back with the 4 cut off.
I believe it to be a performance issue because I have timed some of these methods and when they take 0 or less seconds everything works but they quite often take a couple seconds. With the converting of bytes and encoding taking the most.
Any idea how I can optimize this code to get better performance? Or maybe a way to deal with lag (possibly build a cache)?

Record data from mobile phone's audio jack

I am trying to record data from my mobile phone's audio interface. I used audiorecord function. Following is my code:
public void Initialize() {
buffersizebytes = AudioRecord.getMinBufferSize(SAMPPERSEC,channelConfiguration, audioEncoding); // 4096 on ion
buffer = new short[buffersizebytes];
buflen = buffersizebytes / 2;
audioRecord = new AudioRecord(
android.media.MediaRecorder.AudioSource.MIC, SAMPPERSEC,
channelConfiguration, audioEncoding, buffersizebytes);
acquire();
for(int i=0; i<4096; i++) buffer[i]=1;
}
public void acquire() {
try {
audioRecord.startRecording();
mSamplesRead = audioRecord.read(buffer, 0, buffersizebytes);
audioRecord.stop();
} catch (Throwable t) {
// Log.e("AudioRecord", "Recording Failed");
}
}
I want to put my acquired data into a buffer of 4096 bytes. But my program only put data into 1024 bytes. Also first 432 bytes also zeros. But I am sending data continuously. What could be the issue?

getMinBufferSize, as the name implies gives you the minimum buffer size. You can set anything bigger, including 4096.
As for the first samples after initialization, my phone gives two gigantic peaks that last for about 0.5 seconds, so I guess it is caused by the recorder starting up. Try skipping a few samples (let's say 500) before processing real data.
Furthermore, the size of buffer should be buffersizebytes/2.

AudioRecord and AudioTrack echo

I'm streaming the mic audio between two devices, everything is working but i have a bad echo.
Here what i'm doing
Reading thread
int sampleFreq = 22050;
int channelConfig = AudioFormat.CHANNEL_CONFIGURATION_MONO;
int audioFormat = AudioFormat.ENCODING_PCM_16BIT;
int minBuffer = 2*AudioTrack.getMinBufferSize(sampleFreq, channelConfig, audioFormat);
AudioTrack atrack = new AudioTrack(AudioManager.STREAM_MUSIC,
sampleFreq,
channelConfig,
audioFormat,
minBuffer,
AudioTrack.MODE_STREAM);
atrack.play();
byte[] buffer = new byte[minBuffer];
while (true) {
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
atrack.write(buffer, 0, buffer.length);
atrack.flush();
} catch (IOException e) {
Log.e(TAG, "disconnected", e);
break;
}
}
Here the recording thread
int sampleRate = 22050;
int channelMode = AudioFormat.CHANNEL_CONFIGURATION_MONO;
int audioFormat = AudioFormat.ENCODING_PCM_16BIT;
int buffersize = 2*AudioTrack.getMinBufferSize(sampleRate, channelMode, audioFormat);
AudioRecord arec = new AudioRecord(MediaRecorder.AudioSource.MIC,
sampleRate, channelMode,
AudioFormat.ENCODING_PCM_16BIT, buffersize);
buffer = new byte[buffersize];
arec.startRecording();
while (true) {
arec.read(buffer, 0, buffersize);
new Thread( new Runnable(){
#Override
public void run() {
try {
mOutputStream.write(buffer);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}).start();
}
Am I doing something wrong?

You need echo cancellation logic. Here is what I did on my Arm5 (WM8650) processor (Android 2.2) to remove the echo.
I wrapped Speex with JNI and called echo processing routines before sending PCM frames to encoder. No echo was canceled no matter what Speex settings I tried.
Because Speex is very sensitive to delay between playback and echo frames I implemented a queue and queued all packets sent to AudioTrack. The size of the queue should be roughly equal to the size of internal AudioTrack buffer. This way packet were sent to echo_playback roughly at the time when AudioTrack send packets to the sound card from its internal buffer. The delay was removed with this approach but echo was still not cancelled
I wrapped WebRtc echo cancellation part with JNI and called its methods before sending packets to encoder. The echo was still present but the library obviously was trying to cancel it.
I applied the buffer technique described in P2 and it finally started to work. The delay needs to be adjusted for each device though. Note also that WebRtc has mobile and full version of echo cancellation. The full version substantially slows the processor and should probably be run on ARM7 only. The mobile version works but with lower quality
I hope this will help someone.

Could be this:
bytes = mmInStream.read(buffer);
atrack.write(buffer, 0, buffer.length);
If the buffer remains full from previous call and the new one is not full (so bytes < buffer.length) you re-play hold part of track.