I am trying to recieve an streaming audio from my app.
below is my code for recieving audio stream:
public class ClientListen implements Runnable {
private Context context;
public ClientListen(Context context) {
this.context = context;
}
#Override
public void run() {
boolean run = true;
try {
DatagramSocket udpSocket = new DatagramSocket(8765);
InetAddress serverAddr = null;
try {
serverAddr = InetAddress.getByName("127.0.0.1");
} catch (UnknownHostException e) {
e.printStackTrace();
}
while (run) {
try {
byte[] message = new byte[8000];
DatagramPacket packet = new DatagramPacket(message,message.length);
Log.i("UDP client: ", "about to wait to receive");
udpSocket.setSoTimeout(10000);
udpSocket.receive(packet);
String text = new String(packet.getData(), 0, packet.getLength());
Log.d("Received text", text);
} catch (IOException e) {
Log.e(" UDP clien", "error: ", e);
run = false;
udpSocket.close();
}
}
} catch (SocketException e) {
Log.e("Socket Open:", "Error:", e);
} catch (IOException e) {
e.printStackTrace();
}
}
}
In Received text logger i can see data as coming as
D/Received text: �������n�����������q�9�$�0�/�G�{�������s�����JiH&������d�����Z���������d�����E������C�+
��l��y�����������v���9����������u��f�j�������$�����K���������F��~R�2�����T������������� �L�����!��G��8������s�;�"�,�R�����(��{�����*_��Z�������5������������\������x���j~������������/��=�����%�������
How can store this data into a wav file ?
What you see is the string representation of single udp packet after it was received and the received block has just being released.
It is a very small fraction of the sound you want to convert to wave.
Soon the while loop will continue and you will receive another packet and many more..
You need to collect all the packets in a buffer and then when you think it is ok - convert them to wave file.
Remember Wave is not just the sound bytes you get from udp but also 44 bytes of prefix you need to add to this file in order to be recognized by players.
Also if the udp is from another encoding format such as G711 - you must encode these bytes to PCM – if not you will hear heavy noise in the
Sound of the wave or the stream you play.
The buffer must be accurate. if it will be too big (many empty bytes in the end of the array) you will hear a sound of helicopter. if you know exactly what is the size of each packet then you can just write it to AudioTrack in order to play stream, or accumulate it and convert it to wave file when you will see fit. But If you are not sure about the size you can use this answer to get a buffer and then write the buffer to AudioTrack:
Android AudioRecord to Server over UDP Playback Issues.
they use Javax because it is very old answer but you just need to use AudioTrack instead in order to stream. It is not in this scope so I will just present the AudioTrack streaming replacements instead of Javax SourceDataLine:
final int SAMPLE_RATE = 8000; // Hertz
final int STREAM_TYPE = AudioManager.STREAM_NOTIFICATION;
int channelConfig = AudioFormat.CHANNEL_OUT_MONO;
int encodingFormat = AudioFormat.ENCODING_PCM_16BIT;
AudioTrack track = new AudioTrack(STREAM_TYPE, SAMPLE_RATE, channelConfig,
encodingFormat, BUF_SIZE, AudioTrack.MODE_STREAM);
track.play();
//.. then after receive UDP packets and the buffer is full:
if(track != null && packet != null){
track.write(audioStreamBuffer, 0, audioStreamBuffer.length);
}
You must not do this in the UI thread (I assume you know that).
In the code I will show you - I am getting udp of audio logs from PTT radio. It is encoded in G711 Ulaw . each packet is of 172 bytes exactly. First 12 bytes are for RTP and I need to offset (remove) them in order to eliminate small noises. rest 160 bytes are 20MS of sound.
I must decode the G711 Ulaw bytes to PCM shorts array. Then to take the short array and to make a wave file out of it. I am taking it after I see there was no packet receiving for more than one second (so I know the speech ended and the new block release is because of a new speech so I can take the old speech and make a wave file out of it). You can decide of a different buffer depends on what you are doing.
It works fine. After the decoding the sound of the wave is very good. If you have UDP with PCM so you don’t need to decode G711 - just skip this part.
Finally I want to mention I saw many old answers with code parts using javax.sound.sampled that seems great because it can convert easily an audio file or stream to wave format with AudioFileFormat
And also convert G711 to pcm with AudioFormat manipulations. But unfortunately it is not part of current java for android. We must count on android AudioTrack instead (and AudioRecord if we want to get the sound from the mic) but AudioTrack play only PCM and do not support G711 format – so when streaming G711 with AudioTrack the noise is terrible. We must decode it in our code before writing it to the track. Also we cannot convert to wave file using audioInputStream – I tried to do this easily with javax.sound.sampled jar file I added to my app but android keep giving me errors such as format not supported for wave, and mixer errors when try to stream – so I understood latest android cannot work with javax.sound.sampled and I went to look for law level decoding of G711 and law level creation of wave file out of the buffer of byte array received from the UDP packets .
A. in manifest add:
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.INTERNET"/>
B. in the worker thread:
#Override
public void run(){
Log.i(TAG, "ClientListen thread started. Thread id: " + Thread.currentThread().getId());
try{
udpSocket = new DatagramSocket(port);
}catch(SocketException e){
e.printStackTrace();
}
byte[] messageBuf = new byte[BUF_SIZE];
Log.i(TAG, "waiting to receive packet in port: " + port);
if(udpSocket != null){
// here you can create new AudioTrack and play.track
byte pttSession[] = null;
while (running){
packet = new DatagramPacket(messageBuf, 0, messageBuf.length);
Log.d(TAG, "inside while running loop");
try{
Log.d(TAG, "receive block: waiting for user to press on
speaker(listening now inside udpSocket for DatagramPacket..)");
//get inside receive block until packet will arrive through this socket
long timeBeforeBlock = System.currentTimeMillis();
udpSocket.receive(packet);
Log.d(TAG, "client received a packet, receive block stopped)");
//this is for sending msg handler to the UI tread (you may skip this)
sendState("getting UDP packets...");
/* if previous block release happened more than one second ago - so this
packet release is for a new speech. so let’s copy the previous speech
to a wave file and empty the speech */
if(System.currentTimeMillis() - timeBeforeBlock > 1000 && pttSession != null){
convertBytesToFile(pttSession);
pttSession = null;
}
/* let’s take the packet that was released and start new speech or add it to the ongoing speech. */
byte[] slice = Arrays.copyOfRange(packet.getData(), 12, packet.getLength());
if(null == pttSession){
pttSession = slice;
}else{
pttSession = concat(pttSession, slice);
Log.d(TAG, "pttSession:" + Arrays.toString(pttSession));
}
}catch(IOException e){
Log.e(TAG, "UDP client IOException - error: ", e);
running = false;
}
}
// let’s take the latest speech and make a last wave file out of it.
if(pttSession != null){
convertBytesToFile(pttSession);
pttSession = null;
}
// if running == false then stop listen.
udpSocket.close();
handler.sendEmptyMessage(MainActivity.UdpClientHandler.UPDATE_END);
}else{
sendState("cannot bind datagram socket to the specified port:" + port);
}
}
private void convertBytesToFile(byte[] byteArray){
//decode the bytes from G711U to PCM (outcome is a short array)
G711UCodec decoder = new G711UCodec();
int size = byteArray.length;
short[] shortArray = new short[size];
decoder.decode(shortArray, byteArray, size, 0);
String newFileName = "speech_" + System.currentTimeMillis() + ".wav";
//convert short array to wav (add 44 prefix shorts) and save it as a .wav file
Wave wave = new Wave(SAMPLE_RATE, (short) 1, shortArray, 0, shortArray.length - 1);
if(wave.writeToFile(Environment.getExternalStoragePublicDirectory
(Environment.DIRECTORY_DOWNLOADS),newFileName)){
Log.d(TAG, "wave.writeToFile successful!");
sendState("create file: "+ newFileName);
}else{
Log.w(TAG, "wave.writeToFile failed");
}
}
C. encoding/decoding G711 U-Law class:
taken from: https://github.com/thinktube-kobe/airtube/blob/master/JavaLibrary/src/com/thinktube/audio/G711UCodec.java
/**
* G.711 codec. This class provides u-law conversion.
*/
public class G711UCodec {
// s00000001wxyz...s000wxyz
// s0000001wxyza...s001wxyz
// s000001wxyzab...s010wxyz
// s00001wxyzabc...s011wxyz
// s0001wxyzabcd...s100wxyz
// s001wxyzabcde...s101wxyz
// s01wxyzabcdef...s110wxyz
// s1wxyzabcdefg...s111wxyz
private static byte[] table13to8 = new byte[8192];
private static short[] table8to16 = new short[256];
static {
// b13 --> b8
for (int p = 1, q = 0; p <= 0x80; p <<= 1, q += 0x10) {
for (int i = 0, j = (p << 4) - 0x10; i < 16; i++, j += p) {
int v = (i + q) ^ 0x7F;
byte value1 = (byte) v;
byte value2 = (byte) (v + 128);
for (int m = j, e = j + p; m < e; m++) {
table13to8[m] = value1;
table13to8[8191 - m] = value2;
}
}
}
// b8 --> b16
for (int q = 0; q <= 7; q++) {
for (int i = 0, m = (q << 4); i < 16; i++, m++) {
int v = (((i + 0x10) << q) - 0x10) << 3;
table8to16[m ^ 0x7F] = (short) v;
table8to16[(m ^ 0x7F) + 128] = (short) (65536 - v);
}
}
}
public int decode(short[] b16, byte[] b8, int count, int offset) {
for (int i = 0, j = offset; i < count; i++, j++) {
b16[i] = table8to16[b8[j] & 0xFF];
}
return count;
}
public int encode(short[] b16, int count, byte[] b8, int offset) {
for (int i = 0, j = offset; i < count; i++, j++) {
b8[j] = table13to8[(b16[i] >> 4) & 0x1FFF];
}
return count;
}
public int getSampleCount(int frameSize) {
return frameSize;
}
}
D. Converting to wave file:
Taken from here:
https://github.com/google/oboe/issues/320
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
public class Wave
{
private final int LONGINT = 4;
private final int SMALLINT = 2;
private final int INTEGER = 4;
private final int ID_STRING_SIZE = 4;
private final int WAV_RIFF_SIZE = LONGINT+ID_STRING_SIZE;
private final int WAV_FMT_SIZE = (4*SMALLINT)+(INTEGER*2)+LONGINT+ID_STRING_SIZE;
private final int WAV_DATA_SIZE = ID_STRING_SIZE+LONGINT;
private final int WAV_HDR_SIZE = WAV_RIFF_SIZE+ID_STRING_SIZE+WAV_FMT_SIZE+WAV_DATA_SIZE;
private final short PCM = 1;
private final int SAMPLE_SIZE = 2;
int cursor, nSamples;
byte[] output;
public Wave(int sampleRate, short nChannels, short[] data, int start, int end)
{
nSamples=end-start+1;
cursor=0;
output=new byte[nSamples*SMALLINT+WAV_HDR_SIZE];
buildHeader(sampleRate,nChannels);
writeData(data,start,end);
}
/*
by Udi for using byteArray directly
*/
public Wave(int sampleRate, short nChannels, byte[] data, int start, int end)
{
int size = data.length;
short[] shortArray = new short[size];
for (int index = 0; index < size; index++){
shortArray[index] = (short) data[index];
}
nSamples=end-start+1;
cursor=0;
output=new byte[nSamples*SMALLINT+WAV_HDR_SIZE];
buildHeader(sampleRate,nChannels);
writeData(shortArray,start,end);
}
// ------------------------------------------------------------
private void buildHeader(int sampleRate, short nChannels)
{
write("RIFF");
write(output.length);
write("WAVE");
writeFormat(sampleRate, nChannels);
}
// ------------------------------------------------------------
public void writeFormat(int sampleRate, short nChannels)
{
write("fmt ");
write(WAV_FMT_SIZE-WAV_DATA_SIZE);
write(PCM);
write(nChannels);
write(sampleRate);
write(nChannels * sampleRate * SAMPLE_SIZE);
write((short)(nChannels * SAMPLE_SIZE));
write((short)16);
}
// ------------------------------------------------------------
public void writeData(short[] data, int start, int end)
{
write("data");
write(nSamples*SMALLINT);
for(int i=start; i<=end; write(data[i++]));
}
// ------------------------------------------------------------
private void write(byte b)
{
output[cursor++]=b;
}
// ------------------------------------------------------------
private void write(String id)
{
if(id.length()!=ID_STRING_SIZE){
}
else {
for(int i=0; i<ID_STRING_SIZE; ++i) write((byte)id.charAt(i));
}
}
// ------------------------------------------------------------
private void write(int i)
{
write((byte) (i&0xFF)); i>>=8;
write((byte) (i&0xFF)); i>>=8;
write((byte) (i&0xFF)); i>>=8;
write((byte) (i&0xFF));
}
// ------------------------------------------------------------
private void write(short i)
{
write((byte) (i&0xFF)); i>>=8;
write((byte) (i&0xFF));
}
// ------------------------------------------------------------
public boolean writeToFile(File fileParent , String filename)
{
boolean ok=false;
try {
File path=new File(fileParent, filename);
FileOutputStream outFile = new FileOutputStream(path);
outFile.write(output);
outFile.close();
ok=true;
} catch (FileNotFoundException e) {
e.printStackTrace();
ok=false;
} catch (IOException e) {
ok=false;
e.printStackTrace();
}
return ok;
}
/**
* by Udi for test: write file with temp name so if you write many packets each packet will be written to a new file instead of deleting
* the previous file. (this is mainly for debug)
* #param fileParent
* #param filename
* #return
*/
public boolean writeToTmpFile(File fileParent , String filename)
{
boolean ok=false;
try {
File outputFile = File.createTempFile(filename, ".wav",fileParent);
FileOutputStream fileoutputstream = new FileOutputStream(outputFile);
fileoutputstream.write(output);
fileoutputstream.close();
ok=true;
} catch (FileNotFoundException e) {
e.printStackTrace();
ok=false;
} catch (IOException e) {
ok=false;
e.printStackTrace();
}
return ok;
}
}
Related
My app saves WAVE MONO files at 44100 and 16BIT PCM captured with AudioRecord using a RODE VideoMicro. Everything works just fine, except for the fact that I can see some very subtle "ticks" in the WAVE files uniformly spaced by 0.135 seconds. You can hear the ticks if your volume is very loud, or you can see them clearly using Audacity. Here is a sample WAVE file for you to analyze, and here is a screenshot of Audacity with 4 ticks in the soundfile:
Any idea what is happening?
I am not sure where those ticks come from, but I suspect that there is something strange in my code, since the tics are so uniformly spaced. But of course it could be either the phone, or the external microphone, or the interaction between the two. The BufferSize is 3528 and is retrieved with
static int BufferSize = AudioRecord.getMinBufferSize(SampleRate, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT);
This is the relevant part of my code:
...
recorder = new AudioRecord((denoise.isChecked()) ? MediaRecorder.AudioSource.VOICE_RECOGNITION : MediaRecorder.AudioSource.MIC,
SampleRate, AudioFormat.CHANNEL_IN_MONO, AudioEncoding, 2 * BufferSize); // 2 bytes in 16bit format
try {
recordingThread = new Thread(this::writeAudioDataToPCMFile, "AudioRecorder Thread");
recordingThread.start();
} catch (Exception e) {
e.printStackTrace();
}
}
}
private void writeAudioDataToPCMFile() {
String auxfile = AveActivity.REC_DIR + "/aux" + new Random().nextInt() + ".pcm";
short[] sData = new short[BufferSize];
int product;
int i, readSize;
int tooloud;
int tooloudlim = BufferSize / 50; // 2% clip
long mean;
FileOutputStream os;
try {
os = new FileOutputStream(auxfile);
} catch (FileNotFoundException e) {
runOnUiThread(() -> textsound.setText(r.getString(R.string.errstart)));
e.printStackTrace();
return;
}
waveformLastList = new ArrayList<>();
//int j=0;
while (isRecording) {
readSize = recorder.read(sData, 0, BufferSize);
tooloud = 0;
mean = 0;
for (i = 0; i < readSize; ++i) {
product = (int) (sData[i] * gain);
if (Math.abs(product) <= Short.MAX_VALUE) {
sData[i] = (short) product;
} else {
// Audio clipping!
sData[i] = (short) (Integer.signum(product) * Short.MAX_VALUE);
++tooloud;
}
mean += sData[i] * sData[i];
}
if (tooloud > tooloudlim)
runOnUiThread(this::tooLoudWarning);
try {
os.write(short2byte(sData), 0, 2 * BufferSize); // 2 bytes in 16bit format
} catch (IOException e) {
e.printStackTrace();
}
}
try {
os.close();
CharSequence time = DateFormat.format("yyMMdd-kkmmss", new Date(System.currentTimeMillis()));
song = birdname + "_" + time + ((denoise.isChecked()) ? "_NR" : "") + ((db == 0) ? "" : "_db" + db) + "." + songFormat;
runOnUiThread(() -> textsound.setText(r.getString(R.string.saving, songFormat)));
PCMtoWAV(new File(auxfile), new File(AveActivity.REC_DIR, song));
} catch (IOException e) {
runOnUiThread(() -> textsound.setText(r.getString(R.string.errsaving, songFormat)));
e.printStackTrace();
}
runOnUiThread(() -> {
activatebtns();
micColor(Color.WHITE);
});
}
private void PCMtoWAV(File input, File output) throws IOException {
// Bien rapido: 1/10 sec para cada minuto de gravacion
byte[] data = new byte[2 * BufferSize];
try (FileOutputStream outStream = new FileOutputStream(output)) {
FileInputStream inStream = new FileInputStream(input);
// Write WAVE header
outStream.write(buildWAVHeader(input.length()));
// Write audio data
while (inStream.read(data) != -1)
outStream.write(data);
// Write Metadata to LIST....INFO chunk
outStream.write(buildWAVMeta());
inStream.close();
outStream.close();
input.delete();
}
}
EDIT:
Following Uriel advice, I implemented a BlockingQueue like this:
public void RecordSong() {
if (isRecording) {
isRecording = false;
recorder.stop();
recorder.release();
recordingThread.interrupt();
writingThread.interrupt();
...
} else {
recorder = new AudioRecord(...);
try {
LinkedBlockingDeque<Byte> queue = new LinkedBlockingDeque<>();
recorder.startRecording();
recordingThread = new Thread(() -> captureAudioData(queue), "AudioRecorder Capture Thread");
writingThread = new Thread(() -> writeAudioDataToPCMFile(queue), "AudioRecorder Writing Thread");
recordingThread.setPriority(Thread.MAX_PRIORITY);
writingThread.setPriority(Thread.MAX_PRIORITY);
recordingThread.start();
writingThread.start();
...
}
}
private void captureAudioData(LinkedBlockingDeque<Byte> q) {
...
while (isRecording) {
capture,gain,etc...
try {
for (byte b : short2byte(sData)) q.put(b);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void writeAudioDataToPCMFile(LinkedBlockingDeque<Byte> q) {
String auxfile = AveActivity.REC_DIR + "/aux" + new Random().nextInt() + ".pcm";
FileOutputStream os;
try {
os = new FileOutputStream(auxfile);
} catch (FileNotFoundException e) {
e.printStackTrace();
return;
}
try {
while (true) os.write(q.take());
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
try {
os.close();
} catch (IOException e1) {
e.printStackTrace();
}
}
}
EDIT 2:
I tried the new approach (locked in the bathroom at 2am!). Unfortunately the problem persists. No change. It seems it is either a phone issue or a mic issue. In fact, removing the mic seems to get rid of the ticks, so probably it is a mic issue.
What you are experiencing is probably a pop sound that is called "popcorn". It usually happen when there is gaps in the wave file. what I suggest you to do is 2 things.
set the thread priority to the highest.
don't read and write on the same thread. use one thread to read, put it inside a queue, and on the second thread write the queue data into a file.
This is what i do in my app and there are no pops. LMK if it helped.
I'm trying to record audio and detect silence to stop recording and write file. Below is the code snippet already available here: Android audio capture silence detection
public class RecordAudio extends AsyncTask<Void, Double, Void> {
#Override
protected Void doInBackground(Void... arg0) {
Log.w(TAG, "doInBackground");
try {
String filename = getTempFilename();
try {
os = new FileOutputStream(filename);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
bufferSize = AudioRecord.getMinBufferSize(frequency,
channelConfiguration, audioEncoding);
AudioRecord audioRecord = new AudioRecord( MediaRecorder.AudioSource.MIC, frequency,
channelConfiguration, audioEncoding, bufferSize);
short[] buffer = new short[bufferSize];
audioRecord.startRecording();
while (started) {
int bufferReadResult = audioRecord.read(buffer, 0,bufferSize);
if(AudioRecord.ERROR_INVALID_OPERATION != bufferReadResult){
//check signal
//put a threshold
int foundPeak = searchThreshold(buffer,threshold);
if (foundPeak >- 1) { //found signal
//record signal
byte[] byteBuffer =ShortToByte(buffer,bufferReadResult);
try {
os.write(byteBuffer);
} catch (IOException e) {
e.printStackTrace();
}
} else {//count the time
//don't save signal
Log.d(TAG, "Silence...");
}
} else {
Toast.makeText(getBaseContext(), "Error!!!!", Toast.LENGTH_SHORT).show();
}
}
audioRecord.stop();
//close file
try {
os.close();
} catch (IOException e) {
e.printStackTrace();
}
copyWaveFile(getTempFilename(),getFilename());
deleteTempFile();
} catch (Throwable t) {
t.printStackTrace();
Log.e("AudioRecord", "Recording Failed");
}
return null;
}
byte [] ShortToByte(short [] input, int elements) {
int short_index, byte_index;
int iterations = elements; //input.length;
byte [] buffer = new byte[iterations * 2];
short_index = byte_index = 0;
for(/*NOP*/; short_index != iterations; /*NOP*/)
{
buffer[byte_index] = (byte) (input[short_index] & 0x00FF);
buffer[byte_index + 1] = (byte) ((input[short_index] & 0xFF00) >> 8);
++short_index; byte_index += 2;
}
return buffer;
}
int searchThreshold(short[]arr,short thr){
int peakIndex;
int arrLen=arr.length;
for (peakIndex=0;peakIndex<arrLen;peakIndex++){
if ((arr[peakIndex]>=thr) || (arr[peakIndex]<=-thr)){
Log.d(TAG, String.valueOf(arr[peakIndex]) + " - " + String.valueOf(peakIndex));
return peakIndex;
}
}
return -1; //not found
}
The problem is I'm not able to detect silence in the signal. Every time I read the buffer I get values greater than 32000 or less than -32000 (which I interpreted as maximum possible amplitudes for the signal, since the buffer is made up of short integer values), even when there is silence, so the threshold that determines what is "noise" and what is silence is always reached, and the searchThreshold function never returns -1.
Did anyone experience the same? I'm using a Samsung S5 device for tests.
Thanks in advance
In fact, values near 32000 or -32000 in buffer represents silence, and the values tends to zero when signal is detected. So I inverted the conditions inside searchThreshold function as below and I achieved what I was looking for.
int searchThreshold(short[]arr,short thr_upper, short thr_lower){
int peakIndex;
int arrLen=arr.length;
for (peakIndex=0;peakIndex<arrLen;peakIndex++){
if ((arr[peakIndex] <= thr_upper) && (arr[peakIndex] >= thr_lower )){
return peakIndex;
}
}
return -1; //not found
}
I'm developing a JAVA RTP Streaming App for a company project, which should be capable of joining the Multicast Server and receive the RTP Packets.Later I use the H264 Depacketizer to recreate the a complete frame from the NAL FU (Keep append the data until End Bit & Marker Bit set )
I want to decode and display a raw h264 video byte stream in Android and therefore I'm currently using the MediaCodec classes with Hardware Decoder configured.
The Application is Up and running for the Jeallybean (API 17). Various Resolutions which I need to decodes are :
480P at 30/60 FPS
720P/I at 30/60 FPS
1080P/I at 30/60 FPS
Recently, Due to System Upgrade we are porting the App to Android L Version 5.0.2. My App is not capable of playing the high resolutions videos like 720p#60fps and 1080p#60fps.
For the debugging purpose I started feeding the Elementary H264 Frames with size from the dump file to MediaCodec and found out the Video is Lagging.
There are timestamps on the sample video I used and it seems the actual time taken to proceed by 1 sec in Rendered Video is more
Below is my sample code and links to sample video
h264 video https://www.dropbox.com/s/cocjhhovihm8q25/dump60fps.h264?dl=0
h264 framesize https://www.dropbox.com/s/r146d5zederrne1/dump60fps.size?dl=0
Also as this is my question on stackoverflow, Please bear with me on Bad code formatting and Direct references.
public class MainActivity extends Activity {
static final String TAG = "MainActivity";
private PlayerThread mPlayer = null;
private static final String MIME_TYPE = "video/avc";
private byte[] mSPSPPSFrame = new byte [3000];
private byte[] sps = new byte[37];
File videoFile = null;
File videoFile1 = null;
TextView tv ;
FileInputStream videoFileStream = null;
FileInputStream videoFileStream1 = null;
int[] tall = null ;
SpeedControlCallback mspeed = new SpeedControlCallback();
int mStreamLen = 0;
FrameLayout game;
RelativeLayout rl ;
#Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
//mVideoSurfaceView = (SurfaceView)findViewById(R.id.videoSurfaceView);
setContentView(R.layout.activity_main);
SurfaceView first = (SurfaceView) findViewById(R.id.firstSurface);
first.getHolder().addCallback(new SurfaceHolder.Callback() {
#Override
public void surfaceCreated(SurfaceHolder surfaceHolder) {
Log.d(TAG, "First surface created!");
}
#Override
public void surfaceChanged(SurfaceHolder surfaceHolder, int i, int i2, int i3) {
Log.d(TAG, "surfaceChanged()");
surfaceHolder.getSurface();
if (mPlayer == null) {
mPlayer = new PlayerThread(surfaceHolder.getSurface());
mPlayer.start();
}
}
#Override
public void surfaceDestroyed(SurfaceHolder surfaceHolder) {
Log.d(TAG, "First surface destroyed!");
}
});
tv = (TextView) findViewById(R.id.textview);
videoFile = new File("/data/local/tmp/dump60fps.h264");
videoFile1 = new File("/data/local/tmp/dump60fps.size");
}
private class PlayerThread extends Thread {
private Surface surface;
public PlayerThread(Surface surface) {
this.surface = surface;
}
#Override
public void run() {
try {
decodeVideo(0, 1920,1080, 50, surface);
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (Throwable e) {
e.printStackTrace();
}
}
}
private void decodeVideo(int testinput, int width, int height,
int threshold, Surface surface) throws Throwable {
MediaCodec codec = null;
MediaFormat mFormat;
final long kTimeOutUs = 10000;
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
boolean sawInputEOS = false;
boolean sawOutputEOS = false;
MediaFormat oformat = null;
int errors = -1;
long presentationTimeUs = 0L;
boolean mVideoStart = false;
byte[] byteArray = new byte[65525*5*3];
int i;
int sizeInBytes = 0, index, sampleSize = 0;
try {
byte[] bytes = new byte[(int) videoFile1.length()];
FileInputStream fis = new FileInputStream(videoFile1);
fis.read(bytes);
fis.close();
String[] valueStr = new String(bytes).trim().split("\\s+");
tall = new int[valueStr.length];
mStreamLen = valueStr.length;
Log.e(TAG, "++++++ Total Frames ++++++"+mStreamLen);
for ( i = 0; i < valueStr.length; i++) {
tall[i] = Integer.parseInt(valueStr[i]);
}
} catch (IOException e1) {
e1.printStackTrace();
}
index =1;
try {
videoFileStream = new FileInputStream(videoFile);
} catch (FileNotFoundException e1) {
e1.printStackTrace();
}
System.currentTimeMillis();
if (mVideoStart == false) {
try {
sizeInBytes = videoFileStream.read(mSPSPPSFrame, 0,37);
Log.e(TAG, "VideoEngine configure ."+sizeInBytes);
//for (i = 0 ; i < sizeInBytes; i++){
// Log.e(TAG, "VideoEngine ."+mSPSPPSFrame[i]);}
} catch (IOException e1) {
e1.printStackTrace();
}
sampleSize = sizeInBytes;
index++;
index++;
mFormat = MediaFormat.createVideoFormat(MIME_TYPE, 1920,1080);
mFormat.setByteBuffer("csd-0", ByteBuffer.wrap( mSPSPPSFrame,0, sizeInBytes));
codec = MediaCodec.createDecoderByType(MIME_TYPE);
codec.configure(mFormat, surface /*surface*/ , null /* crypto */, 0 /* flags */);
codec.start();
codec.getInputBuffers();
codec.getOutputBuffers();
}
// index = 0;
while (!sawOutputEOS && errors < 0) {
if (!sawInputEOS) {
int inputBufIndex = codec.dequeueInputBuffer(kTimeOutUs);
//Log.d(TAG, String.format("Archana Dqing the input buffer with BufIndex #: %d",inputBufIndex));
if (inputBufIndex >= 0) {
ByteBuffer dstBuf = codec.getInputBuffers()[inputBufIndex];
/*
* Read data from file and copy to the input ByteBuffer
*/
try {
sizeInBytes = videoFileStream.read(byteArray, 0,
tall[index] /*+ 4*/);
sampleSize = tall[index]/*+ 4*/;
index++;
} catch (IOException e) {
e.printStackTrace();
}
if (sizeInBytes <= 0) {
codec.queueInputBuffer(
inputBufIndex,
0 /* offset */,
0,
presentationTimeUs,
MediaCodec.BUFFER_FLAG_END_OF_STREAM );
sawInputEOS = true;
}
else {
dstBuf.put(byteArray, 0, sizeInBytes);
if (mVideoStart == false) mVideoStart = true;
codec.queueInputBuffer(
inputBufIndex,
0 /* offset */,
sampleSize,
presentationTimeUs,
mVideoStart ? 0:MediaCodec.BUFFER_FLAG_CODEC_CONFIG );
//Log.d(TAG, String.format(" After queueing the buffer to decoder with inputbufindex and samplesize #: %d ,%d ind %d",inputBufIndex,sampleSize,index));
}
}
}
int res = codec.dequeueOutputBuffer(info, kTimeOutUs);
//Log.d(TAG, String.format(" Getting the information about decoded output buffer flags,offset,PT,size #: %d %d %d %d",info.flags,info.offset,info.presentationTimeUs,info.size));
//Log.d(TAG, String.format(" Getting the output of decoder in res #: %d",res));
if (res >= 0) {
int outputBufIndex = res;
//Log.d(TAG, "Output PTS "+info.presentationTimeUs);
//mspeed.preRender(info.presentationTimeUs);
//mspeed.setFixedPlaybackRate(25);
codec.releaseOutputBuffer(outputBufIndex, true /* render */);
//Log.d(TAG, String.format(" releaseoutputbuffer index= #: %d",outputBufIndex));
if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
Log.d(TAG, "saw output EOS.");
sawOutputEOS = true;
}
} else if (res == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
codec.getOutputBuffers();
Log.d(TAG, "output buffers have changed.");
} else if (res == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
oformat = codec.getOutputFormat();
Log.d(TAG, "output format has changed to " + oformat);
}
}
codec.stop();
codec.release();
this.finish();
}
#Override
public boolean onCreateOptionsMenu(Menu menu) {
getMenuInflater().inflate(R.menu.activity_main, menu);
return true;
}
}
There are couples of workaround to problem with the above sample test.
Instead of feeding One Full frame to the decoder Inout, I was feeding single of NAL Units at a time. But still the playback was slow and could not match 60FPS
Google has changed the Implementation of Surface BufferQueue from Asynchronous to Synchronous.Hence when we call MediaCodec.dequeueBuffer to get decoded data, the server side (SurfaceTexture::dequeueBuffer) will wait for a buffer to be queued, and the client side waits for that, so that SurfaceTextureClient::dequeueBuffer will not return until a buffer has actually been queued on the server side. Where as in the Asynchronous Mode, a new GraphicBuffer is allocated.
Displaying h264 video from an mpegts stream over udp:// on android.
I've been trying for a few days to get this to work with no success. What I have is a device that produces a h264 video stream that it multicasts over in an mpegts container over raw udp (not rtp). I'm trying to get this to display in a custom app on android.
I read that android's built in MediaPlayer supports both h264 (avc) and mpegts, but that it does not handle udp:// streams, so I cannot use that (which would be by far the simplest). Instead, I have tried to manually parse the mpegts stream into an elementary stream and pass that to a MediaCodec that's been passed the surface of a SurfaceView. No matter what I seem to try, two things always happen (once I fix exceptions, etc):
The SurfaceView is always black.
The MediaCodec always accepts about 6-9 buffers and then dequeueInputBuffer just starts instantly failing (returning -1) and I cannot queue anything else.
I can split the mpeg stream into TS packets and then join their payloads into PES packets. I've tried passing full PES packets (minus the PES header) into MediaCodec.
I've also tried splitting the PES packets into individual NAL units by splitting on \x00\x00\x01 and passing them individually into the MediaCodec.
I've also tried holding off on passing in NAL unit until I've received the SPS NAL unit and passing that first with BUFFER_FLAG_CODEC_CONFIG.
All of these result in the same thing mentiond above. I am out of ideas about what to try, so any help would be greatly appreciated.
Some points I'm still not sure about:
Nearly all the examples I've seen get the MediaFormat from MediaExtractor, which I can't use on the stream. The few that don't use MediaExtractor explicity set csd-0 and csd-1 from bytestrings that aren't explained. I read that SPS packet can be put in the buffer instead so that's what I tried.
I'm not sure what to pass into presentationTimeUs. The TS packets have a PCR and the PES packets have a PTS, but I don't know what's expected by the api and how these relate.
I'm not sure how the data needs to be passed into MediaCodec (is this why it stops giving me buffers?). I got the idea of passing in individual NAL units from this so post:
Decoding Raw H264 stream in android?
other references I used to make this example:
MPEG-TS Format
PES Format
PES Format
code (sorry it's pretty long):
I just created a test app from the basic template in AndroidStudio, most of it is boilerplate so I'll just paste the video related stuff.
SurfaceView is defined in the xml, so grab it and get the surface when it's created/changed
public class VideoPlayer extends Activity implements SurfaceHolder.Callback {
private static final String TAG = VideoPlayer.class.getName();
PlayerThread playerThread;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_video_player);
SurfaceView view = (SurfaceView) findViewById(R.id.surface);
view.getHolder().addCallback(this);
}
...
#Override
public void surfaceCreated(SurfaceHolder surfaceHolder) {
Log.d(TAG,"surfaceCreated");
}
#Override
public void surfaceChanged(SurfaceHolder surfaceHolder, int i, int i2, int i3) {
Log.d("main","surfaceChanged");
if( playerThread == null ) {
playerThread = new PlayerThread(surfaceHolder.getSurface());
playerThread.start();
}
}
...
PlayerThread is an internal class that reads data from a multicast port and passes it to a parsing function on a background thread:
class PlayerThread extends Thread {
private final String TAG = PlayerThread.class.getName();
MediaExtractor extractor;
MediaCodec decoder;
Surface surface;
boolean running;
ByteBuffer[] inputBuffers;
public PlayerThread(Surface surface)
{
this.surface = surface;
MediaFormat format = MediaFormat.createVideoFormat("video/avc",720,480);
decoder = MediaCodec.createDecoderByType("video/avc");
decoder.configure(format, surface, null, 0);
decoder.start();
inputBuffers = decoder.getInputBuffers();
}
...
#Override
public void run() {
running = true;
try {
String mcg = "239.255.0.1";
MulticastSocket ms;
ms = new MulticastSocket(1841);
ms.joinGroup(new InetSocketAddress(mcg, 1841), NetworkInterface.getByName("eth0"));
ms.setSoTimeout(4000);
ms.setReuseAddress(true);
byte[] buffer = new byte[65535];
DatagramPacket dp = new DatagramPacket(buffer, buffer.length);
while (running) {
try {
ms.receive(dp);
parse(dp.getData());
} catch (SocketTimeoutException e) {
Log.d("thread", "timeout");
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
The receiving works fine, each datagram packet contains two TS packets. They get passed to the parse function:
boolean first = true;
ByteArrayOutputStream current = new ByteArrayOutputStream();
void parse(byte[] data) {
ByteBuffer stream = ByteBuffer.wrap(data);
// mpeg-ts stream header is 4 bytes starting with the sync byte
if( stream.get(0) != 0x47 ) {
Log.w(TAG, "got packet w/out mpegts header!");
return;
}
ByteBuffer raw = stream.duplicate();
// ts packets are 188 bytes
raw.limit(188);
TSPacket ts = new TSPacket(raw);
if( ts.pid == 0x10 ) {
processTS(ts);
}
// move to second packet
stream.position(188);
stream.limit(188*2);
if( stream.get(stream.position()) != 0x47 ) {
Log.w(TAG, "missing mpegts header!");
return;
}
raw = stream.duplicate();
raw.limit(188*2);
ts = new TSPacket(raw);
if( ts.pid == 0x10 ) {
processTS(ts);
}
}
TS packets are parsed by the TSPacket class:
public class TSPacket {
private final static String TAG = TSPacket.class.getName();
class AdaptationField {
boolean di;
boolean rai;
boolean espi;
boolean hasPcr;
boolean hasOpcr;
boolean spf;
boolean tpdf;
boolean hasExtension;
byte[] data;
public AdaptationField(ByteBuffer raw) {
// first byte is size of field minus size byte
int count = raw.get() & 0xff;
// second byte is flags
BitSet flags = BitSet.valueOf(new byte[]{ raw.get()});
di = flags.get(7);
rai = flags.get(6);
espi = flags.get(5);
hasPcr = flags.get(4);
hasOpcr = flags.get(3);
spf = flags.get(2);
tpdf = flags.get(1);
hasExtension = flags.get(0);
// the rest is 'data'
if( count > 1 ) {
data = new byte[count-1];
raw.get(data);
}
}
}
boolean tei;
boolean pus;
boolean tp;
int pid;
boolean hasAdapt;
boolean hasPayload;
int counter;
AdaptationField adaptationField;
byte[] payload;
public TSPacket(ByteBuffer raw) {
// check for sync byte
if( raw.get() != 0x47 ) {
Log.e(TAG, "missing sync byte");
throw new InvalidParameterException("missing sync byte");
}
// next 3 bits are flags
byte b = raw.get();
BitSet flags = BitSet.valueOf(new byte[] {b});
tei = flags.get(7);
pus = flags.get(6);
tp = flags.get(5);
// then 13 bits for pid
pid = ((b << 8) | (raw.get() & 0xff) ) & 0x1fff;
b = raw.get();
flags = BitSet.valueOf(new byte[]{b});
// then 4 more flags
if( flags.get(7) || flags.get(6) ) {
Log.e(TAG, "scrambled?!?!");
// todo: bail on this packet?
}
hasAdapt = flags.get(5);
hasPayload = flags.get(4);
// counter
counter = b & 0x0f;
// optional adaptation field
if( hasAdapt ) {
adaptationField = new AdaptationField(raw);
}
// optional payload field
if( hasPayload ) {
payload = new byte[raw.remaining()];
raw.get(payload);
}
}
}
Then passed to the processTS function:
// a PES packet can span multiple TS packets, so we keep track of the 'current' one
PESPacket currentPES;
void processTS(TSPacket ts) {
// payload unit start?
if( ts.pus ) {
if( currentPES != null ) {
Log.d(TAG,String.format("replacing pes: len=%d,size=%d", currentPES.length, currentPES.data.size()));
}
// start of new PES packet
currentPES = new PESPacket(ts);
} else if (currentPES != null ) {
// continued PES
currentPES.Add(ts);
} else {
// haven't got a start pes yet
return;
}
if( currentPES.isFull() ) {
long pts = currentPES.getPts();
byte[] data = currentPES.data.toByteArray();
int idx = 0;
do {
int sidx = idx;
// find next NAL prefix
idx = Utility.indexOf(data, sidx+4, data.length-(sidx+4), new byte[]{0,0,1});
byte[] NAL;
if( idx >= 0 ) {
NAL = Arrays.copyOfRange(data, sidx, idx);
} else {
NAL = Arrays.copyOfRange(data, sidx, data.length);
}
// send SPS NAL before anything else
if( first ) {
byte type = NAL[3] == 0 ? NAL[4] : NAL[3];
if( (type & 0x1f) == 7 ) {
Log.d(TAG, "found sps!");
int ibs = decoder.dequeueInputBuffer(1000);
if (ibs >= 0) {
ByteBuffer sinput = inputBuffers[ibs];
sinput.clear();
sinput.put(NAL);
decoder.queueInputBuffer(ibs, 0, NAL.length, 0, MediaCodec.BUFFER_FLAG_CODEC_CONFIG);
Log.d(TAG, "sent sps");
first = false;
} else
Log.d(TAG, String.format("could not send sps! %d", ibs));
}
} else {
// put in decoder?
int ibs = decoder.dequeueInputBuffer(1000);
if (ibs >= 0) {
ByteBuffer sinput = inputBuffers[ibs];
sinput.clear();
sinput.put(NAL);
decoder.queueInputBuffer(ibs, 0, NAL.length, 0, 0);
Log.d(TAG, "buffa");
}
}
} while( idx >= 0 );
// finished with this pes
currentPES = null;
}
}
PES packets are parsed by the PESPacket class:
public class PESPacket {
private final static String TAG = PESPacket.class.getName();
int id;
int length;
boolean priority;
boolean dai;
boolean copyright;
boolean origOrCopy;
boolean hasPts;
boolean hasDts;
boolean hasEscr;
boolean hasEsRate;
boolean dsmtmf;
boolean acif;
boolean hasCrc;
boolean pesef;
int headerDataLength;
byte[] headerData;
ByteArrayOutputStream data = new ByteArrayOutputStream();
public PESPacket(TSPacket ts) {
if( ts == null || !ts.pus) {
Log.e(TAG, "invalid ts passed in");
throw new InvalidParameterException("invalid ts passed in");
}
ByteBuffer pes = ByteBuffer.wrap(ts.payload);
// start code
if( pes.get() != 0 || pes.get() != 0 || pes.get() != 1 ) {
Log.e(TAG, "invalid start code");
throw new InvalidParameterException("invalid start code");
}
// stream id
id = pes.get() & 0xff;
// packet length
length = pes.getShort() & 0xffff;
// this is supposedly allowed for video
if( length == 0 ) {
Log.w(TAG, "got zero-length PES?");
}
if( id != 0xe0 ) {
Log.e(TAG, String.format("unexpected stream id: 0x%x", id));
// todo: ?
}
// for 0xe0 there is an extension header starting with 2 bits '10'
byte b = pes.get();
if( (b & 0x30) != 0 ) {
Log.w(TAG, "scrambled ?!?!");
// todo: ?
}
BitSet flags = BitSet.valueOf(new byte[]{b});
priority = flags.get(3);
dai = flags.get(2);
copyright = flags.get(1);
origOrCopy = flags.get(0);
flags = BitSet.valueOf(new byte[]{pes.get()});
hasPts = flags.get(7);
hasDts = flags.get(6);
hasEscr = flags.get(5);
hasEsRate = flags.get(4);
dsmtmf = flags.get(3);
acif = flags.get(2);
hasCrc = flags.get(1);
pesef = flags.get(0);
headerDataLength = pes.get() & 0xff;
if( headerDataLength > 0 ) {
headerData = new byte[headerDataLength];
pes.get(headerData);
}
WritableByteChannel channel = Channels.newChannel(data);
try {
channel.write(pes);
} catch (IOException e) {
e.printStackTrace();
}
// length includes optional pes header,
length = length - (3 + headerDataLength);
}
public void Add(TSPacket ts) {
if( ts.pus ) {
Log.e(TAG, "don't add start of PES packet to another packet");
throw new InvalidParameterException("ts packet marked as new pes");
}
int size = data.size();
int len = length - size;
len = ts.payload.length > len ? len : ts.payload.length;
data.write(ts.payload, 0, len);
}
public boolean isFull() {
return (data.size() >= length );
}
public long getPts() {
if( !hasPts || headerDataLength < 5 )
return 0;
ByteBuffer hd = ByteBuffer.wrap(headerData);
long pts = ( ((hd.get() & 0x0e) << 29)
| ((hd.get() & 0xff) << 22)
| ((hd.get() & 0xfe) << 14)
| ((hd.get() & 0xff) << 7)
| ((hd.get() & 0xfe) >>> 1));
return pts;
}
}
So I eventually figured out that, even though I was using an output surface, I had to manually drain the output buffers. By calling decoder.dequeueOutputBuffer and then decoder.releaseOutputBuffer, the input buffers worked as expected.
I was able to also able to get output by passing in both individual NAL units as well as full access units (one per PES packet), but I got the clearest video by passing in full access units.
I believe the major problem comes from the read method in MicrophoneManager but cant see where the problem is. My console output for bytesread is 0 (this is in the other class AudioTransmitter). It seems to me like its not streaming audio data since the data sent off is none changing and like I said bytes read is 0.
public class MicrophoneManager{
// private TargetDataLine targetDataLine;
private float sampleRate = 8000.0F; //8000,11025,16000,22050,44100
private int sampleSizeInBits = 16; //8,16
private int channels = 1; //1,2
private boolean signed = true; //true,false
private boolean bigEndian = false; //true,false
private AudioFormat audioFormat;
// private AudioRecord audioRecord;
// private AudioInputStream ais;
private static MicrophoneManager singletonMicrophoneManager = null;
public AudioRecord audioRecord;
public int mSamplesRead; //how many samples read
public int buffersizebytes;
public int buflen;
public int channelConfiguration = AudioFormat.CHANNEL_CONFIGURATION_MONO;
public int audioEncoding = AudioFormat.ENCODING_PCM_16BIT;
public static byte[] buffer; //+-32767
public static final int SAMPPERSEC = 8000; //samp per sec 8000, 11025, 22050 44100 or 48000
public class MicrophoneManager{
// private TargetDataLine targetDataLine;
private float sampleRate = 8000.0F; //8000,11025,16000,22050,44100
private int sampleSizeInBits = 16; //8,16
private int channels = 1; //1,2
private boolean signed = true; //true,false
private boolean bigEndian = false; //true,false
private AudioFormat audioFormat;
// private AudioRecord audioRecord;
// private AudioInputStream ais;
private static MicrophoneManager singletonMicrophoneManager = null;
public AudioRecord audioRecord;
public int mSamplesRead; //how many samples read
public int buffersizebytes;
public int buflen;
public int channelConfiguration = AudioFormat.CHANNEL_CONFIGURATION_MONO;
public int audioEncoding = AudioFormat.ENCODING_PCM_16BIT;
public static short[] buffer; //+-32767
public static final int SAMPPERSEC = 8000; //samp per sec 8000, 11025, 22050 44100 or 48000
public MicrophoneManager() {
System.out.println("Initializing");
// audioFormat = new AudioFormat(sampleRate,sampleSizeInBits,channels,signed,bigEndian);
// audioRecord = new AudioRecord(MediaRecorder.AudioSource.DEFAULT, (int) sampleRate, channels, AudioFormat.ENCODING_PCM_16BIT, buffersizebytes);
buffersizebytes = AudioRecord.getMinBufferSize(SAMPPERSEC,channelConfiguration,audioEncoding); //4096 on ion
buffer = new short[buffersizebytes];
buflen=buffersizebytes/2;
audioRecord = new AudioRecord(android.media.MediaRecorder.AudioSource.MIC,SAMPPERSEC,
channelConfiguration,audioEncoding,buffersizebytes); //constructor
}
public static MicrophoneManager getMicrophoneManager() throws Exception {
if (singletonMicrophoneManager == null) {
singletonMicrophoneManager = new MicrophoneManager();
singletonMicrophoneManager.initialize();
}
return singletonMicrophoneManager;
}
public void initialize() throws Exception {
}
public void startAudioInput(){
try {
audioRecord.startRecording();
mSamplesRead = audioRecord.read(buffer, 0, buffer.length);
audioRecord.stop();
} catch (Throwable t) {
// Log.e("AudioRecord", "Recording Failed");
System.out.println("Error Starting audio input"+t);
}
}
public void stopAudioInput(){
audioRecord.stop();
System.out.println("Stopping audio input");
}
public void finishAudioInput(){
audioRecord.release();
System.out.println("Finishing audio input");
}
public boolean available() throws Exception {
return true;
}
public int read(byte[] inBuf) throws Exception {
return audioRecord.read(inBuf,0,inBuf.length);
}
}
AudioTransmitter:
public class AudioTransmitter extends Thread{
private MicrophoneManager mm=null;
private boolean transmittingAudio = false;
private String host;
private int port;
private long id=0;
boolean run=true;
public AudioTransmitter(String host, int port, long id) {
this.host = host;
this.port = port;
this.id = id;
this.start();
}
public void run() {
System.out.println("creating audio transmitter host "+host+" port "+port+" id "+id);
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(
java.security.cert.X509Certificate[] chain, String authType) {
for (int j=0; j<chain.length; j++)
{
System.out.println("Client certificate information:");
System.out.println(" Subject DN: " + chain[j].getSubjectDN());
System.out.println(" Issuer DN: " + chain[j].getIssuerDN());
System.out.println(" Serial number: " + chain[j].getSerialNumber());
System.out.println("");
}
}
}
};
while (run) {
if(transmittingAudio) {
try {
if(mm==null) {
mm = new MicrophoneManager();
// mm.initialize();
}
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
SSLSocketFactory sslFact = sc.getSocketFactory();
SSLSocket socket = (SSLSocket)sslFact.createSocket(host, port);
socket.setSoTimeout(10000);
InputStream inputStream = socket.getInputStream();
DataInputStream in = new DataInputStream(new BufferedInputStream(inputStream));
OutputStream outputStream = socket.getOutputStream();
DataOutputStream os = new DataOutputStream(new BufferedOutputStream(outputStream));
PrintWriter socketPrinter = new PrintWriter(os);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
// socketPrinter.println("POST /transmitaudio?patient=1333369798370 HTTP/1.0");
socketPrinter.println("POST /transmitaudio?id="+id+" HTTP/1.0");
socketPrinter.println("Content-Type: audio/basic");
socketPrinter.println("Content-Length: 99999");
socketPrinter.println("Connection: Keep-Alive");
socketPrinter.println("Cache-Control: no-cache");
socketPrinter.println();
socketPrinter.flush();
// in.read();
mm.startAudioInput();
int buffersizebytes = AudioRecord.getMinBufferSize(8000, AudioFormat.CHANNEL_CONFIGURATION_MONO,AudioFormat.ENCODING_PCM_16BIT); //4096 on ion
System.out.println("audio started");
byte[] data = new byte[buffersizebytes];
while(transmittingAudio) {
// byte[] data = new byte[mm.available()];
int bytesRead = mm.read(data);
os.write(data,0,bytesRead);
os.flush();
// ca.transmitAxisAudioPacket(data);
// System.out.println("read "+data);
System.out.println("bytesRead "+bytesRead);
System.out.println("data "+Arrays.toString(data));
}
os.close();
mm.stopAudioInput();
} catch (Exception e) {
System.out.println("excpetion while transmitting audio connection will be closed"+e);
transmittingAudio=false;
}
}
else {
try {
Thread.sleep(1000);
} catch (Exception e){
System.out.println("exception while thread sleeping"+e);}
}
}
}
public void setTransmittingAudio(boolean transmittingAudio) {
this.transmittingAudio = transmittingAudio;
}
public void finished() {
this.transmittingAudio = false;
mm.finishAudioInput();
}
}
You are calling
mSamplesRead = audioRecord.read(buffer, 0, buffersizebytes);
There are a few problems with that.
audioRecord.read() wants the length of the array you are reading in to, not the size in bytes. You have set your encoding to 16 bit. You should really be doing something like:
new short[] buffer = short[1024]; // or whatever length you like
mSampleRead = audioRecord.read(buffer,0,buffer.length);
You are calling to read buffersizebytes but you set buffer = new byte[1024];. No particular reason to think buffersizebytes is the right number is there? You want a short[] array with 16 bit encoding, and you want the number of SAMPLES (not bytes) you read to be less than or equal to the length of that short[] buffer.
Also, you will be in better shape if you print out the exception you get when they are thrown, change
System.out.println("Error Starting audio input");
to
System.out.println("Error Starting audio input" + t);
and you will at least have a hint why android is throwing you in the dustbin.
Thank you guys for you help however I found a better way to do whats needed (here). Here is the code. Notice this also uses http://www.devdaily.com/java/jwarehouse/android/core/java/android/speech/srec/UlawEncoderInputStream.java.shtml to convert the audio to ulaw
public class AudioWorker extends Thread
{
private boolean stopped = false;
private String host;
private int port;
private long id=0;
boolean run=true;
AudioRecord recorder;
//ulaw encoder stuff
private final static String TAG = "UlawEncoderInputStream";
private final static int MAX_ULAW = 8192;
private final static int SCALE_BITS = 16;
private InputStream mIn;
private int mMax = 0;
private final byte[] mBuf = new byte[1024];
private int mBufCount = 0; // should be 0 or 1
private final byte[] mOneByte = new byte[1];
////
/**
* Give the thread high priority so that it's not canceled unexpectedly, and start it
*/
public AudioWorker(String host, int port, long id)
{
this.host = host;
this.port = port;
this.id = id;
android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_URGENT_AUDIO);
// start();
}
#Override
public void run()
{
Log.i("AudioWorker", "Running AudioWorker Thread");
recorder = null;
AudioTrack track = null;
short[][] buffers = new short[256][160];
int ix = 0;
/*
* Initialize buffer to hold continuously recorded AudioWorker data, start recording, and start
* playback.
*/
try
{
int N = AudioRecord.getMinBufferSize(8000,AudioFormat.CHANNEL_IN_MONO,AudioFormat.ENCODING_PCM_16BIT);
recorder = new AudioRecord(AudioSource.MIC, 8000, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, N*10);
track = new AudioTrack(AudioManager.STREAM_MUSIC, 8000,
AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT, N*10, AudioTrack.MODE_STREAM);
recorder.startRecording();
track.play();
/*
* Loops until something outside of this thread stops it.
* Reads the data from the recorder and writes it to the AudioWorker track for playback.
*/
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
SSLSocketFactory sslFact = sc.getSocketFactory();
SSLSocket socket = (SSLSocket)sslFact.createSocket(host, port);
socket.setSoTimeout(10000);
InputStream inputStream = socket.getInputStream();
DataInputStream in = new DataInputStream(new BufferedInputStream(inputStream));
OutputStream outputStream = socket.getOutputStream();
DataOutputStream os = new DataOutputStream(new BufferedOutputStream(outputStream));
PrintWriter socketPrinter = new PrintWriter(os);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
// socketPrinter.println("POST /transmitaudio?patient=1333369798370 HTTP/1.0");
socketPrinter.println("POST /transmitaudio?id="+id+" HTTP/1.0");
socketPrinter.println("Content-Type: AudioWorker/basic");
socketPrinter.println("Content-Length: 99999");
socketPrinter.println("Connection: Keep-Alive");
socketPrinter.println("Cache-Control: no-cache");
socketPrinter.println();
socketPrinter.flush();
while(!stopped)
{
Log.i("Map", "Writing new data to buffer");
short[] buffer = buffers[ix++ % buffers.length];
N = recorder.read(buffer,0,buffer.length);
track.write(buffer, 0, buffer.length);
byte[] bytes2 = new byte[buffer.length * 2];
ByteBuffer.wrap(bytes2).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().put(buffer);
read(bytes2, 0, bytes2.length);
// os.write(bytes2,0,bytes2.length);
os.write(bytes2,0,bytes2.length);
System.out.println("bytesRead "+buffer.length);
System.out.println("data "+Arrays.toString(buffer));
}
os.close();
}
catch(Throwable x)
{
Log.w("AudioWorker", "Error reading voice AudioWorker", x);
}
/*
* Frees the thread's resources after the loop completes so that it can be run again
*/
finally
{
recorder.stop();
recorder.release();
track.stop();
track.release();
}
}
/**
* Called from outside of the thread in order to stop the recording/playback loop
*/
/**
* Called from outside of the thread in order to stop the recording/playback loop
*/
public void close()
{
stopped = true;
}
public void resumeThread()
{
stopped = false;
run();
}
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(
java.security.cert.X509Certificate[] chain, String authType) {
for (int j=0; j<chain.length; j++)
{
System.out.println("Client certificate information:");
System.out.println(" Subject DN: " + chain[j].getSubjectDN());
System.out.println(" Issuer DN: " + chain[j].getIssuerDN());
System.out.println(" Serial number: " + chain[j].getSerialNumber());
System.out.println("");
}
}
}
};
public static void encode(byte[] pcmBuf, int pcmOffset,
byte[] ulawBuf, int ulawOffset, int length, int max) {
// from 'ulaw' in wikipedia
// +8191 to +8159 0x80
// +8158 to +4063 in 16 intervals of 256 0x80 + interval number
// +4062 to +2015 in 16 intervals of 128 0x90 + interval number
// +2014 to +991 in 16 intervals of 64 0xA0 + interval number
// +990 to +479 in 16 intervals of 32 0xB0 + interval number
// +478 to +223 in 16 intervals of 16 0xC0 + interval number
// +222 to +95 in 16 intervals of 8 0xD0 + interval number
// +94 to +31 in 16 intervals of 4 0xE0 + interval number
// +30 to +1 in 15 intervals of 2 0xF0 + interval number
// 0 0xFF
// -1 0x7F
// -31 to -2 in 15 intervals of 2 0x70 + interval number
// -95 to -32 in 16 intervals of 4 0x60 + interval number
// -223 to -96 in 16 intervals of 8 0x50 + interval number
// -479 to -224 in 16 intervals of 16 0x40 + interval number
// -991 to -480 in 16 intervals of 32 0x30 + interval number
// -2015 to -992 in 16 intervals of 64 0x20 + interval number
// -4063 to -2016 in 16 intervals of 128 0x10 + interval number
// -8159 to -4064 in 16 intervals of 256 0x00 + interval number
// -8192 to -8160 0x00
// set scale factors
if (max <= 0) max = MAX_ULAW;
int coef = MAX_ULAW * (1 << SCALE_BITS) / max;
for (int i = 0; i < length; i++) {
int pcm = (0xff & pcmBuf[pcmOffset++]) + (pcmBuf[pcmOffset++] << 8);
pcm = (pcm * coef) >> SCALE_BITS;
int ulaw;
if (pcm >= 0) {
ulaw = pcm <= 0 ? 0xff :
pcm <= 30 ? 0xf0 + (( 30 - pcm) >> 1) :
pcm <= 94 ? 0xe0 + (( 94 - pcm) >> 2) :
pcm <= 222 ? 0xd0 + (( 222 - pcm) >> 3) :
pcm <= 478 ? 0xc0 + (( 478 - pcm) >> 4) :
pcm <= 990 ? 0xb0 + (( 990 - pcm) >> 5) :
pcm <= 2014 ? 0xa0 + ((2014 - pcm) >> 6) :
pcm <= 4062 ? 0x90 + ((4062 - pcm) >> 7) :
pcm <= 8158 ? 0x80 + ((8158 - pcm) >> 8) :
0x80;
} else {
ulaw = -1 <= pcm ? 0x7f :
-31 <= pcm ? 0x70 + ((pcm - -31) >> 1) :
-95 <= pcm ? 0x60 + ((pcm - -95) >> 2) :
-223 <= pcm ? 0x50 + ((pcm - -223) >> 3) :
-479 <= pcm ? 0x40 + ((pcm - -479) >> 4) :
-991 <= pcm ? 0x30 + ((pcm - -991) >> 5) :
-2015 <= pcm ? 0x20 + ((pcm - -2015) >> 6) :
-4063 <= pcm ? 0x10 + ((pcm - -4063) >> 7) :
-8159 <= pcm ? 0x00 + ((pcm - -8159) >> 8) :
0x00;
}
ulawBuf[ulawOffset++] = (byte)ulaw;
}
}
public static int maxAbsPcm(byte[] pcmBuf, int offset, int length) {
int max = 0;
for (int i = 0; i < length; i++) {
int pcm = (0xff & pcmBuf[offset++]) + (pcmBuf[offset++] << 8);
if (pcm < 0) pcm = -pcm;
if (pcm > max) max = pcm;
}
return max;
}
public int read(byte[] buf, int offset, int length) throws IOException {
if (recorder == null) throw new IllegalStateException("not open");
// return at least one byte, but try to fill 'length'
while (mBufCount < 2) {
int n = recorder.read(mBuf, mBufCount, Math.min(length * 2, mBuf.length - mBufCount));
if (n == -1) return -1;
mBufCount += n;
}
// compand data
int n = Math.min(mBufCount / 2, length);
encode(mBuf, 0, buf, offset, n, mMax);
// move data to bottom of mBuf
mBufCount -= n * 2;
for (int i = 0; i < mBufCount; i++) mBuf[i] = mBuf[i + n * 2];
return n;
}
}