Develop Reference

Custom byteArray data to WebRTC videoTrack

I need to use WebRTC for android to send specific cropped(face) video to the videoChannel. I was able manipulate Camera1Session class of WebRTC to get the face cropped. Right now I am setting it to an ImageView.
listenForBytebufferFrames() of Camera1Session.java
private void listenForBytebufferFrames() {
this.camera.setPreviewCallbackWithBuffer(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera callbackCamera) {
Camera1Session.this.checkIsOnCameraThread();
if(callbackCamera != Camera1Session.this.camera) {
Logging.e("Camera1Session", "Callback from a different camera. This should never happen.");
} else if(Camera1Session.this.state != Camera1Session.SessionState.RUNNING) {
Logging.d("Camera1Session", "Bytebuffer frame captured but camera is no longer running.");
} else {
mFrameProcessor.setNextFrame(data, callbackCamera);
long captureTimeNs = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime());
if(!Camera1Session.this.firstFrameReported) {
int startTimeMs = (int)TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - Camera1Session.this.constructionTimeNs);
Camera1Session.camera1StartTimeMsHistogram.addSample(startTimeMs);
Camera1Session.this.firstFrameReported = true;
}
ByteBuffer byteBuffer1 = ByteBuffer.wrap(data);
Frame outputFrame = new Frame.Builder()
.setImageData(byteBuffer1,
Camera1Session.this.captureFormat.width,
Camera1Session.this.captureFormat.height,
ImageFormat.NV21)
.setTimestampMillis(mFrameProcessor.mPendingTimeMillis)
.setId(mFrameProcessor.mPendingFrameId)
.setRotation(3)
.build();
int w = outputFrame.getMetadata().getWidth();
int h = outputFrame.getMetadata().getHeight();
SparseArray<Face> detectedFaces = mDetector.detect(outputFrame);
if (detectedFaces.size() > 0) {
Face face = detectedFaces.valueAt(0);
ByteBuffer byteBufferRaw = outputFrame.getGrayscaleImageData();
byte[] byteBuffer = byteBufferRaw.array();
YuvImage yuvimage = new YuvImage(byteBuffer, ImageFormat.NV21, w, h, null);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
//My crop logic to get face co-ordinates
yuvimage.compressToJpeg(new Rect(left, top, right, bottom), 80, baos);
final byte[] jpegArray = baos.toByteArray();
Bitmap bitmap = BitmapFactory.decodeByteArray(jpegArray, 0, jpegArray.length);
Activity currentActivity = getActivity();
if (currentActivity instanceof CallActivity) {
((CallActivity) currentActivity).setBitmapToImageView(bitmap); //face on ImageView is set just fine
}
Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, data, Camera1Session.this.captureFormat.width, Camera1Session.this.captureFormat.height, Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(data);
} else {
Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, data, Camera1Session.this.captureFormat.width, Camera1Session.this.captureFormat.height, Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(data);
}
}
}
});
}
jpegArray is the final byteArray that I need to stream via WebRTC, which I tried with something like this:
Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, jpegArray, (int) face.getWidth(), (int) face.getHeight(), Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(jpegArray);
Setting them up like this gives me following error:
../../webrtc/sdk/android/src/jni/androidvideotracksource.cc line 82
Check failed: length >= width * height + 2 * uv_width * ((height + 1) / 2) (2630 vs. 460800)
Which I assume is because androidvideotracksource does not get the same length of byteArray that it expects, since the frame is cropped now.
Could someone point me in the direction of how to achieve it? Is this the correct way/place to manipulate the data and feed into the videoTrack?
Edit:bitmap of byteArray data does not give me a camera preview on ImageView, unlike byteArray jpegArray. Maybe because they are packed differently?

Can we use WebRTC's Datachannel to exchang custom data ie cropped face "image" in your case and do the respective calculation at receiving end using any third party library ie OpenGL etc? Reason I am suggesting is that the WebRTC Video feed received from channel is a stream in real time not a bytearray . WebRTC Video by its inherent architecture isn't meant to crop video at other hand. If we want to crop or augment video we have to use any ar library to fulfill this job.
We can always leverage WebRTC's Data channel to exchange customized data. Using Video channel for the same is not recommended because it's real time stream not the bytearray.Please revert in case of any concern.

WebRTC in particular and video streaming in general presumes that the video has fixed dimensions. If you want to crop the detected face, your options are either to have pad the cropped image with e.g. black pixels (WebRTC does not use transparency), and crop the video on the receiver side, or, if you don't have control over the receiver, resize the cropped region to fill the expected width * height frame (you should also keep the expected aspect ratio).
Note that JPEG compress/decompress that you use to crop the original is far from efficient. Some other options can be found in Image crop and resize in Android.

Okay, this was definitely a problem of how the original byte[] data was packed and the way byte[] jpegArray was packed. Changing the way of packing this and scaling it as AlexCohn suggested worked for me. I found help from other post on StackOverflow on way to pack it. This is the code for it:
private byte[] getNV21(int left, int top, int inputWidth, int inputHeight, Bitmap scaled) {
int [] argb = new int[inputWidth * inputHeight];
scaled.getPixels(argb, 0, inputWidth, left, top, inputWidth, inputHeight);
byte [] yuv = new byte[inputWidth*inputHeight*3/2];
encodeYUV420SP(yuv, argb, inputWidth, inputHeight);
scaled.recycle();
return yuv;
}
private void encodeYUV420SP(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ( ( -38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ( ( 112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
// NV21 has a plane of Y and interleaved planes of VU each sampled by a factor of 2
// meaning for every 4 Y pixels there are 1 V and 1 U. Note the sampling is every other
// pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[uvIndex++] = (byte)((V<0) ? 0 : ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte)((U<0) ? 0 : ((U > 255) ? 255 : U));
}
index ++;
}
}
}`
I pass this byte[] data to onByteBufferFrameCaptured and callback:
Camera1Session.this.events.onByteBufferFrameCaptured(
Camera1Session.this,
data,
w,
h,
Camera1Session.this.getFrameOrientation(),
captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(data);
Prior to this, I had to scale the bitmap which is pretty straight forward:
int width = bitmapToScale.getWidth();
int height = bitmapToScale.getHeight();
Matrix matrix = new Matrix();
matrix.postScale(newWidth / width, newHeight / height);
Bitmap scaledBitmap = Bitmap.createBitmap(bitmapToScale, 0, 0, bitmapToScale.getWidth(), bitmapToScale.getHeight(), matrix, true);

Implementing a high pass audio filter in android

I'm trying to implement an high pass audio filter on the microphone data that I get form the audioRecord.
The data I get form the microphone is a 16-bit PCM audio byte-array. I was trying to use TarsosDSP which provides a API for high pass filtering. However, as an input it requires a float-array so I converted the byte into a float array and ran the highpass filter. To confirm the results I saved the filtered data in a wave file but it sounds totally distorted.
public static byte[] highPassFilter( byte[] buffer, WaveHeader waveHeader, float frequency) {
HighPass highPass = new HighPass(frequency, waveHeader.getSampleRate());
TarsosDSPAudioFormat format = new TarsosDSPAudioFormat(waveHeader.getSampleRate(),waveHeader.getBitsPerSample(),waveHeader.getChannels(),true, false);
AudioEvent audioEvent = new AudioEvent(format);
float[] f_buffer = bytesToFloats(buffer);
audioEvent.setFloatBuffer(f_buffer);
highPass.process(audioEvent);
buffer = audioEvent.getByteBuffer();
byte[] data = PCMtoWav(buffer, waveHeader.getSampleRate(), waveHeader.getChannels(), waveHeader.getBitsPerSample());
writeWavFile(data);
return buffer;
}
public static float[] bytesToFloats(byte[] bytes) {
float[] floats = new float[bytes.length / 2];
for(int i=0; i < bytes.length; i+=2) {
floats[i/2] = bytes[i] | (bytes[i+1] < 128 ? (bytes[i+1] << 8) : ((bytes[i+1] - 256) << 8));
}
return floats;
}
The data in the waveHeader is:
Sample rate = 11025
getBitsPerSample = 16
getChannels = 1
My best guess is that the bytesToFloats conversion is wrong. To verify this I just set the float buffer of the audioEvent with audioEvent.setFloatBuffer and then retrieved it with audioEvent.getByteBuffer which also resulted in a totally distorted audio file.
The byte buffer is read from the audioRecord:
audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC, 11025, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT, 220500);
....
buffer = new byte[frameByteSize];
byte[] audioRecord.read(buffer, 0, frameByteSize);
Anybody have any idea how to fix this or suggestions for different high pass filters that I could use on a byte array in android.
Update: I figured it out. This is my updated function to convert from bytes to floats:
public static float[] bytesToFloats(byte[] bytes) {
float[] floats = new float[bytes.length / 2];
short[] shorts = new short[bytes.length/2];
ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts);
for(int i=0; i < bytes.length; i+=2) {
floats[i/2] = shorts[i/2] / 32768f;
}
return floats;
}

Do the two bytes samples represent float values? They could be signed short within the range of -32,768 to 32,767. Also, for floating point representation of samples the values within the range of -1.0 to 1.0 are common.
I would try:
short sample = bytes[i] | (bytes[i+1] < 128 ? (bytes[i+1] << 8) : ((bytes[i+1] - 256) << 8));
floats[i/2] = (float)sample / 32,768f;

You need to convert pairs of bytes into signed short and then scale it to a float in the range of -1.0 to 1.0.
One of the following lines depending on the endianness of the data will convert to signed 16-bit.
short shortSample = (short)(bytes[i]) | (short)(bytes[i+1]) << 8);
short shortSample = (short)(bytes[i] << 8) | (short)(bytes[i+1]));
And then scale to float:
float sample = shortSample / 32768f;

QCom Decoder Output YUV Buffer Distrubed

I'm trying to get images from HD Live Stream. Getting OMX Decoder YUV Streams and converting them into JPG. JPEG is completely disturbed. Tried some suggestions from group but not working.
My resolution is 320x240.
i will get buffer length is (386 * 256 * 1.5) for configured 320 * 240 resolution. I'm not getting how to get this new width and height information.
JPG conversion code i have in Java and using OMXCodec is in Native. Please help me.
final int frameSize = width * height;
final int qFrameSize = frameSize/4;
int padding = 0;/*(width*height + 2047) & ~2047;
if ((width % 32) != 0) {
padding = (width*height) % 1024;
} else {
padding = (width*height) % 2048;
}
System.arraycopy(input, 0, output, 0, frameSize); // Y
for (int i = 0; i < qFrameSize; i++) {
output[frameSize + i*2 + padding] = input[frameSize + i + qFrameSize ]; // Cb (U)
output[frameSize + i*2 + 1 + padding] = input[frameSize + i ]; // Cr (V)
}
return ;
}
thank you,
Raghu

The output of QCom video decoder is usually a specific custom color format which is typically known as tiled format. Please refer to these questions which have more inputs on how to convert the data to a more cleaner frame
QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka converter
QOMX_COLOR_FormatYUV420PackedSemiPlanar64x32Tile2m8ka color format

android AudioTrack setloop invalid value

I generate a PCM and want to loop the sound.
I follow the documentation, but Eclipse keep telling me that
08-05 15:46:26.675: E/AudioTrack(27686): setLoop invalid value: loopStart 0, loopEnd 44100, loopCount -1, framecount 11025, user 11025
here is my code:
void genTone() {
// fill out the array
for (int i = 1; i < numSamples - 1; i = i + 2) {
sample[i] = Math.sin(2 * Math.PI * i / (sampleRate / -300));
}
// convert to 16 bit pcm sound array
// assumes the sample buffer is normalised.
int idx = 0;
for (double dVal : sample) {
short val = (short) (dVal * 32767);
generatedSnd[idx++] = (byte) (val & 0x00ff);
generatedSnd[idx++] = (byte) ((val & 0xff00) >>> 8);
}
//write it to audio Track.
audioTrack.write(generatedSnd, 0, numSamples);
audioTrack.setLoopPoints(0, numSamples, -1);
//from 0.0 ~ 1.0
audioTrack.setStereoVolume((float)0.5, (float)1); //change amplitude
}
public void buttonPlay(View v) {
audioTrack.reloadStaticData();
audioTrack.play();
}
please help ~~

From the documentation: "endInFrames loop end marker expressed in frames"
The log print indicates that your track contains 11025 frames, which is less than the 44100 that you're trying to specify as the end marker (for 16-bit stereo PCM audio, the frame size would be 4 bytes).
Another thing worth noting is that "the track must be stopped or paused for the position to be changed".

Android way to use Bitmaps from Camera.setPreviewCallback

camera.setPreviewCallback(new Camera.PreviewCallback() {
private long timestamp=0;
public synchronized void onPreviewFrame(byte[] data, Camera camera) {
Log.e("CameraTest","Time Gap = "+(System.currentTimeMillis()-timestamp));
timestamp=System.currentTimeMillis();
Bitmap mFaceBitmap = BitmapFactory.decodeByteArray(data, 0, data.length);
if (mFaceBitmap!=null) FaceDetection.calculate(mFaceBitmap);
camera.addCallbackBuffer(data);
return;
}
});
I have a camera View, and in front of a simple View (where I can draw something).
I'd like to draw on the front of View, when I can find the face of a human.
But mFaceBitmap is ever and ever return null, why?
If this is a bad idea, how can I do this better?

When you set-up the camera you will need to set the preview size and the preview format. Here is some sample code to give the rough idea:
int previewFormat = 0;
for (int format : parameters.getSupportedPreviewFormats()) {
if (format == FORMAT_NV21) {
previewFormat = FORMAT_NV21;
} else if (previewFormat == 0 && (format == FORMAT_JPEG || format == FORMAT_RGB_565)) {
previewFormat = format;
}
}
// TODO: Iterate on supported preview sizes and pick best one
parameters.setPreviewSize(previewSize.width, previewSize.height);
if (previewFormat != 0) {
parameters.setPreviewFormat(previewFormat);
} else {
// Error on unsupported format
}
Now in the callback you can do something like:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
Bitmnap bitmap;
if (previewFormat == FORMAT_NV21) {
int[] previewPixels = new int[previewSize.width * previewSize.height];
decodeYUV420SP(previewPixels, data, previewSize.width, previewSize.height);
bitmap = Bitmap.createBitmap(rgbPixels, previewSize.width, previewSize.height, Bitmap.Config.RGB_565);
} else if (previewFormat == FORMAT_JPEG || previewFormat == FORMAT_RGB_565) {
// RGB565 and JPEG
BitmapFactory.Options opts = new BitmapFactory.Options();
opts.inDither = true;
opts.inPreferredConfig = Bitmap.Config.RGB_565;
bitmap = BitmapFactory.decodeByteArray(data, 0, data.length, opts);
}
}
And finally, the conversion
static void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width, int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
}

You can't use Bitmap.decodeByteArray to convert a camera's preview output into a bitmap, unfortunately.
decodeByteArray is designed for converting JPEG/PNG/etc images into bitmaps, and it doesn't have any way of knowing what the data in the preview callback is like, because it's a simple raw array of pixel values with no identifying header.
You have to do the conversion yourself. There are many ways to do this, of various degrees of efficiency - I'll write out the simplest one here, but it's also probably the slowest.
The data byte array from the camera is encoded in some particular pixel format, which is specified by Camera.Parameters.setPreviewFormat. If you haven't called this, the default format is NV21. NV21 is guaranteed to work on all Android devices; on Android versions >= 3.0, the YV12 format is also guaranteed to work.
Both of these are YUV formats, meaning the color is encoded as a luminance (brightness) channel and two chroma (color) channels. The functions for setting pixel values on a Bitmap (primarily setPixels) require information in the RGB color space instead, so a conversion is required. In addition, both NV21 and YV12 subsample the chroma channels - if you have a 640x480 image, for example, there will be 640x480 pixels in the luminance channel, but only 320x240 pixels in the two chroma channels.
This means you need to create a new int[] array of the right size, and then loop over the byte[] data array, collecting up a set of Y, U, and V values, convert them to RGB, and write them to the int[] array, and then call setPixels on your destination bitmap. The color conversion matrix you need is the JPEG YCbCr->RGB matrix, which you can find at Wikipedia, for example. You can find out about the layout of NV21 or YV12 at fourcc, as one example
If you really don't want to mess with all that, you can also use the YuvImage class, albeit in a roundabout way. You can construct a YuvImage instance from the preview data, as long as you're using the NV21 format, and then save a JPEG from it into a ByteArrayOutputStream. You can then get the byte[] from the stream, and decode it into a bitmap using Bitmap.decodeByteArray. This is a completely unnecessary roundtrip to JPEG and back, so it's quite inefficient and can cause quality loss, but it only requires a few lines of code.
In the latest version of Android, you can also use Renderscript to efficiently do this conversion. You'll need to copy the data into an Allocation, and then use the YUV to RGB script intrinsic to do the conversion.
Finally, you can pass the data and destination bitmap into JNI code, where you can access the Bitmap directly, and write the conversion function there in C or C++. This requires a lot of scaffolding, but is very efficient.