Develop Reference

Custom byteArray data to WebRTC videoTrack

I need to use WebRTC for android to send specific cropped(face) video to the videoChannel. I was able manipulate Camera1Session class of WebRTC to get the face cropped. Right now I am setting it to an ImageView.
listenForBytebufferFrames() of Camera1Session.java
private void listenForBytebufferFrames() {
this.camera.setPreviewCallbackWithBuffer(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera callbackCamera) {
Camera1Session.this.checkIsOnCameraThread();
if(callbackCamera != Camera1Session.this.camera) {
Logging.e("Camera1Session", "Callback from a different camera. This should never happen.");
} else if(Camera1Session.this.state != Camera1Session.SessionState.RUNNING) {
Logging.d("Camera1Session", "Bytebuffer frame captured but camera is no longer running.");
} else {
mFrameProcessor.setNextFrame(data, callbackCamera);
long captureTimeNs = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime());
if(!Camera1Session.this.firstFrameReported) {
int startTimeMs = (int)TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - Camera1Session.this.constructionTimeNs);
Camera1Session.camera1StartTimeMsHistogram.addSample(startTimeMs);
Camera1Session.this.firstFrameReported = true;
}
ByteBuffer byteBuffer1 = ByteBuffer.wrap(data);
Frame outputFrame = new Frame.Builder()
.setImageData(byteBuffer1,
Camera1Session.this.captureFormat.width,
Camera1Session.this.captureFormat.height,
ImageFormat.NV21)
.setTimestampMillis(mFrameProcessor.mPendingTimeMillis)
.setId(mFrameProcessor.mPendingFrameId)
.setRotation(3)
.build();
int w = outputFrame.getMetadata().getWidth();
int h = outputFrame.getMetadata().getHeight();
SparseArray<Face> detectedFaces = mDetector.detect(outputFrame);
if (detectedFaces.size() > 0) {
Face face = detectedFaces.valueAt(0);
ByteBuffer byteBufferRaw = outputFrame.getGrayscaleImageData();
byte[] byteBuffer = byteBufferRaw.array();
YuvImage yuvimage = new YuvImage(byteBuffer, ImageFormat.NV21, w, h, null);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
//My crop logic to get face co-ordinates
yuvimage.compressToJpeg(new Rect(left, top, right, bottom), 80, baos);
final byte[] jpegArray = baos.toByteArray();
Bitmap bitmap = BitmapFactory.decodeByteArray(jpegArray, 0, jpegArray.length);
Activity currentActivity = getActivity();
if (currentActivity instanceof CallActivity) {
((CallActivity) currentActivity).setBitmapToImageView(bitmap); //face on ImageView is set just fine
}
Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, data, Camera1Session.this.captureFormat.width, Camera1Session.this.captureFormat.height, Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(data);
} else {
Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, data, Camera1Session.this.captureFormat.width, Camera1Session.this.captureFormat.height, Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(data);
}
}
}
});
}
jpegArray is the final byteArray that I need to stream via WebRTC, which I tried with something like this:
Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, jpegArray, (int) face.getWidth(), (int) face.getHeight(), Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(jpegArray);
Setting them up like this gives me following error:
../../webrtc/sdk/android/src/jni/androidvideotracksource.cc line 82
Check failed: length >= width * height + 2 * uv_width * ((height + 1) / 2) (2630 vs. 460800)
Which I assume is because androidvideotracksource does not get the same length of byteArray that it expects, since the frame is cropped now.
Could someone point me in the direction of how to achieve it? Is this the correct way/place to manipulate the data and feed into the videoTrack?
Edit:bitmap of byteArray data does not give me a camera preview on ImageView, unlike byteArray jpegArray. Maybe because they are packed differently?

Can we use WebRTC's Datachannel to exchang custom data ie cropped face "image" in your case and do the respective calculation at receiving end using any third party library ie OpenGL etc? Reason I am suggesting is that the WebRTC Video feed received from channel is a stream in real time not a bytearray . WebRTC Video by its inherent architecture isn't meant to crop video at other hand. If we want to crop or augment video we have to use any ar library to fulfill this job.
We can always leverage WebRTC's Data channel to exchange customized data. Using Video channel for the same is not recommended because it's real time stream not the bytearray.Please revert in case of any concern.

WebRTC in particular and video streaming in general presumes that the video has fixed dimensions. If you want to crop the detected face, your options are either to have pad the cropped image with e.g. black pixels (WebRTC does not use transparency), and crop the video on the receiver side, or, if you don't have control over the receiver, resize the cropped region to fill the expected width * height frame (you should also keep the expected aspect ratio).
Note that JPEG compress/decompress that you use to crop the original is far from efficient. Some other options can be found in Image crop and resize in Android.

Okay, this was definitely a problem of how the original byte[] data was packed and the way byte[] jpegArray was packed. Changing the way of packing this and scaling it as AlexCohn suggested worked for me. I found help from other post on StackOverflow on way to pack it. This is the code for it:
private byte[] getNV21(int left, int top, int inputWidth, int inputHeight, Bitmap scaled) {
int [] argb = new int[inputWidth * inputHeight];
scaled.getPixels(argb, 0, inputWidth, left, top, inputWidth, inputHeight);
byte [] yuv = new byte[inputWidth*inputHeight*3/2];
encodeYUV420SP(yuv, argb, inputWidth, inputHeight);
scaled.recycle();
return yuv;
}
private void encodeYUV420SP(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ( ( -38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ( ( 112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
// NV21 has a plane of Y and interleaved planes of VU each sampled by a factor of 2
// meaning for every 4 Y pixels there are 1 V and 1 U. Note the sampling is every other
// pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[uvIndex++] = (byte)((V<0) ? 0 : ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte)((U<0) ? 0 : ((U > 255) ? 255 : U));
}
index ++;
}
}
}`
I pass this byte[] data to onByteBufferFrameCaptured and callback:
Camera1Session.this.events.onByteBufferFrameCaptured(
Camera1Session.this,
data,
w,
h,
Camera1Session.this.getFrameOrientation(),
captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(data);
Prior to this, I had to scale the bitmap which is pretty straight forward:
int width = bitmapToScale.getWidth();
int height = bitmapToScale.getHeight();
Matrix matrix = new Matrix();
matrix.postScale(newWidth / width, newHeight / height);
Bitmap scaledBitmap = Bitmap.createBitmap(bitmapToScale, 0, 0, bitmapToScale.getWidth(), bitmapToScale.getHeight(), matrix, true);

Get yuv size size stored by Mediabuffer

I am use OpenMax to decode the video frame,my sample is like this:
FILE* fp = fopen("/data/local/tmp/test.yuv", "wb");
while(!isExit)
{
MediaBuffer *mVideoBuffer;
MediaSource::ReadOptions options;
status_t err = mVideoDecoder->read(&mVideoBuffer, &options);
if (err == OK)
{
if (mVideoBuffer->range_length() > 0)
{
// If video frame availabe, render it to mNativeWindow
int w = 0;
int h = 0;
int dw = 0;
int dh = 0;
int stride = 0;
sp<MetaData> metaData = mVideoBuffer->meta_data();
sp<MetaData> outFormat = mVideoDecoder->getFormat();
outFormat->findInt32(kKeyWidth , &w);
outFormat->findInt32(kKeyHeight, &h);
int64_t timeUs = 0;
metaData->findInt64(kKeyTime, &timeUs);
metaData->findInt32(kKeyDisplayHeight, &dh);
metaData->findInt32(kKeyDisplayWidth, &dw);
metaData->findInt32(kKeyStride, &stride);
printf("out format w:%d h:%d dw:%d dh:%d stride:%d timestamp:%lld\n",
w, h, dw, dh, stride, timeUs);
if(fp)
{
printf("decode a frame, range_length = %d range_offset = %d size = %d width = %d height = %d\n",
mVideoBuffer->range_length(), mVideoBuffer->range_offset(), mVideoBuffer->size(), w, h);
fwrite(mVideoBuffer->data( ) + mVideoBuffer->range_offset( ), 1, mVideoBuffer->range_length(), fp);
}
metaData->setInt32(kKeyRendered, 1);
}
mVideoBuffer->release();
}
else
{
printf("end of file\n");
isExit = true;
}
}
fclose(fp);
the output is like this:
out format w:1280 h:720 dw:0 dh:0 stride:0 timestamp:44044
decode a frame, range_length = 1417216 range_offset = 0 size = 1417216 width = 1280 height = 720
My question is how to know the yuv realy size stored by the MediaBuffer,
because 1280 x 736 (stride is 32, i guess) x 1.5 = 1413120,but mVideoBuffer's range_length() is 1417216, there is no regular i can get the yuv size, please help me, thanks !

The MediaBuffer which is being read as part read method is initialized here. Please note that the info.mSize is set to def.nBufferSize which is same as allocated size. This information is not updated to reflect the actual size filled by the decoder.
The allocation is 1280 x 736 x 1.5 with the height aligned to 32. The stride is really not required as 1280 is a very well aligned number.

onPreviewFrame YUV grayscale skewed

I'm trying to get the picture from a surfaceView where I have the camera view running,
I've already implemented onPreviewFrame, and it's called correctly as the debug shows me.
The problem I'm facing now, it's since the byte[] data I receive in the method, it's in YUV space color (NV21), I'm trying to convert it to grayscale to generate a Bitmap and then storing it into a file.
The conversion process that I'm following it's:
public Bitmap convertYuvGrayScaleRGB(byte[] yuv, int width, int height) {
int[] pixels = new int[width * height];
for (int i = 0; i < height*width; i++) {
int grey = yuv[i] & 0xff;
pixels[i] = 0xFF000000 | (grey * 0x00010101);
}
return Bitmap.createBitmap(pixels, width, height, Bitmap.Config.ARGB_8888);
}
The importing procedure for storing it to a file, it's:
Bitmap bitmap = convertYuvGrayScaleRGB(data,widht,heigth);
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 50, bytes);
File f = new File(Environment.getExternalStorageDirectory()
+ File.separator + "test.jpg");
Log.d("Camera", "File: " + f.getAbsolutePath());
try {
f.createNewFile();
FileOutputStream fo = new FileOutputStream(f);
fo.write(bytes.toByteArray());
fo.close();
bitmap.recycle();
bitmap = null;
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
Altough, the result I've got it's the following:

I can't find any obvious mistake in your code, but i've already met this kind of skewed images before. When this happened to me, it was due to:
At some point in the code, the image width and height are swapped,
Or the original image you're trying to convert has padding, in which case you will need a stride in addition of the width and height.
Hope this helps!

Probably the Width of the image you are converting is not even. in that case
it is padded in memory.
Let me have a look at the docs...
It seems more complicated than this. if you want your code to work as it is now, you will have to have the width
a multiple of 16.
from the docs:
public static final int YV12
Added in API level 9 Android YUV format.
This format is exposed to software decoders and applications.
YV12 is a 4:2:0 YCrCb planar format comprised of a WxH Y plane
followed by (W/2) x (H/2) Cr and Cb planes.
This format assumes
an even width an even height a horizontal stride multiple of 16 pixels
a vertical stride equal to the height y_size = stride * height
c_stride = ALIGN(stride/2, 16) c_size = c_stride * height/2 size =
y_size + c_size * 2 cr_offset = y_size cb_offset = y_size + c_size

I just had this problem with the S3. My problem was that I used the wrong dimensions for the preview. I assumed the camera was 16:9 when it was actually 4:3.
Use Camera.getParameters().getPreviewSize() to see what the output is in.

I made this:
int frameSize = width * height;
for (int i = 0; i < height; i++) {
for (int j = 0; j < width; j++) {
ret[frameSize + (i >> 1) * width + (j & ~1) + 1] = 127; //U
ret[frameSize + (i >> 1) * width + (j & ~1) + 0] = 127; //V
}
}
So simple but it works really good and fast ;)

Android way to use Bitmaps from Camera.setPreviewCallback

camera.setPreviewCallback(new Camera.PreviewCallback() {
private long timestamp=0;
public synchronized void onPreviewFrame(byte[] data, Camera camera) {
Log.e("CameraTest","Time Gap = "+(System.currentTimeMillis()-timestamp));
timestamp=System.currentTimeMillis();
Bitmap mFaceBitmap = BitmapFactory.decodeByteArray(data, 0, data.length);
if (mFaceBitmap!=null) FaceDetection.calculate(mFaceBitmap);
camera.addCallbackBuffer(data);
return;
}
});
I have a camera View, and in front of a simple View (where I can draw something).
I'd like to draw on the front of View, when I can find the face of a human.
But mFaceBitmap is ever and ever return null, why?
If this is a bad idea, how can I do this better?

When you set-up the camera you will need to set the preview size and the preview format. Here is some sample code to give the rough idea:
int previewFormat = 0;
for (int format : parameters.getSupportedPreviewFormats()) {
if (format == FORMAT_NV21) {
previewFormat = FORMAT_NV21;
} else if (previewFormat == 0 && (format == FORMAT_JPEG || format == FORMAT_RGB_565)) {
previewFormat = format;
}
}
// TODO: Iterate on supported preview sizes and pick best one
parameters.setPreviewSize(previewSize.width, previewSize.height);
if (previewFormat != 0) {
parameters.setPreviewFormat(previewFormat);
} else {
// Error on unsupported format
}
Now in the callback you can do something like:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
Bitmnap bitmap;
if (previewFormat == FORMAT_NV21) {
int[] previewPixels = new int[previewSize.width * previewSize.height];
decodeYUV420SP(previewPixels, data, previewSize.width, previewSize.height);
bitmap = Bitmap.createBitmap(rgbPixels, previewSize.width, previewSize.height, Bitmap.Config.RGB_565);
} else if (previewFormat == FORMAT_JPEG || previewFormat == FORMAT_RGB_565) {
// RGB565 and JPEG
BitmapFactory.Options opts = new BitmapFactory.Options();
opts.inDither = true;
opts.inPreferredConfig = Bitmap.Config.RGB_565;
bitmap = BitmapFactory.decodeByteArray(data, 0, data.length, opts);
}
}
And finally, the conversion
static void decodeYUV420SP(int[] rgb, byte[] yuv420sp, int width, int height) {
final int frameSize = width * height;
for (int j = 0, yp = 0; j < height; j++) {
int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
for (int i = 0; i < width; i++, yp++) {
int y = (0xff & ((int) yuv420sp[yp])) - 16;
if (y < 0)
y = 0;
if ((i & 1) == 0) {
v = (0xff & yuv420sp[uvp++]) - 128;
u = (0xff & yuv420sp[uvp++]) - 128;
}
int y1192 = 1192 * y;
int r = (y1192 + 1634 * v);
int g = (y1192 - 833 * v - 400 * u);
int b = (y1192 + 2066 * u);
if (r < 0)
r = 0;
else if (r > 262143)
r = 262143;
if (g < 0)
g = 0;
else if (g > 262143)
g = 262143;
if (b < 0)
b = 0;
else if (b > 262143)
b = 262143;
rgb[yp] = 0xff000000 | ((r << 6) & 0xff0000) | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff);
}
}
}

You can't use Bitmap.decodeByteArray to convert a camera's preview output into a bitmap, unfortunately.
decodeByteArray is designed for converting JPEG/PNG/etc images into bitmaps, and it doesn't have any way of knowing what the data in the preview callback is like, because it's a simple raw array of pixel values with no identifying header.
You have to do the conversion yourself. There are many ways to do this, of various degrees of efficiency - I'll write out the simplest one here, but it's also probably the slowest.
The data byte array from the camera is encoded in some particular pixel format, which is specified by Camera.Parameters.setPreviewFormat. If you haven't called this, the default format is NV21. NV21 is guaranteed to work on all Android devices; on Android versions >= 3.0, the YV12 format is also guaranteed to work.
Both of these are YUV formats, meaning the color is encoded as a luminance (brightness) channel and two chroma (color) channels. The functions for setting pixel values on a Bitmap (primarily setPixels) require information in the RGB color space instead, so a conversion is required. In addition, both NV21 and YV12 subsample the chroma channels - if you have a 640x480 image, for example, there will be 640x480 pixels in the luminance channel, but only 320x240 pixels in the two chroma channels.
This means you need to create a new int[] array of the right size, and then loop over the byte[] data array, collecting up a set of Y, U, and V values, convert them to RGB, and write them to the int[] array, and then call setPixels on your destination bitmap. The color conversion matrix you need is the JPEG YCbCr->RGB matrix, which you can find at Wikipedia, for example. You can find out about the layout of NV21 or YV12 at fourcc, as one example
If you really don't want to mess with all that, you can also use the YuvImage class, albeit in a roundabout way. You can construct a YuvImage instance from the preview data, as long as you're using the NV21 format, and then save a JPEG from it into a ByteArrayOutputStream. You can then get the byte[] from the stream, and decode it into a bitmap using Bitmap.decodeByteArray. This is a completely unnecessary roundtrip to JPEG and back, so it's quite inefficient and can cause quality loss, but it only requires a few lines of code.
In the latest version of Android, you can also use Renderscript to efficiently do this conversion. You'll need to copy the data into an Allocation, and then use the YUV to RGB script intrinsic to do the conversion.
Finally, you can pass the data and destination bitmap into JNI code, where you can access the Bitmap directly, and write the conversion function there in C or C++. This requires a lot of scaffolding, but is very efficient.

Simple Image Sharpening algorithm for Android App

I am looking for a simple image sharpening algorithm to use in my Android application. I have a grayscale image captured from video (mostly used for text) and I would like to sharpen it because my phone does not have auto focus, and the close object distance blurs the text. I don't have any background in image processing. But as a user, I am familiar with unsharp masking and other sharpening tools available in Gimp, Photoshop, etc. I didn't see any support for image processing in the Android API, and hence am looking for a method to implement myself. Thanks.

This is a simple image sharpening algorithm.
You should pass to this function width, height and byte[] array of your grayscale image and it will sharpen the image in this byte[] array.
void sharpen(int width, int height, byte* yuv) {
char *mas;
mas = (char *) malloc(width * height);
memcpy(mas, yuv, width * height);
signed int res;
int ywidth;
for (int y = 1; y < height - 1; y++) {
ywidth = y * width;
for (int x = 1; x < width - 1; x++) {
res = (
mas[x + ywidth] * 5
- mas[x - 1 + ywidth]
- mas[x + 1 + ywidth]
- mas[x + (ywidth + width)]
- mas[x + (ywidth - width)]
);
if (res > 255) {
res = 255;
};
if (res < 0) {
res = 0;
};
yuv[x + ywidth] = res;
}
}
free(mas);
}

If you have access to pixel information, your most basic option woul be a sharpening convolution kernel. Take a look at the following sites, you can learn more about sharpening kernels and how to apply kernels there.
link1
link2

ImageJ has many algorithms in Java and is freely available.