In the process of tracking severe memory issues in my app, I looked at several heap dumps from my app, and most of the time I have a HUGE bitmap that I don't know of.
It takes 9.4MB, or 9,830,400 bytes, or actually a 1280x1920 image at 4 bytes per pixels.
I checked in Eclipse MAT, it is indeed a byte[9830400], that has one incoming reference which is a android.graphics.Bitmap.
I'd like to dump this to a file and try to see it. I can't understand where is it coming from. My biggest image in all my drawables is a 640x960 png, which takes less than 3MB.
I tried to use Eclipse to "copy value to file", but I think it simply prints the buffer to the file, and I don't know any image software that can read a stream of bytes and display it as a 4 bytes per pixel image.
Any idea?
Here's what I tried: dump the byte array to a file, push it to /sdcard/img, and load an activity like this:
#Override
public void onCreate(final Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
try {
final File inputFile = new File("/sdcard/img");
final FileInputStream isr = new FileInputStream(inputFile);
final Bitmap bmp = BitmapFactory.decodeStream(isr);
ImageView iv = new ImageView(this);
iv.setImageBitmap(bmp);
setContentView(iv);
Log.d("ImageTest", "Image was inflated");
} catch (final FileNotFoundException e) {
Log.d("ImageTest", "Image was not inflated");
}
}
I didn't see anything.
Do you know how is encoded the image? Say it is stored into byte[] buffer. buffer[0] is red, buffer[1] is green, etc?
See here for an easier answer: MAT (Eclipse Memory Analyzer) - how to view bitmaps from memory dump
TL;DR - Install GIMP and load the image as raw RGB Alpha
OK -- After quite some unsuccessful tries, I finally got something out of this byte array. I wrote this simple C program to convert the byte array to a Windows Bitmap file. I'm dropping the code in case somebody is interested.
I compiled this against VisualC 6.0 and gcc 3.4.4, it should work on any OS (tested on Windows, Linux and MacOS X).
#include <stdio.h>
#include <math.h>
#include <string.h>
#include <stdlib.h>
/* Types */
typedef unsigned char byte;
typedef unsigned short uint16_t;
typedef unsigned int uint32_t;
typedef int int32_t;
/* Constants */
#define RMASK 0x00ff0000
#define GMASK 0x0000ff00
#define BMASK 0x000000ff
#define AMASK 0xff000000
/* Structures */
struct bmpfile_magic {
unsigned char magic[2];
};
struct bmpfile_header {
uint32_t filesz;
uint16_t creator1;
uint16_t creator2;
uint32_t bmp_offset;
};
struct bmpfile_dibheader {
uint32_t header_sz;
uint32_t width;
uint32_t height;
uint16_t nplanes;
uint16_t bitspp;
uint32_t compress_type;
uint32_t bmp_bytesz;
int32_t hres;
int32_t vres;
uint32_t ncolors;
uint32_t nimpcolors;
uint32_t rmask, gmask, bmask, amask;
uint32_t colorspace_type;
byte colorspace[0x24];
uint32_t rgamma, ggamma, bgamma;
};
/* Displays usage info and exits */
void usage(char *cmd) {
printf("Usage:\t%s <img_src> <img_dest.bmp> <width> <height>\n"
"\timg_src:\timage byte buffer obtained from Eclipse MAT, using 'copy > save value to file' while selecting the byte[] buffer corresponding to an android.graphics.Bitmap\n"
"\timg_dest:\tpath to target *.bmp file\n"
"\twidth:\t\tpicture width, obtained in Eclipse MAT, selecting the android.graphics.Bitmap object and seeing the object member values\n"
"\theight:\t\tpicture height\n\n", cmd);
exit(1);
}
/* C entry point */
int main(int argc, char **argv) {
FILE *in, *out;
char *file_in, *file_out;
int w, h, W, H;
byte r, g, b, a, *image;
struct bmpfile_magic magic;
struct bmpfile_header header;
struct bmpfile_dibheader dibheader;
/* Parse command line */
if (argc < 5) {
usage(argv[0]);
}
file_in = argv[1];
file_out = argv[2];
W = atoi(argv[3]);
H = atoi(argv[4]);
in = fopen(file_in, "rb");
out = fopen(file_out, "wb");
/* Check parameters */
if (in == NULL || out == NULL || W == 0 || H == 0) {
usage(argv[0]);
}
/* Init BMP headers */
magic.magic[0] = 'B';
magic.magic[1] = 'M';
header.filesz = W * H * 4 + sizeof(magic) + sizeof(header) + sizeof(dibheader);
header.creator1 = 0;
header.creator2 = 0;
header.bmp_offset = sizeof(magic) + sizeof(header) + sizeof(dibheader);
dibheader.header_sz = sizeof(dibheader);
dibheader.width = W;
dibheader.height = H;
dibheader.nplanes = 1;
dibheader.bitspp = 32;
dibheader.compress_type = 3;
dibheader.bmp_bytesz = W * H * 4;
dibheader.hres = 2835;
dibheader.vres = 2835;
dibheader.ncolors = 0;
dibheader.nimpcolors = 0;
dibheader.rmask = RMASK;
dibheader.gmask = BMASK;
dibheader.bmask = GMASK;
dibheader.amask = AMASK;
dibheader.colorspace_type = 0x57696e20;
memset(&dibheader.colorspace, 0, sizeof(dibheader.colorspace));
dibheader.rgamma = dibheader.bgamma = dibheader.ggamma = 0;
/* Read picture data */
image = (byte*) malloc(4*W*H);
if (image == NULL) {
printf("Could not allocate a %d-byte buffer.\n", 4*W*H);
exit(1);
}
fread(image, 4*W*H, sizeof(byte), in);
fclose(in);
/* Write header */
fwrite(&magic, sizeof(magic), 1, out);
fwrite(&header, sizeof(header), 1, out);
fwrite(&dibheader, sizeof(dibheader), 1, out);
/* Convert the byte array to BMP format */
for (h = H-1; h >= 0; h--) {
for (w = 0; w < W; w++) {
r = *(image + w*4 + 4 * W * h);
b = *(image + w*4 + 4 * W * h + 1);
g = *(image + w*4 + 4 * W * h + 2);
a = *(image + w*4 + 4 * W * h + 3);
fwrite(&b, 1, 1, out);
fwrite(&g, 1, 1, out);
fwrite(&r, 1, 1, out);
fwrite(&a, 1, 1, out);
}
}
free(image);
fclose(out);
}
So using this tool I was able to recognise the picture used to generate this 1280x1920 bitmap.
I found that starting from latest version of Android Studio (2.2.2 as of writing), you can view the bitmap file directly:
Open the ‘Android Monitor’ tab (at the bottom left) and then Memory tab.
Press the ‘Dump Java Heap’ button
Choose the ‘Bitmap’ Class Name for the current snapshot, select each Instance of bitmap and view what image exactly consume more memory than expected. (screens 4 and 5)
Choose the Bitmap class name…
Select each Instance of bitmap
and right click on it, select View Bitmap
Just take the input to the image and convert it into a bitmap object by using the fileinput stream/datastream. Also add logs for seeing data for each image that gets used.
You could enable an usb connection and copy the file to an other computer with more tools to investigate.
Some devices could be configured to dump the current screen to file system when the start button is pressed. Maybe this happens to you.
Related
Update #6 Discovered I was accessing RGB values improperly. I assumed I was accessing data from an Int[], but was instead accessing byte information from a Byte[]. Changed to accessing from Int[] and get the following image:
Update #5 Adding code used to get RGBA ByteBuffer for reference
private void screenScrape() {
Log.d(TAG, "In screenScrape");
//read pixels from frame buffer into PBO (GL_PIXEL_PACK_BUFFER)
mSurface.queueEvent(new Runnable() {
#Override
public void run() {
Log.d(TAG, "In Screen Scrape 1");
//generate and bind buffer ID
GLES30.glGenBuffers(1, pboIds);
checkGlError("Gen Buffers");
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, pboIds.get(0));
checkGlError("Bind Buffers");
//creates and initializes data store for PBO. Any pre-existing data store is deleted
GLES30.glBufferData(GLES30.GL_PIXEL_PACK_BUFFER, (mWidth * mHeight * 4), null, GLES30.GL_STATIC_READ);
checkGlError("Buffer Data");
//glReadPixelsPBO(0,0,w,h,GLES30.GL_RGB,GLES30.GL_UNSIGNED_SHORT_5_6_5,0);
glReadPixelsPBO(0, 0, mWidth, mHeight, GLES30.GL_RGBA, GLES30.GL_UNSIGNED_BYTE, 0);
checkGlError("Read Pixels");
//GLES30.glReadPixels(0,0,w,h,GLES30.GL_RGBA,GLES30.GL_UNSIGNED_BYTE,intBuffer);
}
});
//map PBO data into client address space
mSurface.queueEvent(new Runnable() {
#Override
public void run() {
Log.d(TAG, "In Screen Scrape 2");
//read pixels from PBO into a byte buffer for processing. Unmap buffer for use in next pass
mapBuffer = ((ByteBuffer) GLES30.glMapBufferRange(GLES30.GL_PIXEL_PACK_BUFFER, 0, 4 * mWidth * mHeight, GLES30.GL_MAP_READ_BIT)).order(ByteOrder.nativeOrder());
checkGlError("Map Buffer");
GLES30.glUnmapBuffer(GLES30.GL_PIXEL_PACK_BUFFER);
checkGlError("Unmap Buffer");
isByteBufferEmpty(mapBuffer, "MAP BUFFER");
convertColorSpaceByteArray(mapBuffer);
mapBuffer.clear();
}
});
}
Update #4 For reference, here is the original image to compare against.
Update #3 This is the output image after interleaving all U/V data into a single array and passing it to the Image object at inputImagePlanes[1]; inputImagePlanes[2]; is unused;
The next image is the same interleaved UV data, but we load this into inputImagePlanes[2]; instead of inputImagePlanes[1];
Update #2 This is the output image after padding the U/V buffers with a zero in between each byte of 'real' data. uArray[uvByteIndex] = (byte) 0;
Update #1 As suggested by a comment, here are the row and pixel strides I get from calling getPixelStride and getRowStride
Y Plane Pixel Stride = 1, Row Stride = 960
U Plane Pixel Stride = 2, Row Stride = 960
V Plane Pixel Stride = 2, Row Stride = 960
The goal of my application is to read pixels out from the screen, compress them, and then send that h264 stream over WiFi to be played be a receiver.
Currently I'm using the MediaMuxer class to convert the raw h264 stream to an MP4, and then save it to file. However the end result video is messed up and I can't figure out why. Lets walk through some of processing and see if we can find anything that jumps out.
Step 1 Set up the encoder. I'm currently taking screen images once every 2 seconds, and using "video/avc" for MIME_TYPE
//create codec for compression
try {
mCodec = MediaCodec.createEncoderByType(MIME_TYPE);
} catch (IOException e) {
Log.d(TAG, "FAILED: Initializing Media Codec");
}
//set up format for codec
MediaFormat mFormat = MediaFormat.createVideoFormat(MIME_TYPE, mWidth, mHeight);
mFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible);
mFormat.setInteger(MediaFormat.KEY_BIT_RATE, 16000000);
mFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 1/2);
mFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
Step 2 Read pixels out from screen. This is done using openGL ES, and the pixels are read out in RGBA format. (I've confirmed this part to be working)
Step 3 Convert the RGBA pixels to YUV420 (IYUV) format. This is done using the following method. Note that I have 2 methods for encoding called at the end of this method.
private void convertColorSpaceByteArray(ByteBuffer rgbBuffer) {
long startTime = System.currentTimeMillis();
Log.d(TAG, "In convertColorspace");
final int frameSize = mWidth * mHeight;
final int chromaSize = frameSize / 4;
byte[] rgbByteArray = new byte[rgbBuffer.remaining()];
rgbBuffer.get(rgbByteArray);
byte[] yuvByteArray = new byte[inputBufferSize];
Log.d(TAG, "Input Buffer size = " + inputBufferSize);
byte[] yArray = new byte[frameSize];
byte[] uArray = new byte[(frameSize / 4)];
byte[] vArray = new byte[(frameSize / 4)];
isByteBufferEmpty(rgbBuffer, "RGB BUFFER");
int yIndex = 0;
int uIndex = frameSize;
int vIndex = frameSize + chromaSize;
int yByteIndex = 0;
int uvByteIndex = 0;
int R, G, B, Y, U, V;
int index = 0;
//this loop controls the rows
for (int i = 0; i < mHeight; i++) {
//this loop controls the columns
for (int j = 0; j < mWidth; j++) {
R = (rgbByteArray[index] & 0xff0000) >> 16;
G = (rgbByteArray[index] & 0xff00) >> 8;
B = (rgbByteArray[index] & 0xff);
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
//clamp and load in the Y data
yuvByteArray[yIndex++] = (byte) ((Y < 16) ? 16 : ((Y > 235) ? 235 : Y));
yArray[yByteIndex] = (byte) ((Y < 16) ? 16 : ((Y > 235) ? 235 : Y));
yByteIndex++;
if (i % 2 == 0 && index % 2 == 0) {
//clamp and load in the U & V data
yuvByteArray[uIndex++] = (byte) ((U < 16) ? 16 : ((U > 239) ? 239 : U));
yuvByteArray[vIndex++] = (byte) ((V < 16) ? 16 : ((V > 239) ? 239 : V));
uArray[uvByteIndex] = (byte) ((U < 16) ? 16 : ((U > 239) ? 239 : U));
vArray[uvByteIndex] = (byte) ((V < 16) ? 16 : ((V > 239) ? 239 : V));
uvByteIndex++;
}
index++;
}
}
encodeVideoFromImage(yArray, uArray, vArray);
encodeVideoFromBuffer(yuvByteArray);
}
Step 4 Encode the data! I currently have two different ways of doing this, and each has a different output. One uses a ByteBuffer returned from MediaCodec.getInputBuffer();, the other uses an Image returned from MediaCodec.getInputImage();
Encoding using ByteBuffer
private void encodeVideoFromBuffer(byte[] yuvData) {
Log.d(TAG, "In encodeVideo");
int inputSize = 0;
//create index for input buffer
inputBufferIndex = mCodec.dequeueInputBuffer(0);
//create the input buffer for submission to encoder
ByteBuffer inputBuffer = mCodec.getInputBuffer(inputBufferIndex);
//clear, then copy yuv buffer into the input buffer
inputBuffer.clear();
inputBuffer.put(yuvData);
//flip buffer before reading data out of it
inputBuffer.flip();
mCodec.queueInputBuffer(inputBufferIndex, 0, inputBuffer.remaining(), presentationTime, 0);
presentationTime += MICROSECONDS_BETWEEN_FRAMES;
sendToWifi();
}
And the associated output image (note: I took a screenshot of the MP4)
Encoding using Image
private void encodeVideoFromImage(byte[] yToEncode, byte[] uToEncode, byte[]vToEncode) {
Log.d(TAG, "In encodeVideo");
int inputSize = 0;
//create index for input buffer
inputBufferIndex = mCodec.dequeueInputBuffer(0);
//create the input buffer for submission to encoder
Image inputImage = mCodec.getInputImage(inputBufferIndex);
Image.Plane[] inputImagePlanes = inputImage.getPlanes();
ByteBuffer yPlaneBuffer = inputImagePlanes[0].getBuffer();
ByteBuffer uPlaneBuffer = inputImagePlanes[1].getBuffer();
ByteBuffer vPlaneBuffer = inputImagePlanes[2].getBuffer();
yPlaneBuffer.put(yToEncode);
uPlaneBuffer.put(uToEncode);
vPlaneBuffer.put(vToEncode);
yPlaneBuffer.flip();
uPlaneBuffer.flip();
vPlaneBuffer.flip();
mCodec.queueInputBuffer(inputBufferIndex, 0, inputBufferSize, presentationTime, 0);
presentationTime += MICROSECONDS_BETWEEN_FRAMES;
sendToWifi();
}
And the associated output image (note: I took a screenshot of the MP4)
Step 5 Convert H264 Stream to MP4. Finally I grab the output buffer from the codec, and use MediaMuxer to convert the raw h264 stream to an MP4 that I can play and test for correctness
private void sendToWifi() {
Log.d(TAG, "In sendToWifi");
MediaCodec.BufferInfo mBufferInfo = new MediaCodec.BufferInfo();
//Check to see if encoder has output before proceeding
boolean waitingForOutput = true;
boolean outputHasChanged = false;
int outputBufferIndex = 0;
while (waitingForOutput) {
//access the output buffer from the codec
outputBufferIndex = mCodec.dequeueOutputBuffer(mBufferInfo, -1);
if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
outputFormat = mCodec.getOutputFormat();
outputHasChanged = true;
Log.d(TAG, "OUTPUT FORMAT HAS CHANGED");
}
if (outputBufferIndex >= 0) {
waitingForOutput = false;
}
}
//this buffer now contains the compressed YUV data, ready to be sent over WiFi
ByteBuffer outputBuffer = mCodec.getOutputBuffer(outputBufferIndex);
//adjust output buffer position and limit. As of API 19, this is not automatic
if(mBufferInfo.size != 0) {
outputBuffer.position(mBufferInfo.offset);
outputBuffer.limit(mBufferInfo.offset + mBufferInfo.size);
}
////////////////////////////////FOR DEGBUG/////////////////////////////
if (muxerNotStarted && outputHasChanged) {
//set up track
mTrackIndex = mMuxer.addTrack(outputFormat);
mMuxer.start();
muxerNotStarted = false;
}
if (!muxerNotStarted) {
mMuxer.writeSampleData(mTrackIndex, outputBuffer, mBufferInfo);
}
////////////////////////////END DEBUG//////////////////////////////////
//release the buffer
mCodec.releaseOutputBuffer(outputBufferIndex, false);
muxerPasses++;
}
If you've made it this far you're a gentleman (or lady!) and a scholar! Basically I'm stumped as to why my image is not coming out properly. I'm relatively new to video processing so I'm sure I'm just missing something.
If you're API 19+, might as well stick with encoding method #2, getImage()/encodeVideoFromImage(), since that is more modern.
Focusing on that method: One problem was, you had an unexpected image format. With COLOR_FormatYUV420Flexible, you know you're going to have 8-bit U and V components, but you won't know in advance where they go. That's why you have to query the Image.Plane formats. Could be different on every device.
In this case, the UV format turned out to be interleaved (very common on Android devices). If you're using Java, and you supply each array (U/V) separately, with the "stride" requested ("spacer" byte in-between each sample), I believe one array ends up clobbering the other, because these are actually "direct" ByteBuffers, and they were intended to be used from native code, like in this answer. The solution I explained was to copy an interleaved array into the third (V) plane, and ignore the U plane. On the native side, these two planes actually overlap each other in memory (except for the first and last byte), so filling one causes the implementation to fill both.
If you use the second (U) plane instead, you'll find things work, but the colors look funny. That's also because of the overlapping arrangement of these two planes; what that does, effectively, is shift every array element by one byte (which puts U's where V's should be, and vice versa.)
...In other words, this solution is actually a bit of a hack. Probably the only way to do this correctly, and have it work on all devices, is to use native code (as in the answer I linked above).
Once the color plane problem is fixed, that leaves all the funny overlapping text and vertical striations. These were actually caused by your interpretation of the RGB data, which had the wrong stride.
And, once that is fixed, you have a decent-looking picture. It's been mirrored vertically; I don't know the root cause of that, but I suspect it's an OpenGL issue.
I need to use WebRTC for android to send specific cropped(face) video to the videoChannel. I was able manipulate Camera1Session class of WebRTC to get the face cropped. Right now I am setting it to an ImageView.
listenForBytebufferFrames() of Camera1Session.java
private void listenForBytebufferFrames() {
this.camera.setPreviewCallbackWithBuffer(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera callbackCamera) {
Camera1Session.this.checkIsOnCameraThread();
if(callbackCamera != Camera1Session.this.camera) {
Logging.e("Camera1Session", "Callback from a different camera. This should never happen.");
} else if(Camera1Session.this.state != Camera1Session.SessionState.RUNNING) {
Logging.d("Camera1Session", "Bytebuffer frame captured but camera is no longer running.");
} else {
mFrameProcessor.setNextFrame(data, callbackCamera);
long captureTimeNs = TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime());
if(!Camera1Session.this.firstFrameReported) {
int startTimeMs = (int)TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - Camera1Session.this.constructionTimeNs);
Camera1Session.camera1StartTimeMsHistogram.addSample(startTimeMs);
Camera1Session.this.firstFrameReported = true;
}
ByteBuffer byteBuffer1 = ByteBuffer.wrap(data);
Frame outputFrame = new Frame.Builder()
.setImageData(byteBuffer1,
Camera1Session.this.captureFormat.width,
Camera1Session.this.captureFormat.height,
ImageFormat.NV21)
.setTimestampMillis(mFrameProcessor.mPendingTimeMillis)
.setId(mFrameProcessor.mPendingFrameId)
.setRotation(3)
.build();
int w = outputFrame.getMetadata().getWidth();
int h = outputFrame.getMetadata().getHeight();
SparseArray<Face> detectedFaces = mDetector.detect(outputFrame);
if (detectedFaces.size() > 0) {
Face face = detectedFaces.valueAt(0);
ByteBuffer byteBufferRaw = outputFrame.getGrayscaleImageData();
byte[] byteBuffer = byteBufferRaw.array();
YuvImage yuvimage = new YuvImage(byteBuffer, ImageFormat.NV21, w, h, null);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
//My crop logic to get face co-ordinates
yuvimage.compressToJpeg(new Rect(left, top, right, bottom), 80, baos);
final byte[] jpegArray = baos.toByteArray();
Bitmap bitmap = BitmapFactory.decodeByteArray(jpegArray, 0, jpegArray.length);
Activity currentActivity = getActivity();
if (currentActivity instanceof CallActivity) {
((CallActivity) currentActivity).setBitmapToImageView(bitmap); //face on ImageView is set just fine
}
Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, data, Camera1Session.this.captureFormat.width, Camera1Session.this.captureFormat.height, Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(data);
} else {
Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, data, Camera1Session.this.captureFormat.width, Camera1Session.this.captureFormat.height, Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(data);
}
}
}
});
}
jpegArray is the final byteArray that I need to stream via WebRTC, which I tried with something like this:
Camera1Session.this.events.onByteBufferFrameCaptured(Camera1Session.this, jpegArray, (int) face.getWidth(), (int) face.getHeight(), Camera1Session.this.getFrameOrientation(), captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(jpegArray);
Setting them up like this gives me following error:
../../webrtc/sdk/android/src/jni/androidvideotracksource.cc line 82
Check failed: length >= width * height + 2 * uv_width * ((height + 1) / 2) (2630 vs. 460800)
Which I assume is because androidvideotracksource does not get the same length of byteArray that it expects, since the frame is cropped now.
Could someone point me in the direction of how to achieve it? Is this the correct way/place to manipulate the data and feed into the videoTrack?
Edit:bitmap of byteArray data does not give me a camera preview on ImageView, unlike byteArray jpegArray. Maybe because they are packed differently?
Can we use WebRTC's Datachannel to exchang custom data ie cropped face "image" in your case and do the respective calculation at receiving end using any third party library ie OpenGL etc? Reason I am suggesting is that the WebRTC Video feed received from channel is a stream in real time not a bytearray . WebRTC Video by its inherent architecture isn't meant to crop video at other hand. If we want to crop or augment video we have to use any ar library to fulfill this job.
We can always leverage WebRTC's Data channel to exchange customized data. Using Video channel for the same is not recommended because it's real time stream not the bytearray.Please revert in case of any concern.
WebRTC in particular and video streaming in general presumes that the video has fixed dimensions. If you want to crop the detected face, your options are either to have pad the cropped image with e.g. black pixels (WebRTC does not use transparency), and crop the video on the receiver side, or, if you don't have control over the receiver, resize the cropped region to fill the expected width * height frame (you should also keep the expected aspect ratio).
Note that JPEG compress/decompress that you use to crop the original is far from efficient. Some other options can be found in Image crop and resize in Android.
Okay, this was definitely a problem of how the original byte[] data was packed and the way byte[] jpegArray was packed. Changing the way of packing this and scaling it as AlexCohn suggested worked for me. I found help from other post on StackOverflow on way to pack it. This is the code for it:
private byte[] getNV21(int left, int top, int inputWidth, int inputHeight, Bitmap scaled) {
int [] argb = new int[inputWidth * inputHeight];
scaled.getPixels(argb, 0, inputWidth, left, top, inputWidth, inputHeight);
byte [] yuv = new byte[inputWidth*inputHeight*3/2];
encodeYUV420SP(yuv, argb, inputWidth, inputHeight);
scaled.recycle();
return yuv;
}
private void encodeYUV420SP(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ( ( 66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ( ( -38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ( ( 112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
// NV21 has a plane of Y and interleaved planes of VU each sampled by a factor of 2
// meaning for every 4 Y pixels there are 1 V and 1 U. Note the sampling is every other
// pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0 : ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[uvIndex++] = (byte)((V<0) ? 0 : ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte)((U<0) ? 0 : ((U > 255) ? 255 : U));
}
index ++;
}
}
}`
I pass this byte[] data to onByteBufferFrameCaptured and callback:
Camera1Session.this.events.onByteBufferFrameCaptured(
Camera1Session.this,
data,
w,
h,
Camera1Session.this.getFrameOrientation(),
captureTimeNs);
Camera1Session.this.camera.addCallbackBuffer(data);
Prior to this, I had to scale the bitmap which is pretty straight forward:
int width = bitmapToScale.getWidth();
int height = bitmapToScale.getHeight();
Matrix matrix = new Matrix();
matrix.postScale(newWidth / width, newHeight / height);
Bitmap scaledBitmap = Bitmap.createBitmap(bitmapToScale, 0, 0, bitmapToScale.getWidth(), bitmapToScale.getHeight(), matrix, true);
I want to call a function and build a video out of list of images, and then save it locally on the device:
public void CreateAndSaveVideoFile(List<Bitmap> MyBitmapArray)
{
// ..
}
Trials:
Following java/xuggle - encode array of images into a movie, the link in the answer is a dead link
Following How to encode images into a video file in Java through programming?, The suggested library in the accepted answer does not support Android.
The next answer in the above has an approach for Android users however it is not clear for me the input and the output of that function (where did he give the images? and where did he get the video?) - I left a question comment
The next answer in the above provides a whole class, however the required library to be included has a corrupted file (when I try and download it from the provided link) - I left a question comment
Following Java: How do I create a movie from an array of images?, the suggested library in the top answer uses commands that I am not familiar with and I don't even know how to use them. Like:
Creating an MPEG-4 file from all the JPEG files in the current
directory:
mencoder mf://*.jpg -mf w=800:h=600:fps=25:type=jpg -ovc lavc \
-lavcopts vcodec=mpeg4:mbd=2:trell -oac copy -o output.avi
I don't know how can I use the above in a Java / Android project..
Can anyone help in guiding me or/and providing me with an approach to my task? Thanks in advance.
You can use jcodec SequenceEncoder to convert sequence of images to MP4 file.
Sample code :
import org.jcodec.api.awt.SequenceEncoder;
...
SequenceEncoder enc = new SequenceEncoder(new File("filename"));
// GOP size will be supported in 0.2
// enc.getEncoder().setKeyInterval(25);
for(...) {
BufferedImage image = ... // Obtain an image to encode
enc.encodeImage(image);
}
enc.finish();
It's a java library so it's easy to import it into Android project, you don't have to use NDK unlike ffmpeg.
Refer http://jcodec.org/ for sample code & downloads.
Using JCodec as demonstrated by Stanislav Vitvitskyy here.
public static void main(String[] args) throws IOException {
SequenceEncoder encoder = new SequenceEncoder(new File("video.mp4"));
for (int i = 1; i < 100; i++) {
BufferedImage bi = ImageIO.read(new File(String.format("img%08d.png", i)));
encoder.encodeImage(bi);
}
encoder.finish();}
Now to convert your Bitmap to BufferedImage you can use this class:
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.awt.image.DataBufferInt;
import java.io.IOException;
import java.io.InputStream;
/**
* Utility class for loading windows bitmap files
* <p>
* Based on code from author Abdul Bezrati and Pepijn Van Eeckhoudt
*/
public class BitmapLoader {
/**
* Static method to load a bitmap file based on the filename passed in.
* Based on the bit count, this method will either call the 8 or 24 bit
* bitmap reader methods
*
* #param file The name of the bitmap file to read
* #throws IOException
* #return A BufferedImage of the bitmap
*/
public static BufferedImage loadBitmap(String file) throws IOException {
BufferedImage image;
InputStream input = null;
try {
input = ResourceRetriever.getResourceAsStream(file);
int bitmapFileHeaderLength = 14;
int bitmapInfoHeaderLength = 40;
byte bitmapFileHeader[] = new byte[bitmapFileHeaderLength];
byte bitmapInfoHeader[] = new byte[bitmapInfoHeaderLength];
input.read(bitmapFileHeader, 0, bitmapFileHeaderLength);
input.read(bitmapInfoHeader, 0, bitmapInfoHeaderLength);
int nSize = bytesToInt(bitmapFileHeader, 2);
int nWidth = bytesToInt(bitmapInfoHeader, 4);
int nHeight = bytesToInt(bitmapInfoHeader, 8);
int nBiSize = bytesToInt(bitmapInfoHeader, 0);
int nPlanes = bytesToShort(bitmapInfoHeader, 12);
int nBitCount = bytesToShort(bitmapInfoHeader, 14);
int nSizeImage = bytesToInt(bitmapInfoHeader, 20);
int nCompression = bytesToInt(bitmapInfoHeader, 16);
int nColoursUsed = bytesToInt(bitmapInfoHeader, 32);
int nXPixelsMeter = bytesToInt(bitmapInfoHeader, 24);
int nYPixelsMeter = bytesToInt(bitmapInfoHeader, 28);
int nImportantColours = bytesToInt(bitmapInfoHeader, 36);
if (nBitCount == 24) {
image = read24BitBitmap(nSizeImage, nHeight, nWidth, input);
} else if (nBitCount == 8) {
image = read8BitBitmap(nColoursUsed, nBitCount, nSizeImage, nWidth, nHeight, input);
} else {
System.out.println("Not a 24-bit or 8-bit Windows Bitmap, aborting...");
image = null;
}
} finally {
try {
if (input != null)
input.close();
} catch (IOException e) {
}
}
return image;
}
/**
* Static method to read a 8 bit bitmap
*
* #param nColoursUsed Number of colors used
* #param nBitCount The bit count
* #param nSizeImage The size of the image in bytes
* #param nWidth The width of the image
* #param input The input stream corresponding to the image
* #throws IOException
* #return A BufferedImage of the bitmap
*/
private static BufferedImage read8BitBitmap(int nColoursUsed, int nBitCount, int nSizeImage, int nWidth, int nHeight, InputStream input) throws IOException {
int nNumColors = (nColoursUsed > 0) ? nColoursUsed : (1 & 0xff) << nBitCount;
if (nSizeImage == 0) {
nSizeImage = ((((nWidth * nBitCount) + 31) & ~31) >> 3);
nSizeImage *= nHeight;
}
int npalette[] = new int[nNumColors];
byte bpalette[] = new byte[nNumColors * 4];
readBuffer(input, bpalette);
int nindex8 = 0;
for (int n = 0; n < nNumColors; n++) {
npalette[n] = (255 & 0xff) << 24 |
(bpalette[nindex8 + 2] & 0xff) << 16 |
(bpalette[nindex8 + 1] & 0xff) << 8 |
(bpalette[nindex8 + 0] & 0xff);
nindex8 += 4;
}
int npad8 = (nSizeImage / nHeight) - nWidth;
BufferedImage bufferedImage = new BufferedImage(nWidth, nHeight, BufferedImage.TYPE_INT_ARGB);
DataBufferInt dataBufferByte = ((DataBufferInt) bufferedImage.getRaster().getDataBuffer());
int[][] bankData = dataBufferByte.getBankData();
byte bdata[] = new byte[(nWidth + npad8) * nHeight];
readBuffer(input, bdata);
nindex8 = 0;
for (int j8 = nHeight - 1; j8 >= 0; j8--) {
for (int i8 = 0; i8 < nWidth; i8++) {
bankData[0][j8 * nWidth + i8] = npalette[((int) bdata[nindex8] & 0xff)];
nindex8++;
}
nindex8 += npad8;
}
return bufferedImage;
}
/**
* Static method to read a 24 bit bitmap
*
* #param nSizeImage size of the image in bytes
* #param nHeight The height of the image
* #param nWidth The width of the image
* #param input The input stream corresponding to the image
* #throws IOException
* #return A BufferedImage of the bitmap
*/
private static BufferedImage read24BitBitmap(int nSizeImage, int nHeight, int nWidth, InputStream input) throws IOException {
int npad = (nSizeImage / nHeight) - nWidth * 3;
if (npad == 4 || npad < 0)
npad = 0;
int nindex = 0;
BufferedImage bufferedImage = new BufferedImage(nWidth, nHeight, BufferedImage.TYPE_4BYTE_ABGR);
DataBufferByte dataBufferByte = ((DataBufferByte) bufferedImage.getRaster().getDataBuffer());
byte[][] bankData = dataBufferByte.getBankData();
byte brgb[] = new byte[(nWidth + npad) * 3 * nHeight];
readBuffer(input, brgb);
for (int j = nHeight - 1; j >= 0; j--) {
for (int i = 0; i < nWidth; i++) {
int base = (j * nWidth + i) * 4;
bankData[0][base] = (byte) 255;
bankData[0][base + 1] = brgb[nindex];
bankData[0][base + 2] = brgb[nindex + 1];
bankData[0][base + 3] = brgb[nindex + 2];
nindex += 3;
}
nindex += npad;
}
return bufferedImage;
}
/**
* Converts bytes to an int
*
* #param bytes An array of bytes
* #param index
* #returns A int representation of the bytes
*/
private static int bytesToInt(byte[] bytes, int index) {
return (bytes[index + 3] & 0xff) << 24 |
(bytes[index + 2] & 0xff) << 16 |
(bytes[index + 1] & 0xff) << 8 |
bytes[index + 0] & 0xff;
}
/**
* Converts bytes to a short
*
* #param bytes An array of bytes
* #param index
* #returns A short representation of the bytes
*/
private static short bytesToShort(byte[] bytes, int index) {
return (short) (((bytes[index + 1] & 0xff) << 8) |
(bytes[index + 0] & 0xff));
}
/**
* Reads the buffer
*
* #param in An InputStream
* #param buffer An array of bytes
* #throws IOException
*/
private static void readBuffer(InputStream in, byte[] buffer) throws IOException {
int bytesRead = 0;
int bytesToRead = buffer.length;
while (bytesToRead > 0) {
int read = in.read(buffer, bytesRead, bytesToRead);
bytesRead += read;
bytesToRead -= read;
}
}
}
If the minimum version of you application Android SDK is greater or equal to 16 (Android 4.1) the best way of video encoding is use Android Media Codec API.
From Android 4.3 APIs.
When encoding video, Android 4.1 (SDK 16) required that you provide
the media with a ByteBuffer array, but Android 4.3 (SDK 18) now allows
you to use a Surface as the input to an encoder. For instance, this
allows you to encode input from an existing video file or using frames
generated from OpenGL ES.
Media Muxer added in Android 4.3 (SDK 18) so for convenient way of writing mp4 file with Media Muxer you should have SDK>=18.
Using Media Codec API way you will get hardware accelerated encoding and you are easily encode up to 60 FPS.
You can start from 1) How to encode Bitmaps into a video using MediaCodec?
or use 2) Google Grafika or 3) Bigflake.
Starting from Grafika RecordFBOActivity.java. Replace Choreographer event with you own containing bitmap to encode, remove On Screen drawing, load you bitmap as Open GL Texture and draw it on Media Codec Input Surface.
jCodec has added Android support.
You need to add these to your gradle...
implementation 'org.jcodec:jcodec:0.2.3'
implementation 'org.jcodec:jcodec-android:0.2.3'
...and
android {
...
configurations.all {
resolutionStrategy.force 'com.google.code.findbugs:jsr305:3.0.2'
}
}
I can confirm this works as expected, but with caveats. First being I tried some full size images and the file wrote, but gave an error on playback. When I scaled down, I would get an error if the width or height of the image was not even because it requires a multiple of 2 for YUV420J colorspace.
Also worthy of note, this makes your package HEAVY, heavy. My small project went over the dex limit by adding this and required enabling multidex.
FileChannelWrapper out = null;
File dir = what ever directory you use...
File file = new File(dir, "test.mp4");
try { out = NIOUtils.writableFileChannel(file.getAbsolutePath());
AndroidSequenceEncoder encoder = new AndroidSequenceEncoder(out, Rational.R(15, 1));
for (Bitmap bitmap : bitmaps) {
encoder.encodeImage(bitmap);
}
encoder.finish();
} finally {
NIOUtils.closeQuietly(out);
}
You can use Bitmp4 to convert sequence of images to MP4 file.
Sample code :
...
val encoder = MP4Encoder()
encoder.setFrameDelay(50)
encoder.setOutputFilePath(exportedFile.path)
encoder.setOutputSize(width, width)
startExport()
stopExport()
addFrame(bitmap) //called intervally
It's a java library so it's easy to import it into Android project, you don't have to use NDK unlike ffmpeg.
Refer https://github.com/dbof10/Bitmp4 for sample code & downloads.
I created a project that should be able to handle this. The code is light and fairly straight forward.
https://github.com/dburckh/bitmap2video
Abhishek V was right, more information about jcodec SequenceEncoder:
see Android make animated video from list of images
Recently I have built a real-time video system using raspberry pi and Android devices, met the same problem as yours. Instead of saving a list of image files, I used some real-time streaming protocols like RTP/RTCP to transfer data stream to user. If your requirement is something like this, maybe you could change your strategies.
Another suggestion is that you may explore some C/C++ libraries, using NDK/JNI to break the limitation of Java.
Hope the suggestions make sense to you :)
I'm currently trying to understand one of the sample provided in the FastCV package. there is a function doing memory allocation fcvMemAlloc()which takes as input the number of bytes and the byte alignment.
In the sample called FastCVSample.cpp, memory has to be allocated to a data block of size w x h, however, while allocating the memory they divide by 2 the total amount. I don't understand why? If someone has a clue, I'll be really happy to hear from him :-)
Here is the function call - see bellow the call to fcvMemAlloc()
JNIEXPORT void
JNICALL Java_com_qualcomm_fastcorner_FastCVSample_update
(
JNIEnv* env,
jobject obj,
jbyteArray img,
jint w,
jint h
)
{
jbyte* jimgData = NULL;
jboolean isCopy = 0;
uint32_t* curCornerPtr = 0;
uint8_t* renderBuffer;
uint64_t time;
float timeMs;
// Get data from JNI
jimgData = env->GetByteArrayElements( img, &isCopy );
renderBuffer = getRenderBuffer( w, h );
lockRenderBuffer();
time = getTimeMicroSeconds();
// jimgData might not be 128 bit aligned.
// fcvColorYUV420toRGB565u8() and other fcv functionality inside
// updateCorners() require 128 bit memory aligned. In case of jimgData
// is not 128 bit aligned, it will allocate memory that is 128 bit
// aligned and copies jimgData to the aligned memory.
uint8_t* pJimgData = (uint8_t*)jimgData;
// Check if camera image data is not aligned.
if( (uintptr_t)jimgData & 0xF )
{
// Allow for rescale if dimensions changed.
if( w != (int)state.alignedImgWidth ||
h != (int)state.alignedImgHeight )
{
if( state.alignedImgBuf != NULL )
{
DPRINTF( "%s %d Creating aligned for preview\n",
__FILE__, __LINE__ );
fcvMemFree( state.alignedImgBuf );
state.alignedImgBuf = NULL;
}
}
// Allocate buffer for aligned data if necessary.
if( state.alignedImgBuf == NULL )
{
state.alignedImgWidth = w;
state.alignedImgHeight = h;
state.alignedImgBuf = (uint8_t*)fcvMemAlloc( w*h*3/2, 16 ); <-----Why this and not fcvMemAlloc( w*h*3, 16 )
}
memcpy( state.alignedImgBuf, jimgData, w*h*3/2 ); <---- same here
pJimgData = state.alignedImgBuf;
}
// Copy the image first in our own buffer to avoid corruption during
// rendering. Not that we can still have corruption in image while we do
// copy but we can't help that.
// if viewfinder is disabled, simply set to gray
if( state.disableVF )
{
// Loop through RGB565 values and set to gray.
uint32_t size = getRenderBufferSize();
for( uint32_t i=0; i<size; i+=2 )
{
renderBuffer[i] = 0x10;
renderBuffer[i+1] = 0x84;
}
}
else
{
fcvColorYUV420toRGB565u8(
pJimgData,
w,
h,
(uint32_t*)renderBuffer );
}
// Perform FastCV Corner processing
updateCorners( (uint8_t*)pJimgData, w, h );
timeMs = ( getTimeMicroSeconds() - time ) / 1000.f;
state.timeFilteredMs =
((state.timeFilteredMs*(29.f/30.f)) + (float)(timeMs/30.f));
// RGB Color conversion
if( !state.enableOverlayPixels )
{
state.numCorners = 0;
}
// Have renderer draw corners on render buffer.
drawCorners( state.corners, state.numCorners );
unlockRenderBuffer();
// Let JNI know we don't need data anymore. this is important!
env->ReleaseByteArrayElements( img, jimgData, JNI_ABORT );
}
I've found the answer at the following site:
How to render Android's YUV-NV21 camera image on the background in libgdx with OpenGLES 2.0 in real-time?
he explained there that the format of a YUV frame is (w x h x 3)/2 that's why this specific amount of memory is allocated.
NOTE: there is another example here:
http://www.codeproject.com/Tips/691062/Resizing-NV-image-using-Nearest-Neighbor-Interpo
I'm trying to get the picture from a surfaceView where I have the camera view running,
I've already implemented onPreviewFrame, and it's called correctly as the debug shows me.
The problem I'm facing now, it's since the byte[] data I receive in the method, it's in YUV space color (NV21), I'm trying to convert it to grayscale to generate a Bitmap and then storing it into a file.
The conversion process that I'm following it's:
public Bitmap convertYuvGrayScaleRGB(byte[] yuv, int width, int height) {
int[] pixels = new int[width * height];
for (int i = 0; i < height*width; i++) {
int grey = yuv[i] & 0xff;
pixels[i] = 0xFF000000 | (grey * 0x00010101);
}
return Bitmap.createBitmap(pixels, width, height, Bitmap.Config.ARGB_8888);
}
The importing procedure for storing it to a file, it's:
Bitmap bitmap = convertYuvGrayScaleRGB(data,widht,heigth);
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 50, bytes);
File f = new File(Environment.getExternalStorageDirectory()
+ File.separator + "test.jpg");
Log.d("Camera", "File: " + f.getAbsolutePath());
try {
f.createNewFile();
FileOutputStream fo = new FileOutputStream(f);
fo.write(bytes.toByteArray());
fo.close();
bitmap.recycle();
bitmap = null;
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
Altough, the result I've got it's the following:
I can't find any obvious mistake in your code, but i've already met this kind of skewed images before. When this happened to me, it was due to:
At some point in the code, the image width and height are swapped,
Or the original image you're trying to convert has padding, in which case you will need a stride in addition of the width and height.
Hope this helps!
Probably the Width of the image you are converting is not even. in that case
it is padded in memory.
Let me have a look at the docs...
It seems more complicated than this. if you want your code to work as it is now, you will have to have the width
a multiple of 16.
from the docs:
public static final int YV12
Added in API level 9 Android YUV format.
This format is exposed to software decoders and applications.
YV12 is a 4:2:0 YCrCb planar format comprised of a WxH Y plane
followed by (W/2) x (H/2) Cr and Cb planes.
This format assumes
an even width an even height a horizontal stride multiple of 16 pixels
a vertical stride equal to the height y_size = stride * height
c_stride = ALIGN(stride/2, 16) c_size = c_stride * height/2 size =
y_size + c_size * 2 cr_offset = y_size cb_offset = y_size + c_size
I just had this problem with the S3. My problem was that I used the wrong dimensions for the preview. I assumed the camera was 16:9 when it was actually 4:3.
Use Camera.getParameters().getPreviewSize() to see what the output is in.
I made this:
int frameSize = width * height;
for (int i = 0; i < height; i++) {
for (int j = 0; j < width; j++) {
ret[frameSize + (i >> 1) * width + (j & ~1) + 1] = 127; //U
ret[frameSize + (i >> 1) * width + (j & ~1) + 0] = 127; //V
}
}
So simple but it works really good and fast ;)