Develop Reference

Android Creating Video from Screen Scraping: Why is output Image wonky?

Update #6 Discovered I was accessing RGB values improperly. I assumed I was accessing data from an Int[], but was instead accessing byte information from a Byte[]. Changed to accessing from Int[] and get the following image:
Update #5 Adding code used to get RGBA ByteBuffer for reference
private void screenScrape() {
Log.d(TAG, "In screenScrape");
//read pixels from frame buffer into PBO (GL_PIXEL_PACK_BUFFER)
mSurface.queueEvent(new Runnable() {
#Override
public void run() {
Log.d(TAG, "In Screen Scrape 1");
//generate and bind buffer ID
GLES30.glGenBuffers(1, pboIds);
checkGlError("Gen Buffers");
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, pboIds.get(0));
checkGlError("Bind Buffers");
//creates and initializes data store for PBO. Any pre-existing data store is deleted
GLES30.glBufferData(GLES30.GL_PIXEL_PACK_BUFFER, (mWidth * mHeight * 4), null, GLES30.GL_STATIC_READ);
checkGlError("Buffer Data");
//glReadPixelsPBO(0,0,w,h,GLES30.GL_RGB,GLES30.GL_UNSIGNED_SHORT_5_6_5,0);
glReadPixelsPBO(0, 0, mWidth, mHeight, GLES30.GL_RGBA, GLES30.GL_UNSIGNED_BYTE, 0);
checkGlError("Read Pixels");
//GLES30.glReadPixels(0,0,w,h,GLES30.GL_RGBA,GLES30.GL_UNSIGNED_BYTE,intBuffer);
}
});
//map PBO data into client address space
mSurface.queueEvent(new Runnable() {
#Override
public void run() {
Log.d(TAG, "In Screen Scrape 2");
//read pixels from PBO into a byte buffer for processing. Unmap buffer for use in next pass
mapBuffer = ((ByteBuffer) GLES30.glMapBufferRange(GLES30.GL_PIXEL_PACK_BUFFER, 0, 4 * mWidth * mHeight, GLES30.GL_MAP_READ_BIT)).order(ByteOrder.nativeOrder());
checkGlError("Map Buffer");
GLES30.glUnmapBuffer(GLES30.GL_PIXEL_PACK_BUFFER);
checkGlError("Unmap Buffer");
isByteBufferEmpty(mapBuffer, "MAP BUFFER");
convertColorSpaceByteArray(mapBuffer);
mapBuffer.clear();
}
});
}
Update #4 For reference, here is the original image to compare against.
Update #3 This is the output image after interleaving all U/V data into a single array and passing it to the Image object at inputImagePlanes[1]; inputImagePlanes[2]; is unused;
The next image is the same interleaved UV data, but we load this into inputImagePlanes[2]; instead of inputImagePlanes[1];
Update #2 This is the output image after padding the U/V buffers with a zero in between each byte of 'real' data. uArray[uvByteIndex] = (byte) 0;
Update #1 As suggested by a comment, here are the row and pixel strides I get from calling getPixelStride and getRowStride
Y Plane Pixel Stride = 1, Row Stride = 960
U Plane Pixel Stride = 2, Row Stride = 960
V Plane Pixel Stride = 2, Row Stride = 960
The goal of my application is to read pixels out from the screen, compress them, and then send that h264 stream over WiFi to be played be a receiver.
Currently I'm using the MediaMuxer class to convert the raw h264 stream to an MP4, and then save it to file. However the end result video is messed up and I can't figure out why. Lets walk through some of processing and see if we can find anything that jumps out.
Step 1 Set up the encoder. I'm currently taking screen images once every 2 seconds, and using "video/avc" for MIME_TYPE
//create codec for compression
try {
mCodec = MediaCodec.createEncoderByType(MIME_TYPE);
} catch (IOException e) {
Log.d(TAG, "FAILED: Initializing Media Codec");
}
//set up format for codec
MediaFormat mFormat = MediaFormat.createVideoFormat(MIME_TYPE, mWidth, mHeight);
mFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible);
mFormat.setInteger(MediaFormat.KEY_BIT_RATE, 16000000);
mFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 1/2);
mFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
Step 2 Read pixels out from screen. This is done using openGL ES, and the pixels are read out in RGBA format. (I've confirmed this part to be working)
Step 3 Convert the RGBA pixels to YUV420 (IYUV) format. This is done using the following method. Note that I have 2 methods for encoding called at the end of this method.
private void convertColorSpaceByteArray(ByteBuffer rgbBuffer) {
long startTime = System.currentTimeMillis();
Log.d(TAG, "In convertColorspace");
final int frameSize = mWidth * mHeight;
final int chromaSize = frameSize / 4;
byte[] rgbByteArray = new byte[rgbBuffer.remaining()];
rgbBuffer.get(rgbByteArray);
byte[] yuvByteArray = new byte[inputBufferSize];
Log.d(TAG, "Input Buffer size = " + inputBufferSize);
byte[] yArray = new byte[frameSize];
byte[] uArray = new byte[(frameSize / 4)];
byte[] vArray = new byte[(frameSize / 4)];
isByteBufferEmpty(rgbBuffer, "RGB BUFFER");
int yIndex = 0;
int uIndex = frameSize;
int vIndex = frameSize + chromaSize;
int yByteIndex = 0;
int uvByteIndex = 0;
int R, G, B, Y, U, V;
int index = 0;
//this loop controls the rows
for (int i = 0; i < mHeight; i++) {
//this loop controls the columns
for (int j = 0; j < mWidth; j++) {
R = (rgbByteArray[index] & 0xff0000) >> 16;
G = (rgbByteArray[index] & 0xff00) >> 8;
B = (rgbByteArray[index] & 0xff);
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
//clamp and load in the Y data
yuvByteArray[yIndex++] = (byte) ((Y < 16) ? 16 : ((Y > 235) ? 235 : Y));
yArray[yByteIndex] = (byte) ((Y < 16) ? 16 : ((Y > 235) ? 235 : Y));
yByteIndex++;
if (i % 2 == 0 && index % 2 == 0) {
//clamp and load in the U & V data
yuvByteArray[uIndex++] = (byte) ((U < 16) ? 16 : ((U > 239) ? 239 : U));
yuvByteArray[vIndex++] = (byte) ((V < 16) ? 16 : ((V > 239) ? 239 : V));
uArray[uvByteIndex] = (byte) ((U < 16) ? 16 : ((U > 239) ? 239 : U));
vArray[uvByteIndex] = (byte) ((V < 16) ? 16 : ((V > 239) ? 239 : V));
uvByteIndex++;
}
index++;
}
}
encodeVideoFromImage(yArray, uArray, vArray);
encodeVideoFromBuffer(yuvByteArray);
}
Step 4 Encode the data! I currently have two different ways of doing this, and each has a different output. One uses a ByteBuffer returned from MediaCodec.getInputBuffer();, the other uses an Image returned from MediaCodec.getInputImage();
Encoding using ByteBuffer
private void encodeVideoFromBuffer(byte[] yuvData) {
Log.d(TAG, "In encodeVideo");
int inputSize = 0;
//create index for input buffer
inputBufferIndex = mCodec.dequeueInputBuffer(0);
//create the input buffer for submission to encoder
ByteBuffer inputBuffer = mCodec.getInputBuffer(inputBufferIndex);
//clear, then copy yuv buffer into the input buffer
inputBuffer.clear();
inputBuffer.put(yuvData);
//flip buffer before reading data out of it
inputBuffer.flip();
mCodec.queueInputBuffer(inputBufferIndex, 0, inputBuffer.remaining(), presentationTime, 0);
presentationTime += MICROSECONDS_BETWEEN_FRAMES;
sendToWifi();
}
And the associated output image (note: I took a screenshot of the MP4)
Encoding using Image
private void encodeVideoFromImage(byte[] yToEncode, byte[] uToEncode, byte[]vToEncode) {
Log.d(TAG, "In encodeVideo");
int inputSize = 0;
//create index for input buffer
inputBufferIndex = mCodec.dequeueInputBuffer(0);
//create the input buffer for submission to encoder
Image inputImage = mCodec.getInputImage(inputBufferIndex);
Image.Plane[] inputImagePlanes = inputImage.getPlanes();
ByteBuffer yPlaneBuffer = inputImagePlanes[0].getBuffer();
ByteBuffer uPlaneBuffer = inputImagePlanes[1].getBuffer();
ByteBuffer vPlaneBuffer = inputImagePlanes[2].getBuffer();
yPlaneBuffer.put(yToEncode);
uPlaneBuffer.put(uToEncode);
vPlaneBuffer.put(vToEncode);
yPlaneBuffer.flip();
uPlaneBuffer.flip();
vPlaneBuffer.flip();
mCodec.queueInputBuffer(inputBufferIndex, 0, inputBufferSize, presentationTime, 0);
presentationTime += MICROSECONDS_BETWEEN_FRAMES;
sendToWifi();
}
And the associated output image (note: I took a screenshot of the MP4)
Step 5 Convert H264 Stream to MP4. Finally I grab the output buffer from the codec, and use MediaMuxer to convert the raw h264 stream to an MP4 that I can play and test for correctness
private void sendToWifi() {
Log.d(TAG, "In sendToWifi");
MediaCodec.BufferInfo mBufferInfo = new MediaCodec.BufferInfo();
//Check to see if encoder has output before proceeding
boolean waitingForOutput = true;
boolean outputHasChanged = false;
int outputBufferIndex = 0;
while (waitingForOutput) {
//access the output buffer from the codec
outputBufferIndex = mCodec.dequeueOutputBuffer(mBufferInfo, -1);
if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
outputFormat = mCodec.getOutputFormat();
outputHasChanged = true;
Log.d(TAG, "OUTPUT FORMAT HAS CHANGED");
}
if (outputBufferIndex >= 0) {
waitingForOutput = false;
}
}
//this buffer now contains the compressed YUV data, ready to be sent over WiFi
ByteBuffer outputBuffer = mCodec.getOutputBuffer(outputBufferIndex);
//adjust output buffer position and limit. As of API 19, this is not automatic
if(mBufferInfo.size != 0) {
outputBuffer.position(mBufferInfo.offset);
outputBuffer.limit(mBufferInfo.offset + mBufferInfo.size);
}
////////////////////////////////FOR DEGBUG/////////////////////////////
if (muxerNotStarted && outputHasChanged) {
//set up track
mTrackIndex = mMuxer.addTrack(outputFormat);
mMuxer.start();
muxerNotStarted = false;
}
if (!muxerNotStarted) {
mMuxer.writeSampleData(mTrackIndex, outputBuffer, mBufferInfo);
}
////////////////////////////END DEBUG//////////////////////////////////
//release the buffer
mCodec.releaseOutputBuffer(outputBufferIndex, false);
muxerPasses++;
}
If you've made it this far you're a gentleman (or lady!) and a scholar! Basically I'm stumped as to why my image is not coming out properly. I'm relatively new to video processing so I'm sure I'm just missing something.

If you're API 19+, might as well stick with encoding method #2, getImage()/encodeVideoFromImage(), since that is more modern.
Focusing on that method: One problem was, you had an unexpected image format. With COLOR_FormatYUV420Flexible, you know you're going to have 8-bit U and V components, but you won't know in advance where they go. That's why you have to query the Image.Plane formats. Could be different on every device.
In this case, the UV format turned out to be interleaved (very common on Android devices). If you're using Java, and you supply each array (U/V) separately, with the "stride" requested ("spacer" byte in-between each sample), I believe one array ends up clobbering the other, because these are actually "direct" ByteBuffers, and they were intended to be used from native code, like in this answer. The solution I explained was to copy an interleaved array into the third (V) plane, and ignore the U plane. On the native side, these two planes actually overlap each other in memory (except for the first and last byte), so filling one causes the implementation to fill both.
If you use the second (U) plane instead, you'll find things work, but the colors look funny. That's also because of the overlapping arrangement of these two planes; what that does, effectively, is shift every array element by one byte (which puts U's where V's should be, and vice versa.)
...In other words, this solution is actually a bit of a hack. Probably the only way to do this correctly, and have it work on all devices, is to use native code (as in the answer I linked above).
Once the color plane problem is fixed, that leaves all the funny overlapping text and vertical striations. These were actually caused by your interpretation of the RGB data, which had the wrong stride.
And, once that is fixed, you have a decent-looking picture. It's been mirrored vertically; I don't know the root cause of that, but I suspect it's an OpenGL issue.

Media Codec - YUV_YV12 to YUV420SP/NV21

I'm trying to develop an android application that encodes opencv array of Mats of resolution 1200x1200 4 channel images to a mp4 video using Android media codec. The problem I'm facing is that, when I'm trying to use emulator(which uses YUV420P color format - I'm using OpenCV COLOR_BGRA2YUV_I420 conversion) the video output color is same as the array of images, but on real android devices the color output is completely different, so I've debugged and found out that my android devices color format is YUV420SP. Since there are no inbuilt opencv functions to convert RGBA/BGRA to YUV420SP, I've converted image to YUV_YV12 and then to YUV420SP/NV21 using the below code
public byte[] YV12toNV21(final byte[] input, final int width, final int height) {
byte[] output = input;
final int size = width * height;
final int quarter = size / 4;
final int vPosition = size; // This is where V starts
final int uPosition = size + quarter; // This is where U starts
System.arraycopy(input, 0, output, 0, size); // Y is same
for (int i = 0; i < quarter; i++) {
output[size + i*2 ] = input[vPosition + i]; // For NV21, V first
output[size + i*2 + 1] = input[uPosition + i]; // For Nv21, U second
}
return output;
}
But still I'm facing the same issue.
This is the Original RGBA Picture
This one is the output of video
UPDATE:
After swapping U and V

Memory allocation for YUV buffer to convert into RGB

I'm facing issue on few android devices while copying the data from DecodeFrame2()
This is my code:
uint8_t* m_yuvData[3];
SBufferInfo yuvDataInfo;
memset(&yuvDataInfo, 0, sizeof(SBufferInfo));
m_yuvData[0] = NULL;
m_yuvData[1] = NULL;
m_yuvData[2] = NULL;
DECODING_STATE decodingState = m_decoder->DecodeFrame2(bufferData, bufferDataSize, m_yuvData, &yuvDataInfo);
if(yuvDataInfo.iBufferStatus == 1)
{
int yStride = yuvDataInfo->UsrData.sSystemBuffer.iStride[0];
int uvStride = yuvDataInfo->UsrData.sSystemBuffer.iStride[1];
uint32_t width = yuvDataInfo->UsrData.sSystemBuffer.iWidth;
uint32_t height = yuvDataInfo->UsrData.sSystemBuffer.iHeight;
size_t yDataSize = (width * height) + (height * yStride);
size_t uvDataSize = (((width * height) / 4) + (height * uvStride));
size_t requiredSize = yDataSize + (2 * uvDataSize);
uint8_t* yuvBufferedData = (uint8_t*)malloc(requiredSize);
// when i move yuvData[0] to another location i am getting crash.
memcpy(yuvBufferedData, yuvData[0], yDataSize);
memcpy(yuvBufferedData + yDataSize, yuvData[1], uvDataSize);
memcpy(yuvBufferedData + yDataSize + uvDataSize, yuvData[2], uvDataSize);
}
The above code snippet is working on high end android devices. but on few android devices after processing first frame, second frame onwards i am getting crash in first memcpy() statement.
What is wrong in this code? and how to calculate the buffer size from the output of DecodeFrame2().
If i process alternative frames(instead of 30, just 15 frames alternative ones),
it is copying fine.
Please help me to fix this?

yDataSize and uvDataSize is very huge based on the above formula,
This issue has been fixed by modifying the size.

32 bpp monochrome bitmap to 1 bpp TIFF

My android app uses an external lib that makes some image treatments. The final output of the treatment chain is a monochrome bitmap but saved has a color bitmap (32bpp).
The image has to be uploaded to a cloud blob, so for bandwidth concerns, i'd like to convert it to 1bpp G4 compression TIFF. I successfully integrated libTIFF in my app via JNI and now i'm writing the conversion routine in C. I'm a little stuck here.
I managed to produce a 32 BPP TIFF, but impossible to reduce to 1bpp, the output image is always unreadable. Did someone succeded to do similar task ?
More speciffically :
What should be the value of SAMPLE_PER_PIXEL and BITS_PER_SAMPLE
parameters ?
How to determine the strip size ?
How to fill each strip ? (i.e. : How to convert 32bpp pixel lines to 1 bpp pixels strips ?)
Many thanks !
UPDATE : The code produced with the precious help of Mohit Jain
int ConvertMonochrome32BppBitmapTo1BppTiff(char* bitmap, int height, int width, int resx, int resy, char const *tifffilename)
{
TIFF *tiff;
if ((tiff = TIFFOpen(tifffilename, "w")) == NULL)
{
return TC_ERROR_OPEN_FAILED;
}
// TIFF Settings
TIFFSetField(tiff, TIFFTAG_RESOLUTIONUNIT, RESUNIT_INCH);
TIFFSetField(tiff, TIFFTAG_XRESOLUTION, resx);
TIFFSetField(tiff, TIFFTAG_YRESOLUTION, resy);
TIFFSetField(tiff, TIFFTAG_COMPRESSION, COMPRESSION_CCITTFAX4); //Group4 compression
TIFFSetField(tiff, TIFFTAG_IMAGEWIDTH, width);
TIFFSetField(tiff, TIFFTAG_IMAGELENGTH, height);
TIFFSetField(tiff, TIFFTAG_ROWSPERSTRIP, 1);
TIFFSetField(tiff, TIFFTAG_SAMPLESPERPIXEL, 1);
TIFFSetField(tiff, TIFFTAG_BITSPERSAMPLE, 1);
TIFFSetField(tiff, TIFFTAG_ORIENTATION, ORIENTATION_TOPLEFT);
TIFFSetField(tiff, TIFFTAG_PLANARCONFIG, PLANARCONFIG_CONTIG);
TIFFSetField(tiff, TIFFTAG_PHOTOMETRIC, PHOTOMETRIC_MINISWHITE);
tsize_t tbufsize = (width + 7) / 8; //Tiff ScanLine buffer size for 1bpp pixel row
//Now writing image to the file one row by one
int x, y;
for (y = 0; y < height; y++)
{
char *buffer = malloc(tbufsize);
memset(buffer, 0, tbufsize);
for (x = 0; x < width; x++)
{
//offset of the 1st byte of each pixel in the input image (is enough to determine is black or white in 32 bpp monochrome bitmap)
uint32 bmpoffset = ((y * width) + x) * 4;
if (bitmap[bmpoffset] == 0) //Black pixel ?
{
uint32 tiffoffset = x / 8;
*(buffer + tiffoffset) |= (0b10000000 >> (x % 8));
}
}
if (TIFFWriteScanline(tiff, buffer, y, 0) != 1)
{
return TC_ERROR_WRITING_FAILED;
}
if (buffer)
{
free(buffer);
buffer = NULL;
}
}
TIFFClose(tiff);
tiff = NULL;
return TC_SUCCESSFULL;
}

To convert 32 bpp to 1 bpp, extract RGB and convert it into Y (luminance) and use some threshold to convert to 1 bpp.
Number of samples and bits per pixel should be 1.

onPreviewFrame YUV grayscale skewed

I'm trying to get the picture from a surfaceView where I have the camera view running,
I've already implemented onPreviewFrame, and it's called correctly as the debug shows me.
The problem I'm facing now, it's since the byte[] data I receive in the method, it's in YUV space color (NV21), I'm trying to convert it to grayscale to generate a Bitmap and then storing it into a file.
The conversion process that I'm following it's:
public Bitmap convertYuvGrayScaleRGB(byte[] yuv, int width, int height) {
int[] pixels = new int[width * height];
for (int i = 0; i < height*width; i++) {
int grey = yuv[i] & 0xff;
pixels[i] = 0xFF000000 | (grey * 0x00010101);
}
return Bitmap.createBitmap(pixels, width, height, Bitmap.Config.ARGB_8888);
}
The importing procedure for storing it to a file, it's:
Bitmap bitmap = convertYuvGrayScaleRGB(data,widht,heigth);
ByteArrayOutputStream bytes = new ByteArrayOutputStream();
bitmap.compress(Bitmap.CompressFormat.PNG, 50, bytes);
File f = new File(Environment.getExternalStorageDirectory()
+ File.separator + "test.jpg");
Log.d("Camera", "File: " + f.getAbsolutePath());
try {
f.createNewFile();
FileOutputStream fo = new FileOutputStream(f);
fo.write(bytes.toByteArray());
fo.close();
bitmap.recycle();
bitmap = null;
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
Altough, the result I've got it's the following:

I can't find any obvious mistake in your code, but i've already met this kind of skewed images before. When this happened to me, it was due to:
At some point in the code, the image width and height are swapped,
Or the original image you're trying to convert has padding, in which case you will need a stride in addition of the width and height.
Hope this helps!

Probably the Width of the image you are converting is not even. in that case
it is padded in memory.
Let me have a look at the docs...
It seems more complicated than this. if you want your code to work as it is now, you will have to have the width
a multiple of 16.
from the docs:
public static final int YV12
Added in API level 9 Android YUV format.
This format is exposed to software decoders and applications.
YV12 is a 4:2:0 YCrCb planar format comprised of a WxH Y plane
followed by (W/2) x (H/2) Cr and Cb planes.
This format assumes
an even width an even height a horizontal stride multiple of 16 pixels
a vertical stride equal to the height y_size = stride * height
c_stride = ALIGN(stride/2, 16) c_size = c_stride * height/2 size =
y_size + c_size * 2 cr_offset = y_size cb_offset = y_size + c_size

I just had this problem with the S3. My problem was that I used the wrong dimensions for the preview. I assumed the camera was 16:9 when it was actually 4:3.
Use Camera.getParameters().getPreviewSize() to see what the output is in.

I made this:
int frameSize = width * height;
for (int i = 0; i < height; i++) {
for (int j = 0; j < width; j++) {
ret[frameSize + (i >> 1) * width + (j & ~1) + 1] = 127; //U
ret[frameSize + (i >> 1) * width + (j & ~1) + 0] = 127; //V
}
}
So simple but it works really good and fast ;)