Related
Update #6 Discovered I was accessing RGB values improperly. I assumed I was accessing data from an Int[], but was instead accessing byte information from a Byte[]. Changed to accessing from Int[] and get the following image:
Update #5 Adding code used to get RGBA ByteBuffer for reference
private void screenScrape() {
Log.d(TAG, "In screenScrape");
//read pixels from frame buffer into PBO (GL_PIXEL_PACK_BUFFER)
mSurface.queueEvent(new Runnable() {
#Override
public void run() {
Log.d(TAG, "In Screen Scrape 1");
//generate and bind buffer ID
GLES30.glGenBuffers(1, pboIds);
checkGlError("Gen Buffers");
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, pboIds.get(0));
checkGlError("Bind Buffers");
//creates and initializes data store for PBO. Any pre-existing data store is deleted
GLES30.glBufferData(GLES30.GL_PIXEL_PACK_BUFFER, (mWidth * mHeight * 4), null, GLES30.GL_STATIC_READ);
checkGlError("Buffer Data");
//glReadPixelsPBO(0,0,w,h,GLES30.GL_RGB,GLES30.GL_UNSIGNED_SHORT_5_6_5,0);
glReadPixelsPBO(0, 0, mWidth, mHeight, GLES30.GL_RGBA, GLES30.GL_UNSIGNED_BYTE, 0);
checkGlError("Read Pixels");
//GLES30.glReadPixels(0,0,w,h,GLES30.GL_RGBA,GLES30.GL_UNSIGNED_BYTE,intBuffer);
}
});
//map PBO data into client address space
mSurface.queueEvent(new Runnable() {
#Override
public void run() {
Log.d(TAG, "In Screen Scrape 2");
//read pixels from PBO into a byte buffer for processing. Unmap buffer for use in next pass
mapBuffer = ((ByteBuffer) GLES30.glMapBufferRange(GLES30.GL_PIXEL_PACK_BUFFER, 0, 4 * mWidth * mHeight, GLES30.GL_MAP_READ_BIT)).order(ByteOrder.nativeOrder());
checkGlError("Map Buffer");
GLES30.glUnmapBuffer(GLES30.GL_PIXEL_PACK_BUFFER);
checkGlError("Unmap Buffer");
isByteBufferEmpty(mapBuffer, "MAP BUFFER");
convertColorSpaceByteArray(mapBuffer);
mapBuffer.clear();
}
});
}
Update #4 For reference, here is the original image to compare against.
Update #3 This is the output image after interleaving all U/V data into a single array and passing it to the Image object at inputImagePlanes[1]; inputImagePlanes[2]; is unused;
The next image is the same interleaved UV data, but we load this into inputImagePlanes[2]; instead of inputImagePlanes[1];
Update #2 This is the output image after padding the U/V buffers with a zero in between each byte of 'real' data. uArray[uvByteIndex] = (byte) 0;
Update #1 As suggested by a comment, here are the row and pixel strides I get from calling getPixelStride and getRowStride
Y Plane Pixel Stride = 1, Row Stride = 960
U Plane Pixel Stride = 2, Row Stride = 960
V Plane Pixel Stride = 2, Row Stride = 960
The goal of my application is to read pixels out from the screen, compress them, and then send that h264 stream over WiFi to be played be a receiver.
Currently I'm using the MediaMuxer class to convert the raw h264 stream to an MP4, and then save it to file. However the end result video is messed up and I can't figure out why. Lets walk through some of processing and see if we can find anything that jumps out.
Step 1 Set up the encoder. I'm currently taking screen images once every 2 seconds, and using "video/avc" for MIME_TYPE
//create codec for compression
try {
mCodec = MediaCodec.createEncoderByType(MIME_TYPE);
} catch (IOException e) {
Log.d(TAG, "FAILED: Initializing Media Codec");
}
//set up format for codec
MediaFormat mFormat = MediaFormat.createVideoFormat(MIME_TYPE, mWidth, mHeight);
mFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible);
mFormat.setInteger(MediaFormat.KEY_BIT_RATE, 16000000);
mFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 1/2);
mFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
Step 2 Read pixels out from screen. This is done using openGL ES, and the pixels are read out in RGBA format. (I've confirmed this part to be working)
Step 3 Convert the RGBA pixels to YUV420 (IYUV) format. This is done using the following method. Note that I have 2 methods for encoding called at the end of this method.
private void convertColorSpaceByteArray(ByteBuffer rgbBuffer) {
long startTime = System.currentTimeMillis();
Log.d(TAG, "In convertColorspace");
final int frameSize = mWidth * mHeight;
final int chromaSize = frameSize / 4;
byte[] rgbByteArray = new byte[rgbBuffer.remaining()];
rgbBuffer.get(rgbByteArray);
byte[] yuvByteArray = new byte[inputBufferSize];
Log.d(TAG, "Input Buffer size = " + inputBufferSize);
byte[] yArray = new byte[frameSize];
byte[] uArray = new byte[(frameSize / 4)];
byte[] vArray = new byte[(frameSize / 4)];
isByteBufferEmpty(rgbBuffer, "RGB BUFFER");
int yIndex = 0;
int uIndex = frameSize;
int vIndex = frameSize + chromaSize;
int yByteIndex = 0;
int uvByteIndex = 0;
int R, G, B, Y, U, V;
int index = 0;
//this loop controls the rows
for (int i = 0; i < mHeight; i++) {
//this loop controls the columns
for (int j = 0; j < mWidth; j++) {
R = (rgbByteArray[index] & 0xff0000) >> 16;
G = (rgbByteArray[index] & 0xff00) >> 8;
B = (rgbByteArray[index] & 0xff);
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
//clamp and load in the Y data
yuvByteArray[yIndex++] = (byte) ((Y < 16) ? 16 : ((Y > 235) ? 235 : Y));
yArray[yByteIndex] = (byte) ((Y < 16) ? 16 : ((Y > 235) ? 235 : Y));
yByteIndex++;
if (i % 2 == 0 && index % 2 == 0) {
//clamp and load in the U & V data
yuvByteArray[uIndex++] = (byte) ((U < 16) ? 16 : ((U > 239) ? 239 : U));
yuvByteArray[vIndex++] = (byte) ((V < 16) ? 16 : ((V > 239) ? 239 : V));
uArray[uvByteIndex] = (byte) ((U < 16) ? 16 : ((U > 239) ? 239 : U));
vArray[uvByteIndex] = (byte) ((V < 16) ? 16 : ((V > 239) ? 239 : V));
uvByteIndex++;
}
index++;
}
}
encodeVideoFromImage(yArray, uArray, vArray);
encodeVideoFromBuffer(yuvByteArray);
}
Step 4 Encode the data! I currently have two different ways of doing this, and each has a different output. One uses a ByteBuffer returned from MediaCodec.getInputBuffer();, the other uses an Image returned from MediaCodec.getInputImage();
Encoding using ByteBuffer
private void encodeVideoFromBuffer(byte[] yuvData) {
Log.d(TAG, "In encodeVideo");
int inputSize = 0;
//create index for input buffer
inputBufferIndex = mCodec.dequeueInputBuffer(0);
//create the input buffer for submission to encoder
ByteBuffer inputBuffer = mCodec.getInputBuffer(inputBufferIndex);
//clear, then copy yuv buffer into the input buffer
inputBuffer.clear();
inputBuffer.put(yuvData);
//flip buffer before reading data out of it
inputBuffer.flip();
mCodec.queueInputBuffer(inputBufferIndex, 0, inputBuffer.remaining(), presentationTime, 0);
presentationTime += MICROSECONDS_BETWEEN_FRAMES;
sendToWifi();
}
And the associated output image (note: I took a screenshot of the MP4)
Encoding using Image
private void encodeVideoFromImage(byte[] yToEncode, byte[] uToEncode, byte[]vToEncode) {
Log.d(TAG, "In encodeVideo");
int inputSize = 0;
//create index for input buffer
inputBufferIndex = mCodec.dequeueInputBuffer(0);
//create the input buffer for submission to encoder
Image inputImage = mCodec.getInputImage(inputBufferIndex);
Image.Plane[] inputImagePlanes = inputImage.getPlanes();
ByteBuffer yPlaneBuffer = inputImagePlanes[0].getBuffer();
ByteBuffer uPlaneBuffer = inputImagePlanes[1].getBuffer();
ByteBuffer vPlaneBuffer = inputImagePlanes[2].getBuffer();
yPlaneBuffer.put(yToEncode);
uPlaneBuffer.put(uToEncode);
vPlaneBuffer.put(vToEncode);
yPlaneBuffer.flip();
uPlaneBuffer.flip();
vPlaneBuffer.flip();
mCodec.queueInputBuffer(inputBufferIndex, 0, inputBufferSize, presentationTime, 0);
presentationTime += MICROSECONDS_BETWEEN_FRAMES;
sendToWifi();
}
And the associated output image (note: I took a screenshot of the MP4)
Step 5 Convert H264 Stream to MP4. Finally I grab the output buffer from the codec, and use MediaMuxer to convert the raw h264 stream to an MP4 that I can play and test for correctness
private void sendToWifi() {
Log.d(TAG, "In sendToWifi");
MediaCodec.BufferInfo mBufferInfo = new MediaCodec.BufferInfo();
//Check to see if encoder has output before proceeding
boolean waitingForOutput = true;
boolean outputHasChanged = false;
int outputBufferIndex = 0;
while (waitingForOutput) {
//access the output buffer from the codec
outputBufferIndex = mCodec.dequeueOutputBuffer(mBufferInfo, -1);
if (outputBufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
outputFormat = mCodec.getOutputFormat();
outputHasChanged = true;
Log.d(TAG, "OUTPUT FORMAT HAS CHANGED");
}
if (outputBufferIndex >= 0) {
waitingForOutput = false;
}
}
//this buffer now contains the compressed YUV data, ready to be sent over WiFi
ByteBuffer outputBuffer = mCodec.getOutputBuffer(outputBufferIndex);
//adjust output buffer position and limit. As of API 19, this is not automatic
if(mBufferInfo.size != 0) {
outputBuffer.position(mBufferInfo.offset);
outputBuffer.limit(mBufferInfo.offset + mBufferInfo.size);
}
////////////////////////////////FOR DEGBUG/////////////////////////////
if (muxerNotStarted && outputHasChanged) {
//set up track
mTrackIndex = mMuxer.addTrack(outputFormat);
mMuxer.start();
muxerNotStarted = false;
}
if (!muxerNotStarted) {
mMuxer.writeSampleData(mTrackIndex, outputBuffer, mBufferInfo);
}
////////////////////////////END DEBUG//////////////////////////////////
//release the buffer
mCodec.releaseOutputBuffer(outputBufferIndex, false);
muxerPasses++;
}
If you've made it this far you're a gentleman (or lady!) and a scholar! Basically I'm stumped as to why my image is not coming out properly. I'm relatively new to video processing so I'm sure I'm just missing something.
If you're API 19+, might as well stick with encoding method #2, getImage()/encodeVideoFromImage(), since that is more modern.
Focusing on that method: One problem was, you had an unexpected image format. With COLOR_FormatYUV420Flexible, you know you're going to have 8-bit U and V components, but you won't know in advance where they go. That's why you have to query the Image.Plane formats. Could be different on every device.
In this case, the UV format turned out to be interleaved (very common on Android devices). If you're using Java, and you supply each array (U/V) separately, with the "stride" requested ("spacer" byte in-between each sample), I believe one array ends up clobbering the other, because these are actually "direct" ByteBuffers, and they were intended to be used from native code, like in this answer. The solution I explained was to copy an interleaved array into the third (V) plane, and ignore the U plane. On the native side, these two planes actually overlap each other in memory (except for the first and last byte), so filling one causes the implementation to fill both.
If you use the second (U) plane instead, you'll find things work, but the colors look funny. That's also because of the overlapping arrangement of these two planes; what that does, effectively, is shift every array element by one byte (which puts U's where V's should be, and vice versa.)
...In other words, this solution is actually a bit of a hack. Probably the only way to do this correctly, and have it work on all devices, is to use native code (as in the answer I linked above).
Once the color plane problem is fixed, that leaves all the funny overlapping text and vertical striations. These were actually caused by your interpretation of the RGB data, which had the wrong stride.
And, once that is fixed, you have a decent-looking picture. It's been mirrored vertically; I don't know the root cause of that, but I suspect it's an OpenGL issue.
I am making an app where I need to read continues stream of sound which is sent in the form of an byte array. The server side records sound like this (based on an example here on SO):
// Get the minimum buffer size required for the successful creation of an AudioRecord object.
int bufferSizeInBytes = AudioRecord.getMinBufferSize(RECORDER_SAMPLERATE, RECORDER_CHANNELS,
RECORDER_AUDIO_ENCODING);
bufferSizeInBytes = 30000;
// Initialize Audio Recorder.
_audio_recorder = new AudioRecord(MediaRecorder.AudioSource.MIC, RECORDER_SAMPLERATE,
RECORDER_CHANNELS, RECORDER_AUDIO_ENCODING, bufferSizeInBytes);
// Start Recording.
_audio_recorder.startRecording();
int numberOfReadBytes = 0;
byte audioBuffer[] = new byte[bufferSizeInBytes];
boolean recording = false;
float tempFloatBuffer[] = new float[3];
int tempIndex = 0;
byte totalByteBuffer[] = new byte[60 * 44100 * 2];
while (true)
{
float totalAbsValue = 0.0f;
short sample = 0;
numberOfReadBytes = _audio_recorder.read(audioBuffer, 0, bufferSizeInBytes);
for (int i = 0; i < bufferSizeInBytes; i += 2)
{
sample = (short) ((audioBuffer[i]) | audioBuffer[i + 1] << 8);
totalAbsValue += Math.abs(sample) / (numberOfReadBytes / 2);
}
tempFloatBuffer[tempIndex % 3] = totalAbsValue;
float temp = 0.0f;
for (int i = 0; i < 3; ++i)
temp += tempFloatBuffer[i];
if ((temp >= 0 && temp <= _sensitivity) && recording == false)
{
Log.i("TAG", "1");
tempIndex++;
continue;
}
if (temp > _sensitivity && recording == false)
{
Log.i("TAG", "2");
recording = true;
}
if(temp < _sensitivity && recording == true)
{
recording = false;
continue;
}
for (int i = 0; i < numberOfReadBytes; i++)
totalByteBuffer[i] = audioBuffer[i];
if (prepare_sound(totalByteBuffer, numberOfReadBytes))
{
totalByteBuffer = new byte[60 * 44100 * 2];
tempIndex++;
}
}
The example this is taken from is recording sound and saves it to a file when there is no more sound to record. My goal on the other hand is to record sound when there is sound and send this sound on the fly when still recording. Hence, I want to send sounds right a way and not store it to a file when there is no more sound to record. So far I am taking the byte[] with data and stores it in an object an sends it to a client using ObjectOutputStream. The client will then create a temp sound file and play it using MediaPlayer. But I feel that this is not the most effect way to achieve my goal. So, is there any more efficient way to do this with respect to send an continues stream of data as media player does not support playing pure byte[] of data?
Thanks for any help and tips!
Found out that the best solutions for me is to record the sound and when the buffer is full it is sent to the client side. The client then uses an AudioTrack instance to play the byte[] that contains the data like this:
public void onSoundReceived(byte[] sound)
{
_audio_input_stream.write(sound, 0, sound.length);
}
This also makes the sound more "non-lagging" as this is not a MediaPlayer instance which will stop the sound after each time the data was done playing.
I'm editing an MP4 on Android using MediaExtractor to fetch audio and video tracks then creating a new file using MediaMuxer. It works fine. I can play the new MP4 on the phone (and other players) but am unable to stream the file on the web. When I stop the MediaMuxer it generates a log message
"The mp4 file will not be streamable."
I looked at the underlying native code (MPEG4Writer.cpp) and it would appear that the writer is having trouble calculating the needed moov box size. It tries to guess using some heuristic if a bit rate is not supplied as a parameter to the writer. The problem is the MediaMuxer doesn't provider the ability to set MPEG4Writer's parameters. Am I missing something or am I stuck looking a some other means of generating the file (or header)? Thanks.
In MPEG4Writer.cpp:
// The default MIN_MOOV_BOX_SIZE is set to 0.6% x 1MB / 2,
// where 1MB is the common file size limit for MMS application.
// The default MAX _MOOV_BOX_SIZE value is based on about 3
// minute video recording with a bit rate about 3 Mbps, because
// statistics also show that most of the video captured are going
// to be less than 3 minutes.
This is a bad assumption on how MediaMuxer might be used. We are recording a max of 15 seconds of higher res video and MIN_MOOV_BOX_SIZE is way too small. So to make the file streamable I have to rewrite the file to move the moov header before mdat and patch up some offsets. Here is my code. It's not great. Error paths aren't handled correctly and it makes assumptions about the order of the boxes.
public void fastPlay(String srcFile, String dstFile) {
RandomAccessFile inFile = null;
FileOutputStream outFile = null;
try {
inFile = new RandomAccessFile(new File(srcFile), "r");
outFile = new FileOutputStream(new File(dstFile));
int moovPos = 0;
int mdatPos = 0;
int moovSize = 0;
int mdatSize = 0;
byte[] boxSizeBuf = new byte[4];
byte[] pathBuf = new byte[4];
int boxSize;
int dataSize;
int bytesRead;
int totalBytesRead = 0;
int bytesWritten = 0;
// First find the location and size of the moov and mdat boxes
while (true) {
try {
boxSize = inFile.readInt();
bytesRead = inFile.read(pathBuf);
if (bytesRead != 4) {
Log.e(TAG, "Unexpected bytes read (path) " + bytesRead);
break;
}
String pathRead = new String(pathBuf, "UTF-8");
dataSize = boxSize - 8;
totalBytesRead += 8;
if (pathRead.equals("moov")) {
moovPos = totalBytesRead - 8;
moovSize = boxSize;
} else if (pathRead.equals("mdat")) {
mdatPos = totalBytesRead - 8;
mdatSize = boxSize;
}
totalBytesRead += inFile.skipBytes(dataSize);
} catch (IOException e) {
break;
}
}
// Read the moov box into a buffer. This has to be patched up. Ug.
inFile.seek(moovPos);
byte[] moovBoxBuf = new byte[moovSize]; // This shouldn't be too big.
bytesRead = inFile.read(moovBoxBuf);
if (bytesRead != moovSize) {
Log.e(TAG, "Couldn't read full moov box");
}
// Now locate the stco boxes (chunk offset box) inside the moov box and patch
// them up. This ain't purdy.
int pos = 0;
while (pos < moovBoxBuf.length - 4) {
if (moovBoxBuf[pos] == 0x73 && moovBoxBuf[pos + 1] == 0x74 &&
moovBoxBuf[pos + 2] == 0x63 && moovBoxBuf[pos + 3] == 0x6f) {
int stcoPos = pos - 4;
int stcoSize = byteArrayToInt(moovBoxBuf, stcoPos);
patchStco(moovBoxBuf, stcoSize, stcoPos, moovSize);
}
pos++;
}
inFile.seek(0);
byte[] buf = new byte[(int) mdatPos];
// Write out everything before mdat
inFile.read(buf);
outFile.write(buf);
// Write moov
outFile.write(moovBoxBuf, 0, moovSize);
// Write out mdat
inFile.seek(mdatPos);
bytesWritten = 0;
while (bytesWritten < mdatSize) {
int bytesRemaining = (int) mdatSize - bytesWritten;
int bytesToRead = buf.length;
if (bytesRemaining < bytesToRead) bytesToRead = bytesRemaining;
bytesRead = inFile.read(buf, 0, bytesToRead);
if (bytesRead > 0) {
outFile.write(buf, 0, bytesRead);
bytesWritten += bytesRead;
} else {
break;
}
}
} catch (IOException e) {
Log.e(TAG, e.getMessage());
} finally {
try {
if (outFile != null) outFile.close();
if (inFile != null) inFile.close();
} catch (IOException e) {}
}
}
private void patchStco(byte[] buf, int size, int pos, int moovSize) {
Log.e(TAG, "stco " + pos + " size " + size);
// We are inserting the moov box before the mdat box so all of
// offsets in the stco box need to be increased by the size of the moov box. The stco
// box is variable in length. 4 byte size, 4 byte path, 4 byte version, 4 byte flags
// followed by a variable number of chunk offsets. So subtract off 16 from size then
// divide result by 4 to get the number of chunk offsets to patch up.
int chunkOffsetCount = (size - 16) / 4;
int chunkPos = pos + 16;
for (int i = 0; i < chunkOffsetCount; i++) {
int chunkOffset = byteArrayToInt(buf, chunkPos);
int newChunkOffset = chunkOffset + moovSize;
intToByteArray(newChunkOffset, buf, chunkPos);
chunkPos += 4;
}
}
public static int byteArrayToInt(byte[] b, int offset)
{
return b[offset + 3] & 0xFF |
(b[offset + 2] & 0xFF) << 8 |
(b[offset + 1] & 0xFF) << 16 |
(b[offset] & 0xFF) << 24;
}
public void intToByteArray(int a, byte[] buf, int offset)
{
buf[offset] = (byte) ((a >> 24) & 0xFF);
buf[offset + 1] = (byte) ((a >> 16) & 0xFF);
buf[offset + 2] = (byte) ((a >> 8) & 0xFF);
buf[offset + 3] = (byte) (a & 0xFF);
}
Currently MediaMuxer does not create streamable MP4 files
You can try Intel INDE on https://software.intel.com/en-us/intel-inde and Media Pack for Android which is a part of INDE, tutorials on https://software.intel.com/en-us/articles/intel-inde-media-pack-for-android-tutorials. It has a sample that shows how to use media pack to create and stream files over the network
For example for camera streaming it have sample CameraStreamerActivity.java
public void onCreate(Bundle icicle) {
capture = new CameraCapture(new AndroidMediaObjectFactory(getApplicationContext()), progressListener);
parameters = new StreamingParameters();
parameters.Host = getString(R.string.streaming_server_default_ip);
parameters.Port = Integer.parseInt(getString(R.string.streaming_server_default_port));
parameters.ApplicationName = getString(R.string.streaming_server_default_app);
parameters.StreamName = getString(R.string.streaming_server_default_stream);
parameters.isToPublishAudio = false;
parameters.isToPublishVideo = true;
}
public void startStreaming() {
configureMediaStreamFormat();
capture.setTargetVideoFormat(videoFormat);
capture.setTargetAudioFormat(audioFormat);
capture.setTargetConnection(prepareStreamingParams());
capture.start();
}
In addition there are simular samples for files streaming or game process capturing and streaming
I have a bunch of local images saved as jpeg files. My images are captured using CameraPreview and the PreviewFormat is as default: NV21. I want to generate a small video from a fixed number of images.
I am not going to use FFMpeg because it requires NDK and will introduce compatibility issues.
MediaCodec and MediaMuxer seems work but there are not one working solutions on the web.
There are a few references lead to my current solution.
1.EncodeAndMuxTest: http://bigflake.com/mediacodec/EncodeAndMuxTest.java.txt
This one is written by fadden. It quite suits my needs except he is using createInputSurface not queueInputBuffer.
2.Convert bitmap array to YUV (YCbCr NV21)
I do the conversion following this answer. https://stackoverflow.com/a/17116985/3047840
3.Using MediaCodec to save series of images as Video
This question looks much similar as mine but I don't bother using MediaMuxer.
My code is the following:
public class EncodeAndMux extends Activity {
private static final String TAG = "EncodeAndMuxTest";
private static final boolean VERBOSE = false;
private static final File OUTPUT_DIR = Environment
.getExternalStorageDirectory();
private static final String MIME_TYPE = "video/avc";
private static final int FRAME_RATE = 10;
// 10 seconds between I-frames
private static final int IFRAME_INTERVAL = 10;
private static final int NUM_FRAMES = 5;
private static final String DEBUG_FILE_NAME_BASE = "/sdcard/test";
// two seconds of video size of a frame, in pixels
private int mWidth = -1;
private int mHeight = -1;
// bit rate, in bits per second
private int mBitRate = -1;
private byte[] mFrame;
// largest color component delta seen (i.e. actual vs. expected)
private int mLargestColorDelta;
// encoder / muxer state
private MediaCodec mEncoder;
private MediaMuxer mMuxer;
private int mTrackIndex;
private boolean mMuxerStarted;
private Utils mUtils;
private float mPadding;
private int mColumnWidth;
private static final int TEST_Y = 120; // YUV values for colored rect
private static final int TEST_U = 160;
private static final int TEST_V = 200;
private static final int TEST_R0 = 0; // RGB equivalent of {0,0,0}
private static final int TEST_G0 = 136;
private static final int TEST_B0 = 0;
private static final int TEST_R1 = 236; // RGB equivalent of {120,160,200}
private static final int TEST_G1 = 50;
private static final int TEST_B1 = 186;
private static final boolean DEBUG_SAVE_FILE = false; // save copy of
// encoded movie
// allocate one of these up front so we don't need to do it every time
private MediaCodec.BufferInfo mBufferInfo;
private ArrayList<String> mImagePaths = new ArrayList<String>();
byte[] getNV21(int inputWidth, int inputHeight, Bitmap scaled) {
int[] argb = new int[inputWidth * inputHeight];
scaled.getPixels(argb, 0, inputWidth, 0, 0, inputWidth, inputHeight);
byte[] yuv = new byte[inputWidth * inputHeight * 3 / 2];
encodeYUV420SP(yuv, argb, inputWidth, inputHeight);
scaled.recycle();
return yuv;
}
void encodeYUV420SP(byte[] yuv420sp, int[] argb, int width, int height) {
final int frameSize = width * height;
int yIndex = 0;
int uvIndex = frameSize;
int a, R, G, B, Y, U, V;
int index = 0;
for (int j = 0; j < height; j++) {
for (int i = 0; i < width; i++) {
a = (argb[index] & 0xff000000) >> 24; // a is not used obviously
R = (argb[index] & 0xff0000) >> 16;
G = (argb[index] & 0xff00) >> 8;
B = (argb[index] & 0xff) >> 0;
// well known RGB to YUV algorithm
Y = ((66 * R + 129 * G + 25 * B + 128) >> 8) + 16;
U = ((-38 * R - 74 * G + 112 * B + 128) >> 8) + 128;
V = ((112 * R - 94 * G - 18 * B + 128) >> 8) + 128;
// NV21 has a plane of Y and interleaved planes of VU each
// sampled by a factor of 2
// meaning for every 4 Y pixels there are 1 V and 1 U. Note the
// sampling is every other
// pixel AND every other scanline.
yuv420sp[yIndex++] = (byte) ((Y < 0) ? 0
: ((Y > 255) ? 255 : Y));
if (j % 2 == 0 && index % 2 == 0) {
yuv420sp[uvIndex++] = (byte) ((V < 0) ? 0
: ((V > 255) ? 255 : V));
yuv420sp[uvIndex++] = (byte) ((U < 0) ? 0
: ((U > 255) ? 255 : U));
}
index++;
}
}
}
public static Bitmap decodeFile(String filePath, int WIDTH, int HIGHT) {
try {
File f = new File(filePath);
BitmapFactory.Options o = new BitmapFactory.Options();
o.inJustDecodeBounds = true;
o.inPurgeable = true;
o.inInputShareable = true;
BitmapFactory.decodeStream(new FileInputStream(f), null, o);
final int REQUIRED_WIDTH = WIDTH;
final int REQUIRED_HIGHT = HIGHT;
int scale = 1;
while (o.outWidth / scale / 2 >= REQUIRED_WIDTH
&& o.outHeight / scale / 2 >= REQUIRED_HIGHT)
scale *= 2;
BitmapFactory.Options o2 = new BitmapFactory.Options();
o2.inSampleSize = scale;
o2.inPurgeable = true;
o2.inInputShareable = true;
return BitmapFactory.decodeStream(new FileInputStream(f), null, o2);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
return null;
}
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_encode_and_mux);
mUtils = new Utils(this);
mImagePaths = mUtils.getBackFilePaths();
mPadding = TypedValue.applyDimension(TypedValue.COMPLEX_UNIT_DIP,
AppConstant.GRID_PADDING, getResources().getDisplayMetrics());
mColumnWidth = (int) ((mUtils.getScreenWidth() - ((AppConstant.NUM_OF_COLUMNS + 1) * mPadding)) / AppConstant.NUM_OF_COLUMNS);
try {
testEncodeDecodeVideoFromBufferToSurface720p();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (Throwable e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
/**
* Returns the first codec capable of encoding the specified MIME type, or null if no
* match was found.
*/
private static MediaCodecInfo selectCodec(String mimeType) {
int numCodecs = MediaCodecList.getCodecCount();
for (int i = 0; i < numCodecs; i++) {
MediaCodecInfo codecInfo = MediaCodecList.getCodecInfoAt(i);
if (!codecInfo.isEncoder()) {
continue;
}
String[] types = codecInfo.getSupportedTypes();
for (int j = 0; j < types.length; j++) {
if (types[j].equalsIgnoreCase(mimeType)) {
return codecInfo;
}
}
}
return null;
}
/**
* Returns a color format that is supported by the codec and by this test code. If no
* match is found, this throws a test failure -- the set of formats known to the test
* should be expanded for new platforms.
*/
private static int selectColorFormat(MediaCodecInfo codecInfo, String mimeType) {
MediaCodecInfo.CodecCapabilities capabilities = codecInfo.getCapabilitiesForType(mimeType);
for (int i = 0; i < capabilities.colorFormats.length; i++) {
int colorFormat = capabilities.colorFormats[i];
if (isRecognizedFormat(colorFormat)) {
return colorFormat;
}
}
Log.e("","couldn't find a good color format for " + codecInfo.getName() + " / " + mimeType);
return 0; // not reached
}
/**
* Returns true if this is a color format that this test code understands (i.e. we know how
* to read and generate frames in this format).
*/
private static boolean isRecognizedFormat(int colorFormat) {
switch (colorFormat) {
// these are the formats we know how to handle for this test
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar:
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedPlanar:
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar:
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedSemiPlanar:
case MediaCodecInfo.CodecCapabilities.COLOR_TI_FormatYUV420PackedSemiPlanar:
return true;
default:
return false;
}
}
/**
* Returns true if the specified color format is semi-planar YUV. Throws an exception
* if the color format is not recognized (e.g. not YUV).
*/
private static boolean isSemiPlanarYUV(int colorFormat) {
switch (colorFormat) {
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Planar:
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedPlanar:
return false;
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar:
case MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420PackedSemiPlanar:
case MediaCodecInfo.CodecCapabilities.COLOR_TI_FormatYUV420PackedSemiPlanar:
return true;
default:
throw new RuntimeException("unknown format " + colorFormat);
}
}
/**
* Does the actual work for encoding frames from buffers of byte[].
*/
private void doEncodeDecodeVideoFromBuffer(MediaCodec encoder, int encoderColorFormat,
MediaCodec decoder, boolean toSurface) {
final int TIMEOUT_USEC = 10000;
ByteBuffer[] encoderInputBuffers = encoder.getInputBuffers();
ByteBuffer[] encoderOutputBuffers = encoder.getOutputBuffers();
ByteBuffer[] decoderInputBuffers = null;
ByteBuffer[] decoderOutputBuffers = null;
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
MediaFormat decoderOutputFormat = null;
int generateIndex = 0;
int checkIndex = 0;
int badFrames = 0;
boolean decoderConfigured = false;
OutputSurface outputSurface = null;
// The size of a frame of video data, in the formats we handle, is stride*sliceHeight
// for Y, and (stride/2)*(sliceHeight/2) for each of the Cb and Cr channels. Application
// of algebra and assuming that stride==width and sliceHeight==height yields:
// Just out of curiosity.
long rawSize = 0;
long encodedSize = 0;
// Save a copy to disk. Useful for debugging the test. Note this is a raw elementary
// stream, not a .mp4 file, so not all players will know what to do with it.
if (toSurface) {
outputSurface = new OutputSurface(mWidth, mHeight);
}
// Loop until the output side is done.
boolean inputDone = false;
boolean encoderDone = false;
boolean outputDone = false;
while (!outputDone) {
Log.e(TAG, "loop");
// If we're not done submitting frames, generate a new one and submit it. By
// doing this on every loop we're working to ensure that the encoder always has
// work to do.
//
// We don't really want a timeout here, but sometimes there's a delay opening
// the encoder device, so a short timeout can keep us from spinning hard.
if (!inputDone) {
int inputBufIndex = encoder.dequeueInputBuffer(TIMEOUT_USEC);
Log.e(TAG, "inputBufIndex=" + inputBufIndex);
if (inputBufIndex >= 0) {
long ptsUsec = computePresentationTime(generateIndex);
if (generateIndex == NUM_FRAMES) {
// Send an empty frame with the end-of-stream flag set. If we set EOS
// on a frame with data, that frame data will be ignored, and the
// output will be short one frame.
encoder.queueInputBuffer(inputBufIndex, 0, 0, ptsUsec,
MediaCodec.BUFFER_FLAG_END_OF_STREAM);
inputDone = true;
Log.e(TAG, "sent input EOS (with zero-length frame)");
} else {
generateFrame(generateIndex, encoderColorFormat, mFrame);
//generateFrame(generateIndex);
ByteBuffer inputBuf = encoderInputBuffers[inputBufIndex];
// the buffer should be sized to hold one full frame
inputBuf.clear();
inputBuf.put(mFrame);
encoder.queueInputBuffer(inputBufIndex, 0, mFrame.length, ptsUsec, 0);
Log.e(TAG, "submitted frame " + generateIndex + " to enc");
}
generateIndex++;
} else {
// either all in use, or we timed out during initial setup
Log.e(TAG, "input buffer not available");
}
}
// Check for output from the encoder. If there's no output yet, we either need to
// provide more input, or we need to wait for the encoder to work its magic. We
// can't actually tell which is the case, so if we can't get an output buffer right
// away we loop around and see if it wants more input.
//
// Once we get EOS from the encoder, we don't need to do this anymore.
if (!encoderDone) {
int encoderStatus = encoder.dequeueOutputBuffer(info, TIMEOUT_USEC);
if (encoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
// no output available yet
Log.e(TAG, "no output from encoder available");
} else if (encoderStatus == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
// not expected for an encoder
encoderOutputBuffers = encoder.getOutputBuffers();
Log.e(TAG, "encoder output buffers changed");
} else if (encoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// not expected for an encoder
if (mMuxerStarted) {
throw new RuntimeException("format changed twice");
}
MediaFormat newFormat = encoder.getOutputFormat();
Log.e(TAG, "encoder output format changed: " + newFormat);
// now that we have the Magic Goodies, start the muxer
mTrackIndex = mMuxer.addTrack(newFormat);
Log.e(TAG, "muxer defined muxer format: " + newFormat);
mMuxer.start();
mMuxerStarted = true;
} else if (encoderStatus < 0) {
Log.e("","unexpected result from encoder.dequeueOutputBuffer: " + encoderStatus);
} else { // encoderStatus >= 0
ByteBuffer encodedData = encoderOutputBuffers[encoderStatus];
if (encodedData == null) {
Log.e("","encoderOutputBuffer " + encoderStatus + " was null");
}
// It's usually necessary to adjust the ByteBuffer values to match BufferInfo.
encodedData.position(info.offset);
encodedData.limit(info.offset + info.size);
encodedSize += info.size;
if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
// Codec config info. Only expected on first packet. One way to
// handle this is to manually stuff the data into the MediaFormat
// and pass that to configure(). We do that here to exercise the API.
MediaFormat format =
MediaFormat.createVideoFormat(MIME_TYPE, mWidth, mHeight);
format.setByteBuffer("csd-0", encodedData);
decoder.configure(format, toSurface ? outputSurface.getSurface() : null,
null, 0);
decoder.start();
decoderInputBuffers = decoder.getInputBuffers();
decoderOutputBuffers = decoder.getOutputBuffers();
decoderConfigured = true;
Log.e(TAG, "decoder configured (" + info.size + " bytes)"+format);
} else {
// Get a decoder input buffer, blocking until it's available.
int inputBufIndex = decoder.dequeueInputBuffer(-1);
ByteBuffer inputBuf = decoderInputBuffers[inputBufIndex];
inputBuf.clear();
inputBuf.put(encodedData);
decoder.queueInputBuffer(inputBufIndex, 0, info.size,
info.presentationTimeUs, info.flags);
encoderDone = (info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0;
Log.e(TAG, "passed " + info.size + " bytes to decoder"
+ (encoderDone ? " (EOS)" : ""));
Log.e("encoderDone",encoderDone+"");
}
encoder.releaseOutputBuffer(encoderStatus, false);
}
}
// Check for output from the decoder. We want to do this on every loop to avoid
// the possibility of stalling the pipeline. We use a short timeout to avoid
// burning CPU if the decoder is hard at work but the next frame isn't quite ready.
//
// If we're decoding to a Surface, we'll get notified here as usual but the
// ByteBuffer references will be null. The data is sent to Surface instead.
if (decoderConfigured) {
int decoderStatus = decoder.dequeueOutputBuffer(info, 3*TIMEOUT_USEC);
if (decoderStatus == MediaCodec.INFO_TRY_AGAIN_LATER) {
// no output available yet
Log.e(TAG, "no output from decoder available");
} else if (decoderStatus == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
// The storage associated with the direct ByteBuffer may already be unmapped,
// so attempting to access data through the old output buffer array could
// lead to a native crash.
Log.e(TAG, "decoder output buffers changed");
decoderOutputBuffers = decoder.getOutputBuffers();
} else if (decoderStatus == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// this happens before the first frame is returned
decoderOutputFormat = decoder.getOutputFormat();
Log.e(TAG, "decoder output format changed: " +
decoderOutputFormat);
} else if (decoderStatus < 0) {
Log.e(TAG, "unexpected result from deocder.dequeueOutputBuffer: " + decoderStatus);
} else { // decoderStatus >= 0
if (!toSurface) {
ByteBuffer outputFrame = decoderOutputBuffers[decoderStatus];
outputFrame.position(info.offset);
outputFrame.limit(info.offset + info.size);
mMuxer.writeSampleData(mTrackIndex, outputFrame,
info);
rawSize += info.size;
if (info.size == 0) {
Log.e(TAG, "got empty frame");
} else {
Log.e(TAG, "decoded, checking frame " + checkIndex);
if (!checkFrame(checkIndex++, decoderOutputFormat, outputFrame)) {
badFrames++;
}
}
if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
Log.e(TAG, "output EOS");
outputDone = true;
}
decoder.releaseOutputBuffer(decoderStatus, false /*render*/);
} else {
Log.e(TAG, "surface decoder given buffer " + decoderStatus +
" (size=" + info.size + ")");
rawSize += info.size;
if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
Log.e(TAG, "output EOS");
outputDone = true;
}
boolean doRender = (info.size != 0);
// As soon as we call releaseOutputBuffer, the buffer will be forwarded
// to SurfaceTexture to convert to a texture. The API doesn't guarantee
// that the texture will be available before the call returns, so we
// need to wait for the onFrameAvailable callback to fire.
decoder.releaseOutputBuffer(decoderStatus, doRender);
if (doRender) {
Log.e(TAG, "awaiting frame " + checkIndex);
outputSurface.awaitNewImage();
outputSurface.drawImage();
if (!checkSurfaceFrame(checkIndex++)) {
badFrames++;
}
}
}
}
}
}
Log.e(TAG, "decoded " + checkIndex + " frames at "
+ mWidth + "x" + mHeight + ": raw=" + rawSize + ", enc=" + encodedSize);
if (outputSurface != null) {
outputSurface.release();
}
if (checkIndex != NUM_FRAMES) {
Log.e(TAG, "awaiting frame " + checkIndex);
}
if (badFrames != 0) {
Log.e(TAG, "Found " + badFrames + " bad frames");
}
}
private void generateFrame(int frameIndex) {
Bitmap bitmap = decodeFile(mImagePaths.get(frameIndex), mColumnWidth,
mColumnWidth);
mFrame = getNV21(bitmap.getWidth(), bitmap.getHeight(), bitmap);
}
/**
* Generates data for frame N into the supplied buffer. We have an 8-frame animation
* sequence that wraps around. It looks like this:
* <pre>
* 0 1 2 3
* 7 6 5 4
* </pre>
* We draw one of the eight rectangles and leave the rest set to the zero-fill color.
*/
private void generateFrame(int frameIndex, int colorFormat, byte[] mFrame) {
final int HALF_WIDTH = mWidth / 2;
boolean semiPlanar = isSemiPlanarYUV(colorFormat);
// Set to zero. In YUV this is a dull green.
Arrays.fill(mFrame, (byte) 0);
int startX, startY, countX, countY;
frameIndex %= 8;
//frameIndex = (frameIndex / 8) % 8; // use this instead for debug -- easier to see
if (frameIndex < 4) {
startX = frameIndex * (mWidth / 4);
startY = 0;
} else {
startX = (7 - frameIndex) * (mWidth / 4);
startY = mHeight / 2;
}
for (int y = startY + (mHeight/2) - 1; y >= startY; --y) {
for (int x = startX + (mWidth/4) - 1; x >= startX; --x) {
if (semiPlanar) {
// full-size Y, followed by UV pairs at half resolution
// e.g. Nexus 4 OMX.qcom.video.encoder.avc COLOR_FormatYUV420SemiPlanar
// e.g. Galaxy Nexus OMX.TI.DUCATI1.VIDEO.H264E
// OMX_TI_COLOR_FormatYUV420PackedSemiPlanar
mFrame[y * mWidth + x] = (byte) TEST_Y;
if ((x & 0x01) == 0 && (y & 0x01) == 0) {
mFrame[mWidth*mHeight + y * HALF_WIDTH + x] = (byte) TEST_U;
mFrame[mWidth*mHeight + y * HALF_WIDTH + x + 1] = (byte) TEST_V;
}
} else {
// full-size Y, followed by quarter-size U and quarter-size V
// e.g. Nexus 10 OMX.Exynos.AVC.Encoder COLOR_FormatYUV420Planar
// e.g. Nexus 7 OMX.Nvidia.h264.encoder COLOR_FormatYUV420Planar
mFrame[y * mWidth + x] = (byte) TEST_Y;
if ((x & 0x01) == 0 && (y & 0x01) == 0) {
mFrame[mWidth*mHeight + (y/2) * HALF_WIDTH + (x/2)] = (byte) TEST_U;
mFrame[mWidth*mHeight + HALF_WIDTH * (mHeight / 2) +
(y/2) * HALF_WIDTH + (x/2)] = (byte) TEST_V;
}
}
}
}
}
/**
* Sets the desired frame size and bit rate.
*/
private void setParameters(int width, int height, int bitRate) {
if ((width % 16) != 0 || (height % 16) != 0) {
Log.w(TAG, "WARNING: width or height not multiple of 16");
}
mWidth = width;
mHeight = height;
mBitRate = bitRate;
mFrame = new byte[mWidth * mHeight * 3 / 2];
}
public void testEncodeDecodeVideoFromBufferToSurface720p() throws Throwable {
setParameters(1280, 720, 6000000);
encodeDecodeVideoFromBuffer(false);
}
}
Logcat:
12-17 18:25:47.405: E/EncodeAndMuxTest(16415): found codec: OMX.qcom.video.encoder.avc
12-17 18:25:47.405: I/OMXClient(16415): Using client-side OMX mux.
12-17 18:25:47.455: E/EncodeAndMuxTest(16415): found colorFormat: 21
12-17 18:25:47.455: E/EncodeAndMuxTest(16415): format: {frame-rate=10, bitrate=6000000, height=720, mime=video/avc, color-format=21, i-frame-interval=10, width=1280}
12-17 18:25:47.465: I/OMXClient(16415): Using client-side OMX mux.
12-17 18:25:47.495: E/ACodec(16415): [OMX.qcom.video.encoder.avc] storeMetaDataInBuffers (output) failed w/ err -2147483648
12-17 18:25:47.495: I/ACodec(16415): setupVideoEncoder succeeded
12-17 18:25:47.535: I/OMXClient(16415): Using client-side OMX mux.
12-17 18:25:47.545: E/EncodeAndMuxTest(16415): loop
12-17 18:25:47.545: E/EncodeAndMuxTest(16415): inputBufIndex=0
12-17 18:25:47.655: E/EncodeAndMuxTest(16415): submitted frame 0 to enc
12-17 18:25:47.655: E/EncodeAndMuxTest(16415): encoder output format changed: {csd-1=java.nio.ByteArrayBuffer[position=0,limit=8,capacity=8], height=720, mime=video/avc, csd-0=java.nio.ByteArrayBuffer[position=0,limit=18,capacity=18], what=1869968451, width=1280}
12-17 18:25:47.655: E/EncodeAndMuxTest(16415): muxer defined muxer format: {csd-1=java.nio.ByteArrayBuffer[position=0,limit=8,capacity=8], height=720, mime=video/avc, csd-0=java.nio.ByteArrayBuffer[position=0,limit=18,capacity=18], what=1869968451, width=1280}
12-17 18:25:47.655: I/MPEG4Writer(16415): limits: 2147483647/0 bytes/us, bit rate: -1 bps and the estimated moov size 3072 bytes
12-17 18:25:47.655: E/EncodeAndMuxTest(16415): inputBufIndex=2
12-17 18:25:47.795: E/EncodeAndMuxTest(16415): submitted frame 1 to enc
12-17 18:25:47.825: E/EncodeAndMuxTest(16415): decoder configured (26 bytes){csd-0=java.nio.DirectByteBuffer[position=0,limit=26,capacity=692224], height=720, width=1280, mime=video/avc}
12-17 18:25:47.855: E/EncodeAndMuxTest(16415): no output from decoder available
12-17 18:25:47.855: E/EncodeAndMuxTest(16415): inputBufIndex=0
12-17 18:25:47.976: E/EncodeAndMuxTest(16415): submitted frame 2 to enc
12-17 18:25:48.136: E/EncodeAndMuxTest(16415): passed 3188 bytes to decoder
12-17 18:25:48.176: E/EncodeAndMuxTest(16415): no output from decoder available
12-17 18:25:48.176: E/EncodeAndMuxTest(16415): inputBufIndex=1
12-17 18:25:48.296: E/EncodeAndMuxTest(16415): submitted frame 3 to enc
12-17 18:25:48.296: E/EncodeAndMuxTest(16415): passed 1249 bytes to decoder
12-17 18:25:48.326: E/EncodeAndMuxTest(16415): no output from decoder available
12-17 18:25:48.326: E/EncodeAndMuxTest(16415): loop
12-17 18:25:48.326: E/EncodeAndMuxTest(16415): inputBufIndex=2
12-17 18:25:48.396: E/EncodeAndMuxTest(16415): submitted frame 4 to enc
12-17 18:25:48.396: E/EncodeAndMuxTest(16415): passed 3085 bytes to decoder
12-17 18:25:48.436: E/EncodeAndMuxTest(16415): no output from decoder available
12-17 18:25:48.436: E/EncodeAndMuxTest(16415): inputBufIndex=0
12-17 18:25:48.436: E/EncodeAndMuxTest(16415): sent input EOS (with zero-length frame)
12-17 18:25:48.436: E/EncodeAndMuxTest(16415): passed 3056 bytes to decoder
12-17 18:25:48.466: E/EncodeAndMuxTest(16415): no output from decoder available
12-17 18:25:48.466: E/EncodeAndMuxTest(16415): passed 1085 bytes to decoder (EOS)
12-17 18:25:48.476: E/EncodeAndMuxTest(16415): decoder output buffers changed
12-17 18:25:48.496: E/EncodeAndMuxTest(16415): decoder output format changed:
Reading the Jpegs, decompressing them, and then recompressing them is going to cause loss of Image quality (and take CPU effort / time) simply appending them altogether and tossing them in a Video Container will be faster and produce a better Video.
The MJpeg Video Format is quite old so (almost) any Program can play Mjpeg Videos.
I suggest a Solution similar to this http://sourceforge.net/projects/jpegtoavi/ IE: make an Mjpeg Movie from your Jpegs. There is more than that one Program to choose from, use a Search Engine (or our Search Bar) to find more Source Code.
.
I tested my Phone to see if it can understand Mjpegs by creating a File using this command:
ffmpeg.exe -i test_in.mp4 -vcodec mjpeg -acodec copy test_out.mp4
In: Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1568 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)
Out: Stream #0:0(und): Video: mjpeg (l[0][0][0] / 0x006C), yuvj420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30k tbn, 29.97 tbc (default)
Unfortunatley the Android "Gallery Player" is one of the Programs that does not understand that format, but BSPlayer, VLC, and MPlayer for Android can play that Format if you want the resulting Video to play on your Phone (without writing more Code).
I have made an application that records from the phones microphone using the AudioRecord and 16-bit encoding, and I am able to playback the recording. For some compatibility reason I need to use 8-bit encoding, but when I try to run the same program using that encoding I keep getting an Invalid Audio Format. my code is :
int bufferSize = AudioRecord.getMinBufferSize(11025,
AudioFormat.CHANNEL_CONFIGURATION_MONO,
AudioFormat.ENCODING_PCM_8BIT);
AudioRecord recordInstance = new AudioRecord(
MediaRecorder.AudioSource.MIC, 11025,
AudioFormat.CHANNEL_CONFIGURATION_MONO, AudioFormat.ENCODING_PCM_8BIT,
bufferSize);
Any one knows what is the problem? According to the documentation AudioRecord is capable of 8-bit encoding.
If you look at the source, it only supports little endian, but Android is writing out big endian. So you have to convert to little endian and then 8-bit. This worked for me and you can probably combine the two:
for (int i = 0; (offset + i + 1) < bytes.length; i += 2) {
lens[i] = bytes[offset + i + 1];
lens[i + 1] = bytes[offset + i];
}
for (int i = 1, j = 0; i < length; i += 2, j++) {
lens[j] = lens[i];
}
Here is a simpler version without endian
for (int i = 0, j = 0; (offset + i) < bytes.length; i += 2, j++) {
lens[j] = bytes[offset + i];
}