Handle MediaCodec video with dropped frames - android

I'm currently doing fast precise seeking using MediaCodec. What I currently do to skip frame by frame is, I first get the total frames:
mediaInfo.totalFrames = videoTrack.getSamples().size();
Then I get the length of the video file:
mediaInfo.durationUs = videoTrack.getDuration() * 1000 *1000 / timeScale;
//then calling:
public long getDuration() {
if (mMediaInfo != null) {
return (int) mMediaInfo.durationUs / 1000; // to millisecond
}
return -1;
}
Now, when I want to get the next frame I call the following:
mNextFrame.setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View view) {
int frames = Integer.parseInt(String.valueOf(getTotalFrames));
int intervals = Integer.parseInt(String.valueOf(mPlayer.getDuration() / frames));
if (mPlayer.isPlaying()) {
mPlayer.pause();
mPlayer.seekTo(mPlayer.getCurrentPosition() + intervals);
} else {
mPlayer.seekTo(mPlayer.getCurrentPosition() + intervals);
}
}
});
Here is the info about the file I'm testing with:
Frames = 466
Duration = 15523
So the interval between frames are
33,311158798283262
In other words, each time I press the next button the intervals will be rounded to 33, when I press the next button it will call mPlayer.seekTo(mPlayer.getCurrentPosition() + 33 meaning that some frames will be lost, or that is what I thought. I tested and got the following back when logging getCurrentPosition after each time the button is pressed and here is the result:
33 -> 66 -> 99 -> 132 -> 166
Going from 132 to 166 is 34ms instead of 33, so there was a compensation to make up with the frames that would have be lost.
The above works perfectly fine, I can skip through frames without any problem, here is the issue I facing.
Taking the same logic I used above I created a custom RangeBar. I created a method setTickCount (it's basically the same as seekbar.setMax) and I set the "TickCount" like this:
int frames = Integer.parseInt(String.valueOf(getTotalFrames));
mrange_bar.setTickCount(frames);
So the max value of my RangeBar is the amout of frames in the video.
When the "Tick" value changes I call the following:
int frames = Integer.parseInt(String.valueOf(getTotalFrames));
int intervals = Integer.parseInt(String.valueOf(mPlayer.getDuration() / frames));
mPlayer.seekTo(intervals * TickPosition);
So the above will work like this, if my tickCount position is, let's say 40:
mPlayer.seekTo(33 * 40); //1320ms
I would think that the above would work fine because I used the exact same logic, but instead the video "jump/skip" back to (what I assume is the key frame) and the continues the seeking.
Why is happening and how I can resolve this issue?
EDIT 1:
I mentioned above that it is jumping to the previous key frame, but I had a look again and it is calling end of stream while seeking (at spesific points during the video). When I reach end of stream I release my previous buffer so that one frame can still be displayed to avoid a black screen, by calling:
mDecoder.releaseOutputBuffer(prevBufferIndex, true);
So, for some reason, end of stream is called, where I then restart mediacodec causing a "lag/jump" effect.
If I remove the above, I don't get the frame "jump", but there is still a lag while mediacodec is being initialized.
EDIT 2:
After digging deeper I found that readSampleData is -1:
ByteBuffer[] inputBuffers = mDecoder.getInputBuffers();
int inIndex = mDecoder.dequeueInputBuffer(TIMEOUT_USEC);
if (inIndex >= 0) {
ByteBuffer buffer = inputBuffers[inIndex];
int sampleSize = mExtractor.readSampleData(buffer, 0);
if (sampleSize < 0) {
mDecoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);
mIsExtractorReachedEOS = true;
} else {
mDecoder.queueInputBuffer(inIndex, 0, sampleSize, mExtractor.getSampleTime(), 0);
mExtractor.advance();
}
}
For some reason my sampleSize is -1 at a specific point during seeking.
EDIT 3
This issue is definitely regarding the time that I pass, I tried 2 different approaches, the first:
mPlayer.seekTo(progress);
//position is retrieved by setting mSeekBar.setMax(mPlayer.getDuration); ....
and the second approach, I determine the frame intervals:
//Total amount of frames in video
long TotalFramesInVideo = videoTrack.getSamples().size();
//Duration of file in milliseconds
int DurationOfVideoInMs = mPlayer.getDuration();
//Determine interval between frames
int frameIntervals = DurationOfVideoInMs / Integer.parseInt(String.valueOf(TotalFramesInVideo));
//Then I seek to the frames like this:
mPlayer.seekTo(position * frameIntervals);
After trying both the above methods, I realised that the issue is related to the time being passed to mediaCodec because the "lag/jump" happens at different places.
I'm not sure why this doesn't happen when I call:
mPlayer.seekTo(mPlayer.getCurrentPosition() + intervals);

Related

PixelBuffer Object and glReadPixel on Android(ARCore) blocking

I know that the default glReadPixels() waits until all the drawing commands are executed on the GL thread. But when you bind a PixelBuffer Object and then call the glReadPixels() it should be asynchronous and will not wait for anything.
But when I bind PBO and do the glReadPixels() it is blocking for some time.
Here's how I initialize the PBO:
mPboIds = IntBuffer.allocate(2);
GLES30.glGenBuffers(2, mPboIds);
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, mPboIds.get(0));
GLES30.glBufferData(GLES30.GL_PIXEL_PACK_BUFFER, mPboSize, null, GLES30.GL_STATIC_READ); //allocates only memory space given data size
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, mPboIds.get(1));
GLES30.glBufferData(GLES30.GL_PIXEL_PACK_BUFFER, mPboSize, null, GLES30.GL_STATIC_READ);
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, 0);
and then I use the two buffers to ping-pong around:
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, mPboIds.get(mPboIndex)); //1st PBO
JNIWrapper.glReadPixels(0, 0, mRowStride / mPixelStride, (int)height, GLES30.GL_RGBA, GLES30.GL_UNSIGNED_BYTE); //read pixel from the screen and write to 1st buffer(native C++ code)
//don't load anything in the first frame
if (mInitRecord) {
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, 0);
//reverse the index
mPboIndex = (mPboIndex + 1) % 2;
mPboNewIndex = (mPboNewIndex + 1) % 2;
mInitRecord = false;
return;
}
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, mPboIds.get(mPboNewIndex)); //2nd PBO
//glMapBufferRange returns pointer to the buffer object
//this is the same thing as calling glReadPixel() without a bound PBO
//The key point is that we can pipeline this call
ByteBuffer byteBuffer = (ByteBuffer) GLES30.glMapBufferRange(GLES30.GL_PIXEL_PACK_BUFFER, 0, mPboSize, GLES30.GL_MAP_READ_BIT); //downdload from the GPU to CPU
Bitmap bitmap = Bitmap.createBitmap((int)mScreenWidth,(int)mScreenHeight, Bitmap.Config.ARGB_8888);
bitmap.copyPixelsFromBuffer(byteBuffer);
GLES30.glUnmapBuffer(GLES30.GL_PIXEL_PACK_BUFFER);
GLES30.glBindBuffer(GLES30.GL_PIXEL_PACK_BUFFER, 0);
//reverse the index
mPboIndex = (mPboIndex + 1) % 2;
mPboNewIndex = (mPboNewIndex + 1) % 2;
This is called in my draw method every frame.
From my understanding the glReadPixels should not take any time at all, but it's taking around 25ms (on Google Pixel 2) and creating the bitmap takes another 40ms. This only achieve like 13 FPS which is worse than glReadPixels without PBO.
Is there anything that I'm missing or wrong in my code?
EDITED since you pointed out that my original hypothesis was incorrect (initial PboIndex == PboNextIndex). Hoping to be helpful, here is C++ code that I just wrote on the native side called through JNI from Android using GLES 3. It seems to work and not block on glReadPixels(...). Note there is only a single glPboIndex variable:
glBindBuffer(GL_PIXEL_PACK_BUFFER, glPboIds[glPboIndex]);
glReadPixels(0, 0, frameWidth_, frameHeight_, GL_RGBA, GL_UNSIGNED_BYTE, 0);
glPboReady[glPboIndex] = true;
glPboIndex = (glPboIndex + 1) % 2;
if (glPboReady[glPboIndex]) {
glBindBuffer(GL_PIXEL_PACK_BUFFER, glPboIds[glPboIndex]);
GLubyte* rgbaBytes = (GLubyte*)glMapBufferRange(
GL_PIXEL_PACK_BUFFER, 0, frameByteCount_, GL_MAP_READ_BIT);
if (rgbaBytes) {
size_t minYuvByteCount = frameWidth_ * frameHeight_ * 3 / 2; // 12 bits/pixel
if (videoFrameBufferSize_ < minYuvByteCount) {
return; // !!! not logging error inside render loop
}
convertToVideoYuv420NV21FromRgbaInverted(
videoFrameBufferAddress_, rgbaBytes,
frameWidth_, frameHeight_);
}
glUnmapBuffer(GL_PIXEL_PACK_BUFFER);
glPboReady[glPboIndex] = false;
}
glBindBuffer(GL_PIXEL_PACK_BUFFER, 0);
...
previous unfounded hypothesis:
Your question doesn't show the code that sets the initial values of mPboIndex and mPboNewIndex, but if they are set to identical initial values, such as 0, then they will have matching values within each loop which will result in mapping the same PBO that has just been read. In that hypothetical/real scenario, even if 2 PBOs are being used, they are not alternated between glReadPixels and glMapBufferRange which will then block until the GPU completes data transfer. I suggest this change to ensure that the PBOs alternate:
mPboNewIndex = mPboIndex;
mPboIndex = (mPboNewIndex + 1) % 2;

Smoother Android SeekBar

I'm not strictly looking for an implementation of this idea but if someone has already made it, that would be awesome and I'd like to see it. Otherwise:
I'm trying to implement a couple SeekBars in my Android's waveform generator app. I have a couple controls such as: volume, frequency, low pass filter cutoff frequency, resonance, pitch bend, etc.
The problem with my SeekBars are that they sound too step-y and I want it to sound more analog-ish (smoother if you will). In my iOS implementation of the app, the native UISliders did a good job and I didn't hear any step-like movements. However, the SeekBars aren't very smooth and tend to jump value to value (lets say like from 10 to 100 with a max value of 1000).
I was wondering if it might be best if I just design my own custom UI for a smoother slider or if there is one already. Also, is it possible that my audio thread is interrupting the SeekBar's functionality and causing these jumps/step-like behavior?
Things I've tried already:
Lowpass the seekbar progress in the listener's onProgressChanged. This doesn't really work (if it jumped from 5 to 100 for example, this would give me a value in between but that still doesn't give a full smooth-like behavior).
// b = 0.99, a = 0.01
// Follows simple lowpass: yn = (xn * b) + (yn1 * a)
public double lowpass(int xn) {
double yn = (xn * b) + (lastProgress * a);
lastProgress = yn;
return yn;
}
If there is a huge jump (like 5 to 100), I would call a while loop to increment the audio context's variables by 1. The problem with this is if I was trying to do a pitch bend (a 14bit number so that's 16384 values total), it would take too long to get to the target value (the pitch bend does sound cool though). for example (obviously this only accounts for progress going up):
public void onProgressChanged(SeekBar seekBar, int progress, boolean fromUser) {
int myProgress = seekBar.getProgress();
while (myProgress < progress) {
// This will increase the audio context's frequency variable by one every loop until we've reached our target progress
audioContext.setFrequency(myProgress++);
}
}
Thanks!
First, figure out what is the fastest you want the volume to increase. For this example, I'll use 1 second (1000ms) to change from 0 to 1.0. (0% to 100%)
When you enter the loop, record the current time. Then, for each iteration of your loop, check the time passed and increment the necessary amount.
// example, myLoop(0.25, 0.75, startTime);
double myLoop(double start, double end, long startTime) {
long now = System.currentTimeMillis(); // (100ms since startTime)
double span = end - start; // the span in value, 0 to 1.0 (=0.5)
double timeSpan = (now - startTime) / (span * 1000); // time since started / total time of change (100/500 = 0.2)
return timeSpan * span + start; // (0.2 * 0.5 = 0.1) + 0.25 = 0.35, the value for this loop
}
(untested code)

Is it possible to set dynamic frame-rate with cocos2d-x?

I want to write a text reader which has special effects with cocos2d-x, so the most time the graph will be static. If I use cocos2d-x, it's just heavily consuming battery power.
So is it possible to adjust cocos2d-x's frame rate by coding? And how? I want to reduce frame rate when text's static, and increase frame rate when paging up or down.
Or any good idea for this goal on Android? (Page turning animations and more efficient text rendering.)
You can change frame rate using cocos2d::Director::setAnimationInterval method.
https://github.com/cocos2d/cocos2d-x/blob/1361f2c6195ce338a70b65c17e3d46f38e6bcce2/cocos/base/CCDirector.h#L140-L141
/** Set the FPS value. */
virtual void setAnimationInterval(double interval) = 0;
However, I wonder, if you set framerate low, your C++ code wasn't called immediately when paging up or down because the framerate is low. So you might need to modify onDrawFrame to call Cocos2dxRenderer.nativeRender immediately when user touched the screen.
https://github.com/cocos2d/cocos2d-x/blob/1361f2c6195ce338a70b65c17e3d46f38e6bcce2/cocos/platform/android/java/src/org/cocos2dx/lib/Cocos2dxRenderer.java#L84-L106
#Override
public void onDrawFrame(final GL10 gl) {
/*
* No need to use algorithm in default(60 FPS) situation,
* since onDrawFrame() was called by system 60 times per second by default.
*/
if (sAnimationInterval <= 1.0 / 60 * Cocos2dxRenderer.NANOSECONDSPERSECOND) {
Cocos2dxRenderer.nativeRender();
} else {
final long now = System.nanoTime();
final long interval = now - this.mLastTickInNanoSeconds;
if (interval < Cocos2dxRenderer.sAnimationInterval) {
try {
Thread.sleep((Cocos2dxRenderer.sAnimationInterval - interval) / Cocos2dxRenderer.NANOSECONDSPERMICROSECOND);
} catch (final Exception e) {
}
}
/*
* Render time MUST be counted in, or the FPS will slower than appointed.
*/
this.mLastTickInNanoSeconds = System.nanoTime();
Cocos2dxRenderer.nativeRender();
}
}

Android MediaMetadataRetriever.getFrameAtTime skips frames

I wrote a simple Android application that is using MediaMetadataRetriver class to get frames. It works fine, except that I realized that it skips frames.
The video clip I am trying to decode is one shot with the phone camera. Follow relevant code snippets:
MediaMetadataRetriever mediaDataRet = new MediaMetadataRetriever();
mediaDataRet.setDataSource(path);
String lengthMsStr = mediaDataRet
.extractMetadata(mediaDataRet.METADATA_KEY_DURATION);
final long lenMs = Long.parseLong(lengthMsStr);
String widthStr = mediaDataRet
.extractMetadata(mediaDataRet.METADATA_KEY_VIDEO_WIDTH);
int width = Integer.parseInt(widthStr);
String heightStr = mediaDataRet
.extractMetadata(mediaDataRet.METADATA_KEY_VIDEO_HEIGHT);
int height = Integer.parseInt(heightStr);
note the variable lenMs, it holds the clid duration in milliseconds. Then for every frame I do:
int pace = 30; // 30 fps ms spacing
for (long i = 0; i < lenMs; i += pace) {
if (is_abort())
return;
Bitmap bitmap = mediaDataRet.getFrameAtTime(i * 1000); // I tried the other version of this method with OPTION_CLOSEST, with no luck.
if (bc == null)
bc = bitmap.getConfig();
bitmap.getPixels(pixBuffer, 0, width, 0, 0, width, height);
[...]
}
After checking visually I noticed that some frames are skipped (like short sequences). Why? And ho do I avoid this?
Use:
mediaDataRet.getFrameAtTime(i * 1000, MediaMetadataRetriever.OPTION_CLOSEST);
The getFrameAtTime(n) uses OPTION_CLOSEST_SYNC which would give you key frames only.

AudioRecord read returns strange values

I am trying to build some kind of sound-meter to the Android platform. (i am using HTC wildfire) I use the AudioRecord class for that goal, however it seems that the
values that are being returned from its "read" are not reasonable.
This is how i created the AudioRecord object:
int minBufferSize =
AudioRecord.getMinBufferSize(sampleRateInHz,
android.media.AudioFormat.CHANNEL_IN_MONO,
android.media.AudioFormat.ENCODING_PCM_16BIT);
audioRecored = new AudioRecord( MediaRecorder.AudioSource.MIC,
44100,
android.media.AudioFormat.CHANNEL_IN_MONO,
android.media.AudioFormat.ENCODING_PCM_16BIT,
minBufferSize );
This is how i try to read data from it:
short[] audioData = new short[bufferSize];
int offset =0;
int shortRead = 0;
int sampleToReadPerGet = 1000;//some value in order to avoid momentaraly nosies.
//start tapping into the microphone
audioRecored.startRecording();
//start reading from the microphone to an internal buffer - chuck by chunk
while (offset < sampleToReadPerGet)
{
shortRead = audioRecored.read(audioData, offset ,sampleToReadPerGet - offset);
offset += shortRead;
}
//stop tapping into the microphone
audioRecored.stop();
//average the buffer
int averageSoundLevel = 0;
for (int i = 0 ; i < sampleToReadPerGet ; i++)
{
averageSoundLevel += audioData[i];
}
averageSoundLevel /= sampleToReadPerGet;
What are those values? are they decibels?
Edit:
The values goes from -200 to 3000.
The value of shortRead is sampleToReadPerGet (1000).
Not sure what "those values" you are referring to, the raw output or the averaged values, but the raw output are instantaneous amplitude levels. It's important to realize that such values are not referenced to anything in particular. That is, just because you are reading 20, does not tell you 20 of what.
Taking the average of these values doesn't make any sense, because those values swing above and below zero. Do it long enough and you'll just get zero.
It might make sense to take the average of the squares, and then find the square root of the average. This is called the RMS. However, without a fixed buffer size to average over, this is hazardous at best.
To measure dB, you will have to use the formula dB = 20 log_10 (|A|/A_r) where A is the amplitude and A_r is the reference amplitude -- clearly, you must decide what you are referencing (you can calibrate the HTC, or measure against the maximum level or something like that).
You should not get negative values. The values span 16 or 8 bits, so your max is about 32000 or something. The values have no units.
Also, I recommend root-mean-squared instead of an average for determining volume. It is more stable.
What you should try:
Increase the buffer size by 3: Your app may not be reading it fast
enough so you need some space. Otherwise you might be getting some
buffer overflow errors (which you are not checking for in your code)
Try the code in gast-lib: It helps you periodically record audio and also provides you an AsyncTask.
Root mean squared:
public static double rootMeanSquared(short[] nums)
{
double ms = 0;
for (int i = 0; i < nums.length; i++)
{
ms += nums[i] * nums[i];
}
ms /= nums.length;
return Math.sqrt(ms);
}

Categories

Resources