I m under delphi, under android, and it look like an openGL problem. I have problem to obtain smooth scrolling. i always have Jerks :( To render the frame we do like this :
will running do begin // << main loop
doSomething // << calculate for example position of controls regarding velocity
executeAllTimerProcedure;
eglMakeCurrent(..)
paintEverything // << paint all texture, this took around 5 ms
eglSwapBuffers(...) // << stuck waiting the vsync signal
end;
The problem is that eglSwapBuffers wait for vsync signal. as the vsync signal is fired every 16.6 ms (60fps) the main loop is blocking at the step eglSwapBuffers. This can cause some jerk in the scrolling if the calculation made in dosomething are delayed. Also all timer are delayed.
i try to set eglSwapInterval(eglGetCurrentDisplay, 0); and this make eglSwapBuffers non blocking but now (and i don't know why) the jerks are even worse :(
So what i can do to make my scrolling /animation very smooth without any jerks ?
Related
under android with delphi, I have a background thread. in this background thread i call a procedure in synchronize that just do a repaint:
TThread.Synchronize(nil,
procedure
begin
repaint;
end);
to resume you the delphi source code, under android repaint simply put a bool to true saying to the main looper to repaint the form on next loop :
procedure TPlatformAndroid.InvalidateWindowRect(const AForm: TCommonCustomForm; R: TRectF);
begin
TAndroidWindowHandle(AForm.Handle).NeedsUpdate := True;
end;
I have also a mousemove event on my form in with i catch the mouse move and where i also call repaint
procedure TMyFrame.FrameMouseMove(Sender: TObject; Shift: TShiftState; X, Y: Single);
begin
VPos := Y;
repaint;
end;
What i absolutely not understand now, is why this repaint (in the background thread) cause some jerks in the animation if (and only if) i also move at the same time my finger on the screen (if i do not move the finger then i don't have jerks)
How this could be possible? because the repaint simply put to true something that was already to true via the mousemove (or vice versa). how this can delay sometime by more than 100 ms the call to the onpaint of my frame (producing jerks). Removing the onpaint from the background thread remove also the jerks ... no idea what could be wrong
NOTE:
if in TPlatformAndroid.InternalProcessMessages if i replace
if TWindowManager.Current.RenderIfNeeds then
HasEvents := True
by
TWindowManager.Current.RenderImmediately;
HasEvents := True;
then the lag disappears a lot (but not completely)
ok, i find out why.
in delphi the problem is that, even if repaint is a trivial operation with set a flag to true, the following operation that will actually render the form is not trivial and in any case (even if the form didn't change) it's will take 16 ms (the vsync signal penalty). my problem is also connected to ui event because when their is ui event, delphi fire every timer every 1 ms instead of every 10 ms
so you can understand that if i call the repaint at a bad time (say for exemple the nanosecond before normally the timer must be fired), then we will have 16ms delay to wait and then when the timer will fire, it's will be too late, we will have already one frame dropped (in fact not but last paint did nothing), and then we will have lag :(
also it's because of this bug: https://quality.embarcadero.com/browse/RSP-18982
To be specific, I'll present my question in an example:
Say at t = 0 ms, a frame was completed and became visible to the user on the screen. From that point on, I began the work to draw the next frame. However, the work took too long that I missed the frame's due time of t = 16 ms. If finally this next frame was ready at t = 23 ms. When would it actually be visible to the user? t = 23 ms or t = 32 ms (at the next drawing "heart beat" if any)?
And also, where in the Android source code can I find the answer myself?
You actually get 16.666ms per frame, so if you aren't ready to draw at that point then the next attempt would be made at 34ms. Colt from Google has a good video explaining this actually. https://www.youtube.com/watch?v=HXQhu6qfTVU
I am developing an Android video player. I use ffmpeg in native code to decode video frame. In the native code, I have a thread called decode_thread that calls avcodec_decode_video2()
int decode_thread(void *arg) {
avcodec_decode_video2(codecCtx, pFrame, &frameFinished,pkt);
}
I have another thread called display_thread that uses aNativeWindow to display a decoded frame on a SurfaceView.
The problem is that if I let the decode_thread run continuously without a delay. It significantly reduces the performance of avcodec_decode_video2(). Sometimes it takes about 0.1 seconds to decode a frame. However if I put a delay on the decode_thread. Something likes this.
int decode_thread(void *arg) {
avcodec_decode_video2(codecCtx, pFrame, &frameFinished,pkt);
usleep(20*1000);
}
The performance of avcodec_decode_video2() is really good, about 0.001 seconds. However putting a delay on the decode_thread is not a good solution because it affects the playback. Could anyone explain the behavior of avcodec_decode_video2() and suggest me a solution?
It looks impossible that the performance of video decoding function would improve just because your thread sleeps. Most likely the video decoding thread gets preempted by another thread, and hence you get the increased timing (hence your thread did not work). When you add a call to usleep, this does the context switch to another thread. So when your decoding thread is scheduled again the next time, it starts with the full CPU slice, and is not interrupted in the decode_ video2 function anymore.
What should you do? You surely want to decode packets a little bit ahead than you show them - the performance of avcodec_decode_video2 certainly isn't constant, and if you try to stay just one frame ahead, you might not have enough time to decode one of the frames.
I'd create a producer-consumer queue with the decoded frames, with the top limit. The decoder thread is a producer, and it should run until it fills up the queue, and then it should wait until there's room for another frame. The display thread is a consumer, it would take frames from this queue and display them.
I'm writing a simple NDK OpenSL ES audio app that records the users touches on a virtual piano keyboard and then plays them back forever over a set loop. After much experimenting and reading, I've settled on using a separate POSIX loop to achieve this. As you can see in the code it subtracts any processing time taken from the sleep time in order to make the interval of each loop as close to the desired sleep interval as possible (in this case it's 5000000 nanoseconds.
void init_timing_loop() {
pthread_t fade_in;
pthread_create(&fade_in, NULL, timing_loop, (void*)NULL);
}
void* timing_loop(void* args) {
while (1) {
clock_gettime(CLOCK_MONOTONIC, &timing.start_time_s);
tic_counter(); // simple logic gates that cycle the current tic
play_all_parts(); // for-loops through all parts and plays any notes (From an OpenSL buffer) that fall on the current tic
clock_gettime(CLOCK_MONOTONIC, &timing.finish_time_s);
timing.diff_time_s.tv_nsec = (5000000 - (timing.finish_time_s.tv_nsec - timing.start_time_s.tv_nsec));
nanosleep(&timing.diff_time_s, NULL);
}
return NULL;
}
The problem is that even using this the results are better, but quite inconsistent. sometimes notes will delay for perhaps even 50ms at a time, which makes for very wonky playback.
Is there a better way of approaching this? To debug I ran the following code:
gettimeofday(&timing.curr_time, &timing.tzp);
__android_log_print(ANDROID_LOG_DEBUG, "timing_loop", "gettimeofday: %d %d",
timing.curr_time.tv_sec, timing.curr_time.tv_usec);
Which gives a fairly consistent readout - that doesn't reflect the playback inaccuracies whatsoever. Are there other forces at work with Android preventing accurate timing? Or is OpenSL ES a potential issue? All the buffer data is loaded into memory - could there be bottlenecks there?
Happy to post more OpenSL code if needed... but at this stage I'm trying figure out if this thread loop is accurate or if there's a better way to do it.
You should consider seconds when using clock_gettime as well, you may get greater timing.start_time_s.tv_nsec than timing.finish_time_s.tv_nsec. tv_nsec starts from zero when tv_sec is increased.
timing.diff_time_s.tv_nsec =
(5000000 - (timing.finish_time_s.tv_nsec - timing.start_time_s.tv_nsec));
try something like
#define NS_IN_SEC 1000000000
(timing.finish_time_s.tv_sec * NS_IN_SEC + timing.finish_time_s.tv_nsec) -
(timing.start_time_s.tv_nsec * NS_IN_SEC + timing.start_time_s.tv_nsec)
I'm trying to profile my renderer, and I'm seeing some weird profiling behavior that I can't explain.
I'm using a glSurfaceView, which I have set to render continuously.
This is how my onDrawFrame() is structured
public void onDrawFrame(GL10 unused) {
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);
executeAllDrawCommands();
}
This was behaving slowly under light load, so I created a timer class and started to profile this some. I was quite surprised by what I saw.
I put some probes on my onDrawFrame method like so:
public void onDrawFrame(GL10 unused) {
swapTimer.end();
clearTimer.start();
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);
clearTimer.end();
drawTimer.start();
executeAllDrawCommands();
drawTimer.end();
swapTimer.start();
}
clearTimer measures the time it takes to call glClear, drawTimer measures the time it takes to run all my draw calls, and swapTimer measures the time from when onDrawFrame exits and when it returns (the time taken to call eglSwapBuffers).
When I ran a very lightly loaded scene, I got some really strange numbers I can't explain:
swapTimer : 20ms (average)
clearTimer : 11ms (average)
drawTimer : 2ms (average)
I expected the swap time to be somewhat largish, as I believe the device has vsync forced enable at ~30fps, though I don't know why the actual 'clear' call is blocking for 11 milliseconds? I thought it was just supposed to issue an asynchronous command and return?
When I draw a much more busy scene, the numbers change quite a bit:
swapTimer : 2ms (average)
clearTimer : 0ms (average)
drawTimer : 44ms (average)
In this scene my draw calls are taking so much time that it looks like its hiding a lot of the vsync period, and the block on the clear call totally goes away.
Is there any explanation for why glClear is blocking on my lightly loaded scene?
Link to my 'Timer' class source code in case someone is suspicious of my measuring technique: http://pastebin.com/bhXt368W
I put a glFinish (and finishTimer.start()/end() around it), and it takes all the time away from glClear. Instead now glFinish takes some number of milliseconds, and glClear becomes instant.
That explains it.
When your scene is very light and the drawings are rendered very fast, the time to clear and fill the pixels with the new color will take some time (it will always take time, otherwise the renderer is behind and is currenty drawing new stuff). The newer Android devices have fillrate limits. For example, Nexus One has a fillrate lock at 30 Hz - the screen will be synced at that frequency no matter how fast your actual drawings are going. If the drawings finishes under 30 Hz the renderer will sync up with the screen. This is why you notice this delay, which you should notice even if you remove the glClear() call. The renderer is before and faster than the screen's updates.
When the renderer have many objects to draw, the synchronization will halt (given your busy scene's profile data) because the renderer is now after the screen's updates.
When you use glFinish(), it removes the time the glClear() function otherwise would cause, which, by following the fillrate logic, means that glFinish() is now the function that is ensuring synchronization with the screen.
Calculations:
F = 1/T
Easy scene:
F = 1/T = 1/((20+11+2)*10^-3) =~ 30 Hz
The synchronization's delay time appears in your profiler. Renderer is being synchronized with the screen. That means that if you remove the glClear() or the glFinish() call, the delay would appear somewhere else.
Heavy scene:
F = 1/T = 1/((2+0+44)*10^-3)) =~ 22 Hz
The synchronization's delay time does not appear in your profiler. Renderer is after the screen's update frequency.
It seems like this is all related to vsync
That seems to be correct.