I've run across this peculiar "feature" using my HTC One and the ShutterCallback for the Android Camera (I'll try to get my hands on other devices to see if this is repeatable on other hardware). For the record, I'm not focusing, I've locked the exposure, turned off antibanding, and set the JPEG qaulity to 100.
The first time I take a picture and measure the delay from when takePicture is called to when onShutter is called, the delay is consistently around 850 +/- 50 milliseconds. However, after the first picture, the shutter lag for subsequent pictures becomes around 530 +/- 50 milliseconds. Any thoughts what might be causing this? I've tried looking through the JNI documentation, but nothing stands out. I wonder if there is some cache in the camera hardware that points to a buffer. However, I don't see why it would make a difference of 300 milliseconds.
EDIT: I see a gap of about 150 for the shutter lag between the first picture and subsequent pictures on a Galaxy Nexus.
Related
I'm trying to achieve the optimal settings for decoding barcodes using Android's tricky Camera2 API, using TEMPLATE_STILL_CAPTURE and CONTROL_SCENE_MODE_BARCODE WHILE playing with all the other settings like FPS, AF and AE.
So far however, I have been unable to remove the banding seen when reading barcodes from screens.
What would be the way to remove, or reduce, banding when using the Camera2 API taking pictures of screens?
You should set the anti-banding mode, see CONTROL_AE_ANTIBANDING_MODE
...
.setCaptureRequestOption(
CaptureRequest.CONTROL_AE_ANTIBANDING_MODE,
CameraMetadata.CONTROL_AE_ANTIBANDING_MODE_AUTO
)
.setCaptureRequestOption(
CaptureRequest.CONTROL_MODE,
CameraMetadata.CONTROL_MODE_AUTO
)
.setCaptureRequestOption(
CaptureRequest.CONTROL_SCENE_MODE,
CameraMetadata.CONTROL_SCENE_MODE_BARCODE
)
...
https://developer.android.com/reference/android/hardware/camera2/CaptureRequest#CONTROL_AE_ANTIBANDING_MODE
Depending on the technology used by the screen, the screen itself may be flickering at some rate. Unfortunately, the screen is also likely to be bright, meaning the camera has to reduce its exposure time so it doesn't overexpose.
A shorter exposure time makes it harder to apply anti-banding, since anti-banding normally involves having an exposure time that's a multiple of the flicker period. That is, if the flicker is at 100 hz (10 ms period), you need an exposure time that's a multiple of 10 ms. But if the screen is so bright that the camera needs a 5 ms exposure, then there's no way to cancel out the flicker.
In addition, most camera flicker detectors only handle 50 and 60-hz main power flicker rates. Increasingly they do more, since LED lighting has flicker rates that often unrelated to the power line frequencies, but if the device only handles 50 and 60hz, it probably can't compensate for most displays that flicker.
Besides turning on antibanding mode (which should be on by default anyway), you could try increasing exposure compensation to make the image brighter, which may give the device enough room to set an exposure time for flicker reduction. But beyond that there's not much you can do, unfortunately.
I've written an OpenGL ES 2 app using NativeActivity that renders around 6 textured triangles that you can drag around using touch input. The app is rendering as fast as possible, calling glClear() before and eglSwapBuffers() after rendering. As expected the frame rate is capped to the display frequency of around 60 fps and the scene moves around smoothly. However, after some time (10 to 60 seconds) the movement starts to stutter even though the frame rate is still 60 fps. The device I'm using is OnePlus 3.
At first I suspected the culprit was battery saving mode, but investigation revealed no correlation between cpus' frequencies and the stutter.
I did notice something interesting though, the stutter starts when queueBuffer() execution time goes up.
This is the systrace before stutter:
This is the systrace after stutter:
This is the systrace during the transition from before to after:
How do I stop the app from stuttering?
I still don't know the exact cause of the stuttering, but I managed to find a work-around: Only render frames on Choreographer.postFrameCallback and set eglPresentationTimeANDROID to two vsyncs in the future. This avoids filling up the buffer queue, which seems to be one of the causes of the stuttering.
As a bonus it also improved the touch input latency a little.
More info here: Minimize Android GLSurfaceView lag
I have an OpenGL game for Android. It runs at a good 60fps when the screen is touched. When I release my finger it goes back down to around 30fps. Does the touch event/release raise/lower a thread's priority and if so how can I replicate this to keep it at a constant 60fps. This only seems to be an issue on Galaxy Note 2 so far.
I'll assume you are using onDrawFrame and setRenderMode(RENDERMODE_CONTINUOUSLY).
30 and 60FPS indicates that your implementation of onDrawFrame is called as the device's screen refreshes. Most displays refresh at 60Hz, giving you 60FPS.
It is likely that the Galaxy Note 2 has some power saving feature that limits screen refresh to 30Hz when there are no touches on screen. Check if there's any way to disable this feature.
AFAIK, OpenGL ES does not specify a standard for screen refresh rates, you will need a throttling function to ensure that your game runs/feels the same (i.e. at the same speed) despite differences in FPS.
Yes.
The best way to observe this phenomena is to use systrace with the "freq" tag enabled. You probably need a rooted device, and you definitely need one on which systrace is enabled.
systrace will record changes in the clock frequency for various components. It varies by device, but you can usually get the per-core CPU clocks and GPU memory rate. You will likely see several of them drop significantly at the same time your frame rate drops.
The motivation for doing this is to reduce power requirements and extend battery life. The assumption is that, while your finger is in contact with the screen, you're actively doing something and the device should be as responsive as possible. After a brief period of time, the clocks will slow to a level appropriate for the current workload. The heuristics that determine how long to wait before slowing, and how much to slow down, are tuned for each device.
(This has caused some people to create a thread that just sits and spins on multi-core devices as a way to artificially prop up the CPU clock rate. Not recommended. See also this answer.)
The bottom line is that this isn't a simple matter of adjusting thread priorities. You have to choose between recognizing that the slowdown will happen and adapting to it (by making your game updates independent of frame rate), or figure out some way to fool the device into staying in a higher-power mode when you want smooth animation.
(For anyone who wants to play along at home: build a copy of Grafika and start the "Record GL app" activity. If you drag your finger around the screen all will be well, but if you leave it alone for a few seconds you may start to see the dropped-frame counter rising as the app falls behind. Seen on Nexus 5, Nexus 7 (2013), and others.)
I'm developing a GL live wallpaper that uses very little CPU and only modest GPU. On my older test phone, it can run at a full 58fps or so most of the time. But occasionally the effects ramp up, and then the render times jitter between 16ms and 50ms per frame. For example, it'll render several frames at 16ms, slide up to 50ms over a dozen frames or so, render several more frames at 50ms, then slide back down to 16ms and repeat. I discovered that if I set the CPU governor to "performance" (or "conservative", curiously enough) instead of the default "ondemand" it'll render with full effects at full speed. Alternatively, if I leave the governor alone and insert a busy loop in my code (increment a variable 100,000 times per frame) that bumps my CPU usage up enough to transition to a higher clock rate and render smoothly as well.
So it seems on this phone my app is bottlenecked by the GPU, but only when it throttles down. Now, I wouldn't mind if the GLSurfaceView rendered at a slower rate according to the GPU clock, but my problem here is that I'm getting the bursts of alternating high and low frame rates which makes my animation look fluid/frameskippy/fluid/frameskippy/etc. several times per second. It seems like the GPU clock is ramping up and down like crazy?
I got a visible improvement by using RENDERMODE_WHEN_DIRTY and calling requestRender() on a strictly timed thread, but the darn GPU keeps ramping up and down. Why won't it either render as fast as it can at the slower clock, or just jump to and STAY AT the higher clock?
The best solution I've come up with so far is using a sliding window to detect the average frame update time, then applying the difference from the target frame time until the two values converge. The time between render updates is slower but at least it's roughly constant. So that works in theory, but it takes several seconds to reach a steady state and it looks bad in the meantime.
I think a third option might be to cannibalize the GLSurfaceView source and make a custom version. From what I understand, the blocking GL calls are made in there, so it would be much easier for me to time render calls and react accordingly. I'm not very comfortable attempting that though because there's a lot of code in there that I'd have to spend a lot of time understanding before I could even begin to mess with it. Plus I'd then have to worry about how well version X of GLSurfaceView plays with any version Y of Android.
So, with all that said, do I have any other options here? Is there an easier fix to this?
try fixing the frame rate by pausing the thread (thread sleep) for the remaining time to reach a constant frame rate.
Given a known periodic motion (e.g., walking), I'd like to take a full resolution snapshot at the same point in the motion (i.e., the same time offset within different periods). However on the Nexus S (currently running OS 4.1.1 but the same was true of previous OS versions), I'm seeing so much variability in the shutter lag that I cannot accurately plan the timing of the snapshot. Here is a histogram of the shutter lags of 50 photographs. (I measured the shutter lag with one System.nanoTime() just before Camera.takePicture() and another System.nanoTime() at the beginning of the shutter callback. The camera lens was consistently covered to remove any variability due to lighting.)
Is there anything I can do in the application to reduce this shutter lag variability? (In this application, the mean lag can be any duration but the standard deviation must be small ... much smaller than the 0.5 s standard deviation of the shutter lags shown in the above histogram.) I'm hoping someone has a clever suggestion. If I don't get any suggestions, I'll post a feature request in the Android bug tracker.
UPDATE:
I turned off auto-focus (by setting focus to infinity) following RichardC's suggestion at https://groups.google.com/forum/?hl=en&fromgroups#!topic/android-developers/WaHhTicKRA0
It helped, as shown in the following histogram. Any ideas to reduce the shutter lag variability even more?
UPDATE 2:
I disabled the remaining auto parameters: white balance, scene mode, flash. The following histogram of shutter lag time variability seems to be about as good as it gets for a Nexus S running OS 4.1.1. It would be nice to have much less variability in the shutter lag time, perhaps by specifying an optional minimum shutter lag time in Camera.takePicture() which would delay the shutter if it were ready before the specified minimum. I've posted a feature request to the Android issue tracker. If you are also interested in getting the feature, "star" it at http://code.google.com/p/android/issues/detail?id=35785
what's the x axis and what's the y axis?
you can use low level coding (like NDK/renderscript) instead of using java .
you might be able to change the settings of the camera to make it faster .