I would just like to ask if SurfaceFlinger is always called for any type of drawing into the screen?
Example, displaying of JPG file to the screen.
SurfaceFlinger is not what draws your window. It allocates a frame buffer for your window, which the framework running in your application draws directly to without interacting with SurfaceFlinger. The only interaction SurfaceFlinger is involved with when you draw your window is to composite the final new frame buffer to the screen once you are done drawing a frame.
http://pierrchen.blogspot.jp/2014/02/what-is-surfaceflinger-in-android.html
SurfaceFlinger is an Android system service, responsible for
compositing all the application and system surfaces into a single
buffer that is finally to be displayed by display controller.
Let's zoom in above statement.
SurfaceFlinger is a system wide service but it is not directly
available to application developer as Sensor or other services can
be. Every time you want to update your UI, SurfaceFlinger will kick
in. This explains why SurfaceFlinger is a battery drainer.
Besides your application surfaces, there are system surfaces,
including status bar, navigation bar and, when rotation happens,
surfaces created by the system for rotation animation. Most
applications have only one active surface - the one of current
foreground activity, others have more than one when SurfaceView is
used in the view hierarchy or Presentation mode is used.
SurfaceFlinger is responsible for COMPOSITING all those surfaces. A
common misunderstanding is that SurfaceFinger is for DRAWING. It is
not correct. Drawing is the job of OpenGL. The interesting thing is
SurfaceFlinger used openGL for compositing as well.
The composition result will be put in a system buffer, or native
window, which is the source for display controller to fetch data from.
This is what you see in the screen.
Yes, SurfaceFlinger is Android's compositor so it takes everything that will get displayed, figures out what the resulting frame will look like and then sends it off to be displayed on the screen via the graphics card's EGL interface.
You can get the idea that it controls the result of everything you see in a post by Android developer Jeff Sharkey where he tints the whole screen for nightmode. I also found a beamer presentation that looks good about this topic.
Related
I'm using a Texture widget, rendering its content from native code using OpenGL ES. In native code I call ANativeWindow_fromSurface and from that create an EGL surface. AIUI what happens is:
The ANativeWindow represents the producer side of a buffer queue.
Calling eglSwapBuffers causes a texture to be sent to this queue.
Flutter receives the texture and renders it using Skia when the TextureLayer is painted.
The texture is scaled to match the size of the TextureLayer (the scaling happens in AndroidExternalTextureGL::Paint()).
I'm trying to figure out how to synchronise the OpenGL rendering. I think I can use the choreographer to synchronise with the display vsync, but I'm unclear on how much latency this bufferqueue-then-render-with-skia mechanism introduces. I don't see any means to explicitly synchronise my native code's generation of textures with the TextureLayer's painting of them.
The scaling appears to be a particularly tricky aspect. I would like to avoid it entirely, by ensuring that the textures the native code generates are always of the right size. However there doesn't appear to be any direct link between the size of the TextureLayer and the size of the Surface/ANativeWindow. I could use a SizeChangedLayoutNotifier (or one of various alternative hacks) to detect changes in the size and communicate them to the native code, but I think this would lag by at least a frame so scaling would still take place when resizing.
I did find this issue, which talks about similar resizing challenges, but in the context of using an OEM web view. I don't understand Hixie's detailed proposal in that issue, but it appears to be specific to embedding of OEM views so I don't think it would help with my case.
Perhaps using a Texture widget here is the wrong approach. It seems to be designed mainly for displaying things like videos and camera previews. Is there another way to host natively rendered, interactive OpenGL graphics in Flutter?
As we know, we can choose TextureView, SurfaceView and GLSurfaceView for android camera preview.
Which one is best choice for camera preview ? I'm focused on the camera performance.
From a performance perspective, SurfaceView is the winner.
With SurfaceView, frames come from the camera and are forwarded to the system graphics compositor (SurfaceFlinger) with no copying. In most cases, any scaling will be done by the display processor rather than the GPU, which means that instead of scanning the pixels once for scaling and again for scan-out, they're only scanned once.
GLSurfaceView is a SurfaceView with some wrapper classes that handle EGL setup and thread management. You can't use OpenGL ES on a Surface that is receiving camera frames, so you're doing extra work with no benefit. (The overhead is minor one-time setup, not per-frame, so you likely won't be able to measure the difference.)
TextureView receives the frames in a SurfaceTexture as an "external" OpenGL ES texture, then uses GLES to render them onto the app's UI surface. The scaling and rendering are performed by the GPU, and the result is then forwarded to SurfaceFlinger. This is the slowest option, but also the most flexible of the Views.
If you'd like to learn more about how the system works, see the Android Graphics Architecture document.
" The SurfaceView creates a new window in the Android Windowsystem. Its advantage is, that if the SurfaceView gets refreshed, only this window will be refreshed. If you additionally update UI Elements (which are in another window of the windowsystem), then both refresh operations block themselfes (especially when ui drawing is hardwaresupported) because opengl cannot handle multi thread drawing properly.
For such a case it could be better using the TextureView, cause it's not another window of the Android Windowsystem. so if you refresh your View, all UI elements get refreshed as well. (Probably) everything in one Thread.
Hope I could help some of you! "
Source : stackoverflow.com
GLSurfaceView is a SurfaceView with a wrapper class that does all the EGL setup and inter-thread messaging for you.
Its completely upto you what you put to use.. They have their pros and cons over eachother :)
I found some information about graphic stack android and saw this picture:
But I cannot understood why there was twice OpenGL was used? Is it possible to use openGl only one time after SurfaceFlinger? And after what in this picture does EGL library execute?
OpenGLES can be used for drawing both 2D and 3D shapes in current systems. In the picture you are showing, in the application stack, GL can be used to render objects by the application. Say, this output goes to a buffer "B". There can be many such applications, so they all create buffers, say B1, B2, B3. Now, there needs to be some framework, that is responsible for deciding which of these buffers gets shown on the display screen, or what combination of buffers gets shown. This is popularly called as "compositor". In the compositor, GL is again used to show content on to display.
So, GL can be used in both applications and compositors, which is what is shown in the stack above.
EGL is an API (from Khronos, like OpenGL, OpenGLES) for interfacing to the window system, in this case the Android window system. It creates the buffers B1, B2 etc, into which applications can draw, and also the final display buffers.
So, EGL creates/manages buffers/display, GL is a platform independent API that is responsible for 2D/3D drawing. Hope this helps.
I would like to write an application for Android which displays stuff on screen using the framebuffer. This will run only on a specific rooted device, so permissions etc is not a problem. Same application (simple test version anyway) is already running okay on PC/Linux.
The questions:
How to avoid the Android OS from accessing the framebuffer? I would like, while my application is running, to have the OS never touch the framebuffer, no writes and no ioctls. What do I need to do to get exclusive use of the framebuffer, and then (when my application quits) give it back to the OS?
Are there any differences between Android framebuffer and Linux framebuffer to watch out for?
P.S. I would like to start my application as a regular Android application (with some native code), it just has no visible UI except for framebuffer draws which take over the whole screen. It would be nice to still be able to get events from the OS.
See also:
http://www.kandroid.org/online-pdk/guide/display_drivers.html
Hi Alex Not sure why / how to stop android OS from writing to framebuffer. As long as your android application is visible and on top you have the control as what you want to display.
Your application should have an activity with a SurfaceView ( you may want your application to hide notification bar call this function in oncreate of your activity)
requestWindowFeature(Window.FEATURE_NO_TITLE); )
your activity should have SurfaceHolder.Callback implementation to handle callbacks as when the surface is ready to filled with framebuffer. Get the surface holder object as SurfaceView.getHolder() incase you want set pixel formats of the view etc.
Once "surfaceCreated" callback is called you can safely pass your surfaceview object(passing width and height maybe a good idea too) to the native so that you can fill it framebuffer using "ANativeWindow" class.
Check NDK sample code to see how to use the class NDK documentation
SurfaceHolder.Callback documentation
SurfaceHolder documentation
essentially you need to these (ON JB /Kitkat)
get the native window (ANativeWindow) associated with the surfaceview by ANativeWindow_fromSurface.
Acquire a lock on the ANativeWindow by ANativeWindow_acquire .
Set geometry parameters(window,width,height,pf) for the nativewindow by ANativeWindow_setBuffersGeometry
Load the nativewindow with the frambuffer stored (apply dirty rectangle if any here)
by ANativeWindow_lock
Final step to unlock and post the changes for rendering by ANativeWindow_unlockAndPost
Go through the ndk sample examples in case you need sample code.NDK documentation
Is there any tradeoff of add some OpenGL to a "serious" (not game) Android app?
The reason why I want to use OpenGL, is to add some 3d behaviour to a few views.
According to this http://developer.android.com/guide/topics/graphics/opengl.html OpenGL 1.0 is available in every Android device and doesn't require modification of manifest file. So there will never be compatibility issues.
The only 2 things I can think about is 1. mantainability by other developers which can't OpenGL. And possible 2. Integration problems with other components / not well reusable (although, not sure).
Is there also anything else, unexpected things, overhead of some sort, complications, etc.?
Asking because it seems not to be a very popular practice, people seem to prefer to "fake" the 3d with 2d or give it up. Don't know if it's only because they don't want to learn OpenGL.
I use OpenGL for some visualization in a released app, and I have an uncaught exception handler in place to catch any exception coming from the GLThread and disable OpenGL the next time the app is run, since I had some crash reports in the internals of GLSurfaceView.java coming in from buggier devices. If the 3D rendering is not crucial to your app, this is one approach you can take so that users with these devices can continue to use the app.
From Android 3.0+ you can also preserve the EGL context by calling GLSurfaceView. setPreserveEGLContextOnPause(true);. You'll only really need to do this if your renderer is very expensive to initialize, and it only works if you're not destroying the GLSurfaceView in between (i.e. the default behavior of an activity when rotating the device). If you're not loading that many resources then initializing OpenGL is usually fast enough.
From the SurfaceView docs (emphasis mine):
The surface is Z ordered so that it is behind the window holding its SurfaceView; the SurfaceView punches a hole in its window to allow its surface to be displayed. The view hierarchy will take care of correctly compositing with the Surface any siblings of the SurfaceView that would normally appear on top of it. This can be used to place overlays such as buttons on top of the Surface, though note however that it can have an impact on performance since a full alpha-blended composite will be performed each time the Surface changes.
The advantage is that your GL thread can update the screen independently of the UI thread (i.e. it doesn't need to render to a texture and render the texture to the screen); the disadvantage is that something needs to composite your view with the screen. If you're lucky, this can be done in the "hardware composer"; otherwise it is done on the GPU and may be a bit wasteful of GPU resources (see For Butter or Worse: Smoothing Out Performance in Android UIs at 27:32 and 40:23).
If your view is small, it may be better to use a TextureView. This will render to a texture and render the texture as part of the normal view hierarchy which might be better, but can increase latency. The downside is it's only available since API level 14.