Develop Reference

Android TextureView / Drawing / Painting Performance

I am attempting to make a drawing/painting app using TextureView on Android. I want to support a drawing surface of up to 4096x4096 pixels, which seems reasonable for my minimum target device (which I use for testing) which is a Google Nexus 7 2013 which has a nice quad core CPU and 2GB memory.
One of my requirements is that my view must be inside of a view that allows it to be zoomed in and out and panned, which is all custom code I have written (think UIScrollView from iOS).
I've tried using a regular View (not TextureView) with OnDraw and performance was absolutely horrible - less than 1 frame a second. This happened even if I called Invalidate(rect) with only the rect that changed. I tried turning off hardware acceleration for the view, but nothing rendered, I assume because 4096x4096 is too big for software.
I then tried using TextureView and performance is a little better - about 5-10 frames per second (still terrible but better). The user draws into a bitmap, which is then later drawn into the texture using a background thread. I'm using Xamarin but hopefully the code makes sense to Java people.
private void RunUpdateThread()
{
try
{
TimeSpan sleep = TimeSpan.FromSeconds(1.0f / 60.0f);
while (true)
{
lock (dirtyRect)
{
if (dirtyRect.Width() > 0 && dirtyRect.Height() > 0)
{
Canvas c = LockCanvas(dirtyRect);
if (c != null)
{
c.DrawBitmap(bitmap, dirtyRect, dirtyRect, bufferPaint);
dirtyRect.Set(0, 0, 0, 0);
UnlockCanvasAndPost(c);
}
}
}
Thread.Sleep(sleep);
}
}
catch
{
}
}
If I change lockCanvas to pass null instead of a rect, performance is great at 60 fps, but the contents of the TextureView flicker and get corrupted, which is disappointing. I would have thought it would simply be using an OpenGL frame buffer / render texture underneath or at least have an option to preserve contents.
Are there any other options short of doing everything in raw OpenGL in Android for high performance drawing and painting on a surface that is preserved in between draw calls?

First off, if you want to understand what's going on under the hood, you need to read the Android Graphics Architecture document. It's long, but if you sincerely want to understand the "why", it's the place to start.
About TextureView
TextureView works like this: it has a Surface, which is a queue of buffers with a producer-consumer relationship. If you're using software (Canvas) rendering, you lock the Surface, which gives you a buffer; you draw on it; then you unlock the Surface, which sends the buffer to the consumer. The consumer in this case is in the same process, and is called SurfaceTexture or (internally, more aptly) GLConsumer. It converts the buffer into an OpenGL ES texture, which is then rendered to the View.
If you turn off hardware acceleration, GLES is disabled, and TextureView cannot do anything. This is why you got nothing when you turned hardware acceleration off. The documentation is very specific: "TextureView can only be used in a hardware accelerated window. When rendered in software, TextureView will draw nothing."
If you specify a dirty rect, the software renderer will memcpy the previous contents into the frame after rendering is complete. I don't believe it sets a clip rect, so if you call drawColor(), you will fill the entire screen, and then have those pixels overwritten. If you aren't currently setting a clip rect, you may see some performance benefit from doing so. (I didn't check the code though.)
The dirty rect is an in-out parameter. You pass the rect you want in when you call lockCanvas(), and the system is allowed to modify it before the call returns. (In practice, the only reason it would do this would be if there were no previous frame or the Surface were resized, in which case it would expand it to cover the entire screen. I think this would have been better handled with a more direct "I reject your rect" signal.) You're required to update every pixel inside the rect you get back. You are not allowed to alter the rect, which you appear to be trying to do in your sample -- whatever is in the dirty rect after lockCanvas() succeeds is what you're required to draw on.
I suspect the dirty rect mis-handling is the source of your flickering. Sadly, this is an easy mistake to make, as the behavior of the lockCanvas() dirtyRect arg is only documented in the Surface class itself.
Surfaces and buffering
All Surfaces are double- or triple-buffered. There is no way around this -- you cannot read and write simultaneously and not get tearing. If you want a single buffer that you can modify and push when desired, that buffer will need to be locked, copied, and unlocked, which creates stalls in the composition pipeline. For best throughput and latency, flipping buffers is better.
If you want the lock-copy-unlock behavior, you can write that yourself (or find a library that does it), and it will be as efficient as it would be if the system did it for you (assuming you're good with blit loops). Draw to an off-screen Canvas and blit the bitmap, or to a OpenGL ES FBO and blit the buffer. You can find an example of the latter in Grafika's "record GL app" Activity, which has a mode that renders once off-screen, and then blits twice (once for display, once for recording video).
More speed and such
There are two basic ways to draw pixels on Android: with Canvas, or with OpenGL. Canvas rendering to a Surface or Bitmap is always done in software, while OpenGL rendering is done with the GPU. The only exception is that, when rendering to a custom View, you can opt to use hardware acceleration, but this does not apply when rendering to the Surface of a SurfaceView or TextureView.
A drawing or painting app can either remember the drawing commands, or just throw pixels at a buffer and use that as its memory. The former allows for deeper "undo", the latter is much simpler, and has increasingly better performance as the amount of stuff to render grows. It sounds like you want to do the latter, so blitting from off-screen makes sense.
Most mobile devices have a hardware limitation of 4096x4096 or smaller for GLES textures, so you won't be able to use a single texture for anything larger. You can query the size limit value (GL_MAX_TEXTURE_SIZE), but you may be better off with an internal buffer that is as large as you want, and just render the portion that fits on screen. I don't know what the Skia (Canvas) limitation is offhand, but I believe you can create much larger Bitmaps.
Depending on your needs, a SurfaceView may be preferable to a TextureView, as it avoids the intermediate GLES texture step. Anything you draw on the Surface goes directly to the system compositor (SurfaceFlinger). The down side to this approach is that, because the Surface's consumer is not in-process, there is no opportunity for the View system to handle the output, so the Surface is an independent layer. (For a drawing program this could be beneficial -- the image being drawn is on one layer, your UI is on a separate layer on top.)
FWIW, I haven't looked at the code, but Dan Sandler's Markers app might be worth a peek (source code here).
Update: the corruption was identified as a bug and fixed in 'L'.

UPDATE I ditched TextureView and now use an OpenGL view where I call glTexSubImage2D to update changed pieces of a render texture.
OLD ANSWER
I ended up tiling TextureView in a 4x4 grid. Depending on the dirty rect each frame, I refresh the appropriate TextureView views. Any view that is not updated that frame I call Invalidate on.
Some devices, such as the Moto G phone have an issue where the double buffering is corrupted for one frame. You can fix that by calling lockCanvas twice when the parent view has it's onLayout called.
private void InvalidateRect(int l, int t, int r, int b)
{
dirtyRect.Set(l, t, r, b);
foreach (DrawSubView v in drawViews)
{
if (Rect.Intersects(dirtyRect, v.Clip))
{
v.RedrawAsync();
}
else
{
v.Invalidate();
}
}
Invalidate();
}
protected override void OnLayout(bool changed, int l, int t, int r, int b)
{
for (int i = 0; i < ChildCount; i++)
{
View v = GetChildAt(i);
v.Layout(v.Left, v.Top, v.Right, v.Bottom);
DrawSubView sv = v as DrawSubView;
if (sv != null)
{
sv.RedrawAsync();
// why are we re-drawing you ask? because of double buffering bugs in Android :)
PostDelayed(() => sv.RedrawAsync(), 50);
}
}
}

How to efficiently display images in Android

I'm making a simple 2d game in android. However, even with only a few images on screen, I'm running into problems with fps(its around 15-25 and quite stuttery).
Currently I use canvas, and simply set up an image each frame with everything I need to display on it.
Here's some of my render code:
public void render(Painter g) {
g.setColor(Color.rgb(208, 244, 247));
g.fillRect(0, 0, GameMainActivity.GAME_WIDTH, GameMainActivity.GAME_HEIGHT);
g.setColor(Color.rgb(139,69,19));
g.fillRect(0, 1400, GameMainActivity.GAME_WIDTH, GameMainActivity.GAME_HEIGHT - 1400);
g.drawImage(Assets.background3stone, currentBackground3X, 141);
g.drawImage(Assets.background3stone, currentBackground3X + 3459, 141);
g.drawImage(Assets.background2stone, currentBackground2X, 141);
g.drawImage(Assets.background2stone, currentBackground2X + 3459, 141);
g.drawImage(Assets.ground, currentBackground1X, 760);
g.drawImage(Assets.ground, currentBackground1X + 3000, 760);
renderNews(g);
renderButtons(g);
g.drawImage(Assets.leaderboard, 40, 180, 148, 148);
g.drawImage(Assets.achievements, 40, 354, 148, 148);
g.drawString("FPS: " + frameValue, 50, 200);
}
This code runs for every iteration of the game loop.
Does anyone know any ways to optimize my performance? Currently I redraw literally everything every frame, is there a way to not redraw static images? Would switching to openGl help a lot?
Thanks!

There's also a lot of good and efficient game engines available for Android (For example LibGDX). Game engines are usually well optimized for showing, moving and animating multiple images and they usually come with plenty other useful features.
However, if you prefer not to use game engines, I'd recommend using OpenGL to get some boost. You may also get some boost by optimizing your images. Try to reduce the sizes of images and don't use too large images. Also try to load images efficiently to the program.

Switching to openGL would always help a lot. But your problem is that you're drawing directly to the screen. Don't do that. Draw to a bitmap, then blit the bitmap to the screen in a single operation. The performance of that will be MUCH higher. It also would allow you to do your drawing on a 2nd thread and only draw to the screen on the main one.

Before switching to openGL you may want to try to not redraw the image itself, but just to move it. You just need to set new LayoutParams to your image. You may also want to look at TranslateAnimation in case you are moving stuff softly. Or you use a matrix transformation.

Implementing Anti-Aliasing (FSAA?) on Android, OpenGL

I'm currently using OpenGL on Android to draw set width lines, which work great except for the fact that OpenGL on Android does not natively support the anti-aliasing of such lines. I have done some research, however I'm stuck on how to implement my own AA.
FSAA
The first possible solution I have found is Full Screen Anti-Aliasing. I have read this page on the subject but I'm struggling to understand how I could implement it.
First of all, I'm unsure on the entire concept of implementing FSAA here. The article states "One straightforward jittering method is to modify the projection matrix, adding small translations in x and y". Does this mean I need to be constantly moving the same line extremely quickly, or drawing the same line multiple times?
Secondly, the article says "To compute a jitter offset in terms of pixels, divide the jitter amount by the dimension of the object coordinate scene, then multiply by the appropriate viewport dimension". What's the difference between the dimension of the object coordinate scene and the viewport dimension? (I'm using a 800 x 480 resolution)
Now, based on the information given in that article the 'jitter' coordinates should be relatively easy to compute. Based on my assumptions so far, here is what I have come up with (Java)...
float currentX = 50;
float currentY = 75;
// I'm assuming the "jitter" amount is essentially
// the amount of anti-aliasing (e.g 2x, 4x and so on)
int jitterAmount = 2;
// don't know what these two are
int coordSceneDimensionX;
int coordSceneDimensionY;
// I assume screen size
int viewportX = 800;
int viewportY = 480;
float newX = (jitterAmount/coordSceneDimensionX)/viewportX;
float newY = (jitterAmount/coordSceneDimensionY)/viewportY;
// and then I don't know what to do with these new coordinates
That's as far as I've got with FSAA
Anti-Aliasing with textures
In the same document I was referencing for FSAA, there is also a page that briefly discusses implementing anti-aliasing with the use of textures. However, I don't know what the best way to go about implementing AA in this way would be and whether it would be more efficient than FSAA.
Hopefully someone out there knows a lot more about Anti-Aliasing than I do and can help me achieve this. Much appreciated!

The method presented in the articles predates the time, when GPUs were capable of performing antialiasing themself. This jittered rendering to a accumulation buffer is not really state of the art with realtime graphics (it is a widely implemented form of antialiasing for offline rendering though).
What you do these days is requesting an antialiased framebuffer. That's it. The keyword here is multisampling. See this SO answer:
How do you activate multisampling in OpenGL ES on the iPhone? – although written for the iOS, doing it for Android follows a similar path. AFAIK On Android this extension is used instead http://www.khronos.org/registry/gles/extensions/ANGLE/ANGLE_framebuffer_multisample.txt

First of all the article you refer to uses the accumulation buffer, whose existence I really doubt in OpenGL ES, but I might be wrong here. If the accumulation buffer is really supported in ES, then you at least have to explicitly request it when creating the GL context (however this is done in Android).
Note that this technique is extremely inefficient and also deprecated, since nowadays GPUs usually support some kind of multisampling atialiasing (MSAA). You should research if your system/GPU/driver supports multi-sampling. This may require you to request a multisample framebuffer during context creation or something similar.
Now back to the article. The basic idea of this article is not to move the line quickly, but to render the line (or actually the whole scene) multiple times at very slightly different (at sub-pixel accuracy) locations (in image space) and average these multiple renderings to get the final image, every frame.
So you have a set of sample positions (in [0,1]), which are actually sub-pixel positions. This means if you have a sample positon (0.25, 0.75) you move the whole scene about a quarter of a pixel in the x direction and 3 quarters of a pixel in the y direction (in screen space, of course) when rendering. When you have done this for each different sample, you average all these renderings together to gain the final antialiased rendering.
The dimension of the object coordinate scene is basically the dimension of the screen (actually the near plane of the viewing volume) in object space, or more practically, the values you passed into glOrtho or glFrustum (or a similar function, but with gluPerspective it is not that obvious). For modifying the projection matrix to realize this jittering, you can use the functions presented in the article.
The jitter amount is not the antialiasing factor, but the sub-pixel sample locations. The antialiasing factor in this context is the number of samples and therfore the number of jittered renderings you perform. And your code won't work, if I assume correctly and you try to only jitter the line end points. You have to draw the whole scene multiple times using this jittered projection and not just this single line (it may work with a simple black background and appropriate blending, though).
You might also be able to achieve this without an accum buffer using blending (with glBlendFunc(GL_CONSTANT_COLOR, GL_ONE) and glBlendColor(1.0f/n, 1.0f/n, 1.0f/n, 1.0f/n), with n being the antialiasing factor/sample count). But keep in mind to render the whole scene like this and not just this single line.
But like said this technique is completely outdated and you should rather look for a way to enable MSAA on your ES platform.

Android, fastest way to draw a bitmap to canvas

Just wondering what the fastest way is to draw a bitmap to canvas?
Currently I have a bitmap (and canvas for drawing) which i use to double buffer drawing calls, and then when i draw to canvas have a scrolling effect by applying a 1px canvas translation. This alone will reduce the framerate from 60+ FPS to ~40, quite a hit. Im not using surfaceView (or GLSurfaceView) at the moment but just wondering if im missing anything that would improve the speed. onDraw() code below
#Override
public void onDraw(Canvas canvas)
{
//update fps text
mFpsTracker.frameTouch();
if(mBufferedBitmap == null)
{
mBufferedBitmap = Bitmap.createBitmap(getWidth(), getHeight(), Bitmap.Config.ARGB_4444);
mBufferedCanvas = new Canvas(mBufferedBitmap);
}
paint.setColor(Color.BLUE);
mBufferedCanvas.drawLine(0, getHeight(), getWidth(), getHeight(), paint);
mBufferedCanvas.translate(0, -1);
canvas.drawBitmap(mBufferedBitmap, 0, 0, null);
//draw fps
mTextPaint.setColor(Color.WHITE);
canvas.drawText(mFpsTracker.getFPSString(), 40, 40, mTextPaint);
invalidate();
}

please see this blog post by Romain Guy.
A video version is available here.
Don't use ARGB_4444 anymore. It is deprecated. Each pixel is only allocated 4 bits per channel (hence the name). ARBG_8888 offers 16,777,216 colors instead of ARBG_4444's 4,096, but uses 4 bytes per pixel instead of 2.
In Gingerbread, Android made ARGB_8888 the standard format for Surface and increased memory allotment per process because of it.
It is more efficient to set your Window's and (assuming you are using streamlined SurfaceView) SurfaceHolder's format to RGBA_8888. This avoids format changes which are noticeably slower.
Other tips include:
Limit alpha compositing, as this requires comparatively expensive blending from Skia.
Request Bitmap Options that prefer the ARGB_8888 Config and disable dithering.
Remove the Window background if possible.
Enable hardware acceleration, but beware of unsupported operations.
On a 2.1 device, I am able to draw at least 300 bitmaps on-screen at 50 fps.

I got same problem as yours, please tell me when you found some thing new.
This is what I founded so far:
For android version > 3 better not to use double buffer, because you are getting hardware acceleration(need to set true in manifest)
set paint.setDither(true) it will work better on any device with different color then ARGB_4444, witch are most of the devices. Check this out for more info.

In onSizeChange you can resize or create bitmaps according to the canvas size, then the frame drawing will be much faster about 60fps, however using custom view in an endless loop slows down on some and becomes jumpy on some android devices hence I do not recommend it. Instead, it is better to use SurfaceView.
Check this example: How can I use the animation framework inside the canvas?

U need create your bitmap somewhere ( example onCreate or other place (constructor will be good)) because when u do scrolling effect you again and again created new bitmap. So just need create in constructor and than using this bitmap.
it's good solution for me when i have similar problems.
try created this mBufferedBitmap = Bitmap.createBitmap(getWidth(), getHeight(), Bitmap.Config.ARGB_4444); in other place where no invalidate .
hope it's help you.

I think you can get good performance out of a canvas.. but it takes a lot of work..
if you start off with a good performance graphics library, then even if you get a lot of things wrong, you probably will still end up with good performance :) lol
there is a competition running for the fastest drawing library... libgdx is currently winning...
http://code.google.com/p/libgdx/wiki/SimpleApp#Project_Setup

Drawing rotated Bitmaps on Canvas is slow: any other options?

I'm developing a live wallpaper app which will draw an analog clock. So far I reached the point where I draw a clock using three bitmaps: dial, hour hand, minute hand.
When I draw hours and minutes I do it like this (pseudo code):
// without this it looks weird after rotate
Paint smoothPaint = new Paint(FILTER_BITMAP_FLAG);
canvas.drawBitmap(dialBitmap, 0, 0, null);
canvas.rotate(ANGLE_FOR_HOUR_HAND, w/2, h/2);
canvas.drawBitmap(hourBitmap, 0, 0, smoothPaint);
canvas.rotate(ANGLE_FOR_MINUTE_HAND, w/2, h/2);
canvas.drawBitmap(minuteBitmap, 0, 0, smoothPaint);
This works okay except that I'm getting a really low frame rate: about 13 fps.
If I turn off bitmap filtering, it increases to about 24 fps, but still low for me.
I know that 2d drawing on Canvas is not accelerated in any way, but still want to ask are there any other options for me to speed up this code? Drawing to bitmap natively somehow? Something else maybe?
Using OpenGL is an option, but its not that simple since LiveWallpapers do not support GL drawing, so I need to use 3rd party hacks (which I know exist) for that which I would rather not do...

"Any other options?" you ask. You have two:
Don't use bitmaps. The second Cube SDK sample, for example, pops out a rotating 20-line dodecahedron at 25 fps.
As you've mentioned, you can sometimes use a "hack" for special-purpose graphics within live wallpaper. I suggest you at least take a look at the following three...you may find them useful one day: AndEngine (has a live wallpaper extension), LibGDX (allows you to prototype in a desktop environment; live wallpaper facilities still under development, methinks), and GLWallpaperService (designed specifically to bring GLSurfaceView to live wallpaper).
In your post you say that 24 fps is "still low" for you. It is my understanding that 24 fps is approximately the limit of human perception (it's what Flash uses by default, anyway). I suggest you go no higher, even if you can, in order to preserve battery life.
Edit: One other "hack" to consider: renderscript. That's how some of the sophisticated wallpapers (Galaxy, Grass, Water) that come with many phones work. See renderscript enabling java files here: https://android.googlesource.com/platform/packages/wallpapers/Basic/+/master/src/com/android/wallpaper

How about rendering your hand configuration to an intermediate bitmap each time the minute-hand changes, and then rendering that static bitmap to your canvas on each frame?

Found few things that helped to increase frame rate, will post here as an answer to myself :)
First is to use Canvas.drawBitmap(Bitmap, Matrix, Paint) overload to supply the matrix, instead of using canvas.save() + canvas.rotate() + canvas.restore() - more efficient.
Second: reducing the size of bitmaps. Earlier they were padded to near screen-size of hdpi (480pix) with transparent pixels added around them. After I cropped these transparent pixels away and modified the rendering code to account for that, rotating became much more efficient (as expected).