Develop Reference

How to efficiently scale up video frame using NDK

I am doing an Android project about dealing with video frame, I need to handle every frame before display it. The process includes scaling up frames from 1920x1080 to 2560x1440 resolution, color space conversion and some necessary image processing based on RGB, and all these works should be finished within 33ms~40ms.
I have optimized the yuv->rgb and other processing with arm neon, they worked well. But I have to scale up frame firstly from 1080p to 2k resolution, it's the bottleneck of performance now.
My question is how to efficiently scale up image from 1080p to 2k resolution within 20ms, I don't have much experience about scaling algorithm, so any suggestions are helpful.
Could I use arm neon to optimize the existing algorithm?
The hardware environment:
CPU: Samsung Exynos 5420
Memory: 3GB
Display: 2560X1600 px
Update:
I will describe my decoding process, I use MediaCodec to decode the normal video(H.264) to YUV(NV12), the default decoder is hardware, it's very fast. Then I use arm neon to convert NV12 to RGBW, and then send RGBW frame to surfaceflinger to display. I just use normal SurfaceView rahter than GLSurfaceView.
The bottleneck is how to scale up YUV from 1080p to 2K fast.

I find that examples work well, so allow me to lead with this example program that uses OpenGL shaders to convert from YUV -> RGB: http://www.fourcc.org/source/YUV420P-OpenGL-GLSLang.c
What I envision for your program is:
Hardware video decodes H.264 stream -> YUV array
Upload that YUV array as a texture to OpenGL; actually, you will upload 3 different textures-- Y, U, and V
Run a fragment shader that converts those Y, U, and V textures into an RGB(W) image; this will produce a new texture in video memory
Run a new fragment shader against the texture generated in previous step in order to scale the image
There might be a bit of a learning curve involved here, but I think it's workable, given your problem description. Take it one step at a time: get the OpenGL framework in place, try uploading just the Y texture and writing a naive fragment shader that just emits a grayscale pixel based on the Y sample, then move onto correctly converting the image, then get a really naive upsampler working, then put a more sophisticated upsampler into service.

I'd also recommend opengl es too, mainly because of the project I'm currently working on, also playing video. For me, the display is 1920 x 1080, so the texture I'm using is 2048 x 1024. I get approx 35 fps on a quad core arm7.
Use a GLSurfaceView and your own custom renderer. If you're using ffmpeg then once you've decoded your video frames, use sws_scale to scale your frame and then just upload it into the opengl texture. The larger your texture/display, the less fps you will get because it a lot of time taken uploading large images to the gpu every frame.
Depending on your needs for decoding your video input is what you will have to research. For me, I had to compile ffmpeg for android and start from there.

my apologies for putting this in an answer. i dont have enough points to make a comment.
I'd like to add that you might run into OGL texture limitations. I have tried to use OGL for the opposite problem; scaling down from the camera in real time. the problem is that the max OGL texture is 2048x2048. Not sure if this is true for all devices. this limit was true on newer kit like N72013 and LG2. in the end, i had to write in in the NDK without OGL by optimising the hell out of it by hand.
good luck, though.

Use OpenGL to draw shader to a stream?

I'm working on this photo app that uses OpenGL 2.0 with a Renderer, an off-screen GLSurfaceView and some shader scripts (*.fsh and *.vsh).
after loading the shader scripts from Assets folder, preparing the GL surface and context, etc, etc we finally call GLES20.glDrawArrays(GLES20.GL_TRIANGLE_FAN, 0, 4); and it works quite nicely and it generates the bitmaps with the effects.
The problem, OF COURSE, is the memory limitations and any large enough bitmap (regardless of device, not so big for old Gingerbread and very large images for the Nexus 10) and it will produce and OutOfMemoryException.
I'm not so knowledgeable in OpenGL and the way I know to deal with very large amounts of data is to use a stream so it's not necessary to hold it all in memory.
So the question is, is there a way to do apply an openGl shader/renderer through a Stream instead of a in-memory Bitmap ? If yes, any pointer to a link or base procedure?

Not exactly sure what you mean by Stream but here's another solution. Split rendering up into multiple passes. Fore instance, if you have a 512x512 texture and a corresponding quad to texture but can only afford to upload a 256x256 due to memory restrictions do the following:
split up the texture into 4 chunks
create a single, fitting texture object
for each chunk
upload the current chunk into the tex objects data store
draw 1/4 of the quad, e.g. top-left and texture accordingly
Note that the above example assume a 512x512 texture and screen-size. In any case, I think you get the idea.
Obviously, this is the usual memory/performance trade-off.You circumvent memory restrictions by using more bandwidth for transfers and do more rendering.
Note: I'm a desktop GL guy and I'm not quite sure how memory is split up betweem the GPU and the rest, or if there even is some dedicated VRAM. I assume you've got a limited amount available for GL resources which is even smaller than the overall system memory.

Hardware Acceleration pre-Honeycomb

I'm playing around with the Android API with a long term prospect of developing a 2D game. Normally, I could live with just using the Canvas for drawing sprites as the graphics for my game. I'd like to be able to perform lots of drawing for visual effects, but it seems that the Android versions prior to Honeycomb (3.0, API level 11) don't support hardware acceleration.
I'm not sure what that exactly means though. I can't get myself round to believe that the drawing is done by the CPU, pixel by pixel !?! If I end up having effects like glow, lens effects etc... I'll be drawing over each pixel quite a few times. Am I right to believe that a typical smartphone CPU will not be able to cope with that at ~30 FPS?
I don't have the luxury to target Android versions >=3.0 as they constitute 8% of the already not SO big Android market. Should I take my time to go the OpenGL way (I'm a beginner at OpenGL)? If I do so, do you think I'll gain anything by overlaying a GLSurfaceView taking care of the effects on top of a custom android view using a Canvas to do the drawing otherwise. Or is it for any reason a bad idea to mix the two?

Oh God yes. Esepecially if you're targetting pre Android 3 devices, going from SurfaceView (with Canvas.drawXxx() calls) to OpenGlSurface works great. Not only do you have faster frames (updates) per second but memory consumption is A LOT better.
Here's some of the points I've noticed:
If you want to do (multi) sprite animations, doing them with the images loaded as OpenGL textures and displayed on an OpenGL quad gives you a lot more memory space. This is because while regular Bitmap objects are capped by the Android-per-process memory limit (which is something like 16-48 Mb, depending on device and Android version), creating OpenGL textures out of those images (and clearing the iamges right after) doesn't have this limitation. You're only limited by the total memory on the device which is A LOT more then 16-48 megs.
Secondly, but still related to this, with Android 2 and below tracking how much memory a Bitmap instance takes is a lot trickier since those intances are not reported against the Java heap memory. They are allocated in some other memory space. In short, another hassle less if you use OpenGL.
Simple animations such as rotating an image become a breeze with OpenGL. You just texture a quad, then roate it any way you want. Equivalent with Sprite animation is to sequentially display different (rotated versions ) of the image. This is better for memory consumption and speed.
If you're doing a 2D-like game, using OpenGL's orthogonal projection not only simplifies a lot of the (useless, in this case) hassle you'd have with a regular OpenGL perspective projection, but it actually alleviates A LOT of the issues you'd get with regular SurfaceView when needing to scale all your graphical elements so they'd look the same size on different screen resolutions/proportions. With OpenGL's ortho projection you effectifelly create a fixed area of desired widht and height and then have OpenGL project it on the device screen area automatically.
It comes without saying that making simple effects such as a pulsating light affecting some graphic element is a lot easier to do with OpenGL (where you just make the light and have it pulsate, and everything is lit accordingly) rather than simulate this with SurfaceView and baked in sprites.
I've actually started a small asteriod defence-like game with SurfaceView and Canvas and then quickly switched to OpenGL for the above mentioned reasons. Long story short, the game now looks better, and while it initially ran at 16 UPS on a Samsung TEOS and 30 UPS on an Optimus LG 2x it now runs at 33 UPS on the Teos and about 80 UPS on the LG 2x.

Tips for improving OpenGL ES fill rate on Android

In my live wallpaper I'm drawing 3 textured quads that covers the whole screen. On Nexus One I get 40fps. I'm looking for ways to improving performance.
The quads are blended on top of each other, textures are loaded from RGB_8888 bitmaps. Textures are 1024x1024.
I've got
glDisable(GL_DITHER);
glHint(GL_PERSPECTIVE_CORRECTION_HINT, GL_FASTEST);
glDisable(GL10.GL_LIGHTING);
glDisable(GL_DEPTH_TEST);
glEnable(GL_TEXTURE_2D);
glEnable(GL_BLEND);
glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
Things I've tried, all resulting in the same 40fps:
Reduce texture size to 512x512 and 256x256
Use draw_texture extension
Disable blending
Change texture filtering from GL_LINEAR to GL_NEAREST
Use VBOs (desperate try, since there are just 3 quads...)
Run the drawing code in standalone activity (in case being a live wallpaper somehow affects performance)
If I draw 2 layers instead of 3, fps rises to 45 or so,
then drawing just 1 layer sees fps rise to 55. I guess I'm getting limited by fill rate, since turning off and on the potentially costly features result in the same fps, and the only thing that seems to improve fps is just drawing less...
I'm mulling over the idea of texture compression, but supporting the different compression formats doesn't seem like fun. ETC1 has no alpha channel which I need, and I'm not sure if PVRTC and ATITC can even be used from Java and OpenGL ES 1.0 or 1.1.
I'd be glad to hear ideas on what else to try.
I can give pointer to the current version of wallpaper and screenshots if that's of use.

You probably already thought of this, but just in case you didn't:
calling glClear at the start of your frame probably isn't necessary
you could do the first pass with blending disabled
Also, did you try doing it in 1 pass with a multi-texturing approach?
edit: And another thing: writing to the z-buffer is not needed, so either use a context without z-buffer, or disable depth writing with glDepthMask(GL_FALSE).

glCompressedTexImage2D is available in Java (and NDK), however available compression format depends on GPU.
The AndroidManifest.xml File > supports-gl-texture
PowerVR SGX - Nexus S, Galaxy S/Tab, DROID - PVRTC
Adreno - Nexus One, EVO - ATITC
Tegra2 - Xoom, Atrix - S3TC
If you use these compression format and want to support various Android devices, you must prepare for many compressed textures, but the GPU native compression texture should improve rendering performance.
GPU Profiling and callbacks in OpenGL ES
OpenGL ES, Changing texture format
from RGBA8888 to RGBA4444 will
improve fill rate?

The android opengl framework min3d is able to draw these objects or scenes at a full 60fps.
The framework is opensource and is available for download and use at: http://code.google.com/p/min3d/
I would recommend comparing your code to it to see what you have done wrong/differently in order to improve your performance.

Is Android (read typical devices) fast enough for a game that requires plotting pixel by pixel rather than blitting

i have an idea for an Android game which is a little different from the typical game that usually moves sprites(bitmaps) around the screen. Id want to plot lots of little pixels to create my visuals.
PROS
no bitmaps required
pixel plotting of stuff like "fire" can react to wind.
no need to scale bitmaps, works w/ any screen res (lets pretend device can handle more drawing because its got a bigger screen).
CONS
slower to plot pixels than blit bitmaps
need lot of animation frames.
WISHES
id like to update my game in real time, more is better 30fps is good but not essential, 15fps is enough.
PERFORMANCE Q...
Is the typical Android device fast enough to plot say half a screenful of pixels w/ a default background ?
if full screen is not practical what window size should be able to handle such refreshes

You can achieve it in native code using C/C++.
You would create offscreen bitmap, get its memory address, draw raw RGB pixels to it, then blit entire offscreen bitmap to screen in View.onDraw method. You can get nice framerates in fullscreen.
Btw, Android also manipulates pixels in low level by native code (using Skia library). Java classes like Canvas, Bitmap etc just jump to native code to get the work done.
Using Android's Canvas in Java to paint pixels would be unrealistically slow.