Develop Reference

Android Camera.PreviewCallback scheduling (with OpenGL and OpenCV)

I'm developing an application which requires heavy image processing using camera input and real-time results display. I've decided to use OpenGL and OpenCV along with Android's normal camera API. So far it has become a bit of a multithreading nightmare, and unfortunately I feel very restricted by the lack of documentation on the onPreviewFrame() callback.
I am aware from the documentation that onPreviewFrame() is called on the thread which acquires the camera using Camera.open(). What confuses me is how this callback is scheduled - it seems to be at a fixed framerate. My current architecture relies on the onPreviewFrame() callback to initiate the image processing/display cycle, and it seems to go into deadlock when I block the camera callback thread for too long, so I suspect that the callback is inflexible when it comes to scheduling. I'd like to slow down the framerate to test this, but my device doesn't support this.
I started with the code over at http://maninara.blogspot.ca/2012/09/render-camera-preview-using-opengl-es.html. This code is not very parallel, and it is only meant to display exactly the data which the camera returns. For my needs, I adapted the code to draw bitmaps, and I use a dedicated thread to buffer the camera data to another dedicated heavy-lifting image processing thread (all outside of the OpenGL thread).
Here is my code (simplified):
CameraSurfaceRenderer.java
class CameraSurfaceRenderer implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener,
Camera.PreviewCallback
{
static int[] surfaceTexPtr;
static CameraSurfaceView cameraSurfaceView;
static FloatBuffer pVertex;
static FloatBuffer pTexCoord;
static int hProgramPointer;
static Camera camera;
static SurfaceTexture surfaceTexture;
static Bitmap procBitmap;
static int[] procBitmapPtr;
static boolean updateSurfaceTex = false;
static ConditionVariable previewFrameLock;
static ConditionVariable bitmapDrawLock;
// MarkerFinder extends CameraImgProc
static MarkerFinder markerFinder = new MarkerFinder();
static Thread previewCallbackThread;
static
{
previewFrameLock = new ConditionVariable();
previewFrameLock.open();
bitmapDrawLock = new ConditionVariable();
bitmapDrawLock.open();
}
CameraSurfaceRenderer(Context context, CameraSurfaceView view)
{
rendererContext = context;
cameraSurfaceView = view;
// … // Load pVertex and pTexCoord vertex buffers
}
public void close()
{
// … // This code usually doesn’t have the chance to get called
}
#Override
public void onSurfaceCreated(GL10 unused, EGLConfig config)
{
// .. // Initialize a texture object for the bitmap data
surfaceTexPtr = new int[1];
surfaceTexture = new SurfaceTexture(surfaceTexPtr[0]);
surfaceTexture.setOnFrameAvailableListener(this);
//Initialize camera on its own thread so preview frame callbacks are processed in parallel
previewCallbackThread = new Thread()
{
#Override
public void run()
{
try {
camera = Camera.open();
} catch (RuntimeException e) {
// … // Bitch to the user through a Toast on the UI thread
}
assert camera != null;
//Callback set on CameraSurfaceRenderer class, but executed on worker thread
camera.setPreviewCallback(CameraSurfaceRenderer.this);
try {
camera.setPreviewTexture(surfaceTexture);
} catch (IOException e) {
Log.e(Const.TAG, "Unable to set preview texture");
}
Looper.prepare();
Looper.loop();
}
};
previewCallbackThread.start();
// … // More OpenGL initialization stuff
}
#Override
public void onDrawFrame(GL10 unused)
{
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
synchronized (this)
{
surfaceTexture.updateTexImage();
}
// Binds bitmap data to texture
bindBitmap(procBitmap);
// … // Acquire shader program ttributes, render
GLES20.glFlush();
}
#Override
public synchronized void onFrameAvailable(SurfaceTexture surfaceTexture)
{
cameraSurfaceView.requestRender();
}
#Override
public void onPreviewFrame(byte[] data, Camera camera)
{
Bitmap bitmap = markerFinder.exchangeRawDataForProcessedImg(data, null, camera);
// … // Check for null bitmap
previewFrameLock.block();
procBitmap = bitmap;
previewFrameLock.close();
bitmapDrawLock.open();
}
void bindBitmap(Bitmap bitmap)
{
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, procBitmapPtr[0]);
bitmapDrawLock.block();
if (bitmap != null && !bitmap.isRecycled())
{
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, bitmap, 0);
bitmap.recycle();
}
bitmapDrawLock.close();
previewFrameLock.open();
}
#Override
public void onSurfaceChanged(GL10 unused, int width, int height)
{
GLES20.glViewport(0, 0, width, height);
// … // Set camera parameters
camera.startPreview();
}
void deleteTexture()
{
GLES20.glDeleteTextures(1, surfaceTexPtr, 0);
}
}
CameraImgProc.java (abstract class)
public abstract class CameraImgProc
{
CameraImgProcThread thread = new CameraImgProcThread();
Handler handler;
ConditionVariable bufferSwapLock = new ConditionVariable(true);
Runnable processTask = new Runnable()
{
#Override
public void run()
{
imgProcBitmap = processImg(lastWidth, lastHeight, cameraDataBuffer, imgProcBitmap);
bufferSwapLock.open();
}
};
int lastWidth = 0;
int lastHeight = 0;
Mat cameraDataBuffer;
Bitmap imgProcBitmap;
public CameraImgProc()
{
thread.start();
handler = thread.getHandler();
}
protected abstract Bitmap allocateBitmapBuffer(int width, int height);
public final Bitmap exchangeRawDataForProcessedImg(byte[] data, Bitmap dirtyBuffer, Camera camera)
{
Camera.Parameters parameters = camera.getParameters();
Camera.Size size = parameters.getPreviewSize();
// Wait for worker thread to finish processing image
bufferSwapLock.block();
bufferSwapLock.close();
Bitmap freshBuffer = imgProcBitmap;
imgProcBitmap = dirtyBuffer;
// Reallocate buffers if size changes to avoid overflow
assert size != null;
if (lastWidth != size.width || lastHeight != size.height)
{
lastHeight = size.height;
lastWidth = size.width;
if (cameraDataBuffer != null) cameraDataBuffer.release();
//YUV format requires 1.5 times as much information in vertical direction
cameraDataBuffer = new Mat((lastHeight * 3) / 2, lastWidth, CvType.CV_8UC1);
imgProcBitmap = allocateBitmapBuffer(lastWidth, lastHeight);
// Buffers had to be resized, therefore no processed data to return
cameraDataBuffer.put(0, 0, data);
handler.post(processTask);
return null;
}
// If program did not pass a buffer
if (imgProcBitmap == null)
imgProcBitmap = allocateBitmapBuffer(lastWidth, lastHeight);
// Exchange data
cameraDataBuffer.put(0, 0, data);
// Give img processing task to worker thread
handler.post(processTask);
return freshBuffer;
}
protected abstract Bitmap processImg(int width, int height, Mat cameraData, Bitmap dirtyBuffer);
class CameraImgProcThread extends Thread
{
volatile Handler handler;
#Override
public void run()
{
Looper.prepare();
handler = new Handler();
Looper.loop();
}
Handler getHandler()
{
//noinspection StatementWithEmptyBody
while (handler == null)
{
try {
Thread.currentThread();
Thread.sleep(5);
} catch (Exception e) {
//Do nothing
}
};
return handler;
}
}
}
I want an application which is robust, no matter how long it takes for the CameraImgProc.processImg() function to finish. Unfortunately, the only possible solution when camera frames are being fed in at a fixed rate is to drop frames when the image processing hasn't finished yet, or else I'll quickly have a buffer overflow.
My questions are as follows:
Is there any way to slow down the Camera.PreviewCallback frequency on demand?
Is there an existing Android API for getting frames on demand from the camera?
Are there existing solutions to this problem which I can refer to?

onPreviewFrame() is called on the thread which acquires the camera
using Camera.open()
That's a common misunderstanding. The key word that is missing from this description is "event". To schedule the camera callbacks to a non-UI thread, you need and "event thread", a synonym of HandlerThread. Please see my explanation and sample elsewhere on SO. Well, using a usual thread to open camera as in your code, is not useless, because this call itself may take few hundred milli on some devices, but event thread is much, much better.
Now let me address your questions: no, you cannot control the schedule of camera callbacks.
You can use setOneShotPreviewCallback() if you want to receive callbacks at 1 FPS or less. Your milage may vary, and it depends on the device, but I would recommend to use setPreviewCallbackWithBuffer and simply return from onPreviewFrame() if you want to check the camera more often. Performance hit from these void callbacks is minor.
Note that even when you offload the callbacks to a background thread, they are blocking: if it takes 200 ms to process a preview frame, camera will wait. Therefore, I usually send the byte[] to a working thread, and quickly release the callback thread. I won't recommend to slow down the flow of preview callbacks by processing them in blocking mode, because after you release the thread, the next callback will deliver a frame with undefined timestamp. Maybe it will be a fresh one, or maybe it will be one buffered a while ago.

You can schedule the callback in later platform releases (>4.0) indirectly. You can setup the buffers that the callback will use to deliver the data. Typically you setup two buffers; one to be written by the camera HAL while you read from the other one. No new frame will be delivered to you (by calling your onPreviewFrame) until you return a buffer that the camera can write to. It also means that the camera will drop frames.

UI hangs after sleep in thread after lockcanvas

I´m trying to implement a fractal renderer on android. It uses a SurfaceView to display the results.
I use a seperate Thread for rendering to keep the UI working during processing, but it locks up until unlockCanvasAndPost is called. Is this the expected behavior? How can I keep the UI from locking up?
In this example I simulate the long processing by Thread.sleep.
My SurfaceView-Class looks like this:
public class MyView extends SurfaceView implements SurfaceHolder.Callback {
RenderThread rt = null;
public MyView(Context context, AttributeSet attrs) {
super(context, attrs);
getHolder().addCallback(this);
}
#Override
public void surfaceChanged(SurfaceHolder holder, int format, int width,
int height) {
}
#Override
public void surfaceCreated(SurfaceHolder holder) {
// TODO Auto-generated method stub
rt = new RenderThread(holder);
rt.start();
}
#Override
public void surfaceDestroyed(SurfaceHolder holder) {
rt.setStop();
}
}
And this is the RenderThread-class:
public class RenderThread extends Thread {
SurfaceHolder holder;
boolean running = true;
public RenderThread(SurfaceHolder holder) {
this.holder = holder;
}
public void setStop() {
running = false;
}
#Override
public void run() {
while(running) {
Canvas canvas = holder.lockCanvas();
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
if(canvas == null) return;
Paint p = new Paint();
p.setARGB(255, 255, 0, 0);
canvas.drawLine(0, 0, 100, 100, p);
holder.unlockCanvasAndPost(canvas);
}
}
}

Several possibilities.
You don't show how you've incorporated your MyView class into the activity that contains it. Because you've subclassed SurfaceView, you have to add code to your activity to specifically deal with what you've added with RenderThread. The activity's onPause() method, for example, should be sending a signal to your MyView class to stop its RenderThread.
The surfaceDestroyed() method gets called just before the SurfaceHolder's surface is destroyed. You cannot return from that method until your RenderThread has stopped touching it. This is accomplished through synchronization. Official Android documentation for surfaceDestroyed().
There's a terrific example of how all of this is properly done in WindowSurface.java in the ApiDemos project (for Eclipse, installed with "Samples for SDK" via the Android SDK Manager). Once you've loaded the project into Eclipse (with File->New->Project->Android Sample Project), find WindowSurface.java in com.example.android.apis.graphics within the project.
Finally, if your actual code in the thread's run method (what you've replaced with Thread.sleep in the interest of brevity here) does anything with the canvas then your check for canvas==null should be immediately below the attempted locking of that canvas. Ideally, it's better form to do that anyway.

Android: Canvas not being redrawn

There are multiple similar questions like mine, but these questions didn't help me.
I'm making a game. The game thread, SurfaceView and Activity is already finished and works so far. The problem is that the canvas is not redrawn. At startup, it draws the icon on the background, but at every tick, the icon doesn't move (It should move once a second). I want to mention, that I never needed to call postInvalidate. I have a working example where I never called it, but it doesn't work in my current example (I don't want to go into it deeper, since I actually don't need to call it). I copied the current code from my working example, the concept and the way of implementation is exactly the same, but my current code doesn't refresh the canvas. When I log the drawing positions in onDraw method, I see that it's coordinates are updated every second as expected, so I can be sure it's a canvas drawing problem. I have searched for hours but I didn't find what's different to my working example (except that I'm using another Android version and I don't extend thread but implement Runnable, because it's a bad style to extend thread. Nevertheless, I also extended thread to see if there is any difference, but it doesn't help). I already tried to clean the canvas by using canvas.drawColor(Color.BLACK), but that didn't help either. I already tried to use background colors instead of a background image which changes randomly every tick, but it didn't change but stays always the same.
I figured out that the canvas at the very first call has a density of (for example) 240. After the second tick, the canvas density is always 0. I know that the density will not help me here, but maybe it's an important information for someone.
Here are the important classes....
game layout, R.layout.game
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:id="#+id/gameContainer"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical" >
<com.mydomain.mygame.base.game.GameSurface
android:id="#+id/gameSurface"
android:layout_width="match_parent"
android:layout_height="0dp"
android:layout_weight="1.0"
android:background="#drawable/background_game" >
</com.mydomain.mygame.base.game.GameSurface>
<!-- ...more definitions -->
</LinearLayout>
GameActivity (contains layout)
public class GameActivity extends Activity
{
#SuppressWarnings("unused")
private GameSurface gameSurface;
#Override
protected void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
setContentView(R.layout.game);
gameSurface = (GameSurface)findViewById(R.id.gameSurface);
//TODO on button click -> execute methods
}
}
GameSurface (log in onDraw shows updated coordinates every tick)
public class GameSurface extends SurfaceView implements SurfaceHolder.Callback
{
private GameThread thread;
protected final static int TICK_FREQUENCY = 100;// ms, stays always the same. It's a technical constant which doesn't change
private static final String TAG = GameSurface.class.getSimpleName();
public GameSurface(Context context, AttributeSet attrs)
{
super(context, attrs);
ShapeManager.INSTANCE.init(context);
SurfaceHolder holder = getHolder();
holder.addCallback(this);
setFocusable(true); // make sure we get key events
thread = new GameThread(holder, this);
}
public void updateStatus()
{
GameProcessor.INSTANCE.updateShapes();
}
#Override
protected void onDraw(Canvas canvas)
{
for (Shape shape : GameProcessor.INSTANCE.getShapes())
{
Log.i(TAG, "getX()=" + shape.getX() + ", getY()=" + shape.getY());
canvas.drawBitmap(shape.getBitmap(), shape.getX(), shape.getY(), null);
}
}
#Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height)
{
//will never invoked since we only operate in landscape
}
#Override
public void surfaceCreated(SurfaceHolder holder)
{
// start the thread here so we don't busy-wait in run
thread.setRunning(true);
new Thread(thread).start();
}
#Override
public void surfaceDestroyed(SurfaceHolder holder)
{
Log.i(TAG, "executing surfaceDestroyed()...");
thread.setRunning(false);
}
}
GameThread
public class GameThread implements Runnable
{
private SurfaceHolder surfaceHolder;
private boolean running = false;
private GameSurface gameSurface;
private long lastTick;
public GameThread(SurfaceHolder surfaceHolder, GameSurface gameSurface)
{
this.surfaceHolder = surfaceHolder;
this.gameSurface = gameSurface;
lastTick = System.currentTimeMillis();
}
#Override
public void run()
{
Canvas canvas;
while (running)
{
canvas = null;
if (System.currentTimeMillis() > lastTick + GameSurface.TICK_FREQUENCY)
{
long timeDifference = System.currentTimeMillis() - (lastTick + GameSurface.TICK_FREQUENCY);
try
{
canvas = surfaceHolder.lockCanvas(null);
synchronized (surfaceHolder)
{
gameSurface.updateStatus();
gameSurface.draw(canvas);
}
}
finally
{
if (canvas != null)
{
surfaceHolder.unlockCanvasAndPost(canvas);
}
}
lastTick = System.currentTimeMillis() - timeDifference;
}
}
}
public void setRunning(boolean running)
{
this.running = running;
}
}
Any ideas why this code doesn't update my canvas? I can't explain it. I do not post ShapeManager and GameProcessor since they don't have anything to do with the problem (and they only load and control the current states and speed of the game).
[UPDATE]
I figured out that onDraw() is invoked before the game thread has started. That means that canvas is passed to this method before thread is using it. The interesting thing is that, after the thread has started, it always uses the same canvas, but it's not the canvas reference which is passed the very first time. Although canvas = surfaceHolder.lockCanvas(null); is assigned every tick, it's always the same reference, but it's not the original reference.
In a working example of mine, the reference is always the same, since I create the bitmaps at constructor initialization time. I can't do that in my current implementation, since I have to do calculations with values I get from onMeasure() which is invoked much later than the constructor.
I tried to somehow pass the original canvas to the thread, but the reference still changes. Meanwhile I think this is the problem, but I don't know how to solve it yet.

As often happens, I found the solution on my own.
Obviously it's really a problem that it draws to different canvas instances. I'm still not sure why this happens. Didn't have the problem before. Nevertheless, I can avoid drawing to canvas by setting setWillNotDraw(true); in my GameSurface constructor and I must not invoke gameSurface.draw(canvas) in my thread, but gameSurface.postInvalidate() instead.

Android surfaceview/canvas flickering after sleep in thread

This is my first post here. Sorry if it is not well done and thanks in advance.
There is something that is driven me crazy.
I try to draw something and show it slowly on screen. For this I use Thread.sleep as shown in the code. But it is showing two different versions of the canvas in each iteration.
Can anybody explain what is wrong?
I only want to draw something and after a few seconds draw something else. And so on.
This is my code:
public class Vista extends SurfaceView implements Callback {
private Hilo hilo;
private SurfaceHolder sf;
class Hilo extends Thread {
public boolean running;
private Vista view;
Canvas c = null;
Bitmap btm = null;
public Hilo(SurfaceHolder holder, Vista view) {
sf = holder;
btm = Bitmap.createBitmap(480, 800, Config.ARGB_8888);
c = new Canvas(btm);
}
#Override
public void run() {
Paint paint = new Paint();
paint.setStyle(Style.FILL_AND_STROKE);
paint.setColor(Color.BLUE);
int k = 0;
while (running) {
c = sf.lockCanvas();
k+= 10;
c.drawText(String.valueOf(k/10), k, 20, paint);
sf.unlockCanvasAndPost(c);
try {
sleep(1000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
public Vista(Context context) {
super(context);
SurfaceHolder holder = getHolder();
holder.addCallback(this);
hilo = new Hilo(holder, this);
}
#Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
// TODO Auto-generated method stub
}
#Override
public void surfaceCreated(SurfaceHolder holder) {
hilo.running = true;
hilo.start();
}
#Override
public void surfaceDestroyed(SurfaceHolder holder) {
boolean retry = true;
hilo.running = false;
while (retry) {
try {
hilo.join();
retry = false;
} catch (InterruptedException e) {
}
}
}
}

Your code is sleep()ing the Main thread, also known as the UI thread. This is something you absolutely mustn't do, as the main thread is responsible for handling all "stuff" in Android: UI rendering, UI event dispatch, etcetera. You can read the Processes and Threads guide on the Android Developer website for more information. Choice quote:
Thus, there are simply two rules to Android's single thread model:
Do not block the UI thread
Do not access the Android UI toolkit from outside the UI thread
The question then becomes, how do you animate a view without sleeping the main thread? The trick is to tell Android to go do something else and come back to you (redraw you) after a little while. Each time you draw, you calculate by how much to advance your animation.
One (crude) way to do this is with postInvalidateDelayed(long), which does exactly that: redraw your view after the given number of milliseconds. It's not a precise delay, it'll fluctuate a bit, so you can't assume that's exactly how much time passed. A way to do this is to store the System.nanoTime() in an instance field, so that the next time you're drawn, you can take the difference to a new reading of nanoTime(). This difference is how much time has passed and thus tells you by how much you should update your animation state. If your animation is supposed to last 1 second in total and 100 milliseconds went by since the last time you drew your scene, you must advance everything by 10%.
Android provides the a lot of utilities to make this easier.

Why is lockCanvas() slow?

I'm implementing a SurfaceView subclass, where I run a separate thread to draw onto a SurfaceHolders Canvas.
I'm measuring time before and after call to lockCanvas(), and I'm getting from about 70ms to 100ms.
Does anyone could point me why i'm getting such high timings?
Here the relevant part of the code:
public class TestView extends SurfaceView implements SurfaceHolder.Callback {
....
boolean created;
public void surfaceChanged(SurfaceHolder holder, int format, int width,
int height) {
mThread = new DrawingThread(mHolder, true);
mThread.onWindowResize(width, height);
mThread.start();
}
public void surfaceCreated(SurfaceHolder holder) {
created = true;
}
public void surfaceDestroyed(SurfaceHolder holder) {
created = false;
}
class DrawingThread extends Thread {
public void run() {
while(created) {
Canvas canvas = null;
try {
long t0 = System.currentTimeMillis();
canvas = holder.lockCanvas(null);
long t1 = System.currentTimeMillis();
Log.i(TAG, "Timing: " + ( t1 - t0) );
} finally {
holder.unlockCanvasAndPost(canvas);
}
}

You're creating a thread every time the surface is changed. You should start your thread in surfaceCreated and kill it in surfaceDestroyed. surfaceChanged is for when the dimensions of your surface changes.
From SurfaceView.surfaceCreated docs:
This is called immediately after the surface is first created. Implementations of this should start up whatever rendering code they desire. Note that only one thread can ever draw into a Surface, so you should not draw into the Surface here if your normal rendering will be in another thread.
The multiple threads are probably getting you throttled. From SurfaceHolder.lockCanvas docs:
If you call this repeatedly when the Surface is not ready (before Callback.surfaceCreated or after Callback.surfaceDestroyed), your calls will be throttled to a slow rate in order to avoid consuming CPU.
However, I'm not convinced this is the only problem. Does surfaceChanged actually get called multiple times?

This is related to how lockCanvas is actually implemented in the android graphic framework.
You should probably already know that lockCanvas will return you an free piece of memory that you will be used to draw to. By free, it means this memory has not be used for composition and not for display. Internally, simply speaking, an SurfaceView is backed up by double buffer, one is for drawing , one is for composition/display. This double buffer is managed by BufferQueque. If composition/display is slow than drawing, we have to wait until we have free buffer available.

read this:
What does lockCanvas mean (elaborate)