Develop Reference

In Android Screen Recording - How can I get each frame?

I am using the MediaRecorder and MediaProjection Api for recording the screen in android app. I am not able to get each frame that I can send over a rtmp stream. For publishing over an RTMP stream I am using JAVACV Android library.
For eg - In case of live streaming through camera, we can get each frame in onPreviewFrame() callback. After getting each frame I simply use the FFmpegFrameRecorder of JAVACV library for streaming each frame on to the rtmp url.
How can I achieve the same with screen recording?
Any help would be appreciated here. Thanks in advance!

Firstly there is no callback interface with MediaProjection which can give you buffer of screen captured frame, But it is quite possible with SurfaceTexture. Here is one implementation ScreenCapturerAndroid of screen capturing with SurfaceTexture.
An implementation of VideoCapturer to capture the screen content as a video stream.
Capturing is done by MediaProjection on a SurfaceTexture. We interact with this
SurfaceTexture using a SurfaceTextureHelper.
The SurfaceTextureHelper is created by the native code and passed to this capturer in
VideoCapturer.initialize(). On receiving a new frame, this capturer passes it
as a texture to the native code via CapturerObserver.onFrameCaptured(). This takes
place on the HandlerThread of the given SurfaceTextureHelper. When done with each frame,
the native code returns the buffer to the SurfaceTextureHelper
But if you want to send stream of your app view then following screen capturer will help you.
public class ScreenCapturer {
private boolean capturing = false;
private int fps=15;
private int width,height;
private Context context;
private int[] frame;
private Bitmap bmp;
private Canvas canvas;
private View contentView;
private Handler mHandler = new Handler();
public ScreenCapturer(Context context, View view) {
this.context = context;
this.contentView = view;
}
public void startCapturing(){
capturing = true;
mHandler.postDelayed(newFrame, 1000 / fps);
}
public void stopCapturing(){
capturing = false;
mHandler.removeCallbacks(newFrame);
}
private Runnable newFrame = new Runnable() {
#Override
public void run() {
if (capturing) {
int width = contentView.getWidth();
int height = contentView.getHeight();
if (frame == null ||
ScreenCapturer. this.width != width ||
ScreenCapturer.this.height != height) {
ScreenCapturer.this.width = width;
ScreenCapturer.this.height = height;
if (bmp != null) {
bmp.recycle();
bmp = null;
}
bmp = Bitmap.createBitmap(width,
height, Bitmap.Config.ARGB_8888);
canvas = new Canvas(bmp);
frame = new int[width * height];
}
canvas.saveLayer(0, 0, width, height, null);
canvas.translate(-contentView.getScrollX(), - contentView.getScrollY());
contentView.draw(canvas);
bmp.getPixels(frame, 0, width, 0, 0, width, height);
//frame[] is a rgb pixel array compress it to YUV if want and send over RTMP
canvas.restore();
mHandler.postDelayed(newFrame, 1000 / fps);
}
}
};
}
Usage
...
//Use this in your activity
private View parentView;
parentView = findViewById(R.id.parentView);
capturer = new ScreenCapturer(this,parentView);
//To start capturing
capturer.startCapturing();
//To Stop capturer
capturer.stopCapturing();
Using this you can send a view contents to RTMP stream, You may use parent view of your activity to capture all contents of an activity.

Providing video recording functionality with ARCore

I'm working with this sample (https://github.com/google-ar/arcore-android-sdk/tree/master/samples/hello_ar_java), and I want to provide the functionality to record a video with the AR objects placed.
I tried multiple things but to no avail, is there a recommended way to do it?

Creating a video from an OpenGL surface is a little involved, but is doable. The easiest way to understand I think is to use two EGL surfaces, one for the UI and one for the media encoder. There is a good example of the EGL level calls needed in the Grafika project on GitHub. I used that as starting point to figure out the modifications needed to the HelloAR sample for ARCore. Since there are quite a few changes, I broke it down into steps.
Make changes to support writing to external storage
To save the video, you need to write the video file somewhere accessible, so you need to get this permission.
Declare the permission in the AndroidManifest.xml file:
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
Then change CameraPermissionHelper.java to request the external storage permission as well as the camera permission. To do this, make an array of the permissions and use that when requesting the permissions and iterate over it when checking the permission state:
private static final String REQUIRED_PERMISSIONS[] = {
Manifest.permission.CAMERA,
Manifest.permission.WRITE_EXTERNAL_STORAGE
};
public static void requestCameraPermission(Activity activity) {
ActivityCompat.requestPermissions(activity, REQUIRED_PERMISSIONS,
CAMERA_PERMISSION_CODE);
}
public static boolean hasCameraPermission(Activity activity) {
for(String p : REQUIRED_PERMISSIONS) {
if (ContextCompat.checkSelfPermission(activity, p) !=
PackageManager.PERMISSION_GRANTED) {
return false;
}
}
return true;
}
public static boolean shouldShowRequestPermissionRationale(Activity activity) {
for(String p : REQUIRED_PERMISSIONS) {
if (ActivityCompat.shouldShowRequestPermissionRationale(activity, p)) {
return true;
}
}
return false;
}
Add recording to HelloARActivity
Add a simple button and text view to the UI at the bottom of activity_main.xml:
<Button
android:id="#+id/fboRecord_button"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_alignStart="#+id/surfaceview"
android:layout_alignTop="#+id/surfaceview"
android:onClick="clickToggleRecording"
android:text="#string/toggleRecordingOn"
tools:ignore="OnClick"/>
<TextView
android:id="#+id/nowRecording_text"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_alignBaseline="#+id/fboRecord_button"
android:layout_alignBottom="#+id/fboRecord_button"
android:layout_toEndOf="#+id/fboRecord_button"
android:text="" />
In HelloARActivity add member variables for recording:
private VideoRecorder mRecorder;
private android.opengl.EGLConfig mAndroidEGLConfig;
Initialize mAndroidEGLConfig in onSurfaceCreated(). We'll use this config object to create the encoder surface.
EGL10 egl10 = (EGL10)EGLContext.getEGL();
javax.microedition.khronos.egl.EGLDisplay display = egl10.eglGetCurrentDisplay();
int v[] = new int[2];
egl10.eglGetConfigAttrib(display,config, EGL10.EGL_CONFIG_ID, v);
EGLDisplay androidDisplay = EGL14.eglGetCurrentDisplay();
int attribs[] = {EGL14.EGL_CONFIG_ID, v[0], EGL14.EGL_NONE};
android.opengl.EGLConfig myConfig[] = new android.opengl.EGLConfig[1];
EGL14.eglChooseConfig(androidDisplay, attribs, 0, myConfig, 0, 1, v, 1);
this.mAndroidEGLConfig = myConfig[0];
Refactor the onDrawFrame() method so all the non-drawing code is executed first, and the actual drawing is done in a method called draw(). This way during recording, we can update the ARCore frame, process the input, then draw to the UI, and draw again to the encoder.
#Override
public void onDrawFrame(GL10 gl) {
if (mSession == null) {
return;
}
// Notify ARCore session that the view size changed so that
// the perspective matrix and
// the video background can be properly adjusted.
mDisplayRotationHelper.updateSessionIfNeeded(mSession);
try {
// Obtain the current frame from ARSession. When the
//configuration is set to
// UpdateMode.BLOCKING (it is by default), this will
// throttle the rendering to the camera framerate.
Frame frame = mSession.update();
Camera camera = frame.getCamera();
// Handle taps. Handling only one tap per frame, as taps are
// usually low frequency compared to frame rate.
MotionEvent tap = mQueuedSingleTaps.poll();
if (tap != null && camera.getTrackingState() == TrackingState.TRACKING) {
for (HitResult hit : frame.hitTest(tap)) {
// Check if any plane was hit, and if it was hit inside the plane polygon
Trackable trackable = hit.getTrackable();
if (trackable instanceof Plane
&& ((Plane) trackable).isPoseInPolygon(hit.getHitPose())) {
// Cap the number of objects created. This avoids overloading both the
// rendering system and ARCore.
if (mAnchors.size() >= 20) {
mAnchors.get(0).detach();
mAnchors.remove(0);
}
// Adding an Anchor tells ARCore that it should track this position in
// space. This anchor is created on the Plane to place the 3d model
// in the correct position relative both to the world and to the plane.
mAnchors.add(hit.createAnchor());
// Hits are sorted by depth. Consider only closest hit on a plane.
break;
}
}
}
// Get projection matrix.
float[] projmtx = new float[16];
camera.getProjectionMatrix(projmtx, 0, 0.1f, 100.0f);
// Get camera matrix and draw.
float[] viewmtx = new float[16];
camera.getViewMatrix(viewmtx, 0);
// Compute lighting from average intensity of the image.
final float lightIntensity = frame.getLightEstimate().getPixelIntensity();
// Visualize tracked points.
PointCloud pointCloud = frame.acquirePointCloud();
mPointCloud.update(pointCloud);
draw(frame,camera.getTrackingState() == TrackingState.PAUSED,
viewmtx, projmtx, camera.getDisplayOrientedPose(),lightIntensity);
if (mRecorder!= null && mRecorder.isRecording()) {
VideoRecorder.CaptureContext ctx = mRecorder.startCapture();
if (ctx != null) {
// draw again
draw(frame, camera.getTrackingState() == TrackingState.PAUSED,
viewmtx, projmtx, camera.getDisplayOrientedPose(), lightIntensity);
// restore the context
mRecorder.stopCapture(ctx, frame.getTimestamp());
}
}
// Application is responsible for releasing the point cloud resources after
// using it.
pointCloud.release();
// Check if we detected at least one plane. If so, hide the loading message.
if (mMessageSnackbar != null) {
for (Plane plane : mSession.getAllTrackables(Plane.class)) {
if (plane.getType() ==
com.google.ar.core.Plane.Type.HORIZONTAL_UPWARD_FACING
&& plane.getTrackingState() == TrackingState.TRACKING) {
hideLoadingMessage();
break;
}
}
}
} catch (Throwable t) {
// Avoid crashing the application due to unhandled exceptions.
Log.e(TAG, "Exception on the OpenGL thread", t);
}
}
private void draw(Frame frame, boolean paused,
float[] viewMatrix, float[] projectionMatrix,
Pose displayOrientedPose, float lightIntensity) {
// Clear screen to notify driver it should not load
// any pixels from previous frame.
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);
// Draw background.
mBackgroundRenderer.draw(frame);
// If not tracking, don't draw 3d objects.
if (paused) {
return;
}
mPointCloud.draw(viewMatrix, projectionMatrix);
// Visualize planes.
mPlaneRenderer.drawPlanes(
mSession.getAllTrackables(Plane.class),
displayOrientedPose, projectionMatrix);
// Visualize anchors created by touch.
float scaleFactor = 1.0f;
for (Anchor anchor : mAnchors) {
if (anchor.getTrackingState() != TrackingState.TRACKING) {
continue;
}
// Get the current pose of an Anchor in world space.
// The Anchor pose is
// updated during calls to session.update() as ARCore refines
// its estimate of the world.
anchor.getPose().toMatrix(mAnchorMatrix, 0);
// Update and draw the model and its shadow.
mVirtualObject.updateModelMatrix(mAnchorMatrix, scaleFactor);
mVirtualObjectShadow.updateModelMatrix(mAnchorMatrix, scaleFactor);
mVirtualObject.draw(viewMatrix, projectionMatrix, lightIntensity);
mVirtualObjectShadow.draw(viewMatrix, projectionMatrix, lightIntensity);
}
}
Handle the toggling of recording:
public void clickToggleRecording(View view) {
Log.d(TAG, "clickToggleRecording");
if (mRecorder == null) {
File outputFile = new File(Environment.getExternalStoragePublicDirectory(
Environment.DIRECTORY_PICTURES) + "/HelloAR",
"fbo-gl-" + Long.toHexString(System.currentTimeMillis()) + ".mp4");
File dir = outputFile.getParentFile();
if (!dir.exists()) {
dir.mkdirs();
}
try {
mRecorder = new VideoRecorder(mSurfaceView.getWidth(),
mSurfaceView.getHeight(),
VideoRecorder.DEFAULT_BITRATE, outputFile, this);
mRecorder.setEglConfig(mAndroidEGLConfig);
} catch (IOException e) {
Log.e(TAG,"Exception starting recording", e);
}
}
mRecorder.toggleRecording();
updateControls();
}
private void updateControls() {
Button toggleRelease = findViewById(R.id.fboRecord_button);
int id = (mRecorder != null && mRecorder.isRecording()) ?
R.string.toggleRecordingOff : R.string.toggleRecordingOn;
toggleRelease.setText(id);
TextView tv = findViewById(R.id.nowRecording_text);
if (id == R.string.toggleRecordingOff) {
tv.setText(getString(R.string.nowRecording));
} else {
tv.setText("");
}
}
Add a listener interface to receive video recording state changes:
#Override
public void onVideoRecorderEvent(VideoRecorder.VideoEvent videoEvent) {
Log.d(TAG, "VideoEvent: " + videoEvent);
updateControls();
if (videoEvent == VideoRecorder.VideoEvent.RecordingStopped) {
mRecorder = null;
}
}
Implement the VideoRecorder class to feed images to the encoder
The VideoRecorder class is used to feed the images to the media encoder. This class creates an off screen EGLSurface using the input surface of the media encoder. The general approach is during recording draw once for the UI display, and then make the same exact draw call for the media encoder surface.
The constructor takes recording parameters and a listener to push events to during the recording process.
public VideoRecorder(int width, int height, int bitrate, File outputFile,
VideoRecorderListener listener) throws IOException {
this.listener = listener;
mEncoderCore = new VideoEncoderCore(width, height, bitrate, outputFile);
mVideoRect = new Rect(0,0,width,height);
}
When recording starts, we need to create a new EGL surface for the encoder. Then notify the encoder that a new frame is available, make the encoder surface the current EGL surface, and return so the caller can make the drawing calls.
public CaptureContext startCapture() {
if (mVideoEncoder == null) {
return null;
}
if (mEncoderContext == null) {
mEncoderContext = new CaptureContext();
mEncoderContext.windowDisplay = EGL14.eglGetCurrentDisplay();
// Create a window surface, and attach it to the Surface we received.
int[] surfaceAttribs = {
EGL14.EGL_NONE
};
mEncoderContext.windowDrawSurface = EGL14.eglCreateWindowSurface(
mEncoderContext.windowDisplay,
mEGLConfig,mEncoderCore.getInputSurface(),
surfaceAttribs, 0);
mEncoderContext.windowReadSurface = mEncoderContext.windowDrawSurface;
}
CaptureContext displayContext = new CaptureContext();
displayContext.initialize();
// Draw for recording, swap.
mVideoEncoder.frameAvailableSoon();
// Make the input surface current
// mInputWindowSurface.makeCurrent();
EGL14.eglMakeCurrent(mEncoderContext.windowDisplay,
mEncoderContext.windowDrawSurface, mEncoderContext.windowReadSurface,
EGL14.eglGetCurrentContext());
// If we don't set the scissor rect, the glClear() we use to draw the
// light-grey background will draw outside the viewport and muck up our
// letterboxing. Might be better if we disabled the test immediately after
// the glClear(). Of course, if we were clearing the frame background to
// black it wouldn't matter.
//
// We do still need to clear the pixels outside the scissor rect, of course,
// or we'll get garbage at the edges of the recording. We can either clear
// the whole thing and accept that there will be a lot of overdraw, or we
// can issue multiple scissor/clear calls. Some GPUs may have a special
// optimization for zeroing out the color buffer.
//
// For now, be lazy and zero the whole thing. At some point we need to
// examine the performance here.
GLES20.glClearColor(0f, 0f, 0f, 1f);
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);
GLES20.glViewport(mVideoRect.left, mVideoRect.top,
mVideoRect.width(), mVideoRect.height());
GLES20.glEnable(GLES20.GL_SCISSOR_TEST);
GLES20.glScissor(mVideoRect.left, mVideoRect.top,
mVideoRect.width(), mVideoRect.height());
return displayContext;
}
When the drawing is completed, the EGLContext needs to be restored back to the UI surface:
public void stopCapture(CaptureContext oldContext, long timeStampNanos) {
if (oldContext == null) {
return;
}
GLES20.glDisable(GLES20.GL_SCISSOR_TEST);
EGLExt.eglPresentationTimeANDROID(mEncoderContext.windowDisplay,
mEncoderContext.windowDrawSurface, timeStampNanos);
EGL14.eglSwapBuffers(mEncoderContext.windowDisplay,
mEncoderContext.windowDrawSurface);
// Restore.
GLES20.glViewport(0, 0, oldContext.getWidth(), oldContext.getHeight());
EGL14.eglMakeCurrent(oldContext.windowDisplay,
oldContext.windowDrawSurface, oldContext.windowReadSurface,
EGL14.eglGetCurrentContext());
}
Add some bookkeeping methods
public boolean isRecording() {
return mRecording;
}
public void toggleRecording() {
if (isRecording()) {
stopRecording();
} else {
startRecording();
}
}
protected void startRecording() {
mRecording = true;
if (mVideoEncoder == null) {
mVideoEncoder = new TextureMovieEncoder2(mEncoderCore);
}
if (listener != null) {
listener.onVideoRecorderEvent(VideoEvent.RecordingStarted);
}
}
protected void stopRecording() {
mRecording = false;
if (mVideoEncoder != null) {
mVideoEncoder.stopRecording();
}
if (listener != null) {
listener.onVideoRecorderEvent(VideoEvent.RecordingStopped);
}
}
public void setEglConfig(EGLConfig eglConfig) {
this.mEGLConfig = eglConfig;
}
public enum VideoEvent {
RecordingStarted,
RecordingStopped
}
public interface VideoRecorderListener {
void onVideoRecorderEvent(VideoEvent videoEvent);
}
The inner class for the CaptureContext keeps track of the display and surfaces in order to easily handle multiple surfaces being used with the EGL context:
public static class CaptureContext {
EGLDisplay windowDisplay;
EGLSurface windowReadSurface;
EGLSurface windowDrawSurface;
private int mWidth;
private int mHeight;
public void initialize() {
windowDisplay = EGL14.eglGetCurrentDisplay();
windowReadSurface = EGL14.eglGetCurrentSurface(EGL14.EGL_DRAW);
windowDrawSurface = EGL14.eglGetCurrentSurface(EGL14.EGL_READ);
int v[] = new int[1];
EGL14.eglQuerySurface(windowDisplay, windowDrawSurface, EGL14.EGL_WIDTH,
v, 0);
mWidth = v[0];
v[0] = -1;
EGL14.eglQuerySurface(windowDisplay, windowDrawSurface, EGL14.EGL_HEIGHT,
v, 0);
mHeight = v[0];
}
/**
* Returns the surface's width, in pixels.
* <p>
* If this is called on a window surface, and the underlying
* surface is in the process
* of changing size, we may not see the new size right away
* (e.g. in the "surfaceChanged"
* callback). The size should match after the next buffer swap.
*/
public int getWidth() {
if (mWidth < 0) {
int v[] = new int[1];
EGL14.eglQuerySurface(windowDisplay,
windowDrawSurface, EGL14.EGL_WIDTH, v, 0);
mWidth = v[0];
}
return mWidth;
}
/**
* Returns the surface's height, in pixels.
*/
public int getHeight() {
if (mHeight < 0) {
int v[] = new int[1];
EGL14.eglQuerySurface(windowDisplay, windowDrawSurface,
EGL14.EGL_HEIGHT, v, 0);
mHeight = v[0];
}
return mHeight;
}
}
Add VideoEncoder classes
The VideoEncoderCore class is copied from Grafika, as well as the TextureMovieEncoder2 class.

Draw text or image on the camera stream (GLSL)

I have a live broadcasting app based off grafika's examples, where I send my video feed over RTMP to be live broadcast.
I now want to watermark my video by overlaying text or a logo on my video stream. I know this can be done with GLSL filtering, but I have no idea how to implement this based on the sample that I linked.
I tried using Alpha blending but it seems the two texture formats are somehow incompatible (one being TEXTURE_EXTERNAL_OES and the other one TEXTURE_2D) and I just get a black frame in return.
EDIT:
I based my code on Kickflip API:
class CameraSurfaceRenderer implements GLSurfaceView.Renderer {
private static final String TAG = "CameraSurfaceRenderer";
private static final boolean VERBOSE = false;
private CameraEncoder mCameraEncoder;
private FullFrameRect mFullScreenCamera;
private FullFrameRect mFullScreenOverlay; // For texture overlay
private final float[] mSTMatrix = new float[16];
private int mOverlayTextureId;
private int mCameraTextureId;
private boolean mRecordingEnabled;
private int mFrameCount;
// Keep track of selected filters + relevant state
private boolean mIncomingSizeUpdated;
private int mIncomingWidth;
private int mIncomingHeight;
private int mCurrentFilter;
private int mNewFilter;
boolean showBox = false;
/**
* Constructs CameraSurfaceRenderer.
* <p>
* #param recorder video encoder object
*/
public CameraSurfaceRenderer(CameraEncoder recorder) {
mCameraEncoder = recorder;
mCameraTextureId = -1;
mFrameCount = -1;
SessionConfig config = recorder.getConfig();
mIncomingWidth = config.getVideoWidth();
mIncomingHeight = config.getVideoHeight();
mIncomingSizeUpdated = true; // Force texture size update on next onDrawFrame
mCurrentFilter = -1;
mNewFilter = Filters.FILTER_NONE;
mRecordingEnabled = false;
}
/**
* Notifies the renderer that we want to stop or start recording.
*/
public void changeRecordingState(boolean isRecording) {
Log.d(TAG, "changeRecordingState: was " + mRecordingEnabled + " now " + isRecording);
mRecordingEnabled = isRecording;
}
#Override
public void onSurfaceCreated(GL10 unused, EGLConfig config) {
Log.d(TAG, "onSurfaceCreated");
// Set up the texture blitter that will be used for on-screen display. This
// is *not* applied to the recording, because that uses a separate shader.
mFullScreenCamera = new FullFrameRect(
new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_EXT));
// For texture overlay:
GLES20.glEnable(GLES20.GL_BLEND);
GLES20.glBlendFunc(GLES20.GL_SRC_ALPHA, GLES20.GL_ONE_MINUS_SRC_ALPHA);
mFullScreenOverlay = new FullFrameRect(
new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_2D));
mOverlayTextureId = GlUtil.createTextureWithTextContent("hello!");
mOverlayTextureId = GlUtil.createTextureFromImage(mCameraView.getContext(), R.drawable.red_dot);
mCameraTextureId = mFullScreenCamera.createTextureObject();
mCameraEncoder.onSurfaceCreated(mCameraTextureId);
mFrameCount = 0;
}
#Override
public void onSurfaceChanged(GL10 unused, int width, int height) {
Log.d(TAG, "onSurfaceChanged " + width + "x" + height);
}
#Override
public void onDrawFrame(GL10 unused) {
if (VERBOSE){
if(mFrameCount % 30 == 0){
Log.d(TAG, "onDrawFrame tex=" + mCameraTextureId);
mCameraEncoder.logSavedEglState();
}
}
if (mCurrentFilter != mNewFilter) {
Filters.updateFilter(mFullScreenCamera, mNewFilter);
mCurrentFilter = mNewFilter;
mIncomingSizeUpdated = true;
}
if (mIncomingSizeUpdated) {
mFullScreenCamera.getProgram().setTexSize(mIncomingWidth, mIncomingHeight);
mFullScreenOverlay.getProgram().setTexSize(mIncomingWidth, mIncomingHeight);
mIncomingSizeUpdated = false;
Log.i(TAG, "setTexSize on display Texture");
}
// Draw the video frame.
if(mCameraEncoder.isSurfaceTextureReadyForDisplay()){
mCameraEncoder.getSurfaceTextureForDisplay().updateTexImage();
mCameraEncoder.getSurfaceTextureForDisplay().getTransformMatrix(mSTMatrix);
//Drawing texture overlay:
mFullScreenOverlay.drawFrame(mOverlayTextureId, mSTMatrix);
mFullScreenCamera.drawFrame(mCameraTextureId, mSTMatrix);
}
mFrameCount++;
}
public void signalVertialVideo(FullFrameRect.SCREEN_ROTATION isVertical) {
if (mFullScreenCamera != null) mFullScreenCamera.adjustForVerticalVideo(isVertical, false);
}
/**
* Changes the filter that we're applying to the camera preview.
*/
public void changeFilterMode(int filter) {
mNewFilter = filter;
}
public void handleTouchEvent(MotionEvent ev){
mFullScreenCamera.handleTouchEvent(ev);
}
}
This is the code for Rendering the image on the screen (GLSurfaceView), but this is not actually overlayed over the video. If I am not mistaken, this is done on CameraEncoder.
Thing is, replicating the code from CameraSurfaceRenderer into CameraEncoder (they both have similar code when it comes to filters) does not provide an overlayed text/image.

The texture object uses the GL_TEXTURE_EXTERNAL_OES texture target, which is defined by the GL_OES_EGL_image_external OpenGL ES extension. This limits how the texture may be used. Each time the texture is bound it must be bound to the GL_TEXTURE_EXTERNAL_OES target rather than the GL_TEXTURE_2D target. Additionally, any OpenGL ES 2.0 shader that samples from the texture must declare its use of this extension using, for example, an "#extension GL_OES_EGL_image_external : require" directive. Such shaders must also access the texture using the samplerExternalOES GLSL sampler type.
https://developer.android.com/reference/android/graphics/SurfaceTexture.html
Post your code that you used to do alpha blending and I can probably fix it.
I would probably override the Texture2dProgram and pass that to the FullFrame Renderer. It has example code for rendering using the GL_TEXTURE_EXTERNAL_OES extension. Basically, #Override the draw function, call the base implementation, bind your watermark and draw.
That should be between camera and the video encoder.

Playing Video with OpenGL and MediaCodec

I'm trying to play the same video at the same time in two different textureviews. I've used code from grafika (MoviePlayer and ContinuousCaptureActivity) to try to get it to work (thanks fadden). To make the problem simpler, I'm trying to do it with just one TextureView first.
At the moment I've created a TextureView, and once it get a SurfaceTexture, I create a WindowSurface and make it current. Then I generate a TextureID generated using a FullFrameRect object.
#Override
public void onSurfaceTextureAvailable(SurfaceTexture surface, int
width, int height) {
mSurfaceTexture = surface;
mEGLCore = new EglCore(null, EglCore.FLAG_TRY_GLES3);
Log.d("EglCore", "EGL core made");
mDisplaySurface = new WindowSurface(mEGLCore, mSurfaceTexture);
mDisplaySurface.makeCurrent();
Log.d("DisplaySurface", "mDisplaySurface made");
mFullFrameBlit = new FullFrameRect(new Texture2dProgram(Texture2dProgram.ProgramType.TEXTURE_EXT));
mTextureID = mFullFrameBlit.createTextureObject();
//mSurfaceTexture.attachToGLContext(mTextureID);
clickPlayStop(null);
}
Then I get an off-screen SurfaceTexture, link it with the TextureID that I got above and create a surface to pass to a MoviePlayer thus:
public void clickPlayStop(#SuppressWarnings("unused") View unused) {
if (mShowStopLabel) {
Log.d(TAG, "stopping movie");
stopPlayback();
// Don't update the controls here -- let the task thread do it after the movie has
// actually stopped.
//mShowStopLabel = false;
//updateControls();
} else {
if (mPlayTask != null) {
Log.w(TAG, "movie already playing");
return;
}
Log.d(TAG, "starting movie");
SpeedControlCallback callback = new SpeedControlCallback();
callback.setFixedPlaybackRate(24);
MoviePlayer player = null;
MovieTexture = new SurfaceTexture(mTextureID);
MovieTexture.setOnFrameAvailableListener(this);
Surface surface = new Surface(MovieTexture);
try {
player = new MoviePlayer(surface, callback, this);//TODO
} catch (IOException ioe) {
Log.e(TAG, "Unable to play movie", ioe);
return;
}
adjustAspectRatio(player.getVideoWidth(), player.getVideoHeight());
mPlayTask = new MoviePlayer.PlayTask(player, this);
mPlayTask.setLoopMode(true);
mShowStopLabel = true;
mPlayTask.execute();
}
}
The idea is that the SurfaceTexture gets a raw frame which I can use as an OES_external texture to sample from with OpenGL. Then I can call DrawFrame() from my EGLContext after setting my WindowSurface as current.
private void drawFrame() {
Log.d(TAG, "drawFrame");
if (mEGLCore == null) {
Log.d(TAG, "Skipping drawFrame after shutdown");
return;
}
// Latch the next frame from the camera.
mDisplaySurface.makeCurrent();
MovieTexture.updateTexImage();
MovieTexture.getTransformMatrix(mTransformMatrix);
// Fill the WindowSurface with it.
int viewWidth = mTextureView.getWidth();
int viewHeight = mTextureView.getHeight();
GLES20.glViewport(0, 0, viewWidth, viewHeight);
mFullFrameBlit.drawFrame(mTextureID, mTransformMatrix);
mDisplaySurface.swapBuffers();
}
If I wanted to do it with 2 TextureViews, the idea would be to call makeCurrent() and draw into each buffer for each view, then call swapBuffers() after the drawing is done.
This is what I want to do, but I am pretty sure this is not what my code is actually doing. Could somebody help me understand what I need to change to make it work?
#Fadden
Update: This is interesting. I changed the code in onSurfaceTextureAvailable to this:
#Override
public void onSurfaceTextureAvailable(SurfaceTexture surface, int
width, int height) {
mSurfaceTexture = surface;
TextureHeight = height;
TextureWidth = width;
//mEGLCore = new EglCore(null, EglCore.FLAG_TRY_GLES3);
Log.d("EglCore", "EGL core made");
//mDisplaySurface = new WindowSurface(mEGLCore, mSurfaceTexture);
//mDisplaySurface.makeCurrent();
Log.d("DisplaySurface", "mDisplaySurface made");
//mFullFrameBlit = new FullFrameRect(new Texture2dProgram(Texture2dProgram.ProgramType.OPENGL_TEST));
//mTextureID = mFullFrameBlit.createTextureObject();
//clickPlayStop(null);
// Fill the SurfaceView with it.
//int viewWidth = width;
//int viewHeight = height;
//GLES20.glViewport(0, 0, viewWidth, viewHeight);
//mFullFrameBlit.drawFrame(mTextureID, mTransformMatrix);
//mFullFrameBlit.openGLTest();
//mFullFrameBlit.testDraw(mDisplaySurface.getHeight(),mDisplaySurface.getWidth());
//mDisplaySurface.swapBuffers();
}
So, it shouldn't call anything else, just show the empty TextureView - and this is what I see...

Thanks to Fadden for the help.
So there seemed to be some unknown issue that was resolved when I used a new thread to decode and produce the frames. I haven't found out what caused the original problem, but I have found a way around it.

setPreviewDisplay and setDisplayOrientation

I'm puzzled by OpenCV's Android camera sample code. They make a custom class which implements SurfaceHolder.Callback and put the following line inside the method surfaceChanged:
mCamera.setPreviewDisplay(null);
The Android documentation for setPreviewDisplay explains:
This method must be called before startPreview(). The one exception is
that if the preview surface is not set (or set to null) before
startPreview() is called, then this method may be called once with a
non-null parameter to set the preview surface. (This allows camera
setup and surface creation to happen in parallel, saving time.) The
preview surface may not otherwise change while preview is running.
Unusually, OpenCV's code never calls setPreviewDisplay with a non-null SurfaceHolder. It works fine, but changing the rotation of the image using setDisplayOrientation doesn't work. This line also doesn't appear to do anything, since I get the same results without it.
If I call setPreviewDisplay with the SurfaceHolder supplied to surfaceChanged instead of null, the image rotates but does not include the results of the image processing. I also get an IllegalArgumentException when calling lockCanvas later on.
What's going on?
Here are the (possibly) most relevant parts of their code, slightly simplified and with methods inlined. Here is the full version.
Class definition
public abstract class SampleViewBase extends SurfaceView
implements SurfaceHolder.Callback, Runnable {
When the camera is opened
mCamera.setPreviewCallbackWithBuffer(new PreviewCallback() {
public void onPreviewFrame(byte[] data, Camera camera) {
synchronized (SampleViewBase.this) {
System.arraycopy(data, 0, mFrame, 0, data.length);
SampleViewBase.this.notify();
}
camera.addCallbackBuffer(mBuffer);
}
});
When the surface changes
/* Now allocate the buffer */
mBuffer = new byte[size];
/* The buffer where the current frame will be copied */
mFrame = new byte [size];
mCamera.addCallbackBuffer(mBuffer);
try {
mCamera.setPreviewDisplay(null);
} catch (IOException e) {
Log.e(TAG, "mCamera.setPreviewDisplay/setPreviewTexture fails: " + e);
}
[...]
/* Now we can start a preview */
mCamera.startPreview();
The run method
public void run() {
mThreadRun = true;
Log.i(TAG, "Starting processing thread");
while (mThreadRun) {
Bitmap bmp = null;
synchronized (this) {
try {
this.wait();
bmp = processFrame(mFrame);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
if (bmp != null) {
Canvas canvas = mHolder.lockCanvas();
if (canvas != null) {
canvas.drawBitmap(bmp, (canvas.getWidth() - getFrameWidth()) / 2,
(canvas.getHeight() - getFrameHeight()) / 2, null);
mHolder.unlockCanvasAndPost(canvas);
}
}
}
Log.i(TAG, "Finishing processing thread");
}

I ran into this same problem. Instead of using a SurfaceView.Callback, I subclassed their class JavaCameraView. See my live face detection and drawing sample here. It was then trivial to rotate the matrix coming out of the camera according to the device's orientation, prior to processing. Relevant excerpt of linked code:
#Override
public Mat onCameraFrame(Mat inputFrame) {
int flipFlags = 1;
if(display.getRotation() == Surface.ROTATION_270) {
flipFlags = -1;
Log.i(VIEW_LOG_TAG, "Orientation is" + getRotation());
}
Core.flip(inputFrame, mRgba, flipFlags);
inputFrame.release();
Imgproc.cvtColor(mRgba, mGray, Imgproc.COLOR_RGBA2GRAY);
if (mAbsoluteFaceSize == 0) {
int height = mGray.rows();
if (Math.round(height * mRelativeFaceSize) > 0) {
mAbsoluteFaceSize = Math.round(height * mRelativeFaceSize);
}
}
}

I solved the rotation issue using OpenCV itself: after finding out how much the screen rotation needs to be corrected using this code, I apply a rotation matrix to the raw camera image (after converting from YUV to RGB):
Point center = new Point(mFrameWidth/2, mFrameHeight/2);
Mat rotationMatrix = Imgproc.getRotationMatrix2D(center, totalRotation, 1);
[...]
Imgproc.cvtColor(mYuv, mIntermediate, Imgproc.COLOR_YUV420sp2RGBA, 4);
Imgproc.warpAffine(mIntermediate, mRgba, rotationMatrix,
new Size(mFrameHeight, mFrameWidth));
A separate issue is that setPreviewDisplay(null) gives a blank screen on some phones. The solution, which I got from here and draws on this bugreport and this SO question, passes a hidden, "fake" SurfaceView to the preview display to get it to start, but actually displays the output on an overlaid custom view, which I call CameraView. So, after calling setContentView() in the activity's onCreate(), stick in this code:
if (VERSION.SDK_INT < VERSION_CODES.HONEYCOMB) {
final SurfaceView fakeView = new SurfaceView(this);
fakeView.setLayoutParams(new LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.MATCH_PARENT));
fakeView.setZOrderMediaOverlay(false);
final CameraView cameraView = (CameraView) this.findViewById(R.id.cameraview);
cameraView.setZOrderMediaOverlay(true);
cameraView.fakeView = fakeView;
}
Then, when setting the preview display, use this code:
try {
if (VERSION.SDK_INT >= VERSION_CODES.HONEYCOMB)
mCamera.setPreviewTexture(new SurfaceTexture(10));
else
mCamera.setPreviewDisplay(fakeView.getHolder());
} catch (IOException e) {
Log.e(TAG, "mCamera.setPreviewDisplay fails: "+ e);
}
If you are only developing for Honeycomb and above, just replace setPreviewDisplay(null) with mCamera.setPreviewTexture(new SurfaceTexture(10)); and be done with it. setDisplayOrientation() still doesn't work if you do this, though, so you'll still have to use the rotation matrix solution.