I am referring to the demo app Grafika, in which the CameraCaptureActivity records a video while showing a live preview of the effects applied.
While recording in the CameraCaptureActivity, any effect that is applied to the frame that comes from the camera is done twice.
Once for the preview and once while saving the video to the file.
Since the same frame that is previewed is being saved to the file, it would save a lot of processing if this could be done just once.
The rendering of the frames happens directly on the two surfaces, one being the GLSurfaceView (for preview) and the other being MediaCodec (saving part).
Is there a way to render the OpenGL effect only once?
If I could copy the contents of one surface to the other it would be great.
Is there a way to do this?
Yes: you can render to an FBO, then blit the output twice, once for display and once for recording.
Grafika's "record GL app" activity demonstrates three different approaches to the problem (one of which only works on GLES 3.0+). The doFrame() method does the work; the draw-once-blit-twice approach is currently here.
Related
To play a media using Android MediaPlayer or MediaCodec, most of the time, you use SurfaceView or GLSurfaceView (There is another way to achieve this using TextureView, but let's not talk about it here, since it's a bit different type of view)
And as far as I know, capturing the video frame from SurfaceView is not possible - you don't have access to hw overlay.
How about GLSurfaceView? Since we have access to YUV pixels (we're, right?), is it possible?
Can anyone point me where i can find a sample code to do it?
I don't think below explanation can work, because it's assuming the color format is RGBA, and in above case, I think it's YUV.
When using GLES20.glReadPixels on android, the data returned by it is not exactly the same with the living preview
Thank you and have a great day.
You are correct in that you cannot read back from a Surface. It's the producer side of a producer-consumer pair. GLSurfaceView is just a bunch of code wrapped around a SurfaceView that (in theory) makes working with GLES easier.
So you have to send the preview somewhere else. One approach is to send it to a SurfaceTexture, which converts every frame sent to its Surface into a GLES texture. The texture can then be rendered twice, once for display and once to an offscreen pbuffer that can be saved as a bitmap (just like this question).
I'm not sure why you don't want to talk about TextureView. It's a View that uses SurfaceTexture under the hood, and it provides a getBitmap() call that does exactly what you want.
I was able to decode an mp4 video. If I configure the decoder using a Surface I can see the video on screen. Now, I want to edit the frame (adding a yellow line or even better overlapping a tiny image) and encode the video as a new video. It is not necessary to show the video and I don't care now about the performance.(If I show the frames while editing I could have a gap if the editing function takes a lot of time), So, What do you recommend to me, configure the decoder with a GlSurface anyway and use OpenGl (GLES), or configure it with null and somehow convert the Bytebuffer to a Bitmap, modify it, and encode the bitmap as a byte array? Also I saw in Grafika page that you cand use a Surface with a custom Rederer and use OpenGl (GLES). Thanks
You will have to use OpenGLES. ByteBuffer/Bitmap approach can not give realistic performance/features.
Now that you've been able to decode the Video (using MediaExtractor and Codec) to a Surface, you need to use the SurfaceTexture used to create the Surface as an External Texture and render using GLES to another Surface retrieved from MediaCodec configured as an encoder.
Though Grafika doesn't have an exactly similar complete project, you can start with your existing project and then try to use either of the following subprojects in grafika Continuous Camera or Show + capture camera, which currently renders Camera frames (fed to SurfaceTexture) to a Video (and display).
So essentially, the only change is the MediaCodec feeding frames to SurfaceTexture instead of the Camera.
Google CTS DecodeEditEncodeTest does exactly the same and can be used as a reference in order to make the learning curve smoother.
Using this approach, you can certainly do all sorts of things like manipulating the playback speed of video (fast forward and slow-down), adding all sorts of overlays on the scene, play with colors/pixels in the video using shaders etc.
Checkout filters in Show + capture camera for an illustration for the same.
Decode-edit-Encode flow
When using OpenGLES, 'editing' of the frame happens via rendering using GLES to the Encoder's input surface.
If decoding and rendering+encoding are separated out in different threads, you're bound to skip a few frames every frame, unless you implement some sort of synchronisation between the two threads to keep the decoder waiting until the render+encode for that frame has happened on the other thread.
Although modern hardware codecs support simultaneous video encoding and decoding, I'd suggest, do the decoding, rendering and encoding in the same thread, especially in your case, when the performance is not a major concern right now. That will help avoiding the problems of having to handle synchronisation on your own and/or frame jumps.
This is my scenario: I am trying to take a picture from the front camera when someone puts in the incorrect password in the lock screen. Basically, I need to be able to take a picture out of the front cam without a preview.
After much googling, I figured out that the way to do it is opengl and SurfaceTexture. You direct the camera preview to a SurfaceTexture, and later extract the picture from this texture somehow. I found this out from the following resources:
https://stackoverflow.com/a/10776349/902572 (suggestion 1)
http://www.freelancer.com/projects/Android-opengl/Android-OpenGL-App-Access-Raw.html, which is the same as (1)
https://groups.google.com/forum/#!topic/android-developers/U5RXFGpAHPE (See Romain's post on 12/22/11)
I understand what is to be done, but i have been unable to correctly put them into code, as I am new to opengl.
The CameraToMpegTest example has most of what you need, though it goes well beyond your use case (it's recording a series of preview frames as a video).
The rest of what you need is in ExtractMpegFramesTest. In particular, you want to render to an offscreen pbuffer (rather than a MediaCodec encoder input surface), and you can save the pbuffer contents as a PNG file using saveFrame().
The above are written as small "headless" test cases. You can see similar code in a full app in Grafika.
I am trying to decode video samples using MediaCodec API. I am using surfaceView to show rendered samples. If i press home button, app going into pause state and surface destroyed. When i coming back to resume state, new surfaceView reference is creating, but decoder is unable to pump samples on surfaceView. so screen appearing as black.
video configure:
videoDecoder.configure(format, surface, null, 0);
So how can i reconfigure videoDecoder in above statement. It is similar to the following problem
How to keep decoding alive during screen orientation?
The MediaCodec API does not currently (API 19) provide a way to replace the output Surface.
As in the other question you refer to, I think the way to deal with this will be to decode to a Surface that isn't tied to the view hierarchy (and, hence, doesn't get torn down when the Activity is destroyed).
If you direct the output of the MediaCodec to a SurfaceTexture, you can then render that texture onto the SurfaceView. This will require a bit of GLES code. You can find the necessary pieces in the Grafika sources, but there isn't currently a full implementation of what you want (e.g. PlayMovieActivity decodes video to a SurfaceTexture, but that ST is part of a TextureView, which will get torn down).
The additional rendering step will increase the GPU load, and won't work for DRM-protected video. For most devices and apps this won't matter.
See also the bigflake examples.
Update: I've added this to Grafika, with a twist. See the "Double decode" example. The output goes to a SurfaceTexture associated with a TextureView. If the screen is rotated (or, currently, blanked by hitting the power button), decoding continues. If you leave the activity with the "back" or "home" button, decoding stops. It works by retaining the SurfaceTexture, attaching it to the new TextureView.
Ideally I'd like to accomplish two goals:
Pass the Camera preview data to a MediaCodec encoder via a Surface. I can create the Surface using MediaCodec.createInputSurface() but the Camera.setPreviewDisplay() takes a SurfaceHolder, not a Surface.
In addition to passing the Camera preview data to the encoder, I'd also like to display the preview on-screen (so the user can actually see what they are encoding). If the encoder wasn't involved then I'd use a SurfaceView, but that doesn't appear to work in this scenario since SurfaceView creates its own Surface and I think I need to use the one created by MediaCodec.
I've searched online quite a bit for a solution and haven't found one. Some examples on bigflake.com seem like a step in the right direction but they take an approach that adds a bunch of EGL/SurfaceTexture overhead that I'd like to avoid. I'm hoping there is a simpler example or solution where I can get the Camera and MediaCodec talking more directly without involving EGL or textures.
As of Android 4.3 (API 18), the bigflake CameraToMpegTest approach is the correct way.
The EGL/SurfaceTexture overhead is currently unavoidable, especially for what you want to do in goal #2. The idea is:
Configure the Camera to send the output to a SurfaceTexture. This makes the Camera output available to GLES as an "external texture".
Render the SurfaceTexture to the Surface returned by MediaCodec#createInputSurface(). That feeds the video encoder.
Render the SurfaceTexture a second time, to a GLSurfaceView. That puts it on the display for real-time preview.
The only data copying that happens is performed by the GLES driver, so you're doing hardware-accelerated blits, which will be fast.
The only tricky bit is you want the external texture to be available to two different EGL contexts (one for the MediaCodec, one for the GLSurfaceView). You can see an example of creating a shared context in the "Android Breakout game recorder patch" sample on bigflake -- it renders the game twice, once to the screen, once to a MediaCodec encoder.
Update: This is implemented in Grafika ("Show + capture camera").
Update: The multi-context approach in "show + capture camera" approach is somewhat flawed. The "continuous capture" Activity uses a plain SurfaceView, and is able to do both screen rendering and video recording with a single EGL context. This is recommended.