In a NativeScript application, I'm trying to apply a real-time effect to an applied video (specifically, grayscale video playback). I am currently playing the video in app using a fork of the nativescript-videoplayer nativescript plugin. The relevant source code that creates the video view in android is here:
https://github.com/nstudio/nativescript-videoplayer/blob/master/src/videoplayer.android.ts
In essence, I want to modify it to apply an effect / shader, similar to the answer given here:
https://stackoverflow.com/a/31958741/192694
However, I'm not sure where this shader code would hook up to my existing stream of MediaPlayer creation and setting its SurfaceTexture surface.
The original video player plugin uses a TextureView on MediaPlayer instance, instead you may use the GLSurfaceView as showcased in the other SO example #Bill linked.
Related
I have an encoded video stream that I'm playing through exoplayer. What I want to do is get each frame of the video and edit it before it is displayed (e.g. changing some pixels).
Is it possible to do this with exoplayer? I've been looking at the implementation of MediaCodecVideoRenderer.java in the exoplayer source, but it seems that each MediaCodec releases its output buffer to a surface itself, without possibility of editing the frame before rendering.
It will depend on exactly what you want to modify, but it is possible to use a GLSurface view and listen for each frame and then transform the frame, assuming it is not encrypted (with encrypted you van usually still apply transformation bit you definitely should not be able to read the frame itself).
There is a good example project which does something similar to apply filters to videos, extending ExoPlayer - take a look at the EPlayerRenderer class in particular.
https://github.com/MasayukiSuda/ExoPlayerFilter
You can also do a similar thing with openCV - read in a frame, modify it and then display it. This may be easier if you are doing compilacted image manipulations.
I am struggling with recording and editing videos. For editing, I found a useful library in Android named ffmpeg4Android. However, I am still get stuck in recording video. Here is what I want:
1/ Add text or images in video while recording
2/ Add filter while recording video
I found there is a library GPUImage for Android but it has only some examples related to taking a photo, not to recording video)
Please let me know if you have any ways or any suggested libraries which can do it.
Thank you in advance!
Links which I read when researching:
Add overlay while record video on Android
How to Record video with GPUImage?
FFMpeg add text to actual video file after recording in Android
You can add an overlay image on video using a blend filter.
You cannot add text directly but you can add the text to the image writing on a canvas.
android-gpuimage library does not natively support video recording but you can try using the android-gpuimage-videorecording library. It is a fork of the gpu-image for android that provides also the video recording functionality
android-gpuimage-videorecording
see the GPUImageMovieWriter class
It should point you in the right direction for developing your own video writer on top of GPUImage.
The idea is to:
draw on current screen surface
switch to encoder input surface and draw previous frame buffer again on it
switch back to screen surface
other useful links: EGL surface helper, Media encoder
I want to record video by camera in my Android device. I need to add overlay image over recorded movie. In iOS I would use GPUImage. In Android I found Android GPUImage. I tried to use it but I didn't found any way to add any filter while video recording. In provided example I could add filters only for taking photos. Is there any ways to record video with filters with Android GPUImage? Is there any other ways to add images overlay over recording video in realtime? If not, is there any ways to add images overlay over recorded video in postprocessing?
You can add an overlay image on video using a blend filter.
About video recording: android-gpuimage library does not support it but you can try using the android-gpuimage-videorecording library. It is a fork of the gpu-image for android that provides also the video recording functionality
android-gpuimage-videorecording
see the GPUImageMovieWriter class
It should point you in the right direction for developing your own video writer on top of GPUImage.
The idea is to:
draw on current screen surface
switch to encoder input surface and draw previous frame buffer again on it
switch back to screen surface
other useful links: EGL surface helper, Media encoder
This project (MagicCamera) has many multiple input filters. You can write your own fragment shader to overlay an image on camera texture (similar to MagicN1977Filter). It also includes video recording.
I want to use GPUImage Android to process video in real time. I see example that creating pictures with different filters but I didnt find any example of recording video with filters. Is this possible with GPUImage Android?
android-gpuimage library does not support video recording but you can try using the android-gpuimage-videorecording library. It is a fork of the gpu-image for android that provides also the video recording functionality
android-gpuimage-videorecording
see the GPUImageMovieWriter class
It should point you in the right direction for developing your own video writer on top of GPUImage.
The idea is to:
draw on current screen surface
switch to encoder input surface and draw previous frame buffer again on it
switch back to screen surface
other useful links:
EGL surface helper,
Media encoder
GPUVideo-android
This library apply video filter on generate an Mp4 and on ExoPlayer video and Video Recording with Camera2.
Android MediaCodec API is used this library.
This library has many types of filters.
Filters link
how to play video using opengl es in android?
Your question is a bit vague. If all you want to do is to play a video in a GL surface, it's possible. See the Media Player sample code that Dave was referring to in his sample. All you have to do there is to replace the SurfaceView with a GLSurfaceView in both the MediaPlayerDemo_Video.java file as well as in the corresponding layout file (mediaplayer_2.xml).
Also you need to create a custom Renderer class (one that implements the GLSurfaceView.Renderer interface) and set it to your GLSurfaceView.
After you do all this, you will have your video playing on a GL surface, but that would be all.
If, on the other hand, you want to manipulate the video, i.e. to draw the video frames in a GL texture and add effects to it or transform it (for example scale, rotate, etc.), then I'm afraid this can't be done. The reason is that you don't have access to the raw video frames in your application.
I'm not sure why you're metioning OpenGL here, but probably the easiest way to play video is using the VideoView class. You'll want to have a look at the MediaPlayer class too.
You might find the Audio and Video page in the Android Developer Documentation helpful.
There's VideoView sample code and MediaPlayer sample code in the API demos provided with the Android SDK.