Is there an API call or library to embed graphic into Android video during recording. I want to imprint a timestamp directly onto the video. I have seen discussions of intercepting the bit stream but I am concerned about a home-grown solution like this having compatibility and performance issues. Currently I add it in post-processing using ffmpeg which has issues with file timestamps.
I have some ideas for you:
Record using GLSurfaceView(check Rotoscope for example) or TextureView
Use GLES20.glReadPixels() for GLSurfaceView or getBitmap() for TextureView to get bitmap data
Create Canvas from bitmap then draw anything you want
Have fun!
Related
Assuming we have a Surface in Android that displays a video (e.g. h264) with a MediaPlayer:
1) Is it possible to change the displayed saturation, contrast & brightness of the displayed on the surface video? In real time? E.g. Images can use setColorFilter is there anything similar in Android to process the video frames?
Alternative question (if no. 1 is too difficult):
2) If we would like to export this video with e.g. an increased saturation, we should use a Codec, e.g. MediaCodec. What technology (method, class, library, etc...) should we use before the codec/save action to apply the saturation change?
For display only, one easy approach is to use a GLSurfaceView, a SurfaceTexture to render the video frames, and a MediaPlayer. Prokash's answer links to an open source library that shows how to accomplish that. There are a number of other examples around if you search those terms together. Taking that route, you draw video frames to an OpenGL texture and create OpenGL shaders to manipulate how the texture is rendered. (I would suggest asking Prokash for further details and accepting his answer if this is enough to fill your requirements.)
Similarly, you could use the OpenGL tools with MediaCodec and MediaExtractor to decode video frames. The MediaCodec would be configured to output to a SurfaceTexture, so you would not need to do much more than code some boilerplate to get the output buffers rendered. The filtering process would be the same as with a MediaPlayer. There are a number of examples using MediaCodec as a decoder available, e.g. here and here. It should be fairly straightforward to substitute the TextureView or SurfaceView used in those examples with the GLSurfaceView of Prokash's example.
The advantage of this approach is that you have access to all the separate tracks in the media file. Because of that, you should be able to filter the video track with OpenGL and straight copy other tracks for export. You would use a MediaCodec in encode mode with the Surface from the GLSurfaceView as input and a MediaMuxer to put it all back together. You can see several relevant examples at BigFlake.
You can use a MediaCodec without a Surface to access decoded byte data directly and manipulate it that way. This example illustrates that approach. You can manipulate the data and send it to an encoder for export or render it as you see fit. There is some extra complexity in dealing with the raw byte data. Note that I like this example because it illustrates dealing with the audio and video tracks separately.
You can also use FFMpeg, either in native code or via one of the Java wrappers out there. This option is more geared towards export than immediate playback. See here or here for some libraries that attempt to make FFMpeg available to Java. They are basically wrappers around the command line interface. You would need to do some extra work to manage playback via FFMpeg, but it is definitely doable.
If you have questions, feel free to ask, and I will try to expound upon whatever option makes the most sense for your use case.
If you are using a player that support video filters then you can do that.
Example of such a player is VLC, which is built around FFMPEG [1].
VLC is pretty easy to compile for Android. Then all you need is the libvlc (aar file) and you can build your own app. See compile instructions here.
You will also need to write your own module. Just duplicate an existing one and modify it. Needless to say that VLC offers strong transcoding and streaming capabilities.
As powerful VLC for Android is, it has one huge drawback - video filters cannot work with hardware decoding (Android only). This means that the entire video processing is on the CPU.
Your other options are to use GLSL / OpenGL over surfaces like GLSurfaceView and TextureView. This guaranty GPU power.
Is there any existing support to Tensorflow on Android for locally saved videos? The demo provided is tightly coupled with the camera, and porting it to work for videos will be non-trivial and time-consuming, at the very least. The task it is intended for, is to process raw frames from a stream being broadcast live.
You have to take bitmaps for frame in video using MediaMetadataRetriever or something appropriate and then pass them to the tensorflow library for image recognition.
Currently there is no existing support for "video" stream itself in tensorflow AFAIK, even the demo takes screenshots of the camera preview to recognize.
If you really want to recognize video stream itself then you have to build a model of your own.
Otherwise, Process to analyze video would be as follows assuming you already have your graph and label file and playing video is not needed ( If you want to show video during the analysis, then you should implement surfaceview or textureview in your activity):
Initialize tensorflow & load desired video using MediaMetadataRetriever
extract bitmaps for desired frames using getFrameAtTime & scale the bitmap to appropriate size
patternize the bitmap and run inference method ( you can pass the bitmap directly if you copy to use TensorflowImageClassifier.class from the demo )
store the result and loop to another frame (2~4)
It's somewhat simplified overall process but I hope you can get a hint from this.
I am working on a project to read a video file from sdcard then process frames and re diplay as a video in real time. So far I didn't manage to come up with a solution for directly extract frames from the MediaPlayer like MediaPlayer.getCurrentFrame();. MediaMetadataRetriever.getFrameAtTime() is super slow, difficult to get a descent frame rate.
The only thing I have right now is using a TextureView surface with MediaPlayer. Here I start the MediaPlayer and in real time read the bitmap form TextureView asTextureView.getBitMap(), then process BitMap and display it on another ImageView. Here this gives me a a descent frame rate.
The problem here is TextureView has to be in the xml layout and should visible, Which I do not want.
Can some one please shed some light here? Is it possible to somehow hide the TextureView which is attaching to the MediaPlayer, without fake hiding like using RelativeLayouts :). iOS has a solution for this which is AVPLAyerItemVideoOutput, I need something like that with android.
Or any other work around to extract frames from video file?
Thank you
For video processing ...... you can use the FFMPEG Library for getting frames of videos but for that you have the knowledge of android native integration.
I hope this will help you.enter link description here
Is there a way to record square (640x640) videos and concat them in Android? I looked up in the Internet and found some solutions. The solution seems to be "ffmpeg". However, to use ffmpeg I need to dive into NDK and build ffmpeg from its sources. Is there a solution by only using the Android SDK?
My basic needs are:
Record multiple videos (square format)
Resize captured videos (i.e. 480x480 to 640x640)
Concat captured videos
Rotate final video (clockwise 90)
Final output will be in mp4 or mpg format
Is there a solution by only using the Android SDK?
Not really.
Your primary video recording option is MediaRecorder, and it supports exactly nothing of what you list. For example, there is no requirement for any Android device to support taking square videos.
You are also welcome to use the camera preview stuff to assemble your own videos from individual frames. Vine does this, AFAIK. There, you could perhaps use existing Bitmap facilities to handle the cropping, resizing, and rotating. However, this will be slow, and doing this work in a way that can keep up with a reasonable frame rate will be difficult. Also, I do not know if there is a library that can stitch those frames together into a video, or blend in any sort of audio (camera previews are pure images).
I'm looking at the class MediaRecorder of the Android SDK, and I was wondering if it can be used to record a video made from a Surface.
Example: I want to record what I display on my surface (a video game?) into a file.
As I said in the title: I'm not looking to record anything from the camera.
I think it is possible by overriding most of the class, but I'd very much like some ideas...
Beside, I'm not sure how the Camera class is used in MediaRecorder, and what I should get from my Surface to replace it.
Thank you for your interest!
PS: I'm looking at the native code used my MediaRecorder to have some clue, maybe it will inspire someone else:
http://www.netmite.com/android/mydroid/frameworks/base/media/jni/
The ability to record from a Surface was added in Android Lollipop. Here is the documentation:
http://developer.android.com/about/versions/android-5.0.html#ScreenCapture
Android 4.3 (API 18) adds some new features that do exactly what you want. Specifically, the ability to provide data to MediaCodec from a Surface, and the ability to store the encoded video as a .mp4 file (through MediaMuxer).
Some sample code is available here, including a patch for a Breakout game that records the game as you play it.
This is unfortunately not possible at the Java layer. All the communication between the Camera and Media Recorder happens in the native code layers, and there's no way to inject non-Camera data into that pipeline.
While Android 4.1 added the Media Codec APIs, which allow access to the device's video encoders, there's no easy-to-use way to take the resulting encoded streams and save them as a standard video file. You'd have find a library to do that or write one yourself.
You MAY wish to trace the rabbit hole from a different folder in AOSP
frameworks/av/media
as long as comfortable with NDK (C/C++, JNI, ...) and Android (permissions, ...)
Goes quite deep and am not sure about how far you can go on a non-rooted device.
Here's an article on how to draw on a EGLSurface and generate a video using MediaCodec:
https://www.sisik.eu/blog/android/media/images-to-video
This uses OpenGL ES but you can have MediaCodec provide a surface that you can then obtain a canvas from to draw on. No need for OpenGL ES.