Is there any existing support to Tensorflow on Android for locally saved videos? The demo provided is tightly coupled with the camera, and porting it to work for videos will be non-trivial and time-consuming, at the very least. The task it is intended for, is to process raw frames from a stream being broadcast live.
You have to take bitmaps for frame in video using MediaMetadataRetriever or something appropriate and then pass them to the tensorflow library for image recognition.
Currently there is no existing support for "video" stream itself in tensorflow AFAIK, even the demo takes screenshots of the camera preview to recognize.
If you really want to recognize video stream itself then you have to build a model of your own.
Otherwise, Process to analyze video would be as follows assuming you already have your graph and label file and playing video is not needed ( If you want to show video during the analysis, then you should implement surfaceview or textureview in your activity):
Initialize tensorflow & load desired video using MediaMetadataRetriever
extract bitmaps for desired frames using getFrameAtTime & scale the bitmap to appropriate size
patternize the bitmap and run inference method ( you can pass the bitmap directly if you copy to use TensorflowImageClassifier.class from the demo )
store the result and loop to another frame (2~4)
It's somewhat simplified overall process but I hope you can get a hint from this.
Related
Assuming we have a Surface in Android that displays a video (e.g. h264) with a MediaPlayer:
1) Is it possible to change the displayed saturation, contrast & brightness of the displayed on the surface video? In real time? E.g. Images can use setColorFilter is there anything similar in Android to process the video frames?
Alternative question (if no. 1 is too difficult):
2) If we would like to export this video with e.g. an increased saturation, we should use a Codec, e.g. MediaCodec. What technology (method, class, library, etc...) should we use before the codec/save action to apply the saturation change?
For display only, one easy approach is to use a GLSurfaceView, a SurfaceTexture to render the video frames, and a MediaPlayer. Prokash's answer links to an open source library that shows how to accomplish that. There are a number of other examples around if you search those terms together. Taking that route, you draw video frames to an OpenGL texture and create OpenGL shaders to manipulate how the texture is rendered. (I would suggest asking Prokash for further details and accepting his answer if this is enough to fill your requirements.)
Similarly, you could use the OpenGL tools with MediaCodec and MediaExtractor to decode video frames. The MediaCodec would be configured to output to a SurfaceTexture, so you would not need to do much more than code some boilerplate to get the output buffers rendered. The filtering process would be the same as with a MediaPlayer. There are a number of examples using MediaCodec as a decoder available, e.g. here and here. It should be fairly straightforward to substitute the TextureView or SurfaceView used in those examples with the GLSurfaceView of Prokash's example.
The advantage of this approach is that you have access to all the separate tracks in the media file. Because of that, you should be able to filter the video track with OpenGL and straight copy other tracks for export. You would use a MediaCodec in encode mode with the Surface from the GLSurfaceView as input and a MediaMuxer to put it all back together. You can see several relevant examples at BigFlake.
You can use a MediaCodec without a Surface to access decoded byte data directly and manipulate it that way. This example illustrates that approach. You can manipulate the data and send it to an encoder for export or render it as you see fit. There is some extra complexity in dealing with the raw byte data. Note that I like this example because it illustrates dealing with the audio and video tracks separately.
You can also use FFMpeg, either in native code or via one of the Java wrappers out there. This option is more geared towards export than immediate playback. See here or here for some libraries that attempt to make FFMpeg available to Java. They are basically wrappers around the command line interface. You would need to do some extra work to manage playback via FFMpeg, but it is definitely doable.
If you have questions, feel free to ask, and I will try to expound upon whatever option makes the most sense for your use case.
If you are using a player that support video filters then you can do that.
Example of such a player is VLC, which is built around FFMPEG [1].
VLC is pretty easy to compile for Android. Then all you need is the libvlc (aar file) and you can build your own app. See compile instructions here.
You will also need to write your own module. Just duplicate an existing one and modify it. Needless to say that VLC offers strong transcoding and streaming capabilities.
As powerful VLC for Android is, it has one huge drawback - video filters cannot work with hardware decoding (Android only). This means that the entire video processing is on the CPU.
Your other options are to use GLSL / OpenGL over surfaces like GLSurfaceView and TextureView. This guaranty GPU power.
Is there an API call or library to embed graphic into Android video during recording. I want to imprint a timestamp directly onto the video. I have seen discussions of intercepting the bit stream but I am concerned about a home-grown solution like this having compatibility and performance issues. Currently I add it in post-processing using ffmpeg which has issues with file timestamps.
I have some ideas for you:
Record using GLSurfaceView(check Rotoscope for example) or TextureView
Use GLES20.glReadPixels() for GLSurfaceView or getBitmap() for TextureView to get bitmap data
Create Canvas from bitmap then draw anything you want
Have fun!
Is there a way to record square (640x640) videos and concat them in Android? I looked up in the Internet and found some solutions. The solution seems to be "ffmpeg". However, to use ffmpeg I need to dive into NDK and build ffmpeg from its sources. Is there a solution by only using the Android SDK?
My basic needs are:
Record multiple videos (square format)
Resize captured videos (i.e. 480x480 to 640x640)
Concat captured videos
Rotate final video (clockwise 90)
Final output will be in mp4 or mpg format
Is there a solution by only using the Android SDK?
Not really.
Your primary video recording option is MediaRecorder, and it supports exactly nothing of what you list. For example, there is no requirement for any Android device to support taking square videos.
You are also welcome to use the camera preview stuff to assemble your own videos from individual frames. Vine does this, AFAIK. There, you could perhaps use existing Bitmap facilities to handle the cropping, resizing, and rotating. However, this will be slow, and doing this work in a way that can keep up with a reasonable frame rate will be difficult. Also, I do not know if there is a library that can stitch those frames together into a video, or blend in any sort of audio (camera previews are pure images).
How can I put text or transparent image into a video? I can display text overlayed camera output but how can I record it? Using opencv is an alternative but I don't really want to use opencv manager(or 25+ mb binary).
Is there a way to record overlayed video with Android SDK or 3rd party library. What are my options?
Update: I'm not looking for a "record to disc then load recorded video and process every frame" solution. I'm trying to find a way to process every camera frames before recording. Something like opencv.
You can get help from here to get video byes array of each frame from camera and then save them using some third party encoder. Now you can create bitmap from byte array, and using bitmap you can write a overlay text on it. Example code here and here is the link for the third party encoder AndroidFFmpeg
I would like to convert multiple images(frames) to a video(MP4) in an android device. Also, I would like to convert video(MP4) into multiple images(for each frame). I have limited knowledge on FFMPEG, but installing FFMPEG in Android may consume more time. I Would like to ask experienced engineers to suggest a better strategy which can take less time to complete this task. Please point me to some open source code which I may modify to complete this task quickly.
First you need to convert Image to YUV format using Image Decoder.
Then you can feed each YUV Image as a Video Input to Media Recorder Engine.
Go through the Media Recorder Source Code to get more info.