I have to get the live stream video from DJI Phantom 3 camera in my C++ application, in order to do a Computer Vision processing in OpenCV.
First I tried sending the H264 raw data through an UDP socket, inside this callback:
mReceivedVideoDataCallBack = new CameraReceivedVideoDataCallback() {
#Override
public void onResult(byte[] videoBuffer, int size) {
//Here, I call a method from a class I created, that sends the buffer through UDP
if (gravar_trigger) controleVideo.enviarFrame(videoBuffer, size);
if (mCodecManager != null) mCodecManager.sendDataToDecoder(videoBuffer, size);
}
};
That communication above works well. However, I haven't been able to decode that UDP H264 data in my C++ desktop application. I have tested with FFmpeg lib, but couldn't get to alocate an AVPacketwith my UDP data, in order to decode using avcodec_send_packet and avcodec_receive_frame. I also had problems with AVCodecContext, since my UDP communication wasn't a stream like RTSP, where it could get information about its source. Therefore, I had to change how I was trying to solve the problem.
Then, I found libstreaming, in which can be associate to stream the android video camera to a Wowza Server, creating something like a RTSP stream connection, where the data could be obtained in my final C++ application easily using OpenCV videoCapture. However, libstreaming uses its own surfaceView. In other words, I would have to link the libstreaming surfaceView with the DJI Drone's videoSurface. I'm really new to Android, so don't have any clue of how to do that.
To sum up, is that the correct approach? Someone has a better idea? Thanks in advance
I'm going to wager a couple things. Well, Mostly one thing. One typically needs to handle creating fragmented video packets before sending them. The IDRs of h264 are too large for udp streaming.
Having a solid com link between endpoints, you can add a method which converts a single potentially large packet input to one or more small packet outputs.
The packets that are larger than perhaps 1000 bytes need to be broken into several h264 NALU type 28s. The packets that are small and have the same timestamp can be sent in STAP-A type 24s. Typically you can find inband sps/pps in a stap-a.
Once you have a packetizer for IDRs and large slices, write you depacketizor on the receiver and then you should get clear decoded pictures.
Refer to the h264 RTP specs for how to make the type 28s.
After a long time, I finally developed a system that can stream the DJI drone camera correctly
https://github.com/raullalves/DJI-Drone-Camera-Streaming
Related
I need to send some video data from a drone's camera to a server and then on to a client web page. I am working on an Android app that interacts with the drone - it successfully gets the drone's camera output and renders it. I already have a Java server (using Java Lightweight HTTP Server) working with the client web page that could be used for this purpose. I have come up with two possible methods:
h264 and Wowza
I have a callback method available from the drone's camera that provides a h264 encoded byte array. This should be streamable via RTP, however I haven't found any Android/Java libraries to support streaming this from the source I have available (libstreaming only supports streaming from the camera afaik). If it is possible to stream the incoming h264 byte arrays to Wowza, I could then figure out how to access Wowza from the client and render the stream. This method would not need to use the existing server.
Sampling Bitmaps - The camera output is rendered onto a Texture View in the Android app. This can be sampled to produce Bitmap images, which could be encoded as Base64 and sent to the server. The server then passes the latest Bitmap onto the client and the client renders the Bitmap using the encoded Base64:
document.getElementById('drone_feed')
.setAttribute(
'src', 'data:image/png;base64,' + cameraBase64String
);
I have successfully implemented the second method - the Texture View is sampled every 100ms and the client fetches the current encoded image from the server every 100ms. Obviously this method gives very poor latency and low FPS, but it does work.
So I have two questions:
1. Is the first method (h264 and Wowza) possible, and if so how do I go about streaming the incoming h264 video buffers to Wowza?
2. If the first method is not possible, can I make the second method more efficient, or is there an alternative method I have not considered?
You can accomplish this using OpenGL with Android GoCoder SDK to grab the frames from the drone and then forward them to Wowza Streaming Engine or Cloud.
Examples can be found here:OpenGL
I am trying to develop a simple application that show the video stream from an IP camera into a surfaceview.
I am totally new to video decode/encode. In the last few days I have read a lot of information about mediacodec API and about how to implement it, but I can not find the right way. I still have to fully understand how buffers works and how depacketize the RTP packets from UDP and pass each frame to MediaCodec
I have a couple of Sony EP521 IP camera. From the CGI Command Manual I get that the cameras support Mpeg-4/H264 HTTP bit stream ("GET /h264...", the camera will send H.264 raw data as its response.) or RTP (UDP) bit stream.
My problem is that I do not know where to start:
Which is the "best" way to implement this? (with best I mean the most reliable/correct but still easy way)
Should I use HTTP bit stream or RTP?
Are MediaCodec strictly needed or can I implement this in another way? (ie, android.media.mediaplayer class already support h.264 raw data over RTP (I do not if it actually does or not))
How can I extract the video data from an HTTP bit stream?
I know that there are a lot if similar question, but no one seems to fully answer my doubts.
The camera also support MJpeg. This would be easier to implement, but for the moment I do not want to use MJpeg encoding.
Here the Camera CGI manual: http://wikisend.com/download/740040/G5%20Camera%20CGI%20manual.pdf
Thank you, and sorry If already been discussed.
Updating my previous question...
Currently I am working on a live media streaming application. I have been able to use the Media Recorder class to record medias and store it in a file. Currently I am trying to stream this media from android to Adobe Media streamer using RTP. What I have done us using the parcel file descriptor I have packaged the video contents which are captured from the device camera. Now I would want to use the RTPPackets class provided by Sipdroid to convert this packets into RTP packets.
I am having problems integrating this RTP Packets and using the same in to my application as I see there are multiple steps to be done
1) Creating RTP packets, even though I have got the code for RTP packetizer I am not exactly sure how to use the same and where to use this. Because I clearly not sure on what would be the default payload value, ssrc and ccrc values. May the first time these will carry the default values but the second time onward what would set the values for these parameters would it be the parcel file descriptor?
2) Creating a simple server kind of code on the mobile which continues to create the RTP packets and keep sending them to the Adobe media server?
Any help would be much appreciated.
Thanks,
K.Saravanan.
Hi everybody I hope you can help me out with this.
The problem:
I have an RTP stream which I'm multicasting on my private network (WiFi). I would like to use a number of android tablets for displaying the stream. The number of tablets cannot be restricted and the quality should not degrade with increasing number of clients. This explains why I need multicasting rather than unicasts.
The approach:
Theoretically by creating a RTSP or HTTP stream on the server side I should be able to serve the video to my clients. However, my understanding is that the server would take a performance hit when too many clients are connecting at the same time, which I need to avoid. Ideally I would like all clients to simply be listening on the very same multicast. That way the number of clients would have no impact on server performance. [NOTE: The IP is local and TTL is set to 0/1 so no danger of clogging anything else than my own network with the multicast packets.]
The implementation
To implement the approach above I thought to write a multicast client in Android that receives the RTP packets and stitches together the stream. I tried this with JPEG payload and it works quite well. The problem with JPEG, however, is that the BitmapFactory.decodeByteArray call to decode each frame is very expensive (almost 100ms!) which limits the frame rate considerably. The load on the network is also quite high since JPEG is not a good video streaming protocol.
What I would like to do is to do for video what I already did for pictures, i.e. stitch together the payload stream (e.g. MPEG4) from the RTP packets and feed it to "something". Initially I thought VideoView would work with a raw input stream but I was wrong, VV seems to work only with a rtsp or http url (correct?).
Solution?
Now, what are my options? I'd like to avoid setting up a RTSP server from the raw RTP stream and serve all tablets for the reasons above. I did look around for 2 days and checked all the solutions proposed on SO and on the net but nothing seemed to apply to my problem (the RTSP url or a unicast was the solution in most cases, but I don't think I can use it) so I thought it was finally time to ask this question.
Any help is very appreciated!
cheers
After reading your post again, I picked up on something I missed the first time. I used BitmapFactory.decodeByteArray for MJPEG over HTTP from an Axis camera multicast. The call can be done in a few ms. The problem there is that it normally wants to make a new Bitmap every call. There is a way to make the Bitmap persist and that will get the times way down. I just can't remember the call offhand and my normal dev computer is currently being destroyed... err, 'upgraded' by our IT, so I can't tell you off the top of my head, but you should find it if you search a bit. I was able to get 30fps on a Xoom and Galaxy Tab 10.1 and some others no problem.
Mark Boettcher
mboettcher#ara.com
OK, I checked and I used the overloaded BitmapFactory.decodeByteArray to use an immutable Bitmap with the InBitmap flag set in BitmapFactory.Options. May have been something else I had to do for the Bitmap itself, probably made it Static at the very least. May have been some other flags to set also, but you should definitely have enough to go on now.
We had a problem trying to play MJPEG steaming over RTSP on the Android. The multicast video server we had was not able to send MJPEG over HTTP and we did not want to use H.264 over RTSP because of latency. The application was a ROV sending live video back to a Droid for display.
Just to save you a lot of trouble, if I understand the problem correctly, you simply cannot do it with anything in the Android SDK, like MediaPlayer, etc. In the end we got it working by paying a guy to do some custom code using MPlayer, ffmpeg and Live555.
Hope this helps.
I'm trying to build a system that live-streams video and audio captured by android phones. Video and auido are captured on the android side using MediaRecorder, and then pushed directly to a server written in python. Clients should access this live feed using their browser, so the I implemented the streaming part of the system using flash. Right now both video and audio content appear on the client side, but the problem is that they are out of sync. I'm sure this is caused by wrong timestamp values in flash (currently I increment ts by 60ms for a frame of video, but clearly this value should be variable).
The audio is encoded into amr on the android phone, so I know exactly each frame of amr is 20ms. However, this is not the case with video, which is encoded into H.264. To synchronized them together, I would have to know exactly how many millisecs each frame of H.264 lasts, so that I can timestamp them later when delivering content using flash. My question is is this kind of information available in NAL units of H.264? I tried to find the answer in H.264 standard, but the information there is just overwhelming.
Can someone please point me at the right direction? Thanks.
Timestamps are not in NAL units, but are typically part of RTP. RTP/RTCP also takes care of media synchronisation.
The RTP payload format for H.264 might also be of interest to you.
If you are not using RTP, are you just sending raw data units over the network?