Develop Reference

Android camera image transport to ROS in real-time have an OOM trouble?

I'm trying to transport the image-info from Android Camera to ROS in real-time. However, I got a OOM problem. I'm new to Android-ROS, nearly have no experiences of dealing with such problem.
Here're some information of my demo: (if you guys need more, pls comment)
1.
public class MainActivity extends RosActivity implements NodeMain, SurfaceHolder.Callback, Camera.PreviewCallback
2.Dependencies Opencv-for-Android(3.2.0).
3.ROS messages type: android_cv_bridge.
I'm trying to publish the image-messages in onPreviewFrame() function. Code like this:
#Override
public void onPreviewFrame(byte[] data, Camera camera) {
Camera.Size size = camera.getParameters().getPreviewSize();
YuvImage yuvImage = new YuvImage(data, ImageFormat.NV21, size.width, size.height, null);
Bitmap bmp = null;
if(yuvImage != null){
ByteArrayOutputStream baos = new ByteArrayOutputStream();
yuvImage.compressToJpeg(new Rect(0, 0, size.width, size.height), 80, baos);
bmp = BitmapFactory.decodeByteArray(baos.toByteArray(), 0, baos.size());
try{
baos.flush();
baos.close();
}
catch(IOException e){
e.printStackTrace();
}
image = imagePublisher.newMessage();
Time curTime = connectedNode.getCurrentTime();
image.setEncoding("rgba8");
image.getHeader().setStamp(curTime);
image.getHeader().setFrameId("camera");
curTime = null;
if(isOpenCVInit){
Mat mat_image = new Mat(bmp.getHeight(), bmp.getWidth(), CvType.CV_8UC4, new Scalar(0));
Bitmap copyBmp = bmp.copy(Bitmap.Config.ARGB_8888, true);
// bitmap to mat
Utils.bitmapToMat(copyBmp, mat_image);
// mat to cvImage
CvImage cvImage = new CvImage(image.getHeader(), "rgba8", mat_image);
try {
imagePublisher.publish(cvImage.toImageMsg(image));
} catch (IOException e) {
e.printStackTrace();
}
mat_image.release();
mat_image = null;
if(!bmp.isRecycled()) {
bmp.recycle();
bmp = null;
}
if(!copyBmp.isRecycled()) {
copyBmp.recycle();
copyBmp = null;
}
cvImage =null;
image = null;
}
}
yuvImage = null;
System.gc();
}
The imagePublisher are initialized here:
#Override
public void onStart(ConnectedNode connectedNode) {
this.connectedNode = connectedNode;
imagePublisher = connectedNode.newPublisher(topic_name, sensor_msgs.Image._TYPE);
}
Well, I had try my best to avoid the OOM problem. I had also trying to not apply the OpenCV, and just dealing with the bitmap like this:
ChannelBufferOutputStream cbos = new ChannelBufferOutputStream(MessageBuffers.dynamicBuffer());
bmp.compress(Bitmap.CompressFormat.JPEG, 80, baos);
cbos.buffer().writeBytes(baos.toByteArray());
image.setData(cbos.buffer().copy());
cbos.buffer().clear();
imagePublisher.publish(image);
Unfortunately, it's get worse. I'm doubt the way I'm trying to achieve this target. Or is there a better way to do?

I think your problem might be that your network can't transfer this amount of image data and the OOM is caused by the data stuck in buffers that is not yet transferred.
I had similar issues when I wanted to transfer image from my android device. If your problem is the same, you could solve it in several ways:
transfer data via usb tethering, it's generally much faster than wifi or cellular and can transfer even raw image stream without compression with 30 fps 640x480. For Jpeg I think you will be able to stream FullHD at 30 fps.
save data on the phone to a ROS Bag http://wiki.ros.org/rosbag and then work with the data. Here you miss realtime, but sometimes it's not needed. To make it I actually wrote an application for android https://github.com/lamerman/ros_android_bag and you can also download it directly from Google Play https://play.google.com/store/apps/details?id=org.lamerman.rosandroidbag&hl=en
try to decrease the bandwidth even further (decrease image size, fps) or increase the network quality
About your second attempt with transferring JPEG instead of RAW data, have a look at this source code, here it's implemented correctly https://github.com/rosjava/android_core/blob/kinetic/android_10/src/org/ros/android/view/camera/CompressedImagePublisher.java#L80
The problem of transferring via network is for sure actual for raw images, but may also be for compressed ones if the size of image is big and the frame rate is high.

Android Ndk: Uploading a Large File through NDK

I am trying to upload a file (20MB of size) but while uploading, logcat shows
Out of Memory Exception
So I thought to use NDK for this. But i dont know how to proceed. So help me on this
static int chunkSize = 512;
static final byte[] chunks = new byte[chunkSize];
.....
......
while (true)
{
synchronized (chunks)
{
int amountRead = fileInputStream.read(chunks);
System.out.println("========amount read========="+amountRead);
if (amountRead == -1)
{
break;
}
bufferOutputStream.write(chunks, 0, amountRead);
bufferOutputStream.flush();
}
}

When you upload, you need to use an InputStream rather than loading the whole file into memory.

How to get MJPG stream video from android IPWebcam using opencv

I am using the IP Webcam program on android and receiving it on my PC by WiFi. What I want is to use opencv in Visual Studio, C++, to get that video stream, there is an option to get MJPG stream by the following URL: http://MyIP:port/videofeed
How to get it using opencv?

Old question, but I hope this can help someone (same as my answer here)
OpenCV expects a filename extension for its VideoCapture argument,
even though one isn't always necessary (like in your case).
You can "trick" it by passing in a dummy parameter which ends in the
mjpg extension:
So perhaps try:
VideoCapture vc;
ipCam.open("http://MyIP:port/videofeed/?dummy=param.mjpg")

Install IP Camera Adapter and configure it to capture the videostream. Then install ManyCam and you'll see "MPEG Camera" in the camera section.(you'll see the same instructions if you go to the link on how to setup IPWebCam for skype)
Now you can access your MJPG stream just like a webcam through openCV. I tried this with OpenCV 2.2 + QT and works well.
Think this helps.

I did a dirty patch to make openCV working with android ipWebcam:
In the file OpenCV-2.3.1/modules/highgui/src/cap_ffmpeg_impl.hpp
In the function bool CvCapture_FFMPEG::open( const char* _filename )
replace:
int err = av_open_input_file(&ic, _filename, NULL, 0, NULL);
by
AVInputFormat* iformat = av_find_input_format("mjpeg");
int err = av_open_input_file(&ic, _filename, iformat, 0, NULL);
ic->iformat = iformat;
and comment:
err = av_seek_frame(ic, video_stream, 10, 0);
if (err < 0)
{
filename=(char*)malloc(strlen(_filename)+1);
strcpy(filename, _filename);
// reopen videofile to 'seek' back to first frame
reopen();
}
else
{
// seek seems to work, so we don't need the filename,
// but we still need to seek back to filestart
filename=NULL;
int64_t ts = video_st->first_dts;
int flags = AVSEEK_FLAG_FRAME | AVSEEK_FLAG_BACKWARD;
av_seek_frame(ic, video_stream, ts, flags);
}
That should work. Hope it helps.

This is the solution (im using IP Webcam on android):
CvCapture* capture = 0;
capture = cvCaptureFromFile("http://IP:Port/videofeed?dummy=param.mjpg");
I am not able to comment, so im posting new post. In original answer is an error - used / before dummy. THX for solution.

Working example for me
// OpenCVTest.cpp : Defines the entry point for the console application.
//
#include "stdafx.h"
#include "opencv2/highgui/highgui.hpp"
/**
* #function main
*/
int main( int argc, const char** argv )
{
CvCapture* capture;
IplImage* frame = 0;
while (true)
{
//Read the video stream
capture = cvCaptureFromFile("http://192.168.1.129:8080/webcam.mjpeg");
frame = cvQueryFrame( capture );
// create a window to display detected faces
cvNamedWindow("Sample Program", CV_WINDOW_AUTOSIZE);
// display face detections
cvShowImage("Sample Program", frame);
int c = cvWaitKey(10);
if( (char)c == 27 ) { exit(0); }
}
// clean up and release resources
cvReleaseImage(&frame);
return 0;
}
Broadcast mjpeg from a webcam with vlc, how described at http://tumblr.martinml.com/post/2108887785/how-to-broadcast-a-mjpeg-stream-from-your-webcam-with

How to encode non-camera video in Android

I am working on an android application in which a video is dynamically generated by compositing a sequence of animation frames. I tried to use the Android Media Recorder API for this but have not found a way to get it to accept a non-camera source as input. I have been attempting to use a FFMPEG port (based on the Rockplayer build) but am running into difficulties with missing functions since I am using it as an encoder, not a decoder.
The iPhone version of this app uses AVAssetWriter from the AVFoundation framework.
Is there an easier way to do this or am I stuck slugging it out with FFMPEG?

This may help (see the note on resolution though):-
How to encode using the FFMpeg in Android (using H263)
I'm not sure if they did a custom build of ffmpeg, or not, if so they may be able to offer advice on porting a more feature complete version.
-Anthony

Opencv has ViewBase class which takes the input from the camera as a frame and represent the frame as a bitmap , you can extand the class View base and make it for your own use , even though installing opencv on the android isn't very easy.
When you extend SampleCvViewBase you will have the following function which you can use pretty much hard work but the best I can think of.
#Override
protected Bitmap processFrame(VideoCapture capture) {
capture.retrieve(picture, Highgui.CV_CAP_ANDROID_COLOR_FRAME_RGBA);
if (Utils.matToBitmap(picture, bmp))
return bmp;
bmp.recycle();
return null;
}

You can use a pure Java open source library called JCodec ( http://jcodec.org ).
It contains a simple yet working H.264 encoder and MP4 muxer. The class below uses JCodec low level API and should be what you need ( CORRECTED ):
public class SequenceEncoder {
private SeekableByteChannel ch;
private Picture toEncode;
private RgbToYuv420 transform;
private H264Encoder encoder;
private ArrayList<ByteBuffer> spsList;
private ArrayList<ByteBuffer> ppsList;
private CompressedTrack outTrack;
private ByteBuffer _out;
private int frameNo;
private MP4Muxer muxer;
public SequenceEncoder(File out) throws IOException {
this.ch = NIOUtils.writableFileChannel(out);
// Transform to convert between RGB and YUV
transform = new RgbToYuv420(0, 0);
// Muxer that will store the encoded frames
muxer = new MP4Muxer(ch, Brand.MP4);
// Add video track to muxer
outTrack = muxer.addTrackForCompressed(TrackType.VIDEO, 25);
// Allocate a buffer big enough to hold output frames
_out = ByteBuffer.allocate(1920 * 1080 * 6);
// Create an instance of encoder
encoder = new H264Encoder();
// Encoder extra data ( SPS, PPS ) to be stored in a special place of
// MP4
spsList = new ArrayList<ByteBuffer>();
ppsList = new ArrayList<ByteBuffer>();
}
public void encodeImage(BufferedImage bi) throws IOException {
if (toEncode == null) {
toEncode = Picture.create(bi.getWidth(), bi.getHeight(), ColorSpace.YUV420);
}
// Perform conversion
for (int i = 0; i < 3; i++)
Arrays.fill(toEncode.getData()[i], 0);
transform.transform(AWTUtil.fromBufferedImage(bi), toEncode);
// Encode image into H.264 frame, the result is stored in '_out' buffer
_out.clear();
ByteBuffer result = encoder.encodeFrame(_out, toEncode);
// Based on the frame above form correct MP4 packet
spsList.clear();
ppsList.clear();
H264Utils.encodeMOVPacket(result, spsList, ppsList);
// Add packet to video track
outTrack.addFrame(new MP4Packet(result, frameNo, 25, 1, frameNo, true, null, frameNo, 0));
frameNo++;
}
public void finish() throws IOException {
// Push saved SPS/PPS to a special storage in MP4
outTrack.addSampleEntry(H264Utils.createMOVSampleEntry(spsList, ppsList));
// Write MP4 header and finalize recording
muxer.writeHeader();
NIOUtils.closeQuietly(ch);
}
public static void main(String[] args) throws IOException {
SequenceEncoder encoder = new SequenceEncoder(new File("video.mp4"));
for (int i = 1; i < 100; i++) {
BufferedImage bi = ImageIO.read(new File(String.format("folder/img%08d.png", i)));
encoder.encodeImage(bi);
}
encoder.finish();
}
}
You can get JCodec jar from a project web-site.

Android: Jpeg saved from camera looks corrupted

I'm writing an Android application that saves a JPEG snapshot from the camera when the user clicks a button. Unfortunately, when I look at the JPEG file my code is saving looks corrupted. It appears to be caused by my call to parameters.setPreviewSize (see code snippet below) - if I remove that then the image saves fine; however without it I can't set the preview size and setDisplayOrientation also appears to have no effect without it.
My app is targeting API Level 8 (Android 2.2), and I'm debugging on an HTC Desire HD. Not quite sure what I'm doing wrong here... any help would be very much appreciated!
Cheers,
Scottie
public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
// Now that the size is known, set up the camera parameters and begin
// the preview.
Camera.Parameters parameters = mCamera.getParameters();
Camera.Size size = getBestPreviewSize(w,h);
// This next call is required in order for preview size to be set and
// setDisplayOrientation to take effect...
// Unfortunately it's also causing JPEG to be created wrong
parameters.setPreviewSize(size.width, size.height);
parameters.setPictureFormat(ImageFormat.JPEG);
mCamera.setParameters(parameters);
mCamera.setDisplayOrientation(90);
mCamera.startPreview();
}
// This is the snapshot button event handler
public void onSnapshotButtonClick(View target) {
//void android.hardware.Camera.takePicture(ShutterCallback shutter,
// PictureCallback raw, PictureCallback jpeg)
mPreview.mCamera.takePicture(null, null, mPictureCallback);
}
// This saves the camera snapshot as a JPEG file on the SD card
Camera.PictureCallback mPictureCallback = new Camera.PictureCallback() {
public void onPictureTaken(byte[] imageData, Camera c) {
if (imageData != null) {
FileOutputStream outStream = null;
try {
String myJpgPath = String.format(
"/sdcard/%d.jpg", System.currentTimeMillis());
outStream = new FileOutputStream(myJpgPath);
outStream.write(imageData);
outStream.close();
Log.d("TestApp", "onPictureTaken - wrote bytes: "
+ imageData.length);
c.startPreview();
Toast.makeText(getApplicationContext(), String.format("%s written", myJpgPath), Toast.LENGTH_SHORT).show();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
}
}
}
};

Another workaround is to match the aspect ratio between preview and picture sizes (i.e. setPreviewSize(w1,h1); setPictureSize(w2,h2) with w1/h1 ~ w2/h2 (small differences seem to be ok)). E.g. for Desire HD S w1=800,h1=480, w2=2592,h2=1552 works as well as w1=960,h1=720,h2=2592,h2=1952 (if you don't mind distorted images ;-)

I assume that you are using a common implementation of the getBestPreviewSize(w,h) method that is floating about, where you cycle through the different getSupportedPreviewSizes() to find the best match. Although I am not certain as to why it causes the images to be distorted, I have found that calling the parameters.setPreviewSize(size.width, size.height) method with the output of the getBestPreviewSize method is what is causing the problem on the HTC Desire. I have also verified that by commenting it out, the distorted image issue goes away.