Develop Reference

Screen Sharing between Devices using Media Projection API

I am developing an app which has the functionality of sharing screens with other apps.
I used the Media projection API for this. I also used MediaMuxer to combine the audio and video outputs for screen sharing.
I know that Media Projection APIs are used for screen recording but all I want is to share the screen while recording.
For this, I have modified the writeSampleData method of the MediaMuxer class to send bytes via a socket to the other device over the network.
Below is the code for that:
OutputStream outStream;
outStream = ScreenRecordingActivity.getInstance().socket.getOutputStream();
void writeSampleData(final int trackIndex, final ByteBuffer byteBuf, final MediaCodec.BufferInfo bufferInfo) {
if (mStatredCount > 0) {
mMediaMuxer.writeSampleData(trackIndex, byteBuf, bufferInfo);
if (bufferInfo.size != 0) {
byteBuf.position(bufferInfo.offset);
byteBuf.limit(bufferInfo.offset + bufferInfo.size);
if (outStream != null) {
try {
byte[] bytes = new byte[byteBuf.remaining()];
byteBuf.get(bytes);
//Send the data
outStream.write(bytes);
outStream.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}
The bytes are successfully transferred via socket and I am also able to receive these bytes at the receiver's end.
Below is the code for receiving bytes at the receiver's end:
private class SocketThread implements Runnable {
#Override
public void run() {
Socket socket;
try {
serverSocket = new ServerSocket(SERVER_PORT);
} catch (IOException e) {
e.printStackTrace();
}
if (null != serverSocket) {
while (!Thread.currentThread().isInterrupted()) {
try {
socket = serverSocket.accept();
CommunicationThread commThread = new CommunicationThread(socket);
new Thread(commThread).start();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
class CommunicationThread implements Runnable {
InputStream in;
DataInputStream dis;
public CommunicationThread(Socket clientSocket) {
updateMessage("Server Started...");
}
public void run() {
while (!Thread.currentThread().isInterrupted()) {
try {
byte[] data = new byte[512];
} catch (Exception e) {
e.printStackTrace();
try {
fos.close();
} catch (Exception e1) {
e1.printStackTrace();
}
}
}
}
}
}
I followed the following links for Screen Sharing:
Screen capture
screenrecorder
Screen recording with mediaProjection
I used some code from the above examples to make an app.
All I want to know is how to handle the bytes at the receiver. How do I format these bytes to play a live stream from the sender's side?
Am I following the correct approach for sending and receiving byte data?
Does MediaProjection allow one to stream the Screen while recording between applications?
Any help will be deeply appreciated.

Generally for streaming, including screen sharing, the audio and video tracks are not muxed. Instead, each video frame and audio sample is sent using a protocol like RTP/RTSP, in which each data chunk is wrapped with other things like timestamps.
You can take a look at spyadroid which is a good starting point for streaming audio and video over RTSP to a browser or VLC. It streams the camera and microphone but you can adapt it for your own use case.
If you want to go with sockets for the moment, you have to get rid of the MediaMuxer and send frames/samples directly from the Encoder output, appended with timestamps at least to synchronize the playback in the receiver side, after sending CSDs - assuming that you encode in h.264 format - data (SPS PPS aka csd-0 and csd-1 that you can get when the encoder format is changed) to the receiver Decoder, which you can configure with an output surface to render your stream.
Some extra links :
android-h264-stream-demo
RTMP Java Muxer for Android
RTSP
RTP
WebRTC

Android: Video-streaming with Mediarecorder over Sockets using ParcelFileDescriptor

I'm struggling with this quiet long and I decided I need help.
The basis idea is described here: Link
I want to stream the output from the MediaRecorder directly to my Server over Sockets while recording. (In this specific case I want to see if this is possible without using any streaming protocol just via a HTTP-POST.)
The error of my current approach is: E/MediaRecorder: start failed: -2147483648 - RuntimeException
Before I had an error pointing me to this topic mediarecorder-issue-on-lollipop
I use a AsyncTask class to call my recordData() function:
new sendAsync().execute("test");
private void recordData(Socket socket){
try{
OutputStream out = socket.getOutputStream();
InputStream is = null;
ParcelFileDescriptor[] parcelFileDescriptors = ParcelFileDescriptor.createPipe();
parcelRead = new ParcelFileDescriptor(parcelFileDescriptors[0]);
parcelWrite = new ParcelFileDescriptor(parcelFileDescriptors[1]);
try{
is = new ParcelFileDescriptor.AutoCloseInputStream(parcelRead);
} catch (Exception e){
e.printStackTrace();
}
//pfd = ParcelFileDescriptor.fromSocket(socket);
sft = new SurfaceTexture(0);
sf = new Surface(sft);
if(recorder == null) {
recorder = new MediaRecorder();
recorder.reset();
recorder.setVideoSource(MediaRecorder.VideoSource.CAMERA);
recorder.setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP);
recorder.setVideoEncoder(MediaRecorder.VideoEncoder.DEFAULT);
recorder.setVideoFrameRate(30);
recorder.setPreviewDisplay(sf);
recorder.setOutputFile(parcelWrite.getFileDescriptor());
recorder.setMaxDuration(10000); // 10 seconds
try {
recorder.prepare();
recorder.start();
System.out.println("This is Recorder running");
} catch (IOException e) {
e.printStackTrace();
}
byte[] buffer = new byte[16384];
int byteread = 0;
while ((byteread = is.read(buffer, 0, buffer.length)) != -1) {
out.write(buffer, 0, byteread);
}
out.flush();
}else{
stopRecording();
}
} catch (Exception e) {
e.printStackTrace();
}
}
Sorry for the bad coding-style.
I tried to change the OutputFormat and VideoEncoder following other solutions found to the topic. But nevertheless I'm still not sure if I'm going in the right direction.
Even after coming around this bug I think I somehow need to read out and send the Stream to the server in its own thread.
Every hint could help.

I'm not sure if this is an error due to latest versions on Android (tested on Nougat and Oreo). But as much I tried to find a workaround I still ended up on the same error message with trying to use the MediaRecorder.
Thanks to libstreaming I use now the MediaCodec API explained in their example how to use the Library: Example 2.
Then I take the Inputstream of the MediaCodec API and read it into a byte array until a certain size is reached and send this to my http-server.

How to add Media Recorder to Android Google Play Service Vision Barcode scanner

This question was asked but never answered here -- but it is somewhat different than my need, anyway.
I want to record video, while running the Google Vision library in the background, so whenever my user holds up a barcode (or approaches one closely enough) the camera will automatically detect and scan the barcode -- and all the while it is recording the video. I know the Google Vision demo is pretty CPU intensive, but when I try a simpler version of it (i.e. without grabbing every frame all the time and handing it to the detector) I'm not getting reliable barcode reads.
(I am running a Samsung Galaxy S4 Mini on KitKat 4.4.3 Unfortunately, for reasons known only to Samsung, they no longer report the OnCameraFocused event, so it is impossible to know if the camera grabbed the focus and call the barcode read then. That makes grabbing and checking every frame seem like the only viable solution.)
So to at least prove the concept, I wanted to simply modify the Google Vision Demo. (Found Here)
It seems the easiest thing to do is simply jump in the code and add a media recorder. I did this in the CameraSourcePreview method during surface create.
Like this:
private class SurfaceCallback implements SurfaceHolder.Callback
{
#Override
public void surfaceCreated(SurfaceHolder surface)
{
mSurfaceAvailable = true;
try
{
startIfReady();
if (mSurfaceAvailable)
{
Camera camera = mCameraSource.getCameraSourceCamera();
/** ADD MediaRecorder to Google Example **/
if (camera != null && recordThis)
{
if (mMediaRecorder == null)
{
mMediaRecorder = new MediaRecorder();
camera.unlock();
SurfaceHolder sh = mSurfaceView.getHolder();
mMediaRecorder.setPreviewDisplay(sh.getSurface());
mMediaRecorder.setCamera(camera);
mMediaRecorder.setAudioSource(MediaRecorder.AudioSource.CAMCORDER);
mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.CAMERA);
mMediaRecorder.setProfile(CamcorderProfile.get(CamcorderProfile.QUALITY_HIGH));
String OutputFile = Environment.getExternalStorageDirectory() + "/" +
DateFormat.format("yyyy-MM-dd_kk-mm-ss", new Date().getTime()) + ".mp4";
File newOutPut = getVideoFile();
String newOutPutFileName = newOutPut.getPath();
mMediaRecorder.setOutputFile(newOutPutFileName);
Log.d("START MR", OutputFile);
try { mMediaRecorder.prepare(); } catch (Exception e) {}
mCameraSource.mediaRecorder = mMediaRecorder;
mMediaRecorder.start();
}
}
}
}
catch (SecurityException se)
{
Log.e(TAG, "Do not have permission to start the camera", se);
}
catch (IOException e)
{
Log.e(TAG, "Could not start camera source.", e);
}
}
That DOES record things, while still handing each frame off to the Vision code. But, strangely, when I do that, the camera does not seem to call autofocus correctly, and the barcodes are not scanned -- since they are never really in focus, and therefore not recognized.
My next thought was to simply capture the frames as the barcode detector is handling the frames, and save them to the disk one by one (I can mux them together later.)
I did this in CameraSource.java.
This does not seem to be capturing all of the frames, even though I am writing them out in a separate AsyncTask running in the background, which I thought would get them eventually -- even if it took awhile to catch up. The saving was not optimized, but it looks as though it is dropping frames throughout, not just at the end.
To add this code, I tried putting it in the
private class FrameProcessingRunnable
in the run() method.
Right after the FrameBuilder Code, I added this:
if (saveImagesIsEnabled)
{
if (data == null)
{
Log.d(TAG, "data == NULL");
}
else
{
SaveImageAsync saveImage = new SaveImageAsync(mCamera.getParameters().getPreviewSize() );
saveImage.execute(data.array());
}
}
Which calls this class:
Camera.Size lastKnownPreviewSize = null;
public class SaveImageAsync extends AsyncTask<byte[], Void, Void>
{
Camera.Size previewSize;
public SaveImageAsync(Camera.Size _previewSize)
{
previewSize = _previewSize;
lastKnownPreviewSize = _previewSize;
}
#Override
protected Void doInBackground(byte[]... dataArray)
{
try
{
if (previewSize == null)
{
if (lastKnownPreviewSize != null)
previewSize = lastKnownPreviewSize;
else
return null;
}
byte[] bitmapData = dataArray[0];
if (bitmapData == null)
{
Log.d("doInBackground","NULL: ");
return null;
}
// where to put the output file (note: /sdcard requires WRITE_EXTERNAL_STORAGE permission)
File storageDir = Environment.getExternalStorageDirectory();
String imageFileName = baseFileName + "_" + Long.toString(sequentialCount++) + ".jpg";
String filePath = storageDir + "/" + "tmp" + "/" + imageFileName;
FileOutputStream out = null;
YuvImage yuvimage = new YuvImage(bitmapData, ImageFormat.NV21, previewSize.width,
previewSize.height, null);
try
{
out = new FileOutputStream(filePath);
yuvimage.compressToJpeg(new Rect(0, 0, previewSize.width,
previewSize.height), 100, out);
}
catch (Exception e)
{
e.printStackTrace();
}
finally
{
try
{
if (out != null)
{
out.close();
}
}
catch (IOException e)
{
e.printStackTrace();
}
}
}
catch (Exception ex)
{
ex.printStackTrace();
Log.d("doInBackground", ex.getMessage());
}
return null;
}
}
I'm OK with the mediarecorder idea, or the brute force frame capture idea, but neither seem to be working correctly.

How to implement Android mirroring application, using spydroid library and media projection?

I want know how to send my screen over RTP? Here is my approach.
First, I'm using media projection to capture my screen. Referenced url is http://mattsnider.com/video-recording-with-mediaprojectionmanager/
Second, I'm using and try to modify spydroid library, to stream my screen not camera view.
Referenced url is https://github.com/fyhertz/spydroid-ipcamera
I've done screen capture using media projection. Here is some part of sample code.
private boolean drainEncoder() {
mDrainHandler.removeCallbacks(mDrainEncoderRunnable);
while (true) {
int bufferIndex = mVideoEncoder.dequeueOutputBuffer(mVideoBufferInfo, 0);
if (bufferIndex == MediaCodec.INFO_TRY_AGAIN_LATER) {
// nothing available yet
break;
}
else if (bufferIndex == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// should happen before receiving buffers, and should only happen once
if (mTrackIndex >= 0) {
throw new RuntimeException("format changed twice");
}
mTrackIndex = mMuxer.addTrack(mVideoEncoder.getOutputFormat());
if (!mMuxerStarted && mTrackIndex >= 0) {
mMuxer.start();
mMuxerStarted = true;
}
}
else if (bufferIndex < 0) {
// not sure what's going on, ignore it
}
else {
ByteBuffer encodedData = mVideoEncoder.getOutputBuffer(bufferIndex);
if (encodedData == null) {
throw new RuntimeException("couldn't fetch buffer at index " + bufferIndex);
}
if ((mVideoBufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
mVideoBufferInfo.size = 0;
}
if (mVideoBufferInfo.size != 0) {
if (mMuxerStarted) {
encodedData.position(mVideoBufferInfo.offset);
encodedData.limit(mVideoBufferInfo.offset + mVideoBufferInfo.size);
mMuxer.writeSampleData(mTrackIndex, encodedData, mVideoBufferInfo);
}
else {
// muxer not started
}
}
mVideoEncoder.releaseOutputBuffer(bufferIndex, false);
if ((mVideoBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
break;
}
}
}
}
As you can see,
mMuxer.writeSampleData(mTrackIndex, encodedData, mVideoBufferInfo);
If it possible to send encodedData via RTP then mirroring will work fine.
Here is spydroid code, VideoStream.Java
protected void encodeWithMediaRecorder() throws IOException {
Log.d(TAG, "Video encoded using the MediaRecorder API");
// We need a local socket to forward data output by the camera to the packetizer
createSockets();
// Reopens the camera if needed
destroyCamera();
createCamera();
// The camera must be unlocked before the MediaRecorder can use it
unlockCamera();
try {
mMediaRecorder = new MediaRecorder();
mMediaRecorder.setCamera(mCamera);
mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.CAMERA);
mMediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP);
mMediaRecorder.setVideoEncoder(mVideoEncoder);
mMediaRecorder.setPreviewDisplay(mSurfaceView.getHolder().getSurface());
mMediaRecorder.setVideoSize(mRequestedQuality.resX, mRequestedQuality.resY);
mMediaRecorder.setVideoFrameRate(mRequestedQuality.framerate);
// The bandwidth actually consumed is often above what was requested
mMediaRecorder.setVideoEncodingBitRate((int) (mRequestedQuality.bitrate * 0.8));
// We write the ouput of the camera in a local socket instead of a file !
// This one little trick makes streaming feasible quiet simply: data from the camera
// can then be manipulated at the other end of the socket
mMediaRecorder.setOutputFile(mSender.getFileDescriptor());
mMediaRecorder.prepare();
mMediaRecorder.start();
}
catch (Exception e) {
throw new ConfNotSupportedException(e.getMessage());
}
// This will skip the MPEG4 header if this step fails we can't stream anything :(
InputStream is = mReceiver.getInputStream();
try {
byte buffer[] = new byte[4];
// Skip all atoms preceding mdat atom
while (!Thread.interrupted()) {
while (is.read() != 'm') ;
is.read(buffer, 0, 3);
if (buffer[0] == 'd' && buffer[1] == 'a' && buffer[2] == 't') break;
}
}
catch (IOException e) {
Log.e(TAG, "Couldn't skip mp4 header :/");
stop();
throw e;
}
// The packetizer encapsulates the bit stream in an RTP stream and send it over the network
mPacketizer.setDestination(mDestination, mRtpPort, mRtcpPort);
mPacketizer.setInputStream(mReceiver.getInputStream());
mPacketizer.start();
mStreaming = true;
}
Can anyone advise me? Some more information how to modify spydroid library to send mirroring data?

MediaRecorder issue on Android Lollipop

I'm testing libstreaming on new Android Lollipop, and this code that worked on previous release, seems to launch exception.
try {
mMediaRecorder = new MediaRecorder();
mMediaRecorder.setCamera(mCamera);
mMediaRecorder.setVideoSource(MediaRecorder.VideoSource.CAMERA);
mMediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.THREE_GPP);
mMediaRecorder.setVideoEncoder(mVideoEncoder);
mMediaRecorder.setPreviewDisplay(mSurfaceView.getHolder().getSurface());
mMediaRecorder.setVideoSize(mRequestedQuality.resX,mRequestedQuality.resY);
mMediaRecorder.setVideoFrameRate(mRequestedQuality.framerate);
// The bandwidth actually consumed is often above what was requested
mMediaRecorder.setVideoEncodingBitRate((int)(mRequestedQuality.bitrate*0.8));
// We write the ouput of the camera in a local socket instead of a file !
// This one little trick makes streaming feasible quiet simply: data from the camera
// can then be manipulated at the other end of the socket
mMediaRecorder.setOutputFile(mSender.getFileDescriptor());
mMediaRecorder.prepare();
mMediaRecorder.start();
} catch (Exception e) {
throw new ConfNotSupportedException(e.getMessage());
}
Launched exception is:
MediaRecorder: start failed -38
11-18 09:50:21.028: W/System.err(15783): net.majorkernelpanic.streaming.exceptions.ConfNotSupportedException
11-18 09:50:21.028: W/System.err(15783): at net.majorkernelpanic.streaming.video.VideoStream.encodeWithMediaRecorder(VideoStream.java:442)
11-18 09:50:21.028: W/System.err(15783): at net.majorkernelpanic.streaming.MediaStream.start(MediaStream.java:250)
I've tried to comment:
mMediaRecorder.setOutputFile(mSender.getFileDescriptor());
no exception launched, but when I start streaming a dialog tell me that need an outputfile.
Help appreciated.

I filed a bug report on AOSP.
https://code.google.com/p/android/issues/detail?id=80715
"The current SELinux policies don't allow for mediaserver to handle app generated abstract unix domain sockets.
Instead, I'd recommend you create a pipe-pair ( http://developer.android.com/reference/android/os/ParcelFileDescriptor.html#createPipe() ) which is allowed by the Android 5.0 policy.
"
I don't know why they did this or how we were supposed to know.
I'm using a very old/modified (can't tell) version of libstreaming where mediastream is still extended from mediarecorder, but looking at the current version, in MediaStream you'll probably want to change createSockets to something including the following:
ParcelFileDescriptor[] parcelFileDescriptors =ParcelFileDescriptor.createPipe();
parcelRead = new ParcelFileDescriptor(parcelFileDescriptors[0]);
parcelWrite = new ParcelFileDescriptor(parcelFileDescriptors[1]);
then in your video/audio stream
setOutputFile(parcelWrite.getFileDescriptor());
and in that same file
change
// The packetizer encapsulates the bit stream in an RTP stream and send it over the network
mPacketizer.setInputStream(mReceiver.getInputStream());
mPacketizer.start();
to
InputStream is = null;
try{ is = new ParcelFileDescriptor.AutoCloseInputStream(parcelRead);
}
catch (Exception e){}
mPacketizer.setInputStream(is);
As andreasperelli pointed out in the comment, make sure to close the ParcelFileDescriptors in closeSockets(), or depending on your implementation and version, before closeSockets() and before you call MediaRecorder.stop().

at Android 6.0 I resolve this problem with the code
new Thread(new Runnable() {
#Override public void run() {
FileInputStream inputStream = null;
try {
inputStream = new FileInputStream(path);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
while (true) {
byte[] buffer = new byte[0];
try {
buffer = new byte[inputStream.available()];
} catch (IOException e) {
e.printStackTrace();
}
try {
inputStream.read(buffer);
} catch (IOException e) {
e.printStackTrace();
}
try {
mSender.getOutputStream().write(buffer);
mSender.getOutputStream().flush();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}).start();
I use a file as buffer and write bytes at another thread.the MediaRecorder output to the file.