I'm trying to make screen sharing app using WebRTC. I have code that can get and share video stream from camera. I need to modify it to instead get video via MediaProjection API. Based on this post I have modified my code to use org.webrtc.ScreenCapturerAndroid, but there is no video output shown. There is only black screen. If I use camera, everything works fine (I can see camera output on screen). Could someone please check my code and maybe point me in right direction? I have been stuck on this for three days already.
Here is my code:
public class MainActivity extends AppCompatActivity {
private static final String TAG = "VIDEO_CAPTURE";
private static final int CAPTURE_PERMISSION_REQUEST_CODE = 1;
private static final String VIDEO_TRACK_ID = "video_stream";
PeerConnectionFactory peerConnectionFactory;
SurfaceViewRenderer localVideoView;
ProxyVideoSink localSink;
VideoSource videoSource;
VideoTrack localVideoTrack;
EglBase rootEglBase;
boolean camera = false;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
rootEglBase = EglBase.create();
localVideoView = findViewById(R.id.local_gl_surface_view);
localVideoView.init(rootEglBase.getEglBaseContext(), null);
startScreenCapture();
}
#TargetApi(21)
private void startScreenCapture() {
MediaProjectionManager mMediaProjectionManager = (MediaProjectionManager) getApplication().getSystemService(Context.MEDIA_PROJECTION_SERVICE);
startActivityForResult(mMediaProjectionManager.createScreenCaptureIntent(), CAPTURE_PERMISSION_REQUEST_CODE);
}
#Override
public void onActivityResult(int requestCode, int resultCode, Intent data) {
if (requestCode != CAPTURE_PERMISSION_REQUEST_CODE) { return; }
start(data);
}
private void start(Intent permissionData) {
//Initialize PeerConnectionFactory globals.
PeerConnectionFactory.InitializationOptions initializationOptions =
PeerConnectionFactory.InitializationOptions.builder(this)
.setEnableVideoHwAcceleration(true)
.createInitializationOptions();
PeerConnectionFactory.initialize(initializationOptions);
//Create a new PeerConnectionFactory instance - using Hardware encoder and decoder.
PeerConnectionFactory.Options options = new PeerConnectionFactory.Options();
DefaultVideoEncoderFactory defaultVideoEncoderFactory = new DefaultVideoEncoderFactory(
rootEglBase.getEglBaseContext(), true,true);
DefaultVideoDecoderFactory defaultVideoDecoderFactory = new DefaultVideoDecoderFactory(rootEglBase.getEglBaseContext());
peerConnectionFactory = PeerConnectionFactory.builder()
.setOptions(options)
.setVideoDecoderFactory(defaultVideoDecoderFactory)
.setVideoEncoderFactory(defaultVideoEncoderFactory)
.createPeerConnectionFactory();;
VideoCapturer videoCapturerAndroid;
if (camera) {
videoCapturerAndroid = createCameraCapturer(new Camera1Enumerator(false));
} else {
videoCapturerAndroid = new ScreenCapturerAndroid(permissionData, new MediaProjection.Callback() {
#Override
public void onStop() {
super.onStop();
Log.e(TAG, "user has revoked permissions");
}
});
}
videoSource = peerConnectionFactory.createVideoSource(videoCapturerAndroid);
DisplayMetrics metrics = new DisplayMetrics();
MainActivity.this.getWindowManager().getDefaultDisplay().getRealMetrics(metrics);
videoCapturerAndroid.startCapture(metrics.widthPixels, metrics.heightPixels, 30);
localVideoTrack = peerConnectionFactory.createVideoTrack(VIDEO_TRACK_ID, videoSource);
localVideoTrack.setEnabled(true);
//localVideoTrack.addRenderer(new VideoRenderer(localRenderer));
localSink = new ProxyVideoSink().setTarget(localVideoView);
localVideoTrack.addSink(localSink);
}
//find first camera, this works without problem
private VideoCapturer createCameraCapturer(CameraEnumerator enumerator) {
final String[] deviceNames = enumerator.getDeviceNames();
// First, try to find front facing camera
Logging.d(TAG, "Looking for front facing cameras.");
for (String deviceName : deviceNames) {
if (enumerator.isFrontFacing(deviceName)) {
Logging.d(TAG, "Creating front facing camera capturer.");
VideoCapturer videoCapturer = enumerator.createCapturer(deviceName, null);
if (videoCapturer != null) {
return videoCapturer;
}
}
}
// Front facing camera not found, try something else
Logging.d(TAG, "Looking for other cameras.");
for (String deviceName : deviceNames) {
if (!enumerator.isFrontFacing(deviceName)) {
Logging.d(TAG, "Creating other camera capturer.");
VideoCapturer videoCapturer = enumerator.createCapturer(deviceName, null);
if (videoCapturer != null) {
return videoCapturer;
}
}
}
return null;
}
}
ProxyVideoSink
public class ProxyVideoSink implements VideoSink {
private VideoSink target;
synchronized ProxyVideoSink setTarget(VideoSink target) { this.target = target; return this; }
#Override
public void onFrame(VideoFrame videoFrame) {
if (target == null) {
Log.w("VideoSink", "Dropping frame in proxy because target is null.");
return;
}
target.onFrame(videoFrame);
}
}
In logcat I can see, that some frames are rendered, but nothing is shown (black screen).
06-18 17:42:44.750 11357-11388/com.archona.webrtcscreencapturetest I/org.webrtc.Logging: EglRenderer: local_gl_surface_viewDuration: 4000 ms. Frames received: 117. Dropped: 0. Rendered: 117. Render fps: 29.2. Average render time: 4754 μs. Average swapBuffer time: 2913 μs.
06-18 17:42:48.752 11357-11388/com.archona.webrtcscreencapturetest I/org.webrtc.Logging: EglRenderer: local_gl_surface_viewDuration: 4001 ms. Frames received: 118. Dropped: 0. Rendered: 118. Render fps: 29.5. Average render time: 5015 μs. Average swapBuffer time: 3090 μs.
I'm using latest version of WebRTC library: implementation 'org.webrtc:google-webrtc:1.0.23546'.
My device has API level 24 (Android 7.0), but I have tested this code on 3 different devices with different API levels, so I don't suspect device specific problem.
I have tried building another app that uses MediaProjection API (without WebRTC) and I can see correct output inside SurfaceView.
I have tried downgrading webrtc library, but nothing seems to work.
Thanks for any help.
I was faced same issue using WebRTC library org.webrtc:google-webrtc:1.0.22672. I am using android 7.0 device. Video call is working fine. Issue is with screen sharing. Screen sharing showing black screen always.
Then I added following:
peerConnectionFactory.setVideoHwAccelerationOptions(rootEglBase.getEglBaseContext(), rootEglBase.getEglBaseContext());
Now it is working perfectly.
Related
Right now I made my own camera using CameraX. The preview works fine and the image is getting saved. However the preview is blurry and there is hardly any documentation on how to auto focus or even manually focus. It's really important for my app to be very sharp since I am using Firebase's Machine learning kit to recognize text with it.
private void startCamera() {
CameraX.unbindAll();
Rational aspectRatio = new Rational (textureView.getWidth(), textureView.getHeight());
Size screen = new Size(textureView.getWidth(), textureView.getHeight()); //size of the screen
PreviewConfig pConfig = new PreviewConfig.Builder().setTargetAspectRatio(aspectRatio).setTargetResolution(screen).setTargetRotation(Surface.ROTATION_0).build();
preview = new Preview(pConfig);
preview.setOnPreviewOutputUpdateListener(
new Preview.OnPreviewOutputUpdateListener() {
//to update the surface texture we have to destroy it first then re-add it
#Override
public void onUpdated(Preview.PreviewOutput output){
ViewGroup parent = (ViewGroup) textureView.getParent();
parent.removeView(textureView);
parent.addView(textureView, 0);
textureView.setSurfaceTexture(output.getSurfaceTexture());
updateTransform();
}
});
ImageCaptureConfig imageCaptureConfig = new ImageCaptureConfig.Builder().setCaptureMode(ImageCapture.CaptureMode.MIN_LATENCY)
.setTargetRotation(getWindowManager().getDefaultDisplay().getRotation()).build();
imgCap = new ImageCapture(imageCaptureConfig);
findViewById(R.id.imgCapture).setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View v) {
cameraImage.setImageResource(R.drawable.cameraon);
CameraX.unbind(preview);
loadingDialog.startLoadingDialog();
file = new File(Environment.getExternalStorageDirectory() + "/" + System.currentTimeMillis() + ".png");
imgCap.takePicture(file, new ImageCapture.OnImageSavedListener() {
#Override
public void onImageSaved(#NonNull File file) {
String filePath = file.getPath();
Bitmap bitmap = BitmapFactory.decodeFile(filePath);
rotateImage(bitmap);
}
#Override
public void onError(#NonNull ImageCapture.ImageCaptureError imageCaptureError, #NonNull String message, #Nullable Throwable cause) {
String msg = "Image Capture Failed : " + message;
Toast.makeText(getBaseContext(), msg,Toast.LENGTH_LONG).show();
if(cause != null){
cause.printStackTrace();
}
}
});
}
});
//bind to lifecycle:
CameraX.bindToLifecycle((LifecycleOwner)this, preview, imgCap);
}
If you could share some code sample I think it'd be more help full to answer the question. Overall the CameraControl class (link to source) has two relevant APIs
ListenableFuture<FocusMeteringResult> startFocusAndMetering(FocusMeteringAction action);
ListenableFuture<Void> cancelFocusAndMetering();
I believe you can use startFocusMetering with relevant argument to trigger a center focus or face focus (if you have face coordinates). Now if you hook this function to be called each time before the shot is taken or run this in a loop to be called every 500 ms, it may solve your problem.
Some Code references after Github lookup:
Example 1
Example 2
Example 3
I'm in the final stages of creating my Foley app! (my first ever created app during my internship, I learned how to make it myself, so sorry if any questions are stupid!)
My game is finished and working (also thanks to many of you!) and now I am trying to make the second part.
The idea is that it will show a video without sound. People then will have the opportunity to use our Foley room to record sounds over the video.
So far so good, but I still have a few issues.
First, I can't seem to find a way to stop the video/audio and replay it. When you take all the correct steps, there is no issue, but as most of you will know you want to make your code as waterproof as possible, so it would be nice to find a solution to this.
Second, I'd like to know if it's possible to export the combination of the video and recording audio to youtube. (I found a code to upload, but haven't tried if my idea is possible, does anyone have any experience with this?)
//<-----------------------------------Aangeven variabelen--------------->
MediaRecorder _recorder;
MediaPlayer _player;
Button _start;
Button _stop;
public static bool recorderplaying = false;
protected override void OnCreate(Bundle savedInstanceState)
{
base.OnCreate(savedInstanceState);
SetContentView(Resource.Layout.VideoActie);
//Videovariabelen
var videoView = FindViewById<VideoView>(Resource.Id.PlaceHolder);
Button play = FindViewById<Button>(Resource.Id.Play);
Button stop = FindViewById<Button>(Resource.Id.stop);
//opnamevariabelen
_start = FindViewById<Button>(Resource.Id.record);
_stop = FindViewById<Button>(Resource.Id.stop);
//<-----------------------------------Opnemen audio---------------->
//Opslaan opname
string path = $"{Android.OS.Environment.ExternalStorageDirectory.AbsolutePath}/test.3gpp";
//Toegang vragen tot opslag android telefoon
if (Build.VERSION.SdkInt > BuildVersionCodes.M)
{
if (CheckSelfPermission(Manifest.Permission.WriteExternalStorage) != Android.Content.PM.Permission.Granted
|| CheckSelfPermission(Manifest.Permission.RecordAudio) != Android.Content.PM.Permission.Granted)
{
RequestPermissions(new[] { Manifest.Permission.WriteExternalStorage, Manifest.Permission.RecordAudio }, 0);
}
}
//<-----------------------------------Video afspelen-------------->
//video source
videoView.SetMediaController(new MediaController(this));
videoView.SetVideoPath($"android.resource://{PackageName}/{Resource.Raw.puppy}");
videoView.RequestFocus();
//<-----------------------------------Buttons--------------------->
//opname start
_start.Click += delegate
{
_stop.Enabled = !_stop.Enabled;
_start.Enabled = !_start.Enabled;
_recorder.SetAudioSource(AudioSource.Mic);
_recorder.SetOutputFormat(OutputFormat.ThreeGpp);
_recorder.SetAudioEncoder(AudioEncoder.AmrNb);
_recorder.SetOutputFile(path);
_recorder.Prepare();
_recorder.Start();
videoView.Start();
recorderplaying = true;
};
//opname stop
stop.Click += delegate
{
_stop.Enabled = !_stop.Enabled;
videoView.Pause();
_player.Stop();
if (recorderplaying == true)
{
_recorder.Stop();
_recorder.Reset();
recorderplaying = false;
}
else
{
int pass = 0;
}
};
play.Click += delegate
{
_stop.Enabled = !_stop.Enabled;
_player.SetDataSource(path);
_player.Prepare();
_player.Start();
videoView.Start();
};
}
//<-----------------------------------OnResume, OnPause---------------->
protected override void OnResume()
{
base.OnResume();
_recorder = new MediaRecorder();
_player = new MediaPlayer();
_player.Completion += (sender, e) => {
_player.Reset();
_start.Enabled = !_start.Enabled;
_stop.Enabled = !_stop.Enabled;
};
}
protected override void OnPause()
{
base.OnPause();
_player.Release();
_recorder.Release();
_player.Dispose();
_recorder.Dispose();
_player = null;
_recorder = null;
}
}
}
This is what I have so far.
I also tried to make a global variable for stopping the video, since that worked fine for the audio in the game, but sadly that didn't work.
note: don't mind the puppy video, it's a placeholder haha!
If anyone has any idea if and how this is possible, that would be amazing!!
I am trying to scan an image taken from resources using a Recognizer with a RegerParserSettings inside a fragment. The problem is that BaseRecognitionResult obtained through the callback onScanningDone is always null. I have tried to set up the RecognitionSettings with MRTDRecognizer and worked fine, so I think that the library is properly integrated. This is the source code that I am using:
#Override
public void onAttach(Context context) {
...
try {
mRecognizer = Recognizer.getSingletonInstance();
mRecognizer.setLicenseKey(context, LICENSE_KEY);
} catch (FeatureNotSupportedException | InvalidLicenceKeyException e) {
Log.d(TAG, e.getMessage());
}
buildRecognitionSettings();
mRecognizer.initialize(context, mRecognitionSettings, new DirectApiErrorListener() {
#Override
public void onRecognizerError(Throwable t) {
//Handle exception
}
});
}
private void buildRecognitionSettings() {
mRecognitionSettings = new RecognitionSettings();
mRecognitionSettings.setRecognizerSettingsArray(setupSettingsArray());
}
private RecognizerSettings[] setupSettingsArray() {
RegexParserSettings regexParserSettings = new RegexParserSettings("[A-Z0-9]{17}");
BlinkOCRRecognizerSettings sett = new BlinkOCRRecognizerSettings();
sett.addParser("myRegexParser", regexParserSettings);
return new RecognizerSettings[] { sett };
}
I scan the image like:
mRecognizer.recognizeBitmap(bitmap, Orientation.ORIENTATION_PORTRAIT, FragMicoblink.this);
And this is the callback handled in the fragment
#Override
public void onScanningDone(RecognitionResults results) {
BaseRecognitionResult[] dataArray = results.getRecognitionResults();
//dataArray is null
for(BaseRecognitionResult baseResult : dataArray) {
if (baseResult instanceof BlinkOCRRecognitionResult) {
BlinkOCRRecognitionResult result = (BlinkOCRRecognitionResult) baseResult;
if (result.isValid() && !result.isEmpty()) {
String parsedAmount = result.getParsedResult("myRegexParser");
if (parsedAmount != null && !parsedAmount.isEmpty()) {
Log.d(TAG, "Result: " + parsedAmount);
}
}
}
}
}`
Thanks in advance!
Helllo Spirrow.
The difference between your code and SegmentScanActivity is that your code uses DirectAPI, which can process only single bitmap image you send for processing, while SegmentScanActivity processes camera frames as they arrive from the camera. While doing so, it can utilize time redundant information to improve the OCR quality, i.e. it combines consecutive OCR results from multiple video frames to obtain a better quality OCR result.
This feature is not available via DirectAPI - you need to use either SegmentScanActivity, or custom scan activity with our camera management.
You can also find out more here:
https://github.com/BlinkID/blinkid-android/issues/54
Regards
I am preparing one android application in which I have to detect to movement of eyes. Somehow I am able to achieve the above thing on images but I want this on live eyes.
I am not able to understand that if we can use the proximity sensor to detect the eyes. Just like smartStay feature.
Please suggest the ideas to implement the same.
We can use the front camera to detect the eyes and eyes blink. Use the Vision api for detecting the eyes.
Code for eye tracking:
public class FaceTracker extends Tracker<Face> {
private static final float PROB_THRESHOLD = 0.7f;
private static final String TAG = FaceTracker.class.getSimpleName();
private boolean leftClosed;
private boolean rightClosed;
#Override
public void onUpdate(Detector.Detections<Face> detections, Face face) {
if (leftClosed && face.getIsLeftEyeOpenProbability() > PROB_THRESHOLD) {
leftClosed = false;
} else if (!leftClosed && face.getIsLeftEyeOpenProbability() < PROB_THRESHOLD){
leftClosed = true;
}
if (rightClosed && face.getIsRightEyeOpenProbability() > PROB_THRESHOLD) {
rightClosed = false;
} else if (!rightClosed && face.getIsRightEyeOpenProbability() < PROB_THRESHOLD) {
rightClosed = true;
}
if (leftClosed && !rightClosed) {
EventBus.getDefault().post(new LeftEyeClosedEvent());
} else if (rightClosed && !leftClosed) {
EventBus.getDefault().post(new RightEyeClosedEvent());
} else if (!leftClosed && !rightClosed) {
EventBus.getDefault().post(new NeutralFaceEvent());
}
}
}
//method to call the FaceTracker
private void createCameraResources() {
Context context = getApplicationContext();
// create and setup the face detector
mFaceDetector = new FaceDetector.Builder(context)
.setProminentFaceOnly(true) // optimize for single, relatively large face
.setTrackingEnabled(true) // enable face tracking
.setClassificationType(/* eyes open and smile */ FaceDetector.ALL_CLASSIFICATIONS)
.setMode(FaceDetector.FAST_MODE) // for one face this is OK
.build();
// now that we've got a detector, create a processor pipeline to receive the detection
// results
mFaceDetector.setProcessor(new LargestFaceFocusingProcessor(mFaceDetector, new FaceTracker()));
// operational...?
if (!mFaceDetector.isOperational()) {
Log.w(TAG, "createCameraResources: detector NOT operational");
} else {
Log.d(TAG, "createCameraResources: detector operational");
}
// Create camera source that will capture video frames
// Use the front camera
mCameraSource = new CameraSource.Builder(this, mFaceDetector)
.setRequestedPreviewSize(640, 480)
.setFacing(CameraSource.CAMERA_FACING_FRONT)
.setRequestedFps(30f)
.build();
}
No you can't use proximity sensor for eye detection or tracking . Give a shot to OpenCV .
Link : OpenCv
github : OpenCv github
I have successfully been running WebRTC in my Android app for a while, using libjingle.so and PeerConnectionClient.java, etc., from Google's code library. However, I am now running into a problem where a user starts a connection as audio only (i.e., an audio call), but then toggles video on. I augmented the existing setVideoEnabled() in PeerConnectionClient as such:
public void setVideoEnabled(final boolean enable) {
executor.execute(new Runnable() {
#Override
public void run() {
renderVideo = enable;
if (localVideoTrack != null) {
localVideoTrack.setEnabled(renderVideo);
} else {
if (renderVideo) {
//AC: create a video track
String cameraDeviceName = VideoCapturerAndroid.getDeviceName(0);
String frontCameraDeviceName =
VideoCapturerAndroid.getNameOfFrontFacingDevice();
if (numberOfCameras > 1 && frontCameraDeviceName != null) {
cameraDeviceName = frontCameraDeviceName;
}
Log.i(TAG, "Opening camera: " + cameraDeviceName);
videoCapturer = VideoCapturerAndroid.create(cameraDeviceName);
if (createVideoTrack(videoCapturer) != null) {
mediaStream.addTrack(localVideoTrack);
localVideoTrack.setEnabled(renderVideo);
peerConnection.addStream(mediaStream);
} else {
Log.d(TAG, "Local video track is still null");
}
} else {
Log.d(TAG, "Local video track is null");
}
}
if (remoteVideoTrack != null) {
remoteVideoTrack.setEnabled(renderVideo);
} else {
Log.d(TAG,"Remote video track is null");
}
}
});
}
This allows me successfully see a local inset of the device's video camera, but it doesn't send the video to the remove client. I thought the peerConnection.addStream() call would do that, but perhaps I am missing something else?
To avoid building an external mechanism of communication between peers that will involve an answer from the second peer that the new stream can be added, you can always start with existing (but sometimes empty) video stream. Now it is just the matter of filling this stream with content when (and if) necessary.