Android as an UVC Camera - android

I'm stuck at home with a rather bad webcam. I was considering upgrading, but then it struck me: phones these days have really good cameras embedded in them. So why not use it as a webcam?
However, as I was researching this further I was really disappointed with the available apps for this. As far as I was able to find, we have Android apps that work roughly as follows:
Present phone camera as a network attached camera. Then you can use local software to use that feed as a webcam. See e.g., IP Webcam. This may be sufficient, but it's a complicated setup, and network latency makes this far from ideal.
The Android app sends the camera feed to an custom host application that in turn creates a virtual web camera. See e.g., DroidCam. This mostly solves the latency problem, but it is still rather complicated, and requiring us to install a specific third party application is troublesome in regard to user privacy. Especially since the applications are closed source.
So, I took the engineering approach and tried to see if it was even possible to improve the situation. As far as I was able to find, Android supports being used as a custom USB accessory. And looking over the USB video class documentation, it strikes me that it should be possible to create an Android app that presents the phone as a generic UVC webcam, such that we do not have to resort to tricks such as the ones above.
Ideally, I would have liked Android to add another USB device option ("Use USB connection as webcam") in addition to debug mode, file-transfer, etc. This seems quite unlikely to happen in the short term however.
So, my question is this: Does an application that does the above already exist? My searching thus far haven't yielded any results, but I might be missing something as googling for this turned out a bit harder than I expected.
Alternatively, am I wrong in my assumption above, such that there is some fundamental issue why an Android application cannot be made to work in that way?

There does not seem to be any complete app yet as of 2020-10, but the parts are mostly there:
https://github.com/tejado/android-usb-gadget has code to switch the Android device into gadget mode (but no UVC yet)
https://git.ideasonboard.org/uvc-gadget.git feeds v4l2 into the uvc gadget output
Sources:
http://www.davidhunt.ie/raspberry-pi-zero-with-pi-camera-as-usb-webcam/
https://www.raspberrypi.org/forums/viewtopic.php?t=148361
https://www.reddit.com/r/androiddev/comments/iabc2o/can_i_use_my_android_as_wired_camera_ie_as_a/g1nrijl/

It appears Google has started to take notice on this issue and are currently working on a "DeviceAsWebcam" service, which is exactly the solution to this problem, as seen in the Android review below:
https://android-review.googlesource.com/c/platform/system/sepolicy/+/2410788
Naturally though, this is a Android 14 feature, so it will like take a while before this is usable on a lot of devices. Hopefully, someone is able to backport this feature to older versions of Android.

If android / the version of Android that comes on your target phone provides / permits use of the USB gadget driver, then libguvc,
https://developer.ridgerun.com/wiki/index.php?title=USB_Video_Class_Gadget_Library_-_libguvc
can be used to "make an application appear as a USB webcam".
Potentially relevant to get you started would be https://stackoverflow.com/search?q=Android+USB+gadget (other SO references to the use of the USB gadget driver on Android).

Related

Simulate cellular network's behaviour on Eclipse emulator

Is there any way to simulate the behaviour of a cellular mobile network (GSM/UMTS) in the Eclipse Android emulator?
More specifically: I want to develop an app to show information about the network, such as coverage level, parameters of neighbouring cells etc, as provided by the TelephonyManager and other related classes. I would like to do as much debugging as possible with the emulator, before testing in an actual phone.
For that I would need the emulated phone to "see" a functioning cellular network, with its protocols and parameters. Is that possible?
Emulator doesn't give you the option of setting that things. I have developed a toy app for showing network parameters and the only way I found was using a real phone.
By the way, not all phones implement all methods of Android's API, so you will find that some parameters will not be shown. And also many parameters are not available from API and you will need to get them throw AT commands (you will need a rooted device)
Note: I realize this question is two years old, but since this question received a slight edit from its original author 5 minutes ago and so got bumped back up to the front of the site, I am assuming that he may still be looking for an answer.
Yes, AT&T provides such a free tool to simulate and analyze different network conditions to a certain extent. And it can be used in conjunction with Eclipse ADT, Android Studio, or something else entirely. So it's not tied to any particular IDE.
http://developer.att.com/application-resource-optimizer
That being said, I have no idea if it will give you any cell tower related meta data.

Android: any-screen capture without ROOT and constant USB connection

I'm trying to make an app that takes a video capture of the screen by taking individual snapshots. I've been looking into taking the snapshot without root, and seems that most solutions use ddmlib and require the device to be constantly connected via USB.
The only solution that doesn't require root and a constant usb connection is the android screenshot library. However, from the forum comments as well as my experience with the demo app, this library doesn't work very reliably/consistently.
Is there any other solution?
Note: I've also looked at the Noroot Screenshotit app (https://play.google.com/store/apps/details?id=com.edwardkim.android.screenshotitfullnoroot&hl=en), and seem like they have a similar approach to the android screenshot library, but have no idea how they make it more reliable. If anyone has an idea, that would be helpful as well.
Thanks
Edit: We've already got the USB-connected implementation, but we want to be able to do it on the device alone, without needing any connection to another PC.
As of Android 5.0+, you can use the new functionality in the SDK called media projections to do screen capture without root privileges or a USB connection. There is sample code by Google available here.

Analog video capture to Android phone

I am looking for a way of displaying an analog video stream on an android phone. On a pc/mac/etc you can achieve this using a cheap usb analog-digital converter such as a grabby: http://www.terratec.net/en/products/Grabby_82248.html, and then view on VLC, for example.
Would such a thing work (in theory) on android if the proper drivers were available? (ie. are there any hardware issues which make this impossible?)
Does anyone know if such a device with android drivers is available?
Ultimately I want to make an app which interfaces with the grabby (or similar device) and allows the user to view video on the android and capture and send short clips.
First of all the Android device needs to support USB Host. This limits your userbase significantly.
Then there is the problem with power. Some USB Host devices will be incompatible simply because an Android phone will not be able to push enough power through to port to get it running properly.
I'm not sure about the drivers, but I'm 99% sure it won't work "out of the box".
You should certainly take a look at THIS project. It is pretty similar to what you are trying to do. Maybe you should consider getting in touch with that person.
EDIT:
Based on what it took to get that DVB-T dongle running in the project i mentioned above the chances of creating an app that everyone will be able to simply download and use are EXTREMELY slim. Getting that dongle running required using a modified kernel and special scripts. Of course I could be wrong. You can continue the research yourself or wait for someone with more experience than me to reply.

How to take a screenshot of other app programmatically without root permission, like Screenshot UX Trial?

How to take a screenshot of other app programmatically without root permission, like Screenshot UX Trial?
I know I can capture the bitmap of the root view in my app. But I can't get the root view of the other app when my app is running in background
bitmap = Bitmap.createBitmap(rootview.getDrawingCache());
There is a permission for capturing current frame buffer in Manifest: android.permission.READ_FRAME_BUFFER. But some website says it's for signature app only.
Check Android Permissions - Protection Levels
After trying Screenshot UX Trial, I read the permission:
INTERNET: for connect to localhost screenshot server for rooted phone.
SYSTEM_ALERT_WINDOW: for topmost camera button.
VIBRATE: for vibrate feedback.
WRITE_EXTERNAL_STORAGE: to save the screenshot.
GET_TASKS: for detect foreground Develoment setting activity for non rooted&non preloaded capture method.
It seems either SYSTEM_ALERT_WINDOW or GET_TASKS allow the app to take screenshot.
I have two guess of how it works:
It may be able to access the Activity of the foreground activity, it gets the root view of the Activity, capture its screenshot.
Calling glreadpixels
If you try one of my guess, please let me know the result.
This is extremely difficult. I spent several years trying to do it. I eventually succeeded, but any solution will involve commercial as well as technical effort.
Update March 2015
Most of the stuff below is no longer up-to-date. There's now, after all these years, an android.media.projection package
https://developer.android.com/reference/android/media/projection/package-summary.html
which finally allows what you need!
Capturing the screen image of your own application
For completeness, I want to include your own comment that you can capture an image of your own application using Bitmap.createBitmap(rootview.getDrawingCache()); and similar mechanisms.
Capturing the screen of another application whilst you're in the background
Using the READ_FRAMEBUFFER permission
Firstly, you're right that a normal application can't make use of the READ_FRAMEBUFFER permission, because it's "signature"-level. That means you must be signed by the same key as the Android system ROM in order to be able to take such a screenshot.
I thought this was a bit sad, so back in 2009 I made an Android open-source project submission to ask that it be opened up1. The response from Dianne Hackborn, the Android architect was:
Um, no. Absolutely positively not.
So, that went well, then! Hence this permission is still signature-level to this day.
If you had this permission, however, you could call the captureScreen member of ISurfaceComposer2. You'd need to write some native code to access this function, using the Android NDK and also some undocumented APIs. However, it's possible.
Internally within the Android graphics subsystem, this uses a glReadPixels call to retrieve the pixels from the GPU back to the CPU. (The GPU is used for most of the compositing on Android. In fact Android 4.0+ supports extra hardware compositors, and the Surface Flinger has to do even more work to pull those pixels back to the CPU.)
This call works beautifully, except for a few small problems:
The risk of using an unsupported API which might break at any moment;
The hassle of calling it in C++
It causes the GPU pipelines to stall, which can upset the GPU designers but doesn't really cause problems in reality
It relies on a large bandwidth from the GPU back to the CPU. This is sometimes problematic because memory architectures are designed to send data in the opposite direction. However, I seem to recall that all modern Android chipset architectures directly share memory between the GPU and CPU, except for one (it may be Broadcom? - I can't remember) where this may cause this mechanism to be very slow.
... and one large problem ...
Most importantly, as a normal application writer, you can't even call this API due to the signature-level permissions required.
Still, on most Android devices, you can get 10 frames per second out of this. Better still, this API actually supports scaling the resulting image in hardware on the GPU, so if you're clever you can pre-scale the image to just the size you need, before the pixels even hit the CPU. So it can be extremely high performance.
Note, of course, that you as an application writer can't call glReadPixels because you don't have access to the relevant OpenGL context. It's owned by the surface flinger.
Using /dev/graphics/fb0 and similar
Some are tempted to try to read these Linux device files which represent the framebuffer. However, there are three problems:
You need root.
Sometimes they're not even there.
Often, they don't represent the real screen image. Remember on Android that the graphics are composited on the GPU. So there's no reason why the CPU should have access to a copy of the full composited screen image, and it often doesn't. This file sometimes contains tearing (at best) and a garbage image (at worst). Interestingly, some of the tools for rooted phones do use this method, which I think is a mistake. If you've got root, you by definition have all Android permissions and can therefore call the above captureScreen API to get a correct image.
Using hardware partners
Now we get into the solutions which require commercial action.
Talking to the Android chipset makers often presents a solution. Since they design the hardware, they have access to the framebuffer - and they often are able to provide libraries which entirely avoid the Android permissions model by simply accessing their custom kernel drivers directly.
If you're aiming at a specific phone model, this is often a good way forward. Of course, the odds are you'll need to cooperate with the phone maker as well as the silicon manufacturer.
Sometimes this can provide outstanding results. For example I have heard it's possible on some hardware to pipe the phone hardware framebuffer directly into the phone hardware H.264 video encoder, and retrieve a pre-encoded video stream of whatever is on the phone screen. Outstanding. (Unfortunately, I only know this is possible on TI OMAP chips, which are gradually withdrawing from the phone market3).
Using security holes
Android rigidly enforces its permission model, and has few security holes. However the Android OEMs can sometimes be more careless.
For example a major OEM whose name begins with S has implemented a way to capture the screen using a keystroke. It saves it to a world-readable file on the SD card. Hypothetically you might be able to find what intercepts those keys and see how it works. Perhaps you could do something similar.
And perhaps there's a way for another major OEM whose name also begins with S.
No, I'm not going to go into any more detail on this section. To work out how to do those things, I'd need to have reverse-engineered software, and that might be illegal. Good luck, though.
Working with the phone makers
As described previously, the phone makers have ready access to an API which does work. And the phone makers have the signature-level permissions required.
So, all you need to do is to arrange to get your software signed by the phone maker.
This is, however, hard. By signing the software, the phone maker is guaranteeing its quality - so they should want to audit your source code. Also, due to the nature of Android - if they sign the software, they need to be the ones distributing it. You can't put it on the Market if it is signed by someone else's signature.
However, the OEM need not include it on the ROM - they can still distribute it on the Android market. But you can't.
A good solution would be if each vendor signed a small library which then could be accessed by a common SDK. Which leads me onto...
Work with software partners who have solved this already
I know a lot about this because I used to work at RealVNC. We worked with all the major Android phone vendors to get access to these signature-level APIs. I cannot overemphasise the many, many man-years of effort (commercially and technically) required to achieve this. Some of the OEMs have publicised this work - for example 4.
I do not work at RealVNC any more, so I have nothing to gain from advertising their software. But if you really really want to be able to capture the screen on multiple Android devices, you may wish to approach them about re-using their Remote Control Service or Android VNC SDK 5. It is not open-source so you should expect to pay, and believe me this is fair enough given the epic effort involved in working with all these Android OEMs.
In the interests of balance I should point out that other vendors have also worked with the phone makers on this - e.g. Soti. But I believe they all offer specific device management solutions, rather than a general remote control/event injection SDK.
Over USB
Another option - the adb daemon which listens for debugging connections over USB has slightly more privileges than a normal application, which is why it's able to grab the screen (you can see its image using the ddms tool). If you're able to run any command using adb then you too can gain those privileges (as per the android-screenshot-library linked previously).
Contribute to the Android open-source project
Eventually this problem reduced me to dust, and I left for greener pastures which didn't involve trying to squeeze pixels out of Android phones.
Before I left RealVNC though, we tried again to contribute these APIs to the Android open-source project. This time we got a more positive reaction6. In short, it was suggested that our security approach was almost right, but that the graphics system was in too much turmoil to accept our patches. Well, the great news is that the graphics system is no longer in turmoil - in fact it now has that captureScreen API which means no graphics system changes are needed whatsoever. It may therefore be possible to submit a new security mechanism to AOSP around this API which finally solves this problem.
Maybe the android-screenshot-library can help. But well in their Usage page it says that it needs a native service started with adb (from the android sdk).
PS: Remember that Screenshot UX does not work for every unrooted phone.
I don't think Android will allow you to access another app's frame buffer. This is just part of the security of Android. Each app should keep to its own resources.
If you really need to get a screen capture of any app, I would suggest using the native screen grab "gesture". For the the Nexus 7 for example, simply "... hold the power button and the volume down button at the same time for approximately 2 seconds."
A Google search will usually find the trick with your device.

Android Tablet or iPad for Kiosk Device

We want to place a device in a store that operates as sort of a kiosk device. As in customers walk up to it and start interacting with our custom app. The app could be developed for Android or the iPad, so I'm trying to decide which one to use and would like comments on the following ideas:
Is it possible for Android or iOS to access services over the USB port? This would enable us to disable the network.
Is one particularly better for 24 hour always on?
I like the iPad as I think its supply will be more constant as we move forward and need to replace devices due to ones gone bad. Also, our app will probably work on future generations of the iPad. With Android, I'm not sure there will be that sort of consistency from the tablet vendors.
Kiosk mode? I think with the iPad by putting it in a kiosk case that removes access to the home button and turning on the restrictions we'll get what we want. What about Android? I'd rather not have to get into rooting devices and replacing their firmware.
Remote control? Any way to remotely control iOS or Android in a standard means? Our app will be a client to a master server which will obviously be able to control the app somewhat (when used purely as a display device to a customer, controlled from behind the counter).
My feeling is that neither Android tablets or the iPad is best suited for this. Are there other options?
I will try to answer your points, but know that I am probably biased towards Android, because that is where my experience lies.
With Android 3.1+, at least with the Xoom, you have full USB host capabilities. Things like USB flash/hard drives, keyboards, mice, even digital cameras, all work. If you need custom interop with a USB device, you could go as far as to write a driver for it.
24 hour always on is not good for any device with a battery, but neither is better in this situation.
While android apps are forward-compatible, bad programming practices and/or deviations from "vanilla" Android software and hardware CAN break forward compatibility. That being said, if you grab a Google Experience device like the Xoom, you won't meet as many surprises.
In Android 3.0, the navigation bar is built-in at a low level, and it is not possible for apps to remove it. Therefore, it is trivial for anyone to break a "software nanny."
I know that it is possible to control android devices remotely, but without knowing your specific needs, I can't really offer more information than that.
Good luck!
iPad NOOO believe me I am a convert to Apple for my home and business but when we went to launch kiosk the iPad FAILED Big Time.. Here are a couple of little (Big) issues we ran into.
If the device reboots you cannot auto launch you iPad app since Apple does not allow that.
There is a serious memory leak in the iPads browser. We were running javascript / CSS3 and it cratered intermittently. I literally spent 2 hours "today" on the phone with Apple getting the MAJOR run around. I finally said let me speak to an Enterprise Sales Manager as my project could mean thousands of iPads and I got NO WHERE. One Apple employee even told me they don't have enterprise sales managers.
If those weren't enough even though we are just in the proof of concept phase, we are already getting request for other options. These other options are going to require access to the OS which Apple yea right. We are moving to Android immediately.
Sorry Apple I love you but you loose here.
If your using an Ipad you should consider if it can support the power for the USB thing. Watch this Using Powered USB Port
Your idea about putting the tablet behind another piece of glass/plastic is neat. To then deal with remote controlling, you might consider doing some Bluetooth programming.
My mobile development has been primarily with iOS, so I am biased toward that SDK. I will mention that the data/sync/charge port for iOS has (I believe) never changed. Your Gen 1 iPhone sync cord works on your iPhone 4... and your iPad or iPad 2. So, in terms of third party hardware, you may see more consistency with Apple.
I haven't found a good answer regarding whether it is easier to do Bluetooth programming for iOS or Android, but I think to stay cost-effective, you might see which one is more open to third-party devices. Here is an SO post/answer about iOS and third-party Bluetooth devices; I've not found anything on Android regarding third-party Bluetooth remotes, but considering a lot of hardware running Android is third-party, your chances from a naive perspective seem pretty good. Here's the Android Bluetooth API.
Buying an iOS or Android handheld to remote control an iOS or Android tablet does seem a bit steep, but then again, maybe not. Cost also depends on your ratio of remotes to tablets. 1:1? 1:N? N:1? N:M?
The lowest end iPod goes for $229 as of May 20, 2011. Android does have more variety in terms of hardware. You may be able to get a cheap Android phone with no service plan to act as a Bluetooth remote for an Android tab.
I have provided a solution for the kiosk mode using iPad here Lock-down iPhone/iPod/iPad so it can only run one app
I am afraid that I don't really know for Android if the same thing is possible.
To address the issue of crashing applications you can use an exception trampoline (see discussion here https://blog.compeople.eu/apps/?p=275) to catch the crash and reboot your app.
If the entire device is restarted however then other apps that are on the device can be started and will subsequently be locked in.
To answer your other points:
You can use a configuration profile to control network access. Force it to use a VPN or Proxy that only allows your custom app with embedded credentials to use. That way other network access can be prevented.
Your concern over future compatibility is spot on. The Android marketplace is so fragmented then maintaining a fleet is difficult.
If you have an app that is behaving as a server and is locked in then remote control is possible.
We manufacture tablet kiosks that support both android and iPad devices. In fact we are the only iPad kiosk that has achieved apple approval.
Generally speaking i think you will have an easier time with an iPad as the software and hardware will remain more consistent over time. Which is important if you have to change out a fault unit or deploy more kiosks 6 months or year from now when the original device is no longer manufactured.

Categories

Resources