I came across this relatively old post which describes how impressively Nexus One's noise cancellation works and I was wondering where can I find more information about its implementation in the OS software.
In particular:
How much of it is done using software and how much of it is done in
hardware?
Which modules in the Android source code are responsible for noise
cancellation?
Can I control its behavior via Android's API? (if so, which ones)
Does it also work with the microphone in the headset that comes with
Nexus One (4-pin 3.5mm jack) or does it work with the built-in
microphone only?
I only know the answer for the Nexus One, but:
It's done in hardware.
Not sure.
Nope.
Maybe?
For the N1, it works using a second microphone in the back, and comparing the two signals. I don't know exactly how this process is done (hardware or software), but I know there isn't an API for it. Also, it probably doesn't work for the external headset, since there's no second sound source to compare the first one to (unless the headset has two mics too, but I don't think it does).
About the Nexus One:
All hardware only configuration in software.
Sound drivers and sound system but only configuration.
No API possibly some prop configuration but I haven't been able to get that to work.
No, longer reply following.
I haven't found any indication that it uses the other microphone to do noise reduction for the headset. It wouldn’t make much sense either as it would most likely just try to cancel out with the noise from your pocket.
For most other android phones and for headset on the Nexus One I'm pretty sure that there is only some sort of filter to reduce input of sound that is not speech.
I have done some research on this that I tried to get some help with on the android porting and dev lists. There is a little further info:
http://groups.google.com/group/android-porting/browse_thread/thread/fe1b92065b75c6da?pli=1
With the reservation that I haven't looked at the latest and greatest versions of android.
Related
Background
Phone recording is not really supported on Android, yet some devices support it to some extend.
This made various call recording apps gather as much possible information about devices and what should be done to them, and decide upon this what to do.
Some even offer root solutions.
One such example is boldbeast Call Recorder app, which offers a lot of various configurations to change:
"record mode" . Shows 14 modes for non-rooted devices, and up to 34 for rooted. Also shows "Alsa mode" as an option for it, for rooted devices.
Has "Tune Audio Effect ("auto tune a groupd of parameters") .
Has "Tune Audio Route", with the possible values of "Disabled", "Group1", "Group2", "Group3"
For rooted devices:
"change audio controls" ("auto change audio controls")
"change audio driver" (change audio drive settings to enable record mode 21,22,23,24,31,32,33,34")
For rooted devices: "start input stream"
The problem
If I'm in need to create a call recording app, there is no other way than to find the various workarounds for various devices, but as it seems other apps use terms that don't appear in the API.
I can't find any of those of the app I've mentioned, for example.
What I've found
Other than tons of questions of how to record calls on Android, showing that it doesn't work on all devices, I could find some interesting things. Here are my tries and insights so far:
There are some Audio recording sources we can use while preparing the recording (docs here) , but sadly in each device it might be different. For some, VOICE_CALL works, and for some, others. But at least we can try...
On OnePlus 2 with Android 6.0.1, incoming calls can be recorded using VOICE_CALL, but I can't make outgoing calls be recorded there, unless I use MIC as audio source together with speaker turned on. Somehow, the app I've mentioned succeeds recording it without any issues. I'm sure I will see other issues with other Android devices, as I've tried to address this whole topic in the past. Update: I've found this sample project (also here), which for some reason sleeps for 2 seconds on the UI thread between prepare and start calls of the mediaRecorder. It works fine, and when I did something similar (wait using Handler.postDelayed for 1 second), it worked fine too. The comment that was written there is "Sometimes prepare takes some time to complete".
On Galaxy S7 with Android 8, I've failed to get sound of the other side for outgoing calls AND incoming calls (even with MIC and speaker), no matter what I did, yet the app I've mentioned worked fine.
To let you try my POC of call recording, I've published an open source github repository here, having a sample that will record a single call, and let you listen to the most recent one, if all works well.
This "ViktorDegtyarev - CallRecLib" SDK , which doesn't seem to work at all, and crashes on various Android versions
These 2 old sample projects : rvoix , esnyder-callrecorder , both fail to actually record. The second doesn't even seem to work on Android 6.0.1 device, which it's supposed to support.
aykuttasil - CallRecorder sample and axet - android-call-recorder sample - both, just like on my POC, don't have any tweaking except for AudioSource, and because of this they fails to record on some cases, such as OnePlus 2 output-audio of outgoing calls.
Most third party apps only offer the AudioSource tweaking, but some (like "boldbeast") do offer more. One example is "Automatic Call Recorder" which has "configuration" (10 values to choose from, first is "default") and "method" (5 vales to choose from, first is "default"). Those apps probably do not want others to understand what those configurations mean, so they put general names. Or, it's just too complicated for everyone (especially for users), so they generalize the names.
There is an API of "setMode" here, but it doesn't seem to change upon calling it. I was thinking of maybe change the "channel" of where the call is being used, this way, but it doesn't work. It stays on the value of "2" during call, which is MODE_IN_CALL.
There are customized parameters that are available for various devices (each OEM and its own parameters), which can be set here and maybe even via JNI (here and here) , but I don't get where to get this information from (meaning which pairs of key-value are available). I've searched in a lot of places, but couldn't find any website that talks about which possible parameters are available, and for which devices.
I was thinking of using AudioRecord instead of MediaRecorder class for recording, thinking that it's a bit low level, so it could give me more power and access to customized capabilities, but it seems to be very similar to MediaRecorder, and even use the same audio sources (example here).
Another try I had with low level API, was even further, of using JNI (OpenSL ES for Android). For this, I couldn't find much information (except here and here), and only found the 2 samples of Google here (called "audio echo" and "native audio"), which are not about recording sound, or at least I don't see them occur.
Android P might have official way to record calls (read here and here). Testing on my Android P DP3 device (Pixel 2), I could record both sides fine in both incoming and outgoing calls, using "DEFAULT" as audio source, so maybe the API will finally be official and work on all Android versions. I wrote about it here and here.
I was thinking that maybe the Visualizer class could be a workaround of recording, but according to some StackOverflow post (here), the quality it extremely low, so I decided that maybe I shouldn't try it. Plus I couldn't find a sample of how to record from it.
I've found some parameters that might be available on some devices, here (found from here), all start with "AUDIO_PARAMETER_", but testing on Galaxy S7, all returned empty string. I've also found this website, that gave me the idea of using audioManager.setParameters("noise_suppression=off") together with MIC audio source, but this didn't seem to do anything in the case of Galaxy S7.
The questions
As opposed to other similar questions about this topic, I'm not asking how to record calls. I already know it's a very problematic and complex problem. I already know I will have to address various configurations, and that I will probably use a server to store all of them and find there the best match for each one.
What I want to ask is more about the tweaking and workarounds :
Is there a list of configurations for the various devices, Android versions, and what to choose for each?
Besides Audio source, which other configuration is possible to be used?
Which parameters are possible for the various devices and Android versions ? Are there any websites of the OEMs describing them?
What are the various terms in the app I've mentioned? Where can I find information of how to change them?
Which tools are available for rooted devices?
Is it possible to know which device supports call recording and which not, by using the API ?
About the workaround of OnePlus 2, to wait a moment till we start recording, why is it needed? Is it needed on all Android versions? Is it a known issue? Would 1 second be enough?
How come on the Galaxy S7 I've failed to record the other side even when using MIC&speaker?
EDIT: I've found this of accessibility service being able to help with call recording:
https://developer.android.com/guide/topics/media/sharing-audio-input#voice_call_ordinary_app
Not sure how to use it though. It seems "ACR Phone Dialer" uses it. If anyone knows how it can be done, please let me know.
I spent many weeks working on a Voicecall Recording App so I faced all your issues/questions/problems.
Moreover: my project had a low-priority so I didn't spent much time every day on it, so I worked on this App for many months while Android was changing under the hood (minor an major releases).
I was developing always on the same Galaxy Note 5 using its stock ROM (without Root) but I discovered that on the same device the behaviour was changing from one Android release to another without any explanation.
For example from Nougat 7.0 to 7.1.2 I was unable to record a voicecall using the same code as before.
Google has enforced_or_changed restrictions about voicecall recording many times.
At the beginning it was sufficient to use use VOICE_CALL AudioSource. Then manufactures has started to interprete this Value as they wanted, and the result was that one implementation was working well but another was not.
Then Reflection was needed to run undocumented/hidden methods to start voicecall recording.
Then Google has added a Runtime check, so calling them directly was not more possible even using Reflection.
However this method lack of stability because it was not guarantee that a method was using the same name on all devices.
Then I started to reverse-engineer currently working Apps that were working on newer Android version and I discovered that them were using a complete different and more secure approach. This takes me many weeks because all these Apps uses JNI Libraries trying to hide this method between Assembler code.
When I succesfully create a Test App which was recording well I tried the SAME code in many different devices and ROMs/Versions and surprisely it was working well.
This means that all those different methods you can see in these App Settings (I'm 98% sure about it) are just "fake" or just refers to OLD methods not more used.
A small different metion should be done for Rooted devices:
these devices could change AudioRoutes so a different approach can be used in this case.
[1] There isn't any list or website listing all supported devices or best method to do a successfully voicecall record
[6] It's not possibile to know which device supports Voicecall Recording
just using an API call. You have to try and catch Excepions...
[8] Recording by MIC+speaker suffers of many issues: (1) the caller will hear all your ambient sound so the privacy-bug is a big issue (2) the echo is a big problem (3) the recording volume is very low as the quality of recordered voice
According to my tests, one way to improve this is to have an AccessibilityService being active (no need to write there anything at all) while choosing voice-recognition as the audio source. Also it's recommended to have the speaker turned on because this will record the audio from the microphone.
This seems to exist in some call-recording apps.
Weird thing is that Google has written this as a rule on the Play Store:
The Accessibility API is not designed and cannot be requested for
remote call audio recording.
https://support.google.com/googleplay/android-developer/answer/11899428
No idea what the "remote" means here.
Anyway, I've updated the Github repository to include these additions.
I want to connect speaker output to mic input on Android phone without using any physical cable. Such routing should be transparent to all apps. With this function I can design a bunch of interesting apps, e.g., measuring round-trip delay of a voip call, automatically replying to an incoming call etc.
Some similar questions can be found here:
Background music for call
Outputting audio stream into microphone
It seems that there is no solution on Android right now, but the tool JACK running on Linux is the closet one. Can someone provide me some scripts or codes to create a JACK client that realizes this (speaker -> mic) on Linux? And how hard would it be to port JACK server and client to Android?
Thanks!
Getting Jack on Linux to connect the mic to the speakers:
Start Jack with QJackCtl. This is not as easy as it sounds:
Start QJackCtl.
Click the "Setup..." button.
Choose "alsa" from the "Driver:" dropdown.
Choose your soundcard/soundcards that you want to work with from the "Input Device" and "Output Device" dropdowns. Note that if you have a relatively capable soundcard, there may be a number of possible devices for it (my Audigy 2 ZS has four). Choosing the correct one is beyond the scope of this tutorial.
Check the latency in the bottom-right corner. Realize that Jack itself will take some time to process the sound. This gives you information about that (and I'm sorry, but I don't know precisely what it means). Running Jack is often a tradeoff between getting a super low latency and avoiding "xruns", which are where Jack has to give up on processing a section of audio. Xruns often result in a "click" sound.
Click "OK."
Click the "Connect" button.
This opens the "Connections" panel. This is basically a place that you can plug different Jack applications into one another so they process audio from each other.
Choose the input ("system" on the left) and the output ("system" on the right) and click "Connect". You may need to look through the various things listed under "system" on one or both sides to find the thing that's actually capturing from your mic/outputting to your speakers, in which case . Again, this is beyond the scope of this tutorial; Google is your friend, as is trial and error.
Enjoy the noise of feedback when you bring the microphone near the speakers.
As far as porting Jack to Android goes, you might take a look at this, which basically seems to indicate that Android isn't able to handle the low-latency requirements of Jack (which might also cause you problems in trying to do latency measurements). You could also look at this and this, which might offer an alternative way to do what you're trying to do (I'm not sure where it stands regarding the latency issues, also, I've never used either PD or libpd myself).
I want to control the aperture, shutter speed and ISO on my android phone. Is there a way in which I can access the hardware features?
I won't say it's impossible to do this, but it IS effectively impossible to do it in a way that's generalizable to all -- or even many -- Android phones. If you stray from the official path defined by the Android API, you're pretty much on your own, and this is basically an embedded hardware development project.
Let's start with the basics: you need a schematic of the camera subsystem and datasheets for everything in the image pipeline. For every phone you intend to support. In some cases, you might find a few phones with more or less identical camera subsystems (particularly when you're talking about slightly-different carrier-specific models sold in the US), and occasionally you might get lucky enough to have a lot of similarity between the phone you care about and a Nexus phone.
This is no small feat. As far as I know, not even NEXUS phones have official schematics released. Popular phones (especially Samsung and HTC) usually get teardowns published, so everyone knows the broad details (camera module, video-encoding chipset, etc), but there's still a lot of guesswork involved in figuring out how it's all wired together.
Make no mistake -- this isn't casual hacking territory. If terms like I2C, SPI, MMC, and iDCT mean nothing to you, you aren't likely to get very far. If you don't literally understand how CMOS image sensors are read serially, and how bayer arrays are used to produce RGB images, you're almost certainly in over your head.
That doesn't mean you should throw in the towel and give up... but it DOES mean that trying to hack the camera on a commercial Android phone probably isn't the best place to start. There's a lot of background knowledge you're going to need in order to pull off a project like this, and you really need to acquire that knowledge from a hardware platform that YOU control & have proper documentation for. Make no mistake... on the hierarchy of "hard" Android software projects, this ranks pretty close to the top of the list.
My suggestion (simplified and condensed a bit): buy a Raspberry Pi, and learn how to light up a LED from a GPIO pin. Then learn how to selectively light up 8 LEDs through an 74HC595 shift register. Then buy a SPI-addressed flash chip on a breakout board, and learn how to write to it. At some point, buy a video image sensor with "serial" (fyi, "serial" != "rs232") interface from somebody like Sparkfun.com & learn how to read it one frame at a time, and dump the raw RGB data to flash. Learn how to use i2c to read and write the camera's control registers. At this point, you MIGHT be ready to tackle the camera in an Android phone for single photos.
If you're determined to start with an Android phone, at least stick to "Nexus" devices for now, and don't buy the phone (if you don't already own it) until you have the schematics, datasheets, and sourcecode in your possession. Don't buy the phone thinking you'll be able to trace the schematic yourself. You won't. At least, not unless you're a grad student and have one hell of a graduate-level electronics lab (with X-Ray capabilities) at your disposal. Most of these chips and modules are micro-BGA. You aren't going to trace them with a multimeter, and every Android camera I'm aware of has most of its low-level driver logic hidden in loadable kernel modules whose source isn't available.
That said, I'd dearly love to see somebody pull a project like this off. :-)
Android has published online training which contain all the information you need:
You can find it here - Media APIs
However, there are limitations, not all hardware's support all kind of parameters.
And if I recall correctly, you can't control the shutter speed and ISO.
Many tablets and some smart phones use an array of microphone for things like noise cancellation. For example Motorola Droid X uses three microphone arrays and even allows you to set "audio scenes". An example is discussed here.
I want to be able to record from all the microphones that are available on the tablet/phone at the same time. I found that using AudioSource we can choose the mic (I do not know which mic this is specifically but it might be the one facing the user) or the mic that is in same orientation as the video camera, but could not find anyway of accessing all the other mic in the mic array. Any help that points me in the right direction to investigate this will be great. Thanks in advance for your time.
It's seems like you've verified that there isn't a standard Android API for accessing specific mics in an array. I couldn't find anything either.
As is the case with custom additions to the Android system, it's up to the manufacturer to release developer APIs. Motorola has done this before. I took a look at all of the ones they have listed and it seems they simply don't expose it. Obviously, they have code somewhere which can do it (the "audio scenes" uses it).
So the quick answer: you're out of luck.
The more involved answer: you can go spelunking around the source code for the Droid X because it's released as open source. If you can actually find it, understand that you're using an undocumented API which could be changed at any time. Plus, you'll have to do this for every device you want to support.
I am taking this crazy class on Moble Programming. We have to do a final project and I would like to do some sort simple guitar processor app.
I wanted to do this in IOS, but it seems like the learning curve for IOS is to impractical for a short class.
No offense to anyone but Droid is easier to program, at least to me, but I am confused if you can even get guitar input from a jack (not mic) and then do some processing on the input and feed it to the output.
I'm aware of latency, which may or may not be a big deal for a class.
Does anyone know if Droid can do anything like this? If so any articles or somewhere to start? I know with IOS you can at least buy a jack and it seems to have tons of open source processing code, but I can't seem to find anything for Droid. All I have seen is "Ghetto Amp" for guitar stuff.
Any ideas?
Thanks
You may want to look at this project:
http://code.google.com/p/moonblink/wiki/Audalyzer
should be pretty useful :)
However the core class you will be using to pick up and look at audio streams is: http://developer.android.com/reference/android/net/rtp/AudioStream.html
I wrote a MIDI guitar for a college project a long time ago, in assembly for a Texas Instruments DSP. As long as you just played exactly one note, and were really careful about it, it could tell what you'd played.
Not much amplification was needed. In fact, I could get some notes even on an unamplified signal. I had oscilloscopes and a pretty generalized ADC to work with, you might have to amplify the signal...but if you do, be careful not to fry your input. Start low...and really, the more you can read up on the tolerances the better.
Looks like they never made any hi-fi micro-USB 24-bit ADCs or wrote drivers for them. I guess there's no market. :) But if you're doing a school project and not producing the latest Muse album, get a path from your guitar to the headset line in:
http://androidforums.com/android-media/194740-questions-about-audio-recording-droid.html
I'd probably just sacrifice a cheap or broken headset to get the headset plug. ( Maybe they sell appropriate tips at Radio Shack but I've learned not to assume such things anymore :-/ ) After building a cable I'd I'd feed it an amplified signal from the guitar so I could control the gain level to whatever I wanted.
Depending on latency requirements you can use Java or NDK. Note this answer:
Need help about sound processing
(I have one of the original Droids sitting around in a drawer, I'm sure I could use it for something but I just haven't figured out what!)