I am getting this issue where I am recording the PCM_16_bit audio, with sampling rate of 44.1 kHz.
The raw files that I am getting in the Lollipop and Marshmallow versions of Android are not the same (I compared the spectrum of the two files using Audacity)
I am running the same app in two different versions of Android OS.
Both the devices are of the same model (Lenovo A7000).
What to do?
Are your AudioSource.MIC or do you record from a file etc?
If you are using AudioSource.MIC you would need to be in a very controlled environment to get the same results, I think you'd need to be in a isolated box and have 100% control over voice activity to even get the same results on two recordings on the same OS.
If you are using any other source, like a file/inputstream, you should probably notify android about the issue at the issue tracker. (though this is probably already fixed, after a year?)
I'm working on a Android app that uses MediaCodec to encode H.264 video using the Surface method. I am targeting Android 5.0 and I've followed all the examples and samples from bigflake.com (I started working on this project two years ago, so I kind of went through all the gotchas and other issues).
All is working nice and well on Nexus 6 (which uses the Qualcomm hardware encoder for doing this), and I'm able to record flawlessly in real-time 1440p video with AAC audio, in a multitude of outputs (from MP4 local files, upto http streaming).
But when I try to use the app on a Sony Android TV (running Android 5.1) which uses a Mediatek chipset, all hell breaks loose even from the encoding level. To be more specific:
It's basically impossible to make the hardware encoder work properly (that is, "OMX.MTK.VIDEO.ENCODER.AVC"). With the most basic setup (which succeeds at MediaCodec's level), I will almsot never get output buffers out of it, only weird, spammy, logcat error messages stating that the driver has encountered errors each time a frame should be encoded, like this:
01-20 05:04:30.575 1096-10598/? E/venc_omx_lib: VENC_DrvInit failed(-1)!
01-20 05:04:30.575 1096-10598/? E/MtkOmxVenc: [ERROR] cannot set param
01-20 05:04:30.575 1096-10598/? E/MtkOmxVenc: [ERROR] EncSettingH264Enc fail
Sometimes, trying to configure it to encode at a 360 by 640 pixels resolution will succeed in making the encoder actually encode stuff, but the first problem I'll notice is that it will only create one keyframe, that is, the first video frame. After that, no more keyframes are ever, ever created, only P-frames. Ofcourse, the i-frame-interval was set to a decent value and is working with no issues on other devices. Needless to say, this makes it impossible to create seekable MP4 files, or any kind of streamable solution on top.
Most of the times, after releasing the encoder, logcat will start spamming endlessly with "Waiting for input frame to be released..." which basically requires a reboot of the device, since nothing will work from that point on anyway.
In the case where it doesn;t go havoc after a simple release(), no problem - the hardware encoder is making sure that it cannot be created a second time, and it falls back to the generic SOFTWARE avc google encoder. hich ofcourse is basically a mockup encoder which does little to nothing than spit out an error when trying to make it encode anything larger than 160p videos...
So, my question is: is there any hope of making this MediaCodec API actually work on such a device? My understanding was that there are some CTS tests performed by Google/manufacturers (in this case, Sony) that would allow a developer to actually think that an API is supported on a device which prouds itself as running Android 5.1. Am I missing something obvious here? Did anyone actually ever tried doing this (a simple MediaCodec video encoding test) and succeeded? It's really frustrating!
PS: it's worth mentioning that not even Sony provides yet a recording capability for this TV set, which many people are complaining anyway. So, my guess is that this sounds more like a Mediatek problem, but still, what exactly are the Android's CTS for in this case anyway?
I'm building an internal shared library for Android platform. I have the signing keystore from the device manufacturer.
My library is making use of ScreenRecord.cpp internal file from Android source. Recording works fine with the MediaCodec encoder; however I want to access each frame so that I can apply some image overlay logo on to each frame before it gets passed to the encoder.
There's an overlay example in Android source too, but that only works for newer versions of Android (5.0 / API 21+). I want to have an overlay solution for Android Kitkat (4.4 / API 19)
Here's a code example that I obtained from minicap.
mVirtualDisplay = android::SurfaceComposerClient::createDisplay(
android::String8("minicap"),
true);
LOGI("Creating buffer queue");
mScreenshotClient.getCpuConsumer();
mBufferQueue = mScreenshotClient.mBufferQueue;
LOGI("Creating CPU consumer");
mConsumer = new android::CpuConsumer(mBufferQueue, 3, false);
mConsumer->setName(android::String8("minicap"));
mConsumer->setDefaultBufferSize(targetWidth, targetHeight);
mConsumer->setDefaultBufferFormat(android::PIXEL_FORMAT_RGBA_8888);
mConsumer->setFrameAvailableListener(mFrameProxy);
//mFrameProxy is from:
//class FrameProxy: public android::ConsumerBase::FrameAvailableListener
LOGI("Publishing virtual display");
android::SurfaceComposerClient::openGlobalTransaction();
android::SurfaceComposerClient::setDisplaySurface(mVirtualDisplay, mBufferQueue);
android::SurfaceComposerClient::setDisplayProjection(mVirtualDisplay,
android::DISPLAY_ORIENTATION_0, layerStackRect, visibleRect);
android::SurfaceComposerClient::setDisplayLayerStack(mVirtualDisplay, 0);// default stack
android::SurfaceComposerClient::closeGlobalTransaction();
I set up the above code, but onFrameAvailable() method of FrameAvailableListener gets called only once. It never gets called again even when I do stuff on the screen. What am I missing here?
Isn't there any lesser tricky way to access the frames before passing to the encoder?
An example of adding an overlay is built into the screenrecord sources for Lollipop. As far as I can recall it doesn't rely on any features added in Lollipop, so you should be able to build and run it on 4.4. As noted on bigflake, the --bugreport mode was added to AOSP in the 4.4 time frame, but didn't actually ship with the system until 5.x. (With a minor tweak, it should even run on 4.3, but I haven't tried it.)
The key source files are Overlay.{cpp,h}. It does the same thing that you would do from code written in Java: create a GLConsumer (SurfaceTexture), use that to convert the incoming frames to GLES textures, then render texture + overlay to the video encoder.
Sample video is here. Note it adds a block of text to the very start, and a running timestamp / frame-counter to the top left corner.
Note for anyone else reading this: this code is using internal private APIs that have been changing in recent releases, so any binaries must be built for specific versions of Android, and may not be portable to devices built by different manufacturers even if they run the same version of Android (sometimes the OEMs like to mess with things).
Update: My earlier statements about working on KitKat weren't accurate -- there was a major API shift before the Lollipop version went out. The trick is to grab the sources before this change went in, as that was when the BufferQueue API rewrite reached screenrecord. You can see from the change list that the --bugreport option went in about five months before that.
I'm writing an Android application that uses a Buffer Queue Audio Player via OpenSL ES. My application runs just fine on recent devices but I'm having troubles with the HTC Wildfire S.
In particular, calling the slCreateEngine functions produces a SL_RESULT_RESOURCE_ERROR. There isn't a lot of information in the documentation about this error.
Right before that call to slCreateEngine, I am also seeing these errors in LogCat:
Failed to open MM_PARSER_LIB, dlerror = Cannot load library: load_library[1105]: Library 'libmmparser.so' not found
Failed to open MM_PARSER_LITE_LIB, dlerror = Cannot load library: load_library[1105]: Library 'libmmparser_lite.so' not found
Some theories I have:
The device is too memory-constrained to instantiate the engine object
The device does not support OpenSL ES audio interface (this thread
suggests that not all devices support it:
https://groups.google.com/forum/#!topic/android-ndk/Px-7NvaLmjo)
The specs on the HTC Wildfire S are as follows:
Android 2.3.4 (as far as I can take it through carrier updates)
Qualcomm MSM7227 (ARMv6 processor)
512MB RAM
I know that Open SL ES is supported as of Gingerbread, but that doesn't mean all devices have the capability. Due to other requirements for this application, I must handle by audio processing and playback with NDK and cannot use Media Player or AudioTrack.
Questions:
Does anyone know what could be causing the reported error on this
device?
Is there a way of figuring out what devices are compatible with OpenSL ES?
Are ARMv6 devices necessarily incompatible with OpenSL ES?
EDIT
See my comments to HerrLip
NativeAudio example from the NDK folder runs on this device and slCreateEngine is executed successfully. This eliminates my suspicions that the device and other ARMv6 devices are not supported.
The libmmparser.so and libmmparser_lite.so libraries still cannot be loaded in the NativeAudio example, but this does not seem to be the problem as it works on the device.
My application has quite a bit more going on than the NativeAudio example. Perhaps memory constraints are preventing the slCreateEngine function call from getting the required resources.
According to the docs, that error code means "Operation failed due to a lack of resources (usually a result of object realization)." So, I'd assume you're right about constraints.
You could try somebody else's app and see what happens. (Try https://code.google.com/p/high-performance-audio/source/browse/audio-buffer-size )
I don't know
I'm currently attempting to minimize audio latency for a simple application:
I have a video on a PC, and I'm transmitting the video's audio through RTP to a mobile client. With a very similar buffering algorithm, I can achieve 90ms of latency on iOS, but a dreadful ±180ms on Android.
I'm guessing the difference stems from the well-known latency issues on Android.
However, after reading around for a bit, I came upon this article, which states that:
Low-latency audio is available since Android 4.1/4.2 in certain devices.
Low-latency audio can be achieved using libpd, which is Pure Data library for Android.
I have 2 questions, directly related to those 2 statements:
Where can I find more information on the new low-latency audio in Jellybean? This is all I can find but it's sorely lacking in specific information. Should the changes be transparent to me, or is there some new class/API calls I should be implementing for me to notice any changes in my application? I'm using the AudioTrack API, and I'm not even sure if it should reap benefits from this improvement or if I should be looking into some other mechanism for audio playback.
Should I look into using libpd? It seems to me like it's the only chance I have of achieving lower latencies, but since I've always thought of PD as an audio synthesis utility, is it really suited for a project that just grabs frames from a network stream and plays them back? I'm not really doing any synthesizing. Am I following the wrong trail?
As an additional note, before someone mentions OpenSL ES, this article makes it quite clear that no improvements in latency should be expected from using it:
"As OpenSL ES is a native C API, non-Dalvik application threads which
call OpenSL ES have no Dalvik-related overhead such as garbage
collection pauses. However, there is no additional performance benefit
to the use of OpenSL ES other than this. In particular, use of OpenSL
ES does not result in lower audio latency, higher scheduling priority,
etc. than what the platform generally provides."
For lowest latency on Android as of version 4.2.2, you should do the following, ordered from least to most obvious:
Pick a device that supports FEATURE_AUDIO_PRO if possible, or FEATURE_AUDIO_LOW_LATENCY if not. ("Low latency" is 50ms one way; pro is <20ms round trip.)
Use OpenSL. The Dalvik GC has a low amortized cost, but when it runs it takes more time than a low-latency audio thread can allow.
Process audio in a buffer queue callback. The system runs buffer queue callbacks in a thread that has more favorable scheduling than normal user-mode threads.
Make your buffer size a multiple of AudioManager.getProperty(PROPERTY_OUTPUT_FRAMES_PER_BUFFER). Otherwise your callback will occasionally get two calls per timeslice rather than one. Unless your CPU usage is really light, this will probably end up glitching. (On Android M, it is very important to use EXACTLY the system buffer size, due to a bug in the buffer handling code.)
Use the sample rate provided by AudioManager.getProperty(PROPERTY_OUTPUT_SAMPLE_RATE). Otherwise your buffers take a detour through the system resampler.
Never make a syscall or lock a synchronization object inside the buffer callback. If you must synchronize, use a lock-free structure. For best results, use a completely wait-free structure such as a single-reader single-writer ring buffer. Loads of developers get this wrong and end up with glitches that are unpredictable and hard to debug.
Use vector instructions such as NEON, SSE, or whatever the equivalent instruction set is on your target processor.
Test and measure your code. Track how long it takes to run--and remember that you need to know the worst-case performance, not the average, because the worst case is what causes the glitches. And be conservative. You already know that if it takes more time to process your audio than it does to play it, you'll never get low latency. But on Android this is even more important, because the CPU frequency fluctuates so much. You can use perhaps 60-70% of CPU for audio, but keep in mind that this will change as the device gets hotter or cooler, or as the wifi or LTE radios start and stop, and so on.
Low-latency audio is no longer a new feature for Android, but it still requires device-specific changes in the hardware, drivers, kernel, and framework to pull off. This means that there's a lot of variation in the latency you can expect from different devices, and given how many different price points Android phones sell at, there probably will always be differences. Look for FEATURE_AUDIO_PRO or FEATURE_AUDIO_LOW_LATENCY to identify devices that meet the latency criteria your app requires.
From the link at your point 1:
"Low-latency audio
Android 4.2 improves support for low-latency audio playback, starting
from the improvements made in Android 4.1 release for audio output
latency using OpenSL ES, Soundpool and tone generator APIs. These
improvements depend on hardware support — devices that offer these
low-latency audio features can advertise their support to apps through
a hardware feature constant."
Your citation in complete form:
"Performance
As OpenSL ES is a native C API, non-Dalvik application threads which
call OpenSL ES have no Dalvik-related overhead such as garbage
collection pauses. However, there is no additional performance benefit
to the use of OpenSL ES other than this. In particular, use of OpenSL
ES does not result in lower audio latency, higher scheduling priority,
etc. than what the platform generally provides. On the other hand, as
the Android platform and specific device implementations continue to
evolve, an OpenSL ES application can expect to benefit from any future
system performance improvements."
So, the api to comunicate with drivers and then hw is OpenSl (in the same fashion Opengl does with graphics). The earlier versions of Android have a bad design in drivers and/or hw, though. These problems were addressed and corrected with 4.1 and 4.2 versions, so if the hd have the power, you get low latency using OpenSL.
Again, from this note from the puredata library website, is evident that the library uses OpenSL itself to achieve low latency:
Low latency support for compliant devices
The latest version of Pd for
Android (as of 12/28/2012) supports low-latency audio for compliant
Android devices. When updating your copy, make sure to pull the latest
version of both pd-for-android and the libpd submodule from GitHub.
At the time of writing, Galaxy Nexus, Nexus 4, and Nexus 10 provide a
low-latency track for audio output. In order to hit the low-latency
track, an app must use OpenSL, and it must operate at the correct
sample rate and buffer size. Those parameters are device dependent
(Galaxy Nexus and Nexus 10 operate at 44100Hz, while Nexus 4 operates
at 48000Hz; the buffer size is different for each device).
As is its wont, Pd for Android papers over all those complexities as
much as possible, providing access to the new low-latency features
when available while remaining backward compatible with earlier
versions of Android. Under the hood, the audio components of Pd for
Android will use OpenSL on Android 2.3 and later, while falling back
on the old AudioTrack/AudioRecord API in Java on Android 2.2 and
earlier.
When using OpenSL ES you should fulfil the following requirements to get low latency output on Jellybean and later versions of Android:
The audio should be mono or stereo, linear PCM.
The audio sample rate should be the same same sample rate as the output's native rate (this might not actually be required on some devices, because the FastMixer is capable of resampling if the vendor configures it to do so. But in my tests I got very noticeable artifacts when upsampling from 44.1 to 48 kHz in the FastMixer).
Your BufferQueue should have at least 2 buffers. (This requirement has since been relaxed. See this commit by Glenn Kasten. I'm not sure in which Android version this first appeared, but a guess would be 4.4).
You can't use certain effects (e.g. Reverb, Bass Boost, Equalization, Virtualization, ...).
The SoundPool class will also attempt to make use of fast AudioTracks internally when possible (the same criteria as above apply, except for the BufferQueue part).
Those of you more interested in Android’s 10 Millisecond Problem ie low latency audio on Android. We at Superpowered created the Android Audio Path Latency Explainer. Please see here:
http://superpowered.com/androidaudiopathlatency/#axzz3fDHsEe56
Another database of audio latencies and buffer sizes used:
http://superpowered.com/latency/#table
Source code:
https://github.com/superpoweredSDK/SuperpoweredLatency
There is a new C++ Library Oboe which help with reducing Audio Latency. I have used it in my projects and it works good.
It has this features which help in reducing audio latency:
Automatic latency tuning
Chooses the audio API (OpenSL ES on API 16+ or AAudio on API 27+)
Application for measuring sampleRate and bufferSize: https://code.google.com/p/high-performance-audio/source/checkout and http://audiobuffersize.appspot.com/ DB of results