I'm using mp3 files to play sounds in my Android game, developed in Libgdx. The sounds play fine when they happen every now and then, but when I play them fast (footsteps in a running animation for example) the game freezes/stutters.
Every time a sound is played, I get this in the logs:
W/AudioTrack: AUDIO_OUTPUT_FLAG_FAST denied by client; transfer 4, track 44100 Hz, output 48000 Hz
I use libktx AssetStorage to store the sounds. I've been searching for this issue for a few days now and haven't gotten luck with any of the following solutions:
Override createAudio in AndroidLauncher and use AsynchronousAndroidAudio
Convert mp3 to ogg (using Audacity)
Convert to 48k rate sample (using Audacity)
Add 1 or seconds of silence to the file
I test it on my own device, Samsung Galaxy S5, which is quite old and has version Android 6.0.1.
What can I do to resolve this error and stuttering?
Decoding compressed audio can be a significant processing load. If it's a short recording (e.g., one footstep that is being repeated), I'd either package the sound file as a .wav or decode it into PCM to be held in memory, and use it that way. IDK if it's possible to output PCM directly with libgdx, though, but I do recall inspecting and tinkering with an ogg utility to have it decode into an array, and outputting it with a SourceDataLine for a non-libgdx Java project. I realize that SourceDataLine output is not an option with Android, but Android does have provisions for playing back raw PCM.
Another idea to explore is raising the priority of the thread that is processing the audio to Thread.MAX_PRIORITY if libgdx allows this. Theoretically, the audio processing thread spends most of its time in a blocked state, so doing this shouldn't hurt the performance, unless you are really going overboard with your audio requests.
I just saw the mismatch of sample rates. It's wasteful to repeatedly do conversions on the fly when you can do the conversion once in Audacity. I'm guessing the difference between outputting at 48000 vs 44100 isn't that big of a load difference. Seems to me 44100 should be fine, but I doubt using 48000 for everything adds much in terms of cpu load (or perceivable audio fidelity). So, whichever one you pick, spend a little time making sure all the assets match the format.
Related
I have troubles in my current project which requires video processing. Basically crop function (video should be squared), trimming (video shouldn't be longer than 30 seconds) and quality reduction (bitrate should be equal 713K).
I've succesfully embedded FFmpeg into application, all functions are working quite fine except one major detail - processing as per my boss is taking too long time. For video that have around 52 MB for 36 seconds it's taking 50 seconds to perforn all the operations (I'm trimming video to 30 seconds before any other operation obviously). The problem is that on parallel project on iOS video processing takes like 10-15 seconds for greater files. I assume that it's related to fact that they're using Apple QuickTime format which obviusly was developed by Apple so it's not surprising that it's working quite fast.
So well, it was introduction, now my question: is there any way for Android to process any video in any quality (for now we can assume that all videos are in h264) in time of 10-15 seconds (not more then in 30 seconds, as my boss said)? Some alternative to FFmpeg, that can perform operations faster? I'nm pretty sure that there is no possibility to perform such work in a such short time, since I already feel like I searched thought while Internet, but I want to make sure that there is really no possibility to do such work. If anyone can provide me links to solution more faster than FFmpeg or confirm that there is no such solution, I will be very gratefull.
Update
Thanks to Alex Cohn I've resolved this with MediaCodec. After a while, I got 20 seconds processing on 52MB video with cropping to square and lowering bitrate. For any future Googlers out of here I can suggest to take a look at this respository:
Many stuff about MediaCodec
and more precisely at this file: Extract, edit and encode again, video and audio
If the video has been recorded on the same device, you have a very good chance that MediaCodec and native Android media APIs will be much faster (running both decoder and encoder in HW). Otherwise, you can try to decode the video with MediaCodec, and fall back to FFmpeg software decoder if it fails. Even then, if you can use MediaCodec for compression, this alone may deliver performance that will satisfy your boss.
There exists a (deprecated) project called libstagefright that builds FFmpeg with support for the hardware codec, and it was designed to work on API 10+.
Don't forget to compare the CPU characteristics of your Android device and the iOS platform (if that's a 6S, they have a significant fore). Consider multithreaded encoding and decoding.
BTW: Note that FFmpeg does not come with H264 encoder, and the typical bundle of FFmpeg+x264 is GPL, and requires all your app to be open sourced (or pay a hefty license fee for x264, but still be forced to comply with LGPL of FFmpeg).
Note that you can make square video by manipulating the MP4 headers only, without transcoding!
So I have an app where mp3 file is being played using the MediaPlayer. On most devices everything is fine but on Samsung and some other (like HTC One S) devices the same mp3 plays "too fast" (skipping gaps): looks like player does not handle sound gaps (silence) correctly. These mp3s are just speech and speech naturally has gaps (silence) between spoken words. And these gaps are not played correctly in terms of time - MediaPlayer just skips them. As result mp3 is played faster by the duration of all gaps it contains.
What could be a reason and solution for this?UPDATEI'd found that its about frequency+VBR. Somehow if mp3 is of 22050/24000/32000 Hz instead of 44100 or 48000 and VBR or ABR is used the issue raises up. Im using LAME for mp3 encoding. If I remove "--resample 22.05" option so the resulting mp3 becomes 44.1kHz there is no issue playing this mp3 on samsung phone. However the resulting size of mp3 becomes twice bigger which is not acceptable for me cuz in this case my apk becomes bigger than 50Mb. So now the question is how to properly compress mp3 as 22kHz/VBR/MONO.
The issue was fixed in the following way: I added a white noise to an original sound and then encoded it to MP3 format. Resulting files became bigger in size but also they become more compatible (with Samsung devices) The original audio file (made at recording studio) is too clean meaning that silence/pauses in speech (between pronounced words) has no waveform if look in sound editor, its like an ideal silence. So on variuos Samsung devices such MP3-encoded files played with described issue. However on most other devices and PCs such MP3 files played just fine. Once again - Samsung "rules"!
You need to Google our for controlling playback speed in your application I mean to say that there must be some sort of 'playback rate' variable which must be a floating point value something between 0 to 1. This might help you in some workarounds for your app hope you find this somewhat helpful in anyway . O by the way here are some useful links that might help you out as well and if not then we have to keep waiting in the waiting queue for Samsung ;-) if its specifically related to them happy coding
http://code.google.com/p/android/issues/detail?id=1961
play an mp3 with MediaPlayer class on Android issues
Regards
Anas.
I am working on an implementation of one of the Android Test Cases regarding previewTexture recording with the new MediaCodec and MediaMuxer API's of Android 4.3.
I've managed to record the preview stream with a framerate of about 30fps by setting the recordingHint to the camera paremeters.
However, I ran into a delay/lag problem and don't really know how to fix that. When recording the camera preview with quite standard quality settings (1280x720, bitrate of ~8.000.000) the preview and the encoded material suffers from occasional lags. To be more specific: This lag occurs about every 2-3 seconds and takes about 300-600ms.
By tracing the delay I was able to figure out the delay comes from the following line of code in the "drainEncoder" method:
mMuxer.writeSampleData(mTrackIndex, encodedData, mBufferInfo);
This line is called in a loop if the encoder has data available for muxing. Currently I don't record audio so only the h264 streams is converted to a mp4 format by the MediaMuxer.
I don't know if this has something to do with that delay, but it always occurs when the loop needs two iterations to dequeue all available data of the encoder (to be even more specific it occurs always in the first of these two iterations). In most cases one iteration is enough to dequeue the encoder.
Since there is not much information online about these new API's any help is very appreciated!
I suspect you're getting bitten by the MediaMuxer disk write. The best way to be sure is to run systrace during recording and see what's actually happening during the pause. (systrace docs, explanation, bigflake example -- as of right now only the latter is updated for Android 4.3)
If that's the case, you may be able to mitigate the problem by running the MediaMuxer instance on a separate thread, feeding the H.264 data to it through a synchronized queue.
Do these pauses happen regularly, every 5 seconds? The CameraToMpegTest example configures the encoder to output an I-frame every 5 seconds (with an expected frame rate of 30fps), which results in a full-sized frame being output rather than tiny deltas.
As #fadden points out, this is a disk write issue that occurs mostly on devices with lower writing flash speeds or if you try to write to the SD card.
I have written a solution on how to buffer MediaMuxer's write in a similar question here.
I need to be able to play two or more (let's say, up to 5) short ogg files simultaneously. And by simultaneously I mean in perfect synchrony. I am able to load them to SoundPool and play, but this sometimes creates a noticeable difference in playback start time, which I want to get rid of.
From my understanding this can be avoided if mixing PCMs into one buffer and playing. But OGG's are not PCMs and need to be somehow efficiently decoded before playing and latency must be very low, ideally as soon as user presses the button. So I figured I need a way to stream OGG into PCM and as I receive buffers I would mix them and feed to AudioTrack. My requirement is Android 2.3.3+, so I cannot use any new codecs provided in Jelly Bean.
Also although OGGs themselves are small, there is a lot of them. So keeping them all decoded in memory (SoundPool or some pre-decoding) may case problems too.
Can someone give me a tip where to dig? Can OpenSL ES do that for me? Or should I think about integrating ffmpeg? And is it even possible to stream simultaneus files with low latency?
Thanks
You can play sounds using AssetPlayers, but this sometimes creates a noticeable difference in playback start time, yeh...
So, i recomend to decode ogg using Ogg Vorbis (like here) and then using this PCM buffer for BufferPlayer.
Btw, check this OpenSL ES wrappers
https://github.com/Suvitruf/Android-ndk/tree/master/OpenSLES
I'm writing an audio streaming app that buffers AAC file chunks, decodes those chunks to PCM byte arrays, and writes the PCM audio data to AudioTrack. Occasionally, I get the following error when I try to either skip to a different song, call AudioTrack.pause(), or AudioTrack.flush():
obtainbuffer timed out -- is cpu pegged?
And then what happens is that a split second of audio continues to play. I've tried reading a set of AAC files from the sdcard and got the same result. The behavior I'm expecting is that the audio stops immediately. Does anyone know why this happens? I wonder if its an Audio latency issue with Android 2.3.
edit: The AAC audio contains an ADTS Header. The header + audio payload constitute what I'm calling ADTSFrame. These are fed to the decoder one frame at a time. The resulting PCM byte array that gets returned from the C layer to the Java Layer gets fed to Android's AudioTrack API.
edit 2: I got my nexus 7 (Android 4.1 OS) today. Loaded the same APP onto the device. Didn't have any of these problems at all.
it is highly possible about sample rate. one of your devices might be supporting the sample rate u used while the other could not. Please check it. I had the same issue, it was about sample rate. use 44.1kHz (44100) and try again please.