My team and I are nearly done developing a music application for iPhone and Android that allows users to create their own music, built by playing and overlapping sampled sounds (up to 16 at a time). We are looking for a way to allow users to share these songs by embedding an audio player in our website which will (like the Android and iPhone applications already do) take the songs, which are expressed as a string representing pitch, duration, start time, and instrument, and convert them into a single playable audio file (any format).
We have looked into SoundManager 2 and WebAudio, and have run into the same problem with both: stopping sounds creates beeping or popping sounds that cannot be removed. Does anyone know of another framework or API that we should look into? A little googling also made sfplay stand out, but there isn't very much documentation on it. Any other suggestions?
Thanks!
There are still a lot of problems with javascript/html5 audio. WebAudio is very powerful but not very portable. Soundmanager 2 is less powerful, but more portable. You should not be hearing clicks/discontinuities with either library unless you are doing something wrong, but there are problems with synchronization and so on with browser-based audio.
That's why most people doing "serious" audio on the web are using java applets or flash, which won't work on devices.
Related
I'm a biomedical engineering student and I'm doing my Final project on an assistive app for people with sensory issues. I'm also unfortunately not familiar with android apps coding (I have done some C and C++ plus a tiny bit of front-end on JS).
Long story short, I'm looking to analyze every single audio that will be played to the user (through headphones or peripheral speakers) before they are actually played to the user (could be a video playing from any app, games, music etc.) and filter out certain groups of frequencies from it that could cause overload or overstimulation. (the frequencies will be guessed through questions answered by user, the goal is to drastically decrease these frequencies amplitude).
I have found open-source code for the frequencies analyzer both in kotlin and java.
Now my issue is: can I have access to and manipulate other apps audio output? (I have found that the audiostream can be paused or prioritized through audio focus but didn't find an answer for this specific need)
sorry it got long and I thank everyone in advance!
I want to create an Android app that plays multiple mp3s simultaneously, with precise sync (less than 1/10 of a second off) and independent volume control. Size of each mp3 could be over 1MB, run time up to several minutes. My understanding is that MediaPlayer will not do the precise sync, and SoundPool can't handle files over 1MB or 5 seconds run time. I am experimenting with superpowered and may end up using that, but I'm wondering if there's anything simpler, given that I don't need any processing (reverb, flange, etc.), which is superpowered's focus.
Also ran across the YouTube video on Android high-performance audio, from Google I/O 2016. Wondering if anyone has any experience with this.
https://www.youtube.com/watch?v=F2ZDp-eNrh4
Superpowered was originally made for my DJ app (DJ Player in the App Store), where precisely syncing multiple tracks is a requirement.
Therefore, syncing multiple mp3s and independent volume control is definitely possible and core to Superpowered. All you need is the SuperpoweredAdvancedAudioPlayer class for this.
The CrossExample project in the SDK has two players playing in sync.
The built-in audio features in Android are highly device and/or build dependent. You can't get a consistent feature set with those. In general, the audio features of Android are not stable. That's why you need a specialized audio library which does everything "inside" your application (so is not a "wrapper" around Android's audio features).
When you are playing compressed files (AAC, MP3, etc) on Android in most situations they are decoded in hardware to save power, except when the output goes to a USB audio interface. The hardware codec accepts data in big chunks (again, to save power). Since it's not possible to issue a command to start playing multiple streams at once, what will often be happening is that one stream will already send a chunk of compressed audio to hardware codec, and it will start playing, while others haven't yet sent their data.
You really need to decode these files in your app and mix the output to produce a single audio stream. Then you will guarantee the desired synchronization. The built-in mixing facilities are mostly intended to allow multiple apps to use the same sound output, they are not designed for multitrack mixing.
I'm making an app that needs to send a video feed from a single source to a server where it can be accessed by desktop browsers and mobile apps.
So far, I've been using Adobe Media Server 5 with a live RTMP stream. This gives me about a 2.5 second delay on desktop browsers, which gives me no native support for iOS, but leaves me with the option to use Air to export the app for iOS, which produces a minimum 5-6 second delay.
The iOS docs strongly recommend the use of HTTP Live Streaming which segments the stream into chunks and serves it using a dynamic playlist in a .m3u8 file. Doing this produces a 15+ second delay in desktop browsers and mobile devices. A Google search seemed to reveal that this is to be expected from HLS.
I need a maximum of 2-4 second delays across all devices, if possible. I've gotten poor results with Wowza, but am open to revisiting it. FFMpeg seems inefficient, but I'm open to that as well, if someone has had good results with it. Anybody have any suggestions?? Thanks in advance.
I haven't even begun to find the most efficient way to stream to Android, so any help in that department would be much appreciated.
EDIT: Just to be clear, my plan is to make an iOS app, whether it's written natively or in Air. Same goes for Android, but I've yet to start on that.
In the ios browser HLS is the only way to serve live video. The absolute lowest latency would be to use 2 second segments with a 2 segment windows in the manifest. This will give you 4 seconds latency on the client, plus another 2 to 4 on the server. There is no way to do better without writing an app.
15 Second delay for HLS streams is pretty good, to provide lower latency you need to use a different streaming protocol.
RTP/RTSP will give you the lowest latency and is typically used for VoIP and video conferencing, but you will find it very difficult to use over multiple mobile and WiFi networks (some of them unintentionally block RTP).
If you can write an iOS app that supports RTMP then that is the easiest way to go and should work on Android too (only old Androids support Flash/RTMP natively). Decoding in software will result in poor battery life. There are other iOS apps that don't use HLS for streaming, but I think you need to limit it to your service (not a generic video player).
Also please remember that higher latency equals higher video quality, less buffering, better user experience etc. so don't unnecessarily reduce latency.
I've been experimenting with making an android app for the past week or two and have reached a point where I need to decide on a general plan of action concerning the code I'm writing.
started with SoundPool, easy to use, not very flexible. went on to AudioTrack, seems good but slow.
So now I'm looking at the ndk..
Does the ndk have direct access to AudioTrack? or something else?
What is the general concensus on this kind of thing?
A guess is to make the UI in java and the 'sound engine' in C++
I'd like to get started on the right track before I write too much.
edit:
It is a music studio that plays and manipulates wav files from the sdcard as well as realtime sound synthesis.
The Android SDK doesn’t offer a fully featured audio processing solution like Core Audio on iOS. The available features are exposed through OpenSL ES interface. Any audio processing feature's availability is dependent on the device manufacturer’s Android flavor and configuration, so you can not rely on them.
To wit, the infamous Android fragmentation problem is even bigger in audio.
Even worse, if a device reports an audio feature available, it may still not work properly. For example: audio playback rate on a Samsung Galaxy Note III.
Realtime sound synthesis? None of the Android SDK’s features are designed for that.
The best way is doing the UI in Java and dealing with sound in C++.
There are some 'audio engine' offers on the web, but many of them are just “wrappers” around the same thing.
As cofounder of Superpowered, allow me to recommend the use the Superpowered Audio SDK, which is a total audio processing solution designed for real-time and highest performance, without any dependency on Android’s audio offerings, and runs on all.
http://superpowered.com/
There are a ton of Audio Latency issues in Android. There's really not anything that can be done about it. It seems like ICS (4.0) may have done some improvements on it, from what I've read.
You could subscribe to Andraudio and you'd actually be better off directing Android Audio questions through their emailing list than through Stackoverflow:
http://music.columbia.edu/mailman/listinfo/andraudio
I'm currently implementing a sound effect mixing on Android via OpenSL. I have an initial implementation going, but I've encountered some issues.
My implementation is as follows:
1) For each sound effect I create several AudioPlayer objects (one for each simultaneous sound) that uses an SLDataLocator_AndroidFD data source that in turn refers to an OGG file. For example, if I have a gun firing sound (lets call it gun.ogg) that is played in rapid succession, I use around 5 AudioPlayer objects that refer to the same gun.ogg audio source and also the same outputmix object.
2) When I need to play that sound effect, I search through all the AudioPlayer objects I created and find one that isn't currently in the SL_PLAYSTATE_PLAYING state and use it to play the effect.
3) Before playing a clip, I seek to the start of it using SLPlayItf::SetPosition.
This is working out alright so far, but there is some crackling noise that occurs when playing sounds in rapid succession. I read on the Android NDK newsgroup that OpenSL on Android has problems with switching data sources. Has anyone come across this issue?
I'm also wondering if anyone else seen or come up with a sound mixing approach for OpenSL on Android. If so, does your approach differ from mine? Any advice on the crackling noise?
I've scoured the internet for OpenSL documentation and example code, but there isn't much out there with regards to mixing (only loading which I've figured out already). Any help would be greatly appreciated.
This is probably not the best approach (creating many instances of audio players). Unfortunately the Android version (2.3) of OpenSL ES doesn't support SLDynamicSourceItf. Which would be similar to OpenAL's source binding interface. One approach would be to create multiple stream players. You would then search for a stream player that isn't currently playing and start streaming your sound effect to it from memory. It's not ideal but it's doable.
You should probably not use the ogg format for sound effects either. You're better off with WAV (PCM) as it won't need to be decoded.
Ogg is fine for streaming background music.