Trying to read from the same MP4 file with a MediaCodec (decoder) instance and a MediaPlayer instance doesn't work as expected. When pausing (or seeking) and then resuming playback, the MediaPlayer's position is unpredictable and will often jump ahead of the MediaCodec decoder by 100-500 ms.
In my application, the decoder is handling H.264 video frames and the MediaPlayer is playing AAC audio. Both have been initialized with the same on-device MP4 file.
Some observations:
Playback from the beginning of the file (without pausing or seeking) works fine
The audio nearly always jumps ahead of the video after seeking or pausing
Profiling showed that the "jump" appears to happen just after resuming playback; prior to resuming, both components report the same playback position. Just after resuming, there is a large and sudden change in the MediaPlayer's position.
The audio remains ahead of the video by a constant amount after the jump
Letting each component use its own identical copy of the MP4 works fine, and that's how I solved the problem for now. I am aware that MediaCodec can be used to process audio, but I would prefer to avoid that approach.
After combing through the relevant AOSP code (MediaCodec, MediaPlayer, and some JNI classes), it isn't clear to me how these components would be interfering with one another. Do they share a low-level Binder object? Is the MediaPlayer somehow (re)using the cursor that the MediaCodec decoder is using to fill its input buffers?
Related
How can I play background audio, in Android, without interrupting the MediaPlayer playback, by either using MediaPlayer (preferred) or OpenSL ES?
I know SoundPool is able to play sound effects without interrupting any MediaPlayer playback, but the size is limited to 1M per effect, which is way to less. Not requesting audio focus, via AudioManager doesn't seem to work either, audio doesn't play at all in this case.
And in the case of OpenSL ES, all audio generally stops when I start to play a longer asset file. It's similar to the behaviour of SoundPool described above.
Edit from the comments:
I don't want to interrupt other music players, it's the background
audio of a game, which shall play without interrupting the, for
example, music of a player. Games like Subway Surfer, Clash Royale and
such seem to have this achieved somehow, but I could not achieve it
via OpenSL ES, or MediaPlayer.
In order to play sound in background you can use SoundPool, AudioTracks and OpenSlES.
Soundpool: Use small files and make a sequence. In my last project i use 148 sound files (all small) in different scenarios and played using soundpool. Make a list and play one by one or in parallel. Also in games usually you have a small set of sound for particular scenario and it loops. Best is soundpool for that. You can also perform some effects like rate change. Also ogg files are way small, so use them.
AudioTrack: You can use RAW PCM data in audio track. If you want you can extract pcm data using MediaExtractor from almost all formats and use it. But it will be a little work for you in your case, but it is really good (supports huge data, even live streaming).
OpenSLES: Apparently android uses opensles in background for all its purpose. So using it will help you a lot. But it's not easy to get everything done on it. You need to learn more for lesser work.
I have been deeply working on OpenSlES for about 20 days and still i will say for small purpose use soundpool, medium to high level implementation use AudioTracks and for kickass implementation use OpenSLES.
PS: It will be bad effect on your user if you play game sound in background while they are playing their music or their call. Personal experience.
Has anyone tried using ExoPlayer to achieve this?
I tried looking online with no success.
When I say gapless playback, I am referring to the problem of using the media player to play local videos back to back. After the first video is done playing, there is a noticeable delay of 1 second before the second video starts.
Hoping this question helps in understanding this issue further.
For reference please look at the following question:
Android: MediaPlayer gapless or seamless Video Playing
ExoPlayer 2, which is now officially released, seems to support gapless playback using the ConcatenatingMediaSource class. From its developer guide:
Transitions between sources are seamless. There is no requirement that the sources being concatenated are of the same format (e.g. it’s fine to concatenate a video file containing 480p H264 with one that contains 720p VP9). The sources may even be of different types (e.g. it’s fine to concatenate a video with an audio only stream).
And the example code:
MediaSource firstSource = new ExtractorMediaSource(firstVideoUri, ...);
MediaSource secondSource = new ExtractorMediaSource(secondVideoUri, ...);
// Plays the first video, then the second video.
ConcatenatingMediaSource concatenatedSource =
new ConcatenatingMediaSource(firstSource, secondSource);
EDIT: ExoPlayer 2 supports gapless playback, but as of the time of writing is still unreleased as a stable version.
You will most likely never be able to achieve perfect gapless playback of multiple tracks with ExoPlayer or Android Media Player. Neither have been written to support starting multiple tracks and I imagine it will stay out of scope for both of them.
You can achieve gapless playback by using 2 different player instances, once you have started and played the first, you can load the second and start playback once the first finishes. Using this method you could have a gapless solution, as long as you prepare the second video during playback of the first.
To take it further, you can also use 2 different surface textures for rendering the multiple videos, once the first video reaches the end you could fade out the texture and fade in the new one. Resulting in a nice seamless video effect.
Because of the nature of playing multiple videos at once you will most likely want to create your own timer for incrementing the time and deciding when to switch to the next video, rather than trying to use the callbacks from ExoPlayer or Android Media. This will allow you to keep track of the time in a more accurate fashion, without needing to keep talking to multiple video codecs.
I know this is not the answer you've been looking for, but it's the only reasonable answer. The sole way to ensure no gaps in playback is to download the entire file first and begin playback when it's done. Otherwise, in the event that you lose connectivity before the file is finished downloading, pausing is inescapable.
I just tried switching to ExoPlayer from the standard MediaPlayer implementation and the gap is the same if not worse. However I have used a very simple method of restarting the player when the status changes to ended. I don't know if there's a better proper way to do it, perhaps with 2 different ExoPlayers.
My question is about the relative latency of playing, pausing/stopping, and setting volume of audio in Android. Specifically: whether it's the same or lower latency to pause/stop an audio clip than to play it, and likewise whether it's the same or lower latency to set the volume of a clip (or of the system volume) than to play it.
For context, suppose the latency of playing an audio clip in Android is 150ms, i.e. SoundPool.play is executed at T=0m and the end-user hears the sound at T=150ms.
At T=200m, the program executes SoundPool.pause. If the pause latency is also 150m, that means the pause won't be heard by the end-user until T=350m, after they have heard 200m of the clip. If, however, the pause latency is, say 50m, then the sound will stop at T=250m, after only 100m has been head by the end-user.
Obviously latency isn't constant, exact, or consistent across devices, so to be more precise, what I'm really asking is whether Android uses a separate pathway or technique to pause/stop/change volume of audio (either program-specific or system-wide volume) that is inherently lower-latency than the way audio is played.
Setting up Play takes more time as it has to initialize the play the following actions takes path
find the MIME type of the media file, this needs parsing of the media format and looks for specific header
initialize audio decoder(usually hardware), the OMX decoder has to be loaded into memory
setup the buffers say allocate 10 buffers in the parser and 10 buffers in the decoder.
Setup the paths between parser and decoder and playback audio device (Speaker)
Play happens at this step, data flows from parser buffers to decoder buffers, when the decoder buffers are filled, OMX (decoder framework) will notify player engine, engine passes the buffer data to AudioManager -> AudioTrack etc.
Decoder will again process the data from Parser buffers and this process goes on until EOF or user press pause/stop
During pause latency should be much low than play because, only the data exchnage is paused, but buffers are not released.
During stop buffers are released and player is also released, so need to do same process for play again if user needs to play again.
Volume up and down is simple calls to AudioManager to adjust the call voleumes. So its latency should be lower than play/stop
I have two media files located locally - mp4 video and m4a audio, which have to be played in sync. I use MediaPlayer objects for this purpose, all start/pause methods are called simultaneously.
Sometimes I see the difference between audio and video right after players start, sometimes after tapping on pause/resume.
I added logs, and see that after pausing media players, their position differs (e.g. MediaPlayer with audio file: 1820ms, MediaPlayer with video file: 1760ms).
One more interesting thing is that seekTo operation with audio file works good, while with video it's really unpredictable.
Please suggest what is the reason of such a behavior and which solutions are available in order to fix that?
I think that this effect with the video timing depends on how the " key frames", which contain a full frame of information have been encoded. It's only possible to locate to one of these keyframes.
Video editing software gets over this by rolling the deltas from intervening i-frames to the exact seek position.
Is there any way to intercept or just-read the audio output in android device?
I need to read the whole audio output in PCM from inside myActivity, including media player application in background, voice from calls, MediaPlayer istances inside myACtivity, etc., everything that's going to played by speakers. Actually, if it was possible to read them separately, would be great as well.
I tried with AudioRecord, giving it as audioSource parameter every constant found in MediaRecorder.AudioSource with no luck, should I try different audioSources?
Is it a so low-level task that has to be handled within native layer?
The visualizer class is helpful. I use it to playback immediately played audio here.
This audio comes in very low quality, however, so it's only really good for visualization.