I want to implement loop video recording (e.g., DVR devices for vehicles/cars work this way).
MediaRecorder has setMaxDuration method:
After recording reaches the specified duration, a notification will be
sent to the MediaRecorder.OnInfoListener with a "what" code of
MEDIA_RECORDER_INFO_MAX_DURATION_REACHED and recording will be
stopped. Stopping happens asynchronously, there is no guarantee that
the recorder will have stopped by the time the listener is notified.
So when it reaches that "max duration" it stops recording but asynchronously and how can I start a new recording session if previous one can still be in progress?
Should I create a new instance of MediaRecorder for the next recording session? Will it work fine?
private val infoListener: MediaRecorder.OnInfoListener =
MediaRecorder.OnInfoListener { mr, what, extra ->
when (what) {
MediaRecorder.MEDIA_RECORDER_INFO_MAX_DURATION_REACHED -> {
// I want to start a new recording session
}
...
}
}
For a continuous recording application, setMaxFileSize() is more useful, because the MediaRecorder will send a MEDIA_RECORDER_INFO_MAX_FILESIZE_APPROACHING code to OnInfoListener, at which point your application can call setNextOutputFile() to set the next filename and allow the MediaRecorder to continue into the new file without stopping and restarting the recording from the application. If you know the video and audio bitrates, you can estimate the file size corresponding to your desired duration. It is will not be the exact duration, but still useful for accomplishing basic chunked recording.
Your application will need to keep track of the files you create and delete the old ones if you want to implement a circular recording scheme with a limited total storage size.
Related
My application has a feature that records audio using Media Recorder that is started through a service (IntentService) so that it is recorded in the background.
Because some device and API limitations I need to implement a limit for the files so that once I reach the configured limit, generate the file and start recording again.
Example, upon reaching 1 minute, you must stop recording, generate the file, and start another recording until the service is stopped.
I tryed use setOnInfoListener, but this event is never called, when I defined maxDuration to one minute, after this duration time, the logcat show this message and stop audio record:
Handler (android.media.MediaRecorder$EventHandler) {ee2b629} sending message to a Handler on a dead thread
java.lang.IllegalStateException: Handler (android.media.MediaRecorder$EventHandler) {ee2b629} sending message to a Handler on a dead thread
at android.os.MessageQueue.enqueueMessage(MessageQueue.java:545)
at android.os.Handler.enqueueMessage(Handler.java:661)
at android.os.Handler.sendMessageAtTime(Handler.java:630)
at android.os.Handler.sendMessageDelayed(Handler.java:600)
at android.os.Handler.sendMessage(Handler.java:537)
at android.media.MediaRecorder.postEventFromNative(MediaRecorder.java:1709)
mediaRecorder = new MediaRecorder();
mediaRecorder.setAudioSource(MediaRecorder.AudioSource.MIC);
mediaRecorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
mediaRecorder.setOutputFile(file.getPath());
mediaRecorder.setAudioEncoder(MediaRecorder.AudioEncoder.AMR_NB);
mediaRecorder.setMaxDuration(60000);
mediaRecorder.prepare();
mediaRecorder.start();
mediaRecorder.setOnInfoListener((mr, what, extra) -> {
if(what == MediaRecorder.MEDIA_RECORDER_INFO_MAX_DURATION_REACHED){
//Do somethings
}
});
Start a timer ,which will trigger after a given amount of time, say you want the recording to be max of 20 minutes ,then start timer when you call mediaRecorder.start(), and will trigger after 20 mins calling the stop() and release() method there. You can even cancel if you feel recording is done
When I play a sound in my app it comes off as clippy and distorted. Here is a recording: recording.
Here is the sound file as it was uploaded to android studio: success sound cue
Here is the function that calls the sound from the .raw
public void playCorrectAnswerSound() {
final MediaPlayer mp = MediaPlayer.create(this, R.raw.correct);
mp.start();
}
Heres how I call it:
Thread t = new Thread(){
public void run(){
playCorrectAnswerSound();
}
};
t.start()
This is my first time debugging a sound related issue. I don't know what else to include in this post, so if you need more info please say so.
EDIT: I was asked to record more of the distortion. Here it is. Also I should say that after more testing, my physical device lacks the sound distortion while the sound distortion is present on 3 different emulators.
I'm going to say this is stuttering due to underrun (starvation) of the MediaPlayer's internal playback buffer. That is, the MediaPlayer can't supply data fast enough to keep up with the sound hardware (which suggests a severe lack of processing power). If the buffer starves, it'll start to play back old data (because it's a circular buffer). This causes a sharp phase transition, which sounds like a "click". Presumably the MediaPlayer recovers quickly enough that the "correct" sound resumes playing shortly thereafter.
Here is a picture of the spectrum from Audacity. 0-4KHz. The first row is the clean .mp3; the next four rows are the distorted recordings (in no particular order). All rows have been aligned in time, and are roughly the same amplitude. The large vertical stripes in the last four rows represent the distortion/clicks that you hear.
In the application which I want to create, I face some technical obstacles. I have two music tracks in the application. For example, a user imports the music background as a first track. The second path is a voice recorded by the user to the rhythm of the first track played by the speaker device (or headphones). At this moment we face latency. After recording and playing back in the app, the user hears the loss of synchronisation between tracks, which occurs because of the microphone and speaker latencies.
Firstly, I try to detect the delay by filtering the input sound. I use android’s AudioRecord class, and the method read(). This method fills my short array with audio data.
I found that the initial values of this array are zeros so I decided to cut them out before I will start to write them into the output stream.
So I consider those zeros as a „warmup” latency of the microphone. Is this approach correct? This operation gives some results, but it doesn’t resolve the problem, and at this stage, I’m far away from that.
But the worse case is with the delay between starting the speakers and playing the music. This delay I cannot filter or detect. I tried to create some calibration feature which counts the delay. I play a „beep” sound through the speakers, and when I start to play it, I also begin to measure time. Then, I start recording and listen for this sound being detected by the microphone. When I recognise this sound in the app, I stop measuring time. I repeat this process several times, and the final value is the average from those results. That is how I try to measure the latency of the device. Now, when I have this value, I can simply shift the second track backwards to achieve synchronisation of both records (I will lose some initial milliseconds of the recording, but I skip this case, for now, there are some possibilities to fix it).
I thought that this approach would resolve the problem, but it turned out this is not as simple as I thought. I found two issues here:
1. Delay while playing two tracks simultaneously
2. Random in device audio latency.
The first: I play two tracks using AudioTrack class and I run method play() like this:
val firstTrack = //creating a track
val secondTrack = //creating a track
firstTrack.play()
secondTrack.play()
This code causes delays at the stage of playing tracks. Now, I don’t even have to think about latency while recording; I cannot play two tracks simultaneously without delays. I tested this with some external audio file (not recorded in my app) - I’m starting the same audio file using the code above, and I can see a delay. I also tried it with MediaPlayer class, and I have the same results. In this case, I even try to play tracks when callback OnPreparedListener invoke:
val firstTrack = //AudioPlayer
val secondTrack = //AudioPlayer
second.setOnPreparedListener {
first.start()
second.start()
}
And it doesn’t help.
I know that there is one more class provided by Android called SoundPool. According to the documentation, it can be better with playing tracks simultaneously, but I can’t use it because it supports only small audio files and that can't limit me.
How can I resolve this problem? How can I start playing two tracks precisely at the same time?
The second: Audio latency is not deterministic - sometimes it is smaller, and sometimes it’s huge, and it’s out of my hands. So measuring device latency can help but again - it cannot resolve the problem.
To sum up: is there any solution, which can give me exact latency per device (or app session?) or other triggers which detect actual delay, to provide the best synchronisation while playback two tracks at the same time?
Thank you in advance!
Synchronising audio for karaoke apps is tough. The main issue you seem to be facing is variable latency in the output stream.
This is almost certainly caused by "warm up" latency: the time it takes from hitting "play" on your backing track to the first frame of audio data being rendered by the audio device (e.g. headphones). This can have large variance and is difficult to measure.
The first (and easiest) thing to try is to use MODE_STREAM when constructing your AudioTrack and prime it with bufferSizeInBytes of data prior to calling play (more here). This should result in lower, more consistent "warm up" latency.
A better way is to use the Android NDK to have a continuously running audio stream which is just outputting silence until the moment you hit play, then start sending audio frames immediately. The only latency you have here is the continuous output latency.
If you decide to go down this route I recommend taking a look at the Oboe library (full disclosure: I am one of the authors).
To answer one of your specific questions...
Is there a way to calculate the latency of the audio output stream programatically?
Yes. The easiest way to explain this is with a code sample (this is C++ for the AAudio API but the principle is the same using Java AudioTrack):
// Get the index and time that a known audio frame was presented for playing
int64_t existingFrameIndex;
int64_t existingFramePresentationTime;
AAudioStream_getTimestamp(stream, CLOCK_MONOTONIC, &existingFrameIndex, &existingFramePresentationTime);
// Get the write index for the next audio frame
int64_t writeIndex = AAudioStream_getFramesWritten(stream);
// Calculate the number of frames between our known frame and the write index
int64_t frameIndexDelta = writeIndex - existingFrameIndex;
// Calculate the time which the next frame will be presented
int64_t frameTimeDelta = (frameIndexDelta * NANOS_PER_SECOND) / sampleRate_;
int64_t nextFramePresentationTime = existingFramePresentationTime + frameTimeDelta;
// Assume that the next frame will be written into the stream at the current time
int64_t nextFrameWriteTime = get_time_nanoseconds(CLOCK_MONOTONIC);
// Calculate the latency
*latencyMillis = (double) (nextFramePresentationTime - nextFrameWriteTime) / NANOS_PER_MILLISECOND;
A caveat: This method relies on accurate timestamps being reported by the audio hardware. I know this works on Google Pixel devices but have heard reports that it isn't so accurate on other devices so YMMV.
Following the answer of donturner, here's a Java version (that also uses other methods depending on the SDK version)
/** The audio latency has not been estimated yet */
private static long AUDIO_LATENCY_NOT_ESTIMATED = Long.MIN_VALUE+1;
/** The audio latency default value if we cannot estimate it */
private static long DEFAULT_AUDIO_LATENCY = 100L * 1000L * 1000L; // 100ms
/**
* Estimate the audio latency
*
* Not accurate at all, depends on SDK version, etc. But that's the best
* we can do.
*/
private static void estimateAudioLatency(AudioTrack track, long audioFramesWritten) {
long estimatedAudioLatency = AUDIO_LATENCY_NOT_ESTIMATED;
// First method. SDK >= 19.
if (Build.VERSION.SDK_INT >= 19 && track != null) {
AudioTimestamp audioTimestamp = new AudioTimestamp();
if (track.getTimestamp(audioTimestamp)) {
// Calculate the number of frames between our known frame and the write index
long frameIndexDelta = audioFramesWritten - audioTimestamp.framePosition;
// Calculate the time which the next frame will be presented
long frameTimeDelta = _framesToNanoSeconds(frameIndexDelta);
long nextFramePresentationTime = audioTimestamp.nanoTime + frameTimeDelta;
// Assume that the next frame will be written at the current time
long nextFrameWriteTime = System.nanoTime();
// Calculate the latency
estimatedAudioLatency = nextFramePresentationTime - nextFrameWriteTime;
}
}
// Second method. SDK >= 18.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED && Build.VERSION.SDK_INT >= 18) {
Method getLatencyMethod;
try {
getLatencyMethod = AudioTrack.class.getMethod("getLatency", (Class<?>[]) null);
estimatedAudioLatency = (Integer) getLatencyMethod.invoke(track, (Object[]) null) * 1000000L;
} catch (Exception ignored) {}
}
// If no method has successfully gave us a value, let's try a third method
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
AudioManager audioManager = (AudioManager) CRT.getInstance().getSystemService(Context.AUDIO_SERVICE);
try {
Method getOutputLatencyMethod = audioManager.getClass().getMethod("getOutputLatency", int.class);
estimatedAudioLatency = (Integer) getOutputLatencyMethod.invoke(audioManager, AudioManager.STREAM_MUSIC) * 1000000L;
} catch (Exception ignored) {}
}
// No method gave us a value. Let's use a default value. Better than nothing.
if (estimatedAudioLatency == AUDIO_LATENCY_NOT_ESTIMATED) {
estimatedAudioLatency = DEFAULT_AUDIO_LATENCY;
}
return estimatedAudioLatency
}
private static long _framesToNanoSeconds(long frames) {
return frames * 1000000000L / SAMPLE_RATE;
}
The android MediaPlayer class is notoriously slow to begin audio playback, I experienced an issue in an app I was creating where there was a greater than one second delay to begin playing an audio clip. I resolved it by switching to ExoPlayer which resulted in the playback starting within 100ms. I've also read that ffmpeg has even faster start audio startup time than ExoPlayer but I haven't used it so I can't make any promises.
I am implementing an Application that includes the functionality of saving Recorded Video in to Different Video Files based on a certain amount of Time.
For Achieving that i have implemented a Custom Camera and used the MediaRecorder.stop() and MediaRecorder.start() in a certain Loop.
But this approach is creating a Lag Effect while restarting Media Recorder (Stop and Start). Is it possible to seamlessly Stop and Start Recording using Media Recorder or any Third Party Library ?
Any help is Highly Appreciated.
I believe the best solution to implement chunks recording is to set maximum time in MediaRecorder Object
mMediaRecorder.setMaxDuration(CHUNK_TIME);
then you can attach an info listener, it will intimate you when it will hit maximum chunk time
mMediaRecorder.setOnInfoListener(new MediaRecorder.OnInfoListener() {
#Override
public void onInfo(MediaRecorder mr, int what, int extra) {
if (what == MediaRecorder.MEDIA_RECORDER_INFO_MAX_DURATION_REACHED) {
// restartVideo()
}
}
});
in restartVideo you should firstly clear previous MediaRecorder Object and start video again.
You can create two instances of MediaRecorder which will overlap slightly (i.e. when the stream is close to the end of the first chunk you can prepare and start the second one). It is possible to record 2 video files using 2 MediaRecorders at the same time if they capture only the video. Unfortunately sharing the mic between 2 MediaRecorder instances is not supported.
When using MediaPlayer, I noticed that whenever my phone stucks, the MediaPlayer glitches and then continues playing from the position in the audio it glitched.
This is bad for my implementation since I want the audio to be played at a specific time.
If I have a song of 1000 millisecond length, I want is the ability to set MediaPlayer to start playing at some specific time t, and then exactly stop at at time t+1000.
This means that I actually need two things:
1) Start MediaPlayer at a specific time with a very small delay.
2) Making MediaPlayer glitches ignore the audio they glitched on and continue playing in order to finish the song on time.
The delay of the functions is very important to me and I need the audio to be played exactly(~) at the time it was supposed to be played.
Thanks!
You will need to use possibly mp.getDuration(); and/or mp.getCurrentPosition(); although it's impossible to know exactly what you mean by "I need the audio to be played exactly(~) at the time it was supposed to be played."
Something like this should get you started:
int a = (mp.getCurrentPosition() + b);
Thanks for the answer Mike. but unfortunately this won't help me. Let's say that I asked MediaPlayer to start playing a song of length 3:45 at 00:00. At 01:00 I started using the phone's resources, due to the heavy usage my phone glitched making MediaPlayer pause for 2 seconds.
Time:
00:00-01:00 - I heard the audio at 00:00-01:00
01:00-01:02 - I heard silence because the phone glitched
01:02-03:47 - I heard the audio at 01:00-03:45 with 2 second time skew
Now from what I understood MediaPlayer is a bad choice of usage on this problem domain, since MediaPlayer provides a high level API.I am currently experimenting with the
AudioTrack class which should provide me with what I need:
//Creating a new audio track
AudioTrack audioTrack = new AudioTrack(...)
//Get start time
long start = System.currentTimeMillis();
// loop until finished
for (...) {
// Get time in song
long now = System.currentTimeMillis();
long nowInSong = now - start;
// get a buffer from the song at time nowInSong with a length of 1 second
byte[] b = getAudioBuffer(nowInSong);
// play 1 second of music
audioTrack.write(b, 0, b.length);
// remove any unplayed data
audioTrack.flush();
}
Now if I glitch I only glitch for 1 second and then I correct myself by playing the right audio at the right time!
NOTE
I haven't tested this code but it seems like the right way to do it. If it will actually work I will update this post again.
P.S. seeking in MediaPlayer is:
1. A heavy operation that will surely delay my music (every millisecond counts here)
2. Is not thread safe and cannot be used from multiple threads (seeks, starts etc...)