Android's MediaRecorder class introduces a significant (order of magnitude a second) delay when recording video. See e.g. Delay in preparing media recorder on android.
After extensive searching, I can't find a way to get rid of this, only workarounds avoiding MediaRecorder and using onPreviewFrame instead.
In my particular application, MediaRecorder's delay wouldn't be so bad if I could measure it programatically -- assuming its standard deviation wasn't too high under everyday background load conditions. Does anyone know how to do that?
I thought of using a FileObserver on the recorded file to find out when frames start being written but, since I don't know the pipeline delays, I couldn't draw a firm conclusion from that.
On a related note, does anyone know if the 'recording alert' sound is played before the first frame is recorded? Is there a way of turning that sound off?
Related
I try to make an android app which will record yaw of phone into a binary file per frame of video capture. I initially tried to make that work with OpenCV, but the fact that video writer doesn't work seem to introduce a lot of complications. Sequentially I looked into camera2 API and CameraX in hopes of using them instead. However I found no way to implement per frame callbacks. I thought of making a service app which will intercept camera events in non blocking manner and then transfer seconds to frames since my framerate is fixed 30 at the moment but I'm struggling to find a way here as well. Can anyone suggest a way to achieve the expected result?
I also tried to see the composition of android mp4 (because from my understanding it could carry per frame metadata) but to no avail.
Update:
I.ve managed to record video with cameraX while calling the writing of the yaw on background thread from sensor callback set to duration of one frame from framerate. But this is still not very precise. I've tried to add ImageAnalysis UseCase to my UseCaseGroup since its called per frame, but I have an issue which was already mentioned here. Anyone knows how to resolve this, or any workaround to achieve per frame callback?
Camera2 provides per-frame callbacks in various ways - the simplest probably being the onCaptureComplete callback.
That callback will get all the frame metadata, which includes the start of exposure timestamp. You can use that timestamp to match up to sensor yaw information and its timestamps, and then send that on to whatever recording mechanism you have.
CameraX also lets you hook into this via:
https://developer.android.com/reference/androidx/camera/camera2/interop/Camera2Interop.Extender?hl=en#setSessionCaptureCallback(android.hardware.camera2.CameraCaptureSession.CaptureCallback)
if there's no existing convenient path.
So I've been trying to figure out a way to get the timestamp of a video frame as it's being recorded.
All of the samples online and in the API documentation tell you to use MediaRecorder to record video from the camera. The problem is that no timestamp data is returned, nor is there a callback called when it records a frame.
I started investigating the Camera app in the Android 4.2 source code, and was able to make significant progress on this. I successfully recorded video from the camera, saving frame timestamps to a file when the SurfaceTexture's onFrameAvailable listener was called, since SurfaceTexture has a timestamp property.
Upon further investigation though, I figured out that I was receiving these callbacks when the frame was being displayed not when recorded. In hindsight, this makes a lot of sense and I should have spotted it earlier.
In any case, I continued further into the Camera app and started looking at the EffectsRecorder class that it has. I found that it uses some undocumented APIs via the filter framework to allow the use of OpenGL shaders. For my use case, this is helpful, so I likely will continue down this path. I'm still trying to make my way through it to get it recording video correctly, but I feel like I will still only get timestamps on display, rather than on recording.
Looking further, I see that it uses a filter that has a MediaRecorder class underneath it, using an undocumented second video source other than camera called GRALLOC_BUFFER = 2. This is even more intriguing and useful, but ultimately, I still need the timestamp for the frame being recorded.
I'm appealing to Stack Overflow in hopes that someone in the community has encountered this. I simply need a pointer of where to look. I can provide source code, since most of it is just lifted from the AOSP, but I haven't as yet simply because I'm not sure what would be relevant.
Help is greatly appreciated.
I'm trying to use a C library (Aubio) to perform beat detection on some music playing from a MediaPlayer in Android. To capture the raw audio data, I'm using a Visualizer, which sends a byte buffer at regular intervals to a callback function, which in turn sends it to the C library through JNI.
I'm getting inconsistent results (i.e. almost no beats are detected, and the only ones who are are not really consistent with the audio). I've checked multiple times, and, while I can't exactly rule out what I'm doing on my own, I'm wondering how exactly the Android Visualizer behaves, since it is not explicit in the documentation.
If I set the buffer size using setCaptureSize, does that mean that the captured buffer is averaged over the complete audio samples? For instance, if I divide the capture size by 2, will it still represent the same captured sound, but with 2 times less precision on the time axis?
Is it the same with the capture rate? For instance, does setting twice the capture size with half the rate yield the same data?
Are the captures consecutive? To put it another way, if I take too long to process a capture, are the sounds played during the processing ignored when I receive the next capture?
Thanks for your insight!
Make sure the callback function gets the entire audio signal, for instance by counting the frames that get out of it, and the ones that reach the callback.
It would help to be pointed at Visualizer documentation.
I have searched for this online, but am still a bit confused (as I'm sure others will be if they think of something like this). I'd like to preface by saying that this is not for homework and/or profit.
I wanted to create an app that could listen to your microwave as you prepare popcorn. It would work by sounding an alarm when there's a certain time interval between pops (say 5-6 seconds). Again, this is simply a project to keep me occupied - not for a class.
Either way, I'm having trouble trying to figure out how to analyze the audio intake in real-time. That is, I need a way to log the time when a "pop" occurs. So that you guys don't think I didn't do any research into the matter, I've checked out this SO question and have extensively searched the AudioRecord function list.
I'm thinking that I will probably have to do something with one of the versions of read() and then compare the recorded audio every 2 seconds or so to the recorded audio of a "pop" (i.e. if 70% or more of the byte[] audioData array is the same as that of a popping sound, then log the time). Can anyone with Android audio input experience let me know if I'm at least on the right track? This is not a question of me wanting you to code anything for me, but a question as to whether I'm on the correct track, and, if not, which direction I should head instead.
I think I have an easier way.
You could use the MediaRecorder 's getMaxAmplitude method.
Anytime your recorder detects a big jump in amplitude, you have detected a corn pop!
Check out this code (ignore the playback part): Playing back sound coming from microphone in real-time
Basically the idea is that you will have to take the value of each 16-bit sample (which corresponds to the value of the wave at that time). Using the sampling rate, you can calculate the time between peaks in volume. I think that might accomplish what you want.
this may be a bit overkill, but there is a framework from MIT media labs called funf: http://code.google.com/p/funf-open-sensing-framework/
They already created classes for audio input and some analysis (FFT and the like), also saving to files or uploading is implemented as far as I've seen, and they handle most of the sensors available on the phone.
You can also get inspired from the code they wrote, which I think is pretty good.
I have been scratching my head for the past week to do this effect on the text. http://www.youtube.com/watch?v=gB2PL33DMFs&feature=related
Would be great if someone can give me some tips or guidance or tutorial on how to do this.
thankz for reading and answering =D
If all you want is to display a movie with video and sound, a MediaPlayer can do that easily.
So I assume that you're actually talking about synchronizing some sort of animated display with a sound file being played separately. We did this using a MediaPlayer and polling getCurrentPosition from within an animation loop. This more or less works, but there are serious problems that need to be overcome. (All this deals with playing mp3 files; we didn't try any other audio formats).
First, your mp3 must be recorded at 44,100 Hz sampling rate. Otherwise the value returned by getCurrentPosition is way off. (We think it's scaled by the ratio of the actual sampling rate to 44,100, but we didn't verify this hypothesis.) A bit rate of 128,000 seems to work best.
Second, and more serious, is that the values returned by getCurrentPosition seem to drift away over time from the sound coming out of the device. After about 45 seconds, this starts to be quite noticeable. What's worse is that this drift is significantly different (but always present) in different OS levels, and perhaps from device to device. (We tested this in 2.1 and 2.2 on both emulators and real devices, and 3.0 on an emulator.) We suspected some sort of buffering problem, but couldn't really diagnose it. Our work-around was to break up longer mp3 files into short segments and chain their playback. Lots of bookkeeping aggravation. This is still under test, but so far it seems to have worked.
Ted Hopp: time drifting on MP3 files is likely caused by those MP3 files being VBR. I've been developing Karaoke apps for a while, and pretty much every toolkit - from Qt Phonon to ffmpeg - had problems reporting correct audio position on variable MP3 files. I assume this is because they all try to calculate the current audio position by using the number of decoded frames, which makes it unreliable for VBR MP3s. I described it in a user-friendly way in the Karaoke Lyrics Editor FAQ
Unfortunately the only solution I found is to recode MP3s to CBR. Another was to ditch the current position completely, and rely only on system clocks. That actually produced a better result for VBR MP3s, but still not as good as recoding them into CBR.