I'm trying to add HMS automatic speech recognition (ASR) to my app. I already have SpeechRecognizer implemented, but it requires GMS to work.
The current HMS implementation works on a non-huawei device with HMS core installed, but does not work on my Huawei Mediapad T5.
Things I've tried
The methods are called from different threads (main thread and graphics thread), so I've tried synchronizing the methods on a lock or posting a Runnable to the activity handler, without making much a difference. I.E., wrapping the functions in synchronized(lock) or activity.post.
Code:
fun init(activity: Activity)
speechRecognizer = MLAsrRecognizer.createAsrRecognizer(activity)
speechIntent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH)
.putExtra(
MLAsrCaptureConstants.LANGUAGE,
"en-US"
)
.putExtra(MLAsrConstants.FEATURE, MLAsrConstants.FEATURE_ALLINONE)
speechRecognizer?.setAsrListener(listener)
fun startListening()
speechRecognizer?.startRecognizing(speechIntent)
fun destroy()
speechRecognizer?.destroy()
Logs
4945-4945 W/HmsSpeechRecognitionHolder#c1cafe: init on Thread[main,5,main]
4945-4945 W/InputMethodManager: startInputReason = 1
4945-4945 W/HmsSpeechRecognitionHolder#c1cafe: startListening on Thread[main,5,main]
4945-4945 E/HaLogProvider: forbiddenHiLog openHa = false
4945-4945 E/HaLogProvider: forbiddenHiLog.getVenderCountry=ca
4945-4945 E/HaLogProvider: forbiddenHiLog openHa = false
4945-4945 E/MLASR_HaAdapter_MLKitAsr: mEventsToBeReported: has no response event isInfoGatherStart:falsemsg: 0
4945-4945 E/HwCustAudioRecordImpl: isOpenEC : false
634-985 E/HuaweiProcessing: ProcessingLib_Create: the algo have already been created.
634-985 W/EffectsFactory: EffectCreate() library huawei_processing: could not create fx Huawei Audio Preprocessing Effect, error -22
634-985 E/EffectFactoryHAL: Error creating effect e707d040-6b79-11e2-b16a-0002a5d5c51b: Invalid argument
721-8060 E/AudioEffect: set(): AudioFlinger could not create effect, status: -19
721-8060 W/AudioPolicyEffects: addInputEffects(): failed to create Fx huawei_pre_processing on source 1
634-8049 E/baidu_asr_interface: asr_baidu_set_parameters_data-not baidu asr
634-7993 W/DeviceHAL: Device 0x78c2d00000 get_mic_mute: Function not implemented
634-987 W/DeviceHAL: Device 0x78c2dc4680 get_mic_mute: Function not implemented
721-8009 W/HuaweiAudioFlinger: soundtrigger is now disable or not support, pls enable it first from setting
721-8009 W/APM_AudioPolicyManager: startInput(78) failed: other input 70 already started
721-8009 E/AudioFlinger: RecordThread::start error,setCallingAppName -1
4945-4945 E/AudioRecord: start() status -38
4945-4945 E/MLASR_A: getVendorCountry=ca
500-8480 W/libc: Set property "hw.wifi.dns_stat" to "99,14,14044,1,34759"
1472-1472 W/HwKeyguardDragHelper: AnimationBlocked
4945-5079 W/libEGL: EGLNativeWindowType 0x79e0317010 disconnect failed
1140-2290 E/WindowManager: win=Window{d80c651 u0 ProjectActivity} destroySurfaces: appStopped=true win.mWindowRemovalAllowed=false win.mRemoveOnExit=false
767-767 E/wificond: Failed to get NL80211_RATE_INFO_NOISE
767-767 E/wificond: Failed to get NL80211_RATE_INFO_SNR
767-767 E/wificond: Failed to get NL80211_STA_INFO_CNAHLOAD
1140-1316 E/WificondControl: Noise: 0, Snr: -1, Chload: -1
767-767 E/wificond: Failed to get NL80211_RATE_INFO_SNR
767-767 E/wificond: Failed to get NL80211_STA_INFO_CNAHLOAD
767-767 E/wificond: Failed to get NL80211_RATE_INFO_NOISE
767-767 E/wificond: Failed to get NL80211_RATE_INFO_SNR
767-767 E/wificond: Failed to get NL80211_STA_INFO_CNAHLOAD
1140-1316 E/WificondControl: Noise: 0, Snr: -1, Chload: -1
761-8466 W/ACodec: forcing OMX state to Idle when received shutdown in ExecutingState
769-8467 W/SimpleSoftOMXComponent: onChangeState mState= 3, mTargetState = 3, state = 2
769-8467 W/SimpleSoftOMXComponent: checkTransitions port buf count is not zero
769-1826 W/SimpleSoftOMXComponent: checkTransitions port buf count is not zero
769-1826 W/SimpleSoftOMXComponent: checkTransitions port buf count is not zero
769-1826 W/SimpleSoftOMXComponent: checkTransitions port buf count is not zero
769-1826 W/SimpleSoftOMXComponent: checkTransitions port buf count is not zero
769-1826 W/SimpleSoftOMXComponent: checkTransitions mState = 2, mTargetState = 1
721-8060 W/HuaweiAudioFlinger: soundtrigger is now disable or not support, pls enable it first from setting
1900-3437 E/HSM: BMNCaller:is not PermissionEnabled.
721-6695 W/AudioFlinger::EffectModule: EffectModule 0x7ba4f22a00 destructor called with unreleased interface
634-941 E/audio_hw_primary: in_remove_audio_effect error effect is null
634-941 W/StreamHAL: Error from HAL stream in function remove_audio_effect: Function not implemented
721-6695 E/AudioFlinger::EffectModule: Error when removing effect: -38
721-6695 W/AudioFlinger::EffectHandle: disconnect Effect handle 0x7ba4e45800 disconnected after thread destruction
1640-1796 W/AudioState: session release and not found sessionId: 81
4945-4945 W/HmsSpeechRecognitionHolder#c1cafe: destroy on Thread[main,5,main]
4945-8481 E/HwCustAudioRecordImpl: isOpenEC : false
4945-8481 E/HwCustAudioRecordImpl: isOpenEC : false
Things I found suspicious in the logs
634-985 E/HuaweiProcessing: ProcessingLib_Create: the algo have already been created.
634-985 W/EffectsFactory: EffectCreate() library huawei_processing: could not create fx Huawei Audio Preprocessing Effect, error -22
634-985 E/EffectFactoryHAL: Error creating effect e707d040-6b79-11e2-b16a-0002a5d5c51b: Invalid argument
721-8060 E/AudioEffect: set(): AudioFlinger could not create effect, status: -19
721-8060 W/AudioPolicyEffects: addInputEffects(): failed to create Fx huawei_pre_processing on source 1
721-8009 W/APM_AudioPolicyManager: startInput(78) failed: other input 70 already started
721-8009 E/AudioFlinger: RecordThread::start error,setCallingAppName -1
4945-4945 E/AudioRecord: start() status -38
Note: the HMS demo apps I've tried works correctly on my Mediapad T5.
Update: After some fixes pointed out by #shirley, ASR seems to be working reliably on a P30Lite. But still facing the same issue on an older Mediapad T5.
According to the logs you provided, the voice of the user is not detected.
The meanings of logs and status codes are as follows:
solution
It is recommended that you add logs to the callback method of the MLAsrListener listener to view the speech recognition process.
You are advised to check mSpeechRecognizer.destroy(). Check whether this method is invoked prematurely and has ended before it starts.
Check whether the device is faulty or whether the microphone of the device is invalid. Replace the device and perform the test.
After reviewing your logs, the following errors were found:
The reason for this error is:The Languagecode for speech recognition exceeds 10.
You can view the code here:
Ensure that the speech recognition Languagecode does not exceed 10.
11203 ,subError code: 3002,errorMessage: Service unavailable
The cause of this error is that the app_id information is not found in the project.
You are advised to check whether the agconnect-services.json file exists in the project, as shown in the following
If the file does not exist, you need to add it to the project. If the file exists, ensure that app_id is correct.
For details, see the following Docs.
Check whether Automatic Speech Recognition fails to be enabled.
If Automatic Speech Recognition fails to be enabled, you can obtain the cause by using the onError(int error, String errorMessage) method of the MLAsrListener class, as shown in the following figure.
You can add the above method to the listener's class:
2.If speech recognition is enabled successfully but the speech recognition result is not obtained:
The MLAsrConstants.FEATURE parameter is set to FUNCTION_ALLINONE. Therefore, you need to obtain the speech recognition result in the onResults(Bundle results) method, as shown in the following figure.
Some models of phones and tablets could have resource limitation issues when using the ML ASR. The symptom of the problem is that the phone/tablet does not respond after the microphone button is clicked, or an error message is displayed indicating that the speech recognition service is not installed on the device. Not only HMS ASR, I tried using native Android SpeechRecognizer to implement speech recognition and the sample app hangs after click on the button on a limited hardware resource phone simulator.
To fix your issue, I would suggest to switch using the HMS ML Kit ASR to using HMS ML Kit Real-Time Transcription (RTT). RTT provides similar features as ASR for speech recognition and convert the speech to text. For more details, see the HMS ML Kit RTT document at ML Kit-Real-Time Transcription (huawei.com).
The code for RTT is similar to ASR, you need to provide a SpeechRecognitionListener class or anonymous class to implements MLSpeechRealTimeTranscriptionListener. And there is sample code at the document link too. I tested the sample code on my Huawei phone Mate 30 Pro and it is working just fine.
Related
I am working on my first Android application using Kotlin. The activity is simple, connect to an audio stream from a given URL and allow the user to pause, resume, and/or stop the stream. I've been able to connect to and play the requested audio stream using ExoPlayer, but have a problem and question that I have not found addressed in ExoPlayer documentation. I've followed this documentation as best as I can to write the following:
class StreamAudio : AppCompatActivity() {
private lateinit var playerView: StyledPlayerView
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_stream_audio)
// Bind PlayerView to appropriate element in layout
playerView = findViewById(R.id.audio_player)
// Initialize instance of ExoPlayer
var player = ExoPlayer.Builder(applicationContext).build()
// Set MediaItem to stream_url and play
var stream: MediaItem = MediaItem.fromUri("http://stream_url")
player.setMediaItem(stream)
player.prepare()
player.play()
}
}
When running the application the audio streams, but no StyledPlayer UI ever appears, only a solid black view. Tapping the screen does not cause any UI to appear. When substituting with a StyledPlayerControlView, the UI appears briefly before disappearing and leaving a blank white view.
It seems like bad form to initialize the player within onCreate(), how can I access the player in another function? For example, to stop and release the player upon returning to the parent activity?
Thanks in advance for any help, the logs below only show an error described here that doesn't seem likely to cause UI issues.
Log Output:
W/Codec2Client: query -- param skipped: index = 1342179345.
query -- param skipped: index = 2415921170.
E/FMQ: grantorIdx must be less than 3
grantorIdx must be less than 3
D/CCodecBufferChannel: [c2.android.mp3.decoder#722] Created input block pool with allocatorID 16 => poolID 17 - OK (0)
I/CCodecBufferChannel: [c2.android.mp3.decoder#722] Created output block pool with allocatorID 16 => poolID 34 - OK
D/CCodecBufferChannel: [c2.android.mp3.decoder#722] Configured output block pool ids 34 => OK
E/ion: ioctl c0044901 failed with code -1: Inappropriate ioctl for device
E/FMQ: grantorIdx must be less than 3
E/FMQ: grantorIdx must be less than 3
D/AudioTrack: getTimestamp_l(24): device stall time corrected using current time 4430536774140
D/BufferPoolAccessor2.0: bufferpool2 0xe830a4d8 : 5(40960 size) total buffers - 4(32768 size) used buffers - 0/5 (recycle/alloc) - 4/410 (fetch/transfer)
D/BufferPoolAccessor2.0: evictor expired: 1, evicted: 1
I'm working on a DSP project on Android which requires low latency audio I/O. For this reason, I'm using Oboe library. In the LiveEffect example, the synchronous recording and playback is demonstrated. However, for acoustic feedback neutralization, I need the other way around, that is to generate White Noise signal through a built-in speaker first, then record it using a mic. I tried to modify LiveEffect example using this asked question, i.e setting the recording stream as Master (callback) and using non-blocking write method for the playback stream. But I got the following error when I run my code on Pixel XL (Android 9.0):
D/AudioStreamInternalCapture_Client: processDataNow() wait for valid timestamps
D/AudioStreamInternalCapture_Client: advanceClientToMatchServerPosition() readN = 0, writeN = 384, offset = -384
--------- beginning of crash
A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x5800003f666c66 in tid 2852 (AAudio_1), pid 2796 (ac.oiinitialize)
Here is my callback:
oboe::DataCallbackResult
AudioEngine::onAudioReady(oboe::AudioStream *oboeStream, void *audioData, int32_t numFrames) {
assert(oboeStream == mRecordingStream);
int32_t framesToWrite = mPlayStream->getFramesPerBurst();
oscillator_->whiteNoise(framesToWrite); // write white noise into buffer;
oboe::ResultWithValue<int32_t> result = mPlayStream->write(oscillator_->write(), framesToWrite, 0);
// oscillator_->write() returns const void* buffer;
if (result != oboe::Result::OK) {
LOGE("input stream read error: %s", oboe::convertToText(result.error()));
return oboe::DataCallbackResult ::Stop;
}
// add Adaptive Feedback Neutralization Algorithm here....
return oboe::DataCallbackResult::Continue;
}
Is my approach correct for generating a signal and then capturing it through a mic? If so, can anyone help me with this error? Thank you in advance.
However, for acoustic feedback neutralization, I need the other way around, that is to generate White Noise signal through a built-in speaker first, then record it using a mic
You can still do this using an output stream callback and a non-blocking read on the input stream. This is the more common (and tested) way of doing synchronous I/O. A Larsen effect will work fine this way.
Your approach should still work, however, I'd stick to the LiveEffect way of setting up the streams since it works.
In terms of your error SIGSEGV usually means a null pointer dereference - are you starting your input stream before the output stream? This could meant you're attempting to write to the output stream which hasn't yet been opened.
I'm working on a video processing app. The app has one Activity that contains a Fragment. The Fragment in turn contains a VideoSurfaceView derived from GLSurfaceView for me to show the preview of the video with effect (using OpenGL) to users. After previewing, users can start processing the video.
To process the video, I mainly apply the method described in here.
Everything works fine on most devices, but the Oppo Mirror 3 (Android 4.4). On this device, everytime I try to create an Surface using MediaCodec.createInputSurface(), it throws out java.lang.IllegalStateException with code -38.
E/OMXMaster: A component of name 'OMX.qcom.audio.decoder.aac' already exists, ignoring this one.
E/SoftAVCEncoder: internalSetParameter: StoreMetadataInBuffersParams.nPortIndex not zero!
E/OMXNodeInstance: OMX_SetParameter() failed for StoreMetaDataInBuffers: 0x80001001
E/ACodec: [OMX.google.h264.encoder] storeMetaDataInBuffers (output) failed w/ err -2147483648
E/OMXNodeInstance: createInputSurface requires COLOR_FormatSurface (AndroidOpaque) color format
E/ACodec: [OMX.google.h264.encoder] onCreateInputSurface returning error -38
E/VideoProcessing: java.lang.IllegalStateException
at android.media.MediaCodec.createInputSurface(Native Method)
at com.ltpquang.android.core.processing.codec.VideoEncoder.<init>(VideoEncoder.java:46)
at com.ltpquang.android.core.VideoProcessing.setupVideo(VideoProcessing.java:200)
at com.ltpquang.android.core.VideoProcessing.<init>(VideoProcessing.java:167)
at com.ltpquang.android.ui.activity.PreviewEditActivity.lambda$btNext$12(PreviewEditActivity.java:723)
at com.ltpquang.android.ui.activity.PreviewEditActivity.access$lambda$12(PreviewEditActivity.java)
at com.ltpquang.android.ui.activity.PreviewEditActivity$$Lambda$13.run(Unknown Source)
at java.lang.Thread.run(Thread.java:841)
Playing around a little bit, I observed that:
BEFORE creating and adding the VideoSurfaceView to the layout, I can create MediaCodec encoder and obtain the input surface successfully. And I can create as many as I want if I release the previous MediaCodec before creating a new one, otherwise I can only obtain one and only one input surface regardless how many MediaCodec I have.
AFTER creating and adding the VideoSurfaceView to the layout, there is no chance that I can get the input surface from the MediaCodec, it thows java.lang.IllegalStateException always.
I've tried removing the VideoSurfaceView from the layout, set it to null, before creating the surface, but no luck for me.
I also tried with suggestions from here, or here, but they didn't help.
From this, it seems that my device can only get the software codec. So that I cant create the input surface.
My question is:
Why was that?
If the device's resources is limited, what can I do (release something for example) to continue the process?
If it is related to the software codec, what should I do? How can I detect and release the resource?
Is this related to GL contexts? If yes, what should I do? Should I manage the contexts my self?
I implemented a video encoder by MediaCodec API to encode 2560x720 videos. I verified it on several devices but I encountered a configuration problem on Samsung Note 2 (Android 4.4.2). The following is my code to prepare the codec instance:
Format = MediaFormat.createVideoFormat("video/avc", 2560, 720);
Format.setInteger(MediaFormat.KEY_BIT_RATE, (int) (2560 * 720 * 20 * 0.5f));
Format.setInteger(MediaFormat.KEY_FRAME_RATE, RefreshRate);
Format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar);
Format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
try {
Codec = MediaCodec.createEncoderByType("video/avc");
Codec.configure(Format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
CodecFound = true;
} catch (Exception e) {
CodecFound = false;
if(DEBUG) Log.e(CodecTag, "Failed to find available codec", e);
}
No exception is thrown during this try-catch block. But after starting codec operation, neither the encoded data nor the output format is output by the encoder even if the input buffers are filled ("dequeueOutputBuffer" always returns -1). The following is the log after starting codec, where some errors are reported:
05-04 11:29:17.071: E/MFC_ENC(1006): SOMXVE_Init: isSEIOn : 0
05-04 11:29:17.071: E/MFC_ENC_APP(1006): SsbSipMfcEncInit] IOCTL_MFC_ENC_INIT failed
05-04 11:29:17.111: A/libc(1006): Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1), thread 1403 (AvcEncBufPrc)
05-04 11:29:19.326: E/VideoCodecInfo(1353): Failed to get track format due to codec error
05-04 11:29:25.606: E/ACodec(1353): OMX/mediaserver died, signalling error!
05-04 11:29:25.606: E/MediaCodec(1353): Codec reported an error. (omx error 0x8000100d, internalError -32)
05-04 11:29:25.606: E/AudioService(2462): Media server died.
05-04 11:29:25.606: E/Camera(1353): Error 100
I finally find these commands to get the supported media profile from Note 2
adb shell cat /system/etc/media_codecs.xml
adb shell cat /system/etc/media_profiles.xml
According to the profile list, the max resolution the h264 encoder can support is 1920x1080
<VideoEncoderCap name="h264" enabled="true"
minBitRate="64000" maxBitRate="20000000"
minFrameWidth="176" maxFrameWidth="1920"
minFrameHeight="144" maxFrameHeight="1080"
minFrameRate="1" maxFrameRate="30" />
My questions:
Why the codec can be config successfully with a unacceptable format? Is that a bug or a limitation related to API level?
I heard that there are some APIs for checking this kind of capabilities available from Android 5.0, but I must support 4.4 in my application. Is there any other way to obtain detailed capability info during run-time? (except for parsing the media_profile.xml)
Any help would be really appreciated.
The mediaserver process crashed, apparently due to a null pointer dereference. This process manages all interaction with the media codec hardware. A crash like this indicates a bug in the platform, possibly in the OEM's driver implementation.
The configure() call should fail if the resolution is not supported. My guess is that this particular implementation has a faulty check.
I don't think there's a way to check to see if a particular width x height pair works in Android 4.4. You can generally rely on common resolutions like 720p, but some sizes have strange behaviors, especially when you get non-multiples of 16.
I'm processing a live stream via MediaCodec and have a scenario where the MediaFormat changes mid-stream (ie: resolution of the video being decoded changes). Given I'm attaching the decoder to a Surface to render it as soon as I detect the change in resolution on the incoming stream I recreate the decoder before feeding it the new resolution buffer (providing it with the proper new MediaFormat).
I've been getting some weird errors which don't give me too much info as to what could be wrong, ie when calling MediaCodec.configure with the new format and same Surface:
android.media.MediaCodec$CodecException: Error 0xffffffea
at android.media.MediaCodec.native_configure(Native Method)
at android.media.MediaCodec.configure(MediaCodec.java:577)
Which when fetching the CodecException.getDiagnosticInfo it shows nothing that I can really use to understand the reason for the failure: android.media.MediaCodec.error_neg_22
I've also noted the following on the logs and found some related information and am wondering if there's something I need to do regarding the Surface itself (like detaching it from the old instance of the decoder being giving it to the new one):
07-09 15:00:17.217 E/BufferQueueProducer( 139): [SurfaceView] connect(P): already connected (cur=3 req=3)
07-09 15:00:17.217 E/MediaCodec( 5388): native_window_api_connect returned an error: Invalid argument (-22)
07-09 15:00:17.218 E/MediaCodec( 5388): configure failed with err 0xffffffea, resetting...
Looks like calling stop() and release() as well as reinitializing any references I had to the getInputBuffers() and getOutputBuffers() did the trick. At least I don't get the messages/exceptions anymore. Now I just need to figure out the Surface reference part as it seems the resized stream (when resolution changes) is still being fit in the original surface dimensions instead of adjusting the Surface for the new resolution.
If your encoder supports adaptive playback, then apparently you can alter some codec paramaters on the fly:
https://stackoverflow.com/a/34427724/1048170