Develop Reference

Is ExoPlayer’s audio offload mode supported while using MediaSessionService from Media3?

I am working on a native music player app for android using ExoPlayer and MediaSessionService from Media3. Now I want to make playback more energy efficient while the screen is off by using experimentalSetOffloadSchedulingEnabled, but it seems like I’m not able to get the offloading to work.
From the main activity of the app I send ACTION_START_AUDIO_OFFLOAD in the onStop() method to my service (the relevant parts of the service are show below), and ACTION_STOP_AUDIO_OFFLOAD in the onStart() method. In this way I have been able to get correct true/false responses from the onExperimentalOffloadSchedulingEnabledChanged listener, but I do not get any responses from the onExperimentalOffloadedPlayback or onExperimentalSleepingForOffloadChanged listeners, so it seems like the player never enters power saving mode.
My tests were made with Media3 version 1.0.0-beta03 on Android 13 (emulator) and Android 10 (phone) using MP3 files. I am aware that Media3 is in beta and that the offload scheduling method is experimental, but I'm not sure if that is the limitation or if I have done something wrong. Any ideas what could be the issue?
#androidx.media3.common.util.UnstableApi
class PlaybackService: MediaSessionService(), MediaSession.Callback {
private val listener = object : ExoPlayer.AudioOffloadListener {
override fun onExperimentalOffloadSchedulingEnabledChanged(offloadSchedulingEnabled: Boolean) {
Log.d("PlaybackService","offloadSchedulingEnabled: $offloadSchedulingEnabled")
super.onExperimentalOffloadSchedulingEnabledChanged(offloadSchedulingEnabled)
}
override fun onExperimentalOffloadedPlayback(offloadedPlayback: Boolean) {
Log.d("PlaybackService","offloadedPlayback: $offloadedPlayback")
super.onExperimentalOffloadedPlayback(offloadedPlayback)
}
override fun onExperimentalSleepingForOffloadChanged(sleepingForOffload: Boolean) {
Log.d("PlaybackService","sleepingForOffload: $sleepingForOffload")
super.onExperimentalSleepingForOffloadChanged(sleepingForOffload)
}
}
private lateinit var player: ExoPlayer
private var mediaSession: MediaSession? = null
override fun onCreate() {
super.onCreate()
player = ExoPlayer.Builder(
this,
DefaultRenderersFactory(this)
.setEnableAudioOffload(true)
)
.setAudioAttributes(AudioAttributes.DEFAULT, /* handleAudioFocus = */ true)
.setHandleAudioBecomingNoisy(true)
.setSeekBackIncrementMs(10_000)
.setSeekForwardIncrementMs(10_000)
.setWakeMode(C.WAKE_MODE_LOCAL)
.build()
player.addAudioOffloadListener(listener)
mediaSession = MediaSession
.Builder(this, player)
.setCallback(this)
.build()
}
override fun onGetSession(controllerInfo: MediaSession.ControllerInfo): MediaSession? =
mediaSession
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
when(intent?.action) {
ACTION_START_AUDIO_OFFLOAD -> startAudioOffload()
ACTION_STOP_AUDIO_OFFLOAD -> stopAudioOffload()
}
return super.onStartCommand(intent, flags, startId)
}
private fun startAudioOffload() {
player.experimentalSetOffloadSchedulingEnabled(true)
}
private fun stopAudioOffload() {
player.experimentalSetOffloadSchedulingEnabled(false)
}
override fun onDestroy() {
mediaSession?.run {
player.release()
release()
mediaSession = null
}
super.onDestroy()
}
companion object {
const val ACTION_START_AUDIO_OFFLOAD = "ACTION_START_AUDIO_OFFLOAD"
const val ACTION_STOP_AUDIO_OFFLOAD = "ACTION_STOP_AUDIO_OFFLOAD"
}
}

Can we do Speech Recognition and Video Capture in Android using camerax simultaneously?

I want to implement an app that can convert text to speech and record videos (with audio) simultaneously.
But when I am calling both function either one of them working (the recent one that has been called). Can anyone suggest some ways to implement these two together.
`
speechRecognizer = SpeechRecognizer.createSpeechRecognizer(activity)
val speechRecognizerIntent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH)
speechRecognizerIntent.putExtra(
RecognizerIntent.EXTRA_LANGUAGE_MODEL,
RecognizerIntent.LANGUAGE_MODEL_FREE_FORM,
)
speechRecognizerIntent.putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_COMPLETE_SILENCE_LENGTH_MILLIS,10000)
// speechRecognizerIntent.putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_MINIMUM_LENGTH_MILLIS,30000)
speechRecognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault())
speechRecognizerIntent.putExtra("android.speech.extra.GET_AUDIO_FORMAT", "audio/MP3")
speechRecognizerIntent.putExtra("android.speech.extra.GET_AUDIO", true)
speechRecognizer.setRecognitionListener(object : RecognitionListener {
override fun onReadyForSpeech(bundle: Bundle?) {
speechRecognizer.startListening(speechRecognizerIntent)
}
override fun onBeginningOfSpeech() {}
override fun onRmsChanged(v: Float) {}
override fun onBufferReceived(bytes: ByteArray?) {}
override fun onEndOfSpeech() {
// changing the color of our mic icon to
// gray to indicate it is not listening
// #FF6D6A6A
}
override fun onError(i: Int) {}
override fun onResults(bundle: Bundle) {
}
override fun onPartialResults(bundle: Bundle) {
val result = bundle.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
if (result != null) {
for (i in 0 until result.size){
text.add(result[0])
Log.d("Record",result[0])
//binding.tvtext.text = binding.tvtext.text.toString() + result[0]
}
}
}
override fun onEvent(i: Int, bundle: Bundle?) {}
})
speechRecognizer.startListening(speechRecognizerIntent)
}
`

How to detect android speech is in offline or online

I follow this question to do the offline speech on android.
I downloaded the language in google voice and it can work in offline.
The problem is that I want to know it's current running on offline or online speech, (just like Apple speech to text, there is an api to check for that) to display the speech stream in my app correctly
I wonder is there anyway to do that?
Here is my code:
val intentSpeech = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH)
intentSpeech.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, "en-US")
intentSpeech = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH)
intentSpeech.putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true);
intentSpeech.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
intentSpeech.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE,
packageName)
val recognizer = SpeechRecognizer.createSpeechRecognizer(this)
recognizer.setRecognitionListener(this)
P/s: I can see the Read Along app by google works perfectly in offline or online mode.
I'm trying to do the same with the android speech api. Is it possible?

For offline speech to text, you can use Google's default STT model but it seems to be non-continuous.
private fun startSpeechToText() {
val speechRecognizer = SpeechRecognizer.createSpeechRecognizer(this)
val speechRecognizerIntent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH)
speechRecognizerIntent.putExtra(
RecognizerIntent.EXTRA_LANGUAGE_MODEL,
RecognizerIntent.LANGUAGE_MODEL_FREE_FORM
)
speechRecognizerIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault())
speechRecognizer.setRecognitionListener(object : RecognitionListener {
override fun onReadyForSpeech(bundle: Bundle?) {}
override fun onBeginningOfSpeech() {}
override fun onRmsChanged(v: Float) {}
override fun onBufferReceived(bytes: ByteArray?) {}
override fun onEndOfSpeech() {}
override fun onError(i: Int) {}
override fun onResults(bundle: Bundle) {
val result = bundle.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
if (result != null) {
// result[0] will give the output of speech
}
}
override fun onPartialResults(bundle: Bundle) {}
override fun onEvent(i: Int, bundle: Bundle?) {}
})
// starts listening ...
speechRecognizer.startListening(speechRecognizerIntent)
}
If you don't want to use google as it requires to download the offline model for speech to text.
The other option for offline STT is the Vosk API as it has pretrained models in english and other language for live STT.
https://github.com/alphacep/vosk-android-demo
Reference: https://www.geeksforgeeks.org/offline-speech-to-text-without-any-popup-dialog-in-android/

Play music automatically when starting app

I was wondering how to make the music play automatically when starting the app and how to make it stop playing in the background when pressing the home button. Right now, it starts and stops by pressing the toggle button. I was also wondering if its possible to automatically switch to other music when going to another activity?
MainActivity.kt
private lateinit var player: MediaPlayer
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
val secondActivity = findViewById<Button>(R.id.secondActivity)
secondActivity.setOnClickListener {
val intent = Intent(this, MainActivity2::class.java)
startActivity(intent)
}
val toggle: ToggleButton = findViewById(R.id.toggleButton)
toggle.setOnCheckedChangeListener { _, isChecked ->
val svc = Intent(this, MusicService::class.java)
if (isChecked) {
startService(svc)
} else {
stopService(svc)
}
}
}
MusicService.kt
class MusicService : Service() {
private lateinit var player: MediaPlayer
override fun onBind(intent: Intent?): IBinder? {
TODO("Return the communication channel to the service.")
return null
}
override fun onCreate() {
super.onCreate()
player = MediaPlayer.create(this, R.raw.music)
player.setLooping(true)
}
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
player.start()
return START_STICKY
}
override fun onDestroy() {
super.onDestroy()
player.stop()
}
}

Depending on your specific requirements, you should or shouldn't use a MediaService as part of your solution.
To be more clear, a Service should only be used if you want the audio to keep going whenever you're outside the app. This solution will usually be accompanied by a media Notification which you should populate with controls, image assets, etc. (Think of Spotify or SoundCloud) If this is the solution you're looking for, take a look at this doc page from Google and follow it through. Beware that this is a longer and tougher process to maintain.
On the other hand, if all you want to do is play music/sounds while your user is inside your app, then a simple
private lateinit var localMedia: MediaPlayer
override fun onCreate() {
...
localMedia = MediaPlayer.create(context, R.raw.your_audio_file)
}
override fun onResume() {
...
localMedia.start()
}
override fun onPause() {
...
localMedia.release()
}
Furthermore, if you want different audio files to be played on different Activities/Fragments, you might want to abstract the code I provided above into it's own Manager class or so and access it the same but changing the specific .mp3 file (or whatever format) as you see fit.
EDIT:
For a Manager class, you'll have to create your own functions and handle the MediaPlayer inside of it
private class MediaPlayerManager(private val context: Context) {
private lateinit var mediaPlayer: MediaPlayer
fun setupPlayer() {
mediaPlayer = MediaPlayer.create(context, R.raw.your_audio_file)
}
fun play() {
mediaPlayer.start()
}
fun stop() {
mediaPlayer.stop()
}
}
And call these functions from their respective lifecycle method inside your Activity/Fragment, depending on your specific needs
class YourActivity {
val mediaPlayerManager = MediaPlayerManager(context)
override onCreate() {
...
mediaPlayerManager.setupPlayer()
}
override fun onResume() {
...
mediaPlayerManager.play()
}
override fun onPause() {
...
mediaPlayerManager.stop()
}
}
I should add that I'm not necessarily providing a fully-fledged answer here, but a starting point for you to massage to your own needs. The Manager class is nothing but an abstraction of the concept I'm trying to communicate. Lastly, if you want to use a different audio resource file in another Activity/Fragment, you would have to create a method to re-assign the MediaPlayer object inside it with the appropriate file.
E.g.
fun setupPlayer(audioRes: Int) {
mediaPlayer = MediaPlayer.create(context, audioRes)
}

I have done this before
create a Class and named it (for example I named it C) and extend that from Application like following (don't forget put android:name=".C" in <application> tag in manifest.xml):
class C:Application() {
fun onCreate() {
super.onCreate()
context = getApplicationContext()
app = this
}
companion object {
private val context:Context
var currentActivity:Activity
var currentActivities:ArrayList<Activity> = ArrayList()
var handler:Handler
var app:C
fun get():C {
return app
}
fun getContext():Context {
if (currentActivity != null)
{
return currentActivity
}
return context
}
}
}
I created a class for parent of AppCompatActivity and (named UAppCompactActivity)extend it form AppCompatActivity then extend all activities from UAppCompactActivity:
abstract class UAppCompatActivity:AppCompatActivity() {
fun onCreate(#Nullable savedInstanceState:Bundle, #Nullable persistentState:PersistableBundle) {
super.onCreate(savedInstanceState, persistentState)
}
protected fun onResume() {
super.onResume()
C.setCurrentActivity(this)
C.currentActivities.add(this)
/* you can play your music here or do any action you desired */
}
protected fun onPause() {
super.onPause()
C.currentActivities.remove(this)
/*
you can stop your music here or do any action you desired
if (UBase.currentActivities.size() === 0)
G.backgroundMusics.get(G.app.musicNumberBackAndNowPlay).pause()
else
play music or switch to new music
*/
}
}

How can I make speech recognition listen forever?

I have a program here, This is a function that is called on a click of a button. I would like the speech recognition to keep recognizing speech until stoplistening() is called. I keep getting the error codes 6, 7 and every time I speak it stops listening after a few seconds of silence, I even defined the silence and input values:
listenerintent.putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_POSSIBLY_COMPLETE_SILENCE_LENGTH_MILLIS , 200000)
listenerintent.putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_MINIMUM_LENGTH_MILLIS , 5000000)
but the application still closes after 2 seconds of silence if something is said before the silence. If nothing is said then the speech-recognition listens for the maximum amount of time I defined above.
I would like to have the speech recognition to be active regardless of the silence after something is recognized. Any parameter I am missing that could solve this issue? or is this library incapable of going forever and must stop after something is recognized? Is there another end pointer I am overlooking?
var recogobj = SpeechRecognizer.createSpeechRecognizer(this)
var listenerintent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH)
listenerintent.putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_POSSIBLY_COMPLETE_SILENCE_LENGTH_MILLIS , 200000)
listenerintent.putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_MINIMUM_LENGTH_MILLIS , 5000000)
listenerintent.putExtra(RecognizerIntent.EXTRA_SPEECH_INPUT_POSSIBLY_COMPLETE_SILENCE_LENGTH_MILLIS, 5000000)
val recoglistener = object : RecognitionListener {
override fun onReadyForSpeech(params: Bundle?) {
}
override fun onRmsChanged(rmsdB: Float) {
}
override fun onBufferReceived(buffer: ByteArray?) {
}
override fun onPartialResults(partialResults: Bundle?) {
Toast.makeText(this#Recordinit , RecognizerIntent.EXTRA_PARTIAL_RESULTS , Toast.LENGTH_LONG).show()
var data = partialResults!!.getStringArray(RecognizerIntent.EXTRA_PARTIAL_RESULTS)
for (result in data.orEmpty()){
if (result == "test") {
recogobj.stopListening()
Toast.makeText(this#Recordinit, "ok ", Toast.LENGTH_SHORT).show()
}
}
}
override fun onEvent(eventType: Int, params: Bundle?) {
}
override fun onBeginningOfSpeech() {
}
override fun onEndOfSpeech() {
}
override fun onError(error: Int) {
if (error ==6 || error == 7){
recogobj.startListening(listenerintent)
}
else{
Toast.makeText(this#Recordinit ,"${error}" , Toast.LENGTH_LONG).show()
}
}
override fun onResults(results: Bundle?) {
}
}
recogobj.setRecognitionListener(recoglistener)
recogobj.startListening(listenerintent)
}