I'm trying to incorporate Pepper's built in Android tablet more in DialogFlow interactions. Particularly, my goal is to open applications installed on the tablet itself for people to use while they're talking with Pepper. I'm aware there is a 'j-tablet-browser' app installed on Pepper's end that can let a person browse the tablet like an ordinary Android device, but I would like to take it one step further and directly launch an Android app, like Amazon's Alexa.
The best solution I can up with is:
User says specific utterance (e.g. "Pepper, open Alexa please")
DialogFlow launches the j-tablet-browser behavior
{
"speak": "Sure, just a second",
"action": "startApp",
"action_parameters": {
"appId": "j-tablet-browser/."
}
}
User navigates the Android menu manually to tap the Alexa icon
My ideal goal is to make the process seamless:
User says specific utterance (e.g. "Pepper, open Alexa please")
DialogFlow launches the Alexa app installed on the Android tablet
Does anyone have an idea how this could be done?
This is quite a broad question so I'll try and focus on the specifics for launching an app with a Dialogflow chatbot. If you don't already have a QiSDK Dialogflow chatbot running on Pepper, there is a good tutorial here which details the full process. If you already have a chatbot implemented I hope the below steps are general enough for you to apply to your project.
This chatbot only returns text results for Pepper to say, so you'll need to make some modifications to allow particular actions to be launched.
Modifying DialogflowDataSource
Step 2 on this page of the tutorial details how to send a text query to Dialogflow and get a text response. You'll want to modify it to return the full reponse object (including actions), not just the text. Define a new function called detectIntentFullResponse for example.
// Change this
return response.queryResult.fulfillmentText
// to this
return response.queryResult
Modifying DialogflowChatbot
Step 2 on this page shows how to implement a QiSDK Chatbot. Add some logic to check for actions in the replyTo function.
var response: DetectIntentResponse? = null
// ...
response = dataSource.detectIntentFullResponse(input, dialogflowSessionId, language)
// ...
return if (reponse.action != null) {
StandardReplyReaction(
ActionReaction(qiContext, response), ReplyPriority.NORMAL
)
} else if (reponse.answer != null) {
StandardReplyReaction(
SimpleSayReaction(qiContext, reponse.answer), ReplyPriority.NORMAL
)
} else {
StandardReplyReaction(
EmptyChatbotReaction(qiContext), ReplyPriority.FALLBACK
)
}
Now make a new Class, ActionReaction. Note that the below is incomplete, but should serve as an example of how you can determine which action to run (if you want others). Look at SimpleSayReaction for more implementation details.
class ActionReaction internal constructor(context: QiContext, private val response: DetectIntentResponse) :
BaseChatbotReaction(context) {
override fun runWith(speechEngine: SpeechEngine) {
if (response.action == "launch-app") {
var appID = response.parameters.app.toString()
// launch app at appID
}
}
}
As for launching the app, various approaches are detailed in other questions, such as here. It is possible to extend this approach to do other actions, such as running or retrieving online data.
Related
I have implemented Google Onetap SignIn in my application. Everything is working fine, the only issue that I am observing is that on certain devices the pop-up often takes 7-10 seconds to display. Especially in case of Sign-In popup.
Since I have multiple login options available in the app it might so happen that before I can show the user his last used google account to login (via OneTap popup), he gets enough time to click on any other option (eg, Facebook) & it becomes a poor experience.
Since this pop-up is displayed by play-services, I don't see how I can optimise this time taken.
As per the code, it seems the call to
contract
.getOneTapClient()
.beginSignIn(getSignInRequest(isRegistering))
is the one taking the most time. It seems the code that queries for user's on device Google Accounts.
Using below code structure. Adding for reference
contract.getOneTapClient().beginSignIn(getSignInRequest(isRegistering))
.addOnSuccessListener { result: BeginSignInResult ->
try
{
contract.startIntentSenderForResult(
result.pendingIntent.intentSender, requestCode,
null, 0, 0, 0, null)
successCallback?.onSuccess(isRegistering, "Rendering Popup")
val timeTaken = if(isRegistering) System.currentTimeMillis() - signUpTime
else System.currentTimeMillis() - signInTime
BBLogUtils.logWithTag(TAG, "Completed in ${timeTaken/1000.0}s")
}
catch (e: IntentSender.SendIntentException)
{
failureCallback?.onFailure(isRegistering, e, ERROR_INTENT_SENDER_EXCEPTION)
}
}
.addOnFailureListener { e: Exception ->
// No saved credentials found.
// OR Temporarily blocked due to too many canceled sign-in prompts.
BBLogUtils.logWithTag(TAG, "Exception | registering=$isRegistering|rCount=$rCount | Error= ${e.message}")
failureCallback?.onFailure(isRegistering, e, ERROR_NO_CREDENTIALS_FOUND)
}
SignIn request object is the standard as prescribed by the docs
private fun getSignInRequest(isRegistering: Boolean): BeginSignInRequest
{
return BeginSignInRequest.builder()
.setGoogleIdTokenRequestOptions(BeginSignInRequest.GoogleIdTokenRequestOptions.builder()
.setSupported(true) // So that we receive the idToken in result
.setServerClientId(contract.getGoogleAndroidClientId())
/*
* true: for Registration ie. showing all accounts
* false: for return user signIn, ie. showing only previously used accounts
**/
.setFilterByAuthorizedAccounts(!isRegistering)
.build())
.build()
}
Another related question to this feature.
On the first launch of the app on device I saw this additional pop-up
Is there someway this can be skipped ?
Answering 2nd part of my own query here.
After a lot search I still haven't bee able to find a workaround for skipping the process. Turns out that this popup is medium for play services to inform the user about the new sign-in experience.
I observed that if I installed another app using Google Onetap (eg. Reddit or PIntrest), the same pop-up appeared there as well. Also, its shown only once to a user, ie. if the pop up was shown in the Reddit app then it won't come in my app & vice-versa.
In addition, if you wan't to repro this scenario, you can clear the storage of Google Play Services. For some duration, it might show error: 16: App is not whitelisted, but after a while, you will get the How It Works pop-up again.
So I'm just wondering why my code isn't working. How do I give AppShorcutIntent a specific intent with an action and data and stuff like that?
This is my code so far:
val appShortcutIntent = AppShortcutIntent.builder()
.setIntentName("actions.intent.OPEN_APP_FEATURE")
.setPackageName("com.app.name")
.setIntentParamName("feature")
.setIntentParamValue("")
.build()
shortcutsClient.lookupShortcut(appShortcutIntent)
.addOnSuccessListener { shortcutLookupResult ->
if (shortcutLookupResult.isShortcutPresent) {
shortcutsClient.createShortcutSettingsIntent().addOnSuccessListener { intent ->
requireActivity().startActivity(intent)
}
return#addOnSuccessListener
}
val signalShortcut = AppShortcutSuggestion.builder()
.setAppShortcutIntent(appShortcutIntent)
.setCommand("feature on")
.build()
shortcutsClient.createShortcutSuggestionIntent(signalShortcut).addOnSuccessListener { intent ->
requireActivity().startActivity(intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK))
}
}
I have tried so many different things and none of it seems to want to work the way I want. I know the question doesn't have anything specific as the parameter value but no matter what I set the param value too it still just doesn't get recognized as a unique intent when I use the shortcut.
The in-app promo library API doesn't deal with Android intents. It deals with Assistant's built-in intents, which are an entirely different things (even though they are both called "intents"). In the example you copied, it refers to the BII called OPEN_APP_FEATURE.
By using this API, you are telling Assistant how to create a shortcut that launches the app using a BII that it is already configured to handle. This BII is important because it carries the ability to recognize natural language queries associated with it. Android intents don't have that context.
How can I go to recent apps menu and How to select a particular app from the recents by using Espresso Android Instrumentation Test
Since selecting app from recent apps menu, it need controller over device. I think it can't be done with Espresso alone.
But you can achieve this by using android uiautomator.
fun selectAppFromRecentApps(appTitle:String){
mDevice.pressHome()
mDevice.pressRecentApps()
var uiSelector = UiSelector().className("android.widget.ScrollView") //scroll view listing all recent apps
var count = mDevice.findObject(uiSelector).childCount
for (i in 0 until count step 1) {
val child = UiScrollable(uiSelector.childSelector(UiSelector().resourceId("com.android.systemui:id/task_view_bar").instance(i))).getChild(UiSelector().resourceId("com.android.systemui:id/title")) // app framelayout
val text = child.text
if (text == appTitle) {
child.click()
break
}
if(i==count-1){
throw RuntimeException("App : "+ appTitle +" not found in
recent apps")
}
}
}
Unfortunately VIGNESHs answer did not work for me. I expect different devices to have different implementations of the overview view, or probably it's the Android version.
Anyway the following worked for me on different devices:
getInstrumentation().waitForIdleSync();
mDevice.pressHome();
mDevice.pressRecentApps();
mDevice.waitForIdle();
// If the application is listed, there needs to be an element with a content description
// containing the package name
if (!mDevice.wait(Until.hasObject(By.descStartsWith(getTargetContext().getPackageName())), 1000))
{
fail("Overview did not open");
}
// As the app to be tested was the last one opened, we can simply press the button again.
mDevice.pressRecentApps();
// Wait until the activity under test is back.
// If you skip that, you might be to early, doing further tests.
mDevice.wait(Until.hasObject(By.pkg(getTargetContext().getPackageName())), 1000);
getInstrumentation().waitForIdleSync();
I'm currently developing an Android application in order to display home screen widgets. Those ones are related to Microsoft Outlook (Events + Messages) in order to show incoming events and unread new messages in a kind of dynamic tiles.
The Msal graph library helps me a lot to authenticate and retrieve in formations which contains an identifier for each event / message results
But now I want to know if the outlook application is installed on the user device and if there is a way to open Outlook when the user click on the widget. Moreover if the user can open the corresponding clicked event or message with the identifier.
For example the Event widget currently displaying a birthday event. The user click on it. Then it opens Outlook and display directly that birthday event.
Regards
I don't think this is officially documented somewhere. But here's what you can do to find out about it.
You can list all Microsoft applications installed on your device...
val packages = context.packageManager
.getInstalledApplications(PackageManager.GET_META_DATA)
for (info in packages) {
if(info.packageName.startsWith("com.microsoft", true)){
Log.d("package name:" + info.packageName)
Log.d("Launch Activity: " + context.packageManager.getLaunchIntentForPackage(info.packageName))
}
}
Take a note of the "launch intent" displayed in the LogCat. You can use that to launch Outlook. Just make sure you don't hard-code those values because Microsoft can change those values at any point, for example the activity class can change. So, instead of doing this...
context.startActivity(
Intent().apply {
action = Intent.ACTION_MAIN
addCategory(Intent.CATEGORY_LAUNCHER)
setPackage("com.microsoft.office.outlook")
component = ComponentName("com.microsoft.office.outlook", "com.microsoft.office.outlook.MainActivity")
}
)
Do this...
context.startActivity(
Intent().apply {
action = Intent.ACTION_MAIN
addCategory(Intent.CATEGORY_LAUNCHER)
component = ComponentName(
outlookLaunchIntent?.component?.packageName,
outlookLaunchIntent?.component?.className
)
setPackage(outlookLaunchIntent.package)
}
)
Also, remember that getLaunchIntentForPackage and component can return null, so make sure you check for null values properly
I am relaying a suggestion from a couple of internal folks:
Please try to open the event using one of the following URLs:
ms-outlook://events/open?restid=%s&account=test#om.com (if you have a regular REST id)
ms-outlook://events/open?immutableid=%s&account=test#om.com (if you are using an immutable id)
Since immutable IDs are still in preview stage in Microsoft Graph, and customers should not use preview APIs in their production apps, I think option #1 applies to your case.
Please reply here if the URL works, or not, and if you have other related questions. I requested the couple of folks to keep an eye on this thread as well.
Well, i managed to open the outlook android application with the help of your code #Leo. As im not developping with Kotlin, ill post the JAVA code below :
Intent outlookLaunchIntent = context.getPackageManager().getLaunchIntentForPackage("com.microsoft.office.outlook");
if (outlookLaunchIntent != null) {
context.startActivity(outlookLaunchIntent );
}
Below code to open event/message in a web browser provided by webLink property of the graph API. (I only test for event and the url provided not working. Ill post a new issue on StackOverFlow for that but you already see the issue over there : https://github.com/microsoftgraph/microsoft-graph-docs/issues/4203
try {
Intent webIntent = new Intent(Intent.ACTION_VIEW).setData(Uri.parse(calendarWebLink));
webIntent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
context.startActivity(webIntent);
} catch (RuntimeException e) {
// The url is invalid, maybe missing http://
e.printStackTrace();
}
However im still stuck on the decicive goal of my widget item click which is to open the relative event/email in the Microsoft Outlook Android application.
Microsoft Outlook Android app contains widgets which can achieve what im looking for. So i wonder if it is possible to list its broadcast receivers.
The best thing i found is an old manifest for that app but it doesnt help me.
https://gist.github.com/RyPope/df0e61f477af4b73865cd72bdaa7d8c2
Hi may you try to open the event using one of the url:
ms-outlook://events/open?restid=%s&account=test#om.com (If the
user is having rest id)
ms-outlook://events/open?immutableid=%s&account=test#om.com (If
the user is having immutable id)
After setting up a printer through services plugin (e.g. Steps 1..3) on Android OS, my working flow should:
Press "print" button
Show dialog with available printers (previously defined on Android settings)
Execute pdf printing
Is it possible to acquire this list of available printers inside my own application? How?
So far, closest solution I got running through google's documentation was to open my pdf on a web preview and from there let Android handle everything. However, if possible, I wouldn't like to break my UX. After selecting my printer, ideal scenario would be to print pdfs directly.
Thanks in advance
------- STEPS -------
Android Printing Settings
Installed Services
Printers List
Is it possible to acquire this list of available printers inside my own application? How?
Yes you can use PrintService: https://developer.android.com/reference/android/printservice/PrintService
A print service is responsible for discovering printers, adding discovered printers, removing added printers, and updating added printers.
The Android docs also have (somewhat outdated, but still useful) lessons on using the printing-related APIs: https://developer.android.com/training/printing
This sample includes code related to printing a PDF: https://developer.android.com/training/printing/custom-docs
Sample code from the docs:
// Connect to the print manager
private fun doPrint() {
activity?.also { context ->
// Get a PrintManager instance
val printManager = context.getSystemService(Context.PRINT_SERVICE) as PrintManager
// Set job name, which will be displayed in the print queue
val jobName = "${context.getString(R.string.app_name)} Document"
// Start a print job, passing in a PrintDocumentAdapter implementation
// to handle the generation of a print document
printManager.print(jobName, MyPrintDocumentAdapter(context), null)
}
}
// Compute print document info
override fun onLayout(
oldAttributes: PrintAttributes?,
newAttributes: PrintAttributes,
cancellationSignal: CancellationSignal?,
callback: LayoutResultCallback,
extras: Bundle?
) {
// Create a new PdfDocument with the requested page attributes
pdfDocument = PrintedPdfDocument(activity, newAttributes)
// Respond to cancellation request
if (cancellationSignal?.isCanceled == true) {
callback.onLayoutCancelled()
return
}
// Compute the expected number of printed pages
val pages = computePageCount(newAttributes)
if (pages > 0) {
// Return print information to print framework
PrintDocumentInfo.Builder("print_output.pdf")
.setContentType(PrintDocumentInfo.CONTENT_TYPE_DOCUMENT)
.setPageCount(pages)
.build()
.also { info ->
// Content layout reflow is complete
callback.onLayoutFinished(info, true)
}
} else {
// Otherwise report an error to the print framework
callback.onLayoutFailed("Page count calculation failed.")
}
}
So far, no proper solution was found. Possible alternatives are:
Try to access printers through shared preferences stored by services plugins (e.g. PrintHand).
If you just to want to ease acquisition of printer's IP address & case you're using devices with bar code scanner, you could place a label on the printer with its IP Address and make a scan to store the address on a session/singleton field.
I took the second path.