How to implement an Assistant with Google Assist API - android

I have been checking out and reading about Google Now on Tap (from http://developer.android.com/training/articles/assistant.html).
It was very interesting to find from that article that Now on Tap is based on Google's Assist API bundled with Marshmallow and it seems possible for us to develop our own assistant (the term Google used in the article to refer to app like Now on Tap) using the API.
However, the mentioned article only very briefly discusses how to use Assist API and I couldn't find any additional information about how to use it to develop a custom assistant even after spending a few days searching for it on the Internet. No documentation and no example.
I was wondering if any of you have experience with Assist API that you could share? Any help appreciated.
Thanks

You can definitely implement a personal assistant just like the Google Now on Tap using the Assist API starting Android 6.0. The official developer (http://developer.android.com/training/articles/assistant.html) guide tells exactly how you should implement it.
Some developers may wish to implement their own assistant. As shown in Figure 2, the active assistant app can be selected by the Android user. The assistant app must provide an implementation of VoiceInteractionSessionService and VoiceInteractionSession as shown in this example and it requires the BIND_VOICE_INTERACTION permission. It can then receive the text and view hierarchy represented as an instance of the AssistStructure in onHandleAssist(). The assistant receives the screenshot through onHandleScreenshot().
Commonsware has four demos for basic Assist API usage. The TapOffNow (https://github.com/commonsguy/cw-omnibus/tree/master/Assist/TapOffNow) should be enough to get you started.
You don't have to use the onHandleScreenshot() to get the relevant textual data, the AssistStructure in onHandleAssist() will give you a root ViewNode which usually contains all you can see on the screen.
You probably need to also implement some sorts of function to quickly locate the specific ViewNode that you want to focus on using recursive search on the children from this root ViewNode.

There is a complete example here but it's too complicated to start.
This is my example works on android 7.1.1
AndroidManifest.xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.eaydin79.voiceinteraction">
<application
android:allowBackup="true"
android:icon="#mipmap/ic_launcher"
android:label="#string/app_name"
android:roundIcon="#mipmap/ic_launcher_round"
android:theme="#style/AppTheme" >
<service
android:name="voiceInteractionService"
android:permission="android.permission.BIND_VOICE_INTERACTION" >
<meta-data
android:name="android.voice_interaction"
android:resource="#xml/interaction_service" />
<intent-filter>
<action android:name="android.service.voice.VoiceInteractionService" />
</intent-filter>
</service>
<service
android:name="voiceInteractionSessionService"
android:permission="android.permission.BIND_VOICE_INTERACTION" >
</service>
</application>
</manifest>
this is interaction_service.xml file stored in res\xml folder
<?xml version="1.0" encoding="utf-8"?>
<voice-interaction-service xmlns:android="http://schemas.android.com/apk/res/android"
android:sessionService="com.eaydin79.voiceinteraction.voiceInteractionSessionService"
android:recognitionService="com.eaydin79.voiceinteraction.voiceInteractionService"
android:supportsAssist="true" />
voiceInteractionService.java
package com.eaydin79.voiceinteraction;
import android.service.voice.VoiceInteractionService;
import android.service.voice.VoiceInteractionSession;
public class voiceInteractionService extends VoiceInteractionService {
#Override
public void onReady() {
super.onReady();
}
}
voiceInteractionSessionService.java
package com.eaydin79.voiceinteraction;
import android.os.Bundle;
import android.service.voice.VoiceInteractionSession;
import android.service.voice.VoiceInteractionSessionService;
public class voiceInteractionSessionService extends VoiceInteractionSessionService {
#Override
public VoiceInteractionSession onNewSession(Bundle bundle) {
return new voiceInteractionSession(this);
}
}
voiceInteractionSession.java
package com.eaydin79.voiceinteraction;
import android.app.VoiceInteractor;
import android.content.Context;
import android.os.Bundle;
import android.service.voice.VoiceInteractionSession;
import android.media.AudioManager;
public class voiceInteractionSession extends VoiceInteractionSession {
voiceInteractionSession(Context context) {
super(context);
}
#Override
public void onShow(Bundle args, int showFlags) {
super.onShow(args, showFlags);
//whatever you want to do when you hold the home button
//i am using it to show volume control slider
AudioManager audioManager = (AudioManager) getContext().getSystemService(Context.AUDIO_SERVICE);
if (audioManager != null) audioManager.adjustStreamVolume(AudioManager.STREAM_MUSIC, AudioManager.ADJUST_SAME, AudioManager.FLAG_SHOW_UI);
hide();
}
}

Related

App not being recognized by Android Auto

So, I followed the official guide here https://developer.android.com/training/auto/start/index.html to create a very basic Android Auto Audio App. For the moment it does nothing, other then declaring what needs to be declared in the manifest and implementing empty onGetRoot() and onLoadChildren().
Problem is, that it is not being recognized by the Android Auto app.
Any idea where to get a working example? What could be wrong?
Manifest:
<service
android:name=".MyService"
android:enabled="true"
android:exported="true">
<intent-filter>
<action android:name=
"android.media.browse.MediaBrowserService"/>
</intent-filter>
</service>
automotive_app_desc.xml:
<automotiveApp>
<uses name="media" />
</automotiveApp>
service:
public class MyService extends MediaBrowserServiceCompat {
public static final String MEDIA_ID_ROOT = "__ROOT__";
#Override
public BrowserRoot onGetRoot(String clientPackageName, int clientUid,
Bundle rootHints) {
//TODO: check if the client is allow access
return new BrowserRoot(MEDIA_ID_ROOT, null);
}
#Override
public void onLoadChildren(final String parentMediaId,
final Result<List<MediaBrowserCompat.MediaItem>> result) {
// Assume for example that the music catalog is already loaded/cached.
List<MediaBrowserCompat.MediaItem> mediaItems = new ArrayList<>();
// Check if this is the root menu:
if (MEDIA_ID_ROOT.equals(parentMediaId)) {
// build the MediaItem objects for the top level,
// and put them in the mediaItems list
} else {
// examine the passed parentMediaId to see which submenu we're at,
// and put the children of that menu in the mediaItems list
}
result.sendResult(mediaItems);
}
You have to go to Android Auto settings, tap many times on the Version entry (the last one) to unlock Developer settings. Then tap on Developer settings menu item and enable Unknown sources. Restart Android Auto and if your app it's ok it will be listed. Worked for me
I didn't see this included in your snippet from the manifest, but double check that this line is also there.
<application>
...
<meta-data android:name="com.google.android.gms.car.application"
android:resource="#xml/automotive_app_desc"/>
...
</application>
I created a sample app matching everything you have (plus the line above), and it appears in Android Auto on the mobile device, as well as the Desktop Head Unit.

Using accessibility

I want to create an android app which will "scan" other applications'layouts and find out if there are buttons etc.
Some apps, like whatsapp, doesn't allow third parts to move into the app, so I though that using accessibility could be the solution.
Now the problem is that I've never used Accessibility, so can someone of you can please show me how to "scan" an app's layout to find buttons?
Thanks a lot
What you're looking for is an Accessibility Service. Configuring an accessibility service is somewhat complicated. I have set up a repository of accessibility boilerplate code that sets up an AccessibilityService that logs the node heirarchy to LogCat, and has the default project settings activity set up as it's settings activity. Here are some of the highlights, as I dislike just posting github repos as answers. Note that I use a lot of my own libraries. CLog is a logging library and AndroidAccessibilityUtils wraps node infos with some common utility functions. You can find references to the dependencies in the build.gradle file on the github repo at the bottom. Here are some code highlights.
Your manifest.xml file is going to be significantly different from an Activity, and should contain an entry like this in your Application element:
<application .... >
...
<service
android:name="com.moba11y.basicaccessibilityservice.BasicAccessibilityService"
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE" >
<intent-filter>
<action android:name="android.accessibilityservice.AccessibilityService" />
</intent-filter>
<meta-data
android:name="android.accessibilityservice"
android:resource="#xml/service_config" />
</service>
...
</application>
First you need to create a subclass of Accessibility Service:
public class BasicAccessibilityService extends AccessibilityService {
static {
CLog.initialize(BasicAccessibilityService.class.getSimpleName(), BuildConfig.DEBUG);
}
#Override
public void onAccessibilityEvent(AccessibilityEvent event) {
CLog.d(event.toString());
switch (event.getEventType()) {
//On Gesture events print out the entire view heirarchy!
case AccessibilityEvent.TYPE_GESTURE_DETECTION_START:
CLog.d(A11yNodeInfo.wrap(getRootInActiveWindow()).toViewHeirarchy());
default: {
//If the event has a source, let's print it out separately.
if (event.getSource() != null) {
CLog.d(A11yNodeInfo.wrap(event.getSource()).toViewHeirarchy());
}
}
}
}
#Override
public void onInterrupt() {
CLog.e("Service Interrupted: Have never actually had this happen.");
}
}
That is the bulk of the highlights. You also should have a "service_config" XML with properties, as is referenced in the changes to the Manifest XML file. More details, and a reasonable starer point can be found in the open source repo on GitHub.
https://github.com/chriscm2006/Android-Accessibility-Service-Boilerplate

Paid and Free App using Android Library Project - How to code effectively?

I am planning to release a paid version of my free android app using the android library project approach.
http://developer.android.com/tools/projects/index.html
My app has several stand-alone (non-user interface) classes and resources that are easy to reuse between the paid and free version.
My question is about the best way to manage the user interface logic (code in Activity classes). Let's say my free app has one button and my paid app has two buttons in the same activity. Is the best way to achieve this is to have the following setup?
Android library project
1)Layout with one button
2) an Ativity.java file containing logic for when the button is clicked
Free app
Use layout and source code from the library project
Paid app
1) A new layout file with two buttons
2) A new Activity.java which has the exact same code for handling button1 clicks and new code for handling button2 clicks.
This does not seem right because button1's logic in paid app seems to be a wasteful copy ... Is there a better way to do this?
You can make a single project library with all the functionality
Just you need a one method that can identify that if application is paid or free
For that follow the steps
1) Create a new application suppose testFree
2) Create a new Application Class as follow in the library project
package com.example.testlib;
import android.app.Application;
public class App extends Application{
private static App mInstance;
public App() {
mInstance = this;
}
public static App getInstance() {
return mInstance;
}
public boolean isFree()
{
return true;
}
}
3) create a new application suppose testPaid
4) create a new Application class in the testPaid Application as Follow
package com.example.testpaid;
import com.example.testlib.App;
public class AppPaid extends App {
#Override
public boolean isFree() {
// TODO Auto-generated method stub
return false;
}
}
5) set Application name on testFree app to Application class that we created on the library class and also set main and launcher activity from library class
<application
android:name="com.example.testlib.App"
....
<activity
android:name="com.example.testlib.MainActivity"
android:label="#string/app_name" >
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
6) now set the application name to create application class of testPaid app and also main and launcher activity from library project as follows
<application
android:name="com.example.testpaid.AppPaid"
....
<activity
android:name="com.example.testlib.MainActivity"
android:label="#string/app_name" >
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
</application>
7) All you have set now in any class of library project you have a method that will check if you app is free or paid you can check as following way and based on that you can make visible some paid functionality to paid app and some free to free
if(App.getInstance().isFree())
{
Toast.makeText(getActivity(), "Free App", Toast.LENGTH_SHORT).show();
}
else
{
Toast.makeText(getActivity(), "Paid App", Toast.LENGTH_SHORT).show();
}
Let me know if you still find any problem.....

Android Building Your First App Basics

I'm having a difficult time getting the results that should be displayed as described in the second step of the First App project on the android developer website: developer.android.com/training/basics/firstapp/starting-activity.html#receivetheintent
I've created the first intent and copied all other code however upon running the project I receive a blank android screen with no input elements. Here's what the emulator looks like:
http://s1278.beta.photobucket.com/user/cetmrw791346/media/1_zps116f17a9.png.html
I've set the Run Configuration under the Nexus type with an allocation of 512MB RAM so I'm not exactly sure if this might have something to do with an installation problem regarding the Java SDK (7.0) (JDK not the JRE) or if it could possible be the Android SDK. I'm fairly certain I've set everything up correctly. I'm using The Eclipse (I'm pretty sure it's an IDE) for Mobile Developers then creating a new Android App project from File, New Project. Here's what my Package Explorer looks like: http://s1278.beta.photobucket.com/user/cetmrw791346/media/2_zps0f2b94a2.png.html
I'm unsure as how to further troubleshoot the problem and would really appreciate any additional help. Thanks again for the help.
And here are the relevant files:
**AndroidManifest.xml**
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.firstapp"
android:versionCode="1"
android:versionName="1.0" >
<uses-sdk
android:minSdkVersion="8"
android:targetSdkVersion="17" />
<application
android:allowBackup="true"
android:icon="#drawable/ic_launcher"
android:label="#string/app_name"
android:theme="#style/AppTheme" >
<activity
android:name="com.example.firstapp.MainActivity"
android:label="#string/app_name" >
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
<activity
android:name="com.example.firstapp.DisplayMessageActivity"
android:label="#string/title_activity_display_message"
android:parentActivityName="com.example.firstapp.MainActivity" >
<meta-data
android:name="android.support.PARENT_ACTIVITY"
android:value="com.example.firstapp.MainActivity" />
</activity>
</application>
</manifest>
**MainActivity.java**
package com.example.firstapp;
import android.app.Activity;
import android.content.Intent;
import android.os.Bundle;
import android.view.Menu;
import android.view.View;
import android.widget.EditText;
public class MainActivity extends Activity {
public final static String EXTRA_MESSAGE = "com.example.myfirstapp.MESSAGE";
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
}
#Override
public boolean onCreateOptionsMenu(Menu menu) {
// Inflate the menu; this adds items to the action bar if it is present.
getMenuInflater().inflate(R.menu.main, menu);
return true;
}
/** Called when the user clicks the Send button */
public void sendMessage(View view) {
Intent intent = new Intent(this, DisplayMessageActivity.class);
EditText editText = (EditText) findViewById(R.id.edit_message);
String message = editText.getText().toString();
intent.putExtra(EXTRA_MESSAGE, message);
startActivity(intent);
}
}
**activity_main.xml**
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="horizontal"
tools:context=".MainActivity" >
<EditText android:id="#+id/edit_message"
android:layout_weight="1"
android:layout_width="0dp"
android:layout_height="wrap_content"
android:hint="#string/edit_message" />
<Button
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="#string/button_send"
android:onClick="sendMessage" />
</LinearLayout>
**strings.xml**
<?xml version="1.0" encoding="utf-8"?>
<resources>
<string name="app_name">My First App</string>
<string name="edit_message">Enter a message</string>
<string name="button_send">Send</string>
<string name="menu_settings">Settings</string>
<string name="title_activity_main">MainActivity</string>
<string name="title_activity_display_message">DisplayMessageActivity</string>
<string name="action_settings">Settings</string>
<string name="hello_world">Hello world!</string>
</resources>
activity_display_message.xml
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:paddingBottom="#dimen/activity_vertical_margin"
android:paddingLeft="#dimen/activity_horizontal_margin"
android:paddingRight="#dimen/activity_horizontal_margin"
android:paddingTop="#dimen/activity_vertical_margin"
tools:context=".DisplayMessageActivity" >
<TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="#string/hello_world" />
</RelativeLayout>
**DisplayMessageActivity.java**
package com.example.firstapp;
import android.os.Bundle;
import android.app.Activity;
import android.view.Menu;
import android.view.MenuItem;
import android.support.v4.app.NavUtils;
import android.annotation.TargetApi;
import android.os.Build;
public class DisplayMessageActivity extends Activity {
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_display_message);
// Show the Up button in the action bar.
setupActionBar();
}
/**
* Set up the {#link android.app.ActionBar}, if the API is available.
*/
#TargetApi(Build.VERSION_CODES.HONEYCOMB)
private void setupActionBar() {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.HONEYCOMB) {
getActionBar().setDisplayHomeAsUpEnabled(true);
}
}
#Override
public boolean onOptionsItemSelected(MenuItem item) {
switch (item.getItemId()) {
case android.R.id.home:
// This ID represents the Home or Up button. In the case of this
// activity, the Up button is shown. Use NavUtils to allow users
// to navigate up one level in the application structure. For
// more details, see the Navigation pattern on Android Design:
//
// http://developer.android.com/design/patterns/navigation.html#up-vs-back
//
NavUtils.navigateUpFromSameTask(this);
return true;
}
return super.onOptionsItemSelected(item);
}
}
It doesn't look like your emulator has started up yet.
Wait for it to boot to the homescreen, and then your app should run.
A couple of points:
It seems you aren't letting your app actually start. The first screen you posted is just the "boot" screen of your emulator
Have you tried switching to the debug perspective in Eclipse? At the bottom you'll see what Eclipse is actually doing. You have to switch to the console view and/or view the logcat to see a bit more detail, but that should actually help you in your efforts.
If you have trouble starting up your Emulator, you can test it by itself. You have (for instance) the option to select the second of the two Android icons that are in the upper bar in Eclipse. It should be the one that says "Android Virtual Device Manager". When you select it, it shows you your configured Emulators, though you can configure new ones as well. You can start one of those in advance and see how they work.
It seems that you have not still executed your app (the emulator is still booting).
I'm quite new to both Java and Android (just a few weeks on it, following an online course) but I found the emulator really slow and I'd really advice you to plug in a real device and use it for running the app.
When connecting my Galaxy S2 to Linux and clicking RUN, Eclipse allows you to use it for execute the app. In the examples of the course I'm following, the apps starts in just a couple of seconds, while running them in the emulator is painful.
If you still need to use the emulator, you can speed it up by editing the properties of your virtual device in ADT and switching on the flag "[X] Use snapshot". By activating this flag, you won't "power off" and "power on" the "virtual device" each time: when you close it, its current state will be saved to disk as an snapshot and when you run it again, you won't need to wait for it to boot. The snapshot will be used and the virtual device will startup very fast.
Got some similar problem with real device. After have been working well on helloworld, keep on displaying HelloWorld after some changes in the code(building the UI). That is the stack i've no idea to resolve...

Sony Small App SDK

So i've just started to use the new Sony Xperia Tablet S Small App SDK. I'm no realy noob, developed many personal apps before but this has got me stumped.
I've loaded up the Sample project in Eclipse (all correctly configured), sorted some of the errors out, compiled to my device and when I launch the Small App from the launcher at the bottom of the device, it force closes - it's the same with every sample/app that I tried making with the SDK.
I'm attached below my MainApplication.java and AndroidManifest.xml in the hope that someone may be able to see where the issue lies. Don't understand as it's created as per the book. Any help really is appreciated please.
MainApplication.java:
package com.sony.thirdtest;
import android.content.pm.ActivityInfo;
import android.content.res.Configuration;
import android.view.View;
import android.widget.Toast;
import com.small.sonyapptest.R;
import com.sony.smallapp.SmallAppWindow;
import com.sony.smallapp.SmallApplication;
public class MainApplication extends SmallApplication {
private Configuration mConfig;
#Override
public void onCreate() {
super.onCreate();
mConfig = new Configuration(getResources().getConfiguration());
setContentView(R.layout.activity_main);
setTitle(R.string.app_name);
SmallAppWindow.Attributes attr = getWindow().getAttributes();
attr.minWidth = 200;
attr.minHeight = 200;
attr.width = 400;
attr.height = 300;
attr.flags |= SmallAppWindow.Attributes.FLAG_RESIZABLE;
getWindow().setAttributes(attr);
findViewById(R.id.button).setOnClickListener(new View.OnClickListener() {
#Override
public void onClick(View v) {
Toast.makeText(MainApplication.this, R.string.hello, Toast.LENGTH_SHORT).show();
}
});
}
#Override
public void onDestroy() {
super.onDestroy();
}
#Override
public void onStart() {
super.onStart();
}
#Override
public void onStop() {
super.onStop();
}
#Override
protected boolean onSmallAppConfigurationChanged(Configuration newConfig) {
int diff = newConfig.diff(mConfig);
mConfig = new Configuration(getResources().getConfiguration());
// Avoid application from restarting when orientation changed
if ((diff & ActivityInfo.CONFIG_ORIENTATION) != 0) {
return true;
}
return super.onSmallAppConfigurationChanged(newConfig);
}
}
AndroidManifest.xml
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.small.sonyapptest"
android:versionCode="1"
android:versionName="1.0">
<uses-sdk android:minSdkVersion="15" />
<uses-permission android:name="com.sony.smallapp.permission.SMALLAPP" />
<application android:icon="#drawable/ic_launcher" android:label="#string/app_name">
<uses-library android:name="com.sony.smallapp.framework" />
<service
android:name="com.small.sonyapptest.MainApplication"
android:exported="true" >
<intent-filter>
<action android:name="com.sony.smallapp.intent.action.MAIN" />
<category
android:name="com.sony.smallapp.intent.category.LAUNCHER" />
</intent-filter>
</service>
</application>
</manifest>
It seems that there is nothing wrong with your code. So I think the problem is with how you build the project. More exactly, how the small apps framework is included: it shouldn't be!
If you simply add the jar (from the Sony SDK) via "Java build path" -> "Add external Jar", then the classes of the api jar will be included with the application. The problem is those are only stub classes, so you can get one of two possible exceptions.
A simple and quick way to get around this (and still using the standard android sdk, and not switch to the Sony SDK) is the following:
Create a java project, and call it "SmallAppApi" for example
Inside the java project add the small app jar via "Add external jar"
In the last tab in the "Java build path" screen, called "Order and Export" make sure the small app jar is exported.
In the android project, in the "Java build path" screen, in "Projects" tab simply add the java project SmallAppApi (and remove the small app jar).
With this setup the small app jar will be used only when building. This worked fine for me.
In the service tag change android:name="com.small.sonyapptest.MainApplication" to android:name="MainApplication"
Thanks for the responses, all working now after chaning the service tag to just MainApplication.

Categories

Resources