Develop Reference

Error 200 (job canceled) when streaming data from Android to BigQuery

I've been trying to build some functionality into my app too allow user-generated data (EEG recordings) to be sent to a central BigQuery database.
I've never done any networking code in Java before, so I shied away from doing the POST or REST-based strategies recommended here. The BigQuery Java client library seemed to be exactly what I needed, though I was completely confused why it wouldn't officially support Android.
Still, I came across this example Android app (from Google no less) that promised to do exactly what I wanted with the BigQuery Client library. I incorporated it into my app as follows:
// .... in an AsyncTask
#Override
protected String doInBackground(String... params) {
String CSV_CONTENT = params[0];
try {
AssetManager am = MainApplication.getInstance().getAssets();
InputStream isCredentialsFile = am.open(CREDENTIALS_FILE);
BigQuery bigquery = BigQueryOptions.builder()
.authCredentials(AuthCredentials.createForJson(isCredentialsFile))
.projectId( PROJECT_ID )
.build().service();
TableId tableId = TableId.of(DATASET,TABLE);
Table table = bigquery.getTable(tableId);
int num = 0;
Log.d("Main", "Sending CSV: ");
WriteChannelConfiguration configuration = WriteChannelConfiguration.builder(tableId)
.formatOptions(FormatOptions.csv())
.build();
try (WriteChannel channel = bigquery.writer(configuration)) {
num = channel.write(ByteBuffer.wrap(CSV_CONTENT.getBytes(StandardCharsets.UTF_8)));
} catch (Exception e) {
Log.d("Main", e.toString());
}
Log.d("Main", "Loading " + Integer.toString(num) + " bytes into table " + tableId);
} catch (Exception e) {
Log.d("Main", "Exception: " + e.toString());
}
return "Done";
}
This runs without any errors and fires off an API call that is detected by Google Cloud Storage. However, it returns error 200 (job was cancelled) every time. I don't understand how this could be since I'm not doing anything in the code to cancel the request and I don't see how the async task I put the call in could be cancelled.
Was this just a bad example app I copied and a bad usage of the BigQuery Client? If so, what's the best way to send data to BigQuery from Android?

Not fully listing Google Drive folders and files in Android app

I am trying to access all files and folders from google drive to a arraya list. But I can get only one file from Drive. What to do get all files and folders from google drive. I am using the following code..
Thanks
Arun
public void onConnected(Bundle connectionHint) {
// Log.i(TAG, "API client connected.");
Toast.makeText(getActivity(), "Successfully logged in", Toast.LENGTH_LONG).show();
DriveFolder s = Drive.DriveApi.getRootFolder(mGoogleApiClient);
String s1 = (Drive.DriveApi.getRootFolder(mGoogleApiClient)).getDriveId().toString();
DriveId sFolderId2 = DriveId.decodeFromString(s1);
DriveId sFolderId = (Drive.DriveApi.getRootFolder(mGoogleApiClient)).getDriveId();
DriveFolder folder = Drive.DriveApi.getFolder(mGoogleApiClient, sFolderId);
folder.listChildren(mGoogleApiClient).setResultCallback(rootFolderCallback);
// findAll(folder);
}
public ResultCallback<DriveApi.MetadataBufferResult> rootFolderCallback = new
ResultCallback<DriveApi.MetadataBufferResult>() {
#Override
public void onResult(DriveApi.MetadataBufferResult result) {
if (!result.getStatus().isSuccess()) {
return;
}
resultarray = new ArrayList<String>();
int hh = result.getMetadataBuffer().getCount();
for (int i = 0; i < result.getMetadataBuffer().getCount(); i++) {
resultarray.add(result.getMetadataBuffer().get(i).getTitle());
}
Toast.makeText(getActivity(), "Successfully listed files.", Toast.LENGTH_LONG).show();
}
};

UPDATE (Aug 25, 2015, 10:39 MST)
Based on your comment below, you have 2 options:
1/ Stay with the GDAA, use one of the INTENTS:
- Pick a file with opener activity
- Pick a folder with opener activity
See, GDAA does not let your app see anything it did not create (SCOPE_FILE only), but it still allows user to browse everything. If the user selects a file, it will become visible to you app. I don't know your app's intentions, so I can't say if this approach is usable.
2/ Switch to the REST with the DRIVE scope and your app will see everything (user has to approve up front). The basic CRUD implementation can be found here but make sure you change the scope in the init() method to 'DriveScopes.DRIVE'.
In case your app needs to iterate down the folder tree, collecting files in the process, both 'testTree()' and 'deleteTree()' methods in the MainActivity() do exactly that.
You may also stay with the GDAA and add REST functionality to it by adding
com.google.api.services.drive.Drive mGOOSvc = new Drive.Builder(AndroidHttp.newCompatibleTransport(), new GsonFactory(),
GoogleAccountCredential.usingOAuth2(appContext, Collections.singletonList(DriveScopes.DRIVE))
.setSelectedAccountName(email)
but you will sooner or later run into problems caused by GDAA caching / latency.
ORIGINAL ANSWER
Try this approach:
private static GoogleApiClient mGAC;
/****************************************************************************
* find file/folder in GOODrive
* #param prnId parent ID (optional), null searches full drive, "root" searches Drive root
* #param titl file/folder name (optional)
* #param mime file/folder mime type (optional)
* #return arraylist of found objects
*/
static void search(String prnId, String titl, String mime) {
if (mGAC != null && mGAC.isConnected()) {
// add query conditions, build query
ArrayList<Filter> fltrs = new ArrayList<>();
if (prnId != null){
fltrs.add(Filters.in(SearchableField.PARENTS,
prnId.equalsIgnoreCase("root") ?
Drive.DriveApi.getRootFolder(mGAC).getDriveId() : DriveId.decodeFromString(prnId)));
}
if (titl != null) fltrs.add(Filters.eq(SearchableField.TITLE, titl));
if (mime != null) fltrs.add(Filters.eq(SearchableField.MIME_TYPE, mime));
Query qry = new Query.Builder().addFilter(Filters.and(fltrs)).build();
// fire the query
Drive.DriveApi.query(mGAC, qry).setResultCallback(new ResultCallback<MetadataBufferResult>() {
#Override
public void onResult(MetadataBufferResult rslt) {
if (rslt != null && rslt.getStatus().isSuccess()) {
MetadataBuffer mdb = null;
try {
mdb = rslt.getMetadataBuffer();
if (mdb != null ) for (Metadata md : mdb) {
if (md == null || !md.isDataValid()) continue;
String title = md.getTitle();
DriveId driveId = md.getDriveId();
//.......
}
} finally { if (mdb != null) mdb.close(); }
}
}
});
}
}
Call it first with NULLs
search(null,null,null)
To list all the files in your Google Drive. You will see all the files your Android App created. But only those - FILE scope does not see anything else.
If you need to scan the directory tree, you may look closer at this GDAA demo, in MainActivity, there is are 'testTree()' / 'deleteTree() methods that recursively scan the directory tree structure.
Also, you may look at the answer here, it deals with a similar issue (especially the comments exchange under the answer).
Good Luck

Please note that you can use GDAA to retrieve the files and folder that you have either uploaded from the Android Device or downloaded via the drive app. This is to have more security (as quoted by Google).
In he code you need to ensure that you are trying all possible combinations for the files that may be present in your Google Drive account. For example, check if you are tracking the parent of a file or a folder. If this condition is not met your app wont be able to retrieve those specific files.
/** Get the list of parents Id in ascending order. */
private List<String> collectParents(String folderId, Map<String, String> folderIdToParentId){
String parentId = folderIdToParentId.get(folderId);
if (logger.isTraceEnabled()){
logger.trace("Direct parent of {} is {}", folderId, parentId);
}
List<String> ancestors = new ArrayList<String>();
ancestors.add(parentId);
if (folderIdToParentId.containsKey(parentId)){
ancestors.addAll(collectParents(parentId, folderIdToParentId));
return ancestors;
}
return ancestors;
}
See the full code here.

Create GoogleCredential by Access Token

I want to design an Android file viewer for Google Drive.
At first, I implemented the app by using of the Google Android API, as follows,
private void retrieveNextPage(){
if(mHasMore == false)
return;
Query query = new Query.Builder().setPageToken(mNextPageToken).build();
com.google.android.gms.drive.Drive.DriveApi.query(getGoogleApiClient(), query).setResultCallback(metadataBufferResultResultCallback);
}
However, the Android Drive API only allows the app to view and fetch the files that created by itself. I cannot access other files on the drive through the app.
Therefore, I turned to another option, directly manipulate the Java Drive API.
According to the example on developer guide for Java,
https://developers.google.com/drive/web/quickstart/quickstart-java
The users have to manually copy and paste the "Authorization Code" between the browser and app, which is not a practical way to acquire the Access Token in Android.
To come out a new way, I used the GoogleAuthUtil in Android API to acquire the Access Token, coincided with the GoogleCredential and Drive in Java API to fetch the file list, as follows,
private static List<File> retrieveFiles(Drive service) throws IOException{
List<File> result = new ArrayList<File>();
Files.List request = service.files().list();
do {
try{
FileList fileList = request.execute();
result.addAll(fileList.getItems());
request.setPageToken(fileList.getNextPageToken());
}catch (IOException e){
Log.d(dbgT + "JavaRetrieveFiles", "Retrieved Failed");
request.setPageToken(null);
}
}while (request.getPageToken() != null && request.getPageToken().length() > 0);
return result;
}
private class RetrieveTokenTask extends AsyncTask<String, Void, String>{
#Override
protected String doInBackground(String... params){
String accountName = params[0];
String scopes = "oauth2:" + "https://www.googleapis.com/auth/drive";
String token = null;
try{
token = GoogleAuthUtil.getToken(getApplicationContext(), accountName, scopes);
}
catch (IOException e){
Log.e(excpTAG, "IO Exception: " + e.getMessage());
}
catch (UserRecoverableAuthException e){
startActivityForResult(e.getIntent(), REQ_SIGN_IN_REQUIRED);
}
catch (GoogleAuthException e)
{
Log.e(excpTAG, "GoogleAuthException: " + e.getMessage());
}
return token;
}
#Override
protected void onPostExecute(String s){
super.onPostExecute(s);
//Get Access Token
Log.d( dbgT + "Token", s);
EditText tokenText = (EditText) findViewById(R.id.tokenText);
tokenText.setText(s);
EditText fileNameText = (EditText) findViewById(R.id.editTextMeta);
GoogleCredential credential = new GoogleCredential().setAccessToken(s);
HttpTransport httpTransport = new NetHttpTransport();
JsonFactory jsonFactory = new JacksonFactory();
Drive service = new Drive.Builder(httpTransport, jsonFactory, null).setHttpRequestInitializer(credential).build();
List<File> fileList;
try{
fileList = retrieveFiles(service);
for(int i=0; i< fileList.size(); i++)
fileNameText.append(fileList.get(i).getTitle());
}catch(IOException e){
Log.d(dbgT + "RetrieveFileList", "IO Exception" );
}
}
}
Unfortunately, the app always crashes by the causing of NetworkOnMainThreadException when request.execute() in retrieveFiles is invoked.
I checked my access token s, it is usually in form of ya29.xxx...etc., and it can also be passed to my other .NET program for retrieving files from Google Drive. Therefore I can certain the access token is correct.
So my question is, how to create a correct GoogleCredential by using of access token, instead of applying authorization code in setFromTokenResponse ?
Thanks in advance.

Many thanks for Andy's tips, this problem is simply caused by the network operations occurs on the main thread, which is a very basic newbie error.
The Drive in Google Drive SDK for Java, using network libraries without any background/thread worker, and now it is functional after I put the retrieveFiles() into background.
Applying the GoogleAuthUtil in Google Play Android SDK to acquire the access token, and followed by GoogleCredential+Drive in Java SDK that use the token to do the file operation in Google Drive.
This is a right way to avoid the scope restriction in Android SDK for Google Drive, allowing the developers to acquire the full permissive of accessing Google Drive.

Google Drive API - Android - How to obtain a drive file id?

I'm trying to develop an android app that can read a xml file stored in my google drive folder, the idea at first is trying to open the file and handle the content.
I've read the Google Drive API docs for android and i reached a point that I'm lost, it's working with file contents.
According to this guide the way to open a file from drive is this:
DriveFile file = ...
file.open(mGoogleApiClient, DriveFile.MODE_READ_ONLY, null).setResultCallback(contentsOpenedCallback);`
Searching I realized that the complete code (that they not include there is):
DriveFile file = Drive.DriveApi.getFile(mGoogleApiClient,DriveId.bg(id));
file.open(mGoogleApiClient, DriveFile.MODE_READ_ONLY, null).setResultCallback(contentsOpenedCallback);`
Well the problem there is that I don't know the file "id". I've tried the id from the web link of google drive, something like this (https://drive.google.com/open?id=1EafJ-T6H4xI9VaUuUO5FMVb9Y30xyr7OHuISQ53avso&authuser=0) but didn´t work.

You could use the DriveAPI Query method, to retrieve any information about an specific file. you will need to define a query object as the following:
Query query = new Query.Builder()
.addFilter(Filters.eq(SearchableField.TITLE, "HelloWorld.java"))
.build();
And set a callback function to iterate on the results:
Drive.DriveApi.query(googleApiClient, query)
.setResultCallback(new OnChildrenRetrievedCallback() {
#Override
public void onChildrenRetrieved(MetadataBufferResult result) {
// Iterate over the matching Metadata instances in mdResultSet
}
});
You can find more information on the topic here: https://developers.google.com/drive/android/queries

The solution i found for this problem was creating the file from the app. Using the class ("CreateFileActivity.java") from google drive api demo app.
With this class i save the returning Driveid from the new file in a global DriveId variable.
final private ResultCallback<DriveFolder.DriveFileResult> fileCallback = new
ResultCallback<DriveFolder.DriveFileResult>() {
#Override
public void onResult(DriveFolder.DriveFileResult result) {
if (!result.getStatus().isSuccess()) {
Log.e("","Error while trying to create the file");
return;
}
Id=result.getDriveFile().getDriveId();
Log.e("","Created a file with content: " + Id);
}
};
Then with this id in another method i call the file and read it (If i want i can edit this file information from Google Drive Web App):
public void leer(){
DriveFile file = Drive.DriveApi.getFile(getGoogleApiClient(),Id);
file.open(mGoogleApiClient, DriveFile.MODE_READ_ONLY, null)
.setResultCallback(contentsOpenedCallback);
}
ResultCallback<DriveApi.DriveContentsResult> contentsOpenedCallback =
new ResultCallback<DriveApi.DriveContentsResult>() {
#Override
public void onResult(DriveApi.DriveContentsResult result) {
if (!result.getStatus().isSuccess()) {
Log.e("Error:","No se puede abrir el archivo o no se encuentra");
return;
}
// DriveContents object contains pointers
// to the actual byte stream
DriveContents contents = result.getDriveContents();
BufferedReader reader = new BufferedReader(new InputStreamReader(contents.getInputStream()));
StringBuilder builder = new StringBuilder();
String line;
try {
while ((line = reader.readLine()) != null) {
builder.append(line);
}
} catch (IOException e) {
e.printStackTrace();
}
String contentsAsString = builder.toString();
Log.e("RESULT:",contentsAsString);
}
};

I've been playing with this stuff a few months back, and still have some code on GitHub. It may be VERY outdated (libver 15 or so), but it may serve as a reference point, and it is simple. Look here. Pull it down, plug in, step through. Fix what's not working anymore :-). I've abandoned it some time ago.
Be aware of the fact that there are 2 different IDs for Google Drive Android API objects, see SO 22841237.
In general, you usually start with knowing the file/folder name, query GDAA to get a list of objects. Each of them will yield DriveID and ResourceID. DriveID is used in your app to manipulate the objects (does not mean anything outside your Android App and/or device). ResourceID is the string that appears in different forms in URLs and can be used outside your app (web browser for instance...). Look at this wrapper to get some feeling how it works. But again, it's been a few versions back, so there are no guaranties.

The Google Drive API is deprecated, now its Google Drive V3 and for Query we use
String pageToken = null;
do {
FileList result = driveService.files().list()
.setQ("mimeType='image/jpeg'")
.setSpaces("drive")
.setFields("nextPageToken, files(id, name)")
.setPageToken(pageToken)
.execute();
for (File file : result.getFiles()) {
System.out.printf("Found file: %s (%s)\n",
file.getName(), file.getId());
}
pageToken = result.getNextPageToken();
}
while (pageToken != null);
You can Learn more here Officals Docs

Create or Open native Google documents using GoogleDriveApi

I have been closely following the documentation for the Google Drive Android API and, all works great. I can create new text documents and read them back in using the mime type of text/plain.
What I cannot do is create a native Google "Document" or "Spreadsheet." Actually, I can create them by using the mime type to application/vnd.google-apps.document or application/vnd.google-apps.spreadsheet as per Supported MIME Types documentation.
If, however, I try to write content to these documents, the documents never get uploaded.
If I try to read documents that have content (content I created via a web browser) my openContents call fails.
Again, I can create text/plain documents and write to them, but they are not native Google Documents. I have scowered the documentation and sample files, but nothing describes what I'm looking for.
This seems so basic. Does the new GoogleApiClient not support doing this? What am I missing or doing wrong?
Here is the core code for creating. I have a similar issue when trying to read a application/vnd.google-apps.document but I'm sure the two issues are related. I'll spare the verbosity of "read" code.
private void exportToGDriveFile() {
Drive.DriveApi.newContents(getGoogleApiClient()).setResultCallback(createNewFileCallback);
}
final private ResultCallback<ContentsResult> createNewFileCallback = new ResultCallback<ContentsResult>() {
#Override
public void onResult(ContentsResult result) {
if (!result.getStatus().isSuccess()) {
writeLog("Error while trying to create new file contents");
return;
}
String fileName = getIncrementedFileName();
MetadataChangeSet changeSet = new MetadataChangeSet.Builder()
.setTitle(fileName)
.setMimeType("text/plain") // <-- This works! I can write and read back :)
//.setMimeType("application/vnd.google-apps.document") <-- can create if no contents are included.
//.setMimeType("application/vnd.google-apps.spreadsheet")
.setStarred(true)
.build();
writeLog("creating file: " + fileName);
// create a file on root folder
Drive.DriveApi.getRootFolder(getGoogleApiClient())
.createFile(getGoogleApiClient(), changeSet, result.getContents())
.setResultCallback(afterCreateFileCallback);
}
};
private ResultCallback<DriveFileResult> afterCreateFileCallback = new ResultCallback<DriveFileResult>() {
#Override
public void onResult(DriveFileResult result) {
if (!result.getStatus().isSuccess()) {
writeLog("Error while trying to create the file");
return;
}
DriveFile driveFile = result.getDriveFile();
writeLog("Created file " + driveFile.getDriveId());
new WriteFileAsyncTask().execute(driveFile);
}
};
private class WriteFileAsyncTask extends AsyncTask<DriveFile, Void, Boolean> {
#Override
protected Boolean doInBackground(DriveFile... args) {
DriveFile file = args[0];
try {
ContentsResult contentsResult = file.openContents(getGoogleApiClient(), DriveFile.MODE_WRITE_ONLY, null).await();
if (!contentsResult.getStatus().isSuccess()) {
return false;
}
/************************
If I try to write content here, `application/vnd.google-apps.document` files will not upload.
*************************/
String contents = "Hello World";
OutputStream outputStream = contentsResult.getContents().getOutputStream();
outputStream.write(contents.getBytes());
com.google.android.gms.common.api.Status status = file.commitAndCloseContents(
getGoogleApiClient(), contentsResult.getContents()).await();
return status.getStatus().isSuccess();
} catch (IOException e) {
// toast("IOException while appending to the output stream");
}
return false;
}
#Override
protected void onPostExecute(Boolean result) {
if (!result) {
// toast("Error while editing contents");
return;
}
// toast("Successfully uploaded Quizifications!");
}
}

It's not currently possible to read or edit the contents of Google Documents, Spreadsheets or Presentation files. They files are of a special type that don't have standard binary content, so you can't read and write from them in the same way you can from other files.
You can, however, interact with the metadata of existing files.
Sorry for the confusion, we should update the behavior so that its clear that its not possible.

Updating Google Docs with HTML is simple. Just make an api request with html-formatted text in the body (html tag is required) and content-type to be google docs, then your created/updated file will be available to the user as a Google Doc with all the formatting options.
request({
uri: 'https://www.googleapis.com/upload/drive/v2/files/'+fileId,
method: 'PUT',
qs: {
uploadType: 'media'
},
form: '<html> Hello <b>World!</b> </html>',
headers: {
'Content-Type': 'application/vnd.google-apps.document',
'Authorization': 'Bearer ' + access_token
}}, function (error, response, body){
})