I am trying to use python to help do some automation around an incremental build function in the Android build system. Generally, from a given directory, I would execute the following command to build whatever is in that directory and subdirectories:
mm -j8
This is analogous to a "make" command, only it is incremental build and is defined as a function in a bash file called envsetup.sh. What is does it not important, just know that it's a function defined in a bash script somewhere in the file system. To execute this, I can also do:
bash -c ". /path/to/envsetup.sh; mm -j8"
This method of calling it will be important in calling the function from python. I have followed the solution here which shows how to call a function within a bash script from python. I have used this method in a simple script that, in theory, should just spit out the STDOUT and STDERR from executing the command:
import subprocess
command = ['bash', '-c', '. /path/to/envsetup.sh; mm -j8']
(stdout, stderr) = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
print 'stdout: ' + stdout
print 'stderr: ' + stderr
The call to Popen, however, never returns. What am I doing wrong that would allow bash to execute the command properly, but Python hangs when executing the command?
tl; dr:
Your issue is the use of shell=True. Set it to shell=False and it'll work.
With this option set, python will just run the first element of the command array, i.e. bash as a shell script. So currently, python is launching a shell of its own, in order to run your command (bash). It'll run bash with no arguments, and bash will then wait for input, blocking your python script.
The shell=True setting is for use cases where you are passing a shell script in as a single string. When you're explicitly specifying a shell and its parameters as the process to invoke, as you are doing above, you should set shell=False.
>>> import subprocess
>>> subprocess.Popen(['bash', 'whatever'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
Here's what the proces tree looks like when I run the above:
\_ python
\_ /bin/sh -c bash whatever
\_ bash
The whatever is actually passed in, but it's a parameter to the sh, not a parameter to the inner bash, so the command being run is effectively ['/bin/sh', '-c', 'bash', 'whatever'], which is quite different from ['/bin/sh', '-c', 'bash whatever']
Related
I am trying to connect two devices to my pc and run some commands on them using python and adb.
when I run the command from the command prompt, it goes through fine, but when i put those in the python script, they give me errors.
this is causing the errors all the time:
from subprocess import check_output, CalledProcessError
try:
adb_ouput = check_output(["adb","devices","-l","|", "grep", "\"model\""])
print adb_ouput
except CalledProcessError as e:
print e
The error message I get is this:
Usage: adb devices [-l]
Command '['adb', 'devices', '-l', '|', 'grep', '"model"']' returned non-zero exit status 1
When I try the same code without the grep command, it works
adb_ouput = check_output(["adb","devices","-l"])
It gives me the right output.
When I try the same in windows command prompt, It works fine (I am replacing grep with FINDSTR because i'm using it in windows, and I have tried doing the same in the python script as well, with 'shell = True' and also without.)
eg:
adb devices -l | FINDSTR "model"
This gives me an ouptut without any problems.
The output I get is
123ab6ef device product:xxxxxxxxx model:xxxxxxxxx device:xxxxxxxxx
bd00051a4 device product:yyyyyyyyyy model:yyyyyyyyyy device:yyyyyyyyy
I am trying to understand where I am going wrong here, but can't figure it out.
So far I have checked the docs: https://docs.python.org/3/library/subprocess.html
https://docs.python.org/3/library/subprocess.html#subprocess.CalledProcessError
These just give me the error codes.
I have also looked at these answers:
Python, adb and shell execution query
I took some error checking from here and added to my code.
Python subprocess.check_output(args) fails, while args executed via Windows command line work OK
python check_output fails with exit status 1 but Popen works for same command
I think i am close but just can't put my finger on it.
Any help would be appreciated.
First
adb_ouput = check_output(["adb","devices","-l","|", "grep", "\"model\""])
certainly requires shell=True, but even with that it's not equivalent to
adb devices -l | FINDSTR "model"
When using check_output, you're passing "model" as grep argument literally but you should pass just model. "model" is not in your output (with quotes) so grep fails to find it, and returns exitcode 1 which is not really an error for grep but makes check_output trigger an exception because it expects 0.
So I would to this as a quickfix:
adb_ouput = check_output(["adb","devices","-l","|", "grep", "model"],shell=True)
And as a longfix I'd perform the grep command directly with python.
adb_output = check_output(["adb","devices","-l"])
for l in adb_output.splitlines():
if "model" in l:
print(l)
I'm almost certain i am overlooking something.
I have an android gradle project with a build.gradle file. Inside here, I specify the task:
task doSomething(type: Exec) {
println("okay clearly you have got to be getting in here")
commandLine 'sh /Users/dzt/Desktop/create_a_file_on_desktop.sh'
}
and that doesn't run at all. the shell file just literally does:
#!/bin/sh
echo "hi" > /Users/dzt/Desktop/i_am_a_byproduct.txt
and i ran chmod u+x on it so it is executable (i double checked on regular bash shell).
I also tried to use the groovy command:
"cd ../ && sh /Users/dzt/Desktop/create_a_file_on_desktop.sh".execute()
which does not work either. I'm a little stumped. i do NOT see the output file. however, i do see my print statement in the gradle console.
What is going on here?
** EDIT **
okay, i drilled it down more ->
cd ../ does not work at all. why is this? i need to use a relative path, at least relative to this directory
The call must be
commandLine 'sh', '/Users/dzt/Desktop/create_a_file_on_desktop.sh'
or else this is considered one command. But you want to start the sh with the script as param. On the other hand, since you have set the execute-bit, you can as well just call the shell script directly.
See http://gradle.org/docs/current/dsl/org.gradle.api.tasks.Exec.html
Running cd like you want with cd ../ && sh script does also not work like this, since && is a shell script command. If you want to run like this, you have to run the shell and make it run as a command. E.g.
commandLine 'sh', '-c', 'cd ~/scripts && sh myscript.sh'
Gradle does not allow cd command for some reason. some commands just do NOT work using groovy.
instead, i used cd inside my shell script. that seems to work just fine.
First, you have to put in the root level gradle.build file. The you need to write it like this, to actually be able to execute the task.
task doSomething << {
group 'yourGroupName'
description 'Runs your bash script'
exec {
workingDir "$projectDir/../pathto/"
commandLine 'bash', '-c', './bashscript.sh'
}
}
Then you can execute with: ./gradlew -q doSomething. In this case I used bash, but you can use any supported scripting shell, like sh, perl, python etc.
I've had 0 exposure to BASH scripting and this is something I would love to learn. I can't figure out how to run a conditional statement based on the output of ant debug on an Android build.
I would like to essentially say something like
if(`ant debug` == SUCCESS) {
// EXECUTE THESE COMMANDS
} else {
// EXECUTE THESE COMMANDS
}
How can I determine if the ant debug has passed or failed in shell script?
SOLUTION
Okay here is what I have:
ant clean
if ant debug; then
echo "success"
else
echo "failure"
fi
I'll give a quick summary for you.
In Bash, conditionals are based around the exit codes of programs. An exit code of 0 is accepted as true, while everything else is accepted as false.
For example, the true program always exits with an exit code of 0, which means that something like this is possible:
if true;
echo "It is true"
fi
Most commands honor this system, but not every program does. The first thing to check is what exit code ant returns on success and failure. You can check the exit code of the previous command with $?.
Here is an example:
$ true
$ echo $?
0
$ false
$ echo $?
1
If ant does honor the exit code system properly, then something like the following should be possible:
if ant debug; then
echo success
else
echo failure
fi
I know nothing about Ant debugging, but there are two approaches to doing what you want to do in Bash. The first is to test output like you've shown:
if test $(ant debug) == 'SUCCESS'; then
# do stuff
else
# do other stuff
fi
You can make your shell script portable to other variants on the Bourne shell by using backticks instead of $(.....) like you wrote in your question, but that starts to become a hassle if your commands later involve nested quotes.
The second way, which is a little more robust, is to test the exit value of the commands instead of their output. This depends on Ant (or whatever) having exit codes that are documented and stable, but it means that if the output of the commands changes, your scripts will continue to work. For example, the POSIX standard says that if a programs succeeds in doing whatever it's supposed to do, it should exit() with a value of zero:
ant debug > /dev/null
ant_exit_code=$?
# other commands can safely go here now that $? is captured
if test $ant_exit_code -eq 0; then
# do stuff
else
# do other stuff
fi
And yes, Bourne shell really does end an if block with "fi". :-)
A quick man ant shows that ant debug invokes Ant with the debug task. Ant's tasks are kinda of user-defined XML scripts, and by default Ant searches a build.xml file in the current directory. You can generate the file with the android tools, however a template is kept in android-sdk/tools/ant and you can view it online (line 1005 defines the debug target).
So ant debug isn't really a command, and should not be put in a script toolchain. However, if you find your output to be stable and feel brave, you can always compare string. This is the definitve guide.
if [ `ant debug` = $SOMETHING ]; then
echo Success
else
echo Failure
fi
I was trying to run a a sample python program using monkey runner but unfortunately throwing an error of this type :
Can't open specified script file
Usage: monkeyrunner [options] SCRIPT_FILE
-s MonkeyServer IP Address.
-p MonkeyServer TCP Port.
-v MonkeyServer Logging level (ALL, FINEST, FINER, FINE, CONFIG, INFO, WARNING, SEVERE, OFF)
Exception in thread "main" java.lang.NullPointerException
so any one can guide me how to resolve this and how to use monkey runner to execute this type of things
I've found that making the path to the script absolute helped monkeyrunner.
I am invoking the runner from a Python script where a helper class has a startMonkey method:
def startMonkey(self, monkeyScript):
command = [ self.env + '\\tools\\monkeyrunner.bat' ]
command.append( os.path.abspath( monkeyScript ) )
return startProcess( command )
This seems to be working for me.
Script file should be a full path file name try below monkeyrunner c:\test_script\first.py
So go to the folder \ sdk \ tools
and press shift and right click to open command prompt and type monkeyrunner c:\test_script\python_file_name.py
Obviously this shell script is calling itself as a Python script:
#!/bin/sh
## repo default configuration
##
REPO_URL='git://android.git.kernel.org/tools/repo.git'
REPO_REV='stable'
magic='--calling-python-from-/bin/sh--'
"""exec" python -E "$0" "$#" """#$magic"
if __name__ == '__main__':
import sys
if sys.argv[-1] == '#%s' % magic:
del sys.argv[-1]
del magic
:
:
(Whole script: https://android.googlesource.com/tools/repo/+/v1.0/repo)
Can anyone explain
the purpose of calling it this way?
Why not having #!/usr/bin/env python in the first line so it gets interpreted
as Python script from the beginning?
the purpose of adding that magic last command line argument, that is removed afterwards in the beginning of the Python code?
Your first question: this is done to fix unix systems (or emulations thereof) that do not handle the #! correctly or at all. The high art is to make a script that is correct in shell as well as in the other language. For perl, one often sees something like:
exec "/usr/bin/perl"
if 0;
The exec is interpreted and executed by the shell, but the perl interpreter sees a conditional statement (.... if ...) and does nothing because the condition is false.