Develop Reference

Trouble with sed: "bad pattern" - android

I'm working on a script to run on Android (hence the weird shebang path). It involves using sed to comment out certain blocks of code from specified files. Currently, I'm trying to pass an entire sed command to a function, but I'm having a lot of trouble doing so.
This is the script:
#!/system/bin/sh
REM_RCTD=$1
REM_CCMD=$2
REM_TRITON=$3
DEVICE_CODE=$4
COLOR_GRN_PRE="<font color='#00ff00'>"
COLOR_YEL_PRE="<font color='#ffff00'>"
COLOR_POS="</font>"
YELLOW=0
GREEN=1
echoAndExec() {
CMD="$2"
if [ "$1" = ${YELLOW} ]; then
echo "$COLOR_YEL_PRE $CMD $COLOR_POS"
elif [ "$1" = ${GREEN} ]; then
echo "$COLOR_GRN_PRE $CMD $COLOR_POS"
fi
${CMD} || exit 1
}
if [ "$REM_RCTD" = "true" ]
then
CMD="sed -ir -e \_^# LG RCT(Rooting Check Tool)$_,/^$/{/^(#\|$)/!s/^/#/} init.lge.rc"
echoAndExec ${YELLOW} "${CMD}"
fi
if [ "$REM_CCMD" = "true" ]
then
CMD="sed -ir -e \_^service ccmd /system/bin/ccmd$_,/^$/{/^(#\|$)/!s/^/#/} init.lge.rc"
echoAndExec ${YELLOW} "${CMD}"
fi
if [ "$REM_TRITON" = "true" ]
then
CMD="sed -ir -e /# triton service/,\_chmod 644 /sys/devices/system/cpu/triton/enable_s/^/# / init.${DEVICE_CODE}.power.rc"
echoAndExec ${YELLOW} "${CMD}"
fi
Every command sent to the echoAndExec() function works fine, except for sed, which returns
sed: bad pattern '\_^##1 (\)
For the third sed command, it has the same issue with the /.
I've tried a bunch of combinations of quotes, with and without the usage of variables, with different syntax for the pattern beginnings (/#, _), but I'm really lost. As is probably obvious from my script, I'm pretty much a beginner in shell scripting.
Is there maybe a POSIX alternative to sed I could use that might work better? Or is there some combination of quotes and escapes that'll make this work? I just can't figure it out.
If anyone needs any more details, I'll be happy to provide them; I just don't know what would be relevant.

TL;DR: You can use eval in the function and include correct internal quotation marks in the command you pass, which for complex commands is cumbersome and may make code that uses your function very hard to read, but which has some advantages. Or you can pass the commands as multiple arguments to your shell function, after the initial color argument, shift off the color argument after storing or otherwise using it, and run the command as "$#". See below for full details. (There are also likely methods not covered here at all; perhaps others will write answers about them.)
The Problem
The ${CMD} in ${CMD} || exit 1 in not quoted, so word splitting and globbing are performed. That only does what you actually want in the very simplest of cases. Each of your sed commands contains text that you intend to be passed to sed as a single argument, but which contains spaces. The spaces cause it to split into multiple words, each of which is passed to sed as a separate command-line argument.
Consider the simplified situation where the command you want to run is:
printf '%s\n' 'foo bar' 'baz quux'
When you run printf with %s\n as its first argument, it prints each of its subsequent arguments on a line by itself. This is a convenient way to check the effects of word splitting and globbing. The output of that particular command is:
foo bar
baz quux
If you assign the whole command to CMD without internal quotes, the simplicity of this particular command makes the problem immediately obvious: there is no way for the shell to know where you do and do not intend for it to perform word splitting.
$ CMD="printf %s\n foo bar baz quux"
$ $CMD
foo
bar
baz
quux
But this is exactly the situation you have with each of your sed commands. Some of the spaces in your sed commands are intended to separate arguments, while others are not, and the shell has no way to know what you want.
Embedding internal quotation marks into the value of CMD will not solve the problem. Not by itself, anyway. Since they are themselves quoted, their special meaning is suppressed. Quote removal does not occur in this situation, so they will simply remain at the edges of the words where you put them. Furthermore, the whitespace you intended for them to quote still causes word splitting:
$ CMD="printf %s\n 'foo bar' 'baz quux'"
$ $CMD
'foo
bar'
'baz
quux'
You can, of course, expand $CMD inside double quotes ("$CMD"). But that prevents all word splitting, and attempts to run a program whose name is your entire command, spaces and all. That is not what you want:
$ "$CMD"
printf %s\n 'foo bar' 'baz quux': command not found
There are multiple ways to solve this problem. I'll show two.
Way 1: You can make the function run the command with eval
One solution is to expand $CMD in double quotes to prevent word splitting, but instead of running that, pass it as an argument to the eval shell builtin. That causes the shell to parse the contents of CMD the same way it does when those contents actually appear as a line in a script. Note that, since the actual text stored in CMD is not quoted, you must include internal quotes for everything that requires quoting. In simple cases, you can get away with using " " for either the inner quotes or the outer quotes:
$ CMD='printf "%s\n" "foo bar" "baz quux"'
$ eval "$CMD"
foo bar
baz quux
$ CMD="printf '%s\n' 'foo bar' 'baz quux'"
$ eval "$CMD"
foo bar
baz quux
However, for commands that contain characters like $ that may be treated specially if expanded inside double quotes -- like what you are doing with sed -- it is often necessary to use single quotes for both. To achieve this, you can end quoting for just long enough to write a single quote that is itself quoted with \, then resume quoting again. That is, you can write a single quote "inside" single quotes as '\''.
$ CMD='printf '\''%s\n'\'' '\''foo bar'\'' '\''baz quux'\'''
$ eval "$CMD"
foo bar
baz quux
This approach has the major disadvantage that it can be hard to quote your commands correctly to pass them to your shell function, and extremely hard for you (or someone else) to verify they are correct by inspection. However, it has the advantage that the function has a string that can be run, exactly as written, as your command. Normally this is unimportant, but in your case it may be important, because you are showing the user what the command was. If you want the user to see a command that can be run, verbatim -- that is, you want to show proper quoting to the user, not just run the command correctly -- then this eval-based approach is probably the simplest way to achieve that.
Here's a modified version of your shell function. It takes the color attribute value that goes in your start tag, rather than a number, as the first argument. I actually suggest you do that, as there's no reason $YELLOW and $GREEN cannot just be #ffff00 and #00ff00, respectively. But however you choose to write it, this should demonstrate how you can use eval in it.
echoAndExec() {
printf '<font color='\''%s'\''> %s </font>\n' "$1" "$2"
eval "$2" || exit 1
}
As you can see, I've also replaced echo with printf, since some echo implementations expand escape sequences, which you probably don't want here. (Some can also treat the first argument you pass as though it were an option, but that's not an issue here, since your first argument starts with <, never -.) Finally, I have made the variables lower-case. I suggest using lower-case names for your shell variables, unless you plan to export them as environment variables. It is common both for environment variables and for variables treated specially by shells (e.g., PS1) to be named in upper case, and using lower case helps avoid conflicts. You can use upper case if you want to, though.
Here's how you might call that function:
$ green='#00ff00'
$ cmd='printf '\''%s\n'\'' '\''foo bar'\'' '\''baz quux'\'''
$ echoAndExec "$green" "$cmd"
<font color='#00ff00'> printf '%s\n' 'foo bar' 'baz quux' </font>
foo bar
baz quux
Or just:
$ green='#00ff00'
$ echoAndExec "$green" 'printf '\''%s\n'\'' '\''foo bar'\'' '\''baz quux'\'''
<font color='#00ff00'> printf '%s\n' 'foo bar' 'baz quux' </font>
foo bar
baz quux
Of course, whether or not you assign the command to a variable first, you do not have to redefine green each time.
Way 2: You can pass the command as multiple arguments to the function
Normally, when you perform parameter expansion in double quotes, word splitting is suppressed entirely. However, the # parameter is special. * and # both contain the text of all your positional parameters, one after the other, but they behave differently when expanded in double quotes. With "$*", you get no word splitting--the positional parameters are joined, with single spaces (or whatever is the first character of $IFS) between them.
In contrast, with "$#", word splitting is performed between the positional parameters, but not within them. This is to say that each positional parameter becomes its own word, but a positional parameter that contains whitespace still won't be split any further (as it would be with $* or $# outside " ").
This provides exactly the functionality you need to pass a command to your function, in a way that allows you to write the command in a readable manner, and have the function run it correctly. Here's how you might write your function, if you want to take this approach:
echoAndExec() {
printf '<font color='\''%s'\''> ' "$1"
shift
printf '%s </font>\n' "$*"
"$#" || exit 1
}
Running "$#" initially would not do what you want, because it would have included the first positional parameter at the beginning. To remove the first positional parameter while shifting each of the others down by one (or you may prefer to think of it as shifting them left by one), I used the shift builtin.
That code is somewhat less readable than it could be, because I have avoided introducing any variables in the shell function. The reason I have done this is that the declare and local builtins are not actually required by POSIX, and I am not sure if your shell--and other shells on which you might need to run this script--support them. Without them, assigning to a variable in a shell function causes them to be set for the caller as well. For the particular script you've shown, that doesn't seem like it would be a problem, but I don't know if (or how) you might end up extending the script or what variable names you might use in it later.
Some of the changes shown above, compared to your version of the function, are specific neither to this approach of expanding "$#" nor to the preceding approach of using eval. I have explained those changes in the preceding section. You do not have to write your function exactly this way, though you can; the purpose of the above code is to serve as an example.
It's important to remember that you must call this function in a different way from the way you would call the function you wrote originally (and also differently from the eval-based version shown above). Don't store your commands in a variable first. Just pass each word of the command to the shell function:
$ green='#00ff00'
$ echoAndExec "$green" printf '%s\n' 'foo bar' 'baz quux'
<font color='#00ff00'> printf %s\n foo bar baz quux </font>
foo bar
baz quux
You will notice that this is much, much easier to call, because you don't have to use any quoting beyond what you would use just to run the command directly. In fact, you must not use such additional quoting. It is enormously easier to use this function and to understand what you have written and verify that it is correct.
However, this does have the disadvantage that it is no longer trivial to print the command that was passed in with its original quoting. The shell removes those quotes when the function is called. The function does not have access to them.
You might not care about this, but if you do, then you can make your shell function insert quotes around each argument. They won't necessarily be the same quotes that were used originally, and they might not even be correct if the arguments themselves contained single quotes. But they should typically indicate, in a reasonably unambiguous way, what arguments were passed:
echoAndExec() {
printf '<font color='\''%s'\''> ' "$1"
shift
printf \''%s'\'' ' "$#"
printf '</font>\n'
"$#" || exit 1
}
You would use that function the same way. It will look like this:
$ green='#00ff00'
$ echoAndExec "$green" printf '%s\n' 'foo bar' 'baz quux'
<font color='#00ff00'> 'printf' '%s\n' 'foo bar' 'baz quux' </font>
foo bar
baz quux
(As mentioned above, you don't have to redefine green each time. I've just done so, so that it's clear what green means in the versions of the shell function that I have written, and so that it is clear how to test this easily.)
Although you can do that, I underscore that the command it shows will not always be possible to run as shown, because it does not handle internal single quotes correctly. The command shown to the user is pretty good for humans, but not so great for computers. Therefore, although you can use this modified method, it is probably better to go with the eval-based way shown above, if you need to show the user a command with original (or otherwise proper) quoting.
Of course, it is also possible to process the arguments in a more sophisticated way that, for example, properly converts internal occurrences of ' into '\''.
Further Reading
Even though the shell being used here is probably not Bash, I nonetheless recommend the Bash reference manual for its clear, relatively accessible explanations of the important concepts in Bourne-style shell scripting.
For a more official source for such shells in general, links to specific topics within the Shell Command Language chapter of POSIX are included directly in the text above.
To see some of the information included here presented in another form, my chat with the OP about the question can be consulted.

Related

I'm trying to build a GNU Make Canned Recipe, like so:
define this-will-fail
my_var=1
$(info your variable is $(my_var))
endef
$(this-will-fail)
This causes an error *** missing separator. Stop.
However, the following works as expected:
define why-does-this-work
$(eval my_var=1)
$(info your variable is $(my_var))
endef
$(why-does-this-work)
by printing your variable is 1.
I'm looking at AOSP's build system and frequently see usage of eval paired with define. What's the relationship between these two items, and why can I not create variables "normally" when using define?

The answer stems from a simple point with big ramifications - a makefile has two dialects, 1) the make language itself and 2) the shell language you use for stuff like invoking gcc.
A reminder of the normal make "rule" syntax:
targets : prerequisites
recipe
While the make target and prerequisites is in make
syntax, the stuff in the recipe is in the shell syntax, not make syntax.
This explains why you cannot do stuff like assign make variables as part of a recipe, as so:
all: the_dependencies_of_all
this_will_not_work := because_the_shell_does_not_know_what_this_line_means
In the shell syntax bits, make does three special text manipulations, and then the remaining string is passed verbatim to the shell. The three steps are
deal with escaped newlines (e.g. backslash followed by newline)
run any make functions (like $(filter ....) and $(eval ..)
expand any make target variables (like $#)
In the normal case, this 3-steps-then-shell processs happens once per recipe line, so each line of your makefile recipe is run in a different shell (hence why some calls to make will spawns many /bin/sh sub-processes).
Manipulation #2 above explains why you can use eval inside a recipe to add some make syntax to your recipe. Eval will turn into an empty string, which will cause no problems with the shell, but make will have evaluated the string inside your statement, so stuff like variable definitions is possible.
The precise moment this evaluation happens is a bit tricky to pin down - it seems to happen when the variable is first expanded, which itself depends upon where you reference that variable in your makefile. Perhaps someone can clarify this a bit more, since when this happens has been ramifications for invocations of make that use the -j flag to run multiple recipe lines in parallel
More Info:
https://www.gnu.org/software/make/manual/html_node/Rule-Introduction.html#Rule-Introduction
https://www.gnu.org/software/make/manual/html_node/Recipe-Syntax.html#Recipe-Syntax
https://www.gnu.org/software/make/manual/html_node/Splitting-Recipe-Lines.html#Splitting-Recipe-Lines

I have a script to monitor the Notifications screen page.
I can open it via "Culebra" option "UiDevice" -> "Open Notifications".
The notifications error message from some apps have starting characters which are changing, only a constant pattern like "error for" is common but located at different position of the TextView error messages.
Therefore I can't use a regex with the method findViewWithText(regex) as it seems to use a regex match() instead of regex search(). Another solution for my problem is to use traverse() method with my own transform method which can do a regex search() of the view attribute text, but I can't figure out how to pass a parameter like a regex to my own transform method!?

This works for me to touch on a notification with text USB debugging connected:
vc.findViewWithTextOrRaise(re.compile('.*USB.*'), root=vc.findViewByIdOrRaise('id/no_id/3')).touch()
vc.sleep(_s)
notice this is a modified culebra script, that's why findViewWithTextOrRaise() is using the root argument to limit the search to the subtree which may not be needed in all cases, but it's safer to use.
It's worth to mention, that this works too
vc.findViewWithTextOrRaise(re.compile('.*debugging.*'), root=vc.findViewByIdOrRaise('id/no_id/3')).touch()

I want to do the frequent operations about the the camera. The frequent operations are consist of launching camera -> taking picture -> doing onBackPressed -> launching camera -> ...
Is there a way to do this ?

This is an interesting example to demonstrate the use of AndroidViewClient/culebra and how they can simplify such a task.
Run (using the long options to be self-explanatory):
culebra --start-activity=com.google.android.gallery3d/com.android.camera.Camera \
--verbose --verbose-comments \
--find-views-with-content-description=on \
--output myscript.py
Once culebra finishes, myscript.py contains the autogenerated script. This script will start Camera and try to find all the Views. One of these Views is the shutter button, the one we would like to touch to take a picture. It's something like
# class=android.widget.ImageView
no_id29 = vc.findViewWithContentDescriptionOrRaise('Shutter button')
Edit the autogenerated script and add at the end (your Camera application may be different, just check the generated script to see if your values are the same as mine)
no_id29.touch() # take the picture
vc.sleep(3) # wait a bit
device.press('BACK', MonkeyDevice.DOWN_AND_UP) # exit
Save it.
That's it!, run myscript.py and all the steps will be done.
You can add a loop inside the script if you want to repeat it many times or even run myscript.py inside a loop in the shell.

Anyone knows a good tool for crawling the GUI of an android app? I found this but couldn't figure out how to run it...

Personally, I don't think it would be too hard to make a simple GUI crawler using MonkeyRunner and AndroidViewClient.
Also, you may want to look into uiautomator and UI Testing
Good is a relative term. I have not used Robotium, but it is mentioned in these circles a lot.
EDIT - Added example based on comment request.
Using MonkeyRunner and AndroidViewClient you can make a heirarchy of views. I think AndroidViewClient has a built-in mechanism to do this, but I wrote my own. This code tries to produce a layout similar to that used by the Linux tree command. The space and line variables are used to setup the "prefix" prepended on each line. Of course, this code uses a depth-first, recursive traversal.
def printViewListDepth(view, depth=0):
space = '|' * int(not depth == 0)
space += (' ' * 2 * (depth-1)) + '|' * int(not depth-1 <= 0)
line = '_' * int(not depth == 0) * 2
text = view.getText()
text = text[:10] + int(len(text) > 10) * '...'
print " [*] %s%s%s %s %s" % (
space, line, view.getUniqueId(),
view.getClass().replace('android.widget.', ''), text)
for ch in view.children:
printViewListDepth(ch, depth+1)
You call printViewListDepth as follows, using a ViewClient returned by AndroidViewClient:
printViewListDepth(viewClient.root)
Note that in the above implementation, the class of View is truncated, by removing "android.widget." and the the text of a View is truncated at 10 characters. You can change these to suit your needs.
Edit Crawling the GUI
With AndroidViewClient you can query whether a View is clickable, someView.isClickable(). If it is clickable, you can invoke a touch event on it, someView.touch(). Assuming most button clicks open a different Activity, you will need to come up with a mechanism of getting back to where you came from to do the recursion.
I imagine that it can be done with some effort, but you may want to start with something as simple as invoking a BACK button press, device.press('KEYCODE_BACK', MonkeyDevice.DOWN_AND_UP). You will probably need to handle application-specific special cases as they arise, but this should get you off to a decent start.

You can take a look at Testdroid App Crawler
This is available in Testdroid Cloud as project type and you can easily run it on 15 free devices.

I am trying to toggle Bluetooth using AndroidviewClient. Below is part of the code. I am able to "find Bluetooth" and get the id and text also. Then I want to get get the view for ON/OFF to toggle. When I print in the below for loop, I get 3 views and one of them is the view for ON or OFF based on the current state. How to check in the for loop for this view to toggle it?
Thanks for your help,
BR
Srini
view = vc.findViewWithText('Bluetooth')
print "Bluetooth id :", view.getId()
print "Blutooth Text :", view.getText()
for i in view.parent.parent.children:
print str(i)

The solution is to get the 'Bluetooth' View grandparent and then search for the ToggleButton in that subtree.
To find the View in this case we use the pattern 'ON|OFF', so it doesn't matter the state as toggling it is our objective:
parent = vc.findViewWithTextOrRaise('Bluetooth').getParent().getParent()
vc.findViewWithTextOrRaise(re.compile('ON|OFF'), root=parent).touch()
This is probably the best solution. However there is an alternative that may also suit your needs and it's much easier to obtain.
Run culebra to generate a script template:
$ culebra -i off -t on -d on -C -j on -o ~/tmp/bluetooth.py
then, you see in the script something like (may vary depending on the API level and device):
# class=android.widget.TextView text="Bluetooth"
no_id26 = vc.findViewWithTextOrRaise('Bluetooth')
# class=android.widget.Switch text="OFF"
no_id27 = vc.findViewWithTextOrRaise('OFF')
so, if you just append
no_id27.touch()
to the script it will toggle Bluetooth every time you run it.