Zsh-autoquoter makes shell quoting slightly less annoying(ianthehenry.com)
ianthehenry.com
Zsh-autoquoter makes shell quoting slightly less annoying
https://ianthehenry.com/posts/zsh-autoquoter/
26 comments
It's puzzling to me that people still think of zsh as "that newfangled shell I'm not sure about yet".
Bash came out in 1989. Zsh came out in 1990.
Bash came out in 1989. Zsh came out in 1990.
i use zsh, but i often invoke bash just to make sure some behavior is "normal" rather than zsh-specific, or specific to how i wrote my .zshrc last decade [and haven't changed].
i don't think it's about the newfangledness; i think it's about the potential for subtle differences in syntax which scare people away from zsh.
i don't think it's about the newfangledness; i think it's about the potential for subtle differences in syntax which scare people away from zsh.
I think the initial setup can be quite daunting; unlike bash, which never seemed to have moved beyond 1990, zsh added a lot of features, many of which are very useful (some of which less so, like csh-compatibility things) but there's a lot of it, and the defaults are somewhat bare-bones. Hence things like oh-my-zsh, which is quite intimidating in its own way.
The fish shell "fixed" a lot of that by adding more or less the same feature-set as zsh but without the extensive configurability, which is a really great trade-off if the fish defaults work well for you (they don't for me personally though).
The fish shell "fixed" a lot of that by adding more or less the same feature-set as zsh but without the extensive configurability, which is a really great trade-off if the fish defaults work well for you (they don't for me personally though).
Write your scripts using bash or sh in the hashbang and use zsh for your daily work. Also, use shellcheck.
It is useful to try your scripts in a POSIX shell that avoids non-standard extensions. The best known is the Debian dash reroll of the Almquist shell, but there are other implementations (I've heard of mrsh and gsh, and the one in OCaml).
POSIX shells do not support arrays, and many other common bash extensions. However, dash is very small, and very fast (4x faster than bash according to some sources).
Scripts written for the POSIX shell are maximally-portable.
POSIX shells do not support arrays, and many other common bash extensions. However, dash is very small, and very fast (4x faster than bash according to some sources).
Scripts written for the POSIX shell are maximally-portable.
I use Zsh for CLI and Bash for scripting. Never ran into any problem, production-wise.
More flexible would be a double-quote-region widget which can be applied arbitrarily to any region of a command line:
emulate -L zsh
if [[ "$REGION_ACTIVE" -eq 0 ]]; then
return
fi
if [[ "$MARK" -le "$CURSOR" ]]; then
left_region_bound="$MARK"
right_region_bound="$CURSOR"
else
left_region_bound="$CURSOR"
right_region_bound="$MARK"
fi
length_region="$((right_region_bound - left_region_bound))"
before_region="${BUFFER:0:$left_region_bound}"
region="${BUFFER:$left_region_bound:$length_region}"
after_region="${BUFFER:$right_region_bound}"
quoted_region="${(qqq)region}"
cursor_offset="$(($#quoted_region - $#region))"
BUFFER="$before_region$quoted_region$after_region"
CURSOR="$((CURSOR + cursor_offset))"
REGION_ACTIVE=0
zle -N double-quote-region
bindkey '\e"' double-quote-regionThat's truly an impressive Moth. Good find. The zsh plugin looks nice too!
One of the nice things about fish is that it does this when pasting: https://github.com/fish-shell/fish-shell/issues/967
So can Zsh -- this functionality has shipped with Zsh since for several years, though not enabled by default.
See https://github.com/zsh-users/zsh/blob/master/Functions/Zle/b... or https://github.com/zsh-users/zsh/blob/master/Functions/Zle/b...
See https://github.com/zsh-users/zsh/blob/master/Functions/Zle/b... or https://github.com/zsh-users/zsh/blob/master/Functions/Zle/b...
Honestly, the "when pasting" part seems very much a case of solving the wrong problem there. For my shell, I just had any unquoted ':/' or ':?' mark the word as a url, causing glob characters (and some others like '&', for obvious[0] reasons) to be implicitly quoted until the end of the word.
0: assuming you've seen a url like http://example.com/foo?bar=1&baz=2 before
0: assuming you've seen a url like http://example.com/foo?bar=1&baz=2 before
PowerShell does this too, at least for URLs. That said, it’s a true pain in the ass when it kicks in incorrectly, which unfortunately does happen.
Try fish. The quoting rules are simple. I never have to read the docs or try repeatedly to get quoting right. The shell syntax in general is small and simple.
The only downside to fish is that it's not POSIX compliant, but if you're familiar with POSIX shells this is rarely a problem for interactive use, and you can use Bash for scripting if you like.
The only downside to fish is that it's not POSIX compliant, but if you're familiar with POSIX shells this is rarely a problem for interactive use, and you can use Bash for scripting if you like.
I'd recommend skipping directly to how the matching configuration works: https://github.com/ianthehenry/zsh-autoquoter#behavior
The shell quoting rabbithole goes pretty deep, especially when you want to preserve the whitespace within arguments. Has this been solved generally yet? I suspect it requires full knowledge of whatever shell is running on the other side of SSH.
Just thinking of one of OP's examples, his (very cool) autoquoter allows him to type:
> ssh user@host awk '{print $1}' file.txt
And have the "$1" quoted or escaped so that it's interpreted as an awk keyword on the remote system. But what if I actually want it interpreted as a shell variable that should be expanded on the local or remote system? How does it know?
Why is shell quoting still so hard?
> ssh user@host awk '{print $1}' file.txt
And have the "$1" quoted or escaped so that it's interpreted as an awk keyword on the remote system. But what if I actually want it interpreted as a shell variable that should be expanded on the local or remote system? How does it know?
Why is shell quoting still so hard?
This is a good question without a good answer. zsh-autoquoter doesn't know -- it doesn't know anything, really. The only thing it does is pass the literal string you type as an argument to the command you run (in this case, ssh).
So you could get the "treat it as a shell variable on the remote server" behavior by writing this instead:
So you could get the "treat it as a shell variable on the remote server" behavior by writing this instead:
ssh user@host awk "{print $1}" file.txt
> > ssh user@host awk '{print $1}' file.txt
That's the behaviour in bash (and I think pretty much every other vaguely Bourne-like shell). That's what single quotes do.
> what if I actually want it interpreted as a shell variable that should be expanded on the local [...] system?
$ ssh user@host awk "{print $1}" file.txt
> what if I actually want it interpreted as a shell variable that should be expanded on the [...] remote system?
$ ssh user@host sh -c 'awk "{print $1}" file.txt'
I mean, yes I know how to quote and escape. My point was that only I know what my intent was. Do I want local shell interpretation, remote shell interpretation, awk interpretation or a literal string? I'm not sure how great an autoquoter is going to be if it doesn't know my intent. In the end, I'll need to do it myself, since I know what I want.
And the shame of that is that there aren't a lot of tools to help. Utility functions like shell_escape() would be nice, but you still need to handle escaping for the local shell. That's essentially true of any programming language.
And the shame of that is that there aren't a lot of tools to help. Utility functions like shell_escape() would be nice, but you still need to handle escaping for the local shell. That's essentially true of any programming language.
> My point was that only I know what my intent was. [...] In the end, I'll need to do it myself, since I know what I want.
I think we're in violent agreement here; my point was that the autoquoter is superfluous, since the command in your previous comment already does what
> his (very cool) autoquoter [supposedly] allows
even without a autoquoter.
> Utility functions like shell_escape() would be nice
Note that in this case you specifically do not want shell_escape(). You're trying to produce awk code that evaluates to a given string, and that requires knowing awk syntax, not shell syntax; if you escape according to shell syntax, a attacker may be able to find a string where the shell expression for that string, when interpreted as awk code, does something other than evalute to a string.
You could have a generic_escape() function that (say) replaced any non-alphanumeric byte with '\xHH' or '\B' (for 'B'==0xHH), but there will always some language where whatever generic strategy you picked doesn't work.
I think we're in violent agreement here; my point was that the autoquoter is superfluous, since the command in your previous comment already does what
> his (very cool) autoquoter [supposedly] allows
even without a autoquoter.
> Utility functions like shell_escape() would be nice
Note that in this case you specifically do not want shell_escape(). You're trying to produce awk code that evaluates to a given string, and that requires knowing awk syntax, not shell syntax; if you escape according to shell syntax, a attacker may be able to find a string where the shell expression for that string, when interpreted as awk code, does something other than evalute to a string.
You could have a generic_escape() function that (say) replaced any non-alphanumeric byte with '\xHH' or '\B' (for 'B'==0xHH), but there will always some language where whatever generic strategy you picked doesn't work.
$ echo "\x3E"
x3E # shell doesn't accept \xHH
$ echo '12+3<45' | grep -oE '\<.' | paste -sd' '
1 3 4 # posix ERE doesn't accept \B> Why is shell quoting still so hard?
The problem with the awk example here specifically is that you're mixing two different languages (shell and awk), and that's always going to be painful to a degree. Doing something like running Python code directly from a Ruby script also isn't fun, especially not if you want Python variables in that Ruby script.
And to make it extra fun in the ssh you're also adding a second shell (on the system you're sshing to), so you have to think about 1) your machine's shell, 2) the host's shell, and 3) awk.
So yeah ... that's going to be tricky, but in this case it's not really a shell problem as such.
The problem with the awk example here specifically is that you're mixing two different languages (shell and awk), and that's always going to be painful to a degree. Doing something like running Python code directly from a Ruby script also isn't fun, especially not if you want Python variables in that Ruby script.
And to make it extra fun in the ssh you're also adding a second shell (on the system you're sshing to), so you have to think about 1) your machine's shell, 2) the host's shell, and 3) awk.
So yeah ... that's going to be tricky, but in this case it's not really a shell problem as such.
> The problem with the awk example here specifically is that you're mixing two different languages (shell and awk)
Two languages, but three environments altogether. I'm running an awk script, in a remote shell script, in a local shell script. So a string with a meta-character or keyword within the awk script needs to be triply escaped. And there's no shell_escape(shell_escape(awk_escape(a literal dollar one $1))) available to make it less painful.
Two languages, but three environments altogether. I'm running an awk script, in a remote shell script, in a local shell script. So a string with a meta-character or keyword within the awk script needs to be triply escaped. And there's no shell_escape(shell_escape(awk_escape(a literal dollar one $1))) available to make it less painful.
An autoquoter is a bit too much magic for me, but I wish there was a quote explainer. Much like 'cdecl' for C declarations, a program that explains expressions with quotes in plain English. Problem, I guess, is that shell quoting depends on the context where it is used.
Is something like this available for bash?