The case of the mysterious --help directory (2020)(leahneukirchen.org)
leahneukirchen.org
The case of the mysterious --help directory (2020)
https://leahneukirchen.org/blog/archive/2020/05/the-case-of-the-mysterious-help-directory.html
60 comments
Smart completion is the first thing I remove or disable on any system I work on.
"Complete with a path on <tab>" is simple to understand and reason with and becomes second nature after working a lot in the shell prompt.
"Run a complex set of possibly incomplete/broken rules to come up with a string" is the exact opposite. Commonly bumping at "why isn't this command-line completing/is completing with something unexpected" just negates all benefits of <tab> sometimes working for stuff like git branches, long command-line flags, etc.
"Complete with a path on <tab>" is simple to understand and reason with and becomes second nature after working a lot in the shell prompt.
"Run a complex set of possibly incomplete/broken rules to come up with a string" is the exact opposite. Commonly bumping at "why isn't this command-line completing/is completing with something unexpected" just negates all benefits of <tab> sometimes working for stuff like git branches, long command-line flags, etc.
Why would you want "cd folder/<TAB>" instead of "ls folder" ?
If you are navigating a complex folder structure, it's easy to start typing
> cd path/to/
and then realizing that you don't remember whether you need subdirectory1 or subdirectory2 next. With tab completion, you can just continue typing
> cd path/to/subdirectory1<tab><tab>
to check whether what's in there is what you're looking for. Without tab completion, you need to move to the start of the line, change "cd" to "ls", run that, go back up in the history, change "ls" to "cd" and continue from there. It's much more inconvenient.
> cd path/to/
and then realizing that you don't remember whether you need subdirectory1 or subdirectory2 next. With tab completion, you can just continue typing
> cd path/to/subdirectory1<tab><tab>
to check whether what's in there is what you're looking for. Without tab completion, you need to move to the start of the line, change "cd" to "ls", run that, go back up in the history, change "ls" to "cd" and continue from there. It's much more inconvenient.
You don't need to choose, fex bash has complete-filename which is bound to M-/ by default¹. You can rely on smart completion for most uses, but should you suddenly desire to complete a filename in an odd place you simply use the direct binding. There are a variety of other completion mechanisms(user, var, etc) with default bindings available too for other out-of-context uses.
¹ zsh users can imitate the same effect with a custom zle function using `completer _files`
¹ zsh users can imitate the same effect with a custom zle function using `completer _files`
Then what you want is the autocd shell option enabled. See info bash
For reasons like this, I almost wish software came with a "list of unexpected side-effects of using this software".
Take bash, for instance - each time you run a command (this list is nonexhaustive; I'd love to know others):
1. If you have a `PROMPT_COMMAND`, it'll be run.
2. The command will be echoed to your history file.
3. If you hit <TAB>, arbitrary functions might run (for completion).
I don't know what zsh may do on top of this, but my guess is that it's significantly more than bash. That's not a demerit to zsh - that's exactly what it should do - but visibility would be nice.
Take bash, for instance - each time you run a command (this list is nonexhaustive; I'd love to know others):
1. If you have a `PROMPT_COMMAND`, it'll be run.
2. The command will be echoed to your history file.
3. If you hit <TAB>, arbitrary functions might run (for completion).
I don't know what zsh may do on top of this, but my guess is that it's significantly more than bash. That's not a demerit to zsh - that's exactly what it should do - but visibility would be nice.
> 2. The command will be echoed to your history file.
Important caveat: this goes into an in-memory history buffer; the history file is written when bash exits. This means that if you have multiple shells open, you will lose the history from all but the last shell to exit.
Important caveat: this goes into an in-memory history buffer; the history file is written when bash exits. This means that if you have multiple shells open, you will lose the history from all but the last shell to exit.
The .bash_history truncation feature of bash I find quite annoying
I am used to NetBSD's ash where I set HISTSIZE very high in .profile and save history with
A large number of GNU/Linux use a fork of NetBSD's ash called dash. It is the installed default scripting shell on Debian, Ubuntu, etc. and the default shell in Busybox, OpenWRT, etc.
https://en.wikipedia.org//wiki/Dash_(shell)
Dash can be compiled with history support using libedit.^1 The static dash binary I use is 448k compiled with musl.
Dash is not as nice as NetBSD's ash for interactive use. For example, there's no tab completion in dash^2, hash -v does not work despite being listed in the manpage; there are various other small differences. In any event, I am using it as an interactive shell.
The Debian project decided dash is faster than bash as a scripting shell. For me, it is also faster as an interactive shell, because I am used to libedit and most of the interactive use I do is editing and running scripts/command line history
1. Instructions
I am used to NetBSD's ash where I set HISTSIZE very high in .profile and save history with
fc -l 0 > .history
A large number of GNU/Linux use a fork of NetBSD's ash called dash. It is the installed default scripting shell on Debian, Ubuntu, etc. and the default shell in Busybox, OpenWRT, etc.
https://en.wikipedia.org//wiki/Dash_(shell)
Dash can be compiled with history support using libedit.^1 The static dash binary I use is 448k compiled with musl.
Dash is not as nice as NetBSD's ash for interactive use. For example, there's no tab completion in dash^2, hash -v does not work despite being listed in the manpage; there are various other small differences. In any event, I am using it as an interactive shell.
The Debian project decided dash is faster than bash as a scripting shell. For me, it is also faster as an interactive shell, because I am used to libedit and most of the interactive use I do is editing and running scripts/command line history
1. Instructions
# install libedit - may already be installed for other programs such as openssh or sqlite
wget https://git.kernel.org/pub/scm/utils/dash/dash.git/snapshot/dash-0.5.11.5.tar.gz
tar xzf dash-0.5.11.5.tar.gz
cd dash-0.5.11.5
./configure --with-libedit
make
cd src
gcc -static -Wall -g -O2 -o dash alias.o arith_yacc.o arith_yylex.o cd.o error.o eval.o exec.o expand.o histedit.o input.o jobs.o mail.o main.o memalloc.o miscbltin.o mystring.o options.o parser.o redir.o show.o trap.o output.o bltin/printf.o system.o bltin/test.o bltin/times.o var.o builtins.o init.o nodes.o signames.o syntax.o -ledit -lcurses
s/tcsh was the first/& UNIX/
It took the command completion idea from TENEX and TOPS-20
It took the command completion idea from TENEX and TOPS-20
This is the default, but you can get around this behavior by writing to history immediately.
How do you do that?
Yes! If you want to maintain a full history (and you should!), you can make bash consistently log it:
promptFunc() {
# right before prompting for the next command, save the previous
# command in a file.
echo "$(date +%Y-%m-%d--%H-%M-%S) $(hostname) $PWD $(history 1)" \
>> ~/.full_history
}
PROMPT_COMMAND=promptFuncInstead of using that echo, you can tell bash to append the history:
history -a
shopt -s histappend
HISTFILESIZE=-1
HISTSIZE=-1Unfortunately, with this approach it only takes one invocation of bash without your .bashrc (or otherwise without HISTFILESIZE=-1) to truncate your history to 500 lines:
$ man bash
/HISTFILESIZE
The maximum number of lines contained in the history file.
When this variable is assigned a value, the history file is
truncated, if necessary, by removing the oldest entries, to
contain no more than that number of lines. The default
value is 500. The history file is also truncated to this
size after writing it when an interactive shell exits.
It is for this reason that I also keep my bash history in git. I know other folks that do this too. There have also been solutions on HN in recent months for keeping your shell history in an sqlite database.
The history will not be written if either the shell/OS crashes, or if you `unset HIST_FILE` before exiting the shell.
Also, in most common cases, won’t be written to history file if the command is prefixed with a space (see HISTCONTROL).
You have a typo: it should be `HISTFILE`, not `HIST_FILE`.
> or if you `unset HIST_FILE`
I figured I'd learn something by putting in the "I'd love to know others" bit, but this is something I didn't know I could do, and is genuinely useful to know!
I figured I'd learn something by putting in the "I'd love to know others" bit, but this is something I didn't know I could do, and is genuinely useful to know!
[deleted]
No, it will append the newly run commands. Or at least on SUSE it does, maybe other distros configure it differently.
For that matter, it would be nice if software with side effects of this sort disclosed them at runtime by default. For all its faults, this is something npm gets (partly) right. Partly, in this case, because it does print the commands it runs and their output, but it doesn’t tell you why.
I’m sure this would be overly verbose for most users, who would likely turn it off, but it would still be educational in terms of the huge set of implicit side effects normal every day software produces.
I’m sure this would be overly verbose for most users, who would likely turn it off, but it would still be educational in terms of the huge set of implicit side effects normal every day software produces.
Semi related: if you write software that takes only one of -h or --help, but not both, you are bad and you should feel bad.
GNU cat is on the naughty list, supporting --help but not -h. less is the same (and is not part of GNU).
edit Looks like the GNU echo, sort, and uniq programs are the same. GNU has an official policy endorsing --help but not -h. [0]
Also related: --version should print the version numbers (and perhaps build details) of the software, and immediately terminate.
-v lacks the (at least somewhat) universal meaning of -h, though. Some commands, such as curl, use -v as shorthand for --verbose rather than --version, but all commands should support --version.
I believe Java HotSpot used to use -version, supporting neither -v nor --version. Now it supports both --version and -version, but not -v.
[0] https://www.gnu.org/prep/standards/html_node/_002d_002dhelp....
edit Looks like the GNU echo, sort, and uniq programs are the same. GNU has an official policy endorsing --help but not -h. [0]
Also related: --version should print the version numbers (and perhaps build details) of the software, and immediately terminate.
-v lacks the (at least somewhat) universal meaning of -h, though. Some commands, such as curl, use -v as shorthand for --verbose rather than --version, but all commands should support --version.
I believe Java HotSpot used to use -version, supporting neither -v nor --version. Now it supports both --version and -version, but not -v.
[0] https://www.gnu.org/prep/standards/html_node/_002d_002dhelp....
And do not forget the most important flag:
/?Even more so, the worst coworker I ever had once wrote a script that accepted neither, and (when run) would default to trying to process data (and deleting it whether or not it was fully handled) _in production_. The first time we ever ran it (unsuspectingly) was after he'd left for Facebook. Good riddance.
One place I worked had very explicit rules about that that would be drilled into new recruits. Besides --help, running without argument was also required to print the help.
I was told that this rule came about when they "discovered", presumably at nontrivial cost, a disk imaging tool that would default to unconditionally mirroring the first hard disk onto the second.
I was told that this rule came about when they "discovered", presumably at nontrivial cost, a disk imaging tool that would default to unconditionally mirroring the first hard disk onto the second.
Oooo, not failing on unknown arguments is evil. It's almost guaranteed at that point that what you're about to do is not what the user expects to happen.
Not as bad as blindly assuming -h prints help.
To be honest, it should, or if you feel like your program is very simple then you could do what feh(1) does and simply print "see man $PROGRAM". Running -h or --help on a program which spits out an "unknown argument" error is one of my pet peeves.
That reminds me of a function I have in my environment called `dg` that does `cd` then `ls`. The name `dg` is short for dig, since the name `dig` was taken.
I also sometimes write `2>1` instead of `2>&1` and get a mysterious file named `1` in my current directory. Luckily it I figured that one out pretty quick.
I also sometimes write `2>1` instead of `2>&1` and get a mysterious file named `1` in my current directory. Luckily it I figured that one out pretty quick.
Indirectly related story from my workplace: the application that I maintain changes some terminal line settings, such that after it is run, we have to use Ctrl-H instead of backspace (unless we also change PuTTY settings to send Ctrl-H for backspace).
This means that sometimes people accidentally create files whose name is a backspace.
These cannot be seen directly as '\127' via ls (or at least the default Solaris ls; not sure what GNU does), but instead as the output missing a character, usually padding, so people do not realise it is a problem.
Then it causes issues when the file gets caught in globs that process all files in the directory, but the offending file is almost invisible, leaving the user confused.
And we cannot change the program not to change terminal settings because who knows what that will break!
This means that sometimes people accidentally create files whose name is a backspace.
These cannot be seen directly as '\127' via ls (or at least the default Solaris ls; not sure what GNU does), but instead as the output missing a character, usually padding, so people do not realise it is a problem.
Then it causes issues when the file gets caught in globs that process all files in the directory, but the offending file is almost invisible, leaving the user confused.
And we cannot change the program not to change terminal settings because who knows what that will break!
> Indirectly related story from my workplace: the application that I maintain changes some terminal line settings, such that after it is run, we have to use Ctrl-H instead of backspace (unless we also change PuTTY settings to send Ctrl-H for backspace).
Things like that is why I use PROMPT_COMMAND="stty sane"
Things like that is why I use PROMPT_COMMAND="stty sane"
The one good thing about having a file/directory called "--help" lying around is that it protects you against running "rm -rf *".
Yeah I figured it was shell customization gone awry early on, with 'mkdir --' in the mix. But if they'd figured that out immediately it wouldn't have been a fun post; busting out some custom unix plumbing tools and showing how cool they are is really what the article is about. Also gotta admit they have a pretty darn fun github.
> showing how cool they are
That would've been writing a custom kernel extension to track this down, but fortunately tracing exists these days.
Seriously, I don't see anything wrong here, it's about the same route I would've gone down. Except the other way round because writing a custom ~/bin/mkdir would've saved me reading up.
That would've been writing a custom kernel extension to track this down, but fortunately tracing exists these days.
Seriously, I don't see anything wrong here, it's about the same route I would've gone down. Except the other way round because writing a custom ~/bin/mkdir would've saved me reading up.
[deleted]
I ran into a similar problem on windows, using process monitor with the appropriate filter made it pretty easy to find the problem. (I had a batch file containing a forgotten copy command getting called as a side-effect of some other tooling on my PC).
another datum in the "why you should never customise your environment" argument.
Actually it should be: "Customize your environment, but take some damn notes FFS."
another datum in the "you can fix everything if you can control your environment"
Nah, shell environments are really begging to be customized. Just be mindful of it. In a sense, just installing command line tools is customization already.
More like "why you should be aware of all the hidden side-effects if you do".
Didn't push a fix upstream?
My understanding is that this function was in her dotfiles, so by virtue of fixing it there it was fixed upstream.
That isn't what it sounds like to me.
> Jannis Harder told me that the zsh completion for mkdir would run mkdir --help (to see if it’s the GNU variant), which I could verify.
Completions are typically installed as an add-on or as part of a core library of a shell.
I found this line, maybe it's the culprit? https://github.com/zsh-users/zsh/blob/master/Completion/Unix...
> Jannis Harder told me that the zsh completion for mkdir would run mkdir --help (to see if it’s the GNU variant), which I could verify.
Completions are typically installed as an add-on or as part of a core library of a shell.
I found this line, maybe it's the culprit? https://github.com/zsh-users/zsh/blob/master/Completion/Unix...
> > Jannis Harder told me that the zsh completion for mkdir would run mkdir --help (to see if it’s the GNU variant), which I could verify.
> Completions are typically installed as an add-on or as part of a core library of a shell.
Right, but the completions were doing the correct thing for `mkdir`. The problem is that in their dotfiles they told it to use the `mkdir` completions for a command that isn't `mkdir`; and what is correct for `mkdir` isn't correct for that other command.
This fix was to make this edit to their own dotfiles:
> Completions are typically installed as an add-on or as part of a core library of a shell.
Right, but the completions were doing the correct thing for `mkdir`. The problem is that in their dotfiles they told it to use the `mkdir` completions for a command that isn't `mkdir`; and what is correct for `mkdir` isn't correct for that other command.
This fix was to make this edit to their own dotfiles:
-compdef mkcd=mkdir
+_mkcd() { _path_files -/ }
+compdef _mkcd mkcdThe interesting question here (to me) is what an actual fix would even be - given the proximate trigger was basically "lying to zsh about the argument parsing of the thing you were completing" (and obviously we all lie to software all the time, being a cursed sand herder be like that) I'm not sure how you could make the completion setup in question avoid triggering that problem without being less robust for the majority of users.
(this said more in a spirit of morbid fascination with these sorts of weird interactions than anything else, I'm genuinely unsure if there's a right answer here, only "pick which sort of wrong will be least intolerable for your userbase as a whole")
(this said more in a spirit of morbid fascination with these sorts of weird interactions than anything else, I'm genuinely unsure if there's a right answer here, only "pick which sort of wrong will be least intolerable for your userbase as a whole")
I think you've pretty much said why there is nothing to "fix" upstream here. zsh comp was expecting mkdir.
If I'd use a completely unrelated program (e.g. tar) and told zsh completion to handle it as if it were mkdir, nobody would be surprised if things misbehaves spectacularly.
If I'd use a completely unrelated program (e.g. tar) and told zsh completion to handle it as if it were mkdir, nobody would be surprised if things misbehaves spectacularly.
Just today, I deleted a directory called --process in my home. Most likely, I just forgot a -- or otherwise messed up a command. We'll see if it comes back!
Why does zsh want to know where mkdir came from?
The tab completion sniffs the `--help` output to determine whether it should tab-complete `--long` flags and if so whether it should include `--context` (SELinux) among them.
https://github.com/zsh-users/zsh/blob/master/Completion/Unix...
https://github.com/zsh-users/zsh/blob/master/Completion/Unix...
Presumably to figure out which options the completion engine allows you to select. Being able to select GNU extensions on a BSD mkdir would be silly.
As someone who routinely uses tab completion to look into the contents of directories before entering them to confirm that they are the correct ones, I was very confused when a recent system I had to use briefly (I don't remember if it was zsh) decided to not show files if the command started with 'cd'. It wasted more time trying to be "helpful" because it required an extra 'ls' which I could essentially get for free via tab completion.