The roots of an obscure Bourne shell error message(utcc.utoronto.ca)

utcc.utoronto.ca

The roots of an obscure Bourne shell error message

https://utcc.utoronto.ca/~cks/space/blog/unix/BourneShellObscureErrorRoots

39 comments

pinusc3 anni fa

Excellent explanation! However, I would disagree both that there isn't an easy fix and that each of the decisions are sensible.

If the syntax to temporarily set variables was different, say using : instead of =, or requiring some delimiter between assignments and operations, or maybe enclosing the statement in braces (something like: {foo=bar; echo $foo}...) to delimit temporary assignments would have been reasonable alternatives, perhaps even more readable than this. Or having the sntax disallow `foo=`, requiring, say, `foo=null` or `foo=''`, which are both more explicit, or even `unset foo`. These are more sensible individual choices, and that's partly because they're more explicit and partly because they avoid a (now) common error. I'd argue good language design needs to think slightly bigger picture than "individually not catastrophic" choices. Sh is littered with a myriad of small bad choices, which add up to a language that's hard to code or script without bugs. Nowadays with shellcheck it's easier, but it shouldn't be necessary. The consensus is: it's a scripting language, it shouldn't be used for complex stuff. But if it was a slightly better scripting language, it _could_ be used more safely for larger or more complex scripts.

The truth is that it's an ancient language, its (bad) syntax is set in stone, and there's nothing we can do about it except slowly move to alternatives. Sad that there's nothing established to take its place (Perl is read-only, python is not good enough as unix glue, everything else is too obscure).

I know how and love to write in bash. But oh god was it painful to learn

Galanwe3 anni fa

I don't think your comment fairly represents the situation.

Bash is not _just_ a scripting language, it's also (and primarily) a shell.

That means it needs to be good at both interactive REPL-style usage _and_ at simple scripting.

Being a shell implies that it has to be exceptionally succinct and flexible about executing programs and manipulating environment variables.

In that light, your comparison with Python and Perl don't make much sense.

Would you enjoy replacing:

    $ cd /tmp
    $ ls -l

with:

    $ os.chdir('/tmp')
    $ subprocess.run(['ls', '-l'])

I guess not.

Shells will always be a different breed of languages. Bash is maybe not the best at what it could be, but it's not the end of the world either.

The example in the article is a bit silly to be honest. I think the result is kind of obvious for anyone half bash litterate.

pinusc3 anni fa

You are right to point out that in my comment I talk about bash in terms of a language rather than just a shell; but I did have its shell-nature in mind while writing it, and in fact your example is exactly what I meant when I said that python is not good enough unix glue... there's a certain level of script complexity where python starts making more sense than bash, but because the ergonomics of calling other programs (unix glue!) or building pipelines are pretty poor, that threshold is pretty high.

Some of bash syntax makes perfect sense in the context of a shell-but-also-scripting-language. Some of it makes perfect sense in the context of "everything is a string", which I think is an absolutely great abstraction---for example, unset variables being equivalent to empty strings. Other parts are just jargony, and might make the life of a few experienced shell-slingers slightly easier, while complicating many users' interactions with the shell.

I suppose part of my point is that succintness here comes at the cost of readability. A shell could very well be _slightly_ more verbose than bash (say, requiring assignment operators to have two sides, so `foo=''` instead of `foo=`), still be plenty succint as a shell, but being slightly more explicit and readable in a scripting context. `foo=''` is only _two_ characters longer than `foo=`, and is already-implemented behavior. But the `foo=` syntax has certainly confused generations of unix users... that's just not a good tradeoff imo. I feel pretty comfortable calling this lack of foresight, or suboptimal design; that's not bashing (!) the language or its creators, rather, merely noting that we can do better.

And if you don't think so... check out fish shell! It's a beloved and successful shell, and they forgo the use of = entirely (instead, you have to `set foo bar`, which is more verbose and a lot more explicit!)

Edit: fish does allow `foo=bar echo $foo` to override variables, so they actually manage to keep bash ergonomics while avoiding mistakes while scripting.

jhoechtl3 anni fa

> That means it needs to be good at both interactive REPL-style usage _and_ at simple scripting.

The creators of powershell obviously were of a different opinion

Kwpolska3 anni fa

It is fairly good for interactive usage, there are aliases for common commands, case-insensitivity, tab completion etc.

paddim83 anni fa

I'm working on a shell language that combines these things.

You can write things like: let column = du -h | str::column 0 echo(column[1..3])

kpcyrd3 anni fa

For programming, I would gladly go with the second option in return for proper flow-control mechanics and data structures.

Izkata3 anni fa

> or maybe enclosing the statement in braces (something like: {foo=bar; echo $foo}...) to delimit temporary assignments

As long as you don't need to get anything out of it, it's a bit heavy but dropping to a subshell works like that:

  $ (foo=bar; echo $foo)
  bar
  $ echo $foo

__float3 anni fa

> I know how and love to write in bash. But oh god was it painful to learn

This is very well said :)

> Sad that there's nothing established to take its place (Perl is read-only, python is not good enough as unix glue, everything else is too obscure).

Is there anything notable? Either particularly well designed, or just popular? I think I've only ever heard of Oil (https://oilshell.org)

gavinray3 anni fa

Ruby has very ergonomic and pleasant UX for doing the sorts of things you typically want to do in bash scripts IMO.

bdamm3 anni fa

And today I learned about shellcheck.

Thank you for that fun and concise trip into shell evaluation :)

da39a3ee3 anni fa

For what it’s worth, the explanation of the error was obvious to me after seeing the code for a few seconds. So maybe my point is… don’t worry if stuff like that in shell programming seems obscure to you; it becomes much less obscure with practice (and along the way you do learn genuine lessons about unix-like operating systems, not just shell programming details).

rob743 anni fa

The problem is that most people (me included) don't want to become proficient in writing shell scripts, but they need to write a script (or, God forbid, extend and/or debug a script someone else wrote) from time to time. At least I do it so rarely that I forget most things from one time to the next. I understand that Bash is the way it is because of historical reasons, but maybe some less obscure error messages would be nice? ShellCheck is all nice and well, but Bash itself being more helpful would be better. Or, no, please don't tell me that other tools rely on these error messages and therefore they can't be changed?

da39a3ee3 anni fa

> maybe some less obscure error messages would be nice?

Bear with me here! The error message is exactly what we’d expect in this situation. The author literally tried to use the string “107” as an executable name, and bash says ‘I can’t find an executable named “107”’.

It’s impossible to be a good software engineer in 2023 without understanding the basics of shell programming. And the absolute number one thing to understand about bash and related shell languages is that “everything is text”. I know one hears people saying that a lot, but you’ve got to really internalize what it means. bash is just constructing strings of text and doing stuff with it. It barely has any non-string data structures.

Joker_vD3 anni fa

> The author literally tried to use the string “107”

Well, no. That string was not literally used there at all: if it was, there would have been a literal 107 in the user's input. There wasn't.

> It’s impossible to be a good software engineer in 2023 without understanding the basics of shell programming

> bash is just constructing strings of text and doing stuff with it

It's a reassuring knowledge in 2023, after 40 years of countless bugs, exploits, vulnerabilities and accidentally rm-ed important files, the paradigm of "just constructing strings of text and (blindly) doing stuff with it" is still ubiquitously used and one can not be a good software engineer without extensive knowledge of it. Bash truly deserves description of "a mistake, carried through to perfection. It is the language of the future for the programming techniques of the past: it creates a new generation of coding bums" much more than APL.

da39a3ee3 anni fa

Agreed, my use of literally was incorrect. But otherwise I don't think we disagree! Bash/zsh etc is a terrible language and it's absurd that knowing it is still required. Nushell is a better direction, but I don't know how the transition's going to be made; I just went back to zsh from nushell.

da39a3ee3 anni fa

Actually, my use of "literally" is correct as hibbelig points out. I never said that they used the string literal "107". They literally tried to do something which would clearly lead to attempting to use a string representation of an integer as an executable name.

hibbelig3 anni fa

The author literally tried to use 107 as a command.

The author did not try to use literally 107 as a command.

[deleted]3 anni fa

throwaway462813 anni fa

I agree with you. We can complain about Bash all we want, but Bash is everywhere and isn't going anywhere anytime soon (as much as I'd like something better). So if you want to be a good engineer in today's world, you have to learn it. And if the error message isn't immediately obvious to you yet, don't worry, it just means you haven't yet formed the right mental models. But you will, if you continue to educate yourself by reading articles like these!

jlv23 anni fa

Sorry, that's not obscure!

rob743 anni fa

...not if you live and breathe shell scripts, no. But most people coming from other programming languages first need to understand the concept of whitespace being significant, and even if they do, it's easy to insert a space accidentally. In this case, if you get this error message, you would probably first look at the part in the "$(...)" (which, you think, is the only thing resembling a "command" on that line) and wonder why on earth it doesn't work, overlooking the problem slightly to the left of it.

naniwaduni3 anni fa

I would expect to look at the part inside the $(), probably attempt to run it to confirm that it's working at all, and be a bit curious as to why the error message implies I was trying to run the output of that command!

blueflow3 anni fa

I remember yaml and python also having syntactically relevant whitespaces.

1vuio0pswjnm73 anni fa

This does not seem obscure to me at all. Having learned about C from reading djb's programs, I write programs that read their arguments from environmental variables. (See djbdns for example.) When I run one of these programs it's common to set variables on the command line, e.g.,

   echo https://example.com|Connection=keep-alive yy025|nc -vvn 127.127 80

yy025 sets the HTTP Connection header according to the value of the corresponding environmental variable. If I want to send a POST request I set the variable httpMethod to POST. The address 127.0.0.127 is a TLS forward proxy. Original netcat allows one to type 127.127 without the zeros.

Using djb's envdir utility, one can put a selection of different HTTP headers into a directory so that there's no need to type them on the command line.

    cd /path/to/dir
    echo close > connection
    echo aplication/x-www-form-urlencoded > content-type
    echo id=123\;key=xyz > cookie
    echo 19 > content-length
    echo POST > httpMethod
    echo /api/whatever > path
    echo "{ \"key\": \"value\" }" > post-data
    cd
    echo https://example.com \
    |if cd /path/to/dir;envdir . yy025;then 
    cat post-data;fi \
    |nc -vvn 127.127 80

It's strange to me to see this author use printenv instead of set (a built-in) to display Bourne shell environmental variables. As someone who uses Bourne shell every day and on a variety of computers, I never use printenv. On the smaller systems I make, I do not include printenv in the userland.

The use of fgrep to match a single word in all caps is also strange.

Maybe I am just missing some UNIX wisdom here. If so, please disregard this comment.

From the article

  $ export FRED=value
  $ printenv | fgrep FRED
  FRED=value
  $ FRED=
  $ printenv | fgrep FRED
  FRED=
  $ unset FRED
  $ printenv | fgrep FRED
  $ # ie, no output from printenv

Without using printenv or grep -F (in shell script /bin/fgrep)

  $ export FRED=value
  $ set|grep FRED
  FRED='value'
  FRED=
  $ set|grep FRED
  FRED=''
  $ unset FRED
  $ set|grep FRED
  $ # ie, no output from set

siebenmann3 anni fa

In this case, one reason to use printenv is that it is an external command, and so it is clearly and unambiguously seeing (and reporting) whatever the Bourne shell would export into the environment for a real command. In this specific case, it appears that set does not report such single-command variables (whether or not they have a value, eg 'FRED=barney set | grep FRED').

(I am the author of the linked-to entry.)

1vuio0pswjnm73 anni fa

Missing semicolon.

   FRED=barney; set | grep FRED

Also there is no need for grep, nevermind grep -F (fgrep), if using printenv.

   FRED=barney printenv FRED

Without semicolon

   sh -c 'FRED=barney set|grep FRED'

   echo 'FRED=barney set|grep ^FRED'|sh

_TwoFinger3 anni fa

> Missing semicolon.

It's not missing. PP is making a point about "single-command variables."

Izkata3 anni fa

Single-command environment variables, to be precise. The same thing that gets created when you use "export", but isn't created when you use "=" on a line on its own:

  $ FOO=BAR env | grep FOO   # Exists as an environment variable passed to env
  FOO=BAR
  $ env | grep FOO           # But only for that command
  $ FOO=BAR                  # Creates a variable (not an environment variable)
  $ env | grep FOO           # As shown by not being passed down
  $ echo $FOO                # But still accessible in this context
  BAR
  $ export FOO=BAR           # Promotes it to an environment variable (the "=BAR" is optional since it's already defined)
  $ env | grep FOO           # And as such is now visibly passed down
  FOO=BAR

(Yes, bash does mix these together so it's easy to accidentally replace something you don't intend to)

dcassett3 anni fa

>Missing semicolon.

If you add a semicolon then FRED is no longer a single command variable (it persists)

> on NetBSD, one can do this

Linux also does the expected thing:

$ FRED=barney printenv FRED

barney

1vuio0pswjnm73 anni fa

   Without using printenv or grep -F (in shell script /bin/fgrep)

   $ export FRED=value
   $ set|grep FRED
   FRED='value'
   $ FRED=
   $ set|grep FRED
   FRED=''
   $ unset FRED
   $ set|grep FRED
   $ # ie, no output from set

(Missed a dollar sign.)

Another thing is that printenv does not make it easy to see spaces or other nonprinting characters in the values of environmental variables. For example PS="# " displays as PS=#. TAB=$(printf '\11') displays as TAB=. For me, that's not really helpful. Whereas the builtin set command displays the values in single quotes so one can easily the presence of spaces and other non-printing characters.

A workaround is to use printenv with sed, something like

    printenv|sed -n l

If I was going to investigate "single command environmental variables" I might just use getenv and printf. That way I can see the non-printable characters. But not being an IT person or a programmer, maybe I am missing the reason why printenv, written in 1979, would be preferable to getenv. (That, IMO, would be an interesting blog post.) For example,

   int printf(const char *__restrict, ...);
   char *getenv(const char *);
   int main()
   {
   if(getenv("FRED"))
   printf("%s=%s\n","FRED",getenv("FRED"));
   return 0;
   }

usage: FRED= a.out

zerocrates3 anni fa

A lot of people are probably just not familiar with the single-command envvar syntax, and of people that do know it some of them probably don't know the syntax for unsetting.

So it's just not immediately obvious to many why a simple extra space is a problem at all, and why it's this specific error. It can also just generally be hard to see past what you intended to write to what you actually did and how that will be parsed.

[deleted]3 anni fa

Izkata3 anni fa

"printenv" (or just "env", the one I habitually use) shows only environment variables, not all available functions and variables like "set" does.

But otherwise yeah, I also don't see this as obscure at all. Was expecting something confusing when, say, fiddling with arrays or something.

1vuio0pswjnm73 anni fa

Good point about functions. Normally it's not something I notice because I'm not using Bash. But every once a while I use set in Bash and I'm blasted with a slew of preloaded functions.

[deleted]3 anni fa

cmeacham983 anni fa

Can I suggest a simple fix that doesn't involve changing behavior at all but probably improves the lives of the many users who hit this?

Just add an additional line to the error message for lines that look like this pattern:

    sh: 107: command not found
    Did you mean "AVAR=$(... | wc -l)"?

hnben3 anni fa

I feel like that approach would create much effort to do properly. I.e. to be consistent, you would add a lot of such messages, and that would add a lot of complexity to the overall code base.

kazinator3 anni fa