Shell Style Guide(google.github.io)
google.github.io
Shell Style Guide
https://google.github.io/styleguide/shell.xml
335 comments
It's because so many here on hn are the compsci theoritical programmer, sysadmins are dying devops is everything, kind of people who don't actually support production systems other than maybe their particular depts single page webapp, etc...
Hn has a problem with people living in the SV filter bubble seeing everything through that lens... in the real world, bash is still a king.
When your devs shit out some bad code that causes your sysadmin to get a call at 3am... he's using bash to fix your fucking nodejs or whatever fadofthemoment bullshit you decided to run with. Of course entire applications generally shouldn't be in bash... it's like a kit of duct tape, epoxy, and rivets. Of course you should have made the product better, but when it does break, bash can keep it together until you get to the shop.
I also think there is a certain amount of eliteism. It has such a low barrier to entry, it's like some people hate the idea of people being able to program without being programmers.
I've heard it all so much here it just goes in one ear and out the other.
/end bofh rant
Hn has a problem with people living in the SV filter bubble seeing everything through that lens... in the real world, bash is still a king.
When your devs shit out some bad code that causes your sysadmin to get a call at 3am... he's using bash to fix your fucking nodejs or whatever fadofthemoment bullshit you decided to run with. Of course entire applications generally shouldn't be in bash... it's like a kit of duct tape, epoxy, and rivets. Of course you should have made the product better, but when it does break, bash can keep it together until you get to the shop.
I also think there is a certain amount of eliteism. It has such a low barrier to entry, it's like some people hate the idea of people being able to program without being programmers.
I've heard it all so much here it just goes in one ear and out the other.
/end bofh rant
Shell code can run in a whole bunch of environments that even getting python into can be tricky. initramfs before your drives are mounted, for example...
A lot of the discussion on here is mind-blowing, so many pushing to throw out perfectly good tech because it doesn't fit their (limited) worldview.
A lot of the discussion on here is mind-blowing, so many pushing to throw out perfectly good tech because it doesn't fit their (limited) worldview.
Don't know if I would say that bash has a low barrier of entry. I find doing stuff in Python far more intuitive than in bash, it certainly has its quirks. I agree on the rest though.
You make an awful lot of assumptions about population of the site where you can be practically anonymous to other users.
The Python hate here is ridiculous. This is Google's [bash] style guide -- Google has chosen to use Python as their general purpose development language so "setting up environments" and pip, virtualenv, pyenv, pipenv all of that stuff is resolved through their standard processes.
So, if you're not Google, then replace "Python" with "our organization's general purpose language", whether that's Ruby, C#, PHP or what-have-you.
The statement is essentially, don't try to do everything in Bash, it's not well-suited for everything and whoever comes along next that has to deal with your 1000 lines bash script is going to want to kill you.
So, if you're not Google, then replace "Python" with "our organization's general purpose language", whether that's Ruby, C#, PHP or what-have-you.
The statement is essentially, don't try to do everything in Bash, it's not well-suited for everything and whoever comes along next that has to deal with your 1000 lines bash script is going to want to kill you.
Exhibit A: When I joined Google Fiber, one of my first projects was converting a (POSIX) shell script that had grown to 1200 lines to Python. It became something that anyone could modify, including interns, rather than something that required at least a code review from our L7 tech lead, whose time was better spent elsewhere (he was the author and our only competent shell programmer).
That is mind blowing for me in a couple ways.
* You have (at least) seven tiered support structures.
* You have only one 'competent' shell programmer in your 'reachable' project scope.
No wonder these types of language policies are in place.
No wonder these types of language policies are in place.
It's been decades since "writing a 1000 line script" was best done in shell, if there ever was a time. Someone with that much experience ought to be a very senior engineer.
I have shell libraries that generate code via here documents and other templating that are > 500 lines. That is about as far as I reasonably go. Note that these are function (and scope specific variables (with hopefully unique namespaces)) only code bases with copious comments and references to underlying tools.
The script in question was an interface for controlling WiFi (hostapd, wpa_supplicant) in a variety of ways, abstracting across different chipsets.
I don't know what you mean by seven tiered support structures. My tech lead was a Senior Staff Software Engineer, L7, level seven, etc. Just meant that he was a relatively senior guy (not that he didn't get his hands dirty, there were just better places for him to do so). There are lots of levels on Google's SWE job ladder.
And maintaining a 1200 line shell script that can brick hardware in people's homes and disable their WiFi (or internet access altogether) is something to be careful with. There were five or so SWEs doing wifi-related things for Fiber (including me and one other person new to the team), and yes, he was the only one of us who knew shell well enough to reliably catch mistakes. He was a few years older than the rest of us, maybe that's why.
I have a BS in CS from Georgetown and a MS in CS from Stanford. I don't think I ever had a class at either school that required me to write a single conditional statement or loop in shell. I probably never would have picked it up at Google either if I hadn't joined an embedded team.
And maintaining a 1200 line shell script that can brick hardware in people's homes and disable their WiFi (or internet access altogether) is something to be careful with. There were five or so SWEs doing wifi-related things for Fiber (including me and one other person new to the team), and yes, he was the only one of us who knew shell well enough to reliably catch mistakes. He was a few years older than the rest of us, maybe that's why.
I have a BS in CS from Georgetown and a MS in CS from Stanford. I don't think I ever had a class at either school that required me to write a single conditional statement or loop in shell. I probably never would have picked it up at Google either if I hadn't joined an embedded team.
You have deep technical ladders and they may make sense in a place like Google. Who knows. I've seen a couple young men I've mentored go to work at Google and don't hear anything negative from them.
Any code anywhere is worth being careful with and a shell script can be more dangerous than your average glue language (tcl,python,perl) script. I don't see why you need 1200 lines of shell script unless most of it is error handling and safe execution wrapping.
Funny, I've worked at Georgetown and my grandfather went to Stanford. I've written (over 20+ years) good and bad shell scripts. They get better the older I get and they also seem to be (mostly) less than 50 lines. I don't understand why people say these things about edu unless it's intended to awe the easily impressed. Smart is smart and well rounded is well rounded anywhere.
Any code anywhere is worth being careful with and a shell script can be more dangerous than your average glue language (tcl,python,perl) script. I don't see why you need 1200 lines of shell script unless most of it is error handling and safe execution wrapping.
Funny, I've worked at Georgetown and my grandfather went to Stanford. I've written (over 20+ years) good and bad shell scripts. They get better the older I get and they also seem to be (mostly) less than 50 lines. I don't understand why people say these things about edu unless it's intended to awe the easily impressed. Smart is smart and well rounded is well rounded anywhere.
Sorry, I wasn't trying to show off. All I meant was that I had a pretty thorough mainstream CS education. If I didn't learn shell on the way then it's safe to assume most CS students these days don't either.
> due to the fact that it’s [sic] creators are very much of the old 1970s and 1980s Unix mindset
Firstly, Bash is relied upon to conform to a standard; it's not an exploratory project which expresses the maintainers' views about what a shell should look like.
Secondly, Stallman didn't start Bash because he loved Unix. It was a pragmatic choice. Stallman really wanted a free platform based on Lisp. He went with cloning the Unix user space because that was becoming a dominant OS. It was proprietary software, and he wanted to replace that directly with workalike free software, rather than to provide incompatible free software and then have to evangelize not only freedom, but also a different platform at the same time.
There were some technical reasons. Stallman rejected the idea of copying the Lisp Machine approach:
https://www.gnu.org/gnu/rms-lisp.en.html
At first, I thought of making a Lisp-based system, but I realized that wouldn't be a good idea technically. To have something like the Lisp machine system, you needed special purpose microcode. That's what made it possible to run programs as fast as other computers would run their programs and still get the benefit of typechecking. Without that, you would be reduced to something like the Lisp compilers for other machines. The programs would be faster, but unstable. Now that's okay if you're running one program on a timesharing system — if one program crashes, that's not a disaster, that's something your program occasionally does. But that didn't make it good for writing the operating system in, so I rejected the idea of making a system like the Lisp machine.
I decided instead to make a Unix-like operating system that would have Lisp implementations to run as user programs. The kernel wouldn't be written in Lisp, but we'd have Lisp.
https://www.gnu.org/gnu/thegnuproject.en.html
For example, we developed the GNU C library because a Unix-like system needs a C library, BASH because a Unix-like system needs a shell, and GNU tar because a Unix-like system needs a tar program. The same is true for my own programs—the GNU C compiler, GNU Emacs, GDB and GNU Make.
Firstly, Bash is relied upon to conform to a standard; it's not an exploratory project which expresses the maintainers' views about what a shell should look like.
Secondly, Stallman didn't start Bash because he loved Unix. It was a pragmatic choice. Stallman really wanted a free platform based on Lisp. He went with cloning the Unix user space because that was becoming a dominant OS. It was proprietary software, and he wanted to replace that directly with workalike free software, rather than to provide incompatible free software and then have to evangelize not only freedom, but also a different platform at the same time.
There were some technical reasons. Stallman rejected the idea of copying the Lisp Machine approach:
https://www.gnu.org/gnu/rms-lisp.en.html
At first, I thought of making a Lisp-based system, but I realized that wouldn't be a good idea technically. To have something like the Lisp machine system, you needed special purpose microcode. That's what made it possible to run programs as fast as other computers would run their programs and still get the benefit of typechecking. Without that, you would be reduced to something like the Lisp compilers for other machines. The programs would be faster, but unstable. Now that's okay if you're running one program on a timesharing system — if one program crashes, that's not a disaster, that's something your program occasionally does. But that didn't make it good for writing the operating system in, so I rejected the idea of making a system like the Lisp machine.
I decided instead to make a Unix-like operating system that would have Lisp implementations to run as user programs. The kernel wouldn't be written in Lisp, but we'd have Lisp.
https://www.gnu.org/gnu/thegnuproject.en.html
For example, we developed the GNU C library because a Unix-like system needs a C library, BASH because a Unix-like system needs a shell, and GNU tar because a Unix-like system needs a tar program. The same is true for my own programs—the GNU C compiler, GNU Emacs, GDB and GNU Make.
Yeah, when you dig down into the specifics I was not doing the history justice. My broader point was that it was definitely designed in a different era, with a different mindset, and has still managed to persist to this day, without that mindset shifting very much (usually for good reasons, or backward compatibility, which may or not be a good reason depending on specific circumstances)
I often wonder if shell scripts would get such a bad wrap if all the backwards compatible code was strip out and only the latest standards were put in. I think thats where it gets people so very confused & it definitely feels awkward.
I think its actually a perfectly fine language, syntactically speaking.
I often wonder if shell scripts would get such a bad wrap if all the backwards compatible code was strip out and only the latest standards were put in. I think thats where it gets people so very confused & it definitely feels awkward.
I think its actually a perfectly fine language, syntactically speaking.
> no single language is objectively superior to another
I would say some languages are objectively superior to other for specific tasks. The problem is that very often the task to solve doesn't exactly fall within the realm of any given programming language and one has to make compromise.
As for bash, I use it when I feel it's the best option, but I also think it's the worst programming language that I have to use. I wish there were better options. Are there any modern promising replacements for bash?
I would say some languages are objectively superior to other for specific tasks. The problem is that very often the task to solve doesn't exactly fall within the realm of any given programming language and one has to make compromise.
As for bash, I use it when I feel it's the best option, but I also think it's the worst programming language that I have to use. I wish there were better options. Are there any modern promising replacements for bash?
Yeah, trivially, a language like Brainfuck is clearly not a good choice for anything professional or maintainable; therefore, there must be some selection criteria that makes some languages objectively better than other languages.
Whether PHP is a "bad" language or C# is better than Java is harder (and generally a waste of time -- pick something that works 95% of your use-case and figure out the other 5%), but refining the phrase to "no commercially used language is objectively superior to another" starts to make the statement useless to the point of irrelevance.
Whether PHP is a "bad" language or C# is better than Java is harder (and generally a waste of time -- pick something that works 95% of your use-case and figure out the other 5%), but refining the phrase to "no commercially used language is objectively superior to another" starts to make the statement useless to the point of irrelevance.
Try fish shell. Not too different but not as bad.
https://fishshell.com
And an effort to get rid of bash as Oil shell. (They really need a better name for Googleability.)
https://www.oilshell.org
https://fishshell.com
And an effort to get rid of bash as Oil shell. (They really need a better name for Googleability.)
https://www.oilshell.org
Sure, I agree, for specific tasks. Just like a hammer is a better tool than a screw driver for putting nails in a block of wood.
Thats my point. They're tools not lifestyle choices. Learn your tools, know your tools, don't be afraid of other people's tools.
They all, usually, have some sort of value.
I might as well note this too:
I am speaking in broad strokes, you're going to be able to pinhole this as much as you want because of that. I'm simply suggesting that if your immediate reaction to how another organization uses these tools to replace A with B at a certain level is "this is clearly inferior" instead of "well, what can I learn from this" well, there's the problem.
I used to think programmers/computer scientists where open minded people. Then I realized as a whole it tends to be we're only open minded if its within the domain we chose to be most active in
Still far and away better than any other folks I've met, of course, but this is really a community culture problem, which is why I find it so counter productive.
Alternatives to bash? Many do exist. zsh is bash compatible but does implement its own tools and subsets.
If you're looking for something that removes the bash syntax there is powershell, which does run on Linux/macOS now as well:
https://github.com/PowerShell/PowerShell
Then of course, there is oil shell:
https://github.com/oilshell/oil
which is more like a subset of bash (and somewhat close to the idea of stripping away all the legacy stuff)
Then there is xonsh: lets you use python as a basis for most everything
http://xon.sh/
The stalwart fish:
https://fishshell.com/
fish shell has alot of niceties and a more object oriented syntax (kind of)
and for the daring there is ergonomica: https://ergonomica.readthedocs.io/en/latest/
which has a lisp like syntax
as does https://github.com/dundalek/closh which uses clojure and its syntax
and i'm sure there are more.
Thats my point. They're tools not lifestyle choices. Learn your tools, know your tools, don't be afraid of other people's tools.
They all, usually, have some sort of value.
I might as well note this too:
I am speaking in broad strokes, you're going to be able to pinhole this as much as you want because of that. I'm simply suggesting that if your immediate reaction to how another organization uses these tools to replace A with B at a certain level is "this is clearly inferior" instead of "well, what can I learn from this" well, there's the problem.
I used to think programmers/computer scientists where open minded people. Then I realized as a whole it tends to be we're only open minded if its within the domain we chose to be most active in
Still far and away better than any other folks I've met, of course, but this is really a community culture problem, which is why I find it so counter productive.
Alternatives to bash? Many do exist. zsh is bash compatible but does implement its own tools and subsets.
If you're looking for something that removes the bash syntax there is powershell, which does run on Linux/macOS now as well:
https://github.com/PowerShell/PowerShell
Then of course, there is oil shell:
https://github.com/oilshell/oil
which is more like a subset of bash (and somewhat close to the idea of stripping away all the legacy stuff)
Then there is xonsh: lets you use python as a basis for most everything
http://xon.sh/
The stalwart fish:
https://fishshell.com/
fish shell has alot of niceties and a more object oriented syntax (kind of)
and for the daring there is ergonomica: https://ergonomica.readthedocs.io/en/latest/
which has a lisp like syntax
as does https://github.com/dundalek/closh which uses clojure and its syntax
and i'm sure there are more.
I actually like python, the language. I think it balances the needs of a dynamic language well and I like that it is, at least, strongly typed. My hate is entirely directed at python, the environment and tooling around the language. Other languages have figured this stuff out and python just hasn't. And the horribly managed 2.x -> 3.x migration made things even worse.
I get that it is enjoyable to program in and I think it makes a fine choice for instances where a whole team is bought into the decision to use it. But when it's used for things that are distributed to others or force other people who aren't experts in python to wade into the ugliness of the python ecosystem, that's what makes me hate it. I've had so many python-related problems with the aws command-line client alone, let alone all the other python I've been forced to run, that I never want to touch another product written in python again.
To me, it's just an instance of developers prioritizing themselves over their users (hello electron devs).
I get that it is enjoyable to program in and I think it makes a fine choice for instances where a whole team is bought into the decision to use it. But when it's used for things that are distributed to others or force other people who aren't experts in python to wade into the ugliness of the python ecosystem, that's what makes me hate it. I've had so many python-related problems with the aws command-line client alone, let alone all the other python I've been forced to run, that I never want to touch another product written in python again.
To me, it's just an instance of developers prioritizing themselves over their users (hello electron devs).
There's a bit of bias in that you're most likely to really notice something's in Python when it breaks... And the AWS client is terrible itself. And then you have a lot of stuff like Deluge where you'd never notice it's in Python until you checked under the hood.
Because python was the cool thing a few years ago, and it managed to get traction in a lot of places because it was pretty easy and included a lot of tools. It's also taught as an intro language now. So now you have to deal with people who are poopooing it because it doesn't have enough nerd cred like some like some other dynlang.
Shell scripting has some other issues though. While most Bourne derived shells look similar, they have different extensions and capabilities. So only a subset of scripts will work everywhere.
Shell scripting has some other issues though. While most Bourne derived shells look similar, they have different extensions and capabilities. So only a subset of scripts will work everywhere.
I like bash, but I think I understand the hate -- it's the slippery slope of "gluing real applications" to "I can write this all in bash!". Just because you can, doesn't mean you should.
I enjoy the section of the Advanced Bash-Scripting Guide which explains how Bash has sparse arrays which "are useful in spreadsheet-processing software".[0] Because someone out there is apparently writing spreadsheet-processing software in Bash?
[0] http://tldp.org/LDP/abs/html/arrays.html
[0] http://tldp.org/LDP/abs/html/arrays.html
If the application can be reduced to some form of simple piping of existing programs...
Just because all languages have flaws does not mean one language cannot be superior to another. Don't respond to one fallacy by employing another one.
> I think this due to the fact that it’s creators are very much of the old 1970s and 1980s Unix mindset
The Bourne Again shell was created in 1989.
That said, the 1980s Unix mindset was not what you clearly think it to be. It was full of Sun workstations, NextStep, Orthodox File Manager clones, Terminals with graphics, GNU Screen, Threaded ReadNews, ...
The Bourne Again shell was created in 1989.
That said, the 1980s Unix mindset was not what you clearly think it to be. It was full of Sun workstations, NextStep, Orthodox File Manager clones, Terminals with graphics, GNU Screen, Threaded ReadNews, ...
Yes indeed, it sure was. I realize that. Don't forgot those awesome Amiga workstations
https://en.wikipedia.org/wiki/Amiga#/media/File:Amiga500_sys...
Way ahead of their time, those guys.
I don't think my point glosses over this at all though, in that the mindset back then derived from the pioneering work done at Xerox Labs & on original Unix. I know bash itself was not created as the first shell implementation, however its syntax is clearly heavily derived from the original Bourne shell and ascribed to its overall philosophy: do one thing, and do one thing well. Besides, at that time, you either dropped into Smalltalk or C if you needed to do more than text processing, which is what bash et. al. are still insanely good for.
https://en.wikipedia.org/wiki/Amiga#/media/File:Amiga500_sys...
Way ahead of their time, those guys.
I don't think my point glosses over this at all though, in that the mindset back then derived from the pioneering work done at Xerox Labs & on original Unix. I know bash itself was not created as the first shell implementation, however its syntax is clearly heavily derived from the original Bourne shell and ascribed to its overall philosophy: do one thing, and do one thing well. Besides, at that time, you either dropped into Smalltalk or C if you needed to do more than text processing, which is what bash et. al. are still insanely good for.
I absolutely hate coding in bash or looking at bash or suddenly being in Windows and being up a tree because bash isn't supported well (it is now if you install WSL). I only use bash to set environment variables and maybe string together 2 or 3 build commands. If I need so much as an if statement, I'm going to switch to a real programming language.
This document agrees with you.
- If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python.
- If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date.
- If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python.
- If you are writing a script that is more than 100 lines long, you should probably be writing it in Python instead. Bear in mind that scripts grow. Rewrite your script in another language early to avoid a time-consuming rewrite at a later date.
Exchanging Bash for Python is a folly.
Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.)
Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".
Yet Python provides no way to sanity check itself. Mypy is wonderful, but doesn't really help with scripts, where the enemy is raised exceptions, not simple type errors.
And Python lacks efficient, quick and dirty process control. Sure, there are nice packages for handling subprocesses, but that's really a lot more batteries-included in Bash.
Do I like Bash? No, but debugging a bash script is -x, even the most vile many thousand lines of Bash will eventually stop somewhere with a clean-ish error or finish running.
I have no great recommendation for alternative (maybe Ammonite, maybe scripting in Rust), but I'm interested in what people think about this.
Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.)
Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".
Yet Python provides no way to sanity check itself. Mypy is wonderful, but doesn't really help with scripts, where the enemy is raised exceptions, not simple type errors.
And Python lacks efficient, quick and dirty process control. Sure, there are nice packages for handling subprocesses, but that's really a lot more batteries-included in Bash.
Do I like Bash? No, but debugging a bash script is -x, even the most vile many thousand lines of Bash will eventually stop somewhere with a clean-ish error or finish running.
I have no great recommendation for alternative (maybe Ammonite, maybe scripting in Rust), but I'm interested in what people think about this.
>Python seems to handle complexity better, but only on the surface. (Basically Python has nice data structures, great string formatting, and .. that's it.)
It's a complete scripting language, with tons of features, from a comprehensive standard library with something for most needs, to a packaging system, isolated environments, async io, and more. And lots of expressivity in structuring your program (including optional types).
So, not sure what the above "and that's it" means. That it's not Haskell? Yes, it's not.
>Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".
If you're doing dev-ops with such scripts, you already need to handle the environment in general.
It's a complete scripting language, with tons of features, from a comprehensive standard library with something for most needs, to a packaging system, isolated environments, async io, and more. And lots of expressivity in structuring your program (including optional types).
So, not sure what the above "and that's it" means. That it's not Haskell? Yes, it's not.
>Setting up the Python environment requires something, because Python comes in many flavors (2 vs 3, system installed, user installed, /usr/local) and there are a bunch of factors (pip, pyenv, pipenv, PYTHONPATH, current directory with modules, requirement for python-dev headers, and GCC to compile native parts of packages, and then of course maybe extra shared libraries too) that can affect the "script".
If you're doing dev-ops with such scripts, you already need to handle the environment in general.
> not sure what the above "and that's it" means. That it's not Haskell? Yes, it's not.
That scripting in Python is pretty bad. It makes thing harder and at the same time you still have to think a lot about what can go wrong, because there's no enforced error handling.
That scripting in Python is pretty bad. It makes thing harder and at the same time you still have to think a lot about what can go wrong, because there's no enforced error handling.
Yeah, even Java doesn't force you to handle errors. Please, calm down. "Scripting" in a compiled language is not practical. People use the word "script" for a reason - it's because they don't want to deal with a language that forces them to think about everything that could go wrong. For some tasks that's perfectly sensible. For other tasks, there is Rust, Haskell and many others.
Checked exceptions are nice. You can use them if you want to make them part of your API surface.
> "Scripting" in a compiled language is not practical.
The premise is that scripts grow, and you should switch to Python. Which might be okay for Google, but seems to be only a very small step up from Bash, if any at all. (I've outlined the reasons, why I think it's not worth it.)
I do a lot of scripting, in Bash, and when that stars to grow on me, it's time to think about the problem for real. And usually a blind rewrite in Python is not the way to go.
> For some tasks that's perfectly sensible.
Those are the one-off scripts, no? Anything that should run with some regularity, anything that should be reproducible, that handles valuable data, is not something that should be cobbled together in bash/python and thrown into /etc/cron.d . Or apparently that's how Google rolls.
Yeah, well, maybe not everyone is an SRE, so the use case here seems to be build scripts and other not-so-critical scripts, that still have to be maintained, where knowledge of Bash/POSIX-sh is not really assumed, it should just work for anyone who checks out the repo.
> "Scripting" in a compiled language is not practical.
The premise is that scripts grow, and you should switch to Python. Which might be okay for Google, but seems to be only a very small step up from Bash, if any at all. (I've outlined the reasons, why I think it's not worth it.)
I do a lot of scripting, in Bash, and when that stars to grow on me, it's time to think about the problem for real. And usually a blind rewrite in Python is not the way to go.
> For some tasks that's perfectly sensible.
Those are the one-off scripts, no? Anything that should run with some regularity, anything that should be reproducible, that handles valuable data, is not something that should be cobbled together in bash/python and thrown into /etc/cron.d . Or apparently that's how Google rolls.
Yeah, well, maybe not everyone is an SRE, so the use case here seems to be build scripts and other not-so-critical scripts, that still have to be maintained, where knowledge of Bash/POSIX-sh is not really assumed, it should just work for anyone who checks out the repo.
Some actual examples would be nice; this just sounds like nirvana fallacy.
Dunno, why do "scripting", it's a rather clean oop language. For scripting use bash or perl maybe.
"That it's not Haskell" -- gold
It's really not about data structure or anything like that. The big problem with any large shell script is that it's utterly difficult to do proper error handling. Usually the best you can do is check return values to detect an error and 'exit' immediately, hoping that your "trap" works correctly to clean up behind you.
>Python lacks efficient, quick and dirty process control.
Yeah, quick and dirty, that's kind of the problem really. It's great for 20-line scripts but it turns into a huge mess for larger projects unless you enforce super strict guidelines and everybody has their doctorate in bash.
Python, perl and friends make it harder to spawn 3rd party programs and pipe them together for instance, but it also makes it massively easier to handle errors and give decent diagnostics to the user.
>Python lacks efficient, quick and dirty process control.
Yeah, quick and dirty, that's kind of the problem really. It's great for 20-line scripts but it turns into a huge mess for larger projects unless you enforce super strict guidelines and everybody has their doctorate in bash.
Python, perl and friends make it harder to spawn 3rd party programs and pipe them together for instance, but it also makes it massively easier to handle errors and give decent diagnostics to the user.
> trap
I don't have more faith in Python's except than in Bash's trap.
If you use 3rd party code or a subprocess, then both are pretty weak compared to something kernel enforced.
> makes it massively easier to handle errors and give decent diagnostics to the user.
I don't really find that. You have to fight for input/output for subprocesses in Python.
Sure, logging is easier in Python/perl/PHP/whatever, than in Bash, but error handling is about the same. You have to do everything manually, and there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something.
I don't have more faith in Python's except than in Bash's trap.
If you use 3rd party code or a subprocess, then both are pretty weak compared to something kernel enforced.
> makes it massively easier to handle errors and give decent diagnostics to the user.
I don't really find that. You have to fight for input/output for subprocesses in Python.
Sure, logging is easier in Python/perl/PHP/whatever, than in Bash, but error handling is about the same. You have to do everything manually, and there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something.
Let's be clear, I'm not a huge fan of python or exception-based error handling either but I'll take it any day over... whatever you want to call shell ad-hoc error handling or lack thereof.
Trap is not like an exception, it's more like "atexit" or a signal handler. You can't really nest them, you can't really recover from them. So I definitely disagree that error handling is about the same, python's a lot better.
>there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something.
Well that's the whole static vs. dynamic debate but that's a different story. Use C# or Rust or whatever you feel like if you really can't stand python or scheme. Maybe there's a niche to be filled for statically typed scripting languages, there are no popular ones that I'm aware of.
Still, that's besides the point, while I personally prefer static typing like you there's really no contest between dynamic typing and "everything is a string and you can only concatenate them" typing.
Trap is not like an exception, it's more like "atexit" or a signal handler. You can't really nest them, you can't really recover from them. So I definitely disagree that error handling is about the same, python's a lot better.
>there's no compiler, nor type (or other magical) system, that'd warn you if you have missed something.
Well that's the whole static vs. dynamic debate but that's a different story. Use C# or Rust or whatever you feel like if you really can't stand python or scheme. Maybe there's a niche to be filled for statically typed scripting languages, there are no popular ones that I'm aware of.
Still, that's besides the point, while I personally prefer static typing like you there's really no contest between dynamic typing and "everything is a string and you can only concatenate them" typing.
> You have to do everything manually
You are aware that error handling is done via exceptions python?
You are aware that error handling is done via exceptions python?
You are aware that there's no compiler, and that you have to write the try-except-finally?
I'm talking about how weak guarantees Python gives compared to Bash about the correctness of the program. It's easy to get None-s, it's too easy to forget to handle file not found or lack of permission.
I'm talking about how weak guarantees Python gives compared to Bash about the correctness of the program. It's easy to get None-s, it's too easy to forget to handle file not found or lack of permission.
Maybe you should start raising exceptions instead of returning None.
I've almost never wanted to have my program do anything but crash upon file not found or permission error.
I've almost never wanted to have my program do anything but crash upon file not found or permission error.
For most scripts, handling file not found isn't gonna happen anyway. Often the result is just printing to standard out "hey I couldn't find file X" then bailing. An uncaught exception works well enough for this in many cases, such as build scripts.
Same goes for permissions issues and a whole slew of similar scenarios. For many use cases, the lack of forced error checking is a feature.
Concrete example: build scripts. As long as the cause of the build failure is clear from the uncaught exception, there's no reason to handle it. I don't need a friendly UI to avoid scaring off non technical users, I just need something that handles the happy path well and crashes with a meaningful stack trace when something unexpected happens.
Same goes for permissions issues and a whole slew of similar scenarios. For many use cases, the lack of forced error checking is a feature.
Concrete example: build scripts. As long as the cause of the build failure is clear from the uncaught exception, there's no reason to handle it. I don't need a friendly UI to avoid scaring off non technical users, I just need something that handles the happy path well and crashes with a meaningful stack trace when something unexpected happens.
I always add ‘set -ex’ to the top of my bash scripts to exit on any non zero top level return code and to print out all statements being run including expanded variables or inner statements.
That's because Python is not a scripting language. Dynamic is not scripting.
They have a standard setup everywhere, with the same Python version, and the same libs available.
So basically, for them, Python requires no fiddling and works as is.
Same for the shell. They say "use [[]]" because they know that compat is not a problem: it's the same bash everywhere.
Google has millions of servers, billions of lines of code and surely of lot of experience with deployment, managing complexity and handling teams.
I'm kinda trusting them on this choice.
So basically, for them, Python requires no fiddling and works as is.
Same for the shell. They say "use [[]]" because they know that compat is not a problem: it's the same bash everywhere.
Google has millions of servers, billions of lines of code and surely of lot of experience with deployment, managing complexity and handling teams.
I'm kinda trusting them on this choice.
[deleted]
I would be remiss if I didn't mention shellcheck, which will also warn you against doing stupid things.
Very useful when debugging.
https://github.com/koalaman/shellcheck
Very useful when debugging.
https://github.com/koalaman/shellcheck
Also fantastic for asynchronous linting, if you're into that.
You’ve hit most of it. The desire is a lowest common denominator. Python, Ruby, etc would be great except that in some of the places there arent current versions and beyond the most mundane you end up wanting 3rd party packages, which complicates your packaging or removes that lowest common denominator property.
What it begs is for a new tool. A single file, statically linked “super shell” that lets you do structured scripting and has builtins for a bunch of common things (reading in addresses, daemonizing processes, reading the process table, etc. and that tool needs to be build able on nearly everything. Now I like rust and the attention it is getting and the interest in safety and correctness. I can’t think of many places where I’d reach for bash but be happier writing rust, a lot of it is 15minute disposable programming jobs. At a glance, I think I’d trade 5minutes of set -e to find some error I skipped for 45 minutes of trying to rejigger things to properly borrow some shared value.
What it begs is for a new tool. A single file, statically linked “super shell” that lets you do structured scripting and has builtins for a bunch of common things (reading in addresses, daemonizing processes, reading the process table, etc. and that tool needs to be build able on nearly everything. Now I like rust and the attention it is getting and the interest in safety and correctness. I can’t think of many places where I’d reach for bash but be happier writing rust, a lot of it is 15minute disposable programming jobs. At a glance, I think I’d trade 5minutes of set -e to find some error I skipped for 45 minutes of trying to rejigger things to properly borrow some shared value.
Tcl meets all of these requirements. I have a Linux distribution whose PID 1 is a Tcl shell and I can setup everything from there without spawning any new processes.
Yeah, but Tcl is old and quite crufty. The docs are a mess, everything is a string, a lot of concepts it uses are not very familiar and/or mainstream (upvar?).
A modern take on Tcl is what we need, in my opinion.
Surely we should be able to build something nicer 30 years after the first release of Tcl.
A modern take on Tcl is what we need, in my opinion.
Surely we should be able to build something nicer 30 years after the first release of Tcl.
I agree that Tcl isn't mainstream and thus a lot of concepts used by Tcl are not mainstream, but other than that it's got a lot going for it.
Tcl's string representation for objects is just a serialization/deserialization mechanism, which seems to be pretty popular in other languages as well.
Additionally all of the cool things in Tcl such as coroutines, threads (as an additional, but included, package), great documentation delivered as man pages, virtual filesystem access, virtual time, safe interpreters for running untrusted code, a stable ABI (stubs) so that compiled extensions can be used for decades, a small memory as well as disk footprint, extremely cross-platform, easy to embed into larger programs as an extension language, easy to compile a large static program with Tcl as the "main" entrypoint, native BigNum support, many thousands of great packages, ....
What would a modern take on Tcl improve on Tcl that Tcl couldn't just build easier ?
Tcl's string representation for objects is just a serialization/deserialization mechanism, which seems to be pretty popular in other languages as well.
Additionally all of the cool things in Tcl such as coroutines, threads (as an additional, but included, package), great documentation delivered as man pages, virtual filesystem access, virtual time, safe interpreters for running untrusted code, a stable ABI (stubs) so that compiled extensions can be used for decades, a small memory as well as disk footprint, extremely cross-platform, easy to embed into larger programs as an extension language, easy to compile a large static program with Tcl as the "main" entrypoint, native BigNum support, many thousands of great packages, ....
What would a modern take on Tcl improve on Tcl that Tcl couldn't just build easier ?
The modern take on Tcl arrived almost 20 years ago, when "everything is a string" turned into "everything is a typed object with a string representation". The docs are clear and complete, and the concepts it uses make it a powerful and elegant tool.
The docs are clear and complete?!?
Has something changed suddenly in the last 5 years? Yes, the docs had a decent API reference but everything slightly higher up than "what parameters this command has" was on that wiki with comments from 2004 marked as out of date by out of date comments from 2007...
Has something changed suddenly in the last 5 years? Yes, the docs had a decent API reference but everything slightly higher up than "what parameters this command has" was on that wiki with comments from 2004 marked as out of date by out of date comments from 2007...
Hard to see what your standard is. In what sense do the official docs (https://www.tcl.tk/man/tcl8.6/TclCmd/contents.htm) fall short in your opinion?
>I have a Linux distribution whose PID 1 is a Tcl shell
Interested! Available?
Interested! Available?
Not really, it's really used to setup the environment for a network boot, it supports VLANs and network configuration and loading modules and stuff (even from Tcl virtual filesystems), it's basically just Tcl+TUAPI[1]
[1] https://chiselapp.com/user/rkeene/repository/tuapi/doc/trunk...
[1] https://chiselapp.com/user/rkeene/repository/tuapi/doc/trunk...
I thought of Rust because of the strict typing. If something TypeScript-like could compile to Bash, that'd be amazing. (Maybe there's something like that already.)
But now, to think of it, just using Node + TS (+ yarn) sounds better than Python.
But now, to think of it, just using Node + TS (+ yarn) sounds better than Python.
Its possible to write 2-3 compatible Python code. Furthermore, for basic "scripting" its reasonable to assume no external dependencies, so no need to worry about packaging, pip, pipenv, etc.
The major disadvantage of any "comfortable" language aside from Python is that they aren't installed by default on nearly every distribution of every OS.
This is why, for example, Ansible only requires Python 2.6 on managed nodes, allowing literally zero installation on almost any host you want to manage.
If you already have a good mechanism for packaging and distributing software to all your machines (which I think is a reasonable expectation for Google) then go ahead and use whatever language you're well equipped for. But know you'll be sacrificing the easy and wide distribution that using Bash or Python brings you.
The major disadvantage of any "comfortable" language aside from Python is that they aren't installed by default on nearly every distribution of every OS.
This is why, for example, Ansible only requires Python 2.6 on managed nodes, allowing literally zero installation on almost any host you want to manage.
If you already have a good mechanism for packaging and distributing software to all your machines (which I think is a reasonable expectation for Google) then go ahead and use whatever language you're well equipped for. But know you'll be sacrificing the easy and wide distribution that using Bash or Python brings you.
> This is why, for example, Ansible only requires Python 2.6 on managed nodes, allowing literally zero installation on almost any host you want to manage.
Yes, Ansible transfers everything. It's ugly and slow :(
I pretty much gave up on [vanilla] OpenStack because their idea of DevOps is custom python script generated Ansible plays.
Yes, Ansible transfers everything. It's ugly and slow :(
I pretty much gave up on [vanilla] OpenStack because their idea of DevOps is custom python script generated Ansible plays.
Slow? Maybe. Ugly? I find it rather elegant.
Agree. If you want to do shell work in something closer to the shell and have a real programming language, Perl is the winner.
Re. a recommended alternative to quick and dirty bash scripts, do you think golang would do? I've barely dabbled with the language, but its default packaging (a single, static binary) and build system (`go build x`) seem well suited to rapid setup, deployment and testing, which is presumably what you'd want in this scenario.
Many people use go for such things. I simply don't like go, it's too crude (too C), and the packaging system is. Well, it's not Cargo.
But probably for deploy scripts, go with a static binary (hosted on, let's say your GitLab instance - using the GitLab HTTP API with a token to wget/curl it) is nigh unbeatable in end-to-end development and deployment time.
But probably for deploy scripts, go with a static binary (hosted on, let's say your GitLab instance - using the GitLab HTTP API with a token to wget/curl it) is nigh unbeatable in end-to-end development and deployment time.
Aah, makes. I'm not a huge fan of it for similar reasons, but the appeal of a language that's quick, dirty, and relatively acceptable by coworkers is strong.
I had some success with Node.js (though it was in a Node-based project to begin with). I haven't tried piping commands, but to run a bunch of commands and then have all the logic in JavaScript it's quite convenient.
Here's an example that builds a React Native app, while updating version numbers and Git tags:
https://github.com/laurent22/joplin/blob/master/Tools/releas...
Here's an example that builds a React Native app, while updating version numbers and Git tags:
https://github.com/laurent22/joplin/blob/master/Tools/releas...
My main problem with scripting in Node vs python is the event loop gets in your way when just trying to write a quick and dirty sequence of steps. Sure you can use async await but now you are transpiling your script.. seems easier to just bang it out in python.
I love Node but I use it for backend where the event-loop model is useful.
I love Node but I use it for backend where the event-loop model is useful.
Personally I've found myself writing lots of simple command line utilities in Go. YMMV but the standard library handles file IO, network stuff, running external commands and a bunch of standard encodings really well. It also has static typing, and awesome concurrency features.
It's obviously not perfect, but it has replaced lots of Bash scripts for me.
It's obviously not perfect, but it has replaced lots of Bash scripts for me.
Based on the rest of your comment (which I agree with), I think you meant "Exchanging Bash for Python is a folly."
I'm one of those people who finds perverse pleasure in writing portable Bourne shell, and doing so correctly. I know I spend far more brain cycles on it than I should; once it gets past a certain complexity limit, I should just switch to Python. But I always say "It's just a little bit more code, there must be a good way to do this," and there generally is, but only after passing over several bad ways to do it first.
Oh, and of course in doing so, I always try to minimize the amount of calling out to other processes. Why call awk when you could use `while read` instead?
You only get one array to work with in portable Bourne shell, $@. You can parse lines using read, and strip prefixes and suffixes using parameter expansion (`${parameter%pattern}`). It's a restrictive little language, with a lot of quirks and edge cases, but it always makes a nice little challenge to figure out how you can use it safely.
One of the constraints I impose is that spaces in pathnames should always be handled correctly; the biggest form of trouble you can easily get in is to forget to quote something or to parse something on space delimiters when it can include spaces. I've seen someone write a script that accidentally turned into an `rm -rf /path/to/important/data` because of poor handling of spaces.
I generally do not attempt to handle newlines in pathnames. While it's possible to do so in some cases with null delimiters, I have never seen a case of newlines in pathnames in the wild. Of course, this means that I have to make sure that these scripts are never exposed to anything that could contain malicious filenames.
Also, I move away from portable shell if it will really make the code too convoluted or fragile. For instance, if I need to parse long options (--foo), I use GNU getopt rather than the built-in getopts (with appropriate test to make sure the getopt I'm using is GNU, so it won't break silently on a system without GNU getopt), or move to a real language.
Anyhow, while I know it's counterproductive, the puzzle-solving part of me really likes trying to solve the problem of implementing something in portable Bourne shell.
Oh, and of course in doing so, I always try to minimize the amount of calling out to other processes. Why call awk when you could use `while read` instead?
You only get one array to work with in portable Bourne shell, $@. You can parse lines using read, and strip prefixes and suffixes using parameter expansion (`${parameter%pattern}`). It's a restrictive little language, with a lot of quirks and edge cases, but it always makes a nice little challenge to figure out how you can use it safely.
One of the constraints I impose is that spaces in pathnames should always be handled correctly; the biggest form of trouble you can easily get in is to forget to quote something or to parse something on space delimiters when it can include spaces. I've seen someone write a script that accidentally turned into an `rm -rf /path/to/important/data` because of poor handling of spaces.
I generally do not attempt to handle newlines in pathnames. While it's possible to do so in some cases with null delimiters, I have never seen a case of newlines in pathnames in the wild. Of course, this means that I have to make sure that these scripts are never exposed to anything that could contain malicious filenames.
Also, I move away from portable shell if it will really make the code too convoluted or fragile. For instance, if I need to parse long options (--foo), I use GNU getopt rather than the built-in getopts (with appropriate test to make sure the getopt I'm using is GNU, so it won't break silently on a system without GNU getopt), or move to a real language.
Anyhow, while I know it's counterproductive, the puzzle-solving part of me really likes trying to solve the problem of implementing something in portable Bourne shell.
> One of the constraints I impose is that spaces in pathnames should always be handled correctly;
I tend to do the exact opposite. If a filename has spaces in it, i like my script to fail spectacularly while insulting the user (who is typically me....)
Filenames are variable names. It makes no sense to allow spaces in them.
I tend to do the exact opposite. If a filename has spaces in it, i like my script to fail spectacularly while insulting the user (who is typically me....)
Filenames are variable names. It makes no sense to allow spaces in them.
I need to deal with end user data a lot; which frequently has spaces in it.
D can be used as a scripting language:
#! /usr/bin/env rdmd
import std.stdio;
void main() { writeln(2); }
C also thanks to Fabrice Bellard
#!/usr/bin/tcc -run
#include <stdio.h>
int main() {
printf("Hello, tcc\n");
return 0;
}tcc is neat software and I used it for some time almost exclusively for "-run", but after many years I ultimately replaced it with a small shell rc-function for compiling, linking and running a C/C++/x86/etc. file from the shell.
I think it's nicer.
I think it's nicer.
#!/bin/sh
crun() {
local file="$1"
shift
local exepath="$(mktemp)"
if [[ "$file" =~ \.c$ ]]; then
gcc -g -Wall "$file" -o "$exepath" || return $?
else
echo "no filetype detected"
return 126
fi
"$exepath" "$@" & fg
}
#!/usr/bin/env zsh
function crun {
zparseopts -E -D -- -gcc::=use_gcc \
c:=custom_compiler \
o+:=copts \
Wl+:=lopts \
-dump::=dump_asm \
v::=verbose \
h::=usage \
g::=debug
if [[ -n $usage ]]; then
cat <<- EOF
usage: crun [options]... <filename>
--clang (default) use clang for C & C++ files
--gcc use GCC for C & C++ files
--dump dump assembly of program
-o supply an option (e.g -o -Wall)
-v verbose
-g debug
Compiles and runs a C, C++ or x86 Assembly file.
EOF
return 126
fi
# select unique entries of `copts` and then slice copts[2..] (copts[1]
# contains the flag, e.g "-o")
local file=${@[-1]}
local options=${${(u)copts}[2,-1]}
local exepath="$(mktemp)"
if [[ $file =~ \.(cc|cpp|cxx)$ ]]; then
local compiler="clang++"
$compiler -std=c++1z -g -Wall -Weffc++ ${=options} $file -o $exepath
elif [[ $file =~ \.c$ ]]; then
local compiler="clang"
[[ -n $use_gcc ]] && ccompiler="gcc"
$compiler -g -Wall ${=options} $file -o $exepath
elif [[ $file =~ \.(s|asm)$ ]]; then
local objpath="$(mktemp)"
nasm -felf64 $file -o $objpath && ld $objpath -o $exepath
else
echo "no filetype detected"
return 126
fi || return $?
if [[ -n $dump_asm ]]; then
objdump -S -M intel -d $exepath
else
[[ -n $verbose ]] && echo "exepath: $exepath"
if [[ -n $debug ]]; then
gdb --args "$exepath" "$@"
else
"$exepath" "$@" & fg
fi
fi
}Not really. It has rules for function naming and loops. I would set my rule as "never define a function or a loop". Once you get past a totally linear, stateless and deterministic script, stop writing bash. Roast me if you want, but my default language for this kind of stuff is Perl.
I really don't understand the "switch to Python" thing. Bash scripts are good for calling sequences of command line programs. That is not particularly convenient in Python.
It's perfectly convenient. The subprocess module works well enough for such things. Or even the old os.system.
Perfectly convenient, safe and fast. Pick any two.
https://docs.python.org/2/library/subprocess.html
I've told developers that if I catch them writing shell scripts in python they must commit a shell script doing the same set of operations. 2x the work usually dissuades from this type of nonsense.
I've told developers that if I catch them writing shell scripts in python they must commit a shell script doing the same set of operations. 2x the work usually dissuades from this type of nonsense.
As usual, "it depends." If you have complex logic, it's generally much cleaner to implement in python.
(If you told me I had to rewrite a working Python script in shell, I'd probably think you were joking. Talk about nonsense.)
(If you told me I had to rewrite a working Python script in shell, I'd probably think you were joking. Talk about nonsense.)
'It depends..' is a quagmire for the unwary.
My route is to make a good rule about writing python with more than 'n' os., subprocess. and *.Popen calls automatically requiring shell script counterparts in case we find your need to pythonize unsafe, unreadable and slow|broken|buggy.
My route is to make a good rule about writing python with more than 'n' os., subprocess. and *.Popen calls automatically requiring shell script counterparts in case we find your need to pythonize unsafe, unreadable and slow|broken|buggy.
There are obviously programs that are better written as a shell script, no denying it! I'm just saying use the best tool for the job. If you're writing complex logic in a shell script, you're probably barking up the wrong tree. Likewise, if all you're doing is calling external programs from python, you're probably doing it wrong.
Often times it's easier to use binaries and manage their input/output through shell code.
Having a good understanding of shell code will also allow you to easily write slick one-liners that will save you lots of time.
Having a good understanding of shell code will also allow you to easily write slick one-liners that will save you lots of time.
For basic file system operations it’s a necessity. Write 10-20 line python script or type a one line grep/awk/sed piipeline to spit out the contents of some files?
It's a 10-20 line Python script if you write it without any support code. If you're writing any amount of Python code that does shell things, you should either write or get an existing library.
Also, it really isn't 10-20 lines to spit out the contents of some files in Python:
For instance, if one of my files has a space in it, shell becomes a conceptual minefield, requiring you to understand the half-a-dozen ways it might process strings and how that interacts with filenames and arguments passed to commands, whereas the Python just keeps doing what you probably meant, because it knows what's a string and what's something else. Shell makes a lot of sacrifices for that concision, and there's a lot of "Well, this will probably be OK on anything I run it on...". I never put spaces in my file name, but use periods or underscores instead, precisely so I don't screw myself over with shell. I shouldn't have to do that. Personally I think the crossover point is in the high single digit number of bash lines. The only advantage bash still holds over Python at that point is large pipelines, and 80% of those can be replaced by Python functions. For the remainder, use a pipelining library.
Also, it really isn't 10-20 lines to spit out the contents of some files in Python:
>>> filelist = ["tmp.pl", "tmp.go", "Tmp.hs"]
>>> for f in filelist:
... print(open(f).read())
For instance, if one of my files has a space in it, shell becomes a conceptual minefield, requiring you to understand the half-a-dozen ways it might process strings and how that interacts with filenames and arguments passed to commands, whereas the Python just keeps doing what you probably meant, because it knows what's a string and what's something else. Shell makes a lot of sacrifices for that concision, and there's a lot of "Well, this will probably be OK on anything I run it on...". I never put spaces in my file name, but use periods or underscores instead, precisely so I don't screw myself over with shell. I shouldn't have to do that. Personally I think the crossover point is in the high single digit number of bash lines. The only advantage bash still holds over Python at that point is large pipelines, and 80% of those can be replaced by Python functions. For the remainder, use a pipelining library.
for i in ‘ls /tmp/foo’; do echo $i | grep bar | cut -d’-‘ -f3,4; done
I said Python wasn't going to be more concise. I said it has a variety of other useful properties. For instance, at least as I write this, you have "cut -d’-‘"; I assume you meant apostrophes (something getting too polite in a c&p, I assume) but the apostrophes are unnecessary, but you're so used to the compromises in shell I talked about you probably put those in there automatically. I'm not criticizing that; it's a reasonable accommodation to shell's quirks. But it is a quirk.
Your code also breaks if the filenames have spaces:
(HN is forcing me to say [asterisk]. AFAIK there's still no escaping for them.)
You lucked out a bit here too; I don't think I can get echo to do anything very interesting based on file names. Had you interpolated that somewhere more interesting I'd give you some security issues too, if anybody not trusted can create files, which, again, the naive Python code would be much more robust to, because in Python, you have to explicitly ask to bring a string up to an execution context or a function parameter context, unlike shell's complicated rules for interleaving them. You pay a steep price for the convenience.
Of course, this is a matter of scale. I use that sort of pipeline interactively every day. I just start shuddering when it goes in a file for a script, and shudder more if it goes into source control.
Your code also breaks if the filenames have spaces:
$ ls -1
bar-def-gh i-jkl-mno-p
bar-def-ghi-jkl-mno-p
$ for i in `ls`; do echo $i | grep bar | cut -d- -f3,4; done
gh
ghi-jkl
(HN is forcing me to say [asterisk]. AFAIK there's still no escaping for them.)
You lucked out a bit here too; I don't think I can get echo to do anything very interesting based on file names. Had you interpolated that somewhere more interesting I'd give you some security issues too, if anybody not trusted can create files, which, again, the naive Python code would be much more robust to, because in Python, you have to explicitly ask to bring a string up to an execution context or a function parameter context, unlike shell's complicated rules for interleaving them. You pay a steep price for the convenience.
Of course, this is a matter of scale. I use that sort of pipeline interactively every day. I just start shuddering when it goes in a file for a script, and shudder more if it goes into source control.
It's hilarious in these discussions that every time someone comes to show how they can do the equivalent code in much less lines of bash there are always always always bugs of these sorts.
And then the parent even specifically pointed out that files with spaces would be a minefield.
Also complaining about looking up docs is just a matter of where you have your knowledge. I would have the opposite problem to understand the "idiosyncrasies" of what the -d flag into cut does whereas I know the python standard library by heart.
And then the parent even specifically pointed out that files with spaces would be a minefield.
Also complaining about looking up docs is just a matter of where you have your knowledge. I would have the opposite problem to understand the "idiosyncrasies" of what the -d flag into cut does whereas I know the python standard library by heart.
You’re taking HN too seriously, it’s not a formal argument for nitpicking - it’s an opinion.
Sorry I wrote that in a taxi from my phone, not a serious usecase but that’s beside the point I was trying to make- I personally have a hard time using python over bash. Thanks for pointing out the security aspect, I don’t typically think about that.
I figured; you clearly knew enough bash to know what the real backticks are. Those... frontticks?... aren't even on most keyboards.
On the plus side, when I fixed the typographical issues, it did work. I'm pretty sure I could do this more-or-less first-time right in Python, too, but it clearly took me longer to get there.
On the plus side, when I fixed the typographical issues, it did work. I'm pretty sure I could do this more-or-less first-time right in Python, too, but it clearly took me longer to get there.
You seem to be imagining some sort of situation where bash is used to deal with input from customers (random users). You do not use bash for that. Filenames with spaces (lol) do not happen unless you do it yourself. Bash is useful for productivity when getting stuff done. For the work I do (where code never comes close to a customer and one-off tasks are common), if a candidate were to write a python script for the example line of bash I would be strongly biased towards rejecting them.
My real point about security is not that it is always in play per se, but that the naive Python will tend to do the right thing and the naive bash will not, and that's a non-trivial difference between the two. It goes beyond security. Generally as soon as you step out of bash into perl/python/ruby/whatever, you get an immediate boost to correctness of all kinds, unless you go out of your way to be sloppy. Yes, even Perl is an instant correctness boost.
As for file names not having spaces in them, after I said I personally don't use spaces in my file names,
(Also, because I know someone's gonna try to go for the zinger here, I am not being hypocritical. I do not have an objection to one-off scripting jobs, especially ones like this that are read-only so there is no permanent consequence for error.)
As for file names not having spaces in them, after I said I personally don't use spaces in my file names,
find -name '* *' | wc -l
(Also, because I know someone's gonna try to go for the zinger here, I am not being hypocritical. I do not have an objection to one-off scripting jobs, especially ones like this that are read-only so there is no permanent consequence for error.)
> Filenames with spaces (lol) do not happen unless you do it yourself.
This is a cop out. File names can totally contain spaces, and the difference between the Bash and Python solution is that one will continue to work when a file with spaces inevitably shows up and one will fail, possibly silently.
This is a cop out. File names can totally contain spaces, and the difference between the Bash and Python solution is that one will continue to work when a file with spaces inevitably shows up and one will fail, possibly silently.
What do you do that causes filenames to appear with spaces? And why do you not fix the cause instead of forcing yourself to deal with the possibility of a filename with a space in every possible situation?
[deleted]
This attitude is where it brings unexpected bugs in the long run. Don't expect but just be sure.
This is going to come down to what you're most familiar with. I personally can never remember what the arguments to `cut` are due to rarely using it, while I'd be able to write the equivalent python from memory.
Os.listdir isn't much harder to remember, and of course os.walk is actually probably easier for recursive filesystem actions.
One nice thing about using a pipeline is that it will be multi-threaded by default. That is each process will have it's own thread(s) and distribute any work across multiple cores.
Also with something like Gnu Parallel you can easily scale across multiple machines. Maybe there are more efficient approaches but a pipeline and gnu parallel is pretty easy to scale up without thinking much about it.
Also with something like Gnu Parallel you can easily scale across multiple machines. Maybe there are more efficient approaches but a pipeline and gnu parallel is pretty easy to scale up without thinking much about it.
That's just one personal anecdote though. I can do the Python example from memory but not the bash example.
[deleted]
There's a great talk by bwk where he complains about exactly this kind of behavior...before admitting that anything useful in a language will be abused in order to do silly things.
https://www.youtube.com/watch?v=Sg4U4r_AgJU
I'll admit, I'm no fan of Python, but this has little to do with the language itself. I found Bash and Ruby first, so Python just seems like too much work to get right.
Requisite xkcd: https://xkcd.com/1987/
https://www.youtube.com/watch?v=Sg4U4r_AgJU
I'll admit, I'm no fan of Python, but this has little to do with the language itself. I found Bash and Ruby first, so Python just seems like too much work to get right.
Requisite xkcd: https://xkcd.com/1987/
Well, I love and hate bash at the same time.
I love the way of writing scripts. You start with a simple command and keep piping and saving the output to other functions and commands until you reach the desired outcome. While doing so, you can simply wrap every command into wrapper functions and have the entire arsenal of cli applications at you disposal. While not having to care about such esoteric/complicated/efficient stuff like data types as everything is a string until you have a command which knows how to interpret it otherwise.
But at the same time those scripts tend to be very slow, lavish about resources (e.g. compared to compiled languages C/Go/Rust) and the practical/secure syntax is just awful. When simply saving the output of a command to a variable uses six special characters, you know you are coding (s)hell:
Despite the rough syntax I really enjoy writing bash scripts. Maybe its just that I am able to solve everyday problems with just a very few lines of code.
I love the way of writing scripts. You start with a simple command and keep piping and saving the output to other functions and commands until you reach the desired outcome. While doing so, you can simply wrap every command into wrapper functions and have the entire arsenal of cli applications at you disposal. While not having to care about such esoteric/complicated/efficient stuff like data types as everything is a string until you have a command which knows how to interpret it otherwise.
But at the same time those scripts tend to be very slow, lavish about resources (e.g. compared to compiled languages C/Go/Rust) and the practical/secure syntax is just awful. When simply saving the output of a command to a variable uses six special characters, you know you are coding (s)hell:
a="$(echo fun)"
Despite the rough syntax I really enjoy writing bash scripts. Maybe its just that I am able to solve everyday problems with just a very few lines of code.
Sure, you'd enjoy for the first few lines of bash code.
And on Windows there is something better than Bash, called PowerShell.
Not to put too fine a point on it, but that's nonsense.
Historically, PowerShell was supposed to be a replacement for cmd.exe and roughly equivalent to bash, yet suitable for the Windows environment. They tried early on to port some of the UNIX & Linux tools to Windows and it didn't fly, so this is what we're left with.
Although parts of Powershell are open source now, I suspect the mere existence of WSL means that Microsoft understands it has more to gain from compatibility with the rest of the world (Bash is nigh-everywhere) than imposing their particular philosophy on people.
(See also, Apple's decision to adopt UNIX underpinnings)
I'm prepared to be wrong if PowerShell actually gets used outside of Windows, but I wouldn't bet on it.
Historically, PowerShell was supposed to be a replacement for cmd.exe and roughly equivalent to bash, yet suitable for the Windows environment. They tried early on to port some of the UNIX & Linux tools to Windows and it didn't fly, so this is what we're left with.
Although parts of Powershell are open source now, I suspect the mere existence of WSL means that Microsoft understands it has more to gain from compatibility with the rest of the world (Bash is nigh-everywhere) than imposing their particular philosophy on people.
(See also, Apple's decision to adopt UNIX underpinnings)
I'm prepared to be wrong if PowerShell actually gets used outside of Windows, but I wouldn't bet on it.
> Historically, PowerShell was supposed to be a replacement for cmd.exe and roughly equivalent to bash
What? PowerShell has radically different ideology. Plus it is super consistent (helps discover commands), extensible and works with objects rather plain strings, use .NET functions or call custom .dll functions...
> They tried early on to port some of the UNIX & Linux tools to Windows and it didn't fly
Actually they just use aliases to map well-known unix commands to powershell counterparts. It is not expected to have command switches work like that etc. Just a convenience for folks used to rm, ls, tee, wget, whatever. Maybe bad decision because of confusion, maybe a good one because helps people get started with PowerShell.
For me as a Windows admin, PowerShell is one of the best things within Windows Ecosystem.
What? PowerShell has radically different ideology. Plus it is super consistent (helps discover commands), extensible and works with objects rather plain strings, use .NET functions or call custom .dll functions...
> They tried early on to port some of the UNIX & Linux tools to Windows and it didn't fly
Actually they just use aliases to map well-known unix commands to powershell counterparts. It is not expected to have command switches work like that etc. Just a convenience for folks used to rm, ls, tee, wget, whatever. Maybe bad decision because of confusion, maybe a good one because helps people get started with PowerShell.
For me as a Windows admin, PowerShell is one of the best things within Windows Ecosystem.
There is an episode of the "To Be Continuous" podcast with Jeff Snover (PS's inventor) that gets into this a bit more. Because of the difference in philosophy, a straight port to Windows didn't work well.
Basically, text files (UNIX) v structured data from API's (Windows), thus PowerShell was born.
https://www.heavybit.com/library/podcasts/to-be-continuous/e...
Basically, text files (UNIX) v structured data from API's (Windows), thus PowerShell was born.
https://www.heavybit.com/library/podcasts/to-be-continuous/e...
Powershell's philosophy is the same as the one behind core-utils: Simple tools you can chain together to do complex tasks. Only it was made with 30 years of hindsight, so it actually does it much better.
Powershell on Windows is better than bash on Windows.
> Not to put too fine a point on it, but that's nonsense.
In PowerShell you pass structured data around instead of globs of text, which is much more maintainable. Using Bash on Windows is like going backwards, and it's sad that most developers don't even realize it.
> Historically, PowerShell was supposed to be a replacement for cmd.exe and roughly equivalent to bash, yet suitable for the Windows environment. They tried early on to port some of the UNIX & Linux tools to Windows and it didn't fly, so this is what we're left with.
Nonsense, Powershell is a fundamental improvement. It's not a "this is what's we're left with" situation.
> Although parts of Powershell are open source now, I suspect the mere existence of WSL means that Microsoft understands it has more to gain from compatibility with the rest of the world (Bash is nigh-everywhere) than imposing their particular philosophy on people.
Yeah, let's just drag everything down to the lowest common denominator.
> (See also, Apple's decision to adopt UNIX underpinnings)
Which is sad, since it could have been so much more. The Unix way of doing things, isn't the be all and all.
https://www.youtube.com/watch?v=rmsIZUuBoQs
https://news.ycombinator.com/item?id=16813796
In PowerShell you pass structured data around instead of globs of text, which is much more maintainable. Using Bash on Windows is like going backwards, and it's sad that most developers don't even realize it.
> Historically, PowerShell was supposed to be a replacement for cmd.exe and roughly equivalent to bash, yet suitable for the Windows environment. They tried early on to port some of the UNIX & Linux tools to Windows and it didn't fly, so this is what we're left with.
Nonsense, Powershell is a fundamental improvement. It's not a "this is what's we're left with" situation.
> Although parts of Powershell are open source now, I suspect the mere existence of WSL means that Microsoft understands it has more to gain from compatibility with the rest of the world (Bash is nigh-everywhere) than imposing their particular philosophy on people.
Yeah, let's just drag everything down to the lowest common denominator.
> (See also, Apple's decision to adopt UNIX underpinnings)
Which is sad, since it could have been so much more. The Unix way of doing things, isn't the be all and all.
https://www.youtube.com/watch?v=rmsIZUuBoQs
https://news.ycombinator.com/item?id=16813796
I used Powershell extensively before I ever had experience with a linux or unix shell. I still have more experience with Powershell than bash or zsh, but I can't imaging going back. I'm already much more capable in a *nix shell.
There are advantages to having a pipe based on structured data, but that also means you have to know what structure to expect at ever step of the way and whether or not other tools can work with that structure. There are also tons of little quirks about Powershell; enough so that I eventually just started writing most stuff in C# and then just using the Powershell scripts as glue.
Passing text means that every single one of the thousands and thousands of unix cli tools that accept stdio will work in your pipeline.
I think this industry needs to get away from the terrible/horrible, absolute thinking. Most tools have positive and negative
There are advantages to having a pipe based on structured data, but that also means you have to know what structure to expect at ever step of the way and whether or not other tools can work with that structure. There are also tons of little quirks about Powershell; enough so that I eventually just started writing most stuff in C# and then just using the Powershell scripts as glue.
Passing text means that every single one of the thousands and thousands of unix cli tools that accept stdio will work in your pipeline.
I think this industry needs to get away from the terrible/horrible, absolute thinking. Most tools have positive and negative
Passing text means that every single one of the thousands and thousands of unix cli tools that accept stdio will work in your pipeline.
It also means every one of those thousands and thousands of Unix clip tools needs to have a parser to turn that text into some sort of structure to operate on.
It also means every one of those thousands and thousands of Unix clip tools needs to have a parser to turn that text into some sort of structure to operate on.
Which allows them to determine exactly the best way to do that for their purposes. I just run the command and it magically works.
It'll be a long time before the Powershell ecosystem has a fraction of tools that are compatible with it's .NET object pipeline.
It'll be a long time before the Powershell ecosystem has a fraction of tools that are compatible with it's .NET object pipeline.
Which allows them to determine exactly the best way to do that for their purposes. I just run the command and it magically works.
Up until someone changes the text output of the program and it all goes bad. Never mind the maintenance headaches that basically get solved by people moving their command lines into databases where typing happens.
It'll be a long time before the Powershell ecosystem has a fraction of tools that are compatible with it's .NET object pipeline.
That's more a function of age and developers than the typing itself.
Up until someone changes the text output of the program and it all goes bad. Never mind the maintenance headaches that basically get solved by people moving their command lines into databases where typing happens.
It'll be a long time before the Powershell ecosystem has a fraction of tools that are compatible with it's .NET object pipeline.
That's more a function of age and developers than the typing itself.
Sad to see this downvoted.
As a 20 year bash veteran: powershell is the first OS default shell that outputs structured data, and looking at objects and selecting the properties you want is a massive improvement than combining bash with sed/grep/awk to scrape text.
Someone bizarrely responds that cmd still exists on Windows for compatibility purposes (though even Win+X starts powershell now) doesn't change this at all.
The README for my powershell profile (which is written from a *nix PoV) has a little more info comparing the two: https://github.com/mikemaccana/powershell-profile
As a 20 year bash veteran: powershell is the first OS default shell that outputs structured data, and looking at objects and selecting the properties you want is a massive improvement than combining bash with sed/grep/awk to scrape text.
Someone bizarrely responds that cmd still exists on Windows for compatibility purposes (though even Win+X starts powershell now) doesn't change this at all.
The README for my powershell profile (which is written from a *nix PoV) has a little more info comparing the two: https://github.com/mikemaccana/powershell-profile
I'm always amused by these PowerShell threads on HN.
How is it that objects are an accepted thing for basically every programming environment in modern use, yet somehow controversial when it comes to the shell?
The prayer-based text parsing toolchain sucks. It has always sucked, regardless of platform. We put up with it because it was all that we had.
Jeffrey Snover came up with something better and thanks to PS Core we'll be able to use it everywhere!
How is it that objects are an accepted thing for basically every programming environment in modern use, yet somehow controversial when it comes to the shell?
The prayer-based text parsing toolchain sucks. It has always sucked, regardless of platform. We put up with it because it was all that we had.
Jeffrey Snover came up with something better and thanks to PS Core we'll be able to use it everywhere!
Yeah - most people would agree that GraphQL is a better way to access data than, say headless Chrome via Puppeteer. Many folk here prefer TeX over Word because the former encourages seperating content from presentation. But when it comes to the shell, suddenly everyone hates the idea.
> How is it that objects are an accepted thing for basically every programming environment in modern use,
They aren't accepted for basically every programming environmental nment in modern use; peak OOP-all-the-things is well in the past, and it's no longer the one paradigm to rule them all, irrespective of use case.
They aren't accepted for basically every programming environmental nment in modern use; peak OOP-all-the-things is well in the past, and it's no longer the one paradigm to rule them all, irrespective of use case.
Whether you have methods or functions, you still have hashmaps - bash you scrape values, pwsh you select keys. Pipelines can be considered quite functional too.
> Nonsense, Powershell is a fundamental improvement. It's not a "this is what's we're left with" situation.
It would be, if cmd.exe was long dead, PowerShell had stolen the good parts and continued on its way, but we're not there yet. Cmd.exe still exists on Win 10 and let's not even get into the fact that 64 bit only versions of Windows are not yet everywhere. Sure, Microsoft's ecosystem is larger and more complex, but at some point Win32 becomes a liability and unnecessary baggage.
> Which is sad, since it could have been so much more. The Unix way of doing things, isn't the be all and all.
Cultural compatibility matters - See C & C++. How else do you expect to get users and developers on board if you can't show them how your system is better in a few key ways, but isn't a giant leap from what they're currently using?
Tl;dr Change is scary, try to make it less so.
It would be, if cmd.exe was long dead, PowerShell had stolen the good parts and continued on its way, but we're not there yet. Cmd.exe still exists on Win 10 and let's not even get into the fact that 64 bit only versions of Windows are not yet everywhere. Sure, Microsoft's ecosystem is larger and more complex, but at some point Win32 becomes a liability and unnecessary baggage.
> Which is sad, since it could have been so much more. The Unix way of doing things, isn't the be all and all.
Cultural compatibility matters - See C & C++. How else do you expect to get users and developers on board if you can't show them how your system is better in a few key ways, but isn't a giant leap from what they're currently using?
Tl;dr Change is scary, try to make it less so.
Which actually supports linux with PS 6.0+
Unless you want secure strings.. then it complains about "Crypt32.dll" not being available.
Or want to pipe data without PowerShell mangling it[0]
PowerShell is materially worse than bash for a great many purposes because of this design choice alone.
[0] https://brianreiter.org/2010/01/29/powershells-object-pipeli...
PowerShell is materially worse than bash for a great many purposes because of this design choice alone.
[0] https://brianreiter.org/2010/01/29/powershells-object-pipeli...
The particular example this user complains about is piping binary data to file. It makes use of the Out-File cmdlet which is making (by default) Windows based assumptions as to how to encode this content. They would have been better served by using cURL's native -O parameter. To say that the pipe corrupts it is another thing entirely.
Indeed.
I'm a huge PS fan and like to stay as far away as possible from Bash but still, I wouldn't call it better than Bash. At least not always and everywhere. More like different. Both come with a list of advantages and disadvantages. And once you learnt one, the list of disadvantages of the other probably seems too big to even start with it, at least that's what it is now for me.
Its too verbose for me and I would rather use Python or Bash. WSL is amazing for these things.
It has aliases. Both built in and user definable. In fact, there are default aliases that match many of the utilities you're used to.
You might know that if you'd spent more than 30 seconds with it before shouting "it's not bash!".
You might know that if you'd spent more than 30 seconds with it before shouting "it's not bash!".
I'll write fairly long bash scripts from time to time, but my rule is more: "Do I plan on running this next week?" or "Do intend this for anybody else to run?", and not so much "How many lines is this".
[deleted]
i'm surprised it doesn't recommend using set -e /other flags. [http://redsymbol.net/articles/unofficial-bash-strict-mode/] maybe it is not covered because it is not considered 'style' but i think those flags are some of the most important things you can set when writing a bash script. i guess if you are writing scripts that start other things, and then check their error codes it can be annoying because you have turn them off/on.
`set -e` is basically a way to find where you need better error-handling, not a way to make scripts safe. I'd much rather a crummy script run into trouble calling some new featuref, shrug, and move on to the core business-critical code, instead of erroring out and triggering a bunch of pages.
Do you normally protect every command with || exit?
cd foobar || exit
Why wouldn't I rewrite the script to avoid the cd instead? The point of `set -e` isn't to insert all those pointless guards, it's to point out "hey this might fail, so think of a way to do this that doesn't rely on cd".
For your extremely thin example, if it (say) removed some files, I would only remove the files from the current directory or an absolute path.
For your extremely thin example, if it (say) removed some files, I would only remove the files from the current directory or an absolute path.
Please don't blindly follow style guides. Use what works right for you. I generally strongly dislike the decions codified in Google "style guides"
I'm really surprised they went with:
#!/bin/bash
as opposed to:
#!/usr/bin/env bash
the latter feels more flexible and dependable for a script to be passed around.
#!/bin/bash
as opposed to:
#!/usr/bin/env bash
the latter feels more flexible and dependable for a script to be passed around.
It is, but don't forget it's a guide for Google engineers on the Google machines.
People who share code for the world to use should use:
- `#!/usr/bin/env bash`
- or `#!/bin/sh` for POSIX shell scripting
People who share code for the world to use should use:
- `#!/usr/bin/env bash`
- or `#!/bin/sh` for POSIX shell scripting
/bin/sh is most often either the C shell, Dash, or Ash, not the Bourne shell. Bourne shell extensions will cause the script to fail.
edit: bourne again shell
edit: bourne again shell
You certainly meant /bin/sh is not the Bourne again shell. I updated my comment to clear any possible misunderstanding
https://en.wikipedia.org/wiki/Bourne_shell
https://en.wikipedia.org/wiki/Bourne-again_shell
https://en.wikipedia.org/wiki/Bourne_shell
https://en.wikipedia.org/wiki/Bourne-again_shell
/bin/sh shouldn't be a C shell, it's always a POSIX-compatible shell IIRC
In the wild, hopefully not, but I do remember coming across /bin/sh executing csh in an obscure RTOS-ish / proprietary flavor of Linux that we were prototyping at a previous employer. Would've been around 2008. I can't remember the variant, but I remember being caught off-guard by it.
AFAIK /bin/sh is standard; if you cannot trust that /bin/sh is a Bourne-like sh, you might as well have to assume /usr/bin/env can not be what it was meant to be too.
I suppose you're right, though the main point is it may not have Bourne-again extensions
I once triggered an automatic deploy that ran a bash script that minified a bunch of css/html/js assets and pushed them to S3, at some path like `/assets/vX.X.X/whatever.css`. Unfortunately, a few of our deploy boxes had Dash instead of Bourne shell, and when it ran on dash it failed to parse the version tag to push and ended up pushing an empty dir to /assets - deleting everything. Fun experience.
[deleted]
> /bin/sh is most often either the C shell,
Never once have I ever seen this. In 25+ years.
Never once have I ever seen this. In 25+ years.
Doesn't explain why Google wouldn't use env unless there's a good reason to avoid env.
Top of my head:
- it works
- it's simple
- they know that bash is in /bin
- maybe it avoids a needless call to env, resulting in better startup time (see e.g. https://news.ycombinator.com/item?id=16978932 for a similar situation with python)
- it works
- it's simple
- they know that bash is in /bin
- maybe it avoids a needless call to env, resulting in better startup time (see e.g. https://news.ycombinator.com/item?id=16978932 for a similar situation with python)
Google's linux systems are standardized, there isn't a concern about running a script on 10 different OS variants
Could you please explain what #!/use/bin/env bash does?
Why is it better than the other statement?
Thanks!
Why is it better than the other statement?
Thanks!
I really don't like the `/usr/bin/env bash` approach (it came from the ruby world I think?). It's less portable than /bin/bash in my experience - I've been in environments, usually older ones, where it doesn't work at all. But /bin/bash works everywhere. You also can't pass command line arguments. And it's slightly more complex than just invoking /bin/bash.
> It's less portable than /bin/bash in my experience
/bin/bash breaks on FreeBSD, which installs Bash at /usr/local/bin/bash. /usr/bin/env bash works though.
> You also can't pass command line arguments.
Are you sure about that? Given this script:
/bin/bash breaks on FreeBSD, which installs Bash at /usr/local/bin/bash. /usr/bin/env bash works though.
> You also can't pass command line arguments.
Are you sure about that? Given this script:
#!/usr/bin/env bash
echo "args: $@"
$ ./foo 1 2 3
args: 1 2 3
You can do “/bin/bash -e” but not “/usr/bin/env bash -e”. Same for python etc.
Good point about freebsd - I remember running into that exact problem. But I think I just symlinked it.
https://unix.stackexchange.com/questions/29608/why-is-it-bet... has more info on the pitfalls.
In my experience env causes problems, whereas explicit binary call always works (assuming the path exists). Not sure why I was downvoted.
Good point about freebsd - I remember running into that exact problem. But I think I just symlinked it.
https://unix.stackexchange.com/questions/29608/why-is-it-bet... has more info on the pitfalls.
In my experience env causes problems, whereas explicit binary call always works (assuming the path exists). Not sure why I was downvoted.
Is this a problem with env? What does the "-e" flag do anyway? "man bash" proved unenlightening.
It's not because of env. It's just that shebangs are limited to passing one optional argument to the executable specified. The way they're parsed is the executable path, and if there's a space, then the rest of the line (including any additional spaces) is interpreted as 1 argument to the executable.
EDIT: -e causes bash to exit on the first command that returns with an error (returns with a non-zero return code). It's documented in `man bash` where it documents the `set` builtin command.
EDIT: -e causes bash to exit on the first command that returns with an error (returns with a non-zero return code). It's documented in `man bash` where it documents the `set` builtin command.
Thanks for clarifying. In that case, one could imagine an env capable of parsing its single argument into the multiple intended arguments? I admit that I never dreamed that the "set" builtin would have the information about "bash -e" while the initial "OPTIONS" section did not. Why not look under the "cd", "echo", "fc", "read", or "ulimit" builtins? Somewhat helpfully, the "COMMAND EXECUTION ENVIRONMENT" section talks about how "-e" is inherited but not what it means. Ah, man pages.
Well, to be fair, the first sentence in OPTIONS is:
> All of the single-character shell options documented in the description of the set builtin command can be used as options when the shell is invoked.
It's easy to miss though, when one's accustomed to quickly searching with "/".
EDIT: On the imagined env, it might be possible. I wonder what aspects of shebangs are portable across the various unix derived OSs. It may be that there's some systems where it only takes non-whitespace characters as the first argument and drops everything else after encountering another space. There might be also other limitations that one should consider when making more use of shebangs, like the character limit of the lines. For example, I think linux only takes the first 127 characters of the shebang and ignores everything else; other systems might take less. I encountered that limit when writing this answer:
https://unix.stackexchange.com/questions/365436/choose-inter...
EDIT 2: If you were to do such a program, it will probably also need to implement quoting and escaping syntax. The difficult part is that, since it wouldn't be a standard program to have, it'd have to be listed as a dependency of whatever projects you use it in, and I wonder if the benefit really outweighs the cost of having yet another dependency. I can't see something like this gaining wide adoption.
> All of the single-character shell options documented in the description of the set builtin command can be used as options when the shell is invoked.
It's easy to miss though, when one's accustomed to quickly searching with "/".
EDIT: On the imagined env, it might be possible. I wonder what aspects of shebangs are portable across the various unix derived OSs. It may be that there's some systems where it only takes non-whitespace characters as the first argument and drops everything else after encountering another space. There might be also other limitations that one should consider when making more use of shebangs, like the character limit of the lines. For example, I think linux only takes the first 127 characters of the shebang and ignores everything else; other systems might take less. I encountered that limit when writing this answer:
https://unix.stackexchange.com/questions/365436/choose-inter...
EDIT 2: If you were to do such a program, it will probably also need to implement quoting and escaping syntax. The difficult part is that, since it wouldn't be a standard program to have, it'd have to be listed as a dependency of whatever projects you use it in, and I wonder if the benefit really outweighs the cost of having yet another dependency. I can't see something like this gaining wide adoption.
Haha thanks that is why I missed it.
Interestingly, FreeBSD used to allow for multiple arguments.
Do you know why they stopped?
So they did it for compatibility. In the end, it seems better to take everything as 1 argument instead of simply splitting on spaces, without having support for quoting and escaping. That way you can have arbitrary code with spaces in the shebang if that's ever a good idea.
It is probably silly to dream that kernels will change their behavior, if FreeBSD had to revert such an improvement. However as you describe env is actually downstream of the kernel, so if it were improved we could envision shebangs like the following:
#!/usr/bin/env --new-env-behavior-parse-all-args=bash -e
> You can do “/bin/bash -e” but not “/usr/bin/env bash -e”.
Makes sense. Thanks for clarifying. I had a feeling I wasn't understanding your point.
Makes sense. Thanks for clarifying. I had a feeling I wasn't understanding your point.
[deleted]
He probably means you can't:
#!/usr/bin/env bash -ex
#!/bin/bash -exAh, makes sense then. Thanks for clarifying.
> I really don't like the `/usr/bin/env bash` approach (it came from the ruby world I think?). It's less portable than /bin/bash in my experience - I've been in environments, usually older ones, where it doesn't work at all.
Its not a ruby thing, its the way you're supposed to write portable scripts where the location of the executable might not be predetermined. Lets say you're on a distribution where bash is in /usr/bin/bash instead of /bin/bash. Well you'll be modifying the shebang line to either do /usr/bin/env bash or pointing it to that spot.
What old systems didn't it work on? I've used everything from SunOS, Irix, HPUX, AiX, Solaris, Linux, bsd's, anything claiming to be posix has to support that.
Its not a ruby thing, its the way you're supposed to write portable scripts where the location of the executable might not be predetermined. Lets say you're on a distribution where bash is in /usr/bin/bash instead of /bin/bash. Well you'll be modifying the shebang line to either do /usr/bin/env bash or pointing it to that spot.
What old systems didn't it work on? I've used everything from SunOS, Irix, HPUX, AiX, Solaris, Linux, bsd's, anything claiming to be posix has to support that.
[deleted]
Funny that if env is such a crucial part of the OS, why isn't it in /bin?
It didn't work because there was no /usr/bin/env in these systems, or because you wanted a bash different from what was found with $PATH to execute? What were these systems?
I think many lightweight container images don’t have /usr/bin/env for example but they all have /bin/sh. Which I think is the best option for portability.
/usr/bin/env is not POSIX anymore than /bin/bash etc.
/usr/bin/env is not POSIX anymore than /bin/bash etc.
http://manpag.es/RHEL6/1p+env
Mentions that the env command is POSIX, but it doesn't mention the absolute path. Are you saying that /use/bin/env is not POSIX because there is no guarantee that it will be available at that particular path?
Mentions that the env command is POSIX, but it doesn't mention the absolute path. Are you saying that /use/bin/env is not POSIX because there is no guarantee that it will be available at that particular path?
Interestingly enough, not even the path of /bin/sh is guarenteed. From the POSIX:
> Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in.
EDIT: I wondered what POSIX recommended to do for any use of the shebang and this is what I found:
> Furthermore, on systems that support executable scripts (the "#!" construct), it is recommended that applications using executable scripts install them using getconf PATH to determine the shell pathname and update the "#!" script appropriately as it is being installed (for example, with sed).
Wow.
> Applications should note that the standard PATH to the shell cannot be assumed to be either /bin/sh or /usr/bin/sh, and should be determined by interrogation of the PATH returned by getconf PATH, ensuring that the returned pathname is an absolute pathname and not a shell built-in.
EDIT: I wondered what POSIX recommended to do for any use of the shebang and this is what I found:
> Furthermore, on systems that support executable scripts (the "#!" construct), it is recommended that applications using executable scripts install them using getconf PATH to determine the shell pathname and update the "#!" script appropriately as it is being installed (for example, with sed).
Wow.
/usr/bin/env anything is a pain to use with cronjobs
I don't have much experience with cronjobs, but is $PATH not set-up properly in their environment?
I don't remember exactly but it may be $PATH.
I run into this trying to run rvm-managed Ruby scripts in cronjobs. The solution (IIRC) is to explicitly reference the interpreter you want to run in the cronjob line.
I run into this trying to run rvm-managed Ruby scripts in cronjobs. The solution (IIRC) is to explicitly reference the interpreter you want to run in the cronjob line.
This quote on whether to always use braces with variable names (e.g. "${var}" or "$var"):
> These are meant to be guidelines, as the topic seems too controversial for a mandatory regulation.
It's nice to see that even Google has bike-shedding arguments.
> These are meant to be guidelines, as the topic seems too controversial for a mandatory regulation.
It's nice to see that even Google has bike-shedding arguments.
> If you find you need to use arrays for anything more than assignment of ${PIPESTATUS}, you should use Python.
This is exactly why I wrote my last script in Python. I started googling for how to do some simple stuff with arrays in Bash and after 10 minutes decided "I'll write it in Python".
This is exactly why I wrote my last script in Python. I started googling for how to do some simple stuff with arrays in Bash and after 10 minutes decided "I'll write it in Python".
I've never seen this guide before today but skimming through I absolutely love it. It aligns with my exact bash scripting style. Everything from the variable and function naming rules, stderr logging, pipeline indentation, and even having a "main () { ...}" wrapper and invocation. It's almost creepy!
It's not creepy if you wrote it while working at google (as a mathematician, i couldn't help but notice that you are not excluding this possibility).
S/he did exclude this possibility by saying "I've never seen this guide before today" :-)
Maybe koolba was blind and has been cured.
[deleted]
The only thing I'd quibble with is the recommendation of [[ ... ]] over [ ... ], because shell programmers used to [ will be surprised that [[ "foo" == "f*" ]] does pattern matching. But that is more or less an arbitrary style preference.
One thing I'm curious about is whether Google machines have followed Debian etc. in making /bin/sh a faster non-bash shell, or if /bin/sh is bash.
One thing I'm curious about is whether Google machines have followed Debian etc. in making /bin/sh a faster non-bash shell, or if /bin/sh is bash.
> will be surprised that [[ "foo" == "f*" ]] does pattern matching
It will not do pattern matching, because you quoted the right-hand side.
It will not do pattern matching, because you quoted the right-hand side.
... ... okay, now I extremely object to [[ "foo" == f* ]] not doing globbing and doing pattern matching. But, I guess Google folks have experience that this is non-confusing?
Which is totally fucking indane
> One thing I'm curious about is whether Google machines have followed Debian etc. in making /bin/sh a faster non-bash shell, or if /bin/sh is bash.
Since the guide says "Executables must start with #!/bin/bash" it doesn't really matter.
Since the guide says "Executables must start with #!/bin/bash" it doesn't really matter.
Same here. Dropping non-bash support just for [[ ... ]] over [ ... ] seems a little odd.
I'm on a google machine now and it's a symlink to /bin/bash.
> Shell should only be used for small utilities or simple wrapper scripts.
Oh you mean we weren’t supposed to write an etl solution in bash?
Oh you mean we weren’t supposed to write an etl solution in bash?
90% of the time ETL is trivial and not system-critical and can be done with a 3 line bash script.
And in the other 10% a simple typo can cost your company millions of dollars.
And in the other 10% a simple typo can cost your company millions of dollars.
If it wasn’t a toy project in which the transformation was elementary then no, you probably shouldn’t have.
If it went well for your company in case it wasn’t a toy project then I would say that it was despite bash, not thanks to bash.
I’ve actually seen this happen. It’s terrifying.
You don't know true terror.
I've seen an "ETL" solution (minus the "T") written entirely in SQL but using XML instead of tables and columns. After traveling through multiple servers via OpenQuery it eventually gets loaded into a domain-specific piece. That piece passes the data into a frontend that can only run in quirks mode.
That frontend has 60-80 separate projects that all parse out basically the same data but from different fields and with different parsing/mapping logic. There is no ability to load shared libraries or otherwise de-duplicate code and no authorization to call out to any internal web services that could be written to handle it. Nor is there any form of templating available, it's all static files with no nesting of folders allowed. And the whole thing is managed in a domain-specific editor that effectively precludes the use of version control thanks to the way it stores files.
Two people are responsible for managing that frontend piece and more.
I've seen an "ETL" solution (minus the "T") written entirely in SQL but using XML instead of tables and columns. After traveling through multiple servers via OpenQuery it eventually gets loaded into a domain-specific piece. That piece passes the data into a frontend that can only run in quirks mode.
That frontend has 60-80 separate projects that all parse out basically the same data but from different fields and with different parsing/mapping logic. There is no ability to load shared libraries or otherwise de-duplicate code and no authorization to call out to any internal web services that could be written to handle it. Nor is there any form of templating available, it's all static files with no nesting of folders allowed. And the whole thing is managed in a domain-specific editor that effectively precludes the use of version control thanks to the way it stores files.
Two people are responsible for managing that frontend piece and more.
They should have fired the author/s of this abomination much earlier instead of allowing that thing to evolve up to that size.
> They should have fired the author/s of this abomination much earlier instead of allowing that thing to evolve up to that size.
The main author of the SQL side and original author of the frontend has been promoted to the top because the system is critical to the business needs, and since everyone else struggles to work with this monstrosity they must be a pretty great developer. Can't afford to lose them.
The frontend team have threatened to quit several times unless things get better. They've been demanding version control, CI, automated deployments, a shared library, and to not use the domain-specific editor piece that prevents all this.
Management (including (or because of?) the OG author) is convinced that they're pushing for these things because of job security and refuse to let them overcomplicate things. But every time they they threaten to quit they get pretty substantial raises because no one else is willing/able to deal with the mess they were left with.
If you ask me, no amount of money is worth that environment, but all parties involved still work there...
The main author of the SQL side and original author of the frontend has been promoted to the top because the system is critical to the business needs, and since everyone else struggles to work with this monstrosity they must be a pretty great developer. Can't afford to lose them.
The frontend team have threatened to quit several times unless things get better. They've been demanding version control, CI, automated deployments, a shared library, and to not use the domain-specific editor piece that prevents all this.
Management (including (or because of?) the OG author) is convinced that they're pushing for these things because of job security and refuse to let them overcomplicate things. But every time they they threaten to quit they get pretty substantial raises because no one else is willing/able to deal with the mess they were left with.
If you ask me, no amount of money is worth that environment, but all parties involved still work there...
Sounds very much like IBM AS400
I’m not sure when you worked on AS/400, but from what I remember there was no xml involved at all, although you had to deal with DB2 and the almost comical table and column names...
I meant the difficulty in storing source files in version control, the data input separated to 60-70 projects with very little code shared between them and the number of people responsible for understanding the system being very low, sounds very much like an AS400+rpg environment created for parsing XML files.
[deleted]
Sure, but not as terrifying as people rewriting shell one-liners in python or (gasp) java.
Then a lot of people don’t understand the limits of what they’re doing.
For instance piping though commands works fine until one character sequence is interpreted as EOF. The fun part is it will work most of the time, and when it fails nobody will understand why (“we didn’t touch anything”), and rewriting the thing will be a political nightmare (“it was working before and was only 3 lines, why you need so much time to redo it ?”)
I think most people (me included) don’t do enough shell programming to really know the trade-off of the one-liner vs doing it in groovy, so the latter becomes the safer option (rightly so IMO)
For instance piping though commands works fine until one character sequence is interpreted as EOF. The fun part is it will work most of the time, and when it fails nobody will understand why (“we didn’t touch anything”), and rewriting the thing will be a political nightmare (“it was working before and was only 3 lines, why you need so much time to redo it ?”)
I think most people (me included) don’t do enough shell programming to really know the trade-off of the one-liner vs doing it in groovy, so the latter becomes the safer option (rightly so IMO)
one character sequence is interpreted as EOF
I know this is just an example, but how would that happen?
I know this is just an example, but how would that happen?
You can hit EOF or EOT in a stream of UTF-8 or other multi-byte encoding as part of a character. There must be ways to have it handled through something that understands character encodings and workaround the issue, but I know that dumb shell piping will fumble on those.
% printf '\x04\necho EOT is not EOF in a pipe.\n' | bash
bash: line 1: $'\004': command not found
EOT is not EOF in a pipe.
%
If you can do it as a one-liner in Bash, then you're not using the JVM. The benefit of using Apache Groovy for scripting is access to the JVM (and of course defining Gradle builds). If you don't need the JVM, then do it in Python or Ruby.
I have been there, porting ksh scripts to Java.
As much as I might like Java, this surely did not make any sense, but when people want to pay you for doing it, oh well.
As much as I might like Java, this surely did not make any sense, but when people want to pay you for doing it, oh well.
I hope there was a reason for rewriting them..
Anyway Java is for sure NOT a scripting language.
F# is much much better from that point of view, with the right tools, but sadly it is not famous enough.
Well, rewriting one liners is kind of stupid, but it's probably not going to result in a nearly incomprehensible mess running in production.
As have I. It actually worked for a one time migration, but it required a lot of handholding.
> If performance matters, use something other than shell.
I wrote a script that wrapped compiler calls to make dummy output files to debug a huge compilation. Since startup time was the most important metric, bash actually had by far the best performance.
I wrote a script that wrapped compiler calls to make dummy output files to debug a huge compilation. Since startup time was the most important metric, bash actually had by far the best performance.
People rag on bash a lot, but it now forms the core of a major deployment system I built at work. I'm not happy that it's in bash, but every other solution was worse, and it calls out to other tools (e.g. Ansible and Terraform) when they're appropriate.
People have forgotten the subtle art of realizing what tool is best for the job. If all you're doing is stitching together other tools and doing some logging, bash is hard to beat.
People have forgotten the subtle art of realizing what tool is best for the job. If all you're doing is stitching together other tools and doing some logging, bash is hard to beat.
^ this!
Bash is the best stitching together language you'll get. Sure it sucks at scripting.. So stitch a script in there for God's sake!
Bash is the best stitching together language you'll get. Sure it sucks at scripting.. So stitch a script in there for God's sake!
Fish shell?
Not portable enough. The point of bash (well, really sh if we want to be pedantic) is that it's a stitching tool available on just about every conceivable platform.
I write long bash and perl scripts. I keep things neat and tidy most of the time. My "temporary workarounds" are still in use today, in production, so I must have written them well enough for people to understand. I see a lot of debate around preferences at the risk of becoming a funny title on n-gate. Just use whatever languages you know best. If it follows the best practices of that language, someone can port it to their preferred language.
Maybe it is just my browser, but that xml file should be renamed to .txt or have the mime type set differently. There is no formatting for me. It is just one really long line.
Maybe it is just my browser, but that xml file should be renamed to .txt or have the mime type set differently. There is no formatting for me. It is just one really long line.
My "temporary workarounds" are still in use today, in production, so I must have written them well enough for people to understand.
Maybe the reason the temporary workarounds persist is because nobody understands them well enough to replace them ;)
Maybe the reason the temporary workarounds persist is because nobody understands them well enough to replace them ;)
I completely understand. In my case, every one of my scripts has my name and email in it. They won't hesitate to reach out to me if something is broken. :-)
Nothing against you or your long perl scripts (wrote perl for a few years as well), but at my current job there are perl scripts as well, for legacy reasons. The scripts where written 20+ years ago to compliment our mail systems and identy management. Today nobody fixes those scripts or replaces them and finding a replacement solution for the underlying systems is nearly impossible, because nobody understands all the perl scripts or dares to touch them. Our dns management can't handle wildcards and sub-sub domains (easily) and every time something like this is needed there is one person who needs to manually add this.
Anyway the point I was trying to make is that temporary soluyions often stay because of other reasons than perfection.
On the flip side I solve every problem at first through bash or python scripts as well and aolutions tend to be permanent, so we are in the same boat.
On the flip side I solve every problem at first through bash or python scripts as well and aolutions tend to be permanent, so we are in the same boat.
> Indent 2 spaces
It's interesting how quickly the 2-space indent became the new norm
It's interesting how quickly the 2-space indent became the new norm
I'm very curious about why. In 8-space tab sizes, you have to very, very carefully think about what you're fitting on that 80-col line after having in most cases, already removed 10% of your column space. A lot of C resulted in single character variable names and abbreviations that sacrificed immediate readability, but with domain knowledge that gets resolved with time.
On the flip side of things, you have developers who universally write 240+ col lines in HTML and don't know what a punched card is.
On the flip side of things, you have developers who universally write 240+ col lines in HTML and don't know what a punched card is.
…I'm not seeing a strong argument for either in your response.
All Google style guides use two spaces.
not the python one! https://google.github.io/styleguide/pyguide.html
anything other than 4 spaces for python look out of place for me.
anything other than 4 spaces for python look out of place for me.
I've seen a lot of Python from Google using two spaces, which I find very hard to read. Many of the samples here use two-space indentation: https://github.com/google/google-api-python-client/blob/mast...
Here's a specific example: https://github.com/google/google-api-python-client/blob/mast...
Here's a specific example: https://github.com/google/google-api-python-client/blob/mast...
This is either outdated or only for the public version of the guide: the internal guide, and all our python code, has 2-spaces indents.
Meanwhile, the rest of the Python world probably follows PEP8, which uses 4 spaces.
https://www.python.org/dev/peps/pep-0008/#indentation
https://www.python.org/dev/peps/pep-0008/#indentation
Yeah, I'm not following that one. 2-space tabs are visually difficult.
Original HN title had the word "Google" in it.
According to this document, Google requires Bash.
I do not use Bash. I use Almquist shell. I try to avoid using uncommon "features" or utilities.
I am not a frequent Linux user but whenever I have to use it, all of the hundreds of scripts I wrote using another OS and Almquist shell still work. They all work in Bash.
Over the years, I used this site as a reference:
https://www.in-ulm.de/~mascheck/
According to this document, Google requires Bash.
I do not use Bash. I use Almquist shell. I try to avoid using uncommon "features" or utilities.
I am not a frequent Linux user but whenever I have to use it, all of the hundreds of scripts I wrote using another OS and Almquist shell still work. They all work in Bash.
Over the years, I used this site as a reference:
https://www.in-ulm.de/~mascheck/
They require bash for scripts.
You can use whatever shell you like on your machines.
You can use whatever shell you like on your machines.
"They require Bash for scripts."
I do not write scripts to run with Bash. I write them to run with an Almquist-derived shell, which also means I can run them with Bash and other shells without any problems. However a script written in Bash may not run correctly under Almquist shell and other shells.
I do not write scripts to run with Bash. I write them to run with an Almquist-derived shell, which also means I can run them with Bash and other shells without any problems. However a script written in Bash may not run correctly under Almquist shell and other shells.
"One example of this is Solaris SVR4 packages which require plain Bourne shell for any scripts."
Such as?
Such as?
" Indent 2 spaces. No tabs."
Oh dear, here we go....
Oh dear, here we go....
>Bash is the only shell scripting language permitted for executables.
Whelp, wrong right off the bat.
I'm gonna get down my my knees here and beg everyone reading: use POSIX shell. Do not write scripts with bash. Do not write scripts with zsh. Do not write scripts with fish. Use POSIX shell.
sh has a really bad interactive mode (so does bash), so I'm not gonna give anyone a hard time for using another shell as their day-to-day interactive shell. But, for the love of god, write your scripts with POSIX sh.
Whelp, wrong right off the bat.
I'm gonna get down my my knees here and beg everyone reading: use POSIX shell. Do not write scripts with bash. Do not write scripts with zsh. Do not write scripts with fish. Use POSIX shell.
sh has a really bad interactive mode (so does bash), so I'm not gonna give anyone a hard time for using another shell as their day-to-day interactive shell. But, for the love of god, write your scripts with POSIX sh.
> >Bash is the only shell scripting language permitted for executables.
> Whelp, wrong right off the bat.
It really is a Google guide, tailored to Google engineers. If they know for sure that bash is available on any *n?x machine, then why not? It's just a local policy.
And if it's that much of a problem, the other places where you get bash instead of POSIX sh from Google (chromium?) the license clearly allows you to rewrite said script to suit your needs.
All other points are valid though.
Note that I chose to write all my scripts in POSIX (e.g. https://gitlab.com/moviuro/moviuro.bin).
> Whelp, wrong right off the bat.
It really is a Google guide, tailored to Google engineers. If they know for sure that bash is available on any *n?x machine, then why not? It's just a local policy.
And if it's that much of a problem, the other places where you get bash instead of POSIX sh from Google (chromium?) the license clearly allows you to rewrite said script to suit your needs.
All other points are valid though.
Note that I chose to write all my scripts in POSIX (e.g. https://gitlab.com/moviuro/moviuro.bin).
>It really is a Google guide, tailored to Google engineers.
And posted in a public forum, which is why I made sure to protest its adoption by the broader public.
And posted in a public forum, which is why I made sure to protest its adoption by the broader public.
I don't get that from "Whelp, wrong off the bat", which seems to imply that there is a more unambiguous right answer here, and Google is wrong for deviating from it.
Well, even within Google I think this policy is stupid. Any software they make open source or have to port to new platforms that was based on this policy is going to break, and for what? Bashisms provide a dubious value-add.
I have the opposite opinion. I have seen developers proudly putting /bin/sh instead of /bin/bash, without any other shell to test. When the script was run on ubuntu instead of redhat, this called dash and we discovered that the script was finally not compliant on subtle details. Writing POSIX compliant scripts is harder. Where is the reward when bash is present (almost) everywhere?
>Where is the reward when bash is present (almost) everywhere?
It's really not. It's only present on some desktop Linux distributions. It's not present on almost any other OS or anywhere in the embedded space. POSIX shell, on the other hand, is everywhere.
It's really not. It's only present on some desktop Linux distributions. It's not present on almost any other OS or anywhere in the embedded space. POSIX shell, on the other hand, is everywhere.
Why wouldn't you install Bash then, if you want to run shell scripts on those platforms?
I mean, what are the use cases? Where universality is important (so let's say when someone embeds scripts in Makefiles), I understand the need for POSIX, but other that I can't even come up with a good use case for having sh as default. (Maybe to save space, you ship the embedded device or Docker container with sh, sure, but then what kind of script you'd want to run there? Okay, maybe all of them, but then even on alpine installing bash is one quick command.)
I mean, what are the use cases? Where universality is important (so let's say when someone embeds scripts in Makefiles), I understand the need for POSIX, but other that I can't even come up with a good use case for having sh as default. (Maybe to save space, you ship the embedded device or Docker container with sh, sure, but then what kind of script you'd want to run there? Okay, maybe all of them, but then even on alpine installing bash is one quick command.)
> Why wouldn't you install Bash then, if you want to run shell scripts on those platforms?
Busybox is a thing. Many embedded devices don't have the luxury of installing Bash–you're stuck with what you've got.
Busybox is a thing. Many embedded devices don't have the luxury of installing Bash–you're stuck with what you've got.
But what would a script do in such a limited situation? There are a lot of narrow use cases, but those are very much platform dependent, so no reason to use POSIX everywhere. Use whatever is best on that specific busybox thing.
>but then what kind of script you'd want to run there? Okay, maybe all of them
I think you answered your own question here.
I can elaborate, though. bash isn't portable, it uses Glibc-isms. It has to be explicitly ported to new platforms. I also pointed out elsewhere that the value-add of bash is dubious at best - if you need to use bashisms you're better off not writing a shell script at all. So really, the question is: why do you need bash? You have to justify using non-standard tools, not the other way around.
I think you answered your own question here.
I can elaborate, though. bash isn't portable, it uses Glibc-isms. It has to be explicitly ported to new platforms. I also pointed out elsewhere that the value-add of bash is dubious at best - if you need to use bashisms you're better off not writing a shell script at all. So really, the question is: why do you need bash? You have to justify using non-standard tools, not the other way around.
The mentioned "file transfer shell-program" somehow was so important that it has to run on AIX and Linux and every other OS under the sun, yet it was somehow not important enough to be written in non-sh.
And I feel the same with targeting POSIX sh.
Unless you have a really good business niche - like the aforementioned ftp script, use a proper language.
Yes, sysadmins were opposed to installing Bash. But if your core product depends on Bash and only on Bash, maybe I'd be opposed to everything about that deployment too.
And at the same time, I understand that there are IT tasks well suited to be solved with a script. But those don't seem to be the ones that require uber-portability.
If I need to setup something on a fleet of VMS-es, I'll use whatever I can get away with (including a lot of controlled substances to handle the dread of handling VMS), and won't think about what happens if we port this to HP-UX in the year 2525.
And I feel the same with targeting POSIX sh.
Unless you have a really good business niche - like the aforementioned ftp script, use a proper language.
Yes, sysadmins were opposed to installing Bash. But if your core product depends on Bash and only on Bash, maybe I'd be opposed to everything about that deployment too.
And at the same time, I understand that there are IT tasks well suited to be solved with a script. But those don't seem to be the ones that require uber-portability.
If I need to setup something on a fleet of VMS-es, I'll use whatever I can get away with (including a lot of controlled substances to handle the dread of handling VMS), and won't think about what happens if we port this to HP-UX in the year 2525.
There are many, many cases where a shell script is appropriate. I don't see a shell script as inherently evil, especially if you use one as conservative as POSIX sh. I ship shell scripts with my software often and for a wide variety of tasks.
> POSIX shell, on the other hand, is everywhere.
Indeed! The significance of this, I believe, is underappreciated.
During 2008-2015, when I used to work for a network security company, we had a file transfer script[1] for Unix and Linux systems. It was written entirely as a shell script. It began as a simple wrapper around the scp and sftp commands for which a shell script was appropriate. The fact that every Unix and Linux system comes with a shell meant that we could ship a single file to all target Unix and Linux systems without requiring the administrator to install any new packages. The alternative was writing the file transfer functionality in C or C++ which would have required us to build and test the solution for every possible Unix system that our customers used or writing it in Java which would have required us to require the customer to install JVM on their systems. Many customer admins were averse to installing any additional packages on their systems. Given these constraints, a shell script was a good solution.
Unfortunately, the initial shell script relied on several bashisms.[2] Some customers began complaining that their Solaris or AIX system did not have bash, and they needed the script to work with Bourne shell or Korn shell. The customers refused to install bash. So I decided to clean up the script, remove all bashisms from it, and ensure that only POSIX specified features were used, only POSIX specified commands were invoked with POSIX specified options only (with the exception of the scp and sftp commands of course because that was the primary reason why the customers wanted the script).
I am not fond of bashisms. If I need bashims like arrays or regex substitutions in parameter expansions, it's a sign that the script is becoming complex and I switch to a modern programming language. These days, when I write a shell script[3], I restrict myself to POSIX.2001 (2004 edition)[4], write unit tests to exercise every line of code, run the unit tests with bash, ksh, zsh, dash, posh, and yash on Debian, run the tests with bash, ksh, and zsh on macOS, and use Kcov[5] to measure test coverage.
[1]: https://community.rsa.com/docs/DOC-53124
[2]: https://mywiki.wooledge.org/Bashism
[3]: https://github.com/susam/vimer
[4]: http://pubs.opengroup.org/onlinepubs/009695399/
[5]: https://github.com/SimonKagstrom/kcov
Indeed! The significance of this, I believe, is underappreciated.
During 2008-2015, when I used to work for a network security company, we had a file transfer script[1] for Unix and Linux systems. It was written entirely as a shell script. It began as a simple wrapper around the scp and sftp commands for which a shell script was appropriate. The fact that every Unix and Linux system comes with a shell meant that we could ship a single file to all target Unix and Linux systems without requiring the administrator to install any new packages. The alternative was writing the file transfer functionality in C or C++ which would have required us to build and test the solution for every possible Unix system that our customers used or writing it in Java which would have required us to require the customer to install JVM on their systems. Many customer admins were averse to installing any additional packages on their systems. Given these constraints, a shell script was a good solution.
Unfortunately, the initial shell script relied on several bashisms.[2] Some customers began complaining that their Solaris or AIX system did not have bash, and they needed the script to work with Bourne shell or Korn shell. The customers refused to install bash. So I decided to clean up the script, remove all bashisms from it, and ensure that only POSIX specified features were used, only POSIX specified commands were invoked with POSIX specified options only (with the exception of the scp and sftp commands of course because that was the primary reason why the customers wanted the script).
I am not fond of bashisms. If I need bashims like arrays or regex substitutions in parameter expansions, it's a sign that the script is becoming complex and I switch to a modern programming language. These days, when I write a shell script[3], I restrict myself to POSIX.2001 (2004 edition)[4], write unit tests to exercise every line of code, run the unit tests with bash, ksh, zsh, dash, posh, and yash on Debian, run the tests with bash, ksh, and zsh on macOS, and use Kcov[5] to measure test coverage.
[1]: https://community.rsa.com/docs/DOC-53124
[2]: https://mywiki.wooledge.org/Bashism
[3]: https://github.com/susam/vimer
[4]: http://pubs.opengroup.org/onlinepubs/009695399/
[5]: https://github.com/SimonKagstrom/kcov
Note that there are several variants of the Korn shell. And there are even more shells than that.
You can test ksh93 and the MirOS Korn shell on Debian. For the Heirloom Bourne shell, though, look to FreeBSD, which has everything except the Watanabe shell.
* https://svnweb.freebsd.org/ports/head/shells/heirloom-sh/
* https://svnweb.freebsd.org/ports/head/shells/pdksh/
* https://svnweb.freebsd.org/ports/head/shells/ksh93/
* https://svnweb.freebsd.org/ports/head/shells/mksh/
* https://svnweb.freebsd.org/ports/head/shells/oksh/
* https://svnweb.freebsd.org/ports/head/shells/dash/
* https://www.freebsd.org/ports/shells.html
I appreciate what this sort of thing involves, as you can infer. (-:
* https://unix.stackexchange.com/a/434839/5132
You can test ksh93 and the MirOS Korn shell on Debian. For the Heirloom Bourne shell, though, look to FreeBSD, which has everything except the Watanabe shell.
* https://svnweb.freebsd.org/ports/head/shells/heirloom-sh/
* https://svnweb.freebsd.org/ports/head/shells/pdksh/
* https://svnweb.freebsd.org/ports/head/shells/ksh93/
* https://svnweb.freebsd.org/ports/head/shells/mksh/
* https://svnweb.freebsd.org/ports/head/shells/oksh/
* https://svnweb.freebsd.org/ports/head/shells/dash/
* https://www.freebsd.org/ports/shells.html
I appreciate what this sort of thing involves, as you can infer. (-:
* https://unix.stackexchange.com/a/434839/5132
Yes exactly, I had this experience and wrote about it here:
http://www.oilshell.org/blog/2018/01/28.html#limit-to-posix
http://www.oilshell.org/blog/2018/01/28.html#limit-to-posix
bash is slower and having it as /bin/sh was how so many (so so many) servers were taken over via shellshock
A factor two or three in memory consumption.
I'm genuinely curious, why? If you're writing generic scripts designed to run across multiple platforms then fine, but in the context of Google (or most other companies) they'll have a standard set of tooling available on all their machines, which presumably includes Bash if they're making that statement.
What about POSIX shell makes it superior for scripting?
What about POSIX shell makes it superior for scripting?
I don't feel as strongly about it as Sir_Cmpwn does but I agree with him. The logic goes something like: you should only use shell scripts for small scripts and not complex applications. Ergo the added functionality of bash scripts should not be necessary anyway and might actually lead you to believe that it's a proper programming language.
Or to put it the other way around: if you find yourself writing a POSIX shell script and find that you'd like to use more advanced features then one of two things can happen:
- You need portability to SunOS 4, so you suck it up and keep hacking your script because bash is not an option anyway;
- You don't need portability, so then why not use a proper programming language like python, perl or ruby rather than upgrading from crappy sh to slightly-less-crappy bash?
Or to put it the other way around: if you find yourself writing a POSIX shell script and find that you'd like to use more advanced features then one of two things can happen:
- You need portability to SunOS 4, so you suck it up and keep hacking your script because bash is not an option anyway;
- You don't need portability, so then why not use a proper programming language like python, perl or ruby rather than upgrading from crappy sh to slightly-less-crappy bash?
This logic is insane. This is like a company that mainly does C++ or Java saying that if you must use a scripting language it must be Tcl, not Perl or Python, least you get too comfortable with it.
If you want a policy against shellscripts growing out of hand have a policy against that, not that you must use the lowest common denominator even though you have Bash everywhere.
And I'm saying this as someone who almost exclusively writes POSIX sh for portability reasons, but my software isn't meant to target just Google's infra.
If you want a policy against shellscripts growing out of hand have a policy against that, not that you must use the lowest common denominator even though you have Bash everywhere.
And I'm saying this as someone who almost exclusively writes POSIX sh for portability reasons, but my software isn't meant to target just Google's infra.
I think it would be perfectly sane for a company to dictate which scripting language their environment will support. Shell scripts are a bit different though because if you write a large enough software system on a unix-like OS you'll probably want to write a few shell scripts here and there to glue things together, so forbidding them completely would be a bit silly and counterproductive. Saying "avoid shell scripts, if you really have to write one it must be simple and use POSIX sh" is not particularly insane to me.
But anyway my argument was more about my opinion about a developer talking for myself, not a company style guide. Life's too short to learn two shell dialects so I can either learn POSIX which will work basically everywhere and will let me write any simple script just fine or write Bash whose added complexity I probably won't need or even want.
The fact that I'm an embedded dev probably biases my view though, I generally target environments where bash isn't an option anyway.
But anyway my argument was more about my opinion about a developer talking for myself, not a company style guide. Life's too short to learn two shell dialects so I can either learn POSIX which will work basically everywhere and will let me write any simple script just fine or write Bash whose added complexity I probably won't need or even want.
The fact that I'm an embedded dev probably biases my view though, I generally target environments where bash isn't an option anyway.
Google also makes it difficult to compile Chromium on platforms where /usr/bin/python is python 3, because their shebangs are:
>What about POSIX shell makes it superior for scripting?
To address this specifically, I have written a blog article on the subject:
https://drewdevault.com/2018/02/05/Introduction-to-POSIX-she...
The main points are:
- bash is far from as ubiquitous as some people seem to think it is
- sh is formally standardized and has multiple competing implementations, whereas bash is defined by its implementation and there is only one
- sh is perfectly competent and easy to target, bash's value add is questionable
- By the time you need the things bash offers you might as well not use a shell script at all
#!/usr/bin/env python
#!/usr/bin/env python3
>What about POSIX shell makes it superior for scripting?
To address this specifically, I have written a blog article on the subject:
https://drewdevault.com/2018/02/05/Introduction-to-POSIX-she...
The main points are:
- bash is far from as ubiquitous as some people seem to think it is
- sh is formally standardized and has multiple competing implementations, whereas bash is defined by its implementation and there is only one
- sh is perfectly competent and easy to target, bash's value add is questionable
- By the time you need the things bash offers you might as well not use a shell script at all
If you find yourself compiling Google-stuff a lot on Arch, you might find this useful: https://github.com/mortie/nixConf/blob/master/bin/py2
I wrote that script exactly because of the issue you described, and I got tired of changing what /usr/bin/python is symlinked to constantly. With that script, I just run `py2 gn gen out/Debug` or whatever, and my /usr/bin/python stays intact so everything else which expects python3 works, and only the `gn` process and all subprocess sees python2.
This definitely doesn't excuse Google's behavior, but I don't expect it to be fixed soon. My bug report about it (https://bugs.chromium.org/p/webrtc/issues/detail?id=7376) was closed as wontfix (though I'm not entirely sure the person who closed it understood that I was asking for the python scripts to explicitly use python 2, not for them to port the scripts to python 3).
I wrote that script exactly because of the issue you described, and I got tired of changing what /usr/bin/python is symlinked to constantly. With that script, I just run `py2 gn gen out/Debug` or whatever, and my /usr/bin/python stays intact so everything else which expects python3 works, and only the `gn` process and all subprocess sees python2.
This definitely doesn't excuse Google's behavior, but I don't expect it to be fixed soon. My bug report about it (https://bugs.chromium.org/p/webrtc/issues/detail?id=7376) was closed as wontfix (though I'm not entirely sure the person who closed it understood that I was asking for the python scripts to explicitly use python 2, not for them to port the scripts to python 3).
Clearly the WONTFIX was wrongly issued. The issue should be re-opened.
The python v python3 thing is their (Py's implementers) own stupid fault; although the argument could be made that they didn't have any choice.
People have been on 2.7 for so long that a breaking change might be the only thing to get them to move over.
As Chromium is open source, I'm surprised they haven't really explained how people outside of the Googleplex should handle this.
Seems like they're left to their own devices.
People have been on 2.7 for so long that a breaking change might be the only thing to get them to move over.
As Chromium is open source, I'm surprised they haven't really explained how people outside of the Googleplex should handle this.
Seems like they're left to their own devices.
I believe your claim that
/bin/sh
Aye, I was incorrect. I have pushed a correction, but it will take a moment to appear on the site.
The second thing is not more portable than the first (unfortunately).
That, unfortunately, only works if you don't need to maintain backwards compatibility with older systems. `python` will exist and point to python2 on any system. `python2` may not if it predates or ignores the PEP. `python3` didn't even come installed by default on a lot of nixes until recently, making it the worst of option of the three.
Of course, it will take time for this to become effective. But this is the portable solution. Systems for which this does not work are broken and should be fixed, software should not be updated to accomodate for broken systems.
>software should not be updated to accomodate for broken systems.
Its not. The software was written, and maintains, backwards compatibility with systems that predate or partially predate the spec. The spec was written to be backwards compatible with those systems. Using the backwards compatibility option is correct if you wish to maintain backwards compatibility. Consider that Arch also technically violates Pep 394,
> for the time being, all distributions should ensure that python, if installed, refers to the same target as python2
We shouldn't update the shebangs in all of our software ever to accommodate a misbehaving, spec-non-compliant distro, should we?
Its not. The software was written, and maintains, backwards compatibility with systems that predate or partially predate the spec. The spec was written to be backwards compatible with those systems. Using the backwards compatibility option is correct if you wish to maintain backwards compatibility. Consider that Arch also technically violates Pep 394,
> for the time being, all distributions should ensure that python, if installed, refers to the same target as python2
We shouldn't update the shebangs in all of our software ever to accommodate a misbehaving, spec-non-compliant distro, should we?
Competing implementations sounds like a potential downside to me. That’s just more possible configurations where a bug could be hiding. Even if you use standard posix shell, you’d be wise to target a single implementation and stick to it.
This is exactly backwards. If you don't have separation of specification and application, you don't have a specification.
I mean, standards aren't necessary; interpreters can absolutely be snowflakes. But multiple implementations is a sign of strength, and intentionally targeting the bugs of one implementation is short sighted, if sometimes required.
I mean, standards aren't necessary; interpreters can absolutely be snowflakes. But multiple implementations is a sign of strength, and intentionally targeting the bugs of one implementation is short sighted, if sometimes required.
So you think that competition between gcc and clang is harmful to the C and C++ ecosystems? If you find bugs in an implementation of POSIX sh, you should report it to that implementation. Multiple competing implementations is a sign of a good standard, it proves the standard's correctness and demonstrates its maturity.
There is no standard to hold bash to. Any strange behavior of it might be decided as by design and kept forever (and replicated by anything which tries to be bash compatible) or it could be determined to be a bug and, when fixed, break the scripts which worked around it or depended on it. There's no way to know how this is going to shake out because there is no objective specification for the language. At least when there's a bug in one of these competing sh implementations that you're so worried about, it's objectively a bug and can be fixed.
There is no standard to hold bash to. Any strange behavior of it might be decided as by design and kept forever (and replicated by anything which tries to be bash compatible) or it could be determined to be a bug and, when fixed, break the scripts which worked around it or depended on it. There's no way to know how this is going to shake out because there is no objective specification for the language. At least when there's a bug in one of these competing sh implementations that you're so worried about, it's objectively a bug and can be fixed.
To be more precise, my point was that you should only be targeting one of the implementations anyway. I don't see how it's a good use of people's time to be testing their scripts against every shell that claims to be sh-compatible, especially at a place like Google where the scripts are almost all being written to run on completely Google-controlled environments.
But I didn't think about the bug vs. by-design issue and how that is remedied by a well-defined standard. That does seem to be a factor in favor of using a standardized language. But it doesn’t seem different in principle than depending on any software. It is liable to change and leave you with a difficult decision of forking the library or modifying your code that consumes it. Given bash’s ubiquity, it seems reasonable to expect a good level of stability.
If you only target standards, the lowest-common-denominator of functionality, you will be poorer for it. Linux, for example, has all sorts of performance and functionality enhancing features over what POSIX provides.
But I didn't think about the bug vs. by-design issue and how that is remedied by a well-defined standard. That does seem to be a factor in favor of using a standardized language. But it doesn’t seem different in principle than depending on any software. It is liable to change and leave you with a difficult decision of forking the library or modifying your code that consumes it. Given bash’s ubiquity, it seems reasonable to expect a good level of stability.
If you only target standards, the lowest-common-denominator of functionality, you will be poorer for it. Linux, for example, has all sorts of performance and functionality enhancing features over what POSIX provides.
>If you only target standards, the lowest-common-denominator of functionality, you will be poorer for it.
In this case, you won't. I addressed this a few comments ago:
>By the time you need the things bash offers you might as well not use a shell script at all
In this case, you won't. I addressed this a few comments ago:
>By the time you need the things bash offers you might as well not use a shell script at all
> competition between gcc and clang is harmful to the C and C++ ecosystems?
Yes. GCC will die eventually.
Yes. GCC will die eventually.
Maybe one day there will be another cool and dominating shell. That shell is more likely to be POSIX compliant than bash compliant.
All the cool new shells I've seen are either completely POSIX incompatible (powershell, xonsh, etc.) or they try to be bash supersets (zsh, osh, etc.).
Bash is a de facto modern standard.
The only new purely POSIX shells I know are for embedded stuff (dash) and that space is neither fast moving nor in dire need of a better shell, since most modern shell features target interactive use. Or they make coding easier and/or safer, in which case we're back to point 1, with the POSIX incompatibility.
Bash is a de facto modern standard.
The only new purely POSIX shells I know are for embedded stuff (dash) and that space is neither fast moving nor in dire need of a better shell, since most modern shell features target interactive use. Or they make coding easier and/or safer, in which case we're back to point 1, with the POSIX incompatibility.
Maybe the missing POSIX compatibility is the reason why they haven't replaced bash yet.
Nothing will replace bash within my lifetime. Zsh didn't even though it was superior for over a decade.
Heck, Microsoft barely managed to supplant cmd 1-2 years ago, and that's in a non-standard, proprietary environment by what is arguably the biggest commercial platform maker in the world.
Heck, Microsoft barely managed to supplant cmd 1-2 years ago, and that's in a non-standard, proprietary environment by what is arguably the biggest commercial platform maker in the world.
[deleted]
None of this rant actually says why we should use POSIX shell over anything else though.
Portability doesn’t make sense for an operation like Google. Even if some new shell promises compatibility, you still have to test every single script against it. Why bother? Write scripts in bash, run them in bash. In the future, switch to whatever you want for new scripts.
Only if you need portability. If you don't need portability, write it in whatever you want. I prefer bash because it's everywhere. Zsh is too esoteric for me, but I wouldn't judge anyone for using it.
You're getting voted down (likely for only saying the article is wrong but not why), but I'm curious your reasoning. I've tended to write in POSIX sh (for maximum compatibility - and because it was just the way I learned) - but am happy to use bash too when the project allows and it makes sense so not sure I get the hard line stance. Wondering what I might be missing to evoke such a response.
I wrote in some greater detail in response to someone else:
https://news.ycombinator.com/item?id=17073703
https://news.ycombinator.com/item?id=17073703
Nobody cares about non-bash shells if they're not dealing with legacy systems. (And if you are, you have bigger problems)
Wrong. *BSD are not legacy systems, and many Linux installs don't have Bash. On top of that, new systems can't really implement Bash (because bash is defined by the implementation) but could easily implement POSIX sh (because POSIX sh is defined by the standard).
Agreed with BSD, but they usually have bash. (And you usually can compile it for your legacy application - I have done this)
Though I agree that if you're doing something bash specific you're most likely doing something that would be done better using Python/Perl/Ruby, etc
Though I agree that if you're doing something bash specific you're most likely doing something that would be done better using Python/Perl/Ruby, etc
Default *BSD installs do not have bash, it needs to be installed (IDK what DragonFly and TrueOS do, but FreeBSD, NetBSD and OpenBSD don't ship with it). They all have it in the ports trees though. Many distros use BusyBox (IIRC Alpine is one). And AFAIK Debian base only has dash at /bin/sh.
Preferably the complexity in a shell script should be mostly handled by the programs invoked in it. All it has to provide is variables, basic control flow, subshells, and maybe some globbing.
Preferably the complexity in a shell script should be mostly handled by the programs invoked in it. All it has to provide is variables, basic control flow, subshells, and maybe some globbing.
> Do not write scripts with bash.
It's a bit too late for that. Bash is pretty much the de facto default shell for linux. Telling people not to use bash scripts is like telling web devs not to use javascript or OS devs not to use C.
It may change in the future, but bash is so entrenched, it's going to take an extraordinary use case for people to move from bash.
It's a bit too late for that. Bash is pretty much the de facto default shell for linux. Telling people not to use bash scripts is like telling web devs not to use javascript or OS devs not to use C.
It may change in the future, but bash is so entrenched, it's going to take an extraordinary use case for people to move from bash.
Bash is nowhere close for being de facto default shell for Linux. This is a bad analogy. In this case it is more like telling people that want to code Javascript to only write it in React. You got to go from something generic to something specific. Scripts written with bash will often rely on bash-specifics and can therefore only be interpreted by bash. There are entire projects dedicated to stop this plague. If you want to write portable shell scripts, use dash.
https://linux.die.net/man/1/checkbashisms https://mywiki.wooledge.org/Bashism https://wiki.ubuntu.com/DashAsBinSh https://en.wiktionary.org/wiki/bashism
https://linux.die.net/man/1/checkbashisms https://mywiki.wooledge.org/Bashism https://wiki.ubuntu.com/DashAsBinSh https://en.wiktionary.org/wiki/bashism
> Bash is nowhere close for being de facto default shell for Linux.
Oh, yes it is. Every important distribution uses bash as the default.
Oh, yes it is. Every important distribution uses bash as the default.
Ubuntu has dash as /bin/sh.
> It may change in the future, but bash is so entrenched, it's going to take an extraordinary use case for people to move from bash.
You don't know your history. Exactly such a move has already happened once, a decade ago. And it was a success.
* https://lists.debian.org/debian-release/2007/07/msg00027.htm...
* http://initscripts-ng.alioth.debian.org/soc2006-bootsystem/b...
* https://wiki.debian.org/BootProcessSpeedup#Using_a_faster_sy...
You don't know your history. Exactly such a move has already happened once, a decade ago. And it was a success.
* https://lists.debian.org/debian-release/2007/07/msg00027.htm...
* http://initscripts-ng.alioth.debian.org/soc2006-bootsystem/b...
* https://wiki.debian.org/BootProcessSpeedup#Using_a_faster_sy...
bash is completely out of date on macOS (GPL2 vs. GPL3), and not shipped on any *BSD AFAICT.
Some Linux distro don't ship it either (void, alpine IIRC).
And, most of the time, bash is used because people don't know any better, see e.g. https://github.com/OpenRA/OpenRA/pull/8405/files
Some Linux distro don't ship it either (void, alpine IIRC).
And, most of the time, bash is used because people don't know any better, see e.g. https://github.com/OpenRA/OpenRA/pull/8405/files
> bash is completely out of date on macOS
I love finding and removing all the GNU/Linux-ism in my bash code when I move it from a dev host to my personal laptop. macOS coreutils don't have GNU longopts, surprise!
I love finding and removing all the GNU/Linux-ism in my bash code when I move it from a dev host to my personal laptop. macOS coreutils don't have GNU longopts, surprise!
In case you didn't know: if you really need some GNU tool, you can do `brew install coreutils` (assuming you use Homebrew).
The installed binaries will have their names prefixed with `g`; e.g. `gdate` instead of `date`.
If something you want isn't part of coreutils, you can often install it directly; e.g. `brew install gawk` or the weirdly-named `brew install gnu-sed` (which names its installed binary `gsed`... go figure.)
The installed binaries will have their names prefixed with `g`; e.g. `gdate` instead of `date`.
If something you want isn't part of coreutils, you can often install it directly; e.g. `brew install gawk` or the weirdly-named `brew install gnu-sed` (which names its installed binary `gsed`... go figure.)
I know this; but can I teach every potential user of my scripts this same fact through documentation I have to maintain? What if: I write the code so that documenting it is easier?
[deleted]
Quite some people use Zsh interactively.
>Telling people not to use bash scripts is like telling web devs not to use javascript or OS devs not to use C.
Wow, this comparison is totally whack. Javascript and C are both standardized and the former is the only option when using a web browser.
It's never too late. We managed to get people off of ANSI C and there's a similar amount of difference between ANSI* C and C89 as there is between bash and sh.
* correction: pre-ANSI
Wow, this comparison is totally whack. Javascript and C are both standardized and the former is the only option when using a web browser.
It's never too late. We managed to get people off of ANSI C and there's a similar amount of difference between ANSI* C and C89 as there is between bash and sh.
* correction: pre-ANSI
thankthunk(1)
Truly portable shell scripts cannot define functions. Have you ever seen autotools' output? No thanks, I'll be impure and wrong and get the job done. Also your comment is off-topic... not sure why you see every thread about Bash as the right place to start this pedantic flameskirmish.
Of course portable shell scripts can define functions. What are you talking about? Shell functions are defined in POSIX. Maybe you should consult the standard before you proceed:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...
I'm technically wrong per POSIX but practically right per your principle motivation of portability: https://www.gnu.org/software/autoconf/manual/autoconf-2.66/h...
> Unfortunately, even in 2008, where shells without any function support are far and few between, there are pitfalls to avoid when making use of them. Also, finding a Bourne shell that accepts shell functions is not trivial, even though there is almost always one on interesting porting targets.
> Unfortunately, even in 2008, where shells without any function support are far and few between, there are pitfalls to avoid when making use of them. Also, finding a Bourne shell that accepts shell functions is not trivial, even though there is almost always one on interesting porting targets.
This document is 10 years old, and even then it acknowledged that finding shells without function support is a tough ask. You're not practically right, either.
When a system does not (correctly) implement portable standards, it is a bug in the system and software should not be corrected to accomodate for it. autotools disagrees with me on this point.
When a system does not (correctly) implement portable standards, it is a bug in the system and software should not be corrected to accomodate for it. autotools disagrees with me on this point.
Yea, I'm going to side with autotools' maintainers on matters of software portability. And because I'm not aiming for autotools-levels of portability, I just write v3-compatible Bash, to help out my GPL2 OSX friends.
I don't see how the comment is off-topic. While the HN title is "Google's bash style guide", the actual guide has the title "Shell Style Guide". To some, that reads as if a "Programming Style Guide" started by saying only C is allowed.
These are programming languages. Like all programming languages one must understand how to deploy them and use them properly. Yes bash has faults absolutely. It can be very arcane and esoteric and I think this due to the fact that it’s creators are very much of the old 1970s and 1980s Unix mindset and largely bash tools still feel like they work the way they did back then
Python is also a tool. I have written scripts in python to manage processes do forking etc and it too has some warts. It is also used by companies tohat deploy thousands of lines of code or more. Yes it isn’t statically typed and yes you can get into a box but guys no single language is objectively superior to another.
So learn them like you would any other tool and move on.