Please stop writing shell scripts(pythonspeed.com)
pythonspeed.com
Please stop writing shell scripts
https://pythonspeed.com/articles/shell-scripts/
60 comments
Defaults matter. This came up in a search:
https://www.jason.af/defaults-matter/
"Even the most well-intentioned people have a finite amount of energy to fight against a system that pushes toward bad outcomes."
https://www.jason.af/defaults-matter/
"Even the most well-intentioned people have a finite amount of energy to fight against a system that pushes toward bad outcomes."
> Y'all writing bash scripts without set -u and error checking?
Just last week someone pointed to all of the cases where -u doesn't do what one would expect.
Just last week someone pointed to all of the cases where -u doesn't do what one would expect.
> Y'all writing bash scripts without set -u
Yes. I can account for unset variables.
> and error checking?
No. Everything which can error is checked and handled.
Yes. I can account for unset variables.
> and error checking?
No. Everything which can error is checked and handled.
Try:
set -euxo pipefail
Enjoy!
set -euxo pipefail
Enjoy!
What does the pipefail part do?
Say you have `command1 | command2` in your script, and command1 fails. Bash will just skip over the failure and continue.
`set -o pipefail` tells Bash to exit the program on such a failure.
`set -o pipefail` tells Bash to exit the program on such a failure.
I've seen five line bash scripts replaced with thousand line python programs. In an extreme case a pipeline typed directly into slack was productized over weeks into some hundreds of lines of python that did the same thing. I'm going to keep using bash for trivial things.
I routinely replace such 1000 line Python scripts with five lines of bash. Usually, it's in the middle of some production incident, and the script is written for Python that's too new / old for the production machine, or pip does what pip does, or whatever.
Protip: These monstrosities usually end up accidentally invoking bash at the bottom, so just copy paste what it would pass to bash and reimplement the argv parsing.
Protip: These monstrosities usually end up accidentally invoking bash at the bottom, so just copy paste what it would pass to bash and reimplement the argv parsing.
And I've replaced several lines of unreadable, dangerous bash (see article) with a Python expression. Especially anything with string-manipulation, which is typically an atrocity in shell or batch languages. Shall we go on?
That's irrelevant though. The above posters aren't saying "stop using Python." They are defending the use of shell scripts. Both are tools and both have appropriate times where they should be used.
There should be, could be, possibly are safer shells which would be appropriate. Not bash in its present form, for non-trivial problems.
I bet you most bash scripts that don't operate on fixed file names have security vulnerabilities.
Most bash scripts aren't setuid root, so this is mostly a moot point.
None are, afaik. Linux flatly refused to let me elevate a script to run with root permissions.
I ended up porting said script to c++ just so I could have ./gpu_recover work from a test suite which I'm not going to run as root or type passwords into.
I ended up porting said script to c++ just so I could have ./gpu_recover work from a test suite which I'm not going to run as root or type passwords into.
I don't think local privilege escalation worth worrying in most cases. It is remote code execution that you should beware. A .ZIP archive containing files with funny file names can go a long way in achieving RCE when handled by a shell script without proper input sanitation (which is exactly what I was referring to).
This is not a BASH problem. Ok, well this example is, but more generally it is an example of an "untyped" vs. "typed" problem.
Types are the way to write software that has longevity. Even better, while writing software that is typed, learning of type errors sooner is better than learning about them later, and learning about them at runtime when your tools tell you everything is OK is the worst of all.
Statically typechecked languages win the day, here. BASH has value, Python has value, JavaScript has value, and typed languages all have more value, more longevity, and more provability than any untyped or dynamically typechecked languages.
Please use statically typechecked, compiled languages when writing load-bearing software.
/soapbox
Types are the way to write software that has longevity. Even better, while writing software that is typed, learning of type errors sooner is better than learning about them later, and learning about them at runtime when your tools tell you everything is OK is the worst of all.
Statically typechecked languages win the day, here. BASH has value, Python has value, JavaScript has value, and typed languages all have more value, more longevity, and more provability than any untyped or dynamically typechecked languages.
Please use statically typechecked, compiled languages when writing load-bearing software.
/soapbox
Sadly, alas -- not only are there plenty of ways that code written in typechecked/compiled languages can (despite these benefits) easily begin the irreversible slide into becoming unmaintainable (or otherwise high-maintenance) slop -- but many of the more popular choices for such languages seem almost expressly designed to facilitate this slide.
Which also presents a risk to your "load-bearing" software (whatever that means).
We get the theory and correctness parts. But in the trenches and on the ground, there are other factors to consider.
Which also presents a risk to your "load-bearing" software (whatever that means).
We get the theory and correctness parts. But in the trenches and on the ground, there are other factors to consider.
I'm on the ground, too, you know. fighting different battles, I'm sure, but I'm fighting, same as everyone else.
except I have something of a superpower that not everyone has: two and a half decades of hindsight.
my experience tells me, without question, that smart development practices (which are never ever what you hear among the technological fetishists in this field, who chase newness and do not chase improvements) are almost always the ugliest and least structured up-front, and the most pleasant projects of all time by the time those projects are mature.
there is no rule to guide you, other than to simply write what you need right now, and no more. when you need more, write more. Do not try to predict what you will need, because today is the day you are the least experienced you will be from this point onwards. you will be a better developer tomorrow. so write today only what you need today. tomorrow you will know more, need more, and you can add more, and fix what was already written by the less experienced you.
there is no framework, no paradigm, no approach of any kind that creates good software (of any size) that does not evolve as the project itself evolves and as the developers evolve. none.
and finally, the curse: when you're at an age no one listens to you, you will realize this as well.
except I have something of a superpower that not everyone has: two and a half decades of hindsight.
my experience tells me, without question, that smart development practices (which are never ever what you hear among the technological fetishists in this field, who chase newness and do not chase improvements) are almost always the ugliest and least structured up-front, and the most pleasant projects of all time by the time those projects are mature.
there is no rule to guide you, other than to simply write what you need right now, and no more. when you need more, write more. Do not try to predict what you will need, because today is the day you are the least experienced you will be from this point onwards. you will be a better developer tomorrow. so write today only what you need today. tomorrow you will know more, need more, and you can add more, and fix what was already written by the less experienced you.
there is no framework, no paradigm, no approach of any kind that creates good software (of any size) that does not evolve as the project itself evolves and as the developers evolve. none.
and finally, the curse: when you're at an age no one listens to you, you will realize this as well.
Totally agree with you there (except for the 3 percent of the time when actually we do want to apply predictive branching, as it were, and brace ourselves for tomorrow's need rather than simply today's. That is arguably how the future gets built, in fact. But that there are outlier cases doesn't invalidate the main point of the principle you cite, of course).
The rest -- frameworks, methodologies -- are window dressing (or at best training wheels), mostly.
The rest -- frameworks, methodologies -- are window dressing (or at best training wheels), mostly.
> Please use statically typechecked, compiled languages when writing load-bearing software.
load-bearing ... gone steal that phrase. excellent.
load-bearing ... gone steal that phrase. excellent.
I love how the author airily mentions that python will be on your machine.
Some python, maybe, but which one? What a nightmare.
Shell scripts are extremely back compatible. And if it's more than a few lines I'll just, you know, write a program.
Some python, maybe, but which one? What a nightmare.
Shell scripts are extremely back compatible. And if it's more than a few lines I'll just, you know, write a program.
I have found that writing high quality bash scripts correlates with general competence. All of the issues mentioned can be mitigated once you actually take the time to learn how to use your tools. Instead, I read this as excuses for not learning how to code well, instead expecting the tools and languages to do the job that the developer should be doing.
Famous last words. Allowing the possibility of errors in growing, complicated systems built by meat-bags is just short of guaranteeing them.
Are you trying to imply that this is a way to grow a bug-free complicated system in another language? In truth, incompetence transcends language. Developers are going to write bugs until they learn how to avoid them. In my experience, such competence is largely independent of the language or tools.
Sorry, this Slashdot '97 attitude is obsolete, the world has moved on. It's about safer, more robust languages now, supplemented by tests and tools, and getting a good night's sleep. Prevention is simply a lot cheaper and more comfortable than eternal vigilance.
This sounds sort of like a combination of Murphy's Law and the somewhat lesser-known Gall's Law[1].
[1] https://personalmba.com/galls-law/
[1] https://personalmba.com/galls-law/
So, the article describes reasons why bash is not good for shell scripts. Then describes some ways how to avoid them, using bash. Which kinda invalidates the idea that bash is a bad tool for scripting. You just have to know how to use it.
For me, I use both. For simple stuff I use bash. For more complex things I use a higher-level language. Like when I have really complex logic, structured data (I wish a new Linux shell would come up that can send objects through a pipe in a standard format like powershell can). Or when I need to do extensive chatting with a database or REST API or something.
Sometimes I use PHP (even locally), sometimes Python. On a Mac, PHP comes out of the box with some nice extensions that Python doesn/didn't offer out of the box on the same platform. So it was easy for me to do scripts talking to e.g. LDAP or some other things. Of course you can pull extensions in with pip but in my work we have a pain-in-the-ass MITM proxy that breaks most of that if it's not set up correctly. So I tend to rely on pushing scripts that need nothing additional to run but the script itself. Also, pulling stuff in from public repos is a bad idea for security.
But for more simple stuff especially stuff that just ends up running other programs anyway, really there's no point. Otherwise you'll just end up with a row of shell() (PHP) or os.system (python) commands.
For me, I use both. For simple stuff I use bash. For more complex things I use a higher-level language. Like when I have really complex logic, structured data (I wish a new Linux shell would come up that can send objects through a pipe in a standard format like powershell can). Or when I need to do extensive chatting with a database or REST API or something.
Sometimes I use PHP (even locally), sometimes Python. On a Mac, PHP comes out of the box with some nice extensions that Python doesn/didn't offer out of the box on the same platform. So it was easy for me to do scripts talking to e.g. LDAP or some other things. Of course you can pull extensions in with pip but in my work we have a pain-in-the-ass MITM proxy that breaks most of that if it's not set up correctly. So I tend to rely on pushing scripts that need nothing additional to run but the script itself. Also, pulling stuff in from public repos is a bad idea for security.
But for more simple stuff especially stuff that just ends up running other programs anyway, really there's no point. Otherwise you'll just end up with a row of shell() (PHP) or os.system (python) commands.
The industry has been moving steadily for many decades from unsafe technology to safer variants to increase reliability of increasingly mission critical systems. Typescript anyone?
This has been in response to the many high-profile failures that have been and are experienced routinely in the IT industry.
This has been in response to the many high-profile failures that have been and are experienced routinely in the IT industry.
Statically typed bash (whatever that means) would be awesome.
Statically typed python would probably defeat the purpose of the language (just use java, etc.)
Statically typed python would probably defeat the purpose of the language (just use java, etc.)
Ugh no not java please.
I'm so glad we're not having to deal with different components of major server systems requiring different JRE versions because backwards compatibility in Java is so absolutely horribly broken. And of course those different versions you installed in different places try to 'helpfully' find and upgrade each other causing the various components to break.
Java was a nice idea with very poor execution. It's been surpassed by many other technologies now, even .NET does it better.
I'm so glad we're not having to deal with different components of major server systems requiring different JRE versions because backwards compatibility in Java is so absolutely horribly broken. And of course those different versions you installed in different places try to 'helpfully' find and upgrade each other causing the various components to break.
Java was a nice idea with very poor execution. It's been surpassed by many other technologies now, even .NET does it better.
It's great when your prototype grew into a giant system with large teams of developers with mixed-experience. See pydantic etc.
Just in case you weren't aware, PowerShell runs on macOS and Linux now :)
No, I won't. Piping is a nice way to compose programs. I like that, just use shell check and proper settings.
I usually go to python or some other scripting language when I have a lot of string processing to do. For the rest something with a more advanced type system. Scala, rust and the like.
I usually go to python or some other scripting language when I have a lot of string processing to do. For the rest something with a more advanced type system. Scala, rust and the like.
It's a process that's all too familiar to me - a script starts as a few lines of bash and very quickly turns into a few hundred lines of bash before being replaced by a few lines of Python. For some reason in my head I think "this is so simple, Python would be overkill" when in fact it takes more effort (for me) to do anything non-trivial in bash.
The real benefit of bash for me is that it makes it easy to invoke and bind together the many excellent tools we all know and love such as grep, sed, awk, curl, wget and so on. Chances are those tools can do their thing better than my hacked-together Python scripts can, so as long as I leave the heavy lifting to them and use bash only to shuffle text between them, a bash script can work well.
It's true bash has a lot of gotchas, but it's just a matter of knowing them. As an alternative to `set -e` I tend to use `somecmd && echo "Success" || echo "Fail"`, and I usually explicitly check whether a variable is non-empty, as an alternative to `set -u`. Interested to hear whether these approaches have any major drawbacks as against `set -eu`.
The real benefit of bash for me is that it makes it easy to invoke and bind together the many excellent tools we all know and love such as grep, sed, awk, curl, wget and so on. Chances are those tools can do their thing better than my hacked-together Python scripts can, so as long as I leave the heavy lifting to them and use bash only to shuffle text between them, a bash script can work well.
It's true bash has a lot of gotchas, but it's just a matter of knowing them. As an alternative to `set -e` I tend to use `somecmd && echo "Success" || echo "Fail"`, and I usually explicitly check whether a variable is non-empty, as an alternative to `set -u`. Interested to hear whether these approaches have any major drawbacks as against `set -eu`.
Yeah, bash is ugly.
PowerShell however is my go to language for integrations. Its cross-platform now, can do anything higher language can (although that is not the point), any script is way shorter then equivalent [insert-non-shell-language-here] program, and you don't need any toolset, just any editor. On any Windows its there OTB, while its one package manager command away from any Linux et. all systems, you have nice debugger in vs code etc.
I cringe when I have to leave that beauty because I must use 'serious' language. It happens only when I need great performance. Number of integrative components far exceed number of any other components so I am very grateful to have this beast on my side all the time. From invoking REST API functions (Redmine, GitLab/Hub, Grafana, Rundeck) or SOAP functions, over complete deployments and IaC, automatic tests in Pester, build routines via Invoke-Build, to data analytics and processing with ConvertTo/From-JSON,CSV,XML,YAML, to parallel task invocations via multiple processes or threads etc... its all there and it all obeys clear standard. Epic.
If you don't love it, you didn't grok it.
PowerShell however is my go to language for integrations. Its cross-platform now, can do anything higher language can (although that is not the point), any script is way shorter then equivalent [insert-non-shell-language-here] program, and you don't need any toolset, just any editor. On any Windows its there OTB, while its one package manager command away from any Linux et. all systems, you have nice debugger in vs code etc.
I cringe when I have to leave that beauty because I must use 'serious' language. It happens only when I need great performance. Number of integrative components far exceed number of any other components so I am very grateful to have this beast on my side all the time. From invoking REST API functions (Redmine, GitLab/Hub, Grafana, Rundeck) or SOAP functions, over complete deployments and IaC, automatic tests in Pester, build routines via Invoke-Build, to data analytics and processing with ConvertTo/From-JSON,CSV,XML,YAML, to parallel task invocations via multiple processes or threads etc... its all there and it all obeys clear standard. Epic.
If you don't love it, you didn't grok it.
PowerShell has similar issues, see the various incantations you need to invoke in order to avoid brokenness.
This article seems to have been started from the position of "I want `set -euo pipefail`" and the points are just geared towards it.
And those options have big issues!
`-e` ("errexit") can just kill a script for no obvious reason, and doesn't even give a good error to figure out what happened.
`-o pipefail` makes it worse by causing pipelines to sometimes fail depending on how much input was buffered - the classic example is `something | head -n 1`. It also doesn't give a proper error, and with errexit this kills the entire script.
`-u` is probably okay, but also triggers for `$1` and friends, so it basically requires you to write `${foo-}` everywhere. And if you've done that, you can still misspell your variable name.
See http://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail
And those options have big issues!
`-e` ("errexit") can just kill a script for no obvious reason, and doesn't even give a good error to figure out what happened.
`-o pipefail` makes it worse by causing pipelines to sometimes fail depending on how much input was buffered - the classic example is `something | head -n 1`. It also doesn't give a proper error, and with errexit this kills the entire script.
`-u` is probably okay, but also triggers for `$1` and friends, so it basically requires you to write `${foo-}` everywhere. And if you've done that, you can still misspell your variable name.
See http://mywiki.wooledge.org/BashPitfalls#set_-euo_pipefail
And, -x gives you a trace of what the script did before exiting, mostly solving the debugging issues you listed.
I write most of my CLI tools in Rust. I don't care for learning "good bash" like many of these comments are suggesting is the solution. It is not the solution.
You most likely work in a team of people. Expecting everyone to understand the nuances of bash and not get them wrong is wishful thinking.
Some people say that bash is portable, and maybe they're kinda right, but the commands you use like sed and awk might have different args supported. Have fun supporting Linux/mac (and Windows).
I use Rust because it most likely works if it compiles. The easiest way to right functions is usually the correct way to proper error handling etc. Also its what we use for our services anyway so people know it. The only issue is binary size, but that's not a huge issue, we either just ship them in docker images or just store them in github releases where we can download them in CI
You most likely work in a team of people. Expecting everyone to understand the nuances of bash and not get them wrong is wishful thinking.
Some people say that bash is portable, and maybe they're kinda right, but the commands you use like sed and awk might have different args supported. Have fun supporting Linux/mac (and Windows).
I use Rust because it most likely works if it compiles. The easiest way to right functions is usually the correct way to proper error handling etc. Also its what we use for our services anyway so people know it. The only issue is binary size, but that's not a huge issue, we either just ship them in docker images or just store them in github releases where we can download them in CI
> I don't care for learning "good bash"
What language do you use for administering your own systems and automating your own tasks?
In my experience, people who oppose shell scripting spend little if any time administering their own systems (because they have basic requirements?). Your comment, which includes multiple references to your co-workers and zero references to your own systems or tasks, is an example of what I've observed.
What language do you use for administering your own systems and automating your own tasks?
In my experience, people who oppose shell scripting spend little if any time administering their own systems (because they have basic requirements?). Your comment, which includes multiple references to your co-workers and zero references to your own systems or tasks, is an example of what I've observed.
True, I'm not a sys-admin. But I do monitor a few k8s clusters.
Most automated tasks I will write jobs for in Rust (as mentioned, it's a language I and others know well). That's what the rest our company tends to agree with.
I don't know anyone in our tech stack that makes grand use of bash to admin our systems. We're very strict that everything is in K8s. No ssh access to nodes.
Only really CI uses bash and I'd rather it didn't
Most automated tasks I will write jobs for in Rust (as mentioned, it's a language I and others know well). That's what the rest our company tends to agree with.
I don't know anyone in our tech stack that makes grand use of bash to admin our systems. We're very strict that everything is in K8s. No ssh access to nodes.
Only really CI uses bash and I'd rather it didn't
At least with a shell-script, you're unlikely to get SHOUTY DEPRECATION WARNINGS that you can't turn off.
I totally agree with not writing shell scripts.
As far as I can tell this person is trying to sell me python scripts. My personal take is that I don't want that either, for almost exactly the same set of reasons. Just write the darn thing in whatever your real, main project language is (java, go, c++, whatever). I wouldn't use python just for scripts unless all my stuff was in python, I think code interop and working with all the tooling you've already set up is too valuable.
As far as I can tell this person is trying to sell me python scripts. My personal take is that I don't want that either, for almost exactly the same set of reasons. Just write the darn thing in whatever your real, main project language is (java, go, c++, whatever). I wouldn't use python just for scripts unless all my stuff was in python, I think code interop and working with all the tooling you've already set up is too valuable.
Or maybe we should START writing shell scripts in Windows environments now that it is supported natively
I find shell scripts and one liners very accessible and intuitive, while I basically can't do anything with a real programming language besides calculating numbers or printing text to the screen.
"please stop writing shell scripts" - no
A full-featured BDD unit testing framework for bash, ksh, zsh, dash and all POSIX shells
https://github.com/shellspec/shellspec
https://github.com/shellspec/shellspec
right, guess I'll go back to batch files.
If one can imagine a thing it probably exists [1]
[1] - https://www.npmjs.com/package/bash-converter
[1] - https://www.npmjs.com/package/bash-converter
So what should I use instead? Python? No, I write shell scripts because shell is everywhere. I might not know everything about writing bulletproof stuff but I do generally try to target posix shell.
This post reads like a preachy Reddit post. Don’t tell me what to do!
Has anyone here had any experience with using Google zx?
Dear dang, does this qualify as a flamebait post?
[deleted]
One of the first technical docs I read online after discovering Linux was a best practices guide for bash scripts (or some similar version of these if my memory is lapsing)
http://mywiki.wooledge.org/BashGuide/Practices
http://mywiki.wooledge.org/BashPitfalls