Writing Safe Shell Scripts (2019)(sipb.mit.edu)
sipb.mit.edu
Writing Safe Shell Scripts (2019)
https://sipb.mit.edu/doc/safe-shell/
165 comments
Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing.
Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed).
1: https://www.oilshell.org/
Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed).
1: https://www.oilshell.org/
FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :)
Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left.
So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squigglies, they will say "fix this".
The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly.
In other words, it was the typical "static analysis has annoying false positives" issue.
Most of the shell scripts I write deal with "trusted" filenames -- i.e. those checked into the repo. If you want to deal with untrusted filenames, then there are a lot of ugly techniques to do that.
So Oil is basically meant to make the default thing the right thing, and the common thing the short thing (Huffman coding, as Larry Wall calls it).
Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left.
So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squigglies, they will say "fix this".
The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly.
In other words, it was the typical "static analysis has annoying false positives" issue.
Most of the shell scripts I write deal with "trusted" filenames -- i.e. those checked into the repo. If you want to deal with untrusted filenames, then there are a lot of ugly techniques to do that.
So Oil is basically meant to make the default thing the right thing, and the common thing the short thing (Huffman coding, as Larry Wall calls it).
> The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly.
Once it becomes habit the cost is zero, though.
And the alternative you're implicitly suggesting is: "Follow shellcheck except you are allowed to make an expert judgement and ignore it when you know it won't be dangerous and have determined that the code is unlikely to change in the future so that this exception will be safe going forward too."
I think it's clear that this isn't a practical rule on a team with many people -- especially where they are not all experts in shell, which is 100% of teams of many people.
Making the experts "add those annoying double quotes they don't really need in this case" is the far lesser of the two sins.
Once it becomes habit the cost is zero, though.
And the alternative you're implicitly suggesting is: "Follow shellcheck except you are allowed to make an expert judgement and ignore it when you know it won't be dangerous and have determined that the code is unlikely to change in the future so that this exception will be safe going forward too."
I think it's clear that this isn't a practical rule on a team with many people -- especially where they are not all experts in shell, which is 100% of teams of many people.
Making the experts "add those annoying double quotes they don't really need in this case" is the far lesser of the two sins.
Yeah I don't disagree with that. If you're working in a big group with some shell scripts, and want to get stuff done, use ShellCheck.
I just find it annoying, hence the new shell :)
There's a lot more to Oil, but that's definitely one of the surface annoyances I want to fix. Although zsh actually does fix that, and for some reason I've almost never seen a zsh script.
I just find it annoying, hence the new shell :)
There's a lot more to Oil, but that's definitely one of the surface annoyances I want to fix. Although zsh actually does fix that, and for some reason I've almost never seen a zsh script.
Many oh-my-zsh plugins are just collection of functions, which I think could be considered scripts.
https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins
https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins
Yes sorry I should have said "I rarely see a batch program in zsh". zsh is customized with zsh scripts for sure, and they are very extensive.
I guess it's because zsh is not POSIX compatible by default.
I guess it's because zsh is not POSIX compatible by default.
Exactly. If one "expert" is developing only for himself he can allow himself not to bother writing a script "clearly" (i.e. failing the programs that perform source checks).
But as soon as the script is something that should be potentially sooner or later maintained by anybody else than the original author (and in any successful project that's practically sure) the only right policy is to produce the source code that's obvious and passes the source checks.
An "expert" would have no problem to in no time adjust his three-line code to fulfill all the expectations of ShellCheck.
If it takes more than that, that only proves that the checks are actually beneficial and even more important than the "expert" tells to himself: it proves that writing that specific code "cleanly" is actually hard for everybody.
But as soon as the script is something that should be potentially sooner or later maintained by anybody else than the original author (and in any successful project that's practically sure) the only right policy is to produce the source code that's obvious and passes the source checks.
An "expert" would have no problem to in no time adjust his three-line code to fulfill all the expectations of ShellCheck.
If it takes more than that, that only proves that the checks are actually beneficial and even more important than the "expert" tells to himself: it proves that writing that specific code "cleanly" is actually hard for everybody.
"The double quotes are of course technically correct, but when ... it's overkill and makes everyrthing ugly."
The only way you know this is being a person who routinely writes shell scripts.
There is no substitute for knowing what you're doing.
Try running shellcheck on configure scripts.
Or try this
These scripts are "safe" enough for hundreds or thousands of competent users to be running them every day.
The only way you know this is being a person who routinely writes shell scripts.
There is no substitute for knowing what you're doing.
Try running shellcheck on configure scripts.
Or try this
curl -4O https://ftp.netbsd.org/pub/NetBSD/NetBSD-release-8/src/build.sh
shellcheck build.sh
These scripts are "safe" enough for hundreds or thousands of competent users to be running them every day.
>all I get are false positives
double quotes, mixing string and array, unused variables. Why is this normal?
double quotes, mixing string and array, unused variables. Why is this normal?
> in the repo that only operates on 3 files in that subdir
It does today. Will it change tomorrow? Will someone rename one of the files to include a space? It sounds like people saying "I don't need to escape this value in a query, it comes from a static list and it's safe." Yes - it's safe right now...
It does today. Will it change tomorrow? Will someone rename one of the files to include a space? It sounds like people saying "I don't need to escape this value in a query, it comes from a static list and it's safe." Yes - it's safe right now...
It's a burden to you only because you're used to not putting quotes in some cases. But to some of us who always put the quotes, it's not really about that at all, and it's not a subjective issue - missing them is literally wrong. It's like sprinkling random .split(" ") calls in your Python code for no reason, which you'd (hopefully?) never do. And when something is semantically so wrong, the elegance of its syntax compared to a correct solution becomes pretty much irrelevant.
Quotes are just a tool, not a religious requisite. Chubot is talking from experience, and he is right. Putting quotes on all shell expressions is a burden, because it's more keystrokes, and more characters on the display which make the source on display disorienting. When you use shell every day, you know where the quotes are needed and where they are just a noise. Use a tool - when it's needed.
Yes, of course they're tools, but "religious requisite" is a pretty disingenuous representation of the argument I'm making here? Something being a tool doesn't mean its use or lack thereof can't be incorrect. Bikes are tools too, but you don't ride them on the highway, right?
Quotes are tools that affect the correctness of your control flow quite vividly. Of course if there are multiple tools for solving the correctness issue you're welcome to choose any of them (and maybe the choice between the tools is something you can ascribe to religion), but that doesn't absolve you of the need to solve the issue itself. When the argument is that a piece of code is executing objectively wrong instructions (and ones that would be so obviously wrong in other languages!), that's pretty much the diametrical opposite of what you can casually dismiss as religious dogma...
Quotes are tools that affect the correctness of your control flow quite vividly. Of course if there are multiple tools for solving the correctness issue you're welcome to choose any of them (and maybe the choice between the tools is something you can ascribe to religion), but that doesn't absolve you of the need to solve the issue itself. When the argument is that a piece of code is executing objectively wrong instructions (and ones that would be so obviously wrong in other languages!), that's pretty much the diametrical opposite of what you can casually dismiss as religious dogma...
Which is best for simply catching typing errors? Shell check? I never saw the instillation instructions for oil.
The idea is that with Oil there will be better solutions to the problems that ShellCheck points out.
As mentioned, the default should be right in Oil. The defaults are wrong in shell, and ShellCheck is meant to alert you of that.
Oil should have a style linter / formatter, but if it's designed correctly it shouldn't need the analogue of ShellCheck, which is more about correctness.
As mentioned, the default should be right in Oil. The defaults are wrong in shell, and ShellCheck is meant to alert you of that.
Oil should have a style linter / formatter, but if it's designed correctly it shouldn't need the analogue of ShellCheck, which is more about correctness.
I really want someone to integrate fish and oil, and call the project "fish-oil" or something along those lines.
(Oil author here) Yes I think that's a great idea, and I hope to release a "liboil" so that people can do that. Let a thousand flowers bloom, etc.
The most likely thing is that Oil is going to be a much better language this year (more expressive, with safety features, and it should be fast), but the interactive UI will still be more bash-like than fish-like:
https://www.oilshell.org/blog/2020/01/making-plans.html
So I encourage anyone who's interested in that to get involved now as the interactive shell is also a years-long effort :) There are many links on the home page about how to get involved, and I wrote many blog posts so people can understand how it works.
----
Here are a couple posts on why Oil is a good foundation for an interactive shell:
https://www.oilshell.org/blog/2020/01/history-and-completion...
https://www.oilshell.org/blog/2020/01/alias-and-prompt.html
The most likely thing is that Oil is going to be a much better language this year (more expressive, with safety features, and it should be fast), but the interactive UI will still be more bash-like than fish-like:
https://www.oilshell.org/blog/2020/01/making-plans.html
So I encourage anyone who's interested in that to get involved now as the interactive shell is also a years-long effort :) There are many links on the home page about how to get involved, and I wrote many blog posts so people can understand how it works.
----
Here are a couple posts on why Oil is a good foundation for an interactive shell:
https://www.oilshell.org/blog/2020/01/history-and-completion...
https://www.oilshell.org/blog/2020/01/alias-and-prompt.html
I want to say that I am continually impressed at your long-term vision and commitment you have taken to the Oil project. Your posts are a treat to read, and the care you have given to the problem is appreciated. We need to stop paving cow-paths, and discard historical baggage if we are ever going to build robust systems.
Thank you!
Although I have to say I do feel the tradeoff right now between having a strong design / globally consistent code vs. allowing local variation / a lot of people to contribute quickly. That is, allow people to get in, add their useful patch, their useful bit of knowledge, and get out, without caring at all about the rest of the program.
I think Linux, git, and GNU are the prototypical examples of the latter. Those projects are too big for one person, so the code is set up in a way to allow hacking locally. They're also arguably a mess. Anyone who's ever written against the container APIs in Linux knows what I mean (namespaces, cgroups, etc.)
The git UI is another famous example, i.e. the plethora of commands and flags that make little sense. Empirically it seems that you can move faster if you disregard consistency and coherence.
The shell language of course grew like that over 50 years. Bourne shell is pretty consistent, but one thing I learned is that ksh added a lot of bash misfeatures (they're ksh-isms not bash-isms), and it's pretty bad.
But shell is actually a significantly smaller problem than git (after all bash is maintained more or less by one person). So it should be possible to fix it and redesign it. Unfortunately I'm not sure if the lessons generalize to bigger projects, but I would like it if they do.
-----
On a more practical note, feel free to send more people this way if you want to see the project succeed :) The design is there, and it will hold up, but it needs a bit more manpower, especially for things like the interactive shell.
Although I have to say I do feel the tradeoff right now between having a strong design / globally consistent code vs. allowing local variation / a lot of people to contribute quickly. That is, allow people to get in, add their useful patch, their useful bit of knowledge, and get out, without caring at all about the rest of the program.
I think Linux, git, and GNU are the prototypical examples of the latter. Those projects are too big for one person, so the code is set up in a way to allow hacking locally. They're also arguably a mess. Anyone who's ever written against the container APIs in Linux knows what I mean (namespaces, cgroups, etc.)
The git UI is another famous example, i.e. the plethora of commands and flags that make little sense. Empirically it seems that you can move faster if you disregard consistency and coherence.
The shell language of course grew like that over 50 years. Bourne shell is pretty consistent, but one thing I learned is that ksh added a lot of bash misfeatures (they're ksh-isms not bash-isms), and it's pretty bad.
But shell is actually a significantly smaller problem than git (after all bash is maintained more or less by one person). So it should be possible to fix it and redesign it. Unfortunately I'm not sure if the lessons generalize to bigger projects, but I would like it if they do.
-----
On a more practical note, feel free to send more people this way if you want to see the project succeed :) The design is there, and it will hold up, but it needs a bit more manpower, especially for things like the interactive shell.
The trick is separating out external facing (api/ux) consistency from code style/data-structure/etc consistency. It doesn’t really matter if you don’t have coherency across the whole project for the latter because that can easily be swapped out later if it’s even a problem.
Most single-person project maintainers have trouble relaxing the latter and end up staying a single-person project because of it.
Most single-person project maintainers have trouble relaxing the latter and end up staying a single-person project because of it.
Couldn’t agree more! I also really enjoy shfmt, integrating both ShellCheck and shfmt had really helped me to save time and write safer, readable scripts.
https://github.com/mvdan/sh
https://github.com/mvdan/sh
Using shellshock.net can actually teach you a bunch about scripting as well
Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0 license. For many professionals your employer will often block / avoid GPL code, tools, and libraries.
Sorry, but what employers block the use of GPL tools? Most employers that write proprietary software don't want GPL code in their code, but using a tool like shellcheck to perform static analysis on your code doesn't make that software that was analyzed now subject to the GPL, so why would they block it?
Apple is strongly anti-GPL. And they’re pretty open about that.
Most companies that are anti-GPL aren’t so open about it.
Most companies that are anti-GPL aren’t so open about it.
I get that Apple doesn't want GPL code included in their software, but their engineers can still use GPL tools without fear of opening up their proprietary code. I know of plenty of companies where you can not INCLUDE GPL code in the software the company makes, but I don't know of any company that says you can not USE any GPL software to do your job. Those are two different things.
For the six month contract I did at Apple, they were violently opposed to using any GPL software in any way, shape, or form.
Just because you can do something, doesn't necessarily mean you should. That rule applies to the implementation of corporate policies regarding licensing, too.
Just because you can do something, doesn't necessarily mean you should. That rule applies to the implementation of corporate policies regarding licensing, too.
> For many professionals your employer will often block / avoid GPL code, tools, and libraries.
Indeed, which is why Linux is never found in a professional setting. /s
Indeed, which is why Linux is never found in a professional setting. /s
1) The sarcasm isn't appreciated here on HN.
2) Not everything in the FOSS world is GPL'd.
3) There are plenty of users who have this restriction in their work and it is a real restriction to be aware of when discussing tooling.
4) That's just how the code is licensed and is related to the topic at hand.
2) Not everything in the FOSS world is GPL'd.
3) There are plenty of users who have this restriction in their work and it is a real restriction to be aware of when discussing tooling.
4) That's just how the code is licensed and is related to the topic at hand.
I'm disappointed in you HN. The downvoting behavior over just discussing the facts of a codebase or industry is unacceptable and unbecoming. Your feelings of how software licensing should be does not matter here.
This is the kind of ignorance (just use python for anything it's better + and discover subprocess(): it's great) that burns me up. Not so many years ago actively insisting that python/tcl/perl were to be used instead of a 50 line shell script would have gotten you kicked out of a serious discussion. You used the power and reflection of these languages when you needed something that didn't integrate cleanly with the rest of the unix toolkit and/or required sophisticated data structures, controls and abstraction. These days I won't even write python or tcl for trivial projects (including http API work just use curl/jq and bash/(g)awk). The number of shell one liners that just get things done in 30 seconds (rather than loading 5 libraries in python to write the same code) are innumerable. This is google thought policing the historically ignorant yet again. They are terribly opposed to the unix toolkit and C pragma that prevailed for 20 years...because they believe that they know the best way to make the world safe and productive. Self serving, smug rejections that serve the bottom line and coincidentally gain mind share for their products and approaches.
Hear hear! It tears at my heart that the next generation of admins think Python is the answer to everything. It's a systemic rot of the core of *nix understanding.
I'll probably get flamed for this, but Python is an absolute fucking trainwreck of a platform. Even at its core the 2.x vs 3.x is going to break all kinds of stuff in Jan 1, 2020. Just look at the package system: cross-system portability is a mess (don't believe me? run on Arm for a week), and unfortunately because there IS a package system people think they HAVE to use it, which means: BLOAT BLOAT BLOAT your BOAT, gently down the stream!
It's kinda like what Dewalt did to miter saws. We've gone from something that could be kept perfectly square and did a great job and took up a little bit of space, to 30+kg behemoths with sliding rails and double bevels, tons of blade travel and deflection, and are a challenge to keep true and take up 3x the space and weigh a ton.
Keep your mission-critical tools simple. Period.
I'll probably get flamed for this, but Python is an absolute fucking trainwreck of a platform. Even at its core the 2.x vs 3.x is going to break all kinds of stuff in Jan 1, 2020. Just look at the package system: cross-system portability is a mess (don't believe me? run on Arm for a week), and unfortunately because there IS a package system people think they HAVE to use it, which means: BLOAT BLOAT BLOAT your BOAT, gently down the stream!
It's kinda like what Dewalt did to miter saws. We've gone from something that could be kept perfectly square and did a great job and took up a little bit of space, to 30+kg behemoths with sliding rails and double bevels, tons of blade travel and deflection, and are a challenge to keep true and take up 3x the space and weigh a ton.
Keep your mission-critical tools simple. Period.
I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.
Different tools are better at different things.
Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV.
Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The two Pythons are basically incompatible unless you're willing to put in an effort. You have to work to make Python scripts compatible with both versions, especially if you can only use built-ins, and the results are way clunkier than that 2-line shell script you're trying to replace.
Python is a terrible choice if your goal is a relatively short script that calls other programs. You can call out to other programs in Python, but it's much clunkier and more painful to maintain. And while it often doesn't matter (because neither are speedy), CPython 3 takes more than 50x more time to start than dash and 27x time to start than bash (see: https://github.com/bdrung/startup-time , which shows Python3 3.6.4 at 197.79 ms, Bash 4.4.12 at 7.31 ms, and dash 0.5.8 at 2.81 ms). That speed is helpful when you want to have a "simple script in a loop".
Python is much better than shell when you start needing more sophisticated data types, libraries, modularity, etc. But the typical use for a shell script doesn't need any of that. If you need that, shell is a terrible tool, because it's the wrong tool for the job, not because it's never useful.
If you're writing shell, use shellcheck. Once you do that, your error likelihood goes way down. Many of the reports of problems writing shell come from a time when shellcheck didn't exist.
Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV.
Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The two Pythons are basically incompatible unless you're willing to put in an effort. You have to work to make Python scripts compatible with both versions, especially if you can only use built-ins, and the results are way clunkier than that 2-line shell script you're trying to replace.
Python is a terrible choice if your goal is a relatively short script that calls other programs. You can call out to other programs in Python, but it's much clunkier and more painful to maintain. And while it often doesn't matter (because neither are speedy), CPython 3 takes more than 50x more time to start than dash and 27x time to start than bash (see: https://github.com/bdrung/startup-time , which shows Python3 3.6.4 at 197.79 ms, Bash 4.4.12 at 7.31 ms, and dash 0.5.8 at 2.81 ms). That speed is helpful when you want to have a "simple script in a loop".
Python is much better than shell when you start needing more sophisticated data types, libraries, modularity, etc. But the typical use for a shell script doesn't need any of that. If you need that, shell is a terrible tool, because it's the wrong tool for the job, not because it's never useful.
If you're writing shell, use shellcheck. Once you do that, your error likelihood goes way down. Many of the reports of problems writing shell come from a time when shellcheck didn't exist.
These are fair points.
POSIX shell being fixed forever though also means it will /never/ get "fixed".
Shell's silent-failure-by-default, implicit-by-default, principle-of-most-surprise semantics will always be there, and they are /bad/.
I have read lots of shell scripts, by many people, from the experienced UNIX greybeard to to the novice web developer, and they were almost all flawed. Pointing out the flaws, the devlopers of each level were surprised by them. It is close to impossible to write correct shell script, no matter the level of expertise, because the language offers neither semantics nor tooling that foster correctness.
Forgetting `set -e`, forgetting that it gets turned back off in subshells, quoting mistakes, pipefail semantics, you have to have made many mistakes to even /discover/ that these pitfalls exist (or use shellcheck, which will find some, but certainly not all of them).
Shell is a language that looks simple on the surface, but has 1000s of exceptions you have to master to write even minimal non-bugged scripts.
The lack of any form of reasonable data structures and strings being the prevalent data types also do not help. For example, NixOS's linker wrapper script (written in bash) was accidentally quadratic because of quadratic append-to-strings (https://github.com/NixOS/nixpkgs/issues/27609); and that was despite NixOS having many competent bash programmers. When scripts grow, these things happen.(Note that shellcheck does /not/ point out quadratic string appends; it can only find obvious mistakes and this is hard to detect in general). Fixing it was very difficult, and was not understood by many. If you have a language like Python that offers reasonable set/dict data structures that people understand and use intuitively, this is trivially avoided.
Finally, POSIX does not help much when the main thing shell does is calling other programs, and there are no guarantees on what the APIs for those programs are. For example, I have encountered many shell scripts that themselves are technically "portable", but they use arguments to e.g. `rm` that do not work on e.g. busybox, thus crashing the boot processe and resulting in hours of time lost for hundreds of people.
By not using shell, we can support that the next SOMETHING-standard will NOT mandate shell, but something better.
POSIX shell being fixed forever though also means it will /never/ get "fixed".
Shell's silent-failure-by-default, implicit-by-default, principle-of-most-surprise semantics will always be there, and they are /bad/.
I have read lots of shell scripts, by many people, from the experienced UNIX greybeard to to the novice web developer, and they were almost all flawed. Pointing out the flaws, the devlopers of each level were surprised by them. It is close to impossible to write correct shell script, no matter the level of expertise, because the language offers neither semantics nor tooling that foster correctness.
Forgetting `set -e`, forgetting that it gets turned back off in subshells, quoting mistakes, pipefail semantics, you have to have made many mistakes to even /discover/ that these pitfalls exist (or use shellcheck, which will find some, but certainly not all of them).
Shell is a language that looks simple on the surface, but has 1000s of exceptions you have to master to write even minimal non-bugged scripts.
The lack of any form of reasonable data structures and strings being the prevalent data types also do not help. For example, NixOS's linker wrapper script (written in bash) was accidentally quadratic because of quadratic append-to-strings (https://github.com/NixOS/nixpkgs/issues/27609); and that was despite NixOS having many competent bash programmers. When scripts grow, these things happen.(Note that shellcheck does /not/ point out quadratic string appends; it can only find obvious mistakes and this is hard to detect in general). Fixing it was very difficult, and was not understood by many. If you have a language like Python that offers reasonable set/dict data structures that people understand and use intuitively, this is trivially avoided.
Finally, POSIX does not help much when the main thing shell does is calling other programs, and there are no guarantees on what the APIs for those programs are. For example, I have encountered many shell scripts that themselves are technically "portable", but they use arguments to e.g. `rm` that do not work on e.g. busybox, thus crashing the boot processe and resulting in hours of time lost for hundreds of people.
By not using shell, we can support that the next SOMETHING-standard will NOT mandate shell, but something better.
Compared to a proper language, shellscripts work for 80% of inputs with 20% of the code.
This.
A Python script is not the answer when I'm trying to do something quick and dirty at the shell.
The beauty of shell scripting is that it evolves seamlessly from trying to solve simple problems at my command prompt. I pipe a couple of things together, and then I realize I could use a loop and a few conditionals, and suddenly it makes sense to store this in file in case I want to do this again. Boom, program done.
It now works on every computer I own.
I love Python. I use it almost every day. But I'll be damned if I'm going to go rewrite every shell snippet I hack together into Python just because it exists.
Different problems, different solutions.
A Python script is not the answer when I'm trying to do something quick and dirty at the shell.
The beauty of shell scripting is that it evolves seamlessly from trying to solve simple problems at my command prompt. I pipe a couple of things together, and then I realize I could use a loop and a few conditionals, and suddenly it makes sense to store this in file in case I want to do this again. Boom, program done.
It now works on every computer I own.
I love Python. I use it almost every day. But I'll be damned if I'm going to go rewrite every shell snippet I hack together into Python just because it exists.
Different problems, different solutions.
Your observation about the seamless evolution of a basic shell script from a few lines typed interactively is very insightful.
This kind of automation adds a lot of value for low marginal effort and probably explains a lot of the short scripts I have laying around in directories.
This kind of automation adds a lot of value for low marginal effort and probably explains a lot of the short scripts I have laying around in directories.
True that. But everyone should keep in mind that script bloat is real. If you're not careful you'll end up like me, and accidentally have to maintain what is essentially npm but written entirely in bash.
I've primarily used Python 2 for 10+ years and I often find cases where shell scripts are preferable.
The major differentiator is usually "shelling out" in Python kind of sucks. It's verbose, output collection and error handling suck, and escaping can be miserable. I often will reimplement things in pure Python if I have the time.
A recent example was I needed to tar+split large files. `tar cf - -C / $filename | split --bytes ${size}MB --verbose - $tmpfile.` My pure-Python implementation used the Tarfile library and I wrote a custom file-like object to split it--at least 50 lines. Both methods have pros/cons, but I had both implementations handy depending on the context I needed it in.
Another recent example was something I wrote to merge multiple video file "parts" into a single video using ffmpeg. It's 4 lines of shell script and would at least be 3x as long to write in Python.
The rule of thumb I have is if it's longer than 20 lines reevaluate if a shell scripting language is appropriate and I haven't really seen a need to change that in ~15 years I've used it.
The major differentiator is usually "shelling out" in Python kind of sucks. It's verbose, output collection and error handling suck, and escaping can be miserable. I often will reimplement things in pure Python if I have the time.
A recent example was I needed to tar+split large files. `tar cf - -C / $filename | split --bytes ${size}MB --verbose - $tmpfile.` My pure-Python implementation used the Tarfile library and I wrote a custom file-like object to split it--at least 50 lines. Both methods have pros/cons, but I had both implementations handy depending on the context I needed it in.
Another recent example was something I wrote to merge multiple video file "parts" into a single video using ffmpeg. It's 4 lines of shell script and would at least be 3x as long to write in Python.
The rule of thumb I have is if it's longer than 20 lines reevaluate if a shell scripting language is appropriate and I haven't really seen a need to change that in ~15 years I've used it.
I agree, but this lib does make it easier: https://github.com/amoffat/sh
Yeah, most of that comes from the verbose process needed to invoke a process, right? That's something I noticed when going back and forth between PowerShell and C# - that if C# had clean support for invoking a process and collecting the results as an IEnumerable like PowerShell does, PS wouldn't really need to exist, since 90% of the time you're dropping into C#/.net objects to get anything done anyways.
I actually invested some time a while back in building a nicer API for C# to invoke shell commands and process the results. The only downside IMO is the Rx library dependency for STDOUT/STDERR; I personally try to avoid depending on libraries which themselves have extra dependencies.
Since I did this at work it belongs to my employer, so I can't currently publish it freely, but it's not part of our core product so they may be open to publishing it under MIT licence or similar at some stage.
So it can be done, and has been done, but I guess most people are sufficiently happy with Powershell, Python, etc to not bother bridging the gap for C# too.
Since I did this at work it belongs to my employer, so I can't currently publish it freely, but it's not part of our core product so they may be open to publishing it under MIT licence or similar at some stage.
So it can be done, and has been done, but I guess most people are sufficiently happy with Powershell, Python, etc to not bother bridging the gap for C# too.
> Yeah, most of that comes from the verbose process needed to invoke a process, right?
I think it's more incongruence between the languages. os.system() will technically call a command. subprocess is a big step up from popen2, but Perl was much more streamlined and terse. That also applies to one-liners when you're in a shell.
With Python you just have differences in a bunch of things; error handling? I often have to look up error codes, catch the exception then check for error codes. Pipe data in? Test for sys.stdin.isatty() then read from sys.stdin or fallback to sys.argv--it's not obscure, just not particularly Pythonic. The list goes on and on. It will probably take ~20 lines to properly deal with shelling out and if you do things wrong you could deadlock[1]. On the plus side, discouraging shelling out means your code is more portable =P
[1] https://docs.python.org/3.8/library/subprocess.html#subproce...
I think it's more incongruence between the languages. os.system() will technically call a command. subprocess is a big step up from popen2, but Perl was much more streamlined and terse. That also applies to one-liners when you're in a shell.
With Python you just have differences in a bunch of things; error handling? I often have to look up error codes, catch the exception then check for error codes. Pipe data in? Test for sys.stdin.isatty() then read from sys.stdin or fallback to sys.argv--it's not obscure, just not particularly Pythonic. The list goes on and on. It will probably take ~20 lines to properly deal with shelling out and if you do things wrong you could deadlock[1]. On the plus side, discouraging shelling out means your code is more portable =P
[1] https://docs.python.org/3.8/library/subprocess.html#subproce...
> Practically every Unix/Linux box in existence has at least some version of Python
If we understand "box" as "environment", in general, and not literally a physical or virtual machine...
Not really. E.g. see: every clean Ubuntu Docker container. They don't have Python, nor should have to include it by default.
And if you are going to do something like this:
I recently did the opposite of what everybody seems to be doing: moved an 800-line Python script, which you had to read carefully to understand what it was doing, to a 150-line Bash script that you only have to look at, because the important things to know is to see what other, external programs are being called. Just the apt-get stuff in Python was a mess with the poorly documented Python API for Apt, and some 50 lines to just do the same of an apt-get install... effectively wasted mental cycles.
(EDIT: formatting)
If we understand "box" as "environment", in general, and not literally a physical or virtual machine...
Not really. E.g. see: every clean Ubuntu Docker container. They don't have Python, nor should have to include it by default.
And if you are going to do something like this:
apt-get update && apt-get install -y stuff
wget https://file.tgz
tar xf file.tgz
cd file
make
make install
value="$(cat result | grep something)"
mvn clean install -Dapp.value="$value"
I recently did the opposite of what everybody seems to be doing: moved an 800-line Python script, which you had to read carefully to understand what it was doing, to a 150-line Bash script that you only have to look at, because the important things to know is to see what other, external programs are being called. Just the apt-get stuff in Python was a mess with the poorly documented Python API for Apt, and some 50 lines to just do the same of an apt-get install... effectively wasted mental cycles.
(EDIT: formatting)
> Python API for Apt
There seems to be something very wrong with this expression. Why would one need API for apt? It's a command-line tool.
There seems to be something very wrong with this expression. Why would one need API for apt? It's a command-line tool.
I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does. Process substitution is where the shell really shines, in my opinion.
Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to install and maintain other interpreters or runtimes on an embedded platform.
Even job management and parallelism are easier in Bash and GNU Parallel.
[1] https://www.tldp.org/LDP/abs/html/process-sub.html
Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to install and maintain other interpreters or runtimes on an embedded platform.
Even job management and parallelism are easier in Bash and GNU Parallel.
[1] https://www.tldp.org/LDP/abs/html/process-sub.html
> I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does.
Next Generation Shell - mostly there. If something of the above is missing - it's a bug and you are welcome to open an issue. I'm the author.
https://github.com/ngs-lang/ngs
Next Generation Shell - mostly there. If something of the above is missing - it's a bug and you are welcome to open an issue. I'm the author.
https://github.com/ngs-lang/ngs
I like this a lot. Definitely very cool, and I plan on giving it a shot when I have some time.
Just curious, but how would one do process substitution with NGS?
Just curious, but how would one do process substitution with NGS?
I don't know about process substitution, but Dart makes connecting processes together pretty easy and you can do way more fancy things than Bash let's you do, in a clean fast language with basically no gotchas.
Check out dshell: https://pub.dev/packages/dshell
Check out dshell: https://pub.dev/packages/dshell
Pretty cool, thanks for pointing it out. I have some experience with Dart via Flutter development, so I'll give this a go when I have some time.
And now we have 3 languages...
If your OpenWRT router has a bash shell, then you optionally installed it from packages. The default shell is Busybox, which is a minimal shell that supports few if any "bashisms." In other words, you certainly do have to be wary about what shell features you use.
PowerShell, mostly because PowerShell was designed as a mashup of Bash and C#.... And it's kind of a trainwreck in a lot of ways.
It really feels like piping and easy process invocation and compile-time directory awareness wouldn't be massively onerous to add to an existing full-featured programming language so you wouldn't have to sacrifice a good type system and powerful syntax when you want to do scripty things.
It really feels like piping and easy process invocation and compile-time directory awareness wouldn't be massively onerous to add to an existing full-featured programming language so you wouldn't have to sacrifice a good type system and powerful syntax when you want to do scripty things.
As I understand it, PowerShell came about because porting the standard Linux/Unix utilities to Windows didn't work out.
PowerShell meets both requirements as it's tuned for the Windows environment, but can still play nicely with the rest of the world.
Of course, now that WSL is a thing, it's a real question why someone would want to use PS unless they had no other options.
https://www.heavybit.com/library/podcasts/to-be-continuous/e...
PowerShell meets both requirements as it's tuned for the Windows environment, but can still play nicely with the rest of the world.
Of course, now that WSL is a thing, it's a real question why someone would want to use PS unless they had no other options.
https://www.heavybit.com/library/podcasts/to-be-continuous/e...
PowerShell has real data structures (arrays and hash tables), built-in functional programming tools (Select-Object, ForEach-Object, Where-Object), GUI cmdlets (Out-GridView) and direct access to .NET libraries. It’s a viable tool for many complex tasks where on Unix, you’d need to reach for Python rather than a shell.
>Of course, now that WSL is a thing, it's a real question why someone would want to use PS unless they had no other options.
I use it because I only have to learn one tool, and not a ton of different ones with different UX.
The 'language' is sane, at least to me, as it's object oriented doesn't revolve around parsing strings.
I use it because I only have to learn one tool, and not a ton of different ones with different UX.
The 'language' is sane, at least to me, as it's object oriented doesn't revolve around parsing strings.
Some of the features of PowerShell seem really enticing, and I wish they would make their way over to Unix-like shells. Unfortunately, I don't know how much utility I'd get out of PowerShell on Linux, so I haven't tried it.
edit: your comment inspired me so I installed PowerShell via Snap and am going to give it a go on Linux.
edit: your comment inspired me so I installed PowerShell via Snap and am going to give it a go on Linux.
To me PowerShell mixes easy access to command-line tools and higher-order object-oriented map/filter style functions. The mix between Unix-style piping from command-line tools and modern object-y mapping and filtering can feel like a dream.
That's the good part. The bad part is that it's easily one of the wartiest languages I've ever used, especially as a young language it's really got an inexcusable amount of legacy problems.
A colleague of mine (who also uses it heavily) says "two Googles per line".
Just to get sane parameter and variable checking you have to throw up a bunch of attributes and set a bunch of flags.
Be sure to set up strict mode, use cmdletbinding on your parameter blocks, and set ErrorActionPreference to "stop" in every script you write.
That and the quality of many of the first-party official PowerShell modules is appallingly low by Microsoft standards. The good side is that the .NET framework is accessible so whenever the PowerShell module is failing you, you can drop down into leveraging raw c#-style assemblies, which are much higher quality but they speak PowerShell with a very thick accent.
But I'm using it on Windows, YMMV on Linux.
I'm infatuated with the concept, but the implementation leaves a lot to be desired.
That's the good part. The bad part is that it's easily one of the wartiest languages I've ever used, especially as a young language it's really got an inexcusable amount of legacy problems.
A colleague of mine (who also uses it heavily) says "two Googles per line".
Just to get sane parameter and variable checking you have to throw up a bunch of attributes and set a bunch of flags.
Be sure to set up strict mode, use cmdletbinding on your parameter blocks, and set ErrorActionPreference to "stop" in every script you write.
That and the quality of many of the first-party official PowerShell modules is appallingly low by Microsoft standards. The good side is that the .NET framework is accessible so whenever the PowerShell module is failing you, you can drop down into leveraging raw c#-style assemblies, which are much higher quality but they speak PowerShell with a very thick accent.
But I'm using it on Windows, YMMV on Linux.
I'm infatuated with the concept, but the implementation leaves a lot to be desired.
Right. Powershell is a shell though not a programming language.
That is false.
Why do you think it is not a programming language?
Why do you think it is not a programming language?
Shell scripts are readable by just about anyone, they're available on every UNIX system, not just the Red Hat/Debian-derivatives of the last twenty years, they're fast as long as you're not doing stupid things, they're easily maintainable, they don't handle dependencies terribly (unlike Python), and so forth.
There's a reason AT&T used to run ads that showed their secretaries, managers, and so on using and writing shell scripts and there's never been a Python ad claiming that just anyone could write it.
There's a reason AT&T used to run ads that showed their secretaries, managers, and so on using and writing shell scripts and there's never been a Python ad claiming that just anyone could write it.
Having seen many professionals writing shell scripts. No they can’t.
Is this because both languages are from different eras? When did you last see a consumer ad about scripting?
> Shell scripts are readable by just about anyone
I reckon that less than 0.1% of the population can read shell scripts.
I reckon that less than 0.1% of the population can read shell scripts.
I don't think that's a far comment. It was a different era with different expectations about computer user. The equivalent these days would probably be Excel macros
Ok, go ahead and port all 200 of these scripts to Python please: https://github.com/shawwn/scrap
PR's welcome. ;)
Python makes sense for scripts that need good argument parsing, or complicated intermediate input processing. But it's pretty annoying to get a shell pipeline working in Python. `foo | bar | baz` is about 15 characters in bash.
Here's an example. I use `llbranch` all the time: https://github.com/shawwn/scrap/blob/master/llbranch
It lists all branches in a project. It's nicely colorized. What would the equivalent Python be? Who knows, but it'd be much longer.
PR's welcome. ;)
Python makes sense for scripts that need good argument parsing, or complicated intermediate input processing. But it's pretty annoying to get a shell pipeline working in Python. `foo | bar | baz` is about 15 characters in bash.
Here's an example. I use `llbranch` all the time: https://github.com/shawwn/scrap/blob/master/llbranch
It lists all branches in a project. It's nicely colorized. What would the equivalent Python be? Who knows, but it'd be much longer.
Correct me if my first impression was incorrect, but scanning through your scripts they all seemed like sub-kilobyte one-liners. Which, yeah, just use Shell for those and you'll be fine.
To my mind, the parent post was referring to lengthier, more complicated scripts. Speaking as one who wrote and routinely maintained such scripts at one point, I cannot agree more with the sentiment that most if not all of them could and should have been written in Python or Ruby or some other scripting language instead.
To my mind, the parent post was referring to lengthier, more complicated scripts. Speaking as one who wrote and routinely maintained such scripts at one point, I cannot agree more with the sentiment that most if not all of them could and should have been written in Python or Ruby or some other scripting language instead.
Cool repo, but the example is lame because a shell, or a git alias would do just fine.
A shell script that consists of a one-line exec isn't what people concerned about.
FWIW I'm a 99.9% POSIX masochist (probably closer to 4N)
A shell script that consists of a one-line exec isn't what people concerned about.
FWIW I'm a 99.9% POSIX masochist (probably closer to 4N)
Looks like a lot of your scripts couldbe aliases, I store mine in ~/.aliases, and slightly more complicated things (e.g. take arguments) in ~/.functions, and source both those files from bashrc.
I have maybe 3 standalone shell scripts in my PATH, despite writing thousands of lines of shell.
I have maybe 3 standalone shell scripts in my PATH, despite writing thousands of lines of shell.
I've moved from aliases to scripts as I can't seem to have found the (if any exists...) way to ensure aliases can be used from within vim's ":!..." or ":'<,'>!...", and as I often find myself wanting to either execute something while I'm amending code, or wanting to filter text with a purpose-built command, I end up writing (sometimes very short!) shell scripts.
I had a mess of shell functions. It is nice that they run more quickly, but I don't have any shell commands that need to run that often.
What moved me to putting them in a bin directory was that it's just easier to edit a standalone script.
What moved me to putting them in a bin directory was that it's just easier to edit a standalone script.
Write a shell script to concat them into a single file to source ;)
I used to do that with a help of Fabric/Invoke, and it was a pleasant experience overall, but sometimes required too much of verbosity. Then I discovered Plumbum (https://plumbum.readthedocs.io/en/latest/) with its concept of combinators, that I liked a lot more.
Eventually it motivated me to switch the language for shell scripting for the second time, and nowadays I recommend Haskell's Turtle (https://hackage.haskell.org/package/turtle-1.5.16/docs/Turtl...) to anyone who is interested in safer shell scripting and still likes a concise and terse syntax (and it's blazingly fast too).
Eventually it motivated me to switch the language for shell scripting for the second time, and nowadays I recommend Haskell's Turtle (https://hackage.haskell.org/package/turtle-1.5.16/docs/Turtl...) to anyone who is interested in safer shell scripting and still likes a concise and terse syntax (and it's blazingly fast too).
The shell is the true user interface to the OS. I like to think of writing shell scripts as having some mouse clicking automation tool on a GUI shell. It should be viewed as just as kludgy of a solution. But there really isn't anything more convenient most of the time. Shell scripts are to be called by humans to automate behavior they otherwise would have had to do manually.
An application should never rely on a shell script. If you have to execute another program, use the syscalls (fork and exec on posix).
An application should never rely on a shell script. If you have to execute another program, use the syscalls (fork and exec on posix).
> An application should never rely on a shell script. If you have to execute another program, use the syscalls (fork and exec on posix).
Could you explain why? I'm very happy with programs calling shell commands/scripts. You execute another program the same way whether interactively from a shell, or by calling system() from another program. The simplicity and universality of the call syntax is an advantage.
Could you explain why? I'm very happy with programs calling shell commands/scripts. You execute another program the same way whether interactively from a shell, or by calling system() from another program. The simplicity and universality of the call syntax is an advantage.
Security. system() is one of the most common targets for hacking (getting a shell by manipulating the string passed to system() by various means). Calling programs from the kernel directly is a lot more well-behaved. You're limited to only executing one program.
I love python, and half my ‘shell’ scripts are python, but the mere fact that I have to import something to exec something or read a file means that bash is easier to build and test one line at a time. And good bash pipelined one-liners are a reason I’ll never write all of my scripts in python. It’d be hard for me to calculate how often I use constructs like ‘grep | sort | uniq | cut’. That takes a lot of python code.
I still think for very small stuff it's easier to just write a shell script. For instance a docker entry point, where you set a few env vars, download some required files and start your main application.
It’s a real pain to handle subprocesses in Python. If you need to automate certain command based workflows, Bash scripts are much easier, both to write and read, until they reach certain size. At my previous job it was a daily task and scripts involved tricky stuff related with Subversion, Git and builds (complex CI/CD, generally).
The problem is that very few people know shell scripting well, but once you get to know it, it’s not that bad, in quite specific cases.
Always use Shellcheck, though.
The problem is that very few people know shell scripting well, but once you get to know it, it’s not that bad, in quite specific cases.
Always use Shellcheck, though.
If it's a small thing (100 lines or less) I use shell. Anything bigger goes to Perl.
Here's a comment on that from someone working to make a better bash:
https://www.oilshell.org/blog/2018/01/28.html#shouldnt-scrip...
https://www.oilshell.org/blog/2018/01/28.html#shouldnt-scrip...
Practically speaking, my shell scripts tend to be something like a tenth the size of the equivalent python
It's horses for courses. The Unix shell is an interface to system commands. Python is general purpose programming language.
You don't have to pick between the two. Shell scripts are great and give you the option of using battle tested commands that have stood the test of time.
When you don't need to do any actual logic and just want to run a sequence of commands that you've been typing in by hand, and you already know the commands, and you don't want to go look up the Python versions of them all.
IMO, the power of shell is really the ability to leverage command-line tools (e.g. git, curl, jq) with very little code. These tools are very fast, well-tested, feature-rich, and often easy to install in a reproducable way.
IMO, The inflection point where you should stop using shell is very low, though. The Google style guide for shell recommends that you should rewrite scripts once they are more than 100 lines, and I think that it probably too generous.
IMO, The inflection point where you should stop using shell is very low, though. The Google style guide for shell recommends that you should rewrite scripts once they are more than 100 lines, and I think that it probably too generous.
[deleted]
git extensions are an unfortunate counter-example. git relies heavily on shell scripts, and it even has a shell "library" [1].
I wanted to create a custom git command and started with Rust/libgit2, but found it was missing too much. Shell scripting proved to be the most natural, best supported approach (though it was nevertheless very painful).
1: https://github.com/git/git/blob/master/git-sh-setup.sh
2: https://github.com/ridiculousfish/git-prev-next
I wanted to create a custom git command and started with Rust/libgit2, but found it was missing too much. Shell scripting proved to be the most natural, best supported approach (though it was nevertheless very painful).
1: https://github.com/git/git/blob/master/git-sh-setup.sh
2: https://github.com/ridiculousfish/git-prev-next
For anything that isn't super trivial and I mean super trivial < 10 lines - Perl is a better solution probably more standardised that Python.
until they don't and python 3 is there instead?
The problem with Python vs shell scripts is that Python is a bloated mess of a platform. Python tries to be all things to all people, through a messy package system that isn't cross compatible with different platforms. bin/sh|bash is 1/100th the size of python and rock-solid.
I think the main issue is: just because someone gets something to work on their system with their Python install, they think it works everywhere. And that is not at all true. There's a reason bin/sh|bash are so pervasive in administration, they are small, simple, well-understood, and resistant to the kind of bloat the Python chokes on.
python is basically this:
https://www.amazon.com/Wenger-16999-Swiss-Knife-Giant/dp/B00...
I think the main issue is: just because someone gets something to work on their system with their Python install, they think it works everywhere. And that is not at all true. There's a reason bin/sh|bash are so pervasive in administration, they are small, simple, well-understood, and resistant to the kind of bloat the Python chokes on.
python is basically this:
https://www.amazon.com/Wenger-16999-Swiss-Knife-Giant/dp/B00...
Shell scripts have their well-deserved place in Unix systems. A general recommendation to use Python or other high-level scripting languages without a discussion about the reasons why and when to use shell may lead to wrong conclusions.
I'd love to read a good blog post on when to use shell scripts vs Python or other programming languages, which seem far more accessible to me.
If all you need to do is chain together external programs, set environment variables, redirect filehandles, iterate over files, etc with the tiniest bit of logic, and don't want to have to set up an execution environment or download something to do it, and want ultimate Unixy portability, and you want virtually anyone to be able to read and maybe modify it, you want shell.
If you need to do a very specific task that involves interpreting/modifying data structures/formats, if you have non-trivial logic, if you need to use a module, if you need to access an operating system primitive which Unix shells don't really expose, if there's no Unix tool that does what you want, or you just need more control/certainty/reliability, you want Python or something else.
If you need to do a very specific task that involves interpreting/modifying data structures/formats, if you have non-trivial logic, if you need to use a module, if you need to access an operating system primitive which Unix shells don't really expose, if there's no Unix tool that does what you want, or you just need more control/certainty/reliability, you want Python or something else.
The funny thing though is that the "virtually anyone" part seems to me more likely to happen with python - or with any mainstream programming language - than with what seem to me like deeply arcane shell incantations.
[deleted]
I agree, there is not just one true answer to it. For me, I would prefer shell scripts in many bootstrapping processes targeted on system administration. Also as glue language for many GNU tools, shell (then probably bash) is indispensable. With pure shell I can reduce the number of abstraction layers and dependencies. Also POSIX cross-platform compatiblity is a big point, just think of ``./configure`` in software builds.
I'm not sure I'm comfortable with python for a typical shell script. But then again, what alternatives exist?
I've been thinking that we are kind of stuck with it. Much like javascript.
So, we could go the route of typescript and have a translation layer and outputting shell scripts. I actually think that could work quite well from a technical standpoint.
A big part of shellscripts though is having the source available. So maybe even have the translated, original code, and the generated shell output in the same file.
The source code on top (out of view from a shell interpreter) and the generated code below. This would allow it to be readable by anyone and executable by anyone, but modifying it would require the transpiler (which by itself would probably be pretty lightweight as well - certainly compared to python).
Certainly not perfect, but preferable to rediscovering the nuances and gotchas of shell-scripts every other day. Or maybe such a solution would only prolong the suffering and we should start all over.
I've been thinking that we are kind of stuck with it. Much like javascript.
So, we could go the route of typescript and have a translation layer and outputting shell scripts. I actually think that could work quite well from a technical standpoint.
A big part of shellscripts though is having the source available. So maybe even have the translated, original code, and the generated shell output in the same file.
The source code on top (out of view from a shell interpreter) and the generated code below. This would allow it to be readable by anyone and executable by anyone, but modifying it would require the transpiler (which by itself would probably be pretty lightweight as well - certainly compared to python).
Certainly not perfect, but preferable to rediscovering the nuances and gotchas of shell-scripts every other day. Or maybe such a solution would only prolong the suffering and we should start all over.
> I'm not sure I'm comfortable with python for a typical shell script. But then again, what alternatives exist?
perl still exists, and is a wonderful alternative to python for "shell scripts plus". The syntax should be at least vaguely familiar to anyone who has touched bash, awk, sed, etc.
perl still exists, and is a wonderful alternative to python for "shell scripts plus". The syntax should be at least vaguely familiar to anyone who has touched bash, awk, sed, etc.
When these threads come up, it’s staggering to see how much mindshare Perl has lost. This particular case of code that grows too klunky for the Unix shell was a key motivator for the language’s invention and ought to be THE final outpost, but people barely seem aware of the option.
What’s wrong with python?
Feels like the wrong tool for the job with the whole virtual environment that has to be brought up.
Performance for shell scripts isn't super critical in my eyes but the startup time of python feels too much for something you might want to run in an inner-loop from find or something. Not sure if python IO performance is suitable for shell scripts either.
Also isn't nearly as universal as shell-scripts nor something you necessarily have/want on small systems.
But I will admit that I'm channeling prejudices and don't feel I have enough of a complete picture to say anything definite about it. Others in this thread touch upon it though.
Performance for shell scripts isn't super critical in my eyes but the startup time of python feels too much for something you might want to run in an inner-loop from find or something. Not sure if python IO performance is suitable for shell scripts either.
Also isn't nearly as universal as shell-scripts nor something you necessarily have/want on small systems.
But I will admit that I'm channeling prejudices and don't feel I have enough of a complete picture to say anything definite about it. Others in this thread touch upon it though.
You don’t need a virtual env.
I probably wouldn’t wrap GNU find and python, I’d just walk the directory tree in python and use regex.
IO is suitable for everyday tasks and shouldn’t be an issue in Python anyway, if it is either you’re doing it wrong or your shell script should be written in C and not touch interpreted/bytecode compiled languages at all.
I probably wouldn’t wrap GNU find and python, I’d just walk the directory tree in python and use regex.
IO is suitable for everyday tasks and shouldn’t be an issue in Python anyway, if it is either you’re doing it wrong or your shell script should be written in C and not touch interpreted/bytecode compiled languages at all.
Me neither, but a big point of shell scripts is that they are the same commands you'd run on the shell itself.
And there are lots of really optimized tools that are as quick as can be, throwing them out is a hefty price to pay.
And there are lots of really optimized tools that are as quick as can be, throwing them out is a hefty price to pay.
It's too high level for a shell - the unix concepts are too many keystrokes and modules away in python. Also, bad performance.
Some time ago I started to doubt if using 'set -e' is a good idea.
I mean, if it would work as you expect it to, it certainly is a good idea to exit a script as soon as something fails. But sadly not all implementations behave similarly [1] and if you call a function from within a condition, 'set -e' gets deactivated/doesn't work.
For illustration, take a look at the following example:
[1] https://www.in-ulm.de/~mascheck/various/set-e/
I mean, if it would work as you expect it to, it certainly is a good idea to exit a script as soon as something fails. But sadly not all implementations behave similarly [1] and if you call a function from within a condition, 'set -e' gets deactivated/doesn't work.
For illustration, take a look at the following example:
#!/bin/bash
foo() {
set -e
false
echo "Sucks"
}
printf 'Normal: %s\n' "$(foo)"
printf 'Condition: %s\n' "$(foo || true)"
Normal:
Condition: Sucks
[1] https://www.in-ulm.de/~mascheck/various/set-e/
If I have something I know can fail, and I want it to be able to, then calling set +e before it, and recalling set -e afterwards is the way I handle it.
But, in this case, because you're running it in a subcommand, you need to add pipefail.
But, in this case, because you're running it in a subcommand, you need to add pipefail.
set -euo pipefail
As far as I can tell, adding pipefail or nounset doesn't change the behavior in this case.
In my opinion, the problem is that calling a function from a test expression, evaluates it in a different manner than calling it from elsewhere. That is so absurd that I wonder how it wasn't changed over the years.
In my opinion, the problem is that calling a function from a test expression, evaluates it in a different manner than calling it from elsewhere. That is so absurd that I wonder how it wasn't changed over the years.
Yeah there's no good solution for this in shell -- you're damned if you do, and damned if you don't.
bash has inherit_errexit to fix a longstanding bug, but even that's not enough.
Oil has shopt -s strict_errexit and more_errexit that fixes even more.
https://github.com/oilshell/oil/issues/476
I'm still looking for feedback and there's probably one more thing I want to change.
Although the fundamental problem is that errors (success/fail) and boolean values (true/false) are conflated in the status code, so it's a tough problem.
In Oil you will be encouraged to use expressions for booleans, not statements, e.g.
bash has inherit_errexit to fix a longstanding bug, but even that's not enough.
Oil has shopt -s strict_errexit and more_errexit that fixes even more.
https://github.com/oilshell/oil/issues/476
I'm still looking for feedback and there's probably one more thing I want to change.
Although the fundamental problem is that errors (success/fail) and boolean values (true/false) are conflated in the status code, so it's a tough problem.
In Oil you will be encouraged to use expressions for booleans, not statements, e.g.
ls / # success/fail
if (x > 0) { } # true/false
ls /
if test $x -lt 0; then ...Obsessively checking for failures like people do in go is the "correct" way. Using && to chain groups of your commands that depend on each other also helps.
Well, I am not sure what the correct way of handling errors is. I like the concept of Exceptions, but not the results when people use them. Somehow they encourage devs to care less about error checking.
And while I appreciate the thoroughness of handling errors the Go way, I find it tiring and verbose. In many cases, you just want to exit with a specific exit code and adding 4 lines just to specify that exit code doesn't feel right.
The problem I have with errexit is that you can't rely on it. You can write your function with it (even enabling it explicitly), but later someone comes along, calls it in the wrong context and boom, your function runs without errexit.
And while I appreciate the thoroughness of handling errors the Go way, I find it tiring and verbose. In many cases, you just want to exit with a specific exit code and adding 4 lines just to specify that exit code doesn't feel right.
The problem I have with errexit is that you can't rely on it. You can write your function with it (even enabling it explicitly), but later someone comes along, calls it in the wrong context and boom, your function runs without errexit.
[deleted]
Can mktemp fail? I recently wrote a script that ensures that the directory created by mktemp actually exists and starts with /tmp so that when the script wipes out its temporary data at the end, it will not ever run something like "rm -rf /". Using fully-qualified paths for everything is also (not) fun.
In another script I was leery about running rm -f -- /path/foo* so I instead used
Most likely set -e is enough protection against mktemp failing and the rm -f glob probably would've been just fine, but shell scripting has enough footguns that sometimes I can't help but go overboard with paranoia.
In another script I was leery about running rm -f -- /path/foo* so I instead used
find /path -mindepth 1 -maxdepth 1 -name "foo*" -delete
Most likely set -e is enough protection against mktemp failing and the rm -f glob probably would've been just fine, but shell scripting has enough footguns that sometimes I can't help but go overboard with paranoia.
Yes, mktemp can fail.
For example, "TMPDIR=/dev/null mktemp -d" will reliably fail.
You shouldn't be validating it starts with "/tmp" though, because on quite a few systems, people set TMPDIR=/var/tmp.
You absolutely should check if mktemp exited nonzero, but if it exited 0, the directory should be safe to use. If you want to be really paranoid, you can check if the output is an empty string too after checking the exit code.
I think you're shying a little too far away from bash's globbing. Yeah, bash has footguns, but using clever workarounds can also be a recipe for creating confusing and error prone code in its own way.
For example, "TMPDIR=/dev/null mktemp -d" will reliably fail.
You shouldn't be validating it starts with "/tmp" though, because on quite a few systems, people set TMPDIR=/var/tmp.
You absolutely should check if mktemp exited nonzero, but if it exited 0, the directory should be safe to use. If you want to be really paranoid, you can check if the output is an empty string too after checking the exit code.
I think you're shying a little too far away from bash's globbing. Yeah, bash has footguns, but using clever workarounds can also be a recipe for creating confusing and error prone code in its own way.
Please tell me it's enough to check the exit code! That's all I ever do and I have hundreds of mktemp scripts out there.
In the end I think shelling is mostly about controlling your inputs.
In the end I think shelling is mostly about controlling your inputs.
Yes, it's enough to check the exit code. The parent poster's being overly paranoid.
The only thing that checking the exit code won't catch is bugs in mktemp/bash, or bad memory / solar flare bitflips.
The only thing that checking the exit code won't catch is bugs in mktemp/bash, or bad memory / solar flare bitflips.
Not just inputs, unfortunately. TOCTOUs (and races in general), for instance, are very common problem in shell scripts.
Races and TOCTOUs are generally common problems in most concurrent systems that are not considering atomicity by design (e.g. rdbms). It's just that shell can very easily become concurrent...
> Can mktemp fail?
Yes it can. You should use its exit status (or set -e). See http://man.openbsd.org/mktemp
Note that there is a nasty trap if you're using local to declare and set a variable:
Yes it can. You should use its exit status (or set -e). See http://man.openbsd.org/mktemp
Note that there is a nasty trap if you're using local to declare and set a variable:
a () { local M=`ls /asdf` && echo y; }
b () { N=`ls /asdf` && echo y; }
a ; b ;
Shellcheck helpfully warns about declaring and assigning in the same statement exactly because it masks exit status.
The point of mktemp is to use the right location on the current system, which may not be /tmp. If you're going to enforce that tmpdir must be /tmp, you might as well not bother with mktemp and just directly create a file or directory in /tmp.
With the "--" and assuming GNU rm, rm -f -- /path/foo* is safe. If you use a glob pattern that does not match itself, then you will want nullglob:
rm foo.[cC]
Wouldn't the glob still have issues with files that contain spaces?
My defensiveness generally takes a sharp upturn when I write scripts that will be run as root or ones that delete data, but I'll admit not trusting mktemp's exit code is probably taking it a bit too far.
My defensiveness generally takes a sharp upturn when I write scripts that will be run as root or ones that delete data, but I'll admit not trusting mktemp's exit code is probably taking it a bit too far.
I’m constantly surprised that we got a safe language that compiles to Javascript (TypeScript) but never got a safe language that compiles to shell script. Why can’t I write in (a subset of) some other scripting language $lang, but with restricted-to-pure-/bin/sh semantics, and then cross-compile it to actual portable shell script for distribution?
Hopefully not by generating a megabyte of polyfill runtime code; more just by the compiler refusing to compile code in $lang unless it has a direct equivalent in /bin/sh. Any $lang source file the compiler accepted, would just look like “a shell script translated into $lang” already. But the compiler would inject $lang’s safe semantics (exceptions, type-checking, etc.) during the compilation, so you’d at least get that benefit.
Alternately, I’d also be satisfied with “emscripten for shell”: a compiler that generated shell scripts containing a small WASM-like emulator and an embedded bytecode stream to feed it. As long as it was lightweight enough. (A large point of these environments’ use of /bin/sh is that they’re small and embedded and can’t load too much into memory at once.)
My only guess for why this hasn’t happened, is that the people who write things like shell scripts that execute in initramfs, or shell scripts that are intended to install stuff even on lesser-known UNIXes, are all old-hand ops people who know shell-scripting cold, and certainly aren’t developers with any experience in compiler theory.
Hopefully not by generating a megabyte of polyfill runtime code; more just by the compiler refusing to compile code in $lang unless it has a direct equivalent in /bin/sh. Any $lang source file the compiler accepted, would just look like “a shell script translated into $lang” already. But the compiler would inject $lang’s safe semantics (exceptions, type-checking, etc.) during the compilation, so you’d at least get that benefit.
Alternately, I’d also be satisfied with “emscripten for shell”: a compiler that generated shell scripts containing a small WASM-like emulator and an embedded bytecode stream to feed it. As long as it was lightweight enough. (A large point of these environments’ use of /bin/sh is that they’re small and embedded and can’t load too much into memory at once.)
My only guess for why this hasn’t happened, is that the people who write things like shell scripts that execute in initramfs, or shell scripts that are intended to install stuff even on lesser-known UNIXes, are all old-hand ops people who know shell-scripting cold, and certainly aren’t developers with any experience in compiler theory.
Because doing so would be hard for shell scripts to express. real functions, try/catch, and other common language features are not often part of shell languages. As such, the generated code may not support many features you typed in <other lang> and/or the generated code would bundle a runtime with it to emulate what we'd otherwise consider pragmatic scripting feature support.
I mean, like I said, I don’t want to use $lang features that don’t exist in shell script. E.g., I want “exceptions” in the sense that adding a string-typed variable to an integer-typed variable will blow up in a descriptive way (or better, not compile); but I don’t want real exceptions in the sense of being able to catch them. I just want everything to translate to the shell script aborting in verbose and helpful ways before doing something stupid with invalid inputs; or not compiling at all if there’s a code-path that can’t possibly be valid.
Also, I wouldn’t mind if this $lang that compiles to shell script is its very own language I’d have to learn, just like TypeScript is its very own language you have to learn. As long as I don’t have to manually write five layers of guards using impenetrable [ “y${x:-1}” -eq “f” ] style code, and then duplicate the hierarchy of cleanup behaviours after each failed guard, I’d be happy.
Also, I wouldn’t mind if this $lang that compiles to shell script is its very own language I’d have to learn, just like TypeScript is its very own language you have to learn. As long as I don’t have to manually write five layers of guards using impenetrable [ “y${x:-1}” -eq “f” ] style code, and then duplicate the hierarchy of cleanup behaviours after each failed guard, I’d be happy.
This does exist but bash/sh is missing proper types. Examples include, null and numbers and anonymous functions.
Mathematica and other software platforms use bash as universal installer for Mac and Linux. I have seen languages that compile to bash but use is limited based on the language.
Mathematica and other software platforms use bash as universal installer for Mac and Linux. I have seen languages that compile to bash but use is limited based on the language.
In many (not all) times, these "universal installers" are only a small number of POSIX-compilant shell lines followed by some Base64-encoded binary. The whole purpose of the shell commands is to extract this binary.
Checking $PATH or explicitly calling outside resources is probably also advisible.
In general, this article seems very light. No mention of LD_LIBRARY_PATH, for example. I imagine there's probably a better guide to writing secure scripts somewhere else.
In general, this article seems very light. No mention of LD_LIBRARY_PATH, for example. I imagine there's probably a better guide to writing secure scripts somewhere else.
These sorts of security measures are only really necessary when you have a setuid or setgid shell script. (However, a much better suggestion would be to simply not write setuid or setgid shell scripts because they are a security nightmare.)
For more concrete recommendations in a similar spirit, check out https://github.com/pesterhazy/blissful-bash
Instead of
Caveat: some of the longer forms might be Bashisms (e.g. not present in ksh, dash, etc.)
set -euf -o pipefail
set -o errexit
set -o nounset
set -o noglob
set -o pipefail
Caveat: some of the longer forms might be Bashisms (e.g. not present in ksh, dash, etc.)
"which is easier to lookup/search for"
From my experiment:
Googling "set euf" (without quotes) -- 838,000 results, first 6 results are all helpful and explain it in detail.
Googling "set o nounset" (without quotes) -- 38,700 results, entire first page seems helpful.
I dunno, looks like they're identically easy to learn the meaning of.
From my experiment:
Googling "set euf" (without quotes) -- 838,000 results, first 6 results are all helpful and explain it in detail.
Googling "set o nounset" (without quotes) -- 38,700 results, entire first page seems helpful.
I dunno, looks like they're identically easy to learn the meaning of.
"easier to understand/remember the meaning of" would have been better phrasing. I can remember/infer what `set -o noglob` does, but not `set -f`.
I find exceptionally difficult to write anything but trivial shell scripts without bugs. This one took years, and I suspect #bash could still find a bug: https://github.com/jakeogh/commandlock
On the other hand, this is amazing: https://github.com/speed47/spectre-meltdown-checker
On the other hand, this is amazing: https://github.com/speed47/spectre-meltdown-checker
Hi. Author of Next Generation Shell here.
Here is my take on bash vs Python and friends. I am frustrated by both.
bash - domain specific language and huge library (CLI utilities) and lets you get the job done but not a language I would like to use (syntax, error handling, very limited structured data support).
Python - okayish as a language but doing domain specific things (working with files and processes) is so much more verbose than bash.
My solution in NGS - have a high level, modern language with the typical goodies like ... exceptions (wow, completely new concept!) and the language is still domain specific. Working with processes for example has it's own syntax and is much more concise and straightforward (stole bits from bash).
Right now the project has the language, which is useful enough to write scripts (which we do at work). Regarding contribution to the project, my idea is that as much as possible should be in NGS language (as opposed to the lower level C). The UI will be implemented completely in NGS, allowing contributing to the project using the same language you are writing your scripts in.
https://github.com/ngs-lang/ngs
As a side note, while the UI is not there yet, I do plans for it which include interaction with objects on the screen as opposed to current "here is your dump of text" situation... and in general be more considerate of the user.
https://github.com/ngs-lang/ngs/wiki/UI-Design
Here is my take on bash vs Python and friends. I am frustrated by both.
bash - domain specific language and huge library (CLI utilities) and lets you get the job done but not a language I would like to use (syntax, error handling, very limited structured data support).
Python - okayish as a language but doing domain specific things (working with files and processes) is so much more verbose than bash.
My solution in NGS - have a high level, modern language with the typical goodies like ... exceptions (wow, completely new concept!) and the language is still domain specific. Working with processes for example has it's own syntax and is much more concise and straightforward (stole bits from bash).
Right now the project has the language, which is useful enough to write scripts (which we do at work). Regarding contribution to the project, my idea is that as much as possible should be in NGS language (as opposed to the lower level C). The UI will be implemented completely in NGS, allowing contributing to the project using the same language you are writing your scripts in.
https://github.com/ngs-lang/ngs
As a side note, while the UI is not there yet, I do plans for it which include interaction with objects on the screen as opposed to current "here is your dump of text" situation... and in general be more considerate of the user.
https://github.com/ngs-lang/ngs/wiki/UI-Design
Go write a shell script just like any other piece of code. Have fun!
[deleted]
Here's my boilerplate bash header with comments describing each option, which I find practically helpful for starting new scripts
https://gist.github.com/crizCraig/f42bc250754bed764ada5f95d1...
raw: https://gist.githubusercontent.com/crizCraig/f42bc250754bed7...
https://gist.github.com/crizCraig/f42bc250754bed764ada5f95d1...
raw: https://gist.githubusercontent.com/crizCraig/f42bc250754bed7...
I personally think "don't" is right. For those who are interested in trying to use Python instead, I found this to be a helpful resource: https://github.com/ninjaaron/replacing-bash-scripting-with-p...
I disagree, Python scripting will tie you to Python development uncertainties, while shell works with most operational environments.
That's a good point. Shell would not announce an EOL date for a specific version. Scripts that used to work 25 years ago, still work just the same. Python scripts that used to work 10 years ago, now need maintenance/porting, no matter how trivial, to make them work with Python 3.
You probably have one operating environment. It is the environment that you wrote that script for. Generalization in 99 out of 100 is introduction of an unnecessary flow that won't ever be executed outside the test but it will eat time.
[deleted]
I've done a bit of porting bash scripts to posix recently and I've found the following boilerplate pretty useful:
case "$(readlink /proc/$$/exe)" in */bash) set -euo pipefail ;; *) set -eu ;; esac
More useful tips in the same spirit
http://robertmuth.blogspot.com/2012/08/better-bash-scripting...
http://robertmuth.blogspot.com/2012/08/better-bash-scripting...
Since it's bash-specific, I think people should use zsh instead. No need to quote variables as by default, splitting is not done. And you get access to arrays and associative arrays.
Bash has arrays.
x=(1 2)
echo "${x[0]}"
declare -A x
x[hi]=20
x[bye]=30
echo "${x[hi]}"
for i in "${!x[@]}"; do
echo "k: $i, v: ${x[$i]}"
doneI love zsh’s variable & filename expansion features! But I switched to bash after using zsh for a long time because zsh is less standard, not installed by default, can be a bit of a pain when doing lots of ssh work or on machines I don’t control. Bash is always there.
Zsh is also really heavy, or it was last time I checked. Larger binary & slower startup time than bash, and lots of features I don’t use. Modules, calendar, tcp, ftp client ... that’s a lot of stuff in the shell that is arguably better put into separate executables.
Zsh is also really heavy, or it was last time I checked. Larger binary & slower startup time than bash, and lots of features I don’t use. Modules, calendar, tcp, ftp client ... that’s a lot of stuff in the shell that is arguably better put into separate executables.
On set -e:
This is like adding a setting to your Java or Python program to immediately exit if any method call throws any kind of exception, with no possibility of catching and handling that exception.
This does not seem to be a smart thing to do as compared to checking return codes, etc.
Every script I've seen with set -e has been buggier than without it.
The main problem with scripts is that they often do not go through a normal software review and testing process.
This is like adding a setting to your Java or Python program to immediately exit if any method call throws any kind of exception, with no possibility of catching and handling that exception.
This does not seem to be a smart thing to do as compared to checking return codes, etc.
Every script I've seen with set -e has been buggier than without it.
The main problem with scripts is that they often do not go through a normal software review and testing process.
[deleted]
The first piece of advice, "Don't", is the best. "shellcheck myscript.sh" should return a fatal error if the shell script file you're checking exists, and "shellcheck -f myscript.sh" should fix your shell script by removing it.
Amen.
deno, node, python, julia--anything but shell scripting.
deno, node, python, julia--anything but shell scripting.
At the beginning of each file:
#!/bin/bash
set -euf -o pipefail
cd $(dirname $0)
Your recommendations are a little off in the following ways:
You should use /usr/bin/env bash for the shebang. Some distros, such as nixos, don't have /bin/bash.
You need more quoting in the dirname line. cd "$(dirname "$0")" is what you need. The outer quotes are in case the directory has a space or special character in it, the inner ones are in case the script does.
That being said, you may also want $BASH_SOURCE rather than $0, but the reasons behind that are complicated. You may also want to support cases where the script is a symlink depending on what you're doing.
You should use /usr/bin/env bash for the shebang. Some distros, such as nixos, don't have /bin/bash.
You need more quoting in the dirname line. cd "$(dirname "$0")" is what you need. The outer quotes are in case the directory has a space or special character in it, the inner ones are in case the script does.
That being said, you may also want $BASH_SOURCE rather than $0, but the reasons behind that are complicated. You may also want to support cases where the script is a symlink depending on what you're doing.
Using 'env bash' to find the local path is definitely the way to go. It also supports that occasional case where the user has installed modern bash somewhere in their local path (under their home directory) and finds that rather than the system copy.
#!/usr/bin/env bash
case "$(readlink /proc/$$/exe)" in */bash) set -euo pipefail ;; *) set -eu ;; esac
PATH=/usr/sbin:/usr/bin:/sbin:/bin
\unalias -a
Not to be that guy but this is mostly garbage. I suggest simply paying attention and testing. Switches can be a useful part of the tool but they can break things. It doesn't make things better rather it alters behaviors of the shell in sometimes unintended ways. Try some loops with some -eo. If you have a pipe that fails your script you're just ripping the heart and potential out of your script. I think you might just be looking for one-liners. It can be good to catch your failures and try other things vs "woop, damn, nope." Maybe you need things to fail first.
"Quote liberally" - also ripe. Mind your aPostrophes and Quotes because they do things.
I like writing for me and what I need done, not what or how others might have me write or how others might think it should be done.
The shell is a base orchestration and interface tool for your os. If you think it's just that bad then just stick to your language of choice but please refrain from suggesting lazy habits make a better shell, they don't.
Fly your own kite. It's much more fun.
"Quote liberally" - also ripe. Mind your aPostrophes and Quotes because they do things.
I like writing for me and what I need done, not what or how others might have me write or how others might think it should be done.
The shell is a base orchestration and interface tool for your os. If you think it's just that bad then just stick to your language of choice but please refrain from suggesting lazy habits make a better shell, they don't.
Fly your own kite. It's much more fun.
Ok, "garbage" may have been harsh but how about a fun example? Try this with and without -e
#!/usr/bin/env bash
bell=`tput bel`
tock='Blastoff!'
do_ring()
{
if [ "$1" ]; then # true
echo -n $bell; sleep 0.1
echo -n $bell; sleep 0.1
echo -n $bell; sleep 0.1
echo $tock
else
echo -n $bell; sleep 0.1
fi
}
i=3
while [ "$i" -ge "0" ];
do
if [ $i = 0 ]; then
sleep 0.5
echo ok
else
echo $i $bell
sleep 0.5
fi
i=`expr $i - 1`
done
do_ring $1 # do something different if arg
sleep 1
echo neat.
That's why nobody uses expr for arithmetic. But yes, it is annoying that -e breaks some habits that work fine without it.
I can appreciate the quip, especially given my curmundgeonly entry. But the point remains. The shell needs to be able to call any command on the system reliably. Even aged old counting methods. At no point in that loop would the expr subprocess return other than zero. Try to break out of that loop with that bash switch.
One should not rely on these magics, the ones that alter runtime behaviors, without truly understanding what you're after and what you're writing. Or mostly understand. They're fine and good if you have narrow routines with a very strict focus - "I'm in, I'm out, bang."
The best switches available to the shell without a bunch of feature and package and library this or that bloat are exactly
Every single utility and package on that system is at your fingertips. Demand accuracy. I do.
One should not rely on these magics, the ones that alter runtime behaviors, without truly understanding what you're after and what you're writing. Or mostly understand. They're fine and good if you have narrow routines with a very strict focus - "I'm in, I'm out, bang."
The best switches available to the shell without a bunch of feature and package and library this or that bloat are exactly
-x
-n
Every single utility and package on that system is at your fingertips. Demand accuracy. I do.
It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script.
https://github.com/koalaman/shellcheck