Bashing the Bash – Replacing Shell Scripts with Python (2017)(medium.com)
medium.com
Bashing the Bash – Replacing Shell Scripts with Python (2017)
https://medium.com/capital-one-tech/bashing-the-bash-replacing-shell-scripts-with-python-d8d201bc0989
137 comments
We had some problem, roughly, "run a tool on all YAML files in the directory" or something. The first attempt that led to a production outage:
(It is late, bash examples are approximate. Don't sweat the syntax.)
There weren't any YAML files, so argc was == 0, which the program considered an error. Mea culpea, perhaps.
So finally the contorted,
Use it on the command line in your day to day, fine; but if it's getting automated, jump to at least Python.
(It is late, bash examples are approximate. Don't sweat the syntax.)
the_tool *.yaml
shopt -s nullglob
the_tool *.yaml
There weren't any YAML files, so argc was == 0, which the program considered an error. Mea culpea, perhaps.
AN_ARRAY=(*.yaml)
if [[ < length of the array is not 0 > ]]; then
the_tool "${AN_ARRAY[@]}"
fi
So finally the contorted,
AN_ARRAY=(*.yaml)
if [[ "${AN_ARRAY+x}" = x && < length is not 0 > ]]; then
the_tool "${AN_ARRAY[@]}"
fi
Use it on the command line in your day to day, fine; but if it's getting automated, jump to at least Python.
find . -maxdepth 1 -name '*.yaml' -print0 | xargs -r0 the_tool
If you have a problem that involves "run a command with a list of parameters but not if the list is empty" and you don't immediately think of `xargs -r`, then consider that it's because you don't understand the tools, not because the tools are bad.
> If you have a problem that involves "run a command with a list of parameters but not if the list is empty" and you don't immediately think of `xargs -r`, then consider that it's because you don't understand the tools, not because the tools are bad.
find has -exec, xargs isn't necessary.
find has -exec, xargs isn't necessary.
`-exec` would run the command for each file separately, which isn't what the original problem wanted, and I didn't want to assume that it would be equivalent.
I'll submit that that's a solution.
But it's like, I have to turn my head around and stop doing one thing (globs), and switch over to another thing (find, xargs). The language has baited me down one way of doing it, when I shouldn't be doing it that way. I'm well aware of xargs (but not the -r flag). I typically avoid xargs, as our code has to¹ work on macOS & Linux, and xargs is one of those utilities whose available options differ between the two for even simple use cases. (Such as here, for the "does it exec on no args?".)
But it still proves the point: the equivalent Python is just so much more straight-forward, easier to read, and not apt to shoot me at a moments notice, and doesn't have bugs. A reader only familiar with Python's syntax can read and understand Python, and can write Python that is correct. Not so with bash, you have to understand, "oh, no, globs will screw you here, you need this flag from this utility" — no, it's lunacy.
Like, I can go down the road you're suggesting (and in the past, I did) and keep learning the nooks and crannies and intricacies of bash, each one adding to the nightmare fuel that is my understanding of shell. I can keep tweaking the script around various failure modes, letting each line fail in prod 3 different ways. Or, I can just re-write it in Python, or Rust, or any other sane language and have it never fail again, for reasons that would also either apply to bash, or apply and have worse consequences than bash. (E.g., the Python would raise, bash would just plow on with it.)
Your response is exactly the same as my response before I saw the light on PHP, C, etc.: "you're holding it wrong". No, it's a bad tool that guiles the user into holding it wrong. There are better tools that are easier to hold.
¹b/c despite all work being done for Linux, devs get MBPs, b/c the company can't manage more than exactly 1 type of machine, for everyone.
But it's like, I have to turn my head around and stop doing one thing (globs), and switch over to another thing (find, xargs). The language has baited me down one way of doing it, when I shouldn't be doing it that way. I'm well aware of xargs (but not the -r flag). I typically avoid xargs, as our code has to¹ work on macOS & Linux, and xargs is one of those utilities whose available options differ between the two for even simple use cases. (Such as here, for the "does it exec on no args?".)
But it still proves the point: the equivalent Python is just so much more straight-forward, easier to read, and not apt to shoot me at a moments notice, and doesn't have bugs. A reader only familiar with Python's syntax can read and understand Python, and can write Python that is correct. Not so with bash, you have to understand, "oh, no, globs will screw you here, you need this flag from this utility" — no, it's lunacy.
Like, I can go down the road you're suggesting (and in the past, I did) and keep learning the nooks and crannies and intricacies of bash, each one adding to the nightmare fuel that is my understanding of shell. I can keep tweaking the script around various failure modes, letting each line fail in prod 3 different ways. Or, I can just re-write it in Python, or Rust, or any other sane language and have it never fail again, for reasons that would also either apply to bash, or apply and have worse consequences than bash. (E.g., the Python would raise, bash would just plow on with it.)
Your response is exactly the same as my response before I saw the light on PHP, C, etc.: "you're holding it wrong". No, it's a bad tool that guiles the user into holding it wrong. There are better tools that are easier to hold.
¹b/c despite all work being done for Linux, devs get MBPs, b/c the company can't manage more than exactly 1 type of machine, for everyone.
Bash is amazing at being expressive and allowing for a ton of common things sysadmins / Devops / power users need to do with just one or a few lines of code vs a “proper” language that might require libraries, compiling stuff, or just a ton more SLOC + time debugging. If you don’t know what a glob is going to match, try it first. Understand the semantics. If you need portability, do that too, but a ton of what I end up using bash for is along the lines of slicing up poorly formatted inconsistent lists and running tools against stuff, possibly in conjunction with args from those lists, and for that it’s amazing. Stuff like jq helps a lot, and tools like xmlstarlet before that were great too. I remember trying to rewrite a script that was getting long and AWKward in Python and getting pretty frustrated with how explicit and error-prone it was there. Sure, the result was probably more robust, but it came at the expense of an order of magnitude more dev / debugging time. That’s not always necessary; sometimes “good enough” is good enough, and usually bash is good enough.
Also, how much code would it take to do the same thing in python?
Probably something along the line of
Popen(["the_tool"] + list(Path().glob("*.yml")))This is reasonable in general, though in this case, I think this will replicate the problem of the second Bash version, as the problem there is with the_tool requiring at least one argument.
While dethanatos's point about Bash's default behavior with a non-matching glob is on target, the remainder of the example demonstrates a problem in the_tool, which will need to be worked around regardless of how you invoke it.
While dethanatos's point about Bash's default behavior with a non-matching glob is on target, the remainder of the example demonstrates a problem in the_tool, which will need to be worked around regardless of how you invoke it.
[deleted]
> To say "you're holding it wrong" is just folly. Bash is worthy of the same treatment that "A Fractal of Bad Design" gave PHP, and it has no place in anything that wants to call itself engineering.
is this a cloud AMD64 opinion that ignores the 2 trillion devices which Gartner predicts will never run python?
I recently finished a ~600 lines ash script and the kids at work which just graduated complained that *"Y u no python?" For them it was difficult to understand because apparently I avoided cat/grep/dirname/basename/etc and used shell syntax wherever possible to avoid shelling out. (the thing I did would only run when system reaches a certain load)
Everyone forgot that the target hardware has only busybox (ash) and was massively resource constrained. My only options were to implement what I was doing as an eBPF, in C, or shell (ash). The shell script took 4 days (and was refactored/optimized several times within that period), the C (or eBPF) program would have taken ~10 days and the eBPF which is the best solution is unmaintainable once I leave the company.
from a performance perspective I take a bash script any time. Also python programs over time have a risk of eventually pulling in a dozen dependencies.
is this a cloud AMD64 opinion that ignores the 2 trillion devices which Gartner predicts will never run python?
I recently finished a ~600 lines ash script and the kids at work which just graduated complained that *"Y u no python?" For them it was difficult to understand because apparently I avoided cat/grep/dirname/basename/etc and used shell syntax wherever possible to avoid shelling out. (the thing I did would only run when system reaches a certain load)
Everyone forgot that the target hardware has only busybox (ash) and was massively resource constrained. My only options were to implement what I was doing as an eBPF, in C, or shell (ash). The shell script took 4 days (and was refactored/optimized several times within that period), the C (or eBPF) program would have taken ~10 days and the eBPF which is the best solution is unmaintainable once I leave the company.
from a performance perspective I take a bash script any time. Also python programs over time have a risk of eventually pulling in a dozen dependencies.
Judging by the reaction of the kids at work, what you did is already unmaintainable.
Plus, if you are running shell out of necessity on a constrained system, that doesn't mean shell deserves to be a programming language where there's actual choice.
Plus, if you are running shell out of necessity on a constrained system, that doesn't mean shell deserves to be a programming language where there's actual choice.
> what you did is already unmaintainable.
how so? there is no python or perl or ruby etc on the target. what would have been a maintainable option other than /bin/sh ? C/C++ ?
> where there's actual choice.
again what choice? :) as I said there is no other solution than what I've given. the system literally doesn't have anything else regardless what "I think" it should support.
how so? there is no python or perl or ruby etc on the target. what would have been a maintainable option other than /bin/sh ? C/C++ ?
> where there's actual choice.
again what choice? :) as I said there is no other solution than what I've given. the system literally doesn't have anything else regardless what "I think" it should support.
If the kids refuse to learn anything new (to them), they chose the wrong occupation.
How is bash performant? It is literally interpret by lines, and starting a whole process per if statements..
> How is bash performant?
Typically, time spent interpreting control flow in a bash script is dwarfed by time spent in the programs called by the script.
> It is literally interpret by lines
It's just splitting lines into words to build up commands. It's not compiling C++.
> starting a whole process per if statements..
In the common pattern `if command1; then command2; fi`, if command1 is a builtin (e.g., `test` or `[`), then bash does not create a subprocess.
Typically, time spent interpreting control flow in a bash script is dwarfed by time spent in the programs called by the script.
> It is literally interpret by lines
It's just splitting lines into words to build up commands. It's not compiling C++.
> starting a whole process per if statements..
In the common pattern `if command1; then command2; fi`, if command1 is a builtin (e.g., `test` or `[`), then bash does not create a subprocess.
Still doesn’t make it fast in any meaning of the word, which was my original criticism.
You asked how bash is performant. I responded with reasons its performance is good in many applications. Here”s one more: bash starts up faster than other interpreters (e.g., Python), and startup time can often be a significant factor in overall performance.
Guessing: Automatic parallelization in pipelines, just as long as there isn't a wall that has to consume all its input like "sort".
On the contrary, I am 100% open to suggestions that BASH sucks. (Personally, I would have picked the inadequacies of `set -e` as my favorite ... unfortunate aspect.) My objection was very narrow: this specific article does a poor job at saying why shell sucks, and a worse job still at presenting Python as a superior alternative. That's not to say that good reasons to use something else don't exist, and it's not even to say that Python can't be that alternative. (Your sibling comments suggesting xonsh and https://amoffat.github.io/sh/ paint a fairly compelling way to actually move to python without losing the advantages of shell, for instance.)
Personally, I never use bash for scripting. I use the system's scripting shell, Debian's Dash or NetBSD Ash. Rhetorical question: If Bash was great for shell scripting, why doesn't the OS project choose it for scripting. When I compile the OS kernel and userland, the project (NetBSD) uses shell scripts not Python. Not suggesting the shell is necessarily better but I am suggesting it works and people understand it.
There are many ways to run a command on every .yaml file in a directory. Personally I like using xtrace. Something like:
There are many ways to run a command on every .yaml file in a directory. Personally I like using xtrace. Something like:
echo cd the_directory > 1.sh
echo "test -f *.yaml" >> 1.sh
ls -1 *.yaml|sed 's/^/the_tool /' >> 1.sh
chmod +x 1.sh
less 1.sh
sh -ex 1.sh
My preferred way under those conditions would be more along the lines of "see if a safe command fails, do the desired command if not", like:
if ls *.yaml > /dev/null; then
the_tool *.yaml
fiThat only looks simple because we don't know what problem was being solved.
Is it really right to not process any files just because one of them just disappeared?
In general, situations like this suffer from race conditions and time-of-check/time-of-use problems.
Is it really right to not process any files just because one of them just disappeared?
In general, situations like this suffer from race conditions and time-of-check/time-of-use problems.
> Is it really right to not process any files just because one of them just disappeared?
No idea what you mean here, the glob can expand to one or many.
> In general, situations like this suffer from race conditions and time-of-check/time-of-use problems.
I mean, that's a problem in every version presented so far, one that I don't think can be fixed externally without something extra like hardlinking to a new directory, because "the_tool" apparently can't handle missing files and they can go missing when "the_tool" starts up. That's why I didn't even bother.
No idea what you mean here, the glob can expand to one or many.
> In general, situations like this suffer from race conditions and time-of-check/time-of-use problems.
I mean, that's a problem in every version presented so far, one that I don't think can be fixed externally without something extra like hardlinking to a new directory, because "the_tool" apparently can't handle missing files and they can go missing when "the_tool" starts up. That's why I didn't even bother.
This tool of yours can not cope with files not existing, and causes production outages? Yet somehow you manage to blame bash for this? The outcome would not have been different if systemd or python started your tool.
In fact, handling that types of errors is what bash is suitable for. "If file is readable, process it" is not an unusual construct. Please do take care however to avoid races if this directory is a spool directory that can be populated at any time, in which case it is useful that rename() is atomic, sometimes an additional working directory is required. Again, this is how the operating system is designed and nothing the shell can gloss over.
I have seen many problems from previously unknown race conditions when concurrency is increased. Scripts have to be kept composable and trivial to understand in order to guard against that.
To be honest, I would argue to remove any script that looks like the last example above. It is not clear what side effects of that bizarre array construct are useful to the system. Keep scripts as simple and readable as possible, a trivial truth perhaps, applicable to every language, but one that needs to be reiterated until the end of our profession.
In fact, handling that types of errors is what bash is suitable for. "If file is readable, process it" is not an unusual construct. Please do take care however to avoid races if this directory is a spool directory that can be populated at any time, in which case it is useful that rename() is atomic, sometimes an additional working directory is required. Again, this is how the operating system is designed and nothing the shell can gloss over.
I have seen many problems from previously unknown race conditions when concurrency is increased. Scripts have to be kept composable and trivial to understand in order to guard against that.
To be honest, I would argue to remove any script that looks like the last example above. It is not clear what side effects of that bizarre array construct are useful to the system. Keep scripts as simple and readable as possible, a trivial truth perhaps, applicable to every language, but one that needs to be reiterated until the end of our profession.
> This failed, b/c bash can't represent an empty array!
It absolutely can, I have no idea where you’re getting this idea.
It absolutely can, I have no idea where you’re getting this idea.
What happens if someone runs `the_tool this-file-does-not-exist.yaml` or the yaml file is removed by some other means before you get a chance to process it? Does your world also come tumbling down in these cases? If yes, please stop blaming Bash for it.
Didn't know "Fractal of Bad Design" even existed. I like aspects of PHP. Sorry.
This isn’t you are holding it wrong
It is in the “do as I want not as I say” category
I agree with your sentiment. People like to complain about tools. The nice things about bash (and even sh) are that you can combine a bunch of simple tools together quickly because the syntax doesn't get in the way of it. The language designers didn't intend for people to be writing thousands of lines of scripts; they intended them to be using them locally and focused on interactivity. So naturally the syntax is nice for interactive work. It's a stretch to pretend that working with python paths is easier than working with shell paths, just like it's a stretch to say your bash machine learning library is better than a python one. The author should let things excel at what they're good at and stop pretending everything needs to be unified.
After going over it again, I think I can be more precise: What annoys me is that it's inconsistent to the point of being disingenuous. Like... You want to say error handling is hard to do right? That's fine, it is. You wanna write "Checking the status of programs using $? can be accidentally left off, leading to inconsistent behavior" and then in the same article write in explicit exception handlers, and act like somehow those can't be forgotten!? Get lost.
If you forget an error handler in python, execution terminates. So I don't see your point.
> I was hoping someone finally had a library or something to make python good at the things shell does well (mostly, running stuff and piping output around). If only.
xonsh (https://xon.sh/)
Been using it for a few years now. It's worth it.
xonsh (https://xon.sh/)
Been using it for a few years now. It's worth it.
> I was hoping someone finally had a library or something to make python good at the things shell does well (mostly, running stuff and piping output around). If only.
Good news, they did! Even better news: it doesn't require python, it's cross platform, and supports an easy-to-read verbose style for scripts but a quick-to-write terse style for interactive usage. It's an absolute joy to use for nested data structures, everything is an object and it has full native tab completion.
Unfortunately it came out of Microsoft so a lot of people haven't given it the time of day yet. It's honestly one of the joys of modern CLI usage though.
Good news, they did! Even better news: it doesn't require python, it's cross platform, and supports an easy-to-read verbose style for scripts but a quick-to-write terse style for interactive usage. It's an absolute joy to use for nested data structures, everything is an object and it has full native tab completion.
Unfortunately it came out of Microsoft so a lot of people haven't given it the time of day yet. It's honestly one of the joys of modern CLI usage though.
What is Powershell
I just added powershell support to my Debian.
$> ll /bin/powershell
lrwxrwxrwx 1 root root 9 Mar 26 14:40 /bin/powershell -> /bin/dash
$> cat ~/pwr_to_the_people.sh
#!/bin/powershell
echo "Hello Powershell"
$> powershell ~/pwr_to_the_people.sh
Hello Powershell
A dotnet based shell where you can share around dotnet object instead of plain text.
> obscure and difficult-to-predict
I mean, I think there's something to that. Bash has a lot of arcane rules; I personally never write an `if` statement right, what with the `[[` vs `[` and having to surround the condition with whitespace. I find that bash scripts carry a million little gotchas that will cause variables to not evaluate/expand the way you'd expect.
Say what you will about Python, but it's much easier to reason about what a script will do as compared to bash (unless the bash is _extremely_ simplistic).
I mean, I think there's something to that. Bash has a lot of arcane rules; I personally never write an `if` statement right, what with the `[[` vs `[` and having to surround the condition with whitespace. I find that bash scripts carry a million little gotchas that will cause variables to not evaluate/expand the way you'd expect.
Say what you will about Python, but it's much easier to reason about what a script will do as compared to bash (unless the bash is _extremely_ simplistic).
> I was hoping someone finally had a library or something to make python good at the things shell does well (mostly, running stuff and piping output around). If only.
I've only kicked the tires on both, but plumbum:
https://plumbum.readthedocs.io/en/latest/
and sh: https://amoffat.github.io/sh/
both feel pretty good in that regard.
I've only kicked the tires on both, but plumbum:
https://plumbum.readthedocs.io/en/latest/
and sh: https://amoffat.github.io/sh/
both feel pretty good in that regard.
Even if you have a great library that abstracts away shell like things, you’d still probably be better served with a shell script. Shell scripts aren’t supposed to be big. They are quick one offs that solve immediate problems. This means that they are also spread around the filesystem. I’m of the opinion that one shouldn’t install libraries into the system Python path. I also tend to make heavy use of virtual envs. Setting up a venv for each off scripts doesn’t seem like a great plan.
For more centralized workflows, I could see Python shell replacements being helpful. Especially project specific scripts where the overhead of managing a venv is negligible.
But realistically replacing the majority of bash scripts? I just don’t see it.
For more centralized workflows, I could see Python shell replacements being helpful. Especially project specific scripts where the overhead of managing a venv is negligible.
But realistically replacing the majority of bash scripts? I just don’t see it.
https://github.com/MoserMichael/subb
https://pypi.org/project/subb/
I wrote this to simplify just that for my own tools; the subprocess module that comes with the batteries has a very general interface, i think that it is a bit complex for a quick script.
My objective was to get an abstraction, for a one line process run and extraction of the result, similar to what we had in Perl5 with the system library function. https://perldoc.perl.org/functions/system
Also the shell is impractical, when it comes to slightly more complex programs. There is a limit on what you can do with pipes. Maybe that's the reason why perl is that flexible, as they tried to bridge both realms: Perl had to be useful as a replacement for the quick shell like script, and to be useful as a general purpose programming language.
https://pypi.org/project/subb/
I wrote this to simplify just that for my own tools; the subprocess module that comes with the batteries has a very general interface, i think that it is a bit complex for a quick script.
My objective was to get an abstraction, for a one line process run and extraction of the result, similar to what we had in Perl5 with the system library function. https://perldoc.perl.org/functions/system
Also the shell is impractical, when it comes to slightly more complex programs. There is a limit on what you can do with pipes. Maybe that's the reason why perl is that flexible, as they tried to bridge both realms: Perl had to be useful as a replacement for the quick shell like script, and to be useful as a general purpose programming language.
I think this article is supposed to commend Python as a better choice than Bash? I think the author forgot to make a convincing argument for that, though.
> The point of bash-bashing is to reduce use of the shell.
That's a tautology. I get it, we're supposed to reduce the use of bash ... but why?
> Without much real work, it’s easy to replace shell scripts with Python code.
But you're writing the same thing again? There needs to be more reason than this.
> The revised code is easier to read and maintain, runs a little faster, and can have a proper unit test suite.
Three claims. Any data to support it?
Some of the examples of "bash is bad" are also not convincing to me. Here's an example:
> An example of a shell obscurity is the way the current working directory is set. The cd command is clear enough, but in the presence of sub-shells using (), can make it difficult to discern a stack of nested shell invocations and how the working directory changes when the sub-shells exit.
So...why do you have "a stack of nested shell invocations" that all need to be working directory aware? And how would this be any better in Python? (It could be better in either Python or Bash but either one still requires the developer to avoid simple mistakes.)
I don't know. I don't get it. I feel like the author had the unfortunate experience of knowing more Python than Bash and inheriting someone else's (who also didn't have much Bash experience) crappy Bash code.
> The point of bash-bashing is to reduce use of the shell.
That's a tautology. I get it, we're supposed to reduce the use of bash ... but why?
> Without much real work, it’s easy to replace shell scripts with Python code.
But you're writing the same thing again? There needs to be more reason than this.
> The revised code is easier to read and maintain, runs a little faster, and can have a proper unit test suite.
Three claims. Any data to support it?
Some of the examples of "bash is bad" are also not convincing to me. Here's an example:
> An example of a shell obscurity is the way the current working directory is set. The cd command is clear enough, but in the presence of sub-shells using (), can make it difficult to discern a stack of nested shell invocations and how the working directory changes when the sub-shells exit.
So...why do you have "a stack of nested shell invocations" that all need to be working directory aware? And how would this be any better in Python? (It could be better in either Python or Bash but either one still requires the developer to avoid simple mistakes.)
I don't know. I don't get it. I feel like the author had the unfortunate experience of knowing more Python than Bash and inheriting someone else's (who also didn't have much Bash experience) crappy Bash code.
I've written extensive programs in Haskell, and yet I also have many Bash scripts.
I go back and forth. For the longest time I ignored Perl and just wrote scripts in C. Then I learned Perl, then Python. I prefer Ruby for no better reason than it's less boring. These scripting languages share a common strength: nested hash tables make everything easier. Still, I'd revert to Bash, explaining that if you can't accomplish the same thing with virtual text files in Bash you don't really understand Bash.
Nevertheless, switching from Bash to Ruby is like taking off a heavy backpack after a long day hiking. Who needs a joint when the weight is off your shoulders?
I go back and forth. For the longest time I ignored Perl and just wrote scripts in C. Then I learned Perl, then Python. I prefer Ruby for no better reason than it's less boring. These scripting languages share a common strength: nested hash tables make everything easier. Still, I'd revert to Bash, explaining that if you can't accomplish the same thing with virtual text files in Bash you don't really understand Bash.
Nevertheless, switching from Bash to Ruby is like taking off a heavy backpack after a long day hiking. Who needs a joint when the weight is off your shoulders?
And all this time I thought the common strength of "scripting languages" is that they're JITted.
Otherwise they'd just be plain old compiled "languages."
Otherwise they'd just be plain old compiled "languages."
bash is anti-bicycle: no matter how long you use, it trips you up. https://www.oilshell.org/why.html
Shell is great for one liners to run commands, cancerous for anything more complex. Fabric combines the best of both worlds: shell for one liners, Python for more complex logic https://docs.fabfile.org/en/2.6/getting-started.html#addendu...
Shell is great for one liners to run commands, cancerous for anything more complex. Fabric combines the best of both worlds: shell for one liners, Python for more complex logic https://docs.fabfile.org/en/2.6/getting-started.html#addendu...
In my experience, it trips me up no more than Python or Fabric do. I have used Fabric a lot too. At another job, we built all of our pipeline tooling in it.
I somehow keep returning to Bash, though. I enjoy it.
I somehow keep returning to Bash, though. I enjoy it.
[deleted]
Interesting, I used fabric many years ago but thought it had been deprecated. However, it still seems remote focused. Is that a problem for a shell replacement?
As far as I understand, Invoke is pretty much local Fabric. Or rather, Fabric is a networking layer on top of Invoke.
Thanks, looks like it has some similarities with make, however doesn't skip things already built.
Any data to support it?
There isn't any "data" on these things. It's a qualitative, holistic assessment of the risk-benefit tradeoff between two choices. Like much of life in general, in fact.
There isn't any "data" on these things. It's a qualitative, holistic assessment of the risk-benefit tradeoff between two choices. Like much of life in general, in fact.
I have some bad news for you, from a long, long, long time CS/EE/Sysadmin person:
I can probably run my bash script on every linux I ever touched.
I can probably have that python code break on half of the linux boxen I use. This one does not have module X installed. This one is too old, this one is too new, this python is not holding its mouth just at the right angle to work.
Yeah, bash bash all you want. I have pulled out scripts decades old and run them. I can not say the same for other things I have written.
I can probably run my bash script on every linux I ever touched.
I can probably have that python code break on half of the linux boxen I use. This one does not have module X installed. This one is too old, this one is too new, this python is not holding its mouth just at the right angle to work.
Yeah, bash bash all you want. I have pulled out scripts decades old and run them. I can not say the same for other things I have written.
I have the same issue with bash scripts. One host has jq, another is missing it. One host has yq version 2 with completely different syntax than yq version 4.
If you switch from Linux to *nix where you introduce BSD variants (like macOS) it gets even worse. Python dependency management leaves much to be desired but bash has none.
If you switch from Linux to *nix where you introduce BSD variants (like macOS) it gets even worse. Python dependency management leaves much to be desired but bash has none.
That's why you try to write scripts for standard tools and shell features
What are standard tools and shell features? Bash + GNU? Those might be consistently installed on common Linux distros but that doesn't work across *nix systems (Linux, macOS, BSD). That probably won't work against less-common Linux setups like anything embedded (busybox or container images) either
But which one is more likely to also run on Windows? /s
... Hilariously, I think shell might actually win that; neither is installed by default, but you get some sort of bash/sh with any of interix, WSL1/2, git, cygwin, or mingw.
isn't WSL like a hypervisor that runs a guest? I'm clueless about Windows but when I start a WSL it is usually some form of Debian or Ubuntu that I can download from the Microsoft "store". Therefore isn't the shell you have whatever comes with that guest?
>I can probably run my bash script on every linux I ever touched.
Exactly THIS !
Exactly THIS !
yeah python (and before perl with cpan) had these issues that maintaining a consistent system is more effort than writing the damn thing in the first place. I loved Perl but tried to avoid modules like the plague. And python is almost worse because the same mentality (avoid modules) was abandoned by most of the community so now you get incompatibilities with what the OS provides and what gets installed via pip.
Perhaps it's unfair to pick at python for this when many of these issues are not the language but because people picking python when they should have just used shell script.
Perhaps it's unfair to pick at python for this when many of these issues are not the language but because people picking python when they should have just used shell script.
I have ksh scripts I wrote over 20 years ago on xenix, which still work today on bash or ksh. Meanwhile I have python apps that didn't make it a year.
Any particular version of Python is a saner choice, if you could pick one and freeze it retroactively for the last 20 years and for at least the next 20. But that is not how Python works. Python breaks every 11 minutes.
I would not write anything in Python that I cared the slightest bit about longevity or portability.
Maybe if we made a subset of python, locked down the syntax and feature set, discouraged any use of plugins or libraries in any fancy way that relies on any kind of repository or package manager system, and gave it a new name to distinguish it from normal python, maybe that would be an ok replacement for any of the shells.
But that is not happeing and don't even try to pretend like it could.
Any particular version of Python is a saner choice, if you could pick one and freeze it retroactively for the last 20 years and for at least the next 20. But that is not how Python works. Python breaks every 11 minutes.
I would not write anything in Python that I cared the slightest bit about longevity or portability.
Maybe if we made a subset of python, locked down the syntax and feature set, discouraged any use of plugins or libraries in any fancy way that relies on any kind of repository or package manager system, and gave it a new name to distinguish it from normal python, maybe that would be an ok replacement for any of the shells.
But that is not happeing and don't even try to pretend like it could.
Over the past decade I've been drifting towards doing more and more in bash that I'd've normally done one one language or another just because of this stability. Perl is another candidate if I need something more complicated, but I'm not as familiar with it so I don't really reach for it often.
Are there any other languages with the same stability that would also fill this niche?
Are there any other languages with the same stability that would also fill this niche?
I've learnt Perl for this very reason. It is extremely capable, extensible and still very backwards-compatible. It feels like an evolution of bash but much more performant.
A bit of a shame that it's seen as an anachronism to many. Perl soon rubs off on you and becomes a delight.
A bit of a shame that it's seen as an anachronism to many. Perl soon rubs off on you and becomes a delight.
For shell-scripting sorts of tasks, there’s also awk which I find a nice sort of halfway between bash and Perl.
Also, I’ve been using various lisps (elisp in emacs org-mode and Common Lisp) and they both can be written in ways that will probably last a long time.
Also, I’ve been using various lisps (elisp in emacs org-mode and Common Lisp) and they both can be written in ways that will probably last a long time.
Awk is a strange seeming suggestion, but I agree it's probably the next most universal, installed, stable over time option, and more readable than sh.
Using awk as a general purpose language means slightly bending it out of shape, by mostly ignoring the main section (which runs once per selected record of input) and writing your entire program in a big END{} section.
BUT
* It's installed everywhere and has been forever just like sh
* A 20 year old awk script from hpux or something, works in todays gawk, just like sh.
* Unlike sh, the language is more like a normal generic language than, making it more readable and less arcane for non-trivial jobs.
For example, bash has a handy substitute feature, in the form of a brace expansion. ${FOO//a/b}
Awk has an actual sub() or gsub() function.
A lot of things in bash require tricks essentially abusing various fancy brace expansions and messing with IFS to hijack the line parser into doing things there is no explicit command for, especially if you're avoiding external commands. IE you're not simply writing code that does what it says, you're running a bash interpreter in your head and manipulating strings so that when they are expanded and resolved, they mean something to the the parser and results in the parser doing something more useful. It's like your always writing code that writes code, instead of just writing code.
External dependencies is another whole point too. Shells main explicit job is to run other programs. You actually have to work pretty hard not to run an external binary by accident, ie you have know which commands are built in and which are executables. In awk or anything else, you have to go out of your way to exec() or system().
Using awk as a general purpose language means slightly bending it out of shape, by mostly ignoring the main section (which runs once per selected record of input) and writing your entire program in a big END{} section.
BUT
* It's installed everywhere and has been forever just like sh
* A 20 year old awk script from hpux or something, works in todays gawk, just like sh.
* Unlike sh, the language is more like a normal generic language than, making it more readable and less arcane for non-trivial jobs.
For example, bash has a handy substitute feature, in the form of a brace expansion. ${FOO//a/b}
Awk has an actual sub() or gsub() function.
A lot of things in bash require tricks essentially abusing various fancy brace expansions and messing with IFS to hijack the line parser into doing things there is no explicit command for, especially if you're avoiding external commands. IE you're not simply writing code that does what it says, you're running a bash interpreter in your head and manipulating strings so that when they are expanded and resolved, they mean something to the the parser and results in the parser doing something more useful. It's like your always writing code that writes code, instead of just writing code.
External dependencies is another whole point too. Shells main explicit job is to run other programs. You actually have to work pretty hard not to run an external binary by accident, ie you have know which commands are built in and which are executables. In awk or anything else, you have to go out of your way to exec() or system().
That's what Python2.7 is and why every major proprietary product won't adopt anything else. It's also why it won't go away.
All of my Python 2.6-2.7 scripts haven't been touched in 10 years in production and they aren't likely to ever be updated. I actually now refuse to write new Python that isn't compatible with both 2&3 for this reason. Python3 refuses to stabilize.
All of my Python 2.6-2.7 scripts haven't been touched in 10 years in production and they aren't likely to ever be updated. I actually now refuse to write new Python that isn't compatible with both 2&3 for this reason. Python3 refuses to stabilize.
I’ve been slowly embracing the mantra “only use unmaintained software”: maintenance is great when it’s fixing bugs and such, but eventually people insist on breaking backwards compatibility.
Python2.7 is still maintained.
If you mean my software I mentioned if something breaks I'm still required to fix it. I just doubt that will happen at this point and have other tasks to do.
If you mean my software I mentioned if something breaks I'm still required to fix it. I just doubt that will happen at this point and have other tasks to do.
Isn’t Python 2.7 EOL as of January 2020 or something? It’s maintained in the sense that there are still people shipping patches, but the core Python developers no longer maintain it.
That is news to me, and good to know.
https://www.bleepingcomputer.com/news/software/python-27-rea...
Apparently RHEL will maintain any security problems until 2024, but yeah it's pretty much dead. Huh. Not sure how to react to that.
https://www.bleepingcomputer.com/news/software/python-27-rea...
Apparently RHEL will maintain any security problems until 2024, but yeah it's pretty much dead. Huh. Not sure how to react to that.
That is a captivating idea!
> It's also why it won't go away.
Python 2 went away in macOS. Bash still exists.
Python 2 went away in macOS. Bash still exists.
https://www.python.org/downloads/release/python-2718/
There's still a macOS installer for the latest release of Python2.7 (4/20/2020).
There's still a macOS installer for the latest release of Python2.7 (4/20/2020).
The discussion was around longevity, and Python 2 is dead as of January 2020 [0]. Installer might exist, but it has already become more difficult to maintain Python 2. It will become even more difficult in time. Bash, on the other hand, will more likely to be around for longer.
[0] Sunsetting Python 2. https://www.python.org/doc/sunset-python-2/
[0] Sunsetting Python 2. https://www.python.org/doc/sunset-python-2/
Did it really?
`/usr/bin/python` invokes Python 2.7.18 on my just-bought Macbook Pro 14 running MacOS 12.2. Maybe that's because I installed the Xcode command line tools?
`/usr/bin/python` invokes Python 2.7.18 on my just-bought Macbook Pro 14 running MacOS 12.2. Maybe that's because I installed the Xcode command line tools?
IIRC, Apple announced that Python 2 will be removed in a future version of macOS, without specifying a version. Looks like they removed it in macOS 12.3 [0].
[0] https://www.macrumors.com/2022/01/28/apple-removing-python-2...
[0] https://www.macrumors.com/2022/01/28/apple-removing-python-2...
I myself kind of went full circle. I started mostly python, then started to really embrace bash, mainly because I joined a company where everyone else used it a ton, got pretty decent at it. Even wrote basically a primitive container orchestrator in Bash because my PM didn't want to use anything "newfangled". Then I joined a startup at basically the ground floor, and it's almost entirely pure python, with some Makefiles to automate common operations. All that to say, I have a lot of experience with both paradigms.
And frankly...bash kinda sucks as a programming language and environment. It's a footgun factory. Even when I was most fluent, I constantly had to deal with nested quotes, escaping, stringly types, empty variables, implicit behavior.
Python's biggest weak point is definitely its packaging and dependency ecosystem... but it at least has one. Bash doesn't even have modules. And it's gotten MUCH better over the years, with pyproject.toml, poetry, actual version resolvers, pipx makes managing venvs for cli tools way easier.
"but bash is everywhere" says many folks. yeah, and? There was a point in time where it wasn't. Python is seeing way more market penetration.
Xonsh (python based shell) and plumbum (python library which makes pipe-operating and subprocessing easier) are both great.
And frankly...bash kinda sucks as a programming language and environment. It's a footgun factory. Even when I was most fluent, I constantly had to deal with nested quotes, escaping, stringly types, empty variables, implicit behavior.
Python's biggest weak point is definitely its packaging and dependency ecosystem... but it at least has one. Bash doesn't even have modules. And it's gotten MUCH better over the years, with pyproject.toml, poetry, actual version resolvers, pipx makes managing venvs for cli tools way easier.
"but bash is everywhere" says many folks. yeah, and? There was a point in time where it wasn't. Python is seeing way more market penetration.
Xonsh (python based shell) and plumbum (python library which makes pipe-operating and subprocessing easier) are both great.
I'm fully in favor of replacing shell scripts with Python3 scripts wherever possible. In fact that's part of my job. That being said, the author of this article is using confusing / wrong terminology to discuss bash.
This article says that, "The [bash] shell isn't a complete programming language". While I agree that bash lacks any real data types besides strings, bash is a Turing-complete programming language[1], so it's theoretically possible to write _any_ program in bash that can be written in Python. It might be a hell of a lot uglier and lack things like imports, modules, etc., but it can be done.
For example, there is an implementation of an HTTP daemon written purely in bash[2]. Once again-- terrible idea, but great execution.
[1]: https://en.wikibooks.org/wiki/Bash_Shell_Scripting#A_few_not...
[2]: https://github.com/avleen/bashttpd
edit: added newline to separate references
This article says that, "The [bash] shell isn't a complete programming language". While I agree that bash lacks any real data types besides strings, bash is a Turing-complete programming language[1], so it's theoretically possible to write _any_ program in bash that can be written in Python. It might be a hell of a lot uglier and lack things like imports, modules, etc., but it can be done.
For example, there is an implementation of an HTTP daemon written purely in bash[2]. Once again-- terrible idea, but great execution.
[1]: https://en.wikibooks.org/wiki/Bash_Shell_Scripting#A_few_not...
[2]: https://github.com/avleen/bashttpd
edit: added newline to separate references
> For example, there is an implementation of an HTTP daemon written purely in bash[2]. Once again-- terrible idea, but great execution.
It's not pure bash, as it relies on netcat or socat plus some other external programs (ls, tree, cat, date).
There does exist a pure bash httpd though. It relies on a loadable builtin (from the bash tree though not built by default).
https://github.com/dzove855/Bash-web-server
It's not pure bash, as it relies on netcat or socat plus some other external programs (ls, tree, cat, date).
There does exist a pure bash httpd though. It relies on a loadable builtin (from the bash tree though not built by default).
https://github.com/dzove855/Bash-web-server
As much as I appreciate Python, I much rather use Perl as a replacement for complex shell scripts. Perl 5 is ubiquitous, it has been stable for the best part of the last 20 years and it's unbeatable for all those tasks that involve heavy text processing.
Python is good for complex, structured applications, but shell scripts are not that. Shell scripts are glue, and there's arguably no better glue than Perl.
Python is good for complex, structured applications, but shell scripts are not that. Shell scripts are glue, and there's arguably no better glue than Perl.
I am well versed in the shell, perl and python.
I first learned the shell, and pretty much stuck to it conservatively avoiding bash extensions.
Later, I learned perl (around version 4)
my first impression of perl was annoyance.
$foo = 123;
seemed silly because there was a dollar sign on the left side of the assignment. and... $foo, @foo, $', $_, s/abc/def/;
all the syntax seemed needlessly cryptic and reinforced the "perl is a write-only language" idea in my mind.
But then something happened.
At some point all of these idioms went away as I became fluent and could think in perl.
Because there were many ways to do something in perl, I found that it become VERY easy to get an idea in my head out into working code.
So perl was to me my most expressive language.
But what I've noticed is that many people haven't overcome the syntax barrier and perl looks like line noise to them. More worrying is the fact that everyone can get their ideas into perl, but because people solve problems so differently their perl can be vastly different from what I am accustomed to.
So a few years later I learned python. The indentation requirement was a minor annoyance, but that was quickly overcome by the consistency and visual structure it added.
A more major annoyance was that many things were harder to implement in python. Regular expressions is a huge one - they are a major feature of perl and I had learned to use them liberally.
Yet I persisted and with help from the many "batteries included" with python, I was easily writing portable maintainable scripts in python.
And they were still readable after 6 months.
Now I write only small shell scripts, and past a certain size all other scripting is in python. It's quite easy to write meaningful scripts in python. (I've pretty much lost my fluency in perl via python -- I fumble around now looking at or modifying perl scripts)
by the way argparse is quite easily my favorite import in python.
I first learned the shell, and pretty much stuck to it conservatively avoiding bash extensions.
Later, I learned perl (around version 4)
my first impression of perl was annoyance.
$foo = 123;
seemed silly because there was a dollar sign on the left side of the assignment. and... $foo, @foo, $', $_, s/abc/def/;
all the syntax seemed needlessly cryptic and reinforced the "perl is a write-only language" idea in my mind.
But then something happened.
At some point all of these idioms went away as I became fluent and could think in perl.
Because there were many ways to do something in perl, I found that it become VERY easy to get an idea in my head out into working code.
So perl was to me my most expressive language.
But what I've noticed is that many people haven't overcome the syntax barrier and perl looks like line noise to them. More worrying is the fact that everyone can get their ideas into perl, but because people solve problems so differently their perl can be vastly different from what I am accustomed to.
So a few years later I learned python. The indentation requirement was a minor annoyance, but that was quickly overcome by the consistency and visual structure it added.
A more major annoyance was that many things were harder to implement in python. Regular expressions is a huge one - they are a major feature of perl and I had learned to use them liberally.
Yet I persisted and with help from the many "batteries included" with python, I was easily writing portable maintainable scripts in python.
And they were still readable after 6 months.
Now I write only small shell scripts, and past a certain size all other scripting is in python. It's quite easy to write meaningful scripts in python. (I've pretty much lost my fluency in perl via python -- I fumble around now looking at or modifying perl scripts)
by the way argparse is quite easily my favorite import in python.
parser = argparse.ArgumentParser(argument_default=None)
parser.add_argument('-d', '--debug', action='store_true', help='debug flag')
parser.add_argument('-c', '--config', default="~/.config", help='config file name')
arg = parser.parse_args()
if arg.debug:
print('config file: %s'%arg.config)
...The one thing about Python that makes it so much more compelling over Bash is that thing you discover you need once your script starts getting long and repetitive: functions, with arguments, and return values. Python does this much better than Bash. It’s the one thing that makes me no longer use Bash. Oh and exceptions. Exceptions!
The TWO things about Python that make it more compelling are functions and exceptions. You stand a chance of handling errors in Python. In bash there’s very little else you can do, when encountering an exceptional situation that is an error, except stop. The nice thing about all these functions is you can put them in modules as well. Modules! Of course! A namespaced way of laying out non trivial amounts of code!
Ok so real functions, exceptions, modules. Three things that makes Python my default tool of choice. And libraries. Bash doesn’t have pip. Sure, bash’s “pip” is the commands it can run aka /usr/bin, but if that’s your interface then it’s hardly as flexible as say gitlab.py or requests.py.
So: apart from the functions, exceptions, modules, libraries. And speed. SPEED! And code formatting. And a debugger. And stack traces. …what has Python ever given us that makes it a no brainer replacement for bash scripts?
“Brought Unicode”
”Unicode! Oh shut up!”
— After MP
The TWO things about Python that make it more compelling are functions and exceptions. You stand a chance of handling errors in Python. In bash there’s very little else you can do, when encountering an exceptional situation that is an error, except stop. The nice thing about all these functions is you can put them in modules as well. Modules! Of course! A namespaced way of laying out non trivial amounts of code!
Ok so real functions, exceptions, modules. Three things that makes Python my default tool of choice. And libraries. Bash doesn’t have pip. Sure, bash’s “pip” is the commands it can run aka /usr/bin, but if that’s your interface then it’s hardly as flexible as say gitlab.py or requests.py.
So: apart from the functions, exceptions, modules, libraries. And speed. SPEED! And code formatting. And a debugger. And stack traces. …what has Python ever given us that makes it a no brainer replacement for bash scripts?
“Brought Unicode”
”Unicode! Oh shut up!”
— After MP
> And speed. SPEED!
Maybe I'm just a terrible python programmer and also all those around me suck at it but in 20+ years I have yet to see a well optimized python script faster than a well optimized bash/ash/csh/ksh/zsh script. Maybe your point is true if a script constantly does foo=`some thing` and constantly shells out other commands.
Maybe I'm just a terrible python programmer and also all those around me suck at it but in 20+ years I have yet to see a well optimized python script faster than a well optimized bash/ash/csh/ksh/zsh script. Maybe your point is true if a script constantly does foo=`some thing` and constantly shells out other commands.
I was thinking of the time I calculated IPV6 EUI64 suffixes in bash for a kind of janky IPAM thing, and how even the barest minimum of arithmetic turned “instantaneous” into “takes seconds”.
The bash-centric solution was to write a little eui64.c and call that from the script. Bash isn’t for doing things, it’s for making other programs do things. That worked fine for a bit, but in the end it just became easier to factor and reason about the project when it wasn’t a weird mixture of clever things, and one Pythonic lump of boring and predictable things.
The bash-centric solution was to write a little eui64.c and call that from the script. Bash isn’t for doing things, it’s for making other programs do things. That worked fine for a bit, but in the end it just became easier to factor and reason about the project when it wasn’t a weird mixture of clever things, and one Pythonic lump of boring and predictable things.
Typically you don't want to do dependency management for simple scripts, but that's fine because python is batteries included
In my experience, bash makes the simple things simple and the hard things (like complex pipelines, validating arguments, or “pure bash”) really hard. YMMV.
Python makes everything medium difficulty which can be a big win depending on the size of automation required.
Python makes everything medium difficulty which can be a big win depending on the size of automation required.
I decided a while ago to write all my future scripts in Python whenever possible instead of sh or bash due to the overall better syntax and power of Python.
Typer is one of the better packages for running Python scripts from CLI. Fabric & Invoke are pretty good too.
To run any bash commands in Python I have a Python script that creates a temporary bash file with the code to execute, makes it executable, runs it with the commands in the built-in subprocess package, then deletes the file. This makes any complex bash commands with piping and whatnot runnable straight from Python.
This way I don't have to learn all the different bash-based Python commands from pathlib and whatnot that throw unexpected errors, and I can run them using pure bash syntax from Python. But then you also get all the benefits of Python's looping syntax, classes etc.
Typer is one of the better packages for running Python scripts from CLI. Fabric & Invoke are pretty good too.
To run any bash commands in Python I have a Python script that creates a temporary bash file with the code to execute, makes it executable, runs it with the commands in the built-in subprocess package, then deletes the file. This makes any complex bash commands with piping and whatnot runnable straight from Python.
This way I don't have to learn all the different bash-based Python commands from pathlib and whatnot that throw unexpected errors, and I can run them using pure bash syntax from Python. But then you also get all the benefits of Python's looping syntax, classes etc.
This feels like:
"Moving furniture by throwing it in the back of a pickup truck is obviously bad. Here's how to do it with a forklift and 18-wheeler instead, because this is definitely better."
"Moving furniture by throwing it in the back of a pickup truck is obviously bad. Here's how to do it with a forklift and 18-wheeler instead, because this is definitely better."
Beware of Python devs writing Python shell scripts. They're used to using all the brand new (backwards incompatible) language features as soon as they come out. The vast majority have no consideration for a 10 year old PC running a 10 year old OS. The python3.x they write will not run.
Bash, surprisingly, gets backwards incompatible language features too. But the vast majority of bash developers are writing for all computers, not just computers with software from the last 3 years.
Python language could work if you stuck to an old target. But python culture is a problem and makes this very unlikely. What python is, and what python devs deal with, changes constantly. That doesn't work for shell scripts where stability is king.
Bash, surprisingly, gets backwards incompatible language features too. But the vast majority of bash developers are writing for all computers, not just computers with software from the last 3 years.
Python language could work if you stuck to an old target. But python culture is a problem and makes this very unlikely. What python is, and what python devs deal with, changes constantly. That doesn't work for shell scripts where stability is king.
> The vast majority have no consideration for a 10 year old PC running a 10 year old OS. The python3.x they write will not run.
Python3 is just under 14 years old.
Python3 is just under 14 years old.
Yeah, but nobody's targeting Python 3.0; I think the point was that the Python crowd is likely to exclusively target current versions and lean on 3.6 and under being officially EOL.
If you are running a 10-year old version of python 3, and you try and install some recently written python 3 application, it will never work - you'll be stuck in dependency hell, even getting pip to run may be a challenge due to outdated certs or TLS versions.
I wrote this to simplify just that for my own tools; the subprocess module that comes with the batteries has a very general interface, i think that it is a bit complex for a quick script.
https://github.com/MoserMichael/subb
https://pypi.org/project/subb/
Python doesn't have the problem of shell scripting language, it doesn't get impractical, as the program is getting more complex. In bash you have arrays, and even maps, but these aren't pretty. Also the shell scripting language is being evaluated by an parse tree/AST interpreter, that's significantly slower than even python, in it's byte code interpreted form.
My objective was to get an abstraction, for a one line process run and extraction of the result, similar to what we had in Perl5 with the system library function. https://perldoc.perl.org/functions/system
Also the shell is impractical, when it comes to slightly more complex programs. There is a limit on what you can do with pipes. Maybe that's the reason why perl is that flexible, as they tried to bridge both realms: Perl had to be useful as a replacement for the quick shell like script, and to be useful as a general purpose programming language.
https://github.com/MoserMichael/subb
https://pypi.org/project/subb/
Python doesn't have the problem of shell scripting language, it doesn't get impractical, as the program is getting more complex. In bash you have arrays, and even maps, but these aren't pretty. Also the shell scripting language is being evaluated by an parse tree/AST interpreter, that's significantly slower than even python, in it's byte code interpreted form.
My objective was to get an abstraction, for a one line process run and extraction of the result, similar to what we had in Perl5 with the system library function. https://perldoc.perl.org/functions/system
Also the shell is impractical, when it comes to slightly more complex programs. There is a limit on what you can do with pipes. Maybe that's the reason why perl is that flexible, as they tried to bridge both realms: Perl had to be useful as a replacement for the quick shell like script, and to be useful as a general purpose programming language.
[deleted]
I didn't even read the article. The reason is that it depends what you need to do, from there you should decide what's the best tool to solve something. Recently I've started to automate all the package installations and configurations for some of the machines I have. I can't even imagine doing that in Python despite I've been using Python for a long time. I don't know that much about bash, but I got everything I needed and expected from bash scripts for the task I needed to do.
Yes, there are moments when you try to solve something in bash that you think is trivial in other languages but with bash is painful. Few days ago tried to test if a value is present in a bash array using a function. Couldn't get it working after trying few things. Then I realized I could perform a very similar (and good enough) check just seeing if a directory and a file do exist in my relative path to the script I was running: `if [-d DIR ] && [ -f FILE ]; then ...`.
So I needed to change my mindset a bit. Another example of where bash is really good is for tiny small tasks: I wrote a small script to increase/decrease volume that is coupled in i3 and calls the `mixer` command in FreeBSD. Works perfectly and never touched once it was working. Another one: I get an acoustic alert when my battery drops down below certain percentage. And a couple more: set a random background every time I log in to i3, download a page for offline reading.
Again, doing any of these things (or similar ones) with something different than bash (or maybe your favorite shell lang) seems like picking up the wrong tool for the job. There was a time when I cursed shell scripting a lot and that was because I couldn't see when to use shell scripting.
EDIT: Paragraph spacing.
Yes, there are moments when you try to solve something in bash that you think is trivial in other languages but with bash is painful. Few days ago tried to test if a value is present in a bash array using a function. Couldn't get it working after trying few things. Then I realized I could perform a very similar (and good enough) check just seeing if a directory and a file do exist in my relative path to the script I was running: `if [-d DIR ] && [ -f FILE ]; then ...`.
So I needed to change my mindset a bit. Another example of where bash is really good is for tiny small tasks: I wrote a small script to increase/decrease volume that is coupled in i3 and calls the `mixer` command in FreeBSD. Works perfectly and never touched once it was working. Another one: I get an acoustic alert when my battery drops down below certain percentage. And a couple more: set a random background every time I log in to i3, download a page for offline reading.
Again, doing any of these things (or similar ones) with something different than bash (or maybe your favorite shell lang) seems like picking up the wrong tool for the job. There was a time when I cursed shell scripting a lot and that was because I couldn't see when to use shell scripting.
EDIT: Paragraph spacing.
OK, I'll bite... The author would complain about my python3 in the same way and conclude it would be better to write it in my Bash.
Without questioning the "why" of the script, sans my customary morning coffee, and without having tested the code below.
Maybe I'd do something like this.
For the small price of function invocation overhead, the nice benefit of doing it this way is that one can `source` the functions into one's Bash shell and use each one as a standalone Unix tool, complete with tabtab completion, pipelines etc.
Without questioning the "why" of the script, sans my customary morning coffee, and without having tested the code below.
Maybe I'd do something like this.
For the small price of function invocation overhead, the nice benefit of doing it this way is that one can `source` the functions into one's Bash shell and use each one as a standalone Unix tool, complete with tabtab completion, pipelines etc.
#!/usr/bin/env bash
stop_app() {
pkill ${1:?Fail. App name required.}
}
today() {
date +%Y%m%d
}
analyse() {
local analyser=${1:?Fail. Analyser script name required.}
local out_dir=$(printf "results_%s" $(today))
while yaml_file
do local out_file="${out_dir}/summary_$(basename yaml_file).txt"
python3 ${analyser} ${yaml_file} > ${out_file}
printf "%s\n" ${out_file}
done
}
exec_analytics() {
local analyser=${1:?Fail. Analyser script name required.}
local source_dir=${2:-"~/Documents/ExtFin-EFS/smoke/"}
find ${source_dir} -type f -name *.yaml |
sort -r |
analyse ${analyser} |
tail -1
}
update_current() {
local latest_outfile=${1:?Fail. Provide latest output file.}
ln -sf ${latest_outfile} "current.txt"
}
stop_app "whatever_app" && (
trap "rm -f /tmp/module_design_analytics_outfile" 0 HUP TERM PIPE INT
if exec_analytics module_design_analytics.py |
tail -1 > /tmp/module_design_analytics_outfile
then update_current $(printf /tmp/module_design_analytics_outfile)
echo "Done"
else echo "Oops. Something went wrong."
fi
trap - 0 HUP TERM PIPE INT
)(2017)
Previous discussion: https://news.ycombinator.com/item?id=14998213
Previous discussion: https://news.ycombinator.com/item?id=14998213
Showing the code as images instead of text and not even with a fixed width font for code ... and I am suppose to trust that person? :)
Places that I've implemented shell/utility scripts in Python tend to do so because their other application code was also in Python and Python has much more testing support than bash scripts will ever have.
That's why I've done it before, to write tools like custom log rotating / clean-up stuff; another project that comes to mind was a data pipeline that used some Python regex stuff to clean up non-printable characters and ugly stuff from text files before importing data. Both of these projects had lots of tests to verify that the code did what we wanted it to do.
That's why I've done it before, to write tools like custom log rotating / clean-up stuff; another project that comes to mind was a data pipeline that used some Python regex stuff to clean up non-printable characters and ugly stuff from text files before importing data. Both of these projects had lots of tests to verify that the code did what we wanted it to do.
I love to write Makefile recipes that wrap bash into small make commands. My make boilerplate generates help from comments in the make file. I get completions for free. I want all make vars and some internal make recipes to not complete, so I prepend the name with an underscore. Make is great because it handles errors and dependency trees.
Im dealing with some python3 crap today on my main laptop.
Primarily, because of a library I compiled and installed, I'm getting other nasty side effects form it. Primarily that protonvpn-cli isnt now running becasue of some broken library.
I've never had bash break like that. Quirks, sure. But never this sort of brokenness.
Primarily, because of a library I compiled and installed, I'm getting other nasty side effects form it. Primarily that protonvpn-cli isnt now running becasue of some broken library.
I've never had bash break like that. Quirks, sure. But never this sort of brokenness.
I had a similar problem with the YouTube plugin for Kodi recently. Turned out to be some weird Python 3.10 regression that took over a month to fix.
For ProtonVPN you can also use the OpenVPN config files they provide to avoid any Python client issues. Ironically I have a Bash script written around that to generate individual ovpn files as needed from the zip for myself.
For ProtonVPN you can also use the OpenVPN config files they provide to avoid any Python client issues. Ironically I have a Bash script written around that to generate individual ovpn files as needed from the zip for myself.
For folks that like the idea of porting over shell scripts to python: https://plumbum.readthedocs.io/en/latest/
I guess, if you really want a "coding-lang" rather than "just" bash. I'd give Golang a try, just because of the static-binaries.
No environment issues or missing and clashing libraries.
PS. I'd still go for bash 99/100.
No environment issues or missing and clashing libraries.
PS. I'd still go for bash 99/100.
Yo - I'm sure there are advantages, but "that python ecosystem" !
Bash is usually everywhere even on "new installations". I'd hate to fight package compatibility, virtual environments etc..
YMMV
Bash is usually everywhere even on "new installations". I'd hate to fight package compatibility, virtual environments etc..
YMMV
[deleted]
I agree with the general thesis that Bash is unsuited for production work of any complexity, but the coverage of pipelines is incomplete. The only two examples I have found (the examples are images, so I may have missed something) may be implemented in Python by calling sorted(), which is not the case in general. I feel it would be more persuasive if it had at least one example of setting up and running a pipeline.
Depends on what you’re doing.
I wouldn’t want to worry about python environment when dealing with OS post-install scripting for instance.
I wouldn’t want to worry about python environment when dealing with OS post-install scripting for instance.
I think some of the examples of Python being better are places you just shouldn't use Bash.
Like testing (Bats sucks).
Like string manipulation (use other utils or even languages callable from Bash).
Like complicated logic. Just don't do that in Bash.
Bash is for (glorified) one-liners. And for I-can't-believe-you-did-that-in-Bash-ers.
Like testing (Bats sucks).
Like string manipulation (use other utils or even languages callable from Bash).
Like complicated logic. Just don't do that in Bash.
Bash is for (glorified) one-liners. And for I-can't-believe-you-did-that-in-Bash-ers.
I've spent many years becoming quite fluent in bash. It's not trivial. But all objections here are surmountable if not outright features. Python is the right tool for many jobs. But so is bash, for someone willing to truly learn it.
> Without much real work, it’s easy to replace shell scripts with Python code. The revised code is easier to read and maintain, runs a little faster, and can have a proper unit test suite.
And give a wonderful syntax error.
And give a wonderful syntax error.
I once worked for a company that had a product which had a fairly complicated and time-consuming build process. Someone had written a distributed build process using the traditional CMD.exe shell scripting language. It was running multiple processes in parallel, and they communicated by writing and reading text files. Later they added PowerShell scripts. All in all, there were tens of thousands of lines of PS and BAT scripts. It was quite a nightmare.
The team was supposed to maintain this mess. We decided to create a compiler that translated .BAT scripts to C#. The idea was that the C# code would be easier to understand and modify. I left before it was complete, so I'm not sure how that worked out. But last I heard, they were pleased that some parts of it now ran thousands of times faster.
The team was supposed to maintain this mess. We decided to create a compiler that translated .BAT scripts to C#. The idea was that the C# code would be easier to understand and modify. I left before it was complete, so I'm not sure how that worked out. But last I heard, they were pleased that some parts of it now ran thousands of times faster.
having a hard time being sold here.
mkdir -p would create the directory without error. checking if a directory exists is trivial before creating it. Python throws an error if the path given to os.mkdir exists too.
This article is way too long to digest, so i should stop now.
Perhaps the argument should be, rewrite hacky things in bash as programs and build them into your apps?
mkdir -p would create the directory without error. checking if a directory exists is trivial before creating it. Python throws an error if the path given to os.mkdir exists too.
This article is way too long to digest, so i should stop now.
Perhaps the argument should be, rewrite hacky things in bash as programs and build them into your apps?
I've written a lot of bash scripts in my time, even some 10,000 liners. Christ - I have probably written over 100k lines of them. The big ones (by big I mean say over 100 lines) all started life when I thought they would remain under 100 lines.
I'm a professional programmer by trade. Admitting to starting a program I know would be over 100 lines in shell script would be close to an admission of incompetence, because technically shell script is one of the worst computer languages out there. So I'd never admit to it. But it's so damned portable, and ubiquitous and the batteries it comes with (the 'nix cli tools) are to powerful and complete, it makes irresistible at times.
Where this article falls down is they are recommending Python3. Python2 would be fine. But in 'nix environments, where file names and configuration files can't be treated as text (Unicode) because there is no well defined system encoding, Python3 manages to introduce more rare bugs than shell script. It encourages you to treat everything a text, the dies ignominiously when decoding the 'nix byte stream (file name or whatever) fails. (On Windows where everything is UTF16, this isn't a problem - for local files. It remains a problem for data from external systems, like the internet.)
Pull off making a programming language less reliable than shell was a mean achievement - but the Python3 devs did it. Hats off to 'em.
I'm a professional programmer by trade. Admitting to starting a program I know would be over 100 lines in shell script would be close to an admission of incompetence, because technically shell script is one of the worst computer languages out there. So I'd never admit to it. But it's so damned portable, and ubiquitous and the batteries it comes with (the 'nix cli tools) are to powerful and complete, it makes irresistible at times.
Where this article falls down is they are recommending Python3. Python2 would be fine. But in 'nix environments, where file names and configuration files can't be treated as text (Unicode) because there is no well defined system encoding, Python3 manages to introduce more rare bugs than shell script. It encourages you to treat everything a text, the dies ignominiously when decoding the 'nix byte stream (file name or whatever) fails. (On Windows where everything is UTF16, this isn't a problem - for local files. It remains a problem for data from external systems, like the internet.)
Pull off making a programming language less reliable than shell was a mean achievement - but the Python3 devs did it. Hats off to 'em.
>This article is way too long to digest, so i should stop now.
Lol one of the top problems with having "scripts" in python, their reliability-lifetime" (a.k.a as execute-and-forget) is very limited. Packages are outdated, api changes etc etc
Lol one of the top problems with having "scripts" in python, their reliability-lifetime" (a.k.a as execute-and-forget) is very limited. Packages are outdated, api changes etc etc
That's basically what I did with this project.
https://github.com/Mylab6/PiBluetoothMidSetup
Of course I could of done this in bash, but Python is just so much cleaner.
https://github.com/Mylab6/PiBluetoothMidSetup
Of course I could of done this in bash, but Python is just so much cleaner.
More Lillipythonian flame baiting. Summoning the great dang.
TCL would be a better choice, more so back then.
Rock The Bash-Bah!
(Sorry, Oingo Boingo)
(Sorry, Oingo Boingo)
Oingo Boingo? You mean The Clash...
The only real alternatives to bash are shit-tier languages like python and perl.
I will never use a programming language which has significant white space, specially if I may have to view and edit the scripts in a remote terminal with vi.
I am also not too interested in using a programming language which looks the same before and after encryption: https://www.goodreads.com/quotes/tag/437174-perl-the-only-la...
I will never use a programming language which has significant white space, specially if I may have to view and edit the scripts in a remote terminal with vi.
I am also not too interested in using a programming language which looks the same before and after encryption: https://www.goodreads.com/quotes/tag/437174-perl-the-only-la...
This really just reads like someone who doesn't know shell and does know python, but thinks the problem is the tool. They continually describe shell with phrases like "obscure and difficult-to-predict" while talking through things that are either obvious with experience or from context, and then write something different in python (i.e. something you could write in BASH that does something different) that's not less obscure but is only using python obscurity.
> An example of a shell obscurity is the way the current working directory is set. The cd command is clear enough, but in the presence of sub-shells using (), can make it difficult to discern a stack of nested shell invocations and how the working directory changes when the sub-shells exit.
Er... yeah? So you expect PWD to be a global and it was actually a local?
Or more explicitly:
> One of the shell’s ickier features is that variables tend to be global. There are some exceptions and caveats, however, that lead to shell scripts that are broken or behave inconsistently.
That's not a shell feature, that's just how people frequently write it - you can make functions and declare your variables local if you want. I can easily write a python script with all my variables at the top, too.