Xonsh: I don't remember how to write a for loop in Bash [video](youtube.com)
youtube.com
Xonsh: I don't remember how to write a for loop in Bash [video]
https://www.youtube.com/watch?v=x85LSyCxiw8
134 comments
You can use brace expansion to avoid calling out to seq in a subshell (`{1..5}` expands to `1 2 3 4 5`)
Awesome, thanks!! Just found the section of the man page, looks like a pretty feature-filled expansion technique:
A sequence expression takes the form {x..y[..incr]}, where x and y are either integers or
single characters, and incr, an optional increment, is an integer. When integers are sup-
plied, the expression expands to each number between x and y, inclusive. Supplied integers
may be prefixed with 0 to force each term to have the same width. When either x or y
begins with a zero, the shell attempts to force all generated terms to contain the same
number of digits, zero-padding where necessary. When characters are supplied, the expres-
sion expands to each character lexicographically between x and y, inclusive, using the
default C locale. Note that both x and y must be of the same type. When the increment is
supplied, it is used as the difference between each term. The default increment is 1 or -1
as appropriate.
> ...although I just tested it, and the increment and 0-padding only work in non-interactive mode ='(
Works for me...
For completeness, here is a POSIX sh approach that doesn't require `seq` (in fact I notice this is what your "Uber Fancy" version uses internally):
Works for me...
$ echo $0 # determine current shell (alternative: "ps -p $$")
bash
$ echo {01..100..20}
001 021 041 061 081
$ n=5
$ echo {1..$n}
{1..5}
# intended output: 1 2 3 4 5
For completeness, here is a POSIX sh approach that doesn't require `seq` (in fact I notice this is what your "Uber Fancy" version uses internally):
i=1
while [ "$i" -le 5 ]; do
echo "$i"
i="$((i+1))"
done
Ahhh, my login shell is bash v3 (friggin' MacOS!) but my PATH has bash v5! On Bash v5 it works interactively! (probably v4+ too)
You can always do double eval for variables:
var=1234
echo `echo {1..$var}`Yup, which you could also write as eval echo {1..$var} . The second argument's variable is being interpolated as if it's a quoted string, before it gets passed to eval, so eval is actually seeing "{1..1234}" and not doing the interpolation itself (because it wouldn't, because the variable isn't valid in a range). This should prevent the need for a subshell and the second echo call.
With an array:
With an array:
var=12
declare -a array=()
eval "array=( {01..$var} )"
echo "${array[@]}"
# 01 02 03 04 05 06 07 08 09 10 11 12Surely the version you would actually use is:
seq 5Presumably the point is to do something with i rather than just printing.
seq 5 | xargs -n1 <...>Very beautiful. Of course now you have to deal with xargs' wacky syntax and options for string interpolation, which is probably why people don't do this.
Compare:
for i in {1..5}; do echo file_$i; done
with
seq 5 | xargs -I % echo file_%
I had to look up the manpage for xarg's -I, while the for loop uses a small number of orthogonal features entirely in my "working set".
This is also why people "uselessly" use cat.
Compare:
for i in {1..5}; do echo file_$i; done
with
seq 5 | xargs -I % echo file_%
I had to look up the manpage for xarg's -I, while the for loop uses a small number of orthogonal features entirely in my "working set".
This is also why people "uselessly" use cat.
I'm not sure whether you're trying to prove the authors point or not?
For myself I can generally write basic python (or c) without having to double check. Shell not so much. Eg Does the semi colon come before or after the do? Then there's the quoting rules and expansions etc.
For myself I can generally write basic python (or c) without having to double check. Shell not so much. Eg Does the semi colon come before or after the do? Then there's the quoting rules and expansions etc.
All shell scripting is just running commands in a shell terminal. The language is designed as if you're at a VT100 terminal typing commands in by hand. Pressing 'Enter' key tells the shell to evaluate whatever was just typed in. A semi-colon is the equivalent of the 'Enter' key. Most commands allow you to pass arguments to them (and so don't need a semi-colon after) unless they are the end of a block, like done, fi, esac, etc, which don't take arguments, so they would get a semi-colon or newline after them. Most things want some whitespace between each element, so for example "[1-lt2]" doesn't parse, but " [ 1 -lt 2 ] " does.
If you read the man page from top to bottom, pretty much all your questions will go away. Bash is only as confusing and mysterious as every other language. Imagine trying to use C++ without ever reading a book on how C++ works. Bash is much simpler! But you should read the man page!
If you read the man page from top to bottom, pretty much all your questions will go away. Bash is only as confusing and mysterious as every other language. Imagine trying to use C++ without ever reading a book on how C++ works. Bash is much simpler! But you should read the man page!
For me, the most beautiful representation of a loop is like this:
for((x=1;x<=5;x++))
do if [[ -v x ]]
then echo $x
fi
doneI strongly dislike this style. Way too easy to commit a one off error.
Personally, I never use naked loops if I can avoid them. Either foreach or higher order functions if the language has them.
Personally, I never use naked loops if I can avoid them. Either foreach or higher order functions if the language has them.
Tip: You can use { } instead of do/done to emulate C syntax.
Aside: [[ -v x ]] tests if the variable x is set. It should probably be [[ x -ne 0 ]] to match the example from the parent.
Aside: [[ -v x ]] tests if the variable x is set. It should probably be [[ x -ne 0 ]] to match the example from the parent.
"You can use { } instead of do/done"
In sh ? In bash ? In zsh ? I am very glad if it is so in any of them
In sh ? In bash ? In zsh ? I am very glad if it is so in any of them
It works in bash/zsh/ksh but not in sh/dash. Examples:
for i in 1 2 3 4 5; { echo $i; }
for ((i=1;i<=5;i++)) { echo $i; }
if you like pipes and a more functional style:
seq 5 | xargs -n1 <...>I was a bash sympathizer until a post made the rounds about how, even if you `set -euxo pipefail`, `declare x="$(err)"` will silently eat the return code of the subshell. This applies to `local` too, and tbf is mentioned somewhat offhandedly on some man page, but was the straw that solidified my opinion. There's just no reasonable way that even an experienced user of Bash (ie. the kinds of folks who think that set command has them covered) could be expected to know about this footgun. The solution is to always declare and assign variables separately. Still use it all the time, and I guess "now I know", but, wtf Bash. Safety options are poorly bolted on.
I agree, and that was out of the motivations for https://www.oilshell.org/
It runs your bash scripts but you can also opt into correct error handling. The simple invariant is that it doesn't lose an exit code, and non-zero is fatal by default.
Oil 0.10.0 - Can Unix Shell Error Handling Be Fixed Once and For All?
https://www.oilshell.org/blog/2022/05/release-0.10.0.html
This took quite awhile, and I became aware of more error handling gotchas than I knew about when starting the project:
https://www.oilshell.org/release/latest/doc/error-handling.h...
e.g. it's impossible in bash to even see the error status of a process sub like diff <(sort left.txt) <(sort OOPS)
If you have bash scripts that you don't want to rewrite, try
1) run them with OSH
2) Add shopt --set oil:upgrade at the top to get the error handling fixes.
Tell me what happens :) https://github.com/oilshell/oil
I spent a long time on that, but the post didn't get read much. I think it's because it takes a lot of bash experience to even understand what the problem is.
It runs your bash scripts but you can also opt into correct error handling. The simple invariant is that it doesn't lose an exit code, and non-zero is fatal by default.
Oil 0.10.0 - Can Unix Shell Error Handling Be Fixed Once and For All?
https://www.oilshell.org/blog/2022/05/release-0.10.0.html
This took quite awhile, and I became aware of more error handling gotchas than I knew about when starting the project:
https://www.oilshell.org/release/latest/doc/error-handling.h...
e.g. it's impossible in bash to even see the error status of a process sub like diff <(sort left.txt) <(sort OOPS)
If you have bash scripts that you don't want to rewrite, try
1) run them with OSH
2) Add shopt --set oil:upgrade at the top to get the error handling fixes.
Tell me what happens :) https://github.com/oilshell/oil
I spent a long time on that, but the post didn't get read much. I think it's because it takes a lot of bash experience to even understand what the problem is.
> Add shopt --set oil:upgrade at the top to get the error handling fixes
How does this work? It seems magic
How does this work? It seems magic
Right now there are 22 options that form the fixes and enhancements to bash:
OSH is a bash-compatible shell, from scratch! :) Let me know what happens
oil$ shopt -p oil:upgrade
shopt -s command_sub_errexit
shopt -u dashglob
shopt -s errexit
shopt -u expand_aliases
shopt -s inherit_errexit
...
OSH is a bash-compatible shell, from scratch! :) Let me know what happens
Shellcheck warns you about it:
I almost never use set -o pipefail, and often not even set -e. Your scripts end up failing for mysterious reasons and it takes forever to figure out which part of which pipeline failed. Instead, I simply check the result of the end of the pipe (did it return any data? does it look valid?) and that is good enough 99% of the time.
$ cat foo.sh
#!/usr/bin/env bash
set -euxo pipefail
declare x="$(err)"
$ shellcheck foo.sh
In foo.sh line 3:
declare x="$(err)"
^-- SC2034 (warning): x appears unused. Verify use (or export if used externally).
^-- SC2155 (warning): Declare and assign separately to avoid masking return values.
For more information:
https://www.shellcheck.net/wiki/SC2034 -- x appears unused. Verify use (or ...
https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to ...
I almost never use set -o pipefail, and often not even set -e. Your scripts end up failing for mysterious reasons and it takes forever to figure out which part of which pipeline failed. Instead, I simply check the result of the end of the pipe (did it return any data? does it look valid?) and that is good enough 99% of the time.
I don't think you can capture/handle exit codes of subshells anyway, right? I've been operating under that assumption for years, and just don't do subshells if I can avoid them.
$? contains the return status for subshells, commands, pipelines, functions, etc. From the man page, under Special Parameters :
? Expands to the exit status of the most recently executed foreground pipeline.
$ foo="$(echo foobar ; exit 3)" ; ret=$? ; echo "output: $foo" ; echo "return status: $ret"
output: foobar
return status: 3
Conditional statements are even harder to remember. How many square brackets do I need? 1 or 2? Do I use == or -eq? Where does the semicolon go again?
[ is an alias for test, a very simple program
[[ is the built-in
this means you need to write everything separate, test doesn't parse, plus it needs the closing bracket if you use the alias
but if-then works with any expression, and you put ; after expressions
this means you need to write everything separate, test doesn't parse, plus it needs the closing bracket if you use the alias
but if-then works with any expression, and you put ; after expressions
if test 1 -eq 2 ; then echo "oh noes" ; fiSee that's even stupider than just having syntax for it
I somehow already read that in the past but here, in this condensed form, it now actually makes sense. I think you just cured me of my bash fear. Thanks for that explanation.
I'll make sure to either use test or [[ and never [ again so that I remember the difference. And I am currently reading man test
I'll make sure to either use test or [[ and never [ again so that I remember the difference. And I am currently reading man test
I think using test would be the best, even if it's awkward, but [[ is a Bashism (so not POSIX shell compatible), and [ is jut an alias for test and requires that dumb closing bracket, and you are already execve()-ing a separate program for comparison, you are already depending on the exit code, so why not make it explicit by using test.
(I continue to use [[ and never look back, just forward, trying to picture the shining city on a conveniently sloped hill, where people live without fear in their hearts, where they have compile time checking for their scripts, and the compiler gently pushes them toward sane error handling. Not too much, but definitely not too little. Something like Rust's Result type.)
(I continue to use [[ and never look back, just forward, trying to picture the shining city on a conveniently sloped hill, where people live without fear in their hearts, where they have compile time checking for their scripts, and the compiler gently pushes them toward sane error handling. Not too much, but definitely not too little. Something like Rust's Result type.)
> How many square brackets do I need? 1 or 2?
2
> Do I use == or -eq?
For string comparison, use '='. For arithmetic comparison, use -eq, -gt, -lt, etc.
> Where does the semicolon go again?
Whenever you're putting multiple lines on one. If you use multiple lines, you don't need semicolons.
For example:
2
> Do I use == or -eq?
For string comparison, use '='. For arithmetic comparison, use -eq, -gt, -lt, etc.
> Where does the semicolon go again?
Whenever you're putting multiple lines on one. If you use multiple lines, you don't need semicolons.
For example:
if [[ "$result" = 'SUCCESS' ]] && [[ "$count" -gt 0 ]]
then
echo 'Hooray!'
fiPeople complain about === in JS but at least I can just use it everywhere and stop using ==. Needing to remember which equality check to use based on type is so frustrating
Type man bash. Go to section "SHELL GRAMMAR". Go down to "Compound Commands". Then go read the section "CONDITIONAL EXPRESSIONS".
If you want a much-reduced version, read man dash instead. It is very small and close to POSIX semantics. You can then later go find the Bashisms.
If you want a much-reduced version, read man dash instead. It is very small and close to POSIX semantics. You can then later go find the Bashisms.
In bash or bash-like shells you always use the double square bracket tests! They support everything the regular test syntax does, but have some extra features that make them "safer" to use.
Some code style guides for Linux distros or work places like Google actually enforce this!
Some code style guides for Linux distros or work places like Google actually enforce this!
Wow all the bash hate. It takes practice and repetition like with most programming languages. I have no problems writing for loops in bash because I do it all the time. Similarly if it’s something I don’t use frequently I will feel the syntax is arcane and unintuitive and have to look it up anyway.
It’s fine to prefer one tool over the other but bashing the one you’re less familiar with is unnecessary :)
It’s fine to prefer one tool over the other but bashing the one you’re less familiar with is unnecessary :)
I remember how to write loops and conditionals for most of the languages I’ve used for practical purposes. I can even write decent perl after 15 years of not using it. But for some reason bash couldn’t stick and I google it every time. It’s obviously my laziness, but it probably has natural causes.
Anecdata is anecdata and I’ll counter yours with mine: I almost never have to google bash syntax (I do for things like arrays which I don’t use often). So no, there are no “natural causes”.
Face it folks, it’s not bash’s fault if you use it seldom enough that you need a reminder of the syntax when you do. Nothing wrong with that!
Face it folks, it’s not bash’s fault if you use it seldom enough that you need a reminder of the syntax when you do. Nothing wrong with that!
Arrays are pretty basic. I won't have to Google how to create an array in Python for the rest of my life - I've done it in Bash and I've forgotten every single time. I think I had it memorised permanently for Python within about five minutes. I think when I find myself using arrays in Bash (or Powershell) it's usually time to take a step back and reevaluate.
I use bash frequently and I can't remember the syntax.
I think your comment convinced me to never use bash again. Now onto finding a replacement... If people say they use bash because it is preinstalled they can go screw themselves. Install whatever replacement I'm going to pick and shut up.
I think your comment convinced me to never use bash again. Now onto finding a replacement... If people say they use bash because it is preinstalled they can go screw themselves. Install whatever replacement I'm going to pick and shut up.
I’ve heard good things about fish. Some of my colleagues swear by zsh.
> It takes practice and repetition like with most programming languages
To me the issue is why learn a different language for scripting? Python is the obvious replacement, but even Java write scripts in, I'm pretty sure any major language can compile and run easily.
To me the issue is why learn a different language for scripting? Python is the obvious replacement, but even Java write scripts in, I'm pretty sure any major language can compile and run easily.
Running one command after another is trivial in most languages. The hard part is pipelines, which are inherently concurrent. For example, to pipe one subprocess into another in Python requires spawning off separate threads to shuttle the data around (there are libraries which make this easier, but then your scripts will require a package manager...).
Shells are pretty decent at piping and spawning subprocesses. They suck at anything algorithmic, numeric, etc. Thankfully those tasks can usually be handed off to a separate tool (e.g. classic Unix text processing tools, like sed/grep/cut/etc., or more recent data processing tools like jq, mlr, etc.)
Shells are pretty decent at piping and spawning subprocesses. They suck at anything algorithmic, numeric, etc. Thankfully those tasks can usually be handed off to a separate tool (e.g. classic Unix text processing tools, like sed/grep/cut/etc., or more recent data processing tools like jq, mlr, etc.)
Right, I really meant in a proper programming language you'd use high level libraries to do this work, rather than shelling sub processes. You wouldn't need sed/grep/cut.
The ergonomics of calling other commands filtering and piping the output is better in sh, till some complexity limit and error tolerance
Of course python comes with its own package management and system version clashes
Of course python comes with its own package management and system version clashes
Not to mention a slow startup problem, a pipe of python tools spends some significant time there where (ba)sh is close to instantaneous.
Python is a great language, but not if you want to glue together shell commands. It is very verbose and quite obviously not a first-class citizen. That is what you use bash for.
I have no problem writing brainfuck either. Once you get used to it and developed your building blocks for developing more complex algorithms, it is really easy!
"It takes practice and repetition like with most programming languages. "
Except with Bash it really isn't worth it. Learning PowerShell is much better for long scripts.
Except with Bash it really isn't worth it. Learning PowerShell is much better for long scripts.
“Really isn’t worth it” - hearsay. It totally is. Also - power shell is better for long scripts, you say, but what are you going to use for short ones?
I’m not buying what you’re selling tbh.
I’m not buying what you’re selling tbh.
" hearsay. It totally is."
No it is an opinion, and IMHO the RoI for time invested in learning bash is far less than Python or Powershell.
No it is an opinion, and IMHO the RoI for time invested in learning bash is far less than Python or Powershell.
PowerShell might be better, but it won't be already installed in your typical Linux environment.
dnf install powershell
You can put that in a bash script because guess what - bash is already there ;)
$ bash
/bin/ksh: bash: not found
$ pkglocate powershell | grep \^power
$
dnf: Bad command or file name
:)
I'm not sure if it's sarcasm but in case you are serious, Unix shell has been around for over 50 years and its variants are installed on millions systems around the world, and it's still the fastest way to accomplish simple tasks on these servers. PowerShell is something vastly different.
Some arguments would be nice…
Bash makes simply things simple and hard things impossible while being ugly as sin.
Discussions about syntax devolve rather quickly into arguing about sources, because there are so few studies done and everyone has an opinion (or cause). Bash syntax is horrid and that's enough for me.
Just wait a couple decades and languages slowly improve or they die. Sometimes it takes a long time (in comparison to a person's lifespan) and that's ok.
Just wait a couple decades and languages slowly improve or they die. Sometimes it takes a long time (in comparison to a person's lifespan) and that's ok.
Even giving an opinion would be something. What's something you like better about PowerShell? I can easily say something I dislike about it:
This form `Verb-LongNoun` for function names used in PS is somewhere between alien and maddening. It's difficult to type, difficult to remember, and isn't an obvious improvement over `lowercasename` which is typical of basically every other shell.
I guess the idea is that you don't have to worry about naming collisions with programs in your path? That's never been an issue for me when writing shell scripts though, not even once.
This form `Verb-LongNoun` for function names used in PS is somewhere between alien and maddening. It's difficult to type, difficult to remember, and isn't an obvious improvement over `lowercasename` which is typical of basically every other shell.
I guess the idea is that you don't have to worry about naming collisions with programs in your path? That's never been an issue for me when writing shell scripts though, not even once.
You don’t have to use Verb-LongNoun. PowerShell will fuzzy match as long as it’s not ambiguous. Verb-L might be enough. If you’re in an interactive shell they’re already aliased to shorter forms like “gal” for Get-Alias or “gci” for Get-ChildItem.
I’m not OP but things I like about PowerShell:
- Objects instead of text
- The entire C# std lib is accessible
- One tool with the same UX vs several different tools with different UX.
I’m not OP but things I like about PowerShell:
- Objects instead of text
- The entire C# std lib is accessible
- One tool with the same UX vs several different tools with different UX.
The mostly consistent Verb-Noun naming convention is a good thing about powershell and isn't an issue due to very good autocompletion is PowerShell editors.
But the single best feature of PS is that you pipe structured objects to the next command, instead of raw text that must be parsed, which is prone error. Every command output object can be exported to JSON with ConverTo-JSON. You can export the entire Active Directory as JSON with
Get-ADObject -Filter * | ConvertTo-JSON > AD.json
Some more examples.
Get-ChildItem -Path *.txt | Where-Object {$_.length -gt 10000} | Sort-Object -Property length | Format-Table -Property name, length
Get-Item -Path HKLM:\Software\MyCompany | New-ItemProperty -Name NoOfEmployees -Value 8124
Get-Process | Sort-Object -Property handles
But the single best feature of PS is that you pipe structured objects to the next command, instead of raw text that must be parsed, which is prone error. Every command output object can be exported to JSON with ConverTo-JSON. You can export the entire Active Directory as JSON with
Get-ADObject -Filter * | ConvertTo-JSON > AD.json
Some more examples.
Get-ChildItem -Path *.txt | Where-Object {$_.length -gt 10000} | Sort-Object -Property length | Format-Table -Property name, length
Get-Item -Path HKLM:\Software\MyCompany | New-ItemProperty -Name NoOfEmployees -Value 8124
Get-Process | Sort-Object -Property handles
> Just wait a couple decades and languages slowly improve or they die.
Bash has been around for 33 years. Much of its syntax has been around for 52 years through sh.
Bash has been around for 33 years. Much of its syntax has been around for 52 years through sh.
Almost everything that Bash is good at, other languages have already adopted with a superior syntax. The only thing Ba/sh type syntax has going for it is the *nix ecosystems (note the parallels with powershell). That will keep it used for the rest of our lifetimes, if you're only interested in short-term thinking.
Ofc this is bigger thinking than the Powershell vs Bash debate, per se. Powershell is more useable as a language, if not by pure syntactic maturity, than by features alone. There will always be someone who will disagree with the obvious, but there's no doubt that the algol syntax is going away...however long that will take.
Ofc this is bigger thinking than the Powershell vs Bash debate, per se. Powershell is more useable as a language, if not by pure syntactic maturity, than by features alone. There will always be someone who will disagree with the obvious, but there's no doubt that the algol syntax is going away...however long that will take.
Powershell is good at managing windows boxes. I've tried running powershell on linux in order to manage a windows network from the output of a (bash) script that ran on a linux machine. It didn't work. Powershell is a husk of a language on non-windows machines. The important modules like AD are missing, so it won't work properly on non-windows machines.
[deleted]
You're out of your f'n mind.
> Wow all the bash hate. It takes practice and repetition like with most programming languages.
Except I have no issues with other programming languages. Bash is really different. I only tolerate it because it comes pre-installed on just about every system.
Except I have no issues with other programming languages. Bash is really different. I only tolerate it because it comes pre-installed on just about every system.
I have issues with lisp because it’s really different. I could never grok it. I don’t go around swearing off it and calling it names like people do with bash. :)
Your "different" and my "different" are different :)
I also have issues with COBOL.
Sometimes a language is just too inelegant to pour time into learning it.
I also have issues with COBOL.
Sometimes a language is just too inelegant to pour time into learning it.
Yes - like Java. And yet people learn it.
Hmmm … feels like a lot of missing the bigger point here. Xonsh, IMO, isn’t about hating bash or loving Python.
It’s about making a new language (and shell) that tries to bring what’s good about a nice shell to Python so that when we hit our “this bash script is too big/complex and I need a better language” threshold, we can pick up Python (a reasonable choice) but retain some of the ergonomics of bash for calling and piping data to/from external programs, which Python simply doesn’t have (where subprocess + shlex + stdlib utils just don’t cut it IMO).
From my minimal experience/playing around, xonsh seems quite nice and I find it a very believable future where it is a mainstream system scripting tool.
Another thing to keep in mind is that AFAICT it is coming out of the researchy-sciency crowd where Python is mainstream and bash a necessary evil. Finding a way to make subprocess calls easy seems an obvious win in that context.
Going even further, how nice would it be if something similar were part of the Python standard library? Being a wonderfully ergonomic shell script alternative seems like an instance where python could finally be the actual best at something.
It’s about making a new language (and shell) that tries to bring what’s good about a nice shell to Python so that when we hit our “this bash script is too big/complex and I need a better language” threshold, we can pick up Python (a reasonable choice) but retain some of the ergonomics of bash for calling and piping data to/from external programs, which Python simply doesn’t have (where subprocess + shlex + stdlib utils just don’t cut it IMO).
From my minimal experience/playing around, xonsh seems quite nice and I find it a very believable future where it is a mainstream system scripting tool.
Another thing to keep in mind is that AFAICT it is coming out of the researchy-sciency crowd where Python is mainstream and bash a necessary evil. Finding a way to make subprocess calls easy seems an obvious win in that context.
Going even further, how nice would it be if something similar were part of the Python standard library? Being a wonderfully ergonomic shell script alternative seems like an instance where python could finally be the actual best at something.
Weird that no one has yet to actually discuss `xonsh`, which the video is actually about.
However, I'm about to do the same lol, and come out as a bash apologist against the sea of derision here. I've spent a decade writing code in maybe 10 different languages, and bash is still what I write the most of -- and its by choice! If you're writing a lot of code that calls other code, bash is kind of the default choice, since that's literally what the shell is for.
Once you spend enough time with it, I actually find bash to be surprisingly elegant in what it affords the author in its terseness. Combined with other CLI utilities' own concision, it's an incredibly powerful environment. Sure it can start to appear arcane to unfamiliar folks, but you can work to make it significantly more readable than most people would have you believe. And with e.g. shellcheck, you can do away with a great deal of common footguns.
However, I'm about to do the same lol, and come out as a bash apologist against the sea of derision here. I've spent a decade writing code in maybe 10 different languages, and bash is still what I write the most of -- and its by choice! If you're writing a lot of code that calls other code, bash is kind of the default choice, since that's literally what the shell is for.
Once you spend enough time with it, I actually find bash to be surprisingly elegant in what it affords the author in its terseness. Combined with other CLI utilities' own concision, it's an incredibly powerful environment. Sure it can start to appear arcane to unfamiliar folks, but you can work to make it significantly more readable than most people would have you believe. And with e.g. shellcheck, you can do away with a great deal of common footguns.
I worked at one company where our build engineer wrote 7500 lines of exceedingly complex Bash code to analyze our source tree and identify "code owners" - i.e. whoever had worked the most on each directory and was most familiar with it.
I asked him about it, and he said, "It is a lot of code, but it does a lot. And if I'd written it in C++ it would have been a lot more code than that!"
We were a Python shop.
I asked him about it, and he said, "It is a lot of code, but it does a lot. And if I'd written it in C++ it would have been a lot more code than that!"
We were a Python shop.
That sounds like some poor programing in bash. The essence of the problem from what you wrote is something like:
display the blame for all lines of all files per folder | sort by name | sum counts by name
A single command-line can contain all the bash code, since the necessary tools are all available and can be placed together in a pipeline.
display the blame for all lines of all files per folder | sort by name | sum counts by name
A single command-line can contain all the bash code, since the necessary tools are all available and can be placed together in a pipeline.
In all fairness, the Bash code really did quite a bit more than that proverbial one-liner. But much of it wasn't calling and tying together those external tools, it was data analysis code that would have been so much simpler in Python.
Plus, being a Python shop, any of us could have maintained a Python version, instead of the one engineer who was truly a Bash wizard.
Some common sense and manual work might have been even better: just ask everyone in the company to identify the code areas they were familiar with and manually populate the CODEOWNERS files, instead of spending months on a mostly-failed attempt to automate the whole thing.
Or, just teach people how to use git blame!
After all, many of us touched many different parts of the code. Git blame would tell you immediately who was familiar with the different bits, at a much more granular level than a CODEOWNERS file in a directory.
Plus, being a Python shop, any of us could have maintained a Python version, instead of the one engineer who was truly a Bash wizard.
Some common sense and manual work might have been even better: just ask everyone in the company to identify the code areas they were familiar with and manually populate the CODEOWNERS files, instead of spending months on a mostly-failed attempt to automate the whole thing.
Or, just teach people how to use git blame!
After all, many of us touched many different parts of the code. Git blame would tell you immediately who was familiar with the different bits, at a much more granular level than a CODEOWNERS file in a directory.
You've pointed out several ways things could have been done better in that project. They are all clearly important and hard-won knowledge.
But, I have found that one of those is super-duper-extremely important: know when not to program! You can just ask other SMEs (subject matter experts) for the answer!
But, I have found that one of those is super-duper-extremely important: know when not to program! You can just ask other SMEs (subject matter experts) for the answer!
that's like... a ~30 line program...maybe
take the output of
`git ls`
group it by directory
iterate over the grouped directories passing each file to
`git log --follow $1 | git shortlog -sn`
sum the numbers next to each name
sort the name + number combos within the directory.
print the info for each directory.
take the output of
`git ls`
group it by directory
iterate over the grouped directories passing each file to
`git log --follow $1 | git shortlog -sn`
sum the numbers next to each name
sort the name + number combos within the directory.
print the info for each directory.
Imagine how many lines it would have been in assembly!
If he couldn't do it in 300 lines of Python, C++, or bash, he was incompetent and should have been reassigned. At best. 7500 lines for that is not an asset, it is a liability, regardless of language.
At least while he was writing his 7500 lines of bad code, he wasn't doing any other damage. You can throw it out and get somebody else to write a good one in a week.
At least while he was writing his 7500 lines of bad code, he wasn't doing any other damage. You can throw it out and get somebody else to write a good one in a week.
Whenever I start writing bash, if it goes beyond 50 lines I rewrite it in Python.
I really prefer to write no bash at all its so confusing.
I really prefer to write no bash at all its so confusing.
If you want to understand bash, it’s best to spend some time using basic posix sh (dash on some systems) it is a much less powerful shell, but that highlights the power of composition that it does have. Also use shellcheck its an amazing linter, it will teach you many things about the shell, and actually explains the why for each lint.
I always yearn for compile time checking when writing scripts, because my experience is that these scripts become unmaintainable in about 3 hours.
Recently I started using TS for scripting. Not the best, but Ammonite (a Scala scripting env) is a bit too big of a lift.
Recently I started using TS for scripting. Not the best, but Ammonite (a Scala scripting env) is a bit too big of a lift.
In the Google style guide for Shell[1], they use 100 lines as the limit.
[1] -- https://google.github.io/styleguide/shellguide.html#when-to-...
[1] -- https://google.github.io/styleguide/shellguide.html#when-to-...
Whenever I encounter a 100+ line Python script, I rewrite it into a ~10 line bash script.
Whenever I encounter 20 lines bash script, I rewrite it as 15 lines Next Generation Shell script. It is more correct because of better error handling, more concise because of stdlib, more readable because of saner syntax. Disclosure: I'm the author.
You're not supposed to do it because it is terribly insecure but you can implement iteration in bash by writing a program that writes a bash script and piping into bash. For instance,
ls | awk '{print "command " $1}' | bash
[deleted]
Kinda strange and related but one of my first “wow cool” memories of working life was working at a company where we used Unix/Linux servers as workstations. Yeah big iron HP/DEC/Sun boxes.
So anyway my co-worker did a for loop directly as a CLI command, probably using Tcsh at the time but it stuck with me.
It was like a “wow you can just”:
So anyway my co-worker did a for loop directly as a CLI command, probably using Tcsh at the time but it stuck with me.
It was like a “wow you can just”:
for F in `seq 1 10`; do
echo $F
done
Yet, that is the wrong way.
for I in {1..10}; do
....
doneI have a shortcut in my browser bookmark bar just for this. I have been writing bash loops for more than 20 years, but only very sporadically (like 2 times per year).
For some reason, I cannot remember this structure.
P.s. never had this problem with PHP, Python, Java...
P.s.2 I have this shortcut for batch (Windows) and PowerShell, too. :)
For some reason, I cannot remember this structure.
P.s. never had this problem with PHP, Python, Java...
P.s.2 I have this shortcut for batch (Windows) and PowerShell, too. :)
Well bash does kind of have archaic syntax
Was there ever a time when that kind of syntax was normal in other languages? Or is it just arcane syntax, that never had a time when other languages were built like it? Maybe the problem isn't that it's old, but just that it's bad.
Bash is trying to mimic algol68 (poorly).
Allegedly the guy who created bash had a bunch of C macros that made C more algol68-like.
Allegedly the guy who created bash had a bunch of C macros that made C more algol68-like.
You might be thinking of bsh rather than bash: http://oldhome.schmorp.de/marc/bournegol.html
No, that was sh. Steve Bourne. I met him once.
#define IF if(
#define THEN ){
#define ELSE }else{
#define ELSEIF }else if(
#define END }
#define WHILE while(
#define FOR for(
I think it was just cobbled together. I think from the start the shell scripts were a way to arrange and call actual programs. If I remember correctly the first conditionals in shell scripting were literally just calls to the program named "test" I believe or something like that. This resulted in a language with little planning and thought put into it as it expanded to include more and more features.
The biggest problem really seems to be that they wanted to avoid the use of esoteric syntax like double quotes and brackets, which resulted in a very unspecific syntax that is continually uncertain of if you are type strings or commands, so we have to use even more quotes and brackets and things of that nature.
It really makes me wonder why no one seemed to be working on guiding Linux over to some JIT compiled C scripting like TempleOS uses.
The biggest problem really seems to be that they wanted to avoid the use of esoteric syntax like double quotes and brackets, which resulted in a very unspecific syntax that is continually uncertain of if you are type strings or commands, so we have to use even more quotes and brackets and things of that nature.
It really makes me wonder why no one seemed to be working on guiding Linux over to some JIT compiled C scripting like TempleOS uses.
If I remember correctly the first conditionals in shell scripting were literally just calls to the program named "test" I believe or something like that
They still are, in a sense. [ is a synonym for test, and ] is a mandatory no-op argument to it, when used as a “built-in” syntax. You can use [ outside of if as well, like
They still are, in a sense. [ is a synonym for test, and ] is a mandatory no-op argument to it, when used as a “built-in” syntax. You can use [ outside of if as well, like
[ <condition> ] && <then-action>
test <condition> && <then-action>
I really want to use xonsh as my daily but everytime I try to do a git rebase it doesn't handle the inputs properly for my editor.
Sorry for unrelated but I've been curious about this for the last few days! Does anyone know why when I have an alias like
alias jc='python3.10 -m jupyter console'
It still shows python 3.9 if I do print(sys.version)
alias jc='python3.10 -m jupyter console'
It still shows python 3.9 if I do print(sys.version)
Jupyter runs a "kernel" as a child process that actually runs the code in your notebooks. The python version that runs Jupyter can be/is different from the kernel python version.
This makes sense, this would explain the difference.
Is python3.10 an actual binary or a shim or another alias?
which python3.10
which python3.10
python3.10 just points to python installation I manually stuck into a location. I think your sibling comment on my GP post is what I should look at next. Makes sense that the kernel needs to also be installed with the right version. I will check this, thank you and your sibling for the idea.
It’s got parsers for several versions of Python iirc. I’ve used it for a while, but somehow it feels slower than even bash.
Github Copilot has largely solved the “Bash weirdness problem” for me personally, frankly. it’s also made it quite a bit more enjoyable to work with.
Hah that sounds a decent way to throw together some bash scripts if, like me, you're not super fluent in bash syntax! Might actually have to give Copilot a try.
Saw this vid: https://youtu.be/xqQCOigizw4
Saw this vid: https://youtu.be/xqQCOigizw4
So Copilot is its own topic of discussion, but in so far as it and this sentiment are relevant to this thread, I think there's a serious issue here and one particularly relevant to that of Copilot in general.
At some point, I think we have to protect the ability for a human to understand and reason about a technology or a piece of software. Thus the need for better and more economic languages etc.
To sweep the the difficulties of reasoning about a language under the rug of a non-human AI tool that itself isn't providing human readable information on the software or language in question seems to be, quite frankly, an egregious shortcut for short-term gains.
Whatever gains and utility there are in Copilot, bugs creeping in without the developer even understanding them is exactly what many would fear would happen with it. More broadly though, training the developer to be agnostic or ignorant of the issues with the very technology they're using because Copilot is "taking care of it" surely has some negative knock-on effects in terms of developers being able to manage and improve their tech-stacks.
At some point, I think we have to protect the ability for a human to understand and reason about a technology or a piece of software. Thus the need for better and more economic languages etc.
To sweep the the difficulties of reasoning about a language under the rug of a non-human AI tool that itself isn't providing human readable information on the software or language in question seems to be, quite frankly, an egregious shortcut for short-term gains.
Whatever gains and utility there are in Copilot, bugs creeping in without the developer even understanding them is exactly what many would fear would happen with it. More broadly though, training the developer to be agnostic or ignorant of the issues with the very technology they're using because Copilot is "taking care of it" surely has some negative knock-on effects in terms of developers being able to manage and improve their tech-stacks.
Yes, Copilot sounds great for companies who want to spare dev time but bad for society. If software is eating the world and software is a copy of a flawed piece of code, then the foundation of our world may become a bit shaky.
But maybe this is me not wanting to be replaced by the time, I finish studying. Maybe I am just a woodworker and github is Ikea.
Speaking as someone who's used copilot for months...
Programmer jobs are safe.
Brogrammer jobs are not.
It's just a very intelligent autocomplete- your creativity and intelligence is still required.
Programmer jobs are safe.
Brogrammer jobs are not.
It's just a very intelligent autocomplete- your creativity and intelligence is still required.
bash is worse than perl. I usually avoid it with python whenever possible.
I've hated Perl until I had to write significant Bash code.
I still dislike Perl (it has pointers! pointers!), but its syntactic sugar for running shell commands and regexes makes shell-style scripting easier than Python.
I still dislike Perl (it has pointers! pointers!), but its syntactic sugar for running shell commands and regexes makes shell-style scripting easier than Python.
Perl has symbolic references and hard references (pointers). And if you have symbolic reference to a collection of pointers the syntax gets confusing really fast.
May I ask what seemingly horrible meaning do you ascribe to “pointers!” here? Perl references reference values like list, hash, scalar just like Python references lists, dicts, tuples, strings or numbers or any values through a hierarchy of these. The only difference is that in Python you can’t have a list/dict itself, only a reference to it. Perl doesn’t “have” references, it has actual container values instead, which are inaccessible in Python. E.g. @mylist is not an “object”, it is literally a collection of values that will splat into a location of its use. `func(@mylist)` will naturally take n arguments, where n == scalar @mylist. (Unless a prototype of func demands a reference.)
Or maybe you’re mistaking it for pointer arithmetic?
Or maybe you’re mistaking it for pointer arithmetic?
Let me get at this Socratically. What's the syntax to pass a hash to a function, and use it there? Syntax for putting a hash inside an array, or vice-versa? How do you get the keys of a hash that's inside an array?
What's the syntax to pass a hash to a function, and use it there?
None that I know of. But you can put a hashref into a list and vice versa:
But my perl fu weakened after years of wandering and I also remember reading that refs became first-class for common operators and functions (including keys), so dereferencing is not mandatory anymore. Not sure if that’s true, could be a dream easily.
func %h;
sub func {
my %h = @_;
None that I know of. But you can put a hashref into a list and vice versa:
push @a, \%h; # or {…}, $hr
$h{key} = \@a; # or […], $ar
keys %{$a[0]}
keys %{{@a}}
But my perl fu weakened after years of wandering and I also remember reading that refs became first-class for common operators and functions (including keys), so dereferencing is not mandatory anymore. Not sure if that’s true, could be a dream easily.
func %h;
sub func {
my %h = @_;
In any case, these are examples of my discontent with Perl's syntax, which gets unreasonably complicated for non-trivial datastructures. To be clear, Perl's references, the \@a or \%h with the backslashes, are what I derisively call "pointers", because they introduce the same kind of complexity.
It's a problem when you must use different syntaxes to update/refer to a hash, depending on the scope you're using it:
$hash{$key}
$hashref->{$key}
It's a problem when you have to "cast" a type, like in `keys %{$a[0]}` (for the audience: where $a[0] refers to a hashref inside an array, which has to be dereferenced via the %{} syntax, to be used as a hash). This is a simple example, but replace $a[0] with a more sophisticated datastructure, and things get exponentially unwieldy.
The above make it harder to keep track of data as you're browsing code.
In contrast, Python's everything-is-call-by-reference approach means that, whatever the scope, you can refer to something the same way. There never is differing syntax between 'something' and 'a reference to something'.
> I also remember reading that refs became first-class for common operators and functions (including keys), so dereferencing is not mandatory anymore.
This is intriguing, but I can't find any indication of this in Perl's own canonical `perlref` doc, or elsewhere
This is intriguing, but I can't find any indication of this in Perl's own canonical `perlref` doc, or elsewhere
Bash, like Vogon spaceships, doesn't look as if it has been designed so much as congealed.
....just so the author doesn't have to look at StackOverflow, here is his Python example in the video:
Here it is in POSIX Shell/Bourne Shell/Bash/whatever:
The fancy version:
Fancier:
Mucho Fancy (Bash-only):
Uber Fancy: Internal Seq w/Arrays & References (Bash v4+ only):
The version I actually use as a one-liner: