Shell-ish scripting in Go with ease(github.com)
github.com
Shell-ish scripting in Go with ease
https://github.com/bitfield/script
67 comments
>A Sysadmin would take 5 minutes with a shitty language and a shitty tool and get way more done in less time.
Most people aren't sysadmins, but occasionally have to do sysadmin-like things. I've been programming with python and go for years. I've never been able to get the "core" command line utilities to really stick in my head. A sysadmin uses them every day, whereas I rarely have to reach for them. On the rare occasion when I _do_ have to reach for them, it is excruciating (what was the flag I need for `find` again?). If it were life or death and I had to debug even the simplest sed/awk command, it would be death for me! But this package makes perfect sense to me and really enables me to write this sort of quick and dirty thing in a language I'm familiar with and can confidently maintain.
This isn't for everyone, but there's definitely a population that can get a lot of value out of this.
Most people aren't sysadmins, but occasionally have to do sysadmin-like things. I've been programming with python and go for years. I've never been able to get the "core" command line utilities to really stick in my head. A sysadmin uses them every day, whereas I rarely have to reach for them. On the rare occasion when I _do_ have to reach for them, it is excruciating (what was the flag I need for `find` again?). If it were life or death and I had to debug even the simplest sed/awk command, it would be death for me! But this package makes perfect sense to me and really enables me to write this sort of quick and dirty thing in a language I'm familiar with and can confidently maintain.
This isn't for everyone, but there's definitely a population that can get a lot of value out of this.
I completely understand this perspective. But it helps to consider your broader choices.
Let's say you're an "Engineer" engineer, and you deal with units of measurement. You grow up in the US, so you first learn the Imperial system. But so much of the rest of the world uses Metric measurements. Do you find and acquire Imperial versions of every tool, fastener, etc, because it's what you're familiar with? Or do you learn Metric as well, so you can access all of the tools, fasteners, etc found all over the world?
Or take an example from a thousand years ago. Say you're a trader in the Mediterranean and you want to sell your wares. Do you only sell in your local town, where everyone speaks your dialect? Or do you pick up the "Frankish Language", the pidgin language spoken by sailors and other Western Europeans? Learning this mix of Venetian, Catalan, Portuguese, Tamazight, Turkish, Greek, and Arabic will give you extra skills you can use to trade with a vast array of peoples.
POSIX is the closest we have to a 'Mediterranean Basin' for operating systems, and shell scripting is its pidgin language. You don't have to learn it to make a living, but it sure as hell helps.
Let's say you're an "Engineer" engineer, and you deal with units of measurement. You grow up in the US, so you first learn the Imperial system. But so much of the rest of the world uses Metric measurements. Do you find and acquire Imperial versions of every tool, fastener, etc, because it's what you're familiar with? Or do you learn Metric as well, so you can access all of the tools, fasteners, etc found all over the world?
Or take an example from a thousand years ago. Say you're a trader in the Mediterranean and you want to sell your wares. Do you only sell in your local town, where everyone speaks your dialect? Or do you pick up the "Frankish Language", the pidgin language spoken by sailors and other Western Europeans? Learning this mix of Venetian, Catalan, Portuguese, Tamazight, Turkish, Greek, and Arabic will give you extra skills you can use to trade with a vast array of peoples.
POSIX is the closest we have to a 'Mediterranean Basin' for operating systems, and shell scripting is its pidgin language. You don't have to learn it to make a living, but it sure as hell helps.
>what was the flag I need for `find` again
This is not on you, `find`'s command line syntax is awful even if you have to use it everyday.
This is not on you, `find`'s command line syntax is awful even if you have to use it everyday.
[deleted]
> `find`'s command line syntax is awful
What's awful about it?
What's awful about it?
Everything? It, just like tar, don't follow the convention of many other CLI utilities.
That alone is why I prefer `fd` or plocate for most tasks I previously used find for.
That alone is why I prefer `fd` or plocate for most tasks I previously used find for.
Never heard of plocate but I thought fd a step down from find; it's very limited in functionality, colorful output alone can't justify that.
> [It doesn't] follow the convention of many other CLI utilities.
Because unlike other CLI utilities, find evaluates an expression specified as arguments. Maybe a small DSL like JQ would be better, but how would you embed shell code in it? Meh.
> [It doesn't] follow the convention of many other CLI utilities.
Because unlike other CLI utilities, find evaluates an expression specified as arguments. Maybe a small DSL like JQ would be better, but how would you embed shell code in it? Meh.
Find has its uses, I use it frequently to correct improper unix file permissions and owners and when I need to execute a command on a list of files.
However for the vast majority of queries, I'm only looking for something based on the filename, and that's where having sane regex and standard argument structure (like fd has) is great.
However for the vast majority of queries, I'm only looking for something based on the filename, and that's where having sane regex and standard argument structure (like fd has) is great.
> for the vast majority of queries, I'm only looking for something based on the filename
I'd manage with find|grep but yes, nothing wrong with wanting something more compact.
> sane regex
For me that's POSIX ERE, I don't know which flavor fd uses.
I'd manage with find|grep but yes, nothing wrong with wanting something more compact.
> sane regex
For me that's POSIX ERE, I don't know which flavor fd uses.
LLMs are pretty good at using the standard utilities.
That, combined with reading the man pages if it doesn’t work first try has been really effective for me.
That, combined with reading the man pages if it doesn’t work first try has been really effective for me.
I mean, no one really remembers all the flags. They remember a few common ones, due to using them over and over again.
If you are making a genuine effort to avoid the shell, you won't ever learn these basic flags -- you are holding yourself back. It is like a person complaining about using the stairs -- "On the rare occasion when I _do_ take the stairs, it is excruciating..." -- the problem is not necessarily with the stairs.
If you are making a genuine effort to avoid the shell, you won't ever learn these basic flags -- you are holding yourself back. It is like a person complaining about using the stairs -- "On the rare occasion when I _do_ take the stairs, it is excruciating..." -- the problem is not necessarily with the stairs.
I totally agree. My counterpoint is that this tool is the equivalent of portable stair lift. Yes, it may hinder me from developing the ability to walk up the stairs, but I'm fine with that. The only thing that matters is that I get up the stairs.
I believe point 2 is the best. Shell scripts are usually software coordinators. Before even starting you already have a collection of software that already does most of the work. The script is just to speed the execution. As an analogy, it's like serving already cooked dishes you ordered. While a programming language is like having the ingredients, and cooking everything yourself. More versatile, but not that easy. And as you say, the first option is better when everyone's hungry.
Good points. 3rd one reminds me of the Knuth vs McIllroy story about 10+ page pascal vs 6 bash pipes[1]
[1]: https://leancrew.com/all-this/2011/12/more-shell-less-egg/
[1]: https://leancrew.com/all-this/2011/12/more-shell-less-egg/
> I don't know Go well, but it probably doesn't support a similar flexibility.
I know Go okay-ish, and you can remove the 'probably' from that sentence.
Hell, even using Python for shell scripting is a pain in the rear, and that's way more flexible than Go.
I know Go okay-ish, and you can remove the 'probably' from that sentence.
Hell, even using Python for shell scripting is a pain in the rear, and that's way more flexible than Go.
Does shell scripting really need to go hand in hand with the language flexibility ?
You can run a shell command from a language you like so they're just as flexible. Go and many other languages have Exec (or equivalent).
The major reason shell scripting is nice is because its portable with a copy/paste and HelloWorld.sh doesn't need a compiler or a VM or even any preceding incantation to run.
As much as I hate that this is where my hopes are, I do hope that Powershell gets to a point where you can write single C# files and run them.
The major reason shell scripting is nice is because its portable with a copy/paste and HelloWorld.sh doesn't need a compiler or a VM or even any preceding incantation to run.
As much as I hate that this is where my hopes are, I do hope that Powershell gets to a point where you can write single C# files and run them.
D language rdmd wrapper allows to compile-and-execute directly [1].
Together with intuitive function calling facility using Uniform Function Call Syntax or UFCS you can easily has natively compiled scripting environment [2],[3].
[1] Pragmatic D Tutorial:
https://qznc.github.io/d-tut/hello.html
[2] Why I use the D programming language for scripting (2021) (50 comments):
https://news.ycombinator.com/item?id=36928485
[3] Uniform Function Call Syntax:
https://en.wikipedia.org/wiki/Uniform_Function_Call_Syntax
Together with intuitive function calling facility using Uniform Function Call Syntax or UFCS you can easily has natively compiled scripting environment [2],[3].
[1] Pragmatic D Tutorial:
https://qznc.github.io/d-tut/hello.html
[2] Why I use the D programming language for scripting (2021) (50 comments):
https://news.ycombinator.com/item?id=36928485
[3] Uniform Function Call Syntax:
https://en.wikipedia.org/wiki/Uniform_Function_Call_Syntax
You can write executable “scripts” in C by JIT compiling them with the shebang.
Tcc has the `-run` flag for easily doing this with a normal-ish shebang.
With a nasty polyglot preamble of C and bash at the top of your file, you can do it with any compiler.
Tcc has the `-run` flag for easily doing this with a normal-ish shebang.
With a nasty polyglot preamble of C and bash at the top of your file, you can do it with any compiler.
That's a good thing but due to D syntax it is very intuitive and pythonic due to the default GC, in addition to the UFCS feature that I've mentioned compared to C with its agrarian syntax. That's the main reason we have C shell scripting in the form of csh with more intuitive syntax for example foreach loop that D has already supported [1].
But if you insist, D now supports and can compile C language that you can perform using rdmd [2].
[1] C shell:
https://en.wikipedia.org/wiki/C_shell
[2] Adding ANSI C11 C compiler to D so it can import and compile C files directly (105 comments):
https://news.ycombinator.com/item?id=27102584
But if you insist, D now supports and can compile C language that you can perform using rdmd [2].
[1] C shell:
https://en.wikipedia.org/wiki/C_shell
[2] Adding ANSI C11 C compiler to D so it can import and compile C files directly (105 comments):
https://news.ycombinator.com/item?id=27102584
[deleted]
almost all of the most useful scripting languages are some sort of bridge to native functionality . PHP, awk, Perl , to a lesser degree python.
The author is trying to bring the better type checking, concurrency, deterministic syntax and error handling of go to the world of shell scripting.
Most scripts end up becoming programs themselves and go beyond their original scope. It would be nice to force better concurrency & error handling early on before the script takes on 5k lines of functionality.
The author is trying to bring the better type checking, concurrency, deterministic syntax and error handling of go to the world of shell scripting.
Most scripts end up becoming programs themselves and go beyond their original scope. It would be nice to force better concurrency & error handling early on before the script takes on 5k lines of functionality.
I just rewrote a tangled 500 line shell script in go.
It was my first time writing a golang project at work, so I'm sure it could have been better. But writing it the naive way, with all the required golang error handling, it ended up taking about 10x more lines of code in golang than the original bash script.
It does have a dramatically better UX (largely thanks to spf13's cobra and viper), and is way faster than the original, and the codebase is a lot cleaner and more maintainable. So I think it was worthwhile for the users and maintainers.
But still, 10x more lines of code. I like the OP, but I'm still not sure I would reach for golang for short shell scripts.
It was my first time writing a golang project at work, so I'm sure it could have been better. But writing it the naive way, with all the required golang error handling, it ended up taking about 10x more lines of code in golang than the original bash script.
It does have a dramatically better UX (largely thanks to spf13's cobra and viper), and is way faster than the original, and the codebase is a lot cleaner and more maintainable. So I think it was worthwhile for the users and maintainers.
But still, 10x more lines of code. I like the OP, but I'm still not sure I would reach for golang for short shell scripts.
It depends. For single scripts its too much, but if there are dozen or so scripts with related/similar tasks, there can be common code or pattern to be shared. I have one Go project with ~3kloc and does 20 or so operations. But if were to do just single operation it would still need ~1.5K line of code.
Yeah, the original script I rewrote was doing about 15 different operations depending on the user input/arguments, so I guess it indeed reached the point you're describing where there were a lot of common patterns. It's just that instead of being 15 separate scripts, it was one gigantic one with a lot of conditionals and case statements.
This always seemed like the sweet spot for Perl.
500 lines of bash could have been rewritten as 50 lines of dense Perl. Pro: fewer lines of code. Con: even the author would find the Perl script inscrutable after a few weeks.
A more experienced Perl hacker would probably rewrite it as 100 lines of less dense code that people could actually read and comprehend.
I'd probably end up rewriting it as 50 lines of code that used a bunch of pure perl libraries and then use https://p3rl.org/App::FatPacker to bolt the dependencies onto the front for distribution so it was still a single file to install for everybody else.
(there's a lot of perl out there that uses techniques I would switch away from as soon as I got past a one liner in the middle of a pipeline, alas, but it doesn't have to be that way, perl just doesn't stop you blowing both feet off ;)
I'd probably end up rewriting it as 50 lines of code that used a bunch of pure perl libraries and then use https://p3rl.org/App::FatPacker to bolt the dependencies onto the front for distribution so it was still a single file to install for everybody else.
(there's a lot of perl out there that uses techniques I would switch away from as soon as I got past a one liner in the middle of a pipeline, alas, but it doesn't have to be that way, perl just doesn't stop you blowing both feet off ;)
It is. Very much so.
One neat thing about Go that makes it superior to a shell script is that it compiles a statically-linked binary. One self-contained file! Or N if you support N platforms. Did I mention that cross-compilation is trivial?
As someone who once inherited a static binary (without debug symbols, gotta save those few bytes) that should have been a shellscript: Please don't. If your logic reasonably fits into a shell script, then put it there.
Posix shell-compatible scripts will also likely work on all platforms where you go program would've been run.
Posix shell-compatible scripts will also likely work on all platforms where you go program would've been run.
> Posix shell-compatible scripts will also likely work on all platforms where you go program would've been run.
While I see your point, writing a Posix compatible shell script is not trivial. Little errors creep in that "work on my machine" because /bin/sh is a symlink to /bin/bash, but break the script when someone runs it on macOS.
In my experience, you get a lot of cross-platform compatibility when writing Go for zero effort.
While I see your point, writing a Posix compatible shell script is not trivial. Little errors creep in that "work on my machine" because /bin/sh is a symlink to /bin/bash, but break the script when someone runs it on macOS.
In my experience, you get a lot of cross-platform compatibility when writing Go for zero effort.
If a static binary's --help doesn't tell me where the repository lives then I hope the author steps on a lego.
This is a good idea, will add this to my cli tool :)
It is no help to you or your team if they step on the Lego, though.
I mean this entirely depends on how things are done where you work. I wouldn't think the back-reference is necessary if all projects have their own repo in some central location (and it's trivial to match the binary's name to a repo).
Or just make sure the source is available.
Did you do thorough error handling in Bash?
pairing this with yaegi [1] would be interesting. You could having a REPL open doing os operations and when you get the data looking like you want, you select which lines to save to a file.
[1] https://github.com/traefik/yaegi
[1] https://github.com/traefik/yaegi
See also go-script: https://til.simonwillison.net/bash/go-script
I hate go-lang with passion, but these two libs are really cool
How come the hate? That's a pretty strong emotion for something as benign as a programming language.
Not OP, but the usual talking points were covered pretty well in this thread from yesterday: https://news.ycombinator.com/item?id=42884337
To sum it up, the biggest complaints are error handling, null handling, and dependency management. And y’know, being backed by a company of ghouls hellbent on extracting value for themselves at the expense of society.
To sum it up, the biggest complaints are error handling, null handling, and dependency management. And y’know, being backed by a company of ghouls hellbent on extracting value for themselves at the expense of society.
Huh, people are extracting whole bunch of value in Python at expense of society. Should I blame Python or its contributor for it?
I fundamentally disagree with the assertion that Python operates at the expense of society.
Google is an exploitative monopoly. There’s been plenty of ink spilled on the subject to the point that I feel no obligation to repeat it.
While it’s true that GVR is currently employed by Microsoft, the ecosystem of Python is far more anarchic and decentralized.
Google is an exploitative monopoly. There’s been plenty of ink spilled on the subject to the point that I feel no obligation to repeat it.
While it’s true that GVR is currently employed by Microsoft, the ecosystem of Python is far more anarchic and decentralized.
I think he was probably making a comment about Python in regards to PyTorch and AI's energy usage.
i.e. Whether a language exists "at the expense of society" probably depends less on who makes the language, and more on what you do with it.
i.e. Whether a language exists "at the expense of society" probably depends less on who makes the language, and more on what you do with it.
dependency management in Go is best in class what are you talking about? go mod is that good.
Referring to this thread: https://news.ycombinator.com/item?id=42885476
I don’t personally have a bone to pick with Go mod, save for how the GOPROXY DoS issue was handled.
I don’t personally have a bone to pick with Go mod, save for how the GOPROXY DoS issue was handled.
I use Go a lot and I completely disagree. I have also used Ruby a lot and even though I prefer writing Go most of the time (it depends on the task) bundler is far better.
go mod is the second best I've used for sure, but if someone releaed bundler-but-for-go I'd switch to it in a heartbeat.
go mod is the second best I've used for sure, but if someone releaed bundler-but-for-go I'd switch to it in a heartbeat.
Strongly opinionated languages beget strong opinions on the same, both positive and negative.
These sorts of comments always make me wonder what you prefer.
I make a living with go, scala, python, rust, java, and sometimes ts and Js. I prefer scala and sometimes rust.
Too much syntax for a scripting language IMHO.
If anyone wants to experiment with this lib + yaegi interpreter I put up a trivial example at [1]. Composing scripts with LSP support and such might be doable with a proper abstraction, in the example a main package with a main function is required. Interpreting might break some functionality for `script` so perhaps rerunning their test suite with yaegi is a good idea if you get serious about this.
1. https://github.com/danicc097/yaegi-script
1. https://github.com/danicc097/yaegi-script
This is awesome! I enjoy writing Go and like all the tooling around it. Think shell scripts are hard to read and was already planning to adjust some shell-like-Go-tool this weekend, so this post has perfect timing :-)
I'm not going to knock the usefulness of the library, but I am going to knock its application.
Architecturally, shell scripts should _exclusively_ be for bootstraps, configs, or extremely localized (individual developer) automation.
The New York Minute you need non-trivial error-handling/flow-control it's no longer a "shell script" and deserves a proper rewrite in a proper programming language.
Ian Malcom's quote from Jurassic Park comes to mind:
"Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should."
Architecturally, shell scripts should _exclusively_ be for bootstraps, configs, or extremely localized (individual developer) automation.
The New York Minute you need non-trivial error-handling/flow-control it's no longer a "shell script" and deserves a proper rewrite in a proper programming language.
Ian Malcom's quote from Jurassic Park comes to mind:
"Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should."
This. And this is why I like the init system on FreeBSD and OpenRC on Alpine Linux, at least on personal computers. Systemd maybe useful on a server, but I only got a few "services" on my PC and I much prefer something that I can easily understand and hack upon.
Truth! Preach!
a killer feature would be adding a dep tree like make and using goroutines to process the tree concurrently .
Plenty of languages have a framework to assist with shell/Bash scripting. Nothing wrong with that.
If you look at the Python build-in API, it's pretty terrible. In all fairness, Python strives to work equally well on Windows, Linux, and some other things. Good luck with that.
I prefer doing systems scripting in Python over Bash. But writing converting a Bash script to Python without "Pythonizing" it is a bad time.
If you look at the Python build-in API, it's pretty terrible. In all fairness, Python strives to work equally well on Windows, Linux, and some other things. Good luck with that.
I prefer doing systems scripting in Python over Bash. But writing converting a Bash script to Python without "Pythonizing" it is a bad time.
Very nice, bookmarking this!
for larger scripts,just use python or lua.
otherwise,bash is perfect
1. Shell scripting offers infinite functionality. You can shell script with any program in any language. All it needs to do is take input and produce output. And if the functionality doesn't exist, you can create it on the fly, without having to follow any of the traditional rules of programming.
2. Shell scripting is a combination of a grammar, operators, a few simple functions, and an extremely loose coupling with generic i/o and logic. I don't know Go well, but it probably doesn't support a similar flexibility. (most languages are very proscriptive about how you can use the language, so you usually can't make things as easy as they are in a different, more tailored language/interface/paradigm. this is why we have DSLs)
3. Programmers don't really understand the concept of productivity [outside of programming itself]. A programmer would solve a problem by taking 6 weeks to design a perfect program to do the thing. A Sysadmin would take 5 minutes with a shitty language and a shitty tool and get way more done in less time. And re-writing everything into a Go library would always be slower than shell scripting, because it requires re-implementing what a shell script would just use as-is.
Scripting is duct-taping the wheel rather than reinventing it. If you want to save yourself a whole lot of time and trouble, just use the duct tape.
(also: don't go templates exist? why isn't that used for scripting)