The Dune Shell(adam-mcdaniel.github.io)
adam-mcdaniel.github.io
The Dune Shell
https://adam-mcdaniel.github.io/dune-website/
83 comments
I like the ideas of Dune and Nu, which I think were trailblazed by powershell to a degree. The problems of pure *NIX pipe philosophy with text as the lingua franca were persistent parsing problems and escaping. However, the problems of going to far in the way powershell went was it was too radical having to reimagine/replace/relearn almost everything, and generally more difficult to accomplish simple text manipulation. Nu at least maintains most of the *NIX shell UX. Maybe at some point in the future, we will reach parity to where there will fusion of the main DSLs, shells, and programming languages into a runtime interaction system that is somewhere between psql, Erlang shell, nu, and Squeak where there is programming language-OS parity obliterating the distinctions between programming languages, command shells, and DBMS shells. One can dream of a future where perhaps it will be possible to write a function, library, or program in any language one chooses and call it from any other shell or language without swig or ceremony.
The problem of powershell is not merely thet it's new and different, and not merely that you have to learn a new and different way to get a given task done.
Powershell makes some things possible or easier that weren't possible or easy, but only at the expense of making other things either impossible or at least vastly less practical, and all in all it's a bad trade off and a wrong set of priorities for the job it claims to fill, or at least for the job bash currently fills. It's not that bash is a gold standard, just that it's not fundamentally wrong.
Powershell makes some things possible or easier that weren't possible or easy, but only at the expense of making other things either impossible or at least vastly less practical, and all in all it's a bad trade off and a wrong set of priorities for the job it claims to fill, or at least for the job bash currently fills. It's not that bash is a gold standard, just that it's not fundamentally wrong.
It's also just absurdly verbose for a language intended mainly for manual input. I get annoyed having to type out `sudo systemctl` multiple times in a row, in Powershell every single command is at minimum that long. Which makes it way more difficult to memorize the commands too.
There are ton of aliases. It's "impolite" [0] to use aliases in scripts and documentation so looking at PowerShell examples is way more verbose than actually using it day to day in a REPL. Sure I could write `Set-Location` a million times a day, but I just use `cd` in the old ways. Same with `Get-ChildItem`, I tend to just use `ls` myself because of ancient habits. (I find it interesting how many have been moving to `gci` instead as the more PowerShell-native `ls` alias. I've not been convinced to do that myself, but I think it has to do with `gci` is way more powerful than most shells' `ls` and is used to navigate everything from folder structures to object structures including that its not a bad `jq` if you convert JSON files to PowerShell objects.)
[0] It's from a version of the Python ethos that code is read way more often than it is written, so when you are polishing PowerShell code to share with others you expand all the aliases so that it is easier to read.
[0] It's from a version of the Python ethos that code is read way more often than it is written, so when you are polishing PowerShell code to share with others you expand all the aliases so that it is easier to read.
Using a different set of commands in the terminal versus scripts and documentation makes no sense to me, why would I want to learn two different ways do the same thing? Python does not expect me to use `open` in the REPL but prefer `get_file_handle` in published scripts, their ethos works because the entire language has an emphasis on relatively concise readability.
Every single command starting with `Get-` or `Set-` adds so much noise and makes things really hard to visually distinguish too. I don't want overly verbose commands in scripts for the same reason I don't want them in the terminal, it's still a pain to write and even if you have some autocomplete it makes editing harder. I don't think `ls` is any less clear or harder to learn than `Get-ChildItem` really, that name doesn't even actually give me any clue what the command does.
Every single command starting with `Get-` or `Set-` adds so much noise and makes things really hard to visually distinguish too. I don't want overly verbose commands in scripts for the same reason I don't want them in the terminal, it's still a pain to write and even if you have some autocomplete it makes editing harder. I don't think `ls` is any less clear or harder to learn than `Get-ChildItem` really, that name doesn't even actually give me any clue what the command does.
It's not hard to learn. And if you don't want to, then don't. That's really not something someone cares about, unless you're working in a team.
What about the part of the Python ethos where "There should be one-- and preferably only one --obvious way to do it"
A) PowerShell isn't Python, it doesn't have to (and doesn't) import the whole Python ethos, just the parts that make sense for a Shell and Shell language with different end goals.
B) PowerShell doesn't care as much about the first part because shells have always had aliases and macros, but it does care about the second half "obvious way to do it". The other reason for starting with verbose names and aliasing them for day to day REPL use is to aid discovery. It's often very easy to discover which command or cmdlet does the thing you need simply because it is right there in the name. Sometimes you can just guess the name because there are only so many "allowed" verbs and you just need the noun you want to "verb". `Get-Help` can be quite powerful, `Get-Verb` can help explain some bits about what the verbosity is meant to mean. You can search for commands with `Get-Command`.
B) PowerShell doesn't care as much about the first part because shells have always had aliases and macros, but it does care about the second half "obvious way to do it". The other reason for starting with verbose names and aliasing them for day to day REPL use is to aid discovery. It's often very easy to discover which command or cmdlet does the thing you need simply because it is right there in the name. Sometimes you can just guess the name because there are only so many "allowed" verbs and you just need the noun you want to "verb". `Get-Help` can be quite powerful, `Get-Verb` can help explain some bits about what the verbosity is meant to mean. You can search for commands with `Get-Command`.
Python itself doesn't follow that at all, ignore it.
IMO, getting comfortable with and tuning autocomplete makes this a non-issue, and the benefit of the verbose commands is that they are very discoverable.
Any tips or links for tuning? zsh has a powerful engine and lots of great work out there, but when I've tried cooking my own it's all just too dense to get "comfortable" for long.
For VS Code there is a good first-party PowerShell extension which helps a lot at writing scripts in the verbose style.
There's experimental new IntelliSense available in the VS Code terminal: https://code.visualstudio.com/docs/terminal/shell-integratio...
There's experimental new IntelliSense available in the VS Code terminal: https://code.visualstudio.com/docs/terminal/shell-integratio...
Really? I only have to remember Item instead of touch, mkdir, RM, rmdir, cp, mv. All commands that have just different verbs before the Noun in Powershell. And that's for every single cmdlet out there.
It sounds to me that you're unfairly putting your existing knowledge about Linux shells into your criticism of PowerShell. If you want to be fair that criticism should come from the POV of someone who does not now both and now wants to memorise it. And whether or not commands are short or long does not matter. People memorise lyrics, or names of hundreds of people. Length is not a factor, objectively.
And alias do exist.
It sounds to me that you're unfairly putting your existing knowledge about Linux shells into your criticism of PowerShell. If you want to be fair that criticism should come from the POV of someone who does not now both and now wants to memorise it. And whether or not commands are short or long does not matter. People memorise lyrics, or names of hundreds of people. Length is not a factor, objectively.
And alias do exist.
Do you know that there are short versions? e.g. gc for Get-Content
> sudo systemctl
Make an alias like sctl. Takes 5 seconds.
Or an abbr if you are using fish. They are more like text-expanders than aliases.
Make an alias like sctl. Takes 5 seconds.
Or an abbr if you are using fish. They are more like text-expanders than aliases.
True. PS has platform-specific API hooks that are orthogonal to the capabilities of other shells, and more-less replaces JScript, VBScript, and .bat files.
Putting that stuff aside, it is/was one of the first command shells/REPLs outside of SQL DBMS ones in a long time to support manipulation of structured information. For a smoother transition, shells like nu that can straddle structured and file-/line-/word-oriented will be more successful than leaping to too radical of upheaval.
Putting that stuff aside, it is/was one of the first command shells/REPLs outside of SQL DBMS ones in a long time to support manipulation of structured information. For a smoother transition, shells like nu that can straddle structured and file-/line-/word-oriented will be more successful than leaping to too radical of upheaval.
Curious as to what's "vastly less practical" in powershell?
> too radical having to reimagine/replace/relearn almost everything
I switched to zsh because at the time I thought it had better autocomplete, but honestly I use it as I would bash. And I don't think it was worth it to change a little bit.
To change shell I think the change _should_ be radical.
I switched to zsh because at the time I thought it had better autocomplete, but honestly I use it as I would bash. And I don't think it was worth it to change a little bit.
To change shell I think the change _should_ be radical.
Try fish or xonsh.
But it is bash…
Not quite. It has some incompatible default settings, and is missing a small percentage of obscure bash features that I sometimes like to rely on. I'm sure I could do it all with zsh just fine, but it would be more work for me mentally.
Incompatible with what?
Which obscure features?
I can’t actually name them either
It's been several years since I've dug into it and implemented the features I cared about for my personal shell in bash, but I recall that implementing a function to determine if an item exists in an array is significantly shorter and easier in bash 5 due to a feature that is not present in zsh. Probably too obscure for most people to care about, but at the time I was maintaining some rather preposterous bash glue code that relied on that, and compensating for zsh on some people's machines was tedious.
I was scripting on a windows machine which didn’t have WSL setup and decided to learn PowerShell. After a week or so of scripting with PowerShell and seeing how intuitive and robust the syntax is, something changed in my brain and I decided I never again wanted to deal with bash and its endless quirks and brittleness. It’s unnatural to have to wrap every variable access with “${}”, that unnatural key sequence always caused a break in my flow. Anyway I reach for Python sooner and really hunger for a clean, robust nix shell. I hope powershell has instilled this in many people and it drives adoption of something better
I have tried to do the same, but i have opposite experience. I just can't grok the syntax, it always seems like a magician pulling out another "Haha! I bet you haven't seen this one coming !"
Like, i can't even construct the abstract model of how it's supposed to work
Like, i can't even construct the abstract model of how it's supposed to work
Would you be able to provide an example? I'm not doubting your experience, just incredibly curious since I had the opposite experience.
Sure thing :
$env::Path (the semicolon? so Path is not quoted here? But when I assign a value it's quoted?)
Dir -r | %($_.Name.ToLower) ( what is this? statement dreamed by utterly deranged)
They took us for absolute fools
$env::Path (the semicolon? so Path is not quoted here? But when I assign a value it's quoted?)
Dir -r | %($_.Name.ToLower) ( what is this? statement dreamed by utterly deranged)
They took us for absolute fools
I mean you don't need to use the convenience aliases, or use even use it exactly that way. Here's it without any aliases:
Get-ChildItem -Recurse | ForEach-Object { $PSItem.Name.ToLower() }
This might be a better mix of both worlds:
gci -r | foreach { $PSItem.Name.ToLower() }
Get-ChildItem -Recurse | ForEach-Object { $PSItem.Name.ToLower() }
This might be a better mix of both worlds:
gci -r | foreach { $PSItem.Name.ToLower() }
I'm even a windows .NET developer and I also find Powershell extremely weird. I'm not sure what it is.
Interesting thought experiment. Reminds me of my cousin's project Xonsh https://xon.sh/
Happily been using xonsh for the last 6 years. Strongly recommend.
A day or so there was this: https://news.ycombinator.com/item?id=41558081
Imagine your shell's model not being a command line console but a spreadsheet!
Imagine your shell's model not being a command line console but a spreadsheet!
I wonder if there's a good middle ground where unix commands could opt into a structured output, and until they exist, you could shim via `ls -la | shim "ls -la" |> ".modified_time"`. Then one day it could be `ls -la |> ".modified_time"` when structured data is supported natively by the program or command. I feel every project ends up dying since it needs to support every command all at once to be useful.
Shades of Q, APL and other array languages, but visual by default. Maybe an APL spreadsheet-shell would fit the bill? :-D
> One can dream of a future where perhaps it will be possible to write a function, library, or program in any language one chooses and call it from any other shell or language without swig or ceremony
I mean, there's a reason text was selected, it's the lowest common denominator. What you're asking for is universal FFI, which we already have that for the most part: C. The problem ultimately turns into runtime and resource management issues when crossing language boundaries, that is, unless you use OS primatives (pipes, sockets, shared memory, processes, etc.) and use message passing/pipelines exclusively. Then we're back to plaintext.
There's a ton of non-printable characters in ASCII that are useful as delimiters for non-keyboard interactive programs that are largely vestigial. One could at least consider reusing them for delimiters and special control sequences for message passing, but without some sort of standardization it's limited in practical use.
I mean, there's a reason text was selected, it's the lowest common denominator. What you're asking for is universal FFI, which we already have that for the most part: C. The problem ultimately turns into runtime and resource management issues when crossing language boundaries, that is, unless you use OS primatives (pipes, sockets, shared memory, processes, etc.) and use message passing/pipelines exclusively. Then we're back to plaintext.
There's a ton of non-printable characters in ASCII that are useful as delimiters for non-keyboard interactive programs that are largely vestigial. One could at least consider reusing them for delimiters and special control sequences for message passing, but without some sort of standardization it's limited in practical use.
The problem with those ascii values is editors don't display them and there are no keyboard keys to type them.
The reason csv exists is because you can actually see and type all parts of it, on anything, any platform, any hardware, any software, any age, even a mechanical typewriter, even a pencil. That is not merely nice, it's essntially priceless, infinitely, incalculably valuable. The utility outweighs the problems as big as they absolutely are. Actually the same is true for json, yaml, xml...
The reason csv exists is because you can actually see and type all parts of it, on anything, any platform, any hardware, any software, any age, even a mechanical typewriter, even a pencil. That is not merely nice, it's essntially priceless, infinitely, incalculably valuable. The utility outweighs the problems as big as they absolutely are. Actually the same is true for json, yaml, xml...
> The problem with those ascii values is editors don't display them and there are no keyboard keys to type them.
Did you read my post? Like, every word of it? Being able to type a control sequence is not particularly useful for ephemeral data in a message-passing between programs context. It's absolutely important for persistent, editable data. It seems like more of a feature to me to use these special characters for special contexts and not have escaping typeable characters be load-bearing.
Further, my editor seems to print control characters just fine, though entering them would be a bit of a pain and the behavior is likely configurable.
Did you read my post? Like, every word of it? Being able to type a control sequence is not particularly useful for ephemeral data in a message-passing between programs context. It's absolutely important for persistent, editable data. It seems like more of a feature to me to use these special characters for special contexts and not have escaping typeable characters be load-bearing.
Further, my editor seems to print control characters just fine, though entering them would be a bit of a pain and the behavior is likely configurable.
Instead of inventing some new delimited formats, we also already have nearly ubiquitous interchange formats like JSON. If you wanted to reduce the in memory footprint versus plaintext encoding the JSON there are good options like BSON and CBOR.
JSON isn't streamable. Neither is CBOR for that matter, though it's not quite as bad.
Individual JSON records are streamable
JSON itself, no, that's not the case. If you accept an extension like JSON Lines, it can be. Many decoders will barf if you have multiple top level objects in the same file. If you want portable, non-ephemeral data JSON Lines is a poor choice.
Nitpick but I wouldn’t call C a universal FFI, it’s merely the de facto FFI. C can’t meaningfully represent any complex language semantics (not that it should or even could)
It's a near universal de facto FFI if we're being pedantic. My whole point was the reason that's the case is it has essentially no runtime expectations at all, which is a requirement for a truly universal FFI.
> One can dream of a future where perhaps it will be possible to write a function, library, or program in any language one chooses and call it from any other shell or language without swig or ceremony.
Was that not VMS?
Was that not VMS?
There are shells out there that sit between PowerShell / Nu and the old guards like Bash / Zsh.
Ones that support data structures out-of-the-box (like any et al) but also work perfectly fine with existing UNIX commands (like Bash).
Ones that support data structures out-of-the-box (like any et al) but also work perfectly fine with existing UNIX commands (like Bash).
Here is a nice list: https://github.com/oils-for-unix/oils/wiki/Alternative-Shell...
PowerShell seems to work fine calling a nix executable, unless I'm missing something:
> ls -t *.js | head -n 10 | foreach { echo $_ ; Get-ChildItem $_ | fl Name, Length } product_original.js
Name : product_original.js Length : 3353
gulpfile.js
Name : gulpfile.js Length : 382
> ls -t *.js | head -n 10 | foreach { echo $_ ; Get-ChildItem $_ | fl Name, Length } product_original.js
Name : product_original.js Length : 3353
gulpfile.js
Name : gulpfile.js Length : 382
> One can dream of a future where perhaps it will be possible to write a function, library, or program in any language one chooses and call it from any other shell or language without swig or ceremony.
That's a neat idea, and sounds like something that could be built on graalvm.
That's a neat idea, and sounds like something that could be built on graalvm.
To me the silly hang-up for PowerShell is none of that, but the strange syntax. I'd taken (even!) JavaScript-ish syntax over that.
Maybe tabular data is a credible intermediate step. Many shell tools already export data in a tabular format.
"Think of it as an unholy combination of bash and Lisp" reminded me of Scheme Shell: https://scsh.net/
My Ph.D. advisor Olin Shivers created this. It's more pulling bash into Lisp than Lisp into bash.
It doesn't seem to be maintained anymore, but some of its ideas live on in Racket.
My Ph.D. advisor Olin Shivers created this. It's more pulling bash into Lisp than Lisp into bash.
It doesn't seem to be maintained anymore, but some of its ideas live on in Racket.
And now there’s Rash: The Reckless Racket Shell
http://rash-lang.org/
https://github.com/willghatch/racket-rash
http://rash-lang.org/
https://github.com/willghatch/racket-rash
I’ve always wanted to ask about the notorious introduction to scsh’s manual… [1] could Olin have been less than 100% satirical there?
[1] https://scsh.net/docu/html/man.html
[1] https://scsh.net/docu/html/man.html
Nice, it even has Muad'Dib (the desert mouse from the “Dune” movies) as its mascot. The question “but does it have Muad'Dib?!” was the first thing I thought of when I saw the title.
More white rat than desert mouse, I was disappointed. The mouse is also featured in the source material - the books :)
Also the one in the film looked more like a jerboa.
Don’t know what a jerboa is but I quite liked the depiction in the film.
In particular the attention to detail, when they showed a bead of sweat running down its optimally shaped ear, whereupon it was pawed into its mouth.
I loved that little scene so much.
In particular the attention to detail, when they showed a bead of sweat running down its optimally shaped ear, whereupon it was pawed into its mouth.
I loved that little scene so much.
Jerboas ride on your shoulder to alert you of an incoming storm.
In the domain and history that shells inhabit, the design should not prioritize arithmetic operations over indirection. '>>' is a biiig footgun. Elevation of a symbol to non-destructive operation into a destructive one is obviously a wrong choice. I wonder if it was a deliberate decision.
edit: just leave '>>' as append and make '>>>' destructive instead.
edit: just leave '>>' as append and make '>>>' destructive instead.
I was thinking the same.
They should at least make it an option to prompt the user when interactively redirecting to an already existing file. ZSH has this, except for the prompt (you'll need to run the command again using `>!`)
They should at least make it an option to prompt the user when interactively redirecting to an already existing file. ZSH has this, except for the prompt (you'll need to run the command again using `>!`)
I like the ideas behind Powershell, but the DX has always felt ugly and clunky compared to standard shell. Unix/Posix are full of great tools, but they are all extremely painful for newbies and take years to master. There's got to be a middle ground where we can have the power and approachability of SQL with the beautiful elegance of standard Unix pipes.
Dune seems to take a step in that direction which is great.
Dune seems to take a step in that direction which is great.
It kind of sucks that the default shell language is so clunky on Linux O/Ses. Bash marginally better than POSIX sh, and Zsh marginally better.
Even worse is that POSIX forces this to be used.
Using a non POSIX shell is much easier to write (e.g. fish xonsh - and possibly Powershell or rash but I have not used the last two)
sh and derivatives are difficult for me to follow the syntax is impenetrateable - and I was a professional APL programmer.
Using a non POSIX shell is much easier to write (e.g. fish xonsh - and possibly Powershell or rash but I have not used the last two)
sh and derivatives are difficult for me to follow the syntax is impenetrateable - and I was a professional APL programmer.
bash has its own issues but my main gripe is handling of arrays and dictionaries - the syntax is so awkward and hard to remember that I inevitably have to have a reference open in another window. I wish they added some syntactic sugar to make that more ergonomic while retaining backwards compatibility.
This is where Copilot (or other similar tool) help a lot!
The author writes:
With my last shell, Atom, I had accomplished some of the coziness that bash was missing, but I also introduced a lot of really fatal flaws in the syntax and the type system.
I would be interested to read more about this -- what the author learned from that experiment.
With my last shell, Atom, I had accomplished some of the coziness that bash was missing, but I also introduced a lot of really fatal flaws in the syntax and the type system.
I would be interested to read more about this -- what the author learned from that experiment.
Obligatory mention of Emacs Eshell. https://www.gnu.org/software/emacs/manual/html_mono/eshell.h...
The advertisement promises a combination of bash and lisp. So it makes me wonder -- why couldn't we just use lisp?
What are the affordances needed for a "scripting" language (or for interactive use) vis-a-vis more "production use? Is it just about having minimal boilerplate, and a large corpus of ready-to-use functions in the namespace?
What are the affordances needed for a "scripting" language (or for interactive use) vis-a-vis more "production use? Is it just about having minimal boilerplate, and a large corpus of ready-to-use functions in the namespace?
"Is it just about having minimal boilerplate"
That's a lot of it, but I think people don't realize how every keystroke counts with shell.
There's only a handful of languages where the "apply function" operator is space. Shell is one of them. (Haskell & Forth are the other two I know off the top of my head, possibly Factor (concatenative in general tends this way).) Most new shells that are successful copy this. I don't think that's a coincidence. Lisp's abundance of parens is something that people will have trouble with, even Lisp programmers, because this is not the usual whining about a foreign language paradigm and not being used to reading parens... this is literally about the effort required to physically enter them with a keyboard.
That's a lot of it, but I think people don't realize how every keystroke counts with shell.
There's only a handful of languages where the "apply function" operator is space. Shell is one of them. (Haskell & Forth are the other two I know off the top of my head, possibly Factor (concatenative in general tends this way).) Most new shells that are successful copy this. I don't think that's a coincidence. Lisp's abundance of parens is something that people will have trouble with, even Lisp programmers, because this is not the usual whining about a foreign language paradigm and not being used to reading parens... this is literally about the effort required to physically enter them with a keyboard.
Regarding minimal syntax: https://yosefk.com/blog/i-cant-believe-im-praising-tcl.html
Yeah, it really is like that.
I've tried a couple of times to switch to things like IPython as my shell. And what nukes it every time is that while it may be better at the complicated commands I run maybe 1% of the time if you actually look at an unmodified history,
It doesn't look like it when you're doing it once, but when you're doing it dozens of times a minute it adds up fast.
I've tried a couple of times to switch to things like IPython as my shell. And what nukes it every time is that while it may be better at the complicated commands I run maybe 1% of the time if you actually look at an unmodified history,
cd("some dir")
cd som TAB ENTER
It doesn't look like it when you're doing it once, but when you're doing it dozens of times a minute it adds up fast.
In addition to "space is function application" another important shell affordance seems to be "words are just words" -- you don't have to quote simple strings like `README.md`. If you want a single string with more than one word (where there is a space in between), that is in conflict with "space is function application", so then you have to put it in quotes.
https://github.com/babashka/babashka
Granted, if you're saying why can't we have the shell also... well, now you're talking about Emacs.
Granted, if you're saying why can't we have the shell also... well, now you're talking about Emacs.
Process composition is one thing that shell languages do better than traditional programming languages
There’s plenty of LISP packages that support process composition. Also LISPs syntax better suits write once type environments like REPL shells than your average C-derived syntax.
Any examples? Searching for this is hard.
What would `foo | bar | baz` look like in a lisp?
What would `foo | bar | baz` look like in a lisp?
Something like Clojure’s threading macro, probably:
(-> (foo) (bar) (baz))
(-> (foo) (bar) (baz))
Maybe the ultimate would be default to bash but if you start a command with a prefix everything following is parsed and evaluated as an s-expression.
ipython is the complete opposite: python but enhanced with bash. For example:
x = !ls # capture the output of `ls` into x
x[0] # the first filename
...The shell avatar is a rodent but not a desert mouse, not a muad'dib. It's "a shell by the beach", not inspired by any intellectual property you might be thinking of. Yet memorable. Well played.
Looks like it needs some contributors; no releases in almost a year.
Hm, even if I download the sample .dune-prelude, dunesh keeps asking me if this is my first time.
Also, can I make it use a 24h clock and non-US date formatting?
Also, can I make it use a 24h clock and non-US date formatting?
[deleted]