Crush: A command line shell that is also a powerful modern programming language(github.com)
github.com
Crush: A command line shell that is also a powerful modern programming language
https://github.com/liljencrantz/crush
216 comments
I agree 100 %, shells should provide convenient, high level native access to all major RPC mechanisms.
There should simple be a grpc command that you use to instantiate a connection to a grpc server with, and that lets you do discovery, send commands, etc.
It's on my todo list for crush to do that for grpc and dbus.
There should simple be a grpc command that you use to instantiate a connection to a grpc server with, and that lets you do discovery, send commands, etc.
It's on my todo list for crush to do that for grpc and dbus.
This is a shell for people who either don't know Unix or don't like Unix. The normal Unix tools do all these things, often with less typing, and they're a lot more flexible and transferrable to other environments.
I get that many Unix commands are obscure, and the syntax is often inconsistent, but once you're familiar with them, you can do everything this shell does and a whole lot more. And for anything more complex you've got a million other languages to choose from.
I get that many Unix commands are obscure, and the syntax is often inconsistent, but once you're familiar with them, you can do everything this shell does and a whole lot more. And for anything more complex you've got a million other languages to choose from.
I am the author of Crush, I designed Crush for myself: a very happy Unix user for 23 years and someone who can honestly say I love Unix and the Unix philosophy. As such, I feel that I can say with some authority that your first statement is wrong.
The rest of your command is subjective, but I find that being able to see e.g. what user has used a lot of CPU and has many processes running via the command `ps | group ^user cpu={sum ^cpu} proc_count={count}` is both useful and far harder to do in a traditional shell. YMMV.
The rest of your command is subjective, but I find that being able to see e.g. what user has used a lot of CPU and has many processes running via the command `ps | group ^user cpu={sum ^cpu} proc_count={count}` is both useful and far harder to do in a traditional shell. YMMV.
A better PowerShell? I'll take it! What's the story behind the name 'Crush'?
I constructed a list of all words that end in 'sh', filtered out all the words already used for existing shells, sorted the remaining names by how cool they sound and picked the top entry.
cRUSh shares a few letters with Rust (the prog lang of the project). The author made Fish before, a new shell+lang in C++.
A similar idea is Mario, for Python code in shell pipelines. Its most novel feature is (optional) async commands, so you can pipe in a list of urls, make many concurrent requests, and then process them in a single line.
Nushell also seems cool, it's somewhat similar to Powershell. https://github.com/nushell/nushell
cat urls.txt | mario async-map 'await asks.get ! x.json()' apply len
Nushell also seems cool, it's somewhat similar to Powershell. https://github.com/nushell/nushell
This sounds similar to the ideas found in PowerShell, which I really quite like in principle
Neat! Definitely going to try this out.
if the only problem with PowerShell is that it feels clunky and annoying, why not try to extend/improve it (like Kotlin over Java)? Why start a new project that will maybe catch up in X years?
I can imagine lots of reasons. To build on PowerShell means either integrating your changes upstream (not likely, given fixing some of the clunkiness would require breaking changes) or maintaining a separate fork and incorporating patches from upstream constantly (a lot of work, and constant conflicts aren't fun).
Plus the fact that PowerShell is .NET-based and not the most snappy shell out there either.
Plus the fact that extending something requires investing a lot of work just understanding how things tie together, and then avoiding messing up all the hacks and idiosyncrasies added along the way instead of having a clean slate to build on.
Plus the fact that PowerShell is .NET-based and not the most snappy shell out there either.
Plus the fact that extending something requires investing a lot of work just understanding how things tie together, and then avoiding messing up all the hacks and idiosyncrasies added along the way instead of having a clean slate to build on.
I'm not the author but it's probably for the simple reason that writing something from scratch without dependencies can be extremely rewarding and fun.
PowerShell is tied to the CLR APIs. You shouldn't have to run a Mono instance just to get basic shell behavior.
Pretty sure you don't need Mono to use any modern version of PowerShell.
This looks absolutely brilliant.
Reminds me of elvish (elv.sh), which has a similar focus.
If you like the sql-like filesystem interaction, check out:
https://github.com/jhspetersson/fselect
I've integrated this into my workflow, it's a polished tool.
https://github.com/jhspetersson/fselect
I've integrated this into my workflow, it's a polished tool.
This is really cool! I've been waiting for something like this for a long time since I don't use `find` often enough to remember its command line arguments. Finally
If you like the idea of find, but can’t be bothered to remember the args (me neither), might I recommend `fd`! It’s like find, only faster and with a more intuitive, grep-like interface
I think that these 'improved shell' projects (incl. crush and nushell) that try to copy powershell somewhat miss the point of the shell in the first place. If I want a high-level language that's tightly integrated with the unix environment, I already have perl and tcl. Little more power (maybe less integration, but greater capacity to build new integration) and I can use raku or lisp. The point of shell is that it interoperates freely with all programs and all programs share a language. If now, suddenly, I can only use commands that have been explicitly rewritten to support your environment; or regular commands are second-class citizens and need support code to interact with your environment's higher-level functionality——
Then your environment is useless to me. Because I have better programming languages which, despite being able to interact with the system and its commands, don't pretend to be one with the system. As a result, the core high-level language constructs mesh better, because they're not trying to serve two goals at once. It's not possible for something that's not a 'real' shell (including fish, zsh, even csh) to integrate with the system the way a real shell does, because the system itself was designed around the shell.
Then your environment is useless to me. Because I have better programming languages which, despite being able to interact with the system and its commands, don't pretend to be one with the system. As a result, the core high-level language constructs mesh better, because they're not trying to serve two goals at once. It's not possible for something that's not a 'real' shell (including fish, zsh, even csh) to integrate with the system the way a real shell does, because the system itself was designed around the shell.
New shells seem unlikely to catch fire unless they:
* Preserve compatibility with popular shells like bash.
* Address some of the broken parts in a way that makes the world better. Quote handling is the original sin of *nix shells. If somebody fixed that you would have my full attention.
Otherwise I can just stick with Bash + Python3.
* Preserve compatibility with popular shells like bash.
* Address some of the broken parts in a way that makes the world better. Quote handling is the original sin of *nix shells. If somebody fixed that you would have my full attention.
Otherwise I can just stick with Bash + Python3.
Which is impossible, which is why no new shells have caught fire.
In order to preserve compatibility, you must handle quotes, namespaces, etc in the same way as Bourne-derived shells, because they will be expected to run scripts that use nested interpolation, or string declarations without quotes, or whatever. Shells that use a more robust syntax, like Fish or Powershell, can't ever be compatible.
In order to preserve compatibility, you must handle quotes, namespaces, etc in the same way as Bourne-derived shells, because they will be expected to run scripts that use nested interpolation, or string declarations without quotes, or whatever. Shells that use a more robust syntax, like Fish or Powershell, can't ever be compatible.
It's possible, and http://www.oilshell.org/ has done it:
http://www.oilshell.org/release/latest/doc/simple-word-eval....
You can try it right now. Turn on 'shopt -s simple_word_eval' to get the better behavior.
Or turn on the group 'shopt -s oil:basic' and get even more better behavior.
When those options are off, Oil runs some of the biggest shell scripts in the world. It's the shell that's most compatible with bash by a mile.
https://github.com/oilshell/oil/wiki/Shell-Programs-That-Run...
http://www.oilshell.org/release/latest/doc/simple-word-eval....
You can try it right now. Turn on 'shopt -s simple_word_eval' to get the better behavior.
Or turn on the group 'shopt -s oil:basic' and get even more better behavior.
When those options are off, Oil runs some of the biggest shell scripts in the world. It's the shell that's most compatible with bash by a mile.
https://github.com/oilshell/oil/wiki/Shell-Programs-That-Run...
Fish is plenty popular regardless of all that. Sure, it won't replace Bash, in a sense that you'd have Fish handle all the scripts on the system - that's impossible for backwards compatibility reasons. But it's used as a primary interactive shell, and for custom system & user scripts, by a lot of people.
I can't speak to the implementation details but there must be a middle ground where you can set compatibility using 'set -N' or some similar approach to enable new behavior.
Scripting languages like Python, Ruby, Perl, etc., have dealt with upgrade many times. It has not always been pretty, but the alternative is stagnation.
Scripting languages like Python, Ruby, Perl, etc., have dealt with upgrade many times. It has not always been pretty, but the alternative is stagnation.
Yup, it's already been done with 'shopt -s simple_word_eval' in Oil, see right above:
https://news.ycombinator.com/item?id=24086213
And yes I drafted a blog post called "Unix Shell Should Evolve Like Perl 5" which says exactly what you are saying:
http://www.oilshell.org/blog/2020/07/blog-roadmap.html#the-u...
And Oil has already done it. It needs to be faster, but that's an inevitability based on progress so far.
Why Use Oil? http://www.oilshell.org/why.html
https://news.ycombinator.com/item?id=24086213
And yes I drafted a blog post called "Unix Shell Should Evolve Like Perl 5" which says exactly what you are saying:
http://www.oilshell.org/blog/2020/07/blog-roadmap.html#the-u...
And Oil has already done it. It needs to be faster, but that's an inevitability based on progress so far.
Why Use Oil? http://www.oilshell.org/why.html
Not exactly true since PowerShell is doing quite good regarding popularity nowdays...
From my perspective, that's partially because CMD is so stunningly incapable as what a reasonable person might think of as a shell. I'd guess the other half is the amount of pressure Microsoft has placed upon management tooling using PS
If Apple shipped some Swift-powered shell and all of their management documentation started using it, I'd bet a non-trivial portion of administrators would start to use AppleSwiftShell (although I do deeply appreciate that's a severely flawed comparison since macOS already has a sane functioning shell, but it's the best I could think of as a huge company that owns the platform pushing such a concept down)
If Apple shipped some Swift-powered shell and all of their management documentation started using it, I'd bet a non-trivial portion of administrators would start to use AppleSwiftShell (although I do deeply appreciate that's a severely flawed comparison since macOS already has a sane functioning shell, but it's the best I could think of as a huge company that owns the platform pushing such a concept down)
I doubt. Most people still use cmd because they dont care.
Its doing good because it is good and people started to notice once MS propriatery chains went down.
Its doing good because it is good and people started to notice once MS propriatery chains went down.
The way Crush is trying to "fix" quoting is to do less of it.
String interpolation isn't done via quoting but via the format method on the string type, e.g. `"foo {} baz":format "bar"` generates the string "foo bar baz".
Similarly, to execute a command on a remote host, pass in a closure as an argument to the `remote:exec` command, like `remote:exec {1+1} "example.com"`, which will return the value 2.
I find that when quoting becomes a problem it usually means that the wrong solution has already been picked. If you're doing double quoting, you have already lost.
String interpolation isn't done via quoting but via the format method on the string type, e.g. `"foo {} baz":format "bar"` generates the string "foo bar baz".
Similarly, to execute a command on a remote host, pass in a closure as an argument to the `remote:exec` command, like `remote:exec {1+1} "example.com"`, which will return the value 2.
I find that when quoting becomes a problem it usually means that the wrong solution has already been picked. If you're doing double quoting, you have already lost.
fish is pretty popular?
To clarify, I am not saying that everything except for bare /bin/sh without line-editing is Wrong. I wasn't clear enough here. I think that bourne shell, bash, fish, zsh, and csh are all able to interact with the unix system in a way that crush, nushell, and other powershell-alikes cannot.
Can you elaborate on that more of interaction? What can bash, etc do that crush, etc can’t. Is this limitation inherent or easily amended?
I am the original author of both fish and Crush, and I honestly don't know in what way fish interacts with the OS in a way that Crush by virtue of it's design never could.
I know that Crush currently doesn't interact very well with external commands, but I don't see where that is an inherent restriction in the design rather than a missing feature.
I know that Crush currently doesn't interact very well with external commands, but I don't see where that is an inherent restriction in the design rather than a missing feature.
Your ideas are something I wrestled with while I was writing (a very similarly named) crsh[0]. I chose to avoid trying to reimplementing commands and instead stuck with an unstructured environment, but with the ability to parse output using an expandable set of parsers and use inline JavaScript to operate on it. I agree with what you say about perl, but the number of users with experience in perl is -- at least in my bubble -- very small compared with more modern languages.
[0] https://crwi.uk/2020/07/12/crsh.html
[0] https://crwi.uk/2020/07/12/crsh.html
While I have some relatively extensive experience with Perl, I'd rather write assembler using morse than touch that ever again. The language is one guy's not-too-well-thought-through hack that gained enormous traction by filling a void, and from there grew far too organically. There are lots and lots of cryptic ways to do the most mundane things, lots of inconsistencies, the language really makes it hard to write software that isn't terrible to maintain.
Exactly. This is common with shell scripting languages. They feel like some awful duct tape, when in fact the perfect primitive for joining separate concerns already exists (the pipe). Using a nicer language (like modern JS) to perform procedural logic makes perfect sense to me.
> Using a nicer language (like modern JS) to perform procedural logic makes perfect sense to me.
# ls | grep foo
This involves two programs written in nicer language (C).
# ls | rg foo
This involves two programs written in nicer language (C & Rust).
Feel free to come up with JS program that does something well and/or is usable in pipeline. I think you're missing the point of the shell.
# ls | grep foo
This involves two programs written in nicer language (C).
# ls | rg foo
This involves two programs written in nicer language (C & Rust).
Feel free to come up with JS program that does something well and/or is usable in pipeline. I think you're missing the point of the shell.
Aren't we talking about "a better shell scripting language"? At the point where a Perl program becomes "hard to maintain", what would the bash alternative even look like?
Perl code is often written once and then runs for years without issue. I have on occasion fixed broken Perl scripts despite having practically no experience with the language. It wasn't a problem at all, because your average maintenance job is "something was moved" or "we're switching out X for Y", but not "we need to rewrite this complex system that is a Perl script for some reason".
Perl code is often written once and then runs for years without issue. I have on occasion fixed broken Perl scripts despite having practically no experience with the language. It wasn't a problem at all, because your average maintenance job is "something was moved" or "we're switching out X for Y", but not "we need to rewrite this complex system that is a Perl script for some reason".
>It wasn't a problem at all, because your average maintenance job is "something was moved" or "we're switching out X for Y", but not "we need to rewrite this complex system that is a Perl script for some reason".
I would intuitively agree with you as there are hundreds of unnoticeable perl scripts doing their job, but as someone with moderate experience with the language, I can say that when I did maintenance tasks, they were mostly the second kind.
I would intuitively agree with you as there are hundreds of unnoticeable perl scripts doing their job, but as someone with moderate experience with the language, I can say that when I did maintenance tasks, they were mostly the second kind.
You are right, but none of those criticisms apply to Tcl, which is both beautiful and integrates really well with the Unix environment. I won't say that it is perfect, but it is really, really good. A better choice than Perl where Perl is used, a better choice than Lua almost everywhere Lua is used, often a better choice than Python.
Tcl is horribly, terribly under-appreciated. It is one of the great might-have-beens of modern computing, almost as much as is Lisp.
Tcl is horribly, terribly under-appreciated. It is one of the great might-have-beens of modern computing, almost as much as is Lisp.
Eh, having used both Tcl and Lua, I definitely think Lua is the better of the two when it comes to a scripting engine for games.
And I think Lua is more elegant than TCL overall. TCL's stringiness really gets clumsy in certain types of programs, and it's a much larger target than Lua overall.
Not to say that TCL is bad, but I'm glad that dict-based languages prevailed over TCL's string-based-ness
And I think Lua is more elegant than TCL overall. TCL's stringiness really gets clumsy in certain types of programs, and it's a much larger target than Lua overall.
Not to say that TCL is bad, but I'm glad that dict-based languages prevailed over TCL's string-based-ness
This is a very interesting project. I really like the combination of languages and the way you've implemented it. I already see some opportunities to use it.
Thank you! It's pretty buggy at the moment, and maybe frustrating to use since it's missing some important shell features, but any feedback is always welcome.
Yeah I've just installed it and seen eg. the autocompletion on ls isn't how I'm used to. I will keep playing around and watch for progress because I really like the idea.
I especially see this being useful in my CI environment which has a handful of functions, some json and yaml parsing (using jq and python respectively), etc. It's a little cumbersome at the minute but not so much to switch from bash and the rest of the project is JS anyway so it would be good to be consistent.
I especially see this being useful in my CI environment which has a handful of functions, some json and yaml parsing (using jq and python respectively), etc. It's a little cumbersome at the minute but not so much to switch from bash and the rest of the project is JS anyway so it would be good to be consistent.
Excellent. If there are things you notice, please feel free to raise as an issue, and pull requests are always welcome!
> useful in my CI environment
For sure. I've been meaning to play around with getting a docker image with the shell set as the default. It's definitely a use-case I had in mind.
> useful in my CI environment
For sure. I've been meaning to play around with getting a docker image with the shell set as the default. It's definitely a use-case I had in mind.
I'm very impressed with what works so far by the way.
Not many young people know Perl or Tcl (or even lisp for that matter).
>> The point of shell is that it interoperates freely with all programs and all programs share a language.
Also doesn't mean people should stop trying to come up with these new shells. May be some "new" shell will catch up and tomorrow's "all" programs will be compatible with that shell. For example, now on Mac OS X, now ZSH is the default shell.
>> The point of shell is that it interoperates freely with all programs and all programs share a language.
Also doesn't mean people should stop trying to come up with these new shells. May be some "new" shell will catch up and tomorrow's "all" programs will be compatible with that shell. For example, now on Mac OS X, now ZSH is the default shell.
> Not many young people know Perl or Tcl
There are more young people who know Perl or Tcl or Lisp than know Crush. I've never understood this reasoning - "Hardly anyone knows X. Let's replace it with something nobody knows." At most, you're making an argument about a lack of sunk cost.
> May be some "new" shell will catch up and tomorrow's "all" programs will be compatible with that shell
Think of it like transitioning to life in a wheelchair. You can't "catch up" with your house - it just wasn't designed for wheelchair use. Renovating all the narrow corridors, random steps, tight angles and vertical reaches probably costs more than rebuilding.
In this comparison, the mobility mode is your shell.
There are more young people who know Perl or Tcl or Lisp than know Crush. I've never understood this reasoning - "Hardly anyone knows X. Let's replace it with something nobody knows." At most, you're making an argument about a lack of sunk cost.
> May be some "new" shell will catch up and tomorrow's "all" programs will be compatible with that shell
Think of it like transitioning to life in a wheelchair. You can't "catch up" with your house - it just wasn't designed for wheelchair use. Renovating all the narrow corridors, random steps, tight angles and vertical reaches probably costs more than rebuilding.
In this comparison, the mobility mode is your shell.
"Hardly anyone knows X. Let's replace it with something nobody knows."
Because there's a value proposition. Just like TCL, Perl, and Lisp had value propositions over other available languages. It's fine if you don't see the value proposition in Crush but progress sometimes forces us to learn new things instead of sticking with what we have.
Because there's a value proposition. Just like TCL, Perl, and Lisp had value propositions over other available languages. It's fine if you don't see the value proposition in Crush but progress sometimes forces us to learn new things instead of sticking with what we have.
Crush has been around < 1 year. Perl has been around for 32. Of course more people know Perl.
If I’m being honest, you’re not adding anything constructive at all here - it seems you’re in a competition with yourself as to how much you can degrade Crush for whatever reason.
You act like it’s a choice between Crush and Perl. People don’t have to learn Crush. People don’t have to learn Perl. Will picking between the two have any substantial effect on one’s career outcomes these days? Probably not, as candidates will have already learned the important languages for getting a job anyway.
If I’m being honest, you’re not adding anything constructive at all here - it seems you’re in a competition with yourself as to how much you can degrade Crush for whatever reason.
You act like it’s a choice between Crush and Perl. People don’t have to learn Crush. People don’t have to learn Perl. Will picking between the two have any substantial effect on one’s career outcomes these days? Probably not, as candidates will have already learned the important languages for getting a job anyway.
If I'm being honest, I have no idea where you got the belief that either I'm some big Perl partisan or that I have something against Crush, or why you're getting so personal.
But it is cute to couch it in a complaint about not being constructive; points for shamelessness.
But it is cute to couch it in a complaint about not being constructive; points for shamelessness.
>> Hardly anyone knows X. Let's replace it with something nobody knows
By that logic no one should make anything new to replace the incumbent.
In case of Perl, PHP came and it replaced Perl as mainstream web development language.
In sysadmin stuff where Perl used to be mainstream, now significant chunk is taken away by Python.
These days JavaScript appeals to younger people more. Times change and with it the tools people use also change.
There is no guarantee that any new project will succeed but you won't know unless you try.
By that logic no one should make anything new to replace the incumbent.
In case of Perl, PHP came and it replaced Perl as mainstream web development language.
In sysadmin stuff where Perl used to be mainstream, now significant chunk is taken away by Python.
These days JavaScript appeals to younger people more. Times change and with it the tools people use also change.
There is no guarantee that any new project will succeed but you won't know unless you try.
The original point was "why would I learn a new thing, when I already know something which works". The response "some people don't have something they already know" is valid.
You're really bringing up a separate point which is "should we be reinventing the wheel?", I think that's a fair point, but it's not a response to the original commentator.
You're really bringing up a separate point which is "should we be reinventing the wheel?", I think that's a fair point, but it's not a response to the original commentator.
It only took Zsh 29 years (initial release year: 1990, adoption in Mac OS: 2019) to become the default shell on one of the major nixes, while being POSIX compatible and having just an extension layer on top.
I wouldn't hold my breath for something else to replace sh/bash as the default shell on the major nixes within this century. Pessimist? More like realist :-)
I wouldn't hold my breath for something else to replace sh/bash as the default shell on the major nixes within this century. Pessimist? More like realist :-)
That wasn’t even driven by zsh’s actual benefits, but by the fact that bash changed to GPL3. If it had continued on GPL2 we’d 100% still have bash as a default.
Most other major UNIXes, before of BSD/Linux takeover weren't on sh, rather tsch and ksh.
Many young people don't know history, science, other languages, or even the grammar of their native language. That is why we teach.
The "ecosystem" of programming languages is healthy. There are many options, some to master, some to consider. But not knowing how to use a tool -- especially a widely-used and powerful tool -- is no badge of honour or discernment.
The "ecosystem" of programming languages is healthy. There are many options, some to master, some to consider. But not knowing how to use a tool -- especially a widely-used and powerful tool -- is no badge of honour or discernment.
> That is why we teach.
but what if people don't know how to learn?
instead of getting wrapped up in all of this 'knowledge preservation' nonsense, shouldn't we just mimic what they are already doing and assume it to be better, because the activity occurs in a year number that is higher than the previous behavior pattern?
now that would be real progress.
or, not.
but what if people don't know how to learn?
instead of getting wrapped up in all of this 'knowledge preservation' nonsense, shouldn't we just mimic what they are already doing and assume it to be better, because the activity occurs in a year number that is higher than the previous behavior pattern?
now that would be real progress.
or, not.
Sarcasm duly noted. (o;
Societal results sadly noted.
Societal results sadly noted.
> Not many young people know Perl or Tcl (or even lisp for that matter).
I don't know perl and I'm not young, but I do understand and appreciate the role perl -e plays as an extension of bash. When I run into an issue where I need more than bash, but less than python, I'll search around SO for a perl -e command for what I need.
This is often how people decide to learn a tool: they have a problem for which it is the best solution. They get some exposure to the capability and slowly expand on their knowledge.
I don't know perl and I'm not young, but I do understand and appreciate the role perl -e plays as an extension of bash. When I run into an issue where I need more than bash, but less than python, I'll search around SO for a perl -e command for what I need.
This is often how people decide to learn a tool: they have a problem for which it is the best solution. They get some exposure to the capability and slowly expand on their knowledge.
I explicitly included zsh. The point is not the language of the shell itself; that has never been the point. The point is that zsh interoperates with the other programs on your system in a way that crush doesn't and can't.
> a way that crush doesn't and can't
What do you mean by can't? If it misses some mode of interaction, that can be always added.
What do you mean by can't? If it misses some mode of interaction, that can be always added.
Consistent behavior when piping through all commands. A lack of modes of operation to consider. Simplicity.
Simplicity exists on more than a single dimension. Choosing one column from "ls" output for example is not simple, bordering on impossible using the consistent behaviour of standard pipes.
Sometimes the inconsistent solution will be simpler by moving the complexity to the choice of the mode of interaction. Sometimes not.
Sometimes the inconsistent solution will be simpler by moving the complexity to the choice of the mode of interaction. Sometimes not.
> Choosing one column from "ls" output for example is not simple, bordering on impossible using the consistent behaviour of standard pipes.
ls -l | awk '{print $3}'
ls -l | cut -f 3 -d ' 'Maybe a shell is the wrong tool for that job, and hiding the modes that the language is operating in complicates things.
You are correct. It is all about how you think about things.
"ls" lists files. Sure. Simple interactive command. But, thinking about this, "echo " also lists files. "ls" provides information about files. Sure. Simple interactive command. But, "stat" specifically provides information.
So, I would reach for
stat --format "%n %a"
To get name and octal permissions. You are correct, and wrong at the same time. Easy enough to accomplish -- but using * instead of ls. Note that the "space in filename" becomes the big issue (use %N in stat format).
So your conclusion is 100% correct. Thank you for your post and insight
"ls" lists files. Sure. Simple interactive command. But, thinking about this, "echo " also lists files. "ls" provides information about files. Sure. Simple interactive command. But, "stat" specifically provides information.
So, I would reach for
stat --format "%n %a"
To get name and octal permissions. You are correct, and wrong at the same time. Easy enough to accomplish -- but using * instead of ls. Note that the "space in filename" becomes the big issue (use %N in stat format).
So your conclusion is 100% correct. Thank you for your post and insight
Nor do they know that before bash takeover of the UNIX world via BSD and GNU/Linux clones, it was just another shell among many others, with tsch and ksh also having a place on the table.
when he says Perl or Tcl I think he means general purpose languages. It doesn't have to be Perl, or Lisp or Tcl, those are examples.
“somewhat miss the point of the shell in the first place”
This presupposes that the Unix shell is the way it is because of careful, considered, forward-thinking design, rather than being a haphazard heap of mouldering hacks upon lazy corner-cutting upon quick-n-dirty bodges that should’ve been thrown out and replaced decades ago.
I agree that trying to improve the *nix shell is a bad idea… because the _Unix shell itself_ is a bad idea. Fifty years is long enough. Instead of bending over backwards for that obsolescent garbage heap (for which no-one will thank you, as we see), study it, learn its lessons, and draw a line under it.
Then go build a clean, modern shell architecture from scratch, using modern tooling and techniques, unhobbled by the myriad defects and deficiencies of the old. It’ll take a tenth of the time and be ten times better (if not, you’re still doing it wrong).
See also:
https://www.google.com/search?q=sunk+cost+fallacy
https://homes.cs.washington.edu/~weise/uhh-download.html
--
p.s. As for driving early adoption, don’t even think of trying to win over the Unix old guard (who will only bring regressive practices and complaints with them), but sell it hard to the new generations of users who are accustomed to modern UI/UX standards and much less impressed by what the “Old Unix” dogpile has to offer.
This presupposes that the Unix shell is the way it is because of careful, considered, forward-thinking design, rather than being a haphazard heap of mouldering hacks upon lazy corner-cutting upon quick-n-dirty bodges that should’ve been thrown out and replaced decades ago.
I agree that trying to improve the *nix shell is a bad idea… because the _Unix shell itself_ is a bad idea. Fifty years is long enough. Instead of bending over backwards for that obsolescent garbage heap (for which no-one will thank you, as we see), study it, learn its lessons, and draw a line under it.
Then go build a clean, modern shell architecture from scratch, using modern tooling and techniques, unhobbled by the myriad defects and deficiencies of the old. It’ll take a tenth of the time and be ten times better (if not, you’re still doing it wrong).
See also:
https://www.google.com/search?q=sunk+cost+fallacy
https://homes.cs.washington.edu/~weise/uhh-download.html
--
p.s. As for driving early adoption, don’t even think of trying to win over the Unix old guard (who will only bring regressive practices and complaints with them), but sell it hard to the new generations of users who are accustomed to modern UI/UX standards and much less impressed by what the “Old Unix” dogpile has to offer.
> “because the system itself was designed around the shell.”
Windows wasn’t designed around the command prompt, it wasn’t designed around shell utilities. When I call “cat”, it’s not important that I’m writing those three letters, they are a library way of calling a set of file open/read OS functions. When I write “ping”, it’s not the ping tool I care about, it’s the network api calls it does behind the scenes. The Unix/Linux shell is then seen like a “Python standard library” of ways the user can control the OS, and it’s a hugely inconsistent, undesigned, ad-hoc agglomeration of names and parameters and bodges, which cheats out on dealing with the complexity and instead offloads that straight on to the user in a user hostile way.
What powershell tries to do for Windows is make a somewhat designed, consistent, and more user-approachable set of wrappers around the Windows OS system calls, along with guidelines and support for vendors to build in that style, a human interface that is introspectable, built to be programmable, but still composable and modular.
The objection that trying to do such a user-focused thing on Unix makes a “useless” system is weird because that - a user interface to tools/library of OS call patterns - is what the shell is. In what way is “resolve-dnsname” pretending to be one with the system that dig and nslookup aren’t? They’re names which trigger a pattern of network traffic and show a result. In what way is piping text from dig to sed “real” and piping resolve-dnsname to format-list “pretend”?
In what way is running “docker compose” from sh real, but running it from zsh is pretend?
In what way is it a better design to have (a pile of shell tools and Python library wrappers and Python wrapping shell tool calls), instead of (a Python based shell with transparent shell tool calls)?
Like, “I want what I’m familiar with, too many things to relearn”, or “too many things are awkward in this approach (cough PowerShell)” are reasonable objections, but “trying for a more consistent, more human friendly, interface with more high level language constructs at one’s fingertips makes a thing useless to me” feels heel dragging or Stockholm syndromeish. Presumably you don’t object to there being a choice of multiple command line tools to do a task? Why object on principle to a choice of other OS “front end experiences”? Is there any version of things which you can imagine as being both “better” (less arbitrary and inconsistent than shell) and “not pretend”?
> “The point of shell is that it interoperates freely with all programs and all programs share a language.”
What language? Some treat piped bytes as bytes (e.g. piping a binary file), others as ASCII, others as UTF-8. All support different parameter options, names and values and ways of specifying them. Many handle output formatting in different ways, many trigger internal states from environment variables. If the point is a common shared language, the point doesn’t seem to have been achieved from a user perspective. And if the point is that “I have to suffer a system that’s tied to a shared language with an AIX box in a bank basement last touched by a human in 1997” the point is a bad one. I use my computer a lot more than I use AIX and I have more computing power to spare on human comforts and conveniences.
Windows wasn’t designed around the command prompt, it wasn’t designed around shell utilities. When I call “cat”, it’s not important that I’m writing those three letters, they are a library way of calling a set of file open/read OS functions. When I write “ping”, it’s not the ping tool I care about, it’s the network api calls it does behind the scenes. The Unix/Linux shell is then seen like a “Python standard library” of ways the user can control the OS, and it’s a hugely inconsistent, undesigned, ad-hoc agglomeration of names and parameters and bodges, which cheats out on dealing with the complexity and instead offloads that straight on to the user in a user hostile way.
What powershell tries to do for Windows is make a somewhat designed, consistent, and more user-approachable set of wrappers around the Windows OS system calls, along with guidelines and support for vendors to build in that style, a human interface that is introspectable, built to be programmable, but still composable and modular.
The objection that trying to do such a user-focused thing on Unix makes a “useless” system is weird because that - a user interface to tools/library of OS call patterns - is what the shell is. In what way is “resolve-dnsname” pretending to be one with the system that dig and nslookup aren’t? They’re names which trigger a pattern of network traffic and show a result. In what way is piping text from dig to sed “real” and piping resolve-dnsname to format-list “pretend”?
In what way is running “docker compose” from sh real, but running it from zsh is pretend?
In what way is it a better design to have (a pile of shell tools and Python library wrappers and Python wrapping shell tool calls), instead of (a Python based shell with transparent shell tool calls)?
Like, “I want what I’m familiar with, too many things to relearn”, or “too many things are awkward in this approach (cough PowerShell)” are reasonable objections, but “trying for a more consistent, more human friendly, interface with more high level language constructs at one’s fingertips makes a thing useless to me” feels heel dragging or Stockholm syndromeish. Presumably you don’t object to there being a choice of multiple command line tools to do a task? Why object on principle to a choice of other OS “front end experiences”? Is there any version of things which you can imagine as being both “better” (less arbitrary and inconsistent than shell) and “not pretend”?
> “The point of shell is that it interoperates freely with all programs and all programs share a language.”
What language? Some treat piped bytes as bytes (e.g. piping a binary file), others as ASCII, others as UTF-8. All support different parameter options, names and values and ways of specifying them. Many handle output formatting in different ways, many trigger internal states from environment variables. If the point is a common shared language, the point doesn’t seem to have been achieved from a user perspective. And if the point is that “I have to suffer a system that’s tied to a shared language with an AIX box in a bank basement last touched by a human in 1997” the point is a bad one. I use my computer a lot more than I use AIX and I have more computing power to spare on human comforts and conveniences.
"What language?"
The UNIX philosophy was to write a solid system by taking most of the hard problems in kernel space and kicking them into user space. This is not a slur, this is a description. Especially at the time, it wasn't at terrible solution. I kind of think of it the same way as many programming languages have observed that "the standard library is where things go to die"; the code in a system that is hardest to change, well, it doesn't change! So any bad decision that ever gets in there is almost permanent. Part of the reason UNIX won is that it enshrined fewer bad decisions in the kernel than its competition did. I think you could even argue it had more bad decisions overall than its competition, at least towards the beginning; it's just that far fewer of them were enshrined in the kernel.
This is also the philosophy of the pipe. It disclaims all layout and moves that responsibility on to the consumers. There is no "shell language", at all. This is, simultaneously, a big strength and a big weakness.
One aspect of that is precisely that the strength of this approach tends to inhibit anyone who tries to beat it with something like Powershell. All Powershell had to be was better than any shell on Windows. This was not hard. For something like Crush, to take off it has to be better than any existing shell on UNIX, and as annoying as unstructured pipes may be, the ecosystem as a whole still has a lot of positives about it, because it's been tuned for 50+ years. It's going to take a lot of work to match that, and so far, it doesn't seem like any of these projects are achieving take-off velocity.
Arguably, the best thing to do would be for all these projects to get together and try to define a standard UNIX-level non-textual rich serialization of some sort, using lessons from Powershell and arguably things like D-Bus to do it, because none of these projects individually seem to be on a success trajectory.
The UNIX philosophy was to write a solid system by taking most of the hard problems in kernel space and kicking them into user space. This is not a slur, this is a description. Especially at the time, it wasn't at terrible solution. I kind of think of it the same way as many programming languages have observed that "the standard library is where things go to die"; the code in a system that is hardest to change, well, it doesn't change! So any bad decision that ever gets in there is almost permanent. Part of the reason UNIX won is that it enshrined fewer bad decisions in the kernel than its competition did. I think you could even argue it had more bad decisions overall than its competition, at least towards the beginning; it's just that far fewer of them were enshrined in the kernel.
This is also the philosophy of the pipe. It disclaims all layout and moves that responsibility on to the consumers. There is no "shell language", at all. This is, simultaneously, a big strength and a big weakness.
One aspect of that is precisely that the strength of this approach tends to inhibit anyone who tries to beat it with something like Powershell. All Powershell had to be was better than any shell on Windows. This was not hard. For something like Crush, to take off it has to be better than any existing shell on UNIX, and as annoying as unstructured pipes may be, the ecosystem as a whole still has a lot of positives about it, because it's been tuned for 50+ years. It's going to take a lot of work to match that, and so far, it doesn't seem like any of these projects are achieving take-off velocity.
Arguably, the best thing to do would be for all these projects to get together and try to define a standard UNIX-level non-textual rich serialization of some sort, using lessons from Powershell and arguably things like D-Bus to do it, because none of these projects individually seem to be on a success trajectory.
> For something like Crush, to take off it has to be better than any existing shell on UNIX, and as annoying as unstructured pipes may be, the ecosystem as a whole still has a lot of positives about it, because it's been tuned for 50+ years. It's going to take a lot of work to match that, and so far, it doesn't seem like any of these projects are achieving take-off velocity.
Who cares about "taking off"? If it's useful for 10s of users, it has taken off.
Consider xonsh (https://xon.sh/). I don't think it will ever be "mainstream" the way bash or zsh are. I doubt it will ever come preinstalled in any major Linux distribution. But it's used by enough and has enough momentum that I don't worry it will die. That alone is enough for me to use it as my primary shell. I don't care that xonsh will never "take off".
Who cares about "taking off"? If it's useful for 10s of users, it has taken off.
Consider xonsh (https://xon.sh/). I don't think it will ever be "mainstream" the way bash or zsh are. I doubt it will ever come preinstalled in any major Linux distribution. But it's used by enough and has enough momentum that I don't worry it will die. That alone is enough for me to use it as my primary shell. I don't care that xonsh will never "take off".
Failure to take off means that instead of getting a long tail of things that work, you get a smaller island of reduced functionality, even if that is enough to keep you using it.
It doesn't have to be the goal of the project or the author, but there are real advantages to achieving a takeoff, even to the users of a product.
It doesn't have to be the goal of the project or the author, but there are real advantages to achieving a takeoff, even to the users of a product.
A philosophy that is more cargo cult than anything else, as anyone with work experience in commercial UNIXes knows.
What is the one thing that GNU ls does well, by the way?
What is the one thing that GNU ls does well, by the way?
"What is the one thing that GNU ls does well"
Backwards compatibility with itself, probably.
If you came away from my post with the impression that this system produces a lot of elegant software, well, I must have screwed up in my writing somehow. What it produces is bodges upon bodges, which are then polished by years of ramming into the corner cases and putting in another bodge to fix them. There is actually something to be said for this design methodology, and it's not all bad, but it rarely produces elegance.
Backwards compatibility with itself, probably.
If you came away from my post with the impression that this system produces a lot of elegant software, well, I must have screwed up in my writing somehow. What it produces is bodges upon bodges, which are then polished by years of ramming into the corner cases and putting in another bodge to fix them. There is actually something to be said for this design methodology, and it's not all bad, but it rarely produces elegance.
Which has nothing to do with "UNIX philosophy", as expected.
Oh, heh, if you were expecting a defense of the "UNIX philosophy" of doing one thing well from me, you're not going to get it. I think one small step by one small step that has been abandoned over time, and because it was so incremental there's no one place to point at where it went over the line and it's easy for people to miss it because there is no such point, but cross the line it has.
Now, at least GNU ls doesn't send or receive email or something. A quick scan of its man page doesn't show many any features grossly out of line with its primary task. But yeah, it has a lot of features for sure, many of which nominally overlap with tasks that could be done by other tools (sort, ignoring backups, -D ("generate output designed for Emacs' dired mode", lol), quoting styles...).
I feel much the same way about Python. Very little of its philosophy expressed back around 2000 or so is in effect today. And there's no one release you can point at and say, "this is it, this is when it stopped being simple", yet, clearly, today it is no longer simple.
Now, at least GNU ls doesn't send or receive email or something. A quick scan of its man page doesn't show many any features grossly out of line with its primary task. But yeah, it has a lot of features for sure, many of which nominally overlap with tasks that could be done by other tools (sort, ignoring backups, -D ("generate output designed for Emacs' dired mode", lol), quoting styles...).
I feel much the same way about Python. Very little of its philosophy expressed back around 2000 or so is in effect today. And there's no one release you can point at and say, "this is it, this is when it stopped being simple", yet, clearly, today it is no longer simple.
Agreed, that was my whole point.
The "UNIX philosophy" gets cargo cult by younger generations thanks to a couple of statements in a few worshipped books, yet there is hardly any UNIX clone post V6 where it was actually followed upon.
The "UNIX philosophy" gets cargo cult by younger generations thanks to a couple of statements in a few worshipped books, yet there is hardly any UNIX clone post V6 where it was actually followed upon.
This seems to strike a good compromise as "third party" commands are "lines" type which is basically what we are used to send and receive in unix tools. Of course a lot of tools already support emitting JSON so it seems the Crush is already more useful than the typical unix shell as you can still use "lines" for every program (and convert to a table by parsing if you want to use the more advanced features) and there are already a number of third-party programs that you take directly take advantage of these feature with even though they didn't know anything about this shell.
I think the fact that you can already use this with most programs and it is never less powerful than bash seems like good indication that the system doesn't need to be reworked around this shell. It fits in cleanly to existing paradigms.
I think the fact that you can already use this with most programs and it is never less powerful than bash seems like good indication that the system doesn't need to be reworked around this shell. It fits in cleanly to existing paradigms.
You really aren't kidding about Tcl. I recently discovered that with tclreadline, Tcl can be used completely in place of Bash in interactive mode, in a way that something like Python absolutely cannot.
A lot of people seem to have negative opinions about Tcl (maybe because it's more of a LISP than its syntax seems to imply), but as someone who is just discovering it, I'm shocked at how powerful it is for how unpopular it seems to have become. It's like some kind of unexpected cross between Bash, LISP, and (with Tk) Visual Basic. There's something about it that's just kind of fun.
A lot of people seem to have negative opinions about Tcl (maybe because it's more of a LISP than its syntax seems to imply), but as someone who is just discovering it, I'm shocked at how powerful it is for how unpopular it seems to have become. It's like some kind of unexpected cross between Bash, LISP, and (with Tk) Visual Basic. There's something about it that's just kind of fun.
Tcl is awesome. I used it constantly in the 1990s. It's outstanding for things like testing and build scripts.
I think some of the pushback might be because people tried to use it to build large applications and failed. Languages like Java and (dare I say it) Visual Basic were more comprehensive and had good IDE support.
I think some of the pushback might be because people tried to use it to build large applications and failed. Languages like Java and (dare I say it) Visual Basic were more comprehensive and had good IDE support.
Tcl has features that are very powerful, but are really hard for a beginner, and really easy for a novice programmer to shoot themselves in the foot with.
Yet Tcl tends to be forced on undergraduate programmers, who probably just learn to loathe it. I lost several toes to it myself decades ago, and haven’t really felt inclined to pick it up again.
Yet Tcl tends to be forced on undergraduate programmers, who probably just learn to loathe it. I lost several toes to it myself decades ago, and haven’t really felt inclined to pick it up again.
Right, I'm often asked this about Oil [1]: Why do you want to write programs in shell?
That's not the idea of shell. The idea is that I write programs in Python, JavaScript, R, and C++ regularly, and about 10 different DSLs (SQL, HTML, etc.) And I work on systems written by others, consisting of even more languages.
I need a language to glue them together. A language to express build automation and describe deployed systems. Most big systems consist of more than one language.
Shell is the best language for that, but it's also old and crufty, with poor implementations.
When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library".
[1] http://www.oilshell.org/
That's not the idea of shell. The idea is that I write programs in Python, JavaScript, R, and C++ regularly, and about 10 different DSLs (SQL, HTML, etc.) And I work on systems written by others, consisting of even more languages.
I need a language to glue them together. A language to express build automation and describe deployed systems. Most big systems consist of more than one language.
Shell is the best language for that, but it's also old and crufty, with poor implementations.
When you program in shell, gcc, git, pip, npm, markdown, rsync, diff, perf, strace, etc. are part of your "standard library".
[1] http://www.oilshell.org/
Update: also see the philosophy for structured data in Oil below: https://news.ycombinator.com/item?id=24083931
yeah perl with qx/cmd/ calls and similar is just really really good for "shell scripting".
Exactly!
This is why I started my shell. Eventually it will have the ability to run both POSIX and other shell languages in the same program.
Assuming everything is a table is overly restrictive. Assuming everything is text is the UNIX way. It's then the job of the terminal to render it.
This is why I started my shell. Eventually it will have the ability to run both POSIX and other shell languages in the same program.
Assuming everything is a table is overly restrictive. Assuming everything is text is the UNIX way. It's then the job of the terminal to render it.
Because I have better programming languages which, despite being able to interact with the system and its commands, don't pretend to be one with the system. As a result, the core high-level language constructs mesh better, because they're not trying to serve two goals at once.
The power and usefulness of a given language is often the ability to serve multiple goals at once. Python as tool is cool specifically because of a large-ish but accessible-ish library covering a everything from regex-ex to a web-server.
This project might not serve the goal of writing high level programs and interacting effortless with the command line/system. But I don't one should automatically dismiss the goal of making stuff-glued-together look like a coherent whole.
The power and usefulness of a given language is often the ability to serve multiple goals at once. Python as tool is cool specifically because of a large-ish but accessible-ish library covering a everything from regex-ex to a web-server.
This project might not serve the goal of writing high level programs and interacting effortless with the command line/system. But I don't one should automatically dismiss the goal of making stuff-glued-together look like a coherent whole.
You are forgetting about how shell scripts start. "I don't want to keep writing these 5 shell commands, I'll put them in a file. Ok that is useful, I'll add some parameters. This shell script works great, I'll add more functionality."
And before you know it you have a buggy monster written in a batshit insane language that nobody is going to want to touch.
If your shell is at least vaguely sane, the situation is a lot better. I don't really get why you would argue against something that fixes the many many many mistakes of Bash.
And before you know it you have a buggy monster written in a batshit insane language that nobody is going to want to touch.
If your shell is at least vaguely sane, the situation is a lot better. I don't really get why you would argue against something that fixes the many many many mistakes of Bash.
To a first approximation, people who are ok with Bash as a scripting language do not know enough Bash to safely use it as a scripting language.
I see many insane constructs that accidentally work. I also know how much time it would take to explain expansion rules. And people don't even want to hear it because it shakes their simplistic (and inconsistent) understanding.
So yes, I hope we'll see something like crush replacing Bash. But I don't see how it will happen.
I see many insane constructs that accidentally work. I also know how much time it would take to explain expansion rules. And people don't even want to hear it because it shakes their simplistic (and inconsistent) understanding.
So yes, I hope we'll see something like crush replacing Bash. But I don't see how it will happen.
It's my understanding that commands don't have to be specially written for Crush. As long as they speak an understandable format, they can be used. https://github.com/liljencrantz/crush#reading-and-writing-fi...
I don’t think nushell or crush for that matter copy PowerShell. They have learnt from MS and built upon what they didn’t get right. Strongly typed objects on the command line is theoretically interesting, but slow and complex in practice (why does it still take 3-4 seconds to run ls / gci in my homedir at home?). Text in columns gives you the simplicity of text coupled with some level of structure.
They haven't learned everything, because the other part is integration with all sorts of native libraries and IPC.
3 to 4 s to run ls?
U should fix ur computer rather then spreading FUD.
Also its clearly stated that copy of pwsh is intention on both projects.
Nushell is still one man show just like crush so they need decade or so to get to pwsh level.
Its nice to make stuff and offer alternatives but lets keep perspective here.
U should fix ur computer rather then spreading FUD.
Also its clearly stated that copy of pwsh is intention on both projects.
Nushell is still one man show just like crush so they need decade or so to get to pwsh level.
Its nice to make stuff and offer alternatives but lets keep perspective here.
Column orientation is very different to object orientation.
My computer runs just fine. Do you really think pwsh is fast?
My computer runs just fine. Do you really think pwsh is fast?
No. Its acceeptable, and mostly startup related. Certainly not even close to the level u claim. Once started it is more or less OK depending on specific cmdlet and task in question. I use it in production on numerious gov level services serving millions.
Vim is slower for example but rarelly anybody compalain about it given its power.
Vim is slower for example but rarelly anybody compalain about it given its power.
The author, Axel Liljencrantz, was also the initial developer of fish, so has form in shell development.
Had a quick look: Fish is in C++, Crush is in Rust.
I also feel Crush addresses some issues I had with Fish. Still on Zsh though (I like pastable scripts).
I also feel Crush addresses some issues I had with Fish. Still on Zsh though (I like pastable scripts).
I was curious how crush compared to fish (having glanced cursorily once or twice at the latter). Want to expand on the "issues you had with fish"?
Upon further thinking, I think the fact that running and integrating non-built-ins is so clunky makes this less a shell and more just a shell-like programming language, and at that point why not python?
Like it or not, the Unix pipe model of command integration is what enables programs to "know" about each other without ever really knowing about each other. Crush looks like its foregoing that.
Like it or not, the Unix pipe model of command integration is what enables programs to "know" about each other without ever really knowing about each other. Crush looks like its foregoing that.
The author was very clear that they're focusing on the programming language part first and the running non-built-ins part is just a proof of concept to be fleshed out later.
Has anyone got this to compile on macOS?
Nope, error: aborting due to 3 previous errors; 31 warnings emitted
Yes, not working for me too. Ran the
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
command also which upgraded by Rustc to rustc 1.45.2 (d3fb005a3 2020-07-31).
This is one problem I have faced with Rust quite frequently. Try to clone and run a random (not famous) project from Github and it doesn't compile.
Love Java when it comes to this. It's easy to see which version of the JDK is needed from the Maven file. If you have that version of JDK, the projects almost always run.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
command also which upgraded by Rustc to rustc 1.45.2 (d3fb005a3 2020-07-31).
This is one problem I have faced with Rust quite frequently. Try to clone and run a random (not famous) project from Github and it doesn't compile.
Love Java when it comes to this. It's easy to see which version of the JDK is needed from the Maven file. If you have that version of JDK, the projects almost always run.
While I do like the new wave of the structured shells(elvish, uxy, nushell, crush, etc...), I don't think anything will actually succeed unless there's a big change in the operating system.
(I'm not suggesting these aren't valuable - these are valuable experiments, but I'm saying for them to succeed, a new OS is needed.)
The native OS doesn't really embrace structured communication, and so commands will still have to treat the structured shell as a second-level citizen.
In an ideal world, for the structured shell to work, 'npm list' should return a table (or whatever the native data structure is) and you should be able to do 'npm list | where {deduped==false}'. In the real world, 'npm list' won't support the structured shell, so you would be writing 'npm list --json | jq '<obscure commands>' | json:from | where {deduped==false}'. And then 'npm list | grep -v 'deduped' is more simpler, so you don't really feel the advantages of the structured shell.
I think for this to be solved, the OS's primary communication system should be structured, so you can guarantee that every command will be a table. Then the programmers for the tools of that OS will output a table, and the users will be able to take advantage of this. Until then... it's basically a fun experiment.
(I'm not suggesting these aren't valuable - these are valuable experiments, but I'm saying for them to succeed, a new OS is needed.)
The native OS doesn't really embrace structured communication, and so commands will still have to treat the structured shell as a second-level citizen.
In an ideal world, for the structured shell to work, 'npm list' should return a table (or whatever the native data structure is) and you should be able to do 'npm list | where {deduped==false}'. In the real world, 'npm list' won't support the structured shell, so you would be writing 'npm list --json | jq '<obscure commands>' | json:from | where {deduped==false}'. And then 'npm list | grep -v 'deduped' is more simpler, so you don't really feel the advantages of the structured shell.
I think for this to be solved, the OS's primary communication system should be structured, so you can guarantee that every command will be a table. Then the programmers for the tools of that OS will output a table, and the users will be able to take advantage of this. Until then... it's basically a fun experiment.
> I think for this to be solved, the OS's primary communication system should be structured
You mean the STDOUT format of all processes? I disagree. `npm list --json` already does what you want. Your only argument is that ` | jq '<obscure commands>' | json:from | where {deduped==false}` is too obscure. The solution is (probably) a simplified `jq` that has the query format you want. Problem solved. Keep it simple.
You mean the STDOUT format of all processes? I disagree. `npm list --json` already does what you want. Your only argument is that ` | jq '<obscure commands>' | json:from | where {deduped==false}` is too obscure. The solution is (probably) a simplified `jq` that has the query format you want. Problem solved. Keep it simple.
I kinda like the mystical feeling of jq.
As with haskell, it'll feel as though I've arrived somehow, should I achieve real understanding.
As with haskell, it'll feel as though I've arrived somehow, should I achieve real understanding.
To me it's a good example of Opportunity Cost. We could spend the time and effort memorising jq's special snowflake language, but that time and effort would almost certainly be better spent on something else. I deleted it a while back. Can't be bothered with it.
>jq's special snowflake language
If you'll bother condescending to such, you'll find one of the most interesting languages I've encountered in years. jq's syntax and execution model is beautiful. It has some rough edges, leave no doubt. Some function names, date handling, and the obtuseness of some of the errors it returns can be frustrating. But its data streaming layers of nested closures are an extremely terse, readable, and powerful way of expressing operations over structured data.
If you'll bother condescending to such, you'll find one of the most interesting languages I've encountered in years. jq's syntax and execution model is beautiful. It has some rough edges, leave no doubt. Some function names, date handling, and the obtuseness of some of the errors it returns can be frustrating. But its data streaming layers of nested closures are an extremely terse, readable, and powerful way of expressing operations over structured data.
There are other tools like jq that instead use more common syntax. For example, jello[0] is just like jq except it uses python list and dictionary syntax. (disclaimer: I am the author)
[0] https://github.com/kellyjonbrazil/jello
[0] https://github.com/kellyjonbrazil/jello
I usually use jq just to format the blob, and then some unholy combination of grep -A/-B, tr, and sed to get the output I want. It’s usually faster (for me) than relearning jq’s query language.
> would almost certainly be better spent on something else.
I'd would suggest to spend that time on learning how to effectively learn new languages. There's more than one way, and I guarantee that afterwards you'll conquer any given "snowflake" in no time at all.
I'd would suggest to spend that time on learning how to effectively learn new languages. There's more than one way, and I guarantee that afterwards you'll conquer any given "snowflake" in no time at all.
I deliberately wrote "memorise" rather than learn in the usual sense, because there's a higher than normal degree of familiarity needed to get working code flowing from the fingers, when the venue is between a pair of single quotes in a shell.
[deleted]
This is a very tempting feeling, but when I see someone else who feels this way about something that I do understand, I can see that it doesn't serve them. In a tool that you use every day, you should be able to quickly and confidently demystify anything, or you're not really in command of the tool. Your temporary uncertainty should "bottom out" on something you don't have to think about further.
There are plenty of mysteries "out there" in the problem domain; you don't need them in your tools.
There are plenty of mysteries "out there" in the problem domain; you don't need them in your tools.
The problem, as always, is that a table isn't the natural output for a computer. It's either serialized text, or a graph. Unix tools work ok-ish for serialized text and you need general recursive functions to deal with graphs. Something that all shell like languages are extremely bad at.
It doesn't have to be this black and white, in my opinion. The most common data structures I've interacted with on the command line are _newline separated_ or _JSON_. What if your shell allowed parsing of JSON natively?
npm list --json | ({ json }) => json.somePackage.author.name
What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command?
curl www.example.com | ({ xml }) => xml.find('.importantInformation') > scraped_info.txt
npm list --json | ({ json }) => json.somePackage.author.name
What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command?
curl www.example.com | ({ xml }) => xml.find('.importantInformation') > scraped_info.txt
Yeah this is probably it right here. Just being able to parse various common formats would give a lot of the structure people are looking for. The builtins can just operate on that. Maybe it could optionally detect the formats too. If such a shell became popular, it would also incentivize developers adding common formats as outputs to more programs too.
This is what jc[0] does.
[0] https://github.com/kellyjonbrazil/jc
(disclaimer: I am the author)
[0] https://github.com/kellyjonbrazil/jc
(disclaimer: I am the author)
jq makes handling json in existing bash scripts simple. It can even do some simple data wrangling via built-in functions like map. It's not a universally standard command yet, but it's available for most modern systems.
I'm not sure about similar xml cli tools, but they probably exist.
Point is, both of these should be cli tools, not shell built-ins.
I'm not sure about similar xml cli tools, but they probably exist.
Point is, both of these should be cli tools, not shell built-ins.
> I'm not sure about similar xml cli tools, but they probably exist.
I :heart: xmlstarlet (sometimes installed as that name, sometimes as just "xml"): https://formulae.brew.sh/formula/xmlstarlet#default => https://xmlstar.sourceforge.io/
The xsltproc that ships with libxslt1 is also pretty handy, although more verbose
While not quite xml, pup is super handy for doing quick html selections: https://formulae.brew.sh/formula/pup#default => https://github.com/EricChiang/pup#readme
I :heart: xmlstarlet (sometimes installed as that name, sometimes as just "xml"): https://formulae.brew.sh/formula/xmlstarlet#default => https://xmlstar.sourceforge.io/
The xsltproc that ships with libxslt1 is also pretty handy, although more verbose
While not quite xml, pup is super handy for doing quick html selections: https://formulae.brew.sh/formula/pup#default => https://github.com/EricChiang/pup#readme
> I'm not sure about similar xml cli tools, but they probably exist.
xsltproc of course
xsltproc of course
Ooh that would be pretty damn sweet tbh.
Newline separated what? Basically all Linux config files are newline separated lines, but that won’t get you the equals separated data pairs, indented blocks, pretend-xml sections, ad-hoc boolean representations, INI-style sections, ad-hoc list-of-values style data out of them.
> What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command?
Powershell’s Invoke-RestMethod parses xml natively to a structured output; it’s useful but not enough to make XML fun or trivial to work with beyond root.node.foo.title simple scraping.
> What if it allowed parsing of XML natively, allowing instant scrapers to be written in a single command?
Powershell’s Invoke-RestMethod parses xml natively to a structured output; it’s useful but not enough to make XML fun or trivial to work with beyond root.node.foo.title simple scraping.
> Newline separated what?
What I meant by this is that the newline character is the primary delimiter of the data, but I take your point. IMO, the best way to tackle the myriad ways of structuring data is to write parsers for them, not to rewrite the programs themselves to work with your shell's idea of structured data. That's a losing battle.
What I meant by this is that the newline character is the primary delimiter of the data, but I take your point. IMO, the best way to tackle the myriad ways of structuring data is to write parsers for them, not to rewrite the programs themselves to work with your shell's idea of structured data. That's a losing battle.
Interesting note: newlines are perfectly valid characters in file names. In fact, the only byte that cannot be in a file name is '/'[0], which means that you're asking for trouble trying to parse the in-band signaling you have to do with a stream that's pretending to be structured.
[0] with a hex editor it is possible to create a file system entry using '/' in its name. Linux does not handle this situation with grace.
[0] with a hex editor it is possible to create a file system entry using '/' in its name. Linux does not handle this situation with grace.
This is how I felt powershell ought to go on Linux - parsers from every config format to a common intermediate and back, argument completers for every common shell command to a standard intermediate representation.
I think that’s a losing battle as well due to the brittle nature, any tweak to any command output could break something, meaning endless maintenance work on a large scale.
Rewriting concepts into new commands at least bring consistency and a chance of shrinking maintenance work as it settles on a nice design - and if not the maintenance won’t be endless rebuilds of the same parsers.
I think that’s a losing battle as well due to the brittle nature, any tweak to any command output could break something, meaning endless maintenance work on a large scale.
Rewriting concepts into new commands at least bring consistency and a chance of shrinking maintenance work as it settles on a nice design - and if not the maintenance won’t be endless rebuilds of the same parsers.
jc[0] does this for many commands and file-types, including .ini and key/value pair files with various delimiters and comment identifiers.
[0] https://github.com/kellyjonbrazil/jc
(disclaimer: I am the author)
[0] https://github.com/kellyjonbrazil/jc
(disclaimer: I am the author)
> The native OS doesn't really embrace structured communication
OS system calls are highly structured, down to the argument position and data type.
OS system calls are highly structured, down to the argument position and data type.
Right, but the unix philosophy is that everything consumes and emits text.
Having something more than a blob is great.
Of course, there will be a performance tax.
Having something more than a blob is great.
Of course, there will be a performance tax.
What you want is tagged and/or typed pipes. Once a pipe declares how its data is encoded, you can build all kinds of modern tooling and high-level automation on top of that.
The whole “But everything’s a text file!” is an outrageous lie, and always has been: just an excuse to cover up a massive omission in the design of the Unix file system (untyped, untagged “file” resources). Such egregious corner-cutting might have been understandable—even forgivable—back in K&R’s day, when they were bootstrapping C and Unix on hardware with less processing power than a modern microwave oven controller. But we’re decades on from that now and there’s no excuse for perpetuating it: we know how to build safe powerful modern static and dynamic type systems, and we’re spoiled rotten for the hardware resources to support it.
Fifty years ago, K&R built C and Unix, in a cave, with scraps. Nowadays you can bootstrap a complete hardware stack for less than the cost of a 360, and there are modern kernels (L4) and systems language (Rust) already available for free off the shelf. So pick a target market and build the userland for that. All that’s missing is a modern-day Kernighan and Ritchie to step up and get on with it, so it’d be a sorry indictment of this generation if they do not exist.
The whole “But everything’s a text file!” is an outrageous lie, and always has been: just an excuse to cover up a massive omission in the design of the Unix file system (untyped, untagged “file” resources). Such egregious corner-cutting might have been understandable—even forgivable—back in K&R’s day, when they were bootstrapping C and Unix on hardware with less processing power than a modern microwave oven controller. But we’re decades on from that now and there’s no excuse for perpetuating it: we know how to build safe powerful modern static and dynamic type systems, and we’re spoiled rotten for the hardware resources to support it.
Fifty years ago, K&R built C and Unix, in a cave, with scraps. Nowadays you can bootstrap a complete hardware stack for less than the cost of a 360, and there are modern kernels (L4) and systems language (Rust) already available for free off the shelf. So pick a target market and build the userland for that. All that’s missing is a modern-day Kernighan and Ritchie to step up and get on with it, so it’d be a sorry indictment of this generation if they do not exist.
> untyped, untagged “file” resources
I think the problem then, and the problem today, is that these “types” are a very very mobile target. How many implementations of JSON are there? XML? HTML? CSV?
It doesn’t make sense to put these types in the operating system, or else you’ll start TypeError’s every time you download anything. Not to mention the Computational costs of parsing the entire structure of a document every time you edit it to verify that it’s “still of type X”.
I’m not at all sold on the idea that categorizing files and streams in this way is a step forward (in the shell, at least). This is what applications are for: to define their own formats, parsers, and generators. The point of the operating system is not to be one of these applications, but to facilitate their development and use. IMO, typed files and streams would actually impede more than enable this work.
I think the problem then, and the problem today, is that these “types” are a very very mobile target. How many implementations of JSON are there? XML? HTML? CSV?
It doesn’t make sense to put these types in the operating system, or else you’ll start TypeError’s every time you download anything. Not to mention the Computational costs of parsing the entire structure of a document every time you edit it to verify that it’s “still of type X”.
I’m not at all sold on the idea that categorizing files and streams in this way is a step forward (in the shell, at least). This is what applications are for: to define their own formats, parsers, and generators. The point of the operating system is not to be one of these applications, but to facilitate their development and use. IMO, typed files and streams would actually impede more than enable this work.
“It doesn’t make sense to put these types in the operating system”
The OS doesn’t need to hardcode individual types, and to do so would defeat growth. It only has to provide a standard mechanism by which files can declare the type of data they contain, and executables can declare the type(s) of data they consume and/or produce. BeOS, for instance, tagged files with MIME types. The original Mac OS used “four-char codes”. Even DOS had file name extensions, as awful as that particular design choice was.
Storing arbitrary data with no way to indicate how that data is encoded is rank insanity. Saying “it’s the app’s job to deal with encodings” is not wrong, but if apps have no way to know what encoding a given file was written in, how can they know how to decode it?
Please do not rationalize away the longstanding and well-known deficiencies of Unix. These deficiencies do not aid progress; they hinder it. As do apologetics.
The OS doesn’t need to hardcode individual types, and to do so would defeat growth. It only has to provide a standard mechanism by which files can declare the type of data they contain, and executables can declare the type(s) of data they consume and/or produce. BeOS, for instance, tagged files with MIME types. The original Mac OS used “four-char codes”. Even DOS had file name extensions, as awful as that particular design choice was.
Storing arbitrary data with no way to indicate how that data is encoded is rank insanity. Saying “it’s the app’s job to deal with encodings” is not wrong, but if apps have no way to know what encoding a given file was written in, how can they know how to decode it?
Please do not rationalize away the longstanding and well-known deficiencies of Unix. These deficiencies do not aid progress; they hinder it. As do apologetics.
> files can declare the type of data they contain [...] executables can declare the type(s) of data they consume and/or produce.
I think you're conceding that the OS would not be responsible for enforcing data's compliance with it's declared type.
I'd be OK with this world, but I am not convinced that this hypothetical OS is substantially better than the UNIX descendants we have today.
> Storing arbitrary data with no way to indicate how that data is encoded is rank insanity.
I think we all can agree on this... it's just a matter of how we encode the data type. UNIX descendants follow the pattern of encoding data types with filename endings. While I agree that this approach feels primitive, it's hard to argue that it doesn't work, because it does. This is fundamentally identical to MIME-types, imo, because I believe extensions (and mime-types!) are fundamentally "names", and don't have much semantic above that, given the diversity of "name"-implementations, as stated earlier.
I don't believe that we gain a lot of practical value from compelling executables to register their accepted "names" with the Operating System: I'm trying to imagine what big value this gives you, beyond transforming `some_program --out_type=json | jq` to `some_program | jq`.
I hope you understand I'm not trying to apologize for UNIX. I agree that many things about it should be better. But I do think they got "everything is plaintext, from the OS's point of view" right.
(I think it's good discussion and I want to learn from it)
I think you're conceding that the OS would not be responsible for enforcing data's compliance with it's declared type.
I'd be OK with this world, but I am not convinced that this hypothetical OS is substantially better than the UNIX descendants we have today.
> Storing arbitrary data with no way to indicate how that data is encoded is rank insanity.
I think we all can agree on this... it's just a matter of how we encode the data type. UNIX descendants follow the pattern of encoding data types with filename endings. While I agree that this approach feels primitive, it's hard to argue that it doesn't work, because it does. This is fundamentally identical to MIME-types, imo, because I believe extensions (and mime-types!) are fundamentally "names", and don't have much semantic above that, given the diversity of "name"-implementations, as stated earlier.
I don't believe that we gain a lot of practical value from compelling executables to register their accepted "names" with the Operating System: I'm trying to imagine what big value this gives you, beyond transforming `some_program --out_type=json | jq` to `some_program | jq`.
I hope you understand I'm not trying to apologize for UNIX. I agree that many things about it should be better. But I do think they got "everything is plaintext, from the OS's point of view" right.
(I think it's good discussion and I want to learn from it)
“UNIX descendants follow the pattern of encoding data types with filename endings.”
An ad-hoc arrangement with little formal support at the GUI level and none at all at the shell level.
`some_program --out_type=json | jq` to `some_program | jq`.
JSON only tells you how the data is encoded, not what it means. Contrast this MIME type:
application/atom+json
Knowing data is JSON-encoded only enables (unsafe) generic JSON operations. Knowing the data is, say, ATOM allows task-specific operations. Consider a search tool. Having precise type information enables format-specific parsing operations to be decoupled from generalized tree/table searching operations.
Not only that, but looking up parsers and inserting them into the pipeline can be completely automated away. `some_program` doesn’t need to be told to output JSON; it can negotiate that itself with the program that follows, the shell acting as broker. (Content negotiation is one of the Great Ideas of HTTP; sadly utterly ballsed by the WWW, but that doesn’t mean others couldn’t do it right.)
Or consider `ls`. How many options/arguments does that have? (If you don’t know off the top of your head, I totally understand and am happy to wait.) Now, how many out of those options relate to recursion, filtering, and presentation? Because none of those should be built-ins; they should all be separate, general, composable tools. How much duplication of effort would be eliminated across the entire Unix command line? How much easier would it be to learn the entire system?
This is Unix Philosophy 101, yet even a simple standard tool like `ls` utterly fails it. Why? Because for effective, efficient, safe composition to be the easiest route, it needs to Just Work. Which it can’t do, because lack of IO typing and tagging means that What You See Is All You Can Get. Consider: for some consumers it’d make more sense for `ls` to output a list of inodes; for others, a list of full paths; for others, URLs. Outputting a list of file names is crap (assumes shared global state—cwd—that will not change between time of creation and time of use). Outputting a list of file names arranged in multiple vertical visual “columns” each offset by N spaces is insane. This is the very definition of Big Ball Of Mud.
Like I’ve said, there is no point trying to “fix” Unix because fundamentally it does not want to be fixed; never mind whether or not fixing it is even technically feasible. But that’s fine: there’s value in stability and it is a sunk cost so nothing is lost. Just leave Old Unix to be itself, which is what it’s good at, and build a new system unencumbered by legacy baggage and baked-in mistakes, making full use of all the wisdom and resources gained over the decades since.
Okay, so a brand new system will have to be radically better than the old in order to attract audience, but that’s totally doable. Heck, there were already way better CLIs forty years ago (VMS, Plan9, Symbolics Lisp, Smalltalk), so it’s not like there isn’t a wealth of knowledge and experience to purloin for free. It’s just a question of motive: who now benefits from propping up all the old complexity and makework vs who’ll benefit by completely disrupting it. And you just have to look at how an aggressive reborn Apple handed the mighty untouchable Microsoft its own ass back in the 2000s to work out the answer to that. Or Kodak. Or Xerox. Or the US car industry. Or the British Empire. And so on.
Qui audet adipiscitur. She who dares, wins.
An ad-hoc arrangement with little formal support at the GUI level and none at all at the shell level.
`some_program --out_type=json | jq` to `some_program | jq`.
JSON only tells you how the data is encoded, not what it means. Contrast this MIME type:
application/atom+json
Knowing data is JSON-encoded only enables (unsafe) generic JSON operations. Knowing the data is, say, ATOM allows task-specific operations. Consider a search tool. Having precise type information enables format-specific parsing operations to be decoupled from generalized tree/table searching operations.
Not only that, but looking up parsers and inserting them into the pipeline can be completely automated away. `some_program` doesn’t need to be told to output JSON; it can negotiate that itself with the program that follows, the shell acting as broker. (Content negotiation is one of the Great Ideas of HTTP; sadly utterly ballsed by the WWW, but that doesn’t mean others couldn’t do it right.)
Or consider `ls`. How many options/arguments does that have? (If you don’t know off the top of your head, I totally understand and am happy to wait.) Now, how many out of those options relate to recursion, filtering, and presentation? Because none of those should be built-ins; they should all be separate, general, composable tools. How much duplication of effort would be eliminated across the entire Unix command line? How much easier would it be to learn the entire system?
This is Unix Philosophy 101, yet even a simple standard tool like `ls` utterly fails it. Why? Because for effective, efficient, safe composition to be the easiest route, it needs to Just Work. Which it can’t do, because lack of IO typing and tagging means that What You See Is All You Can Get. Consider: for some consumers it’d make more sense for `ls` to output a list of inodes; for others, a list of full paths; for others, URLs. Outputting a list of file names is crap (assumes shared global state—cwd—that will not change between time of creation and time of use). Outputting a list of file names arranged in multiple vertical visual “columns” each offset by N spaces is insane. This is the very definition of Big Ball Of Mud.
Like I’ve said, there is no point trying to “fix” Unix because fundamentally it does not want to be fixed; never mind whether or not fixing it is even technically feasible. But that’s fine: there’s value in stability and it is a sunk cost so nothing is lost. Just leave Old Unix to be itself, which is what it’s good at, and build a new system unencumbered by legacy baggage and baked-in mistakes, making full use of all the wisdom and resources gained over the decades since.
Okay, so a brand new system will have to be radically better than the old in order to attract audience, but that’s totally doable. Heck, there were already way better CLIs forty years ago (VMS, Plan9, Symbolics Lisp, Smalltalk), so it’s not like there isn’t a wealth of knowledge and experience to purloin for free. It’s just a question of motive: who now benefits from propping up all the old complexity and makework vs who’ll benefit by completely disrupting it. And you just have to look at how an aggressive reborn Apple handed the mighty untouchable Microsoft its own ass back in the 2000s to work out the answer to that. Or Kodak. Or Xerox. Or the US car industry. Or the British Empire. And so on.
Qui audet adipiscitur. She who dares, wins.
Incidentally, your jq example is a perfect illustration of Zawinski’s Law† in action:
“jq 1.5 released, including new datetime, math, and regexp functions, try/catch syntax, array and object destructuring, a streaming parser, and a module system”
These big bloated behavioral silos that keep popping up on Unix are neither good nor accidental: they’re the symptom of a fundamental fault in the underlying host. Because when the most complex concept it is capable of communicating on clients’ behalf is “Ugg bang rock!”, it should come as no surprise when its clients all go “fuckit, I’ll just do it all myself”. And so it goes.
--
† http://www.catb.org/jargon/html/Z/Zawinskis-Law.html
“jq 1.5 released, including new datetime, math, and regexp functions, try/catch syntax, array and object destructuring, a streaming parser, and a module system”
These big bloated behavioral silos that keep popping up on Unix are neither good nor accidental: they’re the symptom of a fundamental fault in the underlying host. Because when the most complex concept it is capable of communicating on clients’ behalf is “Ugg bang rock!”, it should come as no surprise when its clients all go “fuckit, I’ll just do it all myself”. And so it goes.
--
† http://www.catb.org/jargon/html/Z/Zawinskis-Law.html
I don’t think you need an OS to pass structured objects, but rather simply a runtime that supports it, a la dotnet. Basically anything that can allow different programs to run and share memory as well as information about the shape of the objects in that shared memory. This can run on existing OSes and in the case of powershell, it already does. I think Go could probably do this as well via its plugin interface and runtime type system except that you aren’t allowed to unload plugins (so every program you ever run hangs out in memory).
Microsoft went far down this road in the early 2000s with WinFS [1] but abandoned it. [Edit: WinFS and Longhorn together]
[1] https://en.wikipedia.org/wiki/WinFS
[1] https://en.wikipedia.org/wiki/WinFS
I keep having the same thoughts, but I'm not sure it needs to be a full OS solution.
Also a related problem that always frustrates me with input is what do I use for the help command? Is it '-h', '-help', '--help', or just 'help', or even man pages only. Each script/bin has its own way of doing things and this seems like something that should be standardized, but I realize there's no great way to do that currently when its just input to the program.
I wonder if instead of an OS solution we can just add more metadata to the actual binary files we compile. For example, if we have a new ELF format that has an optional list of commands, options, full input/output types, etc, any one of the shells can start to provide better tooling. Output format types can then be passed along with the stream, and potentially checked that they work as the proper inputs for another command. Let the shell auto generate the simple cli help docs, or generate the bash completions, or figure out how to better convert data to desired types.
I wonder if instead of an OS solution we can just add more metadata to the actual binary files we compile. For example, if we have a new ELF format that has an optional list of commands, options, full input/output types, etc, any one of the shells can start to provide better tooling. Output format types can then be passed along with the stream, and potentially checked that they work as the proper inputs for another command. Let the shell auto generate the simple cli help docs, or generate the bash completions, or figure out how to better convert data to desired types.
Something relatively simple would be borrowing the idea of the HTTP "accepts" header, and standardizing on an environment variable. For example:
The other trick is knowing what an application will accept via STDIN - ideally at runtime, but before actually writing anything.
If the shell understands all this, it could automatically format JSON/csv output for display, integrate jq for formatting, or even convert between formats (eg: when one app writes in yaml, the other understands JSON input).
STDOUT_ACCEPT=application/json,application/csv,text/plain
The other trick is knowing what an application will accept via STDIN - ideally at runtime, but before actually writing anything.
If the shell understands all this, it could automatically format JSON/csv output for display, integrate jq for formatting, or even convert between formats (eg: when one app writes in yaml, the other understands JSON input).
Speaking of fancy alternative shells, I tried http://xon.sh/ for a while, whose selling point is that its language is a superset of Python.
While it was neat for interactive use to have stuff like proper string manipulation and list comprehensions, writing scripts with it (.xsh files) was horrible, because the sorta-but-not-really-Python-ness meant that no tooling worked. Syntax highlighting was messed up, code autoformatters barfed, linters barfed, autocompletion didn't work.
I actually find writing scripts in Bash, aided by the amazing linter https://www.shellcheck.net/ to feel almost like it's now a compiled language. It goes from being a fraught endeavor, to being kinda fun and educational. It's like pair programming with someone who has an encyclopedic knowledge of the language quirks. Hook it up to automatically run on-save in your editor!
While it was neat for interactive use to have stuff like proper string manipulation and list comprehensions, writing scripts with it (.xsh files) was horrible, because the sorta-but-not-really-Python-ness meant that no tooling worked. Syntax highlighting was messed up, code autoformatters barfed, linters barfed, autocompletion didn't work.
I actually find writing scripts in Bash, aided by the amazing linter https://www.shellcheck.net/ to feel almost like it's now a compiled language. It goes from being a fraught endeavor, to being kinda fun and educational. It's like pair programming with someone who has an encyclopedic knowledge of the language quirks. Hook it up to automatically run on-save in your editor!
Take a look at marcel: https://marceltheshell.org. It is also a shell that pipes objects, and is based on Python, but it sounds like scripting will be more to your taste. There is a marcel.api module that allows you to do shell-like commands neatly integrated into python code, e.g.
from marcel.api import *
for file, size in (ls('marcel', 'test',
file=True, recursive=True) |
map(lambda f: (f.path, f.size))):
print(f'{file}: {size}')I check in on Xonsh every once in a while. The first test I do is try and suspend a sleep into the background. This has never worked on xonsh afaict. I am excited to give Xonsh a spin, but I can't until it gets its job control story sorted out.
They are unfortunately not implementing posix correctly. Instead of using proper processes they use threads to control certain things which makes <ctrl-z> not work properly.
This would actually be possible to fix, but it's quite a bit of work.
This would actually be possible to fix, but it's quite a bit of work.
Oil shell should be mentioned here, an alternative approach at a modern shell. It tries to be a (more sane?) upgrade path to bash: https://www.oilshell.org/
Thanks for the mention! That's an accurate way of describing it. I updated the home page with the one page description: Oil is our upgrade path from bash to a better language and runtime.
The ability to run existing POSIX and bash scripts is the biggest difference between Oil and other new shells (something I'm often asked). Unfortunately that's also why it's such a long implementation effort. But in reading the comments here, I still think that was the right choice.
----
FWIW Oil's approach to structured data is to use interchange formats over pipes. You can already pipe JSON, HTML, and CSV/TSV over pipes, and people do. There are many command line toolkits that do this.
https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil
However the shell is also missing few things which would help that approach, so I'd like to build some native understanding of those formats into it, without requiring any special "plugin interface". Plugins are simply processes in Oil.
For example, Oil already has a QSN "Quoted String Notation" to properly escape byte strings like filenames (this happens to be the format of Rust string literals, which are similar to C and JSON string literals). So you don't have to do parsing and splitting.
The approach to tables will likely be built on QSN in TSV or QSV. And then you can enhance "npm list" and "dpkg" just to print that to stdout, without having to integrate into any special "plugin system". It just works with normal Unix processes and the kernel.
https://github.com/oilshell/oil/wiki/TSV2-Proposal (not implemented)
The ability to run existing POSIX and bash scripts is the biggest difference between Oil and other new shells (something I'm often asked). Unfortunately that's also why it's such a long implementation effort. But in reading the comments here, I still think that was the right choice.
----
FWIW Oil's approach to structured data is to use interchange formats over pipes. You can already pipe JSON, HTML, and CSV/TSV over pipes, and people do. There are many command line toolkits that do this.
https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil
However the shell is also missing few things which would help that approach, so I'd like to build some native understanding of those formats into it, without requiring any special "plugin interface". Plugins are simply processes in Oil.
For example, Oil already has a QSN "Quoted String Notation" to properly escape byte strings like filenames (this happens to be the format of Rust string literals, which are similar to C and JSON string literals). So you don't have to do parsing and splitting.
The approach to tables will likely be built on QSN in TSV or QSV. And then you can enhance "npm list" and "dpkg" just to print that to stdout, without having to integrate into any special "plugin system". It just works with normal Unix processes and the kernel.
https://github.com/oilshell/oil/wiki/TSV2-Proposal (not implemented)
Update: I just wrote a draft of "Why Use Oil?"
http://www.oilshell.org/why.html
Let me know what you think! (feedback link in the article)
http://www.oilshell.org/why.html
Let me know what you think! (feedback link in the article)
Your shell is not for me. Here's why.
1) I don't want shell to be full-blown high level programming language. POSIX, Bash scripts/languages are enough for my shell uses.
2) For anything high level or for tasks shell would become complex, We do have dozens of fully matured languages. I would rather use those instead of this/any shell script.
3) Features desirable for shell include syntax highlighting, autosuggestions,command-not-found helper,tab autocomplete, flag complete. My shell already has it.
4) I like bash and zsh as shell. Bash is simple,i like it,but it doesn't have few features that i mentioned. So,I don't use bash as my main shell. zsh also is compatible with most bash/POSIX and it has all tte features that i mentioned in (3).fish also has these features but it's syntax is different which i don't tend to use. So,i don't use fish. zsh has all features that i like in fish too(mentioned in (3)) .
1) I don't want shell to be full-blown high level programming language. POSIX, Bash scripts/languages are enough for my shell uses.
2) For anything high level or for tasks shell would become complex, We do have dozens of fully matured languages. I would rather use those instead of this/any shell script.
3) Features desirable for shell include syntax highlighting, autosuggestions,command-not-found helper,tab autocomplete, flag complete. My shell already has it.
4) I like bash and zsh as shell. Bash is simple,i like it,but it doesn't have few features that i mentioned. So,I don't use bash as my main shell. zsh also is compatible with most bash/POSIX and it has all tte features that i mentioned in (3).fish also has these features but it's syntax is different which i don't tend to use. So,i don't use fish. zsh has all features that i like in fish too(mentioned in (3)) .
> Your shell is not for me.
And that's okay. Personally, I'm thrilled by the idea of a new shell. I don't plan to use Crush in production (yet?!) but I definitely plan on installing it on my laptop.
I'm excited to see how things go. I understand, for a while, it won't have all the bells and whistles of zsh. But that's okay. Everything has to start somewhere. I remember back when alacritty didn't even have easy scrollback!
Best of luck to Crosh and its author, liljencrantz!
And that's okay. Personally, I'm thrilled by the idea of a new shell. I don't plan to use Crush in production (yet?!) but I definitely plan on installing it on my laptop.
I'm excited to see how things go. I understand, for a while, it won't have all the bells and whistles of zsh. But that's okay. Everything has to start somewhere. I remember back when alacritty didn't even have easy scrollback!
Best of luck to Crosh and its author, liljencrantz!
I'm curious if you've taken the time to learn Powershell?
It's a supercharged version of bash that works with Windows, but instead of piping character-only data, it can pipe structured data between commands in the form of objects. The reason I ask is because, while it adds a ton of flexibility to shell scripting (to the point where I consider it to be a language), it is also quite complex.
The benefit of shell scripting is that it is simple enough to have as a tertiary language that you can pickup and use after a six month hiatus. Once you get into more powerful language territory, there are a lot more components to memorize and it becomes difficult (for me at least) to keep those idioms in my head if they go unused for months.
It's a supercharged version of bash that works with Windows, but instead of piping character-only data, it can pipe structured data between commands in the form of objects. The reason I ask is because, while it adds a ton of flexibility to shell scripting (to the point where I consider it to be a language), it is also quite complex.
The benefit of shell scripting is that it is simple enough to have as a tertiary language that you can pickup and use after a six month hiatus. Once you get into more powerful language territory, there are a lot more components to memorize and it becomes difficult (for me at least) to keep those idioms in my head if they go unused for months.
> The benefit of shell scripting is that it is simple enough to have as a tertiary language that you can pickup and use after a six month hiatus.
I definitely did not find that to be the case. I've learned and relearned shell scripting a bunch of times. I always forget it and need to relearn it. Worse than perl.
I definitely did not find that to be the case. I've learned and relearned shell scripting a bunch of times. I always forget it and need to relearn it. Worse than perl.
> The benefit of shell scripting is that it is simple enough to have as a tertiary language that you can pickup and use after a six month hiatus.
I've been using/abusing traditional shell scripting languages for over a decade now and I still struggle with the rough edges (especially quoting/escaping/argument passing).
I've been using/abusing traditional shell scripting languages for over a decade now and I still struggle with the rough edges (especially quoting/escaping/argument passing).
Every time I learn more about Powershell I get more jealous of Windows users. Not so jealous I'd switch back, but man would I like some of those features on Linux. I know it's ported, but it just doesn't have the tooling on Linux and I'll be damned if my standard scripting language doesn't run natively.
For Clojure aficionados, there's a flavour of Clojure made for scripting and the command line: Babashka https://github.com/borkdude/babashka
And a proper shell, Closh: https://github.com/dundalek/closh
And a proper shell, Closh: https://github.com/dundalek/closh
I believe a good fit for Unix commands is using a pipeline where the output of your command (say, ls) is converted to JSON, so that later in the pipeline you can use jq to filter / count / etc in a structured, non-whitespace-dependent way. A combination of jc (https://github.com/kellyjonbrazil/jc) and jq fits the bill for me:
ls -l /usr/bin | jc --ls | jq '.[] | select(.size > 50000000)'I think this should honestly be higher. I'm not sure if jc is THE tool -- the Python dependency irks me slightly -- but I like the simplicity of the idea, wrapping around what is already available and functional, while improving my experience. I plan to play around with this idea.
jc is also available as a standalone binary with python 'packaged in'. https://kellyjonbrazil.github.io/jc-packaging/
Most commands that you would want to put into a pipeline already have some sort of machine parsable output flag (tabular, json, format specifier, etc). Still, there is not much reason to call ls from a script at all. In python you would be better off with the native libraries. In bash, you would probably be best off with `find /usr/bin -maxdepth 1 -size +50M` in your usecase.
IMO people have been trying to do the same thing with these tools since they were invented. Better to learn how to use them the way they were intended before reinventing them.
Edit: re-read my comment. Seems a little negative. Sorry. To be clear, I think the jc/jq approach is much better than the roll-your-own parser in shell approach.
IMO people have been trying to do the same thing with these tools since they were invented. Better to learn how to use them the way they were intended before reinventing them.
Edit: re-read my comment. Seems a little negative. Sorry. To be clear, I think the jc/jq approach is much better than the roll-your-own parser in shell approach.
One use case that has come up is to parse the output of commands your run on remote hosts that may not even have python installed. I have a PR into Ansible community.general plugins to add this functionality and it works pretty sweet. There are some other automation tools that are looking into this approach as well.
Nice idea that corresponds with my idée fixe - a sane, usable output format for Unix utilities. There are differences, of course - I prefer a text-table format as universal panacea, but I'm glad that more and more people notice a mess in Unix environment ("simple" doesn't mean "consistent" and even Plan9 suffers from that).
See also https://github.com/aniou/cof/wiki/Draft
See also https://github.com/aniou/cof/wiki/Draft
I'm struggling today as well with this shit:
I just wanna write a few small lines of a shell script which downloads stuff;
It is very very stupid through what loops you hvae to jump to do this with curl and bash script in a good way including readability and error handling.
Is a perl script really better?
Perl 5,6 and 7 did not make it very sane to me, managing and installing and working with perl is also less trivial then having a shell like bash, sh or dash available to you.
Something inbetween bash and go is missing, i like the approach of crush. I think thats the biggest misstake happening in bash: playing around with strings and auto expansion and substition etc. Its weird, its error prone, it doesn't have a proper clean well defined ecosysem.
I just wanna write a few small lines of a shell script which downloads stuff;
It is very very stupid through what loops you hvae to jump to do this with curl and bash script in a good way including readability and error handling.
Is a perl script really better?
Perl 5,6 and 7 did not make it very sane to me, managing and installing and working with perl is also less trivial then having a shell like bash, sh or dash available to you.
Something inbetween bash and go is missing, i like the approach of crush. I think thats the biggest misstake happening in bash: playing around with strings and auto expansion and substition etc. Its weird, its error prone, it doesn't have a proper clean well defined ecosysem.
If you're open to learning it, Tcl is actually a good middle ground here. The
best way I can describe it is "structured shell script" - still very string
oriented, but the syntax and command set is much more well thought out, and
its better for building libraries in (having namespacing, among other things).
You even have a choice among object systems if you need one.
You don't lose much in terms of shell interop though - the native exec command understands a lot of the basics (file redirection, pipelining, background execution). In the simplest case, it's just one more word:
You don't lose much in terms of shell interop though - the native exec command understands a lot of the basics (file redirection, pipelining, background execution). In the simplest case, it's just one more word:
exec wget $URL
# Dead
error "IO error"
# Not dead
if {catch [error "IO error"] error_or_result} {
# The error
puts "Caught: $error_or_result"
} else {
# The result
puts "OK: $error_or_result"
}Since there are lots of alternative shells being mentioned, I thought I'd give my tuppence. I want two things from a shell:
- Easy process execution, including branching on exit codes, overriding env vars, providing a working directory, etc. This must support concurrent processes (e.g. pipelines).
- Easy file and stdio management (i.e. piping). This must support streaming (passing along chunks of data as they arrive, rather than gathering them into big intermediate values).
Scripting language REPLs provide everything else I care about (string handling, looping, variables, etc.). Their built-in support for these two things is usually really bad, but libraries can plug that gap. I've tried a few for Python, Haskell, Scala, etc. but the best experience I had was Racket's shell-pipeline library ( https://docs.racket-lang.org/shell-pipeline it's also used by the Rash shell that I've not used https://docs.racket-lang.org/rash ). This makes really good use of Racket features, e.g.
- Quasiquoting for mixing commands with normal functions in a simple, sane way
- S-expression syntax works well for shell-like code, e.g. no comma-separation, distinguishing between (unquoted) symbols for names, flags, etc. and quoted strings for data, allowing special characters in names (e.g. to allow things like > in our pipeline), etc. Of course, Racket can be written with I-expressions, sweet-expressions, etc. too, but I've never bothered.
- Stdio redirection works with Racket's existing "ports" mechanism
- Env vars can be managed using Racket "parameters" (dynamically-scoped variables)
(Note that the API seems to have changed a bit since the version I used, so I'm not sure how the new stuff like laziness fits in)
- Easy process execution, including branching on exit codes, overriding env vars, providing a working directory, etc. This must support concurrent processes (e.g. pipelines).
- Easy file and stdio management (i.e. piping). This must support streaming (passing along chunks of data as they arrive, rather than gathering them into big intermediate values).
Scripting language REPLs provide everything else I care about (string handling, looping, variables, etc.). Their built-in support for these two things is usually really bad, but libraries can plug that gap. I've tried a few for Python, Haskell, Scala, etc. but the best experience I had was Racket's shell-pipeline library ( https://docs.racket-lang.org/shell-pipeline it's also used by the Rash shell that I've not used https://docs.racket-lang.org/rash ). This makes really good use of Racket features, e.g.
- Quasiquoting for mixing commands with normal functions in a simple, sane way
- S-expression syntax works well for shell-like code, e.g. no comma-separation, distinguishing between (unquoted) symbols for names, flags, etc. and quoted strings for data, allowing special characters in names (e.g. to allow things like > in our pipeline), etc. Of course, Racket can be written with I-expressions, sweet-expressions, etc. too, but I've never bothered.
- Stdio redirection works with Racket's existing "ports" mechanism
- Env vars can be managed using Racket "parameters" (dynamically-scoped variables)
(Note that the API seems to have changed a bit since the version I used, so I'm not sure how the new stuff like laziness fits in)
crush> ls | where {type == "directory"}
[deleted]
What is wrong with:
ls -l | grep '^d'It is minimal, so does not meet the modern aesthetics of computer world.
Until you want something more specific like listing absolute paths of those directories without the metadata
ls -l | grep '^d' | awk '{print $9}' | xargs realpath
Or
ls -ld "$PWD"/* | grep '^d' | awk '{print $9}'
Guessing Crush could just add another property called Fullpath you can echo in the next pipe after filtering.
ls | where {type == "directory"} | select %Fullpath
Maybe he adds a --dir flag
ls --dir | select %Fullpath
ls -l | grep '^d' | awk '{print $9}' | xargs realpath
Or
ls -ld "$PWD"/* | grep '^d' | awk '{print $9}'
Guessing Crush could just add another property called Fullpath you can echo in the next pipe after filtering.
ls | where {type == "directory"} | select %Fullpath
Maybe he adds a --dir flag
ls --dir | select %Fullpath
% find . -maxdepth 1 -type d | xargs realpath
How long do these tools exist? Over 20 years?
How long do these tools exist? Over 20 years?
The example works but you need to understand the idiosyncrasies of all the tools involved and the text format. Once you learn that you're passing around structured data and you access it the same way it becomes trivial get data.
Yes, you have to learn the idiosyncrasies of the tools involved. My point was, these are standard tools that are there for more than 20 years, and they work just fine. People should just RTFM and learn by experience instead of inventing something new which looks shiny on surface but most certainly will have unhandled hidden edge cases and serious limitations.
Outputs are not equivalent. But this will work:
find . -type dI've read most of the comments in this thread and will read TFA, but haven't yet.
What's always bothered me about piping to grep is losing the original syntax highlighting. Particularly annoying when the first command, whose highlighting I want to preserve, is itself grep; this leads to awkward contortions like
What's always bothered me about piping to grep is losing the original syntax highlighting. Particularly annoying when the first command, whose highlighting I want to preserve, is itself grep; this leads to awkward contortions like
git grep "a" | grep -v "b" | grep "a"We need SQshell, a shell that uses SQL for operations such as listing files, processes, all that stuff where a structured query language is useful. SQL join syntax is not that different from piping. In my opinion, SQL knowledge is timeless, most other IT knowledge seems to have a "best before" date, but SQL never expires.
You can do that kind of things already, albeit not from within a single universal shell:
- https://github.com/jhspetersson/fselect
- https://github.com/osquery/osquery
- https://github.com/jhspetersson/fselect
- https://github.com/osquery/osquery
The thing is with SQL is many devs seem to be in one of 3 states with it. Allergic to it, will not touch it. Enough to be dangerous, and make very bad decisions with it. Amazing at it and fixing poor decisions. I am not sure what it is about SQL but it really seems to rub people the wrong way. I think it is the order of the statements. I think if they had switched the from and select statement around people would 'get' it more.
SQL is a neat tool for what it is. But command line bashing does not seem like a good fit to me. I get what you are saying though. I just would be kind of miffed to have to do something like 'select * from folders(/foldera/folderb/folderc) where lastcreatedate > getutcdate('1970') order by lastcreatedate desc' when I can currently do something like 'ls -ltar'. It is not that I can't do that. But it takes a cognitive load to do it for a command I use a lot. Of course I could make a program to do it. But then I am kind of back where I started? I could get behind SQL in addition to what I have. But in replacement I think that would be a tough sell.
SQL is a neat tool for what it is. But command line bashing does not seem like a good fit to me. I get what you are saying though. I just would be kind of miffed to have to do something like 'select * from folders(/foldera/folderb/folderc) where lastcreatedate > getutcdate('1970') order by lastcreatedate desc' when I can currently do something like 'ls -ltar'. It is not that I can't do that. But it takes a cognitive load to do it for a command I use a lot. Of course I could make a program to do it. But then I am kind of back where I started? I could get behind SQL in addition to what I have. But in replacement I think that would be a tough sell.
I am definitely in the "enough to be dangerous" category, but isn't it technically possible to write SQL queries the way you describe? IE, the only reasons to write them in SELECT .. FROM order are (1) tradition / what others expect, and (2) optimizations made by the dbs that assume the traditional order? But in terms of returning correct results, FROM .. SELECT should work fine, I think.
Having not written a SQL parser. But I would assume there is a grammar parser in there that puts the items into some IR like language or structure that runs it. The big ones call it a query plan. So you in theory could flip them around as long as the query plan was the same. You would just need to basically convince MS and Oracle and all the other DB projects to do/try it. Not sure they would see the benefit though. In my head they are backwards but in their design it may be fine. Ideally the keywords could be in any order. But that would probably break some implicit ideas that those keywords convey (looking at you having and group by).
Hmm, I thought I remembered reading something to the effect that this was currently possible, but now trying to find more information about it I'm coming up empty.
With oracle if it exists would probably be a compiler directive. In MSSQL would be some sort of DBCC command. Not sure with the other DBs as I do not use them as much and have not needed to tweak them at that level.
Curious how this compares to Xonsh (IIUC, pretty similar idea, tho I Xonsh is actual Python, vs Crush using its own new lang).
Somebody just give me a rust REPL already (that is not the slow as snails evcxr)
The main innovation I see is tokenization into columns instead of character streams. If I want to do that in a traditional shell, I use awk.
Doesn't AWK already do all of this?
Looks like a unix-ish take on PowerShell.
I love the idea behind PowerShell, but the verbosity turns me off.
PowerShell, of course, has the pipelines but also COM integration, which is a big deal for Windows (you can script "the desktop" via COM; for instance you could create a new instance of Microsoft Word and call objects inside Word to do something to a document.)
Unix has dbus and some similar things, but nothing quite so critical as COM is to Windows.
I love the idea behind PowerShell, but the verbosity turns me off.
PowerShell, of course, has the pipelines but also COM integration, which is a big deal for Windows (you can script "the desktop" via COM; for instance you could create a new instance of Microsoft Word and call objects inside Word to do something to a document.)
Unix has dbus and some similar things, but nothing quite so critical as COM is to Windows.
The verbosity is one of PowerShell's best features. There is so much less guess work about what a command or argument does, or whether it is '-a' or '-A', or any of that garbage. You can usually guess how to do things because there's a reasonable consistency to the Verb-Noun structure. The extra typing is mitigated by everything being tab-complete including commands, arguments, and even variables. Not to mention you can just alias commonly used commands into shorter versions anyway!
Did I mention it also makes readings scripts at least 100 times easier?
Did I mention it also makes readings scripts at least 100 times easier?
Around the time powershell came out I got into the (bad) habit of writing super-concise bash scripts that write bash scripts, for instance:
ls *.txt | awk '{print "mv" $1 "/somewhere/"}' | bash
One could build a UNIX distribution that would feel exactly like those Xerox PARC experiences, on top of DBUs/gRPC and a couple of managed languages of choice, but it is a project doomed to failure thanks fragmentation.
Maybe the closest one gets of it are GNOME, XFCE and KDE based distributions.
Although broading the view a bit, macOS has XPC/Open Scripting Architecture and ChromeOS/Android/Fuschia have their Binder infrastructure.
Maybe the closest one gets of it are GNOME, XFCE and KDE based distributions.
Although broading the view a bit, macOS has XPC/Open Scripting Architecture and ChromeOS/Android/Fuschia have their Binder infrastructure.
I really want something like this as a replacement for sh in tiny busybox-backed use-cases, so I find it really annoying that developers insist I install rust and compile it instead of offering a statically compiled binary I can use.
And if it can't do that because of a reliance on shared libs, then that's even worse!
And if it can't do that because of a reliance on shared libs, then that's even worse!
The improved-shell-piping-objects space is getting crowded. My entry is marcel: https://marceltheshell.org.
Trying to create a combined shell/programming language is a mistake, in my opinion. Marcel exposes bits of python on the command line, e.g. for writing predicates. For more serious programming, marcel offers a Python API that allows the use of shell commands inside Python, neatly integrated into the language. For example you can write a series of commands, connected by pipes, and the result defines a Python iterator so that you can use the pipeline results in a for loop.
See the website for more details and discussions.
Trying to create a combined shell/programming language is a mistake, in my opinion. Marcel exposes bits of python on the command line, e.g. for writing predicates. For more serious programming, marcel offers a Python API that allows the use of shell commands inside Python, neatly integrated into the language. For example you can write a series of commands, connected by pipes, and the result defines a Python iterator so that you can use the pipeline results in a for loop.
See the website for more details and discussions.
Publishing a project at this level of immaturity doesn't do it any good. While it definitely "works", it works as an experiment about techniques to write an interactive command interpreter in Rust, not as a shell for people to use.
Currently there is no documentation beyond the introductory GitHub page. But this isn't a fork of sh that only needs documentation of the minor ways in which it departs from POSIX compliance: there is an unknown set of internal commands, each with a complex table format for its output (it is necessary to know all relevant column names to do anything), which depends on a large set of non-text types, which have complex object-oriented behaviour, with "serialization" to a variety of file formats, with an unspecified script and command line syntax.
Such complex features, on par with serious frameworks in full-fledged programming languages, need to be defined and documented very thoroughly, or nobody will understand, trust and use them. Unfortunately, there are many hints that not only documentation is vastly behind schedule, but that important aspects and practical considerations haven't been addressed.
For example: how can I quote file names containing nasty characters? How can I use environment variables? What is my home directory? Why is executing an executable program difficult, with complex and unclear mechanisms that mess with the command line? Given a very simple and very portable traditional shell script (e.g. ps -eaf | grep foo | sort) how can I do the same in Crush with minimal effort? If I load a very large file, does the shell choke attempting to load it all to memory, or does it process it in a proper streaming fashion?
Other issues are more fundamental. How can I perform relational queries between multiple tables, e.g. a join between ls output and a file listing users? If I can't, what can I do with tables that cannot be done more easily and flexibly with awk, jq, and other established tools?
Currently there is no documentation beyond the introductory GitHub page. But this isn't a fork of sh that only needs documentation of the minor ways in which it departs from POSIX compliance: there is an unknown set of internal commands, each with a complex table format for its output (it is necessary to know all relevant column names to do anything), which depends on a large set of non-text types, which have complex object-oriented behaviour, with "serialization" to a variety of file formats, with an unspecified script and command line syntax.
Such complex features, on par with serious frameworks in full-fledged programming languages, need to be defined and documented very thoroughly, or nobody will understand, trust and use them. Unfortunately, there are many hints that not only documentation is vastly behind schedule, but that important aspects and practical considerations haven't been addressed.
For example: how can I quote file names containing nasty characters? How can I use environment variables? What is my home directory? Why is executing an executable program difficult, with complex and unclear mechanisms that mess with the command line? Given a very simple and very portable traditional shell script (e.g. ps -eaf | grep foo | sort) how can I do the same in Crush with minimal effort? If I load a very large file, does the shell choke attempting to load it all to memory, or does it process it in a proper streaming fashion?
Other issues are more fundamental. How can I perform relational queries between multiple tables, e.g. a join between ls output and a file listing users? If I can't, what can I do with tables that cannot be done more easily and flexibly with awk, jq, and other established tools?
I have zero interest in this project, but disagree that it "doesn't do any good" to share. People with similar inclinations and interests can collaborate on software that isn't ready, but that scratches their itch.
Theres nothing wrong with that, as long as you don't create unrealistic expectations.
Theres nothing wrong with that, as long as you don't create unrealistic expectations.
The problem with novel shells, compared with other kinds of itch to scratch, is that existing shells have had about 50 years of evolution: doing better in a few months of work is quite unlikely.
A new shell must be similar to existing ones for compatibility with an enormous ecosystem of command line tools that work well in a typical shell; it must do almost everything that traditional shells do equally well (this includes documentation; documentation is non-negotiable) but it must also have some compelling advantage.
The right way to collaborate on designing a shell is to discuss possible designs of command syntax and behaviour, then if some promising idea comes out of the brainstorming refine it to a full specification, then if the idea survives (without mutating into a less ambitious scripting language, or a library for an existing language or shell, or a small change to an existing shell) begin an implementation. Starting with a proof of concept without a well developed and well documented design is the wrong way to offer a "product" but also the wrong way to learn.
A new shell must be similar to existing ones for compatibility with an enormous ecosystem of command line tools that work well in a typical shell; it must do almost everything that traditional shells do equally well (this includes documentation; documentation is non-negotiable) but it must also have some compelling advantage.
The right way to collaborate on designing a shell is to discuss possible designs of command syntax and behaviour, then if some promising idea comes out of the brainstorming refine it to a full specification, then if the idea survives (without mutating into a less ambitious scripting language, or a library for an existing language or shell, or a small change to an existing shell) begin an implementation. Starting with a proof of concept without a well developed and well documented design is the wrong way to offer a "product" but also the wrong way to learn.
You either have no idea about open source software, or you have no idea what it feels like to love programming, or both.
Please define what you mean by "publishing". Where has the author said anything more than "this is a project on my Github and here's what it does"?
Assuming the answer is nowhere:
Please retract. You appear to be one of those people with no understanding of modern open source culture, and only the ability to criticize. Do you expect everyone to keep their projects private until they meet your personal standards? (a) why the fuck should they? (b) what about a Github user who simply lacks the skills to ever meet your standards? (Evidently not the case here.) It's a shame I'm being overtly aggressive, but I think it's commensurate with the less overt unpleasantness of your comment.
I bet you're one of those people who thinks it's "irresponsible" to publish software on Github that could have bad consequences if someone were to rely on it in an important context. Repositories such as Github are free. They are a place for anyone to store whatever creations they come up with. It's a huge and uncuratable mixture of essential backbones of modern computing, art, utter incompetence, and technical genius, and a lot of stuff in the middle. How that is hard for people like you to understand I do not know. I guess you have a very boring job at a very boring company.
Assuming the answer is nowhere:
Please retract. You appear to be one of those people with no understanding of modern open source culture, and only the ability to criticize. Do you expect everyone to keep their projects private until they meet your personal standards? (a) why the fuck should they? (b) what about a Github user who simply lacks the skills to ever meet your standards? (Evidently not the case here.) It's a shame I'm being overtly aggressive, but I think it's commensurate with the less overt unpleasantness of your comment.
I bet you're one of those people who thinks it's "irresponsible" to publish software on Github that could have bad consequences if someone were to rely on it in an important context. Repositories such as Github are free. They are a place for anyone to store whatever creations they come up with. It's a huge and uncuratable mixture of essential backbones of modern computing, art, utter incompetence, and technical genius, and a lot of stuff in the middle. How that is hard for people like you to understand I do not know. I guess you have a very boring job at a very boring company.
I love this. Yes, there are higher level languages that can be used for accomplishing these things, but people find themselves at the command line a lot, and having the ability to efficiently express your intent using a familiar language, without getting hand-cuffed by a poor underlying representation seems very useful.
This looks to be in a similar vein to the Ion shell, which was developed for Redox [1]. Both written in Rust, both not (seemingly) perfectly POSIX-compatible, etc. I'm not sure I'd personally choose to go with either one over shell if I knew anyone else was going to need to use it, but it's cool to see iteration and experimentation on things that I usually consider to be the fundamentals of the computing environment.
[1] https://github.com/redox-os/ion
[1] https://github.com/redox-os/ion
I like these attempts at innovating the shell. Hardcore Unix users will say it is practically impossible to replace universal text streams and sh. But what if we had said the same thing before Unix wasn’t a thing? Then we wouldn’t have come this far today.
One advantage projects like Nushell and Crush have is that they have second-mover advantage after seeing what PowerShell didn’t get right. For instance whilst PowerShell is highly consistent, it’s also very verbose as a result (unless you tweak it to mimic bash with aliases and then you might as well use bash anyway). Another example is that, whilst the idea of objects in pipes is theoretically interesting, it creates a lot of complexity in creating meaningful pipes (it’s a lot faster to do xargs over a series of words rather than remembering how to do a for loop in pwsh). Maybe these people are striking the right balance with the flexibility of text and the rigour of tables / columns.
All nonsence. 2nd mover ? This would be like 100th mover... a testament its very hard to maintain such shell which is the reason nobody else succeded so far.
This is repeated ad infunum by the anti posh crew, while good arguments are presented against such attitude all the time.
Also, xargs is simply awful compared to what posh gives and awful without any comparrison.
This is repeated ad infunum by the anti posh crew, while good arguments are presented against such attitude all the time.
Also, xargs is simply awful compared to what posh gives and awful without any comparrison.
xargs is quite powerful and, most importantly, simple.
Not simpler then PowerShell way. And not even close in power.
I have recently been partial to python and the sh library. This and the other associate projects look interesting but are they secure and will they catch on?
This is not correct. Powershell allows calling into COM, .NET or straight DLLs for that matter.
It is also what I see as missing opportunity in most UNIX shells that failed to build up on Xerox PARC, Lisp Machines and ETHZ experiments.
The ability to directly plug into shared libraries, UNIX IPC, and modern mechanisms like DBUS and gRPC, alongside live debugging capabilities, thus exposing the whole OS stack to the CLI, making it a proper REPL experience.
Other than that, it looks quite cool.