Lsd: The next gen ls command(github.com)
github.com
Lsd: The next gen ls command
https://github.com/Peltoche/lsd
131 comments
Unfortunately, the Unix terminal is a gigantic pile of hacks accumulated since the late 1960s. It's a miracle the software stack manages to stay upright at all; adding features and getting the entire ecosystem to support them would be tough. Introducing the concept of an external resource that has to be fetched doesn't seem impossible to me, but there are a lot of issues that would have to be thought through.
Anyway, one nice thing about TrueType/OpenType fonts is that the vector representation is extremely well compressed. SVG or similar would be an efficiency loss. In this light, fonts are actually a fairly decent and practical way to represent vector symbols that will be cached and reused. Maybe a minimal practical solution might be to add an ANSI extension that requests an appropriate font for a particular area of the screen, perhaps with a URL from which a specific webfont can be fetched.
Anyway, one nice thing about TrueType/OpenType fonts is that the vector representation is extremely well compressed. SVG or similar would be an efficiency loss. In this light, fonts are actually a fairly decent and practical way to represent vector symbols that will be cached and reused. Maybe a minimal practical solution might be to add an ANSI extension that requests an appropriate font for a particular area of the screen, perhaps with a URL from which a specific webfont can be fetched.
> Unfortunately, the Unix terminal is a gigantic pile of hacks accumulated since the late 1960s
stty sane> Introducing the concept of an external resource that has to be fetched doesn't seem impossible to me, but there are a lot of issues that would have to be thought through.
I agree that an external resource is a lot of hurdles. What I had in mind while writing the reply was a stripped-down version of TrueType, with only a single `glyf` table allowed (probably with a minimal version of the `head` table as well). Judging from how icon fonts are used today, this might even be possible without a persistent font state.
I agree that an external resource is a lot of hurdles. What I had in mind while writing the reply was a stripped-down version of TrueType, with only a single `glyf` table allowed (probably with a minimal version of the `head` table as well). Judging from how icon fonts are used today, this might even be possible without a persistent font state.
>Unfortunately, the Unix terminal is a gigantic pile of hacks accumulated since the late 1960s.
Agreed, and I don't understand why most of the things still done in the Unix terminal have not transitioned to newer, less hacky environments.
I managed to replace the vast majority of the things I was using the terminal for with uses of about 400 lines of Emacs Lisp code that I wrote about 5 years ago.
A large fraction of my uses of the terminal since then have been for the purpose of my getting my Emacs environment and my data files installed on a fresh install of my OS and for recovering from those times when I inadvertently introduced a bug into my Emacs environment that my regression tests did not catch.
The 400 lines of Emacs Lisp code submits a string I typed to bash with the -c flag then asynchronously (i.e., without blocking Emacs's UI) waits for output from the bash process. I also added a way for the user to arrange for a bell to ring when the process finishes. Notably, there is no provision for the user to cause anything to be sent to the process's stdin after the process is started, which allows the 400 lines of code to use a pipe instead of a pty (which is relevant to this comment thread because if OSes didn't need to support the pty abstraction, they could be significantly simpler). In the rare case when I want to send a string to the stdin of some process I use the shell to set things up:
The closest analog in standard Gnu Emacs to the 400 lines I wrote is "shell mode" (shell.el) which consists of about 3 or 4 thousand lines of Emacs Lisp code. My code re-uses shell mode's code for interpreting (ignoring in my case) the escape sequences for ANSI color and for interpreting the escape sequences for "carriage control", but (like shell mode) does not handle the escape sequences for cursor addressing. IIRC the main reason I was dissatisfied with shell mode was that it contains hacky code to keep the bash process's "opinion" on what the current working directory is in sync with the shell mode buffer's "opinion" on what it is. In contrast, in my code, the buffer's "opinion" is the only one that matters (because every command line is processed by a fresh bash process on that bash process inherits its current working directory from the Emacs process when it is created).
If I ever were to start to need to issue a great many command lines on a variety of remote Unix servers, I would probably take the time to write a replacement for ssh along lines similar to what I just described.
Agreed, and I don't understand why most of the things still done in the Unix terminal have not transitioned to newer, less hacky environments.
I managed to replace the vast majority of the things I was using the terminal for with uses of about 400 lines of Emacs Lisp code that I wrote about 5 years ago.
A large fraction of my uses of the terminal since then have been for the purpose of my getting my Emacs environment and my data files installed on a fresh install of my OS and for recovering from those times when I inadvertently introduced a bug into my Emacs environment that my regression tests did not catch.
The 400 lines of Emacs Lisp code submits a string I typed to bash with the -c flag then asynchronously (i.e., without blocking Emacs's UI) waits for output from the bash process. I also added a way for the user to arrange for a bell to ring when the process finishes. Notably, there is no provision for the user to cause anything to be sent to the process's stdin after the process is started, which allows the 400 lines of code to use a pipe instead of a pty (which is relevant to this comment thread because if OSes didn't need to support the pty abstraction, they could be significantly simpler). In the rare case when I want to send a string to the stdin of some process I use the shell to set things up:
echo "here is the string" | some process
The closest analog in standard Gnu Emacs to the 400 lines I wrote is "shell mode" (shell.el) which consists of about 3 or 4 thousand lines of Emacs Lisp code. My code re-uses shell mode's code for interpreting (ignoring in my case) the escape sequences for ANSI color and for interpreting the escape sequences for "carriage control", but (like shell mode) does not handle the escape sequences for cursor addressing. IIRC the main reason I was dissatisfied with shell mode was that it contains hacky code to keep the bash process's "opinion" on what the current working directory is in sync with the shell mode buffer's "opinion" on what it is. In contrast, in my code, the buffer's "opinion" is the only one that matters (because every command line is processed by a fresh bash process on that bash process inherits its current working directory from the Emacs process when it is created).
If I ever were to start to need to issue a great many command lines on a variety of remote Unix servers, I would probably take the time to write a replacement for ssh along lines similar to what I just described.
I've been wanting to prototype the idea of an HTML-based terminal. Basically take the graphical user interface of the web and connect it to the system administration capabilities of the Unix command line. I'd probably draw inspiration from PowerShell: have programs produce some kind of data structure as output that can be rendered into HTML and displayed using stylesheets.
It would not be a trivial amount of work to build this since, to take good advantage of the UI capabilities, you'd need an entire new CLI OS - something to replace GNU Core Utilities. I haven't been sure where to start since the choice of programming language or expression language will significantly affect the overall experience.
It would not be a trivial amount of work to build this since, to take good advantage of the UI capabilities, you'd need an entire new CLI OS - something to replace GNU Core Utilities. I haven't been sure where to start since the choice of programming language or expression language will significantly affect the overall experience.
Add an out of band channel to ssh and your favorite ssh client/terminal emulator. Teach it that information in the out of band channel is used to render html, jupyter notebooks, or whatever. On the remote side, have your sshd set an environment variable that lets apps know the side channel is available, and then df can produce cute little pie charts or historic usage graphs. Ls can produce thumbnails if run in a directory with that kind of media and the appropriate args are used.
Honestly, it's likely that nothing special is needed on the ssh part of things as simple port forwarding and environment variables would do the trick.
Honestly, it's likely that nothing special is needed on the ssh part of things as simple port forwarding and environment variables would do the trick.
you have it already: https://hyper.is/
Great so now in addition to having to support everything the terminal does (the whole pile of garbage hacks that were added since the 60s) you have this electron crap running which on its own is probably more complex than 99% of software out there. This basically took the terminal and made it a hundred times worse.
OTOH, if you do what your parent suggests and start from scratch (no terminal support) you have a platform that isn't supported by anything yet, so unless you port or reimplement everything you need you have to keep using both in parallel which sounds just cumbersome.
OTOH, if you do what your parent suggests and start from scratch (no terminal support) you have a platform that isn't supported by anything yet, so unless you port or reimplement everything you need you have to keep using both in parallel which sounds just cumbersome.
[deleted]
Do you do anything to preserve the environment between calls to bash such that this behaves similarly to a typical shell?
No. Specifically, if the user issues the command line "cd ./foo; bar" a bash process is created that arranges for the wd to be ./foo during the execution of bar, but that cd command will have no effect on any subsequent command line (which will be evaluated by a different bash process).
If the target audience for this code were users other than just me, then it might not pay to arrange things that way, but all the motivation I personally needed to remember to treat the cd command differently while using the code I wrote was the very pleasant thought that my giving up on maintaining normal behavior of the cd command means that for my usage patterns, which again do not including using the code I wrote to interact with REPLs[1], no state need be maintained by any code whatsoever outside of the Emacs process except for the usual, expected state of the file system and any state internal to any bash processes and any of their child processes that are currently executing a command line.
Standard Gnu Emacs contains a M-x cd command, and that does modify the wd in a way that persists through the issuance of multiple command lines. (Again, I arranged for Emacs alone to keep track of the wd.)
[1]Standard Gnu Emacs contains the command M-x run-scheme, M-x run-python, etc, for interacting with a Scheme REPL, a Python REPL, etc. I do not happen to use any of them, but if I did start to use one of them, I probably would not feel tempted to re-write it the way I re-wrote shell mode (a.k.a. M-x shell) because I can think of no easy way to arrange things so that no state persists inside the REPL between command lines. What is special about interactive shells is that there is this single dinky piece of state, the wd, that if treated differently, makes the interface between the terminal or terminal-like thing and the shell process "connectionless" in the way that HTTP is connectionless: namely, after a command line finishes, the shell process and the terminal-like thing can just forget about each other.
I would imagine that this arrangement would be even more worthwhile for running shell command lines on lots of remote machines, but haven't written any code to effect it because I personally don't do that often.
ADDED. If I found myself writing "VAR=3 foo command" more than twice I would probably arrange for "VAR=3" to be permanently added the appropriate emacs variable (namely, process-environment IIRC). Basically, in my life the only environment variables that matter can be set once in a config file somewhere and forgotten.
If the target audience for this code were users other than just me, then it might not pay to arrange things that way, but all the motivation I personally needed to remember to treat the cd command differently while using the code I wrote was the very pleasant thought that my giving up on maintaining normal behavior of the cd command means that for my usage patterns, which again do not including using the code I wrote to interact with REPLs[1], no state need be maintained by any code whatsoever outside of the Emacs process except for the usual, expected state of the file system and any state internal to any bash processes and any of their child processes that are currently executing a command line.
Standard Gnu Emacs contains a M-x cd command, and that does modify the wd in a way that persists through the issuance of multiple command lines. (Again, I arranged for Emacs alone to keep track of the wd.)
[1]Standard Gnu Emacs contains the command M-x run-scheme, M-x run-python, etc, for interacting with a Scheme REPL, a Python REPL, etc. I do not happen to use any of them, but if I did start to use one of them, I probably would not feel tempted to re-write it the way I re-wrote shell mode (a.k.a. M-x shell) because I can think of no easy way to arrange things so that no state persists inside the REPL between command lines. What is special about interactive shells is that there is this single dinky piece of state, the wd, that if treated differently, makes the interface between the terminal or terminal-like thing and the shell process "connectionless" in the way that HTTP is connectionless: namely, after a command line finishes, the shell process and the terminal-like thing can just forget about each other.
I would imagine that this arrangement would be even more worthwhile for running shell command lines on lots of remote machines, but haven't written any code to effect it because I personally don't do that often.
ADDED. If I found myself writing "VAR=3 foo command" more than twice I would probably arrange for "VAR=3" to be permanently added the appropriate emacs variable (namely, process-environment IIRC). Basically, in my life the only environment variables that matter can be set once in a config file somewhere and forgotten.
I recently collected a list of image support in terminal emulators here: https://github.com/albertz/wiki/blob/master/terminal-escape-...
The only more or less widely used solutions are iTerms image support (PNG images, no vector), or Sixel (xterm and some other exotic ones).
There is also ReGIS (https://en.wikipedia.org/wiki/ReGIS), which is a vector format, which might be supported by xterm (not sure).
The only more or less widely used solutions are iTerms image support (PNG images, no vector), or Sixel (xterm and some other exotic ones).
There is also ReGIS (https://en.wikipedia.org/wiki/ReGIS), which is a vector format, which might be supported by xterm (not sure).
We have some standardized emoji like . Seems like the partial 2019 solution would be to use emoji that should work in any modern font (you could make a more UNIXy-feeling font if you want everything to look less skeuomorphic and more stripped-down). This doesn't cover every programming-specific use case, but it's a nice compromise I haven't seen mentioned here.
Indeed, there is a WG2/UTC proposal to officially add Powerline symbols to Unicode [1]. The problem of this approach is that those standards are very strongly (and probably rightly) governed with a lot of crufts and limitations, for example you can't put Github symbol there because it is trademarked.
[1] https://www.unicode.org/L2/L2019/19068-powerline-syms.pdf
[1] https://www.unicode.org/L2/L2019/19068-powerline-syms.pdf
Note that Hacker News strips emoji out from comments.
the partial 2019 solution would be to use emoji that should work in any modern font
This usually works not by having fonts define emojis (most don't) - it's just that systems reach in and render emojis from a dedicated emoji-defining font when they encounter them. So it's not like the world is full of fonts that define 'standard emojis' or funky powerline symbols.
This usually works not by having fonts define emojis (most don't) - it's just that systems reach in and render emojis from a dedicated emoji-defining font when they encounter them. So it's not like the world is full of fonts that define 'standard emojis' or funky powerline symbols.
> So... is there any way or proposal to encode a vector image in the terminal?
For raster images there are sixels, and for vector there are the tektronix codes. These are standard in xterm since forever.
Of course, they could be modernized, using real raster pixels and something like a subset of svg. I guess before standarization it would be better to have a reference implementation included in a major standard terminal.
For raster images there are sixels, and for vector there are the tektronix codes. These are standard in xterm since forever.
Of course, they could be modernized, using real raster pixels and something like a subset of svg. I guess before standarization it would be better to have a reference implementation included in a major standard terminal.
hterm[1] supports ReGIS which is vectors graphics.
A standard xterm supports vt340 if you start it with the option "-ti vt340" which supports ReGIS as well.
[1] http://41j.com/hterm/
A standard xterm supports vt340 if you start it with the option "-ti vt340" which supports ReGIS as well.
[1] http://41j.com/hterm/
xterm has a Tektronix 4014 mode, but that's not exactly what's needed or desired..
xterm also has sixels, which give reasonably high-resolution color graphics in normal VT340 mode.
https://github.com/saitoha/libsixel
By high-resolution I mean you can display JPEGs using sixels. You can run OpenGL programs using sixels.
https://github.com/saitoha/libsixel
By high-resolution I mean you can display JPEGs using sixels. You can run OpenGL programs using sixels.
Personally I always found using icons as defeating the whole point of using a terminal; One of the primary reasons I prefer terminal UIs over GUIs is that I don't have to decipher a bunch of strange hieroglyphs.
Even though of course nothing beats the power of a set of standardized and ubiquitous tools that you can always count on whenever you find yourself on a new machine, I must admit that I'm favorably inclined toward this trend (or what I perceive as one) of rewriting classic CLI tools with sometimes radical performance improvement promises, but most importantly (for me at least) better UIs: `grep` has always been acceptable for me, but I prefer `ag` these days, and I recently discovered `fd`, which seems like it will replace `find`, as I could never remember its weird syntax, even for the most basic cases.
Try rg (ripgrep) over ag:
https://github.com/BurntSushi/ripgrep
https://github.com/BurntSushi/ripgrep
Never managed to switch to ripgrep over ag. Performance difference is not that much of a dealbreaker in practice, and rg is not totally backwards-compatible to ag. I don't exactly remember, what was the problem. I guess it has this annoying practice to ignore everything that's in .gitignore, while for ag I could set up an additional .agignore per-project.
For ripgrep, if you want to use PCRE2 regexes (assuming that's what you meant by backwards compatible), then the -P flag will do that.
Otherwise, ripgrep's support for .gitignore should be substantially better than what's in ag. You can use .rgignore just like .agignore. Both ripgrep and ag also support .ignore. These can override whatever is in your .gitignore.
See the guide for more details: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#a...
> Performance difference is not that much of a dealbreaker in practice
This is true for smallish inputs. But the difference should become larger for bigger inputs. Hell, sometimes ag just refuses to search inputs that are too big. e.g., For a ~9.5GB file:
Otherwise, ripgrep's support for .gitignore should be substantially better than what's in ag. You can use .rgignore just like .agignore. Both ripgrep and ag also support .ignore. These can override whatever is in your .gitignore.
See the guide for more details: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#a...
> Performance difference is not that much of a dealbreaker in practice
This is true for smallish inputs. But the difference should become larger for bigger inputs. Hell, sometimes ag just refuses to search inputs that are too big. e.g., For a ~9.5GB file:
$ time rg '\w+\s+Sherlock Holmes' OpenSubtitles2016.raw.en -c
3003
real 1.608
user 1.105
sys 0.496
maxmem 9473 MB
faults 0
$ time ag '\w+\s+Sherlock Holmes' OpenSubtitles2016.raw.en -c
ERR: Skipping OpenSubtitles2016.raw.en: pcre_exec() can't handle files larger than 2147483647 bytes.
real 0.016
user 0.004
sys 0.011
maxmem 6 MB
faults 0Thank you for your answer.
What I mean, it doesn't seem to be possible to make it forget about .gitignore files completely. For me, it doesn't make any sense for my grep tool to pay any attention to .gitignore files (let alone info/exclude and such), these are 2 completely separate tools. I might occasionally want for some project to put into .rgignore the same line as I put in .gitignore, but that's an explicit choice. Actually, most of the time I have a few global ignores (which is possible to do with ripgrep) and an occasional .agignore file, while pretty much every git directory has .gitignore of some sorts. This is not something I can get around with, since while .agignore (.rgignore) is totally personal, .gitignore affects everyone who works on the project and serves explicitly to indicate that you ought not commit these files.
If I put "--no-ignore" in my .ripgreprc, it skips both .gitignore and .rgignore files, which isn't obviously what I'm trying to achieve.
One more thing (even thought it it possible to get around it by writing my own bash function that would invoke rg) is that unlike the ag, rg doesn't seem have "--pager" option, which I always use heavily, by setting half a dozen of parameters for `less` when grepping.
So these are what I consider a deal breakers and reasons why I didn't switch to rg: while it's true that in some cases I would appreciate a better performance, it's extremely rare that I grep something over a 9.5GB file. And conveniently search for a substring in a project is something that I'm in need multiple times a day.
BTW, noticed one more minor quirk right now: export RIPGREP_CONFIG_PATH="~/.ripgreprc" doesn't work (but RIPGREP_CONFIG_PATH="/home/$USER/.ripgreprc" does, so it's probably almost never an issue).
What I mean, it doesn't seem to be possible to make it forget about .gitignore files completely. For me, it doesn't make any sense for my grep tool to pay any attention to .gitignore files (let alone info/exclude and such), these are 2 completely separate tools. I might occasionally want for some project to put into .rgignore the same line as I put in .gitignore, but that's an explicit choice. Actually, most of the time I have a few global ignores (which is possible to do with ripgrep) and an occasional .agignore file, while pretty much every git directory has .gitignore of some sorts. This is not something I can get around with, since while .agignore (.rgignore) is totally personal, .gitignore affects everyone who works on the project and serves explicitly to indicate that you ought not commit these files.
If I put "--no-ignore" in my .ripgreprc, it skips both .gitignore and .rgignore files, which isn't obviously what I'm trying to achieve.
One more thing (even thought it it possible to get around it by writing my own bash function that would invoke rg) is that unlike the ag, rg doesn't seem have "--pager" option, which I always use heavily, by setting half a dozen of parameters for `less` when grepping.
So these are what I consider a deal breakers and reasons why I didn't switch to rg: while it's true that in some cases I would appreciate a better performance, it's extremely rare that I grep something over a 9.5GB file. And conveniently search for a substring in a project is something that I'm in need multiple times a day.
BTW, noticed one more minor quirk right now: export RIPGREP_CONFIG_PATH="~/.ripgreprc" doesn't work (but RIPGREP_CONFIG_PATH="/home/$USER/.ripgreprc" does, so it's probably almost never an issue).
You want --no-ignore-vcs. There are several other --no-ignore-* flags. ripgrep is a lot more flexible than ag there. And its implementation of the gitignore matching rules should work a lot better than ag. (Search ag's issue tracker to see just how many bugs there are. I'd be surprised if you weren't hitting any of them!)
As for a pager, that's pretty easy to solve: `rg foo | less -F`. Add `-R` to make colors work.
You don't need to search a 9.5GB file to make ag upset. All it takes is ~2GB. A key advantage of ripgrep over ag is that you don't have to just use it for code searching. It actually works just as well any place you'd use standard grep. ag just doesn't have an implementation that's good enough to be that robust.
As for a pager, that's pretty easy to solve: `rg foo | less -F`. Add `-R` to make colors work.
You don't need to search a 9.5GB file to make ag upset. All it takes is ~2GB. A key advantage of ripgrep over ag is that you don't have to just use it for code searching. It actually works just as well any place you'd use standard grep. ag just doesn't have an implementation that's good enough to be that robust.
> --no-ignore-vcs
Oh, thanks, I've missed it somehow.
> rg foo | less -F
Yeah, that's what I meant by writing my own wrapper, since I wouldn't want to type that in every time I search for something. And since I use bash, $@ in the middle of an alias is not possible, so: rg() { /usr/bin/rg $@ | less -LFXRqn }
It doesn't seem to be working as expected for me without explicitly adding --pretty though (obviously, rg sees it doesn't operate in a terminal and tries to be smart). End even then it's... weird. I don't know if that's `less` issue of `rg` issue, I'm guessing something about it being multi-threaded: it sometimes doesn't display some found items at the top of the terminal, but if I scroll screen down and up again -- it's there! Cannot reproduce with ag. Gonna play with this for some more time later, but it surely doesn't work as solidly with a pager, as ag --pager='whatever' does.
EDIT: actually found a bug report that seems to describe exactly the same problem. https://github.com/BurntSushi/ripgrep/issues/513
Oh, thanks, I've missed it somehow.
> rg foo | less -F
Yeah, that's what I meant by writing my own wrapper, since I wouldn't want to type that in every time I search for something. And since I use bash, $@ in the middle of an alias is not possible, so: rg() { /usr/bin/rg $@ | less -LFXRqn }
It doesn't seem to be working as expected for me without explicitly adding --pretty though (obviously, rg sees it doesn't operate in a terminal and tries to be smart). End even then it's... weird. I don't know if that's `less` issue of `rg` issue, I'm guessing something about it being multi-threaded: it sometimes doesn't display some found items at the top of the terminal, but if I scroll screen down and up again -- it's there! Cannot reproduce with ag. Gonna play with this for some more time later, but it surely doesn't work as solidly with a pager, as ag --pager='whatever' does.
EDIT: actually found a bug report that seems to describe exactly the same problem. https://github.com/BurntSushi/ripgrep/issues/513
Interesting. I can't reproduce that problem. Piping to less works perfectly fine on both my Linux and macOS machines.
If you're willing, I'd definitely appreciate it if you could update #513 with your environment. In particular, by answering these questions: https://github.com/BurntSushi/ripgrep/issues/513#issuecommen...
If it's convenient and you could try it in a different environment (perhaps a different terminal emulator or a different shell), then that would be great.
Also, this doesn't sound like a multi-threading issue to me, but if you want to rule that out, then `-j1` will force ripgrep into single threaded mode.
If you're willing, I'd definitely appreciate it if you could update #513 with your environment. In particular, by answering these questions: https://github.com/BurntSushi/ripgrep/issues/513#issuecommen...
If it's convenient and you could try it in a different environment (perhaps a different terminal emulator or a different shell), then that would be great.
Also, this doesn't sound like a multi-threading issue to me, but if you want to rule that out, then `-j1` will force ripgrep into single threaded mode.
Rg similarly supports it's own ignore file, and optionally disabling the default of respecting .gitignore.
Is there a way to get the results inline? e.g. first filename, then line number then the matched string? I don't want the filename, newline, matches syntax.
My sibling comment suggested --vimgrep, but that will print each match on its own line. For example:
$ echo foofoo | rg foo --vimgrep
<stdin>:1:1:foofoo
<stdin>:1:4:foofoo
that's it, thank you!
You might want to use the --vimgrep option:
Show results with every match on its own line, including line numbers and column numbers. With this option, a line with more than one match will be printed more than once.
Show results with every match on its own line, including line numbers and column numbers. With this option, a line with more than one match will be printed more than once.
Thanks, this works, I didn't see that option. --no-heading is also useful. (other comment)
ripgrep is incredible
Thank you. Find is so complex and looking at fd is a revelation here.
Now I feel like all my hard earned cli-fu is a lie.
Now I feel like all my hard earned cli-fu is a lie.
Care to share a link? Using fd as a utility name for an os that already uses the word fd as file descriptor is maddeningly hard to find
I'm assuming it's this -
https://github.com/sharkdp/fd
[deleted]
It's the first result for a maddeningly simple search of "cli fd".
I remember posting a bug to WSL about supposedly incorrect behavior with find, only to realize that I had incorrect syntax. I forgot to include the escape on the terminating semicolon (also, conveniently never shown the man pages), or maybe it was the enclosing parens around an or for name pattern matching.
Anyways, Id been away from *nix for a few years at the time, so I was a bit rusty and soon felt like a moron and closed the bug once I realized my error.
Anyways, Id been away from *nix for a few years at the time, so I was a bit rusty and soon felt like a moron and closed the bug once I realized my error.
> Find is so complex and looking at fd is a revelation here.
> Now I feel like all my hard earned cli-fu is a lie.
fd is also a lie. If you want a decent interface to find just use "find | grep ..."
> Now I feel like all my hard earned cli-fu is a lie.
fd is also a lie. If you want a decent interface to find just use "find | grep ..."
Shameless plug - I also wrote https://github.com/chmln/sd to replace my use of sed.
> While sed does a whole lot more, sd focuses on doing just one thing and doing it well.
I'm sure you already know this, but sed is meant to provide the features that ed does, so sd is quite a bit more limited. Still, I can see how this can be useful, as most invocations of sed end up only really using a handful of its features.
I'm sure you already know this, but sed is meant to provide the features that ed does, so sd is quite a bit more limited. Still, I can see how this can be useful, as most invocations of sed end up only really using a handful of its features.
Yeah, there's no hiding that, the fact that it's a find & replace CLI is declared upfront.
Although personally, I've only really used sed for find & replace so sd solved my gripes with sed perfectly.
Although personally, I've only really used sed for find & replace so sd solved my gripes with sed perfectly.
alias sd="perl -pe"
Have you tried RipGrep (rg)? I think it's the fastest one these days.
fd is excellent for casual interactive use, i've had a great UX with it and performance feels on par with GNU find.
I don't know if it's because I'm so used to find, but I had a bad UX with fd. This is my thought process each time I use it: "Why separating the extension from the filename? Is this a default? Can we change it? What is the option, -e? Or -e is the option to 'exec'? What's the flag to include hidden files? Or are they already included? Does the pattern comes before the directory? Does it need a directory or does it search it the current one? man fd, man fd, man fd."
I guess my brain just refuses to remember these for some reason.
I guess my brain just refuses to remember these for some reason.
Author of fd here. Thank you for your feedback.
If you feel that there is anything we could to to improve fd's UX, it would be great if you could share it on GitHub: https://github.com/sharkdp/fd
> Why separating the extension from the filename? Is this a default
It does not need to be! You can just also just use "fd README.md". Admittedly, the "." will actually search for any character (regex), so if you want to be precise, you need to escape it.
If you feel that there is anything we could to to improve fd's UX, it would be great if you could share it on GitHub: https://github.com/sharkdp/fd
> Why separating the extension from the filename? Is this a default
It does not need to be! You can just also just use "fd README.md". Admittedly, the "." will actually search for any character (regex), so if you want to be precise, you need to escape it.
Right, there's always going to be some additional cognitive overhead with every new tool you adopt. The question is how much you value its benefits for your use case, and is it worth those costs. If you're so used to find, and it works great for you, then keep using it! It's indeed more powerful and standard than fd, which I think is well-suited for vast majority of simple use cases (casual use).
However, here's my reasoning for the first of your questions, as a happy fd user who switched from GNU find. Maybe it can give you insight into why the UX is designed as it is.
The extension is distinguished from the filename because it's a common use case (-e jpg less keystrokes than -iname '*.jpg'), it's semantically separate from the filename (IMO) and therefore should not be treated as part of it. And unlike filenames, extensions most often do not need to be pattern matched. If you want to match multiple extensions, just repeat the -e flag e.g. -e jpg -e png.
However, here's my reasoning for the first of your questions, as a happy fd user who switched from GNU find. Maybe it can give you insight into why the UX is designed as it is.
The extension is distinguished from the filename because it's a common use case (-e jpg less keystrokes than -iname '*.jpg'), it's semantically separate from the filename (IMO) and therefore should not be treated as part of it. And unlike filenames, extensions most often do not need to be pattern matched. If you want to match multiple extensions, just repeat the -e flag e.g. -e jpg -e png.
One major showstopper I have with this utility is that it doesn't obey my terminal color scheme and instead hardcodes some colors that probably just look good on author's terminal. [1]
Would love to see this resolved, cause it seems rather nice otherwise.
https://github.com/Peltoche/lsd/issues/90
Would love to see this resolved, cause it seems rather nice otherwise.
https://github.com/Peltoche/lsd/issues/90
This project becomes pretty ridiculous when you realize you can already accomplish the same features with regular ls, if you just bother to read a bit. That means the target audience for this is very junior engineers.
This has been my experience with the majority of shell toys and dev ecosystem tools that have come out in the past 20+ years; if people simply knew Unix better, they'd be able to do them all with the same tools that have been around since the 80s/early-90s. There's nothing wrong with coming up with these alternatives, but they are almost universally minor improvements over what's available by simply becoming better at Unix.
There's also nothing wrong with trying to rethink the interface to be more simple and intuitive. UX is important and can be improved (or worsened), even though we may sometimes pretend it's not so.
Can ls use nerdfont? Does ls color coding display anywhere near the granularity that lsd and colorls feature?
So please drop the dismissive patronizing tone.
So please drop the dismissive patronizing tone.
OP's point is that if you know how to use ls you don't need icons and extra granularity in color scheme to understand the output.
The same is true of reading a hex dump of the inode table. Improved UIs can matter.
It’s not a valid point nor was it properly articulated.
It’s obviously not a matter of needing, but of aesthetics and usability, and that’s very much a valid pursuit.
It’s obviously not a matter of needing, but of aesthetics and usability, and that’s very much a valid pursuit.
Yes. Learn to use dircolors.
Prove it. Show me the output of BSD or GNU ls and compare it with lsd.
Go to your dircolors file in /etc/dircolors, if it’s not there create it first with the dircolors command. Make sure your terminal is using a nerdfont patched font. Set colors in dircolors for various file types to whatever you want and append or prefix whatever icons you want to use. Done. Define granular rules for other extensions if you wish.
Have you looked at lsd?
How does dircolors add support to nerdfont?
How does it add different colors to user, group, permissions in the equivalent of ls -l?
Also, dircolors is not included in OS X.
How does dircolors add support to nerdfont?
How does it add different colors to user, group, permissions in the equivalent of ls -l?
Also, dircolors is not included in OS X.
I’m sorry, I didn’t read your answer right. I’ll try what you suggested when get some time on a Linux box
Why would a senior engineer have time to bother with all that reading and configuring?
I think that was supposed to be the point: a senior engineer doesn't have to.
This is pretty and all, and there's nothing wrong with it, but...why? Do people run ls enough that they need something more than a plain ASCII list of filenames? Are peoples' directories so disorganized that this adds enough value to justify it over /bin/ls?
I probably run ls a few dozen times a day, but it's usually just to answer "what did I call that file again?" or "did I remember to copy that over?"
I can't imagine a scenario where I'd require too many extra visual cues. I use color ls, mostly because I find the optional symbols like '/' to be distracting, and differentiating files and directories via color is about all I care about.
There's overhead in becoming too attached to non-standard tools: you have to go out of your way to install them, your muscle memory needs retraining, etc.
Most importantly, tools like this break a bunch of core command line standards, like doing one thing and doing it well, and producing clean output that can be pipeline-filtered with all the other Unix standards.
I probably run ls a few dozen times a day, but it's usually just to answer "what did I call that file again?" or "did I remember to copy that over?"
I can't imagine a scenario where I'd require too many extra visual cues. I use color ls, mostly because I find the optional symbols like '/' to be distracting, and differentiating files and directories via color is about all I care about.
There's overhead in becoming too attached to non-standard tools: you have to go out of your way to install them, your muscle memory needs retraining, etc.
Most importantly, tools like this break a bunch of core command line standards, like doing one thing and doing it well, and producing clean output that can be pipeline-filtered with all the other Unix standards.
One need only consider that Orthodox File Managers have been in existence for decades to realize that yes indeed people do like to work with more than plain lists of filenames.
Playing the "doing one thing" card when it comes to ls is rather risky, as ls, with its options for doing things like sorting its output, is one of the favourite examples that people point to of tools that do not adhere to this philosophy in practice.
* https://news.ycombinator.com/item?id=4023265
* https://news.ycombinator.com/item?id=9673975
* https://news.ycombinator.com/item?id=8484718
... and so on.
Playing the "doing one thing" card when it comes to ls is rather risky, as ls, with its options for doing things like sorting its output, is one of the favourite examples that people point to of tools that do not adhere to this philosophy in practice.
* https://news.ycombinator.com/item?id=4023265
* https://news.ycombinator.com/item?id=9673975
* https://news.ycombinator.com/item?id=8484718
... and so on.
Has anyone made an "awesome" list of new CLI tools designed to replace ancient ones?
This might be the sort of thing you're looking for:
https://github.com/agarrharr/awesome-cli-apps
https://github.com/agarrharr/awesome-cli-apps
Found some links.
https://github.com/alebcay/awesome-shell
Only for rust but not a bad list I think.
https://lib.rs/command-line-utilities
Some blog post introducing cli tools.
https://darrenburns.net/posts/tools/
https://github.com/alebcay/awesome-shell
Only for rust but not a bad list I think.
https://lib.rs/command-line-utilities
Some blog post introducing cli tools.
https://darrenburns.net/posts/tools/
How does it compare with exa?
https://the.exa.website
https://the.exa.website
I've noticed a lot of command line utilities are being rewritten in Rust. Is there a reason for this?
Possibly because
1. People predict Rust will be more popular in the future, so they're trying to learn/master the language by practicing it.
2. People think speed and memory safety are important guarantees for unix programs.
3. People are testing whether Rust deserves the fame it has
1. People predict Rust will be more popular in the future, so they're trying to learn/master the language by practicing it.
2. People think speed and memory safety are important guarantees for unix programs.
3. People are testing whether Rust deserves the fame it has
> 2. People think speed and memory safety are important guarantees for unix programs.
I don't think any of the typical unix tools ever segfaulted on me in the last 15 years I use Linux.
I don't think any of the typical unix tools ever segfaulted on me in the last 15 years I use Linux.
Segfaults are the ideal case for memory errors, and those are the most easily caught and fixed, so you're least likely to see them. But, often those memory errors result in silent corruption which can be exploited, and that's harder to detect, especially if it relies on very specific corner cases. `curl` has had a number of these vulnerabilities over the last several years, for example.
Something as simple as `ls` is probably so small and battle tested that it's not an issue, but if you're writing an all-new, not-battle-tested tool, why wouldn't you want stronger guarantees? The new tool is being written for the features, but it's not fun to write vulnerabilities into something that should be simple and "just work."
Languages like Go and Ruby are also mostly memory safe, so those are generally fine picks here too, but every language has trade offs. In this case, the author clearly cares about performance, which Ruby does not care about.
Rust also has a built-in testing harness, which is a lot more convenient IMHO than using whatever C testing framework you might have a predisposition towards.
Something as simple as `ls` is probably so small and battle tested that it's not an issue, but if you're writing an all-new, not-battle-tested tool, why wouldn't you want stronger guarantees? The new tool is being written for the features, but it's not fun to write vulnerabilities into something that should be simple and "just work."
Languages like Go and Ruby are also mostly memory safe, so those are generally fine picks here too, but every language has trade offs. In this case, the author clearly cares about performance, which Ruby does not care about.
Rust also has a built-in testing harness, which is a lot more convenient IMHO than using whatever C testing framework you might have a predisposition towards.
Regarding segfaults, when I was interviewing devs for C++ roles, I'd ask questions about a simple function like this:
* if it crashes, where, when and why does it crash? (Technically since its undefined, nearly any answer here suffices, if it can be backed up. Since practically, most optimizing compilers assume UB can never happen, when you return nothing from a non-void function, the compiler will attempt to invoke the destructor of a non existent object instance (assuming non POD) and boom.)
I asked this because it was a distilled example of a real world rare crash that was extremely difficult bug to track down because the crash location is often know where near the offending function.
I remember getting into a heated argument with a coworker when I claimed it should have been a compilation error. IIRC, he claimed it to be a halting problem and that the compiler couldn't determine that all paths didnt return a value. I called BS, citing at the time (circa 2004) that the new compiler on the block for C# could reliably emit errors when not all return paths returned a value.
I also like it that in C++, it's a very rare example of a very terse example dealing with a number of topics such as undefined/implementation defined behavior, debugging, compilation settings (warning levels, etc) all in a mere 4 lines of codes. With 4 LOC, which is straight forward and simple for the candidate to mentally parse, I can gleam a lot about their understanding of the language (and it's potential pitfalls).
Sorry if this got a little long winded and ranting.
std::string foo(bool flag)
{
if (flag)
return "true";
}
* if it crashes, where, when and why does it crash? (Technically since its undefined, nearly any answer here suffices, if it can be backed up. Since practically, most optimizing compilers assume UB can never happen, when you return nothing from a non-void function, the compiler will attempt to invoke the destructor of a non existent object instance (assuming non POD) and boom.)
I asked this because it was a distilled example of a real world rare crash that was extremely difficult bug to track down because the crash location is often know where near the offending function.
I remember getting into a heated argument with a coworker when I claimed it should have been a compilation error. IIRC, he claimed it to be a halting problem and that the compiler couldn't determine that all paths didnt return a value. I called BS, citing at the time (circa 2004) that the new compiler on the block for C# could reliably emit errors when not all return paths returned a value.
I also like it that in C++, it's a very rare example of a very terse example dealing with a number of topics such as undefined/implementation defined behavior, debugging, compilation settings (warning levels, etc) all in a mere 4 lines of codes. With 4 LOC, which is straight forward and simple for the candidate to mentally parse, I can gleam a lot about their understanding of the language (and it's potential pitfalls).
Sorry if this got a little long winded and ranting.
> he claimed it to be a halting problem and that the compiler couldn't determine that all paths didnt return a value.
Theoretically we can't determine whether a function will return a value or not. In practice, heuristics get 99% of the way and the last 1% you can make the programmer put in a possibly redundant return statement.
Theoretically we can't determine whether a function will return a value or not. In practice, heuristics get 99% of the way and the last 1% you can make the programmer put in a possibly redundant return statement.
There are two different questions: "do all paths lead to returning a value?" and "will the function return a value?"
Answering the second in the general case is equivalent to solving the halting problem. But answering the first question is much simpler. Static analyzers aren't using a heuristic approach to the second question, they're solving a completely different problem.
Answering the second in the general case is equivalent to solving the halting problem. But answering the first question is much simpler. Static analyzers aren't using a heuristic approach to the second question, they're solving a completely different problem.
Don't get me wrong, but if whole classes of errors, can be avoided by not choosing C++, why should I choose to use it?
Because you already know C++ and don't want or can't justify learning another language. That's a very valable reason.
Of course you can decide the type a function returns, to overkill this problem, just run any type inference algorithm powered by unification (e.g. hindley milner). Give undeclared variables the type NULL and you're all set since you can decide whether the function returns type NULL.
Your friend was confused because you cannot decide whether a certain line in your program will be executed. Since this is equivalent to the halting problem. Let's prove this! Assume we have a blackbox B(P,x,N) decides whether line N of program P will be executed given input x. Here, we can solve the halting problem:
```pseudocode
def Helper(sourcecode P, input x):
This means, given programs like your interview question, compiler cannot decide -- in general -- whether the program will crash or work. Of course, it's not too hard to find "good enough" heuristics that'll catch some cases and/or restricting your language to make it "harder" to encode such programs.
Your friend was confused because you cannot decide whether a certain line in your program will be executed. Since this is equivalent to the halting problem. Let's prove this! Assume we have a blackbox B(P,x,N) decides whether line N of program P will be executed given input x. Here, we can solve the halting problem:
```pseudocode
def Helper(sourcecode P, input x):
[0] P(x)
[1] print "What Do We Say to the God of Death?"
if B(Helper,(P,x),1):
return true
else:
return false
This means, given programs like your interview question, compiler cannot decide -- in general -- whether the program will crash or work. Of course, it's not too hard to find "good enough" heuristics that'll catch some cases and/or restricting your language to make it "harder" to encode such programs.
> Is the function well formed (Yes - functions need not return a value on all paths due to C ancestory, even if the return type has a non trivial actor - not sure if this is still considered well formed, but I think it was as least to C++11)
In C falling off the end of a function without returning is legal, as long as you don't use the return value. In C++ this is illegal.
In C falling off the end of a function without returning is legal, as long as you don't use the return value. In C++ this is illegal.
That's due to no shortage of effort by the maintainers, however: one small bug can easily lead to memory safety problems.
Rust is great in this sense because you don't have to think about those issues.
That said, I'd pick rust over C for most projects simply because it's a nicer language to write IMHO. It feels like a modern high-level language with things like iterators, traits, etc, and it has a great package manager too. In comparison, I find writing C to be a huge chore, especially if you're doing much string processing
Rust is great in this sense because you don't have to think about those issues.
That said, I'd pick rust over C for most projects simply because it's a nicer language to write IMHO. It feels like a modern high-level language with things like iterators, traits, etc, and it has a great package manager too. In comparison, I find writing C to be a huge chore, especially if you're doing much string processing
Very importantly, Rust's safety guarantees help with "fearless concurrency", and better multithreading accounts for most of the performance gain we see in these new tools.
You can have fearless concurrency in C too if you use async queues and handover data completely without sharing references to data between threads (other than via the queues) - including using thread safe functions.
Rust will not help you with other multi-threadding issues, that arise from how OS implements processes/threads/signals/syscalls etc.
So I wouldn't call it fearless. Concurrent data access is just one issue you have to tak care around.
Rust will not help you with other multi-threadding issues, that arise from how OS implements processes/threads/signals/syscalls etc.
So I wouldn't call it fearless. Concurrent data access is just one issue you have to tak care around.
Well, they certainly have for me. As an example, GNU awk has some rather easy to the trigger segfaults which do not seem like they are going to be fixed soon.
Have you tried piping /dev/random to them? I bet you could get some of coreutils and a lot more BSD tools to crash by doing that.
The time window is very important here. csh, for instance, famously crashed all the time.
I think its worth mentioning that rust has the gold standard argument parsing library now:
https://clap.rs
That library is used by most(?) rust cli tools, so they all have a similar feel. In addition, there is a library built on top of clap called struct-opt that makes bringing up a new cli exceedingly easy: https://docs.rs/structopt/0.2.15/structopt/
Seriously, checkout struct-opts page even if you don't know any rust. I think their demonstration on the front page summarizes a lot of things I like about idiomatic rust.
That library is used by most(?) rust cli tools, so they all have a similar feel. In addition, there is a library built on top of clap called struct-opt that makes bringing up a new cli exceedingly easy: https://docs.rs/structopt/0.2.15/structopt/
Seriously, checkout struct-opts page even if you don't know any rust. I think their demonstration on the front page summarizes a lot of things I like about idiomatic rust.
I haven't needed it in a long time, but I still think trollop for ruby is magical. It seems it's been renamed to optimist now: http://manageiq.github.io/optimist/
I'd say Rust is a productive language for providing competitive CLIs.
- Compiled so easy to distribute compared to Python, JS, etc - Cargo makes it easy to pull in libraries compared to C/C++ - Smart people made some great libraries that make it trivial to get performance and/or a nice UX. e.g. Everything in ripgrep is available for reuse, down to how it walks the filesystem.
- Compiled so easy to distribute compared to Python, JS, etc - Cargo makes it easy to pull in libraries compared to C/C++ - Smart people made some great libraries that make it trivial to get performance and/or a nice UX. e.g. Everything in ripgrep is available for reuse, down to how it walks the filesystem.
I’d say it’s about 80% projects for people wanting to learn the language and 20% about safety, speed and concurrency.
Because it's a low level system's programming language with Haskell influences (makes it good for modelling a well-defined problem domain) with the "rewrite it in Rust" meme behind it. It's meant to be a replacement for C, so it makes sense that rustaceans would want to write replacements for programs written in C.
In this case it appears they are getting better performance than colorls, whose features the author presumably likes.
1. Hype
2. Rust is, despite reasons for which I don't use it, a good language which seems to achieve it's goals
3. Why not
Given the name, I was surprised at how much sense the choices of colours made.
How does this compare to exa.
It has a much better name for one.
I don’t think exa uses nerdfont for one.
It’s also a lot faster, according to the benchmarks.
It’s also a lot faster, according to the benchmarks.
exa does have a fork which does but exa's development seems to have paused.
https://github.com/ogham/exa/pull/368
https://github.com/ogham/exa/pull/368
Wish it would disable ANSI stuff (--classic) if it's not a vt100/xterm/etc TERMinal.
wow this is amazing... I love the file icons and it's beautiful... ls --tree ran almost instantly on a laravel folder which has a huge number of packages....
I use below sometimes. Hope you guys like these.
A modern replacement for ls, written in Rust. https://the.exa.website/
ls with coloring and icons https://github.com/Electrux/ls_extended
A modern replacement for ls, written in Rust. https://the.exa.website/
ls with coloring and icons https://github.com/Electrux/ls_extended
Is there a list of features? Or can I assume it implements the same features as colorls?
Why would I use this over exa? https://github.com/ogham/exa
We really need a way for terminfo/termcap to indicate whether a user has Powerline fonts. Would be great for programs like this.
Looking at the example -- do the symbols really help any more than just normal highlighting, and, `x --> y` and `/` at the end?
[deleted]
There is an easier way to do this - Use File Manager UI
Is that something you can use from the terminal?
Of course https://ranger.github.io/
At that point, just use MC... https://midnight-commander.org/
to install "sudo snap install lsd --classic"
It doesn't install a man page for itself, requires weird-ass fonts, has a fraction of the features of the real ls [0], and runs about half as fast ...
However it's written in our lord and savior RUST and has lots of colors so this shit is definitely "next gen."
0: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls...
However it's written in our lord and savior RUST and has lots of colors so this shit is definitely "next gen."
0: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls...
Would you please stop posting snarky comments here? The site guidelines explicitly ask you not to: https://news.ycombinator.com/newsguidelines.html.
I'd call this a "shallow dismissal" (https://news.ycombinator.com/newsguidelines.html): perhaps you could reword your response to be a bit nicer?
I think there was a number of reasons given.
It wasn't particularly nice, though, and I'm not quite sure all of the comments given qualify as non-shallow.
[deleted]
I will stick with exa instead. https://github.com/ogham/exa
I am using exa now, and was wondering the same thing. Could you explain why you made the decision? what's the difference?
I would try it if @burntsushi wrote it...
> our lord and savior RUST
So you're using this as an opportunity to bash an entire language ecosystem? The merits of this project have no bearing on Rust or its adoption.
So you're using this as an opportunity to bash an entire language ecosystem? The merits of this project have no bearing on Rust or its adoption.
I think they were referring to the fact that when a new popular language arises, people rush to build things in that language and then act as if the language itself is sufficient to make the resulting product good instead of any inherent merits.
I don't think the parent was bashing Rust.
I think the parent was bashing the attendant hype and the resulting reinventions but worse.
There was a blog I read about the Rust Zealot Brigade (or similar name) that would reliably descend on any HN conversation.
I think the parent was bashing the attendant hype and the resulting reinventions but worse.
There was a blog I read about the Rust Zealot Brigade (or similar name) that would reliably descend on any HN conversation.
I'm geekin lol
They’re gonna need a better list of reasons to try it out than:
1. It’s written in the author’s preferred language
1. It’s written in the author’s preferred language
[1] https://github.com/ryanoasis/nerd-fonts
[2] https://github.com/ryanoasis/nerd-fonts/wiki/Codepoint-Confl...
[3] https://www.iterm2.com/documentation-images.html