WezTerm – A GPU-accelerated cross-platform terminal emulator and multiplexer(wezfurlong.org)
wezfurlong.org
WezTerm – A GPU-accelerated cross-platform terminal emulator and multiplexer
https://wezfurlong.org/wezterm/
127 comments
This is an honest question. What is the advantage of GPU acceleration for a terminal emulator?
I don’t do much low level graphics stuff, but a terminal seems fairly simple graphically, you are just drawing glyphs at xy cords. Would love it if someone had a good explanation.
Obviously there are some terminal applications that could be redrawing very quickly, but that seems like a solved problem.
I don’t do much low level graphics stuff, but a terminal seems fairly simple graphically, you are just drawing glyphs at xy cords. Would love it if someone had a good explanation.
Obviously there are some terminal applications that could be redrawing very quickly, but that seems like a solved problem.
Is it more complicated than wanting to update the display as often as its fresh frequency? Faster screen updates, lower CPU & power usage.
You might not notice the difference until you accidentally unleash a lot of output at once - then you'll see the difference. Some terminals start to update infrequently to save themselves, some hang completely while burning the CPU.
Psychologically I feel more in control of my computer, more confident to try things out if I can see my typing updating instantly than 10s or 100s of milliseconds later.
You might not notice the difference until you accidentally unleash a lot of output at once - then you'll see the difference. Some terminals start to update infrequently to save themselves, some hang completely while burning the CPU.
Psychologically I feel more in control of my computer, more confident to try things out if I can see my typing updating instantly than 10s or 100s of milliseconds later.
Lower CPU usage but higher GPU usage. On my laptop I get muuuch more battery out of it (going from 6 hours to ~2 hours) if I disable everything that may wake up the GPU and force software rendering everywhere (since, unlike GPUs, it's much more often able to rerender only the parts of the screen which changed)
> rerender only the parts of the screen which changed
e.g. Firefox is already good at doing that on the GPU. WezTerm isn't yet, but We're Working On It™ (i.e. someday it will be done), follow https://github.com/wez/wezterm/issues/1134
e.g. Firefox is already good at doing that on the GPU. WezTerm isn't yet, but We're Working On It™ (i.e. someday it will be done), follow https://github.com/wez/wezterm/issues/1134
That isn't an inherent limitation of GPUs, though. More likely just a missing feature in whatever GPU rendering path.
Wall socket
> until you accidentally unleash a lot of output at once
The only time this happens, is when I do something wrong, like `cat`-ing the 12MB binary instead of the config.json, or `tail -f` into the output device of the core logging endpoint and forget to pipe into `grep`.
If that happens, I usually terminate the process anyway.
The only time this happens, is when I do something wrong, like `cat`-ing the 12MB binary instead of the config.json, or `tail -f` into the output device of the core logging endpoint and forget to pipe into `grep`.
If that happens, I usually terminate the process anyway.
You can also run ‘reset’ which will clear escape codes and reset your session.
This is also pretty common when running test runners a lot
So I pipe their output into a file, or into a logger engine. Problem solved.
I believe that there is a difference. I would just love to understand it better.
Does anyone have a benchmark of the differences? Is it like a 2-3x improvement or only much smaller? What is the actual latency?
Does anyone have any links to a low level explanation?
Does anyone have a benchmark of the differences? Is it like a 2-3x improvement or only much smaller? What is the actual latency?
Does anyone have any links to a low level explanation?
A really basic test is just type "base64 < /dev/urandom" in your terminal. How fast does the screen update? And if you press ctrl+C, how long does it take to get your shell prompt back?
(ofc the differences might not be something you ever notice or care about)
(ofc the differences might not be something you ever notice or care about)
e.g. for me on Windows 11, Wezterm updates slower than both cmd.exe and Windows Terminal. It feels like about 300-400ms to give me control back when I hit ctrl+C. Windows Terminal must be about 100-200ms, cmd.exe maybe a tiny bit slower.
I'm not sure speed benchmarks mean much for terminals though, there's going to be a lot of personal preference involved.
I'm not sure speed benchmarks mean much for terminals though, there's going to be a lot of personal preference involved.
The new terminal in windows 11 has GPU acceleration.
Tho control character latency likely doesn’t have anything to do with the responsiveness that GPU text rendering provides, since ctrl+c is a sequence captured by the OS it’s more likely to have something to do with which ever API the terminal uses to capture input or even the OS itself since I guess the OS still captures the sequence first then has some checks if the object can be universally copied and if not passes the sequence to the active window in focus for it to handle that.
Tho control character latency likely doesn’t have anything to do with the responsiveness that GPU text rendering provides, since ctrl+c is a sequence captured by the OS it’s more likely to have something to do with which ever API the terminal uses to capture input or even the OS itself since I guess the OS still captures the sequence first then has some checks if the object can be universally copied and if not passes the sequence to the active window in focus for it to handle that.
But what's the use case for this?
When do I actually need to DISPLAY megabytes worth of output as lag-free as possible on a terminal?
There is no way for the intended recipient of displayed information (aka. the user) to process any of it, so what's the point of eliminating lag?
When do I actually need to DISPLAY megabytes worth of output as lag-free as possible on a terminal?
There is no way for the intended recipient of displayed information (aka. the user) to process any of it, so what's the point of eliminating lag?
What about when you don't know that it ends up being megabytes ? Say you're running Gradle on a large project that does a full recompilation. You don't really particularly care, you know that module X has warnings, etc. But it still prints it all out by default. With carriage returns to update the current percentage, ANSI codes and more. Any millisecond that you spend blocking on outputting your terminal is time that your build can't take. This is compounded by other factors, but ultimately, writing to the terminal is not an asynchronous thing, so your process _will_ be impacted by it being slow.
>Any millisecond that you spend blocking on outputting your terminal is time that your build can't take.
Many Build tools don't just pipe the output of every program to their own STDOUT, but redirect them to a logger, which usually runs asyncronously, so from the PoV of the compiler, its output is processed without delay.
Secondly, if I know that the build produces a huge amount of output, I redirect it to a file instead. If I don't know the first time, I interrupt, correct my command and re-build.
Many Build tools don't just pipe the output of every program to their own STDOUT, but redirect them to a logger, which usually runs asyncronously, so from the PoV of the compiler, its output is processed without delay.
Secondly, if I know that the build produces a huge amount of output, I redirect it to a file instead. If I don't know the first time, I interrupt, correct my command and re-build.
> But what's the use case for this?
telnet towel.blinkenlights.nl
telnet towel.blinkenlights.nl
well... come to the future: https://www.youtube.com/watch?v=dcjkezf1ARY
And what exactly is the use case for this?
If I want to work with images, animations, etc. I use a GUI library. These in turn already use hardware accelerated rendering.
And I already have powerful, optimized, tested, reliable software that enables me to work with terminal and GUI functionality side by side; the DESKTOP ENVIRONMENT.
If I want to work with images, animations, etc. I use a GUI library. These in turn already use hardware accelerated rendering.
And I already have powerful, optimized, tested, reliable software that enables me to work with terminal and GUI functionality side by side; the DESKTOP ENVIRONMENT.
> And what exactly is the use case for this?
Terminal User Interfaces. There's technically no reason i can't check out an image in a folder when i'm browsing on an SSH session (yes i know i can rsync/scp && xdg-open but it's not exactly as fast to type as viu [0].
> powerful, optimized, tested, reliable software that enables me to work with terminal and GUI functionality side by side
If you have tested and reliable desktop environments to recommend, i'm all ears. All the ones i've tried over the years have their own quirks and memory leaks (yes, that includes GNOME and KDE).
But as you said, both approaches are complementary. I'm glad notcurses exists and works via graceful degradation, so people with a modern terminal can get the best while others can still get a featureful ncurses-like experience.
[0] https://github.com/atanunq/viu
Terminal User Interfaces. There's technically no reason i can't check out an image in a folder when i'm browsing on an SSH session (yes i know i can rsync/scp && xdg-open but it's not exactly as fast to type as viu [0].
> powerful, optimized, tested, reliable software that enables me to work with terminal and GUI functionality side by side
If you have tested and reliable desktop environments to recommend, i'm all ears. All the ones i've tried over the years have their own quirks and memory leaks (yes, that includes GNOME and KDE).
But as you said, both approaches are complementary. I'm glad notcurses exists and works via graceful degradation, so people with a modern terminal can get the best while others can still get a featureful ncurses-like experience.
[0] https://github.com/atanunq/viu
Please be aware that all uppercase comes across as quite aggressive.
Writing a lot of data to a terminal can happen accidentally or by having an application running that throws out a burst of log messages. In both cases, you'll want to have your terminal to be responsive again as soon as possible and you want it to be light on your cpu s.t. it doesn't interrupt other applications.
Writing a lot of data to a terminal can happen accidentally or by having an application running that throws out a burst of log messages. In both cases, you'll want to have your terminal to be responsive again as soon as possible and you want it to be light on your cpu s.t. it doesn't interrupt other applications.
> In both cases, you'll want to have your terminal to be responsive again as soon as possible.
My terminal emulator doesn't become unresponsive just because the renderer cannot keep up. It still processes input, so if I accidentially
> it doesn't interrupt other applications.
It can't do that anyway in an environment that does preemptive multitasking.
My terminal emulator doesn't become unresponsive just because the renderer cannot keep up. It still processes input, so if I accidentially
cat some-linux-distro-1of4.iso
> it doesn't interrupt other applications.
It can't do that anyway in an environment that does preemptive multitasking.
Android logs.
For a basic example of why you would want GPU acceleration, have a look at refterm: https://github.com/cmuratori/refterm
Even processes that you wouldn't think would be impacted by a terminal can be hurt relaly bad by your terminal's performance: your compiler's logs, etc. The GPU rendering part merely guarantees that your terminal sticks at 60FPS (or, whatever your refresh rate is) if the processing behind is efficient.
Even processes that you wouldn't think would be impacted by a terminal can be hurt relaly bad by your terminal's performance: your compiler's logs, etc. The GPU rendering part merely guarantees that your terminal sticks at 60FPS (or, whatever your refresh rate is) if the processing behind is efficient.
This was a solved problem in the 1980's with Amiga consoles separating input processing, high level terminal handling and rendering into separate threads.
I tested kitty (GPU accelerated) and rxvt on my laptop, and rxvt beat kitty on throughput. Now, kitty looks much nicer, and I prefer it over rxvt any day. My local copy applies a shader to add a subtle glow so the transparent background doesn't make the text unreadable, for example. That is where GPU rendered terminal shines - not speed. High terminal throughput you get from good architecture (do decouple IO and rendering in separate threads; don't trigger your rendering pipeline for every character output if the buffer is growing; very few terminals bother to do this because a more naive approach is generally fast enough on modern hardware), and even a brutally naive CPU rendered terminal can render fast enough if your architecture is decent.
Well since GPU acceleration minimizes latency (because of zero-copy, all character textures are resident in video memory), that is a boon to perceived responsiveness. IMO that is the single most important feature in a terminal, next to correctness. It’s like typing on an old serial terminal.
Latency and throughput benchmarks don't confirm this. Eshell has one of the best latencies (that's why it feels so responsive compared to VS Code) but low throughput
Generally if you are barfing meg's of data to screen you're prolly doing something wrong... (You prolly wanna be using less?)
And of course your next keystroke is not zero copy. It needs to be sent to the GPU and the texture needs to be updated. GPU terminals seem to do this kinda slowly last I checked
Generally if you are barfing meg's of data to screen you're prolly doing something wrong... (You prolly wanna be using less?)
And of course your next keystroke is not zero copy. It needs to be sent to the GPU and the texture needs to be updated. GPU terminals seem to do this kinda slowly last I checked
> And of course your next keystroke is not zero copy. It needs to be sent to the GPU and the texture needs to be updated.
No textures need to be updated. The texture stays in VRAM across each frame, just need to change which texture the character cell points to. That is zero-copy.
If you were to do this with a software rendered terminal, at minimum the software would tell the windowing system which region of the window changed and then copy that region to VRAM. That’s only if the window system supports region updates, if not you’d need to copy the entire window to VRAM each frame. Much slower than just twiddling a pointer to a texture.
No textures need to be updated. The texture stays in VRAM across each frame, just need to change which texture the character cell points to. That is zero-copy.
If you were to do this with a software rendered terminal, at minimum the software would tell the windowing system which region of the window changed and then copy that region to VRAM. That’s only if the window system supports region updates, if not you’d need to copy the entire window to VRAM each frame. Much slower than just twiddling a pointer to a texture.
Show me the numbers..
https://danluu.com/term-latency/
https://danluu.com/term-latency/
Well all these data tell us is that alacritty has more latency than some terminals and less latency than others. It does not tell us why it displays the latency that it does. In other words, correlation is not causation. Many other factors are at play here.
I’ve given you a rationale for why the GPU rendering technique inherently minimizes latency over the software rendered technique. Can you provide a rationale for why software rendered terminals would have an inherent advantage in terms of latency?
I’ve given you a rationale for why the GPU rendering technique inherently minimizes latency over the software rendered technique. Can you provide a rationale for why software rendered terminals would have an inherent advantage in terms of latency?
> Generally if you are barfing meg's of data to screen you're prolly doing something wrong... (You prolly wanna be using less?)
Why? Many commands I've run have produced a lot of log output in my day.
Why? Many commands I've run have produced a lot of log output in my day.
Bc you wanna see the output? With less you can go page by page and read the output. If there is so much output you can't read it.. then what's the point of printing it to the screen?
If you're looking for something in the stream then maybe you should grep for it?
I'm not saying you never ever should barf to screen - but as far as I can see, you kinda have to strain yourself to concoct a "valid" usecase
If you're looking for something in the stream then maybe you should grep for it?
I'm not saying you never ever should barf to screen - but as far as I can see, you kinda have to strain yourself to concoct a "valid" usecase
The irony of this is that with a faster terminal you'll be even less likely to actually read the data scrolling by. And the solution is very simple: Update a text buffer in one thread and render updates as fast as possible on another. If something barfs massive amounts of text you won't ever see all of it as not all of it will get rendered as it'll have "scrolled past" before the renderer gets to it. But it won't matter because it will be in scrollback if configured and you'd never have been able to read it as it passes by anyway, even with much faster rendering.
Terminals don't just render what you see, but everything that programs output, which can be enormous amounts of text, grinding non-accelerated terminals to a halt.
I don't know if they actually have to do this (It might be necessary for seamless scrolling?), but building some partial rendering might be more complicated than GPU acceleration as well.
I'd be interested if someone knows more about this.
I don't know if they actually have to do this (It might be necessary for seamless scrolling?), but building some partial rendering might be more complicated than GPU acceleration as well.
I'd be interested if someone knows more about this.
I don't get it either. The 2D APIs used for font rendering built into every window system (xrender, etc.) are already hardware accelerated and more than fast enough. Going to OpenGL or whatever this uses just seems ridiculous. There's no use case for rendering your terminal emulator at 10,000 frames per second.
"cat 1_gb_of.txt" will push your terminal far beyond what just a CPU can comfortably handle.
And when do I want to `cat` 1GiB of text?
I cannot process it with my human eyes, so what does it matter if the CPU rendering can keep up or not?
I cannot process it with my human eyes, so what does it matter if the CPU rendering can keep up or not?
cat 1_gb_of.txt | less
When you don't actually realise it is 1 GB before you cat it.
So then I hit Ctrl-C and issue the correct command. How does a GPU based renderer improve on that use-case?
Before you press Ctrl+C and type the correct command, your neighbour running Kitty has already got the last line of the output (which is often the only thing that one really needs).
I switched to Kitty because I felt that the increased speed produced a perceived improvement in my workflow.
I switched to Kitty because I felt that the increased speed produced a perceived improvement in my workflow.
>which is often the only thing that one really needs)
tail 1_gb_of.txt
As other said, sometimes you do not know that there is going to be a lot of output. In my case, I often save the time it takes to press Ctrl+C and add "tail" at the beginning of the previous command
I see you posted the same comment again and again, every time asking for a possible use case, and when people provided you with theirs, you answered, «well, but I would…», with the tone of who is saying that mine is the only rightful way to do it.
Well, it's not.
I see you posted the same comment again and again, every time asking for a possible use case, and when people provided you with theirs, you answered, «well, but I would…», with the tone of who is saying that mine is the only rightful way to do it.
Well, it's not.
[deleted]
It improves it by me not having to press Ctrl-C and issue the correct command. Surely you can see how that is an improvement.
... if the terminal is poorly designed and renders every character update even when the buffer is filling, rather than batch updates.
Very little unless you use shaders for effects. . Updating terminals fast enough was a solved problem on 30 year old hardware. Most of the slowness of some terminal apps is down to poor design not CPU rendering (see the speed differences between various CPU rendered terminals)
30 year old hardware didn't have to render fancy antialiased fonts at 3840x2160 @ 144Hz.
My computer in the 80s ran at 7.16MHz and a single core. The CPU speedup has increased far more than the amount of data to be moved even at the kind of setup you're describing.
Data rates for display have increased very much too. Don't underestimate the, uhh, "how rectangles work" factor – doubling the resolution means doubling both width and height, so quadrupling the pixel count.
80s computers also used lots of special hardware for graphics output, they didn't software render everything ;)
80s computers also used lots of special hardware for graphics output, they didn't software render everything ;)
Yes, but as I said the increase in data rates has not been anywhere near as fast as the increase in CPU speed, even with the unusal example you gave.
As for special hardware for graphics output, that's not really true. Some, sure. My Amiga had a basic blitter. But all that allowed for that helped a terminal was scrolling at a few times higher speed than the CPU - it was limited by the memory bus speed. That was revolutionary at the time. Most other 80's hardware had much less, except for having character modes, but I was specifically talking about ones using bitmap screens like the Amiga for example.
But a blitter does not help you with the worst case scenario of filling the terminal with text from scratch as the setup cost is typically higher than the cost of having the CPU render individual characters (it certainly was on the Amiga - the blitter helped when moving larger chunks of data, but not for character sized areas).
I have written terminals for 1980s hardware and for modern hardware. I know where the bottlenecks are and there is no issue whatsoever keeping up with the graphical display. When a terminal is slow it's a design issue.
As for special hardware for graphics output, that's not really true. Some, sure. My Amiga had a basic blitter. But all that allowed for that helped a terminal was scrolling at a few times higher speed than the CPU - it was limited by the memory bus speed. That was revolutionary at the time. Most other 80's hardware had much less, except for having character modes, but I was specifically talking about ones using bitmap screens like the Amiga for example.
But a blitter does not help you with the worst case scenario of filling the terminal with text from scratch as the setup cost is typically higher than the cost of having the CPU render individual characters (it certainly was on the Amiga - the blitter helped when moving larger chunks of data, but not for character sized areas).
I have written terminals for 1980s hardware and for modern hardware. I know where the bottlenecks are and there is no issue whatsoever keeping up with the graphical display. When a terminal is slow it's a design issue.
Well, atleast the CPU will be free for doing other things by offloading some work to GPU.
> What is the advantage of GPU acceleration for a terminal emulator?
I haven't found any. Seems like it's just a fashion accessory for $THIS_YEAR.
I haven't found any. Seems like it's just a fashion accessory for $THIS_YEAR.
I don’t think there is anything wrong with that.
Also building a GPU accelerated terminal emulator is probably a brilliant project to learn loads of low level skills.
Also building a GPU accelerated terminal emulator is probably a brilliant project to learn loads of low level skills.
I switched from Alacritty to WezTerm a few months ago. I cannot overemphasize what a healthier project/community this is. The developer is surprisingly responsive and seems to be a good guy. Join the Matrix room (linked from the GitHub repo readme) if you have any questions.
I don't know anything about Alacritty.
What is unhealthy about the project?
What is unhealthy about the project?
I don't want to speak uncharitably of Alacritty, which was my daily driver for a couple years. It's FOSS, the developers don't owe me/us anything, etc. And I never contributed (though I tried once or twice and was summarily rejected).
It's better to focus on the positives of other projects. I will say, on a more specific note, that if you use macOS and have been frustrated with certain longstanding issues in Alacritty, you're in for a pleasant surprise with WezTerm.
It's better to focus on the positives of other projects. I will say, on a more specific note, that if you use macOS and have been frustrated with certain longstanding issues in Alacritty, you're in for a pleasant surprise with WezTerm.
I see. Poor MacOS support isn't indicative of an unhealthy project, but I can see why MacOS users might want to avoid such projects.
Support for macOS is not poor—though it could go that way eventually. It would be more accurate to say that Alacritty qua macOS software is now almost unmaintained. If you encounter a problem, don't hold your breath; deal with it in your own fork. (I did this for many months.)
The lead developer responding like this to bug reports is a good indicator.
https://github.com/alacritty/alacritty/issues/1561
https://github.com/alacritty/alacritty/issues/1561
I side with the lead developer. The "reporter" clearly don't use any nuances and think that his opinions are "aligned with the general public".
"Green should look like green and not yellow by default" is not an entitled opinion, and definitely shouldn't be met with the hostility it was.
The lead explains in more details
> The initial issue description basically said "the colors are wrong" with no description of what specifically is wrong, what you've tried (if anything), and a non-representative screenshot where your "expected" results are being skewed by the non-focused window dimming. You put the onus on us to engage and poke at your problem to find out what you actually want.
> The defaults don't match your urxvt installation, and because they don't match your previous terminal, you insist they are somehow wrong.
> "Aparently the defaults are poor" - attacking a decision based on your personal bias without consideration for why things are the way they are or that others may actually enjoy the defaults
> "I think it would be a good idea to copy the default colors from urxvt." - this is basically saying the same as above but with more emphasis on expecting the project to change for and accomodate you.
> "Should I open an issue to review the default colors?" - A harmless question by itself, but with context from previous comments, this is again expecting the project to change for you.
> The initial issue description basically said "the colors are wrong" with no description of what specifically is wrong, what you've tried (if anything), and a non-representative screenshot where your "expected" results are being skewed by the non-focused window dimming. You put the onus on us to engage and poke at your problem to find out what you actually want.
> The defaults don't match your urxvt installation, and because they don't match your previous terminal, you insist they are somehow wrong.
> "Aparently the defaults are poor" - attacking a decision based on your personal bias without consideration for why things are the way they are or that others may actually enjoy the defaults
> "I think it would be a good idea to copy the default colors from urxvt." - this is basically saying the same as above but with more emphasis on expecting the project to change for and accomodate you.
> "Should I open an issue to review the default colors?" - A harmless question by itself, but with context from previous comments, this is again expecting the project to change for you.
https://github.com/alacritty/alacritty/issues/1561#issuecomm... describes why the maintainer thinks the opinion was entitled. I don't think your description is accurate.
You mean the original developer? When was he last involved? The problems I ran into, and saw other people experiencing, related to more recent maintainers. I'm not sure it's helpful to keep dredging up old issues like this.
There was a thread about this roughly two weeks ago:
https://news.ycombinator.com/item?id=29349240
https://news.ycombinator.com/item?id=29349240
Thanks. The OP of the link relates to the high bar that Alacritty has for contributions.
I can understand this helps to keep Alacritty manageable. Leading projects sometimes involves saying no to features, as they do: https://github.com/alacritty/alacritty/issues/1309
That's just a tradeoff.
I can understand this helps to keep Alacritty manageable. Leading projects sometimes involves saying no to features, as they do: https://github.com/alacritty/alacritty/issues/1309
That's just a tradeoff.
I chose WezTerm because Altacritty is optimized for dumping large amounts of text to the terminal which is something I don't care much about. There are a lot of other terminal emulators that have lower latency for just normal typing, etc. Wezterm, kitty, zutty, uxterm, xterm.
Same. Mostly because I've read in many places that it is the fastest, despite not being GPU-accelerated. Wonder if someone can chime in and say if it's true. What I want, in order of importance, is good font rendering (Kitty, for instance, doesn't have the best), speed/responsiveness, and maybe Sixel support. Not interested in tabs at all.
I absolutely adore foot. I find its got all the features I actually want and use and nothing more. Its also easy to build and package. The only problem I have with it is when I'm using xorg I can't have it :(
Any Kitty users tried this? I’ve found Kitty to be exemplary so would be curious if anyone finds this an improvement?
I actually went alacratty => wezterm => kitty. I tried and liked wezterm for a while after alacratty, but after some version, it did something that caused all heavy GPU-using programs(like firefox) to slow down severely, or seize up or something, on multiple computers, after running for more than 24-48 hrs. I had been using the default config, except for my preferred font (the VGA font from https://int10h.org/oldschool-pc-fonts/fontlist/). I unfortunately did not have the time to bisect the options or figure out what exactly was going on, but switching to kitty, cleared up the issue.
I tried looking for a bug reports that matched my symptoms, but found none, and I really did not have the the know-how to diagnose GPU issues, but I'd love to try to help isolate the bug.
It honestly felt like a GPU infinite loop, or GPU OOM issue, because the main CPU was idle, but input and redraws on FF would eventually seize up.
I tried looking for a bug reports that matched my symptoms, but found none, and I really did not have the the know-how to diagnose GPU issues, but I'd love to try to help isolate the bug.
It honestly felt like a GPU infinite loop, or GPU OOM issue, because the main CPU was idle, but input and redraws on FF would eventually seize up.
I'm currently using termite. However Termite is no longe rmaintened. I plan to switch to alacratty or kitty. I'm wondering: Why did you choose kitty / wezterm over alacratty?
I started my journey with urxvt, and was seeking better rendering. Alacratty+tmux (with urxvt keybindings) was working well, and I learned to deal with the "auto resize" when moving windows between screens with different DPIs. I tried wezterm from a thread here, and the emoji rendering (a feature I've never relied on before, but nice to have) was better than on alacritty, and stuck with it. Learned to work with it's tabs + extra features, and stuck around for a bit. Liked that I could run the same terminal on Linux + Windows.
I liked wezterm, and it's flexibility, but like I mentioned that GPU bug was killer. I'd have love to help diagnose it, but I'm not sure where to begin. Kitty did most of what wezterm did, and I just stuck with it since. The timestamp on my kitty.conf is 2021-09-11, so it looks like that was the last time I had to tweak it after starting to use it, and I'm happy thus far.
I liked wezterm, and it's flexibility, but like I mentioned that GPU bug was killer. I'd have love to help diagnose it, but I'm not sure where to begin. Kitty did most of what wezterm did, and I just stuck with it since. The timestamp on my kitty.conf is 2021-09-11, so it looks like that was the last time I had to tweak it after starting to use it, and I'm happy thus far.
Wezterm has more features and for me being written entirely in Rust made it easy to send a bug fix patch. Kitty however seems to have a noticeably lower latency at least on mac.
I agree… I’m using Kitty and am very satisfied with it. I haven’t tried this one yet, though.
I found kitty faster on my system
As I said in another comment, I tried to use WezTerm a few days before switching back to Kitty for the poorer performance. I ran a few benchmarks and discovered if Kitty takes 1, Konsole takes 2 and WezTerm takes 2.5 (using the same fonts and font sizes.)
Looks promising! It seems fast, and I like the idea of a more stripped-down terminal app. I use iTerm, which simply has too much.
That said, it has a long way to go to beat iTerm for me. Seems more like an early alpha than a finished product?
For example, out of the box, it does not support European keyboard layouts:
* I cannot type "|" because the Alt key doesn't seem to work, not even with "send_composed_key_when_left_alt_is_pressed" enabled.
* Similarly, I also cannot use Super+"[" or Super+"]" to switch tabs, because on my keyboard, "[" and "]" are Alt-7 and Alt-8 respectively.
* While Cmd+"-" works to reduce the font size, Cmd+"=" does nothing, probably because "=" requires shift on my keyboard.
There's no GUI as such. For example, attempting to close a tab brings up a text mode dialog inside the terminal window, which feels strange and antiquated. Same with the find function; the latter is more concerning, since you don't get access to normal UI functions such as like selection.
Right-clicking also doesn't bring up the standard macOS context menu.
That said, it has a long way to go to beat iTerm for me. Seems more like an early alpha than a finished product?
For example, out of the box, it does not support European keyboard layouts:
* I cannot type "|" because the Alt key doesn't seem to work, not even with "send_composed_key_when_left_alt_is_pressed" enabled.
* Similarly, I also cannot use Super+"[" or Super+"]" to switch tabs, because on my keyboard, "[" and "]" are Alt-7 and Alt-8 respectively.
* While Cmd+"-" works to reduce the font size, Cmd+"=" does nothing, probably because "=" requires shift on my keyboard.
There's no GUI as such. For example, attempting to close a tab brings up a text mode dialog inside the terminal window, which feels strange and antiquated. Same with the find function; the latter is more concerning, since you don't get access to normal UI functions such as like selection.
Right-clicking also doesn't bring up the standard macOS context menu.
> It seems fast
I was intrigued by WezTerm, but after a few tests I realized that it was ~3 times slower than Kitty (my current terminal of choice) in printing random stuff on the screen, so I decided to stick with the latter. It is a pity, because I love the way WezTerm can be configured!
I was intrigued by WezTerm, but after a few tests I realized that it was ~3 times slower than Kitty (my current terminal of choice) in printing random stuff on the screen, so I decided to stick with the latter. It is a pity, because I love the way WezTerm can be configured!
Serious question: how much does it benefit to accelerate terminal on GPU? Is it really worth the effort?
Lots and yes.
The effort has already been made, there are lots of very capable and stable GPU accelerated terminal emulators available.
The speed boost makes them worth using. If you've ever accidentally returned a minified file from grep you'll know most terminal emulators grind to a halt.
The effort has already been made, there are lots of very capable and stable GPU accelerated terminal emulators available.
The speed boost makes them worth using. If you've ever accidentally returned a minified file from grep you'll know most terminal emulators grind to a halt.
> If you've ever accidentally returned a minified file from grep you'll know most terminal emulators grind to a halt.
This doesn't sound like a rendering issue, though.
Many text editors can't cope with extremely long lines either, terminal-based or not. That happens when they store lines as flat buffers without any algorithmic optimization.
GPU has the potential to speed up terminal rendering a little bit, but in the absence of antialiasing (cache glyphs with the right background color) you can probably achieve 60fps on a 4K screen full of glyphs. It's just memcpy(), a 4K screen is about 32MB of data. 32MB * 60 = 1.8GB, today's memory is much faster.
This doesn't sound like a rendering issue, though.
Many text editors can't cope with extremely long lines either, terminal-based or not. That happens when they store lines as flat buffers without any algorithmic optimization.
GPU has the potential to speed up terminal rendering a little bit, but in the absence of antialiasing (cache glyphs with the right background color) you can probably achieve 60fps on a 4K screen full of glyphs. It's just memcpy(), a 4K screen is about 32MB of data. 32MB * 60 = 1.8GB, today's memory is much faster.
Yeah, it's not a rendering issue, it's an architectural issue. It's people writing terminals imperatively: Text comes in from the app, and they trigger a set of updates to state ending in rendering the text straight to screen.
The first fix is that if you want a maximally responsive terminal you'll want to decouple input process, output processing and rendering into separate threads - this was established already in the 1980's with the Amiga console handling. E.g. any modern terminal that struggles with handles Ctrl+C is an example of why you want this.
The second fix is to "render" into a text buffer and have the renderer look at how many lines it has to scroll and rerender each time it starts a rendering pass, blit whatever is still intact, and then render what is left, rather than process character by character. When the producer overwhelms the terminal in that scenario you end up scrolling multiple lines at once. This will be jerky if the rendering is very slow but it'd need to be very slow for it to be noticeable, and even then it's generally much preferable to the terminal lagging behind.
And when you then have a process barfing gigabytes of data to the terminal, most text will not be visible more than a frame on a terminal that insists on showing it all, and won't be visible at all on one that updates faster than the frame rate anyway, and the fix above will simply entirely skip rendering most of the text. As long as your terminal can render a page full of text at a high enough frame rate it will be seamless.
With respect to the actual rendering, glyph size tends to be small enough that there's not that much to be gained in terms of acceleration other than a rectangular copy to accelerate scrolling, which is mostly relevant when the terminal doesn't handle huge chunks of data being barfed. The main exception being if you use shaders. E.g. on my personal copy of kitty I experimented with a shader to add a "glow" to characters to make it more legible on semi-transparent windows. But I've applied similar hacks to purely CPU-rendered terminals (created "poor mans outlines" by simply rendering the text multiple times at slight offsets with different colours) and good CPU-rendered terminals can do even that more than fast enough.
The first fix is that if you want a maximally responsive terminal you'll want to decouple input process, output processing and rendering into separate threads - this was established already in the 1980's with the Amiga console handling. E.g. any modern terminal that struggles with handles Ctrl+C is an example of why you want this.
The second fix is to "render" into a text buffer and have the renderer look at how many lines it has to scroll and rerender each time it starts a rendering pass, blit whatever is still intact, and then render what is left, rather than process character by character. When the producer overwhelms the terminal in that scenario you end up scrolling multiple lines at once. This will be jerky if the rendering is very slow but it'd need to be very slow for it to be noticeable, and even then it's generally much preferable to the terminal lagging behind.
And when you then have a process barfing gigabytes of data to the terminal, most text will not be visible more than a frame on a terminal that insists on showing it all, and won't be visible at all on one that updates faster than the frame rate anyway, and the fix above will simply entirely skip rendering most of the text. As long as your terminal can render a page full of text at a high enough frame rate it will be seamless.
With respect to the actual rendering, glyph size tends to be small enough that there's not that much to be gained in terms of acceleration other than a rectangular copy to accelerate scrolling, which is mostly relevant when the terminal doesn't handle huge chunks of data being barfed. The main exception being if you use shaders. E.g. on my personal copy of kitty I experimented with a shader to add a "glow" to characters to make it more legible on semi-transparent windows. But I've applied similar hacks to purely CPU-rendered terminals (created "poor mans outlines" by simply rendering the text multiple times at slight offsets with different colours) and good CPU-rendered terminals can do even that more than fast enough.
I had to stop using windows terminal because it has GPU acceleration, which ate up GPU memory I needed for pytorch/GPU accelerated neural nets. When doing regular development work, it’s another story.
Hmm, I do wonder why would a terminal use so much GPU ressources. Maybe it gets better with reducing scrollback? Seems extreme anyway.
Caching glyphs, into a texture. Then they don't need to be re-rendered and can just be copied.
Let's say 512 glyphs should be a pretty conservative upper bound for normal usage. Just pulling this number out of thin air, but I assume it's reasonable. normal glyphs should be about 100-400 pixels (20x20), for 8-bit grayscale that means 2MB of texture cache should be plenty for normal use.
Multiply by 3 if you want to support subpixel rendered glyphs, that still leaves us with a very comfortable upper bound of 6MB.
Note that optionally you can keep the glyphs on CPU memory as opposed to GPU memory, and just push the needed data in a streaming fashion to the GPU for each frame. Since there is no texture filtering for fonts (GL_NEAREST for OpenGL), no need to create a font-atlas and no needed for complicated font atlas packing. Glyph data can be allocated in a simple 1-dimensional space, and streaming a glyph's data up is just a matter of bumping a pointer and a single memcpy.
Multiply by 3 if you want to support subpixel rendered glyphs, that still leaves us with a very comfortable upper bound of 6MB.
Note that optionally you can keep the glyphs on CPU memory as opposed to GPU memory, and just push the needed data in a streaming fashion to the GPU for each frame. Since there is no texture filtering for fonts (GL_NEAREST for OpenGL), no need to create a font-atlas and no needed for complicated font atlas packing. Glyph data can be allocated in a simple 1-dimensional space, and streaming a glyph's data up is just a matter of bumping a pointer and a single memcpy.
> 512 glyphs
You're forgetting about colored glyphs. Although a fragment shader should be able to deal with that, shouldn't it?
You're forgetting about colored glyphs. Although a fragment shader should be able to deal with that, shouldn't it?
One typical approach is that glyphs are rastered (e.g. by Freetype) as 8-bit grayvalues. Basically Alpha channel if you will. You can add RGB color channels from there and blit RGBA with Alpha blending enabled. Not sure if there are finer points to consider here, it's what I've always done and it works for me.
Perhaps less used, and becoming less and less relevant as higher DPI screens are becoming common: With subpixel-rasterized glyphs, you get RGB channel output from the font rasterizer instead of the Alpha channel. These are properly balanced values that are arranged to appear as "white" pixels to most viewers - the idea is that this increases the perceived horizontal resolution by a factor of up to 3. Not sure how to add color to these, maybe just multiplying each component with the respective component of a text color RGB value would work.
In any case, if you have a 4K screen full of flickering glyphs that need to be of varying colors and backgrounds, that might indeed indicate a GPU approach. On the other hand, nobody is getting value of flickering glyphs, so CPU rasterization with some rate limiting would be just as good in these situations, practically speaking.
Perhaps less used, and becoming less and less relevant as higher DPI screens are becoming common: With subpixel-rasterized glyphs, you get RGB channel output from the font rasterizer instead of the Alpha channel. These are properly balanced values that are arranged to appear as "white" pixels to most viewers - the idea is that this increases the perceived horizontal resolution by a factor of up to 3. Not sure how to add color to these, maybe just multiplying each component with the respective component of a text color RGB value would work.
In any case, if you have a 4K screen full of flickering glyphs that need to be of varying colors and backgrounds, that might indeed indicate a GPU approach. On the other hand, nobody is getting value of flickering glyphs, so CPU rasterization with some rate limiting would be just as good in these situations, practically speaking.
I replaced iTerm with Wezterm 6 months back. It feels lot more lighter and faster. As SRE I need to work a lot on terminal. I tried Alacritty for a while but Alacritty not supporting horizontal and vertical splits was a major setback for me.
I moved from alacritty to wezterm for the same reason. While alacritty definitely feels snappier, the features of wezterm more than compensate for me the speed difference.
https://wezfurlong.org/wezterm/config/keys.html
Happy to see that it includes support for clearing scroll back and that the default key binding for that is the same as it is in the Terminal.app that comes with macOS; Super + K.
I just might give this Wezterm a go on my own computers. And since Wezterm is cross platform with support for all three computer platforms that I use; macOS, Linux, FreeBSD, I may finally have a coherent terminal emulator on all of these operating systems. Provided that I find it a comfortable application to use :)
Happy to see that it includes support for clearing scroll back and that the default key binding for that is the same as it is in the Terminal.app that comes with macOS; Super + K.
I just might give this Wezterm a go on my own computers. And since Wezterm is cross platform with support for all three computer platforms that I use; macOS, Linux, FreeBSD, I may finally have a coherent terminal emulator on all of these operating systems. Provided that I find it a comfortable application to use :)
I've tried it a little bit now with a custom config, and have it almost the same way that Terminal.app is on my system now.
My configuration for Wezterm at the moment:
- I set the color scheme to Afterglow, which is a lot like the default color scheme of Terminal.app.
- I downloaded the SF Mono font from https://developer.apple.com/fonts/ so that it is available to Wezterm, and I defined the font size to be the one that I prefer. This is the font that Terminal.app uses by default but I guess in Terminal.app this font is actually bundled inside of the application because the font did not show up in Font Book and Wezterm couldn't find it either until I downloaded it from said link and installed it even though Terminal.app is already using it.
- I configured Command + K to clear not just scrollback (Wezterm default), but to also clear the viewport like Terminal.app does.
- I turned off tabs.
The are only a few things at the moment that I dislike about Wezterm:
1. The title of each window is simply "Wezterm", and tabs provide additional info but since I am not using tabs and the windows still are titled just "Wezterm" I am missing having the title of the Window be set to things like the current working directory and the currently running commands the way that Terminal.app does.
2. https://wezfurlong.org/wezterm/faq.html says that WezTerm version 20200620-160318-e00b076c and newer will automatically set $LANG appropriately on macOS, but I am using version 20211205-192649-672c1cc1 installed from homebrew on macOS Monterey 12.0.1, and it is not setting $LANG for me.
3. I am unable to input characters like æ, ø, å the way that I usually do which is Option + ', Option + o, and Option + a respectively on macOS with the macOS input language set to US. I don't know if this is a consequence of $LANG not being set that I mentioned above or if it's due to the application perhaps not using the operating system provided input methods.
4. When I close the last window the application closes, which is counter to how most applications on macOS work by default, and counter to how Terminal.app works by default. I often close terminals including the last terminal and then later like to tab back to Terminal.app and open a new window. With Wezterm I have to relaunch the application before I can open a new window after I have closed the last window.
Is there a way to make the Wezterm application stay running on macOS when I close the last window?
My configuration for Wezterm at the moment:
- I set the color scheme to Afterglow, which is a lot like the default color scheme of Terminal.app.
- I downloaded the SF Mono font from https://developer.apple.com/fonts/ so that it is available to Wezterm, and I defined the font size to be the one that I prefer. This is the font that Terminal.app uses by default but I guess in Terminal.app this font is actually bundled inside of the application because the font did not show up in Font Book and Wezterm couldn't find it either until I downloaded it from said link and installed it even though Terminal.app is already using it.
- I configured Command + K to clear not just scrollback (Wezterm default), but to also clear the viewport like Terminal.app does.
- I turned off tabs.
The are only a few things at the moment that I dislike about Wezterm:
1. The title of each window is simply "Wezterm", and tabs provide additional info but since I am not using tabs and the windows still are titled just "Wezterm" I am missing having the title of the Window be set to things like the current working directory and the currently running commands the way that Terminal.app does.
2. https://wezfurlong.org/wezterm/faq.html says that WezTerm version 20200620-160318-e00b076c and newer will automatically set $LANG appropriately on macOS, but I am using version 20211205-192649-672c1cc1 installed from homebrew on macOS Monterey 12.0.1, and it is not setting $LANG for me.
3. I am unable to input characters like æ, ø, å the way that I usually do which is Option + ', Option + o, and Option + a respectively on macOS with the macOS input language set to US. I don't know if this is a consequence of $LANG not being set that I mentioned above or if it's due to the application perhaps not using the operating system provided input methods.
4. When I close the last window the application closes, which is counter to how most applications on macOS work by default, and counter to how Terminal.app works by default. I often close terminals including the last terminal and then later like to tab back to Terminal.app and open a new window. With Wezterm I have to relaunch the application before I can open a new window after I have closed the last window.
Is there a way to make the Wezterm application stay running on macOS when I close the last window?
Thanks for giving it a try! I recently moved house and have limited mental bandwidth while I'm setting up the new place--since I don't want to drop the ball on resolving these, I'd appreciate it if you would file separate GitHub issues for those items you'd like to follow up on so that I can find and work through them as time allows!
1. Tab titles are set using escape sequences, typically OSC 0, 1 or 2. You can find examples for configuring shells to emit those online, for example: https://superuser.com/questions/84710/window-title-in-bash Terminal.app defaults to snooping the local process list to set titles; wezterm doesn't do this as the terminal may not be local (eg: could be remote ssh or multiplexer session) and using escape sequences gives definitive control.
2. I'd love to troubleshoot this with you; please file an issue!
3. There are currently a set of open and related IME/macOS issues at the moment. You may want to try looking at https://wezfurlong.org/wezterm/config/keys.html#macos-left-a... and fiddle with those options, but you may be left disappointed until the root of those issues are resolved!
4. This is a consequence of being a cross platform program rather than a macOS-first application. There isn't a way to keep the GUI running without any Windows today. Perhaps in the future?
1. Tab titles are set using escape sequences, typically OSC 0, 1 or 2. You can find examples for configuring shells to emit those online, for example: https://superuser.com/questions/84710/window-title-in-bash Terminal.app defaults to snooping the local process list to set titles; wezterm doesn't do this as the terminal may not be local (eg: could be remote ssh or multiplexer session) and using escape sequences gives definitive control.
2. I'd love to troubleshoot this with you; please file an issue!
3. There are currently a set of open and related IME/macOS issues at the moment. You may want to try looking at https://wezfurlong.org/wezterm/config/keys.html#macos-left-a... and fiddle with those options, but you may be left disappointed until the root of those issues are resolved!
4. This is a consequence of being a cross platform program rather than a macOS-first application. There isn't a way to keep the GUI running without any Windows today. Perhaps in the future?
Wezterm seems to have its own take on multiplexing built-in. And it supports terminal graphics (But not both at the same time for the moment).
Is there a way to use Emacs on Wezterm using all the same keystrokes as Emacs GUI? (e.g., Meta/Control/Shift + Arrows, Control + Backspace, etc.).
Iterm2 is GPU accelerated if I remember correctly as well isn't it? Seems incredibly fast for everything I throw at it.
Yes, optionally: https://gitlab.com/gnachman/iterm2/-/wikis/Metal-Renderer
It's just a tick box in the app, I think it's defaulted to on for a few years now
Yes, I included the link because it has information on options that disable it, like transparent windows, ligatures, etc.
Don't get me started on ligatures - now they make iterm run like dog! Otherwise I'm quite happy with iterm - but also welcome alternatives (except those ones I've seen based on electron).
Criticism is cheap, so I wanna first say that this looks like good work! I am somewhat put off when people don't list dependencies anywhere, but rather hide them away in some script (or, in this case, scripts + Cargo manifest).
For interest's sake, I did a "time find /usr" (ran it first once so it's all in memory) in both Terminal.app on the Mac and then in WezTerm
Terminal.app: 6.174s Wezterm: 7.150s
Terminal.app: 6.174s Wezterm: 7.150s
i've been using wezterm for six months or so as my daily driver on three operating systems and i love it.
Whenever I see a terminal emulator, my first question is whether it can do the VT-100 torture test.
As much as I like the idea, I feel it's not very relevant to most peoples use.
I'm writing my own editor. A little toy terminal I have sitting around can run it with just support for a handful of escape codes. To see how broken the terminal still was I fired up emacs. It took next to nothing to make emacs work fine (while having TERM set to vt100). I've seen very little software try to use more than a very tiny subset of vt100.
I'm writing my own editor. A little toy terminal I have sitting around can run it with just support for a handful of escape codes. To see how broken the terminal still was I fired up emacs. It took next to nothing to make emacs work fine (while having TERM set to vt100). I've seen very little software try to use more than a very tiny subset of vt100.
> I've seen very little software try to use more than a very tiny subset of vt100.
One thing I really love in command-line interfaces is to use the double-wide, double-height characters VT-100 offers.
Even xterm supports that.
Another thing I'd love to have is smooth scrolling. Physical terminals could do that and, when a fast connection was 2400, it even made sense. In newish terminals, I'd love to have that, with progressive fallbacks to faster scrolling as the terminal backlog fills up. If your ls -l is a few lines short, it just feels nicer.
One thing I really love in command-line interfaces is to use the double-wide, double-height characters VT-100 offers.
Even xterm supports that.
Another thing I'd love to have is smooth scrolling. Physical terminals could do that and, when a fast connection was 2400, it even made sense. In newish terminals, I'd love to have that, with progressive fallbacks to faster scrolling as the terminal backlog fills up. If your ls -l is a few lines short, it just feels nicer.
Do you have any examples of command line apps that actually uses the double-wide/double-height characters? I'm aware of them, but I don't think I've ever seen it used. I'm always curious to see good uses of terminal capabilities that are under-used.
Smooth-scrolling I think is generally ignored because as you can see from this thread people are obsessed with throughput over all else, and assumes the raw rendering is the bottleneck, but in my experience with terminals it rarely is - few terminals are in any way optimised for speed.
Elsewhere in this thread I've advocated for a pretty old-school solution to the throughput and responsiveness issue: Decouple IO and rendering into separate threads. The Amiga did that in 1985, and I have no reason to think it was a particular revelation back then - just necessary to produce something that'd become unresponsive too easily. That split then makes it easy to have the renderer batch updates when there is much waiting. E.g. if so much text comes that it'd scroll two lines in a frame there's no reason to scroll one line twice.
If more terminals first did that, then the converse of doing what you propose with defaulting to smooth scrolling when there is little in the buffer, and falling back on faster scrolling would also be "trivial": Just peek at the buffer of how many lines of output are waiting, and adjust scroll speed accordingly. The only difficulty, I suspect, would be to find the optimal rate of change to prevent it from looking jerky.
Smooth-scrolling I think is generally ignored because as you can see from this thread people are obsessed with throughput over all else, and assumes the raw rendering is the bottleneck, but in my experience with terminals it rarely is - few terminals are in any way optimised for speed.
Elsewhere in this thread I've advocated for a pretty old-school solution to the throughput and responsiveness issue: Decouple IO and rendering into separate threads. The Amiga did that in 1985, and I have no reason to think it was a particular revelation back then - just necessary to produce something that'd become unresponsive too easily. That split then makes it easy to have the renderer batch updates when there is much waiting. E.g. if so much text comes that it'd scroll two lines in a frame there's no reason to scroll one line twice.
If more terminals first did that, then the converse of doing what you propose with defaulting to smooth scrolling when there is little in the buffer, and falling back on faster scrolling would also be "trivial": Just peek at the buffer of how many lines of output are waiting, and adjust scroll speed accordingly. The only difficulty, I suspect, would be to find the optimal rate of change to prevent it from looking jerky.
I will give Wezterm a try wheb tmux CC mode(show tmux panel as native window) support is done
> WezTerm is a GPU-accelerated cross-platform terminal emulator and multiplexer written by @wez and implemented in Rust
This is great. I’ve been looking for a GPU-accelerated terminal written in Rust and haven’t been able to find one. Glad to see someone finally wrote one.
This is great. I’ve been looking for a GPU-accelerated terminal written in Rust and haven’t been able to find one. Glad to see someone finally wrote one.
Alacritty?
Oh nice, thanks! This wasn’t humor, I really was looking for a modern performant terminal written in Rust. I wasn’t aware of Alacritty, I’ll definitely check it out :)
I think that was humor, because there seem to be at least three GPU-accelerated terminal emulators written in Rust.
What font is being used in the screenshot?
Wow… and I thought Iterm(2) is the best terminal on MacOS
https://news.ycombinator.com/item?id=26633708 (181 points/8 months ago/128 comments)