Modernizing the OpenBSD Console(cambus.net)
cambus.net
Modernizing the OpenBSD Console
https://www.cambus.net/modernizing-the-openbsd-console/
61 comments
At least on pictures HP 9000 / HP-UX "boot scroll" looks the nicest: https://blog.pizzabox.computer/img/hp712/hpux-init.png https://upload.wikimedia.org/wikipedia/commons/thumb/8/87/HP...
If I recall correctly, there was an article on HN not so long ago about Carmack discovering BSD. Edit: found it - https://news.ycombinator.com/item?id=23224584
s/discovering/committing to/
Carmack has been aware of and actively involved in UNIX systems for a long time now. This isn't a recent discovery. However it has been recently noticed that he's committing to OpenBSD.
Carmack has been aware of and actively involved in UNIX systems for a long time now. This isn't a recent discovery. However it has been recently noticed that he's committing to OpenBSD.
Contributing to OpenBSD, not committing. He is not a committer.. yet.
Apologies, I hadn't realised there were a distinction. What's the difference?
I guess it's about having a direct commit access to their CVS tree.
Precisely. OpenBSD developers have committed John's diffs.
> you'd think Apple would be the first one to fix this…
To be fair, they have one of the very few terminal apps that support double-width/height characters and that passes the VT torture test with flying colors.
To be fair, they have one of the very few terminal apps that support double-width/height characters and that passes the VT torture test with flying colors.
I mean, I would think this would just be using a larger font size for their bitmap, no?
Apple's terminal doesn't seem to use a specific bitmap codepath - it renders the characters at a doubled size for DECDHL and it looks like it stretches the geometry of the font horizontally for DECDWL.
What really disappoints me is that, with all those pixels, and bits per pixel we now have, it still can't have proper bold, italic, underline, overline, Tek graphics, sixels, and smooth scrolling like any ancient terminal made by any well regarded hardware company.
indeed :O
here are the VT100 ansi escape codes: https://espterm.github.io/docs/VT100%20escape%20codes.html
Bold, Underline, Double height, double width, etc... are all there. VT100, from 1978.
here are the VT100 ansi escape codes: https://espterm.github.io/docs/VT100%20escape%20codes.html
Bold, Underline, Double height, double width, etc... are all there. VT100, from 1978.
ncurses gives you all of that in a much easier-to-use fashion.
True, but not really relevant in this context. The fact is that graphical consoles could support all that because, in reality, there was a time they actually did - they were just independent embedded computers connected to the host via a serial port.
In my dreams, when switching to Tek mode, the framebuffer would have a dark green background, the lines would have a brighter green, and drawing would push the HDR for the brightest green it'd be able to do (because to store a vector in a Tektronix DVST, you flash it with a high energy beam). Erasing the screen would flash it light green and there would be no scrolling.
In my dreams, when switching to Tek mode, the framebuffer would have a dark green background, the lines would have a brighter green, and drawing would push the HDR for the brightest green it'd be able to do (because to store a vector in a Tektronix DVST, you flash it with a high energy beam). Erasing the screen would flash it light green and there would be no scrolling.
"Optimize character rendering in 32bpp mode (John Carmack, June 2020)"
Awesome
Awesome
It's the sort of optimization that one can read about in games and DirectX programming books.
A better optimization comes with the realization, as MacOS people had many years ago, that copying from system memory to video memory is slow. So one instead pre-loads the glyphs into off-screen video memory, possibly on-demand, and then uses the display hardware to blit them into the on-screen part.
Of course, this requires that code know about blitter hardware, which does not necessarily exist. If one is doing an in-kernel terminal emulator, such as wscons in OpenBSD here, then one can rely upon the fact that the EFI Graphics Output Protocol provides a Blt() function that is supposed to abstract away the hardware. Indeed, EFI GOP can in theory not provide direct framebuffer access in the first place and require that everything be blitted (PixelBltOnly).
We who write application-mode terminal emulators are not so fortunate. (-:
A better optimization comes with the realization, as MacOS people had many years ago, that copying from system memory to video memory is slow. So one instead pre-loads the glyphs into off-screen video memory, possibly on-demand, and then uses the display hardware to blit them into the on-screen part.
Of course, this requires that code know about blitter hardware, which does not necessarily exist. If one is doing an in-kernel terminal emulator, such as wscons in OpenBSD here, then one can rely upon the fact that the EFI Graphics Output Protocol provides a Blt() function that is supposed to abstract away the hardware. Indeed, EFI GOP can in theory not provide direct framebuffer access in the first place and require that everything be blitted (PixelBltOnly).
We who write application-mode terminal emulators are not so fortunate. (-:
GOP functions aren't available after ExitBootServices() has been called, so the kernel doesn't get to take advantage of Blit() - the assumption is that the kernel will have a native driver for the hardware. Additionally, you can't really use off-screen graphics RAM in the EFI environment because GOP doesn't tell you how much video RAM is actually available, only the size of the framebuffer. You can guess which PCI device is associated with the framebuffer and look at the size of the BAR, but you have no way of knowing which addresses the EFI driver is using for its own purposes. Apple have enough control over their platform that they could possibly make this work, but it's tough when you're dealing with a vendor-provided driver on an add-in card.
Hah! I've been doing too much load-time stuff, obviously. (-:
But if one is looking at PCI device IDs, then one is already at the level of the code having detailed knowledge of the specific hardware (even if only via some "miniport"-like system) and the point stands.
But if one is looking at PCI device IDs, then one is already at the level of the code having detailed knowledge of the specific hardware (even if only via some "miniport"-like system) and the point stands.
Oh, you can look at the device that the GOP instance is bound to and get the PCI device from there. At that point you just walk the BARs looking for one that's mapped in a region that covers the framebuffer address that GOP gives you. This doesn't require you to know anything about the device in question, and most of the time you'd probably get lucky by throwing glyphs in space that exists outside the framebuffer and blitting them onto the screen - up until you find a driver that implements blt() by using the 3D engine and mapping some buffer objects on top of the space you're trying to use.
The code will still have to know about blitter hardware, as I said.
Any design that depends from "you'd probably get lucky" is not a wise one. (-:
Any design that depends from "you'd probably get lucky" is not a wise one. (-:
Will it support "true colors" (16M colors)[1][2]?
[1] https://github.com/termstandard/colors
[2] https://gist.github.com/XVilka/8346728
[1] https://github.com/termstandard/colors
[2] https://gist.github.com/XVilka/8346728
The article uses 32 bits per pixel to illustrate the growth in performance requirements:
> In framebuffer however, to fill a 4K UHD-1 (3840x2160) screen in 32bpp mode we need to send 3840 * 2160 * 4 bytes of data, which is 33177600 bytes (approximately 33 MB).
but I don't know if that is in fact the definition a "framebuffer console" versus a terminal that uses character addressing.
That's a different issue that actually supporting 32-bit ANSI codes, but I think it's an indication of the scope of a an effort to update the console implementation.
> In framebuffer however, to fill a 4K UHD-1 (3840x2160) screen in 32bpp mode we need to send 3840 * 2160 * 4 bytes of data, which is 33177600 bytes (approximately 33 MB).
but I don't know if that is in fact the definition a "framebuffer console" versus a terminal that uses character addressing.
That's a different issue that actually supporting 32-bit ANSI codes, but I think it's an indication of the scope of a an effort to update the console implementation.
Some of the code linked in the article seems to emulate the VGA style text console (1 value for char value, 1 value for color and attributes) and render it onto a 32bit framebuffer.
The diffs that are attributed to Carmack seem to deal with this. scr->rs_bs[off].uc and scr->rs_bs[off].attr appear to represent the gylph and attributes/color respectively.
So I would expect that would severely limit the amount of colors available to the input buffer. OTOH I just found the declaration in the code and the attributes are 32 bits wide which is more than the 8 that VGA provides:
Edit: after reading some more it seems like the color values range from 0-7.
The diffs that are attributed to Carmack seem to deal with this. scr->rs_bs[off].uc and scr->rs_bs[off].attr appear to represent the gylph and attributes/color respectively.
So I would expect that would severely limit the amount of colors available to the input buffer. OTOH I just found the declaration in the code and the attributes are 32 bits wide which is more than the 8 that VGA provides:
struct wsdisplay_charcell {
u_int uc;
uint32_t attr;
};
Edit: after reading some more it seems like the color values range from 0-7.
32 bits isn't really enough. One uses 24 bits for RGB, and the remaining 8 bits aren't really enough for the remaining graphics renditions that a modern terminal emulator should support.
Factor in things like the KiTTY underlining system, invisible, the little-recognized new (sic!) ECMA-48:1991 graphic renditions (encircled, framed, and overlined), and strikethrough; and one actually needs 13 to 14 bits (depending from how far one takes the KiTTY underlining system).
Moreover: No-one doing this style of terminal emulator really does Unicode properly. To do so actually involves having a string of one or more code points per cell, not simply exactly one. Some of the terminal emulators based upon external GUIs (e.g. X11) do do that, or approach it. (Although at the same time they take a giant leap backwards from video terminal semantics to paper terminal semantics, so it is a matter of swings and roundabouts. See https://unix.stackexchange.com/a/593613/5132 .)
Factor in things like the KiTTY underlining system, invisible, the little-recognized new (sic!) ECMA-48:1991 graphic renditions (encircled, framed, and overlined), and strikethrough; and one actually needs 13 to 14 bits (depending from how far one takes the KiTTY underlining system).
Moreover: No-one doing this style of terminal emulator really does Unicode properly. To do so actually involves having a string of one or more code points per cell, not simply exactly one. Some of the terminal emulators based upon external GUIs (e.g. X11) do do that, or approach it. (Although at the same time they take a giant leap backwards from video terminal semantics to paper terminal semantics, so it is a matter of swings and roundabouts. See https://unix.stackexchange.com/a/593613/5132 .)
> 32 bits isn't really enough. One uses 24 bits for RGB,
Yes, and as I mentioned you need to divide between FG and BG colors.
All a little moot because it seems to me like the code uses 3 bits for color.
> No-one doing this style of terminal emulator really does Unicode properly
Indeed, both multi-char and multi-codepoint glyphs. I also think the whole "matrix of glyphs" thing breaks down pretty severely when you want BiDi support for instance. The article mentions someone working on UTF-8 support. I wonder what that looks like.
Yes, and as I mentioned you need to divide between FG and BG colors.
All a little moot because it seems to me like the code uses 3 bits for color.
> No-one doing this style of terminal emulator really does Unicode properly
Indeed, both multi-char and multi-codepoint glyphs. I also think the whole "matrix of glyphs" thing breaks down pretty severely when you want BiDi support for instance. The article mentions someone working on UTF-8 support. I wonder what that looks like.
The "I" in "ITU" does not stand for "ANSI". (-:
* https://itu.int/rec/T-REC-T.416
As M. Kochkov will attest, supporting 24-bit RGB (or even 32-bit ARGB) colour involves (a) recognizing and processing the new (sic!) control sequences from the 1990s, and (b) actually supporting having the display hardware in a non-palettized true colour mode (as opposed to using a 256-colour palettized mode and pretending).
The definition of a framebuffer virtual terminal is, rather, that the display hardware is in its graphics mode rather than its text mode. (This was the norm for some architectures, which didn't even have text mode display hardware. But the MDA/Hercules/CGA/EGA/VGA on the IBM PC and compatibles did.) The operating system, that is providing the virtual terminal, draws the glyphs by manipulating pixels, rather than relying upon "character generation" hardware to do it in the video output stage of the adapter.
* https://itu.int/rec/T-REC-T.416
As M. Kochkov will attest, supporting 24-bit RGB (or even 32-bit ARGB) colour involves (a) recognizing and processing the new (sic!) control sequences from the 1990s, and (b) actually supporting having the display hardware in a non-palettized true colour mode (as opposed to using a 256-colour palettized mode and pretending).
The definition of a framebuffer virtual terminal is, rather, that the display hardware is in its graphics mode rather than its text mode. (This was the norm for some architectures, which didn't even have text mode display hardware. But the MDA/Hercules/CGA/EGA/VGA on the IBM PC and compatibles did.) The operating system, that is providing the virtual terminal, draws the glyphs by manipulating pixels, rather than relying upon "character generation" hardware to do it in the video output stage of the adapter.
I expect none of this (text mode, indexed colors, 16 bpp, hardware GDI/2D etc.) actually exists in hardware any more. I expect GPUs emulate all of this through shaders baked into the VBIOS/driver.
I doubt that very much. I cannot see people successfully making a business case to spend development time and money re-implementing MDA/CGA/EGA/VGA text mode, with the concomitant on-going support costs when inevitably the re-implementation misses stuff, for pretty much zero additional revenue. No, this wouldn't be implementable through drivers. It has to work during POST and IPL.
Just curious, but does anyone know a business using OpenBSD in production other than the Pushover App?
I used OpenBSD in production at my previous job (an ISP), in a few different roles. Lots of ISPs use OpenBSD for various purposes.
HSMs from nCipher used to run on top of (a slightly modified, IIRC) OpenBSD. Just as JunOS (the OS on (some) Juniper network devices) runs on FreeBSD, I'm sure there are many other commercial products built on OpenBSD as well -- we just don't hear about them very much.
---
Related (from the same author as this submission): "Why OpenBSD?" [0].
---
[0]: https://www.cambus.net/why-openbsd/
HSMs from nCipher used to run on top of (a slightly modified, IIRC) OpenBSD. Just as JunOS (the OS on (some) Juniper network devices) runs on FreeBSD, I'm sure there are many other commercial products built on OpenBSD as well -- we just don't hear about them very much.
---
Related (from the same author as this submission): "Why OpenBSD?" [0].
---
[0]: https://www.cambus.net/why-openbsd/
An acquaintance is currently building the tech part of his and some friends' start-up, entirely on OpenBSD - PHP-based web front end, back end, Postgres-based database, mail server etc.
Not sure if this counts, but I use it in production in yakdocs.com and yak-mu.com. I love how easy it is to maintain and I love the pf firewall. Hetzner has images for OpenBSD on their cloud service and you can use a custom image on the dedicated servers.
Vultr also has support for OpenBSD
Works well too. We've moved a lot of boxes from local to them. I do wish their console supported cut and paste on a Mac.
Netflix uses freeBSD does that count?
https://www.youtube.com/watch?v=KP_bKvXkoC4
Virtually everyone uses bits of OpenBSD. Android uses a lot of its C standard library, LLVM uses its regular expression library, and Windows 10 uses OpenSSH with LibreSSL.
Armorlogic, Calyptix Security, GeNUA, RTMX, and .vantronix make network appliances based on the complete OS.
Armorlogic, Calyptix Security, GeNUA, RTMX, and .vantronix make network appliances based on the complete OS.
am I the only one who prefers a non-framebuffer console?
I hate the framebuffer console on most of the systems I have.
In some cases, the font is way too small. Changing the font size is confusing, because it isn't clear what size of text you will end up with.
And on some of the systems I have, the framebuffer scrolls so slow!!!!! Drives me up a wall. If it takes more than a tiny fraction of a second to display an entire screen of text, that is too slow!!!! Hate it, hate it, hate it.
In some cases, the font is way too small. Changing the font size is confusing, because it isn't clear what size of text you will end up with.
And on some of the systems I have, the framebuffer scrolls so slow!!!!! Drives me up a wall. If it takes more than a tiny fraction of a second to display an entire screen of text, that is too slow!!!! Hate it, hate it, hate it.
Its not even an option on newer UEFI machines, all consoles are a framebuffer.
Let me rephrase, I don't want a huge screen with tiny fonts for my console, the default should be at most 132x48
Interesting, I didn't know that. Thanks!
Not the only one, but one of the few.
glances at the VT-320 hooked to the OBSD firewalls
glances at the VT-320 hooked to the OBSD firewalls
I have a WY-185 sitting around, I feel like I should set it up again, on a terminal server maybe
It's a moral imperative. :-)
An elegant tool for a more civilized age.
its also alot of effort and consumes more of my limited desk space ;-)
What is desktop space compared to having a proper console? Do it!
I'm personally after a small HP terminal that was nicknamed the "munchkin" for its diminutive size. My chances of finding one that can be shipped here are slim, but from the US you could have a better chance.
I'm personally after a small HP terminal that was nicknamed the "munchkin" for its diminutive size. My chances of finding one that can be shipped here are slim, but from the US you could have a better chance.
Almost certainly not, but on the other hand there are people who conversely enjoy proper boldface and italics, as Unix text processing tools encompassed pretty much from the get-go; or more than 256 different glyphs in use at once, with none of that faffing about with loadable Unicode-to-8-bit character maps; or MouseText. (-:
Hopefully not too off-topic, I have a question: shouldn't we stop using terminal emulators already (i.e. why do we still require emulating the terminal hardware designed in 1970)? Wouldn't a text editor-like user interface be more sensible? (Perhaps, something similar to emacs' M-x shell but more universally adopted.)
I want to note kmscon, it's really distraction free terminal. Best of all it easily supports unicode points with no added user effort.
While it's not quite related to consoles, I've noticed that the non-GUI interface to a lot of operating systems often don't get a lot of love. Linux has had a fairly nice boot scroll, but I think part of that is because it's become an icon and as such is part of the public face of the OS; but on macOS for example it's an tiny awful experience because the text doesn't scale to 2x (you'd think Apple would be the first one to fix this…) and the output isn't well formatted at all–it's just the various subsystems spewing text as they start up. And a lot of modern systems use kind of old fonts that in many cases seemed to be picked as an afterthought…